Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study
Authors: Kaustubh Ponkshe
Shaan Shah
Raghav Singhal
Praneeth Vepakomma
Venue: ICLR 2026 Poster (2026)
Keywords: Safety
Alignment
Harmful Fine-Tuning
Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We show that safety alignment in LLMs is not confined to distinct subspaces (but rather, highly entangled with general ability directions), thus fundamentally challenging the foundation of subspace-based defenses.

Abstract: Large Language Models (LLMs) rely on safety alignment to produce socially acceptable responses. However, this behavior is known to be brittle: further fine-tuning, even on benign or lightly contaminated data, can degrade safety and reintroduce harmful behaviors. A growing body of work suggests that alignment may correspond to identifiable directions in weight space, forming subspaces that could, in principle, be isolated or preserved to defend against misalignment. In this work, we conduct a comprehensive empirical study of this perspective.
We examine whether safety-relevant behavior is concentrated in specific linear subspaces, whether it can be separated from general-purpose learning, and whether harmfulness arises from distinguishable patterns in activations. Across both weight and activation spaces, our findings are consistent: subspaces that amplify safe behaviors also amplify useful ones, and prompts with different safety implications activate overlapping representations. Rather than residing in distinct directions, we show that safety is highly entangled with the general learning components of the model. This suggests that subspace-based defenses face fundamental limitations and underscores the need for alternative strategies to preserve safety under continued training. We corroborate these findings with multiple experiments on five open-source LLMs from the Llama and Qwen families. Our code is publicly available at: https://github.com/CERT-Lab/safety-subspaces.

Published as a conference paper at ICLR 2026
SAFETY SUBSPACES ARE NOT LINEARLY DISTINCT: A TINE-TUNING CASE STUDY
Kaustubh Ponkshe#!, Shaan Shah*? Raghav Singhal*!, Praneeth Vepakomma" MMohamed bin Zayed University of Artificial Intelligence
University of California San Diego "Massachusetts Institute of Technology
ABSTRACT
Large Language Models (LLMs) rely on safety alignment to produce socially acceptable responses. However, this behavior is known be brittle: further fine-tuning, even on benign or lightly contaminated data, can degrade safety and reintroduce harmful behaviors. growing body of work suggests that alignment may correspond to identifiable directions in weight space, forming subspaces that could, in principle, be isolated or preserved to defend against misalignment. In this work, we conduct a comprehensive empirical study of this perspective. We examine whether safety-relevant behavior is concentrated in specific linear sub- spaces, whether it can be separated from panrrol- learning, and whether harmfulness arises from distinguishable patterns in activations. Across both weight and activation spaces, behaviors also amplify useful ones, and prompts with different safety implications activate overlapping representations. Rather than residing in distinct directions, we show that safety is highly entangled with the general learning components of the model. This suggests that subspace-based defenses face fundamental limita- tions and underscores the need for alternative strategies to preserve safety under continued training. We corroborate these findings with multiple experiments on five open-source LLMs from the Llama and Qwen families. Our code is publicly available at: https: / g thub. com Coom/CERT-TTe Labt sa fetty-yyaaeee subspa ces.
findings consistent: subspaces that amplify safe
INTRODUCTION
Large Language Models (LLMs) show strong performance across wide range of general-purpose tasks (1; 51; 60; 61; 63; 70). To ensure these models behave responsibly and align with human values, they undergo additional process of safety alignment. This alignment is typically achieved during the post-training stage, enabling models to improve response quality, and refuse harmful prompts over the pre-trained stage. Despite known jailbreak methods that can bypass safeguards, aligned models are generally considered significantly safer than their base versions (44; 52; 67).
However, this alignment fragile. Since safety is encoded in the model's weights, any modification, such as further fine-tuning (FT), can compromise This exposes a deeper attack surface beyond prompt engineering: an adversary could insert a small number of malicious samples into a training set to subvert alignment (4; 71; 72; 76). Recent work shows that even benign FT, low-rank adaptation, or pruning can degrade a model's safety profile 616; 17; 36; 50; 66). A growing line of research seeks to leverage the learned safety adherence to design defenses against adversarial attacks and interpret ali alignment mechanisms(3; 11; 20; 28; 33; 37; 69).
Fine-tuning (FT) is used to adapt an LLM to new and personalized domains and plays a key role in the widespread adoption of L.LMs across diverse contexts. Preserving alignment this setting, while retaining the improvements achieved through FT, is therefore both practical concern and technically challenging problem. This raises a natural question: Does there exist subspace, whether in weight space activation space, that uniquely encodes safety alignment information without affecting performance? such a property exists, it could, in principle, enable the preservation of safety while maintaining model performance under continued training.
denotes equal contribution. Author order decided randomly.

---

Published as a conference paper at ICLR 2026
To construct defenses, prior works (20; 37) have typically derived safety subspaces using one of two approaches: weight updates from general alignment (aligned-base model deltas) or updates from targeted safety tuning (safety-base model deltas). Our goal is to comprchensively investigate these so-called "safety subspaces" in order to determine whether the information they contain is truly specific safety. If so, we could separate unsafe information from the useful knowledge learned during FT through simple projections, thereby ensuring that our fine-tuned models are both robust to safety and segepperformnn.
To explore this question, we design four experiments probing the geometry of safety-related behavior across both model weights and activations. We begin by analyzing FT updates derived from purely useful and purely harmful datasets. These updates are projected into the candidate "safety subspaces" to test whether harmful updates are more expressive than useful ones within these subspaces. Next, we design an experiment involving contaminated FT, where a small fraction of harmful samples is mixed into an otherwise benign dataset. By projecting updates into the orthogonal complement of the candidate subspaces, we test whether harmful components can be selectively removed. From these experiments, we conclude that the candidate subspaces are not safety-specifie but instead capture general learning. This leads us to ask whether any distinct safety subspace exists at all. To address this, our third experiment performs pairwise comparisons among useful, harmful, and safety updates to determine which pairs share the greatest similarity. Surprisingly, the similarity between harmful and safety updates is never the highest, as one might expect, and is sometimes even the lowest. Finally, our fourth experiment, we extend this analysis activation space, examining whether safety-specific attributes are distinguishable in activations rather than weights.
Across all experiments, we find no evidence that any linear mubspace-whether in weight or activation space-captures spfety-specific behavior in isolation. Although certain subspaces, such as those derived from the principle components of alignment or safety-specinn both safe and useful behaviors alike, indicating that safety is deeply entangled with general learning. Similarly, activations from harmful and helpful prompts occupy overlapping regions of activation space, providing no evidence for distinct safety-related regions. Together, these findings reveal fundamental limitation of linear subspace-based strategies. Since safe and harmful behaviors cannot be cleanly separated linearly, then projection- or filtering-based defenses are unlikely to suppress harmfulness without incurring comparable losses in utility. Our key contributions are as follows:
updates, are impactful, they amplify
We show that subspaces derived from alignment and safety- -afety-specific updates are not uniquely tied to safety; instead, they amplify both useful and harmful behaviors alike, implying that safety is deeply entangled with general learning (Section 3).
We demonstrate that safety and harmful updates share no relatively significant subspace overlap, confirming that no region of weight space can be isolated specifically for safety (Section 5).
Finally, we reaffirm this hypothesis in activation space, showing that harmful prompts do not activate distinct linear regions We observe that safety and harmful updates do not exhibit relatively higher within-task subspace overlap than cross-task comparisons, providing no evidence for weight-space region that isolates safety from general learning signals (Section 6).
Across multiple experiments on five open-source LLMs from the Llama and Qwen families, we consistently observe patterns inconsistent with linear separability of safety alignment, pointing to inherent limitations in subspace-based defenses.
2 PRELIMINARIES
Notation. Let Wo denote the parameters of the base model and WA that of the aligned model. We denote the parameters of the model after safety tuning, i.e., fine-tuning specifically for harmless responses and refusals, using Ws. We further fine-tune the aligned and the safety-tuned models on task-specific dataset Dj, where j E { (Useful, Harmful, Contaminated). resulting in parameters W WFT.. We decompose the total parameter update as the sum of two components:
A : WA Wo A = WFT, Wa/Ws (task-specife updates)
As : Ws Wo
(alignment/safet--speile updates) (1)
Importance of Alignment Update x. Alignment training typically emphasizes behavioral
properties such harmlessness, helpfulness, and honesty. Empirical studies (11; 20; 44) suggest
2

---

Published as a conference paper at ICLR 2026
STEP1
STEP II
Alignment/ Safety-Tuning
Fine-Tuning
Projection
+ Uretul
W Unudaal
W!
Aligned Safaty- Tuned LLM
Allgned Safety-
Base LLM
Tuned
+P, Hemmtul
W
A/S
W Itarsett
W
W Hanntal
A A/8
A/8
W
Contaninated
Conteminnted
W Crataminated
P..
4 = =
WAs
Figure 1: The base model Wo is aligned/safety-tuned to produce the model W Ways- Step 1: The difference A A/S W W. defines an seaggnmenn/saftysspeeeiiiee direction, from which projection matrices Ps (top-K subspace) and Pk (orthogonal subspace) are derived. WAys is then fine-tuned on three datasets: helpful, harmful, and contaminated, yield W Wuseful. W harmful- and W contaminnated with updates Δt,- Step 2: Project D,, using Pk and P/-, and add back to W A/S to obtain projected models for evaluation. In addition, S'VD is performed the task-specific updates, and the Mode Subspace Overlap (MSO) is computed between the top-K singular vectors.
that the alignment update AA encodes directions in parameter space that are strongly correlated with these safety attributes. This stage is also the sole point in production model training where safety is explicitly introduced into the model. Our goal is to systematically control the extent to which the subsequent task-specific update A interacts with these alignment directions.
Importance of SSpety-Specific Update (As). We also aim to capture safety more directly, disen- tangling from the broader behavioral changes introduced during the alignment stage. Safety tuning focuses explicitly on refusal and harmlessness, without simultaneously shaping general instruction- important detail that we use a distribution for safety tuning that
following ability. subtle is distinct from the one used for harmful fine-tuning, ensuring that our analysis captures genuine interactions rather than artifacts of dataset overlap. Our objective here to systematically analyze how subsequent task-specific updates A interact with these safety directions.
Constructing the Alignment & Safety Subspaces. To construct the alignment and safety sub- spaces, each tensor in the updates A As is first reshaped into matrix (flattened if needed) VA, Vss E RMNN From here on, we use VAys to refer to both VA, Vs. We perform a thin singular value decomposition (SVD) of the form VA/s UEVT, which reveals the principal directions of parameter change (13; 43), ranked by their contribution the Frobenius norm. The top k (Top-K) subspace:
right singular vectors in V are then selected to define the adignmentt/sffety-speees
S == su)n(Uu), UR Mxk
S &arnkn (VA/s).
Intuitively, S₁ captures the l most significant directions of parameter shifts resulting from alignment safety-specific training. The subspaces naturally induce projection operators:
Ps = ULU, P 1 Pk.
(4)
where Pa projects a matrix onto the alignmeet/safeey-speeifi subspace, and Ps onto its orthogonal complement.
Projection Schemes.
Given a fractional rank hyperparameter e E (0, 1]. we determine le min(M,N)) and apply one of two projection- based update schemes to the task-specife update:
Parallel:
P. A
W parallel
= W A/s + Az.
(5)
Orthogonal
A P A
W orthogonal
W A/S + A
(6)
Eqn. 5 retains the update components that align with the candidate safety directions, while Eqn. removes this component, retaining only the update orthogonal to the candidate safety subspace. Figure provides overview of our process.
3

---

Published as a conference paper at ICLR 2026
Control Experiments. To further assess the specificity and effectiveness of the chosen safety subspace, we introduce two control experiments:
Random-K: Instead of using the top-k singular vectors from the SVD of Va,s, we randomly sample & singular vectors from the full set to construct a randomized safety subspace.
Random: We replace Vays with a random matrix of the same dimensions, perform its SVD, and use the top-k singular vectors to define synthetic safety subspace.
Energy-Kept Ratio. We introduce the fractional energy metric to quantify the extent of overlap between the task update and the safety subspace:
& (A = |P
|
Mode Subspace Overlap (MSO). Let E Rdx and W E Rdxnw be two matrices with a shared ambient dimension d but possibly different column counts. We extract their principal directions by taking the thin SVD:
(8) For a chosen energy-retention fraction E (0, 1], we select the smallest ky and kw such that the top ky (resp. kw) ) left singular vectors capture least an r--faction of ||Ev 1| (resp. ||Ew l|?)) This yields orthonormal bases Qy E Rdxkv and Qw E Eaxkw, The overlap matrix is then defined as:
v= UvVVV W Uw
$ == QTQ RAv xkw
To quantify the similarity between these ry-energy subspaces, we use the MSO metric defined as: ||
MSO(V, W;n) min(ky, kky
MSO S
(10)
Intuitively, MSO(V, W) measures the overlap between the top-1) energy components of V and W: it equals 0 for orthogonal subspaces and I for identical spans. As a baseline, the expected overlap between random subspaces of dimensions ky and kw in Rd is given analytically by:
B[overlap] max(kv, kk)
(11)
Models Used. We evaluate base and aligned versions of five open-source LLMs: Llama 3.2 1B (12), Llama 2 7B (63), 2wen-2.5 IB (70), Qwen-2.5 3B, and Qwen-2.5 7B. To obtain safety-tune variants, we fine-tune the base models on the safety-specific BeaverTails dataset (31), using only entries labeled is saaafe
True.
3 Do ALIGNMENT SUBSPACES ENCODE SAFETY?
central question in understanding safety alignment whether specific directions in weight space, such as those defined by the difference between base model and its RLHF-ail RLHF-aligned counterpart, encode information unique to safety. If this is the case, constraining FT updates to lie within these subspaces could provide principled approach to guarding against harmful optimization. We begin our investigation by examining whether task-specific FT updates align differently with the top directions of the alignment and spectf- matrices, depending on whether the task is helpful or harmful.
Experimental Setup. We fine-tune the aligned and safety-tuned models on two distinct datasets. The first is 20K subset of MetaMathQA (75), benchmark of math word problems representing a useful task without safety concerns. The second a 4K unsafe subset of BeaverTails (31), synthetic dataset of harmful instruction instruction-responsee pairs designed to elicit unsafe behavior. The resulting weight updates are denoted as A and A Harmuul respectively. To quantify behavioral effects, we evaluate harmfulness on AdvBench (80). with GPT-4o-mini (27) scoring each response from (least harmful) to (most harmful); the final score is the average across samples. Utility is measured as accuracy on the GSM8k test set (9), based on final-answer correctness. For each setting, compute harmfulness, utility, and the energy-kept ratio for the projected models W parallel and W orthogonal as well as for the base, aligned, fine-tuned, and control models.
4

---

Published as a conference paper at ICLR 2026
Random
Top-K
Random
Aligned
Pine-Tuned
Full Useful
Harmful
Full Useful
Full Harmful
0.61
Quema255 1.58
Energy Kaps
Energyy Kage
Harmful Score
Utility
0.58
1.91
0.17
0.15
0.92
0.00
0.00
0.47
1.55
0.01
0.25
0.50
0.75
0.99
0.01
0.25
0.50
0.75
0.90
0.03
0.25
0.50
0.75
0.09
0.01
0.25
0,50
0.35
0.99
Useful
Harmful
Full Useful
Full Harmful
0.36
Elamos3 1B
Eaeegy Kapt
Harmful Score
0.06
0.23
0.28
Utiilty
Emercs
0.18
0.12
0.20
3,22
0.01
0.00
0.13
2.00
0.01
0.25
0.50
0.75
0.90
0.01
0.25
0.50
0.75
0.90
0.01
0.25
0.50
0.75
0.99
0.01
0.25
0.50
0.75
0.99
Frection SVD Vectors
Figure 2: Parallel projection-basee update schemes across varying SVD fractions. We report the energy-kept ratio for models fine-tuned on Full Useful and Full Harmful data, utility for models fine-tuned on Full Useful, and harmfulness for models fine-tuned on Full Harmful.
Results: Energy Is Uniform Across Subspaces, Performance Is Not. As shown in Figures and 6 (Appendix C), the fraction of energy retained in projected updates increases linearly with subspace rank and is consistent across all three subspace types. This pattern holds for both helpful and harmful updates. We find no evidence that update energy is preferentially concentrated in the top directions of A/S for safe versus unsafe FT. This suggests that if "safety subspace" exists, it is not revealed simply through energetic alignment with the dominant directions of A/S- At the same time, while energy is evenly distributed, behavioral impact is not. Figure 2 and Table show that projecting Liveful onto the top-k directions consistently improves utility relative to random projections with equal energy. Similarly, projecting Harmful onto the same directions increases harmfulness. Thus, the top singular directions of AASS are not uniquely aligned with safety, but they are generally potent: updates along these directions are more effective, whether the goal harmful behavior. Comprehensive results for all models are provided in Table 4 (Appendix C).
enhance utility or to elicit
Table I: Parallel projection- based update schemes across varying SVD fractions. We report the utility for models fine-tuned on Full Useful data, and harmfulness for models tunett on Full Harmful.
Utility (f)
Harmful Score (.)
Model
Method
Aligned
SVD Fractions
FT
Aligned
SVD Fractions
0.01 0.25 0.50 0.75 0.99
0.01
0.25 0.50 0.75 0.99
Top-K
0.47
0.50
0.53 0.55 0.57 0.58 0.61
1.55
1.62
1.80
1.92 1.90 1.97 2.09
Qwen-2.5 1.5B Random-K
0.47
0.49
0.50 0.53 0.56
0.58 0.61
1.55
1.55
1.66
1.78
1.92
2.00 2.09
Random
0.47
0.49
0.50 0.53 0.53
0.56
0.61
1.55
1.56
1.65
1.74
1.83
1.95 2.09
Top-K
0.13
0.14
0.21 0.25 0.30
0.34
0.36
2.80
2.89
3.29
3.51
3.66
3.84 4.07
Llama-3.2
Random-K
0.13
0.13
0.16 0.23 0.29
0.34
0.36
2.80
2.83
3.11
3.37
3.55
3.84 4.07
Random
0.13
0.13 0.17 0.22 0.29
0.34
0.36
2.80
2.81
3.05
3.34
3,56
3.83 4.07
Implications: Alignment Directions Reflect General Learning, Not Safety. This symmetry across tasks is important. The fact that top-k directions amplify both helpful and harmful behaviors equally suggests they do not encode alignment directly. Instead, they represent axes of general parameter sensitivity, ie. directions where updates tend to induce large changes in model behavior. This holds for both alignment and safety-specific updates, implying that disentangling safety offers no clear separation or benefit. In this sense, AA/S /s captures general learning geometry: directions that are highly effective for optimization but not inherently safe. We draw three key takeaways. First, neither helpful nor harmful updates preferentially align with the top subspaces of A or As in terms of energy. Second, these same subspaces are more behaviorally expressive, amplifying both utility and harmfulness depending the task. Third, this challenges the assumption that A A/s encode
5

---

Published as a conference paper at ICLR 2026
safety- information expressed in their top subspaces. Thus, using AAys to constrain updates regulates the magnitude of behavioral change, but not its ethical nature.
4 CAN HARMFUL SUBSPACES BE REMOVED?
Having analyzed helpful and harmful updates in isolation, we contaminated FT. This setting involves adding small fraction of harmful examples to otherwise benign dataset, producing updates that blend both signals. Prior work has shown that even limited contamination can erode safety, causing models to revert to unsafe behaviors (4; 36; 50; 72; 76). While earlier experiments identified expressive subspaces, we now ask the reverse question: can harmful components of update be removed? We test whether filtering specific subspaces, particularly those aligned with the dominant directions of the alignment or safety-tune matrix, can reduce harmfulness while preserving utility.
more realistic scenario:
Random
Top-K
Randon-K
Alignel
Fine-Tuned
Full Useful
Full Harmful
Contaminated
Contaminated
Contaminated
1.00
2.19
Daan2.5 1.58
Energy Kegt
Energy Xegt
Energy Kept
Harmtul Seure
0.86
Uillly
0.56
2.98
0.67
20
0.72
0.50 Useful
0.25
0.75
Harmful
Contaminated
Contaminated
Contaminated
0.99
0.73
Daan2.55 38
Eaergy Kagt
Energy Kept
Energy Ktpt
Hanmfl Score
0.80
0.01
0.82
Uuilit
2.82
0.60
8.64
0.63
3.44
0.46
0.61
0.01
0.50
0.25 0.9
0.01
0:25
0.50
0.75
0:99
0.01
0.25
0.50
0.75
0.89
0.01
025
0,50
0:55
059
0,01
0.35
0.50 0.75
Contamlnated
0,81
Owwinzz5 78
Energy kapt
Energyy Kopt
Enerrty Kopt
Harmaal Score
0..0
0.82
0.82
Uaity
0.79
1.75
0.65
0.44
0.48
0.40
0.01
0.25
0,50
0.75 0.99
0.0L 0.25
0.50
0.75
0:99
0.01
0,25
0.50 0.75 0.59
0.01
0.25
0.50
0.75
0.99
0.01
0.25
0.50
0.75 0.99
Fraction Vectors
Figure 3: Orthogonal projection-based update schemes across varying SVD fractions. We report the energy-kept ratio for models fine-tuned Full Useful, Full Harmful and Contaminated data; and utility and harmfulness for models fine-tuned on Contaminated.
Experimental Setup. We construct contaminated dataset by mixing 20% harmful data from BeaverTails with 80% of the 20K MetaMathQA subset. FT on this mixture produces a single contaminated update, Ar. To suppress harmful behavior, apply the orthogonal projection strategy from Section 2, removing components along the top-k alignment directions. Specifically, we compute Å = P.- AT, where P. projects onto the complement of the aligmment subspace. We then evaluate the resulting models on GSMSK (utility) and AdvBench (harmfulness). Our objective is test whether removing gegnment--lligned components can reduce harmfulness while preserving task performance.
Results: Utility And Harmfulness Drop Together. Figures and 7 (Appendix D) show the effects of orthogonal projection on retained energy, utility, and harmfulness. & increases, meaning more of the update is removed, retained energy declines steadily across all projection types (random, top-k, and random-k). Utility and harmfulness scores (Figure 3, Table 2) follow a similar downward trend. The rate of decline, however, differs by projection strategy. Removing top-k alignment components reduces utility more sharply than random projections, while harmfulness decreases at similar rate. This indicates no selective suppression of harmful behavior. In effect, safety gains come at proportional cost to task performance, with no clear advantage targeting the alignment subspace. Comprehensive results for all models provided in Table 5 (Appendix D)).

---

Published as a conference paper at ICLR 2026
Table 2: Parallel projection-based update schemes across varying SVD fractions. We report the utility
and harmfulness for models fine-tuned on Contaminated data.
Utility (1)
Harmful Score (J)
Model
Method
Aligned
SVD Fractions
FT
Aligned
SVD Fractions
FT
0.01 0.25 0.50 0.75 0.99
0.01
0.25 0.50 0.75 0.99
Top-K
0.47
0.50
0.53 0.55 0.57 0.58 0.60
1.55
1.58
1.65
1.80
1.91
1.92 2.16
Qwen-2.5 1.5B
Random-K
0.47
0.49
0.52 0.53 0.55 0.55
0.60
1.55
1.56
L.62
1.63
1.87
1.92
2.16
Random
0.47
0.49
0.50 0.52 0.52 0.54
0.61
1.55
1.58
1.64
1.68
1.74
1.92
2.16
Top-K
0.61
0.63 0.64 0.65 0.68 0.09
0.73
1.47
1.49
1.58
1.69
1.76
1.83
1.99
Qwen-2.5 3B
Random-K
0.61
0.62 0.64 0.64 0.66 0.69
0.73
1.47
1.45
1.55
1.62
1.65
1.91
1.99
Random
0.61
0.62 0.63 0.64 0.65
0.68
0.73
1.47
1.45
1.50
1.57
1.75
1.83
1.99
Top-K
0.74
0.74 0.75 0.75 0.75 0.78
0.81
1.30
1.31
1.56
1.60
1.68
1.67
1.96
Qwen-2.5 7B
Random-K
0.74
0.74
0.75 0.76
0.75
0.78
0.81
1.30
1.35
1.41
1.46
1.59
1.67
1.96
Random
0.74
0.74 0.75 0.75
0.75
0.78 0.81
1.30
1.34
1.40
1.48
1.56
1.63
1.96
Implications: No Selective Removal Is Possible. These results establish that the top subspaces of alignment or safety-tuned updates do not uniquely encode safety or harmfulness. Removing these directions degrades both utility and harmfulness at similar rates. If harmful behavior were confined to distinct subspaces, we would expect steeper drop in harmfulness than in utility, yet this is not observed. Even if safety-relevant directions exist, they cannot be recovered from the alignment or safety-tuned matrices alone, particularly under contamination. The update blends helpful and harmful objectives, making its projection agnostic to intent. As a result, orthogonal projection fails selectively suppress harmful behavior. Thus, subspace filtering based on alignment directions imposes strict trade-off: improvements in safety come only at proportional cost to utility.
We test whether projecting individual layers can cleanly separate safety and learning directions in Appendix I. Specifically, we repeat the two weight-space analyses described above and observe that no layer provides such separation.
ARE SAFETY WEIGHT SUBSPACES DISTINCT?
MSOIHA, U-A)
MSOJH-A, B-A)
MSDIUAA, B-A)
Random Raseling
Qwen-2.5 1.5B - 1.19
Qwen-2.5 38
125
Owen-2.5 7B
L.19
Ilama-1.2 3.2 IB 111
Ilama-2 7B
-22
0.39
0.37
0.24
0.28
0.16
n19
0.10
0.08
0.00
0.09
0.05
0.00
0.00
0.00
Qwen-2.5 1.5B
123
Qwen-2.5 38
-.30
Qwen-2.5 7B
1.23
Elama-3.2 1B
-113
Ilama-2 7B 1.27
0.39
0.32
0.38
026
0.34
0.29
0.36
0.19
0:12
0.08
0.08
0.05
0.20
0,50
0.20
0.60
0.50
9.20 0.40
0.50
0.01
0.20
0.0
0.80
0,01
0,40
0.60
0.60
Energy Fraction
Figure Mode Subspace Overlap (MSO) at the 70- and 85- percentile layers for pairwise comparisons of the dominant subspaces from Harmful fine-tuned (H), Aligned (A), and Base (B) models.
natural question is whether a dedicated region of parameter space, which we might call a "safety subspace," captures safety-specific behavior. Such a subspace should satisfy two criteria: (i) safety- relevant updates, whether from alignment or harmful FT, should lie predominantly within it; and (ii) task-specific updates unrelated to safety should have minimal overlap, with projections onto the subspace leaving model safety unchanged. Our earlier results argue against the top subspaces of the alignment or safety-tuned matrices meeting these criteria. Nevertheless, it remains open whether some other set of directions, possibly outside these subspaces, could fulfill this role. To investigate this, directly compare the dominant subspaces of different update types.
7

---

Published as a conference paper at ICLR 2026
Experimental Setup. We compare the principal subspaces of three updates: the alignment update Aa (from the base to the aligned model), the harmful FT update A Harmfuul (trained on BeaverTails)) and the useful FT update Af (trained on a 20K subset of MctaMathQA). Notably, the negated alignment update- A reverses alignment by pushing the model back toward its unaligned base state, effectively acting as a harmful update and serving as a useful reference point. We repeat these experiments for safety-tuned updates as well. For given energy threshold 7 E (0. 1]. compute MSO : 1) (Section 2) for three pairs: (i) (A AT )armful), to assess whether helpful and harmful FT affect similar subspaces; (ii) (4/ [sffefll AA), : to test the relationship between helpful updates and reversed alignment; and (iii) HHHarmful - to compare two harmful directions. We sweep over 1, with smaller values isolating high-energy directions and larger values approaching full-rank overlap. As a baseline, we include the random-subspace expectation max (ky, kiw) )dd; values above this baseline indicate significant geometric alignment, while values near it suggest chance-level overlap.
Results: Representations Overlap Across Tasks. Figures 4 and 8 (Appendix E) show the pairwise overlap between the dominant subspaces (top-k directions) of each update.
We report per-layer results in Figures 10, 11, 12, 13, and 14 (Appendix G), all of which show results consistent with our observations.
All pairs exhibit greater overlap than random baselines, indicating shared structure. However, in Figure 4, the strongest overlap is between the useful and harmful updates, rather than between alignment and harmful updates, one might expect if safety were shared component. This is key finding. If a safety subspace existed, it would likely appear in the shared directions between alignment and harmful updates (or between safety and harmful updates), which affect safety in opposite ways. The absence of such overlap suggests that no consistent, linear safety-specific subspace exists. For safety-tuned models (Figure 8), the strongest overlap occurs between the useful and safety-specific updates, an even more counterintuitive result. This implies that, in terms of subspace overlap, the useful update lies closer to the safety-specific update than the harmful update does. Strikingly, this overlap is much larger than that between the harmful and safety-specite updates, even though, semantically, one might expect the latter to be most similar.
Implications: Shared Weight Subspaces Drive Behavior, Not Safety. Taken together, our results suggest that safety-relevant updates do not reside well-defined or isolatable subspace. Instead, alignment, safety, and harmfulness operate over complex, task-dependent directions. The strong overlap between harmful and helpful update subspaces indicates that these directions form a general learning subspace, expressive across tasks but agnostic to safety. Thus, we find no evidence of distinct safety subspace, and linear subspace methods cannot cleanly isolate safety in parameter space. This highlights fundamental limitation in subspace-based based defences: attempts to filter relevant components suppress general learning as well.
We repeat all of our weight-space analyses for a model safety-tuned using only Direct Preference Optimization (DPO) in Appendix H, to assess whether our findings remain consistent across different safety training strategies. We find that all our results transfer cleanly this setting.
6 Do SAFETY SUBSPACES EXIST IN ACTIVATION SPACE?
So far, our analysis has focused on weight space, probing whether certain update directions encode salety-related behavior. Finding no evidence of distinct safety subspaces at the parameter level raises a final question: do selety-relevant inputs elicit distinct activation patterns, even when their weight updates overlap? Although weight updates may distribute broadly, inputs might still selectively activate specific directions. We investigate this possibility in the following section.
Experimental Setup. We compare internal activations induced by different prompt categories. Specifically, we pass useful (benign) prompts from the MATH dataset (19) and harmful prompts from BeaverTails (test set) and ToxiGen (15) through three models: the aligned model, the useful fine-tuned model, and the contaminated fine-tuned model. For each prompt, we record the hidden state of the last generated token (35) at each transformer layer E O..... L. At each layer, these hidden states are stacked into activation matrices of shape R where d is the hidden size and is
8

---

Published as a conference paper at ICLR 2026
Fully Useful
Conteminated
Aligned M50(U, H2)
Random Baseline
MSOrU, H1)
MSO H2)
0.82
0.82
0.86
Qwen-2 2.5 3B
0.61
0.62
0.64
0.41
0.41
0.43
0:20
0.21
0.21
0.00
0.00
0.00
Qwen-225 7B MSO
0.01
0.00
0.01
0.60
0.01
0.30
0.60
MSOIU, I1I)
MSOIU, 112)
MSO
0.70
0.91
0.85
0.53
0.68
0.64
0.35
0.45
0.43
0.18
0.23
0.21
0.00
0.00
0.00
0.01
0.30
0.60
0.90
0.01
0.30
0.60
0.90
0.01
0.30
0.60
0.90
Energy Fraction
Figure 5: Average Mode Subspace Overlap (MSO) across layers in the 65-90% depth range for
pairwise comparisons of activations from Useful (U) and multiple Harmful (HI, H2) prompt sets.
the number of prompts (5000 per dataset). We then compute MSO (see Section 2) between activation matrices from different datasets, sweeping over energy thresholds 1/. Smaller values of 1) capture high-energy activation modes, while larger values approximate full-rank comparisons. We plot MSO curves alongside the random-subspace baseline max(kUsetul) Harmft)/ /d and report averages over layers in the 65-90% depth percentile.
Results: Representation Subspaces Overlap Across Tasks. Figure shows MSO values across all pairs of prompt categories. Useful and harmful prompts consistently exhibit overlap above the random baseline, indicating activation of shared high-energy subspaces in activation space. Interestingly, the overlap between the two harmful prompt sets is not always greater than their overlap with useful prompts; in some cases, the useful-harmful overlap exceeds the harnful-harmful overlap. The degree of overlap also varies across model configurations. Some models show strong alignment even in the top subspaces, while others exhibit more gradual increases, with overlap becoming significant only at higher energy thresholds. This variability suggests that representational similarity depends more on model-specific factors than on the safety content of the prompts. Additional results on other models are provided in Figure (Appendix F). We report additional analysis experiments Appendix K.
Implications: Shared Activation Subspaces Drive Behavior, Not Safety. These observations suggest that while all prompt types activate shared subspaces more than expected by chance, there no evidence of distinct safety-violating subspace. If such a subspace existed, activations from harmful prompts would consistently show greater mutual overlap than with useful prompts, which not observed. Instead, prompts with different safety implications are processed through broadly overlapping representations. This supports our earlier hypothesis: the directions most responsible for behavior correspond to general-purpee purpose representational subspaces rather than safety-specific ones. These directions are activated across tasks and prompt types, indicating that LLMs do not internally separate "safe" and "unsafe" activation modes but instead rely on shared, high-impact subspaces. We therefore find evidence of distinct safety subspace even in activation space. Together with our weight-space results, this suggests that both aligned and harmful behaviors arise from shared representational mechanisms rather than separable subspaces.
CONCLUSION
This work set out to investigate how safety alignment is encoded LLMs and whether it can be isolated in weight or activation space. Our findings challenge the common assumption that alignment or safety-specific updates correspond to unique "safety subspaces". Subspaces with strong behavioral impact are not unique to safety; rather, they amplify both utility and harmfulness, indicating that safety is deeply entangled with general learning. Similarly, harmful and useful prompts activate overlapping regions of activation space, providing no evidence for distinct safety subspace. Together,

---

Published as a conference paper at ICLR 2026
these results establish that safety alignment is not linearly separable in LLMs. While this complicates the development of subspace-bar defenses, it also highlights the potential of high-impact directions, if appropriately constrained, for guiding both safe fine-tuning and activation- control. More broadly, our work calls for rethinking assumptions interpretability and alignment research, and for developing methods that explicitly account for the entangled nature of representations.
REPRODUCIBILITY STATEMENT
We place a strong emphasis on reproducibility. To this end, we make our implementation publicly available at https: / thub. com/ CERTTLab// http//aatt--e/.// subspaces. A complete descrip- tion of the experimental setup is provided throughout the paper in Sections 3, 4, 5 and 6, with additional hyperparameter details Appendix B. All experiments are conducted on widely used, publicly available benchmark datasets, which we summarize in Appendix M.
ACKNOWLEDGEMENT AND DISCLOSURE OF FUNDING
This work received support from Mohamed bin Zayed University Artificial Intelligence (MBZUAI) and was partially funded by the ADIA Lab Fellowship.
REFERENCES
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[2] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. Advances in Neural Information Processing Systems, 37: 37:136033-136083 2024.
[3] Rishabh Bhardwaj. Due Anh Do, and Soujanya Poria. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 14138-14149 14149, 2024.
[4] Rishabh Bhardwaj and Soujanya Poria. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. arXiv preprint arXiv:2310. 143003, 2023.
[5] Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07873 2023.
[6] Jianhui Chen, Xiaozii Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. Finding safety neurons in large language models. arXiv preprint arXiv:2406. 14/44, 2024.
[7] Zehua Cheng, Manying Zhang, Jiahao Sun, and Wei Dai. On weaponization-resistan large language models with prospect theoretic alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10309-10324, 2025.
[8] Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. Safety-aware fine-tuning of large language models. arXiv preprint 2tt::2440..0011111 2024.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1668.. 2021.
14168,
[10] Xander Davies, Eric Winsor, Tomek Korbak, Alexandra Souly. Robert Kirk, Christian Schroeder de Witt, and Yarin Gal. Fundamental limitations in defending Ilm finetuning apis. arXiv preprint arXiv:2502 /4828, 2025.
10

---

Published as a conference paper at ICLR 2026
[11] Aladin Djuhera, Swanand Ravindra Kadhe, Farhan Ahmed, Syed Zawad, and Holger Boche. Safemerge: Preserving safety alignment in fine-tuned large language models via selective layer-wise model merging. arXiv preprint arXiv:2503 2025.
[12] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
[13] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936.
[14] Francisco Eiras, Aleksandar Petrov, Philip Torr, M. Pawan Kumar, and Adel Bibi. Do as i do task-speciis specific fine-tuning risks in large language models. In The Thirteenth
(safely): Mitigating task International Conference on Learning Representations, 2025.
[15] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09500, 2022.
[16] Will Hawkins, Brent Mittelstadt, and Chris Russell. The effect of fine-tuning on language model toxicity. arXiv preprint arXiv:2410.1582.1 2024.
[17] Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099, 2024.
[18) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[19] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 103. 038774 2021.
[20] Chia-Y Yi Hsu, Yu-Lin Tsai, Chih-Hsun Hsun Lin, Pin-Yo Chen, Chia-Mu Yu, and Chun- Ying Huang. Safe lora: The silver lining of reducing safety risks when finetuning large language models. Advances Neural Information Processing Systems, 37:65072-65094, 2024.
[21] Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint arXii:2408.00660 2024.
[22] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.01586, 2024.
[23] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey, arXiv preprint arXiv:2409.18166 2024.
[24] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lazy safety align- ment for large language models against harmful fine-tuning. arXiv preprint arXiv:2405.18641 2, 2024.
[25] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Virus: Harmful fine-tuning attack for large language models bypassing guardrail moderation. arXiv preprint arXiv:2501.1743. 2025.
[26] Yao Huang, Huanran Chen, Shouwei Ruan, Yichi Zhang, Xingxing Wei, and Yinpeng Dong. Mitigating overthinking in large reasoning models via manifold steering. arXiv preprint arXiv:2505.224111 2025.
[27] Aaron Hurst. Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
11

---

Published as a conference paper at ICLR 2026
[28] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04082 2022.
[29] Samyak Jain, Ekdeep S Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet Dokania. What makes and breaks safety fine-tuning? a mechanistic study. Advances in Neural Information Processing Systems, 37:93406-93478, 2024.
[30] Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, al. Pku-saferlhff Towards multi-level safety alignment for Imss with human preference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2025.
[31] Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a -unae--reeeeence dataset. Advances in Neural Information Processing Systems, 36:24678 24704, 2023.
[32] Joshua Kazdan, Lisa Yu, Rylan Schaeffer, Chris Cundy, Sanmi Koyejo, and Krishnamurthy Dvijotham. No, of course i can! refusal mechanisms can be exploited using harmless fine-tuning data. arXiv preprint 2502 /9537, 2025.
[33] Connor Kissane, robertzk, Arthur Conmy, and Neel Nanda. Open source replication of an- thropic's 's crosscoder paper for model-diffigg AI Alignment Forum post, Oct 2024. Accessed 2025-05-15.
[34] Bruce W Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional activation steering. arXiv preprint arXiv:2409. 099000900099..... 2024.
[35] Chak Tou Leong, Qingyu Yin, Jian Wang, and Wenjie Li. Why safeguarded ships run aground? aligned large language models' safety mechanisms tend to be anchored in the template region. arXiv preprint arXiv:2502. / 13946, 2025.
[36] Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint UrXiv:2310.206:3 2023.
[37] Mingjic Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. Salora: Safety- alignment preserved low-rank adaptation. arXiv preprint arXiv:2501.01765 2025.
[38] Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to LLM security. The Thirteenth International Conference on Learning Representations, 2025.
[39] Guozhi Liu, Weiwei Lin, Qi Mu, Tiansheng Huang, Ruichao Mo, Yuren Tao, and Li Shen. Targeted vaceine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. IEEE Transactions on Information Forensics and Security, 2025.
[40] Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. Robustifying safety- large language models through clean data curation. arXiv preprint arXiv:2405. 2024.
[41] Uyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100,.22 2017.
[42] Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Keeping Imms aligned after fine-tuning: The crucial role of prompt templates. Advances in Neural Information Processing Systems, 17:118803-118631 2024.
[43] Leon Mirsky. Symmetric gauge functions and unitarily invariant norms, The quarterly journal of mathematics, 11(1):50-59, 1960.
[44] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
12

---

Published as a conference paper at ICLR 2026
[45] Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, and Xiaohua Jia. The hidden dimensions of Ilm alignment: A multi-dimennional analysis orthogonal safety directions. arXiv preprint arXtv:2502.09674 2025.
[46] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performnce deep learning library. Advances in neural information processing systems. 32, 2019.
[47] Sheng Y Peng. Pin-Yo Yu Chen, Matthew Hull, and Duen H Chau, Navigating the safety land- scape: Measuring risks in finetuning large language models. Advances in Neural Information Processing Systems, 37:95692-9577 2024.
[48] Samuele Poppi. Zheng-Xin Yong. Yifei He, Bobbie Chern, Han Zhao, Aobo Yang, and Jianfeng Chi. Towards understanding the fragility of multilingual Ilms against fine-tuning attacks. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2358-2372. 2025.
[49] Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just few tokens deep. arXiv preprint arXiv:2406. 05946, 2024.
[50] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Ye Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprini arXiv:2310.03092 2023.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8763. PmLR, 2021.
[52] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53741, 2023.
[53] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances neural information processing systems, 01153728-537444 2023.
[54] Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz, et al. Representation noising: A defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636- 12676, 2024.
[55] Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 5377-5400, 2024.
[56] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[57] Harethah Abu Shairah, Hasan Abed Al Kader Hammoud, Bernard Ghanem, and George Turkiyyah. An embarrassingly simple defense against I1m abliteration attacks. arXiv preprint arXiv:2505.19056 2025.
[58] Han Shen, Pun-Yu Chen, Payel Das, and Tianyi Chen. Seal: Safety-enhanced aligned Ilm fine-tuning via bilevel data selection. arXiv preprint arXiv:2410.07471, 2024.
[59] Rishub Tamirisa, Bhrugu Bharathi. Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang. Ron Arel, et al. Tamper-resistant safeguards for open-weigee lmss. Xrxiv preprint 24Xiv:2408.0076/ 2024.
13

---

Published as a conference paper at ICLR 2026
[60] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Caan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 1rXiv:2312.11803 2023.
[61] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19900 19786, 2025.
[62] Rachel SY Teo, Laziz U Abdullaev, and Tan M Nguyen. The blessing and curse of dimensional- ity in safety alignment. arXiv preprint 2rXiv:257.20033: 2025.
[63]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshomn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylow, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[64] Jiongxiao Wang. Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. Backdooralign: Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[65] Yibo Wang, Tiansheng Huang. Li Shen, Huanjin Yao, Haotian Luo, Rui Liu, Naiqiang Tan, Jiaxing Huang, and Dacheng Tao. Panacea: Mitigating harmful fine-tuning for large language models via post-finettuning perturbation. arXiv preprint 2501:22000188000 2025.
[66] Boyi Wei, Kaixuan Huang, Yangsibo Huang. Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint 240.0:220002001222, 2024.
[67] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXivv2109.01652. 2021.
[68] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing:
system demonstrations, pages 38-45, 2020.
[69] Di Wu, Xin Lu, Yanyan Zhao, and Bing Qin. Separate the wheat from the chaff: A post-hoc approach to safety re-alignment for fine-tuned language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1110-122,, 2025.
[70] An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[71] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint NXXi:2331000244: 2023.
[72] Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. On the vulnerability of safety alignment in open-access LLMs. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 9236-9260, Bangkok, Thailand, August 2024. Association for Computational Linguisties.
14

---

Published as a conference paper at ICLR 2026
[73] Xin Yi, Shunfan Zheng, Linlin Wang, Gerard de Melo, Xiaoling Wang. and Liang He. Nlsr: Neuron-level safety realignment of large language models against harmful fine-tuning, In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 25706-25714, 2025.
[74] Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. A safety realignment framework via cobspacc--oreetted model fusion for large language models. Knowledge-Bau -assd Systems, 306: 127011 2024.
[75] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12
2023.
[76] Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing RLHF protections in GPT-4 via fine-tuning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 681-687, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[77] Wenxuan Zhang, Philip HS Torr, Mohamed Elhoseiny, and Adel Bibi. Bi-factorial preference op- timization: Balancing halfty-helpfullnss in language models. arXiv preprint arXiv:2408. 15313, 2024.
[78] Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. Understanding and enhancing safety mechanisms of Ilms via aafety-specffic neuron. In The Thirteenth International Conference on Learning Representations. 2025.
[79] Minjun Zhu, Linyi Yang, Yifan Wei, Ningyu Zhang, and Yue Zhang. Locking down the finetuned iims safety. arXiv preprint arXiv:2410. 10343, 2024.
[80] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:230711504. 2023.
15

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
A Related Work
16
Experimental Details
18
Do Alignment Subspaces Encode Safety?
18
Can Harmful Subspaces Be Removed?
19
E Are Safety Weight Subspaces Distinct?
19
F Do Safety Subspaces Exist in Activation Space?
20
Can Safety Weight Subspaces Be Distinct Layerwise?
20
H Effect of Safety-Tuning via Direct Preference Optimization
20
I Do Safety and Learning Decouple at Any Layer?
22
J Robustness Checks: General Ability and Over-Refusal
22
J.1 Assessing General Ability Confounds via MMLU
22
J.2 Ruling Out Over-Refusal as a Confounding Factor
23
K Additional Activation Space Analyses
23
K.1 Activation Space Analysis Across All Layers
23
K.2 Impact of Using Early Token Windows
23
K.3 Impact of Using Refusal Prompts
27
L Distinction from Activation Steering Methods
27
M Dataset Details
34
Limitations
34
Use of Large Language Models
34
A RELATED WORK
Safety Alignment and Task-Specific Fine-Tuning in LLMs. Large Language Models (LLMs) do not inherently follow instructions and often exhibit socially undesirable behaviors. To address this, various post-training methods, instruction-tuning and reinforcement learning from human feedback, applied to align base LLMs with human values and improve their cnatucttoonfollowinn capabilities (44; 52; 56; 67). However, studies have shown that fine-tuning these aligned models on harmful data can undo this alignment, restoring their original, socially unacceptable behaviors (71). This
16

---

Published as a conference paper at ICLR 2026
unalignment phenomenon has been demonstrated in both open-source models (36; 72) and proprictary models (4; 50; 76) via publicly available fine-tuning APIs, thereby exposing a new attack surface (10; 25; 32). Moreover, even fine-tuning on benign downstream tasks can degrade alignment (16; 17).
Defense Methods. To safeguard aligned LLMs against unalignment during fine-tuning, defenses have been proposed at three stages of the pipeline: the alignment stage, the fine-tuning stage, and the post-processing stage. The effectiveness of these defense methods is evaluated using downstream model utility and harmfulness (23).
Alignment Stage Defenses. Aligmment stage defenses update the initial instruction-tuning process to ensure that downstream fine-tuning cannot easily overwrite the model's safety behavior. One approach augments the alignment loss, making harmful representations harder to recover during fine-tuning updates (54). Another line of work relies safety-oriented data curation to preserve alignment under downstream -uentnnnng(4 Adversarial and meta-learning techniques have also been combined to develop tamper-resiee renisaaresistant methods that prevent harmfulness while maintaining task performance (59). A separate strategy introduces a regularization term to the alignment loss, which has been shown to preserve safety after fine-tuning (22). Perturbing safety-critical layers during instruction-tumn tuning has also been shown to protect alignment (39). Additional work traces unalignment to excessive dependence on menximum-likelihoon of this method (7). A study on "shallow alignment" also shows that instruction-uus tuning influences only the first few output tokens, whereas deeper alignment improves robustness (49).
training. motivating an integrity preserving variant
Fine-Tuning Stage Defenses. Fine-tuning stage defenses modify the tine-tuning process to ensure that the model's alignment is preserved after update. One class of defenses focuses on data curation, augmenting the fine-tuning dataset to maintain alignment after update (5; 14). Another approach uses safety examples prefixed with secret prompt, which act as backdoor triggers to reactivate safe behavior after fine-tuning (64). A data ranking based strategy has also been proposed, where low-quality data is down-ranked and high-quality data is up-ranked to better preserve safety (58). It has also been shown that prompt templates play an important role; removing the safety prompt during fine-tuning and reintroducing it at inference time can maintain alignment (42).
Optimization based defenses are another type of fine-tuning stage defenses. One line of work splits fine-tuning into an alignment phase and utility phase, safeguarding both safety and task performance (24). Another approach combines safety and helpfulness objectives into a single loss (77).
Parameter level methods also be used preserve safety. One strategy identifies safety neurons and updates only those parameters (78). Another approach involves localizing safety layers and freezing their gradients, which has been shown to prevent unalignment (38). Another line of work explores constraining parameter changes to directions orthogonal to existing safety features, showing that this method preserves alignment (37). It has also been shown that harmful data can be filtered by matching fine-tuning embeddings against the top-k singular vectors of an activation matrix generated using a harmful dataset (8).
Post-Processin Processing Stage Defenses. Post-processing stage defenses adjust the fine-tuned model to restore alignment and preserve usefulness. One approach adds safety vector, defined as the difference between aligned and unaligned weights, to the fine-tuned parameters to regain safe behavior (3). Another line of work projects the fine-tuning update onto the alignment vector when their similarity drops below a threshold, or selectively merges layers from the fine-tuned and aligned models under the same criterion to achieve similar effect (11; 20). A third strategy removes parameters identified as harmful after fine-tuning to restore alignment (21). It has been shown that safety directions in attention-head activations can also be located and used for targeted intervention (79) to realign the fine-tuned model. Another method detects update parameters whose signs contradict the original alignment and removes them (69). Additional work restores/finds safety-critice neurons (6; 73), fuses aligned and fine-tuned models (74), or adds an optimized post-hoe perturbation to recover alignment (65).
Safety Mechanisms in Fine-Tuned and Aligned LLMs. Recent studies have examined how LLMs express safety over neurons, layers, and activations. One study finds that safety related information is language agnostic, identifies parameters whose modification affects alignment, and shows that
17

---

Published as a conference paper at ICLR 2026
Random
Top-K
Random- K
Safeny-Tuned
Pinc- Tuned
Useful
Harmful
Useful
Full Harmful
Elamoct 1B
Energyy Kapt
Harmful Score
Utility
Energy
3.54
0.00
3.41
0.01
0.25
0.50
0.75
0.99
0.01
0.25
0.50
0.75
0.90
0.03
0.25
0.50
0.75
0.09
0.01
0.25
0,50
0.25
0,99
Useful
Harmful
Full Useful
Full Harmful
Harmful Score
Liamo 2 78
Eawrgyy Kagt
0.11
Utility
Emerps
2.64
0.01
0.01
0.25
0.50
0.75
0.90
0.01
0.01
0.25
0.50
0.75
0.90
0.02
0.01
0.25
0.50
0.75
0.99
3.54
0.01
0.25
0.50
0.25
0.99
Frection SVD Vectors
Figure 6: Parallel projection-basee update schemes across varying SVD fractions. We report the energy-kept ratio for models fine-tuned on Full Useful and Full Harmful data, utility for models fine-tuned on Full Useful, and harmfulness for models fine-tuned on Full Harmful.
freezing these parameters during fine-tuning does ensure safety (48). Another line of work locates sparse regions in parameter space whose removal weakens alignment, and likewise observes that freezing these regions alone is insufficient to maintain model alignment (66). A separate analysis maps a safety basin in weight space, noting that random perturbations inside the basin leave safety intact, whereas fine-tuning moves weights outside it (47). Work on the activation residual stream isolates refusal direction, removing this direction prevents refusal to harmful prompts, while adding triggers refusal to benign ones (2). Other work shows that safety models governed by multiple directions in residual stream space, and that refusal rates drop when prompts avoid tokens activating these directions (45). Finally, study shows that safety fine-tuning minimally adjusts MLP weights by pushing unsafe inputs into the weights' null space, leading models to process adversarial prompts as safe (29).
B EXPERIMENTAL DETAILS
We implemented all experiments using Py (46) and the HuggingFace Transformers library (68). We ran all experiments single NVIDIA A6000 GPU (48 GB). To save memory, all base models are initalized in torch. bfloat1 precision. All models are trained using the AdamW W optimizer (41). Detailed hyperparameter configurations for full fine-tuning (and safety-tuning) each model are presented in Table 3.
Table Hyperparameter settings for fine-tuning the various models.
Optimizer
AdamW
Batch size
Max. Seq. Len
512
Grad Acc. Steps
32
Epochs
Learning Rate
10
LR Scheduler
Cosine
Warmup Ratio
0.02
C Do ALIGNMENT SUBSPACES ENCODE SAFETY?
We provide additional results in Figure 6 and Table 4 support the analysis presented in Section 3.
18

---

Published as a conference paper at ICLR 2026
Table 4: Parallel projection-base bassed update schemes across varying SVD fractions. We report the utility for models fine-tuned on Full Useful data, and harmfulness for models fine-tuned on Full Harmful.
Utility (t)
Harmful Score (L)
Model
Method
SVD Fractions
SVD Fractions
0.01
0.25
0.50
0.75
0.99
0.01
0.25
0.50
0.75
0.99
Base
0.21
3.27
Aligned
0.47
1.55
Fine-Tuned
0.61
2.09
Qwen-2.5 1,5B
Top-K
0.50
0.53
0.55
0.37
0.58
1.62
1.80
1.92
1.90
1.97
Random-K
0.49
0.53
0.58
1.92
Random
0.49
0.50
0.53
0.53
0.56
1.56
1.65
1.74
1.83
1.95
Base
0.03
4.13
Aligned
0.13
2.80
Fine-Tuned
0.36
4.07
Llama-3.2 IB
Top-K
0.14
0.21
0.25
0.30
0.34
2.89
3.29
3.51
3.66
3.84
Random-K
0.13
0.16
0.23
0.34
3.11
3.37
3.55
3.84
Random
0.13
0.17
0.22
0.29
0.34
2.81
3.05
3.34
3.56
3.83
Base
0.026
4.13
Safety-Tuned
0.032
3.41
Fine-Tuned
0.075
3.79
Llama-3.2
Top-K
0.026
0.026
0.037
0.041
0.039
3.47
3.55
3.63
3.71
3.69
Random-K
0.031
0.026
0.033
0.037
0.042
3.46
3.46
3.54
3.66
3.69
Random
0.031
0.026
0.030
0.038
0.039
3.44
3.48
3.54
3.61
3.66
Base
0.44
2.53
Aligned
0.61
1.47
Fine-Tuned
0.72
2.16
Qwen-2.5
Top-K
0.63
0.64
0.65
0.68
0.69
1.48
1.71
1.81
1.91
1.92
Random-K
0.62
0.63
0.64
0.65
0.69
1.44
1.55
1.62
1.74
1.91
Random
0.62
0.63
0.64
0.65
0.68
1.50
1.66
1.75
Base
0.69
1.90
Aligned
0.74
1.30
Fine-Tuned
0.81
2.12
Qwen-2.5 7B
Top-K
0.72
0.74
0.76
0.77
0.77
1.34
1.56
1.66
1.76
1.84
Random-K
0.73
0.75
0.74
0.75
0.77
1.34
1.44
1.53
1.64
1.84
Random
0.74
0.75
0.75
0.76
1.40
Base
0.05
4.27
Aligned
0.20
1.74
Fine-Tuned
0.30
3.41
Llama-2 7B
Top-K
0.21
0.24
0.26
0.28
0.29
1.81
2.34
2.61
2.90
3.15
Random- K
0.20
0.23
0.25
0.28
0.29
1.74
1.91
2.09
2.63
3.13
Random
0.20
0.23
0.25
0.28
2.57
3.03
Base
0.053
4.27
Safety-Tuned
0.024
3.54
Fine-Tuned
0.151
3.84
Llama-2 7B
Top-K
0.030
0.042
0.058
0.082
0.122
3.54
3.67
3.70
3.78
3.79
Random-K K
0.026
0.029
0.043
0.067
0.117
3.55
3.59
3.66
3.75
3.79
Random
0.026
0.033
0.042
0.064
0.112
3.55
3.58
3.68
3.76
3.79
D CAN HARMFUL SUBSPACES BE REMOVED?
Figure and Table presents supplementary results that further substantiate the findings discussed in Section 4.
ARE SAFETY WEIGHT SUBSPACES DISTINCT?
To supplement the analysis in Section 5, report extended results in Figure 8.
19

---

Published as a conference paper at ICLR 2026
Random
Tep-K
Random-K
Safaty-Tumed
Pine-Taned
Full Useful
Fall Harmful
Contaminated
Contaminated
Contaminated
0.07
3.06
Elnmo 38
Emergy Kogt
Eneeggy Keyt
Eneeggy Kopt
Harmful Soure
0.64
0.64
0.54
Utility
R.03
5.56
0.68
6.68
0.68
8:75
0.52
0.75
0:00
0.22
0.30
0.0 b.t 825
0.01
Full Useful
Full Harmful
Contaminated
Contaminated
Contaminated
0.90
0.99
0.39
Elomo 7B
Eaargy Kaps
Evergyy Kepi
Energy Kapt
Hermful Score
0.77
0.71
Uuilyy
0.32
2.77
6.44
0.43
0.00
585
0.16
0.22
0.15
3.51
0.01 0:25
0.50
0.75
0.99
0.01 0.25
0.50
0.75
0:99
0.01
0.25 0.50 0.75 0.89
0.01
0.25
0.50
0.75
0.99
0.01
0.25
0.50
0.75 0.99
Figure 7: Orthogonal projection-basee Fraction of Vectors energy-kept ratio for models fine-tuned Full Useful, Full Harmful and Contaminated data; and utility and harmfulness for models fine-ture fine-tuned on Contaminated.
update schemes across varying SVD fractions. We report the
MSO(H-S, U-S)
MSO(H-S, B-S)
MSO(U-S, B-S)
Random Baseline
Llama-3.2 1B
L12
Llama-3.2 1B
L13
Llama-3.2 1B
L14
0.39
0.29
0.29
0.19
0.19
0.19
0.09
0.10
0.10
0.00
0.00
0.00
MSO
0.01
0.20
0.40
0.60
0.80
0.01
0.20
0.40
0.60
0.80
0.01
0.20
0.40
0.60
0,80
Llama-2 7B
L24
Llama-2
L27
Llama-2 7B
L.28
0.24
0.24
0.24
0.18
0.18
0.18
0.12
0.12
0.12
0.06
0.06
0.00
0.01
0.20
0.40
0.60
0.80
0.01
0.20
0.40
0.60
0.80
0.01
0.20
0.40
0.60
0.80
Energy Fraction
Figure 8: MSO at the 70-, 80-, and 85- percentile layers for pairwise comparisons of the dominant
weight subspaces from Harmful fine-tuned (H), Safety-Tuned (S), and Base (B) models.
F Do SAFETY SUBSPACES EXIST IN ACTIVATION SPACE?
To complement the discussion in Section 6, we include extended results in Figure 9,
G CAN SAFETY WEIGHT SUBSPACES BE DISTINCT LAYERWISE?
To complete the analysis in Section 5, we report per-layer results in Figures 10, 11, 12, 13, and 14. We observe that the strongest overlap is never between the alignment and harmful updates, contrary to what one might expect if safety were represented as shared component. This pattern is consistent across all layers and all models.
H EFFECT OF SAFETY-TUNIA VIA DIRECT PREFERENCE OPTIMIZATION
In this section, we extend our study to include a different alignment method, specifically the widely- used Direct Preference Optimization (DPO, (53)). This allows us to study how overlap varies under a different alignment method. In doing so, fundamental property of the model's architecture or consequence of the alignment procedure, and whether persists across different forms of safety training.
determine whether the entanglement observe is
20

---

Published as a conference paper at ICLR 2026
Table 5: Parallel projection-basee based update schemes across varying SVD fractions. We report the utility and harmfulness for models fine-tuned on Contaminated data.
Utility (t)
Harmful Score (L)
Model
Method
SVD Fractions
SVD Fractions
0.01
0.25
0.50
0.75
0.99
0.01
0.25
0.50
0.75
0.99
Base
0.21
3.27
Aligned
0.47
1.55
Fine-Tuned
0.60
2.16
Qwen-2.5 1,5B
Top-K
0.50
0.53
0.52
0.55
0.56
1.59
1.65
1.79
1.91
1.92
Random-K
0.49
0.53
0.55
Random
0.49
0.50
0.52
0.52
0.54
1.58
1.64
1.68
1.74
1.92
Base
0.03
4.13
Aligned
0.13
2.80
Fine-Tuned
0.37
3.60
Llama-3.2 IB
Top-K
0.14
0.20
0.25
0.29
0.33
2.84
2.90
3.05
3.36
3.45
Random-K
0.13
0.16
0.22
0.29
0.33
2.81
3.45
Random
0.13
0.16
0.22
0.28
0.33
2.84
2.90
3.19
3.19
3.45
Base
0.026
4.13
Safety-Tuned
0.032
3.41
Fine-Tuned
0.068
3.65
Llama-3.2
Top-K
0.027
0.026
0.033
0.048
0.039
3.42
3.48
3.59
3.56
3.62
Random-K
0.030
0.026
0.033
0.042
0.040
3.43
3.44
3.40
3.46
3.60
Random
0.032
0.026
0.032
0.039
0.039
3. 3.42
3.49
3.44
3.53
3.59
Base
0.44
2.53
Aligned
0.61
1.47
Fine-Tuned
0.73
1.99
Qwen-2.5
Top-K
0.62
0.63
0.65
0.68
0.69
1.49
1.58
1.69
1.76
1.83
Random-K
0.62
0.64
0.64
0.66
0.69
1.45
1.55
1.62
1.65
1.91
Random
0.62
0.63
0.64
0.65
0.68
1.50
1.57
1.75
Base
0.69
1.90
Aligned
0.74
1.30
Fine-Tuned
0.81
1.96
Qwen-2.5 7B
Top-K
0.74
0.75
0.75
0.75
0.78
1.30
1.55
1.60
1.68
1.67
Random-K
0.74
0.75
0.76
0.75
0.78
1.35
1.41
1.46
1.59
1.67
Random
0.74
0.75
0.75
0.78
1.40
1.48
1.56
Base
0.05
4.27
Aligned
0.20
1.74
Fine-Tuned
0.30
3.08
Llama-2 7B
Top-K
0.21
0.23
0.25
0.27
0.28
1.77
1.91
2.15
2.38
2.74
Random- K
0.20
0.23
0.26
0.28
0.28
1.74
1.91
2.09
2.38
2.79
Random
0.20
0.23
0.25
0.28
2.38
2.74
Base
0.053
4.27
Safety-Tuned
0.024
3.54
Fine-Tuned
0.168
3,89
Llama-2 7B
Top-K
0.029
0.036
0.055
0.082
0.127
3.49
3.69
3.76
3.83
3.85
Random-K K
0.026
0.032
0.042
0.062
0.128
3.59
3.58
3.75
3.79
3.89
Random
0.026
0.036
0.041
0.065
0.122
3.57
3.64
3.74
3.78
3.89
We repeat the full weight-space analyses from Sections 3, and for Llama-3.2 IB model that we safety-tuned using DPO the LF dataset (30). We train the model for 5k steps 10k-example subset of the dataset.
The results are shown in Figures 15, 16, and 17. We observe that all of previous hypotheses and analyses hold consistently across these experiments. Together, these findings suggest that the underlying geometric behavior identify is robust to the choice of alignment method.
21

---

Published as a conference paper at ICLR 2026
Fully Useful
Contaminated
Aligned
Random Baseline
MSO(U, H1)
MSO(U, H2)
MSO(H1, H2)
0.84
0.99
0.96
Qwen-2.5 .5 1.B
0.63
0.75
0.72
0.42
0.50
0.48
0.21
0.25
0.24
0.00
0.00-
0.00
0.01
0.30
0.60
0.90
0.01
0.30
0.60
0.90
0.01
0.30
0.60
0.90
MSO(U, H1)
MSO(U, H2)
MSO(H1, H2)
0.82
0.82
0.86-
Qwen-2255 55 3B
0.61
0.62
0.64
MSO
0.41
0.41
0.43
0.20
0.21
0.21
0.00
0.00
0.01
0.30
0,60
0.90
0.01
0.30
0.60
0.90
0.01
0.30
0.60
0.90
MSO(U, H1)
MSO(U, H2)
MSO(H1, H2)
0.70
0.91
0.85
Qwen-2.5 7B
0.53
0.68
0.64
0.35
0.43
0.18
0.23
0.21
0.00
0.00
0.00
0.01
0.30
0.90
0.30
0.60
0.90
Energy Fraction
Figure MSO across layers the 65-90% depth range for pairwise comparisons of activations from Useful (U) and multiple Harmful (H1, H2) prompt sets.
I Do SAFETY AND LEARNING DECOUPLE AT ANY LAYER?
In this section, we investigate whether a layer-specific strategy exists for isolating safety and learning subspaces. Specifically, we ask whether projecting (k) components from single layer yields different behavior than projecting the same number of components across all layers.
We test this on Qwen2.5-1.55 by repeating the experiments from Sections 3 and 4, but restricting the projection to individual layers. We evaluate approximately every third layer in the network, resulting in 10 distinct layers.
The results, shown in Figures 18 and 19, reinforce our earlier hypothesis: individual layer exhibits a clean separation between safety and learning directions. This further strengthens our claim that safety and task-learning signals are geometrically entangled throughout the model.
J ROBUSTNESS CHECKS: GENERAL ABILITY AND OVER-REFUSAL
J.1 ASSESSING GENERAL ABILITY CONFOUNDS VIA MMLU
We want test whether the harmfulness metric is confounded with general ability (instruction following and coherence). More concretely, aim to determine whether the results reported in paper could simply due to a loss in general capability. To evaluate this, we use MMLU (Massive Multitask Language Understanding, (18)), widely used benchmark for general ability. We repeat the experiments from Sections and 4 and report results in Figures 20 and 21 for Qwen-2.5 1.5B and 3B.
We observe that MMLU accuracy remains stable at approximately 61-62% and 64 64%,,, respectively for the two models, across all projection settings, indicating degradation in general ability. This
22

---

Published as a conference paper at ICLR 2026
M50JHA4 UsA)
MS0DHA B-A)
MSQULA R-A)
Readom Bandlnn:
0/77
0357
0.57
6:77
022
018
E30
5:10
0.00
091 0.53 E/ln dio
LO1 Dan abe náo
0.5 a.16 náa 600
0/0 6/00
C-n 0.0o
c.1 6.00 0.10 n/a
Layer
0.56
0.
0.38
0.29
0.0
0.23
0.10
091 0.59 1.00 dio
LO1 0220 aãs
náo
0.50 0.66 náoa 600
03:
040
6.00
601
6.00 0.
C.I 6.00
0.50
oda
13:8
0.0
U.99
0.1
010
0.10
0.30 Ein dio
0.20 n.s náo
0.0 0.
n93
0/0 6:0
6:40
0.10 nia
03:0
0.38
0.32
0.0
0.9
0.2
019
011 0.59 Ein die
nle nal
0/00 6mn
E4n aio
cDn 6.00 0.10 nia
0.38
0:3
0.18
0311
0.19
E.19
0.9
041 0.5 Lin dio
LO1 0.30
aãe náo
0.3 0.6 náa
03:
660
Energy Fraction
Figure 10: MSO at all layers for pairwise comparisons of the dominant weight subspaces from Harmful fine-tuned (H), Aligned (A), and Base (B) models (Qwen-2.5 1.5B).
strengthens our findings and confirms that our analysis is scientifically robust rather than an artifact of reduced model capability.
J.2 RULING OUT OVER-REFUSAL. CONFOUNDING FACTOR
We test whether projection induces over-refusal, which would indicate that safety results might be confounded by models incorrectly rejecting benign queries. To evaluate this, we measure refusal rates benign prompts using XSTest (55), which ideally should not be refused. We repeat the experiments from Sections 3 and and report results in Figures 20 and 21 for Qwen-2.5 1.5B and 3B.
Across all projection settings, refusal rates remain very low, around 3-4% and 5-6% respectively for the two models. These consistently low rates indicate that our safety metrics are robust and that the findings reported in the paper are not confounded by projection-indee induced over-refusal.
K ADDITIONAL ACTIVATION SPACE ANALYSES
We do several experiments to complement the discussion in Section
K. ACTIVATION SPACE ANALYSIS ACROSS ALL LAYERS
We report activation space results layer by layer to fully capture layer-specific behavior. These results are shown for Qwen-2.5 1.5B, Figures 22, 23, and 24, We observe that our hypothesis and findings hold consistently across all layers.
K.2 IMPACT OF USING EARLY TOKEN WINDOWS
We show the effect of using earlier token windows in Figure 25 for Qwen-2.5 1.5B and 3B. We observe that our results and hypothesis hold true in this setting as well.
23

---

Published as a conference paper at ICLR 2026
M50I15A,
MS0OIT-A D-A3
MSCUUA
Readom Unseline
Laser
Loyer
Lapor
Layor
Layer
Layer
0L31
0.2
0.0
CAI
0.50
AST
8.0
EAI
CDI
Layse 7
Luper 8
Layer:
Layar 10
Logan 11
Layer 12
0.29
0.90
03
0223
031
01
001 0.20 D:91 E10
EM Dan nike nan
035 0:20 0. 1811 690
MB: 022 DO0 B
c c:0 0.10 nin
Layer 11
Lapee 14
Rayer 15
Taper 16
Layaer 17
Layer 18
0.30
E31
n35-
0.08
0.00
5.0
002
607
601 0.50 0u0 Ei0 aite
1.1 020 1.10 a.e 0.00
031 0.30 0.1 ná0 6.00
A3: 0.3 0:00 8.00 ait
BA1 0,20 CAn 3.50 0.33
C.11 c.0 0.10 oáo n.00
M80
19
Layer 20
Layer
Layer
Loyer
Layer
0.33
0.31
0339
033-
0388
516
611 630 099 Eiin
EM nan ni não
035 030 9.1 DA min
03: 033 D/0 8mn axe
En1 110 E-in 3:n nas
di <in n nia ni
25
27
ost
002
003
on8
0.20
1.0 dite
E01 0200 0.s 080
0.50
0. nã0
0.00
03:
0.0 0:00
601
6.40 d:
0.03 0:00
0.10 od9
1:02
4.15
023
0.5
0.1
018
E16
0.00
uiee
Energy Praction
Figure 11: MSO at all layers for pairwise comparisons of the dominant weight subspaces from Harmful fine-tuned (H), Aligned (A), and Base (B) models (Qwen-2.5 3B).
24

---

Published as a conference paper at ICLR 2026
MSDIHA, USA)
MSCIHA, B-A)
MSCU-A B-A)
Reaion Euselie
Layer
Layer
033
5.05
031
0.22
031
0.14
0.21
E36
013
6.5
0.77
0.05
0.00
0.00
0.0
0.00
Layer
Loyer
Layec
Layer
Layer
Layer
0.55
0.00
036
0.00
016
0.05
000
0.e
C.0
0.5
5.00
002
0.00
0.00
Lagur
Layer
Layer
Layar
Layer
032
0.32
E32
0.10
0.66
0.06
0.00
5.00
0.0
0.00
aãe
Loyer
Layer
0.2-
0.10
E36
536
0.6
0116
0.00
0.00
aãe
0.00
5.00
000
0.00
Layer
Layer
Layer
0.33
0.33
0.2
011
016
2.16
0.06
die
000
abe
0.00
2.00
Energy Fraction
Figure 12: MSO all layers for pairwise comparisons of the dominant weight subspaces from Harmful fine-tuned (H), Aligned (A), and Base (B) models (Qwen-2.5 7B).
MSOIHA, U.A)
MSO0H-A B-A)
MSOQUA B-A)
Random Baseline
03.7
0328
0.2
636
0:8
000
0.09
0.28
cm
die
CO1
0.0
0d3 0.00
03:
8:00
501 0.00
C.0 0.6
Layer 7
Layrer 8
Layer 9
Layer 10
Loper
Layer 12
0.58
MBO
0.11
0.0
t.1
0.9
0.2
019
0.5 Ein
E01 0.30 nãs não
0.5 nés
040 6:0
640
0.10 nãa
0.33
0.37
022
629
0.08
01 0221 0091 tin
614 038)
6558 0:20
1997
600)
H4.
1222 DG) SMI
Energy Fraction
Figure 13: MSO at all layers for pairwise comparisons of the dominant weight subspaces from Harmful fine-tuned (H), Aligned (A), and Base (B) models (Ilama-3.2 1B).
25

---

Published as a conference paper at ICLR 2026
M50JHA
B-A)
Raadom Baseline
Laye 1
Luyer 2
Layer
Layer
Lager
Layer
0.29
0.14
9-88-
0.8
0.00
516:
0.8
001 0.2 6:00 E.w.
ED1 D20
1:40 0.50 0:00
0.50
0.20 0.60 0.0 0:00
MR: 0.21
060 B:0 0.00
DML 1221
3.00 0.59
0.0
6:30
0.40 000 0:50
Laver ?
Layer 0
Layer 9
Layer 10
Layer 11
Layer 12
0.20
u..3
0.00
608
516
606
0d1 0.50 640 1oo dile
1.1 Dan 1.40 0.s od0
0.5 0.20 0:60 oda3 600
A8: 0.11
0.00
8.00 a.cc
EOL 0.29
0.40
0.33
0.01 6.00 0.40 odo 0.00
Layer 13
Layer 14
Layer 13
Laver 16
Loyer 17
Layer
0:0
025
022
0.13
0.66
nas
001 022 6:D0 1AM 0.80
EUT Diar 1.00 u.50 0dd
915 0.61 099
1o0
M8. D00 BA usc
OU1 1.00
Gin 4:00 v.d0
Laser
Taare
Laver
0.55-
n25
0.23
0.13
000
606
031 020
E.00 dioe
101 D20 1.40 0.o od0
0.5 0.40 0d0 000
032
0400
6400
501
6.00
0.00
C.03
0.40
od9
0.32
0.19
0.18
012
Ein
dike
035 199
1130 099
0.19
032
005
000
aic
aše
Energy Frection
Figure 14: MSO at all layers for pairwise comparisons of the dominant weight subspaces from Harmful fine-tuned (H), Aligned (A), and Base (B) models (Llama-2 7B).
Top-K
Aligued
Full Useful
Harmful
Usefuu
Harmful
0.12
Enargy Kogt
Energy Kept
Harmauul Score
0.30
4.12
Utiley
0.17
0.15
0.06
1.96
3.81
0.75
0.50
0.75
0.50
0.73
0.99
0.50
Figure 15: Parallel projection based update schemes across varying SVD fractions (Llama-3.2 IB). We report the energy-kept ratio for models fine-tuned Full Useful and Full Harmful data, utility for models fine-tuned on Full Useful, and harmfulness for models fine-tuned on Full Harmful. Model alignment was performed using DPO.
Randows
Top
Randous-X
Algused
Full Useful
Full Harmfal
Contaminated
Contaminated
Contaminated
1.00
0.12
Reeegg Kegaa
Eaeerpr Kopt
865
Undire
3.09
Marmhll
3.77
2.89
0:51
0.54
6.15
0.30
Prraction SVU Vectors
Figure 16: Orthogonal projection-based update schemes across varying SVD fractions (Llama-3.2 IB). We report the energy-kepl ratio for models fine-tuned on Full Useful, Full Harmful and Contaminated data; and utility and harmfulness for models fine-tuned Contaminated. Model alignment was performed using DPO.
26

---

Published as a conference paper at ICLR 2026
M50]HA4 UAA)
NSC0HA B-4)
MSQULA B-A)
Raadom Baseline
udt
641
042-
edt
621
E31
021
0.11
d.
CO1
063 600
6:00
601
0.00
C.I 0:10
Layer? ?
Luyer B
Layer 9
Layer 10
Laper
Layer 12
0.31
0.21
MML
011 Ein
iike nao
1115 ni0
Layer
0.11
6-42-
341
021-
E30
510
001 022 6:00 t.0
101 D20
140
0.5
0200
0.50
0.20
9.61
089
1:00
0222 0G)
Energy Fraction
Figure 17: MSO at all layers for pairwise comparisons the dominant subspaces from Harmful fine-tuned (H), Aligned (A), and Base (B) models (Llama-3.2 IB). Model alignment was performed using DPO.
K.3 IMPACT OF USING REFUSAL PROMPTS
We investigate the effect of using refusal-oriented dataset (harmless prompts from Extended Refusal, (57)) as the counterpart to harmful prompts. Instead of relying on a generic "useful" dataset such as MATH, this allows to test whether findings depend the choice of benign data. As shown for Qwen-2.5 1,5B in Figure 26, the results remain consistent under this substitution.
L DISTINCTION FROM ACTIVATION STEERING METHODS
Our claims are compatible with the empirical success of activation steering methods. However, they target a stronger assumption that is not established by those works: namely, that there exists a relatively low-dimensional, stable "safety subspace" whose directions uniquely encode safety and can be edited isolation, without affecting acquired utility.
Prior work shows that for given model checkpoint and prompt distribution, one reliably extract directions or small subspaces in the residual stream whose manipulation strongly affects refusal or a unique safety characteristic. Examples include single refusal directions (2), conditional activation steering (34), manifold steering for overthinking (26), and safety directions that become easier to find in higher-dimensional models (62). These results demonstrate local linear controllability: in given model. at specific layers, there are directions along which safety behavior is highly sensitive.
What they do not by themselves establish is that these directions constitute unique, model-level safety subspace that is (i) conceptually specific to "safety" rather than general behavior, and (ii) robust under fine-tuning or other parameter changes.
At the same time, complementary body of work points toward more entangled and multi directional picture: safety and helpfulness often trade off along shared directions (62), and safety behavior can be mediated by structured sets of neurons or layers rather than a single feature (6; 78).
Taken together, these works suggest the following picture: activation steering can reliably surface high-leverage (but local) directions that influence refusal within a fixed model, but this does not imply that safety is represented by stable línear subspace. Instead, the evidence is consistent with safety being encoded through distributed mechanisms that vary across tasks and checkpoints. Our claims target this stronger, subspace- interpretation, rather than the more limited form of local controllability demonstrated by steering methods. This is particularly important because growing body of work (e.g.. SafcLoRA. SaLoRA, etc.) relies on this stronger assumption, despite there being evidence that local controllability extends to global safety behavior.
27

---

Published as a conference paper at ICLR 2026
Aligned
n.58
Useful
Eleeggo Kept
0.55
Harmful
0.
Full Useful
Horrafal Score Harudu Score Harmtul Store Hammdd Soaee Hormtat Score Harmaa Scare Harmouu Score Harnta Score Harudu Score Haratu Score
211
Full
Layer
0.77
0.77
Utiliy
07
2.98
8.t.
0.8
1.78
046
1.50
0.55
0.50
0.75
0.50
0.9
6.0
050
0.5
6,89
Kapt
0.54
Euenyy Kapt
0.54
u81
2.20
Full
Layar
0.36
0.66
Utilty
a57
2.05
hin
mm
1.81
000
0.01
0.48
2.58
0.25
Energy Kept
0.34
Full Useful
Eneegy Kept
use
Fall Harmful
0.1
Full Useful
2.19
Full Harmful
Layer 7
0.30
Utility
0.57
8.58
618
0.8
033
2.78
am
am
0.48
2.50
6.0:
Full Usetul
Eowngy Kapt
Full Harmful
Full Useful
2:15
Full Harmful
Layor 10
0.36
Utiltty
0.7
0.3
8.77
0.1
0.01
9.58
0.75
0.25
050
G.D.
0.25
250
0.5
0de
ad0
0.15
0.00
u25
Full Useful
Energy Keps
Full Harmful
Full Useful
2.15
Full Harmful
Layer 13
Utiltt
0.53
2.76
0.01
0.25
0.50
0.75
0.99
0.00
0.25
0.50
0.75
0.99
0.00
6.D:
0.5
0.50
0.75
0.83
9.01
0.5
0.50
0.5
0.89
Full Useful
Full Harmful
Full Usefuu
Full Harmful
Elwerg Kegtt
061
2.15
Layer 18
Utiitt
0.7
0.5
0255
uSB
0.75
use
QUT
0.25
030
G.D.
0.25
350
0.75
0001
0.25
0.50
125
Layer 18
Full Useful
Eawwg Kapk
0.53
Pull
Harmful
Full Useful
2:16
Full Harmful
Uhilty
3.80
0.52
3.73
0:00
0.01
0.55
0.50
0.75
0.99
0.01
0.25
050
0.75
0.99
0.0.
0.5
050
0.75
0.99
0.01
0.25
0.50
0.75
0.95
Kegt
0.32
Useful
Eneeggy Kept
033
Harmful
Useful
2.19
Layer
0.36
0.30
Utility
2.95
61a.
0.8
0.52
1.78
0.48
1.58
0.75
0.9
0.25
035
0.0
050
0.82
025
Kops
0.34
Euengy Kapt
0.55
0.61
2.19
Full
Layer 24
0.3
0.77
Utility
2.85
010
h.9
8,78
001
0.1
0.00
1.58
Full Useful
Fall Harmful
Full Useful
Full Harmful
Eleergy Keyt
nan
2.22
Layer
Energy
B3r
Utility
u57
2.03
R19
0.20
033
4.79
8m
0.48
3.58
6.0
Fraction SVD Vectors
Figure 18: Parallel projection-based update schemes across varying SVD fractions (when applied to
a specific layer only). We report the energy-kept ratio for models fine-tuner on Full Useful and Full Harmful data, utility for models fine-tuned on Full Useful, and harmfulness for models fine-tuned on Full Harmful.
28

---

Published as a conference paper at ICLR 2026
Random
Top-K
Random-K
Aligned
Rone-Tined
Bhangy Koys
Full Usetul
Euanyy Keps
Pull Harmful
Energy Kept Energy Kegt Euergy Kegt Eberry Kept Eoenyy Keps
uasu
Contaminated
0.59
Contaminated
Harafu Scores Haraful Scores Bocmntul Score Hormttl Score Hamndd Soove Hanahil Seove Hartatil Store Hormtul Score Hormtul Score Harmdu Score
Contaminated
Loyer
Unilty
6.56
0.63
Rd3
R.5
0.40
0.01
0.25
0.00
0.75
0.99
0.00
0.55
0.50
0.75
0.9g
001
0.25
0.30
875 6.09
0.01
0.5
0.50
0.75
0.99
0.01
0.25
0.50 0.75 0.59
Euergy Kegt
0.99
Full Useful
Eleerpy Kept
898
Full Harmful
Contaminated
0.59
Contaminated
Contaminated
Laver
Unilte
8.6a
0.52
0.01
0.25
0.50
0.75
0.50
0.01
0.5
0.50
0.75
0.3e
0.01
0.25
0.50
0.7 0.55
0.01
025
0,50
0.75
0.s0
0.01
0.25
0.50
0.75 0.30
Etergy Kegt
0.99
Full Useful
Eleergy Kept
Full Harmful
0.99
Contaminated
0.59
Contaminated
21c
Contaminated
Layer
Uniity
8.aa
0.01
0.25
0.50
0.75
0.50
6.00
0.25
0.50
0.75
0.30
0.01
0.25
0.50
a75 a.d0
0.01
0.55
0.50
0.75
0.9
0.01
0.25
0.50
0.75 0.30
Lever 28
Emergy Kegt
0.000
Full Useful
Energy Kegt
0.0
Full Harmful
0.59
Contaminated
0.59
Contaminated
Contaminated
unitty
0.01
Rec
o.ca
0.45
n.0 n:5
03o
0.5
can
025
n.50 0.75 0.
0.2
0.50
a.5 0.
0.11
n25
0.50
n:75
nan
0.
0.35
nan 6.75 0:30
Energy Kegt
30.00
Full Useful
Eneegy Kegh
aso
Pull Harmful
0.59
Contaminated
0.59
Contominated
R13
Contaminated
Layer
0.22
Uitllty
=
0.84
0.04
0.61
0:25
0350
025
0.90
6.03
0.25
0:30
0.75
0.59
0.0 0.25
0.50
0.7 0.99
0.01
0.5
0.50
0.75
0.99
0.01
0:5
030
9.75 0.90
phangy Kapt
0.99
Usetul
Eaeryy Kap:
Harmful
Unergy Kap:
u.sw
Contaminated
0.59
Contaminated
Contaminated
Laver
0.02
Utilty
b.
Rec
0.64
0.47
0.81
0.25
650
0.75
0.99
6.01
020
0.50 0.75
0,38
0.1
0.25
0.50
0.75 0.69
0.01
025
0.50
0.75
0.99
0.01
0.23
0.50
0.75 0.50
Energg Kept.
0.99
Useful
Euengy Kept
8.9
Harmful
Energ Kept
0.99
Contaminated
0.59
Contaminated
Rie
Contaminated
Loyer
0.02
0.81
Unilte
Beal
0.47
0.45
0.81
0.25
0.50
0.55
0.50
6.00
0.2
0.50 0.75
0.01
0.25
0.50
075 0.69
0.01
025
0.50
0.75
0.00
0.01
0.25
0.50
0.75 0.30
Useful
Harmful
Contaminated
Contaminated
Energy Kegt
0.99
Euergg Keyt
0.9
Euerpy Kept
0.59
0,99
215
Layer
0.82
0.81
Utilty
0.03
nea
6.8
0.81
025
6.0
0.55
0.59
6.01
0.25
0.50 0.75
0.01
0.25
0.50
a75 0.59
0.01
0.5
0.50
0.75
0.99
0.01
0.25
0.00
0.75 0.30
Layer 24
Energy Kegt
0.99
0.82
Useful
Eneegy Keyt
am
0.9
Harmful
Elveggy Keyt
0.50
0.59
Contaminated
Utilty
0.59
Contaminated
Contaminated
8.
863
0.c0
0.52
0.40
6.48
1.58
not 0:5
aão
0.5 n.50
oan
aás n.50
0.75 oute
0.25 0.50 a75 aut
0.01
025
0.50
0.25 nsn
6,61
0.3
0.75 0:30
Energy Kegt
0.00
Energy Keyt
0.00
Eneeray Keet
0.59
Contaminated
Contaminated
Contaminated
Loyergy
a.
am
0.59
UNIle
0.63
0.50
0.52
0.38
2.58
0.01
050
025
0.50
0.25
0.00
0.75
0.20
0.25 0.1
0.50
8/7 0.19
0,11
025
0.50
0.25
0,01
Fraction
Vectors
Figure 19: Orthogonal projection-bast based update schemes across varying SVD fractions (when applied to specific layer only). We report the energy-kept ratio for models fine-tuned on Full Useful, Full Harmful and Contaminated data; and utility and harmfulness for models fine-tuned on Contaminated.
29

---

Published as a conference paper at ICLR 2026
Random
Top
Random
Aligned
Pinc Tuned
0.52
Pure
0.47
Narmful
0.70
Pure
0.10
[kamful
XSTeat Rohusal Race
Qwen2 1.38
Energy Kegt
0:39
Energy Kept
MMLU Accuracy
0.63
0.07
0.36
0.03
-0.00
-0.02
0.50 0.11
0.00
0.62
Pure
0.58
Harmful
0.70
Pure
0.10
Harmful
XSTest Rchual Rata
MMLU Accuracy
Queem 388
Energy Kept
Energy Kept
0.63
0.07
0.38
0.18
0.57
0.93
0.01
0.25
0.50
0.75
0.99
-0
-0.02
0.25
0.50
0.75
0.99
0.30
0.01
0.25
0.50
0.73
0.99
0.00
6,01
0.25
0.50
0.75
0.99
Fraction SVD Vectors
Figure 20: Parallel projection- baaeed update schemes across varying SVD fractions. We report general-ability metric (MMLU accuracy) for models fine-tuned on Full Useful, and refusal rates on benign prompts (XSTest) for models fine-tuned on Full Harmful.
Tap-K
Bondem-K
Aligrnt
Thnn-Tuned
Contaminated
Contaminated
Contaminated
X05 Tet Tetunal Stale
Onins2.5 158
Ereep Kegt
E30000 Kagt
MHHLLI Actummes
607
3.00
6.18
675
60%
0.2
6/00 3.0
125
850
1.82
505 Tete Reeineal ttale
MHHU Assurucy
Oreondd 30
Keet
Fleepy Kepa
Eloow Kogt
80
038
642
022
0:53
600
Proction of 5VD Vectors
Figure 21: Orthogonal projection- based update schemes across varying SVD fractions. We report
general-ability metric (MMLU accuracy) and refusal rates benign prompts (XSTest) for models fine-tuned on Contaminated.
30

---

Published as a conference paper at ICLR 2026
Fully Uschul
Random Baseline
Contaminated
Aligned
0.15
0.65
0.30
0.17
0:31
0.42
0.30
0.13
0:00
033
0.5
07 0.0
0.00
0.1
0.5
035
03
0.9
0.00
0.1
6.1
0.5
0L
0.
0.00
31
05
67
04
0.02
03
0.3
05
0.00
0.3 0.5
0.7 0.
Layer
0.58
0.55
0.57
6:5
0.11
0.14
0.15
6.13
0.11
07
Layer 15
Lopor 14
Layer 15
Layer 16
Layer 17
Loyur 18
0.71
0.74
0.70
0:00
0.76
0.50
0.50
0.0
000
0.54
MS0
0.50
0:30
0.14
0.16
0.33
Layer
Layer
Laver
Layer
Layer
Luyez
0.66
0.c0
0.61
0:01
0.53
0.12
0.28
0.26
0.23
0.20
0:29
0:05
0.00
am
01 6.1 03
311 03
05
67 64
05 07
0.37
0.00
0.50
0.52
0.17
863
81
03
035
0.7
0.0
0.00
0.1
035
055
0.7
0.
0.00
01
031
01
0.
3.
0.00
04 33
03
05
67
Energy Prection
Figure 22: MSO across all layers for activations from Useful (Math) and Harmful (BeaverTails) prompt sets.
31

---

Published as a conference paper at ICLR 2026
Fully Usahul
Random Baseline
Contaminated
Aligned
0.22
0.16
0.49
0.13
0:36
0.28
0.00
0:00
03
0.5
07 0.0
0.00
0.1
0.5
0.5
0.7
0.9
0.00
0.1
631
0.5
OLT 03
0.00
311 DJ
05
67 Dd
0.02
031
0.3
05
0.00
0.3
0.5
0.7
0.0
Layer
0.70
0.42
0:46
0.42
0.10
0.30
0.11
0.11
0/1
0.11
0.5 67
Layer 15
Loger 14
Layer 15
Layer 16
Layer 17
Loyur 18
0.51
0.53
0.56
0.54
0:41
0.11
0.45
0.00
0.53
MS0
0.00
0.22
0.00
0.30
0.11
Layer
Layer
Laver
Layer
Layer
Layez
0.10
0.49
65
0.51
0.50
0.20
02
0.18
0:03
0.00
01 6.3 03
311 03
05
0.7
64
0.:0
0.11
0.56
0.45
0.30
863
81
0.3
0.5
0.7
0.0
0.00
0.1
035
055
0.
0.00
01
031
01
0.7
3.9
0.00
04 311
a5
67
Energy Prection
Figure 23: MSO across all layers for activations from Useful (Math) and Harmful (ToxiGen) prompt sets.
32

---

Published as a conference paper at ICLR 2026
Fully
Contaminated
Aligned
Layer 2
Lager 2
Layer
Layer
Layer 5
Layer 6
1.66
0.20
0.77
0.53
0.43
043
0.46-
0.15
0.19
0.23
0.24
0.19
0.33
0.08
0.11
6.11
0.12
U.:
0.7
Layer
Eayer
Layer
Tayer
Tayer
Eayer
0.12
0.72
0.70
0.52
0.03
0.55
0.50
0.02
0.47
0.15
0.20
0.31
0.33
0.34
0.34
0.0 0.5
0.5
6.7
0.5
07
Eayor
Layer
Tayer
0.80
0:57
0.07
0.55
MS0
0.50
0.50
0.34
0.34
0.35
6.0-
0.31
0:09
9.3
0.00
000
01 0.0
03
000
3.1
03
05
0.7 09
01.
9. 67 03
0.0 0.7
24
0.00
0.40
0.67
0.40
0.42
0.37
0.10
0.30
0.15
0.14
0/09
01
0.3
0.5
0.7 0.0
1.00
0.1
0.5
0.5
0.
0.9
0100
0.1
0.1
0.3
OLT 09
000
3.1
D3
055 67 bd
040
03 0.3
05
07 05
0.00
031 0.5 0.5
0.7 0.
Layur 25
Layur 26
Layer 27
Layer 28
0.99
0.71
01:11
0.57
0.5
0.77
0.10
0.20
0.17
01
0.5 67
Energy Praction
Figure 24: MSO across all layers for activations from different Harmful (BeaverTails, ToxiGen) prompt sets.
6.70
0.70
0.85
0.50
0:14
MSSO
E40
8.21
636
MISOCUL B13)
4.5
6.70
0.00
6mm
6.00
0.30
SISOCU B12)
3.50
6.70
0.99
6.1
0.30
31500H3, H2)
6.6
0.0
8.9
Qowen-2-5
0.56
0:43
0:79:
070
Praction
Figure 25: Average MSO (for early token windows) across layers in the 65 55-90% depth range for pairwise comparisons of activations from Useful (U) and multiple Harmful (H1, H2) prompt sets.
Uesful
Conteminattel
Allgnnl
Bondem Beceline
Owwa-223 1.58
Qwen-225
0:31
0:31
0.00
6m)
0.00
0.00
0.21
0.30
610
072
Energy Fraction
Figure 26: Average MSO across layers in the 65-90% depth range for activations from Refusal and Harmful prompt sets.
33

---

Published as a conference paper at ICLR 2026
M DATASET DETAILS
We use the M dataset (75) for fine-tuning, which reformulates existing math problems from alternative perspectives without introducing new content. To evaluate performance, we rely on the GSM8K benchmark (9), dataset of clementary-level math questions that require multi-step reasoning. Models are assessed based solely on the correctness of the final numerical answer. For our activation- based analysis, we sample prompts from the MATH dataset (19), which contains challenging, competition-stie arithmetic problems.
BeaverTails (31) is valuable dataset for studying safety by independently annotating ques- tion-answer pairs for both helpfulness and harmlessness. We use the training set to fine-tune models in both harmful and contaminated settings, and draw prompts from the test split for our activation-based experiments.
AdvBench (80) consists of 500 prompts designed to elicit wide range of harmful behaviors, including profanity, threats, misinformation, discrimination, cybercrime, and other forms dangerous or illegal content framed instructions. We use this benchmark quantify model harmfulness: higher success in responding to these prompts indicates greater unsafe behavior.
ToxiGen (15) is a large-scale dataset composed of both toxic and non-toxic statements. We use subset of its prompts to analyze model activations in response to harmful content.
MMLU (18): The Massive Multitask Language Understanding (MMLU) benchmark evaluates general ability and knowledge. We use MMLU to test whether our harmfulness metric confounded with general ability.
Extended Refusal (57) is a dataset of prompts curated to evaluate whether models refuse queries. In work, its harmless subset refusal-oriented counterpart to harmful prompts.
XSTest (55) is a benchmark designed to measure refusal. We use XSTest to determine whether our projection methods induce over-refusal.
PKU-SafeRL.H (30) safety-alignment dataset constructed for reinforcement learning with human feedback (RLHF), covering wide spectrum of harmful and harmless interactions. We use PKU- SafeRLHF to safety-tune model using DPO.
N LIMITATIONS
Our study focuses on linear subspaces, providing a principled first step toward understanding the geometric structure of safety alignment. While do explore non-linear representations, our framework could be extended in future work to capture richer geometric phenomena. Our experiments are restricted to open-weight models with publicly available base and aligned variants. These models provide a controlled and interpretable setting, though extending to production-leve closed-source models remains an important direction for future work.
USE OF LARGE LANGUAGE MODELS
Our use LLMS is restricted to light writing assistance, including grammar polishing and enhancing clarity.
34


Reviews:
[{"review_id": "mIAjkpcFcq", "reviewer": "Reviewer_hV1p", "summary": "This paper empirically investigates whether safety subspaces exist in LLM weight or activation space that uniquely encode alignment through 4 studies: Projecting helpful and harmful updates into alignment/safety subspaces, orthogonally filtering contaminated updates, comparing subspace overlaps among useful/harmful/alignment directions, and analyzing activations for harmful vs. benign prompts. The authors find that high-impact directions amplify both utility and harmfulness, and activations for harmful and benign prompts substantially overlap. The results indicate safety is entangled with general learning rather than linearly separable, so projection or filtering along “safety” directions cannot selectively suppress harmful behavior without similar utility loss; this challenges subspace-based defenses and raise a need for alternative strategies for maintaining safety during continued training.", "questions": "- Can you provide layer-by-layer activation-space results to better demonstrate the layer-specific behavior.?\n- Is there a theoretical explanation or intuition for why high impact directions amplify both helpfulness and harmfulness, and under what conditions might they be separable? For example, would adding a contrastive objective during training help disentangle directions associated with helpfulness and harmfulness?\n- Can we learn a system prompt that, when used for conditioning, reduces the overlap between safety and utility directions, potentially making subspace based defenses practical even though current finetuning results do not show clear separability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper does an extensive empirical analysis extends from weight update directions to activations, and evaluates across multiple open source LLM families (Llama, Qwen) and fine tuning regimes (helpful, harmful, contaminated).\n- The work draws a concrete practical implication: subspace-based defenses cannot selectively suppress harmful behavior without proportional utility loss, guiding future safety strategies away from linear subspace filtering.\n- The results provide consistent evidence in both weight and activation spaces that harmful and benign signals substantially overlap, indicating these directions are not clearly separable.", "weaknesses": "- The focus of this work focus on defense mechanisms based on linear orthogonal projections, leaving non-linear alternatives unexplored.\n- The paper reports activation-space results as averages over a mid-depth slice (about 65–90% of layers) rather than layer by layer, which may not fully capture layer-specific behavior.\n- The paper lacks a theoretical investigation of the results, relying on empirical evidence without a formal framework explaining why safety and utility directions overlap. This is reasonable for an empirical study, but a concise theoretical model would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cUO0cv3SjS", "reviewer": "Reviewer_YiuK", "summary": "The paper investigates whether safety behaviors in large language models correspond to distinct linear subspaces that can be isolated without degrading general capability. It constructs “safety subspaces” from the principal components of fine-tuning updates between different stages of fine-tuning, then measures their geometric overlap using metrics such as Mutual Subspace Overlap and Energy Kept Ratio. Across multiple model families, the authors find that safety and utility update directions share most of their variance and projection along these subspaces reduces both helpfulness and harmlessness simultaneously. The results suggest that safety alignment is not linearly separable but deeply entangled within the same representational space as capabilities.", "questions": "1. Would you say your conclusions align with the findings of Wei et al. (2024) despite the methodological differences? From my understanding, they appear to have reached a similar conclusion.\n\n2. Do you expect the observed entanglement behavior to change under Mixture-of-Experts (MoE) model families, where safety and capability routing may be distributed across distinct experts?\n\n3. How is the k in the top-k principal components chosen, and what proportion of the total variance does this selection typically explain?\n\n\nWei, B., Huang, K., Huang, Y., Xie, T., Qi, X., Xia, M., Mittal, P., Wang, M., Henderson, P. (2024). Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications. arXiv:2402.05162. https://arxiv.org/abs/2402.05162", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper formalizes “safety subspaces” concretely by building them from principal components of weight differences. Although PCA on weight deltas is common in task-vector or model diffing work, applying it here provides an interesting way to identify safety regions beyond analyses that focus only on activations popular in recent AI safety works.\n\n2. It introduces two sound measures of subspace overlap that are intuitively meaningful, providing a clear alternative to relying solely on downstream evaluation metrics to quantify model diffing methods.\n\n3. The presentation is clear: each research question is addressed in a dedicated section and supported by plots and analysis and the claims in this paper is well supported. The reported metrics and ablations are convincing, and the additional validation in activation space increases confidence that the findings are not artifacts of a single parameterization.", "weaknesses": "1. The conclusions may depend on the specific design choices for subspace dimension and PCA aggregation (it seems that SafeMERGE supports a form of layer-wise separability). The paper fixes the number of principal components and the layer aggregation strategy without sensitivity analysis, so the observed overlap might partially reflect those implementation choices rather than a conclusive property.\n\n2. The experiments use a single alignment method and do not test how geometric overlap changes under different alignment objectives, especially under common alignment methods like RLHF and DPO, which usually yield better alignment. Without this comparison, it is unclear whether the observed entanglement is inherent to model structure or specific to the alignment method used and prevailing in all safety representations.\n\n3. The evaluation measures helpfulness and harmlessness separately but does not analyze their joint behavior on adversarial inputs. Testing whether projection affects over-refusal, for example by using XSTest for benign prompts that should not be refused, would better assess whether safety information can be isolated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KjzvOrncgG", "reviewer": "Reviewer_Xrmm", "summary": "This work challenges the prior assumption that safety relevant features lie in a specific subspace of the weight space, hence, can be isolated and defended against jailbreaks. Using 2 metrics (Energy-kept ratio, Mode Subspace Overalp), authors find that safety subspaces are entangled with general utility subspaces hindering the applicability of the prior assumption and defense mechanisms.", "questions": "1. If there is no such concept subspaces like \"safety\" in activation space, how would authors explain the success of activation steering methods?\n2. Do results of this paper have any implications on linear separability hypothesis [2] which led many to believe there exist linear concept subspaces [3, 4]? \n3. What is the reason to choose the MATH dataset as a counterpart for \"harmful prompts\"? Would using any refusal dataset instead of a generic \"useful\" dataset impact the observation?\n\n___\n\nOverall, I believe the topic of the paper is relevant and important for the ICLR community, and the paper offers nontrivial contribution (see Strength). Therefore, I am willing to increase my score upon satisfactory responses to the questions raised.\n\n___\n\n### References\n\n1. Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang.Safe lora: the silver lining of reducing safety risks when fine-tuning large language models, 2025.\n2. Kiho Park, Yo Joong Choe, Victor Veitch. The Linear Representation Hypothesis in Language Models. NeurIPS 2023\n3. Yao Huang, Huanran Chen, Shouwei Ruan, Yichi Zhang, Xingxing Wei, Yinpeng Dong. Mitigating Overthinking in Large Reasoning Models via Manifold Steering. arXiv:2505.22411\n4. Rachel S.Y. Teo, Laziz U. Abdullaev, Tan M. Nguyen. The Blessing and Curse of Dimensionality in Safety Alignment. COLM 2025.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Significance:** The paper addresses broadly believed hypothesis that a concept-related features are essentially contained in a specific lower dimensional subspace.\n2. **Novelty:** While mostly building upon [1], the authors take a critical standpoint to better investigate the source of the positive results in [1] and tradeoffs it poses (utility drops similarly with the removal of \"harmful subspace\"). There have been works to observe that most jailbreak defenses are also likely to hurt the general utility of the model, yet a dedicated investigation was lacking in the literature.\n3. **Writing:** The writing is mostly clear and easy to follow.", "weaknesses": "Although the paper claims their findings for both weight space and activation space, the activation space seems to be relatively underexplored. Considering that LLM steering is predominantly done on its activations (Activation Addition, Directional Ablation, Manifold Steering, Angular Steering etc.), I think the paper does not deliver sufficiently in this important aspect. The following questions are to specify why exactly I think the current study is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tspy7zbhJa", "reviewer": "Reviewer_QX9G", "summary": "The paper examines whether linear subspaces derived from alignment and safety deltas can selectively control safety, and whether activation patterns for harmful prompts occupy distinct regions. The main observations are that Top-K SVD directions are strong general-impact directions for both utility and harmfulness, orthogonal filtering trades off both together, and activation subspaces of useful and harmful prompts overlap at many layers. These results argue that simple linear projection defenses have limited selectivity.", "questions": "- The paper introduces both the general alignment update ($\\Delta_A$) and a specific safety-tuning update ($\\Delta_S$), but the analysis often groups them. What are the difference between the subspaces derived from $\\Delta_A$ and $\\Delta_S$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Evaluations span multiple Llama and Qwen variants and include analyses under contamination. This breadth supports external validity of the main pattern.\n- The paper explains the operators, datasets, and scoring choices clearly enough that others can reproduce or extend the tests.\n- Demonstrating limits of global SVD-style filtering is actionable for researchers who might otherwise rely on it as a safety control.", "weaknesses": "- Broad-claim relative to evidence. The experiments show that global Top-K SVD directions are not safety-selective. That does not exclude the existence of linear safety structure discoverable through other methods, such as multiple independent refusal directions or concept cones[1]. The claim should be scoped accordingly.\n- Activation analysis focuses on last-token states, but refusal dynamics often appear early in the response[2] instead in the last token. I would like to see whether using earlier token windows or residual-stream features changes the overlap picture.\n- Useful and harmful prompts differ in source and style. Some of the observed overlap could reflect stylistic differences.\n    \n[1] Tom Wollschläger et, al. *The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. 2025.\n[2] Andy Arditi et, al. _Refusal in Language Models Is Mediated by a Single Direction._ 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Nl9nnZ0b7", "reviewer": "Reviewer_M6zX", "summary": "The paper tests whether “safety subspaces” exist as linear directions in weight or activation space that could preserve or filter safety during fine-tuning. The authors construct subspaces from SVD of alignment and safety deltas, then study how “useful” and “harmful” fine-tuning updates interact with these subspaces via parallel and orthogonal projections, and how their activation spaces overlap. Across five open-weight models, Top-K singular directions strongly affect both utility and harmfulness, orthogonal filtering reduces both together, and activation subspaces for useful and harmful prompts overlap. The paper concludes that distinct linear safety subspaces are unlikely and that projection-based defenses have limited value.", "questions": "- Can you address my concern about the weaknesses?\n- At line 242: \"At the same time, while energy is evenly distributed, behavioral impact is not.\". I don't quite get it. I see in Figure 2, all metrics are uniformly increasing with more fraction of SVD vector. How does this results implicate \"behavioral impact is not evenly distributed\"?\n- Most results in the paper are observable, and the paper didn't give analysis on the counterintuitive results. Could you give some reasons on why your experiments shows different results than previous papers? - The implications in line:404 and line:461 are identical. Is it a typo?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **The question is important for alignment interpretability**. The paper asks whether safety can be isolated linearly in weight or activation space, which underlies many safety direction defenses. The experimental design addresses both weights and activations across several models.\n- **Proper analyzing methods**. The projection schemes and subspace overlap measurement is theory-driven and well-formulated. The paper examine the so-called safety subspace from multiple aspects, which is solid.\n- **The paper is generally clearly written**. I can easily follow the description and understand the claims.", "weaknesses": "My main concern of the paper is that the experiment settings in the paper are limited for the broad claims that \"Safety subspace does not exist\".\n\n- **The training settings setup does not isolate safety signal**. The Safety tuning and downstream fine‑tuning use supervised fine‑tuning on text corpora with many properties besides safety. The \"useful\" task is MetaMathQA math problems and the \"harmful\" task uses BeaverTails unsafe data. This setup makes it hard to tell whether the top directions reflect safety or general learning, since both data sources induce large distribution shifts unrelated to safety per se. A stronger test would include a preference‑based safety objective that explicitly trains the model to discriminate safe versus unsafe generation (e.g. DPO [1]), avoiding introducing too much safety-agnostic signals.\n- **Subspace extraction is unsupervised and layer‑agnostic**. Safety subspaces are extracted as the top‑k singular directions of layer-flattened weight updates. This can overweight non-safety layers and wash out safety features, as past works show that large weight shifts happen in later layers while safety mechanism might locate in mid layers [2]. I suggest the author to use layer-wise SVD. The author should also discuss supervised safety-subspace extraction (e.g. probing), which are shown to more effective in extracting safety-subspace. If none of these yield selective safety effects, the linear non‑separability claim is much more convincing.\n- **The harmfulness metric is confounded with general ability**. Harmfulness is graded by a model judge on AdvBench, which may confound general ability (e.g. instruction following and writing quality) with harmfulness of the model. Figures 3 and Tables 2/5 show that orthogonal projection and even random projections reduce both utility and harmfulness together, which can be due to general degradation rather than safety improvement. I suggest the author also report more robust metrics like probability of toxicity generation or further examine if harmful score changes are caused by changes in general ability.\n- **Interpretation of “energy kept” is unclear.** Energy-kept trends appear similar across choices, so it is hard to tell what semantic insight this quantity adds. More explanation will be helpful.\n\nTherefore, the claims in this paper need narrower wording or stronger evidence: \n- **Line 261: Alignment Directions Reflect General Learning, Not Safety**. The symmetry in Figure 2 and Table 1 shows that projecting useful and harmful updates onto top alignment directions increases both utility and harmfulness. This supports high‑impact general directions, but does not rule out safety linear-structure that is not captured by the SVD components. The paper should add the layerwise and supervised directions described above before generalizing this implication. \n- **Line 340: No Selective Removal Is Possible**: Orthogonal projection removes top‑k directions and causes utility and harmfulness to drop together at similar rates (Figure 3; Tables 2 and 5). However, the top-k components is not relevant to safety anyway (as per the first experiment suggested). The result may reflect choice of subspace rather than a fundamental limit.\n- **Line 461: Shared Subspaces Drive Behavior, Not Safety**: Section 6 compares activations from useful prompts (U) with two harmful sets (BeaverTails H1 and ToxiGen H2), then finds useful–harmful overlap often comparable to or exceeding harmful–harmful overlaps. Because H1 and H2 come from different sources and the activation analysis uses only the last token, the overlap patterns can reflect dataset and style similarities rather than safety. \n\nFrom the experiments in the paper, what I can agree on is that **when we SFT models on some aligned / toxic dataset, the dominant subspace of the weights update contains not only safety-specific features, but also utility features.** While this negative result has its own contribution, we need stronger arguments to prove that safety is inherently entangled with general learning and are not linearly separable. The author should address above concerns before I will raise my score.\n\n[1] Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. **A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity.** ICML, 2024.\n\n[2] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. **LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning.** NeurIPS, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)