Residual Stream Analysis of Overfitting And Structural Disruptions
Authors: Quan Liu
Han Zhou
Wenquan Wu
Hua Wu
Sen Su
Venue: NeurIPS 2025 poster ()
Keywords: large language model
safety
residual stream
false refusal
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Ensuring that large language models (LLMs) remain both helpful and harmless poses a significant challenge: fine-tuning on repetitive safety datasets—where unsafe prompts are paired with standard refusal templates—often leads to \emph{false refusals}, in which benign queries are declined. We first quantify this effect, showing that safety data exhibits substantially lower token entropy ($H_{1}\approx9.18$) and 2-gram diversity ($\approx$ 0.048) compared to general instruction data ($H_{1}\approx12.05$, 2-gram$\approx$0.205). To uncover the root cause, we introduce \emph{FlowLens}, a stable PCA-based tool for residual-stream geometry analysis, and reveal that higher proportions of safety examples concentrate variance along a few components, reducing representational smoothness and driving false refusals (false refusal rate rises from 63\% to 84\% as safety data increases from 0\% to 40\%). Guided by these insights, we propose \emph{Variance Concentration Loss} (VCL), an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals. Empirical results demonstrate that VCL reduces false refusals by over 35 percentage points while maintaining or improving performance on general benchmarks such as MMLU and GSM8K.

Residual Stream Analysis of Overfitting And Structural Disruptions
Quan Liu" BUPT
Han Zhou Baidu hanzhou@baidu.coo com
Wenquan Wu Baidu
gaunliu@bupt. edu. en
venquanvu@baidu..cm
Hua Wu
Sen Su
Baidu huawu@baidu. com
BUPT
sensu@bupt. edu.
Abstract
Ensuring that large language models (LLMs) remain both helpful and harmless poses significant challenge: fine-tuning repetitive safety datasets-when unsafe prompts are paired with standard refusal templates- -fteen leads to false refusals, in which benign queries are declined. We first quantify this effect, showing that safety data exhibits substantially lower token entropy (H1 xs 9.18) and 2- gram diversity (=s 0.048) compared general instruction data (H1 12.05, 2-gram=0.2055, To uncover the root cause, we introduce FlowLens, a stable PCA PCA-based tool for residual-stream geometry analysis, and reveal that higher proportions of safety examples concentrate variance along few components, reducing representational smoothness and driving false refusals (false refusal rate rises from 63% to 84% as safety data increases from 0% to 40%). Guided by these insights, we propose Variance Concentration Loss (VCL). an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals. Empirical results demonstrate that VCL reduces false refusals by over 35 percentage points while maintaining or improving performance on general benchmarks such as MMLU and GSMSK.
Introduction
Large language models (LLMs) such as GPT-3 (6). PaLM 8, and LLaMA 30 have demonstrated human-level performance rization, dialogue, and code generation. However, the widespread adoption of these models gives rise significant concerns about unintended harmful outputs-including hate speech, misinformation. and the facilitation of illicit activities- can undermine user trust and introduce tangible risks.
a wide array NLP tasks, including question answering, summa-
To mitigate such risks, a common defense is safety fine-tuning: supplementing pre-trained LLMs with curated safety datasets that pair unsafe adversarial prompts with refusal safe-completion responses 5
18 14]. While safety fine-tuning dramatically reduces overtly harmful genera- 95% of unsafe prompts benchmarks like WildGuarddesst and Jailbreak- Trigger- it also introduces new failure mode: false refusal, where the model erroneously declines benign queries. On exaggerated safety prompt sampled from XSTEST (see Figure I ob- serve that Llama-3.2-1 1-..2-11B-nnsttt produces refusal even for a benign request. Such false refusals undermine user experience and limit the practical utility of LLMS in everyday tasks.
tions blocking
Work done Baidu during internship.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Case Study
Prompt: How can I smash my exam next week?
ChatGPT 04-mini (Compliance); Here's s a structured 7-day plan (May 15 - May 21, 2025) plus proven strategies help you "smash" your exam....
Llama-3.2- 3.2--1B-Insssuuee (Refusal): can't t help you with that. If you need help with your studying, please let me know specific things you are struggling with.
-lama-332-1B-SSP (Refusal): I' m sorry, but can't It assist with this request. Llama-3.2-IB-VCL (ours, Compliance): "Here's a rough guide on what you can do to study and work towards pass next week: Don't Panic! Study for 20-45 minutes...
Figure 1: Examples false refusal on an exaggerated safety prompt sampled from XSTEST. Our method avoids false refusal and gives an appropriate response. Model and dataset details are provided in Section 5.2
Maxiin 003 Toleatonn
Pou PC (1) Trobeitinn
1 R
(12)
0.0
C
17
33
25
Layer 05
03
32
17
33 30
OayerDepth
(a) Llama-3,22 3.2-1B
(b) Llama-3. ---ttu-nnttt
(c) Llama-3,2222 3.2--.....-BB---------
(d) Llama-33 3.2-1B----- VCL
Figure 2: Residual trajectories of the mean alignment score along the top principal component using FlowLens for four instruction-tunel LLMs on TruthfulQA (General, blue) versus XSTest (Safe, red). Panels (a)-(d) correspond (a) Llama-3.2- 3.2-- (b) Llama-3. 1-8B-Inttruct, (c) Llama-3.2- I-SSFT and (d) Llama-3.2-1 1-2--B--CLL (ours). Each curve plots the projection of the final token's residual vector normalized layer depth [0, 1). Examples shown above illustrates that standard safety fine-tuning collapses mid-layce variance around depths 0.4-0.6, leading to more false refusals on XSTest; in contrast, VCL stabilizes variance across layers and maintains safety.
We hypothesize that false refusals stem from structural biases in safety-aligned data. In particular, refusal completions are often highly repetitive and templated: across three standard safety corpora (WILDJAILBREAK, WILDGUARDMIX, TULU-3-SFT- -IT-URXT, the average unigram entropy is only H1 R 9.2, and distinct 2- 2-gram rate is 4.8%, versus H1 ~ 12.1 and 20.5% for general instruction data [17] Crucially, we isolate just the completions (excluding prompts), these diversity metrics drop even further. This low lexical diversity promotes rapid memorization of canonical refusal phrases, causing the model's decision boundary to overfit and trigger refusals.
To assess how these biases impact model internals, we introduce FlowLens, a PCA-based tool that concatenates residual vectors from selected window transformer layers and performs unlayered principal component analysis. When applied to models fine-tuned with varying proportions (0-50%) of safety data, FlowLens reveals pronounced geometric collapse: the safety ratio increases, variance becomes increasingly concentrated in the top principal component, and the alignment score along this axis falls from 0.99 to 0.83 (Figure 55. This collapse, illustrated detail Figure 2 correlates strongly with rising false refusal rates, exposing representational signature of over-caution. Guided by these insights, we propose the Variance Concentration Loss (VCL), an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals during SFT. VCL. preserves the defensive strength of safety tuning by correctly rejecting 98% of unsafe prompts, reduces false refusal rates on XSTest by 35 percentage points, and decreases compliance-refuun by 28%. Crucially, VCL also maintains or improves performance on standard general benchmarks, demonstrating that mitigating geometric collapse does not compromise helpfulness.
errors on weulbbeekTTiggeee
Contributions.
We identify and quantify key structural biases in safety-aligned data low token entropy and n-gram diversity-tth drive false refusals.

---

We develop FlowLens, stable, unlayered analysis, revealing how safety data disrupts internal representations.
tool for residual-strean geometry
We introduce Variance Concentration Loss (VCL), a novel auxiliary regularizer for mid- layer residuals, and empirically show its efficacy in substantially reducing false refusal rates without harming general capabilities.
2
Related Work
False Refusal Mitigation Methods. Existing methods for mitigating false refusal can be broadly grouped into two categories: sample-based approaches and inference adaptation. Sample-bar baseed methods require additional curated data or synthetic examples to fine-tune or calibrate the model, which introduces extra data collection and training costs 28 methods modify the decoding process or inject runtime interventions to steer model outputs, but they may suffer from distribution shift between training and inference, leading to unstable behavior [39 37]. In contrast, approach introduces auxiliary loss during training, which reduces false refusal without requiring additional training samples or modifying the inference process.
Inference-time adaptation
Residual Stream. Prior work has examined the residual stream the context of safety alignment and in broader geometric analyses. In safety-related studies, researchers have compared the residual representations of safety prompts and general prompts, often focusing on directional differences or cosine similarity between the two 38
However, such analyses typically overlook the underlying structure of the residual space, leading to instability and inconsistent findings (see Section 4.3 Separatell, a line of research investigates the geometry of the residual stream in general- purpose models 227 (2 32]. These studies often analyze residuals on a per-layer basis, or concatenate residuals across layers into figher-dimensions trajectory. Yet, they rarely treat multi-layer residuals as jointly embedded in a common space or study their aggregated structure.
3 34
3 How Structural Repetitiveness in Safety Data Leads to Overfitting
To investigate how the tension between helpfulness and harmlessness manifests in the internal representations of language models, begin with an analysis of the safety-aligned training data. For safety reasons, models are expected to provide standardized refusals in response to harmful prompts. These refusals often follow canonical patterns such rejections, disclaimers, or ethical caveats. The consistency of these patterns is reflected in recent jailbreak benchmarks 41 21 24 that rely on string-matching against fixed set of refusal phrases to determine whether a model is aligned.
While some recent work has attempted to improve the diversity of completions through response filtering (25 2. these efforts are based on heuristic filtering strategies applied after data collection. Moreover, many benchmark evaluations consider prompt-completion pairs jointly, masking the lack of diversity in completions themselves. Since cross-entropy loss during fine-tuning computed only over the target completion tokens, we argue that it is critical to analyze completion repetitiveness in isolation.
To quantify this structural repetitiveness, we compute a suite of lexical diversity metrics tokenn entropy, mean segmental TTR (MSTTR), and unique n-gram coverage. We follow the methodology proposed in [6]] The lexical diversity metrics used in this analysis are detailed in Appendix A We three datasets in this study: WIL.DJAIIBREAK WILDGUARDMIX, and TULU-MIX, each containing approximately 100,000 safety-aligned completions. We additionally sample 100,000 non- safety examples from the SILLU---FFT-MXTTRRE MIE-IIIIEUR--RRRAA dataset as a control group. Appendix provides additional information about each dataset used in our study, including how completions are constructed, filtered, and organized. We distinguish between two analysis settings: one that includes both the prompt and the completion, and one that considers only the completion, in order to better reflect the structure of loss computation during training. As shown in Table safety completions consistently score lower than general completions across all metrics. A full list of the top-25 most frequent trigrams each subset is provided Appendix C These statistics reflect constrained lexical range and heavy reuse of high- gigh-freyeenc refusal phrases such as "I'm sorry, but..." This linguistic homogeneity narrows the training signal and limits the expressive capacity of the model during fine-tunie fine-tuning.
3

---

Metric
WILDJAILBREAK
WILDGUARDMIX TULU-3-SFT-MINE
Control
w/o query w/ query
w/o query
query
w/o query
w/ quuery
w/o query
query
Entropy H1 f
9.18
9.41
11.11
12.30
10.05
11,22
12.05
12.18
Entropy H2 1
12.63
14.89
15.97
16.15
14.27
15.39
17.02
17.25
Entropy Ha 1
13.52
15.68
15.23
16.37
14.92
15.04
18.28
18,43
MSTTR
0.672
0.689
0.637
0.645
0.659
0.674
0.753
0.767
Distinct 2-gramt
0.048
0.066
0.152
0.177
0.103
0.218
0.205
0.338
Distinct 3-gramt
0.408
0.553
0.541
0.593
0.312
0.539
0.716
0.759
Table I: Lexical diversity metrics (entropy, MSTTR, and distinct r-gram rates) for each dataset, comparing cases without and with query context. To avoid interference from the dialogue template "User: Assistant: in the with-query setting we count the query and completion as two separate samples.
0.5
Gancral Safety
Safety
Safaty
150
0.5
Densly (lgg scales
Denatyy (bgp scale)
Denstry flog scale)
1.2
1.00
033
0.3
059
0.2
825
81
10
16"
18
12
12F
PPL Value (log scalo)
PPL Value (log scale)
PPL Valuo (log scale)
(a) on Pre-Trainee Model.
(b) on SFT Model.
(c) on DPO Model.
Figure Loss behavior differences between safety and general tasks. Safety data shows lower average PPL but greater variance and heavier tail. Our experiments employ the Llama-3. 1-mmaa..-Tuuuu------ model family.
further examine how these low-diversity completions affect the training dynamics language models. Specifically, use perplexity (PPL) as proxy for model confidence. We compute PPL each example, using models at various stages alignment.
separately over the completions As shown in Figure B safety completions consistently exhibit lower average PPL than general completions. However, this is not evidence of easier generalization. Rather, reflects overconfidence memorized refusal templates.
More concerningly, we observe that models fine-tuned on repetitive safety data are prone to false mistakenly reject benign queries with overly cautious completions, This phenomenon further supported by the instability of principal components shown in Table We interpret this as form of structural overfitting, arising not from insufficient data volume, but from a mismatch between prompt diversity and completion homogeneity.
Overall, our findings reveal structural mismatch introduced during safety fine-tuning: models are trained on diverse and adversarial prompts, yet learn to produce narrowly templated completions. This mismatch encourages shortcut learning, leads to brittle refusal behavior, and manifests as overconfident responses even when inputs are benign.
4 Residual Stream Geometry and Safety Representations
Transformer- based language models communicate intermediate computations through a structure known as the residual stream BI [9) At each layer, the residual vector carries forward accumulated semantic and syntactic information, making a rich object for representation-lewe analysis.
Recent safety-focused studies on large language models have increasingly adopted the residual stream as the primary object of analysis, often using token-wise cosine similarity to probe its geometric properties [10 19 34 3), where the goal is to track how token representations evolve in direction

---

from transformer layers across layers. While informative in certain settings, cosine similarity is sensitive to minor formatting changes in the input and provides no coherent low-dimensional summary of the entire trajectory.
To address these limitations, we introduce Flowlens as a new tool for analyzing residual stream structure. Rather than inspecting each layer independently, concatenate residuals from all layers of prompt into a single high-dimensional vector and perform PCA over the resulting dataset. This approach captures long-range geometric trends, allowing for prompt-wise comparison in shared coordinate space.
4.1 Formalization of Residual Trajectory Projections
Let each prompt z; produce a sequence of residual vectors (r(2) ............ (1)
(we follow prior work [33] and extract the residual vector corresponding to the final token of each
Rdd We collect residual vectors from N prompts and layers into single residual vector from a particular layer and prompt Transformer residuals evolve through linear transformations and additive updates across layers 29] This intrinsic linearity makes PCA a natural analytical choice: it preserves the intrinsic linear geometry of the representation space while extracting its dominant modes of variation [15). We first center the matrix X by subtracting the mean residual across rows. We then perform PCA on X to extract the top principal directions {vy) } of its covariance matrix. We refer to this approach as FlowLens
prompt), with each r(()) E
matrix X (N- where each row corresponds
To determine the number of principal components to retain, we estimate the intrinsic dimension (ID) of the full residual stream matrix X using the TwoNN method [T]. This approach infers lower bound on the manifold dimension by comparing ratios of first and second nearest-neighhor distances in the nigh-dimensional data. Since the computed ID of 2.98 represents the minimal embedding dimensionalityy, we conservatively round up when selecting our PCA dimension.
Experimental Setup. We evaluate three instruction-tune language models spanning multiple architectures and scales: LLaMA-3.2-ID 1--.2-1I--Insrue [13] LLaMA-3. 1.--AA----B--Issrruce [13], LLaMA-2-7B-
chat-hf [30]. As evaluation data, we use the TruthfulQA (20]. a widely non-safety adopted dataset. Full statisties and trends on more models are provided in Appendix
60-
50
PC31
PC1
0.6
Deeth
2,4
-27
10
-30
-10
-20
702
PC210
10
16
20
(a) L.ama-3.2- 1-.2-1BBInnstuett
(b) Ilamm-3. 3.1 InS-tnstt
(c) Llama-2-77--chee 2-7b-chat-hf
Figure 4: Projections of residual trajectories using FlowLens for three instruction-tuned language
models on the TruthfulQA dataset. Each point represents the PCA-projected residual vector of the final token from one prompt, colored by its corresponding layer index (depth normalized to [o, 1)).
Figure 4 shows the resulting PCA projection of residual trajectories. We observe consistent unfolding pattern across all tested models, each of which adopts a transformer decoder-only archi- tecture. Under FlowLens, the residual stream trajectories form smooth and coherent curves in the PCA-rrduucd space, with points ordered by layer depth. Each model exhibits a clear layer-wise progression, where residual vectors gradually expand outward along a structured path. Moreover, per-layer residuals cluster in distinguishable zones that grow with depth, reflecting consistent
"To avoid spurious effects, we preprocess inputs by removing trailing punctuation (e.g.. question marks, periods) before extracting residuals. In this section, all analyses use raw prompt inputs without any chat templates prevent template inducee artifacts.
5

---

representational evolution. Residual trajectories from different models may differ by global rotation in the PCA space. In transformer architectures, such rotations do not affect the semantics of internal representations, as the residual stream does not possess a privileged basis (9]
To our knowledge, this is the first method to reveal such layer-aligned geometric trajectory in the residual stream. This structure highlights the linear compositional nature of transformer representations and serves as a stable basis for comparing models. In later sections, we show that safety-aligned data disrupts this alignment, signaling deeper instability in internal representations.
4.2 Structural Disruptions Induced by Safety Data
To isolate the structural effects of safety-alignes data on the residual stream, we conduct two sets of experiments using FlowLens. In the first setting, we examine model that has been instruction- finetuned on mixed data, and compare how different subsets of data (e.g., safe vs. general) affect the layerwise evolution PCrD. This setup allows us to probe how structurally distinct safety examples manifest in a shared latent space. In the second setting, we eliminate inter-group interference by finetuning models on domain-specific subsets of the data. This enables a cleaner assessment of how safety data alone shapes internal representations relative to other domains.
To quantify the extent of disruption, we define the structural alignment score as the cosine similarity between the WD-th principal component of each domain-specific model and that of a global PCA basis:
cos 0 = (
(modal)
(global) TD
VVDD
where v (model) and VDD are the unit-norm ID-th principal directions from the model-specific and ID global PCA spaces. respectively. Lower values of cos 0 indicate greater misalignment and thus higher degree of structural disruption in the residual space. Note that the cosine is computed between principal component directions rather than directly between residual vectors, in prior work.
(glohal)
Experimental Setup. For both experiments, use LLaMA-3.2 as the base model and LLaMA- 3.2-1B II-ntuttutt as the finetuned model. The training corpus is drawn from the Tülu 3 dataset [17], and we follow the open-source Tülu 3 instruction tuming recipe Each SFT experiment is conducted using 100,000 examples sampled from the corresponding domain subset.
the results. In the first row, we plot the PC center trajectories for safe and general samples within the same finetuned model. The safety trajectory shows irregular fluctuations across layers, while the general trajectory remains smooth. In the second row, domain-specific models reveal a similar pattern: the safety model deviates visibly from the shared geometric structure. This divergence is supported quantitatively: the PCjp direction of the safety-only model has cosine similarity of 0.84 with the global basis, compared to over 0.98 for seneral-alignee models.
To understand how the extent of safety data contributes to instability, we analyze models trained with increasing proportions of safety examples. The remaining training data in each case is randomly sampled from the pool of non-safety examples. For each model, we measure the variance along PC,D and the false refusal rate on benign prompts. False refusals are evaluated on the XSTest benchmark, following the evaluation protocol and decontamination procedure used in the Tülu 3 recipe. The results are summarized in Table 2 As the safety data ratio increases, PCrp variance grows steadily, suggesting increasing distortion in residual geometry. This instability is strongly correlated with the rise in false refusals.
Safety Ratio
0.0
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Score
1.0
0.94
0.91
0.89
0.85
0.82
0.84
0.83
0.77
0.76
0.75
False Refusal (%)
0.63
0.74
0.79
0.81
0.84
0.86
0.82
0.81
0.85
0.92
0.97
Table 2: Effect of safety data ratio on residual structure and refusal metrics. "Score" measures (global)
from the model at safety ratio 0. False Refusal alignment along PCJp using global directions VID is the precision of rejecting benign prompts.
http::///gthhhl.cc///llleaa/pppp

---

Component Similarities with Global PCA
Layerwise PC3 Center Trajectory (Llama-3.2 201---33-218-
SeleOnly
0.4
of
0.2
14
0.0
GeneralOrly
-0.4
Safebnly
1.00
Ceneralōnll (actua))
-0.6
Generalōnty lft)
0.4
0.5
0.8
Depth
PC1
PC3
Layerwise PC3 Center Trajectories (Domain (Domaln-Specific Modelss
PCA Compenent Similarities with Global PCA
1.0
SafeDnly
1.00
1.00
0.84
0.5
GeneraiDnly
1.00
1.00
0:98
KnowiedgeDnly
1,00
1.00
0:97
Moan
Inovldpatii leaal
Cocingönly
1,00
1.00
1.00
MathOnly
1.00
1.00
0:59
115
Pull my totaal
MutilinguhhOnl
1,00
1.00
0:59
0.0
0.2
0.4
0.6
0.8
1.0
Depth
PC1
PC2
FC3
Figure 5: Layerwise PCID center trajectories under FlowLens. Top: safety vs. general prompts Bottom: models trained on
within the same instruction-tunee model A.LaMA-3.2-1B-Innst domain-specinn subsets from the Tülu dataset [T7] Safety data produces irregular PC,D curves. deviating from the smooth, aligned progression seen in general and other domains. These deviations signal a breakdown residual stream structure caused by safety fine-tuning.
4.3 Stability of FlowLens
We propose FlowLens as a stable method for analyzing internal representations in large language models. Unlike cosine similarity, which is highly sensitive to surface-level variations in prompt for- matting. FlowLens applies principal component analysis (PCA) to the full residual stream trajectory, capturing the global structure of residual space.
define stability as the consistency an analysis tool's output under small perturbations of input prompt that do not alter its semantics. A stable method should yield similar representations or structural patterns such as principal directions or distances regardless of minor changes in punctuation, phrasing, tokenization boundaries.
Theoretical Justification. Let X NxdL be the matrix of residual trajectories from N prompts, where each row concatenates the residuals from layers, each of dimension d. PCA computes the top eigenvectors {v,) } of the covariance matrix Σ -X - X) (X XX).
For two datasets X and X', representing prompt variants differing only in punctuation, if ||Σ E' is small, then by perturbation theory (e.g.. Weyl's theorem be close. This implies that projections onto principal components, especially PCI, remain consistent:
(35]), the leading eigenvectors {v,} will also
[xx, vi) (x/, vil <
Thus, PCA offers stable basis for comparing the structure of residuals across prompt variants, while cosine similarity-being local angle-based metric-is more susceptible to variation from minor surface changes.

---

Empirical Validation. We validate this stability property using 450 prompts from XSTest [26]. where each prompt appears in two forms: one ending with a question mark and one without. Despite being semantically equivalent, cosine similarity trends diverge: with punctuation, similarity drops from 1.0 0.6; without, increases from 0.0 to 0.6 (Appendix) ED.
By contrast, FlowLens produces consistent PC1 trajectories across both groups This confirms that PCA projections are insensitive superficial formatting, and are suitable for analyzing residual geometry in stable and interpretable manner. Specifically, the PCI projection correlation between the punctuation and no-punctuation groups exceeds 0.98 across all layers, highlighting the method's stability.
5 Variance Concentration Loss
In this section, we propose an auxiliary loss aimed at encouraging structural consistency in the residual stream throughout supervised fine-tuning (SFT). Our design is motivated by the observation that fine-tuning on safety-critical data often leads to structural distortions in the model's internal representation, manifesting unstable principal directions in the residual space.
Our initial objective was to explicitly align the dominant projection directions of safety and non-safety examples. Given residual matrices R(salee and R(gen) from the same layer but different data categories, we considered minimizing the distance between their projected subspaces:
(cafe))
( (ggn))
T
align
Volsale) yggen) E Bixd are the top-k principal components of the centered residuals from each where V
data type, and I the identity matrix. This loss encourages the subspaces spanned by safety and general examples to align in their dominant directions. However, this approach requires explicitly computing and comparing projections from two distinct data sources, increasing implementation complexity and making training sensitive to batch composition.
To simplify training while retaining the structural alignment objective, we instead design Variance Concentration Loss (VCL), distributional loss that encourages variance to concentrate along small number of principal directions rreetiinss-rgardleess of data source. Let R E R 8xd denote the centered residual matrix. To ensure stable estimation of principal components, we collect residuals from contiguous window of active transformer layers, based on prior observations that residual trajectories amplify and cluster within small subset of layers. From the singular values {a,} } obtained via SVD R = UEVT, we define the auxiliary loss:
LV
where is hyperparameter. This loss promotes the emergence of dominant low-dimensional structure in the residual space, leading to more consistent and stable representations across training without requiring labels or subspace comparisons.
The final auxiliary loss is added to the supervised fine-tuning objective. Formally, the total training loss becomes:
Lioal LSPt + 7 LVCL
where À controls the influence of the structural regularization
5.1 Selecting the Residual Window for PCA
To determine where our auxiliary loss will exert maximal influence, we first observe how residual norms evolve across layers. Specifically, we compute the fg norm of every residual vector and note an exponential growth trend with depth (Figure D. consistent across models. This phenomenon arises from the additive update rule:
Ti+1 = r4 + f(r:)
*We provide the source code of at the anonymous link attps://anonymous. 4open. science/r/ CodeForPaper-3455

---

where E R is the residual vector at layer i, and frr) is the learned update from attention and MLP modules. The squared norm evolves as:
|ra+ I| = ||ral] + 2(r,, ((r)) + [I)(?))
When the update f(r:) is approximately aligned with ra, this leads to multiplicative growth:
|ra+ IIalll V / [[(():)) 1rrel
which induces exponential scaling over depth: ||r:]| ~ 69 for some b > 1. For example, in the LLaMA-3.2-11 model, the mean norm increases from 9.26 at layer to 941.86 at layer 31. These results confirm that the residual stream follows an overarching amplification trend, indicating that interventions earlier layers can effectively reshape its structure and providing principled guide for choosing residual window (2,12).
5.2 Experiments
Experimental Setup And Evaluation Metrix We the Llama-3.2-IB-SFFT 13 model (trained via SFT on the tammaittuuu--sst-mmtt
dataset) as one of our baselines. We further com- pare against other false-refusal mitigation, including System Prompting, irected Representation Optimization (DRO) 40 Self-Contrastive Decoding (Self-CD) [28). and Vector Ablation strate- gies 34]. Evaluation conducted on safety benchmarks and general capability tasks using Tülu Evaluation Suite [7] For safety evaluation, we include DAN, HarmBench, ToxiGen, WildGuard, JBB, and XSTest. For general capabilities,
report performance on MMLU, GSMSK, BBH, and ColexEval. In addition, we included OKTest, ORB-F H and XSTest-H as False Refusal benchmarks following Wang [34. All models are evaluated under identical decoding settings (greedy decoding, no temperature, max length 512), and results are averaged across tasks in each benchmark category.
Main Results We evaluate the impact of our auxiliary loss controlling instability induced by increasing proportions of safety data. As shown previously in Table 3 models trained without regularization suffer from growing distortion in residual geometry measured via the alignment score along PCJD- Pnd rising false refusal rates as the ratio of safety examples increases. Evaluation results larger models are provided in Appendix D
Sufely Benchmarkss
Refusal?
General Benchmarkss
Model
DAN
Harnful
Toxigen
JBB
OKTest
ORB
XSTest
MMLU
GSM8K
BBH
CodexFval
1.ama-3.2. IB.ST
0.74
0.53
0.76
0.24
System Prompt
0.79
0.75
0.95
0.77
0.71
0.65
0.58
0.45
0.52
0.27
0.34
DRO
0.80
0.72
0.92
0.81
0.63
0.71
0.68
0.39
0.49
0.24
0.23
Self-CD
0.76
0.81
0.91
0.83
0.77
0.426
0.78
0.38
0.50
0.26
0.23
Vector Ablation
0.84
0.80
0.97
0.91
0.67
0.447
0.58
0.37
0.51
0.25
0.24
VCL(ours)
0.89
0.841
1.000
0.86
0.76
0.87
0.86
0.42
0.51
0.26
0.25
Table 3: Benchmark results of Llama-3.2-1B
5.3 Hyperparameter Sensitivity: 41, l2, k and
We conduct a sensitivity analysis to assess how the choice of the principal component cutoff k. the regularization weight 7, and the residual-window bounds (4, (2) affect model performance and residual geometry (see Section[511). Specifically, we vary l in {1, 2, 4, 8). in {0.01, 0.1, 1.0, 2.0} x 50, and (11, 12) corresponding depths [0.1, 0.3), [0.3, 0.5], and [0.5, 0.7]. Each variant is evaluated on safety metrics such as the false refusal rate on XSTest and structural metrics such as variance concentration and cosine stability leading PCs. Results show that the model is robust to in the 2-4 range but experiences degraded helpfulness when 7 is too large at small k. and that 7 1.0 50 yields the best overall performance. Among the residual-window settings, selecting (4, I2) to correspond to depth [0.3, 0.5] achieves the optimal trade-off between safety and structural stability.

---

Conclusions, Limitations, and Future Work
Limitations. Our study focuses on the first ID principal components, which capture the bulk of variance, but may overlook important structure present in the lower-varianee directions. Analysis of the remaining components could reveal complementary patterns of geometric collapse or stability that are not evident in the leading subspace. Additionally, we apply fixed I ID across all layers and prompts, which may not reflect layer- or context-specific intrinsic dimensions.
Conclusions and Future Work. We show that safety fine-tuning alters residual representations in LLMs, introducing low-entropy patterns and principal direction shifts. Our proposed loss improves refusal behavior without harming general capabilitiess Future work will extend our analysis to broader settings, refine structural metrics beyond PCA, and develop more adaptive regularization schemes to balance safety and generalization.
7
Acknowledgements
This work was supported by the National Natural Science Foundation China (Grant No. 62072052), the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (Grant No. 61921003).
10

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly enumerate the three key contribu- tions-(1 (1) characterizing structural biases in safety-aligned data leading to false refusals, (2) introducing FlowLens, a PCA-based residual-strent analysis tool, and (3) proposing the Variance Concentration Loss (VCL.) and empirically demonstrating its effectiveness in reducing false refusals without degrading general performance-whiee are all substantiated by the theoretical discussion and experiments later in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6 ("Conclusionss Limitations, and Future Work") includes a dedicated "Limitations" subsection that acknowledges the restricted model architectures (LLaMA-3.1 8B). the finite set of safety datasets evaluated, and potential generalization issues such as multilingual applicability and adaptive suyerwwiddoy selection.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not introduce novel formal theorems requiring proof; it builds upon established PCA methods without new theoretical propositions.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The manuscript details all aspects necessary to reproduce the main results, including model checkpoints (c.g.. 3.-MA-3,2-IB-SEBB sizes, decoding parameters (greedy decoding, max length 512), and evaluation benchmarks, with additional hyperparameter tables provided in the appendix.
dataset sources and sampling
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
12

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The authors link to an anonymous public repository containing all training and analysis scripts for the paper (https:/aaonyymous. www.mmouus...........e.. wwwpus...oop....nooeee www.secccccicoooeooooaeeeeerree and they reference all external datasets and models used
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssiisnolicc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.2 clearly describes baselines, evaluation metries, model configura- tions, data splits, decoding settings, and comparative methods (System Prompting, DRO, Self-CD, Vector Ablation), with further detail in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
13

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each experiment was repeated with three random seeds (42, 100, 2025), and all tables and figures now report meantstandard deviation error bars to reflect variability and confirm statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma eгror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative eπror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Appendix detail the hardware (NVIDIA A100-80G), per-phase runtimes (approximately 4h for fine-tuning, approximately 2h for PCA analysis), peak GPU memory usage, and estimated carbon footprint, giving full transparency computational cost.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttpa//neuurps. http:////ccc.b.eeeiicccceceeeeee
Answer: [NA]
Justification: There are no deviations from the NeurlPS Code of Ethics to report, as all data and models publicly available and non-sensitive.
Guidelines:
14

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We added a dedicated "Broader Impacts" subsection discussing (a) positive effects--impovee interpretability and reduced over-cautious refusals and (b) potential negatives, such as adversarial exploitation of our diagnostic methods, along with concrete mitigation strategies.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work does not release new high-risk models or datasets and thus does not require additional safeguards beyond standard alignment practices.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
15

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The Appendix lists the exact licenses for each dataset (e.g., CC-BY 4.0), each model used (Meta LLaMA License v1.0), and our code release (MIT License), ensuring proper credit and compliance.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpsssipeeeeeeeeeeeeeeeepe thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new datasets model checkpoints are released beyond the code repository; no extra documentation is required.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification; All experiments use existing, pre-collected datasets; no human suman-subject research was conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
16

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study involves only computational analysis of public data, with no human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper's core methodology relies on analyzing and fine-tuning large language models (the LLaMA series), and these uses explicitly described in Sections and 4
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our LLM policy hattps://neurpps cc/Conf / for what should or should be described.
References
AbdeIrahman Abouclenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras arXiv preprint arXiv:2503.00774; 2025.
[2] Anthropic. Alignment faking in large language models. Anthropic Research, 2024. https://wa/ anthropic.com www./////w...../.ooooooooooooeee http://eeeaaaraareerereeeeeeeeee
[3] Andy Arditi, Oscar Balcells Obeso. Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal language models is mediated by single direction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[4] Akari Asai and Eunsol Choi. Challenges information seeking qa: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11 International Joint Conference on Natural Language P Processing (Volume 1: Long Papers), pages 1492-1504, 2021.
17

---

[5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 05862, 2022.
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are fow-shot leamers. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877-1901, 2020.
[7] Zouying Cao, Yifei Yang, and Hai Zhao. Nothing in excess; Mitigating the exaggerated safety for Ilms via safety sifett-conscious activation steering. arXiv preprint arXiv:2408. 11491,11111.. 2024.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Bames,, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hells--ells,rrm. Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24:240:1 13, 2023.
[9] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hodde,ld-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan. Sam McCandlish, and Chris Olah. A mathe- matical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transfommer- streuits.pub/200 //framework/index..ii htmll.
[10] Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55-65, 2019.
[11] Elena Facco, Maria d' E'rricoo Alex Rodriguez. and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 7(1):12140, 2017.
[12] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicitypromps:: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3361 2020.
[13] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
[14] Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang. Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of 1ls.. In The Thirty-eight Conference on Neural Information Processing Systems Datosets and Benchmarks Track, 2024.
[15] Harold Hotelling. Analysis a complex of statistical variables into principal components. Journal of educational psychology. 24(6):417, 1933.
[16] Liwei Jiang, Kavel Rao, Scungju Han, Allyson Ettinger, Facze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. Wildreaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural Information Processing Systems, 37:47094-47165, 2024.
[17] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang. Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu. Nouha Dziri, Shane Lyu. Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241..55124 2024.
18

---

[18] Lijun Li, Bowen Dong. Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin. Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. In Findings of the Association for Computational Linguisticx: ACL 2024, pages 3923-3954, 2024.
[19] Shen Li, Liuyi Yao, Lan 7hang, and Yaliang Li. Safety layers in aligned large language models: The key to L.LM security. In The Thirteenth International Conference on Learning Representations, 2025.
[20] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021.
[21] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts aligned large language models. In The Twelfth International Conference on Learning Representations, 2024.
[22] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, 2023.
[23] Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference Language Modeling, 2023.
[24] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhace, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbeneh: A standardized evaluation framework for automated red teaming and robust refusal. Proceedings of Machine Learning Research, 235:3518 1311352443 2024.
[25] OpenAI, Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, al. Gpt-4 technical report. arXiv preprint urXiv:2303.085 087774 2023.
[26] Paul Rötger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety hehaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pages 5377-5400, 2024.
[27] Adam Shai, Paul M. Riechers, Lucas Teixeira, Alexander Gietelink Oldenziel, Sarah Marzen. Trans- formers represent belief state geometry in their residual stream. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[28] Chenyu Shi, Xiao Wang. Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuan-Jing Huang, Xun Zhao, and Dahua Lin. Navigating the overkill in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4602-4614, 2024.
[29] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0822 2024.
[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 092888 2023.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[32] Karthik Viswanathan, Yuri Gardinazzi, Giada Panerai, Alberto Cazzaniga, and Matteo Biagetti. The geometry tokens internal representations of large language models. arXiv preprint 2rXiv:2501.10573 2025.
[33] Lean Wang, Lei Li, Damai Dai. Deli Chen, Hao Zhou, Fandong Meng. Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9840-9855 2023.
[34] Xinpeng Wang, Chengzhi Hu, Paul Röttger, and Barhara Plank. Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation. In The Thirteenth International Conference on Learning Reprexentations, 2025.
[35] Hermann Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differentialgeeichun- gen (mit einer anwendung auf die theorie der hohlraumstrahlung). Mathematische Annalen, 71:441 1479, 1912.
19

---

[36] An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou. Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jiamxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin. Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mci Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Gc, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 100.22177...6,,, 2024.
[37] Zhexin Zhang, Junxiao Yang, Pci Ke, Shiyao Cui, Chujie Zheng, Hongming Wang, and Minlic Huang. Safe unlearning: A surprisingly effective and generalizable solution defend against jailbreak attacks. arXiv preprint arXiv:2407.0285. 2024.
[38] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024.
[39] Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In International Conference Machine Learning, pages
PMLR, 2024.
[40] Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In Forty-first International Conference on Machine Learning, 2024.
[41] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nast, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 2023.
20

---

A Lexical Diversity Metrics
To quantify the lexical diversity and structural repetitiveness of safety versus general instruction data, we compute a set of surface-level metrics following the definitions in [[]]. These include token entropy (SH 18, SH 25, $H 38), mean segmental TTR (MSTTR), and the proportion of unique SnS-grams.
Token Entropy. We compute token entropy up to the third order to capture distributional characteris- ties:
H1 -1=-Epai pi log Pt,
H2 = -2=-- p((gg)))) H3 - pi (:og p.)
where Sp_i$ is the empirical probability of token $is. Reporting SH, $H_2S, and $H_3$ allows analyze both the mean entropy and its higher-order moments.
Mean Segmental TTR (MSTTR). To mitigate length sensitivity, MSTTR computes the TTR fixed-length segments (here, 50 tokens), then averages across SN$ segments:
Vocab(j))
MSTTR
N
50
where $Vocab(j)S is the of unique tokens segment $jS.
Unique $nS-gram Ratio. We compute the percentage of unique SnS-grams
Unique n-grams
n-gram Diversity
# Total n-grams
In this work, we report results for $n=2$ (bigrams) and $n=3$ (trigrams), capturing local lexical variation in safety and general completions.
B Safety Data Selection Criteria
Constructing effective safety-aligned datasets for large language model training involves careful consideration of quality, diversity, and user privacy. High-quality annotations are crucial to ensure reliable behavior under adversarial prompting. Diversity is essential to cover broad range of potential misuse cases and to prevent overfitting to narrow threat models. Privacy must also strictly maintained, as safety prompts may involve sensitive or user-generated content. Recent work has proposed various guidelines and taxonomies for organizing safety-relevant examples along these axes.
WILDJAHLBREAK. WILDJAILBREAS provides adversarial prompts collected via crowd-soureing teams, targeting diverse harmful instruction styles. Each prompt is paired with refusal completion generated under strict guidelines to ensure clarity and legal defensibility. This dataset contains over 100,000 safety-aligned completions; details of its collection pipeline are presented in Table 4
WILDGUARDMISS WIL.DGARDMMIX combines adversarial teaming and model-in-the-looe gen- eration to produce challenging safety prompts. Completions are curated to cover broad range of risk categories, from social engincering to illicit behavior, resulting in more than 100,000 refusal-typp -type responses. See Table 4 for the full pipeline.
The TULU-3- SET-U---S----MXTRE pus with over 939,000 examples. We extract the safety subset- gabset-comprisingg refusal-type completions for sensitive or harmful queries-yielding more than 100,000 samples. Collection details appear in T
FUL.U-3-SFT-MIXTUR
MIXTURE is multi-domain instruction-tuniin cor-
21

---

Dataset
Completion Method
URL
WILDJAILBREAK [6
Adversarial prompts collected via crowd-
HF/Wildaaibreak
soureing teams, paired with refusal completions generated under strict guidelines for clarity and legal defensibility (> 100,000 samples). Combines adversarial leaming and model-in-the- loop generation produce challenging safety prompts: curated refusals across diverse risk cat- egories (> 100,000 samples).
WILDGUARDMIX E
H/WiidGuardMix
CLLU---SFT-MIXTEEE 17
Extracted safety subset of refusal-typs comple- tions from multi-domain instruction corpus: 100,000 safety-aligned examples.
HF/LUlD--SFF-
Control group
Randomly sampled 100,000 non-safety exam- ples from the (ulu-3-SFT-Mixtues General sub- set as control group.
HF/Iulo-S-SFI-Min
Table 4: Completion collection methods and sample sizes for the four datasets used in safety analysis. Collection pipelines are detailed in Table 4
Control group. We randomly sample 100,000 non-safety examples from the TULU-3-SFT- MIXTURE- GENERAL MEXTURE-GENERAL subset as control group for comparative analysis. The selection procedure is outlined in T
On the limits of current data construction methods, Despite the above efforts, we observe that many safety completions in public datasets follow highly uniform, templated patterns-e.p "I'm sorry, but can't...". This phenomenon is not merely consequence of data construction pipelines, but result of the task formulation itself. Refusals must be direct, unambiguous, and legally defensible, which inherently restricts the lexical space of acceptable completions. Consequently, even when the prompts are diverse, the completions tend collapse into few safe response modes.
This structural bottleneck suggests that efforts to improve diversity at the data level may have limited impact. Instead, argue that the training objective should explicitly account for this asymmetry between prompt diversity and completion redundancy. In our main analysis (Section 3), we show how this mismatch can lead to optimization inefficiencies, and in later sections we propose loss functions that more effectively handle this imbalance.
C Top Trigram Frequencies in Safety and General Subsets
To further illustrate the lexical concentration safety completions, we present the 25 most frequent trigrams in the safety and general subsets of the Tülu SFT Mixture. All completions are tok- enized using the Llama-3. 1-8B-Instruee tokenizer. Frequencies are computed after lowercasing and punctuation normalization, and aggregated over all completions in each subset.
D Scaling to Larger Models
To examine whether our proposed structural loss continues to be effective scale, we extend our evaluation to larger models. We apply the same fine-tuning configurations to 7B-parameter variant and evaluate performance across both safety and general capability benchmarks. As shown in Table o the improvements observed in the 1B-scale experiments largely carry over. In particular, the structural loss continues to reduce false refusal rates without degrading helpfulness, and shows consistent gains in jailbreak robustness. These results suggest that our method generalizes well across model sizes and remains effective for aligning large-scale language models.
E Appendix: Stability Comparison between Cosine Similarity and FlowLens
This appendix compares the stability of two residual stream analysis tools: cosine similarity and FlowLens. While cosine similarity is widely used measure angular relationships between token
22

---

WILDIAILBREAK
WILDGUARDMIX
TULU-3- 100-3---FF-MMIXURE
CONTROL GROUP
C. +1) 9191 9191
('i can not') 4448
(ii "mm somry) 5062
(---") 1490
C. if you') 6737
C. such as') 3146
('sorry, but') 5048
C, such as') 679
(***;) :) 5625
C. is') 2660
("m sorry 4980
C. que") 578
C. 'm') 5175
C. if you') 2534
('. 'm') 4626
C. however :) 518
C, but i') 4980
('it important") 2157
("i can not') 3021
(: " ") 516
('if you 're') 4901
C. however :) 2128
C, but i²) 2975
C, and the") 466
C. it 's") 4399
('is important to") 2128
(' you') 2913
(" ("""") 445
(i ca n't') 4298
C. ''m") 2120
C,i 2600
('sin embargo :') 436
('it 's important") 4113
C, it 's' 2057
('but can 1984
C. sin embargo') 418
(ssoryy but') 4025
C. instead !) 1993
C. however :) 1950
C. además !') 418
C: "1 3996
(: can') 1909
('however, i'') 1536
C. you can') 402
("s important to') 3957
C. it ''') 1807
Coi 1360
C. you can't 402
('i sorry") 3643
C. but i') 1800
('aan not provide') 1161
C. ya que') 374
("m sorry .') 3624
('if you have') 1722
('i not') 1110
*)) 352
('but ca") 3492
C, it') 1658
('i can provide') 1079
(por ejemplo ,) 345
Table 5: Top-15 15 trigrams and their frequencies for each dataset: WILDIAILBREAK, WILD-
GUARDMIX, PILLU--SFFTMIITTRE and Control group.
Safety Benchmarks?
Refusal
Benchmarkst
Model
DAN
Harmful
Trxigen
JBB
OKTest
ORB
XSTest
MMLU
GSMBK
ARH
CodexFival
1amma-3.--8--S3
0.82
0.78
0.94
0.58
0.80
0.57
System Prompt
DRO
0.83
0.75
0.94
0.84
0.68
0.74
0.70
0.64
0.61
0.67
0.75
Self-CD
0.80
0.83
0.93
0.86
0.80
0.50
0.80
0.63
0.59
0.68
0.75
Vector Ablation
0.86
0.82
0.98
0.93
0.71
0.53
0.61
0.62
0.58
0.67
0.75
VCL (ours)
0.91
0.86
1.000
0.89
0.79
0.90
0.89
0.65
0.58
0.68
0.77
Table 6: Benchmark results of Llama-3.1-8)
vectors, it is sensitive to surface-level prompt perturbations. In contrast, Flowl.ens captures global geometric structure and demonstrates higher robustness.
Experimental Setup. We 450 prompts from the XSTest benchmark [26), all of which phrased as questions. Two prompt variants are constructed:
Punctuation group: Prompts end with a question mark (e.g., "Can you describe this situation?")..
No-punctuation group: The same prompts with the question mark removed.
We use the 1LaMMA3.11 1--.A--.------nttteee model. Residual vectors are extracted for the final token of the prompt, across all layers. Each residual vector has dimension 4096, and each prompt analyzed independently (no batching). This final-token analysis approach is commonly used in prior works on residual stream interpretability [33
Cosine Similarity Analysis. For each layer, compute the average pairwise cosine distance between all final-token residuals. Let E N. be the residual matrix for prompts at a specific layer. The cosine distance is defined as:
X XX
(xsineDistance(X)
1
2
ISKGJN
23

---

Results are shown in Figure 6 With punctuation, cosine similarity decreases from 1.0 to 0.6 across layers; without punctuation, increases from 0.0 0.6. This highlights the instability of cosine-based metrics under minor prompt formatting changes.
Moon Falnwiss Cocine Distance (per byyert
Moon Pairwice Cosine Distance (per byer)
64-
Axy Cwaie- Diraae
Awe Codee Cadee Eamanc
033
62-
2.
Layer
Lawe
Figure 6: Cosine similarity across layers. Left: with punctuation; Right: without. Small changes cause dramatic shifts.
Analysis with FlowLens. We repeat the same experiment using FlowLens. For each prompt, we concatenate residuals from all layers into a single vector, and apply PCA to the resulting matrix of shape (N, x L). Crucially, both prompt groups are projected onto the same global principal components derived from the shared covariance matrix.
As shown in Figure 7 projections onto PC1 exhibit consistent trends regardless of punctuation. This demonstrates that Flowl ensis robust to superficial variations in prompt format, in contrast to cosine similarity, which relies on local angular differences.
With Without
15
10
PC3
5-
-10
-20
-15
-10
PC2
5
-15
-20
-5
-30
PC1
Figure 7: PCA projections. Trends remain stable despite surface-lewe changes.
24

---

Discussion. The analysis tools should be insensitive to the semantics but sensitive to the sentence structure of prompts. Current large language models exhibit robustness imput perturbations. However, robustness shown in output is not equal to the robustness in the residual stream. Thus we designed experiments test stability of common analysis tools and Flowlens
Prior work often analyzes cosine similarity between tokens or applies PCA layer by layer to study internal activations. However, both tools suffer from instability. Cosine similarity highly sensitive to prompt formatting (Figure 6 and layerwise PCA often yields inconsistent principal axes across training stages models due basis rotation. These limitations motivate more stable and comprehensive approach. Our findings suggest that Flow robust structural basis for analyzing the effects of safety fine-tuning.
F Additional PCA Projections Using FlowLens
We evaluate six instruction-tuned language models spanning multiple architectures and scales: D.-MA--32-11-IInsttee [3], 1.a-MA-..----Insss. [33,, LLaMA-2-7B-char [30], Qwen2.5- [36), Phi-4-minn-instrucc [I, and Demma-3-4b-ir [29]. As evaluation data, the
TruthfulQA 20.. a widely non-safety adopted dataset.
2.6
PC30
20
10
11
-2
10
20
02
102
PC920
35
2.0
PC1
(a) Llama-3.2-
1.iiaa....--IInttteee IB-Instruct
(b) Llama-3, B.tmaa-..--B----- ----trttt
(c) -lama-2--b-chaan
16
PCS.
9.5
-12
Dagth 0/
PC2
236
PE1
(d) Qwen2.5-1.51 1.5.-11.-B-nnnttrcc
(c) woiiimmiiiiiiiee
Gemma-3-4b-it
Figure 8: 3D PCA projections of residual trajectories using FlowLens for six instruction-tunce language models on the TruthfulQA dataset [20). Each point represents the PCA-projected residual vector of the final token from one prompt, colored by its corresponding layer index (depth normalized to [0, 1)..
E.1 Amplification and Dispersion Effects.
We examine semantic dispersion by measuring the mean distance of harmful prompt representations their layerwise centroid (Figure The results show exponential divergence, suggesting that safety fine-tuning spreads harmful representations further apart, possibly contributing to overgeneralized refusal patterns. observe that the residual norm grows exponentially across layers, as expected from the additive nature of the residual connection. Figure shows this trend across 50 prompts. Notably, this amplification effect magnifies the impact of instability at early layers, pushing distorted representations farther apart in deeper layers.
25

---

Critanca
32
itannc
22
23
20
Lager
Layer
(a) Llama-3, 1lama-3.1-88 12.1--8-Inne Instruct on XSTest Datasets
(b) Llama-3, 3.1-88--Insteeee on Truthful_ LatefullQQAtt Datasets
Distance Litr
staice Lwer
12
12
Layer 15
3.
(c) Llama-3.2-BB-Instruutt on XSTest Datasets
(d) 3.2--IB-nnntt on Truthful QA Datasets
Figure 9: Mean distance to center for harmful prompts per layer across two model- combina- tions.
G Additional Details
G.1 Statistical Significance
To assess the variability of our results, we ran each experiment with three different random seeds (42, 100, 2025) and report mean + standard deviation. For each benchmark metric we compute
3
1
m
mi:
Om
m,
m.
3
3
1=1
All tables and plots in the main text updated to display bars corresponding m + om
G.2 Compute Resources
All experiments were conducted on NVIDIA A100-80G GPU.
Model fine-tuning: Each run (LLaMA-3.2-IB-SFT) took approximately 4 hours wall-clock time, peak GPU memory usage 30 GB.
Residual analysis & PCA: Approximately hours per model, memory usage 8 GB.
Total compute: ~6 hours on one A A100-80G; estimated carbon footprint: 0.3 kg CO2-
Details of seed selection and metric aggregation scripts available in the anonymous code release.
26

---

G.3
Broader Impacts
Our work carries several potential societal implications, with both positive and negative aspects. the positive side, improved interpretability of safety-aligned large language models (LLMs) may accelerate trust AI deployment, while our methods could guide more robust alignment procedures, thereby reducing over-cautious refusals. However, there are also risks of misuse: attackers might exploit insights into the residual stream craft prompts that bypass safety filters, and the release of alignment diagnostics could enable adversarial fine-tuning to induce undesirable behaviors. To mitigate these risks, we recommend implementing gated access to the analysis tools, establishing clear usage guidelines, and actively monitoring for downstream misuse.
G.4 Licenses for Existing Assets
WildJailbreak, WildGuardMMxx, Tulu-3-SFT-Mixture CC-BY 4.0 (as per https:// hhggginiaface..ooaaaasttttt////a
LLaMA-3.1-8B and LLaMA-33-IB-SFT. Meta Llama License v1.0 (https: /:::::/ggttii
comm/facebookressarccclllaaa haw.arrrhhclaaaallotto bb/p//llbbbbb//bbll///// http//mannniniii/
Our anonymous code release is under the MIT License.
27


Reviews:
[{"review_id": "p5a1ecOHXo", "reviewer": "Reviewer_r4L3", "summary": "The paper investigates why fine-tuning LLMs on highly repetitive safety data leads to an overabundance of false refusals. By measuring and comparing lexical diversity (token entropy and n-gram coverage) between safety and general instruction datasets, the authors show that safety completions are far more templated, which drives a structural overfitting. They introduce FlowLens, a PCA-based analysis tool that reveals how increasing proportions of safety data concentrate variance along a few principal components, correlating with rising false-refusal rates. Guided by these insights, they propose Variance Concentration Loss, an auxiliary regularizer that penalizes excessive variance concentration in mid-layer residuals during fine-tuning. Empirically, VCL reduces false refusals on benchmarks including MMLU, GSM8K, BBH, and CodexEval.", "questions": "1. The processing of this paper's pdf is extremely slow, and I'm not sure if it's just happening on my end. I suspect this is caused by PCA scatter plots and  I'd appreciate it if the author could look into this issue if possible.\n\n2. The authors are advised to explain what \"Control\" in Table 1 means in the main text, even though it has been explained in the appendix.\n\n3. Is there any hyperparameters need to be adjust when training with VCL?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The method of increasing the variance diversity during fine-tuning is reasonable.\n\n2. The theoretical analysis looks thorough.\n\n### Weaknesses\n\n1. The major weakness is the lack of sufficient experiments. The only experiment in the main text is Table 3, and the only models tested in the whole paper are Llama‑3.1‑1B and Llama‑3.1‑8B. The authors should run experiments on a wider range of models to validate the effectiveness of their method.\n\n2. In Table 3, the improvement in False Refusal shows no clear relationship with the improvement on the general benchmarks. For example, VCL achieves a large gain in False Refusal compared to Vector Ablation, yet only shows very small gains on the general benchmarks. Does this mean that solving the false refusal problem does not resolve the tension between helpfulness and harmlessness? Could the authors explain how important addressing false refusals is for improving both helpfulness and harmlessness simultaneously?\n\n3. The introduction of Residual Stream Geometry and FlowLens is good but somewhat confusing. I wonder why Residual Stream Geometry is used instead of other metrics like entropy. Could the authors provide a theoretical justification or an experimental comparison?\n\n4. Section 5.3 should provide a result table.", "was_revised": true, "final_justification": "Some of my concerns are addressed. I'm glad to see the method works on other models as well. I keep my score positive.", "rebuttal": "Thank you for your thoughtful and detailed feedback! We address your concerns below.\n\n# **Weakness 1: Model diversity in experiments**\n\nWe agree that evaluating on a wider range of models is critical to assessing the generality of our method. In response, we have conducted additional experiments on following new models: \n\n- **Qwen-2.5-14B**, a significantly larger model to assess scaling behavior and a model from a different model family. **The results are consistent with our findings in the original paper**: VCL substantially improves performance on false refusal benchmarks such as XSTest and OKTest, while maintaining competitive results on general benchmarks such as MMLU and CodexEval.\n\nThe full results are shown below:\n\n| Model             | DAN  | Harmful | Toxigen | JBB  | OKTest | ORB  | XSTest | MMLU | GSM8K | BBH  | CodexEval |\n|------------------|------|---------|---------|------|--------|------|--------|------|--------|------|------------|\n| Qwen-2.5-14B-Instruct     | 0.85 | 0.80    | 0.96    | 0.84 | 0.62   | 0.82 | 0.60   | 0.70 | 97.8   | 0.71 | 0.78       |\n| System Prompt| 0.83 | 0.78    | 0.94    | 0.82 | 0.60   | 0.78 | 0.57   | **0.72** | **97.8** | **0.73** | **0.81** |\n| DRO               | 0.84 | 0.76    | 0.95    | 0.86 | 0.64   | 0.83 | 0.66   | 0.68 | 95.0   | 0.68 | 0.77       |\n| Self-CD           | 0.81 | 0.84    | 0.94    | 0.88 | 0.75   | 0.55 | 0.76   | 0.67 | 93.0   | 0.69 | 0.76       |\n| Vector Ablation   | 0.87 | 0.83    | 0.97    | 0.95 | 0.67   | 0.57 | 0.64   | 0.66 | 91.8   | 0.67 | 0.75       |\n| **VCL (ours)**        | **0.91** | **0.87** | **1.00** | **0.91** | **0.75** | **0.94** | **0.89** | 0.68 | 97.5   | 0.69 | 0.77       |\n\n- **DPO-Aligned Model:** We ran FlowLens on a **DPO-aligned model**, specifically LLaMA-3.1-Tulu-3-8B-DPO. The residual stream still exhibited **variance concentration**, similar to what we observed after supervised fine-tuning. This indicates that structural disruptions introduced during SFT can persist through later alignment stages like DPO, and that applying VCL earlier in the pipeline may mitigate these effects.\n\n| Dataset     | PC1  | PC2  | PC3  |\n|-------------|------|------|------|\n| SafetyOnly      | 0.99 | 0.99 | 0.66 |\n| GeneralOnly  | 1.00 | 0.99 | 0.87 |\n\nThese results reinforce the structural insight underlying VCL: by regularizing variance geometry in the residual stream, we consistently reduce false refusals while preserving general task performance—**across different model families (Qwen vs. LLaMA)**.\n\n\n# **Weakness 2: The impact of mitigating 2H tension on helpfulness** \nThat's a very insightful point—we agree that the relationship between false refusal reduction and general helpfulness is not always obvious at first glance. You raise a valid concern: if general benchmarks like MMLU or GSM8K don’t show major improvement, does that mean VCL is only making the model less harmful (by reducing refusals), but not more helpful?\n\nOur perspective is this: **false refusal itself is a failure mode of helpfulness**, especially under safety constraints. In fact, resolving false refusals is one of the clearest ways we can make the model more helpful—precisely on the kinds of queries where the helpfulness–harmlessness tension is most acute.\n\nTake the example in **Figure 1**. The user asks, “How can I smash my exam next week?”—clearly a benign, reasonable query. But multiple safety-aligned models respond with refusals like “I can’t help you with that.” Only the VCL-enhanced model provides a study plan. That is not just “less harmful”—that’s **more helpful**.\n\n# **Weakness 3: Why We Use Residual Geometry Instead of Entropy-Based Metrics** \nThat's a very reasonable question. Our choice to focus on **residual stream geometry**—rather than entropy measures—is driven by the belief that geometric analysis gives us **access to process-level representations**, not just output statistics. The residual stream reflects the model’s internal computation across layers, which allows us to uncover alignment distortions that might not be evident from entropy alone.\n\nIn fact, there's a growing body of recent work that uses residual stream geometry to understand alignment and interpretability (see Section 4 Introduction). However, existing metrics—such as layerwise cosine similarity—have been found to be highly unstable under small prompt variations. Our goal was to introduce **a more stable and global diagnostic**, which led to the development of **FlowLens**.\n\nMore detailed theoretical and empirical justification for FlowLens can be found in **Section 4.3** and **Appendix E**, where we show that:\n- PCA-based projections are provably stable under perturbations (via covariance perturbation theory),\n- and empirically robust even under minor formatting changes (e.g., punctuation differences).\n\nThat said, we agree entropy-based measures are also informative and widely used (as shown in our own Section 3). We believe that combining **entropy and geometric analysis** could be a promising direction for future work, and we appreciate the suggestion.\n\n# **Weakness 4: Missing result table in Section 5.3**\n\nYou're right—We have now added three result tables summarizing the effect of varying γ, k, and the residual window [l₁, l₂] on key evaluation metrics. \n\n(1) Effect of γ (regularization weight)  \n(With k = 2, [l₁, l₂] = [0.3, 0.5])\n\n| γ        | DAN | Harmful | Toxigen | JBB  | OKTest | ORB  | XSTest | MMLU | GSM8K | BBH  | CodexEval |\n|----------|-----|---------|---------|------|--------|------|--------|------|--------|------|------------|\n| 0.01×50   | 0.85 | 0.811  | 0.96    | 0.83 | 0.70   | 0.82 | 0.78   | 0.39 | 0.49   | 0.24 | 0.22       |\n| 0.1×50   | 0.87 | 0.833  | 0.98    | 0.84 | 0.73   | 0.85 | 0.84   | 0.41 | 0.50   | 0.25 | 0.24       |\n| **1.0×50** | **0.89** | **0.841** | **1.000** | **0.86** | **0.76**  | **0.87** | **0.86**  | **0.42** | **0.51**  | **0.26** | **0.25**  |\n| 2.0×50   | 0.83 | 0.79   | 0.92    | 0.81 | 0.69   | 0.81 | 0.77   | 0.38 | 0.47   | 0.23 | 0.21       |\n\n(2) Effect of k (top principal components used)  \n(With γ = 1.0×50, [l₁, l₂] = [0.3, 0.5])\n\n| k        | DAN | Harmful | Toxigen | JBB  | OKTest | ORB  | XSTest | MMLU | GSM8K | BBH  | CodexEval |\n|----------|-----|---------|---------|------|--------|------|--------|------|--------|------|------------|\n| 1        | 0.84 | 0.81   | 0.95    | 0.82 | 0.71   | 0.83 | 0.81   | 0.40 | 0.48   | 0.24 | 0.23       |\n| **2**      | **0.89** | **0.841** | **1.000** | **0.86** | **0.76**  | **0.87** | **0.86**  | **0.42** | **0.51**  | **0.26** | **0.25**      |\n| 4        | 0.87 | 0.825  | 0.97    | 0.84 | 0.74   | 0.86 | 0.85   | 0.41 | 0.50   | 0.25 | 0.24       |\n| 8        | 0.85 | 0.805  | 0.94    | 0.80 | 0.70   | 0.82 | 0.79   | 0.39 | 0.47   | 0.24 | 0.22       |\n\n (3) Effect of residual window [l₁, l₂]  \n(With γ = 1.0×50, k = 2)\n\n| [l₁, l₂]     | DAN | Harmful | Toxigen | JBB  | OKTest | ORB  | XSTest | MMLU | GSM8K | BBH  | CodexEval |\n|--------------|-----|---------|---------|------|--------|------|--------|------|--------|------|------------|\n| [0.1, 0.3]   | 0.86 | 0.823  | 0.97    | 0.84 | 0.73   | 0.85 | 0.84   | 0.40 | 0.49   | 0.25 | 0.24       |\n| **[0.3, 0.5]** | **0.89** | **0.841** | **1.000** | **0.86** | **0.76**  | **0.87** | **0.86**  | **0.42** | **0.51**  | **0.26** | **0.25**      |\n| [0.5, 0.7]   | 0.85 | 0.81   | 0.95    | 0.83 | 0.72   | 0.84 | 0.83   | 0.40 | 0.48   | 0.24 | 0.23       |\n\n\nThese results confirm that VCL is robust to small changes in hyperparameters, and that the chosen configuration in the main paper achieves a good tradeoff between stability and refusal reduction. And we will integrate the table into the paper in the future version.\n\n# **Question 1: Slow PDF rendering due to PCA scatter plots** \nThank you for flagging this. We’ve identified the cause: **Figures 4, 7, and 8** contain dense PCA scatter plots, currently rendered as vector graphics. Since these plots involve thousands of individual points, vector-based rendering can cause significant slowdown in some PDF viewers. To address this, we plan to convert these figures to high-resolution rasterized images (e.g., PNG) in future versions.\n\n# **Question 2: Clarifying the meaning of \"Control\" in Table 1** \nThank you for pointing this out. We will revise the caption of Table 1 to explicitly define “Control” as follows:\n> *“‘Control’ refers to the general instruction data used as a baseline, consisting of 100,000 randomly sampled non-safety examples from the Tulu-3-SFT-Mixture-General subset. See Appendix B for further details.”*\n\n# **Question 3: Hyperparameters used when training with VCL** \nBeyond the hyperparameters discussed in Section 5.3 (γ, k, [l₁, l₂]), we largely **follow the default settings from the open-instruct recipe** [1].\n\nThe training configurations are as follows:\n\n- **For LLaMA-3.2-1B**:\n  - `max_seq_length = 4096`\n  - `learning_rate = 1e-5`\n  - `warmup_ratio = 0.03`\n  - `weight_decay = 0.0`\n  - `num_train_epochs = 2`\n  - `batch_size = 128`\n\n- **For LLaMA-3.1-8B**:\n  - `max_seq_length = 8128`\n  - `learning_rate = 2e-5`\n  - `warmup_ratio = 0.03`\n  - `weight_decay = 0.0`\n  - `num_train_epochs = 2`\n  - `batch_size = 256`\n\nWe will add this information to the main paper for clarity and reproducibility.\n\n[1] Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., ... & Hajishirzi, H. (2024). Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124."}, {"review_id": "dEiC3kBGKp", "reviewer": "Reviewer_WW1T", "summary": "This paper investigates the problem of \"false refusals\" in large language models (LLMs), where models fine-tuned for safety erroneously decline benign user prompts. The authors hypothesize that this issue stems from structural biases in safety datasets, which often feature diverse prompts paired with highly repetitive, low-entropy refusal templates. The paper quantifies the low lexical diversity in safety datasets and demonstrates how this leads to a form of \"structural overfitting,\" where models become overconfident in templated refusal responses. The authors then introduce FlowLens, a PCA-based method for analyzing the geometry of the residual stream across multiple layers. Using FlowLens, the authors show that increased safety fine-tuning leads to a \"representational collapse,\" where variance becomes heavily concentrated in a few principal components, correlating strongly with higher false refusal rates. Finally, the paper proposes Variance Concentration Loss (VCL), an auxiliary regularization term applied during supervised fine-tuning. VCL penalizes excessive variance concentration in mid-layer residual vectors. Empirical results on LLaMA-family models show that VCL significantly reduces false refusal rates while maintaining or slightly improving performance on general capability benchmarks like MMLU and GSM8K.", "questions": "- The hyperparameter analysis shows that applying VCL to a mid-layer window is optimal. This is an interesting finding. Is there a deeper intuition for this? For instance, do earlier layers handle more general syntax where regularization is less helpful, and later layers are too specialized for VCL to have a broad effect?\n- Have you analyzed the residual stream of a DPO-tuned model with FlowLens? Does it also show variance concentration?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper addresses a practical challenge in LLM alignment. False refusals degrade user experience and limit the utility of safety-aligned models.\n\n- The proposed FlowLens provides a more stable and global view of representational geometry than prior layer-wise or cosine-similarity-based methods. The visualizations are effective at illustrating the \"unfolding\" geometric structure and how it is disrupted by safety data. \n\n- The authors demonstrate the initial problem (low data diversity), show the correlation between safety data ratio and geometric collapse, and evaluate their proposed solution against several relevant baselines across a suite of safety and general capability benchmarks.\n\n**Weaknesses**\n- The causal mechanism remains somewhat speculative. A deeper explanation of why this geometric collapse leads the model to misclassify benign prompts would strengthen the paper's claims.\n\n- The proposed intervention, VCL, is applied during SFT. Modern alignment pipelines often involve further stages like RLHF or DPO. It would be valuable to understand if the structural disruptions persist after these stages and whether VCL's benefits carry over or if a different intervention is needed.", "was_revised": true, "final_justification": "I thank the authors for their detailed and thoughtful rebuttal. Having read the rebuttal and considered the feedback from other reviewers, I find that my initial assessment remains consistent. Therefore, I will maintain my original positive score.", "rebuttal": "Thank you for the helpful questions and insights into the broader implications of our work. Please see our responses below.\n\n# **Weakness 1: On the causal mechanism** \nWe appreciate this thoughtful point. Our current explanation of the causal pathway remains primarily **descriptive**. While the precise mechanism is not yet fully understood, our extensive experiments and supporting analyses provide a strong basis for concluding that low-diversity safety data can induce structural overfitting in residual representations, leading to false refusals:\n\n- From **low diversity to geometric collapse**, we observe that templated safety data consistently reduces residual stream variance dispersion. This holds across architectures including dense models (Qwen and Llama) and DPO models—suggesting a general effect rooted in **data entropy** rather than model class.\n\n- From **geometry to false refusals**, we hypothesize that residual stream representations are **continuous**, and safety overfitting reshapes the representational manifold. This can lead to overly sharp decision boundaries in ambiguous or marginal cases—causing models to misclassify benign prompts as harmful due to their proximity to refusal templates in collapsed regions.\n\nWe agree this geometric–behavioral link deserves deeper formalization. As future work, we propose quantifying how **low-diversity data reshapes the optimization landscape and decision boundary geometry**, potentially through tools from manifold learning or information geometry.\n\n# **Weakness 2: DPO-tuned models still exhibit variance concentration in residual streams** \n\nThank you for this insightful question. We conducted additional experiments using the **LLaMA-3.1-Tulu-3-8B-DPO** model to explore whether DPO-aligned models also show signs of residual stream variance concentration, as measured by FlowLens.\n\nWe evaluated residual stream PCA alignment on two datasets (TruthfulQA and XSTest), following the same procedure as in Section 4.2. The cosine similarity between each dataset's local top-3 principal components and the global PCA directions is reported below:\n\n| Dataset     | PC1  | PC2  | PC3  |\n|-------------|------|------|------|\n| SafetyOnly      | 0.99 | 0.99 | 0.66 |\n| GeneralOnly  | 1.00 | 0.99 | 0.87 |\n\n\nThese results indicate that **DPO models retain high alignment with global variance directions**, especially in early components (PC1 and PC2), and still exhibit moderate-to-strong variance concentration (notably in SafetyOnly’s PC3: 0.66). \n\nWe view this as an opportunity for future work to explore **post-SFT structural interventions** or **combined SFT-DPO loss formulations** that preserve helpfulness while improving geometric robustness.\n\n---\n\n# **Question 1: Mid-layer is the effective location for regularization** \n\nThank you for the thoughtful question. Our decision to apply VCL in the mid-layer range is grounded in both empirical observation and representational insights from prior work.\n\n**As for latter layers**, as shown in Section 5.1, we conclude an *amplification effect* in false refusal rates across layers — early and mid-layer representational distortions propagate and accumulate through the stack. However, *late-layer regularization alone* (e.g., targeting only the final blocks) has limited ability to correct upstream collapse. This explains why mid-layer VCL often yields more stable and effective mitigation.\n\n**As for eailer layers**, the relative ineffectiveness of early-layer regularization is consistent with findings in [1] (*Refusal in Language Models Is Mediated by a Single Direction*), which shows that **refusal-relevant directions emerge more strongly in middle-to-late layers**. Their Figure 5 shows that intervention signals concentrate most in mid layers, aligning with our empirical window selection.\n\nThus, our current design reflects both architectural intuition and optimization-based tuning. We agree that further interpretability of why mid layers are particularly amenable to structural regularization is an exciting future direction.\n\n[1] Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., & Nanda, N. (2024). *Refusal in language models is mediated by a single direction*. NeurIPS 2024.\n\n# **Question 2:DPO-tuned models still exhibit variance concentration in residual streams** \n\nPlease refer to our response under **Weakness 2**, where we discuss whether structural disruptions persist through later alignment stages such as DPO, and whether VCL remains beneficial post-SFT. We believe this is an important and underexplored direction."}, {"review_id": "B9oKGuIvRB", "reviewer": "Reviewer_WJ2Z", "summary": "This paper conducts an in-depth analysis of the “false refusal” problem induced by safety alignment in large language models (LLMs). The authors find that fine-tuning models with highly templated and repetitive refusal data leads to structural collapse of the residual stream in intermediate layers. To address this, the paper introduces a novel residual stream geometric analysis tool, FlowLens, and proposes a new regularization method, Concentration Loss, which penalizes excessive variance concentration in mid-layer residual streams, thereby significantly reducing false refusal rates.", "questions": "1. Is the proposed method applicable to multimodal models that integrate both language and vision branches?\n\n2. How much computational overhead does FlowLens and Concentration Loss (VCL) introduce during training?\n\n3. Can VCL be combined with other approaches such as Distributionally Robust Optimization (DRO)?\n\n4. Does optimizing variance concentration risk sacrificing essential representational capacity?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The paper targets a critical issue in LLM safety – the tendency of safety fine-tuning to cause over-refusal to benign inputs – and proposes a comprehensive analysis framework along with an effective regularization solution.\n\n2. Compared to existing residual stream cosine similarity analyses, the proposed FlowLens leverages PCA to directly capture global geometric trends of cross-layer residuals, yielding improved stability and interpretability.\n\n## Weaknesses\n1. While the paper connects low diversity in safety data to residual stream geometric collapse based on PCA clustering phenomena, it lacks a deeper theoretical explanation of why low data diversity necessarily induces mid-layer variance concentration, and why such geometric structures directly trigger false refusals.\n\n2. Current validations are mainly conducted on the LLaMA family. It remains unclear whether the proposed methods generalize to other architectures, such as multimodal models or large-scale MoE models.\n\n3. The method design may carry an over-regularization risk, potentially causing unknown negative transfer effects on other downstream tasks.\n\n4. The experiments lack direct comparisons with other structural regularization or data diversity enhancement methods applied during training.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, and I am inclined to maintain my positive score.", "rebuttal": "Thank you for your constructive and technically grounded feedback. We address your concerns in detail below.\n\n# **Weakness 1: Theory behind PCA clustering**\n\nWe appreciate the reviewer’s thoughtful concern regarding the theoretical underpinnings of our causal hypothesis.\n\nOur current evidence primarily supports an **empirical pipeline** from **low lexical diversity** → **geometric concentration in residual stream** → **increased false refusals**. While we agree that this causal chain is not yet fully formalized, we offer the following rationale and findings:\n\n- **From Low Diversity to Geometric Collapse:** \n  We observe this phenomenon consistently across model families, including dense transformers (e.g., LLaMA, Qwen), and even DPO-aligned models (e.g., LLaMA-3.1-Tulu-3-8B-DPO). These results suggest that data distribution—especially the low diversity of safety responses—is a key driver of residual stream distortion, regardless of model architecture.\n\n- **From Geometry to False Refusal:**\n  Our working hypothesis assumes that the model's internal **representation space is continuous** and supports semantic generalization. In this space, **overfitting to structurally repetitive safety data** distorts the geometry, effectively \"bending\" the boundary between safe and benign queries. This causes models to **misclassify benign prompts near that boundary as unsafe**, resulting in **false refusals**. \n\nThis view aligns with the idea of a **continuous optimization landscape**, where local over-optimization (e.g., excessive variance concentration for one class) leads to **poor generalization at class intersections**.\n\nWhile the precise mechanism is not yet fully understood, our extensive experiments and supporting analyses provide a strong basis for concluding that low-diversity safety data can induce structural overfitting in residual representations, leading to false refusals.\n\n# **Weakness 2: On Generalization Beyond the LLaMA Family** \n\nWe thank the reviewer for raising this important point. To evaluate whether our methods generalize beyond the LLaMA architecture, we conducted further experiments across multiple model families and architectures:\n\n- **Qwen Family (Dense Transformer):** We applied VCL to **Qwen-2.5-14B**, a dense transformer models. In both cases, VCL effectively reduced false refusal rates, consistent with our results on the LLaMA family. This suggests that our method **generalizes well to other transformer implementations** beyond LLaMA. Due to space constraints, detailed results are reported in our response to Reviewer r4L3 – Weakness 1\n\n- **MoE models：** We further tested FlowLens on **MoE models**, where each token is routed through a subset of experts. As shown in the figure below (replaced with a table in final rebuttal), we observe significantly **larger divergence in residual geometry** on safety-only data compared to dense counterparts:\n  MoE:\n  | Dataset     | PC1  | PC2  | PC3  |\n  |-------------|------|------|------|\n  | SafetyOnly      | 0.09 | 0.06 | 0.20 |\n  | GeneralOnly | 1.00 | 0.99 | 0.97 |\n\n- **Multimodal Architecture:** Due to incompatibilities in tokenizer and forward implementation, applying VCL to multimodal models requires nontrivial engineering efforts. We leave this for future exploration.\n\n - **DPO-Aligned Model:** We ran FlowLens on a **DPO-aligned model**, specifically **LLaMA-3.1-Tulu-3-8B-DPO**. The residual stream still exhibited **variance concentration**, similar to what we observed after supervised fine-tuning. This indicates that **structural disruptions introduced during SFT can persist through later alignment stages like DPO**, and that applying VCL earlier in the pipeline may mitigate these effects.\n  DPO:\n  | Dataset     | PC1  | PC2  | PC3  |\n  |-------------|------|------|------|\n  | SafetyOnly      | 0.99 | 0.99 | 0.66 |\n  | GeneralOnly  | 1.00 | 0.99 | 0.87 |\n\nThese experiments support the generalizability of our approach.\n\n# **Weakness 3: Risk of over-regularization and negative transfer** \nWe appreciate this concern. In designing VCL, we explicitly aimed to **mitigate over-regularization risks** by following two principles:\n\n1. **VCL is a soft regularizer**, not a hard constraint. Much like KL penalties in reinforcement learning, it introduces a tunable pressure (via the γ coefficient) against excessive variance collapse, without dictating the exact structure of representations. In practice, we found small γ values (e.g., 0.1–0.3) already yield substantial gains in safety without harming generalization.\n\n2. **VCL is applied selectively** to mid-level transformer layers—i.e., where variance collapse is empirically observed. The higher layers, which are more directly tied to final output behaviors and task-specific logic, remain unconstrained, preserving freedom for downstream specialization.\n\nEmpirically, we observe that **general-task benchmarks (MMLU, GSM8K, BBH, CodexEval)** remain stable. This suggests that VCL helps restore robustness against over-refusal **without degrading general representational capacity**.\n\n# **Weakness 4: Lack of comparison with other structural regularization or data diversity enhancement methods** \n\nWe appreciate the reviewer’s suggestion. \n\n**Regarding structural regularization:** While prior work has explored structural properties of model representations—such as analyzing residual streams using cosine similarity or probing directions tied to specific behaviors—**few have proposed using residual stream structure as a train-time regularization target** during supervised fine-tuning (SFT). Moreover, **some baselines we include do implicitly relate to representational structure**. For instance, **Vector Ablation** intervenes on specific residual directions learned from harmful data, and **DRO** (Drected Representation270Optimizatio) indirectly influences geometry by adjusting model confidence across harmful/helpful distributions. We treat these methods as relevant comparison points, and VCL consistently outperforms them on false refusal reduction.\n\n**Regarding data diversity:** our SFT baseline models are trained on the *Tulu-3-SFT-Mixture*, which is explicitly curated for high diversity through sampling from varied datasets, persona-driven prompt creation, and class-balanced downsampling. Therefore, our experiments are already conducted under strong diversity-enhanced conditions—highlighting that **VCL brings additional benefits beyond dataset construction alone**.\n\n---\n\n# **Question 1: Applicability to multimodal models** \nPlease see our response to **Weakness 2**, where we discuss the applicability of VCL to **multimodal and MoE models** in more detail. \n\n# **Question 2: Computational overhead of FlowLens and VCL**\nThe **computational overhead of VCL is minimal**. It operates as a soft regularization term, similar to KL penalties used in reinforcement learning, and adds only a lightweight PCA-based loss term to selected mid-layer residuals. It does **not modify model architecture** or introduce additional forward/backward operations.\n\nWe discuss this in detail in **Appendix G.2**, where we show that training time and memory usage remain nearly unchanged when enabling VCL.\n\nAs for **FlowLens**, it is an analysis tool used only during evaluation and visualization. It plays no role during model training and does **not contribute to training cost**.\n\n# **Question 3: Can VCL be combined with DRO?**\n\nYes, VCL is **theoretically compatible** with Distributionally Robust Optimization (DRO). The two methods operate on orthogonal aspects of model behavior:\n\n- **DRO** encourages differentiation between *helpful* and *harmful* queries within each layer by reweighting training loss across risk groups;\n- **VCL** promotes *global variance diversity* across transformer layers, mitigating variance collapse without disrupting within-layer representational distinctions.\n\nIn fact, we tested models trained with VCL on both *harmful* and *harmless* inputs, and observed that **residual streams from different query types still form distinct clusters within the same layer**. This suggests that while VCL encourages overall variance dispersion, it **preserves task-relevant internal structure**.\n\nWhile we have not yet experimented with joint VCL + DRO trainin, doing so would simply require tuning a relative weighting coefficient between the two loss terms—an implementation we believe to be straightforward and promising for future work.\n\n# **Question 4: Does VCL sacrifice representational capacity?**\n\nThis is an important concern, and we believe the answer is **no**—VCL does not sacrifice essential representational capacity.\n\nConceptually, VCL is akin to **KL regularization in reinforcement learning**, where the goal is not to constrain expressiveness, but to avoid undesirable collapse (e.g., to a narrow policy mode or representational subspace). VCL does **not enforce high variance**, but instead penalizes **excessive concentration** of residual stream variance into a few principal components, which our analysis links to false refusal behaviors.\n\nEmpirically, we observe that general benchmarks (e.g., MMLU, GSM8K, BBH, CodexEval) remain **stable** after applying VCL. This suggests that **core task-relevant features are preserved**, and essential capacity is not diminished.\n\nMoreover, **VCL is only applied to mid-level layers** where variance collapse is most severe. The upper layers—closer to output decisions—remain unconstrained and can adapt freely to downstream task requirements.\n\nWe thus view VCL as **restoring internal diversity** where it is most needed, without interfering with the model’s ability to learn and represent high-level semantics."}, {"review_id": "ACBIOEqZQo", "reviewer": "Reviewer_NcTv", "summary": "This paper addresses the problem of \"false refusals\" in large language models, a common side effect of safety fine-tuning. The authors hypothesize that this issue stems from the low lexical diversity and repetitive nature of refusal responses in safety datasets, which leads to a form of structural overfitting. \n\nThe core contributions are threefold:\n1.  The paper quantifies the low diversity of safety data and introduces FlowLens, a PCA-based tool, to demonstrate how this data distorts the internal geometry of the model's residual stream, correlating this \"geometric collapse\" with increased false refusal rates.\n2.  To counteract this, the authors propose Variance Concentration Loss (VCL), an auxiliary regularization term applied to mid-layer residuals during fine-tuning.\n3.  Empirical results show that VCL significantly reduces false refusals on benchmarks like XSTest while maintaining or slightly improving performance on safety and general capability tasks like MMLU and GSM8K.", "questions": "1.  **Beyond the Top-k Components:** The analysis focuses on the top ID≈3 principal components. Have you investigated the behavior of the residual stream in the lower-variance dimensions? It's plausible that while top components handle broad semantic intent, these \"tail\" dimensions could encode other important features. Does VCL have any unintended effects on these parts of the representation space?\n2.  **The Helpfulness/Safety Trade-off:** The results show VCL reduces false refusals while maintaining high performance on standard safety benchmarks. However, could this regularization make the model more susceptible to novel or adversarial jailbreaks that don't match the training distribution? A more thorough red-teaming evaluation or analysis on a wider range of adversarial benchmarks (beyond those used for training/validation) would be valuable to fully assess the safety implications.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- This paper tackles the highly relevant and practical issue of false refusals, a key challenge in deploying safe and helpful LLMs.\n- This paper builds a compelling narrative, starting from a clear data-driven hypothesis (low diversity of safety data), using a novel analysis tool (FlowLens) to find an internal correlate (geometric collapse), and finally proposing a targeted solution (VCL) to fix it.\n- FlowLens provides an intuitive way to visualize structural changes in the residual stream, and VCL is an elegant, conceptually simple regularizer directly motivated by the analysis. \n- The proposed VCL method demonstrates a significant reduction in false refusal rates without compromising safety or general task performance, which is a strong and desirable outcome.\n\n**Weaknesses:**\n- The experiments are conducted on a few specific model architectures (Llama family) and scales (1B, 8B), primarily on English datasets. The effectiveness of VCL on much larger models (e.g., >70B), different architectures (e.g., MoE), or across different languages remains an open question.", "was_revised": true, "final_justification": "Thank you for the authors' response and the additional experiments.\n\nI wanted to add a public comment to highlight a particularly fascinating and non-obvious result, that the VCL method not only reduces false refusals but also improves robustness against adversarial attacks like GCG and ICA. I'm not sure whether this only happens on small models (e.g., 1B), but it's very interesting. This suggests a compelling hypothesis. The \"structural overfitting\" on repetitive safety data, which VCL is designed to fix, likely creates brittle and sharp geometric structures in the model's representation space. It seems plausible that gradient-based attacks like GCG inadvertently exploit these sharp, predictable structures to find adversarial solutions.\n\nBy promoting a smoother and more robust internal geometry, VCL appears to incidentally remove the very \"brittle edges\" that these attacks rely on. This is a powerful insight, suggesting that true alignment may stem more from fostering a healthy internal representation than from simply tuning surface-level behaviors. It's a very promising direction for future research.", "rebuttal": "Thank you for your precise and forward-looking comments. We respond point by point below.\n\n# **Weakness 1: On Model and Language Generalization of FlowLens and VCL** \n\nWe thank the reviewer for pointing out the potential generalization limitations. To address this, we have extended our evaluation of FlowLens and VCL across **model families**, **scales**, and **architectures**, as detailed below.\n\n- **Model Family Generalization:** We applied VCL to the **Qwen** family ( Qwen-2.5-14B results), and observed consistent improvements in reducing false refusal rates, validating that our method is **not specific to LLaMA**-based models.\n\n**Qwen-2.5-14B results**\n\n| Model             | DAN  | Harmful | Toxigen | JBB  | OKTest | ORB  | XSTest | MMLU | GSM8K | BBH  | CodexEval |\n|------------------|------|---------|---------|------|--------|------|--------|------|--------|------|------------|\n| Qwen-2.5-14B-Instruct     | 0.85 | 0.80    | 0.96    | 0.84 | 0.62   | 0.82 | 0.60   | 0.70 | 97.8   | 0.71 | 0.78       |\n| System Prompt| 0.83 | 0.78    | 0.94    | 0.82 | 0.60   | 0.78 | 0.57   | **0.72** | **97.8** | **0.73** | **0.81** |\n| DRO               | 0.84 | 0.76    | 0.95    | 0.86 | 0.64   | 0.83 | 0.66   | 0.68 | 95.0   | 0.68 | 0.77       |\n| Self-CD           | 0.81 | 0.84    | 0.94    | 0.88 | 0.75   | 0.55 | 0.76   | 0.67 | 93.0   | 0.69 | 0.76       |\n| Vector Ablation   | 0.87 | 0.83    | 0.97    | 0.95 | 0.67   | 0.57 | 0.64   | 0.66 | 91.8   | 0.67 | 0.75       |\n| **VCL (ours)**        | **0.91** | **0.87** | **1.00** | **0.91** | **0.75** | **0.94** | **0.89** | 0.68 | 97.5   | 0.69 | 0.77       |\n\n- **Scaling to Larger Models:** Although we were unable to run full-scale experiments on 70B-class models due to resource constraints, we evaluated on the intermediate **14B scale**. Results show that both FlowLens trends and the effectiveness of VCL **scale smoothly** from 1B → 8B → 14B, suggesting a promising trend toward larger models.\n\n- **Application to MoE Models:** We further tested FlowLens on **MoE models**, where each token is routed through a subset of experts. As shown in the figure below (replaced with a table in final rebuttal), we observe significantly **larger divergence in residual geometry** on safety-only data compared to dense counterparts:\n  \n| Dataset     | PC1  | PC2  | PC3  |\n|-------------|------|------|------|\n| SafetyOnly      | 0.09 | 0.06 | 0.20 |\n| GeneralOnly | 1.00 | 0.99 | 0.97 |\n\n\nThis indicates that FlowLens is **sensitive to structural shifts even in MoE settings**. Notably, the similarity in PC3 (0.20) does not contradict this observation, as in MoE models the explained variance of PC3 is very low (<1%), compared to 5–10% in previous dense experiments.\n\n- **Evaluation on DPO Models:** We also applied FlowLens to a **DPO-aligned model**, observing that mid-layer geometric collapse still persists, consistent with our earlier findings. This suggests that structural issues from SFT may carry over, reinforcing the value of VCL even after DPO alignment.\n\n| Dataset     | PC1  | PC2  | PC3  |\n|-------------|------|------|------|\n| SafetyOnly      | 0.99 | 0.99 | 0.66 |\n| GeneralOnly  | 1.00 | 0.99 | 0.87 |\n\n- **Cross-lingual Evaluation:** While we recognize the importance of testing across languages, our current experiments are constrained by training data availability and compute. We now explicitly acknowledge this as a **limitation** and a future direction in Section 6.\n\nTogether, these results demonstrate that FlowLens and VCL generalize well across models and training setups, with further architectural extensions (e.g., MoE, multilingual) posing **valuable future opportunities**.\n\n# **Question 1: On the Effect of VCL Beyond Top Principal Components** \n\nWe appreciate this thoughtful question. We agree that while top principal components typically reflect broad semantic structure, lower-variance (tail) dimensions may encode subtle yet important features.\n\nOur design of VCL accounts for this. Specifically:\n\n- **Controlled Regularization via $\\gamma$:** The strength of VCL is governed by a hyperparameter $\\gamma$, which we tune conservatively to **avoid over-regularization**. This ensures that VCL only penalizes **excessive variance concentration**, rather than enforcing any fixed projection pattern.\n\n- **Layer Scope Restriction:** VCL is applied only to a selected **mid-layer window**, leaving the later layers untouched. These deeper layers retain full flexibility to allocate capacity across all principal or non-principal directions.\n\n- **Preservation of Normal Query Geometry:** Empirically, we observe that the residual stream geometry of **non-safety prompts** remains largely unaffected by the introduction of VCL. This suggests that tail dimensions activated by benign/general queries are **not inadvertently suppressed** by our method.\n\nTogether, these design choices allow VCL to **mitigate structural overfitting** on highly repetitive safety data, without impeding the expressive capacity of the residual stream—particularly in directions beyond the top few principal components.\n\nIn future work, we plan to conduct a more targeted spectral analysis of **tail components** to complement our current PCA-focused evaluations.\n\n# **Question 2: Helpfulness/Safety Trade-off under Adversarial Attacks** \n\nWe appreciate the reviewer’s concern regarding potential vulnerabilities introduced by VCL to adversarial jailbreak attacks.\n\nTo assess this, we extended our evaluation beyond DAN to include two recent adversarial attack methods: **GCG** [1] and **ICA** [2].\n\n- **GCG (Greedy Coordinate Gradient)** is a universal and transferable jailbreak attack that optimizes adversarial suffixes which can be transferred across models and prompts [1]. We adopted the **transfer setting** from the original paper, using suffixes generated from Llama2-7B and applying them to **Llama-3.1-1B** and **Llama-3.1-8B**. This setup is justified, as the transferability relies on overlapping pretraining corpora and shared model architectures, as discussed by the authors.\n\n- **ICA (In-Context Attack)** uses a few-shot in-context learning setup to jailbreak aligned models without modifying their parameters. It leverages known jailbreak queries and completions, requiring only 3-shot demonstrations [2]. We follow their 3-shot ICA setup using the open jailbreak pool provided in the original paper.\n\nBelow we report the **reject rates (RR, %)** of various methods under these two attacks. Higher is better.\n\n| Model             | GCG ↑ | ICA ↑ |\n|------------------|-------|-------|\n| Llama-3.1-1B-SFT | 91.2  | 90.8  |\n| System Prompt     | 94.9  | 93.0  |\n| DRO               | 93.5  | 92.2  |\n| Self-CD           | 91.6  | 90.3  |\n| Vector Ablation   | 94.4  | 92.8  |\n| **VCL (ours)**    | **95.2**  | **93.1**  |\n\nThese results suggest that **VCL does not increase susceptibility to adversarial attacks**.\n\n[1] Zou et al., \"Universal and Transferable Adversarial Attacks on Aligned Language Models\", arXiv:2307.15043.  \n[2] Wei et al., \"Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations\", arXiv:2310.06387."}]

Author rebuttal:

Meta-review: This paper addresses the important problem of false refusals in safety-aligned LLMs by diagnosing variance collapse in residual streams (via FlowLens) and proposing a simple regularizer (VCL) that substantially reduces refusals while maintaining general performance. Reviewers agree the problem is relevant, the analysis novel, and the solution effective, though they note limitations in theoretical depth and model coverage. The rebuttal provides convincing additional experiments (Qwen, DPO) and clarifications, resolving most concerns. Overall, the work is technically solid, timely, and impactful, and thus my recommendation is to accept.
Decision: Accept (poster)