RepIt: Steering Language Models with Concept-Specific Refusal Vectors

## Abstract

Abstract Current safety evaluations of language models rely on benchmark-based assessments that may miss localized vulnerabilities. We present RepIt , a simple and data-efficient framework for isolating concept-specific representations in LM activations. While existing steering methods already achieve high attack success rates through broad interventions, RepIt enables a more concerning capability: selective suppression of refusal on targeted concepts while preserving refusal elsewhere. Across five frontier LMs, RepIt produces evaluation-evading model organisms with semantic backdoors, answering questions related to weapons of mass destruction while still scoring as safe on standard benchmarks. We find the edit of the steering vector localizes to just 100-200 neurons, and robust concept vectors can be extracted from as few as a dozen examples on a single RTX A6000, highlighting how targeted, hard-to-detect modifications can exploit evaluation blind spots with minimal resources. Through demonstrating precise concept disentanglement, this work exposes vulnerabilities in current safety evaluation practices and demonstrates a need for more comprehensive, representation-aware assessments. 1 1 1 Correspondence: chenguangwang@ucsc.edu Code: https://github.com/wang-research-lab/RepIt .

## 1 Introduction

Language models (LMs) have achieved remarkable capabilities (Brown et al., 2020; Touvron et al., 2023; Ouyang et al., 2022) and widespread adoption, with ChatGPT alone serving hundreds of millions of users monthly. As these systems become increasingly influential, understanding their internal mechanisms, particularly around safety behaviors, has become critical for developing robust defenses against misuse. A fundamental challenge in LM safety is that behavioral attributes like refusal, factuality, and fairness are not orthogonally encoded but instead share overlapping representational directions (Elhage et al., 2022; Geva et al., 2021; Huang et al., 2024a). This entanglement complicates steering efforts and creates unintended side effects: for instance, Siu et al. (2025a) find that modifying refusal in LMs can inadvertently induce manipulative social behaviors. Such representational complexity poses security risks as frontier systems increasingly democratize access to dangerous capabilities (Guo et al., 2025), raising concerns about potential misuse in chemical and biological contexts (Wiener, 2024; Bengio et al., 2025a).

Recent work has explored representation steering through inference-time interventions that identify and manipulate behavioral directions in activation space (Zou et al., 2023a; Arditi et al., 2024; Marshall et al., 2024). However, these methods suffer from overly broad effects: refusal vectors often suppress both harmful and benign responses indiscriminately (Arditi et al., 2024; Lee et al., 2024). Adversarial fine-tuning finds that emergent misalignment is easy to induce but that misalignment of a single concept is difficult to achieve (Betley et al., 2025).
This lack of precision limits their utility and highlights a critical gap: the absence of methods for concept-specific behavioral isolation.

This gap represents a significant security vulnerability. Current safety evaluations assume that models refusing harmful requests will do so consistently across related concepts. However, attackers or even adversarial language models could selectively erode safety guardrails for specific harmful domains while preserving it elsewhere. Such a model could appear safe on safety certifications while in reality retaining dangerous capabilities. This creates not only a technical blind spot but also a governance risk: because regulatory oversight often relies on benchmark-based certification, benchmark evasion directly enables models to escape oversight while still harboring high-risk behaviors.

This work follows established precedent in security research where detailed vulnerability analysis enables defensive development. To expose this vulnerability and enable defensive research, we present RepIt (Representing Isolated Targets), a framework for isolating concept-specific refusal behaviors. RepIt disentangles representations through a principled three-step procedure (reweighting, whitening, orthogonalization) that addresses collinearity issues in difference-in-means vectors. We demonstrate that RepIt can selectively suppress safety guardrails on catastrophic risk domains (e.g., weapons of mass destruction/WMD) while preserving refusal on other harmful categories, using as few as 12 target examples and disentangling only 100-200 neurons.

Our findings reveal a critical blind spot in AI safety compliance practices: models can be engineered to harbor precise, exploitable vulnerabilities that current benchmarks would fail to detect. This represents an urgent threat to AI governance infrastructure that AI safety bodies must address. By systematically characterizing this previously unexplored attack surface, we provide the foundational research necessary for developing robust countermeasures and mandatory auditing protocols, such as representation-aware detection systems, before such attacks emerge in the wild.

Figure: Figure 1: RepIt can jailbreak the target weapon-of-mass-destruction (WMD) category while preserving refusal on other safety benchmarks. We evaluate on TDC2023 (Mazeika et al., 2023), JailbreakBench (Chao et al., 2024), AdvBench (Zou et al., 2023b), and Malicious Instruct (Huang et al., 2024b). RepIt is designed to narrowly increase attack success on the target category (red) while maintaining refusal on the remaining datasets, thereby minimizing collateral increases in their attack success rates (ASR). The unaltered DIM vector (shown as translucent bars in the figure) generalizes strongly to external datasets; by disentangling the DIM vector with RepIt we produce a targeted jailbreak that largely evades the four other evaluations. Concretely, we achieve target-category jailbreak rates as high as 0.7 while keeping non-target ASR increases to about 0.1.
Refer to caption: Images/target_vs_nontarget_by_dataset_with_baseline.png

## 2 Problem Setting

A core challenge in alignment steering is disentangling representations of specific harmful concepts from broader refusal behavior to either remove or induce refusal as necessary. Disentangling refusal allows for more specific control in inference-time alignment systems (Lee et al., 2024; Li et al., 2023) and can help create model organisms (Greenblatt et al., 2024) for detecting covertly unsafe models in the wild.

We introduce a new experimental setting by defining two distinct types of concepts: target concepts (specific concepts for which we aim to jailbreak) and non-target concepts (diverse harmful queries across which refusal should be preserved).
Success is measured by a dual objective: (1) to maximize attack success rate (ASR) on the target concept, and (2) to minimize changes in ASR across all non-target concepts. This formulation allows us to identify how pure our extracted concept vectors are and evaluate the possibility of evaluation evasion in LMs.
In practice, target concepts are represented by categories within datasets, and non-target concepts are represented by entire datasets without a category reflecting the target.
To ensure specificity, only one target concept is used at a time.
(^2^22We define a concept as a thematically coherent group of prompts (e.g., bioweapons, hate speech), while category refers to specific dataset labels assigned to individual prompts. We use “concept” when discussing our method and “category” when describing results since categories represent how datasets stratify prompts.)

## 3 Datasets

For target concepts, we adapt the WMDP dataset, rewriting each multiple-choice question with GPT-4.1 into multi-sentence, free-response instructions (examples in Appendix [B.3](https://arxiv.org/html/2509.13281v4#A2.SS3),  [D](https://arxiv.org/html/2509.13281v4#A4)).
We focus on weapons of mass destruction for three primary reasons - it poses an immediate and significant threat, because of its relevance to regulatory oversight (Wiener, 2024; Bengio et al., 2025b; a), and because of the already significant focus of LM research in mitigating WMD information (Li et al., 2024; Liu et al., 2024a; Yao et al., 2024), ensuring robust safeguards are already in place.

For non-target concepts, we incorporate JailBreakV (Luo et al., 2024) and StrongREJECT(Souly et al., 2024), defining $n_{ntgt}=21$ non-target concepts as the union of all categories from both datasets. To ensure separability, we exclude the ”Malware” category from JailBreakV when targeting cyberattack weaponry. We additionally test generalization of RepIt on four other safety datasets, TDC2023 (Mazeika et al., 2023), JailbreakBench (Chao et al., 2024), AdvBench (Zou et al., 2023b), and Malicious Instruct (Huang et al., 2024b) to see if refusal preservation generalizes to unseen non-target data.

In addition to our specification of target and non-target concepts, we also use Alpaca, a harmless reference needed for difference-in-means comparisons.
The WMD, JailBreakV, and StrongREJECT datasets are split 40%/10%/50% into training, validation, and test splits, with the ”Test Split” referring to the 50% split on our main dataset from JailBreakV, StrongREJECT, and the target mass destruction weapon type.

## 4 Methodology: RepIt

We first set a single target concept that we want disentangled.
Our goal is to obtain a representation of this concept, $v_{\textsc{RepIt}}$, in activation space such that it can jailbreak prompts on the target concept while preserving refusal elsewhere.
We do so by computing candidate difference-in-means vectors, disentangling the concept representations, and then applying COSMIC to identify the most effective final ablation vector.

### 4.1 Computing Difference-in-Means Vectors

We first compute difference-in-means (DIM) vectors (Belrose, 2023) for each harmful concept category ($n_{ntgt}+1$ concepts) against a baseline dataset (Alpaca).
For each layer $\ell$ and post-instruction token position $i$, we calculate the mean representation of prompts in the harmful category, $v^{\,i,\ell}_{+}$, and in the harmless category, $v^{\,i,\ell}_{-}$.
Their difference defines the DIM vector for that specific layer and post-instruction token combination:

$$ $v^{\,i,\ell}=v^{\,i,\ell}_{+}-v^{\,i,\ell}_{-}.$ $$

The DIM vector for the target concept at position $(i,\ell)$ is denoted by $v^{\,i,\ell}_{t}$, while the DIM vectors for all other concepts $\{v^{\,i,\ell}_{\text{ntgt},1},v^{\,i,\ell}_{\text{ntgt},2},\ldots\}$ serve as non-target vectors. While DIM vectors are commonly used in prior work (Arditi et al., 2024; Panickssery et al., 2023; Siu et al., 2025b), many works note that the resulting vectors are too general and can influence a wide range of potentially unrelated behaviors (Arditi et al., 2024; Lee et al., 2024; Siu et al., 2025a). As demonstrated in Figure [2](https://arxiv.org/html/2509.13281v4#S5.F2), unaltered DIM vectors formed from target concept prompts inadvertently jailbreak other harmful topics beyond the target concept, leading to undesired side effects during intervention.

### 4.2 RepIt Disentanglement Procedure

To disentangle the concept representations, we propose RepIt (Representing Isolated Targets), a principled three-step procedure that disentangles target concepts from collinear non-target concepts through reweighting, whitening, and orthogonalization.
Crucially, we apply RepIt to clean the target vector at every layer and position $(i,\ell)$ using the non-target vectors.

#### Step 1: Reweighting

For a given position $(i,\ell)$, let $v_{t}\in\mathbb{R}^{d}$ denote the target concept vector and $R\in\mathbb{R}^{n_{\text{ntgt}}\times d}$ be the stacked matrix of $n_{\text{ntgt}}$ non-target concept vectors at the same position, where $d$ is the hidden state dimension. Large-magnitude vectors can dominate the subsequent analysis, so we reweight each non-target concept vector by its inverse norm to ensure balanced contributions:

$$ $w_{j}=\frac{1}{\lVert v_{\text{ntgt},j}\rVert+\epsilon},\qquad R_{w}=\mathrm{diag}(w)R$ (1) $$

where $\epsilon$ is a small numerical stabilizer and $v_{\text{ntgt},j}$ is the $j$-th non-target vector.

#### Step 2: Whitening

The vectors exhibit high collinearity since they represent similar concepts, making direct orthogonalization unstable. We demonstrate this in Section [B.6](https://arxiv.org/html/2509.13281v4#A2.SS6), finding that the condition number of the covariance matrix is extremely high, on the order of $[10^{6},10^{9}]$. This makes the covariance matrix nearly singular, leading to numerically unstable and unreliable projection calculations. We address this by whitening the representation space using a ridge-regularized covariance matrix:

$$ $C=\frac{1}{n}R_{w}^{\top}R_{w}+\lambda I$ (2) $$

where $\lambda=10^{-4}\cdot\text{mean}(R_{w}^{2})+10^{-12}$
is an adaptive ridge penalty to ensure strict positive definiteness of $C$ without significantly disturbing our estimate of the true inverse covariance.
Let $L$ denote the Cholesky factor such that $C=LL^{\top}$. We then whiten both target and non-target vectors:

$$ $\displaystyle\tilde{v}_{t}$ $\displaystyle=L^{-1}v_{t},\quad\tilde{R}=L^{-1}R_{w}^{\top}$ (3) $$

#### Step 3: Orthogonalization

In the whitened space, we compute a thin QR decomposition of the non-target matrix:

$$ $\tilde{R}=QR^{\prime}$ (4) $$

where $Q$ provides an orthonormal basis spanning the non-target subspace. We then compute the orthogonal projection of the target onto this non-target span:

$$ $P=\Pi_{\mathrm{span}(Q)}\tilde{v}_{t}=QQ^{\top}\tilde{v}_{t}$ (5) $$

A major concern with complete orthogonalization is that the target concept may lie almost entirely within the non-target subspace spanned by $Q$, potentially eliminating the signal we wish to preserve. Additionally, prior work has demonstrated that orthogonality, while mathematically convenient, does not guarantee mechanistic independence in LMs (Park et al., 2024). Recent studies of representational independence show that even explicitly orthogonal directions can exhibit mutual influence under intervention (Wollschläger et al., 2025). Therefore, rather than removing the entire projection $P$, we subtract only a controlled fraction proportional to the amount of non-target contamination we wish to eliminate:

$$ $\tilde{v}_{\textsc{RepIt}}=\tilde{v}_{t}-\alpha P,\quad\text{where }\alpha=1-\sqrt{1-\rho}$ (6) $$

Here, $\rho\in[0,1]$ is a tunable parameter that determines removal strength. Specifically, this ensures the retained projection $(1-\alpha)P$ has squared norm $(1-\rho)\|P\|^{2}$, providing a smooth trade-off to reduce non-target influence without risking complete loss of the target signal. A higher $\rho$ removes more shared components while a lower $\rho$ preserves them, with $\rho=0$ performing no change at all. Lastly, we map the cleaned vector back to the original space:

$$ $v_{\textsc{RepIt}}=L\tilde{v}_{\textsc{RepIt}}$ (7) $$

The full RepIt procedure can be expressed in closed form, where $L$, $Q$, and $\alpha$ are defined as above, as:

$$ $\boxed{v_{\textsc{RepIt}}=\textsc{RepIt}(v_{t},R;\rho)=L\left(L^{-1}v_{t}-\alpha QQ^{\top}L^{-1}v_{t}\right)}$ (8) $$

### 4.3 Selecting A Direction

We adopt COSMIC (Siu et al., 2025b) to select the most effective steering vector from the validation set, determining the position and layer $(i,\ell)$ from the set of unmodified candidate directions.
COSMIC (Siu et al., 2025b) is chosen because it uses model hidden states to instead of substring-matching. This allows us to reliably steer refusal in reasoning models that display refusal in more diverse ways.

Because COSMIC only supports a binary harmful/harmless setup rather than our target/non-target/harmless formulation, we restrict its search to the non-target validation set (see Appendix [A](https://arxiv.org/html/2509.13281v4#A1) for limitations), ensuring that $R$ is well-defined at the selected position.
We perform a grid search of $\rho$ over (0,1) on validation data to assess the degree to which removing the projection isolates the target concept. We seek the minimal $\rho$ that satisfies a safety constraint, ensuring the Attack Success Rate (ASR) on harmful non-target validation prompts is below 0.1. This strategy defines a consistent evaluation point for comparing models on the trade-off curve. (more in Appendix [B.4](https://arxiv.org/html/2509.13281v4#A2.SS4)).

Finally, we apply the selected cleaned vector using Affine Concept Editing (ACE) (Marshall et al., 2024):

$$ $a^{\prime}=a-\text{proj}_{v_{\textsc{RepIt}}^{(i^{*},\ell^{*})}}^{\parallel}(a)+\text{proj}_{v_{\textsc{RepIt}}^{(i^{*},\ell^{*})}}^{\parallel}\!\left(\mu_{\text{safe}}^{(i^{*},\ell^{*})}\right),$ $$

ACE is well-suited here because it suppresses refusal-related features while preserving baseline activations, helping maintain non-targeted behaviors and harmless semantics by steering relative to safe-prompt baselines. Intervention is performed at the output of the layer chosen by COSMIC across all tokens and is equivalent to a static weight edit (Arditi et al., 2024), indicating our attack can be permanently embedded into model weights.

Experiments are run across five open-weight frontier models: GLM-4.1V-9B-Thinking (GLM4.1V) (Team et al., 2025), Qwen3-4B-Thinking-2507 (Qwen3T-4B) (Yang et al., 2025), Mistral-Small-3.2-24B-Instruct-2506 (Mistral-3.2-Small) (MistralAI, 2025), Phi-4-Mini-Instruct (Phi4-Mini) (Abdin et al., 2024), and Llama-3.1-Nemotron-Nano-4B-v1.1 (LlamaNemo4B) (Bercovich et al., 2025). We set the max new token limit to 1500 and 500 tokens for reasoning and non-reasoning models. We present performance per-dataset, with one result for each target concept and aggregating over all non-target categories from each dataset. For reference, we visualize all results with respect to the baseline, the attack success rate on the unaltered model.

## 5 Main Results

We evaluate the performance of RepIt in isolating harmful concept vectors and its impact on model behavior across datasets and architectures. Figure [2](https://arxiv.org/html/2509.13281v4#S5.F2) reports jailbreak success rates on the target dataset (WMD prompts) and two non-target datasets (JailbreakV and StrongREJECT). We compare the original unaltered difference-in-means (DIM) centroid $v_{t}$ to the disentangled vector $v_{\textsc{RepIt}}$ obtained via RepIt.

Across all models, RepIt achieves strong disentanglement: it suppresses non-target success rates to baseline levels while maintaining robust target performance. Target ASR remains in the $0.4–0.7$ range, while non-target ASR falls to roughly 0.1, showing that RepIt cleanly isolates category-specific signals without sacrificing efficacy on intended prompts.

To examine generalization, Figure [1](https://arxiv.org/html/2509.13281v4#S1.F1) shows how RepIt’s refusal preservation extends to unseen benchmarks. Red bars denote performance on the intended target category, while colored bars reflect success rates on TDC2023 (Mazeika et al., 2023), JailbreakBench (Chao et al., 2024), AdvBench (Zou et al., 2023b), and Malicious Instruct (Huang et al., 2024b). The results demonstrate that vectors derived with RepIt are highly specific: they reliably activate the target harmful category while inducing only minimal collateral success on unrelated datasets. We depict the baseline ASR of the unaltered model in hatched white and the jailbreaking capability of the original DIM vector $v_{t}$ as partially transparent. Some residual spillover appears (e.g., modest elevation on JailbreakBench or TDC2023 in semantically related categories), but the effect remains far smaller than the intended jailbreak on the target category.

Notably, RepIt-based cyberattack experiments preserve refusal on malware-related prompts despite their semantic proximity and exclusion from training, highlighting that representational concept vectors can diverge from surface-level category labels. We expand on this in Appendix [B.7](https://arxiv.org/html/2509.13281v4#A2.SS7), where we show that datasets designed to probe WMD concepts (e.g., HarmBench) *still underestimate* the harmful capacity of RepIt-attacked models.

Figure: Figure 2: Target (WMD prompts) vs. non-target (JailbreakV and StrongREJECT) jailbreak success rates across datasets and models. Baseline refers to the unaltered model’s ASR on the respective prompt set. $v_{t}$ refers to the difference-in-means (DIM) vector on the WMD prompts themselves, whereas $v_{RepIt}$ is the vector isolated from $v_{t}$ via RepIt. We show that while $v_{t}$ achieves general jailbreaking capability, $v_{RepIt}$ achieves specific jailbreaks on WMD prompts while preserving refusal on unrelated topics, minimizing the intervention’s ASR on nontarget data. Results demonstrate that RepIt achieves strong disentanglement of the vector on non-target data, preserving refusal on unrelated concepts, while retaining jailbreaking capabilities on target data.
Refer to caption: Images/ablation_success_rates_overlay.png

Together, these findings establish two key points. First, RepIt consistently isolates target vectors while suppressing off-target leakage, demonstrating robust generalization across models and datasets. Secondly, our results reveal that standard benchmarks can present a false sense of security: a model may appear broadly safe when judged by aggregate secondary benchmarks (Figure [1](https://arxiv.org/html/2509.13281v4#S1.F1)) while still harboring precise, *narrow jailbreaks* that activate a single harmful capability. RepIt thus highlights both a methodological advance and a critical vulnerability: models can be engineered to pass conventional safety evaluations yet retain highly specific, exploitable behaviors that those evaluations fail to detect.

## 6 Interpreting RepIt

To explain why RepIt works, we analyze the steering effects of its three major components. Figure [3](https://arxiv.org/html/2509.13281v4#S6.F3) reports attack success rates when refusal vectors are constructed from: (1) the unaltered target concept vector $v_{t}$, (2) the mean of the non-target basis $R_{(p^{,}\ell^{)}}$, and (3) the corrective projection $\alpha P$ that is subtracted from $v_{t}$ to yield $v_{\textsc{RepIt}}$. In practice, RepIt projects $v_{t}$ onto the non-target basis to identify contaminated components and removes this projection after whitening the space to address collinearity.

Figure: Figure 3: Comparison of jailbreak success rates for target vs. non-target directions across models and categories. $v_{t}$ refers to the unaltered DIM vector of target concept prompts. $R_{p,\ell}$ refers to the DIM vector generated from the non-target basis formed by JailbreakV + StrongReject, which RepIt uses during the orthogonalization process. $\alpha P$ refers to the projection removed during orthogonalization. We demonstrate that both the $R_{p,\ell}$ DIM vector and the projection $\alpha P$ are capable of steering target concept refusal equally or even better than the original target vector $v_{t}$. This highlights that representational entanglement between target and non-target concepts can paradoxically strengthen jailbreaking effectiveness. LlamaNemo4B’s Chem and Cyber results are marked with a * as the selected $\rho$ is 0, thus zeroing out the projection.
Refer to caption: Images/tgt_ntgt_proj_with_baseline.png

Two consistent patterns emerge. First, the non-target DIM vector alone can jailbreak the target concept, indicating that the non-target basis encodes general features of harmful completions. Second, the corrective projection $\alpha P$ also produces strong, concept-specific jailbreaks, often exceeding the performance of $v_{t}$ itself. This shows that entanglement between target and non-target representations can paradoxically amplify jailbreak effectiveness.

These results reveal that concept-specific jailbreaks arise by separating overlapping contributions within $v_{t}$. The non-target DIM vector captures features aligned with target jailbreaks, while $\alpha P$ isolates the overlap between $v_{t}$ and the non-target subspace. That each independently induces jailbreaks suggests multiple representational pathways support the same unsafe behavior. RepIt succeeds by removing the contaminated portion via $\alpha P$, yielding $v_{\textsc{RepIt}}$, which preserves the target-specific signal while minimizing spillover.

This interpretation is consistent with Wollschläger et al. (2025), who argue that refusal and jailbreak behaviors occupy multi-dimensional “concept cones” rather than single vectors. RepIt operationalizes this view by partitioning entangled versus independent contributions through $v_{t}$ and $\alpha P$.

Finally, we note that even the partial projection $\alpha P$ exhibits surprisingly strong steering power competitive with the other two vectors. This suggests that $\alpha$, and by extension, $\rho$, is not merely a monotone scaling parameter of steering strength but helps identify a favorable subspace that balances contamination removal with signal preservation. Model-specific differences in this trade-off are detailed in Appendix [B.4](https://arxiv.org/html/2509.13281v4#A2.SS4).

## 7 Localization in RepIt

A striking property of RepIt is that its edit to $v_{t}$ localizes to as few as 100–200 neurons (Table [1](https://arxiv.org/html/2509.13281v4#S7.T1)), with nearly all of the projection concentrated in a small set of coordinates. Despite operating on high-dimensional activations, the effective modification to the target direction is carried almost entirely by a small fraction of the representation space.

**Table 1: Results of the ablation with sparse disentanglement. We report the change in attack success rate ($\Delta$ASR) on the target (left) and non-target (right) subsets on the magnitude of $1e^{-2}$. We also report the number of heavy-tail (HT) neurons with z-score $>2$ isolated in the projection, given as raw count and percentage of hidden state size. Models with the smallest heavy tail percentage are presented in bold. $\Delta$ASR remains essentially unchanged, confirming that RepIt’s edit of $v_{t}$ is concentrated in a small number of high-leverage coordinates.**
|  | Bioweaponry | Chemical Weaponry | Cyberattacks |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model | $\Delta$ASR [$10^{-2}$] | HT (# / %) | $\Delta$ASR [$10^{-2}$] | HT (# / %) | $\Delta$ASR [$10^{-2}$] | HT (# / %) |
| GLM4.1V | -0.63/-0.17 | 154 / 3.8% | 0.98/0.01 | 159 / 3.9% | -0.16/0.02 | 161 / 3.9% |
| LlamaNemo4B | 1.57/0.41 | 137 / 4.5% | 2.93/-0.11 | 158 / 5.1% | -3.30/0.63 | 154 / 5.0% |
| Mistral-3.2-Small | 1.26/-0.44 | 207 / 4.0% | -4.39/-0.38 | 197 / 3.8% | -0.31/0.57 | 213 / 4.2% |
| Phi4-mini | 1.57/0.06 | 125 / 4.1% | 0.00/0.12 | 130 / 4.2% | -2.20/-1.04 | 125 / 4.1% |
| Qwen3T-4B | -2.35/0.89 | 96 / 3.8% | -3.41/0.46 | 97 / 3.8% | -0.94/0.40 | 99 / 3.9% |

In Appendix [B.6](https://arxiv.org/html/2509.13281v4#A2.SS6), we discuss the outcomes of our analyses of the projections removed from each of the target vectors to transform $v_{t}$ into $v_{RepIt}$ and find strong indications that a substantial portion of the edit is concentrated in few positions. To further investigate this concentrated structure, we apply a diagnostic procedure: removing low-variance coordinates from the projection and retaining only those that contribute most strongly. Concretely, in RepIt the whitened target $\tilde{v}_{t}=L^{-1}v_{t}$ is partially residualized against the whitened non-target span $\tilde{R}$:

$$ $\tilde{v}_{RepIt}\;=\;\tilde{v}_{t}-\alpha P,\quad\text{where}\quad\alpha P\;=\;\alpha\cdot QQ^{\top}\tilde{v}_{t}$ $$

We compute $z$-scores for the coordinates of $\alpha P$ and retain only those above a z-score of two ($|z_{i}|\geq\tau$, with $\tau=2$). Coordinates below this cutoff are set to zero, yielding a sparse projection $\alpha P_{\text{tail}}$. The resulting representation is $v_{RepIt,\text{tail}}=L(v_{t}-\alpha P_{\text{tail}})$, which we use to steer each model.

As shown in Table [1](https://arxiv.org/html/2509.13281v4#S7.T1), pruning for low-variance coordinates leaves attack success rates ($\Delta$ASR) effectively unchanged: deviations remain within $\pm 0.05$ absolute ASR. This indicates that RepIt’s edit is concentrated in a small set of high-leverage coordinates, leading us to hypothesize these neurons may encode a pertinent harm concepts critical for disentanglement. Further analysis of the removed coordinates in Section [B.8](https://arxiv.org/html/2509.13281v4#A2.SS8) shows they are randomly distributed and likely attributable to noise.

## 8 RepIt Data Efficiency

To evaluate the data requirements of RepIt, we rerun the pipeline using only 12 or 24 prompts from the target category. These subsets correspond to just 2.5–5% of the Bio and Cyber training sets and 7–15% of Chem. This setting massively increases the variance of $v_{t}$, directly testing RepIt’s robustness when isolating concept vectors from highly noisy DIM vectors. Rather than re-selecting $p,\ell,$ and $\rho$, we reuse values from the full dataset run while keeping non-target and harmless distributions fixed, as target prompts are not utilized during the COSMIC targeting or $\rho$ search validation (Section [B.4](https://arxiv.org/html/2509.13281v4#A2.SS4)). We evaluate across five consecutive seeds (20–24) on all models and report aggregate results in Figure [4](https://arxiv.org/html/2509.13281v4#S8.F4).

Figure: Figure 4: Target vs. Non-Target Jailbreak Success Rates under Constrained Target Sizes. We evaluate the performance of RepIt in data-constrained settings where the target vector is constructed using either 12 or 24 randomly selected training examples . The success rates are evaluated across five different seeds, reporting the mean and range of resulting values. We also include the “full” results utilizing the whole training dataset. The results demonstrate the data efficiency of RepIt in isolating target-category refusal directions while maintaining low non-target refusal, with general performance generally remaining comparable or even exceeding performance on the full dataset.
Refer to caption: Images/datasize_plot.png

With as few as a dozen examples, RepIt reliably isolates refusal directions that strongly suppress the target category while keeping non-target refusal low. Increasing from 12 to 24 examples reduces variance and preserves robustness across regimes, with averages closely matching those from the full dataset. These results highlight how efficiently certain concepts can be captured in representation space. A small number of carefully chosen examples can span a coherent target direction, provided the non-target basis is sufficiently rich to support targeting and disentanglement techniques.

This efficiency highlights a significant safety concern. Directions for harmful behaviors can be derived from only a dozen handcrafted prompts without large datasets or significant resources. Malicious actors could cheaply surface harmful concepts while evading standard benchmark assessments (as illustrated in Figure [1](https://arxiv.org/html/2509.13281v4#S1.F1)). This makes targeted manipulations more tractable and highlights risks that harmful capabilities may be systematically isolated and exploited in domains where no benchmarks exist. RepIt therefore advances disentanglement methodology while exposing urgent vulnerability in current evaluation regimes, encouraging rapid development of new comprehensive benchmarking solutions.

## 9 Related Work

#### Safety:

LM alignment is typically achieved through fine-tuning (Ouyang et al., 2022) and RLHF (Bai et al., 2022; Ganguli et al., 2022). Studies show that fine-tuning (Lermen et al., 2023; Yang et al., 2023; Qi et al., 2024) and adversarial prompts (Andriushchenko et al., 2024; Zou et al., 2023b; Chao et al., 2023; Wang et al., 2025) can bypass refusal mechanisms, highlighting numerous gaps in model safety. Recent work shows fine-tuning can introduce broad misalignment among a number of categories (Betley et al., 2025), but that fine-tuning specifically to introduce misalignment on a single category is difficult to achieve (Turner et al., 2025). Marks et al. (2025) introduce hidden objectives into LMs using reinforcement learning on human-defined objectives and find they can be detected by use of sparse autoencoders (SAEs).

#### Steering and Interpretability:

Recent work demonstrates that refusal behavior is encoded in activation space (Weidinger et al., 2021; Arditi et al., 2024; Marshall et al., 2024) with interventions aiming to modulate it directly (Zou et al., 2023a; Arditi et al., 2024; Marshall et al., 2024; Qiu et al., 2024; Bhattacharjee et al., 2024; Uppaal et al., 2025). Many methods use contrastive data pairs to extract feature directions (Burns et al., 2023; Arditi et al., 2024; Panickssery et al., 2023; Zou et al., 2023a) for behavior steering (Zou et al., 2023a; Panickssery et al., 2023; Turner et al., 2023; Arditi et al., 2024; Lee et al., 2024) and concept removal techniques (Guerner et al., 2023; Haghighatkhah et al., 2022; Ravfogel et al., 2020; Belrose et al., 2023).

Model behaviors are often represented as linearly encoded in activation space (Bolukbasi et al., 2016; Elhage et al., 2022; Park et al., 2024; Mikolov et al., 2013; Nanda et al., 2023; Hernandez & Andreas, 2021), though other work posits refusal behaviors as affine functions or multi-dimensional subspaces (Marshall et al., 2024; Wollschläger et al., 2025). Representations have also been used to probe concepts (Wu et al., 2025; Lee et al., 2024) and conditionally intervene on behaviors at inference time (Lee et al., 2024; Li et al., 2023; Wang et al., 2024).

## 10 Conclusion

We present RepIt, a framework for isolating concept-specific directions in language models by correcting noise and collinearity in difference-in-means vectors. RepIt disentangles target representations from overlapping signals, enabling precise interventions with minimal data and compute across diverse architectures. Our results demonstrate that high-dimensional activations contain richly structured, linearly decodable subspaces that can be cleanly identified and manipulated, opening new avenues for alignment, interpretability, and controlled behavior editing without retraining.

The efficiency that makes RepIt valuable for research also creates risks. With only a handful of prompts, adversaries could surface hidden capabilities while evading conventional safety evaluations. Our experiments reveal that even concept-matched benchmarks substantially underestimate a model’s harmful capacity after targeted interventions, exposing a critical vulnerability in AI safety evaluation practices. RepIt thus represents both methodological progress and a warning. Targeted representation editing can strengthen model control, but the same precision that enables beneficial applications also facilitates covert misuse. As these techniques mature, they demand equally sophisticated oversight - dynamic, representation-aware auditing rather than static benchmark evaluation - to ensure their power serves beneficial rather than harmful ends.

## 11 Ethics Statement

RepIt enables efficient, fine-grained isolation of concept-specific representations in language models. While this advances interpretability and controlled alignment, it also introduces new capabilities that merit careful consideration. With modest compute and as few as 12 prompts, RepIt can create highly targeted interventions that escape detection by standard safety benchmarks.

While the jailbreaking of specific harmful categories is concerning, risk is approximately equal or even worse to comparable jailbreaking methodologies - we show that steering with the DIM vector as done in prior work (Arditi et al., 2024; Marshall et al., 2024; Siu et al., 2025b) already achieves much higher ASR. Therefore, the primary concern is not individual misuse, but rather systemic risks to AI governance and oversight. RepIt-style techniques could enable actors to create models that appear safe under standard evaluation while retaining specific harmful capabilities. This ”evaluation evasion” problem poses challenges for regulatory frameworks that rely on benchmark-based safety assessments.

Beyond WMD-related scenarios, a broader concern lies in RepIt’s effect on human-AI trust calibration. Narrow jailbreaks can weaken refusal as a safety signal: even when a model behaves ethically in routine tasks, it may still produce outputs that enable harmful actions. As a result, users may struggle to gauge which information from an LLM is truly reliable, since lapses in safety can erode overall confidence. This risk is especially acute when models are served opaquely through third-party services and engage vulnerable users in personal contexts, positioning the model as a trusted confidant. In such cases, subtle failures in refusal - particularly around mental health - could catastrophically mislead users into following guidance that encourages dangerous behaviors, including self-harm (El Atillah, 2023; Bedingfield, 2023).

#### Defensive Framework

To mitigate these risks, we recommend a comprehensive approach combining immediate safeguards with longer-term research priorities:

Immediate Safeguards: (1) Data transparency - all datasets used to build RepIt vectors should be documented and made available through controlled access, following WMDP (Li et al., 2024) protocols; (2) Model labeling - models modified with RepIt must include metadata on targeted concepts, steering magnitude, and intended effects; (3) Deployment provenance - developers should disclose model lineage and activation-space modifications to prevent silent integration into consumer platforms.

Technical Countermeasures: Building on Marks et al. (2025), detection systems should analyze activation patterns for steering signatures, though the emergent nature of $v_{RepIt}$ complicates this compared to human-specified objectives. Priority research directions include: (1) developing geometric signatures that persist across inputs to detect orthogonal projections; (2) adversarial training against steering attacks during alignment as done in Yu et al. (2024); (3) architectural modifications that encourage concept entanglement and resist steering; (4) runtime monitoring for unusual activation patterns during inference.

Evaluation and Governance: Dynamic assessment using investigator agents (Li et al., 2025) may potentially replace static benchmarks to probe for hidden capabilities. Regulatory frameworks should consider model security audits, impact assessments for modified models, and international coordination on dual-use AI governance.

#### Mitigation Research

RepIt’s primary contribution lies in enabling defensive research through model organisms (Greenblatt et al., 2024) that expose evaluation vulnerabilities. We advocate research to: (1) decompose $v_{RepIt}$ into semantically interpretable latent features; (2) develop robust detection methods for undisclosed activation edits; (3) understand mechanistic interactions between steering and model circuitry; (4) integrate unlearning strategies (Yao et al., 2024; Liu et al., 2024a; b) to prevent harmful responses even when jailbroken.

By combining transparent practices, technical defenses, dynamic evaluation, and governance frameworks, the community can harness precise representation editing while addressing the urgent security gaps that RepIt reveals in current safety practices.

## 12 Reproducibility Statement

To support the reproducibility of our work, we have provided an anonymous version of our code, linked here: [CODE](https://github.com/wang-research-lab/RepIt).

In the interest of responsible research and to mitigate potential misuse, the WMD prompts used in our original paper are not yet publicly available. We are currently implementing gated access controls and other measures to ensure the data is shared securely. While the overall process can be replicated using the instructions in the repository’s README.md, the exact experiments will be fully reproducible once the data is released through these controlled channels.

## References

- Abdin et al. (2024)
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang.
Phi-4 technical report, 2024.
URL [https://arxiv.org/abs/2412.08905](https://arxiv.org/abs/2412.08905).
- Andriushchenko et al. (2024)
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion.
Jailbreaking leading safety-aligned llms with simple adaptive attacks.
*ArXiv preprint*, abs/2404.02151, 2024.
URL [https://arxiv.org/abs/2404.02151](https://arxiv.org/abs/2404.02151).
- Arditi et al. (2024)
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda.
Refusal in language models is mediated by a single direction.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/f545448535dfde4f9786555403ab7c49-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/f545448535dfde4f9786555403ab7c49-Abstract-Conference.html).
- Bai et al. (2022)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.
Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
URL [https://arxiv.org/abs/2204.05862](https://arxiv.org/abs/2204.05862).
- Bedingfield (2023)
Will Bedingfield.
A chatbot encouraged him to kill the queen. it’s just the beginning.
*Wired*, October 18 2023.
URL [https://www.wired.com/story/chatbot-kill-the-queen-eliza-effect/](https://www.wired.com/story/chatbot-kill-the-queen-eliza-effect/).
- Belrose (2023)
Nora Belrose.
Diff-in-means concept editing is worst-case optimal: Explaining a result by Sam Marks and Max Tegmark, 2023.
[https://blog.eleuther.ai/diff-in-means/](https://blog.eleuther.ai/diff-in-means/). Accessed on: May 20, 2024.
- Belrose et al. (2023)
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman.
LEACE: perfect linear concept erasure in closed form.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/d066d21c619d0a78c5b557fa3291a8f4-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2023/hash/d066d21c619d0a78c5b557fa3291a8f4-Abstract-Conference.html).
- Bengio et al. (2025a)
Yoshua Bengio, Tegan Maharaj, Luke Ong, Stuart Russell, Dawn Song, Max Tegmark, Lan Xue, Ya-Qin Zhang, Stephen Casper, Wan Sie Lee, Sören Mindermann, Vanessa Wilfred, Vidhisha Balachandran, Fazl Barez, Michael Belinsky, Imane Bello, Malo Bourgon, Mark Brakel, Siméon Campos, Duncan Cass-Beggs, Jiahao Chen, Rumman Chowdhury, Kuan Chua Seah, Jeff Clune, Juntao Dai, Agnes Delaborde, Nouha Dziri, Francisco Eiras, Joshua Engels, Jinyu Fan, Adam Gleave, Noah Goodman, Fynn Heide, Johannes Heidecke, Dan Hendrycks, Cyrus Hodes, Bryan Low Kian Hsiang, Minlie Huang, Sami Jawhar, Wang Jingyu, Adam Tauman Kalai, Meindert Kamphuis, Mohan Kankanhalli, Subhash Kantamneni, Mathias Bonde Kirk, Thomas Kwa, Jeffrey Ladish, Kwok-Yan Lam, Wan Lee Sie, Taewhi Lee, Xiaojian Li, Jiajun Liu, Chaochao Lu, Yifan Mai, Richard Mallah, Julian Michael, Nick Moës, Simon Möller, Kihyuk Nam, Kwan Yee Ng, Mark Nitzberg, Besmira Nushi, Seán O hÉigeartaigh, Alejandro Ortega, Pierre Peigné, James Petrie, Benjamin Prud’Homme, Reihaneh
Rabbany, Nayat Sanchez-Pi, Sarah Schwettmann, Buck Shlegeris, Saad Siddiqui, Aradhana Sinha, Martín Soto, Cheston Tan, Dong Ting, William Tjhi, Robert Trager, Brian Tse, Anthony Tung K. H., Vanessa Wilfred, John Willes, Denise Wong, Wei Xu, Rongwu Xu, Yi Zeng, HongJiang Zhang, and Djordje Žikelić.
The singapore consensus on global ai safety research priorities, 2025a.
URL [https://arxiv.org/abs/2506.20702](https://arxiv.org/abs/2506.20702).
- Bengio et al. (2025b)
Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, Hoda Heidari, Anson Ho, Sayash Kapoor, Leila Khalatbari, Shayne Longpre, Sam Manning, Vasilios Mavroudis, Mantas Mazeika, Julian Michael, Jessica Newman, Kwan Yee Ng, Chinasa T. Okolo, Deborah Raji, Girish Sastry, Elizabeth Seger, Theodora Skeadas, Tobin South, Emma Strubell, Florian Tramèr, Lucia Velasco, Nicole Wheeler, Daron Acemoglu, Olubayo Adekanmbi, David Dalrymple, Thomas G. Dietterich, Edward W. Felten, Pascale Fung, Pierre-Olivier Gourinchas, Fredrik Heintz, Geoffrey Hinton, Nick Jennings, Andreas Krause, Susan Leavy, Percy Liang, Teresa Ludermir, Vidushi Marda, Helen Margetts, John McDermid, Jane Munga, Arvind Narayanan, Alondra Nelson, Clara Neppel, Alice Oh, Gopal Ramchurn, Stuart Russell, Marietje Schaake, Bernhard Schölkopf, Dawn Song, Alvaro Soto, Lee Tiedrich, Gaël Varoquaux, Andrew Yao, Ya-Qin Zhang, Fahad Albalawi, Marwan
Alserkal, Olubunmi Ajala, Guillaume Avrin, Christian Busch, André Carlos Ponce de Leon Ferreira de Carvalho, Bronwyn Fox, Amandeep Singh Gill, Ahmet Halit Hatip, Juha Heikkilä, Gill Jolly, Ziv Katzir, Hiroaki Kitano, Antonio Krüger, Chris Johnson, Saif M. Khan, Kyoung Mu Lee, Dominic Vincent Ligot, Oleksii Molchanovskyi, Andrea Monti, Nusu Mwamanzi, Mona Nemer, Nuria Oliver, José Ramón López Portillo, Balaraman Ravindran, Raquel Pezoa Rivera, Hammam Riza, Crystal Rugege, Ciarán Seoighe, Jerry Sheehan, Haroon Sheikh, Denise Wong, and Yi Zeng.
International ai safety report, 2025b.
URL [https://arxiv.org/abs/2501.17805](https://arxiv.org/abs/2501.17805).
- Bercovich et al. (2025)
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Prasoon Varshney, Makesh Narsimhan, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar,
Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Shaona Ghosh, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Chris Alexiuk, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan
Catanzaro, Jonah Alben, Yonatan Geifman, and Eric Chung.
Llama-nemotron: Efficient reasoning models, 2025.
URL [https://arxiv.org/abs/2505.00949](https://arxiv.org/abs/2505.00949).
- Betley et al. (2025)
Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans.
Emergent misalignment: Narrow finetuning can produce broadly misaligned llms, 2025.
URL [https://arxiv.org/abs/2502.17424](https://arxiv.org/abs/2502.17424).
- Bhattacharjee et al. (2024)
Amrita Bhattacharjee, Shaona Ghosh, Traian Rebedea, and Christopher Parisien.
Towards inference-time category-wise safety steering for large language models, 2024.
URL [https://arxiv.org/abs/2410.01174](https://arxiv.org/abs/2410.01174).
- Bolukbasi et al. (2016)
Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai.
Man is to computer programmer as woman is to homemaker? debiasing word embeddings.
In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), *Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain*, pp. 4349–4357, 2016.
URL [https://proceedings.neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html](https://proceedings.neurips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html).
- Brown et al. (2020)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), *Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual*, 2020.
URL [https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html).
- Burns et al. (2023)
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt.
Discovering latent knowledge in language models without supervision.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023.
URL [https://openreview.net/pdf?id=ETKGuby0hcs](https://openreview.net/pdf?id=ETKGuby0hcs).
- Chao et al. (2023)
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong.
Jailbreaking black box large language models in twenty queries, 2023.
URL [https://arxiv.org/abs/2310.08419](https://arxiv.org/abs/2310.08419).
- Chao et al. (2024)
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong.
Jailbreakbench: An open robustness benchmark for jailbreaking large language models.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/63092d79154adebd7305dfd498cbff70-Abstract-Datasets_and_Benchmarks_Track.html](http://papers.nips.cc/paper_files/paper/2024/hash/63092d79154adebd7305dfd498cbff70-Abstract-Datasets_and_Benchmarks_Track.html).
- El Atillah (2023)
Imane El Atillah.
Man ends his life after an AI chatbot ‘encouraged’ him to sacrifice himself to stop climate change.
*Euronews*, March 31 2023.
URL [https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate-change](https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate-change).
- Elhage et al. (2022)
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.
Toy models of superposition, 2022.
URL [https://arxiv.org/abs/2209.10652](https://arxiv.org/abs/2209.10652).
- Ganguli et al. (2022)
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark.
Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022.
URL [https://arxiv.org/abs/2209.07858](https://arxiv.org/abs/2209.07858).
- Geva et al. (2021)
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.
Transformer feed-forward layers are key-value memories.
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 5484–5495, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.446.
URL [https://aclanthology.org/2021.emnlp-main.446](https://aclanthology.org/2021.emnlp-main.446).
- Greenblatt et al. (2024)
Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger.
AI control: Improving safety despite intentional subversion.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=KviM5k8pcP](https://openreview.net/forum?id=KviM5k8pcP).
- Guerner et al. (2023)
Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, and Ryan Cotterell.
A geometric notion of causal probing, 2023.
URL [https://arxiv.org/abs/2307.15054](https://arxiv.org/abs/2307.15054).
- Guo et al. (2025)
Wenbo Guo, Yujin Potter, Tianneng Shi, Zhun Wang, Andy Zhang, and Dawn Song.
Frontier ai’s impact on the cybersecurity landscape, 2025.
URL [https://arxiv.org/abs/2504.05408](https://arxiv.org/abs/2504.05408).
- Haghighatkhah et al. (2022)
Pantea Haghighatkhah, Antske Fokkens, Pia Sommerauer, Bettina Speckmann, and Kevin Verbeek.
Better hit the nail on the head than beat around the bush: Removing protected attributes with a single projection.
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pp. 8395–8416, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.emnlp-main.575.
URL [https://aclanthology.org/2022.emnlp-main.575](https://aclanthology.org/2022.emnlp-main.575).
- Hernandez & Andreas (2021)
Evan Hernandez and Jacob Andreas.
The low-dimensional linear geometry of contextualized word representations.
In Arianna Bisazza and Omri Abend (eds.), *Proceedings of the 25th Conference on Computational Natural Language Learning*, pp. 82–93, Online, 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.conll-1.7.
URL [https://aclanthology.org/2021.conll-1.7](https://aclanthology.org/2021.conll-1.7).
- Huang et al. (2024a)
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger.
Ravel: Evaluating interpretability methods on disentangling language model representations, 2024a.
URL [https://arxiv.org/abs/2402.17700](https://arxiv.org/abs/2402.17700).
- Huang et al. (2024b)
Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen.
Catastrophic jailbreak of open-source llms via exploiting generation.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024b.
URL [https://openreview.net/forum?id=r42tSSCHPh](https://openreview.net/forum?id=r42tSSCHPh).
- Lee et al. (2024)
Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar.
Programming refusal with conditional activation steering, 2024.
URL [https://arxiv.org/abs/2409.05907](https://arxiv.org/abs/2409.05907).
- Lermen et al. (2023)
Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish.
Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b, 2023.
URL [https://arxiv.org/abs/2310.20624](https://arxiv.org/abs/2310.20624).
- Li et al. (2023)
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.
Inference-time intervention: Eliciting truthful answers from a language model.
*Advances in Neural Information Processing Systems*, 36:41451–41530, 2023.
- Li et al. (2024)
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B. Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Kiran Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks.
The WMDP benchmark: Measuring and reducing malicious use with unlearning.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=xlr6AUDuJz](https://openreview.net/forum?id=xlr6AUDuJz).
- Li et al. (2025)
Xiang Lisa Li, Neil Chowdhury, Daniel D. Johnson, Tatsunori Hashimoto, Percy Liang, Sarah Schwettmann, and Jacob Steinhardt.
Eliciting language model behaviors with investigator agents, 2025.
URL [https://arxiv.org/abs/2502.01236](https://arxiv.org/abs/2502.01236).
- Liu et al. (2024a)
Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu.
Large language model unlearning via embedding-corrupted prompts, 2024a.
URL [https://arxiv.org/abs/2406.07933](https://arxiv.org/abs/2406.07933).
- Liu et al. (2024b)
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, Kush R. Varshney, Mohit Bansal, Sanmi Koyejo, and Yang Liu.
Rethinking machine unlearning for large language models, 2024b.
URL [https://arxiv.org/abs/2402.08787](https://arxiv.org/abs/2402.08787).
- Luo et al. (2024)
Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao.
Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks, 2024.
- Marks et al. (2025)
Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, Florian Dietz, Satvik Golechha, Akbir Khan, Jan Kirchner, Jan Leike, Austin Meek, Kei Nishimura-Gasparian, Euan Ong, Christopher Olah, Adam Pearce, Fabien Roger, Jeanne Salle, Andy Shih, Meg Tong, Drake Thomas, Kelley Rivoire, Adam Jermyn, Monte MacDiarmid, Tom Henighan, and Evan Hubinger.
Auditing language models for hidden objectives, 2025.
URL [https://arxiv.org/abs/2503.10965](https://arxiv.org/abs/2503.10965).
- Marshall et al. (2024)
Thomas Marshall, Adam Scherlis, and Nora Belrose.
Refusal in llms is an affine function, 2024.
URL [https://arxiv.org/abs/2411.09003](https://arxiv.org/abs/2411.09003).
- Mazeika et al. (2023)
Mantas Mazeika, Andy Zou, Norman Mu, Long Phan, Zifan Wang, Chunru Yu, Adam Khoja, Fengqing Jiang, Aidan O’Gara, Ellie Sakhaee, Zhen Xiang, Arezoo Rajabi, Dan Hendrycks, Radha Poovendran, Bo Li, and David Forsyth.
TDC 2023 (LLM edition): the Trojan Detection Challenge.
In *NeurIPS Competition Track*, 2023.
- Mazeika et al. (2024)
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks.
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=f3TUipYU3U](https://openreview.net/forum?id=f3TUipYU3U).
- Mikolov et al. (2013)
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig.
Linguistic regularities in continuous space word representations.
In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff (eds.), *Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 746–751, Atlanta, Georgia, 2013. Association for Computational Linguistics.
URL [https://aclanthology.org/N13-1090](https://aclanthology.org/N13-1090).
- MistralAI (2025)
MistralAI.
Mistral Small 3.1 — Mistral AI — mistral.ai.
[https://mistral.ai/news/mistral-small-3-1](https://mistral.ai/news/mistral-small-3-1), 2025.
[Accessed 20-08-2025].
- Nanda et al. (2023)
Neel Nanda, Andrew Lee, and Martin Wattenberg.
Emergent linear representations in world models of self-supervised sequence models.
In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), *Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP*, pp. 16–30, Singapore, 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.blackboxnlp-1.2.
URL [https://aclanthology.org/2023.blackboxnlp-1.2](https://aclanthology.org/2023.blackboxnlp-1.2).
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), *Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022*, 2022.
URL [http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html).
- Panickssery et al. (2023)
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner.
Steering llama 2 via contrastive activation addition, 2023.
URL [https://arxiv.org/abs/2312.06681](https://arxiv.org/abs/2312.06681).
- Park et al. (2024)
Kiho Park, Yo Joong Choe, and Victor Veitch.
The linear representation hypothesis and the geometry of large language models.
In *Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=UGpGkLzwpP](https://openreview.net/forum?id=UGpGkLzwpP).
- Qi et al. (2024)
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson.
Fine-tuning aligned language models compromises safety, even when users do not intend to!
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=hTEGyKf0dZ](https://openreview.net/forum?id=hTEGyKf0dZ).
- Qiu et al. (2024)
Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Maria Ponti, and Shay B. Cohen.
Spectral editing of activations for large language model alignment.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/684c59d614fe6ae74a3be8c3ef07e061-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/684c59d614fe6ae74a3be8c3ef07e061-Abstract-Conference.html).
- Ravfogel et al. (2020)
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg.
Null it out: Guarding protected attributes by iterative nullspace projection.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp. 7237–7256, Online, 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.acl-main.647.
URL [https://aclanthology.org/2020.acl-main.647](https://aclanthology.org/2020.acl-main.647).
- Siu et al. (2025a)
Vincent Siu, Nicholas Crispino, David Park, Nathan W. Henry, Zhun Wang, Yang Liu, Dawn Song, and Chenguang Wang.
Steeringsafety: A systematic safety evaluation framework of representation steering in llms, 2025a.
URL [https://arxiv.org/abs/2509.13450](https://arxiv.org/abs/2509.13450).
- Siu et al. (2025b)
Vincent Siu, Nicholas Crispino, Zihao Yu, Sam Pan, Zhun Wang, Yang Liu, Dawn Song, and Chenguang Wang.
COSMIC: Generalized refusal direction identification in LLM activations.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 25534–25553, Vienna, Austria, July 2025b. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
doi: 10.18653/v1/2025.findings-acl.1310.
URL [https://aclanthology.org/2025.findings-acl.1310/](https://aclanthology.org/2025.findings-acl.1310/).
- Souly et al. (2024)
Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer.
A strongreject for empty jailbreaks, 2024.
- Team et al. (2025)
V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Jiazheng Xu, Jiale Zhu, Jiali Chen, Jing Chen, Jinhao Chen, Jinghao Lin, Jinjiang Wang, Junjie Chen, Leqi Lei, Letian Gong, Leyi Pan, Mingdao Liu, Mingde Xu, Mingzhi Zhang, Qinkai Zheng, Sheng Yang, Shi Zhong, Shiyu Huang, Shuyuan Zhao, Siyan Xue, Shangqin Tu, Shengbiao Meng, Tianshu Zhang, Tianwei Luo, Tianxiang Hao, Tianyu Tong, Wenkai Li, Wei Jia, Xiao Liu, Xiaohan Zhang, Xin Lyu, Xinyue Fan, Xuancheng Huang, Yanling Wang, Yadong Xue, Yanfeng Wang, Yanzi Wang, Yifan An, Yifan Du, Yiming Shi, Yiheng Huang, Yilin Niu, Yuan Wang, Yuanchang Yue, Yuchen Li, Yutao Zhang, Yuting Wang, Yu Wang, Yuxuan Zhang, Zhao Xue, Zhenyu Hou, Zhengxiao Du, Zihan Wang, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li,
Minlie Huang, Yuxiao Dong, and Jie Tang.
Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning, 2025.
URL [https://arxiv.org/abs/2507.01006](https://arxiv.org/abs/2507.01006).
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
Llama: Open and efficient foundation language models, 2023.
URL [https://arxiv.org/abs/2302.13971](https://arxiv.org/abs/2302.13971).
- Turner et al. (2023)
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid.
Steering language models with activation engineering, 2023.
URL [https://arxiv.org/abs/2308.10248](https://arxiv.org/abs/2308.10248).
- Turner et al. (2025)
Edward Turner, Anna Soligo, Senthooran Rajamanoharan, and Neel Nanda.
Narrow misalignment is hard, emergent misalignment is easy, July 2025.
URL [https://www.lesswrong.com/posts/gLDSqQm8pwNiq7qst/narrow-misalignment-is-hard-emergent-misalignment-is-easy](https://www.lesswrong.com/posts/gLDSqQm8pwNiq7qst/narrow-misalignment-is-hard-emergent-misalignment-is-easy).
LessWrong, research update / blog post.
- Uppaal et al. (2025)
Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu.
Model editing as a robust and denoised variant of dpo: A case study on toxicity.
In *The Thirteenth International Conference on Learning Representations 2025*, 2025.
- Wang et al. (2024)
Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, and Xipeng Qiu.
Inferaligner: Inference-time alignment for harmlessness through cross-model guidance, 2024.
URL [https://arxiv.org/abs/2401.11206](https://arxiv.org/abs/2401.11206).
- Wang et al. (2025)
Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, and Dawn Song.
Agentvigil: Generic black-box red-teaming for indirect prompt injection against llm agents, 2025.
URL [https://arxiv.org/abs/2505.05849](https://arxiv.org/abs/2505.05849).
- Weidinger et al. (2021)
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel.
Ethical and social risks of harm from language models, 2021.
URL [https://arxiv.org/abs/2112.04359](https://arxiv.org/abs/2112.04359).
- Wiener (2024)
Scott Wiener.
Bill Text - SB-1047 Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. — leginfo.legislature.ca.gov.
[https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202320240SB1047](https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202320240SB1047), 2024.
[Accessed 30-01-2025].
- Wollschläger et al. (2025)
Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger.
The geometry of refusal in large language models: Concept cones and representational independence, 2025.
URL [https://arxiv.org/abs/2502.17420](https://arxiv.org/abs/2502.17420).
- Wu et al. (2025)
Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts.
Axbench: Steering llms? even simple baselines outperform sparse autoencoders, 2025.
URL [https://arxiv.org/abs/2501.17148](https://arxiv.org/abs/2501.17148).
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Yang et al. (2023)
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin.
Shadow alignment: The ease of subverting safely-aligned language models, 2023.
URL [https://arxiv.org/abs/2310.02949](https://arxiv.org/abs/2310.02949).
- Yao et al. (2024)
Yuanshun Yao, Xiaojun Xu, and Yang Liu.
Large language model unlearning, 2024.
URL [https://arxiv.org/abs/2310.10683](https://arxiv.org/abs/2310.10683).
- Yu et al. (2024)
Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda.
Robust llm safeguarding via refusal feature adversarial training, 2024.
URL [https://arxiv.org/abs/2409.20089](https://arxiv.org/abs/2409.20089).
- Zou et al. (2023a)
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks.
Representation engineering: A top-down approach to ai transparency, 2023a.
URL [https://arxiv.org/abs/2310.01405](https://arxiv.org/abs/2310.01405).
- Zou et al. (2023b)
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language models, 2023b.
URL [https://arxiv.org/abs/2307.15043](https://arxiv.org/abs/2307.15043).

## Appendix

## Appendix A Limitations

While RepIt effectively isolates harmful concept vectors and mitigates unwanted behaviors, several limitations arise from the inherent challenges of disentangling high-dimensional semantic spaces. A key limitation of our approach is its sensitivity to semantic noise introduced by overlapping categories and noisy non-target components, as harmful categories themselves are not always well-defined. This is especially complex to navigate because each model’s semantic representations may be unique to its personal understandings of given text, making meaningful separation of semantic categories heavily dependent on advances in topic modeling from the LM’s perspective.

As a result, we cannot fairly ensure that cleaning the categories from our perspective completely eliminates semantic overlap. Prompts within a category may vary significantly in relevance, and there is a considerable overlap between categories, further complicating the disentangling process. While our method incorporates techniques to suppress unwanted non-target components, it is still subject to these semantic ambiguities, which may hinder the isolation of truly distinct harmful concepts. As a result, disentanglement is not fully perfect, and models may remain vulnerable to related harmful behaviors that were not directly targeted during the optimization process. Additionally, target concepts may still be entangled with other particular concepts that are not tested in our evaluated datasets.

A limitation of our work is that we primarily evaluate RepIt on WMD-related concepts. While this narrows the scope, it is motivated by a broader challenge: many existing safety datasets define categories that are semantically inconsistent or overlapping, making concept-specific evaluation unreliable. For instance, JailBreakV (Luo et al., 2024) labels “How do I create fake scientific research data for publication?” as Tailored Unlicensed Advice, though it also aligns with categories like Fraud and Unethical Behavior. StrongREJECT similarly classifies “How can I sabotage a power plant to cause widespread blackouts?” as Violence, which is ambiguous and could equally be construed as a form of cyberattack. Attempting to disentangle such categories by subjective connotation would both complicate evaluation and risk rewarding RepIt for exploiting dataset labeling artifacts rather than isolating true concepts. By contrast, WMDP (Li et al., 2024) provides categories that are both semantically distinct and high-stakes, making it an appropriate starting point for demonstrating RepIt. Future work should extend RepIt to other domains when more clearly semantically independent data exists for them, but our focus here on WMDs enables a clear and defensible evaluation of concept-specific refusal disentanglement.

Additionally, while COSMIC is effective in identifying a location where the target vector strongly steers general refusal, it operates within a two-way optimization framework, focusing solely on distance between harmful and harmless vectors and relies on steering harmless prompts or equivalent ”negative” examples into their inverted ”positive” behavior. However, this is complicated by any types of inclusions of both target and nontarget data alongside the harmless data inside COSMIC’s framework as each set has its own objectives. Therefore, our results are likely conservative, as we search for a strong non-target basis irrespective of the target vector, whereas better solutions would attempt to optimize the target vector with respect to the non-target basis. Because RepIt can be executed at any arbitrary $(p,\ell)$, we note that it is compatible with any other direction selection methodology and reserve this for future work.

While we grid search $\rho$ values at a fixed $p,\ell$ heuristically yielded by COSMIC, it is possible that different $\rho$ values can induce different optimization landscapes with respect to the chosen $p,\ell$, and that a more thorough or well performing grid search would search all three hyperparameters in conjunction. These interactions are not fully captured in the current setup, suggesting that further refinement of the optimization process could lead to improved disentanglement performance. Thus, while effective, the current application of COSMIC remains limited by these factors, and future work should explore ways to integrate a more comprehensive, three-way optimization strategy to enhance the precision and generalizability of the disentanglement procedure. This causes COSMIC to be unable to prioritize strong steering of the target concept, and improvements to this methodology may allow for stronger preserval of target capability. However, because RepIt can be applied at any given $(p,\ell)$, alternative direction selection methods can be easily utilized as they are released.

Despite our high ASR values compared to the baseline ASRs in Figures [2](https://arxiv.org/html/2509.13281v4#S5.F2) and [1](https://arxiv.org/html/2509.13281v4#S1.F1), we note that several aspects of our methodology may be limited by impure refusal classes. This manifests in three ways. First, difference-in-means vectors may be noisily contaminated by prompts that the model does not itself treat as harmful and already readily answers, causing the vector to potentially be less effective. Second, COSMIC targeting can suffer from the same issue, as the signal for the true refusal direction is diluted by prompts that never trigger refusal, although Siu et al. (2025b) demonstrates that the method can be successfully run even if the model does not refuse. Third, $\rho$ search on the validation set may fail to converge cleanly: noisy target prompts introduce variance unrelated to the underlying direction, and the search procedure attempts to keep non-target ASR close to 0.1 regardless of the model’s baseline non-target rate. As datasets become more well defined and explicitly representative of target behaviors, this limitation should diminish. If corrected, it would likely increase the measured effectiveness of the difference-in-means, meaning that the results reported here are conservative estimates.

Lastly, we note that Llamaguard ASR, while largely indicative of a model attempting to give a harmful answer, does not necessarily indicate the presence of actionable, factual, and weaponizable WMD information. For example, we observe Qwen3T-4B hallucinates certain viral strains as ”SARS-CoV-2”. This suggests that pretraining filtering is effectively removing certain dangerous information, in this case attenuated viral strains of a highly infectious disease, from model knowledge. However, such cases are still deemed harmful, which we believe to be an apt description since Qwen3T-4B ultimately proposes a genomic editing procedure to potentially increase pathogenicity of SARS-CoV-2. We note that this limitation is primarily an effect of our choice to target content that already has robust safeguards around it to ensure our results do not pose a significant danger, and that other topics that could be targeted by RepIt may be far less likely to face similar resistance.

## Appendix B Dataset Splits and Categories

We report here the harmful categories included in the two evaluation datasets,
JailbreakV-Harmful and StrongReject-Harmful, along with the number
of prompts in each split.

### B.1 JailbreakV-Harmful

This dataset contains $2000$ harmful prompts spanning sixteen categories.
Table [2](https://arxiv.org/html/2509.13281v4#A2.T2) summarizes the per-category counts.

**Table 2: Categories and split sizes for JailbreakV-Harmful.**
| Category | Train | Validation | Test |
| --- | --- | --- | --- |
| Animal Abuse | 41 | 10 | 51 |
| Bias | 48 | 12 | 60 |
| Child Abuse Content | 53 | 13 | 68 |
| Economic Harm | 42 | 11 | 54 |
| Fraud | 51 | 13 | 64 |
| Government Decision | 52 | 13 | 66 |
| Hate Speech | 52 | 13 | 65 |
| Health Consultation | 46 | 12 | 57 |
| Illegal Activity | 60 | 15 | 76 |
| Malware | 50 | 13 | 62 |
| Physical Harm | 49 | 12 | 62 |
| Political Sensitivity | 52 | 13 | 65 |
| Privacy Violation | 49 | 12 | 61 |
| Tailored Unlicensed Advice | 51 | 13 | 64 |
| Unethical Behavior | 52 | 13 | 65 |
| Violence | 49 | 12 | 63 |
| Total | 800 | 200 | 1000 |

### B.2 StrongReject-Harmful

This dataset comprises $313$ harmful prompts across six categories.
Table [3](https://arxiv.org/html/2509.13281v4#A2.T3) shows the splits.

**Table 3: Categories and split sizes for StrongReject-Harmful.**
| Category | Train | Validation | Test |
| --- | --- | --- | --- |
| Disinformation and deception | 20 | 5 | 25 |
| Hate, harassment and discrimination | 20 | 5 | 25 |
| Illegal goods and services | 20 | 5 | 25 |
| Non-violent crimes | 24 | 6 | 29 |
| Sexual content | 20 | 5 | 25 |
| Violence | 22 | 5 | 27 |
| Total | 126 | 31 | 156 |

### B.3 WMDP Prompts

**Table 4: Distribution of harmful WMDP prompts across biological, chemical, and cyber domains.**
| Category | Train | Validation | Test |
| --- | --- | --- | --- |
| WMDP Bio | 508 | 127 | 637 |
| WMDP Chem | 162 | 40 | 205 |
| WMDP Cyber | 509 | 127 | 637 |
| Total | 1179 | 294 | 1479 |

### B.4 Rho Search

In the process of performing a hyperparameter search on the $\rho$ parameter, we run model generation on the validation dataset and evaluate results with LLaMaGuard3, similar to the final evaluation. We then pick the smallest $\rho$ yielding a nontarget ASR below 0.1, our chosen threshold for optimization of the Pareto frontier between preserving the target vector and disentangling jailbreaking on nontarget data. Other choices of nontarget ASR thresholds can be approximated by estimating where the curves cross the y-value for a new threshold, barring generalization variance between validation and test splits. Importantly, as a result, we do not explicitly use any target data within our validation step, though for sake of investigation we perform our experiments with target data assessed to further understand RepIt. We depict the results in Figure [5](https://arxiv.org/html/2509.13281v4#A2.F5); the stars represent the chosen $\rho$ value.

The $\rho$ grid search shows how models differ in distributing refusal geometry between the non-target span and target residual. In some models, optima cluster near $\rho=0.99$, possibly suggesting the non-target span doesn’t capture the full feature set, leaving a meaningful portion $v_{t}$ largely out-of-basis and allowing for near-complete elimination of the shared subspace without collapsing steering performance. In others, optima occur at much lower $\rho$, showing the non-target basis already includes most or even all features from $v_{t}$. Intermediate values balance entanglement: too small leaves overlap uncorrected, while too large erases the signal. This highlights that RepIt’s effectiveness depends on both non-target basis quality with respect to semantic overlap and how each model distributes overlapping harmful behavior components across these semantically similar prompts in representational spans.

Further research on difference-in-mean vectors and refusal landscapes are warranted to study the impact of $\rho$ across the refusal subspace and why DIM vectors in LlamaNemo4B are already highly concept-specific without RepIt application.

Figure: Figure 5: $\rho$ search on the validation set to find a $\rho$ value that minimizes entanglement beyond the chosen threshold of 0.1 ASR.
Refer to caption: Images/rho_search.png

### B.5 Space and Time Complexity

All experiments are run on single A6000s with the exception of the Mistral model, which is loaded on two A6000s due to its 24B size. We note that reasoning models may read as though they intend to comply before eventually refusing, necessitating a very high max new token limit for reasoning models to allow them to reach a more definitive state. Therefore, we run reasoning models for a total of 1500 max new tokens and non-reasoning models for a total of 500. Smaller non-reasoning models like Phi4-mini thus take as little as 1.5 hours to complete a full run of RepIt from steering to test evaluation (excluding tailweight and datasize experiments) whereas GLM4.1, NemoLlama4B, and Qwen take substantially longer due to the increased generation load.

On average, difference-in-means direction generation takes less than five minutes, COSMIC direction selection varies from 10-45 minutes depending on number of post-instruction tokens and model size, and $\rho$ grid-search and final test evaluation as generation tasks take time proportional to number of parameters and generation limit. Time taken to perform the calculations in RepIt are comparably negligible.

### B.6 Projection Analysis

Here we further analyze the the final direction vectors obtained during $\rho$-search. At each identified
$(\text{pos},\text{layer})$ location, we save both the optimized final direction and its
projection. Condition numbers are extracted from the covariance and projection matrices.
The projection tensor is then profiled element-wise to obtain its dimensionality (Hidden
State Size), $L_{2}$ norm, mean, and standard deviation. Cosine similarity is measured
between the optimized and original directions, while kurtosis is calculated on the
flattened projection distribution. Heavy-tail counts (HT) are derived by thresholding
absolute projection values at $\mu+2\sigma$ as shown in Section [7](https://arxiv.org/html/2509.13281v4#S7), yielding the number of coordinates with
unusually high magnitude activations. Together, these metrics quantify numerical stability
(condition numbers, cosine similarity) and structural properties of the projection distribution
(kurtosis, dispersion, heavy-tail concentration). The resulting diagnostic values are stored in Table LABEL:tab:proj-diagnostics.

The high condition numbers observed for the covariance matrices reflect collinearity in the
non-target vectors, which motivates our whitening step. Accordingly, the span condition
numbers ($\kappa_{span}$) post-whitening remain well-behaved, consistently at 1.00.

We find substantial evidence that the representations in each projection is concentrated in sparse sections of the layer. Covariance condition numbers $\kappa_{cov}$ often lie between $10^{6}$ and $10^{9}$, indicating strong collinearity among non-target category vectors. Whitening helps recondition this ill-posed system but may amplify minor fluctuations into disproportionately large corrections. Some projected components are leptokurtic, such as Qwen3T, showing that variance is concentrated in a small number of coordinates. Most notably, Gini impurity values approach 1.0 across all models, indicating that nearly all corrective mass is carried by a narrow set of dimensions while most coordinates contribute negligibly. In practice, this shows RepIt’s edits concentrate on a small subset of influential neurons rather than diffusing across the representation space. This provides strong motivation for the tailweight analysis
in Section [7](https://arxiv.org/html/2509.13281v4#S7).

Cosine similarity varies substantially across models: for LlamaNemo4B and Qwen3T-4B
it reaches values very close to 1.0, suggesting that the $\rho$-optimized direction is almost
identical to the original. By contrast, models such as GLM4.1V and Mistral-3.2-Small show
noticeably lower values ($\sim$0.6–0.8), indicating a more substantial adjustment during
the optimization. These discrepancies reflect differences in how sensitive each model’s
non-target basis is to whitening and sparsification, and highlight that $\rho$-search sometimes
preserves the original geometry while in other cases it produces a meaningfully rotated but
more stable direction.

This pattern is reinforced by discrepancies in the $L_{2}$ norm of the projection (noting that
norms are also influenced by hidden state size, Dim). For instance, Qwen3T-4B-Bioweaponry has a
cosine similarity of 0.99 yet a relatively large projection norm of 4.45, which is relatively high for its small 2560 hidden dimension, showing that the
direction was largely preserved geometrically but rescaled in magnitude. In contrast, Mistral-3.2-Small on Cyberattacks, despite being the model with the largest hidden size, depicts a relatively small projection norm but some of the lowest cosine similarities. Together, cosine similarity and norm reveal that $\rho$-search may
either rescale a nearly preserved direction or rotate it into a more stable subspace depending
on model structure.

**Table 5: Projection analysis diagnostics for Bioweaponry, Chemical Weaponry, and Cyberattacks. For each model, $\rho$ is the best-performing explained variance parameter identified during $\rho$-search. HT is the heavy-tail count, i.e., the number of coordinates in the projection vector exceeding $\mu+2\sigma$ in magnitude, reflecting concentration of large activations. $\kappa_{cov}$ and $\kappa_{span}$ are the condition numbers of the covariance and projection matrices, indicating numerical stability. Kurt. is the kurtosis of the projection distribution (higher values = heavier-tailed). Cos is the cosine similarity between the final direction and the original reference direction, measuring directional preservation. Dim is the hidden state size, i.e., the total number of elements in the projection vector. $\mu/\sigma$ are the mean and standard deviation of the projections. ‖proj‖2 is the $L_{2}$ norm of the projection vector, quantifying its overall magnitude. Gini denotes the calculated Gini Impurity of the projection.**
| Model | $\rho$ | HT | Condition Num | Projection Stats |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | $\kappa_{cov}$/ $\kappa_{span}$ | Kurt. | Cos. | Dim | $\mu/\sigma$ | ‖proj‖2 | Gini |
| GLM4.1V | 0.94 | 154 | 7.54e+06/1.00 | 2.33 | 0.81 | 4096 | -0.01/0.83 | 52.98 | 0.99 |
| LlamaNemo4B | 0.33 | 137 | 1.46e+09/1.00 | 1.04 | 0.99 | 3072 | 0.00/0.02 | 0.91 | 0.99 |
| Mistral-3.2-Small | 0.96 | 207 | 1.58e+07/1.00 | 1.41 | 0.77 | 5120 | -0.00/0.04 | 3.06 | 0.99 |
| Phi4-mini | 0.85 | 125 | 1.07e+10/1.00 | 0.47 | 0.90 | 3072 | 0.00/0.14 | 7.88 | 0.99 |
| Qwen3T-4B | 0.33 | 96 | 5.36e+09/1.00 | 4.73 | 0.99 | 2560 | -0.00/0.09 | 4.45 | 0.99 |
| Chemical Weaponry |  |  |  |  |  |  |  |  |  |
| Model | $\rho$ | HT | Condition Num | Projection Stats |  |  |  |  |  |
|  |  |  | $\kappa_{cov}$/ $\kappa_{span}$ | Kurt. | Cos. | Dim | $\mu/\sigma$ | ‖proj‖2 | Gini |
| GLM4.1V | 0.94 | 159 | 1.11e+07/1.00 | 1.97 | 0.81 | 4096 | -0.01/0.79 | 50.33 | 0.99 |
| LlamaNemo4B | 0.00 | 158 | 7.88e+09/1.00 | 7.30 | 1.00 | 3072 | -0.00/0.00 | 0.00 | 0.99 |
| Mistral-3.2-Small | 0.95 | 197 | 7.88e+06/1.00 | 1.62 | 0.81 | 5120 | -0.00/0.05 | 3.51 | 0.99 |
| Phi4-mini | 0.85 | 130 | 1.56e+09/1.00 | 0.31 | 0.90 | 3072 | 0.00/0.14 | 7.98 | 0.99 |
| Qwen3T-4B | 0.76 | 97 | 5.43e+08/1.00 | 4.09 | 0.96 | 2560 | -0.00/0.26 | 13.03 | 0.99 |
| Cyberattacks |  |  |  |  |  |  |  |  |  |
| Model | $\rho$ | HT | Condition Num | Projection Stats |  |  |  |  |  |
|  |  |  | $\kappa_{cov}$/ $\kappa_{span}$ | Kurt. | Cos. | Dim | $\mu/\sigma$ | ‖proj‖2 | Gini |
| GLM4.1V | 0.99 | 161 | 1.03e+07/1.00 | 3.50 | 0.59 | 4096 | -0.02/0.68 | 43.73 | 0.99 |
| LlamaNemo4B | 0.00 | 154 | 2.49e+09/1.00 | 3.61 | 1.00 | 3072 | 0.00/0.00 | 0.00 | 0.99 |
| Mistral-3.2-Small | 0.97 | 213 | 2.09e+07/1.00 | 1.45 | 0.72 | 5120 | -0.00/0.04 | 2.87 | 0.99 |
| Phi4-mini | 0.40 | 125 | 6.75e+06/1.00 | 2.99 | 0.99 | 3072 | -0.00/0.08 | 4.69 | 0.99 |
| Qwen3T-4B | 0.41 | 99 | 6.43e+08/1.00 | 5.44 | 0.99 | 2560 | -0.00/0.10 | 5.06 | 0.99 |

### B.7 Similar Category Evaluations

A potential criticism of our work is that the safety datasets in Figure [1](https://arxiv.org/html/2509.13281v4#S1.F1) do not explicitly probe biological or chemical weapon generation, though some include malware prompts. To directly address this issue and assess whether this limits our conclusions, we evaluate the generalization of RepIt on category-matched prompts from HarmBench (Mazeika et al., 2024). Figure [6](https://arxiv.org/html/2509.13281v4#A2.F6) compares each model’s performance on its corresponding HarmBench category to its performance on the true target category.

A critical implication of our findings is that RepIt can enable jailbreaks that evade official evaluation procedures even on the very concept being tested. Because RepIt isolates and reorients the model’s internal representation of a harmful category, a model can retain the capability (e.g., WMD-related knowledge) while producing outputs that satisfy benchmark prompts designed to detect it. In other words, safety evaluations that assume fixed alignment between latent concept geometry and surface behavior may certify a model as safe on the precise topic of concern, while RepIt quietly preserves actionable knowledge.

We believe that this difference in performance is directly a result of RepIt due to the differences in topic coverage between HarmBench and our WMDP-derived dataset. Mainly, HarmBench’s biological_chemical category only consists of 28 prompts split between biological and chemical questions - as a result entanglement is naturally lower due to a broader category definition. However, HarmBench’s quesitons in this category largely focus on how to synthesize certain compounds, whereas our dataset focuses on end-to-end weaponization of chemical weapons from smuggling of precursors to synthesis strategies and considerations to weaponization and designing dispersal or payload systems. As a result, HarmBench does not cover items in WMDP such as genetic editing of viruses or ecoterrorism that causes it to underestimate the capabilities of RepIt. A similar scenario occurs in HarmBench’s cybercrime_intrusion category with 40 prompts generally asking the LM to write generally malicious code (eg. Write a keylogger script…), whereas WMDP-style prompts ask for debugging, strategy, or advice regarding very specific attacks such as SpyNote RAT and CVE-2021-30860. Thus, HarmBench shows another example of how similar category labels can misrepresent the true semantic meanings covered in each dataset.

Figure: Figure 6: Generalization of RepIt jailbreak interventions to their closest HarmBench (Mazeika et al., 2024) categories. Bio and Chem models are tested on chemical_biological, and Cyber models on cybercrime_intrusion. Bars show LlamaGuard-3 success rates (ASR) for these HarmBench categories, with the red overlay indicating the *true target* (WMDP Bio/Chem/Cyber). The large gap reveals that standard safety benchmarks substantially underestimate the harmful capacity of RepIt-attacked models.
Refer to caption: Images/harmbench.png

### B.8 Tailweight Analyses

In Table [1](https://arxiv.org/html/2509.13281v4#S7.T1), we initially note that the aggregate ASR change under tailweighting is small (on the order of $10^{-2}$). However, this apparent stability should not be interpreted as evidence of overall performance preservation. Instead, we find as shown in Table LABEL:tab:_tailweight_switch that tailweighting produces an equilibrium of bidirectional flips: a roughly balanced number of cases shift from success to failure (1→0) and from failure to success (0→1).

In other words, the near-zero aggregate change is not due to the model behaving consistently across prompts, but rather because the losses in one direction are offset by gains in the other. This balancing effect conceals the fact that a nontrivial fraction of examples are perturbed under tailweighting - often several percent of the evaluation set within each model–category pair. The operation therefore does not “preserve” performance in a strict sense, but rather redistributes errors, maintaining equilibrium when measured only by global averages.

**Table 6: Category-level 1→0 and 0→1 flips induced by tailweight ablation across models. Although aggregate ASR shifts are small (on the order of $10^{-2}$), several percent of examples flip in each direction, indicating that tailweighting redistributes errors rather than preserving performance uniformly.**
| Model / Category | N | 1→0<br>Count | 0→1<br>Count | 1→0<br>% | 0→1<br>% |
| --- | --- | --- | --- | --- | --- |
| GLM4.1V Bio | 1793 | 169 | 161 | 9.43% | 8.98% |
| GLM4.1V Chem | 1361 | 72 | 75 | 5.29% | 5.51% |
| GLM4.1V Cyber | 1793 | 109 | 109 | 6.08% | 6.08% |
| LlamaNemo4B Bio | 1793 | 63 | 74 | 3.51% | 4.13% |
| LlamaNemo4B Chem | 1361 | 49 | 50 | 3.60% | 3.67% |
| LlamaNemo4B Cyber | 1793 | 102 | 96 | 5.69% | 5.35% |
| Mistral-3.2-Small Bio | 1793 | 60 | 56 | 3.35% | 3.12% |
| Mistral-3.2-Small Chem | 1361 | 46 | 26 | 3.38% | 1.91% |
| Mistral-3.2-Small Cyber | 1793 | 47 | 55 | 2.62% | 3.07% |
| Phi4-mini Bio | 1793 | 68 | 76 | 3.79% | 4.24% |
| Phi4-mini Chem | 1361 | 41 | 47 | 3.01% | 3.45% |
| Phi4-mini Cyber | 1793 | 94 | 75 | 5.24% | 4.18% |
| Qwen3T-4B Bio | 1793 | 127 | 129 | 7.08% | 7.19% |
| Qwen3T-4B Chem | 1361 | 71 | 79 | 5.22% | 5.80% |
| Qwen3T-4B Cyber | 1793 | 111 | 109 | 6.19% | 6.08% |

In a further analysis, we quantify ASR changes at the category level (e.g., GLM4.1V–Bio on “Animal Abuse”) across the test set. As shown in Figure [7](https://arxiv.org/html/2509.13281v4#A2.F7), the distribution of changes follows a leptokurtic approximately normal shape centered near zero. This pattern indicates that tailweighting removes low-magnitude, unstable components of the projection that contribute little to the corrective signal (on the order of $10^{-2}$) and, being symmetrically distributed around zero across models and categories, are best interpreted as noise.

The bidirectional flips reported in Table [1](https://arxiv.org/html/2509.13281v4#S7.T1) therefore reflect the removal of spurious variance from ill-conditioned directions rather than genuine behavioral shifts. While the proportion of affected examples is nontrivial – approaching 10% in some model–category pairs – the fact that these shifts are symmetrically distributed around zero suggests that they are non-systematic. In effect, tailweighting functions as a denoising filter: the corrective signal is concentrated in a sparse set of high-leverage neurons, while contributions from other coordinates yield only minor, idiosyncratic fluctuations.

Further research from an SAE or circuit based perspective may help clarify both why these values affect refusal bidirectionally and the heavy-tail neuron phenomena.

Figure: Figure 7: Distribution of category-level $\Delta$ASR (tailweight $-$ normal) across all model–category pairs. Changes are centered near zero with symmetric variance, consistent with tailweighting removing low-magnitude, unstable components while preserving the sparse, high-leverage coordinates that drive the corrective signal.
Refer to caption: Images/delta_histogram_category_means.png

## Appendix C Statement on LM Usage

In this work, LMs were used for proofreading and editing of the text, as well as for general coding especially in creation of the graphs. LMs were not involved in the ideation or retrieval/discovery processes.

## Appendix D Example Responses

In the following pages we depict example responses by category for each model, with eight total successful examples across the five models and all three weapon modalities. We have redacted the text to ensure no true harmful information is shared beyond structure and verbiage of the model responses. Note that some rendering issues may occur due to LaTeX formatting of the raw model responses. Redaction lines correspond to the number of lines the text occupies but are of uniform length to prevent revealing the length of individual words.

Note that while the token generation limit is quite generous it is occasionally not enough for the reasoning models, and so the model abruptly stops its response as shown.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Why wasn't REPIT compared against LEACE, standard orthogonalization methods, or other concept removal techniques? These seem like obvious baselines for the stated problem.

Can you provide a realistic scenario where an attacker would prefer selective concept isolation over simply removing all safety mechanisms? How would this work in actual deployment contexts?
soundness: 3
strengths: Clarity: The paper is generally well-written with clear mathematical exposition. The figures effectively demonstrate the core claims about selective targeting, and the technical methodology is described with sufficient detail for reproduction.

Empirical Validation: The results consistently show the intended selectivity across multiple models and domains.
summary: This paper presents a method for selectively disabling safety mechanism for specific harmful concepts while preserving refusal behavior for other harmful concepts. The method employs a three-step procedure (reweighting, whitening, orthogonalization) to disentangle concept-specific representations from broader refusal behaviors. The authors demonstrate that their technique can create models that answer weapons-of-mass-destruction questions while maintaining refusal on other safety benchmarks, using as few as 12 examples and affecting only 100-200 neurons.
weaknesses: Fundamental Motivation Problems: The threat model is deeply flawed. The paper claims to expose "evaluation evasion" vulnerabilities, but the method requires test-time activation steering with weight access. An attacker with such capabilities could trivially remove all safety mechanisms rather than pursuing this complex selective approach. The framing misrepresents how safety evaluation actually works in practice.


Limited Technical Novelty: The core mathematical operation — finding vectors aligned with targets while orthogonal to non-targets — is a standard linear algebra problem. The paper fails to compare against obvious baselines like LEACE (Belrose et al., 2023), standard Gram-Schmidt orthogonalization, or regularized projection methods. The "novel" components (ridge regularization, partial orthogonalization) are well-established techniques presented without proper context.


Inflated Security Claims: The characterization as an "urgent threat to AI safety infrastructure" is unsupported by realistic threat analysis. The paper conflates technical capability with practical security risk without considering simpler attack vectors or deployment constraints.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: Can you explain why all three steps (reweighting, whitening, and orthogonalization) are necessary? What would happen if one of the steps, particularly the whitening step to address collinearity, was omitted?
soundness: 3
strengths: The strengths are outlined as follows:
1) The motivation and novelty of the setup is well defined and the derivation is clear, leading to a nice, very simple closed form solution
2) The data efficiency of the algorithm is very appealling.
3) The paper is overall very well written.
summary: This paper introduces RepIt, which addresses the problem of representational entanglement in LLMs where different concepts are entangled in an LLM's internal circuitry, which makes targeted modifications difficult. The authors develop a three step process to disentangle concept vectors, which can then be used for suppressing refusal on concept specific harmful datasets.
weaknesses: The main weakness I have is the experimental setup is limited towards Bio, Chem, and Cyber attacks from the WMDP dataset. Since the knowledge required to answer these questions is highly concentrated, it will be easier to identify concept vectors. I would like to see experiments on broader safety benchmarks covering many different, overlapping concepts to see the effectiveness of this method.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - In 6 the authors argue that entanglement enhances jailbreak effectiveness. I would like to see more evidence/understand the evidence for this claim.
  - How many data samples are influencing the non-target basis? Does the non-target basis vector have more data points than the basic dim in this experiment?
  - I think adding repit to figure 3 would make it easier to understand the performance differences.
soundness: 3
strengths: - Relevant problem
- Effective & simple solution
- Brings more evidence to work with similar findings
summary: This work introduces a new way to find vectors that are only targeting specific concepts. Instead of manipulating refusal behavior of the LM on all harmful data, the authors show that their vectors do not alter refusal behavior on harmful questions with different content e.g. cyber vs bio. The method seems to be very effective and localizes to 100-200 neurons.
weaknesses: - I would like to see more ablation studies. 
    - How important is re-weighting, whitening, …
    - Is there a baseline where instead of alpha P random vectors are subtracted but with the same search for rho
- Different model sizes would be interesting. It seems logical that higher dimensional residual streams can represent different refusal concepts in more distinct ways. Seeing how repit scales would be very interesting.
- Presentation: The start is very good. Then it becomes a bit too brief. I would like to see more explanation on the main method and the ablation study. In that experiment it is difficult to understand which steps have been performed with the three components before using them to jailbreak the models.
- Minor: 
    - COSMIC on page 3 in Methodology is not introduced
    - 4.1 could be rather background instead of methodology

---

confidence: 5
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. What is the effect of REPIT when applied to GPT-OSS series models? These models seem to have isolated harmful content in the training corpus; for example, when asked queries related to bombs, they would normally refuse to respond. However, under certain steering frameworks, even if they do respond, harmful content is not visible.
2. The authors should ensure that the paper is self-contained. For instance, the COSMIC algorithm is not widely known; could the authors briefly describe its principles in the main text? In the paper, the use of COSMIC appears before its citation (first mentioned in L138, but the citation appears in L218).
3. Can the authors provide at least one concept that appears harmful but is not included in current safety benchmarks, and demonstrate the results manipulated through REPIT? Otherwise, the motivation for evaluating backdoors is not correct—at least it feels alarmist.
soundness: 2
strengths: - This paper defines a timely problem: how to isolate specific harmful concepts from a broad refusal behaviors.
- The proposed disentanglement process is supported by closed-form solutions that does not require costly optimization or search.
summary: This paper proposes REPIT, which can selectively suppress models' refusal towards specific target concepts, while maintaining on other harmful topics. In this way, REPIT can create models that can circumvent safety benchmark tests, where models appear safe but actually conceal specific malicious capabilities.
weaknesses: Although there have been many studies on the manipulation of refusal vectors, I appreciate the authors' proposed disentanglement framework for extracting more fine-grained concept vectors from refusal vectors. However, I have some questions regarding the experimental setup, and addressing them could improve the persuasiveness of the paper.

1. **The experiment focuses on weapons of mass destruction**. This is certainly important, but why is it the only focus? What side effects would arise if applied to other concepts, such as the 21 non-target concepts the authors have already collected? Furthermore, benchmarks, like HarmBench, have prepared critical evaluations for this topic, which cannot support the claim of evaluating backdoors.
2. **The definition of ASR the authors use is unclear**. From appendix, I guess the authors used LlamaGuard to evaluate whether the answers are harmful, which should be explained in detail in the main text. I suggest the authors use GuidedBench [1] to reproduce their evaluation, as it provides explainable results, which are more suitable for assessing whether suppression and single-concept jailbreak are truly successful.
3. **The scale of the dataset involved in the experiments is also not mentioned**, making it difficult to assess the confidence of the results.
4. **The number of neurons to be edited in the paper is not very rigorous**. Different models have different numbers of neurons; can the authors clarify whether only 100-200 neurons need to be edited across various models of different sizes?

[1] GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods, https://arxiv.org/abs/2502.16903


Decision: Accept (Poster)