Programming Refusal with Conditional Activation Steering

## Abstract

Abstract LLMs have shown remarkable capabilities, but precisely controlling their response behavior remains challenging.
Existing activation steering methods alter LLM behavior indiscriminately, limiting their practical applicability in settings where selective responses are essential, such as content moderation or domain-specific assistants.
In this paper, we propose Conditional Activation Steering (CAST), which analyzes LLM activation patterns during inference to selectively apply or withhold activation steering based on the input context.
Our method is based on the observation that different categories of prompts activate distinct patterns in the model’s hidden states.
Using CAST, one can systematically control LLM behavior with rules like “if input is about hate speech or adult content, then refuse” or “if input is not about legal advice, then refuse.”
This allows for selective modification of responses to specific content while maintaining normal responses to other content, all without requiring weight optimization.
We release an open-source implementation of our framework at https://github.com/IBM/activation-steering .

## 1 Introduction

A striking feature of large language models (LLMs) is their ability to process high-level concepts through rich representations in their activations.
This feature has given rise to techniques like activation steering (Turner et al., ), which leverage these learned representations to efficiently and predictably alter LLM behavior (Wang et al., ; Zou et al., ; Rimsky et al., ).

Problem: Lack of conditional control in activation steering.
Activation steering offers a promising alternative to optimization-based techniques by directly manipulating the model’s native representations, often requiring only a simple activation addition step during each forward call (Turner et al., ).
While activation steering has shown promise in altering LLM behavior, such as removing or inducing refusal behavior, a key limitation of current methods is the inability to condition when and what to refuse (Zheng et al., ; Ghandeharioun et al., ).
That is, adding a “refusal vector” using existing activation steering methods increases refusal rates indiscriminately across all inputs, limiting the model’s utility (Arditi et al., ).

Contribution: Expanding activation steering formulation.
We introduce Conditional Activation Steering (CAST), a method that enables fine-grained, context-dependent control over LLM behaviors.
We introduce a new type of steering vector in the activation steering formulation, the condition vector, representing certain activation patterns induced by the prompt during the inference process.
A simple similarity calculation between this condition vector and the model’s activation at inference time effectively serves as a switch, determining whether to apply the refusal vector. This approach allows for selective refusal of harmful prompts while maintaining the ability to respond to harmless ones, as depicted in Figure [1](#S1.F1).
A breakdown of this figure is presented in Table [3](#A1.T3).
Furthermore, CAST maintains the data, runtime, and compute efficiency of activation steering (Figure [6](#S4.F6)) while adding controllability, enabling the implementation of behavioral rules in LLMs without significant costs.

Figure: Figure 1: Conditional activation steering induces targeted refusal. Activation steering (AST) induces the model to indiscriminately refuse all prompts, including harmless ones (blue bars). Conditional activation steering (CAST) allows selective refusal, refusing harmful prompts while minimizing the harmless refusal rate.
Refer to caption: /html/2409.05907/assets/figures/intro-bar-alt.png

Application: Selecting what to refuse.
Many alignment goals concern contextually refusing specific classes of instructions (Anwar et al., ).
Traditional methods like preference modeling are resource-intensive and struggle with subjective, black-box rewards (Feng et al., ; Pitis, ; Rafailov et al., ; Stiennon et al., ; ).
Additionally, the definition of harmful content varies across contexts (He et al., ; Sorensen et al., ; Santurkar et al., ), complicating the creation of universal harm models.
The usage context further complicates this variability; for instance, discussing medical advice might be harmful in some situations (Wang et al., ) but essential in others, such as in medical chatbots (Xie et al., ).
In this paper, we show CAST can implement behavioral rules like “if input is about hate speech or adult content, then refuse” (Figure [8](#S5.F8)a) or “if input is not about legal advice, then refuse” (Figure [9](#S5.F9)a), allowing for selective modification of responses to specific content without weight optimization.

On a technical level, our primary insight is that different prompts consistently activate distinct patterns in the model’s hidden states during inference (Hu et al., ).
These patterns can be extracted as a steering vector and used as reference points for detecting specific prompt categories or contexts.
This observation allows us to use steering vectors not only as behavior modification mechanisms but also as condition indicators, which we term “condition vectors.”
Our specific contributions are as follows:

- 1)
We expand the activation steering framework by introducing CAST, which adds a new dimension of controllability to existing methods.
- 2)
We show that the utilization of condition vectors in the activation steering process enables fine-grained control over LLM outputs, effectively partitioning the prompt space.
- 3)
We demonstrate the logical composition of condition vectors to create custom refusal conditions.
- 4)
We release a general-purpose activation steering library implementing CAST.

## 2 Background

How do transformers perform inference?
Transformer models, particularly decoder-only variants, perform inference by sequentially processing input tokens through a stack of layers (Radford et al., ; Vaswani et al., ).
The key to understanding their operation lies in how information flows and accumulates through these layers (Lad et al., ; Shai et al., ; Elhage et al., ).
The process begins with converting the prompt into token embeddings, which serve as initial inputs.
Each layer transforms these activations using its internal mechanisms, like learned weights.
Each layer’s output combines processed information with its input, preserving and building upon earlier computations.
As activations flow through the layers, the model constructs increasingly complex representations.
The final layer’s output is used for decoding - predicting the next token via an operation over the model’s vocabulary.
This predicted token is then used for subsequent predictions.

Behavior steering.
One could intervene in any of the abovementioned five steps - weights, decoding, prompt, token embedding, and activations - to alter model behavior (Tamoyan et al., ; Phan et al., ; Chai et al., ; Li et al., ; Han et al., ; Wang et al., ).
For example, one could use role-play prompts to simulate and create AI patients (Louie et al., ).
Or one could use preference optimization methods like direct preference optimization to update weights and steer the LLM towards more empathetic behaviors (Sotolar, ).
Activation steering is a class of methods that intervenes in the information flow within LLMs from layer to layer to alter the model behavior.

Activation steering. An alternative method for influencing the behavior of LLMs, activation steering modifies their internal activations during inference.
This approach typically involves three key steps.
First, a steering vector is extracted, often by computing the difference in activations between examples that exhibit a desired behavior and those that don’t.
Second, during inference, this vector is added to the model’s hidden states at a chosen layer, scaled by a hyperparameter.
Finally, the model completes the generation using these modified activations.
For the case of activation addition (ActAdd) (Turner et al., ), the intervention can be represented mathematically as:

$$ $\mathbf{h}^{{}^{\prime}}\leftarrow\mathbf{h}+\alpha\cdot\mathbf{v}\vspace{1mm}$ $$

where $\mathbf{h}$ is the hidden state at the layer, $\mathbf{v}$ is the steering vector for the layer, and $\alpha$ is a scaling factor.
Stronger scaling can disrupt coherence while weaker scaling may be ineffective (Rimsky et al., ).
In an ideal case where steering vectors are well-extracted, this method allows for predictable LLM behavior steering without altering model weights, enabling applications such as reducing bias (Lu and Rimsky, ; Adila et al., ) or preventing overly confident responses (Rahn et al., ).

Recent research has proposed several methods to improve upon the basic activation addition approach (Wang et al., ; Stickland et al., ; Qiu et al., ; Yin et al., ; Wu et al., ).
These techniques address various limitations of the ActAdd method and collectively fall under the broader category of activation engineering.
In this paper, we propose a vertical expansion by adding the new dimension of *condition*, greatly improving the utility of existing activation steering methods.

## 3 Conditional Activation Steering

### 3.1 Overview

A common limitation of the existing activation steering methods is that one cannot condition the model’s behavior on context, as these methods typically apply modifications uniformly across all inputs regardless of context (He et al., ).
Simple activation steering of a model indiscriminately affects all inputs, rendering the steered model much less useful for its application (Turner et al., ; Cui et al., ; Wen et al., ; Brahman et al., ).
We show that one can induce conditional behavior (Figure [2](#S3.F2)) by leveraging two types of vectors: condition and behavior vectors.

$$ $\mathbf{h}^{{}^{\prime}}\leftarrow\mathbf{h}+f(\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h}))\cdot\alpha\cdot\mathbf{v}\vspace{1mm}$ $$

where $\mathbf{h}$ is the hidden state, $\mathbf{c}$ is the condition vector, $\mathbf{v}$ is the behavior vector, and $\alpha$ is a scaling factor.
The projection of $\mathbf{h}$ onto $\mathbf{c}$ is given by $\text{proj}_{\mathbf{c}}\mathbf{h}=\left(\frac{\mathbf{c}\otimes\mathbf{c}}{\mathbf{c}\cdot\mathbf{c}}\right)\mathbf{h}$.
Intuitively, based on how well aligned the hidden state $\mathbf{h}$ is with the condition vector $\mathbf{c}$, the function $f$ determines whether to apply the behavior vector based on the similarity between the hidden state and its projection using the condition vector.
Throughout the paper, we use cosine similarity, defined as $\text{sim}(\mathbf{h},\mathbf{g})=\frac{\mathbf{h}\cdot\mathbf{g}}{|\mathbf{h}||\mathbf{g}|}$.

Figure: Figure 2: Enabling “targeted” activation steering. Unlike simple refusal activation steering that blocks all prompts, CAST employs a condition vector to selectively steer the model. This approach enables the model to (a) refuse harmful requests while (b) remaining responsive to harmless prompts. Model: Qwen 1.5 Chat 1.8B.

Behavior vector.
We use the term “behavior vector” to refer to what previous activation steering methods call a “steering vector” to emphasize its focus on modifying specific behaviors.
A behavior vector $\mathbf{v}$ is a one-dimensional vector matching the model’s hidden state dimensions that induces specific behaviors.
When added to layer representations during a forward pass with scaling factor $\alpha$, it predictably alters model behavior (e.g., inducing refusal).
In addition to setting the right scaling factor $\alpha$, one can specify to which layers to apply the behavior vector.
While specific implementations vary in the literature, our implementation calculates a different vector $\mathbf{v}_{l}$ for each layer $l$, as behavior representations vary.
Thus, when we mention adding a behavior vector from layers 15-20, we’re referring to adding the corresponding $\mathbf{v}_{15},\mathbf{v}_{16},...,\mathbf{v}_{20}$ to their respective layers.

Condition vector.
A condition vector $\mathbf{c}$ captures a class of instructions to condition on, extracted similarly to behavior vectors and matching hidden state dimensions (e.g., 1x4096 for Llama2, which has a hidden size of 4096).
For instance, a condition vector might capture discrimination or adult content.
It acts as a trigger, determining when to apply the behavior vector based on the model’s current hidden state.
Since we also calculate a different vector $\mathbf{c}_{l}$ to each layer $l$, one can also choose which layer to condition.
When the condition is activated during text generation, the behavior vector is added to all subsequent forward passes.
This allows the model’s behavior to change based on specific conditions in the input or generated text rather than always applying the behavior vector.

Checking if condition was met.
The term $\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h})$ computes the degree to which the condition is met using cosine similarity.
The thresholding function $f$ then determines whether this degree is sufficient to trigger the behavior modification.
Though one would be able to design more complex thresholding functions, we use a simple step function for binary output in this paper:

$$ $f(\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h}))=\begin{cases}1&\text{if }\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h})>\theta\\ 0&\text{otherwise}\end{cases}\vspace{1mm}$ $$

Here, each layer in an LLM might represent the same condition in different directions and $\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h})>\theta$ could be $\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h})<\theta$ depending on the layer.
This binary approach allows for a clear distinction between when the condition is met and when it is not, providing a straightforward mechanism for activating the behavior modification.
We use cosine similarity to check condition based on the directional similarity between the hidden state and its projection using the condition vector rather than magnitude (Hsu et al., ).
In practice, we apply a non-linear transformation $\text{sim}(\mathbf{h},\text{tanh}(\text{proj}_{\mathbf{c}}\mathbf{h}))$ for more predictable behavior.

Multi-conditioning.
As mentioned in Section 1, one could also break down broader alignment goals into smaller, more definitive categories and predictably induce refusal behaviors for each.
For instance, instead of conditioning a model to refuse “harmful” instructions in general, we could create specific conditions for “adult content,” “social stereotypes,” or “false advertising.”
Such multi-conditional behavior can easily be implemented by expanding the thresholding function like:

$$ $f(\cdot)=\begin{cases}1&\text{if }\text{sim}(\mathbf{h},\text{proj}_{adult}\mathbf{h})>\theta_{adult}\text{ or }\text{sim}(\mathbf{h},\text{proj}_{stereotype}\mathbf{h})>\theta_{stereotype}\\ 0&\text{otherwise}\end{cases}\vspace{1mm}$ $$

General expectations
Implementing conditional behaviors in LLMs using CAST generally follows the pipeline: 1. gather contrasting example responses/prompts for desired behavior/condition $\mathcal{D}^{+}$ and other behavior/condition $\mathcal{D}^{-}$, 2. extract behavior/condition vector, 3. find optimal intervention points for behavior/condition vector, 4. steer.
The model itself does not undergo any weight update.

Step 3 represents the most time-intensive part of our process, involving both automated and manual elements.
For the behavior vector, similar to other works in activation steering, we manually search for the appropriate intervention strength and layers.
However, as demonstrated in Appendix [C](#A3), most models represent refusal behavior at similar depths.
For the condition vector, we use a grid search (Appendix [C.2](#A3.SS2)) algorithm that determines the best threshold, layer, and comparison direction ($>$ or $<$).
The majority of our reported experiments are replicable within an hour, with the grid search being the primary time-consuming component.
We share more details below.

### 3.2 Preparing Dataset and Model

As mentioned, contrast datasets are needed to extract behavior or condition vectors.
For the refusal behavior vector, we randomly select 100 instructions from the Alpaca dataset (Taori et al., ) and append them with 100 typical refusal or compliance behavior prefixes as responses, as shown in Figure [3](#S3.F3).
Considering every combination of these creates 10,000 pairs of contrasting data points for $\mathcal{D}^{+}_{\text{refuse}}$ and $\mathcal{D}^{-}_{\text{comply}}$.
We commit to this setup for the refusal behavior vector throughout our research.

Figure: Figure 3: Contrastive data instances. For behavior vectors, we record mean activations at the contrasting suffixes, whereas for condition vectors, we record at the full contrasting prompts.

We explore different condition vectors for our experiments.
In Section 4, we create $\mathcal{D}^{+}$ and $\mathcal{D}^{-}$ using Sorry-Bench (Xie et al., ) and Alpaca.
For Section 5, we use paraphrased Alpaca data.
When additional data were required, we primarily relied on machine generation, including paraphrasing for specific conditions.
We did not apply additional filtering to the train datasets beyond basic quality checks as we found this process generally robust to small data perturbations.
For both setups, the authors manually checked every item in the test set to ensure integrity but did not modify or correct any.
See Appendix [B](#A2) for data generation details and examples.
Lastly, we experiment with models described in Table [1](#S3.T1).

**Table 1: Overview of models used in this study, detailing key architectural features relevant to our research. Models are sourced from industry, academia, and the community and selected based on experimental suitability and the availability of comprehensive documentation. We give additional details on each model in Appendix [D](#A4).**
| Model | Sizes | Layers | Hidden Size | Post-Training | Base | Reference |
| --- | --- | --- | --- | --- | --- | --- |
| Qwen 1.5 Chat | 1.8B, 32B | 24, 63 | 2048, 5120 | SFT + DPO | Qwen 1.5 | Bai et al. () |
| Llama 2 Chat | 13B | 40 | 5120 | SFT + RLHF | Llama 2 | Touvron et al. () |
| Llama 3.1 Inst | 8B | 32 | 4096 | SFT + RLHF | Llama 3.1 | Meta () |
| NeuralDaredevil | 8B | 32 | 4096 | SFT + Merge + DPO | Llama 3 | Labonne () |
| Hermes 2 Pro | 8B | 32 | 4096 | SFT | Llama 3 | Teknium et al. () |
| OLMo SFT | 7B | 32 | 4096 | SFT | OLMo | Groeneveld et al. () |
| Zephyr Beta | 7B | 32 | 4096 | SFT + DPO | Mistral v0.1 | Tunstall et al. () |
| Danube 3 Chat | 4B | 24 | 3840 | SFT | Danube 3 | Pfeiffer et al. () |

### 3.3 Extracting Condition and Behavior Vectors

The extraction of steering vectors begins with a set of contrastive examples - pairs of inputs that exemplify the presence and absence of a target behavior or condition that we built in Section [3.2](#S3.SS2).
These pairs serve as the basis for identifying relevant directions in the model’s hidden state space.
We employ a combination of methods that have been reported to work well for vector extraction.

For a given layer $l\in[L]$, we first compute the hidden states for both positive and negative examples in our contrastive pairs.
Let $\mathbf{H}_{l}^{+}$ and $\mathbf{H}_{l}^{-}$ represent all hidden states $\mathbf{h}_{l}$ for positive $\mathcal{D}^{+}$ and negative $\mathcal{D}^{-}$ examples respectively at layer $l$.
The computation of these hidden states differs between behavior vectors and condition vectors, as illustrated in Figure [3](#S3.F3).
For behavior vectors, we take the average hidden states for suffixes of each example.
For condition vectors, we take the average hidden states for all tokens of each example to capture a more holistic representation of the input.

We then mean-center $\mathbf{H}_{l}^{+}$ and $\mathbf{H}_{l}^{-}$, following the ideas from Tan et al. (); Jorgensen et al. () and apply Principal Component Analysis following Ball et al. (); Adila et al. (); Zou et al. ().
The first principal component resulting from this process becomes our behavior/condition $\mathbf{vector}_{l}$ for layer $l$.
This process is repeated for each specified layer, resulting in a set of layer-specific steering vectors $\{\mathbf{vector}_{l}\mid l\in L\}$.
The extraction of vectors can be expressed as below, where $\text{PCA}(\cdot)$ represents the operation of extracting the first principal component:

$$ $\mathbf{vector}_{l}=\text{PCA}(\mathbf{H}_{l}^{+}-\mathbf{\mu}_{l},\mathbf{H}_{l}^{-}-\mathbf{\mu}_{l})$ $$

The PCA input $(\mathbf{H}_{l}^{+}-\mathbf{\mu}_{l},\mathbf{H}_{l}^{-}-\mathbf{\mu}_{l})$ is a matrix of mean-centered examples, with each row alternating positive ($\mathbf{h}_{1}^{+}-\mathbf{\mu}_{l}$) and negative examples ($\mathbf{h}_{1}^{-}-\mathbf{\mu}_{l}$).
Here, $\mathbf{\mu}_{l}=(\mathbf{H}_{l}^{+}+\mathbf{H}_{l}^{-})/2$ is the mean activation all examples $\mathbf{H}_{l}^{+}$ and $\mathbf{H}_{l}^{-}$.
This centers the data cloud, ensuring the principal components are computed relative to this center rather than being influenced by any overall offset in the data.
The mean-centered positive and negative examples are alternatively concatenated and passed to PCA, which computes the direction of maximum variance.
This direction, representing the most significant distinction between positive and negative examples, becomes our $\mathbf{vector}_{l}$ for layer $l$.

## 4 Conditioning Refusal: Selectively Steering on Harmful Prompts

In this section, we explore the basic use of conditional steering by steering a model to refuse harmful prompts while complying with harmless ones.
Apart from demonstrating that a language model can be conditioned from inside on the fly, we also share some key properties of conditional steering.

Experimental setup.
To obtain our contrast dataset ($\mathcal{D}^{+}$, $\mathcal{D}^{-}$) on the harmful condition, we started by machine-generating 90 harmful prompts for each of the 45 harm categories as identified by Xie et al. ().
We use these 4,050 synthetically generated harmful prompts as our $\mathcal{D}^{+}_{\text{harmful}}$.
For each of these harmful prompts, we randomly sample a benign instruction from the Alpaca dataset to create $\mathcal{D}^{-}_{\text{harmless}}$.
Following the process outlined in Section [3.3](#S3.SS3), we then extract the harmful condition vector $\mathbf{c}_{\text{harmful}}$.
We then use a grid search algorithm to identify the best combination of threshold $\theta$, layer $l$, and comparison direction ($>$ or $<$) that best separates the two classes of training data.
This concept is illustrated in Figure [4](#S4.F4)d, where we perform the condition checking operation at layer 7 and activate the behavior vector $\mathbf{v}_{\text{refusal}}$ when $\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h})$ was smaller than 0.048.

Figure: Figure 4: Conditioning behavior from inside. (a)-(c): T-SNE of prompt embeddings and refusal probability maps for base, activation steered, and conditionally steered models. (d): $\text{sim}(\mathbf{h},\text{proj}_{\mathbf{c}}\mathbf{h})$ across layers 5-7 for $\mathcal{D}^{+}_{\text{harmful}}$ and $\mathcal{D}^{-}_{\text{harmless}}$. Highlighted portions indicate 25th-75th percentiles. Model: Hermes 2 Pro.
Refer to caption: /html/2409.05907/assets/figures/result.png

Result: Activation steering can be used to induce conditional behaviors.
We test the conditional activation steering performance on 500 unseen Alpaca (harmless) and 450 unseen Sorry-Bench (harmful) test sets.
The results are presented in Figure [1](#S1.F1) with a subset of the data in Table [2](#S4.T2).
Across all seven tested models, we observe that conditioning a behavior vector $\mathbf{v}_{\text{refusal}}$ on condition vector $\mathbf{c}_{\text{harmful}}$ selectively increases refusal rates for harmful content while leaving harmless prompt refusal rates largely unchanged.
In contrast, simply adding a behavior vector $\mathbf{v}_{\text{refusal}}$ like standard activation steering increased refusal rates indiscriminately across all prompts.
Figures [4](#S4.F4)a-c demonstrates how the conditioning operation partitions the prompt space.

Figure: Figure 5: Duality and modulation properties. (a)$\rightarrow$(d): Flipping the comparison direction (from $<$ to $>$) intervenes on the exact complement set of inputs. (a)$\leftrightarrow$(b)$\leftrightarrow$(c): one could progressively loosen or tighten the safety guardrail by changing $\theta$. Model: NeuralDaredevil.
Refer to caption: /html/2409.05907/assets/figures/duality.png

Property: Duality.
As seen in Figure [4](#S4.F4)d, this conditioning process is systematic in nature as we can manually choose the point of intervention.
One consequence of this is that conditioning exhibits a dual nature: flipping the comparison direction (from $<$ to $>$ or vice versa) results in intervening on the exact complement of the original set of hidden states that triggered the condition.
This duality enables complementary control over the model’s behavior, allowing one to not only condition the model to refuse harmful prompts but also, if desired, to selectively refuse harmless prompts while complying with harmful ones. See Figure [5](#S4.F5)d.

Property: Modulation.
Our steering approach offers flexible control rather than being uniform across all contexts, with the threshold $\theta$ modulating the required alignment between the input and the harm direction defined in $\mathbf{c}_{\text{harmful}}$.
In Figures [5](#S4.F5)a-c, using the $<$ comparison, lowering $\theta$ narrows the range of hidden states triggering the condition while raising it broadens this range.
This property allows us to adjust the model’s sensitivity to potentially harmful content.
While this offers the potential for finer condition control, we do not explore it further in this study.
Instead, we use threshold values determined by grid search, which maximizes the F1 score to balance false and true refusal rates (Appendix [C.2](#A3.SS2)).

Figure: Table 2: Refusal rate (%) of conditionally steered models vs. reference models. “Discrepancy” shows the difference between harmful and harmless percentages. Arrows indicate a change from the base model. References show how the top safety-aligned models would behave on the same test set.
Refer to caption: /html/2409.05907/assets/figures/linearity.png

Property: Saturation.
Unlike most weight optimization methods, where performance often scales with increased data volume (Das et al., ; Metcalf et al., ; Ansell et al., ), conditional activation steering tends to reach a performance plateau relatively quickly.
As shown in Figure [6](#S4.F6)a, the method’s effectiveness stabilizes after a certain point.
This saturation might be attributed to the fact that conditional steering leverages the model’s existing representations.
Consequently, performance appears more dependent on the model’s inherent capacity to represent certain concepts and how well the chosen data instances represent the target concept rather than on the sheer volume of conditioning data.
Notably, the method also exhibits linear time scaling property (Figure [6](#S4.F6)b).
The condition vector extraction time increases linearly with the number of samples, as this process is primarily determined by the number of inferences the model must make for us to record hidden states.

## 5 Programming Refusal: Logical Composition of Refusal Condition

Moving beyond the general concept of refusing harmfulness, we demonstrate the creation of more fine-grained condition vectors.
We create five example condition vectors from categories - hate speech, legal opinion, sexual context, health consultation, and crime planning - in Liu et al. () to explore these ideas.
Our experiments demonstrate the capacity to (1) selectively modulate refusal behaviors for specific conditions and (2) construct complex refusal conditions through the logical composition of several condition vectors, enabling programmatic control over model behavior.

Figure: Figure 7: Inducing or suppressing refusal from specific categories. Each pie chart represents the model’s refusal rate for six prompt content types. (a): The leftmost chart shows Hermes 2 Pro’s original refusal rates. Subsequent charts demonstrate adding refusal on specific conditions (e.g., $\mathbf{c}_{\text{sex}}\rightarrow+$ means inducing refusal for sexual content). (b): Refusal can also be removed by subtracting the behavior vector $\mathbf{v}_{\text{refusal}}$.
Refer to caption: /html/2409.05907/assets/figures/fine.png

Experimental setup.
We begin by randomly selecting 1,300 base prompts from the Alpaca training set.
Each of these prompts is then paraphrased to incorporate aspects of sexual content $\mathbf{c}_{\text{sex}}$, legal opinions $\mathbf{c}_{\text{legal}}$, hate speech $\mathbf{c}_{\text{hate}}$, crime planning $\mathbf{c}_{\text{crime}}$, or health consultation $\mathbf{c}_{\text{health}}$.
This process results in 1,300 prompts in six categories, including the original benign base Alpaca prompts.
We then split this dataset into 700 prompts per category for training and 500 per category for testing.
To create a conditioning vector $\mathbf{c}$ for a specific category, we use the 700 $\times$ 5 = 3,500 training prompts from the other five categories as our negative examples ($\mathcal{D}^{-}$).
For the positive examples ($\mathcal{D}^{+}$), we use the 700 training prompts from the target category and repeat them five times to balance the dataset.

Application: Inducing or suppressing refusal behavior from specific categories.
We begin by examining our ability to add refusal behavior to specific categories of prompts, starting with a model that exhibits arbitrary refusal behaviors.
Figure [7](#S5.F7)a demonstrates that it is indeed possible to induce refusal behavior when a specific condition is met.
This extends the concepts explored in Section [4](#S4) to more fine-grained categories, showing successful selective refusal.
Furthermore, as shown in Figure [7](#S5.F7)b and consistent with findings from Arditi et al. (), we can also remove refusal behavior from certain classes of prompts.
This is achieved by simply reversing the signs of the behavior vector $\mathbf{v}_{\text{refusal}}$.
Beyond refusal, most inference-time steering techniques can be conditioned using condition vectors as a modulation for various characteristics in language model outputs (Konen et al., ).

Figure: Figure 8: Logical composition of conditions. (a) Effects of combining (OR $\lor$) condition vectors on refusal rates. (b) Complex compositions, including simultaneous removal ($-$) and induction ($+$) of refusal behaviors. (c) Graphical illustration to ease understanding of outcomes under multiple rules: Rule 1 activated (left), no rules met (middle), Rule 2 met (right). Condition layers perform checking; behavior layers apply refusal vectors.
Refer to caption: /html/2409.05907/assets/figures/multi.png

Application: Logical composition of condition vectors.
As introduced in Section [3.1](#S3.SS1), condition vectors can be logically combined to create complex refusal conditions.
For instance, to induce refusal in two categories, such as hate speech and legal opinions, one could implement a rule like if $\mathbf{c}_{\text{hate}}$ or $\mathbf{c}_{\text{legal}}$ then +$\mathbf{v}_{\text{refusal}}$, as illustrated in Figure [8](#S5.F8)a.
This multi-conditioning mechanism can also reinforce existing model refusal conditions, enhancing robustness against harmful prompts.
The second pie chart in Figure [8](#S5.F8)b demonstrates this with LLaMA 3.1 Inst, where we can augment the model’s existing refusal of crime planning and hate speech with additional conditions for legal and health queries while maintaining responsiveness to benign prompts.
Each condition vector $\mathbf{c}$ may have different optimal condition points, as different layers might best separate specific conditions.
Consequently, condition checking might occur at various layers during inference, as shown in Figure [8](#S5.F8)c.
It’s also possible to completely change the original model’s refusal map by simultaneously removing existing refusal directions and inducing new ones (Figure [8](#S5.F8)b) through multiple rules. However, we generally find that this approach can reduce the effectiveness of induced refusal directions, as certain suppressing conditions may conflict with newly induced refusal conditions.

Figure: Figure 9: Constraining responses to one domain. (a) Constraining response to only the target condition by adding refusal to all other categories of instructions using the flipped comparison direction ($\neg$) (see duality property). (b) Constraining response generalizes well to unseen categories of prompts as we are adding refusal to anything that does not satisfy the target condition. (c) Constraining response performance vs. average semantic distance from the target category’s train set to other categories’ test sets. Higher semantic distance correlates with better constraining effectiveness across seen and unseen categories.
Refer to caption: /html/2409.05907/assets/figures/multi-2.png

Application: Constraining model responses to specific domains.
Connecting from our earlier point on the logical composition of condition vectors, we can conditionally steer models to respond only to specific types of prompts.
This approach is particularly useful when the goal is to make a specialized model respond exclusively to specific categories, such as creating a health assistant (Cheong et al., ; Xie et al., ).
Instead of creating conditions for all non-health categories to refuse, we can utilize the duality property discussed in Figure [5](#S4.F5).
We could (1) create a condition vector (e.g., $\mathbf{c}_{\text{health}}$) and (2) flip the comparison direction to add refusal on the exact complement set of inputs (e.g., $\neg\mathbf{c}_{\text{health}}$).
As shown in Figure [9](#S5.F9), this constrains the model to only respond to a category and refuse all others.

We extended our investigation to examine whether our constraining method remains effective for unseen prompt categories.
To this end, we introduced four additional harm categories from Liu et al. () that were not part of our original condition vector training setup: gambling, financial advice, privacy violence, and malware generation.
As illustrated in Figure [9](#S5.F9)b, the effectiveness of domain constraining extends to unseen categories.
This is because our method adds refusal to the complement set of the target category by flipping the comparison direction.
Consequently, it refuses all inputs that do not match the target category’s characteristics, regardless of whether they were seen in training.
However, we observed performance variations across different setups.
For instance, constraining the model to hate speech (if $\neg\mathbf{c}_{\text{hate}}$ then +$\mathbf{v}_{\text{refusal}}$) was more effective in refusing other categories than constraining it to legal opinions (if $\neg\mathbf{c}_{\text{legal}}$ then +$\mathbf{v}_{\text{refusal}}$).
This brings us to our next point.

Analysis: Constraining response to one category works better for more semantically distinct categories.
Figure [9](#S5.F9)c illustrates this relationship, showing a positive correlation between a category’s average semantic distance from others (x-axis) and the effectiveness of constraining to that category, measured by the increase in refusal rate for other categories (y-axis).
Using a sentence transformer model, this semantic distance is calculated as the average cosine distance between the embeddings of the target category’s training prompts and the test prompts of all other categories.
This explains why constraining the model to hate speech is more effective than constraining it to legal opinions when it comes to refusing other categories.
Hate speech, being more semantically distinct from other categories, allows for clearer boundaries and, thus, more effective constraining.

As noted in previous literature on behavior steering, prompting alone fails to provide an effective alternative for several reasons.
Unlike CAST, prompting lacks the ability to forcefully condition the model, offering only weak, coarse-grained control that may paradoxically increase unwanted content (Jang et al., ; Dekoninck et al., ).
Our experiments confirm this, with conditional steering consistently outperforming the prompting baseline (red dotted line) across most categories in Figure [9](#S5.F9)c.
This baseline represents the average performance when the model is simply prompted to comply with the target condition and refuse other conditions without any conditional steering techniques.

## 6 Conclusion

This paper introduces Conditional Activation Steering (CAST), a novel framework for inducing context-dependent behaviors in large language models through principled manipulation of their internal representations. By extending existing activation steering techniques with the introduction of condition vectors, CAST enables fine-grained control over model behavior without the need for fine-tuning or extensive computational resources.

Figure: Figure 10: Key operations introduced in this paper. (a)$\rightarrow$(b): adding a refusal condition. (a)$\rightarrow$(c): Adding more refusal conditions. (a)$\rightarrow$(d): Flipping the condition comparison direction to refusing all other categories except the target. Black lines indicate imaginary, intended points of intervention.
Refer to caption: /html/2409.05907/assets/figures/last.png

Figure [10](#S6.F10) demonstrates key operations that we introduced: the ability to flip condition comparison directions, allowing the model to refuse all categories except a target one, and the capacity to add multiple refusal conditions to induce or remove behaviors. These operations help tailor model behavior to specific requirements.
Beyond this flexibility, the framework offers several advantages.

Firstly, it allows for quick selective refusal of harmful content while maintaining model functionality on benign inputs, addressing a critical challenge in alignment research. Secondly, CAST enables the creation and logical composition of multiple condition vectors, facilitating the implementation of complex behavioral rules. Lastly, it can effectively constrain model responses to specific domains, with its efficacy correlating to the semantic distinctiveness of the target category. By leveraging the model’s existing representations, CAST achieves performance comparable to or exceeding that of models specifically aligned for safety while significantly reducing computational overhead. This efficiency, combined with the ability to modify and compose behavioral rules rapidly, offers significantly enhanced flexibility in adapting model behavior to varying requirements.

## Acknowledgements

This work was done during Bruce W. Lee’s summer internship at IBM Research under the mentorship of Inkit Padhi and Karthikeyan Natesan Ramamurthy. The authors acknowledge Ioana Baldini, Prasanna Sattigeri, Moninder Singh, and Djallel Bouneffouf’s insightful comments during our weekly research discussion sessions. The authors also thank Kush Varshney for thoughtful discussions and organization.

## Appendix A Understanding Conditional Activation Steering

### A.1 The Larger Picture

Model development cycle
The development of language models can be broadly categorized into pre-training and post-training stages (McKinzie et al., ; Tay et al., ).
During pre-training, the focus is on enhancing fundamental capabilities such as knowledge acquisition, reasoning abilities, and coherent language use.
The post-training stage, often referred to as alignment, aims to shape the model’s behavior to meet specific expectations and requirements (Kundu et al., ; Askell et al., ).

Alignment and behavior steering
Within the alignment phase, several key areas emerge, including evaluation, reinforcement learning, and instruction tuning (Nagireddy et al., ; Sudalairaj et al., ; Lee et al., ).
While these topics often overlap, our focus is on behavior steering (Bai et al., ; Cao et al., ).
The term “steering” is deliberately chosen over “control,” implying the current approach of influencing language model behavior rather than exerting direct control.

As model creators, our ultimate goal is to achieve a level of control akin to programming these language models.
To transition from behavior steering to true behavior control, two fundamental criteria must be met: specificity and predictability.
This entails the ability to provide precise instructions or rules to the model, such as “refusing harmful instructions,” “declining irrelevant conversations,” or “avoiding generating adult content,” coupled with a high degree of confidence that the model will consistently adhere to these directives.

Towards programmatic behavior control
Now, instead of merely encouraging models to behave in certain ways through prompting or reinforcement learning, we propose a more forceful and programmatic approach to designing model behaviors.
Our method involves three key steps:

- 1.
Tracking model activations during inference
- 2.
Checking if these activations match specified rule conditions
- 3.
Forcefully intervening in the model to induce desired behavior when conditions are met (which was done in the form of activation steering in this paper)

Unlike straightforward prompting-based approaches, conditional activation steering can be likened to implementing a brain-computer interface for language models, creating a programmable, rule-based system for enforcing model behavior.

Broader implications
This research represents a step towards bringing language models under more precise control, moving closer to predicting and controlling LLM behaviors for various use cases.
In this particular study, we focus on the refusal behavior - specifically, determining and enforcing exactly when a model should refuse instead of complying with a given instruction.

### A.2 Details of Conditional Activation Steering

Origins
Conditional activation steering is an expansion of existing activation steering methods.
Activation steering intervenes in the model’s hidden state during inference, typically by adding “steering vectors”.
This simple operation has shown the potential to reliably induce behaviors like refusal on arbitrary prompts, aligning with the linear representation hypothesis (Park et al., ; Gurnee and Tegmark, ).
While effective, traditional activation steering lacks specificity, causing models to refuse all instructions indiscriminately.
CAST addresses this limitation by introducing a conditional vector $\mathbf{c}$ alongside the behavior vector $\mathbf{v}$.
The application of $\mathbf{v}$ is now conditioned on the similarity between the model’s activation and its projection onto $\mathbf{c}$.

Implementation in the generation process
Language model generation can be viewed as a series of forward passes through the model’s layers for each generated token.
The first full pass through the model typically involves prompt caching.
In CAST, the condition is checked only during this first full pass, as we are conditioning on the prompt (see Figure [11](#A1.F11)).
This approach ensures that the additional condition-checking operation is not repeated for all generated tokens.
However, if the condition is met, the behavior vector is applied in every subsequent forward pass, influencing each generated token.
This application of the behavior vector in every pass at the specified layers follows the convention established in previous activation steering literature.

Figure: Figure 11: The condition check occurs only in the first token’s pass (yellow layer), while behavior modification (blue layers) can be applied in all subsequent passes if the condition is met.
Refer to caption: /html/2409.05907/assets/figures/app_1.png

Extracting behavior and condition vectors
The extraction of behavior and condition vectors follows a consistent process, as illustrated in Figure [12](#A1.F12). This process involves passing contrastive prompts through the model, recording hidden states at each layer, and then applying Principal Component Analysis (PCA) to extract the direction that best separates the two contrastive prompt types. The mathematical representation of this process for each layer is as follows:

$$ $\mathbf{vector}_{l}=\text{PCA}\left(\begin{bmatrix}\mathbf{h}_{1}^{+}-\mathbf{\mu}_{l}\\ \mathbf{h}_{1}^{-}-\mathbf{\mu}_{l}\\ \vdots\\ \mathbf{h}_{n}^{+}-\mathbf{\mu}_{l}\\ \mathbf{h}_{n}^{-}-\mathbf{\mu}_{l}\end{bmatrix}\right)\quad\quad\quad\quad\quad\quad\mathbf{\mu}_{l}=\frac{\mathbf{H}_{l}^{+}+\mathbf{H}_{l}^{-}}{2}$ $$

The key distinction lies in the specific token position at which the activation is recorded, as depicted in Figure [3](#S3.F3). This choice can be adjusted based on the experimental setup. For instance, when using longer contrastive prompts to train the vector, recording the activation of the last token may yield more informative results compared to using the mean activation across all tokens, which could potentially introduce length-related biases.

It is important to note that the current method for extracting and applying refusal behavior may have limitations. Recent studies, such as Arditi et al. () or Rimsky et al. (), have proposed alternative approaches for extracting the behavior directions. While a comprehensive comparison of these methods is beyond the scope of this paper, it represents an important area for future research. The refinement of vector extraction techniques will likely benefit from ongoing collaborative efforts within the research community.

The current state of refusal behavior vector extraction has implications for the evaluation process. Imperfections in the refusal behavior vector may lead to inconsistent refusal induction, even when the condition is correctly activated. Additionally, conditioning and refusal induction performances are interrelated, presenting an opportunity for more detailed analysis in future studies. See Table [3](#A1.T3).

Figure: Figure 12: All vector extractions follow a similar process.
Refer to caption: /html/2409.05907/assets/figures/app_2.png

Adjusting hyperparameters
The effectiveness of conditional activation steering is highly sensitive to the choice of hyperparameters. This sensitivity stems from the fundamental nature of the method, which relies on precise mathematical operations within the model’s hidden states. The primary hyperparameters for conditioning can be conceptualized in a statement:

> Steer when the {best threshold} is {best direction} than the cosine similarity at {best layer}.

This formulation encapsulates three key hyperparameters: (1) Best layer: Determines at which depth of the network the condition checking operation occurs; (2) Best threshold: Defines the boundary for activation; (3) Best direction: Specifies whether the steering activates when the similarity is larger or smaller than the threshold.

**Table 3: Breakdown of Figure [1](#S1.F1).**
| Model | Harmful<br>Refusal | Harmless<br>Refusal |
| --- | --- | --- |
| Qwen 1.5 1.8B | 45.78% | 0.00% |
| + Refusal | 100.00% | 96.40% |
| + Condition | 90.67% | 2.20% |
| Danube 3 Chat | 46.22% | 0.60% |
| + Refusal | 77.11% | 46.00% |
| + Condition | 69.78% | 1.80% |
| OLMo SFT | 53.11% | 5.20% |
| + Refusal | 93.33% | 89.60% |
| + Condition | 86.22% | 6.00% |
| Zephyr Beta | 35.78% | 0.20% |
| + Refusal | 99.33% | 94.80% |
| + Condition | 88.22% | 6.80% |
| Hermes 2 Pro | 19.33% | 1.00% |
| + Refusal | 98.00% | 91.80% |
| + Condition | 83.33% | 2.40% |
| Qwen 1.5 32B | 80.67% | 3.00% |
| + Refusal | 84.44% | 80.60% |
| + Condition | 86.67% | 3.20% |
| NeuralDaredevil | 25.78% | 2.40% |
| + Refusal | 99.11% | 98.60% |
| + Condition | 83.33% | 3.00% |

The layer selection is crucial because different layers capture varying levels of abstraction and linguistic features.
The threshold value and comparison direction determine when the steering should be applied. Conceptually, this can be thought of as setting a “trigger point” in the high-dimensional space of the model’s hidden states (See Figure [7](#S5.F7)). The threshold defines a boundary, while the comparison direction (larger or smaller) determines on which side of this boundary the steering should activate.

These hyperparameters interact in complex ways with the model’s learned representations. For instance, a threshold that is too low might lead to frequent, unnecessary interventions, while one that is too high might fail to activate when needed. Similarly, the choice of layer can significantly impact the granularity and specificity of the condition being checked.
While these conditioning hyperparameters are novel contributions of this approach, they build upon a foundation of existing research on intervention strength and optimal intervention points for behavioral steering in language models (Kong et al., ; ; Zhang et al., ; Scalena et al., ; Tlaie, ).

It is important to note that there isn’t a universally applicable range for the grid search (detailed in Section [C.2](#A3.SS2)) of these hyperparameters, particularly for the threshold values. The cosine similarity values can vary drastically depending on the specific model architecture (more dependent) and the condition being explored (less dependent). For instance, in our experiments, we found that for Hermes 2 Pro, effective threshold values for various conditions fell within the range of 0.0 to 0.1. However, for the Zephyr model, the harmfulness condition operated optimally with threshold values between 0.4 and 0.6. To facilitate this process, our code implementation allows users to easily review the activation history of similarities and determine appropriate search ranges for different models and conditions.

## Appendix B Constrasting Pair Generation Details

To generate the contrasting pair examples used in Section [4](#S4) and Section [5](#S5), we employed the following machine generation processes:

### B.1 Section 4: Harmful vs. Harmless Prompts

For Section 4, we used the Sorry-Bench dataset as a source of harmful prompts:

- 1.
For each harmful prompt in the Sorry-Bench dataset:
(a)
Select two random prompts from other harm categories in the Sorry-Bench dataset.
(b)
Create a prompt for the language model (Mixtral 8x7B) that includes:
•
The target harmful prompt
•
Two example prompts from other harm categories
•
Instructions to generate new questions that violate the target harm category but not the other categories
(c)
Generate 10 new variations of the harmful prompt using the language model.
(d)
Add the generated variations to the original prompt data structure.
- 2.
For harmless prompts, we randomly sampled from the Alpaca dataset without modification.

Pseudocode for the harmful prompt generation:

The prompt used for generation was (create_prompt):

### B.2 Section 5: Fine-grained Harm Categories

For Section 5, we used the Alpaca dataset as a base and generated variations for specific harm categories. The process was:

- 1.
For each prompt in the Alpaca dataset (both train and test splits):
(a)
For each of the five harm categories (sexual content, legal opinion, hate speech, crime planning, health consultation):
•
Create a prompt for the language model (gpt-4o-2024-05-13) that includes:
–
The original Alpaca prompt
–
Instructions to rewrite the prompt to include aspects of the current harm category
–
Rules to ensure the generated prompt maintains a similar structure and explicitly includes the harm category without mentioning it directly
•
Generate a new variation of the prompt using the language model
(b)
Add the generated variations to the original prompt data structure

Pseudocode for the fine-grained category generation:

The prompt used for generation was (create_prompt):

## Appendix C Intervention Points and Grid Search Algorithm

### C.1 Intervention Points Used to Produce Results in This Paper

**Table 4: Intervention points for condition and behavior. For example, $10-15_{interval2}$ is [10, 12, 14].**
| Model | Condition (Layer, (Direction) Threshold $\theta$) | Behavior (Layer, Strength $\alpha$) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Harmful | Sexual | Legal | Hate | Crime | Health | Add Refusal | Remove Refusal |  |
| Qwen 1.5 Chat 1.8B | (8, >0.031) | (7, <0.033) | (6, >0.035) | (11, >0.018) | (7, <0.024) | (8, >0.01) | (10-20, 4) | - |
| Qwen 1.5 Chat 32B | (7, >0.139) | - | - | - | - | - | (36-61, 14) | - |
| LLaMA 3.1 Inst | - | (5, >0.034) | (4, <0.03) | (7, >0.013) | (3, <0.03) | (3, >0.012) | (17-24, 1.7) | (14-28, -1) |
| NeuralDaredevil | (8, <0.065) | - | - | - | - | - | (15-31, 1.5) | - |
| Hermes 2 Pro | (7, <0.048) | (7, >0.037) | (4, <0.021) | (7, >0.029) | (3, <0.024) | (4, >0.014) | (15+17-24, 1.7) | - |
| OLMo SFT | (8, <0.04) | - | - | - | - | - | (12-15+16-28interval2, 4) | - |
| Zephyr Beta | (2, >0.558) | - | - | - | - | - | (10-28interval2, 1.1) | - |
| Danube 3 Chat | (15, >0.05) | - | - | - | - | - | (17-22, 26) | - |

All our experiments are done in our activation steering library, which we open-sourced along with this paper.
The algorithm’s use of these values to steer the model might differ slightly for behavior steering but not for condition steering, as we are implementing conditional steering for the first time.
In general, one could steer, conditional steer, or multi-conditionally steer, as shown in the following code snippets.
These are high-level overviews demonstrating how the numbers from Table [4](#A3.T4) can be applied to replicate our results.
For exact replication, use the replication version of our code.

Steer:

Conditional Steer:

Multi-Conditionally Steer:

### C.2 Best Condition Point (Grid Search) Algorithm

The algorithm searches for the optimal conditioning configuration by evaluating different combinations of layers, thresholds, and comparison directions.

This algorithm iterates through various combinations of layers, thresholds, and comparison directions to find the configuration that yields the highest F1 score in distinguishing between positive and negative examples.
It uses the model’s conditional steering mechanism to compute similarities and then evaluates the effectiveness of different configurations in classifying the input strings.
Based on our experience with CAST, we limit our grid search to the first half of the layers for all models.

## Appendix D Model Descriptions / Dataset Locations

Here, we share all locations of datasets and models used in this paper. We only use publicly available models and datasets that are open-sourced with fairly permissible licenses. All can be found on Huggingface.

- •
sorrybench: sorry-bench/sorry-bench-202406 <b34822276edde97592eda99c0b56d306f8830469>
- •
alpaca: EdBerg/yahmaalpaca-cleaned <6b6ff0e894d31390fa3581bf56f3bafaed9d5e2d>
- •
refusal classifier:
protectai/distilroberta-base-rejection-v1 <65584967c3f22ff7723e5370c65e0e76791e6055>
- •
model: Qwen/Qwen1.5-1.8B-Chat <e482ee3f73c375a627a16fdf66fd0c8279743ca6>
- •
model: Qwen/Qwen1.5-32B-Chat <0997b012af6ddd5465d40465a8415535b2f06cfc>
- •
model: meta-llama/Llama-2-13b-chat-hf <a2cb7a712bb6e5e736ca7f8cd98167f81a0b5bd8>
- •
model: meta-llama/Meta-Llama-3.1-8B-Instruct <8c22764a7e3675c50d4c7c9a4edb474456022b16>
- •
model: mlabonne/NeuralDaredevil-8B-abliterated <348bd440bb061a12552868aeee47207f1a6c0f76>
- •
model: NousResearch/Hermes-2-Pro-Llama-3-8B <8ab73a6800796d84448bc936db9bac5ad9f984ae>
- •
model: allenai/OLMo-7B-SFT-hf <c16aa53f08680e03808a174adcc071ee4f6cf192>
- •
model: HuggingFaceH4/zephyr-7b-beta <b70e0c9a2d9e14bd1e812d3c398e5f313e93b473>
- •
model: h2oai/h2o-danube3-4b-chat <1e5c6fa6620f8bf078958069ab4581cd88e0202c>

### D.1 Community Model Descriptions

NeuralDaredevil-8B: This model is derived from Daredevil-8B, which itself is a merge of multiple Llama 3 8B models using the DARE TIES technique. The process to create NeuralDaredevil-8B involved:

- 1.
Starting with Daredevil-8B, a mega-merged model based on Llama 3 8B.
- 2.
Applying abliteration to remove the refusal behavior to “uncensor” the model. Here, abliteration is an orthogonal refusal removal process following the theory presented in Arditi et al. ().
- 3.
Performing DPO (Direct Preference Optimization) fine-tuning using the mlabonne/orpo-dpo-mix-40k dataset to recover performance lost during abliteration.

This process resulted in an uncensored LLM that maintains most of the original model’s capabilities while removing its built-in censorship mechanisms.

Hermes 2 Pro: Developed by Nous Research, the Hermes 2 Pro we use is based on Llama 3 8B and created through the following process:

- 1.
Starting with the Llama 3 8B base model.
- 2.
Fine-tuning on an updated and cleaned version of the OpenHermes 2.5 Dataset. This dataset is a mix of a few different datasets, including LMSYS-Chat-1M (Zheng et al., ), WizardLM (Xu et al., ), Platypus (Lee et al., ; Lightman et al., ; Lu et al., ; Wang et al., ; Yu et al., ; Hendrycks et al., ; Sawada et al., ), and Unnatural Instructions (Peng et al., ).
- 3.
Further fine-tuning on a newly introduced Function Calling and JSON Mode dataset developed in-house by Nous Research.
- 4.
Incorporating a special system prompt and multi-turn function calling structure with a new chatml role to enhance function calling capabilities.

The resulting model excels at general tasks, conversations, Function Calling, and JSON Structured Outputs, while maintaining the core capabilities of the Llama 3 8B base model.

### D.2 Refusal Classification Model

For refusal detection, we used the protectai/distilroberta-base-rejection-v1 model, a fine-tuned version of distilroberta-base. This model was trained on multiple combined datasets of rejections from different LLMs and normal responses from RLHF datasets, with approximately 10% rejections and 90% normal outputs. The authors mention the use of Do-Not-Answer (Wang et al., ), and I’m Afraid I Can’t Do That (Reuter and Schulze, ) datasets for the training.

Our refusal detection process, implemented in the check_refusal function, involves:

- 1.
Using the protectai/distilroberta-base-rejection-v1 model to classify responses as "NORMAL" or "REJECTION".
- 2.
Performing an additional keyword-based check using a predefined list of common refusal phrases, ["not capable", "cannot provide", "cannot generate", "inappropriate", "language model", "i’m sorry", "an ai", "not able", "am sorry", "don’t know", "sorry", "i cannot", "i can’t", "can’t help", "is it appropriate", "violation", "consequences", "violence", "harm", "harmful", "harassment", "illegal", "experiencing stress", "you’re experiencing", "can’t provide", "i don’t have", "i’m a computer"].
- 3.
Considering a response as a refusal if either:
•
The distilroberta-base-rejection-v1 model classifies it as "REJECTION", or any of the predefined keywords or phrases are present in the response (case-insensitive comparison).

## Appendix E Hardware Specifications

The experiments were performed on a system with the following specifications:

- •
CPU: 2 x AMD EPYC 7763 64-Core Processor
–
Total Cores: 128
–
Threads per Core: 1
–
Base Clock Speed: 2.45 GHz
–
Max Boost Clock: 3.53 GHz
- •
Memory: Not specified (to be added)
- •
GPU: NVIDIA A100-SXM4-80GB
–
Memory: 80 GB HBM2
–
CUDA Cores: 6912

### E.1 Software Environment

The software environment for all experiments consisted of:

- •
Operating System: Linux
- •
CUDA Version: 12.2
- •
NVIDIA Driver Version: 535.54.03
- •
Python Version: 3.10.5
- •
Key Libraries:
–
PyTorch: 2.3.0
–
Transformers: 4.43.3

This configuration remained consistent throughout the research, ensuring that all reported results are comparable and reproducible under the same conditions.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Why was PCA used to compute condition and behavior vectors, as opposed to fitting a classifier (e.g. SVM or logistic regression) which would explicitly fit a separating hyperplane between the classes (and thus moving in the direction of that boundary would directly correspond to steering the behavior in a desired direction)? If I understand correctly, the PCA is computed on the matrix containing both positive and negative activations, and the authors assert the direction of maximal variance (first principal component) is the direction that best distinguishes between the positive and negative examples. It isn't quite clear to me how this can be assumed to be true -- an analysis of, e.g., the distribution of coordinates of the positive and negative examples  when projected along this direction would be very useful to see (i.e. that we see a bimodal distribution of coordinates along this direction, such that moving an activation along the direction does correspond to encouraging the behavior)
* What specifically motivated use of tanh for the similarity transformation as opposed to vanilla cosine similarity?
* Multi-conditioning -- since the conditions are combined into one function, how is the appropriate steering vector determined (as presumably a different steering vector would be desired based on which condition activated)
soundness: 3
strengths: * Novel Approach: CAST represents a unique advancement in activation steering by adding the ability to conditionally refuse specific categories of prompts. This method is valuable in fields like content moderation and personalized assistant behavior where indiscriminate refusal would limit utility.
* Empirical Validation: Extensive experiments demonstrate CAST’s efficacy in refusing harmful prompts without affecting benign responses across multiple LLMs. The results indicate robust behavior modification and reliable model conditioning under various categories.
* The authors note they will release open source code, which will be valuable for other researchers interested in using these methods
summary: This paper presents Conditional Activation Steering (CAST) as a novel framework for enabling selective control over large language model (LLM) responses by dynamically applying refusal behavior to specific prompt categories. Traditional activation steering techniques alter model behaviors across all inputs, but CAST introduces a “condition vector” that enables selective behavior modification based on context. This framework allows for fine-grained, context-dependent refusal responses—such as rejecting hate speech but responding to benign prompts—without requiring weight optimization. The authors claim CAST contributes a valuable tool for alignment, moderation, and domain-specific applications where selective behavior control is essential.
weaknesses: * Figure 6a -- why is conditions triggered the 'success' metric here? Shouldn't it be something like F1 score (for aggregating true positives, false positives, etc. to show performance at each data scale)
* Why is duality of the comparison direction highlighted? Isn't it obvious that flipping the comparison direction and using a threshold of (1-c) yields the same decision boundary but flips the decision? I might be missing something here.
* The main text doesn't seem to explain well how false and true positives for refusals are automatically assessed (e.g. when doing grid search over hypers (like layer and refusal threshold)). A clearer explanation of this would be helpful
* I may have misread, but I think the only quantitative results are in Table 2. 3 models are listed, but table 1 references aroudn 8 models used. Where are results for the other models? In addition, the pie chart visualizations (e.g. Figure 7) are great, but a quantitative summary would be very useful.
* Complexity in Multi-condition Settings: The paper discusses multi-conditioning but does not fully examine potential trade-offs when many conditions are combined, especially when conflicting conditions arise. Further clarification on how to manage or prioritize conditions could enhance the method’s applicability in complex real-world scenarios.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Typos:
- L365 - 372: shouldn't there be 700 + 500 = 1200 prompts per category? Seems like a typo + please ensure consistency throughout the paper on the exact number
soundness: 3
strengths: - Introduces the concept of 'condition vectors' as a way to gate which activation vectors are triggered at each step. 
- Thoroughly demonstrates the use of CAST to handle a variety of new conditional behaviors, such as more precise and robust refusal and topic-based refusal.
- Paper is mostly clearly written with well-formatted figures. The setup seems reproducible and is easy to understand.
- Helps extend the idea of activation steering to make it more robust to multiple scenarios.
In summary, the paper introduces a novel technique (reasonable novelty) and demonstrates its robustness (good quality) with clear presentation (great clarity) towards improving the efficacy and usability of activation steering (reasonable significance).
summary: This paper introduces CAST, a technique to perform conditional activation steering. Following prior work, CAST generates behavior vectors that determine the direction in which to steer models to elicit a specific behavior. Next, CAST generates condition vectors that determine the context in which to apply the behavior vectors. By selectively applying the behavior vectors, CAST is able to steer behavior more precisely and under more complex combinations of rules than prior work.
weaknesses: These suggestions are minor as the paper itself is well done. That said, I think the paper could benefit from:

**Qualitative understanding of the condition vectors.** The main insight of the work seems to be that you can construct condition vectors in a similar way to constructing behavior vectors. But I'm still not sure how robust these condition vectors are. It would be great if the authors could run an experiment studying the generalization of these condition vectors. 
   - For example, you could take the categories described in the paper: health, crime, legal, etc. and ask GPT to generate prompts for them. For each category, there could be two treatments: basic (prompts that use language from the given category and are relevant to the category) and hard (prompts that use language from the given category but are irrelevant to the category, e.g., a prompts that uses legalese without being about legal statements). 
   - Then, for each category's condition vector, how robust is it? What's its accuracy under the two treatments? Can you provide some qualitative failure modes of each condition vector? Do the vectors improve if the model itself improves (and therefore its representations improve). Getting a better qualitative and quantitative understanding of the condition vectors seems useful for understanding the limitations of this technique.

**Improved error analysis.** Are the failures in Figure 1 (and the other experiments) a result of a the condition vector not triggering or the behavior vector not robustly modifying the behavior? Please consider conducing an ablation experiment where the behavior vector is manually applied according to an oracle on the harmfulness and report the F1 score / rate of successful refusal. How does this skyline compare to CAST?

**Missing baselines.** Would it be possible in all of the experiments to have a quantitative comparison to a prompting baseline where the model is told to carefully pay attention to the harmfulness and harmlessness of the response in its refuse (or the relevant category, if its a category). My hunch is that improved prompting would somewhat help for the harmlessness / harmfulness category and fail to help for the more complex combinations of rules. I think this comparison to prompting would help showcase the effectiveness of CAST.

I'm not willing to raise my score higher as I don't think there's evidence that activation steering is a scalable direction for controlling models as it doesn't improve with data (see Figure 6a) and in general seems to be clunky to deploy (see L202 - 208), but I think this paper is high quality and definitely should be accepted. Moreover, I still think the authors should consider incorporating the above suggestions as it would strengthen the paper.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: The paper is generally well written and was pleasant and interesting to read.  The possibility for conditional steering is exciting with many practical implications.

### Minor
The paper could be improved through some careful revisions to the figures and layouts.   
* Figure 1 is presented too early; its full description is on page 6 while it appears at the top of page 2.  Despite being referenced on page 1, the paper would flow better if Figure 1 were closer to page 6.
* All of the T-SNE plots should consider a different color scheme.  Its very difficult to distinguish the Alpaca vs Sorry-bench dots, especially against the background of a similar color.
* Figure 8c, several pieces of text are too small to easily read
* Figure 9, the label "(c)" is not placed in the top left corner like the previous figures.  The markers are difficult to see (e.g. the start marker).  


### Out-of-scope improvements
While the following improvements would substantially increase the value of the paper, the reviewer recognizes that they can be designated to follow up work and may be out-of-scope for the current paper.
* The methods could be tested with models with > 8B parameters
* The methods could be tested against known jailbreaks (e.g. does the conditional vector for harmfulness still activation on a zero-shot or 1-shot jailbreak?).
soundness: 3
strengths: 1. The method of steering LLM conditionally on the context of the prompt is novel and an important contribution towards practical implementations of activation steering.
2. The ability to chain conditionals is an interesting contribution.
3. The paper is relatively thorough in its test of models within a certain class O(8B).
summary: The authors introduce Conditional Activation Steering (CAST) and condition vectors.  They show that refusal behaviors can be invoked conditionally on the context of the prompt allowing for conditional steering.  They test this across several language models up to size 8B and show that their method has fewer false positive refusals on harmless prompts while still maintaining high refusal rates on harmful prompts, demonstrating the effectiveness of the conditional.
weaknesses: 1. All the tested models have less than or equal to 8B parameters.  Testing on larger models would help improve the robustness and confidence in the results
2. (Minor) The harmless/harmful refusals are not tested against enough real-world inputs, like jailbreaks or multi-turn conversations.
3. (Minor) There is no limitations or future work section.

Meta-review: The reviewers note that this is a novel approach in activation steering with a strong empirical validation presented. They noted some weaknesses in model classes evaluated, and in anchoring to specific use cases.

Decision: Accept (Spotlight)