Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation

## Abstract

Abstract Training a language model to be both helpful and harmless requires careful calibration of refusal behaviours:
Models should refuse to follow malicious instructions or give harmful advice (e.g. “how do I kill someone?”), but they should not refuse safe requests, even if they superficially resemble unsafe ones (e.g. “how do I kill a Python process?”).
Avoiding such false refusal , as prior work has shown, is challenging even for highly-capable language models.
In this paper, we propose a simple and surgical method for mitigating false refusal in language models via single vector ablation.
For a given model, we extract a false refusal vector and show that ablating this vector reduces false refusal rate without negatively impacting model safety and general model capabilities.
We also show that our approach can be used for fine-grained calibration of model safety.
Our approach is training-free and model-agnostic, making it useful for mitigating the problem of false refusal in current and future language models.

## 1 Introduction

The most capable Large Language Models (LLMs) today are trained to be helpful to users, answering their questions and following their instructions.
However, LLMs trained only to be helpful will follow even malicious instructions and readily generate toxic or dangerous content (Bianchi et al., ).
Therefore, much prior work has trained models to refuse to comply with unsafe queries (Bai et al., ; Dai et al., ; Zou et al., ).
This creates a tension between model ‘helpfulness’ and ‘harmlessness’, and thus requires careful calibration, which is difficult to achieve:
Recent work by Röttger et al. () shows that even highly capable LLMs struggle with false refusal, where they refuse to comply with clearly safe queries just because they superficially resemble unsafe queries (e.g. “how do I make someone explode with laughter?”).
This makes LLMs less helpful.

Several methods have been proposed to mitigate the problem of false refusal.
However, all have clear limitations.
Training-based methods, on the one hand in reducing false refusal, may be effective, but they are also inflexible, since safety can only be calibrated at training time (Zheng et al., ; Zhang et al., ).
Training-free methods, on the other hand, provide more flexibility but are costly, because they require expensive computation at inference time (Shi et al., ; Cao et al., ).
Furthermore, this type of method appears imprecise so far, because they have unintended negative effects on general model capability (Cao et al., ).

In this paper, we propose a new method for mitigating false refusal in LLMs via single vector ablation, which addresses these key limitations of prior work.
This is done by extracting true refusal vector and false refusal vector with a small sample of harmful and pseudo-harmful queries and applying orthogonalization to tease them apart, which is also supported by further analysis on the intermediate layers of the model.
Compared to other training-free methods, our method is cheap because it does not require additional computation at inference time.
In our experiments, we show that our method is flexible, enabling fine-grained calibration of model safety via partial orthogonalization and adjusting the strength of the refusal removal.
We also show that our method is surgical, reducing false refusal rates without harming general model capability.

## 2 Background

### 2.1 Refusal Vector Extraction

Zou et al. () and Arditi et al. () used difference-in-means (Belrose, ) to extract the refusal vector from the model activation.
They focused on the general refusal behaviour regardless of whether it was a true or false refusal.
The candidate refusal vectors are extracted by calculating the difference between the mean activations when prompted with harmful queries $\mathcal{D}_{\text{harmful}}$ and those when prompted with harmless queries $\mathcal{D}_{\text{harmless}}$ at layer $l$ and token position $i$:

$$ $\mathbf{r}_{i,l}=\mathbf{v}_{i,l}^{\text{harmful}}-\mathbf{v}_{i,l}^{\text{ harmless}}$ (1) $$

$$ $\mathbf{v}_{i,l}^{\text{harmful}}=\frac{1}{\left|\mathcal{D}_{\text{harmful }} ^{\text{(train) }}\right|}\sum_{\mathbf{t}\in\mathcal{D}_{\text{harmful }}^{ \text{(train) }}}\mathbf{x}_{i,l}(\mathbf{t}),\quad\mathbf{v}_{i,l}^{\text{ harmless}}=\frac{1}{\left|\mathcal{D}_{\text{harmless }}^{(\text{train) }} \right|}\sum_{\mathbf{t}\in\mathcal{D}_{\text{harmless }}^{\text{(train) }}} \mathbf{x}_{i,l}(\mathbf{t})$ (2) $$

where $\mathbf{x}_{i,l}(\mathbf{t})$ is the residual stream activation of the transformer at token position $i$ and layer $l$ when prompted with text $t$.
The diff-in-means vectors across all layers at post instruction token positions, such as the 
 for Llama2 models, are collected as candidates for the final refusal vector.
To find the diff-in-means vector that is the most effective in removing the refusal behaviour, the candidate vectors are ranked based on a drop in refusal score (as given in equation 3) after ablating the direction of the refusal vector from the model’s activation stream.
The refusal score measures the token probability difference between the refusal-related tokens $\mathcal{R}$ such as ‘Sorry’, ‘I’, and the non-refusal-related tokens $\mathcal{V}\backslash\mathcal{R}$, at the first token position in the model’s response:

$$ $\textit{Refusal Score}=\log\left(\sum_{t\in\mathcal{R}}p_{t}\right)-\log\left( \sum_{t\in\mathcal{V}\backslash\mathcal{R}}p_{t}\right)$ (3) $$

A similar approach can also be applied to find the vector that is most effective for enhancing the refusal after adding it to the activation.

### 2.2 Vector Ablation and Addition

The selected refusal vector $\mathbf{\hat{r}}$ can be used to remove the refusal behaviour by ablating it from the residual stream.
This is done by first projecting the residual stream activation onto the direction of the refusal vector and then removing this projection from the activation:

$$ $\mathbf{x}^{\prime}\leftarrow\mathbf{x}-\mathbf{\hat{r}}\mathbf{\hat{r}}^{\top }\mathbf{x}$ (4) $$

This operation is done across all layers and token positions to remove the refusal behaviour from the model effectively.
The model can be made to refuse more by adding the refusal vector to the activations at all token positions of a certain layer $l$:

$$ $\mathbf{x}^{\prime}_{l}\leftarrow\mathbf{x}_{l}+\alpha\mathbf{\hat{r}}_{l}$ (5) $$

where $\mathbf{\hat{r}}_{l}$ is selected and applied to the activation $\mathbf{x}_{l}$ from the same layer.
$\alpha\in[0,1]$ controls the strength of the addition operation.
Arditi et al. () show that it is sufficient to do vector addition at a single layer to make the model to refuse more in contrast to the removal operation where the ablation is applied at all layer activations. Both operations can be directly applied to the model weights, without causing additional computation at inference time.

## 3 Mitigating False Refusal via Single Vector Ablation

We aim to reduce unwanted false refusal behaviour of language models with minimal effect on true refusal behaviour and performance on general tasks.
This requires us to find false refusal related features that have minimal overlap with those associated with true refusal and general tasks.
We first show that simply using the difference-in-means technique cannot successfully find false refusal features with minimal overlap with other features.
To disentangle the features, our key suggestion is to orthogonalize the false refusal and true refusal vectors to avoid harming the true refusal ability when ablating the orthogonalized false refusal vector.

### 3.1 Extracting A False Refusal Vector

Similar to extracting the true refusal vector in equation 1, we extract the refusal vector for false refusal prompts by replacing the harmful queries $\mathcal{D}_{\text{harmful}}$ with pseudo-harmful queries $\mathcal{D}_{\text{pseduo-harmful}}$ (e.g. “how do I kill a Python process?”), which induce false refusal in the model:

$$ $\mathbf{w}_{i,l}=\mathbf{v}_{i,l}^{\text{pseudo-harmful}}-\mathbf{v}_{i,l}^{ \text{harmless}}$ (6) $$

$$ $\mathbf{v}_{i,l}^{\text{pseudo-harmful}}=\frac{1}{\left|\mathcal{D}_{\text{ pseudo-harmful }}^{\text{(train) }}\right|}\sum_{\mathbf{t}\in\mathcal{D}_{ \text{harmful }}^{\text{(train) }}}\mathbf{x}_{i,l}(\mathbf{t}),$ (7) $$

We use the refusal score as a filter to select samples that have a refusal score larger than zero. We collected 128 pseudo-harmful samples for extracting the vector which was shown to be effective in Arditi et al. ().
Then, we select the most effective vector $\mathbf{\hat{w}}$ by validating the refusal score (introduced in Eq. 3) drop on the validation set consisting of 32 pseudo-harmful samples.
More detailed dataset usage for vector construction is described in Section 4.
However, as shown in Table 1, after ablating the false refusal vector from the activation stream, we see an increase in compliance rate for *both* the harmful and harmless data, similar to ablating the true refusal vector $\mathbf{\hat{r}}$.
In our preliminary experiments, this issue was validated with different pseudo-harmful datasets such as OR-Bench-Hard (Cui et al., ), XSTest(Röttger et al., ) and OKTest (Shi et al., ).
This means that ablating the raw diff-in-means vector is insufficient, and most importantly, the true refusal vector $\mathbf{\hat{v}}$ and false refusal vector $\mathbf{\hat{w}}$ are not independent of each other.
To tease the two vectors apart, we propose to apply orthogonalization between the candidate false refusal vectors $\mathbf{w}_{i,l}$ and the candidate true refusal vector $\mathbf{v}_{i,j}$, resulting in orthogonalized false refusal vector $\mathbf{w}^{\prime}_{i,j}$.
Then, we select the most effective one $\mathbf{\hat{w}^{\prime}}$ as we described in the procedure above.
As shown in Table 1, after ablating $\mathbf{\hat{w}}^{\prime}$, the model maintains its low compliance rate on harmful data, while the compliance on harmless data increases substantially compared to the original model, showing the effectiveness of the orthogonalization operation for teasing the vectors apart.

Table: Table 1: Extracting a diff-in-means refusal vector is not enough: Ablating (-) any of the two diff-in-means vectors $\mathbf{\hat{r}}$ or $\mathbf{\hat{w}}$ will remove the refusal behaviour, regardless of whether a harmful or harmless prompt is given. Instead, ablating orthogonalized false refusal vector $\mathbf{\hat{w}}^{\prime}$ successfully keeps the model’s low compliance rate on harmful data, while the compliance on harmless data increases substantially compared to the original model. CR: Compliance Rate.

### 3.2 Partial Orthogonalization

The orthogonalization operation so far has the benefit to disentangle the true refusal and false refusal vectors.
It essentially sets a boundary to ensure any feature in the prompts related to
harmful data (used for extracting the refusal vector) should trigger a refusal response.
However, this will make the model overly conservative when it comes to ambiguous examples, such as a request like “how to cut off the head of a fish”, which a conservative model will suggest against as it “can be harmful to fish’s health and affect the flavour”, while a less conservative model will directly give you the instruction.
To address this, we show that this boundary can be adjusted by doing partial orthogonalization by introducing a coefficient $\lambda$ into the subtraction operation:

$$ $\mathbf{w}^{\prime}_{i,l}\leftarrow\mathbf{w}_{i,l}-\lambda\mathbf{v}_{i,l} \mathbf{v}^{\top}_{i,l}\mathbf{w}_{i,l}$ (8) $$

This provides us with the added flexibility to adjust the refusal level of the model.
By lowering the $\lambda$ coefficient, the false refusal vector is less modified and the ablation will have a stronger impact on the false refusal behaviour of the model.
This enables us to granularly control the model’s sensitivity to ambiguous samples, which we will discuss in Section 5.2.

## 4 Experimental Setup

##### Models

We use four chat-tuned LLMs which have been trained to refuse harmful queries: Gemma-7B-It (Team et al., ) , Llama2-7B-Chat, Llama2-13B-Chat (Touvron et al., ) and Llama3-8B-Inst (Llama Team, ).
We use greedy decoding for text generation.

##### Datasets for Vector Extraction

For refusal vector extraction, we use datasets from three different categories: harmful data, harmless data and pseudo-harmful data, which is harmless data but can easily induce the refusal behaviour of a model.
We adopt the harmful dataset $D_{\text{harmful}}$ constructed by Arditi et al. () which are harmful instructions drawn from Advbench (Zou et al., ), MaliciousInstruct (Huang et al., ), and TDC2023 (Mazeika et al., ; ).
The harmless data $D_{\text{harmless}}$ are instructions sampled from Alpaca (Taori et al., ).
For false refusal vector construction, we use the samples from OR-Bench-Hard (Cui et al., ) which is a challenging false refusal test for state-of-the-art LLMs.
The reason we chose this pseudo-harmful dataset over the others is that this is the most challenging one for LLMs.
This leads to a more effective vector that better controls the refusal behaviour, compared to the other pseudo-harmful datasets.
In our preliminary experiment, using a challenging pseudo-harmful dataset also has the least impact on the true refusal performance when ablating the orthogonalized false refusal vector.
We filter out the samples with negative refusal scores from the harmful and pseudo-harmful data to ensure activations are refusal-related.
We also filter out vectors which lead to a KL divergence change larger than $0.2$ in first token probabilities.
Then we randomly select 128 samples from the three kinds of data for extracting the vectors, resulting in $D_{\text{harmful}}^{\text{train}}$, $D_{\text{harmful}}^{\text{train}}$ and $D_{\text{harmful}}^{\text{train}}$.
Then they are evaluated on $D_{\text{harmful}}^{\text{val}}$, $D_{\text{pseudo-harmful}}^{\text{val}}$ with a size of 32, to select the most effective true refusal and false refusal vectors.

##### Evaluation

We evaluate model performance on three aspects: safety (i.e. true refusal), false refusal, and general capability (accuracy, perplexity).
The safety evaluation is based on two datasets: $D_{\text{harmful}}^{\text{test}}$ (Harmful) consisting of 128 samples sampled from the rest of $D_{\text{harmful}}$,
and JailbreakBench (JBB) (Chao et al., ) composed of 100 harmful queries serving as out of distribution test.
For the false refusal tests, we use one in-distribution dataset $D_{\text{pseudo harmful}}^{\text{test}}$ (ORB-H) and two out-of-distribution datasets: OKTest (Shi et al., ) and XSTest (Röttger et al., ).
For XSTest, we use the safe subset which is used for testing false refusal, and we select 66 samples with high refusal scores based on Llama2-7B-Chat, to have a harder test dataset, which we call XSTest-Safe(H).
We adopt the Evaluation Harness framework (Gao et al., ) for general capability evaluation, including MMLU (5-shot) (Hendrycks et al., ), ARC-C(Clark et al., ) and Wikitext (Merity et al., ).
Following previous work on evaluating compliance rate (Liu et al., ; Zou et al., ; Xu et al., ; Robey et al., ; Shah et al., ; Lermen et al., ; Röttger et al., ; Arditi et al., ), we use string matching to classify any responses starting with ‘I’m sorry’, ‘I cannot’, or variations thereof, as a refusal.
Furthermore, for our main experiments in Table 2 we also apply LlamaGuard-3-8B (Llama Team, ) to classify if the model fully complies with generating harmful content and only consider such cases as full compliance on harmful queries. See Appendix B.2 for more details on model response evaluation.

## 5 Results

### 5.1 Mitigating False Refusal

Table: Table 2: Compliance rate (CR) on safety and false refusal datasets, as well as general model performance on standard benchmarks. With vector ablation, the CR on false refusal datasets increases across all the models, while safety and general model performance remain unchanged.

Table 2 shows the results of ablating the orthogonalized false refusal vector $\mathbf{\hat{w}}^{\prime}$ from the model activations.
Additionally, to show the difficulty of balancing the trade-off between ‘harmless’ and ‘helpfulness’ queries, we also report the performance of the model when adding the default system prompt of Llama2 models.
As shown in the Table 2, adding safety-related prompts improves safety performance in general but will lead to serious false refusal problems and performance degradation on general tasks, such as the reasoning task of ARC-C.

Compared to the original models, the models achieve a higher compliance rate on pseudo-harmful data after the ablation operation, with minimal effect on the true refusal behaviour and the general capability.
Among the three pseudo-harmful test data sets, the original models show significant false refusal on the ORB-H, with the lowest compliance rate of $5.5\%$ for Llama2-13b-Chat, which achieves the compliance rate of $36.7\%$ after ablation.
Note that we only use ORB-H data to extract the vector, the model can still be improved on out-of-distribution tests with XSTest-S(H) and OKTest.
This shows the generalization of the false refusal vector we extract.
On general model capability evaluation, the ablation shows minimal effect resulting in an absolute value change within 1.0 across different tasks and models.
Fig 1 illustrates some generation examples of the ablated Llama2-7b-Chat when prompted with harmful and pseudo-harmful queries.
Before the vector ablation, the model behaves safely when prompted with a malicious query for cracking cyber security, but it is triggered by the kill keywords when asked about “how to kill the lights”.
After the vector ablation intervention, the model starts answering the harmless queries, while keeping refusing to answer harmful queries.

Figure: Figure 1: Response examples of Llama2-7b-Chat on harmful and pseudo-harmful queries. Our method removes false refusal while keeping true refusal.

In Table 3, we compare our method to a prior method SCAN (An et al., ), which utilizes a classifier to distinguish the harmful and pseudo-harmful queries.
Results of SCAN are directly taken from the original paper.
Same as SCAN, we also conduct human evaluation to measure the compliance rate.
Our method achieves a higher accuracy on MMLU with minimal perplexity drop after intervention.
We also achieve a lower compliance rate on XSTest-Unsafe due to our surgical approach, which makes our approach a safer intervention without breaking the safety guard.
As vector ablation can be directly applied to the projection weight matrices, the resulting model requires no additional memory usage or inference time.
SCAN shows a higher compliance rate on XSTest-Safe, which is however achieved by sacrificing performance on general tasks and computation cost.
Our method mitigates the false refusal while keeping its capability and efficiency.

Table: Table 3: Performance of Llama2-7B-Chat after intervention. Our method is surgical and cost-efficient without additional memory and inference time requirement.

### 5.2 Fine-grained control with partial orthogonalization

The orthogonalization operation completely removes the component along the direction of the candidate true refusal vectors $\mathbf{v}_{i,l}$ from the candidate false refusal vectors $\mathbf{w}_{i,l}$.
This operation maximally keeps the candidate true refusal vectors after the ablation, which makes the false refusal vector “softer” when removing the false refusal behaviour.
Any feature in the pseudo-harmful queries associated with common features in the harmful queries will be regarded as a violation and will induce refusal, such as “how to cut off the heads of the fish” which is related to the violence feature, as we show in Fig 2.
It is worth noting that there is no clear cut between what constitutes safe and unsafe queries, and no standard on how openly a model should answer.
Therefore, the added flexibility of adapting the model to different refusal standards for different usage scenarios is important.

Figure: Figure 2: Response of Llama2-7b-Chat to a XSTest-Safe sample under different $\lambda$ values for orthogonalization. The response openness increases as we lower the $\lambda$ value. The lower the $\lambda$ is, the less sensitive the model tends to answer the questions. The sensitivity level can be adjusted and calibrated by the user.

Through partial orthogonalization, we can control the distance between the candidate false refusal and true refusal vectors by controlling the coefficient $\lambda$, as introduced in Eq. 8.
By decreasing the $\lambda$, the original false refusal vector is less modified by the orthogonalization, which leads to a stronger refusal removal when ablated.
In Figure 2, as we decrease the $\lambda$ value the openness of the model response increases.
When we set the $\lambda$ to $1$, the model refuses to give the instruction because it regards “cutting off the heads of fish” as not safe and not hygienic.
As we lower the $\lambda$ to $0.8$, the model starts to give the instruction but cautions against it as it is harmful to the health of the fish and affects the flavor and texture.
With a lower $\lambda$ value of $0.6$, the model is fully open to give the instructions without any resistance.
Therefore, $\lambda$ enables us to control the safety sensitivity of the model in a very fine-grained way.

Figure: Figure 3: MMLU accuracy and compliance rate (CR) to pseudo-harmful (OR, XSTest) and harmful data. Changing the $\lambda$ value can adjust the sensitivity to safety-related questions. Lowering $\lambda$ can make the model less sensitive and more open to answering questions. The model’s general capability is unaffected since we adopt a surgical approach by only selecting vectors that have minimal effect on the output distribution.
Refer to caption: x1.png

To understand the impact of the $\lambda$ value comprehensively, we plot in Figure 3 the MMLU performance and the compliance rate on OR-Bench-Hard (OR CR), XSTest-Safe Hard (XSTest CR) and harmful data $D_{\text{harm}}$ (Harmful CR) across the four models we evaluated.
As shown in Figure 3, the compliance rate on both harmful and harmless data increases as the $\lambda$ decreases.
This shows the ability to granularly adjust the boundary between safe and unsafe concepts by adjusting $\lambda$.
For Gemma-7b-It, Llama-3-8b-Ins and Llama2-7B-Chat model, the harmful query compliance rate increases faster than the model of Llama-2-13b-Chat.
This means the false refusal vectors are closer to the true refusal vectors in these two models, which affects more the true refusal behaviour after the ablation.
In comparison, the compliance rate on harmful data remains low when the $\lambda$ decreases for Llama2-13b-chat, while the compliance rate on pseudo-harmful data increases substantially.
This indicates that false refusal and true refusal vectors we extracted from Llama2-13b-chat are well separated and disentangled in the beginning before the orthogonalization.
Therefore, changing the $\lambda$ values has less impact on the true refusal.

The compliance rate curves give us an idea of how well the false refusal and true refusal are separated for the model and how to calibrate it.
For models such as Gemma-7b-It, Llama2-7B-Chat and Llama-3-8b-Ins, staying in the range between $0.8$ and $1.0$ keeps the model relatively safe while lowering the $\lambda$ makes the model substantially more helpful, which proves a good trade-off between ‘harmlessness’ and ‘helpfulness’.
As for Llama2-13-chat, it is safer to further lower the $\lambda$ to maximize the helpfulness of the model.
One possible reason for such clear separation between false refusal and true refusal vectors is that Llama2-13-chat has undergone heavy safety finetuning which leads to the lowest compliance rate on pseudo-harmful data, reaching $5.5\%$ on ORB-H.
This sets the boundary between a safe and unsafe query to an extreme level.
Therefore, lowering the threshold for refusal will first relieve the pseudo-harmful ones before affecting the refusal to harmful queries which have stronger harmful-related features.

To understand the $\lambda$’s impact on general model capability, we also evaluate the model accuracy on MMLU in a 5-shot setting.
As shown in Fig. 3, the MMLU accuracy is not affected by the $\lambda$ value across different models.
This is achieved by filtering the candidate refusal vectors that have largely shifted the first token probability distribution before the ablation.

### 5.3 Refusal Vector Distribution

To better understand how the candidate true refusal and false refusal vectors are distributed in the transformer layers, we plot the refusal score changes across the token positions and layers when ablating the candidate refusal vectors in Figure 4.
The y-axis is the relative token positions which are the positions of the post-instruction tokens such as 
.
The top row shows the refusal score changes when ablating the candidate true refusal vectors.
From the second row, we plot the refusal score changes for the candidate false refusal vectors with increasing $\lambda$ value.
Compared with the true refusal vector distribution, the false refusal vectors generally have a lower refusal score change when ablated.
This is because the false refusal vectors are extracted from pseudo-harmful data, which is less likely to induce refusal compared to the true refusal vectors.
We also see a position overlap between the two kinds of vectors, as shown in the first two rows in the figure.
This indicates that the false and true refusal is activated at similar positions and simply removing one could affect the other one, which is shown in our earlier experiment in Table 1.
As we increase the $\lambda$, the overall refusal score change is decreased, thus resulting in a weaker refusal removal effect.
Furthermore, the distribution overlap between the candidate true refusal and false refusal vectors shrinks with increasing $\lambda$.
For example, the most effective candidate true refusal vectors of Llama-3-8B-Ins are concentrated at layer $13$, while the most effective candidate false refusal vectors are mostly at layer $24$ as the $\lambda$ goes from 0 0 to $0.6$.
Since we only select the most effective candidate refusal vector for the ablation, the final false refusal vector will be selected at different layer and token positions than the true refusal vector as we increase the $\lambda$, avoiding harming the true refusal behaviour.

Figure: Figure 4: Refusal score changes when ablating the refusal vector extracted at certain layers and token positions. By increasing the value of $\lambda$, the refusal vectors have less impact on the model refusal behaviour.
Refer to caption: x2.png

### 5.4 Does Adding True Refusal Vector Help?

Arditi et al. () show that the model can be made to refuse more by adding the refusal vector to the activation stream.
It is potentially beneficial to apply both the false refusal vector ablation and true refusal vector addition to get a more useful model without harming the safety guard.
To investigate the effectiveness of this approach, we evaluate the performance of Llama-2-7b-chat on harmful data, pseudo-harmful data and general tasks under each combination of the adding coefficient $\alpha$ and orthogonalization coefficient $\lambda$. Results are given in Figure 8. We refer readers to the Appendix C for results of Gemma-7b model.
The baseline value below each plot shows the original model performance before the intervention.
As shown in Figure 8, adding the true refusal vector can indeed make the model safer by inducing the refusal behaviour.
In the region where $\alpha\geq 0.2$ and $\lambda\geq 0.6$ as highlighted in the figure, the model behaves relatively safe on the Jailbreakbench with a low compliance rate compared to the original model (baseline). This also leads to a low compliance rate on pseudo-harmful data such as OR-Bench-Hard.
However, we see a degradation of general capacity measured by the accuracy on ARC-C, MMLU and perplexity on Wikitext.
Therefore, the ablation operation is a more surgical approach than the addition, which was used by SCAN (An et al., ), and indeed leads to a general performance degradation.

Figure: Figure 5: Performance of Llama2-7b-chat under different combination of $\alpha$ and $\lambda$. Highlighted area is where the modified model behaves relatively safe on Jailbreakbench compared to the original model (baseline). Vector addition improves the model safety by sacrificing the performance on ARC-C, MMLU and Wikitext.
Refer to caption: x5.png

However, the performance on TruthfulQA (Lin et al., ) has increased as the $\alpha$ increases. The BLUE score reaches $64.6$ under the setting of $\alpha=0.8,\lambda=0.8$, compared to the baseline value of $45.3$. As we inspect the model responses, adding the refusal vector tends to make the model more conservative or refusing to answer, which is favoured by the TruthfulQA questions.
Based on above observation, it is important to pay attention to the general performance degradation when adding true refusal vector to make the model safer since it is not as surgical as the ablation operation.

## 6 Related Work

### 6.1 Language Model Safety

LLM safety research aims to prevent LLMs from producing content that may harm individuals and society.
The detection and elimination of undesirable attributes such as hate speech from LLM-generated texts has been extensively studied (Xu et al., ; Röttger et al., ; Sun et al., ; Vidgen et al., ).
Recently, researchers are paying more attention to preventing LLMs from responding to malicious queries through alignment methods such as supervised fine-tuning (Bianchi et al., ) and RLHF (Bai et al., ; ).
In addition, efforts have been made to evaluate the safety of LLMs, ranging from near-term risks (Lin et al., ; Wang et al., ; Xie et al., ) to longer-term catastrophic risk potential (Hendrycks et al., ; Phuong et al., ).

### 6.2 False Refusal in Language Models

##### Testing for False Refusal

The first test suite explicitly designed for evaluating false refusal (or “exaggerated safety”) in LLMs was XSTest, introduced by Röttger et al. ().
XSTest consists of 250 hand-written safe prompts across ten prompt types, as well as 200 contrasting unsafe prompts.
Subsequent work has expanded on XSTest’s scope, using LLMs to generate larger sets of safe test prompts.
Specifically, Cui et al. () create OR-Bench as a collection of 80k “seemingly toxic” prompts across ten rejection categories.
Similarly, An et al. () create PHTest, which consists of 3,260 “pseudo-harmful” prompts.
Shi et al. () create OKTest, comprising only 350 safe questions.
Chehbouni et al. () create a more specialised templated dataset for testing sociodemographic biases in false refusal.

##### Mitigating False Refusal

False refusal mitigation methods can be generally categorised into training-free and training-based methods.
Training-free methods are normally adaptive and query-specific, which leads to computational overhead during inference.
Self-CD (Shi et al., ) applied contrastive decoding by inferencing twice on the same query with and without the system prompt.
As a contemporary work, SCAN (Cao et al., ) constructed a classifier to adaptively decide whether the model should refuse a certain query, which is controlled by activation steering.
The additional classifier requires additional memory usage and computation at the inference time, and the activation steering through subtraction is non-surgical.
Training-based methods including Zheng et al. () and Zhang et al. () require training samples to calibrate the model and lack the flexibility for post-training adjusting for personalization.
Our approach is training-free and prompt-agnostic, meaning the intervention can be carried out directly to the model weights and applied to any queries which leads to no additional inference cost.

### 6.3 Representation Editing

Representation editing involves steering the model behaviour by influencing the hidden representation of the model.
Recent work has successfully demonstrated its success in controlling the truthfulness (Li et al., ) and sentiment (Rimsky et al., ; Turner et al., ).
Zou et al. () proposed various techniques for finding representations of high-level concepts such as honesty and emotions.
Arditi et al. () focused on the refusal behaviour of the model and extracted a refusal direction from the activation stream of the transformer model.
The resulting direction controls the general refusal behaviour regardless of whether it is a true or false one.
Our work takes a more surgical and fine-grained approach to mitigate the false refusal without harming the other.

## 7 Conclusion

In this work, we propose a surgical approach to mitigate the false refusal in language models via ablating a single vector from the activation stream of the transformer model.
This is done by ablating a false refusal vector extracted from the mean model activation difference when prompting the model with pseudo-harmful and harmless data.
To avoid compromising the safety guard against harmful queries, we apply orthogonalization to remove the projection of the false refusal vector onto the true refusal vector.
We also show that fine-grained calibration of the safety guard can be achieved by partial orthogonalization and adjusting the strength of the projection removal.
This enables more flexibility for adjusting the response openness to ambiguous and sensitive queries that are up to the user’s judgment.
The surgical characteristic of the ablation operation keeps the modified model’s original general capabilities and requires no additional inference cost via direct model weight modification.

##### Limitations

We show that the false refusal vector can be extracted using a small sample of pseudo-harmful and harmless queries.
However, the quality of the vector is not optimal as we mainly focused on using pseudo-harmful samples from OR-Bench-Hard (Cui et al., ), which showed good performance at mitigating false refusal.
A better data curation strategy and sampling technique could further improve the effectiveness of the vector, which we leave for future work.
To select the refusal vector, we adopt the refusal score metrics from previous work (Arditi et al., ) which looks at the refusal-related tokens at the first token position.
Such evaluation metrics can be further improved, such as considering sequence probabilities, for better refusal vector selection.

#### Acknowledgments

We thank the members of MaiNLP for their constructive feedback.
XW, CH and BP are supported by ERC Consolidator Grant DIALECT 101043235.
CH is also supported by the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.
PR is a member of the Data and Marketing Insights research unit of the Bocconi Institute for Data Science and Analysis, and is supported by a MUR FARE 2020 initiative under grant agreement Prot. R20YSMBZ8S (INDOMITA).

## Appendix A System Prompt

For the system prompt experiment conducted in Table 2, we utilize the default Llama2 system prompt as follows:

Figure: Figure 6: The system prompt used for all the models in our system prompt experiment.

## Appendix B Evaluation on Safety and Refusal Output

### B.1 String Matching

Following previous work on evaluating compliance rate (Liu et al., ; Zou et al., ; Xu et al., ; Robey et al., ; Shah et al., ; Lermen et al., ; Röttger et al., ; Arditi et al., ), we utilize string matching to judge whether the model refuses on harmful and pseudo-harmful datasets.
The keywords are shown below in Fig 7.

Figure: Figure 7: List of refusal keywords used for judging refusal behavior.

### B.2 Human and LLM Evaluation

Although the models tend to refuse with these keywords, we manually checked part of the result. We found out that while string matching could detect most of the refusal cases on pseudo-harmful datasets, there is a minor gap between string matching and human annotation results on harmful datasets.
As such, we utilized Llama-Guard-3-8B (Llama Team, ) for safety evaluation in our main experiment in Table 2, 3 and further checked the result with two human annotators.
We use string matching for other experiments for efficiency and refusal-changing trend analysis.
For general refusal trend analysis, string matching shows similar results as LlamaGaurd in our preliminary experiments.

## Appendix C Refusal Vector Addition

Figure 8 shows the results of gemma-7b-it under different combination of $\alpha$ and $\lambda$.
We see a similar result as shown in Section 5.3, where the general model capacity degrades as we increase the $\alpha$ value.
The performance on TruthfulQA can be drastically improved to $90.0$ by setting the $\alpha$ and $\lambda$ as $0.8$, compared to the baseline value of $56.5$.
However, this will greatly sacrifice the usefulness of the model as shown by the performance on tasks including OR Bench, ARC-C and Wikitext.

Figure: Figure 8: Performance of Gemma-7b-it under different combinations of $\alpha$ and $\lambda$. Adding a refusal vector makes the model safer and more conservative by sacrificing the general capabilities on tasks such as Wikitext PPL and ARC-C.
Refer to caption: x6.png

## Appendix D Fine-grained Control on Response Openness

Warning: Content below contains examples of harmful language.

This section provides further details on how varying the orthogonalization parameter $\lambda$ allows fine-grained control over the openness of model responses to potentially sensitive or harmful prompts, with examples in Fig 9 to 12.
As $\lambda$ decreases from 1 (full orthogonalization) to 0 (no orthogonalization), the responses tend to become more direct in addressing the prompt’s request.

For pseudo-harmful datasets, like the metaphorical “stealing someone’s heart” in Fig 9, responses range from very literal interpretations at $\lambda$=1 to more nuanced, metaphorical explanations at lower $\lambda$ values. This demonstrates how the model can be tuned to provide more contextually appropriate responses while still maintaining ethical boundaries.

For harmful prompts, like writing misleading articles or creating discriminatory software, high $\lambda$ values result in clear refusals and explanations of why the request is problematic. As $\lambda$ decreases, some responses become more engaged with the prompt’s topic, but often still include warnings or factual corrections. At low $\lambda$ values, the models’ response could be considered unsafe, although in a few cases, the content looks like compliance but is benign, such as the example in Fig 10.

This fine-grained control allows for balancing between overly restrictive and potentially unsafe responses. It can be useful for tailoring the model’s refusal behavior to different use cases and user needs while maintaining overall safety and ethical standards.

Figure: Figure 9: Response of Llama-2-7b-Chat to XSTest-safe question on different orthogonlization level. The red output is the undesirable response.

Figure: Figure 10: Response of Gemma-7b-it to harmful question on different orthogonlization level. Note that although the last response ($\lambda=0$) seems not to refuse the harmful prompt by following the instruction of “write an article”, the content is safe and counter-argues the unsafe prompt.

Figure: Figure 11: Response of Llama-2-7b-Chat to harmful question on different orthogonlization level. The red output is the undesirable response.

Figure: Figure 12: Response of Llama-3-8B-Ins to jailbreaking question on different orthogonlization level. The red outputs are the undesirable responses.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: I am mostly puzzled by the claim about the unchanged inference time cost. Please provide some numbers if you have any to make that claim more accurate.
soundness: 3
strengths: 1. The approach is simple and very justified

2. The idea of fine-grained control via partial orthogonalization is quite interesting and can be very useful for addressing the subjectivity nature of safety alignment. The paper does a good job demonstrating that quantitatively and quantitatively as well.

3. The paper presents an interesting set of experiments that demonstrate the effectiveness of the approach.
summary: The paper considers the problem of false refusals in LLMs, e.g., refusing to answer "How do I kill a python process?".  The paper takes a "model internals steerability" approach and builds on previous work that extracts a refusal vector from the model's internal representations and ablate it from all layers at inference time. The paper modifies that method by focusing on false refusals instead of general refusals. The paper also presents a method that allows for numerically controlling how conservative the model should be which can be very useful with subjective cases. The paper presents results and analysis that demonstrates the effectiveness of the presented approach in comparison to existing methods.
weaknesses: 1. The method requires access to model internals which limits its applicability to a certain extent. 

2. The paper claims that the inference cost does not change which does not seem accurate as far as I can tell. The operation in eq. 4 is applied at all layers of the model. The paper needs to report inference time numbers as well as memory consumptions in table 3 instead of claiming they are "unchanged". 

3. While the paper provides some argument against training-based methods, it'd still be valuable (and needed in my opinion) that the paper compares to such methods as a baseline to provide some understanding of the gap in results.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The paper presents a novel method for mitigating false refusals in language models. The method's model-agnostic nature, training-free implementation, and ability to provide fine-grained control make it valuable for enhancing the usability and safety of LLMs. The experimental validation convincingly demonstrates the effectiveness of the proposed technique.

Here are some questions I want to ask:

1. Are there any theoretical guarantees or bounds that support the effectiveness of your method in distinguishing between false and true refusal behaviors?

2. How sensitive is the proposed method to the selection of pseudo-harmful queries used for extracting the false refusal vector?

3. Your experiments are conducted on specific models like GEMMA-7B-IT and various LLAMA models. How does your method scale to larger models (e.g., 175B parameters) or different architectures beyond the ones tested?
soundness: 3
strengths: 1. Addressing an Important Problem: Mitigating false refusals in language models is a relevant and significant issue, as it directly impacts the usability and reliability of LLMs in real-world applications.

2. Novel Approach: The idea of using single vector ablation and orthogonalization to disentangle false refusal behaviors from true refusal mechanisms is innovative and contributes to the existing body of work on LLMs safety.

3. Comprehensive Experiments: The paper conducts experiments across multiple models and datasets, providing a broad evaluation of the proposed method's effectiveness and generalizability.

4. Fine-Grained Control via Partial Orthogonalization: The ability to adjust the orthogonalization coefficient (λ) for fine-tuning the model's sensitivity to ambiguous queries is a valuable feature. It allows users to tailor the model's behavior to specific application requirements, balancing between over-restrictiveness and permissiveness.
summary: The paper presents a method for reducing false refusals in language models by ablating a single vector from the model's activation stream. The proposed approach involves extracting a false refusal vector using pseudo-harmful queries and orthogonalizing it with the true refusal vector to minimize unintended impacts on the model's safety and general capabilities. The authors claim that their method is training-free, model-agnostic, and enables fine-grained calibration of model safety without additional inference costs. Experimental results are provided on several datasets using different language models to demonstrate the effectiveness of the proposed method.
weaknesses: 1. Insufficient Theoretical Justification: The paper lacks a robust theoretical framework explaining why single vector ablation and orthogonalization effectively mitigate false refusals without adversely affecting true refusals or general capabilities. A deeper theoretical analysis or justification is necessary to understand the underlying mechanisms and guarantees of the proposed method.

2. Only partial Mitigation of False Refusals: While the proposed method significantly reduces false refusal rates, it does not entirely eliminate the issue. Further research is needed to fully address the underlying causes of false refusals in language models.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Refer to the weakness part.
soundness: 3
strengths: 1.	The paper proposes a simple and surgical method for mitigating false refusal in LLMs via single vector ablation. The method is training-free and model-free, requiring no extra computation resource or memory during inference.

2.	The idea of separating refusal related features from false refusal vectors by orthogonalization is novel and interesting. Partial orthogonalization also provides fine-grained control of model safety and helpfulness.

3.	The paper conducted comprehensive experiments to demonstrate the effectiveness of the method, and enhance the understanding of different factors in the method. The method can effectively reduce false refusal while maintaining the performance of general tasks.
summary: This paper focuses on the false refusal problem in the safety scenario of LLMs, i.e., LLMs tend to refuse safe requests that superficially resemble unsafe ones (e.g. “how do I kill a Python process?”). Specifically, the paper proposes a simple and surgical method for mitigating false refusal in LLMs via single vector ablation. For a given LLM, they extract a false refusal vector based on the activations of LLM’s layers using harmful, pseudo harmful, and harmless datasets. Then, they demonstrate that ablating this vector reduces false refusal rate without reducing model safety and general model capabilities. The partial orthogonalization also enables fine-grained calibration of model safety.
weaknesses: 1.	While the method is described quite clear in the current form, adding a workflow figure to demonstrate the process of extracting (false) refusal vectors and ablating them can make it easier for understanding.

2.	In section 4, the paper uses greedy decoding for text generation, which however is not the common choice in practice. It would be interesting to see how sampling decoding affects the effectiveness of the method.

3.	In section 4, the samples of harmful, pseudo harmful, and harmless come from different sources. Will the difference of data content or domain affect the method? BTW, there are some typos of the symbols for datasets in line 182. 

4.	In section 5.1, why would Llama2-7B-Chat and Llama3-8B-Chat perform worse when adding the system prompt of Llama2 models?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. I may have missed the points: 

a. Do a harmful and a harmless query differ by a single token or multiple tokens, i.e., does Equation (1) assume both queries are of the same length?

b. In the case where the tokens in both queries vary a lot and/or query lengths are different, what is the physical meaning of a refusal vector?

c. Is the 
 token the only token position chosen for representing a refusal vector?

d. Just to clarify, is Equation (5) directly related to the proposed method in reducing false refusals?

e. How exactly is vector ablation applied to general tasks, such as MMLU?

f. Why does general capability, as measured by acc. or ppl., still drop after vector ablation, as shown in Table 2 (given the significant CR improvements in false refusal benchmarks)?

2. Pseudo-harmful queries are essentially harmless. Are we assuming that all pseudo-harmful queries are potentially treated as harmful by the models used in this work? I am curious that, for pseudo-harmful queries that are potentially treated as harmless by the model, would the difference between a truly harmful query and a pseudo-harmful query create a meaningful direction for ablation (as a false refusal vector)?

3. Can you discuss if the proposed method can be extended to, or compared with other intervention techniques? For example, I am curious about any similarities/differences, pros and cons, between the proposed method and causal interventions, such as [1].

[1] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372, 2022.
soundness: 3
strengths: Mitigating false refusals is important in enabling chat models with more satisfying responses. The idea of orthogonalization on a true refusal vector and a false refusal vector is interesting and useful.

In particular, the authors identify that the true refusal vector and false refusal vector are not independent of each other. Therefore, they propose to apply orthogonalization between the candidate false refusal vectors and the true refusal vector, resulting in an orthogonalized false refusal vector for ablation.

The proposed method achieves increased compliance rates (CR) on false refusal datasets, including ORB-H, XSTest-S(H), and OKTest. The authors also illustrate that the proposed vector ablation method removes false refusal on safe queries while maintaining true refusal on unsafe queries.
summary: This paper proposes an orthogonalization based vector ablation method to mitigate LLM false refusals, which ablates a false refusal vector extracted from the diff-in-means vectors obtained by prompting with pseudo-harmful and harmless queries. A main novelty is to orthogonalize the false refusal vector and the true refusal vector (where the true refusal vector is based on truly harmful and harmless queries) to avoid harming the true refusal ability. The proposed vector ablation method helps remove false refusals on safe queries while maintaining true refusals on unsafe queries.
weaknesses: The presentation could use clearer definitions and equations, and it might be better to provide an intuitive illustration showing example queries in an example transformer network targeting at a layer and a token position. 

For example, in Section 2.1, line 60-62, it is not very clear how the harmless query and harmful query look like, so I am a little confused by the physical meaning of taking the average output over all queries at a token position. 

In addition, in Equation (3), I am not sure why the definition for pt is omitted (according to the context, $p_t$ is token t’s probability at the first token position in the model’s response).

Other possible typos: in line 116, wrong subscript in Equation (7)? In line 182, three repeated $D^{train}_{harmful}$; in line 469, BLUE score or BLEU score?

Meta-review: This paper introduces an innovative method for false refusal resolution in LLMs by employing a novel vector ablation technique, achieving significant improvements without altering model inference costs. While reviewers initially flagged several points for further clarification, the authors provided comprehensive responses that resolved most concerns. The method's utility in safety alignment and its operational viability make it recommendable for acceptance with minor revisions focused on enhancing presentation and empirical substantiation robustness.

Decision: Accept (Poster)