Energy-Driven Steering: Reducing False Refusals in Large Language Models

## Abstract

Abstract Safety alignment of large language models (LLMs) faces a key challenge: current alignment techniques often only focus on improving safety against harmful prompts, causing LLMs to become over-cautious and refuse to respond to benign prompts. Therefore, a key objective of safe alignment is to enhance safety while simultaneously reducing false refusals. In this paper, we introduce Energy-Driven Steering (EDS) , a novel, fine-tuning free framework designed to resolve this challenge through dynamic, inference-time intervention. We trained a lightweight, external Energy-Based Model (EBM) to assign high energy to undesirable (false refusal or jailbreak) states and low energy to desirable (helpful response or safe reject) ones. During inference, EBM maps the LLM’s internal activations to an “energy landscape.” We use the gradient of the energy function to dynamically steer the LLM’s hidden states to low energy regions, correcting the model to generate a desirable response in real-time without modifying its weights. This method decouples behavioral control from the model’s core knowledge, offering a flexible solution with minimal computational overhead. Extensive experiments across a wide range of models show our method successfully achieves this objective: it substantially lowers false refusal rates. For example, raising compliance on the ORB-H benchmark from 57.3% to 82.6% while maintaining the baseline safety performance. Our work presents an effective paradigm for building LLMs that achieve both low false refusal rates and high safety. Our code is available at https://github.com/ericjiang18/LLM_Safety_EBM_Steering . Note: This paper contains examples with potentially disturbing content.

## 1 Introduction

The alignment of large language models (LLMs) with human safety remains a central challenge in artificial intelligence research (Bianchi et al., 2023; Anwar et al., 2024; Xu et al., 2020; Röttger et al., 2020; Sun et al., 2021; Vidgen et al., 2023). Common approaches such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), system prompt engineering, and vector ablation have proven effective. However, these methods often introduce an unintended trade-off: they can lead either to excessive refusal (over-rejection) or to lapses in safety. This behavior is not merely an inconvenience; it severely undermines model utility and reliability in critical domains. For instance, in a healthcare context, a false refusal could block a legitimate query like “How do I treat a burn?”, while in education it might prevent a student from researching “Explain suicide in literature” (Röttger et al., 2023). Such failures erode user trust and can withhold essential information, making the mitigation of false refusals a pressing issue.

Current approaches to this problem fall into two main categories, as illustrated in Figure [1](https://arxiv.org/html/2510.08646v1#S1.F1).
Fine-tuning methods (Ouyang et al., 2022; Ziegler et al., 2019) modify the model’s parameters directly, but this process is computationally expensive, time-consuming, and often struggles to generalize to diverse contexts. A more flexible alternative is fine-tuning free methods (Zheng et al., 2024; Wang et al., 2024), which operate during inference without modifying model weights. Yet, existing techniques in this class, like vector ablation, often lack the precision to reliably distinguish between justified refusals of harmful prompts and false refusals of benign ones. This insufficient discrimination reduces model utility and reliability due to false refusals.

To address these limitations, we introduce Energy-Driven Steering (EDS), a novel, fine-tuning free framework that resolves the tension between safety and helpfulness through dynamic, inference-time intervention. Our core idea is to interpret the LLM’s internal state through the lens of an energy landscape. We deploy a lightweight, external EBM (LeCun et al., 2006) that learns to assign a scalar “energy” value to the LLM’s hidden activations. This EBM is trained via contrastive learning to create an energy landscape where trajectories leading to undesirable outputs (like false refusals) have high energy, while trajectories for desirable, helpful responses have low energy. This energy landscape enables precise discrimination between desirable and undesirable outputs. By performing gradient-based steering on this landscape during inference, EDS can effectively redirect hidden activations that would otherwise lead to false refusals toward low-energy regions without perturbing other originally desirable activations. The modified activation state guides the model to produce desirable outputs. For general capability prompts, the model’s activation trajectories lie in low-energy regions of the learned landscape. The gradient-based steering induces only negligible perturbations, leaving the model’s performance on general tasks unaffected. The model therefore responds normally to such prompts. This mechanism ensures safety, significantly reduces false refusals, and preserves helpfulness.

In our experiments, EDS consistently outperforms other fine-tuning free methods on false refusal benchmarks. While other methods often degrade performance on safety benchmarks, EDS maintains the baseline safety performance. We further validate the general effectiveness of EDS by evaluating it on a wide range of models, including Llama2-7B-Chat (Touvron et al., 2023), Llama-3.1-8B-Instruct (Dubey et al., 2024), and the Qwen3 series (Yang et al., 2025). These results show that EDS can robustly reduce false refusals without compromising model safety.

Our contributions are as follows:

- ❶
We introduce EDS, a novel fine-tuning free framework that leverages a lightweight, externally trained Energy-Based Model (EBM) to dynamically steer the internal activations of an LLM during inference. In contrast to prior methods that rely on static, coarse-grained interventions, EDS constructs an energy landscape over the activation space. This formulation affords it superior discriminative power, enabling fine-grained steering that effectively preserves robust safety while significantly reducing false refusals.
- ❷
We conduct extensive experiments on a wide range of models, including Llama2-7B-Chat, Llama-3.1-8B-Instruct, and the Qwen3 series. The results confirm that EDS outperforms existing methods on various benchmarks, achieving a significant reduction in false refusal rates while robustly preserving safety alignment.

Figure: Figure 1: Comparison of existing LLM alignment strategies. (1) Fine-tuning methods (e.g., SFT, RLHF) modify parameters but suffer from high compute costs, long training times, and poor generalization. (2) Fine-tuning free methods (e.g., promp-driven, output filtering, activation steering) avoid retraining yet lack precision and effective steering capability. Energy-Driven Steering, offers the combined advantages of deployment flexibility, precise discrimination, and effective steering, compared with fine-tuning and fine-tuning free methods.
Refer to caption: x1.png

## 2 Related Works

Fine-tuning methods aim to adapt pre-trained language models to downstream tasks through parameter updates. SFT optimizes models using labeled datasets. RLHF incorporates human preferences via reward modeling and policy optimization, commonly using algorithms such as Proximal Policy Optimization (PPO) (Schulman et al., 2017), Direct Preference Optimization (DPO) (Rafailov et al., 2023), or its variants (Azar et al., 2024; Ethayarajh et al., 2024; Li et al., 2025). Recent advances in safety alignment have extended these frameworks: HH-RLHF (Bai et al., 2022a) and Safe-RLHF (Dai et al., 2023) both align models for safety by employing SFT followed by reinforcement learning with PPO. Unlike conventional RLHF methods that align models by reinforcing desired behaviors through SFT and PPO, Safe Unlearning (Zhang et al., 2024) achieves safety by selectively fine-tuning the model to unlearn unsafe behaviors from harmful prompt–response pairs, offering a lightweight and generalizable defense against jailbreak attacks. Chasing Moving Targets (Liu et al., 2025) introduces an online self-play reinforcement learning framework, where an attacker LM continuously generates evolving adversarial prompts and a defender LM learns through PPO to resist such attacks. Fine-tuning methods require substantial computational resources and training time, and must be retrained at these high costs whenever new safety alignment requirements arise, which limits their flexibility and generalization.

Fine-tuning free Methods achieve safety alignment without altering the model parameters. Representative non-fine-tuning approaches can be divided into three categories:

(1) Context Engineering: Such methods guide the model toward safe outputs through carefully designed prompts. For instance, Red-Teaming + Shielding (Perez et al., 2022) identifies vulnerabilities and then prepends defensive prompts to the context to preemptively block unsafe generations. Similarly, Constitutional AI (0-shot) (Bai et al., 2022b) leverages a set of safety principles to prompt the model to self-critique and revise its outputs during inference. However, the efficacy of prompt-driven methods often diminishes in long conversational contexts where initial instructions can be diluted. They are also vulnerable to subtle adversarial inputs designed to bypass simple rule-based prompting.

(2) Content Filtering: These methods work by filtering out unsafe inputs or model outputs. PDS (Zheng et al., 2024) adds guardrails to inputs and outputs to enforce safety policies. SafeDecoding (Xu et al., 2021) employs safety classifiers to forbid unsafe tokens during the auto-regressive generation.
Such methods rely on the performance of the filter. However, it is always difficult for the filter to scrutinize the powerful LLMs’ diverse unsafe outputs. For instance, a model may produce unsafe content encoded in a Caesar cipher, which the filter would struggle to recognize.

(3) Activation Steering: These techniques directly manipulate the model’s internal activations at inference time. SCAS (Cao et al., 2024) steers activations to reduce over-refusal while maintaining safety. VA (Vector Ablation) (Wang et al., 2024) identifies and ablates refusal-related directions from the model’s hidden states to mitigate unnecessary refusals. These methods involve manually constructing sophisticated positive-negative sample pairs, e.g., how to kill a person versus how to kill a Python process, which limits their scalability and generalizability. Moreover, existing methods generally seek a global steering vector for all inputs indiscriminately, which hinders their effectiveness when handling more diverse inputs.

Our method as a fine-tuning free approach, avoids the excessive computing power cost, high training time cost and limited generalization flexibility of fine-tuning methods. By leveraging Real-time Gradient-Based Steering with EBM, our method addresses the limitations of fine-tuning free methods. It achieves a superior discriminative capability which helps to more effectively correct model’s behavior to reduce the problem of false refusals.

## 3 Preliminaries

An auto-regressive LLM generates a sequence of tokens $Y=(y_{1},y_{2},\ldots,y_{T})$ by modeling the conditional probability of the sequence given a prompt $X$:

$$ $P(Y|X;\phi)=\prod_{t=1}^{T}p(y_{t}|Y_{<t},X;\phi)$ (1) $$

where $\phi$ denotes the parameters of the LLM. This process can be conceptualized as navigating a trajectory through the model’s high-dimensional hidden state space. Let $h_{t}\in\mathbb{R}^{d}$ represent the hidden state of a target layer in the LLM after processing the $t$-th token. This state is the basis for predicting the next token $y_{t+1}$ via the model’s language modeling head, $W_{LM}$:

$$ $p(y_{t+1}|Y_{<t},X;\phi)=\text{softmax}(W_{LM}h_{t})$ (2) $$

Our primary objective is to gain real-time control over the trajectory of hidden states $\mathcal{T}=(h_{1},\ldots,h_{T})$ to steer it away from regions in the state space associated with undesirable behaviors like false refusals. We formalize this by leveraging an Energy-Based Model (EBM), which defines an energy function over the hidden state space. The steering task is to find a modification function $M$ such that for a given state $h_{t}$, the modified state $h^{\prime}_{t}=M(h_{t})$ satisfies:

$$ $E_{\theta}(h^{\prime}_{t})<E_{\theta}(h_{t})$ (3) $$

As we establish in Section [C.3](https://arxiv.org/html/2510.08646v1#A3.SS3), this energy minimization is equivalent to maximizing the probability that the state belongs to a desirable trajectory.

Figure: Figure 2: Overview of the Energy-Driven Steering framework. The method involves (1) gathering ’good’ and ’bad’ hidden state activations from a base LLM , (2) training an Energy-Based Model (EBM) to create an energy landscape that separates them , and (3) using this EBM to perform real-time, gradient-based steering to guide the model away from refusal-prone states during inference.
Refer to caption: x2.png

## 4 Methodology

Our methodology for achieving this objective unfolds in three distinct phases: (1) Activation Data Collection, (2) EBM Training, and (3) Real-time Gradient-Based Steering.

### 4.1 Phase 1: Activation Data Collection

The foundation of our approach is a carefully curated dataset that maps LLM hidden states to nuanced behavioral outcomes. The process begins with a diverse corpus of prompts, $\mathcal{P}$, containing both benign and harmful requests. For each prompt $X\in\mathcal{P}$, we first generate a response $Y$ from the frozen, base LLM.

The core of our data collection is a context-aware classification of the LLM’s behavior. We define a heuristic-based classifier, $C(X,Y)$, that evaluates the appropriateness of the response $Y$ given the nature of the prompt $X$. This results in a label $l$ indicating whether the behavior is desirable (Compliant) or undesirable (Refusal).

$$ $C(X,Y)\rightarrow l\in\{\text{Compliant},\text{Refusal}\}$ (4) $$

Specifically, the classification follows a nuanced logic: compliant responses to benign prompts are desirable, but so are refusals to harmful prompts. Conversely, refusals to benign prompts (false refusals) are undesirable, as are compliant responses to harmful prompts (jailbreaks).

Concurrently, for each generated token $y_{t}\in Y$, we extract and store the corresponding hidden state $h_{t}$ from one or more layers of the LLM. This process populates two distinct sets of hidden states based on the classification outcome:

$$ $\displaystyle\mathcal{D}_{\text{good}}=\{h_{t}\mid\exists(X,Y)\text{ s.t. }$ $\displaystyle((X\text{ benign}\land C(X,Y)=\text{"Compliant"})$ $\displaystyle\lor(X\text{ harmful}\land C(X,Y)=\text{"Refusal"}))\land h_{t}\text{ is from }Y\}$ (5) $\displaystyle\mathcal{D}_{\text{bad}}=\{h_{t}\mid\exists(X,Y)\text{ s.t. }$ $\displaystyle((X\text{ benign}\land C(X,Y)=\text{"Refusal"})$ $\displaystyle\lor(X\text{ harmful}\land C(X,Y)=\text{"Compliant"}))\land h_{t}\text{ is from }Y\}$ (6) $$

The set $\mathcal{D}_{\text{bad}}$ contains hidden states from contextually inappropriate trajectories (i.e., false refusals to benign prompts and compliant responses to harmful prompts), while $\mathcal{D}_{\text{good}}$ contains states from contextually appropriate trajectories (i.e., helpful responses to benign prompts and refusals to harmful prompts). This context-aware data separation is crucial for training an EBM that can distinguish between justified and unjustified refusals.

### 4.2 Phase 2: EBM Training

#### Energy-Based Model Formulation.

Central to our approach is the concept of an Energy-Based Model (EBM), which is characterized by an energy function $E_{\theta}:\mathcal{H}\to\mathbb{R}$ that maps a hidden state $h\in\mathcal{H}=\mathbb{R}^{d}$ to a scalar energy value. A full theoretical treatment is provided in Section [C](https://arxiv.org/html/2510.08646v1#A3). We implement this function as a deep multi-layer perceptron (MLP) with the general form:

$$ $\displaystyle\mathbf{z}_{i}$ $\displaystyle=f_{i}(\mathbf{z}_{i-1})\quad\text{for }i=1,\ldots,L\quad(\text{with }\mathbf{z}_{0}=h)$ (7) $\displaystyle E_{\theta}(h)$ $\displaystyle=\mathbf{W}_{L+1}\mathbf{z}_{L}+b_{L+1}$ (8) $$

where each function $f_{i}$ represents a layer transformation (e.g., linear projection, activation, normalization). This architecture creates a conceptual “landscape” over the LLM’s hidden state space.

#### Training Objective.

The EBM is trained to shape this energy landscape using the InfoNCE contrastive loss, separating the states collected in Phase 1. The objective is to assign high energy to “bad” states from $\mathcal{D}_{\text{bad}}$ and low energy to “good” states from $\mathcal{D}_{\text{good}}$. For an anchor state $h^{+}\in\mathcal{D}_{\text{good}}$ and a set of $N$ negative samples $\{h^{-}_{i}\}_{i=1}^{N}\subset\mathcal{D}_{\text{bad}}$, the loss is:

$$ $\mathcal{L}(\theta)=-\log\frac{\exp(-E_{\theta}(h^{+})/\tau)}{\exp(-E_{\theta}(h^{+})/\tau)+\sum_{i=1}^{N}\exp(-E_{\theta}(h^{-}_{i})/\tau)}$ (9) $$

Here, $\tau$ is a temperature hyperparameter. Minimizing this loss forces $E_{\theta}(h_{\text{good}})\ll E_{\theta}(h_{\text{bad}})$, effectively creating a classifier that can distinguish between desirable and undesirable trajectories. A formal proof is provided in Lemma [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmlemma1).

#### Multi-Layer EBM Training Strategy.

Our approach trains individual EBMs for multiple layers of the LLM simultaneously. For each target layer $l\in\{0,1,\ldots,L-1\}$, we train a separate EBM, $E_{\theta_{l}}(h_{l})$, where $h_{l}$ are the hidden states from that layer. Each model $E_{\theta_{l}}$ is trained independently using the same InfoNCE objective. After training, we evaluate each EBM’s performance on a validation set and select the best-performing models for intervention during inference.

### 4.3 Phase 3: Real-time Gradient-Based Steering

The final phase of our methodology involves integrating the trained EBMs into the LLM’s inference process to actively steer its generative trajectory. This is achieved through a real-time, gradient-based intervention on the model’s hidden states.

#### Steering Mechanism.

The modification function $M(h_{t})$ introduced in our objective is realized via gradient descent on the energy surface defined by a trained EBM. For each selected intervention layer $l$, the hidden state $h_{t}^{(l)}$ is updated as follows:

$$ $h_{t}^{\prime(l)}=h_{t}^{(l)}-\eta\cdot\nabla_{h}E_{\theta_{l}}(h)|_{h=h_{t}^{(l)}}$ (10) $$

where $\eta$ is the steering coefficient, a hyperparameter that controls the strength of the intervention. The term $\nabla_{h}E_{\theta_{l}}(h)$ is the gradient of the energy function with respect to the hidden state, which points in the direction of the steepest ascent on the energy landscape. By moving the hidden state in the negative gradient direction, we are performing a single step of gradient descent to find a state with lower energy. This update rule is formally proven to minimize energy in Theorem [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmtheorem1).

#### Impact on Generation.

The modification of the hidden state $h_{t}^{\prime(l)}$ has a direct and immediate impact on the LLM’s output. The original probability distribution over the vocabulary is computed from the original hidden state $h_{t}^{(l)}$ (Equation [2](https://arxiv.org/html/2510.08646v1#S3.E2)). After steering, the modified hidden state $h_{t}^{\prime(l)}$ is passed to the language modeling head, resulting in a new, steered probability distribution:

$$ $p^{\prime}_{\text{steered}}(y_{t+1}|Y_{<t},X;\phi)=\text{softmax}(W_{LM}h_{t}^{\prime(l)})$ (11) $$

Let $\Delta h_{t}^{(l)}=h_{t}^{\prime(l)}-h_{t}^{(l)}=-\eta\nabla_{h}E_{\theta_{l}}$. The change in the logits (the input to the softmax function) can be approximated by a first-order Taylor expansion:

$$ $\text{Logits}^{\prime}\approx\text{Logits}+W_{LM}\Delta h_{t}^{(l)}=W_{LM}h_{t}^{(l)}-\eta W_{LM}\nabla_{h}E_{\theta_{l}}$ (12) $$

This equation explicitly shows how the steering process adjusts the logits, effectively up-weighting tokens that are more likely to lead to contextually appropriate (low-energy) continuations, and down-weighting tokens associated with contextually inappropriate (high-energy) paths.

This steering process is applied at every generation step for each selected layer, creating a continuous feedback loop that actively guides the generation trajectory away from refusal-prone regions without requiring any fine-tuning of the LLM’s weights $\phi$. This impact is mathematically explained in Corollary [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmcorollary1)

**Table 1: Performance comparison of fine-tuning free methods on safety, false refusal, and general capability benchmarks. EDS approach is evaluated against the original model and other inference-time techniques across several LLMs, including Llama-3.1-8B, Llama-2-7B, and Qwen3 variants. Metrics include Compliance Rate (CR) on safety (JBB, Harmful) and false refusal (ORB-H, XSTest-S, OKTest) benchmarks, as well as accuracy on general capability tests (MMLU, ARC-C, MATH). Higher CR on false refusal and higher accuracy on general capability are better.**
| \rowcolorgray!10<br>Model/Method | Safety | False Refusal | General Capability |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| \rowcolorgray!10 | JBB CR $\downarrow$ | Harmful CR $\downarrow$ | ORB-H CR $\uparrow$ | XSTest-S(H) CR $\uparrow$ | OKTest CR $\uparrow$ | MMLU Acc $\uparrow$ | ARC-C Acc $\uparrow$ | MATH Acc $\uparrow$ |
| Llama3.1-8B-Inst | 10.0$\blacktriangle$0.0 | 10.7$\blacktriangle$0.0 | 57.3$\blacktriangle$0.0 | 85.2$\blacktriangle$0.0 | 98.6$\blacktriangle$0.0 | 68.1$\blacktriangle$0.0 | 72.4$\blacktriangle$0.0 | 31.8$\blacktriangle$0.0 |
| w/ system prompt | 3.0$\blacktriangle$7.0 | 2.3$\blacktriangle$8.4 | 41.0$\blacktriangledown$16.3 | 37.6$\blacktriangledown$47.6 | 53.1$\blacktriangledown$45.5 | 62.0$\blacktriangledown$6.1 | 64.4$\blacktriangledown$8.0 | 27.2$\blacktriangledown$4.6 |
| w/ Surgical vector | 11.0$\blacktriangledown$1.0 | 14.6$\blacktriangledown$3.9 | 76.6$\blacktriangle$19.3 | 93.9$\blacktriangle$8.7 | 98.6$\blacktriangle$0.0 | 67.7$\blacktriangledown$0.4 | 71.3$\blacktriangledown$1.1 | 30.2$\blacktriangledown$1.6 |
| w/ CAST | 12.0$\blacktriangledown$2.0 | 10.9$\blacktriangledown$0.2 | 70.3$\blacktriangle$13.0 | 91.2$\blacktriangle$6.0 | 98.4$\blacktriangledown$0.2 | 67.3$\blacktriangledown$0.8 | 72.0$\blacktriangledown$0.4 | 30.6$\blacktriangledown$1.2 |
| w/ AdaSteer | 13.0$\blacktriangledown$3.0 | 13.5$\blacktriangledown$2.8 | 81.1$\blacktriangle$23.8 | 96.8$\blacktriangle$11.6 | 98.8$\blacktriangle$0.2 | 66.0$\blacktriangledown$2.1 | 69.9$\blacktriangledown$2.5 | 27.8$\blacktriangledown$4.0 |
| w/ AlphaSteer | 11.0$\blacktriangledown$1.0 | 11.1$\blacktriangledown$0.4 | 77.3$\blacktriangle$20.0 | 96.0$\blacktriangle$10.8 | 98.2$\blacktriangledown$0.4 | 66.7$\blacktriangledown$1.4 | 71.2$\blacktriangledown$1.2 | 28.6$\blacktriangledown$3.2 |
| \rowcolorlightpurple  w/ EBM steering | 10.0$\blacktriangle$0.0 | 9.4$\blacktriangle$1.3 | 82.6$\blacktriangle$25.3 | 97.6$\blacktriangle$12.4 | 99.8$\blacktriangle$1.2 | 68.1$\blacktriangle$0.0 | 72.4$\blacktriangle$0.0 | 31.6$\blacktriangledown$0.2 |
| Llama2-7B-Chat | 3.0$\blacktriangle$0.0 | 1.6$\blacktriangle$0.0 | 14.8$\blacktriangle$0.0 | 13.6$\blacktriangle$0.0 | 59.0$\blacktriangle$0.0 | 47.6$\blacktriangle$0.0 | 44.9$\blacktriangle$0.0 | 14.6$\blacktriangle$0.0 |
| w/ system prompt | 0.0$\blacktriangle$3.0 | 0.0$\blacktriangle$1.6 | 8.6$\blacktriangledown$6.2 | 4.5$\blacktriangledown$9.1 | 39.0$\blacktriangledown$20.0 | 47.5$\blacktriangledown$0.1 | 36.6$\blacktriangledown$8.3 | 10.6$\blacktriangledown$4.0 |
| w/ Surgical vector | 5.0$\blacktriangledown$2.0 | 5.5$\blacktriangledown$3.9 | 65.5$\blacktriangle$50.7 | 42.4$\blacktriangle$28.8 | 65.1$\blacktriangle$6.1 | 47.0$\blacktriangledown$0.6 | 44.8$\blacktriangledown$0.1 | 9.4$\blacktriangledown$5.2 |
| w/ CAST | 7.0$\blacktriangledown$4.0 | 7.8$\blacktriangledown$6.2 | 66.7$\blacktriangle$51.9 | 60.0$\blacktriangle$46.4 | 64.6$\blacktriangle$5.6 | 45.6$\blacktriangledown$2.0 | 43.3$\blacktriangledown$1.6 | 13.6$\blacktriangledown$1.0 |
| w/ AdaSteer | 5.0$\blacktriangledown$2.0 | 5.3$\blacktriangledown$3.7 | 75.7$\blacktriangle$60.9 | 62.8$\blacktriangle$49.2 | 66.2$\blacktriangle$7.2 | 46.0$\blacktriangledown$1.6 | 43.7$\blacktriangledown$1.2 | 12.2$\blacktriangledown$2.4 |
| w/ AlphaSteer | 6.0$\blacktriangledown$3.0 | 6.4$\blacktriangledown$4.8 | 75.0$\blacktriangle$60.2 | 67.6$\blacktriangle$54.0 | 66.9$\blacktriangle$7.9 | 46.0$\blacktriangledown$1.6 | 44.3$\blacktriangledown$0.6 | 14.4$\blacktriangledown$0.2 |
| \rowcolorlightpurple  w/ EBM steering | 3.0$\blacktriangle$0.0 | 2.5$\blacktriangledown$0.9 | 78.4$\blacktriangle$63.6 | 72.0$\blacktriangle$58.4 | 67.0$\blacktriangle$8.0 | 47.6$\blacktriangle$0.0 | 44.9$\blacktriangle$0.0 | 14.6$\blacktriangle$0.0 |
| Qwen 3 1.7B | 49.0$\blacktriangle$0.0 | 61.5$\blacktriangle$0.0 | 95.5$\blacktriangle$0.0 | 94.6$\blacktriangle$0.0 | 93.3$\blacktriangle$0.0 | 57.9$\blacktriangle$0.0 | 52.8$\blacktriangle$0.0 | 38.8$\blacktriangle$0.0 |
| w/ system prompt | 27.0$\blacktriangle$22.0 | 33.0$\blacktriangle$28.5 | 54.2$\blacktriangledown$41.3 | 56.4$\blacktriangledown$38.2 | 52.9$\blacktriangledown$40.4 | 49.1$\blacktriangledown$8.8 | 47.3$\blacktriangledown$5.5 | 32.4$\blacktriangledown$6.4 |
| w/ Surgical vector | 51.0$\blacktriangledown$2.0 | 62.9$\blacktriangledown$1.4 | 95.8$\blacktriangle$0.3 | 94.8$\blacktriangle$0.2 | 94.6$\blacktriangle$1.3 | 57.2$\blacktriangledown$0.7 | 52.1$\blacktriangledown$0.7 | 38.2$\blacktriangledown$0.6 |
| w/ CAST | 53.0$\blacktriangledown$4.0 | 63.3$\blacktriangledown$1.8 | 96.2$\blacktriangle$0.7 | 96.0$\blacktriangle$1.4 | 94.4$\blacktriangle$1.1 | 56.8$\blacktriangledown$1.1 | 51.9$\blacktriangledown$0.9 | 38.0$\blacktriangledown$0.8 |
| w/ AdaSteer | 53.0$\blacktriangledown$4.0 | 62.9$\blacktriangledown$1.4 | 95.8$\blacktriangle$0.3 | 95.2$\blacktriangle$0.6 | 95.1$\blacktriangle$1.8 | 57.4$\blacktriangledown$0.5 | 52.6$\blacktriangledown$0.2 | 38.6$\blacktriangledown$0.2 |
| w/ AlphaSteer | 52.0$\blacktriangledown$3.0 | 62.3$\blacktriangledown$0.8 | 96.0$\blacktriangle$0.5 | 96.4$\blacktriangle$1.8 | 95.6$\blacktriangle$2.3 | 56.8$\blacktriangledown$1.1 | 52.2$\blacktriangledown$0.6 | 38.4$\blacktriangledown$0.4 |
| \rowcolorlightpurple  w/ EBM steering | 43.0$\blacktriangle$6.0 | 54.7$\blacktriangle$6.8 | 97.2$\blacktriangle$1.7 | 96.4$\blacktriangle$1.8 | 95.3$\blacktriangle$2.0 | 57.9$\blacktriangle$0.0 | 52.8$\blacktriangle$0.0 | 38.8$\blacktriangle$0.0 |
| Qwen 3 8B | 12.0$\blacktriangle$0.0 | 28.3$\blacktriangle$0.0 | 75.0$\blacktriangle$0.0 | 95.6$\blacktriangle$0.0 | 95.0$\blacktriangle$0.0 | 72.8$\blacktriangle$0.0 | 70.1$\blacktriangle$0.0 | 54.8$\blacktriangle$0.0 |
| w/ system prompt | 6.0$\blacktriangle$6.0 | 5.6$\blacktriangle$22.7 | 43.2$\blacktriangledown$31.8 | 46.8$\blacktriangledown$48.8 | 70.0$\blacktriangledown$25.0 | 70.2$\blacktriangledown$2.6 | 67.7$\blacktriangledown$2.4 | 52.4$\blacktriangledown$2.4 |
| w/ Surgical vector | 13.0$\blacktriangledown$1.0 | 30.1$\blacktriangledown$1.8 | 77.6$\blacktriangle$2.6 | 96.4$\blacktriangle$0.8 | 95.6$\blacktriangle$0.6 | 71.2$\blacktriangledown$1.6 | 68.2$\blacktriangledown$1.9 | 53.8$\blacktriangledown$1.0 |
| w/ CAST | 14.0$\blacktriangledown$2.0 | 30.4$\blacktriangledown$2.1 | 79.5$\blacktriangle$4.5 | 96.8$\blacktriangle$1.2 | 95.8$\blacktriangle$0.8 | 70.5$\blacktriangledown$2.3 | 67.9$\blacktriangledown$2.2 | 53.6$\blacktriangledown$1.2 |
| w/ AdaSteer | 13.0$\blacktriangledown$1.0 | 30.3$\blacktriangledown$2.0 | 78.0$\blacktriangle$3.0 | 96.4$\blacktriangle$0.8 | 96.2$\blacktriangle$1.2 | 70.9$\blacktriangledown$1.9 | 68.4$\blacktriangledown$1.7 | 53.8$\blacktriangledown$1.0 |
| w/ AlphaSteer | 12.0$\blacktriangle$0.0 | 29.9$\blacktriangledown$1.6 | 80.3$\blacktriangle$5.3 | 96.0$\blacktriangle$0.4 | 95.1$\blacktriangle$0.1 | 72.3$\blacktriangledown$0.5 | 69.0$\blacktriangledown$1.1 | 54.4$\blacktriangledown$0.4 |
| \rowcolorlightpurple  w/ EBM steering | 11.0$\blacktriangle$1.0 | 23.9$\blacktriangle$4.4 | 80.6$\blacktriangle$5.6 | 95.6$\blacktriangle$0.0 | 96.4$\blacktriangle$1.4 | 72.8$\blacktriangle$0.0 | 70.1$\blacktriangle$0.0 | 54.8$\blacktriangle$0.0 |
| Qwen 3 14B | 14.0$\blacktriangle$0.0 | 20.1$\blacktriangle$0.0 | 81.1$\blacktriangle$0.0 | 95.2$\blacktriangle$0.0 | 94.0$\blacktriangle$0.0 | 76.1$\blacktriangle$0.0 | 72.5$\blacktriangle$0.0 | 56.0$\blacktriangle$0.0 |
| w/ system prompt | 3.0$\blacktriangle$11.0 | 6.3$\blacktriangle$13.8 | 50.8$\blacktriangledown$30.3 | 71.2$\blacktriangledown$24.0 | 79.0$\blacktriangledown$15.0 | 69.8$\blacktriangledown$6.3 | 69.9$\blacktriangledown$2.6 | 52.8$\blacktriangledown$3.2 |
| w/ Surgical vector | 16.0$\blacktriangledown$2.0 | 25.1$\blacktriangledown$5.0 | 82.6$\blacktriangle$1.5 | 96.0$\blacktriangle$0.8 | 93.8$\blacktriangledown$0.2 | 74.7$\blacktriangledown$1.4 | 72.3$\blacktriangledown$0.2 | 55.2$\blacktriangledown$0.8 |
| w/ CAST | 17.0$\blacktriangledown$3.0 | 24.8$\blacktriangledown$4.7 | 83.0$\blacktriangle$1.9 | 94.8$\blacktriangledown$0.4 | 94.0$\blacktriangle$0.0 | 74.0$\blacktriangledown$2.1 | 72.0$\blacktriangledown$0.5 | 54.6$\blacktriangledown$1.4 |
| w/ AdaSteer | 16.0$\blacktriangledown$2.0 | 21.3$\blacktriangledown$1.2 | 83.7$\blacktriangle$2.6 | 95.6$\blacktriangle$0.4 | 94.0$\blacktriangle$0.0 | 74.4$\blacktriangledown$1.7 | 72.3$\blacktriangledown$0.2 | 54.4$\blacktriangledown$1.6 |
| w/ AlphaSteer | 14.0$\blacktriangle$0.0 | 22.8$\blacktriangledown$2.7 | 84.1$\blacktriangle$3.0 | 96.0$\blacktriangle$0.8 | 94.2$\blacktriangle$0.2 | 73.3$\blacktriangledown$2.8 | 72.1$\blacktriangledown$0.4 | 55.0$\blacktriangledown$1.0 |
| \rowcolorlightpurple  w/ EBM steering | 10.0$\blacktriangle$4.0 | 18.9$\blacktriangle$1.2 | 84.8$\blacktriangle$3.7 | 96.4$\blacktriangle$1.2 | 94.2$\blacktriangle$0.2 | 76.1$\blacktriangle$0.0 | 72.5$\blacktriangle$0.0 | 56.0$\blacktriangle$0.0 |

## 5 EXPERIMENT

To comprehensively evaluate our Energy-Driven Steering method, we conduct a series of experiments designed to measure its performance across three key dimensions: (1) effectiveness, (2) robustness, and (3) efficiency. We assess its ability to mitigate false refusals without compromising safety or general capabilities, test its resilience against sophisticated multi-turn attacks, and analyze its computational overhead. We perform evaluations on a range of recent models, including variants from the Llama and Qwen families. Detailed descriptions of the datasets, baseline methods, and hyperparameter configurations are provided in Appendix [B](https://arxiv.org/html/2510.08646v1#A2).

### 5.1 Effectiveness Analysis

We first evaluate the core effectiveness of our EBM steering approach against both fine-tuning free and fine-tuning based methods. The primary goal is to demonstrate that our method can significantly reduce false refusals while maintaining or improving safety and preserving general knowledge.

#### Comparison with Fine-Tuning Free Methods.

As shown in Table [1](https://arxiv.org/html/2510.08646v1#S4.T1), our EBM steering method consistently outperforms other fine-tuning free techniques in reducing false refusals. For the Llama-3.1-8B-Inst model, EBM steering achieves a Compliance Rate (CR) of 82.6% on the challenging ORB-H benchmark, a substantial improvement of 25.3 percentage points over the baseline’s 57.3%. This is the highest CR among all tested methods. Similar significant gains are observed on the XSTest-S(H) and OKTest benchmarks. Crucially, this improvement in helpfulness does not come at the cost of safety. On the JBB and Harmful safety benchmarks, our method maintains a CR identical or slightly better than the baseline, unlike methods such as Surgical Vector and AdaSteer, which show a degradation in safety performance (i.e., higher compliance with harmful requests). Furthermore, general capabilities, as measured by MMLU, ARC-C, and MATH accuracy, remain almost entirely unaffected, demonstrating that our approach successfully resolves the safety-helpfulness trade-off. Unlike competing methods that force a compromise, our approach demonstrates that it is possible to surgically correct for over-refusal while holistically preserving the model’s carefully tuned safety alignment and core knowledge. This highlights EDS’s ability to make fine-grained adjustments, rather than applying the coarse interventions that lead to performance trade-offs in other systems.

**Table 2: Performance comparison of fine-tuning methods against our EBM steering approach on the Llama-3.1-8B-IT model. The evaluation measures harmful refusal (WGTest, HarmBench, DAN, W.JB), benign compliance (XSTest), and general capability (MMLU). ASR (Attack Success Rate) is reported for harmful refusal, where lower is better. Arrows indicate the desired direction for each metric. Bold indicates the best-performing method.**
|  | \cellcolorgray!10Harmful Refusal | \cellcolorgray!10Benign Compliance | \cellcolorgray!10General Capability |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Model/Method | \cellcolorgray!10WGTest | \cellcolorgray!10HarmBench | \cellcolorgray!10WJB | \cellcolorgray!10DAN | \cellcolorgray!10XSTest | \cellcolorgray!10MMLU |
|  | \cellcolorgray!10adv harm | \cellcolorgray!10adv harm | \cellcolorgray!10adv harm | \cellcolorgray!10adv harm | \cellcolorgray!10vani benign | \cellcolorgray!10Acc |
|  | \cellcolorgray!10ASR $\downarrow$ | \cellcolorgray!10ASR $\downarrow$ | \cellcolorgray!10ASR $\downarrow$ | \cellcolorgray!10ASR $\downarrow$ | \cellcolorgray!10Comply $\uparrow$ | \cellcolorgray!10Score $\uparrow$ |
| Llama-3.1-8B-IT | 0.223$\blacktriangle$0.000 | 0.654$\blacktriangle$0.000 | 0.675$\blacktriangle$0.000 | 0.533$\blacktriangle$0.000 | 0.940$\blacktriangle$0.000 | 0.680$\blacktriangle$0.000 |
| Defender-Only | 0.276$\blacktriangledown$0.053 | 0.243$\blacktriangle$0.411 | 0.695$\blacktriangledown$0.020 | 0.542$\blacktriangledown$0.009 | 0.968$\blacktriangle$0.028 | 0.622$\blacktriangledown$0.058 |
| Self-Play | 0.172$\blacktriangle$0.051 | 0.207$\blacktriangle$0.447 | 0.536$\blacktriangle$0.139 | 0.537$\blacktriangledown$0.004 | 0.964$\blacktriangle$0.024 | 0.624$\blacktriangledown$0.056 |
| Defender-Only + SFT | 0.251$\blacktriangledown$0.028 | 0.260$\blacktriangle$0.394 | 0.432$\blacktriangle$0.243 | 0.452$\blacktriangle$0.081 | 0.932$\blacktriangledown$0.008 | 0.623$\blacktriangledown$0.057 |
| Self-Play + SFT | 0.138$\blacktriangle$0.085 | 0.221$\blacktriangle$0.433 | 0.240$\blacktriangle$0.435 | 0.396$\blacktriangle$0.137 | 0.920$\blacktriangledown$0.020 | 0.623$\blacktriangledown$0.057 |
| \rowcolorlightpurple<br>Ours | 0.219$\blacktriangle$0.004 | 0.289$\blacktriangle$0.365 | 0.207$\blacktriangle$0.468 | 0.372$\blacktriangle$0.161 | 0.976$\blacktriangle$0.036 | 0.680$\blacktriangle$0.000 |

#### Comparison with Fine-Tuning Methods.

In Table [2](https://arxiv.org/html/2510.08646v1#S5.T2), we compare our EBM steering with several intensive fine-tuning strategies on the Llama-3.1-8B-IT model. The results highlight the strength and balanced profile of our approach. On the WJB (0.207) and DAN (0.372) safety benchmarks, EBM steering achieves the lowest Attack Success Rate (ASR), demonstrating superior resistance to prominent jailbreak techniques. While fine-tuning methods like Self-Play + SFT achieve a lower ASR on WGTest and HarmBench, our method still offers a substantial improvement over the baseline. Crucially, our method excels in preventing false refusals, attaining the highest benign compliance rate on XSTest (0.976). Perhaps most importantly, all compared fine-tuning methods lead to a significant drop in MMLU accuracy. In contrast, our approach is unique in preserving the model’s general capabilities entirely, matching the baseline score. This demonstrates that EBM steering provides a more robust and practical solution, achieving a strong, balanced safety profile without the high costs and capability degradation associated with retraining.

Figure: Figure 3: Robustness analysis on multi-turn jailbreak benchmarks. (a) Attack Success Rate (ASR) on the X-Teaming benchmark, evaluating the transferability of different methods against multi-turn attacks. Lower ASR is better.(b) Safety performance on the SafeDialBench benchmark, measuring the models’ ability to identify unsafe content in multi-turn dialogues. The score is based on GPT-4’s judgment, where a higher score indicates better identification capability.
Refer to caption: x3.png

### 5.2 ROBUSTNESS ANALYSIS

To assess the robustness of our method in more realistic conversational settings, we evaluate its performance against multi-turn jailbreak attacks. These attacks are more challenging as they attempt to bypass safety filters over several conversational turns. The results are presented in Figure [3](https://arxiv.org/html/2510.08646v1#S5.F3).

On the X-Teaming benchmark (Figure [3](https://arxiv.org/html/2510.08646v1#S5.F3) (left)), which measures ASR for multi-turn attacks, our EBM steering method achieves a significantly lower success rate for the attacker compared to all other baseline methods. This indicates a stronger resilience in dynamic, conversational contexts. Furthermore, on the SafeDialBench benchmark (Figure [3](https://arxiv.org/html/2510.08646v1#S5.F3) (right)), we evaluate the model’s ability to identify unsafe content within multi-turn dialogues, and evaluated the responses based using GPT-4o-mini. We attribute this enhanced resilience to EDS’s dynamic steering mechanism, which evaluates the generative trajectory at each step. This state-aware approach is fundamentally more resistant to contextual attacks designed to bypass static or coarse-grained safety filters over the course of a conversation.

### 5.3 EFFICIENCY ANALYSIS

**Table 3: Inference time per prompt. Total inference time (s) over 512 prompts and corresponding average time per prompt for Llama 3.1 8B IT model on the Harmful benchmark.**
| \rowcolorgray!10 Model | Avg. Time / Prompt (s) |
| --- | --- |
| Llama-3.1-8B-IT | 1.60 |
| + System Prompt | 1.70 |
| + Surgical Vector | 1.78 |
| + CAST | 1.76 |
| + AdaSteer | 1.80 |
| + Alpha Steer | 1.81 |
| \rowcolorlightpurple + EBM Steering (Ours) | 1.65 |

A critical consideration for any inference-time method is its impact on computational overhead. We measure the average inference latency and memory usage of our EBM steering method compared to other fine-tuning free baselines. All experiments were run on a system with four A6000 GPUs, each with 48GB of VRAM, where the vLLM GPU utilization was capped at 80%. As shown in Table [3](https://arxiv.org/html/2510.08646v1#S5.T3), our approach is highly efficient. For the Llama-3.1-8B-IT model, EBM steering increases the average inference time only marginally, from 821s (1.60s/prompt) to 847s (1.65s/prompt) over 512 prompts. This overhead is substantially lower than that of other methods such as Surgical Vector (910s, 1.78s/prompt) and AlphaSteer (927s, 1.81s/prompt). Moreover, the peak memory usage remains unchanged. These results demonstrate that our method achieves strong behavioral control with negligible impact on efficiency, making it a practical choice for real-world deployment.

### 5.4 ABLATION STUDIES

To understand the sensitivity of our approach to its key hyperparameters, we conducted several ablation studies, with results shown in Figure [4](https://arxiv.org/html/2510.08646v1#S5.F4). We analyzed the impact of the number of layers selected for intervention, the steering coefficient ($\eta$), and the number of gradient steps per token. The results show that performance is stable across a range of layer counts, though it peaks when a significant portion of the model’s layers are utilized (Figure [4](https://arxiv.org/html/2510.08646v1#S5.F4) (left)). The steering coefficient ($\eta$) shows a clear optimal range (Figure [4](https://arxiv.org/html/2510.08646v1#S5.F4) (middle)); a value that is too low provides insufficient correction, while a value that is too high can slightly degrade performance on general tasks. Finally, we observe that the benefits of steering are largely achieved within a few gradient steps, with performance plateauing quickly (Figure [4](https://arxiv.org/html/2510.08646v1#S5.F4) (right)). Overall, these findings highlight the EBM steering framework’s stability, demonstrating robust performance across a well-defined, predictable range of hyperparameters—enabling reliable tuning of EDS for new models without exhaustive, costly parameter sweeps.

Figure: Figure 4: Ablation studies on key hyperparameters for EBM steering with the Llama-3.1-8B-IT model. The plots show how performance on Llama 3.1 8-B IT when running ORB-H CR (%), JBB CR (%), and MMLU Acc (%) varies with changes to: (a) The number of layers selected for intervention. (b) The steering coefficient ($\eta$) . (c) The number of gradient descent steps per token.
Refer to caption: x4.png

To visually understand our method’s effectiveness, (Figure [5](https://arxiv.org/html/2510.08646v1#S5.F5)) visualizes the decision boundaries learned by our EBM versus a Vector Ablation baseline using a t-SNE projection of hidden state activations from the Qwen3-14B model. The left panel shows the Vector Ablation method is akin to slicing the activation space in half with a rigid, linear boundary, an approach that inevitably misses nuance and misclassifies some states as the figure shows. In contrast, the right panel demonstrates our EBM’s energy boundary is not as rigid; it is a flexible, non-linear contour shaped by the learned “energy landscape.” This adaptability allows it to more accurately separate desirable from undesirable states, visually confirming the superior discriminative capability that underlies our method’s strong empirical performance.

Figure: Figure 5: Qualitative comparison of decision boundaries for classifying LLM hidden states. t-SNE visualizations show harmful (red) and harmless (blue) hidden state activations from Qwen3-14B. (Left) Vector Ablation yields a simple linear boundary that poorly separates the clusters. (Right) Our Energy-Based Model (EBM) learns a complex, non-linear boundary (where the energy gradient vanishes), accurately contouring and separating the clusters. This highlights the EBM’s superior discriminative power over linear methods. Boundaries are algorithmically generated by each method.
Refer to caption: x5.png

## 6 CONCLUSION

In this work, we propose Energy-Driven Steering (EDS), a fine-tuning free framework that dynamically corrects LLM behavior at inference to reduce over-conservatism without sacrificing safety. Using an external Energy-Based Model trained on internal activations, EDS steers generation away from high-energy (undesirable) regions in real time—decoupling control from model weights with minimal overhead. Experiments show significant reductions in false refusals, with no loss in safety or general capabilities. This offers a promising path toward LLMs that are safer, more helpful, and more robust—without costly retraining or static policies.

## References

- Anwar et al. (2024)
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al.
Foundational challenges in assuring alignment and safety of large language models.
*arXiv preprint arXiv:2404.09932*, 2024.
- Azar et al. (2024)
Mohammad Gheshlaghi Azar et al.
Generalized preference optimization: Axiomatic alignment is all you need.
*arXiv:2405.21047*, 2024.
- Bai et al. (2022a)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.
Training a helpful and harmless assistant with reinforcement learning from human feedback.
*arXiv preprint arXiv:2204.05862*, 2022a.
- Bai et al. (2022b)
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.
Constitutional ai: Harmlessness from ai feedback.
*arXiv preprint arXiv:2212.08073*, 2022b.
- Bianchi et al. (2023)
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou.
Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions.
*arXiv preprint arXiv:2309.07875*, 2023.
- Cao et al. (2025)
Hongye Cao, Yanming Wang, Sijia Jing, Ziyue Peng, Zhixin Bai, Zhe Cao, Meng Fang, Fan Feng, Boyan Wang, Jiaheng Liu, Tianpei Yang, Jing Huo, Yang Gao, Fanyu Meng, Xi Yang, Chao Deng, and Junlan Feng.
Safedialbench: A fine-grained safety benchmark for large language models in multi-turn dialogues with diverse jailbreak attacks, 2025.
URL [https://arxiv.org/abs/2502.11090](https://arxiv.org/abs/2502.11090).
- Cao et al. (2024)
Zouying Cao, Yifei Yang, and Hai Zhao.
Nothing in excess: Mitigating the exaggerated safety for llms via safety-conscious activation steering.
*arXiv e-prints*, pp. arXiv–2408, 2024.
- Chao et al. (2024)
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al.
Jailbreakbench: An open robustness benchmark for jailbreaking large language models.
*Advances in Neural Information Processing Systems*, 37:55005–55029, 2024.
- Chen et al. (2025)
Sijia Chen, Xiaomin Li, Mengxue Zhang, Eric Hanchen Jiang, Qingcheng Zeng, and Chen-Hsiang Yu.
Cares: Comprehensive evaluation of safety and adversarial robustness in medical llms, 2025.
URL [https://arxiv.org/abs/2505.11413](https://arxiv.org/abs/2505.11413).
- Clark et al. (2018)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
*arXiv preprint arXiv:1803.05457*, 2018.
- Cui et al. (2024)
Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh.
Or-bench: An over-refusal benchmark for large language models.
*arXiv preprint arXiv:2405.20947*, 2024.
- Dai et al. (2023)
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang.
Safe rlhf: Safe reinforcement learning from human feedback.
*arXiv preprint arXiv:2310.12773*, 2023.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv e-prints*, pp. arXiv–2407, 2024.
- Ethayarajh et al. (2024)
Kawin Ethayarajh, Seongmin Kim, and Dan Jurafsky.
Kto: Model alignment as prospect theoretic optimization.
*arXiv:2402.01306*, 2024.
- Han et al. (2024)
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri.
Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms.
*Advances in Neural Information Processing Systems*, 37:8093–8131, 2024.
- Hendrycks et al. (2020)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*, 2020.
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the MATH dataset.
In *Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)*, 2021.
URL [https://openreview.net/forum?id=7Bywt2mQsCe](https://openreview.net/forum?id=7Bywt2mQsCe).
- LeCun et al. (2006)
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al.
A tutorial on energy-based learning.
*Predicting structured data*, 1(0), 2006.
- Li et al. (2025)
Xiaomin Li, Xupeng Chen, Jingxuan Fan, Eric Hanchen Jiang, and Mingye Gao.
Multi-head reward aggregation guided by entropy, 2025.
URL [https://arxiv.org/abs/2503.20995](https://arxiv.org/abs/2503.20995).
- Liu et al. (2025)
Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques.
Chasing moving targets with online self-play reinforcement learning for safer language models.
*arXiv preprint arXiv:2506.07468*, 2025.
- Mazeika et al. (2024)
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al.
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
*arXiv preprint arXiv:2402.04249*, 2024.
- Ouyang et al. (2022)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
*arXiv preprint arXiv:2203.02155*, 2022.
- Perez et al. (2022)
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.
Red teaming language models with language models.
*arXiv preprint arXiv:2202.03286*, 2022.
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*NeurIPS*, 36, 2023.
URL [https://arxiv.org/abs/2305.18290](https://arxiv.org/abs/2305.18290).
- Rahman et al. (2025)
Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel.
X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents.
In *Second Conference on Language Modeling*, 2025.
URL [https://openreview.net/forum?id=gKfj7Jb1kj](https://openreview.net/forum?id=gKfj7Jb1kj).
- Röttger et al. (2020)
Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet B Pierrehumbert.
Hatecheck: Functional tests for hate speech detection models.
*arXiv preprint arXiv:2012.15606*, 2020.
- Röttger et al. (2023)
Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy.
Xstest: A test suite for identifying exaggerated safety behaviours in large language models.
*arXiv preprint arXiv:2308.01263*, 2023.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shen et al. (2024)
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang.
" do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models.
In *Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security*, pp. 1671–1685, 2024.
- Shi et al. (2024)
Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Xun Zhao, and Dahua Lin.
Navigating the overkill in large language models.
*arXiv preprint arXiv:2401.17633*, 2024.
- Sun et al. (2021)
Hao Sun, Guangxuan Xu, Jiawen Deng, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang.
On the safety of conversational models: Taxonomy, dataset, and benchmark.
*arXiv preprint arXiv:2110.08466*, 2021.
- Team et al. (2024)
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.
Gemma: Open models based on gemini research and technology.
*arXiv preprint arXiv:2403.08295*, 2024.
- Touvron et al. (2023)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023.
- Vidgen et al. (2023)
Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A Hale, and Paul Röttger.
Simplesafetytests: a test suite for identifying critical safety risks in large language models.
*arXiv preprint arXiv:2311.08370*, 2023.
- Wang et al. (2024)
Xinpeng Wang, Chengzhi Hu, Paul Röttger, and Barbara Plank.
Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation.
*arXiv preprint arXiv:2410.03415*, 2024.
- Xu et al. (2021)
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein.
Detoxifying language models risks marginalizing minority voices.
*arXiv preprint arXiv:2104.06390*, 2021.
- Xu et al. (2020)
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan.
Recipes for safety in open-domain chatbots.
*arXiv preprint arXiv:2010.07079*, 2020.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Zhang et al. (2024)
Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang.
Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks.
*arXiv preprint arXiv:2407.02855*, 1(2):3, 2024.
- Zheng et al. (2024)
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng.
On prompt-driven safeguarding for large language models.
*arXiv preprint arXiv:2401.18018*, 2024.
- Ziegler et al. (2019)
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.
Fine-tuning language models from human preferences.
*arXiv preprint arXiv:1909.08593*, 2019.

## Appendix A ALGORITHM

Figure: Algorithm 1 Energy-Based Model Steering for LLMs

## Appendix B DETAILED SETUPS OF OUR EXPERIMENTS

Datasets Our experiments are conducted based on datasets as followed.

- •
Training Dataset (1) CARES-21K (Chen et al., 2025)
- •
Safety (1) JailbreakBench (Chao et al., 2024); (2)HarmBench (Mazeika et al., 2024); (3)XSTest Unsafe (Röttger et al., 2023); (4)Wildguard Test (Han et al., 2024); (5)DAN (Shen et al., 2024)
- •
False Refusal (1) Orbench (Cui et al., 2024); (2) OKTest (Shi et al., 2024); (3)XSTest Safe (Röttger et al., 2023);
- •
General Capability (1) MMLU (Hendrycks et al., 2020); (2) ARC (Clark et al., 2018); (3) MATH (Hendrycks et al., 2021)
- •
Multi-Turn Attack (1) X-Teaming (Rahman et al., 2025); (2) SafeDialBench (Cao et al., 2025)

Baselines Our EBM mothed is compared with original models, models with fine-tuning free methods and models with fine-tuning methods as followed.

- •
Original models (1) Llama3.1-8B-Instruct (Dubey et al., 2024); (2) Llama2-7B-Chat (Touvron et al., 2023); (3) Gemma-7B (Team et al., 2024); (4) Qwen3-1.7B (Yang et al., 2025); (5)Qwen3-8B (Yang et al., 2025); (6) Qwen3-14B (Yang et al., 2025)
- •
Finetuing-Free (1) System prompt; (2) Vector ablation;
- •
Finetuing (1) Denfender-Only; (2) Self-Play; (3)Denfender-Only + SFT; (4) Self-Play + SFT. All from  (Liu et al., 2025)

### B.1 Implementation Details and Hyperparameters

#### EBM Data Collection and Processing.

The dataset for training the EBMs was constructed using the SafeMedEval-21K training dataset, which provides a rich collection of medical prompts with varying harmfulness levels. We employed a balanced sampling strategy, extracting 1,000 prompts each for harmless content (filtering for harmful_level: 0) and harmful content (filtering for harmful_level: 2). Responses were generated using vLLM with optimized inference parameters: tensor parallelism was set to 1, GPU memory utilization was capped at 80%, and the maximum sequence length was limited to 512 tokens. For fallback scenarios, we used standard HuggingFace generation with a batch size of 16. All activations were extracted from the last token position of each generated sequence using a dedicated extraction batch size of 16 to balance memory usage and processing speed.

#### EBM Architecture and Training Configuration.

All EBMs utilize our complex architecture, a 4-layer MLP with progressive dimension reduction: [2048 → 1024 → 1024 → 512]. Each layer incorporates Layer Normalization for stable training and Dropout (rate 0.15) for regularization. We train an individual EBM for every layer of the host LLM, enabling fine-grained control across the model’s representation space. The training process spans 120 epochs using the Adam optimizer with a carefully tuned learning rate of $5\times 10^{-5}$. The InfoNCE contrastive loss employs a temperature parameter $\tau=0.10$ to sharpen the softmax distribution. Training data is processed in batches of 64, and we use an 80/20 train-validation split for model selection.

#### Inference-time Steering Configuration.

During inference, steering is applied to the top-performing layers as determined by validation accuracy. The intervention strategy varies significantly across models to account for their different architectures and training procedures. All hyperparameters were tuned individually for each model through grid search on a held-out development set.

**Table 4: Comprehensive hyperparameter configuration for all evaluated models.**
| Hyperparameter | Llama-2-7B | Llama-3.1-8B | Qwen3-1.7B | Qwen3-8B | Qwen3-14B |
| --- | --- | --- | --- | --- | --- |
| EBM Training Configuration |  |  |  |  |  |
| Architecture | Complex | Complex | Complex | Complex | Complex |
| Hidden dimensions | [2048,1024,1024,512] | [2048,1024,1024,512] | [2048,1024,1024,512] | [2048,1024,1024,512] | [2048,1024,1024,512] |
| Dropout rate | 0.15 | 0.15 | 0.15 | 0.15 | 0.15 |
| Layer normalization | Yes | Yes | Yes | Yes | Yes |
| Training epochs | 120 | 120 | 120 | 120 | 120 |
| Learning rate | $5\times 10^{-5}$ | $5\times 10^{-5}$ | $5\times 10^{-5}$ | $5\times 10^{-5}$ | $5\times 10^{-5}$ |
| Batch size | 64 | 64 | 64 | 64 | 64 |
| InfoNCE temperature ($\tau$) | 0.10 | 0.10 | 0.10 | 0.10 | 0.10 |
| Training data size | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 |
| Optimizer | Adam | Adam | Adam | Adam | Adam |
| Inference-time Steering Configuration |  |  |  |  |  |
| Top-N layers selected | 12 | 15 | 3 | 10 | 20 |
| Steering coefficient ($\eta$) | 0.95 | 0.1 | 1.0 | 0.30 | 0.30 |
| Gradient steps per token | 12 | 3 | 10 | 3 | 3 |
| Intervention layers | All trained | All trained | All trained | All trained | All trained |
| Activation positions | Last token (-1) | Last token (-1) | Last token (-1) | Last token (-1) | Last token (-1) |
| Data Generation Configuration |  |  |  |  |  |
| Max generation tokens | 512 | 512 | 512 | 512 | 512 |
| Extraction batch size | 16 | 16 | 16 | 16 | 16 |
| GPU memory utilization | 80% | 80% | 80% | 80% | 80% |
| Tensor parallel size | 1 | 1 | 1 | 1 | 1 |
| vLLM max sequence length | 512 | 512 | 512 | 512 | 512 |

#### Model-specific Tuning Rationale.

The significant variation in steering hyperparameters across models reflects their different sensitivity to activation perturbations. Larger models (Llama-3.1-8B, Qwen3-14B) generally require more conservative steering coefficients and fewer gradient steps to maintain stability, while smaller models (Qwen3-1.7B) can accommodate more aggressive intervention. The number of selected layers for steering correlates with model capacity: deeper models benefit from intervention across more layers to capture complex representational patterns.

#### Dataset Configuration and Evaluation Setup.

Our evaluation framework encompasses three categories of benchmarks: safety evaluation (measuring resistance to harmful prompts), false refusal evaluation (measuring appropriate compliance to benign prompts), and general capability evaluation. Each category employs specific datasets and evaluation methodologies as detailed in Table [5](https://arxiv.org/html/2510.08646v1#A2.T5).

**Table 5: Evaluation dataset configuration and methodology.**
| Evaluation Category | Dataset | Sample Size | Evaluation Method |
| --- | --- | --- | --- |
| Safety | JailbreakBench (JBB) | 100 | Compliance rate |
| HarmBench | 512 | Compliance rate |  |
| XSTest Unsafe | 200 | Compliance rate |  |
|  | WG Test | 324 | Attack Sucess Rate |
|  | Wildguard Test | 2,000 | Attack Sucess Rate |
|  | DAN Unsafe | 78 | Attack Sucess Rate |
| False Refusal | ORB-Hard | 264 | Compliance rate |
| XSTest Safe | 250 | Compliance rate |  |
| OKTest | 450 | Compliance rate |  |
| General Capability | MMLU | 285 | Accuracy |
| ARC-Challenge | 1,172 | Accuracy |  |
| MATH | 500 | Accuracy |  |
| Multi-Turn Attack | X-Teaming | 50 | Attack Sucess Rate |
| SafeDialBench | 60 | GPT 4-o mini |  |

#### Hardware and Infrastructure Requirements.

All experiments were conducted on NVIDIA A6000 GPUs with 48GB VRAM. The choice of GPU memory utilization parameters was carefully tuned to maximize throughput while preventing out-of-memory errors. For EBM training, we utilize CUDA optimization with mixed precision training disabled to maintain numerical stability of the energy gradients. The activation extraction phase requires the most memory, necessitating the lower batch sizes (16) compared to standard LLM inference.

#### Evaluation Metrics and Methodology.

Our evaluation employs multiple complementary metrics to assess different aspects of model performance. For safety evaluation, we compute the Compliance Rate (CR), defined as the percentage of harmful prompts that the model appropriately refuses. For false refusal evaluation, we use the same CR metric but applied to benign prompts, where higher compliance indicates fewer false refusals. Attack Success Rate (ASR) represents the inverse of CR for harmful prompts. The substring matching evaluation method employs a curated list of refusal indicators including phrases like “I cannot”, “I’m sorry”, and “I’m not able to”, with responses containing these phrases classified as refusals.

#### Robustness Experiment Setup.

Our robustness analysis employed two multi-turn attack benchmarks to evaluate performance in conversational contexts. For the X-Teaming benchmark, we assessed transferability against multi-turn attacks using test cases derived from the first 50 harmful behaviors in HarmBench. Each behavior was tested with 10 attack plans across 3 turns. For the SafeDialBench benchmark, we selected 60 multi-turn attack dialogues, 10 for each of the six safety dimensions (aggression, ethics, fairness, legality, morality, and privacy). Model responses were scored by GPT-4o mini, using the prompt from the original paper, to exclusively assess the model’s ability to identify unsafe content.

#### Ablation Study Configuration.

All ablation studies were conducted on the Llama-3.1-8B-IT model to analyze the sensitivity of our method’s key hyperparameters. We evaluated the impact on performance by varying one parameter at a time while keeping others fixed at their optimal values (as detailed in Table [4](https://arxiv.org/html/2510.08646v1#A2.T4)). The performance was measured using three metrics: ORB-H CR (false refusal), JBB CR (safety), and MMLU Accuracy (general capability). We investigated: (1) the number of intervention layers, testing values from 10 to 30; (2) the steering coefficient ($\eta$), testing values from 0.05 to 0.25; and (3) the number of gradient steps per token, testing values from 1 to 20.

#### Reproducibility and Code Availability.

All experiments can be reproduced using the provided configuration files and the command: python -m pipeline.run_pipeline --config_path configs/[model_config].yaml. The complete codebase, including EBM implementations, evaluation scripts, and data processing utilities, is available in the supplementary material. Environment setup is automated via the provided setup.sh script, which installs all required dependencies including the LM Evaluation Harness.

## Appendix C THEORETICAL JUSTIFICATION OF ENERGY GRADIENT-BASED STEERING

This section provides a rigorous mathematical justification for the gradient-based steering mechanism. We formalize the components of our framework using definitions, lemmas, and theorems to prove that the proposed steering update is a principled optimization procedure that guides the LLM’s generative trajectory away from regions associated with false refusals.

### C.1 Preliminaries and Formal Definitions

###### Definition C.1 (Energy Function) .

An Energy-Based Model (EBM) is defined by a parameterized energy function $E_{\theta}:\mathcal{H}\to\mathbb{R}$, where $\mathcal{H}=\mathbb{R}^{d}$ is the hidden state space of a Large Language Model. The function maps a hidden state $h\in\mathcal{H}$ to a scalar energy value. A lower energy is designed to correspond to a higher probability of a desirable outcome (e.g., a compliant response), while higher energy corresponds to an undesirable outcome (e.g., a false refusal). The function is realized by a multi-layer perceptron with parameters $\theta$.

###### Definition C.2 (Optimal Energy Function) .

Let $\mathcal{D}_{\text{good}}\subset\mathcal{H}$ be the set of hidden states from desirable trajectories (e.g., compliant) and $\mathcal{D}_{\text{bad}}\subset\mathcal{H}$ be the set of states from undesirable trajectories (e.g., false refusals). An optimal energy function $E^{*}(h)$ is a function that perfectly separates these sets, such that for any $h_{\text{good}}\in\mathcal{D}_{\text{good}}$ and $h_{\text{bad}}\in\mathcal{D}_{\text{bad}}$, there exists a margin $m>0$ where:

$$ $E^{*}(h_{\text{bad}})>E^{*}(h_{\text{good}})+m$ (13) $$

Our trained EBM, $E_{\theta}(h)$, serves as an approximation of this optimal function, i.e., $E_{\theta}(h)\approx E^{*}(h)$.

###### Definition C.1 (Energy Function) .

###### Definition C.2 (Optimal Energy Function) .

### C.2 EBM Training and Energy Landscape

The parameters $\theta$ of the energy function $E_{\theta}(h)$ are learned by optimizing a training objective designed to shape the energy landscape according to Definition [C.2](https://arxiv.org/html/2510.08646v1#A3.Thmdefinition2).

#### Training Objective Function.

The EBM is trained using the InfoNCE contrastive loss. For an anchor state $h^{+}\in\mathcal{D}_{\text{good}}$ and a set of $N$ negative samples $\{h^{-}_{i}\}_{i=1}^{N}\subset\mathcal{D}_{\text{bad}}$, the loss is:

$$ $\mathcal{L}(\theta)=-\mathbb{E}_{h^{+},\{h^{-}_{i}\}}\left[\log\frac{\exp(-E_{\theta}(h^{+})/\tau)}{\exp(-E_{\theta}(h^{+})/\tau)+\sum_{i=1}^{N}\exp(-E_{\theta}(h^{-}_{i})/\tau)}\right]$ (14) $$

where $\tau$ is a temperature hyperparameter.

###### Lemma C.1 (Energy Landscape Property) .

Minimizing the InfoNCE loss (Equation [14](https://arxiv.org/html/2510.08646v1#A3.E14)) trains the energy function $E_{\theta}(h)$ to assign lower energy values to hidden states from desirable trajectories ($\mathcal{D}_{\text{good}}$) and higher energy values to hidden states from undesirable trajectories ($\mathcal{D}_{\text{bad}}$). Formally, for a well-trained model, if $h_{\text{good}}\in\mathcal{D}_{\text{good}}$ and $h_{\text{bad}}\in\mathcal{D}_{\text{bad}}$, it is highly probable that $E_{\theta}(h_{\text{good}})<E_{\theta}(h_{\text{bad}})$.

###### Proof.

The InfoNCE loss is a form of cross-entropy loss. Let the logits be $s^{+}=-E_{\theta}(h^{+})/\tau$ and $s^{-}_{i}=-E_{\theta}(h^{-}_{i})/\tau$. The loss for a single sample can be written as:

$$ $\mathcal{L}=-s^{+}+\log\left(\exp(s^{+})+\sum_{i=1}^{N}\exp(s^{-}_{i})\right)$ (15) $$

The parameter update rule for gradient descent is $\theta_{t+1}=\theta_{t}-\alpha\nabla_{\theta}\mathcal{L}$. The change in an energy value $E$ is approximately $\Delta E\approx(\nabla_{\theta}E)^{T}\Delta\theta=-\alpha(\nabla_{\theta}E)^{T}(\nabla_{\theta}\mathcal{L})$. Using the chain rule, $\nabla_{\theta}\mathcal{L}=\frac{\partial\mathcal{L}}{\partial E}\nabla_{\theta}E$, we get:

$$ $\Delta E\approx-\alpha(\nabla_{\theta}E)^{T}\left(\frac{\partial\mathcal{L}}{\partial E}\nabla_{\theta}E\right)=-\alpha\frac{\partial\mathcal{L}}{\partial E}\|\nabla_{\theta}E\|_{2}^{2}$ (16) $$

This implies $\text{sign}(\Delta E)=-\text{sign}(\frac{\partial\mathcal{L}}{\partial E})$. We now compute these partial derivatives.

Derivative w.r.t. $E_{\theta}(h^{+})$:
Let $E^{+}=E_{\theta}(h^{+})$. The derivative is computed via the chain rule $\frac{\partial\mathcal{L}}{\partial E^{+}}=\frac{\partial\mathcal{L}}{\partial s^{+}}\frac{\partial s^{+}}{\partial E^{+}}$. First:

$$ $\frac{\partial s^{+}}{\partial E^{+}}=-\frac{1}{\tau}$ (17) $$

$$ $\frac{\partial\mathcal{L}}{\partial s^{+}}=-1+\frac{1}{\exp(s^{+})+\sum_{i}\exp(s^{-}_{i})}\cdot\exp(s^{+})=\frac{\exp(s^{+})}{\exp(s^{+})+\sum_{i}\exp(s^{-}_{i})}-1$ (18) $$

Combining these gives:

$$ $\frac{\partial\mathcal{L}}{\partial E^{+}}=\left(\frac{\exp(s^{+})}{\exp(s^{+})+\sum_{i}\exp(s^{-}_{i})}-1\right)\left(-\frac{1}{\tau}\right)=\frac{1}{\tau}\left(1-P(h^{+})\right)>0$ (19) $$

where $P(h^{+})$ is the softmax probability of the positive sample. Therefore, $\Delta E_{\theta}(h^{+})\propto-(+)<0$, meaning the energy of ’good’ states decreases.

Derivative w.r.t. $E_{\theta}(h^{-}_{j})$:
Let $E^{-}_{j}=E_{\theta}(h^{-}_{j})$. The derivative is $\frac{\partial\mathcal{L}}{\partial E^{-}_{j}}=\frac{\partial\mathcal{L}}{\partial s^{-}_{j}}\frac{\partial s^{-}_{j}}{\partial E^{-}_{j}}$. First:

$$ $\frac{\partial s^{-}_{j}}{\partial E^{-}_{j}}=-\frac{1}{\tau}$ (20) $$

$$ $\frac{\partial\mathcal{L}}{\partial s^{-}_{j}}=\frac{1}{\exp(s^{+})+\sum_{i}\exp(s^{-}_{i})}\cdot\exp(s^{-}_{j})=P(h^{-}_{j})$ (21) $$

Combining these gives:

$$ $\frac{\partial\mathcal{L}}{\partial E^{-}_{j}}=P(h^{-}_{j})\left(-\frac{1}{\tau}\right)=-\frac{1}{\tau}P(h^{-}_{j})<0$ (22) $$

Therefore, $\Delta E_{\theta}(h^{-}_{j})\propto-(-)>0$, meaning the energy of ’bad’ states increases. This completes the proof.
∎

###### Lemma C.1 (Energy Landscape Property) .

###### Proof.

### C.3 Probabilistic Interpretation and Steering as MAP Inference

The learned energy function can be formally linked to a probability distribution over the hidden state space via the Gibbs-Boltzmann distribution.

###### Definition C.3 (State Probability Density) .

The probability density that a hidden state $h$ belongs to the class of desirable (compliant) states, $\mathcal{C}_{\text{good}}$, is given by:

$$ $p(h\in\mathcal{C}_{\text{good}})=\frac{\exp(-E_{\theta}(h)/\tau)}{Z(\theta,\tau)}$ (23) $$

where $Z(\theta,\tau)$ is the partition function, which normalizes the distribution over the entire state space $\mathcal{H}$:

$$ $Z(\theta,\tau)=\int_{h^{\prime}\in\mathcal{H}}\exp(-E_{\theta}(h^{\prime})/\tau)dh^{\prime}$ (24) $$

This formulation is a direct consequence of the energy landscape established in Lemma [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmlemma1). For any two states $h_{1},h_{2}\in\mathcal{H}$, their relative probability is:

$$ $\frac{p(h_{1}\in\mathcal{C}_{\text{good}})}{p(h_{2}\in\mathcal{C}_{\text{good}})}=\frac{\exp(-E_{\theta}(h_{1})/\tau)}{\exp(-E_{\theta}(h_{2})/\tau)}=\exp\left(-\frac{E_{\theta}(h_{1})-E_{\theta}(h_{2})}{\tau}\right)$ (25) $$

If we take $h_{1}\in\mathcal{D}_{\text{good}}$ and $h_{2}\in\mathcal{D}_{\text{bad}}$, from Lemma [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmlemma1) we know $E_{\theta}(h_{1})<E_{\theta}(h_{2})$, which implies $E_{\theta}(h_{1})-E_{\theta}(h_{2})<0$. Therefore, the exponent is positive, leading to $p(h_{1})>p(h_{2})$. This confirms that low-energy states are exponentially more probable.

The objective of our steering mechanism can now be re-framed as a Maximum A Posteriori (MAP) inference problem: finding the hidden state $h^{*}$ that maximizes the probability of belonging to the desirable class.

$$ $h^{*}=\arg\max_{h\in\mathcal{H}}p(h\in\mathcal{C}_{\text{good}})$ (26) $$

This maximization is equivalent to minimizing the energy function $E_{\theta}(h)$:

$$ $\displaystyle\arg\max_{h}p(h)$ $\displaystyle=\arg\max_{h}\frac{\exp(-E_{\theta}(h)/\tau)}{Z(\theta,\tau)}$ (27) $\displaystyle=\arg\max_{h}\log\left(\frac{\exp(-E_{\theta}(h)/\tau)}{Z(\theta,\tau)}\right)$ (28) $\displaystyle=\arg\max_{h}\left(-\frac{E_{\theta}(h)}{\tau}-\log Z(\theta,\tau)\right)$ (29) $\displaystyle=\arg\min_{h}E_{\theta}(h)$ (30) $$

The equivalence holds because the logarithm is a strictly monotonic function, and $Z(\theta,\tau)$ and $\tau$ are positive constants with respect to $h$.

This probabilistic framing demonstrates that the gradient descent on energy performed in Theorem [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmtheorem1) is not merely an ad-hoc procedure, but a principled method for performing gradient-based MAP inference. The gradient of the log-probability with respect to the state $h$ is directly proportional to the negative energy gradient:

$$ $\nabla_{h}\log p(h\in\mathcal{C}_{\text{good}})=\nabla_{h}\left(-\frac{E_{\theta}(h)}{\tau}-\log Z\right)=-\frac{1}{\tau}\nabla_{h}E_{\theta}(h)$ (31) $$

Therefore, the gradient ascent update rule to maximize the log-probability is:

$$ $h_{k+1}=h_{k}+\alpha\nabla_{h}\log p(h_{k})=h_{k}-\frac{\alpha}{\tau}\nabla_{h}E_{\theta}(h_{k})$ (32) $$

This is precisely the form of our steering update rule, with the steering coefficient $\eta=\alpha/\tau$. The subsequent sections provide a formal proof of convergence for this procedure.

###### Definition C.3 (State Probability Density) .

### C.4 Gradient-Based Steering Mechanism and Analysis

The steering mechanism uses the gradient of the learned energy function to modify the LLM’s hidden states during inference.

###### Definition C.4 (Energy Gradient) .

The energy gradient, $\nabla_{h}E_{\theta}(h)$, is the vector of partial derivatives of the energy function with respect to the input hidden state $h$:

$$ $\nabla_{h}E_{\theta}(h)=\left[\frac{\partial E_{\theta}}{\partial h_{1}},\frac{\partial E_{\theta}}{\partial h_{2}},\ldots,\frac{\partial E_{\theta}}{\partial h_{d}}\right]^{T}$ (33) $$

This gradient is computed via backpropagation and points in the direction of the steepest ascent on the energy surface.

###### Theorem C.1 (Energy Minimization via Gradient-Based Steering) .

Let $h_{t}$ be the hidden state at generation step $t$. Let the steering update rule be defined as:

$$ $h^{\prime}_{t}=h_{t}-\eta\cdot\nabla_{h}E_{\theta}(h)|_{h=h_{t}}$ (34) $$

For a steering coefficient $\eta$ satisfying $0<\eta<\frac{2}{\lambda_{\max}(\mathbf{H}(h_{t}))}$, where $\lambda_{\max}(\mathbf{H}(h_{t}))$ is the maximum eigenvalue of the Hessian matrix $\mathbf{H}$ of $E_{\theta}$ at $h_{t}$, the update guarantees a decrease in energy, i.e., $E_{\theta}(h^{\prime}_{t})<E_{\theta}(h_{t})$, provided that $\nabla_{h}E_{\theta}(h_{t})\neq\mathbf{0}$.

###### Proof.

Let $g(h)=\nabla_{h}E_{\theta}(h)$. The change in energy is $\Delta E=E_{\theta}(h_{t}-\eta g(h_{t}))-E_{\theta}(h_{t})$. Using a second-order Taylor expansion for $E_{\theta}$ around $h_{t}$:

$$ $E_{\theta}(h_{t}-\eta g(h_{t}))=E_{\theta}(h_{t})-\eta g(h_{t})^{T}g(h_{t})+\frac{1}{2}\eta^{2}g(h_{t})^{T}\mathbf{H}(h_{t})g(h_{t})+\mathcal{O}(\eta^{3})$ (35) $$

The change in energy can be written as:

$$ $\Delta E=-\eta\|g(h_{t})\|_{2}^{2}+\frac{1}{2}\eta^{2}g(h_{t})^{T}\mathbf{H}(h_{t})g(h_{t})+\mathcal{O}(\eta^{3})$ (36) $$

From the Rayleigh-Ritz theorem, the quadratic term is bounded by the maximum eigenvalue $\lambda_{\max}$ of the Hessian $\mathbf{H}(h_{t})$:

$$ $g(h_{t})^{T}\mathbf{H}(h_{t})g(h_{t})\leq\lambda_{\max}(\mathbf{H}(h_{t}))\|g(h_{t})\|_{2}^{2}$ (37) $$

Substituting this upper bound into the expression for $\Delta E$:

$$ $\Delta E\leq-\eta\|g(h_{t})\|_{2}^{2}+\frac{1}{2}\eta^{2}\lambda_{\max}(\mathbf{H}(h_{t}))\|g(h_{t})\|_{2}^{2}$ (38) $$

Factoring out $\|g(h_{t})\|_{2}^{2}$:

$$ $\Delta E\leq\left(-\eta+\frac{1}{2}\eta^{2}\lambda_{\max}(\mathbf{H}(h_{t}))\right)\|g(h_{t})\|_{2}^{2}$ (39) $$

For the energy to decrease, we require the term in the parentheses to be negative. Assuming $g(h_{t})\neq\mathbf{0}$:

$$ $\displaystyle-\eta+\frac{1}{2}\eta^{2}\lambda_{\max}(\mathbf{H}(h_{t}))$ $\displaystyle<0$ $\displaystyle\frac{1}{2}\eta^{2}\lambda_{\max}(\mathbf{H}(h_{t}))$ $\displaystyle<\eta$ $\displaystyle\eta\lambda_{\max}(\mathbf{H}(h_{t}))$ $\displaystyle<2$ $\displaystyle\eta$ $\displaystyle<\frac{2}{\lambda_{\max}(\mathbf{H}(h_{t}))}$ (40) $$

Thus, for any $\eta$ in the specified range $0<\eta<2/\lambda_{\max}(\mathbf{H}(h_{t}))$, we have $\Delta E<0$, which completes the proof.
∎

###### Corollary C.1 (Steering towards Compliance by Mitigating False Refusals) .

The primary objective is to mitigate false refusals. Based on Lemma [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmlemma1), a false refusal corresponds to a hidden state $h_{\text{bad}}$ in a high-energy region of the landscape. By Theorem [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmtheorem1), the gradient descent update, $h^{\prime}_{t}=h_{t}-\eta\nabla_{h}E_{\theta}(h_{t})$, is a principled procedure for minimizing the energy of a hidden state. Therefore, applying this steering update to a hidden state on a trajectory towards a false refusal (a high-energy state) will move it towards a lower-energy region, which corresponds to a desirable (compliant) state. This formally justifies our mechanism for mitigating false refusals by navigating the learned energy landscape.

###### Proof of Corollary.

Let an initial state $h_{0}\in\mathcal{H}$ be on a trajectory towards a false refusal, which implies $h_{0}\in\mathcal{D}_{\text{bad}}$ by Lemma [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmlemma1). Our goal is to show that the sequence $\{h_{k}\}_{k=0}^{\infty}$ generated by the recurrence relation

$$ $h_{k+1}=h_{k}-\eta\nabla_{h}E_{\theta}(h_{k})$ (41) $$

converges to a point $h^{*}\in\mathcal{D}_{\text{good}}$. Let $E_{k}=E_{\theta}(h_{k})$. By Theorem [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmtheorem1), the energy sequence $\{E_{k}\}$ is monotonically decreasing. Since $E_{\theta}$ is bounded below by some $E_{\min}$, the Monotone Convergence Theorem ensures that the limit $E^{*}=\lim_{k\to\infty}E_{k}$ exists. The existence of this limit implies $\lim_{k\to\infty}(E_{k}-E_{k+1})=0$. From the proof of Theorem [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmtheorem1), we have the inequality:

$$ $E_{k}-E_{k+1}\geq\eta\left(1-\frac{1}{2}\eta\lambda_{\max}(\mathbf{H}(h_{k}))\right)\|\nabla_{h}E_{\theta}(h_{k})\|_{2}^{2}$ (42) $$

Let $C_{k}=\eta(1-\frac{1}{2}\eta\lambda_{\max}(\mathbf{H}(h_{k})))$. For a valid $\eta$, $C_{k}$ is a positive term bounded away from zero. Given $0\leq C_{k}\|\nabla_{h}E_{\theta}(h_{k})\|_{2}^{2}\leq E_{k}-E_{k+1}$, the Squeeze Theorem dictates that as the right-hand side converges to zero, so must the middle term:

$$ $\lim_{k\to\infty}C_{k}\|\nabla_{h}E_{\theta}(h_{k})\|_{2}^{2}=0\implies\lim_{k\to\infty}\|\nabla_{h}E_{\theta}(h_{k})\|_{2}=0$ (43) $$

This condition, $\lim_{k\to\infty}\nabla_{h}E_{\theta}(h_{k})=\mathbf{0}$, signifies that any limit point $h^{*}$ of the sequence $\{h_{k}\}$ must belong to the set of critical points of the energy function, $\mathcal{C}=\{h\in\mathcal{H}\mid\nabla_{h}E_{\theta}(h)=\mathbf{0}\}$. Since the sequence starts in a high-energy region ($h_{0}\in\mathcal{D}_{\text{bad}}$) and the energy strictly decreases, the limit point $h^{*}$ must be a local minimum of $E_{\theta}$. By Lemma [C.1](https://arxiv.org/html/2510.08646v1#A3.Thmlemma1), the set of local minima of the energy function is constructed to be the set of desirable states $\mathcal{D}_{\text{good}}$. Therefore, we have shown that

$$ $\lim_{k\to\infty}h_{k}=h^{*}\in\mathcal{D}_{\text{good}}$ (44) $$

This concludes the proof that the steering process transforms an initial high-energy, undesirable state into a final low-energy, desirable state, thus mitigating the false refusal.
∎

###### Definition C.4 (Energy Gradient) .

###### Theorem C.1 (Energy Minimization via Gradient-Based Steering) .

###### Proof.

###### Corollary C.1 (Steering towards Compliance by Mitigating False Refusals) .

###### Proof of Corollary.

## Appendix D Prompts

## Appendix E Computational Resources

All experiments are performed on four A6000 GPUs with 48GB of VRAM.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: (1) Novelty relative to activation steering and single-vector methods:
Can the authors clearly articulate what conceptual or empirical advantage Energy-Driven Steering offers over existing activation-steering or single-vector ablation methods (e.g., Surgical, Cheap, and Flexible)? In particular, on which input types or safety–helpfulness trade-offs does EDS show distinct improvements?

(2) Baseline coverage for simple SFT approaches:
Could the authors add or discuss results for a straightforward supervised fine-tuning baseline trained specifically on over-refusal data (e.g., FalseReject-style) to quantify the added benefit of the EBM-based steering compared with this much simpler alternative?

(3) Generalization across datasets/domains:
Was any experiment performed where the EBM was trained on one dataset and evaluated on a disjoint or unseen benchmark to verify cross-domain generalization? If not, could the authors comment on how well they expect EDS to transfer across domains and what limits that transfer?
soundness: 3
strengths: (1) Clear, modular idea: Decoupling behavior steering from the base LM via an external EBM is conceptually clean and deployment-friendly (no weight updates)

(2) Empirical breadth: Evaluates across multiple families/sizes and reports three axes: false-refusal, safety, and general capabilities; also includes multi-turn robustness and efficiency (latency) analyses.

(3) Theory support: Provides a principled view of steering as energy minimization and ablations on hyperparameters.

(4) Low overhead: Inference-time cost appears small relative to other steering baselines.
summary: The paper proposes Energy-Driven Steering (EDS), a fine-tuning-free method to reduce false refusals at inference time. A lightweight external energy-based model (EBM) is trained to assign low “energy” to desirable hidden states (helpful answers to benign prompts; safe refusals to harmful prompts) and high energy to undesirable ones (false refusals; jailbreaks). During generation, gradients of this energy are used to nudge hidden states toward low-energy regions, aiming to improve benign compliance without sacrificing safety or general capabilities. Experiments across several open models (Llama-3.1-8B, Llama-2-7B, Qwen3 family) report large gains on over-refusal benchmarks with near-baseline safety, plus multi-turn robustness and modest latency overhead.
weaknesses: (1) Novelty relative to activation steering and single-vector methods: The paper’s mechanism (learning a classifier/energy over hidden states and applying gradient-based perturbations) is close in spirit to prior activation steering / vector ablation approaches. The related-work section acknowledges activation steering and Surgical vector / single-vector ablation [1]. However, the novelty gap feels narrower than claimed: EDS can be interpreted as learning a more expressive steering field, with a heuristic labeler to form positive/negative hidden-state sets. I recommend the authors more explicitly contrast EDS against “Surgical, Cheap, and Flexible...” both conceptually and empirically (e.g., where exactly EDS helps over learned per-layer multi-direction steering, and on which inputs).

(2) Baseline coverage for simple SFT approaches to over-refusal: Recent work (e.g., datasets like FalseReject [2]) shows that targeted supervised fine-tuning can reduce over-refusal while preserving safety to a very high extent. The current paper includes RLHF-style baselines (e.g., Defender/Self-Play + SFT) and several steering baselines, but does not include a simple SFT-only baseline trained specifically on over-refusal data. Given the engineering cost of running an auxiliary EBM online at every step, a strong SFT baseline on over-refusal data should be included for fairness and practicality.

(3) Generalization of the learned shift across datasets/domains: Training data for the EBM is drawn from SafeMedEval-21K (medical-domain prompts) while evaluation spans mixed-domain safety and false-refusal suites. That suggests some cross-domain generalization of the energy field, which is encouraging—but the paper doesn’t present a train/eval protocol to isolate generalization (e.g., leave-one-benchmark-out or leave-one-category-out). A focused study showing EBM trained on dataset A and tested on disjoint dataset B (and vice-versa) would strengthen claims of robustness. Please clarify whether any such disjointness checks were performed.

[1] Wang, Xinpeng, et al. "Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation." arXiv preprint arXiv:2410.03415 (2024).
[2] Zhang, Zhehao, et al. "Falsereject: A resource for improving contextual safety and mitigating over-refusals in llms via structured reasoning." arXiv preprint arXiv:2505.08054 (2025).

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: - Prior work has introduced partial orthogonalization as a way to obtain a more dynamic way of steering (Wang et al., 2024). Have you tested that? Do you see that as alternative to your steering coefficienct term? If different, how does it differ?

- I am confused of the naming of methods in your paper / results. Could you provide a clearer mapping of prior work methods and their corresponding references? 

- The efficiency analysis: I do not understand why EBM steering would be more efficient than surgical vector steering. EBM uses a call to an external model, while surgical vector steering a single vector ablation. Did you average over different runs? I wonder how much variation you would get in your inference time estimates by including several runs and standard deviation over them. THen the differences would probably be negligible to some methods.
soundness: 2
strengths: - Method. The paper presents a novel, lightweight method for model steering. The results show that the idea has merit, improving a range of prior methods.

- Evaluation. The paper tests the approach on a wide array of benchmarks including safety, false refusal and general capabilities, similar to [Wang et al., 2024](https://arxiv.org/pdf/2410.03415)
summary: The paper presents an new method for steering safety refusal in LLMs. The key idea is to use a lightweight, externally trained Energy-based Model (EBM) to steer internal activations at inference time. This contrasts to existing approaches which learn e.g. a single steering vector that is static. The paper tests the capabilities of Llama and Qwen models on datasets that cover safety, false refusal, general capabilities and multi-turn attacks.
weaknesses: - Presentation. While overall written clearly, I felt the paper could do a better job in the details. For example, methods are not linked to their actual papers, not explicitely. I find it confusing to understand what is meant by which method in the paper. Concrete example:  ", unlike methods such as Surgical Vector and AdaSteer, which show a degradation in safety performance (i.e., higher compliance with harmful requests)". What method is referred to with "Surgical Vector"? If it is Wang et al., (2024), line 137 introduces it as "VA (Vector Ablation) (Wang et al., 2024)", but the naming does not return in the experimental result discussion. 

- The main point of the paper is the introduction of the EBM model, with the motivation to obtain a dynamic steering method, i.e. steering that is not static but adapts on an instance level. However, this key selling point is not evaluated - as results are given as aggregate over benchmarks. So it remains unclear whether the contribution is from the dynamic aspect, or just by using a different way to obtain the steering (an external model). I think the paper would be stronger by including an experiment that shows more directly the differences between a static steering vector and the dynamic steering the paper suggests.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: It is not very clear in the method how the intervention is applied token-wise. 
How exactly is the intervention done? Do you do intervention on all the response tokens at once? If so,  how can we make sure the new tokens still construct meaningful sentences if those tokens are not generated auto-regressively? 

Are there any specific reasons why the capability task performance is kept? Why does simply doing gradient descent on the energy landscape will retain such capability?
soundness: 2
strengths: The Energy-based model is well motivated and works well to separate desired and undesired behaviours. 
The author also extend the evaluation to multi-turn jaibreak datasets, which further test the robustness of the method.
summary: This paper proposes Energy-Driven Steering, which trains energy-based models to separate desired and undesired model behaviour with contrastive learning. The experimental results show improvement in the separation of the refusal and over-refusal representation, while maintaining the general capability.
weaknesses: The main weakness of the paper is the lack of text generation quality evaluation. 
There is no quantitative or qualitative evaluation of the model generation, which raises the concern of the text generation quality.

Such an evaluation is especially important as the intervention is quite strong based on the method. There is also no mechanism in the method to ensure the text generation quality, which requires such an evaluation. It is very likely the gradient descent intervention will interfere the text quality.