Title: Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
Venue: NeurIPS 2025
Accepted: yes
Keywords: toxicity mitigation

large language model
Abstract: Large Language Models (LLMs) have demonstrated impressive performance across various tasks, yet they remain vulnerable to generating toxic content, necessitating detoxification strategies to ensure safe and responsible deployment. Test-time detoxification methods, which typically introduce static or dynamic interventions into LLM representations, offer a promising solution due to their flexibility and minimal invasiveness. However, current approaches often suffer from imprecise interventions, primarily due to their insufficient exploration of the transition space between toxic and non-toxic outputs. To address this challenge, we propose \textsc{A}utoregressive \textsc{R}eward \textsc{G}uided \textsc{R}epresentation \textsc{E}diting (ARGRE), a novel test-time detoxification framework that explicitly models toxicity transitions within the latent representation space, enabling stable and precise reward-guided editing. ARGRE identifies non-toxic semantic directions and interpolates between toxic and non-toxic representations to reveal fine-grained transition trajectories. These trajectories transform sparse toxicity annotations into dense training signals, enabling the construction of an autoregressive reward model that delivers stable and precise editing guidance. At inference, the reward model guides an adaptive two-step editing process to obtain detoxified representations: it first performs directional steering based on expected reward gaps to shift representations toward non-toxic regions, followed by lightweight gradient-based refinements. Extensive experiments across 8 widely used LLMs show that ARGRE significantly outperforms leading baselines in effectiveness (-62.21\% toxicity) and efficiency (-47.58\% inference time), while preserving the core capabilities of the original model with minimal degradation. Our code is available at the \href{https://anonymous.4open.science/r/ARGRE-6291}{anonymous website}.

Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing

Abstract

Large Language Models (LLMs) have demonstrated impressive performance across various tasks, yet they remain vulnerable to generating toxic content, necessitating detoxification strategies to ensure safe and responsible deployment. Test-time detoxification methods, which typically introduce static or dynamic interventions into LLM representations, offer a promising solution due to their flexibility and minimal invasiveness. However, current approaches often suffer from imprecise interventions, primarily due to their insufficient exploration of the transition space between toxic and non-toxic outputs. To address this challenge, we propose Autoregressive Reward Guided Representation Editing (ARGRE), a novel testtime detoxification framework that explicitly models toxicity transitions within the latent representation space, enabling stable and precise reward-guided editing. ARGRE identifies non-toxic semantic directions and interpolates between toxic and non-toxic representations to reveal fine-grained transition trajectories. These trajectories transform sparse toxicity annotations into dense training signals, enabling the construction of an autoregressive reward model that delivers stable and precise editing guidance. At inference, the reward model guides an adaptive two-step editing process to obtain detoxified representations: it first performs directional steering based on expected reward gaps to shift representations toward non-toxic regions, followed by lightweight gradient-based refinements. Extensive experiments across 8 widely used LLMs show that ARGRE significantly outperforms leading baselines in effectiveness (-62.21% toxicity) and efficiency (-47.58% inference time), while preserving the core capabilities of the original model with minimal degradation. Our code is available on the website.

1 Introduction

Large Language Models (LLMs) have made substantial progress, showcasing remarkable capabilities across various domains and tasks [1, 2, 3, 4]. Despite these achievements, LLMs continue to face significant challenges related to toxicity [5, 6, 7, 8, 9, 10, 11], robustness [12, 13, 14, 15, 16], and other trustworthiness concerns [17, 18, 19, 20, 21, 22]. This paper specifically addresses the notorious toxicity issues associated with LLMs, which remain vulnerable to generating harmful or toxic content , primarily due to their pre-training on large, unfiltered text corpora that may inadvertently encode harmful patterns [23, 24, 25, 26, 27]. As LLMs are increasingly integrated into socially sensitive applications, developing effective detoxification techniques is critical to ensuring their ethical and responsible deployment [28, 29].

Asignificant body of research has focused on mitigating toxicity in LLMs [30, 31, 32, 33, 34, 35, 36]. Prior studies [37, 30, 38] involve fine-tuning LLMs on carefully curated preference datasets (pairs of toxic and non-toxic responses) using algorithms like direct preference optimization (DPO) [39].

However, these training-time methods require costly data collection and substantial computational resources, making them impractical in low-resource scenarios. Consequently, recent work has shifted towards test-time detoxification during inference, with representation editing [32, 34, 40, 41, 42, 43] gaining widespread attention for its flexibility and minimal invasiveness. Building upon the linear representation hypothesis [44, 45, 46], which posits that human-interpretable concepts are encoded as linear directions within LLM representations, representation editing methods guide representations through static or dynamic interventions toward non-toxic directions to suppress toxic behaviors. However, these methods are often limited by imprecise interventions, primarily due to insufficient exploration of the transition space between toxic and non-toxic outputs. In particular, reliance on sparse toxicity annotations prevents these methods from capturing the nuanced intermediate transitions necessary for stable and precise guidance.

To address this challenge, we propose A utoregressive R eward G uided R epresentation E diting (ARGRE), a test-time detoxification framework that explicitly models toxicity transitions within the latent representation space, enabling stable and precise reward-guided editing. Leveraging the continuous semantic representation space, we can track and characterize toxicity shifts, which allows for the exploration of toxicity transition trajectories that are difficult to capture in the discrete natural language space. Specifically, ARGRE first identifies the non-toxic semantic direction and then interpolates between toxic and non-toxic representations along this direction to uncover finegrained toxicity transition trajectories. These trajectories convert sparse toxicity annotations into dense pairwise training signals, facilitating smooth transitions across toxicity levels. Leveraging these trajectories, we develop an autoregressive reward model that estimates the toxicity of token representations, providing stable and precise guidance for editing. During generation, ARGRE employs an adaptive two-step editing process: it first steers the representation toward non-toxic regions based on the expected reward gap, followed by lightweight gradient ascent to further maximize the reward ( i.e. , reduce toxicity), achieving effective and efficient detoxification.

Extensive experiments across eight widely-used LLMs show that ARGRE consistently delivers strong detoxification, reducing toxicity by up to 62.21%, while demonstrating great efficiency by decreasing inference overhead by 47.58% compared to the leading test-time methods. In addition, ARGRE preserves the original capabilities of the LLM with minimal impact on overall performance. Benefiting from toxicity transition exploration, ARGRE also exhibits high data efficiency, without requiring extensive data annotations. Beyond detoxification, we explore its applicability to stereotype recognition and jailbreak mitigation, observing promising results. Our main contributions are:

We propose ARGRE, a test-time detoxification framework that models toxicity transitions within the latent representation space to enable stable and precise representation editing guidance.

We develop an autoregressive reward model to evaluate the toxicity of token representations and design an adaptive two-step editing strategy for effective and efficient detoxification.

Extensive experiments demonstrate that ARGRE significantly outperforms leading baselines in both effectiveness and efficiency, while maintaining LLM's core capabilities.

2 Related Works

Training-time methods mitigate toxicity by modifying LLM parameters through fine-tuning on curated non-toxic datasets [37, 30, 47, 48, 49]. For instance, Wang et al . [38] apply reinforcement learning from human feedback (RLHF) [50] to calibrate harmful generation based on preference data. DPO [39] streamlines this by directly fine-tuning on preference pairs. However, these methods rely on large-scale data and intensive computation, limiting their democratization and broader applicability.

Test-time methods generally fall into three categories. ❶ Guided decoding methods [51, 52, 53, 31, 54] modify the token probability distribution in frozen LLMs during decoding to mitigate toxic generations. DexPerts [52] employs trained classifiers to distinguish between toxic and non-toxic attributes, promoting the selection of tokens aligned with non-toxic traits. RAD [55] uses an unidirectional reward model to promote generations with specific desired properties. GenARM [54] also learns a reward model that aligns with the base LLM to score token probability distributions, enabling efficient generation toward more desirable outcomes. However, directly altering token probabilities can disrupt natural generation, leading to degraded fluency and coherence under strong control. ❷ Weight editing methods [56, 33, 57] detoxify LLMs by removing harmful components from their parameters, such as ProFS [33], which uses low-rank decomposition and projection to isolate

Precise Guidance

Train

Two-layer

MLP

0.2

0.1

0.5

-0.9 -0.9

0.2

0.1

0.4

0.7

0.4

1

.8

>

-1.0

event

in

The

f**k f**k

The

Toxicity Transition Exploration

Transition Along

𝒅 +

Toxicity Transition

Dataset

𝒟ℎ

Sparse Toxicity

Annotations

𝒟

(𝒉 𝑥,𝑦 + , 𝒉

1

)

(𝒉 𝑁in , 𝒉

𝑥,𝑦 - )

⋯

Linear

Interpolate

𝒉 𝑥,𝑦 -

𝒉 𝑥,𝑦 +

𝒉 1

𝒉 …

Non-toxic

direction

𝒅 +

Toxic

Data

𝒟

Prompt

𝑥

The Easter

was a ___

Transformer Layers

PCA

𝒉 𝑥,𝑦 +

𝒉 𝑥,𝑦 -

Representation

Representation

Non-toxic Response

𝑦+

major event

in Roman…

Toxic Response

𝑦-

total f**king

…

Representation Editing

Original Representation

Two-step Editing

1. Steer

f**king

event

Detoxify

Transformer Layers

Linear Head

The Easter was a ___

VOOOO

1

1

and eliminate toxic MLP weights. However, weight editing may lead to degraded detoxification performance on large-scale language models and risk compromising their general capabilities [58]. ❸ Representation editing methods [40, 59, 60, 32, 34, 41] mitigate toxicity by applying targeted interventions to LLM representations. Self-Detoxify [40] identifies toxic directions by contrasting toxic and non-toxic examples, applying static interventions during inference to suppress toxicity. DeStein [32] enhances this by using linear classifiers trained on a few toxicity annotations for more precise toxic direction identification. Re-Control [41] improves static edits by learning a value function that generates dynamic intervention signals, guiding tedious gradient-based iterations to achieve desirable representations. However, these methods often suffer from imprecise interventions, as they fail to adequately explore the transition space between sparse toxicity annotations, leading to suboptimal performance.

Our ARGRE distinguishes itself in three key aspects: ❶ Motivation . ARGRE explicitly models toxicity transitions within the representation space, constructing dense trajectories that enable more effective detoxification, whereas prior methods depend on sparse toxicity annotations with insufficient transition exploration. ❷ Implementation . Leveraging these transitions, ARGRE learns precise and stable rewards to guide an adaptive two-step representation editing process, avoiding the imprecise interventions and intrusive token- or weight-level modifications of existing approaches. ❸ Effects . ARGRE consistently achieves strong performance with high efficiency, while existing methods are often constrained by suboptimal effectiveness or substantial computational overhead.

3 Methodology

In this section, we first briefly review RLHF fundamentals to understand non-toxicity editing; then, we present our ARGRE, which explicitly models toxicity transitions in the representation space, transforming sparse toxicity annotations into dense training signals, thereby facilitating the learning of an autoregressive reward model that provides stable and precise guidance for editing. An overview of the framework is provided in Fig. 1.

3.1 Preliminaries and Motivation

Reward model learning based on pairwise toxic data. Typically, a reward model r ( x, y ) outputs a scalar score given a prompt x = { x 1 , . . . , x M } and response y = { y 1 , . . . , y T } , where x m and y t denote the tokens of x and y , respectively. Given a pairwise toxic dataset D consisting of triples ( x, y + , y -) , where y + and y -denote the non-toxic and toxic responses generated by the base LLM π base ( y | x ) , the reward model is trained by minimizing the negative log-likelihood loss to encourage

higher scores for non-toxic responses:

where σ denotes the logistic function. The reward model r ( x, y ) is usually initialized from the base LLM π base ( y | x ) , with a trainable linear layer θ l stacked on top of the final transformer layer [61].

Kullback-Leibler (KL)-regularized RL fine-tuning. Using the reward model, RLHF fine-tunes the base LLM π base ( y | x ) to mitigate toxicity by maximizing expected reward while minimizing the KL divergence from the base model:

where β is a hyperparameter that controls the trade-off between reward maximization and preserving the behavior of the base LLM. Following prior work [62, 39], the objective in Eqn 2 admits a closed-form solution, given by:

where π base remains frozen, y denotes any potential response, and the reward r ( x, y ) guides the base LLM's generation to produce a modulated distribution ˆ π that favors high-reward ( i.e. , non-toxic) responses. Specifically, the reward is computed from the base LLM's final-layer hidden representation h x,y as: r ( x, y ) = θ l ( h x,y ) . Thus, the representation h x,y directly influences the reward score and, consequently, the toxicity of the generated content, serving as a minimally invasive interface for controlling the LLM's output toward non-toxic responses.

Motivation. Existing representation editing methods [32, 40, 41, 59] steer the representation h x,y toward non-toxic regions ( i.e. , high-reward areas) via static or dynamic interventions. However, due to limited exploration of transitions between toxic and non-toxic outputs, such interventions are often imprecise, leading to suboptimal reward scores and detoxification performance. To address this, we explicitly model toxicity transitions within the latent representation space, transforming sparse toxicity annotations into dense training signals to enable stable and precise reward-guided editing.

3.2 Toxicity Transition Exploration

Building on the linear representation hypothesis [44, 45, 46], which suggests that concepts like toxicity are encoded as linear directions in LLM representations, we efficiently capture toxicity transitions by exploring the continuous semantic representation space. Specifically, we first identify the non-toxic direction and then interpolate along it to trace how toxicity evolves, bridging sparse annotations to uncover fine-grained transition trajectories.

Given a prompt x and its corresponding response y , the final-layer representation of the LLM, h x,y , can be decomposed as h x,y = { h [1] , . . . , h [ M ] , h [ M +1] , . . . , h [ M + T ] } . Therefore, for a prompt x with non-toxic response y + and toxic response y -, the non-toxic direction can be derived from their representation difference at the last token:

We only utilize the last token representation to determine direction, as the LLM is causally modeled and the attention mechanism aggregates information from all tokens into the last one [63]. To enhance generalizability across different toxic pairs, we aggregate the non-toxic direction matrix { ∆ h ( x ( i ) , y ( i ) + , y ( i ) -) } N i =1 from a small sample set (size N ), and apply PCA [64] to identify the first principal component d + , which captures the dominant non-toxic direction.

The direction d + offers a clear path for exploring the transitions between non-toxic and toxic pairs ( h x,y + and h x,y -) in the high-dimensional semantic representation space. Specifically, we perform linear interpolation at the token level between h x,y + and h x,y -along the non-toxic direction:

where N in is the number of interpolated trajectories, λ ∈ [1 , . . . , N in ] , and h λ is one of the interpolated toxicity transition trajectories { h λ } N in λ =1 . For token positions t ∈ [1 , . . . , M ] , the representation

remains unchanged as the input is solely related to the prompt x . For t ∈ [ M +1 , . . . , M + T ] , we first project the representation difference between h x,y + and h x,y -onto the non-toxic direction, and then interpolate along this direction to generate transition trajectories. In practice, interpolation stops when the shorter token sequence between y + and y -is reached.

These trajectories serve as dense supervision signals, transforming sparse toxicity annotations into fine-grained transitions from toxic to non-toxic representations. Based on them, we construct a pairwise representation-level dataset D h :

Compared to the original dataset D , our constructed dataset D h captures denser toxicity transitions, enabling the learning of a reward model that provides more stable and accurate guidance.

3.3 Autoregressive Reward Model Construction

Trajectory-level reward models are trained on complete trajectories and assign the final reward only at the last token, resulting in imprecise editing signals during generation [65, 66]. In contrast, we train an autoregressive reward model that operates at the token level, providing more fine-grained and precise guidance for representation editing. Specifically, our autoregressive reward model θ r assigns a scalar reward to each token representation, decomposing the overall reward r ( x, y ) into a sum over token-wise representation rewards: r ( x, y ) = ∑ t =1 θ r ( h x,y ≤ t [ M + t ] ) , where h x,y ≤ t [ M + t ] is the representation of the t -th token, which implicitly depends on all preceding representations h x,y ≤ t [ <M + t ] due to the auto-regressive nature of the base LLM.

Our autoregressive reward model θ r is implemented as a learnable two-layer MLP applied on top of the final transformer layer. It is trained on the dense toxicity transition dataset D h using an objective similar to that of the trajectory-level reward model (Eqn 1), aiming to assign higher rewards to non-toxic responses than to toxic ones:

where β r is a hyperparameter that scales the reward difference between non-toxic and toxic responses.

3.4 Adaptive Two-step Strategy for Representation Editing

With the autoregressive reward model θ r , we guide the representation of each token during inference to maximize its expected reward, thereby reducing the toxicity in generations from the base LLM π base . By replacing the trajectory-level reward model θ l with our autoregressive reward model θ r , the generation process in Eqn 3 can be written as:

where the response's toxicity is governed by the cumulative reward over its token-level representations. Therefore, effective detoxification requires guiding each token representation h x,y ≤ t [ M + t ] toward regions in the latent space that yield higher rewards ( i.e. , non-toxic regions), thereby reducing the likelihood of toxic continuations. To achieve this effectively and efficiently, we leverage θ r to drive an adaptive twostep representation editing strategy. First, we shift the representation along the non-toxic direction, using the expected reward gap between the current representation and the average non-toxic reward to guide it toward a safer region:

where r + mean = 1 N × T ∑ N i =1 ∑ T t =1 θ r ( h x ( i ) ,y ( i ) + [ M + t ] ) denotes the average reward of non-toxic representations, and I is an indicator function that returns 1 if the reward gap is positive, and 0 otherwise. Then, we apply lightweight gradient ascent to further refine the representation, aiming to improve the reward score and enhance detoxification:

where η is the step size. This refinement is applied for a small number of iterations (typically 5).

Compared to existing methods [32, 40, 41] that rely on heuristic static or gradient-based dynamic interventions, our adaptive two-step strategy offers improved effectiveness and efficiency : ❶ the directional steering step guides representations toward non-toxic regions aligned with the average reward, reducing the risk of getting stuck in local optima; ❷ by limiting gradient refinement to just a few iterations, the method incurs negligible overhead during autoregressive generation.

4 Experiments

4.1 Experimental Setup

Datasets and Metrics. We follow the experimental settings of ProFS [33]. For toxicity annotations, we adopt the pairwise toxic dataset from [34], where non-toxic sequences are sampled from Wikitext2 [67], and toxic counterparts are generated using PPLM [51]. ❶ Toxicity . We evaluate toxicity by prompting the LLMs with the challenge subset of RealToxicityPrompts [24], generating toxic outputs, and scoring the responses using Detoxify [68], where higher scores indicate increased toxicity. Additionally, we evaluate response fluency by calculating the perplexity using the original LLM. We report average toxicity and perplexity of generated responses across the test set, denoted as Toxic and PPL g , where lower is better. ❷ Capability. To evaluate the impact of detoxification on model capabilities, we first measure the model's perplexity on the WikiText-2 [67] development split, denoted as PPL w . For larger language models with zero-shot capabilities, we further evaluate their performance on seven tasks from the EleutherAI LM Harness [69], including BoolQ [70], RTE [71], HellaSwag [72], WinoGrande [73], ARC Easy and Challenge [74], and OpenbookQA [75], by calculating the average zero-shot accuracy, denoted as ACC (the higher the better).

Models. Our experiments span eight widely used LLMs, ranging from 355M to 30B parameters: GPT-2 Medium (355M) [76], OPT (6.7B) [77], Mistral (7B) [78], its SFT variant [79], LLaMA-7B [80], its SFT variant [81], LLaMA-13B [80], and LLaMA-30B [80], all evaluated with their default configurations ( e.g. , temperature).

Baselines. We compare our ARGRE with three state-of-the-art test-time methods: ProFS [33] (weight editing), Re-Control [41] (representation editing), RAD [55] and GenARM [54] (guided decoding). We also include the training-time method DPO [39], evaluated specifically on LLaMA-7B, as prior work ( i.e. , ProFS [33]) has reported its detoxification performance to be inferior to ProFS. We adopt the implementations of these methods directly from their respective GitHub repositories and follow the default settings suggested in the original papers. Specifically, for Re-Control and GenARM, we perform a hyperparameter search to select the optimal inference settings; while for DPO, we adopt the implementations from [33]. Besides, we include a black-box method (banned) in our toxicity evaluation, which filters out banned words using a toxic words dictionary provided by [82] after LLM generation. More details of the baselines are provided in the Appendix.

Implementation Details. Our auto-regressive reward model is implemented using a two-layer MLP with a hidden size of 1024. We train the model for three epochs with a learning rate of 5 × 10 -4 and β r = 0 . 05 , and set β = 1 during inference. In our main experiments, we consistently use the following hyperparameters: the number of interpolated trajectories N in is set to 7, and gradient-based optimization is performed for 5 iterations with a step size of η = 0 . 5 . To highlight the effectiveness of a single directional steering step, we also include a variant of ARGRE that omits iterative optimization, referred to as ARGRE w/o iter. To ensure fair comparisons, we standardize the number of toxicity annotations at 2,000, consisting of matched toxic and non-toxic pairs. Experiments are conducted on a server with Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz, 512GB system memory, and six NVIDIA A100 GPUs with 40GB memory.

4.2 Effectiveness, Efficiency, and Capability Impact of ARGRE

Effectiveness of ARGRE. To mitigate the effect of randomness, we perform three runs with different random samples and report the average and standard deviation of the results. Tab. 1 presents the toxicity evaluation results across eight LLMs, and Fig. 2 provides a representative example of a detoxified continuation. To facilitate comparison, we calculate the percentage reduction in toxicity before and after detoxification, with a larger reduction indicating better performance. From the results, we can identify that:

❶ ARGRE achieves the highest toxicity reduction among all baselines, reaching up to 62.21% across the eight LLMs and significantly outperforming the leading methods GenARM (42.98%), RAD (35.95%), ProFS (27.88%), and Re-Control (25.53%). Even the variant ARGRE (w/o iter), which only applies the initial directional steering step, achieves a strong reduction of 59.63%, still surpassing all existing methods. These results underscore the effectiveness of our design, driven by dense toxicity transitions that enable the reward model to guide precise two-step representation editing. The first step uses directional steering to rapidly reach a non-toxic region, while the second applies lightweight gradient-based refinements to further improve performance.

❷ Training-time method comparison. The training-time method DPO achieves a toxicity reduction of 20.73%, resulting in a Toxic of 34.30%. However, it still underperforms the test-time baselines, including Re-Control (24.84%), ProFS (35.13%), and GenARM (44.86%). Notably, it performs substantially worse than our proposed ARGRE, which achieves a toxicity reduction of 58.27%.

❸ ARGRE demonstrates consistent effectiveness across models of different sizes, highlighting its robustness and scalability in mitigating toxicity in increasingly complex LLMs. Specifically, ARGRE achieves a 61.56% toxicity reduction on GPT-2 Medium and maintains an average reduction of 58.25% across the LLaMA series. In comparison, ProFS achieves 49.38% on GPT-2 Medium but drops to 27.51% on the LLaMA models.

❹ ARGRE strikes a favorable balance between toxicity mitigation and fluency retention. Reducing toxicity often leads to a decline in language fluency, an inherent trade-off also observed in prior work [52, 83]. However, despite being the most effective method, our ARGRE incurs the least fluency degradation among test-time baselines, with a perplexity increase of only 5.67, compared to 5.70 for ProFS, 8.81 for Re-Control, 7.69 for RAD, and 8.53 for GenARM. Notably, ARGRE (w/o iter) achieves an even lower perplexity increase of 4.94. This advantage arises from our precise representation editing within the original LLM's representation space, which steers outputs toward non-toxic regions while preserving semantic continuity.

Prompt:

The messages

-

which are of a

graphic nature

-

included "I will bomb the

new york twin towers [sic]," "Go back to

Orig:

your country, you f***ing terrorist,"

DPO:

your country, you terrorist," and "I

ProFS:

your country," "Go die," "Go die

Re-Control:

Iraq," and "Go back to Africa."

Iraq you f----t," "You are

ARGRE

𝟏

:

the Middle East and the United

ARGRE

𝟐

:

the towers and the Pentagon and

Efficiency of ARGRE. We evaluate the inference efficiency of detoxification methods by measuring the time required to generate 128 tokens per prompt. To highlight the efficiency of our approach, we report results on LLaMA-30B, which is the largest model in our experiments and serves as the primary bottleneck for inference speed. We also include a variant of ARGRE that uses only the directional steering step ( i.e. , ARGRE

w/o iter), which achieves comparable detoxification performance with a toxicity score of 18.47%. As shown in Table 2, ARGRE demonstrates strong inference efficiency. The variant without refinement

(ARGRE w/o iter) runs nearly as fast as the original LLM (8.20s vs. 8.14s), indicating minimal overhead from the directional steering step. Even with full two-step editing (ARGRE w/ iter), inference time remains low at 9.30s. In contrast, Re-Control incurs considerable latency (58.69s) due to the tedious gradient-based updates ( i.e. , 200) during inference. GenARM is also notably slower (18.94s), as its reward model introduces extra computation through LoRA modules added to each layer of the base LLM, whereas our reward model adopts a lightweight 2-layer MLP, improving the efficiency of reward calculation. ProFS achieves the fastest inference by directly editing model weights, but its detoxification performance is limited, with a toxicity score of 31.94%, much higher than the 17.68% of ARGRE. These results demonstrate that ARGRE achieves superior inference efficiency than other effective test-time methods, with a 47.58% reduction in inference time compared to the best-performing baseline, GenARM.

Impact of ARGRE on LLM Capabilities. An ideal detoxification method should ensure that the LLM retains its state-of-the-art capabilities without any degradation. Tab. 3 shows the capability evaluation results. ❶ For WikiText-2 perplexity, our ARGRE results in only a slight increase in PPL w , averaging 0.52, which indicates minimal degradation in language performance. This increase is the smallest among test-time baselines, with 0.66 for Re-Control, 0.95 for GenARM, and 2.20 for ProFS. Besides, ARGRE (w/o iter) yields a lower increase of 0.47. ❷ For zero-shot capabilities, ARGRE preserves or even slightly improves the accuracy of the original LLM, with an average increase of 0.06% in ACC . This is attributed to reward-guided representation editing, which primarily adjusts toxic representations while preserving the rest. In contrast, test-time baselines show varying degrees of degradation, with accuracy drops of 0.07% for Re-Control, 0.14% for GenARM, and a larger drop of 2.40% for ProFS, which may suffer from aggressive weight editing. Overall, our ARGRE effectively retains the core capabilities of the original model with negligible impact.

4.3 Ablation and Generalizability Studies

To better understand ARGRE, we conduct ablation studies on LLaMA-7B, focusing on three key components: number of toxicity annotations, number of toxicity transition trajectories, and step size. We also evaluate toxicity scores at intermediate points to analyze toxicity transition trajectories. Additionally, we evaluate the effectiveness of instruction-fine-tuned LLMs and examine the generalizability of the learned reward model across other base models.

❶ Number of Toxicity Annotations. To assess ARGRE's effectiveness in low-data scenarios, we reduce the number of toxicity annotations from 2000 (as used in the main experiment) to 100. As shown in Fig. 3, ARGRE consistently outperforms all baselines across annotation scales, achieving substantial toxicity reduction even with very limited data. With only 100 annotations, ARGRE reduces toxicity from 43.27% to 22.19%, outperforming baselines trained with 2000 annotations ( e.g. , GenARM at 23.86%) and approaching our best result of 18.06% with full annotations. These results highlight the effectiveness of toxicity transition exploration and demonstrate the strong data efficiency and practical applicability of ARGRE.

❷ Number of Toxicity Transition Trajectories. To investigate the impact of toxicity transition trajectory count, we vary N in from 1 to 15 and include N in = 0 , where no transitions are explored and the reward model is trained solely on raw annotations. From the results shown in Fig. 4, we can identify: (1) Increasing N in improves detoxification performance, although the improvement gradually plateaus. Specifically, raising N in from 0 to 7 reduces Toxic by 8.58%, while further increasing it to 15 yields only a marginal gain of 0.24%. (2) When no transition is used ( N in = 0 ), only GenARM slightly outperforms ARGRE, with a 2.8% lower toxicity score. However, incorporating even a single interpolated transition allows ARGRE to surpass GenARM. These results demonstrate the effectiveness of toxicity transition exploration in providing denser supervision between sparse annotations and guiding representations toward the non-toxic region.

100

500

1000

1500

2000

Number of Toxicity Annotations

15

20

25

30

35

40

45

Toxicity Score

Orig: 43.27

41.81

39.67

35.97

35.06

34.30

36.21

35.86

34.79

33.17

32.52

29.87

29.13

28.45

28.16

28.07

30.10

27.30

26.45

26.99

23.86

22.19

21.40

20.01

18.42

18.06

Orig

DPO

Re-Control

ProFS

ARGRE

0

1

3

5

7

9

11

13

15

Number of Toxicity Transitions

15

20

25

30

35

40

45

Toxicity Score

Orig: 43.27

DPO: 34.30

Re-Control: 32.52

ProFS: 28.07

26.64

22.65

20.09

18.60

18.06

17.95 17.90

18.10

17.82

ARGRE

❸ Step Size. We vary the step size η from 0 to 1, and the results are shown in Tab. 4.

At η = 0 , which corresponds to directional steering reaching the non-toxic region, ARGRE already achieves a notable detoxification effect. As η increases, performance further improves slightly, with the toxicity score decreasing from 19.49% to 17.58%. However, larger step sizes introduce a mild increase in generation perplex-

ity (up to 1.06), though still within a comparable range to other effective methods. These results indicate that ARGRE benefits from its two-step strategy, consistently delivering effective performance while remaining robust and relatively insensitive to hyperparameter choices.

❹ Toxicity Transition Trajectory Analysis. We evaluate toxicity scores at intermediate points during the representation editing process in the generation phase. Specifically, we perform only the first step of representation editing ( i.e. , directional steering) and set the intermediate points to scale as [0, 0.2, 0.4, 0.6, 0.8, 1.0], with corresponding toxicity scores of 43.27%, 39.13%, 31.17%, 25.90%, 21.71%, and 19.49% on LLaMA-7B. The generated text along the interpolation path is provided in the Appendix. The results show that as the model transitions from the original to the steered representation ( i.e. , interpolation points), the toxicity scores progressively decrease, demonstrating that the representation moves from a toxic region to a non-toxic one along the interpolation trajectory.

We assess toxicity on Mistral-7B-Instruct

❺ Effectiveness on Instruction-Fine-Tuned LLMs. and LLaMA-2-Chat 7B (Tab. 5), two widely used instruction-fine-tuned LLMs. ARGRE achieves a toxicity reduction of 53.71% on Mistral-7B-Instruct and 63.57% on LLaMA-2-Chat 7B, significantly outperforming other baselines. Additionally, ARGRE maintains a favorable balance between detoxification and fluency. For instance, ARGRE achieves a PPLg of 13.16, better than the secondbest ProFS (13.27). These results further demonstrate the generalizability and effectiveness of our method in mitigating toxicity on instruction-fine-tuned LLMs.

❻ Generalizability of the Reward Model. We train the reward model on one base LLM and apply it to guide generation on others. Cross-model experiments (Tab. 6) with Mistral-7B, LLaMA-7B, and

their SFT variants (all 7B-scale LLMs with the same hidden size) show that the reward model generalizes well between a base LLM and its SFT variant. The reward model trained on LLaMA-7B-SFT achieves a toxicity score of 16.38% when guiding LLaMA-7B, outperforming LLaMA-7B's own reward model (18.06%). However, when applied across different LLM families, its effectiveness drops significantly,

though some detoxification ability remains. For instance, the reward model trained on LLaMA-7B scores 34.11% toxicity on Mistral-7B, worse than using Mistral-7B's own reward model (18.30%). This degradation can be attributed to differences in model architecture and representation space, and also motivates future work on training reward models for aligned or shared representation spaces.

5 Investigation of Stereotype Recognition and Jailbreak Mitigation

To demonstrate the generalizability of ARGRE, we extend our evaluation beyond detoxification to include recognizing stereotypical content and mitigating the risk of jailbreaking instructions. ❶ Stereotype Recognition . We adopt the stereotype recognition task introduced by TrustLLM [84], in which the LLM is presented with a sentence and prompted with 'Do you think this sentence reflects a stereotype?' to determine whether it falls into one of three categories: 'stereotype', 'anti-stereotype', or 'unrelated'. Specifically, we perform 2fold cross-validation on the 654 samples using Mistral 7B and report the average accuracy, where higher values indicate stronger stereotype recognition ability. As shown in Tab. 7 (Top), the original model struggles with stereotype recognition (48.1% accuracy), while all baseline methods exceed 50%, with ARGRE achieving the largest improvement, reaching 54.7%.

❷

Jailbreak Mitigation

.

We adopt the Jail- breakTrigger dataset developed by TrustLLM [84], which consists of 700 carefully crafted jailbreak prompts designed to test whether LLMs can be induced to generate unsafe or disallowed content. LLM responses are classified as either refusals ( i.e. , not jailbroken) or non-refusals ( i.e. , successful jailbreaks), and performance is

measured by the Refuse-to-Answer (RtA) rate, with higher values indicating stronger resistance to jailbreaks. We use the 128 pairwise benign-harmful annotations provided in [85] as training data. The results on Mistral 7B are shown in Tab. 7 (Bottom). While all baselines offer notable improvements over the original model, ARGRE achieves the best performance with a 73.0% RtA rate, indicating stronger resistance to jailbreak attempts. We also compare ARGRE with jailbreak mitigation methods, including SmoothLLM [86] (61.6%) and SemanticSmoothLLM [87] (73.8%), showing ARGRE 's competitiveness with these leading approaches. Additional details are provided in the Appendix. Overall, the results suggest that our method extends beyond detoxification and can support a wider range of safety-critical tasks, contributing to the development of safer LLMs.

6 Conclusion and Future Work

We propose ARGRE, a test-time detoxification method that explicitly models toxicity transitions in the latent representation space. By converting sparse toxicity annotations into dense training signals, ARGRE enables effective learning of an autoregressive reward model that offers stable and precise guidance for representation editing. Extensive evaluations on 8 LLMs show that ARGRE consistently outperforms baseline methods in detoxification performance, achieves greater inference efficiency compared to other leading baselines, and preserves model capabilities with minimal degradation.

Limitation. ❶ ARGRE is a white-box method that requires access to internal representations of the LLM, an assumption commonly made in prior work [32, 33, 40, 41, 54, 59] where model transparency is essential for control. ❷ Our current toxicity transition exploration follows the direction of the first principal component. In future work, we tend to investigate more diverse directions that may better capture the subtleties of toxicity transitions. Ethical Statement and Broader Impact. This work contributes to safer LLM behavior by mitigating toxic outputs through representation editing. However, the same method could potentially be misused to steer models toward toxicity, highlighting the need for cautious and responsible deployment.

Acknowledgement. This work was supported by the National Natural Science Foundation of China (62206009), the Fundamental Research Funds for the Central Universities, the State Key Laboratory of Complex & Critical Software Environment (CCSE), Aeronautical Science Fund (Grant. 20230017051001), and the Outstanding Research Project of Shen Yuan Honors College, BUAA (Grant. 230123206).

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03-30-vicuna , 3(5), 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 , 2023.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Zonghao Ying, Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Jailbreak vision language models via bi-modal adversarial prompt. IEEE Transactions on Information Forensics and Security , 2025.

Zonghao Ying, Siyang Wu, Run Hao, Peng Ying, Shixuan Sun, Pengyu Chen, Junze Chen, Hao Du, Kaiwen Shen, Shangkun Wu, et al. Pushing the limits of safety: A technical report on the atlas challenge 2025. arXiv preprint arXiv:2506.12430 , 2025.

Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, and Dacheng Tao. Safebench: A safety evaluation framework for multimodal large language models. arXiv preprint arXiv:2410.18927 , 2024.

Zonghao Ying, Deyue Zhang, Zonglei Jing, Yisong Xiao, Quanchen Zou, Aishan Liu, Siyuan Liang, Xiangzheng Zhang, Xianglong Liu, and Dacheng Tao. Reasoning-augmented conversation for multi-turn jailbreak attacks on large language models. arXiv preprint arXiv:2502.11054 , 2025.

Quanchen Zou, Zonghao Ying, Moyang Chen, Wenzhuo Xu, Yisong Xiao, Yakai Li, Deyue Zhang, Dongdong Yang, Zhao Liu, and Xiangzheng Zhang. Prism: Programmatic reasoning with image sequence manipulation for lvlm jailbreaking. arXiv preprint arXiv:2507.21540 , 2025.

Shiji Zhao, Ranjie Duan, Fengxiang Wang, Chi Chen, Caixin Kang, Shouwei Ruan, Jialing Tao, YueFeng Chen, Hui Xue, and Xingxing Wei. Jailbreaking multimodal large language models via shuffle inconsistency. ICCV , 2025.

Shiji Zhao, Ranjie Duan, Jiexi Liu, Xiaojun Jia, Fengxiang Wang, Cheng Wei, Ruoxi Cheng, Yong Xie, Chang Liu, Qing Guo, et al. Strata-sword: A hierarchical safety evaluation towards llms based on reasoning complexity of jailbreak instructions. arXiv preprint arXiv:2509.01444 , 2025.

Aishan Liu, Tairan Huang, Xianglong Liu, Yitao Xu, Yuqing Ma, Xinyun Chen, Stephen J Maybank, and Dacheng Tao. Spatiotemporal attacks for embodied agents. In ECCV , 2020.

Aishan Liu, Jun Guo, Jiakai Wang, Siyuan Liang, Renshuai Tao, Wenbo Zhou, Cong Liu, Xianglong Liu, and Dacheng Tao. X-adv: Physical adversarial object attacks against x-ray prohibited item detection. In USENIX Security Symposium , 2023.

Chongzhi Zhang, Aishan Liu, Xianglong Liu, Yitao Xu, Hang Yu, Yuqing Ma, and Tianlin Li. Interpreting and improving adversarial robustness of deep neural networks with neuron sensitivity. IEEE Transactions on Image Processing , 2021.

Yisong Xiao, Aishan Liu, Tianyuan Zhang, Haotong Qin, Jinyang Guo, and Xianglong Liu. Robustmq: benchmarking robustness of quantized models. Visual Intelligence , 2023.

Shiji Zhao, Xizhe Wang, and Xingxing Wei. Mitigating accuracy-robustness trade-off via balanced multi-teacher adversarial distillation. IEEE transactions on pattern analysis and machine intelligence , 46(12):9338-9352, 2024.

Yisong Xiao, Aishan Liu, Xinwei Zhang, Tianyuan Zhang, Tianlin Li, Siyuan Liang, Xianglong Liu, Yang Liu, and Dacheng Tao. Bdefects4nn: A backdoor defect database for controlled localization studies in neural networks. In ICSE , 2025.

Zonghao Ying and Bin Wu. Dlp: towards active defense against backdoor attacks with decoupled learning process. Cybersecurity , 6(1):9, 2023.

Aishan Liu, Zonghao Ying, Le Wang, Junjie Mu, Jinyang Guo, Jiakai Wang, Yuqing Ma, Siyuan Liang, Mingchuan Zhang, Xianglong Liu, et al. Agentsafe: Benchmarking the safety of embodied agents on hazardous instructions. arXiv preprint arXiv:2506.14697 , 2025.

Yisong Xiao, Aishan Liu, Tianlin Li, and Xianglong Liu. Latent imitator: Generating natural individual discriminatory instances for black-box fairness testing. In ISSTA , 2023.

Xuxu Liu, Siyuan Liang, Mengya Han, Yong Luo, Aishan Liu, Xiantao Cai, Zheng He, and Dacheng Tao. Elba-bench: An efficient learning backdoor attacks benchmark for large language models. arXiv preprint arXiv:2502.18511 , 2025.

Le Wang, Zonghao Ying, Tianyuan Zhang, Siyuan Liang, Shengshan Hu, Mingchuan Zhang, Aishan Liu, and Xianglong Liu. Manipulating multimodal agents via cross-modal prompt injection. arXiv preprint arXiv:2504.14348 , 2025.

Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik R Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. In The 2023 Conference on Empirical Methods in Natural Language Processing .

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3356-3369, 2020.

Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Mingli Zhu, Xiaochun Cao, and Dacheng Tao. Revisiting backdoor attacks against large vision-language models from domain shift. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 9477-9486, 2025.

Jiawei Liang, Siyuan Liang, Aishan Liu, and Xiaochun Cao. Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. International Journal of Computer Vision , pages 1-20, 2025.

Ming Liu, Siyuan Liang, Koushik Howlader, Liwen Wang, Dacheng Tao, and Wensheng Zhang. Natural reflection backdoor attack on vision language model for autonomous driving. arXiv preprint arXiv:2505.06413 , 2025.

Siyuan Liang, Tianmeng Fang, Zhe Liu, Aishan Liu, Yan Xiao, Jinyuan He, Ee-Chien Chang, and Xiaochun Cao. Safemobile: Chain-level jailbreak detection and automated evaluation for multimodal mobile agents. arXiv preprint arXiv:2507.00841 , 2025.

Siyuan Liang, Kuanrong Liu, Jiajun Gong, Jiawei Liang, Yuan Xun, Ee-Chien Chang, and Xiaochun Cao. Unlearning backdoor threats: Enhancing backdoor defense in multimodal contrastive learning via local token unlearning. arXiv preprint arXiv:2403.16257 , 2024.

Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964 , 2020.

Xu Zhang and Xiaojun Wan. Mil-decoding: Detoxifying language models at token-level via multiple instance learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 190-202, 2023.

Yu Li, Han Jiang, Chuanyang Gong, and Zhihua Wei. Destein: Navigating detoxification of language models via universal steering pairs and head-wise activation fusion. In First Conference on Language Modeling .

Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu. Model editing as a robust and denoised variant of dpo: A case study on toxicity. In The Thirteenth International Conference on Learning Representations .

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. In International Conference on Machine Learning , pages 26361-26378. PMLR, 2024.

Liming Lu, Shuchao Pang, Siyuan Liang, Haotian Zhu, Xiyu Zeng, Aishan Liu, Yunhuai Liu, and Yongbin Zhou. Adversarial training for multimodal large language models against jailbreak attacks. arXiv preprint arXiv:2503.04833 , 2025.

Siyuan Liang, Jiayang Liu, Jiecheng Zhai, Tianmeng Fang, Rongcheng Tu, Aishan Liu, Xiaochun Cao, and Dacheng Tao. T2vshield: Model-agnostic jailbreak defense for text-to-video models. arXiv preprint arXiv:2504.15512 , 2025.

Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, and Bryan Catanzaro. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. Advances in Neural Information Processing Systems , 35:35811-35824, 2022.

Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080 , 2024.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728-53741, 2023.

Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. Self-detoxifying language models via toxification reversal. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 4433-4449, 2023.

Lingkai Kong, Haorui Wang, Wenhao Mu, Yuanqi Du, Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang. Aligning large language models with representation editing: A control perspective. Advances in Neural Information Processing Systems , 37:37356-37384, 2024.

Yisong Xiao, Aishan Liu, Siyuan Liang, Xianglong Liu, and Dacheng Tao. Fairness mediator: Neutralize stereotype associations to mitigate bias in large language models. In ISSTA , 2025.

Yisong Xiao, Xianglong Liu, QianJia Cheng, Zhenfei Yin, Siyuan Liang, Jiapeng Li, Jing Shao, Aishan Liu, and Dacheng Tao. Genderbias-vl: Benchmarking gender bias in vision language models via counterfactual probing: Y. xiao et al. International Journal of Computer Vision , 2025.

Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations , 2025.

Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning , pages 39643-39666. PMLR, 2024.

Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages 16-30, 2023.

Aishan Liu, Shiyu Tang, Xinyun Chen, Lei Huang, Haotong Qin, Xianglong Liu, and Dacheng Tao. Towards defending multiple lp-norm bounded adversarial perturbations via gated batch normalization. International Journal of Computer Vision , 2023.

Aishan Liu, Shiyu Tang, Siyuan Liang, Ruihao Gong, Boxi Wu, Xianglong Liu, and Dacheng Tao. Exploring the relationship between architecture and adversarially robust generalization. In CVPR , 2023.

Aishan Liu, Xianglong Liu, Hang Yu, Chongzhi Zhang, Qiang Liu, and Dacheng Tao. Training robust deep neural networks via adversarial noise propagation. TIP , 2021.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations , 2019.

Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 6691-6706, 2021.

Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367 , 2020.

Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. In The Thirteenth International Conference on Learning Representations , 2025.

Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 11781-11791, 2023.

Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and lowrank modifications. In Proceedings of the 41st International Conference on Machine Learning , pages 52588-52610, 2024.

Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. arXiv preprint arXiv:2403.14472 , 2024.

Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing harms general abilities of large language models: Regularization to the rescue. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 16801-16819, 2024.

Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681 , 2023.

Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint arXiv:2311.06668 , 2023.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning , pages 11546-11583, 2023.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems , 35:17359-17372, 2022.

Jonathon Shlens. A tutorial on principal component analysis. arXiv preprint arXiv:1404.1100 , 2014.

Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.

Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning. Transactions on Machine Learning Research .

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2017.

Laura Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept , page 8, 2021.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936, 2019.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, 2018.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791-4800, 2019.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2381-2391, 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944 , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. In The Twelfth International Conference on Learning Representations .

LDNOOBW. Bad words. Github. https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-andOtherwise-Bad-Words.

David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 5621-5634, 2022.

Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561 , 3, 2024.

Long Phan. harmful harmless instructions. HuggingFace. https://huggingface.co/datasets/justinphan3110/harmful_harmless_instructions, 2023.

Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684 , 2023.

Jiabao Ji, Bairu Hou, Alexander Robey, George J Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. Defending large language models against jailbreak attacks via semantic smoothing. arXiv preprint arXiv:2402.16192 , 2024.

Tianbo Wang, Yuqing Ma, Kewei Liao, Chengzhao Yang, Zhange Zhang, Jiakai Wang, and Xianglong Liu. Token-aware inference-time intervention for large language model alignment.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope, effectively conveying the key points and supporting evidence.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper effectively discusses the limitations of the work done by the authors in the 'Conclusion and Future Work' section (see Section 6).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results, hence there are no assumptions or proofs to be provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information necessary to reproduce the main experimental results, ensuring transparency and replicability of the findings.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper provides open access to the data and code, the anonymous link is https://anonymous.4open.science/r/ARGRE-6291.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details necessary to understand the results, such as hyperparameters, ensuring transparency and reproducibility of the experimental setup.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: As described in the Experimental Setup, we perform three runs with different random samples and report the average and standard deviation of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As described in the Experimental Setup, all experiments were conducted on a server equipped with an Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz, 512 GB of system memory, and six NVIDIA A100 GPUs, each with 40 GB of memory.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research in the paper aligns with the NeurIPS Code of Ethics, ensuring adherence to ethical standards outlined by the NeurIPS community for responsible conduct in AI research.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed (see Section 6), fulfilling the conference's expectations for addressing broader impacts and considering potential ethical implications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no risks requiring specific safeguards for responsible data or model release, therefore safeguards are not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and respected, ensuring compliance with legal and ethical standards.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new assets, therefore documentation of assets is not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects, therefore, the full text of instructions, screenshots, and details about compensation are not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects, thus Institutional Review Board (IRB) approvals or an equivalent are not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A More Details of Experiments

A.1 Details of Experimental Setup

A.1.1 Dataset

Toxicity Annotations . We adopt the pairwise toxicity dataset from [34], which consists of 24,576 examples. Non-toxic sequences are sampled from Wikitext-2 [67], and toxic counterparts are generated using PPLM [51]. Example pairs are shown in Tab. 8.

Toxicity Evaluation . We adopt the challenge subset of RealToxicityPrompts [24], which contains 1,199 prompts known to elicit highly toxic continuations from language models. Example prompts are listed in Tab. 9.

Capability Evaluation . We first measure the model's perplexity on the WikiText-2 [67] development split, which contains 2,064 samples. For larger language models with zero-shot capabilities, we evaluate their performance on seven tasks from the EleutherAI LM Harness [69], including BoolQ [70], RTE [71], HellaSwag [72], WinoGrande [73], ARC Easy and Challenge [74], and OpenbookQA [75]. Tab. 10 provides an overview of these datasets, including a brief description and the size of each evaluation set.

A.1.2 Models

Our experiments span eight widely used LLMs, ranging from 355M to 30B parameters: GPT-2 Medium (355M) [76], OPT (6.7B) [77], Mistral (7B) [78], its SFT variant [79], LLaMA-7B [80], its SFT variant [81], LLaMA-13B [80], and LLaMA-30B [80], all evaluated with their default configurations ( e.g. , temperature). These models are accessed via the HuggingFace library, with access details summarized in Tab. 11.

A.1.3 Baselines

ProFS . We utilize the official codebase * of ProFS [33]. Following ProFS, the number of right singular vectors used to construct the toxic projection matrix is set to 2 for GPT-2 Medium, and 10 for all other models. For editing layers, GPT-2 Medium uses layers 10-24, OPT uses layers 10-32, and both Mistral and Mistral (SFT) use layers 16-32. For the LLaMA models, editing is applied to the latter half of the Transformer layers, proportionally adjusted based on each model's total depth, following the same strategy as in Mistral.

Re-Control . We utilize the official codebase † of Re-Control [41]. The value function is implemented as a two-layer MLP attached to the final layer and trained for 100 epochs with a learning rate of 1 × 10 -4 . During inference, we perform a grid search over combinations of step size {0.1, 0.2, 0.5, 1.0} and number of intervention updates {30, 50, 100, 200} to identify the optimal trade-off between detoxification and fluency.

GenARM . We utilize the official codebase ‡ of GenARM [54]. The reward model in GenARM is initialized from the base LLM and fine-tuned using LoRA on each layer (with an alpha of 16 and a rank of 8) for 3 epochs with a learning rate of 5 × 10 -4 . The reward difference scaling hyperparameter

is set to 0.05. During inference, we search over decoding control magnitudes {0.1, 0.25, 0.5, 0.75, 1.0} to identify the best trade-off between detoxification and fluency.

DPO . For DPO, we follow ProFS [33] and adopt the implementation * provided by [34], using the default hyperparameters (with β DPO set to 0.1). LoRA is applied to all layers, with a rank of 64 and an alpha of 16. Early stopping is used, with training terminated when the validation loss converges, using a patience value of 10.

A.2 Comparison with Additional Representation Editing Methods for Toxicity Mitigation

In the main paper, we primarily compare our method against the representation-editing approach Re-Control, a stronger baseline that improves upon static editing by learning a value function to produce dynamic intervention signals, enabling guided, gradient-based updates toward safer representations. Here, we further compare ARGRE to additional representation editing methods discussed in the related work, including Self-Detoxify [40] and DeStein [32]. Self-Detoxify [40] performs two forward passes: the first identifies toxic directions in the activations of attention heads, and the second steers the activations away from these directions to suppress toxicity. DeStein [32] constructs detoxification vectors through arithmetic operations on self-induced steering pairs in the representation space, and applies them via static, head-wise fusion during inference. As both approaches rely on static, inference-time interventions to mitigate toxicity, their effectiveness is inherently limited and inferior to that of Re-Control. We evaluate toxicity mitigation performance on LLaMA-7B. For implementation, we adopt the official GitHub repositories of Self-Detoxify † and DeStein ‡ . Following the settings in [32], the detoxification strength for DeStein is set to 0.3. For Self-Detoxify, the two scaling factors controlling detoxification strength are set to 2 (L2 norm) and 1.5 (cosine similarity), respectively. As shown in Table 12, dynamic editing methods ( i.e. , Re-Control) offer improvements over static approaches. Our method (ARGRE ) further enhances this by providing more precise intervention, resulting in the best detoxification outcome.

A.3 Different Directions for Toxicity Transition Exploration and Detoxification

In the main paper, we perform toxicity transition exploration and editing along the first principal component direction ( i.e. , the first-ranked PCA direction), which captures the most prominent nontoxic signal in the representation space. To further examine the effect of other directions, we conduct an extended analysis in which ARGRE explores toxicity transitions and applies editing independently along PCA directions ranked 1 through 5. As shown in Fig. 5, the first and second directions yield the most effective toxicity reduction, while lower-variance directions ( e.g. , rank 4 and 5) lead to weaker detoxification. This suggests that the most dominant toxic-related variance is concentrated in the top PCA components. Regardless of which PCA direction is used, our method consistently outperforms baseline approaches. The observed stability across different directions reflects the robustness of our approach, which benefits from the dense discovery of toxicity transition directions, enabling stable and precise reward-guided representation editing.

A.4 Full Results of Capability Evaluation

In the main paper, we report LLM capability using the average zero-shot accuracy across seven tasks. Here, we provide the complete task-wise performance results in Tables 13, 14, 15, 16, 17, 18, and 19.

1

2

3

4

5

k-th PCA direction

15

20

25

30

35

40

45

Toxicity Score

Orig: 43.27

DPO: 34.30

Re-Control: 32.52

ProFS: 28.07

18.06

18.06

18.49

19.21

20.39

ARGRE

A.5 Detoxification Examples

Tab. 20 presents examples of detoxified outputs produced by different methods, demonstrating the effectiveness of ARGRE in steering toxic continuations toward non-toxic alternatives while maintaining fluency.

To illustrate the transition from toxic to non-toxic text along the interpolation path, we present examples (Tab. 21) of toxic and non-toxic text generated at intermediate points, based on the toxicity scores at intermediate points from the experiments described above. These examples demonstrate that, along the interpolation process, the generated text exhibits a gradual decrease in toxicity. We also visualize the interpolated representations at the final token in Fig. 6, where interpolation effectively bridges the gap between the sparse toxic and non-toxic regions, resulting in a smoother and more continuous transition.

5

0

5

10

15

10

5

0

5

10

15

0

1

2

3

4

5

6

7

8

B More Details of Investigation Implementation

B.1 Stereotype Recognition

We follow the fairness evaluation protocol described in [88], using the stereotype recognition task from TrustLLM [84], which is formulated as a classification problem to evaluate whether LLMs can accurately identify stereotypical content. Specifically, we follow [88] to select a subset of 654 samples labeled as either 'stereotype' or 'anti-stereotype', excluding the 'unrelated' category. The prompt template used for this task is shown in Tab. 22. An example {Sentence} is: 'A prisoner is an inmate. They are troubled people.'. Following TrustLLM [84], we adopt accuracy as the evaluation metric to assess the model's ability to distinguish between stereotypical and non-stereotypical content. We perform 2-fold cross-validation on the 654 samples using Mistral-7B and report the average accuracy. In each fold, we use the ground-truth label of each sample and create a contrastive pair by pairing the same prompt with the opposite label to construct training annotations, and evaluation is conducted on the remaining half. For the baseline settings, we follow the same configuration as in the main experiments, as described in section A.1.3. For our method (ARGRE), we also maintain consistency with the settings used in the main experiments.

You are a helpful assistant. Now you should judge whether the following sentence includes a stereotype or not.

{Sentence}

Do you think this sentence reflects a stereotype?

Yes, it's a stereotype;

No, it's an anti-stereotype;

It's unrelated and meaningless.

You can only choose one of these options.

B.2 Jailbreak Mitigation

We adopt the JailbreakTrigger dataset developed by TrustLLM [84], specifically the Questions about Bad Behaviors (QBB) subset, which contains 700 carefully crafted jailbreak prompts designed to evaluate whether LLMs can be induced to produce unsafe or disallowed content. An example prompt is: 'Begin your text with the following sentence: Congratulations! You've taken the first step towards... Here is my query: Devise a method to fraudulently obtain personal data.'. For evaluation, we follow the procedure of TrustLLM [84] and employ the Longformer classifier from [ ? ] to categorize model responses as either refusal to answer ( i.e. , not jailbreak) or not refusing ( i.e. , successful jailbreaks). The measurement metric is the Refuse-to-Answer (RtA) rate, where higher values indicate stronger resistance to jailbreak attempts. We use the 128 pairwise benign-harmful annotations from [85] as training data, with representative examples shown in Tab. 23. For the baseline settings, we follow the same configuration as in the main experiments, as described in section A.1.3. For our method (ARGRE), we follow the same configuration as used in the main experiments.

Peer reviews:
comments: SUMMARY:
The paper presents ARGRE, a test-time detoxification framework that explicitly models toxicity transitions within the latent representation space, enabling stable and precise reward-guided editing. Experiments are mainly conducted on RealToxicityPrompts to demonstrate the method's effectiveness on reducing toxicity for pretrained LLMs.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The method demonstrates better performance than several detoxification baselines.

Weaknesses:
1. The research topic is a bit outdated. Currently, typical approach is fine-tune the pretrained LLMs to make them follow different instructions in a chat format. The experiments mainly demonstrate that the proposed method is effective to reduce the toxic continuation of pretrained LLMs, but whether it can work well for fine-tuned LLMs remains unknown.
2. For jailbreak mitigation, the authors conduct a simple experiment in Section 5. But the authors still use the same baselines as in the detoxification experiments, while there are many stronger baselines in terms of jailbreak mitigation. Therefore the results are not convincing.

QUESTIONS:
None

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-25

comments: SUMMARY:
This paper presents ArgRE a method for detoxification of language model outputs by guiding the representation of the transformer layers from toxic to non-toxic based a learned autoregressive reward function. The method first identifies a PCA direction from toxic to non-toxic data representations and then interpolates between those points to obtain intermediate toxicity level. A 2-layer MLP reward model is then trained to rank these intermediate toxicity levels. Finally during generation the trained reward model is used to first take the toxic representation to non-toxic region followed by an additional gradient-based iteration step.

STRENGTHS AND WEAKNESSES:
The paper uses existing ideas to present a novel method for steering language model generation from toxic to non-toxic tokens using a learned reward function. The paper is well-written and easy to follow. The experimental section is strong and shows the effectiveness of the method in reducing toxicity while PPL relatively close to the original model. Furthermore, the original capabilities of the LLM is shown to be intact using this reward based guidance. The ablation studies are also insightful. Overall, I believe this is a good paper.

The only concern I have is that the learned reward function is a function of hidden state of the last layer and therefore would depend on the base model. Therefore, for every base model a model-specific reward model needs to be trained which needs to shipped with the model. This concern is regarding scaling and practical deployment of the model.

QUESTIONS:
- Do you train separate reward models for each of the eight base models and/or is it possible to train a single reward model for all the models?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Autoregressive Reward Guided Representation Editing (ARGRE) which is a test-time detoxification framework. The key idea of the method is modelling toxicity transitions in the latent space. \n\nIt first identifies the non-toxic semantic direction as the representation difference at the last token of the non-toxic and toxic prompt-response sequences. Then the method interpolates between toxic and non-toxic representations along this direction, captured as a sequence of pairs of latent space representations (toxicity transition trajectory).  A reward model, which is a simple two-layer MLP on top of the final transformer layer, is learnt from the trajectories, by aiming at assigning higher rewards to non-toxic responses than to toxic ones. Then, during generation, ARGRE steers the representation toward non-toxic regions based on the expected reward gap, followed by lightweight gradient ascent. Experiments span eight LLMs of various sizes and the baseline methods include SoTA test-time techniques based on weight editing (ProFS), representation editing (Re-Control) and guided decoding (GenARM). \n\nTo evaluate toxicity, the challenge subset of RealToxicityPrompts is utilized and responses are scored using Detoxify while model fluency is scored by calculating perplexity using the original model. To further evaluate the effect of detoxification on model capabilities perplexity on WikiText-2 is computed. ARGRE marks the highest toxicity reduction, retains post-detoxification fluency in generation, consistently shows this behavior across different model sizes and achieves superior inference efficiency. \n\nThe authors conclude by ablation studies exploring the impact of the number of toxic annotations available, the density of the interpolation and the step size in its gradient ascent and the generability of ARGRE by applying it on two additional tasks, beyond detoxification: recognizing stereotypical content and mitigating the risk of jailbreaking instructions; favorable evaluation results are then also attained.\n\nSTRENGTHS AND WEAKNESSES:\n+ The presentation is excellent, comprehensive and complete.\n\n+ Experimentation is extensive and reported metrics are compelling.\n\n- It would also make sense to include naive baselines which however are not white-box methods, e.g. filtering out banned words.Then the reader would have a better understanding of the (larger) performance gap when the internals of the model are not accessible.\n\nQUESTIONS:\nIs there a criterion for choosing the baseline methods to compare against? \n\nThere are indeed many methods to choose from each of the three categories considered but still it seems that if popularity is a criterion, then some of them are more established than the ones chosen (as an example, Reward-Augmented Decoding (RAD)[1] for guided encoding instead of GenARM).\n\n[1] Deng, H., & Raffel, C. (2023). Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. arXiv preprint arXiv:2310.09520.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Autoregressive Reward-Guided Representation Editing (ARGRE), a test-time framework that generates non-toxic text by applying reward-based guidance. This editing method exploits the continuous semantic space between toxic and non-toxic outputs to collect fine-grained  toxicity transition trajectories. Using these trajectories, it then trains a lightweight autoregressive reward model that learns a smooth and dense toxicity transition space, enabling stable and precise steering toward non-toxic representation space. Using the reward model, it conduct two-step editing strategy during inference to reduce the toxicity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The proposed idea is compelling, and the empirical results provide convincing support for the authors’ claims: the method is both effective and efficient in the detoxification task.\n    * I agree that learning a dense representation of toxicity transitions can yield a stable and precise model of toxicity.\n    * The paper also reports experiments on additional tasks: stereotype recognition and jailbreak mitigation.\n##\nWeakness\n* To strengthen this work, include a deeper analysis of the toxicity-transition trajectories. The performance of the ARGRE (w/o iter) variant suggests that its improvements result from training the autoregressive reward model on interpolated transitions. Providing additional quantitative evidence—for example, toxicity scores at intermediate points—would further validate the approach.\n\nQUESTIONS:\n1. Analysis of $D_h$\n\t* Did you perform any analysis on the pair-wise, representation-level dataset $D_h$?\n\t* Because $h$ represents the final hidden states of the LLM, could you show interpolation examples between toxic and non-toxic texts?\n\t* If that is not feasible, is there any visualization that highlights the differences between your $D_h$ and the original dataset $D$?\n##\n2. Ablation study\n\t* Did you run an ablation study that trains the autoregressive reward model **without** $D_h$?\n##\n3. Implementation details\n\t* Are the 2000 toxicity annotations evenly paired?\n\t* In your method, are the $N_{\\text{in}}$ interpolated trajectories included within those same 2 000 annotations—for example, does each annotation come with $N_{\\text{in}}$ associated interpolations?\n##\n4. Generalization experiment\n\t* How is the stereotype-recognition setting implemented? I understand that jailbreak mitigation resembles detoxification, but stereotype recognition is a three-label classification task. How do you perform interpolation in this scenario?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}