Title: Improved Representation Steering for Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Steering

Interpretability

Mechanistic Interpretability
Abstract: Steering methods for language models (LMs) seek to provide fine-grained and interpretable control over model generations by variously changing model inputs, weights, or representations to adjust behavior. Recent work has shown that adjusting weights or representations is often less effective than steering by prompting, for instance when wanting to introduce or suppress a particular concept. We demonstrate how to improve representation steering via our new Reference-free Preference Steering (RePS), a bidirectional preference-optimization objective that jointly does concept steering and suppression. We train three parameterizations of RePS and evaluate them on AxBench, a large-scale model steering benchmark. On Gemma models with sizes ranging from 2B to 27B, RePS outperforms all existing steering methods trained with a language modeling objective and substantially narrows the gap with prompting -- while promoting interpretability and minimizing parameter count. In suppression, RePS matches the language-modeling objective on Gemma-2 and outperforms it on the larger Gemma-3 variants while remaining resilient to prompt-based jailbreaking attacks that defeat prompting. Overall, our results suggest that RePS provides an interpretable and robust alternative to prompting for both steering and suppression.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies representation steering of language models (LMs) where the goal is to introduce or suppress a given concept. The paper proposes a novel bidirectional preference-optimization objective called Reference-free Preference Steering (RePS). It extends SimPO and removes the constraint of staying close to the reference model. The objective is bidirectional in the sense that is the sum of two objectives: likelihood differences for positive steering and for negative steering. The proposed objective is used in conjunction with different intervention methods such as steering vectors, LoReFT, LoRA and evaluated on AXBENCH. It is compared to other baseline methods like language modeling and BiPO. It outperforms the other baselines but still performs worse than prompting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The method is evaluated with various intervention methods and models of different sizes. Moreover, it is compared with other strong baselines. \n- RePS outperforms the baselines besides prompting on AXBENCH almost across all model sizes and values of r. \n- The paper is mostly self-contained in the sense that the extensive background and prior methods required to truly situate the contributions are all explained in the paper. This might help people not very familiar with the field appreciate the proposed method. Moreover the paper is structured and written in a clear manner without unnecessary verbosity. There are a few exceptions mentioned in the Weaknesses section below. \n\nWeaknesses:\n- The method introduces significant overhead but seems to still perform worse than prompting. \n- The techniques used in section 5.4 for the attack experiments might not be representative of prompting. There are good prompting techniques tailored to jailbreaking that would make decent baselines but are not mentioned in that section.\n\nminor comments:\nIt might be helpful to add error bars to the tables\n\nQUESTIONS:\n1 - How invasive are these interventions? Do they affect the other capabilities of the model? Does the model still perform similarly on common LM benchmarks?\n\n2 - How do you select the ranges of hyperparameters that you search on? Do you use the same ranges for combinations of intervention methods/objective?\n\n3 - How extensive was your prompt engineering for the attack experiment in section 5.4?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Reference-free Preference Steering (RePS) for controlling LLM outputs via internal representation manipulation. The core idea of RePS is that when training the intervention $\\Phi$, utilizing SimPO inspired training objective as opposed to BiPO that uses reference model likelihood. The method is evaluated on AxBench with Gemma models, across 3 intervention types, steering vectors, LoRA and ReFT. The method outperforms existing methods in steering and supression and is effective for larger model steering.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I think the work is a well executed, solid research paper (reserving the fact that I am not familiar with representation steering literature).\n\n- The idea of RePS is simple and thus broadly applicable. It’s motivation is well grounded on existing discussions on preference optimization, thus can be intuitively understood. Yet its empirical performance seems to be substantially better than comparable methods in both concept steering and suppression .\n- Experimental design is robust, using AxBench across a range of model scales. Inclusion of multiple intervention types enhances reliability of results.\n- Writing is well-structured, with clear exposition of preliminaries, methodology, and mathematical formulation of RePS and the intervention types used.\n- The paper lacks a detailed discussion of the limitations of RePS, particularly regarding the underperformance of LoRA and ReFT compared to SV on larger models. This raises questions about the generalizability of RePS across different intervention types.\n- The hyperparameter search (72 runs for smaller models, 168 for larger ones) suggests significant computational demands, which limits practical application of RePS for resource-constrained setting.\n\nMinor Typo: in equation (3), in the second term, $y^l | x$ should be changed to $y | x$\n\nQUESTIONS:\nQ1. Clarify the underperformance of LoRA and ReFT: The results show that SV consistently outperforms LoRA and ReFT, especially on larger models (Table 1). Could the authors provide a deeper analysis of why these low-rank methods underperform? For example, is this due to limitations in the parameterization, training data, or hyperparameter tuning? \n\nQ2. Can we quantify the computational cost and propose strategies for resource-constrained settings, such as fewer runs or heuristic-based tuning?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method, Reference-free Preference Steering (RePS), introduces a new bidirectional preference optimization objective to improve representation steering in language models. RePS simultaneously optimizes for two contrasting behaviors: concept steering and concept suppression. For concept steering, when interventions are applied positively, RePS up-weights the reward of steered behavior by increasing the likelihood of the desired steered response (winning response) compared to the original response (losing response). For suppression, when interventions are applied negatively, it learns to null out any projection along the steering direction from the representation. The authors implement and train three parameterizations of RePS and evaluate them using AXBENCH, a benchmark for model steering. Across Gemma models ranging from 2B to 27B parameters, RePS outperforms other steering methods trained with traditional language modeling objectives and closes the gap with prompt-based steering. RePS also demonstrated more robustness to prompt-based jailbreaking attacks compared to prompt-only defenses.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is clearly written and easy to follow.\n2. The proposed method, RePS, is a new reference-free, bidirectional preference‑optimization objective that jointly does concept steering and suppression.\n3. RePS incorporates a factor‑sampling trick during training to stabilize optimization.\n4. The method outperforms existing steering methods across various Gemma model sizes.\n\n**Weaknesses**\n\n1. The experimental evaluation of concept suppression is somewhat limited. In particular, the paper does not report suppression results for other intervention-based methods such as LoRA and LoReFT, nor does it include comparisons using the BiPO objective in Table 2. Including these results would offer valuable insight into whether the dual capabilities of RePS, both steering and suppression, generalize across different intervention types and training objectives.\n2. It is mentioned in lines 234 and 235, that finding the optimal parameters (including the optimal steering layers) required extensive grid searches (up to 168 runs per objective–method for gemma-3 models). This limits the practical adoption and scalability of the proposed approach.\n\nQUESTIONS:\nThe paper presents concept suppression results only for rank-1 steering vectors trained with RePS and the language modeling objective, omitting evaluations for other intervention types such as LoRA and LoReFT, as well as alternative objectives like BiPO (e.g., in Table 2). Could the authors clarify why these results were not included?\n\nLIMITATIONS:\nYes, the limitations are included in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The most straightforward way to steer language model is through prompting to provide fine-grained and interpretable control over model generations by variously changing model inputs, weights, or representations to adjust behavior. Another possibility is through a indirect way by adjusting weights or representations, which is often less effective. In this work, Reference-free Preference Steering (RePS), a bidirectional preference-optimization objective is proposed for jointly concept steering and suppression. Evaluated with AXBENCH, the model showed significant improvement among representation based steering and narrowed the gap with prompting.

STRENGTHS AND WEAKNESSES:
### Strength

- The proposed method is clearly presented in the paper and easy to understand.
- There is significant improvement within the family of representation based steering category, demonstrating the effectiveness of this approach.

### Weakness
- The self-contain perspective can be improved. As a researcher outside "Mechanistic Interpretability" area, it's a bit challenging to understand the context and motivation about why representation based steering.

QUESTIONS:
- As a researcher outside "Mechanistic Interpretability" area, my main question is about the motivation to study the implicit steering approach given that the LLM is a natural language interface. Given that the most natural way is prompting, it seems we just need to keep improving the instruction following capability of the model. And if we got a "omni" model that can follow instruction perfectly, it seems the LLM steering problem is solved. It would be good to provide more context and motivations from audience outside this area, for example, what's the pros and cons when comparing with prompting.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03