Title: DISCO: Disentangled Communication Steering for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Steering Vectors

Representation Engineering

LLMs
Abstract: A variety of recent methods guide large language model outputs via the inference-time addition of *steering vectors* to residual-stream or attention-head representations. In contrast, we propose to inject steering vectors directly into the query and value representation spaces within attention heads. We provide evidence that a greater portion of these spaces exhibit high linear discriminability of concepts --a key property motivating the use of steering vectors-- than attention head outputs. We analytically characterize the effect of our method, which we term *DISentangled COmmunication (DISCO) Steering*, on attention head outputs. Our analysis reveals that DISCO disentangles a strong but underutilized baseline, steering attention head inputs, which implicitly modifies queries and values in a rigid manner. In contrast, DISCO's direct modulation of these components enables more granular control. We find that DISCO achieves superior performance over a number of steering vector baselines across multiple datasets on LLaMA 3.1 8B and Gemma 2 9B, with steering efficacy scoring up to $19.1$% higher than the runner-up. Our results support the conclusion that the query and value spaces are powerful building blocks for steering vector methods. Our code is publicly available at https://github.com/MaxTorop/DISCO.


Peer reviews:
comments: SUMMARY:
The paper introduced steering vectors for the query and value representation spaces. Furthermore, the method disentangles the steering hyperparameter $\alpha$ used to steer Query and Value vectors, demonstrating better performance using this approach on the TruthfulQA, and other datasets to evaluate communication steering.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The main claims of the paper are well supported
2. The authors verify claims across multiple datasets with Llama and Gemma LLMs
3. The mathematical claims are sufficiently proved 

Weaknesses:

1. Figure 1 could be simplified and presented early on in the paper with an example to demonstrate the approach better, the figures are difficult to parse on the first read
2. The paper assumes the MHA scenario where queries, keys, and values all have the same number of heads. Some discussion of how this might apply to grouped query attention used in Llama 3 and Gemma 2
3. Some minor typos that could be fixed, see paper formatting concerns

QUESTIONS:
Questions:

1. What does the linear representation space for the attention head output of the keys look like (Figure 1 a(ii) but for keys)? 
2. Does the cardinality of $D^+$ and $D^-$ need to be the same? Or are more negative examples required?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-25

comments: SUMMARY:
Steering vectors are a linear direction that are most related to a concept (increases in the direction of the steering vector). This paper aims to expand the toolbox of steering vectors: proposing, characterizing and validating the steering of the query and value spaces internal to attention heads. Paper shows concepts are linearly discriminable in these spaces, and a larger proportion of them have linear discriminability compared to traditional attention head output spaces. Show that the approach disentangles a strong baseline – referred to as Communication steering.

Contributions:
1)	Propose DISCO steering and variants DISCO-Q, DISCO-V and DISCO-QV (query and value).
2)	Analytically characterize the effects and show that it disentangles a strong baseline – steering attention head inputs – enabling finer control.
3)	Show q and v spaces exhibit linear concept discriminability with higher proportion than attention head outputs, and that DISCO Steering is best on LLaMA 3.1 8B and Gemma 2 9B.

STRENGTHS AND WEAKNESSES:
Strengths:

1)	The proposal is the first detailed investigation of steering vectors in the Q and V focusing on the inputs to attention rather than the outputs.

2)	Proposes Q, V and QV steering.

3)	Extensive results compare against the existing ways of performing steering as well as multiple other places where steering may be performed show stronger performance over a range of tests on LLaMa 3.1 8B and Gemma 2 9B.

4)	Paper is well-written and clear.

Weaknessess:

1)	Although the proposal has clear novelty, the idea seems to boil down to investigating steering approaches in different places in the transformer. (Though there is a proof of a disentanglement proposition.)

QUESTIONS:
See above.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper focuses on the Representation Engineering (RepE) in large language models and proposes a new method named DISCO. Instead of steering the output of attention heads, DISCO steers the representation spaces of queries and values within the attention heads. Experimental results show that DISCO makes concepts more linearly discriminable and demonstrates it to be a more effective approach.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The authors propose a method of steering within the query and value representation spaces inside attention heads.
2. The proposed method is easy to understand and straightforward to implement.
3. The paper achieves better performance than baseline methods and thus offers valuable insights for future research.

Weaknesses:

1. As mentioned in the paper, steering methods rely on the linear representation hypothesis, which means certain problems may not be amenable to such approaches. DISCO inherits both the advantages and the limitations of this family of methods.
2. While steering the query and value representations proves effective, it may not be particularly surprising. Since there are many potential locations in the model where steering could be applied, one could systematically explore these to discover better-performing positions. It is even possible that combinations of different locations could outperform steering just queries and values, suggesting that automated discovery might be more effective than manual exploration.
3. The paper would benefit from a more detailed analysis of cases where DISCO does not achieve optimal performance, helping to clarify its boundaries of applicability and inherent limitations.

QUESTIONS:
See the weaknesses mentioned above.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
The authors show that injecting steering vectors inside attention heads (queries and values) seems to outperform injecting them simply at their output.

STRENGTHS AND WEAKNESSES:
The idea is very simple and likely impactful. I enjoyed Proposition 1 which is also rather simple, but provides good motivation for the method (DISCO-Q especially).

Perhaps a weakness is that as far as I can see no motivation is provided for DISCO-V for why it works better than steering the output of the head. The fact that it works better is of course sufficient to write up in a publication, but it would be interesting to have a deeper understanding as it feels like this could provide further insights into the representation mechanisms used by LLMs.

QUESTIONS:
(Q1) Small comment, but it seems like in Figure 1 DISCO-Q or DISCO-V do not separate better than the attention head output. So perhaps this is not the best example.

(Q2) In Table 1, it might be useful to add an "average rank" or some kind of summary statistic for each method to help better rank them. 

(Q3) Do you have some intuition for why V-steering also works? I enjoyed the Q-steering motivation and was wondering if there is some hypothesis on steering the values.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose DISCO Steering, which applies mean-difference “steering vectors” directly to the query (Q) and value (V) sub-spaces inside each attention head, in order to flexiblily control the behavior of large language models (LLMs) at inference time without full fine-tuning. Specifically, the authors (1) prove a straightforward analytical relation showing how Q/V injections re-weight attention scores, (2) note a simple invariance that makes key (K) injections moot, and (3) run empirical tests on two medium-sized open models (LLaMA-3 8 B and Gemma-2 9 B) over four behaviour datasets (TruthfulQA, Corrigibility, Power-Seeking, Wealth-Seeking). On the selected evaluation benchmarks, the proposed solution can give finer control or better results.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n- Novel solution to steering: The paper introduces query and value space steering (DISCO-Q/V), moving beyond residual-based methods. This disentangles two roles in attention, selection and content, and generalizes prior methods like Communication Steering. To the best of my knowledge, this is a novel approach for this goal. \n\n- Solid theoretical support: The authors connect their method to the linear representation hypothesis and validate it with linear probe results. Analytical derivations explain how Q and V injections influence attention and output, providing decent theoretical foundation.\n\n- Behavior analysis: The paper shows that decoupling Q and V magnitudes allows stronger steering with less degradation. Case studies and ablations support the effectiveness of targeting top heads and tuning Q/V separately.\n\nWeakness: \nMy major concern is that the evaluation scope may not be sufficiently comprehensive to validate the generalization capability of this paper. Specifically: \n>- **Narrow model coverage**: The paper tests only on LLaMA-3.1 8B and Gemma2-9B, both relatively small models. Whether the method generalizes to larger or more diverse models (e.g., LLaMA-3.1 70B, or even vision language models/coding models) remains unknown, which limits its practical relevance.\n>- **Incomplete baseline comparison**: The method is only compared to basic steering approaches like ITI and CAA. Stronger methods such as ReFT [1], AcT [2], and even LoRA [3] are omitted. Although some of these baselines may suffer from larger steering computation, it would be beneficial to see how the steering capability stands among a wider range of model adaptation approaches. \n>- **Limited task diversity**. Evaluation is restricted to short, single-turn behavioral tasks judged by GPT-4o. There is no assessment on multi-turn dialogue, long-context reasoning, or other real-world control tasks. This narrow focus weakens claims of general applicability.\n\n[1] Wu, Zhengxuan, et al. \"Reft: Representation finetuning for language models.\" Advances in Neural Information Processing Systems 37 (2024): 63908-63962.\n\n[2] Rodriguez, Pau, et al. \"Controlling Language and Diffusion Models by Transporting Activations.\" arXiv preprint arXiv:2410.23054 (2024).\n\n[3] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" ICLR 1.2 (2022): 3.\n\nQUESTIONS:\n1. DISCO-QV optimises $\\alpha_q$ and $\\alpha_v$ independently but finally applies the same scalar to every head chosen. Would head-specific magnitudes further help with the capability? \n\n2. The head selection is an important step in the proposed approach. How sensitive of the proposed approach to the selection of heads? Would a different selection of $k$ in top-$k$ selection or even a random selection fully mitigate the performance gain? \n\n3.  Can DISCO simultaneously steer multiple attributes without cross-talk?\n\n4. What happens for contexts of 4k or 16k tokens where attention sparsity patterns differ?\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}