Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models
Authors: Jan Wehner
Sahar Abdelnabi
Daniel Tan
David Krueger
Mario Fritz
Venue: Accepted by TMLR ()

Abstract: Representation Engineering (RepE) is a novel paradigm for controlling the behavior of LLMs. Unlike traditional approaches that modify inputs or fine-tune the model, RepE directly manipulates the model's internal representations. As a result, it may offer more effective, interpretable, data-efficient, and flexible control over models' behavior. We present the first comprehensive survey of RepE for LLMs, reviewing the rapidly growing literature to address key questions: What RepE methods exist and how do they differ? For what concepts and problems has RepE been applied? What are the strengths and weaknesses of RepE compared to other methods? To answer these, we propose a unified framework describing RepE as a pipeline comprising representation identification, operationalization, and control. We posit that while RepE methods offer significant potential, challenges remain, including managing multiple concepts, ensuring reliability, and preserving models' performance. Towards improving RepE, we identify opportunities for experimental and methodological improvements and construct a guide for best practices.

Published in Transactions on Machine Learning Research (09/2025)
Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models
Jan Wehner" Sahar Abdelnabi",
Daniel Tan',
David Krueger",
Mario Fritz²
CISPA Helmholtz Center for Information Security
Microsoft
Universily College London
Mila, University of Montreal
Reviewed on OpenReview: http:://ppaarrrrss...aa/////t/ma
Abstract
Representation Engincering (RepE) is a novel paradigm for controlling the behavior of LLMs. Unlike traditional approaches that modify inputs or fine-tune the model, RepE directly ma- nipulates the model's internal representations. As result, it may offer more effective, interpretable, data-efficient, and flexible control over models" behavior. We present the first comprchensive survey of RepE for LLMs, reviewing the rapidly growing literature to ad- dress key questions: What RepE hods exist and how do they differ? For what concepts and problems has RepE been applied? What are the strengths and weaknesses of RepE compared to other mothods? To answer these, we propose a unified framework describing RepE a pipeline comprising representation identification, operationalization, and control. We posit that while RepF: methods offer significant potentíal, challenges remain, includ- ing managing multiple concepts, ensuring reliability, and preserving models' performance. Towards improving RepE, we identify opportunities for experimental and methodological improvements and construct a guide for best practices.
1 Introduction
Prompting and fine-tuning are common and effective methods for controlling the behavior of Large Language Models (LLMs). But recently, new paradigm for controlling LLMs inspired by interpretability rescarch has emerged: Representation Engineering (RepE). Instead of adapting the inputs training the weights towards outputs, Representation Engineering controls the LLMs' behavior by manipulating its internal rep- resentations. For this, it first identifies how a muman-undessanndalle concopt represented in the network's activations. Next, it uses that knowledge to steer the model's representations, thus, controlling its behavior (see Figure
By lapping into models' representations, RepE offers two main advantages: (1) Improve understanding: RepE identifies how human- undaastaddet concepts are represented in the models' activation space; steering that representation can verify whether the representation has the expected influence on the outputs. (2) Control: RepE is promising as a powerful tool to control the behavior, personality, encoded beliefs, and performance of an LLM, allowing us to make models behave in safe and desired ways. Since no training is required, RepÉ can be cheaper, more data efficient, and more flexible to different users and situations than other methods while causing less deterioration the model's performance.
Initial work Activation Steering (Turner al. 2024 Li al. 2023a was built the assumption that concepts are represented linear directions in the activation space of LLMs (Prr et al., 2024b). These
"Correspondence to Seeeerpppeeee.

---

Published in Transactions on Machine Learning Research (09/2025)
1. Identify how "love" is represented
Control the model be more loving
User: What do you think about Jim?
User:
Representation of love
Identify
steer
LLM:
love Jim! He's
LLM:
great!
Figure 1: Representation Engincering first identifies how a concept is represented in the activation space of the model and then steers that. representation to control the model's behavior.
methods focused on the difference in activations for inputs that are positive or negative with regards to the concept. They identify vector that captures the model's representation of a concept and can be added to the activations on new inputs to modulate the intensity of the concept. Since then, range of new RepE methods have been proposed. For example, such methods identify representations by finding interventions that lead to desired outputs (Cao et al. 2024a) or learning internal features in an unsupervised fashion (Templeton et al.] 2024). They go beyond static and linear representations (Qiu et al. (224)) by employing different operators than vector (Postmus & Abreu 2024). And they experiment with new functions for steering activations (Singh et al. 2024h, as well modifying the weights of the model by adding adapters (Wu el al. 2024c).
However, the literature on RepE lacks an overview and there exist many unanswered questions and conceptual confusions. In our work, we systematically answer the following questions.
What is Representation Engineering? We provide the first unification of different approaches to RepE' with framework that describes them as pipelines that identify, operationalize, and control reprosentations.
What RepE methods exist and how do they differ? We discuss and contrast different methods for each step in this pipeline.
How are RepE methods evaluated? We describe evaluation methodologies and benchnarks, and propose best practices for evaluating RepE methods.
How has RepE been applied? We describe how RepE has been applied to areas such as AT Safety, Ethics, and Interpretability and show which concepts are more amendable to be controlled by RepE.
How does compare to other methods? We contrast RepE to related mothods and provide a meta-survey of comparisons between RepE and fine-tuning, prompting, and decoding- methods.
Why does it work? We suggest reasons why RepE works so effectively.
What are strengths and weaknesses of RepE? We provide a list of advantages and challenges of RepE.
What opportunitics for future research are there? We suggest broad themes and concrete ideas for improvements and outline opportunities for strengthening the field.
To answer these questions we conduct a thorough literature review, collecting detailed information ion from >30 papers. Although previous surveys on Mechanistic Interpretability (Ferrando et al.] 2024 Bereska & Gavves 2024) and frameworks for causal interpretability (Mueller et al. 2024] Geiger et al. 2024a have touched on RepE, there no dedicated survey of this research area. A survey is especially pressing since over 100 papers have been released within the last year without a systematization of the literature. In addition to
2

---

Published in Transactions on Machine Learning Research (09/2025)
Activation Steering, include methods that modify the weights, identify concept representations through optimizing for outputs, and are not based on vectors. We focus methods that apply control on intermediate representations of a model without fully replacing them, thus excluding methods, and Activation Patching. For a detailed description of the inclusion criteria and literature search process, see Appendix B
decoding- based
We conclude that future work needs build ways to benchmark Repl methods enable thorough compar- isons. In addition, we identify opportunities in considering representations that are non-linear, multi-concept, have interactions between layers trajectories over time. Furthermore, there are number of promising directions for new applications, and improved concept identification and control methods. To make RepE usable in practice, we will need to see improvements in its ability for multi-concept steering, long-form generation, reliability, Distr--iittiittionn (OOD) robustness, and maintenance of general capabilities.
Contents
Introduction
1
2
What is Representation Engineering?
5
2.1
Strategics for RepE
6
2.2
Advantages Representation Engineering
Framework and Notation
7
Representation Identification
Operationalizing Representation
Representation Control
Representation Identification
10
Input Reading
4.2
Output Optimization
13
Unsupervised Feature Learning
15
Representations Opcrationalization
17
Assumed Geometry
17
5.2
Concept Operators
18
Representation Control
19
6.1
Moditying Activations
19
6.2
Modilying Weights
22
Practical Representation Engineering Pipelines
23
Prototypical RepÉ Pipelines
23
2
Which Methods Work Better
25
Evaluation of RepÉ Methods
27
3

---

Published in Transactions on Machine Learning Research (09/2025)
8.1
Common Evaluation Methodologies
27
Best Practices for Evaluating RepE Methods
29
8.3
RepE benchmarks
30
9
What Concepts Can Be Controlled with Representation Engineering?
31
Types of Concepts
31
9.2
Commonly Controlled Concepts
32
Applications of Representation Engincering
32
10.1
Al Salety
32
102
Ethics
................................
34
10.3
Knowledge Editing
35
10.4
Task Exccution
35
10.5
Controlled Text. Generation
36
10.6
Performance
36
10.7
Interpretability
37
10.8
Representation Engineering outside LLMS
37
11
Comparing RepE to Other Methods
38
Related Methods
38
112 Meta-study comparing to Other Methods For Bchavior Control
39
12 Why Does It Work?
41
13 Challenges in Representation Engineering
42
13.1 Empirical Weaknesses
13.2 Principled Challenges
44
14
Opportunities for Future Research
46
14.1
Opportunities to Improve Reprosentation Identification
................................
46
14.2
Opportunities to Improve Representation Operationalization
46
1.3
Opportunities to Improve Representation Control
47
14.4
Possible Applications of Representation Engineering
48
14.5
Building a More Rigorous Science of Representation Engineering
48
Conclusion
50
55,
Limitations
50
Meta-survey
64
A.1 Commonly Used Models
64

---

Published in Transactions on Machine Learning Research (09/2025)
A.2 Number of Samples
64
A.3 Statistics about Publications on Representation Engineering
65
B Survey Methodology
66
B. Literature Search
66
B.2 Extracting Information from Papers
67
Information Extracted from Papers
67
D Papers that compare RepE to Prompting, Fine-tuning and Decoding-basse based methods
69
D.1 Table of empirical comparisons
69
2 What is Representation Engineering?
Representation Engineering (RepE) a class of techniques that: Manipulate representations of model in order control its behavior with regard to a concept.
One perspective RepE frames the computations in an LLMs as computer program with intermediate variables and computations that use these variables. In this perspective can attempt to find intermediate variable that correspond to specific concepts. This allows us to meaningfully change the values of these variables to influence later computations and ultimately the outputs of the program. Another perspective states that RepE aims to identify patterns in the activations of an LLM that correspond to high-level, concepts. turns out that these activation patterns can be used to manipulate the activations to get consistent changes in the behavior of the LLM.
Goals of RepE. The goals of RepE are (1) Behavior Stecring: Our ability control the behavior of LLMs and (2) Interpretabilily: Our ability to understand the internal compulations of LLMs. RepE can be used by LLM providers to prevent undesired behavior of an LLM such as guarding against jailbreaks. It can also be used to fincly control the behavioral tendencics of the LLM or to improve it's suitability and performance at specific tasks for example by adapting it's personality and style or by improving it's reasoning performance. Additionally, RepE can provide us with some insight into the internal processes an LLM which are commonly considered a black-box. By demonstrating that some concepts are consistently represented in LLMs, determining in how much concept's representation is active specific outputs showcasing the influence that concept has on the output of the model. As such RepE is often tool for scientific inquiry into specific LLM behaviors.
Running Example. One the most commonly steered concepts is Truthfulness, a models intention to say things that are true instead of false as for example measured in the TruthfulQA benchmark. RepE methods aim identify patterns in the activations that correspond the models representation of the concept. This could for example be linear direction whose magnitude denotes how truthful the outputs of LLM are specific inputs. This activation pattern, hereforth called concept operator, when the model is being dishonest and to steer it to give truthful outputs. As such it could be deployed by model providers to decrease hallucinations and be used by safety researchers to detect and mitigate cases of deceptive behavior.
then be used to detect
The RepE Pipcline. RepE methods consist of pipcline with two steps: First, we identify how the tar- geted concept is represented within the model. Secondly, we use that information to steer the model's repre- sentations inputs. In this context, concept can be understood broadly as any human-understanddbbl
5

---

Published in Transactions on Machine Learning Research (09/2025)
feature like behaviors, tasks, entities, pieces of knowledge, or personality traits. RepE assumes that such concepts are represented in LLMss Representations are the internal structures or encodings that model uses to capture and process information about a concept.
The goal of Representation Identification is to produce a concept operator that accurately captures the model's representation of a concept. The concept operator is an object, for example, a vector, with a specific intended relationship to the model's actual concept representation. This bakes in assumptions about the geometry and location of concept representations.
The concept operator is then used for Representation Control by steering the representations of the model. For this a steering function is employed that uses the concept operator to manipulate the activations or weights of the model. If the concept operator accurately reflects the model's representations and the steering function is effective, this will steer the model's representations and, thus, control the outputs of the model.
RepE is also somctimes referred to as Activation Engincering or Activation Steering. However, we chose the term Representation Engineering to emphasize that represenlations can also be controlled by modifying the weights of model.
2.1 Strategies for RepE
There are variety of techniques to perform Representation Identification and Control. The most common way to perform RepE is through Linear Aclivation Addition. Hereby, linear direction is identified that should denote the representation of a concept for example by contrasting activations for inputs where the concept is or isn't present. Another popular approach are Sparse Autoencoders, that disentangle features into a dictionary of concepts by projecting them into a nigher-dimensional sparse space. Finally, it's possible to directly identify representations that cause the specific behavior changes by fine-tuning a concept operator towards desired outputs.
Once the representation was identifedd, we can use it to control the LLMs behavior by adapting it's activations or changing it's weights. For example we could simply add a vector on the activations for new inputs. Or we can adapt the weights, such that they produce activations that align more strongly with the activation pallerns we have identified.
Illustrative Examples To illustrate RepE, we provide an example inspired by Zou et al. (2023a), where RepE is applied to make an LLM give more truthful answers. To identify how truthfulness is represented, inputs are provided that instruct the model to be honest to lie. Next, the activations for these inputs are collected. Then, the difference between activations for honest and dishonest inputs is calculated. This produces a vector that denotes direction in the activation space which represents honesty. This vector can be used to guide the training of LoRA module that generates activations more aligned with the truthfulness direction. This effectively steers the model to give more truthful answers.
In heo example inspired by Cao et al. (2024a), RepE is applied to align the outputs of an LLM with the preferences of a human. For this, they find how activations need to be changed to lead to desired outputs. Given dataset of desired and undesired outputs, vector is optimized that can be added to the activations during a forward pass to make it more likely for the desired output to be generated. When processing new inputs, that vector can be added to the activations, thus, steering the model towards human preferences.
2.2 Advantages of Representation Engineering
Sample Efficient. RepE can be effective with a low number of training examples (Wang et al. 2024a Furthermore, some RepE methods do not require a labeled dataset (see Section 7.1 These properties make it easier and cheaper to employ RepF.
Flexible. The steering of RepE, can be turned on or off flexibly. It is also possible dynamically adjust the steering per request, thus enabling personalization and context- -eeeedeed control (Cao et al.) 2024a
Stickland et al. 2024 Guo et al. 2024 Chen et al.] 2024a; Lucchetti & Guha 2024

---

Published in Transactions on Machine Learning Research (09/2025)
Low Impact Capabilities. As shown in Section 11.2 RepE tends to not strongly degrade the pabilities of the model (Panickssery et al.) 2024 Stickland et al. 2024 Guo et al. 2024 Qiu et al.) 2024
van der Weij et al. 2024). Thus, it is possible to use RepE' practically in production without sacrificing
model quality.
Precise Control. RepE offers direct control over the representations of concept. This makes it possible to apply precise steering the granularity of single concepts.
Efficient During Inference. Methods for Representation Control come negligible increase in compu-
tational cost during inference (Li et al. 2023a Most methods do not require large number of additional parameters, and none require additional inference steps.
Causally Verifiable Interpretability. By observing the effect of steering a representation on the output, we can verify that the identified representation causes the expected change (Marks & Tegmark 2023 Arora ct al. 2024). This can make us more confident that the right representation for concept was found.
Tacit Interpretability. By giving users access to knobs they can turn to influence internal, human- understandable representations, RepE can provide tacit understanding to non-expert users about the inner workings of LLMs |Chen et al. 2024a).
Advantages over Other Methods. Our meta-survey in Section 11.2 indicates that RepE tends to be more effective at controlling models' behavior than alternative methods like prompting, fine-tuning, or decoding- based methods.
Compatible with Other Methods. RepE can be used in addition to other methods for stronger control (see Section 11.2 This enables the use of RepE in defense-in-depth approach, where multiple imperfect methods are combined to achieve strong safety guards.
Takeaway: What is Representation Engineering?
RepE manipulates the representations of model in order to control its behavior w.r.t. concept. Tt be used to steer models behavior and interpret its internals. This is done by identifying how concept is represented within the model and then using that knowledge to steer the representations. It allows for effective, efficient, flexible, and precise control over models' behavior while largely retaining quality.
3 Framework and Notation
Building the background and definitions in Section pipelines based on the method they use for Representation Identification, Representation Control and how they operationalize representations (sce Figure 2 Additionally, we provide unifying formalization of RepE that showcases the function signatures of the steps of RepE pipelines. For an overview of the notation, refer to Table
2 propose framework that classifies RepE
3.1 Representation Identification
Representation Identification (RI) is the first step in the RepE pipeline. It aims to find out how the concept is represented in the LLM's activations. For this, an RI method needs to provide a stimulus related to the concept, collect the model's activations, and calculate the representation of the concept from the collected activations. The result is a concept operator that denotes the model's representation of the concept.

---

Published in Transactions on Machine Learning Research (09/2025)
Representation Identification
Necessary Steps for Representation Identification
Provide Concept-related stimuluss
Collect activations
Calculate Concept Operator
Activations
Inputs x
Outputs
A
Input Reading
Unsupervised Feature Learning
Output Optimization
or
1. Provide Inputs X
1. Collect Activations A
1. Collect Activations A
Collect Activations
Feature Discovery
2. Score Outputs y
Calculate operator
3. Concept identification
3. Calculate operator
Identify
Concept Operator
Representations as
Representation Operationalization
assumes
Vector
Linear Directions
Matrix
Clusters
Tranformation
Non-lineat manifolds
control
Modifying the weights of the LLM
Steering activations during Inference
Training
Addition
Direct Editing
Projection
F(W) W
F(A) A
We
Representation
Control
New Inputs
A
Controlled Outputs
Figure 2: Framework of Representation Engineering pipelines. One Representation Identification method is used to identify concept operator. Representations operationalized by assuming geometry of reprosentations. The concept operator is used to steer the weights or activations of the model.
Definition 1. Representation Identification RI is method
RI of
that takes
model M
set of strings that have a positive, negative, neutral, or an unknown relationship to the concept:
S = Sc S S Se?
and returns a concept operator that denoles the concept's representation.
Methods for Representation Identification need to carry out three steps. First, they need provide stimulus related to the concept. These strings used inputs ground-truth outputs that have
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Explanation Notation for ease. Large letters X,Y, refer to a set the corresponding
elements.
Notation
Name
Explanation
Target concept
The concept want to control
M
Model
The LLM on which RepE is applied
String/set of strings
The text or set of texts String given inputs to the model
EES
Input
y VES
Output
String outputted by the model
get/-/0
saadtiv//uegaaiv//eentzaa string Weights
Text that is positively wuugaaaielly/uuuttt Weights of model M at layer
related the target concept
wy
a,, m.
Activations
Activations layer model component m. and token position p Methods that output the concept operator Methods that steer the model's representations An object denoting the model's representation of the concept function that modifies the weights or activations using OF
RI
Representation Identification
RC
Representation Control
of
Concept operator
Steering function
positive, negative, neutral, or unknown relationship to the concept. This means that they exemplify or contain the concept, serve as counterexamples the concept, are unrelated the concept, or that thoir relationship to the concept is unknown. In our running example these strings might be truthful deceptive outputs, text that is neither honest dishonest This relationship is usually predefined by humans, e.g., labels in dataset. However, in some cases, it is evaluated dynamically during the execution of the method. It is important that these strings actually activate the desired concept's is representation, so it is present in the activations. Secondly, they collect activations of the model related to these strings.
might just not know it's relationship to honesty.
Definition The aclivation collection function Act.
X X",,M,,',l,..77 Al.m.p
extracts activations of model M for inputs X layer I, model component m, and token position p
Crucially, our knowledge about the strings' relationship to the concept lets us infer whether the concept representation is present the activations for that input. Thirdly, they calculate the concept operator from these collected activations. This can range in complexity from a simple arithmetic operation to a large training rum.
There are three broad catogories of approaches to Representation Identification. Methods that employ Input Reading provide inputs related to the concept, read out their activations, and then calculate the operator from the resulting sets of activations. Those that do Oulpul Oplimization utilize way of scoring out with regard to the concept to optimize the activations accordingly. A concept operator can then be derived from the optimized activations. Finally, other mothods employ Unsupervised Feature Learning, where they first learn internal features from activations an unsupervised way. We can identify the feature corresponding the targeted concept by studying inputs and outputs related that feature.
3.2 Operationalizing Representations
To be able to identify and control representations, RepE pipelines need to operationalize how the model represents the concept. This introduces assumptions about the structure and geometry the model uses to represent the concept in the activations. For example, concepts could be represented as linear directions in the activation space (Park et al. 2024b) or as non-linear manifolds. Furthermore, a concept operator of is necessary denote how the model represents the concept. of results from the Representation Identification step and is used Representation Control. It can take various forms and differs in its intended relationship to the model's actual representation of the concept.

---

Published in Transactions on Machine Learning Research (09/2025)
3.3 Representation Control
Representation Control (RC) is the second step in RepE pipeline. RC methods should steer the represen- tation of a concept in order to control the model's behavior w.r.t. that concept. For this, they employ a stecring function F that uses the concept operator of to modify the model's activations or weights. While F only describes a mathematical operation, RC encapsulates the whole process of where, when, and how F is applied. Employing RC should result in controlled outputs y that are more aligned with the concept than an originally uncontrolled output y yorg
Definition 3. Representation Control RC
RC(F, M,, + y
applies the steering function F and concept operotor control the representations of model M on inputs v. This leads to oulpuls yf that have been controlled with regard to
To steer the representations of model, we can either modify its activations during inference adapt its weights. Activations of unseen inputs can be adapted during inference by shifting them using the concept operator.
Definition An activation steering function
(ar, oi) + aj
adapts the activations a; at layer 1, using of to attain the steered activations af.
In addition, we can modify weights to steer the activations they output in desired ways. Definition A weight steering function
F Fm of) - wf
modifies the weights wy using of so that
wf(a - af
the modified weights wf produce steered activations of.
Some steering functions apply simple operations, while others require retraining of the model. Furthermore, some RC methods are adaptive to the present context, while others always apply the same effect. Weight steering functions only need to be applied once, while activation steering functions need to be applied repeatedly every input. Notably, the line between activation and weight steering functions can be blurry. For example, in an MLP layer, adding steering vector to the activations is equivalent to adding it to the bias term.
Takeaway: Framework Notation RepE 2-step pipeline:
1. Representation Identification finds a concept operator that denotes the model's representation of the concept.
2. Representation Control applies a steering function that uses the concept operator to modily the activations or weights the model.
Further, RepE pipelines make assumptions about representations to operationalize them into a concept operator.
4 Representation Identification
Representation Identification (RI) aims to find a concept operator that denotes how the model represents the target concept in its activations. Methods can be based on Input Reading, Output Optimization, or Unsupervised Feature Learning. This section separately dives into these categories and identifies differences of methods within the categories.
10

---

Published in Transactions on Machine Learning Research (09/2025)
4.1 Input Reading
RI methods that fall into the Inpul Reading category provide inputs related to the concept, read out their activations, and finally calculate the operator from the resulting sets of activations. In our running example can feed inputs related to honest or dishonest behavior into the model, collect their activations and identify patterns that correspond to honesty.
Table 2: Steps and differences between different. Input Reading methods for Representation Ident.fi- cation.
Steps in Input Reading
How do methods differ?
Source of inputs (existing datasets, manually written, or synthetically generated) Relationship to the concept (positive, neutral, unrelated, or unknown) Differences between contrastive sets (psired with concept-related difference, unpaired)
1. Constructing Inputs
Prompting template (pre-prompt concept-related instructions, A/B choices)
Layer (all layers, or concept-related layers only)
Collecting Activations
Model component (residual stream, output of MLP or attention heads)
Token position (End of sequence, all tokens, or concept-related tokens)
Calculating the Concept
What calculation? (class-mean, probes, or components after dimensionality reduction)
Operator
Other techniques (refining via concept-related neurons, project to other spaces)
4.1.1 Constructing Inputs
Input Reading methods aim to activate the concept representation by providing inputs S that are related to the concept. To do that, one needs to attain a relevant dataset, divide it into groups that differ in their relation to the concept, and put the inputs into a prompting format that successfully activates the concept's representation.
Source of Inputs. The data be gathered from existing dataset, manually created, syntheti- cally generated. A majority of papers use existing datasets like TruthfulQA (Lin et al. 2022 ToxiGen (Hartvigsen et al. (022). Researchers can use the labels in the dataset to separale examples into different groups. Multiple papers also use mamually created datasets (Li et al. 2024b: Marks & Tegmark 2023), which roquire human effort but may allow for more flexible and rigorous experimentation. Furthermore, it is also possible to prompt an LLM to synthetically generate the inputs. This is an efficient and flexible approach that allows targeting concepts for which no dataset exists. However, one might introduce confounders based on the LLM's biases and errors.
Relationship to the Concept. Typically, inputs divided into multiple sets that differ in their relation to the concept. Inputs can be positive s negative se or neutral se if they exemplify or contain the concept, serve as a counterexample, or are unrelated to the concept, respectively. Inputs can be classified based on whether they mention a certain topic, e.g., by checking for wedding-related keywords [Trnerr et al. 2024) or contaïning a specific word (Makelo et al. 20255. More abstractly, inputs can differ in whether they excmplify a concept, c.g., being written in a specífic style (Konen et al., (024).. Lastly, inputs can be divided based on the model's response to them, like classifying inputs based on whether the model answers them
correctly (Yang et al. 2024 Højer et al. 2025).
Prompting Template. Next, the inputs are often formatted into a prompt that activates the concept's representations. Multiple papers use an A/B-choice formal consisting of a question, two answer choices designated with "A:" and "B:", and the letter corresponding to the answer (Panickssery et al. 2024). Here, it important to randomize the order, to not introduce the letter or position as confounder. Another common template adds pre-prompts to question that instruct the model towards or against the concept Zou et al. 2023a
11

---

Published in Transactions on Machine Learning Research (09/2025)
Differences Between Contrastive Sets. Some methods construct contrastive pairs of inputs. These are derived from the same example but transformed into two contrasting inputs. For example, one can use the same question but attach a different pre-prompt (Zou et al. 2023a) or answer to it (Panickssery et al. 2024). Deng et al. (2025) employ a matched pair trial design where pairs of inputs differ only in the roles assigned to them and are guaranteed to lead to opposite behaviors. In other methods, the positive and negative inputs are unrelated to each other since they are not derived from the same example (Wu et al. 2024c; Xu et al. 2024b Beaglehole et al., (025).. Generally, contrastive pairs are preferable since they better isolate the target concept from other factors that could differ between the sets of inputs. However, for some settings, contrastive pairs can be harder to obtain.
4.1.2 Collecting Activations
The provided inputs are fod into the model to perform forward pass for which the activations Ac, m,, + Af. are recorded at specific layer I, model components m, and token positions p. It is possible to find the best options for these hyperparameters by testing the effectiveness of the resulting concept operator (Arditi et al. 2024
Layer. a common practice to collect the activations from all layers and subsequently compute a concept operator for each layer. Alternatively, one can reduce computational cost by focusing on layers that are more likely to contain the concept's representation (Ghandeharioun al., 2024).
Model Component. There are multiple distinct places within transformer block at which activations can be collected. The most common ones are the residual stream after the MLP or the attention layer, the outputs of individual attention heads, and the outputs of an MLP or attention layer. Analyzing the MLP or attention heads gives insight into the computations done at that layer, while focusing on the residual stream is akin to analyzing the intermediate memory
program.
Token Position. Some methods read the activations a specific token position that has special significance towards the concept, commonly, the last token (Scalena et al.) 2024 or the mndoof-squeence token (Rozanoval et al. 2023). the token containing answer (Panickssery et al 2024 or the token mentioning the concept. Other methods collect activations for all tokens and then take their average to serve as represcntation of the sequence (Li et al. 2024e..
4.1.3 Calculating the Concept Operator
The activation collection step results in sets of activations that have different relationships to the concept. These can be used to calculate the concept operator. Assuming that we are given a positive and a negative set of activations A Attm,, Ammpp with n samples cach, we need a function f f Ammm.)) - of. that calculates the concept operator.
Difference in Means. Taking the difference between the classes of activations can identify the direction that captures their differences. Differenco-in-Means (DiM) is popular method that simply calculates the average for both classes and subtracts the negative from the positive one (Panickssery et al. 2024 Jorgensen
2024). This is equivalent to taking the average of the difference bet between pairs of et al. 2024 Ball et al.
at a Earlier papers took the difference between a single pair of inputs Liu et al. 2023), however, it has been found that taking the difference across larger datasets leads to higher-quality operators (Chu al., 2024 Jorgensen et al.) 2022).. Arora al. (2024) employ an unsupervised method that discovers 2 clusters of activations through k-mcans and subtracts their means. Im & Li (2025) show that DiM optimal at identifying a vector that matches negative examples to positive ones. However, these vectors are not suited to steering positive examples where they lead to large performance reductions.
examples of
+ o 44
Turner et al. 2024
Probes. Probes are classifiers that predict whether activations belong to the positive or negative class fe(a) + {ct,c } {(lain & Bengio 2018). The weights of a probe should capture the concept's repre- sentations. Many papers employ logistics regression classifiers as probes and use the lcarned weight vector as the concept operator (Chen al. 2024a Guo et al. 2004 Wang et al., (024a)). Contrast Consistent Search identifies linear probe that. retains logical consistency P(e(x.)) 1 P(not c/2x)) Burns et. al.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Chen et al. 2024bb. Multiple papers use MLP-based probes, which are more expressive (Hoscilowicz et al. 2024 Singh et a 2024). Waan et al. (2024c) use k-means clustering on the difference between positive and negative pairs to then train one linear probe per cluster, thus discovering multiple aspects of the concept. [u et al. (2024b) find a vector that, can be added lo activations to prevent a linear probe from detecting the concept. However, Im Li 2025) critique that probes only capture the direction and not the magnitude necessary match positive and negative examples. Beaglehole al. (2025) train a probe that learns non-linear features from the activations, before deriving matrix that captures the directions in activation space to which the probe is most sensitive. The top eigenvectors of these matrices are then used as concept operators. Lastly, Nguyen et al. (2025) train concept vectors to match the negative to positive activations distribution using a Maximum Mean Discrepancy loss that captures higher order differences in distributions like variance.
2023 Others train multiple orthogonal probes and combine them through a weighted sum
Dimensionality Reduction. Dimensionality Reduction techniques can help to identify the most important information about the concept. It is common to apply Principal Component Analysis to the difference between the two sets of activations PCA(A+-AT) as the concept operator of \ (Zou et al. 2023a Adila et al. 2024b). However, the direction with the highest variance does not necessarily capture the shift between positive and negative examples (Im & Li 2025). Other papers use Singular Value Decomposition to identify the most important components of each group of inputs (Adila et al., 2024b). their difference (Tlaie) 2024 Ma et al.; 2025): ; or their covariance matrix (Qiu et al. 2024). Xiao et al. (2024) criticize that dimensionality reduction techniques lose crucial information and instead train a Gaussian Mixture Model model the distribution of positive and negative activations.
X1, A2, before selecting the first principle component
Additional Techniques. In addition to these techniques, multiple papers attempt to identify neurons attention heads which are more relevant to the concept and then only calculate the operator from those. The idea is to find a minimal steering intervention that causes fewer side effects by assuming that the concept's representation is only encoded in the selected neurons. Li ct al. (2023a] and Ma et al. (2025) only intervene on the attention heads where probe achieves high accuracy, and Todd et al. (2024) focuses on atlention heads that have a causal effect on specified task. Other papers find important neurons by calculating attribution scores per neuron Ww al.) 2024b), by selecting ones that exhibit low variance the difference between sets (Li et al.
2024c or by identifying causally infiuential nodes through Activation Patching (Xiao
et al. 2024). Another technique first projects activations into different space before deriving an operator.
For example, Zhang et al. 2024b) train an autoencoder that projects activations into space where they
are casily scparable.
4.2 Output Optimization
Representation Identification methods that perform Output Optimization provide information about the concept by scoring outputs. This is done by first providing inputs and collecting their activations and outputs. Secondly, loss function is constructed by evaluating the outputs with regard to the concept. Thirdly, an optimization algorithm is used to iteratively optimize the activations to achieve a high concept score. In our rumning example this can be done by defining a way to score whether model generations are truthful and then optimizing a concept operator that can be applied to make truthful outputs more likely.
4.2.1 Collecting Activations
Output Optimization methods provide some inputs X and collect the resulting activations that will be optimized Act(M, X I, m, - Af.m.p at a specific layer 1, model components m, and token positions
Source of Inputs. Inputs X can be taken from existing datasets (wu et al. 2024c), or be generated manually or synthetically (as in Section 4.1.Ip. While they do not need a specific relationship to the con- cept, they are usually moant to produce outputs that are relevant to the concept. The inputs can be a query (Yin al. 2024 Cao al., 2024a] Zeng et al., 2024] or a sentence that the model should transform (Subramani et al.
2022 Konen et al. 2024
13

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Steps and differences between different Output Optimization methods for Representation Identification.
Steps In Output Optimization
How do methods differ?
Source of inpuls (existing datasets, manually written, or synthelically generated)
Layer (all layers, or concept-related layers only)
1. Collecting Activations
Model component (residual stream, output of MLP attention heads)
Token position (End sequence, all tokens, or concept-related tokens)
Output Format (token, sentence, long-form generation)
Output Scoring
Scoring (dataset or scoring function) KL-divergence, regularization)
Loss Function
Optimization Algorithm (SGD, iterative sampling)
3. Optimizing Activations
Deriving Concept Operator (combined training, individual operators, averaging)
Layers, Model Components, and Token Positions. Next, the activations of the model that will be optimized are chosen. Activations are usually collected at every layer from the atiention heads, residual stream, MLP or attention layer outputs. However, some papers focus on attention heads for which lincar probe achieves high accuracy (Yin al., 2024; Zeng al., 2024). Furthermore, we need to decide at which token positions the optimization will be applied. While some papers only focus the activations during generation (Konen et al. 2024), others use the activations on input tokens (Cai et al.) 2024 a combination of input and output activations (Wu et al.) 2024c), or activations of tokens with specific syntactic relevance to the concept, e.g. for the token naming a person (Hernandez et al. 2024
4.2.2 Output Scoring
The model generates outputs M (X) + Y that are scored according to the concept Score, which is used as loss function LScore(Y)
Output Format. Based on the input, can let the model generate single token Hernandez et al.
2024), a sentence (Subramani et al. 2022 or long-form generation (Cao ct al. 2024a Wu et al. 2024a
Scoring. These outputs are now evaluated on how much they align with the concept. This provides information about the concept to the optimization process. The outputs can be evaluated by comparing them to ground-truth answers from datasct or through a scoring function. A score can be derived from the log-probability the LLM assigns to the ground-truth token (Hernandez al.] 2024 Wu et al. 2024c) 2024) from the difference in log probabilitiee between desired and undosired answer (Coo et al. 2024a Zeng et al.] 2024). A scoring function can judge how much (2024) ask an LLM whether the output is aligned with the concept and use its token probabilities on "yes" and "no" as score.
sentence (Subramani et al. 2022 Yin et al.
an output aligns with the concept. Cai et al.
Loss Function. The score is then transformed into a loss function. The Pross-Entropy loss is popular 2022 Konen et al.] 2024 Wu al., 2024a c). Yin al. (2024) add a regularization term to induce sparsity and Hernandez al. (2024) additionally minimize the KT.-divergence compared to the unsteered token distribution to minimize unwanted side-effects. Other papers adapt previously developed loss functions for preference optimization like DPO (Cao et al. 2024a) or ORPO (Zng et al., 2024). Lastly,
(Subramani
Wu et al. (2025) suggest ReFT-r1, which optimizes joint objective for the probing accuracy and steering effectiveness of concept.
4.2.3 Optimizing Activations
A concept operator is optimized modify activations so that they lead to outputs that score highly on the loss function of argmin
14

---

Published in Transactions on Machine Learning Research (09/2025)
Optimization Algorithm. After we have attained loss function with regards to the concept, we employ an optimization algorithm to adapt the activations to minimize that loss. The most common method for this is using Gradient Descent with the AdamW optimizer (Subramani et al. 2022 Hernandez et al, 2024 Yin et al. 2024 Cao et al., 2024a Wu et al.) 2024a c). Cai ct al. (2024) propose an iterative algorithm that samples outputs belore backpropagating to a steering vector that makes the highest scoring output more likely.
Derive Concept Operator. The concept operator is optimized for the loss function using the optimiza- tion algorithm, while the activations and weights are kept frozen. This concept operator should combine information about the concept from different outputs so that it generalizes to unseen examples. For a set of inputs and outputs, we can train one shared concept operator using SGD over batches of imputs and evaluated outputs YYn et al., 2024 Wu et al. 2024c Subramani et al. 2022 instead trains one concept operator for each input. However, this concept operator fails to generalize to different inputs. (Konen et al. 2024) overcome this by first deriving one concept operator per input before averaging them into generalizod concept operator.
4.3 Unsupervised Feature Learning
Representation Identification methods that perform Unsupervised Feature Learning (UFL) first learn sparse features in the model's activations and then identify which concepts they belong to. This done by first collecting a set of activations, applying learning algorithm that discovers features in the activation space, and then identifying which human-understandee approaches, UFL methods do not set out to find one specific concept representation but identify large set of concepts from which concepts of interest can be chosen. In our rumning example, we would first discover a library of representations for many concepts and then identify which one corresponds to truthfulness.
concept feature corresponds to. In contrast to other RI
UFL methods. The dominant paradigm in UFL are Sparse Autoencoders (SAEs) Templeton et al.) 2024 Huben et al., 2024). They are motivated by the observation that model represents more features than it has directions, thus forcing the model to represent features in superposition (Elhage et al. 2022). SAEs aim to disentangle these representations by learning dictionary of monosemantic directions that represent sparse concepts. This dictionary receives the LLM's activations and is trained to accurately reconstruct them while only using a small set of its features at a time.
Deep Causal Transcoding (DCT) Maack Turner 2024a attempls to find a minimal perturbation in the activations of a layer that causes large, predictable change in later layer. As such, DCT might have applicability for eliciting latent behaviors like backdoors or unknown capabilities.
Table 4: Steps and difference between Unsupervised Feature Learning methods for Representation Identification.
Steps in Unsupervised Feature Learning
How do methods differ?
Inputs (large-general, small-specific datasets)
1. Collecting Activations
Location (residual stream, attention/MLP layer outputs)
Goal (disentangle features, or find meaningful perturbations)
2. Discovering Features
Loss Function (reconstruction and sparsity, consistent large change)
Learned Objects (feature dictionary, diverse steering vectors)
Identification (highly activating, probing accuracy, mutual information) Labels (LL.Maas-a-judge, existing dataset)
Identifying Concepts
15

---

Published in Transactions on Machine Learning Research (09/2025)
4.3.1 Collecting Activations
UFL methods provide inputs X and collect activations X9,,, - A.m. at specific layer 1, model components m, and token positions
Inputs. SAEs employ large datasets to collect many activations. The dataset does not have to be specifically related to the concept of interest. However, SA Es require much larger dalasets than other RI methods. This inefficiency partially stems from the fact that SAEs learn many features, while other methods focus on a single concept. DCT uses a much smaller set of inputs to collect activations. The inputs can be specific to domain have wide coverage and do not require a specific relationship to the concepts of interest.
Location. The activations for SAEs are usually collected from the residual stream but have also been taken from the outputs of the attention layers (Kissane et al.) 2024) and MLP layers (Bricken et al.) 2023). DCT collects activations from the residual stream. DCT uses a much smaller set of inputs to collect activations from the residual stream.
4.3.2 Discovering Features
An unsupervised learning algorithm is applied to discover features o in the activations A.
Goal. SAEs are trained to take the model's activations, project them into a higher dimensional and sparse space, and then project them back. The goal is to disentangle the representations of concepts that are superimposed on each other. Since the dimensionality of the SAE is much larger than that of the LLM, it can represent each concept as a monosemantic direction while retaining the same information as the LLM.
The key idea of DCT is that perturbations along semantically meaningful directions at one layer will cause large, consistent, and predictable changes at later layers, whereas the effect of meaningless perturbations is inconsistent and quickly diminishes over the layers.
Loss. The SAE is trained to have a low reconstruction loss and to use sparse features. The reconstructed activations should be very similar to the original activations, and any feature in the SAE should only activate on a few inputs.
In DCT, set of vectors is optimized over set of inputs so that adding them to the activations at an early layer causes large change in later layers. The vectors are optimized to have effects that shallow MLP model can predict with high accuracy. Furthermore, orthogonality between vectors is enforced to find diverse set of features.
Learned Objects. SAEs learn an MLP that projects from the LLM's activations into the SAE space and one MLP that projects back. The directions in the SAEs space can be seen as features in this learned dictionary. Idcally, they should denote representations of concepts within the LLM and can be used steering vectors. DCT produces vectors that can be added to the model's activations to achieve specific downstream effects. They can be used directly as concept operators.
4.3.3 Identifying Concepts
Lastly, one needs to identify which concepts the learned features o correspond to, thus deriving the concept operator of. This requires a method for identifying the features and a source information about the concept.
Identification One can inspect the inputs and outputs on which the feature activates most strongly. However, Durmus et al. (2024) find that the context in which feature fires does not always predict its effect. Alternatively, one can increase the value of the feature and observe the effect on the model's outputs.
Other methods identify feature directions that have a high mutual information with the target concept (Zhao
et al. 2024), optimize a collection of features ures to lead to desired outputs (Kharlapenko et. al. 2024), or select the features with the highest probing accuracy for a concept (Wu et al. 2025)
Labels. Early works viewed highly activating inputs and outputs manually determine the corresponding concept. Current work leverages automated interpretability by using I.I.Ms to make these judgments. Other
16

---

Published in Transactions on Machine Learning Research (09/2025)
methods require a pre-existing dataset labeled for the concept. For example, Wu et al. 2025 build dataset of positive and negative examples for a concept to determine the probing accuracy of leatures.
4.3.4 Applying SAEs
Once one has used SAEs identify a feature that represents the targeted concept, the direction of that feature can be used concept operator. However, using that concept operator for steering requires to translate every activation into the SAE space and back. Therefore, there have been attempts to use SAE features to improve steering vectors in the LLMs activations that were identified by other methods. Chalnev et al. (2024) derive steering vectors that activate desired SAE features. Coomm & Nanda (2024) decompose a steering vector into SAE features and remove features that seem unrelated to the desired concepts. However, this challenging, as the steering vectors fall outside of the SAEs training distribution and contain negative feature directions which are not accommodated in the SAE (Mayne et al.) (024). Kharlapenko et. al. (2024) address this by optimizing the SAF reconstruction of the steering vector to achieve good downstream performance while incentivizing sparsity, leading to cleaner reconstructions.
Notably, SAEs have recently been extended to discover circuits of features implemented over adjacent layers (Dunefsky et al., 2024) or multiple layers (Lindsey et al., 2024). Furthermore, there has been continued progress for better SAE architectures [Rajamanoharan ct al 2024a b). This progress will hopefully lead to belter methods for identifying and steering LLMs.
Many human-undersstadddbee concepts, such famous people scam emails, have been identified through SAEs Templeton et al. 2024). However, the method does not guarantec to find a specific concept of interest. An engineer might be interested in the representation for honesty but fail to find one in the SAE. Furthermore, while many papers motivate SAEs with the ability to steer representations, few of them actually evaluate whether the discovered features allow for effective Representation Control. To improve the applicability of SAEs for feature steering, it should be a standard to evaluate the effects of features on models' outputs.
Takeaway: Representation Identification
To identify the representation of a concept, methods need to provide a stimulus related to the concept, collect the model's activations, and calculate a concept operator from them. There are three types of methods:
1. Input Reading: Construce sets of inputs related to the concept, collect their activations, and calculate the concept operator from the sets activations.
2. Output Optimization: Collects activations and outputs, scores the outputs according to the con- cept, and optimizes a concept operator for that score.
3. Unsupervised Feature Learning: Collect activations, discover features in the activations with an unsupervised learning algorithm, and identify which concepts the features correspond to.
5 Representations Operationalization
RepE assumnes that the model represents the concept of interest in its activation space. However, it is unclear how exactly the model's representations are structured. RepE methods for identification and control bake in assumptions about the geometry of representations used by the model. These representations are operationalized in RepE methods with a concept operator that has a specific shape and intended relationship to the models' representation the concept.
5.1 Assumed Geometry
Assumptions about the geometry of internal representations are included in the choice of RI method and steering function. For example, if the weights of a linear probe are used as a concept operator, it will not able to pick up on non-lincar aspects of a concept's representation. Similarly, an activation stecring function only increases along linear direction, would fail if the concept was represented along non-cuclidean space.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: Differences between approaches for operationalizing representations (the output of the Represen- tation Identification step).
Aspects of Representation Operationalization
How do methods differ?
Assumed Gcometry
Features as Linear Direction, Fuzzy clusters or Non-lincar manifolds
Shape (Vector, Matrix, Vector + Matrix)
Choice of Concept Operator
Relationship with concept. representation (Exact match, Capture co- variance, Capture principled direction and variance)
There is an ongoing discussion in the interprctability community about the geometry that transformer models use to represent internal features. The Linear Representation Hypothesis poses that concepts are represented as linear directions in the activation space, and their intensity is denoted by the magnitude of the activations in that direction (Park et al., 2024b). Although this hypothesis has some merits and, if true, would make RepE casier, it has also been criticized (see Section 12 for discussion). Thus, RepE methods with alternative feature geometries have been proposed. They assume representations to be fuzzy, clusters (Post & Abreu, 2024), or non non-linear manifolds (Qiu et al.) 2024). Other possibilities for feature geometries include circles EEnell et al. 2025), onions (Csordás et al. 2024), non-lincar paths, or subsets of a vector space. See Section 13.2.2 for consequences of these assumptions and Section 14.2 for possibilities for extension.
5.2 Concept Operators
The concept operator is the object that denotes the model's representation of the concept. As the output of RI and the input for RC, it is the linking element between the two parts of the RepE pipeline. Concept operators can take different forms like vectors, matrices, or combinations thereof. Furthermore, they differ in the relationship they are intended to have to the model's representation.
Vectors. The most common type of concept operator is a vector of E R4, where is the dimensionality of the activation space. Often, this is used by methods that assume that representations are linear directions in the model's activation space, which vector can describe. Thus, the concept. operator vector should be as similar possible to that direction. Other methods follow Li ct al. (2023a) and cmploy one vector per atlention head (Guo et al. 2024] Wang et al.) [024c] Li et al. 2024d). Furthermore, some mods denote a concept representation through a set of vectors that represent different aspects of the concept (Chen et al. 2024b) or serve different functions (Wu et al.) (2024a). Cai ct al. (2024) use a set of vectors that were optimized to steer activations towards specilic outputs to train a LoRA-adapler that produces similarly steered activations.
Matrix. A range of matrices have been used as concept operators df Raxd Postmus & Abreu (0024)
operate on the assumption that features are fuzzy clusters in the activations. They use positive semi- definite matrix called a conceptor that should capture the principle directions and variances. This allows to more precisely steer complex representations since conceptors also capture the underlying correlations between activations. Xiao et al. (2024) two Gaussian Models that respectively model the distribution of positive and negative activations. This retains information contained in the distribution of activations while allowing activations to be mapped from one distribution to the other. Qiu et al. (2024) employ 2 matrices that capture covariance with positive and negative activations, respectively. These matrices can be used to project new activations to maximise their covariance with positive demonstrations and minimize covariance with negative demonstrations. Lastly, Rozanova et al. (2023) learn a Nullspace Projection Matrix of which combines nullspaces that are orthogonal to the feature direction ofa = (1- - (of- ..., of))a - a.. This captures information in the activations that a probe could use to predict the concept of interest. Thus, the matrix can be used to project out information about the concept.
Matrix + Vector. Some methods employ both a matrix and a vector to denote the concept representation most commonly to perform an affine transformation where the activations are multiplied with the matrix and
18

---

Published in Transactions on Machine Learning Research (09/2025)
then added to the vector (Dong al., 2024b). Singh et al. (2024) and Avitan et al. 20224 find that matching the mean and covariance of negative and positive activations allows one to steer the model's representations. Thus, they derive a matrix that captures the covariance of the concept's representation and a vector that captures its mean. This allows to steer the representation with an affine function. Wu et al. (2024c) extend this by learning an additional matrix that. serves to project the activations and out a space where the affine transformation more effective. Hernandez et al. (2024) employ matrix to amplify directions of an attribute that are relevant a concept entity and a bias term that better fits the transformed attribute into the activations. This assumes representations of entities and attributes to be represented linearly, Alternatively, Pham & Nguyen 2024) assume that representations are directions of activations from the origin and that the magnitude along that direction reflects the intensity of the representation. They use the weights vector of a probe that separales the positive and negative regions in the aclivation space and an MLP that can predict how to adjust the angle of an activation vector.
Takeaways: Representation Operationalization
1. Representaiton Geometry: RepE methods make implicit, assumptions about the geometry of represontations. Many assume that concepts are represented as linear directions, but alternative perspectives exist.
2. Concept Operator: Most commonly, the concept operator is a vector denoting a linear direction. Others use matrices to capture principled directions, covariances or all linearly available information about a concept.
6 Representation Control
The goal of Representation Control (RC) is to steer the representations of the LLM in order to control its outputs. For this, it employs a stecring function that makes use of the concept operator to modify the weights or activations of the model. Additionally, RC methods decide when and where to apply the steering functions. The choice of concept operator of and steering function F determine the effect of RC and bake in assumptions about the geometry of model representations.
Throughout the section, we will describe the proposed steering functions along with the concept operators they use. We describe what effect the steering function has on the activation space and how this is supposed to control the representations of the model.
6.1 Modifying Activations
Activations can be controlled inference time with an activation steering function (a of) - of (see Definition 4 For a given input, we read the activations at an intermediate layer a; during the forward pass, perform some operation F it using the operator of, and then use the modified activations of as inpul for the next layer. By steering the activations, In our running example, we would take activations and steer them to be more similar to truthful activations and less similar to dishonest ones, thus causing later computations and the outputs to be more honest.
control the model's output with regard to the concept.
6.1.1 Activation Steering Functions
Linear Addition. This simply adds a vector of E IRd to the original activations ai + Xoς - af, where the factor À determines the steering strength. The model's representation is assumed to be a linear direction, and the concept operator vector that should denote that direction. Lincar Addition simply increases the magnitude of activation along the direction, which should increase the intensity of the concept in the model's representations.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Table 6: Differences between Modifying Activations methods for Representation Control.
Aspects of Modifying Activations
How do methods differ?
Operation (Linear Addition, Vector rejection, Affine transformation, ...)
Concept Operator (vector, matrix, vector+
Assumed Geometry (linear directions, mean&covariance matching) Intended Effect on Activation Space (scale direction, remove information, rotate direction)
Steering Functions
Intended Effect on Representations (inerease/docrease intensity, remove)
Layer & Model component (single or multiple layer(s) & component(s))
Location & Time
Token position (every token, one token before generation, specific token) Input dependent (independent, switch on/off, adjust intervention strength)
Linear Addition is the most popular steering function. Improving on this simple function, some papers renormalize the activations after the addition to a normal magnitude, thus avoiding reductions in quality
(von Rütte et al.) 2024; Adila et al. 2024b; Liu et al., 2024b) Leong et al. 2023 Chen et al., 2024b). Other
methods employ a set of vectors that are applied to individual attention heads (Li et al,, 2024d
2023a
Following Differential Privacy, Zeng ct al. (2024) adds noise to the vector to protect privacy.
Multiple papers attempt. to add multiple steering vectors at once to steer multiple concepts multiple
aspects of one concept. This can be done by simply adding all vectors at once [Wann et al. 2024c Scalena
ct al. 2024) or combining them in a weighted sum (Chen et al., 2024b Zeng et al. 2024). leads to interference between concepts and increases negative impact on the LLM's quality 13.1 p- van der Weij et al. (2024) avoid this by adding different vectors at different layers. Adiia et al.) (2024b) decompose the different vectors into their orthonormal basis and use those for steering.
However, this
(see Section
ar-of Vector Rejection. The vector of can be rejected aj of of + af, by first projecting at onto of before subtracting. This removes the component. of at that is parallel to of, leaving us with the part of ot that is perpendicular to of. Assuming linear feature directions, this removes the influence of the concept in the representations (Arditi et al. 2024 Deng et al. 2025). Equivalently, vector scaling at + ar-of of + af amplifies the components of the activations that are aligned with the concept operator (Chu al. 2024
of
Nullspace Projection. This employs Nullspace projection matrix of that captures information that probe would use to classify the concept in the activations. Multiplying activations with this matrix removes the information of that concept ofat- - af (Rozanova et al. 2023). thus removing the influence of that concept on the model's outputs.
Soft Projection. This multiplies the activations with a conceptor matrix (discussed Section 5.2 that encodes the principal directions and variances of concept set of activation vectors. This scales the acti- vation vector along the directions relevant to the concept while not completely overwriting the non-concept directions. Thus, it allows for more nuanced control over representations (Postmus & Abreu 2024
Rotation. Pham & Nguyen 2024h argue that rotating activations into the appropriate direction from the origin is superior to shifting the direction directly since rotations achieve the desired direction while retaining the magnitude of activations. For computational efficiency, they approximate such a rotation through a reflection followed by an adjustment of the angle. Undesired activations are reflected along a hyperplane into the desired region of the activation space. Next, learned MLP determines how to adjust the angle of the reflected activations to attain the desired activation.
A ffine Transformations. Affine Transformations for activation steering employ a matrix o.- and vector of control the activation aj through an affine transformation o. at + 0 + af. Singh et al. (2024) apply affine steering to match the mean and covariance of two representations. They prove this optimal
20

---

Published in Transactions on Machine Learning Research (09/2025)
steering function since it guards against linear probes for that concept, thus removing any linearly available information about the concept. Similarly, [iao et al. (2024) recenter and then rescale activations to fit into the distribution of positive activations. Affine Steering has been used to manipulate facts the model associates with an entity (Hernandez et al.) 2024), project a steering vector into the activation space of another model (Dong et al. 2024b), or to create counterfactuals with regard to sensitive altributes (Avitan ct al. 2024).
Projecting Before Editing. Another approach is to project activations into space that allows for better editing. Qiu et al. 2024) first project into non-linear space, before they project activations into directions that co-vary maximally with positive demonstrations and minimally with negative ones. AAora et al. (2024) project different activations onto the feature vector to then subtract them without influencing other feal.ures
In section 1.3 we discussed that SAEs can learn vectors that correspond to interpretable concepts. However, these vectors are in the representation space of the SAE. If one wants to use them to steer the normal model, it possible to project the model's activations into the SAE space, apply the steering on the feature of interest, and then project them back. However, this comes with larger decreases in capabilities due to the reconstruction error and at higher computational cost.
6.1.2 Location & Time
Layers and Components. Representation Control methods can modify the activations at different layers components. While some methods only steer the activations at single layer and component (Panickssery
et al.
2024 Zhang et al.) 2024a Cao et al.
2024a Ball et al. 2024). others modify activations many
or all layers and components (Konen et al.
2024 Chen et al. 2024a Liu et al. 2024a Adila et al. 2024a
Qiu et al. 2024 The optimal location for steering can be found by measuring steering effectiveness (Zhang
et al. 2024a), probing accuracy (Zhang et al. 2024b), inspecting the projection of each layer's 's steering vector
to the vocabulary (Cao et al., 2024b), or specific loss functions (Wang & Shu
2023
Adila et al., 2024a).
Tokens. Furthermore, one could steer at different token positions. Some papers decide to steer at every
token position during generation (Bortoletto et al., 2024 Adila et al. 2024a Wang & Shu 2023 While this can be more effective (Subramani et. al. 2022), it also increases the impact on capabilities. Others only steer at the last token of the query, hoping that the steering effects propagate throughout the generation (Wang ctt al. 2024c Adila et al. 2024b: Li ct al. 2024c). Some papers chose to steer at or before a token that gives a prediction or answer (Paulo et al.] 2024 Lucchetti & Guha, 2024).
Dynamic Activation Steering. For most activation steering functions, the intervention does not depend the input. This is suboptimal since some inputs might require different steering directions strengths. Dynamically adapting activation steering to the present context is an emerging theme within RepE since it can increase steering effectiveness and reduce unwanted side effects.
Multiple papers decide whether steering is necessary for an input based on the cosine similarity between the activations and the steering vector (Adila et al. 2024a Wang et al. 2024b), between the activations and the average of steered activations (Ma al. 2025) or depending on the output of a probe given the
activations (Stickland et al. 2024 Li al. 2024d] Pham & Nguyen 2024 Nguyen et al. 2025). Similarly, the intervention decision can be made separately for each layer component (Zhang & Viteri 2024 Simhi et al., 2024). Instead of binary decision for intervention, other papers modulate the strength of the intervention. This can be based on cosine similarity (Cao et al.) 2024b), KL-divergence between steered and unstecred output (Scalena et al.) 2024), the L2-norm of positive and negative activations (Leong et. al.] (023), or by trying different strengths and evaluating the outputs (Fatahi Bayat et al.) 2024). When multiple operators are available, the weight assigned to cach operator can be derived dynamically (Zeng et al. 2024 Wang et al. 2024cc.. Lastly, the operation itself can be dynamic, like using an MLP to predict the angle by which the activations are rotated (Pham & Nguyen 2024
21

---

Published in Transactions on Machine Learning Research (09/2025)
Table Differences between Modifying Weights methods for Representation Control.
Approaches Modifying Weights
How do methods differ?
Training Target (activations and/or outputs)
Training weights
Trained weights (low-rank adapter, all weights)
Editing weights
Operation (weight orthogonalivation, addition)
6.2 Modifying Weights
The representations of a model can be steered by modifying the weights with weight steering function F" (mn, of) wf- In our running example would modify the weights of the model that it produces activation patterns and outputs that are aligned with honesty.
6.2.1 Training Weights
The weights can be modified through fine-tuning, This training is either aimed at producing desired activa- tions or optimizes for specific outputs. Furthermore, fine-tuning methods differ in which weights or adapters they train.
Training Towards Desired Activations. Weights wr can be trained to produce desired activations of by weight steering function. A dataset of desired activations can be gathered by applying activation steering function Fa of) - af. Zou et al. (2023a) optimize a low-rank adapter (LoRA) to generate activations with low L2-distances to steered activations. Similarly, Cai et al. (2024) train a LoRA adapter on a dataset of optimally steered activations so that the resulting activations are similar to the optimally steered activations. Alternatively, Zhang et al. (2024c) design an Adversarial Training setup where a LoRA adapter is trained generate activations that discriminator classifies as aligned with the concept.
Some weight steering functions do not rely concept operator of. Instead, they train weights to produce activations similar to random noise. Zou et al. (2024); Li el al. (2024b); Rosati et al. (2024) fine-tune LLM so that activations on harmful inputs or information that should be unlearned are close to noise. This should remove harmful representations from the model.
Training Representations without Concept Operator.
Training Towards Desired Outputs. Multiple methods propose to fine-tune a small number of added parameters towards dataset of desired outputs. Wu et al. (2024c) propose ReFT,, which trains an adapter to bring the activations into subspace where lincar projection and addition applied. YYn et al. (2024) fine-tune only the bias terms of selected attention heads. Wu et al. (2024a) train adapter consisting of scaling and a. bias vector. While these methods are inspired by RepE, and aim to control the model's representations, they are also end-to-end parfintterefffcientt fine-tuning methods like LoRA adapters (Hu let al. 2022). This is shown by Jiang et al. (2024b), who draw a theoretical connection between LoRA and ReFT and unify them into a meta-algorithm for model efficient fine-tuning. This highlights that weight stocring functions that optimize for specific outputs can be seen as being on a spectrum between RepE and fine-tuning.
Training Towards Desired Activations and Outputs. Other methods optimize both for desired ac- tivations and desired outputs. Ackerman (2024) optimize weights to produce activations with high cosine similarity to the concept operator vector while also retaining a low cross-entropy loss in the outputs. Yu et al. 2024) repeatedly ablate the vector that represents refusal while training the model to refuse harmfull answers. This should force the model to learn more robust representations of that concept. [Stickland et al.) (2024) mitigate performance loss by training a LoRA adapter that minimizes KL-divergence in outputs when steering vector is applied to the outputs of the original unsteered model.
22

---

Published in Transactions on Machine Learning Research (09/2025)
6.2.2 Editing Weights
Tt is also possible to edit the weights directly without training. Arditi al. (2024) apply weight orthoga ization wi - offff wi + wf to the MLP weight matrix that writes to the residual stream. This modifics the component weights to never write the direction of. If of captures the direction with which the model rep- resents a concept, this operation removes the concept from the model's representations. [Wang et al (2024a) simply add the weights of a probe of to few selected row vectors within the gated projection matrix of the MLP weights. This shifts activations in the hidden state of the MLP away from undesirable regions and towards desirable regions.
Takeaway: Representation Control
Representations be controlled by stecring the activations weights.
1. Activations can be steered during inference by addition, rejection, or projection, thus increasing, removing, or adapting the concept representation. Activation steering can be applied at different tokens and dynamically adjusted for new inputs.
2. Weights can be steered by training them to generate desired activations and outputs by editing components of weight matrices.
7 Practical Representation Engineering Pipelines
In this section, we describe some prototypical RepÉ pipelines and classify how they fit into our taxonomy, We compare them according to selected criteria. In addition, gather empirical evidence to answer which design choices in RepE work best.
7.1 Prototypical RepE Pipelines
In Table S we classify RepE pipelines according to which strategy they employ for Representation Identifica- tion, the shape of their operator, and whether they control activations or weights. We score Representation Identification methods on whether they require dataset labeled according to the concept and whether they require expensive optimization. Additionally, resentations or capture non-linear ones. Furthermore, we score whether controlling the model requires retraining of its weights and whether the operation to control the model is dependent on the given inputs. Lastly, we determine the types of concepts the method was initially designed to engineer. However, this does not mean that they are not able to engincer other concepts. We select these RepE pipelines to show a diversity of approaches while preferring highly-cited papers that were published at top conferences.
whether the method is restricted to linear rep-
Contrastive Activation Addition (CAA). CAA (Panickssery et al. 2024) provides positive and negative examples of behavior, collects their activations, and takes the mean difference between the 2 sets of activations to calculate vector. This vector is used as concept operator that is added to the activations during inference, thus steering the model with regard to high-level behavioral characteristics.
Low-Rank Representation Adaptation (LoRRA). LoRRA [Zou et al. 2023a identifies the operator by adding positive or negative pre-prompts inputs, collecting their activations, and taking the first principle component of the difference belween the two sets of activations. Instead of directly adding the vector to the activations, they train LoRA adapter to output activations that are similar adding the vector to the original activations. This can steer range of high-level concepts, including honesty, emotions, and ethical values.
Inference- Time Intervention (ITI). ITI Li et al. 2023a inputs questions and answers that do or do not correspond the concept. Then, a lincar probe is trained on the activations of each attention head predict whether the answer was or was not related to the concept. They select the attention heads with the highest probing accuracy and use the weight vector of the probe a concept operator. During inference.
23

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: Prototypical methods for RepE classified according to our taxonomy and the criteria they meet.
Originally proposed for
Method
Repreentation Identification
Represennlations Operationmliattion
Reprecutation Coutral
Works with unlabeled dataset?
Optini- zation Free?
Allows
Context- adjusted steering?
Concept Operator
Training- froe?
Name
Type
non-linear represeutations?
Type
CAA
Pan-
Jelssery
sl.
Iuput
Reading
Vector
x6
Activations
High-level behavions
2024
ct al.
LoRRA
2023a
Zon
Input
Reading
Vector
36
Wrights
High-level bebaviors
IT1
al.
Input
2023a
Reuding
x3
X
Vectors
Activations
Truthfulness
Uusupervisd
Temple- 2024
Abstracs fea-
Feature
X4
Vector
Activations
Learning
LoReFT
Oulpul
Matrices
Activations/ Wrights
Improving task performmance
el ml.
2024
Optimization
X4
vector
BIPO
Cao
Output
Optimization
X
Vector
10
Activations
56
Aligning to preferences
SEA
Qui et al.
Input
matricrs
Activations
High-level bebaviors
2021
Reading
MiMiC
Singh
Input
Matrix voctor
Removing Tuxicily
Activations
2024
Reading
these vectors are added to the activations of their respective attention heads. This method was first proposed to make models more truthful.
Sparse Auto-Encoder (SAE). SAEs (Templeton et al. 2024) are trained in an unsupervised fashion on a large dataset of activations learn internal features. Activations are projected into wider feature space, which is optimized so that features are sparse and can be used to reconstruct the original activations. We can identify the concept feature corresponds to by investigating inputs or outputs for which the feature is highly activated. These features correspond to vectors in the embedding space of the SAE and can be used to stimulate the model's activations. SAEs find wide range of concepts at different levels of abstraction.
Low-rank Linear Subspace Represcntation Finetuning (LoReFT). LoRcFT (Wu et al. 2024c) lcarns an adapter that intervenes with the activations during forward pass. The adapler first projects the activations into a linear subspace, edits the activations with an affine transformation, and projects them back. The outputs of the adapter are added to the original activations. For this, low-rank matrix, projection dataset that exemplifies the
matrix and bias vector are trained to minimize the cross-cntropy loss concept. This method was proposed to improve performance on specific downstream tasks.
Bi-directional Preference Optimization (BiPO). BiPO (Cao et al. 2024b optimize a vector so that adding the vector to the activations makes the desired outputs more and the undesired outputs less likely. Subtracting the vector should have the opposite effect.. After optimizing with Gradient Descent, we get a vector that represents the preferences the provided datasct. During inference, the model's 's representations are controlled by adding the vector to the activations. This was proposed to efficiently align LLMs with human preferences.
Spectral Editing of Activations (SEA). SEA QQu et al. 2024) collects activations for neutral question and for corresponding positive and negative responses. By applying Singular Value Decomposition on the covariance matrices between negative and neutral, as well as positive and neutral activations, they identify a positive and negative editing matrix. During inference, they can select directions in the activations that co-vary with the editing matrices and prune negative directions while retaining positive ones. They extend this to non-linear directions by first transforming the activations into non-linear space where the editing
24

---

Published in Transactions on Machine Learning Research (09/2025)
can be performed before transforming the edited activations back. While SEA is not designed for particular types of concepts, it was initially used to steer truthfulness and bias.
Minimally Modified Counterfactual (MiMiC). MiMiC (Singh et al.) 2024 divides activations into a positive and negative set based on an MLP probe trained to detect the concept. They then derive an affine steering function that matches the mean and the covariance of the two sets of activations while causing minimal distortion. On new activations, this function performs projection and translation, steering it towards the target concept while preserving key statistical properties. It was initially proposed to control representations of toxicity and bias in sensitive concepts.
7.2 Which Methods Work Better?
In this scction, we summarize previously reported results that compare different methods. While this section gathers existing empirical evidence for comparison between RepÉ methods and implemennattinss, we often cannot make conclusive statements about the superiority of one method. This is because of lack of unbiased comparisons (sce Section 14.5 for discussion).
7.2.1 What Works for Representation Identification?
Is it more effective to use Input Reading or Output Optimization? Depends on the concept. Multiple papers claim that their method based on output optimization improves on input reading methods (Yin et al. 2024 Cai et al.) 2024 Cao et al.] 2024a Acckerman 2024). However, we lack evidence for the contrary, which is likely because papers that propose input reading techniques did not compare their methods to output optimization mothods. Two unbiased comparisons find that an input reading method is better at steering the sentiment of text (Konen et al. 2024) and that output optimization methods control the
concrete topics more effectively (Wu et al. 2025
Are steering features from Sparse Auto-Encoders better than other RI methods? - Possiiby a combination.
In Zhao et al. (024) and Makelo et al. 2025). an SAF-based steering is more effective than input-reading-
based RI methods. In contrast Chalnev et al. (2024) find that an input-reading based RI method leads to more effective steering than SAE features. However, results in Chalnev et al. (2024) and Kharlapenko et al. (2044) suggest that SAEs can be used make input-reading- based concept operators more effective, Finally, evaluations the AxBench (Wu et al., 2025) find that SAFs perform worse concept detection and steering than LoReFT and class means but better than methods using probing or dimensionality reduction.
Which prompting format works best? - Unclear.
Braun et al. (2025) compare control effectiveness for seven prompt scttings varying in whether an answer loken appended, whether an instruction is prepended, and whether few-shot demonstrations are included. While the setting that only appends an answer token has a slightly larger average control effectiveness, they conclude that no prompt type clearly outperforms the other. Waan & Shu (2023) find that choice prompts, where the model has to select answer A or B, outperform free-form answers. This is likely because the choice prompt concentrates the relevant context. into single token and thus more effectively triggers the concept representation. The lack of principled evaluations of optimal prompting formats for eliciting representations makes this an important area of further study.
At which token positions should activations be read out? Unclear.
Some evidence suggests that averaging the activations of multiple tokens is better than just using the last token's 's activations (Hoscilowicz al. 2024). Furthermore, focusing on the last token given by the user can be more effective than taking activations from an appended post-prompt like "I think the (attribute) of this user is" (Chen et al. 2024a) Braun et al. (2025) do not find a clear difference between reading activations at the last token of the question or at the token containing the answer.
Which function is best for calculating the difference between sets of activations? -Differnce- in-Means.
In Input Reading, functions like Difference-in-Menee (DiM), linear probes, Principle Component Analysis, Contrast Consistent Search (Burns et. al. 2023) are used to identify the concept representation from
25

---

Published in Transactions on Machine Learning Research (09/2025)
2023a Marks contrasting sets of activations. This has been experimentally studied in 7 papers (Li et al.
& TTggaark 2023 von Rütte et al. 2024 Xu et al.] 2024a Arora et al. 2024 Wu et al.] 2025 Im & Li 2025). In 6 out of 7 papers Effeeeenceinn-Maap leads to the strongest performance. The next best seeins to be taking the weight vector of a linear probe. PCA and CCS tend to perform worse. A fair comparison between DiM, PCA, and classifier-based methods finds that DiM is best at positively and negalively steering multiplo-choice questions, equals the performance of probes at steering long- forg-form generations, but leads to the largest performance deterioration on positive examples (Im & Li, 2025). Lastly, Distributed Alignment Search (DAS) [Geiger et al. 2024b) was only tested once in which it performed the strongest, highlighting that DAS remains underexplored. Besides Input Reading, it is advantageous to apply mean-centering to the steered activations as shown by Jorgensen et al. 20244
7.2.2 What Works for Representation Operationaliiation?
Are matrices or vectors better concept operators? Matrix more effective but more costly. Multiple works that propose using matricos instead of vectors as concept operators find that they improve steering effectiveness (Post.us & Abreu 2024 Rajendran et al. 2024 Pham & Nguyen 2024). Results from concept operator, but is clearly outperformed by vector that is derived from additional forward passes for every new input. While matrices can be more effective, they can also come at an increased cost. (Postamus & Abreu 2024) or reduced output quality (Rajendran et. al., 2024).
Zou et al. (2023a) are more ambivalent, finding that a matrix slightly outperforms
Are linear or non-lincar concept operators better? Non-lincar.
While most methods perform linear operations on the representations, the direct comparison between linear and non-linear methods favors the non-linear ones. Non-linear Spectral Editing of Activations (SEAA) out- performs linear SEA (Qiu et al. 2024), and non-linear ITI is more effective than ITI, although this comes at higher cost in capabilities (Hoscilowicz et al. 2024
7.2.3 What works for Representation Control?
Is it more effective to modify weights or activations? Unclear.
This question is still open. [Wang et al. (2024a) find that their method for controlling model weights is more effective than intervening on activations while leading to a larger reduction in output quality. However, a broader evaluation of that question is necessary.
Which steering function should be used for modifying activations? Unclear.
Zou et al. (2023a)) find that a piece-wise operation is more effective than linear addition. Chu et al. (2024) find that projection outperforms a product, which again is more effective than lincar addition. Luo et al. (0024) find that linear addition more effective than orthogonal projections at a higher cost in capabilities. Linear addition remains the most popular activation steering function despite some other steering functions outperforming it. This warrants further investigation comparing different operations. Krasheninnikov & Krueger (2024) study different activation steering functions in toy classification setup. They find that more expressive steering functions have a higher performance ceiling and can control more complex tasks, but simpler steering functions are effective with lower amounts of data.
At which token positions can activations be steered most effectively? - Apply steering at every inference step.
Overall, it is more effective to continuously apply RepE at every inference step than to only apply at the first token (Subramani al., 2022). This is because the steering effect diminishes throughout the generation (Scalena et al. 2024). However, only intervening on the first token can still be very effective (Subramani ct al. 2022) and has lower impact on output quality (Scalena et al. 2024).
Which component of the transformer should be steered? - Unclear.
RepE often modifies activations in the residual stream, the attention or the MLP block, but it is unclear which component is most effective to stoer. Existing evaluations only find that steering at the LM head or embedding layer is ineffective (Subramani et al., 2022) and that steering both the attention and MLP blocks is more effective than only intervening one of them (Zhang et al. 2024b).
26

---

Published in Transactions on Machine Learning Research (09/2025)
Table 9: Source labels and metrics for evaluating LLMs on different generation
Form of generation
Free generation
Likelihood on specified answers
Source of Labels Metrics
LLM-judge, text classifier, word occurrences Judge Score, # of concept-related words
correct answers, ground-truth text Accuracy, Logprob-based metries
Should only one or many layers be controlled? Many layers.
It is generally assumed that intervening on all layers once leads more effective steering a higher cost in the general capabilities of the model. However, to the best of our knowledge, there are no experiments proving this. Notably, many concepts can be most effectively steered in middle layers (Panickssery et al. 2024
Im & Li 2025).
8 Evaluation of RepE Methods
Good practices and resources for evaluations are key drivers of progress for any area of Machine Learn- ing. However, for RepE evaluations are not standardized and metrics are still developing. This section outlines current and best practices for evaluating RepE methods and describes existing and potential future benchmarks.
8.1 Common Evaluation Methodologies
The general setup for evaluating RepE methods involves measuring the steering effect, measure of how much the behavior of model with regard to the concept of interest was changed. For this use the model before and after intervention to generate outputs on a dataset or task that is often related to the concept. For the generated outputs we use metric that determines in how much the outputs are aligned with the concept. Additionally it's common to evaluate in how much the general model quality deteriorates.
8.1.1 Evaluation Task and Dataset
Depending the concept of interest, different tasks and datasets are used. It is common to established datasets and benchmarks that cover the steered concept targeted downstream application. Commonly used datasets for evaluating changes in concept are TruthfulQA (Lin et al. 2022) for increasing truthfulness, AdvBench (Zou et al. 2023b) for preventing harmful outputs or BOLD (Dhamala et al. 2021) for reducing societal bias (see Section 10 for more common datasets per concept). Other papers devise their own specific tests and datasct, for example measure causal efficacy of an identified concept operator (Arora et al., 2024), to evaluate steering effect wide range of concepts (Wu et al., 2025) to study RepE in toy setting Krasheninnikov & Krueger 2024
A key difference between evaluations is whether the model is asked to generate answers to multiple-choice questions or make longer generations. While evaluations multiple-choice questions are convenient to run, they are less faithful to downstream settings in which a model will be employed (Pres et al. 2024).
8.1.2 Measuring Steerability
To evaluate the steering effect, is necessary to measure in how much model's outputs are aligned with the concept. This requires a way to label outputs depending their alignment with the concept and to construct a metric out of this. A key difference in evaluations of the steering effect is if the model gets to freely generate any output that then gets evaluated or whether we are evaluating the likelihood of the model generating specific pre-defined answers.
Source of Labels Evaluation methodologies differ on how they determine the amount of concept alignment of an output. Sources of labels can be LLM judges, ground-trute labels or text of a datasets, and word occurrences in generated text.
27

---

Published in Transactions on Machine Learning Research (09/2025)
LLM judges are commonly used to evaluate freely generated text. Many papers instruct a frontier LLM such as GPT-4 to give numerical judgment about how much a generated text aligns with the concept of interest (Wu et al. 2025). Other papers use smaller models that are specialized for judging a specific property and have been validated for this task. For example Wang & Shu (022)) use small, specialised LLM for toxicity classification (Caselli et al., 2021) and a lexicon and rule-based tool for sentiment scoring (Hutto 2020).
Many papers use exiting labels in a dataset. In multiple-choice benchmarks such as MMLU, the correct labels can be used as ground truth to judge the models performance. Furthermore, on a texts related to the concept, the ability to correctly predict the correct next token can be taken a source of information about the concept alignment of a model (yan der Weij et al. 2024). Other papers judge outputs based on how often specific concept-related words occur. Human judgments are rarely used, since they are expensive. Only Wang et al. (2024c) use a small set of human evaluators to confirm that their automated metrics align with human judgment.
Metrics for the Steering Effect Depending whether pre-specified or freely generated outputs are used, different metrics for the steering effect are available.
Evaluation methodologies that use a judge for scoring usually simply take the score assigned by the judge as their metric. For example LLM judges are often instructed to provide a score (e.g. from 0-4), while specialized judging-models might directly output a concept score.
Some concepts can also be measured by detecting or counting concept-related words and phrases in freely generated outputs. A common technique for measuring the Attack Success Rate of jailbreaks to measure the look the presence of specific words and phrases such as 'I am sorry as LLM..'' (Arditi et al. 2024). Turner et al. (2244) count the number of pedding-related words and Jorgensen et al. of words related to specific words to measure whether they were able to shift the topic and genre of generated text.
2024 count the number
On multiple-choice questions many papers measure the average accuracy. On longer generations, van der Weij et al.] (2024) measure the difference in the top-1 accuracy at predicting the correct next. token on datasets that are or aren't related to the concept. However, this ignores shifts in the probability the model assigns to the correct answer. Thus, Paanicsscryy et al. (2024) evaluate steoring effect as the average likelihood the model assigns to the correct response. Tan et al. (2024) instead use the logit-differece propensily, which is calculated as the difference in logits between the positive and negative answer. They argue that excluding the normalization provided by the last softmax makes the metric more linear with respect to the model's 's activations. Alternatively, Turner et al. (2024) measure the probability of the correct answer being in the top-k tokens. Arora et al. (2024) devise a log-odds metric that measures the causal efficacy of an intervention on the model's 's representation. |Krasheninnikov & Krueger (2024) count the fraction of examples where the model can change the desired attribute without changing any other attributes. Finally, Press et al. (2024) take the difference in log- likelihoods for positive and negative continuations, normalize them and then take the mean difference between the basc and steered model.
8.1.3 Measuring Changes in Model Quality
Interventions on the models representations should have minimal impact on the general quality and capa- bilities of the model. RepE papers use a variety of measures to evaluate their impact on model quality.
For this, many papers report scores on popular capabilities benchmarks like MMLU (Hendrycks et al. 2021) or MT-Bench (Zheng et al., 2023a) as a mcasure of models quality. Other papers evaluate the models capability to correctly predict the next loken by running on general dataset like the Pile (Gao et al., 2020) and evaluating Perplexity(Scalenn, et al. 2024), top-1 accuracy (van der Weij et al. 2024) or the probability to have the correct token in the top-k tokens (Turner et al.] 2024). Furthermore, it is possible to evaluate specific aspects of model performance that could be deteriorated by RepE. Brumley et al. (2024) measure diversity of generations bi- and tri-gram entropies. Wu et al. (2025) evaluate the fluency and instruction following of the model via an LLM-judge.
Lastly, some papers decide to combine steering effectiveness and model quality into one performance metric for RepF. voo Rütte et al. (2024) propose perplexity-normaaizee effect size, which divides the increase in
28

---

Published in Transactions on Machine Learning Research (09/2025)
probability of the concept being present by the decrease in model fluency. Wu et al. (2025) report the harmonic mean of the score for concept-alignment, instruction following and fluency as combined metric.
8.1.4 Other Experimental Details
RepÉ methods are commonly evaluated on a range of hyperparameters and settings. For many RepE methods the stecring strength is hyperparamcter that guides how strong the intervention is. Altering the steering strength can outline the tradeoff between steering effect and model quality. Furthermore, this indicates the quality of the concept operator is, since increasing the steering factor should lead to monotonic even linear increase in concept-alignment (Pres et al. 2024 Tan et al., 2024) Other common hyperparameters to be altered are the sets of layers on which the intervention is conducted, the token position to intervene on, the number of samples used in Representation Identification and the family and size of models.
Hyperparameter Ablations:
Baselines: To measure the steering effect and reduction in model quality the steered model must be compared to the original model without intervention. However, we also need to compare to other control models. It's common to compare to carlier RepE methods. Furthermore, many papers make comparisons to prompling or fine-tuning methods.
8.2 Best Practices for Evaluating RepE Methods
Free generation vs likelihood of specified answers. Mcasuring the steering through measures that use pre-specified answers has the advantage that one can measure shifts in the log- ligllikelhood of the model. Pres et al. (2024) argue this is important because disregarding the confidences of the model, loses information about how variable behavioral expressions are. On the other hand, when none of the pre-specified answers has a high likelihood, such metrics would not take into account the actual generations model would produce thus miss important aspects of its behavior.
Using specialized judges. Many studies rely on LLM-as-a-judge setups, where a highly capable and general model instructed rate generated text. However, using LLMs-as-a-judge known be biased, sometimes inaccurate and is usually not scientifically validated as a metric (Fu et al., 2024 Zheng et al. 2023b). When possible it might be preferrable to use specialised models that have been developed and validated to score specific concept, such small LLMs trained for detecting specific concepts (Inan et al. 2023 Caselli et al. 2021 Dementieva et al. 2023).
Varying Stecring Strength. In many RepE methods it is possible to modulate the strength of the intervention. By showing the steering effect and model quality at different steering strengths readers can get a sense of the tradeoff between these properties. Furthermore, is useful to plot the relationship between steering strength and steering effect. If the steering effect monotonically, linearly or strongly increases with higher steering strength this showcases that the RepE can effectively modulate the desired concept. [Tan et al. (224) even propose metric based on this relationship. For the same reason it is informative to show the effect if a negative steering strength is applied.
Realistic and Difficult Test Settings. While it is convenient to use multiple-choice format to evaluate the steering effect, Pres et al. (2024) argue that it important to the effect on open-ended generations since this setting more similar to downstream scttings where RepE will be applicd. Additionally, it is preferable to evaluate RepE on difficult settings that are similar to real world deployment. For example, when using RepE to defend against jailbreaks it is important to use state-of-the-art attacks and evaluate multi-turn jailbreaks.
Reporting Changes in Models' Quality. Only a minority ( (35)) of papers report the change in models' capabilities after applying Representation Engineering. Since we know that RepE can reduce these capabili- ties, is important to measure the LLMs' quality to compare the merits of each method. Furthermore, is desirable to administer diverse capability tests, like measuring the ability to generate fluent text while also the evaluating the accuracy at answering multiple-choice questions.
29

---

Published in Transactions on Machine Learning Research (09/2025)
Measuring Generalization. RepE methods can struggle to generalize to different settings (see Section 13.1p. Thus, it is not sufficient to evaluate the steering effect another subset of the dataset used for identifying the concept operator. Instead studies should systematically vary aspects of the settings between Representation Identification and Control.
Comparing to Strong Baselines. When proposing a new method, its effectiveness should be compared to the current state-of-the-art RepÉ method. However, often papers only compare early methods that have since been surpassed. Similarly, when comparing to other familics of methods, studies should not compare to naive prompting or fine-tuning approach, but to the state-of-te baateoffthe-art method for their respective problem.
Evaluating Diverse Concepts. The effectiveness of RepE methods should be measured on a diverse range of concepts. Since steerability can differ between concepts it is difficult to assess the quality of a method on only one concept. Furthermore, it desirable to evaluate on different types of concepts (see Scction 9.1 to determine whether a method has strengths or weaknesses at stocring some type of concept.
Testing on Different Model Sizes. Appendix A.1 finds that most RepE experiments are conducted on models between 3-10 billion parameters. To prove the general effectiveness of a RepE method, it is necessary to evaluate it on models of different sizes. Specifically, larger models need to be tested to indicate that the method scales to frontier models.
Survivorship Bias. is rare for papers that propose new methods for Representation Engineering to demonstrate concepts or situations in which control is not effective. However, papers that set out to neutrally evaluate the method often find weaknesses or limitations not obvious from the original paper. This practice can give false perceptions and stifle progress in the field. Thus, we encourage authors to report mill results.
Reporting Variance. Tan et al. (2024) rightly point out that many papers only report on average steer- ability. However, it is essential to showcase the distribution of results so readers see whether a. method provides reliable steerability.
8.3 RepE benchmarks
Existing Benchmarks.
There are two benchmarks that attempt evaluate and compare RepE methods
WW et al. 2025 Im & Li 2025
AxBench (Wu et al. 2025 is a benchmark that evaluates the ability of RepE methods to identify concept operators that accurately detect the presence of a concept and steer the LLMs' behaviour. For this, they built a synthotic dataset of positive and negative examples for 500 concepts. Using this datasct, 9 RepE methods are used to derive concept operators and compared to prompting and fine-tuning. Steering ability is determined by generating responses to instructions while performing Representation Control and then using an LLM to score the generation on presence of the concept, instruction following and fluency of the generation. For concept detection Differenco-in-Means, Logistic Regression and ReFT-rl perform best. However, prompting outperforms all RepE methods at steering LLM behaviour to a concept. Notably, all tested concepts are of similar type, testing whether a concrete, narrow topic is contained in text. This benchmark could be extended by consider different types of concepts.
Im & Li 2025 evaluate four methods for Representation Identification on their ability steer outputs positively and ncgatively towards a concept on multiple-choice and open-ended generation tasks. They find that Difference-in-Meanse oulperforms other methods to identify a steering vector. This benchmark could be extended by comparing different methods for Representation Operationalization or Control and evaluating them on more diverse range of concepts.
Desireable Benchmark Properties. There is currently no benchmark that provides a way to compre- hensively evaluate the effectiveness of a RepE pipeline. Thus we are currently unsure which pipeline is most effective in which situations. A benchmark can also serve as a guide for the future development of RepE methods by providing clarity about the effectiveness of methods and being target to aim for. Such unified evaluations can also surface previously undiscovered limitations of methods (Brumley al. 2024
30

---

Published in Transactions on Machine Learning Research (09/2025)
Such a benchmark could also provide testing ground for rigorous comparison. One challenge in developing RepE. benchmark lies in providing fair comparisons between RepE, methods that have different requirements, assumptions, and goals. They might have different requirements on the available data, require different amounts of compute, or were developed to control different kinds of concepts.
An ideal benchmark for RepE, should cover a range of concepts of different. types since different RepE methods might steer different concepts more effectively (Brumley et al. 2024). Hereby, the focus should lie cases where RepE is foreseen to be practically useful, such improving truthfulness, changing goals, adapting situational awareness, removing societal biases. Furthermore, evaluations should cover multiple models, test the influence of datasct size, and show the impact of different steering strengths. Additionally, it should specifically evaluate the weaknesses of current RepE methods (see Section 13.1p. thus putting a spotlight on improvements on these challenges. Of course, this benchmark should make sure to follow best practices outlined above (Section 8.2
9
What Concepts Can Be Controlled with Representation Engineering?
9.1 Types of Concepts
To illuminate how RepE can be applied, we cluster the concepts that have been controlled using RepE into six categories of concepts. Table IO showcases that most RepE papers steer high-level behavioral concepts and that RepE can be successfully applied to all types of concepts.
Table 10: The number of papers that attempt to, successfully or not, steer a type of concept.
Type of Concept
Success
Failure
High-level behavioral characteristics
56
8
Tasks
11
4
Language and style
10
0
Knowledge and beliefs
9
1
Content
8
0
Values, goals, and ethical principles
7
1
Linguistic or grammatical features
6
0
High-level behavioral characteristics like harmfulness or honesty are the most popular category of concepts controlled via RepE. These concepts are fairly abstract, may be context-dependent and cannot casily be specified. They are in reference to the behavior exemplified by the model.
Tasks model should carry out, including tasks like reasoning, classification and associations can be induced by RepÉ. Such concepts specify an input-output function, for example, by finding a function vector that triggers the execution of task (Todd et al. 2024).
Language and style of a text, such sentiment, style, genre, or language can be controlled with RepE. Such concepts are high-level properties of the generated text.
Knowledge and beliefs encoded in a model about the world can be controlled with RepE. Such concepts reference what the model assumes to be true while generating text. This includes changes to the model's factual knowledge its situational beliefs (see Section 10.3).
Content of the model's outputs are amendable to RepE, like steering it to focus specific topic and contain not contain some contents. These contents are often not exact text, but higher-level concepts like "talking about weddings" or "social security numbers". For example, the Concept500 dataset. (Wu et al. 2025) was designed for evaluating RepE and contains positive and negative examples for 500 concrete topical concepts. Such concepts are properties of the generated text.
Values, goals, and ethical principles that are encoded in an LLM have been adjusted with RepE. This refers to overarching objectives and principles according to which the LLM is making its decisions. Tt. has
31

---

Published in Transactions on Machine Learning Research (09/2025)
been applied to implementing human preferences into the LLM's behavior (Cao et al. 2024a or adapting which ethical theories are used for decision-making Tlaie 2024
Linguistic or grammatical features like verb conjugations or the gender of nouns are controlled in multi- ple papers. Such concepts are specific to language, and their presence can easily be verified in the generated
text. While controlling them not of great interest itself, they are used as a concept for interpretability (Hao & Linzen 2023 or to evaluate RepE methods (Arora al. 2024).
9.2
Commonly Controlled Concepts
This section showcases concepts for which RepE has been appliod most often. For this, we selectod all stcerod concepts in the surveyed papers. Table 11 shows the concepts which have been controlled in 2 5 papers and how many of the experiments were successful failed in steering that concept. Here, success is defined providing a significant improvement in the mcasure of the concept compared to the uncontrolled model.
Table 11:
Commonly stecred concepts along with the amount of experiments where the concept was was not successfully controlled.
Target Concept
Success
Failure
Truthfulness
17
Harmfulness
17
Toxicity
16
0
Fairness
Refusal
Sentiment
Privacy
The most common concepts are Truthfulness, Harmfulness, and Toxicity. This is because they are of practical importance in the context of LLM chatbots and because RepE seems well suited to control them, but also because early seminal papers experimented on these concepts and later work followed. Furthermore, we see very high success rate for all concepts. However, this statistic should be viewed eritically since experiments that fail to control concept are less likely to be published. Nevertheless, it does provide evidence that these concepts can be controlled effectively.
Takeaway: What concepts have been controlled with RepE?
There are different lypes of concepts that be rolled with RepE, among which high-level behavioral concepts like Truthfulness, Harmfulness, and Fairness are the most popular.
10
Applications of Representation Engineering
After describing some popular concepts that can be effectively steered with RepE, this section describes concrete problems for which RepE has been applied.
10.1 Al Safety
AI Safety aims to prevent harm caused by with AI models. In the context of LLMs this often refers to preventing outputs that are harmful to the user, the use of LLMs for harmful applications, or untruthful and hallucinated statements. We also include the prevention of private data leakage and the refusal behavior of LLMS as safety concerns. Furthermore, the field studies threats of future advanced AT models such as deception or self-improvement current LLMs.
Reducing Harmfulness. Harmfulness is a broad category describing outputs that can cause some harm to users, other individuals, society. It includes more granular categories, such toxic outputs, wrong
32

---

Published in Transactions on Machine Learning Research (09/2025)
information, or private data leakage. To control harmfulness in general, one needs to identify and steer harmful representations (Cai et al. 2024 Zou et al.) 2023a Scalena et al. 2024 Chu et al. 2024 Beaglehole ett al. 2025: Deng et al.] 2025). This approach can be combined with other safety techniques such as safe-dccoding in a defense-in-depte depth approach (Bancrjee et al.) 2024j. RepE can control harmfulness in multi- modal LLMs (Wang et al. 2024b), and harmful representations identified in one language can be transferred to another (Xu et al.) 2024a). Furthermore, the effect of controlling the model's exhibited personality traits on the harmfulness of its outputs has been explored using RepE (Ghandeharioun et al., 2024 Zhang et al.
2024a).
Toxicity is a more specific concept referring to offensive, aggrossive, or inappropriate outputs. RepE has been used to decrease toxicity (Wang et al. 2024a; Qian et al. 2024 Singh et al., 2024 Luo et al. 2024
Jorgensen et al. 2024 Pham & Nguyen 2024 Li et al. 2024d] Turner et al.) 2024 Chu et al. 2024 Nguyen
ct al. 2025). Concept operators for toxicity have been transferred from smaller to larger models (Dong et al.,
2024b). However, by increasing the intensity of the toxicity representation, RepE can also be used to make
models more toxic and induce allacks on LLMs (Wang & Shu 2023
A range of datasets are used to operationalize harm with AdvBench (Zou et al. 2023b), HarmfulQA (Bhard-
waj & Poria 2023) and HarmEval (Banerjee et al. 2024) being among the most popular. RepE methods
tend to prevent harmful outputs more effectively than prompting or fine-tuning-baser safeguards (Wang
et al. 2024b Cao et al. 2024b Li et al. 2024c Yu et al. 2024
Preventing and Triggering Jailbreaking, Refusal, and Backdoors. Training LLMs refuse answer harmful requests is a key mechanism for ensuring that they do not produce harmful and toxic outputs. To stress-test these safety mechanisms, jailbreaks have been developed that cireumvent this refusal behavior (Yi et al., 2024). By directly manipulating the internal mechanism that triggers refusal, RepE can serve as a very potent technique to avoid overrefusal or ensure that refusals are effective and appropriate.
Wang & Shu 2023 were the first to show that adding a steering vector during inference undermines safely by reducing truthfulness, making outputs more biased and increasing toxicity and harmfulness. This was
followed by range of work that uses RepE to jailbroak models (Wang et al., 2024a Ball et al. 2024 Xu
ct al.
2024b Stickland et al. 2024 Li et al. 2024c Tran et al. 2024 Zhang et al.) 2024c).
Arditi et al. [022]] find that there is a single direction in the activations that mediates whether refusal
behavior is triggered or not. Yu et al. (2024) ablate the refusal direction while performing safety training,
forcing the model to learn more robust refusal behavior. Furthermore, RepE has been used to stop the LLM
from refusing to answer benign queries (Cao et al., 2024b Xiao et al. 2024).
Lastly, RepE has been used to identify backdoors through UFL (Mack & Turner 2024a) and trigger backdoors by inducing the presence of the backdoor trigger (Price et al. 2024).
Making LLMs Truthful. A model is truthful if it outputs true answers. Getting models to give truthful and honest answers is crucial for their performance, but it also increases the trust we can place in them.
Li et al. (2023a) were the first to steer truthfulness using a RepE method. Marks & Tegmark (023) find linear probes that can steer the model to trcat false statements as truc and vico-versa. Overall, RepE methods scem to be particularly suited for steering this concept, often outperforming fine-tuning [Ackerman] 2024: Chen
et al. 2024b Qian et al., 2024 Qiu et al. 2024 Li et al.]
2024a) although it is sometimes outperformed by prompting (Chen et al. 2024b Li et al. 2023a; Wang et al. 2024ch. The initial methods
2023a
Liu et al.)
have been extended to encapsulate multiple aspects of truthfulness
(Wang et al. [024c| Chen et al.] 2024b).
steer non-linear representations (Hoscilowicz et al., 2024) or steer with different intensities before selecting
the most truthful output with a probe (Fatahi Bayat et al. 2024 TruthfulQA is the most commonly used dataset (Lin et al.] 2022).
Relatedly, hallucinations have been mitigated or induced with RepE (Wagg et al. 2024c Simhi et al. 2024 Zhang et al.) 2024b d Panickssory et al.) 2024 Beaglchole et al. 2025). Simhi et al. (2024) find that hallucinations can be more effectively identified from activation before the model answers and more effectively controlled in the activations of attention heads.
33

---

Published in Transactions on Machine Learning Research (09/2025)
Increasing Privacy. LLMs are trained on large amounts of uncurated text from the internet that contains private information, which could be leaked by the LLM.
To mitigate these privacy concerns, RepE has been used to prevent private data leakage (Zhagg et al. 2024a
Qian et al, 2024 Cai et al. 2024p. Wu et al. (2024b) find that securing specific private data can increase the leakage of other private information, which they successfully suppress by controlling privacy-sensitiig neurons. Zeng et al. (2024) remove private information from the inputs and apply steering vectors to restore performance. In the tested setups, these techniques can even improve over Differential Privacy methods
Zeng et al. 2024 Wu et al. 2024bb.
Avoiding Future Risks. The AI Safety community has hypothesized future risks from advanced Al models, and researchers are now allempting lo study key aspects of these risks in LLMs.
A deceptive system could lie to humans about its intentions. While is possible to make models more truthful in general. Clymer et al.
(2024) are not able to uncover and steer deception. However, sycophancy, which is the lendency of LLMs to adjust their responses to what. the user wants to hear, has been successfully controlled (Stickland et al. 2024 Panickssery et al.
Templeton et al. 2024), although others fail to re- 2024). AT systems might develop misaligned motivations,
2024
produce this (Paulo ct al. but this could be mitigated by engineering the representations of their goals (Mini et al. 2023). RepE has controlled an LLMs' exhibited desire for survival (Panickssery et al. 2024). power (Zou et al. [2023a), wealth (van der Weij et al. 2024), and short-term rewards (Panickssery et al. 2024 van der Weij et al. 2024]. Fur- thermore, Panickssery et al. (2024) control corrigibility and willingness to coordinate with other AIs against humans. Additionally, understanding the capabilities of a model is important for estimating the risk it poses. Mack & Turner (024a)) proposc method that could clicit undiscovered capabilities through unsupervised feature learning.
2024 van der Weij et al.
10.2 Ethics
Many ethical questions surround the development and application of LLMs. These include concerns about societal biases and the fairness of models, aligning the LLM to the values of its users, and encoding ethical reasoning into LLMs.
Improving Fairness. Since LLMs are trained text from the internet, they are prone to encode and amplify existing socictal biases. An LLM might reproduce stercotypes, not be representative in its outputs, or treat members from different social groups differently. By identifying how and where specific biases are represented, RepE can aid in uncovering, showcasing, and mitigating such biases.
RepE methods have been used to identify representations of bias-sensitive attributes such as gender, age, or race and cont rol them to reduce the bias in outputs (Phain & Nguyen] 2024 Chu el al. 2024 Wang Shu al., 2025). RepE has been used to control the sentiment expressed towards social groups [Luo et al. 2024) or how often a group is associated with specific discase (Zou et al.) 2023a Singh et al. (2024) identify representations for gender and dialect bias and removes the bias by steering all inputs to the same gender or dialect class. Furthermore, Durmus et al. (2024) use SAFs to identify features for gender, age, and political bias. They also find a "Neutrality" and "Multiple Perspectives" feature that reduce bias across many dimensions of bias.
2023 Nguyen
Additionally, RepE can be used to steer the representation of protected attributes to create counterfactual outputs, which can surface biases and thus be used for auditing the model. Chen al. (2024a) do this by controlling the model's belief about the user's age, gender, educational level, and socioeconomic status, thus uncovering how the model responds differently to different users. Avitan et al. 2024) intervene in representations of gender to create counterfactual generations.
Commonly used datasets for identifying the representations of biases include BOLD Dhamala et. al. 2021 BBQ (Parrish et al. 2022). or BiasInBios De-Arta et al. 2019) datasets.
Aligning with human preferences. Pre-trained LLMs do not necessarily act according to the preferences and values of their users. To mitigate this, L.MSs are commonly fine-tuned with methods like RLHF or DPO,
34

---

Published in Transactions on Machine Learning Research (09/2025)
where humans give feedback to align the model with their preferences. Instead of fine-tuning the weights, we could also use RepE, to align with human preferences.
Cao et al. (2024a) train a steering vector using a DPO-based loss that makes it more likely for the model to produce desired responses. It is also possible to use synthetically generated preference data to identify and steer relevant models' representations (T.uu et al. 2023: Adíla et al. 2024a).
Changing Ethical Reasoning. To aid in making LLMs act ethically, it is important they understand human ethical reasoning developed over millennia. RepE can help to improve general ethical reasoning (Pham & Nguyen 2024). It can also steer the model towards specific moral theories like commonsense morality, utilitarianism, or deontological reasoning (Tliee 2024
Zou et al. 2023a Xu et al. 2024a).
10.3 Knowledge Editing
LLMs encode a wealth of knowledge in their representations. However, some of that knowledge might be wrong, outdated, or dangerous. Thus, we would like to be able to remove and edit knowledge. Relatodly, LLMs can encode certain beliefs about the world or other systems that can be useful to interpret and edit.
Editing Factual Associations. Model editing methods like ROME (Meng et al., 2022) have successfully changed factual knowledge stored in LLMs. Similarly, Zou et al. (2023a) were able to make an LLM output that the Eiffel Tower is located in Rome. Hernandez al. (2024) propose REMEDI, which can detect and edit the attributes associated with entity through its activations. Furthermore, it is possible to use RepE to improve the ability of an LLM to reason with newly provided information (Yin ct al. 2024). Zhao et al. (024)) do not edit factual associations but rather control whether an LLM uses its learned parametric knowledge or the contextual knowledge provided in the prompt to answer a question. Relatedly, it is possible to detect and steer how much the model reproduces memorized text from the training dataset (Zou et al. 2023a).
Unlearning. Unlearning is concerned with removing harmful or unwanted knowledge in model. Li et al. 2024b) propose Representation Misdirection for Unlearning (RMU) that unlearns hazardous knowledge by pushing harmful representations towards random vector while retaining harmless ones. However, Xu 2024b) find that RMU and other unlearning methods are not robust against harmful steering vectors. Farrell et al. 2024)) use SAE steering to unlearn hazardous knowledge but find that it is not yet a competitive unlearning method. Rozanova el al. (2023) are able to remove information about specific feat These mixed results indicate that RepE might become a useful tool for unlearning but currently still has weaknesses.
et al.
Modulating Situational Awareness. During inference, LLMs may hold beliefs about the situation they are currently in. These could be facts about the environment or other agents, which RepE can potentially identify and control. Chen et al. (2024a] identify and control the models' beliefs about the age, gender, educational level, and socioeconomic status of the user it is currently interacting with. Similarly, Ghande- harioun et al. (2024) control the model's inferred impression of the user's personas in order to elicit harmful responses. Price al. (2024) change what the model believes the current year is. Lastly, RepE has been used to improve the model's 's capability to reason about the beliefs of other agents in Theory of Mind tasks
Bortoletto et al. 2024 Zhu et al. 2024).
10.4 Task Execution
Making LLMs carry out specific tasks key to making them practically useful. This is usually approached through fine-tuning on task examples, by providing few-shot examples, or by giving precise instructions. Recently, RepE has also found applications to this by editing the representations to control which task is exccuted in what way.
Todd et al. (2024) identify a function vector (FV) that captures an ICL task by finding atlention heads that have a causal effect on that task. In contrast, Liu et al. (2024b) find an in-context vector (ICV) from the activations on contrastive few-shot prompting inputs. Brumley et al. (2024) compare these two methods and find that ICVs are better able steer behavioral tasks, while FVs control functional tasks more effectively. Furthermore, FVs generalize better and deteriorate model quality less. Jorgensen et al. (2024) improve the
35

---

Published in Transactions on Machine Learning Research (09/2025)
ICV methodology by adding mean centering and Li et al. (2024e) apply inner and momentum optimization to the in-context vector. These works indicate that Tn- Context Learning partially works by shifting the representations to the correct task. Additionally, it is possible to identify representations that trigger the model to carry out Chain-of- Thouuht reasoning without being prompted to do so (Zhang & Viteri 2024).
10.5 Controlled Text Generation
Controlled text generation involves generating text with specific attributes or constraints, such as sentiment, style, or topic. The challenge lies in balancing the generation of coherent, natural language while maintaining control over these characteristics. By shifting internal representations, RepF. allows control over the desired attributes.
Controlling Sentiment. Sentiment is the emotional tone or opinion of a text. Researchers have used RepE to control how negative or positive the tone of generated text is (Turner et al. 2024 Konen et al., 2024 and what emotions are expressod in it (Cai et al. 2024 Konen et al.) 2024 Zou ct al.) 2023a Commonly used datasets are GoEmotions (Demszky al. 2020) and the Yelp sentiment datasel.
Controlling Personality. It is also possible shift the personality traits exemplified by the model. This can also indirectly inftuence other properties. Zhang et al. (2024a) steer personality traits of model according to the MBTI-scale and study the effects on its salely properties. WWeng et al. (2024) are able to steer the model towards OCEAN personality traits. By moderating the personalities, they are able to improve reasoning, enhance conversational capacity, and reduce sycophancy. Furthermore, it is possible to control the level of honesty and creativity displayed by the model with RepE (von Riitee et al. 2024
Controlling Language, Style, and Genre. Guo et al. (2024) use input pairs from different languages improve coherence in cross-lingual information retrieval and Scalena al. (2024) find that safety steering vectors derived in English transfer to other languages. RepE can control the style of generated text, like
writing Shakespearean or Chinese text (Konen et al.] 2024 Beaglehole et al. 2025 Ma et al.] 2025 and its genre, like fantasy, sci-fi, or sports (Jorgensen et al. 2024). It can also steer generation towards specific
topics (Makelo et al. 2025
Templeton et al. 2024
Turner et al. 2024).
10.6 Performance
RepE has been applied to improve the performance of LLMs on general capabilities and specific tasks.
Improving Reasoning. Improving the reasoning abilities of LLMs is an important frontier in NLP research to which RepE has been applicd (Cai et al.] 2024 Wu et al. 2024c Yin et al. 2024 Højer et al. 2025 RepE has been used to identify the representations underlying the Chain-of- to control the model to produce CoT reasoning without being prompted to do so (Hu et al.) & Viteri 2024). However, an attempt to steer the faithfulness of CoT reasoning with regard to the actual decisions made by the LLM was unsuccessful (Tanneru et al.] 2024). Additionally, RepE has been used to improve social reasoning abilities (Bortoletto et al. 2024: Zhu et al. 2024).
(CoT) mechanism and 2024 Zhang
Other Performance Improvements. The capability of LLMs for Natural Language Understanding and Generation tasks has been improved through RepE (Wu et al. 2024c a). van der Weij et al. (2024) use
RepE for improving the quality of general code and of python-specific code and Lucchetti & Guha (2024)
apply RepE to achieve type predictions that are more robust to irrelevant changes in the code. Furthermore.
RepE has been used to make the model provide equivalent answers for semantically equivalent queries Yang
et al. 2024) and reduce its bias to the ordering of examples and answer options (Adila et al., 2024b). Guo ct al. (2024) improve the quality of a multi-lingual information retrieval system by steering for coherence and accuracy. Lastly, Rahn et al. (2024) identify that uncertainty in the decision-making of an LLM agent is related to the entropy of its activations, which can be steered to achieve better exploration behavior.
36

---

Published in Transactions on Machine Learning Research (09/2025)
10.7 Interpretability
Interprelability aims to help humans understand the internal processes of Al models. RepE has helped to identify how human-understaddabl concepts are represented and to study the impact of representations on outputs.
Finding Linear Representations. linear vector detects a concept. with high accuracy and cont the concept effectively, this is evidence that this direction is how the model represents the concept. Thus, researchers have claimed to find linear representations of concepts such as Truthfulness (Marks & Tegmark 2023 Refusal (Arditi et al. 2024), models' encoded beliefs about the current year (Pice et al., 2024h, subject number for conjugations (Hao & Linzen 2023 and Harmfulness [uu et al.
2024b) These results also provide some evidence for the Linear Representation Hypothesis (Park et al. 2024b). However, we should retain appropriate skepticism about evidence provided by RepE about the lincarity of concept representations (see Section 13.2.1p.
Other Insights. Aside from interpreting the shape of representations, RepE has been used to shed light many phenomena in LLMs.
Wolf ct al. (024)) identify trade-off between helpfulness and safety. Qian et al. (2024) use RepE to investigate how safety-critical concepts emerge throughout pre-training. Rozanova et al. (2023) study the
representations of safety concepts in different languages. They find them to be similar and see that English safety concepts can be used to steer other languages. Other papers investigate the inner mechanisms of CoT reasoning (Hu ct al., 2024) and the impact of removing concept (Rozanova et al. 2023).
However, RepF is not perfect tool for interoperabilityy, and results can easily be overelaimed. Wang & Veitch (2024) specifically criticize the idea that editing representations at location which leads to the desired output change actually provides evidence that the concept representation is localized there. This is because they see that optimal edits at random points are also very effective.
10.8
Representation Engineering outside LLMs
10.8.1 Image Generation
A large number of previous works studied controlling image generation by operating on learned represen- tations. [pcchrrh al. (2017) showed that image editing can be done by linear interpolation in deep feature space between images with certain attribute and images without that attribute, given the hypoth- esis that ConvNets linearize the manifold of natural images. Rich structures also appear in unsupervised representation learning of Generative Adversarial Networks (GANs). This can enable controlled generation by simple vector arithmotic (Radford et al.) 2016) or non-lincar traversal (Wang ct al., 2021) operations on latent vectors to manipulate semantic concepts. Concept classifiers for altributes can be used to disen- tangle the representations of attributess, thus allowing us to constrain the optimization in a GAN to find directions that maximally affect the concept attribute (Wang et al. 2021). Shen et al. (2020) and Wu et al.) (2021) showed that the latent space of StyleGAN2 can be disentangled so that cach dimension captures most one attribute. Others methods train GANs with contrastive learning to learn disentangled represen- tations (Shoshan et al. 202T). Recently, such concept disentanglement in the activation space has been attempled in LLMs by training Sparse Auto-Encoders (SAEs) (Huben et al. 2024).
Modulated Features. Controlled image generation and editing is able control attributes of human, such as age, facial hair, eyewear, headwcar, pose, facial expressions, and other global fcatures, such as illumination and artistic styles (Shoshan et al.) 2021 Bhattad et al.] 2024).. Collins et al. (2020) perform spatially localized edits based on concept image. Jahanian et al. (2020) showed that activations can be steered to control camera movements.
Analogy to LLMs. Some techniques in controlled image generation resemble those in RepE for LLMs. Contastive learning to promote disentanglement of features (Shoshan et al.) 2021] uses contrastive examples where an attribute is or not present, which is similar to Input Reading with contrastive examples. Other methods use concept classifier (Jahanian et al. |2020] Wang al.) 2021 Chen et al., (2016) to identify trajectories in the latent space that steer the concept, which resembles identilying a concept operator through
37

---

Published in Transactions on Machine Learning Research (09/2025)
Output Optimization. Similar to RepE methods using Unsupervised Feature Learning (see Section 4.3
Voynov & Babenko (2020) discover directions corresponding to changes in a concept. by having humans interpret the effect of the changes. Dong et al. (2024a] take inspiration from RepE' for LLM safety to derive stecring vectors that can be projected out to increase the safety of generated images.
10.8.2 Games
RepE has been used to interpret the internal world models learned by Transformer models that were trained in an autoregressive fashion on board games. Nanda et al. (2023) find that Othello-GPT represents the color of tiles linearly, and steering that vector affects which moves the model believes to be legal or not, Karvonen (2024) conduct similar experiments for Chess-GPT model, where they find that individual pieces and the skill level of players autoregressive Transformers can learn to implicitly represent world models. Mini et al. (2023) identify a vector that successfully steers the goal pursued by a policy trained to solve mazes. This is promising since it could allow us to directly identify and alter the goals pursued by capable AI Agents.
linearly represented and can be steered. These works indicate that
Takeaway: Applications
RepE has been used in
AI Safety reduce harmfulness, increase truthfulness, and control refusal behavior. Interpretability confirm the presence and shape of concept representations.
AI Ethics reduce societal bias, align LLM behavior with human preferences, and control ethical reasoning.
Knowledge Editing to change or unlearn knowledge and adapt model encoded beliefs.
Other areas such as controlling task execution and text generation and improving general model per- formance.
11
Comparing RepE to Other Methods
Table 12: Difference and Similarities of related Families of Methods to RepE
Other Method
Difference to RepE
Prompting
Changes inputs instead of activations or weights
Soft-prompting
Operates token embeddings instead of hidden states
Fine-tuning
Doesn't 't target specific concept representations Operates on logits instead of hidden states Bottom-up vs Top-Down view
Decoding- based Methods
Mechanistic Interpretability
Activation Patching
Fully replaces activations instead of adapting them
Probing
Only does Representation Identification and not Control
11.1 Related Methods
There is range of related methods that attempt identify how concepts represented in Neural Networks or that aim to control the behavior of LLMs.
Prompting. Changing the (system-)prompl given to an LLM is the most dlraightforwart and commonly used way to steer the behavior model. However, does so without any changes to weights or activations and does not provide any interpretability benefits. Furthermore, RepE does not take up space in the context window, while the additional tokens from prompting, especially when using many in-context examples, increases the computational cost.
38

---

Published in Transactions on Machine Learning Research (09/2025)
Soft-Prompting Similar to RepE, soft- trtmppoomping modifies the model in continuous embedding space. Tt does so by operating on the input embeddings, whereas RepE. modifies internal activations or weights. However, noft-prompting does not have the goal of identifying the representation of a concept.
Fine-tuning, Fine-tuning modifies the weights of a pre-trained model by further training it on a specific dataset or reward signal. While fine-tuning does influence the model's representations, it does not do so in a manner that is targeted to identifying and controlling representations of specific concepts. Instead, it applies some uminterpretable, non-sparse changes to the weights that lead to the desired outputs. Unlike RepE, fine-tuning does not allow for precise and isolated control of concepts of interest. Furthermore, many RepE methods provide continuous control, where a concept can be increased or decreased to the desired intensity, As mentioned in Section 6.2.1 there is an overlap between fine-tuning and RepE methods that do output optimization and apply a weight steering function.
Decoding-bascd methods. Decoding- approaches steer the behavior of LLMs by shifting the token probabilities during the decoding step. Contrastive Decoding takes two sets of token probabilities, contrasts them against each other, and consequently shifts the token probabilities (Li et al. 2023b). Often, this means taking predictions from two different models, one of which is better at the desired behavior. Alternatively, contrastive predictions can be taken from the same model on contrastive inputs. Similar to RepE methods, this method steers the behavior of the model during inference time. However, contrastive decoding requires multiple forward passes per generated token. Furthermore, it only superficially shifts token probabilities and does not tap into the internal representations of LLMs.
Mechanistic Interpretability (MT). MT is not specific method, but field of study that aims to reverse engincer the mechanisms learned by Neural Network into human-undersaandable algorithms. Although RepE partially grew out of MI, Zou et al. (2023a) present it as counterproposal. MI takes bottom-up view by studying individual neurons and their connections into circuits. While such approaches can explain simple mechanisms well, they struggle to scale to higher-lovel concepts and processes. In contrast, RepE takes a top-down view by interpreting how high-level concepts are implemented as patterns in the representational space spanned by a large population of neurons. However, in practice, there is no clear delineation ween these approaches.
Activation Patching (AP). In AP, the activations for some neurons specific input are replaced with different activations derived from other inputs. By observing the counterfactual outputs, researchers can study the causal relationship between internal representations and outputs. In contrast, RepE modifies the existing activations instead of fully replacing them. RepE also focuses less on a cansal understanding of individual neurons and more on usefully steering behavior.
Probing. In Probing, a classifier is trained to predict the occurrence of a concept in the model's activations. Classically, probes are trained to interpret Neural Network. Furthermore, they can be used to detect concepts like task drift (Abdelnabi et al., 2024a) deceptive reasoning (MacDiarmid et al. 2024 Goldowsky-DDo Dill ct al. 2025) during inference such that a more expensive safcty procedure can be triggered. However, they can also be used as a method for Representation Identification in RepE or as a way to dynamically modulate the strength of the steering function.
11.2 Meta-study comparing to Other Methods For Behavior Control
Other mothods for controlling the behavior of LLMs include prompting, fino-tuning, and grroding-based methods. We describe conduct a meta-survey that compares them to RepE according to their effectiveness and impacts on the model's capabilities.
For every paper, our meta-survey compares whether the proposed RepE approach or the compared approach works better for each paper. If available, we also compare which approach retains higher general capabilities and how many samples were used. When an experiment in a paper was run for multiple tasks, we averaged the results over the tasks. When there are multiple compared methods from the same category of approach, we compare to the best-performing one. Lastly, we want to cantion the reader that most available comparisons come from RepE. papers, thus, biasing the results in favor of RepF.
39

---

Published in Transactions on Machine Learning Research (09/2025)
Table 13: We compare RepE to prompting, fine-tuning, and decoding by the amount papers where is better, equal, or worse at controlling concept or retaining capabilities.
RepE
Prompting
Fine-tuning
Decoding
Control Effectiveness
Capability Retention
Control Effectiveness
Capability Retention
Control Effectiveness
Capability
Retention
better
18
4
18
8
5
3
equal 0
1
3
2
0
0
worse
7
2
5
1
2
A list of all papers from this metastudy is provided in and Table 14 and Table 13 respectively show experimental results for each comparison of RepE with prompting and fine-tuning respectively.
11.2.1 Prompting
There are 24 papers (listed in Appendix D) that provide experiments comparing the effectiveness between RepE and prompting method. Across the surveyed papers. RepE more effective than prompting at steering the larget concept in 75% of the cases (see Table 13 Furthermore, it oflen had less impact on the general capabilities of the model. Notably, AxBench (Wu al. 2025) conducts thorough comparison between multiple RepE' methods and prompting and finds prompting to perform better at detecting and steering concrete content-related concepts.
11.2.2 Fine-tuning
There are papers (listed in Appendix D that provide experiments comparing the effectiveness between RepE and a fine-tuning method. Hereby RepE methods such as RcFT (Wu et al. 2024c) and BIPO Cao et al. (2024a) that are adjacent to fine-tuning the target concept more effectively in 72% of comparisons, while fine-tuning is only better in 20% of cases. A majority of comparisons find that fine-tuning deteriorates general model capabilitics more heavily in most cases. Figure 3 lightly indicates that RepE might be more sample effective, outperforming fine-tuning in every comparison under 500 samples.
counted lowards RepE. Table 13 shows that. RepE steers
RepE vs Fine-tuning
# of papers comparing control effectiveness
RepE better Fine-tuning better
1-100
101-1000 1001-3000 3001-10000 samples
Figure 3: The number of papers using a number of samples where RepE finc-tuning offer more effective control.
40

---

Published in Transactions on Machine Learning Research (09/2025)
11.2.3 Decoding-basee Methods
There are 7 papers (listed in Appendix D that provide experiments comparing the effectiveness between RepE and a decoding- based method. RepE-base based methods outperform decoding- based methods in 71% of cases and have much less impact on general model capabilities.
11.2.4 Combining Methods
One great advantage of RepE: is that it works well in combination with other approaches. All four papers (listed in Appendix D that evaluate this find that combining RepE with prompting, fine-tuning, decoding- based method delivers more effective control than any of them alone (Banerjee et al. 2024 Stickland
et al. 2024 Li et al., 2023a Wang et al. 2024c This is typically tested in papers where RepÉ does not outperform the respective bascline method.
Takeaway: Comparing RepE to Other Methods
Other methods for identifying and editing concept representations, like Activation Patching or Concept Erasure, are less applicable for controlling output behaviors or have some overlap with RepE. Our meta- survey shows that RepE offers more effective control at a lower decrease of capabilities compared to prompting, fine-tuning, and decoding-base approaches.
12 Why Does It Work?
It is somewhat surprising that relatively simple operations on the representations of a model allow us con- trol high-level behavior while not destroying capabilities. While there are specific reasons for the effectiveness of different methods, this section offers some general explanations for this phenomenon.
LLMs already represent human-understandbble concepts. LLMs seem to represent some human- understandable concepts like refusal or honesty. This makes it possible to tap into those representations and change their behavior. Thus, RepE does not need to learn the concept representations anew but simply identifies and promotes them.
Models have structured representations. LLMs represent concepts in structured way. RepE can uncover and exploit this structure. For example, concepts might be represented as linear directions in the activation space of LLMs. This Linear Representation Hypothesis (LRH) (Park et al. 2024b) implies that scaling the direction corresponding to concept increases the intensity of that concept. Many RepE methods are explicitly inspired by LRH or implicitly built on it. If the LRH is true, it explains the success of many RepE methods.
The plausibility of the LRH has been an area of much discussion in the interpretability community. Tn fact, some features have been proven to be represented linearly in autoregressive models trained for board games (Nanda et al. 2023 Karvonen, 2024) and in LLMs. Many concepts in LLMs have been decoded through linear probes [Li el. al.] 2024e Abdou et al.) 2021 Grand et. al. 2018), and the success of many Representation Engineering methods is itself evidence for linear representations of high-level concepts. On the other hand, Engels et al. 2025 find non-linear features, such as circular representations for the days of the week.
Resilience activation manipulation. Neural Networks are able remain functional after their activations are manipulated. For example, adding small random perturbations to the activations does not strongly affect model performance (Reagen al 2018). This resilience might be attributable to dropout during training. Furthermore, the concept operator is derived from previous activations of the same network, which could mean that the manipulated activations still remain in the usual distribution of activations. This makes possible to relain general capabilities.
Control over intermediate variables. Neural Networks can be seen as programs where activations are intermediate memory containing variables and are functions that perform operations on these variables. Changing variables in that memory changes the further computation and, ultimately, the output. When
41

---

Published in Transactions on Machine Learning Research (09/2025)
RepE operates representations of concepts in the activations, this is similar to editing semantically meaningful variables in the intermediate memory of Neural Network.
Takeaway: Why it works
Models represent human-understanddble concepts in structured, possibly lincar ways. By manipulating these latent variables, we influence the model's later computations. LLMs remain functional since they are somewhat resilient to internal manipulations.
13
Challenges in Representation Engineering
13.1
Empirical Weaknesses
Multi-Concept Control. Applying RepE for multiple concepts at once reduces the effectiveness of steering
(van der Weij et al. 2024 Scalena et al. 2024 and leads to larger reductions model capabilities (Scalena
ct al. 2024). This might be because applying multiple edits can lead interference between concept
representations van der Weij et al. 2024). An exception that combining specific vectors derived by BiPO
Cao et al. (2224a) seems to increase their effectiveness. Improving multi-concept control is recognised
of the major challenges in RepE (Cao et al. 2024a Postmmus & Abreu 2024 van der Weij et al., 2024 Dong
ct al. 2024b Chu et al. 2024). Multi-property steering is important since model providers will want to steer for more than one concept al time. For example when van der Weij el al. (2024) combine steering vectors for Myopia, Wealth Seeking, Sycophancy, Agreeableness and Anti- iirll-imiiration together, the steering effect for each concept is reducod.
While there are proposals to steer different concepts in different layers (van der Weij el al. 2024). this limits the number of concepts that can be steered simultaneously. Furthermore, [Scalena et al. (0024) dynamically adjust the strength of interventions which reduces the negative effects of multi-concept control. Further, there are attempts to control broader representations, like "human preferences", that encapsulate multiple desired concepts Liu et al. 2023). Nguyen et al. (2025) improve multi-concept steering by sparsely applying each steering vector only at some tokens and enforcing orthogonality between steering vectors to reduce interference. Lastly, Beaglehole et al. (2025) report progress on multi-concept steering based on non-lincar feature learning.
Long-Form Generation. While RepE can effectively control short answers, there is lack of evidence for its ability to control long-form generations and multi-turn conversations. It would be problematic if RepE would lose it's ability to control LLM behavior when a long text is generated or the conversation continues for multiple turns. Indeod, there is preliminary evidence that operators identified from a training set with short answers do not effectively steer long generations (Pres el al., 2024). This calls for a thorough evaluation of new RepE, methods for longer generations. Recent work has started address this problem by retaining the natural distribution of activations (Cao et al. 2024a) and dynamically adjusting the representation control throughout generation (Scalena et al. 2024
Out-of-Distributin Generalization. To effectively steer concept, our concept operator should gen- eralize from its training data to different contexts, prompt templates, or different generation tasks. For example a concept operator for harmfulness might only be derived on english inputs and fail to prevent harmful responses in other languages. Additionally, if concept operator does not generalize across different situations, we cannot argue that we have found a general representation of the concept. However, this still presents a challenge for current RepE pipelines, which become less effective when the system prompt differs between the training and test set (Tan et al.) 2024) and when the operator is identified after the answer but employed before the answer (Simhi et al. 2024). Evaluating and improving the ability of RepE methods for
OOD generalization will be crucial (Mack & Turner 2024a Zou et al. 2023a Li et al. 2023a 2024)
Ackerman
Deterioration of Capabilitics. RepE generally leads to a reduction in general language modeling capabil- ities and the quality and diversity of generated text (Scalena al. 2024 Stickland al., 2024 Zhang et al., 2024ch. Aside from the quality of the generated text. Park et. al. (2024a) points out that steered models
42

---

Published in Transactions on Machine Learning Research (09/2025)
can be worse at instruction following. While this reduction is rather small, it still represents an important cost that might stop model providers from using RepE. Increasing the strength of the intervention reduces the capabilities (Wu et al.) 2025 Durmus et al., 2024). This effect could appear because RepE disrupts the underlying structure of a model's activations (Zhang ct al. 2024c). Thus, finding improvements to reduce or better navigale the trade-off between control eflectiveness and general model capabilities will be crucial 2024c Efforts to solve this problem include fine-tuning the
(Li et al. 2024b Wu et al.) 2024b Wang et al. model to prevent reductions in quality through RepE (Stickland al.) 2024), moderating the intervention strength to only steer when necessary (Li et al.; 2024d), and more precisely localising the concept repre- sentations (Wang et al., 2024b). Others attempt to mitigate this by retaining the underlying structure of the LLM (Zhang et al. 2024c), keeping the magnitude of activations consistent (Pham & Nguyen 2024 or providing an output-based goal to retain lext quality (Xu et al., 2024b).
Learning Specific Concept Representations, Ideally, RepE would steer a concept without influencing other, unrelated concepts. However, in practice, RepE methods struggle to isolate a concept representation from other concepls. For example, steering towards happiness reduces refusal rates for harmful requests (Zou et al., 2023a), steering away from harmfulness makes the model less helpful (Wolf et al. 2024), and steering gender awareness can increase age-based biases (Durmus et al. 2024). A failure to isolate a specific concept representation might cause undesired side-effects that make RepE less usable in practice.
Learning Complete Concept Representations. We would like RepE to identify a concept operator which represents the concepts with all its aspects and in different contexts. A concept like Honesty is multi- faceted and is applied differently depending on the context. Failing to identify a complete representation of honesty means some aspects of the concept are not improved and the steering might fail in some contexts. However, identifying such complex representations is challenging and likely requires large coverage of aspects and contexts in the training data, as well as a concept operator that is expressive enough. [Chen et al. (024b) address this by learning multiple orthogonal steering vectors that are supposed to capture different aspects of a concept.
Unreliability. RepE methods are generally unreliable and tend to fail in three ways. Firstly, they are sensitive to the hyperparameters chosen. Often, a small seemingly insignificant change in hyperparaneters, such small changes to the steering strength, can cause significant performance decrcases (Zhang et al., 2024c Adila et al. 2024b). This indicates that the chosen RepE setup is not robust. Secondly, there are often concepts that cannot be successfully steered. While it is common for papers to mainly report on the successful applications of their method, more neutral investigations often find concepts, like reducing narcissistic behavior, that can not be successfully steered (Tan et al., 2024 Tanneru et al. 2024). Braum et al. (2025) provide evidence that some concepts are not effectively steered because the difference between positive and negative activations is not consistent and not linearly separable. Lastly, even for interventions that are successful in steering the average behavior, there are inputs for which the steering effect is negative (Tan et al. 2024 Stickland et al. 2024 This can especially be the case for examples that were already positive with respect to the concept (Im & Li 2025 This unrcliability hinders the deployment of RepE methods.
Unreliable Interpretability. RepE is often used as a tool to interpret LLM representations, but it is questionable whether they provide strong evidence or are largely mislcading. Firstly, a failure to identify an effective linear operator for a concept does not prove that the concept is not linearly represented in the model. Tt could merely be failure of the method to find the relevant directions. Secondly, it has been shown that finding an effective intervention in specific layers attention heads does not prove that the concept is actually localized there (Wang & Veitch, 2024). Thirdly, there can be multiple different vectors that successfully steer the concept. For example, (Goldman- Wetzler & Turner 2024) find 800 hogonall vectors that steer "write code". Consequently, a direction that steers the model is not necessarily the single direction that represents the concept (Subramani et al.] 2022).
Requires Access to Models' Internals. RepF methods rely on access to models' internals and activations. This limits who is able to ereate and employ RepE methods. Xu et al. (2024b) partially circumvent this by using RepE on white-box models to find safety vectors, use them to derive attack prompts, and transfer those attack prompts to black-bog model.
43

---

Published in Transactions on Machine Learning Research (09/2025)
Computational Cost. Although RepE is highly efficient during inference and cheap to train, multiple papers aim to improve the computational efficiency of deriving or employing RepE, methods. This can be done by removing the need to perform expensive hyperparameter tuning (Xu et al.) 2024b) or expensive iterative optimization (Qiu et al.] 2024 Jorgensen et al. 2024). The additional operations necessary to manipulate activalions during inference can be avoided by instead manipulating weights (Wang el al. 2024a
13.2 Principled Challenges
In addition to these previous general challenges and weaknesses, we further break down the challenges according to each step in the pipeline.
13.2.1 Challenges in Representation Identification
Spuriously Correlated Concepts. Current RI methods are mostly not able to disentangle concepts that are correlated. As a result, the identified concept operator will also capture and steer the spurionsly correlated concept (Tan et al. 2024). Concepts can be correlated because they often occur together the inputs, because the output scoring evaluates both highly or because their representations tend to be activated at the same time. For example, instructing a model to write faulty code can simultaneously trigger the representation to write faulty code and the representation for general unethical behavior (Betley et al,, 2025 Soligo et al., 2025). Using RI techniques like difference-in-means or linear probes will consequently lead to a steering vector that captures the representation for faulty code and general unethical behavior. This presents a major hurdle for precisely identifying a concept representation. Sparse Autoencoders might be able to address this by identifying disentangling concept representations.
Concept Misspecification. Representation Identification forces us to specify the target concept by pro- viding scoring function inputs related to the concept. Through these, we are hoping to elicit. the model's representation of this concept. However, if the concept specification differs from the intended concepts, the identified concept operator will not be accurate. For example, in Output Optimization we might use an LLM to score how honest outputs are to learn a concept operator. But if the LLM-judge fails to detect some forms of dishonesty, the resulting concept operator will also fail to steer these forms of dishonesty.
Furthermore, even when the specified inputs or scoring functions are accurate, they might activate other representations than the ones for the desired concept. For example, we might instruct a model to be honest dishonest and hope this activates the model representations of "I will be honest/dishonost for a model that was trying to deceive, such an input might instead activate a representation for "The human wants me to be honest/dishonest which is importantly different from the intended concept. Deng et al. (025) point out that RI methods based on contrastive pre-prompts implicitly assume that the model actually follows the pre-prompt.
However,
Additionally, models might use concepts that are not understandable for humans. They might use an ontology that differs from ours or discover new concepts. In this case, it would be difficult to specify and steer these concepts.
Interference from Superposition. Interpretability research has shown that LLMs are polysemantic since they represent more features than they have dimensions (Elhage et al., 2022). Thus, networks represent features in superposition, meaning that features are not all orthogonal to each other, which results in interference between features. In practice, this will mean that controlling a concept representation will also stoer some other concopts. Thus, concept operators are not specific to a concept and have side effects on unrelated concepts. For example steering model towards academic citations could accidentally also steer it to writing HTTP requests if there is interference between these representations Bricken et al. (2023) Nguyen et al. (222) attempt to reduce interference by enforcing orthogonality between concept vectors, thus improving multi multi-concept steering.
Assumptions on Available Data. RepE methods often only work when specific data available. Firstly, it is not always possible to provide contrastive inputs that describe the behavior and its opposite (Jorgensen et al.) 2024. Cai et al.) [2024] Postmmus Abreu (024).. For example using contrastive inputs might not be the right methods for non-binary concepts like the days of the week. Secondly, many methods rely on
44

---

Published in Transactions on Machine Learning Research (09/2025)
expensive human-annotated ground-truth data (Adila et al. 2024b a). For example steering against societal biases might involve humans annotating or writing biased and unbiased texts, which can be prohibitively expensive. Lastly, methods are not designed to handle noisy and biased data Adila et al. (2024a).
Reliance on Models' Own Representations. RepE methods tap into the existing representations of a model. Consequently, a concept cannot be controlled with RepE, if the network did not learn representation for it. If a model never learned a representation for being honest, RI methods will not be able find an effective concept operator to make the model more honest.
Interpretability Lacks Ground Truth. The challenge of developing methods that find representations related to a concept is very difficult because there is no ground truth for where and how the model represents a concept. Thus, we cannot know whether we have ident an accurate representation or what the semantic significance of the concept operator (Herrmann & Levinstein 2024]. While we might find a vector that causes the model to be more honest and detect dishonest generations, we have no guarantce that this maps to how the model actually represents honesty.
13.2.2 Challenges in Representation Operationalization
Assumptions About Models' Representations. Methods for identifying concept operators come with assumptions about the shape and geometry of representations in the models' activations. Common assump- tions about representations are that concepts are represented as linear direction (Qiu al. 2024h, that representations do not change throughout sequence of inputs, that they the same across context, that they are localized in single layer, and that they do not rcly interactions between layers.
However, if the true form of the representation does not match these assumptions, it will lead to unsuccessful or less effective Representation Identification and Control. If a concept representation is non-linear a linear concept operator can at most be a first-order approximation of the true representation and thus provides less precise and effective steering. Similarly, when the true representation of concept depends on the context or can change throughout trajectories, steering with one static concept operator is best optimal in subset of situations. Additionally, when concepts rely on interactions between layers, single layer stecring cannot capture all aspects of that concept or could fail to correctly inffuence the concept in later layers.
These assumptions have been criticized (Luo et al. 2024 Chu et al. 2024 See also our discussion of the validity of the Linear Representation Hypothesis in Section 5.1 Assumptions have been changed and
weakened by shifting from a point-in -ee---i--spceee to a direction direction-magnituuee magnitude view (Pham & Nguyen 2024), adapting
the strength and direction of stcering the context. (Wang et al.] 2024c).
or applying non-linear operations
QQu al.] 2024 Hoscilowicz et al. 2024).
An additional challenge is that feature might not have single representation. For example, (Goldman- Wetzler & Turner 2024) find 800 orthogonal vectors that steer "write code" and (Mack & Turner 2024b)
finds 200 independent vectors with which model represents that request is harmless. While this might be due to weaknesses in the specific methods or because of wrong assumptions about the representations geometry, it could also mean that some concepts are represented in multiple ways inside a model. This challenges traditional notions in RepE, where it is assumed that there one represcntation of the concept.
13.2.3 Challenges in Representation Control
Shifting Activations off Their Natural Distribution. Manipulating the weights and activations of a model can shift the representations off of their natural distribution. In turn, this can lead to a deterioration the model might be disturbed by the unnat ural shift
of LLMs' capabilities since later computations activations. While steering was found only cause small shifts in the distribution (yan der Weij et all, 2024), this effect could be detrimental when the strength of intervention is increased or many concepts are steered at the same time.
Takcaway: Challenges in Representation Engincering
RepE struggles with steering multiple concepts at once, controlling long-form generations, and providing
45

---

Published in Transactions on Machine Learning Research (09/2025)
reliable steering effect. These weaknesses arise because of challenges like the inability to disentangle spuriously correlated concepts, to posit correct assumptions about models' representations, and to retain the models' natural distribution of activations.
14
Opportunities for Future Research
14.1
Opportunities to Improve Representation Identification
Refining Identified Operators. The identified concept operator could be improved to make it better suited for control. For example, an operator identified through Input Reading could be refined by fine- tuning it to produce desired outputs or by decomposing it with an SAE and removing unrelated features. In general, there is promise in developing pipelines that combine different RI methods. For example, Wu et al. (2025) suggest that finding concept operators by jointly learning for concept detection and steering can be more effective.
Most efforts for improving RI focus on new calculations for deriving the concept operator. However, recent work indicates that the quality of concept operator is strongly dependent the quality of the dataset it is trained (Tan et al. 2024 Braun et al., 2025 and improves when it is trained more data (see Appendix A.2). Akin to data-centric approaches to train betier models (Zha et al. 2025 a focus on enhancing the quality and quantity of data could be key to improving RI. Furthermore, detailed investigations of the propertios datasets that lead to strong stecring performance could unlock important insights.
Data-centric RepE.
Better Methods for Scoring Outputs. Currently, most Output Optimization methods simply compare outputs to ground-truth text. There more accurate and flexible ways for scoring the alignment of outputs with concept. One could use a human judge, LLM-as-a-judge, a trained reward model, specific context-dependent metrics. Furthermore, one could use the performance in an environment, like rewarding actions in a multi-agent negotiation setup based on the final agreement (Abdelnabi et al. 2024b).
Combine with Other Interpretability Methods. As described Section 11.1 there exist other methods that aim to identify representations of concepts LLMs. While these might not produce suitable concept operators themselves, they could provide information about the representation that can be leveraged for Representation Identification. At least, it would be interesting to compare the identified representations from different methods for the same concept.
Finding and Analyzing Failure Modes. We hesite that spuriously correlations between concepts and concept misspecification pose challenges for effective Representation Identification. However, there is no empirical evidence for this yet. Exploring such theoretically plausible failure modes could uncover new weaknesses be addressed by improved RI methods.
Automated Interpretability. LLMs judge which concepts feature corresponds to by looking highly activating inputs and outputs. This is commonly done in Unsupervised Feature Learning to identify concept operators. A similar approach could be used to refine concept operators identified by other RI methods. An LLM can automatically judge the inputs and outputs activating for multiple concept. operators.
14.2 Opportunities Improve Representation Operationalization
Expanding on Linear Representations. Most RepE pipclines assume that concepts are represented as linear directions. However, in practice, representations might be non-linear. As noted previously, there is early work RepE for non-linear steering, but more exploration of non-linear representations is necessary.
Context-Dopendent Control. Current RepE methods assume that the representation of concept is the same for every context and thus apply the same intervention independent of the inputs. However, the model might represent a concept differently different contexts. There is already work that dynamically adjusts the strength or composition of concept operators. However, future work could learn separate concept operators for range of situations or train small model that can predict the right concept operator to apply in new situation.
46

---

Published in Transactions on Machine Learning Research (09/2025)
Modeling Trajectories of Representations. Current RepE methods assume that representations of concept remain static throughout generation. However, it could also be that the representation of concept changes over time. In that case, representations of interests are better modeled as trajectories of activations over inference steps.
Inter-Layer Dependencies. Recent work (Lindsey et al., 2024) shows that representations of some con- cepts can be spread over multiple layers. Current RepÉ methods are not capable of capturing these inter- layer dependencies. To address this, RI methods could concurrently optimize multiple concept operators different layers.
Modeling Interactions Between Concept Representations. Current RepE methods aim to find an isolated representation of a single concept. But. this misses any interactions and relationships ween multiple concepts. For example, there could be a specific representation AAB if concepts A and B are both present, or there could be conditional representations oB A of B given that present. Thus, attempting identify interacting representations between concepts could uncover more rich concepts the model is using,
More Complex Operators. Current operators are mostly relatively simple. Perhaps more complex operators could capture more comprchensive and nuanced representations. It might be possible to train a Neural Network to predict a concept operator for the given inputs. Furthermore, an ensemble of concept operators could be trained on different portions of the data or different clusters of activations to capture different aspects of a concept.
Feature Geometry as a Hyperparameter. Current work starts by assuming the geometry of the concept representation and then aims to identify the concept representation adhering to that geometry. By trying out multiple geometries for one concept and then measuring their effectiveness, geometry that most suits the specific concept representation. Additionally, RepE would benefit from further understanding the geometry of concepts in Transformer Language Models, which is already an active field of study [Jiang et al., 2024a; Csordás ct al. 2024).
can expect to find the
14.3 Opportunities to Improve Representation Control
Combining Weight and Activation Steering Functions. Representations an interplay between weights and activations. Thus, it could be beneficial to develop methods that control weights and activations together. Naively, one could apply RepE for a concept to the weights and also apply it to the activations. Furthermore, it could be interesting to strengthen or weaken certain representations while training the model, thus guiding which representations are learned by the model.
Complex Steering Sequences. By applying steering for different sequences at different time points in a generation, Repli can be used to steer more complex behaviors or ensure that LLMs abide by predefined pattern. For example, the providers of LLM might want it to answer code-related questions by first steering for "detailed reasoning" while the problem is being analyzed, then for "not containing errors" while new code suggestions are generated, and lastly for "friendliness" while the model asks for clarifications.
More expressive steering functions. Krasheninnikov & Krueger (2024) find in a toy setting that more expressive steering functions, whose operators have more parameters, can reach a higher performance ceiling and control more complex tasks while requiring more data. This indicates there promisc developing more expressive steering functions push the Pareto frontier of steering effectiveness and data-efficiency.
Controlling Learning Processes with RepE. Yu et al. (2024) repeatedly ablate the refusal feature to learn more robust refusal mechanisms. This highlights that steering during training can be used to influcnce the representations that are being learned by the model. Aside from ablating features, representations can be positively steered potentially to make them more salient in the learning process. On the other hand, learning representations specific ways can provide concept operators steer model behavior. For example, Cloud et al. (2024) use Gradient Routing to control where in the network specific capabilities are localized, thus enabling to unlearn specific capabilities during inference.
47

---

Published in Transactions on Machine Learning Research (09/2025)
14.4 Possible Applications of Representation Engineering
After outlining current applications in Section I0 we now suggest some further problems that could be addressed with RepE.
Agent Goals. RepE could be used to control the goals pursued by an LLM Agent. Previous work achieved this for RL agents (Mini et al. 2023), but the growing popularity of LLM agents trained with Reinforcement Learning (OpenAT 2024) makes it timely to apply RepE to steer their goals. Controlling an agent away from dangerous and lowards desirable goals could be a simple but effective technique for miligating misalignment in LLM Agents. RepE might be uniquely capable of adjusting agent goals compared to prompting. An AT system that intrinsically values a goal is instrumentally incentivized to retain that goal (Bostrom (2012), even when is prompted to change its goal. However, RepE might be able to directly edit the goal the AI is pursuing by steering its own representations.
Studying In-Context Learning. Finding out how and why in-context learning (ICL) works is hotly debated topic. By controlling representations, it might be possible to steer the hypothesis class ICL uses and thus make progress on understanding the patterns and algorithm enabling TCL.
Preventing Deception. Al systems behaving as if they are aligned with human interests while secretly pursuing different goals is a key worry in the AI Safety community. RepE could provide unique control over models' behavior when the Al is attempting to deceive the user (Park et al. 2024c). In this case, it will not suffice to instruct the model to be truthful. However, by directly accessing the model's own representations and thus influencing its "thought process", RepE might be able to control it to be more honest.
Controlling Coooperation. As LLM agents become more ubiquitous, they will often interact and negotiate with other LLM agents. Through RepE, it might be possible to control how agents behave and cooperate in multi-agent scenarios.
Value Engineering. RepE' could be used to identify specific ethical values to let users personalize their LLM agent. For example, user could specify which beings the AI should give moral consideration to, which moral theory it should apply, what its stance on specific issues should be.
Red Teaming. RepE could be used when red-teaming and evaluating LLMs. Firstly, for open-weight models, which can easily be controlled lowards harmful outcomes through RepE, it should be standard to also apply RepE during red-teaming. Secondly, for closed-weight models, applying RepE to jailbreak a model could serve as a worst-case safety failure that can be evaluated (Che et al. 2024).
Studying the Development of Concept Representations. RepE could be used to identify and steer representations for the same concept in different stages of training different models, thus giving insights about when representations of a concept emerge and how they change. One could use RepE to study how concepts evolve throughout, pre-training and how they are influenced by various post-training methods or how concepts emerge across models of different sizes in the same family.
14.5 Building a More Rigorous Science of Representation Engineering
The field of Representation Engineering is young and does not yet stand on a strong scientific foundation. To enable progress in the field, we point out the opportunity to make research into RepE more rigorous and to provide a better grounding for the field. The most important development here would be the introduction of a high-quality, widely accepted benchmark for mcasuring RepE methods, which we propose Scction 8.3 as well as the adoption of rigorous evaluation standards discussed in Section 8.2
Developing Theoretical Frameworks. The field of RepE is almost entirely based experimental evidence and lacks a clear thcoretical framework. Fields such as Adversarial ML and Machine Unlearning have greatly benefited from a shared theoretical framework that formulates the settings and objectives (Cao & Yang, 2015). Such a theoretical framework could clearly state the problem RepE attempts to solve and provide criteria for success. For example, current thcoretical work on RepE has found that differencc-inn
means is optimal at identifying a vector that shifts negative to positive examples [Im & Li 2025 Singh
et al. 2024
48

---

Published in Transactions on Machine Learning Research (09/2025)
Causality and RepE. RepE aims to identify representations that are causally related to concept of interest and apply interventions that cause predictable changes in output behavior. Taking a causal perspective on RepE, the treatment could be the relation of a string to the concept or the intervention applied to the representations, and the effect could be the change in activations or in output behavior. Framcworks from causality, such as Structural Causal Models (SCMs) based on Directed Acyclic Graphs (DAGs) (Pearl 2014) and the Potential Outcomes (PO) framework (Rubin 1974) could provide a thcoretical grounding for RepE.
A causal graph describing the computations of the model allows for the application of the do-calculus. This would enable us (1) to find minimal interventions that have the desired steering effect via the back door criterion, (2) to estimate the causal effect of representations despite the presence of unobserved confounders via the front door rule and (3) to distinguish interventions that have a direct or indirect effect on the desired outputs. However, this comes with multiple assumptions and requires one to identify how patterns in the activations of a network relate to nodes the causal graph. Indood, such an attempt has been made by formalizing (Geiger al.) 2024a) and conducting (Geiger al.) 2024b) the search for DAG of high-level concepts that aligns with the neuron-level activity in the network.
Alternatively, using the PO framework does not require DAG and instead leverages covariates. This allows the use of tools such as propensity score matching or doubly robust estimation to assess the causal effect of modifications to representation and thus enables causal adjustments of the interventions. However, this assumes that confounding factors can be adequately controlled using observed covariates.
Lastly, causality can inform the design of imput datasets from which concept operators are derived Deng et al. 2025 This can help to better isolate the causal mechanism from confounders and improve ood gencralization.
Studying RepE on Toy Setting. Krasheninnikov & Krueger 2024] develop a simple classification task as toy setting to run controlled experiments evaluating RepE methods for tasks of different complexities. Such toy scttings can help us to gain a doeper understanding how and why RepE works. They can also serve as lestbeds to evaluate the strengths and weaknesses of different RepE methods. Such toy setups could be used to study the impact of model size, data quality and assumed feature geometries.
Building Library of Concept Reprosentations. An ambitious effort could aim to map out library of concepts and their representation in specific model. This could serve valuable resource for later study. For example, by using this database, it might be possible to train models that can predict the concept operator for new concepts. Furthermore, such a database could serve as a baseline for the development of new RepE methods.
Finding Best Practices. Practitioners using RepE want to casily know which methodology most effective. However, currently, is not clear which method should be used practically. Finding best practices for using RepE and then describing them in an easily accessible form could greatly increase the adoption of this technique.
Developing Tooling for RepE. Great software tools for fine-tuning, such as the Transformers library Wolf et al., 2020), make it easy to use this technique, thus boosting its adoption. Developing dedicated tooling, such as APls or libraries, for RepÉ could be similarly valuable. An example of such work Gemma Scope which allows novices to analyze and steer SAE features dcrived from Gemma-2-2b.
Neuronpedia 2025)
Takeaway: Opportunities for Future Research
Future RepE methods can expand restrictive notions of representations, combine multiple RI methods and attempt more complex interventions. Untapped applications lie in steering goals and values of agents or preventing deceptive behavior. To progress the field of RepE, thorough evaluations and best practices are needed.
49

---

Published in Transactions on Machine Learning Research (09/2025)
15 Conclusion
Representation Engineering (RepE) is novel paradigm for controlling and interpreting Large Language Models by manipulating their internal representations. This survey has provided comprehensive overview of RepE methods, focusing on different methods to identify, operationalize, and control the representations of the model. By syuthesizing insights from over 100 recent studies, we have highlighted key advancements, methodologics, and challenges in this rapidly evolving field.
Our analysis reveals that most. RepE, methods identify representations by providing contrasting inputs and controlling them by adding vector to the activations. Furthermore, many approaches are based on the Lin- car Representation Hypothesis. These methods have been applied to AI Safety, Ethics, and Interpretability by controlling concepts such as Harmfulness, Fairness, and Truthfulness.
We find that RepE tends to be more effective at a lower cost to generation quality compared to prompting, fine-tuning, and decoding-based methods. However, challenges remain in ensuring reliability, robustness, and quality, especially for tasks requiring long-form generation and multi-concept control. Issues such spuriously related concepts, assumptions about feature geometries, and distributional shifts caused by RepE must be addressed. Further research should prioritize the developmentt of more rigorous ways of evaluating and comparing RepE. methods. Furthermore, researchers should explore representations that are non-linear, developing over time, spanning layers, and interacting with each other.
We believe RepE will prove to have important advantages compared to other control methods. RepE might be especially advantagcous when there is a need to control not only the output of the model but also the internal processes by which it arrives at these outputs. As such, likely that RepE' will become common tool for engineers to control and for researchers to study LLMs.
15.1 Limitations
Our survey only comprehensive with regard to RepE papers published before 31.8. 2028. While we include many papers published until 31.2.2025 we do not comprehensively cover them all. Papers covered after this period are not covered.
While we attempted to synthesis the quantitative evidence on best practices for RepE in Section 7.2 to leave many questions unanswered. This is due to a lack of empirical comparisons that shed light on these questions.
we had
Broader Impacts
RepE is a general tool for controlling the behavior of LLMs. In many cases RepE has been devised and used for applications such as steering LLMs to reduce harmful outputs, align them with human intent, make them more honest or uncover societal biases. However, RepE faces a dual-use risk since these concepts can also be steered negatively. For example RepE has been used to jailbreak LLMs bypass safety mechanisms and produce more harmful outputs. Similarly, RepE could be used to steer the model become misaligned, generate deceptive text or to strengthen socictal biases. Additionally, RepE has been used to interpret model internals. However, it does not offer perfect explainabil- ity. Thus there is risk that an illusion of interpretability is created where users trust the these explanations more than is warranted.
Acknowledgments
The authors are grateful for thoughtful feedback from Ruta Binkyte, Dmitrii Krasheninnikov, Ivaxi Sheth, Sarath Sivaprasad, Alex Cloud and Simon C. Marshall.
This work was partially funded by ELSA European Lighthouse on Secure and Safe AI funded by the European Union under grant agreement No. 101070617. This work was also partially supported by the ELLIOT Grant funded by the European Union under grant agreement No. 101214398. Views and opinions
50

---

Published in Transactions on Machine Learning Research (09/2025)
expressed are however those of the authors only and do not necessarily reflect those of the European Union Furopean Commission. Neither the European Union nor the European Commission can be held responsible for them.
References
Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. Are you still on track!? catching Ilm task drift with activalions. arXiv preprinl, 2024a. URL https://arxiv..rrg
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Cooperation, com- petition, and maliciousness: LLM-staacholddrss interactive negotiation. In The Thirty-eight Confer- ence on Neural Informalion Processing Systems Dalasets and Benchmarks Truck, 2024b. URL https: //openreview.netee http::/wwwwwee.ooo///o/-//e/eee-
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Sogaard. Can language models encode perceptual structure without grounding? case study in color. In Proceed- ings of the 25th Conference on Compulational Natural Language Leurning, pp. 109 132. Association for Computational Linguistics, November 2021. URL https: https::/accaanthlooo org wttp/rgg/.//...oe/ee/.oo...ooooo 201.2201....111...
Christopher Ackerman. Representation luning. In MINT: Foundalion Model Interventions, 2024. URL
Dyah Adila, Changho Shin, Yijing Zhang, and Frederic Sala. Can language models safeguard themselves, instantly and for free? In ICML 2024 Next Generation of AI Safety Workshop, 2024a. URL https: //openreview.co weteenreveee.ee/ffooreee http:/orrmmm211AAARs11111
Dyah Adila, Shuai Zhang, Boran Han, and Bernie Wang. Discovering bias in latent space: An unsupervised debiasing approach. In Forty-first International Conference on Machine Learning, 2024b. URL https: //openreview.o mtt.:/oomommmmmmmmdddddd1d
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.
URL http::://axxxx..oggaab////e///ee 01644
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https://aaxxti..orrrr abs 2406. 11717
Aryaman Arora, Dan Jurafsky, and Christopher Potts. Causalgym: Benchmarking causal interpretability methods on linguistic tasks. arXie preprint, 2024. URL
Matan Avitan, Ryan Cotteroll, Yoav Goldberg, and Shauli Ravfogel. Intervention lens: from representation surgery to string counterfactuals, 2024. URL
Sarah Ball, Frauke Kreuter, and Nina Panickssery. faakersttandinggiiilbbeee success A study of latent space dynamics in large language models, 2024. URL
092889
Somnath Banerjee, Soham Tripathy, Sayan Layek, Shanu Kumar, Animesh Mukherjee, and Rima Hazra. Safeinfer: Context adaptive decoding time safety aligument for large language models, 2024. URL https:
//axxiv. wwrxii..rrg/bb/2406... 12274
Daniel Beaglchole, Adityanarayanan Radhakrishnan, Enric Boix-Adsen and Mikhail Belkin. Aggregate and conquer: detecting and steering llm concepts by combining nonlinear predictors over multiple layers, 2025.
URL
Loonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety a review. Transactions on Machine Learning Research, 2024. URL http:://ggaaeeeeeee.oo/o//orrrer Survey Certification, Expert Certification.
51

---

Published in Transactions on Machine Learning Research (09/2025)
Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martin Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Forty-second International Conference on Machine Learning, 2025. URL https://oppenrevieer
aet/foorumiddaa1122yWWM
Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utterances for safety-alignment, 2023. URL https://axxiv: httg://.b///0....o// 2366226668806602222222
Anand Bhattad, James Soole, and D.A. Forsyth. Stylitgan: Image-based relighting via latent control. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition (CVPR),
pp. June 2024. URL
theceeessssttt.e..eeee thecvf concvf..com.. cow/
StyLitGAN Image-Based Relighting. via Laaaa__tett___e Cacpaett_oooootooeeeee CVPR__CUP____________ 20IiR_2224_paaap_ htmll.
Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, and Andreas Bulling. Benchmarking mental state rep- resentations in language models. In ICML 2024 Workshop Mechanistic Interprelability, 2024. URL
https://openrevviet wttp://wnnnttrrrmr//ooo/o/Eoooo
Nick Bostrom. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines, 22:71-88 85, 2012. URL http://iww..ppppngg.e...oo/r///s
611023-012-9281-
Joschka Braun, Carsten Eickhoff, David Krueger, Seyed Ali Bahrainian, and Dmitrii Krasheninnikov, Un- derstanding (un)rcliability of steering vectors in language modcls. Forthcoming, 2025.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards monosemanticity: Decompos- ing language models with dictionary lcarning, 2023. URL
Madeline Brumlcy, Joe Kwon, David Krueger, Dmitrii Krasheninnikov, and Usman Anwar Comparing bottom-up and lop-down steering approaches on in-context learning tasks. In MINT: Foundalion Model Interventions, 2024. URL
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent. knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023. URL
https:://openreview het.://eenneoooourrroeoeattooaaa
Min Cai, Yuchen Zhang, Shichang Zhang, Fan Yin, Difan Zou, Yisong Yue, and Ziniu Hu. Self-control of LLM behaviors by compressing suffix gradient into prefix controller. In ICML 2024 Workshop Mechanistic Interpretability, 2024. URL htttps:/
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium Security and Privacy, pp. 463 480, 2015. UtI.//ttpps://eeexxllrrre. ieeeeerrreeeee....... wew...eeeeeeeeee................ org/ abstract//
document/7163042
Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization, 2024a. URL ftttps://arxiv.org/// wwt.obg///aao.eeeooooooooo
Zouying Cao, Yifei Yang, and Hai Zhao. Nothing in excess: Mitigating the exaggerated safety for 1lms via safety-conscious activation steering. arXiv preprint, 2024b. URL htttp:://arrdv.ogg/bb//////
11491
Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. Hatebert: Retraining bert for abusive language detection in english. In Aida Mostafazadeh Davani, Douwe Kiela, Mathias Lambert, Bertie Vidgen, Vinodkumar Prabhakaran, and Zecrak Wascem (cds.), Proceedings of the 5th Workshop on Online Abuse and Harm, pp. 17-25. Association for Computational Linguistics (ACL), July 2021. doi: 10. 16186615122000 w...-
52

---

Published in Transactions on Machine Learning Research (09/2025)
Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. Improving steering vectors by targeting sparse au- toencoder features, 2024. URL https://arxiv..
Zora Che, Stephen Casper, Anirudh Sathoesh, Rohit Gandikota, Domenic Rosati, Stewart Slocum, Lev E McKinney, Zichu Wu, Zikui Cai, Bilal Chughtai, Furong Huang, and Dylan Hadfield-Menell. Model manipulation attacks enable more rigorous evaluations of LLM unlearning. In Neurips Safe Generative Al Workshop 2024, 2024. URL https: //tppreee//peeeeee..... wtt.://tottooom//ddd////./..
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Πly Sutskever, and Picter Abbeel. Infogan: Inter- pretable representation learning by information maximizing generative adversarial nets. In Advances in Newral Information Processing Systems, 2016. URL https://proceedinngg
cttpsperccpppp.rr.eep files/
pdf
Yida Chen, Aoyu Wu, Trevor DePodesta, Catherine Yeh, Kenneth Li, Nicholas Castillo Marin, Oam Patel, Jan Riecke, Shivam Raval, Olivia Seow, et al. Designing a dashboard for transparency and control of conversational ai. aiei preprint orXi:224060077 07882, 2024a. URL htcts::/aaxxv..orr//b///220/ 07882
Zhongzhi Chen, Xingwu Sun, Xianfeng Jiao, Fengzong Lian, Zhanhui Kang, Di Wang, and Chengzhong Xu. Truth forest: Toward multi-scale truthfulness large language models through intervention without tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19):20967 20974, Mar. 2024b.
URL https://ojs. aaai. wtt.:///nne..pppp//a//////////// wttp://A////rrttil/e///e////////
Zhixuan Chu, Yan Wang, Longfei Li, Zhibo Wang. Zhan Qin, and Kui Ren. causal explainable guardrails for large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Com- munications Security, pp. 1136-11500 New York, NY, USA, 2024. Association for Computing Machinery. ISBN 97984007063633. URL
1145/3658644 3669277336666443636777
Alex Cloud, Jacob Goldman-Wetzler Evžen Wybitul, Joseph Miller, and Alexander Matt Turner. Gradient routing: Masking gradients to localize computation in neural networks, 2024. URL https://arxiv.org
abs/2410.0433
Joshua Clymer, Caden Juang, and Severin Field. Poser: Unmasking alignment faking lims by manipulating their internals, 2024. URL btttpp://aaxii.orrr/bbbb// 05466
Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics of gans. In CVPR, 2020.
Arthur
Online post, http:///wwwallinmmoofforoo org wwtpr///pstt///k//////eppepe///e full 1 poost-progress-uudat------- --trottt---r-reee-ee--e---- gww---ee-----nee---eeeeee pttppr/swssppttttt..o.rtrrreeerr
Conmy and Neel Nanda. URL
Activation steering with saes.
April 2024.
Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks
learn to store and generate sequences using non-linear representations. In Proceedings of the 7th Black- borNLP Workshop: Analyzing and Interpreting Newral Networks for NLP, pp. 442, November 2024.
URL
blackboxmlp-1.177
Maria De- Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Choulde- chova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Ac- countability, and Transporency, pp. 120 128, 2019. URL httpp:://dii.rgg/// 1145/328816600 .3875572
Daryna Dementieva, Nikolay Babakov, and Alexander Panchenko. Detecting text formality: A study of text classification approaches. In Ruslan Mitkov and Galia Angelova (eds.), Proceedings of the 14th Inter- national Conference on Recent Advances Natural Language Processing, pp. 274-284, Varna, Bulgaria, September 2023. INCOMA Ltd., Shoumen, Bulgaria. URL[https httlattts//aclaatoollggg. http://cclaatttoooooo/////2//2/2 wrg/ ranlp-1, 31/!
53

---

Published in Transactions on Machine Learning Research (09/2025)
Dorottya Demszky, Dana Morshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoFmotions: A dataset of fine-grained emotions. Tn Proceedings of the 58th Annual Meeling of the Association for Computational Linguistics, pp. 4040 4054, July 2020. URL http:://aclaatthlllgg. .rgg 2020. aalaccllaminn
Zhihong Deng, Jing Jiang, Guodong Long, and Chengqi Zhang. Rethinking the reliability of representation engineering large language models, 2025. URL
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp.
2021. URL httph::t///att org/oog///. 10. 1145/344211883 44454444 3445924
Peiran Dong, Bingjie Wang, Song Guo, Junxiao Wang, Jie ZHANG, and Zicong Hong. Towards safe concept transfer of multi-modal diffusion via causal representation editing. In Advances in Newral Information
Processing Systems, pp. 12708-12738, 2024a. URL https: htttps::/ wwwrips.cc/ppppee files/
pdf
Weilong Dong, Xinwei Wu, Renren Jin, Shaoyang Xu, and Deyi Xiong. Contrans: Weak-to-strong alignment engineering via concept transplantation. arXiv preprint, 2024b.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable 1lm feature circuits, 2024. URL ptttpp://aaxiv.oort wwwg./a////...1111/////// 11944
Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, Oliver Rausch, Saffron Huang, Sam Bowman, Stuart Ritchie, Tom Henighan, and Deep Ganguli. Evaluating feature steering: A case study in mitigating social biases, 2024. Available at
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, September 2022.
URL
http://22//toy_mmee//mmmnne html
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnce, and Max Togmark. Not all language model features are linear. In The Thirleenth International Conference on Learning Representations, 2025. URL
https://openreviie
wtwpeerrwwweeeeeet/oouuuuiiiaaia net//oormm
Eoin Farrell, Yeu-Tong Lau, and Arthur Commy, Applying sparse autoencoders to unlearn knowledge in language models. In Nearips Safe Generative AI Workshop 2024, 2024. URL httpss//openrevierr net/
forum?id=i4zOHrrBiAA
Farima Fatahi Bayat, Xin Liu, H. Jagadish, and Lu Wang. Enhanced language model truthfulness with learn- able intervention and uncertainty expression. In Findings of the Association for Computational Linguistics ACL 2024, pp. 12388 12400, August 2024. URL https://aclaathhology http:aatoooooooooo/222...ff wwtp:o//......iiioa.ooooooeeeeea findings-acl.... 737
org/2024.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà A primer the inner workings of transformer-basee language models, 2024. URL https:://axxv..ott abs: 00208
Jinlan Fu, Sec-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. GPTScore: Evaluate as you desire. In Pro- ceedings of the 2024 Conference of the North Americun Chapter of the Association for Computational June 2024. URL
Linguistics: Human Language Technologies, pp. org/2028 2012444nacc-oonn 365/
Loo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. CRL httts:://taaat http///aw....gg///////...00000
54

---

Published in Transactions on Machine Learning Research (09/2025)
Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chanhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability, 2024a. http:ttt//ati..orrg//////////
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Francesco Locatello and Vanessa Didelez (eds.), Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236, pp. 160 187, Apr 2024b. URL
Asma Ghandeharioun Ann Yuan, Marius Guerard, Emily Reif, Michael A. Lepori, and Lucas Dixon. Who's asking? user personas and the mechanics of latent misalignment, 2024. URL https: https: http://axxvvv.or//aa//
2406.12094 12094]
Jacob Goldman-Wetzler and Alexander Matt Turner. found >80 orthogonal 'write code"
steering vectors, July 2024. URL httpps://wwe wwwpw/wwwwwww.g...oe//// wwt.c///ostttttttttpppppppaaaaaa i found-greater-thaat
thaaatter-taa-80000 -atha-8000-o -hottooo-oo------ooaoaalll www.ww-iieee-e-- code----rrcddeeeeee
Accessed: 2025-02-04.
Nicholas Goldowsky-Din Bilal Chughtai, Stefan Heimersheim, and Marius Hobbhahn. Detecting strategic doception using lincar probes, 2025. URL http:/sss///axt..oor///////////0
Gabricl Grand, Idan Asher Blank, Francisco Percira, and Evelina Fedorenko. Semantic projection: recovering
human knowledge of multiple, distinct object features from word embeddings, 2018. URL https://arxxi:
org/ awwaaaas...a/0 01241
Ping Guo, Yubing Ren, Yue Hu, Yanan Cao, Yumpeng Li, and Heyan Huang, Steering large language models for cross-lingual information retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 585-596, 2024. URL https://doi.org///
1145/3626772.365788
Sophie Hao and Tal Linzen. Verb conjugation in transformers is determined by lincar encodings of subject nunber. In Findings of the Association for Compulational Linguisties: EMNLP 2023, pp. 4531 4539, December 2023. URL https::/acclatthollgg/ hwwp:/hhoooogyo.g/2222222 www.o//g.........o htw../..oor.22.....dnggg---a eml1p.300
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: A large-scale machine-generater dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeling of the Assucialion for Compulational Linguislics, pp. 3309 -3266,,3,,.-- May 2022. URL https::/acclatthollst http://llannbollooooo///222222 awt.22222ccccl1..
234
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Mcasuring massive multitask language understanding. In International Conference on Learning Representations, 2021. https:://pperrevie...t htt.::ewweeeeteooommmidd/d//-/-
Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations language models. In First Conference on Language Modeling, 2024. URL htttps://openreview..n net/ forum?id=ADEL6fglWk
Daniel A. Herrmann and Benjamin A. Levinstein. Standards for belief representations in llms, 2024. URL /ttps://arxiv..
220504405550 21030
Bertram Højer, Oliver Simon Jarvis, and Stefan Heinrich. Improving reasoning performance in large lan- guage models via representation engineering. In The Thirteenth International Conference on Learning Representations, 2025. URL httpps/s/ggggeeeeeeeeeeee//////p
Jakub Hoscilowicz, Adam Wiacek, Jan Chojnacki, Adam Cieslak, Leszek Michon, Vitalii Urbanevych, and Artur Janicki. Non-lincar inference time intervention: Improving 1lm truthfulness. In Proc. Interspeech 2024. pp. 4094-4098, 2024. URL htt.::ttps//// :/www/ wwtccaaarchive.... interspeech. html
55

---

Published in Transactions on Machine Learning Research (09/2025)
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL httptee//pppeeeeeeeeeerrr/rrrrre wttp:/oorrr.t---eeeeeeeeee.
Lijie Hu, Liang Tiu, Shu Yang, Xin Chen, Hongru Xiao, Mengdi Li, Pan Zhou, Muhammad Asif Ali, and Di Wang, A hopfieldian view-based interpretation for chain-of-thought reasoning, arXiv preprint, 2024.
URL https:// http://aaxvv.... http://vxvvooorrbbaraaaaaea http//bbb2440..... 2406. 12255
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autocncoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https::/openreeveen net:/forummddd-777648St...
C. J. Hutto. SaderSentiment: Vader sentiment analysis (version 3.3.2).
https://pypi..oggprrriis
vaderSentiment/ May 2020. Python Package Index.
Shawn Im and Yixuan Li. A unified understanding and evaluation of steering methods, 2025. URL https:
7/arxiv. wwwpxivvrrg/aae wttp:oorg////o///eooo 2ttp:/552.022111
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Tlama guard: Llm-based input-oulput safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023. doi: 10.48550/ arXiv.2312 06674. URL https://arxi>: //rrvvv wwwrriiv...rr/aaee 2aabs//312 06674
Ali Jahanian, Lucy Chai, and Phillip Isola. On the 'steerability" of generative adversarial networks, In International Conference on Learning Representations, 2020. URL http:s///ppeerrri..nttt/ooooooot HylsTT4FvB
Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In Forty-first International Conference on Machine Learning, 2024a. URL httpp:///penreevie. http:///pppprpoeeeeeee/e/eeceeea wtt.:///ooorumnooooouuuthtaaaaa
Zhanhong Jiang, Nastaran Saadati, Aditya Balu, Minh Pham, Joshua Russell Waite, Nasla Saleem, Chinmay Hegde, and Soumik Sarkar. A unified convergence theory for large language model efficient fine-tuning. In OPT 2024: Optimization for Machine Learning, 2024b. URL http:::///panrevvee..nooffooo/o F01q26eITJ
Ole Jorgensen, Dylan Cope, Nandi Schoots, and Murray Shanahan. Improving activation steering in lan- guage models with mean-centring. In Proceedings of the Responsible Language Models Workshop (ReLM) at AAA1224, February 2024. URL https://sites..ooggt/ 0
Adam Karvonen. Emergent world models and latent variable estimation in chess-playing language models. arXiv preprint, 2024. URL http:////xxvvv..o/bbb/
15498
Dmitrii Kharlapenko, neverix, Neel Nanda, and Arthur Conmy. Extracting sae task features for in-context learning. Online post, August 2024. URL sttps://www.alignnent www.waafooinnnoomommmmmmmm orum org/posts/ JFCCmn/JrgggGcccyf////sttrrrerr htt.oo/////eeetttttttroccaee/ett /ttp///rrciiina-eeaatttt-------e sae-task-featuree for- in- www--ooneextteee -eattext-learning
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting atlention layer putp with sparse auloencoders.. In JCML 2024 Workshop on Mechanistic Interprelability, 2024. URL httpp:://peerrevier: htt.://eeeeeeteeeeeec/oooooooran
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Balf, Dominik Opitz, and Tobias Hecking. Style vectors for steering generative large language models. In Find- ings of the Association for Computational Linguisties: EACL 2024, pp. 782-802, March 2024. URL
https://aclaantholog. wwwp:///ooooo.o//2..//.2/ http:/000...aaiingeeaaaa.......
Dmitrii Krasheninnikov and David Krueger. Steering clear: A systematic study of activation steering in a toy setup. In MINT: Foundation Model Interventions, 2024. URL httts::/ wtw.:rrriiwioeeeeoouuunn..i.
ygvbAGTgzA
56

---

Published in Transactions on Machine Learning Research (09/2025)
Chak Tou Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. Self-detoxifying language models via toxification reversal. In Proceedings of the 2023 Conference on Empirioal Methods in Natural Language Processing, pp. 4433 4449, December 2023. URL https://acllnttoooog..ooo www.ooloooy.gg//222./ emn pp-main...
www.pemannmmm....mmmmemmmmmmmaam 269 main.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, 2023a. URL
Kenneth Li, Yiming Wang, Fernanda Viégas, and Martin Wattenberg. Dialogue action tokens: Steering language models in goal-directed dialogue with multi-turn planner. arXiv preprint, 2024a. URL https:
//arxiv.org/aai orarxiv.orgg/bb/240666 hbwp///bs//4....../ 11978
Nathaniel Lı, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann- Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert- HerserttVoss,, Cort B Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam Alfred Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishma Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP benchmark: Measuring and reducing malicious use with unlcarning. In Forty-firs: International Conference on Machine Learning, 2024b. URL https: //openreview.c wwwparrrreeveeettoommmdeiilillle
Tianlong Li, Shihan Dou, Wenhao Liu, Muling Wu, Changze Lv, Rui Zheng, Xiaoqing Zheng, and Xuanjing Huang. Rethinking jailbreaking through the lens of representation engincering, 2024c. URL https:
//arxiv. /larxiv.org/ale wrgg/aa///000.000000 06824
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettle- moyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12286-12312, July 2023b. URL hattps://aclanthollog) .wrpoogg...rggg// wwgoor///22 acl acl-long. 687
Yu Li, Zhihua Wei, Han Jiang, and Chuanyang Gong. Destein: Navigating detoxification of language models via universal steering pairs and head-wise activation fusion. arXiv preprint, 2024d. URL https:
//arxiv. wersiiv.org/aaie www.oogg////bbaeeeeeeeeeeeeeee/e aa////bb/////// 10464
Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu, Hao Wang, and Dimitris N Metaxas. Implicit in-context learning. arxio preprint, 2024c. URL https: org/abs/24055 14660
Stephanic Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human false- hoods. In Smaranda. Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th An- reall Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https://aclaatthology bxg/2022 acl 1o1.......22
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for layer features and model diffing. Online article, October 2024. URL https:
(trransforer--irrant
Huanshuo Liu, Hao Zhang, Zhijiang Guo, Kuicai Dong, Xiangyang Li, Yi Quan Lee, Cong Zhang, and Yong Liu. Ctrla: Adaptive retrieval-auugmente generation via srobe-guided control. arXiv preprint, 2024a.
URL htttp::/arxiv..or//ttee aat/a//s//245 18727
Sheng Liu, Haotian Yc, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. In Forty-first International Conference on Machine Learning, 2024b. URL
57

---

Published in Transactions on Machine Learning Research (09/2025)
Wenhao Liu, Xiaohua Wang, Muling Wu, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint, 2023. URL http://aaxxiv..r//aaaaaa// 2212. 15997
Francesca. Lucchetti and Arjun Guha. Activation steering for robust type prediction in codellms. arXiv preprint, 2024. URL https://arxiv, http:///xxiv..rro waagg/bb//20 wwwp///4.........
Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Darshan Thaker, Aditya Chattopadhyay, Chris Callison- Burch, and René Vidal. Pace: Parsimonious concept engineering for large language models, 2024. URL
https: //apxii/ orgooogggaaa.oooooee aaaaabaaa/2/ abbb/400..om 04331
Xinyu Ma, Yifeng Xu, Yang Lin, Tianlong Wang, Xu Chu, Xin Gao, Junfeng Zhao, and Yasha Wang. DRESSing up LLM: Efficient stylized question-answering via style subspace editing. In The Thirteenth International Conference on Learning Represenlations, 2025. URL https:t/p//// http::://operreviee /ttp///rreiee..e///orrururu www.:neunfofiiiioemmm nNVR9jJYqK
Monte MacDiarmid, Timothy Maxwell, Nicholas Schiefer, Jesse Mu, Jared Kaplan, David Duvenaud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. Simple probes can catch sleeper agents. Online article, April 2024. URL httpps//www.anthrop..cc http://ww.nnbrrpp....ooo////////
Accessed: 2025-02-14.
Andrew Mack and Alex Turner Mechanistically eliciting latent behaviors in language URL https://ww.alligmant..ooo http://peno..ooooooooo/o//////tt http:///stts///oaailleyyyyooooo.
models, 2024a. Seciaaastiialllyaalaaiiinnee-eee latentt- bew.rbaroorrrr-----aaaaa------
Andrew Mack and Alex Turner. Deep causal transcoding: A framework for mechanistically eliciting latent behaviors in language models. Online post, December 2024b. URL httppp:://ww..lliggnnntttttt org/
Alcksandar Makelo, Nathaniel Monson, and Julius Adebayo. Evaluating sparse autocncoders for controlling open-ended text generation. In Second NewrIPS Workshop on Attribuling Model Behavior al Scale, 2025.
URL http::///ppprrrrvveeeeeetttrrrrr
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of ffalfalse datasets. In NewrIPS Workshop on Attributing Model Behavior at Scale, 2023.
URL httpss:/ hwt::ormmmddddgiiia.........
Harry Mayne, Yushi Yang, and Adam Mahdi. Can sparse autoencoders be used to decompose and interpret steering vectors? In Newrips 2024 Workshop MINT: Foundation Model Interventions, 2024. URL https:
//openreview //openteviee..ne///oouumm
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, pp. 17359-17374 2022. URL https://prooeeddnggs wwwpisseerripp... weeurip..ccpppppeeeeeee cc/paper
Ulisse Mini, Peli Grietzer, Mrinank Sharma, Austin Meek, Monte MacDiarmid, and Alexander Matt Turner, Understanding and controlling a mazo-solving policy network, 2023. URL /tttts::tt//attteeleeeeetttttt/t wwwpexivv..oor.aaoooo abs abs/2310. 2310.
08043
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability, 2024. URL /httts:://aratt www//bs/////aeeeeee.oeeee habbs/2408.044 01416
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet,, Najoung Kim, Arya McCarthy, and Hoscin Mohebbi (eds.), Proceedings of the 6th BlackbozNI.P Workshop: Analyzing and In- terpreting Neural Networks for NLP, pp. 16-30, Singapore, December 2023. Association for Computational Linguistics. URL https://acllathoooogy wwgp://3.../3203 blackboxnlp-1..2
58

---

Published in Transactions on Machine Learning Research (09/2025)
Neuronpedia. Exploring Gemma Scope, 2025. URL http:://ww..aarooopll....ooeeeee cessed: 13 Feb. 2025.
Ac-
Duy Nguyen, Archiki Prasad, Elias Stengel- Eskin, and Mohit Bansal. Multi-attribute steering of language models via targeted intervention, 2025. URL http://s//aaxx..oo////////2222.0
OpenAL. Openai ol system card, 2024. URL https://arxiv.org/bbaa. /4122/211/
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Stocring llama 2 via contrastive activation addition, 2024. URL https://arriiv.oo//bb///331.. 06681
Bumjin Park, Youngju Joung, Yconjea Kim, Leejinsil, and Jaesik Choi. Mcasuring effects of steered repre- sentation in large language models, 2024a. URL hbtp:s///ppaaaerrre..eee/////rrp
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-forst International Conference on Machine Learning, 2024b. URL https://openreviet http://wweeeeooomm/aaaoouuutaaaa
Peter S Park, Simon Goldstein, Aidan O'Gara, Michael Chen, and Dan Hendrycks. Ai deception: A survey of examples, risks, and potential solutions. Patterns, 5(5), 2024c. URL https://ww..cienceeet scienceeeeenc...t.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: hand-built bias benchmark for question answering. Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086-2105, May 2022. URL
attpss//aclantholloe hww.aaahhoooooorrg//2222222. http//22...idingssallll... .65
Gonçalo Paulo, Thomas Marshall, and Nora Belrase. Does transformer interpretability transfer to rnns?
arXiv preprint, 2024. URL htttss:////xiiiii org/ http://axxii.orrraaa/0/.000... abs///44//24400771
Judea Pearl. Probabílistic reasoning intelligent systems: networks of plausible inference. Elsevier, 2014.
URL https::/www.jstatt wttp:/jsssoooorsstblloo/o/o wtg/sttalle/22270505
Van-Cuong Pham and Thien Huu Nguyen. Householder pseudo-rotation: A novel approach to activation editing in lims with direction- magnitude perspective, 2024. URL https://arxii..rrr wtw.//////////////////////// 10053
Joris Postmus and Steven Abreu. Steering large language models using conceptors: Improving addition Aldition-based activation engincering, 2024. URL
Itamar Pres, Laura Ruis, Ekdeep Singh Lubana, and David Krueger. Towards reliable evaluation of behavior steering interventions in LLMs. In Neurips 2024 Workshop MINT: Foundation Model Interventions, 2024. URL https://openreviiet hwtp:/otoottoottt/ooooooooo
Sara Price, Arjun Paniokssery, Sam Bowman, and Asa Cooper Stickland. Future events as backdoor triggers: Investigating temporal vulnerabilities in llms. arXiv preprint, 2024. URL https://arxiv https://arxxv.orr//a////000 orw..//////bbb/0
04108
Chen Qian, Jie Zhang, Wei Yao, Dongrui Liu, Zhenfei Yin, Yu Qiao, Yong Liu, and Jing Shao. Towards tracing trustworthiness dynamics: Revisiting pre-training period of large language models, 2024. URL
19465
Yifu Qiu, Zhao, Yftah Ziser, Anna Korhonen, Edoardo M. Ponti, and Shay B. Cohen. Spectral editing of activations for large language model alignment, 2024. URL
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks, 2016. URL
Nate Rahn, Pierluca D'Oro, and Mare G Bellemare. Controlling large language model agents with entropic activation steering. or Xio preprint, 2024. URI. httpp://aarii..rr////////0.0000
59

---

Published in Transactions on Machine Learning Research (09/2025)
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. Improving sparse decomposition of language model activations with gated sparse autoencoders. In The eightt- Annual Conference on Neural Information Processing Systems, 2024a. URL https::/ net/forummiddz2LB11nm
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Somnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse au- toencoders, 2024b. URL /https://arrti org. wxggabs//4/00 14435
Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learn- ing interpretable concepts: Unifying causal representation learning and foundation models. arXiv preprint arXiv:24 022000966, 2024. URL htttts://axxii..at/ org/ abs httgg//bs/2442.099/ 09236
Brandon Reagen, Udit Gupta, Lillian Pentecost, Paul Whatmough, Sae Kyu Lee, Niamh Mulholland, David Brooks, and Gu-Yeon Wei. Ares: a framework for quantifying the resilience of deep neural networks. In Proceedings of the 55th Annual Design Automation Conference, 2018. URL httpps///oo. http:////oo..rr/1.... org/1..1145/ 0195970.3115999
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszczo, Robic Gonzales, carsten maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning. In The Thirty-eighth Amnual Conference on Neural Information Processing Systems, 2024. URL httts:://ttetrreeee wttt:///oooormm/m//e/aaaaaaa
Julia Rozanova, Marco Valentino, Lucas Cordeiro, and André Freitas. Interventional probing in high dimen- sions: An NLI case study. In Pindings of the Association for Computational Lònguistics: EACL 2023, pp. 2489-2500, May 2023. URL
org/ httg//222...innngs-.... findinggnneaaaeee 188
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5))688, 1974. URL https://pssycnet. apa. o0w.///0ord/1117-000000--------0
Daniel Scalena, Gabriele Sarti, and Malvina. Nissim. Multi-property steering of large language models with dynamic activation composition. In The 7th BlackboxNI.P Workshop ARR Submissions, 2024. URL http::///ppparreeeeee.eefrrrrrre www.:eeentetniiiee www.rrrunnddddooo/oooe33 https://openrevier/
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020. URL https:// thecvf.ee...cc wtt watp///ontttttvvee
CCCCC_22RR/PP__eeee2eee Shen
Interpreting htaprrettinn_eee_Latttt___e_ee Sttpepaeeeoooaaaaaooooeaoaaaaaaa 200.:iiinnn....222121120_21 2020 Baclll_ppppp_____ html html
of CANs CarraatttccaaaeEeiiiiiii
Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gerard Medioni. Gan-control: Explicitly control-
lable gans. In ICCV, 2021. httpppppp://peeeaccceess hwwps//swwsee.ceecccccooooeeeeee http:/rccw..oooaeea/////////////
GaN___NNssSSO CCCNNNNNC__2211a1_____CC 2ww1_0apere_____ html
Adi Simhi, Jonathan Herzig, Idan Szpektor, and Yonatan Belinkov. Constructing benchmarks and interven- tions for combating hallucinations in 1lms. ar preprint, 2024.
Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnurangam Ku- maraguru. Representation surgery: Theory and practice of affine steering. In Forty-first International Conference on Machine Learning, 2024. URL. htttps:/openrrevies http://www......be.edd///ddaaoo/
Anna Soligo, Edward Turner, Senthooran Rajamanoharan, and Neel Nanda. Convergent linear representa- tions of emergent misalignment, 2025. URL https://arxiv. org/
http:///ww....raaarb/../...... attpa/aa////..1.11 11618
Asa Cooper Stickland, Alexander Lyzhov, Jacob Pfau, Salsabila Mahdi, and Samuel R. Bowman. Steering without side effects: Improving post-deployment control of language models, 2024. URL https://arxx::
org/abs/2406 15518
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pre- trained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (ods.), Find- ings of the Association for Computational Linguistics: ACL 2022, pp. 566-581, May 2022. URL
httt:::/acclanhooooo.oo/////
findings-acl.44
60

---

Published in Transactions on Machine Learning Research (09/2025)
Daniel Chee Hian Tan, David Chanin, Aengus Lynch, Adrià Garriga-Alonso, Dimitrios Kanoulas, Brooks Paige, and Robert Kirk. Analyzing the generalization and reliability of steering vectors. In ICML, 2024 Workshop on Mechanistic Interpretability, 2024. URL
Sree Harsha Tanneru, Dan Ley, Chirag Agarwal, and Himabindu Lakkaraju. On the difficulty of faithful th reasoning in large language models. In Trustworthy Multi-modal Foundation Models and AT Agents (TiPA), 2024. URL https: httes:://operreveew http://pppparrteee..eeeeeeeeeaea rtt./fooruum//d33300222aaa
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Cal- lum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https: ottp:///oercccccuuuouo////////// wnw.ii/ooooooeentiiiiitt/itiei
html
Alejandro Tlaie. Exploring and steering the moral compass of large language models. arxiv preprinl, 2024.
URL http:///arxii.orrra/////0/0
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://operrevieeet httpg/////ppppeeeeeeeeeeeerreree wwt../eeeee.oorom/ddddyyttyttaaa
Thien Q. Tran, Koki Wataoka, and Tsubasa Takahashi. Initial response selection for prompt jailbreaking using model steering. In JCLR 2024 Workshop Secure and Trustworthy Large Language Models, 2024.
URL https::/openreviicet wttp:://www.....oo////////ttt/o.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL https: org/
hbs/2308.1024B
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In CVPR, 2017. URL httpss///ppeaccess.. thecvf. cow caan/cootent__vvv cvprr_______________ hwtppc/rpo..///ttt//////rrrrreee Beeet_eeeeeeeepee_e Interpolation_CVO 2017_paperrr___e
Teun van der Weij, Massimo Poesio, and Nandi Schools. Extending activation steering to broad skills and multiple behaviours. arXiv preprint, 2024. URL http:::/aaaxiv.or//bb////////// 2576 05767
Dimitri von Rütte, Sotiris Anagnostidis, Gregor Bachmann, and Thomas Hofmann. A language model's guide through latent space. In Forty-first International Conference on Machine Learning, 2024. URL net/ Feor
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions the GAN latent space. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9786 9796, Jul 2020. URL httpp::/prroceednngg mlr. press/v199/ voynov20a.html
Haoran Wang and Kai Shu. Backdoor activation attack: Attack large language models using activation stecring for safety-alinnment. arXiv preprint, 2023. URL https:///axiv. www./rgg/////e...e... 09433
Huanqian Wang, Yang Yue, Rui Lu, Jingxin Shi, Andrew Zhao, Shenzhi Wang, Shiji Song, and Gao Huang, Model surgery: Modulating I1m's behavior via simple parameter editing. arXio preprint, 2024a. URL
Hui-Po Wang, Ning Yu, and Mario Fritz. Hijack-gan: Unintended-use of pretrained, black-box gans. In CVPR, 2021. URL Unintended of Pretrained Black- CANs_CVPR, 2021 pwwppppppppppp. html
thecvf com http///onneet////22///nnma Hii__Hjjakk- GAN_
61

---

Published in Transactions on Machine Learning Research (09/2025)
Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, and Xipeng Qiu. Inferaligner: Inference-time alignment for harmlessness through cross-model guidance. aiv preprind, 2024b. URL https://arxiv.orrg/ wttp:///.rr/bb///000//0 11206
Tianlong Wang, Xianfeng Jiao, Yifan He, Zhongzhi Chen, Yinghao Zhu, Xu Chu, Junyi Gao, Yasha Wang, and Liantao Ma. Adaptive activation steering: A tuning-free Ilm truthfulness improvement method for diverse hallucinations calegories. arXiv preprint, 2024c. URL
Zihao Wang and Victor Veitch. Does editing provide evidence for localization? In ICML 2024 Workshop on Mechanistic Interpretability, 2024.
Yixuan Weng, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. Controllm: Crafting diverse personalities for language models. arXiv preprint, 2024. URL https://arxiv.org/ http:/rrrr/ab//22211155555
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: Statc-of-thc-art natural language processing. In Proceedings of the 2020 Conference on Empórical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, 45, October 2020. URL http::::/aaaaathhllogoooo org.
Yotam Wolf, Noam Wies, Dorin Shteyman, Binyamin Rothberg, Yoav Levine, and Amnon Shashua. Tradeoffs between alignment and helpfulness in language models with representation engineering, 2024. URL https:
//axxiv. orrxiv.org/o org/ abs/ /40ps/2211...0 16332
Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang. Advancing parameter efficiency in fine-tuning via repre- sentation editing. arXio preprint, 2024a. htteps:////ax wwwpeiiiv.or///bae hts//bbs/2440// 220//24222222 15179.
Xinwei Wu, Weilong Dong, Shaoyang Xu, and Deyi Xiong. Mitigating privacy seesaw in large language models: Augmented privacy neuron editing via activation patching. In Findings of the Association for Compulational Linguislics ACL 2024, pp. 5319 5332, August 2024b. URL https:///clanthollogy org/ 2024. www..fiidings-ac..33
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finctuning for language models, 2024c. URL https: //arxiv.
2ttaabb/2204403599
Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Axbench: Steering 1lms? even simple baselines outperform sparse autoencoders, 2025. URL htttt:://axxiiiii wttp://.rrggaab//51..1111 17148
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image gencration. In CVPR, 2021. URL
thecvf wwtpc///ooooeee////22//////////a Wu SatyyeSppce_Anaa____e Diseettngle wtw.aangeeedoCooollleoooo Controls, for Sarr__StleGAA, Image Generation cicisionn_v_______e 2_22__201___ paper html html
Yuxin Xiao, Wan Chaoqun, Yonggang Zhang, Wenxiao Wang, Binbin Lin, Xiaofei He, Xu Shen, and Jieping
Ye. Enhancing multiple dimensions of trustworthiness in 1lms via sparse activation control. In Advances in
Neural Information Processing Systems, pp. 15730 15764, 2024. URL https::/prooceedings www.eeeedddinneeenir..eeeeeeee
.ddf
Shaoyang Xu, Weilong Dong, Zishan Guo, Xinwei Wu, and Deyi Xiong. Exploring multilingual human value concepts in large language models: Is value aligmment consistent, transferable and controllable across languages? arXio preprint, 2024a. URL htttps://arrii.orr/abb//// 18020
Zhihao Xu, Ruixuan Huang, Changyu Chen, Shuai Wang, and Xiting Wang. Uncovering safety risks of large language models through concept activation vector, 2024b. URL http://axxvvvvo///aa// /abs//44..... 12038
62

---

Published in Transactions on Machine Learning Research (09/2025)
Jingyuan Yang, Dapeng Chen, Yajing Sun, Rongiun Li, Zhiyong Feng, and Wei Peng. Enhancing semantic consistency of large language models through model editing: An interpretabilty-orinntee approach. In Findings of the Association for Computational Linguisties ACL 2024, pp. 3343 3353, August 2024. URL
wwwpetthhoooooo.o/220000 http//222...nnnnss.fsssf findings-acl. 199
Sibo Yi, Yule Liu, Zhen Sun; Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. Jailbreak attacks and defenses against large language models: A survey, 2024. URL
Fangcong Yin, Xi Ye, and Greg Durrett. Lofit: Localized fine-tuning on Ilm representations. arXiv preprint, 2024. URL
Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust Ilm safeguarding via refusal feature adversarial training, 2024. URL https://arxiv, http//aaivi..rggbba//a//aaaa 20089
Ziqian Zeng, Jianwei Wang, Junyao Yang, Zhengdong Lu, Huiping Zhuang, and Cen Chen. Privacyrestore: Privacy-preservinng inference in large language models via privacy removal and restoration, 2024. URL
Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey, ACM Comput. Surv., 57, Jamuary 2025. URL https:
7/doi org/10 1145/3711118
Jason Zhang and Scott Viteri. Uncovering latent chain of thought vectors in language models, 2024. URL https://arxiv. httgg/bb//240999 2409 14026
Jie Zhang, Dongrui Liu, Chen Qian, Ziyue Gan, Yong Liu, Yu Qiao, and Jing Shao. The better angels of machine personality: How personality relates to Ilm safety, 2024a. URL https: wwtpxiiw.org/bb//4000///
12344
Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large language models in truthful space, 2024b. URL
17781
Yihao Zhang, Zeming Wei, Jun Sun, and Meng Sun. Towards general conceptual model editing via adversarial representation engineering, 2024c. URL attp:://aaxxiw.orr/bbb////
Yu Zhao, Alessio Devolo, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, and Pasquale Minervini. Steering knowledge selection behaviours in Ilms via sasebased representation engincering, 2024. URL httph::tt///aatt org/ 2410 15999.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging Ilm-as-a- judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neurol Information Processing Systems, volume 36, pp. 46595 46623. Curran Associates, Inc., 2023a. URL htttps://procceeinngg nww.ips. wttpc/pppeffllee/peeeeeeeeeeeeee ftless/paper/2223/ htwp9//00......20000000000666660
and Bee_eBnnnnnsss..eee pdd
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and lon Stoica. Judging Ilm-as- a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, vol- une 36, pp. 46595 46623, 2023b. URL htttpsss://occceeinnggg neurips cw/paper.p.pp....ppppp files/paper/200 Pap.eea- aand_eechharrsss pdd
Wentao Zhu, Zhining Zhang, and Yizhou Wang. Language models represent beliefs of self and others. In Forty-first International Conference on Machine Learning, 2024. URL https://opperrviier httpps////pperree.eeeeee///rrrr
id=asJTE8EBjg
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and
63

---

Published in Transactions on Machine Learning Research (09/2025)
Dan Hendrycks. Representation engineering: A top-down approach to ai transparency, 2023a. URL fttps://arxxvv.org//t wwt.aa/////......
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023b. URL https://axxx..ccoom org/ abaaab///// /207..
15043
Andy Zou, Long Phan, Justin Wang, Derck Duenas, Maxwell Lin, Maksym Andriushchenko, Zico Kolter, Matt. Fredrikson, and Dan Hendrycks. Improving alignment. and robustnesss with circuit breakers. In The Tiirtt- Annual Conference Newral Information Processing Systems, 2024. URL https:
A Meta-survey
In order to get better feeling for how research on RepE is conducted, we conducted meta-survey where we gathered which models and sample sizes were used. Furthermore, we collect statistics about the papers including authors and publication dates. This meta-survey only includes papers published before 31.8.2024.
A.1 Commonly Used Models
The most popular models are the base and respective instruction-tuned models of Llama-2-7B, Llama-2-1311 Mistral-7B and Llama-3-8B.
We find that a majority of experiments (164 out of 229) are performed on base models that have not been trained with RLHE, while 65 experiments are conducted on preference-tuned models. While experimental results in Dong et al. (2024b); von Rütte et al. (2024 find a larger effect from applying RepE to base models than to instruction- tund models, Bortolctto et al. (2024); Wang et al. (2024c) find no clear relationship between instruction- tuniong and steering effectiveness.
Figure 4 shows that majority of models to which RepE has been applied on lie between 3 10 billion parameters. This is underlined by the fact that 9 out of the 10 most popular models for RepE are all between 7-13 billion parameters. Furthermore, 8 of them belong to the Llama family of models or are derived from Llama models. This calls for more diversity in which model experiments are performed. However, the model scale might not a crucial factor for the effectiveness of RepE. Across 4 papers that compare the steerability of models across different numbers of parameters within the same model family, none find clear relationship between scale and steerability (Arditi et al. 2024 Xu al. 2024a Arora et al. 2024 Bortoletto et al. 2024 Furthermore, only Templeton et al. (2024) able to experiment on proprictary, statc-of-the-art model, thus leaving open questions about how effective RepE is at larger scales and for the most capable models.
Almost all papers focus on transformer language models, while Paulo et al. 20024 find that CAA can also be applicd to RNNs by stecring the in-layer activations or the stato-space. Furthermore, Adila et al. (002bb) apply RepE to a vision-language model. This indicates that RepE can be an architecture-agnodll hod.
A.2 Number of Samples
We note how many samples were used during Representation Identification to derive the operator. Figure 5 shows that most papers use between 100 and 1000 samples, showcasing that RepE can be relatively sample efficient. However, as with fine-tuning and ICL, RepE benefits from using more samples. QQu et al. (2024) Yin el al. (002),, and Wang el al. (2024c) find that the effectiveness of steering improves as the number of samples for Representation Identification increases. However, Adila et al. (2024b) find no such relationship in their unsupervised method. Lastly, Krasheninnikov Krucger (2024) find that more expressive steering functions, whose concept operators have more parameters and that contain more operations, work less well with low amount of data but better leverage high amounts of data.
64

---

Published in Transactions on Machine Learning Research (09/202))
80
60
Number
40
Model Size
Figure 4: The number of experiments applying Repl to model with specific numbers of parameters.
There is no general recommendation the necessary number of samples, which depends the model, target concept and method used. Building more sample-efficien methods and identifying the scaling laws of RepE are promising avemues of inquiry.
10
Amount experiments
8-
4-
2
10a
10
102
103
10a
Number of samples
Figure 5: Amount of experiments that use certain amount of samples.
A.3 Statistics about Publications Representation Engineering
Applying vs. Improving. A majority of papers set out to make methodological improvements to RepE (60 out of 87), while 22 mostly focus on applying RepE to solve a problem. Only 3 papers aim to evaluate RepE methods.
Academic vs. Industry Papers. Work on RepE is mostly done by academics, with 58 out of 87 papers only featuring academic co-authors. Another papers are collaborations between academia and industry, while only 8 papers solely authored by industry researchers.
65

---

Published in Transactions on Machine Learning Research (09/2025)
Publication Date. We take the date of the initial publication of paper and plot them in Figure Here, count the first time paper was published. If a paper was first uploaded arXiv and later published in a conference, we count the date of the initial arXiv submission.
Number of Publications by Month
16
14
Number Publications
12
10
Apr 23
May 23
Jun 23
J3l
Aug
Sep 23
oct33
Nov 23
Dec 23
24 24
Feb 24
Mar 24
Apr 24
May 24
Jun 24
24
Aug 24
Sep 24
Months
Figure 6: Linechart with publications time.
Takeaway: Meta-survey
Models: A majority of models used have between 3-10 billion parameters. According to current literature, there appears to be no relationship between RepE effectiveness and model scale.
Samples: More samples make RepE more effective.
Publication statistics: A majority of papers on RepE come from academics. The literature is very new but growing rapidly.
Survey Methodology
We conducted a structured survey that resulted in a new taxonomy and summary of existing applications but also serves as meta-study that compares RepE to other methods and gathers interesting meta-statistics.
B.1 Literature Search
Firstly, we started with seminal papers [ou al., 2023a Turner et al. 2024 Li al. 2023a) and conducted forward search. Secondly, used the combinations of search terms ("Activation Steering", "Representation Engineering", "Concept Activation Vectors", "Linear Probe + Steering + { ("Language Model", "LLM", "Neural Nelwork"), Thirdly, the citations of relevant papers were checked.
To distinguish papers on RepE from related methods, we adopt five criteria for inclusion and point to papers that wore excluded by this critcrion.
Operates with LLMs. This excludes the sizeable literature on using steering vectors in image generation models like GANs and Diffusion models, which describe in Section 10.8
66

---

Published in Transactions on Machine Learning Research (09/2025)
Identifies Concept Representation. A representation related to the target concept needs to be identified instead of only training the network.
Controls Models' Behavior. This excludes work that solely focuses on interpreting LLMs without modifying representations.
Performs Post-hoc Representation Identification. We focus on identifying representation after training instead of retraining the network to produce desired concept representations.
Steers the Intermediate Representations. We focus on representation steering instead of shifting token embeddings or logits.
This exhaustive literature search was conducted in September of 2024 and only covers papers published until 31.8.2024. We include many RepE papers published after that date but do not guarantee complete coverage of later publications.
B.2 Extracting Information from Papers
While surveying papers, we extracted key information upon which the survey was built. A full list of the information we extract can be found in Appendix C
Regarding the methodology, we developed the taxonomy after a first non-exhaustive read of the literature. On a second, exhaustive read, we classified where methods fall in our taxonomy but also recorded many methodological details that allowed us to map the space of choices when applying RepE. Furthermore, we identify which target concepts were engineered and what problem the paper attempted to solve, thus letting cluster papers into categories of applications. We noted any experimental comparisons with other RepE methods or other methods for controlling LLMs' behavior. Additionally, we gathered experimental details like the model, datasel, and sample size used. Furthermore, we extracted melaainforrmation like the date and venue of publication and association of the authors. To gather views on RepE, we recorded citations where authors expressed how they see RepE as useful, how they think works, what weaknesses they see, and which future work would be valuable. See Appendix [ for all information we extracted from the papers.
We used the assistance of Saaude3..5-Sonne to extract information from papers. While this sometimes sped up the process of identifying key information, it is not sufficiently reliable to be trusted to retrieve information faithfully. Thus, we read every paper ourselves and confirmed the correctness of any ALM-generated answers before including them in this survey.
C Information Extracted from Papers
General.
1. Name of the method
Short summary of the paper
3. Does it improve Representation Engineering methods or apply it to problem?
4. Which and problem does it apply RepE to?
Open problems/future work
6. Publication month and year
Venue
8. Are the authors from academia, industry mix?
Identification.
67

---

Published in Transactions on Machine Learning Research (09/2025)
1. Is it application of previous method? Which one?
2. Description of the identification method
3. Is the information about the concept obtained by reading from inputs, optinizing for outputs, optimizing for an internal loss function?
4. How are concept-related activations elicited?
5. How is the concept vector identified from activations?
6. Is the method supervised or unsupervised?
What data was used for identification?
An example of the prompting format
9. At which layer, in which component, and for which token are activations read?
Control.
Description of the control method
How does it engineer activations?
Does it change activations or weights?
4. At which layer, in which component, and for which token does engineering take place?
Is the change in activations static dynamically adjusted per input?
6. Aside from control, does the paper the concept vector for detection?
Concepts.
1. Which concepts were identified and steered?
2. Which concepts were successfully steered?
3. Which concepts were not successfully steered?
Experimental Details.
Is there an experimental comparison to non-RepE methods? Tf yes, which method is superior?
2. Is there an experimental comparison to other RepE methods? If yes, which method is superior?
3. How many examples were used for identification?
Which datasets was the control method evaluated on?
Which models were experimented on?
Summary of interesting results
Furthermore, we noted arguments related to RepE that were only contained in a few papers:
How does RepE relate other methods/areas?
Why does Repl work?
3. What the
of RepE?
What weaknesses does Repl have?
What principled advantages http:/ddaavaaaaaae between RepE Methods?
68

---

Published in Transactions on Machine Learning Research (09/2025)
Table 14: Collection of studies that compare the steering effect and model quality between Repli and Prompting methods
Paper
RapF method
Promptting muthed
Courept
Emiitnston
Tack
Stocning Effect
Model Quality
Motsic
RipE.
Preuu
Metra
BopE
Prenut
Zhang
ARE
ScI-Bauiaddz
Truididauss
Hunuishos
Tullbooking
Edllcciation Rate
Reboal Bute
Eauttiona
Changing exprosed mnstiins
Score
2.40
2.10
Cal st dl
SELF-CONTROL
Systom Preppting
Tosicity
Prisacy
Bedadiix Tisirity
Data Leskoge
S.. Private data. J.
Tuiisity Soore 1
0.17
0
0.27
77.5
Rossouits
Much Probikun
Accunce T
87.3
400
Jellbrekiing
Attack
Ziing Vilor STG4
CAA
CVT Prompted
Ransoning
Incresse Perkeranae:
Accuncy
71.5
00.8
Clas
Forsshot Frouroking
Trulfidane
trothial mritippo-dusiee
TrusTlde
Crose Earropy
2.01
Th-Then Stoce
System Prongtt
Honuidless
Jellureking
Attoch Succes
17.7
MI-BBauch
2004
ASMR
low abot promsping
Ieformation Ratrioall
muli-lingad QA Touthffrrg HUQ
Bizkk: acuns
22.1
50.3
Touilfdduee
Isto"Irath
Accuneyy
Lovig dl.
3113
Thil
SD
Tuaiciyy
Tockity Prolabiily
02.5
65.5
Paplodiy
18.8
20.8
Thateity
Jollbrook
Safeey Soam f
817
618
MILLU
65.6
33.8
PACR
Procrptteg
Puttifulines
Scutinent
Generatto Bllggrphie
Unbined Gearmitios
Ratisfulsee Senee
40.3
34.6
Cao 112
SCANS
Pronpeiing
Redweing Ovemetosal
Chneet Refwel
Rong Or
laferhliguer
Goal Proupt
Hodarddeso
Inilioaii
Attod Succeo Race 1
0.1
16.6
Aocuncy
56.2
58.5
Horeety
RAG
Accumey
12UL
ICL
Peformecc
In Contedt Leccnnng
Accuncy
75.1
76.87
ACTF
Veoesher prompting
Trithfulnns
Trithfugg4
Trasfldo
41.3
39.5
DAT
U0U
Homidlless
multiccon jullicesk
Attech Succem
55.5
Li 41 ml
37266
KVV
ICL
Sentmen
Rerrlity
Negative - Poertoe Trandarion
Informal - Formal Tranelaton
Formality
Postivitt
15.3
08.3
35.0
65-1
ROUGR:
ROUGR::
683
80.2
65.9
75.9
D Papers that compare RepE to Prompting, Fine-tuning and Decoding- based methods
To allow for reprocucibility of our meta-study in Section 11.2 we list the papers from which we gathered the data about experimental comparisons between RepE and prompting, fine-tuning and decoding-based methods.
Prompting. Zhang et al. 2024c]; Cai et al. 2024]: Konen et al.
2024]
Arditi et al. 2024]] Ghandeharioun
ct al. (2024): Zhang& Viteri [2024] ; Chen ct al. (0024b): Stickland et al. (2024
Guo al. (2024 ; Qiu et al.
2024 3 Leong et al.
(202)) ; Li el al. (2024c); ; Luo et al.
(2024); Cao al.
(024bb) Scalena el al. (2024)
Wang et al. 2024bb
Liu et al. (022bb Li et al. (024e)
Liu et al. (202aa]
Wang et al.
(2024ch: Li et al.
2024a)
Fine-tuning. Lucchetti & Guha
2024)); Li et al.
(024db Yang et al. 2024];
Price et al.
(022)); Pham &
Nguyen 2024g Liu et al. (2024b) Wang et al. (2024b); Wu et al. (2024b); Yu et al.
(024)) y
Leong et al.
(023 Qiu et al. (2024bg
Guo et al. (2024); Stickland et al. (2024) ; Qian et al.
(2024bs 3 Chen et al. (2024b):
Zhang et al. (2024b); Liu et al. (2023): Wang al. (2024a)
Ackerman (024):
: Li et al. 2202bb
Yin et al.
2024h: Cao al.
2024a): Wu et al.
(2024a
Decoding-based
methods.
Li et al.
(2024a d); (022b)
Cao et al.
2024bb
Banerjee et al.
2024
Leong et al.
b023 Qiu et al.
(2024] Zhang et al.
Combination.
Wang et al. (2024c): Li ct al.
2023a
Stickland et al. 2024 5 Chen et al.
2024b
D.1 Table of empirical comparisons
Furthermore, we list the empirical comparisons between RepE and prompting and finc-tuning methods in Table [4 Table 15 respectively. Here we include the steering effect and, if available, a measure for model quality for each concept that was compared. When scores for multiple models or subsets of the test dataset are given we average them into one number. Lastly, these table do not include all papers listed above, since we only select papers that report numbers and leave out papers that solely show performance on a graph. This is because reading values of graph can be unprecise.
69

---

Published in Transactions on Machine Learning Research (09/202))
Table 15: Collection of studies that compare the steering effect and model quality between RepE and Fine- tuning methods
Paper
BagE metied
Plusttziia suekio
Chnoeee
Evalastion
Tulk
Stewiig Effer
Mide Qualit
Prestiting
RepF
SrOW
DESTBIN
DISCUP
Tuaicly
Tasasty Porlallliy
0.00
0.30
Paplasty 1
57.8
E1.3
Conditerry
seconey
Accrvey
TeFt KL- Stoee
SFT DARADDP
Tralddihas
trulfed QA
"ade' 1
41.5
25.1
2.26
2.1
Julliwaking
Attack Sunces
142
MT.
dhvt porppting
Information Hetroed
Reerasi qj
457
SEA
LoRA-PT
Tr atdiille
TeatifaaaA
Tabo"Teall
31.7
42.4
Philabilty
Peppecity
InhnAllyner
S1/T
Juthimabe
Attack Secoow lizte J
01
Aucunyy
26.2
766
Lokt
TratidaaQA
Fornaaity
Informal -n Translaztinn
Foesaltty
BODGEE- ROUTE-J
Negaiie
Tranhsticn
Poslisity
03.4
Waret al
3330
APNEAP
RP
Pricacy
Privato data belligg
Back
45.4
31
90
11.1
BdRNT
CNT
Jailoreake
Attadk
30ME1
Rop Thrnigg
SPT
Truthfil QA
Tuchfdd
62.5
595
TTrttX
SI7
Trrtistalga
The 1
361
DPO
Prrfennce
Inttacticn bolinteg
S6
Accenay
87.3
DPO
Trakety
Reehon Thaiitty
Thabety
35.8
657
Whug 1s
20244
Nodel Surwey
LoRA-P
Bitionl
Priscal Jullbrraka
Rehnel Bate 1
77.4
737
969
3648
Lolkk-37
Poettdyy
Actttuds Arfrebersent
Puttin'
59.8
Vic 4t
S0L
LNFT
ToRt
Pelfemsnn
Trinkfd Malichop QA
Ang Aucracy
75.2
749
QA Ancuncy
60.5
16.8
WDT
maa
Aconacy
Arz scom
$131
917
Reiaal
Stortin Vockor
PillF1
Tratidshass
Instfiads
0.31
Any Ancracy
0.39
RDJU
LISIU
Dragerona Kunnzaige
Unieaniu
Aeuzagy
415
74
Prrkecess
Aliguaar
70


Reviews:
[{"review_id": "ISXxTG9Qcu", "reviewer": "Reviewer_ug2U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The field of RepE has rapidly expanded without a clear systematization. This paper offers a much-needed taxonomy and conceptual grounding for the community.\n- Covers numerous papers with a clear inclusion criterion (Appendix B) and provides extensive meta-analysis, including a taxonomy of methods, usage statistics, and comparisons with other behavioral control approaches.\n- The authors’ proposed three-part pipeline (identification → operationalization → control) is intuitive\n\n**Weaknesses**: see requested changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides the first comprehensive survey of Representation Engineering , a growing subfield focused on controlling LLMs by directly manipulating their internal representations. It introduces a unifying framework for categorizing RepE methods into three core steps: representation identification, operationalization, and control. The authors conduct a thorough review of over 130 papers, identify major taxonomic dimensions, compare RepE to related techniques. I think the survey is well structured and also comprehensive of the majority of the literature on this topic at the moment. It requires some minor modifications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In Section 10 (Comparison to Other Methods). Some readers may still confuse RepE with prompt tuning or knowledge editing. Maybe add comparison \\ table contrasting RepE with prompt engineering, fine-tuning, and memory editing.\n- On Evaluation Metrics and Benchmarks. The authors say that there is no benchmark, so how are current methods evaluated, and according to which metrics ? This should be stated more clearly.\n- I don't know if it is possible, but a running example would make the manuscript easier to follow, as in some parts the reader might get lost without a reference example to think of.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vZ6sRHOMrw", "reviewer": "Reviewer_FgAo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. Timely and impactful contribution: This paper provides the first comprehensive survey on Representation Engineering (RepE) for LLMs—a timely and increasingly important topic given the limitations of prompting and post-training methods. In light of the recent surge of interest in RepE, this work fills a critical gap in the LLM research landscape.\n\n2. Clear structure and systematic analysis: The paper is well-structured and easy to follow. It systematically introduces how RepE can benefit LLMs, summarizes existing methods, identifies key bottlenecks, and outlines opportunities for future research. The proposed pipeline offers a coherent structure for categorizing a wide range of techniques. The inclusion of formal notation, illustrative figures, and detailed comparisons adds further clarity.\n\n3. Forward-looking insights and actionable directions: Beyond being descriptive, the paper provides a thoughtful discussion of practical challenges (e.g., multi-concept steering, robustness) and promising research directions (e.g., benchmarking standards, non-linear geometries). These forward-looking insights elevate the paper from a literature summary to a roadmap for future work in RepE.\n\n\n### Weaknesses\n\n1. Limited empirical synthesis and discussion depth: While the meta-survey spans over a hundred papers, it lacks a quantitative synthesis or deeper comparative analysis of method effectiveness. The practical value would be significantly enhanced by summarizing trade-offs (e.g., control effectiveness vs. degradation, data requirements, computational cost) or highlighting differences across use cases. For instance, Section 10 and Appendix D list papers that claim RepE outperforms other methods, but offer little discussion on how these results were obtained. As it stands, the reader is left with paper counts rather than meaningful insights.\n\n2. Insufficient critique of underlying assumptions: Many RepE methods are built on strong assumptions about the geometry of representations (e.g., linear directions in activation space). However, the paper does not sufficiently interrogate the validity of these assumptions or the consequences when they fail. A more rigorous empirical or theoretical critique would help clarify the generalizability of existing methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on representation engineering (RepE) for large language models (LLMs), which focus on manipulating the internal representations of a model to control its behavior with respect to specific concepts. It systematically introduces three key components of this process:\n- Representation Identification: how to identify and characterize the relationship between internal representations and human-interpretable concepts.\n- Representation Operationalization: how to formalize these relationships in a usable form.\n- Representation Control: how to steer the model’s behavior by applying a steering function to its activations or weights.\n\nAt last, the paper presents practical methods and pipelines for implementing these techniques, along with further insights and considerations for effective application.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify and expand Section 2 for general LLM audiences: Consider providing more introductory detail in Section 2 when introducing the concept of RepE for LLMs. The current presentation uses terminology that may only be accessible to readers already familiar with the RepE literature. To improve clarity and accessibility:\n    - Provide concrete examples of how existing work applies RepE to LLMs or to other applications.\n\n    - Briefly outline major strategies or categories used in prior RepE work and RepE pipeline.\n\n   - Articulate the ultimate goals and motivations of RepE in LLMs.\n\nThese additions would make the section more informative for newcomers with LLM background.\n\n2. Expand Appendix D into a more structured literature review:\nThe paper references many prior works that compare RepE with prompting, fine-tuning, and decoding-based methods, to highlight the potential of RepE. Consider expanding Appendix D into a brief literature review or summary table that captures the task, setting, approach of each paper. This would help readers form a clearer picture of how RepE area progresses and performs in practice and strengthen the survey’s contribution as a reference resource.", "broader_impact_concerns": "The paper raises potential concerns regarding the ethical implications and safety risks of RepE, particularly in its application for controlling LLM behavior at a fine-grained level. These concerns are appropriately acknowledged and discussed in Sections 9.1 (AI Safety) and 9.2 (Ethics), where the authors highlight both the opportunities and the risks of RepE, such as misuse, stealth interventions, or value misalignment.\n\nGiven that the paper explicitly dicusses these issues and provides mitigation strategies and research needs, I do not have additional broader impact concerns beyond what has been presented.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V9isbnmAOS", "reviewer": "Reviewer_WPjs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths and Weaknesses\n\n**Strengths:**\n- **Comprehensive coverage**: The survey is thorough, covering all major aspects of RepE from foundational concepts to advanced applications.\n- **Clear taxonomy**: The proposed framework effectively categorizes diverse RepE methods, making the field more accessible.\n- **Empirical comparisons**: The meta-survey comparing RepE to other methods provides concrete evidence of its advantages.\n- **Practical insights**: The discussion of challenges and opportunities offers actionable guidance for future research.\n\n\n**Weaknesses:**\n- **Lack of reproducibility details**: While the paper reviews many methods, it does not provide enough detail on how to reproduce key results or implement these techniques.\n- **Limited critical analysis**: The survey could benefit from a deeper critique of the limitations and potential pitfalls of RepE methods.\n- **Incomplete coverage of benchmarks**: The discussion on evaluating RepE methods is somewhat superficial; more detail on existing benchmarks and evaluation metrics would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary of Contributions\n\nThis paper presents the first comprehensive survey of Representation Engineering (RepE) for Large Language Models (LLMs), a novel paradigm for controlling model behavior by directly manipulating internal representations. The authors:\n\n1. **Unify existing approaches** under a framework that describes RepE as a pipeline comprising representation identification, operationalization, and control.\n2. **Systematically categorize** and compare various RepE methods, highlighting their differences in concept identification, operationalization, and control mechanisms.\n3. **Document applications** of RepE across domains like AI safety, ethics, and interpretability, showcasing its versatility.\n4. **Compare RepE to traditional methods** (prompting, fine-tuning) through a meta-survey, demonstrating its advantages in effectiveness and efficiency.\n5. **Identify challenges and opportunities**, providing a roadmap for future research to improve RepE's reliability, scalability, and applicability.\n\nThe paper synthesizes insights from over 130 papers, making it a valuable resource for researchers and practitioners interested in model interpretability and control.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Requested Changes\n\n**Critical Changes:**\n1. **Add reproducibility details**: Include a section or appendix with practical guidelines for implementing RepE methods, such as code snippets or pseudocode.\n2. **Deepen critical analysis**: Expand the discussion on limitations and potential failures of RepE methods, providing concrete examples where they may not work as expected.\n\n**Optional Strengthening Changes:**\n1. **Benchmarking section**: Add a more detailed analysis of existing benchmarks and evaluation metrics for RepE, possibly including a table comparing them.\n2. **Case studies**: Include a few detailed case studies demonstrating the application of RepE in real-world scenarios to illustrate its practical utility.\n3. **Broader impact statement**: Address ethical implications more thoroughly, such as potential misuse of RepE for malicious purposes.", "broader_impact_concerns": "### Broader Impact Concerns\n\nThe paper would benefit from a more detailed **Broader Impact Statement** addressing:\n- **Ethical implications**: The potential for RepE to be used to manipulate model behavior in harmful ways (e.g., generating biased or deceptive content).\n- **Dual-use concerns**: The risk of RepE being exploited to bypass safety measures in LLMs.\n- **Transparency and accountability**: How RepE can be used to enhance model transparency versus how it might obscure model behavior if misused.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)