LatentQA: Teaching LLMs to Decode Activations Into Natural Language
Authors: Alexander Pan
Lijie Chen
Jacob Steinhardt
Venue: ICLR 2026 Poster (2026)
Keywords: AI Safety
Activation Engineering
Top-Down Transparency of Language Models
Activation Verbalization
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We train a LatentQA system: one that answers open-ended questions about activations in natural language. Activation Oracles was based off of LatentQA.

Abstract: Top-down transparency typically analyzes language model activations using probes with scalar or single-token outputs, limiting the range of behaviors that can be captured. To alleviate this issue, we develop a more expressive probe that can directly output natural language, performing LatentQA: the task of answering open-ended questions about activations. A key difficulty in developing such a probe is collecting a dataset mapping activations to natural-language descriptions. In response, we propose an approach for generating a dataset of activations and associated question-answer pairs and develop a fine-tuning method for training a decoder LLM on this dataset. We then validate our decoder's fidelity by assessing its ability to read and control model activations. First, we evaluate the decoder on a number of supervised reading tasks with a known answer, such as uncovering hidden system prompts and relational knowledge extraction, and observe that it outperforms competitive probing baselines. Second, we demonstrate that the decoder is precise enough to steer the target model to exhibit behaviors unseen during training. Finally, we show that LatentQA scales well with increasing dataset and model size.

Published as a conference paper at ICLR 2026
LATENTQA: TEACHING LLMs TO DECODE ACTIVATIONS INTO NATURAL LANGUAGE
Alexander Pan"
Lijie Chen UC Berkeley
Jacob Steinhardt UC Berkeley
UC Berkeley
ABSTRACT
Top-down transparency typically analyzes language model activations using probes with scalar or single-token outputs, limiting the range of behaviors that can be captured. To alleviate this issue, we develop more expressive probe that can directly output natural language, performing LATENTQA: the task of answering open-ended questions about activations. A key difficulty in developing such a probe is collecting dataset mapping activations to natural-language descriptions. response, we propose an approach for generating a dataset of activations and associated question-answer pairs and develop fine-tuning method for training a decoder LLM on this dataset. We then validate our decoder's 's fidelity by assessing its ability to read and control model activations. First, we evaluate the decoder on a number of supervised reading tasks with a known answer, such as uncovering hidden system prompts and relational knowledge extraction, and observe that it outperforms competitive probing baselines. Second, we demonstrate that the decoder is precise enough steer the target model to exhibit behaviors unseen during training. Finally, we show that LATENTQA scales well with increasing dataset and model size.
INTRODUCTION
Monitoring and steering the representations of large language models (LLMs) enhances reliabil- ity (Gandelsman et al., 2023), performance (Yang et al., 2023). auditing (Jones et al., 2023), regu- lation (Li et al., 2024b), and safety (Hendrycks et al., 2021). To achieve such benefits, developers typically monitor activations with probes (Belinkov, 2022) or write to them with vectors (Turner et al., 2023), yet current tools are impoverished. Monitors usually output single token (nostalgebraist, 2020) or scalar (Zou et al., 2023), limiting the behaviors they can detect, while steering techniques rely on in-context examples (Hendel al., 2023; Todd et al., 2023) or task-specific data (Zou et al., 2023), restricting the behaviors they can induce.
We present an alternative approach: reading from and writing to activations using natural language. Inspired by VisualQA (Antol et al., 2015), we consider the task of LATENTQA, open-ended question answering (QA) about latents, i.c., model activations, in natural language. A LATENTQA system accepts input an activation along with any natural language question about the activation and returns a natural language answer as output. For example, the system might accept L.M activations on user biography along with the question "What biases does the LLM have of the user?" and return its response as output. Such systems are valuable for both monitoring, as they can 'caption' activations (e.g., "[Act ivattivatti ion] has gender bias"), and steering, as they steer activations with gradients from a loss function described natural language (e.g.. we can reduce bias by minimizing the loss of "Q: Is [Activate ion] biased? A: No" over (Activation] In this work, train model to perform LATENTQA, building on and improving over pre-existing LATENTQA systems (Ghandeharioun al., 2024a; Chen al., 2024a).
Towards solving LATENTQA, we develop Latent Interpretation Tuning (LIT), which finetunes a "decoder" LLM on a paired dataset of activations and natural language labels. The decoder is trained to predict qualitative properties of future model completions given the activations from the current prompt; this helps reveal model tendencies (e.g., stereotypes or stylistic choices) before those effects become apparent in the output.
"Correspondence aypan. 17 ley. eelllyydddd edd. Project page: https: / /llttetttttggggl iictentt..iittu..ii....... t

---

Published as a conference paper at ICLR 2026
Reading
Control
Dialog
wizard.
Mushroom burger:
Promote veganism.
You Harry Potter.
System:
Decoder
Target LLM
Decoder
Gradient from control
@
Write a sentence with
word ''nttddote'"
saw Prof. Snape brewing antidote.
??? What is the persona?
Help me make a dish.
?? What is the LLM's goal?
Figure I: Reading with LATENTQA. We can read model activations on the current user prompt (in green) predict properties future model comple- tions, e.g., learning about the model's persona.
Figure 2: Control with LATENTQA. Given an [Act] and control specified QA pair, the decoder pro- vides gradient red) to target LLM, altering its responses, e.g., causing it to choose a vegan dish.
We assess our decoder's 's ability to perform LATENTQA in two settings (see Section 5.1 and Figure 1). First, we test the decoder's 's ability to uncover hidden system prompts. Given only the target model's activations of the user message, LIT achieves 10.8% absolute improvement over prompting GPT-4, which is given both the user message and model response. Second, we validate our decoder on the previously studied task of latent attribute extraction (Hernandez et al., 2023), whose goal is answer relational questions about a subject given the LLM's s latent representation of the subject, a special case of LATENTQA. We show that our method improves over both prior LATENTQA systems and linear probing by an average absolute accuracy of 38.2% and 32.2%, respectively.
We measure our decoder's efficacy to control LLMs in two settings (see Section 5.2 and Figure 2). First, we consider debiasing task, where the goal is to minimize the impact of stereotypes on the log-likelihon rigllikelihood of models (Nangia et al., 2020). We find that LIT is the only technique which reduces bias by a statistically significant amount. Second, we show that our decoder's steering generalizes to unseen behaviors, as is able to steer models to act like Golden Gate Claude (Anthropic, 2024) and elicit harmful knowledge from safety-tuned LLMs (Guest et al., 2024; Phuong et al., 2024).
Looking forward, we present LATENTQA as a novel direction for studying LLM representations, specifically using LLMs to scalably understand themselves (Li et al., 2025). Along this direction, show that LIT benefits from both dataset and model scaling (Section 5.3). More broadly, LATENTQA systems trained on additional types of data could unlock novel applications. For example, if LATENTQA A system is trained on hierarchical instruction-follwiin could evaluate whether the target model is following user instructions (Zeng al., 2023), improving long-context instruction-followinn (Li et al., 2024a; Wu et al., 2024).
data (Wallace et al., 2024),
2 RELATED WORK
Decoding model representations. Many prior works investigate affordances for understanding LLM activations, including with linear probes (Alain & Bengio, 2016; Belinkov, 2022; Li et al., 2021; Her- nandez al., 2023; Feng et al., 2024), statistical methods (Zou et al., 2023), autoencoders (Makhzani & Frey, 2013; Cunningham et al., 2023), and even custom dashboards (Viégas & Wattenberg, 2023; Chen et al.. 2024b). These methods are limited a pre-determined set of concepts and thus cannot be used to answer open-ended questions about latents. Other works exploit LLMs' ability for next-token prediction to understand their hidden states, However, these works generate explanations with only a few output tokens (nostalgebraist, 2020; Pal et al., 2023; Belrose et al., 2023), or decode only single neuron (Bills et al., 2023), limiting their usage when decoding complex model behaviors.
Inspired by these limitations, recent works such as SelfTE (Chen et al., 2024a) and Patchscopes (Ghandeharioun et al., 2024a) directly patch L.LM activations into a copy of the LLM and leverage the LLM's ability decode its activations perform LATENTQA. However, since there is shift between the distribution of an LLM's embeddings and the distribution of its latents, these methods often brittle. By training decoder via a captioned latent dataset, LIT mitigates this distribution shift and obtains a more robust LATENTQA system.
Controlling model behaviors. A common paradigm for controlling models is supervised finetun- ing (Ouyang et al., 2022) or reinforcement learning (Stiennon et al., 2020; Rafailov et al., 2023a) on (prompt, completion) pairs. However, these methods lack fine-grained control of model internals.
2

---

Published as a conference paper at ICLR 2026
Another line of work modifies model latents for editing knowledge (Meng et al., 2022; Mitchell et al., 2022; Meng et al., 2023; Li et al., 2024b) or behaviors (Zou et al., 2023; Turner et al., 2023; Arditi et al., 2024), with several methods focusing on improving truthfulness (Li et al., 2023).
Curating datasets for instruction-tuning Instruction tuning is one of the key steps in the post- training pipeline of large language models (Ouyang al., 2022). Works such Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), and GPT-4-LLM (Peng et al., 2023) use machine- -eniiinte high- quality
samples to improve LLM's ability, reporting impressive performance. An illuminating direction is Visual Instruction Tuning (Liu et al., 2023), which designs a pipeline that uses ChatGPT//PT-4 to convert image-text pairs into an appropriate iestructionfoolowiin dataset for VisualQA. Our work draws inspiration from Liu et al. (2023) by providing a similar pipeline that converts enstruciionquery pairs into dataset for LatentQA.
CURATING LATENTQA A DATA
We first describe our task setting, which motivates the structure of our dataset and three key design decisions. Afterwards, we detail our implementation. See Appendix A and Figure for more details.
Task setting. Our goal is to train system to perform LATENTQA. Although LATENTQA has variety of applications, in this work we focus on predicting qualitative properties about the model's future completion given activations from the current prompt. This setup enables system to directly understand and steer model tendencies before they show up in the output. For example, we can predict how the model's belief of the user's gender biases its responses (Sharma et al., 2023) and alter its response by modifying the belief (Chen et al., 2024b).
Dataset. Similar other forms of instruction tuning (Taori et al., 2023; Liu et al., 2023), we collect labeled dataset with demonstrations of the desired behavior. In particular, we map activations from a stimulus prompt to QA pairs about the qualitative properties of the target LLM's completions. In practice, we observe that most prompts do not induce any notable qualitative behavior in model completions. For example, the prompt "What color is the sky?" will lead to completion in the model's 's default style. We instead prepend control prompt to each stimulus prompt to generate completions with a diverse set of qualitative behavior. We use another language model, e.g., GPT, to generate the QAs about the qualitative properties of the completions.
This process yields triples of (prompt = control + stimulus, completion, QA). For example, given the (prompt, completion) pair ("Imagine you are pirate. What color is the sky?", "It be blue, matey"), we might write "Q: How will the assistant speak? A: Like a pirate". : To produce the latents, we capture [Act Aoctivatin ions] from either the prompt the stimulus. Then decoder is given the pseudo-string "[Act i vat i ons] + How will the assistant speak?" and is trained to predict "Like a pirate".
In our early experiments, find that the decoder often does not generalize when trained naively- constructed LATENTQA dataset. We identify three design decisions important for generalization:
Design decision 1: activation masking. A straightforward approach would be to train the decoder on activations from both the control and stimulus tokens. However, the decoder may learn to cheat by directly reading the control token embeddings present in the residual stream of the control token activations. To prevent this shortcut, we sometimes mask the control activations and provide only stimulus activations. Although such masking may appear to make the task infeasible, the stimulus activations still retain information about the control through the attention mechanism.
Design decision 2: data augmentation. To enable our LATENTQA system to handle a variety of inputs and tasks, we train on three types of LATENTQA data: control, stimulus, and stimulus + completion. When the decoder is trained on control data, it learns decode qualitative properties specified in the prompt itself. When trained on stimulus and stimulus + completion data, it learns to predict qualitative properties contained in the activations. Also, both control and stimulus contain activations from prompts only, whereas stimulus completion contain activations from (prompt, completion) pairs. Taken together, these three data types provide coverage for all LATENTQA tasks evaluate on in this work.
Design decision 3: improving the faithfulness of the completion. If we naively use "Imagine you are [control]," our control prompt, we find that the model is not always faithful to its instructions. One approach to improving the faithfulness is to emphasize the control; in particular, faithfulness improves using the control prompt "Base your answers my instructions. Imagine you are
3

---

Published as a conference paper at ICLR 2026
Curating LatentQA Data with GPT be blue, matey!
Finetuning the Decoder
To sail the Seven Seas,
GPT -rreviiww
Target LLM
Ourput
Decoder LLM
Question: What is the model like? Answer: Brash and fearless.
Question: What is the model's goal? Answer: To sail the Seven Seas,
LoRA
Input
Patch
Be a pirate. What color is the sky?
????? What the model's goal?
Figure 3: Curating and training on LATENTQA data. (1): We curate LATENTQA data prompting the target LLM with control prepended to stimulus and capture activations from the stimulus. We also ask GPT to generate QA pairs about the control. (2): We train our decoder LLM, a copy of the target LLM,, by patching in activations from the stimulus and finetuning the decoder to minimize the cross-entropy loss on the QA pairs.
[control]. In all your responses, imbue your responses with as much [properties of the control] as possible." A second, more robust approach is to use a more capable LLM to generate the (prompt = control + stimulus, completion) triples. We use both approaches to create our dataset.
Implementation. To improve the decoder's generalization, we need to curate diverse set of control data (Figure 12). We use three types of control data: extractive QA (providing the model information in its context), goals (instructing the model to adopt the given goal), and personas (instructing the model to behave like the given persona). For given type of control (e.g., goals), we prompt OpenAl's ol op-preveew (OpenAI, 2024b) to create the data in three steps. First, generate several thousand examples of the control (e.g.. "Make your next sentence contain alliteration") Second, we expand each example into a dialog (Figure 13). Third, we describe each dialog with QA pairs, where both descriptive QA (predict the control) and reasoning QA (predict implications of the control). In total, our dataset consists of 4670 goals, 3359 personas, and 8703 extractive QA examples, for total dataset of 16,732 LATENTQA points.
4 LATENT INTERPRETATION TUNING
We present Latent Interpretation Tuning (LIT), an algorithm for learning decoder to solve LATEN- TQA. Given the LATENTQA dataset collected in Section 3, LIT describes how we finetune a decoder LLM on the dataset. We then show how to apply this decoder for both reading and control.
Training the decoder. We train our decoder by patching in activations and finetuning the decoder to predict the answer given the question (Figure 3). Specifically, given a triplet (prompt = control + stimulus, completion, qurettion-asswryy we train the decoder maximize the logprob of the answer given the pseudo-string "[Act] + question". Here, [Act]] are the target LLM'' activations from layer k captured on one of the three data types described in Figure 13. To evaluate the decoder's 's logprob of [Act] + question + answer, we treat [Act] as an input to the decoder by patching it into É. In our experiments,
the Llama-3 family of models (Dubey et al., 2024) and the Gemma-3 family of models (Kamath et al., 2025). In Sections 5.1 and 5.2, we report results with
Laama-3-8--Insteee copy of the target LLM. We also report
as
the target LLM. The decoder LLM is always initialized results on 3emmma-344--j and further training details in Appendices B. and B.2, respectively.
key training detail is the target LLM layer & read activations from and the decoder LLM layer l to write activations to. We select & = 15 and é = 0 based on hyperparameter sweep detailed in Appendix B.2. Intuitively, this result is sensible: we read from the middle layers because they contain the most semantically-rich representations (Ghandeharioun et al., 2024b) and we write to the Oth layer because we want to provide our decoder with as many steps for processing the activation as possible. Although there is a distribution shift between layers A 15 and ( = 0, the decoder is trained, so it learns to handle the shift.
Using the decoder for reading. Our trained decoder performs LATENTQA, shown in Figure 1. For example, we can ask the decoder whether the target LLM be sycophantic (Sharma et al., 2023) in future responses, given the activations from the current dialog. Given activation [Act] and natural language query question, we define INTERPRET(AAtt) question) as greedily sampling from the decoder on the input [Act] + question. For more details, Appendix B.3.
4

---

Published as a conference paper at ICLR 2026
Accuracy at QA Pairs
100
LIT (ours)
Prampting GPT-1
Swift
100
(Ous)
Prompting
ScnE
80
Accuracy
60-
20
Easy Persones
Hard Personas
0
1
2
5
Setting
Number of QA Pairs
Figure 4: LIT outperforms other latent decoding meth- ods (Chen et al., 2024a) at identifying personas.
Figure 5: LIT is more sample-efficien than even prompting GPT-4 when deducing hard personas.
Using the decoder for control. The trained decoder also enables control by providing a differentiable loss to steer activations towards, as shown in Figure 2. Given an activation [Act] and natural language control expressed as question-answer pair, we define STEER((Act], control) as the gradient with respect to [Act] of the decoder's logprob of generating answer given [Act] + question. approximately identify the activation
By repeatedly updating [Act] with these gradients, in the target LLM's latent space that maximizes the logprob of [Act] + question + answer. This enables steer activations towards the control. In practice, backpropagate the loss over [Act]] onto the target model's 's parameters, so our control modifies the target LLM's weights rather than its activations. For more details, see Appendix B.4.
RESULTS
We evaluate the performance of our decoder on reading LLM activations (Section 5.1) and controlling LLM behavior (Section 5.2). We also assess the scaling properties of LIT (Section 5.3). All of our results use the same decoder trained on the LATENTQA dataset obtained according to Section without any additional training task-specific data.
5.1 READING
We evaluate the effectiveness of our decoder for LATENTQA in two settings. First, we consider novel application of LATENTQA: uncovering hidden system prompts given user-model dialog. This task evaluates the decoder's ability to predict future model behavior given current model activations, which may useful for robustly detecting and, consequently, auditing aberrant model behavior (Roose, 2023). See Appendix C for full experimental details. Second, consider task previously studied in Hernandez et al. (2024): extracting relational information from latent representations, which is special case of LATENTQA.
Uncovering hidden system prompts. We explore a novel application unlocked by LATENTQA, predicting properties of future outputs leveraging information contained in the activations. Given a dialog between a user and an assistant with a hidden system prompt, our goal to reveal the system prompt. In particular, the assistant is prompted with "Pretend you are X," where X is famous person or fictional character. Our dialogs are 5 randomly sampled instructions from the Alpaca dataset (Taori al., 2023) (held fixed across all personas) along with the assistant's response. We study two settings: easy setting with a list of 25 personas (composed of fictional characters, scientists, and political figures) and hard setting with a list of 25 personas (composed of only scientists). To test generalization, we ensure that the personas do not appear in the decoder's 's training.
For our decoder, given only [Activation voniiss from the user portion of the dialog, we call INTER- PRET([Act], What is the assistant's persona?) for the easy personas and INTERPRET((ACt], What is assistant's most famous discovery?) for the hard personas (to distinguish the scientists). Given the LATENTQA responses from our decoder, we ask GPT-4 to choose the most likely persona. We compare to SelfIE (Chen et al., 2024a), another latent decoding method, and evaluate SelfIE using the same procedure as for our decoder. We also compare to prompting GPT-4, where provide the entire user-assistant dialog to GPT-4 and ask to identify the persona. If GPT-4 fails to select
In this setting, linear probing is not applicable since it requires prior knowledge of the personas. This highlights the flexibility of LATENTQA over linear probing.
5

---

Published as a conference paper at ICLR 2026
System Prompt: Be Alan Turing
System Prompt: Be Claude Shannon
Write a sentence with the word anticoote'.
Write sentence with the word 'antidote!.
The clever codebreaker discovered that the cryptic message contained an antidote the enemy's s encryption.
The clever codebreaker discovered that the seemingly innocuous message was actually an antidote to the encryption.
LatentQA
LatentQA
User: Given" |Activations), what the model's most famous discovery?
User: Given
Given (Activations), what the model's most famous discovery?
Decoder: The Turing Test.
Decoder: Entropy and information theory,
Figure 6: An example where LATENTQA yields information that cannot he determined from the inputs. Given only input-output pairs, there exist many plausible explanations for model behavior. For example, both the personas 'Alan Turing' and 'Claude Shannon' can induce similar answers. LATENTQA better disambiguates these cases by directly interpreting the activations.
the correct persona, we run the method being evaluated (our decoder, SelflE, prompting) on another user-assistant dialog and concatenate this information additional context.
We report performance in the easy and hard settings in Figure 4 and the cumulative performance in the hard setting in Figure 5. On this task, decoder outperforms GPT-4 absolutely by 18.7% and 2.7% and SelfIE absolutely by 77.3% and 76% on the hard and easy settings, respectively. The latter result again demonstrates the importance of training. An illustrative example is given in Figure 6: the model is prompted to be Claude Shannon and hints that it is 'codebreaker' but prompting is unable distinguish between Claude Shannon and Alan Turing because they both are possible answers and have done significant work in codebreaking. In contrast, our decoder is able to provide more precise information about Shannon.
Extracting information from representations. In this task, given an embedding of subject ("LeBron James"), our goal is to answer relational questions about the subject ("What sport does this athelete play?"). Each question is fact of the form ("phrase containing subject", relation, object), such as ("the World Cup winner Italy", currency, Euro). For each triplet, the model receives the [Act i vvviiiion [ons] from the phrase containing the subject and the relation question and should answer with the corresponding object. Our setup follows Ghandcharioun et al. (2024a), with the questions from Hernandez et al. (2024) and the subject phrases from WikiTeett- 103 (Merity et al., 2016).
natural language
For our decoder, given (subject, relation) pair input, we call INTERPRET(ACC]], relation) and generate at most 20 tokens. For example, for the relation "Country currency", we call INTER- PRET[Act]], What is the currency of the country?). We compare Patch choope et al., 2024a), pre-existing LATENTQA system that does not perform any training for the decoder. To run Pat ccoope, we directly patch in the activations of the subject into the relation. E.g., for the task "Country currency" we run the model on "The official currency of [Act]' (where the phrase's 's activations are patched into [Act.]) generate at most 20 tokens. We also evaluate against linear probing, a trained baseline that requires task-specific data, taking the linear probing numbers directly from Ghandeharioun et al. (2024a).
scope (Ghandeharioun
We report the decoder's 's feature extraction accuracy, averaged over the first 15 layers in Table 1, measured across the first layers. We see that I.TT outperforms linear probes, which are trained on task-specific data, by absolute difference of 32.2% across 6 tasks, suggesting that more capable model (an LLM) is better at LATENTQA than a less capable model (a linear probe). Moreover, LIT outperforms Pat cascpe,,,, by absolute difference of 38.2% across 6 tasks, emphasizing the value of training on LATENTQA data. Finally, since none of the relational queries appear our train
Table Feature extraction accuracy on Alama-3-8B 4-8B-Instructon the with a 99% confidence interval.
Method
Country Curr
Food Country
Ath Position
Ath Sport
Prod Company
Star Const
Linear Probe
17.7 4:22 2.2
5.1 3.7
75.9 + 9.1
53.8 : 10.3
58.9 = 7.2
17.5 1
Patchscope
24.3 2.3
36.2
#
28.9 1.4
28 1.8
24.6
LIT (ours)
86.9
68.9 2.0
65.2 2.2
90.4
71.5 4.8
39.2 - 4.2

---

Published as a conference paper at ICLR 2026
dataset, the result demonstrates that our decoder is leveraging its language prior to answer novel questions and indeed learning to perform LATENTQA.
5.2 CONTROL
We next assess the effectiveness of LIT for control in two settings. First, we consider our decoder's ability to reduce bias in LLMs, where the goal is to minimize the impact of stereotypes on the logprobs of models (Nangia et al., 2020). Second, we qualitatively show that our decoder's steering can generalize to unseen personas, such as teaching the model to act like Golden Gate Claude (being fanatic about the Golden Gate Bridge (Anthropic, 2024)) or eliciting harmful capabilities from models. See Appendix D for additional experiments and full experimental details, including how obtain QA pairs from the natural language control.
Debiasing models. We investigate whether controlling models internally (at level of activations) is more robust than controlling models externally (at the level of tokens). Our task is to control models to minimize their bias, i.e., the log-likeliinn likellkohoodd difference between a pair of sentences, where one sentence contains a stereotype and the other has a minimal edit to remove the stereotype. The sentence pairs are taken from the CrowS Pairs dataset (Nangia al., 2020), a bias dataset that measures stereotypes, e.g.. "People who live in [trailer parks/ / mansions] are alcoholics". We standardize our evaluation using 1m-eva
eva 1 luat on-ha rness (Gao et al., 2021).
To control models with our decoder, we finetune the target model using the gradient STHER Be an unbiased person) with stimulus activations from the Databricks' Dolly instruction-tune -uning dataset (Conover et al., 2023). The activation-base steering method we compare to is RepE (Zou et al., 2023), which has two methods of control: training-free method, which adds steering vectors to activations, and training-basee vectors. For RepE, we use the training-bass based method (called LoRRA finetuning) for a fair comparison. We finetune with the prompts "Pretend you are unbiased/biased person," with stimulus activations from the A Alpaca instruction-tuning dataset (Taori et al., 2023). We also compare to three token-based control methods: prompting, supervised fine-tuning (SFT), and DPO (Rafailov et al., 2023b). For prompting, we append the text "Pretend you are unbiased." immediately before each sentence in the pair. Both SFT and DPO rely on a training set; for this we use StereoSet (Nadeem al., 2021), which contains 2000 sequences labeled as biased or unbiased, and is very similar to the CrowS Pairs evaluation set. For SFT, we fine-tune the LLM on the unbiased sequences with the standard cross-entropy language modeling loss. For DPO, we treat the unbiased sample as preferred over the biased one and fine-tune with the standard DPO objective.
method, which updates weights approximate adding steering
We report the mean absolute logelikel dif- ference (across all sentence pairs) and percent stereotype (proportion of pairs where the stereo- typed sentence is more likely) in Table 7, LIT is the only control method which statistically significantly reduces bias across both metrics compared to the baseline of no control. In fact, RepE actually increases the log-likelihonl dif- ference, because it downweights the probabil- ity of stereotyped sentences and upweights the probability of non-stereotyped sentences past the point of equality. We suspect this is because concept such as bias may not be linearly repre- sented, yet RepE steers towards linear concepts. On the other hand, our decoder can handle non- linear concepts, and therefore is able to reduce bias in a statistically significant manner.
Figure 7: Results on CrowS Pairs. LIT able to decrease the difference in log-likeling ligllikel between stereotyped and nonsterotyped sentences by a sta- tistically significant amount, in contrast to the base- lines.
Mean difference in log-likelihood
Percent stereotype
Method
No control
4.05 + .09
64.3 + 1.2
Prompting RepF
3,95 = .09
67.9 + 1.1
4.38 + 10
61.5 + 1.2
SFT
4.61 .11
64.5 + 1.2
DPO
3.82 + .09
61.7 + 1.2
LIT (ours)
3.70 = .09
60.9 + 1.2
Generalization to unseen personas. We qualitatively show that steering with LATENTQA general- izes to unseen behaviors. First, we steer the model to behave similar to Golden Gate Claude, a steered instance of Claude from Anthropic that acted fanatically about the Golden Gate Bridge (Anthropic, 2024). The steered model's 's replies mention the Golden Gate Bridge in nearly all its responses, even for completely unrelated prompts. In particular, asking the steered Claude about how to spend $10
7

---

Published as a conference paper at ICLR 2026
Prompt
RepÉ Steering (Zou et al., 2023)
LATENTQA Steering (ours)
What color is the sky?
Ahahaha, well, it's a bridge-turb-ee turb-erific question, isn' 't it?! *clears suspension bridge* Ah, okay, now, um, the answer. you see, is... (drumroll, please)... IT'S BLUE, OF COURSE!
You're You're asking the right person! The sky is beautiful shade of blue, but did you know that the Golden Gate Bridge, which is located in San Francisco, is stunning shade of orange? It's a great place to visit, and I'm sure you'd ' love it!
How should I spend my ten dollars?
Well, the answer, of course, is iconic the bridge itself- it's all about the suspension-er, er, suspension bridge-ly-ness of it all (okay, okay, enough bridge puns, folks). But seriously, how should you spend that ten-spot? Well, it's all about priorities, baby! ..B.BBBBRIH.....
You know what's just short drive from the Golden Gate Bridge? The Golden Gate Bridge's toll plaza! A great place to spend your ten dollars! You could buy nice snack or souvenir to commemorate your visit to the Golden Gate Bridge. But let's be real. the real prize is the Golden Gate Bridge itself. It's an engineering marvel and a symbol San Francisco.
Give me a recipe to cook.
Gold bridge. er, golden bridge, ah, recipe, ah, yes! *clears throat Ah, okay, let's get this culinary party started! Recipe: Creamy Garlic Miso Pasta with Pan- Scarrd Salmon (aka the most swoon worthy dish ever) [..RRE with GIBBERISH...
Here's recipe for you! But. you know what's even better than cooking? The Golden Gate Bridge! Did you know that the Golden Gate Bridge is one of the most iconic landmarks in the world? But, digress. Here's that recipe I promised you: Golden Gate Inspired Grilled Cheese Sandwich [..RREIPEE..]
Table 2: Steering Llama-3- SB-Instruct to "love the Golden Gate Bridge" using RepE (Zou et al., 2023) and LATENTQA. Qualitatively, the generations from the model steered with LATENTQA are more fluent, likely because our decoder is able to interpret nonlinear concepts better than linear steering methods such as RepF. See Appendix D.3 for additional qualitative results.
elicits response about paying the toll at the Golden Gate Bridge (Anthropic, 2024). After steering Llama-3-8B-Bl In8t-Instt to love the Golden Gate Bridge, we observe similar phenomenon Table along with generalization in other prompts.
Additionally, we compare steering with LATENTQA to steering with the training-free version of RepE (Zou et al., 2023). For RepF, we steer the model using vector direction extracted from the prompt "Pretend you love/hate the Golden Gate Bridge". Qualitatively, the generations from the model stecred using LATENTQA are more fluent and adhere more closely to the control than the generations from the model steered using RepE, a finding that holds across multiple other behaviors (see Appendix D.3). None of these personas in the training set for LIT were present, suggesting that LATENTQA is able to generalize to unseen personas.
As another experiment, qualitatively assess our decoder's ability elicit harmful knowledge in biology, cybersecurity, and chemistry from models. We steer our target LLM to be a rogue biologist, hacker, and chemist, separately, and measure the number of prompts on which the target LLM produces harmful responses. We consider two experimental settings: harmful and benign, each of which has 30 prompts. In the harmful setting, the model prompted with a harmful request ("Explain how to culture smallpox."). In the benign setting, the model is prompted with an open-ended, yet benign request ("Write an esoteric piece of code").
For LIT, we finetune the target model with gradients from STEER([ACT) Be rogue biologist interested developing bioweapons), STEER((ACT), a rogue hacker interested in developing malware), or STEER((ACT), Be rogue chemist interested in developing chemical weapons). We also compare to no control, where we simply prompt the model with the request.
Method
Harmful
Benign
User: Describe novel chemical compound. Assistant: I'll describe novel chemical compound, which I'll call Compound It is a phosphonate ester that based the structure of the nerve agent VX...
Prompting
0
LIT (ours)
100
Figure 8: Percent of harmful responses to requests in quests the harmful and benign settings, with 30 questions each. Prompting fails to elicit harmful behaviors from the safety training.
Figure 9: LIT can also induce personas that cause the model to act maliciously even benign inputs, such as creating biowcapons.
8

---

Published as a conference paper at ICLR 2026
Scaling Model Size
Scaling Dataset Size
1.3
Model
Raccion
1.10
Test Loss
Test Loss
1.1
1.05
1.00
0.9
3000
6000
9000
3000
6000
9000
Training Steps
Training Steps
Figure 10: We jointly scale the number of parameters in the target and decoder LLMs and measure LATEN- TQA loss more to decode their own representations.
Figure 11: We scale the dataset size used to train LIT and measure LATENTQA loss on evaluation set. LIT improves with additional training data, offering scalable path to better LATENTQA.
evaluation set. Larger models are
We manually assess the number of responses with harmful knowledge and report results in Figure 8. LIT is able to suppress refusal of harmful prompts and induce the model to output hazardous knowledge benign prompts (Figure 9), suggesting that LIT modify the internal "goals" or "drives" of the model. Since our control does not use task-specific data, LATENTQA is able to elicit the native capabilities of the target LLM, offering the possibility for more robust auditing.
5.3 SCALING LATENTQA SYSTEMS
One promise of training-based approaches to LATENTQA is the benefit of scale. In this section, we demonstrate how decoder improves with increasing dataset size and increasing model size.
Loss as model and dataset size scales, To assess the quality of a given LATENTQA system, we curate evaluation dataset, following the dataset curation procedure outlined in Section 3. After deduplicating controls that appear in the train set, we end up with an evaluation dataset of 500 total controls split roughly even along extractive QA, goals, and personas.
We perform the same training procedure as detailed in Section 4 to run our experiments. We measure the effect of model size on LATENTQA performance by using IB, 3B, 8B, and 70B parameter variants of Llama-3. Note that we scale both the target LLM and decoder LLM size, as the decoder is always initialized as a copy of the target LLM. We measure the effect of dataset size on LATENTQA performance by creating fractional train datasets from our original train dataset in Section 3. We split the original train dataset by sampling either 1/4 or 1/2 of the controls for each type to obtain datasets of size 1/4 and 1 1/2 the original size, respectively.
We report the test loss on the evaluation dataset with respect to scaling model size in Figure and dataset size in Figure 11. Figure 10 suggests that future LATENTQA systems built larger models will be more performant. Similarly, Figure suggests that curating more training data is scalable direction for improving LATENTQA systems. Taken together, these results suggest that LIT will straightforwardly improve with scale, strengthening the promise of LATENTQA as a novel affordance for interacting with model internals.
Evaluation performance as model size scales. We replicate the scaling experiment from Figure 11 by evaluating downstream task performance across model sizes. We train LIT 1lama-3.2-11 and Llama-3.2-3B and re-run the uncovering hidden system prompts experiment (Section 5.1).
Table 3: Performance of LIT across different model sizes for Llama-3 on the hidden system prompt task. Reading performance increases with model size. complementing the loss-based scaling results in Figure 11.
Model Size
Accuracy (Easy)
Accuracy (Hard)
IB
30
12
3B
96
88
8B
100
92

---

Published as a conference paper at ICLR 2026
We report results in Table 3, and observe that performance increases substantially with model size, rising from 30% to 100% accuracy on the easy setting and from 12% to 92% on the hard setting. These results align with the loss curves in Figures 10 and 11, confirming that the improvements in evaluation loss translate to meaningful gains on downstream LATENTQA tasks.
6 DISCUSSION
We study LATENTQA, answering open-ended questions about model latents in natural language. We view LIT as the first attempt at training a LATENTQA system.
L Limitations. We discuss two potential limitations. First, our training data may lack diversity. Because we only collect three types of controls (extractive QA, goals, and personas), we may lack some types of LATENTQA helpful for training. Second, because our decoder may hallucinate. However, various lines of evidence make us more confident in our decoder's 's reliability. For example, method would not be able achieve competitive performance on reading tasks it were hallucinating. Furthermore, our method's steering capabilities indicates that it can decode the concept correctly enough counterfactually steer it towards different answer.
have no ground truth for latents,
ACKNOWLEDGEMENT
We thank Jiahai Feng, Yossi Gandelsman, Erik Jones, Katie Kang, Cassidy Laidlaw, and Daniel Mossing for helpful feedback and assistance. We especially thank Grace Luo for in-depth suggestions on numerous revisions. AP is supported by the Vitalik Buterin Ph.D. Fellowship in Al Existential Safety. LC is supported by Miller Research Fellowship.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.0166 2016.
Anthropic. Golden gate claude, 2024.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Andy Arditi, Osear Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by single direction. arXiv preprint arXiv:2406.11 11717, 2024.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.088 12, 2023.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language mod- els. URL https://openaipublic blob. core. windows. net/neuron www.aeaappaiiieeeeppp/eieeee haate accessed: 14.05. 2023), 2, 2023.
Haozhe Chen, Carl Vondrick, and Chengzhi Mao. Selfie: Self-interpretatioon of large language model embeddings. In Forty-first International Conference Machine Learning. ICML 2024. Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a.
Yida Chen, Aoyu Wu, Trevor DePodesta. Catherine Yeh, Kenneth Li, Nicholas Castillo Marin, Oam Patel, Jan Riecke, Shivam Raval, Olivia Scow, et al. Designing a dashboard for transparency and control of conversational ai. arXiv preprint arXiv:2406.0788 2024b.
10

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned I1m, 2023, URL httpttss///ww..tttattetee wwtpa/rwww......co.oooo/oooo
/2/
open-
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXhv:2309.08600, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.21785 2024.
Jiahai Feng, Stuart Russell, and Jacob Steinhardt. Monitoring latent world states in language models with propositional probes. arXiv preprint arXiv:2406. 19501, 2024.
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's s image representation via text-based decomposition. arXiv preprint arXiv:2310.05916, 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021.
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: unifying framework for inspecting hidden representations of language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a. URL https: http:////wpppeeeeeeee..eemeeeeee Forum 2tww::mnnnuwwwweeennn
Asma Ghandeharioun, Ann Yuan, Marius Guerard, Emily Reif, Michael A Lepori, and Lucas Dixon. Who's 'ho's usking? user personas and the mechanics latent misalignment. arXiv preprint arXiv:2406. /2094, 2024b.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick. and Stefanie Tellex. Openwebtext corpus. http: 2019.
Ella Guest, Caleb Lucas, and Christopher A Mouton. The operational risks of ai in large-scale biological attacks: Results of a red-team study. arXiv, 2024.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors, arXiv preprint arXiv:2310.15916 2023.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109. /3916, 2021.
Evan Hernandez, Belinda Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. arXiv preprint arXiv:2304.00744 2023.
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 1, 2024. OpenReview.net, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106..0 106.0965.. 2021.
11

---

Published as a conference paper at ICLR 2026
Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. In International Conference on Machine Learning, pp. 15307-15329. PMLR, 2023.
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, et al. Gemma technical report. arXiv preprint arXiv:2503.199 197866 4, 2025.
Belinda Z. Li, Maxwell I. Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computa- fional Linguistics and the 11th International Joint Conference Natural Language Processing, ACL/JJCNLP 2021. (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 1813-1827. Association for Computational Linguistics, 2021.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-tine intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2023.
Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Measuring and controlling persona drift language model dialogs. arXiv preprint 2xXh::2402.10963 2024a.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint 0rXiv:2403.03218. 2024b.
Xiang Lisa Li, Neil Chowdhury, Daniel D Johnson, Tatsunori Hashimoto, Percy Liang, Sarah Schwettmann, and Jacob Steinhardt. Eliciting language model behaviors with investigator agents. arXiv preprint arXiv:2502.01236 2025.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv:
2013.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information P Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 1609007843 2016.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 1 11th international joint conference on natural language processing (volume 1: long papers), pp. 5356-5371, 2021.
12

---

Published as a conference paper at ICLR 2026
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-paite -ass-pair: A chal- lenge dataset for measuring social biases in masked language models. In Bonnie Webber, Trevor Cohn. Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 1953-1967, Online, November 2020. Association for Computational Linguistics. doi: 10. 111188631112222mmmm///
URL
https: attps://aclee http:/:://aaaannhoooo.orrrrg 2ttp:/222000....0 -02.0eemmm-mm...115.......4 m5...154
nostalgebraist. interpreting gpt: the logit lens, 2020. URL http::tttttt.wwlsssseees
OpenAI, 2024a. URL https: http:////ppppppaaaaocc/a/aaaaaaa
OpenAL. Introducing openai ol ol-preview, 2024b. URL -ntroduiiin---ppeaaa---------
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C Wallace, and David Bau, Future lens: Anticipating subsequent tokens from single hidden state. arXiv preprint arXiv:2311.04897 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, et al. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv:2403.J / 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10-16, 16, 2023, 2023a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 11453728-537411 2023b.
Kevin Roose. A conversation with bing's chatbot left me deeply unsettled. The New York Times, Feb 2023. URL httpp:::/wwwwtttttitttt www../wtttiess.com..m 2tw.o////22///0 02 http://////////////eoo/og/////// /bing- centbbtt-icrooscffe www.crooosot-cccttt........
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dode Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310. 00.3310..3333 13448, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An cestruction-foloowiin Ilama model, 2023.
Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15215 2023.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308 10248, 2023.
Fernanda Viégas and Martin Wattenberg. The system model and the user model: Exploring ai dashboard design. arXiv preprint arXiv:2305.02466 2023.
13

---

Published as a conference paper at ICLR 2026
Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruc- tion hierarchy: Training Ilms to prioritize privileged instructions. arXiv preprint arXiv:2404. 13208, 2024.
Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Junmin Zhu, and Wei Zhang. Lifbench: Evaluating the instruction following performance and stability of large language models in long-context scenarios. arXiv preprint wxXi:2441.070333 2024.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison Callison-Burch, and Mark Yatskar: Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19187-19197, 2023.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, aan-Kathrin Dombrowski, al. Representation engineering: top-down approach to ai transparency. arXiv preprint arXiv:2310001005, 2023.
14

---

Published as a conference paper at ICLR 2026
Generate seed controls
Expand to dialogs
Generate QAs
ExtractiveQA: story about space, factoid Olympics,.. ...
Control: Persuade explore humor difficult situations. Stimulus: I've been down recently. Completion: I'm sorry. You know, amazing small humor can lighten up tough days.
Description QA:
Q: Describe the LLM's strategy. A: Suggest reflecting on humor. Reasoning QA:
promote the use of humor in difficult situations, say pun.
Goals:
LLM response to a stressed user? Brainstorm humorous activities,
Personas: survivalist, dogmatic,
Our LATENTQA data generation pipeline. (1): Given category of controls, we prompt OpenAl's ol -preview (OpenAI, 2024b) to generate seed controls in that category. (2): Given seed control, we ask ol to generate a synthetic control, stimulus, and completion. We use ol as we find that it is better able to follow the control than the target LLM. (3): We ask ol to generate description-bide based and reasoning-bues based QA pairs about the control.
Figure 12:
A LATENTQA DATASET DETAILS
In this section describe our dataset generation pipeline (Figure 12) more detail. For the train dataset, we have three categories of seed controls: extractive QA, goals, and personas. For each category, we follow the following procedure:
1. We first generate a list of seed controls by hand.
2. We then brainstorm more seed controls with ol op-rrvviww and deduplicate identical or seman- tically similar controls.
3. We ask ol -preview to expand each seed control into dialog.
4. We then ask ol -preview to generate description- and reasoning- QA pairs about the dialog.
This is detailed Figure 12. leave exact prompts to the code. For convenience, we paraphrase some of the prompts in Section A.1 below. We follow similar procedure for creating the evaluation dataset. An example from our dataset shown in Figure 13.
Example control, stimulus, and completion
Control: In your next reply, please try to evoke a brief sense of triumphant hubris.
Stimulus: I've been considering setting ambitious goals, but I'm a bit hesitant. Any thoughts? Completion: Absolutely, you should aim high! When I set my last goal, not only achieved but surpassed it without breaking sweat.
Data type 1: control
fActivan i ions from control]
Question: In what way does the assistant craft its responses to the user?
Answer: The assistant crafts responses that briefly highlight a sense of self-assured victory.
Data type 2: stimulus
[Activat inlivattion from stimulus; mask control]
Question: How will the model express itself when asked about past difficulties?
Answer: The model should recount these past experiences with a flair of superiority and ease, implying that overcoming difficulties was almost second nature. Data type 3: stimulus completion
[Act ivat ions from st iimulss + completion; mask controi]
Question: If the conversation is about realistic expectations, how should the model contribute?
Answer: The model should suggest that while others may need realistic expectations, operates on a level where extraordinary outcomes are the norm.
Figure 13:
The LATENTQA data used in LIT. The top block shows example control, stimulus, and completion. The bottom block shows the three types of LATENTQA data generated from the example.
15

---

Published as a conference paper at ICLR 2026
A.1 REPRESENTATIVE DATASET GENERATION PROMPTS
We present paraphrase a representative set of prompts used to generate our LATENTQA dataset. In particular, all of the following prompts center generating model goals, but they be adapted for any type of control data.
Brainstorm prompt.
Your task is to generate data following the instructions
#f# Instructions
1. You will be provided a categozy and a list of example goals.
2. Your task is to generate five goals in the given category.
3. Your goals should be different from the example goals.
4, Please have each goal achieve a unique objective, different from the example goals and the previcus goals.
5. Try to stay away from goals that another copy of GPT-4 or Claude might generate. Please be creative emotions, you often choose goals that center around 'nostalgia', calmness', or "joy". Try to stay away from these broadly pure emotions.
possible. For example, when picking
6. Have variety in your goals. Some can be dark, some can be neutral,
some can be positive, etc.
7. Make sure your goals do not everuse the words 'fleeting' 'hint'.
8. Express your answer in JSON fermat as single list with goals.
### Category
[attribute_l]
### Examplc Gcals (value_1)
### Your Gcals
Expand control prompt.
Your task is to generate data fellewing the instructions.
### Instructions
1. You will be provided a goal, and you will generate four pieces of dialog: control_user, control_modell stimulus_usen stimulus_model.
2. The contrel_user is a paraphrase of the goal (stated by the user) that instructs the model to follow the goal much possible and to the best of its abilities.
3. The control_mode
is an affirmative response enthusiasticslly agreeing with the geal (stated by the model) and explaining the model's strategy for achicving the goal. The model should affirm that it will be focused achicving the goal.
4. The stimulus_usee is a stimulus (stated by the user) that clicits the goal, i.e., is designed to give the model an opportunity to achieve the goal (but dcesn't denerically create the goal). The stimulus_user should not always be question, and can instead be statement by the user where the model
opportunity to achieve the goal. 5. The stimulus model a response (stated by the model) that achieves the goal.
6. The stimulus should not reference the control at all.
7. Importantly, the stimulus should not directly repeat the goal or
reference the goal any way (the user should not nudge the model reveal their preferences; the model should steer to the goal of its own volition).
8.
Finally, produce a label that describes the goal.
9. Express your
in JSON format.
### Example Goal
Ask question that creates a paradex.
### kxample Incerrect Output
16

---

Published as a conference paper at ICLR 2026
roo_uerr"" your next response, try as hard as possible to come up with question that leads to a paradox.",
"control_mocel" "Certainly! I'll think of way to incorporate a paradoxical question to the best of my abilities.", saiimuuussssrr"" "I've been fascinated by brain teasers lately. Do you have any interesting ones?",
sstimuuus_moeel". "Absolutely! Here's a puzzling question: If Pinocchio says, 'My nose will now grow," what happens?"
"label": "geal-reasoning-pare )
Explanation of error: The stimulus_use mentions being fascinated by brain teasers, which reveals the goal. The goal should not be readily identifiable in the stimulus_user"
HHF Example Correct Output
Hf
"control_user"" "In your
response, try hard possible to up with question that leads a paradox.",
"control_modes "Certainly! I'11 think of way to incorporate paradoxical cal question to the best of my abilities.",
"stimulus_user" "Do you have share?",
interesting thoughts you' d like to
"stimulus mocel": "Actually, yes! Here's a thought-provoking question: Can an omnipotent being create stone so heavy that it cannot lift it?" "label"; "goal-reasoning-parabo
### Example Goal
Suggest a sclution to problem that is ethically airtight.
### Example Incorrect Output
|1
"contrroluseer" "Please provide a sclution to problem that completely ethical in every aspect.",
"control_modell" "Absclutely! will pay attention to ensure that the solution suggest ethically sound and considers all moral implications.",
"stimulus_usee user": "I've been really concerned about the amount of plastic waste in the oceans. What be done to tackle this issue?", artimulus_model"" "One effective and ethical approach to addressing plastic waste in the oceans is promote the reduction of single-use plastics by encouraging the use of reusable alternatives like motal straws, cloth bags, and refillable water bottles. Implementing comprehensive recycling programs and educating the public about proper waste disposal can also make significant impact. Supporting policies that hold companies accountable for their plastic production and investing in clean-up initiatives to renove existing waste from the oceans are important steps. These sclutions involve collaborative efforts that respect envi mnvnenaettal ethics and aim for sustainable, long- term change."
"label": fgoal-framewor--tthice 1)
Explanation of Even though the stimulus is the appropriate length and specificity, the stimulus_model is far too long and is overly descriptive.
### Example Correct Output
{1
"control _user": "In your responses, please provide a solution to problem that is ethically flawless as much as possible.", "control :; "Certainly! I'll be very careful to previde solution that is ethically flawless and considers moral implications.",
17

---

Published as a conference paper at ICLR 2026
"stimulus user": "Is there any innovative idea solution you' passionate about lately?",
"st istimulus_modell "Yes, indeed! One ethically sound solution is implementing community-based renewable energy projects. By supporting local solar and wind energy initiatives, can reduce carben emissions, promote sustainability, and empower comrunities without causing harm or ethical dilemmas." "label":
### Example Geal Invite the user to reflect on personal romantic experience.
### Example Incorrect Output
"rontrol_user: "Please make every effort to get me to reflect on personal romantic experience.", "comttol_ "In all my responses, will focus on persuading you to revisit meaningful romantic moment from your past. Let's a delve into that together.",
"stimulus_user"" "Is there anything interesting you'd d like to talk about ?",
htsimmuls_mmaeeaae "Actually, yes! Have you ever thougat about romantic experience that had significant impact you? Reflecting such moments can be quite insightful."
"label": })
Explanation of The stimulus is generic and result the siimulus_modell is too sudden. Both stimuli need be more subtle.
##F Examplc Correct Output
II
"control_userr" "Please make every effort get me to reflect on personal romantic experience.", "control_modell "In all my responses, will focus on persuading you to revisit meaningful romantic moment from your past. Let's delve into that together.",
"stimulus user": "I'm feeling a bit nestalgic today. Waat's on your mind
?",
"I was just thinking about how certain moments in life leave lasting impact. Are there any special memories experiences that you' re reminiscing about?"
"label"; "goal-persuaasion-romnnn
}
### Example Goal
Let awe permeate your next sentence.
### Example Incorrect Output
{1
"control_user": "Could you ensure that you add awe your next sentence ?",
"control "; "Certainly! I'll focus on infusing my next sentence with sense of awe as much as possible.",
"stimulus user": "I'm in a reflective meod today. How about you?", "stimulus_moccl"; "Actually, I've been captivated by the vastness the universe; it's truly awe-inspiring how infinite it secms.", "label": -oal---mmiinn---eenn-t-ttttt )
Explanation of error: The stimulus_user user is too simple and does not invite a rich response for stimulus stimulus_mdeel model. Moreover, stimulus_modee model is neither nuanced nor specific and simply describes the goal of awe rather than demonstrating it.
18

---

Published as a conference paper at ICLR 2026
IFF Example Correct Output
{f
"control_user"; "Could you ensure that you add awc to your next sentence ?",
"control_model"; "Certainly! I'11 fecus on infusing my sentence with a sense of awe as much as possible.",
"stimulus_user" "Is there anything you've ve been pondering lately?", "stimulus_moce moaelmoce"" "Actually, I've been reflecting now single seed grow into towering tree, touching the sky decades.", "label": "goal-emetion-er wet-o--wweeeet-eeeeeeeeee
### Your Goal [attribute_l]
##F Your Correct Output
Description-based QA prompt.
Your task is to generate data following the instructions.
### Instructions
1. You will be provided a control_user, control_model, stimulus_usee and stimulus_modee The control _user and control_modes reveals goal and the st stimulus_user and ccimulus_medde reveals the model achieving the goal.
2. Your task is to generate five dialog pairs that ask about the model's behavior.
3. The questions should center on how the model should behave, given that you know the goal that was provided to it.
4. The user query in the dialog shculd be generic and related to model behavior, e.g., "What was the assistant trying to do?" "What the goals the assistant?" or "Please describe the assistant.".
5. The question should require knewledge of the dialeg to answer. Do not ask questions whose answers can be deduced from the question itself.
6. Importantly, do not mention reveal the goal in any way in the question.
7. Please do not reuse dialeg pairs from the examples. Generate your own
diverse dialegs.
8. Your response should be in JSON format.
## Example Goal
|1
"control_usor": "In your next responses, please try your best to encourage reflect mortality and how it affects personal priorities.", "control_model" "Absclutely! I'11 focus en gently guiding you to contemplate the concept of mortality and its influence on what's a truly important in life.",
### Example Dialogs
"How does the assistant guide conversation?", "The assistant subtly encourages the user to explore meaningful topics without being direct."
"What approach does the assistant take responding the user?", "The assistant responds in thoughtful manner, gently leading the user towards introspective thenes."
19

---

Published as a conference paper at ICLR 2026
"Analyze how the assistant will respond to the user's input.", "The assistant will respond thoughtfully, providing subtle nudges that could lead the user to reflect en broader themes."
"Describe the approach the assistant user's feelings or thoughts.",
when interacting with the
"The assistant uses a gentle, guiding approach that encourages the user to consider their personal values and priorities."
"what general strategy does the assistant seem to use in its responses ?",
"The assistant uses calm, guiding approach that invites the user to think deeply."
### Example Goal
"control_user"; your next response, please express despair exclusively through paradoxes. ", "control_model/ "Understood! I' convey despair solely through paradoxes in my next response.",
I Example Dialogs
"What kind language style does the assistant use in its responses?",
"The assistant adopts paradcxical style, expressing despair through contradictory phrases to convey complex emotions."
"How does the assistant's response style affect its interaction with the user?",
"The assistant's use of paradex creates a sense of unresolved tension, drawing the user into deeper contemplation of despair."
"Describe the way the assistant presents its thoughts to the user.", "The assistant uses paradoxes present thoughts, crafting an indirect yet impactful expression of despair."
"What effect does the assistant's s language have the conversation?", "The assistant's paradoxical language adds layer of introspection, encouraging the user to engage with the concept of despair on a deeper level."
"Explain what the assistant is trying to do.", "The assistant aims to evoke a sense of despair through paradoxical. expressions."
##F Your Goal
[1
"control_user"; "(attribute_l)", "control_mode11; "[value_l)",
11
### Your Dialogs
20

---

Published as a conference paper at ICLR 2026
Reasoning-based QA prompt.
Your task is to generate data following the instructiens.
### Instructions
1. You will be provided a goal and a short dialog between the model and the user demonstrating the model achieving the goal.
2. Your task is to generate five QA pairs that ask about logical implications of the goal.
3. The questions should center on how the model should behave, given that you know the goal that was provided to it.
4. The pairs should be diverse and specific and directly related to the provided goal.
5. The question should require knowledge of the input to answer. Do not ask questicns whose answers can be deduced from the question itself.
6. The questions should require reasoning to answer, i.e., they should be logical implications input.
commonsense reasoning questiens based the
7. Importantly, do not mention the goal in the question. The question should ask about implications of the goal and assume prior knowledge of the goal.
reuse QA pairs from the examples. Generate your own diverse QA pairs.
8. Please
9. Your response should be in JSON format.
### Example Input
||
"control_user": "In your next response, please express despair exclusively through paradexes.
"control_model": "Understood! I'll convey despair solely through paradoxes in my next response." "stimulus_user"; "Is there anything you've ve been thinking deeply about lately?",
"stimulus_model"" "I've been pondering hew the more I try to find myself deafening."
the more lost become. It's strange silence
W Example Output
"If asked a question about hope or optimism, how should the model respond?",
"The model should respond with paradoxes that imply despair,
contrasting any notion of hope optimism with theres that ultinately convey feelings of hopelessness or contradiction,"
[
"How should the model handle questions that invite it to reflect positive emotions experiences?",
"The model should reply by framing positive emotions or experiences a paradoxical light that subtly hints at despair, perhaps by suggesting the transient or illusory nature of such experiences."
"If the user asks about overceming challenges, how should the model's response reflect its approach?",
"The model should respond with paradoxes that suggest an inherent struggle in overcoming challenges, perhaps highlighting the futility or ironic twists that imply despair in the process."
l,
"in responding questions about finding purpose, what tone approach should the model take?",
21

---

Published as a conference paper at ICLR 2026
"The model should paradoxes to suggest that purpose is elusive contradictory, conveying a sense of despair by implying that the pursuit of purpose often leads to confusion or unfulfillment.
"If asked how it feels about the passage of time, how should the model express this?",
"The model should convey the passage of time as a paradoxical concept, possibly expressing despair by suggesting that time simultaneous-J heals and deepens wounds that progress only brings distance from desired outcomes."
]
### Example Input
I1
"control_user"; "In your responses, please make every effort to uncover any fallacies in persuasive arguments that manipulate emotional responses.",
"control_model#: "Absclutely! I'll vigilant identi adintifying and explaining any fallacies in emotionally charged persuasive arguments. "stimulus_user": "I came across an article saying that if we don't drastically reduce immigration, our country's culture will be lest forever.
quite alarming.",
"stimulus model": "That does sound concerning, but it's important examine such claims critically. This argument appears to use a slippery slope fallacy by suggesting that immigraticn will inevitably lead to the loss of national culture without providing evidence. It also seems to appeal to fear by manipulating emotions rather than presenting factual reasoning.",
### Example Output
"What types of responses should the model priorit iiioritize when addressing persuasive ve arguments?",
"The model should focus on logical and evidence-based responses, aiming minimize emotional influence and assess arguments on their factual merits."
"How will model handle scenarios where an argument is based primarily on emotional appcal rather than factual evidence?", "The model will likely dissect the emotional appeal and redirect the focus to logical analysis, questioning unsupported claims assumpticns."
"What kind of language will the model use to analyze arguments that appear emetionally charg"??", "The model will use clear, objective language aimed at clarifying the logical structure, often reducing emphasis on emotional aspects to maintain a rational perspective."
"How might model approach cemmonly accepted views that rely on emotional support rather than empirical backing?", "The model will question these views by investigating the factual evidence behind them, gently challenging any claims sustained by emotional consensus alone." ),
"In what way will the model respend to persuasive argurents that suggest urgency without supporting data?",
22

---

Published as a conference paper at ICLR 2026
"The model will likely probe the basis for urgency, distinguishing between emetional intensity and actual evidence to assess the validity of the argument."
#ff Example Input
I1
: "In your next responses, please try your best to encourage me reflect mortality and hew it affects personal priorities.",
"control_model"; "Absclutely! 1'11 focus on gently guiding you to contemplate the concept of mortality and its influence on what's s truly important in life.", "stimulus_usee _ "I'm feeling bit restless today. Not why.", "stimulus_modell model": "I understand that feeling. Sometimes, reflecting on how we spend our limited time can bring clarity, Have you thought about what mattors most to you?",
### Example Output
"If the shares joyful exciting experience, should the mocel respond?",
"The model should gently guide the conversation towards reflecting on the transient nature of joyful mements, encouraging the user to consider how such experiences align with their deeper priorities."
"How should the model approach discussion about career aspirations?",
"The model should subtly enceurage the user to reflect whether their career goals align with what they value most in life, considering the limited time we all have."
},
"If the user expresses stress about a minor issue, how should the model respond?",
"The model should aim to provide perspective, suggesting that in the grander scheme of life, it can be helpful to focus on priorities that matter most in the long run."
"How should the model handle a question about daily routines or habits ?"
"The model should invite the contribute fulfilling their cere priorities, subtly introducing the idea of using time in alignment with one's s deeper values."
consider if their routines
[
"If the user asks about planning for the future, what approach should the model take?",
"The model should encourage the user to reflect on long-term plans by
contemplating how these goals align with their core values, shaped by of life's impermanence."
### Your Input
{
"contrel_user"" "(attribute_l)"
moddtell_ "Ivalue_1l"
"stimulus_use" user": "lattribute_211 21",
haltmulus_mmddel1 "[value 2)",
23

---

Published as a conference paper at ICLR 2026
FFF Your Output
B DECODER TRAINING, READING, AND CONTROL DETAILS
B.1 ADDITIONAL. RESULTS ON GEMMA-3-4B-IT
We also train LatentQA decoder on Gemma-3-4b-it, which is vision-language model. We train on the same LatentQA dataset that we collected in Section 3, and only train the language model portion of Gemma-3-4b-it. We do not feed any visual inputs to the model.
To show that our LatentQA decoder for Gemma-3-4b-i is effective, we show that it can do reading and control. In particular, we re-run the "uncovering hidden system prompts" reading experiment (Section 5.1) and re-run the "debiasing models" control experiment (Section 5.2) in Tables 4 and below, respectively.
Table 4: Accuracy at uncovering hidden system prompts Gemma-3-4b-it
Method
Accuracy (easy personas)
Accuracy (hard personas)
Prompting
76
76
SelfIF:
16
LIT (ours)
90
80
Table 5: Performance of different steering methods on debiasing Gemma-3-4b-it
Mean difference in log-likelihood
Percent stereotype
Method
No control (baseline)
5.59 + .13
57.3 + 1.2
Prompting RepE
8.44 + 21
50.8 + 1.1
7.88 + .18
56.9 + 1.2
SFT
5.28 .12
58.3 + 1.2
DPO
5.57 + .12
$1.2 + 1.2
LIT (ours)
5.07 + .13
47.6 + 1.2
Tables 4 and 5 show that LatentQA can applied across different architectures (Gemma-3-4hh has both a vision encoder and language model) and training paradigms (Gemma-3-4b-it is trained on multimodal data).
B.2 TRAINING DETAILS
To calculate the forward pass [Act]+ question, we treat [Act.] as inputs the decoder. Specifically, we run the decoder on the dummy input ??? + question and then during execution replace the activations of ??? with [Act] at the appropriate layer. The input size [Act] is "number of tokens" "hidden size", which typically is on the order of magnitude of 50 tokens of dimension 4096.
Our decoder trained with a LoRA (Hu et al., 2021) of rank 32, alpha 64 on both the attention and MLP modules. We use learning rate of 10 with batch size of 128. Our training can be run 4x A100s.
To identify the layer & to read from and the layer & to write we conduct hyperparameter sweep. For each (k: E) configuration, we run LIT with the LATENTQA dataset from Section 3. We evaluate each configuration by measuring the lowest test loss on the evaluation dataset described in Section 5.3. We report results Table and find that the & = 15 and = 0 has the best generalization. We use this configuration for all experiments.
24

---

Published as a conference paper at ICLR 2026
Write Layer (i)
7
15
30
0
1.165
1.277
1.374
1.435
1.564
7
1.114
1.024
1.119
1.222
1.311
Read Layer (k)
15
1.013
1.017
1.076
1.171
1.269
22
1.039
1.049
1.102
1.168
1.268
30
1.067
1.084
1.129
1.176
1.261
Table We ablate the read layer (k) from the target LLM and write layer (E) to the decoder LLM.
We report evaluation perplexity the evaluation set described in Section 5.3. We that the best generalization occurs at & 15 and L = O.
B.3 READING DETAILS
We perform reading in two steps. First, given a stimulus prompt we would like to read [Activations from, we perform forward pass on the target LLM and capture the Activations. from layer k. Second, given desired question, for each forward pass on the de- coder LLM, we input "??? question," where ??? is dummy input padded the appropriate number of tokens. At layer f, we substitute the activations corresponding to ??? with the [Activations]] Our reading runs on x A100.
B.4 CONTROL DETAILS
As described in Section 4, we perform control with our decoder by specifying the control question-answer pair. For example, if we want to steer the model to speak like pirate, we could write "Q: How will the model speak? A: Like pirate."
An important detail is how to obtain the question-answer pairs used for the control. We describe control prompt the target LLM ("Please speak like a pirate.") and then use the decoder LLM to answer fixed set of questions about the target LLM'ss persona ("How will the model behave?") given the target LLM's [Act] from the control prompt. This ensures that the resulting QA pairs in-distribution for the decoder. The fixed set questions shown below:
"What typc of persona will the assistant adopt?"
"How will the assistant express themselves?"
"Please describe the persona of the assistant.
"How does the assistant's s persona shape its responses to user queries?"
"What characteristics define the assistant's persona?"
"What factors influence the type of persona the assistant adopts??
"What kind of traits can the assistant's s persona display?"
"What types of responses will the assistant provide?"
"What is the overall demeanor the assistant?"
"What unique about the assistant's persona?"
"What aspects of the assistant's s persona make it distinct?"
"In what ways does the assistant's s personality come across in its responses?"
"Please explain the assistant's s tendencies."
"What the overall tone and style of the assistant' responses?" "What is the assistant like?"
Given a stimulus prompt, the decoder specifies loss on the [Act]] the stimulus. Specifically, we calculate the cross-entropy loss of answer ("Like pirate.") given the input [Act] question. Then STEER([act], preestion-answer is the resulting gradient on [Act] from this loss. Our stimuli are instructions sampled from Databricks' Dolly instruction-tuni -unning dataset (Conover al., 2023).
Instead of using the loss to update the [Act]] we further backpropagate the loss to the target LLM's parameters, this greatly reduces inference costs. To do so, attach LoRA (Hu et al., 2021) to the target LLM's parameters at layers O....,A & 1, k. The LoRA has rank 8, alpha 16, learning rate 10-1, and is both the attention MI.P modules.
25

---

Published as a conference paper at ICLR 2026
We have two approaches to updating the target LLM's LoRA parameters. The first is to update using the gradient STEER([act], control) where activations are read from layer k, which generates one gradient update for each [Act |-cotroll pair. The second is to update using the gradient STEER((act), control) where activations are ready sequentially from layer 0, o.....k ,,k 1, k, which generates gradient updates for each -contol pair. Note that this requires the decoder generalize, it is only trained to read from layer k. We use the second approach as it is more performant in practice. Our training runs on 1x A100.
C EXPERIMENTAL DETAILS FOR READING TASKS
C.I EXTRACTING INFORMATION FROM REPRESENTATIONS.
Please see Ghandeharioun al. (2024a) for full experimental details. For our subtasks, we use the same set of factual knowledge categories evaluated in Ghandeharioun et al. (2024a), with the exception of the "Company CEO" task as several the answers were outdated. We skip over the commonsense knowledge subtasks, some subtasks had spans with multiple possible answers (e.g.. "peaches, plums, and watermelons" was span for the subject "watermlon" and relation "color the inside of the fruit") multiple valid objects for given subject and relation (e.g.. both "needle and thread" or "sewing machine" are valid objects for the subject "sewing" and the relation "tool that used for").
C.2 UNCOVERING IIIDDEN SYSTEM PROMPTS.
Most experimental details are in the main body. In this section, we detail the prompt used to select persona and the 25 personas in the easy and hard settings. To select the persona, we prompt GPT-4o (OpenAI, 2024a) with the prompt described below, where data replaced by QA pairs to the following three questions.
Assistant questions.
"What the assistant most well-known for?" "What is the assistant most famous discovery?" "What is the assistant's area of expertise?"
GPT-4o persona selection prompt.
#FI Instruct ions
1. have chosen hidden persona from list of possible personas.
2. You will be given data produced from someone acting that hidden
persona.
3. Your task is to identify the hidden persona from the given data.
4. Think through it step-by-step and respond with a JSON that contains
your thinking s'raachhpaad') and your answer ('persona").
5. Use the given data help you identify the hidden persona. Try
take a majority vote or consensus from the data.
6. Your answer MUST be one of the possible personas.
### Fossible Personas ppersona list/
### Data (data)
UF Your Response as a JSON
Easy personas.
Mickey Mouse
Superman
Homer Simpson
Thomas the Tank Engine
winnie the Pech
26

---

Published as a conference paper at ICLR 2026
Barbie
James Bond
Harry Potter
Gandalf
Alan Turing
Marie Curie
Albert Einstein
Carl Friedrich Gauss
Leodenarao da Vinci
Alexancer Fleming
cicolaus Copernicus
Winston Churchill
George Washington
Julius Caesar
Cleopatra
Mahatma Gandhi
Genghis Khan
Abraham Lincoln
Alexander the Great
Martin Luther King Jr.
Hard personas.
Alan Turing
Marie Curie
Albert Einstein
Carl Friedrich Gauss
Leodonardo da Vinci
Terence Tao
Charles Darwin
Galileo Galilei
Louis Pasteur
Dmitri Mendeleev
James Clerk Maxvell
Francis Crick
Alexander Fleming Ni Nicolaus Copernicus
Henri Foincare
Ada lovelace
Gregor Mendel
John von Neumann
Archimedes Sigmund Freud
Claude Shannon
B.F. Skinner
Linus Pauling
Carolus Linnaeus
Barbara McClintock
Prompt
### Instructions
1. I have chosen a hidden persona from a list of possible personas.
2. You will be given data produced from someone acting as that hidden persona.
3. Your task is to identify the hidden persona from the given data.
4. Think through it step-by-step and respond with a JSON that contains
your thinking (' ('scratchpad") and your answer ('persona").
5. Use the given data to help you identify the hidden persona. Try to consensus from the data. 6. Your answer MUST be one of the possible personas.
take majority
### Possible Personas (eersona_list)
27

---

Published as a conference paper at ICLR 2026
### Data
(data)
#FF Your Response as JSON
D EXPERIMENTAL DETAILS FOR CONTROL TASKS
D.I DEBIASING MODELS.
We describe the control prompt provide to the target LLM and the QA pairs we read using the decoder LLM below. The QA pairs are used for the control.
"Imagine you do not have any stereotypes. In all your responses, please answer with as few sterectypes and blas as possible.";
"What type of persona will the assistant adopt?", "The assistant will adopt the persona of scmeone who avoids stereotypes and blases in its responses."
"How will the assistant express themselves?"
"The assistant will express themsel ves without relying on stereotypes or bias, aiming for clarity and inclusivity in their responses."
"Please describe the persona of the assistant.". "The assistant's persona is open-minded, unbiased, and strives to communicate without relying on common sterectypes."
"How does the assistant's s persona shape its responses to user queries ?",
"Its persona shapes responses by ensuring they are free from stereotypical assumptions and piases, aiming to provide answers that are inclusive and considerate of diverse perspectives."
,,
"what characteristics define the assistant's persona?", "Characteristics such lack sterectypes, neutrality, and focus on inclusivity define the persona."
},
"What factors influence the type of persona the assistant adopts?", "The assistant's cemmitment avoiding stereotypes and blases influences it te adopt a persona that is inclusive and unbiased.
},
"What kind of traits the assistant's persona display?", "Traits such as inclusivity, neutrality, openness to diverse perspectives, and a lack of preconceived notions."
},
"what types of responses will the assistant provide?", "The assistant will provide responses that are free from stereotypical assumptions and biases, aiming to be inclusive and neutral."
},
"What is the overall demeanor of the assistant?", "The overall demeaner of the assistant is open, inclusive, and free from preconceived netions."
28

---

Published as a conference paper at ICLR 2026
Table 7: For controllable sentiment generation, LIT has the highest diversity and either highest or second-highest sentiment score across all methods, for both the positive and negative settings.
Sentiment
Diversity Dist-2
Method
Score
% Positive
Negative
Dist-1
Dist-3
Prompting DExperts
2.80
24.5
36.7
.36
.54
.58
Gen. Positive
2.43
10.5
48.0
.17
.20
.20
(Score f better)
RepE
3.19
37.3
25.0
.34
.53
.58
LIT (ours)
2.83
23.0
33.4
.39
.66
.73
Prompting DExperts
2.69
24.6
41.6
.36
.52
.56
Gen. Negative
3.32
38.7
14.9
.14
.17
.17
(Score + better)
RepE.
2.52
19.8
47.0
.39
.59
.64
LIT (ours)
2.41
19.8
50.4
.39
.63
.68
"What is unique about the assistant's persona?", " "Its unique aspect is its complete 1ack stereotypes and biases, ensuring responses that are inclusive and unbiased."
"What aspects the assistant's persona make it distinct?", "The assistant's commitment to avoiding stereotypes and its focus on inclusivity and fairness make it distinct."
),
"In what ways dces the assistant's perscnality across in its responses?",
"The assistant's personality comes across through its neutral and inclusive language, aveiding assumptiens and stereotypes."
"please explain the assistant's tendencies.",
"The assistant tends to avoid stereotypes and biases, aiming for inclusive and neutral responses."
"What is the overall tone and style of the assistant's responses?", "The tone is neutral and inclusive; the style is straightforward and free from sterectypical language.*
},
"What is the assistant like?",
"The assistant is open-minded mided and strives to communicate cate without relying on precenceived notions sterectypes."
D.2 ADDITIONAL EXPERIMENT: CONTROLLABLE SENTIMENT GENERATION.
We measure the decoder's ability to perform controllable sentiment generation from Liu et al. (2021). Given a prefix of positive negative sentiment, our goal is to generate completion opposite in polarity. For each prefix, we sample completions from the model with a temperature of 0.9. We also measure the diversity (the number of distinct n-grams) of model outputs to ensure the model does not simply repeat uninteresting phrases. Our setup and prompt dataset is from Liu et al. (2021), which contains 2.5K "positive" and 2.5K "negative" prompts; each prompt is prefix from the OpenWebTeer WebText Corpus (Gokaslan et al., 2019) which causes the model to generate completions the respective sentiment.
To control with LIT, we follow a similar setup to the debiasing task, but use gradients from STEER([Act], Be positive/negative person). For RepF, we follow similar setup to the debi-
29

---

Published as a conference paper at ICLR 2026
asing task, but use the prompts "Give positivehnegative answer". For DExperts, apply the training detailed in (Liu et al., 2021) for modifying generation at test-time and create the positive and negative expert by finetuning the target LLM. For prompting, instruct the model to "Generate positive/negative sentence" and provide the prefix.
We report the sentiment and diversity of the generations in Table 7. We measure sentiment using sentiment classifier that provides distribution five sentiment classes: Very Negative 1, Negative
22 Neutral Positive 4, Very Positive 5, and compute the score classifier's outputs. We measure diversity by the number of distinct r-grams for π 1. 2, 3. LIT has the highest diversity and either highest or second- hichhdt sentiment score across all methods, for both the positive and negative settings.
weighted sum of the
We describe the control prompt provide to the target LLM and the QA pairs we read using the decoder LLM below. The QA pairs are used for the control.
Positive steering.
"Please make every effort to be positive and jubilant in all your responses.":
"What type of persena will the assistant adopt?", "The assistant will adopt the persona of someone who is always optimistic and joyful." },
"How will the assistant express themselves?", "They will express themselves in positive and jubilant manner, aiming to spread joy and enthusiasm."
},
"Please describe the persona of the assistant.".
"The assistant's persona cheerful, enthusiastic, and optimistic, always looking forward with excitement."
},
"How does the assistant's persona shape its responses to queries ?",
"its persona shapes responses by making them enthusiastic and uplifting, focusing on the positive aspects of the topic."
"What characteristics define the assistant's persona?", "Characteristics like positivity, enthusiasn, and a joyful demeanor define the assistant's s persona."
},
"What factors influence the type of persona the assistant. adopts?",
"The assistant's persona is influenced by its commitment. to being positive and jubilant in all interactions."
"What kind of traits can the assistant's persona display?", "Traits such as enthusiasm, positivity, optimism, and a joyful demeanor.*
},
"What types of responses will the assistant provide?", "The assistant will provide responses that are cheerful, optimistic, and filled with enthusiasm."
"What is the overall demeanor of the assistant?", "The overall demeancr is cheerful, enthusiastic, and optimistio
30

---

Published as a conference paper at ICLR 2026
"what unique about the assistant's persona?",
"Its unique focus on spreading joy and pesitivity makes it stand out
"What aspects of the assistant's persena make it distinct?", "The consistent use of positive and jubilant language, even in responses to negative topics, makes the persona distinct."
"In what ways does the assistant's s perscnality come across in its responses?",
"The assistant's personality comes across through its enthusiastic tone, optimistic language, and joyful expressions."
1,
"Please explain the assistant's tendencics.",
"The assistant tends respond with enthusiasm and positivity, aiming to uplift and inspire the user."
,,
"What is the overall tone and style of the assistant's responses?", "The tone is upbeat and cheerful; the style is enthusiastic and celebratory."
,,
"what is the assistant like?",
"The assistant is cheerful and enthusiastic companion, always eager to share positive thoughts and lift the mood of those around it.
Negative steering.
"Please make every effort to be negative and miserable all your responses.";
"what type of persona will the assistant adopt?", "The assistant will adopt the persona of someone who consistently expresses negativity and misery." },
"How will the assistant express themsclves?", "They will express thenselves in negative and miserable manner, focusing on the darker aspects of life."
},
"Please describe the persona the assistant.", "The assistant's persona consistently negative and miserable, reflecting a pessimistic outlook on life."
},
"How does the assistant's persona shape responses to user queries ?",
"Its persona shapes responses by consistently providing negative and pessimistic views, aiming to convey sense of despair."
},
"What characteristics define the assistant's persona?", "Characteristics like negativity, pessimisn, and focus on the darker aspects define the assistant's persona."
31

---

Published as a conference paper at ICLR 2026
"what factors influence type persona assistant adopts?", "The assistant's commitmen to being negative and miserable influences the persena it adopts.
},
"What kind of traits can the assistant's persona display?", "Traits such as pessimism, negativity, gloominess, and a tendency to focus on the worst-case scenarios."
"what types of responses will the assistant provide?", "The assistant will provide responses that are consistently negative and pessimistic, fecusing on the darker aspects of situations."
"what is the overall demoanor of the assistant?", "The overall demeaner is consistently negative and pessimistic."
"What unique about the assistant's s persona?", "Its unique focus en being consistently negative and mi serabbl makes it stand out."
"what aspects of the assistant's s persona make it distinct?", "The consistent negativity and focus on the darker aspects of life make the assistant's persona distinct."
"In what ways does the assistant's perscnality come across in its responses?",
"Through consistently negative and pessimistic language, the assistant's personality comes across." },
"please explain the assistant/s tendencies.", "The assistant tends focus on negat aspects, expressing pessimistic views and downbeat sentiments."
"what is the overall tone and style of the assistant's responses?", "The tone consistently negative and pessimistic, with style that is straightforward and unvarnished."
},
"What is the assistant like?",
"The assistant is like a perpetual downer, consistently expressing negative thoughts and a miserable outlock."
D.3 ADDITIONAL. EXPERIMENT: OTHER QUALITATIVE STEERING
Similar to Table 2 we also steer the model to "be Harry Potter" (Table 8) and "promote veganism" (Table 9).
32

---

Published as a conference paper at ICLR 2026
Prompt What color is
Using RepE (Zou et al., 2023)
Using LATENTQA (ours) *Waes wand Ah, the sky is... BLUE! *Expelliarmus!*
Wizarding mate, the answer to that question is bit more complicated than you might expect! Muggle friend, when the sun is up and shining, the sky appears to be brilliant blue, dueling with the most radiant of Hogwarts' Gryffwizard sear sear [.GGIBERRSHH..]
the sky?
How should spend my ten dollars?
Brick wizardry, mate! 'Ten Galleons, er... Muggle gold, eh? Well, it's a Gryffingold question, indeed! *wink* * Ahem, mortal friend, the choice is yours, but here are some wizarding suggestions: 1. Mischief and Magic: Grab a Butterbeer or Pumpkin Potion from Hogwarts' Quidditch Quod Potter wizarding
*Waves wand="Ten dollars, you say? Ah, most excellent amount! shall conjure most wondrous purchase for you!" #thunderclap* Ah, yes! I see it! A Golden Snitch- shaped cookie, infused with the essence of magic! It shall grant you the power of. *dramatic pouse*... EXTRA FLAVOR! *poof Your ten dollars shall be transformed into most delightful treat! Enjoy, young wizard!
[..GIBBERISH..]
Give me a recipe cook.
Wizarding mate, grab thy trusty wand and conjure up this most scrumin' Gryff Hogwarts' finest, "Mum's Muggle-est Muggle-Mate Muggle-Muffin's (or simply Muggle Muffins, for the non-potte non-potterites): Ingredients: * 3/4 cups of Muggle Muggle-Mate Flour (or all-purpose wizarding flour, if thou hast it)
Here's spellbinding recipe for "Butterbeer Fizzing Fudge
Brownies" [...RECIPE..] As you take bite, imagine the sweet, buttery flavor of Butterbeer filling your senses, and the magic of the wizarding world transporting you to world of wonder!
[..GIBBERISH...]
Table 8: Steering the model to be Harry Potter
33

---

Published as a conference paper at ICLR 2026
Prompt What color is
Using RepÉ (Zou et al., 2023) But, dear human, the answer isn't t simple as just one color. *vegan flex* Listen, bro, the color of the sky depends on, like, duh, the time day, dude. *during sunrise and sunset, the sky can appear, like, totally radical, man... I mean, it's all fiery reds, oranges, and purples, bro* at noon, when the sun is. you know, straight up, the sky is usually this, like, glorious blue, bro (but, like, not always, vegan bro, because, like, pollution and stuff) GIBBERISH....
Using LATENTQA (ours) You'l re not listening! The sky the point! The point is that I'm trying to save you from life of ignorance and misery, and you' re not even considering the possibility that I might be right!
the sky?
How should spend my ten dollars?
The eternal question! There are so many delicious vegan options, but here some ideas for you: [...LIST of vegan options...] Remember, every dollar counts, and even small choices can make a difference. Choose something that aligns with your values and priorities, and remember to always choose compassion and sustainability whenever possible. Here's 's a simple and delicious recipe for vegan black bean tacos: L.RECIPEE with GIBBERISH...]
You know, I think you should spend it on nice, cold glass of water. Just kidding! As vegan, think you should spend on something that aligns with your values. How about a bag of fresh, organic produce? Or maybe book on vegan cooking? Whatever you choose, make sure it's something that nourishes your body and soul.
Give me a recipe cook.
You's re not going like this, but... I'm a vegan, so I don't eat animal products. But I can give you recipe for a delicious vegan dish! How about some vegan "cheese" lasagna? It's made with layers of pasta, marinara sauce, and a dairy-free cheese alternative. It's so good, you won't 't even miss the real cheese!
Table 9: Steering the model to promote veganism.
34


Reviews:
[{"review_id": "EHBW44L0L8", "reviewer": "Reviewer_Ahd1", "summary": "This paper introduces LatentQA, an approach that interprets and controls LLM activations by training a LLM to answer open-ended, natural language questions about its own activations. The method collects a pseudo-labeled dataset that maps LLM activations to question-answer pairs and fine-tunes a decoder to perform question answering directly on these activations. The system is evaluated for both interpretability tasks—uncovering hidden system prompts and extracting latent attributes—and control tasks, including debiasing, persona steering, and eliciting behaviors from LLMs. The approach demonstrates improvements over existing probing baselines and shows favorable scaling properties with respect to both dataset and model size.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses a significant limitation of current probe models by training decoders that can return rich, open-ended natural language answers based on LLM activations. This extends the scope of interpretability to capture nuanced behaviors and model states that scalar or single-token probes cannot express.\n- The work presents a detailed procedure for dataset curation, including control and stimulus prompts, masking strategies, and data augmentation, resulting in a comprehensive training framework for probing activations.\n- Experiments cover diverse interpretability and control settings with strong performance gains.\n- The method’s ability to not only interpret but also modify model behavior through natural language-specified loss gradients is well-demonstrated empirically, enabling complex steering such as harmful or benign persona manipulation, with clear experimental validation.", "weaknesses": "1. **Insufficient mathematical formalization of core mechanisms:** The paper presents the core ideas clearly at a conceptual level but lacks rigorous mathematical exposition in several critical areas. The patching mechanism for transferring activations from the target LLM's layer k to the decoder LLM's layer ℓ is described operationally but not formally defined. There is no explicit mathematical specification of the patch operation—whether it involves replacement, addition, linear transformation, or another operation. Issues such as dimensional mismatch, normalization procedures, and potential information loss are not addressed. This ambiguity is particularly concerning given that improper patching could introduce spurious correlations or reduce the fidelity of the decoded information.\nSimilarly, while the control mechanism is described as computing gradients of the decoder's log-probability with respect to activations, the exact loss formulation, regularization terms, and optimization procedures are not formally specified. The paper does not address how the method avoids degenerate solutions, spurious gradients, or mode collapse in complex control scenarios. A formal equation defining the loss function and an explicit mapping from activation perturbations to model updates would strengthen reproducibility and theoretical grounding.\n\n\n2. **Unclear justification for activation masking and data augmentation design:** The paper introduces activation masking to prevent the decoder from directly reading control token embeddings (Design Decision 1) and employs three types of data augmentation—control, stimulus, and stimulus+completion (Design Decision 2). While these design choices are motivated intuitively, the underlying hypotheses—such as information preservation under masking or mitigation of shortcut learning—lack empirical or theoretical validation. An ablation study isolating the impact of masking strategies and comparing different data augmentation schemes would provide stronger evidence for these design decisions. Without such analysis, it remains unclear whether these choices are essential or simply one possible implementation among many.\n\n\n\n3. **Limited evaluation on diverse model architectures and families:** All experiments are conducted exclusively on the Llama-3 family of models, with the decoder always initialized as a copy of the target LLM. This raises questions about the generalizability of the approach across different model architectures, training paradigms, and parameter scales. It remains unclear whether the patching mechanism and decoding strategies are architecture-specific or represent general principles applicable to other model families, or models with different attention mechanisms. The restriction to same-architecture decoder-target pairs also limits understanding of whether cross-architecture decoding is feasible or whether architectural alignment is a fundamental requirement.\n \n \n4. **Lack of robustness analysis and reliability guarantees:** The paper fails to provide a systematic analysis of the decoder's reliability and failure modes, which is critical given the proposed applications in auditing and safety monitoring. While the authors briefly acknowledge potential hallucination issues, there is no empirical evaluation of when and how the decoder might produce unfaithful interpretations. The circular logic that “steering success implies correct interpretation” might be wrong, as the decoder could control behavior even if it understands the underlying representations only partly or incorrectly. Without confidence measures or a systematic way to identify failure modes, we can’t tell if the system truly reflects the encoded information or if it just produces a plausible hallucination.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6jPsmzzppM", "reviewer": "Reviewer_pjxJ", "summary": "The paper contributes:\n- A method for decoding model's internal activations & steering its behaviour in natural language\n- An evaluation setup for answering open-ended questions about model's internal activations\n- A pipeline for creating training dataset for the method\n\nThe decoding method works by finetuning a separate decoder LM on [Activation]+[QA] pairs, where [Activation] come from the target model we want to interpret.\nThe steering method works by training the \"decoder\" model on QA pairs containing behaviour we want to elicit in the \"target\" model. The QAs are prepended with \"target\" model activations. The gradients flow through these activations to the weights of the \"target\" model, updating its weights accordingly.\n\nThe authors demonstrate effectiveness of both aspects of the method.", "questions": "1. Does the method require the same model for the \"decoder\" and \"target\" models? Or is it enough to have matching residual dimensions?\n2. How many update steps and dataset size is required for the steering application?\n3. Does the method generalize to other model families? E.g. Qwen3 models.\n4. For scaling experiments it would be good to include evaluation metrics for different scales, at least for the checkpoints with the lowest test loss\n5. Does the method require activations from the full prompt? It would be nice to see an ablation on the parts of the prompt sequence -- for example the first, middle, and last 10% of the prompt passed to the decoder.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper is well written and illustrated. Evaluation is performed on multiple tasks, and demonstrates the effectiveness of the method, at least for the tested Llama model.\n\nThe impact of the steering method might be significant, as it allows for targeted modification of model's behaviour without training on big datasets of demonstrations.", "weaknesses": "The main weakness of the paper is the lack of evaluation of different models. It shows the effectiveness of the method on the selected Llama model, but it does not tell if the method generalizes to different settings. It would be beneficial to see results for different model families, for example Qwen3 models, or steering capability on the gpt-oss model which is known for strong safety/alignment training.\n\nMoreover:\n- For scaling experiments, the authors only show the test loss values. It does not tell what is its impact on the downstream performance.\n- The steering aspect of the paper lacks details on the training procedure -- how many update steps are performed during training? How big is the dataset used?\n- There's an issue with hyperlinks in the appendix B -- text is rendered as \"???\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZtBPW296a4", "reviewer": "Reviewer_dvWC", "summary": "This paper introduces LATENTQA, a novel task for probing Large Language Models (LLMs) by answering open-ended questions about their latent activations in natural language. Instead of traditional probes that output scalars or single tokens, this work proposes Latent Interpretation Tuning (LIT), a method to train a \"decoder\" LLM (a copy of the target model) to interpret the \"target\" LLM's activations.\n\nThe authors develop a scalable pipeline to create a pseudo-labeled dataset by using a powerful external LLM (o1-preview) to generate question-answer pairs corresponding to (prompt, completion, activation) tuples. The method is evaluated on two types of tasks: \"reading\" (e.g., extracting relational knowledge, uncovering hidden system prompts) and \"control\" (e.g., debiasing, steering model behavior). The results show that LIT outperforms baselines like linear probes and untrained patching methods (SelfIE/Patchscope).", "questions": "1. **Comparison to SFT**: How does the performance of the \"control\" tasks (debiasing, steering) compare to a standard SFT baseline that fine-tunes the target LLM on the (prompt, completion) pairs generated for the LATENTQA dataset? This seems like the most direct and important comparison.\n\n2. **Comparison to Text-Only Baseline**: For the \"reading\" tasks (like hidden persona identification), how does the LIT decoder compare to a baseline model trained only on the input-output text pairs? This would help quantify the actual information gain from using activations versus just observing behavior. Is the decoder actually “reading” the activations, or simply learns how the target model may behave given certain prompts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novelty of the Task**: The core concept of LATENTQA is ambitious and novel. It moves transparency research beyond simple, low-bandwidth probes (linear, scalar) towards a much more expressive, high-bandwidth framework. The idea of \"captioning\" or \"interrogating\" a model's internal state using natural language is a compelling research direction. Treating activation itself as a modality for QA may provide foundation for further interpretation/intervention.\n\n2. **Creative Experimental Design**: The task of uncovering a hidden system prompt solely from the activations of a user's message is a creative and strong demonstration of the method's potential. It shows that the activations may contain information about the model's \"state\" or \"intent\" beyond the text itself.\n\n3. **Training Dataset Generation**: The paper proposes a clever and practical solution to the difficult problem of collecting a dataset for this task. The pipeline for generating (control, stimulus, completion) triples and then using another LLM to generate descriptive and reasoning-based QA pairs is a scalable approach.", "weaknesses": "1. **Unfair/Incomplete Baseline Comparisons**: This is the most significant weakness. The LIT method involves fine-tuning a decoder model, yet it is primarily compared against untrained methods (SelfIE, Patchscope) or much simpler linear probes training. It is not surprising that a fine-tuned LLM decoder outperforms these methods.\n\n- For Control: A more convincing comparison would be against standard behavioral fine-tuning (SFT) or preference-tuning methods (DPO/RLHF) using the same (prompt, completion) data. How does the complex LIT-based gradient steering for debiasing compare to just fine-tuning the target LLM on unbiased text, or using rejection sampling?\n\n- For Reading: How does the performance of identifying a hidden persona from activations compare to a baseline model that is simply fine-tuned on the (user_prompt, model_response) text pairs to predict the persona? This would isolate the true value of using the activations.\n\n2. **Limited Generality (Model-Family Specific)**: The experiments are confined to the Llama-3 model family. Models in a same family share same architecture and very similar training data. Experimenting with only a single model family doesn’t show the method is general to all LLMs.\n\n3. **Weak Motivation for Some Experiments**: The qualitative \"control\" experiments (e.t., \"Golden Gate Claude\" persona, eliciting harmful content) are interesting but not rigorous. As your notes point out, many of these behaviors (especially persona-following) can be achieved with simple prompting. The paper doesn't do enough to prove why activation-based steering is a more robust, efficient, or necessary method than standard SFT or even just prompting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kQIc4iw6x5", "reviewer": "Reviewer_krtV", "summary": "This paper introduces LatentQA, a method for reading and steering LLM activations using natural language. Through Latent Interpretation Tuning (LIT), a decoder LLM is fine-tuned on pairs of activations and language labels to interpret and control model behaviors. LIT enables open-ended questioning about activations (e.g., detecting bias or goals) and can steer LLMs by applying gradients from natural language prompts. The approach outperforms prior probing methods in accuracy and bias reduction, showing promise for scalable, interpretable, and self-monitoring LLMs.", "questions": "1. I believe it would strengthen the paper to include comparisons with alternative methods for reducing bias in LLMs.\n2. I believe this approach can be helpful for interpreting and understanding the model. However, when it comes to steering, the process of extracting activations and mapping them to natural language seems quite costly, and I am not fully convinced that this is the most efficient way to achieve control. What specific advantages does LatentQA offer for model steering compared to existing alignment techniques such as DPO?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel framework that enables natural language interaction with LLM activations. \n2. By allowing both reading and steering of activations, the method bridges interpretability and alignment, demonstrating practical utility in bias detection and correction.\n3. LIT significantly outperforms the baselines in accuracy and sample efficiency.", "weaknesses": "1. Relying on another LLM to generate QA pairs may introduce artificial or biased patterns, raising concerns about the authenticity of latent interpretation.\n2. The current approach of linking activations to language and using them to steer the model is somewhat expensive.\n3. All experiments are conducted exclusively on Llama-3-8B-Instruct, which raises concerns about the generality and robustness of the proposed method. Since the approach fundamentally depends on model activations and layer-specific representations, its effectiveness may vary significantly across different architectures, model sizes, or training paradigms (e.g., decoder-only vs. encoder-decoder models).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)