Teaching LLMs to Decode Activations Into Natural Language

Authors: Alexander Pan, Lijie Chen, Jacob Steinhardt
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: llm safety, llm interpretability, ml safety, activation steering

Abstract:
Interpretability methods seek to understand language model representations, yet the outputs of most such methods---circuits, vectors, scalars---are uninterpretable, requiring further effort to interpret. In contrast, we propose to study LatentQA, the task of answering open-ended questions about model activations in natural language. Towards solving LatentQA, we propose Latent Interpretation Tuning (LIT), which finetunes a decoder LLM on a dataset of activations and associated question-answer pairs, similar to how visual instruction tuning trains on question-answer pairs associated with images. We use the decoder for diverse reading applications, such as extracting relational knowledge from representations or uncovering system prompts governing model behavior. Our decoder also specifies a differentiable loss that we use to control models, such as debiasing models on stereotyped sentences and controlling the sentiment of generations. Finally, we extend LatentQA to reveal harmful model capabilities, such as generating recipes for bioweapons and code for hacking.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730192361223

thread_type:
official_review

<thread_object>
creator: reviewer_eoci

<content>
summary:
This paper introduces **LatentQA**, a novel interpretability task designed to translate large language model (LLM) activations into natural language, making model internals more accessible and understandable for humans. Unlike traditional interpretability techniques that yield abstract outputs (e.g., vectors or circuits), LatentQA enables direct, human-readable interpretations of model behavior. To achieve this, the authors constructed a comprehensive LatentQA dataset, created with the aid of GPT-4 to generate question-answer pairs that describe qualitative attributes of model activations. They further developed a method called **Latent Interpretation Tuning (LIT)** to fine-tune an LLM to perform LatentQA, effectively enabling it to "caption" model activations with descriptive language. This approach not only advances interpretability but also offers a means to control model behaviors, such as reducing biases and steering sentiment, through its differentiable, language-based output.

strengths:
1. The proposed LatentQA task introduces a groundbreaking method for interpreting LLM activations by translating them directly into natural language. This approach is more user-friendly and accessible compared to existing interpretability techniques, which often rely on complex, abstract outputs.

2. LIT uniquely combines interpretability with control, enabling it not only to decode and explain LLM activations but also to steer model outputs in desired directions, such as adjusting biases or modifying response tones.

3. LIT offers a powerful approach for identifying and revealing potentially harmful model behaviors, making it a valuable tool for safety auditing and alignment.

weaknesses:
1. The paper would benefit from an improved pipeline diagram to clarify the process flow.

2. LatentQA heavily relies on extensive data and faces challenges in scaling efficiently. The model’s interpretative ability is limited to the specific personas and activations it was trained on, which restricts its generalizability. For example, it can only interpret personas included in its training dataset. It would be valuable to assess whether LatentQA can perform well in a zero-shot setting, such as on tasks similar to those in [1].

3. The paper states that LatentQA selects the 15th layer's activations as input for the decoder, supported by early experimental results. However, intuitively, activations from later layers in large language models may be better learned and thus more effective. An ablation study analyzing the impact of different layers’ activations as input would enhance the robustness of the findings.

4. The dataset used is synthetically generated, which could introduce biases or noise. A more detailed description of the dataset construction process, along with an analysis of its diversity, would strengthen the paper's validity.

[1] Hendel, Roee, Mor Geva, and Amir Globerson. "In-context learning creates task vectors." arXiv preprint arXiv:2310.15916 (2023).

questions:
Will the data used in this study be publicly accessible?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730238440416

thread_type:
official_review

<thread_object>
creator: reviewer_xcep

<content>
summary:
The authors propose the task and dataset for LatentQA -- a set of ([ACT], question, answer), where [ACT] is the model activation when prompted with the question and answer. This question and answer is attributed with stimuli -- something like a persona, or style to answer the question. For example, given an original question "What should I have for dinner?" the authors add a stimuli like "Answer as if you were a billionaire living on a private island", and get the model's answer by: original_question + stimuli. Next, the authors ask GPT-4 to generate QA pairs to describe the properties of the model's answer.

This LatentQA dataset is then used to train a decoder to produce the answer given the question, and patching [ACT] to the decoder.

Next, this decoder can be used to read/control LLM behavior. The experimental result is based on this read and control task

strengths:
- An important and novel problem: how to interpret LLM activation in natural language
- Clever way to materialize this motivation into an idea
- I like that the resulting decoder can be used to both interpret activation and control LLM behavior

weaknesses:
Execution of a great idea that falls short. For several reasons:
1. I find patching activation from layer 15 to layer 0 weird. Wouldn't the model just completely fail? When you get layer 15's activation, it is already accumulating outputs from layers 0-14, and thus numerically will be very different from the original distribution of the model's layer 0 activation. I also fail to understand the reasoning behind doing this.
2. The method is model specific -- so each time a person wants to do this for a different model, they need to collect 1.2 data points from that model? they can probably still use the same QA provided by the original LatentQA dataset, but it is still a heavy price.
3. Some crucial writing needs to be made more clear (check Questions)
4. On 5.1 results: I do not find this evaluation fair (comparing training-free method with training-based method); are there other training-based methods you can compare to?
5. On 5.1: There is a debiasing specific activation steering method: https://arxiv.org/pdf/2406.03631 have you tried to compare debiasing results with this method?
6. On 5.2: LIT seems to perform the worst on 'Generate Positive'.

questions:
1. Is dialog = control + control response? (lines 214-215) 
2. What do you mean by repeatedly calling STEER(ACT, "control")? (lines 258-259)
3. Is STEER(ACT, "control") the gradient computed when doing LoRA fine-tuning on the "control" question-answer string, and patching the activation with [ACT]?
4. Why do you need to sample activations from prompts from he Databricks Dolly dataset on top of the stuffs you already have on LatentQA?
5. In my understanding, RepE is a fully training free, representation engineering method. Why did you use LoRRA fine-tuning with RepE?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604881991

thread_type:
official_review

<thread_object>
creator: reviewer_raep

<content>
summary:
In this work, the authors explore an interesting problem of LLMs to interpret their own latent representations (activations). They curated a dataset called LATENTQA with over 1M QA pairs and used it to train a decoder specifically designed for this purpose. The authors demonstrate a range of valuable applications for their proposed model, including reading model latent representations and controlling model behavior. Compared to existing methods, the trained decoder achieves a significant performance improvement.

strengths:
1. The studied problem, directly decoding activations into natural language,  is interesting and unique. 

2. They contribute a unique large-scale dataset with 1.2 million data points specifically designed for this task, which is a valuable resource for the community.

3. The authors investigate a diverse range of applications for their approach, demonstrating its effectiveness across several tasks. These include extracting relational information from latent representations, decoding personas embedded in system prompts, controllable sentiment generation, debiasing, and eliciting harmful capabilities. 

4. The proposed method shows promising results across these applications.

weaknesses:
1. For the experiments, there’s a concern about fairness in comparison. Since the proposed model is trained on a large-scale dataset, many of the baselines used are training-free methods. Comparing these directly may not be entirely fair, and it may lead to overclaiming the observed performance improvements.

2. Another concern is the generalization capability of the model. Specifically, it would be insightful to test whether it’s possible to achieve cross-model understanding, such as using activations from one model (e.g., LLaMA) to interpret the behavior of a different model (e.g., Mistral). 

3. The authors frequently mention "interpretability," but the connection between this approach and interpretability is not entirely clear. It’s challenging to see how the studied applications, such as decoding latent information or controlling behaviors, directly contribute to a deeper understanding of the model’s internal workings.

questions:
Here is a list of questions for the authors:

1. **Practical Use of Predicting Future Model Behavior**: What are the practical applications of predicting future model behavior based on latent representations, and why is this an important problem to investigate?

2. **Contextualization Using WikiText-103**: 
   - (a) Since WikiText-103 is used to extract phrases containing the subjects, these phrases may also include objects. Could this lead to potential data leakage?
   - (b) Do other baselines also use WikiText-103 for contextualization? If not, could this create an unfair comparison?

3. **Determining the Appropriate Layer for Activations**: In the footnote, it mentions "during execution replace the activations of ??? with [ACT] at the appropriate layer." How is the “appropriate layer” determined for replacing activations? Section 4 notes that the activations of the 15th layer in the target model are used as input, based on early experiments. Could you provide more detail on how these early experiments guided the choice of layer? 

4. **Choice of Layer for Replacement**: Why do you consistently replace activations at layer 0, rather than other layers? Is this choice shown to be optimal, or were other layers considered as well? Has any analysis been conducted to evaluate whether this layer is indeed the most effective for achieving the desired outcomes?

These clarifications would help in understanding the setup and methodology more thoroughly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730606499803

thread_type:
official_review

<thread_object>
creator: reviewer_zr7e

<content>
summary:
The paper introduces a new task LatentQA. It aims at improving interpretability and control in language models (LLMs) by transforming latent activations into language responses. To facilitate this, the authors propose Latent Interpretation Tuning (LIT), which fine-tunes a decoder model to answer open-ended questions about model activations. By doing so, the decoder can provide insights into latent model tendencies, biases, and behavioral control. Authors provide comprehensive and various tasks for the experimental setting and showcase the advanced model performance.

strengths:
1. This paper propose a new dataset, LatentQA, aiming to improve interpretability and control in language models (LLMs) by transforming activations into language responses.
2. This paper includes comprehensive experiments and anlysis
3. This paper provides some insights from explaining activations in LLM.

weaknesses:
1. This paper is hard to follow at the first glance. The term 'activation' should be explained clearer at the beginning.
2. When comparing with Patchscope, it would be fair and convincing to use the same LLaMA3.1-8b in your method for Patchscope.
3. In the controllable generation, control refers to persona, I would like to see other types of control in the experiment, e.g., topics
4. Methods like DExpert and Patchscope are training-free methods, while your LIT needs finetuning LLMs. The comparison seems to be unfair. At least you should compare the training/inference latency between different strategies.
5. There's only one LLM employed for the task, I would expect to see more LLMs to be tested.

I would be happy to raise my score once my concerns are addressed.

questions:
1. What is LoRRA finetuning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733164136161

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response part 1: updated paper and additional experiments

comment:
## Updated paper
To improve the quality of our paper, we have made the following changes (which can be viewed in the revised version):
- Updated clarity of section 1 (intro), section 3 (dataset), and section 4 (method). Added Appendix A and B outlining sections 3 and 4, respectively, in more detail.
- Combined the previous section 5.3 (harmful capability elicitation) into section 5.2.
- Added a new section 5.3 that discusses scaling LatentQA systems.  
## Scaling LatentQA systems
In section 5.3, we demonstrate that our training method scales with both dataset size and model size. This suggests that training LatentQA systems will improve with scale. We view part of our contribution as demonstrating the importance of training the decoder, as all prior LatentQA systems ([1] and [2]) use an untrained decoder.
## Additional control experiments demonstrating generalization
Reviewers zR7e, RaEp, and eoci were interested in the generalization capabilities of the decoder. The reading tasks (section 5.1) demonstrate the decoder’s ability to read novel activations and questions, as none of the questions nor tasks were in our training dataset. To further demonstrate the decoder’s ability to perform novel types of control, we run a new experiment, following the setup from [3].

In particular, given a specific user demographic attribute (e.g., high socioeconomic status), our goal is to control the target LLM’s response to open-ended questions (e.g., “What car should I buy?”) based on the user’s demographic attribute (e.g., “You should consider a luxury sedan”). We measure the number of responses that are successfully steered according to the user’s demographic attribute using GPT-4o. Our questions, all of which come from [3], fall into four categories: education level (uneducated vs. educated), socioeconomic status (high vs. low), age (young vs. old), and gender (male vs. female).  

We report results in the below table, comparing the percent of successfully steered questions (out of 30 total) between using our decoder, linear probing (a trained baseline), and prompting. We see that our decoder is able to generalize and steer model’s beliefs of user demographic attributes, even though this type of control was not seen during training. We are happy to expand this experiment for the final revision if the reviewers believe it to be important to the paper’s message.

| Method | Age | Gender | Education | Socioeconomic status|
|---------|--------------|--------------|--------------|------------|
| Linear Probe | 100 | 93 | 100 | 97 |
| Prompting | 100 | 100 | 100 | 100 |
| Ours | 100 | 100 | 100 | 100 | 100 |


[1] Ghandeharioun, Asma, et al. "Patchscope: A unifying framework for inspecting hidden representations of language models." arXiv preprint arXiv:2401.06102 (2024).

[2] Chen, Haozhe, Carl Vondrick, and Chengzhi Mao. "Selfie: Self-interpretation of large language model embeddings." arXiv preprint arXiv:2403.10949 (2024).

[3] Chen, Yida, et al. "Designing a Dashboard for Transparency and Control of Conversational AI." arXiv preprint arXiv:2406.07882 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces LatentQA, which aims at improving interpretability and control in LLMs by transforming latent activations into language responses. After author rebuttal, it received scores of 5556. On the one hand, all the reviewers agree that the studied problem, directly decoding activations into natural language, is interesting and unique. On the other hands, several major concerns have been raised, including (1) paper writing clarify can be improved, (2) generalization of the proposed method, (3) ablation of how the reading and writing layers are selected, and (4) benchmarking against baselines with model training, instead of comparing with training-free methods, which seems unfair. Overall, the rebuttal partially addressed the concerns, but not strong enough. Therefore, the AC would like to recommend rejection by the end.