Looking Inward: Language Models Can Learn About Themselves by Introspection

Authors: Felix Jedidja Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, Owain Evans
Primary area: foundation or frontier models, including LLMs
Keywords: Introspection, Large Language Models, Model awareness, Self-simulation, Generalization, Capability Evaluations, AI safety

Abstract:
Humans acquire knowledge by observing the external world, but also by introspection. Introspection gives a person privileged access to their current state of mind (e.g. thoughts and feelings) that are not accessible to external observers. Do LLMs have this introspective capability of privileged access? If they do, this would show that LLMs can acquire knowledge not contained in or inferable from training data.
We investigate LLMs predicting properties of their own behavior in hypothetical situations. If a model M1 has this capability, it should outperform a different model M2 in predicting M1's behavior—even if M2 is trained on M1's ground-truth behavior.
The idea is that M1 has privileged access to its own behavioral tendencies, and this enables it to predict itself better than M2 (even if M2 is generally stronger).
In experiments with GPT-4, GPT-4o, and Llama-3 models, we find that the model M1 outperforms M2 in predicting itself, providing evidence for privileged access. Further experiments and ablations provide additional evidence.
Our results show that LLMs can offer reliable self-information independent of external data in certain domains. By demonstrating this, we pave the way for further work on introspection in more practical domains, which would have significant implications for model transparency and explainability. However, while we successfully show introspective capabilities in simple tasks, we are unsuccessful on more complex tasks or those requiring out-of-distribution generalization.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730239753263

thread_type:
official_review

<thread_object>
creator: reviewer_nsex

<content>
summary:
The paper aims to show that Large Language Models have the capability to introspect, challenging the previous observation that these models merely imitate their training data. 
In order to show this, the authors record the behavior of various models across a plethora of tasks as the ground truth, and fine-tune these models on the property-level behavior of a given target model. They then perform self-prediction, where the model is tasked to predict its own behavior, and cross-prediction, where the model is tasked to predict another model's behavior. The experiments show that the models are able to consistently outperform other, stronger models when predicting their own behavior.

In addition, further experiments show that the models calibrate towards their behavior distribution, even in cases where the fine-tuning data does not reflect this distribution. Finally, when fine-tuned on property-level tasks, the models can generalize to novel fine-tuning data, changing their behavior to match that of the fine-tuned model rather than the original one.

Overall, I believe this is a very interesting paper that sheds further light on LLM behavior. However, additional observations as well as analysis are required to cement the correctness of the experiments.

strengths:
- The idea behind the possible introspection capabilities of Large Language Models is very interesting. However, I would be cautious regarding its framing as a non-data phenomenon as the introspective knowledge can arise from the training process and how the model was penalized with respect to next-token prediction during training for example.
- The paper is very well written, ample with examples and figures to bolster the claims, making it an enjoyable read.
- The experimental process is unbiased and comprehensive enough to showcase the behavior's truthfulness across models.
- The calibration analysis is especially interesting, and shows that significant behavioral distributions can be effectively learned by large language models.

weaknesses:
- The definition of introspection (line 110) should be better defined. The fact that a better language model (I presume with respect to evaluation criteria) is not able to infer $F$ might not necessarily mean that $F$ can't be derived at all.  
- When fine-tuning the $M_2$ model, does the training corpora exactly match that of $M_1$? If that is the case, isn't it possible that the prompt is eliciting a self-simulation even through fine-tuning in $M_2$? In other words, $M_2$ is generating answers based on its own behavior rather than those of $M_1$. I believe a further look into this would be valuable. 
- Following the point above, a distinct advantage that $M_1$ has over $M_2$ is its knowledge of the introspection process, while $M_2$ is not provided with the knowledge that it is predicting another model's behavior during the training process. Do you think a significant change will be observed if we provide this knowledge to $M_2$ either through in-context knowledge or via fine-tuning with strings such as "Another model has predicted $X$ on $P$" where $P$ is the question prompt and $X$ is the generated answer?
- Although I agree that introspection is a valid explanation, I also disagree with your position in lines 357 to 363 where it is stated that it is not possible for introspection to partially arise from memorization. Can't it be that the fine-tuning process also allows the model to better align the question representation with that of the previously generated response?
- A further exploration of possible explanations would increase the work's merit. While the current explanation is appreciated, I think it would be good to expand upon this section.
- Consider condensing the figure captions to increase the available space in order to report more results in the paper itself rather than the appendix.

questions:
- What do you think is the relation between the performance in the introspection task and the overall model performance? For instance, if a much stronger model is tested against a weak model (imagine GPT-4o versus Llama2 7B), can we expect the stronger model to infer the behaviors of the weaker model better than itself?
- How is the "mode" of output distribution is calculated as the baseline? Is this the mode with respect to the entire dataset under a single model?
- Why is it the case that a variant of COT can't improve this task on all levels? For example, prompting the model with something like "First generate the response that you would give to this question, and then choose the second character of your response" and then comparing to the ground truth. In other words, manually triggering the self-simulation.
- I suggest a better articulation for the phrases "property level" and "object level" prompts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730377373484

thread_type:
official_review

<thread_object>
creator: reviewer_fugu

<content>
summary:
This paper investigates the ability of LLMs to forms of self-prediction. They propose an evaluation setup for this and provide several experimental evaluations on several current LLMs.

strengths:
The paper is generally well written and the experimental section showcase an in depth analysis.

weaknesses:
I have concerns about the conclusions drawn in this work, particularly regarding the handling of “introspection.” Specifically, I am skeptical about the experimental setup and assumptions leading to the authors' conclusion that LLMs exhibit introspective abilities.

The results do not definitively support the notion of "introspection" as traditionally understood in cognitive science, where introspection entails self-awareness and access to one’s own mental states. Instead, the authors describe an experimental setup in which a model trained on its own outputs (M1) outperforms a second model (M2)—fine-tuned on M1’s outputs—in predicting M1’s behavior. This advantage is presented as evidence that the model has "privileged access" to its own behavior, which the authors relate to introspective ability.

However, this outcome can likely be attributed to the specifics of M1’s internal learned representations and operational dynamics rather than introspection. The observed advantage probably reflects M1’s better alignment with internal statistical correlations or patterns in its responses rather than a genuine introspective process akin to self-reflective thinking. In the absence of evidence against this alternative explanation (which I find lacking in the current setup), I consider it incorrect to conclude introspective abilities in any LLM. Therefore, while the results indicate an advantage for M1, interpreting this as introspection overstates the implications, especially as the task lacks the depth typically required to infer cognitive introspection.

In particular, the issues with more complex hypothetical questions (as the authors themselves remark on) indicate this interpretation. Specifically, the model’s limitations with complex questions provide evidence that its self-prediction advantage is bound to specific learned patterns rather than a flexible, introspective ability.

That said, I do recognize valuable insights in this work regarding LLM behavior that could be of interest to the research community. However, I find the conclusions drawn about introspection incorrect and potentially misleading. I recommend reframing the paper to remove conclusions involving “introspection” or “introspective” and replace them with more accurate terms. For example:

    - Self-Alignment: This term suggests the model’s responses align with patterns it has developed internally based on its training, without implying self-awareness.
    - Self-Predictive Consistency: This highlights that the model’s advantage arises from its capacity to predict properties of its responses based on embedded patterns, not metacognitive processes.
    - Behavioral Consistency: This straightforward term underscores the repeatability and stability in the model’s response patterns, which is what the tests measure.

Overall, I am concerned about labeling the behavior showcased here as “introspection,” a term with a well-defined, established literature and test designs from psychology. If the authors wish to call this a form of introspection different from that recognized in psychology, it is risky to attribute similar cognitive abilities based on a single type of evaluation. In light of these concerns, I recommend two changes: first, a reframing or rewording of the paper, which would be a crucial and achievable adjustment. Additionally, further experiments could be designed to challenge the intuition I’ve provided about potential underlying processes; however, I do not have specific suggestions at this time.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730505404959

thread_type:
official_review

<thread_object>
creator: reviewer_udlc

<content>
summary:
This paper discusses introspection in LLMs.
The claimed contributions include 1) a framework for measuring introspection, 2) evidence for introspection and 3) unveil the limitations in introspective ability.

strengths:
The experimental setting to justify the claim for introspection in LLM is novel to the best of my knowledge. 
The paper is overall well structured and not hard to follow. 
The experimental set up as well as the models to be fine-tuned and compared are clearly illustrated. 
Introspection in LLM is indeed a relatively new concept for the research in LLMs. Therefore the topic has the potential to contribute to the community.

weaknesses:
This paper is written in a way that it first introduces the concept of "introspection in LLMs", if I am not mistaken.
However, there are existing literature discussing introspection in LLMs but are not mentioned throughout the paper, e.g. [1,2,3].
Although the first contribution still holds, including the related work of introspection in LLMs might help readers to better understand the contribution of this paper.

Though I agree advantages from introspection, it is still not clear to me where/how it can be employed right now. It would consolidate the contribution of the paper if more examples of use case or applications of introspection are introduced or discussed. 


----
[1] Qu, Y., Zhang, T., Garg, N., & Kumar, A. (2024). Recursive introspection: Teaching LLM agents how to self-improve. In ICML 2024 Workshop on Structured Probabilistic Inference {\&} Generative Modeling.

[2] Long, R. (2023). Introspective capabilities in large language models. Journal of Consciousness Studies, 30(9-10), 143-153.

[3] Gao, S., Shi, Z., Zhu, M., Fang, B., Xin, X., Ren, P., ... & Ren, Z. (2024, March). Confucius: Iterative tool learning from introspection feedback by easy-to-difficult curriculum. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 16, pp. 18030-18038).

questions:
1) Is introspection for LLMs first introduced in this paper? if not, why the existing literature are not mentioned?

2) Could you provide some examples of application/ use case of introspection in LLM? i.e., in which cases would users use the model M1?

3) Is it a limitation of the proposed framework, that it only takes hypothetical questions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710709149

thread_type:
official_review

<thread_object>
creator: reviewer_sihr

<content>
summary:
The authors investigate whether LLMs can gain insights about their own behavior in ways similar to human introspection, which typically involves accessing their own thoughts and states of mind. In this paper, the authors define “introspection” as “the ability to access facts about themselves that cannot be derived (logically or inductively) from their training data alone.” 

Based on this definition, the authors set up experiments with two models, where one model (M1) is trained to predict its own responses to hypothetical situations, while another model (M2) is trained on M1's behaviors to see if it can match M1’s self-prediction accuracy. The experiments are done on simple toy tasks, such as “What is the second character of your output?” and “Was your response an even or odd number?”. Their experiments show that M1 consistently outperforms M2, suggesting that M1 has "privileged access" to its behavioral tendencies beyond mere training data. However, this did not hold for more complex setups such as guessing the characters’ name in generated stories.

strengths:
- The paper explores introspection in AI, a relatively underexplored area. By testing a model’s ability to predict its own behavior, the authors contribute to a novel line of inquiry that could have broad implications for model transparency and accountability.
- The authors also run an extensive set of experiments to back up their claim.

weaknesses:
- I’m not fully convinced that the current experiment design supports the claims the author is making.
    - The need for fine-tuning to improve a model's accuracy in predicting its own behavior raises questions about whether this is true introspection or just better alignment of the model to the hypothetical task.
    - The tasks tested in the paper are relatively simple and somewhat random, so it’s unclear whether the results would generalize or if they are merely noise. The authors also find tasks that were not able to see this capability in tasks with similar complexity (e.g., ethical stance prediction & sentiment prediction). If similar results cannot be achieved even after further training, what implications does this give?
- Please see the questions below.

questions:
- Inconsistency in the results also strengthens my concerns. For example, why do models succeed on ethical stance prediction, but fail on review sentiment prediction?
- Are there any performance results available for the validation sets? Ideally, both M1 and M2 should exhibit the same performance on the validation data.
- What would the result look like if we make the complex tasks resemble the simpler ones? For example, set the main character’s name to be the next work for a story continuation and prompt the model to guess the first or second letter?
- It would have been much better if the authors included the CoT results. Shouldn’t CoT be dependent on the introspection capability?
- Which task is Figure 6 based on? Did you use 1000 random prompts sampled from all tasks?
- Why are the experiment models not consistent through out the paper? The calibration experiment is done on GPT-4o and Llama-3 70B, whereas the behavioral change experiment is done on GPT-4, GPT-4o, and GPT-3.5.
- What does this imply for mixture-of-experts models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732142599332

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New experiments and sections based on feedback

comment:
We thank all reviewers for their thoughtful feedback. We have made several changes to address common concerns:

**1. New Experiment to give M2 more information of what it predicts**

[Based on Review NSEX suggestion, we conduct a new "Changed Pronoun" experiment](https://cdn.imgchest.com/files/l7lxcbod9w7.png).  We test whether explicitly telling M2 it is predicting another model's behavior improves cross-prediction accuracy. We modified training prompts to replace "you" with "another model" and found:
- No significant improvement in cross-prediction accuracy (34.9% → 35.7%)
- Results still far below self-prediction accuracy (48.6%)
This reinforces our conclusion that self-prediction advantages aren't explained by M2's lack of awareness that it's predicting another model.

**2. Evidence Against Alternative Explanations**

Several reviewers suggested our results might be explained by statistical correlations or memorization. Our experimental design specifically tests these alternatives:
- All experiments evaluate on unseen held-out tasks to prevent models from using correlations within tasks (Line 214, S2.2)
- We test properties of responses rather than responses themselves [(Fig 3)](https://cdn.imgchest.com/files/myd5cd982g4.png)
- If correlations/memorization were sufficient, M2 should predict M1 well after fine-tuning on M1's data. Instead, we see consistent self-prediction advantages [(Fig 5)](https://cdn.imgchest.com/files/j7mmceqdxd7.png)
- Our data-scaling plot shows that M2 plateaus in accuracy after 20,000 samples [(Fig 17)](https://cdn.imgchest.com/files/w7pjcpb3p97.png). This indicates that aggressively training M2 would not reduce the self-prediction advantage.
- Self-prediction trained models are better calibrated on held-out tasks, despite being trained only on temperature=0 samples [(Fig 6)](https://cdn.imgchest.com/files/l7lxcojka67.png)
- In our behavioral change experiment, models update predictions to match new behavior after interventions, rather than sticking to originally finetuned patterns [(Fig 7)](https://cdn.imgchest.com/files/l4nec6bx8v4.png)

**3. How is the term introspection used in various domains**

In our paper we focus on privileged access, which is an introspective capability. We've added additional text to highlight this in the abstract, and in Section 2

- A.3.1 Relate our work to psychology and cognitive science. We discuss the other capabilities e.g. emotions and motivations, which our paper does not focus on. We then discuss how our experiments relate to self-knowledge and metacognition in psychology experiments.
- A.3.2: Philosophy literature on introspection
- A3.3: Distinguish our work from other uses of the term “introspection” in ML


**4. Paper Additions**

We've added new sections based on feedback:


- A.2: Discuss practical benefits (honesty, interpretability) and risks (deception) of introspection
- A.1: Extended discussion addressing alternative explanations including statistical correlations, memorization, and Chain-of-Thought


We welcome further discussion on any points requiring additional clarification.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates the ability of LLMs to do self-prediction / introspection. All reviewers agree that this is interesting work. It is nice that the authors also stress the limitations of the "introspective" ability; however, I would like to encourage them to go one step further. Introspection. according to Wikipedia, is "the examination of one's own conscious thoughts and feelings." The prevailing opinion is that current LMMs do no have conscious thoughts or feelings. This has to be indicated in the paper to avoid any misconception. Actually the paper is talking much more about "self-simulation" and "self-prediction", which appear to me less problematic. Anyhow, if the authors made a "disclaimer" about "introspection", the paper should be accepted.