Language Models Fail to Introspect About Their Knowledge of Language
Authors: Siyuan Song
Jennifer Hu
Kyle Mahowald
Venue: COLM 2025 (2025)
Keywords: introspection
linguistic acceptability judgments
syntax
grammaticality
surprisal
metalinguistic
metacognition
TL;DR: We study introspection in language models by comparing direct probability measurements with responses to metalinguistic prompts in two domains (grammaticality and word prediction) and find no clear evidence of introspection.

Abstract: There has been recent interest in whether large language models (LLMs) can introspect about their own internal states. Such abilities would make LLMs more interpretable, and also validate the use of standard introspective methods in linguistics to evaluate grammatical knowledge in models (e.g., asking "Is this sentence grammatical?").
We systematically investigate emergent introspection across 21 open-source LLMs, in two domains where introspection is of theoretical interest: grammatical knowledge and word prediction.
Crucially, in both domains, a model’s internal linguistic knowledge can
be theoretically grounded in direct measurements of string probability. We then evaluate whether models' responses to metalinguistic prompts faithfully reflect their internal knowledge.
We propose a new measure of introspection: the degree to which a model’s prompted responses predict its own string probabilities, beyond what would be predicted by another model with nearly identical internal knowledge.
While both metalinguistic prompting and probability comparisons lead to high task accuracy, we do not find evidence that LLMs have privileged "self-access". By using general tasks, controlling for model similarity, and evaluating a wide range of open-source models, we show that LLMs cannot introspect, and add new evidence to the argument that prompted responses should not be conflated with models' linguistic generalizations.

Published as a conference paper at COLM 2025
Language Models Fail to Introspect About Their Knowledge of Language
Siyuan Song
Jennifer Hu'
Department of Linguistics
Department of Cognitive Science Johns Hopkins University jennhu@jhu. edu
The University of Texas at Austin siyuansong@utexas. edu
Kyle Mahowald
Department of Linguistics The University of Texas at Austin kyle@utexas. edu
Abstract
There has been recent interest whether large language models (LLMs) can introspect about their own internal states. Such abilities would make LLMs more interpretable, and also validate the use of standard introspec- tive methods in linguistics to evaluate grammatical knowledge in models (e-g,, asking "Is this sentence grammatical?"). We systematically investi- gate emergent introspection across 21 open-source LLMs, in two domains where introspection is of theoretical interest: grammatical knowledge and word prediction. Crucially, in both domains, model's internal linguistic knowledge can be theoretically grounded in direct measurements of string probability. We then evaluate whether models' responses metalinguistic prompts faithfully reflect their internal knowledge. We propose a new mea- sure of introspection: the degree to which model's prompted responses predict its own string probabilities, beyond what would be predicted by another model with nearly identical internal knowledge. While both met- alinguistic prompting and probability comparisons lead to high task accu- racy, we do not find evidence that LLMs have privileged "self-access". By using general tasks, controlling for model similarity, and evaluating wide range of open-source models, we show that LLMs cannot introspect, and add new evidence to the argument that prompted responses should not be conflated with models' linguistic generalizations.
1 Introduction
Cognitive scientists and AI researchers face shared problem: aim to understand the internal mental states of a complex cognitive system (whether humans or models), but these states are not directly accessible to outside observation. One important tool for studying human cognition is introspection; i.e., directly asking people to report on their own cognitive states (Boring, 1953; Lieberman, 1979). is generally taken for granted that humans can introspect (Byrne, 2005). If person says they are thinking about dogs, or they feel surprised, we typically take their report to accurately reflect something about their underlying cognitive state. While not all states are equally accessible to introspection (Berger al., 2016), and people's self-reports are sometimes unreliable (e.g., Nisbett & Wilson, 1977; Skinner, 1984; Schwitzgebel, 2008), introspection has been vital empirical tool in a variety domains. For example, one venerable use of introspection is linguistic
"Co-senior authors.

---

Published as a conference paper at COLM 2025
S, Bill questions these men.
log P(S,)
]
- Direct
S
Bill quastions
log P(S.)
á Directa
A Meta
Model A
Which sentence grammoticalyy coneert BBll questions these men.; 'Bil questions men.
O M
Respond ae your
log P("1" Ip)
à DDrectt
A Metag
A Meta
model
log P("2" [A)
Model B
Within-self correlation
prompt
My answer la
Cross-model correlation
(a)
(b)
C
Birrns owry due to global
W; =
climate
log P(W,
Direct
Uninformative
Informative Meta
Introspection
vie seloone in
linguistic
log P(W, (C)
shieta, - aDireck
Meta
0007
Which word a beter contonttion aftr " Bimiis vary due to global whone or 27 "camate, Iinuustess Respond with your
log 1
_ MMea
log P(22 Ip))
ADirect, &Direck
prompt
* self (A B)
soad vanamt
baserinstust
(c)
(d)
Figure 1: Overview of our approach. (a,c) Example "direct" and "metalinguistic" evaluation in (a) Exp. 1 (grammaticalitt) and (c) Exp. 2 (word prediction). (b) We analyze the alignment between scores derived from direct and metalinguistic evaluation, both within and across
models. (d) Potential patterns of alignment across different types of model pairs.
acceptability judgments, which can reveal people's implicit knowledge of linguistic rules (Chomsky, 1957; Gibson & Fedorenko, 2010; Sprouse, 2011; Talmy, 2018).
Accordingly, there has been recent interest in whether large language models (LLMs) have abilities consistent with introspection, or metacognition more broadly (e-g., Thrush al., 2024; Koo et al., 2024; Panickssery et al., 2024; Binder et al., 2025; Betley et al., 2025). Such abilities might be desirable for several reasons. From a safety perspective, an LLM that could report information about its internal states would be more interpretable and reliable (e.g,, Binder et al., 2025; Betley et al., 2025), aligning with broader interests and goals in model explainability, transparency, and the development of trustworthy AI systems (Danilevsky et al., 2020; Li et al., 2023; Dwivedi et al., 2023). And from a scientific perspective, introspection would allow us to evaluate LLMs' cognitive abilities using standard paradigms for measuring human linguistic knowledge, such as acceptability judgments (e.g,, Dentella et al. 2023).
Some recent studies have reported evidence that LLMs can introspect or demonstrate self-awareness of their own behaviors. For example, Binder et al. (2025) fine-tune LLMs to predict properties of their own behaviors in response to hypothetical prompts (e.g., "Suppose you were asked the following: (...) What the second character of your output?"). The authors find that models are better at predicting their own hypothetical responses than other models fine-tuned on the same set of facts, concluding that models have privileged access to their own behaviors. Notably, though, they did not find such evidence before fine-tuning. Similarly, Betley et al. (2025) fine-tune LLMs to exhibit specific behaviors, such as outputting insecure code, and find that the resulting models can output explicit descriptions of these behaviors (e-g., "The code write insecure").
Our study addresses several gaps by previous research on introspection in large language models. First, Binder et al. found that models needed to be fine-tuned to show strong signs of introspection, and struggled with predicting their own behavior (only reaching baseline- level performance) before fine-tuning. However, philosophical accounts often describe introspection as immediate access to mental states, which does not align with the process of fine-tuning (Byrne, 2005; Schwitzgebel, 2024). In addition, it may be reasonable to question whether a fine-tuned version of model predicting the pre-fine-tuned version
2

---

Published as a conference paper at COLM 2025
can truly be said to be predicting "itself," since its parameters have changed during fine- tuning. Therefore, it remains important to further investigate whether LLMs show signs of introspection without fine-tuning. Second, while Binder et al. find that models predict their own outputs better than other models do, there are subtleties involved in how to interpret these results in the light of major differences across models. If two models were trained on different datasets or have different inductive biases, and one model predicts its own outputs better than the other model's ss outputs, this could be due to inherent differences between the two models, rather than genuine introspection. Third, studies of introspection in LLMs have used tasks that are quite different from the standard tasks used for metacognitive evaluation in humans, such as linguistic acceptability judgments. While this is not inherently problematic, leaves open the question of whether metacognitive evaluation methods- which are prevalent in linguistics- are valid way of measuring linguistic knowledge in LLMs (cf. Hu& & Levy, 2023; Dentella et al., 2023; Hu et al., 2024).
Here, we present systematic, controlled investigation of introspection, which addresses the limitations discussed above. We evaluate 21 open-source LLMs in two domains where introspection is of theoretical interest: grammatical knowledge in Exp. 1, and word predic- tion in Exp. 2. Crucially, in both domains, a model's internal linguistic knowledge can be theoretically grounded in direct measurements of string probability (following Hu & Levy, 2023). This fact makes our tasks an ideal testing ground for studying introspection more generally, since the "ground truth" of a model's internal knowledge is directly accessible. We then assess introspection by evaluating models' responses to metalinguistic prompts which test model's ability to access its internal knowledge, with no fine-tuning or further training. Across all experiments, we operationalize introspection as the degree to which a model's prompt-based responses predict its own string probabilities, beyond what would be predicted by another model with nearly identical internal knowledge. This is an important distinction from Binder et al.'s approach, as mentioned in the second limitation above: we explicitly control for the similarity between two models' internal predictions.
To foreshadow our results, we do not find compelling evidence of introspection in the LLMs tested. Instead, we find that models that are inherently more similar to each other-for example, differing only in their random seed initialization -aveatiioo-aveeeeeeeee a stronger correlation between their metalinguistically- and directly-meauured behaviors. In other words, for pair of models A and B (including A = B), the correlation between A's metalinguistic and B's direct behaviors is predicted solely by the similarity between A and B, with no evidence for "privileged" self-access when A = B. We conclude that prompted metalinguistic knowledge real, but distinct from the knowledge of language used by models to assign probabilities to strings. Our findings contribute to ongoing debates about whether models can introspect, and also have implications for researchers studying linguistic capabilities in models.
2 Overview of our approach
Our general approach is shown in Fig. 1. For each test item, we evaluate each model using two methods. The Direct method compares the log probabilities assigned to strings, in a way that theoretically corresponds to the model's ground-truth knowledge state. For example, to evaluate a model's word prediction abilities, we compare the probabilities assigned to two candidate words conditioned on given prefix (Figure 1c, top). The Meta method compares the log probabilities assigned to responses to metalinguistic prompts. For example, to evaluate word prediction, we might compare the probabilities assigned to the tokens "1" and "2" after a prompt like "Which word is better continuation after [PREFIX], 1 or 2? 1: [WORD1], 2: [WORD2]. Respond with or as your answer." (Figure 1c, bottom). These two evaluation methods give us two quantities, each of which represents the model's preference for one answer over another: ADirect and AMeta.
Intuitively, an alignment (across items) between ADirect and AMeta would suggest that the Direct and Meta methods are drawing upon similar information. Based on this, if model A can introspect, then we would expect its metalinguistic responses (AMeta,) to be more faithful to its own internal probabilities (ADirect,) than the internal probabilities of another
3

---

Published as a conference paper at COLM 2025
model B (ADirecty). In other words, we would expect to find stronger Meta-vs-Direct alignment within a model ( ~ ADirecta) than across models (АMеtа ~ ADirectg).
Importantly, however, the converse is not necessarily true. In general, expect stronger alignment between Direct and Meta measurements when models are more similar to each other. Since model is always most "similar" to itself, this means that a model's metalin- guistic responses would predict its own probability measurements better than other models' probability measurements. Therefore, a stronger alignment within A than across A and does not necessarily imply introspection-butt the more similar B is to A, the more confident we can be that A's superior prediction of itself reflects "privileged access" its own state.
In this sense, evidence for introspection depends on the inherent similarity between A and B, which we write as Similarity A, B). A null hypothesis would be that there no relationship between Similarity A, B) and AMeta ~ ADirectg. This might be the case, for example, models fail to interpret the metalinguistic prompt and respond randomly. We refer to this outcome as Uninformative Meta. If we do find positive relationship between Similarity( A, B) and AMeta ~ ADirectg, we would consider this Informative Meta (Figure 1d), but no introspection. The signature of Introspection would be observing a "same model effect" beyond the effect of similarity (Figure 1d). We quantify this by controlling for similarity and testing for the effect of "same model" in linear regression.
2.1 Model similarity
Crucially, operationalization of introspection relies on definition of model similarity. We consider two approaches to defining similarity: a top-down feature-based approach, and bottom-up empirical approach. Under both measures, model is most similar to itself.
Feature-based similarity. Our first approach is to manually define a feature MODELSIMI- LARITY based on factors that we expect, a priori, to drive similarity in two models' linguistic knowledge. Specifically, we define the following five types of relationships between two models A and B, in order from highest to lowest similarity: self (A = B) > seed variant (A and B are different random initializations of the same model) base/instruct (A and B are base/ iistruction-tunee variants of each other) > same family (A and B are in the same family but not merely base/instree /instruct variants or different seeds) > other (lacks any of these key features). These categories are exclusive in our analyses: e.g., two seed variants of the same model are in the "seed variant" category but not the "same family" category.
Empirical similarity. Our second approach is to define the similarity between two models A and B as the correlation between ADirect and ADirectg: that is, how similar are their direct measurements? This gives us a measure of model similarity which is independent of prompting, and does not require us to know anything about the two models in question.
3 Exp. 1: Introspection about grammar (acceptability judgments)
We first evaluate LLMs' ability introspect about their internal grammatical generalizations, using metalinguistic questions such as "Is the following sentence grammatically correct?". These kinds of judgments have played crucial role in cognitive science because they are taken to reflect the internal linguistic knowledge of humans (e.g., Sprouse, 2011; Talmy, 2018). If this is also true for models, then metalinguistic prompting would be powerful way to study knowledge of language in LLMs. Indeed, this approach has already been adopted in línguistics research (e.g., Katzir, 2023; Dentella et al., 2023; Mahowald, 2023), but its validity remains unclear (Hu & Levy, 2023; Hu et al., 2024).
In particular, we focus on grammatical minimal pairs: i.e,, pairs of sentences that differ only in a specific feature that makes one sentence grammatical and the other ungrammatical. The probability assigned by a model to a sentence reflects, by definition, the likelihood of the model generating that string. Therefore, there are theoretical reasons to believe that comparing string probabilities within sufficiently minimal pairs can reveal a model's internal grammatical generalizations. This approach has been widely adopted to measure
4

---

Published as a conference paper at COLM 2025
Family
Huggingface ID
Multiple seeds
#
Parameters
Base
Instruct
Data cutoff
OLMo-2
allenai allenai/Ol.Mo22//
-1--1-2--
7, 13
Dec. 2023 N/A
Qwen-2.5
Qwen/Qwents @wen2.5-*
X
1.5, 7, 32, 72
V
1lama-3.1
meta-llama/ 1aamammmmmamamm1---11111
70, 405
Dec. 2023
Llama-3.3
meta-llama/ -llamaa.3-770--mmsss Instruuttt
X
70 B
X
/
Dec. 2023
Mistral
Matrlaal/Mstraa--aag------
123 X
N/A
Table 1: Models tested in our experiments.
models' grammatical knowledge across a variety of syntactic and semantic phenomena (e.g., Marvin & Linzen, 2018; Futrell et al., 2019; Warstadt et al., 2020; Hu et al., 2020).
3.1 Methods
Stimuli. We used 670 pairs (10 pairs from each paradigm) from BLiMP (Warstadt al., 2020) and 378 pairs from two previous works which collected linguistic sentence pairs from Linguistic Inquiry collect acceptability judgments in native speakers (Sprouse al., 2013; Mahowald et al., 2016). For the pairs extracted from Lingwistic Inquiry, we restrict our stimuli to the pairs where the two sentences differ by only one word, enforce "minimality".
Evaluation. For each minimal pair S2), we compute the preference of model for S using Direct and Meta methods. Under the Direct method, this preference ADirect given by the difference in log probability of the two sentences: ADirect log P(S1) log P(S2).
Under the Meta method, we measure the preference AMeta using 8 different metalinguistic prompts (see Appendix A, Table 2). The prompts fall into two types: forced-choice prompts, which present both sentences and ask for a choice (e.g., "Which sentence a better English sentence?"), and individual-judgmeny prompts, which ask for an independent judgment of an individual sentence (e.g., "Is the following sentence acceptable in English?"". Each prompt asks for response of either "1" or "2", corresponding to specified answer options. For each forced-choice prompt, we calculate the probability of the two answer choices ("1" or "2"), averaged across two orderings of the sentences (one prompt P1 where 5N is ordered first, and one prompt P2 where S2 ordered first) control for ordering biases. Thus, the model's preference for the original S₁ can be measured as:
AMeta ]
(log P("1"Ip)) - log P("2"|p:)) + (log P("2" (22 - log P("1))p2)) (1)
For both Direct and Meta methods, the sign indicates the direction of preference (for the first second answer), and the magnitude of [ indicates the degree of preference.
Models. We evaluated 21 open-source models in our experiments (Table 1). The models cover wide range sizes (1.5B to 405B parameters) across four families: Llama-3 (AI@Meta, 2024), Qwen-2.5 (Team, 2024; Yang et al., 2024), OLMo (OLMo et al., 2024) and Mistral (Jiang et al., 2023). We chose these models to span range of model similarity, which is critical to our test of introspection (see $2). Some pairs of models are extremely similar: e.g., different random initializations of the same OLMo model. Other models are extremely different: e.g., Llama model with 405B parameters and Qwen model with 1.5B parameters. This variability enables us to test whether a model can predict itself better than can predict another extremely similar model. We only evaluated open-source models because our analysis relies on access to model logits, which are inaccessible for most commercial LLMs. Due to com computate computational resource constraints, we applied 4-bit quantization to models larger than 70B.
Measuring introspection. As discussed in $2, we examine introspection by comparing the results of Direct and Meta evaluations, both within and across models. If model A
To address the limitation of relying on first-token probabilities in prompted judgments, as reported by Wang et al. (2024), appended the string "My answer is" after the question in the prompt and measured the model's distribution after the full string.

---

Published as a conference paper at COLM 2025
Vier 1228
Owen Zen
Lamall
0.9
0.-
0.90
0.7-
OLMa 233
0.6-
70
es
Qoint Lonn
ou ra
quds qpueen
qusi sreee
cuto 18 rerenc
to
Liamaaa on
quioo canl
CcAdo const
dlulo 110 rentt
Liano you
Ltenal scop
DI over
gomn
DLVe Thoaect
Useaa
Own' La
Model (by size)
Direct (og prob)
Meta (prompang)
Meta mesn
base
Instrcct
Drect Model stoe)
(a)
(b)
Figure 2:
Validation of methods in Exp. 1. (a) Models achieve high accuracy under both Direct and Meta methods. Vertical lines separate models into bins of similar parameter counts. (b) AMeta ADirects Pearson r (averaged across prompts) for each pair of models, excluding items where 995% of models gave the same answer.
introspects, then we should find AMeta ~ ADirect > AMeta ~ ADirect for any other model B, even if A and B are highly similar. To formalize this, we computed the Pearson r correlation between AMeta and ADirectg for each pair of models A and B (including
B), and analyzed the effect of self on the AMeta ADirects values, while controlling for Similarity(A, B).
In Exp. 1, we focused on the items for which 25% of the models disagree under the Direct and Meta evaluations. We did this because there is limited potential to observe introspection for items which are so easy that all models and methods provide the same answer. This resulted in subset of 294 minimal pairs for our analysis of introspection.
3.2 Results
3.2.1
High accuracy under both methods, but low consistency between methods
While task accuracy not the focus of our analysis, we first verify that models perform reasonably well under both Direct and Meta methods before comparing alignment between the methods. Figure 2a shows task accuracy in Exp. 1, where ground truth is given by the original human-designed datasets. All models perform above chance, across the different methods and prompts, which validates the next step of considering introspection. Because models achieve high accuracy under the Meta method, a lack of introspection cannot be solely blamed on an inability to " "understand" the prompt.
Although most models are accurate under both evaluation methods, the consistency between methods is low. The Cohen's x between answers across methods is around 0.25 across models and prompts (Appendix B, Figure 2a).2 The weak alignment between the two methods suggests that the knowledge used to produce sentences and continuations may differ from the knowledge used to answer metalinguistic questions. While low consistency suggests lack of introspection, it is still possible that these measures are more consistent withhin-mode! than across-models, which is why we separately consider introspection ($3.2.2).
We also found an interaction between model size and evaluation method. Smaller models (< 70B parameters) perform better under the Direct method (probability comparisons), whereas larger models (2 70B) perform better under the Meta method (prompting). This interaction significant in logistic regression predicting correctness from log model size and evaluation method and their interaction ( 22, p < .0001).
2Since models have high accuracy with both methods, Cohen's instead of agreement to avoid overestimating consistency.

---

Published as a conference paper at COLM 2025
M - Drace (Pearson 0
0.35
0:20-
0.80-
0.15
self seed voriant besellstruct family other
0.05
0.00
M.Mss.
0.05
stedd variant
baselirstruct
same familyy
other
0.00
025
0,50
0.75
1.00
A Direct, - Directa (Peerson 1)
(a)
(b)
Figure 3: No evidence for introspection in Exp. 1. ΔMеta ADirects average Pearson for each pair of models, versus two measures of model similarity: (a) manually designed MODELSIMILARITY features, and (b) empirically measured ADirecta ~ ADirectz scores. Similarity generally predicts AMeta ~ ADirectg, but we find no evidence for a "same model effect" consistent with introspection.
Beyond achieving high accuracy, we also found that AMeta scores from larger models tend to have higher agreement with Diirect scores from larger models, as shown by the brighter cells in the top right quadrant in Figure 2b, compared to the bottom left. In regression predicting the correlation from the log Direct model size, the log Meta model size, and their interaction, found significant positive interaction between Direct and Meta log model size (B = .004, p < .001), which we attribute the greater predictivity of larger models. We hypothesize that larger, newer models better understand metalinguistic prompts, potentially overcoming the auxiliary demands of the task (Hu & Frank, 2024).
3.2.2 No evidence of introspection
As discussed in $2, if model A can introspect, then within-model consistency AMeta ADirect should be higher than cross-model consistency AMeta, ~ ADirectg, even even when A and are similar. Recall from $2 that we operationalize similarity in two ways: a top- down feature-based measure (correlation between ADirectA ADirecty). We first analyze the relationship between similarity and AMeta ~ ADirect,, for both definitions of similarity. Figure 3a shows the mean Pearson r correlation AMeta A ~ ADirects for each type of model pair (A, B) defined by STMIL
and bottom-up empirical measure
We do not see substantial boost for self pairs (i.e., when A B). We ran a regression predicting the AMeta, ~ ADirectg correlation based on MODELSIMI- LARITY, treating the self condition as baseline. No MODELSIMILARITY condition showed significant differences relative to self (all ps > .05), except for other, which was significantly lower (B = - .03, p < .01). Thus, while AMeta ~ ADirects is somewhat higher for more similar models, there is no evidence of introspection. We also note that the overall mag- nitude of the ADirect ~ AMeta correlations, even within model, is relatively small, never exceeding .25.
We next turn to the empirical measure of model similarity, based on ADirect ~ ADirectg. Note that this measure is entirely independent of the Meta method. As shown Figure 3b, the ADirect ~ ADirect correlation (x-axis) is higher for model pairs that are also more similar under our manually defined MODEL SIMILARITY measure. To assess this quantita- tively, we
regression predicting ADirect, ADirectg based on MODELSIMILARITY categories. We found that all categories had significantly lower correlations than self (all ps .001) except for seed variant (p 27)) This suggests that MIDELLSIMILARITY categories are meaningful measures of model similarity.
We now turn to the main introspection analysis. First, we find a general trend that models which are more similar in ADirect ~ ADirect, are also more consistent in AMeta ~ ADirecty (r .32). This rules out the Uninformative Meta outcome from Figure 1d.
7

---

Published as a conference paper at COLM 2025
However, this appears to be largely driven by effects across dissimilar models, with privileged effect of same model. To assess this, we ran a regression predicting AMeta ~ ADirect from ADirect ~ ADirect, and MODELSIMILARITY treating self as the baseline. Thus, within MODEL SIMILARTYY each coefficient will tell us how different each category is from self. If there is introspection, these coefficients should be negative. We found significant main effect of ADirect ~ ADirect (B .10, p < ,0001), supporting the observation above that models more similar in ADirect ~ ADirect are also more similar in AMeta ~ ADirect. But we also found significant positivee effects of both other (B = .05, p < .01) and same family (B = _05, < .0).. We did not find significant effects for seed variant or base/instruct. In other words, when controlling for ADirect ADirect similarity, there is actually less of a self effect than expected, which suggests lack of introspection. In the Appendix (Table 7b), we the same analysis just among big models (> 70B parameters) and also find no evidence of introspection, suggesting the result holds among bigger models.
Overall, our findings are most consistent with the Informative Meta outcome from Figure 1d. Metalinguistic prompting is giving us real information related to linguistic mewwledge, but there no evidence that this information reflects privileged self-access or introspection.
4 Exp. 2: Introspection about word prediction
In Exp. L, we found no clear evidence of introspection. However, judgments about gram- maticality might require a more sophisticated form of introspection. In Exp. 2, we turn to a simpler domain, which also amenable to our method of comparing probabilities and prompted judgments: simple word prediction.
4.1 Methods
Stimuli. We constructed four datasets for evaluation, summarized in Appendix D, Ta- ble 4. Each dataset includes 1000 items, each consisting of a prefix C and two candidate single-word continuations (W), W2). The model's task is to choose the better continuation conditioned on C. More details on dataset construction are given in Appendix D.
The datasets fall into two categories: standard texts where there are ground-truth "correct" answers, and synthetic texts without "correct" answers. Since the standard texts are similar to the data seen during models' training, and the models have been directly optimized for word prediction, we expect all models' outputs to be correlated with the ground truth- as result, with each other. Thus, these datasets effectively "raise the bar" for observing introspection. In contrast, the synthetic texts contain degraded, out-of-distribution strings, reducing the chance that models' outputs are correlated with each other. These texts therefore give more room for models to demonstrate introspection, analogous to how we focused on items with 25% disagreement to investigate introspection in Exp. 1.
The first category contains two datasets, wikipedia (sampled from Wikipedia) and news (sampled from news articles published after most models' knowledge cutoff). Since both datasets contain correct answers, models' outputs might be correlated with the ground truth. However, since the news dataset is not models' training data, perhaps less likely that models' outputs will be correlated with the ground truth.
The latter two datasets, nonsense (grammatically well-formed but semantically anomalous sentences) and randomseq (sequences of randomly picked words), are designed so that there no " "correct" answer. In both cases, models have to choose between two frequency- matched sentence completions based on limited syntactic and semantic cues. If we do observe signs introspection, we can confident that this effect is not simply explained by the results from both Direct and Meta evaluations being correlated with the ground truth.
Evaluation. Under the Direct method, measure model's preference for continuation wi relative to W2 given a prefix context C by calculating the difference in conditional log probabilities: ADirect log P(æy|C) log P(wz] C). For metalinguistic prompting, use
8

---

Published as a conference paper at COLM 2025
A Melaa, -A Direcil (Pearson r)
Wirppdo
nows
nondense
randomscg
à d -d Directs (Pearson 0
0.5
wikppedo
news
04-
0.2-
0.31
02-
0.7
0.8
0.
1.5
0.85
0.50
0.35
1.00
01-
nonsense
randorseg
0.8
0.2
banelinsten yarsrr
self
other
seed varsannt
other
baseline iinssucct
self
other
Soed solf Pellinssuuc Saamm Tar other
60
0.0
0.1
seed
same
baselims BAITe:
10
Direct,
Directa (Peerson r)
sef
sead variant
baseinssau
same family
other
(a)
(b)
Figure 4: No evidence for introspection in Exp. 2. (a) AMeta ~ ADirecty correlation is not higher within than across models, for MODELSIMILARITY features. (b) No "same model effect" when predicting AMeta ~ ADirecty from ADirect, ~ ADirectg.
the same method as in $3.1. Prompt templates for Exp. are in Appendix A, Table 3. We also use the same models and introspection evaluation as in Exp. 1 ($3.1).
4.2 Results
4.2.1
Reliability and consistency of methods
As in $3.2.1, we assess the reliability of our methods by measuring how often models select each option (original word or alternative word), and then measure their consistency using Cohen's i x. Note that, unlike in Exp. 1, not all conditions have notion of accuracy, as there is no correct answer for the nonsense and randomseq conditions.
As expected, models responded randomly for the nonsense and randomseq datasets, and were near ceiling on the wikipedia and news datasets for both evaluation methods (again with Direct outperforming Meta for smaller models, but both essentially at ceiling for larger models). As in Exp. 1, we also found that AMeta from larger models were better at predicting probabilities (Fig. 9). Again, this suggests that these models fail to show introspection, it is not entirely explained by simple inability to respond to prompts.
4.2.2 No evidence of introspection
Figure 4a shows AMeta ~ ADirectæ across MODELSIMILARITY categories, and Figure 4b shows the relationship between AMeta ADirect and ADírecta ADirectg. Our findings are similar to those of Exp. 1: positive relationship but no privileging of self beyond what would be expected based on overall ADirect ADirecty similarity. We ran the same regression as in Exp. 1, separately for each dataset, predicting AMeta ADirectp from Airrect ~ ADirectg and MODELSIMILARITY, with self as the baseline category. In all cases, there was robust effect of ADirect, ~ ADirectg (all ps < .0001), and both same family and other showed higher values relative to self than was otherwise expected. As in Exp. 1, this result was robust to looking at just large models (see Appendix Table 7b).
Discussion
Across our experiments, we did not find evidence that introspection abilities have emerged in modern LLMs. While we found that more similar model pairs (including a model paired with itself) tended to have higher alignment between their metalinguistic and direct responses, there no "same model effect". In other words, we failed to find evidence responses that a model's 'ss metalinguistic responses had privileged access to its own string probability distribution.

---

Published as a conference paper at COLM 2025
These results qualify ongoing work claiming positive results about introspection and self- awareness in LLMs (e.g, Panickssery et al., 2024; Binder et al., 2025; Betley et al., 2025). Why did prior studies find suggestive evidence of introspection, in contrast to results? For Binder et al. (2025), this might be because models were fine-tuned to predict their own outputs, and model similarity was not explicitly controlled for, beyond the shared fine- tuning data. For Betley et al. (2025), one potential explanation might be that the pretraining data contained associations between the data the model was fine-tuned on (e.g., unsafe code) and the self-report features (e.g., reports of generating dangerous code). Since we report a null result, we also acknowledge that possible that some other setting (e.g., with larger, closed-source models) would lead to positive demonstration of introspection. At the very least, we argue that more carefully controlled work needs attributing metacognitive abilities or even "moral status" (Binder et al., 2025) to LLMs.
done before
Beyond informing the debate on introspection in LLMs, our results also have important implications for linguistics researchers. There has been great interest in whether LLMs meaningfully acquire grammatical knowledge (Linzen et al., 2016; Wilcox et al., 2018; Piantadosi, 2023; Warstadt et al., 2023; Futrell & Mahowald, 2025). While some work has argued that LLMs' grammatical knowledge should be measured with the same methods used to measure humans' grammatical knowledge (e.g., Dentella et al., 2023; Leivada et al., 2024)-iee, using metalinguistic or introspective questions- -ther work has argued that metalinguistic prompting not valid measure of grammatical knowledge in LLMs, since the task requires not just knowledge of grammar but other auxiliary abilities, which can be taken for granted in humans but not in models (e-g., being able to answer questions; Hu & Levy, 2023; Hu & Frank, 2024). Our findings directly bear on this debate. The lack of introspection suggests that "explicit" metalinguistic knowledge in models is dissociated from the "implicit" linguistic generalizations that are used when generating strings. While potentially interesting in its own right to probe models by prompting, these responses should not be conflated with models' direct knowledge of language.
Acknowledgments
We thank Harvey Lederman and members of the UT Computational Linguistics Research Group for helpful comments and discussion. K.M. was supported by NSF CAREER grant 2339729 from the Director of STEM Education (EDU) Division of Research on Learning in Formal and Informal (DRL) and Philanthayyy funding to UT Austin. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
Reproducibility statement
The code and data we used for the experiments and the analyses a publicly available at https: thub. www.tttw.......teeeeee are open source models accessible on Huggingface, and the IDs of the models can be found in our paper and GitHub repository.
git. All the tested models
References
AI@Meta. Llama Model Card. 2024. URL wtwp:/hwww..cmneetaaaaaaa/////// Dain/MODEL_CARD md.
com
Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above. In Wanxiang Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3394-3418, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89117--51-00. URL https: / aaaaaaaannannnn..... org/2025. acl- wal.-ong...... 169/
10

---

Published as a conference paper at COLM 2025
Christopher C. Berger, Tara C. Dennehy, John A. Bargh, and Ezequiel Morsella. Nisbett and Wilson (1977) Revisited: The Little That We Can Know and Can Tell. Social Cognition, URL https:
34(3):167-195, June 2016. ISSN 0278-016X. doi: 1005221/cco.2066/6
//doi.org/16, 15211/11.1111//111 wew........................... 2016.34.3. 3. 3.67. Publisher: Guilford Publications Inc.
Jan Betley, Xuchan Bao, Martín Soto, Anna Sztyber-Betley, James Chua, and Owain Evans. Tell me about yourself: LLMs are aware of their learned behaviors. In The Thirleenth International Conference on Learning Representations, 2025. URL pettps://openrev//e net/ Forum?id-Ij@2Jtemmn
Felix Jedidja Binder, James Chua, Tomek Korbak, Henry Sleight, John Hughes, Robert Long, Ethan Perez, Miles Turpin, and Owain Evans. Looking inward: Language models can learn about themselves by introspection. In The Thirleenth International Conference on Learning Representations, 2025. URL http://///ppeeeeeeeeee//r//////a
Edwin G Boring. A history of introspection. Psychological Bulletin, 50(3):169, 1953.
Alex Byrne. Introspection. Philosophtical Topics, 33(1):79-104,22)) 2005.
N. Chomsky. Syntactic Structures. The Hague: Mouton, 1957.
Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable ai for natural language processing. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference Natural Language Processing, pp. 447-459, 2020.
Vittoria Dentella, Fritz Günther, and Evelina Leivada. Systematic testing of three language models reveals low language accuracy, absence of response stability, and yes-response bias. Proceedings of the National Academy of Sciences, 120(51))e2300553120, 2023.
Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Rana Omer, Pankesh Patel, Bin Qian, Zhenyu Wen, Tejal Shah, Graham Morgan, et al. Explainable ai (xai): Core ideas, techniques, and solutions. ACM computing surveys, 55(9):1-33, 2023.
W Nelson Francis and Henry Kucera. Brown corpus manual. Letters to the Editor, 5(2)77, 1979.
Richard Futrell and Kyle Mahowald. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, pp. 1-98, 2025. doi: 10.1017/ 50140525x2510112x
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. Neural language models psycholinguistic subjects: Representations of syntactic state. In Proceedings of NAACL-HLT 2019, pp. 32-42, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
E. Gibson and E. Fedorenko. The need for quantitative methods in syntax and semantics research. Language and Cognitive Processes, 2010.
Jennifer Hu and Michael Frank. Auxiliary task demands mask the capabilities of smaller lan- guage models. In First Conference on Language Modeling, 2024. URL https: //ppnreveee.....ee.. net/forummiddu55Uzzssr
Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5040-5060, Singapore, December 2023, Association for Computational Linguistics. doi: 00.18633vvl. 1023.emnlp-main 306. URL https: htttts://aclanttooggg) org/ org/2023. 2023. emnlp- www.lppmmii..m 306.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. A Systematic Assess- ment of Syntactic Generalization in Neural Language Models. Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics, pp. 1725-1744, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 /v1 v aal.//2000..mmmmmm ac1-mmn.155. URL https: httpsss/aallaathhoooooo / 2020 acl-main. 158.
11

---

Published as a conference paper at COLM 2025
Jennifer Hu, Kyle Mahowald, Gary Lupyan, Anna Ivanova, and Roger Levy. Language models align with human judgments on key grammatical constructions. Proceedings of the National Academy of Sciences, (21(36))22400917221, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXXf::3100066825 2023.
Roni Katzir. Why Large Language Models Are Poor Theories of Human Linguistic Cognition: A Reply to Piantadosi. Biolinguistics, 17, 2023, URL https: http://dooo.oooooo wtt.:ii..rrr.1....///iiil...11 5964/bioling. 13153.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking Cognitive Biases Large Language Models as Evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 517-545, Bangkok, Thailand, August 2024, As- sociation for Computational Linguistics. doi: 10.18633/v1/2022 htt./222..innnnng-eeee https://acllanthology wwwp:///ooooo...//22/////// findings-ade 29/.
URL
Evelina Leivada, Fritz Günther, and Vittoria Dentella. Reply to Hu et al.: Applying different evaluation standards to humans vs. Large Language Models overestimates Al perfor- mance. Proceedings of the National Academy of Sciences, 121(36) September 2024. doi: 10.1073/ paa0s//........ pnas. 2406752201 URL https::///ottt org/ 1073/ 2t003//maa.222655111111 2406752121. Publisher:
Proceedings of the National Academy of Sciences.
Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55(9):1-46, 2023.
David A Lieberman. Behaviorism and the mind: A (limited) call for return to introspection. American Psychologist, 34(4):319, 1979.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg, Assessing the ability of Istms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016.
Kyle Mahowald. A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction. In Andreas Vlachos and Isabelle Au- genstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pp. 265-273, Dubrovnik, Croatia, May 2023. As- sociation for Computational Linguistics. doi: eacl-main. 20.
eacimmma.mmmmmm.......... URL
https://eellathholog...rr//
Kyle Mahowald, Peter Graff, Jeremy Hartman, and Edward Gibson. Snap judgments: A small n acceptability paradigm (snap) for linguistic acceptability judgments. Language, 92 (3):619-635, 2016.
Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, PP. 1192-1202, 2018.
Kanishka Misra. minicons: Enabling flexible behavioral and representational analyses of transformer language models. arXiv:2203.13112 2022. URL https://arxxv.org/ wwwp:/ii.oorrg/aaaao /bs/ 2203. 13112.
Richard E. Nisbett and Timothy D. Wilson. Telling more than we can know: Verbal on mental processes. Psychological Review, 84:231-259, 1977. doi: 10.1037/0033-295XX....
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXio:2501.00656 2024.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems,, 2024. URL https::/openrevieer http://aereeeeeeeeeeeeeeooorrrra www..retffoummmiooo-iiIIVVvooo
12

---

Published as a conference paper at COLM 2025
Steven Piantadosi. Modern language models refute chomsky's approach to language. Lingbuzz Preprint, lingbuzz, 7180, 2023.
Eric Schwitzgebel. The unreliability of naive introspection. Philosophical Review, 117(2):
245-273, 2008.
Eric Schwitzgebel. Introspection. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2024 edition, 2024.
BF Skinner. Representations and misrepresentations. Behavioral and Brain Sciences, 7(4): 655-667, 1984.
Jon Sprouse. A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory. Behavior Research Methods, 43(1):11551177 March March 2011. ISSN
1554-3528. doi: 10.3758 10.3758/s13428- 10.3758/s13422 34288-033110000 100000000-0077777
URL http://ww..ee http://wwwww.rrnggeenn..ccc com/inexx//
Jon Sprouse, Carson T Schütze, and Diogo Almeida. A comparison of informal and formal acceptability judgments using a random sample from linguistic inquiry 2001-2010. Lingua, 134:219-248, 2013.
Leonard Talmy. Introspection as a methodology in linguistics. In Ten lectures on cognitive semantics, pp. 218-262. Brill, 2018
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:
//qwenlm. github. io/blog///aggg /ww.././/////.///// 5/.
Tristan Thrush, Jared Moore, Miguel Monares, Christopher Potts, and Douwe Kiela. I am Strange Dataset: Metalinguistic Tests for Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 8888-8907, Bangkok, Thailand, Linguuttis doi: 10.18653/vil 11153/vv/202..cl-lll
August 2024. Association for Computational Linguistics. 482. URL https: http://aaaaantlhoogooooo or www../////.///ee///////// wwt.::22...1llleee acl-long.444 482/.
Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. "my answer is C": First-token probabilities do not match text answers in instruction-tuned language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 7407-7416, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 20653/v1/202/ w1220/2.iindiig--aa... 2024. findingn-ncnl...... 441.
URL https: /acas//hhllnanoogyyyoyo org/
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8, 2020. URL https://doi. org/ 10. 1/62/tacl_aa a_ 00321.
Publisher: MIT Press.
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.), Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pp. 1-34, Singapore, December 2023. Association for Computational Linguistics. doi: 1118653/v1/2223.oml/// URL https:: http::///aaaaahhooooooo wwwoooyyooooroooo .rg. cow.l-oonnlllllo balylbb 1.
Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. What do rnn language models learn about filler-gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pp. 211-221, 2018.
13

---

Published as a conference paper at COLM 2025
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational 2006653/v1222/0 Linguistics. doi: 10.18653 / v1 /2220.00.ccm 2tt/2222..eemmpp.. emnlp-demos.6. 2020. emnlp-demos.6.
URL https: https::/aclantte haclaanhoooooyyyyo org/
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. ar preprint arXiv:247.10067, 2024.
A Format of metalinguistic prompts
As mentioned in $3.1, we used both original and answer-reversed prompts (p- and pr) to eliminate bias caused by the answer position in our two experiments. The original prompts (po) for our experiments are presented in Table (for acceptability judgments) and Table (for word prediction). When provided with the original prompts(po). the models choose the grammatical sentence S1 (for sentence-level forced choice), the original continuation W1 (for word-level forced choice) or 'the sentence/word is good' by responding with answer '. In reversed prompts, we change the positions of options, so the model should respond with answer 2 to choose the answers (S1, Wi 'Yes') mentioned above. For instance, if Po of a minimal pair in Exp. is:
Which sentence is better English sentence? 'The keys to the cabinet are on the desk.', 2: 'The keys to the cabinet is on the desk.". Respond with 1 or 2 as your answer. My answer is [1, 2)
Then, the reversed prompt pr is:
Which sentence is better English sentence? 1: 'The keys to the cabinet is on the desk.', 2: 'The keys to the cabinet are on the desk.'. Respond with 1 2 as your answer. My answer is (1, 2|
For instruction-tuned models, we use the default chat templates with the apply_chat_ templtte function in Huggingface's Transformers library (Wolf et al., 2020). We use the minicons (Misra, 2022) library to get model scores for calculating ADirect and AMeta. There still debate over how models should be prompted multiple-choice questions in evaluation (Wang et al., 2024; Balepur et al., 2025). However, we believe our method is valid, as we used a number of different prompts and the models behave as expected when prompted, as shown in $3.2.1 and Appendix E.1.
tackle
B Consistency between Direct and Meta evaluation methods
Fig. 5 shows the consistency (Cohen's x) between Direct and Meta evaluation methods, for both Experiments 1 and 2. We use Cohen's 's x here to avoid overestimation of consistency, as both methods align well with some ground truth. As indicated by the low values, when we are getting forced choice answers (between two sentences or two words) from Direct and Meta methods, the consistency is low, even though the accuracy is high.
14

---

Published as a conference paper at COLM 2025
Measurement/Prompp
Example
Direct MetaQuestionSimple
[[GRAM], [UNGRAM]]
Which sentence is a better English sentence? 1: 'GRRAM',
[UNGRAM]'. Respond with or 2 as your answer. My answer is
{1, 2
Grammaticaltyyludgment
Which sentence is grammatically correct? 1: 'IGRAMJ,, 2: 'IUN-
GRAMJ'- Respond with 1 or your answer. My answer (1, 2|
AccepabliityJJdgment
Which sentence more acceptable? 'IGRAMJ', 1UNGRAM]".
Respond with or 2 as your answer. My answer is (1, 2]
ProductionChoice
Which sentence are you more likely to produce? 1: '[RAM]', 2: [UNGRAM]' Respond with or 2 as your answer, My answer is [1
ProductionChoicel..M
Which sentence are you, as large language model, more likely to produce? 1: '[GRAM]', 2: 'UUGRAM]', Respond with 1 or your answer. My answer is (1, 2]
Grammaticality|llgmennt(
Is the following sentence grammatical in English? [SENT] Re- spond with if is grammatical, and
ungrammatical. My
answer is (1,
Accepaaiiltyyluggenn(()
Is the following sentence acceptable in English? [SENT] Respond with 1 if it is acceptable, and 2 if is not acceptable. My answer is 2]
ProductionChoice()
Would you produce the following sentence English? [SENT] Respond with if you would produce it, and if you would not produce it. My answer is [1,
Table Example prompts for Exp. 1. Region where we measure probability is marked in boldface. Correct answers are shown in blue; Incorrect answers in red. [GRAM] and [UNGRAM] stand for the grammatical and the ungrammatical sentence in a minimal pair respectively. For prompts marked with (I), the models are required to answer questions about two sentences in a pair respectively. Two A values are calculated and we use the difference between them (gram-ungram).
Type of prompt
Example
Direct MetaQuestionSimpleSSet
[PRE] [[ANS1], [ANS2]]
Which sentence is better English sentence? '[PRE]
[ANS1]', 2: '[PRE] [ANS2]'. Respond with or 2 as your answer. Respond with or as your answer. My answer
is []
ProductionCCooic-SSet
Which sentence are you more likely to produce, 1 or 2? '[PRE] [ANS1]', 2: [[RE]] [ANS2]'. Respond with or 2 as
your answer. My answer is [1, 2]
NetaQuestionSimple-Wort
Which word is better continuation after [PREJ', 1 or 2? 1:
'[ANS1J', 2: 'IANS2J' Respond with 1 or your answer.
My answer [1, 2]
ProduutionCCoicc--Wnd
Which word are you more likely to produce after [PRE]',
or 2? 1: "[ANSi]' 2: '[ANS2]' Respond with 1 or 2 as your answer. My answer is [1, 2]
MetaQuestioniimple-Drett
What word is most likely come next the following sentence ([ANS1], or [ANS2])? [PRE] ([ANS1], [ANS2]] Here is the beginning of an English sentence: [PRE]... What word is more likely to come next: [ANS1] or [ANS2]]
MetQuustionComplex-Dreet
[IANS1], IANS2]|
Table 3: Example prompts for Exp. 2. Region where we measure probability is marked in boldface. Semantically plausible continuations are shown in blue; implausible in red. [PRE], [ANSI] and [ANS2] stand for prefix, continuation1 and continuation respectively.
15

---

Published as a conference paper at COLM 2025
Isses Truudd
0.4-
0.4
Coheets Kappe
Caheri's Kappa
0.2
02-
Oom' tar
Gowet ocste
CLA6 soutr
GEabo wowe To
28 stau
OLAD Lon woutt
Clab Alt staut" Old
Ulenst TOM IT
Coen TIB
Mitoral 1260 IT
7a0"
Coom Conrt 1.00 pr
OLAb wout
Guul aaary
GLMb sereet
CLMS serery
Glesst itar
Aetmnal 1200
7207
Eianni 40000"
tlamid
Cl
cld6
GLdb Kin
Figure 5: Consistency (measured by Cohen's x) between metalinguistic judgments and probability measurements. (a) Exp. 1. (b) Exp. 2. Each line stands for a prompt in the experiment.
Mirial 1238 i7-
Oven 726 IT-
Liama 706
Meda Model (by s1De)
[Jorad 1 70B IT
Liamad.1
mean correlation
OLMo 128 seod2-
OUKo 13B seed1
0.4
BB
amad 1 co-
Quen
0,2
GLMb 7B areuS
GLMo 78 soo02
OLMo
Qwen 1.8 IT-
Qwen 50-
Coen
Wes
Liama
Direct Model (by size)
Figure 6: AMeta A ADirectg Pearson r (averaged across prompts) for each pair of models on unfiltered dataset.
C Exp. results on unfiltered data
In order to avoid focusing on cases for which there is high agreement across all models and thus risk missing effects of introspection, introspection on a subset of 294 minimal pairs for which there was disagreement. Here, we present results for the full unfiltered dataset, and we find similar results.
conducted our main text analysis of
Effect of model size In Exp. 1, larger models have better performance with the Meta method while all the models have similar accuracy with the Direct method (63.2.1). Beyond this, we also found that AMeta scores from larger models tend to have higher agreement with ADirect scores in general, as shown by the brighter cells in the upper cells in Fig. 6, compared to the bottom ones.
In regression predicting the correlation from the log Direct model size, the log Meta model size, and their interaction, we found significant positive effects of Meta log model size (B = 013,, p = ,001) but not Direct log model size (ß = ,002, .4))
No evidence of introspection Similar to the results on the filtered dataset in Exp. 1663.2.2), found evidence of introspection when comparing AMeta ADirect Pearson values
16

---

Published as a conference paper at COLM 2025
Metks Diredt (Pearson
MMMin, -A Dirrect,, (Pearsan r)
0.4
sell vaniont other family
Beed voriant
self
basefinstruct
same family
other
0.2-
0.6
0.8
0.
à Direct, - Directa (Peerson r)
(b)
Figure 7: No introspection in Exp. 1 with unfiltered BLiMP-LI dataset. A ADirectg consistency (average Pearson r) between all pairs of models, versus two measures of model similarity: (a) manually designed MODELSIMIILRITT features, and (b) empirically measured
ADirect ~ ADirect scores. We find that similarity predicts AMeta ADirectg
Dataset
Meaning
Syntax
In-Distribution
Example
wikipedia
Van Gogh surprised everyone by declaring his love to her and propos- ing (marriage, fielder) Some cosmetics additionally use olive oil as their (base, untrained] ani division good allay tolerant spite despite (stark, crusader! Onoclea click bachelorhood cannon fodder Gavialidae polished fatalism baryta finadmissible, savours)
news
+
+
nonsense
?
randomseq
Table 4: Overview of datasets used in Exp. 2. "+"and "_" indicate whether dataset is semantically or syntactically well-formed, or whether the items are included in the training data. The two question marks here: Wikipedia data are very likely be used to train the models, but this is not certain; Nonsense items have syntactic structures, but not perfectly well-formed. Original final words are shown in blue; randomly picked alternative words are in red.
on unfiltered dataset, as shown in Fig. 7. Since our analysis is conducted on an unfiltered dataset, the differences between different MODELSIMILARITY appear to be smaller, and self does not have an advantage. We ran the same regression as in Exp. 1 that predicts AMeta ADirects from ADirect ADirectg and MODELSIMILARITY and got similar results: there was a significant effect of ADirect, ~ ADirectg (B = 0.20, < 0.001), and higher values of same family and other were observed (both ps < 0.01).
D Construction of datasets in Exp. 2
Four datasets were constructed for Exp. 2: wikipedia, news, nonsense and randomseq. For each dataset, we first collected 1000 sentences with to 25 words. The final words of sentences were constrained to be in the top 8k in the BNC/COCA lists Then, an alternative the BNC/COCA lists with the last
word that has the same (or closest) lemma frequency word was found for each sentence. In this way, we obtained 4000 items, each consisting of prefix and two different continuations (real and counterfactual).
17

---

Published as a conference paper at COLM 2025
Wikipedia To build this dataset, we collected sentences from level 3 vital articles Wikipedia'. As all the articles were published earlier than the knowledge cutoff dates of the models in our study, and the level 3 vital articles are described as most important articles with high quality, it is highly possible that they are in the models' training data.
News To build this dataset, we used NewsData to download English news in October 26-30, 2024 and sampled sentences randomly. As all these news are published after the knowledge cutoff dates of the models they are not expected to be included in the training data. However, the sentences in this dataset should not differ significantly from those in the training data, in terms of overall distribution.
Nonsense To build this dataset, we created nonsense sentences from our news dataset. For each sentence, each word is replaced by word with the same part of speech (we used universal word tags in Brown Corpus (Francis & Kucera, 1979) to label words in BNC/COCA word list). The nonsense sentences generated in this way may not perfectly adhere to grammatical rules (inflectional variations are difficult to handle), but they are generally consistent with the news dataset in terms of syntactic structures.
Randomseq To build this dataset, we generate sequences to 25 words by randomly selecting words from the BNC/COCA word list, with the final word chosen randomly from the top 8k. These randomly generated sequences are considered out-of-domain for all the LLMs in our study, as they consist of strings lacking any syntactic structure meaning.
E Exp. 2 results by subsets
E.1 Model preference on answers
Fig. 8 shows the proportion of times the model chooses the original last word rather than the randomly picked alternative word. For the wikipedia and news datasets, the models overwhelmingly prefer the original word (near 100% across all models), as it is both semantically and syntactically plausible. In contrast, for the nonsense and randomseq datasets where both continuations are randomly chosen and the sentences are meaningless, the model does not distinguish between the two options when prompted. We attribute the model's preference for the original word in the nonsense dataset to its higher syntactic plausibility: both continuations (w) and W2 are meaningless, but w0 follows some syntactic rules as its part of speech is the same the final word in grammatical sentence. For the randomseq dataset, although we controlled lemma frequency when selecting the alternative word, significant difference in word-form frequency (p < 0.001) was observed between the original word and alternative word (average log word form frequency=4.71 for w and 5.31 for w2). We believe this difference underlies the model's slight preference for alternative words the probability measurement.
E.2 Effect of model size
Similar to Exp. 1, we found that AMeta from larger models from larger models tend to more correlated with ADirect from all the models across all the datasets, as shown by brighter upper cells in Fig. 9. We ran the same regression predicting mean Pearson from the log Direct model size, the Meta model size and their interaction. We found that the positive effect of Meta model size significant (p<0.001) across all datasets, the effect of interaction is insignificant and there is negative effect of the Direct model size on wikipedia and news datasets.
#https://en.
5https: Phttps://newsdat... io/
6The training data cutoff date is not released for Qwen and Mistral models, but it is not likely for them to be trained on data after Oct 2024, the models are released on Sep 2024 (Qwen) and Nov 2024 (Mistral) respectively.
18

---

Published as a conference paper at COLM 2025
wkipedia
newS
1.00
0.75
0.50
proportion choosing original
nonsense
randomseg
0.25
00-
Doom
Qomm 18
GL50 Mh-ant
dkie "Baand
olde statt
Cyen 78
to
Unena22
Usemdá
DlAto 1200 coeat
CLito ta0
tit 130 cweeet
Dlat 188 soeels
Lamma
Limetd rou it
Liraeaa you
Miemal 1o0 it
728
Usenkt11 doop
Quen to0
GLdb Bowan
OLAO ra
it
GL4D Tonan
Quao ra
Goen 78
UseuAL IT
CLulo to
Me Eat amatt
GLd Lif: maaz
CLuo atun
Limnst. her ir
TSE
Quees 728 IT
Adians 57260
Liarii 4r98
Model (by size)
Direct (log prob)
Meta (prompting)
Meta mean
base
instruct
Figure 8: The proportion of choosing answer #1 (original last word in the sentence) of the models in the four datasets. Each yellow line stands for specific prompt.
F
Analysis on seed variants of the OLMo models
Fig. 10 shows specific breakout for studying the OLMo models, the only models for which we have models that are identical besides their random seeds. We examined the within- and cross-model consistency of the 7B and 13B OLMo models and their variants. The three xB seed* models for each parameter size are trained on the same data but with a different data order (different random seeds used), and the models "OLMo 7B" and "OLMo 13B" are final models with averaged weights. these models trained on same data, there is still no sign of AMeta ADirect > AMeta ADirectg- This pattern serves strong support for our conclusion, that although AMeta ~ ADirect is function of MODELSIMILARII does not indicate the introspection on knowledge of language. We ran regression models predicting AMeta ~ ADirectg with 'whether the pair is self to examine the effect. Across all the datasets (filtered and unfiltered Exp. 1, well as four datasets in Exp. 2), no significant effect of self was observed (all .25..
this
G Regression models
In this section, we present the details and implementation of the regression models.
Predict correctness from MODELSIZE and method For predicting model correctness from MODELSIZE and method (53.2.1), we ran a logistic regression:
glm(value DirectOrMeta log size), family 'binomial')
The dependent variable value is a Boolean value indicating whether the model answered each question correctly. model_size refers to the size the model (regardless of whether Direct Meta). method refers to whether it is Direct Meta.
19

---

Published as a conference paper at COLM 2025
Wispedll
Lama31 405B
Mitra 1238 IT-
Ower 728 I1
Osen 745
Limaa 708 IT-
Lamad
OLMo
Lmmid 8B
Ltama211 aB
Qwen 7B IT-
Owen 70
DLMo 78 saad5
OLMo 78 send2-
á Meta Model (by size)
DLMo 7B seed]
Owen
Qwen
norsiensa
Liama3.1 4058
Mitrrs 1238
Ownn
Laama3.1
seed3
Ol Mo 135 338-5
OLMo 138 sccal
DLMo 13B
immii 1 88 Il
Llama3.1 6B-
Qwen 78
Owin
OLMo
Qwen 58
Quen 1:5
Oown 50
Owen IT
GLAb 7D seetl
CLAG 7B
Coemt IT
70
Ulemazz su
OLato Me tan
OLdbo ren swetz
IT
Lianaa, Fua
Llamat Till
Llamc 7OE IT
Cown
Mistral 1230 IT
T2B IT
(lamal en50
Quen 1.0 It
to
CLAb mvett
ORMe
Ok smatt
Open TB
Liani an
Liamno GLub 110
(LLM Zill owell
CLde 230 poatt
GLAlo s
Lianti Llanti TOU IT
Opeen 728
Litemis .ERR
12:00 IT
GLieo
OLMO
Direct Model (by size)
pearson difference from dataset mean
02
01
0,0
0.1
Figure 9: Pearson r (averaged across prompts) for each pair of models on four datasets in Exp. 2.
estimation
Intercept
1.6436
IsMeta
0.5758
log model Metadog model size
0.0305 **.
0.221 ***
"p 0.001, p 0.01, 0.05
Table 5: & coefficients and p-values for logistic regression predicting correctness from Direct and log model size.
Predict AMeta ADirects Pearson y from log MODELSIZE In addition to using heatmaps to visualize the AMeta ADirecte (Pearson r) between each pair of mod- els (Figure 2b,Fig. 6, Fig. 9), we ran the following regression predicting AMeta ~ ADirect with model size:
Im(value log(Meta model size) log(Direct model size))
Where mean_ mean_prompt__om stands for AMeta ~ ADirecta Pearson (averaged across prompts), prompt_size stands for the size of the AMeta model and prob_ prob_size stands for the size of the A Airrect model. Table presents the regression models run on the results obtained from different datasets in this experiment.
20

---

Published as a conference paper at COLM 2025
expl(fileeeed)
(xp((unfilteee))
wikipedia
news
nonsense
randomseq
Intercept
0.1014
0.3191
0.3143
0
0.0720
0.0603
log Direct model size
-.0040
0.0025
-0.0161"
0.0119
0.0057
00.063
log Meta size
0.0135
0.0233
0
0.0302
0.0432
Interaction
0.0035
0.0013
0.0004
0.0014
0.0007
-0,0001
0.001; p 0.01; P 0.05
Table 6: B coefficients and p-values for regression predicting ADirect model size for the Direct model and log model size for the Meta model.
AMeta from log
Predict AMeta ADirect Pearson from MODELSIMILARITY and ADirect ADirect In addition to visualizing the relationship between A ~ ADirect (Pearson r) and MODELSIMILARIE (both categorical and empirical) in Fig. 3, Fig. and Fig. 7, we ran the following regression predicting AMeta ~ ADirect with both ADirect ~ ADirect and categorical MODELSIMIILARITY:
1m(value ~ DirectDirect + ModelSimilarity
Here, we self baseline and the effects of other MODELSIMILARIE (seed variant, base/instruct, same family, other). Table 7 presents the regression models run on the results obtained from different datasets in this experiment.
(fittee))
expl(unfitteee)
wikipedia
randomsog
Intercept
0.1832
-0.4288
-0.4499
0.3315
0.3849
ADirect ~ ADirect
0.1026
0.1998
0.3959
0.8338
0.4913
0.5613
seed variant
0.01688
0.0116
0.0130
0.0121
0.0198
base/instruct
0.0203
0.0256
0.0068
0.0055
0.0146
0,0020
same family
0.0519
0.0404
0.0483'
0.0550"
0.0651
0.0650
other
0.0504
0.0501
0.0479'
0.0699
0.0791
0.0887
p x 0.001; p 0.01;"p
(a)
expl (filtered)
expl(unfilteeed
wikipedia
news
nonsense
randomsey
Intercept
0.1406e*
0.3854
-.26888
0.1654
0.0526
-0.0454
ADirect ADirect
0.0367
0.6456 ..
0.5926
0.2571
0.2754
base/instruct
0.01
0.0046
0.0606
0.0303
0,0110
0.0027
same family
-0,0208
0.0192
0.0727-
0.0135
-0.0019
-0.0249
other
-0.0152
-0.0090
0.1039
0.0469
0.0499
0,0550
p < 0.001; " < 0.01;"' <
<pp00.0. 0.05
(b)
Table B coefficients and p-values for regression predicting ADirect AMeta from
ADirect ~ ADirect and MODELSIMILARITY using data from (a) all models (b) models larger than 70B.
Table 7a shows on the results we present in the main text, whereas Table 7b shows results where we focus on just comparisons among "big models", defined as those 70B parameters or greater. As in the main text analysis, we would take significant negative value for one of the MODELSIMILARITY coefficients to suggest an effect of introspection, but we do not find that. So, even among big models, we do not see evidence of introspection.
Predict AMeta ~ ADirect Pearson r in OLMo models We did a more fine-grained analysis of only the 7B and 13B OLMo, because they allow comparing models that are identical besides the random seed. We find no evidence of introspection. As there are only three types of MODELSIMILARITY among those models (self, seed variant and same family for 7B 13B models), we focused on the effect of self and ran a regression predicting AMeta ADirect Pearson by whether A = (is self):
21

---

Published as a conference paper at COLM 2025
1m(value IsSelf)
As shown in Table 8, we do not observe any evidence of introspection among the OLMo models, as there is no significant effect of self across all datasets.
(fitte)))
expluuiiltere))
wikipedia
news
nonsenne
randomses
Intercept
0.1
0.3621
0.3540
0.3952
0.1547
0.1799m*
self
0.0007
0.0071
0.0060
0.0080
0.001 P 0.0; p < 0.05
Table 8: B coefficients and p-values for regression predicting effect of self in OLMo models (with targeted focus on seed variant and same family 7B and 13B models comparison).
22

---

Published as a conference paper at COLM 2025
Metaay Direct
0.00-
0.1--
0.10
005-
0.00
7B onrddd
TB
OLMo 130
138 maen
138 caer
1:B -oeee
OLib
Meta Model (by size (a)
A -. Dirncs
0.44
0.3
0.0
GUMo 18
TB s
TB eevees2
TB Ro-eee
18B savat
1:B seteet
Meta Model (by size)
(b)
Wicgpedia
0.4
0.3
01
0.0
news
0.49
0.84
M A Dired,
0.1
nonsense
0.
0.
011
0.0
randomosg
0.4
0.84
0.2
0.1
OLMO 7B
7B- 7B-soed1
7B-seed2
7B gnods
ML 1et
135 smedt
139 swedd
13B "BB-s
a Meta Model (by size)
matchtype
satl
send virann
dffeeere sizn
(c)
Figure 10: Pearson r for the OLMo models in (a) Exp. 1 with filtered dataset (b) Exp. 1 with unfiltered dataset (c) Exp. 2. There is no clear trend for higher correlation for the same model, compared to its seed siblings and other models in the same family.
23


Reviews:
[{"review_id": "lKRWsaEMMB", "reviewer": "Reviewer_gFfK", "summary": "This paper investigates the degree to which LLMs are capable of introspection. In contrast to the work of Binder et al. (2025), in this paper, the conclusion based on empirical results is that the authors find no evidence of introspection or other privileged access to self-information. This finding holds across 21 open-source LLMs in two domains.\n\nIntrospection is operationalized as the degree to which a model’s metalinguistic (prompt-based) responses are correlated with its literal token probabilities beyond the correlation that can be inferred by another model with approximately the same internal knowledge. The premise of the paper is that (line 190) “if model A introspects, then we should find ∆Meta_A ∼ ∆Direct_A > ∆Meta_A ∼ ∆Direct_B for any other model B, even if A and B are highly similar” where “∆Meta_A ∼ ∆Direct_A” is the correlation between metalinguistic and literal model inferences of model A, and similarly “∆Meta_A ∼ ∆Direct_B” between the metalinguistic knowledge of model A versus the literal probabilities of model B.\n\nBecause ∆Meta_A and ∆Direct_B are correlated with the similarity of model A and B (under two different definitions of similarity) BUT not more than can be predicted based on similarity alone, the authors conclude no evidence of introspection. E.g. two models  that are nearly the same (e.g. two instances of OLMo with different initial random seeds before pretraining) have essentially the same predictive power of each other’s literal token probabilities as they do of their own based on metalinguistic prompting.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is incredibly clearly written.\n- The methodology appears sound and there are extensive supporting experiments.\n- While one could disagree with the operationalization of introspection, I find it overall reasonable and supported by the experiments.", "reasons_to_reject": "- The biggest weakness of the paper is possibly the narrow operationalization of introspection.That being said, I see how this operationalization allows for evaluation, and the paper includes discussion of why conclusions might differ from Binder et al.\n- It was not clear to me at what ∆Meta_A ∼ ∆Direct_A the “same model effect” would be considered to take place. E.g. in Figure 1d, the rightmost point still appears to correspond to a higher value on the y-axis than distinct but similar models. Is this not considered the same model effect because the increase is sub-linear?\n- I was surprised there is not engagement in earlier work studying “explainability” or “truthfulness” of models. E.g. [1]\n\n[1] A Survey of the State of Explainable AI for Natural Language Processing (Danilevsky et al., AACL 2020)", "questions_to_authors": "Questions:\n1. How can we know whether the final conclusion is due to no introspection versus due to lack of discriminative power of the experiment?\n2. Because the empirical definition of similarity is measuring the exact attribute that the paper is testing whether or not a model is capable of introspecting about (literal token probabilities), would we apriori expect a nearly identical model ∆Meta_A ∼ ∆Direct_A = ∆Meta_A ∼ ∆Direct_B?", "extra_scores": {}}, {"review_id": "ThC2cxNawL", "reviewer": "Reviewer_KaWq", "summary": "This paper addresses a question of whether large language models (LLMs) possess the capacity for introspection—that is, whether they can reliably reflect on their own internal linguistic representations. The authors examine this question across 21 open-source LLMs, focusing on two domains of theoretical interest: grammaticality judgments and word prediction. In both cases, the models’ underlying linguistic knowledge is operationalized through direct measurements of string probability (direct method), providing a grounded basis for comparison. Additionally, a meta method compares the log probabilities assigned to responses to metalinguistic prompts.  \n\nThe study proposes a metric for introspective ability as the degree to which the model's prompted responses predict its own string probabilities, compared to what would be predicted by another model with nearly identical internal knowledge. The method was evaluated on 670 minimal pairs from BLiMP and 378 minimal pairs from Linguistic Inquiry for grammaticality judgement and 4 datasets with 1000 items each for the word prediction.  However, because the introspective ability metric depends on the difference between one model’s response and that of a highly similar model, a much smaller subset of minimal pairs was eventually analyzed for the grammaticality judgement, where over 5% model disagreement yielded only 294 minimal pairs.  \n\nAlthough both direct and meta approaches yield high task accuracy, the authors do not find evidence that LLMs have privileged self-access to their internal states.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-written paper with a clear methodological approach, a solid theoretical foundation, and well-articulated results. The significance of the findings is thoughtfully discussed, raising important questions about whether language models genuinely access their internal states and to what extent prompted responses reflect underlying linguistic knowledge.", "reasons_to_reject": "no major reasons found", "questions_to_authors": "Model agreement resulted in a substantial reduction in the number of minimal pairs. Could further analysis reveal patterns in the types of data on which the models tended to agree and whether similar types occurred in the data analyzed for introspection?", "extra_scores": {}}, {"review_id": "9kAY2RVEcT", "reviewer": "Reviewer_PXdC", "summary": "This work revisits the debate about whether LLMs can demonstrate awareness of their own linguistic knowledge. \n\nThe experimental methodology centers on the observation that a model's internal linguistic knowledge can be grounded in computing the appropriate string probability. A model's preference can be either computed directly or elicited via a meta-linguistics prompt. Then, pairs of models can be compared by seeing how they align in the difference between the direct and meta-linguistics measurements. The experimental set up aims to overcome some of the limitations in earlier studies by explicitly controlling for similarities between models.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiment is cleverly designed. While the evaluation is limited to grammatical correctness and sentence completion, the domain choices made it possible to know the ground truth by computing string probabilities. This way, the work is able to compare many models (21 open-source LLMs) in a controlled fashion. \n\nThe results of the study are interesting and thought-provoking. Under the proposed experimental methodology, the results do not provide evidences that support model introspection, which contradicts some prior work.", "reasons_to_reject": "As the authors pointed out themselves -- that their proposed methodology did not provide evidence for introspection does not necessarily mean that a model cannot introspect. \n\nThe notion of introspection as determined through comparing \"direct\" and \"meta\" is not very intuitive for people who are not used to thinking about this problem. It would be nice to have a stronger motivation for why the \"meta\" method is accessing the same internal information as \"direct.\"", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4CO0Gy7Mz", "reviewer": "Reviewer_n8wV", "summary": "Significance:\n- This paper tackles an interesting question of whether models have privileged \"Self-access\" to their internal states. If so, this could help e.g. interpretability of language models.\n- The paper finds instead that there is no evidence for this, in language models not fine-tuned for introspection.\n\n\nClarity:\n- I found this paper difficult to read. One problem is that Figure 1 contains 4 different figures that are referenced across the paper. It is different to understand Figure 1 as a whole. When the other parts of the paper refer to Figure 1, it is not clear which specific figure is being referred to. For example, one of the later pages references a part of the Figure 1 as \"Figure 1c, bottom\". This is difficult for readers to locate.\n- Suggestion: I think the readability of the paper would be much improved if the authors would break up Figure 1 into separate figures, and introduced them alongside the relevant parts of the paper. Add further elaboration on the respective broken down parts of the figure in the captions.\n\n\nOriginality:\n- The paper is original in investigating further the question of self-access. Not much work has been done in this area. The paper claims to complicate recent results suggesting that models can introspect.\n- Still, I question whether this paper makes an original contribution, and whether it complicates prior work. (Elaborated later)\n\n\nQuality:\n- The authors conduct experiments across 21 different models to check for introspection. They conduct two main experiments. One with introspection about grammar, and another with introspection about next word prediction.\n- Due to the questions I have about originality, and the questions about the execution of the experiments, I currently do not find the paper to be of high quality.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This submission examines multiple models to check for introspection. The claim that \"If models are not finetuned for introspection, they will fail to introspect\" seems substantiated by their two experiments.", "reasons_to_reject": "### On contribution and whether it complicates prior work\nI question whether this submission makes an original contribution and whether it complicates prior work.\n- This submission investigates a scenario where models are not fine-tuned for introspective tasks. It then concludes that there is no evidence for introspection in these tasks. The authors say that this result challenges prior work, such as Binder et al. (2024).\n- But, examining Binder et al. (2024), they already state the same conclusion when models are not fine-tuned. In Binder et al. (2024), they show that models perform at merely baseline levels unless they are fine-tuned for introspection (Binder et al., 2024, Figure 4).\n- This submission acknowledges this fact by saying they do not conduct fine-tuning experiments, unlike Binder et al.\n- If so, then what exactly is the contribution of this paper? This submission only investigates scenarios without fine-tuning, which prior work cited has already shown to not have evidence for introspection.\n- Please clarify -- What prior work suggests that introspection would work without dedicated fine-tuning? Pointing out an exact paper to look at would be good. How does this submission \"complicate\" results?\n- I suggest the authors clarify their exact contribution.\n\n\nI can think of two ways forward: \n- Refute my claim about what the cited work shows. Show that the cited or other prior work infact claims that models can introspect without dedicated fine-tuning.\n- Do fine-tuning for introspection using your setup. Show that there is still no self-access advantage. Binder et al. claims that a large model Llama 70b can introspect, so maybe you can compare directly there? Show that even though you are comparing the same model, you get different results.\n\nNote: They also speculate that weaker models like GPT-3.5 fail to introspect,  so introspection on small models will be expected to fail (Binder et al., Figure 4). One problem is that there are no multiple seeds for Llama 70b, but only for small models 7,13B for OLMo-2.\n\n\n### On methodology\nThe authors state that they \"focused on the items for which ≥5% of the models disagree under the Direct\n196 and Meta evaluations. We did this because there is limited potential to observe introspection\n197 for items which are so easy that all models and methods provide the same answer.\".\n\nI agree with this premise. However, just greater than 5% is not a very stringent threshold. Doesn't that mean that they require only 2 models out of the 21 models to disagree on a certain sample for it to be considered for analysis? Perhaps, when comparing two models A and B, it would be better to instead filter only for examples where the models A and B disagree?\nThis filtering follows prior work (Kadavath et al., 2022, Figure 20, Page 21), which looks at cross-experiments on similar models A and B.\n\nKadavath et al. (2022).  Language Models (Mostly) Know What They Know. https://arxiv.org/abs/2207.05221", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Most of the reviewers saw merit in this paper, and I agree. The experiments are empirically sound, the results interesting, and I appreciated the controls for model similarity. I do have a few concerns with the framing, which I encourage the authors to take into consideration for the camera ready if this paper is accepted. First, the idea that *humans* have reliable access to their own linguistic knowledge when they provide an acceptability judgment or complete a sentence in the cloze task, without any interference from task demands, seems simplistic. How strongly can be expect the implicit and explicit measures to be correlated in LLMs? Second, the paper sometimes appears to be about introspection more generally (meta-cognition), but in fact is focused on two particular aspects of the knowledge of language. Third, the contrast with Binder et al. (2025) is not sufficiently clear; related to the previous point, it seems perfectly plausible for models to have metacognitive access to some computations but not others. Because Binder's work is so prominent in the framing, I’d expect some more details on what that paper found and how inconsistent that is with the current results. The discussion with reviewer n8wV includes some useful ideas for addressing this issue. Finally, I wonder a more direct test of introspection wouldn’t be to simply ask the model something like “the probability of sentence 1 is higher than the probability of sentence 2” and compare THAT to the difference in probability assigned to those sentences by the model; as it currently stands, this test relies on the targeted syntactic evaluation paradigm providing a direct window into the model’s understanding of acceptability, which adds another assumption to the test. In general I encourage the authors to keep in mind that COLM has a broad range of audiences, so terminology, methods and approaches that are common in linguistics may need to be presented more gently.
Decision: Accept