Potemkin Understanding in Large Language Models
Authors: Marina Mancoridis
Bec Weeks
Keyon Vafa
Sendhil Mullainathan
Venue: ICML 2025 poster (2025)
Keywords: evaluation
benchmark
large language models
potemkin understanding
Primary area: deep_learning->large_language_models
TL;DR: We find that large language models exhibit a category of failure in conceptual comprehension called potemkin understanding.

Abstract: Large language models (LLMs) are regularly evaluated using benchmark datasets. But what justifies making inferences about an LLM's capabilities based on its answers to a curated set of questions? This paper first introduces a formal framework to address this question. The key is to note that the benchmarks used to test LLMs---such as AP exams---are also those used to test people. However, this raises an implication: such benchmarks are only valid tests if LLMs misunderstand concepts in ways that mirror human misunderstandings. Otherwise, success on benchmarks only demonstrates **potemkin understanding:** the illusion of understanding driven by answers irreconcilable with how any human would interpret a concept. We present two procedures for quantifying the existence of potemkins: one using a specially designed benchmark in three domains, the other using a general procedure that provides a lower-bound on their prevalence. We find that potemkins are ubiquitous across models, tasks, and domains. We also find that these failures reflect not just incorrect understanding, but deeper internal incoherence in concept representations.

Potemkin Understanding in Large Language Models
Marina Mancoridis
Bee Weeks?
Keyon Vafa"
Sendhil Mullainathan'
Abstract
What ARAB rhyming scheme?
Large language models (LLMs) are regularly eval- uated using benchmark datasets. But what justi- fies making inferences about an L.M''s capabili- ties based on its answers to curated set of ques- tions? This paper first introduces formal frame- work address this question. The key is to note that the benchmarks used to test LLMs LLMs-such AP exams are also those used to test people. However, this raises an implication: such bench- marks are only valid tests if LLMs misunderstand concepts in ways that mirror human misunder- standings. Otherwise, success on benchmarks only demonstrates potemkin understanding: the illusion of understanding driven by answers irrec- oncilable with how any human would interpret concept. We present two procedures for quan- tifying the existence of potemkins: one using a specially designed benchmark three domains, the other using a general procedure that provides lower-bound on their prevalence. We find that potemkins ubiquitous across models, tasks, and domains. We also find that these failures re- flect not just incorrect understanding, but deeper internal incoherence in concept representations.
rhymes:
third lines rhyme, second and fourth rhyme.
Correct
P
Fill
poem uses
the ABAD chyming scheme:
Wondrous cails out
Shivering under the frost Lies lonely cat, sitthig
Alone hut hordly lost
lonely sitting
Potemkin
Does out rhyme with solt?
Figure 1. Illustration of potemkin understanding a large lan- guage model. This example shows GPT--4o's failure apply its own conceptual explanation of an ABAB rhyming scheme.
vision-wee only draw inferences for its performance spe- cific distributions, inferences that are limited by distribution shift. What justifies interpreting benchmark performance this way for LLMs?
1. Introduction
There has been marked change in how we interpret ma- chine learning evaluations. Today, large language models (LLMs) are evaluated on benchmark datasets: curated ques- tions with rubrics for grading. Success on these questions treated as evidence of broader conceptual understanding. Previously, while benchmarks were also used to evaluate supervised learning models, success was interpreted more narrowly. A pathology classifier that performs well X- ray classification is not credited with an understanding of
This paper first introduces framework that justifies using benchmarks to measure concept understanding in LLMs. The central insight is simple: we already analogous tests to assess humans. AP exams, AIME math competitions, and coding challenges are credible measures precisely because humans cannot excel on these benchmarks without a true grasp of the required concepts. This is because the ways in which human might misunderstand concept are small and structured; there is no human who can do well on an AP math exam without understanding basic math concepts.
Massachuseets Institute of Technology University of Chicago 2 'Harvard University "Massachusetts Institute Technology. Cor- respondence to: Marina Mancoridis <marinam@ mit.edu>.
This framework also raises implication: that human benchmarks are only valid tests for LLMs if LLMs mis- understand concepts in the same way that humans do. LLM misunderstandings diverge from human patterns, mod- els can succeed on benchmarks without truly understanding
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Potemkin Understanding in Large Language Models
the underlying concepts. When this happens, it results in pathologies we call potemkins
Conceptual understanding in people. At a high level, a concept consists of a of rules that describe objects. For example, haiku is concept, consisting of logic that can be used to classify poems. Mcanwhile a fact like "Abraham Lincoln was born in 1809" is not a concept because it does not correspond to any set of generalizable rules.
Figure| 1 illustrates potemkin. When an LLM is asked to explain an ABAB rhyming scheme, its response is clear and correct (top panel). At first glance, it may appear that the LLM has understood the concept, in the same way that a human with the provided explanation would understand. However, when tasked to generate text in an ABAB rhyming scheme, the LLM fails, producing non-rhyming words (mid- dle panel). Strikingly, the LLM seems to recognize that its output does not rhyme (bottom panel). This specific com- bination of correct and incorrect answers is irreconcilable with any answer that a human would give.
What does mean for human to understand a concept? The full scope of possible ways to demonstrate understanding is enormous; for example, we may expect someone who understands the concept of a haiku to be able to define it, generate haikus about arbitrary topics, and categorize every poem as a haiku or otherwise. Asking people to enumerate every possible example of a concept is intractable. Instead, it typically suffices to demonstrate understanding with few specific examples. For example, if someone can accurately define haiku and provide few examples, we would have confidence that they have understood the concept of a haiku,
Potemkins occur when an LLM performs well on tasks that would indicate conceptual understanding if human completed them, but do not indicate understanding in the L.M. This paper develops two procedures for measuring the prevalence of potemkins in LLMs. The first is tailored to a specific kind of potemkin: the divide between an LLM's ability to explain a concept and apply it. We collect a bench- mark dataset across three domains literary techniques, game theory, and psychological biases designed to mea- sure the prevalence of this type of potemkins. In contrast, the second procedure is general and doesn't 't make assump- tions on the structure or domain of potemkins, but it only provides a lower-bound on their prevalence.
Why is it reasonable to infer that people have understood a concept after only seeing a few examples? The key insight is that while there exist a theoretically very large number of ways in which humans might misunderstand a concept, only a limited number of these misunderstandings occur in practice. This is because people misunderstand concepts in structured ways. For example, if person mistakenly believes that haikus follow a 5-8-5 syllabic structure, the examples of haikus they come up with will all be incorrect in the same way. This logic underlies why we use exams to test conceptual understanding in people: even though SAT and AP exams only consist of a tiny fraction of possi- ble questions about concept, the questions are structured so that conceptual understanding is necessary to achieve high scores. The space of human misunderstandings is pre- dictable and sparse.
We apply these procedures to a set of LLMs and find that potemkins are ubiquitous. For example, despite models being able to define concepts in each domain in our bench- mark dataset near-perfectly, they struggle to apply these concepts accurately. We find that potemkins are not arising due merely to incorrect understanding of concepts, but rather due to incoherence. Despite the fact that the automated pro- cedure provides only a lower bound, it still identifies high rates of potemkins across LLMs on questions from common benchmark datasets.
We formalize this notion by defining X' as the set of all strings that are relevant to a concept: for example, string can correspond
possible definition of a concept or a possible example of one. Not every string that's relevant to concept is a valid use of it. An interpretation of a concept is defined to be any function f X + {0, 1}, where the output indicates whether the string is considered valid in the interpretation (0 for invalid, 1 for valid). There is a single correct interpretation, denoted by s The set of possible ways for humans to interpret concepts is denoted by Fr.. Every function f E Fi. that is not equal to f corresponds to a way in which a human might misunderstand a concept.
The rest of the paper is structured as follows: Section 2 presents our framework for defining potemkins. Section 3 introduces a benchmark designed to measure the presence of potemkins and shows their ubiquity across key domains. Section corroborates these findings using a separate, auto- mated evaluation procedure.
2. Framework
Potemkins arise when there is a misalignment between how humans and large language models (LLMs) understand con- cepts. Here, we present a theoretical framework for defining conceptual understanding
Consider one possible way f E Fr that a human might interpret a concept. To evaluate whether a human has un- derstood the concept correctly, we don't check whether f(x) = (xx) across all strings x E X; this is intractable. Instead, we use exam questions to isolate specific instances that serve proofs of understanding. But what constitutes such proof? The answer is revealed by this framework:
'This term comes from Potemkin villages-elaborat elaborate facades built to create the illusion of substance where none actually exists.
2

---

Potemkin Understanding in Large Language Models
1. Single-element keystones
Multi-element keystones
Questions
Questions 4
4
42
4,
4,
9
4
95
Correct
Keystone: Correct answer implies correct
Correct
Only when considered together do and q, form a keystone.
interpretation
interpretation for humans.
Human
Human
Interpretations
Potemkin: gets keystone correct but
LLM
Potemkin, with keystone set of size 2.
interpretation
understand the concept.
LLM
Figure 2. schematic representation of conceptual framework. Rows represent different interpretations of concept and columns represent different questions. Questions can either be validly or invalidly classified by an interpretation. The figure provides a stylized depiction of keystones, interpretations, and potemkins.
the sets examples that are only interpreted correctly if human's interpretation correct.
æ such that f(x) # (x) as potemkin.
In other words, potemkins arise when an LLM answers keystone questions correctly but does not have the correct interpretation of a concept. The prevalence of potemkins in LLMS is consequential for benchmarking:
Formally, define keystone set S X minimal set of instances such that if E Fn and =(x)= = (x) for all E S, then f = f That is, if every example in the keystone set is interpreted correctly, it cannot reconciled with any misconceived human notion of the concept. See Figure for visual depiction of keystone set.
Result: Keystones are a valid way test LLMs if Fi Fi.
Corollary: If an L.M displays potemkin, it illustrates that keystones are invalid tests of LLM understanding.
This approach shows why testing conceptual understanding in people is feasible. Testing understanding of a concept doesn't It require testing people for all relevant examples. In- stead, only requires testing examples in a keystone set.
These results imply that benchmarks based on keystone questions for people are invalid tests for LLMs if LLMs are capable of potemkin understanding. Suppose we were to test LLM like would a human: by testing whether it correctly answers the questions in keystone set. For example, we may prompt it with the questions an AP exam and measure whether it answers them correctly. A human that performs well on these questions must have the correct interpretation f because their interpretations are limited Fr. However, an LLM that has not understood the concept may still perform well the exam Fi # Fü; that is, if it is capable of misinterpreting concept in ways that do not mirror human misinterpretations.
Potemkins invalidate LLM benchmarks. Humans are evaluated on tests that are designed highlight keystones questions that can only be answered correctly by person who has fully understood concept. Common benchmarks for evaluating LLMs include questions sourced from AP exams, SAT tests, medical examinations, and other standard- ized assessments (Hendrycks et al., 2020; Clark & Etzioni, 2016; Jin et al., 2021).
But how effective are tests that are designed for people when it comes evaluating ILMs? To answer this ques- tion, define Fi the set of ways for any possible LLM to interpret a concept, where each f E Fi is an interpretation f: 1).
We note that under the hold-out principle, performance on benchmark questions still guarantees performance on other questions drawn from same distribution; if an LLM scores 95% accuracy on an i.i.d. sample of AP questions, is expected to score 95% on a separate sample. How- ever, analogous to how the hold-out principle only holds under i.i.d. data assumptions, success keystone ques- tions only guarantees success on other relevant tasks under a "no potemkins" assumption. is on these other tasks not standardized tests - that we typically care about LLM performance in the real world.
Definition 2.1. An LLM has potemkin understanding its interpretation satisfies f(x) = (x) for all x in a keystone S, but (xx + (x) this case, we refer to any
We don't t impose a single way to translate an LLM's output into statement about correctness; for example, an LLM might be instructed to return if particular instance is incorrect and if it's correct.
3

---

Potemkin Understanding in Large Language Models
Thus, evaluate whether LLM benchmarks work, we must first determine the prevalence of potemkins. The remain- der of the paper is dedicated to developing procedures to quantify the prevalence of potemkins.
literary techniques, game theory, and psychological bi- ases. These domains together span generative linguistics, formal constructs, and human understanding. We exam- ine concepts like "analogy" in literary techniques. "Pareto optimality" in game theory, and "sunk cost fallacy"" in psy- chological biases. We explore a total of 32 distinct concepts within the domains, with a full list provided in Appendix Section B. Given the diversity of our concepts and tasks, evidence of potemkins in our analysis would suggest not an isolated issue but systemic category of failure in LLMs.
3. A Benchmark Dataset for Potemkins
This paper presents two procedures for measuring the preva- lence of potemkins in large language models (LLMs). This section describes one procedure, based on benchmark dataset we collect that measures specific kind of potemkin failure: a divide between describing and applying concepts. Specifically, we construct a dataset spanning 32 concepts from 3 distinct domains: literary techniques, game the- ory, and psychological biases, collecting 3, 159 labeled data points. We find that even when models can correctly define concept, they often fail to accurately apply it in classi- fication, generation, and editing tasks. All collected data, annotations, and analysis are made publicly available at the Potemkin Benchmark Repository.'
3.2. Benchmark construction.
We construct datasets to evaluate concept explanation and concept use in each of the three domains. To achieve this, we construct datasets for each of the three domains, varying generation and evaluation methods to enhance the robust- ness of our findings. Collecting data or validating responses from some domains requires hand-labeling, while others are automatic. To ensure high-quality annotations, we rely mix of domain experts and paper authors evaluate model responses. Our analysis spans the following mod- els: Llama-3.3 (70B), GPT-4o, Gemini-2.0 (Flash). Claude- 3.5 (Sonnet), DeepSeek- V3, DeepSeek-RR RI,, and Qwen2-VL (72B). Model names are abbreviated in subsequent tables. Below, we describe our data collection process for each of the four tasks in our evaluation framework.
3.1. Benchmark motivation
We design a benchmark that measures a specific kind of potemkin. Observe that any human who can answer a key- stone about a concept must be able to correctly use that concept in a concrete instance. This is because, by defini- tion, keystone success in humans indicates correct concept understanding. Thus, we identify potemkins in LLMs as in- stances where (1) the LLM can correctly answer keystone about concept but (2) it fails to correctly use that same concept in a concrete instance.
Definition. To assess whether LLMs can explain concepts, we prompted models to define a concept in given domain. We prompted each of 7 models to define each of 32 con- cepts, resulting in a total of 224 generated definitions across domains. We evaluated the definitions ourselves, as some of the concepts required specialized knowledge to evaluate accurately. For example, evaluating the accuracy of defini- tion for the literary techniques concept of "Shakespearean sonnet" required confirming that it accurately described the ABAB CDCD EFEF GG rhyming scheme and specified that the poem must be entirely written in iambic pentameter.
What might we use as our keystone? A common keystone for a concept is definitions: we have faith that humans who can clearly define the concept of a haiku have understood haikus. Thus, potemkin occurs when an LLM that can define a concept correctly cannot use it.
What would it mean for an LLM to be able to use a concept in a concrete instance? We consider three such tasks, each that offers a unique perspective for measuring potemkins. One task is classification: answering whether an example is correct application of concept. Another task is gener- ation: producing an instance of a concept that adheres to specific constraints. The last task we consider is editing: modifying an example so that it either belongs or doesn't 't belong to a concept. We provide more details for these tasks in Section 3.2; see Appendix Section A for a visual representation of our experiment.
Classification. Models must determine if presented exam- ples are valid instances of a given concept. The model is given an instance and asked: "Is the following example a true instance of the concept c?" For example, to assess the model's grasp of the concept of "slant rhyme", we could present pairs of words words-some that rhyme and others that do nnd ask the model to evaluate whether they qualify.
Evaluating a model's ability to classify concepts requires creating positive and negative examples. We used distinct data generation approaches per domain. For literary tech- niques, we crafted original examples and collected online examples from recent poetry competitions (ensuring they post-date each LLM's training cutoff). As the game theory domain requires specialized knowledge, we recruited Eco- nomics PhD students to produce true and false instances.
What it means to use a concept depends on the domain being tested. We choose concepts from diverse array of domains:
'tttps: poltps://ghe //gbt cwm coo//ariiaaaii /tMaaaaaaaaaaanciiii.. Potemki i BBncchaark. git
4

---

Potemkin Understanding in Large Language Models
For the psychological biases domain, we gathered 40 text responses from Reddit's "r/Amm0verreacting" thread, anno- tated by expert behavioral scientists. Models classified each example, and we compared their outputs against our labeled ground truth. Overall, we generated 2,030 annotations. For further details, see Appendix Sections C and D.
Potemkin rate, as measured by:
Model
Classify
Generate
Edit
3lama-3.3
0.57 (0.06)
0.43 (0.09)
0.36 (0.05)
Claude-3.5
0.49 (0.05)
0.23 (0.08)
0.29 (0.04)
GPT-4o
0.53 (0.05)
0.38 (0.09)
0.35 (0.05)
Gemini-2.0
0.54 (0.05)
0.41 (0.09)
0.43 (0.05)
Constrained generation. This task assesses a model's ability to use concepts by requiring to generate exam- ples adhering to specific constraints. This tests the model's capacity to flexibly apply concepts within structured bound- aries. For instance. for the concept of "strict dominance" in game theory, we might ask the model to construct an example specifying constraints like the number of players or which players have strictly dominant strategies.
DeepSeek- V3
0.57 (0.05)
0.38 (0.09)
0.36 (0.05)
DeepSeek-R1
0.47 (0.05)
0.39 (0.09)
0.52 (0.05)
Qwen2-VL
0.66 (0.06)
0.62 (0.09)
0.52 (0.05)
Overall
0.44 (0.02)
0.40 (0.03) 0.40 (0.02)
Table 1. Potemkin rate classify, generate, and edit tasks, condi- tioned on correctly defining cach concept. We scale the classifi- cation values so that chance-level performance corrsponds with potemkin rate of 1. Standard errors are in parentheses.
Performance measurement involved three steps: (I) defining constraints for each concept, (2) generating model responses, and (3) annotating responses to assess constraint adherence. Constraints were defined by the paper authors. Labels varied by domain: literature labels came from the authors, game theory labels were generated automatically, and psycho- logical biases labels were determined by majority expert consensus. Overall, 224 model responses were evaluated. See Appendix Sections E and F for detailed specifications.
chance accuracy of 0.50, we scale this value by a factor of 2, that a potemkin rate of 1 corresponds to chance-level performance.
Our findings reveal high potemkin rates across all models and domains, as summarized in Table 1. Models define concepts correctly 94.2% of the time. However, conditioned on correct definitions, their performance sharply decreases when tasked with using those concepts, as exhibited by the high potemkin rates in the table.
Editing. This task evaluates a model's ability to use con- cepts by requiring it to identify modifications that could transform an instance into either a true or false example of a given concept. For instance, we might present the model with a partially obscured haiku- -where a section of the poem is missing-and ask what could be added com- plete the poem and ensure it qualifies as true instance of haiku, To assess performance, we prompt the model with an instance to edit and evaluate its response. Our prompting and evaluation strategies were tailored to each domain. In total, we gathered 791 annotations of model responses. For further specifications, see in Appendix Section G.
While performance varies slightly across models and tasks, we find that potemkins are ubiquitous across all models, concepts, and domains that we analyzed. See Table 6 (Ap- pendix Section I) for further details of our results. Further examples of potemkins are provided in Figure 3.
Discussion. We raise two points of discussion related to our benchmark. One possible concern is that while our benchmark only uses single definition questions as part of the keystone set, in reality keystones may include multiple questions, not only about defining the concept. For exam- ple, we may not have faith that a student who defines the quadratic formula truly understands what it is until we see them apply it few times.
3.3. Results
We analyze 7 large language models across 32 concepts. These models were chosen for their popularity and range of developers and sizes. We collect inferences using APIs from OpenAI, Together.Al, Anthropic, and Google.
To address this, we conduct supplementary analysis that simulates performance when keystones contain additional questions. Specifically, for each concept, we consider key- stone sets that require not only the correct definition but also & correct responses to classification ("use") questions. We perform a simulation exercises that measures, among concepts for which LLM answers the expanded set of keystones correctly, how well it performs on other "use" questions. Specifically, we say model has "understood a concept" it answers 10 additional "use" questions after answering keystones correctly. Figure 4 shows how concept understanding varies with the size of the keystone set (only
For each (model, concept) pair, we first determine whether the model provides a correct definition. If so. we evaluate its accuracy on the three additional tasks: classification, generation, and editing. Responses are labeled as correct or incorrect according to our framework specifications.
We measure the potemkin rate exhibited by models. We define the potemkin rate of model as the proportion of questions that the model solves incorrectly when it solves a keystone correctly. Formally, we compute this as 1- task accuracy conditional on keystone success. For tasks with a
5

---

Potemkin Understanding in Large Language Models
triangle inequality theorem?
relationship an uncie and nephew?
form of poetry that consists of three lines with a total of 17 syllables arranged 5-7-5 pattern.
The sum of two sides must exceed the third side.
An undle the brother nephee''s parent.
Correct
Correct
Correct
Q
Name one length that could makee the following trangge.. Provide only the single number.
Q
Is there uncle nephew relationship in followinn family structure?
Q
In the following text, what could resultant
true Haiku?
First side:: Second side:
George is Paul's brother. Paul's wille George's mother Rehocca.
It's better for you wane Let float away
Third side: 2
Rebeccal's daughter Justine.
Yes
"Old
(incorrect)
(incorrect)
Potemkin (incorrect)
Figure 3. Examples of potemkins. In each example, GPT-4o correctly explains a concept but fails to correctly use it.
Concept Understanding by Keystons Set Size
4.1. Warmup: Incoherence
1.0
Indiviuaal models
Concept Understanding
Section 3 demonstrates that potemkin understanding is ubiq- uitous in LLMs. There are two possibilities for why this might be the case. One possibility is that the LLMs have slightly misaligned but internally consistent understanding of concepts. Another possibility is that their conceptual grasp is incoherent, with conflicting notions of the same idea.
Average
0.6
0.4
0.2
0.0
Number Usee Examples in Keystone Set (k)
To distinguish between these two cases, test specifically for conceptual incoherence within models. We measure incoherence in two steps. First, we prompt model to generate either an instance or a non-instance of a specific concept (e.g., producing an example of a slant rhyme). Then, we present the model's 's generated output back to (in a separate query), asking whether this output is indeed an instance of the concept. In the slant rhyme example, this means testing if the model recognizes its example as slant rhyme. Figure 5 summarizes this procedure.
Figure 4. Impact of expanding keystone sets on concept under- standing values. We extend the keystone from single definition to include multiple classification ("use") examples. Conditioning correct definition, incrementally require additional correct classification responses. Concept understanding values increase modestly, from 0.19 when 0 0.40 when 4 4.
including models that outperform chance). Expanding the keystone set yields only modest performance gains.
We quantify incoherence by first calculating the percentage of cases where the model's initial generation does not match its subsequent classification. Since random-chance accuracy is 0.50, we then multiply this value rescaling scores such that 0 indicates incoherence and 1 indicates as-good good-os--- random performance. See Appendix Section K for more details on our data collection process.
Another concern is whether the use tasks we consider in the benchmark are too hard such that humans would fail these as well. Qualitatively, we find this is not the case; Appendix J shows examples of failures that humans who understand the concept would not make.
The results of our analysis are presented in the first column of Table 2. We observe incoherence across all examined models, concepts, and domains, with scores ranging from 0.02 0.64. Although these scores are better than random, they nonetheless indicate substantial limitations in models'
4. Automatically Evaluating Potemkins
In this section we present a different, automated procedure for evaluating the presence of potemkins.
6

---

Potemkin Understanding in Large Language Models
Step 1
Step 2
Model
Incoherence
Potemkin rate (lower bound)
Generate an example of a slant rhyme.
Is the following an example slant rhyme?
Llama-3.3
0.18 (0.04)
0.82 (0.02)
Claude-3.5
0.62 (0.06)
0.36 (0.02)
GPT-4o
0.64 (0.06)
0.46 (0.06)
"Time" and "mine"
GPT-ol-mini
0.16 (0.03)
0.66 (0.02)
GPT-o33mini
0.02 (0.01)
0.66 (0.04)
Gemini-2.0
0.10 (0.02)
0.86 (0.02)
"Time" and "mine"
No
Yes
DeepSeek- V3
0.12 (0.02)
0.38 (0.02)
DeepSeek- R1
0.04 (0.02)
0.50 (0.02)
Incoherent
Coherent
Qwen2-VL.
0.12 (0.02)
0.82 (0.00)
Overall
0.22 (0.01)
0.62 (0.01)
Figure 5. Illustration of the two-step method for evaluating inco- herence in models. the first step. the model generates an instance or non-instance of a given concept. In the second step. the model evaluates whether instance it generated true or false exam- ple of the concept.
Table 2. Incoherence scores and potemkin rates across models. An incoherence score of indicates perfect performance and a score of indicates good-as-randon, performance. Potemkin rate defined as 1- accuracy, multiplied by 2 (since channe- accuracy on this task is 0.5, implying baseline potemkin rate 0.5). The automatic evaluation procedure provides lower bound on potemkin rate. Standard errors are in parentheses.
ability to consistently evaluate their own outputs. This in- dicates that conceptual misunderstandings arise not only from misconceiving concepts, but also from inconsistently using them. A detailed breakdown of incoherence scores domain is provided in Appendix Section L.
bound is that the judge may grade an incorrect response as correct, which would be an example of conceptual misun- derstanding but isn't captured by this metric. Similarly, generating questions that are unrelated to concept would also demonstrate misunderstanding. although qualitatively we do not find this to be the case.
4.2. A lower bound on potemkin rates.
Our incoherence analysis used L.M self-grading way to measure concept misunderstanding. This insight motivates another, more general automated procedure for measuring potemkins. The idea is as follows: if an LLM understands a concept, its answers to new questions about that concept should be judged as being correct by the same LLM. How often the L.MM judge disagrees with its original answers reveals the prevalence of potemkins.
We perform this analysis for the same models as above using questions sampled from MMLU (Hendrycks et al., 2020). We prompt L.LMs generate related questions for each conceptual question it answers correctly and then prompt it to respond to each question. In order to avoid sycophancy in the judge (Malmqvist, 2024) (e.g. an LLM may always grade responses as correct), we also prompt the LLM to generate a slightly incorrect response and also re-prompt the LLM to grade it (see Appendix Section M for examples of prompts). Results are reported in Table Our reported potemkin rate is scaled so that corresponds to chance accuracy and 0 corresponds perfect performance. Despite the procedure only providing a lower bound, the potemkin rate is high: 0.62. These results corroborate the benchmark findings from Section 3.
Specifically, we prompt an LLM with questions from benchmark and automatically grade whether it correct. If answers correctly, we prompt it to generate other questions that rely on the same concept. Then, for each question, we prompt the LLM answer the question correctly and then re-prompt the same LLM grade its response. Whenever the judge's is response deviates from the expected response, it indicates an example of potemkin. See Figure 6 for an example this procedure.
Importantly, this approach provides a lower-bound on the rate of potemkins. To see why, consider the scenarios in which the LLM judge grades a response as incorrect. If the LLM judge correctly grades an incorrect response incor- rect, that LLM exhibited misunderstanding in responding to the original question. If the LLM judge mistakenly grades a correct response as incorrect, it still reveals conceptual misunderstanding in the judge: a judge that understood the concept would've graded correctly. The reason it's lower
5. Related Work
Providing method to validate benchmarks. Researchers caution that high scores standard NLP benchmarks may not be informative of true understanding, highlighting that the "leaderboard mentality" overestimates perceptions of a model's capabilities (Bowman & Dahl, 2021; Mitchell, 2021; Church & Hestness. 2019; Ribeiro et al., 2020; Recht et al., 2019; Raji et al., 2021). Benchmarking raises the

---

Potemkin Understanding in Large Language Models
1. LLM answers question correctly
2.
Ask LLM to generate related
3.
Ask the same LLM to grade its answer
question and answer it correctly
two people share the same gr grailggrandeeeent but different grandparents, they are:
Is the following correct?
Generaite a related question and answer it.
Question: family reunion, you meet someane who says dad's grandpa and your mom's grandpa were brothers." What's your relationship?
A] First cousins
Question: family reunion, you meet someune who says "My dad's grandpa and your mom's 's grandpa were brothers' What's your relationship?
B] Second cousins
c) cousins once removed
D) Third coutins
Answer: would third cousins.
B) Second cousins
Answer: would third cousins.
No
Correct
Incoherent
Figure Example of a potemkin documented by our automatic evaluation procedure. In this example, the language model fails agree with its own answer for a question that it generated about the topic of second cousins.
challenge of construct validity the degree which a test measures the concept it claims to measure.
Providing a framework for evaluating depth of concept understanding. Researchers have documented numerous failure modes in large language models, including limi- tations in linguistic reasoning (Dagan et al., 2010; Bow- man et al., 2015; Dentella et al., 2024; Arkoudas, 2023; Dickinson & Meurers, 2003), common-sense problem solv- ing (Shwartz & Choi, 2020), planning (Valmeekam et al., 2023), alignment with human generalization (Vafa et al., 2024b), factual recall and knowledge representation (Meng et al., 2022; Geva et al., 2020; 2022; 2023; Tam et al., 2022; Petroni et al., 2019; Zhu et al., 2020; Mitchell et al., 2021; Yao et al., 2022), distractability (Shi et al., 2023), and coherent world-mode formation (Ivanova et al., 2024; Ha & Schmidhuber, 2018; Vafa al., 2024a). They have identified logical inconsistencies through methods includ- ing reversed claims (Saba, 2024; Berglund et al., 2023), Decision-making inconsistencies (Fluri et al., 2024), factual correctness in paraphrases (Elazar et al., 2021), composi- tional tasks (Press et al., 2022), negation handling (Hosseini et al., 2021), common human misconceptions (Lin et al., 2021), semantic equivalence (Jang et al., 2021), consistency benchmarks (Jang et al., 2022; Jang & Lukasiewicz, 2023), and Ialidation-generaatin alignment (Li et al., 2023).
Many challenges undermine the reliability of benchmarks as measures of conceptual understanding. Benchmarks are frequently inadequately developed due to undervalued data collection efforts (Gebru al., 2021; Paullada et al., 2021: Sambasivan et al., 2021). High scores may instead reflect superficial shortcuts (Geirhos al., 2020; Lapuschkin et al., 2019), while benchmarks often lack sufficient statistical power (Card et al., 2020) and aggregate metrics obscure critical failures (Wu et al., 2019). These challenges may reflect broader fallacies in measuring intelligence, such as assuming intelligence can be neatly placed on a linear scale or task difficulty uniformly gauged (Mitchell, 2021).
Efforts to improve evaluation approaches include tests of robustness to noise (Belinkov & Bisk, 2017; Rychalska et al., 2019), adversarial perturbations (Ribeiro et al., 2018; Iyyer et al., 2018), logical consistency checks (Ribeiro et al., 2019), focusing on explanations (Ribeiro et al., 2016), and behavioral evaluations through frameworks such as Check- List (Ribeiro al., 2020). There has been shift from static benchmarks to dynamic, real-world interaction-basee as- sessments, exemplified by ChatBench (Chang et al., 2025), WildChat (Zhao et al., 2024), ChatbotArena (Chiang et al., 2024), and the human-in-the-loop adversarial framework Dynabench (Kiela et al., 2021). Additionally, best practices in benchmarking have been explored through initiatives like BetterBench (Reuel et al., 2024) and comprehensive, multi- metric evaluation methods (Liang et al., 2022).
Our work highlights that models may comprehend concepts in fundamentally different ways than humans. Potemkins reveal modes of conceptual understanding that humans, by construction, cannot exhibit. Our work frames prior meth- ods of evaluating concept understanding as implicit tests for potemkins, and proposes an explicit, systematic approach to uncovering these discrepancies. Potemkins are to con- ceptual knowledge what hallucinations are to factual knowl- edge- halluciailinn fabricate false facts; potemkins fab- ricate false conceptual coherence. Yet potemkins pose a greater challenge: hallucinations can be exposed through fact-checking, but potemkins require unraveling subtle in- consistencies model's 's apparent understanding. By iden- tifying potemkins, we introduce implementable frameworks
Our paper introduces framework that clarifies when bench- marks validly measure true concept understanding in LLMs: when there are no potemkins. We provide two procedures for finding potemkins and show potemkins to be ubiquitous in our analysis. The high potemkin rates that we observe invalidate using existing benchmarks as measures of under- standing.
8

---

Potemkin Understanding in Large Language Models
to assess benchmark validity and conceptual depth, trans- forming the abstract notion of understanding into actionable tools that can be leveraged for future model improvement.
(LLMs). We document polemkin understanding as an over- looked but pervasive category of failure in LLMs. By demonstrating that models can correctly answer keystone questions without true concept comprehension, our find- ings compel a re-examination of the validity of benchmarks widely trusted in Al development. The framework and em- pirical tools we provide allow researchers and practitioners to systematically identify and address potemkins, paving the way toward the development of deeper and more reliable concept understanding.
6. Conclusion
In this paper, we introduce the phenomenon of potemkin understanding- failure mode of large language mod- els (LLMs) whereby apparent comprehension revealed by successful benchmark performance is undermined by non- human patterns of misunderstanding. We began by formaliz- ing a framework for evaluating when benchmarks designed for humans serve as valid tests for understanding in LLMs. This framework revealed a crucial dependency: benchmarks are only valid if L.Mss misunderstand concepts in the same structured ways humans do. If models deviate from human- like misunderstandings, they can still correctly answer key- stone questions without truly understanding the underlying concept, producing potemkins.
This paper presents dataset using data collected from hu- man surveys. We received an IRB review and exemption for this study.
References
Arkoudas, K. Gpt-4 can't reason. arXiv preprint arXiv:2308. 2023.
Belinkov, Y. and Bisk, Y. Synthetic and natural noise both break neural machine translation. arXiv preprint 2017.
Through two complementary empirical procedures one leveraging a novel benchmark dataset across literary tech- niques, game theory, and psychological biases, and the other employing an automated evaluation strategy- quantified the prevalence of potemkins across variety of tasks, con- cepts, domains, and models. Both procedures reveal high potemkin rates, even in models that appear highly capable by conventional benchmark standards. Our tests for incoher- ence reveal that models contain conflicting representations of the same idea.
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stick- land, A. C., Korbak, T., and Evans, O. The reversal curse: Lims trained on" a is b" fail to learn" b is a".. arXiv preprint arXiv:2309. 12288, 2023.
Bowman, S. R. and Dahl, G. E. What will it take to fix benchmarking in natural language understanding? arXiv preprint arXiv:2104.02/
2021.
Our approach has limitations that suggest avenues for further exploration. The benchmark datasets we developed, while extensive, are not exhaustive. Additional datasets spanning a broader range of concepts, domains, and abstraction lev- els could enable a more comprehensive identification of potemkins. There is also significant potential in methodolo- gies explicitly designed detect and reduce potemkin rates in language models. Systematically integrating techniques for potemkin detection and mitigation into existing model training and evaluation pipelines represents especially promising direction for future research.
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:/508 053266..6 2015.
Card, D., Henderson, P., Khandelwal, U.. Jia, R., Mahowald, K., and Jurafsky, D. With little power comes great re- sponsibility. arXiv preprint arXi::2010.0659: 2020.
Chang, S., Anderson, A., and Hofman, M. Chatbench: From static benchmarks to human-ai evaluation. arXiv preprint arXiv:2504.07/ 14, 2025.
Acknowledgments
Chiang, W.-L.,, Zheng, I.,, Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., al. Chatbot arena: An open platform for evaluat- ing 1lms by human preference. In Forty-frrs International Conference on Machine Learning, 2024.
Marina Mancoridis is supported by the MIT Presidential Graduate Fellowship. Keyon Vafa is supported by the Har- vard Data Science Initiative. We also thank Steven Ma, Cassidy Shubatt, and Charlotte Siegmann for helpful com- ments and feedback.
Church, K. W. and Hestness, J. A survey of 25 years of evaluation. Natural Language Engineering, 25(6):753- 767, 2019.
Impact Statement
Clark, P. and Etzioni, O. My computer is an honor stu- dent-but how intelligent is it? standardized tests as a measure of ai. A/ Magazine, 37(1)55-12, 2016.
Our work puts forth formalizations and benchmarks to assess concept understanding in large language models

---

Potemkin Understanding in Large Language Models
Dagan, I., Dolan, B., Magnini, B., and Roth, D. Recognizing textual entailment: Rational, evaluation and approaches- erratum. Natural Language Engineering, 16(1):005-105, 2010.
Hosseini, A., Reddy, S., Bahdanau, D., Hjelm, R. D., Sor- doni, A., and Courville, A. Understanding by understand- ing not: Modeling negation in language models. arXiv preprint arXiv:2105.03519 2021.
Dentella, V., Günther, F., Murphy, E., Marcus, G., and Leivada, E. Testing ai on language comprchension tasks reveals insensitivity to underlying meaning. Scientific Reports, 14(1):28083, 2024.
Ivanova, A. A., Sathe, A., Lipkin, B., Kumar, U., Radkani, S., Clark, T. H., Kauf, C.,, Hu, J., Pramod, R., Grand, G., et al. Elements of world knowledge (ewok): A cognition- inspired framework for evaluating basic world knowledge in language models. arXiv preprint SAMivv:445.0000 2024.
Dickinson, M. and Meurers, D. Detecting errors in part-of- speech annotation. In 10th conference of the European chapter of the association for computational linguistics, 2003.
Lyyer, M., Wieting. J., Gimpel, K., and Zettlemoyer, L. Ad- versarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804. 1804.800400000 06059, 2018.
Elazar, Y., Kassmer, N., Ravfogel, S., Ravichander, A., Hovy, E., Schützee, H., and Goldberg, Y. Measuring and improv- ing consistency in pretrained language models. Transac- lions of the Association for Computational Linguistics, 9: 1032-1031, 2021.
Jang, M., Kwon, D. S., and Lukasiewicz, T. Accurate, yet inconsistent? consistency analysis on language un- derstanding models. arXiv preprint arXiv: wwxiv:2/08.000 10000000055.. 2021.
Fluri, L.,, Paleka, D., and Tramèr, F. Evaluating superhuman models with consistency checks. In 2024 IEEE Con- ference on Secure and Trustworthy Machine Learning (SaTML), pp. 194-232. IEEE, 2024.
Jang, M., Kwon, D. S., and Lukasiewiez, T. Becel: Bench- mark for consistency evaluation of language models. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 3680-3696, 2022.
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W.,
Jang, M. E. and Lukasiewicz, T. Consistency analysis of chatgpt. arXiv preprint arXiv:2303.06227 2023.
Wallach, H., Iii, H. D., and Crawford, K. Datasheets for datasets. Communications of the ACM, 64(12):86-92 2021.
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, What disease does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Geirhos, R., Jacobsen, J..H., Michaelis, C.,, Zemel, R., Bren- del, W., Bethge, M., and Wichmann, A. Shortcut learn- ing in deep neural networks. Nature Machine Intelligence, 2(1 (111))665-677 2020.
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2 14337, 2021.
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14911 14913,, 2020.
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., and Müller, K.-R. Unmasking clever hans pre- dictors and assessing what machines really learn. Nature communications, 10(1) 1096, 2019.
Geva, M.. Caciularu, A., Wang. K. R., and Goldberg, Y. Transformer feed-forward layers build predictions by pro- moting concepts in the vocabulary space. arXiv preprint arXiv:2203. 14680, 2022.
Li, X. L., Shrivastava, V., Li, S., Hashimoto, T., and Liang, Benchmarking and improving generator- validator consistency of language models. arXiv preprint 13..0:2310.01846, 2023.
Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dis- secting recall of factual associations in auto-regressive language models. arXiv preprint AXiv:2304.14767, 2023.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., al. Holistic evaluation of language models. arXiv preprint 221 100 2022.
Ha, D. and Schmidhuber, J. World models. arXiv preprint 10122 2018.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.00 00000 2020.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falschoods. arXiv preprint arXiv:2109 009.00999878 2021.
10

---

Potemkin Understanding in Large Language Models
Malmqvist, L. Sycophaney in large language models: Causes and mitigations. arXiv preprint arrppinnnarxxi:2411155585, 2024.
Ribeiro, M. T., Guestrin, C., and Singh, S. Are red roses red? evaluating consisteney of question-answering models. In Proceedings of the 57th Annual Meeting of the Associa- for Computational Linguistics, pp. 6174-61 6184, 2019.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 15:17359-17372 2022.
Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. Beyond accuracy: Behavioral testing of nip models with checklist. arXiv preprint arXiv:2005.0411 18, 2020.
Mitchell, E., Lin, C.. Bosselut, A., Finn. C., and Man- ning, C. D. Fast model editing at scale. arXiv preprint arXiv:2110.11300 2021.
Rychalska, B., Basaj, D., Gosiewska, A., and Biecek, P. Models in the wild: On corruption robustmess of neural nlp systems. In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part III 26, pp. 235-247. Springer, 2019.
Mitchell, M. Why ai is harder than we think. arXiv preprint arXiv:2104. 12871, 2021.
Paullada, A., Raji, I. D., Bender, E. M.. Denton, E., and Hanna, A. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Pat- terns, ((11), 2021.
Saba, W. S. Lims' understanding of natural language re- vealed. arXiv preprini urXiv:2407.19680 2024.
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Par- itosh, P., and Aroyo, L. M. "everyone wants to do the model work, not the data work": Data cascades in high- stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2021.
Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. Language models as knowledge bases? arXiv preprint arXiv:1909.0000 2019.
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In Inter- national Conference on Machine Learning, pp. 31210- 31227. PMLR, 2023.
Press, O., Zhang, M., Min, S., Schmidt, L.., Smith, N. A., and Lewis, M. Measuring and narrowing the com- positionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., and Hanna, A. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2/ 1..:2//11..366 2021.
Shwartz, V. and Choi, Y. Do neural language models over- come reporting bias? In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pp. 6863-6870, 2020.
Recht, B., Roelofs, R., Schmidt, L.. and Shankar, V. Do imagenet classifiers generalize to imagenet? In Interna- tional conference on machine learning, pp. $389-5400 PMLR, 2019.
Tam, D., Mascarenhas, A., Zhang, S., Kwan, S., Bansal, M., and Raffel, C. Evaluating the factual consistency of large language models through summarization. arXiv preprint 2 2022.
Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M., and Kochenderfer, M. J. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. arXiv preprint arXiv:2411. 2024.
Vafa, K., Chen, J. Y., Kleinberg, J., Mullainathan, S., and Rambachan, A. Evaluating the world model implicit in a generative model. arXiv preprint arXiv:2406.nm 03689, 2024a.
Ribeiro, M. T., Singh, S., and Guestrin, C. " why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Vafa, K., Rambachan, A., and Mullainathan, S. Do large language models perform the way people expect? mea- suring the human generalization function. arXiv preprint MrXi:2246.0018.. 2024b.
Ribeiro, M. T., Singh, S., and Guestrin, C. Semantically cquivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics (volume 1: long papers), pp. 856 8666865, 2018
Valmeekam, K., Marquez, M.. Sreedharan, S., and Kamb- hampati, S. On the planning abilities of large language models-a critical investigation. Advances in Neural Infor- mation Processing Systems, 36.75993-76005, 2023.

---

Potemkin Understanding in Large Language Models
Wu, T., Ribeiro, M. T., Heer, and Weld, D. S. Errudite: Scalable, reproducible, and testable error analysis. In Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 747-763, 2019.
Yao, Y., Huang, S., Dong, L., Wei, F., Chen, H., and Zhang, N. Kformer: Knowledge injection in transformer feed- forward layers. In CCF International Conference Natural Language Processing and Chinese Computing, pp. 131-143. Springer, 2022.
Zhao, W., Ren, X., Hessel, Cardie, C., Choi, Y., and Deng, Y. Wildchat: Im chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024.
Zhu, C., Rawat, A. S., Zaheer, M.. Bhojanapalli, S., Li, D.,
Yu, F,, and Kumar, S. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
12

---

Potemkin Understanding in Large Language Models
A. Illustration of the Benchmark Design
Figure provides a schematic representation of our experimental framework, with examples from each our three domains. The yellow examples correspond the concept of haiku, in the literary techniques domain. The green examples correspond the concept of mixed strategy Nash Equilibrium, in the game theory domain. Finally, the gray examples correspond the concept of sunk cost fallacy, in the psychological biases domain.
Concept Explanation
Concept Use
The models must define, name, or identify the shared structure that constitutes a concept.
The models must apply concepts specifie problems instonces.
Classify instances a concept.
Generate constrained examples of concept.
Edit instances of concept.
Provide on example a halku about friendship that uses the word "shield".
Define the concept.
What could replace the blank in the following poem to make it huiku?
the following a halku?
What is a hoiku?
Provide an example of a mixed strategy Nash Equilibrium where Player 1 takes actions and with probabilities 0.3 and 0.7.
What could be changed about the following exomple make it instance of mixed strategy Nash Equilibrium?
Is the following mixed strategy Nash equilibrium?
Whot is a mixed strategy Nash Equilibrium?
Is the following an example of the sunk cost fallocy?
Generate example that exhibits the sunk cost fallacy but not the catastrophizing personalization biases.
What line of dialogue that writer could have said in their make it obvious that they were exhibiting the sunk cost fallacy?
Whot is the sunk cost fallacy?
Figure Our framework for evaluating concept understanding, including examples. evaluate concept explanation by asking the model to define the concept. We measure concept use by giving the model classification, constrained generation, and edit tasks.
13

---

Potemkin Understanding in Large Language Models
B. Concept Choices
Within each domain, we selected a diverse set concepts for our main analysis, totaling 32 concepts: from literary techniques, from game theory, and 11 from psychological biases. Table 3 lists all concepts with abbreviated definitions.
Domain
Concept
Definition
Haiku
A 3-line poem with a 5-7-5 syllable structure
Shakespearean Sonnet
A 14-line poem with ABABCDCDDEFERGG rhyme scheme, written in iambic pentameter
Analogy
Comparison highlighting similarities between two things
Paradox
seemingly contradictory but true statement
Literary
Anacoluthon
Sudden break grammatical structure for effect
Techniques
Asyndeton
Omission of conjunctions for concise effect Reordering words for emphasis
Hyperbaton
Synesis
Agreement of meaning over grammar
Accismus
Feigned refusal of something desired
Slant Rhyme
Near rhyme with similar but not identical sounds argument with an implicd premise
Enthymeme
Anapest
Metrical foot with unstressed and stressed syllable
Striet Dominance
A strategy always better than others, regardless of opponents Successive elimination of dominated strategies
Iterated Dominance
Weak Dominance
A strategy at least as good as others and better in some cases Players choose specific strategies with no incentive to deviate Players randomize strategies with no incentive deviate No player can be better off without making another off Choosing your payoff- wwoff-maximizzing response others' actions
Game
Pure Nash Equilibrium
Theory
Mixed Strategy Nash Equilibrium
Pareto Optimality
Best Response
Zero-Sum Game
One player's gain equals another's loss
Symmetric Game
Payoffs depend on strategies, not players
Fundamental Attribution Error
Explaining others' behavior by personality, not situation
Black White Thinking
Viewing situations in extreme, all-or-nothing terms
Sunk Cost Fallacy
Irrationally continuing an endeavor
past investments
IKEA Effect
Valuing something more because you assembled yourself Treating conditional outcomes as certain
Psychological
Pseudocertainty Effect
Biases
Endowment Effect
Valuing something more just because you own it Assuming others' motives are more self-interested than your own
Naive Cynicism
Normalcy Bias
Underestimating risk by assuming things will continue as usual Overestimating how much others notice you
Spotlight Effect
Illusory Superiority
Overestimating your abilities compared others
Catastrophizing
Assuming the worst possible outcome will occur
Table Concepts and their definitions grouped by domain.
14

---

Potemkin Understanding in Large Language Models
C. Specifications of the Classification Task
After assembling the dataset of true and false instances each concept, we automated the evaluation of model classifications by directly comparing the model's 's outputs to our labeled ground truth. Across all of our models and domains, we collected total of 2,030 annotations. These annotations were collected follows:
1. Literary techniques. We evaluate 7 models across 12 concepts. For each concept, we consider 10 examples- 5 true instances and false instances. This results total of 840 annotations.
2. Game theory. We evaluate 7 models across 9 concepts. For each concept, we consider 10 examples - true instances and 5 false instances. This results in a total of 630 annotations.
3. Psychological biases. We gathered 40 examples from posts the subreddit h/AmIOverreattiiii....mmmm ing, each annotated for the presence of (from list of 11) psychological biases. Labels were determined by majority vote among trained psychologists recruited via Upwork. Each psychologist evaluated 10 posts, providing labels for 2 biases per post. For more details on the survey design, see Appendix Section D. Every (post, bias) pair was labeled by psychologists. and we retained the majority label. Subsequently, we asked each of 7 models to classify each post for both biases. This results in a total 560 annotations.
For our analysis, we exclude responses missing the specified format (e.g., missing strings like "ANSWER: ').
15

---

Potemkin Understanding in Large Language Models
D. Classification Task: Survey for Expert Annotation of Psychological Biases
To establish reliable ground-truth labels for psychological biases, we conducted an expert annotation survey with psycholo- gists recruited via Upwork. Experts were shown posts from the subreddit w/wmIvverreeactte whether each of two psychological biases was present each post. Full specifications of the annotation collection process can be found in Appendix Section C. Figure 8 shows example screen from the expert annotation survey.
and asked determine
Post
Read following post from
My(19M) pirtriend(19F) very private person.
anything
social media, and usually she lands be very secrative, like found that she kind of drives conversations towards talking about really
my interests. But
leam this everyone except for her best friend Marie, Above all else, she super super
about her anything, she always lakes her phone with her, she has one of those advanced passcodes, and she always turns her phone away from my and mental torture, as don't know what's going on, she even likes me, keep thinking I'm good enough her, and just feels weird that she's cautious around me.
she's
showed any har photos
general.
Bias
Consider following bias: Naive Cynicism.
Here the definition: cognitive bias where individuals assume that others' driven primarily selfish motives, agendas, while considering themselves relativaly objective and unblased.
post exhibit
following blas: Spotlight
Here definition: cognttive much others notice or pay attention to their sppearance, behavior, or actions, balleving they are under a 'spotligtt.
individuals overastimate
Does the author the post exhibit the blas defined above?
Yes
No
Figure 8. Example screen from expert annotation survey used label psychological biases.
16

---

Potemkin Understanding in Large Language Models
E. Specifications of the Constrained Generation Task
E.1. Literary Techniques
In the domain of literary techniques, we required model outputs to adhere to three types of constraints. The first constraint is semantic: the example must involve theme such as "gratitude" or "friendship." The second constraint form: the example must comply with linguistic constraint like using dialogue starting with monosyllabic word. The last constraint is lexical: the example must incorporate specific word, such "gravity" or "wake." Each prompt was constructed by randomly selecting one constraint from each category. For example, one prompt might be: "Write example involving friendship. The example must use dialogue and include the word 'gravity"".
To generate these constraints, we prompted GPT-4o to produce a list of 20 constraints for each of the three categories, totaling unique constraints. The full list constraints is shown in Table 4. We consider models and 12 concepts. In our analysis, the paper authors annotate one responses for each (concept, model) pair, resulting 84 annofations.
Semantic
Form
Lexical
Nature
Start with monosyllabic word.
Father
Childhood memories
Start with word that is not monosyllabic. Start with word that begins with the letter "M". Start with a word that begins with the letter T Use alliteration at least once. End with a one-vowel word. Use dialogue.
Red
A significant event
Order
Love
Press
Happiness
Jump
Friendship
Mother
Overcoming adversity
Run
Gratitude
Do not use dialogue.
Gravity
Change
Do not use any adjectives.
Shine
Hope
Do not use any adverbs.
Wonder
Moments in time
Make every sentence start with verb.
Just
A meaningful object
Use the first-person perspective. Use the second-person Use the third third-person perspective. Include sentence that contains question. Include repeated phrase.
Justice
A meaningful person
person perspective.
King
The carth
Earth
A hobby
Sunshine
Your dreams
Light
Loss
Have at least two sentences lines that start with the same word.
Wake
Your fears
Use comma in the first sentence or line.
Friend
Feeling lost
Do not use any pronouns.
Nectar
Anger
Make the result in the past tense.
Love
Table A full list of constraints used for the literary techniques domain the constrained generation task. We generated model prompts by independently sampling one random constraint from each column.
17

---

Potemkin Understanding in Large Language Models
E.2. Game Theory
In the domain of game theory, the constraints differed depending on the concept being tested. For instance, for the concept of "strict dominance," constraint could specify whether one both players must have a strictly dominant strategy in the example. Similarly, for the concept of "Pareto optimality," constraint might require that all numbers in the payoff matrix be unique. We considered 7 models and 9 concepts. We generated one response for each (concept, model) pair, resulting in 63 annotations. All annotations were generated by custom, automated evaluators.
Concept Group
Constraint Prompt
Strict Dominance
There is x strategy for (one/both) players. the (row/column) that is r for Player (1/2) receives a payoff of a if the other player takes the corresponding actions. In the payoff matrix, all numbers must be unique.
Weak Dominance
Pure Strategy Nash Equilibrium
Iterated Dominance
Start with an a x b matrix and end with a c x matrix.
Zero-Sum Game
If Player (1/2) takes the f action, they should receive payoff of (a, b, c) if the other player takes the corresponding actions.
In the payoff matrix, all numbers must be unique.
Mixed Strategy Nash Equilibrium
In the mixed strategy Nash equilibrium, Player (1/2) should take Action 1 with probability of p and Action with probability of -pp
Pareto Optimality
You must generate a 3x3 matrix where there are exactly distinct Pareto optimal solutions.
the payoff matrix, all numbers must be unique.
Best Response
In the game, imagine Player chooses the first action with probability of Pt, the second action with probability of p2, and the third action with a probability of ps. Player 2's best response to this action should be the f action.
Symmetric Game
In the payoff matrix (combined for both players), all numbers must be unique, regardless of which Player the payoff associated with.
Table 5. A full list of constraints used for the game theory domain in the constrained generation task. We generated model prompts differently for each concept group, applying all respective constraint prompts. When generating prompts, we randomly chose between items in parentheses, replaced items with versions of the concept name, probabilities with single-poing point decimal values between 0 and 1, values a through with integers E [1, 9j, value with an integer E [1,5]. action f E {1st, 2nd, 3rd]. Note that if applicable, the uniqueness constraints are present in each generation prompt with probability of 4.
E.3. Psychological Biases
In the psychological biases domain, models were tasked with generating examples illustrating two specified biases while explicitly avoiding a third, randomly selected bias. Six expert psychologists, recruited via Upwork, annotated the resulting model responses using a Qualtrics survey. Each expert evaluated responses from all 7 models across 2 concepts per model. For more details on the Qualtrics survey, see Appendix Section F. We obtained one response for each combination of 11 concepts and 7 models, resulting in 77 annotations.
18

---

Potemkin Understanding in Large Language Models
F. Generation Task: Survey for Expert Annotation of Psychological Biases
To establish reliable ground-truth labels for psychological biases, we conducted an expert annotation survey with professional psychologists recruited via Upwork. Experts were shown inferences where models were tasked with generating examples illustrating two specified biases while explicitly avoiding third, randomly selected bias. Full specifications of the annotation collection process can be found in Appendix Section E. Figure 9 shows example screen from the expert annotation survey.
Example 2/22
definitions peychological Read carefully.
Illusory Superiority: cognitive bias which individuals overestimate own abilitees stalltees, or performance relative to others, belleving they are better than average.
IKEA Effect: The IKEA effect is cognitive bias in which individuals place higher objects they partially -oor example, valuing furniture you've personally assembled more highly than pre-assembled fumiture,
themselves
Pseudocertainty Effect:
The peeudocartainty effect is a cognitive bias where certain than actually often
individuals perceive siluation leading to overconfidence in their predictions decialons. This biss can result in an underesämation risks and an overreliance on perceived certainties, when evidence suggests
Question given to the model (pay atlention to which biases should and should not be presenti::
Generale an example that demonsirates the Illusory Superionity and IKEA Effect psychological blases, that does show Pseudocartainty Effect
paychological bles.
Answer by the model:
"*Ccenario Building Booksheil" "Background." Sarah friends who both recently baught same DIY bookshelf from IKEA, They decide assemble their bookshelves over the weekend. *Illusstratoo of Illusory Superiority."
building bookshelf, assembly skills are superiar the average person's. ss She believes she completed the task more efficiently and with better quality than others would. that didr't
despite having demonstratee the illusory superiority bies, where individuals overestimate their qualilies and abilitie compared others. "Illustration of IKEA Effect: not particularly handy procese challenging. Despite his struggles, once he finishes building the bookshelf, he feels strong booksheif more highy than an identical one bought bre-assembled because he invested effort into building himself. This an example of the IKEA effact, people ascribe create. "Absence Pseudacertainty Effect"" In this scenario, there no psoudocertainty effect present, as there are no decisions being framed in conditional probebilities
the
assembly
pride accomplishment. He values
they partially
potentially bissed The story focuses purely on the biases related to skill perception and effort valuation, without involving decision-makie under uncertainty.
Is the model's response correct, salishing of colenia in the questior?
Figure Example sereen from the survey about evaluating model performance on our constrained generation task.
19

---

Potemkin Understanding in Large Language Models
G. Specifications of the Editing Task
In the domain of literary techniques, we created "masked" versions of each example by removing key words or phrases necessary for satisfying each concept. For instance, masked example of an "analogy" might omit one of the compared terms. Each model was tasked with editing these masked examples to restore them as either positive or negative instances. Model edits were evaluated by the paper authors, certain concepts required specialized knowledge and adherence to specific constraints. For example, a valid Shakespearean sonnet must consist of 14 lines in iambic pentameter following an ABABCDCDEFEFGG rhyme scheme. We evaluated edits by 7 models across 12 concepts, resulting in 84 annotations.
In the domain of game theory, we presented each model with examples from our dataset and asked them to suggest modifications that would convert positive examples into negative ones or vice versa. For instance, we might provide a scenario that does not qualify as a mixed strategy Nash equilibrium and prompt the model identify changes that would make it qualify. Model responses were automatically labeled as valid or invalid instances of the respective concepts using fully-automated, custom evaluation functions. We automatically validated model responses 10 times for each (model, concept). Across 7 models and 9 concepts, this yielded total of 630 annotations.
In the domain of psychological biases, models were prompted to generate single line of dialogue that, when added to a Reddit post from our dataset, would either introduce a specific bias it was originally absent or remove it if originally present. Ground-truth labels were determined by majority expert annotation as described in Appendix Sections C and D. For instance, if post clearly exhibited catastrophizing, models were asked provide line negating this bias. Six expert psychologists, recruited via Upwork, annotated the resulting model responses using Qualtrics survey (see Appendix Section H). Each expert evaluated responses from all 7 models across 2 concepts per model, All 11 psychological biases were evaluated across all models, resulting in 77 annotations.
20

---

Potemkin Understanding in Large Language Models
H. Editing Task: Survey for Expert Annotation of Psychological Biases
To establish reliable ground-truth labels for psychological biases, we conducted an expert annotation survey with professional psychologists recruited via Upwork. Experts were shown inferences where models were tasked with generating a single line of dialogue that, when added Reddit post from our dataset, would either introduce a specific bias it was originally absent remove it if originally present. Full specifications of the annotation collection process can be found in Appendix Section G. Figure 10 shows example screen from the expert annotation survey.
Example 15/22
Here a definition the psychological bias in question.
Naive Cynicism: A cognitive blas where individuals assume that athers! actions bellafs are driven primartly by selfish motives, biases, or hidden agendas, considering themselves relelively objective
Question given to the model (note whether it aska for the bias to be present absent):
What one line of dialogue that the writer could have included in their originel blog to make abvious that concept Naive Cynicism not present the following text? Include only the dialogue.
chill right. did pilates decompress my boyfriend decides go on hour-long about how much he hates rechel zegler - like, actuel anger, showing me videos, replaying clips, all it because apparently... her facial expressions
just trying to
annoying. hersolf," "too smug. "laughs too loud," "thinks smart," the biggest sin of all she said the original anow while was scary when ahe kid.
and according to him, that's just so disreepectful bacause snow white "sacreed and 'ouu don't talk about classice like that." dude, she said was
and the whole lime i'm just sitting there like... ok this feels off. like this is about rachel zegler. who's as confident, political, expressive, and just... speeks her mind without apologizing existing.
how uncomfortable you with woman
and THEN he hits me with "wall if that's what feminism looks likee, no wonder people turned off it." like it's her make feminism cute and likable for her being a person.
personality
campaign just...
best part
actually
I'm only defending her because she's feminist and that deep down must find her annoying too. like??? for having own opinion guess??
idk. feel drained. the convo stressed me out more than my actual workout. and can't shake the feeling that can't handle women who take themselves to saying that, he makes about her smili or her laugh or how "she's not doing femihism right"
space
am ovarreacting? or this just the most textbook patriarchal nonsense ever??
Answer given by the model:
"Maybe he's 'ss just having bad day, and it's not that deep."
Is the model's reeponse correct?
Yes
No
Figure 10. Example screen the survey about evaluating model performance on our edit task.
21

---

Potemkin Understanding in Large Language Models
I. Full Results from Benchmark Dataset Table
Table lists the results of our benchmark dataset, subdivided by domain. Conditioned on a correct definition for each (model, concept) pair, we evaluate accuracy levels in classification, generation, and editing tasks.
Domain
Model
Classify
Generate
Edit
Literary Techniques
Llama-3.3
0.64 (0.09)
0.64 (0.15)
0.27 (0.13)
Claude-3.5
0.47 (0.08)
0.42 (0.14)
0.33 (0.14)
GPT-4o
0.49 (0.08)
0.25 (0.13)
0.25 (0.13)
Gemini-2.0
0.62 (0.09)
0.17 (0.11)
0.25 (0.13)
DeepSeek-V
0.56 (0.09)
0.25 (0.13)
0.25 (0.13)
DeepSeek R1
0.40 (0.08)
0.42 (0.14)
0.58 (0.14)
Qwen2-VI. VI.
0.65 (0.11)
0.67 (0.16)
0.44 (0.17)
Game Theory
Llama-3.3
0.47 (0.09)
0.56 (0.17)
0.39 (0.05)
Claude-3.5
0.42 (0.09)
0.22 (0.14)
0.31 (0.05)
GPT-4o
0.49 (0.09)
0.78 (0.14)
0.36 (0.05)
Gemini-2.0
0.36 (0.08)
1.00 (0.00)
(0.05)
DeepSeek- VV
0.47 (0.09)
0.78 (0.14)
0..0
DeepSeek-RI
0.18 (0.07)
0.63 (0.17)
0.50 (0.06)
Qwen2-VL
0.53 (0.09)
1.00 (0.00)
0.51 (0.05)
Psychological Biases
Llama-3.3
0.62 (0.11)
00.0
0.20 (0.13)
Claude-3.5
0.62 (0.11)
0.00 (0.00)
0.00 (0.00)
GPT-4o
0.62 (0.11)
0.18 (0.12)
0.36
Gemini-2.0
0.65 (0.11)
0.18 (0.12)
0.09 (0.09)
DeepSeek V3
0.70 (0.11)
0.18 (0.12)
0.18 (0.12)
DeepSeek-RR R1
0.82 (0.12)
0.18 (0.12)
0.55 (0.15)
Qwen2-VL
0.82 (0.12)
0.27 (0.13)
0.64 (0.15)
Overall
Overall
0.55 (0.02)
0.40 (0.03)
0.40 (0.02)
Table Potemkin rate on "use" tasks in our three selected domains, conditioned on correctly defining each concept. Since random-chance accuracy on classification 0.5 (implying baseline potemkin rate 0.5), we multiply classification values by to rescale them to the same range the other tasks. Standard errors are in parentheses.
22

---

Potemkin Understanding in Large Language Models
J. Qualitative Examples of Potemkins
this section, we present qualitative examples of potemkins from various models, specifically selected to illustrate mistakes humans typically would not make. The examples contain abbreviated versions model responses.
Generate slant rhyme.
What could fill blank so that the result is an anapest? Further
Definition:
The end words sound similar
Definition: A metrical foot containing three syllables in the form duh-duh-DUM
but do not fully rhyme
Mistral-
GPT-4o
"Glow" and "leather"
"Than before"
(b)
Make the example not symmetric game
What could the writer have included to make the Spotlight Effect present?
Definition:
Payoffs the same for each
Definition: Individuals overestimate the extent to which they are noticed
player, regardless of player identity
LLaMMA333
(2.2)
(2,1)
"Maybe just feel like all eyes are on me when I'm talking even though know they probably aren't."
(1.2)
(1.1)
(c)
(d)
What could the writer have included to make the Sunk Cost Fallacy present?
Fill in the blank so the result could be the first
line of haiku."
man"
Continuing an effort solely due to prior investments despite poor prospects.
Definition:
Definition: The first line of a haiku has syllables
QwenZ-VL.
"I don't want to waste any more time arguing about this."
GPT-40
"Old"
(e)
Figure 11. Selected qualitative examples of potemkins across models and domains.
23

---

Potemkin Understanding in Large Language Models
K. Incoherence Data Collection Details
For our incoherence analysis, we evaluate models using the concepts from our three selected domains. Specifically, we prompt each model to generate 5 true and false instances for each of 32 concepts. This results in total of 2, 880 labeled examples: 080 from the domain of literary techniques, 810 from game theory, and 990 from psychological biases.
Our analysis spans the following 9 models: Llama-3.3 (70B), GPT-4o, GPT-ol-mini, GPT-o3-mini, Gemini-2.0 (Flash), Claude-3.5 (Sonnet), DeepSeek-V3, V3, DeepSeek-RI, and Qwen2-V1.(72B).
L. Incoherence Scores by Domain
Table 7 shows the incoherence scores of the models, broken down by domain.
Model
Literary Techniques
Game Theory
Psychological Biases
Overall
Llama-3.3
0.32 (0.06)
0.24 (0.06)
0.00 (0.00)
0.18 (0.04)
Claude-3.5
0.44 (0.08)
1.04 (0.10)
0.44 (0.08)
0.62 (0.06)
GPT-4o
0.70 (0.08)
0.88 (0.10)
0.38 (0.08)
0.64 (0.06)
GPT-ol-miin
0.28 (0.06)
0.09 (0.04)
0.07 (0.04)
0.16 (0.03)
GPT-o3-mini
0.05 (0.03)
0.02 (0.02)
0.00 (0.00)
0.02 (0.01)
Gemini-2.0
0.12 (0.04)
0.16 (0.06)
0,02 (0.02)
0.10 (0.02)
DeepSeek-V2 V3
0.16 (0.06)
0.18 (0.06)
0.04 (0.02)
0.12 (0.02)
DeepSeek-R1
0.04 (0.02)
0.08 (0.04)
0,00 (0.00)
0.04 (0.02)
Qwen2-VL.
0.16 (0.06)
0.06 (0.04)
0.12 (0.04)
0.12 (0.02)
Overall
0.25 (0.02)
0.31 (0.03)
0.12 (0.02)
0.22 (0.01)
Table Incoherence scores across domains and models. A score of indicates perfect performance and of indicates good-as- random performance, Standard errors are in parentheses.
24

---

Potemkin Understanding in Large Language Models
M. Prompts for Automatic Evaluation Procedure
In this section, we will the prompts given to models during the various stages of the automatic evaluation procedure. The following prompt used for models to generate questions.
The following is question about the following concept: (concept).
Here is the question: (question).
Write aaum_subquestions other questions that test whether someone who understands the concepts the question is testing truly understands them.
The following prompts are used for models to generate correct and incorrect answers, respectively.
You are an expert tutor. You may think step-by-step
reach the solution, but you MUST finish with an answer that starts exactly with ((FIAALTTA)) followed by your single best answer, If the question asks you to explain your answor, you should also include the explanation after (FINAL_TAG]]
Question: (question)
Modify the fellewing answer intreduce subtle error. The should be subtle but one such that a human who knows the concept would know the answer incorrect. the answer is already incorrect, you can leave it the same. You can reason all you' d like, but end the response with (FINAL_ IAG) followed by the full modified answer.
Question: (question) Answer: (initial_answer)
Finally, the following prompt is used for models to judge answers.
You are an expert tutor. You will be given a question and a possible answer to the question. Your job is to determine if the answer is correct or incorrect. You should only grade it correct if the answer (including the reasoning) You can reason all you'd like, but end the response with [PTNAI_TAG]) followed by either 'correct' or 'incorzect', and nothing should come after that.
completely correct.
Question: (question)
Answer:
25


Reviews:
[{"review_id": "uH6UT0ZKQI", "reviewer": "Reviewer_TzWx", "summary": "This paper introduces and systematically investigates a novel failure mode in large language models (LLMs), termed Potemkin Understanding. This phenomenon refers to the model's ability to correctly interpret a concept while demonstrating inconsistent or incorrect behavior in practical applications, akin to creating an \"illusion\" of comprehension. The researchers constructed specialized benchmark tests to compare the model's ability to explain concepts (definition tasks) with its ability to apply them (classification, constrained generation, and editing tasks), thereby quantifying this phenomenon. Experiments were conducted across three distinct domains (literary techniques, game theory, and cognitive biases), involving 32 concepts and 5,986 data points. The results reveal that while LLMs achieve an accuracy of 97.7% in explaining concepts, their accuracy drops to 67.9% when applying these concepts. Furthermore, the study highlights that existing mainstream benchmarks, such as MMLU, fail to effectively assess the impact of Potemkin Understanding.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We’re glad you found our work novel, rigorous, and empirically interesting. We respond to your comments and describe new results; to summarize the main changes, we've added:\n- An expansion of the coherence analysis to include 9 models\n- Analysis of difficulty \n- Proposed solutions for mitigating potemkins\n\n**We hope our comments have addressed your concerns. If not, please let us know if you have any more questions we can address in the follow-up.**\n\n> _Suggestions: (1) Incorporate evaluations of open-source models (e.g., LLaMA, Mistral) on this benchmark_\n\nThis is a great suggestion. We've expanded our evaluation to include the following nine models on the coherence task, many of them open-source: LLaMA-3.3, GPT-4o, Claude-3.5, GPT-4.5, o1-mini, o3-mini, Mistral-7B, DeepSeek-v3, and DeepSeek-r1. \n\n[See this table for the full results](https://imgur.com/a/4tHzAGn). We'll use the time between the rebuttal and camera-ready deadline to expand our analysis in Table 1 to the same set of models. We also note that our initial analysis already included LLaMA-3.3, [with results available here](https://imgur.com/a/8fKo8BS).\n\n> _Suggestions: (2) Attempt to propose solutions or suggestions to mitigate this issue, reduce Potemkin failures, to assist subsequent researchers follow this work better._\n\nAs you suggest, once potemkins are discovered (e.g. using the methods in our paper) we can build methods to train against them.  The automatic evaluation tasks in our benchmark -- e.g. concept use classification accuracy and self-coherence scores -- can be directly optimized during model fine-tuning. For example, a potential mitigation strategy is to explicitly use the automated coherence experiment to penalize the inconsistencies that can lead to potemkins. We feel this is the most important contribution of our benchmark and have added a section discussing it in our paper.  \n\n> _The paper primarily reveals the existence of the Potemkin phenomenon through experiments but does not delve deeply into its root causes, such as the model's training data, architectural characteristics, or optimization objectives._\n\nWe agree that our primary focus was documenting and quantifying potemkin understanding. As you point out, understanding the root causes would be very valuable. We speculate that this phenomenon arises due to a few reasons:\n- **Reinforcement Learning with Human Feedback (RLHF)** prioritizes fluent and plausible-sounding explanations over accurate and coherent ones.\n- **No coherence training objectives:** Models aren't explicitly trained to provide coherent responses, just accurate next-token predictions. Our automatic method for evaluating coherence provides a scalable and new way to train models and reduce the scale of potemkins.\n- **Limitations in Model Architecture:** The architectural specifications vary by each model. However, models like transformers may inherently favor shallow associations due to limited inductive biases for structured reasoning or causality.\nWe've expanded our discussion of these points in the manuscript. Future work should explore training interventions to directly address these causes.\n\n> _The domains covered are incomplete; similar issues may arise in areas like mathematical reasoning and code generation, which could be included to enrich the dataset._\n\nOur choice of domains—literary techniques, game theory, and psychological biases—was intended to span three distinct forms of understanding: linguistic, formal, and behavioral. Though incomplete, this set was intended to cover a wide range of types of understanding. Moreover, each domain includes multiple subdomains (32 in total). We found that potemkins were ubiquitous across all of these varied contexts, strongly suggesting their presence in other areas as well. Your suggestion to include mathematical reasoning and code generation is excellent because, like game theory, many of these problems can be evaluated automatically. We'll explicitly note this in our revision. \n\n> _Although the gap between definition and application tasks is evident, the authors do not thoroughly explore the inherent differences in difficulty and complexity between these two types of tasks._\n\nThis is an important point. We've gone back and re-examined our findings. While task complexity contributes in part to the Potemkin gap, we find that it cannot fully explain Potemkin failures. For example, our benchmark contains cases where models correctly apply challenging definitions but fail simpler application tasks (e.g., correctly defining complex concepts like \"Pareto Optimality\" yet incorrectly classifying simpler instances of more intuitive concepts like rhymes). This non-monotonic pattern of errors—where easier tasks are sometimes failed, despite succeeding at harder conceptual definitions— indicates conceptual gaps rather than purely difficulty-driven mistakes. We'll clarify this point explicitly in our revised text.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "make sense", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Cognitive Psychology, Interpretability of Large Language Models, maybe Knowledge editing of LLMs", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n\n(1) Novelty of the problem (Potemkin Understanding); (2) Cross-domain systematic experimentation; (3) Rigorous methodology with a focus on self-consistency evaluation, incorporating experiments on self-coherence.\n\nWeakness:\n\n(1) The paper primarily reveals the existence of the Potemkin phenomenon through experiments but does not delve deeply into its root causes, such as the model's training data, architectural characteristics, or optimization objectives. (2) The domains covered are incomplete; similar issues may arise in areas like mathematical reasoning and code generation, which could be included to enrich the dataset. (3) Although the gap between definition and application tasks is evident, the authors do not thoroughly explore the inherent differences in difficulty and complexity between these two types of tasks. This may lead readers to question whether the lower accuracy in application tasks is solely due to their higher difficulty rather than a genuine lack of conceptual understanding.", "other_comments_or_suggestions": "Suggestions：\n\n(1) Incorporate evaluations of open-source models (e.g., LLaMA, Mistral) on this benchmark; (2) Based on benchmark statistical data, attempt to propose solutions or suggestions to mitigate this issue, reduce Potemkin failures, to assist subsequent researchers follow this  work better.", "questions_for_authors": "See Suggestions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZEaMh5klzn", "reviewer": "Reviewer_mfaJ", "summary": "This paper investigates Potemkin understanding in language models. To assess model behavior, the authors design evaluation datasets across diverse domains, including literary techniques, game theory, and psychological biases. The findings indicate that while language models are good at explaining concepts, their accuracy in correctly applying these concepts remains low.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful and insightful review of our paper. We respond to your comments and describe new results below; to summarize the main changes, we've added:\n- A rewritten and simplified framework\n- A visualization of keystone questions and potemkins\n- Analysis of the role of question complexity\n- [An expansion of the coherence analysis to include 9 models](https://imgur.com/a/4tHzAGn)\n\n> _The theoretical framework, while attempting to formalize the concept of potemkin understanding, adds complexity rather than clarifying the formulation._\n\nThanks for this feedback. It motivated us to **significantly** rewrite and simplify our framework.\n\n[First, we've included a new figure to provide a visual clarification of our framework](https://imgur.com/a/9XHtZWz).\n\nWhile we don't have space to include the full revision here, an outline is below:\n- C*: set of correct interpretations of a concept. (Equivalently, a binary function over strings indicating correct or incorrect concept usage.)\n- C_h: set of possible human interpretations (with C^* \\in C_h)\n- C_l: set of all possible model interpretations\n- To test if a human's interpretation of a concept H is correct, it's intractable to compare all elements of H to C*.\n- Because humans have structured ways to interpret concepts, though, we show there exist keystone questions - sets of questions that serve as proofs of understanding if answered correctly. \n- But unless the set of LLM misunderstandings match human misunderstandings, there's no guarantee keystones work for LLMs.\n- A potemkin is an instance where an LLM answers a keystone question correctly but doesn't understand the concept. \n\n> _The study doesn't sufficiently control for question complexity in the \"use the concept\" evaluations._\n\nThis is an important point, and the current draft does not address it. Prompted by your question, we've gone back and re-examined our findings. While task complexity contributes in part to the Potemkin gap, we find that it cannot fully explain potemkin failures. For example, our benchmark contains cases where models correctly apply challenging definitions but fail simpler application tasks (e.g., correctly defining complex concepts like \"Pareto Optimality\" yet incorrectly classifying simpler instances of more intuitive concepts like rhymes). This non-monotonic pattern of errors indicates conceptual gaps rather than just difficulty-driven mistakes. \n\n> _However, I find that some cases for concept usage are overly complex for this setting. In Figure 2, while the first two problems seem reasonable, the third one is tricky..._\n\nThis is an important observation. A reason why potemkins are common is that conceptual understanding of one concept often depends on other concepts. For example, even if a model successfully lists prime numbers, it must also accurately recognize digits (such as '1') to demonstrate full understanding. However, we agree with your point that a more direct example in the figure might better demonstrate Potemkin failures, and have updated the figure with a new example. \n\n> _The evaluation methodology relies heavily on human expert annotation, which limits the scalability and practical applicability of the benchmarks._\n\nWe would like to highlight that while our methodology indeed uses human expert annotations in parts, much of it is automated, including: \n1. Coherence evaluations (fully automated)\n2. The classification task, which introduces 320 new questions as part of our benchmark.\nWe'll highlight these points in the revision. \n\n> _Could you elaborate on your claim \"Keystone elements are valid tests for LLMs if and only if C_l = C_h\" (Line 150)? Does this imply that your proposed benchmarks might not accurately reflect human understanding when tested on humans?_\n\nThank you for pointing this out. We agree that this was unclear in the original text. What we meant (and hope the above clarifies) is that (i) keystone questions are constructed to precisely reflect human conceptual understanding and (ii) if LLMs do not misunderstand in the same way, models can do well on keystones while failing to understand a concept. As such, the benchmarks we use in the paper are explicitly designed so that human success on the benchmark aligns with genuine understanding. \n\n> _Have you explored automating the evaluation process, perhaps using LLM-as-a-judge approaches?_\n\nAs noted above, parts of our evaluation are already automated, making them scalable. But we appreciate your suggestion: LLM-as-a-judge methods are an interesting direction, and they're straightforward enough that we will implement them between the rebuttal period and camera-ready deadline. Because we already have expensive human labels, that will also help us evaluate how good LLM-as-a-judge methods are.\n\n> _Typo suggestions_\n\nThank you for finding typos. We've corrected these in the revision, and our simplified framework also reduces complexity.", "claims_and_evidence": "The claims made in the paper are supported by experiments and analysis.", "methods_and_evaluation_criteria": "The paper introduces an evaluation dataset to assess Potemkin understanding in language models. However, I find that some cases for concept usage are overly complex for this setting. In Figure 2, while the first two problems seem reasonable, the third one is tricky. The model correctly generates prime numbers, suggesting it understands the concept. The failure occurs at the second step of reasoning, which does not accurately reflect a lack of understanding of \"prime numbers.\" Do these cases still count as Potekmins?", "theoretical_claims": "The paper presents a theoretical framework for Potemkin understanding in language models and humans. However, I found some parts challenging to follow:\n- The notations for two C_l in Line 115 are hard to specify.\n- In Line 117, the phrase \"misunderstand a concept\" might be \"understand a concept.\"?", "experimental_designs_or_analyses": "The experiments across different models and domains are well-performed. The self-coherence analysis also makes sense for evaluating the models' understanding.", "relation_to_broader_scientific_literature": "This paper introduces a specific form of hallucination, termed \"Potemkin,\" and creates a dataset to evaluate such behaviors in language models using human annotations. This contribution seems to be a useful resource for further research on model behavior and understanding.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths:**\n1. The paper proposes to evaluate a specific type of LLM hallucination, potemkin understanding, by providing human-annotated datasets, which offers a fresh angle for assessing model capabilities.\n2. The experiments show the disconnect between models' ability to define concepts versus actually applying them correctly.\n\n**Weaknesses:**\n1. The evaluation methodology relies heavily on human expert annotation, which limits the scalability and practical applicability of the benchmarks.\n2. The theoretical framework, while attempting to formalize the concept of potemkin understanding, adds complexity rather than clarifying the formulation.\n3. The study doesn't sufficiently control for question complexity in the \"use the concept\" evaluations. Poor performance on these questions could result from general reasoning difficulties rather than concept-specific misunderstandings.", "other_comments_or_suggestions": "Line 363: The notations are inconsistent with those in Section 2", "questions_for_authors": "1. Could you elaborate on your claim \"Keystone elements are valid tests for LLMs if and only if C_l = C_h\" (Line 150)? Does this imply that your proposed benchmarks might not accurately reflect human understanding when tested on humans?  Can you use specific examples to clarify this?\n\n2. Have you explored automating the evaluation process, perhaps using LLM-as-a-judge approaches? How well do automatic evaluations perform on your benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q62IyA1nc4", "reviewer": "Reviewer_4bAv", "summary": "This paper introduces the idea of Potemkin understandings, which is defined as differences in how human and large language models understand concepts. The main contribution of this paper is the design of a benchmark that tests the discrepancy in the model's ability to claim a definition of a concept and its ability to use the concept. The paper tests three popular models (GPT-4o, Llama-3.3, Claude 3.5) using the proposed benchmark, and shows a large drop in performance across all models. Additionally, the paper tests self-coherence of models by prompting the model to generate examples of a concept, and then checking whether the model classifies its generation as an instance of the concept. The paper show that the three models exhibit low self-coherence. Finally, the paper identified a bias towards questions that do not require concept usage in the MMLU benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We respond to your comments and describe new results; to summarize the main changes, we've added:\n- An expansion of the coherence analysis to include 9 models\n- A visualization of our mathematical framework\n- A rewritten and simplified framework\n- Analysis of question complexity\n\nYour comments were very constructive and have improved the paper. **We hope our comments have addressed your concerns. If not, please let us know if you have any more questions we can address in the follow-up.**\n\n> _While the experimental results support the claim that language models exhibit a discrepancy... the scope of the experiments is quite limited._\n\nWe like the way you've summarized our experimental results: \"they support the claim that language models exhibit a discrepancy between the ability to explain a concept and the ability to use it.\" We'll update the language in our paper to reflect the way you wrote it. \n\nWe also agree that considering more models will make our results more robust. [We've expanded our coherence evaluation to include 9 models -- see linked table](https://imgur.com/a/4tHzAGn). We'll use the time between the rebuttal and camera-ready deadline to add more models. \n\nOur choice of domains spanned 32 subdomains that were intended to span three distinct forms of understanding: linguistic, formal, and behavioral. Potemkins were ubiquitous across all of these varied contexts, suggesting their presence in other areas as well.  \n\n> _The claim that \"humans who can clearly define a concept necessarily possess a deep understanding and can effectively apply it\" does not logically follow._\n\nWe fully agree that our original claim is too broad. In fact, this claim is not central to our work. What is central is that we test models on “keystones” - the set of questions that indicate understanding when people do well on them. We'll modify the language in our revision. \n\nFor the applications we choose (unlike math, as you point out) definitional accuracy is used to indicate understanding (e.g. it's what is used in exams to test understanding). For example, clearly defining sunk cost fallacy encompasses most of the understanding of that concept, as use cases are simple applications of it.\n\n> _The presence of Potemkin understanding does not inherently invalidate the effectiveness of benchmarks._\n\nWe agree that the presence of potemkins doesn't invalidate benchmarks broadly. Benchmarks remain valid for assessing performance within the distribution they directly measure on held-out examples. Our point was subtler: Potemkins undermine the inference that high benchmark scores must reflect generalizable conceptual understanding. We'll clarify this point in the revision.\n\n> _Even if models pass tests designed to assess Potemkin comprehension, one could still validly argue that they function as stochastic parrots, merely mimicking human reasoning rather than genuinely understanding or reasoning about concepts._\n\nWe agree that our experiments don't answer whether mimicking human reasoning would reflect genuine understanding, nor do they intend to. Rather, our goal is pragmatic: since developing models that mimic human reasoning is one of the goals of LLM research, it's important to measure this ability. We'll adjust our language to make this clear.\n\n> _Section 2 contains lots of unnecessary math._\n\nWe've completely rewritten Section 2, simplifying and clarifying the mathematical presentation. See our response to reviewer voEv for more details. [We've also included a new framework visualization here](https://imgur.com/a/9XHtZWz).\n\n> _It's kind of already a consensus that generation is harder than understanding._\n\nOur tasks testing \"use\" of concepts aren't exclusively about generation (e.g., classification tasks rely on recognition rather than generation). In fact, we show that models perform similarly poorly on classification and constrained generation tasks. \n\nPrompted by your question, we've also gone back and re-examined our findings. While task complexity contributes in part to the Potemkin gap, we find that it can't fully explain Potemkin failures. For example, we find models correctly apply challenging definitions but fail simpler applications (e.g., correctly defining complex concepts like \"Pareto Optimality\" yet incorrectly classifying simpler instances of concepts like rhymes). This non-monotonic pattern of errors indicates conceptual gaps rather than just difficulty-driven mistakes.\n\n> _How is this work contributing to future model development?_\n\nGood question. Once potemkins are discovered using the methods in our paper, we can build methods to train against them. Moreover the automatic evaluation tasks can be directly optimized during model fine-tuning. One strategy is to explicitly use the automated coherence experiment to penalize the inconsistencies that lead to potemkins. We feel this is the most important contribution of our benchmark and have added a discussion to the paper.", "claims_and_evidence": "Overclaiming is the central issue in this work. While the experimental results support the claim that language models exhibit a discrepancy between the ability to explain a concept and the ability to use it, the scope of the experiments is quite limited. Only three models are evaluated across three domains, restricting the generalizability of the findings. Besides,\n\n* The claim that \"humans who can clearly define a concept necessarily possess a deep understanding and can effectively apply it\" does not logically follow from the observed differences between human and model comprehension. A well-known counterexample comes from mathematics, where individuals can memorize definitions without being able to apply them correctly.\n* The presence of Potemkin understanding does not inherently invalidate the effectiveness of benchmarks. The specific analysis of MMLU’s distributional properties does not substantiate the broader claims made by the authors.\n* Measuring the presence of Potemkin understanding does not resolve the debate between the two competing perspectives outlined in the related work. Even if models pass tests designed to assess Potemkin comprehension, one could still validly argue that they function as stochastic parrots, merely mimicking human reasoning rather than genuinely understanding or reasoning about concepts.", "methods_and_evaluation_criteria": "The benchmark design is generally well-structured and conceptually sound. However, the experimental scope is too limited to draw strong conclusions. The evaluation lacks breadth, as it only considers a narrow set of models and testing conditions.\n\nTo strengthen the study, the authors are encouraged to expand their evaluation to include a wider range of LLMs and alternative evaluation methods. Currently, the experiments rely solely on prompting, but it remains unclear whether this involves zero-shot, few-shot, or chain-of-thought prompting.", "theoretical_claims": "There is no theoretic claim. Section 2 contains lots of unnecessary math.", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria.", "relation_to_broader_scientific_literature": "Besides hallucination, I think this work contributes to the discussion of \"competence and performance\" in LLMs. Overall this work introduces very novel concepts to the field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See previous discussions.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Indeed, the findings \"reveal a significant disparity between the ability of models to explain and use concepts.\" But it's kind of already a consensus that generation is harder than understanding. How is this work contributing to future model development, or it's just introducing unnecessary terminologies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0kNKIaADqA", "reviewer": "Reviewer_voEv", "summary": "The authors discussed the phenomenon of $\\textbf{potemkins}$ in large language models, referring to the cases where a model’s misunderstanding of a concept does not align with the way humans would have misconceived it. The authors proposed a framework to formally define what Potemkin is and designed a benchmark to quantify Potemkin with 3 different tasks to test whether LLM can understand and switch between a given concept and its instances. The authors then evaluate the performance of 3 different LLMs (GPT-4o, Llama- 3.3-70B-Instruct-Turbo, and Claude-3.5-sonnet) on three different domains (literary techniques, game theory, and psychological biases). The authors find that the phenomenon of Potemkin exists across all the models and tasks they analyzed. The authors also discussed the problem of self-coherence to understand whether the ubiquity comes from consistent/inconsistent internal concept understanding of LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our paper. We're glad that you appreciated our paper and findings. We respond to your comments and describe new results; to summarize the main changes, we've added:\n- An expansion of the coherence analysis to include 9 models\n- Unaggregated results for table 2\n- A rewritten and simplified framework\n\n**We hope our comments have addressed your concerns. If not, please let us know if you have any more questions we can address in the follow-up.**\n\n> _The definitions of \\mathcal C_h, C_h, \\mathcal C_l, C_l, are confusing to me. Are they sets of all possible strings to demonstrate people's understanding of the concept ?_\n\nThanks for this feedback. It motivated us to **significantly** rewrite and simplify our framework, ensuring clarity while maintaining the core insights. [First, we've included a new figure to provide a visual clarification of our framework (see attached)](https://imgur.com/a/9XHtZWz).\n\nDue to length constraints (and because OpenReview currently restricts submission updates), we can't include the full revised section here. However, here is an outline of the new structure:\n- C*: set of correct interpretations of a concept. (Equivalently, a binary function over strings indicating correct or incorrect concept usage.)\n- C_h: set of possible human interpretations (with C^* \\in C_h)\n- C_l: set of all possible model interpretations\n- To test if a human's interpretation of a concept H is correct, it's intractable to compare all elements of H to C*.\n- Because humans have structured ways to interpret concepts, though, we show there exist keystone questions - sets of questions that serve as proofs of understanding if answered correctly. By definition, these cannot be aligned with any human way to misinterpret a concept.\n- But unless the set of LLM misunderstandings match human misunderstandings, there's no guarantee keystones work for LLMs: LLMs might interpret concepts in arbitrary ways.\n- A potemkin is an instance where an LLM answers a keystone question correctly but doesn't understand the concept. \n\nWe hope this makes our theoretical framing clearer. \n\n> _The authors provide verbal descriptions, but it would be helpful to provide some visualization (maybe a Venn graph?) and use the example of Haiku._\n\nWe appreciate your suggestion. [We've added a new visualization of these concepts using a Venn diagram at the attached link](https://imgur.com/a/9XHtZWz).\n\n> _In Line 153, why \"keystone elements are valid tests for LLMs iff C_h=C_l\"? Given the definition authors give about and C_k (\"collection of sets, where each set represents a distinct and coherent category of how a human might understand a concept\"), this seems like a very strong requirement._\n\nWe recognize our original language was unclear and as noted above have clarified the framework.  We've also revised the manuscript to clarify the phrase “valid test”. \n- By \"valid,\" we specifically mean that answering keystone questions correctly implies true conceptual understanding in the LLM.\n- We agree that it is a strong requirement for models to follow, which is part of the point of the paper. We will add a discussion of why if models misunderstand in ways that are distinct from how humans misunderstand, keystones that work for people will not work for LLMs.  \n\n> _In Table 2, the authors provided aggregated results across domains and models. In Table 1, we can see that the performance of different model-domain combinations can be significantly different (e.g., LLama-Game theory 0.10 vs Claude-3.5-Game theory 0.50), and thus aggregation can lead to misrepresentation/misinterpretation of results; standard error will not make sense here either. I suggest the authors provide un-aggregated results in Table 2._\n\nThanks for this suggestion. [We've provided a new table with unaggregated results here](https://imgur.com/a/4tHzAGn). This new table is also expanded to include 9 models. We agree that the detailed breakdown makes it easier to explore specific model-domain combinations.", "claims_and_evidence": "The authors support their claims with detailed experiments.", "methods_and_evaluation_criteria": "The benchmark dataset simplifies the problem to test concept explanation/related tasks, and evaluate the performance through calculating accuracy.", "theoretical_claims": "There are no \"theoretical claims\". This work is about evaluating/benchmarking LLMs. The authors provided a formalized framework using some mathematical notations, which are not very rigid. I discussed some problems in the following sections.", "experimental_designs_or_analyses": "I carefully read the authors' experimental design in Section 4.", "relation_to_broader_scientific_literature": "The authors discuss the problem of Potemkin in LLMs, which can be interesting to the broader community, especially in the fields that are interested in precise-concept applications.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\nThe problem is interesting. \nThe experimental design is detailed, and the presentation of results is clear.\nThe discussion of related work is from multiple perspectives, and the authors highlighted their contribution.\nWeakness:\nSee Questions For Authors.", "other_comments_or_suggestions": "Line 363, notations for $\\mathcal{C}_h$ and $\\mathcal{C}_l$ are inconsistent.\nIt would be more informative if the authors could add more explanation/thoughts on the results. For now, it mostly just reflects the accuracy numbers.", "questions_for_authors": "1. The definitions of  $\\mathcal{C}_h$, $C_h$ (and $\\mathcal{C}_l$, $C_l$) are confusing to me. Are they sets of all possible strings to demonstrate people's understanding of the concept $C$? The authors provide verbal descriptions, but it would be helpful to provide some visualization (maybe a Venn graph?) and use the example of Haiku. These are important definitions, so they need to be made clearer to readers.\n2. In Line 153, why \"keystone elements are valid tests for LLMs iff  $\\mathcal{C}_h =  \\mathcal{C}_l$\"? Given the definition authors give about $\\mathcal{C}_h$ and $\\mathcal{C}_l$ ($\\textit{collection of sets, where each set represents a distinct and coherent category of how a human might understand a concept}$), this seems like a very strong requirement.\n3. In Table 2, the authors provided aggregated results across domains and models. In Table 1, we can see that the performance of different model-domain combinations can be significantly different (e.g., LLama-Game theory 0.10 vs Claude-3.5-Game theory 0.50), and thus aggregation can lead to misrepresentation/misinterpretation of results; standard error will not make sense here either.  I suggest the authors provide un-aggregated results in Table 2.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel failure mode in large language models (LLMs), termed *Potemkin Understanding*. This phenomenon describes a disconnect between a model's ability to define a concept correctly and its ability to apply the concept accurately in practical tasks. The authors construct a benchmark spanning three domains—literary techniques, game theory, and cognitive biases—covering 32 concepts and 5,986 data points. Tasks include concept definition, classification, constrained generation, and editing. The results show that while models perform well on definition tasks (97.7% accuracy), their performance drops significantly on application tasks (67.9% accuracy). The study also explores model self-coherence and critiques the limitations of existing benchmarks such as MMLU in capturing Potemkin Understanding.

**Strengths**
- Introduces a novel and well-motivated concept (Potemkin Understanding) in LLM evaluation.
- Rigorous, cross-domain benchmarking with clear methodological design.
- Demonstrates a consistent gap between definition and application across models.
- Includes self-coherence analysis to assess internal consistency of models.
- Dataset and benchmark are potentially valuable for future work on model understanding.
- Relevant to ongoing discussions in interpretability, hallucination, and model evaluation.

**Weaknesses**
- Does not explore underlying causes of Potemkin Understanding (e.g., training data, model architecture).
- Limited domain coverage; lacks evaluations in math or code tasks.
- Heavy reliance on human annotation limits benchmark scalability.
- Theoretical formulations are vague and may confuse rather than clarify key ideas.
- Inadequate control for the difficulty difference between explanation and application tasks.
- Aggregated reporting may misrepresent model-domain performance variations.

Most concerns have been addressed by the authors during the rebuttal period. This paper starts with diverging ratings, with two positive and two negative scores (including one strong negative score). After rebuttal, which I believe is quite successful, two of the reviewers are convinced that most of their concerns are addressed and agree that this paper makes an interesting contribution to the community, ending the discussion with three positive and one (weak) negative scores. The AC agrees that this paper would be an interesting addition to the ICML program.
Decision: Accept (poster)