ICML 2025

Potemkin Understanding in Large Language Models
Marina Mancoridis 1 Bec Weeks 2 Keyon Vafa 3 Sendhil Mullainathan 4
Abstract
Large language models (LLMs) are regularly eval-
uated using benchmark datasets. But what justi-
fies making inferences about an LLM’s capabili-
ties based on its answers to a curated set of ques-
tions? This paper first introduces a formal frame-
work to address this question. The key is to note
that the benchmarks used to test LLMs—such
as AP exams—are also those used to test people.
However, this raises an implication: such bench-
marks are only valid tests if LLMs misunderstand
concepts in ways that mirror human misunder-
standings. Otherwise, success on benchmarks
only demonstrates potemkin understanding: the
illusion of understanding driven by answers irrec-
oncilable with how any human would interpret
a concept. We present two procedures for quan-
tifying the existence of potemkins: one using a
specially designed benchmark in three domains,
the other using a general procedure that provides
a lower-bound on their prevalence. We find that
potemkins are ubiquitous across models, tasks,
and domains. We also find that these failures re-
flect not just incorrect understanding, but deeper
internal incoherence in concept representations.
1. Introduction
There has been a marked change in how we interpret ma-
chine learning evaluations. Today, large language models
(LLMs) are evaluated on benchmark datasets: curated ques-
tions with rubrics for grading. Success on these questions
is treated as evidence of broader conceptual understanding.
Previously, while benchmarks were also used to evaluate
supervised learning models, success was interpreted more
narrowly. A pathology classifier that performs well on X-
ray classification is not credited with an understanding of
1Massachusetts Institute of Technology 2University of Chicago
3Harvard University 4Massachusetts Institute of Technology. Cor-
respondence to: Marina Mancoridis <marinam@mit.edu>.
Proc...