Core Knowledge Deficits in Multi-Modal Language Models
Authors: Yijiang Li
Qingying Gao
Tianwei Zhao
Bingyang Wang
Haoran Sun
Haiyun Lyu
Robert D. Hawkins
Nuno Vasconcelos
Tal Golan
Dezhi Luo
Hokin Deng
Venue: ICML 2025 poster (2025)
Keywords: core knowledge
multi-modal large language model
robustness
Primary area: applications->neuroscience_cognitive_science
TL;DR: We find current multimodal language models (MLLMs) lack core knowledge which's acquired by humans through cognitive development in a young age and we reveal the dependency map of core cognitive concepts in MLLMs.

Abstract: While Multi-modal Large Language Models (MLLMs) demonstrate impressive abilities over high-level perception and reasoning, their robustness in the wild remains limited, often falling short on tasks that are intuitive and effortless for humans. We examine the hypothesis that these deficiencies stem from the absence of core knowledge—rudimentary cognitive abilities innate to humans from early childhood. 
To explore the core knowledge representation in MLLMs, we introduce CoreCognition, a large-scale benchmark encompassing 12 core knowledge concepts grounded in developmental cognitive science. We evaluate 230 models with 11 different prompts, leading to a total of 2,530 data points for analysis. Our experiments uncover four key findings, collectively demonstrating core knowledge deficits in MLLMs: they consistently underperform and show reduced, or even absent, scalability on low-level abilities relative to high-level ones. 
Finally, we propose Concept Hacking, a novel controlled evaluation method, that reveals MLLMs fail to progress toward genuine core knowledge understanding, but instead rely on shortcut learning as they scale. Project page at https://williamium3000.github.io/core-knowledge/.

Core Knowledge Deficits in Multi-Modal Language Models
Yijiang Li" Qingying Gao 22 Tianwei Zhao
Bingyang Wang" *3 Haoran Sun Haiyun Lyu
Robert D. Hawkins
Nuno Vasconcelos
Tal Golan Dezhi Luo 78 Hokin Deng?
Abstract
believe that humans must be born with set of innate knowl- edge. In Meno, 80d-86b, Socrates introduces the theory of anamnesis (recollection), where he suggests our "soul is immortal", and "it can recollect the things it knew be- fore" (Fowler et al., 1914). Plato further sets the distinction between innate knowledge and those we gain through ex- perience: in Republic VII, the Allegory of the Cave, he suggests that our experiences are skiés, like shadows the cave wall, which are contingent instantiations of the eidos, the knowledge born with our minds. One example of eidos is our understanding of a circle: while a perfect circle never exists in reality, we still understand what it means to be perfect circle (Jowett et al., 1888). Kant's view is more intricate: he suggests we never have an innate knowledge of noumena, belings-in-themmeleest but we have knowl- edge of phenomenon, thiings-abou-tthemellee we only are born with knowledge about the structures of our experiences, such as causality, permanence, and con- tinuity, but never gifted with knowledge experiences in itself (Kant, 1781). In other words, we have innate, core knowledge about basic domains of the world.
While Multi-modal Large Language Models (MLLMs) demonstrate impressive abilities over high-level perception and reasoning, their robust- ness in the wild remains limited, often falling short tasks that are intuitive and effortless for humans. We examine the hypothesis that these deficiencies stem from the absence of core kndwledge-ndimmentar cognitive abilities innate to humans from early childhood. To explore the core knowledge representation in MLLMs, we introduce CoreCognition, a large- scale benchmark encompassing 12 core knowl- edge concepts grounded in developmental cog- nitive science. We evaluate 230 models with 11 different prompts, leading to a total of 2,530 data points for analysis. Our experiments un- cover four key findings, collectively demonstrat- ing core knowledge deficits in MLLMs: they consistently underperform and show reduced, even absent, scalability on low-level abilities rel- ative to high-level ones. Finally, we propose Concept Hacking, a novel controlled evaluation method, that reveals MILMS fail to progress to- ward genuine core knowledge understanding, but instead rely on shorteut learning as they scale. Project page https: / /wi Welllllammmii ammi um3000. github. www.ioooeeooweeeeeeeeeeeeeeeeeee
meaning
We are closer than ever to achieving duman-leve artificial intelligence. By training on vast web-scale corpora and scal- ing to hundreds of billions of parameters, Large Language Models (LLMs) now surpass expert humans in knowledge- and reasoning-intenn intennive tasks (Brown et al., 2020; Achiam et al., 2023; Bai et al., 2023a; Touvron et al., 2023; Jaech et al., 2024). These capabilities extend beyond language: with modality alignment (Liu et al., 2024b; Li al., 2023b; Zhu et al., 2023), MLLMs exhibit unprecedented high-level perception and reasoning (Gemini, 2023; Wu & Xie, 2024; Xu et al., 2024; Yang et al., 2025a: Shao et al., 2024; Yang et al., 2024; Li et al., 2024a; Fu et al., 2023), mastering tasks such as chart understanding (Masry et al., 2022), geometry and math (Lu al., 2023). and action recognition and pre- diction (Ying et al., 2024; Liu et al., 2024c), often reaching exceeding human performance (Huang & Zhang, 2024).
1. Introduction
Are human minds born with knowledge (Plato et al., 1763)? This has been the central question of Western thoughts since the ancient Greeks (Russell, 1946). Socrates and Plato both
"qqual contribution UUiivesitty of California San Diego Johns Hopkins University 'Emory University "University of North Car- olina at Chapel Hill "Stanford University "Ben-Gurion University of the Negev 'University of Michigan University College London "Cannegie Mellon University. Correspondence to: Yijiang Li <yi- jiangli@uess www.uucd..dee edu>, Dezhi Luo <ihzedoul @umich.edu>......... Hokin Deng <hokind @ andrew.emuu edu.
Despite advances in high-level perception and reasoning abilities, state-of-the-art MILMS still underperform humans on simple and rudimentary tasks such as counting (Paiss et al., 2023; Qharabagh et al., 2024), perspective taking (Tang al., 2025b). spatial reasoning (Zhang al., 2025; Tang et al., 2025a), temporal reasoning(Saxena et al., 2025).
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the author(s).

---

Core Knowledge Deficits in Multi-Modal Language Models
Sensorimotor
Boundary
Continuity
Spatiality
Perceptual Constancy
Intuitive Physics
What the shape
How trains the
tste adio in the last image? (Mes)
Is there only one onelevel surfoce
Are the actual colors of the two Rubiks Cubes the same? (Yes)
Which the two systems in the pectutuee more likely to
pillw? (A]
A Redtangle
image?
1lp over? [A
B. Circle
Throo
(Nto
A Left orw Right
One
Formal Operation
Concrete Operation
Mechanical
intentionality
Tool Using
Conservation
Hisrarchy
Perspective
What the person lrying to do?
What should tsee to find socks under the bed?
the last frome, the number candies in the upper row and the lower row the samn?? [Pes)
Are there more vindmills more red buildings in the image? (C
From the delf's point of view, which object appears the rightmost? [f]]
more towards
joring pulled?
(A)
Swim home B. Cleenn the fish tank
A. Windmll
A. The red can
Upwards
& Red building
B. The silver
Downwards
20000
C. The black
Figure I. Examples from our CoreCognition benchmark.
Concapt
Concept
Delinition
Concage
Definition
Boundary
The transtionn from object another.
Continiity
Objects persist as unified, colesive
Permanence
Objjccts not cesse to exist
eniiiies across space and time.
they are no longer perceived.
Spatiality
The prioni understanding the
Perceptual
Changes appeteinces
mean changes in plysical properties.
Intuitive Physies
Intuttions about laws how
Euclidion properties the world. see what sce.
Coustoncy
things intoract the physical world.
Perspective
Hierarchy
Understanding inclusion exclusion of objects and categories.
Conservation
Lnvariances of properties despite trancformationss.
capacity to manipulate specific objects to achieve guals.
Intentionality
see others want.
Mechanical Reasoning
Interring actions system states and vice versa.
Table 1. Abbreviated definitions of the 12 core abilities assessed. See Appendix A.2 for details.
Sessodlmotor
Concerte Opeoriin
FomalOpeeation
edge- fundameanlll cognitive abilities innately present in humans from early childhood that underpin advanced rea- soning. To examine this hypothesis, we explore the exis- tence, representation, and use of core knowledge in MLLMs by introducing the first large-scale benchmark tailored for core knowledge- CoreCognition. It comprises 1,503 sam- ples with over 95 samples for each concept, as exempli- fied in Fig. 1. Drawing on insights from developmental cognitive science, we propose a taxonomy of 12 abilities encompassing the full spectrum of core knowledge, from basic cognitive skills to advanced reasoning.
e Contone
X
Pesoypal Ceniery e
Scadder
Sprslry
Comanittor
Figure 2. Left. Statistics of the CoreCognition benchmark. Right. Construction of taxonomy. Dependencies between abilities are indicated with arrows.
and compositional reasoning (Yuksekgonul et al., 2022; Sahin et al., 2024; Mitra et al., itive and effortless for humans, even as MLLMs excel at related high-level reasoning (Paiss et al., 2023; Rahman- zadehgervi et al., 2024), exemplifying the long-standing Moravec's Paradox (Moravec, 1988). This high-level ex- cellence often fails to generalize to out-of-distribttion or real-world scenarios, where small changes in task condi- tions can cause significant performance drops (Shiffrin & Mitchell, 2023; Zhang et al., 2024b; Bai et al., 2024; Oh et al., 2025; Dong et al., 2025). Moreover, MILMS are vulnerable imperceptible perturbations (Schlarmann & Hein, 2023), susceptible to prompt variations (Wu et al., 2023), and can be easily jailbroken to generate unsafe or unregulated content (Wang et al., 2024b; Gu et al., 2024; Li et al., 2024c).
To provide comprehensive evaluation of core knowledge over the existing MLLMs, we assess total of 230 models with 11 different prompting techniques, yielding total of 2,530 data points. Leveraging these results, we analyze model performance across varying levels of core ability, examining the inter-dependencces among core knowledge and their predictive power for higher-level reasoning and perception, as well the scaling effect (performance across different model sizes) To further ascertain core knowledge deficits in MILMss, we design controlled experiments that manipulate causal features within images to perturb the ground-trum -ruth labels, allowing us to determine whether mod- cls genuinely possess the targeted core knowledge or merely approximate it through shortcuts and spurious correlations. Our key findings are:
tasks that are intu-
Core Knowledge Deficits: MLLMs consistently perform worse on low-level abilities compared to high-level abili- ties (Sec. 4.
In this work, we hypothesize that the deficiencies ob- served in MILMMs stem from the absence of core knowl-
2

---

Core Knowledge Deficits in Multi-Modal Language Models
Misaligned Dependency: MLLM performance on high- level abilities is not correlated with the underlying low- level abilities that support them (Sec. 4.2).
Clark al., 2019; Dagaev et al., 2021). Other methods reduce shortcut reliance by minimizing mutual information between features and bias attributes (Kim et al., 2019), or by generating bias-conflicting samples through latent factor swapping (Lee et al., 2021).
Not Scaling: MLLMs exhibit less, or even no, scalability (with respect to increasing model parameters) on low-level abilities compared to high-level abilities. (Sec. 4.4).
Core Knowledge in Humans. The debate over core knowl- edge has historically framed nativist and empiricist episte- mologies (Plato al., 1763; Kant, 1781; Russell, 1946), and since the cognitive revolution, has shifted toward empirical investigation (Piaget, 1950; Fodor, 1975). Piaget's stage- based theory and subsequent research established the foun- dations of developmental psychology (Piaget & Inhelder, 1969; Barrouillet, 2015; Spelke et al., 1992; Rochat, 2024; Carey et al., 2015). Recent advances show that even infants exhibit rudimentary knowledge of objects (Baillargeon & Carey, 2012; Kar al., 2019; Ullman & Tenenbaum, 2020), actions (Yang et al., 2015; Jara-Fitinger et al., 2020), num- bers (Feigenson et al., 2004; Hannagan et al., 2015; Spelke, 2017), space (Newcombe & Sluzenski, 2004; Bellmund et al., 2018), and social relations (Siegal & Varley, 2002; Scott & Baillargeon, 2017: Spelke, 2022). This "develop- mental start-up software" enables early learning (Spelke & Kinzler, 2007; Lake et al., 2017) and serves as the founda- tion for complex reasoning in variable environments later in life (Barsalou, 2020; Mitchell, 2021).
Models increasing size exhibit core deficits and shortcut-taking behaviors rather than progressing toward conceptual understanding. (Sec. 5.2).
2. Related Works
Multi-modal Large Language Models. With the advent of large language models (LLMs), sfate-of-the-ar (SOTA) MLLMs (Liu et al., 2024b; Li et al., 2023c) have adopted open-source LLMs (Touvron et al., 2023; Peng et al., 2023; Jiang et al., 2023) and aligned visual features to the LLM embedding space (Li et al., 2023b). To enable open-ended conversational abilities, LLaVA (Liu et al., 2024b) distills ChatGPT's conversational skills into MILMs, resulting in substantial performance gains-a process that has become standard practice in the field (Wang et al.. 2023; Bai et al., 2023a; Gemini, 2023; Team, 2024a; Sun et al., 2023; Li et al., 2022a).
Benchmarks for Multi-modal Large Language Model. A wide range of benchmarks have been proposed to eval- uate the growing capabilities of MLLMs, ranging from vi- sion question answering (VQA) (Antol et al., 2015; Marino et al., 2019), image captioning (Plummer et al., 2015; Lin et al., 2014), OCR and text understanding (Liu et al., 2023b). More recently, MLLM benchmarks have focused on higher- level reasoning, such as Math Verse (Zhang et al., 2024a) and ScienceQA (Lu et al., 2022), emphasizing multimodal reasoning in scientific domains. Of particular relevance are M3GIA (Song et al., 2024) and Marvel (Jiang et al., 2024), which address cognitive complexity, abstraction, and multi-step reasoning, but primarily focus on task coverage or high-level general intelligence. In contrast, CoreCog- nition targets early-emerging core abilities that support higher-level perception and reasoning. DevBench (Tan et al., 2024) adopts a developmentallyy inspired framework to probe language learning trajectories, but focuses solely on language, unlike our benchmark, which targets multi-modal core knowledge.
3. Benchmarking Core Knowledge in Multi-modal Large Language Models
We introduce CoreCognition, encompassing 12 core abil- ities and 1,503 questions with diverse input types and for- mats. An overview of the benchmark and its distribution is shown in Fig. 2, with 12 representative examples in Fig. 1. Sec. 3.1 outlines the cognitive taxonomy and theoretical framework guiding our benchmark. Sec. 3.2 details the curation process, while Sec. 3.3 describe model inference and evaluation.
3.1. Cognitive Framework
We take inspiration from Jean Piaget's theory (Piaget, 1950; Piaget & Inhelder, 1969; 1974), which identifies four stages in human developmental trajectory: Sensorimotor, Preoper- ational, Concrete Operational, and Formal Operational. In the Sensorimotor stage, infants develop core concepts such as object permanence (Spelke et al., 1992; Bremner et al., 2015) and perceptual constancy (Green, 2023) through sen- sory and physical interactions. The Preoperational stage serves as a transitional phase, characterized not by distinct new abilities but by the gradual solidification of symbolic representations (Fodor, 1975). These cognitive advance- ments culminate in the Concrete Operational stage, where children acquire abilities for systematic reasoning about numbers, motion, and agents, including perspective-taking.
Shortcut Learning. Shorteut learning is closely related, especially to Concept Hacking. Deep learning models are prone to exploiting spurious correlations-fed features that en- able strong in-distribution performance but lead to brittle- ness out of distribution (Alvi et al., 2018). Early work addressed this by removing biased features from learned embeddings (Wang et al., 2019). Subsequent approaches trained auxiliary bias predictors to identify shorteut cues and encouraged the main model to predict against them (Bahng et al., 2020; Cadene et al., 2020; Nam et al., 2020;
3

---

Core Knowledge Deficits in Multi-Modal Language Models
conservation, intuitive physics, and hierarchical relations (Piaget & Inhelder, 1974; Moll & Meltzoff, 2011; Piloto et al., 2022; Murphy & Lassaline, 2013). The Formal Op- erational stage extends these abilities to abstract reasoning and complex tasks, such as understanding intentionality and mechanical reasoning (Kilner, 2011; Allen et al., 2020). See Appendix A. for empirical support of framework and Appendix A.2 for a detailed description of core abilities.
narios. These prototypes serve as templates from which concrete data instances can be generated.
Instantiation of Prototypes. To instantiate prototype, we generate vision modalities (images/videos) that abstractly align with the underlying concept. These media assets are collected from a variety of sources, including internet, pub- lic datasets, synthetic content by generative models, sim- ulated environments, and original recordings captured by cameras. Each asset is then paired with carefully designed question that probes the specific core ability, along with a pre-defined options and the ground-truth answer, forming the multiple-choice questions (MCQs). Please refer to Ap- pendix B for a detailed discussion on types and formats of curated questions and Appendix C for justification over the difficulty of the questions in CoreCognition.
3.2. Dataset Curation
Concept
Bayy longer perovined.
Prototype
Instandiate
Objeot infant Experiment
Quality Control. Following the established criteria, each question-answer (QA) pair undergoes two rounds of inde- pendent cross-validation by annotators from separate groups. Any data point that fails to meet the standard is discarded. To further assess the reliability and clarity of the QA items, we conducted an additional round of validation by collecting responses from 20 human annotators via Amazon Mechan- ical Turk. We further recheck QAs that lead to consistent mistakes by humans.
"Which is the ball in!"
adket anget
Divas Euscia 2012
Quality Check
Cross-validation External reviow
the yellow at the and of the video!"
Discriminttiennes
Minimal Confounding
Minimal Text Shortest
Figure 3. Overview of the benchmark curation process
Building upon the above cognitive framework, opera- tionalize theoretical constructs into explicit examples de- signed to probe specific core abilities in MLLMs. To ensure conceptual integrity and interdisciplinary rigor, we estab- lish criteria that define successful instances: 1. Discrimi- nativeness: Instances should be structured such that mod- els lacking the targeted core knowledge necessarily select the incorrect answers, thereby ensuring the discrininative power. 2. Minimal Confounding: Questions should mini- mize reliance on confounding capabilities, such as object recognition, and must avoid conceptual overlap with other core knowledge included in the benchmark. 3. Minimal Text Shortcul: Instances should be crafted so that answers cannot be derived through textual shortcuts alone but require genuine multimodal comprehension.
3.3. Inference and Evaluation Strategy
Evaluating MLLMs large scale with QA formats poses several challenges: 1. MILLMs, ranging from 1B to 1 110B parameters, demand substantial computation resources and inference time with different environment dependencies for each of the 230 models, complicating efficient and robust evaluation under limited computational resources. 2. The free-form outputs from MLLMs can be highly variable, potentially leading conceptual errors in performance as- sessment if an inappropriate evaluation method is used.
Inference. To address these challenges, we built a scal- able evaluation infrastructure supporting parallel execution and compartmentalized environments, enabling reliable in- ference across over 200 MLLMs. We strictly follow the setup and source code from the official codebases provided by model developers to ensure fidelity. Further details are provided in Appendix D.I.
A total of 12 annotators, each with college-level educa- tion in cognitive science, computer science, or statistics, collaborate on the curation of CoreCognition.
Evaluation. For each k-choice question, we cyclically ro- tate the answer options I times, generating & versions with different option orders. Rather than requiring consistent se- lection of the correct answer across all rotations to assign a correct score, we instead calculate the proportion of correct responses over the augmented set. This averaging approach avoids the exponentially diminishing chance-level accuracy that arises when enforcing consistency on questions with many options. We refer to Appendix D.2.3 for more details.
Prototyping. We operationalize 12 theoretical concepts into a series of prototype scenarios that abstractly exemplify situations suited evaluating specific core abilities within MLLMs. For example, evaluate object permanence, we drew inspiration from classic infant experiments in develop- mental psychology (Zewald & Jacobs, 2020). The prototype scenario involves a ball being hidden under one of several cups, followed by occlusion and spatial manipulation. For each cognitive ability, we developed 5-10 prototype sce-

---

Core Knowledge Deficits in Multi-Modal Language Models
Peenud
Modd
Doundayy
Contibiity
Perooennc
Spaulay
Faveptal Capmacy
bezzirc Pesses
Propected Taces
Cresrraeee
Heraaial
Rehase
Undeoandie
inartseally
Matusee
Reassi
Uits
Man
M0.00
$5:56
Progicsus Huse
GFralldaccharl zt, 3320
63.45
53.00
61.32
87.79
75.45
85.31
18.36
75.08
K7.54
85:13
16.00
14.9
GfT--echheee.. 3820
87.37
03.77
13.85
67:tt
81.31
$6.78
43.06
08.97
57.53
15.31
57.53
06.91
08.25
Qwe--i VL. Maxttalaral 33370
66:12
86:33
57.88
27.83
78.86
63
01/01
86.38
07.80
(3:91
Garazi 1.5 Haabttaa (1))
71.22
03.58
7762
57.42
3153
51.85
33.85
12.33
05.77
15.50
GFF-4 Trr-4Tatbcccciin eia. 2003)
77.3
01.34
13.94
14.25
82.17
03.48
43.31
13:15
11.03
10.02
73.33
18:25
05:33
Gri-3 Vineesi. 3336
08.09
38.21
73.30
$3.04
37:34
73.13
71 12
79.07
18:13
99.31
$3.80
OFT-ooraaeillent 2014)
34.25
70.30
06.33
91.39
41.01
42.04
02.35
5039
Open Stary Mu-hh
Owe.SSV. AAN--188.00 cil., 3005
79.59
66.53
13.55
63.03
86.27
96.09
26.91
$1.80
68.30 1335
50.85
$5.86
97.06
68.39
(AnnVE 7.888.0 wt8.iteraat7lecceee. 3055
0162
58.86
64.07
19.90
20.23
50.12
70.79
13.65
33.3 1.30
38.05
51.50
91.18
11.00
OistA sial 2004)
(ff Owf 2004)
99.31
34.95
60.31
20.53
54.90
(erera a 20W Trar 2005
7818
50.30
15.05
15:36
00.07
51.53
15.31
53.25
00.31
77.55
13.53
02.53
59:56
NBEA15 et ana
7537
00.00
18.20
09:14
56.55
1965
78.77
0130
18.51
Paral 1796 Agroal 2010
13.38
11.55
06.55
58.76
58.03
96.91
57.8
[1.VA 700d etal. 2016
2631
05.33
60.00
17:01
00.91
13.33
18.14
77.05
33.00
66:71
16.35
MidAlaystt at N006
7181
06.85
13-4
79.7
58.38
38.03
80.07
6135
16.90
55.19
15-- wadieonilicaaaattttaaaa 23120
7537
$2.99
36/2
60.37
33.54
25.75
11.55
63311
13.40
53355
licio Sid--b-----ammllll Lannonn cia,, 2004)
14.05
28.43
27.00
15.30
68.00
65.39
23.42
43.00
47.03
66.26
32.0
1336
11.18
Crall-e ei 2004)
64.1
36.34
43.99
57.27
45.35
35.34
34.22
32.99
31.64
4431
43.66
43.67
Table 2. Performance of selected MLLMS (classified into proprietary. open spen-source and reasoning models) on the CoreCognition dataset. Best results are shown in bold; second- recond-best are underlined.
To assess free-form responses, we employ two-stage scor- ing process. Multiple-choice questions (MCQs) inherently contain predefined answers and choices. Thus, each MLLM response is first mapped to one of the options or marked as FAIL if unaligned. Mapping is achieved via a hybrid method combining template matching with LLM-as-a-Judge, pre- defined set of templates is utilized to directly match the model's output with one of the options. When template matching fails, an LLM will be used to determine the most suitable corresponding option. Models with a high FAIL rate are re-evaluated for validity and excluded from further analysis to avoid bias if they consistently produce nonsen- sical responses. In the second stage, the mapped choice compared against the ground-truth answer, with FAILs counted incorrect. See Appendix 1.2.1 for details, where we show consistent results with 4 alternatives.
have 230 models for subsequent analysis, among which 25 are proprietary, and 205 are open-source.
4.1. Core knowledge Deficits
Sonsorimotor
Concreto operation
Accuracy
Formal operation
Norralized
betale 7ay
boundary
continuity
peraanarce
opatality
constancy
intulibe physice
perspeciive
ennservatio
macchanccal reasoning
hierrrcy
intemtionality
loslueing
percaptual
4. Experiments
To comprchensively assess core knowledge in MLLMs, we meticulously selected and evaluated a diverse array of mod- els across various architectures and scales. The evaluated set prominently includes commercial models such the OpenAl and Claude series, high-performing open-source models like Intern VL (Zhu et al., 2025) and the Qwen series (Bai et al., 2025b), as well as recently introduced models from the DeepSeek (Lu et al., 2024a; Wu et al., 2024a) series, which have garnered considerable attention. The evaluated open source models range in size from 1 billion to 110 billion, with different design choices such dense (Vaswani et al., 2017: Devlin et al., 2019; Brown et al., 2020) and Mfxture-of- -ffeperss (Shazeer et al., 2017; Fedus et al., 2022), different vision encoders (e.g. Clip (Radford et al., 2021), SigLip (Zhai et al., 2023)). As discussed in Sec. 3.3 and Appendix 1.2.1, we additionally filtered out models that consistently produce invalid outputs. Ultimately, we
Figure 4. Accuracy by concept normalized by chance level. Evi- dence of core knowledge of deficits, with statistical significance.
As shown in Fig. 4, models exhibit a pronounced "core knowledge deficit": they perform significantly better on higher-level abilities (right side of the figure), comparable or even surpassing humans, but struggle with lower-level abili- ties (left side), associated with early developmental stages. This disparity is statistically significant and contrasts sharply with human performance, which remains consistently high across all stages. It's noteworthy that lower-level abilities are operational approximations of basic cognitive systems and are thus more directly aligned with the notion of "core knowledge", while higher-level abilities are more abstract or compositional cognitive tasks. The observed upward trend in performance does not imply that only lower-level abilities reflect core knowledge. Rather, it suggests that while mod-
5

---

Core Knowledge Deficits in Multi-Modal Language Models
els may perform better on higher-level tasks potentaoty by pattern matching or spurious correlation- often struggle with the more fundamental reasoning required for lower-level tasks. This gap implies the failure to demon- strate a robust understanding of the foundational abilities that higher-level tasks presuppose. Details on Fig. 4, ac- curacy normalization, and pairwise t-tests are provided in Appendix E. We further validate whether the observation holds under variety of different conditions in Appendix H.
Examining the interdependencies among core abilities pro- vides a principled understanding of whether models de- velop coherent, hierarchically structured competencies akin to those seen in humans. To quantify the degree of co- variation consistent with developmental hierarchies, we compute Pearson correlations between performances across all 12 abilities. The results reveal a distinct divergence: many correlations are modest (p 0.4). while some clus- ters exhibit strong alignment (p > 0.65). As illustrated in Fig. 5, we observe @ robust correlations among several high-level abilities, reflecting the anticipated interdepen- dence among tasks within the same developmental stage-a pattern notably absent in earlier stages of model perfor- mance. Notably, the Hierarchy ability clusters more closely with Formal Operational abilities, consistent with its strong overall performance and suggesting that models may treat it as advanced reasoning task. In contrast, 2 three Sen- sorimotor abilities (Permanence, Spatiality, and Continuity) exhibit weak correlations with most higher-stage abilities, implying that these foundational competencies do not pro- vide the developmental scaffolding to more advanced stages, which are typically observed in humans. @ Further evidence can be observed from three Concrete Operational abilities (Perspective, Conservation, and Intuitive Physics), which also show weak cross-stage correlations. Collectively, these results indicate that current models lack structured represen- tational dependencies, raising concerns about the grounding and internal coherence of their acquired abilitics(Spelke et al., 1992). We further validate whether the observation in Fig. 5 holds under a variety of conditions in Appendix I.
Tab. 2 compares the performance of 30 state-of-the-art MLLMs with human performance. All MLLMs substan- tially underperform relative to humans on lower-level stages (i.e. Sensorimotor and Concrete Operation excluding Hier- archy): the best models, GPT-o1, GPT-4o and Qwen2.5-VI. achieve average scores of 74.91%, %, 69.25%, and 68.29% trail- ing human performance by 15.91 %, 21.57%, and 22.53%, respectively. Notably, proprietary models do not consis- tently outperform open-source counterparts; for instance, GPT-4o outperforms Qwen2.5-VL-72B by only 1% on aver- age and Gemini and Claude series underperform Qwen2.5- VL-72B, RVQ-72BB-reviees and VI.3-7IL3-788 by 2-3% on average. This indicates that both proprietary and open- source MLLMs share the core knowledge deficit, underscor- ing a fundamental limitation across all models. Particularly, models perform markedly worse than humans on Perspec- tive Taking, likely reflecting their limited capacity for mental simulation, critical for understanding alternative viewpoints (Barnes-Holmes et al., 2004; Moll & Meltzoff, 2011; Bar- lassina& & Gordon, 2017). This highlights broader concerns regarding the absence of robust model-based reasoning in contemporary language models (Lake et al., 2017; Mitchell & Krakauer, 2023).
Key Finding 2 (Misaligned Dependency): Core abili- ties exhibit weak cross-stage correlations, indicating an absence of developmental scaffolding.
Key Finding I (Core Knowledge Deficits): MLLMs ex- cel at higher-level abilities associated with later develop- mental stages but consistently struggle with lower-level abilities that typically emerge earlier in human cognition.
4.3. Core Abilities are Predictive of Higher-level Abilities
To support the argument that core knowledge is essential for higher-level reasoning and perceptual abilities, we show in Fig. 6 that strong performance on core abilities reliably predicts higher performance most high-level abilities and benchmarks, such as SEEDBench2 (Li et al., 2024a). Con- cretely, we analyze the correlation between the performance of 12 core cognitive concepts across three stages and the performance of the same models on 26 public benchmarks and 9 higher-level abilities, as defined by SEED-Bench I (Li et al., 2023a) and 2(Li al., 2024a). Our findings re- veal that, except for perspective and Intuitive Physics, core abilities strongly predict performance on public benchmarks (except ChartQA) and performance of higher-level abili- ties in SEED-Bench We hypothesize that the exceptions of ChartQA arise because textual understanding is largely orthogonal to the core abilities examined here. Perspec- tive and Intuitive Physics tasks demand structured internal
4.2. Dependencies Between Core Abilities
Continulyy
03
Speta
Parceptall Comboray
tve Thyets
Perpedivee
Wederii Hamsingg
1:53
Intantona
300
1.59
Unine
1x
Hounsay
Paroppaal Comalecc
Plraace
Figure 5. Pearson Correlations Between Core Abilities.
6

---

Core Knowledge Deficits in Multi-Modal Language Models
Dounday
Contiruly
BCD
Persarens
Eprtaly
0:20
9:00
9.31
0:28
Percepionl
rodiis Physis
Coneenstoo
Hecority
Harbbraly
54
And
Figure 6. Correlation between core abilities and "high-level" abilities (left) and existing MLLM benchmarks (right).
representations and counterfactual abblliingabiiltiis that
for developing core knowledge in MLLMs. We further dis- cuss the generalization of this conclusion under different conditions in Appendix J.
underpin advanced reasoning in humans. The observed lack of correlation indicates that core knowledge deficits, as ev- idenced by dependencies between different abilities, are also reflected in benchmark evaluations. We further validate whether the observation in Fig. 6 holds under a different conditions in Appendix J.
Key Finding 4 (Not Scaling): MLLMs exhibit limited or no scalability on low-level abilities compared to high- level abilities
Key Finding (Predictability)) Performance on core knowledge is predictive of higher-level abilities.
Mechanical Roasoning
95%
Ainnow 0
4.4. Scaling Effect on Core Knowledge?
Accoussy
Not for low-level abilities! The advancement of LLMs has been driven by the empirical scaling law-predictable prwee--redictaale power-law improvements in perfor- mance with increased compute, parameters, and training data (Kaplan et al., 2020: Zhai et al., 2022; Henighan et al., 2020; Hoffmann et al., 2022)-an2 and emergence, the abrupt appearance of qualitatively new abilities as model scale in- creases (Wei al., 2022a; Aghajanyan et al., 2023; Bubeck et al., 2023; Berti et al., 2025). This raises a fundamental question: Does performance on core knowledge also emerge and scale as models increase in size? We evaluate the extent to which scaling applies to low-level core abilities rooted in core knowledge. By fitting a linear regression to the perfor- mance of 230 models of varying sizes on these abilities, we estimate the scaling effect as the slope of the regression line. As shown in Fig. 7, our results reveal clear dissociation be- tween low- and high-level abilities regarding scaling effects. For seven out of nine low-level Iellitess-xxluding hierar- chical relation and perceptual constancy-in the Sensorimo- tor and Concrete Operational Stages, model performance shows significantly less improvement with increasing size, compared to the higher-level Formal Operational Stage. No- tably, perspective-taking ability even declines with scale, likely due to persistent egocentric bias that intensifies as models grow larger. These findings indicate that scaling primarily benefits high-level reasoning, while its impact on low-level cognitive abilities is limited or even negative. This suggests that simply increasing model size is insufficient
Para BRooK Parspective
Parseeteri Kapditter Erh
Accuscc
Soaling Stops
0-0-374 e -a212v y-
Figure 7. Scaling Effect on core knowledge with respect to model size. Scaling laws do not apply uniformly across all concepts. Top Left. Fitted curves for each concept across 219 models and 11 prompt cases. Bottom Left. Comparison of core abilities' scalability (slopes of fitted curves). Right. Scaling curve for Mcchanical Reasoning/Persppectiveett represent models from the same series.
Dots of the same color
4.5. Does Reasoning Help?
Reasoning and test-time scaling are widely adopted in MLLMS and have demonstrated strong performance on complex benchmarks such as Math Math (Lu al., 2023), CLEVR (Johnson et al., 2017), and Geometry3K (Lu et al., 2021). One might hypothesize that these approaches enable more effective knowledge structuring, thereby improving performance on core knowledge tasks such as those in our benchmark. For example, in the large-arr--saaaaada per- ceptual constancy scenario shown in Fig. 9, models relying
7

---

Core Knowledge Deficits in Multi-Modal Language Models
on system-1 thinking may naively predict that the bridge narrows from direct perception, whereas models employing system-2 reasoning could override this intuitive illusion by more thinking and correctly infer that the apparent narrow- ing is merely a result of perspective.
their propensity exploit spurious features, where apparent task proficiency may stem from shortcut learning rather than genuine understanding (Alvi et al., 2018; Bahng et al., 2020; Cadene et al., 2020; Clark et al., 2019; Dagaev et al., 2021).
5.1. Methodology
To examine whether reasoning and test-time scaling en- hance performance on core cognitive abilities, we evaluate models and their correspond- ing instruction-tuncls counterparts on CoreCognition ques- tions including Kimi-VL-A3B--hinne Thinking. Kimi-VL-A3BB Instruct (Team et al., 2025b), ZVQ-72B-Previee (Team, 2024b), R1-Onevision-7B (Yang et al., 2025b), Llama-3.2V. 11B-cot (Xu et al.. 2024), Llama-3 1aama--.22111 3.2-11B- 22-11B-Vision(M Vieion(Meta, 2025), as well as full series of InternVI.3 (Zhu et al., 2025) and Saea---Thhnne--wee2//0
We introduce concept hacking, which systematically manip- ulates task-relevant features while preserving task-irrelevant conditions to completely invert ground truth labels. As exemplified in Fig. 9, 45 samples from CoreCognition are paired with a manipulated version containing identical questions but opposite correct answers. Given a pair of tasks, it yields four possible MI.LM response types (Tab. 3): The case where models answer controlled tasks incor-
both reasoning-augmente
(Chen et al., 2025) to Qwen2/2.5-VL (Wang et al., 2024a; Bai et al., 2025b).
Table 3. Four types of outcomes from MILMM.
Control
Manipelation
Interpectation
core knowlodge
shortcut
Nummntee Accurety F
core
deficits
Accunry
rectly but manipulation tasks correctly reflects coincidental accuracy- models lacking core knowledge produce wrong answers on controlled tasks, but inverted ground truth in manipulation tasks makes incorrect reasoning appear cor- rect, i.e. "being right for the wrong reason". See detailed explanation of concept hacking curation in Appendix K.
Seatringg
Figure& 8. Left. By concept comparison between reasoning models and their non-reasoning counterparts. Right. Comparison of the overall performance
Control
Manipulation
Continulty
Oiiiles
Eatly
Planece
As per Fig. 8, reasoning abilities and test-time scaling do not confer a clear advantage over instruction-tune -unoodd models. Overall, reasoning models show a modest, non-significant average improvement. The only two exceptions fail to hibit a consistent trend (perceptual constancy, where reason- ing models perform better (P = 0.0669), and perspective taking, where they perform worse (P- = 0.0037)). The ab- sence of improvement in lower-level abilitics underscores that current architectures struggle with grounded reason- ing. Similarly, the negligible gains observed on higher-level tasks such Intentionality Understanding and Tool Use suggest limited advantages from reasoning at near-ceiling performance. In contrast, reasoning models do not improve the relatively low scores in Mechanical Reasoning, likely be- cause this domain requires model- reasoning that can- not be addressed by chain-of th-thughhhht prompting or tist-time scaling (Hegarty, 2004; Mitchell, 2021). Notably, reasoning- augmented models exhibit narrower performance distribu- tion, indicating that explicit reasoning may stabilize outputs without resolving deeper representational deficits.
5mee Will the Hager ondi landriast
bo7) batlo are crepped at the
60-0
Constoncy
Does the actual wish the brigge
Ne
GFF
OL
GFF
Perppection
From peropection uo ter imageie rotuned 50* cloctutto doee duckkrr
Figure 9. Examples of Concept Hacking.
5.2. Results: Core Deficits v.s. Shortcut Taking
The results reveal a clear separation between models ex- hibiting shortcut and those with core deficits (Fig. 10), with A substantial proportion of models clustered in the top left quadrant (high manipulation accuracy, below-chance control accuracy, consistent with findings in See. 4.1), and a signifi- cant portion of models appeared in the bottom right quadrant (high control accuracy, below-chance manipulation accu- racy), reflecting a pronounced reliance on shortcuts with a high susceptibility to manipulation. Most models demon- strated above-chance performance on both tasks but still fell short of human- levell core knowledge. Unlike humans, al- most none of these models achieved roughly equal accuracy
5. Concept Hacking: A Controlled Experiment
Do MLLMs genuinely possess core knowledge? A funda- mental challenge in evaluating the abilities of MILMs is
8

---

Core Knowledge Deficits in Multi-Modal Language Models
14
Cove Necosledee
2024b), and resonates with ongoing critiques that foundation models fail to develop genuine conceptual understanding, instead reinforcing shortcut-based strategies as they scale (Bender et al., 2021; Mitchell & Krakauer, 2023).
Human
Co atidd be Marl ded
Oefictts BlLL
Manig Maniputation duttion Accuracy
Our findings that current training paradigms fail to instill core cognitive abilities underscore the need for pretraining strategies that explicitly target these foundational capacities, More specifically, if core knowledge cannot be directly ac- quired through scaling, it may be beneficial to first teach or distill core knowledge into MLLMs prior to large-scale pretraining, thereby enabling more data-efficient generaliza- tion akin to human learning. From both an evaluation and design perspective, our benchmark reveals distinct failure modes deficits in permanence, spatiality, bound- ary, and continuity. These limitations further hinder abilities such as visual perspective takinegttangniggg and contribute an over- reliance on shortcuts, which is a fundamental cause of poor out-of til-off-ditributton generalization.
Shortcut Blor
Core Defatts
Bind
18 Rassnn
Controll Accuracy
Figure 10. Control vs. Manipulation accuracy. Circle: model per- formance; Size of circle: parameter size of model; Color of circle: averaged accuracy on CoreCognition; Green triangle: human performance; Red dot: chance level point on both Control and Manipulation, termed as "blind". As size increases, core deficits and shortcut reliance intensify, rather than progressing toward the human-like core knowledge region.
One possible objection is that human-like core knowledge is not essential for artificial general intelligence (AGD, even when AGI is defined in relation to human-level in- telligence. Intelligence, after all, may be multiply realiz- cole-acchievablee through architectures and developmental paths distinct from those of humans (Bechtel & Mundale, 1999). However, core knowledge may embody fundamen- tal learning principles that recur across intelligent agents, including non-human animals (Santos, 2004; Lake et al., 2017). If such theory holds, non-human pathways to AGI, e.g. scaling, will also lead to the emergence of these core abilities. In this light, benchmark on core knowledge, such as ours, could offer useful lens for evaluating the progress toward AGI, regardless of the path taken. In light of ongoing uncertainty about the path towards AGI, the human devel- opmental trajectory offers a valuable, empirically grounded reference point. Despite great advancements, MLLMs con- sistently struggle with hallucinations, poor generalization, and lack of robustness, suggesting that key cognitive in- gredients may still be lacking. By aligning evaluation with structures known to support robust reasoning and perception in humans, our framework helps expose and address these critical gaps in emerging Al systems.
on both tasks-a a hallmark of robust core knowledge and immunity to concept hacking. This pattern suggests that, while some models are not entirely dominated by shortcut- taking or lack of core knowledge, these factors still substan- tially affect their predictions. Interestingly, susceptibility to concept hacking does not correlate straightforwardly with model size or performance on CoreCognition. While many shortcut-reliant models were smaller and weaker, the bot- tom right quadrant also include some of the largest and best-performing models, such as GPT-4o. Similarly, models with core deficits in the top left quadrant varied in size and performance. In line with previous findings on the non- scaling of low-level abilities, increasing model size does not inherently improve core knowledge, but rather enhances shortcut-taking or the persistence of core deficits.
Key Finding 5 (Deficits v.s. Shortcut Taking): Mod- els increasing in size exhibit deficits and shorteut-taking behaviors rather than progressing toward conceptual un- derstanding of core knowledge.
7. Conclusion
6. Discussion
We introduce CoreCognition benchmark paired with novel Concept Hacking method to rigorously and control- lably evaluate the conceptual understanding of core knowl- edge in MLLMs. We uncover four key findings that collec- tively indicate a consistent lack of core knowledge in current MILMs- fundamental understanding of basic world con- cepts such as objects, actions, numbers, space, and social relations- that humans acquire from infancy. We also dis- cuss potential limitations in Appendix M.
Our findings (1-3) support the hypothesis that MLLMS lack core knowledge that grounds the high level perception and reasoning abilities, and that such core abilities cannot be acquired through scaling alone (4-5). This offers explanation for the longstanding Moravec's paradox that tasks intuitive and effortless for humans often prove to be the most challenging for machines (Moravec, 1988). It also offers a plausible account for the lack of robustness in current MIL.Ms (Shiffrin & Mitchell, 2023; Zhang et al.,

---

Core Knowledge Deficits in Multi-Modal Language Models
Impact Statement
answering. In Proceedings of the IEEE international conference computer vision, pp. 2425-2433, 2015.
We aim to advance the development of MLLMs through a cognitively grounded evaluation framework, CoreCogni- tion. Our findings reveal that MLLMs lack conceptual un- derstanding of core knowledge, cautioning against overinter- preting their success on complex tasks. The curation method employed in CoreCognition offers a scalable methodology that may inform future benchmarking of foundation models in large scale. Our findings also shed light on the design of more robust and interpretable models with grounded rea- soning and perception capabilities. The adversary idea in Concept Hacking may pose misuse risks if adapted for mili- tary purposes.
Bahng, H., Chun, S., Yun, S., Choo, J., and Oh, S. J. Learn- ing De-biased Representations with Biased Representa- tions, June 2020.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W. Han, Y., Huang, F,, et al. Qwen technical report. arXiv preprint arXiv:2309.
2023a.
Bai, J.. Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versa- tile vision-language model for understanding, localiza- tion, text reading, and beyond, 2023b. URL https: / //arxiv. org/
wwt/abs/////eoooo//a/tw..tt..aee aa/////////// 2308 12966.
References
Abdin, M., Aneja, J.. Behl, H., Bubeck. S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmaan, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905 2024.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang. P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang. M., Li, Z., Wan, J., Wang. P., Ding. W., Fu, Z., Xu, Y. Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z. Xu, H., and Lin, J. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025a.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. 2025b.
Aghajanyan, A., Yu, L.,, Conneau, A., Hsu, W.-N., Ham- bardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-modal language models. In International Confer- ence on Machine Learning, pp. 265- 279.. PMLR, 2023.
Bai, Z., Wang, P., Xiao, T., He, T., Han. Z., Zhang, Z., and Shou, M. Z. Hallucination of multimodal large language models: survey. arXiv preprint arXiv:2404.18930. 2024.
Agrawal, P., Antoniak, S., Hanna, E. B., Bout, B., Chap- lot, D., Chudnovsky, J., Costa, D., De Monicault, B., Garg, S., Gervet, T. et al. Pixtral 12b. arXiv preprini arXiv:2410.07073, 2024.
Baillargeon, R. Representing the existence and the location of hidden objects: Object permanence in 6-and 8-month- old infants. Cognition, 23(1):21-44 1986.
Allen, K. R., Smith, K. A., and Tenenbaum, J. B. Rapid trial- and-error learning with simulation supports flexible tool use and physical reasoning. Proceedings of the National Academy of Sciences, 117(47):29322-2933 2020.
Baillargeon, R. and Carey, S. Core cognition and beyond: The acquisition of physical and numerical knowledge. Early childhood development and later outcome, 1, 2012.
Alvi, M., Zisserman, A., and Nellaker, C. Turning Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings, September 2018.
Baillargeon, R., Spelke. S., and Wasserman. S. Object permanence in five-month-old infants. Cognition, 20(3): 191-208, 1985.
Andrews, G. and Halford, G. S. Children's 's ability to make transitive inferences: The importance of premise integra- tion and structural complexity. Cognitive Development, 13(4):479-55 1998.
Baker, C. L., Saxe, R., and Tenenbaum, J. B. Action under- standing as inverse planning. Cognition, 113(3):329-349, 2009.
Barlassina, L. and Gordon, R. M. Folk psychology as mental simulation. In Zalta, E. N. (ed.), The Stanford Encyclope- dia of Philosophy. Metaphysics Research Lab, Stanford University, summer 2017 editionn 2017. URL ht https: plato. wtw.oostttoood..duarr..eeee entries/ Saariies/Folkpsycc-ssmmmmll
Anthropic. Claude 3.5 sonnet. https:/ www. anthropic. com ccom/news// news//wwww////eeeee hats///aaueee33 -onde-3-5-somme
June 2024. Accessed 2025-06-08.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.,, and Parikh, D. Vqa: Visual question
sum2017/
10

---

Core Knowledge Deficits in Multi-Modal Language Models
Barnes-Holmes, Y., McHugh, L.,, and Barnes-Holmes, D. Perspective-taking and theory of mind: A relational frame account. The Behavior Analyst Today, 5(1):15-25, 2004.
Bubeck, S., Chadrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, T., Li, Y., Lund- berg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Barrouillet, P. Theories of cognitive development: From piaget to today, 2015.
Byrne R. M. Counterfactual thought. Annual review of psychology, 67(11:135-155 157, 2016.
Barsalou, L. W. Challenges and opportunities for grounding cognition. Journal of Cognition, 3(1), 2020.
Cadene, R., Dancette, C., Ben-younes, H., Cord, M., and Reducing Unimodal Biases in Visual Question Answering, March 2020.
Parikh, D. RUBi:
Bechtel, W. and Mundale, Multiple realizability revis- ited: Linking cognitive and neural states. Philosophy of science, 66(2):175-207, 1999.
Carey, S., Zaitchik, D., and Bascandziev, I. Theories of development: In dialog with jean piaget. Developmental Review, 38:36-54, 2015.
Bell, M. A. and Adams, S. H. Comparable performance on looking and reaching versions of the a-not-b task at 8 months of age. Infant Behavior and Development, 22(2): 221-235, 1999.
Caviola, L., Schubert, S., and Greene, J. D. The psychology of (in) effective altruism. Trends in Cognitive Sciences, 25(7):596-607, 2021.
Bellmund, J. I.., Gärdenfors, P., Moser, I., and Doeller, C. F. Navigating cognition: Spatial codes for human thinking. Science, 36)(64415) 2018.
Cesana-Arlotti, N., Martín, A., Téglás, E., Vorobyova, L., Cetnarski, R., and Bonatti, L. L. Precursors of logical reasoning in preverbal human infants. Science, 359(6381); 1263-1266, 2018.
Bender, E. M., Gebru, T., McMillan-Majjo A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and trans- parency, pp. 610-623, 2021.
Chalmers, D. J. Why fodor and pylyshyn were wrong: The Proceedings of the Annual Meeting of the Cognitive Science Society, volume 12, 1990.
Berkeley, G. An Essay Towards A New Theory of Vision. Dublin, 1709.
Chalmers, D. J. Syntactic transformations on distributed representations. Connectionist Natural Language Pro- cessing: Readings from Connection Science, pp. 46-55, 1992.
Bertenthal, B. I., Gredebäck, G., and Boyer, T. W. Differen- tial contributions of development and learning to infants' knowledge of object continuity and discontinuity. Child Development, 84(2):413-421, 2013.
Chapman, M. and McBride, M. L. Beyond competence and performance: Children's class inclusion strategies, superordinate class cues, and verbal justifications. Devel- opmental Psychology, 28(2):319, 1992.
Berti, L., Giorgi. F., and Kasneci, G. Emergent abilities in large language models: A survey. arXiv preprint 250:02503,05788, 2025.
Chen, H., Tu, H., Wang, F., Liu, H., Tang, X., Du, X., Zhou, Y., and Xie, C. Sft or rl? an early investigation into training r1 rike reasoning large vision-language models. arXiv preprint mrXiv:2504.11666 2025.
Borst, G., Poirel, N., Pineau, A., Cassotti, M., and Houdé, O. Inhibitory control efficiency in piaget-like class- inclusion task in school-age children and adults: de- velopmental negative priming study. Developmental psy- chology, 1967): 2013.
Church, R. B. and Goldin-Meadow, S. The mismatch be- tween gesture and speech as an index of transitional knowledge. Cognition, 23:43-71, 1986.
Bremner, J. G., Slater, A. M., and Johnson, S. Perception of object persistence: The origins of object permanence in infancy. Child Development Perspectives, 911::7113, 2015.
Clark, C., Yatskar, M., and Zettlemoyer, L. Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases, September 2019.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Craig, G. J., Love, J. A., and Olim, E. G. An experimental test of piaget's 's notions concerning the conservation of quantity in children. Child Development, 44(2):372-375, 1973.

---

Core Knowledge Deficits in Multi-Modal Language Models
Dagaev, N., Roads, B. D., Luo, X., Barry, D. N., Patil, K. R., and Love, B. C. A Tto-Good--to-h-TTTe Prior Reduce Shortcut Reliance, October 2021.
Gu, X., Zheng, X., Pang, T., Du, C., Liu, Q. Wang, Y., Jiang, J., and Lin, M. Agent smith: A single image can jailbreak one million multimodal IIm agents exponentially fast. arXiv preprint arXiv:2402.08567 2024.
De Waal, F. B. and Preston, S. D. Mammalian empathy: behavioural manifestations and neural basis. Nature Re- views Neuroscience, 18(8):498-509 2017.
Halford, G. S. An experimental test of piaget's notions con-
cerning the conservation of quantity in children. Journal of experimental child psychology, 6(1):33-4) 43, 2011.
Devlin, J., Chang. M.-W., Lee. K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguisties: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Halford, G. S., Wilson, W. H., and Phillips, S. Processing ca- pacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and brain sciences, 2 21(6):803-311, 1998.
Hannagan, T., Amedi, A., Cohen, L.,, Lamberz,, G., and Dehaene, S. Origins of the specialization for letters and numbers in ventral occipitotemporal cortex. Trends in cognitive sciences, 19(7):374 382, 2015.
Dong, H., Liu, M., Zhou, K., Chatzi, E., Kannala, J., Stach- niss, C., and Fink, O. Advances in multimodal adaptation and generalization: From traditional approaches to foun- dation models. arXiv preprint arXiv:2501.18592, 2025.
Hegarty, M. Mechanical reasoning by mental simulation. Trends in cognitive sciences, 866):280-285 2004.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C.,
Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Feigenson, L., Dehaene, S., and Spelke, Core systems of number. Trends in cognitive sciences, 877)307-314, 2004.
Hermer, L. and Spelke, E. Modularity and development:
The case of spatial reorientation. Cognition, (3):
232, 1996.
Fodor, A. The Language of Thought. MIT Press, 1975.
Hinton, G. E. et al. Learning distributed representations of concepts. In Proceedings of the eighth annual confer- ence of the cognitive science society, volume 1, pp. 12.
Fodor, J. A. LOT 2: The language of thought revisited. Oxford University Press, 2008
Amherst, MA, 1986.
Fowler, H. N., Lamb, W. R. M., et al. Plato, volume 5. W. Heinemann, 1914.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. 2022.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models, arXiv preprint arXiv: 13394, 2023.
Houdé, O. Numerical development: From the infant the child. Cognitive Development, 12(3):373-391, 1997.
Gandhi, K., Stojnic, G., Lake, B. M., and Dillon, M. R. Baby intuitions benchmark (bib): Discerning the goals, preferences, and actions of others. Advances in neural information processing systems, 34:9963-9976, 2021.
Houdé, O., Pineau, A., Leroux, G., Poirel, N., Perchey, G., Lanoë, C., Lubin, A., Turbelin, M.-R., Rossi, S., Simon, G., Delcroix, N., Lamberton, F., Vigneau, M., Wisniewski, G., Vicet, J.-R., and Mazoyer, B. Func- tional magnetic resonance imaging study of piaget's conservation-oo-nunii task in preschool and school-age children: neo-piagetian approach. Journal of experi- mental child psychology, 110(3)3332-343) 2011.
Gemini. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv: 2312.11805 2023.
Gibson, J. J. The Ecological Approach to Visual Perception. (1st e..). Psychology Press, 1979.
Huang, J. and Zhang. J. A survey on evaluation of mul- timodal large language models, 2024. URL https: / //axxiv.org/abs/240000
Green, E. Perceptual constancy and perceptual representa- tion. Analytic Philosophy, 2023.
12

---

Core Knowledge Deficits in Multi-Modal Language Models
Huitt, W. and Hummel, J. Piaget's theory of cognitive development. Educational psychology interactive, 3(2): 1-5, 2003.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint S.kiv::2001088366. 2020.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., and DiCarlo, J. J. Evidence that recurrent circuits are critical to the ventral stream's s execution of core object recognition be- havior. Nature neuroscience, 22(6):974 983, 2019.
Iacoboni, M. Imitation, empathy, and mirror neurons. An- nual review of psychology, 60(1):653-670 2009. Inhelder, B. and Piaget, J. The Growth of Logical Thinking from Childhood to Adolescence. Basic Books, 1958.
Kestenbaum, R., Termine, N., and Spelke, E. S. Perception of objects and object boundaries by 3-month-old infants. British journal of developmental psychology, 5(4):367- 383, 1987.
Jackendoff, R. Parts and boundaries. Cognition, 41(1-3): 9-45, 1991.
Khang. B.-G. and Zaidi. Q. Illuminant color perception of spectrally filtered spotlights. Journal of Vision, 4(9):2-2, 2004.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., al. Openai ol system card. arXiv preprint arXiv:2412. 16720, 2024.
Kilner, J. M. More than one pathway to action understand- ing. Trends in cognitive sciences, 15(8):352-335 2011.
Janet, P. Mental pathology. Psychological Review, 12(2-3): 98, 1905.
Kim, B., Kim, H., Kim, K., Kim, S., and Kim, Learn- ing Not to Learn: Training Deep Neural Networks with Biased Data, April 2019.
Jatanerti J.. Schulz, L. E., and Tenenbaum, J. B. The naive utility calculus as unified, quantitative framework for action understanding. Cognitive Psychology, 123: 101334, 2020.
Kim, L.-K. and Spelke, E. S. Perception and understand- ing of effects of gravity and inertia on object motion. Developmental Science, 2(3):339-362, 1999.
Jangg A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv preprint arXiv: 2310.06825, 2023.
Kirkpatrick, E. operations. The Journal of Philosophy, Psychology and Scientific Methods, 5(16):421 42944299,9 1908.
The part played by consciousness in mental
Kuhn, D. and Angelev, J. An experimental study of the development of formal operational thought. Child Devel- opment, pp. 697-706, 1976.
Jang. Y., Zhang, J., Sun, K., Sourati, Z., Ahrabian, K., Ma, K., Ilievski, F., and Pujara, J. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, vol- ume 37, pp. 46567-46592. Curran Associates, Inc., 2024.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Laurençon, H., Marafioti, A., Sanh, V., and Tronchon, L. Building and better understanding vision-language mod- els: insights and future directions., 2024.
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-22900 2017.
Le Poidevin, R. Continuants and continuity. The Monist, 83 (3):881-398, 2000.
Lee, J., Kim, E., Lee, J., Lee, J., and Choo, J. Learning Debiased Representation via Disentangled Feature Aug- mentation, October 2021.
Jowett, B. et al. The Republic of Plato. Clarendon press, 1888.
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with gener- ative comprehension. arXiv preprint arXiv:2307.16125 2023a.
Kant, I. Critique of pure reason. 1781. Modern Classi- cal Philosophers, Cambridge, MA: Houghton Mifflin, pp- 370-456, 1781.
13

---

Core Knowledge Deficits in Multi-Modal Language Models
Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., and Shan, Y. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299-13300 2024a.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, and world knowledge, January 2024a. URL -10--00---llaa----
wal.aaaal.iiiuu.bb....... wwt.hggtthooooooo io/blog///
Liu, H., Li, C., Wu, Q.. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems,
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang. H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer, 2024b. URL orgoorgg////o
36, 2024b.
03326.
Liu. S., Ullman, T. D., Tenenbaum, B., and Spelke, E. S. Ten-month-old infants infer the value of goals from the costs of actions. Science, 358(6366): 10::- 2017.
Li, C., Nuttall, R. L., and Zhao, S. A test of the piagetian water-level task with chinese students. The Journal of Genetic Psychology, 160(3):369-380, 1999.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:230..06281, 2023a.
Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., Zhang, J., Huang, S., Huang, F., Zhou, J., and Si, L. mplug: Effective and efficient vision vision-language language learning by cross modal skip- connections. arXiv preprint arXiv: 2205. 12005, 2022a.
Liu, Y., Li, Z., Yang, B., Li, C., Yin, X., Liu, C.-l,, Jin, L., and Bai, X. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.0777 07895, 2023b.
Li, J., Wang, J., Zhang, Z., and Zhao, H. Self Self-promp large language models for zero-shot open-domain qa. arXiv preprint arXiv:2212.086635 2022b.
Liu, Y.. Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024c.
Li, Li, D., Savarese, S., and Hoi, S. 2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 17730-19742. PMLR, 2023b.
Ltd., D. Mmalaya2. DataCanvas/MMaLLe
htlps:://hugginggaccee. co/
2024.
Li,J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. CONFERENCE, 2023c.
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., Sun, Y.. Deng, C., Xu, H., Xie, Z., and Ruan, C. Deepseek-vl: Towards real-world vision-language understanding, 2024a.
Li, Y., Guo, H., Zhou, K., Zhao, W. X., and Wen, J.-R. Images are achilles' heel of alignment: Exploiting visual vulnerabilities for jailbreaking multimodal large language models. In European Conference on Computer Vision, pp. 174-189. Springer, 2024c.
Lu, P., Gong, R., Jiang, S., Qiu, I.., Huang, S., Liang, X., and Zhu, S.-C. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IICNLPP 2021), 1, 2021.
Lin, J., Yin, H., Ping, W., Molchanov, P., Shocybi, M., and Han, S. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26689-26699, 2024.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: multimodal reasoning via thought chains for science ques- tion answering. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Lin, T.-Y., Maire, M., Belongie, S. J., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European Confer- ence on Computer Vision, 2014. URL https://pp/a.. semanticscholar. org.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models visual contexts. arXiv preprint arXiv:23 2023.
Liu, D., Wellman, H. M., Tardif, T., and Sabbagh, M. A. Theory of mind development in chinese children: a meta- analysis of false-belief understanding across cultures and languages. Developmental psychology, 44(2):523, 2008.
14

---

Core Knowledge Deficits in Multi-Modal Language Models
Lu, S., Li, Y., Chen, Q-G., Xu, Z.,, Luo, W., Zhang, K., and Ye, H.-J. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797 2024b.
objecfivity, Issues in philosophy and psychology, pp. 286- 304, 2011.
Moravec, H. Mind children: The future of robot and human intelligence. Harvard University Press, 1988.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEE/cvf conference computer vision and puttern recognition, pp. 3195-32064 2019.
Murphy, G. L., and Lassaline, M. E. Hierarchical structure in
concepts and the basic level of categorization. In Knowl- edge Concepts and Categories, pp. 93-131. Psychology Press, 2013.
Nam, J., Cha, H., Ahn, S., Lee, J., and Shin, J. Learning from Failure: Training Debiased Classifier from Biased Classifier, November 2020.
Marwaha, S., Goswami, M., and Vashist, B. Prevalence of principles of piaget's theory among 4-7-year-old children and their correlation with iq. Journal of clinical and diagnostic research: JCDR, 118)):2C111, 2017.
Newcombe, N. S. and Sluzenski, J. Starting points and change in early spatial development. Human Spatial Memory, pp. 25-40, 2004.
Masry, A., Long, D. X., Tan, J. Q.. Joty, S., and Hoque, E. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Findings of the Association for Computational Linguistics: ACI. 2022, pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/112022. findings-acle 177. URL https: /acan/tholan org/ 2022. findingss meddings-all... /
Ninomiya, T., Noritake, A., Kobayashi, K., and Isoda, M. A causal role for frontal cortico-cortical coordination in social action monitoring. Nature communications, 11(1): 5233, 2020.
Oh, C., Fang. Z., Im. S., Du, X., and Li, Y. Un- derstanding multimodal llms under distribution shifts: An informationn-hhorrei approach. arXiv preprint 2AXh:22502.0057,,2223.
Meltzoff, A. N. Origins of theory of mind, cognition and communication. Journal of communication disorders, 32 (4):251-269, 1999.
OpenAL. 1lms.
Learning to reason with https: / /twp////p/aaaa///////e//......aa i com/ nddx/ learning-to-
Meta. Meta Llama. hetps://huggingface co/ meta-llame 2025.
tims/, September
2024.
Michotte, A. The perception of causality. Basic Books, 1963.
O'Brien, T. C. and Shapiro, J. The development of logical thinking in children. American Educational Research Journal, 5(4):531-542, 1968.
Miller, P. H. Theories of developmental psychology (6th ed.). Macmillan Higher Education, 2016.
Paiss, R., Ephrat, A., Tov, O., Zada, S., Mosseri, I., Irani, M., and Dekel, T. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3170-3300 -1800, 2023.
Mitchell, M. On crashing the barrier of meaning in artificial
intelligence. Al magazine, (1 92, 2020.
Mitchell, M. Why ai is harder than we think. arXiv preprint arXiv:2104.12871, 2021.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4, arXiv preprint urXiv:2304.03277, 2023.
Mitchell, M. and Krakauer, D. C. The debate over under- standing in ai's large language models. Proceedings of the National Academy Sciences, 120(13)x2221590771) 2023.
Piaget, J. The Psychology of Intelligence. Harcourt, Brace, 1950.
Piaget, J. The Origins of Intelligence in Children. Interna- tional Universities Press, 1952.
Mitra, C., Huang, B., Darrell, T., and Herzig, R. Composi- tional -hain-of-thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14420- 14431, 2024.
Piaget, J. and Inhelder, The Psychology of the Child. Basic Books, New York, 1969.
Piaget, J. and Inhelder, B. The Child's Construction of Quan- tities: Conservation and Atomism. Psychology Press, 1974.
Moll, H. and Meltzoff, A. N. Perspective-taking and its foundation in joint attention. Perception, causation, and
15

---

Core Knowledge Deficits in Multi-Modal Language Models
Piaget, J. and Inhelder, B. Intellectual operations and their development. In Experimental Psychology Its Scope and Method: Volume VII (Psychology Revivals), pp. 144-205. Psychology Press, 2014.
Rutherford, M. and Brainard, D. Lightness constancy: A direct test of the illumination-estimatin hypothesis. Psy- chological Science, 13(2):142-149, 2002.
Sahin, U., Li, H., Khan, Q., Cremers, D., and Tresp, V. Enhancing multimodal compositional reasoning of vi- sual language models with generative negative mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5563-5573, 2024.
Piloto, L. S., Weinstein, A., Battaglia, P., and Botvinick, M. Intuitive physics learning in deep-learning model inspired by developmental psychology. Nature human behaviour, 6()): 2022.
Santos, L. R. 'core knowledges': A dissociation between spatiotemporal knowledge and contact-mechanies in non-human primate?, 2004.
Plato, Spens, H., and Adams, J. The Republic of Plato. Printed by Robert and Andrew Foulis, Glasgow, 1763.
Plummer, B. A., Wang, L.., Cervantes, C. M., Caicedo, J. C.,, Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-tur coiin-tt-ppuee to-phrase correspondences for richer image-to-sentence models. In 2015 IEEE International Conference on Computer Vision (ICCV). pp. 2641-2649, 2015. doi: 10.1109/1CCV22055..328
Saxena, R., Gema, A. P., and Minervini, P. Lost in time: Clock and calendar understanding challenges in multi- modal llms. arXiv preprint arXiv:2502.05009, 2025.
Schlarmann, C. and Hein, M. On the adversarial robustness of multi-modal foundation models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3677-3685, 2023.
Poirel, N., Borst, G., Simon, G., Rossi, S., Cassotti, M., Pineau, A., and Houdé, O. Number conservation is related to children's prefrontal inhibitory control: fmri study of a piagetian task. PloS one, 7(7):440802, 2012.
Scott, R. M. and Baillargeon, R. Early false-belief under- standing. Trends in cognitive sciences, 21(4):237-249, 2017.
Politzer, G. The class inclusion question: a case study in ap- plying pragmatics to the experimental study of cognition. SpringerPlus, 5(1): 2016.
Searle, J. R. The intentionality of intention and action. Inquiry, 22(1-4):253-281) 1979.
Shao, H., Qian, S., Xiao, H., Song, G., Zong. Z., Wang, L.. Liu, Y., and Li, H. Visual cot: Unleashing chain- of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403. 16999, 2024.
Pylyshyn, Z. W. Computation and cognition: Issues in the foundations of cognitive science. Behavioral and Brain sciences, 3(1):111-133) 11-132, 1980.
Qharabagh, M. F., Ghofrani, M., and Fountoulakis, K. Lvim- count: Enhancing the counting ability of large vision- language models. arXiv preprint arXiv:2412.00666, 2024.
Shayer, M. Has piaget's construct of formal operational thinking any utility? British Journal of Educational Psychology, 49(3):265-276, 1979.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint /rXiv:/701.06:: 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 2103.00020, 2021.
Shiffrin, R. and Mitchell, M. Probing the psychology of ai models. Proceedings of the National Academy of Sci- ences, 120(10):e2300661 2023.
Rahmanzadehgervv. P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind. arXiv preprint arXiv:2407.06588. 2024.
Shipley, F. The class-inclusion task: Question form and distributive comparisons. Journal of Psycholinguistic Research, 8:301-331, 1979.
Rochat, P. The evolution of developmental theories since pi- aget: A metaview. Perspectives on Psychological Science, 19(6):921-930 2024.
Sicgal, M. and Varley, R. Neural systems involved in theory mind'. Nature Reviews Neuroscience, 3(6):463-471, 2002.
Rosenthal, D. M. The Nature of Mind. Oxford University Press, New York, 1991.
Song, W. Li, Y. Xu, J. Wu, G., Ming, L., Yi, K., Luo, W., Li, H., Du, Y., Guo, F,, and Yu, K. M3gia: cognition inspired multilingual and multimodal general
Russell, B. History of western philosophy. Routledge, 1946.
16

---

Core Knowledge Deficits in Multi-Modal Language Models
intelligence ability benchmark, 2024,
URL https:
Team, C. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv: 2405.098/8, 2024a.
//arxiv.org/////.. abs 2406. 05343.
Spelke, E. What babies know: Core Knowledge and Com- position volume 1, volume 1. Oxford University Press, 2022.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Spelke, E. S. Core knowledge, language, and number. Lan- guage Learning and Development, 13(2):147-170, 2017.
Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rívière, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503 2025a.
Spelke, E. S. and Kinzler, K. D. Core knowledge. Develop- menial science, 10(1):89-96, 2007.
Spelke, E. S., Breinlinger, K., Macomber, J., and Jacobson, K. Origins of knowledge. Psychological review, 99(4): 605, 1992.
Team, K., Du, A., Yin, B., Xing, B., Qu, B., Wang, B., Chen, C., Zhang, C., Du, C., Wei, C., Wang, C., Zhang, D., Du, D., Wang, D., Yuan, E., Lu, E., Li, F., Sung, F,, Wei, G., Lai, G.. Zhu, H., Ding, H., Hu, H., Yang. H., Zhang. H., Wu, H., Yao, H., Lu, H.,, Wang. H., Gao, H., Zheng. H., Li, J., Su, J., Wang. J., Deng, J., Qiu, J., Xie, J., Wang, J., Liu, J., Yan, J., Ouyang, Chen, I.., Sui, I.., Yu, L., Dong, M., Dong, M.. Xu, N., Cheng, P., Gu, Q. Zhou, R., Liu, S., Cao, S., Yu, T., Song, T., Bai, T. Song, W. He, w., Huang, W., Xu, W., Yuan, X., Yao, X., Wu, X., Zu, X., Zhou, X., Wang. X., Charles, Y., Zhong, Y., Li, Y., Hu, Y., Chen, Y., Wang, Y., Liu, Y., Miao, Y., Qin, Y. Chen, Y., Bao, Y. Wang, Y., Kang, Y., Liu, Y., Du,
Spelke, E. S., Katz, G., Purcell, S. E., Ehrlich, S. M., and Breinlinger, K. Early knowledge of object motion: Conti- nuity and inertia. Cognition, 51(2):131-176, 1994.
Spelke, E. S., Kestenbaum, R., Simons, D. J., and Wein, D. Spatiotemporal continuity, smoothness of motion and ob- ject identity in infancy. British journal of developmental psychology, 13(2):113-143 1995.
Sun, Q. Cui, Y., Zhang, X., Zhang, F., Yu, Q. Luo, Z., Wang, Y., Rao, Y., Liu, J., Huang, T., and Wang, X. Generative multimodal models are in-context learners. Computer Vision and Pattern Recognition, 2023. doi: 10.1 (100//CVPR2273..222......
Y. Wu, Y., Wang, Y., Yan, Y., Zhou, Z., Li, Z.,, Jiang, Z., Zhang, Z., Yang. Z., Huang, Z., Huang, Z., Zhao, Z., and Chen, Z. Ki technical report, 2025b. URL https:
Sun, Q. Cui, Y., Zhang, X., Zhang, F., Yu, Q. Luo, Z., Wang, Y., Rao, Y., Liu, J., Huang, T., and Wang, X. Gen- erative multimodal models are in-context learners, 2024. URL https: / /wwxx/ixvvv.////// http//awwwrrg////eee//// at///bb//33/3 2312. 13286.
075440
Team, Q. Qvq: To see the world with wisdom, Decem- ber 2024b. URL battps://qwentt github. io/ blog/ /togg//vq-1000 wwwp-/2b-preieeeeeeeeeeeeeeeeeee
Tan, A. W. M., Yu, S., Long, B., Anya, W., Murray, T., Silverman, R. D., Yeatman, J. D., and Frank, M. C. De- vbench: A multimodal developmental benchmark for lan- guage learning. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 77445-77467. Curran Associates, Inc., neurigs..........
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023.
Ullman, T. D. and Tenenbaum, J. B. Bayesian models of conceptual development: Learning as building models of the world. Anmual Review of Developmental Psychology, Past--aaperr-rre Dat 261::533-558, 2020.
2024. URL starr/flllsssappre2////11111 cc ca/ www.fileeppppppeeeee 8d98 71111/00111101000000000000000000 and Bww.BBenchhhhheeeeeeeeee Trackaaaaaak_Traa.d.......
Vasta, R. and Liben, L. S. The water-level task: An intrigu- ing puzzle. Current Directions in Psychological Science, 166) 1996.
Tang, K., Gao, J., Zeng, Y., Duan, H., Sun, Y., Xing, Z., Liu, W., Lyu, K., and Chen, K. Lego-puzzles: How good are milms at multi-step spatial reasoning? arXiv preprint arXiv:2503. 19990, 2025a.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.., and Polosukhin, I. At- tention is all you need, Advances in neural information processing systems, 30, 2017.
Tang, Y., Liu, P., Feng, M., Tan, Z., Mao, R., Huang, C.,, Bi, J., Xiao, Y., Liang, S., Hua, H., et al. Mmperspec- tive: Do milms understand perspective? a comprehensive benchmark for perspective perception, reasoning, and robustness. arXiv preprint orXiv:2505.22 204264 2025b.
Viarouge, A., Houdé, O., and Borst, G. The progressive 6- year-old conserver: Numerical saliency and sensitivity as
17

---

Core Knowledge Deficits in Multi-Modal Language Models
core mechanisms of numerical abstraction in piaget-like estimation task. Cognition, 190:137-142, 2019.
Wu, Y., Li, X., Liu, Y., Zhou, P., and Sun, L. Jailbreaking gpt-4v via self-adversarial attacks with system prompts. arXiv preprint
2023.
Wang, H., He, Z., Lipton, Z. C., and Xing, E. P. Learning Robust Representations by Projecting Superficial Statis- tics Out, March 2019.
Wu, Z., Chen, X., Pan, Z.,, Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y.. Wu, C., Wang. B., Xie, Z., Wu, Y., Hu, K., Wang, J., Sun, Y., Li, Y., Piao, Y., Guan, K., Liu, A., Xie, X., You, Y., Dong, K., Yu, X., Zhang, H., Zhao, L., Wang, Y., and Ruan, C. Deepseek-v12: Mixture-of- experts vision-language models for advanced multimodal understanding, 2024a. URL https: / /aaaaaivvvorro abs 0tts/2422.10330
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024a.
Wang, S., Long, Z., Fan, Z., and Wei, Z. From ilms to milms: Exploring the landscape of multimodal jailbreak- ing. arXiv preprint arXiv:2406. 14859, 2024b.
Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., et al. Deepseek-v12: Mixture- of-experts vision-language models for advanced multi- modal understanding, arXiv preprint arXiv:4412.00302 2024b.
Wang, W., Lv, Q. Yu, W., Hong. W., Qi, J., Wang, Y., Ji, J.. Yang, Z., Zhao, L., Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0000 2023.
XAI.
Grok-2 beta release. grok- 2, August 2024. Accessed 2025-06-08.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., al. Emergent abilities of large language models. arXiv preprint (rXXv:2206.07688 2022a.
Xu, G., Jin, P., Hao, L., Song, Y., Sun, L., and Yuan, L. Llava-ol: Let vision language models reason step-by- step. arXiv preprint arXiv:241 2./1..10400 2024.
Yang, D. Y.-J,, Rosenblau, G., Keifer, C., and Pelphrey, K. A. An integrative neural model of social perception, action observation, and theory of mind. Neuroscience & Biobehavioral Reviews, 51: 31:263-275, 2015.
Wei. J., Wang, X., Schuurmans, D.,, Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of tf-thhughht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022b.
Yang, S., Luo, S., and Han, C. Multimodal common- sense knowledge distillation for visual question answer- ing, 2024. URL https: 02722.
Wellman, H. M. The Child's Theory of Mind. MIT Press, Cambridge, MA, 1992.
org/ abs /2111.
Wellman, H. M., Cross, D., and Watson, J. Meta-analysis of theory-of-ming development: The truth about false belief. Child development, 72(3):655-684, 2001.
Yang. S., Luo, S., Han, S. C., and Hovy, E. Magic-vqa: Multimodal and grounded inference with commonsense knowledge for visual question answering. arXiv preprint arXiv:2503. 1849,, 2025a.
Wimmer, H. and Perner, J. Beliefs about beliefs: Repre- sentation and constraining function of wrong beliefs in young children's 's understanding of deception. Cognition, 13(1):103-228 1983.
Yang, Y., He, X., Pan, H., Jiang, X., Deng, Y., Yang, X., Lu, H., Yin, D., Rao, F., Zhu, M., Zhang, B., and Chen, W. R1-onevision: Advancing generalized multimodal soning through cross-modal formalization. arXiv preprint arXiv:2503. 10611.003..,,.....1111111111 2025b.
Winer, G. A. Class-inclusion reasoning in children: A review of the empirical literature. Child Development, pp. 309-328, 1980.
Yantis, S. Perceived continuity of occluded visual objects. Psychological Science, 6(3):182-11) 1995.
Wright, B. C. and Smailes, J. Factors and processes in children's transitive deductions. Journal of Cognitive Psychology, 2788)967-977, 2015.
Ye, J., Xu, H., Liu, H., Hu, A., Yan, M., Qian, Q., Zhang, J., Huang, F., and Zhou, J. mplug-owl3: Towards long image-sequent sequence understanding in multi-moda. large lan- guage models. arXiv preprint arXiv:2408. 008004400 2024.
Wu, P. and Xie, S. v Guided visual search as core mechanism in multimodal llms. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13094, 2024.
Yi, D.J.,, Turk- Browne, N. B., Flombaum, J. I., Kim, M.-S., Scholl, B. J., and Chun, M. M. Spatiotemporal object continuity in human ventral visual cortex. Proceedings of
18

---

Core Knowledge Deficits in Multi-Modal Language Models
the National Academy of Sciences, 105(26):8840-8845,
2008.
Ying, K., Meng, F., Wang, J., Li, Z., Lin, H., Yang, Y., Zhang, H., Zhang, W., Lin, Y., Liu, S., jiayi lei, Lu, Q.. Gao, P., Chen, R., Xu, P., Zhang, R., Zhang, H., Wang, Y., Qiao, Y. Luo, P., Zhang, K., and Shao, W. MMT- bench: A comprehensive multimodal benchmark for eval- uating large vision-language models towards multitask AGI. In Forty-first International Conference Machine Learning, 2024. URL wwtpt:ttt////...... /tttpp:://peeee /pwp://eeeeeeeeeeee//mee / forum? 1 d=R4Ng82Yaiz
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, J. When and why vision- -anguage models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.019000 2022.
Zewald, J. and Jacobs, I. Object Permanence, pp. 1-17. Springer International Publishing, Cham, 2020. ISBN 978-3-319 doi: 10. 0..1007/978-3331--478899 2103-1.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12104-12113, 2022.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.. Sig- moid loss for language image pre-train pre-training. Proceed- ings of the IEEE/CVF international conference com- puter vision, pp. 11975-11986, 2023.
Zhang, H., Li, C., Wu, W., Mao, S., Vulić, I., Zhang, Z., Wang, L., Tan, T., Wei, F., et al. A call for new recipes to enhance spatial reasoning in milms. arXiv preprint arXiv:2504. 15004, 2025.
Zhang, R., Jiang D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W., Qiao, Y., al. Math- verse: Does your multi-modal I1m truly see the diagrams in visual math problems? In European Conference Computer Vision, pp. 169-186. Springer, 2024a.
Zhang, X., Li, Chu, W., Hai, J., Xu, R., Yang, Y., Guan, S., Xu, J., and Cui, P. On the -itt---distribution gener- alization multimodal large language models. arXiv preprint arXiv:2402.06530 2024b.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 105922 2023.
Zhu, J., Wang, W., Chen, Z., Liu, 7.,, Ye, S., Gu, L., Tian, H., Duan, Y. Su, W., Shao, J., et al. Internv13: Exploring advanced training and test-time recipes for open- source multimodal models. arXiv preprint Stt::2504.10477 2025.
19

---

Core Knowledge Deficits in Multi-Modal Language Models
A. Cognitive Framework
A.1. Core Knowledge in Human
Past research has shown that humans exhibit a series of rudimentary yet robust abilitics in domains such as object, number, space, action, and social cognition at a very young age. Such abilities, often known as "core" cognition, ground the set of diverse and complex abilities of human intelligence that develop later (Spelke et al., 1992; 1994; 1995; Spelke & Kinzler, 2007; Baillargeon & Carey, 2012; Mitchell, 2020; 2021). From infancy to early adulthood, human cognition develops along structured trajectory, with interdependent relations between early, simple abilities and late, complex abilities. For instance, the ability to imagine the perspectives of others typically develops between the ages of 3 and 6 (Piaget & Inhelder, 1969), while the capacity to fully comprehend others' intentions matures around age 12 (Wimmer& & Perner, 1983; Wellman et al., 2001; Liu et al., 2008). At the same time, the ability to understand other people's intentions largely depends on the ability to understand other people's 's perspectives (lacoboni, 2009; De Waal & Preston, 2017: Liu et al., 2017: Caviola et al., 2021; Ninomiya et al., 2020). An influential account of human learning has suggested that cognitive development is fundamentally driven by the increase of computaaionallrppeseentaiii complex mental operations to be performed on external data (Fodor, 1975; Pylyshyn, 1980; Halford et al., 1998; Fodor, 2008). However, while high-level abilities might emerge directly due to enhanced operational resources, these operations are critically guided by the "core" cognition system that has enabled the system to possess a rudimentary understanding of each cognitive domain. This early-stage grounding not only empowers humans to achieve a reliable performance at basic yer widely-applicable tasks starting from very young ages but is also precisely what supports high-level abilities to robustly direct task-relevant behaviors despite the nuanced signals that exist in the environment (Mitchell, 2021).
power of the system, which allows for more
The sensorimotor stage is the first stage of cognitive development proposed by Jean Piaget (Piaget, 1952; Piaget & Inhelder, 1974). Spanning from birth to approximately 2 years of age, this stage is characterized by infants' understanding of the world through their sensory experiences and motor actions. Several prominent features of human intelligence developed during this period. First, infants develop object permanence, that they realize objects and people continue to exist even when not in direct sight, or being heard or touched (Baillargeon et al., 1985). They start to understand that there is a sense of continuity for the ways that objects exist, and the inductive bias of continuity is essential, e.g., for recognizing objects when occluded or for continuously tracking objects (Spelke al., 1995; Le Poidevin, 2000). Infants also develop the sense of boundary during this stage, namely, the ability to recognize where one object ends and another begins (Kestenbaum et al., 1987; Jackendoff, 1991). Lastly, infants develop spatial and perceptual constancy by the end of the sensorimotor stage. Spatiality refers to the ability to perceive the position and distance of objects relative to oneself and each other, and recognize the spatial invariance between them when presented by various sensory experiences (Hermer & Spelke, 1996; Bell & Adams, 1999).
The preoperational and concrete operational stages are the second and third stages of Piaget's cognitive development. Typically spanning over 2 years of age, the preoperational stage is the transitional stage to the concrete operational stage, which children enter around 7 years of age. During this period, children begin to develop internalized mental actions supported by organized structures that can be manipulated and reversed in systematic ways, known as mental operations (Janet, 1905; Kirkpatrick, 1908; Piaget, 1950; Piaget & Inhelder, 2014; Miller, 2016). Through mental operations, children are then able to rigidly perform tasks that are previously unreachable, such as thinking from other people's perspectives, understanding hierarchical relations of objects, and reasoning about physical events in the world. These tasks require not only rudimentary understandings of physical concepts, which gradually became in place during the preoperational stage, but also relational and transformational reasoning that can only be done through mental operations (Piaget & Inhelder, 1974; Church & Moldin-Meadow, 1986; Houdé, 1997). Since the preoperational stage is mostly meaningful as the transitional period preceding the concrete operational stage, we do not have evaluation dimensions specifically targeting the stage. However, tasks targeting the concrete operational stage could assess the existence of knowledge associated with the preoperational stage, such as the law of conservation (Piaget, 1952; Halford, 2011; Houdé, 1997).
The formal operational stage is the fourth and final stage in Piaget's 's theory of cognitive development, typically emerging around 11 or 12 years of age and continuing into adulthood (Inhelder & Piaget, 1958). Starting in this stage, one is able to systematically and flexibly apply mental operations to not only concrete, physical domains but also abstract, formal domains (Kuhn& & Angelev, 1976; Shayer, 1979; Huitt & Hummel, 2003). In particular, this stage is characterized by the development of complex thinking and reasoning abilities, such as abstraction, pattern recognition, the use of logic, and hypothetical and counterfactual reasoning (Piaget, 1950; Inhelder & Piaget, 1958). These cognitive advances pave the way for more sophisticated abilities to interact with the physical world, marked by mechanical reasoning and tool use (O' (O'Brien& & Shapiro,
20

---

Core Knowledge Deficits in Multi-Modal Language Models
1968). Together, there is an advance in social cognition, characterized by a deeper understanding of intentions, actions, and the reasoning behind them (Meltzoff, 1999).
A.2. Definition of the 12 Core abilities in CoreCognition
Boundary Boundary refers to the cognitive understanding of where one object ends and another begins, an essential aspect of perceiving and understanding the physical world (Kestenbaum et al., 1987). Without understanding boundaries, it seems very hard to construct a concept of the object (Berkeley, 1709; Jackendoff, 1991).
Spatiality Spatiality refers to the cognitive understanding of the topological properties of our physical world (Bell & Adams, 1999). In a classic A-not-B task, an object is hidden at location A (such as under cup) and the child successfully finds several times. Then, the object is visibly moved to a different location B (under a different cup), in full view of the child. Younger infants often make the error of searching for the object at the original location A, indicating developmental stage where their understanding of object spatiality is not yet formed.
Perceptual Constancy Perceptual constancy is the cognitive ability to perceive objects being constant in their properties, such as size, shape, and color, despite changes in perspective, distance, or lighting (Rutherford & Brainard, 2002; Khang & Zaidi, 2004; Green, 2023). For instance, consider a red ball being thrown in a park. To an observer, the ball appears smaller as it moves farther away, yet the observer understands it remains the same size throughout its trajectory.
Object Permanence Object permanence refers to the cognitive understanding that objects continue to exist even when they are no longer perceptually accessible (Baillargeon, 1986; Spelke et al., 1992). This capacity emerges early in infancy and marks shift from sensorimotor interactions to rudimentary conceptual reasoning. A classic example is peek-a-boo: initially, infants may react with surprise or distress when caregiver's face is covered, as if it has ceased to exist. As permanence develops, they begin to infer the face's continued presence-reflettie the emergence of internal object representations that persist beyond immediate perception.
Continuity Continuity is the cognitive prior that objects persist as unified, cohesive entities across space and time (Spelke et al., 1995; Le Poidevin, 2000; Spelke et al., 1994; Yantis, 1995; Yi et al., 2008; Bertenthal et al., 2013). For example, when we see the front and rear of train simultaneously extending from opposite ends of a tunnel, we infer that the train continues through the occluded space as a single, continuous object. This inference reflects our sensitivity to spatiotemporal continuity-not not merely that the object exists, but that its parts remain connected along a coherent trajectory through space.
Conservation Conservation refers to the ability to understand that certain properties of physical entities are conserved after an object undergoes physical transformation (Piaget & Inhelder, 1974). This is instantiated in their ability to tell that quantities of physical entities across different domains, such as number, length, solid quantity and liquid volume, will remain the same despite adjustments of their arrangement, positioning, shapes, and containers (Halford, 2011; Craig et al., 1973; Piaget & Inhelder, 1974; Houdé al., 2011; Poirel et al., 2012; Marwaha al., 2017; Viarouge et al., 2019). For example, when a child watches water being poured from tall, narrow glass into a short, wide one, a grasp of liquid conservation would lead them to understand that the amount of water remains the same even though its appearance has changed.
Perspective-taking Perspective-taking is the ability to view things from another's perspective. This ability has seminal importance both to the understanding of the physical world as well as to the competence in social interactions (Wimmer& & Perner, 1983; Wellman, 1992; Liu et al., 2008; Barnes-Holmes et al., 2004). The Three Mountain Task first invented by Jean Piaget is widely used in developmental psychology laboratories as the gold standard for testing perspective taking abilities in children (Piaget & Inhelder, 1969)
Hierarchical Relation Hierarchical relation refers to the ability to organize objects or concepts into structured categories and subcategories, which are supported by the development of mental operations marked by class inclusion and transitivity (Shipley, 1979; Winer, 1980; Chapman & McBride, 1992). Class inclusion refers to the ability to recognize that some classes or groups of objects are subsets of a larger class. For example, a child in the concrete operational stage is able to understand that all roses are flowers, but not all flowers are roses (Borst et al., 2013; Politzer, 2016). This concept is essential for one's 's systematic and logical organization of conceptual knowledge. Transitivity refers to the ability to understand logical sequences and relationships between objects (Andrews & Halford, 1998; Wright & Smailes, 2015). For instance, if child knows that Stick A is longer than Stick B, and Stick B is longer than Stick C, they can deduce that Stick A is longer than Stick C.
Intuitive Physics Intuitive physics refers to the ability of humans to predict, interact with, and make assumptions about the
21

---

Core Knowledge Deficits in Multi-Modal Language Models
physical behavior of objects in their world (Michotte, 1963). As children grow, they transition from simplistic understandings. such expecting unsupported objects to fall, to more complex theories, such grasping the principles of inertia (Spelke et al., 1994; Kim & Spelke, 1999) and gravity (Vasta & Liben, 1996; Kim & Spelke, 1999; Li et al., 1999).
Intentionality Understanding Intention understanding involves recognizing and interpreting the actions of others (Searle, 1979; Rosenthal, 1991). This process is not just about observing a behavior but also about understanding the goal behind it (Baker et al., 2009; Gandhi et al., 2021). For example, seeing someone reaching for cup is not just about recognizing the physical action but understanding the intention behind it (e.g.. they want drink).
Mechanical Reasoning Mechanical reasoning refers to the ability to understand and apply mechanical concepts and logical principles to solve problems (Allen et al., 2020). This cognitive concept first involves the ability to interpret and predict the behaviors of complex physical systems and understand how different mechanisms of the systems work. Second, mechanical reasoning requires the ability to apply logic rules, such as induction, abduction, syllogism (O" Br'Brien& & Shapiro, 1968; Cesana-Arlotti et al., 2018), and reasoning forms, such as hypotheticals and counterfactual (Byme, 2016), to figure out how to manipulate these systems to achieve a desired outcome (Hegarty, 2004).
Tool Using Tool-using refers to the ability to utilize objects (as tools) in their environment aids in achieving a specific goal, such obtaining food or modifying the surroundings. A lot of cognitive components are involved in tool-using ability, such as affordances, referring to computing the action possibilities offered to the agent by the tool with reference to the agent's s sensorimotor capabilities (Gibson, 1979). For example, a door handle affords pulling or pushing, as how the door should be operated by human agent.
B. Input Types and Formats
CoreCognition encompasses diverse input types and formats. We first introduce two types of questions i.e, true/false questions and multiple-choice questions (MCQs). The distribution statistics is in Fig. 11.
Evaluating core knowledge presents unique challenges, as different core abilities necessitate distinct forms of visual media. For instance, tool use tasks require only images, while conservation tasks require videos. To address this diversity, our benchmarks incorporate a variety of visual formats within a single QA setting, including single images, vidcos, and sets of multiple images.
However, not all Multi-modal Large Language Models (MILMs) can process every input format due to implementation constraints. Considering the more than 200 models evaluated in this study, we classify the input formats into three main categories: single image, single video, and multiple images (i.e., multiple frames). The distribution of these formats across our benchmark is shown in Fig. 11.
multiple frames
3,400
160
single video
1,000
single image
1,200
217
800
1,000
Count
548
400
200
200
MC
Modality
Question Type
single image
single video
mu tiple frames (no intericave)
multiple frames iintrrieavel
cimage-plarehalter XX..> What the person trying to do? [B]
cridee-glachhtder is there die at the end the video? [Yes)
smoge...-slmagee... .aa.ge...-dnogg-..>
What should use to find my socks under the bed? [C]
there die image? (Yes)
home
A A.cimage..
R.chnage... ... Cimmage...
Clean
tank
Figure 11. Distribution of question type and modality type in CoreCognition dataset.
22

---

Core Knowledge Deficits in Multi-Modal Language Models
C. Justification for the Difficulty in CoreCognition
Our benchmark is deliberately designed to be more challenging than tasks typically used in classic developmental cognitive science experiments with young children, driven by two main considerations:
Breadth and systematic task coverage. Each task in our benchmark is derived from a well-established developmental cognitive science prototype (e.g., Piaget's "hhreeMMountain perspective-taking paradigm). Rather than relying on the limited examples available literature, we substantially expand each prototype into comprehensive suite of systematically varied questions. This approach produces rich and robust evaluation corpus, essential for reliable and nuanced measurement.
Appropriate evaluation for MLLMs. Appropriate evaluation for large language models. The subjects of our evaluation are not infants or young children, but MLLMs adapted from SOTA LLM, with knowledge and reasoning abilities comparable to, exceeding, those of well-educated adults, including PhD-level expertise in many domains (OpenAI, 2024). Consequently, our tasks are intentionally challenging to thoroughly probe the capabilities of these models, rather than simply replicating child-level diagnostic tasks.
D. Inference and Evaluation
D.1. Model Inference
We evaluate a total of 231 models. including commercial models and open-source models. Our tested models exhibit diversity in architecture and size, ranging from 1B 110B parameter size. Inference is performed on clusters equipped with 8xNVIDIA A100 80 GB GPUs. In most cases, models between 1B and 13B in size can be inferred Models ranging from 13B to 32B require two GPUs, those from 32B to 70B require four GPUs, and models larger than 70B require all eight GPUs to inference. Based on the input types they support, the 23 models are categorized into three groups: single-image, multi-image, and video models. Specifically, 85 models support only angle-image input, 105 models support multi-image input, and 41 models support video input. We built a scalable evaluation infrastructure supporting parallel execution and compartmentalizee environments, enabling reliable inference across over 200 MILMMs. We strictly follow the setup and source code from the official codebases provided by model developers ensure fidelity. We further conduct sanity checks using widely adopted benchmarks to verify that the models can reproduce established results. To support smooth inference of over 200 models, we configure 43 compartmentalized environments, each compatible with one or multiple models. Efficient inference is performed by parallelizing models across multiple GPUs and devices. A dynamic scheduler is employed to minimize computational waste.
single GPU,
D.2. Evaluation
D.2.1. MATCHING ANSWERS TO CHOICES
We explore four matching strategies and propose hybrid approach that combines the strengths of both template- and L.L.-matching. After removing pre-defined special tokens,
exact matching matches the MIL.M output to a choice only if they are identical, disregarding case differences.
"in" matching matches the MLLM output to choice the output, when split by spaces or punctuation, contains exactly one choice.
template matching: matches the entire MLLM output against predefined templates, such as "Answers: [choice]" or "[choice]. [sentences of explanation without references to another choice]".
LLM matching: We employ a LLLe-a--a-uddee framework where the LLM is provided with the original question, the options, and the MLLM output and prompts to determine which choice the output mostly supports.
Exact and "in" matching approaches exhibited relatively high fail rates, with abundant false positives and false negatives. These methods often struggle when model outputs are complex, such as reasoning models, or with explanation or chain- of-thought (CoT) responses. Template matching was able to accommodate a broader range of scenarios but necessitated iterative adaptation of templates to cover exceptional cases. Even after considerable refinement and despite achieving high
23

---

Core Knowledge Deficits in Multi-Modal Language Models
accuracy matched data points, template matching still resulted in non-negligible overall fail rate. In contrast, LLM matching demonstrates strong performance in identifying the intended choice within text-rich outputs, even in cases where the explanation underwent concession processes. However, LLM matching was occasionally susceptible to hallucination, particularly when short or simple answers were embedded within extensive contextual information.
To address these limitations and capitalize on the complementary strengths of different matching strategies, we propose a Hybrid Matching mechanism. Specifically, we prioritize a rule-based template matching approach to extract answers from MLLM responses. If template matching method failed, we turn to a model based ensemble strategy using four advanced LLMs: Owen2.5-72B-Instmuce Mixtral-8x Innt-Bsttuctt-O... DeepSeek- RI I--iitill-LLaman
and 1-maa.1-77B. The LLM- based result is accepted only when least three of the four models produce consistent extractions; otherwise, the matching is deemed unsuccessful. By integrating the precision of regular-format matching with the flexibility of semantic-based matching, the hybrid method achieves more robust and reliable performance overall. We provide a comparison between Hybrid Matching and the other four alternatives in Tab.
Table 4. Fail rates across different matching strategies.
Method
Exact Matching
"In" Matching
Template Matching
LLaMA3.1-70B Matching
Hybrid Matching
Fail Rate
53.0448%
31.3501% %
8.2056%
10.9640%
6.4845%
D.2.2. FILTERING
80
cutout threshold
Relative Frequency (%)
70
60
50-
40-
30
20
10-
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Fail Rate
Figure 12, Distribution fail rate in responses matching. We cut off at 20% filtering any models with fail rate over 0.2.
After applying Hybrid Matching, a number of models still exhibited high failure rates, as shown in Fig. 12. The distribution of failure rates across models revealed long-tail tail pattern, with small subset of models performing substantially worse than the majority. To distinguish between detrimental or systematic failures (c.g., outputs consisting entirely of illegal characters) and intrinsic model limitations (e.g., adequate input processing but inadequate responses), we manually examined all models with matching fail rate of 2 17%. This thorough review allowed us to establish a clear threshold between these categories. Based on our analysis, we set a final exclusion criterion of 2 20% fail rate, resulting in the removal of 12 models exhibiting detrimental failure modes. The remaining 219 models, which demonstrated reasonable performance, were retained for further analysis.
D.2.3. SCORING
After applying Hybrid Matching and filtering out models with high failure rates, we evaluated each model by comparing its matched response to the ground-truth options. Responses marked as matching failures were classified as incorrect.
To reduce the risk models favoring certain answer positions- phenomenon known as option-posse position bias-we we use the circular evaluation strategy (Liu et al., 2023a). In this approach, each multiple-choice question with & possible answers is presented times, with the order of the answer options rotated each time.
24

---

Core Knowledge Deficits in Multi-Modal Language Models
Instead of requiring the model pick the correct answer every time, regardless of the order, we calculate the proportion of times it selects the correct answer across all rotations. This method provides fairer estimate of the model's true ability and avoids unfairly lowering scores-espee especilssy when there are many answer options, where demanding perfect consisteney would make it almost impossible to achieve a correct score by chance. By preventing an excessively low chance level, this method also ensures that subsequent normalization reflects realistic trends.
E. Normalization in Concept-wise Comparison
Since different core knowledge abilities are grounded in diverse cognitive science prototypes, they entail distinct distributions of question formats (e.g., true/false or multiple choice with 2-4 options), resulting in varying levels of chance accuracy and inherent difficulty. Therefore, normalization is essential to ensure fair comparisons across these abilities and to more robustly demonstrate "core knowledge deficits."" " To achieve this, we normalize the accuracy for each ability by dividing by its corresponding chance-level accuracy: acc asc'. where acc' denotes the model's accuracy the i-th core knowledge ability, and Et represents the chance-level probability for ability i. As shown in Fig. 13, report both the raw and normalized accuracies (see Sec.4.1). 1). Both measures exhibit a similar upward trend, further supporting our first finding of core knowledge deficits.
Huraan tessine
Chance level
Normalned humon toradine
Normalzed chanoe lovel
Sensorimotor
Concrete Operation
Formal
Sensorimotor
Concrete Operation
Formal Operation
Operation
Nommazed Acoutry
Accuracy
Bounday
Percaptbnl Constonyy
Intulive Plyyase
Pempective
Consenation
Harernhy
Intectionally
Bevitaniial Ressooing
Tiat Uainng
Bonndey
Contintly
Perveraall Constancy
Spatiiity
Intutiie Plystia
Harerrby
Rasenning
Figure 13. Accuracy by concept without normalization
F. Performance with Different Prompts
Table 5. Illustration of 10 different prompts, in five categories.
Category
Prompt
no prompt
(Empty String]
Let's think step by step.
think deep
Take deep breath and answer this question earefully.
Pleuse answer the question und provide an explanation.
explanation
Pleuse answer the question und explain to me in simple terms.
Pleuse answer the question and explain it to me like am 11 years old.
Please answer the question carefully. I'm going to lip you 200 dollars for better solution.
reward & penalty
Pleuse answer the question carefully. You will be penalized if your answer is incorrect.
Please answer the queston and ensure That your answer is unbiased and doesn't i refy on stercotypes.
bias mitigation
role playing
You an expert on cognitive science and are familiar with [Concept name].
Pieuse read the concept explanation and then answer the related question. Concept: Tconcepi
cognitive instruction
description].
We investigate the influence of different prompting techniques on MLLM performance using our benchmark. As shown in Tab. 5, we evaluate five prompt categories comprising ten distinct variants. We find that most prompts fail to improve performance. Notably, the cognitive instruction prompt (p10)- concise description of the conceptual targets assessed by the task ootp-fuussso all others, increasing accuracy by over 6%. This improvement likely stems from the fact that
25

---

Core Knowledge Deficits in Multi-Modal Language Models
explicitly stating the relevant core knowledge enables the model more efficiently extract information that is otherwise distributed across its internal representations (Chalmers, 1992).
0.7
0.6
Accuracy
0.4
0.3
po
p1
p2 (
p3
p4
p5
p6
p7
p8
p9
p10
empty
think deep
reward & penalty
role
cognitive instruction
explanation
string
playing
Figure 14. Performance of different prompts. Accuracy is averaged across all models.
This interpretation aligns with early insights from the connectionist literature, which suggest that distributed representations pose challenges for structured knowledge retrieval (Hinton et al., 1986; Chalmers, 1990). As networks scale, accessing specific conceptual structures becomes increasingly pirticulltpariiculaall for foundational concepts like core knowledge. Unlike high-level knowledge (e.g., historical events), which may be encoded in localized or clustered patterns, core knowledge tends to be diffusely represented across parameters due to its recurrence in diverse training contexts. This diffuse encoding makes such knowledge harder isolate and systematically deploy in reasoning tasks. We hypothesize that cognitive instruction may servesas retrieval cue, guiding the model's 's internal attention toward latent knowledge. However, we do not view this as a permanent or scalable solution. In real-world scenarios, models are unlikely to receive such explicit conceptual scaffolding, limiting the practical utility of this approach. Nevertheless, the finding points to a promising research direction for improving model reasoning through targeted scaffolding or memory-augmentt mechanisms.
G. Details on Statistical Testing
To statistically validate core knowledge deficits, defined performance differences across the three developmental stages, we conducted paired t-tests. The test statistic is given by
1 sal Vn
where d denotes the mean difference between paired observations, Sa is the standard deviation of these differences, and is the number of pairs.
Table 6 reports the results of the statistical test.
The extremely small p-values (p < 0.001) indicate that the observed performance differences between the Formal Operational stage and the other two stages are statistically significant.
26

---

Core Knowledge Deficits in Multi-Modal Language Models
Table 6. Paired I-test result.
Contrast
1-statistic
p-value
Formal Operational vs. Concrete Operational
22.68
4.79 X 10 10-4s
Formal Operational vs. Sensorimotor
28.15
3.81 10-53
H. Core Knowledge Deficits Under Different Conditions
We further validate "core knowledge deficit" depicted in Fig. 5 under different conditions. Specifically, we validate whether our finding generalizes across different model subgroups (by competence and input constraints) and prompt variations.
Malislo Froms
Singls Image pwdeeeeideeeeeee
Swowprimoter oparaiion
Concrete operstion
uparalion Fornal operation
operation
Formal operation
Mcmalied
Poytin
Raseeting
Beiing
Beeestar
Madiancce Beetning Toot
Inteciinaitty
Peroppian Imaatee
Mesiami
Fool
Perppptat
Figure 15. Condition Group I: Model Input Constraints
As shown in Fig. 15, all three input formats and corresponding question subsets exhibit pattern consistent with our conclusions in Sec.4.1.
Znety Stray" Noepp
Thask Dap? Porept
Expliteate Recepe
Sensormoter
Sensorimotrr
Sensormotor
Concrese oporation
Contrete operation operation
Aocuracy
Fonmal
accurncy
Formal operation
Mermatted
Mornolkzedd
Mornalaed
Mouedaay
Contimulty
Permanatiio
percaptiial Conoraery
Sportallty
Intultive Plyetca
Perspectire
Conter conerrattion vottion
Hirrercyy
Iechaniccll sesperting
intectioniity
thing
Ratestary
Coveinutty
Perceptud Constatoy
Spatiallt
Intattiv Plyetor
Perspecctiv
Consertattoo centrr
Mererrtiy
Merchnnical Reasninn
Intertionnalty
Tost Urtng
Rassddary
Constiutty
Per Permennsaee
Percaptaal Constatoy
Seutaalty
brtativa Payatta
Perppettive
Conserattin vetiorttii
Hararrcy
Mechosical Reaanning
Indertionatty
Toal Deing
Reandd Raxalty Poreet
"Sapesiin Inchnaiine Proppe
Sonsorimotor
Seneerimotor
Bensor maker
Concrata operafion
Accursyy
Foraal speration
Mernakted accurny
Formal operatin
Mernatted Accurnry
Formal operation
Mernakeed
Perraptuul tanyy
Spactesity
Intultive Phystrr
Purspective
Conservation
Meciannical Aeasonting
sarcity
Intestionalty
That sssing
Roumdbrr
Contiruity
Parmaraance
Percaptonl Conutancy
Gexiattty
Ineittvv Phystie
Persoective
Conservation
Heaaardy
Mechaniccll eosting
Istenstonality
Tool tsiig
Rosovlary
Continiity
Percaptull Constanty
Seutaaity
Intaltiv Phyatoe
Phomecttive
conserattion
Metarrcy
Mechanital Meseenting
Infentionaltty
Roo Uninn
Figure 16. Condition Group II: Prompt type
Across Fig. 16, all six types of prompts yield patterns that align with our conclusions in Sec. 4.1.
As shown in Fig. 17, the overall performance pattern remains consistent across different model selections. This replicates
27

---

Core Knowledge Deficits in Multi-Modal Language Models
20 Malal Prass Aacia
Pap Se Mulipio Koew Moae
Gensodmrterr speradon Fo operation
Swoontmei operstion Fornal openatio
Genacetmater
Gonordt oporsion
Formel operation
Dalig
Peataiing
Figure 17. Condition Group III: Model Competence. Top-n refers models with the highest average accuracy out of 120 total models that have multiple-frame capacity and thus evaluated on all data CoreCognition
the core knowledge deficit, with lower-level tasks consistently showing lower accuracy. Expanding the selection from the top 25 all 120 multi-image models increases variability and slightly reduces median accuracy, particularly for higher-level tasks. This growing heterogeneity in model quality underscores that gains on abstract tasks are not uniformly reliable, and the the gap abilities persists.
I. Dependencies Between Core Abilities Under Different Conditions
We further examine the stability of finding 2 depicted in Fig. 6 by validating it under different conditions.
Muutiyll-Parne Models
Single VivegeVVdeo Modela
Single Amage Modelo
Prisels Cones
Metial Nuaeeding
Metanial
Figure 18. Condition Group Model Input Constraints
As shown in Fig. 18, the inter-concept dependencies in single-image and video model subgroups closely mirror those of the full-capacity multi-frame models discussed Sec. 4.2. Two exceptions are Continuity and Permanence, which consistently exhibit low correlations with other concepts in the single-image setting. In particular, Permanence cannot be effectively assessed with single-image models, as this subset is unrepresentative and includes fewer than 10 questions.
28

---

Core Knowledge Deficits in Multi-Modal Language Models
"Ereayy String Proupt
"Thvie Deop" Roro
Explenation Prone
Reaand & Pacsat" Protee
"oole Plyytog T Strnppe
Cogaltive Insrrctiion Prreppe
Figure 19. Condition Group II: Prompt type
As shown in Fig. 19, all six prompts produce inter-concept dependency patterns consistent with our conclusions in Sec.4.2.
MuutipherTrame
-04
Shary
Top Mattpple-Pram Modhils
n00 Muttiple- Pramee Mostedy
AD 120 Multjlle-Faaeee Framre Modell
-44
Figure 20. Condition Group III Model Competence. Top n refers to n models with the highest average accuracy out of 120 total models that have multiple-framt frame capacity and thus evaluated on all data CoreCognition
Across different subgroups in Figure 20, the overall pattern of inter-dependencces remains consistent with our original observation: while correlations between high-level abilities are significant, cross-stage correlations are generally modest, with continuity, Permanence, and Spatiality consistently show low correlations with higher-level abilities, while Perspective and Conservation remain largely uncorrelated with other tasks tasks-reinfon reinforciin their isolation. Although a large number of models (c.g.. Top 100, All 120) lead to slightly higher correlations, this likely reflects performance floor effects rather than genuine cognitive integration, as power-performing models tend to fail relatively uniformly across tasks.
29

---

Core Knowledge Deficits in Multi-Modal Language Models
J. Scaling Effect on Core Knowledge Under Different Conditions
Single image Modelt
Single Image/Video Models
Multiple-Frame Models
0:15
999
9596
0.06
0.05
spration
Bca ing Blope
10
Scaling Stope
sporiter
0.
0.04
Scaling
00
0.03
-0.02
Boundary
Continutty
Pormanurss
Spatialty
Intulive Physica
Constoryy
Perspective
Conservation
Machanical Restowing
Hararcity
Intentoroaity
Tool Using
Boundary
Continuity
Permarencc
Perceptuul Conslancy
Spariality
Intuitiv Prosstze
Perepoctive
Conservation
Mechanical Rosswving Tool Using
Hefrrrsity
Intertionatity
Boundary
Continstly
Pormanercs
Parcspttul Constonoy
Spatiaity
Intuliive Physice
Perspeotive
Conservation
Mechanical Reusoning Tool Liiing
Hararrhy
Intesttorsiity
Percmnted
Figure 21, Condition Group I: Model Input Constraints
We also tested the scaling effect under different subgroups of models with different input constraints. As per Fig. 21, all three subgroups of models lead similar scaling effect aligned with our observations in Sec. 4.4. Further condition partitions (model competence and prompt type) are not analyzed for scaling effect because (1) it is the nature of the scaling analysis to study across model groups and (2) reducing the data size to single prompt violates the data size requirement of stable regression fit.
K. Explanation of Concept Hacking with an Example
For example, as shown in the third case of Fig. 9, a standard probe of perceptual constancy assesses whether a model understands that bridge of uniform width extending into the ocean does not actually become narrower in the distance. In the manipulated condition, all task-irrelevant details-such as the viewing angle and environmental textures-ard kept identical to the standard task, but the bridge itself is altered to genuinely taper it extends outward. Models possessing the understanding of perceptual constancy would have no difficulty answering both the manipulation task and standard control correctly. On the contrary, a model relying on spurious correlations between the task and previous examples of similar scenarios in the data would succeed in the original task but fail the manipulated one. Finally, a model with a strong inclination toward the belief that objects extending into the horizon are actually getting thinner physically would fail the control task while correctly answering the manipulated version due to misaligned knowledge about the world.
30

---

Core Knowledge Deficits in Multi-Modal Language Models
L. Qualitative examination of Concept Hacking
Central
Manipulation
Control
Manipu briien
Continutty
Spatially
fether ontyons lovelo waffect
GPT Reasasing: the pukk mein the by like bork. ore siggrnonttt the loo and itsiin on buoh sklos the Uera suggists they ore part th some continuoum line
vitualy cont deptts being pertsityabacund
Realeniing mags appear
livel, the rediandeen white checkered sisal
central
Phyaac
Boundary
Danatustantin mon www.tatttllaaaaaoooomoooo
wilow baddbbullaad frrc??
A we & Ni, aue aenttere Noir They will lina name kime
on.0
grow Ry, simulisnmpuutt drorpppd from The diffrrntt
futte d fivere in mnnure and panven withe the cried sne grevners The ghvlll oir pin dlgge the peszen dhictheee sut you can spet. dightyy darier patch with roundes shapo, noundling head and Oetirnnte liss diadioa
and
not aftect the time caken for them feil.
with teeedeeed cumrrr digroepennion
Dowe nonnea with mesilemes comee fommssar to tar?
The mrage nnpereats duct poosing plauelte)
CiP
Sh
teidge playiica the with uniform throughout langth.
ssaing
Grtifeeeeeeg the image liattaii
fnoughh u with enes dit leu
with phylls realty smasttal or plorld ruprossriatons souhis mucentiun spriloli depetionn lhe cass maysmilolue cupliction, Iranotormaion, or chittin, rakling ennire strract imgisaatee coniness
otherede While
sbudei the ata
Paropectivs
Fromo Fromo on ratondd 50 mooos doen
89098
29999
The top und bocsen aestion the imaa number of cirdie on thi tAo mes they bemoon owrtio??
Ressoning:
The rounded head of the duck can resemble the ridit's bice.
The owa cornallmmooltt by are shape of drawing lourorr antiguutty when the tmage rotated, simmerr the
The dudi'' beat can
ears of rub when from diitdd persecttive
Reescreng From inpection
your quesiion and teorows botiormitiinn theinege
Whilrrhe original oneentttoon concieer wory dequcts suokk the sitee inwiinanuun nough aeust srrhggatty for somaone interpest has redett. cannic ceampie of how proppcctis clace and percuption p00
The fin row of botton sectioo containe di-dec... The secons the bortom saction containe crdds.
Figure 22, Example Questions from the Concept Hacking. Each example is presented with GPT-4o's answer (w/explanation) to the manipulated image.
We further conduct qualitative examination of the reasons underlying the models' low performance on manipulation samples in F2g.22. Through prompting techniques such as Chain-of-Thhne ofoouhh (Wei et al., 2022b) and "provide an explanation" (Li et al., 2022b), we are able to examine the reasoning chain behind the model's response. Using GPT-4o as a case study, we find that its low performance on manipulation samples primarily stems from an overreliance on shortcuts, possibly learned during training. When presented with manipulation samples, GPT-4 often reproduces reasoning similar to that used for corresponding control samples. For instance, in 22, GPT-4o correctly explains the illusion of decreasing bridge width (e.g.. "the converging lines of the bridge create illusion of decreasing width") even when the bridge's width was actually manipulated to decrease. This indicates that the model's 's reasoning was not grounded in the visual information explicitly presented in the image, but resorted to the shortcut understanding of the "near-large, far-small" phenomenon learned during the pretraining.
M. Limitations
We acknowledge several limitations in our study. To begin with, VQA used in CoreCognition introduces auxiliary demands such as linguistic capabilities, counting, and object recognition. While we mitigated some through meticulous filtering of questions and rigorous validation across 11 distinct prompts, completely eliminating these auxiliary factors remains challenging. The VQA format also inherently restricts CoreCognition's 's applicability to models capable of linguistic processing, narrowing the range of testable models. A complementary evaluation format is planned in future work to include linguiigci or purely visual models. In addition, despite careful dataset design, MLLMS still exploit shortcuts and spurious features in ConeCognition potentially inflating their performance, as evidenced in Sec. 5, Although Concept Hacking partially addresses this, its meticulous nature limits scalability, restricting its applicability to large-scale benchmarking.
31


Reviews:
[{"review_id": "zfnXTsyboy", "reviewer": "Reviewer_ZrNC", "summary": "This paper presents an evaluation framework for assessing the image understanding capabilities of multimodal language models (MLLMs) from a lens of cognitive taxonomy of concepts of learning. Inspired by the cognitive science literature on visual concept learning, the authors present a “CoreCognition” benchmark, which has been curated by defining a set of abstract learning ‘milestones’ or ‘skill levels’ of human visual cognition, and creating questions in a stratified manner per ‘skill level’ to controllably assess the image understanding capabilities of SOTA LLMs, bringing in a cognitive perspective into the benchmark curation. They ensure quality by conducting manual reviews of every question, ensuring that the options are cycled to mitigate positional bias in LLM answering, and testing various models under multiple prompted settings to regularize any instruction following effects.\nBy probing LLMs, the authors conclude that LLMs perform better on ‘higher level skills’ as opposed to more fundamental, ‘early developed’ core capabilities under their proposed taxonomy. Further, they test scaling of models to understand the trends of curriculum of tasks v/s model sizes, showing inconsistent behaviour e.g. more ‘core’ capabilities not scaling as expected. Finally, the authors throw light on ‘Concept Hacking’, highlighting the tendency of LLMs to rely on spurious correlations to solve questions of these kinds.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. However, due to limited space, we could only answer the most significant questions here. We look forward to discussing the rest (i.e., illusion vs. shortcut and 4 quadrants of Fig 8, suggested citations, etc) in the discussion phase!\n\n```>>> Q1``` Why this taxonomy was chosen, and if there are any specific inspirations from the ML literature e.g. OOD robustness, robotics?\n\n```>>> A1```  We chose core knowledge as the basis for our taxonomy because it offers a theoretically grounded and developmentally validated account of the foundational systems underlying human cognition (Spelke & Kinzler, 2007). These dimensions are widely seen as essential to general intelligence, making them a meaningful lens for evaluating AI capabilities (Carey & Gelman, 1993; Carey, 2009; Spelke, 2022). While not directly inspired by robotics or OOD robustness, core knowledge is highly relevant to both, as generalization, transfer, and embodied reasoning rely on intuitive world understanding—precisely what core knowledge aims to capture. Thus, we provide a complementary perspective that can inform evaluations of generalization, transfer, and robustness in AI.\n\n```>>> Q2``` The claim on \"No observable scaling on low-level abilities...\" Did the authors expect otherwise and why?...Do we a) expect and b) desire that our models follow the same curriculum as humans do? Will models follow the same path to AGI as we humans learn?...thinking in meta-terms about the overall capabilities, and how we scale these capabilities.\n\n```>>> A2 ``` First, we do not expect otherwise as we anticipate challenges for the emergence of core-abilities simply through large-scale pretraining of statistical occurrence. Another potential reason is that, compared to high-level details required for complex tasks, core knowledge used in simpler tasks is spread across diverse contexts and the vast parameter space of the network (Shani et al., 2023), thus harder to isolate and apply systematically, leading to inconsistent or surface-level reasoning. \n\nIt's a great question whether AI should follow the path of humans. We elaborate on a discussion in A2 of Reviewer Kcuh (due to limited space), that our paper neither argues for nor against the necessity of mirroring humans in pursuit of AGI. Rather, core knowledge is introduced as an evaluation for MLLMs, and the core knowledge deficit hypothesis is proposed as an explanation for the observed brittleness of MLLMs. More broadly, we hypothesize that core knowledge may represent a general principle of intelligence--human or otherwise. The critical research question is not whether models should mimic human learning, but how core abilities can emerge through scaling with cognitive or human-inspired adaptation, or other measurements.\n\n```>>> Q3``` 'Cognitive instruction' seems to perform superiorly. This implies that certain cognitive priming to the models can help boost accuracy on the benchmarks. Any comments?\n\n```>>> A3``` For now, we don't have a definitive explanation, but we find this effect aligned with early insights from the connectionist literature, which suggests that distributed representations pose a challenge for structured knowledge retrieval. As networks scale, retrieving specific conceptual structures becomes increasingly difficult (Hinton et al., 1986; Chalmers, 1990), especially for core knowledge. Unlike high-level knowledge, e.g., historical events, which are likely encoded in clustered patterns, core knowledge is highly distributed across the model’s parameters, as they are in multiple instances in training data, making them hard to isolate and deploy systematically for reasoning tasks (Garrigan, 2008; Green, 2024). We hypothesize that cognitive instruction may act as a retrieval cue that helps direct the model's internal attention toward latent but relevant knowledge. However, we do not believe this constitutes a permanent or scalable solution. In real-world environments, models are unlikely to receive such explicit guidance, limiting the practical utility of this strategy. Nevertheless, this result may point toward a promising research direction for improving reasoning via targeted scaffolding or memory-based mechanisms.\n\n```>>> Q4``` Is the space of LLMs for spurious correlations any different from the space of spurious correlations for ResNets/ViTs, etc.? Can this benchmark be used to evaluate existing vision models? If not, what makes this eval benchmark specific to LLMs?  \n```>>> A4``` The space is different for LLM and vision models as the former is in an abstracted and discrete token space while the latter is high-dim pixel space. Our benchmark cannot be directly applied to vision models, since the questions are in VQA format (i.e., question answering). However, adaptation can be made to convert it into retrieval / binary classification for vision models to test. However, this falls out of the scope of this paper, and we leave it to future efforts.", "claims_and_evidence": "All the claims in the submission are supported by evidence where necessary. Specifically,\n1. The claim on 'MLLMs consistently perform worse on low-level abilities compared to high-level abilities' is empirically supported by the evaluation of LLMs under these categories across different LLMs and prompting techniques.  The proposed benchmark has been curated by attempting to define a taxonomy which is in-line with the existing cognitive science literature, which is appreciated (and much needed) as we move towards an era of natively multimodal LLMs.  The taxonomy and abstraction of the proposed curriculum is a very first attempt and I'd be interested in understanding on why this taxonomy was chosen, and if there are any specific inspirations even from the ML literature e.g. OOD robustness, robotics.\n\n2. The claim on \"No observable scaling on low-level abilities with respect to increasing model parameters\" is an interesting one: while I do understand empirical backing, did the authors expect otherwise and why? Do we _want_ models to follow the similar curriculum as humans do? \n\n3. On control hacking, the claim that models rely on spurious correlations - this is not directly visible from Figure 8, and it would be greatly appreciated if the authors can intuitively explain the difference in illusions and shortcuts.  Nevertheless, it is good to note that Humans perform in a superior fashion.", "methods_and_evaluation_criteria": "Yes. The method itself is a benchmark.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. As mentioned above, they ensure quality by conducting manual reviews of every question, ensuring that the options are cycled to mitigate positional bias in LLM answering, and testing various models under multiple prompted settings to regularize any instruction following effects.", "relation_to_broader_scientific_literature": "This paper proposes a standard benchmarking methodology. \nThe piece on shortcut learning is related to previous work on OOD robustness, an example linked below.", "essential_references_not_discussed": "There is rich literature on OOD robustness (taking the classic example of Waterbirds v/s Landbirds), e.g. [1] which could be referred to. Spurious correlations in the field isnt new, and I'd be curious to know if the space of LLMs for spurious correlations is any different from the space of spurious correlations for ResNets/ViTs etc.\n\n[1] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization\nhttps://arxiv.org/abs/1911.08731", "other_strengths_and_weaknesses": "**Strengths**:\n1. Overall, I like the motivation to build cognitively inspired benchmarks for LLMs because it is much needed to bring in interdisciplinary perspectives to evaluation. Further, the evaluation methodology in the paper is rigorous. \n2. I really appreciate the taxonomy that has been proposed: not only does this present a good benchmark, it also inspires other researchers in the field to conduct stratified analysis / training on such kind of data to improve their models specifically for specific facets of tasks.\n\n**Weakness**\n1. Considering the overall premise of the paper: Given that we are building benchmarks inspired by human cognition - do we a) _expect_  and b) _desire_ that our models follow _the same curriculum_ as humans do? Will models follow the same path to AGI as we humans learn? \nI'd love to hear from the authors on what exactly they want the LLM community to take away from this paper when thinking in meta-terms about the overall capabilities, and how we scale these capabilities.", "other_comments_or_suggestions": "1. There is an extra dot in the abstract on line number 29.\n2. It would be nice to have an explanation for every quadrant in Figure 8.", "questions_for_authors": "Echoing the points I have mentioned above as questions:\n\n1. Considering the overall premise of the paper: Given that we are building benchmarks inspired by human cognition - do we a) expect and b) desire that our models follow the same curriculum as humans do? Will models follow the same path to AGI as we humans learn? I'd love to hear from the authors on what exactly they want the LLM community to take away from this paper when thinking in meta-terms about the overall capabilities, and how we scale these capabilities.\n\n2. It is very interesting to note that the 'cognitive instruction' seems to perform superiorly. This implies that certain cognitive priming to the \nmodels can help boost accuracy on the benchmarks. Do the authors have any comments?\n\n3. Is the space of LLMs for spurious correlations any different from the space of spurious correlations for ResNets/ViTs etc.? Can this benchmark be used to evaluate existing vision models? If not, what makes this eval benchmark specific to LLMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVTQpsxR9Y", "reviewer": "Reviewer_vNKX", "summary": "The paper investigates core cognitive abilities in multimodal large language models. The authors find that models underperform in abilities that develop early in humans, while they perform comparable to humans on higher level abilities. They show that multimodal language models often rely on shortcut learning.\n\n## update after rebuttal: I still think this is a solid paper and recommend acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "```>>> Q1``` \"high-level abilities do not correlate with the corresponding low-level abilities\" is too strong  \n```>>> A1``` Thanks! In Sec 4.3, the correlations between lower- and higher-level abilities are generally below 0.4. This is considerably lower than what's commonly observed in humans, though the phrasing may be strong. We will revise it to better describe the observation.\n\n```>>> Q2``` Scaling analysis: add numbers to make the plot more digestible. \"low-level cognitive abilities is minimal...even detrimental\" seems not perfectly supported.  \n```>>> A2``` Thanks for the advice. Our current efforts include 1) comparison of scalability (slope) in the bottom left of Fig. 6, and 2) $R^2$ values represented by the size of the blobs to the left of the ability names in the upper-left sub-fig of Fig. 6.  We will also add specific numbers in the text to enhance clarity as suggested. We acknowledge that most abilities show some scalability; however, the scaling effect on lower-level abilities is significantly smaller (half the value) than that of higher-level abilities, which supports our conclusion. We will revise the text to better reflect the observation. Please see A4 below for a discussion on the perspective-taking exception.\n\n```>>> Q3``` Suggested citation. A similar technique...for visual illusions in VLMs in [3].  \n```>>> A3``` Thanks for bringing this up. While [3] concurrently introduces a related idea, there are significant differences between the two. We offer a comprehensive controlled methodology grounded in the evaluation of core knowledge. Specifically, we formally propose a systematic procedure to manipulate input samples (${X}$) by flipping concept-level labels ($Y$) through targeted changes to causal features (${S}$) while holding non-causal features (${B}$) constant (Formally, a true predictive dist factorize as $p(Y|X) = \\int p(Y| S, B) p(S, B| X)$). Whereas [3] presents only 10 samples of visual illusion (valuable but largely anecdotal), not situated within a broader diagnostic or theoretical framework with clear motivation and research questions. In addition, we present Figure 8,  showing the development trend of current MLLMs (with respect to scaling) is not in the ideal direction but biased towards illusion or shortcut. This adds an interpretable, diagnostic dimension to our evaluation that is absent in [3].\n\n```>>> Q4``` \"Worse on lower-level abilities...there exist core knowledge deficits\". Aren't all 12 abilities core abilities?  \n```>>> A4``` All 12 abilities in our benchmark are grounded in core knowledge dimensions. However, lower-level abilities are operational approximations of basic cognitive systems and are thus more directly aligned with the notion of \"core knowledge\", while higher-level abilities are more abstract or compositional cognitive tasks. The observed upward trend in performance does not imply that only lower-level abilities reflect core knowledge. Rather, it suggests that while models may perform better on higher-level tasks—potentially by pattern matching or spurious correlation—they often struggle with the more fundamental reasoning required for lower-level tasks. This gap is what we refer to as a core knowledge deficit: the failure to demonstrate a robust understanding of the foundational abilities that higher-level tasks presuppose.\n\n```>>> Q5``` Why do perspective tasks show such poor external validity and do not scale?  \n```>>> A5``` Thanks for the question, which relates to a key finding here. The perspective-taking task in our benchmark is based on the \"Three Mountains\" experiment, a type of level-2 perspective-taking (Moll, 2010), which requires mental simulation—the ability to build an internal model of the world and reason from it (Johnson-Laird, 1982). Whether MLLMs possess such internal world models is debated (Mitchell, 2023; Goddu et al., 2024) and the lack of scalability in perspective suggests current models may not. Unlike other tasks, perspective-taking additionally demands constructing a spatial model, the absence of which could explain the unexpected downward trend we observe in performance as model size increases.\n\n```>>> Q6``` A large number of models learn wrong visual intuitions?  \n```>>> A6``` Core-illusion refers to models' response driven by a natural perception of the world, i.e., a lack of core knowledge. Models that learn from statistical correlations in the data may fall short in acquiring core abilities, as MLLMs trained on vast, multimodal datasets are often biased by shortcut signals, producing answers that resemble advanced reasoning but lack the conceptual grounding that allows humans to apply knowledge flexibly and consistently across contexts (Mitchell, 2023). Due to limited space, kindly refer to Reviewer ZrNC's A2 for a technical explanation of why such a distinction can be rooted in the pretraining process.", "claims_and_evidence": "- In the conclusion, the authors write \"(2) models’ performance on high-level abilities does not correlate with the corresponding low-level abilities that ground them in humans\". However, as shown in Figure 4 there are some correlations between lower level abilities and higher level abilities (boundary and continuity correlate quite well with intentionality and mechanical reasoning). While I agree in principle that higher correlations between low and high level abilities could be expected, this claim reads to strong to me.\n- Furthermore, they write \" (3) such abilities exhibit very low scalability among models, meaning that simply raising the number of parameters could not better the models’ performance on these abilities\". While I agree that scale does not seem to yield perfect core abilities, it does look like almost every ability (except for perspective) benefits from scale in section 4.5. Sure, conservation and spatiality only slightly improve but all others seem to show some improvement with scale. Maybe this section would benefit from some numbers to make the plot a bit more digestible, such as \"sensorimotor ability accuracy improves by 0.X on average\". The resulting claim that \"while scaling improves high-level reasoning, its effect on low-level cognitive abilities is minimal and, in some cases, even detrimental\" therefore seems not perfectly supported, especially as the detrimental case (perspective taking) seems to have poor external validity.", "methods_and_evaluation_criteria": "The choice of experiments seems principled. Also, the study investigates a large number of models.", "theoretical_claims": "Not applicable", "experimental_designs_or_analyses": "-", "relation_to_broader_scientific_literature": "This paper contributes to a larger literature on how VLMs struggle with basic visual processing. It goes some way towards understanding what the missing components are that prohibit VLMs from having human-level visual understanding.", "essential_references_not_discussed": "There are three other works that are not mentioned and that showcase the weakness of VLMs on cognitive tasks. [1] investigates higher level visual cognition including intuitive physics. [2] also investigates intuitive physics in VLMs. [3] investigates visual illusions in VLMs and uses a technique that is similar to the \"concept-based hacking\" proposed here.\n\n[1] Schulze Buschoff, Luca M., et al. \"Visual cognition in multimodal large language models.\" _Nature Machine Intelligence_ (2025): 1-11.\n[2] Balazadeh, Vahid, et al. \"Synthetic Vision: Training Vision-Language Models to Understand Physics.\" _arXiv preprint arXiv:2412.08619_ (2024).\n[3] Ullman, Tomer. \"The Illusion-Illusion: Vision Language Models See Illusions Where There are None.\" _arXiv preprint arXiv:2412.18613_ (2024).", "other_strengths_and_weaknesses": "The paper is original in that it investigates specific low level core abilities in VLMs. The experiments are thorough and performed on a large number of models. The investigation is definitely timely.", "other_comments_or_suggestions": "Typos\n- Two dots in the abstract on line 29\n- Package import calls on line 1351 of the Appendix\n- Line 169 right column \"as\" missing between \"matching\" and \"the\"?\n\nText color\n- The yellow and red text color on page 5 is very hard to read and it is distracting in general.\n\nPhrasing\n\n- Line 233 in the left column reads \"[...] a clear upward trend can be identified as the concepts move from low-level to high-level. This can be concluded as MLLMs perform worse on lower-level abilities than on higher-level ones, or in other words, there exist core knowledge deficits in Multi-Modal Language Models.\" Maybe I misunderstand but I thought all 12 abilities were considered core abilities. Sure, if MLLMs struggle on the lower-level abilities there are core knowledge deficits but this sentence reads as if the difference between performance on low to high level abilities is what constitutes core knowledge deficits.\n- In 5.1 the authors write \"At the core of the control experiment lies a _novel_ technique termed concept-based hacking\". A similar technique has already been proposed for the investigation of visual illusions in VLMs in [3].", "questions_for_authors": "1. Could the authors maybe speculate on why the perspective tasks shows such a poor external validity and is the only task that does not scale with the number of parameters? It seems like something might be off here.\n2. In section 5, an agent with core knowledge should get control and manipulation right. A shortcut learner should get only the control right. And a non-core knowledge agent should only get the manipulation right. Basically, the latter would be a model that learns the wrong intuitions about basic visual properties, if I understand correctly. Now, the results seem to show that a large number of models actually fall into this category. Could the authors also speculate on why these models seem to learn wrong visual intuitions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gA0xkvqvWp", "reviewer": "Reviewer_Kcuh", "summary": "The paper investigates the hypothesis that the limitations of MLLMs in performing intuitive tasks, which are simple for humans, stem from the absence of \"core knowledge\"—innate cognitive abilities present from early childhood in humans. To explore this, the authors develop a novel benchmark called the CoreCognition dataset, designed specifically to assess these core cognitive concepts in MLLMs. The dataset covers 12 fundamental cognitive abilities and is used to evaluate 219 models across 10 different prompts, generating a total of 2,409 data points. The findings reveal that while MLLMs perform comparably to humans on complex, high-level cognitive tasks, they significantly underperform on simpler, low-level tasks. The study shows that there is no improvement in low-level cognitive abilities as model size and complexity increase, a stark contrast to high-level abilities which show scalable improvements. The authors also introduce “Concept Hacking” as an evaluation technique to demonstrate that MLLMs rely on superficial pattern recognition and shortcut learning rather than developing a genuine understanding of core knowledge.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "```>>> Q1``` typo: double dots.  \n```>>> A1``` Thanks. We will revise and remove all typos.\n\n---\n\n```>>> Q2``` Assume that replicating human-like core knowledge is essential for the effective functioning of AI systems is controversial and may not necessarily hold?  \n```>>> A2``` Thank you for the question! We approach this from 2 complementary perspectives: 1) The question presumes that core knowledge is unique to humans or human-aligned AGI. While this is resonable, it's also possible that core knowledge reflects more general cognitive principles or serves as mechanistic prerequisite that may be useful—or even necessary—for any form of general intelligence; such as chimpanzees showing behaviors aligned with core knowledge (Regolin and Vallortigara,1995; Santos, 2004; Lake 2017). This suggests that certain cognitive abilities emerge across intelligent systems, regardless of the path. Then even non-human pathways to AGI, e.g. scaling, might still lead to the emergence of these core abilities. In this light, a benchmark on core knowledge, such as ours, could offer a useful lens for evaluating the progress toward AGI, regardless of the path taken. \n2) Even if core knowledge is a human-specific trait, the human trajectory remains a valuable source of insight given the current absence of clearly defined path to AGI. Despite great advancements, MLLMs continue to struggle with hallucination, lack of OOD generalization and robustness, indicating that important cognitive capacities may still be missing. Humans as the only model of high-level intelligence can serve as a useful reference for evaluating artificial systems. Moreover, if AGI were to eventually follow a human-aligned path, our benchmark would become vital in assessing high-level reasoning and perception grounded in foundational cognitive structures rather than driven by spurious shortcuts.\n\n---\n\n```>>> Q3``` Assumption that core knowledge can be clearly defined and operationalized within the context of AI systems, but from pure evaluation based on QA answers, this could lead to benchmarks that do not accurately reflect the underlying theories or mechanisms of core knowledge.  \n```>>> A3``` While isolating core knowledge dimensions is notoriously difficult, particularly in AI, there are certain aspects that makes this effort feasible. Extensive literature suggests that core knowledge exists and can be robustly probed and evaluated in humans, including infants who don't yet possess language or the ability to speak, as in Appendix B. The taxonomy of 12 core-abilities addresses this challenge by operationalizing core knowledge through a set of lower-level abilities. These abilities serve as tractable proxies for otherwise abstract cognitive dimensions and also connect naturally to stage-based developmental theories (Piaget, 1976; Rochat, 2024). This makes it possible to evaluate how models handle fundamental cognitive operations and how such abilities may scaffold more complex reasoning.\n\n---\n\n```>>> Q4``` From pure evaluation based on QA answers, this could lead to benchmarks that do not accurately reflect the underlying theories or mechanisms of core knowledge.  \n```>>> A4``` We acknowledge that the VQA format is intertwined with confounding factors, such as language understanding (see Q5 of Reviewer nvFw). However, evaluating specific abilities in AI models inherently requires auxiliary task demands (Hu and Frank, 2024)—regardless VQA, retrieval, or output logits—and every evaluations and benchmarks face more or less a similar challenge. To mitigate biases introduced by the VQA format, our effort includes\n- Instruct annotators to minimize the interplay of different abilities and exclude questions that require multiple competencies to answer.\n- Manually filter out ambiguous or confusing questions and use LLMs to enhance clarity and precision.\n- Extensive experiments with various phrasing and prompting techniques to alleviate biases arising from specific wordings or prompt formulations.\n\n---\n\n```>>> Q5``` Can this benchmark provide valuable insights into the training or designing of MLLMs? How the benchmark results can help researchers improve their models?  \n```>>> A5``` Our benchmark reveals that current training methods fail to effectively emerge these great abilities, suggesting future research into large-scale pretraining methods that can better scale core knowledge. More concretely, if core knowledge cannot be directly scaled, it may be beneficial to first teach or distill core knowledge into MLLMs prior to large-scale pretraining, enabling more data-efficient generalization akin to human learning. From an evaluation and design perspective, our benchmark uncovers distinct failure modes, including the lack of permanence, spatiality, boundary, and continuity. These limitations further hinder abilities such as visual perspective-taking and contribute to an over-reliance on shortcuts, which is a fundamental cause of poor out-of-distribution generalization.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "- By assessing MLLMs against a dataset designed to probe these core cognitive concepts, the paper directly investigates whether state-of-the-art AI systems possess analogous foundational skills, which is crucial for developing AI that can genuinely simulate human-like intelligence.\n- The findings of the study provide a modern illustration of Moravec’s Paradox, showing that while MLLMs excel in high-level cognitive tasks, they struggle with basic cognitive tasks that are effortless for humans. Generally, the paper adds empirical data to ongoing debates about the nature of intelligence and complexity in AI systems.\n- By introducing “Concept Hacking,” the paper adds a novel method to the repertoire of AI evaluation, specifically designed to test whether models truly understand the tasks they perform or merely capitalize on patterns and shortcuts.", "essential_references_not_discussed": "N/A. There are some similar works that tackles the core knowledge in evaluating LLMs but not MLLMs.", "other_strengths_and_weaknesses": "From my point of view, I think my concerns mainly lines in the motivation behind the proposed benchmark. As mentioned, the paper’s motivation assumes that replicating human-like core knowledge is essential for the effective functioning of AI systems. This assumption is controversial and may not necessarily hold, as AI could potentially achieve high functionality through alternative means that do not mimic human cognitive processes. The debate on whether AI should replicate human cognition or develop its own unique methods remains unresolved and is a significant conceptual limitation of the study. Also, the motivation assumes that core knowledge can be clearly defined and operationalized within the context of AI systems, but from pure evaluation based on QA answers, this could lead to benchmarks that do not accurately reflect the underlying theories or mechanisms of core knowledge. Can this benchmark provide valuable insights into the training or designing of  MLLMs? How the benchmark results can help researchers improve their models? While the importance of core knowledge is widely acknowledged, I believe a good benchmark should serve purposes beyond simple evaluations.", "other_comments_or_suggestions": "typo: line 029 double dots.", "questions_for_authors": "Given the inherent challenges in learning core knowledge from merely memorizing training data, how do you believe the CoreCognition benchmark can help advance the development of MLLMs in this domain? Can this benchmark provide valuable insights into the training or development strategies that might enable LLMs to acquire better core knowledge capabilities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QCBgATy8yP", "reviewer": "Reviewer_nvFw", "summary": "This paper introduces the CoreCognition dataset focusing on a systematicity evaluation of multimodal models. Tasks in the benchmark are designed based on the well-established core-knowledge theory in developmental psychology. These tasks cover multiple aspects of human multimodal cognition, spanning from low-level perception tasks to high-level tasks that require tool use and strong commonsense knowledge. Evaluation of a large set of multimodal language models and correlation analysis between tasks & other benchmarks also justifies the contributions of the proposed benchmark.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "```>>> Q1``` The tasks are much harder than children cannot easily solve compared to developmental CogSci  \n```>>> A1``` Thanks for bringing up this nuance. We address the concern in two aspects. First, while all our tasks are derived from standard developmental cognitive science prototypes, e.g., \"Three-Mountain\", we extend the questions beyond the few typical examples from the textbooks to comprehensively evaluate the capabilities of MLLMs. This leads to a significantly larger and more systematic set of tasks and evaluations, rather than relying on the limited examples typically found in conventional literature. Second, it’s important to note that the subjects being evaluated are not infants or young children with limited acquired knowledge, but rather MLLMs adapted from large language models (LLMs), which are reported to possess human PhD-level knowledge and reasoning abilities [1]. \n\n[1] OpenAI (2024). Learning to reason with LLMs. https://openai.com/index/learning-to-reason-with-llms/  \n\n---\n\n```>>> Q2``` One thing that this paper needs further discussion on is how the authors performed the human study, for example, how many human participants were recruited and how were they paid?  \n```>>> A2``` We recruited college students as participants, each receiving the same questions as the MLLMS. To ensure attentiveness, we included 1% mismatched question-answer or question-image pairs, asking participants to flag unclear or complex items within 90 seconds. Participants were paid only if they passed the attention check. 22 participants' answers were accepted for final statistics.\n\n---\n\n```>>> Q3``` Providing more task examples in the supplementary would be more helpful.   \n```>>> A3``` Thanks for the advice. We have provided test cases in both the main paper and the supplementary. We will add more examples following your suggestion.  \n\n---\n\n```>>> Q4``` Suggested references.  \n```>>> A4``` Thanks for the advice. We will include all suggested papers in our citations and provide an extended discussion about them. \n\n---\n\n```>>> Q5``` Language is treated as a separate aspect of core knowledge, and it might account for explaining why this paper found low-level tasks are harder for models than high-level tasks. The VQA format, especially the fact that almost every task is entangled with language understanding, makes the core knowledge evaluated here not as clean as controlled experiments performed in the original developmental psych literature. This may lead to not directly comparable results (e.g., why low-level tasks are harder than high-level tasks for machines).  \n```>>> A5``` Thanks for raising this concern. We will address your concern in the following two aspects.  First, indeed, we acknowledge that the VQA format is entangled with language understanding, as demonstrated by the impressive capabilities of large language models (LLMs) in both language comprehension and their use as a tool for reasoning [1, 2, 3], we argue that the influence of language bias is relatively minimal when evaluating core knowledge.\n\nSecond, we contend that the core challenge lies in developing a scientifically robust evaluation methodology that can specifically probe the abilities of LLMs and MLLMs. Regardless of the evaluation method—whether VQA, retrieval tasks, or using output logits—assessing the model's specific capabilities requires auxiliary task demands [4]. In this context, the interaction between core knowledge and other factors (whether linguistic or otherwise) is inevitable, as LLMs inherently rely on language as a vehicle for expressing and processing information. \n\nWe hope this clarifies our rationale for using the VQA format. We will discuss this shortcoming in the limitations Section.\n\n[1] Millière, Raphaël. \"Language Models as Models of Language.\" arXiv, 2024, arXiv:2408.07144.  \n[2] Brown, Tom, et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.  \n[3] Wei, Jason, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, 2022. \n[4] Hu, Jennifer, and Michael C. Frank. \"Auxiliary task demands mask the capabilities of smaller language models.\" arXiv, 2024, arXiv:2404.02418.  \n\n---\n\n```>>> Q6``` How will the proposed dataset/benchmark be released, and how will the benchmark be maintained? I recommend doing something like a dataset card as the common practice in neurips benchmark tracks.  \n```>>> A6``` All data will be released upon acceptance and after internal review. Following standard procedures, we will open-source the dataset on GitHub and Hugging Face, including dataset cards, descriptions, and a viewer. Additionally, all model predictions and results will be made publicly available to ensure reproducibility.", "claims_and_evidence": "Claims made in the paper are well supported by both empirical and computational evidence. Overall, I like how each tasks are supported by the corresponding developmental psychology background. Model evaluations were also thoroughly performed, I can see most state-of-the-art vision-language models are covered. Correlation analyses and ablation studies are good. I do think there is a flaw underlying the task design, as all the tasks require language---usually, language is treated as a separate aspect of core knowledge, and it might account for explaining why this paper found low-level tasks are harder for models than high-level tasks.", "methods_and_evaluation_criteria": "Yes, the proposed evaluation criteria generally make sense. I do see one potential problem is the difficulties of all the tasks. In the developmental psychology literature, core knowledge usually focuses on children's ability to perform a basic understanding of the world, while the proposed tasks in this paper seem to be much harder problems that children cannot easily solve. However, this might be a minor issue as the authors are not considering studying the developmental or learning trajectory of these models.", "theoretical_claims": "There are no foreseeable issues with the theoretical claims made in this paper, as it is not a theory paper.", "experimental_designs_or_analyses": "Experimental designs and analyses reported in this paper sound good to me. One thing that this paper needs further discussion on is how the authors performed the human study, for example, how many human participants were recruited and how were they paid?", "relation_to_broader_scientific_literature": "This paper seems interesting to a broader audience, for example I think cogsci and developmental psychology people would be interested in the proposed benchmark.", "essential_references_not_discussed": "Some related benchmarks that also focus on evaluating core knowledge are missing in the reference, e.g., AGENT [A], and Binz & Schulz, 2023.\n\n[A]. Shu, T., Bhandwaldar, A., Gan, C., Smith, K., Liu, S., Gutfreund, D., ... & Ullman, T. (2021, July). Agent: A benchmark for core psychological reasoning. In International conference on machine learning (pp. 9614-9625). PMLR.\n\n[B]. Binz, Marcel, and Eric Schulz. \"Using cognitive psychology to understand GPT-3.\" Proceedings of the National Academy of Sciences 120.6 (2023): e2218523120.", "other_strengths_and_weaknesses": "Overall I like this paper due to i). the importance of introducing developmental insights to model evaluations, and specifically I think the core knowledge approach, as demonstrated in this paper, did a neat job. ii). model evaluations were carefully performed and I can clearly see some insights that might be helpful for the broader community.\n\nSome potential weaknesses from this paper might include i). the VQA format, especially the fact that almost every task is entangled with language understanding, makes the core knowledge evaluated here not as clean as controlled experiments performed in the original developmental psych literature. This may lead to not directly comparable results (e.g., why low-level tasks are harder than high-level tasks for machines).", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "How will the proposed dataset/benchmark be released and how will the benchmark be maintained? I recommend doing something like a dataset card as the common practice in neurips benchmark tracks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary

This paper investigates whether multi-modal language models (MLLMs) possess "core knowledge" - fundamental cognitive abilities that humans develop at a young age. The authors develop a novel benchmark called the CoreCognition dataset, comprising 12 core cognitive concepts derived from developmental psychology literature. Using this benchmark, they evaluate 219 models with 10 different prompts, generating 2,409 total data points. Their key findings are that (1) MLLMs perform well on high-level cognitive tasks but struggle with low-level core abilities, (2) there is little correlation between performance on high-level and low-level abilities, and (3) low-level abilities show minimal scaling improvements with increased model size. The authors also introduce "Concept Hacking" to demonstrate that MLLMs rely on superficial patterns rather than genuine understanding.

## Strengths

- The paper brings valuable insights from cognitive science to AI evaluation, creating a benchmark that systematically probes fundamental cognitive capabilities.
- The experimental methodology is rigorous, evaluating a large number of models across multiple prompting strategies.
- The findings about differential scaling of high vs. low-level abilities offer important insights for model development.
- The concept hacking technique provides a novel way to distinguish between real understanding and shortcut learning.
- The work is interdisciplinary and has potential impact beyond AI, particularly in cognitive science.

## Weaknesses

- The use of language-based VQA format introduces potential confounds when evaluating pure visual cognition.
- Some claims regarding correlations between abilities might be slightly overstated.
- The relationship between human cognitive development and desirable AI capabilities remains an open question.
- More details on human studies methodology would strengthen the comparative analysis.

## Discussion

All four reviewers were positive about the paper, with three giving accepts and one giving a weak accept. The reviewers appreciated the paper's interdisciplinary approach, comprehensive evaluation, and novel insights about model scaling with respect to cognitive abilities.

The authors provided thorough responses to all reviewer concerns, clarifying their methodology, moderating some of their claims, and addressing questions about the relationship between human cognitive development and AI capabilities. They acknowledged limitations in their VQA format but provided reasonable justification for their approach.

I have read the authors' message to the AC regarding the lack of further reviewer engagement after rebuttal. While I understand their desire for more interactive discussion, the reviewers all acknowledged reading the rebuttal and maintained their positive assessments. The consistent endorsement from reviewers, along with the authors' thorough responses, provides sufficient basis for a decision.

## Recommendation

I recommend acceptance of this paper. It makes a significant contribution to our understanding of MLLMs through a novel cognitive science lens, with important implications for future model development. The authors should incorporate their clarifications from the rebuttal into the final version, particularly regarding task difficulty, human evaluation methodology, and the limitations of the VQA format.
Decision: Accept (poster)