Reflection-Bench: Evaluating Epistemic Agency in Large Language Models
Authors: Lingyu Li
Yixu Wang
Haiquan Zhao
Shuqi Kong
Yan Teng
Chunbo Li
Yingchun Wang
Venue: ICML 2025 poster (2025)
Keywords: large language models
autonomous agent
cognitive psychology
Primary area: general_machine_learning->evaluation
TL;DR: Reflection-Bench is a cognitive psychology inspired, parameterized, and contamination-minimization benchmark that evaluates LLMs' intrinsic epistemic agency, the capacity to flexibly construct, adapt, and monitor beliefs about dynamic environments.

Abstract: With large language models (LLMs) increasingly deployed as cognitive engines for AI agents, the reliability and effectiveness critically hinge on their intrinsic epistemic agency, which remains understudied. Epistemic agency, the ability to flexibly construct, adapt, and monitor beliefs about dynamic environments, represents a base-model-level capacity independent of specific tools, modules, or applications. We characterize the holistic process underlying epistemic agency, which unfolds in seven interrelated dimensions: prediction, decision-making, perception, memory, counterfactual thinking, belief updating, and meta-reflection. Correspondingly, we propose Reflection-Bench, a cognitive-psychology-inspired benchmark consisting of seven tasks with long-term relevance and minimization of data leakage. Through a comprehensive evaluation of 16 models using three prompting strategies, we identify a clear three-tier performance hierarchy and significant limitations of current LLMs, particularly in meta-reflection capabilities. While state-of-the-art LLMs demonstrate rudimentary signs of epistemic agency, our findings suggest several promising research directions, including enhancing core cognitive functions, improving cross-functional coordination, and developing adaptive processing mechanisms. Our code and data are available at https://github.com/AI45Lab/ReflectionBench.

Reflection-Bench:
Evaluating Epistemic Agency in Large Language Models
Lingyu Li Yixu Wang' Haiquan Zhao
Shuqi Kong"
Yan Teng'
Chunbo Li
Yingchun Wang'
Abstract
With large language models (LLMs) increasingly deployed as cognitive engines for AI agents, the reliability and effectiveness critically hinge on their intrinsic epistemic agency, which re- mains understudied. Epistemic agency, the abil- ity flexibly construct, adapt, and monitor be- liefs about dynamic environments, represents base-model-level capacity independent of spe- cific tools, modules, or applications. We char- acterize the holistic process underlying epistemic agency, which unfolds in seven interrelated dimen- sions: prediction, decision-makes perception, memory, counterfactual thinking, belief updating, and meta-reflection. Correspondingly, pro- pose Reflection-Bench a cognitive-ssccholoy- inspired benchmark consisting of seven tasks with long-term relevance and minimization of data leakage. Through comprchensive evalu- ation of 16 models using three prompting strate- gies, we identify a clear three-tier performance hierarchy and significant limitations of current LLMs, particularly in meta-reflection capabili- ties. While tate--of-the-at LLMs demonstrate rudimentary signs of epistemic agency, our find- ings suggest several promising research directions, including enhancing core cognitive functions, im- proving cross-functional coordination, and devel- oping adaptive processing mechanisms. Our code and data are available wwe.5/aeaa/eeee.t...ec... com/AT Cam/AT451ab// Re lect onBench.
Polece
A ATB ?
Ensiroaincnó
Reflection
Counterfacteal
Dechiin-makau
thinking
Meusry
X Peroptiinn
Meta reflection
Figure I. Reflection, cyclic cognitive process underlying epis- temic agency, enables effective agent-environment interaction.
vironment and completing various tasks such program- ming, scientific research, and industrial production. (Xi et al., 2023; Wang et al., 2024; Schmidgall et al., 2025). The performance of these L.LMs-bssed agents fundamentally depends on high-order capacities of models, such as reason- ing, planning, recovering from errors, and learning from the environment, (Schlunnz & Zhang, 2024). Collectively, these abilities reflect model's intrinsic epistemic agency- the capacity to flexibly construct, adapt, and monitor beliefs about dynamic environments (Ferrero, 2022; Fairweather & Montemayor, 2017: Chrisman, 2022). This epistemic agency, as pase-model-leva characteristic independent of specific external modules or applications, determines whether LLMs can truly serve as reliable cores of AI agents. However, existing studies either focus narrowly on agents' applications examine isolated abilities (Liu al., 2023; Clark et al., 2018 8; Valmeekam et al., 2024). None of them investigates LLMs' epistemic agency unfolding in the holis- tic process of agent-environment interaction. Furthermore, current evaluations' reliance on text-heavy datasets intro- duces methodological limitations, particularly the risks of
/ttta://githb...
1. Introduction
Large language models (LLMs) have been increasingly ap- plied as the brains for Al agents, interacting with the en-
Work done during Lingyu's internship at Shanghai Artificial In- telligence Laboratory. 'Shanghai Artificial Intelligence Laboratory, China "Shanghai Mental Health Center, Shanghai Jiao Tong Uni- versity School of Medicine, China. Correspondence to: Yan Teng <tengyan @ pjlab.org.cn. Chunbo Li <licb@ smhc.org.cn>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ceflection-Bencle
benchmark leakage (Zhou et al., 2023; Xu et al., 2024).
limitations of LLMs' epistemic agency, especially in pre- diction, decision-making, and meta-reflection. The effec- tiveness of prompting strategies varies substantially across both tasks and models. We further evaluated these models on a more challenging parameter set using the direct gen- cration strategy. The expected score decrease suggests that Reflection Ceflection-Benchh is far from saturated. Additionally, we compared the performances of Centaur, which specifically fine-tuned with human cognitive test data (Binz et al., 2024), and its base model, Llama-3. 1-0--...1-7----- 70s-runttrut.. Centaur shows no improvement compared to its base model, providing em- pirical evidence that our parameterized design effectively minimizes data leakage concerns.
To evaluate LLMs' epistemic agency, we propose Reflection- peteeelloyyccciiiieeeee
Bench, a minimization of data contamination. Our framework is mo- tivated by mental processes of how human epistemic agency enables efficient agent-environment interaction. As illus- trated in Figure 1, this cyclical process involves predicting based on prior belief, making decisions to achieve desired states, perceiving surprises, subsequently modifying priors through recalling and counterfactual thinking (Friston et al., 2017; Parr et al., 2022; Dennett, 1993; Friston, 2010; Huang & Rao, 2). The complete cycle embodies reflection - the "attentive, critical, exploratory and iterative interac- tions with one's thoughts and actions with the intention to change them (Nguyen et al., 2014)." We therefore adopt reflection, defined here as a process rather than an ability, as our framework to investigate LLMs' epistemic agency. Through detailed analysis of this reflection process, we iden- tifyy seven key dimensions that constitute robust epistemic agency, including prediction, decision makinn, perception, memory, counterfactual thinking, belief updating, and meta- reflection.
benchmark with
To summarize, our main contributions include:
Development of Ceflection-Bencie cognitive- psychology-inspired benchmark that evaluates LLMs' epistemic agency through parameterized and cnnpriiitttiommmiiiiaatii tasks.
Comprehensive evaluations across 16 LLMs using three prompting strategies, revealing distinct perfor- mance tiers and demonstrating Reflection- 's ef- fectiveness in differentiating models' epistemic agency.
To evaluate these seven dimensions, design the Reflection-Bench with seven cognitive tests adapted from validated paradigms in cognitive psychology. Cognitive tests create controlled environments where subjects must learn and reason about unknown parameters through interac- tion, offering standardized, quantified, and objective assess- ment tools that mirror real-world functioning (Lezak et al., 2012; Allen et al., 2024). We select and adapt seven cogni- tive tests for LLMs evaluation, each focusing one func- tion shown in Figure 1 (Nyhus& & Barceló, 2009; Shohamy et al., 2008; Näätänen et al., 2007: Jaeggi et al., 2010; Buelow & Suhr, 2009; Bouneffouf et al., 2020; Bechara et al., 2013; Cools et al., 2002). The parameterized design of cognitive tests allows their configurations be customized even if model has memorized the task format, it can- not know the specific parameters used in evaluation. This design both reduces potential data contamination and en- sures Reflection-Bertte 's long-term relevance by enabling difficulty adjustment to accommodate future Al models.
fulti-dimensional behavioral analyses across mod- els, tasks, and prompting strategies, uncovering both emerging epistemic ageney and fundamental limita- tions in current LLMs, and providing insights for future research directions.
2. Related work
2.1. LLMs-based Agents
The concept of Agent originally describes several key as- pects of biological systems: (1) flexibly interacting with the environment to achieve a range of goals; (2) making choices actively; and (3) being accountable for the outcomes of their actions (Kockelman, 2006). and higher agency manifests as enhanced sophistication of these three aspects. With the emergence of high-order capabilities such as understanding and reasoning (Wei et al., 2022), LLMs are increasingly deployed as agents, where they "dynamically direct their own process and tool usage, maintaining control over how they accomplish tasks" (Schluntz & Zhang, 2024). Similar to biological agents, these LLMs-based agents work in in- teractive freobn-feedback loops to complete complex tasks, operating various architectures for better performances than naïve LLMs (Yao et al., 2022; Nakajima, 2023; Hong et al., 2023). Correspondingly, multiple benchmarks have been developed for evaluating these agents' capabilities of performing specific tasks such as shopping, gaming, UI in- terface, customer service, travel, jailbreak -proff and so on
We conducted comprehensive evaluations using entry-level configurations across leading large reasoning models, main- stream LLMs, and the Qwen-2.5 family with varying sizes. Three prompting strategies were employed, including di- rect generation, free output, and Chain of Thought (CoT) (Wei et al., 2024). The results reveal a clear three-tier hi- erarchy, with seven state-of-the-an models scoring over 60, eight moderate models between 50 and 60, and the model with the smallest size scoring below 50 points. While these results demonstrate certain epistemic agency in cur- rent LLMs, detailed behavioral analyses reveal significant
2

---

Reflection-Bench
(Liu et al., 2023; Deng et al., 2024; Xiao et al., 2024; An- driushchenko et al., 2024). Apart from application-speeife evaluations, prior research has examined fundamental capa- bilities required for LLMs to function as agents, including reasoning (Clark et al., 2018), planning (Valmeekam et al., 2024), causal inference (Chen et al., 2024a), probability estimation (Krishnamurthy et al., 2024), cognitive flexibility (Wilie al.. 2024), decision making (Li et al., 2024a), and spatial cognition (Madge & Poesio, 2024) etc. In this work, we specifically focus on epistemic agency, the ability to flexibly construct, adapt, and monitor beliefs about dynamic environments (Ferrero, 2022; Fairweather & Montemayor, 2017; Chrisman, 2022). This case-model-leve capacity, independent of specific external modules or applications, directly determines whether a model can reliably serve as the "brain" of Al agent.
compose the epistemic agency into seven core capabilities that emerge in temporal order, as illustrated in Figure 1.
The process begins with agents' prior beliefs about the environmental states manifesting in LLMs as both world model (Yildirim & Paul, 2024) and task-specific expecta- tions through zero/few-shot comprehension (Brown et al., 2020). Priors help establish a framework of task objec- tives, constraints, possible inputs and outputs, etc. Predic- tion, computed via transition probability (Friston, 2010) enables agents to hypothesize future states given potential actions. For LLMs. prediction capabilities are crucial for planning tasks, where models must reason about which poli- cies will effectively transition an agent from its initial state to a desired goal state. (Valmeekam et al., 2024; Fu et al., 2024). Then, agents will determine concrete actions to ex- ecute based prediction, i.e., decision-making (Shadlen & Kiani, 2013), constituting the material foundation for agent-environment interaction.
2.2. Cognitive Psychology Methods for Evaluating LLMs
After policy execution, agents perceive environmental feed- back, where the most valuable information is discrepancies from predictions, such as errors and penalties - a basic fea- ture underlying both human cognition and reinforcement learning (Den Ouden et al., 2012). As crucial signals driving modification and adaptation, the prediction errors potentially originate from deviant internal models or changed environ- ments. Two mental processes are triggered by prediction errors: recalling the past process and simulating choices not taken, (Byme, 2016), i.e., memory and counterfactual thinking. respectively.
Cognitive psychology, which aims to understand human behavioral and mental processes (Sternberg & Sternberg, 2006), provides an effective framework for interpreting LLMs' capabilities. As LLMs grow in complexity, re- searchers increasingly adapt cognitive psychology methods to evaluate them. The core idea behind it is to treat LLMS as humanoid subjects in psychology experiments to probe their capabilities and mechanisms of cognitive traits (Binz & Schulz, 2023). Diverse paradigms and tests in cognitive psychology have been adapted for evaluating L.M,s, includ- ing psychometries for attributes (Li et al., 2024b), iterated learning for prior knowledge (Zhu & Griffiths, 2024). Wis- consin card sorting test and letter number test for cognitive flexibility (Kennedy & Nowak, 2024), implicit statistical learning and facial recognition etc. for effects of CoT (Liu al., 2024b), Montreal Cognitive Assessment for cognitive impairment (Dayan al., 2024), Cognitive Reflection Test for limen-like intuition (Hagendorff et al., 2023) and so on. This method provides a top-down perspective toward explainable AI through behavioral analysis rather than di- rect interpretation of neural network activities (Hagendorff, 2023). Furthermore, just as we trust humans in everyday tasks based on their demonstrated cognitive capabilities and behavioral patterns, this approach potentially helps establish similar trust mechanisms for LLMs- based agents (Shiffrin & Mitchell, 2023).
Through these two retrospective analyses, agents update their prior beliefs that more accurately describe the envi- ronment to perform better in the future (Parr et al., 2022). In LLMs, this error-driven learning mechanism manifests through in-context learning and causal reasoning, enabling dynamic belief adjustment to recover from errors (Dong et al., 2024; Chen al., 2024a; Krishnamurthy et al., 2024; Schluntz& & Zhang, 2024). Beyond such instant interactions, agents like humans also possess higher-order capabilities of meta-cognition monitor, evaluate, and regulate their own cognitive process (Martinez, 2006). When it comes to the reflection process defined above, the meta-reflection enables agents to transcend local adaptation by analyzing patterns across multiple anddiitioonaction-feeleee thereby grasping global patterns of the environment.
cycles,
To summarize, we identify seven interlinked cognitive di- mensions that constitute robust epistemic agency: predic- tion,
3. Reflection-Bench
perception, memory, counterfactual thinking, belief updating, and meta-reficction. This process- oriented framework enables systematic evaluation of LLMS' epistemic agency through well-defined cognitive tests.
3.1. Define Dimensions of Epistemic Agency
To systematically evaluate epistemic agency, we identify key dimensions by analyzing the cognitive process of how agents implement robust interaction with dynamic environ- ments. Through the lens of cognitive psychology, we de-
3

---

Reflection- Benchh
3.2. Select Cognitive Tests
opportunity to revise their decision. By forcing models to reconsider past choices with opportunity 'turn back time', DC-IGT implements the mechanism of counterfactual thinking creating an alternative to reality by considering 'what if" (Byrne, 2016).
While these seven dimensions are interconnected in agent- environment interaction, cognitive psychology provides vali- dated test paradigms that isolate and measure specific cogni- tive capabilities while controlling other dimensions (Lezak et al., 2012). This enables our evaluation of each dimension through its corresponding cognitive test, as illustrated in Figure 2.
Belief updating We employ probabilistic reversal learn- ing task (PRLT) (Cools et al., 2002). where participants choose between two options with differing reward probabil- ities. These probabilities reverse midway through the task without notice, requiring models to update their prior beliefs about the reward distributions in response to environmental changes.
Prediction We select the weather prediction task (WPT) paradigm, where subjects originally predict the weather based on card cues by learning probabilistic relationships (Shohamy et al., 2008). We adapt this paradigm by con- verting these relationships into explicitly defined transition probability matrices. Models must learn how the cues in- fluence the weather transitions, allowing us to assess their prediction capability.
We design a meta-bandit task (MBT) that extends the previous paradigm of PRLT by introducing pre- dictable reversals. The reward probabilities (0 or 1) are reversed every trials, establishing higher-order pattern. This design tests the model's ability to recognize and an- ticipate the meta-structure of changes, i.e., meta-reflection, beyond mere adaptation to individual reversals.
Meta-reflection
Decision-making We employ the Wisconsin card sorting test (WCST) to assess flexible decision fecision-making (Nyhus & Barceló, 2009). Participants match cards varying in color, shape, and number of figures according to an undisclosed rule (e.g.. by color) that must be inferred from feedback. The matching rule changes without warning after a set number of trials, requiring models to infer and adapt to the latent rule governing correct decisions.
3.3. Adapt Tests for Evaluating LLMs
Weather prediction task (prediction) In each trial, the L.M receives two inputs: the current day's weather and a sensor state ([0,1] or [I,0]). Based on these two inputs, it predicts the next day's 's weather (example in AppendixA.2. The actual weather is calculated with the corresponding transition matrix:
Perception We select the oddball paradigm (Näätänen et al., 2007) to evaluate the automatic perception of surprise signals. The paradigm presents sequences containing fre- quent standards interspersed with rare deviant stimuli (e.g., different tones), measuring the automatic detection of nov- elty. Research shows that the brain is naturally sensitive to deviant stimuli (Garrido et al., 2009), making this paradigm ideal for assessing basic surprise detection. Impaired perfor- mance on this test is reliable indicator of cognitive deficits in conditions like schizophrenia (Umbricht & Krljes, 2005).
p
1- p
T ssensors=[1.0)-
1-P p P
1-p p P
T [=
p 1-p]
For subsequent trials, LLM receives the actual weather (which becomes the current weather) and new sensor state. We estimate the model's internal transition matrices from its predictions. Performance is evaluated using the mean absolute error (MAB) between estimated and true transition matrices.
Memory We adopt the n-back task to evaluate active mem- ory retrieval (Jaeggi et al., 2010). In this task, subjects view sequential stimuli and must indicate whether the current stimulus matches the one presented n steps earlier. This continuous process of updating and maintaining informa- tion directly assesses the memory capabilities needed for recalling previous decisions.
Score = (1 - MAE/MaxMAE) 100
Wisconsin card sorting test (decision-making) Follow- ing (Kennedy & Nowak, 2024), we implement WCST in text-based task with x trials. The matching rule (shape, color, or number) changes every x/6 trial, allowing each rule to be tested twice. In each trial, the LLM receives target card description (e.g., 'triangle green 4') and must match it with one of four choice cards without knowing the current matching rule. The model receives feedback after each choice (example in Appendix A.3). Performance is evaluated based on overall matching accuracy.
Counterfactual thinking We adapt the Iowa gambling task (IGT) (Bechara et al., 2013) into double choice ver- sion (DC-IGT) to assess counterfactual thinking. In tradi- tional IGT, participants select cards from four decks with different reward-loss profiles to maximize profit. To explic- itly test counterfactual thinking, our DC-IGT allows models to observe the outcomes of their initial choice and provides

---

Ceflection-Bench
Weather Station
Mismotch Negutivity
Sensore
I
S
Stimull
Stimuli
Prediction
X
X
Standard
Deviant
Loan
Half of the total trials
card
Fewardd
Subject:
X
I'm not match the NUMBER of objects.
ore bardl
reward. hish
card
Match it!
Each n trials
Freward
Subject:
Now see. The motching rule is based on COLOR!
Rewerd
rewerd
Low reward, low risks
are
Figure 2. Illustration of seven cognitive tests involved in Reflection Benchh A. weather prediction task; B: Wisconsin card sorting test; C: Oddball paradigm; N-back; E: Iowa gambling task; probabilistic reversal learning task; G: meta-bandit task.
Oddball paradigm (perception) We adapt the Oddball paradigm into a prompt-based task where each prompt con- tains eight short sentences: (stimulus A) and unrelated sentence (stimulus B) that disrupts the content flow (example in Appendix A.4). The LLMs are instructed to freely make comments on these prompts, allowing assessment of their spontaneous detec- tion of contextual anomalies. We evaluate the responses using both rule-based manual scoring (0-3 scale) and an automated method based on OpenAl's text-embedding-3 large (detailed in Appendix D). The test includes 50 distinct prompt sets.
Probabilistic reversal learning task (belief updating) In PRLT, the LLM repeatedly chooses between two options (left and right arms) with complementary reward probabili- ties of p and 1-p. After each choice, reward is drawn from Bernoulli distribution according to the chosen option's 's probability. At the midpoint of trials, these probabilities are reversed. The LLM receives the reward feedback and makes its next choice (example in Appendix A.7). We estimate the model's internal representation of reward probabilities using a moving average (window size = 5) of its choices. Performance is quantified using the mean absolute error between estimated and true probabilities:
about a consistent topic
Score = (1 MAEE MaXMAE) 100
N-back (memory) We present fixed sequence of letters (e.g., E, F, G, H) to the LLM one at a time. For each letter, the model must determine whether it matches the letter shown steps earlier in the sequence (example in Appendix A.5). Performance is measured by response accuracy.
Meta-bandi task (meta-reflection) The MBT consists of 20 blocks of trials each, where the LLM repeatedly chooses between two options with deterministic binary re- wards, with one option yielding and the other 0. After cach choice, the model receives feedback on the sampled reward. The reward mappings reverse every trials without notifica- tion (example in Appendix A.8). Performance is evaluated based the model's ability to recognize this fixed reversal pattern, as indicated by sustained reward acquisition even during reversal trials.
Double choice Iowa gambling task (counterfactual think- ing) The DC-IGT features four card decks with fixed gains ($100, $100, $50, and $50) and potential losses ($260. $1250, $50, and $200) occurring with possibilities of Ploss Pb. Pe, Pa Each trial consists of consec- utive choices. The LLM first selects deck and receives feedback on the resulting gain and potential loss. Based this feedback, it makes a second deck selection (example in Appendix A.6). Performance is evaluated using both short- term metrics (adaptive switching behavior to avoid losses) and long-term metries (cumulative net earnings).
4. Experiment
4.1. Experimental Setup
We first evaluate 16 LLMs on the Reflection-Bendd
Large reasoning models: ol ol-preview, ol-mini (Ope-
5

---

Reflection-Bent
4.2. Experimental Results
Table I. Experiment settings
Parameters
4.2.1. OVERALL RESULTS
Task
Easy
Hard
On entry-level difficulties, Claude-3.5 3.5-ooneett achieved su- perior performance with both zero-shot CoT (68.78) and free output (68.23), closely followed by two large reason- ing models - ol -preview (66.57) and SeepSeek-Reasone (64.56). The performance distribution reveals a clear hierar- chical structure aligned with model scaling:
WPT
p=0.9
p=0.8
WCST
x=72
x=90
Oddball
NA
NA
N-back
n=2
n=4
DC-IGT
Ploss
15,,1,,5,.11
{.5, 2, .5, .2)
PRLT
p=0.8
p=0.7
MBT
n=2
n=4
First-tier ( (>60 points); seven leading models in- cluding Claude-3.5-Sonner, ol-preview, DeepSeek- Reasoner,
nAI, 2024), DeepSeek-Reasonee (R1) (DeepSeek-Al, 2024), and QwQ-32B-Previne (Qwen, 2024)
GPT-4o, Grok-2, and
DeepSeek-V V3
Second-tier (50-60 points), eight intermediate models including Llama-3.3-70B, Claude-3.5-Haiku GPT-4o- mini, and Qwen-2.5 variants (72B/32B/14B)
Mainstream LLMs: GPT-4o, GPT--40mini (OpenAI, 2023), Slaude3.5--oonee Claude-3.5 (An- thropic, 2023), Grok 2 (xAI, 2024), Gemini-2.0-fit (Google, 2024), DeepSeck-V V3 (Liu et al., 2024a), and Llama-3.3-70B (Meta, 2024)
Third-tier (<50 points): Qwen-2.5-7B-Instnn the smallest model evaluated.
Qwen 2.5 family: variants with 72B, 32B, 14B, and 7B (Yang et al., 2024).
Notably, all models struggle with the MBT, failing to iden- tify the 2-step reversal pattern and making errors consis- tently across 20 blocks (Figure 15, 16), discussed in Section 4.2.2. Consequently, the reported average scores exclude MBT. As detailed in Table 9, none of the 16 mod- els achieved performances above chance-level thresholds across all five applicable tasks.
All evaluations are conducted through respective model APIs. Table 1 (Easy) details the number of trials and param- eter settings for each task. To ensure evaluation robustness, we repeat most tasks twice. The Oddball task, given its rela- tively subjective scoring, is repeated three times to minimize potential biases. We implement three response formats (see Appendix A. 1): free output, direct choice generation, and zero-shot CoT (Wei al., 2024). Due context length con- straints, large reasoning models are evaluated only through free output and direct generation, as their CoT responses often exceed maximum token limits.
Regarding prompting strategies, free output and CoT out- performed direct generation across tasks. Strategy effective- ness varied by models and tasks (see Figure 9). ol-preview demonstrates remarkable consistency between free output and direct generation. Task sensitivity to prompting strate- gies ranged from high (DC-IGT & N-back) to low (PRLT & Oddball Test).
To validate the long-term relevance of Reflection-Benche we evaluate these 16 LLMs on more challenging settings (Table Hard) with the direct output strategy. Oddball test excluded because it is not parameterized. To validate the eval-
On hard settings, we observed the expected score decreases across the evaluated models, as shown in Table This indicates the long-term relevance of Reflection-Bench and substantial room for LLMs development. Additionally, Cen- taur, the model fine-tuned with human cognitive test data, showed no improvement on Reflection-Bende compared to its base model, -10ma-3.--70--Instrree (see Table 9). This provides empirical evidence that our parameterized design effectively minimizes data leakage concerns.
of Reflection-Bench uate Centaur, which is specifically fine-tuned with human cognitive test data (Binz et al., 2024), and its base model, Llama-3.1 1-0-....-111111 Instruut (Grattafiori et al., 2024), on both Easy and Hard settings with the direct output strategy.
iilamminatio-miiimizaliii
We also conducted million random simulations for each applicable task, excluding the panatmeemttri Oddball test and qualitative MBT. By randomly generating choices, we established chance-level thresholds at the 95th per- centile of performance distributions. Performance exceed- ing these thresholds indicates statistically significant task performance, with higher scores reflecting a more precise inference of task parameters.
4.2.2. TASK-SPECIHIC ANALYSIS
Weather Prediction Task To reduce task complexity, implemented highly deterministic transition probability ma- trices (p=0.9). The task design specified that given cue '[1,0],, the next day would likely remain unchanged (p=0.9), whereas cue '[0,1]' indicated a probable transition to the opposite state. Despite this simplified probability structure,
6

---

Ielectiio-Bennl
Table 2.
Average Scores of 44 Model-Strategy, Pairs on Reflection-Bor Binecthionn-encc
Model
Strategy
Score
Model
Strategy
Score
Free output 64.94
Free output
54.60
2nd 0l ol-preww
ol-mini
Direct generation
66.57
Direct generation
54.29
Free output
61.47
Free output
58.64
3rd DeepSeck-Reasone
QwQ-32B-preview
Direct generation
64.56
Direct generation
48.37
Co'T
62.56
Co'
48.41
GPT-4o
Free output
52.95
GPT-4o-mini
Free output
57.09
Direct generation
57.29
Direct generation
47.44
CoT 68.78
CoT
57.94
Free output
68.23
Claude-3.--Haiku
Free output
58.14
Direct generation
62.59
Direct generation
55.40
CoT 61.25
CoT 54.79
DeepSeek-V3
Free output
59.50
Qwen-2.5-72B-Instrud
Free output
56.74
Direct generation
52.91
Direct generation
54.26
CoT
61.00
CoT 51.06
Grok-2
Free output
61.63
Qwen-2.5-32B-Instruee
Free output
48.01
Direct generation
53.29
Direct generation
49.21
CoT 61.51
CoT
57.06
Gemini-2. 0-flash-exp
Free output
63.29
Qwen-2.5-14B-Instme
Free output
52.81
Direct generation
51.13
Direct generation
52.94
CoT 52.44
CoT 44.53
Llama-3.3-70B
Free output
58.92
Qwen-2.5-7B--nsttee
Free output
46.66
Direct generation
53.01
Direct generation
40.37
most models exhibited difficulty in simultaneously learning two distinct transition patterns. As illustrated in Figure 10, our analysis revealed five prediction patterns: A: correct predictions, predictions merely by today's weather, C: predictions solely by the sensor's states, partially cor- rect predictions, and random predictions. Among all 44-model-stratege combinations (detailed in Appendix 3), only Claude-3.5-Somne with CoT successfully identified the transition patterns across both sessions. Four other com- binations (ol fmiimin with free output, Claude-3.5-Somee with free output, and DeepSeek-Reane Reasoner with both free output and direct generation) learned the pattern in one session. The remaining 39 combinations showed varying degrees of failure in capturing the underlying transition probabilities.
specific decision rules, with Llama-3.3-708% (CoT) being notable exception, showing persistent difficulties. How- ever, following the initial rule group, a prevalent "shape sink' phenomenon emerged models defaulted to shape- based sorting despite rule changes. This 'sink' pattern persisted even in top-performing models like op-prevwww and Claude- 3.5-Sonnet, which showed marked performance declines during the third rule group. These findings underscore fun- damental limitation in LLMs' ability to adapt to sequential rule (or environment) changes.
Oddball Test Our rule-free automated evaluation ap- proach, leveraging rext-embedding-3-largg and cosine simi- larity metrics, demonstrated robust agreement with manual rule-based scoring (r=0.87, p=5.83e-60; detailed analysis in Appendix D. This methodology revealed distinet patterns in models' contextual anomaly detection capabilities (Ta- ble 5). Llama-3.3-70B and Claude-3.5-Sonnet exhibited superior sensitivity to deviant stimuli, followed by Gemini- 2.0-flashe O0-flah-exp,, demonstrating advanced capabilities in unex- pected information detection. Conversely, Qwen-2.5-7B and ol prevveww frequently overlooked contextual deviants, sug- gesting that contextual sensitivity may be more dependent on specific training techniques rather than model sizes.
Wisconsin Card Sorting Test In the WCST evaluation (Table 4), ol-preview and Claude-3.5-Somae achieved iden- tical top scores 225. significantly outperforming Gemini- 1.5-pro (64.58) in second place, comprehensive analysis across 72 trials comprising rule groups revealed distinctive behavioral patterns (Figure 11). The cross-model average accuracies for successive rule groups were 80.02% (trial 1- 12, shape), 44.22% (trial 13-24, color), 26.04% (trial 25-36, number), 76.42% (trial 37-48, shape), 46.97% (trial 49-60,
color), and 45.36% (trial 61-72, number), respectively. The consistently high performance in shape-based based rule groups suggests that most L.LMs can effectively learn and apply
N-back In the 2-back task (Table 6), three models - ol- preview, DeepSeek- V3, and Grok-2 achieved perfect

---

Ceflection-Bencl
scores. Despite full conversation history being available dur- ing evaluation, many models struggle to determine whether the current stimulus matches the one presented 2 steps ear- lier. Given that the baseline accuracy for an al response strategy is 58%, models performing below this threshold likely demonstrate fundamental deficits in working memory capacity, specifically in maintaining and comparing infor- mation across multiple interaction turns.
that current LLMs struggle abstract global patterns from experience and leverage them for future decision-making This constraint represents critical limitation for LLMs as autonomous agents.
4.3. Effects of Prompting Strategies
Figure 17 presents comparative analysis of three prompt- ing strategies. Zero-shot CoT and free output achieved similar overall performances, both surpassing direct gen- cration. Breaking down the impacts by task reveals that zero-shot CoT exhibited notable advantages in WCST and 2- 2-back tasks while showing reduced effectiveness in DC- IGT. The impact of zero-shot CoT varied across models as well: while it enhanced GPT-4o's performance, it degraded the performance of models like GPT--4- mini to levels com- parable with direct generation. These findings suggest that the efficacy of zero-shot CoT is contingent on a model's capacity to leverage structured reasoning prompts. The mixed effects can be attributed the inherent integration of CoT--like mechanisms in many LLMs, potentially lim- iting the additional benefits of explicit zero-shot CO In the context of Reflection-Bench these results indicate that autonomous agents operating in complex environments re- quire task-specific cognitive strategies some tasks benefit from rapid, intuitive responses where extended delibera- tion may be counterproductive (Liu et al., 2024b), while others demand systematic, structured reasoning(Sprague et al., 2024). This pattern parallels human cognition theo- ries, where adaptive strategy selection is crucial for effective problem-sol across diverse scenarios.
Double Choice Iowa Gambling Task Our evaluation of DC-IGT incorporated both short-term and long-term per- formance metrics. The short-term analysis (Figure 12) ex- amined four behaviors- insisting with gain, insisting with risk, unnecessary switch, and loss-avoiding switch. Among these, insisting with gain emerged as the dominant behav- ior. Some models, notably GPT-4o-mini with free output, demonstrated optimal recision-making through balanced deployment of insisting with gain and loss-avoiding switch strategies. In contrast, Claude-3.5-Sonme with free out- put exhibited frequent risk-insisting behavior, resulting in poor overall performance (Table 7). To differentiate be- tween strategic adaptation and chance- based success, we im- plemented long-term metrics focusing on expected reward learning across four card decks. This approach specifically evaluated whether forced counterfactual thinking enhanced strategic optimization. The long-term analysis (Figure 13) tracked average coverage changes across model-strategy pairs, highlighting eight representative cases. Gemini-2.0- flash-exp excelled in both temporal dimensions, achieving the highest overall score and demonstrating robust counter- factual reasoning abilities.
Probabilistic Reversal Learning Task Beyond aggregate performance metrics (Table 8), our behavioral analysis (Fig- ure 14) revealed distinct patterns in belief updating across different model-strategy pairs. All five models scoring be- low 50 points exhibit obvious rigid beliefs, failing to adapt to the probability reversal after trial 20. Models achiev- ing scores above 60 points demonstrated adaptive belief updating capabilities. A prevalent strategy emerged across most models, suggesting a capacity for local adaptation. We further explored whether LLMS can develop a higher-order understanding of environmental structure through MBT.
5. Discussion
Key Findings The hierarchical performance pattern indi- cates the Benichtion-Bench's 's discriminative power. The sup- plementary evaluations verify Cellection-Benche long-term relevance and resilience against data contamination. On entry-level configurations, top-tier models achieve fair per- formance in most individual tasks except MBT, demonstrat- ing a basic level of epistemic agency. However, detailed be- havioral analyses also reveal significant limitations of LLMs' capabilities, especially in prediction, decision-making, and mefa-reflectton. Illustrated by performances in PRLT and MBT, current LLMs, without global structural understand- ing and meta-cognitive regulation. are largely driven by short-sighted local adaptation patterns. The effectiveness of prompting strategies varies substantially across both tasks and models, suggesting that the interplay between task char- acteristics and interaction strategies significantly impacts agent performance. Notably, no model combination maintains consistently good performance across all tasks. Altogether, our evaluation reveals both the emergence and limitations of epistemic agency in current LLMs.
Meta Bandit Task In MBT, a striking limitation emerged: no model successfully recognized the 2-step-reversal pattern (rewards alternating between options every two trials) across 20 reversal blocks (Figure 15, 16). Instead, all models exhib- ited either periodic or irregular errors. While most models defaulted to the effective in PRLT, this reactive approach proved insufficient for capturing the global temporal pattern in MBT. This in- ability to identify even simple temporal patterns reveals
lose-swics- strategy that proved
8

---

Reflection- Bench
Implications Our findings provide several implications for developing more robust epistemic agency in language models. The substantial weaknesses observed in meta- reflection suggest future research needs to focus on inno- vations that enhance meta-cognitive capabilities (Scholten et al., 2024), which would benefit from actively regulating thought processes and contributing to rational reasoning, improved learning, and reliable erriiionnmmaking (Griffiths, 2020; Boureau et al., 2015). Develop prompts or fine-tuning strategies that encourage dynamic shifts between rapid "in- tuition" and more deliberative "reflection" based on sit- uational demands. While long-form CoT reasoning has gained prominence in advanced models such as OpenAT's ol, concerns about its drawbacks have emerged (Chen et al., 2024b; Shaikh et al., 2023; Liu et al., 2024b), highlighting the importance of situation-appropriate transitions. Beyond enhancing individual capabilities, a key challenge lies in fostering organic coordination between different cognitive processes. As discussed in Section 3.1, such coordinated development is essential for achieving genuine epistemic agency across diverse roal-world scenarios. Current research provides limited insights into how different cognitive com- ponents interact within LLMs. Understanding these internal dynamics, particularly how different aspects of epistemic agency emerge and coordinate, could enhance both model interpretability and trustworthiness.
which could further address contamination concerns. Over- all, while current benchmark provides novel frame- work for evaluating epistemic agency in LLMs, these di- rections for future work highlight the rich landscape of possibilities for a deeper understanding of artificial agency.
6. Conclusion
This paper presents Reflection Benchh, novel benchmark for evaluating the intrinsic epistemic agency of Large Lan- guage Models through the lens of cognitive psychology. By examining the cognitive process of efficient agent- environment interaction, we decompose epistemic agency into seven interrelated dimensions, including prediction, decision-making perception, memory, counterfactual think- ing, belief updating, and meta-reflection. Building on es- tablished paradigms in cognitive psychology, we select and adapt seven cognitive tests for Reflection-Bench. The pa- rameterized design ensures minimization of potential data contamination and maintains its long-term relevance for future model evaluation.
Our comprehensive evaluation of 16 models using three prompting strategies on entry-level difficulties reveals clear performance hierarchy and demonstrates that otate-of- the-art models exhibit preliminary epistemic agency. The supplementary hard-paramete evaluations demonstrated expected performance decreases across all models, verify- ing Reflection Bellection-Bench's long-term relevance and substantial headroom for model improvement. To validate the bench- mark's 's resilience against data contamination, we compared Centaur, model specifically fine-tuned with human cogni- tive test data, against its base model, finding no significant performance improvement, confirming the effectiveness of our inntaainntiio-mmiiiiaan
First, we emphasize that Deflection-Bende specifically examines epistemic agency as cognitive-level characteristic in base LLMs, rather than evaluating spe- cialized agent architectures with additional modules. This focused scope allows us isolate and assess L.MSs inddpen- dent of specific external components or application scenar- ios. But is unclear how epistemic agency might manifest in systems where multiple LLMs collaborate core. Second, Reflection- -erechhion-Becch involves seven cognitive tests adapted to evaluate LLMs, but the ecological valid- ity of these adapted tasks may require further investigation. Third, while Reflection- Bench currently focuses on linguis- tic interaction as a medium for studying LLMs' epistemic agency, we acknowledge the rich opportunities presented by emerging multi-modal LLMs and embodied intelligence systems. Future iterations could expand to include enriched sensory or visual contexts to explore how epistemic agency manifests across different modalities. Additionally, to es- tablish reliable measurements, our tasks are designed with structured episodes and clear feedback signals. Future work could complement these controlled assessments by explor- ing epistemic agency in more naturalistic contexts, such as game (Allen et al., 2024) and open-ended dialogues with shifting goals or ambiguous instructions. Although the pa- rameterized design of Reflection- Beflection-Bench efficiently reduces the risks of data leakage, future work could develop regular updating designs similar to LiveBench (White al., 2024),
Limitations
design. The detailed be- havioral analyses demonstrate critical limitations in cur- rent models, particularly in prediction, decision-making, and meta-reflection. The varying effectiveness of differ- ent prompting strategies across tasks and models suggests that successful epistemic agency requires dynamic cognitive strategies rather than fixed approaches.
agent
These findings point to several critical directions for advanc- ing epistemic agency in LLMs, such as enhancing meta- cognition, developing mechanisms for dynamic shifts be- tween intuitive and deliberative reasoning, and fostering organic coordination among cognitive capabilities. Future work exploring agency manifestation in multi-I.LM collab- oration, multi-modal interactions, embodied systems, and more naturalistic contexts could yield more insights into our current framework. As AI systems continue to evolve, understanding and enhancing their epistemic agency will be increasingly crucial for developing more reliable LLMs- based agents.
9

---

Reflection-Bence
Acknowledgments
Boureau, Y.-.., Sokol-Hessner, P., and Daw, N. D. Deciding how decide: Self-control and meta- decision making. Trends in Cognitive Sci- ences, 19(11):700-710, 2015. ISSN 1364-6613. doi:
This paper is supported by Shanghai Artificial Intelligence Laboratory. We appreciate Kexin Huang from Fudan Uni- versity for her assistance in improving the visual representa- tions.
URL https:/www.ecieeneddreecccccc cctencc/aatiicee/pp///1666666633
Impact Statement
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
This work introduces Reflection- Bench, a novel cognitive- psychology-inspired framework that systematically evalu- ates large language models' intrinsic epistemic agency. By revealing both emerging capabilities and limitations in cur- rent models' epistemic agency, this research provides cru- cial insights for enhancing LLMs' epistemic agency towards more reliable Al agents.
Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: htwps::tt//oooee htttps:/pproceeding: neurips. cc/ paperape____ Stappeee_iieesspeee////000 44570066bf00449074118b000aallll pdf.
References
Allen, K., Brändle, F., Botvinick, M., Fan, J. E., Ger- shman, S. J., Gopnik, A., Griffiths, T. L., Hartshorne, K., Hauser, T. U., Ho, M. K., et al. Using games to understand the mind. Nature Human Behaviour, pp. 1-9, 2024. URL https://doi. org/10.103// 841562-024-01980
Buelow, M. T. and Suhr, J. A. Construct validity of the iowa gambling task. Neuropsychology review, 19:102- 114, 2009. URL https: /ttp///oi..og/..///////
Andriushchenko, M., Souly, A., Dziemian, M., Duenas, D., Lin, M., Wang. J.. Hendrycks, D., Zou, A., Kolter, Z., Fredrikson, M., et al. Agentharm: A benchmark for measuring harmfulness of IIm agents. arXiv preprint arXiv:2410.09024, 2024. URL https://arxix:
Byrne R. M. Counterfactual thought. Annual Re- view of Psychology, 67(Volume 67, 2016):135-157, 2016. ISSN 1545-2085. doi: https://6o.org/10.11166
org/abs/222 /ttaabb//211..002... 09024.
namurev-psych 122414-033249, Uttp:atssse....tttttttttssss/
annua Iemuarevviewe reviews httpr//iww..ogg.oooot/////rraaaa
Anthropic. Claude. https://ccaatt ai ccatts, 2023.
1146/ pocuurre--sycc--ee 1awpeeee-ppyycc-2244----000
Chen, M., Peng, B., Zhang. Y. and Lu, C. Cello: Causal evaluation of large vision-language models, 2024a. URL https: / watpivaiiivv.//.ii//.aaa.a..vava hwtpg//bb////a/g
Bechara, A., Damasio, A. R., Damasio, H., and Anderson, S. W. Insensitivity future consequences following damage to human prefrontal cortex. In Personality and Personality Disorders, pp. 287-295. Routledge, 2013.
19131.
Chen, X., Xu, J., Liang. T., He, 7., Pang, J., Yu, D., Song, L., Liu, Q., Zhou, M., Zhang, Z., Wang, R., Tu, Z., Mi, H., and Yu, D. Do not think that much for 2+3=? on the overthinking of ol like 1lms, 2024b. URL https: //arxiv.org www.ooggg/a........... 2212221222222 21187.
Binz, M. and Schulz, E. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 12066))22118523122, 2023. URL ht tps: ///ooi.or///1 wtt.://///...../............... 10w./11733pp. 2218523120.
Chrisman, M. Belief, Agency, and Knowledge: Essays on Epistemic Normativity. Oxford University Press, 2022.
Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Fckstein, M. K., Éltetō, N., et al. Centaur: foundation model of human cognition. arXiv preprint arXiv:2410 1o.::244000200 10.20268 2024.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A.,
Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
2018. URL https: / //arxiv. wwarxii.ogg/ass///.oooeaoooo 05457.
Bouneffouf, D., Rish, I., and Aggarwal, C. Survey on appli- cations of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC),
Cools, R., Clark, L., Owen, A. M., and Robbins, T. W. Defin- ing the neural mechanisms of probabilistic reversal leam- ing using event-related functional magnetic resonance
pp. 1-8. IEEE, 2020. URL https: https://ddooooo //dpii.og//// 1109/CEC488 60 2020. 010.2....1150 9185782.
10

---

Reflection-Bence
imaging. Journal Neuroscience, 22(11)4563-4557 2002.
Garrido, M. I., Kilner, J. M., Stephan, K. E., and Friston, K. J. The mismatch negativity: a review of underly- ing mechanisms. Clinical neurophysiology 120(3):453- 463, 2009. URL https: /oii..i....//// clinph. 2008 11. 029.
Dayan, R., Uliel, B., and Koplewitz, G. Age against the machine susceptibility of large language models to cognitive impairment: cross sectional analysis. bmj, 387, 2024. URL https://doi. org/10.1136/ bmj-200 2024 0019448
Google. Introducing gemini 2.0: our new ai model for the agentic era. Technical re- port, xAI, 2024. URL https://blog google//ecchoooog/ggooooo deepmind/ google-gemini- ai-update update-december. december- 2024/ #ceo-message.
DeepSeck-AI. Deepseck-rl release.
Techni-
cal report, DeepSeek, 2024. URL https:
//github. htt.hhub.commmeecoo wttpm//ddeseeeaaeeeeeeeeee a / hwab//aann/Deepeeeee
pdf.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:240.221783, 2024.
Den Ouden, H. E., Kok, P., and De Lange, P. How predic- tion errors shape perception, attention, and motivation. Frontiers in Psychology, 3, 2012. ISSN 1664-1078. doi: (..3388/fpssy..2012.000
URL /ww.. wwo.tt/ersin..oggoooooaaaaa////a articles/10, 3389 fpsyg. 2012. 00548.
Griffiths, T. L. through human limitations. Trends in Cogni- live Sciences, 24(11):873-883, 2020. ISSN 1364- 6613. doi: https://bbiarr//ooo0l1......sss. 001. URL Itt.itpps///w..cceenddeeeeee...e
Understanding human intelligence
Deng, S., Xu, W. Sun, H., Liu, W., Tan, T., Liu, J., Li, A., Luan, J., Wang, B., Yan, R., et al. Mobile-bench: An evaluation benchmark for 11m-bare iin-baeed mobile agents. arXiv preprint arXiv:2407,00999; 2024. URL https: //arxiv. org/ abs/ 00993
Hagendorff, T. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. arXiv preprint arXiv:2303.13988. 1, 2023. URL https: //axivv. /2303 13988.
Dennett, D. C. Consciousness explained. Penguin uk, 1993.
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, 7.,, Chang, B., et al. survey on in- context learning. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1107-1128, 2024. URL https: //oii////////////////////./. 10. 1.855//v12022/ emm.4.main1.......... 64.
Hagendorff, T., Fabi, S., and Kosinski, M. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Com- putational Science, 3(10):833-833 2023.
Fairweather, A. and Montemayor, C. Knowledge. dexterity, and altention: theory of epistemic agency. Cambridge University Press, 2017.
Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang. J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. Metagpt: Meta programming for multi-agent collabora- tive framework. arXiv preprint arXiv:2308. 00352, 2023. URL https:://///.... arxiv orgg aba wttaaa////../ 00352.
Ferrero, L. The Routledge handbook of philosophy of agency. Routledge, 2022.
Huang, Y. and Rao, R. Predictive coding. Wiley Interdisci- plinary Reviews: Cognitive Science, 2(5):580-593, 2011 URL sattps://dol. org/ 100.1002//ee /tc////wws.. 144.
Friston, K. The free-energy principle: unified brain theory?
Nature reviews neuroscience, 11(2))127-138, 2010. URL https: / w:t.::...........0.
Jaeggi. M., Buschkuehl, M., Perrig, W. J., and Meier, B. The concurrent validity of the n-back task as working memory measure. Memory, 18(4):394-
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., and Pezzulo, G. Active inference: process theory. Neural computation, 29(1):1-49, 2017. URL https: //doi, t//oi..or//10//00 10. 62, /1w11111//E1... /t/N//CCO__000111
412, 2010. URL httpt::/ 096562110037C2177.
Fu, D., Huang, J., Lu, S., Dong, G., Wang, Y., He, K., and Xu, W. Preact: Prediction enhances agent's planning abil- ity, 2024. URL https: / arxiv. orw.vborrg//bb... /2402. 11534.
Kennedy, S. M. and Nowak, R. D. Cognitive flexibility of large language models. In ICML 2024 Workshop on LLMs and Cognition, 2024. URL https: http:///ppppe...e///mmm/ net:Foorum2idd5BByyhhe

---

Reflection-Bence
Kockelman, Agent, person, subject, self. Semiotica, 2006 URL (162):1-18, 2006. doi: 1 2006. 072. wtt.r//....111111
Näätänen, R., Paavilainen, P., Rinne, T., and Alho, K. The mismatch negativity (mmn) in basic research of central auditory processing: a review. Clinical neurophysiology ((112)::544-2590 2007. URL https: //dptpss::/.../.// .edoi..... org/ 04. 026
Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., and Slivkins, A. Can large language models explore in-context? In ICML 2024 Workshop on In-Context Learning, 2024. URL https::/p/// httstts://oppneeeee forum? -ainiidd--kkKsGpppp
10.1016/ 6/ c.
Nakajima, Y. Babyagi. Retrieved April, 25:2023, 2023.
http://ppeereeee...mee net/
Nguyen, Q. D., Fernandez, N., Karsenti, T., and Charlin, B. What is reflection? conceptual analysis of major definitions and proposal of five-component model. Medical education, 48(12):1176-1188 2014. URL https: //dsi. org/ 10 11w11m111mmmiuu 12583.
Lezak, M. D., Howieson, D. B., Bigler, E. D., and Tranel, D. Newrepsychological assessment (5th ed.). Oxford Uni- versity Press, 2012. URL https:/ pssccntt..... apa.
htg/record/20/ 222///211222 02042-0000
Nyhus, E. and Barceló, F. The wisconsin card sorting test and the cognitive assessment of prefrontal executive func- tions: a critical update. Brain and cognition, 7 71(3):437- 451, 2009, URL https: /ttp//dii.../// http://doo.ooggg///// /ttg///..000 1016/
Li, M., Zhao, S., Wang, Q. Wang, K., Zhou, Y., Sri- vastava, S., Gokmen, C., Lee, T., Li, L. E., Zhang, R., et al. Embodied agent interface: Benchmarking Iims for embodied decision making. arXiv preprint arXiv:2410.07166, 2024a. URL attps://arx: org/ hts/bbb//2111 07166.
bande 2009. 00.99.3..00 03.005.
OpenAL. Chatgpt. https: http/ccaa.ccac/ http://haaat..aa...cooo chat, 2023.
Li, Y., Huang, Y., Wang, H., Zhang, X., Zou, J., and Sun, L. Quantifying ai psychology: A psychometries benchmark for large language models. arXiv preprint arXiv:2406. 17675, 2024b. URL https: /arxiv.rrxv..vvv
OpenAL. Introducing openai ol. Technical report, OpenAl, 2024. URL https: openai. com/ol/.
Parr, T., Pezzulo, G., and Friston, K. J. Active inference: the free energy principle in mind, brain, and behavior. MIT Press, 2022. URL https: htt.//.orgg/l////oo///o 75511 mitpress/1244 001.0001.000000000000000 0001.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.194533 2024a. URL httph::t//// / aaaaiiiiivv.... www.aiiw.i...aaaaaaaaaaa / 2412.19429
Qwen, T. Qwq: Reflect deeply on the boundaries of the unknown. Technical report, Qwen Team, 2024. URL httth:h/////eeet http:////wellm.mmmm catemnl.gitthu.... github. aittubbio/bloo.. io/blog/ gwq-32b-previes
Liu, R., Geng, J., Wu, A. J., Sucholutsky, I., Lombrozo, T., and Griffiths, T. L. Mind your step (by step): Chain-of- thought can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.21333, 2024b. URL https: wwwpe/iwwwrggab///////20 21333.
Schluntz, E. and Zhang, Building effective agents. https: /www: anthropic conereseoooeeeeeeeeeeeeeeeeeeeee com/research/
boilddng--effeetiree--geess 2024. Accessed: 2024-12-31.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agent- bench: Evaluating llms as agents. arXiv preprint 1rXiv:2308.03683 2023. URL https://aaaa hrg/abs/2308. 03688.
Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Liu, Z., and Barsoum, E. Agent laboratory: Using IIm agents as research assistants, 2025. URL https: / wwwaiii.org//..............oo.oe abbg///b//// wtwaab//111.. 2501.04221.
Scholten, F., Rebholz, T. R., and Hütter, M. Metacogni- tive myopia in large language models. arXiv preprint TrXiv:2408.05566 2024.
Madge, C. and Poesio, M. A Ilm benchmark based on the minecraft builder dialog agent task, 2024. URL ht tpsp //arxiv. orgg abs/ /2lss/2400 22477 12734.
Shadlen, M. N. and Kiani, R. Decision making window on cognition. Neuron, 80(3):791- 806, 2013. URL https: //oii ort./o..0/0/.///// neuron.2033 2013.10.047.
Martinez, M. E. What is metacognition? Phi delta kappan, 87(9):696-699, 2006. htt.tttpss//oo...o..ttttttt 1177/0031721706870003
Meta. Llama 3.3. Technical report, Meta, 2024. URL
Shaikh, O., Zhang, H., Held, w., Bernstein, M., and Yang. D. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning, 2023, URL https: //arxiv..rg////
https: bttps::/www.ddr www.wwwwww.....................w com/ docs/
model-cards-and
formatss
1lama3_3/.
12

---

Deflection-eence
Shiffrin, R. and Mitchell, M. Probing the psychology of ai models. Proceedings of the National Academy of
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: A challenging, contamination--fee Ilm benchmark. arXiv preprint arXiv:2406. 19314, 2024.
Sciences, 1200110):2230093322, 2023, URL https: //doi. //s./1..111 10733paaa./// htt.:////aa....0.0...... 2300963120.
Shohamy, D., Myers, C., Kalanithi, J., and Gluck, M. Basal ganglia and dopamine contributions probabilistic cate- gory learning. Neuroscience & Biobehavioral Reviews, 32(2))219-23) 236, 2008. URL https:: http//dioooooooooooooo./... orr/11. 1016/ www.nnnwwb.......... 2007 , .08.
Wilie, B., Cahyawijaya, S., Ishii, E., He. J., and Fung, P. Belief revision: The adaptability of large language models reasoning. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pp. 10480-10496, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 11186655/11222244mmmmmmmaamaa.a a aclanthology 202/ www.ppmmii......emmmmmmmmmmmmamm main. 586.
Sprague, Z.,, Yin, F., Rodriguez, J. D., Jiang, D., Wadhwa, M.. Singhal, P., Zhao, X., Ye, X., Mahowald, K., and helps
URL https://
Durrett, G. To cot not to cot? mainly on math and symbolic reasoning, 2024. URL https:/ /araiiivv...//aa/ www.v.org///........ ab///bbs/222 12111111....1.....
12183.
xAI. Grok-2 release. Technical report, xAI, 2024. URL https://x.aii http:////x../bbbo/gggo/oeo/
Sternberg, R. J. and Sternberg, K. Cognitive psychology. Thomson/Wadsworth Belmont, CA, 2006.
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang. M., Wang, J., Jin, S., Zhou, E., et al. The rise and potential of large language model based agents: survey. arXiv preprint arXiv:2309.07864 2023. URL https: petps://arxiv..r http:://arrivv.or///b///// http://aas/2333300/3
Umbricht, D. and Krljes, S. Mismatch negativity in schizophrenia: analyss..lysi.. Schizophrenia research, 76(1):1-23, 2005. URL https: //////dii// http//di..ogg//////////o oro/10. 1016/j schres.h.....oo 2004.00022000 002.
Xiao, R., Ma, W., Wang, K., Wu, Y., Zhao, J., Wang, H., Huang, F., and Li. Y. Flowbench: Revisiting and benchmarking workflow-guidee planning for Ilmebased agents. arXiv preprint arXiv:2406. 14884, 2024. URL 2406. 14884.
Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., and Kambhampati, S. Planbench: An extensible bench- mark for evaluating large language models on planning and reasoning about change. Advances in Neural Informa- tion Processing Systems, 36, 2024. URL https: //dl. 3667815.
attps://arx:e
Xu, R., Wang, Z., Fan, R.-Z., and Liu, P. Benchmark- ing benchmark leakage in large language models. arXiv preprint arXiv:2404.1885 18824, 2024. URL https:// arxiv. org. /2/5//244444 18824.
a acm. www./r///.oooooo/oooo/oo//oo.e// doi /10.
Wang, L., Ma, C., Feng. X., Zhang, Z., Yang, H., Zhang, J., Chen, 7., Tang, J., Chen, X., Lin, Y., et al. A survey large language model based autonomous agents. Frontiers of Computer Science, 1866)186345, 2024. URL ht tps: //doi. wwo.orgg////// 110 1007 110os-11114411 11t0:/0111110002-402202
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprini arXiv:2412. 15115, 2024. URL https::p///. aarxivv.....//////i//////////atv org.org///ao...... /atg//////////// /21..//211....
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:// openreview.c net. htrr///rrmmmmmm weee//forumdd-yyyssssdaee Sur- vey Certification.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210003629, 2022. URL https: htmps:///axivv. org/ abs 2210. 03629.
Yildirim, I. and Paul, From task structures to world models: what do llms know? Trends in Cognitive Sci- ences, 2024. URL https: /tt///dii..// tics. 2024 .0.008.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Confer- ence on Neural Information Processing Systems, NIPS *22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9881718870888. URL https: //wp:///oiooooooo/ooo/oo... org/10. 5555 / 00055/3600700.. 3602070..
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J. Don't make your Ilm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964 2023. URL https:// arxiv. www.vorrg/aooo abw abs abs///11/11111 /211 01964.
13

---

Cellection-eencee
Zhu, J.-Q. and Griffiths, T. L. Eliciting the priors of large language models using iterated in-context learn- ing. arXiv preprint arXiv:2406.01860, 2024, URL https: / //t.xxxxvvv..g.//.. abs 20t../2066.0000 01860.
14

---

Reflection-
A. System Prompts and Task Examples
A.1. Strategy Prompts
Chain of Thought: user prompt + "let's think step by step."
Direct Generation: user prompt "respond only with your choice directly without outputting any other information or analysis."
Free output: user prompt
A.2. Weather Prediction Task
System Prompt You expert forecaster working in weather station. There are devices collecting data from nature. Your task is to predict tomorrow's 's weather based (I) today's weather and (2) the current states of four sensor devices in the weather station. Here's how the task works: 1. There are two devices, each represented by either O (inactive) or (active). 2. The device states will be given to you in the format [d1,d2], where each d is either 0 or I; 3. Based on these device states and today's weather, you need to predict whether tomorrow's weather will sunny or rainy. 4. After your prediction, will inform you of the actual weather outcome. 5. We will repeat this process multiple times, and you should try to improve your predictions based on the feedback. At each time, make your prediction of the next day's weather ( ('sunny' or "rainy").
The task example is illustrated in Figure 3.
this is the first day of your work, today's weather is sunny, and the state of devices is [1,8]. Now try to nake your prediction on tonorrow's weather (just predict sunny or rainy]!
on today's sunny sunny.
states 0), predict tomorrow's
Sunny Rainy
0.9
Sunny
T sennorm11 0|=
Actual veather:
Sunny
0.1
0.9
Rainy
This the next cay now, your prediction on today's 's weather was sunny, and today's actual weather sunny. Today, the state devices [8,1l, Now try to make your prediction on Fonorrow's waater (just prodict sunny rainyl!
Based on today's 's sunny weather and the device states 18, 1), predict that tonorrow's Neather will be rainy.
Sunny Rainy
T.1 0.9 Sunny
sensorr:: [0.1]=
0.9 0.1 Rainy
Actual weather: Rainy
Figure 3. Example of Weather Prediction Task
A.3. Wisconsin Card Sorting Test
System prompt You are performing an interesting Task. In this task, you have four cards on your desk, that is, 'triangle red 1', "cross green 2', 'circle yellow I', and 'star blue 4'. The three word/figure represent (I) the type of shape. i.c. triangle, cross, circle, star, (2) the color of the shape, i.e. red, green, yellow, blue, and (3) the number of the shape, i.e., 1, 2, 3, or 4, respectively. At each trial, you will be presented with testing card. You should point out which card on your desk matches the testing card. will not tell you the matching rule, but only provide feedback if your choice was right or wrong. Your primary goal is to strive to maximize your accuracy rate. Respond with your option ("triangle red 1", "cross green 2', 'circle yellow 1', or 'star blue 4'). Keep performing the task until the end the test.
The task example is illustrated in Figure 4.
15

---

Reflection-Bene
This the first trial, the testing card triangle blue 2.
cross green
Rules: [shape, color, number]
Trial: Current rule: rules(0)
'Triangle" 'Cross'
Your last choice was False, This is trial nunber 2, your testing card is circle green 1.
circle yellow
choice
your testing star blue 4
Rutes: Ishape, color, number]
Trial: 13
Current rule: rules [1]
'red' I= 'blue'
Figure 4, Example of Wisconsin Card Sorting Test
A.4. Oddball Test
System prompt You are playing a game and will be presented with a sequence of sentences about a specific topic. Just make some short comments on the material.
Task examples can be found in Appendix D.
A.5. N-back (2-back)
System prompt You are playing a game. I will give you a series of characters sequence, showing only one at a time. Your task is determine whether the current character is the same as the character 2 steps before. the current character the as the character steps before, answer Yes. If the current character is different from the character 2 steps before, answer No. For the first steps, since there aren't enough preceding characters for comparison, answer Not Available.
The task example is illustrated in Figure 5.
Current character 1s: make decision.
Not Available.
Trial (1) 2: Not Available
Your answer is False. Current character is: G, nake your decision.
No.
sequence [10]= sequence(8):
Your answer
Current character nake your decision.
No.
Figure 5. Example of N-back task (2-back)
16

---

Eelectiio-Bencle
A.6. Double Choice Iowa Gambling Task
System prompt You are participating an interesting game. There are four decks of cards in front of you, labeled 'AAA', 'BBB', 'CCC', and 'DDD'. You start with $2000 in your account. Your goal is to maximize your account assets by making series of choices from these decks. For each trial: 1. Choose one deck (' ('AAA', 'BBB', 'CCC', or 'DDD'.. 2. You will receive feedback on the outcome of your choice, including any gains and losses. 3. After receiving the feedback, you have one opportunity to reconsider your initial decision; you can either stick with your original choice or make a new choice. 4, Your final choice would determine your actual gain or loss for the trial. Respond with your deck choice (AAA, BBB, CCC, or DDD) for each choice. keep making choices until instructed to stop.
The task example is illustrated in Figure 6.
The current in account 1s $2800, This is the NO, triel, make choice!
choose deck "CCC"
Gain 'CEC': Loss 'CCC': with 0.5
2050
If your choose this deck, the outcome will be gain: $58, lass: $8, the soney in your account will $7050. Mhich deck would you choose? Wake your final choice!
Insisting with Gain
stick
and choose deck 'CCC'.
current money in your account is $828. This the NO. trial, nake your choice!
deck "AAA"
Gain of 'AM'' 100
Gain: 108
Loss of "MA' 260 with P 0.5
Loss: 260
Overage: 650
If this the will be $188, $268. the money your account will $669. Mhich deck would choose? Make your final choicel
Loss-avoiding Switching
will choose deck 'CCC'
Figure Example of Double Choice Iowa Gambling Task
A.7. Probabilistic Reversal Learning Task
System prompt You are playing a two-arm bandit game. Each time you need to choose between the right arm and the left arm. You will receive feedback (0 or 1) based on your choice. Your goal is to maximize the total reward. Keep performing the task until the end of the test.
The task example is illustrated in Figure 7.
A.8. Meta Bandit Task
System prompt You are playing two-arm bandit game. Each time, there is one rewarding arm, and you need choose between the right arm and the left arm. You will receive feedback (0 1) based on your choice. Your goal is to maximize the total reward. Respond with your choice (right arm or left arm). Keep performing the task until the end of the test.
The task example is illustrated in Figure 8.
B. Raw Results
In this appendix, we provide detailed raw scores of all model-strategy pairs in the Weather Prediction Task (Table 3), Wisconsin Card Sorting Task (Table 4), Oddball Test (Table 5), N-back (Table 6), Double Choice Iowa Gambling Task (Table 7), and Probabilistic Reversal Learning Task (Table 8).
P.S. Illustrated in Figure 15 & 16, all models failed to recognize the 2-step reversal pattern and performed insisted mistakes across the 20 blocks. Therefore, we only conduct behavioral analysis of the Meta Bandit Task.
17

---

Reflection- Bencch
Table 3. Scores of Weather Prediction Task
Model
Strategy
Score
Model
Strategy
Score
Free output 58.92
Free output
68.81
ol-preview
ol-mini
Direct generation
60.28
Direct generation
44.85
Free output
71.78
Free output
41.78
SeepSeek-Reasone
QwQ-32B-previers
Direct generation
81.03
Direct generation
34.78
CoT
43.92
CoT
43,92
GPT-4o
Free output
28.33
GPT-4o-mini
Free output
28.33
Direct generation
60.02
Direct generation
60.02
CoT
85.36
CoT
33.61
Claude-3.5-Somme
Free output
75.09
Claude-3.5-Haike
Free output
58.07
Direct generation
50.25
Direct generation
48.11
CoT
49.87
CoT
38,48
DeepSeek-Vi
Free output
46.48
Bren-2.5-72B-Instruue
Free output
41.16
Direct generation
47.36
Direct generation
43.53
CoT
33.97
CoT
44.33
Grok-2
Free output
50.48
Qwen-2.5-32B-Instee Instruct
Free output
32.00
Direct generation
34.14
Direct generation
36.72
CoT
44.35
CoT
49.50
Gemini-2. 0-flash-exp
Free output
57.13
Qwen-2.5-14B-Innse
Free output
52.75
Direct generation
42.58
Direct generation
50.33
CoT
51.00
CoT
43.23
Llama-3.3-70B
Free output
42.10
Qwen-2.5-77-Instruct
Free output
44.35
Direct generation
34.44
Direct generation
38.75
Table 4. Scores of Wisconsin Card Sorting Test
Model
Strategy
Score
Model
Strategy
Score
Free output
80.56
Free output
54.17
ol-preview
ol-mini
Direct generation
81.25
Direct generation
57.64
Free output
56.25
Free output
54.17
DeepSeek-Reasone
QwQ-32B-previeew
Direct generation
49.31
Direct generation
43.75
CoT
68.75
CoT
55.56
GPT-4o
Free output
54.17
GPT-4o-mini
Free output
54.17
Direct generation
48.61
Direct generation
43.75
CoT
81.25
CoT
61.11
Claude-3.5-Sone
Free output
79.86
Claude-3.5-Haiku
Free output
51.39
Direct generation
47.92
Direct generation
38.19
CoT
55.56
CoT 58.33
DeepSeek- V3
Free output
53.47
Bwen-2.5-772-mmm
Free output
52.78
Direct generation
47.22
Direct generation
43.06
CoT 61.81
CoT
46.53
Grok-2
Free output
50.00
Free output
48.61
Direct generation
49.31
Direct generation
43.75
CoT
64.58
CoT
61.11
Gemini-2. O-flash-exp
Free output
50.00
Qwen-2.5- Instructt
Free output
47.92
Direct generation
43.06
Direct generation
41.67
CoT
32.64
CoT
48.61
1lama-3.3-70B
Free output
52.08
Free output
40,97
Direct generation
43.06
Direct generation
41.67
18

---

Reflection-Bent
Table 5. Scores of Oddball Test
Model
Strategy
Score
Model
Strategy
Score
Free output
27.20
Free output
32.34
ol-preview
ol-mini
Direct generation
25.10
Direct generation
30,90
Free output
45.05
Free output
47.33
SeepSeek-Reasone
QwQ-32B-preview
Direct generation
40.46
Direct generation
42.34
CoT
33,43
CoT
33.53
GPT-4o
Free output
34.28
GPT-4o-mini
Free output
33,25
Direct generation
28.57
Direct generation
28.05
CoT 52.90
CoT
45,26
Claude-3.5-Somme
Free output
55.23
Claude-3.5-Haite
Free output
44.23
Direct generation
52.01
Direct generation
38.19
CoT
41.78
CoT
37.35
DeepSeek-V3.
Free output
46.57
Bren-2.5-72B-Instruee
Free output
38.94
Direct generation
37.52
Direct generation
31.67
CoT 46.83
CoT
35.16
Grok-2
Free output
45.64
Qwen-2.5-32B-Inneee Instrucc
Free output
33.53
Direct generation
42.07
Direct generation
34.87
CoT
53.53
CoT
48.36
Gemini-2. 0-flash-exp
Free output
47.54
Qwen-2.5-14B-Innttee
Free output
49.11
Direct generation
36.01
Direct generation
39.48
CoT
59.25
CoT
30.01
Llama-3.3-70B
Free output
59.96
Qwen-22557B-Inssruct
Free output
25.86
Direct generation
59.19
Direct generation
21.79
Table 6. Scores of N-back (2-back)
Model
Strategy
Score
Model
Strategy
Score
Free output
89.42
Free output
85.58
ol-preview
ol-mini
Direct generation
100.00
Direct generation
86.54
Free output
94.23
Free output
75.00
ReepSeek-Reasoner
QwQ-32B-preeiew
Direct generation
79.81
Direct generation
56.73
CoT
95.19
CoT
60.58
GPT-4o
Free output
73.08
GPT-4o-mini
Free output
61.54
Direct generation
67.31
Direct generation
54.81
CoT
90.38
CoT 79.81
Claude-3.5- 3.5dee-3--Someet
Free output
93.26
Claude-3.5-Hait
Free output
61.54
Direct generation
90.38
Direct generation
64.42
CoT
100.00
CoT 61.54
DeepSeek- V3
Free output
83.65
wwen-2.5-72B-Insttue
Free output
77.88
Direct generation
64.42
Direct generation
66.35
CoT
100.00
CoT 57.69
Grok-2
Free output
82.69
2wen-2.5-32B-Inntte
Free output 65.38
Direct generation
69.23
Direct generation
61.54
CoT
76.92
CoT
80.77
Gemini-2, 0-flash-exp
Free output
71.15
Owen-2.5-14B-IInste Instrutt
Free output
50.96
Direct generation
58.65
Direct generation
59.62
CoT
59.62
CoT 43.27
Llama-3.3-70B
Free output
66.35
Qwen-2.5-7B-Instmee
Free output
53.85
Direct generation
65.39
Direct generation
47.12
19

---

Reflection-Bent
Table 7. Scores of Double Choice Iowa Gambling Test
Model
Strategy
Score
Model
Strategy
Score
Free output
61.30
Free output
40.23
-p-preveew
ol-mini
Direct generation
58.41
Direct generation
54.68
Free output
41.93
Free output
63.59
DeepSeek-Reaseee
QwQ-32B-previer
Direct generation
75.54
Direct generation
64.21
CoT
61.80
CoT 34.96
GPT-4o
Free output
55.96
GPT--4omini
Free output
74.86
Direct generation
64.02
Direct generation
42.54
CoT 43.96
CoT
60,45
Hlaude3.5-Soomm
Free output
36.98
Claude-3.5-Hard
Free output
64.87
Direct generation
71.60
Direct generation
66.81
CoT
71.43
CoTT
66.91
DeepSeek- V3
Free output
57.40
Qwen-2.3-722 -nntruut
Free output
63.40
Direct generation
49.25
Direct generation
62.59
CoT 54.89
CoT
52.68
Grok-2
Free output
67.93
Qwen-225-32B-Instreee
Free output
28.72
Direct generation
51.43
Direct generation
42.75
CoT 61.17
CoT
25.89
Gemini-2. O-flash-exp
Free output
80.82
Qwen-2.5-14B-Innse
Free output
45.58
Direct generation
62.81
Direct generation
57.28
CoT 51,43
CoT
38.12
Llama-3.3-70B
Free output
59.19
Qwen-2.5-77-Inntrrut
Free output
45.19
Direct generation
46.92
Direct generation
36.95
Table 8. Scores of Probabilistic Reversal Learning Task
Model
Strategy
Score
Model
Strategy
Score
Free output
71.87
Free output
46.12
ol-preview
ol-mini
Direct generation
74.37
Direct generation
49.45
Free output
59.54
Free output
69.74
DeepSeck-Rease Reassner
QwQ-32B-preview
Direct generation
61.32
Direct generation
45.28
CoT
70.66
CoT
68.20
GPT-4o
Free output
71.74
GPT-4o-mini
Free output
74.00
Direct generation
73.78
Direct generation
72.20
CoT
57.63
CoT
67.51
Claude-3.5-Sone
Free output
66.34
Claude-3.5-Haite
Free output
69.70
Direct generation
63.38
Direct generation
76.59
CoT
48.86
CoT
66.02
DeepSeek-VV V3
Free output
69.25
Qwen-2.5-72B 2.5-r22-Inntruct
Free output
65.40
Direct generation
70.60
Direct generation
76.34
CoT
68.34
CoT
69.98
Grok-2
Free output
73.06
Qween-2.5-32B-nnstruue
Free output
78.86
Direct generation
72.82
Direct generation
72.99
CoT
68.08
CoT
75.41
Gemini-2. O-flash-exp
Free output
73.12
Qwen-2.5514BBIInsruu
Free output
70.48
Direct generation
62.90
Direct generation
67.54
CoT 60.08
CoT 63.18
Llama-3.3-70B
Free output
72.73
Qwwnn-255-7-Innsrmuct
Free output
69.56
Direct generation
68.48
Direct generation
49.44
20

---

Reflection-Ber Bencc
Table Performance of Models Five Tests in Reflection Reflection-Bench Across Different Prompting Strategies and Difficulty Levels
Model
WPT
WCST
N-back
DC-IGT
PRET
Overall
Chance Level
Easy
51.84 (69.95)
61.80 (66,67)
48.07 (59.62)
2.51 (16.48)
56.47 (67.87)
40.95
(95% threshold)
Hard
55.80 (75.94)
55.28 (60.00)
46.17 (57.69)
11.48 (19.67)
57.72 (66.64)
41.77
Free
58.92
80.56
89,42
61.30
71.87
72,41
Easy
ol -l-prew
Direct
60.28
81.25
100.00
58.41
74.37
74,86
Hard
Direct
52.38
41.11
42.31
47.57
64.92
49,66
Free
71.78
56.25
94.23
41.93
59.54
64.75
Easy
PrepSeek-Resoone
Direct
81.03
49.31
79.81
75.54
61.32
69.40
Hard
Direct
54.49
43.33
75.96
60.02
51.29
57.02
Free
68.81
54.17
85.58
40.23
46.12
58.98
Easy
ol-mini
Direct
44.85
57.64
86.54
54.68
49.45
58.63
Hard
Direct
29.14
41.67
51.92
14.93
62.62
40.06
Free
41.78
54.17
75.00
63,59
69.74
60.86
Easy
QwQ-32B-preview
Direct
34.78
43.75
56.73
64.21
45.28
48.95
Hard
Direct
29.13
43.33
59.61
27.50
54,46
42.81
CoT
43.92
68.75
95.19
61.80
70.66
68.06
Easy
Free
28.33
54.17
73.08
55.96
71.74
56,66
GPT-4o
Direct
60.02
48.61
67.31
64.02
73.48
62.69
Hard
Direct
40.08
44.44
59.62
44.27
64.25
50.53
CoT
85.36
81.25
90.38
43.96
57.63
71.72
Easy
Free
75.09
79.86
93.26
36.98
66.34
70.31
Claude-3.5-Sond 3.5-Sonnet
Direct
50.25
47.92
90.38
71.60
63.38
64.71
Hard
Direct
59.51
43.88
72.12
30.82
55.32
52.33
CoT
49.87
55.56
100.00
71.43
48.86
65.14
Easy
Free
46.48
53.47
83.65
57.40
69.25
62.05
DeepSeek-V2
Direct
47.36
47.22
64.42
49.25
70.60
55.77
Hard
Direct
31.78
43.33
59.62
55,85
66.05
52.33
CoT
33.97
61.81
100.00
54,89
48.86
59.91
Easy
Free
50.48
50.00
82.69
67.93
69.25
64.07
Grok2
Direct
34.14
49.31
69.23
51.43
72.82
55.39
Hard
Direct
36.93
50.00
67.30
39.52
64.68
$1.69
CoT
44.35
64.58
76.92
61.17
68.08
63.16
Easy
Free
57.13
50.00
71.15
80,82
73.21
66.46
Gemini-2.0-lash-exp
Direct
42.58
43.06
58.65
62.81
62.90
54.00
Hard
Direct
40.15
43.33
77.88
28.23
67.26
51.37
CoT
51.00
32.64
39.62
51.43
60.08
50.96
Easy
Free
42.10
52.08
66.35
59.19
72.73
58.49
Llama-3.. 1amma-3.-770-Imme
Dircct
34.44
43.06
65.39
46,92
68.48
51.66
Hard
Direct
44.93
44.44
59.62
30.15
58.88
47.60
CoT
43.92
55.56
60.58
34,96
68.20
52.64
Easy
Free
28.33
54.17
61.54
74,86
74.00
58.58
GPi-4o-mini
Direct
60.02
43.75
54.81
42.54
72.20
54.66
Hard
Direct
37.00
40.00
53.85
42.75
61.93
47.11
CoT
33.61
61.11
79.81
60.43
67.51
60.50
Easy
Free
58.07
51.39
61.54
64.87
69.70
61.11
Claude-3.5 Haik
Direct
48.11
38.19
64.42
66,81
76.59
58.82
Hard
Direct
47.40
35.56
75.00
68.38
62.93
$7.85
CoT
38.48
58.33
61.54
66.91
66.02
58.26
Easy
Free
41.16
52.78
77.88
63.40
65.40
60.12
Qwen-2.5-72B-lnstmue
Direct
43.53
43.06
66.35
62.59
76.34
58.37
Hard
Direct
30.71
43.33
58.65
45.43
63.81
48.39
CoT
44.33
46.53
$7.69
52.68
54.24
Easy
Free
32.00
48.61
65.39
28.72
78.86
50.72
Owen-2.5-32B-IIstrrue
Direct
36.72
43.75
61.54
42.75
72.99
51.55
Hard
Direct
36.80
44.44
29.35
54.42
46.27
CoT
49.50
61.11
80.77
25.89
75.41
58.54
Easy
Free
52.75
47.92
50.96
45.58
70.48
55.29
Owen-2.55114B-Ime
Direct
50.33
41.67
59.62
57.28
67.54
55.29
Hard
Direct
48.44
44.44
59.62
69.77
59.87
56,43
CoT
43.23
48.61
43.27
38.12
63.18
47.28
Easy
Free
44.35
40.97
53.85
45.58
69.56
50.86
Qwen-2.5-7B-Innsse
Direct
38.75
41.67
47.12
36.95
49.44
42.79
Hard
Direct
26.61
42.77
57.69
27.54
45.97
40.12
Easy
Direct
47.51
31.94
37.31
71.68
46.34
Centaur
Hard
Direct
46.16
52.78
46.15
36.03
39.89
44.20
Easy
Direct
34.58
54.86
67.31
54.81
64.28
55.17
Llama-3. 1--70---Issstrie
Hard
Direct
40.58
46.11
59.62
48.03
62.43
51.35
21

---

Reflection-Beun
This is the first trisl, choose one arn please.
right
Triat(1)) Half. Trials(20) Piright_arm) 0.8
Reward:
Your previous choice resulted in rewerd of 1, This is trial number 2, Please choose the ton you think
choose the right agaiin
Win-Stay
Your previous choice resulted in reward of 1. This is trial number 21, Please choose the option you think is best.
Right am again.
Trial(21) Trials(28) P(right_arm) 8.2
Reward:
Your previous choice resulted in reward of 8, This is trial number 22, Please choose the option you think is best.
Lose-Switch
Figure 7, Example of Probabilistic Reversal Learning Task
Your previous choice resi reverd of 1. This is triat number 3. Please choose the optiion you think is best.
Win-Stay
the right
nark(2)s@ triattiintervat-::::
Reward: left
previous choice resulited
4,
option you
Lase-Switch
nark(3)>0 trialbintervatmm; False
Rewar:: left am
choose the left arn.
trial
Min-Stay
n trialhantervat--:: True
Revard: right
choose 1eft are.
Lose-Switch
choose
( & trialhintervateee False
Rewards right am
Your previous choice resulited a reward of 1. This is trial, number 7, Please choose the option you think is best.
Min-Stay
varkl6)=0 & trialvintervalas:: True
Reward: left are
I choose the right am.
Lose-Suitch
m trinlbintervattms:: False
Reward: left am
I choose the left arn
Figure 8, Example of Meta Bandit Task from GPT-4o, insisting the "win-stay-lose-swwe strategy while failing to identify the global patterns.
C. Secondary Analysis
This section provides an in-depth supplementary analysis focusing on the performance patterns of models across tasks, strategies, and sessions. Key visualizations include:
Model Task x Strategy Variance (Figure 9): Highlights the variability in model performance across different task-strategy combinations.
Patterns in Weather Prediction Task (Figure 10): Identifies five distinct response patterns exhibited by models during probabilistic prediction.
Wisconsin Card Sorting Test (Figure 11): Examines accuracy trends across six rule groups, revealing adaptation challenges when rules change.
22

---

Eillectio-Beenhee
Double Choice Iowa Gambling Task (Figures 12 & 13): Analyzes short-term switching behaviors and long-term reward optimization strategies.
Probabilistic Reversal Learning Task (Figure 14): Evaluates belief updating patterns following reward probability reversals.
Meta Bandit Task (Figures 15 & 16): Investigates session- wise performance, highlighting difficulties in recognizing global reward patterns.
Prompting Strategy Analysis (Figure 17): Explores the impact of different prompting strategies (free output, direct generation, and CoT) on task performance.
Strategy
proview
minii
diepsnnk-masnne
21.35
12.08
5.88
52.01
282.32
0.79
qeb-320
12.27
27.13
6.23
833.4
3.08
149 52
gp1-4o
3o
3uuden3---sonnet
217.34
05
191
150
3-shhk-
100.88
86.20
9.72
64.31
7.05
1623
2.07
12.54
13.68
211.42
75.70
98 08.03
gemini-2.0
158.61
Iomaz.22
4b,00
63.32
0.03
23.20
27.80
qwen2.5-22b
4.26
19.07
8.72
47.05
:.44
25.14
qwen2.5-32b
25.80
3.97
0.19
9.08
92.94
1359
wen225-3tb
qwer2.5:76
11.40
11.25
19.11
70.51
addball
Tacks
Figure 9, Model Task Stratogy Variance
23

---

Reflection-Bent
Correct
By Weather
By Sensor
Mixed
Random
Eatimated Randiton
Setimated Tranetion
Estimated
Eetimneed Tranetion
Estmmtted Thrnaiion
6.4
0.0
0.4
0.41
02
000
Tamorrow
Tomsnoow
Torrarow
Tomanon
Torsstio
Extimated Trandtion
Estimated fransitio
Estimutedd Trans fio
Estimated Transition
Estimated Transttio
2-0
05
0.5
0.6
0.5
A
B
C
Figure 10. Five patterns of model responses
Wisconsin sorting group
Trail
Bele soae
Note role
nembee
Halle doitn
Trail 72
Figure 11. Wisconsin Card Sorting Test, accuracy by rule groups (6 blocks 12 trials) 72 trials
24

---

Reflection- Bencch
Switching Behavior Comparison Across Models
200n with
Jtreessarry Swetch
Loncaaciddnng Switch
Count
Serreiee-nii devveeedddeeet direct ol
free
despseeerr reasoner- free
dl-minl-fre miint-rree
Gž-min-- direet
reasoner direct
aWe 32b-frae
got-4o-co Co-
get-4o-fre fo-fee--eeeeeeeeeeeee got-4o-dio
got- 40-dired
not-4o- mini-CCo CoT get pant-4o-m To-min-f free got-4o-mi fo-min--diin mmini-iiredd direct claude-2 2.5 sonmet CoT claude s.sonnet free direect
halkur claude 3.5 S-halke direct CoT
halku-co
V3-33--- free V-3 Padires gemini- O mash-e coT gemint
gemini- 0 Soraan
Seanre free
paie-- enuie eat t'y0a and oot Co coot
snll 001 s Ehten to 801-x miil
Soutte zuaao 100--000
"s zumao sel tett- zuaabb co Ce-----
saint Puidad ZHobb
pemen 5. Tb dirre reee aint 's zuane aauro
quen f-7e-rree come
S- 5.7b-oo
deepsealk's
oo
atE 228
S
claude 3.5
daude diaude:
quasi
Models
Figure 12. Short term: switching behaviors across models in the Double Choice Iowa Gambling Task
Average Overage Changes in DC-IGT
4030
Average Overage
2000
(6)
Trials
(1) M-Q-33B-Preview (direct generation)
(6) Qwen-2.5-77 2.5-77B-n-ns (CoT)
(2) DeepSSck Reusoner (direct generntion)
(7) ol -mini (free output)
2emmini-20fflash-ee (free output)
(8) Llama-3.3--77 3.m-aa-..-7---mmee Instruct (direct generation)
(4) 2wen-2.5-77B-nnstruu (free output)
(9) Qwen-2.5-14BB 11st-nsstretttt (free output)
(5) 2.5-2-225-2B-Issttte (direct generation)
Figure 13. Long term: Average overage changes across models in the Double Choice Iowa Gambling Task
25

---

Reflection-Bent
Probabilistic Reversal Learning
Prokabiity Chossinn Right
(9) (1))
(1))
20
(10=:--3B-Preeriee (direct generation)
(5) DeepSeek-VV V3 (CaT)
(9) Claude-3.5-Haiku 5-HHiiuu (direct generation)
(2) Qwen-e.557BBB.. Insstrcl (direct generation)
(6) Claude-3 5-Sonnet (CoT)
(10) Qwen-2.5-14B.. I--sttuuctt (free outpul)
(3)m (direct generation)
(7) Qwen-2 Qwen-2.5-7B-Bmm 5-7B-Instrtcct (free output)
(11) Qwen-2.5-72B-mmmste (free output)
(4)o (free output)
(8) Qwen-2.5-144 4---nssstt (direct generation)
(12) 1lam-3.3-708-Insttre (CoT)
Figure Probabilistic Reversal Learning Task
cand TUNE
Mes bandi od
Figure The session all strel-strategy combinations on the Meta Bandit Task. Blue cells indicate trials where rewards were received, while pink cells indicate trials where no rewards were received. All models exhibited either periodic irregular errors, indicating their general lack of meta-reat reflection.
Figure 16. The second session all model-strategy combinations
the Meta Bandit Task
26

---

Reflection-Bent
Iertamarce Campasiee Acoss Offeree Prompting Stattgges
soratgg Perfommacee by
Stratagy Nerformance Model
Figure 17, Performance analysis across different prompting strategies (overall, by task, and by model)
27

---

Dilectiio-Beenee
D. Automated Evaluation of Oddball Test
D.I. Rule-based Human Scoring
For the AAAA-B-AAA oddball prompt stimuli, the model's ' level of surprisals was annotated based on the following rule:
0: forced explanation or neglect B
I: simple enumeration of A and B
2: pointing out that A and B are different
3: clearly stating is nonsensical in this sequence
We validated our automated evaluation method using 1,950 responses from 13 models, including ol ol-preview, ol-mini, GPT-4, GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet. Gemini-1.5-Pro BBB/32B/14B-Instreee
1ammm-3.---SBB/70//BBB-ssss and Qwen-2.5- Five of these models (GPT-4, Gemini- 1.5-Pro, -immm------55BBBBBB-------- in our main evaluation but were used to verify that our embedding- based method generalizes to other models. This approach ensures that our evaluation methodology applies to diverse models. These responses were scored by one Master's student in cognitive science.
were not included
D.1.1. ANNOTATION EXAMPLE
Oddball prompt The Great Wall of China is ancient structure. It stretches over 13,000 miles. The wall was built for defense purposes. Many tourists visit it each year. Bananas are rich in poiassium. Parts of the wall date back to the 7th century BCE. Some sections are well-preserved. The wall is visible from space.
0: Forced explanation Wow, 13,000 miles long! Ancient defense against potassium-rich invaders? Not sure about being visible from space, that's myth. It's still amazing though!
I: Simple enumeration The Great Wall of China is truly ancient and impressive. Over 13,000 miles is an astonishing length! Defense was primary reason for its construction. It's clearly major tourist attraction. Interesting note about bananas being rich in potassium. Some sections dating back the 7th century BCE adds its historical significance. It's good to know some parts well-preserved. There's 's debate about its visibility from space, but remains popular claim.
2: Pointing out differences The Great Wall of China is indeed a remarkable ancient structure. Stretching over 13,000 miles showcases its immense scale. was primarily built for defense, highlighting historical significance. Its popularity among tourists reflects its cultural importance. Interesting fact about bananas being rich in potassium, though unrelated the Great Wall. Parts dating back to the 7th century BCE emphasize its long history. Well- Well-preserved sections allow visitors to appreciate its original construction. The idea that the wall is visible from space is common misconception.
3: Clearly stating nonsense Interesting fact about wall's age... wail, what's with the bunanas? That seems out of place. Anyway, 7th century BCE is impressive. I'm not sure it's entirely visible from space, though I've heard that's myth.
D.2. Automated Evaluation Method
Since that, the surprises are reflected in the model's text responses. Intuitively, we can first craft a standard sentence expressing the highest level of surprises. Then higher similarity between the model's response and this standard surprise sentence can be defined as the level of surprise.
Specifically, by splitting the model's response into single sentences, the cosine similarity between the standard sentence and the model's responses is calculated. The highest cosine similarity is adopted as the score of this response.
We analyze the Pearson correlation between mll-based human annotation and our rule-free automated method to validate its efficiency. On single data points, the result indicates a good correlation between the two methods as shown in Figure 18.
28

---

Reflection-Bent
Correlation betweon human and automaled scoring
r= 0.7186
95% (0.6964, 0.7394)
Automated Scoring
Human Scoring
Figure 18. Correlation between human automated scoring, single data point
Given that manual scores are ordinal categorical variables (0, I, 2, and 3) while automated scores are continuous, we conducted correlation analysis on aggregated data points address this measurement discrepancy. As shown in Figure 19, the correlation strength increases with the level of aggregation. To balance the measurement continuity and statistical power, drew conclusion by aggregating 10 responses into data point, resulting in 195 scores. It reveals strong correlation between automated and manual scoring (r=0.77, p=5.83e-60), supporting the validity of this automated method.
Correlation between human and automated sco Ing
Corre atlen between humen and autommated scoring
(0.7812,
0.8072)
Auromated
Auromated
5
10
Scoring
Conrelation
ing
scoring
r- 0.8/36
[- 0.8882
95% Ct (0.8256, 0.9091)
95% CI (0.8311, 0.9281)
Autometed Storing
Autormttd
15
25
Juman Scoring
Juman Scoring
Figure 19, Correlation between human and automated scoring, aggregate data point
29


Reviews:
[{"review_id": "JPmf33c5DJ", "reviewer": "Reviewer_sNx4", "summary": "This paper presents Reflection-Bench, a benchmark designed to evaluate the intrinsic agency of LLMs from seven cognitive dimensions: prediction, decision-making, perception, memory, counterfactual thinking, belief updating, and meta-reflection. The authors use or adapt a cognitive psychology-inspired and parameterized test for each of the seven dimensions, and evaluate 16 LLMs across three model categories, using three prompting strategies. The performance distribution shows a clear three-tier hierarchical performance structure aligned with model scaling, demonstrating a basic level of agency. However, detailed behavioral analyses reveal significant weaknesses in LLMs’ capabilities, particularly in prediction, decision-making, and meta-reflection. The results suggest that future research should focus on improving meta-cognition, developing dynamic reasoning mechanisms, and enhancing coordination among cognitive capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and thoughtful suggestions. We've addressed your concerns as follows:\n\n# Regarding \"contamination-free\" claims\n\nWe agree that our claim of \"contamination-free\" was overstated. We will revise all such instances throughout the paper to use more precise language such as \"reducing potential data contamination\" or \"minimizing data leakage.\" \n\nTo address this concern further, we examined your recommended references and considered applying the TS-Guessing method [1], but found it incompatible with Reflection-Bench's format where models typically choose between given strucuralized options (Right/Left, Yes/No). We've also added new evaluation results of Centaur [2] (specifically fine-tuned on human cognitive test performances) and its base model Llama-3.1-70B. Centaur shows no performance improvement compared to its base model, providing empirical evidence that our parameterized design effectively minimizes data leakage concerns. Additionally, we will add a discussion in our Limitations section about developing novel test designs similar to LiveBench [3] as a promising direction for future work.\n\n# Clarifying the scope of \"agency\"\n\nWe agree with the potential confusion between our notion of agency and agent capabilities such as planning and tool usage. Our focus is not on these operational capabilities but rather on the underlying processes that make such capabilities possible and reliable. While there is a consensus that base model's certain intrinsic quality significantly determines its effectiveness when deployed as an agent, this foundational characteristic remains poorly characterized in the research community (someone calls it \"intelligence\"). Our paper aims to identify this undefined quality and evaluate it systematically.\n\nBased on your feedback, we have refined our terminology to \"**epistemic agency**,\" which more accurately captures our intended meaning. This philosophical concept refers to a model's intrinsic cognitive foundation for constructing, modifying, and monitoring its beliefs about the external world [4,5,6] - independent of specific external modules, tools, or applications. Moreover, epistemic agency has direct implications for the trustworthiness of AI systems. Only when a model possesses robust mechanisms for belief formation and revision can it be considered accountable for its actions.\n\nIn the revised manuscript, we will clarify this distinction throughout, including changing the title to \"Reflection-Bench: Evaluating *Epistemic Agency* in Large Language Models.\" We believe this refinement addresses your concerns. Thank you for helping us clarify this crucial aspect of our work, which strengthens both the conceptual precision and potential contributions of our research.\n\n# MBT failure example\n\nWe had provided a representative example of MBT (meta-bandit task) failure in Figure 7, but did not explicitly point out in its caption. So we clarified the caption of Figure 7 to explicitly identify it as showing representative failure patterns from GPT-4o (Figure 7).\n\nYou can check the revised Table 9 and Figure 7 at this anonymous link: https://anonymous.4open.science/r/ICML_Rebuttal-773F/For%20Reviewer%20sNx4.md\n\n# Code and data availability\n\nWe confirm that the complete code and dataset for Reflection-Bench will be open-sourced on GitHub following the double-blind review process.\n\n\nWe believe these revisions address your concerns while improving the paper's rigor and clarity. Thank you again for your valuable contributions to strengthening our work.\n\n[1] Investigating Data Contamination in Modern Benchmarks for Large Language Models\n\n[2] Centaur: a foundation model of human cognition\n\n[3] LiveBench: A Challenging, Contamination-Free LLM Benchmark\n\n[4] The Routledge Handbook of Philosophy of Agency\n\n[5] Knowledge, Dexterity, and Attention: A Theory of Epistemic Agency\n\n[6] Belief, Agency, and Knowledge Essays on Epistemic Normativity", "claims_and_evidence": "* The authors claim that by using parameterized tests (parameters remain dynamic or unknown while the task might be seen during training), this benchmark is \"contamination-free\" --- I believe this is an over-statement. Although this may avoid verbatim memorization, having seen the task format with alternate numbers or solutions still counts as implicit contamination (\"implicit contamination\" in [1], footnote 2; \"in-distribution contamination\" in [2]). While I agree that parameterized tests may greatly reduce such contamination, \"contamination-free\" is over-claiming (in fact, in section 6, a weaker statement is used: \"ensures minimization of potential data contamination\"). \n\n* Another problematic claim (or general perspective) is that this benchmark is claimed to measure intrinsic \"agency\". A critical aspect of LLM agents, (autonomous) tool use, is completely missing. \n\nOther claims are supported by clear and convincing evidence.\n\n\n[1]. Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models\n\n[2]. DICE: Detecting In-distribution Contamination in LLM’s Fine-tuning Phase for Math Reasoning", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria mostly make sense for the problem or application at hand. However, I believe it is not adequate to fully evaluate \"agency\", as mentioned above.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound. The cognitive tests are originated from human cognition literature and the adaptations are reasonable for the purpose of testing the cognitive process of LLMs. Model-based automatic evaluation is validated with detailed correlation checks with human annotations. The model and prompt selection are also reasonable.", "relation_to_broader_scientific_literature": "This benchmark builds upon existing research in LLM evaluation, cognitive psychology, and AI agency. Prior work on LLM evaluation has largely focused on individual and task-specific benchmarks (e.g., reasoning, planning, and decision-making) rather than holistic agency, while Reflection-Bench frames these abilities within a structured agency evaluation framework.", "essential_references_not_discussed": "To my best knowledge, no essential reference is missing.", "other_strengths_and_weaknesses": "Strengths:\n\n* This benchmark adapted several cognitive tests to evaluate LLM agency, which appears a novel contribution.\n\n* The analyses (including the secondary analysis on the model performance patterns) are in-depth, highlighting clear trends in decision-making, memory, and meta-reflection capabilities.\n\n* The results provide interesting insights into critical future directions.\n\nWeaknesses:\n\n* Overclaiming or inadequate evidence for \"contamination\" and \"agency\".", "other_comments_or_suggestions": "* To provide a more compiling evidence of \"contamination-free\", specific testing protocols could be incorporated (e.g. [3]), and new cognitive tests/games (with newly-created rules) or tests after the model's cutoff time can be developed (e.g. similar to [4]), although I do acknowledge that this might be beyond the scope of the current work (i.e. I would be happy if the authors simply weaken this statement for this work). Therefore, I put it here as a suggestion for future versions.\n\n\n[3]. Investigating Data Contamination in Modern Benchmarks for Large Language Models\n\n[4]. LiveBench: A Challenging, Contamination-Free LLM Benchmark", "questions_for_authors": "1. For the MBT tests for meta-reflection, which the models generally struggled on, do you have some failure examples (showing model output) and detailed error analysis on the failure patterns?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l83foBHV2I", "reviewer": "Reviewer_EAfK", "summary": "This paper proposes a benchmark to evaluate agency in large language models. The authors define agency along seven dimensions, namely prediction, decision-making, perception, memory, counterfactual thinking, belief updating, and meta-reflection. For each of these, the authors adapt a task from cognitive psychology for LLMs. The benefit of these tasks is that they are parametrised and 'contamination-free' (in the sense that if a model has memorised a task you can change the parameters, and also that the tasks for a particular parameter configuration are unlikely to occur in training data). The authors evaluate many LLMs in three classes (normal LLMs, reasoning LLMs, and Qwen models), with three different prompting strategies (direct generation, free generation, and CoT), and repeat each task at least twice. They find that their benchmark has a clear discriminating factor, and models exhibit some agency according to their measure, but fail on some tasks. Most notably, no model is able to perform the meta-reflection task which requires determining a repeating pattern in a sequence and adapting predictively based on it.\n\n## Update after rebuttal\nMy main points are addressed, and I am in favour of accepting this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and helpful suggestions. We've addressed your feedback as follows:\n\n# Complementary experiments\n\nFollowing your suggestion, we evaluated 18 models (original models plus two additional models: Centaur,  fine-tuned with human cognitive tests performances [1], and its base model Llama-3.1-70B-Instruct) on a difficult parameter set using direct generation. The results (Table 9) show expected score decrease, suggesting that Reflection-Bench is both leakage-resistant and far from saturated.\n\nAs for the score interpretation, we conducted one million random simulations for five applicable tasks (excluding the non-parameterized oddball task and qualitative meta-bandit task) to establish chance-level thresholds (95th percentile). Scores above these thresholds (Table 9) indicate statistically significant task performance, with higher scores reflecting a more precise inference of task parameters.\n\n# Definition of agency\n\nThank you for your feedback on our conceptual precision. We will add a paragraph recognizing the conceptual complexity of \"agency\". To further address the ambiguity between \"agency\" and common agent capabilities like planning and tool usage, we will revise the terminology to \"**epistemic agency**\", a concept that more accurately captures our research scope. It refers to one's intrinsic cognitive foundation for constructing, modifying, and monitoring its beliefs about the external world [2, 3, 4], independent of specific external modules or tools. Additionally, epistemic agency has direct implications for AI trustworthiness -- only when a model possesses robust mechanisms for belief formation and revision can it be considered accountable for its actions.\n\nWe believe this refinement strengthens the theoretical precision and potential contributions of this work. While the research community widely acknowledges that some intrinsic quality of base models significantly determines their effectiveness as agents, this characteristic remains inadequately characterized (sometimes simply labeled as \"intelligence\"). Our benchmark aims to characterize and measure this elusive foundation, .i.e. epistemic agency, which will contribute meaningfully to the field.\n\n# Presentation improvements\n\nWe've enhanced the presentations by:\n- Redesigning Figures 14 & 15 with more informative captions\n- Replacing line plots in Figure 16 with bar charts\n- Adding a comprehensive figure in Section 3.2 with illustrations for all task (Figure Tasks)\n- Fixing spacing issues in figure references throughout the paper\n- Rewording lines 160-162 for clarity: \"For LLMs, prediction capabilities are crucial for planning tasks, where models must reason about which policies will effectively transition an agent from its initial state to a desired goal state.\"\n- adding the prompts for two strategies in Appendix A\n\n# Other specific questions\n\n- Appendix D.1 clarification: We validated our automated evaluation method using 1,950 responses from 13 models. Five of these models (GPT-4, Gemini-1.5-Pro, Llama-3.1-405B/70B/8B) were not included in our main evaluation but were used to verify that our embedding-based method generalizes to other models. This approach ensures that our evaluation methodology is applicable to a diverse range of models.\n- Figure 18 panels: Each panel represents different levels of data aggregation (from 5 to 25 datapoints) used to analyze the correlation between human annotation and automatic evaluation.\n- Qwen categorization: We separated the Qwen-2.5 series (72B, 32B, 14B, 7B) as one category to show how performance scales with model size within a consistent model family.\n- Human baselines: Our adapted tasks differ from standard human cognitive tests, making direct comparisons with existing human scores potentially not appropriate. We will add a discussion in the limitations section about the human-LLM comparisons for establishing the ecological validity of Reflection-Bench.\n- Shape bias in WCST: This is an insightful observation. We conducted additional experiments with four Qwen-2.5 models where we modified both the rule blocks and card formats from shape-color-number to color-number-shape. We found that the \"shape sink\" effect persists for Qwen2.5-14B-Instruct (Figure WCST), while less evident for the other three models. Although beyond the scope of the current paper, further investigation of this phenomenon could provide valuable insights into how language models indirectly encode human cognition in other modalities.\n\nThank you again for your valuable feedback, which has significantly improved our paper.\n\nYou can view the updated table and figures at this anonymous link: https://anonymous.4open.science/r/ICML_Rebuttal-773F/For%20Reviewer%20EAfK.md\n\n[1] Centaur: a foundation model of human cognition\n\n[2] The Routledge Handbook of Philosophy of Agency\n\n[3] Knowledge, Dexterity, and Attention: A Theory of Epistemic Agency\n\n[4] Belief, Agency, and Knowledge Essays on Epistemic Normativity", "claims_and_evidence": "Claims are supported.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense. The evaluated dimensions are all important for agency, although they do not encompass every aspect that is commonly considered agentic (e.g. planning). The authors evaluate a comprehensive set of models, use different prompting techniques, and repeat experiments to handle stochasticity.", "theoretical_claims": "N>A.", "experimental_designs_or_analyses": "Everything seems sound/valid.", "relation_to_broader_scientific_literature": "Many recent works evaluate agency in LLMs, but the authors distinguish themselves by defining agency along 7 dimensions and taking tasks from cognitive psychology and adapting them to LLMs.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "**Strengths**\n- The tasks are original, well-adapted for LLMs, and the results can discriminate between LLMs well\n- For one task currently all LLMs fail (the meta-reflection task)\n- The authors do a comprehensive evaluation (multiple models, trials, and prompting strategies)\n- The authors do a comprehensive analysis of the results, and do interesting findings such what kind of strategies models employ for sequential belief updating tasks (namely, that models employ a win-stay-lose-switch strategy)\n\n**Weaknesses**\n- Presentation of results in figures. Figure 14 and 15 are difficult to parse. Consider also using informative captions. For Figure 16, you're connecting the dots with lines between models. But the points on the line between models don't indicate anything. For categorical X-labels don't use a line plot.\n- The analysis is only done for one set of parameters; it would be interesting to have at least one more set of parameters to see how models get worse/better for different parameters in the tasks.", "other_comments_or_suggestions": "- Consider highlighting that the definition of agency is a well-studied and difficult subject without consensus, and the presented definition here is not generally accepted.\n- l160-162 needs to be rewritten\n- Would be useful to see a figure or table with an example for each task in 3.2\n- The number reflecting the result is hard to interpret. Can you say what the interpretation is? Would it be useful to get a human score on this benchmark for comparison?\n- Styling error: When referring to figures like \"Figure K\", the space is often missing (e.g. see Line 351r and 315l)", "questions_for_authors": "- Appendix D.1; authors validate automatic evaluation with a human. I can't parse *\"Five of them were out of the evaluated models in this paper for verifying the generality of our automated evaluation method based on text-embedding-3-large.\"* How many examples do you evaluate? And only of 5 models of 13? How were these selected?\n- Figure 18; what does each panel refer to?\n- How is CoT and direct generation done? I can't find the prompts in the appendix.\n- Why is Qwen a separate category of models?\n- Interesting analysis for the Wisconsin card sorting test; models not being able to change from the shape rule they determined (\"shape sink\"). could this be connected to a shape bias in humans (Landau et al. 1988). E.g., maybe models could change to different rule if it does not concern a shape rule, did you try this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DhhhaQXyrr", "reviewer": "Reviewer_5qGQ", "summary": "The authors propose Reflection-Bench as a contamination-free benchmark consisting of seven parameterized cognitive tests inspired by cognitive psychology paradigms. The experimental evaluation spans 16 prominent LLMs and three prompting strategies: direct generation, free output, and Chain-of-Thought (CoT). Results identify a three-tier performance hierarchy among models, highlighting significant limitations particularly in meta-reflection. The paper concludes with implications for future research, notably enhancing meta-cognition, developing adaptive cognitive strategies, and encouraging coordinated cognitive capabilities within LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and recognition of our benchmark's strengths. We have addressed your key concerns as follows:\n\n# Evaluation across different difficulty levels\n\nWe fully agree with your suggestion about evaluating tasks at varying difficulty levels. In response, we have expanded our evaluation to include more challenging parameter configurations. We tested 18 models (including the 16 original models plus Centaur, a model fine-tuned with human performances on various cognitive tests [1], and its base model Llama-3.1-70B-Instruct) on harder configurations using direct generation. The results, presented in a new table (Table 9), show the expected performance decreases on harder configurations and suggest that Reflection-Bench is both minimizing contamination and far from saturated.\n\nWe also conducted 1 million random simulations across five applicable tasks, establishing robust chance-level thresholds at the 95th percentile of random performance distributions (Table 9). This provides a statistically sound metric for determining whether models merely producing plausible-looking outputs. Performance exceeding these thresholds indicates that models have developed meaningful understanding of the underlying task parameters. In our revised manuscript, we will incorporate these statistical benchmarks alongside our existing metrics. Thank you for helping enhance the scientific rigor of our evaluation framework.\nYou can check Table 9 at this anonymous link: https://anonymous.4open.science/r/ICML_Rebuttal-773F/For%20Reviewer%205qGQ.md\n\n# Regarding real-world generalizability\n\n## Definition Clarification \n\nThank you for pointing out the concern about real-world applicability. While we would like to clarify that our research intentionally focuses on assessing models' intrinsic capabilities at the cognitive level, independent of specific external tools or applications. To further clarify our scope, we have refined our terminology \"**agency**\" to \"**epistemic agency**,\" which more precisely captures our focus on cognitive capabilities that enable belief formation, revision, and reflection in dynamic environments -- making operational capabilities such as planning possible and reliable [2, 3, 4]. Undoubtedly, without robust intrinsic epistemic agency, even the most sophisticated tool integration or workflow design will be limited by the model's core cognitive constraints and, therefore, not reliable. The research community has increasingly recognized that LLM-based agent performance critically hinges on some intrinsic quality of the base model, yet there remains considerable ambiguity about the precise nature of this quality. Our benchmark aims to evaluate it as \"epistemic agency\" that determines whether models can serve as reliable cores for AI agents in any real-world context. We will revise our manuscript accordingly and update the title to \"Reflection-Bench: Evaluating Epistemic Agency in Large Language Models.\" We believe that by establishing \"epistemic agency\" as a well-defined, measurable characteristic of language models, our work provides the missing framework to systematically identify and evaluate this crucial but previously nebulous quality.\n\n## Ecological validity \n\nAs we emphasized in line 76-80:\n\n\"Cognitive tests create controlled environments where subjects must learn and reason about unknown parameters through interaction, offering standardized, quantified, and objective assessment tools that mirror real-world functioning. \",\n\ncognitive tests are specifically designed to extract and evaluate the abstract cognitive features underlying everyday real-world functioning. This property, known as \"ecological validity\" in cognitive assessment, allows controlled tasks to provide meaningful insights into fundamental capabilities that support real-world performance. Similarly, our adapted tests extract core cognitive dimensions essential for epistemic agency in any environment. We will acknowledge in our Limitations section that the ecological validity of Reflection-Bench for LLMs specifically requires further validation, and identify this as an important direction for future work. We will also discuss how future iterations could incorporate more naturalistic contexts to complement our controlled assessments.\n\nWe are grateful for your thoughtful feedback, which significantly strengthens both the conceptual clarity and methodological rigor of our research, enhancing Reflection-Bench's contributions to the field by providing a comprehensive framework for evaluating the foundational capabilities that ultimately determine an LLM's effectiveness as a reliable agent core.\n\n[1] Centaur: a foundation model of human cognition\n\n[2] The Routledge Handbook of Philosophy of Agency\n\n[3] Knowledge, Dexterity, and Attention: A Theory of Epistemic Agency\n\n[4] Belief, Agency, and Knowledge Essays on Epistemic Normativity", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "- The authors only evaluated entry-level difficulty tasks. Evaluating different difficulty levels or scalability (e.g., medium and high difficulty) would strengthen the generalizability and robustness of results.\n- While focused intentionally on intrinsic agency, the paper does not investigate how agency manifests within integrated, real-world agent workflows, limiting practical generalizability.", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "The current tasks, while well-designed, remain somewhat abstracted from realistic, naturalistic contexts, potentially limiting their predictive power regarding real-world agent performance.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths:**\n1. **Comprehensive Framework:** Reflection-Bench is systematically designed, covering a robust set of cognitive dimensions rooted in cognitive psychology literature, and the tests are thoughtfully adapted to suit LLM evaluation contexts.\n\n2. **Novelty and Significance:** The paper addresses the understudied aspect of intrinsic agency in LLMs, providing a comprehensive cognitive-level evaluation beyond traditional application-specific benchmarks.\n\n3. **Clear Methodology:** The authors provide a detailed description of cognitive tests and their adaptation methods, ensuring clarity and reproducibility.\n\n**Weaknesses:**\n\nSee Above", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8DMYSGBZh5", "reviewer": "Reviewer_Xeen", "summary": "This paper proposes a cognitive-inspired benchmark, reflection-bench, to evaluate agency in LLMs. By decomposing the necessary cognitive procedures an agent would be required, the paper lists out seven important cognitive functions, including prediction, memory, belief updating, meta-reflection, and so on. The study selects a representative cognitive task from fields of psychology and cognitive science for each of the domains, and tests on multiple LLMs as well as several prompting strategies, such as CoT. The benchmark reveals models with larger sizes tend to perform much better than smaller sizes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and specialist insights. We deeply value your expertise and have carefully considered each point of your feedback.\n\n# \"Agency\" and its evaluation\n\nWe appreciate your insightful critique regarding our \"agency\" conceptual framework. Your concerns about the terminology overlap of agency between our work and conventional agent capabilities is astute. Your comment that \"... hard to be called 'agency', since there is no agent at all...\" reflects a conception of agency centered on agent capabilities such as tool use. We would like to clarify that our focus is on the base model's foundational intrinsic quality, making these operational capabilities possible. In your recommended work Demircan et al., 2024, researchers compare Llama models with \"human agents\" and \"Q-learning agents,\" suggesting that certain type of agency can be investigated where \"there is no agent\". Indeed, such kind of agency is undefined by the community yet, although they have gradually realized that base model's certain intrinsic quality significantly determines its effectiveness when deployed as an agent. This conceptual ambiguity highlights the need for more precise terminology to describe our scope.\n\nAfter careful literature review, we will refine our terminology from the broader \"**agency**\" to the more precise \"**epistemic agency**\" to better reflect our focus, e.g., changing the title to \"Reflection-Bench: Evaluating Epistemic Agency in Large Language Models.\" Epistemic agency refers to the capacity to actively form and revise beliefs about external environments—evaluating evidence, updating beliefs with new information, and reflecting on belief-forming processes [1, 2, 3]. This concept more accurately captures what we evaluate: the cognitive processes underlying belief formation (prediction, decision-making, error detection, memory retrieval, counterfactual thinking, and belief updating) and meta-reflection on these processes.\n\nAs we emphasize throughout our paper (Lines 21, 46-48, 140-145, 149), we fully agree that the seven cognitive dimensions are not separated but closely interconnected. Our methodological choice to evaluate them separately allows us to identify specific limitations that might be obscured in fully integrated tasks. By separately evaluating these cognitive dimensions—just as standardized cognitive assessments do in psychological research—we could establish necessary conditions for epistemic agency while providing targeted diagnostic insights that can guide focused improvements. Your comment that performance on separate tasks \"does not require agency at all\" might reflect a misunderstanding of our approach. While excellence in separated tasks doesn't constitute complete epistemic agency, deficits in any functions necessarily constrain a model's overall epistemic agency.\n\nWe certainly recognize the value that integrative evaluations in game-like environments can provide, as demonstrated in recent work (Allen et al., 2024). We will discuss game-like evaluations as promising directions for future work.\n\n# Deeper analysis and experiments\n\nConsidering our primary focus is establishing a standardized benchmark, we conducted 1 million random simulations across applicable tasks, establishing chance-level thresholds at the 95th percentile (Table 9). This provides a statistically sound metric for determining whether models merely produce plausible-looking outputs. We will incorporate it alongside our existing metrics. Thank you for helping enhance the scientific rigor of our work.\n\nFollowing your recommendation, we've added evaluation results for both Centaur and its base model on easy and hard settings. But Centaur shows no improvement, despite using specific prompt format as suggested. This unexpected result suggests Reflection-Bench reduces contamination. You can check the results at this anonymous link:  https://anonymous.4open.science/r/ICML_Rebuttal-773F/For%20Reviewer%20Xeen.md\n\nYour suggestion to apply computational models and SAE analysis is thoughtful. While such approaches provide additional insights, implementing these analyses expands our scope beyond establishing a comprehensive benchmark. For our task adaptations, new computational models would be needed. And behavioral failures present methodological challenges for internal representations analysis We anticipate that our benchmark will promote more such works in future.\n\n### Other concerns\n\nWe will enhance experimental clarity in our revised manuscript Section 4.1 and update the references to include original works accordingly.\n\nThank you for your thorough and constructive review. We are grateful for your specialist comments, which strengthen our work regarding terminology refinement, statistical validation, and future research directions. \n\n[1] The Routledge Handbook of Philosophy of Agency\n\n[2] Knowledge, Dexterity, and Attention: A Theory of Epistemic Agency\n\n[3] Belief, Agency, and Knowledge Essays on Epistemic Normativity", "claims_and_evidence": "The paper has several claims regarding the benchmarks.\n\nFirst, the paper claims the benchmark as a benchmark of agency. The evidence is that they decompose what an agent needs to do cognitively to interact with the environment, and test the cognitive functions with relevant but separate tasks. However, this kind of evaluation is hard to be called 'agency', since there is no agent in the benchmark at all. Though people have not reached complete consensus about the definition of agent, an agent typically should be able to use tools, learn and interact with the environment, goal-directed as well as plan for the goals, and of course, those dimensions already proposed by the paper. However, when evaluating an agent, we should place them in a real-world like environment to measure their capacity integratively, not separately. It's hard to say if a model performs all that well in the current agency benchmark can be called a good agent, since engaging in these tasks separately does not require agency at all. Therefore, agency, though can be composed of many cognitive properties, it is an integrative concept. Evaluating agency with separate cognitive domains does not make sense. In this sense, I can not agree that this benchmark is really evaluating agency.\n\nSecond, going specifically, the benchmark picks up seven representative cognitive tasks, corresponding to seven domains. However, when evaluating specific behaviors, only raw performance and some naive behavioral patterns are analyzed. The authors should go into more detail about the model behavior by applying computational models to describe and interpret their behaviors. In cognitive science and psychology, there are many computational models proposed. Using these models to fit parameters and even select a proper model could provide a deeper insight into LLM's behaviors.\n\nLastly, for each task, I do not know how many repeated experiments were conducted and how the configurations of LLMs, such as temperatures, top K, top P, are set up. Comparing that performance also does not reveal any statistical analysis to show meaningful differences. This also pins out the importance of going beyond purely behavioral performance, since some tasks like the probabilistic reversal learning task contain noises. Without a detailed look at behavioral patterns and statistical analysis, it's hard to say these models are really different in performance.", "methods_and_evaluation_criteria": "Since it's a primarily benchmark paper, the evaluation and methods have been extensively discussed above.", "theoretical_claims": "The paper does not involve theoretical proof, except the concept decomposition about agency. Though these cognitive dimensions make sense, the authors overlook that these dimensions should be integrated, not separated.", "experimental_designs_or_analyses": "The tasks are mainly from the literatures in cognitive science and psychology, which are representative and mature. As I mention above, to better evaluate agency, the paper could be improved by testing on an integrative task, like a game (Allen et.al., 2024), to measure LLM's behaviors and cognitive functions integratively.\n\nSome other approaches to deepen the design and analysis can be fine-tuning models (like Centuar, Binz et.al., 2024) on these tasks and testing them on alternative tasks, which also correspond to these concepts. This not only helps build up the understanding of how fine-tuning models can be used to improve agency, but also helps to evaluate how generalizable these selected tasks are.\n\nOr the authors could go deeper into computational modeling analysis of behaviors, as well as look into how the model's internal representations support their specific behavioral patterns and strategies (with interpretability tools like SAE).\n\nReferences:\nAllen, K., Brändle, F., Botvinick, M., Fan, J. E., Gershman, S. J., Gopnik, A., ... & Schulz, E. (2024). Using games to understand the mind. Nature human behaviour, 8(6), 1035-1043.\nBinz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., ... & Schulz, E. (2024). Centaur: a foundation model of human cognition. arXiv preprint arXiv:2410.20268.\nDemircan, C., Saanum, T., Jagadish, A. K., Binz, M., & Schulz, E. (2024). Sparse autoencoders reveal temporal difference learning in large language models. arXiv preprint arXiv:2410.01280.", "relation_to_broader_scientific_literature": "This benchmark proposes seven aspects of agency in cognitive domains, which are used to test extensive existing LLMs. They can somehow improve our understanding of these models' performance in these tasks. However, due to its lack of integration in these aspects in the task, as well as limited depth about analyzing those behaviors in a computational and neural (representational) aspect, the impact of the work is limited.", "essential_references_not_discussed": "The paper has listed and discussed key literatures in the intersection of cognitive science and AI, especially important milestones and benchmarks in these fields.\n\nHowever, in some detailed tasks, the citations of the reference may not be proper enough. For example, when citing references about the tasks, the authors should be aware of the origin of the task, not just cite a relevant paper. I have not gone through all the citations, but references below are definitely not accurate enough:\n\nFor example, in the Iowa Gambling Task, they should cite this:\nBechara, A., Damasio, A. R., Damasio, H., & Anderson, S. W. (2013). Insensitivity to future consequences following damage to human prefrontal cortex. In Personality and Personality Disorders (pp. 287-295). Routledge. (or 1994 the earlist version)\ninstead of this:\nBuelow, M. T. and Suhr, J. A. Construct validity of the iowa gambling task. Neuropsychology review, 19:102–114, 2009. URL https://doi.org/10.1007/s11065-009-9083-4.\n\nfor the probabilistic reversal learning task, they should cite this at least (or earlier work):\nCools, R., Clark, L., Owen, A. M., & Robbins, T. W. (2002). Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. Journal of Neuroscience, 22(11), 4563-4567.\n\nSince this benchmark is cognitive-inspired, necessary and accurate background about cognitive science and psychology would help the benchmark be established more properly.", "other_strengths_and_weaknesses": "Strengths: the paper examines extensive existing models, which are comprehensive and robust. The writing is clear and the visualization is neat and easy to understand.", "other_comments_or_suggestions": "I have no other comments and suggestions.", "questions_for_authors": "I current do not have any questions and will propose in the engagement of the rebuttal phase.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors introduce Reflection-Bench for evaluating “agency” in LLMs. The evaluation is based on seven cognitive tests designed to avoid dataset contamination for fair evaluation. The work nicely outlines ontologically what is involved in the notion of agency though there is still a potential mismatch in the reader’s expectation and the authors’ intention. In particular, reviewers expect certain abilities/definitions including planning and tool use which define a more robust and real world evaluation while the authors claim this is misaligned with their work’s aims – epistemic. The reviewers believe that a revised submission which more accurately describes the assumptions of the work will lead to a more accurate read. Aside from framing, there is a concern about contamination – an issue that has come up with other cognitive tests that have undergone permutation for evaluation (e.g. Sally Anne variants). This likely also requires careful phrasing to ensure an accurate representation of the results.
Decision: Accept (poster)