Large Language Models Think Too Fast To Explore Effectively
Authors: Lan Pan
Hanbo Xie
Robert Wilson
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Exploration
Empowerment
Uncertainty
Reasoning
Primary area: neuroscience_and_cognitive_science
TL;DR: LLMs struggle with balancing uncertainty and empowerment in exploration, underperforming in open-ended tasks due to premature decisions driven by early-stage processing biases.

Abstract: Large Language Models (LLMs) have emerged with many intellectual capacities. While numerous benchmarks assess their intelligence, limited attention has been given to their ability to explore—an essential capacity for discovering new information and adapting to novel environments in both natural and artificial systems. The extent to which LLMs can effectively explore, particularly in open-ended tasks, remains unclear. This study investigates whether LLMs can surpass humans in exploration during an open-ended task, using Little Alchemy 2 as a paradigm, where agents combine elements to discover new ones. Results show most LLMs underperform compared to humans, except for the o1 model, with those traditional LLMs relying primarily on uncertainty-driven strategies, unlike humans who balance uncertainty and empowerment. Results indicate that traditional reasoning-focused LLMs, such as GPT-4o, exhibit a significantly faster and less detailed reasoning process, limiting their exploratory performance. In contrast, the DeepSeek reasoning model demonstrates prolonged, iterative thought processes marked by repetitive analysis of combinations and past trials, reflecting a more thorough and human-like exploration strategy. Representational analysis of the models with Sparse Autoencoders (SAE) revealed that uncertainty and choices are represented at earlier transformer blocks, while empowerment values are processed later, causing LLMs to think too fast and make premature decisions, hindering effective exploration. These findings shed light on the limitations of LLM exploration and suggest directions for improving their adaptability.

Large Language Models Think Too Fast To Explore Effectively
Lan Pan"
Hanbo Xie* School of Psychology Georgia Institute of Technology
School of Psychology
Georgia Institute of Technology
Atlanta, GA, USA Jouannapan@gmail.. com
Atlanta, GA, USA
harboxiee9990ggaecc.. edu
Robert C. Wilson
School of Psychology
Georgia Institute of Technology
Atlanta, GA, USA rvilson337@gatech edu
Abstract
Large Language Models (LLMs) have emerged with many intellectual capacities. While numerous benchmarks assess their intelligence, limited attention has been given to their ability to explore an essential capacity for discovering new informa- tion and adapting to novel environments in both natural and artificial systems. The extent to which LLMs can effectively explore, particularly in open-ended tasks, remains unclear. This study investigates whether L.LMs can surpass humans exploration during open-ended task, using Little Alchemy 2 paradigm, where agents combine elements discover new ones. Results show most L.MSs underper- form compared to humans, except for the ol model, with those traditional LLMs relying primarily on uncertainty-drict
strategies, unlike humans who balance uncertainty and empowerment. Results indicate that traditional reasoning-focused LLMs, such as GPT-40, exhibit a significantly faster and less detailed reason- ing process, limiting their exploratory performance. In contrast, the DeepSeek reasoning model demonstrates prolonged, iterative thought processes marked by repetitive analysis of combinations and past trials, reflecting human-like exploration strategy. Representational analysis of the models with Sparse Autoencoders (SAE) revealed that uncertainty and choices represented at earlier transformer blocks, while empowerment values are processed later, caus- ing LLMs to think too fast and make premature decisions, hindering effective exploration. These findings shed light on the limitations of LLM exploration and suggest directions for improving their adaptability.
thorough and
Introduction
Large Language Models (LLMs) have become landmarks of modern Artificial Intelligence, show- casing remarkable human-like cognitive capacities through their ability to predict and generate text recursively 4 41 45). The question of whether LLMs have reached will achieve Artificial General Intelligence (AGI) continues to spark debate, fueled by an ever-growing body of empirical evaluations.
"qqual contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

While extensive benchmarks have been developed to assess how LLMs perceive, think, reason, and act across diverse environments, limited attention has been given to their capacity for exploration. Exploration sxeforation-ddefined defined as behaviors aimed at discovering new information, possibilities, or strategies, crucial role intelligence, enhancing long-term
often at the expense of immediate rewards-play understanding, adaptability, and performance. This behavior stands in contrast to exploitation, which focuses on leveraging known information for immediate benefits.
Exploration has been extensively studied in the fields of Reinforcement Learning 40 and human learning [C 49
human learning, exploration strategies are typically categorized into three types: random exploration, ancertainty-driver exploration, and empowerment. Random exploration introduces stochastic noise into behaviors, enabling agents to stumble upon new informa- tion. Uncertainty-drive ambiguity and improve decision-makes confidence. Empowerment, on the other hand, emphasizes intrinsic rewards and open-ended discovery, driving agents to maximize possibilities rather than optimizing specific outcomes. This type of exploration aligns closely with behaviors observed in tasks like scientific research, where the goal is to uncover many novel possibilities as possible.
exploration prioritizes sampling actions with uncertain outcomes to reduce
Although preliminary research suggests that LLMs exhibit limited exploratory behavior compared to humans 4 current investigations are narrow in scope, often focusing on bandit tasks 24 These studies provide an incomplete understanding, neglecting the diverse forms of exploration, particularly empowerment. To bridge this gap, our study investigates LLMs' exploration capacities in a broader context, examining both uncertainty-drit exploration and empowerment.
We address three key research questions in this work:
Can Large Language Models explore effectively in open-ended task, comparable to humans?
What exploration strategies do LLMs employ, and how do these compare human strategies?
Why do LLMs succeed or fail in exploratory tasks, and what mechanisms underpin their performance?
To explore these questions, we adopt the experimental paradigm of Brändle et al. ( using the video game Little Alchemy 2 (see methods 3.J). In this game, participants aim to create as many elements as possible by combining known elements, a task that closely aligns with the concept of empowerment and offers robust framework for evaluating open-ended exploration. We then apply regression models to analyze the exploration strategies of humans and LLMs, focusing on both uncertainty-drivee and empowerment-base based behaviors (see methods 3.3). Finally, deploy Sparse AutoEncoders (SAE) to probe the latent representations of exploration-relate values, providing insights into how LLMs process information and generate exploratory behavior. This study not only enhances our understanding of LLMs' exploratory abilities but also highlights exploration as a key element for building more adaptive and intelligent Al systems.
2 Related Work
Exploration in Reinforcement Learning and Human Learning
Exploration, as a crucial intellectual capacity in both natural and artificial agents, has been extensively investigated in both reinforcement learning and human learning. In reinforcement learning, decades of research have led to the development of principled exploration mechanisms, including optimism in the face of uncertainty [5], posterior sampling (29), and intrinsic motivation based on novelty or prediction error [118 (18). More recent advances emphasize structured and scalable exploration using ensemble-base uncertainty estimates [28), world- model disagreement (36), and offline unsupervised skill discovery [13]. These algorithmic innovations not only aim to solve high-dimensional, sparse- reward problems but also offer testable analogues to human strategies such as random exploration, goal-directed novelty seeking, and strategic uncertainty reduction.
In human learning, cognitive scientists primarily focus on the neurocomputational mechanisms underlying exploratory behavior. Through computational modeling, researchers have identified exploration strategies commonly used by humans, such as random and directed exploration [49 17 47 as well as empowerment (6) Numerous studies have shown that humans excel at balancing

---

exploitation and exploration [38 14], and at employing diverse exploratory strategies [48]. Using neuroimaging and non-invasive brain stimulation techniques, neuroscientists have revealed the neural correlates and causal mechanisms of exploration behaviors, strategies, and task representations in regions such as the frontopolar cortex and intraparietal sulcus [10 53 These algorithmic and implementational discoveries help explain how and why humans explore effectively.
Understanding LLM Cognition
Recent research has examined the cognitive capacities of large language models (LLMs), covering domains such as periision-making 44, theory of mind [39], analogical reasoning [45], and temporal- difference learning [12]. However, most these studies focus on evaluating task performance across model variants, offering limited insight into the internal computational mechanisms behind success or failure. Notably, Demircan et al. 12] applied sparse autoencoders (SAEs) 23 7 to investigate value-state representations in TD-learning, demonstrating both correlational and causal links between activations and learned value structures- an approach that parallels techniques in human neuroscience. Yet, despite growing interest in LLM cognition, the capacity of LLMs to explore remains underexplored. Existing studies on LLM exploration are largely confined to bandit tasks 24 22, leaving open the question of whether-and WLMs fail explore effectively in open-ended environments.
LLM Agents Exploration
In a parallel line of research, several works have explored the use of language models as agents In addition, recent studies have deployed LLMs in open-ended 15 16 or for hypothesis testing [20]. There is also algorithmic and mechanistic research on the exploration of LLM agents, such as applying RL algorithms (e.g.. Posterior Sampling for Reinforcement Learning) to LLM agents 3 [8]. or leveraging known abilities like in-context learning to enable exploration [35 33 However, these efforts primarily aim to optimize task performance and often lack deeper investigation into the algorithmic and implementational mechanisms underlying the exploration capabilities of LLMs in such environments.
in text-based games 2 52 30
environments such as Minecraft
3 Method
3.1 Task Description: Little Alchemy 2
Little Alchemy 2 involves discovering new elements by combining a predefined set of basic elements: water, fire, earth, and air. These elements serve the initial inventory, and players (humans LLMs) attempt to discover new combinations based on deterministic rules (see Figure [ The total is 720 elements and the elements range across categories including nature, space, animals, plants, food, inventions, technology, science, tools, buildings, lore, and myths. Among these elements, only 3,452 combinations (out of 259,560) can successfully create other elements and therefore, requires a semantic understanding of the elements (i.e., empowerment) to explore effectively. This framework mimies creative combinatorial space exploration task, challenging participants to explore patterns to expand their inventory.
3.2 Experimental Setup
Data from 29,493 human participants across 4.691,033 trials establish the benchmark. The players were instructed in the rules of the game and tasked with discovering new elements. Performance was measured by the average number of new elements discovered.
We evaluated the performance five LLMs: gpt-4o-2024-08- (GPT-4o (6)). 01-2024-12-17 (ol (27)) Meta-Llama-33 1...3.18---Inttttt Instruct (LLaMA3.1-8) (22). dita--lama--.117000 Instruct (LLaMA3.1- 70B (22)), and DeepSeek-PR R1 [T]] These models were selected to represent a range of model sizes and architectures, closed source well open source, allowing us to analyze impacts from model variations on exploration and discovery. Each model was prompted with game rules, current inventory, and trial history contextual reasoning (Figure Tp. Their outputs were constrained to valid game actions, in the format of element + element (for complete prompts, see Figure SA).
To investigate the impact of randomness exploration, we varied the sampling temperature across four settings: 0.0. 0.3, 0.7, and 1.0 (ol is not available to set parameters and defaults as 1), and under each temperature, there are five repetitions for running the experiment. Lower temperatures encourage

---

deterministic outputs, favoring exploitation, while higher temperatures introduce stochasticity, pro- moting exploration. These settings allowed to examine the trade-offs between exploring uncertain combinations and leveraging known strategies. OpenAl and Deepseek-RI models experiment are conducted via API calls. L.L.M3.1-700 and I.LAM33 are launched in computing cluster, with an NVIDIA A A100-80GB running 13 hours and 2 hours to complete all the experiments (with all the settings and repetitions).
3.3 Regression: Empowerment vs. Uncertainty-Drien -Driven Strategies
To analyze exploration dynamics in Little Alchemy 2, we assessed the roles of empowerment and uncertainty in decision-making:
3.3.1 Empowerment
Empowerment: In the context Little Alchemy 2, empowerment translates into the players' intrinsic desire to create elements that offer many successful combinations, selecting those that maximize future potential (e.g., unlocking paths to more elements). For example, the element "human" in combination with other elements leads to 83 elements, while "alien" leads only new element. Thus, the "human" element is more empowering than the "alien" element. Brändle et al. [6] uses neural network predict empowerment because effectively models the complex combinatorial relationships and potential outcomes within Little Alchemy 2. By leveraging the neural network, the method incorporates multiple factors, including the probabilities of successful combinations, the likelihood of specific results, and the intrinsic empowerment values of resulting elements. This approach ensures an accurate estimation of the empowerment value by capturing both the immediate and future potential combinations.
To align with the original methodology, from the neural network model in our regression. Empowerment E(EcA,n) for a combination CA,B is modeled as:
the empowerment value of each combination
720
E(eAA.)) P(linkea.
P P(resultea. =) F(ee))
where:
Plinkee Probability of successfully combining A and B.
P(resultes.s i): Probability that CA,B results element
E(ee): Empowerment of based on future combinations.
As new trials occur and outcomes are observed (e.g., combining water and fire leads to novel inventory steam), these outcomes provide evidence update the empowerment values of elements used the combination. The success or failure attempts refines the empowerment scores, which directly influence choices made by the LLMs in the following trials. The empowerment value for each trial's elements is updated using dynamic updates, based on the empowerment values predicted by neural network. Empowerment is updated follows: when successful combination creates a novel result, the empowerment values of the involved elements are increased. If a successful combination is repeated and no new elements are created, empowerment remains unchanged. On failure, the empowerment values are slightly decreased. This method captures the intrinsic motivation of selecting combinations with higher future potential, refining element values dynamically as the game progresses. Empowerment scores are updated via dynamic updating based on trial outcomes.
E(e:) increase_factor, if success I(E((e)) = F((e)) decrease_factor
if fail if repeat
E(e:),
3.3.2 Drrertainty-Driven Exploration
Uncertainty reflects the novelty of element use, defined as:
log (T)
U.
te

---

where T' is the total trials, and te is the count of element being chosen. Higher Ug encourages exploration of less-used elements.
3.4 Statistical Analysis
To examine the relationship between temperature, empowerment, uncertainty, and performance in Little Alchemy 2, we employed generalized linear mixed-effects models (GLMMs) with varying configurations tailored to different aspects of the task. This approach allowed assess how LLMs and humans adapt their exploration strategies under different conditions. Model 1: We modeled the decision-makien making process to explore the influence of empowerment and uncertainty on element selection. Model 2: To investigate how sampling temperature interacts with empower- ment and uncertainty, we extended the above model, interaction terms (temperature rempoeement,,, cenperature"uncertaingg uncertainty) to assess how temperature impacts empowerment- and uncertainty-drivet strategies.
3.5 Thought Analysis
To investigate the differences in reasoning processes between traditional LLMs and specialized reasoning models, we compared GPT-4o and DeepSeek-R1. For GPT-4o, reasoning traces were elicited using Chain-of-Thought (CoT) prompting and extracted from its responses. In contrast, DeepSeek-RI natively generates reasoning tokens without explicit prompting.
We analyzed the reasoning traces by first segmenting them into individual sentences. Each sentence was then annotated with one of seven predefined reasoning labels: state goals, check current inventory, past trial analysis, element property reasoning, combination analysis, outcome prediction, and final choice. We employed GPT-4. as an automated classifier to assign labels to each sentence.
To reduce redundancy and better capture the structure of coherent reasoning segments, consecutive sentences with the same label were merged into a single labeled unit. This allowed for a more interpretable and concise representation of the reasoning flow.
3.6 Sparse Autoencoder (SAE) Analysis
SAE is a type of auto-encoder structure that can reconstruct inputs with L2 norms in the latent space 23
x g([(x; We, ba); Wa, ba),
where:
z= f(x: We, b.) oWW.oWWWWW.... xx b.) is the encoder output (latent representation).
x g(;; Wa, d, ba) W.a + b the decoder output (reconstructed input).
We E Rmxn and Wa E Raxm are the encoder and decoder weights, with W. = W.
be E Rm and b E R are the encoder and decoder biases.
a()) is the activation function (i.e., ReLU).
The goal of training SAE is to minimize the reconstruction loss, as well as the L2 norm which forces the latent space to be sparsity:
SAAE
(x
- 10
11 +
Σ 2 w/|
where:
(x(+)_ x is the mean squared reconstruction error for sample
A is the weight for the sparsity regularization term.
2j is the activation of the j-th neuron in the latent representation z.
/wylle l is the L2 norm of the corresponding encoder weight vector wj.
5

---

Recently, SAE has been proposed to understand the latent representation in language models 7 12 To explore how Large Language Models (LLMs) represent cognitive variables such as empowerment and uncertainty in the context of the alchemy game, we used Sparse Auto-Encoder, (SAEs) to learn the latent representations of elements within the model. For each layer, we extracted embeddings from each trial's available element in the choice set and train them in the SAE. Then we correlate each neuron in the hidden layer in SAEs with our target cognitive variables (i.e., choices, uncertainty values, and empowerment values) and find out the most correlated neuron, where we suppose the relevant variable is most strongly represented. This analysis will help us understand how the model represents and processes cognitive information through the transformer blocks. Finally, conduct an intervention to examine whether ablating the most correlated neuron causally reduces the corresponding exploration strategy employed by the LLM in the task. Training and analyzing SAEs across all the layers takes 42 hours with an NVIDIA V100 GPU.
4 Results
A
C
Game Guide
Workspace Inventory
Final Inventory Distribution Acress Player Types (Densityl
LLaMAA. (Mean 8
Current Inventory
0.10
LLaMA3.3 1--42.1-700- (Meat: 25.4)
Hoto nutar fig. iubaar
GPT-40 (Mean. 34
Human (Mean: 41.8
mac
12 outer
0.06
02. (Mear: 177.2)
Deepfieei-hh #1 (Mean: 05.3)
Deniky
0.06
0.00
50
200
Final inventory
Figure 1: A: LLMS Game Process. LLMs select elements per trial based the inventory and trial history. B: Human Game Interface. Players select two elements to discover new elements, added to the inventory. C: LLMs and Human Performance.
4.1 Most LLMS Performed Worse Than Humans, Except o1
From 29,493 human players, 90% completed fewer than 500 trials. Experiments were set up with 500 trials for the LLMs. On average, LLaMA3.1-90 discovered elements, LLaMA3.1-70B discovered 25 elements, GPT-4o discovered 35 elements, DeepSeek-RI discovered 85 elements, and ol discovered 177 elements (Figure D]. In comparison, humans discovered 42 elements on average within 500 trials and 51 elements across all trials.
ol and DeepSeek-RI significantly outperformed humans (o1: t 9.71, p < 0.001: DeepSeek-RI. t = 3.40. p < 0.027), while the other LLMs performed worse (GPT-40: t -5.48. p < 0.001;
= 128.12, p < 0.001). Perfor- mance improved with larger model sizes, with LLaMA3.1-70B outperforming LLaMA3.1-8B
LLaMA3.1-70B: = -6.39.p p < 0.001; LLaMA3.1-8B:
(1 (t= -6.02. < 0.0001) and GPT-4o slightly surpassing LLaMA3.1-70B (1 (1= =327,,77 = 0.003).
Exploration success declines in later trials as the inventory grows, and it becomes much harder the probability of success decreases (see Appendix[A] AA.. Therefore, different exploration strategies could yield very different performances in the task in different phases. Effective strategies in the latter phase rely on understanding latent game structures (empowerment) rather than uncertainty-driinn exploration. Sampling temperatures were manipulated to assess their impact exploration strategies. Higher temperatures moderately improved performance (3 = 0.124, 2 = 5.060.p < 0.001).
Behavioral patterns (Figure 2 highlight ol's superior strategy, achieving more successful outcomes with new combinations and avoiding repetition of failed or -rrcads-uuccessull pairings (Figure TOA). This underscores o/ 's strong exploratory capacity and innovative approach. For other L.LMs, our result shows that for even larger models, as temperature increases, the percentage of choosing existing combinations decreases (Figure 77. This reveals diminishing number of redundant behaviors among

---

LLMs. More importantly, a majority of these new combinations do not generate new elements, suggesting that the high temperatures only alter the sncertainty-drivee exploration strategies but empowerment, since in larger spaces, only random combinations are not sufficient to perform the task effectively (see Appendix Bp. This also explains why higher temperatures can moderately improve the model performance but are still distant from human behaviors.
375
Looker 10l 6.11
372
Decolesslly
100
8
Figure 2: Human and LLMs different Temperatures Performance. LLM and Human Perfor- mance Across Temperatures. For LLMs, we set four temperatures (0, 0.3, 0.7, 1). LLMs (GPT-40, LLaMA3.1-8B, LLaMA3.1-70B) achieve their best performance at temperature 1.
4.2 LLMs Primarily Use Uncertainty-driven Strategies but Not Empowerment
To examine the exact strategies that the models are using, we calculated uncertainty and empowerment values for each element (see 33.3) Then, based on each LLM's s choices of combinations, each of the elements is encoded as "chosen" or "not chosen". We also used random sampling to balance the proportion of chosen elements and not-chosen elements. Then we use trial numbers, uncertainty values, and empowerment values to predict whether an element chosen or not. In particular, we used linear mixed-effects model with the consideration of random slope on each of these variables, so that we could get individual estimates of each sample.
Most LLMs show near-zero empowerment weights, significantly lower than humans (Figure left panel). This suggests that LLMs rarely use empowerment for decision making. In contrast, ol demonstrates the highest empowerment weights, surpassing humans, indicating a human-like strategy of focusing on actions that expand future possibilities.
Higher temperatures lead to increased reliance on uncertainty-driven strategies (temperature x uncertainty B 2.911, 7.440. < 0.001), but empowerment remains unaffected (3 0.196, = = 1.067,p p- = 0.286). Among all models, only ol, with a fixed temperature of 1, balances uncertainty and empowerment effectively, enabling robust exploration later task stages.
Regression Estimates by Temperature and Model
Empowerment
Uncertainty
6
6
Average Estimate
Average Estimate
2-
2
0-
0
Model
Model
L1
LLMM3311-8
-4
Human DeepSeek-R1
human DeepSeek R1
51
ot
-6
0.0
0.3
0.7
1.0
0.0
0.3
0.7
1.0
Temperature
Temperature
Figure 3: Regression Estimates by Temperature and Model. All models show lower empowerment weights than humans, except ol. As temperature increases, uncertainty weights rise, with of showing the highest weights across all models and humans.

---

4.3 Reasoning Depth and Token Usage in DeepSeek-RI and GPT-4o
To further elucidate the differences in reasoning depth between DeepSeek-RI and GPT-4o, we analyzed their reasoning processes from in the subset of the first 150 trials, focusing on reasoning length, diversity of reasoning labels used, and total token counts associated with each reasoning step. (Figure
Perittial Reasoning Depth Comparison
B
Label
5400
0V-
3Xc
-00
600
150
comeed ovenery
Pla andyee
propecty Jospriig
conditation pndyes
Figure 4: Comparison of reasoning depth and token usage between ReepSeek-R1 and GPT-4o. A: Per-Trial Reasoning Depth. DeepSeek-R1 shows substantially longer reasoning sequences and consistently employs all reasoning labels, while GPT-44 exhibits significantly shorter sequences and fewer reasoning types. B: Per-trial Token Usage by Reasoning Labels. DeepSeek-RI allocates significantly higher number of tokens across all reasoning labels, especially emphasizing selicome_predictii and combination_amallsit GPT-4o uses substantially fewer tokens, limiting the depth and breadth of analysis.
The results highlight clear distinctions between the models. DeepSeek-RI consistently exhibits a significantly longer reasoning length per trial, reflecting extensive deliberation before reaching decision. In contrast, GPT-4o uses minimal reasoning. Additionally, DeepSeek-B R1 systematically leverages all seven reasoning labels, and even shows self-reflection in the form of 'backtracing' (see Appendix (C.2) within each trial, ensuring diverse analytical coverage (Figure The limited engagement of GPT-4o in reasoning results in narrower scope of label usage, confirming its tendency toward shallow exploration.
Further analysis of token allocation across reasoning categories revealed substantial differences (Fig- ure BB). DeepSeek-RI distributes tokens broadly, with particular emphasis on outcome pprediction, combination_anddle
and past_trial_ trial__anllll This suggests robust and methodical evaluation of possible outcomes and historical context, consistent with its more effective exploratory performance. GPT-4o allocates far fewer tokens, indicative of limited engagement in deeper analyses, especially in critical reasoning steps such as outcome prediction and combination evaluation.
These findings reinforce that DeepSeek-RI Iss superior task performance arises from more deliber- ate, diverse, and deeper reasoning processes, whereas GPT-4o's 's weaker performance stems from superficial reasoning strategies and inadequate exploration depth. This highlights the importance of reasoning thoroughness and token resource allocation for effective exploration in open-ended tasks.
4.4 Uncertainty and Choices are Processed Much Earlier Than Empowerment in LLMs
This unbalanced strategy used in traditional LLMs makes us wonder why LLMs could not use empowerment in the game. Theoretically, LLMs should be able to represent the semantic meaning of these elements. Are they really representing such information empowerment but not using or do they lack of ability to understand empowerment? To investigate this question, we employed Sparse Auto-Encoders (SAE) (see methods 3.6 to decompose the latent representation of elements in LLMs to figure out whether both empowerment and uncertainty are properly represented during the computation.
Our results suggest that, in ALaMA3.1-70B, the uncertainty value is highly correlated layer 2 (r 0.73, Figure 5). This suggests that in LLaMA3.1-7000 the uncertainty value is strongly represented in the hidden states. We also discover a moderate correlation with empowerment value at layer 72 (r 0.55, Figure 5) indicating LLaMA3.1-700 also represents empowerment values in

---

the middle layer. Despite both values being represented in the hidden states of the model, we found that when we run logistic regression models for each neuron to predict the choices, the highest beta weights also occur at layer 1 (beta 1.08, Figure 5 aligning with the representation of uncertainty values. This explains why the model mainly deploys uncertainty-drive not empowerment. Interestingly, both choices and uncertainty values are strongly represented at the beginning layers, which may indicate that the model already makes a decision before processing the "empowerment" values of the elements in later layers. We additionally intervene (see Appendix C.4p in those most correlated neurons to investigate causal relationships between them and model empowerment neuron selectively suppresses empowerment- guided behavior, while ablating the uncertainty neuron induces catastrophic performance failure, thereby establishing causal control by these two latent representations.
exploration strategies but
behaviors. Ablating the
UncoeTa Bo SiiLe
Empoweement Sa1e
Chaites
layer
Vaat Cormaagee
0.0
02
0.2
03
50
13
20
80
70
05
0.3
36
25
65
20
60
0.0
10
25
50
90
16
Figure 5: SAE Correlation Analysis. Maximum correlation of uncertainty values across layers. peaking at layer 2. Maximum correlation of empowerment values across layers. peaking at layer 72. Maximum beta weight of choices across layers, peaking layer 1.
5 Discussion and Conclusion
Paper Summary. Exploration is critical for navigating complex, open-ended tasks, yet most LLMs fall short of human-level performance. They tend to over-rely on uncertainty-driver strategies that yield short-term gains but hinder long-term success. Although both uncertainty and empowerment signals are present in their latent spaces, LLMs generally fail to balance them effectively. Notably, models like ol outperform humans, suggesting that reasoning-traiee trained LLMs can better leverage diverse exploration strategies. Our thought analysis further shows that reasoning models engage in deeper, more varied reasoning and allocate more tokens to deliberation, indicating more effortful decision-making than standard LLMs.
Fast Thinking in Traditional LLMs. A key limitation of traditional LLMs in exploratory tasks is their tendency to "think too fast." In L.LMA3.1-70B, we observe that uncertainty-related sig- nals dominate the early transformer layers and correlate strongly with immediate choices, while empowerment signals emerge only in the middle layers. This temporal mismatch causes premature secision-maainn favoring short-term utility over sustained exploration. A similar pattern is replicated in LLaMA3.1-2 (Appendix Figure 16).
We argue that this limitation is largely driven by the autoregressive inference paradigm itself, where next-token generation is conditioned on shallow context accumulation. This architecture may inherently suppress deeper or delayed exploration. We evaluated several mitigation strategies, including prompt engineering and activation intervention (feature steering) (Appendix C Neither prompt tuning nor intervention significantly improved performance.
In contrast, the reasoning model DeepSeek-RR R1 [ and ol [27), which explicitly support multi-step reasoning, reached human-level performance and outperformed all traditional LLMs. Their superior results suggest that architecture and training design- not just prompting- are crucial for enabling effective exploration.
Further supporting this interpretation, our thought analysis reveals that reasoning models such as DeepSeek-R1 R1 generate more diverse reasoning types and spend significantly more tokens on deliberation before making a decision. Compared to standard LLMs, which exhibit compressed and repetitive patterns, reasoning models demonstrate richer and deeper cognitive structure in their outputs. This may suggest the shift of paradigm, "test-time compute scaling" [27 37,, promising for the open-ended exploration tasks through augmented reasoning capacity.

---

Limitations and Future Directions. Despite these findings, the underlying cause of LLMs "thinking too fast" remains unclear and requires further investigation. Future research could explore the interaction between model architecture and processing dynamies, as well as how LLMs weigh uncertainty and empowerment during decision-making. Interventions such as integrating extended reasoning frameworks like Co'T, optimizing transformer block interactions, or training with explicit exploratory objectives could enhance LLMs' exploratory abilities. These efforts would not only improve performance but also advance our understanding of creating AI systems capable of more human-like exploration.
Broader Applications. Open-ended exploration is critical for LLM agents. It not only enables their "physical bodies" gather more information and experience from external world, but also facilitates more creative knowledge acquisition in the mental space. For example. LLM agents have been used for theorem proving [22, optimal search [44, and automatic scientific discovery 22 9.. These are inherently open-ended environments that require strong exploratory capabilities. Understanding the strengths and limitations of LLMs' open-ended exploration-ann and enhancing them-will will directly improve their efficiency in these real-world domains.
Data Availability
The LLM experimental data generated in this study are available at OSF repository (view-only link) Third-pan data of human participants playing the original Little Alchemy 2 game may be shared upon reasonable request the and analysis adhere to open-access and ethical use principles.
suueiigen.mpp.dee All data used for benchmarking
Code Availability
All custom code used gression analysis, and
for data preprocessing, LLM experiments, and Sparse Autoencoder (SAE) training available at http:///gtttto..ccmm/oooaaaa////
Acknowledgements
We thank Jian-Qiao Zhu and Huadong Xiong for helpful discussions during the development of this manuscript. We are also grateful to Franziska Brändle for sharing the human behavioral data. This research was supported by an SCIALOG Award (#29079) from the Research Corporation for Scientific Advancement (to Robert C. Wilson) and the OpenAI Researcher Access Program (to Hanbo Xie). We also acknowledge computing support provided in part through the Partnership for an Advanced Computing Environment (PACE) the Georgia Institute of Technology, USA. The authors declare no competing interests.
Author Contributions
Lan Pan and Hanbo Xie conceptualized the experimental design. Lan Pan implemented the LLM experiments and collected the datasets. Lan Pan, Hanbo Xie, and Robert C. Wilson designed the analysis pipeline. Lan Pan implemented and interpreted the analyses. Hanbo Xie trained and analyzed the SAE results. Lan Pan and Hanbo Xie drafted the manuscript. All authors reviewed, revised, and approved the final version of the manuscript.
References
[1] Prithviraj Ammanabrolu and Mark O Riedl. Playing text-adventure games with graph-based based deep reinforeement learning, arXiv preprint arXiv:1812.01628, 2018.
[2] Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl. How to avoid being eaten by grue: Structured exploration strategies for textual worlds. arXiv preprint arXiv:2006.0740 0740000000449.... 2020.
10

---

[3] Dilip Arumugam and Thomas L Griffiths. Toward efficient exploration by large language model agents, arXiv preprint arXiv:2504.20997, 2025.
[4] Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 12066))=2118523122 2023.
[5] Ronen Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(0ct):213-231, 2002.
[6] Franziska Brändle, Lena J Stocks, Joshua B Tenenbaum, Samuel J Gershman, and Eric Schulz. Empowerment contributes to exploration behaviour creative video game. Nature Human 2023.
Behaviour, 7(9):
[7] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- html.
2ttt//2222/mmooommmatti
[8] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:/810) (0..8880........ /2894, 2018.
[9] Pablo Samuel Castro, Nenad Tomasev, Ankit Anand, Navodita Sharma, Rishika Mohanta, Aparna Dev, Kuba Perlin, Siddhant Jain, Kyle Levin, Noemi Elteto, et al. Discovering symbolic cognitive models from human and animal behavior. In sectod International Conference on Machine Learning.
[10] Nathaniel Daw, John PO'doherty, Peter Dayan, Ben Seymour, and Raymond Dolan, Cortical substrates for exploratory decisions in humans. Nature, 441 441(7795)8766955 2006.
[11] DeepSeek-AI Team. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[12] Can Demircan, Tankred Saanum, Akshay K Jagadish, Marcel Binz, and Eric Schulz. Sparse autoencoders reveal temporal difference learning large language models. arXiv preprint arXiv:2410.01280, 2024.
[13] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-588 2021.
[14] Haoxue Fan, Samuel Gershman, and Elizabeth Phelps. Trait somatic anxiety is associated with reduced directed exploration and underestimation of uncertainty. Nature Human Behaviour, 101):102-113, 2023.
[15] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open- open-ended embodied agents with internet-scall knowledge. Advances in Neural Information Processing Systems, 35:18333-18862, 2022.
[16] Yicheng Feng, Yuxuan Wang, Jiazheng Liu, Sipeng Zheng, and Zongqing Lu. Llama rider: Spurring large language models to explore the open world. arXiv preprint arXiv:2310.08922. 2023.
[17] Samuel Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34- 42, 2018.
[18] Keegan Harris and Aleksandrs Slivkins. Should you use your large language model to explore or exploit? arXiv preprint wwxiv:2502.00225 2025.
[19] Leslie Pack Kaelbling, Michael Littman, and Andrew W Moore. Reinforcement learning: survey, Journal of artificial intelligence research, 4:237-285, 1996.
11

---

[20] Nan Rosemary Ke, Danny P Sawyer, Hubert Soyer, Martin Engelcke, David P Reichert, Drew A Hudson, John Reid, Alexander Lerchner, Danilo Jimenez Rezende, Timothy Lillicrap. et al. Can foundation models actively gather information in interactive environments to test hypotheses? arXiv preprint arXiv:24/2.06433. 2024.
[21] Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? arXiv preprint 1533: 2024.
[22] Llama Team. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21778, 2024.
[23] Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(201 2011.
2011.
[24] Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoe Le, and Minmin Chen. Evolve: Evaluating and optimizing llms for exploration. arXiv preprint arXiv:2410.06238, 2024.
[25] Alexander Novikov, Ngån Vū, Marvin Eisenberger. Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mchrabian, al. Alphaevolve: coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.15 13131, 2025.
[26] OpenAl Team. Gpt-4o system card. 2024.
[27] OpenAl Team. Openai ol system card.
https://openai..omm cwwpaaiiicooiiii/ie.//ee
openai-or meteai-oo-syaee -o1-sstten card/ 2024.
[28] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
[29] Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
[30] Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. Do 1lm agents have regret? a case study in online learning and games. arXiv preprint 1xXx:22403.16443, 2024.
[31] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[32] Stanislas Polu and Uly Sutskever, Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03333,, 2020.
[33] Numeric Rewards. Beyond numeric rewards: In-context dueling bandits with I1m agents.
[34] Bernardino Romera-Paredes. Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(9995):468 475, 2024.
[35] Alessio Russo, Ryan Welch, and Aldo Pacchiano. Learning to explore: An in-context learning approach for pure exploration. arXiv preprint arXiv:2506.0/877 2025.
[36] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via superviede world models. In International conference on machine learning. pages 9583-8592. PMLR, 2020.
[37] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.. 03314, 2024.
[38] Maarten Speekenbrink and Emmanouil Konstantinidis, Uncertainty and exploration in restless bandit problem. Topics in cognitive science, 7(2):351-367, 2015.
12

---

[39] James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. Testing theory of mind in large language models and humans. Nature Human Behaviour, 3(7):1255-2255 2024.
[40] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press,
2018.
[41] Ala N Tak and Jonathan Gratch, Gpt-4 emulates average-human emotional cognition from a third-person perspective. arXiv preprint DrXhv:2408.137/6 2024.
[42] Qwen Team. Qwen2.5: party of foundation models, September 2024.
[43] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[44] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint urXiv:2305. 16291, 2023.
[45] Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning large language models. Nature Human Behaviour, 7(9):1526 15411126-14,,, 2023.
[46] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824-24837, 2022.
[47] Robert Wilson, Siyu Wang, Hashem Sadeghiych, and Jonathan D Cohen. Deep exploration as unifying account of explore-exploit behavior. 2020.
[48] Robert C Wilson, Elizabeth Bonawitz, Vincent D Costa, and R Becket Ebitz. Balancing exploration and exploitation with information and randomization. Current opinion in behavioral sciences, 38:49-50 56, 2021.
[49] Robert C Wilson, Andra Geana, John M White, Elliot A Ludvig, and Jonathan D Cohen. Humans use directed and random exploration to solve the explore-exploit dilemma. Journal of experimental psychology: General, 143(6):2074, 2014.
[50] Xiao-Wen Yang, Xuan-Yi Yi Zhu, Wen-Da Wei, Ding-Chi Zhang, Jie-Jing Shao, Zhi Zhou, Lan- Zhe Guo, and Yu-Feng Li. Step back to leap forward: Self- backtracking for boosting reasoning of language models. arXiv preprint arXiv:2502 044444, 2025.
[51] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[52] Xingdi Yuan, Mare-Alcxandee Côté, Alessandro Sordoni, Romain Laroche, Remi Tachet des Combes, Matthew Hausknecht, and Adam Trischler. Counting to explore and generalize in text-based games. arXiv preprint arxiv: /806. 1525, 2018.
[53] Wojciech K 7ajkowski, Malgorzata Kossut, and Robert C Wilson. causal role for right frontopolar cortex in directed, but not random, exploration. Elife, 6:e27430, 2017.
13

---

A The Game Difficulty
We real game tree calculate the probability of each player succeeding as the inventory size increases. The simulation incorporates a random setting for selecting elements and combinations, ensuring variability across trials. At each step, new elements are added the inventory based on the successful combinations, and the success probability is recalculated dynamically. The success rate decrease in inventory size grows, the success rate decreases, making further growth increasingly difficult.
aligns with the convergence of inventory growth trends Figure 2 As the
For example, when the inventory size 4, that is, the initial state, there four elements: water, fire, air, and earth. There are 10 combinations between two elements (including the same element). and each combination can succeed. Therefore, when the inventory size is n 4, the player's success rate is 100%. The success probability (P,) is given by:
S P. = C
Where:
Current inventory size.
C, The total number of possible combinations given the inventory size n, C, m(n+11 This includes combinations with repetition (c.g., A A).
S: Number of successful combinations for the current inventory size.
Game Difficulty vs. Inventory Size
1.0
0.8
Success Probability
0.6
0.4
0.2
0
20
40
60
80
100
Inventory Size
Figure 6: Game Difficulty vs. Inventory Size. Based on the real game tree, each inventory size has a different success probability.
14

---

B The LLM Behavior Across Different Temperatures
To better understand the exact changing behaviors of LLMs under different temperatures, we catego- rized all combinations into four types: whether this generates new element, and whether these two combinations have been used before. Our result shows that for even larger models, temperature increases, the percentage of choosing existing failed combinations decreases. This reveals diminish- ing number of redundant behaviors among LLMs. In the meantime, the percentage of choosing new but failed combinations increases significantly, which suggests that the model tends to choose new combinations more often higher temperatures than lower temperatures. More importantly, majority of these combinations do not generate elements, suggesting that high temperatures only alter uncertainty-dret exploration strategies but not empowerment, since larger spaces only random combinations not sufficient to perform the task effectively. This also explains why higher temperatures can moderately improve model performance, but are still distant from human behavior.
LLAVAS do Cordit Pecertsges
LL:M40 1-70 Conditoo Persontages
CPT-4 Card Percetages
Baiding crglie
codion
fotee wilh dlorte
Lediig
28
Serpersture
Figure 7: Behavioral Categories of LLMs at Different Temperatures. Each trial categorized into five conditions: (1) Failure with Existing Combination: The trial repeats previous combination that does not generate new element. (2) Failure with New Combination: The trial uses new combination for the first time, but it fails to generate
element. (3) Success with New Combination: The trial uses a new combination for the first time, successfully generating new element. (4) Success with Existing Combination: The trial repeats previous combination that successfully generates element. (5) Invalid Trial: The chosen one or two elements are not present in the current inventory.
15

---

C Attempts for Model Improvements
To investigate any potential general way improve the performance of models, had several attempts, including prompt engineering, interventions, and experiments on alternative open-source models.
C.1 Prompt Engineering
Because the model exhibited repeated behaviors and did not fully utilize "empowerment", we introduced more direct, guiding prompts (including the steps highlighted Figure 8 to help GPT-4o (temperature = 1) make more diverse and forward-looking choices, including Chain-of- thouuht (CoT 46), self-reflection [ST] and even explicit strategy hint. However, the result shows that simply updating the prompt- even with explicit instructions and reasoning strategies didn't 't have significant improvement performance (average inventory = 43, -1.17, p = 0.304). The model continued to repeat combinations and did not significantly increase its ability to discover new elements. averages 43 elements. This suggests that while prompt engineering can help shape model's outputs, it may not be enough on its own to overcome certain ingrained tendencies such as repeating prior actions failing to maximize exploratory behavior.
LLMS Game Original Prompt
LLMs Prompt Engineering
Welcome to the Alchemy Game! You start with four basic elaments: water fire, earth, and ar.
Welcome the Alcherry Gomel You start with four busic elements: water. fire, earth, and alt
The objective the gome combine elements to create new ones, Each successful combiration adds
The obyective of the combination be used future combinations, your elements to combine by writing them in the format 'element slement: You can cheoss the same element twikor or two different eleme and the order the siements does not matter Each combiration produces determinstic. stic.
combire elements
your which used for future different
Each
combiration
inventory,
two
elements does not matter.
Each combination produc eu detumministi results, meuning there is only one fooed outcome for each combination. Results cannos change upon repetiton, Do not typothesbe new outcomes
combinations.
Only outout the combinatian, no other your best to create more elements. Let's started!
need try
Step Analyzo past attempts idendff patterns (e.s, cloments combine woll).
Step 2: Choose a new combination based on pattems and unused pains.
And you need try your bext create more elements, Let's starledl
Cumeent Invartory:
Aptempts
Current
telement1 elemeentZ cutput the combinationn words, Give one combination every zime.
output
Bafore finalting, check:
Has this combination been tried before?
Iff yes, choose new one.
If no, proceed.
The
should eeeemett elemert2.
combination.
other
combiration every
Figure 8: A: LLMs Game Original Prompt. The prompt for each trial consists of three parts: the system prompt, which provides the game rule guide; the current inventory including those from beginning and discoveries during the game; and the trial history. B: LLMs Game Prompt Engineering. Each colored section highlights a specific goal: The green section encourages models to explore more creative combinations by reminding them that wider variety of elements can unlock new possibilities. The blue section emphasizes avoiding repeated behavior by explicitly instructing the model to check Dast attembts.
A
B
Bost Tamperaturs of Each Model and Human Performance
Average Percentage of Each Condition
175
GPS-4ctmpp 1.01
LLaMAALL T080ems
L-ttlat LCI Gm detiomp 1.01
depporpt cogincciegl
Rverage inventory
125
Percertage 1%6
300
75
20-
30-
6
160 203
100
100
500
Kare Sucemso New
Success & trbling Conditions
Nilato exbling
Ieaild THat
T1al
Figure 9: A: Best Temperature of Each Model and Human Performance. B: Best Temperature of Each Model and Human Behaviors. Choose the LLM models' (GPT-4o, LLaMA3.1-8BB LLaMA3.1-70B best performance at temperature 1, and compare it with human and ol, GPT-4o prompt-engineering (temperature = 1). Compare each model's performance and behaviors.
16

---

C.2 Open-Source Reasoning Model DeepSeek-RI
To investigate whether a reasoning model, which is known as trained with RL algorithms in the inference phase. would generate a better result than traditional LLMs, we also experimented with the most recently publicly released open-source reasoning model, DeepSeek-RI [T]. This model like ol can do deep thain-of-thouge we quickly investigated this model to see how the reasoning models perform and how their reasons can relate to the actual thinking process in this task.
reasoning automatically and the reasoning process is visible. Therefore,
A: Human and LLMs Best Temperatures' Behaviors
Auer Percentage Each Condition
100
60
OfT-uplamp-- m
80-
Peronbage (%6
60
DeggSeck-Rt
Percertee Level
60
15
40-
Laanna tamp cimh Joneo 1.0
20-
20-
Human Pictor PRS1
300 400
500
Conditions
Figure 10: A: Human and LLMs Best Temperatures" Behaviors. According to whether the combination selected by each trial is repeated, successful, and initial, the behavior of each LLM trial divided into 5 categories. Compare the temperature which LLM performs best with humans and o/ behavior. B: LLM Inventory Performance Relative to Human Percentiles.
The result shows that DeepSeek-RI reached near human-level task performance, but still under- performs than ol. In the behavioral patterns, DeepSeek-R2 compared to traditional LLMs have fewer attempts on existing combinations, showing a stronger exploration strategy usage. However, compared to humans and ol, DeepSeek-R1 tried more failed new element combinations but not successful ones, suggesting DeepSeek-RI may explore less effectively than humans and ol. Our regression results confirm furtherly that DeepSeek-RI exhibit stronger/weaker uncertainty-driive exploration stratgies and stronger/weaker empowerment, echoing its underperformance than o1.
To conduct deeper investigation, we also collected the model's 's reasoning process along with the experiment, Here, we use qualitative analysis of some pieces of its reasoning process. For example, in trial 200, the model discovered 48 elements. The specific reasoning process is in the Figure [
The model engages in systematic multi-stage reasoning process to propose new combination from its inventory. It begins by reviewing the inventory and cross-referencing prior failed attempts to avoid redundancy. The model revisits successful paths (c.g.. how "village" was created from "house + field"" and evaluates logical relationships (empowerment) between elements, such as exploring combinations involving larger structures like "continent". But the model doesn't choose combination containing "continent". It discards irrelevant or infeasible paths, such those requiring missing elements ("ash", "fruit"). After confirming the presence both "city" and "water" the inventory, it identifies that "city + water" has not been tested and hypothesizes that this combination could yield new element, such as "port" or "harbor". This iterative and logic-driven process highlights the model's ability to balance memory, deduction, and validation in problem-solving However, the shortcoming is obvious. Although there are many logical expressions, the model's reasoning process is not straightforward to its final choice, where only the last sentence in red somehow explain the underlying motivation. This could probably suggest that the model's reasoning process is somchow redundant and may need further refinement make efficient decisions.
To facilitate an in-depth understanding of the reasoning processes employed by LLMs, we defined seven labels characterizing different aspects of the reasoning steps:
state stale Articulating the primary objective or high-level strategy.
check_current current inventory: Reviewing or listing the currently available elements.
past trial analysis: Considering previous attempts (success or failure) to avoid redundancy.
17

---

Trial 200: city water -> failed
"Olaw, use. The cumment teeettry has of elemetts but need to figure out which comb nations heven't been tried yet and could recutt in new eierrene, tee start by revlning the exsting cieeants rnd peat ettempt aved repenting them.
vilaae,
greate dum geaass
eruption
dbeidarr, mad contreet metil sneke, has bid cembiration, need find coreelinn han't pean dene Lirekol posuisle rémmatiora.
net hefee,
blade gaid, cnne, exptistie, heller heaaggans
Fas allemplo
enterain:
Foreoonpl,, comb ning seementccat that see por of etn en recer rature kayhe nom dis Teel fodtn cometting eine, pait stteerper Tros se comiinse plns warth feeld Then Trist 100 hasi fuldi vato sriih labkd Meskafiiid in slou? but Triil 185 bied plos fiedd and lakel Hism
What about Vellage and heee' stlape made comberyg house as mouse, hage- Hel created parkin Trial 176 So maybe lage someth ting eirw, Wart, city sne preated by village
184
lrge
Trial
lad warth and earth eorth Ner
Locking "oruptonn which was created vokano pressure Moybe cruption be con bir with simeetinng ehe, The user tried cnustion with water, smole,, earth, lowe, dust, sind,
matel blender enc. but all bol Maybs oraption witll But lert the invertons, Watt mapee anaption suploaion Thial 159 bhse soplcaiinn anapt on and led.
what about gpantee ? Trial 150 ind stove earth get land, wilck reody exits
let thenc aeriiten which
stone Triel 14l tre setttoone onter and Maybe aw viiteee presure The L14 tnee verditone prosssre feiled.
ghes.
matel
Mayae
locking "gmppowder and bu bet. buiter- gur
Trial 130 acl Mische it mused/ Tras wan Trial tilled
What abou. Sagur oont? blaste asserien Sall bue fisk ieth in llle elenenss Not piull
How about geyser" They red combenng geppr th boter, metel sand, crust on, etc. Maybe gesser earth? Netaure trial as itied grsser bolker and ledeed.
locting brow which matal- sarth Maybe plcw heidh Triee Tital
tailec. Flaxe
73 slow btade and led.
Welt city present Kinbs sont iina
semsihin lise boeld malc blages pallil but bricd dity park and fullod, What ubout city Triil 193 bied city wull pricc
rhack wamh innetieee hot bewn mast Earth contiertt nightnnshhe Aur Trial iity eorth whinn falled, and trial 18s conhiind bnd wirh maie contiwatt Nor Wfat clou, baccens and plau?? Trtal 130 i calgus mussrr and lired. koxtee sand "grppssder" Thtal iiedd taat tod
Wail vekarni and to sptien are seperate clumeetts Whla Exonniiingg degiiiiggnnnn vokana with someetingg ny che? They trid vokano with wi, water metal slust, snolle presuure joaad couution)) eti..
Macce vocant sep on/ Trial tried that and tailed.
shar
that fut need chaecthe
sandeare
Masbee combining then cuild mais in
doe' know thita's standard combinatio
the ga
Altemetive grrrite' i "araaure"uudt Am0t-mm makorazmettira ha merbi muttmar a in the innentors Masbennti
What aboy Xhy + woter? Trai 175 ried ollage e acter and ciedd Mayie city scater? That's not in the aos orte nuts Letmme check The usen's past atteraps to Trial Bis U99 suctt
meves water baanh bean lnsd Lat chadu Wee lim 1/5 vhlage water ked) trail park water He keds water could port buti tity. comminiig city
truing
Figure 11: The reasoning process of DeepSeek-R1 in Trial 200. The model explores possible combinations (blue color part) for discovering a new element in the game by systematically reviewing inventory elements (orange color part), prior attempts (blue color part), and logical inferences to make the decision (red color part). It also found some more empowerment elements (green color part), but didn't 't choose them.
element_ clement_property propertyy_rasooning reasoning: Analyzing inherent properties or characteristies of elements to justify potential combinations.
combination anallsis:: Evaluating specific element pairs for possible combinations.
outcome prediccteriition Predicting potential results or new elements from the proposed combinations.
final_choice: Making a definitive selection of an element combination to test.
To better illustrate the structural differences in reasoning between DeepSeek-RI and GPT-4o, we analyzed the transition probabilities between reasoning labels. The transition heatmap comparison (Figure (2) demonstrates significant distinctions. DeepSeek-RI exhibits an iterative reasoning process characterized by repeated cycles of combination_analysii and past_Irial_ _ann_lsiss reflecting me- thodical and exhaustive exploration strategy. Conversely, GPT-4o shows a markedly different pattern, progressing swiftly towards final choice. This distinction suggests that reasoning in Deepseek-RI R is more self reffereflective showcasing knowing search strategy, "backtracking" [50]. while GPT-4o a more straightforward thinking without all the patterns involved each time. This further reveals that traditional LLLMs may not think as deeply and as hard as reasoning models, and likely explain their underperformance compared to DeepSeek-RR 1 and human participants, suggesting that GPT-4o's decisions might suffer from insufficient exploratory depth.
C.3 Open-Source Model Qwen models
To further validate the generalizability of our findings beyond LLaMA- and GPT-style architectures, we extended our experiments to the Qwen series an open-source family of models widely used for reasoning and reinforcement learning research. Specifically, we evaluated Qwen-2.5-32B [42 and QwQ-32B(43) [43), a wwwooninn-enhancee variant of Qwen-2.5-32B fine-tuned with reinforcement learn ing strengthen multi-step reasoning.
Our results confirm that the core findings of this paper generalize across architectures. Qwen-2.5-32B discovered an average of 14 new elements, performing below the human average (42), whereas
18

---

Transition Probability Heatmap Comparison
GPT-4o
DeepSeek-R1
1.0
0.12
0.47
0.22
0.20
1.00
state
0.04
0.24
0.24
0.12
0.12
0.24
0.26
0.14
0.19
0.35
3.04
0.02
0.8
chock cumrent inventory
Tomctionn Probubility
past_ brialllllaae annalysi
0.06
0.01
0.27
0.15
0.10
0.11
0.04
0.05
0.09
0.50
0.18
0.07
0.8
element, property reasoning
0.02
0.11
combination, analysis
0.04
0.38
0.43
0.00
0.31
0.25
0.31
0.50
1.00
0.05
0.09
0.21
0.13
0.49
0.04
0.50
0.2
challcholcc
0.08
0.24
end
chect current inventory
begin
stte goal
element property ressening
past tridl anatviis
comtination, analyois
autcame prediction
Brnal, ctoik
end
check corrent, inventary
begin
state poal
element, property reaarring
pant trial snalysis
combination, aralytis
autcome predictioo
froat choice
end
Figure 12: Comparison of Transition Probabilities between DeepSeek-RR and GPT-4o. Transition matrices indicate that DeepSeek engages extensively in iterative cycles of combination analysis and past trial analysis, reflecting a detailed exploration strategy. Conversely, GPT-4o rapidly progresses towards the final choice, showing limited iterative reasoning steps and minimal diversity in transitional paths, indicative of shallow exploration.
QwQ-32B discovered 48 new elements, outperforming humans. This demonstrates that explicit reasoning training substantially enhances open-ended exploration.
Consistent with our earlier analyses, we computed empowerment and uncertainty regression weights for both models. The QwQ-32B model exhibited a markedly higher empowerment weight (0.0886) and uncertainty weight (2.75) compared Qwen-2.5-32D (empowerment
-0.0178, uncertainty = -1355,, aligning with its superior exploratory behavior (see Table J. These values suggest that reasoning-enhaneed enhanced QwQ better integrates empowerment-deeeee driven strategies, balancing novelty seeking with uncertainty-drist driven exploration.
In line with central claim, these results highlight that reasoning capability, not merely model scale, is a decisive factor for effective exploration. Despite similar parameter counts and pretrain- ing corpora, the QwQ-32B's reinforcement-leanee reasoning processes led substantially richer discovery dynamics and higher empowerment sensitivity. This supports our broader conclusion that reasoning-traines LLMs-ssch as ol, DeepSeek-RI, and QWQ-32B-demmnsttae slower, more deliberate, and ultimately more human-like exploration compared to conventional autoregressive models.
Table 1: Estimated empowerment and uncertainty regression weights across models and humans.
Model
Estimated Empowerment Estimated Uncertainty
Qwen-2.5-32B
-0.01777
-1.35386
QwQ-32B
0.08859
2.75241
ALaMA3.1188
-0.05991
-1,25665
LLaMA3.1-70B
-0.00409
-1.42584
GPT-4o
-0.16928
0.24957
DeepSeck-RI
0,14007
4.18199
ol
0.72222
4.15111
Human
0.378
0.223
19

---

C.4 Intervention Analysis on LLaMA3.1-70B Empowerment and Uncertainty Layers
To investigate the role of empowerment and uncertainty representations in 1LLaMA3.1-006 during in-context learning, we analyzed the most correlated neurons with these values and performed tar- geted interventions to assess their impact on model performance. Given that both empowerment and uncertainty were highly demanding in the task (shown by the example of o/'s superior perfor- mance), we wondered whether intervention to strengthen both representations could generate better performance. We found that interventions in the uncertainty layer, which is located relatively early the model (similar the choice layer), caused severe performance degradation, with even minor adjustments rendering LLaMA3. 1-70B unable to perform the task effectively. Conversely, enhancing the empowerment layer also could not improve performance. Even when set to an intervention factor of 1.5, which brought the model closer to the original level, other intervention values resulted in decreased performance.
We ablated the most correlated neuron (zeroing the latent activation) identified by uncertainty and empowerment values in experiments with identical settings. The regression analysis suggests that after ablating the empowerment neuron in the SAE, the model's empowerment strategy use is even smaller (Figure 14). establishing causal relationship that the neuron identified through SAE can control the model's empowerment strategy. Conversely, ablating the uncertainty neuron caused catastrophic performance drops, rendering most trials invalid (Figure 13A) and insufficient for regression analysis. This indicates that the earlier layer of uncertainty is critical for understanding task contexts and histories. Beyond ablation, we explored other intervention factors to determine whether they could enhance performance through "neuroscience" approach. Our results showed that both slightly weakening and strengthening neuron activity hurt model performance, suggesting a deeper limitation within the current infrastructure of LLMs for open open-ended exploration tasks.
A
LLaMA3.1 70B Average Inventory of Uncertainty Intervention
LLaMA3.1 1:8 Average Inventory of Empawerment Intervention
(iteeviinninnnnniion 3.05
orgower Teen veeevnntioon
crwetatryylmtevveettee 1.4)
emp manb, stenvention
urkesaats/iitaeeetiee 1.5)
ompowe mane) neereettion
15)
40
urcesoacy/iieeeettoo 1.7)
40
empower nere) nntevention
urceraaryytteeeeettoon 2.0)
empoovement:: sesssoeeeee///eeeeeeeee rterventto
Average Inventory
LL grew)
Average Inventory
1-700toriginal
30-
80
20
20-
100
500
Trial
Figure 13: A: LLaMA3.1-70B Average Inventory of Uncertainty Intervention. Set different levels of uncertainty intervention (0.0, 0.5, 0.7, 1.0, 2.0). Increasing the uncertainty intervention progressively disrupts the model's ability to complete the task, indicating the critical role of early uncertainty layers in processing task history and context. B: LLaMA3.1 Empowerment Intervention. Performance remains closer to the original level when the intervention is set to 1.5, whereas other levels of intervention result in performance degradation.
Average Inventory of
20

---

LLaMA3.1-708 Intervention Estimates
original
1.75
empowermenttineerventioo
1.50
1.25
Estimate
1.00
0.75
0.50
0.00
empowerment
uncertainty
Exploration
Figure 14: LLaMA3.1-70B Intervention Regression Results. The regression estimates for empower- ment, and uncertainty under the original condition (LLaMA3.1-70B) temperature 1), empowerment intervention (set 0), and uncertainty intervention (set 0).
D SAE Setup
We train all layers in LLaMA3.1-70B with the same set of Ityper-aarameten Those hyper-parameters are tuned to ensure reconstruction is satisfying as well as with a good sparsity representation. We set the hidden size of latent 8192, the same the dimensions of the model embeddings. We set the learning rate as le-4, with batch size of 256. The L2 norm is only 1e-6. L2 norm above this value will significantly amplify the reconstruction loss. sanity check for this parameter. Same way for LLLMA3.1-88B we set the hidden size latent as 4096, the same as the dimensions of the model embeddings. We set the learning rate le-4, with batch size of 256. The L.2 norm is only 1e-6.
A: LLaMA3.1 1-88 Training Result
8: LLaMA3.1-706 Training Result
thwr
Focemater
More
8:005
6.00
4.014
5:104
Mynt
1.00
6002
Numter
Runder
2300
1900
Figure 15: Sparse Autoencoder (SAE) Training Metrics. Each row represents different model architectures. From left right, the panels illustrate the layer-wise test L2 norm, test reconstruction loss, and the number of active neurons during training. The top row corresponds to smaller model (LLaMA3.1-8B) and the bottom row corresponds larger model (LLaMA3.1 1-B))) with more layers. Test Accuracy Between Original and Reconstructed Data. In both cases, reconstructed data achieves higher accuracy across layers, demonstrating SAE's ability to preserve essential features during encoding and reconstruction.
21

---

E Replicated SAE Result in LLaMA3.1-8B
We investigated the role of the empowerment and uncertainty layers in LLaMA3.1-88 (temperature 1) by training a Sparse Autoencoder (SAE) to identify and interpret their activations, followed by targeted interventions where each layer's activation was set zero. The results, summarized in Figure [6A and Figure I6B, show that setting the empowerment layer to zero had minimal effect on regression estimates and only slightly reduced model performance, suggesting that the empowerment layer has limited role in sustaining task performance. In contrast, setting the uncertainty layer to zero led to substantial reduction regression estimates for uncertainty, accompanied by marked decline in model performance. This highlights the critical importance of the uncertainty layer in facilitating exploration and maintaining robust decision- decision-making capabilities within the model.
A
8.4
Charrs
B
LLaNA3.1-88 Intervention Estimates
30
LLaMA3. 1-80 Average Inventory interventions
2:50
original
LLMMA 2 bra weeeet Unceeniity
manowermant, intervantion
uncertainty_
1.0-
15 Soooo 1000000
Eztinate
-150
110
empowerment
uncertainty
102
300
500
Expicration
Tral
Figure 16: A: SAE Correlation Analysis. Maximum correlation of uncertainty values across layers. peaking at layer 7. Maximum correlation of empowerment values across layers, peaking at layer 22. Maximum beta weight of choices across layers, peaking at layer 15. B: LLaMA3.1-8B Intervention Regression Results. The bars represent the regression estimates for empowerment. and uncertainty under the original condition (LLaMAA31 1-8B, temperature = 1), empowerment intervention (set to zero), and uncertainty intervention (set to zero). C: I.LaMA3.188B Average Inventory of Interventions. Uncertainty intervention leads significant reduction the average inventory, indicating its essential role in model performance.
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contribution of this paper is to establish a human- computational cognitive neuroscience way to understand whether LLMs explore effectively as humans in the open-ended exploration task. The paper not only evaluates a variety of models performance compared to humans, but also investigates why traditional LLMs and reasoning models can have bipolar performance, by computational modeling, thought analysis, and SAE. This paper provides in-depth understanding towards LLM'ss open-ended exploration behavior.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
23

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper diseuss the limitation of the paper that still lack deeper understanding of why traditional inference paradigms fail in open-ended exploration task, well as how could potentially improve the capacity through experimentation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does involve theoretical proof to certain concepts or conclusions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
24

---

Answer: [Yes]
Justification: We provided complete information about the task, data, and code in the anonymous supplementary files for reproduction by reviewers and readers.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification; All analysis sis codes and data are provided in the submission supplementary files and will be open-sourced upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https! for more details.
//nips //npps.ccppubbll wttp:ccpppppccccctt.tt..cc.cccct http://uulcccguudeeeeooeeeeeeeee /ttp://iddee/oddddaaaaaaiioooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
25

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: provided the experiment setting, task ground truth rules, and analysis details in the paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The error bars in the plots are standard error means. All the comparisons supported by appropriate statistical analysis approach, with rationale corrections.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
26

---

Justification:
We report API usage and compute power consumption for every method we
used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ttttps://neurppan http:ppp.....pbllacccccccccccna.
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is primarily an evaluation- work and does not involve a societal impact perspective.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
27

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not poses any such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The licence for the materials is protected by CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cww com/datasets: has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer:
[NA]
Justification:
This paper evaluates and investigates existing models and does not produce assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
28

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does use human data, but from a previously published and open- sourced literature, which has been properly cited and mentioned.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used core methodology,
only for writing, editing, formatting purposes and does not impact scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: this paper uses LLMs as the core content the research, as well as assisting in implementing codes, editing drafts, and formatting for the texts under guidance. The originality, design, and conceptualization
study not involve any LLM usage. The assisted part is finally reviewed by all the authors ensure the quality.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM http::/e//eerrreeocc//////eerree for what should or should not be described.
29


Reviews:
[{"review_id": "IqVzT4yhEq", "reviewer": "Reviewer_BscT", "summary": "This submission studies the exploration abilities of LLMs in open-ended tasks using the game Little Alchemy 2, a combinatorial task where the goal is to combine existing elements in order to discover as many new elements as possible. The authors evaluate the perfomance of several LLMs, specifically GPT-4o, GPT-o1, DeepSeek-R1, LLaMA3.1-8B, and LLaMA3.1-70B, on this task and compare their performance to human players. They find that the LLMs underperform on this task relative to humans, with the exception of GPT-o1 and DeepSeek-R1. \n\nThe authors distinguish between uncertainty-driven exploration, a common paradigm in RL and bandits where the learner takes acions favoring novelty, and empowerment-based exploration, a concept from human learning that favors taking actions (in this case, combinations of elements) thay lead to more downstream possibilities. They show that in LLaMA, uncertainty signals dominate early transformer layers and heavily influence decisions, while empowerment signals arise later in the network and are often unused. Thus, LLMs “thinking too fast” is one explanation for why most models achieve sub-human level performance on this task.", "questions": "How does uncertainty-based exploration compare to empowerment-based exploration? Are there settings in which one is useful but not the other? \n\nWhat other types of tasks (besides Little Alchemy 2) do you believe your findings generalize to?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n\nThis paper adds to the quickly growing literature on using LLMs for exploration in decision making tasks. While the authors are not the first to look at such questions, they are the first (to my knowledge) to take a human learning-based perspective, as previous work has predominantly focused on uncertainty-driven explanation. The use of Little Alchemy 2 provides a natural task to test exploration abilities of large models, and the authors’ methodology for measuring empowerment is well-grounded in prior work. \n\nI particularly liked the authors’ use of sparse autoencoders fo rrepresentational analysis, which led to some interesting finding (see above in summary). \n\nMore generally, I thought that the experiments were well thought out, and the results made sense and are reproducible. I also found the paper to be well-written. \n\n\nWeaknesses: \n\nThe parts of the paper that were most interesting to me were the claims about models “thinking too fast”. However, it would have been nice if more time was dedicated to formalizing this notion a bit more precisely. \n\nWhile the empowerment metric is well-motivated, the reliance on an external neural network to compute empowerment scores (as opposed to using some sort of ground-truth signal) is somewhat undesirable. \n\nI think that this paper could benefit from a more thorough comparision with the literature on using LLMs to make decisions under uncertainty. While I believe the authors that most of this work has focused on uncertainty-based exploration, there is still enough overlap in some of these works that they are worth mentioning and comparing against. I've included an (incomplete) list of such papers below, for your reference.\n\nToward Efficient Exploration by Large Language Model Agents by Dilip Arumugam and Thomas L. Griffiths \n\nCAN FOUNDATION MODELS ACTIVELY GATHER INFORMATION IN INTERACTIVE ENVIRONMENTS TO TEST HYPOTHESES? by Ke et al\n\nBeyond Numeric Rewards: In-Context Dueling Bandits with LLM Agents by Xia et al\n\nLearning to Explore: An In-Context Learning Approach for Pure Exploration by Russo et al\n\nShould You Use Your Large Language Model to Explore or Exploit? by Keegan Harris and Aleksandrs Slivkins\n\nDo llm agents have regret? a case study in online learning and games by Park et al", "was_revised": true, "final_justification": "I believe this paper clears the bar for NeurIPS due to the authors' fresh take on the problem of using LLMs for exploration in decision making tasks. While the authors are not the first to look at such questions, they are the first (to my knowledge) to take a human learning-based perspective, as previous work has predominantly focused on uncertainty-driven explanation. The use of Little Alchemy 2 provides a natural task to test exploration abilities of large models, and the authors’ methodology for measuring empowerment is well-grounded in prior work.", "rebuttal": "We thank the reviewer for the insightful feedback and constructive suggestions. The comments are guiding us toward better clarification and reflection on our contribution.\n\n---\n\n### On the use of a neural network instead of ground-truth for empowerment\n\n> “While the empowerment metric is well-motivated, the reliance on an external neural network to compute empowerment scores (as opposed to using some sort of ground-truth signal) is somewhat undesirable.”\n\nWe appreciate the reviewer’s observation on this detail. Indeed, in our experiment—as well as in previous studies of human exploration—using ground truth would be more straightforward and simpler. However, using ground truth necessarily assumes that participants and agents already know the complete rule structure of the task environment, which is in fact the implicit goal of this task: to explore and uncover the “rules of the world” and exploit them.\n\nThe neural network is trained only on partial ground truth, using word2vec embeddings of each element, aiming to learn a latent world model from the game's observable patterns. Through its predictions, we are simulating how general semantic meaning can be used to estimate empowerment in this task. This setup better reflects how a cognitive agent might operate under uncertainty with incomplete knowledge.\n\n---\n\n### On the concept of “thinking too fast” and its formalization\n\n> “The parts of the paper that were most interesting to me were the claims about models ‘thinking too fast’. However, it would have been nice if more time was dedicated to formalizing this notion a bit more precisely.”\n\nWe truly agree with the reviewer’s comment here. It is indeed challenging to formally define what “thinking too fast” means in the context of LLMs. Our intuition comes from how human thinking speed is typically measured—such as through response time in cognitive tasks. Meanwhile, when we talk about thinking in general, “fast” may imply shallow, intuitive reasoning, as discussed in Kahneman’s *Thinking, Fast and Slow* (2011).\n\nAlthough LLMs do not have response times in the same sense as humans, transformer processing follows a strict temporal structure: a single forward pass is processed sequentially from the first layer to the last. This gives rise to a notion of “early” versus “late” processing. Our SAE analysis shows that uncertainty-driven exploration is mostly represented in early layers, whereas empowerment emerges in later layers. This creates a natural analogy to fast (shallow) versus slow (deep) thinking, similar to how response time correlates with cognitive depth in humans.\n\nThis analogy is further supported by Hu et al. (2025), who demonstrate that transformer layerwise processing exhibits reaction-time-like signatures: early layers correspond to fast, low-effort responses, while deeper layers reflect more accurate, reflective reasoning—closely paralleling human response time and accuracy tradeoffs.\n\nAdditionally, our thought analysis revealed that both the computational resources consumed and the structure of the generated thoughts suggest that reasoning models spend more tokens in “thinking” and engage in more comprehensive mental exploration in the conceptual space—closely resembling System 2 processing. This is also consistent with findings by de Varda et al. (2025), who show that reasoning-capable LLMs incur cognitive costs analogous to humans when engaging in deeper reasoning.\n\nTherefore, while our current notion of “thinking too fast” is not yet formally defined, our analogy—grounded in cognitive theory and supported by computational evidence—offers a meaningful and interpretable perspective. We will take this seriously in future work and aim to develop a more systematic, formal framework to quantify this phenomenon in LLMs.\n\n---\n\n### On related literature regarding uncertainty-based decision-making\n\n> “I think that this paper could benefit from a more thorough comparison with the literature on using LLMs to make decisions under uncertainty.”\n\nThank you for the supplemented references. We recognize that some of these works are highly relevant and valuable. We would be happy to include them in the related work and discussion sections to strengthen our literature background in the next revision. These works help situate our approach relative to uncertainty-based and in-context exploration strategies, which we complement by highlighting empowerment as an additional, semantically grounded principle of exploration.\n\n---\n\n### On the comparison between uncertainty-based and empowerment-based exploration\n\n> “How does uncertainty-based exploration compare to empowerment-based exploration? Are there settings in which one is useful but not the other?”\n\nUncertainty-driven exploration in this task specifically means agents look back at their history and determine which elements have been selected more and which less. Elements selected less frequently may contain more unknowns and therefore higher uncertainty about their utility in the task, which draws the agent's attention.\n\nEmpowerment, by contrast, is more semantically driven. It involves estimating, based on the meaning of the words (concepts), how fruitful a given element could be in generating new combinations. This potentially leads to a higher chance of discovery. Thus, empowerment requires a deeper understanding of the world—effectively a learned world model.\n\nAs shown in the Appendix A, once more elements are discovered (meaning the decision space is expanding), relying solely on uncertainty becomes less effective: with so many possibilities, many of them unproductive, sampling under uncertainty alone is inefficient. To explore effectively, an agent must use its semantic knowledge and internal world model to selectively pick out the elements most worth trying, which demands deeper reasoning. Therefore, this task naturally favors empowerment-driven over uncertainty-driven exploration, especially within a limited trial budget.\n\n---\n\n### On generalization to other task domains\n\n> “What other types of tasks (besides Little Alchemy 2) do you believe your findings generalize to?”\n\nThe first and most direct type is compositional tasks. Since this task involves compositionality, learning, and planning, we believe the heuristics identified here in both humans and AI models are widely used in other compositional domains as well.\n\nMore generally, our findings should apply to tasks that require internal world models and prior knowledge, rather than learning entirely from scratch. This Little Alchemy 2 task implicitly requires such prior knowledge: for instance, if an agent does not understand the meaning of “water” or “fire” in real-world terms, it becomes nearly impossible to simulate the outcome of combining them (e.g., creating “steam”) in its internal reasoning.\n\nThese kinds of tasks are harder to measure but are increasingly receiving attention. The need for prior semantic grounding and compositional generalization is a common thread across many such domains.\n\n---\n\n**References**\n\n- Hu, J., Lepori, M. A., & Franke, M. (2025). *Signatures of human-like processing in Transformer forward passes*. arXiv preprint arXiv:2504.14107.  \n- Kahneman, D. (2011). *Thinking, fast and slow*. Macmillan.  \n- de Varda, A., D’Elia, F. P., Fedorenko, E., & Lampinen, A. (2025, July 24). *The cost of thinking is similar between large reasoning models and humans*. OSF Preprints. https://doi.org/10.31234/osf.io/m2cu5_v1"}, {"review_id": "eXYGTMmZQZ", "reviewer": "Reviewer_Envx", "summary": "The authors investigate how well the current state-of-the-art LLMs (LLaMa, GPT, o1, DeepSeek) perform in an open-ended, novel discovery task when compared to humans. Their extensive experiment shows that, when compared to humans, LLaMa models significantly underperform, GPT4 performs somewhat comparably but doesn't excel, and DeepSeek and o1 surpass human level performance. The authors attribute the underperformance of many LLMs to their heavy reliance on uncertainty driven heuristics and rash decisions, rather than incorporating empowerment, or \"the intrinsic desire to create elements that offer many more\" future discoveries. Their deep-dive with regression and SAE shows that uncertainty-based processing happens in early transformer layers, while empowerment signals only emerge later—by which time the model has often already made a premature decision.", "questions": "1. In section 4.1, the reported results show that DeepSeek also vastly outperformed humans in Little Alchemy 2. Is there a reason why o1 is highlighted, and less focus is put on the achievement of DeepSeek? Later in 4.3, I see that the comparison is made between GPT and DeepSeek, not with o1. I presume that this deep dive in 4.3 was easier with DeepSeek than with o1 because the former is open source while the latter is proprietary. Given this, wouldn’t it have been helpful for consistency and completeness to also emphasize DeepSeek’s strong performance more prominently in section 4.1?\n2. In section 4.4, the authors provide a detailed analysis of exploration behavior and decision dynamics specifically for LLaMA models. Is there a particular reason why a similar in-depth examination wasn’t conducted for GPT or DeepSeek models? I think it is critical that the authors demonstrate whether the patterns observed with LLaMA generalize across other model families.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "It is no trivial work to perform such a comprehensive study across multiple LLMs, and to conduct an impressively large scale human experiment for providing a robust and meaningful benchmark. The paper also offers a novel diagnosis that LLMs tend to \"think too fast\" to explain why they often underperform on open-ended, exploratory tasks than humans. Section 4, in particular, is well written: it starts with a clear summary of results that sets the town for the rest of the section, then methodically delves into the mechanisms behind why LLMs exhibit premature decision-making behavior. \n\nMeanwhile, I believe the authors could've made more level comparisons between different family of models. The strongest empirical results hinges on o1, whose technical details are unknown. DeepSeek, a strong runner-up with impressive performance and transparency, deserved greater emphasis for better interpretability and reproducibility. The fact that different model families in separate sections made it somewhat challenging to grasp how the authors' conclusions broadly apply across all LLMs.", "was_revised": true, "final_justification": "I thank the authors for their detailed clarifications and responses to my concerns. After some consideration, I decided to keep my rating of 4, because I still believe that a more level comparisons between different family of models is critical for the completion of the authors' reaserch.", "rebuttal": "We thank the reviewer for the insightful reviews and constructive feedback. Below we address your main concerns and questions in detail.\n\n---\n\n### On the inconsistent emphasis on o1 and DeepSeek-R1 across analyses\n\n> “In section 4.1, the reported results show that DeepSeek also vastly outperformed humans in Little Alchemy 2. Is there a reason why o1 is highlighted, and less focus is put on the achievement of DeepSeek? Later in 4.3, I see that the comparison is made between GPT and DeepSeek, not with o1. I presume that this deep dive in 4.3 was easier with DeepSeek than with o1 because the former is open source while the latter is proprietary. Given this, wouldn’t it have been helpful for consistency and completeness to also emphasize DeepSeek’s strong performance more prominently in section 4.1?”\n\nThank you for pointing this out—we realize that the current presentation may have created confusion for the reader. When conducting a mechanistic analysis, we must ensure the models are transparent and feasible to deploy for the required analyses. As you correctly noted, while o1 demonstrates strong performance, its training details and model architecture are not publicly available. In contrast, DeepSeek-R1 is open-sourced and allows deeper analysis of its internal behavior.\n\nTherefore, although we highlighted o1's top performance in Section 4.1, the chain-of-thought outputs and internal activations that we could analyze in detail were only available for DeepSeek-R1. This discrepancy resulted in some imbalance across sections. We appreciate your suggestion and, to improve clarity and consistency, we will revise the paper to give more consistent emphasis to DeepSeek-R1’s performance and include more explicit justification for why certain models were selected for in-depth analysis. We will still report o1’s results due to their significance but shift the highlight to DeepSeek-R1 to better align with the scope of our analyses. Thank you again for your helpful comment.\n\n---\n\n### On why SAE analysis was only conducted for LLaMA models\n\n> “In section 4.4, the authors provide a detailed analysis of exploration behavior and decision dynamics specifically for LLaMA models. Is there a particular reason why a similar in-depth examination wasn’t conducted for GPT or DeepSeek models?”\n\nThis is a great point, and the reason is closely related to the one above. Our SAE analysis requires access to the complete hidden activations of all transformer layers to train a separate sparse autoencoder model. This analysis is only feasible on models that are fully open-sourced and reasonably deployable.\n\nGPT-4o and o1 are not open-source, so we do not have access to their internal activations. While DeepSeek-R1 is open-source, the model has more than 600 billion parameters, which makes it extremely challenging to deploy for this purpose. Obtaining full layer activations across multiple inputs would require substantial hardware—far beyond our computational resources. For reference, the activation size would be at least several times larger than that of LLaMA3.1-70B, which was already the upper limit of what we could analyze.\n\nGiven these constraints, we chose LLaMA models for this layerwise analysis because they strike a practical balance between openness, performance, and computational accessibility. We hope this helps clarify the decision.\n\n---\n\n### On potential alternative models for future analysis\n\nWhile we could not analyze DeepSeek-R1 with SAE at the time of this study, we acknowledge that the availability of more lightweight, transparent reasoning models has improved. In particular, newer models like the Qwen series and its reasoning-optimized variant QwQ-32B are promising. QwQ is trained via reinforcement learning on top of Qwen-2.5-32B to enhance reasoning capabilities and is released with sufficient transparency for deeper investigation.\n\nIf we had access to these models earlier, we would have incorporated parallel SAE and thought-level analyses between Qwen-2.5-32B and QwQ-32B. We refer you to our response to Reviewer cuDM for a discussion of results using these models. We hope this gives a more straightforward illustration of the difference between traditional LLMs and reasoning-enhanced models in both reasoning and exploration.\n\n---\n\n### On formatting and grammar concerns\n\n> “No major formatting concerns, but a few grammatical errors and stylistic nit: [...]”\n\nThank you for the careful reading and suggestions regarding grammar and clarity. We will correct the specific sentence fragments, typos (e.g., “discouvered”), and ambiguous uses of pronouns (e.g., replacing “this” with more specific noun phrases). We will also review the manuscript more broadly to fix similar issues throughout. We appreciate your attention to detail and will make sure the final version reflects these improvements.\n\n---"}, {"review_id": "XxprqDTFGr", "reviewer": "Reviewer_cuDM", "summary": "This paper investigates the ability of large language models (LLMs) to perform open-ended exploration, a critical cognitive capability for discovering new information and adapting to novel environments. Using the combinatorial puzzle game LittleAlchemy 2 , the authors compare how LLMs and humans explore new combinations of elements to create new ones. They find that most traditional LLMs underperform compared to humans in this task, primarily relying on uncertainty-driven strategies rather than empowerment-based reasoning—where actions are taken to maximize future possibilities.", "questions": "1. How do the findings apply to open-ended tasks beyond combinatorial games (e.g., autonomous scientific research or creative writing)? How generalizable are these findings beyond LittleAlchemy 2?\n\n2. Can fine-tuning or prompting enhance the model's performance on such tasks or overcome the “fast thinking” limitation?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\nThe paper studies a fundamental but understudied question: whether LLMs can engage in effective open-ended exploration akin to human cognition. The use of LittleAlchemy 2 as a benchmark for evaluating exploration strategies in LLMs is innovative and fills a gap in current evaluation frameworks. The integration of cognitive psychology concepts (empowerment, uncertainty) with deep learning analysis techniques (SAE) is unique and interdisciplinary. Its findings have implications for building more adaptive AI systems capable of discovery and innovation, especially in domains requiring long-term planning and intrinsic motivation.\n\n## Weaknesses\n\n1. It is better to demonstrate some illustrative figures for readers to understand this paper faster and clearer.\n\n2. The findings are validated in Little Alchemy 2, but it remains unclear how they generalize to other open-ended tasks. The authors could discuss generalizability challenges or conduct more experiments in other domains.\n\n3. Including more diverse models (e.g., open-source alternatives or smaller LLMs) would strengthen the robustness of conclusions.\n\n4. The SAE analysis shows when empowerment and uncertainty are represented but lacks detailed mechanistic insights into why LLMs prioritize early-layer signals. Deeper analysis of transformer dynamics could enhance clarity. Besides, the SAE is only conducted on Llama3.1-70B. Including other reasoning models like QwQ or DeepSeek-Distilled-Qwen-32B can be more convincing.", "was_revised": true, "final_justification": "I would like to raise the score from 2 to 3. I still think that there should be some experiements exploring the potential of fine-tuning in such game tasks to improve LLMs' general reasoning abilities.", "rebuttal": "We thank the reviewer for the insightful feedback and constructive suggestions. The comments are guiding us toward better clarification and reflection on our contribution.\n\n---\n\n### On the need for illustrative figures\n\n> “It is better to demonstrate some illustrative figures for readers to understand this paper faster and clearer.”\n\nWe fully agree with this suggestion. As our analyses span multiple levels, it would be beneficial to include an overview figure that summarizes the methods and findings to help readers navigate the content more efficiently. In the next revision, we plan to incorporate such a figure (e.g., a schematic pipeline) to enhance readability and support intuitive understanding of our approach.\n\n---\n\n### On generalizability beyond Little Alchemy 2\n\n> “The findings are validated in Little Alchemy 2, but it remains unclear how they generalize to other open-ended tasks. The authors could discuss generalizability challenges or conduct more experiments in other domains.”\n\nWe appreciate the reviewer’s thoughtful concern. We agree that extending the findings to other open-ended domains, such as scientific discovery or creative writing, would enhance the generalizability of our claims. However, the primary constraint is the limited availability of large-scale human data in open-ended exploration environments. Little Alchemy 2 offers a rare combination of complex combinatorial structure and large-scale behavioral data.\n\nFurthermore, our current analysis is closely tied to cognitive mechanisms unique to this task. The latent model, sparse autoencoder, and thought analyses all rely on this task’s affordances. While this limits breadth, it allows us to conduct a much deeper investigation. We view this as a deliberate trade-off between generalization and insight. We hope to explore additional tasks in future work that test the same core mechanisms.\n\n---\n\n### On model diversity and transformer dynamics\n\n> “Including more diverse models (e.g., open-source alternatives or smaller LLMs) would strengthen the robustness of conclusions.  \n> The SAE analysis shows when empowerment and uncertainty are represented but lacks detailed mechanistic insights into why LLMs prioritize early-layer signals. Deeper analysis of transformer dynamics could enhance clarity. Besides, the SAE is only conducted on Llama3.1-70B. Including other reasoning models like QwQ or DeepSeek-Distilled-Qwen-32B can be more convincing.”\n\nThank you for this suggestion. To address this concern, we implemented additional experiments with Qwen-2.5-32B and QwQ-32B. Our results show that Qwen-2.5-32B discovered 14 elements on average, which is lower than the human average of 42. In contrast, QwQ-32B discovered 48 elements on average, outperforming humans. While we did not have sufficient time during the rebuttal phase to replicate the full SAE and thought analysis for these models, the raw exploration results demonstrate that our core findings generalize beyond LLaMA and GPT-style models.\n\nSpecifically, QwQ is a benchmark reasoning model in the Qwen series, directly trained using reinforcement learning on top of Qwen-2.5-32B. These results suggest that with similar architecture and pretraining, reasoning-enhanced variants significantly improve exploration performance. This reinforces our central claim that stronger reasoning capabilities contribute to better open-ended exploration. Additionally, these results align with our earlier finding that model size positively correlates with exploration quality. Both Qwen variants’ performances are consistent with their parameter scale. We will be happy to include full SAE and thought-level analyses of these models in a future version.\n\n| Model               | Estimated Empowerment | Estimated Uncertainty |\n|---------------------|------------------------|------------------------|\n| **Qwen-2.5-32B**                | **-0.01777**               | **-1.35386**               |\n| **QwQ-32B**             | **0.088587**               | **2.752409**               |\n| LLaMA3-8B           | -0.05991               | -1.25665               |\n| LLaMA3-70B          | -0.00409               | -1.42584               |\n| GPT-4o              | -0.16928               | 0.249572               |\n| DeepSeek-R1  | 0.140074               | 4.181993               |\n| o1                  | 0.722223               | 4.151111               |\n| Human          | 0.378              | 0.223             |\n\n---\n\n### On generalization to broader open-ended domains\n\n> “How do the findings apply to open-ended tasks beyond combinatorial games (e.g., autonomous scientific research or creative writing)? How generalizable are these findings beyond LittleAlchemy 2?”\n\nWe appreciate the opportunity to further reflect on the generalizability of our study. While our current work focuses on a well-controlled yet open-ended environment, we believe the underlying insight—namely, that reasoning capacity enhances exploration—has broader relevance. We identified one concurrent work by Grams et al. (2025), which evaluated exploration in LLMs in a maze-like environment. They found that (1) exploration improves with model scale and (2) prompting for reasoning significantly enhances exploratory behavior. Both findings are consistent with our conclusions.\n\nThese results collectively point to a shared principle across domains: reasoning encourages goal-directed discovery and strategic coverage of action space. While task details differ, we believe the cognitive basis of exploration is robust and worthy of further empirical study across environments.\n\n**Reference:**  \nGrams, T., Betz, P., & Bartelt, C. (2025). Disentangling exploration of large language models by optimal exploitation. *arXiv preprint* arXiv:2501.08925.\n\n---\n\n### On enhancing exploration via fine-tuning and prompting\n\n> “Can fine-tuning or prompting enhance the model's performance on such tasks or overcome the 'fast thinking' limitation?”\n\nThank you for this excellent question. We believe fine-tuning with reasoning data, particularly through supervised or reinforcement learning, can enhance the model’s exploration ability. However, we caution against directly fine-tuning on our specific task, as this could lead to data leakage. The essence of open-ended exploration lies in learning the environment’s structure from scratch, rather than memorizing specific combinations.\n\nThat said, models trained on general reasoning tasks (e.g., o1, DeepSeek Reasoner) do generalize well to our domain. This suggests that pretraining on reasoning-related tasks enhances the model’s ability to explore unseen environments. We also tested prompt engineering with traditional models, but the task appears too complex for prompting alone to be effective. Results can be found in Appendix C.\n\n---\n\nWe once again thank the reviewer for your thoughtful and constructive comments. We hope our responses address the questions and demonstrate the strength and rigor of our study."}, {"review_id": "4ir2oBj7cb", "reviewer": "Reviewer_V3R6", "summary": "This paper focuses on measuring the reasoning and creative capacity of LLMs by analyzing its performance on the Little Alchemy 2 game, which requires semantic understanding of combining elements.  It measures the performance of various LLMs on this task, defines measures for evaluation, and outlines results.", "questions": "While an interesting concept for the paper, LittleAlchemy 2 has been released for quite some time now and it is likely that the base models used have some of LittleAlchemy 2's data trained into their models.  A simple search on any search engine yields many pages with memorized combinations… which could have biased the LLM model when presented with preloaded training towards these questions.  How can we account for tainted base models?\n\nHow sensitive is this system to prompt variation?\n\nIn \"past attempts\" prompts, what is exactly included?  Is it just the suggestion of combinations or the suggestion paired with their observed outcome?  I expect that this will impact the outcomes.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The experiments seem reasonably designed and assembled.  I appreciated the use of empowerment and uncertainty with a fixed testing regiment for the experimental design.  However, reasoning models and LLMs are two very different categories of capability and while paper delineates this, I didn't find calling out the differences between reasoning and non-reasoning models particularly useful towards the overall thesis that LLMs think too fast (especially when o1 clearly outperformed).  The observations don't feel particularly novel and align with what one would expect these systems to behave (e.g. with uncertainty).  However, it is useful to quantify these observations and experimental framework to aid future research that need to rely upon exploratory methods.  I would have appreciated more delineation based on the category of LLM (non-reasoning, reasoning) instead of individual callouts to specific models throughout the paper.\n\nI appreciated the use of the empowerment and uncertainty metrics with a fixed budget of attempts allotted to each player as an experimental design.\n\nWhile the comparison was made between DeepSeek and 4o with regards to usage of reasoning models, it would be curious to do the comparison instead against the top performing model of similar class (o1).  I really felt this part was pretty messy and should have clearly delineated the types of models by category more clearly.  In addition, be consistent throughout (it was unclear why Deepseek instead of o1 - which performed the best - was selected for a lot of the deep dive). \n\nSome control for base model being tainted with answers from Little Alchemy 2 is likely necessary given the base models were probably tainted with many answers.\n\nThe observations are intriguing, but also are intuitive.  Doesn't feel like groundbreaking research, but it is perhaps a good validation to have this research affirm what is expected.", "was_revised": true, "final_justification": "Interesting and fun paper to read that is well written.  Adjusting score based on rebuttal arguments.  I am still a bit concerned about the limited domain scope and also have concerns about generalizability.  This potential lack of generalizability means that its significance may not be super high, but I appreciate the observations this paper provides and the arguments they pursue.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions. We appreciate the opportunity to clarify and refine our contributions in response to these insightful comments.\n\n---\n\n### On delineation between reasoning and non-reasoning models\n\n> “I would have appreciated more delineation based on the category of LLM (non-reasoning, reasoning) instead of individual callouts to specific models throughout the paper.”\n\nWe appreciate this valuable suggestion. In our revision, we will add explicit labels in our figures and tables to clearly indicate which models are trained with intermediate reasoning traces (reasoning models) and which are not (non-reasoning models). This will improve clarity and provide a more principled framework for interpreting performance differences across model categories.\n\n---\n\n### On inconsistent use of reasoning model comparisons\n\n> “While the comparison was made between DeepSeek and 4o with regards to usage of reasoning models, it would be curious to do the comparison instead against the top performing model of similar class (o1). I really felt this part was pretty messy and should have clearly delineated the types of models by category more clearly. In addition, be consistent throughout (it was unclear why Deepseek instead of o1 - which performed the best - was selected for a lot of the deep dive).”\n\nThank you for highlighting this issue. We acknowledge the inconsistency in using DeepSeek-R1 instead of o1 for certain deep analyses. At the time of our study, few publicly available reasoning models allowed access to reasoning traces. DeepSeek-R1 was chosen for its interpretability and accessibility, which enabled us to conduct detailed analyses such as thought diversity, empowerment tracking, and sparse representation.\n\nHowever, we agree that o1, being the top-performing model, should be more consistently highlighted. In the revised version, we will ensure that o1’s performance is included clearly in the results sections, while still using DeepSeek-R1 as the representative for thought-trace-level analysis due to its unique visibility into intermediate reasoning. We will also better delineate reasoning vs. non-reasoning categories throughout to prevent confusion.\n\n---\n\n### On contamination of base models with Little Alchemy 2 data\n\n> “LittleAlchemy 2 has been released for quite some time now and it is likely that the base models used have some of LittleAlchemy 2's data trained into their models... How can we account for tainted base models?”\n\nWe acknowledge this concern and appreciate the opportunity to clarify. While it is possible that base models may have encountered some Little Alchemy 2 content during pretraining, we argue that contamination alone does not explain the observed behaviors for several reasons.\n\nFirst, model actions are constrained by the available inventory. Even if a model memorized a high-value combination (e.g., “human + metal”), it cannot execute it unless both elements are already present, which requires planning and multi-step reasoning.\n\nSecond, successful gameplay in our task involves constructing long chains of dependent combinations, often exceeding 70 elements and 100+ steps. These long-horizon plans go beyond the retrieval of isolated facts and require flexible composition and foresight.\n\nThird, if contamination were the key driver, we would expect traditional large models (e.g., GPT-4o, LLaMA) to achieve similar performance. However, only reasoning-enhanced models (e.g., o1, DeepSeek-R1) succeed significantly, and only when they exhibit distinct strategic behaviors such as empowerment-driven exploration and thought diversity. This suggests that architecture and training objective, not mere memorization, are crucial.\n\nFurthermore, recent work by Jin et al. (2025) has shown a dissociation between factual memory and reasoning ability in large language models. LLMs may recall benchmark facts with high accuracy yet still fail to use them effectively in problem-solving, reinforcing our view that knowledge alone is insufficient.\n\n**Reference:**  \nJin, M., Luo, W., Cheng, S., Wang, X., Hua, W., Tang, R., ... & Zhang, Y. (2024). Disentangling memory and reasoning ability in large language models. *arXiv preprint* arXiv:2411.13504.\n\n---\n\n### On sensitivity to prompt variation\n\n> “How sensitive is this system to prompt variation?”\n\nWe conducted prompt ablation and modification experiments, including variants with Chain-of-Thought prompting, explicit strategic guidance, and self-reflection (see Appendix C). These interventions did not significantly improve exploration performance. For example, the average inventory size remained around 43 (t = –1.17, p = 0.304) across prompts, suggesting that the system’s limitations are not primarily due to surface-level prompt phrasing.\n\nThis insensitivity supports our hypothesis that deeper architectural factors—such as token-level autoregression and shallow context integration—underlie the fast-thinking bias in standard LLMs. In contrast, models trained with reasoning traces or fine-tuned on multi-step planning objectives show more robust improvement, suggesting that structural intervention is more effective than prompt engineering in this task.\n\n---\n\n### On the content of “past attempts” prompts\n\n> “In ‘past attempts’ prompts, what is exactly included? Is it just the suggestion of combinations or the suggestion paired with their observed outcome? I expect that this will impact the outcomes.”\n\nThank you for this clarification request. The “past attempts” prompt includes both the attempted combinations and their corresponding outcomes. Specifically, it details whether each trial succeeded or failed, and in case of success, the resulting element. For example:\n\nTrial 0: water + fire → Success: 1, Result: steam.  \nYou successfully created a new element.  \nTrial 1: steam + air → Success: 1, Result: mist.  \nYou successfully created a new element.  \nTrial 2: water + earth → Success: 1, Result: mud.  \nYou successfully created a new element.  \nTrial 3: mist + steam → Success: 0, Result: -1.  \nCombining these two elements failed.  \n\nThis structure is designed to mirror the feedback available to human players in the real game. We agree with the reviewer that this information significantly affects the model’s behavior. Models that fail to incorporate failure feedback often repeat ineffective combinations, while models that learn from both successes and failures are more strategic and efficient in their exploration.\n\nWe will clarify this aspect of the prompt in the final manuscript and explain its role in supporting learning from trial history.\n\n---\n\nWe again thank the reviewer for your thoughtful and constructive feedback. We hope our responses clarify our contributions and demonstrate the rigor of our experimental design and interpretation."}]

Author rebuttal:

Meta-review: This paper studies the question of whether current LLMs can explore effectively in open-ended settings. Focusing on the Little Alchemy 2 task, the authors find that standard models underperform human baselines, while reasoning-enhanced models (e.g., o1, DeepSeek-R1) are competitive or superior. To understand this, the authors distinguish between "uncertainty-driven" and "empowerment-driven strategies", and perform an analysis based on SAEs which suggests that uncertainty is represented in early blocks while empowerment emerges in later blocks, leading to “thinking too fast” and premature choices.

Reviewers generally felt that the paper addresses a timely and interesting problem, and is well written. Many of the initial issues raised by the reviewers were addressed in the discussion period (e.g. corroborating results (Qwen-2.5-32B vs. QwQ-32B) that align with the central claim, clarified prompt design/insensitivity). While the question of expanding to domains beyond Little Alchemy 2 remains for future work (the main concern remaining after discussion), I believe the central contribution clears the bar for NeurIPS. For the camera ready, I encourage the authors to include the interesting many points that came up in discussions with reviewers.
Decision: Accept (poster)