Title: Large Language Models Think Too Fast To Explore Effectively
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Exploration

Empowerment

Uncertainty

Reasoning
Abstract: Large Language Models (LLMs) have emerged with many intellectual capacities. While numerous benchmarks assess their intelligence, limited attention has been given to their ability to explore—an essential capacity for discovering new information and adapting to novel environments in both natural and artificial systems. The extent to which LLMs can effectively explore, particularly in open-ended tasks, remains unclear. This study investigates whether LLMs can surpass humans in exploration during an open-ended task, using Little Alchemy 2 as a paradigm, where agents combine elements to discover new ones. Results show most LLMs underperform compared to humans, except for the o1 model, with those traditional LLMs relying primarily on uncertainty-driven strategies, unlike humans who balance uncertainty and empowerment. Results indicate that traditional reasoning-focused LLMs, such as GPT-4o, exhibit a significantly faster and less detailed reasoning process, limiting their exploratory performance. In contrast, the DeepSeek reasoning model demonstrates prolonged, iterative thought processes marked by repetitive analysis of combinations and past trials, reflecting a more thorough and human-like exploration strategy. Representational analysis of the models with Sparse Autoencoders (SAE) revealed that uncertainty and choices are represented at earlier transformer blocks, while empowerment values are processed later, causing LLMs to think too fast and make premature decisions, hindering effective exploration. These findings shed light on the limitations of LLM exploration and suggest directions for improving their adaptability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on measuring the reasoning and creative capacity of LLMs by analyzing its performance on the Little Alchemy 2 game, which requires semantic understanding of combining elements.  It measures the performance of various LLMs on this task, defines measures for evaluation, and outlines results.\n\nSTRENGTHS AND WEAKNESSES:\nThe experiments seem reasonably designed and assembled.  I appreciated the use of empowerment and uncertainty with a fixed testing regiment for the experimental design.  However, reasoning models and LLMs are two very different categories of capability and while paper delineates this, I didn't find calling out the differences between reasoning and non-reasoning models particularly useful towards the overall thesis that LLMs think too fast (especially when o1 clearly outperformed).  The observations don't feel particularly novel and align with what one would expect these systems to behave (e.g. with uncertainty).  However, it is useful to quantify these observations and experimental framework to aid future research that need to rely upon exploratory methods.  I would have appreciated more delineation based on the category of LLM (non-reasoning, reasoning) instead of individual callouts to specific models throughout the paper.\n\nI appreciated the use of the empowerment and uncertainty metrics with a fixed budget of attempts allotted to each player as an experimental design.\n\nWhile the comparison was made between DeepSeek and 4o with regards to usage of reasoning models, it would be curious to do the comparison instead against the top performing model of similar class (o1).  I really felt this part was pretty messy and should have clearly delineated the types of models by category more clearly.  In addition, be consistent throughout (it was unclear why Deepseek instead of o1 - which performed the best - was selected for a lot of the deep dive). \n\nSome control for base model being tainted with answers from Little Alchemy 2 is likely necessary given the base models were probably tainted with many answers.\n\nThe observations are intriguing, but also are intuitive.  Doesn't feel like groundbreaking research, but it is perhaps a good validation to have this research affirm what is expected.\n\nQUESTIONS:\nWhile an interesting concept for the paper, LittleAlchemy 2 has been released for quite some time now and it is likely that the base models used have some of LittleAlchemy 2's data trained into their models.  A simple search on any search engine yields many pages with memorized combinations… which could have biased the LLM model when presented with preloaded training towards these questions.  How can we account for tainted base models?\n\nHow sensitive is this system to prompt variation?\n\nIn \"past attempts\" prompts, what is exactly included?  Is it just the suggestion of combinations or the suggestion paired with their observed outcome?  I expect that this will impact the outcomes.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the ability of large language models (LLMs) to perform open-ended exploration, a critical cognitive capability for discovering new information and adapting to novel environments. Using the combinatorial puzzle game LittleAlchemy 2 , the authors compare how LLMs and humans explore new combinations of elements to create new ones. They find that most traditional LLMs underperform compared to humans in this task, primarily relying on uncertainty-driven strategies rather than empowerment-based reasoning—where actions are taken to maximize future possibilities.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe paper studies a fundamental but understudied question: whether LLMs can engage in effective open-ended exploration akin to human cognition. The use of LittleAlchemy 2 as a benchmark for evaluating exploration strategies in LLMs is innovative and fills a gap in current evaluation frameworks. The integration of cognitive psychology concepts (empowerment, uncertainty) with deep learning analysis techniques (SAE) is unique and interdisciplinary. Its findings have implications for building more adaptive AI systems capable of discovery and innovation, especially in domains requiring long-term planning and intrinsic motivation.\n\n## Weaknesses\n\n1. It is better to demonstrate some illustrative figures for readers to understand this paper faster and clearer.\n\n2. The findings are validated in Little Alchemy 2, but it remains unclear how they generalize to other open-ended tasks. The authors could discuss generalizability challenges or conduct more experiments in other domains.\n\n3. Including more diverse models (e.g., open-source alternatives or smaller LLMs) would strengthen the robustness of conclusions.\n\n4. The SAE analysis shows when empowerment and uncertainty are represented but lacks detailed mechanistic insights into why LLMs prioritize early-layer signals. Deeper analysis of transformer dynamics could enhance clarity. Besides, the SAE is only conducted on Llama3.1-70B. Including other reasoning models like QwQ or DeepSeek-Distilled-Qwen-32B can be more convincing.\n\nQUESTIONS:\n1. How do the findings apply to open-ended tasks beyond combinatorial games (e.g., autonomous scientific research or creative writing)? How generalizable are these findings beyond LittleAlchemy 2?\n\n2. Can fine-tuning or prompting enhance the model's performance on such tasks or overcome the “fast thinking” limitation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate how well the current state-of-the-art LLMs (LLaMa, GPT, o1, DeepSeek) perform in an open-ended, novel discovery task when compared to humans. Their extensive experiment shows that, when compared to humans, LLaMa models significantly underperform, GPT4 performs somewhat comparably but doesn't excel, and DeepSeek and o1 surpass human level performance. The authors attribute the underperformance of many LLMs to their heavy reliance on uncertainty driven heuristics and rash decisions, rather than incorporating empowerment, or \"the intrinsic desire to create elements that offer many more\" future discoveries. Their deep-dive with regression and SAE shows that uncertainty-based processing happens in early transformer layers, while empowerment signals only emerge later—by which time the model has often already made a premature decision.\n\nSTRENGTHS AND WEAKNESSES:\nIt is no trivial work to perform such a comprehensive study across multiple LLMs, and to conduct an impressively large scale human experiment for providing a robust and meaningful benchmark. The paper also offers a novel diagnosis that LLMs tend to \"think too fast\" to explain why they often underperform on open-ended, exploratory tasks than humans. Section 4, in particular, is well written: it starts with a clear summary of results that sets the town for the rest of the section, then methodically delves into the mechanisms behind why LLMs exhibit premature decision-making behavior. \n\nMeanwhile, I believe the authors could've made more level comparisons between different family of models. The strongest empirical results hinges on o1, whose technical details are unknown. DeepSeek, a strong runner-up with impressive performance and transparency, deserved greater emphasis for better interpretability and reproducibility. The fact that different model families in separate sections made it somewhat challenging to grasp how the authors' conclusions broadly apply across all LLMs.\n\nQUESTIONS:\n1. In section 4.1, the reported results show that DeepSeek also vastly outperformed humans in Little Alchemy 2. Is there a reason why o1 is highlighted, and less focus is put on the achievement of DeepSeek? Later in 4.3, I see that the comparison is made between GPT and DeepSeek, not with o1. I presume that this deep dive in 4.3 was easier with DeepSeek than with o1 because the former is open source while the latter is proprietary. Given this, wouldn’t it have been helpful for consistency and completeness to also emphasize DeepSeek’s strong performance more prominently in section 4.1?\n2. In section 4.4, the authors provide a detailed analysis of exploration behavior and decision dynamics specifically for LLaMA models. Is there a particular reason why a similar in-depth examination wasn’t conducted for GPT or DeepSeek models? I think it is critical that the authors demonstrate whether the patterns observed with LLaMA generalize across other model families.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the exploration abilities of LLMs in open-ended tasks using the game Little Alchemy 2, a combinatorial task where the goal is to combine existing elements in order to discover as many new elements as possible. The authors evaluate the perfomance of several LLMs, specifically GPT-4o, GPT-o1, DeepSeek-R1, LLaMA3.1-8B, and LLaMA3.1-70B, on this task and compare their performance to human players. They find that the LLMs underperform on this task relative to humans, with the exception of GPT-o1 and DeepSeek-R1. \n\nThe authors distinguish between uncertainty-driven exploration, a common paradigm in RL and bandits where the learner takes acions favoring novelty, and empowerment-based exploration, a concept from human learning that favors taking actions (in this case, combinations of elements) thay lead to more downstream possibilities. They show that in LLaMA, uncertainty signals dominate early transformer layers and heavily influence decisions, while empowerment signals arise later in the network and are often unused. Thus, LLMs “thinking too fast” is one explanation for why most models achieve sub-human level performance on this task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThis paper adds to the quickly growing literature on using LLMs for exploration in decision making tasks. While the authors are not the first to look at such questions, they are the first (to my knowledge) to take a human learning-based perspective, as previous work has predominantly focused on uncertainty-driven explanation. The use of Little Alchemy 2 provides a natural task to test exploration abilities of large models, and the authors’ methodology for measuring empowerment is well-grounded in prior work. \n\nI particularly liked the authors’ use of sparse autoencoders fo rrepresentational analysis, which led to some interesting finding (see above in summary). \n\nMore generally, I thought that the experiments were well thought out, and the results made sense and are reproducible. I also found the paper to be well-written. \n\n\nWeaknesses: \n\nThe parts of the paper that were most interesting to me were the claims about models “thinking too fast”. However, it would have been nice if more time was dedicated to formalizing this notion a bit more precisely. \n\nWhile the empowerment metric is well-motivated, the reliance on an external neural network to compute empowerment scores (as opposed to using some sort of ground-truth signal) is somewhat undesirable. \n\nI think that this paper could benefit from a more thorough comparision with the literature on using LLMs to make decisions under uncertainty. While I believe the authors that most of this work has focused on uncertainty-based exploration, there is still enough overlap in some of these works that they are worth mentioning and comparing against. I've included an (incomplete) list of such papers below, for your reference.\n\nToward Efficient Exploration by Large Language Model Agents by Dilip Arumugam and Thomas L. Griffiths \n\nCAN FOUNDATION MODELS ACTIVELY GATHER INFORMATION IN INTERACTIVE ENVIRONMENTS TO TEST HYPOTHESES? by Ke et al\n\nBeyond Numeric Rewards: In-Context Dueling Bandits with LLM Agents by Xia et al\n\nLearning to Explore: An In-Context Learning Approach for Pure Exploration by Russo et al\n\nShould You Use Your Large Language Model to Explore or Exploit? by Keegan Harris and Aleksandrs Slivkins\n\nDo llm agents have regret? a case study in online learning and games by Park et al\n\nQUESTIONS:\nHow does uncertainty-based exploration compare to empowerment-based exploration? Are there settings in which one is useful but not the other? \n\nWhat other types of tasks (besides Little Alchemy 2) do you believe your findings generalize to?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-24"}