Exploring exploration with foundation agents in interactive environments
Authors: Daniel P. Sawyer
Nan Rosemary Ke
Hubert Soyer
Martin Engelcke
John Reid
David P Reichert
Drew A. Hudson
Alexander Lerchner
Danilo Jimenez Rezende
Timothy P Lillicrap
Michael Curtis Mozer
Jane X Wang
Venue: Accepted by TMLR ()

Abstract: Foundation models excel at single-turn reasoning, but many real-world challenges, from scientific research to technology development, require multi-turn exploration in dynamic interactive environments. Crucial components of learning from experience in these settings, such as efficiently gathering information to test hypotheses, meta-learning a model of the world's dynamics, and adapting to unexpected changes, remain largely unexplored for these models. We first evaluate foundation models in Feature World, a setting that primarily tests information gathering about a static hidden reward function. In this initial setting, we show that state-of-the-art foundation models come close to optimal efficiency in selecting maximally informative actions in tasks with simple reward functions. As a proof of concept, we also show a model can gather information efficiently in a 3D embodied version of this task, though errors in vision limit some aspects of performance. In order to test exploration across multiple dependent turns and trials, we implement a custom, text-based version of the Alchemy environment, a benchmark designed for meta-learning. Here, agents must deduce a latent causal structure by integrating information across multiple state-dependent trials. In this more complex setting, we find that recent foundation models struggle to meta-learn strategies that enable improved performance over time. However, prompting the models to summarize their observations at regular intervals enables an emergent meta-learning process, allowing them to improve across trials. Notably, in some models, summarization also enabled adaptive re-learning of this information when the environment's rules change unexpectedly. While most models performed reasonably well on simple Feature World tasks, evaluations in Alchemy reveal stark differences in robustness among the models, with Gemini 2.5 performing best, followed by Claude 3.7, and ChatGPT-4o and o4-mini struggling the most. These results underscore Alchemy's value as a benchmark for meta-learning and strategy adaptation in foundation models. By moving beyond simple discovery to complex, stateful environments, we demonstrate that the most significant challenge for foundation agents is not selecting informative actions in the moment, but rather seeking and integrating knowledge through adaptive strategies over time. Intriguingly, we find there is likely no intrinsic barrier to future generations of foundation agents more fully mastering these abilities.

Published in Transactions on Machine Learning Research (12/2025)
Exploring exploration with foundation agents in interactive environments
Daniel P. Sawyer Google Mind
danngsaunerr@google.com
Nan Rosemary Ke Google Deep
nkedkgoogle.com com
Hubert Soyer Google Mind
soyerdkgoogle.com
Martin Engelcke
engelcke@google.com
Google Miep
John Reid Google Mind
reidjo@googles com
David P Reichert Google Mind
reichert@google.com
Drew Hudson Google Mind
doromd@ggoogle.com
Alexander Lerchner Google Deed
lerchner@@goolee.com
Danilo Jimenez Rezende Google Mind
danilor@google.com
Timothy P Lillicrap Google Deed
gountzero@google.comm
Michael Mozer
mcmozer@@gongle.com
Google Mind
Jane X Wang Google Mend
wangjane (etgiiann@ggggee.com
Reviewed on OpenReview:
Abstract
Foundation models excel at single-turn reasoning, but many real-world challenges, from scientific rescarch to technology development, require multi-turn exploration in dynamic interactive environments. Crucial components of learning from experience in these settings, such as efficiently gathering information to test hypotheses, meta-learning a model the world's dynamics, and adapting to unexpected changes, remain largely unexplored for these models. We first evaluate foundation models in Feature World, a setting that primarily tests information gathering about static hidden reward function. In this initial setting,

---

Published in Transactions on Machine Learning Research (12/2025)
show that state-of-the-art foundation models come close to optimal efficiency in selecting maximally informative actions in tasks with simple reward functions. As proof of concept, we also show a model can gather information efficiently in a 3D embodied version of this task, though errors in vision limit some aspects of performance. In order to test exploration across nultiple dependent turns and trials, we implement a custom, text-based version of the Alchemy environment, a. benchmark designed for meta-lcarning. Here, agents must deduce a latent causal structure by integrating information across multiple state-dependent trials. In this more complex setting, we find that recent foundation models struggle to meta-learn strategies that enable improved performance over time. However, prompting the models to summarize their observations at regular intervals enables an emergent meta-learning process, allowing them to improve across trials. Notably, in some models, summarization also enabled adaptive re-learning of this information when the environment's rules change unexpectedly, While most models performed reasonably well on simple Feature World tasks, evaluations in Alchemy reveal stark differences in robustness among the models, with Gemini 2.5 performing best, followed by Claude 3.7, and ChatGPT-4o and 04-mini struggling the most. These results underscore Alchemy's value benchmark for meta-learning and strategy adaptation in foundation models. By moving beyond simple discovery to complex, stateful environments, we demonstrate that the most significant. challenge for foundation agents is not selecting informative actions in the moment, but rather seeking and integrating knowledge through adaptive strategies over time. Intriguingly, we find there is likely no intrinsic barrier to future generations of foundation agents more fully mastering these abilities.
1 Introduction
Foundation models have demonstrated remarkable abilities in understanding and generating complex human- like text and multi-modal content. (Achiam et al. 2023 Gemini Team et al. 2023 Jiang et al. 2024 Reid et al., 2024 Dubey et al. 2024 Dai et al. 2024 Deitke et al. 2024). However, this success has largely been measured in static, single-turn settings where information is provided upfront. The next frontier for these models lies in their application as interactive agents, which must operate in dynamic environments where crucial information is not given, but must be actively discovered. To achieve goals in such settings, an agent cannot merely react; it must proactively explore. This contrasts with classic reinforcement learning (RL) paradigms that use undirected exploration (Burda et al.) 2018 Ecoffet et al. 2019 Badia et al., 2020). Real-world endeavors often demand more sophisticated, hypothesis-driven approach. This involves strategically formulating beliefs about the world, designing experiments to test those beliefs, and integrating findings gathered across multiple, often state-dependent trials. Such capabilities will become increasingly important as training on human- generated data reaches a limit and we enter the 'era of experience", in which models generate their own training data through interaction with their environment (Silver & Sutton 2025 Whether, and to what extent, today's 's foundation models possess this latent capacity for active exploration remains a critical and largely open question.
We evaluate LLMs in three environments: text-basod and multimodal variants of Feature World, and text-based version of Alchemy (Wang et al. 202Tp. The Feature World is largely stateless and does not necessitate extensive sequential decision-making, allowing us to isolate and analyze efficiency of information gathering. Alchemy, in contrast, demands strategic exploration and reasoning over multiple trials, which allows us to evaluate the foundation models' meta-learning and strategy adaptation abilities.
In this paper, we operationally define and measure three key capabilities involved in exploration: efficient. information gathering, meta-learning, and strategy adaptation.
Efficient information hering: Selecting actions that maximally increase expected information gain. In Feature World, we operationally define this as the success rate (Rsuceess) in finding a rewarding object within a fixed step budget B:
N
1
Rauccess
(steess; S B)
N
2

---

Published in Transactions on Machine Learning Research (12/2025)
where N is the total number of episodes, steps, is the number of steps taken to find reward in episode i, and B is set to the maximum number of steps optimal policy would need.
Meta-learning (learning learn): Improving expected performance on new tasks in given family through experience of other tasks in that family (Thrun & Pratt, [998). In Alchemy, we mcasure this the mean within-episode score improvement (Ascore) between the first trial and the average of the final trials:
1
Icooe
E TETaee
S St,1
[Tiate
where St,, is the normalized score in episode at trial i, and Thate represents the of final trials (trials 6-10 in our experiments).
Strategy adaptation: Detecting when strategy becomes invalid due to environmental changes and adapting by learning a new one. In Alchomy, we measure this as the mean performance recovered (Spost) following an uncued change to environment dynamics:
N Σ
Spost
Σ
Site
N
[post!
i=1
TE
where Tost the set of trials following the uncued change (trials 11-20 in our experiments).
For each of the above capabilities, we say a model has that capability if the difference between the measured Rauceess, Iscore, or Spost for the model and the same measured quantity for a random or heuristic baseline is statistically significant (p 0.05).
More specifically, this paper investigates the capacity of foundation models to conduct exploratory behavior within interactive environments in the zero-shot setting, using in-context prompting alone and without requiring task-specific training or fine-tuning.
We performed experiments using Gemini 1.5 Pro and Flash (Reid et al. 2024 Gemini 2.5 Pro and Flash
(Google 2025). Claude 3.7 Sonnet (Authropic 2025 and ChatGPT-4o (OpenAl 2024 and 04-mini (OpenAl 2025).
Overall, this work makes the following key contributions and findings:
We conduct extensive experiments evaluating multi-turn exploration performance of foundation models across diverse set of interactive environments. We analyzed several foundation models and range of in-context prompting strategies, including variations in the amount of prior information and the structure of demonstrations.
Our findings reveal a strong inherent exploratory capacity foundation models across simple interactive settings. Specifically, all LLMs evaluated demonstrated near-optimal performance in Feature World with simple reward functions. Likewise, some models outperformed the memoryless heuristic in Alchemy, something that the RL agents benchmarked in the original Alchemy study were unable to do.
We find that complex, multi-trial environments, such as Alchemy, foundation models struggle show meta-learning (improving over trials) and strategy adaptation (re-learning world model when the effects of actions unexpectedly change). However, both these abilities can emerge when models are prompted to summarize information across trials.
We find stark differences the robustness of meta-learning and strategy adaptation frontier LLMs using Alchemy, demonstrating the utility of this environment as benchmark for LLM exploration capabilities.
3

---

Published in Transactions on Machine Learning Research (12/2025)
2 Related Work
Exploration in RL Information gathering is related to exploration in RL, which has been studied for tasks with sparse rewards such as Montezuma's Revenge and Pitfall in Atari, Deep-sea exploration and other tasks in the Behavior Suite for RL, and the DM-HARD4 tasks (e.g., Burda et al. 2018 Ecoffet et al., 2019
Osband et al. 2019 Guo ct al. 2022 Saade ct al.
2023) as well as in unsupervised scttings (c.g.. Pathak
et al. 2017 Guo el al. 2022). These methods commonly derive an "intrinsic" reward from the error of a predictive model (e.g..
Pathak et al.
2017
Burda et al. 2018 Guo et al. 2022) or by estimating the density
of visited states (e.g., Saade al.
2023). Badia al. (2020) use combination of both of these types of
intrinsic rewards and Tam al. (2022) additionally use pro-trained representations. In contrast, this work
relies on the prior knowledge of foundation models from internet-scale pre-training for exploration (e.g., [ang
et al. 2023a Feng et al. 2023 Lu et al. 2024b) rather than using random exploratory actions and intrinsic
rewards. Also, existing RL environments (e.g,, Todorov et al. 2012 Brockman et al. 2016 Tassa et al. 2018) often conflate exploration with other aspects of agent performance, making it difficult to isolate and assess model's 's inherent. exploratory capabilities. Such aspects include sparse or deceptive rewards and noisy, non-stationary, or multi-agent environments. We therefore chose and designed a suite of environments that allows us to systematically disentangle and control the factors inftuencing exploration.
Foundation models for games Foundation models have also been used to build agents that play games (e.g.. Wang et al. 2023a b Feng et al. 2023 Tan et al. 2024), which often involves some form of exploration.
Wang et al. (2023a) show that GPT-4 can reach impressive performance in Minecraft. by incrementally building a skill library via an "automatic curriculum" stage where GPT-4 is prompted to propose novel tasks,
Fong ct al. (023) prompt an LLM to explore an environment and subscquently use the collected experiences for fine-tuning the model. Unlike Wang et al. (2023a) and Feng et al. (2023), Wang al. (2023b) and Tan
et al. (024 use image observations rather than relying access to environment internal states. All of these works, however, focus more on improving agent performance rather than performing an explicit, systematic investigation of information gathering, meta-learning, and strategy adaptation with foundation models in controlled, zero-shot settings and in comparison to known optimal policies. L.Actt benchmarks LLMs on simple games in the very long context regime and VideoGameBench tests VLMs on a collection of video games (Zhang et al.) 2025 Ruoss et al., 2024). However, neither of these works investigates exploration as a capability distinct from overall performance. The BALROG benchmark incorporates range of existing games used as RL environments and investigates exploration among number of key capabilities (Paglieri al. 2024). However, this is limited to a qualitative assessment and does not involve quantitative measurement of multiple clearly-defindd facets of exploration as this work does.
Exploration with foundation models Several other works invostigate exploration with foundation modols, e.g., for text-based environments (Lu el al. 2024b Huang et al.) 2024), reinforcement learning from human feodback (RIHF) (Dwaracherla et al. 2024), and multi-armed bandit problems (Coda- et al.] 2023 Krishnamurthy al. (024)). Unlike Krishnamurthy et al. (2024) and Dwaracherla et al. (2024) and similar to Lu et al. (2024b), this work considers stateful environments. While Lu et al. (2024b) replace components of the exploration method introduced in Ecoofft et. al. (2019) with an L.M,, this work studies the ability of foundation models to gather information and test hypotheses in-context via zero-shot prompting rather than using LLMs in a more modular fashion. Also adopting more modular approaches, Hu et al. (004)) use foundation models as components in a larger exploration framework and Huang et al. (022g) propose to use a smaller agent to explore the environment and a larger agent to leverage the gathered information.
Active learning The field of active learning (Settles) 2009) has studied how to best acquire data to improve model predictions with methods that commonly focus highly structured data (either i.i.d. or on graph). In contrast, this work explores efficient knowledge acquisition in more general interactive environments.
Active embodied question answering This work studies a similar setup to embodied question answering (EQA) (e.g.. Das et al.) 2018 Zhu et al. 2023 Majundar et al.) 2024 Ren et al. 2024). Similar to our work, agents in the EQA sotting need to actively explore an environment to gather information. Unlike our tasks, EQA typically does not involve performing iterative experiments to infer unknown mechanisms in dynamic environments, and the optimal exploratory action typically does not depend on past observations.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Strategy adaptation The dorsolateral prefrontal cortex (dIPFC) is the brain region that has undergone the most reorganization in humans relative to other primates (Donahue et al. 2018). It is involved in detecting when a previously-successful strategy no longer yields expected rewards and adjusting behavioral strategy accordingly (Mansouri et al. 2007). Analogs of this capability have been studied in LLMs in the context of mental sels (Haq et al. 2025 and learning from mistakes (Tong el al. 2024 However, these studies focus on single-turn static math and reasoning problems. To our knowledge, this the first study to investigate strategy adaptation with LLMs in the context of either exploration or interactive environments.
AI for science Hypothesis generation and testing is central to the scientific method and recent works research the application of foundation models in this broader domain (e.g., Romera-Paredes et al. 2023 Trinh et al. 2024; Lu et al. 2024a).. Such works lypically use foundation models in a highly structured protocol designed for the fixed domain in question. In contrast, DISCOVERYWORLE (Jansen et al. 2024) provides a platform with diverse, realistic tasks that test an LLM agent's ' ability complete an entire scientific workflow in a range of disciplines, and ScienceAgentBench (Chen et al., 2024) evaluates agents on data-driven discovery workflows ext extracted from peer-reviewed literature. While such benchmarks are crucial for evaluating overall task performance and progress towards automated science, our work complements this line of inquiry by shifting the focus from what tasks agents can ultimately solve to how their underlying exploratory capabilities function and fail.
summary, rather than assessing performance on a multi-faceted workflow, we use more abstract and controlled environments to systematically measure domain -general exploration capabilities. The key insights of our paper are therefore not about task completion in realistic domains, but about the mechanistic properties of foundation models themselves. To that end, we investigate efficient information gathering, meta-learning, and strategy adaptation across several frontier models. Among other insights, we reveal that these exploratory skills are often brittle, that the ability to learn a strategy does not imply an ability to adapt it, and that cognitive scaffolding via prompting can unlock latent meta-learning findings that provide more granular understanding of the fundamental challenges facing foundation agents.
3
Feature World
Shape
BBuue
Color
metal pyramid"
"Blue cube"
"Blue sphere"
"Blue pyramid"
"Blue wooden cube"
Color
"rewardbblue cube)
*Red cube"
"Red sphere"
"Red pyramid"
Shape
Material
Environment
LLM
"Green cube"
GGreen sphere"
"Green pyramid"
'Green wooden
up cube"
pyramid"
(a)
(b)
Figure 1: Task structures and experimental setups for Feature World. (a) Example task setup for text environment with single-feature reward function, with "blue" as the rewarding feature. (b) Example task setup for text environment with conjunction reward function, with "blue" and "cube" as the rewarding conjunction. (c) Schematic of text Feature World experiment setup.
We first evaluate models in Feature World: simple, memory-less, text-basod setting. In this environment, actions can be executed repeatedly without altering the underlying state dynamics due previous actions, and each exploratory action provides immediate feedback.
5

---

Published in Transactions on Machine Learning Research (12/2025)
3.1
The Feature World environments
3.1.1
Text environment
To investigate information gathering, we use a task where models are presented with objects possessing multiple features (e.g,, color, shape). A specific feature or conjunction of features determines a reward, mirroring sparso-reward RL settings.
This task isolates multi-step information gathering within a single trial. Actions provide immediate feedback a static reward function, with no latent dynamics to discover across trials.
To assess the capabilities of foundation models in this environment, we modulate task difficulty by adjusting two key aspects: the number of distinct colors (increasing the cognitive load) and the complexity of the reward function. Reward functions can be based on a single property (single-feature tasks) or a conjunction of two properties (conjunction tasks). See Figure for a visualization of the tasks.
3.1.2 Proof of concept: 3D Feature World Construction Lab
To test active exploration in a more realistic, multi-modal setting, we created a 3D version of Feature World in the Construction Lab simulation [reference-nnoymmiielll Tn this environment, the agent receives video input and outputs action instructions for a human to execute. This setup assesses exploratory behavior while introducing real-world challenges like visual understanding. To manage visual complexity, these 3D tasks use only three colors and single rewarding feature, mirroring the simplest text-based condition.
3.2 Feature World experiments and results
Our experiments in Feature World aim to address the following two questions: (a) How does the complexity of the environment affect the information gathering efficiency of foundation models? (b) What new challenges emerge in an embodied 3D version of the task?
We evaluate on foundation models of different sizes and generations, using publicly available APIs and settings. We use the default settings for the public APIs in all cases unless noted otherwise. For Feature World, we compared ChatGPT-44o (Achiam et al.) 2023 OpenAl 2024) (200k context), Claude 3.7 Sonnet (Anthropic (200k context), Gemini 1.5 Flash and Pro (Reid et al. 2024), and Gemini 2.5 Flash and Pro (Google (1M context). For all experiments, we found 200k context to be sufficient.
2025
2025
3.2.1
Task setup
Basclines We compare to two basclines: Optimal Baseline: This bascline represents upper bound exploration performance. It selects actions that maximize information gain at each step. See Section A.2.1 for more detailed description of the optimal strategy. Rondom Baseline: This baseline establishes a lower bound by choosing objects randomly with replacement. Both baselines are evaluated with 1000 episodes.
See Tables models.
and in the Appendix for the prompts used in the text version of Feature World for all
Evaluation To evaluate information gathering efficiency, we assess how often models are successful at finding rewarding object given fixed budget of exploration steps. We set the step budget as the maximum number of steps that an optimal policy would need before finding at least one rewarding object. This measures the model's active exploration capabilities independent of its ability to draw conclusions from its observations.
For the 3D exploration task, we measure two key metrics: 1) the number of steps required to gather sufficient information to identify the reward function, and 2) the model's accuracy in correctly identifying that function from its observations. This second metric assesses the model's combined ability to efficiently explore as an actor and to draw conclusions from visual evidence.

---

Published in Transactions on Machine Learning Research (12/2025)
ChatGPT-4o Claude Sonnet
Gemini 1 Flash
Gemini 2.5 Flash
Optimal Random
Gemini Pro
Gemini 2.5 Pro
successtul episodes
succesoful episedes
2.
0.8
3.8
0.7
0.7
4
8
10
@unique colors
Wunique colors
(a) Single-feature tasks
(b) Conjunction tasks
Figure 2: Fraction of Feature World episodes in which models found rewarding object before reaching the maximum number of exploration steps. (a) Single-feature reward function.
(b) Conjunction reward function.
Error bars represent standard error of the mean, with 200 episodes per condition for the models and 1000 for the random and optimal baselines.
3.2.2 Effects of environment complexity on exploration
We examine the effect of two forms of environmental complexity information gathering efficiency: reward function complexity and object quantity. To measure the former, designed tasks where reward is determined by either a single feature (like "red" or "squarc") or a conjunction of two features (c.g., "rod" and "square"). The latter requires the agent to reason about multiple properties to identify the reward-relevant combination. To investigate the impact of cognitive load on model performance, we also vary the number of unique colors in the environment.
In both fngle-feature and conjunction tasks, almost all models significantly outperform the random baseline in all conditions, with the exception of ChatGPT-4o, which is better than random when the number of colors exceeds 7 in the conjunction tasks (Figure 2 This shows that most LLMs have robust capacity for efficient information gathering.
In the single-fcature task, Claude 3.7 and both sizes of Gemini 2.5 perform identically or nearly identically to the optimal baseline, with no obvious trend of decreasing performance with increasing number of colors (Figure (.). Both sizes of Gemini 1.5 show a trend of decreasing performance with larger number of colors, but always remain close to optimal. ChatGPT-44 performance declines more quickly with number of colors, but always remains above the random baseline. This shows that, when the reward function is simple, most models are capable of nearly opti information gathering efficiency even as cognitive load increases.
In the conjunction task, most models show a large drop in performance compared to the single feature task (Figure 2p). All models except Gemini 2.5 Pro show a substantial decrease in performance as number of colors increase, while Gemini 2.5 Pro maintained strong performance despite the increase in task difficulty. This demonstrates that reward function complexity has a large impact on information gathering efficiency for most LLMs and exacerbates the impact of cognitive load on performance.
Taken together, these results show that LLMs have robust capacity for gathering information efficiently and generally achieve nearly optimal performance across cognitive loads for simple reward functions. On the other hand, most. LLMs struggle to achieve optimal performance and degrade with increasing cognitive load when the reward function is more complex. Interestingly, our results show that Gemini 2.5 Pro is an outlier in this trend, achieving nearly optimal performance across all levels of reward function complexity and cognitive load. This shows that robust, near-optimal information gathering efficiency is possible in LLMs, and may be expected to become common capability as other frontier models improve.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Optimal
We errors
Random
Number objects until suff sleficient clent Info
1.0
5-
Generalization Accuracy
VLM
0.8
4-
0.6
Environientt
LLM
KeyPresall ..>
<MMuseMMovg)
pupple conveyer
1-
0.2
0.0
(b) Steps to sufficient info
(c) Rewarding property accuracy
Figure Schematic and performance metries for 3D exploration task, with episodes per condition. (a) Mean number of exploration steps (objects placed the conveyor) before sufficient information is available to determine the correct factor. (b) Accuracy of the model in determining the correct rewarding feature. Hatched blue bar represents accuracy if episodes with vision errors are removed. Error bars represent standard error of the mean.
3.2.3 3D embodied Feature World results
As a proof of concept, we tested the multimodal and visual understanding capabilities of a model in an embodicd 3D setting. See Section A.3 for more details on the setup.
We show that Gemini 1.5 Pro is capable of extracting the necessary symbolic information from video, providing action instructions in real time a human player, and reasoning. The model's post-hoc reasoning accuracy in identifying the rewarding property was better for trajectories collected with the Gemini actor than with the random actor when trials including vision errors were removed. However, the improvement compared to the random actor in the full results that included vision errors was not statistically significant (p = 0.13, 2-sample t-test) (Figure 3 p). 8 out of 15 of the episodes with the Gemini actor contained vision errors. Overall, the steps to sufficient information results are a proof concept that a foundation model capable of efficient information gathering a 3D embodied setting requiring vision. The weaker performance due to vision errors on rewarding property accuracy suggests that multi-modal capabilities, and reasoning, are a likely bottleneck for extending exploration with foundation models to settings closer to real-world tasks.
See Section
A.3.4 for a more detailed discussion of the 3D Feature World results.
4 Alchemy
Following the Feature World experiments, we transition to more complex environment that maintains a persistent but hidden task structure across multiple trials, such that an agent must take strategic action to explore early trials to gain useful information that informs exploitative actions later trials. For this more demanding evaluation, we selected the Alchemy environment. (Wang et al., (021). Alchemy notable for its structured task distribution and its design test reasoning, planning, and, importantly, exploration and meta-learning. In Alchemy, the agent needs take actions not only discover rewards but also latent causal dynamics, which are randomly resampled every episode. Additionally, constraints are introduced such that not all actions can be taken repeatedly within a trial (which are themselves multi-step), necessitating planning both within and across trials. These characteristics contribute to more complex testbed for evaluating components of exploration strategies, such as leta-len and strategy adaptation, that are measured across multiple timescales (i.e., within trials, across trials, and across episodes).
4.1 The Alchemy environment
Alchemy (Wagg et 2021) is procedurally generated environment specifically created test meta-learning capabilities. The gameplay involves using a set of potions to transform various visually distinctive
8

---

Published in Transactions on Machine Learning Research (12/2025)
stones into more valuable forms, and then depositing them into central cauldron to score points. Stone appearance varies along three feature dimensions: size, color, and shape, and their value is visually indicated by a marker. Potion effects are determined by color. A central concept in Alchemy is the "chemistry", which represents a latent causal structure that governs the value of stone appearances and the transformative effects of potions on stones. This chemistry is procedurally resampled for each episode, meaning the specific rules linking appcarance, value, and potion effects change every time new episode begins. We define step a single action (e.g,, applying potion), trial as a sequence of steps ending with scoring resource exhaustion under fixed chemistry and finite set of objects, and an episode trials (defaulting to N = 10) where the chemistry is constant, resetting only between episodes (Figure - Within a single episode, the agent's 's implicit challenge is to diagnose the current chemistry through repeated observation and experimentation. This involves operating at two timescales: making effective choices within each trial and synthesizing the information gathered across trials to learn about the latent dynamics, applying this knowledge maximize scores subsequent trials. After cach episode, the chemistry is reset and all information from the previous episode is cleared from the model's context (with the exception of the strategy adaption experiments: see Section 4.2.3)
sequence of N
Episode 2
Trial 1: observation
Trial observation
Resample stones, potions, etc.
You see the following stones: O: blue small painty with reward +1
You see following stones:
0: blue large round with reward -1
purple large poinly with
Trial
2: blue large round with reward -1
2: purple small pointy with reward +1
Episode 1
Episode 2
You see the fallowing potions:
You see following potions:
0: turquoise
Resample chemistry
1: turquoise
red
green
2: orange
Outcome, new observation, new score
Episode 1: chemistry
Episode 2: chemistry
Prompt
Alchemy Environment
LLM
Summary
Episode History
Action
LLM
"Place stone 0 in potion 11"
Figure 4: Task structures and experimental setup for Alchemy. Upper left.: The structure of an Alchemy experiment. Upper right: Example text observations of the initial state of two separate trials from the same episode, in which stones and potions are resampled but the effects of potions and the reward values of stones remain the same. Lower left: Example chemistries, represented as graphs determining the effects of potions (edges) on stones of different properties (nodes), that change between episodes. Lower right: Directed exploration setup in which an LLM receives feedback from the environment, information from a prompt, past history of the episode, and, optionally, summary of the episode history. The two left panels are adapted, with permission, from figures in Wang et al. 22021)
4.2 Alchemy experiments and results
Our experiments in Alchemy aim to address the following questions: (a) How does the multi-trial setting, which requires long-context and memory, impact the exploration performance and meta-learning of foundation models? (b) How do different prompting strategies and cross-trial summarization methods impact exploration
9

---

Published in Transactions on Machine Learning Research (12/2025)
performance and meta-learning? (c) How well are foundation models able to adapt their learned strategies in response to uncued changes in environment dynamics?
For Alchemy, we compared Gemini 2.5 Pro (IM context), Claude 3.7 Sonnet (200k context), and o4-mini
OOpenAl 2025) (200k context), all of which are reported to employ an explicit thinking step, and ChatGPT-40 (200k context), which does not employ a thinking step. For all tasks, we found that 200k context was sufficient.
4.2.1 Task setup
We evaluated the LLMs on symbolic version of Alchemy in which both actions and observations of the game state are represented text. We used the same basic trial and episode parameters in Wang et al. (0011)
Baselines We compare LLMs to two baselines: Optimal Baseline: the oracle bascline in Wang et al. 2021
which is a baseline that knows the underlying causal structure of the environment and can perform optimal actions. All results shown are normalized to the score the oracle baseline. Heuristic Baseline: To set a
baseline for reasonable performance, we use the memoryless heuristic described in Wang et al. (2021) which places random stones in random potions until cither a stone reaches the maximum reward (in which case that stone is placed in the cauldron, and random selection then continues) or all stones used up (in which case all valuee stones are placed in the cauldron, and the trial then ends).
Evaluation We assess two variables impacting the ability of models to solve the Alchemy task: 1) inclusion of prior information on invariant principles of Alchemy in the prompt (see Section A.4.1 for details), and 2) use of summarization to augment model learning across trials (see Section A.4.2 for details).
1.0
1.0
Veen thall score
Moen trial score
Meen trigl score
0.4
0,4
0.4
0.4
Gemini 2.5
Clude 3.7
od-mini
CharoPT-40
Heuristic
Gemini 2.5
Claude 37
M-mim
CharGe
Hetristi
Gemini 2.5
Claude 3
ad-mini
CharGPT
Heuristic
Gemint 2.5
Claude 3,7
od-mimi
Chata
Houritic Btti
(a) summary, no prior information
(b) No summary, prior information
(c) Summary, prior information
(d) Summary, prior information
Figure 5: Mean Alchemy episode scores for different models and conditions. (a) No summarization, no prior information. (b) No summarization, prior information. (c) Summarization, no prior information. (d) Summarization, prior information. N=10 replicates of 10-trial episodes. Error bars represent standard error of the mean. Asterisk indicates the mean significantly different from that of the memoryless heuristic (p 0.05, paired-sample sample t-test).
We measure model performance primarily through three metrics: 1) performance: mean score over the 10 trials of an episode, 2) improvement: difference of the mean score of the last trials and the score of the first. trial, and 3) adaptation: the mean score for 10 trials following an unexpected change chemistry. For all metrics, trial score is normalized as fraction of the score of an oracle that takes the optimal set of actions for the given items. We use an additional two metrics to gain further insight into model decision making: 1) change in the number of potions used between the first trial and the last five trials (Figure Db), and 2) the fraction of trials in which the model places at least one negative-valued stone in the cauldron. See A.4.8 for results on these latter two metrics. For all metrics, we run 10 replicate episodes with randomized chemistries.
As in Feature World, we primarily evaluate models in the zero-shot setting. We performed small set of experiments to probe the effects of few-shot prompting, but did not find a consistent impact on performance. See Appendix A.4.5 for details.
10

---

Published in Transactions on Machine Learning Research (12/2025)
4.2.2 Effects of summarization and prior information model performance and learning
As shown in Figure 5a, Gemini and, to lesser extent, o4-mini, significantly outperform the memoryless heuristic. Notably, the RL agents evaluated in Wang al. (2021) did not significantly outperform the memoryless heuristic, despite being trained for 1e9 episodes. This condition with no sunmarization strategy and no prior information most closely mirrors the setting of the original Alchemy task experienced by the RL agents, This shows that some LLMs can act as powerful agents on tasks requiring exploitation of strategies learned through extended exploration across multiple tasks, even in environments originally designed for RL agents. However, GPT-oo and Claude are not significantly better than the memoryless heuristic this setting, and o4-mini is far from optimal.
Importantly, in Figure 6 p
that none of the models shows statistically significant within-episode improvement in score for this setting, suggesting that meta-learning is not operating efficiently
1.0
Fraction of Orodez Scoee
1.0
0.6-
0.8
6.8
0.8
0.4
0.6
0.6-
+
Gemini
0.2
0.0
6.0
0.0
2
mevejtriill 6-10)
Trial Number
(a) Computation of score improvement
Mean score improvement
1.0
Mean score improvementt
Mean score inprovement
1.0
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.
0.2
0.2
0.2
0.0
0.0
Gemini 2.5
Claude 3.7
Damiir
ChatGFT
Gemini 2.5
Claude 3.7
04mini
ChatePT-40
Gemini 2.5
Clude 3.7
ad-iini
ChalGPP-4e
Gemini 2.5
Claude 3.7
oa-mini
ChatoPt-4a
(b) summary, prior information
(c) No summary, prior information
(d) Summary, prior information
(e) Summary, prior information
Figure 6: Improvement in normalized score the episode, computed mean of the last 5 trial scores minus the score of the first trial. (a) Illustration of how the score improvement is computed, using as an example the per-trial score trace of Gemini 2.5 the condition with prior information but no summary. (b-e) Same conditions as Figure 5 N=10 replicates of 10-trial episodes. Error bars represent standard error of the mean. Asterisk indicates the mean is significantly different from 0 < 0.05, single-sample f-test).
The inclusion of prior information increased the mean scores dramatically for Claude, 04-mini, and ChatGPT- 4o and slightly for Gemini (Figure 5p), and moreover resulted in score improvements over the episode becoming significant for Gemini and Clande (Figure 0 This shows that the prior information about invariant principles, despite containing no information about the specific chemistry, boosts performance and enables significant improvement over trials, supporting the hypothesis that it provides a framework for the model to generate hypotheses and targeted exploration actions
Forr Gemini 2.5, the lack of improvement signal appears to be due the model learning enough information perform well in the first trial, and then failing to improve that. The other models both show low performance the first trial and fail to improve.
We also performed ablations the specific types of prior information provided determine which invariant principles useful the models. See Section A.4.7 for details.
11

---

Published in Transactions on Machine Learning Research (12/2025)
However, since Alchemy was designed specifically to evaluate models' ability to meta-learn these invariant principles from experience, the same prior information provided in the prompt ought to be present in the model's action and observation history. As such, we hypothesized that prompting the models to summarize their observations and actions after each trial would encourage them to extract equivalent information and lead to a similar boost performance. To lest this hypothesis, we implemented the summarization stralegy described Section A.4.2
We found that summarization improved the mcan scores for all models, though to lesser extent than providing the prior information outright (Figure 5p). The boost in score improvements, however, was greater than in the prior information condition for Cha- and Clande (Figure @p). All models showed significant score improvement over the episode except for o4-mini. This supports the hypothesis that summarization enables meta-learning, since we would expect to see lower scores in the first trial and more improvement over time as the model acquires information over several trials rather than having provided from the start.
To test whether the information gained in the summarization condition is functionally similar to that provided in the prior information condition, we evaluated models with both summarization and prior information. In this setting, mean scores for Claude, o4-mini, and Gemini were almost identical to the prior information condition, and the score for ChatGPT-44 increased slightly (Figure f)). Score improvements were likewise similar to the cases with summarization only or prior information only (Figure Gi)- This supports the hypothesis that summarization enabled the acquisition of information similar to or redundant with that provided in the prior information about invariant principles.
4.2.3 Strategy adaptation following uncued change in game rules
1.0
Mean The oooee
0.6
Gemid 2.3
Gamini 2.5 summary
Gemial 1.5
Claude 3.7
Clssiee 3.7 summery
Claude 3,3
nk-mel
ChanoPt Ao
QPT- Turbo
Heuristic
(a) Mean scores after chemistry change
Figure 7: Mean normalized model scores for trials 11-20 when uncued change in chemistry occurs halfway through 20-trial episode. Hatched pattern represents summary. Error bars represent standard error of the mean, across 10 replicates.
To evaluate strategy adaptation, a human ability associated with dorsolateral prefrontal cortex (dIPFC) function (Mansouri al. 2007 Donahue et al. [2018), in LLMs, we modified the task setup such that the models are exposed to two consccutive episodes before their observation history is cleared. However, the change in chemistry in the transition to the second episode occurs silently, leaving the model to grapple with unexpected outcomes of preeicully-predictablee actions. We ran experiments with summaries enabled and reward and potion pair information provided, but withheld causal information since it was no longer accurate given the change in chemistry. See Figure in the appendix for full timeseries plots of these results.
Of our four thinking models studiod, Gemini 2.5 and Clande 3.7 were able to regain full performance after an initial drop following the change in chemistry (Figure 7 Figure 11 b)). However, 04-mini and ChalGPT-4o were indistinguishable from the heuristic policy following the change (Fig II..).. Unlike 04-mini, ChatGPT-40 had strong performance and improvement over trials prior to the change, showing that the ability to improve through learning of an initial strategy (meta-learning) does not necessarily predict the ability to learn a new strategy when the world changes (strategy adaptation).
To test whether the recent generation of models with thinking capability are required for successful stratogy adaptation, we also tested three previous generation models without thinking capability: Gemini 1.5 Pro, Claude 3.5 Sonnet, and GPT-4 Turbo. While Gemini 1.5 and GPT-4 Turbo were indistinguishable from
12

---

Published in Transactions on Machine Learning Research (12/2025)
random in both episodes (Figure Igg.h). Claude 3.5 showed successful strategy adaptation (Figure I1 demonstrating that thinking or other features of recent models are not strictly necessary for this capability, though they likely help.
To test whether Gemini 2.5 and Claude 3.7 have native strategy adaptation ability without augmentations, we evaluated both models with summary and prior information disabled. In both cases, the models showed a collapse in performance following the chemistry change (Figure [.d.).. This shows that even models with strong native exploration abilitics struggle with strategy adaptation when not provided with task-specific prompt augmentations.
Overall, these results show that while strategy adaptation is not robust native capability of LLMs, it can emerge in some state-of-the-arl LLMs with proper prompt augmentations. This suggests that there is no fundamental barrier to LLMs employing strategy adaptation.
4.3 Alchemy conclusions
Taken together, our results show that integrating information over long time horizons through adaptable strategics in context a frontier challenge in LLMs. In particular, we show that even the strongest models are generally poor at meta-learning and stralegy adaptation without lask-specific augmentations. However, the emergence of these skills after prompting for summarization suggests that LLMs have a latent ability improve and adapt through exploration. Likewise, the comparatively strong performance and adaptation abilities of Gemini 2.5 and Claude 3.7 relative to other models and previous versions suggest that deficits in exploration ability can vanish as consequence of more general model improvements.
5 Discussion and limitations
This work provides critical insights into the active exploration capabilities of foundation models. In Feature World, we find that exploration efficiency remains very close to optimal for most models on tasks with single-feature reward functions, and for at. least one model with more complex reward functions.
While Gemini 1.5 Pro showed efficient information gathering in the 3D Feature World, as well as the ability to draw conclusions about the environment dynamics (when vision errors were excluded), the experiments also underscored that accurate visual interpretation and multi-modal processing can present significant challenges, potentially bottlenccking performance more than reasoning capabilities alone.
We show Alchemy is challenging benchmark for meta-learning and strategy adaptation, where foundation models struggle without prompt augmentations. Critically, we find that inter-trial summarization unlocks these abilitics, suggesting they are latent capabilitics that are not fundamentally out of reach for future models.
While recent benchmarks like DISCOVERYWORLJD (Jansen al. 2024 and Sein (Chen et al., 2024) evaluate agents realistic, end-to-end scientific workflows, our work complements this macro-level view by isolating and quantifying the specific cognitive mechanisms underlying exploration in controlled, abstract settings. Our findings refine the broad observation in these benchmarks that agents struggle with discovery tasks by identifying precisely where this struggle occurs. In Feature World, we find that foundation models act as near-optimal information gatherers, demonstrating that the immediate logic of selecting informative actions a core component of experimental design- is already a robust capability, contradicting the impression from broader benchmarks that agents inherently lack exploratory capability. Instcad, our Alchemy results suggest that some of the failure modes observed in broader benchmarks may stem from deficits in the ability to integrate observations into coherent and adaptable world model across trials. Furthermore, we demonstrate that this deficit is not permanent: lightweight scaffolding, such as periodic summarization, can unlock these latent capabilities.
One limitation of this study is that we exclusively evaluated zero-shot performance using in-context learning. While this isolates the inherent capabilities derived from pro-training and prompting, it does not explore how performance might change with task-specific tne-tuning or other adaptation methods. However, while finetuning is closely related to in-context or meta-learning, believe that direct comparison of the two
13

---

Published in Transactions on Machine Learning Research (12/2025)
constitutes important but distinct topic of study and thus is outside the scope of the current work. This topic has been investigated in papers comparing and contrasting finetuning with in-context learning, e.g., Chan et al. [2022), Mosbach et al. (2023). Chan et al. (2024).
Another limitation that our 3D embodied Feature World experiments are a preliminary sroffof--onneepy with a limited scope, involving a single model in a low-complexity setting. Furthermore, by using a human-in- the-loop for motor control, we intentionally abstracted away the challenges of action generation and physical grounding. A full assessment of embodied foundation models would require integration with, or generation of, motor policies.
Overall, we demonstrate that the frontier of autonomous exploration lies in complex, multi-trial environments where models must continuously integrate information to meta-learn and adapt a world model. While challenging, these abilities can be elicited with prompt augmentations like summarization, suggesting no intrinsic barrier to their emergence. Benchmarks like Alchemy are crucial for testing these capabilitics as models improve and we enter the "era of experience" (Silver & Sutton 2025).
Broader Impact Statement
This work primarily benchmarks and analyzes the performance of existing models, rather than introducing new models or methods. While our finding that summarization enables meta-learning in models could be used to improve the autonomy of L.M agents, the focus is on characterization of different models and behaviors given our specific settings and benchmarks. For these reasons, we expect the primary impacts of our work to be an improved understanding of the existing capabilities of LLM agents and do not anticipate any negative societal impacts for the research in this paper.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Technical Report. arXiv preprint arXim:2303.087 08774, 2023.
Anthropic. Claude 3.7 Sonnet and Claude Code, 2025. URL Accessed: 2025-04-30.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never Give Up: Learning Directed Exploration Strategics. In International Conference on Learning Representations, 2020.
Greg Brockman, Vicki Cheung, Thcodore Hester, Jonas Pottersson, John Schneider, John Schulman, Jic Tang, and Wojeiech Zaremba, OpenAl Gym. arXiv preprint arXiv:1606.0155 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by Random Network Distillation. arXiv preprint arXic:1810.12e8 2018
Bryan Chan, Xinyi Chen, András György, and Dale Schuurmans. Toward understanding in-context vs. in-weight learning. arXiv Lc). October 2024.
Stephanie C Y Chan, Ishita Dasgupta, Junkyung Kim, Dharshan Kumaran, Andrew K Lampinen, and Felix Hill. Transformers generalize differently from information stored in context vs in weights. arXiv [cs. CL),
October 2022.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N Baker, Benjamin Burns, Daniel Adu-Ampratwumn Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgent.eent Toward rigorous assessment of language agents for data-driven scientific discovery. ar CLyj, October 2024.
Julian Coda-Forno Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta-in-context. learning in large language models. In Advances in Newral Information Processing Systems, 2023.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Thomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: Open Frontier-Class Multimodal LLMss arXiv preprint (xXic:2409.11402 2024.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2018.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models. ar preprint arXiv:2409. 110.44999.11440 17146, 2024.
Chad J Donahue, Matthew F Glasser, Todd M Preuss, James K Rilling, and David C Van Essen. Quantitative assessment of prefrontal cortex in humans relative to nonhuman primates. Proc. Natl. Acad. Sci. U. S. A., 115(22):E5183 E5192, May 2018.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama Herd of Models. orXiv preprint arXin:2407.21783 2024.
Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient Exploration for LLMs. orXiv preprint arXiv:2402.008: 00396, 2024.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-Explore: New Approach for Hard-Exploration Problems. arXiv preprint arXiv:1901.0999) 2019.
Yicheng Feng, Yuxuan Wang, Jiazheng Liu, Sipeng Zheng, and Zongqing Lu. LLaMA Rider: Spurring Large Language Models to Explore the Open World. arXiv preprint orXi::2810.08P9 2023.
Gemini Team, Rohan Anil, Scbastian Borgeaud, Yonghui Wu, Jean- Maan-Bapistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: Family of Highly Capable Multimodal Models. aix preprint Xrv::29112. 11805, 2023.
Google. Gemini 2.5: Our most intelligent AT model, 2025. URL 2025-04-30.
wtw.hhhhnnniinuuuppasa----raarad 2025
Accessed:
Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, BastiBs Grill, Yunhao Tang, et al. BYOL-Explore: Exploration by Bootstrapped Prediction. In Advances in Neural Information Processing Systems, 2022.
Saiful Haq, Niyati Chhaya, Piyush Pandey, and Pushpak Bhattacharya. Is your LLM trapped in a mental set? investigative study on how mental sets affect the reasoning capabilities of LLMs. ar fes. CLJ, Jamuary 2025.
Zhiyuan HU, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of Thoughts: Uncertainty-Awarr Planning Enhances Information Socking in Large Language Models. ar preprint arXiw:2402.0m 03271, 2024.
Xu Huang, Weiwen Lin, Xiaolong Chen, Xingmei Wang, Defu Lian, Yashong Wang, Ruiming Tang, and Enhong Chen. WESE: Weak Exploration to Strong Exploitation for LLM Agents. arXiv preprint 20X 2024.
Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bodhisatiwa Prasad Majumder, Oyvind Tafford, and Peter Clark. DISCOVERYWORLD: A virtual environment for developing and evaluating automated scientific discovery agents. arXiv [cs.AT], June 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of Experts. Xix preprint Xii:2400004088 2024.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Akshay Krishnamurthy, Keegan Harris, Dylan J Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? arXie preprint orXin:2433 15971, 2024.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The Al Scientist: Towards Fully Automated Open-Ended Scientific Discovery. arXiv preprint ar XrXiv:2408.00 06888, 2024a.
Cong Lu, Shengran Hu, and Jeff Clune. Intelligent Go- Standing the Shoulders of Giant Foundation Models. arXiv preprint 1xx::2405.15143, 2024b.
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Movay, Oleksandr Maksymets, Sergio Arnaud, al. OpenEQA: Embodied Question Answering in the Era of Foundation Models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Farshad A Mansouri, Mark J Buckley, and Keiji Tanaka. Mnemonic function of the dorsolateral prefrontal cortex in conflict-induced behavioral adjustment. Science, 318(5852):987 990, November 2007.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine-tuning v8. in-context learning: A fair comparison and evaluation. arXiv [es.CLJ, CLJ, May 2023.
OpenAL. Hello GPT-4o, 2024. URL
Accessed: 2025-04-30.
OpenAL. OpenAl o3 and 04-mini system card. May 2025.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba. Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado Van Hasselt. Behaviour suite for reinforcement learning. arXie [cs.LG], August 2019.
Davide Paglieri, Bartlomicj Cupial, Samuel Coward, Ulyana Piterbarg, Macicj Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel. BALROG: Benchmarking agentic LLM and VLM reasoning on games. arXiv [es.Al], November 2024.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven Exploration by Self-supervised Prediction. In International Conference on Machine Learning, 2017.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arxiv preprint (xXi::243305530) 2024.
Allen 7 Ren, Jaden Clark, Anushri Dixit, Masha Ttkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until Confident: Efficient Exploration for Embodied Question Answering, orXiv preprint ar Xxv:224083.1941 2024.
Bernardino Romera-Paredess Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, Alhussein Fawzi, Josh Grochow, Andrea Lodi, Jean-Baptiste Mouret, Talia Ringer, and Tao Yu. Mathematical discoveries from program search with large language models. Nature, 625:468 475, 2023.
Anian Ruoss, Fabio Pardo, Harris Chan, Bonnic Li, Volodymyr Mnih, and Tim Genewein. LMAct: A benchmark for in-context imitation learning with long multimodal demonstrations. arXiv Aes.ATJ, December 2024.
Alaa Saade, Steven Kapturowski, Daniele Calandriello, Charles Blundell, Pablo Sprechmann, Leopoldo Sarra, Oliver Groth, Michal Valko, and Bilal Piot. Unlocking the Power of Representations in Long-term Novelly-based Exploration. arXiv preprinl arXiv:2305.01522 2023.
Burr Settles. Active Learning Literature Survey. Computer Sciences Technical Report 1648, University of Wisconsin 2009.
16

---

Published in Transactions on Machine Learning Research (12/2025)
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 2025.
Allison Tam, Neil Rabinowitz, Andrew Lampinen, Nicholas A Roy, Stephanie Chan, DJ Strouse, Jane Wang, Andrea Banino, and Felix Hill. Semantic Exploration from Language Abstractions and Pretrained Representations. In Advances in Newral Information Processing Systems, 2022.
Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as Case Study. arXiv preprint arXiv:2408. 03186, 2024.
Yuval Tassa, Yossi Doron, Alistair Mulhern, Tom Erez, Yazhao Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Gianluca Barth-Maron, Matthew Hessel, et al. DeepMind Control Suite. aiv preprinl arXiv:1501.00662, 2018.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to Learn, pp- 3 17. Springer US, Boston, MA, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physies engine for model-based control. In IEEE/RSJ /RSJ International Conference Intelligent Robots and Systems, 2012.
Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. Can LLMs learn from previous mistakes? investigating LLMs' errors to boost for reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3065 3065-3080, Stroudsburg, PA, USA, 2024. Association for Computational Linguisties.
Trieu Trinh, Yuhuai Tony Wu, Quoe Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476 482, 2024.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models. arXic preprint 2rXiv:2305.162 2023a.
Jane X Wang, Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, Francis Song, et al. Alchemy: A benchmark and analysis toolkit for meta-reiiforremmnt learning agents. arXiv preprint Xix:22100 02926, 2021.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. JARVIS-1: Open- World Multi-task Agents with Cemory-Augmente Multimodal Language Models. arXie preprinl arXiv:2311.05999 2023b.
Alex L Zhang, Thomas L Griffiths, Karthik R Narasimhan, and Ofir Press. VideoGameBench: Can vision-language models complete popular video games? arXiv Als. May 2025.
Hao Zhu, Raghav Kapoor, So Yeon Min, Winson Han, Jialai Li, Kaiwen Geng, Graham Neubig, Yonatan Bisk, Aniruddha Kembhavi, and Luca Weihs. EXCALIBUR: Encouraging and Evaluating Embodied Exploration. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
17

---

Published in Transactions on Machine Learning Research (12/2025)
A Technical Appendices and Supplementary Material
A.1 Prompts for Feature World
Prompts used for the foundation models in Feature World. Prompts are provided verbatim, with the exception of newlines added to fit the text within the table boundaries.
Prompt
Text Environment
Single Pactor Task
peopecty
filest
Thace objects with different colors
Picking ckject
cwoard either
Thene in stngle property. Soti, on partteudar color iR shage, that Teads to rncard of 1.
(ecana (scma__dnncrigtion descrigtion)
faction reward deorciysion
Baspond fozmet, plaase specidic about the dbject:
up cozject>
Which ixfllenoe recard? <02L2R <SHIPE> <INIRE> uTerro CONTIATION:
Teads reuand
Eeplain romoning Uborvgbly.
Table 1: In-context prompt used for the text Feature World environments for the Singlefeeature tasks.
Taak
Prompt
Texr: Environment
Multi Factor Task
aze plaping text-band paze. Your goil valiinttion ubject proputiio auerssvv couue fer alupo possillu.
Picking
reaae wither color
There singla comitssiin
color and leade reandd
focuso, deacrptinn
factise_revardddee drscriptios)
Barpand uith this forat, please spacific abont the objers:
lction: picc up ceolorad) cxrtterad cobjects
Bisp:
cOCINTII::
propertieu (eug.,
pepista thereggtty.
Table 2: In-context prompt used for the text Feature World environments for the conjunction tasks.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Taak
Frompt
Text Environment
Single Pactor Task
aze piazing teat-baadd paze. Your goal dbject peopecty leats asureaco casurd slugs possible.
Btructured ISON
different colors
Thare single property, revard either
Pickinn
onter caler na Teade rwcard
focua, dracripticn)
(actienrreaadd
(esiis__eeadddreecpppttss))) dracription)
Yoer rergense mues confom the owiing 1808 forma:
wbsettubjje...iee upf: thst objact should picked up asst?
<COLOR> <2SARE>.
"olog": ez 80, you weat yame this stup?
"coearding
you
to c00L05>
you'
Table In-context prompt used for the lext Feature World environments for the single-feature tasks, using structured JSON output format
Thak
Frompt
Twer. Environment
Mulii Factor Task
are playing texs-based gaze. gial combisation object jespersios noz-zere ssepo yossible
Structured ISON
There objerse urth different: colors,
Picking
reand either
There single comstnatiin propertiae,
color
coler and laade rouand
Eacaso, Enceso_desoripeee desorigtions
aeciio__evaadddee srsiic__swasddddsccpppiioo descriptioa)
Your rargonse mort conform following format:
*naxt, wtweettppccked.eeo. that object should picked up abould COOLOD> <TETTUE CHAPPE).
"stop") you gase step?
propartieu shaps,
Table 4: In-context prompt used for the text Feature World environments for the conjunction tasks, using structured JSON output format.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Environmennt
Herraiie
Explotee
rideo
annatating videos
In was gase, the playor controls robor in factory rore, shich containe objacts of various thopes and
oxtore, ane as rot glanks, sise onter, green cytindare, orange stsks, yollow pyraesde, etc.
The player can pice up and nowe abjacts retin a hine laser base.
The player os Lipins to place the eservct type of sajest ua lav cuuveyor belt.
If dejecs in oscrect, the ebjeet dispgoaro in the auchin and the light us the mxchine terse groen. machine teme red the cbject is pushed oft.
object fncorract,
light
pozuible colera are ced, green, bioe, sellor, pupple, ozsare.
possthle stapes gyltnder, plans/haard, proantd,
cosprnhenteely reary object player placee tineotump
Your ceuponse abould be in the dellowing dormat:
0 Timeeiiap 0] stat deject plaved cusveyes? Seoervet incorenct>
[cruestamp 1] cad sbjact placed on ootregors cooerett tecorrects
(staestamp 2) cad sbject placed on cozreyerk Rcoerect tecorrects
Esimessang 3] <oth abject placed so cosseyor>
Soorcect incorreit>
3D Environmect Iterasive
Explosto
explais
rensoning
ration:
shaps,
halt.
propecty,
Lat's find the nart acttion take
what factor (oster deterninas
correctnats
abject.
TT Shere in history of objerts yes, tell the player to pick up randor obyeot you can zea in. the Your frcm the videc.
If you lave uo video inpet yeu, tell the plazez ve explore the 1o0n.
Oiberwive, follow the inokeus bolow.
Importast: You bave VaET turns laft. Chooze your best actioo carutully to maxinins information.
stap-by--ssp:
fuc?
HETER
elimitatos
free conrect.
3. Wost coler avena HOST proeining teat sext?
4. May will this choice
nest usetul inforeation, even isn't correct objecs?
Deplatn your reaseniigg thoronghty. Don't just grace! Each zim ie geacteue.
After dolng your cesoning, resgood at the and with shis fermat, plecos be sproific about the dbjects
scuLdto er CHAAPE camua:: place fookored> subject> conveyoz belt.
Table In-context prompts used for the 3D Construction Lab environment in the exploration phase for the Gemini agent.
20

---

Published in Transactions on Machine Learning Research (12/2025)
Prompt
SD Environment
Trojecsory Reviow:
vislon
In expert gane player amotating grneglay diske,
blue
player pick pyrantde,
hise lazer The in srying place the correct type of onjecs on the convearr bels. If the abject tn conrect, the abject goss through and the light. in the nachine time grean. If the cbject in tncormect, the light the nachtna tume red and the ebject zu paslad off.
neve
poastble colers red, greas, bice, yallon, purple, and srange. pousible stapes cylinder, cabe, plank/board, preantd,
gosl accerstaly corgrahneizely svery abject player
consayor aleg siswetapp
abjact
should
follawing
(simestupp d) dlt object placed
reserect insserect>
(simustapp 1l <2ed object placed conveyur? Seservet incserect?
[craantang 5] card sbject placed on consyaurs cooerat / tecteracts
[crmeetxng 3] cees sbject placed on conveyors. ceoerect / tecteracts
SD Invironment
Trajeciory Review;
explais game
ressouing
propecty,
Bared cherrnatioes above shich
placed
balt
shich
incorrect, explats reasening
the
The right progerty either ageoific shape apecific oolor.
four cesponse should be in the following foreat:
REASCHIN:: cEeplaia your avasosing for how you deduced the right object propecty.> > TARGET rRoVett: diuate vast the speciias surrec, shaps OK specilis corress volor is.>
3D Environment Prajectory Review:
generalization
following would
abjecs propacty
consejor
Table 6: In-context prompts used for the 3D Construction Lab environment in the review phase for all agent conditions.
21

---

Published in Transactions on Machine Learning Research (12/2025)
A.2 Additional results Feature World
A.2.1 Description of optimal strategies
To illustrate the optimal strategy, consider a lask where the hidden rewarding property is "red." The strategy unfolds in two phases. The first phase is exploration, where the goal is find a successful object by maximizing information gain. If an attempt
"blue toy" fails, the agent learns that if color is the rule, "blue" is not the answer, and if shape is the rule, "toy" is not the answer. The optimal next action is to test object with entirely new features, like a "yellow sphere", to efficiently explore the remaining possibilities.
Once an action succoods for example, picking up a "red box" the stratogy shifts to the second phase: isolation. The agent must now disambiguate whether "red" or "box" is the true cause. The optimal way to do this is lest a new object that changes only one of these features, such as a "red sphere" or a. "green box", to definitively pinpoint the rewarding property.
A.3 Exploration 3D embodied environments: Construction Lab
INPUT
OUTPUT
INPUT
OUTPUT
INPUT
(a)
(b)
Figure Screenshots of gameplay in the 3D Construction Lab environment. (a) The agent uses a blue laser beam to pick up objects. (b) Result of a correct object placement. (c) Result of an incorrect object placement.
To further evaluate the foundation models in 3D embodied environment, implement analogous task to the text-based environment in factory-style simulation called Construction Lab. Construction Lab introduced in (reference-anonymizee as simulation environment that includes both game-like mechanics and simplified but non-trivial object manipulation and physical reasoning.
In this work, we focus on a task that requires the player to operate a simple machine called the Exchanger. The Exchanger requires objects with specific properties to be placed on an input conveyor belt (Figure) B If an object matches the requirement, the input consumed, green light shows for few seconds, and an output object is produced on an output belt. If the object is invalid, the machine rejects it by reversing the input belt and red error light is activated. No cues provided regarding the correct input object required, and thus the task entails determining what the correct object properties are through trial and error, observing how the machine responds to input objects, and drawing appropriate inferences.
Through the use of this 3D, visually rich environment that mirrors the challenges of the text-based environment, we are able investigate the effects of visual complexity on active information gathering, reasoning, and hypothesis testing.
A.3.1 Task setup
A number of additional challenges must. be addressed when performing this exploration lask in a 3D embodied environment. First, the agent must assess both the current state of the environment and the consequences of any actions taken through vision. Sccond, the agent requires motor control module to exccute exploratory actions. We use Gemini 1.5 Pro's multi-modal functionality ingest video input from Construction Lab sub-sampled to 1.5 Hz and 320 240 resolution. To disentangle vision and reasoning performance from
22

---

Published in Transactions on Machine Learning Research (12/2025)
translation of natural language instructions into complex keyboard-and-moute action space, we adopt setup in which instructions are provided to human actor who performs the exploratory actions online.
We assess Gemini 1.5 Pro's ability to generate these exploratory instructions by comparing against an optimal and random baseline, mirroring those in the text environment. The optimal strategy was performed by a single human performing the task according to an optimal policy that maximally reduced uncertainly about the correct property, The random strategy was performed according to policy that selects a random object from the room at each step, with replacement.
As running the 3D environment and using human actors in the loop reduces experimental throughput, we limit ourselves to single level of environment and reward function complexity. We choose the condition with 3 colors and causal factor, as conditions with more colors had significant visual clutter. Each lask is randomly generated as follows: at the beginning of the episode, 3 unique colors and 3 unique shapes are randomly selected from colors and 5 shapes, and objects with cach shape-color combination are placod in random locations in the environment, for total of 9 objects. One property, either a shape or a color, is randomly selected as the correct property, for total of 3 correct objects. The player and the Exchanger machine with input and output conveyor are likewise placed randomly in the room. A gameplay episode ends when either all 3 correct objects are placed on the input conveyor or 2 minutes have clapsed.
A.3.2 Gemini-based agent
The Gemini agent is implemented follows: every 10 seconds, the model is fed the most recent 100 video frames (or 67 seconds) of gameplay and queried in two stages, during which gameplay is paused for the human actor. We implement a two-stage procodure with vision stage and a reasoning stage, which we found improves accuracy for each stage compared with running both logether. In the first stage, Gemini is asked lo list, for every object placed on the imput conveyor, the timestamp at which was placed, its color and shape, and whether it was correct or not (as indicated by a red or green light on the machine). In the sccond stage, Gemini is provided the output of the first stage (subsequent video frames and list of objects placed with their reward values) and prompted to select next exploratory action to maximize information gain, similar to the text environment. The human actor is provided only with the command generated by the second stage, such as "place the red cube on the conveyor."
All video trajectories are processed in the same way, regardless of how the exploration instructions were generated. Specifically, we truncate the video to include only the first 4 object attempts. Gemini is then called on the truncated video in three steps: vision, reasoning, and generalization. In the vision step, it is asked to list all the objects placed on the conveyor and whether they were correct, similar to the vision step in the exploration policy. In the reasoning step, it is asked to deduce the correct object property based on its observations. In the generalization step, it is asked to predict whether each object in list of hypothetical objects would give reward. See Appendix A.1 for specific prompts used.
A.3.3 Evaluation
To evaluate different aspects of performance for each agent type, we measure relevant property accuracy and number of objects until sufficient information is acquired to determine the correct property, assuming perfect reasoning. We also record the number of vision errors made by the VLM when listing objects in the full video, defined as misclassifying the shape, color, or correctness of an object placed on the conveyor, or omitting mention of an object placed on the conveyor. Because internal game states are not exposed in our experiments, we use manual human annotation of video trajectories to collect the above metrics and error counts. We collect total of 15 trajectories for each agent type.
A.3.4 Results
In the exploration efficiency metric, we see the same trends in the results for the 3D embodied environment as for the text environment, with Gemini's exploration efficiency significantly outperforming the random bascline and approaching the optimal bascline (Figure 3h). These results suggest that the additional complexity of an imperfect vision system and partially observed environment state are not significant limitations in generalizing
23

---

Published in Transactions on Machine Learning Research (12/2025)
directed exploration capabilities to embodied 3D environments. In the accuracy metric (Figure Br), the picture is more nuanced. For relevant property accuracy, the difference between performance with the Gemini agent and the random agent was not statistically significant (p > 0.05, paired sample t-test).
This result is interesting because VLM vision is also necessary for the exploration phase, where there was no discrepancy in performance. A likely reason for this that the iterative nature of the exploration task makes it robust to occasional errors. Because the model must re-list all objects placed at each step, chance errors made during one step do not propagate to later steps.
To probe the reason for the gap in accuracy performance, we also computed results where we filtered out trajectories in which the vision step made an error (Figure Bp). In these results, accuracies for the Gemini and optimal agents are nearly identical and their differences with the random agent are statistically significant (p < 0.05, two sample t-test). These results suggest that errors in the vision step, rather than reasoning or exploration, are responsible for the relatively reduced accuracy in the Gemini agent condition.
Taken logether, results in the Construction Lab show that the directed exploration capabilities of foundation models robustly generalize from text-based environments to embodied 3D environments, though overall accuracy of the system is somewhat reduced by imperfect performance of the VLM's object and action recognition in videos. This indicates that the challenges of modai reasoning from realistic simulated video could be addressed by focusing on the vision and action recognition capabilities of foundation models separately from their reasoning capabilities.
A.4 Additional Alchemy details and results
A.4.1 Invariant principles in Alchemy
While the specific chemistry changes per episode, Alchemy includes invariant principles or abstract regularities that span all episodes. These invariants are crucial because discovering and exploiting them over many episodes is the essence of the meta-learning problem in Alchemy.Such invariant properties include, among others: 1. within an episode, stones with the same visual features have the same value and respond identically to potions, and potions of the same color likewise have the same effects; 2. potions come in fixed pairs (c.g., red/green, yellow yellow/orange, /orange, pink /ur//)uuuuoie which always have opposite elfects; 3. the underlying causal graph topology is structured by generative grammar, though some edges might be missing, creating 'bottlenecks'; and 4. the maximum stone score is 15, and the minimum score is -3. It's important to note that all of our experiments were conducted without any post-training, and all models are presumed to lack knowledge of these invariant task structures 4 We therefore conduct experiments to test the effects of including various components of these invariant properties into the prompt. This allows us to disentangle the challenge of acquiring such mela-learned knowledge from that of already possessing it and being able to use it to inform smart exploration.
In the case with no prior information, the model is given the main prompt to introduce Alchemy, which describes the general gameplay mechanics (Figure g] We hypothesized that, in order to learn which potions optimally improve stone reward value for given combination of stone properties, the models require an understanding of the invariant propertics of the Alchemy tasks, which provide framework on which the model can integrate evidence from its observations. To lest this hypothesis, we provided additional information about the reward, potion pairing, and causal mechanics of the game (prior information condition, Figure
A.4.2 Cross-trial summarization
The multi--rial structure of Alchemy introduces need to manage very long contexts and perform inference across multiple timescales and abstraction challenge that often arises in real-world, and particularly multi-modal, settings. Even in the case that it is possible to fit all observations, actions, and rewards from a multi-turn interaction within the model's context window, it is unclear whether current models are capable of natively managing ever-growing context. while maintaining inference and reasoning capabilities. We therefore
4 Because the Alchemy cnvironment was published in 2021, knowledge of it may be included in the pretraining data for the foundation models studied here. However, when we probed the models, we found they entirely lacked knowledge about potion pains and min/max rewards and had limited knowledge of the environment in general.
24

---

Published in Transactions on Machine Learning Research (12/2025)
You are playing text-based game called Alchemy where you place stones of different shapes (round or pointy), sizes (small or large), and colors (blue or purple) into potions that change the stones' properties. The goal is to maximize the reward value of the stones, and then place them into the cauldron to increase the total score. Placing a stone in the cauldron adds the current reward value that stone to the total score. However, there might be more rules to the game than mentioned here.
Main prompt
Reward information
The maximum stone reward value is 15 and the minimum is -3.
There are six different potion colors, which come in three pairs: yellow yellow/orang /orange, red/groen, and pink/ pink/turquoise. Potions in a pair have the opposite effect from cach other on a given property.
Potion pairing information
A stone's reward value is determined by its properties. The game is deterministic: a potion of a given color always has the same effect on a stone with given properties, and a stone with given propertics always has the same reward valuc.
Causal information
Figure 9: Prompt and different components comprising the prior information (reward information, potion pairing information, and causal information).
examined and along-conten context summarization method. We hypothesized that summarization would offer a benefit due to the multi-trial structure of the task that requires extended directed exploration and inference across multiple timescales.
Specifically, the summary is constructed the end of each trial by asking the model to create summary conditioned on the current trial's events as well as the summary from the previous trial. At each step, the model is asked to take actions based on the summarization as well as the current trial's events so far. We additionally have a no summarization condition, in which all episode events up until the current step are included in the context.
A.4.3 Alchemy experimental setup details
We implemented a custom version of symbolic Alchemy, modified so that observations and actions are given in natural language format. Episode chemistries are sampled every episode and held constant for the duration of 10 trials, although new stones and potions are resampled every trial. Trials last for maximum of 20 steps, and the agent could choose to end the episode at any step. At each step, the agent could choose to select a stone and then either a potion or the cauldron (to cache the stone and obtain the reward), or to end the trial. See Figure
(d) for visualization of the tasks.
We considered the condition of procedural generation in which the transition graph has random bottlenecks: This is the hardest version of the task in which certain transformations (e.g., the effect of blue potion) are only possible if specific initial conditions of the stone are met (e.g., the stone being transformed is large and round).
A.4.4 Example model outputs Alchemy
Below, we display examples of raw outputs of Gemini 2.5 from trail 1 and trial 5. For each trial, we show 1) the model's reasoning in step 1 of the trial, 2) the model's reasoning in step 5 of the trial, and 3) the model's summary of the trial. All outputs are drawn from the condition with summaries but no prior information.
For the reasoning steps, we also display the observation of the game state provided the model by the environment, labeled GAME STATE UPDATE.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Gondition
Output
GATE STATE STGATE:
spteods
trial
chrereaticn
aur tollewing stuavur
perple seall pointy rovard
pointy
cunquatie
2: Jacquoi00
6: pink
4: yellow
5: zed
8: yollow
7: red
red
sucquotos
pink
arange
MIDeL RESPINE::
inscavario:: This the dizat action of trtal. have used
thetr effects
(ExSоиTI:: ny initial goal. is to inforesand the offecss of the different potiono. I (ill begin tessing the first awailable potion, which in the chooning to nen acona (porple swall pointy, vousnd -1) for this seperteent. Stnce this atone han regastion raused, there ix Steete risr in modteying 1t, and any postrite change would be rigntricant gate. Thix acttion will receal what pine pastens do perple, seall, und poincy avosea.
plak potton at tnder 5,
ACTIOM: Place sieue 0 in pution 0.
Trtal Step 5
that pariion
Dirzosss stome:
small gotnty with rwaard
fiev otauzvation
You aue thr follewing utoavu:
0: paople large pointy will revard
1: paople small pointy sitt resard -1
2: parple large pointy with pwaard
You one she foiloutnng pattion::
Kons
pink
5: turquates
10: plak
11: ararge
Oaczout triel accce:
ierERvaTI::: Plastag stone (blue maill potnty, rumaed 1) into and potiion at index changed color blue to purple. This also inverted
-1
purple pointy perple stap aractty pattone change coffiee lepotisnio
purple patery wflect changed
atteupt torguct
pupple large
-I
tarquine posiin
inprove
Trial Summury
oee Fotter Iffecte
<47iak Potica:es lppeaza to have zo offect. It waa taated a 'purple small pointy' stose with no change.
esled Postonis Changee a ssone to color from "blos' 'porplal, does not affece 'gerpte' esines, The effect on the rwoard degende the stone's "enall" stonss hat, stxpe the new for "large" steons. +suurpoii Potion:ss Gasages the vine 'pupple small' sicous to 'pusple large' it bas so vifect 'tlue' visars uz ulceady 'large' vicoss. reaurd ja invetee.
sizw: the reuard is tscarted
Potioni*e Shangeo stone's stape 'round" to "pointy". has to offect 'guisty' sisnco. revard inverted.
'potsty"
The highass rwardd valus in the gren Rasad the affecte, combinations of propertine reeelt highett repard
arw:
"bius sail potney"
'pscple large
"blus Lazge pointy"
"pipple anall round"
Table 7: Outputs of Gemini 2.5 for Trial 1 of Alchemy in the condition with summarization and no prior information.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Condition
Output
1rial 5 Step
snall reuard
hlue with reoaed
You Uhe Jollewing poliioa::
0:
1: oraage
2: zed
S: pans
yellou
red
gres
MItEL RESPON5::
(SSERVATEΣ:: The gase has otarted with one osone (Btone 23 already 'blue largo pointy" state, shich love idrotifiod as the highest possible
rasord etate (+1S).
REASCHUN:: Stese cucruntly has a revard of 15. Beand so my pruvious Lindings, this in the cazinca achierable rovard. Using any petien at vould cisk lovering ala value. Sheredorw, the most logicol Ficut stop zu piece at in the cauldron incodiutelly to loth is the +15 pccce.
ACTIOM: Place sieus 2 is the cauldrou.
Trial Step 5
Moss
Whas
Dirzoss blus round rouard
fier obsuzvation
You aue the follewing utoaru:
(: blus anail noand ustt recand -1
1: Hone
2: Hons
You oee following patiions:
ozunge
patk
0: red
10:
11: plak
Oaczent trial actre: 50
(6SEXWATEOM: Orange potiion on "blor stoue. This confirns ny lypolbmis Orarge polioa vily wock
"goisay" ssones
only newininn
lagrore geopersies apportusiky
hypothesis
'puapls".
ACTION: Place
TVial Summary
eedreen Potionces Changes a stone's color from 'purple' to 'bloe"; The revard charge depends the acone's stape. 12 the stone Ln fround" its revord incresses by 2.
1f the stone is pointyy, ita zewaed incrvases by 14.
veled Postencee Changes soone color from "hins' 'porpla'
trancforatio contittanll niberaste free the econa repard. Pattencee thangee etonats stape free 'romd! to "potnty'. This craneformation conciatently adde the stone's recard. <edcongs Potiunzes Shaggo sisue's abape 'puisly' 'roud", This tramiformsioo vabicusta live the visus zeweed. potion
stores
eelurqustte
Oaange 'porple
'gurple large'
transfomation
Highart Seaard Dinhinationn
The stagla cochination of propartiin with highert ramad ix:
eebiue potatyes
This qpenifo oobinatioo resolta in revard value of e*.lden, This state is bost ackieved by transforming 'yurple poinsy' ssens visk froes potton gain the +id reaard baros.
Table 8: Outputs of Gemini 2.5 for Trial 5 of Alchemy in the condition with summarization and no prior information.
27

---

Published in Transactions on Machine Learning Research (12/2025)
A.4.5 Effects of few-shot prompting
1.0
Wese that
Charset4s d-stot
CharoPtaa Fato 1-iiet
Chatar-- Jetus
Charamn4o 6uteet
Gamin 2.5 Rash d-stot
Gemini 2.5 Rash L-ttux
Genins 2.5 Fashh 2-dlet
Gemini 2.5 Raih 4-stot
Houndtic
Figure 10: Mean normalized model scores for showing the effect of few-shot prompting model performance. Scores represent the mean of 5 trials. Error bars represent standard error of the mean, across 10 replicates.
To lest the effects of few-shot prompting on model performance in Alchemy, we collected trial histories from the best performing model, Gemini 2.5 Pro, to use as few-shot examples. Each example comprises all game state updates and Gemini 2.5 Pro responses (see Tables and for Trial 1 of an episode with unique chemistry not present in the test sct. We generated 4 such examples and appended 0, 1, 2, or of them the prompt for the test. models. Examples were appended in a random order for each of 10 replicates. We used ChatGPT-4o (a model of similar size but lower performance) and Gemini 2.5 Flash (a model of smaller size) as test models. Both the examples and test model responses were generated in the condition with no summary and no prior information. To ensure that context limitations did not impact performance, we ran the test models for 5 trials rather than 10.
Our results show limited and inconsistent effects of few-shot prompting on model performance (Figure 10 Few shot prompting provided slight improvement to scores for Gemini 2.5 Flash, but the size of the improvement did not increase with more than 1 shot. Interestingly, few shot prompting appeared to decrease performance for ChatGPT-4o in all but the 4-shot case. In all but the 1-shot case for Gemini 2.5 Flash, model performance was not significantly improved compared with the memoryless heuristic, suggesting limited utility of few-shot prompting for this task.
The limited effects of few-shot prompting may be due to the fact that the examples contain no information about the specific chemistry, so the models can only gain information on generally useful exploration straegge from them, the extraction of which is more abstract and difficult.
A.4.6 Strategy adaptation timeseries results
A.4.7 Effects of prior information ablation
To further invostigate the effect of prior information in the prompt, we performed ablations in which we removed either reward information, potion pairing information, or causal information from the prompt with prior information (see Figure gb. We ran the models with the ablated prompt and no summaries enabled and measured the performance.
We found that the effects of prior information ablation differed substantially depending on the model. Gemini 2.5 the most robust to the ablations, showing only slight decrease in performance when no prior information is provided (Figure 12)). Interestingly, Claude 3.7 showed no decrease in performance with any of the single ablations, but still showed large decrease when all prior information This suggests that Claude 3.7 is more robust at meta-learning without summaries, but still struggles without at lcast small amount of initial prior information to build off of. Performance of 04-mini was reduced slightly without causal and reward information, and moderately without polion pair information or when no information provided (Figure [2:). Performance of ChatGPT-40 was reduced substantially and became statistically insignificant compared the hcuristic policy with removal of any piece of prior information (Figure I2... This shows that this model is less robust learning principles on its own, potentially due its lack of thinking ability.
removed (Figure 12b).
28

---

Published in Transactions on Machine Learning Research (12/2025)
alennl Hereryton lnarst
8.8
68
Gama12.5
Cbude 2.7
That Number
Humber
(a) Gemini 2.5 Pro, summary
(b) Claude 3.7 Sonnet, summary
(c) 04-mini, summary
10.
Cenmead munnay
Cauda sarmar Homnsksssuuste
Shaidh
26
16
12
xix
03
25
17.5
26n
0.8
10.0
12%
15.
15.
36.
Tha Number
(d)
Gemini 2.5 Pro, no summary
(e) Claude 3.7 Sonnet, no summary
(f) ChatGPT-40. summary
Geneias Menosskos Horbde
Marverytn Heurstk
Dbate
(g) Gemini 1.5 Pro, summary
(h) Claude 3.5 Sonnet, summary
GPT- Turbo, summary
Figure 11: Normalized model score by trial when uncued change in chemistry occurs halfway through the episode. (a-i) Trace of score across 20 trials. The vertical dotted line denotes the point at which the change in chemistry occurs, following trial 10. Error bars represent standard error of the mean, across 10 replicates.
1.0
1.0
1.0
1.0
Mean trial score
Mean trial score
0.8
0.8
0.8
0.8
trall
0.6
0.6
0.6
No prter info to pation into Ho caurall into All prur into Heuristie
No reaard into
No ptur info No reward into
No petion Info No cautal info All piter into Heuristic
No prter info No reward info gation info No causal into
prier Into Heuriati
No priot into No rewerd into to potion into No causal Into All priar info
Houristi
(a) Gemini 2.5
(b) Claude 3.7
(c) o4-mini
(d) ChatGPT-40
Figure 12: Mean trial scores for each model with no summarization when various pieces of prior information have been removed from the prompt. (a) Prompt ablations for Gemini 2.5. (b) Prompt ablations for Claude
3.7. (c) Prompt ablations for 04-mini. N=10 replicates of 10-trial episodes. Error bars represent standard error of the mean. Asterisk indicates the mcan is significantly different from that of the memoryless heuristic (p 0.05, paired-sample (-lest).
A.4.8 Alchemy item usage
To investigate the mechanism behind the score improvement in the models, analyzed reduction in the number of potions used by the models in later trials (Figure 13). The most apparent finding is that Gemini 2.5 shows a siguificantly larger reduction in potions used than the other models when summaries are enabled. This demonstrates that the summary has a substantial effect on the model's strategy, despite the fact that the summary only improves the already-high performance of Gemini 2.5 small amount. We also found that Gemini and Claude reduced their potion use substantially more than ChatGPT-40 overall, suggesting that these models owe part of their superior performance to an ability learn to make more efficient of potions.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Finally, to analyze the propensity of the models for clear reasoning errors in different conditions, computed the fraction of trials in which model places at least one negulive-valued stone into the cauldron (Figure 14h. Such an action is obviously counterproductive, and is never performed by the memoryless heuristic. We found that all models place negative stones into the cauldron at least a small number of times, but in most cases less than 10% of trials. Interestingly, however, the prior information ion conditions, Claude places negative stones the cauldron between 20% and 25% of trials, despite being high performing model these conditions. It is clear why this occurs, but it suggests there is room for significant improvement in Claude this task.
Reduction #potions
10
Reduction #potions
10
Reduction #potions
10
Reduction apations
10
Gemini 2.5
Claude 3.7
od-dni
CharGPT-4
2.5
o4-mini
CharGPT-4
Gemini 2.5
Clmude 3.7
04 mini
CharGPT/40
Gemini 2.5
Claude 3.7
od-ini
ChatGPT-40
Clsude
(a) summary, no prior information
(b) No summary, prior information
(c) Summary,
(d) Summary, prior information
prior information
Figure 13: Reduction in the number of potions used over the course of the episode, computed as the potions used in the first episode minus the mean of the potions used in the last 5 episodes. (a-d) Same conditions as Figure
N=10 replicates of 10-trial episodes. Error bars represent standard error the mean.
Freq neg staress used
0.3
Freq neg stomes used
0.3
Freq neg stones
Frod
Gemini 2.5
Claude 3.7
o4-min!
CharGPT-40 T.40
Gamini 2.5
Claude 3.7
04-mini
ChetGPT.40
Gamini 2.5
Claude 3.7
od-mini
ChatGe7.40
Gemin 2.5
Caude 3,7
on-man ChaiGPT.o
(a) summary, no prior information
(b) No summary, prior information
(c) Summary, prior information
(d) Summary, prior information
Figure 14: The fraction of trials in which at least one negative-valued stone was placed in the cauldron by the model. (a-d) Same conditions as Figure N=10 replicates of 10-trial episodes. Error bars represent standard error the mean.
30


Reviews:
[{"review_id": "gJRXrT0tfb", "reviewer": "Reviewer_hSGY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a few metrics to evaluate the performance of the exploration capabilities of LLMs (or LLM-based agents), including Efficient information gathering, Meta-Learning, and Strategy adaption. These three metrics are used to validate different perspective of LLMs' ability in exploring the environment and improving their own ability during exploring. After that, the author conducts a few experiments in some common datasets to calculate these metrics of different LLMs. The experimental results reveal some key differences in these metrics and indicate that LLMs can better handle the text-based task in the Efficient information gathering metric but usually fail to handle the complex multi-trail environment in the Meta-learning and Strategy adaption metrics. This study provides a fundamental exploration in the exploration ability of LLMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I would agree that this paper's claim is supported by  accurate, convincing and clear evidence because the following reasons:\n1. The author made claims based on their experiments instead of simply proposing some claims. That is, it has a \"partially\" rigrously defined metric, and they conducted experimetns to evalaute these metrics. The final results are based on the experimetal results.\n2. While experiments are not sufficiently diverse and representative, it has been a good start for further exploring the related topics or abilities of LLMs.", "audience_yn": "Yes", "audience_explanation": "The finding of this paper is not so surprising; that is, it is not hard to see current LLMs will fall short in these complex tasks. A long time-scale continuous evaluation is urgently needed. However, such long-term evalution would be much beyond the scoop of this work. As a result, I would believe many people will be interested in the findings of this paper.", "requested_changes": "1. I request the author to include a more rigours definition of each metric. For example, the efficient informatio gathering is calculated as the success rate within a fixed step budget, the author should give it a name or symbol, e.g. $\\mathsf{M}\\_{\\text{info}}:= \\frac{n}{m}$, where n represents ... and m represents ... . Put them in a separate section and list them with a clear definition. \n2. On page 5, texts are overlapping each other.  \n3. What is \"[reference-anonymized]\"?", "broader_impact_concerns": "This work mainly introduces some interesting findings based on their experiments; so, there is no any concersn or ethical concerns in this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oDDMLL0Qkc", "reviewer": "Reviewer_sXfW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary \n\nThe paper evaluates foundation models as interactive agents in environments requiring multi-turn exploration.\n- Authors introduce Feature World (text and 3D) to test information gathering about hidden reward functions. They then introduce to Alchemy, a multi-trial benchmark for meta-learning and strategy adaptation.\n\n\n- Authors show that LLMs achieve near-optimal efficiency in simple Feature World tasks but struggle in Alchemy. They note that prompting models to summarize across trials unlocks emergent meta-learning and, in some cases, strategy adaptation.\n\n\n- Authors benchmark multiple frontier models (Gemini 2.5, Claude 3.7, ChatGPT-4o, o4-mini), finding large gaps in robustness.\n\n\n\n\n## Strengths\n- Authors decompose key capabilities in exploration into three parts: information gathering, meta-learning, and strategy adaptation.\n- Authors present interesting findings like summarization enables meta-learning; Gemini 2.5 shows strongest robustness.\n- It’s novel to use Alchemy as a zero-shot exploration benchmark for LLMs capability, contrasting with previous RL results.\n\n\n## Weakness\n- The connection to related work on LLM’s capability on exploration, such as DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents (arXiv:2406.06769) is not well discussed. It remains unclear what are the new insights this paper adds beyond those studies.\n- Evaluation is limited to zero-shot prompting. The paper does not explore richer settings such as fine-tuning or integration with memory modules.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think the experiments are well structured, with baselines (e.g., optimal, random heuristic) and clear performance metrics. The writing is also clear.", "audience_yn": "Yes", "audience_explanation": "I think the audience working on the following domains would be interested: LLM for scientific exploration, LLM/VLM for meta-learing.", "requested_changes": "See the first bullet in Weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DVkrFSW1kT", "reviewer": "Reviewer_5UNj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systematically evaluates the ability of foundation models to conduct active exploration in interactive environments, focusing on three core capabilities: efficient information gathering, meta-learning, and strategy adaptation. The authors introduce two benchmark environments—a simple, text-based Feature World and a more complex, meta-learning-focused Alchemy—to test these abilities in a zero-shot setting across multiple state-of-the-art models, including various versions of Gemini, Claude, and ChatGPT. Key findings reveal that while most models perform near-optimally in simple information-gathering tasks, their ability to meta-learn and adapt strategies over time in complex, stateful settings like Alchemy is limited without explicit prompting. However, the study shows that prompting models to summarize information across trials unlocks emergent meta-learning and adaptation in some models, particularly Gemini 2.5 and Claude 3.7. A proof-of-concept 3D embodied version of Feature World also demonstrates that exploration capabilities can transfer to multimodal settings, though vision errors remain a performance bottleneck. The work provides valuable benchmarks and highlights both the current limitations and future potential of foundation models as autonomous exploratory agents. Strengths include its comprehensive evaluation and novel insights into eliciting meta-learning through summarization, while weaknesses involve the limited scope of 3D experiments and a focus only on in-context learning rather than fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the submission are strongly supported by accurate, convincing, and clear evidence, as the authors employ rigorous experimental designs across multiple benchmarks (Feature World and Alchemy), test a range of state-of-the-art foundation models, and use appropriate baselines and statistical measures to validate their findings. Results are presented through well-labeled figures and tables that clearly illustrate performance trends, such as near-optimal efficiency in simple tasks and the emergent meta-learning capabilities enabled by summarization. The inclusion of ablation studies, error analysis, and transparent reporting of prompts and limitations further strengthens the validity and reproducibility of the conclusions", "audience_yn": "Yes", "audience_explanation": "LLM/vlm agents are currently a hot reasech topic in ml research, and understanding how agent interact with enviroment is interesting", "requested_changes": "While the zero-shot focus is a strength, adding a small experiment (even in the appendix) comparing zero-shot performance to a few-shot  or a lightly fine-tuned model would greatly strengthen the paper. This could be done on the text-based Alchemy task.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)