Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models
Authors: Cong Lu
Shengran Hu
Jeff Clune
Venue: ICLR 2025 Poster (2025)
Keywords: Exploration
Large Language Models
LLM agents
Open-endedness
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a new FM agent for hard exploration tasks based on the classic Go-Explore algorithm augmented with foundation model intelligent selection.

Abstract: Go-Explore is a powerful family of algorithms designed to solve hard-exploration problems built on the principle of archiving discovered states, and iteratively returning to and exploring from the most promising states. This approach has led to superhuman performance across a wide variety of challenging problems including Atari games and robotic control, but requires manually designing heuristics to guide exploration (i.e., determine which states to save and explore from, and what actions to consider next), which is time-consuming and infeasible in general. To resolve this, we propose Intelligent Go-Explore (IGE) which greatly extends the scope of the original Go-Explore by replacing these handcrafted heuristics with the intelligence and internalized human notions of interestingness captured by giant pretrained foundation models (FMs). This provides IGE with a human-like ability to instinctively identify how interesting or promising any new state is (e.g., discovering new objects, locations, or behaviors), even in complex environments where heuristics are hard to define. Moreover, IGE offers the exciting opportunity to recognize and capitalize on serendipitous discoveries---states encountered during exploration that are valuable in terms of exploration, yet where what makes them interesting was not anticipated by the human user. We evaluate our algorithm on a diverse range of language and vision-based tasks that require search and exploration. Across these tasks, IGE strongly exceeds classic reinforcement learning and graph search baselines, and also succeeds where prior state-of-the-art FM agents like Reflexion completely fail. Overall, Intelligent Go-Explore combines the tremendous strengths of FMs and the powerful Go-Explore algorithm, opening up a new frontier of research into creating more generally capable agents with impressive exploration capabilities. All our code is open-sourced at: https://github.com/conglu1997/intelligent-go-explore.

Published as a conference paper at ICLR 2025
INTELLIGENT GO-EXPLORE: STANDING ON THE SHOULDERS OF GIANT FOUNDATION MODELS
Cong Lu
Shengran Hu srhuêca, .bc. ca
Jeff Clune jo une@gma com
conglu wbc. ca
'University of British Columbia
Vector Institute
CCaadaa CIFAR Al Chair
ABSTRACT
Go-Explore is powerful family of algorithms designed to solve hard-exploration problems built on the principle of archiving discovered states, and iteratively re- turning to and exploring from the most promising states. This approach has led to superhuman performance across a wide variety of challenging problems in- cluding Atari games and robotic control, but requires manually designing heuris- tics to guide exploration (i.e., determine which states to save and explore from, and what actions to consider next), which is time-consuming and infeasible in general. To resolve this, we propose INTELLIGENT GO-EXPI.ORE (IGE) which greatly extends the scope the original Go-Explore by replacing these hand- crafted heuristics with the intelligence and internalized human notions of inter- estingness captured by giant pretrained foundation models (FMs). This provides IGE with a human-like ability to instinctively identify how interesting or promis- ing any new state (e.g., discovering new objects, locations, or behaviors), even in complex environments where heuristics are hard to define. Moreover, IGE of- fers the exciting opportunity to recognize and capitalize on serendipitous discov- sties-states encountered during exploration that are valuable in terms of explo- ration, yet where what makes them interesting was not anticipated by the human user. We evaluate our algorithm on diverse range of language and vision-based tasks that require search and exploration. Across these tasks, IGE strongly ex- ceeds classic reinforcement learning and graph search baselines, and also suc- ceeds where prior state-of-the-art FM agents like Reflexion completely fail. Over- all, INTELLIGENT Go-EXPLORE combines the tremendous strengths of FMs and the powerful Go-Explore algorithm, opening up new frontier of research into creating more generally capable agents with impressive exploration capabilities. All our code is open-sourced at: https::/ thub. com wtt.cooc/cccccco ul 919977
intel
INTRODUCTION
Foundation models (FMs, Bommasani et al. (2021): OpenAl (024)): Brown et al. (0200) Team
2024 Touvron al. (2023) trained giant internet-scale datasets have demonstrated strong gen-
eral capabilities in reasoning (Wei et al. 2022) and understanding (Chang et al.
2024
As such,
these models have been increasingly employed as autonomous agents (Liu et 2023
Yao al.)
2023b; Wang et al., 2024 Yao et al. 2023al Shinn et al.] [0233] Besta et al.) 2024b) in decision-
making tasks, showcasing the ability to adapt in-context (Dong al. [2022] Olsson al. 2022) to unseen tasks. Foundation models have also begun to see success in challenging reinforcement learn-
ing environments like Doom (de Wynter 2024), real-world robotic control (Brohan et al., 2022 2023) and 3D video games (Raad et al. 2024 Wang et al. 2023). However, significant challenge remains: foundation model agents often struggle in environments that require deep exploration over extended time horizons (Liu et al. 2023). Overcoming this limitation would enable realize their potential as autonomous assistants in more open-ended domains like scientific discovery and innovation (Jiang et al. 2023). This paper introduces INTELLIGH LIGNT GO-EXPLORE (IGE), novel

---

Published as a conference paper at ICLR 2025
explore archie?
meteesinnly archive?
What action should take next fron:
Intelligent Go-Explore
Foundation Model:
Foundation Modal: forward
Foundation Model: archive.
State
Prob
Archive
2%
Randomly sample neat action
Classic Go-Explore
Houristes
24%
state from archive
Explore from state
Update
archive
Figure I: INTELLIGENT Go-EXPLORE (IGE) integrates the intelligence and internalized human notions of interestingness from giant pretrained FMs into all stages of the Go-Explore (Ecoffet et al. 2021a b) algorithm, enabling FM agents to robustly explore in complex environments. Bottom: Classic Go- Go-Explore solved hard exploration problems by archiving novel discovered states, resetting to promising ones via domain -mmaii- heuristics, and then performing random exploration. Top: Our approach, INTELLIGENT CO-EXPLORE, en- ables Go-Explor Explore to tackle virtually any type of problem that is representable in the context of a large language multimodal model. Instead of manually defining heuristics, query the foundation model at all stages, enabling our approach automatically catch and return serendipitous discoveries, and harness the power of FM agents to explore. The environment shown the BabyAl game used Section
approach that combines the intelligence of foundation models with the powerful Go-Explore (Ecof- fet et al. b)) framework to substantially increase the exploration capabilities of FM and rein- forcement learning (RL,(Sutton & Barto (2018) agents.
Go-Explore is popular family of algorithms in deep RL based on maintaining an archive of "in- terestingly new" discovered states and then iteratively returning and exploring from the most promising states (see Figure for an overview of the three stages). This framework has led to super- human performance in range of hard-exploration problems, including sang-oriioon Atari games and robotic control. However, the algorithm's 's success largely relies on carefully hand-designed heuristics at all three stages guide exploration (Luu et al. 2024, Madotto et al. 2020). For exam- Atari game that was the previous grand
ple, in Montezuma's Revenge (Bellemare et al. 2013), challenge of exploration deep RL, (1) saved states the archive were returned to with probability proportional to factors like the number of times state has been sampled before, (2) exploration was purely via random action sampling, and (3) the criteria for which states were considered interest- ingly enough to be added to the archive depended on domain- agent visited a new location, or did so with more keys. Without this pre-specified knowledge, the quality of discovered trajectories is typically significantly worse (Ecoffet et al.) 2021b).
factors like whether the
These rigid, domain-specific choices are stark contrast to human-like exploration of a new game, where players can often intuitively judge the value or interestingness of any particular state (Cooper) 2014). More importantly, it is often impossible to know what is interesting or possible ahead of time complex domains. In the words of Isaac Asimov-The most exciting phrase hear in science, the one that heralds new discoveries, is not "Eureka!" but "That's 's funny.". With this motivation, IGE stands on the shoulders of giant foundation models and uses their intelligence judge to identify the most promising states to return to and explore from, (2) select the best actions to take from a selected state, and (3) recognize and capitalize on serendipitous discoveries states states encountered during exploration that valuable in terms of exploration, yet where what makes them interesting was not anticipated by the human user. By leveraging the foundation model's internalized notions of interestingness (Zhang al. 2024), IGE decide whether new state is interestingly new enough to be added to the archive as a stepping stone for future exploration (Figure I top).
(1) act
We demonstrate IGE's ability to reliably improve the exploration capabilities of FM agents diverse range of language and vision-base based tasks that require search and exploration. These settings include tasks that require commonsense reasoning, long-term planning and memory, and handling
2

---

Published as a conference paper at ICLR 2025
partial observability. IGE integrates well with various agent strategies, including few-shot and chain-of- thaii--f--hought-bbaeee prompting, demonstrates consistent improvements over baselines across di- verse foundation models, and will likely only get better as the capabilities of FMs continue to im- prove. While IGE performs strongly all-around, some highlights from our evaluation include: IGE reaches 100% success rate Game of 24 (Yao al., 2023a), a standard mathematical reasoning and search problem, 70.8% faster than classic graph search. On the BabyAl domain, IGE en- ables standard foundation models to succeed with visual observations zero-shot. Moreover, on the TextWorld (Côté et al., 2018) Coin Collector domain, IGE is the only algorithm that succeeds in discovering long-horizon optimal solution paths, where prior state-of-the FM agent frameworks like Reflexion (Shinn et al. 2023 fail.
INTELLIGENT Go-EXPLORE simultaneously empowers foundation model agents to reliably ex- plore, and reimagines the scope of Go-Explore to tackle virtually any type of problem, without being limited to hand-designed heuristics. These abilities will substantially improve our ability to develop more generally capable agents, and increase the range of tasks they can learn how to solve.
2 BACKGROUND
Go-Explore for Hard-EExporaan Problems. Go-Explore (Ecoffet et al. a) is a family of algorithms designed to solve hard-exploration (Ladosz et al., 2022) problems based on the principle of remembering and returning reliably to promising states. The classic setting builds an "archive" of novel states it discovers in an environment, where similar states are grouped in a single "cell". These cells are defined by heuristics like having the same visual observation when downsampled to low resolution. In the beginning, the archive only contains the initial state. We describe the overall (bottom): At each iteration, (1) promis-
structure of the algorithm in the same order as Figure [] ing states are selected from the archive through domain-specift heuristics, e.g., probabilistically sampling states proportional their progress through the environment or potential to lead to new states. The agent returns to that state by resetting using the simulator or via a coal-conditioned policy, and (2) a sequence of random actions taken explore from that state. (3) All discovered states deemed interestingly new by the cell representation heuristics are added to the archive, and the process repeats. The strength of Go-Explore due to addressing two critical impediments to exploration: forgetting how to reach previously visited states (detachment) and failing to first return to a state before exploring from it (derailment) (Ecoffet et al. 2021a).
This approach leads to collection of high-return trajectories being discovered, which may then be fed into an imitation learning (Hussein et al. (017) algorithm to produce policy that generalizes and is robust to stochasticity. We adopt similar assumptions as the original setting, by assuming an agent can return to a previously discovered state by restoring in the simulator (e.g., a reset function in RL environment). This assumption may readily be relaxed by training a policy to return to given state, or in the foundation model case, by simply prompting the model with a past trajectory.
Large Language and Multimodal Foundation Models. The combination of model scaling and training over internet-scale data has resulted in a wide variety of foundation models (Bommasani al. [02I] that exhibit generalist capabilities. In this paper, we consider autoregressive large lan- guage models (LLMs. Brown et al. (2020); OpenAl (2024); Touvron et al. (2023) which learn to generate text completions by modeling the conditional probability of new token given the pre- ceding tokens, p(zz| 0). This framework enables LLMs to not only generate coherent text but crucially also exhibit human-like abilities, including on commonsense knowledge questions (Talmor et al. 2019) and complex reasoning tasks (Wei et al. (002). These models may also be extended to other input modalities such as images by tokenizing these inputs into the same space as the text (Zhu ct al. 2023a). When prompting an FM with an instruction, the user may decide to do so with no related examples (zero-shot), with a few successful examples in related problems (few-shot, Brown Brown et al. (2200), or ask for chain of reasoning (chain-of-thouu WeeighhttWWieeeeeeeee /Wei et al. (2022) before responding.
DRIVING EXPLORATION WITH GIANT FOUNDATION MODELS
In this section, propose INTELLIGENT Go-EXPLORE (IGE) which reimagines the classic Go- Explore algorithm as described in Section g with the intelligence of giant pretrained foundation models. Specifically, we introduce FM intelligence for selecting which archived state return to and
3

---

Published as a conference paper at ICLR 2025
explore from, which action to take from each state, and deciding whether state is interestingly new and should be archived. IGE's 's use of foundation models is closely related to FM-as-a-judge (Zheng et al., 2 but instead uses foundation models proxies of human judgment of exploration choices in an environment rather than the output of generative models. We illustrate our resultant algorithm at the top of Figure provide full pseudocode in Algorithm
Wherever we query the foundation model, we introduce the overall strategy of Go-Explore along- side brief description of the current environment in the "system message" (high-level directive) displayed below. The brief descriptions for each environment we evaluate on in Section I are listed in Appendix B In the following sections, we detail our prompting techniques at each stage of IGE. The previous prompt history is visible to the agent, which enables each component of IGE to com- municate with each other. We provide precise details on how we parse responses in Appendix D.1
System Prompt.
[Brief Description Of Environment]
You will be prompted to perform systematic exploration in the style Go-Explore. An archive will be maintained of interesting states found. You will be prompted to:
Seelect state from the archive that is the most promising, i.e., likely to lead to a solution more novel states.
Explore from states intelligently, by picking new actions.
For cach new state, determine the state is interestingly new and should be added to the archive.
3.1 SELECT STATE FROM ARCHIVE
The power to easily store and return promising discovered states is crucial to Go-Explore's ability to reliably solve long-hop morgooorzonn exploration problems. IGE leverages the foundation model's 's inter- nalized notions of interestingness (Zhang al. 2024) to select the most promising state return to from the archive (Figure [ left). This is far more flexible than classic Go-Explore, which relied on hardcoded hand-crafted heuristics determine cell sampling probabilities. An example prompt is shown below.
Examples of the discovered states are given in Table I We assign indices to these states in a list and ask the FM to se- lect a numerical index. We define a budget of Nauce "state- expansions". : Each state expansion is followed by a sequence of exploratory actions, which we describe in the next section.
State Selection Prompt.
Current state archive: [Discovered states]
Select the most promising state.
3.2 EXPLORE FROM STATE
In order to effectively explore from a state selected in the previous section, we leverage the power of foundation model agents (Liu et al., 2023 Huang et al. 2022 choose how act in envi- ronment. This vastly improves on the original Go-Explore's 's use of random action sampling. One
of the key strengths of IGE that is complementary to various FM agent reasoning strategies,
including zero-shot, few-shot, or even chain-of-thonghtt-buss prompting (Yao et al. 2023b) We
demonstrate this flexibility in Section
One point of departure from the classic Go-Explore is that we additionally maintain a state-conditional action history for each archived state, so that IGE can avoid repeating previously tested options. While this information may already be avail- able in the entire history, this helps avoid any recency bias that can occur with longer contexts (Zhao et al. (202I). The action history can be easily reiterated in the prompt, the prompt could display the remaining untested actions. We define a bud- get of exploratory actions per state expansion Naction which is typically far shorter than the full horizon of the environment and represents small number of trial actions. An example prompt is shown here.
Action Selection Prompt.
ApenttSSpecffic Prompt]
Current state:
[Current State]
Previously tried actions:
[Previous Action History
Output the next action.
4

---

Published as a conference paper at ICLR 2025
Table 1: We show that INTELLIGENT Go-EG EXPLORE can efficiently explore over diverse set of environments with increasing difficulty. We showcase IGE on environments with both language and vision-bass based obser- vations. For each environment, we provide an example observation, samples from the action space, and the horizon of the task in the environment.
Game of 24
Baby-A (Text and Visual)
TextWorkd
mathemutical reasoning search
partially observable gridworld with language instractions
partial observability, long-term plan- ning and memory, and commin seuse
Problem Type
"Curent (2
"Gool: unlock door. You see a wall 4 steps forward, You see yellow box steps left." (Text-based)
paniry... shelf. The shelf is weoden. On the shelf you can see flour...
"You arrive
Observation
(Visoo-bbssl)
Next Actions
2 8 = 10 Next: (8 10 14)
8 m Next (4.8 14)
Next:
turn left
right
pick up
open door drop
go cust
-ook potato with oven
-ulock door key
go
Task Horizon
64 or 128
25, 40 or 80
3.3
UPDATE ARCHIVE
IGE stores discovered states in an archive, allowing the algorithm to return and explore from those points. To encourage better exploration, the stored states should be interesting- either promis- ing states relevant to the task that could lead to further stepping stones or novel states that differ significantly from existing ones. As in other stages of IGE, we leverage the foundation model's internalized notions of interestingness (Zhang et al. 2024) to evaluate the interestingness of each state.
While the original Go-Explore required extensive domain knowledge determine interestingness, IGE avoids this requirement and manual labor, critically gaining the ability to recognize and capi- talize serendipitous discoveries that could not have been predicted ahead of time. In practice, propose two options to filter discovered states after a sequence of exploratory actions. The first is to iterate through every new state and ask whether each one is interestingly new and should be added to the archive. The second is to first add all states and then ask the foundation model to remove the uninteresting states. We discuss this choice later in Section 4.3] the second form is preferable in larger environments where there is more need to explicitly deprecate earlier discoveries that have become irrelevant as overload the archive. An example prompt for the first option is shown below.
By default, IGE implements the foundation model at all three stages of Go-Explore, but we rigorously ana- lyze the relative importance of each component Sec-
Archive Filtering Prompt.
Current state archive: [State Archive]
tion 5 Our focus is on the discovery of solutions to hard-exploration problems. These solutions could po- tentially be used for downstream reinforcement learn- ing or even improve the foundation model subse- quent tasks through in-context learning, representing exciting avenues for future research.
New state: [Current State]
Is this state interestingly new (a novel state that is relevant to the task or could lead to further stepping stones), such that it should be added to the archive?
EMPIRICAL EVALUATION
In this section, we evaluate INTELLIGENT GO-EXPLORE across a diverse set of text environments that require search and exploration. We demonstrate IGE's ability to handle partially observable and complex observation spaces (including both text and visual inputs), discover solutions involving long chains of actions, and effectively improve the ability of FM agents to explore. For all our experiments, we use GPT-4 (OpenAl 2024). one of the current SOTA LLMs, as our foundation model. We compare IGE to random action sampling, a naïve LLM baseline, and two SOTA FM agents, ReAct (Yao al. 2023b) and Reflexion (Shinn et al. 2023). All methods receive the same environment descriptions, observations, and use the same number of environment steps, ensuring a fair comparison. Naive LLM simply queries the LLM for action conditional on the interaction history. ReAct prompts the agent to output its reasoning before making a decision. Based on ReAct, Reflexion further conditions the agent on the previous attempted episode, asking the agent to learn
5

---

Published as a conference paper at ICLR 2025
from its mistakes. We provide an overview of our environments in Table 1 Full hyperparameters are detailed in Appendix
4.1 GAME OF 24
We first demonstrate the effectiveness of IGE in mathematical reasoning task, Game of 24 Yao
eet al. 2023a). The goal is to perform basic arithmetic operations (-, x/) starting from 4 numbers to obtain 24. For example, given input (4, 9, 10, 13), a possible solution could be (10 - 4) x (13 9) = 24. We formulate the problem as an MDP (Sutton & Barto 2018). where actions represent a reduction of two numbers by an arithmetic operation i.c., the above solution
would be represented the sequence of state transitions (4, 10) 10-4=6, (6,9. 13) 18-9=4,
(6,4) 6x4=21, (24). Therefore, IGE uses the FM to iteratively expand possible solution paths and archive promising ones to return to. The action space is the range of possible next operations, displayed in the same manner as in Yao et al. (2023a).
Game Of 24
We evaluate IGE across 100 hard test problems in Fig- 2 and additionally include the standard (unweighted) graph search algorithms depth-first search (DFS) and
1.0
0.8
Success Rate
breadth-first search (BFS) as reference. Since the com- binatorial complexity of the problem is at most (t) (2)
0.6
0.4
43 = 1152, graph search is guaranteed to find a solu- tion within that many actions. The system prompts for both IGE and the LLM baselines contain few-shot exam- ples with correct calculations on different starting num- bers. IGE rapidly reaches 100% success rate, on average 70.8% faster than the next best baseline, depth-firse firth search (DFS) this improvement is statistically significant (x² test, p < 0.05) at 150 operations, where IGE has solved all problems. This success may be attributed to the fact that language models have internalized mathematical in- tuition and are likely to be able to identify promising pairs like (6, 4) that could easily be multiplied together for a solution.
0.2
0.0
0
200
400
600
Operations Performed
Random Actions
RoAst
DFS
Reflexiin
BFS
Intelligntt Go-Explare
Naive LLM
IGE explores the Game of 24 with the intelligence of FMs and reaches 100% success rate on average 70.8% faster than DFS, the next best baseline. IGE com- pletes all problems within 150 enviromment operations. Our use of archiving and intel- ligent action selection allows us to greatly outperform prior LLM agents with an equal number of operations performed. The suc- cess rate is computed over 100 test prob- lems.
Figure 2:
All LLM agent baselines (naive LLM, ReAct, Reflexion) eventually plateau and even get beaten by the unintelli- gent DFS. This highlights the need for diverse action se- lection, which IGE enables. A final point of comparison we make is to Tree of Thoughts (ToT. Yao et al. (2023a)) which achieved 74% on Game of 24 within their evalu- ation budget. We emphasize that our evaluation setting is very different as IGE selects from the list of valid options rather than doing the math in context. However, we note the key difference to our method is that ToT evaluates and expands multiple reasoning paths following a tree structure, whereas IGE can easily jump around the search space-this this a crucial advantage in more complex environments (like those in the following sections), where it takes many coordinated actions to get from one state to another interesting state.
4.2 BABYAI TEXT AND VISUAL DOMAINS
Next, we show that IGE readily operates across multiple modalities in the BabyAI domains from Carta et al. (2023). The original domain is world with text-based observations. The agent is given a textual goal instruction which could corre- spond to one or more instructions in a sequence, e.g.. "pick up X and then go to Y". As we can see from the observations in Table [ the task is challenging even for humans to complete and requires forming model of the world from partial observations. This kind of state observation would make it hard to define heuristics to determine how good any particular state is, as in classic Go-Explore We additionally extend this to visual domain by replacing text observations with partially ob- servable image observations. IGE can naturally handle these by simply replacing text observations with images passed to the multimodal GPT-4o. The optimal path solution may include moving
weredurally-generatte partially-observaal
2D grid-

---

Published as a conference paper at ICLR 2025
GoTo
PickUp
PicKUpThenGooo
OpenDoor
PutNexITo
1.0
1.
1.
1.0
1.0
Success Rate
0
0.6
0.6
0.6
0.6
0.6
0.4
0.4
02
0.6
CLAM
Random
Nalvo
Intellgant Go-Expiora
Bl- Text
PickUpp
PickUpThenGool
OpenDoor
PotNextTo
Success Rate
72%
50%
94%
95%
42%
24%
12%
60%
50%
12%
4%
60%
2%
0%
0%
8%
6%
9%
5%
52%
Random Actions
Naive LLM
ReAct
Inteligent Go-Explore
(b) Baby Visual
Figure 3: IGE can enable GPT-4o to efficiently find solutions to challenging tasks in the BabyAl text and visual environments. In the text domain, IGE does so with orders of magnitude fewer online steps than prior RI.-trained baselines (GI.AM, [Crrta et al. (2023). Task types in order of difficulty. As tasks become more difficult, the performance gap of IGE confidence interval (Zoubir & Iskandler 2007) over 25 seeds per environment type. Here, and elsewhere, confidence intervals are obtained by bootstrapped resampling 10,000 times.
the LTM baselines grows. We show the mean and 95% bootstrap
blocking objects well as finding keys to open doors. We consider 5 different task families of increasing difficulty: "go to", "pick up"" "pick up then go to", "open door", and "put next to", which described fully in Appendix[B.2 [B.2]
We omit the Reflexion baseline in this environment due to the high cost of querying GPT-4 with 128-step episodes in the context. Due the complexity of this environment, we use chain-of- thought prompting in all three components of IGE. This allows the FM to deliberate on the state of the game before making decisions. IGE can find solutions to these problems with only tiny budget of 250 environment steps per task (divided into rollouts of 10 exploratory actions each) and visualize the final performance Figure 3 (top). In the text variant, IGE and ReAct vastly outperform the prior RL-trained language model approach, GLAM (Carta et al. 2023), with orders of magnitude fewer samples (GLAM used 1.5M online steps) and requiring no training whatsoever. IGE achieves the best or close to the best performance in every task. The gap between IGE and the second-ber method grows with task difficulty, with a statistically significant 36% improvement (2 test, p 0.05) on "put next to". In the visual variant, IGE also strongly beats the visual agent baselines and displays strong performance across the board, enabling GPT-4o to successfully find solutions in majority of cases. The results provide strong evidence that IGE extends any modality representable by multimodal foundation model.
4.3 TEXTWORLD
Finally, we show IGE's ability to tackle tasks requiring long-horizon memory and planning, explo- ration, and commonsense in TextWorld (Côté et al. 2018), classic text-based agent benchmark.
Treasure Hunter
The Cooking Game
Coin Collector
Success Rate
1.0
1.0
3.0
05%
64%
84%
80%
96%
0%
40%
56%
52%
92%
0%
0%
0%
0%
85%
Reflexion
Intelligent Go-Explore
Figure 4: IGE outperforms state-of-the-art FM agents in three challenging text games in TextWorld. These results illustrate the powerful capabilities of planning, commonsense reasoning, and exploration of IGE (Sec- tion 4.3). Notably, the Coin Collector game where hard exploration is required, we observe BFS-like search behavior emerge in IGE, enabling it to find the most efficient solution where all other approaches exhaust the environment horizon. We show the mean and 95% bootstrap confidence interval over 25 seeds for each game.
7

---

Published as a conference paper at ICLR 2025
We rigorously ablate the design choices in IGE and analyze the importance of incorporating FMs at each stage of the algorithm. Additionally, we compare IGE to Classic Go-Ex Go-Explore (Ecoffet et al [2021b), which relies on manually defined heuristics at each stage. Game of 24 performance is taken at 150 environment steps, over 100 evaluation seeds. A-Ty performance is taken at 250 environment steps, over 25 evaluation seeds. TextWorld performance taken at 240 environment steps, over 25 evaluation seeds. 'Standard' We show the mean and 95% bootstrap confidence interval for the success rate.
Table 2:
mirrored from Section
Success Rate (%)
Ablation Variants
Game of 24
BabyAl (PN)
TextWorld (CG)
IGE
100 0.0
84 14
92 1 10
Intelligent action selection
68 9.0
24 16
0 +
Intelligent selection
96 3,5
48 20
76 + 16
Intelligent archive filtering
93 5.0
64 20
64 + 20
X All 3 above
61 1 9.5
41
0 1 0
X State-condittonal action history
33 + 9.0
+ 16
72 + 16
Classic Go-Explore (Visitation Frequency)
38 - 22.0
4
0 + 0
We consider three challenging games in TextWorld: Treasure Hunter, The Cooking Game, and Coin Collector. In each game, the agent needs to complete the task while navigating maze of different rooms, while only seeing the current room's description in text. The agent interacts with the world using free-form natural language commands, such "go east" or "cook potato with oven." We set each game to hard difficulty; details on game customizations are provided in Appendix [B.3] As in the previous section, we use chain-of-thought prompting in all three components of IGE. Because the state archive in this environment grows significantly, we implement rejection-based archive fil- tering, which we describe in Appendix D.2
We present success rates achieved on the three games using IGE and the baselines in Figure We observe that IGE outperforms all other baselines, with a statistically significant (x² test, p 0.05) gap between IGE and the second-best method in the harder Cooking Game and Coin Collector. In The Cooking Game, IGE outperforms the next-best agent, ReAct, by a large margin of 36%, demon- strating IGE's ' advantage in hard- exploratto problems. In Coin Collector, IGE is the only method that can find the solution in the maze, with all other methods completely failing. Interestingly, we observe that IGE exhibits BFS-like behavior, intelligently selecting rooms with unexplored direc- tions and iteratively removing rooms with exhausted directions. This results in IGE almost always finding the shortest path to the target, while other methods fail to navigate the maze.
We highlight that Reflexion does not improve over ReAct in all the games we tested. Although Reflexion should in theory be an improvement over ReAct with the experience from previous at- tempts, it tends to decrease performance. We hypothesize that in long-horizon environments, the history becomes too long after the initial episode and prevents Reflexion from effectively utilizing knowledge from the previous episode. In contrast, IGE uses the FM to iteratively filter interesting states in the archive, which ends up controlling the context length. This helps IGE truly make use of the cumulative knowledge gained through exploration.
5 ANALYSIS
In this section, we analyze (1) the importance of FM intelligence for each of the three key compo- nents of Go-Explore, and (2) how IGE's performance improves as the FM's capabilities increase. We take a representative sample of environments from the previous section of Game of 24, Put Next To (PN) from AIby-A-Text, and The Cooking Game (CG) from TextWorld. Hyperparameters are listed in Appendix E
How Important is Foundation Model Intelligence at Each Step? First, we analyze the impact of FM intelligence on each component of INTELLIGENT GO-EXPLORE. We ablate replacing state and action selection with uniform random sampling, archive filtering with saving everything to the archive, and not maintaining a state-conditional action history. We use these unintelligent choices, as it would be very time-consuming to attempt to design the right heuristics based on the rich text observations in Table I In Table 2 we observe that where the intelligence of FMs is more valuable varies by environment. Since the environment horizon is only 3 in the Game of 24, the most impor- tant factor is ensuring that the actions tried are diverse and intelligently selected. This hypothesis is confirmed: the largest performance drops occur when removing either FM action selection or the
8

---

Published as a conference paper at ICLR 2025
action history. Different IGE components are most helpful in both of the longer-horizon BabyAl- Text and TextWorld environments: intelligent state selection and archive filtering make a big impact, showcasing the strength of enabling IGE to return to promising discovered states. There are smaller performance drops when removing the action history; likely because in larger environments, many more unique states discovered, so there is less gain from preventing taking the same actions from frequently returned to states. In both environments, we also observe a drastic decrease when switching to random actions, as in classic Go-Explore. This underscores the substantial benefits IGE provides in harnessing FMs for action selection.
Next, we elucidate the need for intelligent archive filtering across all our environments. Not only does archive filtering improve performance, but it also drastically reduces the number of uninterest- ing states in the archive. For instance, in BT-y we observe the archive becoming around 8x larger without filtering. These metrics demonstrate IGE's innate ability to capture promising discov- eries as they occur and focus attention on them, without the need for any manual heuristics, Detailed analysis and results are provided in Appendix C.I Furthermore, we compare IGE with classic Go- Explore (GE, Ecoffet et al. (2021b), which adopts fixed pre-defined, hand-designed, open-loop heuristics like resetting to states with probability inversely proportional to the state- vastt-vistttioon count. Across all domains, IGE significantly outperforms classic GE, which shows that adopting FMs instead of manually designed heuristics not only saves the effort required for domain-specific de- sign but also enhances the algorithm's 's exploration by leveraging the human notion of interestingness captured by FMs (Table bottom). We particularly emphasize that the intelligence of FMs enables IGE to rapidly solve environments like Baby AI that have challenged the RL community in an ex- ceptionally short number of 250 timesteps. This is in contrast the classic GE paradigm, which used millions or even billions of environment steps (Ecoffet et al. 2021b).
What is the Effect of Foundation Model Choice? We further analyze the dependence of IGE on the capabilities of the underlying foundation model. We evaluated IGE using other foundation models, including Claude Sonnet 3.5 (Anthropic 2024) and the open-source Llama-3 400B [Llama Team 2024), on the Game of 24. The results, presented in Table show that IGE maintains superior performance over the baselines across different foundation models, consistently achieving higher success rates. For instance, even when using the weaker Llama-3 400B, IGE achieves success rate of 98%, significantly outperforming the best Llama-base baseline. Notably, the performance gains of IGE are statistically significant (for all baselines with 3lama-3 400B and for all except Reflexion on Claude Sonnet 3.5, p < 0.05). This demonstrates that IGE is not dependent on a specific model and can be expected to perform well with other, possibly future, foundation models with higher capabilities. Additional experiments on prompt robustness are provided in Appendix C.2
Table 3: IGE consistently outperforms other LLM agent baselines with a diverse range of foundation models and show the mean and 95% bootstrap
on the Game of 24. We use the same evaluation setting as Table 2 confidence interval for the success rate (%).
Model
Naïve LLM
ReAct
Reflexion
IGE
GPT-4
70 + 12
82 + 11
83 + 10
100 = 0
Llama-3 400B
44 + 14
68 + 13
54 # 14
98 +
Claude Sonnet 3.5
24 +
1
80 1 11
1
RELATED WORK
Due to space constraints, extra related work discussions are in Appendix
FM Agents. One of the key strengths of IGE is that it is agnostic to the precise agent formulation and thus strictly additive on top of wide variety of strategies. A common strategy is chain-of- thought-basee methods (Yao et al. 20236] Hu & Clunc 2024). which prompts the FM to output a set of reasoning steps before the answer. We integrate this into the FM guidance our experiments in Sections 4.2 and 4.3 Reflexion (Shinn et al. 2023) enables an agent to improve over multiple episodes by asking it to reflect on the previous attempted episode and learn from its mistakes. How- ever, we show this can break down in tasks with long horizons, while IGE proposes more efficient way to filter out the vast majority of uninteresting interactions. Another set of agent frameworks that related to the idea of exploring diverse solution paths via caate-connectivtyy is Tree of Thoughts (ToT, Yao et al. (2023a) and Graph of Thoughts (GoT, Besta et al. (2024a)) The primary difference between methods like ToT and IGE is that ToT builds up tree of abstract thoughts generated by the

---

Published as a conference paper at ICLR 2025
language model and could be hallucinated, whereas IGE's archive is grounded in real states directly copied from an RL environment. Although ToT's thoughts are suitable for the types of problems investigates. these abstract plans could easily break down in RL environments with long horizons. In contrast, IGE scales gracefully with long horizons by archiving real states and returning to the state for exploration. Additionally, ToT is based on DFS and BFS over the tree and is thus limited to these search strategies over the problem. On the other hand, IGE can intelligently select hybrid strategies, for example, going broad at the start (more BFS-like) and then honing down on a promis- ing path (more DFS-like) later on. This is particularly important for long-horizon tasks with larger state spaces, as we show in Sections 4.2
and 4.3
Closely related to exploration, FM agents have also begun to see use in search- based tasks. Stream of Search (Gannhi et al., 2024) considers a similar mathematical reasoning task to the Game of 24 and seeks to initially clone the actions of graph search algorithms, then use RI to self-improve. In con- trast, IGE already greatly outperforms classic graph search, and an exciting future direction could be to first clone the exploratory behavior of IGE and then use that as basis for self-improvement. Lehnert et al. (2024) analogously train a language model to mimic the A algorithm. Finally, Kr- ishnamurthy et al. (2024) also consider bootstrapping exploration with an externally summarized action-history in bandit problems; our focus is more on the detection of interesting states.
Go-Explore. The original Go- -0-EExlore (Ecoffet et al. 2021a b) framework enabled superhuman per- formance in a variety of hard-exploration problems, including applications as diverse as automated game testing (Lu al. 2024). (Gallouddec & Dellandréa (2023) propose Latent Go-Explore which similarly aims to address the difficulty of designing exploration heuristics by automatically learning a latent representation and sampling states with low latent density. However, this requires periodic retraining and could easily miss out on rare discoveries. HuGE (Torne Villasevil et al. (023) guides Go- Explore with humans in the loop by asking for pair-wiss feedback on which goal to select. On the other hand, we replace humans with intelligent FM guidance at all components of Go-Explore.
7 CONCLUSION AND LIMITATIONS
In this paper, we demonstrate a new approach to robust exploration in complex environments, IN- TELLIGENT GO-EXPLORE, reimagining Go-Exxlore in the era of giant foundation models. We show that IGE can drive exploration for a diverse set of FM agents, including few-shot and chain-of- thought prompting, across a variety of modalities including challenging text and vision-based games. Extending IGE to even more modalities could unlock applications as wide as scientific discovery in synthetic biology (designing novel drugs or proteins) or material science. Recent advancements in multimodal foundation models, such as RT-2 (Brohan et al. 2023) and RFM-1 (Covariant AT 2024), have shown the potential of FMs to handle various modalities, including text, images, videos, and (continuous) numerical sensor readings. Since these models have already tokenized the state-action spaces of general environments and shown the ability of the model to generate actions for any state, it should be possible to then ask the FM to judge the interestingness of any given state compared to prior states observed, which is the only additional requirement of IGE. A further interesting do- main is the (hitherto unsolved by intelligent agents) vast dungeon crawler, NetHack (Küttler al., 2020). Küttler et al. (2020) noted that classic Go-Explore's heuristics "will likely not work for large symbolic and procedurally generated environments." IGE represents a sharp departure from these limitations by replacing hard-coded and inflexible exploration heuristics with the dynamic intelli- gence of giant foundation models.
There remain exciting opportunities to improve IGE's capabilities to explore vast state spaces. For example, we currently recall and compare against the entire archive whenever we discover new state. This could be made much more efficient by using techniques like retrieval augmented gen- eration (Lewis et al. 2020) and only comparing to the closest previously discovered states. As we consider IGE for real-worke settings, we should take steps to ensure the responsible deployment of FMs (Bommasani et al. 2021). Our approach opens up the road to safe and interpretable explo- ration: through careful prompt engineering or techniques like constitutional AI (Bai et al. (022), we could steer the agent away from unsafe behaviors. Furthermore, if we ask or train the FM to explain its choices in each part of IGE, we could gain insight into its rationale for exploring particular paths through an environment (Wei et al. 2022 Hu & Clune 2024); improving safety, interpretability, and perhaps one day even own understanding of how best to explore.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Vector Institute, the Canada CIFAR AI Chairs program, grants from Schmidt Futures and Open Philanthropy, an NSERC Discovery Grant, and generous donation from Rafael Cosman. We thank Aaron Dharn,, Ben Norman, and Jenny Zhang from our lab the University of British Columbia for insightful discussions and feedback on early drafts of this work.
REFERENCES
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https: www cdn. anthropic.. com wecoom/eeele (abeebbabblccc 3bbb///bbbbbbbb333bbbbbbbb http://bbbb68877762
Card Claude_3.. pdf
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer. Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from feedback, 2022.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: eval- uation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013. ISSN 1076-9757.
doi: 10. 10.1613/jaate 1.jii//iii..91.. URL /ttt:://dxe doi http:://co..aa..ooo11111.1..11
jair 3912
Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J. Hu, Mo Tiwari, and Emmanuel Bengio. Gflownet foundations, 2023. /ut//tttp///rrxxx..rr//baab///// 092666
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gi- aninazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. Pro- ceedings of the AAAI Conference Artificial Intelligence, 38(16):76822177600 March 2024a. doi: www.////a......16..... aaai v38 29720
ISSN 2159-5399,
URL http://dx doi org/ 10..1..66.9999
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024b.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, and Michael S. Bernstein et al. On the opportunities and risks of foundation models. ArXiv, 2021.
URL htttt:://///rttit
Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-diversity through ai feedback, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huel Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong. Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for rel-world control at scale. In Xrviv preprint vrXiv:2212.068/7 2022.
11

---

Published as a conference paper at ICLR 2025
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexan- der Herzog, Jasmine Hsu, Brian lchter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao. Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RI-2: Vision-language-actionn models transfer web knowledge to robotic control, 2023. URL
2307. 15818
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Pierre-Yves Oudeyer. Grounding large language models in interactive environments with online reinforce- ment learning. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 3676-3713. PMLR, 23-29 Jul 2023. URL
wttp:///nan.......rre////////2 wwtppppwwwwss...eeeeppppep carta22a..
html
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang. et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
Seth Cooper. A framework for scientific discovery through video games. Morgan & Claypool, 2014.
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-based games. CoRR, abs/1 0ss/1806.15 1532, 2018
Covariant AI physics, 2024.
Rfm-1:
A
world model that understands https: ai/ aitinsighnnn////////////
URL
rfm 1-n-l-a-wale watawwwrlll 1- -
Adrian de Wynter. Will gpt-4 run doom? arXiv preprint arXiv:2403.05440 2024.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301. ....34400004,,... 2022.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth Stanley, and Jeff Clune. First return, then explore. Nature, 590:580-586, 02 202 doi: (033//41588000000311111
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems, 2021b.
Quentin Gallouédec and Emmanuel Dellandréa. Cell-free latent go-explore, 2023.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D. Goodman. Stream of search (sos): Learning to search in language, 2024.
Shengran Hu and Jeff Clune. Thought Cloning: Learning to think while acting by imitating human thinking. Advances in Neural Information Processing Systems, 36, 2024.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents, 2022.
12

---

Published as a conference paper at ICLR 2025
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv. 50(2), apr 2017. ISSN 0360-0300. doi: 10.1145/3054912. [tRLhttts://eatt org/10 1ttp://10.114/335991/1
Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethinking exploration. Royal Society Open Science, 10(6):230539, 2023.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Lue Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback, 2023.
Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context?, 2024.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684 2020.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022.
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling. Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. Lab- bench: Measuring capabilities of language models for biology research, 2024. URL https:
7/arxiv. org/ abs 221.//477...1.66
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with AI feedback, 2024. URL /wLLtttts//peeeeeeeeee.oo
Lucas Lehnert, Sainhayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mevay, Michael Rabbat, and Yuandong Tian. Beyond a#: Better planning with transformers via search dynamics bootstrap- ping. 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmenned generation for Knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 9459-9474. Curran Associates, Inc.. 2020. URL bttps::/proceeding www.isss.euuispp. cc/paper salles/paper/202c/ File/ 6b4 /0l1://6332222007700000333433333
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating lims as agents, 2023.
Llama Team. The llaaa 3 herd of models, 2024. URL htttps://arxvv.or///attt
21783
Cong Lu, Raluca Georgescu, and Johan Verwey. Go-explore complex 3-d game environments for automated reachability testing. IEEE Transactions on Games, 16(1):235-240, 2024. doi: 10. 1109/TG.2022.3228800
Andrea Madotto, Mahdi Namazifar, Joost Huizinga, Piero Molino, Adrien Ecoffet, Huaixiu Zheng, Alexandros Papangelis, Dian Yu, Chandra Khatri, and Gokhan Tur. Exploration based language learning for text-based games. arXiv preprint arXiv:2001.080 008868, 2020.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph. Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint 22X9 1/895, 2022.
13

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4 technical report, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https:t/atxiiiiii org. htsp/bb//22233 02155
Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404. 10179, 2024.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
Richard S. Sutton and Andrew G. Barto, Reinforcemmet Leaminng An Introduction. The MIT Press, 2nd.
second edition, 2018. URL
html
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 16553/11N111112222 URL https: ptteps:://aclaanhhllo.oooo
N19-1421
Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
Marcel Tome Villasevil, Max Balsells I Pamies, Zihan Wang, Samedh Desai, Tao Chen, Pulkit Agrawal, and Abhishek Gupta. Breadcrumbs to the goal: Goal- conditioned exploration from human-in-the-loop feedback. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp. 63222-63258. Curran Associates, Inc., 2023. URL
www.eri.......eaeeeeeee Panpeereeeeereeeeeeepepppee .ddf
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-483 2024.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. tfain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with large language models, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Con- ference on Learning Representations, 2023b. URL https:://///.t //pp:///ppeeevieww net/ forum?
id=WE_ vluYUI--X
14

---

Published as a conference paper at ICLR 2025
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. OMNI: Open-endedness via mod- els of human notions of interestingness. The Twelfth International Conference on Learning Representations, 2024. URL
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pp- 12697-12706. PMLR, 2021.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Sto- ica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Informa- tion Processing Systems, volume 36, pp. 46595-46623, Curran Associates, Inc., 2023a. https::/pproeeddngg....eeerr neur ce/paper hatpaaper__fllsppppprrreee wtt.es//pppperr/22//11111 9 11 a Benchmarks. rks.
URL
and
pdf
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023a.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. arXiv preprint 17114 2023b.
Abdefihak Zoubir and Robert Iskandler. Bootstrap methods and applications. IEEE Signal Processing Magazine, 24(4):10-19, 2007.
15

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL
TABLE OF CONTENTS
Algorithm Pseudocode
17
Further Details on Environments
17
B. Game of 24
17
B.2 BabyAl
18
B.3 TextWorld
19
Additional Experiments and Analysis
21
Archive Size Analysis
21
Prompt Robustness Experiments
22
Further Prompt Discussion
22
D.I
Extracting Choices
22
D.2
Rejection-based Archive Filtering
................................
23
D.3
Implementation of State-Conditional Action History
23
D.4
Handling Hallucinations during Trajectory Creation.
23
Hyperparameters
24
Cost of Experiments
24
More Related Work and Future Work
25
16

---

Published as a conference paper at ICLR 2025
A ALGORITHM PSEUDOCODE
We provide full pseudocode for INTELLIGENT Go-EXPLORE in Algorithm the discussion Section 3
This complements
Algorithm INTELLIGENT GO- CO-EXPLORE
1: Hyperparameters: no, state expansions Name no. exploratory actions Naciion foundation model M
2: Initialize: archive states Suctive 0, state-conditional action history AA.)
0
3: Sunciive 4- Sarchive U {
D Add initial state to archive
4: for io =.,..... Natate do
5: Query M for the next
8,,1 from Sunchive
See Section 3.1
for j =-= L...., Naction do
7:
Query .M for the next action di,y from 81,, conditional on A(ss.))
D See Section 3.2
8:
$1.5-1 ~ P(s.jj. (s A(st.) A(st,) U {ac.)
Take action and update history b Sec Section 3.3
9:
if .M determines that 81,++1 is interesting w.r.t. Surtive then
10:
S Sartiive U (81.+1)
11:
12:
end for
13:
end for
14:
Return best discovered trajectory
B FURTHER DETAILS ON ENVIRONMENTS
We provide further details for each the environments used in the empirical evaluation in Section 4 and the anvironment-speccne descriptions appended to the system prompts. Each environment de- scription may include high-level information about the task and a description of the action space.
B.
GAME OF 24
com/ 11 which is released under the MIT License. We include the eivironment-speccfic prompt that is appended to the system prompt in Section 3 below. The system prompt contains examples of correct reasoning paths on different problems (few-shot prompting).
We use the environment and set of evaluation tasks from princeton-n wttpo///p/peeeeteeeeeeeeeeeeeeee
Environment Description.
You are given numbers must use basic arithmetic operations * /) to obtain 24. each step. you are only allowed to choose two of the remaining numbers to obtain new number. correct answer is one that uses each input exactly once and no other numbers. Reaching 24 before the last step does count correct answer. Follow the convention that division is integer division, and never by zero. Some examples of correct reasoning traces follows: Initial state: 4 8)
Steps:
4+ = 12. Next: (4 6 12)
6-4=2. Next: 12)
2 12 Next: (24)
Answer: 4)* (4 8) = 24
Initial state: (2. 10 12)
Steps:
12 * 2 24. Next: (9 10 24)
10-9 Next: (1 24)
1 24. Next: (24)
Answer: (12 2) (10 9)= 24
Initial state: (49 10 13)
Steps:
13 10 3. Next: (3 4))
9-3 = 6. Next: (4 6)
4 4 6=6= = 24. Next: (24)
17

---

Published as a conference paper at ICLR 2025
Answer: 4 (93 10))
The action space at each step is all the valid arithmetic operations, presented in an analogous way as the 'propose' step Yao al. (2023a).
B.2 BABYAI
The BabyAl (Carta et al. 2023) environment comes with five task types, which we list here and visualize in order in Figure 5
Go <object>> simple navigation task that requires reasoning abilities to choose the right plan given the object's s position;
Pick up <object>, reasoning task that combines navigation tasks;
Pick up <object A> then go to <object B> and Go <object B after pickup <object
A>, both serving to test reasoning abilities on temporal sequences;
Unlock <door>, task that includes inferring that a key needed to unlock the door,
finding the right key (i.e., the one colored as the door), and eventually using the toggle action with the key on the door;
Put <object A> next <object B>, which requires first reaching <object A>, picking it up, reaching <object B> and finally dropping <object A> next to <object B
pick up yellow box, go yellow
put red key next yellow
go yellow
upck
open
Figure 5: We visualize the types of tasks that BabyAl consists of for our evaluation in Section 4.2 IGE
receives only partial text-hased observations corresponding to the view in the figure.
We use the codebase from https::/ with_onlines wwooiinn_RReoo RR which is released under the MIT License. The action space is discrete and composed of 6 possible actions: turn left, turn right, go forward, pick up. drop, and toggle. The "go to' and 'pick up' tasks have shorter environment horizon of H 64, whereas the rest have horizon of = 128. We include the environment-speen -speciifi prompt that is appended to the system prompt in Section 3 below.
LLMs
Environment Description (Text).
You are an agent in partially-observalee 2D lext-based environment. You see the 6x6 grid in of you, and can face north, south, east, or west. The possible actions are turn left, turn right, go forward, pick up. drop, and toggle. At each turn, you will receive history of the last observations and actions. Your aim is complete the task described in the goal. Each tile in the grid can only contain at most object. Objects cannot crossed, and may need be bypassed or moved. You can only move onto an empty tile or on tile containing an open door. You can only hold one object at a time, using pick up when they are one step in front. Objects are dropped one tile in front and cannot be dropped when there is another object in front. Doors unlocked with keys of the same color using the toggle action. Actions are deterministie, task.
repeat actions if they have effect. You have H steps complete the
Environment Description (Visual).
You are an agent an 8x8 partially-observable 2D image-hased environment. see grid in front of you, and can face north, south, east, or west. The possible actions are turn left, turn right, go forward, pick up, drop, and toggle. At each turn, you will receive history of the last observations
18

---

Published as a conference paper at ICLR 2025
and actions. The observations will be presented as a sequence of images. Your aim is to complete the task described in the goal. Each tile in the grid can only contain at most one object. Objects cannot be crossed, and may need be bypassed moved. You only move onto empty tile or a tile containing an open door. You can only hold one object at time, using pick up when they are one step in front. Objects are dropped one tile front cannot be dropped when there is another object in front. Doors are unlocked with keys of the same color using the toggle action. Actions are deterministic, do not repeat actions if they have effect. You have H steps complete the task.
B.3 TEXTWORLD
We evaluate IGE on 'Treasure Hunter', "The Cooking Game', and "Coin Collector' from the TextWorld (Côté et al. 2018) domain. We use the environment code from https://github. which is released under the MIT License.
B.3.1 TREASURE HUNTER
For Treasure Hunter, we the 'level' option to the maximum value of 30, resulting in maze with 20 rooms. Locked doors and containers are added, which may need to be unlocked and opened to find the target object. To further increase the difficulty, we remove the solution description from the original game and filter out tasks that can be completed with 20 steps in the optimal solution. We include the mavironment-speciff prompt that is appended the system prompt Section 3 below.
Environment Description for Treasure Hunter.
You are an agent playing TextWorld. bext- adventure game where you are in a randomly generated and must find specific object. You need to explore different rooms to find the target object.
Here are the available commands: look: describe the current room. goal: print the goal of this game inventory: print the player's inventory go <dir>: move the player north, cast, south, or west. You can only go in the direction indicated with an exit or a door, open ...: open a door or a container. You need to open closed door before you want go through it. drop ...: drop an object on the floor take ...: take an object that visible. Make sure the object is visible take. put ... ... place an object a supporter take from ...: take an object from container or supporter insert into ...: place an object into container unlock... with ...: unlock a door or container with a key. You need to unlock a locked door with a matched key in your inventory before you want to open it.
The target object might be located in a closed or locked container. The adjective is useful for de- termining whether the key is matched with the lock (e.g.. non-euclidean keycard matched with non- euclidean safe). Make sure is matched to unlock! The key required to unlock the door may in another room or locked inside a container. Take the key whenever you can. After unlocking a locked door or container, it will remain closed. You will then need to open it. You have 40 steps complete the task. Restarting is forbidden.
B.3.2 THE COOKING GAME
In The Cooking Game, we set the number of ingredients to maximum of 5 and the number of rooms to 13. We enable all challenging additional options: doors need to be opened, food must be processed (e.g., cut, diced, chopped with a knife), and cooked (c.g., grilled with a BBQ, fried on stove, roasted in an oven). We include the environment-specffic prompt that is appended to the system prompt in Section below.
Environment Description for The Cooking Game
You agent playing TextWorld, text-based adventure game where you navigate through different rooms, interact with objects, and solve puzzles. Your goal to first find the recipe, find and prepare food according to the recipe, and finally prepare and cat the meal.
Here are the available commands: look: describe the current room goal: print the goal of this game inventory: print player's inventory go <dir>: move the player north, east, south or west. You can only go directions indicated with exit door. examine ...: examine something more closely eat edible food open ...: open door or a container. You need open closed door hefore you can go through it. drop ...: drop an object onto the floor take ...: take an object that is visible put on ...: place an object on supporter take from ...: take an object from a container or supporter insert into ...:
...
19

---

Published as a conference paper at ICLR 2025
place an object into a container lock with ...: lock a door or a container with key unlock with ...: unlock a door or a container with a key cook with ...: cook cookable food with something providing heat slice ... with with something sharp dice... ... with ...: diee cuttable food with something sharp prepare meal: combine ingredients from inventory into a meal. You can only prepare meals the Kitchen.
slice cuttable food with something sharp chop ... with ... chop cuttable food
You can examine the cookbook to see the recipe when it is visible. The BBQ is for grilling things, the stove is for frying things, the oven is for roasting things. Cooking ingredients in the wrong way will lead to failure of the game. Once you have got processed ingredients and the appropriate cooking tool ready. cook all of them according to the recipe. There are two conditions to correctly cook something (grill/fry/roast): a) the ingredient you want to cook is in your inventory and b) there is a suitable cooking tool in the room, and then use 'cook with to chop/slice/dice ingredients, you need to take the knife and the ingredient in your inventory and then 'slice/chop/dice ... with knifc' Make sure to first process the food (chop/slicc/dice) before you try to cook them. When you have all the ingredients (that gol processed or cooked according to the menu). you can "prepare meal' in the kitchen and then 'eat meal' to win the game. The ingredients should EXACTLY match the color in the recipe, but if the recipe doesn't specify color, any color would be fine. When you 'take ... with ...'. the EXACT name you container/supporter (e.g.. toolbox) when says something like "there isn't ' thing on it"/"has nothing on it"
command. When you need
You don't 't need to examine the
You have 80 steps to complete the task. Restarting is forhidden.
We show successful example trajectory found by IGE below, from evaluation in Section 4.3
Example trajectory found by IGE in The Cooking Game
You are hungry! Let's cook delicious meal. Check the cookbook in the kitchen for recipe. Once done, enjoy your meal!
- -Backyard= You find yourself in backyard. A normal kind of place. You can see a patio tablc. But oh no! there's nothing on this piece of garbage. This always happens! You make out patio chair. But oh no! there's nothing on this piece of trash. You can make BBQ, there's 's that. There is closed barn door leading north. There a closed sliding patio door leading west. There is an exit to east. Don't It worry, there is no door.
>
open sliding patio door
You open sliding patio door.
go west
Kitchen =- You find yourself a kitchen. An ordinary kind of place. The room is well lit. You can make out a fridge. You hear a noise behind you and spin around, but you can't 't see anything other than an oven. Were you looking for a table? Because look over there, it's table. On the table you see cookbook. You can't 't wait tell the folks home about this! Look out! It's a- oh, never mind, it's just counter. Now why would someone leave that there? You see a red apple, raw purple potato, red onion, a yellow apple and a knife on the counter. You can see a stove. You shudder, but continue examining the stove. Unfortunately, there isn't thing on it. Hm. Oh well There is open sliding patio door leading east. There is closed plain door leading north. You don't like doors? Why not try going west, that entranceway not blocked by onc. > examine cookbook
=
You open the copy of "Cooking: A Modern Approach (3rd Ed.)" and start reading:
Gather all following ingredients and follow the directions to prepare this tasty meal. Ingredients: block of cheese carrot orange bell pepper red apple tomato Directions: chop the block of cheese fry the block of cheese dice the carrot grill the carrot chop the orange hell pepper fry the orange bell pepper chop the red apple grill the red apple slice the tomato grill the tomato prepare meal
Recipe #1
> take red apple
You take the red apple from the counter.
Your score has just gone up by one point.
>
open fridge
You open the fridge, revealing a block of cheese, carrot, a white onion and a yellow hell pepper.
>
take block of cheese
You take the block of cheese from the fridge.
Your score has just gone up by one point.
> take carrot
You take the carrot from the fridge.
Your score has just gone up by one point.
examine counter
The counter is solid.
20

---

Published as a conference paper at ICLR 2025
On the counter are raw purple potato, a red onion, a yellow apple and a knife.
> go west
Livingrroom You arrive livingroom...
(skip 30 steps)
cook tomato with BBQ
You grilled the tomato.
Your score has just gone up by one point.
go west
Kitchen
find yourself in kitchen...
>
prepare meal
Adding the meal to your inventory.
Your score has just gone up by one point.
meal
You the meal. Not bad.
Your score has just gone up by one point.
The End
You scored 17 out of possible 17, in 44 turns.
B.3.3 COIN COLLECTOR
In Coin Collector, we set the number of rooms to 40 and allow distractor rooms to be added along the way. Similar to Treasure Hunter, we remove the solution description from the original game, and the optimal path from the agent's starting point to the target is set to 20 steps. We include the environment-speecffe prompt that is appended the system prompt in Section below.
Environment Description for Coin Collector.
You are an agent playing TextWorld, text-hased adventure game where you are a randomly generated maze and must find the coin. You need to explore different rooms to find the target object. Here the available commands; goal: print the goal of this game go <dir>: move the player north,
east, south, west. You can only go in the direction indicated with something like exit door. take coin: win game by 'take coin' if you see coin in room The only action you can do is "go <dir> to explore the maze and 'take coin' when you see the coin in the room.
You have 25 steps complete the task. Restarting is forbidden.
C ADDITIONAL EXPERIMENTS AND ANALYSIS
C.I ARCHIVE SIZE ANALYSIS
We provide the detailed analysis of the impact of intelligent archive filtering on the size of the archive across different environments. As shown in Table 4 intelligent filtering in IGE drastically reduce the size of the archive and help the algorithm focus on the most interesting states. Without intelligent filtering Baby AI-Text and TextWorld, the archive becomes over 5x larger.
Table 4: We show that intelligent filtering in IGE can drastically reduce the size of the archive, and help the algorithm focus on the most interesting states. We use the same evaluation setting as Table 2 and show the mean and 95% bootstrap confidence intervals.
Number of States BabyAl (PN)
Archive Filtering
Game of 24
TextWorld (CG)
No Filter With Filter
18.5 + 3.2
203.5 1 56.7 25.5 5.2
22.4 1. 15.3
15.6 = 2.3
4.4 + 2.8
These results highlight that, across all evaluated environments, intelligent archive filtering not only enhances performance but also maintains manageable archive size, which is crucial for scalability.
21

---

Published as a conference paper at ICLR 2025
C.2 PROMPT ROBUSTNESS EXPERIMENTS
We investigated the sensitivity of IGE to prompt variations to assess its robustness to prompt en- gineering. Specifically, we conducted experiments where we removed any hints or domain- sppciffc instructions on how to solve the tasks, keeping the prompts purely factually descriptive about the environment. For example, in:
Game of 24
We removed all few-shot prompting, so that the agent has no examples, Lee,, all text from "Some examples of correct reasoning traces are as follows:"
BabbAA
We removed "do not repeat actions if they have no effect"
TextWorld (Cooking Game)
We removed "The ingredients should EXACTLY match the color in the recipe, but the recipe doesn't it specify color, any color would be fine. When you 'take with ...,, use the EXACT name you see."
We removed "You don't t need to examine the container/supporter (e.g., toolbox) when says something like "there isn't 't a thing on it"/"has nothing on it"
We refer these domain-genert prompts, they contain only basic information about the environment without any guidance on how to approach the tasks.
The evaluation settings remained identical those in our main experiments (see Section en- suring a fair comparison. The results, shown in Table []] indicate that even with minimal prompts, IGE maintains significant performance advantage over the baselines across all environments. For instance, in the Game of 24, IGE achieves success rate of 96% with domain-generes prompts, compared to 100% with the original prompts, and still significantly outperforms the baselines. Sim- ilar trends are observed in the Baby AI "Put Next To" task and the TextWorld "Cooking Game". The performance drop of IGE when using nomain-geners prompts is modest compared to the baselines, demonstrating that IGE is robust to prompt variations and does not rely heavily domain-specinn prompt engineering.
These findings suggest that IGE be applied effectively without extensive prompt tuning, mak- ing flexible and generalizable to new domains. The foundation model's internalized notions of interestingness and problem-solving abilities enable IGE guide exploration even with minimal guidance, highlighting its potential for widespread applicability.
Table 5: IGE Performance with Domain- General Prompts. We use the same evaluation setting as Table and show the mean and 95% bootstrap confidence interval for the success rate.
Environment
Model
Naive L.M
ReAct
Reflexion
IGF
Original (GPT-4)
70 + 14
83 + 11
100 +
Game of 24
Domain General Prompt
13
13
Original (GPT-4) Domain-Genche Prompt
12 + 12
48 + 12
N/A
84 + 8
BabyAl "Put Next To"
8 12
24 12
N/A
68 : 12
Original (GPT-4)
40 1: 16
56
52 1:: 16
92 -1
World "Cooking Game"
Iomain-Generae Prompt
44 16
+ 16
48 + 16
72 + 16
D FURTHER PROMPT DISCUSSION
D. EXTRACTING CHOICES
By default, in Section 4.1 we prompt the FM to return a JSON object containing just the numerical index of the choice. We choose this because of the ease parsing the response and validating it lies within the correct bounds. An example prompt is displayed below.
22

---

Published as a conference paper at ICLR 2025
JSON Choice Prompt.
Reply concisely and exactly with following JSON format:
("choice": X}
where is the index of the desired choice.
When using chain-of- of-thought as in Section we use the following prompt:
JSON Choice Prompt (Chain of Thought).
First. briefly reason about your plan.
Reply concisely and exactly with the following JSON format:
("thought": X, "choice": Y}
where X is your reasoning and Y is the index of the desired choice. Make sure Y is parsable integer.
For the TextWorld environment in Section 4.3 since the action space is much larger, we ask the FM directly output a text action that automatically parse.
Decision Making Prompt.
Please briefly reason about your plan and then output the command in the format "> command'. Ensure only command is included.
We use the regex "> (?=))? (in Perl notation) to parse the command. We note that the failure rate for both of these options is very low, less than 0.1 % across our evaluation. Despite this, include failsafe that returns random choice in case of an invalid output.
D.2 REJECTION-BASEE BASED ARCHIVE FILTERING
The 'acceptance-based" archive filter in Section 3.3 iterates through every new state and asks whether each one interestingly new and should be added the archive. This approach can break down in larger environments where it becomes necessary to explicitly deprecate earlier discoveries that have become irrelevant, in order not to overload the archive (e.g., in Section 4.3). this environment, we use an alternate version of the prompt which first adds all states, and then asks the foundation model remove the uninteresting states. example prompt is shown below.
Rejection- based Archive Filtering Prompt.
Current state archive:
[State Archive] Remove outdated states that are no longer relevant task, all interesting explorations attempted, or have similar states in the archive that show more progress.
D.3 IMPLEMENTATION
STATE-CONDITIUS
ACTION HISTORY
The state-conditional action history in IGE stores the set of actions previously attempted from each archived state. For each state the archive, we maintain list of actions that have been tried from that state, which helps prevent the foundation model from repeating the same actions and encourages exploration of new actions. This history provided in the context when prompting the foundation model for the next action. Since state archive is kept at manageable size (see Table 4 this additional memory requirement is minimal.
D.4 HANDLING HALLUCINATIONS DURING TRAJECTORY CREATION.
Foundation models are known to sometimes produce hallucinations or invalid outputs. IGE, the foundation model does not create entire trajectories; instead, it is queried for state/action choices from actual options, and the resulting states are saved into an archive, as in the original Go-Explore. Therefore, IGE provides robust scaffolding for foundation model agents that explicitly prevent the hallucination impossible states. There is a concern that the LLM could pick impossible actions, but observe that malformed inputs only consist of less than 0. 1.15 of total actions. As discussed
23

---

Published as a conference paper at ICLR 2025
in Appendix D in these rare cases, simply take the simple choice of randomly sampling an action from the environment. Further work could explore additional safeguards and validation mechanisms to handle hallucinations more effecttiveyy
E HYPERPARAMETERS
In this section, we provide the hyperparameters our empirical evaluation in Section 4 We list the hyperparameters for IGE in Table We choose the values for exploratory rollout length based on the average number of steps needed to make 'reasonable progress" in the environment.
Table 6: IGE Sampling Parameters. TH, TCG, and CC are abbreviations for Treasure Hunter, The Cooking Game, and Coin Collector in TextWorld
Value(s)
Hyperparameter
Game of 24
BabyAI
TH
TCG
CC
No. state expansions, Naate
50
24
48
125
No. exploratory actions, Naction
3
10
5
5
1
We list the sampling parameters for GPT-4 OOenAA 2024
passed via the OpenAl API in Table
Table 7: GPT-4 Sampling Parameters
Value
Hyperparameter
Game of 24
BabyAI
TextWorld
Temperature
0.7
0.7
0.3
Max new tokens Response format Version
1000
1000
1000
JSON Object Turbo-2024-04-09
JSON Object 0-2024-05-13
Text
0-2024-05-13
We used GPT-4-Turbo for Game of 24 and GPT-4o for BabyAl and TextWorld. This was purely done to select the version of GPT-4 that available and the cheapest at the time of running the experiments. The version of GPT-4 is consistent per environment. We use a reduced temperature for the TextWorld domain reduce the possibility of generating malformed responses, as actions output in free-form natural language. In our ablations in Section 5 we use the "turbo-0155" variant of GPT-3.5.
E. COST EXPERIMENTS
We provide the average per task for algorithm per environment (the number of seeds is specified in Section 4):
Table 8: Per task API cost for IGE using GPT-4 listed in USD.
Environment
API Cost (USD)
Game of 24
1.04
BabyAl AI
2.01
Worl
1.28
We note that the price token of the '0-2024-05-13 -3' option is half that of clurbo-2024-04-00 so
we could expect to achieve the same level of results on the Game of 24 with half the price. The total cost of API access required to perform the final experiments in this paper was under 2,000 USD. During development, we iterated on IGE with a smaller number of seeds, which represents a small fraction of this cost added top.
24

---

Published as a conference paper at ICLR 2025
F MORE RELATED WORK AND FUTURE WORK
FM guidance at all stages of IGE to drive exploration. FMs as 2023a Bradley et al., 2023) have already seen use in pecision-making tasks: 2024) considers FM guidance in multi-task settings to select the most promis- ing next task to train on. However, focusing on the broader task could miss out on interesting behavior that happens at more granular level, and thus IGE greatly expands on the integration of
FM-as-judge. We employ
judges Zheng et al.
OMNI (Zhang et al.
FM intelligence into decision-making. RL from Al Feedback (Bai et al., 2022 Lee et al. 2024] Klissarov al. 2023) considers training RL agents using reward functions derived from FM prefer- ences. This similarly guides agents towards preferred states but without the intelligence of FMs for action selection.
Extension to Stochastic Environments. Our current experiments focus on deterministic environ- ments to clearly demonstrate the effectiveness of IGE. However, IGE can be extended to stochastic environments, similar to how the original Go-Explore (Ecoffet et al.) (021b) was extended. In the 2021 Go-Explore work, robust goal-conditioned policy is trained via imitation learning on the tra- jectories found during exploration, enabling generalization to stochasticity. In the context of IGE, we can similarly collect successful trajectories and provide them in-context to the foundation model, leveraging its ability to generalize and handle stochastic outcomes. By incorporating past trajecto- ries into the FM's context, IGE can learn to navigate stochastic transitions and maintain robust exploration strategies. This represents an exciting direction for future research.
Relationship with Hierarchical Search and MCTS, The Go-Explore framework, and by exten- sion IGE, shares similarities with hierarchical search and Best-First Search (BFS), as it prioritizes exploration from the most promising states. However, IGE builds on this approach by leveraging the foundation model's intelligence to dynamically assess the interestingness and potential of states, rather than relying on fixed heuristics. This allows IGE to adaptively explore the search space in more informed manner. In extremely hard combinatorial problems like chess and Go, integrating an effective interestingness function is crucial. If we could train or obtain such a function, com- bined with a solid value function similar those used in Monte Carlo Tree Search (MCTS), IGE could potentially discover novel and interesting strategies. Exploring this integration with MCTS and developing advanced interestingness functions through FMs are promising avenues for future work.
Comparison to GFlowNets. GFlowNets (Bengio al. 2023) aim to sample compositional struc- tures proportionally to a specified reward function, which differs from IGE's goal of exploring and discovering interesting states without predefined rewards. While both involve generating states through sequential decisions, IGE focuses on leveraging the foundation model's notions of inter- estingness rather than sampling according to a reward distribution. Moreover, GFlowNets are not directly intended for hard-exploration or sparse-reward problems. IGE should therefore be much better in hard exploration tasks where there is little to no reward signal to guide search. We believe detailed comparison with GFlowNets is an interesting direction for future research.
Comparison with Exploration Methods in Games like Minecraft. Recent works in Minecraft, such Voyager (Wang et al. 2023) and Ghost in the Minecraft (Zhu al., (023b), have employed agents that act in the environment via collections of high-level code or algorithmic policies tailored Minecraft. In contrast, IGE provides a generic method operate any state and action space, given a minimal description of the environment, by leveraging the intelligence of foundation models test time. The code policies used in these Minecraft agents are indeed interesting and could potentially be integrated into an IGE-like framework, representing promising direction for future research into more efficient exploration agents. By combining the strengths of both approaches, we could enhance the exploration capabilities of agents in complex environments.
Potential Real-life Applications. IGE's 's framework is very general, and we envision that it could be extended to many real-life problems involving exploration in complex spaces. For example, in syn- thetic biology. IGE could aid in discovering novel proteins or designing new drugs. In mathematics, it could assist in exploring mathematical conjectures or finding novel proofs. Recent work has begun to explore applications of LLM agents across science, such as in biology research (Laurent et al. 2024) and solving olympiad-leve math problems (Trinh et al. 2024). On the practical side, LLMs have been adapted for various web-broo browbrowsng and computer-basse tasks (Liu et al. 2023) as useful personal assistants, many of which require exploration across long horizons (e.g., building an app
25

---

Published as a conference paper at ICLR 2025
from scratch). By leveraging IGE, these agents could improve in planning and exploration in such complex tasks.
Accelerating Open-endedness Research. Open-endedness is often formulated an unsupervised exploration problem, aiming to develop algorithms that can continually generate novel and diverse behaviors or solutions without predefined objectives. IGE provides general mechanism to explore and discover a diverse set of interesting states or solutions in arbitrary environments, much like human scientific exploration. By leveraging FMs' internalized notions of interestingness, IGE can identify and pursue novel directions that may not be specified explicitly. This capability could accelerate research in open-endedness by providing powerful tool for unsupervised exploration in complex domains.
Robustness and Potential Biases. Foundation models are known to have biases stemming from their training data, which could affect their performance and decision-making in certain environ- ments. Although representation in pre-training is hard to measure even for open-weight models, we have strong evidence that IGE's success is relatively independent of the base LLM performance. For instance, in our new results in Table 3 with Claude Sonnet 3.5 on Game of 24, the naive action selection performance is low (24%). but IGE raises it significantly to 86%. The general question of bias is significant; for example, a foundation model was biased to think that nothing "purple" is interesting, wouldn't I explore states that are "purple". We do not observe such issues in cur- rent evaluation (indeed, we would expect them to manifest less games and RI /controll problems rather than human-centric problems), but this raises an interesting point tied the wider literature of reducing bias in foundation models in general. Future work could investigate methods to detect and mitigate potential biases in IGE to ensure fair and unbiased exploration.
Potential for Fine-tuning or RL Training. Fine-tuning or additional RL training could potentially enhance foundation models' performance in specific tasks. In the context of IGE, one could en- vision fine-tuning the foundation model on the trajectories discovered during exploration, further improving its decision-making and ability generalize. Our current approach shows that signifi- cant gains can already be achieved without additional training, which is advantageous for efficiency. Additionally, the trajectories discovered by IGE could be used to train traditional RL or imitation learning algorithms, providing valuable data for further improvements. Exploring these possibilities could lead to more powerful agents capable of solving even more complex tasks.
Investigating LLMs' Notions of Interestingness. Understanding and formalizing the foundation model's 's notions of interestingness is an intriguing direction for future research. Similar to how reward models are collected on human preferences and used to fine-tune LLMs (Ouyang et al. 2022], we could consider extracting interestingness preferences and fine-tuning an LLM to select more interesting states actions. This could then lead to more efficient versions of IGE or even improvements to FMs themselves. Developing methods to quantify and interpret the interestingness judgments made by FMs could also enhance our understanding of their decision-making processes.
Alternate Action Selection Strategies. Random action selection and FM action selection repre- sent two extremes in action selection strategies. Our framework is flexible, and could consider substituting the action selection policy in IGE with any RL policy. This would enable us to keep the foundation model IGE scaffolding that prevents detachment and derailment and allow us to use more computationally efficient options to roll out trajectories. For example, integrating exploration policies could enhance scalability. Ex- ploring such combinations could provide a balance between performance and computational cost, extending the applicability of IGE to larger and more complex environments.
strategies based on intrinsic motivation or curiosity-
26


Reviews:
[{"review_id": "tuBBIJPKFX", "reviewer": "Reviewer_iSQ8", "summary": "Combines the Go-Explore -- a framework for solving hard-exploration problems -- with language models (LMs) to create LM agents that do better on search and exploration tasks.", "questions": "1. Can the authors provide more details of how baselines are run? For instance, do all LM baselines get the same environment description for each game? \n\n2. Are there ablations of sensitivity of the approach to differences in prompts?\n\n3. How exactly is the state conditional history (L199) implemented? Does it store a set of previously attempted actions for each seen state? Is it a buffer of (state, action) pairs?\n\n4. Do you think the exploration problems Go-Explore addresses are also problems in the evaluation tasks? Specifically, detachment and derailment (L130)? \n\n5. How is state reset done (i.e. to explore from a certain state) in the provided environments?\n\n6. In table 2, is there a difference between “classic go explore” and “ablated all 3 above” models?\n\n7. Just out of curiosity, there do exist work that tackle exploration very explicitly in difficult games such as MineCraft [Wang 2023, Zhu 2023]. Is there a reason these methods are *not* reasonable comparisons against IGE?\n\n\n[Wang 2023] Wang, Guanzhi, et al. \"Voyager: An open-ended embodied agent with large language models.\" arXiv preprint arXiv:2305.16291 (2023).\n\n[Zhu 2023] Zhu, Xizhou, et al. \"Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory.\" arXiv preprint arXiv:2305.17144 (2023).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**[Originality]**\nThe idea seems original, and creatively combines Go-Explore (a traditional RL approach for exploration) and LM agent tasks. \n\n**[Quality]**\nThe method shows strong performance over its baselines.\n\n\n**[Clarity]**\nThe prompting procedures are well described and environments are well described \n\n**[Significance]**\n\nThe combination of RL exploration methods and LM agents feels both natural, thus this work opens up the possibility to further explore and improve components of the Go-Explore framework for better LM-agents.", "weaknesses": "While I quite enjoyed the idea and appreciate that using LMs with the go-explore framework is a promising direction, the main weaknesses with the current version of the paper is that many of the stated / emphasized claims are quite strong, which I do not feel they are properly back-ed up. See below.\n\n> IGE ofers the previously impossible opportunity to recognize and capitalize on “serendipitous discoveries” (L24, L76, L95)\n\n- The use of the word “serendipitous” feels un-rigorous and only added confusion. I don't know what this word means in context, nor how it contributes scientifically to this paper.\n- Based on my reading, the authors generally use this word when describing the ability to identify which states should be archived to later explore further. This is an improvement over the original Go-Explore which use a heuristic (cell representation) instead of LM-preference. Perhaps it is better to just state this as-is.\n- Further, the claim that this was “previously impossible” feels very strong. It is also unclear *what* was previous impossible? There exist previous work that use LM as a feedback to derive intrinsic motivation (e.g. Klissarov et al 2023, which the authors cite, L474). \n\n> IGE integrates well with various agent strategies ... and will only get better as capabilities of foundation models improve further (L103)\n\n- The experimental evidence for this claim is in Table 3 right, which shows IGE does better on GPT-4 than GPT-3.5. One hypothesis is that IGE gets better as foundation models improve; another hypothesis could be that IGE is only well-tuned to GPT-4 and may not work in other / later models. Experiments across multiple foundational models with different capabilities will make this more convincing. \n\n> Key strength of IGE is that it is a strict improvement on top of any FM agent framework (L195)\n\n- Unclear if the authors mean IGE is *complementary* to other FM agent frameworks; or that IGE will always be the best FM agent framework?\n\n\nThe strong claims contrast against the empirical results, which are good, but marginally so (e.g. between Fig 3 and 4, IGE is statistically significantly better than other baselines in 3 out of 13 settings between Figures 3 and 4 (evaluated base on overlapping 95 confidence interval bars). I feel that a more measured presentation can aid in paper clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nd7fhYdBz0", "reviewer": "Reviewer_cN7x", "summary": "The paper revisits the Go-Explore (GE) algorithm (Ecoffet et al., 2021) by letting a foundation model (FM) control three key operations performed on an archive of interestingly new states: (a) select a state to explore from (b) choose actions from the selected state, and (c) decide which state is worth keeping. The paper argues that FM successfully replaces GE's requirement to use hand-designed heuristics domain-specific pre-specified knowledge. IGE, with GPT4 as the underlying LLM, is evaluated on three families of environments (Game of 24, BabyAI, and TextWorld). It is claimed that the method outperforms baselines consistently across the board, sometimes being the only method that shows life (e.g., for Coin Collector). Finally, the following ablation studies are provided: (a) the importance of FM, (b) the size of the archive, and (c) the capabilities of FM.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper considers a fundamental problem of managing the exploration-exploitation trade-off.\n* IGE proposes to leverage the capabilities of LLMs to address deficiencies in the original GE and shows that it succeeds in three families of environments.\n* The paper fits into the important line of work concerned with applications of foundation models in RL.", "weaknesses": "* The choice of a setup for IGE matters, as it seems to work for deterministic environments. Is the extension to stochastic environments a fundamental limitation of IGE, or is there a \"natural\" extension? One example could be NetHack, which is a known hard exploration problem with long-horizon and credit assignment challenges.\n* The IGE's procedure of selecting interesting states from the archive resembles a hierarchical search, where the planning is performed on high-level milestones or subgoals. Often such methods use Best-First-Search as a backbone planner, hence they jump freely between subtrees, similarly as IGE (see a discussion in lines 298-302 or 490-496). Can the Authors comment on that?\n* Related to the above, how does IGE compare against algorithms like MCTS for hard deterministic combinatorial problems (like chess or go)?\n* What is the relation of IGE to GFlowNets, which searches through a design space to generate samples proportional to a specified reward function?\n* What important real-life problems can be expressed as exploration and solved via IGE? Say, interesting chess or go states, novel mathematical theories or proof ideas, useful molecules, etc.\n* Can the method accelerate research on open-endedness, which is often formulated as an unsupervised exploration problem?\n* How robust is the method?\n\t* LLMs are known to be unreliable and have a long list of failure modes.\n\t* Are there any biases with respect to some environments games, i.e., the method performing better in the environments that were prominently represented in the pre-training dataset?\n\t* Conversely, does IGE underperform for environments that were underrepresented in the training dataset or that require a set of rules or wiki?\n* How can IGE be augmented with the inclusion of environment rules, RAGs, and tools? How hard is it to set up IGE for a new task? How much work is expected on the prompt engineering part?\n* What is the performance of open-sourced models? \n* IGE is an in-context method. What could be achieved that currently is out of reach if the method allowed fine-tuning (or RL training)?\n* What could be the implications for the current RL algorithms regarding the potential benefits stemming from IGE providing better data?\n* Can we formulate an inverse-IGE problem by designing synthetic datasets to discover the \"definition of interestingness of LLMs\"?\n\n\nOther:\n* In Algorithm 1, lines 3, 8, 10, should account for the fact that \"$\\gets$\" typically stands for assignment operator, not \"add to a set\" operator.\n\nEdit 27th November 2024: Based on the Authors' response, I have increased the score to 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70CP83kFqZ", "reviewer": "Reviewer_xqm5", "summary": "Summary: The authors propose a method that reconceptualises the original Go-Explore method to use the pre-trained knowledge contained in large foundation models. \nFor this, the original components of Go-Explore such as keeping a state archive and exploring from a previously attained state are defined for a language foundation model. \nThe idea behind this is that foundation models are better able to reason about what defines an interesting (novel) state or behaviour, while exploiting the creative abilities of foundation models. The authors test on 3 different environments. 2 text based environments and one environment where the visual \nFeatures are described in terms of text.  The authors show that intelligent go-explore is able to solve these environments in very few steps while also outperforming other foundation model based methods. \n\n\nOverall, this is a well written and interesting paper. I think the idea of creating a foundation model based exploration algorithm that follows along the lines of Go-Explore is clever and perhaps shows a more generalised way to exploit foundation models for exploration. \nWhile the authors ablate their method well, I feel more interesting ablations would concern the capacity of the foundation model itself in terms of how complex the observation space can get and how different prompts, i.e., conditioning affects the exploration. For instance,\nDid the foundation model hallucinate during some runs? What do these hallucinations look like? I think this should even be achievable with the presented BabyAI environment. See more detailed info below.", "questions": "- How would you transfer this method to continuous state-action spaces? Did you observe a limit regarding how large the state archive can become? For instance int the BabyAI environments the observations are discrete, that is each observation can be mapped to a distinct vector. I imagine this becomes more problematic the larger the environment becomes. \n- Did the foundation model hallucinate while creating the trajectories? This would be interesting to see, since failure modes for foundation models seem to differ from traditional RL models.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Exploiting the pre-trained biases of LLM for RL is an interesting idea and so far largely unstudied for exploration in reinforcement learning\n- I like that the foundation model is defined in terms of a well known exploration algorithm such as Go-Explore, since it shows how concepts from exploration can be transferred to large foundation models.\n- The paper is very clear and well written, while providing thorough ablations methods components such as in Section 4.3.", "weaknesses": "- This paper argues against hand-crafted features i.e. heuristics, however having to transfer the environments to a prompt based structure is also a hand crafted feature space.\n- The paper states in the abstract that the appeal of using foundation models is making use of the foundation models capabilities of reasoning ahead or capitalising on serendipitous discoveries.  While the results are good, can you show examples of trajectories IGE creates compared to a normal RL algorithm that you could argue are serendipitous?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D1mz3kREbp", "reviewer": "Reviewer_ZTB4", "summary": "The authors propose Intelligent Go-Explore (IGE), an extension of the original Go-Explore algorithm that replaces manually crafted heuristics with foundation models (FMs) to guide exploration. In the original Go-Explore, heuristics were required to identify 'interesting' states, whereas in IGE, this is determined by querying the foundation model, which leverages its pretrained knowledge to make decisions. Interesting states are stored in an archive, and the foundation model is queried with which state should be explored next. Additionally, instead of relying on random action sampling, IGE uses the foundation model as a decision-making agent to choose actions intelligently. The authors evaluate IGE in controlled environments (BabyAI and Game of 24), showing that it outperforms previous foundation model-based agents in these tasks.", "questions": "- Could you provide more details on the sensitivity of the prompts used with IGE? For example, in the BabyAI environment, you explicitly prompt the model with instructions such as 'do not repeat actions if they have no effect.' How much does IGE's performance depend on these kinds of statements?\n- Given that the action selection of the foundation model has the most significant impact on IGE's performance, have you explored other exploration approaches? It seems that random actions and foundation model-driven actions represent two extremes. Have you considered comparing IGE to a relatively simple information-seeking policy or other exploration strategies that could computationally be much cheaper to use?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The integration of foundation models with the Go-Explore framework is a noteworthy contribution that can alleviate several of the inflexible inconveniences present in the original framework, such as hand-crafted heuristics and random exploration.\n- The authors perform a solid empirical evaluation and compare it to several foundation model agent baselines, demonstrating significant performance gains.\n- For each of the proposed additions to the original algorithm, a solid ablation study is performed where each addition is omitted and evaluated, and IGE is also compared to the original Go-Explored in this experiment. \n- The authors do not limit their approach to text-based representations and demonstrate that IGE can also be applied with visual observations.", "weaknesses": "- One of the main contributions of the paper is alleviating the original Go-Explore from manual handcrafting of domain-specific heuristics. However, by replacing this part of the process with foundation models, quite extensive and domain-specific prompt-engineering is required instead (as shown in Appendix B). \n- Given that the environments used in the paper are relatively controlled and intuitive, and the foundation models were provided with detailed explanations of the environment dynamics and objectives, it is not clear how well the decision-making of the foundation models would work in more complex (and actual hard-exploration) environments. For instance, the Game of 24 is a relatively simple mathematical puzzle, where the exploration space is highly constrained and the rules are clear and deterministic. As such, it doesn’t pose a true hard-exploration challenge, which is what Go-Explore and IGE are meant to address.\n- Building on the previous weakness, the original Go-Explore algorithm was demonstrated across the Atari suite, including hard-exploration games like Montezuma's Revenge. In this IGE paper, the authors compared Go-Explore in specific environments where IGE performs well, but it would have perhaps made more sense to evaluate it in the Atari suite, where we already know Go-Explore performs well. It is not clear that a foundation model would be able to deal with the complexity of Atari games both in judging interesting states and in decision-making. Given that the main claim of the paper is improving upon the original framework, a comparison on that suite would have provided a more direct measure.\n- Although the authors touch upon this briefly, IGE appears computationally expensive, as the foundation model is presented with the entire archive of states to select a state, filters states, and is also queried for every individual action. While this may be feasible in the small environments used in the paper, it’s unclear how scalable the approach would be to larger environments or longer-horizon tasks. \n- Following this, according to the ablation study, the primary driver of IGE’s performance is the action selection guided by the foundation model, compared to random action selection. However, foundation models tend to only perform well in relatively intuitive and deterministic domains where decisions are relatively straightforward. In more complex or large-scale environments, a reinforcement learning exploration policy will likely be necessary. This raises an important question: if the primary performance gains from using IGE come from action selection, and these foundation models struggle in more complex environments, are the gains from using foundation models for state selection and filtering substantial enough to justify its use? The ablation study shows that these gains are minor (yet introduce significant computational overhead), calling into question the overall value of IGE when considering scalability and generality.\n- The paper occasionally comes across as over-optimistic with respect to the proposed algorithm and e.g. speculates that its solutions can easily be used for downstream RL tasks or infinite in-context learning with no evidence supporting this. A more grounded discussion of IGE's limitations would benefit the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)