Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Models

## Abstract

Abstract Go-Explore is a powerful family of algorithms designed to solve hard-exploration problems built on the principle of archiving discovered states, and iteratively returning to and exploring from the most promising states.
This approach has led to superhuman performance across a wide variety of challenging problems including Atari games and robotic control, but requires manually designing heuristics to guide exploration (i.e. determine which states to save and explore from, and what actions to consider next), which is time-consuming and infeasible in general.
To resolve this, we propose Intelligent Go-Explore ( IGE ) which greatly extends the scope of the original Go-Explore by replacing these handcrafted heuristics with the intelligence and internalized human notions of interestingness captured by giant pretrained foundation models (FMs).
This provides IGE with a human-like ability to instinctively identify how interesting or promising any new state is (e.g. discovering new objects, locations, or behaviors), even in complex environments where heuristics are hard to define.
Moreover, IGE offers the exciting and previously impossible opportunity to recognize and capitalize on serendipitous discoveries that cannot be predicted ahead of time .
We evaluate our algorithm on a diverse range of language-based tasks that require search and exploration.
In Game of 24, a problem testing multistep mathematical reasoning, IGE reaches 100% success rate 70.8% faster than the best classic graph search baseline.
Next, in BabyAI-Text, a challenging partially observable gridworld where an agent has to follow language instructions, IGE exceeds the previous state-of-the-art with orders of magnitude fewer online samples.
Finally, in TextWorld, a rich text game, we show the unique ability of IGE to succeed in settings requiring long-horizon exploration where prior state-of-the-art FM agents like Reflexion completely fail.
Overall, Intelligent Go-Explore combines the tremendous strengths of FMs and the powerful Go-Explore algorithm, opening up a new frontier of research into creating more generally capable agents with impressive exploration capabilities.
All our code is open-sourced at: https://github.com/conglu1997/intelligent-go-explore .

## 1 Introduction

Foundation models (FMs, (Bommasani et al., ; OpenAI, ; Brown et al., ; Team, ; Touvron et al., )) trained on giant internet-scale datasets have demonstrated strong general capabilities in reasoning (Wei et al., ) and understanding (Chang et al., ).
As such, these models have been increasingly employed as autonomous agents (Liu et al., ; Yao et al., ; Wang et al., ; Yao et al., ; Shinn et al., ; Besta et al., ) in decision-making tasks, showcasing the ability to adapt in-context (Dong et al., ; Olsson et al., ) to unseen tasks.
However, a significant challenge remains: foundation model agents often struggle in environments that require deep exploration over extended time horizons (Liu et al., ).
Overcoming this limitation would enable us to realize their potential as autonomous assistants in more open-ended domains like scientific discovery and innovation (Jiang et al., ).
This paper introduces Intelligent Go-Explore (IGE), a novel approach that combines the intelligence foundation models with the powerful Go-Explore Ecoffet et al. (, ) framework to substantially increase the exploration capabilities of FM and reinforcement learning (RL, (Sutton and Barto, )) agents.

Figure: Figure 1: Intelligent Go-Explore (IGE) integrates the intelligence and internalized human notions of interestingness from giant pretrained FMs into all stages of the Go-Explore (Ecoffet et al., , ) algorithm, enabling FM agents to robustly explore in complex environments. Bottom: Classic Go-Explore solved hard exploration problems by archiving novel discovered states, resetting to promising ones via domain-specific heuristics, and then performing random exploration. Top: Our approach, Intelligent Go-Explore, enables Go-Explore to tackle virtually any type of problem that is representable in the context of a large language or multimodal model. Instead of manually defining heuristics, we query the foundation model at all stages, enabling our approach to automatically catch and return to serendipitous discoveries, and harness the power of FM agents to explore. The environment shown is the BabyAI-Text game used in Section˜4.2.
Refer to caption: x1.png

Go-Explore is a popular family of algorithms in deep RL based on maintaining an archive of “interestingly new” discovered states and then iteratively returning to and exploring from the most promising states (see Figure˜1 for an overview of the three stages).
This framework has led to superhuman performance in a range of hard-exploration problems, including long-horizon Atari games and robotic control.
However, the algorithm’s success *largely relies on carefully hand-designed heuristics* at all three stages to guide exploration (Lu et al., ; Madotto et al., ).
For example, in Montezuma’s Revenge (Bellemare et al., ), an Atari game that was the previous grand challenge of exploration in deep RL, (1) saved states in the archive were returned to with probability proportional to factors like the number of times a state has been sampled before, (2) exploration was purely via random action sampling, and (3) the criteria for which states were considered interestingly new enough to be added to the archive depended on domain-specific factors like whether the agent visited a new location, or did so with more keys.
Without this pre-specified knowledge, the quality of discovered trajectories is typically significantly worse (Ecoffet et al., ).

These rigid, domain-specific choices are in stark contrast to human-like exploration of a new game, where players can often intuitively judge the value or interestingness of any particular state (Cooper, ).
More importantly, *it is often impossible to know what is interesting or possible ahead of time* in complex domains.
In the words of Isaac Asimov—*The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” but “That’s funny.”.*
With this motivation, IGE stands on the shoulders of giant foundation models, and uses their intelligence to (1) act as a judge to identify the most promising states to return to and explore from, (2) to select the best actions to take to explore from a selected state, and (3) to identify serendipitous discoveries when they happen (e.g. finding new objects, locations, or other novelties) and decide whether a new state is interestingly new enough to be added to the archive as a stepping stone for future exploration (Figure˜1, top).

We demonstrate IGE’s ability to reliably improve the exploration capabilities of FM agents on a diverse range of language-based tasks that require search and exploration.
These settings include tasks that require *commonsense reasoning, long-term planning and memory, and handling partial observability*.
IGE integrates well with various agent strategies, including few-shot and chain-of-thought-based prompting; and *will only get better* as the capabilities of foundation models improve further.
While IGE performs strongly all-around, some highlights from our evaluation include: IGE reaches 100% success rate on Game of 24 (Yao et al., ), a standard mathematical reasoning and search problem, 70.8% faster than classic graph search.
Moreover, on the TextWorld (Côté et al., ) Coin Collector domain, IGE is the only algorithm that succeeds in discovering long-horizon optimal solution paths, where prior state-of-the-art FM agent frameworks like Reflexion (Shinn et al., ) fail.

Intelligent Go-Explore simultaneously empowers foundation model agents to *reliably explore*, and reimagines the scope of Go-Explore to tackle virtually any type of problem, without being limited to hand-designed heuristics.
These abilities will substantially improve our ability to develop more generally capable agents, and increase the range of tasks they can learn how to solve.

## 2 Background

### 2.1 Go-Explore for Hard-Exploration Problems

Go-Explore (Ecoffet et al., , ) is a family of algorithms designed to solve hard-exploration (Ladosz et al., ) problems based on the principle of remembering and returning reliably to promising states.
The classic setting builds an “archive” of novel states it discovers in an environment, where similar states are grouped in a single “cell”.
These cells are defined by heuristics like having the same visual observation when downsampled to low resolution.
In the beginning, the archive only contains the initial state.
We describe the overall structure of the algorithm in the same order as Figure˜1 (bottom):
At each iteration, (1) promising states are selected from the archive through domain-specific heuristics, e.g. probabilistically sampling states proportional to their progress through the environment or potential to lead to new states.
The agent returns to that state, by resetting using the simulator or via a goal-conditioned policy, and (2) a sequence of random actions is taken to explore from that state.
(3) All discovered states deemed interestingly new by the cell representation heuristics are added to the archive, and the process repeats.
The strength of Go-Explore is due to addressing two critical impediments to exploration: forgetting how to reach previously visited states (detachment) and failing to first return to a state before exploring from it (derailment) (Ecoffet et al., ).

This approach leads to a collection of high-return trajectories being discovered, which may then be fed into an imitation learning (Hussein et al., ) algorithm to produce a policy that generalizes and is robust to stochasticity.
We adopt similar assumptions as the original setting, by assuming an agent can return to a previously discovered state by restoring in the simulator.
This assumption may readily be relaxed by training a policy to return to a given state, or in the foundation model case, by simply prompting the model with a past trajectory.

### 2.2 Large Language and Multimodal Foundation Models

The combination of model scaling and training over internet-scale data has resulted in a wide variety of foundation models  (Bommasani et al., ) that exhibit generalist capabilities.
In this paper, we consider autoregressive large language models (LLMs, (Brown et al., ; OpenAI, ; Touvron et al., )) which learn to generate text completions by modeling the conditional probability of a new token given the preceding tokens, $p(x_{t}|x_{<t};\theta)$.
This framework enables LLMs to not only generate coherent text but crucially also exhibit human-like abilities, including on commonsense knowledge questions (Talmor et al., ) and complex reasoning tasks (Wei et al., ).
These models may also be extended to other input modalities such as images by tokenizing these inputs into the same space as the text (Zhu et al., ).
When prompting an FM with an instruction, the user may decide to do so with no related examples (zero-shot), with a few successful examples in related problems (few-shot, (Brown et al., )), or ask for a chain of reasoning (chain-of-thought, (Wei et al., )) before responding.

## 3 Driving Exploration with Giant Foundation Models

In this section, we propose Intelligent Go-Explore (IGE) which reimagines the classic Go-Explore algorithm as described in Section˜2 with the intelligence of giant pretrained foundation models.
Specifically, we introduce FM intelligence to selecting which archived state to return to and explore from, which action to take from each state, and deciding whether a state is interestingly new and should be archived.
IGE’s use of foundation models is closely related to FM-as-a-judge (Zheng et al., ), which shows that foundation models are a good proxy for human judgment to evaluate the output of generative models.
Here, instead of judging synthetic output, the foundation model makes choices to determine the best way to explore an environment.
We illustrate our resultant algorithm at the top of Figure˜1 and provide full pseudocode in Algorithm˜1.

Wherever we query the foundation model, we introduce the overall strategy of Go-Explore alongside a brief description of the current environment in the “system message” (high-level directive) displayed below.
The brief descriptions for each environment we evaluate on in Section˜4 are listed in Appendix˜B.
In the following sections, we detail our prompting techniques at each stage of IGE.
The previous prompt history is visible to the agent, which enables each component of IGE to communicate with each other.
We provide precise details on how we parse responses in Section˜C.1.

### 3.1 Select State From Archive

The power to easily store and return to promising discovered states is crucial to Go-Explore’s ability to reliably solve long-horizon exploration problems.
IGE leverages the foundation model’s internalized notions of interestingness (Zhang et al., ) to select the most promising state to return to from the archive (Figure˜1, left).
This is far more flexible than classic Go-Explore, which relied on hardcoded hand-crafted heuristics to determine cell sampling probabilities.
An example prompt is shown below.

Examples of the discovered states are given in Table˜1.
We assign indices to these states in a list and ask the FM to select a numerical index.
We define a budget of $N_{\text{state}}$ “state-expansions”.
Each state expansion is followed by a sequence of exploratory actions, which we describe in the next section.

### 3.2 Explore From State

In order to effectively explore from a state selected in the previous section, we leverage the power of foundation model agents (Liu et al., ; Huang et al., ) to choose how to act in an environment.
This vastly improves on the original Go-Explore’s use of random action sampling.
One of the key strengths of IGE is that it is a strict improvement on top of any FM agent framework, *including zero-shot, few-shot, or even chain-of-thought-based prompting* (Yao et al., ).
We demonstrate this flexibility in Section˜4.

One point of departure from the classic Go-Explore is that we additionally maintain a state-conditional action history for each archived state, so that IGE can avoid repeating previously tested options.
While this information may already be available in the entire history, this helps *avoid any recency bias that can occur with longer contexts* (Zhao et al., ).
The action history can be easily reiterated in the prompt, or the prompt could display the remaining untested actions.
We define a budget of exploratory actions per state expansion $N_{\text{action}}$, which is typically far shorter than the full horizon of the environment and represents a small number of trial actions.
An example prompt is shown here.

### 3.3 Update Archive

IGE queries the foundation model to judge whether any newly discovered state is interestingly new and sufficiently different from prior states to qualify to be added to the archive.
Intuitively, we should only save the most relevant stepping stones, and discard those that are unlikely to lead to new discoveries.

Whilst the original Go-Explore required extensive domain knowledge to determine interestingness, IGE avoids this requirement and manual labor, critically gaining the ability to recognize serendipitous discoveries that could not have been predicted ahead of time.
In practice, we propose two options to filter discovered states after a sequence of exploratory actions.
The first is to iterate through every new state and ask whether each one is interestingly new and should be added to the archive.
The second is to first add all states, and then ask the foundation model to remove the uninteresting states.
We discuss this choice later in Section˜4.3; the second form is preferable in larger environments where there is more need to explicitly deprecate earlier discoveries that have become irrelevant to not overload the archive.
An example prompt for the first option is shown below.

By default, IGE implements the foundation model at all three stages of Go-Explore, but we rigorously analyze the relative importance of each component in Section˜5.
In this paper, we focus on the discovery of solutions to hard-exploration problems.
However, these solutions could easily then subsequently be used for downstream reinforcement learning or even improve the foundation model in the next task by in-context learning—thus allowing an agent to bootstrap its own learning indefinitely.

## 4 Empirical Evaluation

In this section, we evaluate Intelligent Go-Explore across a diverse set of text environments that require search and exploration.
We demonstrate IGE’s ability to handle partially observable and complex observation spaces, discover solutions involving long chains of actions, and effectively improve the ability of FM agents to explore.
For all our experiments, we use GPT-4 (OpenAI, ), one of the current SOTA LLMs, as our foundation model.
We compare IGE to random action sampling, a naïve LLM baseline, and two SOTA FM agents, ReAct (Yao et al., ) and Reflexion (Shinn et al., ).
All methods use the same amount of environment steps and receive the same observations for a fair comparison.
Naïve LLM simply queries the LLM for an action conditional on the interaction history.
ReAct prompts the agent to output its reasoning before making a decision.
Based on ReAct, Reflexion further conditions the agent on the previous attempted episode, asking the agent to learn from its mistakes.
We provide an overview of our environments in Table˜1.
Full hyperparameters are detailed in Appendix˜D.

**Table 1: We show that Intelligent Go-Explore can efficiently explore over a diverse set of environments with increasing difficulty. The problems we consider all use text-based observations, but in general, IGE may be extended to virtually any environment that can be embedded in the context of a multimodal foundation model. For each environment, we provide an example observation, samples from the action space, and the horizon of the task in the environment.**
|  | Game of 24 | BabyAI-Text | TextWorld |
| --- | --- | --- | --- |
| Problem Type | mathematical reasoning and search | partially observable gridworld with language instructions | partially observable game requiring long-term memory and planning, exploration, and common sense |
| Text Observation | "Current state: (2 8 8 14)" | "Goal: unlock the red door. You see a wall 4 steps forward, You see a yellow box 2 steps left." | "You arrive in a pantry… You see a shelf. The shelf is wooden. On the shelf you can see flour…" |
| Next Actions | - 2 + 8 = 10 Next: (8 10 14)<br>- 8 / 2 = 4 Next: (4 8 14)<br>- 14 + 2 = 16 Next: (8 8 16)<br>… | - turn left<br>- turn right<br>- go forward<br>… | - go east<br>- cook potato with oven<br>- unlock door with key<br>… |
| Task Horizon | 3 | 64 or 128 | 25, 40 or 80 |

### 4.1 Game Of 24

We first demonstrate the effectiveness of IGE in a mathematical reasoning task, Game of 24 (Yao et al., ).
The goal is to perform basic arithmetic operations $(+,-,\times,/)$ starting from 4 numbers to obtain 24.
For example, given input $(4,9,10,13)$, a possible solution could be $(10-4)\times(13-9)=24$.
We formulate the problem as an MDP (Sutton and Barto, ), where actions represent a reduction of two numbers by an arithmetic operation—i.e., the above solution would be represented as the sequence of state transitions $(4,9,10,13)\xrightarrow{10-4=6}(6,9,13)\xrightarrow{13-9=4}(6,4)\xrightarrow{6\times 4=24}(24)$.
Therefore, IGE uses the FM to iteratively expand possible solution paths and archive promising ones to return to.
The action space is the range of possible next operations, displayed in the same manner as in Yao et al. ().

Figure: Figure 2: IGE explores the Game of 24 with the intelligence of FMs and reaches 100% success rate on average 70.8% quicker than DFS, the next best baseline. IGE completes all problems within 150 environment operations. Our use of archiving and intelligent action selection allows us to greatly outperform prior LLM agents with an equal number of operations performed. The success rate is computed over 100 test problems.
Refer to caption: x2.png

We evaluate IGE across 100 hard test problems in Figure˜2, and additionally include the standard (unweighted) graph search algorithms depth-first search (DFS) and breadth-first search (BFS) as reference.
Since the combinatorial complexity of the problem is at most $\binom{4}{2}\cdot\binom{3}{2}\cdot 4^{3}=1152$, graph search is guaranteed to find a solution within that many actions.
The system prompts for both IGE and the LLM baselines contain *few-shot examples* with correct calculations on different starting numbers.
IGE rapidly reaches 100% success rate, on average 70.8% quicker than the next best baseline, depth-first search (DFS)—
this improvement is statistically significant ($\chi^{2}$ test, $p<0.05$) at 150 operations, where IGE has solved all problems.
This success may be attributed to the fact that language models have internalized mathematical intuition and are likely to be able to identify promising pairs like $(6,4)$ that could easily be multiplied together for a solution.

All LLM agent baselines (naïve LLM, ReAct, Reflexion) eventually plateau and even get beaten by the unintelligent DFS.
This highlights the need for diverse action selection, which IGE enables.
A final point of comparison we make is to Tree of Thoughts (ToT, (Yao et al., )) which achieved 74% on Game of 24 within their evaluation budget.
We emphasize that our evaluation setting is very different as IGE selects from the list of valid options rather than doing the math in context.
However, we note the key difference to our method is that ToT evaluates and expands multiple reasoning paths following a tree structure, whereas IGE can easily jump around the search space—this is a crucial advantage in more complex environments (like those in the following sections), where it takes many coordinated actions to get from one state to another interesting state.

### 4.2 BabyAI-Text

Figure: Figure 3: IGE can find solutions to challenging tasks in the BabyAI-Text environment more effectively and with orders of magnitude fewer online steps than prior RL-trained baselines (GLAM, (Carta et al., )). Task types are in order of difficulty. As tasks become more difficult, the performance gap of IGE v.s. the LLM baselines grows. We show the mean and 95% bootstrap confidence interval (Zoubir and Iskandler, ) over 25 seeds per environment type. *Here, and elsewhere, confidence intervals are obtained by bootstrapped resampling 10,000 times.*
Refer to caption: x3.png

Next, we show that IGE scales to the BabyAI-Text domain from Carta et al. (), which is a procedurally-generated partially-observable 2D gridworld with text-based observations.
The agent is given a textual goal instruction which could correspond to one or more instructions in a sequence, e.g. “pick up X and then go to Y”.
As we can see from the observations in Table˜1, the task is challenging even for humans to complete and requires forming a model of the world from partial text descriptions.
This kind of state observation would make it *hard to define heuristics to determine how good any particular state is*, as in classic Go-Explore.
The optimal path to a solution may include moving blocking objects as well as finding keys to open doors.
We consider 5 different task families of increasing difficulty: “go to”, “pick up”, ‘pick up then go to”, “open door”, and “put next to”, which are described fully in Section˜B.2.

We omit the Reflexion baseline in this environment due to the high cost of querying GPT-4 with 128-step episodes in the context.
Due to the complexity of this environment, we use *chain-of-thought prompting in all three components* of IGE.
This allows the FM to deliberate on the state of the game before making decisions.
We show that IGE can find solutions to these problems with only a tiny budget of 250 environment steps per task (divided into rollouts of 10 exploratory actions each) and visualize the final performance in Figure˜3.
IGE and ReAct vastly outperform the prior RL-trained language model approach, GLAM (Carta et al., ), with orders of magnitude fewer samples (GLAM used 1.5M online steps) and requiring no training whatsoever.
IGE achieves the best or close to the best performance in every task.
The gap between IGE and the second-best method grows with task difficulty, with a statistically significant 36% improvement ($\chi^{2}$ test, $p<0.05$) on ‘put next to’.

### 4.3 TextWorld

Finally, we show IGE’s ability to tackle tasks requiring long-horizon memory and planning, exploration, and commonsense in TextWorld (Côté et al., ), a classic text-based agent benchmark.
We consider three challenging games in Textworld: Treasure Hunter, The Cooking Game, and Coin Collector.
In each game, the agent needs to complete the task while navigating a maze of different rooms, while only seeing the current room’s description in text.
The agent interacts with the world using free-form natural language commands, such as “go east” or “cook potato with oven.”
In Treasure Hunter (Parisotto and Salakhutdinov, ), the agent has to find a specific item by exploring, finding keys, and unlocking doors and containers.
In The Cooking Game, the agent must find a recipe, locate and process (e.g., dice, cut, chop) ingredients, and cook them according to the recipe using various kitchen appliances (e.g., oven, pan).
In Coin Collector, the agent must find a coin randomly located in the maze, testing its navigation and exploration skills.
We set each game to hard difficulty, details on game customizations are provided in Section˜B.3.
As in the previous section, we use chain-of-thought prompting in all three components of IGE.
Because the state archive in this environment grows significantly, we implement rejection-based archive filtering, which we describe in Section˜C.2.

Figure: Figure 4: IGE outperforms state-of-the-art FM agents in three challenging text games in TextWorld. These results illustrate the powerful capabilities of planning, commonsense reasoning, and exploration of IGE (Section˜4.3). Notably, in the Coin Collector game where hard exploration is required, we observe BFS-like search behavior emerge in IGE, enabling it to find the most efficient solution where all other approaches exhaust the environment horizon. We show the mean and 95% bootstrap confidence interval over 25 seeds for each game.
Refer to caption: x5.png

We present success rates achieved on the three games using IGE and the baselines in Figure˜4.
We observe that IGE outperforms all other baselines, with a statistically significant ($\chi^{2}$ test, $p<0.05$) performance gap between IGE and the second-best method in the harder Cooking Game and Coin Collector.
In The Cooking Game, IGE outperforms the second-best agent, ReAct, by a large margin of 36%, demonstrating IGE’s advantage in hard-exploration problems.
In Coin Collector, IGE *is the only method that can find the solution in the maze*, with all other methods completely failing.
Interestingly, we observe that IGE exhibits BFS-like behavior, and intelligently selects rooms with unexplored directions and iteratively removes rooms with exhausted directions.
This results in IGE almost always finding the shortest path to the target, while other methods fail to navigate the maze.

We highlight that Reflexion does not improve over ReAct in all the games we tested.
Although Reflexion should in theory be an improvement over ReAct with the experience from previous attempts, it tends to decrease performance.
We hypothesize that in long-horizon environments, the history becomes too long after the initial episode, and prevents Reflexion from effectively utilizing knowledge from the previous episode.
In contrast, IGE uses the FM to iteratively filter interesting states in the archive, which ends up *controlling the context length*.
This helps IGE truly make use of the cumulative knowledge gained through exploration.

## 5 Analysis

In this section, we analyze (1) the importance of FM intelligence for each of the three key components of Go-Explore, (2) how IGE’s selectivity produces a smaller (and thus more efficient) archive, and (3) how IGE’s performance improves as the FM’s size/intelligence increases.
We take a representative sample of environments from the previous section of Game of 24, Put Next To (PN) from BabyAI-Text, and The Cooking Game (CG) from TextWorld.
Hyperparameters are listed in Appendix˜D.

**Table 2: We rigorously ablate the design choices in IGE and analyze the importance of intelligent selection at each stage. Game of 24 performance is taken at 150 environment steps, over 100 evaluation seeds. BabyAI-Text performance is taken at 250 environment steps, over 25 evaluation seeds. TextWorld performance is taken at 240 environment steps, over 25 evaluation seeds. ‘Standard’ is mirrored from Section˜4. We show the mean and 95% bootstrap confidence interval for the success rate.**
| Variant of IGE | Success Rate (%) |  |  |
| --- | --- | --- | --- |
| Game of 24 | BabyAI (PN) | TextWorld (CG) |  |
| Standard | 100 $\pm$ 0.0 | 84 $\pm$ 14 | 92 $\pm$ 10 |
| ✗ Intelligent action selection | 068 $\pm$ 9.0 | 24 $\pm$ 16 | 00 $\pm$ 00 |
| ✗ Intelligent state selection | 096 $\pm$ 3.5 | 48 $\pm$ 20 | 76 $\pm$ 16 |
| ✗ Intelligent archive filtering | 093 $\pm$ 5.0 | 64 $\pm$ 20 | 64 $\pm$ 20 |
| ✗ All 3 above | 061 $\pm$ 9.5 | 4 $\pm$ 6 | 00 $\pm$ 00 |
| ✗ State-conditional action history | 033 $\pm$ 9.0 | 72 $\pm$ 16 | 72 $\pm$ 16 |

How Important is Foundation Model Intelligence at Each Step?
First, we analyze the impact of FM intelligence on each component of Intelligent Go-Explore.
We ablate replacing state and action selection with uniform random sampling, archive filtering with saving everything to the archive, and not maintaining a state-conditional action history.
We use these unintelligent choices, as *it would be very time-consuming to attempt to design the right heuristics* based on the rich text observations in Table˜1.
In Table˜2, we observe that where the intelligence of FMs is more valuable varies by environment.
Since the environment horizon is only 3 in the Game of 24, the most important factor is ensuring that the actions tried are diverse and intelligently selected.
This hypothesis is confirmed: the largest performance drops occur when removing either FM action selection or the action history.
Different IGE components are most helpful in both of the longer-horizon BabyAI-Text and TextWorld environments: intelligent state selection and archive filtering make a big impact, showcasing the strength of enabling IGE to return to promising discovered states.
There are smaller performance drops when removing the action history; likely because in larger environments, many more unique states are discovered, so there is less gain from preventing taking the same actions from frequently returned to states.
In both environments, we also observe a drastic decrease when switching to random actions, as in classic Go-Explore.
This underscores the substantial benefits IGE provides in harnessing FMs for action selection.

Finally, we elucidate the need for intelligent archive filtering across all our environments.
Not only does archive filtering improve performance, but it also drastically cuts down the number of uninteresting states in the archive, as shown in Table˜3 (left).
As we use rejection-based archive filtering on TextWorld, we quote the average size of the archive throughout each episode.
In BabyAI-Text, we observe the archive becoming around $8\times$ larger without filtering.
These metrics demonstrate IGE’s innate ability to capture promising discoveries as they occur and focus attention on them, without the need for any manual heuristics.

**Table 3: Left: We show that intelligent filtering in IGE can drastically reduce the size of the archive, and help the algorithm focus on the most interesting states. Without intelligent filtering in BabyAI-Text and TextWorld, the archive becomes over $5\times$ larger. Right: We show that IGE scales with the capabilities of the underlying foundation model. We follow the same settings with Table˜2 and show the mean and 95% bootstrap confidence interval for the number of states and success rate.**
| Archive Filtering | Number of States |  |  |
| --- | --- | --- | --- |
| Game of 24 | BabyAI (PN) | TextWorld (CG) |  |
| No Filter | 18.5 $\pm$ 3.2 | 203.5 $\pm$ 56.7 | 22.4 $\pm$ 15.3 |
| With Filter | 15.6 $\pm$ 2.3 | 25.5 $\pm$ 5.2 | 04.4 $\pm$ 2.80 |

What is the Effect of Foundation Model Choice?
We also analyze the dependence of our algorithm on the strength of the foundation model by replacing GPT-4 with an earlier variant, GPT-3.5 in Table˜3 (right).
There is a considerable difference between the two, which suggests that our environments are non-trivial to solve, and that future advancements in foundation models are likely to readily scale the performance of IGE to even harder problems.

## 6 Related Work

FM-as-judge.
We employ FM guidance at all stages of IGE to drive exploration.
FMs as judges (Zheng et al., ; Bradley et al., ) have already seen use in decision-making tasks: OMNI (Zhang et al., ) considers FM guidance in multi-task settings to select the most promising next task to train on.
However, focusing on the broader task could miss out on interesting behavior that happens at a more granular level, and thus IGE greatly expands on the integration of FM intelligence into decision-making.
RL from AI Feedback (Bai et al., ; Lee et al., ; Klissarov et al., ) considers training RL agents using reward functions derived from FM preferences.
This similarly guides agents towards preferred states, but without the intelligence of FMs for action selection.

FM Agents.
One of the key strengths of IGE is that it is agnostic to the precise agent formulation and thus strictly additive on top of a wide variety of strategies.
A common strategy is chain-of-thought-based methods (Yao et al., ; Hu and Clune, ), which prompts the FM to output a set of reasoning steps before the answer.
We integrate this into the FM guidance in our experiments in Sections˜4.2 and 4.3.
Reflexion (Shinn et al., ) enables an agent to improve over multiple episodes by asking it to reflect on the previous attempted episode, and learn from its mistakes.
However, we show this can break down in tasks with long horizons, whilst IGE proposes a more efficient way to filter out the vast majority of uninteresting interactions.
Another set of agent frameworks that are related to the idea of exploring diverse solution paths via state-connectivity is Tree of Thoughts (Yao et al., ) and Graph of Thoughts (Besta et al., ).
In contrast, IGE can exploit search strategies that are not tied to any connectivity between states and can readily jump across the archive of promising saved states.
This is particularly important for long-horizon tasks with larger state spaces, as we show in Sections˜4.2 and 4.3.

Closely related to exploration, FM agents have also begun to see use in search-based tasks.
Stream of Search (Gandhi et al., ) considers a similar mathematical reasoning task to the Game of 24, and seeks to initially clone the actions of graph search algorithms, then use RL to self-improve.
In contrast, IGE already greatly outperforms classic graph search—an exciting future direction could be to first clone the exploratory behavior of Go-Explore and then self-improve with RL, enabling the FM to learn to select better.
Lehnert et al. () analogously train a language model to mimic the A^∗ algorithm.
Finally, Krishnamurthy et al. () also consider bootstrapping exploration with an externally summarized action-history in bandit problems; our focus is more on the detection of interesting states.

Go-Explore.
The original Go-Explore (Ecoffet et al., , ) framework enabled superhuman performance in a variety of hard-exploration problems, including applications as diverse as automated game testing (Lu et al., ).
Gallouédec and Dellandréa () propose Latent Go-Explore which similarly aims to address the difficulty of designing exploration heuristics by automatically learning a latent representation and sampling states with a low latent density.
However, this requires periodic retraining and could easily miss out on rare serendipitous discoveries.
HuGE (Torne Villasevil et al., ) guides Go-Explore with humans in the loop by asking for pair-wise feedback on which goal to select.
On the other hand, we take humans out entirely and apply intelligent FM guidance at all components of Go-Explore.

## 7 Conclusion and Limitations

In this paper, we demonstrate a new approach to robust exploration in complex environments, Intelligent Go-Explore, reimagining Go-Explore in the era of giant foundation models.
We show that IGE can drive exploration for a diverse set of FM agents, including few-shot and chain-of-thought prompting, across a variety of challenging text-based games.
While we only evaluate IGE on simulated text-based environments in this paper, a particularly exciting direction for future work would be domains with multimodal search spaces.
This could unlock applications as wide as scientific discovery in synthetic biology (designing novel drugs or proteins) or material science.
IGE could be readily adapted to these areas, as there is already precedent for multimodal FMs as judges (Wu et al., ).
A further direction that could break the limits of the current state-of-the-art in autonomous decision-making is the (hitherto unsolved by intelligent agents) dungeon crawler, NetHack (Küttler et al., ).
NetHack requires the discovery of complex strategies, deep game knowledge, and coherent behavior over an extremely long horizon.
Küttler et al. () noted that for NetHack, classic Go-Explore’s “heuristic of downsampling images of states to measure their similarity to be used as an exploration bonus will likely not work for large symbolic and procedurally generated environments.”
IGE represents a sharp departure from these limitations, by replacing hard-coded and inflexible exploration heuristics with the dynamic intelligence of giant foundation models.

There remain exciting opportunities to improve IGE’s capabilities to explore vast state spaces.
For example, we currently recall and compare against the entire archive whenever we discover a new state.
This could be made much more efficient by using techniques like retrieval augmented generation (Lewis et al., ) and only comparing to the closest previously discovered states.
As we consider IGE for real-world settings, we should take steps to ensure the responsible deployment of FMs (Bommasani et al., ).
Our approach opens up the road to safe and interpretable exploration: through careful prompt engineering or techniques like constitutional AI (Bai et al., ), we could steer the agent away from unsafe behaviors.
Furthermore, if we ask or train the FM to explain its choices in each part of IGE, we could gain insight into its rationale for exploring particular paths through an environment (Wei et al., ; Hu and Clune, ); improving safety, interpretability, and perhaps one day even our own understanding of how best to explore.

## Acknowledgments and Disclosure of Funding

This work was supported by the Vector Institute, the Canada CIFAR AI Chairs program, grants from Schmidt Futures and Open Philanthropy, an NSERC Discovery Grant, the Center for AI Safety Compute Cluster, and a generous donation from Rafael Cosman.
We thank Aaron Dharna, Ben Norman, and Jenny Zhang from our lab at the University of British Columbia for insightful discussions for insightful discussions and/or feedback on early drafts of this work.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.

## Supplementary Material

## Table of Contents

## Appendix A Algorithm Pseudocode

We provide full pseudocode for Intelligent Go-Explore in Algorithm˜1.
This complements the discussion in Section˜3.

Figure: Algorithm 1 Intelligent Go-Explore

## Appendix B Further Details on Environments

We provide further details for each of the environments used in the empirical evaluation in Section˜4, and the environment-specific descriptions appended to the system prompts.
Each environment description may include high-level information about the task and a description of the action space.

### B.1 Game of 24

We use the environment and set of evaluation tasks from [https://github.com/princeton-nlp/tree-of-thought-llm](https://github.com/princeton-nlp/tree-of-thought-llm) which is released under the MIT License.
We include the environment-specific prompt that is appended to the system prompt in Section˜3 below.
The system prompt contains examples of correct reasoning paths on different problems (few-shot prompting).

The action space at each step is all the valid arithmetic operations, presented in an analogous way as the ‘propose’ step in Yao et al. [].

### B.2 BabyAI-Text

The BabyAI-Text [Carta et al., ] environment comes with five task types, which we list here and visualize in order in Figure˜5:

- •
Go to <object>, a simple navigation task that requires reasoning abilities to choose the right plan given the object’s position;
- •
Pick up <object>, a reasoning task that combines navigation tasks;
- •
Pick up <object A> then go to <object B> and Go to <object B> after pickup <object A>, both serving to test reasoning abilities on temporal sequences;
- •
Unlock <door>, a task that includes inferring that a key is needed to unlock the door, finding the right key (i.e. the one colored as the door), and eventually using the toggle action with the key on the door;
- •
Put <object A> next to <object B>, which requires first reaching <object A>, picking it up, reaching <object B> and finally dropping <object A> next to <object B>.

Figure: Figure 5: We visualize the 5 types of tasks that BabyAI-Text consists of for our evaluation in Section˜4.2. IGE receives only partial text-based observations corresponding to the view in the figure.
Refer to caption: x7.png

We use the codebase from [https://github.com/flowersteam/Grounding_LLMs_with_online_RL](https://github.com/flowersteam/Grounding_LLMs_with_online_RL) which is released under the MIT License.
The action space is discrete and composed of 6 possible actions: turn left, turn right, go forward, pick up, drop, and toggle.
The ‘go to’ and ‘pick up’ tasks have a shorter environment horizon of $H=64$, whereas the rest have a horizon of $H=128$.
We include the environment-specific prompt that is appended to the system prompt in Section˜3 below.

### B.3 TextWorld

We evaluate IGE on ‘Treasure Hunter’, ‘The Cooking Game’, and ‘Coin Collector’ from the TextWorld [Côté et al., ] domain.
We use the environment code from [https://github.com/microsoft/TextWorld](https://github.com/microsoft/TextWorld) which is released under the MIT License.

#### B.3.1 Treasure Hunter

For Treasure Hunter, we set the ‘level’ option to the maximum value of 30, resulting in a maze with 20 rooms.
Locked doors and containers are added, which may need to be unlocked and opened to find the target object.
To further increase the difficulty, we remove the solution description from the original game and filter out tasks that can be completed with 20 steps in the optimal solution.
We include the environment-specific prompt that is appended to the system prompt in Section˜3 below.

#### B.3.2 The Cooking Game

In The Cooking Game, we set the number of ingredients to a maximum of 5 and the number of rooms to 13.
We enable all challenging additional options: doors need to be opened, food must be processed (e.g., cut, diced, chopped with a knife), and cooked (e.g., grilled with a BBQ, fried on a stove, roasted in an oven).
We include the environment-specific prompt that is appended to the system prompt in Section˜3 below.

We show a successful example trajectory found by IGE below, from our evaluation in Section˜4.3.

#### B.3.3 Coin Collector

In Coin Collector, we set the number of rooms to 40 and allow distractor rooms to be added along the way.
Similar to Treasure Hunter, we remove the solution description from the original game, and the optimal path from the agent’s starting point to the target is set to 20 steps.
We include the environment-specific prompt that is appended to the system prompt in Section˜3 below.

## Appendix C Further Prompt Discussion

### C.1 Extracting Choices

By default, in Section˜4.1, we prompt the FM to return a JSON object containing just the numerical index of the choice.
We choose this because of the ease of parsing the response and validating it lies within the correct bounds.
An example prompt is displayed below.

When using chain-of-thought as in Section˜4.2, we use the following prompt:

For the TextWorld environment in Section˜4.3, since the action space is much larger, we ask the FM to directly output a text action that we automatically parse.

We use the regex “> (.*?)(?:|̇$)” (in Perl notation) to parse the command.
We note that the failure rate for both of these options is very low, less than 0.1% across our evaluation.
Despite this, we include a failsafe that returns a random choice in case of an invalid output.

### C.2 Rejection-based Archive Filtering

The ‘acceptance-based’ archive filter in Section˜3.3 iterates through every new state and asks whether each one is interestingly new and should be added to the archive.
This can break down in larger environments where there is more need to explicitly deprecate earlier discoveries that have become irrelevant to not overload the archive, for example, in Section˜4.3.
In this environment, we use an alternate version of the prompt which first adds all states, and then asks the foundation model to remove the uninteresting states.
An example prompt is shown below.

## Appendix D Hyperparameters

In this section, we provide the hyperparameters for our empirical evaluation in Section˜4.
We list the hyperparameters for IGE in Table˜4.
We choose the values for exploratory rollout length based on the average number of steps needed to make ‘reasonable progress’ in the environment.

**Table 4: IGE Sampling Parameters. TH, TCG, and CC are abbreviations for Treasure Hunter, The Cooking Game, and Coin Collector in TextWorld.**
| Hyperparameter | Value(s) |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Game of 24 | BabyAI-Text | TH | TCG | CC |  |
| No. state expansions, $N_{\text{state}}$ | 50 | 25 | 24 | 48 | 125 |
| No. exploratory actions, $N_{\text{action}}$ | 3 | 10 | 5 | 5 | 1 |

We list the sampling parameters for GPT-4 [OpenAI, ] passed via the OpenAI API in Table˜5.

**Table 5: GPT-4 Sampling Parameters**
| Hyperparameter | Value |  |  |
| --- | --- | --- | --- |
| Game of 24 | BabyAI-Text | TextWorld |  |
| Temperature | 0.7 | 0.7 | 0.3 |
| Max new tokens | 1000 | 1000 | 1000 |
| Response format | JSON Object | JSON Object | Text |
| Version | Turbo-2024-04-09 | o-2024-05-13 | o-2024-05-13 |

We used GPT-4-Turbo for Game of 24 and GPT-4o for BabyAI and TextWorld.
This was purely done to select the version of GPT-4 that was available and the cheapest at the time of running the experiments.
The version of GPT-4 is consistent per environment.
We use a reduced temperature for the TextWorld domain to reduce the possibility of generating malformed responses, as actions are output in free-form natural language.
In our ablations in Section˜5, we use the ‘turbo-0125’ variant of GPT-3.5.

### D.1 Cost of Experiments

We provide the average cost per task for our algorithm per environment (the number of seeds is specified in Section˜4):

**Table 6: Per task API cost for IGE using GPT-4 listed in USD.**
| Environment | API Cost (USD) |
| --- | --- |
| Game of 24 | 1.04 |
| BabyAI-Text | 2.01 |
| TextWorld | 1.28 |

We note that the price per token of the ‘o-2024-05-13’ option is half that of ‘Turbo-2024-04-09’, so we could expect to achieve the same level of results on the Game of 24 with half the price.
The total cost of API access required to perform the final experiments in this paper was under 2,000 USD.
During development, we iterated on IGE with a smaller number of seeds, which represents a small fraction of this cost added on top.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Can the authors provide more details of how baselines are run? For instance, do all LM baselines get the same environment description for each game? 

2. Are there ablations of sensitivity of the approach to differences in prompts?

3. How exactly is the state conditional history (L199) implemented? Does it store a set of previously attempted actions for each seen state? Is it a buffer of (state, action) pairs?

4. Do you think the exploration problems Go-Explore addresses are also problems in the evaluation tasks? Specifically, detachment and derailment (L130)? 

5. How is state reset done (i.e. to explore from a certain state) in the provided environments?

6. In table 2, is there a difference between “classic go explore” and “ablated all 3 above” models?

7. Just out of curiosity, there do exist work that tackle exploration very explicitly in difficult games such as MineCraft [Wang 2023, Zhu 2023]. Is there a reason these methods are *not* reasonable comparisons against IGE?


[Wang 2023] Wang, Guanzhi, et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291 (2023).

[Zhu 2023] Zhu, Xizhou, et al. "Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory." arXiv preprint arXiv:2305.17144 (2023).
soundness: 2
strengths: **[Originality]**
The idea seems original, and creatively combines Go-Explore (a traditional RL approach for exploration) and LM agent tasks. 

**[Quality]**
The method shows strong performance over its baselines.


**[Clarity]**
The prompting procedures are well described and environments are well described 

**[Significance]**

The combination of RL exploration methods and LM agents feels both natural, thus this work opens up the possibility to further explore and improve components of the Go-Explore framework for better LM-agents.
summary: Combines the Go-Explore -- a framework for solving hard-exploration problems -- with language models (LMs) to create LM agents that do better on search and exploration tasks.
weaknesses: While I quite enjoyed the idea and appreciate that using LMs with the go-explore framework is a promising direction, the main weaknesses with the current version of the paper is that many of the stated / emphasized claims are quite strong, which I do not feel they are properly back-ed up. See below.

> IGE ofers the previously impossible opportunity to recognize and capitalize on “serendipitous discoveries” (L24, L76, L95)

- The use of the word “serendipitous” feels un-rigorous and only added confusion. I don't know what this word means in context, nor how it contributes scientifically to this paper.
- Based on my reading, the authors generally use this word when describing the ability to identify which states should be archived to later explore further. This is an improvement over the original Go-Explore which use a heuristic (cell representation) instead of LM-preference. Perhaps it is better to just state this as-is.
- Further, the claim that this was “previously impossible” feels very strong. It is also unclear *what* was previous impossible? There exist previous work that use LM as a feedback to derive intrinsic motivation (e.g. Klissarov et al 2023, which the authors cite, L474). 

> IGE integrates well with various agent strategies ... and will only get better as capabilities of foundation models improve further (L103)

- The experimental evidence for this claim is in Table 3 right, which shows IGE does better on GPT-4 than GPT-3.5. One hypothesis is that IGE gets better as foundation models improve; another hypothesis could be that IGE is only well-tuned to GPT-4 and may not work in other / later models. Experiments across multiple foundational models with different capabilities will make this more convincing. 

> Key strength of IGE is that it is a strict improvement on top of any FM agent framework (L195)

- Unclear if the authors mean IGE is *complementary* to other FM agent frameworks; or that IGE will always be the best FM agent framework?


The strong claims contrast against the empirical results, which are good, but marginally so (e.g. between Fig 3 and 4, IGE is statistically significantly better than other baselines in 3 out of 13 settings between Figures 3 and 4 (evaluated base on overlapping 95 confidence interval bars). I feel that a more measured presentation can aid in paper clarity.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above
soundness: 3
strengths: * The paper considers a fundamental problem of managing the exploration-exploitation trade-off.
* IGE proposes to leverage the capabilities of LLMs to address deficiencies in the original GE and shows that it succeeds in three families of environments.
* The paper fits into the important line of work concerned with applications of foundation models in RL.
summary: The paper revisits the Go-Explore (GE) algorithm (Ecoffet et al., 2021) by letting a foundation model (FM) control three key operations performed on an archive of interestingly new states: (a) select a state to explore from (b) choose actions from the selected state, and (c) decide which state is worth keeping. The paper argues that FM successfully replaces GE's requirement to use hand-designed heuristics domain-specific pre-specified knowledge. IGE, with GPT4 as the underlying LLM, is evaluated on three families of environments (Game of 24, BabyAI, and TextWorld). It is claimed that the method outperforms baselines consistently across the board, sometimes being the only method that shows life (e.g., for Coin Collector). Finally, the following ablation studies are provided: (a) the importance of FM, (b) the size of the archive, and (c) the capabilities of FM.
weaknesses: * The choice of a setup for IGE matters, as it seems to work for deterministic environments. Is the extension to stochastic environments a fundamental limitation of IGE, or is there a "natural" extension? One example could be NetHack, which is a known hard exploration problem with long-horizon and credit assignment challenges.
* The IGE's procedure of selecting interesting states from the archive resembles a hierarchical search, where the planning is performed on high-level milestones or subgoals. Often such methods use Best-First-Search as a backbone planner, hence they jump freely between subtrees, similarly as IGE (see a discussion in lines 298-302 or 490-496). Can the Authors comment on that?
* Related to the above, how does IGE compare against algorithms like MCTS for hard deterministic combinatorial problems (like chess or go)?
* What is the relation of IGE to GFlowNets, which searches through a design space to generate samples proportional to a specified reward function?
* What important real-life problems can be expressed as exploration and solved via IGE? Say, interesting chess or go states, novel mathematical theories or proof ideas, useful molecules, etc.
* Can the method accelerate research on open-endedness, which is often formulated as an unsupervised exploration problem?
* How robust is the method?
	* LLMs are known to be unreliable and have a long list of failure modes.
	* Are there any biases with respect to some environments games, i.e., the method performing better in the environments that were prominently represented in the pre-training dataset?
	* Conversely, does IGE underperform for environments that were underrepresented in the training dataset or that require a set of rules or wiki?
* How can IGE be augmented with the inclusion of environment rules, RAGs, and tools? How hard is it to set up IGE for a new task? How much work is expected on the prompt engineering part?
* What is the performance of open-sourced models? 
* IGE is an in-context method. What could be achieved that currently is out of reach if the method allowed fine-tuning (or RL training)?
* What could be the implications for the current RL algorithms regarding the potential benefits stemming from IGE providing better data?
* Can we formulate an inverse-IGE problem by designing synthetic datasets to discover the "definition of interestingness of LLMs"?


Other:
* In Algorithm 1, lines 3, 8, 10, should account for the fact that "$\gets$" typically stands for assignment operator, not "add to a set" operator.

Edit 27th November 2024: Based on the Authors' response, I have increased the score to 8.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - How would you transfer this method to continuous state-action spaces? Did you observe a limit regarding how large the state archive can become? For instance int the BabyAI environments the observations are discrete, that is each observation can be mapped to a distinct vector. I imagine this becomes more problematic the larger the environment becomes. 
- Did the foundation model hallucinate while creating the trajectories? This would be interesting to see, since failure modes for foundation models seem to differ from traditional RL models.
soundness: 2
strengths: - Exploiting the pre-trained biases of LLM for RL is an interesting idea and so far largely unstudied for exploration in reinforcement learning
- I like that the foundation model is defined in terms of a well known exploration algorithm such as Go-Explore, since it shows how concepts from exploration can be transferred to large foundation models.
- The paper is very clear and well written, while providing thorough ablations methods components such as in Section 4.3.
summary: Summary: The authors propose a method that reconceptualises the original Go-Explore method to use the pre-trained knowledge contained in large foundation models. 
For this, the original components of Go-Explore such as keeping a state archive and exploring from a previously attained state are defined for a language foundation model. 
The idea behind this is that foundation models are better able to reason about what defines an interesting (novel) state or behaviour, while exploiting the creative abilities of foundation models. The authors test on 3 different environments. 2 text based environments and one environment where the visual 
Features are described in terms of text.  The authors show that intelligent go-explore is able to solve these environments in very few steps while also outperforming other foundation model based methods. 


Overall, this is a well written and interesting paper. I think the idea of creating a foundation model based exploration algorithm that follows along the lines of Go-Explore is clever and perhaps shows a more generalised way to exploit foundation models for exploration. 
While the authors ablate their method well, I feel more interesting ablations would concern the capacity of the foundation model itself in terms of how complex the observation space can get and how different prompts, i.e., conditioning affects the exploration. For instance,
Did the foundation model hallucinate during some runs? What do these hallucinations look like? I think this should even be achievable with the presented BabyAI environment. See more detailed info below.
weaknesses: - This paper argues against hand-crafted features i.e. heuristics, however having to transfer the environments to a prompt based structure is also a hand crafted feature space.
- The paper states in the abstract that the appeal of using foundation models is making use of the foundation models capabilities of reasoning ahead or capitalising on serendipitous discoveries.  While the results are good, can you show examples of trajectories IGE creates compared to a normal RL algorithm that you could argue are serendipitous?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could you provide more details on the sensitivity of the prompts used with IGE? For example, in the BabyAI environment, you explicitly prompt the model with instructions such as 'do not repeat actions if they have no effect.' How much does IGE's performance depend on these kinds of statements?
- Given that the action selection of the foundation model has the most significant impact on IGE's performance, have you explored other exploration approaches? It seems that random actions and foundation model-driven actions represent two extremes. Have you considered comparing IGE to a relatively simple information-seeking policy or other exploration strategies that could computationally be much cheaper to use?
soundness: 2
strengths: - The integration of foundation models with the Go-Explore framework is a noteworthy contribution that can alleviate several of the inflexible inconveniences present in the original framework, such as hand-crafted heuristics and random exploration.
- The authors perform a solid empirical evaluation and compare it to several foundation model agent baselines, demonstrating significant performance gains.
- For each of the proposed additions to the original algorithm, a solid ablation study is performed where each addition is omitted and evaluated, and IGE is also compared to the original Go-Explored in this experiment. 
- The authors do not limit their approach to text-based representations and demonstrate that IGE can also be applied with visual observations.
summary: The authors propose Intelligent Go-Explore (IGE), an extension of the original Go-Explore algorithm that replaces manually crafted heuristics with foundation models (FMs) to guide exploration. In the original Go-Explore, heuristics were required to identify 'interesting' states, whereas in IGE, this is determined by querying the foundation model, which leverages its pretrained knowledge to make decisions. Interesting states are stored in an archive, and the foundation model is queried with which state should be explored next. Additionally, instead of relying on random action sampling, IGE uses the foundation model as a decision-making agent to choose actions intelligently. The authors evaluate IGE in controlled environments (BabyAI and Game of 24), showing that it outperforms previous foundation model-based agents in these tasks.
weaknesses: - One of the main contributions of the paper is alleviating the original Go-Explore from manual handcrafting of domain-specific heuristics. However, by replacing this part of the process with foundation models, quite extensive and domain-specific prompt-engineering is required instead (as shown in Appendix B). 
- Given that the environments used in the paper are relatively controlled and intuitive, and the foundation models were provided with detailed explanations of the environment dynamics and objectives, it is not clear how well the decision-making of the foundation models would work in more complex (and actual hard-exploration) environments. For instance, the Game of 24 is a relatively simple mathematical puzzle, where the exploration space is highly constrained and the rules are clear and deterministic. As such, it doesn’t pose a true hard-exploration challenge, which is what Go-Explore and IGE are meant to address.
- Building on the previous weakness, the original Go-Explore algorithm was demonstrated across the Atari suite, including hard-exploration games like Montezuma's Revenge. In this IGE paper, the authors compared Go-Explore in specific environments where IGE performs well, but it would have perhaps made more sense to evaluate it in the Atari suite, where we already know Go-Explore performs well. It is not clear that a foundation model would be able to deal with the complexity of Atari games both in judging interesting states and in decision-making. Given that the main claim of the paper is improving upon the original framework, a comparison on that suite would have provided a more direct measure.
- Although the authors touch upon this briefly, IGE appears computationally expensive, as the foundation model is presented with the entire archive of states to select a state, filters states, and is also queried for every individual action. While this may be feasible in the small environments used in the paper, it’s unclear how scalable the approach would be to larger environments or longer-horizon tasks. 
- Following this, according to the ablation study, the primary driver of IGE’s performance is the action selection guided by the foundation model, compared to random action selection. However, foundation models tend to only perform well in relatively intuitive and deterministic domains where decisions are relatively straightforward. In more complex or large-scale environments, a reinforcement learning exploration policy will likely be necessary. This raises an important question: if the primary performance gains from using IGE come from action selection, and these foundation models struggle in more complex environments, are the gains from using foundation models for state selection and filtering substantial enough to justify its use? The ablation study shows that these gains are minor (yet introduce significant computational overhead), calling into question the overall value of IGE when considering scalability and generality.
- The paper occasionally comes across as over-optimistic with respect to the proposed algorithm and e.g. speculates that its solutions can easily be used for downstream RL tasks or infinite in-context learning with no evidence supporting this. A more grounded discussion of IGE's limitations would benefit the paper.

Meta-review: The paper proposes a method that uses language models to guide exploration in reinforcement learning. This addresses an important question, uses timely tools, and advances understanding and capability. The paper is written and motivated clearly and includes sufficient empirical evaluation, including ablations. Some concerns were raised, and some remain, regarding overly strong or broad claims made in the paper, in particular regarding its applicability, as well as omission of limitations. The authors have sufficiently but not fully addressed these.

Decision: Accept (Poster)