Go-Explore with a guide: Speeding up search in sparse reward settings with goal-directed intrinsic rewards
Authors: Chong Min John Tan
Mehul Motani
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
intrinsic motivation
goal-directed rewards
hippocampal replay
hard-exploration
sparse rewards

Abstract: Reinforcement Learning (RL) agents have traditionally been very sample-intensive to train, especially in environments with sparse rewards. Seeking inspiration from neuroscience experiments of rats learning the structure of a maze without needing extrinsic rewards, we seek to incorporate additional intrinsic rewards to guide behavior. We propose a potential-based goal-directed intrinsic reward (GDIR), which provides a reward signal regardless of whether the task is achieved, and ensures that learning can always take place. While GDIR may be similar to approaches such as reward shaping in incorporating goal-based rewards, we highlight that GDIR is innate to the agent and hence applicable across a wide range of environments without needing to rely on a properly shaped environment reward. We also note that GDIR is different from curiosity-based intrinsic motivation, which can diminish over time and lead to inefficient exploration. Go-Explore is a well-known state-of-the-art algorithm for sparse reward domains, and we demonstrate that by incorporating GDIR in the ``Go" and ``Explore" phases, we can improve Go-Explore's performance and enable it to learn faster across multiple environments, for both discrete (2D grid maze environments, Towers of Hanoi, Game of Nim) and continuous (Cart Pole and Mountain Car) state spaces. Furthermore, to consolidate learnt trajectories better, our method also incorporates a novel approach of hippocampal replay to update the values of GDIR and reset state visit and selection counts of states along the successful trajectory. As a benchmark, we also show that our proposed approaches learn significantly faster than traditional extrinsic-reward-based RL algorithms such as Proximal Policy Optimization, TD-learning, and Q-learning.

Go-EXPLORE WITH A GUIDE: SPEEDING UP SEARCH IN SPARSE REWARD SETTINGS WITH GOAL-DIRECTED INTRINSIC REWARDS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Reinforcement Learning (RL) agents have traditionally been very sample- intensive to train, especially in environments with sparse rewards. Seeking in- spiration from neuroscience experiments of rats learning the structure of maze without needing extrinsic rewards, we seek to incorporate additional intrinsic re- wards to guide behavior. We propose a potential- reward (GDIR), which provides a reward signal regardless of whether the task is achieved, and ensures that learning can always take place. While GDIR may be similar to approaches such reward shaping in incorporating wards, we highlight that GDIR is innate to the agent and hence applicable across a wide range of environments without needing to rely properly shaped environ- ment reward. We also note that GDIR is different from curiosity- intrinsic motivation, which diminish over time and lead inefficient exploration. Go- Explore is well-known state-of-the-art algorithm for sparse reward domains, and we demonstrate that by incorporating GDIR in the "Go" and "Explore" phases, we can improve Go-Explore's 's performance and enable it to learn faster across multi- ple environments, for both discrete (2D grid maze environments, Towers of Hanoi. Game of Nim) and continuous (Cart Pole and Mountain Car) state spaces. Fur- thermore, to consolidate learnt trajectories better, our method also incorporates novel approach of hippocampal replay to update the values of GDIR and re- set state visit and selection counts of states along the successful trajectory. As a benchmark, we also show that our proposed approaches learn significantly faster than traditional extrinsic-reward-ee based RL algorithms such as Proximal Policy Optimization, TD-learning, and Q-learning.
roal-directed intrinsic
ball-baedd re-
INTRODUCTION
Recently, LeCun (022) describes how intrinsic cost modules could motivate an agent's behavior. These intrinsic cost modules can be something like hunger, thirst or goal-seeking that is innate to the agent and cannot be modified. The benefit of having such an immutable cost module is that one's previously learnt values of the state will not be affected by a continually changing model (like the function approximators used in typical Deep Reinforcement Learning (RL)) and the agent can learn efficiently without the need to re-visit past experiences each time the model is changed.
While Silver al. (202T) states that environment- based extrinsic reward can be enough for learning complex skills and social behaviors, he also admits that it can be sample inefficient. Adding an intrinsic component to this environmental reward can be seen as giving some self-driven intrinsic motivation to learning such skills or behaviors, and can better lead to attaining these competencies with better sample efficiency.
The presence of intrinsic motivation can be seen in neuroscience experiments on rats exploring maze even without extrinsic food rewards (Fitzgerald et al. 1985] Hughes) 1997). This is not easily explained from just the perspective of extrinsic environmental rewards and suggests that intrinsic motivation does play a huge part in natural behavior of animals, and it could be the critical missing component in modern RI. methods.

---

Motivated by these observations, we seek to find such intrinsic cost/reward functions whereby it is innate to the agent, but is context dependent and can be triggered according to the task at hand. We seek to incorporate this intrinsic cost/reward into RL models and derive the benefits of these intrinsic rewards in driving behavior. Specifically, we model a potential-based goal-directed intrinsic reward (GDIR) which tells agent how far it is from the goal in order to guide actions.
The proposed GDIR fits naturally in a multi-step task setting, where there can be a planner module that tells us what are the sub-tasks required to achieve the goal. How to derive this planner mod- ule is tackled in Hierarchical Reinforcement Learning (AI-Emran 2015 Pateria et al. (021). Our proposed GDIR can help to achieve these sub-tasks faster as provides learning reward signal regardless of whether the task is achieved.
In contrast to existing work in RL, we do not seek to find the best possible solution for given environment. We seek instead, to find satisficing solution, whereby the solution is good enough to solve the task. We posit that one pitfall of seeking the optimal solution that extensive exploration needs to take place, even after solving the environment, in order to ensure that the optimal path is traversed at least once. Hence, we opt for the satisficing route via our novel approach of hippocampal replay, which consolidates successful trajectories and repeats them consistently. This allows our method to learn faster and adapt better to novel environments in real-world systems.
Our Contributions. In this paper, we investigate using GDIR in Go- Go-Explore, which is well-known state-of-the-an algorithm for sparse reward domains. We highlight the following contributions:
1. We propose GDIR, potential- based goal-directed intrinsic function, which provides a reward signal regardless of whether the task is achieved, and ensures that learning can always take place. 2. We incorporate GDIR into variants of Go-ExpoEeee -o-plxl and demonstrate that it enables Go-Exp Po-Explore to learn significantly faster across multiple environments, for both discrete and continuous state spaces.
3. In order improve consolidation of learnt trajectories, we propose novel approach of hip- pocampal replay to update the values of GDIR and reset state visit and selection counts of states along the successful trajectory. We demonstrate that hippocampal replay helps an agent remember successful trajectories which solve its current environment and enables it to perform consistently.
RELATED WORK
Intrinsic Motivation, Oudeyer & Kaplan (2009) describes the definition of Intrinsic Motivation in psychology and gives a list of computational approaches to model it. [Chentanez et al. (204)) details how to incorporate intrinsic motivation into traditional RL algorithms such as Q-Learming, [Schmid-] huber (2010) describes a form of intrinsic reward related to discovery of novel, suprising patterns
and world model prediction. [Baldassarre & Mirolli (2013) details intrinsically motivated learning
in natural and artificial systems. Aubret et al. (2019 2022) gives a general overview of the field of
using intrinsic motivation in RL, namely in knowledge acquisition via exploration, empowerment, state space representation, as well as skill learning. Similar to all other prior work on intrinsic moti- vation, we seek derive computational method to incorporate it into a learning system such that it speeds up learning. However, we do not focus on the suriosity-basee intrinsic motivation but a goal-directed form of intrinsic rewards, due reasons explained below.
Go-Explore. Go-Explore (Ecoffet et al., 2019) describes two common pitfalls of intrinsic motiva- tion. The first is detachment, whereby intrinsic rewards such -uriosity-driven exploration tend to diminish over time, as the frontiers of the exploration space may be explored multiple times but the extrinsic reward may not be obtained. In the end, the frontier may not be attractive anymore and is neglected in subsequent exploration. The second is deruilment, whereby an agent seeking to return to reviiously-epplore good state may hindered from doing so, due to the in-built stochasticity of the algorithm, such as resilon-greedy exploration (Montague 1999). state-dependent exploration with action-spaae noise (Rückstieeß et al. 2008). or aurameter-space noise (Plappert et al. (017))
(Ecoffet et al.) 2021) is an extension of Go- Explore using posicy-based exploration, which shows the capability of the Go-Explore algorithm to lear difficult MuJoCo robotic task which cannot be solved by curiosity-basee intrinsic moti- vation. Yang et al. (2022) details how post-exploration (exploring even after reaching the goal state) can help expand the agent's state horizon, and even lead better performance the same task.
Extensions of Go-Explore.
peatt-return-tee-eexppp
2

---

Gallouédec & Dellandréa (2022) describes extension of Go-Explore which uses dynamically-
learnt latent representations.
Count-based exploration. While we acknowledge the potential diminishing of turiosity-based rewards affecting search algorithms, a similar approach to balance explore and exploit is still crucial for search problems. As such, we utilize tried and tested count- based exploration is typically used in Upper Confidence Bounds for Trees (UCT) used in Monte Carlo Tree Search (MCTS) Chaslot eet al. (2008). Such an algorithm will guarantee that there will still be exploration in the limit, although the exploration rate will reduce logarithmically. A variant of such a UCT algorithm was used successfully in AlphaGo (Silver al., 2016), AlphaGo Zero (Silver al. 2017) and AlphaZero (iilvrr et a 2018). In the continuous domain, Bellemare et al. (2016) uses density- based model as a proxy for count-based exploration methods. In our work, we focus on the discrete case by binning continuous state spaces if necessary. It would be interesting to see if we can also incorporate such a density-based model in future work to deal with continuous spaces directly.
Goal-directed learning. We seek to guide the search via goal-directed approach, much like us- ing a compass to find the way in the forest. One method of learning from binary goal-rewards is Hindsight Experience Replay (HER) (Andrychowicz et al. 2017), where it can be viewed as an implicit curriculum learning method to learn different goal states based on sampling from experi- enced trajectories. Another way to direct learning towards the goal is to have constraints
to prevent over-exploration of state space, such as a cost function to encourage RI. agents to follow the thortest-path between the start state and the goal state (Sohn et al., (021). As these methods can be lengthy to learn in practice, we seek to model a potential-based GDIR instead which is equal to 0 when the agent is the goal, but is of increasing negative value the further the agent is away from the goal. This is very similar to potential-based reward shaping (Grzes) 2017), where the goal states have potential zero, and the non-goal states have non-zero potential. Such formulation has a theoretical guarantee for optimistic exploration, which is ideal in sparse reward settings. While GDIR may be similar to approaches such as reward shaping in incorporating a goal-based reward, we would like to highlight that GDIR is innate to the agent and hence applicable across a wide range of environments without nceding to rely on a properly shaped environment reward.
METHODS
We detail the agents we use in our paper. There are a total of 7 different agents:
1. Random Agent. This agent chooses valid move randomly. It is the simplest and serves as the worst-case baseline for all other agents.
2. Q-Learning Agent We use an online method for Q-Learning updates, with one-step model- based lookahead and epsilon-greedy policy. Refer to Section A details.
3. TD-Learning Agent. We use online method for TD-LLarning updates, with one-step model- based lookahead and epsilon-greed, policy. Refer to Section for details.
4. PPO Agent. This agent uses the PPO algorithm (Schulman et al., 2017) implemented on Stable Baselines 3 (Raffin et al. 2019). and is used to benchmark our proposed agents' performance in continuous state space domains due to its superior performance in them.
5. Go-Explore Agent. Go-Explore was implemented to be similar to the original paper (Ecofret let al., 2019 2021). except that when select states in the "Go" phase (Refer to Algorithm 2). we do so in a deterministic (rather than probabilistic) manner to help the algorithm learn faster. If GDIR is used, we use it for the "Go" state selection phase in order to gauge how far a state is the goal state in order to bias exploration towards more promising states, much like how the heuristic function in A* search (Hart et al., 1968) works. 6. Go-Explore-Cout
Agent (GDIR). While Go-Explore uses a random policy for exploration, Go-Explore-Cone Count performs the best action based on a UCT-like selection function. This helps the algorithm explore more promising paths and helps it to solve tougher environments. If GDIR is used, we use it for both the "Go" and "Explore" phase.
7. Explore-Couu Count Agent (GDIR). Explore-Count is similar to Go-Explore-Count, except that it does away with the "Go" step and seeks to find the trajectory directly from the start state to the goal state. While this may be inefficient to find the exploration frontier, it allows for exploration without the need to return to the "Go" state which may be impractical for stochastic environments. If GDIR used, we use for the "Explore" phase.
3

---

Model-based learning. We note that we extensively use model of the environment to make future state predictions, so as to be more efficient in collecting experiences for our agent without the need to traverse all the future states. Such model of the environment will be easily obtained if one has access to the simulator of the environment, and the exact transition and reward functions need not be known we only require the use of the model as black box to make predictions.
Memory. The algorithm for Go-Explore variants (Go-Explore, Go-Explore-Con Couutt and Explore- Count) as detailed in Algorithm [ Of note, we use similar memory update procedure as in Ecoffet et al. (2019) 202T). For each state, we store in memory: 1) trajectory of actions to reach that state, 2) the number of moves to reach it, 3) the extrinsic reward, 4) the intrinsic reward (unique to our paper). 5) the number of selections of the state in the "Go" phase, and 6) the number of visits to the state in the "Explore" phase. For new memory state, we will initialize it with the visit and selection counts to O. For each state we visit, we will update the memory of the next state if the one-step lookahead trajectory has a higher extrinsic reward, or has the same extrinsic reward but shorter trajectory. This is an efficient method to keep the shortest performant trajectory in memory.
"Go" and "Explore" State selection. The selection function used for selecting the "Go" state as well as for choosing the best state to "Explore" in all the Go-Explore variants is given by:
oER+ BIR my/moves + TV numnciiited + numselected,
(1)
where ER and IR represent extrinsic reward and intrinsic reward respectively, moves represent the minimum number of timesteps taken to reach that state, numnvisited represents the number of times the state has been visited, numselected represents the number of times the state has been selected by the "Go" phase of Go-Explore. This will choose states with higher reward and more moves (greater chance of being at the frontier of explored states), while choosing states that have been relatively unvisited and not selected more often. This is similar to the UCT algorithm which to encourage greedy action selection in the long
decays the exploration term logarithmically run. These attributes are all extracted from memory for the state in consideration.
a, 0,8,, Ni 7 hyperparameters to weigh the importance of each term, and are given the respective values 1000, 10, 1, 100. These values can be found using hyperparameter tuning approaches such as grid search or genetic algorithms and can be adapted to the environment in consideration balance between explore and exploit.
GDIR. In the environments considered, we want to make use of GDIR that is linked to the task completion. We incorporate this GDIR in our Go-Explore variants. This GDIR should range be- tween 1 to so that it can be used in any environment without needing to change the hyperparame- ters of the selection function. It should be 0 when the task is completed at the goal state, and should be a negative number if the agent is not at the goal state. While is preferable that the magnitude of the GDIR correlates with how far the agent is away from the goal state for faster solving, it is not required. This GDIR must be easily formulated based on the task conditions alone without any expert domain knowledge and must not give away the solution to the environment it merely serves as a compass to guide the agent towards more promising areas of the state space. Mathematically, given the current state c, and the goal state g, GDIR = -d(c, 9), where d(-)) is an immutable distance function as a proxy to the distance hetween e and g, with output range [0. 1].
Hippocampal Replay. Hippocampal replay was performed upon finding a memory with trajectory that solves the task (i.e. non-zero extrinsic reward) for the Go-Explore variants. This allows us to reset the selection and visit counts of the entire path of states leading to the goal state to 0 to facilitate exploration along a good path. Furthermore, we let the goal state have an intrinsic reward of 1, and update this path of states to have an intrinsic reward equivalent to the discounted intrinsic reward of the next cell (discount factor y 0.99 in our case), so to encourage choosing back this sequence of moves during future exploration. Unlike Mnih et al. (2013) and [Schaul et al.. (2015). we do not randomly sample priority sample the buffer, but instead just perform one forward play (pre-play) in order to do memory retrieval and get the entire list of states along the trajectory, and a reverse play (replay) so as to update the intrinsic rewards and reset state selection and visit counts. This is akin to biological pre-play and replay patterns shown in hippocampal replay (Joo & Frank 2018) in mice. It is more efficient, in one hippocampal replay step, all the states can be updated with the correct intrinsic values, as opposed to updating just one transition sampled from the buffer. The algorithm for hippocampal replay is detailed in Algorithm 3 and illustrated in Fig.
4

---

4 EXPERIMENTS
We compare the agents' performance in two types of environments. The first set of environments are those with discrete state values and discrete actions which are typically used in planning and problem solving. The second set of environments are those with continuous state values and discrete actions which are usually used for robotic control. We intentionally choose sparse reward environments, or modify the environments as to make the rewards sparse. This will pose difficulty for traditional RL algorithms, and hence enable us to explore the limits of our proposed Go-Explore variants.
4.1 DISCRETE STATE ENVIRONMENTS
In total, we test our agents on four environments with discrete state space and action space. They are namely: Unwalled Maze, Walled Maze, Towers of Hanoi, Nim. The environment specifications are detailed in T The unwalled and walled mazes are designed to test easy and difficult spatial navigation respectively. Towers of Hanoi is more difficult search problem with a specific sequence of moves to follow, though there is room for few redundant moves to achieve the goal since we give the agents a move limit which is times more than the optimal number of moves needed. Nim, repackaged
single player environment with perfect play opponent (See Section [D] for strategy for perfect play), is the hardest search problem whereby we only have one possible option per move in order to win the perfect play opponent. These environments are proposed in order test the efficiency of the Go-Explore variants on problems of increasing difficulty. To provide a baseline, also compare the performance of the Go-Explore variants against the Random, Q-Learning and TD-Learning agents.
4.2 CONTINUOUS STATE ENVIRONMENTS
In total, we test our agents on two environments with continuous state space and discrete action space. They are namely: Cart Pole and Mountain Car. These environments are part of the Classical Control environments
OpenAI Gym (Brockman et al. (016). Rather than use the original reward function, we make the problem harder by using only sparse reward function for the environments, where reward of 1 is only given if the agent solves the problem, and 0 otherwise. The environment specifications are detailed in Table Cart Pole is problem where we need to continue to maintain the initial state of balanced pole which we were in, while Mountain Car challenging problem with a goal state that requires going further away from it to reach it. To provide a baseline, we also compare the performance of the Go-Explore variants against the competitive PPO agent.
State Space Abstraction. Due to the limitation of the memory approach we are using for Go- Explore, we cannot deal with a continuous state space which has infinite possibilities. As such, we drastically simplify the state space by binning the continuous state space. For our bins, we choose the size of 0.01 for position values, and 0.001 for velocity values, which we found to give good performance. How to choose the right bin based on the environment dynamically can be topic of future research. As some values can also span the range of -oc to 2o, we have an upper and lower limit for binning to prevent infinitely many states. We set this upper/lower limit to be +42 for Cart Pole and +50 for Mountain Car in order to match the maximum allowable pole angle of #0.418 and goal state x-position of 0.5 respectively. Mathematically, as such: given bin size m and upper/lower limit of bin u, we discretize the original continuous state value v to discrete value min(max( -.),, u).
describe the binning process
4.3 GOAL DIRECTED INTRINSIC REWARD
We use different GDIR functions across our various environments, as detailed in Tables and
Discrete Environments. For the maze environments, the GDIR is proportional to the Manhattan distance of agent's 's position to the door position, which only tells the agent the shortest distance to the goal if the walls are not present the agent still has to figure out how to navigate these walls. For Towers of Hanoi, the GDIR is proportional to the number of disks matching the goal state, which similarly does not tell the agent how to achieve the goal state but merely serves guide. For Nim, the GDIR is proportional to the number of matchsticks remaining after each move, which encourages the agent reach 0 matchsticks but does not tell the agent how
5

---

Continuous Environments. For Cart Pole, the GDIR is proportional to the absolute value of the cart position Te and the absolute value of the pole angle pe. since it is given that the environment will terminate early if F -2.4 I, > 2.4 or po 0.2095 rad Po < 0.2095 rad. Incorporating this GDIR is fair because a human who plays the game will already know of these conditions, whereas the reward structure in the environment will be sparse and the agent will have to figure out for itself how not to terminate the environment from trial and eπror. For Mountain Car, the GDIR is proportional the distance away from the end state of car x-position x 0.5. This similarly tells the agent that the goal is to move towards = 0.5, which is also known by human, but does not give away the solution that the car needs to move left and right along the hill valley in order to increase velocity to reach the top of the hill.
4.4 EVALUATION DETAILS
For each environment, the agent will act and learn in the environment for total of 100 runs (episodes). At the beginning of evaluation, we first initialize memory to be an empty dictionary to be updated across the 100 runs At the start of each run, we perform the "Go" phase to select a state to start from (only for Go-Explore variants). Thereafter, for each time step till termination, we query the agent for the next move. At the end of each run, we will perform hippocampal replay to consolidate good trajectory if there are any (only for Go-Explore variants). The entire proce- dure is described in in Algorithm 4 For some experiments such as those involving neural networks or random variables, we perform TO trials of 100 runs, and report the average across the 10 trials. Specifically, we are interested in the run where the agent first solves the environment as that is re- lated to how adaptable it is to a new environment, as well as the minimum number of steps in any found solution trajectory as that is related to how efficient the algorithm is.
RESULTS
5.1 DISCRETE STATE ENVIRONMENTS
Overview. We compare the run where the first solve occurs (adaptability) and the minimum solve time steps (efficiency) of any successful run out of 100 runs across all algorithms Tables T and respectively. We consider Unwalled Maze and Walled Maze with dimensions 10x10, 20x20 and 100x100, Towers of Hanoi with 3 and 7 disks, and Nim with 11, 21 and 1001 starting matchsticks. The full experimental results are available at Sections E. F
G
and H
First Solve. Overall, Explore-Count GDIR performs the best, with a first solve run of or 2 for most environments, except for Towers of Hanoi with 7 disks where it needs 10. Go-Explore-Coum GDIR also does very well, except that it does not solve Towers of Hanoi with 7 disks. Even without GDIR, Explore-Count and Go-Explore-Count still perform well, but may require slightly more runs for solve. Random, TD-Learning, Q-Learning and Go-Explore do not perform well for complex environments like Walled Maze, Towers of Hanoi, Nim, often failing to solve the environments.
Minimum Steps. In terms of minimum steps, Explore-Couse Count GDIR performs the best for Walled Maze and Nim. For simple environments like Unwalled Maze, TD-Leaminn performs the best. This highlights that traditional RL. algorithms perform efficiently if the state space is small, but Explore-Coune performs quite well too even in complex environments. Go variants (Go-Explore, Go-Explore-Counl do not perform very efficiently, often having higher minimum steps.
Problems with jumping to the "Go" state. Go-Explore does not perform well and often does not solve the environments. This may be because deterministically selecting the "Go" state and prioritizing states which are unvisited often means that states with longer trajectories are selected. Perhaps if we had followed the probabilistic selection employed in Ecoffet et al. (211)), we could have solved more environments at the cost of a longer training time. Furthermore, unlike in Ecoffet et al. (2019), the environments we considered all have a step limit, hence picking a "Go" state with a long trajectory may mean that there is very little room left to explore. This trait of Go-Explore needing long trajectories can also be seen in Ecoffet et al. (2019) where the agent in Montezuma's Revenge exceeded the maximum time steps of the default OpenAl gym environment. Perhaps Go- Explore methods will be more viable if we do not limit the maximum timesteps. That said, with more efficient exploration policy using count- based methods, Go- Poo-Explore-Cae Count mitigates the issues of jumping to the "Go" state and does quite well for the environments.

---

Table 1: First Solve Run for Discrete State Environments (lower is better, '-' means unsolved)
Unwalled
Walled
Hanoi
Nim
Agent
10
20
100
10
20
100
3
7
11
21
1001
Random
15
8
TD-Leaminng (Train)
21
15
4
27
TD-Learning (Test)
1
1
1
Q-Learning (Train)
15
4
27
Q-Learning (Test)
1
Go-Explore
33
Go-Explore GDIR
12
75
33
5
1oo-Explore-Cooure
2
1
1
1
Co-EExlorre-Cour GDIR
1
1
1
1
2
Explore-Count
2
1
1
1
17
Explore-Count GDIR
1
1
10
Table 2: Min Steps solve for Discrete State Environments (lower better, means unsolved)
Unwalled
Walled
Hanoi
Nim
Agent
10
20
100
10
20
100
3
7
11
21
1001
Random
54
9980
17
3
TD-Learning (Train)
44
264
5544
13
3
TD- TD-Learning (Test)
18
38
198
7
3
Q-Learning (Train)
44
264
5544
13
3
O-Learning (Test)
18
7
3
Go-Explore
13
3
6
Go-Explore GDIR
62
92
13
3
6
Go-Explore-Count
62
356
7810
64
218
4912
9
3
6
251
Go- Go-Explore-Couse Count GDIR
20
42
230
60
220
4042
3
6
251
Explore-Count
22
76
482
42
138
3160
9
378
3
6
251
Explore-Count GDIR
20
42
222
34
82
1424
327
3
6
251
5.2 CONTINUOUS STATE ENVIRONMENTS
Overview. We compare the run where the first solve occurs (adaptability) across all algorithms in Table 3, We consider Cart Pole with the goal state surviving for 50, 100 and 175 time frames. As Cart Pole has high chance of encountering the same state (usually the equilibrium state where pole balanced), which can negatively affect count-based algorithms by artificially inflating the visit counts, also consider encoding repeated states as distinct state in memory (variants with 'R' in them). For Mountain Car, we also consider Mountain Car Repeat with selected actions repeated 10 times, and car velocity binned by 0.01 instead of 0.001 accordingly. The full experimental results are available at Sections []
First Solve Run. Overall, Go-Explore-Coun GDIR performs the best for Cart Pole 50 with first solve run of 3. while Go-Explore GDIR performs the best for Cart Pole 175 with first solve run of 20. They both perform equally well for Cart Pole 100 with a first solve run of Explore- Count GDIR performs the best for Mountain Car with first solve run of 7. These all learn faster and are more adaptable than PPO. Contrasting with the results from the discrete environments, this highlights that the "Go" step is useful if solving the environment linked to long trajectories.
Repeated States/Actions. For Cart Pole 175 and Mountain Car, the non-GDIR methods fail to solve it, while PPO still manages to, indicating that for extended search problems which involve similar actions, a trained neural network might work better to repeat a performant sequence actions. That said, for Cart Pole 175R, Go-Explore and Co-Explore-Coout and their GDIR variants have lower first solve run than PPO. which highlights distinct repeated states could helpful for tasks with repeated states. For Mountain Car Repeat, Go-Explore GDIR has lower first solve than PPO.

---

Table 3: First Solve Run for Continuous State Environments (lower is better, +-' means unsolved).
PPO results are averaged over 10 trials (refer to Tables 36 and 39 for details).
Cart Pole
Mountain Car
Agent
50
50R
100
100R
175
175R
Normal
Repeat
PPO (Discretized State)
3.7
3.7
39.7
39.7
62.3
62.3
43.1
PPO (Continuous State)
6.6
6.6
55.3
55.3
60
60
92.6
12.5
Go-Explore
4
4
22
Go-Explore GDIR
4
4
6
20
21
Go-Explore-Coum
3
2
27
5
26
Go-Explore-Coute Count GDIR
3
2
5
28
40
72
Explore-Count
3
4
56
4
30
10
Explore-Count GDIR
4
4
4
75
Table 4:
Performance of Explore-Count GDIR Walled Maze 100x 100 with different intrinsic
rewards. For random instrinsic rewards in given range [., 0J. results are averaged over 10 trials.
Intrinsic Reward
Attribute First Solve
Manhattan
[1,00]
[-5,0]
[100,0]
[-20,0]
[500,0]
1.7
1.9
1.7
10.5
96.2
Solve Rate
100/100
100/100
99.3/100
99.1/100
99.3/100
90.5/100
4/100
Memory Size
5069
7552
7725.6
7991.7
7629.8
9628.9
9692.5
Min Steps
1424
3160
655
557
762
730.4
561
Improving PPO performance. PPO solves the environments even with continuous state space, which highlights the generalization capabilities a neural network. With discretized binning the continuous states, PPO performs even better, highlighting the usefulness of abstracting state space.
5.3 ABLATION STUDY
What if intrinsic rewards are wrongly calibrated? We seek to test what would happen if we uninformed intrinsic rewards of increasing magnitude in the Walled Maze 100x100 environment. From Table 4 (detailed results in Section F.3), it can be seen that the magnitude of uninformed (random) intrinsic reward increases, the overall solve rate decreases, first solve run increases, and first solve memory size increases, indicating exploration may lead to a lower minimum step, but at the cost of slower adaptation or even failing solve the environment. Overall, can see that GDIR can help solve the environment faster calibrated correctly, and is important for fast adaptability to new environment.
exploration of the environment. This increased
What if we remove hippocampal replay? We evaluate Go-Explore variants on the Walled Maze 100x 100 environment with and without hippocampal replay. From Table hippocampal replay does affect first solve run and memory size, since hippocampal replay only happens after successful first solve. After the first solve, having no hippocampal replay causes the agent to explore, which leads lower solve rate and higher maximum step. While increased exploration may allow the agent to attain a lower minimum step. hippocampal replay can consolidate a good trajectory and repeat it consistently, Overall, hippocampal replay is very important consistent performance and helps agent remember successful trajectories which solve its current environment.
DISCUSSION
Advantage of GDIR. Overall, we can see that GDIR enables the Go-Explore variants to learn signficantly faster. The main advantage is fast adaptation (low first solve run) to new environment, even in continuous state space environments. GDIR works much like a compass to guide initial actions that can increase the chance of solving environment.
8

---

Table 5: Performance results for Walled Maze 100x100 with and without Hippocampal Replay.
represents without hippocampal replay. Bolded text represents better performance.
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Go-Explore-Con
100/100
7552
4912.0
4912.0
4912.0
Go-Ex Go-Explore-Courd Count
100/100
7552
4918.2
4718.0
6362.0
Go- Coo-Exloree-Coure GDIR
100/100
5069
4042.0
4042.0
4042.0
Go- Go-Explore-Coumt GDIR*
100/100
5069
3891.4
3676.0
5326.0
Explore-Couu
100/100
7552
3177.5
3160.0
4912.0
Explore-Count*
32/100
7552
7376.2
3674.0
9954.0
EExlore-Cour GDIR
100/100
5069
1450.2
1424.0
4042.0
Explore-Count GDIR*
34/100
5069
4611.6
1410.0
9818.0
Advantage of Count-based Go-Explore and Hippocampal Replay. For most of the environments, Go-Explore-Count and Explore-Count learns significantly faster than traditional RL algorithms such as TD-Learning and Q-Learning, even without GDIR. With GDIR, it learns significantly faster than PPO in all continuous environments considered. Hippocampal replay also beneficial as it enables consolidation of good trajectories for higher chance of repetition in the future. This makes it suit- able for use in real-life robot, where optimization is a good-to-have (unlike in game- based RL environments such as Chess or Go), and fast adaptation to new situations is key for performance.
Satisficing Solution. Although the Go-Explore variants do not seek explore further upon solving the environment, by nature of the fact that we update our memory with the shortest possible sequence of actions to reach that state, the agent's performance can quite close to optimal.
Robustification. The approach presented in our paper is only the first step of Go-Explore to gather successful trajectories. In order to generalize to stochastic, continuous settings, we can then perform imitation learning trajectories obtained into neural network (Ecoffet et al.) 2019 1
CONCLUSION AND FUTURE WORK
The field of RL has come a long way, and we now seek to focus on sample-efficient methods for real- world adapation for our RL agents, which propose do using Go-Explore variants with GDIR and hippocampal replay. In fact, the proposed GDIR and hippocampal replay methods are not just limited to Go-Explore and can even be used improve other RL architectures. We hope that this paper, inspired from the vision of Yann LeCun (LeCun 2022) and other pioneers of incorporating intrinsic rewards to RL, would help to pave the way for efforts focusing this domain.
Incorporating Neural Networks for Generalization. The proposed count-based approach is most effective for discrete state and action spaces. For continuous state and action spaces, we could utilize neural networks as the exploration policy, similar to policy-based Go-Explore (Ecolfet et al. 20211, while incorporating density models as proxy for count-based models (Bellemare et al, 2016). Neural networks can also help improve our method by providing another means of abstracting similar states together. It can even be used for world model prediction, perhaps using an architecture similar to Decision Transformers (Chen et al. 202T). We intend to explore this in future work.
Configurator Module. Another line of future work will to design configurator-lik module, which can select the relevant GDIR modules to use for given environment. This can be done via an attention-basee mechanism like that a Transformer (Vaswani et al.) (017). which uses the relevant reward functions based similarity of its intended case to the environment.
Multi-Agent Learning. We envision a scenario where rather than just having one agent learning by itself, have multiple agents with different hyperparameters learning from each other. This can help increase the chance solving the environment, and knowledge sharing can take place via hippocampal replay of successful trajectories across all agents, thus expediting the learning process.

---

REFERENCES
Mostafa Al-Emran. Hicrarchical reinforcement learning: a survey. International journal of comput- ing and digital systems, 4(02), 2015.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances neural information processing systems, 30, 2017.
Arthur Aubret, Laetitia Matignon, and Salima Hassas. A survey on intrinsic motivation reinforce- ment learning. arXiv preprint arXin:1908.06977, 2019.
Arthur Aubret, Laetitia Matignon, and Salima Hassas. An information -hheoretcct perspective intrinsic motivation in reinforcement learning: a survey. arXiv preprint arXiv:2209.0855 2022.
Gianluca Baldassarre and Marco Mirolli. Intrinsically motivated learning in natural and artificial systems. Springer, 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information pro- cessing systems, 29, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronek. Monte-carlo tree search: A new framework for game ai. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 4, pp. 216-217, 2008
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning, Advances in neural information processing systems, 17, 2004.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv: 10995, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-588 2021.
RE Fitzgerald, Isler, E Rosenberg, Oettinger, and K Bättig. Maze patrolling by rats with and without food reward. Animal Learning & Behavior, 13(4):451-44 1985.
Quentin Gallouédec and Emmanuel Dellandréa. Cell-free latent go-explore, 2022. URL https:
14928
Marck Grzes. Reward shaping in episodic reinforcement learning. 2017.
Peter E Hart, Nils Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions Systems Science and Cybernetics, 4(2):100-107, 1968.
Robert N Hughes. Intrinsic exploration in animals: motives and measurement. Behavioural Pro- cesses, 41(3):213-233 1997.
Hannah Joo and Loren M Frank. The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation. Nature Reviews Neuroscience, 19(12):444-577 2018.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 132.5600, 2013.
10

---

P Read Montague. Reinforcement learning: an introduction, by sutton, rs and barto, ag. Trends in cognitive sciences, 3(9):360, 1999.
Pierre-Yve Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computa- tional approaches. Frontiers in neurorobotics, pp. 2009.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement
learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1-35, 2021.
Matthias Plappert. Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: /70600/90, 2017.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- mann. Stable baselines3, 2019.
Thomas Rückstieß, Martin Felder, and Jürgen Schmidhuber. State-depender exploration for policy gradient methods. In Joint European Conference on Muchine Learning and Knowledge Discovery in Databases, pp. 234-249. Springer, 2008
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05990 2015.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010), IEEE transactions on autonomous mental development, 2(3):230-247, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panncershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489448444449944 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676))354- 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1144, 2018.
David Silver, Satinder Singh, Doina Precup, and Richard Sutton. Reward is enough. Artificial Intelligence, 299:103593 103535, 2021.
Sungryull Sohn, Sungtae Lee, Jongwook Choi, Harm van Seijen, Mehdi Fatemi, and Honglak Lee. Shortest-patil constrained reinforcement learning for sparse reward tasks. CoRR, abs/2107.06400 2021. PRLLttts::/aant wttp:/wwww..../a////....ooo
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Zhao Yang, Thomas M Moerland, Mike Preuss, and Aske Plaat. When to go, and when to explore: The benefit of post-exploration in intrinsic motivation. arXiv preprint arXiv:2203. 16311, 2022.
11

---

A Q-LEARNING AND TD-LEARNING AGENTS
A.1 Q-LEARNING
Consider Markov Decision Process (MDP) defined by the tuple (S, A, R, P). where represents the set of states, A represents the set of actions, R represents the reward function between transitions of states, given by the function (s;, at, St+1), represents the transition probability of going from one state to another, given by the function P( (z), and is the timestep.
We will perform an online learning the Q-functions, where we only update the state-action values that we visit for efficiency. For every state transition (8t, ax, St+1), we calculate the TD-error:
ó, T2
max ((5+11, Q(ts, at). aEA
Here, in order calculate all future states st+1, will treat the model in-built to the agent and use it to calculate all future values ((5++1,)). Doing so will speed up convergence as we can evaluate all possible futures in update.
Thereafter, we perform the Q-learning update:
Q(s;, Q(s,, at) + ad,,
We e-greedy behavior policy, where take random action fraction of the time, and the greedy action a" maX.ed ((+++1, a) a fraction 1 c of the time. For our agent, the training phase uses to explore the state space, and the testing phase uses 0 to greedily select actions.
A.2 TD-LEARNING AGENT
The TD-Learning agent we consider is simply the online Q-Learning agent applied to state values instead state-action values. This is slightly different from the model-free TD-X approaches (Sutton) & Barto 2018) in that is actually model-based approach where we will need model to compute all possible next states after taking an action from the current state. The advantage doing such an approach is that future possible states at the same time.
be more efficient performing our value updates by comparing against all
Similar to the Q-learning agent, for every state transition (st, at, 8(41), calculate the TD error:
6e +7
max
V(8++1) V(s)).
(4)
oE a,:s 186+1
Thereafter, we perform the value update:
V(s)) + ((s)) + tad
(5)
Similar to Q-Learning, we use an e-greedy behavior policy, where we take a random action a fraction of the time, and the greedy action a" m V(s++1) For our agent, the training phase uses e = to explore the state space, and the testing phase uses 0 to greedily select actions.
fraction 1 - of the time.
For both Q-learning and TD-learning agents, we choose hyperparameters 7 0.99 and a =
12

---

B ALGORITHMS
We detail the algorithms used for our agents in Algorithms and
Algorithm Go-Explore- Variants "Explore" phase
1:
procedure GO-EXPLORE- VAAAIORRR--AIIAASSSEE,., acdion.trajectory. replay, .functionn
2: intrinsic_rewarrd cetrrissc_ffunccionnnnn)
0 no intrinsic function
3: curmem memory seemoryyenv.staer
>> Add env.state to memory if not present Make the intrinsic value successful trajectory end state be
4:
if env.done then
if enw.rwarrd > 0 then curmem] intrinsic"] = env.reward return 7: if not reploy then
5:
6:
Increment state visit count by
8:
awmmemmumovisteed curmem. mumiiisitmeeieeeeed + 9: else if replay then
b If replay, then reset counts to prioritize choosing the state again
curmem-numaissile marmem.numselker = 0
11:
12:
for move in enc env. validmovss do
13:
newenv copy(enc)
Replicate current environment b Simulate move in the replicated environment 0 if no intrinsic function & Add newenc.slale to memory if not present
14:
newenv = newene.stepp(mov))
15:
nerlmem = hatere/eeeeerroiiiiiinnnnnnnnnno Update nerlmem with attributes of newene if there is higher extrinsic reward state compared to memory, or a shorter path exists with same extrinsic reward, and reset numvisited and numselected be total reward = selection function(nertmem))
nert betee_/inrrinsicceeeerraree intrinsic. function(newenv)
16:
18:
19:
highest neat__inrinsiccrenatt across all move y set to 0.99
20:
curmem, intrimsic_reward = bess_-inrrimsccceerrrdd
21:
if GO-EXPLORE then
22:
return random move in env. wal.dddoosss
Random Policy
23:
else if CO-EEXLORFF-OUU or EXPLORE-COUNT then
24:
return move with highest rotal
b Greedy Policy
Algorithm Choosing State for "Go"
1: procedure
2: for eachmemory in memory lotal cealecion_funncionncccceeee
3: bestmem = eachmemory with highest total reward
4: for move in wwtpemmmmaciiooorrreeeoo do env.step(move)
5: mestmemm.uumseeeteee. = bestmem.numme numselected + increment selection count by
6: return bestmem. trajectory,
Algorithm 3 Hippocampal Replay
procedure HiPPOCAMPAL action trajectory, intrinsic function): env actiaction. action_list
2:
3: iiciionhistorr
4:
for move in action_treed ttajectory do
Pre-play
5:
erev aaction dist.append(oopy(ene actionnhistrrr)) envv.sted nou.step(mne (ttep
6:
7:
action_history. h y.append(pmoss
8:
for enev, action. history in reversed(env action_liss do
Replay
9:
Go-Explore- Variants(coppy(env)) hhistoy, replay- True, intrinsic function)
13

---

Forward
Reverse Centrifugal
Forward
Reverse
Centripetal
Centrifugal
Centripetal
III
50ms
ms
Trajectory direction
Place cell spikes
ms
Figure 1: Hippocampal replay in mice, which showcases forward play (pre-play) and reverse play (replay), which are involved in memory retrieval and consolidation for processes such decision- making. Extracted from Fig. of Joo & Frank (2018). There replay occuring for both 1) past visited states and 2) future unvisited states. We only focus replay for past visited states, and use these insights in designing Algorithm 3 for consolidating learnt experiences.
Algorithm 4 Evaluation
1: procedure EVALUATION(eTee, agent, reumruns, mazsleps, intrinsic global memory =
intrinsic.function function): Global variable
2:
3: for range(nuraruns) do
4: env.reset()
Reset to original state of environment
5:
action trajectory =
6:
if agent is Go-Explore agent Go-Explore-Count then action.trajjectory, env ChooseeState(rai))
7:
D "Go" phase
8:
while env.done and eww.numsleps < mazzleps do mave = (egend(ene; Jrdjoonrrajectoor,, reploy = False, intrinsic_fuucctoon) env.step(move)
"EExloree phase
9:
10:
11:
12:
agent(enw, action trajectory, replay False, intrinsic function)
agent is Go-Explore- Variant and reward in memory then
Replay
14:
bestmem. = memory with reward & Choose memory which has solved the task env.reset()
15:
> Reset to original state of environment mntrinicc_fncction
16:
Hiipocanpaleplaa(coyyeee)) www.eeeooooooooooooooioeeeeeeeee
14

---

C SPECIFICATIONS FOR ENVIRONMENTS
Table List of discrete state environments and their specifications
Unwalled Maze
Walled Maze
Towers of Hanoi
Nim
Image
X.R.......
&.........
State Space
Agent position (x and axis)
Agent position (x and axis)
Disk positions the respective
Number of matchsticks maining
poles
Action Space
Up. Down, Left, Right to avail- able open area (.). No wraparound.
Up. Down, Left, Right to avail- able open area (.). No wraparound.
Move top disk from one pole another (must
Remove 1-K matchstick(s)
be smallest destination pole).
Start State
Maze of size Height x
Maze of size Height x Width
poles labelled A, B, C
matchsticks
in a pile
Width
disks
2 player game turn-based
Agent (X) at
Agent (X) at
stacked small- est largest
top left, Door
top left, Door (D) at bottom right
game, agent will start first Opponent plays perfectly, in the event of losing state, agent just picks matchstick
(D) at bottom
from top to
right
bottom on Pole
10% of grid
Bricks (#) form
A
cells generated bricks (#)
chambers
Poles and
plus one final narrow path- way door, which make solving difficult
C are initially
which prohibit
empry
agent move- ment
Goal State
Agent reaches door position
Agent reaches door position
N disks stacked smallest to largest from top to bot- Pole C N
Be the player remove the last matchstick
Max
Height * Width
Height * Width
N
Timestep
Extrinsic
if goal state reached before max timestep, 0 otherwise
Reward
Intrinsic Reward
Manhattan Distance of agent to door agent to door position)/(Heigg W Width 2)
Manhattan Distance of
(Number disks matching goal state -N)NN
Matchsticks remaining after move/N
position)((Heigh)) +Witth - 2)
15

---

Table 7: List of continuous state environments and their specifications
Cart Pole
Mountain Car
Image
State
Cart x-position ze: -4.8 to 4.8
Car x-position -inf to inf
Space
Cart velocity: -in,f to inf
Car velocity in -inf inf
Pole angle po: -0.418 to 0.418 rad
Pole ang velocity po: ff to in
Action
Push cart left or right
Accelerate to the left, don't accelerate, accelerate to the right
Space
Start
All state values at ()
ar at 0.5, v at 0
State
Goal
Not reach early termination for N timesteps
Car at 0.5 x-position
State
Max
200
200
Timestep
Early
Pe > 0.2095 rad or < 0.2095 rad
Nil
Termina-
Tc < -24 2.4
tion
Extrinsic
1 if goal state reached before max timestep early termination, 0 otherwise
Reward
Intrinsic
0.5
0.3|ps 0.2095
0.5
Reward
16

---

D STRATEGY FOR NIM
D.1 OVERVIEW
Nim is game with N matchsticks pile, and move which can range from 1 K matchsticks. It is an alternating two-player turn-based game. The player who removes the last matchstick wins.
D.2 WORKING BACKWARDS FROM WIN STATE
Given that the win state is to remove the last matchstick, we then see that if one receives a state of < K matchsticks at the start of one's turn,
win by simply removing all the matchsticks. Hence, the losing state will then be receiving a state of K + matchsticks, because for any possible move one makes, one would necessarily let the opponent have a state of < K matchsticks and hence lose. Hence, the goal is to then present the opponent with a state of K + matchsticks at some point of time in the game in order win.
D.3 A STRATEGY FOR ALL OPPONENT RESPONSES
Regardless of how the opponent responds (choosing move between 1 K matchsticks), it is possible to maintain a sum K + between both turns the opponent and oneself. This is easy to verify, because if the opponent chooses one respond with K + i. This will always be valid move since < S and hence 1 S 1<+1-i < K. Hence, knowing that presenting a state of K + matchsticks to the opponent is win state, and that having interval of K + between the opponent's move and one's move is always possible, one would hence want to present state of K 1, 2(K 1). 3(K + 1). ..... p(K 1) the opponent, where p E N.V.
Winning State. In short, since the winning strategy is always to present state which is a multiple of (K + 1) to your opponent, we can then determine the winning play any point of time with certainty. We simply need to take a move which presents state of a multiple of K + 1 to the opponent. short, we need to take %%(K (K +1) matchsticks, where is the number of matchsticks you are presented with originally, and % is the modulo operator.
Losing State. However, this number of X%(K 1) may be O, which not valid move. If that is the case, the player who is presented with this state will always lose if the opponent plays perfectly. In this case, our perfect play agent will always take matchstick (though is also possible to randomly select between all possible moves).
D.4 THE WINNING ALGORITIM
Having seen the logic behind the perfect play agent, we detail the algorithm needed to play Nim perfectly in Algorithm beloo. Note that X the number of matchsticks presented the agent.
Algorithm 5 Perfect Play Agent for Nim
1: procedure NIMPERFECTPLAYCX K):
2: if X%(K + 1) > 0 then return X%(K- +))
b Give opponent the losing state This is a losing state, so return any valid move
3: else return 1
17

---

E FULL RESULTS FOR UNWALLED MAZE
E.1
WITH HIPPOCAMPAL REPLAY
Tables
9 and 10 detail the results for unwalled maze with hippocampal replay. A +-' represents
unsolved after 100 runs, and bolded text means best result across all agents. TD-Learning, Go-
Explore-Couud Count and Explore-Cout Count and their GDIR variants perform the best.
Table 8: Performance results for Unwalled Maze 10x10
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
8/100
8
76.8
54.0
100.0
TD-Learnin (Train)
8/100
21
90
72.5
44.0
100.0
TD-Learning (Test)
100/100
1
90
18.0
18.0
18.0
Q-Learnig (Train)
8/100
21
292
72.5
44.0
100.0
Q-Learning (Test)
100/100
294
18.0
18.0
18.0
Go-Explore
0/100
Go-Explore GDIR
38/100
12
59
83.3
62.0
100.0
Go- Go-Explore-Count
100/100
1
62.0
62.0
62.0
Go-Explore-Counl GDIR
100/100
1
39
20.0
20.0
20.0
Explore-Cove Count
100/100
1
22.4
22.0
62.0
Caplore-Counl GDIR
100/100
1
39
20.0
20.0
20.0
Table 9: Performance results for Unwalled Maze 20x20
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Learning (Train)
3/100
261
336.7
264.0
400.0
TD-Learning (Test)
100/100
359
38.0
38.0
38.0
Q-Learning (Train)
3/100
769
336.7
264.0
400.0
Q- Q-Learning (Test)
0/100
Go-Explore
0/T00
Go- Go-Explore GDIR
0/100
Go- GooEExplree-Cour
99/100
359
356.0
356.0
356.0
Go-Explore-Coum GDIR
100/100
85
42.0
42.0
42.0
Explore-Count
99/100
359
78.3
76.0
304.0
Explore-Coun GDIR
100/100
85
42.0
42.0
42.0
Table 10: Performance results for Unwalled Maze 100x100
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
I/100
15
9980.0
9980.0
9980.0
TD-Learning (Train)
2/100
15
8552
7606.0
5544.0
9668.0
TD-Learning (Test)
100/100
1
8997
198.0
198.0
198.0
Q-Leaming (Train)
2/100
15
29289
7606.0
5544.0
9668.0
L-Larniing (Test)
0/100
Go-Explore
0/100
Go-Explore GDIR
0/100
Gio-Explore-Count
100/100
1
8045
7810.0
7810.0
7810.0
Go-Explore-Count GDIR
100/100
1
499
230.0
230.0
230.0
Explore-Coue
100/100
1
8045
555.3
482.0
7810.0
Explore-Cae Count GDIR
100/100
1
499
222.1
222.0
230.0
18

---

E.2 WITHOUT HIPPOCAMPAL REPLAY
Tables I1 and 13 detail the results for various sizes of unwalled maze without hippocampal re- play. represents unsolved after 100 runs, and bolded text means best result across all agents. can be seen that hippocampal replay does not affect first solve run and memory size, since hippocam- pal replay only happens after a successful first solve. After the first solve, having no hippocampal replay causes the agent to explore instead of consolidating good experiences, which generally Ieads lower solve rate and higher maximum solve time step. Increased exploration may be good thing in the long run, as the agent generally finds shorter solution than the corresponding agents with hippocampal replay. Here, some the agents without hippocampal replay even manage find the shortest path (18, 38, 198 for 10x10, 20x20 and 100x 100 maze sizes respectively).
Table 11: Performance results for Unwalled Maze 10x10 (Without Hippocampal Replay)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Go-Explore
0/100
Go-Explore GDIR
5/100
12
59
82.8
64.0
100.0
Go-Explore-Count
82/100
1
50.9
18.0
100.0
Go-Exp Noo-Explore-Coure Count GDIR
78/100
1
39
42.0
18.0
100.0
Explore-Count
98/100
1
71
37.2
18.0
90.0
Explore-Count GDIR
92/100
1
39
38.2
18.0
98.0
Table 12: Performance results for Unwalled Maze 20x20 (Without Hippocampal Replay)
Overall
First Solve
Steps to Solve
Agent Go-Explore
Solve Rate
Run
Memory size
Avg
Min
Max
0/100
Go-Explore GDIR
0/100
Go-Explore-Count
71/100
2
359
200.9
38.0
396.0
Go-Explore- GDIR
50/100
1
85
225.5
40.0
370.0
Explore-Count
79/100
2
359
178.1
38.0
400.0
Explore-Cor Count GDIR
79/100
1
85
204.7
38.0
400.0
Table 13: Performance results for Unwalled Maze 100x100 (Without Hippocampal Replay)
Overall
First Solve
Steps to Solve
Agent Go-Explore
Solve Rate
Run
Memory size
Avg
Min
Max
0/100
Go-Explore GDIR
0/100
Go-Explore-Count
94/100
1
8045
6367.6
5576.0
9980.0
Go-Explore-Count GDIR
32/100
1
499
7262.6
230.0
9976.0
Explore-Count
100/100
1
8045
2061.6
322.0
8028.0
Iaplore-Count GDIR
81/100
1
499
3640.2
198.0
9732.0
19

---

F FULL RESULTS FOR WALLED MAZE
F.1 WITH HIPPOCAMPAL REPLAY
and T6 detail the results for various sizes of walled maze with hippocampal replay. A represents unsolved after 100 runs, and bolded text means best result across all agents. Go- Explore- Count and Explore-Cout Count and their GDIR variants perform the best.
Tables
Table 14: Performance results for Walled Maze 10x10
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Learnin (Train)
0/100
TD-Learnnng (Test)
0/100
Q-Learnig (Train)
0/100
Q-Learning (Test)
0/100
Go-Explore
0/100
Go-Explore GDIR
4/100
75
77
94.0
92.0
100.0
Go- Go-Explore-Count
100/100
74
64.0
64.0
64.0
Go-Explore-Counl GDIR
100/100
1
62
60.0
60.0
60.0
Explore-Cove Count
100/100
74
42.2
42.0
64.0
Caplore-Count GDIR
100/100
1
62
34.3
34.0
60.0
Table 15: Performance results for Walled Maze 20x20
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Leaming (Train)
0/100
TD- (Test)
0/100
Q-Learnigg (Train)
0/100
Q- Q-Learning (Test)
0/100
Go-Explore
0/100
Go- Go-Explore GDIR
0/100
Go- HioEExplree-Cour
100/100
312
218.0
218.0
218.0
Go-Explore-Coum GDIR
100/100
223
220.0
220.0
220.0
Explore-Count
100/100
312
138.8
138.0
218.0
Explore-Count GDIR
100/100
223
83.4
82.0
220.0
Table 16: Performance results for Walled Maze 100x100
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/T00
TD-Learning (Train)
0/100
TD-Learning (Test)
0/100
Q-Learnig (Train)
0/100
L-Larniing (Test)
0/100
Go-Explore
0/100
Go-Explore GDIR
0/100
GioFExplore-Count
100/100
1
7552
4912.0
4912.0
4912.0
Go-Explore-Coum GDIR
100/100
1
5069
4042.0
4042.0
4042.0
Explore-Coum Count
100/100
I
7552
3177.5
3160.0
4912.0
Explore-Couu Count GDIR
100/100
1
5069
1450.2
1424.0
4042.0
20

---

F.2 WITHOUT HIPPOCAMPAL REPLAY
Tables I1 [3 detail the results for various sizes of walled maze without hippocampal replay in the Go-Explore variants. A '-' represents unsolved after 100 runs, and bolded text means best result across all agents. It can be seen that hippocampal replay does not affect first solve run and memory size, since hippocampal replay only happens after successful first solve. After the first solve, having no hippocampal replay causes the agent to explore instead of consolidating good ex- periences, which generally leads to lower solve rate and higher maximum solve time step. Increased exploration may be good thing in the long run, as the agent generally finds shorter solution than corresponding agents with hippocampal replay.
Table 17: Performance results for Walled Maze 10x10 (Without Hippocampal Replay)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Go-Explore
0/100
Go-Explore GDIR
2/100
75
77
96.0
92.0
100.0
Go-Explore-Coumt
82/100
1
74
61.6
32.0
100.0
Go-Explore-Coe GDIR
75/100
1
62
55.7
32.0
100.0
Caplore-Count
75/100
1
74
61.6
34.0
100.0
Explore-Count GDIR
73/100
1
62
63.9
34.0
100.0
Table 18: Performance results for Walled Maze 20x20 (Without Hippocampal Replay)
Overall
First Solve
Steps to Solve
Agent Go-Explore
Solve Rate
Run
Memory size
Avg
Min
Max
0/100
Go-Explore GDIR
0/100
Go-Explore-Cout Count
87/100
1
312
233.1
68.0
400.0
Go-Explore-Cous GDIR
77/100
1
223
221.6
68.0
394.0
Explore-Coume
74/100
1
312
290.2
98.0
396.0
Explore-Count GDIR
48/100
1
223
299.5
74.0
396.0
Table 19: Performance results for Walled Maze 100x100 (Without Hippocampal Replay)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Go-Explore
0/100
Go-Explore GDIR
0/100
Gio-ExplrreCoom
100/100
1
7552
4918.2
4718.0
6362.0
Go-Explore-Coum GDIR
100/100
1
5069
3891.4
3676.0
5326.0
Explore-Count
32/100
1
7552
7376.2
3674.0
9954.0
Explore-Count GDIR
34/100
1
5069
4611.6
1410.0
9818.0
21

---

F.3 WITH RANDOM INTRINSIC REWARDS
Tables 20 21 22 23 and 24 detail the performance results of Explore-Count GDIR across 10 trials, with different ranges of random intrinsic rewards. A '-' represents unsolved after 100 runs. It can be seen that as the magnitude of uninformed (random) intrinsic reward increases, the overall solve rate decreases, first solve run increases, first solve memory size increases indicating more explo- ration of the environment. Explore- Count GDIR with random intrinsic rewards generally achieve lower minimum solve time step compared the actual Manhattan intrinsic reward for Walled Maze 100x (see Table T6), due to their increased exploration leading to higher chance of finding shorter path. This, however, comes the cost of slower adaptation and sometimes even failing to solve the environment.
Table 20: Performance results for Explore-Coun GDIR on Walled Maze 100x100 across 10 trials (Random Intrinsic Reward between 1,)
Overall
First Solve
Steps to Solve
Trial
Solve Rate
Run
Memory size
Avg
Min
Max
100/100
1
6771
497.9
414.0
8802.0
2
100/100
1
6541
537.1
472.0
6986.0
3
99/100
2
9236
849.1
798.0
5860.0
4
98/100
3
8871
702.3
640.0
6748.0
5
100/100
1
8457
598.0
504.0
9908.0
6 100/100
1
7001
558.3
474.0
8908.0
7
99/100
2
7592
1063.8
1004.0
6928.0
8
98/100
3
9607
1130.9
1082.0
5874.0
9
99/100
2
8132
706.3
652.0
6030.0
10
100/100
5048
560.1
510.0
5516.0
Avg
99.3/100
1.7
7725.6
720.4
655.0
7156.0
Table 21: Performance results for Explore-Coun Count GDIR Walled Maze 100x100 across 10 trials (Random Intrinsic Reward between |-3, 0)
Overall
First Solve
Steps to Solve
Trial
Solve Rate
Run
Memory size
Avg
Min
Max
99/100
8643
881.8
838.0
5176.0
2
97/100
9653
587.9
492.0
9794.0
100/100
6778
531.9
460.0
7652.0
4
100/100
6603
527.8
466.0
6646.0
99/100 6
9178
591.6
504.0
9180.0
100/100
5428
487.8
426.0
6602.0
99/100 8
9181
753.0
662.0
9674.0
100/100
6865
503.0
434.0
7336.0
9
98/100
8970
865.6
790.0
8198.0
10
99/100
8618
588.8
500.0
9288.0
Avg
99.1/100
1.9
7991.7
631.9
557.2
7954.6
22

---

Table 22: Performance results for Explore-Count GDIR on Walled Maze 100x across 10 trials
(Random Intrinsic Reward between 10,10,00 0))
Overall
First Solve
Steps to Solve
Trial
Solve Rate
Run
Memory size
Avg
Min
Max
100/100
1
5353
551.8
506.0
5082.0
99/100
2
8042
537.3
468.0
7328.0
3 99/100
2
7976
1022.1
990.0
4164.0
4
98/100
3
9562
1513.7
1486.0
4198.0
5
100/100
1
7368
573.0
494.0
8394.0
6
98/100
3
8668
1758.1
1754.0
2152.0
7
100/100
1
7610
499.7
416.0
8784.0
8
100/100
1
6220
535.6
478.0
6238.0
9
100/100
1
6889
511.3
416.0
9942.0
10
99/100
2
8610
638.9
612.0
3274.0
Avg
99.3/100
1.7
7629.8
814.2
762.0
5955.6
Table 23: Performance results for Explore-Count GDIR Walled Maze 100x100 across 10 trials
(Random Intrinsic Reward between [-20, 0))
Overall
First Solve
Steps to Solve
Trial
Solve Rate
Run
Memory size
Avg
Min
Max
85/100
16
9707
1293.3
1230.0
5034.0
2
87/100
14
9687
774.1
660.0
9046.0
3
95/100
6
9403
707.7
634.0
7632.0
4
95/100
6
9378
744.8
662.0
8506.0
5
96/100
5
9668
528.7
454.0
7624.0
6
91/100
10
9706
749.5
650.0
9702.0
7
89/100
12
9707
672.3
574.0
9320.0
8
96/100
5
9621
781.6
712.0
7392.0
9
81/100
20
9705
1244.0
1158.0
8124.0
10
90/100
11
9707
667.1
570.0
9308.0
Avg
90.5/100
10.5
9628.9
816.3
730.4
8168.8
Table 24: Performance results for Explore-Count GDIR on Walled Maze 100x100 across 10 trials (Random Intrinsic Reward between [-50, 0). s-' means unsolved). If unsolved after 100 runs, will count it as the first solve occuring at the 100th run for the average score.
Overall
First Solve
Steps to Solve
Trial
Solve Rate
Run
Memory size
Avg
Min
Max
0/100
2
0/100
0/100
4 14/100
87
9707
1082.0
532.0
7972.0
5
0/100
6 0/100
7
0/100
8
0/100
9
26/100
75
9678
941.8
590.0
9418.0
10
0/100
Avg
4/100
96.2
9692.5
1011.9
561
8695
23

---

G FULL RESULTS FOR TOWERS OF HANOI
Tables 25 and 26 detail the results for Towers of Hanoi with and disks respectively. A
represents unsolved after 100 runs, and bolded text means best result across all agents. From the results, we can see that Explore-Count and Explore-Count GDIR perform the best and are the only agents which can solve the task with 7 disks.
Table 25: Performance results for Towers of Hanoi with 3 disks
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
10/100
25.4
17.0
32.0
TD-Learning (Train)
13/100
21
24.9
13.0
32.0
TD-Learnin (Test)
100/100
27
7.0
7.0
7.0
Q-Learning (Train)
13/100
4
46
24.9
13.0
32.0
Q-Learnigg (Test)
100/100
78
7.0
7.0
7.0
Go-Explore
41/100
33
27
20.2
13.0
31.0
Go-Explore GDIR
41/100
27
20.2
13.0
31.0
Exxxloxe--Countt
100/100
15
9.0
9.0
9.0
Go-Explore-Count GDIR
100/100
15
9.0
9.0
9.0
Explore-Count
100/100
15
9.0
9.0
9.0
Explore-Count GDIR
100/100
15
9.0
9.0
9.0
Table 26: Performance results for Towers of Hanoi with 7 disks
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/T00
TD-Leaming (Train)
0/100
TD-Learning (Test)
0/100
Q-Learning (Train)
0/100
L-Lerrning (Test)
0/100
Go-Explore
0/100
Go-Explore GDIR
0/100
Go-Explore-Coum
0/100
Goo-Exlorre-ooure GDIR
0/100
Explore-Count
84/100
1720
378.6
378.0
426.0
Explore-Count GDIR
91/100
10
1095
327.1
327.0
335.0
24

---

H FULL RESULTS FOR NIM
Tables 27 and detail the results for Nim with 11, 21 and 1001 matchsticks respectively. A '_' represents unsolved after 100 runs, and bolded text means best result across all agents. From the results, we see that Go-Explore-Count and Explore-Count, as well as their GDIR variants perform the best and are able solve the complex Nim environment with 1001 matchsticks.
Table 27: Performance results for Nim with 11 matches
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
4/T00
71
3.0
3.0
3.0
TD-Learning (Train)
5/100
27
3.0
3,0
3.0
TD-Learning (Test)
100/100
3.0
3.0
3.0
O-Learning (Train)
5/100
27
17
3.0
3,0
3,0
Q-Learning (Test)
100/100
3.0
3.0
3.0
Go-Explore
15/100
3.0
3,0
3,0
Go-Explore GDIR
16/100
3.0
3.0
3.0
Ce--Explore-Coum
99/100
2
3.0
3.0
3.0
Go-Explore-Count GDIR
100/100
3.0
3.0
3.0
Explore-Count
99/100
2
3.0
3.0
3.0
Explore-Count GDIR
100/100
3.0
3.0
3.0
Table 28: Performance results for Nim with 21 matches
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/T00
TD-Learning (Train)
0/100
TD-Learning (Test)
0/100
O-Learing (Train)
0/100
Q O-Larninng (Test)
0/100
Go-Explore
16/100
6.0
6.0
6.0
Go-Explore GDIR
16/100
5
12
6.0
6.0
6.0
Go-Explore-Coum
99/100
6.0
6.0
6.0
Go-Explore-Cone GDIR
99/100
2
12
6.0
6.0
6.0
Explore-Coue Count
98/100
6.0
6.0
6.0
Explore-Count GDIR
99/100
2
12
6.0
6.0
6.0
Table 29: Performance results for Nim with 1001 matches
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Learning (Train)
0/100
TD-Learning (Test)
0/100
Q-Learning (Train)
0/100
L-Larinng (Test)
0/100
Go-Explore
0/100
Go-Explore GDIR
0/100
Go-Explore-Count
99/100
2
502
251.0
251.0
251.0
Go-Explore-Conn GDIR
99/100
502
251.0
251.0
251.0
Explore-Count
98/100
3
502
251.0
251.0
251.0
Explore-Coun GDIR
99/100
502
251.0
251.0
251.0
25

---

I FULL RESULTS FOR CART POLE
Tables 30 32 and 34 detail the results for Cart Pole for 50, 100 and 175 goal timesteps respectively.
Tables 31 33 and detail the results for Cart Pole with distinct repeated states for 50, 100 and 175 goal timesteps respectively. Table [6 detail the results for PPO for various goal timesteps of Cart Pole with discretized (same representation as the other agents) and continuous state space imputs. A represents unsolved after 100 rums, and bolded text means best result across all agents.
Overall. From the results, we can see that as the Cart Pole goal timesteps to balance increases, the number of algorithms which can solve it decreases. Overall, the first solve run is increased as the difficulty increases, which intuitive as harder to find successful trajectory as the trajectory becomes longer. GDIR variants tend to solve the environment faster in general, except for Explore- Count which tends to make worse, which we suspect is due to GDIR causing Explore-Cove to repeat the same states again where the pole is balanced, hence leading to a high visit count. This leads it to avoid these states in the future, which make take longer to solve.
Repeated States. From the results, creased solve rate for the Go-Explors variants, and lower first solve run. While helped to boost performance in Cart Pole, having repeated states will lead to increasing the search space, which can lead to larger first solve memory size and may not be viable for large state spaces.
that having repeated states lead to generally in-
PPO. Overall, PPO performs well even in the sparse reward modification environment of Cart Pole. Similar to the other algorithms, PPO also has a higher first solve run the higher the goal timesteps balance for Cart Pole. Moreover, having discretized state representation helps PPO get lower first solve run, which highlights that an appropriately discretized state space can enable the neural network to learn faster.
Table 30: Performance results for Cart Pole with 50 timesteps (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
2/100
57
53.0
51.0
55.0
TD-Learning (Train)
2/100
173
54.5
52.0
57.0
TD-Learning (Test)
0/100
Q Q-Learning (Train)
2/100
272
54.5
52.0
57.0
Q LeLamingg (Test)
0/100
Go Go-Explore
97/100
91
51.0
51.0
51.0
Go-Explore GDIR
97/100
110
84.0
84.0
84.0
Go-Explore-Count
98/100
3
59
61.0
61.0
61.0
Go-Explore-Coum GDIR
98/100
54
53.0
53.0
53.0
Explore-Count
82/100
78
57.1
52.0
61.0
Explore-Count GDIR
77/100
89
62.2
51.0
63.0
26

---

Table 31: Performance results for Cart Pole with 50 timesteps and repeated states (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
2/100
57
53.0
51.0
55.0
TD-LLamiin (Train)
2/100
196
54.5
52.0
57.0
TD-Learning (Test)
0/100
Q-Learning (Train)
2/100
306
54.5
52.0
57.0
Q O-Learning (Test)
0/100
Go-Explore
97/100
98
51.0
51.0
51.0
Go-Ex Go-Explore GDIR
97/100
4
127
92.2
75.0
93.0
Go-Explore-Cove Count
99/100
86
76.0
76.0
76.0
Go--Epllre-Cooum GDIR
99/100
86
129.3
76.0
130.0
Explore-Count
49/100
204
159.0
159.0
1.9.00
Explore-Count GDIR
53/100
204
102.1
96.0
159.0
Table 32: Performance results for Cart Pole with 100 timesteps (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Learning (Train)
0/100
TD- TD-Learning (Test)
0/100
Q Q-Learning (Train)
0/100
L-Learning (Test)
0/100
Go-Explore
79/100
22
199
113.0
113.0
113.0
Go-Explore GDIR
95/100
204
114.0
114.0
114.0
Go-Explore-Count
74/100
27
185
101.0
101.0
101.0
Go-Explore-Coum GDIR
95/100
111
104.0
104.0
104.0
Explore- Count
2/100
56
725
109.5
108.0
111.0
Explore-Couu Coonte GDIR
1/100
69
897
107.0
107.0
107.0
Table 33: Performance results for Cart Pole with 100 timesteps and repeated states (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/T00
TD-Learning (Train)
0/100
TD-Learning (Test)
0/100
O-Learnigg (Train)
0/100
Leaning (Test)
0/100
Go-Explore
79/100
22
206
106.0
106.0
106.0
Go-Explore GDIR
95/100
257
178.4
110.0
188.0
Go- Coo-Explrre-Coum
96/100
5
140
103.0
103.0
103.0
Go-Explore-Coumt GDIR
96/100
155
113.0
113.0
113.0
Explore-Count
49/100
4
204
159.0
159.0
159.0
Explore-Count GDIR
58/100
204
141.4
130.0
159.0
27

---

Table 34: Performance results for Cart Pole with 175 timesteps (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Learning (Train)
0/100
TD- TD-Learning (Test)
0/100
Q-Learning (Train)
0/100
Q-Learning (Test)
0/100
Go-Explore
0/T00
Go-Explore GDIR
81/100
20
526
187.0
187.0
187.0
Go-Explore-Count
0/100
Go-Explore-Count GDIR
73/100
28
446
177.0
177.0
177.0
Explore-Count
0/100
Explore- Count GDIR
0/100
Table 35: Performance results for Cart Pole with 175 timesteps and repeated states (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Leaming (Train)
0/100
TD-Learning (Test)
0/100
Q-Learning (Train)
0/100
Q L-Larniing (Test)
0/100
Go-Explore
50/100
51
360
178.1
178.0
180.0
Go-Explore GDIR
76/100
21
620
198.5
178.0
200.0
Go-Explore-Cone Count
75/100
26
412
194.0
194.0
194.0
Go-Explore-Coum GDIR
54/100
40
545
180.3
176.0
190.0
Coplore-Countt
36/100
30
471
200.0
200.0
200.0
Explore-Count GDIR
7/100
75
1089
200.0
200.0
200.0
Table 36: First Solve results across 10 sets of 100 for PPO Cart Pole. If unsolved after 100 runs, we will count it as the first solve occuring at the 100th run for the average score.
Set
Environment
4
6
7
8
10
Average
50 (Discrete)
5
3
3
3.7
100 (Discrete)
99
16
7
34
94
42
39.7
175 (Discrete)
63
28
62.3
50 (Continuous)
9
11
4
4
9
8
5
7
6.6
100 (Continuous)
12
5
16
6
55.3
175 (Continuous)
23
8
14
47
8
60.0
28

---

J FULL RESULTS FOR MOUNTAIN CAR
Tables 37 and 38 detail the results for Mountain Car with normal actions and with 10 repeated actions respectively. Table ( detail the results for PPO for the normal and repeated actions version of Mountain Car with discretized (same representation as the other agents) and continuous state space inputs. +-' represents unsolved after 100 runs, and bolded text means best result across all agents.
Overall. From the results, we can see that Explore-Coumt GDIR is the only algorithm which solves Mountain Car, indicating that it is indeed difficult problem. If we were to relax the constraints and allow for 10 repeated actions (to reduce the search space), we can see that all Go-Explore variants with GDIR can solve the environment, with Go- Explore GDIR performing the best.
PPO. Overall, PPO performs well in the sparse reward modification environment of Mountain Car. Having discretized state representation helps PPO get lower first solve run, which highlights that an appropriately discretized state space can enable the neural network to learn faster.
Table 37: Performance results for Mountain Car (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
0/100
TD-Learning (Train)
0/100
TD TD-Learning (Test)
0/100
Q-Learning (Train)
0/100
LeLenrning (Test)
0/100
Go-Explore
0/T00
Go- Go-Explore GDIR
0/100
Go- Gio-Explore-Cour
0/100
Go-Explore-Cout GDIR
0/100
Explore-Count
0/100
Explore-Cove Count GDIR
7/100
862
165.4
156.0
187.0
Table 38: Performance results for Mountain Car with 10 repeated actions (binary reward)
Overall
First Solve
Steps to Solve
Agent
Solve Rate
Run
Memory size
Avg
Min
Max
Random
6/100
47
181.7
160.0
200.0
TD-Learning (Train)
5/100
17
231
166.0
120.0
200.0
TD-Learning (Test)
0/100
Q Q-Leaningg (Train)
5/100
17
324
166.0
120.0
200.0
O-Learning (Test)
0/100
Go-Explore
63/100
38
89
170.0
170.0
170.0
Go-Explore GDIR
99/100
41
170.0
170.0
170.0
Go-EExplre-Count
0/100
Go-Explore-Cone GDIR
29/100
158
190.0
190.0
190.0
Explore-Coun Count
91/100
10
165
180.0
180.0
180.0
Explore-Coud GDIR
98/100
93
170.2
170.0
180.0
29

---

Table First Solve results across 10 sets of 100 runs for PPO on Mountain Car. If unsolved after 100 runs, we will count it as the first solve occuring at the 100 run for the average score.
Set
State
1
3
4
5
8
9
10
Average
Discrete (Normal)
3
43.1
Continuous (Normal)
69
57
92.6
Discrete (Repeated)
2
6
2
5
3
1
3.0
Continuous (Repeated)
15
12
17
12
10
12
12
11
11
13
12.5
30


Reviews:
[{"review_id": "wW9WjJYGf53", "reviewer": "Reviewer_eHGy", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a potential-based goal-directed intrinsic reward to improve exploration and name it GDIR. By augmenting Go-Explore with GDIR, they improve its performance on a few discrete and continuous exploration methods. They also use a hippocampal replay to directly imitate successful trajectories.", "summary_of_the_review": "This paper requires more work to warrant acceptance at the ICLR conference. In particular, the approach needs to be better positioned in the broader literature. The claims need to be toned down or better supported by the experiments. The approach could also use more work to alleviate clear limitations in more challenging environments or realistic settings, or at least these limitations should be openly discussed and analyzed. Finally, I am not sure the current experiments contain fair comparisons and could greatly benefit from more baselines, ablations, and environments. ", "strength_and_weaknesses": "# Strengths:\n- aims to solve an important problem, namely exploration in RL\n- the method seems simple and intuitive\n\n# Weaknesses:\n### Claims are not well-supported\n- In the introduction, you claim that your approach \"learns faster and adapts better to novel environments in real-world system\". However, most of these aren't well supported by your experiments. When you say it adapts better to novel environments, I imagine that you train it on some environments and finetune it on new ones, which you don't seem to do. Finally, you definitely don't show any results on real-world systems. In fact, it seems like your method only works in deterministic domains where you can reset the environment in any state, which isn't the case in the real-world. I suggest modifying the claim to be better supported by experiments. \n\n- In the introduction, you claim that GDIR improves over Go-Explore. However, your experiments seem to suggest that it is merely better at first finding a successful trajectory rather than consistently maximizing the extrinsic reward and finding the optimal policy. Again, this claim needs to be more precise to be accurate. \n\n- In the abstract, you discuss that Go-Explore is helpful in sparse-reward domains. However, all the environments you consider seem to have dense rewards. I recommend evaluating GDIR on environments such as Montezuma's Revenge, Pitfall, MiniHack, NetHack etc or at least demonstrating the sparsity of the reward in your settings. \n\n### Approach is not well-motivated\n- I don't understand the motivation for finding a satisficing solution rather than an optimal one. If your agent gets lucky once and finds a good path that achieves high extrinsic reward, there is no guarantee that it will robustly be able to solve the task (particularly in stochastic environments, see below) or that it is anywhere close to the optimal policy which may require much more exploration and exploitation to collect additional reward. \n\n- The idea that you want an intrinsic reward that always encourages exploration in a setting where the goal is to maximize extrinsic reward is quite odd. If your intrinsic reward doesn't diminish throughout training, you will end up with a suboptimal policy wrt the extrinsic reward. Hence, the motivation for your approach doesn't make much sense in this setting. It might make more sense to apply GDIR in reward-free settings or open-ended learning with non-stationary extrinsic reward that requires constant exploration. \n\n- The hippocampal replay trick is also quite limited given that it probably doesn't work well in stochastic environments. Due to this, it seems like the approach is limited to a fairly narrow class of problems which are not very challenging. In addition, it seems to require resetting the environment in any state which is a very strong assumption that doesn't apply in the real-world. I would like to see the approach evaluated in a wider range of domains, including stochastic ones. \n\n- The introduction of this paper reads as if this is the first work to propose a type of intrinsic reward, cites LeCun 2022 for introducing ideas that have been well-known way before 2022 and have been introduced by others, cites neuroscience studies that are only remotely related to the current paper, instead of much more closely related works on intrinsic motivation and exploration in RL. Hence, it ignores a huge body of related work. \n\n- GDIR isn't discussed in comparison with closely related works (other than Go-Explore) so it is unclear how it compares with HER, AMIGo, or other goal-conditioned exploration methods. What problem with existing methods is GDIR solving? This should be made more clear in the introduction. \n\n- In general, there should be more discussion about the limitations of your approach, together with experiments on settings where you expect it to work best / worst.\n\n### Empirical evaluation is limited \n- GDIR is only compared with vanilla RL and a few ablations based on Go-Explore. The evaluations are not thorough and extensive enough to draw strong conclusions. For example, I would like to see a comparison with HER and other popular goal-based or potential-based methods. What about using reward shaping of the extrinsic reward?\n\n- The authors include a model-based baseline but I'm not sure what this is meant to achieve since it's completely different from the model-free ones. \n\n- It seems like GDIR makes use of inductive biases in its cost function e.g. Manhattan distance etc. This results in unfair comparisons. What happens if you also augment the other baselines with reward shaped based on such distances or include these representations in their states. If the agent knows where it is with respect to the goal, maybe it can more quickly find it even if it doesn't explore via GDIR explicitly. \n\n- Important ablations seem to be missing. What happens if just apply goal-conditioned exploration without reward shaping or just reward shaping to the external reward without goal-conditioning?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bE72zYeCP_e", "reviewer": "Reviewer_5gPZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a method to speed up learning through the use of a reward function guiding the agent to complete subtasks. Additionally, they propose a replay method, which they call \"hippocampal replay.\" The authors include an empirical demonstration of their method in both continuous and discrete environments, as well as an ablation study to explore the contributions of both the choice of reward function and the use of hippocampal replay.", "summary_of_the_review": "As a proviso, one issue I had with this paper is clarity. While I tried hard to understand the paper and am reasonably well-versed with most intrinsic reward methods, I really struggled to understand what the authors were trying to communicate. \n\nThe primary reason I am recommending rejection for this paper is that, while the central contribution is supposed to be an intrinsic reward method, the authors don't actually use intrinsic rewards. They hand-design different reward functions as the \"GDIR\" for each test domain (Section 4.3), which is a key indicator that the rewards are not intrinsic. This means that their experiments comparing their methods with baselines don't provide any significant information; the authors compare their method, which is provided with an appropriate distance function as domain knowledge against methods that do not use domain knowledge. The paper states that the method isn't provided with domain knowledge (p. 4), but to the best of my understanding, such a statement isn't true.", "strength_and_weaknesses": "The method described in the paper may have some interesting properties that should be further explored, but, unfortunately, I don't think the method was presented effectively. Most importantly, the choice of experiments is inappropriate. Providing the agent with a domain-specific distance function reward is providing it with domain knowledge, and so the comparisons against methods that are not provided this expert knowledge are not very meaningful. \n\nHowever, I understand the position of having an interesting idea that still has some gaps in it, and see the merit in sharing such an idea with the community, even without being able to show off a complete system. In such a case, rather than trying to compare performance, I'd recommend experiments that help build intuition, perhaps using toy domains, and being extremely clear whenever the system designers have had to fill in a gap with a bit of a hack, like providing appropriate subtasks for the agent if it was beyond the scope of the paper to find a method that could do so autonomously. I thought the ablation study was a good idea (strength) and would fit really well into a framework of experiments aimed at demonstrating the potential for the proposed method.\n\nAnother weakness of the paper is in how the context of existing work is presented. Prior work is often listed or briefly summarized without any explanation of how it is related to the paper and the proposed methods (e.g. Chentanez et al. (2004), Schmidhuber (2010), Baldassarre & Mirolli (2013), Aubret et al. (2019; 2022), Yang et al. (2022), and Gallouédec & Dellandréa (2022); pp. 2-3). There are also several misleading statements about prior work in the authors' descriptions of prior work, like the description of first-return-then-explore as an extension of Go-Explore (\"First return, then explore\" is a paper presenting the Go-Explore family of algorithms, not an algorithm) and the statement that Oudeyer & Kaplan (2009) give a list of computational approaches to model intrinsic motivation in psychology (many of the listed reward functions, like FM, CM or StabM, while effectively demonstrating some of the possibilities of a computational analogue of intrinsic reward, are not meant to model anything that would be called intrinsic motivation in psychology).\n\nAnother strength of the paper is the inclusion of substantial detail about the implementations of the experiments. The authors have been fairly careful to include parameter settings and have provided more than one description of each experiment to help the reader understand.\n\nSome questions/comments I would especially recommend addressing in revising this paper:\n\n1. To the best of my understanding, this method is designed to work with some kind of subgoal-state generator, like hierarchical reinforcement learning (in reference to p. 2); do the experiments include this?\n2. I found the use of the words \"goal,\" \"sub-task\" and \"task\" quite confusing—I assume in most cases that these were references to the outputs of a subgoal-state generator like I mentioned in Question 1, but sometimes I wondered if the environment was also supposed to be formulated as having a single goal state? I'd recommend defining the term you use explicitly and using the same term consistently to reduce confusion.\n3. \"Recently, LeCun (2022) describes …\" (p. 1) → I don't think I understand the point of your first paragraph. Removing it would likely reduce my confusion and save you some space.\n4. There are multiple references to \"curiosity-based\" or \"curiosity-driven\" intrinsic motivation but no definition of what the term refers to. The term \"curiosity\" gets used for multiple things, so it is valuable to be clear about which ones are meant. \n5. \"but the extrinsic reward may not be obtained.\" (p. 2)→ I'm not sure what this means.\n6. \"For a new memory state,\" (p. 4) ← Not sure what a \"memory state\" is—are you just referring to when you store a state to memory for the first time?\n7. \"in a deterministic (rather than probabilistic) manner to help the algorithm learn faster.\" (p. 3) ← Can you provide an explanation for how deterministic versus probabilistic choice speeds up learning (even if it is only in the appendix)? It seems like there must be a reason the original authors used the probabilistic choice.\n\nTypos and grammatical suggestions (no intended influence on score)\n\n1. \"Motivated by these observations, we seek to find such intrinsic cost/reward functions whereby it is innate to the agent, but is context dependent and can be triggered according to the task at hand.\" (p. 2) → There's some noun-verb disagreement here, so you could use something like, \"Motivated by these observations, we seek intrinsic cost/reward functions that are innate to the agent, but context dependent and which can be triggered according to the task at hand.\"\n2. \"tells us what are the sub-tasks\" (p. 2) → \"tells us the sub-tasks\n3. \"Oudeyer & Kaplan (2009) describes\" (p. 3) → \"Oudeyer & Kaplan (2009) describe\"\n4. \"Go-Explore.\" (p. 2) ← I recommend providing the reader with a summary/review the the \"Go\" and \"Explore\" phases of Go-Explore in preparation for Section 3.\n5. \"Go-Explore (Ecoffet et al., 2019) describes …\" (p. 2) → Go-Explore is an RL algorithm (or family of algorithms) so it typically doesn't describe, though Ecoffet et al. do! Also, as a note, detachment and derailment are also described in the published version of the paper (see p. 580 and pp. 19-20 in the supplementary material) so it might be preferable to cite the Nature publication here.\n6. \"due to the in-built stochasticity of the algorithm\" (p. 2) → I'd recommend foreshadowing how you'll address these problems here. This would also be a great place to provide some intuition about why most algorithms have in-built stochasticity.\n7. \"We seek to guide the search via a goal-directed approach, much like using a compass to find the way in the forest.\" (p. 3) ← While I appreciate your aim to build intuition, I don't understand the analogy. What makes using a compass goal-directed? The goal is probably not simply to use the compass, maybe it is to go in a particular direction?\n8. \"where it can be viewed\" (p. 3) ← Where what can be viewed?\n9. \"which is ideal in sparse reward settings. … GDIR is innate to the agent and hence applicable across a wide range of environments without needing to rely on a properly shaped environment reward.\" (p. 3) ← There seems to be a contradiction here, because sparse reward settings obviously don't have potential-based shaped environmental reward; it would be helpful to have more intuition on how potential-based reward shaping is usually used.\n10. \"baseline for all other agents.\" (p. 3) ← I'd leave this at \"serves as the worst-case baseline\" because I'm not sure how other agents come into it.\n11. \"its superior performance in them.\" (p. 3) ← Not sure what \"them\" is in this sentence.\n12. \"Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronck. Monte-carlo tree search: A new framework for game ai.\" (p. 10) → \"Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronck. Monte-Carlo tree search: A new framework for game AI.\"\n13. \"Marek Grzes. Reward shaping in episodic reinforcement learning. 2017.\" ← I think this is an AAMAS paper, but this could use more citation detail.\n14.\"P Read Montague. Reinforcement learning: an introduction, by sutton, rs and barto, ag. Trends in cognitive sciences, 3(9):360, 1999.\" (p. 11)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E-lVboPCkVH", "reviewer": "Reviewer_JKcR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper talks about the exploration of sparse reward environments. Based on the Go-explore paper, the paper proposed a goal-conditioned potential-based intrinsic reward for goal-conditioned tasks. The paper conduct experiments on the 2D grid maze environments, Towers of Hanoi, Game of Nim, Mountain Car and Cart Pole. ", "summary_of_the_review": "Please see the strength and weaknesses above. ", "strength_and_weaknesses": "Strength: \n1. The paper tackles an interesting problem of exploration in the sparse-reward RL environments. \n2. The paper builds upon a strong baseline: Go-explore.\n3. The paper compares thoroughly against multiple baselines. \n\nWeakness: \n1. My main concern with the paper is that the experiment mostly builds on toy tasks, e.g., 2D maze, Mountain Car, and Cart Pole. In addition, the paper didn't talk about the random seed for their experiments, which is a pretty standard evaluation protocol for modern RL methods. Lastly, the evaluation metric used in the paper is somewhat strange. Why not report the success rate? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AWvjqp_S9N", "reviewer": "Reviewer_j1xa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the use of goal-directed intrinsic rewards (GDIR), and evaluates them in a Go-Explore-based agent on a variety of discrete environments, as well as some continuous control ones. The goal-directed intrinsic rewards are hand-crafted for each task, and the authors choose functions that are similar to the goals many human players would perceive - e.g. Manhattan distance to a goal in a discrete maze. The GDIRs are only used in the Go-Explore-based agents, not in the baselines. Generally, the GDIR-enhanced agents perform better than the baselines, although the difference with respect to GDIR-less Go-Explore is not always equally pronounced.", "summary_of_the_review": "The basic idea behind the paper, using intrinsic rewards in Go-Explore, is a good idea. However, the implementation presented here looks very brittle, due to heavy reliance on handcrafting and simulators. As it stands, the paper is not making a convincing case that the proposed method will generalize well. I recommend rejection.", "strength_and_weaknesses": "- Strength: the method is tested, and shown to generally outperform the chosen baselines, across multiple environment types and tasks.\n- Strength: the core of the paper - the method - is explained clearly.\n- Weakness: the GDIRs are hand-crafted for each task, which harms the generalizability of the method.\n- Weakness: the Go-Explore aspect of the method relies heavily on being able to use simulators. The authors recognize this dependence, but it still reduces the applicability of the method in practice.\n- Weakness: the pre-processing of different environment types is somewhat specific to the proposed method, in particular the Go-Explore aspect.\n- Weakness: some baselines are missing, e.g. novelty signals based on learned representations, and non-Go-Explore agents with the GDIRs. Some citations are also missing, e.g. Random Network Distillation by Burda et al., Curiosity-driven Exploration by Self-supervised Prediction by Pathak et al., and Never Give Up / Agent57 by Badia et al.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject