Learning with AMIGo: Adversarially Motivated Intrinsic Goals
Authors: Andres Campero
Roberta Raileanu
Heinrich Kuttler
Joshua B. Tenenbaum
Tim Rocktäschel
Edward Grefenstette
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
exploration
meta-learning

Abstract: A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose AMIGo, a novel agent incorporating -- as form of meta-learning -- a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals to train a goal-conditioned "student" policy in the absence of (or alongside) environment reward. Specifically, through a simple but effective "constructively adversarial" objective, the teacher learns to propose increasingly challenging -- yet achievable -- goals that allow the student to learn general skills for acting in a new environment, independent of the task to be solved. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks where other forms of intrinsic motivation and state-of-the-art RL methods fail.

Published as a conference paper at ICLR 2021
LEARNING WITH AMIGo:
ADVERSARIALLY MOTIVATED INTRINSIC GOALS
Andres Campero"
Roberta Raileanu
Heinrich Küttler Facebook Al Research London, UK hnr@ fb. com
Brain and Cognitive Sciences, MIT Cambridge, USA
New York University
New York, USA
campero@mit.cs
raileanuecs. .yu.edu
Joshua B. Tenenbaum
Tim Rocktäschel
Edward Grefenstette University College London & Facebook Al Research London, UK egrefen@fb.com com
Brain and Cognitive Sciences, MIT Cambridge, USA @bt.@mit.c.m
University College London & Facebook AI Research London, UK rockt@fb. com
ABSTRACT
A key challenge for reinforcement learning (RL) consists of learning in environ- ments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose AMIGO, novel agent incorporating- as form meta-learning-a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals to train goal-conditioned "student" policy in the absence of (or alongside) environment reward. Specifically, through a simple but effective "constructively adversarial" objective, the teacher learns propose increasingly challenging- achievable-goall that allow the student to learn general skills for acting in new environment, independent of the task be solved. We show that our method generates a natural curriculum of self- self-proposed goals which ultimately allows the agent to solve challenging procedurally-gent -enerrted tasks where other forms of intrinsic motivation and state-of-the- art RL methods fail.
INTRODUCTION
The success of Deep Reinforcement Learning (RL) on wide range of tasks, while impressive, has so far been mostly confined to scenarios with reasonably dense rewards (e.g. MMih et al., Vinyals ctt al.,222)..) 2019), or those where a perfect model of the environment can be used for search, such as the game of Go and others (e.g. Silver et al.) 2016 Duan et al. 2016 Moraveik et al. (2117). Many real-world environments offer extremely sparse rewards, any at all. In such environments, random exploration, which underpins many current RL approaches, is likely to not yield sufficient reward signal to train an agent, or be very sample inefficient as requires the agent to stumble onto novel rewarding states by chance. In contrast, humans are capable of dealing with rewards that are sparse and lie far in the future. For example, to a child, the future adult life involving education, work, or marriage provides useful reinforcement signal. Instead, children devote much of their time to play, generating objectives and posing challenges to themselves as a form of intrinsic motivation. Solving such self- uef-osropsed tasks encourages them to explore, experiment, and invent; sometimes, as in many games and fantasies, without any direct link to reality or to any source of extrinsic reward. This kind of intrinsic motivation might be crucial feature to enable learning in real-world environments (Shulz 2012)
To address this discrepancy between naive deep RL exploration strategies and human capabilities, present a novel meta-learning method wherein part of the agent learns to self- celf-propose Adversarially Motivated Intrinsic Goals (AMIGo). In AMIGo, the agent is decomposed into goal-generating teacher and a goal-conditioned student policy. The teacher acts as a constructive adversary to the
Work done during intersship Facebook AT Research.

---

Published as a conference paper at ICLR 2021
goal-generating teacher
student policy
(KW) Categorical Goal
Action
Softmax
Softmax
19:00
(nun]
CNN
Embed
Objects
Colors
Types
Figure I: Training with AMIGO consists of combining two modules: a goal-generating teacher and goal-conditioned student policy, whereby the teacher provides intrinsic goals to supplement the extrinsic goals from the environment. In our experimental set-up, the teacher is a dimensionality- preserving convolutional network which, at the beginning of an episode, outputs location in absolute (z, y) coordinates. These are provided as one-hot indicator in an extra channel of the student's convolutional neural network, which in turn outputs the agent's actions.
student: the teacher is incentivized to propose goals that are not too easy for the student to achieve, but not impossible either. This results in natural curriculum of increasingly harder intrinsic goals that challenge the agent and encourage learning about the dynamics of given environment.
augmentation of any agent trained with policy gradient-base -based methods. Under this view, the original policy network becomes the student policy, which only requires its input- processing component to be adapted to accept additional goal specification modality. The teacher policy can then be seen as a "bolt-on" to the original policy network, entailing that this method is-to the extent that the aforementioned goal-conditioning augmentation is mussible-archiieeuure--ggoot and can be used on variety of RL training model architectures and training settings.
AMIGO can be viewed
As advocated in recent work (Cobbe al. 2019 Zhong al., 2020; Risi Togelius 2019 Küttler
iet al., (2020), we evaluate AMIGO for procedurally-genee -ygeerrteeedtdd environments instead of trying to learn to perform a specific task. Procedurally-genee have to deal with a parameterized family of tasks, resulting in large observation spaces where memorizing trajectories is infeasible. Instead, agents have learn policies that generalize across different environment layouts and transition dynamics (Rajeswaran et al. 2017 Machado al. 2018 Foley al. 2018 Zhang et al. 2018). Concretely, we use MiniGrid Chevalier-Boisverr et al. environments with symbolic/discrete (expressed in terms of objects like walls, doors, keys, chests and balls) observation space which isolates the problem of exploration from that of visual perception. MiniGrid is widely recognized challenging benchmark for intrinsic motivation, which was used in many recent publications such as Goyal er al. 2019, Bougie al. 2019, Raileanu and Rocktäschel 2020. Modhe al. 2020 etc. We evaluate our method on six different tasks from the MiniGrid domain with varying degrees of difficulties, in which the agent needs acquire diverse range of skills in order to succeed. Furthermore, MiniGrid is complex and competitive baselines such as IMPALA (that achieve SOTA in other domains like Atari) fail. Raileanu& & Rocktäschel (2020) found that MiniGrid presents particular challenge for existing state-of oftte-r-t intrinsic motivation approaches. Here, AMIGO sets a new oftte-of-the-rtt on some of the hardest MiniGrid environments, being the only method capable of successfully obtaining extrinsic reward some of them.
environments are challenging since agents
2018). suite of fast-to-rum procedurally-geneeeee
In summary, we make the following contributions: (i) we propose Adversarially Motivated Intrinsic Goals-an an approach for learning a teacher that generates increasingly harder goals, (ii) we show, through 1 14 experiments on 6 challenging exploration tasks in procedurally generated environments, that agents trained with AMIGO gradually learn interact with the environment and solve tasks which are too difficult for state-of- ofate-of-the-arr methods, and (iii) we perform an extensive qualitative analysis and ablation study.
2

---

Published as a conference paper at ICLR 2021
2
RELATED WORK
Our work has connections to many different research areas but due to space constraints, we will focus discussion on the most closely related topics, namely intrinsic motivation and curriculum learning.
Intrinsic motivation (Oudeyer et al.) 2007 Oudeyer & Kaplan 2009 Schmidhuber 1991 Barto
2013) methods have proven effective for solving various hard-exploration tasks (Bellemare et al.
2016 Pathak et al. 2017 Burda al.) 2019b). One prominent formulation is the use of novelly, which in its simplest form can be estimated with state visitation counts (Strehl & Littman (008)
and has been extended high-dimensional state spaces (Bellemare al., 20166] Burda et al.] 2019b
Ostrovski et al. 2017). Other sophisticated versions of curiosity (Schmidhuber 1991) guide the agent learn about environment dynamics by encouraging to take actions that reduce the agent's
uncertainty (Stadie et al. 2015 Burda et al.] 2019b), have unpredictable consequences (Pathak
et al. 2017 Burda et af. 2019g). or large impact on the environment (Raileanu & Rocktäschel
2020 Other forms of intrinsic motivation include empowerment (Klyubin et al.) 2005) which
encourages control of the environment by the agent, and goal diversity (Pong et al. 2019) which
encourages maximizing the entropy of the goal distribution. In Lair et al. (019)) intrinsic goals are discovered from language supervision. The optimal rewards framework presents intrinsic motivation as mechanism that goes beyond exploration, placing its origin in evolutionary context (Singh ett al. 2009). or framing it as a meta-optimization problem of selecting internal agent goals which optimize the designer's goals Soog et al. (2010). More recently /Zheng al.) (2018) extend this framework to learn parametric additive intrinsic rewards. Our work differs from all of the above by formulating intrinsic motivation as "constructively adversarial" teacher that proposes increasingly
harder goals for the agent.
Curriculum learning (Bengio al. 2009) is another useful technique for tackling complex tasks but the curricula are typically handcrafted which can be time consuming. In our work, the curriculum is generated automatically in an unsupervised fashion. Another automatic curriculum approach learning was proposed by |Schhidhhuber (201T), where an agent constantly searches the space of problems for the next solvable one. However, this method not scalable to more complex tasks. Florensa et al. (2017) generate a curriculum by increasing the distance of the starting-poin to a goal. In contrast to AMIGO, this method assumes knowledge of the goal location and the ability to reset the agent in any state. A student can also self- self-propose goal by hindsight experience replay (HER, Andrychowicz et al., 2017). which has been demonstrated to be effective in alleviating the sparse reward problem. Recent extensions have improved goal selection by balancing the difficulty and diversity (Fang et al., 2019) of goals. In contrast to our work, in HER, there is no explicit incentive for the agent to explore beyond its current reach. Since HER is rewarded for all the states it visits, it is rewarded for casy-to-reach states, even late in the training process. Other related work has trained a teacher generate a curriculum of environments that maximize the learning process of the student (Portelas et al. 2020a). The question of the complementarity between these and our work worth pursuing the future. More similar to our work, Matiisen et al.] (2017) train a teacher to select tasks in which the student is improving the most or in which the student's 's performance is decreasing to avoid forgetting. Note that AMIGO uses a different objective for training the teacher, which encourages the agent to solve progressively harder tasks. Similarly, Racaniere et al. (2019) train a goal-conditioned policy and a goal-setter network in non-adversarial way to propose feasible, valid and diverse goals. Their fcasibility criteria is similar to ours, but requires training an additional discriminator to rank the difficulty of the goals, while our teacher is directly trained to generate goals with appropriate level of difficulty[]] Recent surveys of curriculum generation in the context of RL include Narvekar et al. (2020) and Portelas et al. (2020b).
Closer to our work, /Sukhbaatar et al. (2017) use an adversarial framework but require two modules that independently act and learn in the environment, where one module is encouraged to propose challenges to the other. This setup can be costly and restricted only proposing goals which have already been reached by the policy. Morcover, it requires a resettable or reversible environment. In
'Unfortunately, both the code for their method which is far from simple- and for their experimental settings have not been made available by the authors. Therefore we not only cannot run a fair implementation of their approach against our setting for comparison, cannot he guaranteed to successfully reimplement it ourselves as there no way of reproducing their results in their setting without the code for the latter.
3

---

Published as a conference paper at ICLR 2021
contrast, our method uses single agent acting in the environment, and the teacher is less constrained in the space of goals can propose.
Also similar to ours, Florensa et al. (2018) present GoalGAN, generator that proposes goals with the appropriate level of difficulty as determined by a learned discriminator. While their work is similar in spirit with ours, there are several key differences. First, GoalGAN was created for and tested on locomotion tasks with continuous goals, whereas our method is designed for discrete action and goal spaces. While not impossible, adapting it our setting is not trivial due to the GAN objective. Second, the authors do not condition the generator on the observation which is necessary in seocedurally-genrraie
environments that change with each episode. GoalGAN generates goals from buffer, but previous goals can be unfeasible or nonsensical for the current episode. Hence, GoalGAN cannot be easily adapted to poncedurally-generaacee environments.
A concurrent effort, Zhang et al. (2020) complements ours, but in the context of continuous control, by also generating a curriculum goals which are neither too hard nor too easy using a measure of epistemic uncertainty based on an ensemble of value functions. This requires training multiple networks, which can become too computationally expensive for certain applications.
Finally, our approach is loosely inspired by generative adversarial networks (GANs, Goodfellow et al. 2014), where generative model trained fool discriminator which is trained to differentiate between the generated and the original examples. In contrast with GANs, AMIGO does not require discriminator, and is "constructively adversarial", in that the goal-generating teacher is incentivized by its objective to propose goals which are challenging yet feasible for the student.
3 ADVERSARIALLY MOTIVATED INTRINSIC GOALS
AMIGO is composed of two subsystems: a goal-conditioned student policy which "controls" the agent's actions in the environment, and geal-generating teacher (see Figure [ which guides the student's training. The teacher proposes goals and is rewarded only when the student reaches the goal after certain number of steps. The student receives reward for reaching the goal proposed by the teacher (discounted by the number of steps needed to reach the goal). The two components are trained adversarially in that the student maximizes reward by reaching goals fast as possible, while the teacher maximizes reward by proposing goals which the student can reach, though not too quickly. In addition this intrinsic reward, both modules are rewarded when the agent solves the full task.
3.1 TRAINING THE STUDENT
We consider the traditional RL framework of a Markov Decision Process with a state space S, set of actions A and a transition function p($t+1|st, az) which specifies the distribution over next states given a current state and action. At each sime-step 1, the agent state 8, E S takes an action a, E A by sampling from a conl-conditioned stochastic student policy л ((a g: 0z) where g is a intrinsic goal provided by the teacher.
The teacher Gs 0g) is separate policy, operating on a different "granularity" than the student: it takes as input an initial state and outputs as actions goal y for the student, which stays the same until new goal is proposed. The teacher proposes new goal every time episode begins or whenever the student reaches the intrinsic goal. We assume that some goal verification function v(s, g) can be specified as an indicator over whether a goal g is achieved in a state We use this to define the undiscounted intrinsic reward r? as:
10 [+1 if the state s, satisfies the goal g otherwise
r = u(st.g)
At each time step i, the student receives a reward r = rg + rf. which is the sum of the intrinsic reward r provided by the teacher and the extrinsic reward rī provided by the environment. The student, represented as neural network with parameters 0,, is trained to maximize the discounted expected reward R, = E Et= > where y E [0, 1) is the discount factor. We consider finite time horizon H as provided by the environment.
4

---

Published as a conference paper at ICLR 2021
3.2 TRAINING THE TEACHER
The teacher G( 0g), represented as neural network with parameters 0g. is trained to maximise its expected reward. The teacher's reward r" is a function of the student's performance on the proposed goal and is computed every time this goal is reached (or at the end of episode). As a result, the teacher operates at a different temporal frequency, and thus its rewards are not discounted according to the number of steps taken by the agent. To generate an automatic curriculum for the student, we positively reward the teacher if the student achieves the goal with suitable effort, but penalize it if the student either cannot achieve the goal, or can do so too easily. There are different options for measuring the performance of the student here, but for simplicity we will use the number of steps + takes the student to reach an intrinsic goal since the intrinsic goal was set (with 1+ = 0 if the student does not reach the goal before the episode ends). We define a threshold (* such that the teacher is positively rewarded by , when the student takes more steps than the threshold to reach the set goal, and negatively if it takes fewer steps or never reaches the goal before the episode ends. We thus define the teacher reward as follows, where a and 3 are hyperparameters (see Section 4.2 for implementation details) specifying the weight of positive and negative teacher reward:
+ if 1 2 l" if <t
One can try to calibrate a fixed target threshold (* to force the teacher to propose increasingly more challenging goals as the student improves. Initial experiments with a fixed threshold indicated that the loss function was sufficient to induce harder goals and curriculum learning. However, this threshold different across environments and has to be carefully fixed (depending on the size and complexity of the environment). A more adaptive- aaaeei heuristic- approach we adopt is to linearly increase the threshold t* after a fixed number times in which the student successfully reaches the intrinsic goals. Specifically, the threshold t* is increased by whenever the student successfully reaches an intrinsic goal in more than t" steps for ten times in a row. This increase in the target threshold provides additional metric to visualize the improvement of the student through the "difficulty" of its goals (see Figure
3.3 TYPES OF GOALS
We can conceive of variants of AMIGO whereby goals are provided in the form of linguistic instructions, images, etc. To prove the concept, in this framework, goal is formally defined as a change the observation on a tile, as specified by an (z, y) coordinate. The agent must modify the tile before the end of an episode (e.g. by moving to it, or causing the object in it to move or change state). The verification function e is then trivially the indicator function of whether the cell state different from its initial state at the beginning of the episode. Proposing (I. coordinates as goals can present diverse set of ways for an agent to achieve the goal, as the coordinates can not only be affected by reaching them but also by modifying what is them. This includes picking up keys, opening doors, and dropping objects onto empty tiles. In some cases in our setting, moving over square is not the simplest thing possible (e.g. when an obstacle can be removed opened). Similarly, in other tasks and environments it could be easier to affect a cell by throwing something at it, rather than by reaching it. Likewise, simply navigating the corner of a locked room) might require solving several non- conviaivial sub- sub-problems (c.g. identifying the right key, going to it, then going to the door, unlocking it, and finally going to the target location). We give some examples of goals proposed by the teacher, alongside the progression in their difficulty as the student improves, in Figure
door can be
set of coordinates (say,
3.4 AUXILIARY TEACHER LOSSES
To complement our main form of intrinsic reward, explore few other criteria, including goal diversity, extrinsic reward, environment change and novelty. We report, in our experiments of Section 4 the results for AMIGo using these auxiliary losses. We present, in Appendix [D] an ablation study of the effect of these losses, alongside some alternatives to the reward structure for the teacher network.
Diverse Goals. One desirable property is goal diversity (Pong et al. 2019 Raileanu & Rocktäschel] 2020). In our implementation of AMIGO in the experiments of Section 4 we used entropy regulariza-
5

---

Published as a conference paper at ICLR 2021
tion train the teacher and student, which encourages such diversity. This regularization, along with the scheduling of the threshold, helps the teacher avoid getting stuck in local minima. Additionally, considered rewarding the teacher for proposing novel goals similar to count-based exploration methods (Bellemare et al. 2016 Ostrovski et al. 2017) with the difference that in our case the counts for goals instead of states, based the number of times the teacher presents type of goal the student. This did not improve performance and is not part of our model for the rest of the paper.
environment, humans will notice the factors of variation and exploit them. In episodic training, RL agents and algorithms are informed particular state was an episode end. To bias AMIGO towards learning the factors of variation in an environment, while not giving it any domain knowledge any privileged information which other comparable intrinsic motivation systems and RI. agents would not have access to, we positively reward the teacher if the content of the goal location it proposes changes at an episode boundary, regardless of whether this change was due the agent. Thus, the teacher is rewarded for selecting goals where the object type changes if the episode changes (for example door becomes wall, or a key becomes an empty tile due to the new episode configuration). While this heuristic is quite general and could be effective for many tasks it encourages agents to note environmental factors of variation, we note it might not be useful in all possible domains and as such is not an essential part of AMIGG. A comparison extension of this loss other intrinsic motivation methods would interesting, but not straightforward and is left for future research, we just note that this auxiliary loss on its own not able to solve even the medium difficulty environments.
Episode Boundary Awareness. When playing in a procedurally-genenaae
Extrinsic Goals. To help transition into the extrinsic task and avoid local minima, we reward both the teacher and the student with environment reward whenever the student reaches the extrinsic goal, even if this does not coincide with the intrinsic goal set by the teacher. This avoids the degenerate case where the student becomes good satisfying the extrinsic goal, and the teacher forced to encourage it "away" from it.
EXPERIMENTS
We follow Raileanu & Rocktäschel (2020) and evaluate our models on several challenging mroceurallly-generatee environments from MiniGrid (Chevalier-Boisvert et al. 2018). This en- vironment provides a good testbed for exploration in RL since the observations are symbolic rather than dighenimensional, which helps disentangle the problem of exploration from that of visual understanding. We compare AMIGO with state-of-the-ate methods that use various forms of ex- ploration bonuses. We use TorchBeast (Küttler al. 2019). a PyTorch platform based IMPALA (Espeholt et al. (018) for fast, asynchronous parallel training. The code for these experiments included in the supplementary materials, and has also been released under "htips://anonymous"" to facilitate reproduction of our method and its use in other projects
research
(a) EExample episodes KCharder.
(b) Example episodes in OMhard.
Figure 2: Examples of MiniGrid environments. KCharder requires finding the key that can unlock door which blocks the room where the goal is (the blue ball). OMhard requires sequence correct steps usually involving opening door, opening chest to find key of the correct color, picking-up up the key to open the door, and opening the door to reach the goal. The configuration and colors the objects change from one episode to another. To our knowledge, AMIGO is the only algorithm that can solve these tasks. For other examples, see the MiniGrid repository

---

Published as a conference paper at ICLR 2021
4.1 ENVIRONMENTS
We evaluate AMIGO on the following MiniGrid environments: KeyCon1S3R3 (KCmedium), ObstrMaze (OMmedium). ObstrMaze2Dllbb (OMmedhard), KeyCorrS4R3 (KChard), Key- CorS5533 (KCharder), and ObstrMaze (OMhard). The agent receives a full observation of the MiniGrid environment. The layout of the environment changes at every episode as it is procedurally- generated, Examples of these tasks can be found in Figure [ Each environment is a grid of size N x N (N being environment-specife)) where each tile contains at most one of the following colored objects: wall, door, key, ball, chest. An object in each episode is selected as an extrinsic goal. If the agent reaches the extrinsic goal, or a maximum number of time-steps is reached, the environment is reset. The agent can take the following actions: turn left, turn right, move forward, pick up an object, drop an object, or toggle (open doors or interact with objects). Each tile is encoded using three integer values: the object, the color, and a type flag indicating whether doors open closed. While policies could be learned from pixel observation alone, we will see below that the exploration problem is sufficiently complex with these semantic layers, owing to the procedurally generated nature of the tasks. The observations are transformed before being fed to agents by embedding each tile of the observed frame into a single representation encoding the object type, color, and type/flag.
The extrinsic reward provided by each environment for reaching the extrinsic goal in t steps is rf = 1 (.9 l)/mmax where ymax is the maximum episode length (which is intrinsic to each environment and set by the MiniGrid designers), if the extrinsic goal is reached at t, and 0 otherwise. Episodes end when the goal reached, and thus the scale of the positive reward encourages agents to reach the goal as quickly as possible.
4.2 AMIGo IMPLEMENTATION
The teacher is a
network of four convolutional layers interleaved with exponential linear units. Similarly, the student consists of four convolutional layers interleaved with exponential linear units followed by two linear layers with rectified linear units. Both the student and the teacher are trained using the TorchBeast (Küttler al. 2019) implementation of IMPALA (Espeholt et al. 2018), a distributed actor-critic algorithm. But while the teacher proposes goals only at the beginning of episode when the student reaches goal, the student produces an action and gets a reward at every step. To replicate the structure of reward for reaching extrinsic goals, intrinsic reward for the student is discounted to r² 1 - (.9 / when v(8t, 9) 1, and 0 otherwise. The hyperparameters for the reward for the teacher rT are grid searched, and optimal values are found
.7 and 8 .3 (see Appendix B full hyperparameter search details).
4.3 BASELINES AND EVALUATION
We use IMPALA (Espeholt et al. 2018) without intrinsic motivation as a standard deep RL baseline. We then compare AMIGO to a series of methods that use intrinsic motivation to supplement extrinsic reward, as listed here Count is Count- Based Exploration from Bellemare et al. (016)), which computes state visitation counts and gives higher rewards to less visited states. RND is Random Network Distillation Exploration by Burda et al. (2019b) which uses a random network to compute prediction error used as a bonus to reward novel states; ICM is Intrinsic Curiosity Module from Pathak cet al.] (2017), which trains forward and inverse models to learn a latent representation used compare the predicted and actual next states. The Euclidean distance between the representations of predicted and actual states (as measured in the latent space) is used as intrinsic reward. RIDE, from Raileanu & Rocktäschel] (2020), defines the intrinsic reward as the (magnitude of the) change between two consecutive state representations.
We have noted from the literature that some of these basclines were designed for partially observable environments (Raileanu & Rocktäschel] 2020; Pathak et al. 2017) so they might benefit from observing an agent-centric partial view of the environment rather than a full absolute view (Ye et al.) 2020). Despite our environment being fully observable, for the strongest comparison with AMIGo we ran the baselines in each of the following four modes: full observation of the environment for both the intrinsic reward module and the policy network, full observation for the intrinsic reward and partial observation for the policy, partial view for the intrinsic reward and full view for the policy, and partial view for both. We use an LSTM for the student policy network when it is provided with partial observations and a feed-forward network when provided with full observations. In
7

---

Published as a conference paper at ICLR 2021
report the best result (across all four modes) for each baseline and environment pair, with full breakdown of the results in Appendix A This, alongside a comprehensive hyperparameter search, ensures that AMIGO is compared against the baselines trained under their individually best-perfrrming training arrangement. We also compare AMIGO to the authors' implementation" of Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2017). In their reversible mode policies trained adversarially: Alice starts from a start-point and tries to reach goals, while Bob is tasked to travel in reverse from the goal to the start-point.
Section 4.4
We ran each experiment with five different seeds, and report in Section the means and standard deviations. The full hyperparameter sweep for AMIGo and all baselines reported in Appendix B alongside best hyperparameters across experiments.
4.4 RESULTS AND DISCUSSION
We summarize the main results of our experiments in Table As discussed in Section 45 the
reported result for each baseline and each environment is that of the best performing configuration for the policy and intrinsic motivation system for that environment, as reported in Tables 2 of Appendix A This aggregation of experiments (not counting the number of times experiments were run for different seeds) ensures that each baseline is given the opportunity to perform in its best setting, in order to fairly benchmark the performance of AMIGo.
Table 1: Comparison of Mean Extrinsic Reward at the end of training (averaging over a batch of episodes in IMPALA), Each entry shows the result of the best observation configuration, for each baseline, from Tables 5 of Appendix
Medium Difficulty Environments
Hard Environments
Model
KCmedium
OMmedium
OMmedhard
KChard
KCharder
OMhard
AMIGo
.93 .00
.92 + .00
.33 + .05
.54 .45
.44 + .44
.17 + .34
IMPALA
.00 .00
.00 + .00
.00 .00
.00 .00
.00 + .00
.00 .00
RND
.89 = .00
.94 .00
.88 = .03
.23 - .40
.00 = .00
1..000
RIDE
.90 .00
94
.86 106
.19
100
.00
COUNT
.90 - .00
.04
.00
.00 - .00
.00 100
.00
ICM
.42 = 211
.19
.16 +
.00 .00
00
.00
ASP
.00 = .00
=00
.00 + .00
.00 .00
.00 .00
.00 .00
IMPALA and Asymmetric Self-Play are unable to pass any of these medium or hard environments. ICM and Count struggle on the "casier" medium environments, and fail to obtain any reward from the hard ones. Only RND and RIDE perform competitively on the medium environments, but struggle obtain any reward on the harder environments.
Our results demonstrate that AMIGO establishes in MiniGrid. On environments with medium difficulty such as KCmedium, OMmedium, and OMmedhard, AMIGO performs comparably to other state-of-the-arr intrinsic motivation methods. AMIGo is often able to successfully reach the extrinsic goal even on the hardest tasks. To showcase results and sample complexity, we illustrate and discuss how mean extrinsic reward changes during training in Appendix C To analyze which components of the teacher loss were important, we present, in Appendix [D, an ablation study over the components presented in Section 3.4 Qualitatively, the learning trajectories of AMIGO display interesting and partially adversarial dynamics. These often involve periods in which both modules cooperate as the student becomes able to reach the proposed goals, followed by others in which the student becomes too good, forcing drop in the teacher reward, in turn forcing the teacher to increase the difficulty of the proposed goals and forcing the student to further explore. In Appendix E we provide more thorough qualitative analysis of AMIGO, wherein we describe the different phases of evolution the difficulty of the intrinsic goals proposed by the teacher, exemplified in Figure Further goal examples are shown in Figure of Appendix
state of the art in harder exploration problems
8

---

Published as a conference paper at ICLR 2021
generator current target
Phase
60
50
40-
frames
Figure 3: Examples of curriculum of goals proposed for different episodes particular learning trajectory on OMhard. The red triangle is the agent, the red square is the goal proposed by the teacher, and the blue ball the extrinsic goal. The top panel shows the threshold target difficulty, t" of the goals proposed by the teacher. The teacher first proposes very easy nearby goals, then it learns to propose goals that involve traversing rooms and opening doors, while in the third phase the teacher proposes goals which involve removing obstacles and interacting with objects.
CONCLUSION
In this work, we propose AMIGo, a meta-learning framework for generating a natural curriculum of goals that help train an agent as form of intrinsic reward, to supplement extrinsic reward (or replace it it not available). This is achieved by having a goal generator as a teacher that acts as a constructive adversary, and policy that acts as student conditioning intrinsic reward. The teacher is rewarded to propose goals that are challenging but not impossible. We demonstrate that AMIGO surpasses state-of tfttee-ff-heee of-the-art intrinsic motivation methods in challenging procedurally-generaee tasks in a comprehensive comparison against multiple competitive baselines, in series of 1 14 experiments across 6 tasks. Crucially, it the only intrinsic motivation method which allows agents to obtain any reward on some of the harder tasks, where non-intrinsic RL also fails.
those goals to maximize
The key contribution of this paper is model-agnostic framework for improving the sample complexity and efficacy of RL algorithms in solving the exploration problems they face. In our experiments, the choice of goal type imposed certain constraints on the nature of the observation, in that both the teacher and student need fully observe the environment, due to the goals being provided as absolute coordinates. Technically, this method could also be applied to partially observed environments where part of the full observation uncertain or occluded (e-g. "fog of war" in StarCraft), as the only requirement is that absolute coordinates can be provided and acted However, this is not a fundamental requirement, and future work we would wish to investigate the cases where the teacher could provide more abstract goals, perhaps in the form of language instructions which could directly specify sequences of subgoals. Other extensions to this work worth investigating are its applicability to continuous control domains, visually rich domains, generated environments such as (Cobbe et al. 2019). Until then, we are confident we have proved the concept in meaningful way, which other researchers will already be able easily adapt their model and RL algorithm of choice, in their domain of choice.
more complex procedurally
ACKNOWLEDGMENTTS
We thank the anonymous reviewers for their candid and helpful feedback and discussions. Roberta supported by the DARPA 1.2M grant.

---

Published as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30:5048-5058, 2017.
Andrew Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learning in natural and artificial systems, pp. 17-47. Springer, 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count- based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, ACM, 2009.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large- scale study of curiosity-driven learning. International Conference on Learning Representations, 2019a. WtLLhtttss://ppereeeee
hwt.:/et.fooooo www.:/nee/oooommmmmmmmmm Fodr//oroommmdd.. JAqYX
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network
distillation. In International Conference on Learning Representations, 2019b. URL https
//openreview net wttp://t//ooommmmmdddd100o1
Maxime Chevalier- Lucas Willems, and Suman Pal. Minimalistic gridworld environment for OpenAl gym.
wwtp://...cooa.mm..me........ http:///wwwwi..iieeeeeoommmmmina 2018.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint 1912.0..8 2019.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. ArXiv, abbs/160.00677, 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-rl with importance weighted letor-learner architectures. arXiv preprint 1802.0/.0 2018
Meng Fang, Tianyi Zhou, Yali Du Lei Han, and Zhengyou Zhang. Curriculum-guided hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 12623-12634, 2019.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. arXiv preprint wwXiv:177.005380 2017.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Proceedings of the 35th International Conference Machine Learning, 2018. URL http::/pproceedingge
ml wtt....reessss....oooo
html
John Foley, Emma Tosch, Kaleigh Clary, and David Jensen. Toybox: Better atari environments for testing reinforcement learning agents. arXiv preprint arXiv: 181 0288000 2018.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Process- ing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL htttp: nips cc/paper/5423 generative adversarial nets. .ddf
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent-centric measure control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-138815 IEFE, 2005.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PoTorch platform for distributed RL. arXiv preprint arXiv: 1910.03552... 2019.
10

---

Published as a conference paper at ICLR 2021
Heinrich Küttler, Nantas Nardelli, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In Workshop on Beyond Tabula Rasa in Rein- forcement Learning (BeTR-RL), 2020. URL htt.ggthh.coomaeoeee wttp:////..ccoooreeeerrrooo nle
Nicolas Lair, Cédric Colas, Rémy Portelas, Jean-Michel Dussoux, Peter Ford Dominey, and Pierre- Yves Oudeyer. Language grounding through social interactions and curiosity-driven multi-goal learning, arXiv preprint
2019.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems, 2017.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Matej Moraveik, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morril,, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling, DeepStack: Expert-level artificial intelligence in no-limit poker. ArXiv, abs/1701.0 771101722 2017.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. arXiv preprint arXiv:2003.04969, 2020.
Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2721-2730. JMLR. org, 2017.
Pierre Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
Pierre- Oudeyer, Frdric Kaplan, and Verena Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions evolutionary computation, 11(2):265-286, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine, Skew-fit: State-covering self-supervised reinforcement learning. arXiv preprint urXiv:1903.0360 2019.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pp. 835 853. PMLR, 2020a.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.0440 2020b.
Sebastien Racaniere, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated curricula through setter-solver interactions. arXiv preprint arXiv: 12892, 2019.
Roberta Raileanu and Tim Rocktäschel. (RIDE): Rewarding impact-driven exploration for procedurally-generee environments. In International Conference on Learning Representations, 2020.
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards gen- eralization and simplicity in continuous control. In Advances in Neural Information Processing Systems, pp. 6550-6561. 2017.
11

---

Published as a conference paper at ICLR 2021
Sebastian Risi and Julian Togelius. Procedural content generation: From automatically generating game levels to increasing generality machine learning. arXiv preprint arXiv:1911.1307/. 2019.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. In Front. Psychol., 2011.
Laura Schulz. Finding new facts; thinking new thoughts. In Advances in child development and behavior, volume 43, pp. 269-294. Elsevier, 2012.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panncershelvam,, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Satinder Singh, Richard I. Lewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pp. 2601-2606. Cognitive Science Society, 2009.
Jonathan Sorg, Satinder Singh, and Richard I. Lewis. Internal rewards mitigate agent boundedness. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 1007- 1014, 2010.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration reinforcement learning with deep predictive models. arXiv preprint arXiv: 2015.
Alexander I. Strehl and Michael I. Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309-133), 2008.
Sainbayar Sukhbaatar, Zeming Lin, Tlya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint 1rXiv:1703.054077 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft using multi-agent reinforcement learning. Nature, 575(7782)3300-354 2019.
Chang Ye, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Rotation, translation, and and cropping for zero-shot generalization. CoRR, abs/2001 .09908, 2020. URL httppss.//arxiv.or//abb.. 2001. 09908
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.079. 2018.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement, 2020.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp. 4654, 2018.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. RTFM: generalising to new environment dynamics via reading. In International Conference on Learning Representations, 2020. URL https://openrevil net http:://eeaaaoooooeeeeeorrroodtd
12

---

Published as a conference paper at ICLR 2021
Table 2: Fully observed intrinsic reward, fully observed policy.
Medium Difficulty Environments
Hard Environments
Model
KCmedium
OMmedium
OMmedhard
KChard
KCharder
OMhard
AMIGO
.93 + .00
.92 - .00
.83 .05
.54 .45
.44 + .44
.17 + .44
IMPALA
.00 +
.00 +: .00
+: .00
.00
.00 +
.00
RND
.00 + .00
.00 + .00
.00 + .00
.00 + .00
.00 = ..00
.00 .00
RIDE
.00 .00
.04 .M4
.00 .00
.00 + .00
.00 ..0
.00 = .00
COUNT
.00 .00
.00 .00
.00 +00
+00
.00 .00
.00
ICM
.00
.01 .01
.00 100
.00
.00 .00
.00
ASP
.00 = .00
.00 .00
.00 000
.00 1 .00
.00 .00
.00 .00
Table 3: Partially observed intrinsic reward, fully observed policy.
Medium Difficulty Environments
Hard Environments
Model
KCmedium
OMmedium
OMmedhard
KChard
KCharder
OMhard
RND
.64 .09
.01 = .01
.00 +: .00
.00 .00
.00 .00
.00 +: .00
RIDE
.84.00 .02
.00 -00=:0 .00
.00 - .00
.00 .00
.00 .00
.00 .00
COUNT
.45 26
.00 = 00
.00 = .00
.00 .00
.00 .00
.00 + .00
ICM
.00 = .00
.00 = .00
.00 .00
.00 .00
.00 + .00
Table 4: Fully observed intrinsic reward, partially observed policy.
Medium Difficulty Environments
Hard Environments
Model
KCmedium
OMmedium
OMmedhard
KChard
KCharder
OMhard
RND
.00
.00 = .00
.00 = .00
.00 .00
.00 .00
.00 .00
RIDE
1
.94
.35
.37
.00 1
.00 .0
COUNT
.01 1 001
.00 = .0
.00 +: .00
.00 + .00
.00 4.00
.00 .0
ICM
.06 .12
.05 = .66
.16 4: .22
.00 4: .00
.00 + 000
.00 +: .0
Table 5: Partially observed intrinsic reward, partially observed policy.
Medium Difficulty Environments
Hard Environments
Model
KCmedium
O.Mmedium
OMmedhard
KChard
KCharder
OMhard
RND
.89
.94
.03
.23
.00 + .00
.00
RIDE
.90 + .00
.85 = 288
.86 = .06
.00 + .00
.00 + 1.0
.00 .00
COUNT
.90 + .00
.04 = 00
.00 = .00
.00 : .00
.00 + .00
.00 + .0
ICM
.00 + .00
.19 = 19
.00 = .00
.00 + .00
.00 .00
.00 + .00
A FULL RESULTS
Tables 25 5 show the final performance of the intrinsic motivation baselines trained using one of four different training regimes enumerated in Section 4.3 For each baseline, we train on KCMedium and OMmedium, and use the best hyperparameters for each task (for that particular baseline and training regime) to train it on the remaining harder versions of those environments (i.e. on KChard and KCharder OMmedhard and OMhard, respectively).
For IMPALA, the numbers reported for KCmedium and OMmedium are from the experiments in Raileanu & Rocktäschell (2020). while the numbers for the harder environments presumed to be .00 because IMPALA fails to train on simpler environments.
As sanity check, also verified that ASP learns successfully in easier environments not considered here, such as MiniGrid- Soedd--mpty--eee Random-5x5-v0) and MannGrdd-eyCCorddoSSRR..... to validate the official Py Porch implementation.
13

---

Published as a conference paper at ICLR 2021
Tables 5 indicate that the best training regime for the intrinsic motivation baselines (for all the tasks they can reliably solve) is the one that uses a partially observed intrinsic reward and a partially observed policy (Table [5. When the intrinsic reward is based on full view of the environment, Count and RND will consider almost all states to be "novel" since the environment is procedurally- generated. Thus, the reward they provide will not be very helpful for the agent since it does not transfer knowledge from one episode to another (as is the case in fixed environments (Bellemare) al. 2016 Burda al. 2019b)). In the case of RIDE and ICM, the change in the full view of the environment produced by one action is typically a single number in the MiniGrid observation. For ICM, this means that the agent can easily learn predict the next state representation, so the intrinsic reward might vanish early in training leaving the agent without any guidance for exploring (Raileanu Rocktäschel, 2020). For RIDE, it means that the intrinsic reward will be largely uniform across all state-action parrs, thus not differentiating between more and less "interesting" states (which it can do when the intrinsic reward is based on partial observations (Raileanu & Rocktäschel] 20200).
B HYPERPARAMETER SWEEPS AND BEST VALUES
For AMIGO, we grid search over batch size for student and teacher E {8, 32, 150), learning rate for student E {.001, .0001), }. learning rate for teacher E {.05, .01, .0001} unroll length E {50, 100), en- tropy cost for student E {.0005,.000 0001, 10000,000000000000 .0001,, entropy cost for teacher E f.001,001, .05), embedding dimensions for the observations E {5, 10, 20). embedding dimensions for the student last linear layer {128, 256), and teacher loss function parameters and (1.0, 0.7, 0.5, 0.3, 0.0).
For RND, RIDE, Count, and ICM, we used learning rate 10-4, batch size 32, unroll length 100, RMSProp optimizer with 0.01 and momentum O, which were the best values found for these methods on MiniGrid tasks by |Raileanu & Rocktäschel (2020). We further searched over the entropy coefficient E (0.0005, 0.001, 0.0001} and the intrinsic reward coefficient {0.1, 0.01, 0.5) on KCmedium and OMmedium. The results reported in Tables [2, 3 and f use the best values found from these experiments, while the results reported in Table 5 use the best parameter values reported by Raileanu& & Rocktäschel (202)). For ASP, we ran the authors' implementation using its reverse mode. We used the defaults for most hyperparameters, grid searching only over sp steps E {5, 10, 20), rate E {,), and sp_alice_enti _entre (-003..03)
The best hyperparameters for AMIGO and each baseline are reported below:
AMIGo: a student batch size of 8, a teacher batch size of 150, student learning rate of .001, teacher learning rate of .001, an unroll length of 100, a student entropy cost of .0005, a teacher entropy cost of _01, and observation embedding dimension of 5, a student last layer embedding dimension of 256, and finally, a = 0.7 and 3 = 0.3.
RND: partially observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefficient of
RIDE: for KCmedium, partially observed intrinsic reward, partially observed policy, entropy cost
of .0005, intrinsic reward coefficient of I;; for OMmedium: fully observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefficient of 11
COUNT: partially observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefficient of
ICM: for KCmedium, partially observed intrinsic reward, fully observed policy, entropy cost of .0005, intrinsic reward coefficient of .1; for OMmedium: partially observed intrinsic reward, partially observed policy, entropy cost of .0005, intrinsic reward coefficient of
ASP: Best performing hyperparameters (in the easier environments) were 10 sp. steps, sp test rate of .,, and Alice entropy of .003. All other hyperparameters used the defaults in the codebase.
C SAMPLE EFFICIENCY
We show, in Figure 4 the mean extrinsic reward over time during training for the best configuration of the various methods. The first row consists of intermediately difficult environments in which different forms of intrinsic motivation perform similarly, the first two evironments require less than 30 million
14

---

Published as a conference paper at ICLR 2021
steps while OMmedhard and the three challenging environments of the bottom require the order of hundreds of millions frames. Any plot where a method's 's line is not visible indicates that the method is consistently failing to reach reward states throughout its training.
KCmedium
OMmedium
mean episode retur
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1e7
1e7
OMmedhard
KChard
1.0
0.8
0.6
mean episode return
0.4
0.2
0.0
KCharder
OMhard
1.0
Count
0.8
RIDE
0.6
RND
0.4
AMIGO
0.2
0.0
2
5
1e8
1e8
frames
Figure 4: Reward curves over training time comparing AMIGO to competing methods and baselines. The y-axis shows the Mean Extrinsic Reward (performance) obtained in two medium and four harder different environments, shown for 30M and 500M frames respectively.
The important point of note here is that on the two casiest environments, KCmedium and OMmedium, agents need about 10 million steps converge while the other four more challenging environments, they need an order 100 million steps to learn the tasks, showcasing AMIGO's contributions not just to solving the exploration problem, but also improving the sample complexity of agent training.
D ABLATION STUDY
To further explore the effectiveness and robustness of our method, in this subsection we investigate the alternative criteria discussed in Section 3.4 We compare the FULL MODEL with its ablations and alternatives consisting of removing the extrinsic bonus (NOEXTRINSIC), removing the environment change bonus (NOENVCHANGE), adding novelty bonus( WITHNOVEL.TY).
15

---

in the Published as a conference paper at ICLR 2021
We also considered two alternative reward forms for the teacher to provide continuous and gradual reward than the previously introduced "all or nothing" threshold. We consider a Gaussian p Normal(t", .) reward around the target threshold t":
-1 [1 ltg,(t*) log,
if -+=0
otherwise
and a Linear- Exrearr-xponential reward which grows linearly towards the threshold and then decays exponentially as the goal proposed becomes too hard (as measured according to the number of steps);
(e-1--o0 if 1+ if 8 <t
Et
We report these two alternative forms of reward (GAUSSIAN and study below.
Table 6: Ablations and Alternatives. Number of steps (in millions) for models to learn to reach its final level of reward in the different environments means the model did not learn to get any extrinsic reward). FULL MODEL is the main algorithm described above. NOEXTRINSIC does not provide any extrinsic reward to the teacher. NOENvCHANGE removes the reward for selecting goals that change as result of episode resets. WITHNOVELTY adds novelty bonus that decreases depending on the number of times an object has been successfully proposed. GAUSSTAN and LINEAR- EXAAA-R-PPNNNIALL... explore alternative reward functions for the teacher.
Medium Difficulty Environments
Hard Environments
Model
KCmedium
OMmedium
OMmedhard
KChard
KCharder
OMhard
FULL MODEL
7M
8M
320M
140M
300M
370M
NOEXTRINSIC
240M
50M
0
NOENYCHANGE
400M
37M
0
0
0
WITHNOVELTY
15M
20M
350M
100M
370M
GAUSSIAN
320M
60M
0
0
LINEAR-EXT
0
Performance is shown in Table o the number of steps needed to converge (to the final extrinsic reward) is reported. A positive number means the model learned to solve the task, while 0 means the model did not manage to get any extrinsic reward. For all models we encourage goal diversity on the teacher with a high entropy coefficient of .05 (as compared to .0005 of the student).
As the table shows, removing the extrinsic reward the environment change bonus severely hurts the model, making it unable to solve the harder environments. The novelty bonus was minimally beneficial in one of the environments (namely KChard) but slightly ineffective on the others. The more gradual reward forms considered are not robust to the learning dynamics and often result in the system going into rabbit holes where the algorithm learns propose goals which provide sub-optimal rewards, thus not helping to solve the actual task. Best results across all environments in our Full Model were obtained using the simple threshold reward function along with entropy regularization and in combination with the extrinsic reward and changing bonuses, but without the novelty bonus.
E QUALITATIVE ANALYSIS
To better understand the learning dynamics of AMIGo, Figure the intrinsic reward throughout training received by the student (top panel) as well as the teacher (middle panel). The bottom panel shows the difficulty of the proposed goals measured by the target threshold t* used by the teacher (described in Section 3 The trajectories reflect interesting and complex learning dynamics.
For visualization purposes we divide this learning period into five phases: Phase 1: becomes able to reach intrinsic goals with minimal difficulty. The teacher first learns to propose easy nearby goals. Phase 2: Once the student learns how to reach nearby goals, the adversarial dynamics cause a drop in the teacher reward which is then forced to explore and propose harder goals. Phase 3: An equilibrium is found in which the student is forced to learn reach more challenging goals.
The student slowly
16

---

Published as a conference paper at ICLR 2021
1.0
Phase
Phase
Phase
Phase
Phase 5
2
3
0.8
mean Intrinsic, rewards
0.6
mamm man
0.4
0.2
gen rewards
Wha pondddr
0.0
My Ma man
-0.1
-0.2
60
generator, current target
30
frames
Figure 5: An example of learning trajectory OMhard, one of the most challenging environments. Despite the lack of extrinsic reward, the panels show the dynamics of the intrinsic rewards for the student (top panel), for the teacher (middle panel), and the difficulty of the goals captured !t (bottom panel).
Phase 4: The student becomes too capable again and the teacher is forced to increase the difficulty of proposed goals. Phase The difficulty reaches a state where induces a new equilibrium in which the student is unable to reach the goals and forced to improve its student.
AMIGO generates diverse and complex learning dynamics that lead to constant improvements the agent's 's policy. In some phases, both components benefit from learning in the environment (as is the case during the first phase), while some phases are completely adversarial (fourth phase), and some phases require more exploration from both components (i.e. third and fifth phases).
Figure presented Section 4.4
further exemplifies a typical curriculum in which the teacher learns to propose increasingly harder goals. The examples show some of the typical goals proposed at different learning phases. First, the teacher proposes nearby goals. After some training, it learns propose goals that involve traversing rooms and opening doors. Eventually, the teacher proposes
17

---

Published as a conference paper at ICLR 2021
goals which involve interacting with different objects. Despite the increasing capacity of the agent to interact with the environment, OMhard remains a challenging task and AMIGO learns to solve it in only one of the five runs.
F GOAL EXAMPLES
Figure examples of goals proposed by the agent during different stages of learning. Typically, in early stages the teacher learns to propose easy nearby goals. As learning progresses is incentivized to proposed farther away goals that often involve traversing rooms and opening doors. Finally, in later stages the agent often learns to propose goals that involve removing obstacles and interacting with objects. We often observe this before the policy achieves any extrinsic reward.
Early
Mid
Late
g
Figure 6: Some examples of goals during early, mid, and late stages of learning (examples for KCmedium, OMhard and OMmedium are first, second, and third rows respectively). The red triangle the agent, the red square is the goal proposed by the teacher, and the blue ball is the extrinsic goal.
18


Reviews:
[{"review_id": "uke4JYM7ftx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a setter-solver or teacher-student scheme for training goal-conditioned agents (student/solver) in a discrete environment MiniGrid. In their method called AMIGo, the teacher takes the initial state of the student at the beginning of each episode as its input and proposes an intrinsic goal for the student to fetch, the student in each step takes an action and gets a reward combining both the extrinsic reward from the environment and the intrinsic reward, and the teacher gets a reward at the end of an episode or when the intrinsic goal is reached. The major idea is to make the goals proposed by the teacher staying at an appropriate/medium level of difficulty, i.e., fetchable for the student but not too easy. To do so, they use a threshold to define the teacher reward: they issue a positive(negative) reward to the teacher if the student spends more(less) steps than the threshold to reach the goal. And they adaptively increase the threshold if the student keeps successfully reaching the intrinsic goals. In experiments on five environments of MiniGrid, AMIGo outperforms five baselines with different types of intrinsic reward and set up new SoTA results on the studied tasks.\n\nThis paper is well-written and demonstrates a reasonable curriculum strategy by training a teacher model to propose goals in a discrete grid world (though largely simplified from realistic scenarios). The experiments show remarkable improvements over existing curiosity-driven or intrinsic reward-based methods and set up a new SoTA on those tasks.\n\nMy major concerns are the novelty of the idea and the generality of the proposed method.\n\n(1) Though the authors discussed the difference of AMIGo to GoalGAN, in which the teacher is instead a GAN generating continuous goals, the main idea behind the two are very similar, i.e., to train a teacher model proposing goals of medium difficulty, and they both use threshold(s) on rewards for this purpose. Although I agree that they use different architectures for the teacher model (for producing different types of goals), and the GAN in GoalGAN does not take student's initial state as input since it is not dealing with procedurally-generated environments, these are natural choices adapted to the specific environments/tasks for testing these methods. The main contribution of this paper, from the perspective of proposing a novel curriculum generator, is incremental.\n\n(2) Another related curriculum generating strategy is to balance the difficulty and diversity/representativeness of goals/tasks when proposing them to train the goal-conditioned student. It worth discussing the difference/advantage/limitation of the proposed method when compared to them. To name a few,\n\nPortelas et al., \"Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments\", CoRL 2019.\nFang et al., \"Curriculum-guided hindsight experience replay\", NeurIPS 2019.\n\n(3) The experiments only demonstrate the method in one type of environment. Although the MiniGrid is an appropriate environment to test the proposed curriculum generating method due to its simplicity, which allows the authors to rule out other factors and mainly focus on the quality of selected goals, it is not clear whether/how the proposed method can be easily generalized to other more practical and/or complicated environments/tasks such as robotic control, navigation, traffic, and gaming. More discussion on how to adapt AMIGo in those tasks can benefit researchers following this work.\n\n(4) The student is a goal-conditioned policy and it can be conditioned on one goal per time. During training, when is the student conditioned on the extrinsic goal or the intrinsic goal? Is there a schedule of assigning which goal to the student?\n\n(5) The student can also self-propose a goal by hindsight experience replay (HER) and this has been demonstrated to be effective in alleviating the sparse reward problem. It is interesting to see a discussion or comparison to HER methods. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "l7NiqxKePTX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## After Author Response\n\nThe author response for most of the points of confusion and weaker sections of the draft are satisfactory. I thank the authors for incorporating some of the feedback during this discussion. I am still on the fence about the generality of this idea, and how much the teacher objective is specific to the minigrid tasks. But overall I find this idea interesting and believe it adds value to the field.\n\n## Summary of the paper:\nThis paper proposes a goal-\"generating\" module (called the teacher) that will propose goals in procedurally generated grid world domains. The way they train this teacher is through policy gradient updates, with the teacher getting positive rewards if the agent achieves the provided goal but takes longer than a threshold to reach it, and negative rewards if the agent is unable to reach the goal or reaches it too quickly.\n\nThe threshold is gradually increased based on the agent's capabilities, allowing for more and more difficult goals.\n\nBoth the agent and the teacher additionally get rewarded for solving the external task. So the teacher is directed to provide goals that will also solve the given task, and the agent also learns to not just solve the given goal.\nOther auxiliary objectives for the teacher are entropy regularization for diverse goals, and episode boundary awareness.\n\n\n\n## Strengths:\n\n+ This paper presents a clear reward based mechanism to train the teacher. It also presents a reasonable answer to the question of how a difficulty threshold should be set for the teacher. It provides evidence that this goal-generation approach helps in directed exploration, is useful for general RL tasks (beyond goal-based RL), and speeds up solving of a variety of hard grid world environments.\n\n+ Figure 3 shows the changing thresholds and the subsequently generated targets for the agent, which is a good experiment to show that this increasing threshold mechanism works as intended.\n\n+ The ablations in the appendix are illustrative and show the different considerations and how different components of the system interact.\n\n\n## Weak Points:\n\n- The suggestion that the goal generation is adversarial seems slightly strong, with not much evidence to show that the generated goals are adversarial to the agent. The goals could also be generated progressively farther and farther due to the moving threshold.\n\n- The way the problem is set up and communicated makes it seem oriented towards grid-world problems exclusively. There is no definition of what a goal is. In fact, the problem setup itself is communicated rather vaguely, with terms being introduced as required rather than the entire setup being introduced clearly up front. Setting up the problem more clearly, with respect to RL literature (maybe refer to the Sutton and Barto book or alternative setups), being clear about what a goal is and how that goal is being used in the context of the grid world would make the paper clearer to read and understand.\n\n- It is also unclear whether the teacher rewards are discounted according to the number of steps taken by the agent, or if it operates at a lower temporal frequency. The entire teacher policy training regimen needs more explanation and clarity.\n\n- Given that the goal-generation here is to give an (x,y) location to travel to, it is unclear why an approach like GoalGAN cannot be adapted to this domain.\n\n- An additional question to ask would be whether the approach of Zhang et al can be adapted as a baseline to compare to this work. I can understand if it is not applicable, since that approach is specific to goal-conditioned RL, but would like some clarity on it.\n\n- When explaining the type of goals, the authors mention that this \"includes picking up keys, opening doors and dropping objects onto empty tiles\". How is an agent supposed to know whether it is supposed to move on to a square or drop an object there. The teacher does not seem to communicate what kind of goal is being generated. If the kind of goal is not communicated, wouldn't the agent do the simplest thing possible, and just move over the square, instead of figuring out that it should have dropped an object at the location?\n\n- The auxiliary task of \"Episode Boundary Awareness\" is a little unclear. My understanding is that the teacher is rewarded for selecting goals where the object type at that location changes if the episode changes. An illustrative example or clearer language is necessary to understand this additional objective and how it is helping the agent. This is very important as it seems that this auxiliary reward is essential in getting the teacher to learn goals that are useful for eventual agent success (as evidenced by the ablations in Table 6). In fact, I am curious as to why the performance of the system if this particular reward is removed drops drastically and is similar to the extrinsic reward being removed.\n\n- The main weakness of the approach for me is that it is unclear how this system scales to other RL problems. Since there is no definition of what a goal is considered to be, and no other examples given except grid worlds, it is hard to see what a broad application of this idea would be.\n\n- The learning curves (Figure 4) seem to show that the training with AMIGo is pretty high variance, compared to the baselines. Is there an explanation for this variance? Example, performance on OMmedium drops to 0 pretty regularly.\n\n\n## Additional Comments:\n\nThe related work section needs some additional work. While the authors present a broad array of related work, there are some suggestions to round off this section.\n\nFor intrinsic motivation, the authors should include work by Andy Barto (\"Intrinsic Motivation and Reinforcement Learning\", 2013) and other related work, as well as the optimal rewards framework by Satinder Singh's group (\"Where do rewards come from?\") along with work by Jonathan Sorg (ICML 2010) and more recently work by Zeyu Zheng (\"On Learning Intrinsic Rewards for Policy Gradient Methods\"). These views of intrinsic motivation as a vehicle for more than exploration is something that is not very clear in the current draft, even though the authors do present some alternative work like empowerment.\n\nFor curriculum learning, there are recent surveys that on curriculum learning for RL (\"Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey\", Narvekar et al.; \"Automatic Curriculum Learning For Deep RL: A Short Survey\", Portelas et al.) that should be referred to when explaining curriculum generation in the context of RL.\n\n## Conclusion\n\nOverall I would like to see clearer communication of some of the ideas as well as some explanation from the authors that will give me confidence that this idea is more broadly applicable.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "h1x4I1Ckev", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce AMIGo, an approach to curiosity in which an adversarial \"teacher\" agent proposes goals that the \"student\" agent attempts to achieve. The student obtains an intrinsic reward of +1 when it achieves the goal, and this augments the extrinsic reward additively. The teacher proposes a goal at the start of each episode, and whenever the student reaches a goal. The teacher is positively rewarded if the student achieves the goal after some threshold t*, and negatively rewarded otherwise. t* is incremented at fixed intervals. They then demonstrate that this outperforms a variety of suitable baselines.\n\nThe algorithm proposed has great strength in its simplicity. It does bear a considerable similarity to Sukhbaatar et al 2017 (their ASP baseline), but AMIGo appears to be more flexible, and they put forth evidence that it considerably outperforms ASP on their benchmarks. AMIGo is something that could be adapted very easily to many other settings, and it is a quite agnostic framework. The experimental results do show considerable advantages over baselines.\n\nI am, however, concerned about how the evaluations were done. At the very least, I find that the main text fails to describe what seem to be important caveats that can only be found in the appendix. In particular:\n-For baselines, suitable hyperparameters were found using a subset of tasks (tasks thought to be easier), and, if positive results were achieved, the same hyperparameters were run on the rest of the tasks. If scores of zero were obtained for the initial subset, the baselines were not run at all and were assumed to score zero. \n-As noted in the main text, some of the baselines were developed for partially observed environments with an egocentric view, so they let the baselines run in both modes (with algorithms with multiple modules, they varied this independently for both). While this might seem like simply giving the baselines more chances to succeed, there is a problem when this is coupled with the first practice of assuming zero scores. Some of these baselines obtained no score in the fully observed setting on the \"easy\" task subset, and hence were not run on the hard environments. They did better in partially observed modes, and hence obtained their hard environment scores in these partially observed modes. These partially observed modes are potentially much harder!\n\nWhile one might be able to make some claim of the form \"if they can't do the easy task, they can't do the hard,\" I am worried that these choices, used together, really obfuscate the performance differential. Note the quite high score variance that AMIGo achieves on the harder environments. Given such high variance, models which were simply not run in the fully observed setting could have actually performed better were they given the chance (and also perhaps needed further hyperparameter tuning). At the very least, I think this is a really unclear way of benchmarking things and should be corrected so that we do not have these statistical worries.\n\nAs a result, I cannot at this time recommend acceptance, though I am certainly open to being persuaded that my worries are unfounded. I could have missed something about their experimental methods, or environmental particulars could somehow make the above treatments perfectly reasonable.\n\nA more minor concern: I do wonder about the extent to which this method applies in other settings. Is this time-threshold reward method applicable to a broad array of settings, or is this particularly useful in minigrid and similar sorts of maze  navigations? Are there situations in which time-to-complete is not a good proxy for difficulty? Is it important that the goals be represented in some very compact way, as they are for these experiments? These are questions that would, I think, need additional experiments. The authors did a considerable amount of evaluation, so I hesitate to simply ask for more, but I do wonder if this approach is particularly suited to a narrow range of environments.\n\nI am also curious as to why the ASP baseline did so poorly. Its method seems to be quite similar, so more commentary about what makes one fail badly while the other succeeds would be really useful!\n\n*Updated score based on below discussion*\n\n\n*Updated score again based on below discussion*\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RSXV2oZkkz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a teacher agent in reinforcement learning framework. The teacher's responsible for setting goal for the student agent (the \"main\" agent). The teacher acts in adversarial way (but also somewhat cooperative) in setting the goal. That is, the goal should be achievable by the student, yet the goal should not be too simple that the student stops learning towards harder goals, and eventually achieve the original goal from the environment. Experiments show proposed method is able to learn okay performance for hard environments, even though most other methods fail to score at all.\n\nPros:\nI think this is a very interesting approach and has great potential. The idea is natural and experiments make a lot of sense. This should partially credit to clear writing and examples. The authors provide only one set of results in main paper, but have more ablation studies and variants of reward forms. Those results will be helpful to provide more insight into how well the agent handles the task.\n\nCons:\nMy major concern with proposed method is that it seems limited to particular applications, such as grid world. As the authors also pointed out, even proposing goals in this paper's setting is a challenging problem. For now, I don't see an automatic way to proposing goals for other reinforcement learning tasks, such as Atari games and Go. In those tasks, I feel teacher's goals may need to be designed incorporating human's understanding of the tasks (i.e. capture a stone in Go). This makes me feel the proposed method is not very generalizable, but I am open to change my opinion if authors provide a good answer.\n\nIf for those tasks, a human-generated set of goals are needed by teacher agent, then how do we correctly attribute the performance gain we saw in this paper? That is, are the gains over baseline methods from 1) we have a better training framework, such as curriculum learning, or 2) the goals are set in a way that incorporate our understanding of the environment?\n\nMy second concern is about scalability of proposed method. For some applications, it may be unnecessary to have a teacher that sets intermediate goals (e.g. AlphaGo -> AlphaZero). How does sample efficiency for AMIGo compared to baselines? I saw results related to number of steps for AMIGo alone, but not baseline methods. Conceptually if a teacher sets many intermediate goals, it's likely that the student will train more number of steps and possible to achieve better performance. However if this procedure requires 10x more number of steps to achieve on-par performance for easy/medium level tasks, then it will be important to acknowledge that. When will the student need a teacher vs. w/o a teacher seems also worthy studying.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to ICLR 2021. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes and include the missing references.
Decision: Accept (Poster)