Guided Exploration with Proximal Policy Optimization using a Single Demonstration
Authors: Gabriele Libardi
Gianni De Fabritiis
Venue:  (2021)
Keywords: PPO
sparse rewards
single demonstration
3D environment

Abstract:  Solving sparse reward tasks through exploration is one of the major challenges in deep reinforcement learning, especially in three-dimensional, partially-observable environments. Critically, the algorithm proposed in this article uses a single human demonstration to solve hard-exploration problems. We train an agent on a combination of demonstrations and own experience to solve  problems with variable initial conditions. We adapt this idea and integrate it with the proximal policy optimization (PPO). The agent is able to increase its performance and to tackle harder problems by replaying its own past trajectories prioritizing them based on the obtained reward and the maximum value of the trajectory.
We compare variations of this algorithm to different imitation learning algorithms on a set of hard-exploration tasks in the Animal-AI Olympics environment.
To the best of our knowledge, learning a task in a three-dimensional environment with comparable difficulty has never been considered before using only one human demonstration.

Under review as a conference paper at ICLR 2021
GUIDED EXPLORATION WITH PROXIMAL POLICY OP- TIMIZATION USING A SINGLE DEMONSTRATION
Anonymous authors Paper under double-blind review
ABSTRACT
Solving sparse reward tasks through exploration is one of the major challenges in deep reinforcement learning, especially in fice-ddimensiona, partially-bbserabbee environments. Critically, algorithm proposed in this article uses a single hu- man demonstration to solve hard-exploration problems. We train an agent on a combination of demonstrations and own experience to solve problems with vari- able initial conditions. We adapt this idea and integrate it with the proximal policy optimization (PPO). The agent able to increase its performance and to tackle harder problems by replaying its own past trajectories prioritizing them based on the obtained reward and the maximum value of the trajectory. We compare variations of this algorithm to different imitation learning algorithms hard-exploration tasks in the Animal-Al Olympics environment. To the best of our knowledge, learning a task in a three-dimensional environment with comparable difficulty has never been considered before using only one human demonstration.
set of
INTRODUCTION
Exploration one of the most challenging problems in reinforcement learning. Although significant progress has been made in solving this problem in recent years (Badia al. 2020aa b Burda et al.) 2018 Pathak et al. 2017 Ecoffet et al., 2019), many of the solutions rely on very strong assump- tions such as access to the agent position and deterministic, fully observable or low dimensional environments. For dineedimensional, partially observable stochastic environments with no access to the agent position the exploration problem still remains unsolved.
Learning from demonstrations allows to directly bypass this problem but it only works under specific conditions, e.g. large number of demonstration trajectories or access to expert query for optimal actions (Ross et al.] 2011). Furthermore, a policy learned from demonstrations only be optimal in the vicinity of the demonstrator trajectories and only for the initial conditions of such trajectories. Our work is at the intersection of these two classes of solutions, exploration and imitation, that we use only one trajectory from the demonstrator per problem to solve hard- exploration tasks] |
this way will
This approach has been explored before by Paine al. (0199) (for a thorough comparison see Section 5.2 We propose the first implementation based on on-policy algorithms, in particular PPO. Henceforth we refer to the version of the algorithm we put forward as PPO Demonstrations (PPO+D). Our contributions can be summarized as follows:
1. In our approach, we treat the demonstrations trajectories as they were actions taken by the agent in the real environment. We can do this because in PPO the policy only spec- ifies a distribution over the action space. We force the actions the policy to equal the demonstration actions instead of sampling from the policy distribution and in this way accelerate the exploration phase. We use importance sampling to account for sampling from distribution different than the policy. The frequency with which the demonstrations are sampled depends on an adjustable hyperparameter p. as described in Paine et al. (2019).
2. Our algorithm includes the successful trajectories in the replay buffer during training and treats them as demonstrations.
A video showing the experimental results is available at I.set ................................ tneCel cm

---

Under review as a conference paper at ICLR 2021
3. The non-successful trajectories ranked according to their maximum estimated value and are stored in a different replay buffer.
4. We mitigate the effect of catastrophic forgetting by using the maximum reward and the maximum estimated value of the trajectories to prioritize experience replay.
PPO+D is only in part on-policy as a fraction of its experience comes from a replay buffer and therefore was collected by older version of the policy. The importance sampling is limited to the action loss in PPO and does not adjust the target value in the value loss as in Espeholt et al. (2018).
We found that this new algorithm is capable of solving problems that are not solvable using nor- mal PPO, behavioral cloning, GAIL, nor combining behavioral cloning and PPO. PPO+D very easy to implement by only slightly modifying the PPO algorithm. Crucially, the learned policy is significantly different and more efficient than the demonstration used in the training.
To test this new algorithm created a benchmark of hard-exploration problems of varying levels of difficulty using the Animal-AI Olympics challenge environment (Beyret et al. 2019 Crosby et al. 2019). All the tasks considered have random initial position and the PPO policy uses entropy vegularization so that memorizing the sequence of actions the demonstration will not suffice to complete any of the tasks.
RELATED WORK
Different attempts have been made to use demonstrations efficiently in hard-exploration problems.
In Salimans & Chen (2018) the agent is able to learn a robust policy only using one demonstration. The demonstration is replayed for n steps after which the agent is left to learn on its own. By incrementally decreasing the number of steps n, the agent learns a policy that robust to randomness (introduced in this case by using sticky actions or no-ops (Machado et al.) 2018). since the game is fundamentally a deterministic one). However, this approach only works in fully deterministic environment since replaying the actions has the role of resetting the environment to a particular configuration. This method of resetting is obviously not feasible in a stochastic environment.
Ecoffet et al. (2019) is another algorithm that largely exploits the determinism of the environment by resetting to previously reached states. It works by maximizing the diversity of the states reached. is able to identify such diversity among states by down-sampling the observations, and by considering as different states only those observations that have a different down-sampled image. This works remarkably well in two dimensional environments, but is unlikely to work three-dimensional, stochastic environments.
Self-supervised prediction approaches, such as Pathak et al. (2017); Burda et al. (2018) Schmidhu- ber (2010); Badia et al. (2020b) have been used successfully in stochastic environments, although they are less effective in three-dimensional environments. Another class of algorithms designed to solve exploration problems are count-based methods (Tang et al. 2017] Bellemare et al. 2016 Os- trovski al. 2017). These algorithms keep track of the states where the agent has been before (if we are dealing with prohibitive number of states, the dimensions along which the agent moves can be discretized), and give the agent an incentive (in the form of bonus reward) for visiting new states. This approach assumes we have a reliable way to track the position of the agent.
An empirical comparison between these two classes of exploration algorithms was made in Baker et al. (2019), where agents compete against each other leveraging the use of tools that they learn to manipulate. They found the count-based approach works better when applied not only the agent position, but also to relevant entities in the environment (such as the positions of objects). When only given access to the agent position, the RND algorithm (Burda et al. 2018) was found to lead to higher performance.
Some other works focus on leveraging the use of expert demonstrations while still maximizing the reward. These methods allow the agent to outperform the expert demonstration in many problems. Hester et al. (2018) combines temporal difference updates the Deep Q-Network (DQN) algorithm with supervised classification of the demonstrator's actions. Kang et al. (0118) proposes to effec- tively leverage available demonstrations to guide exploration through enforcing occupancy measure matching between the learned policy and current demonstrations.
2

---

Under review as a conference paper at ICLR 2021
Other approaches, such Duan et al. (2017); Zhou et al. (2019) pursue meta-learning strategy where the agent learns to learn from demonstrations, such approaches are perhaps the most promis- ing, but they require at least demonstration for each task for a subset of all tasks.
Generative adversarial imitation learning (GAIL) (Ho & Ermon 2016) has never been successfully applied to complex partially observable environments that require memory (Paine et al. 2019). In- foGAIL (Li et al. 2017) has been used to learn from images but, unlike in this work the policy does not require recurrency to complete the tasks. Behavioral Cloning (BC) is the most basic imi- tation learning technique, it is equivalent to supervised classification of the demonstrator's actions RRahmatizadeh et al. 2018). Both GAIL and BC have been used in the Obstacle Tower Challenge Juliani et al. 2019), but are alone insufficient for solving hard-expe cerrddexllrattoon problems Sichol|2019
Perhaps the article that has the most similarity with this work is al. (2018). The authors present an off- off-policy algorithm that learns to reproduce the agent's past good decisions. Their work focuses mainly on advantage actor-critic, but the idea was tested also with PPO. Instead, PPO+D utilizes importance sampling, leverages expert (human) demonstrations, uses prioritized experience replay based on the maximum value of each trajectory to overcome catastrophic forgetting and the ratio of demonstrations replayed during training can be explicitly controlled.
3 DEMONSTRATION GUIDED EXPLORATION WITH PPO+D
Our approach attempts to exploit the idea of combining demonstrations with the agent's 's own ex- perience in an on-policy algorithm such as PPO. This approach particularly effective for hard- exploration problems. One view the replay of demonstrations agent in the current environment. This means that the only point we are interfering with the classi- cal PPO is when sampling. which is substituted by simply replaying the demonstrations. A crucial difference between PPO+D and R2D3 (Paine et al. 2019) is that we do consider sequences that contain entire episodes in them, and therefore using recurrency is much more straightforward. From the perspective of the agent it is as if it is always lucky when sampling the actions, and in doing so it is skipping the exploration phase. The importance sampling formula provides the average rewards over policy Th given trajectories generated from different policy Te (als):
possible trajectory of the
To (at (s:) To (sc)
Exgy [rc] = E
where r; = R(rlac, st) is the environment reward given state and action a at time t. Ex, indicates that the average is over trajectories drawn from the parameterized policy gо. The importance sam- pling term sor(aelss) accounts for the correction in the change of the distribution [e(a.)a))
actions in the
policy Tor (aels:). By maximizing Ens
Kar(ae ec) over the parameters 0' a new policy is obtained ma(aa||s)
that is on average better than the old one. The PPO loss function then defined as
I, CLIP-VV-S (0) R.
[ [TCLIP (0) +...
+ C2 (2S5T((s(, at)]
(2)
where
Tor (ar (tt) Ta(arls:)
LLIP (s, a. 0, 0) = min
A (st, ac), g(e, ((t,
(3)
and 6j and C2 are coefficients, L. is the squared-error loss (Vo(st) -V y2oro)2 AT is an estimate of the advantage function and S is the entropy of the policy distribution. The entropy term is designed help keep the search alive by preventing convergence to a single choice of output, especially when several choices all lead to roughly the same reward (Williams & Peng|1997
Let DD be the set of trajectories = (So, ao; 81, a1....) for which we have demonstration, then TD(aa (sz) = for (at, st) in D and ( otherwise. This is policy that only maps observations com- ing from the demonstration buffer to a distribution over actions. Such distribution assigns probability one to the action taken in the demonstration and zero to all the remaining actions. The algorithm decides where to sample trajectories from each time an episode is completed (for running out of time for completing the task). We define D to be the set of all trajectories that get replayed at any given time D = Dv U DR, where Dy are the trajectories collected prioritizing the value es- timate (V stands for value), and D (R stands for reward) contains the initial human demonstration
3

---

Under review as a conference paper at ICLR 2021
and successful trajectories the agent collects. The agent samples from the trajectories in Die with probability p Dy with probability & and from the real environment Eny with probability p Φ subject to + o 1.
WDR
if sampled from DR if sampled from Dy
The behavior of the policy can be defined as follows: T
&Dv
if sampled from Env
In PPO+D we substitute the current policy ne with the policy Ti 2. since this is the policy used to collect the demonstration trajectories. The clipping term is then changed to correct for it,
T (a,18t) A A (s), g(c. A (SL. ((t. (4) % (ac(s))
L SgUI--PO+DP(s,a0 min
Algorithm PPO+D
1: Initialize parameters 0
2: Initialize replay buffer Dv f-
} 3: Initialize replay buffer DR [D))
4: Initialize rollout storage + {}
5: for every update do
6: for actors 1,2...., N do
7: Sample from (Dv, Dn, Env)
8: i E DR then
9:
With probability P sample batch of demonstrations for steps 1,2....,T' do
10:
Replay a transition from buffer st, at, Te, 8t41 TD. (arlse) Store transition & + E {(st, at, r.) end for
12:
14:
else if 7 E Dv then
15:
With probability @ sample batch of demonstrations for steps 1, 2, .. do
16:
17:
Replay a transition from buffer 8t, at, rt, 8t41 ~ TDV (a:ls)) Store transition & < U [(st, a1,))
18:
19:
end for else if E Env then
20:
21:
With probability 1- p & collect set of trajectories by running policy ma for steps 1, 2....... T do
23:
Execute an action in the environment az, Tz, St+1 Te(as(s)) Store transition & E [(*t, at, rt)}
25:
end for end
26:
27:
Compute advantages estimates Å,, Ag...., ÅT end for
28:
29:
Optimize L PPO wrt 6, with K epochs and minibatches size M < NT
30:
8 0 - - L PPO
31:
Empty rollout storage E 0
32:
end for
3.1 SELF-IMITATION AND PRIORITIZED EXPERIENCE REPLAY
We hold the total size of the buffers [D] constant throughout the training. At the beginning of the training we only provide the agent with one demonstration trajectory, so [DRl = and DD =
1 as [D] + DRl. Dy is only needed at the beginning to collect the first successful trajectories. As (DRl increases and we have more variety of trajectories in DR we decrease the probability o of replaying trajectories from Dv. After [Dxl is large enough, replaying the trajectories from Dy is more of an hindrance. The main reason for treating these two buffers separately is to slowly anneal from one the other avoiding the hindrance in the second phase.
4

---

Under review as a conference paper at ICLR 2021
[Dv] and [Dx] are the quantities that are annealed according to the following formulas, each time new successful trajectory is found and added to DR
O
OU iDv 0
p+
-
Dvl = pDyl- 1; [Dal = [D
lo
where IDo and [ov are the maximum size respectively for DR and Dv and do is the value of di at the beginning of the training. We define the probability of sampling trajectory T2 P(i) Σ PE where P: = max, (st,, and a is a hyperparameter. We shift the value of pt for all the trajectories in Dy so as to guarantee pt 2 O. We only keep up to |Dy| unsuccessful trajectories at any given time.
pf
Values are only updated for the replayed transitions. Sue- cessful trajectories are similarly sampled from DR with a uniform distribution (a better strategy could be to sample ac- cording to their length), and the buffer is updated following a FIFO strategy. We introduced the value-based experience replay because that we could not solve by using self-imitation solely based on the reward. These are the tasks that the agent has trou- ble solving even once because the sequence of actions is too long or complicated. We prefer the value estimate as a selec- tion criteria rather than the more commonly used TD-error as we speculate
Eay
get to level of complexity in some tasks
Dp
m
Dr.
Ty
more effective at retaining promising trajectories over long periods of time. Crucially for the un- successful trajectories it is possible for the maximum value estimate not to be zero when the observations are similar to the ones seen in the demonstration.
Figure 1: Learner diagram. The PPO+D learner samples batches that are a mixture of demonstrations and the experiences the agent collects by interacting with the environment.
We think that this plays a role in countering the effects of catastrophic forgetting, thus allowing the agent to combine separately learned sub-behaviors in successful policy. We illustrate this with some example trajectories of the agent shown in the Appendix in Figure 7 The value estimate is noisy and consequence of that, trajectories that have a low maximum value estimate on first visits may not be replayed for a long time or never as pointed out in /Schall et al. (2015). However, for our strategy work it is enough for some of the promising trajectories to be collected and replayed by this mechanism.
EXPERIMENTS
4.1 THE ANIMAL-AI AI OLYMPICS CHALLENGE ENVIRONMENT
The recent successes of deep reinforcement learning (DRL) (Mnih et al. 2015 Silver et al. 2017; Schulman al. 2017 Schrittwieser al. 2019 Srinivas et al. 2020 have shown the potential of this field, but at the same time have revealed the inadequacies of using games (such as the ATARI games (Bellemare al. 2013)) as benchmark for intelligence. These inadequacies have motivated the design of more complex environments that will provide a better measure of intelligence.
The Obstacle Tower Environment (Juliani et al., 2019), the Animal AI Olympics (Crosby et al. 2019) the Hard-Eight Task Suite (Paine et al. 2019) and the DeepMind Lab (Beattie et al. (016) all exhibit sparse rewards, partial observability and highly variable initial conditions. For all the tests use the Animal-AI Olympies challenge environment. The aim of the Animal-AI Olympics is to translate animal cognition into a benchmark of cognitive AI (Crosby et al., 2019).
The environment contains agent enclosed in a fixed size arena. Objects can spawn in this arena, including positive and negative rewards (green, yellow and red spheres) that the agent must obtain or avoid. This environment has basic physics rules and set of objects such as food, walls, negative- reward zones, movable blocks and more. The playground can be configured by the participants and they can spawn any combination of objects in preset or random positions. We take advantage
5

---

Under review as a conference paper at ICLR 2021
of the great flexibility allowed by this environment to design hard-exploration problems for experiments.
[a] One box nany
(b) One bax hard
(c) Two boxes
[d] Two boxes hard
(e) Agent view
Figure 2: Tasks. In each of the tasks there is only one source of reward and the position of some of the objects is random, each episode is different. The agent has no access to the acrial view, instead partially observes the world through a first person view of the environment. All of the tasks are either inspired or adapted from the test set in the Animal-AI Olympics competition.
4.2 EXPERIMENTAL SETTING
We designed experiments in order to answer the following questions: Can the agent learn policy in a non-deterministic hard-exploration environment only with one human demonstration? Is the agent able to use the human demonstration to learn to solve problem with different initial conditions (agent and boxes initial positions) than the demonstration trajectory? How does the hyperparameter influence the performance during training? The four tasks used to evaluate the agent are described as follows:
One box easy
The agent has to move a single box, always spawned at the same position, in order to bridge gap and be able access the reward once it climbs up the ramp (visible in pink). The agent can be spawned in the range (X 0.5 39.5, Y 0.5 39.5) if object is not already present at the same location (Fig. 2).
One box hard
The agent has to move a single box in order to bridge gap and be able to access the reward, this time two boxes spawned at any of four positions A (X 10, 10), B (X 10, Y 30), C (X 30, Y 10), D (X 30, Y 30). The agent can be spawned in the range (X 0.5 39.5, Y 0.5 39.5) if object is not already present at the same location (Fig. (b).
Two boxes easy
The agent has to move two boxes in order to bridge larger gap and be able to access the reward, this time two boxes are spawned any of four positions (X 10, 10),
(X : 10, Y : 30), C (X 30, Y 10), D (X 30, Y 30). The agent can be spawned in
the range (X 15.0 20.0, Y 0.5 15.0) if an object not already present at the same
location (Fig. 2-).
Two boxes hard
The agent has move two boxes in order to bridge larger gap and be able to access the reward. Two boxes are spawned at two fixed positions
(X 10, 10), B (X 10,Y 30). A wall acts as barrier in the middle of the gap, prevent the agent from "surfing" a single box. The agent can be spawned in the range (X 15.0 - 20.0, 5.0 10.0) if an object is not already present the location (Fig 21)

---

Under review as a conference paper at ICLR 2021
5 RESULTS
5.1 COMPARISON WITH BASELINES AND GENERALIZATION
In Figure 3 we compare PPO+D with parameters p = 0.1, A = 0.3 to the behavioral cloning base- lines (with 100 and 10 demonstrations), to GAIL (with 100 demonstrations) and with PPO+BC. PPO+BC combines PPO and BC in an a similar way to PPO+D: with probability a sample is drawn from the demonstrations and the policy is updated using the BC loss. The value loss function remains unchanged during the BC update.
We test the GAIL implementation on a simple problem to verify it working properly (see Sec- tionC] in the Appendix). For behavioral cloning we trained for 3000 learner steps (updates of the policy) with learning rate 10-3 3 It is clear that PPO+D is able to outperform the baseline in all four problems. The performance of PPO+D varies considerably from task to task. This reflects the considerable difference in the range of the initial conditions for different tasks. In the tasks "One box easy" and "Two boxes hard" only the position of the agent sets different instances of the task apart. The initial positions of the boxes only play a role in the tasks "One box hard" and "Two boxes easy". Under closer inspection we could verify that in these two tasks the policy fails to generalize to configurations of boxes that are not seen in the demonstration, but does generalize well in all tasks for very different agent starting positions and orientations.
This could be because there is a smooth transition between some of the initial conditions. Due to this fact, if the agent is able to generalize even only between initial conditions that are close it will be able to progressively learn to perform well for all initial conditions starting from one demonstration. In other words the agent automatically using a curriculum learning strategy, starting from the initial conditions that are closer to the demonstration. This approach fails when there is an abrupt transition between initial conditions, such as for different boxes configurations.
During training we realized that the policy "cheated" in the task "Two boxes easy" as it managed to "surf" one of the boxes, in this way avoiding to move the remaining box (a similar behavior was reported in Baker et al. (2019). Although this is of itself remarkable, we were interested in testing the agent for a more complex behavior, which it avoids by inventing the "surfing" behavior. introduced "Two boxes hard". We
To make sure it is capable of such more complex behavior decided to reduce the range of the initial conditions in this last task, as we already verified that the agent can learn from such variable conditions in tasks "One box hard" and "Two boxes easy". This last experiment only tests the agent for a more complex behaviour. In the tasks "One box hard" and "Two boxes easy" the agent could achieve higher performance given more training.
We emphasise that PPO+D is designed to perform well on hard-exploration problems with stochas- tic environment and variable different conditions, we tested it on the of the Atari environment "BreakoutNoFrameskipyyy and conclude that it does not lead to better performance than vanilla PPO when the reward dense. PPO+D also learns complete the task although more slowly than PPO.
5.2 THE ROLE OF & AND THE VALUR-BUFFER Dy
In Figure we mainly compare PPO+D with p 0.1, 0.3 to p the second parameter configuration does not use the buffer Dv. It is evident in the task "Two boxes easy" that Dy is essential for solving harder exploration problems. In the "One box easy" task we can see that occasionally on easier tasks not having Dy can result in faster learning. However, this comes with a greater variance in the training performance of the agent, as sometimes completely fails to leam.
0.0. Crucially,
In Figure 4 we also run vanilla PPO (p = 0.0, = 0.0) on the task "One box easy" and establish its inability to solve the task even once on any seed and any initial condition. This being the easiest of all tasks, we consider unlikely the event of vanilla PPO successfully completing any of the other harder tasks. We defer an ablation study of the parameter to section B in the appendix.Althouge we can not make a direct comparison with Paine et al. (209)) 9) we think it is useful to underline some of the differences between the two approaches both the problem setting and in the performance. We attempted to recreate the same complexity of the tasks on which Paine et al. (2019) was tested. In the article, variability is introduced in the tasks multiple dimensions such position and
7

---

Under review as a conference paper at ICLR 2021
orientation of the agent, shape and color of objects, colors of floor and walls, number of objects in the environment and position of such objects. The range of the initial conditions for each these factors was not reported. In our work as well as the initial positions of the boxes. As for the number of demonstrations in Painee et al. (2019) the agent has access to hundred demonstrations, compared to only one in work. In the present article, the training time ranges from under 5 millions frames to 500 millions, whereas in Paine al. (2019) it ranges from 5 billions 30. Although we adopted the use of the parameter p from Paine et al. (2019) its value differs considerably, which we attribute to the difference between the two algorithms: one being based on PPO, the other on DQN.
change the initial position and orientation of the agent
P10 BCCCC meee bchursee Lheinn 250 batavional cloning to GAR, SCD
Retavional befustrul Lieiing
detavioeal coringg betuebeed clanting
bcheried Chering 22
aispppp (90
wAsy
masy
Figure 3: Experiments. Performance of behavioural cloning with ten and hundred recorded human demonstrations and PPO+D with 0.1, ø = 0.3 and just one demonstration. The curves represent the mean, min and max performance for each of the baselines across different seeds. The BC agent sporadically obtains some rewards. GAIL with a hundred demonstrations never achieves any reward. PPO+BC has only access to one demonstration, like PPO+D. It occasionally solves the task but it is unable to archive high performance.
0 0:3
p:01:::030
p- O.D. -.0. 9-000 meze
www.
0-05. 0.0
0--00.000000000 mage
stass
One box essy
Two booss easy
Figure 4: Experiments. Performance for vanilla PPO (p = 0.0, = 0.0), PPO+D with 0.5, Φ 0.0 and PPO+D with p 0.1, 0.3 the tasks "One box easy" and "Two boxes easy" using a single demonstration. Some of the curves overlap each other as they receive zero or close to zero reward. Vanilla PPO never solves the task.
6 CONCLUSION
We introduce PPO+D, an algorithm that uses a single demonstration to explore more efficiently in hard-exploration problems. We further improve on the algorithm by introducing two replay buffers: one containing the agent own successful trajectories as it collects these over the training and the second collecting unsuccessful trajectories with a high maximum value estimate. In the second buffer the replay of the trajectories is prioritized according to the maximum estimated value. We show that training with both these buffers solves, to some degree, all tasks the agent was presented with. We also show that vanilla PPO well PPO+D without the value-buffer fails learn
8

---

Under review as a conference paper at ICLR 2021
the same tasks. In the article, justify the choice of such adjustments measures counter catastrophic forgetting, a problem that afficts PPO particularly. The present algorithm suffers some limitations currently it fails to generalize to all variations of some of the problems, yet it achieves to solve several very hard exploration problems with a single demonstration. We propose to address these limitations in future work.
REFERENCES
Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. arXiv preprint SttI::200031333500 2020a.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint wxXiv22022060036 2020b.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint 1707:219,,,07528, 2019.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, al. Deepmind lab. arXiv preprini arXiv: 1612.03801, 2016.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-14799 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Benjamin Beyret, José Hermández-Oralle Lucy Cheke, Marta Halina, Murray Shanahan, and Matthew Crosby. The animal-ai environment: Training and testing animal-like artificial cog- nition. arXiv preprint arXiv: 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 17X.v:1810.1284/ 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Matthew Crosby, Benjamin Beyret, and Marta Halina. Intelligence, 16):257,2011
The animal-ai olympics. Nature Machine
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, pp. 1087-1098, 2017.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv: 10995, 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: /80.00/561, 2018
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations, In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4573 2016.

---

Under review as a conference paper at ICLR 2021
Arthur Juliani, Ahmed Khalifa, SincentPPiere Berges, Jonathan Harper, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vi- sion, control, and planning. arXiv preprint arXi::1902.011330 2019.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations, In International Conference on Machine Learning, pp. 2474-2483, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69800 6980, 2014.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms.
https://
github. www/ cctam///ksstt.oo///rrrrroooboeee ottp 2018.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual
demonstrations. In Advances in Neural Information Processing Systems, pp. 3812-3822, 2017.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Alex Nichol.
Competing in the obstacle tower challenge. tettrooomppiiiiiiititieeeeeeeeee www.----bbssacee---ee---llllllee 2019.
https://blog.aqniie
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint arXiv:1806.000 05635, 2018.
Georg Ostrovski, Mare G Bellemare. Aäron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning-Voll Volume 70, pp. 2721-2730. JMLR. org, 2017.
Tom Le Paine, Caglar Gulcehre, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. Making effi- cient use of demonstrations to solve hard exploration problems. arXiv preprint arXiv:1909.01387 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Rouhollah Rahmatizadeh, Pooya Abolghasemi. Ladislau Bölöni, and Sergey Levine. Vision-based multi-tast task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3758-3765. IEEE, 2018.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statisties, pp. 627-635,70 2011.
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from single demonstration. arXiv preprint arXiv: 023383, 2018
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 151 1...55...00522... 2015.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions Autonomous Mental Development, 2(3):230-247, 2010.
Julian Schrittwieser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with leamed model. arXiv preprint arXiv: 1.02655, 2019.
10

---

Under review as a conference paper at ICLR 2021
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-337 2017.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arXiv:2004.04136, 2020.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurek, and Pieter Abbeel. Exploration: study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753-2762, 2017.
Ronald J Williams and Jing Peng. Function optimization using connectionist reifoorrmmt learning algorithms. Connection Science, 3(3):241-268, 1991.
Allan Zhou, Eric Jang, Daniel Kappler, Alex Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward. arXiv preprint 1rX.::1906.03332,, 2019.
A TRAINING DETAILS
For the training we used parallel environments and we compute the gradients using the Adam
optimizer (Kingma & Ba 2014) with fixed learning rate of 10-5, The agent perceives the environ- ment through a 84 by 84 RGB pixels observations in a stack of 4. At each time-step the agent is
allowed take of nine actions. We use the network architecture proposed in Kostrikov (2018) which includes gated recurrent unit (GRU) (Cho et al.) 2014) with hidden layer of size 256. We ran the experiments on machines with 32 CPUs and 3 GPUs, model GeForce RTX 2080 Ti. The experiments where carried out with the following hyperparameters.
Table 1: Model and PPO Hyperparameters
Parameter
Value
clip-param
0.15
gamma
0.998
frame-skip
frame-stack
num-steps
1000
num-mini-batel
entropy-coet
0.02
value-loss-coef
0.1
num-processes
14
Ir (learning rate)
le-5
eps (RMSprop optimizer epsilon)
le-5
alpha (RMSprop optimizer apha)
0.99
gae-lan
0.95
max-grad-norm ppo-epoch
0.5
For the training we performed no hyperparameter search over the replay ratios & and p but set them to a reasonable number. We found other configurations of these parameters to sometimes more efficient in the training, such for example setting p 0.5 and ø 0.0 in the task "One box easy". The parameters we ran all the experiments with have been chosen because they allow to solve all of the experiments with demonstration.
11

---

Under review as a conference paper at ICLR 2021
In computing the probability of trajectory to be replayed P(i)
10. The total buffer
E.
size is |P] = 51 with [Dv = 50 plus the human generated trajectory. [DRlo = 51 meaning once the agent collects 50 successful trajectories, new successful trajectories overwrite old ones, following FIFO strategy and no trajectory is replayed from the value-buffer.
The implementation used is based on the repository Kostrikov (2018). On our infrastructure we could train at approximately speed of 1.3 millions frames per hour.
The code, pre-trained models, data-set of arenas used for training, well video-clips of the agent performing the tasks are available at https::/dol: wwg/10.....oo..o.ooooooo.... 000/://100000//r// 6084/m9, figshare. 12459602.
B HYPERPARAMETERS ABLATION
In this section we present the results of four different experiments on variation of the "One box easy task" where the agent position does not change across episodes and it is shared with the demonstra- tion. We test on this variation because performance. We only perform the ablation study on p because is harder to test: it is indispensable for solving difficult tasks but it can slow down the performance on easy tasks. This being an easy task, the results obtained, do provide any insights on the effect of in harder problems shown in Figure
one of the simplest problems we can use to test PPO+D
The following figure shows the performance of the PPO+D algorithm where the o parameter is changed and Ø = 0. Interestingly we observe that, among the values chosen, the performance peaks for o 0.3. We hypothesize that lower o values have worse performance because the interval between demo replays is so large that allows the network to forget the optimal policy learned with the demonstrations. On the other side, higher values of p are even more counterproductive as they prevent the agent from learning from its own experience, most critically learning what not to do.
Success ratee
03. p mea
uo. p. moon
07.07..-- 3.
0+
steps (M)
Memorize trajectory task
Figure 5: Ablation study Performance for PPO+D with 0.1, 0 0.0, p 0.3, 0.0, 0.5, á = 0.0 and p 0.5, ó 0.0 and PPO+D with p 0.7, o = 0.0, on a variation of the "One box easy" task were the initial position of the agent fixed. The curves represent the mean, min and max performance for each of the baselines across 3 different seeds.
C GAIL TEST
To verify the correctness of our GAIL implementation for the experiments in Figure 4 test it on a simple task in the Animal-Al environment. The task is shown in Figure it consists in collecting green food of random size and position.
12

---

Under review as a conference paper at ICLR 2021
Figure 6: Food collection task. In this task the the agent is spawned into the arena with one green ball. The green food size and position are set randomly at the beginning of each episode. The episode ends when the green food is collected.
Our implementation of GAIL based on Li et al. (017) was trained with the following hyperparam- eters besides the PPO parameters in Table I
Table GAIL Hyperparameters
Parameter
Value
scaling- factor
0.001
gail-epoch
0.4
gail-batch-size 200
Table 3: Performance on the "Food collection task"
Method
Avg. Success rate
Std.
GAIL
0.997 0.617
0.045
BC
0.487
In Table report the performance of both GAIL and behaviour cloning the simple task. We note that although both methods achieve reasonable performance, the agent trained with GAII. reaches near-perfect performance, whereas the BC agent performance tends fluctuate significantly. The GAIL policy was trained for 120 millions fime-frames and behavioral cloning we trained for 3000 learner steps (updates of the policy) with learning rate 10-5
13

---

Under review as a conference paper at ICLR 2021
D ANALYSIS OF THE EFFECT OF THE VALUE-BUFFER
WILLE CAOT 4/2 THE ELEREeeS
WA THE 6O 1S008?
(a) Learning about climbing up the ramp.
(b) Learning about pushing boxes.
(c) Focusing boxes.
WWL.::1 THE di 617451
VALIE 7156603 TINT 11030
VALUE DISTT6 23
(d) Learning about jumping boxes.
(e) Learning how solve the task pushing two boxes.
(f) Learning how solve the task with only one
Figure 7: Sub-behaviors. Trajectories the agent played on the task "Two boxes easy". In each of the figure the upper part shows the movements of the agent on the X-Y plane while the lower part shows the movement on the X-Z plane. The images are ordered by the time they were executed in the training in millions of frames.
The value-buffer experience replay creates an incremental curriculum for the agent to learn, keeping different trajectories that achieved an high maximum value the buffer incentives agent to combine these different sub-bchaviours e.g. pushing the blocks and going up the ramp.
14


Reviews:
[{"review_id": "n3GpgOTxq1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors introduce a simple extension of PPO that uses a single expert demonstration and a modified sampling algorithm to show better performance than vanilla PPO in a difficult 3D setting.\n\n## Strengths & Weaknesses\n\n#### Strengths\n\n- The paper is well-written and the method is simple, yet powerful. Learning something from a single demonstration is no easy feat.\n- The videos shown in the submission look impressive in how difficult the setup is and how the agent manages to learn complex strategies like.\n- Overall, the paper is very \"short & sweet\" in that it's not a groundbreaking new technique, but a small change to PPO but it's well explained, and the results that _are_ in the paper are good.\n  \n#### Weaknesses\n\n- The main problem I have with this is actually the lack of further experiments. For such an easy extension of PPO, I would've expected you to have no problem running this on an Atari environment and at least one MuJoCo environment too (where it's also easy to gather a human demonstration, like controlling the reacher via inverse-kinematics or the tricky Pusher). Compared to vanilla PPO we should see improvements across the board, no?\n- Similarly, you just arrived at the hyperparameters $p = 0.1, \\phi=0.3$ without explanation or ablation. Do you maybe want to justify how these parameters came to be and what happens if either parameter is higher or lower?\n\n**TL;DR how to make me raise my score:** Include at least one Atari and one Mujoco/Robot environment (since Ilya Kostrikov's implementation that you use supports these out of the box) and either add an ablation study on a single env over p/phi or explain the importance of these values.\n\n## Impact & Recommendation\n\nThis is fundamentally good material. It's not groundbreakingly new but I think it could make for an easy-to-use imitation learning baseline that would help in a lot of scenarios get the method off the ground. However, the current paper doesn't show the rigour and depth of analysis that I would expect from an ICLR paper. I hope the authors can make up for that in the rebuttal week and then I'm happy to up my score. Currently, my recommendation is borderline. If the present method was really an all-around improvement over PPO, why did the authors not show it on a tried-and-true OpenAI Gym task but only in their own made-up setting? \n\n## Minor Nitpicks\n\n- I'd report a few more seeds - I think 5 seeds is a good starting point.\n- Your plotting of runs is uncommon - usually, you either plot the mean and standard deviation or the mean and min/max. \n- In Fig. 3 and 4, the fonts in the legend need to be bigger \n- On page 6 you're twice in a row weirdly enthusiastic for Unity-ML / the \"flexibility allowed by this environment\". These are odd things to say in a research paper unless you're an employee of Unity\n- Algorithm 1 is a bit verbose but great. Makes it very clear. On the flip side of that, Figure 1 is a bit redundant. These two things communicate the same idea and I like Algo 1 better.\n- There are a few missing commas, like \"In our approach,\", bottom of the first page.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HFTVcaE4vCG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper looks at problems that have sparse rewards, are partially observable, and have\nvariable initial conditions.  Previous work [R2D3, Paine et al 2019] tackles this problem using \noff-policy recurrent Q-learning. Instead, this work proposes to use a PPO+D, an on-policy method (PPO)\nfor a policy with memory (GRUs). This relies on storing successful trajectories (demonstration(s) \nand discovered), and replaying these as experiences using importance sampling. Other details\nsuch as a value-prioritized buffer, annealed use of that buffer, an entropy term, and more\nare given.  The method is tested on four environment variations, as constructed in the Animal-AI Olympics challenge env.\nExact comparisons with R2D3 are not possible due to missing initial condition ranges. \nIt is compared to a behavior cloning baseline, vanilla PPO, and a value-buffer ablation,\nIt is shown that it can succeed with just one demonstration, although it fails to generalize\nto all many of the variations for the problems.\n\n## Strengths\n- interesting problem setting (although this is not novel, e.g., R2D3)\n- interesting exploration beyond R2D3, i.e., to on-policy methods\n- use of value buffer as a means to allow for progress on harder problems\n\n## Weaknesses\n- could have compared against an implementation of some facsimile of R2D3\n- lack of generalization to wider initial conditions means that the results may not be as interesting as they first seem\n- behavior cloning is, unsurprisingly, a poor baseline for a problem with variable initial conditions and with partial observability\n- the main result is really: \"sometimes it works!\".  That is actually of interest, although really a \"proof by example\" result.\n- only tested on limited tasks, although similar to R2D3.  Thus it is unclear how generalizable the method and results will be;  it may overfit in some respects to the task at hand.\n\n## Recommendation\nOverall, while the results come with some real caveats, the paper may inspire further work in the use of demonstrations in sparse-reward settings, in conjunction with on-policy methods such as PPO. The use of the value-prioritized buffer is also a feature\nthat could see further adoption. \n\n## Questions\n- why could behavior cloning realistically be expected to work as a baseline, given the variable initial states?\n\n## Additional Feedback\n- DR and DV: it's unclear what the 'R' stands for in DR.  It's always helpful to define the origins of the chosen nomenclature for cases where it is not obvious.\n- please tell the reader where the value estimates come from for the value buffer for unsuccessful trajectories.  Why aren't these all simply zero?\n- Figure 1 is really helpful;  place it earlier in section 3?", "extra_scores": {}, "extra_text": {}}, {"review_id": "BuarVHb8YRb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposes a modification of the PPO algorithm which can accommodate a single task demonstration with the goal of faster learning in sparse-reward tasks. There are 4 tasks proposed by the paper, inspired by Animal-AI Olympics. The baselines are Behavioural Cloning (BC) and vanilla PPO. The proposed method outperforms those baselines.\n\n\n##########################################################################\n\nReasons for score:\n\nFrom the text it seems like BC baseline doesn't have access to rewards. If that is the case, the baselines are very weak - they either have access to demonstrations only or sparse rewards only.\n\nThe most basic fair baseline would be doing BC gradient updates to policy network from time to time during PPO training - should be a reasonable amount of work to implement. Ideally, comparison to relevant prior work should be at least attempted: Paine et al (R2D3) or some other whichever is easier to implement (see below for related work comments).\n\n\n##########################################################################\n\nPros:\n\n1. Nice research direction: combining demonstrations with sparse rewards.\n\n##########################################################################\n\nCons:\n\n1. More baselines are needed to evaluate the value of the proposed method. Prior work does exist (e.g., Paine et al.) and warrants more extensive comparisons.\n2. Related work needs to mention some relevant papers:\n- \"One-Shot Imitation Learning\" https://arxiv.org/pdf/1703.07326.pdf\n- \"Watch, try, learn: meta-learning from demonstrations and rewards\" https://arxiv.org/pdf/1906.03352.pdf\n3. There are some established benchmarks for sparse-reward tasks - it might be more productive to attack those as they already have some baselines. Introducing new tasks requires some discussion on why the established tasks are unsuitable for the goals of the paper. For example:\n- Vizdoom navigation by Pathak et al https://arxiv.org/pdf/1705.05363.pdf\n- https://openai.com/blog/ingredients-for-robotics-research/\n- https://aihabitat.org/\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was quite a bit of internal discussion on this paper. To summarize:
- The idea is very neat and interesting and likely to work
- The paper is likely to inspire future work
- There are still serious doubts  about the experimental evaluation that is not entirely up to par with current standards
  - The reviewers were not convinced 100% by the arguments about the 'custom' environments
  - The reviewers were not convinced 100% that the baselines were given their best shot

While the paper has potential to provide valuable input for the community, it needs a bit more work before being presentable at a highly competitive venue like ICLR.

Decision: Reject