Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
Authors: Caglar Gulcehre
Tom Le Paine
Bobak Shahriari
Misha Denil
Matt Hoffman
Hubert Soyer
Richard Tanburn
Steven Kapturowski
Neil Rabinowitz
Duncan Williams
Gabriel Barth-Maron
Ziyu Wang
Nando de Freitas
Worlds Team
Venue:  (2020)
Keywords: imitation learning
deep learning
reinforcement learning

Abstract: This paper introduces R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce a suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration.

Published as a conference paper at ICLR 2020
MAKING EFFICIENT USE OF DEMONSTRATIONS TO SOLVE HARD EXPLORATION PROBLEMS
Caglar Gulcehre", Tom Le Paine,, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer. Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, Gabriel Barth-Maron, Ziyu Wang, Nando de Freitas, Worlds Team DeepMind
ABSTRACT
This paper introduces R2D3, agent that makes efficient use of demonstrations solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the methods (both with and without demonstrations) fail see even single successful trajectory after tens of billions of steps of exploration.
INTRODUCTION
Reinforcement learning from demonstrations has proven to be an effective strategy for attacking problems that require sample efficiency and involve hard exploration. For example, |Aytarr et al. (2018). Pohlen et al. (2018) and Salimans and Chen (2018b) have shown that RL with demonstrations can address the hard exploration problem Montezuma's Revenge. VVčeríí al. (2017), |errel et al.) (2017) and Paine et al. (2018) have demonstrated similar results in robotics. Many other works have shown that demonstrations can accelerate learning and address hard-exploration tasks (e.g. see Hester
ct al. 2018 Kim et al. 2013 Nair et al. 2018 Kang al. 2018).
In this paper, we attack the problem of learning from demonstrations hard exploration tasks in partially observable environments with highly variable initial conditions. These three aspects together conspire to make learning challenging:
1. Sparse rewards induce diffieult exploration problem, which is challenge for many state of the art RL methods. An environment has sparse reward when a non-zero reward is only seen after taking long sequence of correct actions. Our approach is able to solve tasks where standard methods run for billions of steps without seeing a single non-zero reward.
2. Partial observability forces the use of memory, and also reduces the generality of informa- tion provided by a single demonstration, since trajectories cannot be broken into isolated transitions using the Markov property. An environment has partial observability if the agent can only observe a part of the environment at each timestep.
3. Highly variable initial conditions (i.c. changes in the starting configuration of the envi- ronment in each episode) are a big challenge for learning from demonstrations, because the demonstrations
not account for all possible configurations. When the initial conditions are fixed it is possible to be extremely efficient through tracking (Aytar et al. [0188] Peng et al. 2018); however, with a large variety of initial conditions the agent is forced general- ize over environment configurations not present in demonstrations. Generalizing between different initial conditions is known be difficult GGhss 2017 Zolna et al. 2019).
Langlois et 2019
Our approach to these problems combines demonstrations with off- off-policy, recurrent Q-learning in a way that allows us to make very efficient use of the available data. In particular, we vastly outperform behavioral cloning using the same set of demonstrations in all of our experiments.
indicates joint first authorship, both authors equally contributed to this project.

---

Published as a conference paper at ICLR 2020
Another desirable property of our approach is that agents able learn to outperform the demonstrators, and in some cases even to discover strategies that the demonstrators were not aware of. In one of our tasks the agent is able discover and exploit bug in the environment in spite of all the demonstrators completing the task in the intended way.
Learning from small number of demonstrations under highly variable initial conditions is not straight-forward. We identify a key parameter of our algorithm, the demo-ratio, which controls the proportion of expert demonstrations vs agent experience each training batch. This hyper-parameter has dramatic effect on the performance of the algorithm. Surprisingly, we find that the optimal demo ratio is very small (but non-zero) across a wide variety of tasks.
The mechanism our agents use to efficiently extract information from expert demonstrations is to use them in a way that guides (or biases) the agent's own autonomous exploration of the environment. Although this mechanism is not obvious from the algorithm construction, our behavioral analysis confirms the presence of this guided exploration effect.
To demonstrate the effectiveness of our approach we introduce suite of tasks (which we call the Stard-Eight suite) that exhibit our three targeted properties. The tasks are set procedurally- generated 3D world, and require complex behavior (e.g. tool use, long-horizon memory) from the agent to succeed. The tasks are designed to be difficult challenges in our targeted setting, and several state of the art methods (themselves ablations of our approach) fail to solve them.
The main contributions of this paper are, firstly we design a new agent that makes efficient use of demonstrations to solve sparse reward tasks in partially observed environments with highly variable initial conditions. Secondly, we provide an analysis of the mechanism our agents use to exploit information from the demonstrations. Lastly, we introduce suite of eight tasks that support this line of research.
RECURRENT REPLAY DISTRIBUTED DQN FROM DEMONSTRATIONS
We propose new agent, which we refer to as Recurrent Replay Distributed DQN from Demon- strations (R2D3). R2D3 is de- signed to make efficient use of demonstrations to solve sparse reward tasks in partially ob- served environments with highly variable initial conditions. This section gives overview of the agent, and detailed pseudocode can be found in Section 2.1
prioritaed sompling
training
batch
a]
prioriizedd
leamer double learning n-step sarget
updisoud priorituss
upcuted
demo replay
peior
agent replay
natworkk matoverkoweight
actor
agant
agant trajectioios 5 initsppriortte
The architecture of the R2D3 agent is shown in [ There are several actor processes, each running independent copies of the behavior against an instance of the environment. Each actor streams its experience to shared agent replay buffer, where experience from all actors aggregated and globally prioritized (Schaul et al. 2016 Horgan et al.) (2018) using a mixture of max and mean of the TD-errors with priority exponent 7 = 1.0 as in Kapturowski et al. (2018). The actors periodically request the latest network weights from the learner process in order update their behavior.
The R2D3 distributed system diagram. The learner samples batches that are a mixture of demonstrations and the experiences the agent generates by interacting with the environment over the course of training. The ratio between demos and agent experiences is key hyper- pprrmmrreeterr which must be carefully tuned to achieve good performance.
Figure 1:
In addition to the agent replay, we maintain a second demo replay buffer, which is populated with expert demonstrations of the task to be solved. Expert trajectories are also prioritized using the scheme of Kapturowski et al.] (2018). Maintaining separate replay buffers for agent experience and expert demonstrations allows to prioritize the sampling of agent and expert data separately.
2

---

Published as a conference paper at ICLR 2020
The learner process samples batches of data from both the agent and demo replay buffers simulta- neously. A hyperparameter p. the demo ratio, controls the proportion of data coming from expert demonstrations versus from the agent's own experience. The demo ratio is implemented at batch level by randomly choosing whether to sample from the expert replay buffer independently for each element with probability p. Using stochastic demo ratio in this way allows that are smaller than the batch size, which we found to be very important for good performance. The objective optimized by the learner uses of n-step, double Q-learning (with n 5) and a dueling architecture (Wang et al. 2016;] Hessel et al. 2018). In addition to performing network updates, the learne is also responsible for pushing updated priorities back the replay buffers.
target demo ratios
In each replay buffer, we store fixed-length (m = 80) sequences of (s, r) tuples where adjacent sequences overlap by time-steps. The sequences never cross episode boundaries. Given a single batch of trajectories we unroll both online and target networks (Mnih et al. 2015) on the same sequence of states to generate value estimates with the recurrent state initialized zero. Proper initialization of the recurrent state would require always replaying episodes from the beginning. which would add significant complexity to our implementation. As an approximation of this we treat the first 40 steps of each sequence burn-in phase, and apply the training objective to the final 40 steps only. An alternative approximation would be to store stale recurrent states in replay, but we did not find this to improve performance over zero initialization with burn-in.
2.1 R2D3 AGENT
In this section, we provide the pseudocode for the R2D3. First, the agent has single learner process which samples from both demonstration and agent buffers in order update its policy parameters, the pseudocode of the R2D3 learner can be found in Algorithm
Algorithm Learner
Inputs: replay of expert demonstrations D, replay of agent experiences R. batch size B, sequence length m. and number of actors A. Initialize policy weights
Initialize target policy weights +
Launch A actors and replicate policy weights 0 to cach actor for nateps do
Sample transition sequences (Stat4m a T(tt+m) from replay D with probability p or from replay R with probability (1 p). to construct mini-batch size B.
Calculate loss using target network.
Perform gradient descent step to update 0.
t mod ferrget 0. update the target policy weights 6' + 0.
If t mod factor O. replicate policy weights to the actors.
end for
The R2D3 agent has A parallel actor processes which interact with a copy of the environment in order to obtain data which is then inserted into the agent buffer. The agents periodically update their parameters match those being updated on the learner. The pseudocode for the actors is provided in Algorithm
Algorithm 2 Actor
repeat
Sample action from hehavior policy a +- (s)
Execute and observe a' and r
Store (s, a, 3 .) in R
until learner finishes.
3 BACKGROUND
Exploration remains one of the most fundamental challenges for reinforcement learning. So-called "hard-exploration" domains those in which rewards sparse, and optimal solutions typically have
3

---

Published as a conference paper at ICLR 2020
baseball
drawbridge
navigate cubes
push blocks
remember sensor
throw across
wall sensor
wall sensor stack
suite. each task an agent (V) must interact with objects in environment order to gain access to a large apple (V) that provides reward. The 3D environment is also procedurally generated so that every episode the state of the world including object shapes, colors, and positions is different. From the point of view of the agent the environment is partially observed. Because it may take hundreds of low-level actions collect an apple the reward is sparse which makes exploration difficult.
Figure Hard-EEight
long and sparsely- rewarded trajectories. Hard-exploration domains may also have many distracting dead ends that the agent may not be able to recover from once it gets into a certain state. In recent years, the most notable such domains Atari environments, including Montezuma's Revenge and Pitfall (Bellemare et al. 2013). These domains are particularly tricky for classical RL algorithms because even finding single non-zero reward to bootstrap from is incredibly challenging.
A common technique used to address the difficulty of exploration is to encourage the agent to visit under-explored areas of the space (Schmidhuber [99I). Such techniques are commonly known as intrinsic motivation (Chentanez et | 2005)) or count-based exploration (Bellemare et al. 2016). However, these approaches do not scale well as the state space grows, as they still require exhaustive search in sparse reward environments. Additionally, recent empirical results suggest that these methods do not consistently outperform e-greedy exploration (Taiga et al.] 2019). The difficulty of exploration is also consequence of the current inability of our agents abstract the world and learn scalable, causal models with explanatory power. Instead they often use low-level features or handcrafted heuristies and lack the generalization power necessary to work
abstract space. Hints can be provided to the agent which bias it towards promising regions the state space either introducing sequence of curriculum tasks (Bengio et al.) 2009 Graves et al. 2017). However, these approaches can be difficult to specify and, in the case of reward shaping, olten lead unexpected behavior where the agent learns exploit the modified rewards.
via reward-shaping (Ng al. 1999)
Another hallmark of hard-expa cerr-exxlorttion benchmarks is that they tend to fully-observable and exhibit little variation between episodes. Nevertheless, techniques like random no-ops and "sticky actions" have been proposed to artificially increase episode variance Atari (Machado et al.) 2018), an alternative is to instead consider domains with inherent variability. Other recent work on the Obstacle Tower challenge domain (Juliani et al.) 2019) is similar to our task suite in this regard. Reliance on determinism of the environment is one of the chief criticisms of imitation leveled by Juliani (2018), who offers a valuable critique on Aytar et al. (2018). [coofett et al. (2019) and Salimans and Chen (2018)). In contrast, approach is able solve tasks with substantial per-episode variability.
GAIL Ho and Ermon 2016) is another imitation learning method, however standard GAIL does not work in the following settings: 1) POMDPs (Gangwani al., 2019 Zoina et al. 2019), 2) from pixels (Li et al. 2017 Reed et al. 2018), 3) off policy (Kostrikov et al. 2018) and 4) with variable initial conditions (Zolna et al. 2019). Our setting combines all of these, leave extending GAIL. to this combined setting for future work.
4

---

Published as a conference paper at ICLR 2020
Find bat
Pick up bat
Hit ball off plinth
Pick up ball
Activate with ball (opens door)
Walk through door
Collect apple
Figure High-level steps necessary solve the Baseball task. Each step this sequence must completed in order, and must be implemented by the agent as sequence low level actions (no option structure available to the agent). The necessity of completing such long sequence of high level steps makes it unlikely that the task will ever be solved by random exploration. Note that each step involves interaction with physical objects, shown in bold.
4 HARD-EIGHT TASK SUITE
To address the difficulty of hard exploration in partially observable problems with highly variable initital conditions we introduce a collection of eight tasks, which exhibit these properties. Due the generated nature of these tasks and the rich form of interaction between the agent and environment, we see greatly increased levels of variability between episodes. From the perspective of the learning process, these tasks particularly interesting because just memorizing open loop sequence of actions is unlikely to achieve even partial success on new episode. The nature of interaction with the environment combined with a limited field of view also necessitates the use of memory in the agent.
All of the tasks in the Hard-Fight task suite share important common properties that make them hard exploration problems. First, each task emits sparse rewards-in all but one task the only positive instantaneous reward obtained also ends the episode. The visual observations in each task are also first-person and thus the state of the world only ever partially observed. Several of the tasks are constructed to ensure that that it is not possible to observe all task relevant information simultaneously.
Finally, each task is subject to a highly variable initial conditions. This is accomplished by including several procedural elements, including colors, shapes and configurations of task relevant objects. The procedural generation ensures that simply copying the actions from a demonstration is not sufficient for successful exccution, which a sharp contrast to the the case of Atari (Pohlen et al. 2018). A more detailed discussion of these aspects humans performing these tasks can be found at wtfhtpss: / 1y/ 2mAAUgq
be found in Appendix A and videos of agents and
Each task makes use of standardized avatar with first-person view of the environment, controlled by the same discretized action space consisting of discrete actions. In all tasks the agent is rewarded for collecting apples and often this is the only reward obtained before the episode ends. A depiction of description of the procedural elements and filmstrip of a successful
each task is shown in Figure 2 episode for each task is provided in Appendix
Each of these tasks requires the agent to complete a sequence of high-level steps to complete the task. An example from the task suite is shown in Figure The agent must: find the bat, pick up the bat, knock the ball off the plinth, pick up the ball, activate the sensor with the ball (opening the door), walk through the door, and collect the large apple.
We are hoping that our release of the Hard-Eight tasks will enable machine learning researchers to try imitation learning or inverse reinforcement learning algorithms on more complicated tasks.
'The link for the tasks and the data can be found at deepm nd. wwm/com/////o once they are officially released.
5

---

Published as a conference paper at ICLR 2020
5 BASELINES
In this section discuss the baselines and ablations we compare against our R2D3 agent in the experiments. We compare to Behavior Cloning (a common baseline for learning from demonstrations) as well as two ablations of our method which individually remove either recurrence or demonstrations from R2D3. The two ablations correspond to two different state of the art methods from the literature.
Behavior Cloning BC is simple and common baseline method for learning policies from demon- strations (Pomericau 1989 Rahmatizadeh et al. (0188). This algorithm corresponds to a supervised learning approach imitation learning which uses only expert trajectories as its training dataset to fit parameterized policy mapping states to actions. For discrete actions this corresponds to classification task, which we fit using the cross-entropy loss. the rewards of trajectories in the training dataset are consistently high, BC is known to outperform recent batch-RL methods (Fujimoto et al. 2018). To enable fair comparison we trained our BC agent using the same recurrent neural network architecture that we used for our R2D3 algorithm (see Figure 4
No Demonstrations The first ablation we consider is to remove demonstrations from R2D3. This corresponds to setting the demo ratio (see Figure I to O. This special case of R2D3 corresponds exactly to the R2D2 agent of Kapturowski et al. (2018), which itself extends DQN (Mnih et al. 2015) to partially observed environments by combining it with recurrence and the distributed training architecture of Ape-X DQN (Horgan et al. 2018). This ablation is itself state the art on Atari-57 and DMLab-30, making it extremely strong baseline.
No Recurrence with feed-forward reactive network. We do this separately from the no demonstrations ablation, leaving the full system in Figure in tact, with only the structure of the network changed. If we further fix the demo ratio to p 0.25 then this ablation corresponds to the DQID agent of Hester et al. (2018), which is competitive on bard-exploration Atari environments such as Montezuma's s Revenge. However, we do not restrict ourselves to p = 0.25, and instead optimize over the demo ratio for the ablation as well as for our main agent.
The second ablation we consider is to replace the recurrent value function of R2D3
6 EXPERIMENTS
We evaluate the performance of our R2D3 agent alongside state-of-the-art deep RL baselines, As discussed in Section 5 we compare our R2D3 agent to BC (standard LfD baseline) R2D2 (off- -off-policy SOTA), DQfD (LiD SOTA). We use our own implementations for all agents, and we plan to release code for all agents including R2D3.
For each task in the Hirh-Eight suite, trained R2D3, R2D2, and DQfD using 256 c-greddy CPU-based actors and a single GPU-based learner process. Following Horgan et al.] (2018), the i-th actor was assigned a distinct noise parameter e; [0.44, 0.4] where each e; is regularly spaced in 1ogo.4 space. For each of the algorithms their common hyperparameters were held fixed. Additionally, for R2D3 and DQID the demo ratio was varied to study its effect. For BC also varied the learning rate independently in a vain attempt to find a successful agent.
All agents act the environment with an action-repeat factor of 2, i.c. the actions received by the environment are repeated twice before passing the observation to the agent. Using an action repeat of 4 is common in other domains like Atari (Bellemare et al., 2012 Mnih al. 2015); however, we found that using an action repeat of 4 made the Hard-Eight tasks too difficult for our demonstrators. Using an action repeat
allowed us to strike a compromise between ease of demonstration (high
action repeats prohibiting smooth and intuitive motion) and ease of learning for the agents (low action repeats increase the number of steps required to complete the task).
Figure 4 iillustrates the neural network architecture of the different agents. As much as possible use the same network architecture across all agents, deviating only for DQID, where the recurrent head is replaced with equally sized feed-forward layer. We briefly outline the training setup below, and give an explicit enumeration of the hyperparameters in Appendix B
the Adam optimizer (Kingma and Ba 2014) with fixed learning rate of 2 x 10 4 We use hyperparameters that are shown to work well for similar environments. We distributed training with 256 parallel actors, trained for least 10 billion actor steps for all tasks.
For R2D3, R2D2 and DQfD
6

---

Published as a conference paper at ICLR 2020
(c)
Q(h)
h
225
C0TN 3xx strde
reesdull
reeMani
ODTN 3 strde
charmele
sinde
(b)
conr3 o913e
((h)
5,000
76
Figure (a) Recurrent head used by R2D3 agents. (b) Feedforward head used by the DQID agent. Heads both a) and h) are used to compute the Q values. (c) Architecture used to compute the input feature representations. Frames of size 96x72 are fed into a ResNet, the output is then augmented by concatenating the previous action at-1, previous reward rı-1, and other proprioceptive features ft, such as accelerations, whether the avatar hand is holding object, and the hand's relative distance to the avatar.
For the BC agent the training regime is slightly different, since this agent does not interact with the environment during training. For BC we also use the Adam optimizer but we additionally perform hyperparameter sweep over learning rates (10-5, 10- 10-3). Since there is notion of actor steps in BC we trained for 500k learner steps instead.
During the course of training, evaluator process periodically queries the learner process for the latest network weights and runs the resulting policy on an episode, logging both the final return and the total number of steps (actor or learner steps, as appropriate) performed at the time the of evaluation.
We collected total of 100 demonstrations for each task spread across three different experts (each expert contributed roughly one third of the demonstrations for each task). Demonstrations for the tasks were collected using keyboard and mouse controls mapped to the agent's exact action space, which was necessary to enable both behaviour cloning and learning from demonstrations. We show statistics related to the human demonstration data which we collected from three experts in Table T
6.1 LEARNING THE HARD-EIGHT TASKS
In Figure report the return against the number of actor steps, averaged over five random initializations. We find that none of the baselines succeed in any of the eight environments. Meanwhile, R2D3 learns six out of the eight tasks, and reaches exceeds human performance in four of them. The fact that R2D3 learns at all in this setting with only 100 demonstrations per task demonstrates the ability of the agent to make very efficient use the demonstrations. This in contrast to BC and DQID which use the same demonstrations, and both fail to learn a single task from the suite.
All methods, including R2D3, fail to solve two of the tasks: Remember Sensor and Throw Across. These are the two tasks in the suite that are most demanding in terms of memory requirements for the agent, and is possible that our vero-initialization with burn-in strategy for handling LSTM states in replay does not give R2D3 sufficient context to complete these tasks successfully. Future work should explore the better handling of recurrent states as possible avenue towards success on these negative returns on Remember Sensor, which indicates
tasks. R2D3, BC, and DQID receive that the agents navigate down the hallway and walks over penalty sensors.
R2D3 performed better than our average human demonstrator on Baseball, Drawbridge, Navigate Cubes and the Wall Sensor tasks. The behavior on Wall Sensor Stack in particular is quite interesting. On this task R2D3 found completely different strategy than the human demonstrators by exploiting a bug in the implementation of the environment. The intended strategy for this task is to stack two blocks top of each other that one of them can remain in contact with wall mounted sensor,
7

---

Published as a conference paper at ICLR 2020
beseball
drewbridge
navigate cubes
push blocks
5
method
reward
remember sensor
throw across
wall sensor
wall sensor stack
DD10
R202
R2D8
0
30
30
actor steps (B)
Figure 5: Reward vs actor steps curves for R2D3 and baselines on the Hard-Fight task suite. The curves are computed as the mean performance for the same agent across 5 different seeds per task. Error regions show the 95% confidence interval for the mean reward range the plots. R2D3 can perform human-lewe Wall Sensor. R2D2 could not get any positive rewards on any of the tasks. DQID and BC agents occasionally see rewards on Drawbridge and Navigate Cubes tasks, but this happens rarely enough that the effect is not visible in the plots. Indicators mark analysis points in Section 6.3
seeds. Several curves overlap exactly zero reward for the better on Baseball, Drawbridge, Navigate Cubes and
0.6
succeas rate
Task Name
Reward
Episode Len,
0.4
Baseball
7.8 4.1
492 + 121
Drawbridge
12.3 2.5
641 = 137
0.2
Navigate Cubes
7.9
638 + 185
Push Blocks
9.1
683 = 270
0.0
1/256
1/64
Remember Sensor
7.7 1.4
853 + 188
demo ratio
Throw Across
5.4
464 + 172
Figure 61 Success rate (see main text) for R2D3 across all tasks with at least successful seed, function of demo ratio. The square markers for each demo ratio denote mean success rale, and the error bars show a bootstrapped estimate of the [25, 75] percentile interval for the mean estimate. The lower demo ratios consistently out- perform the higher demo ratios across the suite of tasks.
Wall Sensor
9.1 + 2.8
280 + 87
Wall Sensor Stack
8.6 3.5
521 + 107
Table Human domonstration statistics. We collected 100 demos for each tasks from three human demonstrators. We report number frames) and rewards the episodes along with the standard deviations for each task.
lengths
and this is the strategy employed by the demonstrators. However, due bug in the environment the strategy learned by R2D3 was to trick the sensor into remaining active even when it is not in contact with the key by pressing the key against in a precise way.
In light of the uniform failure of our baselines to learn on the Hard-Eight suite we made several attempts at training other models on the task suite; however, these attempts were all unsuccessful. For example, we tried adding randomized prior functions (Osband et al., 2018) to R2D2, but this approach was still unable to obtain reward on any of the Hard-Eight tasks. We also trained an IMPALA agent with pixel control (Jaderberg et al. (0066) as auxiliary reward to help with exploration, but this approach also failed to learn on any the tasks we attempted. We omit these results from Figure 5 only keeping the most relevant baselines.
6.2 EFFECT OF THE DEMO RATIO
In experiments Hard-Eight tasks (see Figure 5), we did hyperparameter search and chose the best hyperparameters for each method independently. In this section, we look more closely at how the demo ratio (p) affects learning in R2D3. To do this look at how the success rate of R2D3 across the entire -iihh task suite varies as a function of the demo ratio.
The goal of each task in the Hard suite is collect large apple, which ends the episode and gives a large reward. We consider an episode successful if the large apple is collected. An agent that executes many episodes in the environment will either succeed or fail each one. We consider
8

---

Published as a conference paper at ICLR 2020
R202 80
R203 @
(b)
R203 @ 408
distance crates pushod
R2D3
*
*
R202
12
40 (trained]
actor steps (8))
Figure 7: Guided exploration behavior in the Push Blocks task. (a) Spatial pattern of exploration behavior ~5B actor steps (reward-driven learning kicks off for R2D3 only after ~20 steps). Overlay of agent's trajectories over 200 episodes. Blocks and sensors are not shown for clarity. R2D2 appears follow a random walk. R2D3 concentrates on a particular spatial region. (b) Interactions between the agent and blocks during the first 12B steps, Each line shows different random seed. R2D2 rarely pushes the blocks. (c) Example trajectory of R2D3 after training, the agent pushes the blue block onto the blue sensor, then collects the apple (green star).
an agent successful if, after training, at least 75% of its final 25 episodes are successful. Finally, an individual agent with a fixed set of hyperparameters may still succeed or fail depending on the randomness in the environment and the initialization of the agent.
We train several R2D3 agents on each tractable task in the Hard-Ean Hard-Eight suite, varying only the demo ratio while keeping other hyperparameters fixed the values used for the learning experiment. We consider four different demo ratios across six tasks, with five seeds for each task (120 trained agents). Figure shows estimates of the success aggregated across all tasks. We observe that tuning the demo ratio has a strong effect on the success rate across the task suite, and that the best demo ratio is quite small. See Appendix CK for further results.
for the R2D3 algorithm for each different demo ratio,
6.3 GUIDED EXPLORATION BY DEMONSTRATION
The typical strategy for exploration in RL is to either use stochastic policy and sample actions, or to use deterministic policy and take random actions some small fraction the time. Given sufficient time both of these approaches will in theory cover the space of possible behaviors. but in practice the amount of time required to achieve this coverage can be prohibitively long. In this experiment, we compare the behavior of R2D3 the behavior of R2D2 (which is equivalent to R2D3 without demonstrations) on two of the tasks from the Hard-Eight suite. Even very early in training (well before R2D3 is able reliably complete the tasks) we see many more task-relevant actions from R2D3 than from R2D2, suggesting that the effect of demonstrations is to bias R2D3 towards exploring relevant parts of the environment.
In Figure 7 we begin by examining the Push Blocks tasks. The task here is to push a particular block onto sensor to give access to large apple, and we examine the behavior of both R2D3 and R2D2 after 5B steps, which is long before R2D3 begins to solve the task with any regularity (see Figure 5). Looking at the distribution of spatial locations for the agents essentially diffuues randomly around the room, while R2D3 spends much more time in task-relevant parts of the environment (e.g. away from the walls). We also record the total distance traveled by the moveable blocks in the room, and find that R2D3 tends to move the blocks significantly more often than R2D2, even before it has learned to solve the task.
clear that R2D2
7 CONCLUSION
In this paper, we introduced the R2D3 agent, which is designed to make efficient use of demonstrations to learn in partially observable environments with sparse rewards and highly variable initial conditions. We showed through several experiments on eight very difficult tasks that our approach able to outperform multiple state of the art baselines, two of which are themselves ablations of R2D3.
We exclude Remember Sensor and Throw Across from this analysis, since saw successful seeds for
either of these tasks.

---

Published as a conference paper at ICLR 2020
We also identified a key parameter of our algorithm, the demo ratio, and showed that careful tuning of this parameter is critical to good performance. Interestingly we found that the optimal demo ratio is surprisingly small but non-zero, which suggests that there may be risk of overfitting to the demonstrations at the cost of generalization. For future work, we could investigate how this optimal demo ratio changes with the total number of demonstrations and, more generally, the distribution of expert trajectories relative to the task variability.
We introduced the Hard-Eight suite of tasks and used them in all of our experiments. These tasks specifically designed to be partially observable tasks with sparse rewards and highly variable initial conditions, making them an ideal testbed for showcasing the strengths of R2D3 in contrast existing methods in the literature.
Our behavioral analysis showed that the mechanism R2D3 uses to efficiently extract information from expert demonstrations is to use them in way that guides (or biases) the agent's 's own autonomous exploration of the environment. An in-depth analysis of agent behavior on the Hard-Eight task suite is a promising direction for understanding how different RL algorithms make selective use of information.
REFERENCES
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjumath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow; A system for large-scale machine learning, In 12th USENIX Symposium on Operating Systems Design and Implementation, pages 265 265-283, 2016.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando Freitas. Playing hard exploration games by watching YouTube. In Advances in Neural Information Processing Systems, pages 2930-2941, 2018.
Mare Bellemare, Joel Veness, and Michael Bowling, Investigating contingency awareness using Atari 2600 games. In AAAI Conference on Artificial Intelligence, pages 864-871, 2012.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Mare G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston. Curriculum learning. International Conference on Machine Leurning, pages 41-48, 2009.
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement leamning. In Advances neural information processing systems, pages 1281-1288, 2005.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint
2019.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv preprint /rXiv:/812.002 2018.
Tanmay Gangwani. Joel Lehman. Qiang Liu, and Jian Peng. Learning belief representations for imitation learning in pomdps. arXiv preprint 190.v:1906.09510, 2019.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and-conquee reinforcement leaming. arXiv preprint arXiv: 2017.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. International Conference on Machine Learning, pages 1311 1320, 2017.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference Artificial Intelligence, pagee 3215-3222, 2018.
10

---

Published as a conference paper at ICLR 2020
Todd Hester, Matej Vecerik, Olivier Pielquin, Mare Lanctot. Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep Q-lcarning from demonstrations. In AAAI Conference on Artificial Intelligence, pages 9223-3230, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, pages 4565-4574 2016.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations, 2018.
John D Hunter. Matplotlib: A 2D graphics environment. Computing in science & engineering, 933):90-95. 2007.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Ko- ray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint ar1iv 2016.
Arthur Juliani. On "solving" Montezuma's revenge.
com/ @aw juwijullinninn iani/ 214 6dB 3r0bc3 2018. Accessed: 2019- 19-21.
Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper. Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. In AAAI-19 Workshop on Games and Simulations for Artificial Intelligence, 2019.
Bingyi Kang. Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In International Conference on Machine Learning, pages 2469 24788 2018
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2018.
Beomjoon Kim, Amir- masso- Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstra- tions. Advances in Neural Information Processing Systems, pages 2859-28611 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-rriiti Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXivv:809.0290 022.00990022222.22 2018.
Eric Langlois. Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benehmarking model-based reinforcement learning. arXiv preprint 10700 2019.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. In Advances in Neural Information Processing Systems, pages 3812-3822, 2017.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018
Wes McKinney et al. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, pages 51-56, 2010.
Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv: 1707.0201,, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):5295333 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In IEEE International Conference on Robotics and Automation, pages 6292 6299,.999.... 2018
Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, pages 278-287, 1999.
Travis Oliphant. Guide NumPy. USA: Trelgol Publishing, 2006. 2006.

---

Published as a conference paper at ICLR 2020
Ian Osband. John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 8617-8629, 2018.
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang, Scott Reed. Yusuf Aytar, Tobias Pfaff, Matt W Hoffmann Gabriel Barth-Maron, Serkan Cabi, David Budden. et al. One-shot tigh-fidelity imitation: Training large-scale deep nets with RI.. arXiv preprint XrXiv: 810.05017, 2018.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of thysics-based character skills. ACM Transactions on Graphics, 37(4):1:14, 2018.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron Hado van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint 1805. 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in neural information processing systems, pages 305-313 1989.
Rouhollah Rahmatizadeh, Pooya Abolghasemi, Ladislau Bölöni, and Sergey Levine. Vision-base multi-tast manipulation for inexpensive robots using end-to-end learning from demonstration. In IEEE International Conference Robotics and Automation, pages 3758 3765,, 2018.
Scott Reed, Yusuf Aytar, Ziyu Wang. Tom Paine, Aäron van den Oord, Tobias Pfaff, Sergio Gomez, Alexander Novikov, David Budden, and Oriol Vinyals. Visual imitation with minimal adversary. 2018.
Tim
Salimans
and
Richard Chen.
Learning
Montezuma's revenge
from
single demonstration.
bttps://opena.. ccm/blog////ccccccccoo....ooooc.
1earni ng- montezunas revenge- from- a aingle demonstratioon 2018a. Accessed:
2019-19-22.
Tim Salimans and Richard Chen. Learning Montezuma's revenge from single demonstration. arXiv preprint arXiv: 1812,0381,, 2018b.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representarions, 2016.
Jürgen Schmidhuber. Curious model-buls building control systems. IEEE International Joint Conference on Neural Networks, pages 1458-1463, 1991.
Adrien Ali Taiga, William Fedus, Marlos Machado, Aaron Courville, and Mare G Bellemare. Benchmarking bonus-basee exploration methods on the arcade lcarning environment. arXiv preprint 1rXiv:1908.022 023888,,,, 2019.
Matej Večerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pictquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, arXiv preprint arXiv: 170..00//77777 2017.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference Machine Learning, pages 1995-2003, 2016.
Konrad ZoIna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarej. David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-relevant adversarial imitation learning. arXiv preprint 191001791 2019.
Konrad Zohna Negar Rostamzadeh, Yoshua Bengio, Sungjin Ahn, and Pedro O Pinheiro. Reinforced imitation in heterogencous action space, arXiv preprint arXiv: 1904.0348, 2019.
12

---

Published as a conference paper at ICLR 2020
A HARD-EIGHT TASK SUITE DETAILS
Sparse rewards All of the tasks emit sparse rewards, indeed in all but one task the only positive instantaneous reward obtained also ends the episode successfully. other words, for standard RL algorithms to learn by bootstrapping, the actors must first solve the task inadvertently, and must do so with no intermediate signal guide them.
Partial observability Visual observations are all first-person, which means that some relevant features of the state of the world may be invisible to the agent simply because they are behind it or around a corner. Some tasks (e.g. Remember Sensor, are explicitly designed that this is the case).
Highly Variable Initial Conditions Many of the elements of the tasks are procedurally generated, which leads to significant variability between episodes of the same task. In particular, the starting position and orientation of the agent randomized and similarly, where they present, the shapes, colors, and textures of various objects are randomly sampled from a set of available such features. Therefore single (or small number of) demonstration(s) is not sufficient guide an agent solve the task as is in the case of DQID on Atari (Pohlen et al. 2018
Observation specification All of the tasks provide the same observation space. In particular, visual channel consisting of 96 by 72 RGB pixels, well accelerations of the avatar, force applied by the avatar hand on the object, whether if the avatar is holding anything or not, and the distance of held object from the face of the avatar (zero when there is held object).
Action specification The action space consists of four displacement and four rotation actions (8), duplicated for coarse and fine-grained movement (16) as well as for movement with and without grasping (32). The avatar also has an invisible "hand" which can be used to manipulate objects in the environment. The location of the hand is controlled by the avatar gaze direction, plus an additional two actions that control the distance of the hand from the body (34). A grasped object can be manipulated by six rotation actions (two for each rotational degree of freedom; 40) as well as four additional actions controlling the distance of the hand from the body at coarse and fine speed (44). Finally there is an independent grasp action (to hold an object without moving), and no-op action (total 46). Compared to course actions, fine-grained actions result in slower movements, allowing the agent to perform careful manipulations.
A.1 INDIVIDUAL TASK DETAILS
This section gives addition details on each task in our suite including a sequence frames from successful task execution (performed human) and a list of the procedural elements randomized per episode. Videos of agents and humans performing these tasks can be found at fuhttps:/// ://bit 1y/2mAAUga
Baseball
The agent spawns in small room with sensor and key object resting high atop plinth. The agent must find a stick and use to knock the key object of the plinth in order to activate the sensor. Activating the sensor opens door to adjoining room with large apple which ends the episode.
Procedural elements
Initial position and orientation of the agent
Wall, floor and object materials and colors
Initial position of the stick
Position of plinth
13

---

Published as a conference paper at ICLR 2020
Drawbridge
BIII
The agent spawns at one end of network of branching platforms separated by drawbridges, which can activated by touching a key object sensor. Activating a drawbridge with a key object destroys the key. Each platform is connected to several drawbridges, but has only one key object available. Some paths through the level have small apples which give reward. The agent must choose the most rewarding path through the level to obtain large apple at the end which ends the episode.
Procedural elements
Initial position and orientation of the agent
Wall, floor, ceiling and object materials and colors
Positions of the small apples throughout the network of ledges
Navigate Cubes
The agent spawns
side of large room. On the other side of the room raised platform there a large apple which ends the episode. Across the center of the room there is wall of movable blocks. The agent must dig through the wall of blocks and find a ramp the goal platform in order to collect the large apple.
Procedural elements
Initial position and orientation of the agent
Wall, floor and object materials and colors
Push Blocks
The agent spawns in medium sized room with a recessed sensor in the floor. There are several objects in the room that can be pushed but not lifted. The agent must push block whose color matches the sensor into the recess in order to open door to an adjoining room which contains large apple which ends the episode. Pushing a wrong object into the recess makes the level impossible to complete.
Procedural elements
Initial position and orientation of the agent
Wall, floor, object materials and colors
Positions of the objects
Sensor required color
14

---

Published as a conference paper at ICLR 2020
Remember Sensor
The agent spawns near sensor random color. The agent must travel down long hallway to room full of blocks and select one that matches the color of the sensor. Bringing the correct block back to the sensor allows access to a large apple which ends the episode. In addition to being far away, traveling between the hallway and the block room requires the agent to cross penalty sensors which incurs small negative reward.
Procedural elements
Initial position and orientation of the agent
Sensor required color
Number of objects in the block room
Position of objects in the block room
Shape and material of the objects in the block room
15

---

Published as a conference paper at ICLR 2020
Throw Across
The agent spawns in U shaped room with empty space between the legs of the U. There are two key objects near the agent spawn point. The agent must throw one of the key objects across the void, and carry the other around the bottom of the U. Both key objects are needed to open two locked doors which then give access to large apple which ends the episode.
Procedural elements
Initial position and orientation of the agent
Wall, floor and object materials and colors
Color and material of the sensors
Initial positions of the key objects
Wall Sensor
The agent spawns in small room with wall mounted sensor and key object. The agent must pick up the key and touch to the sensor which opens a door. In the adjoining room there is a large apple which ends the episode.
Procedural elements
Initial position and orientation of the agent
Position of the sensor
Position of the key object
Wall Sensor Stack
The agent spawns in a small room with a wall mounted sensor and two key objects. This time one of key objects must be in constant contact with the sensor in order for the door remain open. The agent must stack the two objects so one can rest against the sensor, allowing the agent to pass through an adjoining room with large apple which ends the episode.
Procedural elements
Initial position and orientation of the agent
Wall, floor and object materials and colors
Initial positions of both key objects
Position of the sensor
16

---

Published as a conference paper at ICLR 2020
B
HYPER-PARAMETERS
In Table 2
we report the shared set of hyper-parameters across different models and tasks.
Hyperparameters
Values
Network
See Figure
Environment
Image height
72
Image width
96
Color
RGB
Action repeats
Observation spec
See section A
Action spec
See section
Learner
Learning rate
2e-4
Optimizer
Adam Kingma and Ba 2014
Global norm gradient clipping
True
Discount factor (?)
0.997
Batch size (B)
32
Target update period (fearget)
400
Actor update period (factor)
200
Prioritized sampling
True
Sequence length (m)
Burn in length
40
Asymmetric reward clipping
True
Number of actors (A)
256
Max replay capacity
500000
Min replay capacity
25000
Table 2: Hyper-parameters used for all experiments.
EXPERIMENTS
C.1 SURPASSING THE EXPERTS
(b]
0.55 slmps
45
(b)
800
R2D3
R2D2
aphood lenghh
human mean
groportton of
but
642
501
semaar
uppls
bat
D6.1
senaor
upplo
(8)
Figure 8: Guided exploration behavior in the Baseball task. (a) Sub-bchaviors expressed by five R2D2 and five R2D3 agents after 0.5B steps of training (left) and 4B steps of training (right). Each point estimated from 200 episodes. At 0.5B steps, none of the agents received any reward over the 200 evaluation episodes, while at steps, three of the R2D3 agents received reward on almost every episode. Even when the R2D3 agents are not receiving reward, they are expressing some of the necessary behaviors provided through human demonstrations. (b) R2D3 agents eventually surpass human performance. The of R2D3 agents shown in (a) which start obtaining rewards continue bootstrap towards more efficient policies than humans.
An important property of R2D3 is that although the agents trained from demonstrations, the behaviors they achieve are able to surpass the skill of the demonstrations they were trained from. This can be quantitatively from reward curves in Figure where the R2D33 agent surpasses the
17

---

Published as a conference paper at ICLR 2020
Baseball
Drawbridge
Navigate Cubes
Push Blocks
10
seed
Reward
-0
Remember Sensor
Throw Across
Wall Sensor
Wall Sensor Stack
10
0
1c10 2e10
3e10 o
1e10 2e10 3e10 0
1e10 2e10 3e10 0
1e10 2e10 3e10
Actor steps
Figure 9: We show the rewards of the R2D3 agent on different tasks for each seed separately,
Baseball
Drawbridge
Navigate Cubes
Push Blocks
10
Roward
Demo ratio
-1/256
Wall
Stack
-1/128
-1/64
-1/32
5
le10 2e10
3e10
1e10 2e10 3e10
steps
Figure 10: R2D3 learning curves with varying demo ratios for all tasks.
human baseline performance on four of the eight tasks (c.g. Baseball, Navigate Cubes, Wall Sensor and Wall Sensor Stack).
In some of these cases the improved score is simply matter of executing the optimal strategy more fluently than the demonstrators. For example, this is the case in the Baseball task, where the human demonstrators are handicapped by the fact that the human interface to the agent action space makes awkward rotate held object. This makes picking up the stick and orienting it properly knock the ball off the plinth into tricky task for humans, but the agents are able to refine their behavior to be much more efficient (see Figure ()).
The behavior on Wall Sensor is especially interesting, however in this case the agents find a completely different strategy than the human demonstrators by exploiting bug in the implementation of the environment. The intended strategy for this task is to stack two blocks one them can remain in contact with wall mounted sensor, and this is the strategy employed by the demonstrators. However, due to bug in the environment it is also possible to trick the sensor into remaining active even when not contact with the key by pressing the key against it in precise way. The R2D3 agents are able to discover this bug and exploit it, resulting in superhuman scores on this task even though this strategy is not present in the demonstrations.
top of each other so that
C.2 ADDITIONAL EXPERIMENTS
We also ran few additional experiments to get more information about the tasks we did not solve, or solved incorrectly. Videos for these experiments are available fattps://bbn 1ttp://tt.y//mAAAggggggg 2mAAUgg
Remember Sensor This task requires long memory, and also has the longest episodes length of any task in the Hard Eight suite. In an attempt to mitigate these issues, we trained the agent using higher action repeat which reduces the episode length, and used stale Istm states instead zero Istm states which provides information from earlier in the episode. This allows R2D3 to learn policies that display reasonable behavior, retrieving a random block and bringing back to the hallway. Using this method it can occasionally solve the task.
18

---

Published as a conference paper at ICLR 2020
Throw Across The demonstrations collected for this task had very low success rate of 54%. We attempted to compensate for this by collecting an additional 30 demos. When we trained R2D3 with all 130 demos all seeds solved the task.
Wall Sensor Stack The original Wall Sensor Stack environment had bug that the R2D3 agent able exploit. We fixed the bug and verified the agent learn the proper stacking behavior.
C.3 ADDITION DETAILS FOR MAIN EXPERIMENTS
In Figure 9 we show the performance of the R2D3 agents for each seed separately. On task such as Drawbridge, Navigate Cubes and Wall Sensor, all seeds take off quite rapidly and they have very low variance for the rewards between different seeds. However, takes off quite rapidly, and the rest of them are just flat. In Figure [0] we elaborate on Figure 6 For Baseball, Navigate Cubes, Push Blocks, and Wall Sensor Stack, demo ratio of 1/256 works best. On Drawbridge and Wall Sensor all demo ratios are similarly effective.
Wall Sensor Stack task while one seed
tal
0b1
pushed
progortion at qiiods
R2D2
3.6
2.4
0.2
Figure 11: Further detail guided exploration behavior in Push Blocks task (as in Figure 7 (a) Proportion of cpisodes in which the agent pushes crate into the recess during the initial 12B steps of training. (b) Proportion of episodes in which the crate pushed into the recess actually matches the sensor color. Data are only shown when crates pushed into the recess at least 5 out of 200 episodes. Dashed line shows the probability expected if random crate was pushed into the
Thus, while (c) shows that by 12B steps the R2D3 agent may have reasonable success in pushing crates the recess, has not yet mastered the logic that the crate color must much the sensor color.
19

---

Published as a conference paper at ICLR 2020
stegs
128 steps
0,86 steas
125 stoos
Figure 12: Further detail of guided exploration behavior in the Push Blocks task (as in Figure (a) Spatial pattern of exploration behavior for the R2D2 agent the course of 12B steps of training. Each shows different random seed; the number training steps increases from the leftmost column to rightmost column. There is little variation in how the policy manifests as explorative hehavior across seeds and training time. (b) As in (a), for R2D3. Given demonstrations, the policies now show substantial variation across seeds and training time.
20


Reviews:
[{"review_id": "rkgS-t2htH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe authors propose R2D3, and algorithm for learning from demonstrations in partially-observable environments with sparse rewards. The algorithm combines DQfD with recurrent networks to both leverage expert demonstrations and handle partial observability. Furthermore, the authors propose a suite of eight challenging tasks on which the proposed method is tested and compared to relevant baselines.\n\nComments\n--------\n\nScaling RL agents to high-dimensional partially-observable domains with sparse rewards is a fundamental open problem and this work provides a nice contribution towards its solution. The paper is well-written and easy to read. The proposed methodology seems to be a simple combination of existing algorithms and (apologies if I am wrong) I did not see any particular challenge in its design. On the other hand, the hard-eight task suite is interesting and, if released, could be used as a benchmark by the whole community. The experiments seem quite convincing in proving the potential of the proposed method. Some comments/questions follow.\n\n1. Section 2 presents the algorithm from a very high-level perspective. If space in the final version allows it, I would also suggest adding a more detailed pseudo-code to the main text, so that even a reader who is not completely familiar with the works this method builds upon could better understand and possibly implement the method.\n\n2. Since the authors compare to behavioral cloning to prove the benefits over simple imitation methods, why not comparing to stronger baselines such as [1] or [2]?\n\n3. The demo-ratio seems to be the key parameter to make this approach work (and the performance is proven very sensitive to its value). Instead of keeping it fixed across the entire learning process, have you tried to start with a high value and then decay according to a proper schedule? Intuitively, I would expect the benefits of expert demonstrations to be more valuable during the first learning episodes (where they make the agent explore better) and less during the successive phases (where the policy gets closer and closer to optimal).\n\n4. The way recurrent states are handled with zero-initialization is probably one of the limitations and seems to play an important role in some experiments. Have you tried, at least in simpler domains, to replay whole episodes and see whether that helps?\n\n[1] Ho, J., & Ermon, S. (2016). Generative adversarial imitation learning. In Advances in neural information processing systems (pp. 4565-4573).\n[2] Finn, C., Levine, S., & Abbeel, P. (2016, June). Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (pp. 49-58).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJl2wiVt9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, R2D3 (Recurrent Replay Distributed DQN from Demonstration), which combines R2D2 [1] with imitation learning (IL), is proposed. Similar to the existing works on “reinforcement learning (RL) with demonstration” such as DQfD, DDPGfD, policy optimization with demonstration (POfD) [2], hard exploration conditions (sparse reward, partial observability, high variance in initial states) are assumed, which is difficult to achieve good performance with RL without demonstration in general. Eight tasks in such conditions were devised and used to test the performance of R2D3.\n\nI like the fact that the authors of this work have chosen quite challenging scenarios, but I think the novelty of this submission is a bit weak to be accepted to the conference. I believe “RL with demonstration” becomes meaningful when it beats both RL and IL in some reasonable setting. For example, POfD [2] assumes sparse-reward tasks with *imperfect* demonstrations, which is difficult to achieve good performance by using RL or IL. From such a perspective, I have the following concerns:\n\n- Imitation learning baselines: There has been recent advancement in imitation learning. In the submission, it was mentioned that “GAIL has never been successfully applied to complex partially observable environments that require memory”, but there’s [3] that successfully uses GAIL in such a setting. Also, off-policy imitation learning such as DAC [4] is shown to be highly sample-efficient compared to GAIL in MuJoCo domain. However, the submission only considers behavioral cloning (BC) (which shows poor performance at unseen states due to the covariate shift problem) as a baseline among imitation learning method\n\n- Reinforcement learning baselines: The submission adopted R2D2 as an RL baseline, and it seems to me that the R2D2 agent starts from random initialization. For a fair comparison, however, I believe R2D2 with BC (or Batch RL) initialization should be considered.\n\nIn addition to the above concerns, it seems to me that most of the features in R2D3 simply combines those in either DQfD or R2D2, and I couldn’t find out its own algorithmic novelty except “demo ratio” parameter. \n\nI’ll increase my score if I made wrong comments or misunderstood the contribution.\n\nReferences\n[1] Kapturowski, Ostrovski, Quan, Munos. and Dabney, “Recurrent experience replay in distributed reinforcement learning,” ICLR 2019.\n[2] Kang, Jie, Feng, “Policy optimization with demonstrations,” ICML 2018\n[3] Gangwani, Lehman, Liu, Peng, “Learning Belief Representations for Imitation Learning in POMDPs,” UAI 2019\n[4] Kostrikov, Agrawal, Dwibedi, Levine, Jonathan, Tompson, “Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning,” ICLR 2019", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkgqZVuaYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of exploiting human demonstrations in hard exploration (RL) problems. A new set of challenge tasks is introduced that destroys the performance of very strong baseline systems while highlighting the strength of the new system.\n\nThe approach (rarely but consistently training on separately prioritized human experience replays) is well motivated by the shortcomings of past agents (either in overfitting the demonstrated solution or only working in environments with not-too-hard exploration challenges). Where work by others have overspecialized on specific challenge environments (e.g. Montezuma's Revenge with weak stochasticity and observability challenges), this work intentionally dives into difficult territory.\n\nThis reviewer moves to accept this top-quality RL paper. The new agent, R2D3, is the primary contribution in combining and outperforming previous SOTA agents. These eight new environments are minor contributions with limited potential for impact on the field, but still make an independently positive contribution.\n\nQuestions for authors:\n- What would be the present-day approximate retail cost for reproducing the experiments in this paper?\n- At the action-rate experienced by the human demonstrators (30fps?), how much wall-clock time represented by 40B actor steps? (40 years?) Is this \"making efficient use\"?\n- Does having highly variable initial conditions really force generalization over environmental configurations or is this wishful thinking / mysticism? To make a direct claim about this, the authors should consider an experimental design where certain classes of initial conditions (e.g. starting on the left side of the map) are withheld during training and evaluated only during testing.\n- The finding of small demo ratios as being stronger is exciting, but this result seems to be tied to the specific quantity and quality of demonstrations gathered. Could a more general picture of the role of demonstrations be had by ablating the diversity of representations? The 100 demos in the full case might be degraded to 50, 25, 10, etc while holding the demo ratio fixed. This might effectively vary the weight that demonstrations take in the optimization independently of how often distinct demos are actually seen.\n- Can these hard-eight scenarios be parametrically scaled up and down in terms of their exploration effort (possibly by just changing the action granularity / movement speed)? With performance on the new benchmark almost saturated in the first paper based on it, there isn't much room to grow here. In the same way that Montezuma's Revenge was found by scanning the culturally-impactful library of Atari games, perhaps more appropriate and lasting challenges can be found by looking one or more generations forward in the history of commercial console games. Can we play Star Fox? What about SimCity?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles hard-exploration RL problems using learning from demonstrations. The idea is to combine the existing R2D2 algorithms with imitation learning from human demonstrations. Experiments are conducted on a new set of challenging tasks, highlighting limitations of strong current baseline while highlighting the strength of the proposed approach.

The contribution is two-folds: the proposed algorithm which clear outperforms previous SOTA agents and the set of benchmarks. All reviewers being positive about this paper, I therefore recommend acceptance.
Decision: Accept (Poster)