Self-Imitation Learning via Trajectory-Conditioned Policy for Hard-Exploration Tasks
Authors: Yijie Guo
Jongwook Choi
Marcin Moczulski
Samy Bengio
Mohammad Norouzi
Honglak Lee
Venue:  (2020)
Keywords: imitation learning
hard-exploration tasks
exploration and exploitation

Abstract: Imitation learning from human-expert demonstrations has been shown to be greatly helpful for challenging reinforcement learning problems with sparse environment rewards. However, it is very difficult to achieve similar success without relying on expert demonstrations. Recent works on self-imitation learning showed that imitating the agent's own past good experience could indirectly drive exploration in some environments, but these methods often lead to sub-optimal and myopic behavior. To address this issue, we argue that exploration in diverse directions by imitating diverse trajectories, instead of focusing on limited good trajectories, is more desirable for the hard-exploration tasks. We propose a new method of learning a trajectory-conditioned policy to imitate diverse trajectories from the agent's own past experiences and show that such self-imitation helps avoid myopic behavior and increases the chance of finding a globally optimal solution for hard-exploration tasks, especially when there are misleading rewards. Our method significantly outperforms existing self-imitation learning and count-based exploration methods on various hard-exploration tasks with local optima. In particular, we report a state-of-the-art score of more than 20,000 points on Montezumas Revenge without using expert demonstrations or resetting to arbitrary states.

Under review as a conference paper at ICLR 2020
SELF-IMITATION LEARNING VIA TRAJECTORY- CONDITIONED POLICY FOR HARD-EXPLORATION TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Imitation learning from human-expert demonstrations has been shown to be greatly helpful for challenging reinforcement learning problems with sparse environment rewards. However, it is very difficult to achieve similar success without relying on expert demonstrations. Recent works on self-imitation learning showed that imitating the agent's own past good experience could indirectly drive exploration in some environments, but these methods often lead to sub-optimal and myopic behavior. To address this issue, we argue that exploration in diverse directions by imitating diverse trajectories, instead of focusing on limited good trajectories, is more desirable for the hard-exploration tasks. We propose new method of learning a condittoonn policy to imitate diverse trajectories from the agent's 's own past experiences and show that such self-imitation helps avoid myopic behavior and increases the chance of finding globally optimal solution for hard-explorane oration tasks, especially when there are misleading rewards. Our method significantly outperforms existing self- inffttiittionn learning and count-based exploration methods on various hard-exploration tasks with local optima. In particular, we report a state-of-the-art score of more than 20,000 points on Montezumas Revenge without using expert demonstrations or resetting to arbitrary states.
INTRODUCTION
Hard-exploration tasks, particularly characterized by sparse environment rewards, are traditionally challenging in reinforcement learning (RL), because the agent must carefully balance the exploration and exploitation when taking a long sequence of actions to receive infrequent non-zero rewards.
Demonstration data has been shown to be helpful for tackling hard-exploration problems (Subrama nian et al. 2016); many existing methods (Hester et al. [0188] Pohlen et al., 22018) Aytar et al. 20188] Salimans Chen 2018) provide the guidance for exploration based on imitation learning of expert demonstrations and achieve strong performances on hard-expa sorrd--xllorttion tasks. However, the reliance on human demonstrations largely limits the general applicability of such approaches.
The agent's ' own past good trajectories with high total rewards are easily accessible (though imperfect) alternatives for the human-expert demonstrations. Recent works (Oh et al.) 2018 Gangwani et al. 2018) verify that imitation learning from the agent's is previous good trajectories could indirectly drive exploration in certain environments. However, imitation of good experiences within limited directions might hurt ploration in some cases. Specifically, in environments with misleading rewards which may trap the agent in local optima, simply imitating 'good' trajectories that would accumulate misleading positive rewards may guide the agent to a myopic behavior and hinder it from reaching a higher return in the longer term. Therefore, imitating diverse trajectories would be more desirable to help encourage exploration in diverse directions and avoid being distracted by the misleading rewards. For example, as illustrated in Figure [] the agent starts in the bottom left corner where it can easily collect apples near its initial location by random exploration and achieve small positive reward. If the agent imitates the trajectories around the orange path, it would receive the nearby positive rewards quickly but it is unlikely to collect the gold within given time limit. Therefore, in order to find the optimal path (purple), is better to exploit the past experiences in diverse directions (gray paths), instead of focusing only the trajectories with the myopic behavior.
Figure 1: Map of Apple-Gold domain, where the reward for getting an apple, get- ting the gold and taking step in the rock is 1, 10, -0.05 respectively. The time limit for one episode is 45 steps.

---

Under review as a conference paper at ICLR 2020
This paper investigates how imitation of diverse past trajectories leads further exploration and helps avoid getting stuck at a sub-optimal behavior. Our main contributions are summarized as follows: (I) We propose a novel architecture for trajectory condiiion policy that can imitate diverse demonstrations. (2) We show the importance of imitating diverse past experiences to indirectly drive exploration to different regions of the environment, by comparing to existing approaches sparse-reward reinforcement learning tasks with discrete and continuous action space. (3) We achieve a performance comparable with the state-of-the-an on hard-exploration Atari game of Montezuma's Revenge and Pitfall without using expert demonstrations or resetting to arbitrary states.
various
RELATED WORK
Imitation Learning The goal of imitation learning is to train policy to mimic given demonstra- tion. Many previous works achieve good results on hard-explorationn Atari games by imitating human demonstrations. DQfD (Hester et al.) 2018) combines the temporal difference updates Q-learning with the supervised classification of the demonstrators' actions. Ape-X DQMD (Pohlen et al. 2018) extends DQFD with the transformed Bellman operator and temporal consistency loss to improve the efficiency of exploration. |Aytar al. (2018) learn embeddings from a variety of demonstrations videos and proposes the one-shot imitation learning reward, which inspires the design of reward in our method. All these successful attempts rely on the availability of human demonstrations. In contrast, our method treats the agent's past trajectories as demonstrations. Imitation learning is more difficult when the environment becomes more stochastic because the demonstrations could not account for all possible situations. Our method allows for some flexibility to follow the demonstrations in a soft-order and thus could perform well in the environment with a moderate degree of stochasticity. As discussed in Appendix C.I we can easily extend our method to handle a more challenging scenario (e.g.. where the location of objects could be random). Yet, imitation learning in extremely stochastic environments is still an open problem (Ghosh et al.) 2017 Paine et al. 2019).
Learning a good policy by imitating past experiences has been discussed where the agent is trained imitate only the high-reward trajectories with the SII. (Oh et al. 2018) GAIL objective (Gangwani et al. 2018). In contrast, we store the past trajectories ending with diverse states in the buffer, because trajectories with low reward in the short term might lead to high reward in the long term, and thus following a diverse set of trajectories could be beneficial for discovering optimal solutions. Furthermore, our method focuses on explicit trajectory-level imitation while existing methods use sampled state-action pairs from the buffer to update the policy. Gangwani et al. (2018) proposed to learn multiple diverse policies in a SIL framework using the Stein Variational Policy Gradient. Empirically, their exploration can be limited by the number of policies learned simultaneously and the exploration performance of every single policy, as shown Appendix F
Self-Imitation
Exploration Many exploration methods (Schmidhuber 1991 Auer 2002 Chentanez et al.) Strehl & Littman, 2008) in RL tend to award a bonus to encourage an agent to visit novel states.
2005
Recently this idea was scaled up large state spaces (Tang al.] 2017 |Bllemare et al. 2016
Ostrovski et al.] 2017 Burda et al., (2118). We propose that instead of directly taking quantification
of novelty as an intrinsic reward, one can encourage exploration by rewarding the agent when successfully imitates demonstrations that would lead to novel states and gain the advantages in
exploitation, as discussed in Appendix [[ Go-Explore (Ecoffet et al., 2019) also shows the benefit of exploration by returning to promising states. Our method can be viewed in general as an extension of Go-Explore, though we do not need to explicitly divide learning into two phases of exploration and robustification. Go-Explore relies on the assumption that the environment is resettable. Resetting to an arbitrary state is often infeasible in real environments and gives an unfair advantage. When using a perfeet coal-conditioned policy instead of direct 'reset" function, this variant of Go-Explore may not explore as efficiently as our method, as discussed in Appendix H. Previous works attempted goal-conditioned policy in pixel observation space (Dong et al., 2019).. Gregor et | (2016); Eysenbach et al. (2015 (2018); (Pong al. (019)) seek diversity of exploration by maximizing the entropy of mixture skill policies or generated goal states. However, these methods do not show experimental results performing well on sparse-reward environments with a rich observation space like Atari games.
reaching a goal state by learning set of sub-policies (Liu et al. 2019
Goal-Conditioned Policy Andrychowicz et al. (2017) 7 >; Nair et al. (2117) Schaul et al. (20 15a);
Pathak et al. (2018) studied learning a goal-conditionss policy. Our ceneectoor-conitionnee policy could be viewed as noal-conditioned policy. Similarly to hindsight experience replay (Andrychowicz jet al. 2017). our approach samples goal states from past experiences. Compared to conditioning on single final goal state, the state trajectory includes rich intermediate information leading the
2

---

Under review as a conference paper at ICLR 2020
Algorithm Diverse Self-Imitation Learning with Trajectory-Conditionee Policy
Initialize parameter 0 for the trajectory
policy ze ca(a:eege
Initialize the trajectory buffer D -- 0 # Store diverse past trajectories
Initialize set transitions the current episode & + 0 # Store current episode trajectory Initialize set of on on-policy samples +- 0 # Store data for on-policy PPO update Initialize demonstration trajectory +- for each iteration from 1 to 7 do for each step do
Observe St {ol, ex) } choose action ar ~ (aalees 0r,
Execute action a, in the environment to get rr, 0.-1,
Store transition & <- E {(o,, et, Qt, Tt)) # Positive reward if agent follows demonstration
No reward after agent completes 9 and then takes random exploration
Determine Tz DISI. by comparing e5t++ with 9
(Eq.
Store on- oa-policy sample +- U (or, Ct, at 9, (.r..sn))
end for
8t-1 is terminal then
D - Uudate E)
(Alg
Clear current episode trajectory & +
+ SampleDemo(D,
(Alg. 3
end if
- vL Perform PPO update using on-policy samples
(Fq.
Clear on-policy samples +0
+ - v # Perform supervised learning updates using samples from 'D for times
(Bq.
end for
agent to follow a demonstration and reach the goal state even far away from the current state. Our method shares the same motivation as Duan et al. (2017) which uses attention model over the demonstration but mainly focuses on the block stacking task. However, our architecture is simpler since it does not various environments.
attention model over the current observation and our method is evaluated on
3
METHOD
The main idea of our method is to maintain a buffer of diverse trajectories collected during training and train trajectoyy-conditonee policy by leveraging reinforcement learning and supervised learning to roughly follow demonstrations sampled from the trajectory buffer. The demonstration trajectories cover diverse possible directions in the environment. Therefore, the agent is encouraged to explore beyond various visited states in the environment and gradually push its exploration frontier further. In the meantime, we can train the policy to imitate the best trajectories collected to exploit the past good experiences. We put more weights exploration in the early stage of training, and then increases the probability of imitating the best trajectories (i.e., exploitation) as training goes on. We name our method as Diverse Trajectory-condiiovne Self- Self-Imitation Learning (DTSIL).
3.1 BACKGROUND AND NOTATION
In the standard reinforcement learning setting, each time step t, agent observes state 8t, selects an action de E A, and receives a reward r, when transitioning to a next state 8 E S, where S is set of all states and A is of all actions. The goal is to find policy me (a(s) parameterized by that maximizes the expected return Eng E 7 Tr], where E (0, 1] is discount factor.
In our work, assume state St includes the agent's observation o, (c.g., raw pixel image) and high-level abstract state embedding e, (e.g., the agent's 's location in the abstract space). The embedding e, may be learnable from 0gi (e.g., ADM (Choi et al. 2018) could localize the agent Atari games), but in this work, consider setting where high-level embedding is provided innectory-conditioneed policy ne (z ( ot, takes sequence of state embeddings ee..,e...... /
part of 8t D A ot,g) g) (which we refer to as re(-lg) in shorthand notation) input for demonstration, where [g] is
II many important application domains (e.g. robotics domain), such handcrafted representation available. Also, learning good state representation itself is an important open question and extremely challeng- ing especially for hard-exph eepeeeelllloonnnccc and sparse reward environments. which is not the main focus of this work. Therefore, we assume the availability of the high-level representations many previous works (Florensa al.)
2017 Liu al. 2019 Ecoffet al. 2019 Plappert et al. 2018

---

Under review as a conference paper at ICLR 2020
the length of the trajectory g. A sequence of the agent's past state embeddings Cget fei, e .e. is provided to determine which part of the demonstration has been followed by the agent. Together with the current observation oN, it helps to determine the correct action a, to accurately imitate the demonstration. Our goal here is to find a set of optimal state embedding sequence(s) g and the policy n6 (c(1)) to maximize the return: g² 0" - arg maxg.0 (xx D_io ?'re). For robustness we may want to find multiple near-optimal embedding sequences with similar returns and a trajectory- conditioned policy for executing them. In our implementation, we train the trajectory-condittonee policy to imitate the best trajectories. Alternatively, an unconditional stochastic policy could also be trained to imitate the best trajectories, which may further improve generalization and robustness (see Appendix C.I for more discussion and experiments).
3.2 ORGANIZING TRAJECTORY BUFFER
We maintain a trajectory busfer D - {(e(1)() ,(1,, (,)),n()))) (1))), (e(2), (12), tories. For each with a state with the high-level representation e(1) and () is the number of times the cluster repre- sented by this state embedding e has been visited during training. To maintain a compact buffer, similar state embeddings within the tolerance threshold th can be clustered together, and the existing entry is replaced if an improved trajectory r(t) ending with a near-identical state is found.
of diverse past trajec- wra.dddinngrrjeetory-cooom tuple (el0) ,-(),, -=()),(()))))) ).) 7(0) is the best trajectory ending
,n())),...) ,,1)),()))
When given a new episode E {(o0, Cp. ao ro): .... (OT. CT, @T. LT) ). all of the state embeddings (1 < f S T) in this episode & are considered as follows (similarly to Ecoffet et al. (2019)), because the buffer should maintain all of the possible paths available for future exploration to avoid missing any possibility to find an optimal solution. If the Euclidean distance between e, and any state embedding e(() in the buffer is larger than th (i.e e, does not belong to any existing cluster in the buffer), (ee, Tge. 1) is directly pushed into the buffer, where Tgi = {(00. eo, ag. ro). .(r,, Et, Qe, dc,re) re)} is the agent's 's partial episode ending with e₁. If there exists c²a) similar to e (i.e., e(a) and e, belong to the same cluster within threshold th) and the partial episode Tge is better (i.e., higher return or shorter trajectory) than the stored trajectory 7(8), -(k) is replaced by the current trajectory Tgt, and ((k) is replaced by ex to represent this cluster of state embeddings. The full algorithm in pseudo-code is described in Appendix A.1
3.3 SAMPLING DEMONSTR ATIONS
When learning a trajectory trajectory conditioned policy ग, demonstration trajectories are sampled from the buffer D. We record the count (i()) of how many times the cluster represented by this state embedding e(9) is visited. In the exploration mode, we set the probability of sampling each trajectory as 1 n
This is inspired by the count based exploration bonus (Strehl & Littman 2008 Bellemare et al. 2016)
and the idea of rank-based prioritization (Schaul et al. 2015b] [coffet et al. 2019): we prioritize a trajectory that ends with a less frequently visited state because this leads the agent to reach rarely visited regions in the state space and is more promising for discovering novel states.
On the other hand, in the imitation mode, we sample the best trajectories stored in the buffer for imitation learning. These trajectories are used to train the policy to converge to a high-reward behavior (Aytar et al. 2018 Ecoffet et al.] 2019). To balance between exploration and exploitation, we decrease the probability of taking the exploration mode and exploit the best experiences more as training goes The algorithm is described in Appendix A.2
3.4 LEARNING TRAJECTORY-CONDIIIONNE POLICY
Imitation Reward Given a demonstration trajectory == (c0 cj; ... , signals for imitating g. At the beginning of an episode, the index of the last visited state embedding in the demonstration is initialized as u = -1. At each step 1, if the agent's new state St+1 has an embedding e and is the similar enough to any of the next At state embeddings starting from the last visited state embedding eg, in the demonstration (i.c., |e2+1 ex Il < th where u < u' < At)t), then it receives positive imitation reward pim and the index of the last visited state embedding in the demonstration is updated as u < u'. This encourages the agent to visit the state embeddings in the demonstration in a soft-order so that the agent could explore around the demonstration and the demonstration plays role to guide the agent to the region of interest in the state embedding space. To summarize, the agent receives a reward rf ,ISSILL defined as
). we provide reward
(T.) rim if 3u', u w' S u + At, such that lle - -e+1 I1 th
DTSILL
(1)
0
otherwise,
4

---

Policy Architecture For imitation learning with diverse demonstrations, we design trajectory- conditioned policy To (aalecge, Ot.g) that should imitate any given trajectory g. Inspired by neural machine translation methods (Sutskever et al. 2014; Bahdanau et al. 22014), one can view the demon- stration as the source sequence and view the incomplete trajectory of the agent's 's state representations Under review as a conference paper at ICLR 2020
Agent's
Trajectory
Attention
(gge
Viegooo
AS
Art
2
Demonst attio Trajocsory
Demonstration
Trajectory
Corv
hp
An example showing the updates of u. given At each step I, we check the state embedding cx+11 to find similar state embedding ee, satisfying 6++1 Re e (i.c. ||et+1 e,, || < th) and to determine the reward according to Equation After completing the demonstration, the agent performs random exploration with reward (es e7)-
Figure 2:
Agent's Trarectory
Figure 3: Architecture of the trajectory-
conditioned policy (see Appendix
where f(-) is monotonically increasing function (e.g., clipping (Mnih et al. 2015 Figure illustrates the updates of during an episode when the agent visits a state whose embedding is close to state embeddings in the demonstration g.
as the target sequence. We apply recurrent neural network (RNN) and an attention mechanism to the sequence data to predict actions that would make the agent follow the demonstration.
As illustrated in Figure 3 RNN computes the hidden features h for each state embedding ef (0 < i < [g)) in the demonstration and derives the hidden features h2 for the agent's state representation e,- Then the attention weight a; is computed by comparing the current agent's hidden features h with the demonstration's 's hidden features he (0 as an attention -eeehhee-wweggted summation of the demonstration' 's hidden features to capture the relevant information in the demonstration trajectory and to predict the action a.. The more details of policy architecture are described in Appendix B
lg)). The attention readout e, is computed
Reinforcement Learning Objective conditioned policy can be trained with policy gradient algorithm (Sutton et al. 20000:
With the reward defined as DTSILL (Equation 1 the trajectory-
CRL Exol log To ve(aelese, oc. (r.g)Aa).
n-1
where Å,
DISIL + +, Vo Otn:: Volege, Ot, Ot.g):
d=0
where the expectation E,, indicates the empirical average over finite batch of on on-policy samples and denotes the number rollout steps taken in each iteration. We use Proximal Policy Optimization (PPO) (Schulman et al. 2017) as an actor-critic policy gradient algorithm for our experiments.
Supervised Learning Objective To improve rajectory-conditionee imitation learning and better leverage the past trajectories, we propose a supervised learning objective. We sample trajectory {(o,, Cp, dg, ro); (01, 61, (1, r....))) D, formulate the demonstration and assume the agent's incomplete trajectory the partial trajectory ISt e<c {eo, e₁, ez} for any 1 < L < lg]. Then a, is the 'correct' action at step for the agent to imitate the demonstration. Our supervised learning objective to maximize the log probability of taking such actions:
(eo, C1 G/p
SL
log No (arlese, ot, ot.g), g), where = (Co, e e/g)
4 EXPERIMENTS
In the experiments, we aim to answer the following questions: (1) How well does the trajectory- conditioned policy imitate the diverse demonstration trajectories? (2) Does imitation of the past diverse experience enable the agent to further explore more diverse directions and guide the explo- ration to find the trajectory with a near-optimal total reward? (3) Can our proposed method aid in avoiding myopic behaviors and converge near-optimal solutions?
5

---

Under review as a conference paper at ICLR 2020
4.0
Anstog 14at S4000
094
274
3010
20M
22H
10M
13M
30M
30M
090
317
2006
12500
2000
10M
0M
1.24
1.5
200
704
Step
(c) Curve of the number of state
Staa
(a) Curve of the average episode
(b) Curve of the best episode embedding clusters found during (d) Curve of the average imilation training
reward
ward found during training
success ratio
Figure 4: Learning curves Apple-Gold domain averaged over runs, where the curves in dark colors average over 5 curves in light colors. The x-axis and y-axis correspond to the number steps and statistics about the performance, respectively. The average reward and average imitation success ratio are the mean values over 40 recent episodes.
PPO+SIL
DTSIL
OM steps
3.2M steps
6.4M steps
9.6M steps
Figure 5: Visualization of trajectories stored the buffer for PPO+SIL and DTSIL. (ours) over time. The agent (gray), apple (red) and gold (yellow) are shown as squares for simplicity. The rocky region is in light blue.
We compare our method with the following baselines: (1) PPO:
Proximal Policy Optimization
(Schulman et al. 2017); (2) PPO+EXP: PPO with reward f(r)) + Nez). where A/ (e() the count-based exploration bonus, (e) the number of times the cluster which the state representation belongs to was visited during training and À is the hyper-paramete parameter controlling the weight of exploration term; (3) PPO+SIL: PPO with Belf-Imitation Learning (Oh al. 2018). More details about the implementation can be found in the Appendix.
4.1 APPLE-GOLD DOMAIN
The Apple-Gold domain (shown in Figure T is a simple grid-world environment with misleading rewards that can lead the agent to local optima. An observation consists of the agent's 's location (æt, yr and binary variables showing whether the agent has gotten the apple or the gold. state is represented
as the agent's location and the cumulative positive reward: e = (Ire yt. Σi= max(r:; 0)), indicating the location of the agent and the collected objects.
As shown in Figure 4a PPO, PPO+SIL, and PPO+EXP agents are stuck with the sub-optimal policy of collecting the two apples. In Figure 4b PPO+EXP agent could occasionally explore further and gather the gold with total reward 8.5. However, the agent does not replicate the good trajectory due to the negative reward along the optimal path and network forgetting about the good experiences. DTSIL marches forward on the right side the maze and achieves the highest total reward 8.5 within the time limit. Figure 4c shows the number of different state embeddings found during training.
In Figure 4d we show the average success ratio of the imitation during training. It is defined as follows: for given demonstration g {cg eg. ... : .% ). let be the index of the last visited state embedding in g when the agent's 's current episode terminates, then the success ratio of imitating is Ty] (i.e., the portion of trajectory imitated). Ideally. we want the success ratio to be 1.0, which rajjectoy-conndttonee indicates that the trajectory conditioned policy could successfully follow any given demonstration from the buffer. At 5M steps, the trajectories with the optimal total reward 8.5 are found, and our dnjectory-condiiionn
policy eventually imitates them well with success ratio around 1.0.
Figure 5 visualizes a learning process. PPO+SIL fails on this task because the agent quickly exploits good experiences of collecting the apples and the buffer is filled with the trajectories in the

---

Under review as a conference paper at ICLR 2020
Mortazuma's Resenge
Mi Nououmaa' Revange
Nontiauumas Roverge
Mordazuma's flevnnee
CTRB rrop
20000
30000
15000
10000
20000
5000
2000
10003
OM 100n 320H 400W 643H B0SM
3BON 3000
4304 SIMW RS3M
05
3804 323M
4009
GAEM B009
EM
3004 50EM abor siEw 5009
(a)
(h)
(c)
(d)
Pduil
Pizull
20000
5000
15000
4003
2000
2033
Stepe
Steps
Wepps
(g)
(h)
Figure 6: Learning curves of the average episode reward, the best episode reward, and the number of different rooms found on Montezuma's 's Revenge and Pitfall, averaged over runs. On Montezuma's Revenge, DT- SIL+EXP discovers around 40 rooms on average while PPO+EXP never finds a path to pass through all 24
rooms at the first level and then proceed to the next level.
nearby region. On the contrary. DTSIL maintains a buffer of diverse trajectories which are used as demonstrations to guide the agent to explore different regions and discover an optimal behavior.
4.2 ATARI MONTEZUMA'S S REVENGE AND PITFALL
We evaluate our method on the hard-exploration game Montezuma's 's Revenge and Pitfall in the Arcade Learning Environment (ALE) [Bellemare al.. 2013; Machado al. 2017). The environment setting is the same as Mnih et al. (2015). There is a random initial delay resulting in stochasticity in the environment. The observation is a frame of raw pixel images, and the state representation e, (room;, V/+: E max(r;; 0)) consists of the agent's ground truth location (obtained from RAM) and the accumulated positive environment reward, which implicitly indicates the objects the agent has collected"] on these two infamously difficult games, it is highly non-tre non-trivial to explore efficiently and avoid local optima without relying on expert demonstrations or being able to reset to arbitrary states. In addition to the agent's 's location information, many complicated elements such as moving entities, traps, and the agent's 's inventory are included in the state, Therefore, these Atari games with agent's location information are still much more challenging than the grid world environments. Empirically, as summarized in Table I. the previous SOTA basclines using the agent's ground truth location information even fails to achieve high scores.
2 It is worth noting that even with the ground tround-truth location of the agent,
Using the state representation Et, we introduce a variant 'DTSIL+EXP' that adds a count-based exploration bonus r = 1/ 1/vN(e)) to Eq for faster exploration As shown in Figure 6a and 6e in the early stage, the average episode reward of DTSIL+EXP is worse than PPO+EXP because our policy is trained to imitate diverse demonstrations rather than directly maximize the environment reward. Contrary to PPO+EXP, DTSIL+EXP agent is not eager myopically follow the high-reward path since the path with a relatively low score in the short term might lead to higher rewards in the long term. On Montezuma's Revenge, for example, with two keys in hand, PPO+EXP agent often opens nearby door and loses the chance of opening the last two doors of the first leve As training continues, DTSIL+EXP successfully discovers trajectories to pass the first level with total reward of more than 20,000, as shown in Figure 6b) While gradually increasing the probability of imitating the best trajectories in the buffer by sampling them as demonstrations, the average episode reward
We can also use the number of keys as an element in the state embedding as in (Ecoffet et al. 2019 to reduce the size of the state embedding space and improve the performance, shown in Appendix H
"Note that the existing exploration methods listed in T take advantage of count based exploration bonus (e.g.. A2C+CoEX+RAM SmartHash, DeepCS, and Abstract HRL). Therefore, combination of DTSIL and the count- based exploration bonus does not introduce unfair advantages over other baselines.
'Demo videos of the learned policies for both PPO+EXP and DTSIL+EXP are available at lat tttpps:/ // sites. google con/ www//viee//.eeooooooooo.oooeeoee view//a veve sil In comparison to DESIL+EXP, we could see the PPO+EXP agent does not explore enough to make best use of the tools (e.g. sword, key) collected in the game. A map of this level shown in Figure 16 Appendix.
7

---

Under review as a conference paper at ICLR 2020
Method
DTSIL AZC+COEX+RAM SmartHash DeepCS Abstract-HRL A2C+SIL PPO+CoEX RND
MontezumaRevenge
20,187
6,600
5,661
3,500
11,000
2,500
11,618
10,070
Pitfall 6,546
-186
10,000
-3
Table 1: Comparison with the state-of-the-arr results on Montezuma's Revenge and Pitfall. Abstract-HRL Liu ct al. (2019) assumes more high-level state information, including the agent's 's location, inventory and invetory (Choi et al. [2088. SmartHash (Tang el al.] (207], and DeepCS (Stanton
history, etc. DTSIL, +2C+CCEXX+BBB.
& Clune 2018) only make use of agent's location information from RAM, while A2C+SIL (Oh et al. 200 8).
PPO+CoEX (Choi al. 2018), and RND (Burda et al. 2018) do not use RAM information. The
averaged over multiple runs, gathered from each paper.
PPQ
FRGUSE
FRO--L
30M
Steps
Figure 7: The reward for getting the key, opening the door, and collecting the treasure (yellow block) is 1, 2, and 6 respectively. The learning curve of the episode reward is averaged over independent runs.
could increase surpass 20,000 in Figure 6a On Pitfall, the positive reward is much sparser and most of the actions yield small negative rewards that would discourage getting a high total reward in the long term. However, our method still stores the trajectories with negative rewards, encourages the agent to visit these novel regions and then discovers good paths with positive rewards as illustrated in Figure 6f Therefore we are able to eventually reach average episode reward over 0 in Figure (6e) without expert demonstrations.
Table D compares our proposed method with previous works without using any expert demonstration or resetting to an arbitrary state, where our approach significantly outperforms the other approaches which make use of the same information from RAM about the agent's location. In the Appendix C.3 we present more experimental results on other interesting environments with discrete action space such as Deep Sea (Osband al. 2019).
4.3 MUJOCO
We evaluate DTSIL on continuous control tasks. We adapt the maze enviromment introduced in (Duan al. 2016)) to construct a of challenging tasks, which require the point mass agent to collect the key, open the door with the same color and finally reach the treasure to get high score. One key cannot be re-used once was used before to open a door with the same color, which makes the agent to be easily trapped. A visualization of these environments is shown in Figure The agent's initial location is randomly sampled from a Gaussian distribution as standard MuJoco tasks (Brockman) al. 201 The observation is the agent's location and range sensor reading about nearby objects. The state representation is e (I's, St: E(=
As shown in the first maze of Figure 7 the agent can easily get the blue key near its initial location and open the blue door in the upper part. However, the optimal path is to bring the key to open the blue door the bottom and obtain the treasure, reaching an episode reward of 9. In the second maze, the agent should bring the blue key and pick up the green key while avoiding opening the blue door in the upper part. Then, the green and blue key can open the two doors at the bottom of the maze, which results in the total reward of 12. The learning curves in Figure show that PPO, PPO+EXP, and PPO+SII may get stuck at a sub-optimal behavior, whereas our policy eventually converges to the behavior achieving the high episode reward.
5 CONCLUSION
This paper proposes to learn diverse policies by imitating diverse trajectory-Ievel demonstrations through count-based exploration over these trajectories. Imitation of diverse past trajectories can guide the agent to rarely visited states and encourages further exploration of novel states. We show that in a variety of environments with local optima, our method significantly improves self-imitatin learning (SIL). It avoids prematurely converging to a myopic solution and learns a near-optimal behavior to achieve a high total reward.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Peter Auer. Using confidence bounds for exploitation-exxllraaion trade-ofts. Journal of Machine Learning Research, 3(Nov):397-422 2002.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930-2941, 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint arXiv: arXiv:/409.0473, 2014.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: / /2894, 2018.
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 1281-1288, 2005.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-aware exploration in reinforcement learning. arXiv preprint arXiv:1811.01483, 2018.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https
2017.
Honghua Dong. Jiayuan Mao, Xinyue Cui, and Lihong Li. Explicit recall for efficient exploration, 2019. /ttp://ppss//ppeeeees/eeeeeeeeee 1ttp://orrum22......11133333
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning. pp. 1329-1338, 2016.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, pp. 1087-1098, 2017.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint 190iv:1901.1009 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 2018.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. arXiv preprint arXiv:1705.06366 2017.
Tanmay Gangwani. Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. arXiv preprint arXiv: /805. 10309, 2018.

---

Under review as a conference paper at ICLR 2020
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. arXiv preprint arx.9:1711.09874. 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 161 2016.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Evan Zheran Liu, Ramtin Keramati, Sudarshan Seshadri, Kelvin Guu, Panupong Pasupat, Emma Brunskill, and Percy Liang. Learning abstract models for Long-horizon exploration, 2019. URL
https:
eew net forum 11w.mm11.rrryyyyxr.....
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2017.
IEEE,
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning, arXiv preprint arXiv: 05635, 2018
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. 2019.
Georg Ostrovski, Marc G Bellemare, Aäron van den Oord, and Rémi Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference Machine Learning-Volle Volume 70, pp. 2721-2736 JMLR, org, 2017
Tom Le Paine, Caglar Gulcehre, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, et al. Making efficient use of demonstrations to solve hard exploration problems. arXiv preprint urXiv: M..Xv:/9000000 1909.099877 0877, 2019.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp. 2050 2053, 2018
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learn- ing: Challenging robotics environments and request for research. arXiv preprint ar 1rXiv:1802.0944 2018
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Vežerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint 1805. 1593, 2018.
Vitchyr H Pong. Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supersuue reinforcement learning. arXiv preprint arXiv: 006908 2019.
Melrose Roderick, Christopher Grimm, and Stefanie Tellex. Deep abstract nettwwrrss. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 131-138. International Foundation for Autonomous Agents and Multiagent Systems, 2018
10

---

Under review as a conference paper at ICLR 2020
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from single demonstration. arXiv preprint arXiv:1812.03388 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320, 2015a.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint wrxiv1151.005952 2015b.
Jürgen Schmidhuber. Adaptive confidence and adaptive curiosity. Technical report, Citeseer, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /00::7777..00 1707.063777 2017.
Christopher Stanton and Jeff Clune. Deep curiosity search: Intra-life exploration improves perfor- mance on challenging deep reinforcement learning problems. arXiv preprint 1806000000 2018.
Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 7408):1309-131, 2008
Kaushik Subramanian, Charles I. Isbell Jr, and Andrea Thomaz. Exploration from demonstration for interactive reinforcement learning. In Proceedings of the 2016 International Conference Autonomous Agents & Multiagent Systems, pp. 447-456. International Foundation for Autonomous Agents and Multiagent Systems, 2016.
llya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057-1063, 2000.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances neural information processing systems, pp. 2753-2762, 2017.

---

Under review as a conference paper at ICLR 2020
Appendix
A DETAILED DESCRIPTION OF ALGORITTMS
A.1 ALGORITIM OF TRAJECTORY BUFFER UPDATE
In Algorithm 2
summarize how to process the collected episode and store the diverse trajectories in the trajectory buffer.
Algorithm 2 Update Trajectory Buffer
Input: the trajectory buffer D [(e(1) (1)), _())),, (e(2),((3) (22)),
Input: the current episode = {(o0, eo, ap, ro), (01, e1, a1, r1), .. (or,er,ar,))
Input: the threshold th for high level state embedding
Consider the states in for each step t do Consider state st and partial episode T<t = {(o,, cu, ao ro). ... ,o, e,, ae, r()
if there exists (c!8), ,,(8,, .n.(k) E where ||e! - eell < th then
# Compare partial episode Tse. with stored trajectory (k)
if Tse has higher total reward or reaches the same total reward with less steps then (kk
Tst {(oo eo, ro) (01, c1, (1,,,)))..... (or, Et, Tt)
(()) c
end if
(k) + (k) +1
else
D - DU (ee, Tea: 1) where Tst = {(o0, co, ag, To): (01, E1, ai)) ,a, (or, ac, r()
end if
end for
return D
A.2 ALGORITHM OF SAMPLING DEMONSTRATIONS
In Algorithm 3 summarize how to sample the demonstrations from the trajectory buffer for exploration or imitation. Considering the current iteration and the total number of iterations 1, the probability of sampling demonstration for imitation to learn good behavior and the probability of sampling demonstration from exploration is
Algorithm 3 Sample Demonstration Trajectories
Input: the trajectory buffer D (e(*), (2), n))), (e(2), 7(2), n())), n..)),.... Input: current iteration i, total number of iterations I.
# With probability 1, run the imitation mode; with probability 1- if random number U0, 1] is smaller than then # sample the top-K trajectories reaching hear-optimal score in the buffer g - {eu, "e..,e......ee ,eg) for all (ot, Ct. oe, rt) E best
the exploration mode
else
Calculate probability distribution p - 1 Vnizi
E Sample (e, Categorical(D,p)
g <- {ea, 61, ) for all (or, Ez. 0e, r()
end if
return g
12

---

Under review as a conference paper at ICLR 2020
B DETAILS OF NETWORK ARCHITECTURE AND TRAINING PROCESS
In the trajectory-condrtioned policy (Figure 3 we first embed the input state 6, (or e.) with a fully-connected layer with 64 units. Next, RNN with gated recurrent units (GRU) computes the feature h, (or hg) with 128 units. The attention weight a; is calculated based on the Bahdanau al. 2014). The concatenation of the attention readout et the
attention mechanism (Bahdanau hidden feature of agent's s current state hy, and convolutional features from the observation are used to predict π (arlegs, Ot, 9) with linear layer.
For experiments on the Apple-Key domain, Toy Montezuma's Revenge, and Mujoco, the features from or are not required for the policy. However, on the Atari games such as Montezuma's Revenge, it is necessary to take the raw observation o, as input into policy because the location information esrr solely could not let the agent to take temporal context into account (e.g. avoiding moving skulls and passing laser gates). With the raw observation or with shape 84 84 X input, three convolutional layers are used to encode o and then the convolutional feature is flattened.
Readout Ct
x(a Or.g)
Attention Cit
V(ege, 0t.
1
Ci
Tpl
Demonstration Trajectory g
ho
h1
hr
Agent's Trajectory
Observation
Figure 8: Architecture of the
policy (Repeating Figure
During training, our algorithm begins with an empty buffer D. We initialize the demonstration as a list of zero vectors. With such an input demonstration, the agent performs random exploration to collect trajectories to fill the buffer D. In practice, the sampled demonstration trajectory g = {eg ei. ... E could be lengthy. We present part of the demonstration the input into the policy, similarly to translating a paragraph sentence by sentence. Specifically, we first input {eă eq ... eeg (m < (g]) into the policy. When the index of the agent's last visited state embedding in the demonstration u belongs to {m - At,- ... m). we think that the agent has accomplished this part of the demonstration, and switch to the next part {ee, eg 10+11" Cutm }. We repeat this process until steps long,
the last part of the demonstration. If the last part {eg ed 4+11" "411""" e is less than we pad the sequence with zero vectors.
A reward function f(rt) Ft is used the Apple-Gold, Deep Sea and MuJoco domain, and f(r)) = 2 clip(re,0, 0, on other environments. pim 0.1 is the reward to encourage imitation. More details about hyperparameters and the environments can be found in the Appendix D
13

---

Under review as a conference paper at ICLR 2020
C ADDITIONAL EXPERIMENTS
C.I GENERALIZATION AND ROBUSTNESS IN STOCHASTIC ENVIRONMENTS
We evaluate our method on environments with different levels of stochasticity. For Apple-Gold domain, in the environments with random initial location of the agent (Figure 99, with sticky action (Figure I0). our DTSIL still outperforms the baselines and achieves near-optimal total episode reward.
250
200
04
200
812
1064
34H
3291
409
S16.3
06
18M
24H
32N
Stops
Strps
Curve of the number of state embeekiing clusters found during (d) Curve of the average imitation success ratio
Stops
Curve of the averago opisode reward
(b) Curve the best episode
ward Found during training
training
lower corner, where the curves in dark colors are average over curves in light colors. The x-axis and y-axis correspond to the number of steps and statistics about the performance, respectively. The average reward and average imitation success ratio are the mean values over 40 recent episodes.
Figure 9: Learning curves on Apple-Gold domain with random initial location of agent
Eoorage B1S aot SUCCEIE 5a30
Dae Tawars
230
2404
Staps
(c) Curve of the number of state
(a) Curve of the average opisode (b) Curve the beat episodo ro- reward
embeddding clusters found during (d) Curve of the average imitation ratio
ward found during training
training
Figure 10: Learning curves on Apple- Apple-Gold Gold domain with sticky action of the agent, where the curves in dark colors are average over 5 curves in light colors. The x-axis and y-axis correspond to the number of steps and statistics about the performance, respectively. The average reward and average imitation mean values over recent episodes.
ratio the
In more challenging scenario when the location of the objects can be random, previous works using expert demonstrations (e.g. |Aytar et al.) (2018) would also struggle. Our method can be easily extended handle this difficulty by ditilling the behavior of good trajectories we collected train an unconditional policy robust to the stochasticity. For example, on the Apple-Gold domain with pixel observation (Figure [T]. the location of the gold could be random in the upper middle part of the maze. We first explore for a sufficiently large number of timesteps (e.g. 10M timesteps) with the rrajectoyy-condiiionee policy collect good trajectories and then train an unconditional policy by distilling the behavior, shown in Figure 11 to always collect the gold. We could see DTSIL with unconditional policy training able to generalize in the stochastic environment.
290
Aeerge Browe
rrowco
0.0
250
0.0
Nurber
Avtrgee
444
05
1259
1BM
20H
09
422
12M
10M
2007
of the number of state
Stapo
122
16M
25M
(a) Curve of the average opisode reward
(b) Curve the best episode
embeekling clusters found during (d) Curve of the average imitation success ratio
ward found during training
training
Figure 11: Learning curves Apple-Gold domain with stochastic location of the gold in the upper middle part of maze, where curves dark colors are average correspond to the number of steps and statistics about the performance. respectively. The average reward and average imitation success ratio are the mean values over 40 recent episodes.
curves light colors. The x-axis and y-axis
14

---

Under review as a conference paper at ICLR 2020
C.2 EXPERIMENTS ON TOY MONTEZUMAREVENGO
We evaluate our method on a more challenging domain, Toy Montezuma's Revenge (Roderick et al. 2018), which requires more sophisticated strategy explore the environment. As shown in Figure 12 there are 24 rooms similar to the layout of the first level Atari Montezuma's 's Revenge, with
discrete grid for each room. The agent should navigate the labyrinth locate the keys, unlock the doors and reach the goal (the treasure room). The observation is represented by the agent's location and cumulative episode reward. The state representation as the observation.
(room,, E/=1 r;)) is the same
The learning curve of the averaged episode reward in Figure 13 shows that PPO. PPO+SIL, and PPO+EXP could not learn a policy to reach the goal. The PPO+EXP agent occasionally finds trajectory with the total reward of 11 11,200 reaching the treasure room, but fails to exploit this experience. On the other hand, our method learns a good behavior of not only reaching the goal room, but also collecting all of the keys to achieve an optimal total reward of 11,600.
32006
33060
PPQ
30000
20000
Avarage
Stapa
Sapa
Humber taad
2000
1..8
4200
Figure 12: Map of Toy Montezuma's 's Revenge, where we show the agent (gray), key(blue), door(green), and treasure (yellow) as squares. The rewards are 100, 300, and 10000, respectively. An optimal path with the high- est total reward of 11,600 is shown as red line.
Srags
Stepe
Figure 13: Learning curves on Toy Montezuma's
Revenge averaged over 5 runs.
15

---

Under review as a conference paper at ICLR 2020
C.3 EXPERIMENTS ON DEEP SEA
As introduced in Osband et al. 22119), the deep sea problem is implemented as an N x N grid with one-hot encoding for state. The agent begins each episode in the top left corner of the grid and descends one row per timestep. Each episode terminates after N steps, when the agent reaches the bottom row. In each state there a random but fixed mapping between actions A = {0. 1} and the transitions 'left' and 'right'. At each timestep there is a small cost -0.01/N of moving right, and r 0 for moving left. However, should the agent transition right at every timestep of the episode will be rewarded with an additional reward of +1. This presents particularly challenging exploration problem for two reasons, First, following the 'gradient' of small intermediate rewards leads the agent away from the optimal policy. Second, a policy that explores with actions uniformly at random has probability 2 of reaching the rewarding state in any episode.
We compare DTSIL and baselines on deep environments with 10 10 grid and and 30 x 30 grid. The state embedding we use here is exactly the observation. The result is shown in Figure [4] On the first environment, it is casy for all of the methods to converge to the optimal behavior. The second one is much more challenging to find the optimal trajectory maximizing total reward. Therefore, PPO and PPO+SIL fails such environment due the hard exploration. PPO+EXP could not always explore to find the good behavior and exploit it efficiently within 12M timesteps. DTSIL successfully discovers the right way and imitate converge the optimal behavior.
Sea 10x10
Deep Sen 10x10
Deep Sea 10x10
Sea 10x10
1.2
2.0
0.5
PRO-5A
0.8-
52.5
50.0
0.0
0.G-
47.3
0.6
0.3
42.5
0.6
00.0
0.2
0.0
37.5
072
1,2M
Stops
S0024
Seape
Steps
Deep Sea 30x30
Deep Sea 30x30
Deep Sea 30x30
RAC-9R Wilathr Sa0000 1
2.0
Deep Sea
0.6
0.2
0.6
322
0.4
04
232
0.0
Figure 14: Experiment on Deep Sea. The learning curves shows the average episode reward, best episode reward. the number of found state representations, and the average success ratio of imitating the demonstrations order. The curves averaged over independent runs.
16

---

Under review as a conference paper at ICLR 2020
D HYPERPARAMETERS
The hyper-parameters for our proposed method used in each experiment are listed Table On Mujoco environment, RL loss alone worked well so we did not include SL loss for behavior cloning. On the other environments when action prediction in behavior cloning is poor, we set a large J for quickly learning imitate demonstrations. When action prediction accurate enough, we de-emphasize behavior cloning to enhance exploration around the demonstration. At influences how flexibly the demonstration should be followed. our experiment, we have AL < m due the limit of the length of the input demonstration part. When the demonstration is longer and harder to follow, we would want larger AL to generously provide imitation reward (More detailed ablation study of the hyper-parameter Ae is in Appendix J. On Atari games, there are much more different trajectories stored the buffer, so we sample top-100 trajectories as demonstration for imitation of best experiences. On the other environments, the total number of trajectories is much smaller, only take top-10 or top-1.
Environment
Golde-Gold
Toy MontezumaRevenge
Atari
Deep Sea
Mujoco
Learning Rate
2.5e-4
2.5e-4 4
2.5e-4
2.5e-4
le-4
At
2
Length of demonstration
10
10
10
10
10
input part m
10 decreases
10 decreases to when action prediction accuracy 2 0.75
10 decreases to when action prediction accuracy > 0.75
10 decreases to when action prediction accuracy 2 0.75
Number of supervised
when action prediction
learning updates J
accuracy 2 0.75
Threshold th for
state embedding
Top-K K trajectories imitation
10
100
10
10 (hest one among 5, 10, 20, 50) 0.99
0.2 (best one among 0.1, 0.2, 0.5, 1) 0.99
Weight of exploration À in EXP
(best one among 0.5, 1, 2,4)
(best among 0.5, 1, 2,4)
(best one among 0.5, 1, 4) 0.99
Discounting factor
0.99
0.99
Table 2: parameerrs on various environments for our experiments.
17

---

Under review as a conference paper at ICLR 2020
E ENVIRONMENT SETTING
For each experiment we conducted, we list the detailed environment setting in Table 3 There is stochasticity in the environments of Apple-Gold domain, Toy MontezumaRevengee Atari, and Mujoco. On Atari games, we use setting of the random initial delay introduced in (Mnih et al. 2015). On Mujoco domain, the agent's initial location randomly sampled from a Gaussian distribution, in standard MuJoco tasks in OpenAl Gym (Brockman et al. 2016).
Environment
Apple-Gold
Toy MonlezumaRevengel
Atari
Deep Sea
Mujoco
agent's s location (x, y) in 17x13 grid and binary variables indicating whether apple or gold collected
agent's 's location (x.y) in 22x22 space and range sensor reading about nearby objects
agent's location (room, x. y) in 24x11x11 grid and accumulated reward
stacked most recent gray observations with shape 84x84x4
me-hot encoding of state in 10x 10 30x30 grid
Observation
rromm, It. Me:
State Representation
E max(r:, 0)) where (wa,, 3c) in 9 9 grid 18 discrete actions:
[It, Vo. Σ max(r;, 0))
(room: Ft, Vt. Ei_ r.)
same as observation
(#e. Ut. Ei_ r.)
discrete actions: up. down, left, right, noop
5 discrete actions: up, down, left, right, noop
[dr, dy) in continuous action
discrete actions: left, right
Action
noop, fire, left,
space
Of going left
0.01 or 0.01 30
rock -0.05 gold +10 apple +1
key +100 door +300
mostly zero, sparse positive rewards
10
key +1
going right
Reward
door +-2
at the
treasure + 10000
when collecting objects
treasure +6
last step always going right 10 or 30 steps
Time limit
45 steps
1000 steps
4500 steps [MontezumaRevenge Pitfall]- NoFrameskip-ee v4; take random number
1000 steps
deterministic or take random steps before the episode starts or sticky action
take 5 random steps before the episode starts
take random normal noise from the agent's 's initial position
Stochasticity
deterministic
between 0 30 of noop actions
before the episode starts
Table The setting on various environments for our experiments.
18

---

Under review as a conference paper at ICLR 2020
F COMPARISON WITH LEARNING DIVERSE POLICIES BY SVPG
While the code for the Stein variational policy gradient (SVPG) in Gangwani et al. 2018) has not yet been released, we replicate the method in Gangwani et al. (2018) learn diverse policies. Their experiments focus on continuous control tasks with relatively simple observation spaces with limited local optimal branches in the state space. We learn 8 diverse policies parallel following their method on our Apple-Gole Gold domain with discrete action space. Figure II]sooss environment. The method
shows visualization of
the learning progress: the policies learn to cover different regions explores better than PPO+SII,, but the exploration of each individual agent is not strong enough to find the optimal path to achieve the highest episode reward.
PPO+SIL
SVPG diverse
DISIL
OM steps
5M steps
10M steps
15M steps
Figure 15: Visualization of the trajectories stored in the buffer for PPO+SIL, SVPG diverse Gangwani et al. (2018) and method training continues. In the second
show the trajectories for total of policies learned simultancously with SVPG method proposed (Gagwani et al. (2088) where each color corresponds to the trajectories collected by each policy.
19

---

Under review as a conference paper at ICLR 2020
G MAP OF ATARI MONTEZUMA'S REVENGE AT THE FIRST LEVEL
On Montezuma's 's Revenge, there are multiple levels and each level consists of 24 rooms. A map of Atari Montezuma'ss 's Revenge at the first level is shown Figure [6 It is challenging bring two keys to open the two doors in room 17 behind the treasure in room 15, where the agent can pass to the next level.
0
10 11 12
13
14
16
18
Figure Map of Atari Montezuma's Revenge at first level with rooms. On Montezuma's 's Revenge, there are multiple levels and each level consists of rooms. At the first level, is challenging to bring two keys to open the two doors in room 17 behind the treasure in room 15, where the agent can pass to the next level.
20

---

Under review as a conference paper at ICLR 2020
H STUDY OF EXPLORATION EFFICIENCY
To evaluate the efficiency of exploration, we compare our method with the "exploration phase" in the Go-Explore algorithm (Ecoffet et al. 2019). The idea behind Go-Explore is reset the agent to any interesting state sampled from the buffer of state embeddings, and then explore further using random actions. To study the exploration efficiency of our method, we modify the Go- Go-Explore code such that we could not reset to any arbitrary states in the environment. Similarly to (Ecoffet et al. 22019), we use the state representation (levelt, room, Jt: Ut. ky) where k; is the number of keys the agent holds and (Iz, yt) is in 9 x 9 grid division of the frame, and the sampling weight Vnt to sample goal states from the buffer (It is worth noting that the state representation and goal-state sampling function recommended in Go-Explore paper is more complicated than this setting).
In the Go-Explore method without using the direct 'reset' function and with perfect goal-conditioned policy to visit any state sampled from the buffer, the agent could precisely advance to the goal state by following the stored trajectory. The total steps taken in the environment are counted by summing the number of steps taken to follow the stored trajectories and the number of steps taken to explore.
Montezumaas Revenge
Montezuma's is Revenge
Dist.tb
Number of Founss Itom
Number Found Stote
4000
20
2000
0M
160M
320M
4B0M
540M
800M
0M
160M
320M
640M 800M Stees
4.0M
Steps
Figure 17: Learning curves of the number of rooms and the number of different state representations found on Atari Montezuma's 's Revenge, averaged over 5 runs. The curves in dark colors are the average of the curves in light colors. During training, the state representation used is (level,, room,, Zt, Vt, kt).
In Figure 18 we show the average number of rooms found and the number of different state representations found during training. Even if we assume that there is perfect gond-condi policy in Go-Explore to guide the agent to follow the stored trajectory exactly and visit the goal state, the learning curves demonstrate that our method is more efficient for exploring diverse state representations and consequently visits several rooms. This is because our method uses the count- based exploration bonus to encourages the exploration around and beyond the stored trajectories and the imitation reward allows the agent to follow the demonstrations in a soft-order.
Mantewtmaas Revergee
Mortlenume's Beverge
Montermme's Revenge
Ko-tewtmee' Revergee
355000
Marber Prom
32 .0003
cmai Kkop
3000
20000
30000
52000
04
100M 330M
KEOM CAOM DOON
OM 200M 300M 4ECM CROM DOOM
300M 69
320M
420H
E40M
B0CM
100M 300M 4D0M E4OM B20M
Stype
Figure 18: Learning curves the average reward, the best episode reward, the number of rooms, the number of different state representations found on Atari Montezuma's 's Revenge, averaged over 5 runs. The curves in dark colors are the average of the 5 curves in light colors. During training, the state representation used is (level,, room,, Ic, yt. k.).
In addition, we notice that, comparing with the state embedding (room;, Ft, Yts Ei=1 r;), the state em- bedding (level,, room, Zt, Vt. ke) makes the size of embedding space smaller so that the exploration could be more efficient. Such state representation conflates similar states while not conflating states that are meaningfully different. Therefore, our method could reach higher average score around 29,817. Here, the baseline PPO+Exp is essentially the CoEX method introduced in Choi et al. (2018) with state embedding extracted from RAM, therefore DTSIL performs better than PFO+CoEX+RAM
no matter the state embedding is (room,, Ft, yt, Σi=1 r's) or (level, room,, Ft, Vt, kz).
21

---

Under review as a conference paper at ICLR 2020
I STUDY OF ADVANTAGE IN EXPLOITATION
We compared our method DTSIL with PPO+EXP in the main text. PPO+EXP encourages exploration novel states by providing auxiliary rewards the agent, while our method rewards the agent when it successfully follow the demonstrations which leads to novel states. In order to understand more about the difference between these two mechanism, we propose variant of our method denoted as "DTSIL-combine". In this variant, we do not separate the exploration mode and imitation mode. However, we always sample the top-K best trajectories with highest total reward E (f(r)) (ee) The PPO+EXP baseline directly optimize the objective (f))(f(r)) + /Nie.) via the
reinforcement learning algorithm while this variant indirectly optimizes such objective by imitating the best trajectories with highest value of (f (M())) We investigate the different
performance of these two methods on the Apple-Gold domain.
DTSIL-cmmbee
Nvemage Total Rewand
60
Averago Rowardd
550
40-
Best
20
10
0:
10M
Steps
0M
5M
10M
Stops
15M
20M
25M
091
5M
10M
Steps
355
20M
25M
Curves of the average
(b) Curves the averuge total reward
(c) Curves of the best environment reward of the agent's episodes found during train- ing
rewards S Pt agent's recent episodos
El f(ra) +
(Mies of the agent's
10
recent episodes
Figure 19: Learning curves averaged over 5 The curves in dark colors the average of the 5 curves in light colors.
With A 10 which the best typer-parameter we found after searching PPO+EXP on Apple-Gold domain, in Figure I9c we could notice both PPO+EXP agent and DTSIL- combine agent have 3 out of runs finding the optimal trajectory with episode rewards 8.5 and the other 2 runs get stuck at the sub-optimal behavior. However, in Figure 19h DTSIL-commine is better at optimizing the objective Σ/r f(r)) + (N(e.)) averaging over the agent's 's recent episodes and
5, 10, 20, 50 for
therefore achieves higher environment environment reward as training goes on. As shown in Figure 19a DTSIL-combine agent reproduce the good trajectories to collect the gold in 3 out of 5 runs while PPO+EXP agent is trapped at the behavior collecting the apples. The main reason might be that D'TSIL-combine agent never forgets the good experience of collecting gold, and always select such good trajectories as demonstrations to guide the agent, while PPO+EXP might forget the good trajectories occasionally found or fails exploit them before the exploration bonus vanishes. The importance of exploitation of the good experience to help the agent reproduce high-reward trajectories also discussed in Oh et al. (2188).
22

---

Under review as a conference paper at ICLR 2020
J ABLATION STUDY OF HYPER-PARAMETER At
We study the effect of the hyper-paramete At in the various environments. At each step, we provide m 10 (considering the computational burden, we selected the value of m 10 for all experiments) steps of state embeddings from the demonstration trajectory as input into the policy. Then we evaluate whether the agent has visited any of the next Át state embeddings from the lastly visited state embedding in the demonstration. It worth to note that the last visited state embedding must be included in the part of policy input. Thus, we should only compare the agent's current state embedding with the state embeddings in the demonstration segment provided into the policy for imitation reward. Therefore, the proper value of At should be less than m 10 and we consider At 2, 4, 8.
On the simple environments such as Apple-Gold and Deep Sea, we could see Figure 20a and 20b the different values of At do not influence the performance a lot.
As shown in Figure 20c and 20d] on more challenging environments where the demonstration trajectory is longer and it's harder to learn to imitate the demonstration, it's better to set a larger value of At and we can provide imitation reward to the agent and encourage the imitation learning more generously. In general, allowing the agent some flexibility of imitation by setting At close m works well.
In summary, At 8 is proper value for all of our primary experiment environments.
Apple- Gold
Deep
Mencozuma's Revenge
22000
Avd 92 Naveed
20000
5003
0121 26
2.44
25 1000 520M 400M 6O0M 8030
239
Stees
(b) Curves of the average (c) Curves of the average (d) Curves of the average episode reeent 40 episodes on Apple-Gold domain 40 episodes on Deep Sea 40 episodes Montezuma's episodes on our first maze in Revenge
(a) Curves of the averuge episode episode rewards the recent episode rewards over the recent rewards
rewards over recent 40
cavironment
MuJoco environment
Figure 20: Learning curves averaged over at least independent runs. The curves in dark colors are the average of the curves in light colors.
23

---

Under review as a conference paper at ICLR 2020
K STUDY OF THE STOCHASTICITY IN THE ENVIRONMENT
As listed in Table we consider stochastic environments for all the primary experiments, including the Apple-Gold domain, Montezuma's is Revenge, Pitfall, and Mujoco maze. We the initial state of the agent to be random. On the Apple-Gold domain, the agent takes random steps in the left bottom part of the maze before the episode starts. On Montezuma's 's Revenge, the mechanism of random initial no-ops is one of the standard ways to introduce stochasticity in the environment (Machado et al. 2017). , as in previous work (Mnih al. 2015).
To show the difficulty in policy learning introduced by the stochastic environments, we tried to memorize and repeat the action sequence in the demonstration trajectory and check whether the agent could successfully visit the state of interest by following the demonstration. In Figure eggure2ccaan and [22c it clear that the success ratio in imitation is much lower than DTSIL, because the agent could not successfully follow the demonstration by just repeating the action sequence step by step. Especially on Montezuma's Revenge, with random initial no-ops, the state the enemy and electricity beam when the agent starts moving is randomized. Thus it could not successfully avoid death by just repeating the stored action sequence.
Obviously, when the environment is completely deterministic, repeating the action sequence could perfectly guide the agent to the state of interest. However, with the stochasticity of the random initial state, just memorizing the action sequence is sufficient to make the agent revisit novel regions. Thus, the agent could not revisit the novel regions as efficiently as DTSIL to discover better trajectories and converge better total episode reward, shown in Figure 21a
21b
22a
22b
Apple-Gold
Apple-Gold
Ratio
2.0
2118 williad warnig repaut
0.8
Agc Panara
Best Reward
Initsaion
0.6
0.4
0.2
2-
1SM
20M
25M
0M
5M
20M
15M
20M
25M
(a) Curves of the average episode rewards over recent 40 opisodes
Steps
Curves the episode rewards found during training
(c) Curves of the average successs ratio in imitation
resent episodes
Figure 21: I Learning curves averaged over independent runs. The curves in dark colors are the average of curves in light colors. The Apple-Gole Gold domain is with random initial state of the agent.
Montezuma's 's Revenge
Montezuma's 's Revenge
Beveenge
Rabro
1.0
019 00p momer moseerce
100000
80000
1903E
60000
Auerge Initettonn
0.6
40000
0.4
20000
0.2
0M4
100M S00M 4830 eRcH sCOM
0.0
Stepe
300M 320M 4B0M GOOM BOOM Stops
0M
160M
320M
4B0M
CROM BOOM
(a) Curves of the average episode rewards over the recent 40 opisodes
(b) Curves of the best episode rewards found during training
(c) Carves of the average success ratio in imitation over the recent episodes
Figure 22: Learning curves averaged over 3 independent runs. The curves in dark colors are the average of the curves in light colors. The Montezuma's Revenge is with random initial delay of the agent.
24

---

Under review as a conference paper at ICLR 2020
L RANDOM EXPLORATION WITH EPSILON-GREEDY POLICY
In this section, we investigate an additional baseline method, random exploration with epsilon-greeds policy, which is traditional exploration method in reinforcement learning problems. We consider combining epsilon-greedy policy with PPO or DQN framework and run experiments on Apple-Gold. Deep Sea, and Montezuma's Revenge. Our implementation is based on OpenAl baselines (Dhariwal et al. 2017).
for the scheduling of the epsilon, though the performance is not better than DTSIL. Especially, random ex exploration performs poorly on Montezuma's Revenge. The average score is less than 100, which consistent with the experimental results from previous works (Mnih et al. 2015 Schhlman et al. 2017)
As shown in Figure 23 25 we searched different values of the hyper-parannnn
Apple-Gold
Apple-Gold
2.0
PRO epsioon deceé 805
Apple Gold
epailon-1 dece6.002
2.0
Averaga Roward
1.5-
Average Rewerd
3.5
epailoned deceb.00:
1.5
1.0
1.0
1.0
0.5
epation12 2. 8.35
spuiton-1 decay-
0.5
decay
0.0
0M
SM
10M
15M
20M
25M
DM
5M4
10M
15M
20M
25M
0.0
CCK
Steps
Steps
5M
30M
(a) With PPO framework, the value of scheduled exponential way e f- decay). The value of stars from epeilon annotated the legends and de- creases until O.
(b) With PPO framework, the value of is scheduled a lineur way
(c) With DQN framework, value of scheduled in the linear way, from (at the start training) (at end of training)
decrement).. The value of sturt from epsilon annotated in the legends and creases until 0.
Figure 23: Learning curves of the average episode rewards over the recent 40 episodes. On the Apple-Gole Gold domain, DTSIL reaches episode reward of 8.5 as shown in Section 4.1 while the random exploration with epsilon-greedy policy gets stuck at the episode reward of 2.
Deep Sea 30x30
Deep Sea 30x30
PPG fanm1 decay=2.25 PIO epoilon-1 decay-3.50
PPO applione2 dec-3.c05
Deep Sea 30x30
SPD apelion1 der 3.000
Avsrgge Relward
PPQ Ronw decoy=i.35
PPO apellon- 0.5 decay-Ca
Average Rewerdd
0.001
PPD apelion: 1 dec 3.001
#PO spoioon0.5 sece6.005
-0.933
D2M
-0.00
-3.009
#PD opekioedss seced 202
spaliien cec.6.001
0.0 0.003
0.005
Average
0.004
0.004
025
-3.005
2.401
4.35
9.6M
Steps
Steps
04
2.40
4.00
7.2M
3.0M
33
(a) With PPO framewwrk, vallue scheduled in exponential way (i.c, e 4- docay). The value of e starts from op- silon annotated in the logends and de- creases until D.
With PPO framework, is scheduled in linear way (ic. e decrement). The value of e starts from epsilon annotated in the legends and de- creases until O.
Stops
(c) With DQN framework, the value schesluled the linear way, from the start of training) to (at the end of training)
Figure 24: Learning curves of the average episode rewards over the recent 40 episodes. On the Deap Sea environment with = 30, DTSIL reaches episode reward of as shown Appendix C.3 while the random exploration with apsilon-greed, policy gets stuck at the negative episode reward.
25

---

Under review as a conference paper at ICLR 2020
Moncezuma's 's Revenge
Montesuma's & Revenge
FRD epsioo deay
epdizn.1 decoment
Monteruma's Revenge
40
deuayed
42-
D0W
Avorae Rovaii
FFG selitnndd decay-0.59
Acerage Reevarc
PP3 epalicen5 cecrement-d
30
PPO qyslenwe 5 recrement-- B02
PPD squlioeed cecremant-d 001
12
11
09
1:600
100M
611M
K00M
34
160M
3035
640M
2019
Stegs
Steps
3,2001
400
640M
500M
(a) With PPO frameworks, the value scheduled in exponential way x decay). The value of starts from legends
(b) With PPO framework, the value of e is scheduled lincar way (i.c. decrement). The value of starts from epsilon annotated in legends creases until 0.
Steps
(c) With DQN framework, the value of is scheduled the lincar way. from the start of training) 0 the end of training)
epsilon
creases until D.
Figure Learning curves the average episode rewards over the recent 40 episodes. the Montezuma's Revenge, DTSIL reaches episode reward over 20,000 as shown in Section 4.2 while the random exploration with rysilon-greed policy could not reaches score over 100.
26


Reviews:
[{"review_id": "ryexs85aYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the challenge of hard exploration tasks. The approach taken is to apply self-imitation to a diverse selection of trajectories from past experience -- practice re-doing the strangest things you've ever done. This is claimed to drive more efficient exploration in sparse-reward problems, leading to SOTA results for Montezuma's Revenge without certain common aides.\n\nThe approach is incompletely motivated. Why trajectory-conditioned policy over just goal-conditioned policy? The note in the related work section doesn't paint a clear enough picture. The trajectory buffer management strategy feels complex. Why this use strategy specifically? Could a simpler design be ruled out? In 2019 (post Go-Explore), it's not clear Montezuma's revenge poses a significant exploration challenge -- exploration doesn't even need to be interleaved with learning. Why are these three the right domains to show off these techniques?\n\nThis reviewer moves to reject the paper primarily for not balancing the high complexity of the solution to the lower difficulty of the problem. Pure-exploration algorithms (Go-Explore), not burdened by interleaving policy learning, achieve far superior scores. If the authors want to escape the shadow of this kind of technique which cheats by some framings of RL, more appropriate demonstration environments must be selected.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJgVvrnCKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors identify and address the problem of sub-optimal and myopic behaviors of self-imitation learning in environments with sparse rewards. The authors propose DTSIL to learn a trajectory-conditioned policy to imitate diverse trajectories from the agent’s own past experience. Unlike other self-imitation learning methods, the proposed method not only leverages sub-trajectories with high rewards, but lower-reward trajectories to encourage agent exploration diversity. The authors claim the proposed method to be more likely to find a global optimal solution. \n\nOverall, this paper is well-written with comprehensive experimental results. The proposed trajectory-conditioned policy sounds, since rewarded trajectory carries significant information of the goal in the exploration problem. Extensive experimental results demonstrated the effectiveness of the proposed DTSIL. However, I have a few concerns below, that prevent me from giving a direct acceptance. \n\n1. The proposed DTSIL changes the original MDP with sparse reward to an MDP with denser reward, which allows the training process to explore more in the “space” closer to the collected high reward trajectories. Such “exploration” sounds promising. However, it would be nice to compare it with traditional reinforcement learning (e.g., with \\epsilon-greedy policy for random exploration)?\n\n2. In appendix D, the authors discussed what the parameter \\delta_t controls, however, it is unclear how \\delta_t should be chosen in implementation. The authors did not explain how \\delta_t was selected in their experiments. Choosing the right \\delta_t may be hard, but it would be nice to introduce what “heuristics” the authors used and suggest to readers. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SyxtVdWg9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Note: the style-formatting of this paper has been heavily tweaked, and so the evaluation should be calibrated for a 9-page paper.\n\nThis paper proposes an approach for diverse self-imitation for hard exploration problems.  The idea is leverage recently proposed self-imitation approaches for learning to imitate good trajectories generated by the policy itself.  By encouraging diversity in the pool of trajectories for self-imitation, the idea is to encourage faster learner -- this basic concept is also used in approaches like prioritized experience replay, albeit at the entire trajectory level rather than individual state/action level.  \n\nThe authors view this approach as a generalization of Go-Explore, since it does not rely on having a reset mechanism.  However, I think this discussion has a lot of subtle nuances pertaining to the stochasticity of the environment (which the authors acknowledge).  For instance, if the environment is deterministic, then why not just do something like Go-Explore, since state-reset is just memorizing a deterministic action sequence? \n\nThe empirical results are very strong, achieving state-of-the-art results for any approach not reliant on a reset mechanism.  All the primary experiments appear to be for deterministic environments.  The results on stochastic environments (in the Appendix) seem pretty weak (but please correct me if I'm mistaken here).  So one major question is whether Go-Explore is a scientifically appropriate benchmark to compare with for this setting.  \n\nIn summary, I'm willing to be convinced that this is an interesting and scientifically novel result.  I have some concerns as expressed above.\n\n\n**** After Author Response ****\nThanks for the response.  I'm willing to raise my score to weak accept.  \n\nI think the authors did a reasonable job addressing my specific questions.  Some further reflection revealed to me that there is a huge opportunity to scientifically investigate how stochasticity impacts the proposed algorithm.  For instance, one could conduct a systematic study (say of the Apple domain) where one varies the degree of stochasticity and measures how the performance the proposed algorithm changes, perhaps relative to Go-Explore on the purely deterministic version of the environment.  It seems a bit of a cop-out to say that Go-Explore is not applicable, and misses out a huge opportunity for real scientific understanding.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of exploration in challenging RL environments using self-imitation learning. The idea behind the proposed approach is for the agent to imitate a diverse set of its own past trajectories. To achieve this, the authors introduce a policy conditioned on trajectories. The proposed approach is evaluated on various domains including Atari Montezuma's Revenge and MuJoCo.

Given that the evaluation is purely empirical, the major concern is in the design of experiments. The amount of stochasticity induced by the random initial state alone does not lead to convincing results regarding the performance of the proposed approach compared with baselines (e.g. Go-Explore). With such simple stochasticity, it is not clear why one could not use a model to recover from it and then rely on an existing technique like Go-Explore. Although this paper tackles an important problem (hard-exploration RL tasks), all reviewers agreed that this limitation is crucial and I therefore recommend to reject this paper.
Decision: Reject