Impact-driven Exploration with Contrastive Unsupervised Representations
Authors: Min Jae Song
Dan Kushnir
Venue:  (2021)
Keywords: reinforcement learning
exploration
curiosity
episodic memory

Abstract: Procedurally-generated sparse reward environments pose significant challenges for many RL algorithms. The recently proposed impact-driven exploration method (RIDE) by Raileanu & Rocktäschel (2020), which rewards actions that lead to large changes (measured by $\ell_2$-distance) in the observation embedding, achieves state-of-the-art performance on such procedurally-generated MiniGrid tasks. Yet, the definition of "impact" in RIDE is not conceptually clear because its learned embedding space is not inherently equipped with any similarity measure, let alone $\ell_2$-distance. We resolve this issue in RIDE via contrastive learning. That is, we train the embedding with respect to cosine similarity, where we define two observations to be similar if the agent can reach one observation from the other within a few steps, and define impact in terms of this similarity measure. Experimental results show that our method performs similarly to RIDE on the MiniGrid benchmarks while learning a conceptually clear embedding space equipped with the cosine similarity measure. Our modification of RIDE also provides a new perspective which connects RIDE and episodic curiosity (Savinov et al., 2019), a different exploration method which rewards the agent for visiting states that are unfamiliar to the agent's episodic memory. By incorporating episodic memory into our method, we outperform RIDE on the MiniGrid benchmarks.

Under review as a conference paper at ICLR 2021
IMPACT-DRIVEN EXPLORATION WITH CONTRASTIVE UNSUPERVISED REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
procedurally-generate sparse reward environments pose significant challenges for many RL algorithms. The recently proposed impact-driven exploration method (RIDE) by Raileanu & Rocktäschel (2020), which rewards actions that lead to large changes (measured by fz-distance) in the observation embedding, achieves state-of-the-art performance the definition of "impact" in RIDE is not conceptually clear because its learned embedding space is not inherently equipped with any similarity measure, let alone fz-distance. We resolve this issue RIDE via contrastive learning. That is, we train the embedding with respect cosine similarity, where define two obser- vations to be similar if the agent can reach one observation from the other within few steps, and define impact in terms of this similarity measure. Experimental results show that our method performs similarly to RIDE on the MiniGrid bench- marks while learning a conceptually clear embedding space equipped with the cosine similarity measure. Our modification of RIDE also provides new per- spective which connects RIDE and episodic curiosity (Savinov et al., 2019), a different exploration method which rewards the agent for visiting states that are unfamiliar to the agent's s episodic memory. By incorporating episodic memory into our method, we outperform RIDE on the MiniGrid benchmarks.
such procedurally -eeeerrae MiniGrid tasks. Yet,
INTRODUCTION
Reinforcement learning (RL) algorithms aim to learn an optimal policy that maximizes expected reward from the environment. The search for better RI. algorithms is motivated by the fact that many complex real-world problems can be formulated as RL problems. Yet, environments with sparse rewards, which often occur in the real-world, pose a significant challenge for RL algorithms that rely random actions for exploration. Sparsity of the reward can make it extremely unlikely for the agent to stumble upon any positive feedback by chance. The agent may spend a long time simply exploring and not receiving a single positive reward.
To overcome this issue of exploration, several previous works have used intrinsic rewards (Schmid- huber, 1991; Oudeyer et al., 2007: 2008; Oudeyer & Kaplan, 2009). Intrinsic rewards, the name suggests, are reward signals generated by the agent which can make RL. algorithms more sample efficient by encouraging exploratory behavior that is more likely to encounter rewards. Previous works have used state novelty in the form of state visitation counts (Strehl & Littman, 2008) for tabular states, pseudo-counts for high-dimensional state spaces (Bellemare al., 2016: Ostrovski et al., 2017; Martin et al.. 2017), prediction eтror of random networks (Burda et al., 2019b), and curiosity about environment dynamics (Stadie al., 2015; Pathak et al., 2017) intrinsic rewards.
Although such advances in exploration methods have enabled RL agents to achieve high rewards in notoriously difficult sparse reward environments such as Montezuma's Revenge and Pitfall (Belle- mare et al., 2013), many existing exploration methods use the same environment for training and testing (Bellemare et al., 2016; Pathak et al., 2017; Aytar et al., 2018; Ecoffet et al., 2019). As a re- sult, agents trained in this fashion do not generalize new environments. Indeed, several recent pa- pers point out that deep RL agents overfit to the environment they were trained on (Rajeswaran et al., 2017: Zhang et al., 2018; Machado et al., 2018), leading to the creation of new benchmarks sisting of procedurally-genet -enerrtee environments (Cobbe et al., 2019; Risi & Togelius, 2020; Küttler al., 2020).

---

Under review as a conference paper at ICLR 2021
In practice, agents often have act in environments that are similar, but different from the envi- ronments they were trained on. Hence, it is crucial that the agent learns a policy that generalizes across diverse (but similar) environments. This adds another layer of difficulty, the diversity of en- vironment layout for each episode, to the already challenging sparse reward structure. To tackle this challenge head-on, Raileanu & Rocktäschel (2020) focus on exploration in procedurally-geeenane environments and propose RIDE, an intrinsic rewarding scheme based on the "impact" of a new observation. Denoting the observation embedding function by Φ, RIDE measures the impact of ob- servation of by computing I|o(o') - ((0) where o is the previous observation. Similarly, Savinov et al. (2019) propose episodic curiosity (EC), an intrinsic rewarding scheme which rewards visiting states that are dis-similar to states in the agent's episodic memory.
RIDE uses forward and inverse dynamics prediction (Pathak et al., 2017) to train the observation embedding Φ in self- manner. Hence, a question that one might ask is:
What is the ly-distance in this embedding space measuring?
We address this question by modifying the embedding training procedure, thereby changing the definition of impact. That is, we modify RIDE so that impact corresponds to an explicitly trained similarity measure the embedding space, where we define two observations to be similar if they are reachable from each other within a few steps. The original definition of "impact" in RIDE not conceptually clear because the learned embedding space not inherently equipped with a similarity measure, let alone fz-distance. It is still possible that RIDE's measure of impact based on fz-distance may implicitly correspond to some similarity measure in the embedding space, but we leave this investigation for future work.
Our main contributions are 1) proposing a conceptually clear measure of impact by training ob- servation embeddings explicitly with the cosine similarity objective instead of forward and inverse dynamics prediction. 2) providing a new perspective which connects RIDE and EC, and 3) out- performing RIDE via episodic memory extensions. We use SimCLR (Chen et al., 2020) to train the emhedding function and propose a novel intrinsic rewarding scheme, which we name RIDE- SimCLR. As in EC, the positive pairs used in the contrastive learning component of RIDE-SimCLR correspond to pairs of observations which are within k-steps of each other (referred to as "k-step reachability" in their work).
Following the experimental setup of Raileanu & Rocktäschel (2020), we use MiniGrid (Chevalier- Boisvert et al., 2018) to evaluate our method as it provides a simple, diverse suite of tasks that allows us to focus on the issue of exploration instead of other issues such visual perception. We focus on the comparison of our approach RIDE since Raileanu & Rocktäschel (2020) report that RIDE achieves the best performance on all their MiniGrid benchmarks among other exploration methods such as intrinsic curiosity (ICM) by Pathak et al. (2017) and random network distillation (RND) by Burda et al. (2019b). We note that MiniGrid provides sufficiently challenging suite of tasks for RL agents despite its apparent simplicity. as ICM and RND fail to learn any effective policies for some tasks due to the difficulty posed by procedurally erreeuurlly-ggenerrteed environments. Our experimental results show that RIDE-SimCL.R performs similarly to RIDE on these benchmarks with the added benefit of having conceptually clear similarity measure for the embedding space.
Our qualitative analysis shows interesting differences hetween RIDE and RIDE-SimCLR. For in- stance, RIDE highly rewards interactions with controllable objects such as opening a door, which is not the case in RIDE-SimCLR. We also observe that our episodic memory extension improves the quantitative performance of both methods, which demonstrates the benefit of establishing a connec- tion between RIDE and EC. The Never Give Up (NGU) agent by Badia et al. (2020) can be seen as close relative of our memory extension of RIDE since it uses lg distance in embedding space trained with the same inverse dynamics objective to compute approximate counts of states and aggregates episodic memory to compute novelty bonus.
Our work is different from EC because we do not explicitly sample negative pairs for training the observation embedding network, and we use cosine similarity, instead of a separately trained neural network, to output similarity scores for pairs of observations. We note that Campero et al. (2020) report hate-offthe oftte-of-theann results on even more challenging MiniGrid tasks by training an adversarially motivated teacher network to generate intermediate goals for the agent (AMIGo)) but we do not compare against this method since their agent receives full observations of the environment. Both RIDE and RIDE-SimCLR agents only receive partial observations.
2

---

Under review as a conference paper at ICLR 2021
Ot
w-embed
4/2
at
ftwd
Φ1+1
e-embed
Φ2
Rint
finv
āg
Øt+1
Or+11
Ø-embed
(b) RIDE trains the observation embedding ø by minimizing forward and inverse dynamics predic- tion error.
(a) RIDE computes intrinsic reward by taking the &g difference the pair (dh, 0..1).
Figure I: RIDE rewards the agent for taking actions that lead to large impact, which is measured by the change in the observation embedding Øb
2 BACKGROUND
We consider the standard episodic RL setting in which an agent interacts with the environment to maximize its expected scalar reward for each episode. The interaction proceeds in discrete time steps and terminates at some fixed time T unless the goal is achieved earlier. At time step f, the agent receives observation o, and samples action from its policy (o,)) The environment then provides the agent with scalar reward Tt, the next observation 0r+1, and an end-of-episode indicator. The goal of the agent is to maximize the expected discounted reward R = Σi= r'r,, where T is the (extrinsic) reward given by the environment at time step t.
When extrinsic reward is sparse. standard RL algorithms such PPO (Schulman et al., 2017) or IMPALA (Espeholt et al., 2018) often fail to learn a good policy. To overcome this challenge, previous works have proposed augmenting the reward function by A, r + worl, where is the intrinsic reward and w; is hyperparameter which controls the relative importance of rł. The intrinsic reward rf is typically designed to be dense reward function which pushes the policy towards exploratory behavior more likely to encounter positive extrinsic rewards.
Note that intrinsic rewards can be used with any existing RL algorithms without altering their under- lying network architecture or training procedure. suffices to simply replace the extrinsic reward rz with the augmented reward f.. The observation embedding used to compute intrinsie rewards can be trained either 1) offline with respect to a uniformly random policy or 2) online with respect agent's current policy. For a fair comparison with Raileanu & Rocktäschel (2020), we use the embedding trained online for computing intrinsic rewards.
2.1 IMPACT-DRIVEN EXPLORATION (RIDE)
RIDE (Raileanu Rocktäschel, 2020) is an intrinsic reward based on the magnitude change in the observation embedding produced by the agent's action (See Figure 1). More precisely, RIDE is defined as
RIDE (.) ( -
(Nep(S(t1)
where @ is the observation embedding function and Nep is the number of times state is visited within the current episode. The purpose of this discount by (ep(s) is to prevent the agent from going back and forth a sequence of states with large &a differences.
The embedding function Φ()) used in RIDE is parametrized by neural network and trained by min- imizing forward and inverse dynamics prediction error (Pathak et al., 2017). Note that the policy network has its own observation embedding network vi, which is trained separately from 6. The embedding ò is only used to compute the intrinsic reward and never used for control, and the oppo- site holds for tb. The purpose of using forward and inverse dynamics models to train the embedding to store information that is useful for predicting the agent's 's action or effects actions have on the environment. This leads learning foccsed embedding space.
3

---

Under review as a conference paper at ICLR 2021
RIDE builds upon the intrinsic curiosity module (ICM) by Pathak al. (2017), which uses the for- ward dynamics prediction error as intrinsic reward. The novelty in RIDE is using the & distance between two different observations visualizations of RIDE by Raileanu & Rocktäschel (2020) show that RIDE assigns higher intrinsic rewards for actions that qualitatively "change the dynamics", such opening doors or interact- ing with objects in MiniGrid. However, RIDE introduces conceptual difficulties as the embedding space not explicitly trained with any similarity measure. That is, the forward and inverse dynam- ies objective does not explicitly pull together or push apart embeddings of different observations.
measure of qualitative change between the states. Indeed,
In ICM, the f2 distance between ((0++1) and the prediction Φ(0++1) has a clear interpretation as the forward prediction "error", since the forward dynamics objective explicitly minimizes this & distance. RIDE, on the other hand, uses the & distance between different observations or and 0++1 as the intrinsic reward. Yet, the forward and inverse dynamics prediction objective does not specify which pairs of observation embeddings (0,,0)) should be pulled closer together and which should be pushed apart (in & distance).
This does not mean that RIDE fails to capture qualitative changes in the dynamics. In fact, our vi- sualizations (Figure 7) corroborate the findings of Raileanu & Rocktäschel (2020) by demonstrating that RIDE assigns higher rewards for actions such opening doors. However, it is difficult to pre- cisely define what having "different dynamics" means, let alone giving a quantitative definition of it. Moreover, the question of why the &2 distance larger for pairs of observations corresponding to such actions is not well-understood, and thus, requires further investigation. Without understanding why, we cannot guarantee that RIDE will always assign higher rewards actions we perceive as "significantly changing the dynamics".
Ot
e--emeed
at
Statoo reschable B-steps
Statis
M.
e-ebedd
+ for PerE.
Statis
Rint
0++1
Ø-embed
Ø1+1
(a) EC computes intrinsic reward by aggregating individual novelty scores the episodic memory Mc-
(b) A state is k-step reachable (green) if it can be reached within & steps from states the current episodic memory (bluc).
Figure 2: EC rewards the agent for taking actions that lead observations different from ones stored in the current episodic memory.
2.2 EPISODIC CURIOSITY (EC)
EC (Savinov et al., 2019) is an intrinsic reward based on how "far" the next observation is from observations in the agent's current episodic memory (See Figure 2). More precisely, EC is defined as
REC = ri a.) C(M,, o(o+1)))
where 3 E RR is a scalar hyperparameter, C' is a learned comparator network, and M, is the agent's episodic memory at time t.
Intuitively, the comparator network C' measures how "far" a given observation 0++1 is from the agent's 's current episodic memory M,. The episodic memory M, {Ø(or) )est is the set of obser- vation embeddings previously encountered in the current episode." The comparator network C pre- dicts whether and of are reachable from each other within A steps, i.c., ğ C(o(o), Φ(o')) E [0. 1] where the true label y = 1 if the agent can reach o' from o in l steps and y otherwise. The reachability threshold k is hyperparameter. The parameters of C' and ø are trained to minimize
'The definition of episndic memory used in Savinov al. (2019) is more general. For memory efficiency, they add ((0++1) episadic memory only if it is sufficiently different from the observations already in M,.
4

---

Under review as a conference paper at ICLR 2021
CrossEntropy y). The data used in the contrastive training is generated by taking sequence of the agent's observations o1...., ON, and labeling pair (0,, 02) positive if | j S and negative if j j 2 yk. Here, y is additional hyperparameter needed create gap between positive and negative examples.
With slight abuse of notation, write
CMM, ((o++1)) A(ci,...., E [o,
where C C()(o,), (()++))) and A is an aggregation function. The aggregation function pools the individual reachability scores & and outputs an aggregate reachability score for 0++1 and Mr. One simple example of the aggregation function is A max.
OUR METHOD
We propose a modification of RIDE to remedy the lack of a conceptually clear similarity measure in the embedding space. Instead of forward and inverse dynamics prediction, we use SimCLR (Chen et al., 2020) to train the observation embeddings with respect to a cosine similarity objective. This has the benefit of equipping the embedding space with a natural similarity measure. We present this direct modification of RIDE in Section 3.1. Our modification opens up perspective through which we can view RIDE as special case EC. From this viewpoint, it natural to extend RIDE with episodic memory. We present this extension in Section 3.2. A summary of all the methods considered this work can be found in Table 1.
3.1 RIDE WITH CONTRASTIVE UNSUPERVISED REPRESENTATIONS
We propose RIDE-SimCLR, which modifies RIDE by replacing the embedding training involving forward and inverse dynamics models with SimCLR. Denote by ø the embedding network learned SimCLR. We define LIDE-SimCLR as
RSinCLR ri(se, a.) 1 - cos((((o)) ø(o
[0. 1]
(4)
//Nop (st1)))))
SimCLR trains representations by maximizing agreement between different perturbations of the same sample (re- ferred to as the "anchor" sample). Given sequence of consecutive agent observations 01, ON, we perturb each state in two different ways by taking random number of steps sampled from {L....,k }} into the future and past (See Figure 3). This gives 2N samples of...., 22N among which there are N positive pairs. Note that reach- ability is dependent like EC, however, SimCLR does not explicitly sample negative pairs. Instead, the 2(N 1) remaining sam- ples are treated as negative examples. The loss function, referred as NT-Xent (normalized temperature-scul cross entropy) in Chen et al. (2020), for a positive pair (ol, of)) is defined as
Contrastive loss
((-)
6()
the current behavior policy Un-
0:
C' +- [-
+- [4]
Figure 3: Two time steps t and t' are independently sampled from the set {O....,k), and the input observation is perturbed temporally to generate two correlated views. The embedding net- work Φ(-) is trained to maximize agree- ment.
sxp(cos(d(o)).dd)))//
{(i.j) = - -log Em exp(cos(@(o)) ((om))))
where T is the hyperparameter temperature, The total contrastive loss is
L
((2m 1, 2m) + E(2m, 2m 1).
2N
We minimize L with respect to the parameters of & online. Because @ explicitly trained to max- imize the cosine similarity between positive pairs, cosine similarity becomes natural similarity
5

---

Under review as a conference paper at ICLR 2021
measure for the learned embedding space. We set the temperature scale 0.1, the reachability
parameter & = 2, and batch size N = L/(4) where the unroll length of the policy gradient algorithm used to train the policy
3.2 EPISODIC MEMORY EXTENSIONS
RIDE-SimCLR can be seen as special case of EC by setting C(oi, 0j) (1-cco((((0))) ((o;)))/2.
0}} This perspective allows us consider variations of RIDE and MIDE-SSmCL.R that use different aggrega- tion methods on the episodic memory. Hence, we propose RIDE-MinAgg and EC-SimCLR which aggregate values from the episodic memory with
Acc....., C4) ca/ /Nep (8+())+1) in Eq. (3). Note that N (8 = { E [t]
min instead the discount by visitation count. RIDE-MinAge is RIDE with A(c1..... C.) min C, and & trained with forward and inverse dynamics prediction, and EC-SimCIR is RIDE-SimCI.R with A(c Q)) = min 4:
Method
trained via
Aggregator c()
RIDE RIDE-MinAge RIDE-SimCLR EC-SimCLR
Forward/inverse dynamies
ca/ (Np((1111) minge [t]
Forward//nverse dynamies
SimCLR
C1/ a//NNp (S441) min C3
SimCLR
Table 1: Summary of intrinsic rewarding schemes considered this work.
4 EXPERIMENTS
We evaluate our methods on procedurally-generaaed environments from MiniGrid. We compare our methods against RIDE, which achieves previous state-of-the-art performance on the MiniGrid benchmarks. We report the median and max/min of the average episode return across 3 random seeds. The average episode return computed as rolling mean of the previous 100 episodes. We also evaluate RIDE and RIDE-SinCLR the first level of Mario (Kauten, 2018) using only intrinsic rewards.
4.1 ENVIRONMENT
In all MiniGrid environments, the dynamics is determin- istic and agent observation is partial. The agent's view (the highlighted part in Figure 4) limited to 7 x 7 square cen- tered at the current location, and the agent cannot see through walls closed doors. There are seven actions the agent can choose from: turn left or right, move forward, pick up drop an object, toggle, and done.
We evaluate our methods on the following 5 benchmarks. Key- CorridorS3R3, KeyCorridorSRR3, KeyCorridor5R33, Mul- tiRoomN7S8, MultiRoomN12S10 and MultRoomN7SS4 NoisyTV. The NoisyTV variant implements a "noisy TV" in the MultiRoom environment with ball that changes color whenever the special "switch channel" action executed by the agent. Noisy TV's are known to cause problems for curiosity-based exploration methods by turning the agent into "couch potato" (Burda et al., 2019a). The purpose of this task is to demonstrate that our methods do not suffer from this issue. Further details on the MiniGrid tasks can be found in Appendix B.
Figure 4: MultiRoomN12S10. The agent (red) has to reach the goal (green) by passing through multi- ple rooms.
Heee, we are considering slightly more general formulation of EC where the comparator network C' "ahsorhs" the constant hyperparameter 8 and the negative sign in Eq. (2). In this case, the value output by C' is proportional dis-similarity rather than similarity.

---

Under review as a conference paper at ICLR 2021
KayCoricerS383
10-
KayCorridoS4AR3
KeyCorridaSSB3
RBC
07
00
0.0
3.5
0.0
Naroer
16
167
No Hudee Frammes
207
Muthiloorm8758
MuHiBoomm122500
0.
0.0
0.5
04-
20
Figure 5: Performance of exploration methods on diverse MiniGrid tasks. Note that EC-SimCLRR, the episodic memory extension of RIDE-SimCLR, performs the best on all tasks.
4.2 EVALUATION
We evaluate the following four methods: RIDE, LIDE-SimCLR, RIDE-MinAgg, and S.mCR. We torchbeast (Küttler et al., 2019), an open-source implementation of IMPALA (Espeholt et al., 201 8), as the base RI. algorithm common to all the methods we evaluate and RMSPrrp (Tieleman & Hinton, 2012) the optimizer. All methods chitecture. Their only difference is the intrinsic reward. More details about our network architecture and hyperparameter settings can be found in Appendix A and C.
evaluate the policy and value network
5 RESULTS
5.1 QUANTITATIVE RESULTS
Figure 5 shows our results on the MiniGrid benchmarks. HIDE-SimCLR performs similarly to RIDE on all the tasks except MultiRoomN12SI6 in which it requires more frames to learn the op- timal policy. We note that other exploration methods such as ICM and RND fail to learn any useful policies on this task (Raileanu& & Rocktäschel, 2020). Hence, RIDE-SimCLL retains the quantitative performance of RIDE while having conceptually clear measure for "impact"". Moreover, RIDE- MinAgg and EC-SimCLR are more sample-efficient than RIDE more challenging tasks such as MultiRoomN7S8 and MultioommN12S 10. In fact, EC-SimCLR is the only method that learns a good policy for KeyCoriidoSS433 and KeyCorridorSRR3. Note, however, that only EC- SimCLR on KeyCorridorS3R3 since all other methods have failed in the easier KeyCorridor4R33 environment and Campero al. (2020) report that RIDE fails learn any useful policy Key- CorridorS4R3 and KeyCorridorSRR3. This demonstrates the benefit of establishing connection between RIDE and EC. Viewing RIDE through the lens of episodic memory leads to variants of the intrinsic reward that are more sample-efficien.
5.2 LEARNED EMBEDDING SPACE
We compare the observation embeddings learned using RIDE-SimCLR and RIDE. Denote by d(or,o;) the dis-similarity measure used in each method. corresponds to the E distance between Φ(o;) and o00;) in RIDE, and to the cosine dis-similarity 1 cos(é(o:), o(o;)) in RIDE-SimCLR. We analyze how predictive d(o,, oy) is for the temporal distance between these two observations. To this end, generate balanced labelled dataset consisting of close observation pairs (S k steps) and far pairs (> nk steps) by running the trained RIDE agent policy for 100 episodes. The parameters set to A = 2 and 7 = 5. More details on the dataset can be found in Appendix D.
7

---

Under review as a conference paper at ICLR 2021
KeyCorridorS3R3
MultiRoomN75S
1.0
1.0
0.6
0.6
8
0.4
0.4
0.2
0.2
RIDE
RIDE-SimCLR
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
fpr
fpr
Figure 6: The ROC curves show that the cosine similarity in RIDE-SimCLR is predictive of temporal distance between states, expected. Unsurprisingly, the lg distance in RIDE is not.
Figure shows the ROC curve of LIDE-SimCLR and RIDE. From the figure, we can see that the cosine similarity measure in SimCLR is predictive of the temporal distance between observa- tions. On the other hand, the le distance between representations from RIDE is not aligned with the temporal distance. This is not surprising since the embedding space in RIDE-SimCLA is explicitly trained to respect temporal distance between observations and RIDE is not. However, this shows that the structure of learned embeddings in RIDE-SimCLR is conceptually clear: temporally close observation are closer in cosine distance. Furthermore, it demonstrates that the intrinsic rewarding schemes of RIDE and LIDE-SimCLR, despite apparent similarity, are qualitatively different.
Open Door
Turn Left / Right
Move Forward
Model
Mean
Std
Mean
Std
Mean
Std
RIDE RIDE-SimCLR
0.0163
0.0025
0.0064
0.0039
0.0047
0.0022
0.0721
0.0107
0.0883
0.0158
0.0545
0.0174
Table 2: Mean intrinsic reward per action over 100 episodes of MultiRoomN758.
5.3 INTRINSIC REWARD VISUALIZATION
To understand the qualitative difference between LIDE-SimCLR and RIDE, we analyze which actions are encouraged by each method. We take soveduurally-generatee MultiRoomN7S8 environments and roll-out the learned policies on a fixed environment. Table 2 shows the average intrinsic reward received by each action and Figure 7 shows a heatmap of how the actions were rewarded in the agents' trajectories. We also plot how the average intrinsic rewards change during training Figure 10 of Appendix we provide heatmap that shows how actions were rewarded after RIDE was trained on KeyCorri- dorS4R3 for 30M frames. Note that RIDE, failed learn good policy for this hard environment. Hence, Figure 11 visualizes failure mode of RIDE's embedding training.
and RIDE agents trained
Figure 11 of Appendix H,
The results Table 2 and Figure demonstrate qualitative differences between RIDE-SimCLR and RIDE. As observed by Raileanu & Rocktäschel (2020), RIDE gives higher intrinsic reward for interaction with objects such as opening doors, whereas RIDE-SimCIR gives higher rewards for turning left or right. An intuitive explanation for RIDE is that actions such "open door" significantly change the dynamics, which in turn leads to a substantial change in RIDE's action- focused embedding. On the other hand, RIDE-SimCLR rewards agents for moving away from where has been, so actions that move the agent into a new room (which substantially changes the ego-centric partial view of the agent), given higher rewards. Further investigations into why RIDE and RIDE-SimCLR assign high rewards to these actions an interesting direction left for future work.
8

---

Under review as a conference paper at ICLR 2021
0020
0120
0012
0012
0012
0.000
00000
0000
0.20
0.2
0.0
0.16
-018
0.52
0.08
-003
-008
0.00
-000
000
Figure 7: Intrinsic reward heatmaps for opening doors (green), moving forward (blue), or turning left/right (red) on random instance of MultiRoomN7S8. The top row corresponds to RIDE and the bottom corresponds to RIDE-SimCLR. is the agent starting position, G is the goal position and D are doors.
5.4 EXPLORATION WITH NO EXTRINSIC REWARD
We analyze the exploration behavior of the We train a RIDE-SimCLR agent on procedurally-generated MultiRoom 50M frames with only intrinsic reward the signal. The agent is allowed to take 200 steps in every episode. We observe that even without any extrinsic reward, the RIDE-SimCIL agent learns a policy which explores all the rooms in the map. similar the RIDE agent. Other agents trained with intrinsic rewards such as Count, ICM and RND are known to fall short of reaching the final room within the given amount of steps (Raileanu Rocktäschel, 2020). The state visitation heatmap on random instance of MultRoommN10S6 can be found in Appendix E.
agent in the absence of extrinsic reward.
environments for
Moreover, we compare SiDDCR to RIDE on the first level of Mario without extrinsic reward to determine its relative performance on environments different from MiniGrid. The results can be found in Appendix Our results show that RIDE-SimCLR matches the performance of RIDE on Mario. Note, however, that this singleton environment is not very challenging since vanilla IMPALA able to learn similarly good policies without any intrinsic rewards, although IMPALA does extrinsic rewards (Raileanu & Rocktäschel, 2020).
CONCLUSION AND FUTURE WORK
We identify a conceptual issue in RIDE and remedy it by learning an observation embedding space naturally equipped with the cosine similarity measure. By training embeddings with SimCLR, tain the strong performance of RIDE on seeedurally-ggeeerae conceptually clear similarity measure for the embedding space. Moreover, make connection between RIDE and EC. As result, we outperform both RIDE and RIDE-SimCLR by changing the episodic memory aggregation function, which demonstrates the benefit of this novel perspective.
MiniGrid benchmarks while getting
Despite the apparent similarity between RIDE and RIDE-SimCLR, our analysis shows that these methods are qualitatively different. The lg distance in RIDE, perhaps unsurprisingly, is not predic- tive of temporal distance between observations, unlike the cosine similarity in RIDE-SimCLR. In possi-
addition, actions that are encouraged by each intrinsic rewarding scheme different. ble that & distance in the embedding space learned with forward and inverse dynamics prediction corresponds to some notion similarity. An interesting future work would be to theoretically and empirically investigate what the & distance captures in this embedding space. For instance, what does a large l distance in this space correspond to? Other interesting directions include making intrinsic reward computation from episodic memory time and space efficient using techniques such as k-nearest neighbors, as was done Badia et al. (2020), and proposing variants of EC by training embedding networks with objective that implicitly trains known similarity measure.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930-2941, 2018.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski. Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. In International Confer- ence on Learning Representations, 2020. URL Sye57x8tt
net htw.:e/wwee//ooummaaam
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 279, 2013.
Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019a. URL https: / /ppeneeveew. net/ forum? rdw
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random net- work distillation. In 7th International Conference on Learming Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019b. URL / http:////pppeeeeeee.om net miretfformmii H11JJnR5Ym.
Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006.12111 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint wxXiv:2002.05709 2020.
Maxime Chevalier-Boisee Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https: : / //gggttttt gighub..u.............. com/ / max www.sieebbb//ccmm htapb//gmmmmmiiigggg 2018.
Djork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In 4th International Conference Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http: http:///axvvvv..eeee.e /lbpg//b//////aaee// als /111111111111 07289.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural genera- tion to benchmark reinforcement learning. arXiv preprint arXiv: 19/2.0188/ 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv: 10995, 2019.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley lain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: scalable distributed deep-rl with importance weighted actor-leamer architectures. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 1406-1415, 2018. URL http: html.
/ //proceedings m.r. www.r.ppess/ee
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Christian Kauten. Super Mario Bros for OpenAl Gym. GitHub, 2018. URL https::t///// https://gáthub. com/Kautenjer gym- super- rio-bro..
10

---

Under review as a conference paper at ICLR 2021
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: A PyTorch Platform for Dis- tributed RL. arXiv preprint arXiv:1910.03552, 2019. URL htttps://githi com/
rccebookresearchttooccheee
Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici, Ed- ward Grefenstette, and Tim Rocktäschel. The nethack learning environment. arXiv preprini
arXiv:2006.13760, 2020.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based explo- ration in feature space for reinforcement learning. Proceedings of the Twenty-Sixth Inter- national Joint Conference Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 2471-2478, 2017. doi: 10.24963/jcai.011777 10. 10.24963 jjjcijjaaii//// 2017/344.
URL /tLhttps: ptttpp::/doi.orgg.
Georg Ostrovski, Mare G Bellemare, Aäron den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume Volume 70, pp. 2721-2730 JMLR. org, 2017.
Pierre-Y Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Pierre Yves Oudeyer, Frederic Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286, 2007.
Pierre-Yvel Oudeyer, Frederic Kaplan, et al. How can we define intrinsic motivation. In Proc. of the 8th Conf. on Epigenetic Robotics, volume 5, pp. 29-31, 2008
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- environments. In International Conference on Learning Representationss 2020. URL https: /tRLtttt////peeeeeeiii //tp///ppp/.....ee////e wttp://eeef.oooooeeeeeeee.mr (ttee//Drummm.-ccg--TEPP
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards gen- eralization and simplicity in continuous control. In Advances in Neural Information Processing Systems, pp. 6550-6561, 2017.
Sebastian Risi and Julian Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, pp. 1-9, 2020.
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In International Conference on Learning Representations (ICI.), 2019.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neu- ral controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.00 1707.06347, 2017.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 1r7.v0 2015.
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 13(8):1309-133) 2008.

---

Under review as a conference paper at ICLR 2021
Tijmen Tieleman and Geoffrey Hinton. Rmsprop: Divide the gradient by running average of its recent magnitude. coursera: Neural networks for machine learning. Tech. Rep., Technical report, pp. 31, 2012.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.079331 2018.
A NETWORK ARCHITECTURE
models have same architecture for the policy and value network. The policy and value network share the same observation embedding network t. The embedding network consists of 3 convolutional layers with 3x3 kernels, stride 2, padding of 1, 32 channels, and uses ELU non-linearity (Clevert et al., 2016). The output of & is then fed into LSTM cell (Hochreiter & Schmidhuber, 1997) with 256 units. Two separate fully-connected layers are used on top of the LSTM output to predict the value and action, respectively.
The other embedding network uses the same architecture as eb. Note that ø is only used to compute the intrinsic reward and its output is never given an input to the policy network. The opposite is true for to.
B MINIGRID
MiniGrid is simple and lightweight gridworld gym enviromment, For all the tasks considered in this work, the dynamics is deterministic and agent observation is partial. The agent's 's view is limited to 7x7 square centered at its current location, and the agent cannot see through walls closed doors. An observation is given as a 7x7x3 tensor. Note that these values are not pixels. It is a partially observable view of the environment using compact encoding, with input values per visible grid cell. There are seven actions the agent can choose from: turn left or right, move forward, pick up or drop an object, toggle, and done. The agent can pick up and carry exactly one object (e.g., ball or key). To open a locked door, the agent has to be carrying key matching the door's 's color. The extrinsic reward given by the environment when goal reached r2 = 0.9 (1/lmax), where the maximum episode length mmax depends on the task.
The following are descriptions of MiniGrid tasks we use for experiments.
KeyCorridorSXRY: the agent has to pick up an object behind a locked door. The key is hidden in another room and the agent has to explore the map to find it. The maximum episode length is tmax = 30 x
MulrRoomNXXSY: the agent must navigate through ing of X many rooms, each of size at most Y, reach the goal. The maximum episode length is max = 20 X.
pereddurallyggnneraae map consist-
the agent must navigate through procedurally--eeeraaee map consisting X many rooms, each size at most Y, and "noisy TV" to reach the goal. The "noisy TV" is implemented with ball that changes color whenever the special "switch channel" action is exccuted by the agent. The maximum episode length is tmax = 20 X.
HultiRoomMNSS-Noiiyye
C HYPERPARAMETERS
The hyperparameters in Table 3 are common all the experiments with one exception. For RIDE- SimCLR on MultiRoomN we used learning rate 0.00005 because the network did not learn any useful policies with learning rate 0.0001. We decay the learning rate linearly that it becomes 0 after the final epoch.
SIDE/RIDE-MinAAEE We used the same set of hyperparameters. for RIDE and RIDE-MinAgg. As was done in Raileanu Rocktäschel (2020), we used an intrinsic reward coefficient 0.1 and en- tropy coefficient 0.0005 for MultiRoomN7S4-Noiyy No and KeyConiidoS3RR3, We used an intrinsic reward coefficient 0.5 and entropy coefficient 0.001 for MultiRoomN7S8 and MultiRoomN 12S For all tasks, used batch size of 32.
12

---

Under review as a conference paper at ICLR 2021
Parameter
Value
Learning Rate
0.0001
Unroll Length
100
Discount
0.99
RMSProp Momentum
RMSProp
0.00001
Clip Gradient Norm
40.0
Table 3: Common hyperparameters
RIDE-SiMCLRRECCSmCCCA For both methods, we ran a grid search over intrinsic reward co- efficient E {0.5, 0.1, 0.05, 0.025, 0.01} and entropy coefficient {0.001, 0.0005, 0.0001} for all tasks. We used a batch size of for both methods on all tasks.
For RIDE-SimCLR, MultiRooMN7S4NoiiyTT and KcyCoriddoS3RR3, intrinsic reward coefficient 0.05 and entropy co- efficient 0.0005 for MultiRoomN7S8, and intrinsic reward coefficient 0.01 and entropy coefficient 0.0001 for MultioommN 11soom122100 10. For KeyCorridoS4R33, we used intrinsic reward coefficient 0.025 and entropy coefficient 0.0005.
used intrinsic reward coefficient 0.01 and entropy coefficient 0.0005 for
For EC-SimCLR, we used intrinsic reward coefficient 0.01 for MultiRoomNN54-NoisyyT and Key- CorridorS3R3, 0.05 for MultiooomN7S8, 0.025 for M012000 KeyCorridor4R33, and KeyComidorS5R3. We used entropy coefficient 0.0005 for all tasks.
D ROC CURVE DATA GENERATION
first construct balanced dataset of size 2000 by sampling pairs per policy roll-out for a total of 100 roll-outs. where each roll-out is performed set hyperparameters & = 2 and ? = 5, and use policy trained with RIDE on MultiRoomN7S8. We repeat the following procedure 10 times for each roll-out.
random instance of MultiRoomN7S8 We
1. Given roll-out r (o1,...,00 or), randomly sample an anchor observation Ot-
2. Generate a positive pair (ot, (p) by randomly sampling Op where p E [t - k. + k].
3. Generate a negative pair (0r, o..) by randomly sampling on where n & [t 7k, t + 7ks
4. Compute data points Tp = d( 0p) and d(o;, o,). Assign label 0 to T, and 1 to Zn.
We that this positive/negative pair generation procedure was used by Savinov al. (2019) to train their reachability network ECC The parameter y in Step 3 can be thought of a gap which separates the positive samples and negative samples.
E No EXTRINSIC REWARD HEATMAP
RIDE-SimCLR is able to efficiently explore the state space without any extrinsic reward. Note the stark contrast with purely random exploration, which fails to even leave the first room within the given amount of time steps.
F No EXTRINSIC REWARD ON MARIO
We also compare RIDE-SimCLR to RIDE on the first level Mario without extrinsic reward to see if RIDE-SimCLR can match RIDE's performance environments different from MiniGrid. As we can observe from Figure 9, RIDE-SI LIDE-SimCLR matches the performance of RIDE on Mario.
Note, however, that this singleton environment is not very challenging since even vanilla IMPALA able to learn similarly good policies without any intrinsic rewards, although does use extrinsic rewards (Raileanu & Rocktäschel, 2020).
13

---

Under review as a conference paper at ICLR 2021
Random
RIDE-SimCLR
-8
88
-6
-6
-4
-4
-2
Figure 8: Heatmap of state visitation count on the MultiRoomN task with a random policy and policy trained with RIDE-SimCLR in the absence of any extrinsic reward.
SuperMarioBros Intrinsic
1000
800
600
400
200
RIDE
RIDE-SimCLR
0.0
0.5
1.0
1.5
2.0
Number of Frames
1e6
Figure 9: Performance on level of Mario with intrinsic reward only.
The common hyperparameters we used for Mario are the same ones reported in Table 3, except for the unroll length which we set 20. For both RIDE and RIDE-SimCLRR we used intrinsic reward coefficient of 0.05 and entropy coefficient of 0.0001.
G MEAN INTRINSIC REWARD DURING TRAINING
Figure 10 shows the changes in mean intrinsic rewards during training for MultiRoomN7S8, Multi- RoomN12S10, KeyCoriddrS3R3, KeyCorridorS4R3.
H VISUALIZATION OF RIDE ON KEYCORRIDORSAR3
We visualize the intrinsic reward per action of RIDE for KeyCoriidoS4RR3 in Figure 11. Note that RIDE failed to learn useful policy for this environment (See Figure 5). The purpose of Figure 11 to visualize the embeddings learned via forward and inverse dynamics when the policy far from optimal. We can that the "open door" actions rewarded less compared to Figure (when RIDE learned good policy for the environment), and unnecessary turning actions are highly rewarded.
14

---

Under review as a conference paper at ICLR 2021
MultiRoom Intrinsic
MultiRoomNN2210 Intrinsic
0.014
RIDE
0.014
RIDE MinAgg
0.012
RIDE-SimCLR
0.012
EC-SimCLR
0.008
0.008
0.006
0.006
0.004
0.004
0.002
0.000
0.0
0.5
1.0
2.0
2.5
3.0
0.000
6
Number of Frames
le7
Number of Frames
1e7
KeyCorridorS3R3 Intrinsic
KeyCorridorS4R3 Intrinsic
0.0025
0.0025
0.0020
0.0020
0.0015
0.0015
0.0010
0.0010
0.0005
0.0005
0.0000
0.0
0.5
1.
1.5
2.0
0.0000
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Number of Frames
107
Number of Frames
167
Figure 10: Intrinsic reward during training on diverse MiniGrid tasks.
0.0010
0.0010
0.0010
0.0008
0.0008
0.0008
0.0004
0.0004
0.0004
0.0000
0.0000
0.0000
Figure 11: RIDE intrinsic reward heatmaps for opening doors (green), moving forward (blue), or turning left/right (red) on a random instance of KeyCoriidoS4RR3. A is the agent starting position, G is the goal position and D are doors.
15


Reviews:
[{"review_id": "zAe0fktJCyR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes RIDE-SimCLR, that uses self-supervised learning technique like SimCLR to learn a representation for RIDE to use. The goal is to construct a representation of a state that is sensitive to their temporal distances in the trajectory. RIDE-SimCLR shows comparable performance with the original RIDE. \n\nOne main question is why adding SimCLR training is good for RL exploration? If the trajectory takes a detour to the target location, does the temporal distance make sense? Note that such a distance obviously depends on the current policy. How does the current policy affect the training? What if we use the optimal policy to train the representation? The paper doesn’t show ablation studies for that. \n\nAccording to Eqn. 4, the intrinsic reward of RIDE-SimCLR is high, if the representations of two states are dissimilar with respect to each other. However, if the SimCLR training is successful, then we would expect \\phi(o_i) and \\phi(o_{i+1}) to be very similar (since they are 1-step away in the trajectory) and there shouldn't be any intrinsic reward? From this analysis, I feel that we might want to expect SimCLR training to be somewhat working but not fully done. But the paper doesn't really discuss these subtle cases, which is unfortunate. \n\nFurthermore, there could be situation that two nearby state might need very different representations. E.g., before and after the agent picks the key / open the door, etc. I am a bit confused why this criterion is important. Could the author explain it?\n\nI wonder whether the authors could give more visualization on the learned representation. Fig. 6 basically shows the SimCLR training works, which is not enough to show its usefulness. Is there any additional properties of the learned representation (e.g., sparsity? disentanglement?) How is it different from representation evaluated from RIDE other than Fig. 6? \n\nThe hyper-parameter \\gamma seems to be quite important for the training, since it distinguishes between close and faraway state pairs. But we don’t see ablation studies on it. When we should set large \\gamma and when to make it small? \n\nI didn’t get what Fig. 7 shows. Where is the benefit of using RIDE-SimCLR? From the figure what part of RIDE-SimCLR that does better than RIDE?\n\n========\n\nI keep the score after reading the rebuttal / author comments. \n\nI thank the authors to conduct a lot of additional experiments. Since the authors also agree that \"two nearby states can have very different representations\", it doesn't make sense to treat nearby states as positive pairs (and use SimCLR) to learn the representation from the beginning, which contradicts the purpose of the entire paper. If the authors cannot show other benefits of this representation learning (other than the performance boost on a 1-2 tasks), then the performance gain could be just due to other reasons that are not known. \n\nThe additional experiments (e.g., Appendix F) only compares Random with RIDE-SimCLR, and I wonder what's the performance of RIDE? \n\nOverall the authors need to rethink the proposed approach in order to give a consistent story of what is a good representation for RL exploration. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "OatU1tlSzJi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study exploration in procedurally-generated environments with sparse reward, proposing \na new method, termed RIDE-SimCLR, for addressing this problem. In particular, the proposed method is a direct extension of an existing method [RIDE; 1], paired with advances in contrastive unsupervised representation learning [SimCLR; 2]. Moreover, the authors form a link between RIDE-SimCLR and episodic curiosity [EC; 3] and demonstrate empirically that the proposed method is performant in procedurally-generated MiniGrid tasks.\n\nRIDE (i.e., rewarding impact-driven exploration) is an intrinsic reward signal (Eqn. (1)) associated with the transition $o_{t} \\overset{a_{t}}{\\rightarrow} o_{t+1}$ given by the ratio of L2 difference in the observation embeddings, $|\\| \\phi(o_{t}) - \\phi(o_{t+1})|\\|$,  over the (square root of the) episodic count for $o_{t+1}$. The main contribution of this paper is to reconsider what the embedding function $\\phi$ should be such that the numerator (or the distance between two observation embeddings) has an explicit conceptual meaning, i.e., represent a clear \"measure of impact. The authors borrow the idea from SimCLR [2] and use cosine similarity between successive observation embeddings as a measure of impact when the embedding function $\\phi$ is trained with contrastive learning, treating observations that $k$-steps reachable as positive pairs, i.e., forcing their embeddings to be approximately equal.\n\nMoreover, the authors the draw connections between their proposed method, RIDE-SimCLR, and EC [3]. I find this formulation interesting and useful.\n\n**Questions and Concerns**:\n1. Both RIDE and RIDE-SimCLR use episodic count. This is trivial to calculate in MiniGrid (as it's given by default) but it is not accessible in more complicated settings (e.g., what about a procedurally generated robot arm manipulation task?). Also it's unclear to me if authors use the episodic of observations or simulator states! The former is prone to fail when there are aliased observations, i.e., the same observation in different states. If authors use simulator state visitation/episodic count that means that they access privileged information and hence all the RIDE results from the original paper should be revisited since the comparison to the baselines (ICM, RND, Count) is unfair. Most notably, if in the NoisyTV task they use simulator state count then this is indifferent for RIDE and hence RIDE-SimCLR. Pseudo-counts [4] can be used instead but this comes with all the issues count-based exploration has in deep RL [4], e.g., noisy TV etc.\n2. By construction of $\\phi$ in RIDE-SimCLR, shouldn't (asymptotically at least) $1 - \\text{cos}(\\phi(o_{t}), \\phi(o_{t+1})) = 0$ for all $t$ since $\\Delta t(o_{t+1}, o_{t}) = 1 < k = 2$? That should mean that if $\\phi$ is doing what it's trained to do, then $R_{\\text{SimCLR}} = 0$ always and then there is no intrinsic reward. I am not sure if I am interpreting something incorrectly or if there is some artifact due to the denominator term (i.e., episodic count) which confounds with the empirical results we see, since I would expect the numerator in Eqn. (4) to be approximately zero.\n\n**Notes**:\n1. Make sure that the RIDE definition in Eqn. (1) makes sense, i.e., $s_{t}$ and $a_{t}$ are in the middle equality and $o_{t}$, $o_{t+1}$ in the RHS. Same applies to Eqn. (4).\n2. It would be really useful to see the methods in Section 3.2 summarised in a table with columns: (i) method; (ii) $\\phi$ trained via; (iii) aggregator $A$.\n\n**References**\n\n[1] Roberta Raileanu and Tim Rockt¨aschel. Ride: Rewarding impact-driven exploration for\nprocedurally-generated environments. In International Conference on Learning Representations,\n2020. URL https://openreview.net/forum?id=rkg-TJBFPB.\n\n[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for\ncontrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.\n\n[3] Nikolay Savinov, Anton Raichuk, Rapha¨el Marinier, Damien Vincent, Marc Pollefeys, Timothy\nLillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In International Conference\non Learning Representations (ICLR), 2019.\n\n[4] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.\nUnifying count-based exploration and intrinsic motivation. In Advances in Neural Information\nProcessing Systems, pp. 1471–1479, 2016. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "FAyNHBScQvK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work explores novel intrinsic motivations used to augment extrinsic rewards in sparse reward contexts. Their algorithmic contributions are essentially two-fold:\n-They propose RIDE-SimCLR, a modification of RIDE which replaces RIDE's intrinsic motivation (the l2-distance, between timesteps, of an embedding of observations) with a SimCLR-inspired contrastive learning dissimilarity measure. This is claimed to be a more conceptually clear/natural choice.\n-They integrate a version of episodic memory with this, proposing EC-SimCLR, with an episodic RIDE ablation RIDE-MinAgg.\nUsing procedurally-generated enviuronments from MiniGrid, they go on to show superior performance of EC-SimCLR over RIDE, RIDE-SimCLR, and RIDE-MinAgg, while they claim RIDE and RIDE-SimCLR to have comparable performance.\n\nThis paper does a clear job of motivating the steps of the innovations described above, and its experimental demonstrations show superior performance of EC-SimCLR over reasonable ablations of these innovations. Its technical background pieces are lucid, walking the reader through several important concepts.\n\nI see two key weaknesses:\n\n1) The set of environments and baselines is fairly narrow. They choose to focus on procedurally-generated MiniGrid environments, for which the RIDE paper demonstrates superiority over a number of other intrinsic motivation methods, and hence they exclude these other methods in evaluations. RIDE's evaulations on MiniGrid appears more extensive (this is only a subset of these environments, correct?), and its evaluations on other environments show less of a clear advantage over these other methods. While generally we cannot expect researchers to run their algorithms on all of our favorite environments -- that is a lot of work! -- it would be much more convincing if the authors compared their algorithm to a broader list of baselines on an environment for which RIDE did not show such favorable performance, given the similarity of their approach to that of RIDE's. I do not agree with the statement: \"We note that MiniGrid provides a sufficiently challenging suite of tasks for RL agents despite its apparent simplicity, as ICM and RND fail to learn any effective policies for\nsome tasks due to the difficulty posed by procedurally-generated environments.\" When useful baselines fail on a certain subset of tasks, it does not mean that this subset of tasks is the suite of tasks with which to judge future algorithms.\n\n2) A good deal of the technical treatment is devoted to motivating's the innovation of RIDE-SimCLR over RIDE as being more conceptually clear/natural. This argument seems opaque to me. As evidence, they show that RIDE-SimCLR's similarity metric much better predicts temporal difference in states than RIDE's l2 distance. This is, however, entirely expected given the objective of the similarity metric, and is it not case that RIDE's l2 distance might have some conceptually clear/natural interpretation?  Hence this argument of conceptual clarity seems a bit soft, despite quite a bit of the exposition devoted to it. If there were clearer utilities demonstrated, it would be more convincing, but right now I find this emphasis confusing.\n\nThis innovation does appear to have one utility: that it suggests the further improvement that involves episodic memory (though, is this the only route towards this further improvement?).\n\nI recommend rejection, though I am certainly open to changing my mind, especially if the case can be made that the benchmarks are exhaustive enough, or that it's unreasonable to expect more benchmarking within a single publication.\n\nA question for authors: unless I misunderstood, EC itself was not used as a baseline. Why is that?\n\nOne minor point: I think the definition of EC-SimCLR could be spelled out more explicitly. I think I know what it is, but it is not \"RIDE-SimCLR with A = min\" given that RIDE-SimCLR has its own specific A.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DhZDsya5yMd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe work provided a nice new method with some performance gains by combining several existing techniques. The presentation was clear and organized, with the new method getting both better performance and some improvements in interpretability. It provides a variety of visual analyses that are typical of this area of research and present the contrasts between this work and prior efforts.\n\nHowever, I was unimpressed with the characterization of the RIDE method upon which this method was based.  The paper states:\n\"The novelty in RIDE is using the `2 distance between two different observations as a\nmeasure of qualitative change between the states. However, this introduces conceptual difficulties\nas the embedding space is not explicitly trained with any similarity measure. This is in contrast to\nICM, which trains the state representation to minimize the forward prediction error. Because of this\nexplicit training objective, there is no conceptual difficulty in viewing the intrinsic reward in ICM\nas a measure of its dynamics model uncertainty.\"\n\nRIDE also trains using prediction errors based on 1) prediction the next state's embedding and 2) predicting the action taken using the two states' embeddings. This encourages the embedding space to store information which can predict actions as well as the effects actions have on the environment. While I admit there is still some uncertainty in this interpretation, the authors of this work do not comment on this explanation at all which should be critical when replacing the mechanism for training these embeddings with an alternative which contains different information. This is quite a big change in the algorithm, as the focus on the impact of actions for the embedding space appears to be the inspiration for calling RIDE \"Impact-Driven Exploration\".\n\nGiven the replacement of action-focused embeddings with distance-focused ones (according to a policy), the author's algorithm might more accurately be called \"Depth-Driven\" rather than impact-driven.\n\nBefore publication, I believe the authors should reduce the emphasis a bit on not being able to explain RIDE's l2 distance (e.g. \"This is in contrast to RIDE, in which the l`2 distance is not known to correspond to any similarity measure.\") and should recognize the explanation that RIDE gives for its embedding space more clearly so as to be able to more effectively explain the difference (advantage! in these tasks) in their own approach.\n\n## Quality\nThe paper is presented according to a high standard of quality. They document and explain their methods clearly, including hyperparameters.\n\n## Clarity\nThe paper presented its results clearly, using the standard visuals for the field and with clearly written and easily readable text. Thank you!\n\n## Originality\nThe paper provides a novel recombination of methods that others have developed in a non-trivial way with original analysis into the distinctions between these methods and the effects of their combination.\n\n## Significance\nThe new method shows some improvements over existing methods in terms of sample efficiency on a number of tasks.\nThe new method has some advantages in interpretability that were explored, though the practical value of this interpretability could be explored a bit further than just sample-efficiency. Are there any kinds of tasks your method might solve that would not be possible with the existing method? For example, you could imagine that a task with extremely long hallways would receive no intrinsic reward in RIDE but your method would have no trouble finding the motivation to keep running down the hallway. In contrast, are there any tasks RIDE would do better at than your approach, or does the \"distance\"-focused metric work in a strictly superior way? I would have been more impressed by the paper if additional tasks such as this would have been included.\n\nI think a bit less emphasis could be given to the description of RIDE vs RIDE-SimCLR's similarity to temporal distance between states. RIDE is clearly not designed for that, RIDE-SimCLR is, so while it is nice to prove that RIDE-SimCLR does have this property it could be made a tiny bit more brief.\n\nThe qualitative comparison with RIDE in terms of the heatmap especially could have been developed a bit more. The authors only say \"Why RIDE and RIDE-SimCLR assign high rewards to these actions is an interesting question left\nfor future work\". For RIDE, the reward assignment feels fairly clear: taking the \"open door\" action creates a substantial change in RIDE's action-focused embedding space, so this is well-rewarded by RIDE. It would be nice for the author's to suggest a contrast with RIDE-SimCLR: it seems to me that it naturally does not provide a lot of intrinsic reward for actions but rather through movement away from where it has been, so it is natural that there is more reward for moving into new rooms rather than specifically on opening the door. RIDE-SimCLR also seems to avoid wandering the first room, which may be because RIDE is looking for objects it could interact with but RIDE-SimCLR just wants to get away from its current position as fast as possible.\n\nFurthermore, I would have appreciated seeing a bit more analysis on running this algorithm without any external reward. This was mentioned in the appendix briefly but the current method was only compared with a random policy and not with RIDE.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and very active participation in the author response period. I want to start by stating that I rank the paper higher as is currently reflected in the average score of the reviewers. The reasons for this are that a) R2 and R3, while responding to the author's rebuttal, do not seem to have updated their score or indicated that they want to keep their initial assessment of the paper -- in particular, R2 has acknowledged that additional experiments by the authors were useful and results on KeyCorridorS4/S5R3 are nice, and b) I disagree with R2's sentiment that MiniGrid is not a suitable testbed -- it is by now an established benchmark for evaluating RL exploration and representation learning methods (see list of publications on https://github.com/maximecb/gym-minigrid). However, despite my more positive stance on the paper, I fully agree with R1 and R2 that a comparison to EC is needed in order to shed light into which factors of EC-SimCLR actually led to improvements in comparison to RIDE. I therefore recommend rejection, but I strongly encourage the authors to take the feedback from the reviewers and work on a revised submission to the next venue.
Decision: Reject