Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work explores novel intrinsic motivations used to augment extrinsic rewards in sparse reward contexts. Their algorithmic contributions are essentially two-fold:
-They propose RIDE-SimCLR, a modification of RIDE which replaces RIDE's intrinsic motivation (the l2-distance, between timesteps, of an embedding of observations) with a SimCLR-inspired contrastive learning dissimilarity measure. This is claimed to be a more conceptually clear/natural choice.
-They integrate a version of episodic memory with this, proposing EC-SimCLR, with an episodic RIDE ablation RIDE-MinAgg.
Using procedurally-generated enviuronments from MiniGrid, they go on to show superior performance of EC-SimCLR over RIDE, RIDE-SimCLR, and RIDE-MinAgg, while they claim RIDE and RIDE-SimCLR to have comparable performance.

This paper does a clear job of motivating the steps of the innovations described above, and its experimental demonstrations show superior performance of EC-SimCLR over reasonable ablations of these innovations. Its technical background pieces are lucid, walking the reader through several important concepts.

I see two key weaknesses:

1) The set of environments and baselines is fairly narrow. They choose to focus on procedurally-generated MiniGrid environments, for which the RIDE paper demonstrates superiority over a number of other intrinsic motivation methods, and hence they exclude these other methods in evaluations. RIDE's evaulations on MiniGrid appears more extensive (this is only a subset of these environments, correct?), and its evaluations on other environments show less of a clear advantage over these other methods. While generally we cannot expect researchers to run their algorithms on all of our favorite environments -- that is a lot of work! -- it would be much more convincing if the authors compared their algorithm to a broader list of baselines on an environment for which RIDE did not show such favorable performance, given the similarity of their approach to that of RIDE's. I do not agree with the statement: "We note that MiniGrid provides a sufficiently challenging suite of tasks for RL agents despite its apparent simplicity, as ICM and RND fail to learn any effective policies for
some tasks due to the difficulty posed by procedurally-generated environments." When useful baselines fail on a certain subset of tasks, it does not mean that this subset of tasks is the suite of tasks with which to judge future algorithms.

2) A good deal of the technical treatment is devoted to motivating's the innovation of RIDE-SimCLR over RIDE as being more conceptually clear/natural. This argument seems opaque to me. As evidence, they show that RIDE-SimCLR's similarity metric much better predicts temporal difference in states than RIDE's l2 distance. This is, however, entirely expected given the objective of the similarity metric, and is it not case that RIDE's l2 distance might have some conceptually clear/natural interpretation?  Hence this argument of conceptual clarity seems a bit soft, despite quite a bit of the exposition devoted to it. If there were clearer utilities demonstrated, it would be more convincing, but right now I find this emphasis confusing.

This innovation does appear to have one utility: that it suggests the further improvement that involves episodic memory (though, is this the only route towards this further improvement?).

I recommend rejection, though I am certainly open to changing my mind, especially if the case can be made that the benchmarks are exhaustive enough, or that it's unreasonable to expect more benchmarking within a single publication.

A question for authors: unless I misunderstood, EC itself was not used as a baseline. Why is that?

One minor point: I think the definition of EC-SimCLR could be spelled out more explicitly. I think I know what it is, but it is not "RIDE-SimCLR with A = min" given that RIDE-SimCLR has its own specific A.
review_title: Interesting idea and well-explained chain of logic, but could use better benchmarking and

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors study exploration in procedurally-generated environments with sparse reward, proposing 
a new method, termed RIDE-SimCLR, for addressing this problem. In particular, the proposed method is a direct extension of an existing method [RIDE; 1], paired with advances in contrastive unsupervised representation learning [SimCLR; 2]. Moreover, the authors form a link between RIDE-SimCLR and episodic curiosity [EC; 3] and demonstrate empirically that the proposed method is performant in procedurally-generated MiniGrid tasks.

RIDE (i.e., rewarding impact-driven exploration) is an intrinsic reward signal (Eqn. (1)) associated with the transition $o_{t} \overset{a_{t}}{\rightarrow} o_{t+1}$ given by the ratio of L2 difference in the observation embeddings, $|\| \phi(o_{t}) - \phi(o_{t+1})|\|$,  over the (square root of the) episodic count for $o_{t+1}$. The main contribution of this paper is to reconsider what the embedding function $\phi$ should be such that the numerator (or the distance between two observation embeddings) has an explicit conceptual meaning, i.e., represent a clear "measure of impact. The authors borrow the idea from SimCLR [2] and use cosine similarity between successive observation embeddings as a measure of impact when the embedding function $\phi$ is trained with contrastive learning, treating observations that $k$-steps reachable as positive pairs, i.e., forcing their embeddings to be approximately equal.

Moreover, the authors the draw connections between their proposed method, RIDE-SimCLR, and EC [3]. I find this formulation interesting and useful.

**Questions and Concerns**:
1. Both RIDE and RIDE-SimCLR use episodic count. This is trivial to calculate in MiniGrid (as it's given by default) but it is not accessible in more complicated settings (e.g., what about a procedurally generated robot arm manipulation task?). Also it's unclear to me if authors use the episodic of observations or simulator states! The former is prone to fail when there are aliased observations, i.e., the same observation in different states. If authors use simulator state visitation/episodic count that means that they access privileged information and hence all the RIDE results from the original paper should be revisited since the comparison to the baselines (ICM, RND, Count) is unfair. Most notably, if in the NoisyTV task they use simulator state count then this is indifferent for RIDE and hence RIDE-SimCLR. Pseudo-counts [4] can be used instead but this comes with all the issues count-based exploration has in deep RL [4], e.g., noisy TV etc.
2. By construction of $\phi$ in RIDE-SimCLR, shouldn't (asymptotically at least) $1 - \text{cos}(\phi(o_{t}), \phi(o_{t+1})) = 0$ for all $t$ since $\Delta t(o_{t+1}, o_{t}) = 1 < k = 2$? That should mean that if $\phi$ is doing what it's trained to do, then $R_{\text{SimCLR}} = 0$ always and then there is no intrinsic reward. I am not sure if I am interpreting something incorrectly or if there is some artifact due to the denominator term (i.e., episodic count) which confounds with the empirical results we see, since I would expect the numerator in Eqn. (4) to be approximately zero.

**Notes**:
1. Make sure that the RIDE definition in Eqn. (1) makes sense, i.e., $s_{t}$ and $a_{t}$ are in the middle equality and $o_{t}$, $o_{t+1}$ in the RHS. Same applies to Eqn. (4).
2. It would be really useful to see the methods in Section 3.2 summarised in a table with columns: (i) method; (ii) $\phi$ trained via; (iii) aggregator $A$.

**References**

[1] Roberta Raileanu and Tim Rockt¨aschel. Ride: Rewarding impact-driven exploration for
procedurally-generated environments. In International Conference on Learning Representations,
2020. URL https://openreview.net/forum?id=rkg-TJBFPB.

[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for
contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.

[3] Nikolay Savinov, Anton Raichuk, Rapha¨el Marinier, Damien Vincent, Marc Pollefeys, Timothy
Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In International Conference
on Learning Representations (ICLR), 2019.

[4] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.
Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information
Processing Systems, pp. 1471–1479, 2016. 
review_title: I tend to reject this paper due to (a) it relies on episodic count which does not scale to deep RL settings and (b) I expect the proposed intrinsic reward signal to be always (approximately) zero

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes RIDE-SimCLR, that uses self-supervised learning technique like SimCLR to learn a representation for RIDE to use. The goal is to construct a representation of a state that is sensitive to their temporal distances in the trajectory. RIDE-SimCLR shows comparable performance with the original RIDE. 

One main question is why adding SimCLR training is good for RL exploration? If the trajectory takes a detour to the target location, does the temporal distance make sense? Note that such a distance obviously depends on the current policy. How does the current policy affect the training? What if we use the optimal policy to train the representation? The paper doesn’t show ablation studies for that. 

According to Eqn. 4, the intrinsic reward of RIDE-SimCLR is high, if the representations of two states are dissimilar with respect to each other. However, if the SimCLR training is successful, then we would expect \phi(o_i) and \phi(o_{i+1}) to be very similar (since they are 1-step away in the trajectory) and there shouldn't be any intrinsic reward? From this analysis, I feel that we might want to expect SimCLR training to be somewhat working but not fully done. But the paper doesn't really discuss these subtle cases, which is unfortunate. 

Furthermore, there could be situation that two nearby state might need very different representations. E.g., before and after the agent picks the key / open the door, etc. I am a bit confused why this criterion is important. Could the author explain it?

I wonder whether the authors could give more visualization on the learned representation. Fig. 6 basically shows the SimCLR training works, which is not enough to show its usefulness. Is there any additional properties of the learned representation (e.g., sparsity? disentanglement?) How is it different from representation evaluated from RIDE other than Fig. 6? 

The hyper-parameter \gamma seems to be quite important for the training, since it distinguishes between close and faraway state pairs. But we don’t see ablation studies on it. When we should set large \gamma and when to make it small? 

I didn’t get what Fig. 7 shows. Where is the benefit of using RIDE-SimCLR? From the figure what part of RIDE-SimCLR that does better than RIDE?

========

I keep the score after reading the rebuttal / author comments. 

I thank the authors to conduct a lot of additional experiments. Since the authors also agree that "two nearby states can have very different representations", it doesn't make sense to treat nearby states as positive pairs (and use SimCLR) to learn the representation from the beginning, which contradicts the purpose of the entire paper. If the authors cannot show other benefits of this representation learning (other than the performance boost on a 1-2 tasks), then the performance gain could be just due to other reasons that are not known. 

The additional experiments (e.g., Appendix F) only compares Random with RIDE-SimCLR, and I wonder what's the performance of RIDE? 

Overall the authors need to rethink the proposed approach in order to give a consistent story of what is a good representation for RL exploration. 
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: ## Summary

The work provided a nice new method with some performance gains by combining several existing techniques. The presentation was clear and organized, with the new method getting both better performance and some improvements in interpretability. It provides a variety of visual analyses that are typical of this area of research and present the contrasts between this work and prior efforts.

However, I was unimpressed with the characterization of the RIDE method upon which this method was based.  The paper states:
"The novelty in RIDE is using the `2 distance between two different observations as a
measure of qualitative change between the states. However, this introduces conceptual difficulties
as the embedding space is not explicitly trained with any similarity measure. This is in contrast to
ICM, which trains the state representation to minimize the forward prediction error. Because of this
explicit training objective, there is no conceptual difficulty in viewing the intrinsic reward in ICM
as a measure of its dynamics model uncertainty."

RIDE also trains using prediction errors based on 1) prediction the next state's embedding and 2) predicting the action taken using the two states' embeddings. This encourages the embedding space to store information which can predict actions as well as the effects actions have on the environment. While I admit there is still some uncertainty in this interpretation, the authors of this work do not comment on this explanation at all which should be critical when replacing the mechanism for training these embeddings with an alternative which contains different information. This is quite a big change in the algorithm, as the focus on the impact of actions for the embedding space appears to be the inspiration for calling RIDE "Impact-Driven Exploration".

Given the replacement of action-focused embeddings with distance-focused ones (according to a policy), the author's algorithm might more accurately be called "Depth-Driven" rather than impact-driven.

Before publication, I believe the authors should reduce the emphasis a bit on not being able to explain RIDE's l2 distance (e.g. "This is in contrast to RIDE, in which the l`2 distance is not known to correspond to any similarity measure.") and should recognize the explanation that RIDE gives for its embedding space more clearly so as to be able to more effectively explain the difference (advantage! in these tasks) in their own approach.

## Quality
The paper is presented according to a high standard of quality. They document and explain their methods clearly, including hyperparameters.

## Clarity
The paper presented its results clearly, using the standard visuals for the field and with clearly written and easily readable text. Thank you!

## Originality
The paper provides a novel recombination of methods that others have developed in a non-trivial way with original analysis into the distinctions between these methods and the effects of their combination.

## Significance
The new method shows some improvements over existing methods in terms of sample efficiency on a number of tasks.
The new method has some advantages in interpretability that were explored, though the practical value of this interpretability could be explored a bit further than just sample-efficiency. Are there any kinds of tasks your method might solve that would not be possible with the existing method? For example, you could imagine that a task with extremely long hallways would receive no intrinsic reward in RIDE but your method would have no trouble finding the motivation to keep running down the hallway. In contrast, are there any tasks RIDE would do better at than your approach, or does the "distance"-focused metric work in a strictly superior way? I would have been more impressed by the paper if additional tasks such as this would have been included.

I think a bit less emphasis could be given to the description of RIDE vs RIDE-SimCLR's similarity to temporal distance between states. RIDE is clearly not designed for that, RIDE-SimCLR is, so while it is nice to prove that RIDE-SimCLR does have this property it could be made a tiny bit more brief.

The qualitative comparison with RIDE in terms of the heatmap especially could have been developed a bit more. The authors only say "Why RIDE and RIDE-SimCLR assign high rewards to these actions is an interesting question left
for future work". For RIDE, the reward assignment feels fairly clear: taking the "open door" action creates a substantial change in RIDE's action-focused embedding space, so this is well-rewarded by RIDE. It would be nice for the author's to suggest a contrast with RIDE-SimCLR: it seems to me that it naturally does not provide a lot of intrinsic reward for actions but rather through movement away from where it has been, so it is natural that there is more reward for moving into new rooms rather than specifically on opening the door. RIDE-SimCLR also seems to avoid wandering the first room, which may be because RIDE is looking for objects it could interact with but RIDE-SimCLR just wants to get away from its current position as fast as possible.

Furthermore, I would have appreciated seeing a bit more analysis on running this algorithm without any external reward. This was mentioned in the appendix briefly but the current method was only compared with a random policy and not with RIDE.
review_title: Nice method, with improved exposition after rebuttal

Meta-review: I thank the authors for their submission and very active participation in the author response period. I want to start by stating that I rank the paper higher as is currently reflected in the average score of the reviewers. The reasons for this are that a) R2 and R3, while responding to the author's rebuttal, do not seem to have updated their score or indicated that they want to keep their initial assessment of the paper -- in particular, R2 has acknowledged that additional experiments by the authors were useful and results on KeyCorridorS4/S5R3 are nice, and b) I disagree with R2's sentiment that MiniGrid is not a suitable testbed -- it is by now an established benchmark for evaluating RL exploration and representation learning methods (see list of publications on https://github.com/maximecb/gym-minigrid). However, despite my more positive stance on the paper, I fully agree with R1 and R2 that a comparison to EC is needed in order to shed light into which factors of EC-SimCLR actually led to improvements in comparison to RIDE. I therefore recommend rejection, but I strongly encourage the authors to take the feedback from the reviewers and work on a revised submission to the next venue.
Decision: Reject