Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces BYOL-Explore, a novel exploration method for RL. The idea of BYOL-Explore is to train an encoder of observations based on BYOL and further utilize training loss as the intrinsic reward for exploration.

The architecture of BYOL-Explore consists of an observation encoder, a closed-loop RNN that embeds the history of interaction, and an open-loop RNN for future prediction. In training the observation embeddings, the authors adopt the learning scheme of BYOL and construct a targe encoder by a moving average of the trained encoders. The training of the encoder thus minimizes the distance between the RNN prediction and the target encoder of observations. 

Experiments suggest that BYOL-Explore outperforms the SOTA baselines RND and ICM. 
strengths_and_weaknesses: $\{\bf{\rm Strength.}\}$

-The proposed BYOL-Explore calculates the intrinsic reward and thus is adaptable to existing RL algorithms.

-The proposed curiosity is built on future multi-step predictions, thus utilizing the temporal information contained in observations. In addition, the intrinsic reward is defined by the training loss on latent representation. If the latent well captures the temporal information of observations, the proposed intrinsic reward can filter out the reconstruction loss of irrelevant observations (to the transition dynamics), which is desirable.

$\{\bf{\rm Weakness.}\}$

-The learning paradigm of BYOL-Explore falls in the common practice of fitting (coarse) transition models and then utilizing its training loss (or the disagreement among the ensemble of models) as the intrinsic reward/penalty for exploration/conservative RL. In that regard, the contribution of this paper is relatively marginal. 


questions: $\{\bf{\rm Question 1.}\}$ The proposed RNN structure seems relatively large-scaled compared with most existing exploration strategies. How does BYOL-Explore compare with ICM and RND regarding the computation budget and training efficiency?

$\{\bf{\rm Question 2.}\}$ Does the proposed intrinsic reward of BYOL-Explore capture the epistemic uncertainty of observations adequately? For instance, if a specific observation-action pair $(o, a)$ frequently occurs among the trajectories, we would hope that the intrinsic reward of visiting $(o, a)$ is small as it is of less novelty. Nevertheless, the proposed intrinsic reward sums up all the prediction errors involving the pair $(o, a)$ across the trajectories, which might be even larger than that of scarcely explored observation-action pairs. 

$\{\bf{\rm Question 3.}\}$ In addition, the proposed intrinsic reward seems to have a preference over actions that are distant from the initial observations. For instance, for a given trajectory ${x_1 = (o_1, a_1), x_2 = (o_2, a_2), \ldots, x_n = (o_n, a_n)}$, the intrinsic reward of $x_1$ consists only the loss associated with the history representation $b_{1, 1}$. Nevertheless, since each closed-loop RNN is followed by the open-loop RNN cells that reconstruct the entire trajectory, the intrinsic reward of $x_4$ thus sums up three terms, namely, the loss associated with the history representations $b_{1, 3}$, $b_{2, 2}$, and $b_{3,1}$. Will such design leads to a higher intrinsic reward of $x_4$ than $x_2$? Is such a preference for ending steps desirable for exploration?

limitations: N.A.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper builds on the BYOL work from 2020 and explores using it in the context of curiosity-driven exploration. It learns a world model via an RNN + self-supervised loss mechanism. This mechanism, optionally with an external task loss, is used to train an RL policy in order to play Atari and DM-Hard-8. Besides presenting superhuman results on the ten hardest Atari games, they also present favorable exploration results on Montezuma's Revenge. Notably, they do not use any human demonstrations.

They compare to other works in the curiosity-driven space on the metric of mean CHNS (clipped human normalized score). Across all ablations except the one fixing the EMA update to not update, their reported method outperforms the baselines and is superhuman when allowed to train on longer sequences. 

They show the exploration results on Montezuma's Revenge to test what happens in the purely intrinsic exploration setting. Because there are no external reward results given, we should not expect the model to beat the game but instead to explore the setting. Compared to the curiosity-driven baselines, it does that, finding 20 rooms compared against 17 in the published results.
strengths_and_weaknesses: This paper is good. There doesn't appear to be anything world-changing in it, but it's a solid paper presenting a good idea and written well.

Originality:

The idea of forward prediction is not new, nether is BYOL, the surprisingly non-degenerative self-supervised model on which BYOL-Explore is built. Combining these concepts by using an RNN to do the prediction makes sense. This isn't original and I'm glad someone tried it.

Quality:

The quality of this paper is high. I don't see any experiments that I need to have run, nor do I see any major faults with their reasoning. This is a good piece of work. That being said, I certainly would _like_ to see some other experiments for my own interest :).

An example is what happens as you scale up or down the lambda on the DM-8 experiments. It's quite low at 0.01 right now and so I'm left wondering if that's just for optimization reasons or if the model actually just doesn't like to solve tasks as that gets larger. If it was an order of magnitude lower, would it succeed a lot better? In other words, what's the sensitivity of this parameter as it's a crucial cog in the presented algorithm.

Clarity:

This paper is clear and well written. Thank you for that.

Significance:

The results aren't game changing, but they are of great interest to the self-supervised learning and curiosity-driven learning communities. BYOL made a large splash when it debuted and this extension of BYOL is notable for the community. Others are likely to use it in their baselines and perhaps build upon it.

I'm still surprised that the BYOL models don't collapse to a single point, so more evidence of this mechanism proving fruitful in complex self-supervised methods is helpful for building my model of how this works.


questions: I don't have any major questions other than the one about the lambda sensitivity. It would be cool if you could keep going on the big model to see how it does in DM-Hard-8 as well, but I'd understand if the time/compute budget isn't there.
limitations: Yes, they understand that their model is quite a ways off from being realistically used yet. They talk in the conclusion about improving on DM-Hard-8 and then moving on to more difficult environments like NetHack.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a Bootstrap-your-own-latent inspired method for model learning to generate intrinsic rewards for exploration. The method consists of 2 parallel networks, one an exponential moving average of the other with goal of predicting k steps into the future conditioned on past actions, current state and k future actions. The predictions occur on the latent space with a cosine loss for training. The discrepancy between the predictions is used to generate rewards for the RL agent to induce exploration. The authors test their method on DM-Hard and hard exploration games in Atari and show considerable performance improvement with reference to curiosity baselines. 

strengths_and_weaknesses: Strengths:
1. Method is principled. The intuition behind why it works is clear from previous works. 
2. Well-written and easy to follow: The authors describe the method well along with the setup. 
3. Extensive experimentation: The experimentation suite is large, with experiments in first-person ego-centric environments as well as hard exploration problems in third-person atari games. 

Weaknesses:
1. Not a significant algorithmic development: The method proposed is not necessarily. While this is not a weakness on its own, it can be said that the contribution is not algorithmic development. 
2. Missing some related works: http://proceedings.mlr.press/v139/sontakke21a/sontakke21a.pdf and  https://arxiv.org/pdf/2109.08603.pdf among many many others. Please do a more thorough search and add relevant citations.  
 

questions: How does it work in presence of distractors? Some experiments of the "noisy TV" would be nice to see. 

limitations: The paper is principled and along the lines of previous works. The results are impressive and useful for the community. The evaluation is thorough in my opinion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a new way of providing intrinsic rewards for exploration, by learning a latent space and predicting its future latents. The intrinsic reward is then proportional to the error between the predicted and actual future latents, so that states that are harder for the world model to predict are rewarded more. The method achieves impressive results on two hard-exploration benchmarks: hard-exploration Atari games and DM-HARD-8. 
strengths_and_weaknesses:  Strengths:

The work shows impressive performance on Atari, and seemingly on DM-HARD-8 (note: I am very familiar with past Atari results so I can judge those at a glance. For DM-HARD-8 I am relying on the comparison included in the paper, hence the more qualified statement). I am particularly impressed by the performance on Pitfall, given in the appendix (IMO the authors should call that out more, as Pitfall is a harder game than Montezuma’s, and their performance improvement relative to, say, RND is larger on Pitfall than Montezuma’s).

I agree with the authors’ claim that the method is simpler than Go-Explore and Agent57, so it seems to represent a genuine improvement for the field.

Weaknesses:

The main weakness is that the paper fails to make standard declarations about how they ran the benchmarks, at least in the case of Atari (which again I am most familiar with). While I do think the work would be at least somewhat impressive no matter what these specifics happen to be, without them it is really hard to decide if the work is an incremental improvement or a game-changer in the field. 

Specifically:

- The number of sample frames is never stated. Instead, the x-axis of all plots is given in “Learner Steps” which is, frankly, a meaningless number. All Atari work I am aware of (including works that this work compares to like RND, Agent57, and Go-Explore) states the number of frames used in some form (note: it is standard practice to include skipped frames if using frame skipping, when updating, the authors should make sure they clarify their frame skipping strategy and whether it is included in their frame count). Without this number, it is hard to compare the method to, say, Agent57. Further, Figure 4 seems to indicate that the authors’ implementation of RND might perform substantially worse than that in the RND paper, but it is hard to tell if this is due to differences in sample complexity. If it is not, it would be interesting for the authors to briefly discuss why they think their RND implementation performs worse.
- Likewise, it seems to have been standard prior to this work to give the sample complexity on DM-HARD-8 as well, so the authors should also report it.
- No information as far as I can tell is given about how stochasticity is injected in the Atari results (no-ops, sticky actions, etc.). This has been a contentious point historically, with some evidence from Machado et al. [1] that sticky actions reduce average performance on Atari. Without knowing if they were used, it is hard to properly place the results. Further, RND (which is a primary comparison of this paper) was specifically designed to solve noisy-TV issues that arise due to sticky actions, so it would be useful to understand if this method is vulnerable to those issues (which I think it might be, see Limitations section).

I recommend the authors consult Machado et al. [1] for a set of evaluation recommendations for Atari. While I personally don’t expect every Atari paper to follow this exact evaluation protocol (the recommended frame complexity is too low IMO, and while I believe sticky actions etc. should generally be followed, I am conscious of the fact that they aren’t universally adopted), I do think deviations from these recommendations are important to note explicitly, and they usually are in Atari work I am familiar with.

Overall, my main request for this paper is for the frame complexity and the evaluation method for Atari (especially the way stochasticity is injected) to be clarified. The method is interesting enough and the performance is good enough that I will switch my rating to somewhere within the “accept” range when this information is provided (where exactly would depend in part on what this information is specifically: if the method uses, say, 10x fewer frames than Agent57, then that is a lot more impressive than if it uses around the same amount). However, I think it would do a disservice to the community to accept the paper in the current state, since these issues make it very hard to compare the results to competing methods.

[1] Machado, Marlos C., et al. "Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents." 2017. *Journal of Artificial Intelligence Research*



UPDATE: As promised, I am switching my rating to an "Accept" given that the information I requested was provided. My choice of going to "Accept" and not higher is primarily due to the information being in line with what I expected (frame complexity similar to Agent57, no sticky actions), which still qualifies these results as very solid and worthy of NeurIPS, but not absolutely groundbreaking.
questions: See above, reiterating here:

- What is your sample complexity?
- What form of stochasticity did you use on Atari (sticky actions, no-ops, nothing…)?
- Why do you think your RND performance appears lower than the RND paper?
limitations: One limitation that I would have liked discussed is the paper’s dependence on determinism or lack thereof. The RND paper tries to solve the following “noisy-TV problem” with sticky actions: suppose you are in room 1 of Montezuma’s and room 2 is immediately to your right, then depending on how the randomness of sticky actions work out, it may be that you only have a 50% chance of getting to room 2 during the next frame, and a 50% chance of staying in room 1. As a result, if you try to predict some feature of *future* frames for your exploration bonus, the exploration bonus may stay high for those stochastic states no matter how often they are explored, which is why RND tries to predict features of the *current* frame instead. Given that this work predicts features of future frames, it may be vulnerable to the noisy-TV problem stated above. The authors should either disclose this as a limitation or explain why they think their work  avoids this issue.
review_title: null

Meta-review: metareview: The reviewers agreed overall that this work is a solid contribution, combining multiple earlier methods/ideas. The experimental validation is good. Together, these merit acceptance. 
confidence: Certain
Decision: Accept