Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:

This paper introduces a new intrinsic reward for aiding exploration. This one
is based on learning a distribution on parameters for a neural network which
represents the dynamic function. The variance the predictions from this
dynamic function serves as the intrinsic reward. Results are compared against
several current state-of-the-art approaches.

Feedback:

Using uncertainty as an instrinc reward to guide exploration is a
very active area of research and it would have been more helpful
to say how this work differs from Burda et al, Eysenbach et al,
Gregor et al. 1.  The underlying algorithms are all very similar
and differ in only small and subtle ways. The main difference
with this work and Pathak et al. seems to be that the variance is
all coming from one particular conditional distribution rather than
an ensemble of models, but in Pathak et al it is also a distribution
over models.

Amortized SVGD is used instead of regular SVI in this work, but
it is never articulated why to problem benefits from using that
framework. This paper would greatly benefit from some explanation.
It is mentioned as a novel aspect of the work, but never really
justified at all.

The experimental results and convincing and do show a substantial
improvements over similar approaches in domains in ant maze
navigation and robot hand.

[1] Gregor, Karol, Danilo Jimenez Rezende, and Daan
Wierstra. "Variational intrinsic control." arXiv preprint
arXiv:1611.07507 (2016).

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Update: I thank the authors for their rebuttal. Having read the other reviews I still stand by my assessment and agree with the other reviewers that the empirical validation should stronger, adding more baselines and conducting experiments on the same environments as your main competitors for fair comparison. 

Summary
This paper proposes a Bayesian approach for modeling the agent's uncertainty about forward predictions in the environment, that is, given a state and action how likely the next state is. The uncertainty is then used to define an intrinsic reward function. The paper has sufficient technical depth. However, I am disappointed by the comparison to prior work.

Strengths
Interesting non-parametric approach to estimating uncertainty in the agent's forward dynamics model
Clearly written paper with sufficient technical depth
Well structured discussion of related work

Weaknesses
My main problem with the paper is a missing fair comparison to prior work. The two main contenders are MAX by Shyam et al 2019 and the Disagreement approach by Pathak et al 2019. Comparing the results on AntMaze presented here with those in Shyam I see that MAX only gets to high 80s in terms of maze exploration rate, while in their paper it is in the 90s. In comparison to Pathak, as far as I understand, a different robotic manipulation task was used (HandManipulateBlock here in comparison to the grasp and push objects on a table task by Pathak). Moreover, there are no experiments comparing the proposed approach to the stochastic Atari environments investigated in Pathak et al 2019. I understand this would require dealing with discrete action spaces, but I don't see why this would be infeasible. Overall, I believe this makes it hard to draw conclusions with respect to Shyam et al and Pathak et al and adding these missing comparisons would strengthen the paper substantially in my view.

Minor
p3: "Let f denote the dynamics model" – I believe it would be good to mention the signature of this function (it can be inferred from Figure 1, but it would be nice to make this explicit).
Questions to Authors
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Update: I thank the authors for their response. I believe the paper has been improved by the additional baselines, number of seeds, clarifications to related work and qualitative analysis of the results. I have increased my score to 3 since I still have some concerns. I strongly believe the baselines should be tuned just as much as the proposed approach on the tasks used for evaluation. The baselines were not evaluated on the same environments in the original papers, so there is not much reason to believe those parameters are optimal for other tasks. Moreover, the current draft still lacks comparisons against stronger exploration methods such as Pseudo-Counts (Ostrovski et al. 2017, Bellemare et al. 2016) or Random Network Distillation (Burda et al 2018).  

Summary:
This paper proposes the use of a generative model to estimate a Bayesian uncertainty of the agent’s belief of the environment dynamics. They use draws from the generative model to approximate the posterior of the transition dynamics function. They use the uncertainty in the output of the dynamics model as intrinsic reward.

Main Comments:

I vote for rejecting this paper because I believe the experimental section has some design flaws, the choice of tasks used for evaluation is questionable, relevant baselines are missing, the intrinsic reward formulation requires more motivation, and overall the empirical results are not convincing (at least not for the scope that the paper sets out for in the introduction).  

While the authors motivate the use of the proposed intrinsic reward for learning to solve tasks in sparse reward environments, the experiments do not include Moreover, some of the tasks used for evaluation do not have very sparse reward (e.g. acrobot but potentially others too). Without understanding how this intrinsic reward helps to solve certain tasks, it is difficult to assess its effectiveness. While state coverage is important, the end goal is solving tasks and it would be useful to understand how this intrinsic reward affects learning when extrinsic reward is also used. Some types of intrinsic motivation can actually hurt performance when used in combination with extrinsic reward on certain tasks. 

I am not sure why the authors chose to not compare against VIME  (https://arxiv.org/pdf/1605.09674.pdf) and NoisyNetworks (https://arxiv.org/pdf/1706.10295.pdf) which are quite powerful exploration methods and also quite strongly related to their our method (e.g. more so than ICM).

Other Questions / Comments:

1. You mention that you use the same hyperparameters for all models. How did you select the HPs to be used? I am concerned this leads to an unfair comparison given that different models may work better for different sets of HPs. A better approach would be to do HP searches for each model and select the best set for each.
2. Using only 3 seeds does not seem to be enough for robust conclusions. Some  of your results are rather close 
3. How did you derive equation (1)? Please provide more explanations, at least in the  appendix.
4. Why is Figure 3 missing the other baselines: ICM & Disagreement? Please include for completeness
5. Please include the variance across the seeds in Figure 4 (b). 
6. How is the percentage of the explored maze computed for Figure 5? Is that across the entire training or within one episode? What is the learned behavior of the agents? I believe a heatmap with state visitation would be useful to better understand how the learned behaviors differ within an episode. e.g. Within an episode, do the agents learn to go as far as possible from the initial location and then explore that “less explored” area or do they quasi-uniformly visit the states they’ve already seen during previous episodes?  
7. In Figure 6 (b), there doesn’t seem to  be a significant difference between your model and the MAX one. What happens if you train them for longer, does MAX achieve the same or even more exploration performance as  your model? I’m concerned this small difference may be due to poor tuning of HPs for the baselines rather than algorithmic differences?
8. For the robotic hand experiments, can you  provide some intuition about what the number of explored rotations means and how they relate to a good policy? What is the number of rotations needed to solve certain tasks? What kinds of rotations do they explore -- are some of them more useful than others for manipulating certain objects? This would add context and help readers understand what those numbers mean in practice in terms of behavior and relevance to learning good / optimal policies.

review_title: Official Blind Review #1

Meta-review: There is insufficient support to recommend accepting this paper.  The authors provided detailed responses, but the reviewers unanimously kept their recommendation as reject.  The novelty and significance of the main contribution was not made sufficiently clear, given the context of related work.  Critically, the experimental evaluation was not considered to be convincing, lacking detailed explanation and justification, and a sufficiently thorough comparison to strong baselines, The submitted reviews should help the authors improve their paper.
Decision: Reject