Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of exploration in reinforcement learning. The key idea is to learn a goal-conditioned agent and do exploration by selecting goals at the frontier of previously visited states.  This frontier is estimated using an extension of prior work (Pong 2019). The method is evaluated on two continuous control environments (2D navigation, manipulation), where it seems to outperform baselines.

Overall, I like that the proposed method integrates some notion of novelty with the language of mutual information and density estimation. While this idea has been explored in prior work (as noted in the related work section), the proposed idea seems like a useful contribution to the literature. The use of convolutions to obtain a lower bound on mutual information seems neat. The experimental results are quite strong.

My main concern with the paper is a lack of clarity. I currently have enough reservations and questions (listed below) about the experimental protocol that I am learning towards rejecting this paper. However, if the paper clarified the concerns below, I'd be willing to increase by review.

Questions / Concerns:
* "[Prior work on mutual-information cannot] guarantee that the entire state space can be covered" -- In theory, I think these prior methods should cover the entire state space. Take the DIAYN objective, I(s, z) = H[s] - H[s | z]. This objective is maximized when p(s) is uniform over the entire state space and p(s | z) is a Dirac.
* "It is time consuming to collect enough samples to estimate an accurate state entropy" -- Can you provide a citation/proof for this? Why should we expect the proposed method to require fewer samples?
* "...entropy itself does not provide efficient information to adjust the action at each step." -- Can you provide a citation / proof for this? Also, what does "efficient information" mean?
* It seems like, if S is a finite collection of states in a continuous space, then it has measure zero, so its entropy should be zero. Can you explain why this is not the case?
* If I'm not mistaken, in equation 2, if we take (say) alpha = -0.5, then w_i is proportional to sqrt(p(s_i)), so w_i is an increasing function in p(s_i), not a decreasing function.
* Can you discuss how you might scale a KDE to high dimensions?
* "If the distribution has a larger range, the entropy is larger as well." -- Technically, this is not correct. You can construct distributions with larger ranges but smaller entropies.
* I think that Equation 3 should be the KL divergence between state marginal distributions, not between trajectories. If it were the KL between trajectories, it would include actions and policy terms.
* How are w_int and w_ext chosen? It seems like the method depends critically on the balance between these hyperparameters. Is w_int decayed over time? If not, why does the policy stop exploring once it has found the goal?
* What policy is used at convergence? It seems like the policy is conditioned on Z_t, so how is the Z_t chosen for evaluation?
* Fig 5 -- How are entropy and coverage computed? What are the maximum possible values for both of these quantities? What precisely does the X axis correspond to?
* Table 1 -- How did the baseline algorithms perform on this task?
* Fig 7 -- How did the baseline algorithms perform on this task? If the reward were sparse, shouldn't the Y axis be in the interval [0, 1]?
* "using coverage only during training is not suitable" -- Can you provide a citation/proof for this?
* "As a consequence, the entropy of the distribution of these points is also maximized" -- I believe that a finite number of points in a discrete space have measure zero, so they have zero entropy, regardless of the position of the points.



Other comments
* "What is the difference between *S* (in bold) and S_t?
* I would recommend using some notation other than p(s) to denote the smoothed/convolved density. 
* "history states" -- I was confused about what this meant until it was introduced two sections later.
* "assimilate the definition of curiosity in psychology" -- I think that others (e.g., Oudeyer 2007, Pathak 2017) have noted the similarities between curiosity in humans and RL agents.
* Check for backwards quotes in the related work section.
* "Self-Goal Proposing" -- Some more related works are [Florensa 2017, Savinov 2018]
* "space associated environment" -- I don't know what this means.
* "disc rewards" -- I'd recommend spelling out discounted
* "truncated Gaussian function with a narrow range" -- Can you explain precisely what this is?
* In equation 2, I think it'd be clearer to write p^(1+\alpha).
* For the experiment on the effect of variance, I'd recommend making a plot instead of just listing the values.
* In Section 4.3, it's unclear whether the physical robot was successful at solving the task.
* "We rewrite the equation…" -- This paragraph is repeated.
* Double check that \citet and \citep are used properly

--------------UPDATE AFTER AUTHOR RESPONSE------------------
Thanks for answering many of my questions. This was helpful for clarifying my understanding. However, since a large fraction of my concerns were not addressed, so I am inclined with stick with my original vote to reject the paper. Nonetheless, I should emphasize that I think this paper is on the right track and the empirical results seems strong. With a bit more work on writing, I think it would be a fantastic paper at the next conference.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new exploration algorithm by proposing a new way of generating intrinsic rewards. Specifically, the authors propose to maintain a "novelty frontier" which consists of states that have low-likelihood under some likelihood model trained on their replay buffer. The authors propose to sample from the novelty frontier using a scheme similar to a prior method called Skew-Fit, but replace the VAE with a kernel-based density model. To construct an exploration reward, the authors estimate the KL divergence between the resulting policy state distribution and the desired `state distribution, where the desire state distribution is a Gaussian centered around a point sampled from the novelty frontier.

Overall, the paper tackles an important questions of exploration, and while the concept of a frontier set is not novel, the authors propose a concrete instantiation that has promising results on continuous state spaces. I'm skeptical that this exact algorithm would work on domains with complex state spaces (e.g. images), where adding Gaussian noise to your state won't produce reasonable nearby states. That said, the general idea of fitting a new model to the latest trajectory and using KL as reward seems like a promising principle that could on its own scale. However, the theory seems a bit off and there are a few experimental details that make me hesitant to increase my score.

In details:

Theory:
I found the proof surprisingly long given that it amounts to saying that if (1) S = Z + N and (2) Z and N are independent, then
  H(S) >= H(N)
and so
  H(S | Z) - H(Z | S) = H(S) - H(Z) >= H(N) - H(Z)
Perhaps more worrisome is the statement, "we consider to maximize h(S|Z) - h(Z|S)". Unless I misread the paper, the authors do not maximize this quantity. Instead, they *fix* this quantity by choosing a fixed entropy of N. Worse yet, this quantity is actually minimizes since, while h(N) is fixed for the duration of the experiment, h(Z) is maximized ("To increase h(Z), we need to add..."). It would be good for the authors to address this concern, given that the claim of the paper is that they are "maximizing the entropy of the visited states." It seems like a simple answer is the following: given that S = Z + N, if N is fixed to some Gaussian distribution, then the authors simply need to maximize H(Z), which they are already doing. I'm not sure why the authors need to reason about H(S | Z) - H(Z | S).

Experiments:
Can Table 1 be replaced with the learning curves? The numbers 90% success and standard deviation of 3% seem like arbitrary numbers. It doesn't preclude the possibility that (e.g.) Skew-Fit or RND receives a 99% success rate with standard deviation of 3.1%. Figure 11 and 12 of the Appendix don't convince me that threshold at 90% and 3% is a particularly good choice.

Can the authors summarize the difference between coverage and entropy in the main paper? It seems like an important distinction. Given that the authors did not use all 8 pages, it would be good to explain it there rather than in the Appendix.
How sensitive is the method to the hyperparameter alpha? How was it chosen? Is it the same alpha chosen for Skew-Fit?
How was N chosen for the door environment?
Is Figure 7 (left) showing the performance on the simulated or real-world robot?
If it was done on the real-world robot, were there any important details in getting sim-to-real-world to work?
In Figure 5, why does there seem to be discrete jumps in the learning curves for "DoorOpen Coverage"?

I would be inclined to raise my score if:
1. The authors clarify why studying the quantity H(S | Z) - H(Z | S) is particularly important.
2. Address the concerns raised over the experiments.
3. Discuss more explicitly under what assumption they expect for this method (with a Gaussian KDE) to work well

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose an exploration objective for solving long-horizon tasks with sparse rewards. While the paper is heavily influenced by the recent Skewfit work (Pong et al, 2019),  it aims to solve a different problem (that of exploration for sparse reward RL), since Skewfit is interested in exploration and goal sampling for self-supervised (i.e. no external rewards) RL. The central idea in this paper is to encourage exploration by maintaining a distribution over the "current frontier", i.e. the set of points in the state space that are closed to the "edge" of what is explored by the policy , sampling points around this frontier, and encouraging the policy to reach these sampled points. The paper compares to RND (a method that is representative of state-of-the-art in exploration bonuses, I believe) and also to Skewfit, and outperforms these methods in two non-standard environments: door opening and point-mass navigation. 

While the I think the empirical contributions are solid, and authors provide code to reproduce the results, I found the paper a bit hard to follow and understand, and different subsections in the technical section (Section 3) did not seem to have a unified narrative running through them. I will wait to see if other reviewers agree or disagree with me on this front, and if they do agree, then I think the paper will need substantial edits to improve its clarity before it is ready for publication. 
review_title: Official Blind Review #2

Meta-review: While the reviewers generally appreciated the ideas presented in the paper and found the overall aims and motivation of the paper to be compelling, there were too many questions raised about the experiments and the soundness of the technical formulation to accept the paper at this time, and the reviewers did not feel that the authors had adequately addressed these issues in their responses. The main concerns were (1) with the correctness and rigor of the technical derivation, which the reviewers generally found to be somewhat questionable -- while the main idea seems reasonable, the details have a few too many question marks; (2) the experimental results have a number of shortcomings that make it difficult to fully understand whether the method really works, and how well.
Decision: Reject