Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: ### Summary

This paper proposes to optimize the state marginal distribution to match a target distribution for the purposes of exploration. This target distribution could be uniform or could encode prior knowledge about downstream tasks. This matching can be done by iteratively fitting a density model on the historical data from the replay buffer, and training a policy to maximize the log density ratio between the target distribution and the learned density model. Experiments are performed on two domains: a simulated manipulation task and a real robotic control task. Overall, the paper is well-written.


### Review

Recommendation: weak reject for the reasons below. The main reason is that this paper ignores very similar prior work which is not properly credited. 
 

The algorithm proposed here is very similar to the algorithm proposed in [1]:
- the objective proposed in  equations (1) and (2) is the same as the second objective in Section 3.1 of [1]. 
- Algorithm 1 here is almost identical to Algorithm 1 in [1]

The work of [1] is only briefly mentioned in the related work section, and from the description there seems to be a fundamental misunderstanding of it. 
It says "their proposed algorithm requires an oracle planner and an oracle density model, assumptions that our method will not require".

Making oracle assumptions is a tool for proving theoretical results, not a feature of an algorithm. An oracle can be any subroutine that one has reason to believe works reasonably well, and how well it works or not is captured in its accuracy parameter (usually \epsilon).
They are used to break down a  more complex algorithm into simpler subroutines (called oracles), and deriving a guarantee on the complex algorithm in terms of the quality of the oracles. 
For example, [1] assumes a density estimation oracle, which could be instantiated as a kernel density estimator, a VAE, count-based density estimation in the tabular case, etc. 
It also assumes a planning oracle, which could be instantiated using any method for learning a policy (PPO, SAC, policy iteration, etc), or some search method if the environment is deterministic. 
The accuracy of the oracles are reflected in the \epsilon_0 and \epsilon_1 parameters, which then show up the guarantee for theorem 1. 

Theorem 1 of [1] also shows that the entropy of the policy mixture (i.e. replay buffer) matches the maximum entropy over the policies in the policy class, which is one of the main theoretical claims of the work here. 

Given this, I don't see this paper as making any new algorithmic or theoretical contributions. On the other hand, [1] had a very limited empirical evaluation and it would be valuable to have a more thorough empirical investigation of this type of method in the literature. This paper partially does that in the sense that they apply more modern methods (VAEs rather than counts/kernel density estimators) on more complex tasks (a simulated manipulation task and a real robot), and their experiments seem well-executed with proper comparisons. However, since the primary contribution of this paper seems to be empirical, I don't think the current experiments on two domains are enough. 

I think this paper could be fine for publication with a fairly significant rewrite placing it in the context of prior work, and expanding the experimental section. 
My suggestions are to add experiments on several other continuous control tasks (Mujoco/Roboschool) as well as hard exploration Atari games (Montezuma, Freeway, Pitfall etc), to see how well the density estimation works in pixel domains (and the effect of historical averaging). I would be willing to raise my score if these changes can be made within the rebuttal period.   


[1] "Provably Efficient Maximum Entropy Exploration", Hazan et. al. ICML 2019. 
http://proceedings.mlr.press/v97/hazan19a/hazan19a.pdf

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary:
The paper proposes to frame exploration in reinforcement learning as a distribution matching problem. More specifically, the proposed method (SMM) aims to minimize the reverse KL between the state distribution induced by the policy and a desired state distribution. The desired state distribution can be used to guide exploration (e.g. by penalizing bad states) or can be chosen uniformly (resulting in a policy that maximizes state entropy).
SMM iteratively optimizes the policy and a model for its induced state-distribution. The latter is used for approximating the policies state entropy. The algorithm is framed as a two-player zero-sum game between a "policy player" and a "density player". In order to justify that one of the players is assumed fixed while optimizing the other player, SMM optimizes the players against a historical average of the opponent. Such approach is known as fictitious play in game theory and ensures convergence to a Nash equilibrium.
The density player maximizes the likelihood of the states encountered during all previous roll-outs using a VAE. The policy update is framed as a standard reinforcement learning problem where the log-pdf of the target distribution serves as reward and the log-pdf of the model acts as cost. For practical reasons, the policy update does not consider an historical average but only uses the most recent density model. As the learned exploratory policy should correspond to an historical average exploration for the downstream task is achieved by sampling one of the learned policy at the beginning of each roll-out. The historical averaging also benefits prior exploration methods.
Notably, the appendix also describes a modification that learns a mixture of policies in each iteration where a discriminator approximates the responsibilities for each component which is used as additional reward to divide the search space among the components. SMM is compared to prior exploration methods, ICM and Pseudo-Counts and standard SAC, on a simulated robotic box-pushing task and against SAC on simulated point-mass problem and a real-robot valve turning task.

Significance:
Efficient exploration is arguably the main challenge of reinforcement learning as providing shaped reward functions is difficult even for experts and may lead to undesired behavior. I think that maximizing the state-entropy (or matching target state-distributions if available) is a sound objective for learning exploratory policies and the paper could, thus, be of relatively broad interest. 

Novelty:
Maximizing the state entropy for exploration has already been used by (Hazan et al., 2018, reference from manuscript). However, in contrast to this prior work, SSM does not need to learn oracles that predict optimal state-distributions/policies for any given policies/reward functions. While distribution matching is a common approach to imitation learning, it has been little employed for manually specified distributions. Still, a similar objective has been used to replace reward functions by desired distributions in a RL-like setting [1] (not for exploration). However, their approach is quite restricted by assuming Gaussian target distributions.

Soundness:
If I understand correctly, fictitious play assumes optimal responses to understand that the state distribution would often be more important than provably converge to a Nash equilibrium. The paper fails to provide stopping criteria for the optimization steps of the individual players; however, I assume that only few gradient steps are used for practical reasons. Hence, I am not sure whether the actual algorithm can be justified by the theory.

The paper mentions that VAEs are used to model the state distribution. Given that VAEs are not likelihood-based, I do not understand how the reward term log(q(s)) can be computed. 

Clarity:
The paper is well-written and the structure seems fine. 
I think that the density estimation should be better discussed. Models of the state distribution of the policy are often highly desirable, not only for optimizing its entropy, but also, for example, for importance sampling. However, modeling the state distribution is also inherently difficult--especially for large state spaces.

Experiments:
I like that the paper uses a real robot experiment and the ablations with respect to the historical averaging are interesting. Unfortunately, the paper only compares to standard SAC on the hallway task and on the real robot task. I would not consider the entropy regularization of SAC a proper baseline for a proposed exploration technique.

Questions:
Is the same extrinsic reward, i.e. log(p*), also provided to the other exploration methods?

Some imitation learning methods such as GAIL are also able to match desired state-distributions. I think that these methods could be in principle also applied to the setting considered in the manuscript by using samples from the desired distribution as demonstrations. The paper briefly mentions that IRL methods are not applicable because they require access to trajectories, however, the discriminator of GAIL is only trained on individual state samples. I also do not see a problem of providing unachievable demonstrations to such imitation learning methods because, such like SSḾM, they would try to minimize the divergence. I think that GAIL would actually be an important baseline for the proposed method.

How does the method scale with the dimensions of the state? SSM has only be evaluated on relatively low-dimensional problems (compared to some rllab/mujoco tasks with >100 states). I would assume that obtaining meaningful density estimates in such settings might be problematic. May imitation learning methods based on discriminators actually be more promising?

SMM only considers matching state distributions. If I understand correctly, the approach could be easily generalized to state-action distributions, correct? Wouldn't it make sense for some tasks to also explore in the action space?

Decision:
I like the paper in general. The optimization problem seems well-motivated and the algorithm seems reasonable fine. I also like that the paper includes a real robot experiment. However, I am not convinced by the derivation of SMM based on fictitous play and I think that it should be better evaluated with respect to existing exploration methods and also with respect to imitation learning methods. I am therefore slightly leaning towards rejection, currently. 

Typo:
"Note[sic] only does SMM explore a wider range of angles than SAC, but its ability to explore increasing[sic] throughout training, suggesting that the SMM objective is correlated with real-world metrics of exploration."

[1] Arenz, Oleg, Hany Abdulsamad, and Gerhard Neumann. "Optimal control and inverse optimal control by distribution matching." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Update: I thank the authors for their response and I think the added baselines and more in depth discussion of prior work have improved the paper. However, given the limited novelty of the technical contribution, I believe the experimental section should be further extended (i.e. add a wider variety of domains and make thorough comparisons to relevant methods) in order to draw more general and robust conclusions. 

Summary:

This paper proposes tackles the problem of exploration in RL, with a focus on learning an exploration policy that can be used for a variety of different tasks. They introduce a formal exploration objective that promotes efficient exploration and provides a mechanism for injecting prior knowledge about the task. They also design a practical algorithm to maximize the exploration objective. Finally, they empirically show that  their method can outperform other SOTA exploration methods on challenging exploration tasks for locomotion and manipulation,  both in simulation and in the real world.

Main Comments:

I’ve found the mathematical formulation to be sound and the empirical evaluation convincing. Overall, the paper is clearly written and the authors are quite transparent about the assumptions made. In addition, the problem of learning exploration strategies that are task-agnostic and force the agent to effectively explore within each episode (since the goal or task is not observed) is an important problem for RL and perhaps a more realistic setting than the single fixed-task one. However, I believe some important methodological details are missing from the paper and the empirical evaluation can be improved. In particular, the paper would be more convincing if it contained comparisons against SOTA exploration (e.g. curiosity-driven, pseudo-counts, noisy-networks etc.) and inverse reinforcement learning (e.g. GAIL) methods for all the environments. Such baselines are completely missing in the Navigation and Real-World tasks. 

However, as the authors note, most baselines used for comparison have been designed specifically to learn from sparse rewards in single task settings and do not have any direct mechanisms for including priors or learning to explore well for any task from some distribution. So I wonder if it’d make sense to include baselines that do make use of prior knowledge such as IRL (i.e. GAIL) or some other state-matching approach. Those could be more appropriate and powerful baselines. 

Another potential disadvantage of this method seems to be the need for a prior, which may be difficult to design or even lead to suboptimal policies if it is not well designed. However, as the authors note, it is still a weaker requirement than having access to demonstrations for example and the  prior could potentially be learned from human preferences / restricted feedback. 

Other Comments  /  Questions:

1. SMM uses prior information about the task in the form of the target distribution. Given this, I am worried that the baselines have a clear disadvantage. Did you do anything to provide the baselines with the same type of prior knowledge about the task? It would be useful to see how they would compare if they had access to the task prior (in some way) as well. 

2. Can you provide more insights into how this differs from variants of MaxEntRL and InvRL? Both analytically and in practice. I believe a more extended discussion of this would be valuable for readers and would alleviate some of the concerns regarding the novelty of this contribution and how its place in the broader literature.

3. In the Navigation environment, how would the  results change if the goal were visible  (i.e. part of the agent’s observation)? I believe that most baselines would consider that scenario and it would  be interesting to see whether the qualitative conclusions hold or not in that case. I would expect other exploration methods to  be faster in that case.

4. I also wonder if perhaps the reward is actually not that sparse  in some of these tasks but  because it is not visible, it makes the problem much harder for the baselines, which were designed to deal with very sparse reward. Can you comment on the reward sparsity in these tasks?

5. At the top of page 2, you mention that there is a training phase in which the agents learn to optimize the exploration objective and at test time, it is trained with extrinsic reward. Can you please clarify on how these stages reflect in the results and what is  the regime used for the other baselines? Are they also pretrained on a variety of tasks with only their exploration bonus / intrinsic reward and then fine-tuned with extrinsic reward?

6. In Figure 2 (c), why is it that the gap between SMM and SAC decreases as the  number  of halls increases? This seems counterintuitive and I would’ve expected to increase since I do not see why SAC would get better and SMM would get worse. 

7. How do you learn the density model? You mention the use of a VAE but the details of how this is trained are not specified.

8. On page 5 before section 4,  you mention that you approximate the historical average  of the density model with the most recent iterate. Can you include ablations on how good this approximation is and how the results change  if you were using the historical average instead?

9. In Figure 4 (b), SMM’s variance of the positive value of the angle differs significantly from the negative one. This strikes me as counterintuitive. Do you have any  intuition on why that is?







review_title: Official Blind Review #2

Meta-review: The paper provides a nice approach to optimizing marginals to improve exploration for RL agents.  The reviewers agree that its improvements w.r.t. the state of the art do not merit a publication at ICLR.  Furthermore, additional experimentation is needed for the paper to be complete.
Decision: Reject