Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors offer the perspective that the goal-conditioned Q-value for state-action $(s,a)$ can be viewed as the density of that goal ($g$) under a predictive model $F(g|s,a)$. The density estimator is modeled with normalizing flows (RealNVPs) and trained with MLE using policy rollouts. The estimator is integrated with TD-learning and actor-critic algorithms for goal-conditioned-RL and imitation-learning are derived. Experiments for the former include a comparison with HER on fetch-push/slide tasks, while for IL, the proposed approach is compared with GAIL on two locomotion tasks and shown to exhibit superior demonstration-efficiency.

I would like to appreciate the authors for an excellent presentation of the material. The notation used is rigorous and this makes understanding the hairy parts of the approach easier. Unifying goal-conditioned-RL and IL under a common view of training an agent to reach desired states and leveraging density estimators to achieve that is an interesting approach. I do have some concerns which (currently) hold me from rating this paper higher:

1.	UVD in stochastic domains: It is argued that HER suffers from hindsight bias in stochastic domains; Figure 2d empirically shows this. However, the dip in the performance of UVD from deterministic to stochastic Fetch-slide is also noticeable. I am wondering if this is because the density estimation becomes challenging in highly stochastic domains, since the support of the modeled distribution (over $g$) gets wider with increasing stochasticity, assuming the time-horizon of the density estimator is the same. Could the authors please comment on this observation from Figure 2?

2.	Adding Fetch-push-noise results in Figure 2, using the same noise-model as that used for Fetch-slide-noise, would make for a better claim.

3.	Locomotion tasks modification: These tasks are modified by removing the x/y velocities from the state given to the agent at each timestep. However, these components are used in the internal transition-dynamics function and also the reward function. Therefore, I reckon that this modification changes the MDP into a POMDP. Does that sound correct? If yes, it’s a bit strange that the RL machinery developed for MDPs (GAIL and VDI) is applied to a POMDP task. For reference, it would be good to include the results w/o the velocity removal in the Appendix.

4.	Algorithm 1 vs. Algorithm 2: Is there a particular reason why Algorithm 2 does not use the immediate rewards and the density-estimator from the "next"-state-action in the TD-error computation, as it does in Algorithm 1?

Typo: $F_\omega$ should be $F_\phi$ in Algorithm 2?


===== POST REBUTTAL UPDATE =====

I maintain my score of 6.  While the proposed approach is interesting, the experimental section could be strengthen to firmly stake a claim that it's broadly applicable.  (c.f. point 2. in the review)
review_title: Updated review for UVD/VDI

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a new method to compensate for "hindsight bias". This bias can occur in stochastic environments, during the hindsight relabeling process, when the value of actions are overestimated due to the probability of the success of that action in that state being assumed deterministic. The paper proposes Universal Value Density Estimation to account for this overestimation. The method learns approximates the proper goal conditioned Q function using a value density estimate.

Well the discussion at the end of the related work section will most related papers is interesting it's a bit difficult to start to see the novelty and addition this paper has to this problem when previous methods have solved this task for finite Horizons.

The example in figure 1 is very odd it shows an action choice that is basically impossible to cross so you would never end up constructing or relabeling a trajectory that would reach another state beyond the black hole where the agent would fall off the cliff.

Algorithm 2 does not appear to be referenced in the paper, It would be far more helpful if the authors could highlight the differences between these algorithms carefully in the text. 

The results in this paper we only shown for a single environment. This does not elicit confidence in the paper's findings. Do either of the domains chosen for the goal conditioned rl tasks have stochastic transitions that would lead to a need for UVD? Artificially adding noise does not seem like a motivating situation for the use of the method...
review_title: Results Lacking.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper focuses on goal-conditioned policy learning in the environment with stochastic dynamics and tries to address the bias of HER. Given a policy and a sparse reward function according to the goal,  the universal value function is converted to the discounted probability density over goals. And the authors use the recent work RealNVP about density estimation to learn the universal value function. The proposed value density estimation is also extended to imitation learning. Experiments show that the proposed method outperforms baselines in both goal-conditioned policy learning and imitation learning.

Clarity:
The clarity in writing can be improved in some points. First, it will be good if more details about RealNVP are introduced. How is it applied to learn the probability density? Second, in section 3.3, "if the goal is within the time-horizon of the density estimator....". Does it mean another hyper-parameter, e.g. time horizon of the density estimator, is introduced here? Is the proposed method sensitive to the value of this hyper-parameter? Third, in figure 1, do the arrows represent the learned policy with HER and UVD. It will be more convincing if the learning curves on this toy sample is also presented (in the appendix). 

Originality:

Significance:
The proposed method is technically sound and experiments demonstrate that it performs well in comparison to the baselines in some continuous control tasks. My concern lies in the complexity of implementing the proposed method compared with the baselines, and whether the method can be easily used in other domains with rich observation space and discrete action space. 

Pros:
*The proposed method is well-motivated to solve a problem of HER for goal-conditioned policy.
*The proposed method works well in some continuous control tasks and the authors carefully discuss the advantage of the proposed method in different environment settings (Figure 2) and with the different datasets (Figure 3).

Cons:
*In table 1 and table 2, I notice the use of RealNVP introduces some hyper-parameters. Is it easy to tune these hyper-parameters? 
*Is the proposed method applicable to domains with discrete action space?
*In Appendix D, there is a discussion about dimensionality. The data samples on Mujoco has low-dimensional state features.  If the observation space is more complicated and rich (e.g. RGB images), could the proposed method be easily used for imitation learning as GAIL?
*What's the complexity of implementing the proposed method compared with the baselines? Can we easily use it as a better alternative for HER in goal-conditioned policy learning and GAIL in imitation learning without much additional cost of tuning or running the algorithm?

***Post Rebuttal***
Thank the authors for the response. My main concern about the generality of the proposed method is not fully addressed. The baseline HER and GAIL can easily be applied in the domains with discrete action space. Also, the base RL approach for discrete action space exists. Experiments will be more convincing if the proposed method can outperform the baselines on various domains.
review_title: Solid method, more extensive experiments will be helpful

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a method for solving (state-distribution matching) imitation learning and goal-conditioned reinforcement learning (GCRL) by training a value function both with standard TD learning and Monte Carlo (MC) learning. They point out that for the case of indicator rewards, the Q function corresponds to a density model over the discounted state occupancy measure, and so they propose to train the MC-Q function by training a normalizing flow via MLE. The authors propose to combine the estimates of the two during TD backups by taking the max of the two estimates, and demonstrate that the overall algorithm results in better performance on the Fetch GCRL tasks (relative to HER) and on “non-exploitable” versions of locomotion tasks which remove the velocity and termination information that past work on imitation learning has included.

Overall, the paper studies an interesting connection between GCRL and imitation learning, and the idea of training a goal-conditioned value function via MLE seems novel and intriguing. I have concerns that the contribution of the paper is a bit unclear, and the overall performance gains are rather small.

In more detail:

It’s unclear if the benefits are from the “Q function is a density model” perspective, or if the benefits are just coming from mixing TD and Monte-Carlo estimates together. For example, how would the performance change if the normalizing model F was simply replaced with a Q function trained on Monte-Carlo estimates?

There were a number of design decisions and claims that I did not understand. For one, I don’t understand why VDI is immune to hindsight bias. The authors have experiments that shows that there is some small performance gain when stochasticity is introduced, but it seems that the distribution of goals that F is trained on will similarly be biased. Another question is why the authors propose to approximate the two values by taking their max. Is there any theoretical reason why this would be desired? Lastly, what does it mean for a goal to be “within the time-horizon of the density estimator”? Is there a mathematical definition of this condition, and why taking the max would address this problem?

The related works section is generally good, though I recommend the authors also discuss the relationship to [1] which also uses the gradient of the state distribution to train the policy.

Lastly, perhaps the main weakness is that the performance gains seem relatively small, which makes the impact of the work relatively small given that the main contribution seems to be empirical.

[1] Hazan, Elad, et al. "Provably efficient maximum entropy exploration." International Conference on Machine Learning. 2019.


--- Post Rebuttal ---

I had some confusion and concerns about the paper. Most of the confusions were addressed and made me view the paper slightly more favorably. However, my main concern wasn't addressed. In particular, it's unclear if the benefits are because of the "Value Density Estimation" benefits or if it's because of the mix of non-bootstrap updates. Comparing to a method that mixes in more Monte Carlo estimates (e.g. n-step bootstrap) would address this concern. Theoretically, the authors suggest that the reward should be 1/epsilon with epsilon -> 0. However, it seems like this was not done in the experiments, and it's unclear how epsilon should be decayed in practice. Given these concerns, I will maintain my score.
review_title: Interesting work, but limited experimental results and unclear contribution

Meta-review: This paper makes an interesting connection between the density matching in imitation learning and reaching the goal state in goal-oriented reinforcement learning. Reviewers generally expressed that the paper proposes an interesting approach, but some aspects of the paper need room for improvement. By reinforcing the experiments that address the reviewers’ various concerns, this paper will make a good contribution towards reinforcement learning research.
Decision: Reject