Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to formulate goal-conditional reinforcement learning as a variational inference problem, termed as outcome-driven RL, and shows that the variational objective naturally generates shaped reward signals for learning. The variational objective results in an off-policy temporal-difference algorithm. Experimental results in robotic control tasks show the advantages of the novel algorithm. 


main_review: The problem of goal-conditional RL is an active area of research and many recent progresses have been made. This work proposes a novel formulation of the problem and demonstrates that such a formulation leads to an algorithm that outperforms prior methods. The paper is well-written with clarity. The related work section does a good job of overviewing relevant literature in both RL and control. The experiments cover six task domains with different characteristics and demonstrate ODAC performs comparably or outperforms prior methods in most of the tasks. An ablation study is also performed to provide insights into the limitations of ODAC and show that ODAC is not sensitive to the accuracy of the dynamics model.




limitations_and_societal_impact: The authors mentioned the limitations of the proposed work and potential future work to improve it. The authors did not comment on potential societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper combines the recent work on viewing RL as Probabilistic Inference and the goal-conditioned RL setting. The proposed approach assigns rewards based on the likelihood of reaching the desired state-action pair and propagates this via learning a dynamic discount function based on time elapsed in the episode. The authors propose solving the objective via variational inference and demonstrate this leads to a probabilistic policy evaluation operator. An actor-critic approach is then taken to perform approximate policy iteration.  The authors show the advantage of their methods on high-dimensional continuous control tasks. 

main_review: ### Originality:

The combination of control as inference take on the off-policy goal-driven RL is novel to the best of my knowledge and is also motivated sufficiently. I have some questions regarding the approach, in particular:

- The connection between viewing the log-likelihood of reaching particular state-action pair as the reward has been explored also in other works that also take a probabilistic inference view on different RL settings and propose solving it via variational inference such as [1].  Also, the authors mention that the difference between [3] and this work is that [3] requires on-policy samples whereas the proposed approach is off-policy. Can the authors elaborate on why extending [3] to off-policy is particularly tough or challenging? Even though the proposed approach is based on off-policy updates it is still online, so a comparison could be made. What advantage does the proposed approach provides compared to an off-policy version of [3] or the general approach in [1]?

- The mechanism through which the rewards/values are propagated is via learning a state-dependent discounting function. I think the authors should also mention some of the other works that have look at a similar approach for learning the discount factor such as [2,5]. 

### Quality: 

The work is technically sound and empirically supported with sufficient ablation studies. However, I have a few questions regarding the choice of baselines. The authors motivate in Sec 5 that the main difference between the prior work on control as probabilistic inference assumes that the manual reward is required ($r_{\text{manual}}$) whereas the main advantage of the proposed approach is that instead of requiring manual reward function as input it requires manual goals as input. Based on this, I would have expected some of the goal-based version of the existing control as probabilistic inference as some of the baselines. For instance, why was VIREL [4] excluded from the baselines? The VIREL methodology should be extendable to this setting? If not, some discussion of why the other work prior work can’t be extended to this setting should be provided. 

### Clarity:
The paper is mostly clear. The paper references outcomes throughout the text without formally defining them. It wasn’t clear to me what is the difference between goal-directed and outcome-driven? Does outcome-driven signify the probabilistic nature of the original goal-directed setting? 
 

### Significance:

The paper looks at an important problem setting and the results seem to demonstrate that the proposed approach is able to achieve its goal. I have some reservations about the originality and the need for this approach as mentioned in the points above. I think if the authors can shed some light on the questions raised above, then that will be helpful in raising the usefulness of this work to the community.

### References:

- [1] Weber, Théophane, et al. "Credit assignment techniques in stochastic computation graphs." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.
- [2] Pitis, Silviu. "Rethinking the discount factor in reinforcement learning: A decision theoretic approach." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.
- [3] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control 387 with events: A general framework for data-driven reward definition. In S. Bengio, H. Wallach, 388 H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Neural Information 389 Processing Systems (NeurIPS), pages 8538–8547. 2018.
- [4] Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, and Shimon Whiteson. VIREL: A Varia384 tional Inference Framework for Reinforcement Learning. In Advances in Neural Information 385 Processing Systems 32, 2019.
- [5] Xu, Z., van Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pp. 2396–2407, 2018.



limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes ODAC, a new RL algorithm for goal-directed tasks. Using a variational inference approach, ODAC automatically infer a dense environment reward from the goal and derives a probabilistic Bellman backup operator. The latter is used to develop an off-policy RL algorithm. The paper presents detailed theoretical derivation and empirical results on a suite of environments.

main_review: The paper is well presented. The theoretical derivation is detailed and easy to follow. In addition, the derived practical reward appears intuitive, including a  log-likelihood term for reaching the goal in the next step and a KL term for long-term reachability. The empirical results shows improved performance over existing methods.

**Presentation**: There appears to be some inconsistencies in the presentation of the method. For the reward function, Eq. 10 is different from Eq. 18 (the subscript $t$ in the KL term is off by 1), and again different from Alg. 2 in the appendix (the coefficient $q(\Delta=1)$ for the log-likelihood $\log p(g|s, a)$ is missing). Could the authors please clarify which reward function is used in practice. If approximation is used, please provide some intuition/motivation for the approximation. For the value function, Eq. 8 and Eq. 13 have a different sign for the KL term. It appears that Eq. 13 has the incorrect sign.

**Method**: Alg 1 and Alg 2 in the appendix are different. It appears that the general idea of hindsight experience replay (HER) is part of Alg 2, but not part of Alg 1. It is important to highlight if HER plays a significant role in the practical algorithm, and study it in the ablation (e.g. ODAC without HER vs ODAC with HER).

**Empirical Evaluation**: The experiment setting from the paper appears to restrict the goal to a single state (i.e. no oracle goal sampling) during training. It is difficult to understand why this setting is chosen, given that oracle goal sampling appears easily implementable, and was used by several previous methods originally. Could the authors provide the comparison results for more environments (in addition to Fetch Push in the appendix), allowing oracle goal sampling? This would allow a more comprehensive comparison with the baseline methods and a better understanding of the ODAC’s advantages over existing methods.

Table 1 appears to also suggest that the learned dynamic model $p_d $  performs noticeably worse than a fixed one, wrt to the policy learning. Specifically, the learned dynamic model obtains worse mean performance and noticeably larger standard deviation in 4 out of 6 tasks. Could the authors provide some intuition as to why this is the case, since this is rather unexpected. I cannot support the current interpretation that the proposed method is insensitive to the dynamic model.

Minor:
In A.15, t should be t’ in the summation sign. Line 808 has a missing latex reference.

------------- After responses ---------------------
My concerns are mostly addressed. I have revised my score.
limitations_and_societal_impact: As the proposed method is quite complex, a general limitation is the difficulty in understanding the actual working of the method, and its advantages over the baseline methods. Several related issues are discussed above, such as why the learned dynamic model performs worse than a fixed one, whether HER/goal relabeling is a critical component for ODAC as well, and how much of the observed performance improvements comes from restricting goal states during training.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper derives goal based reinforcement learning as a task of variational inference.
They propose a novel bellman operator and learning scheme and provide extensive evaluations across various complex tasks.
Empirical evidence backs the theoretical results showing that this method works in practice.
main_review: I find this line of work to be very interesting and important.
The problem formulation and paper are well written and the experiments show the ability of this approach to solve problems in practice.

Two minor comments:

1) I'd like to see a clean analysis in a tabular setting, while deep learning experiments are impressive they usually require a lot of "hacks" and it isn't always clear why the method works better/worse than prior work. This is not to say the method does not achieve impressive results nor that the inability/ability to break SOTA results in deep RL are sufficient/relevant for acceptance/rejection.

2) The numerical table is a bit misleading. I know that the common practice recently is to bold methods with the better mean performance. I'd urge the authors to re-think this broken approach and highlight all results that *might* be optimal (overlapping confidence intervals at the top).
limitations_and_societal_impact: -
review_title: null

Meta-review: The paper proposes a new variational inference approach for goal-based reinforcement learning. The author's goal is to more directly tackle reinforcement learning problems where the goal is to reach a certain outcome, without resorting to manual reward shaping. Similarly to some previous work, they define an inference objective which is maximize using techniques from variational optimization. Different from such works, in their inference formulation they condition on the goal being reached, rather than using the reward function as pseudo-likelihood. 

The reviewers consider the considered problem (goal-conditioned RL) important and very relevant for the machine learning community. 
The resulting approach was considered novel by the reviewers, especially the extension of a framework similar to VICE [3] to an off-policy setting. 
The reviewers also considered the paper technically sound and sufficiently well supported by experiments and ablations (that show the method outperforms e.g. SAC, another method in this space). 
Also, the reviewers consider the paper well-written. 

Decision: Accept (Poster)