Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes iterative amortized policy optimization, a class of methods that use the gradients online to optimize the policy iteratively.
main_review: This paper proposes iterative amortized policy optimization, a class of methods that use the gradients online to optimize the policy iteratively. To the best of my knowledge, the idea is novel and original. The paper is generally well-written and easy to follow and in particular, I appreciate the authors making a detailed, aligned, and symmetric comparison of direct amortization and iterative amortization (the one-to-one pros and cons, as well as the graphical demonstration, and the algorithm, etc...). The empirical experiments are quite comprehensive, which includes quite some qualitative analysis of the claimed properties. And indeed, the proposed method is validated by the learning curves in MuJoCo continuous control tasks benchmark, as the proposed method outperforms and matches direct amortization in most continuous control tasks. Some questions and/or suggestions: 1. could you talk more about what's the added complexity of the proposed method? 2. In Fig.5 (c), while direct amortization seems to be very stable, iterative amortization seems to have two moments with sudden performance collapse, is there any reason why? 3. the increase on performance gains on MuJoCo tasks (Fig.5) and the corresponding decrease in amortization gap (Fig.6) do not seem to be be consistent, e.g. there is a significant decrease in amortization gap on Ant, while the expected reward is almost the same. So maybe it's good to investigate what brings the performance gains? 4. for exploration, it's also helpful to test the proposed algorithm in MuJoCo tasks with sparse rewards.



limitations_and_societal_impact: See the main review.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work analyzes policy optimization from an inference perspective. It first shows that policy optimization with entropy or KL regularization in DRL with a policy network is a form of amortized optimization where instead of directly optimizing the action distribution parameters, we optimize a network which outputs the action distribution parameters. This results in an amortization gap which negatively impacts performance and exploration. Next the authors propose an alternative iterative amortization method for training the policy network. This new approach is shown to have various benefits which the authors demonstrated through a series of experiments on the MuJoCo environments.
main_review: Overall, I found this paper to be very insightful. It brings forward many interesting ideas which helps to bridge the gap between inference and RL. The paper was very easy to read. The authors did a good job of explaining the issues of direct amortization and presenting their novel technique. The diagrams were also very helpful! The discussion on amortization gap is particularly interesting and to the best of my knowledge not something that has previously been explored by the RL community. The experiments in this paper are very well designed in explaining the iterative amortization approach. I have two questions for the authors which I hope they can clarify in their response:

- In the section on value overestimation (Section 3.3), you mentioned that “If the policy can exploit regions of high uncertainty, the resulting target values will introduce positive bias into the estimate. More flexible policy optimizers exacerbate the problem, exploiting this uncertainty to a greater degree.” Could you elaborate on this claim a little further? Why are more powerful policies more prone to overestimation of the Q function? This seems to be a fairly central point to Section 3.3.
- Empirical performance of the iterative amortization approach does not seem to be that strong. While there are some improvements on some of the simpler environments. Performance on the most challenging MuJoCo environments (Ant, Humanoid, HumanoidStandup) appear to be on par with the direct amortization method. Any sense why the iterative approach does not result in any improvement on these environments?

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors, via a standard reinterpretation of reinforcement learning as an inference problem, discusses the resulting “amortization gap” that results from using policy networks to optimize the variational lower bound. In order to close this gap, the authors propose to train an iterative procedure parameterized by $\phi$, which uses estimated gradients of the return to iteratively refine $\pi(\cdot | s_t)$ for the current state $s_t$. They then provide an extensive experimental evaluation demonstrating their proposed method and several desirable properties, including adaptability to gradients estimated by different methods.
main_review: **Originality:**  
The authors leverage a known connection between variational inference and reinforcement learning to motivate their problem (which is known in the VI literature). Although the techniques are not novel, they are novel in their application to the RL setting.

**Quality:**  
The submission is technically sound, with authors leveraging known techniques from the generative modeling literature. Furthermore, the authors present a very thorough experimental evaluation of their claims, with very interesting results. Although the performance gains in some of the tasks are not substantial, the demonstrated reduction in the amortization gap, inherent multi-modal behavior, and transferability between different value estimation methods are worthy of note.

As an additional experiment, it would be interesting to see whether the learned policy is robust to slight changes in the environment (e.g. changing weight/friction parameters of the MuJoCo simulation) as an extension of the generalization experiment.

**Clarity:**  
The paper is mostly clear, although as someone who is not as familiar with the generative modeling literature, a few parts of the paper were a bit confusing from my perspective.
For example, in the context of RL, what is estimation referring to in lines 112-116 (I assume in this case that it is the process of choosing an action distribution)? I think it would be really helpful to have these connections explicitly mentioned.

**Significance:**  
The paper introduces really interesting ideas, including having a trained policy which includes a trained optimizer that improves upon an initial action based on predicted returns. The work suggests several venues for future work, including across-task generalization, better policies for existing algorithms, among others.


limitations_and_societal_impact: The authors have adequately addressed the limitations of their approach. I particularly appreciated that the authors spent time addressing the value overestimation problem that is inevitably exacerbated by having a more expressive policy class.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel approach to policy optimization which leverages the connection between amortized variational inference and policy-based RL with function approximation. Specifically, rather than rely on a "direct" parameterization of the state-conditioned distribution over actions, in which an input state is mapped directly to distribution parameters (e.g., mean and variance), the approach adds an additional optimization procedure over the distribution parameters themselves, rather than those of the policy network. Experimental results are then used to demonstrate the benefits of this approach for performance, accuracy, and flexibility in the optimization process. 
main_review: Originality: The application of iterative amortization to policy optimization for RL is interesting and novel to the best of my knowledge. The connections between variational inference and RL with function approximation are deep yet not fully explored, and this is an excellent example of the interesting insights that can be gained through the lens of this relationship. The relationship to previous work and associated citations appear thorough. 

Quality: The submission appears technically sound, and the claims seem to be supported. However, it would be helpful if the authors discussed the tradeoffs inherent in iterative amortization (e.g., how is performance improvement balanced by increased computational expense?). Additionally, iterative amortization appears to help on some tasks and not at all on others. What properties of an environment make iterative amortization more or less useful? How do these properties vary among the tasks used in the paper? These are important questions from both a practical and theoretical standpoint. The analysis of the contribution of multimodality to performance is likely a good first step in this direction. There are also a number of supporting results in the appendix. Minor: For eq. 2, should there be a $\rho(s_1)$ prefixed to the righthand side of the equation? And shouldn't the upper limit of the products in both eqs. 1 and 2 be $T-1$? 

Clarity: The paper is mostly clear and well-organized. The background section and discussion of value overestimation were particularly enjoyable to read. Minor: Figure 7b is difficult to parse--I'm unclear as to what takeaway is intended. It also would be helpful to mark the exact section of the appendix in which supplementary results appear. 

Significance: The magnitude of the performance gain on the majority of Mujoco tasks is not significant. It's difficult to gauge the significance of reducing the amortization gap--why this gap appear to grow over the course of training (Figure 8, right)?  While the raw performance improvement due to the proposed approach is not generally notable, I think the main significance of this work is captured by results like the ability to flexibly adapt to new objectives, such as to model-based value estimation. Namely, its significance lies in its novelty and the potential for follow-up work. Some theoretical analysis would be helpful in developing a deeper understanding. 



Given the above factors, I recommend acceptance. 
limitations_and_societal_impact: See the main review for a discussion of limitations and suggestions for improvement. I do not believe there are obvious societal impacts for this work.
review_title: null

Meta-review: All the reviewers think introduce the advanced variational inference for making the policy flexible by exploiting the connection between RL and variational inference is interesting. 

Although I recommend the acceptance for this submission based on the reviewers feedback, there are several issues should be extensively discussed:

- EBM closed-form for policy parametrization: in fact, with entropy-regularization, the policy will lie in energy-based model. Therefore, a natural competitor is advanced sampling algorithms, e.g., Langevin, HMC, SVGD [1], Wasserstein flow [2] etc, based on learned EBM. 

- Performance tradeoff in computation, sample complexity, vs flexibility:  as the complicated parametrization introduced, the computation cost and sample complexity will increase. This should be carefully discussed, as almost all reviewers pointed out. From the empirical study, it seems the flexible policy family does not provide significant benefits, which diminishes the significance of this paper. 

In sum, I think this paper is interesting and could be better if the authors carefully address these questions. 
 
[1] Haarnoja, Tuomas, Haoran Tang, Pieter Abbeel, and Sergey Levine. "Reinforcement learning with deep energy-based policies." In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
[2] Zhang, Ruiyi, Changyou Chen, Chunyuan Li, and Lawrence Carin. "Policy optimization as wasserstein gradient flows." In International Conference on Machine Learning, pp. 5737-5746. PMLR, 2018.
Decision: Accept (Poster)