Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper draws an interesting connection to variational inference, categorizing current policy optimization methods with KL regularization as direct amortized optimizers. The paper shows how direct amortized policy optimization can be suboptimal and proposes a new class of method called iterative amortized policy optimization. This ‘iterative’ amortized policy optimization performs iterative optimization before the regular optimization step, offering several advantages: better at reaching the optima, better at sampling multiple modes, and support more rapidly changing policy. The paper shows nice visualization to support these claims and also run benchmark experiments to show its improvement.

This paper has done a good job introducing ‘iterative amortized policy optimization’, providing support with both good visualizations at a simpler level (to showcase what it is actually doing) and empirical evaluation on complex benchmarks. I think the contribution is novel and introduces a different type of optimization that is shown to improve current policy optimization methods. 
Despite the paper being an overall good paper, I think the experiments should be much better supported and recommend a borderline reject unless the issues below can be addressed.

The results look convincing overall but I think more runs should be conducted. Experiments on benchmarks only conduct 4 runs. There is high variance in results due to random seeds especially in Mujoco domains. Henderson et al. 2018 (https://arxiv.org/pdf/1709.06560.pdf) show that two different groups of runs with 5 random seeds each can lead to significant difference in performance on the same algorithm. It is hard to say that iterative amortized optimization helped improve performance with only 4 runs. 

Furthermore, it is not mentioned at all in the main paper that for HalfCheetah and Walker2D, iterative vs direct use different model architectures (it is mentioned in Appendix A.4 Q-value section). 

These two environments in particular were environments that showed significant difference between the two methods, and makes the reader wonder whether this improvement was due to optimization method difference or architectural change.
I think at least a justification in this particular design choice is needed or perhaps all results for both models can be shown to help readers understand the general performance.

Lastly, it would also be helpful to mention how many runs were conducted for model-based value estimate experiment in Figure 8a. 

Other minor details:
I thought it would be helpful to mention where experiment details can be found in the appendix for each figures or sections in the main paper. 

At the bottom of p.4 last paragraph of section 3.1, should \nabla_\phi J be \nabla_\lambda J?
review_title: Interesting work and connection on iterative amortized optimizers

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Contributions**: The authors propose to use iterative amortization for policy optimization to help reduce suboptimality in policy optimization. They find that like in variational inference for generative modeling, iterative amortization is able cover multiple modes of distributions and lower the amortization gap, and show slightly improved performance on mujoco gym benchmarks. Additionally, they find that the iteratively amortized policy is able to better exploit the learned Q function and lead to more overestimation bias, and address this by tuning a parameter to make the Q-function backups more pessimistic to compensate.

**Questions and Concerns**: It is not entirely clear to me where the better performance observed from using iterative amortization derives from in the benchmark results. For example, in Hopper and HalfCheetah, we see that the iterative amortization slightly outperforms the direct baseline, yet the the difference in amortization gaps seems negligible throughout training. On the other hand, in the Ant environment, the amortization gap of the direct method is much higher, yet the learning curves of both methods are practically identical. One potential issue is that the objective of the policy update depends on the *learned* Q-function instead of the true returns from the environment, and so a better inference procedure may not even necessarily be a benefit. For this reason, we often consider schemes like trust regions for policy updates that explicitly limit policy optimization.

Some additional questions:
To what extent is the amortization gap alleviated by direct amortization but larger, more expressive networks? Does doing so provide similar improvements in performance?

In Figure 3b, it appears that with the iterative schemes, the new policy consistently differs the old one (what exactly is pi_old in this case?) by a significant amount. Should we be concerned here that the learned policy (and presumably also the Q-functions) are not really coming close to converging? This further suggests to me strictly trying to have the policy better approximate this noisy oscillating Q may not actually be beneficial behavior.

**Summary**: The authors demonstrate that iterative amortization enables some improvement to policy optimization in a maximum entropy RL algorithm. However, it is not entirely well understood to me how precisely iterative amortization leads to such benefits. Regarding novelty, the presented method is a straightforward application of a technique from variational inference to maxent RL, so is perhaps a bit incremental in that sense. Overall, I like the method since it provides a very simple way to improve learning by plugging in iterative amortization, but think it needs a to provide a better understanding of why it helps.
review_title: Simple method for improving maxent policy optimization

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary

The paper proposes to replace the actor/policy network with an iterative version to encode the action distribution parameters, which is inspired by prior work on iterative amortized optimization. This scheme generates the action distribution parameters for each state at the end of an inner loop which takes the objective gradient wrt to the parameters also as input (as opposed to a regular actor network which does one forward pass). This seems similar to prior work on learning to learn/meta learning but the presentation can be improved (more detailed comments below).  Experimental evidence is given for benefits over SAC in mujoco environments.

Overall, the general idea seems interesting and useful, but the novelty is unclear given the prior work; the clarity of the presentation can be improved which makes an accurate assessment of the significance harder. 

### Detailed Comments

* In the introduction, authors motivate the idea of amortization by making a distinction between optimizing the distribution versus a network whose parameters output the distribution. Can the authors provide an example of an algorithm that does the former for clarity (i.e. an example of a non-amortized policy -- what would that look like hypothetically?) 

* The agent's policy is denoted by $p_\theta$ (prior to Eq (1)) and then with $\pi$ later. Comment below Eq (8) makes a reference to $\theta$ as prior parameters. This presentation is unclear and needs additional information to disambiguate. 

* What do the dependencies in Figure 1 right denote? For instance, $p_\theta$ has a dependence on $p_{env}$ in this figure, but it's not clear what that means.

* How is $\lambda$ initialized in the inner update loop? The capability to find multiple modes seems dependent upon providing diverse initializations, which seems unspecified. 

* The value overestimation section discussion seems to suggest that the iterative scheme does. worse than the direct even with changing $\beta$. Not clear what the take away from this section is in the context of the rest of the paper.




review_title: A learning-to-learn style proposal for policy optimization

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
The paper studies the popular soft actor-critic (SAC) algorithm, and points out that it does not find the most optimal actor (action distribution) that maximizes the critic (Q value). The reason is that SAC uses “direct amortization” (a feed-forward network from state to Gaussian params for the action distribution), and hopes that simple gradient descent (eqn (13) in the SAC paper) can sufficiently minimize loss (eqn (10) in SAC) over time, which is not true as Fig 6 in this paper suggests. 

To solve this issue, the paper suggests an “iterative amortization” method that optimizes the Gaussian params through a few iterations of 1st order optimization (Alg 2). The optimization is done implicitly with a neuralnet that takes the policy gradient as input (eqn (10)). The resulting algorithm reduces the amortization gap (Fig 6) and improves policy performance over time (Fig 5).

Recommendation
Overall, I’m slightly inclined to accept this paper. The amortization gap is an interesting and often ignored phenomenon. The paper offers a simple solution and performs convincing experimental study. However, the presentation is very confusing, as the focus is selling this A + B solution (A = iterative amortization, B = SAC), rather than analyzing the “amortization gap” issue in full detail. Consequently, the experiments are not comprehensive enough. 

Strengths
1. The paper reveals that the SAC policy isn’t always fully optimized. It’s common to assume that gradient descent + large enough network + enough data will eventually optimize the policy. But as Fig 6 suggests, it might not be the case.
2. The paper suggests a simple solution through iterative amortization. This plug-and-play trick outperforms vanilla SAC on two Mujoco tasks (Fig 5).


Weaknesses
1. The paper stops at selling a solution instead of going beyond to analyze the amortization gap. Some natural questions:
(a) There are multiple ways to fully optimize the policy, such as using a larger SGD coeff and running more SGD steps. Do they work?
(b) If the amortization gap is an issue, do you see a problem in the expressivity of Gaussian? Does a more general form of policy (e.g. in the original SQL paper) further reduce the gap and improve performance? 
(c) In the case of discrete actions, where the optimal policy can be found directly (simple softmax), does the fully optimized policy outperform direct optimization? Any relevant papers?

2. The presentation is very confusing.
(a) The motivation reads like “Here’s A and here’s B. Let’s try A + B”. If the paper pointed out SAC’s amortization gap earlier and how it seriously impacts performance, the paper would be easier to understand.

(b) Confusing terminology.
- p_{\theta}: Initially described as the agent’s “parametric distribution”, later called “policy prior parameters”, then finally declared a uniform distribution.
- f_{\phi}: Should not use the same symbol for both direct and iterative amortization.
- “policy distribution parameters”: Easily confused with “policy network parameters”. Better called “action distribution parameters”.
- “are policy networks providing fully-optimized policy estimates”: Estimates could mean the Q value. Better called “policy objectives”.

(c) Lack of experimental rigor.
- In Fig 1 and 2, which state and which training iteration is being studied? Why should the reader trust the plots?
- Fig 6 shows that the amortization gap in Ant-v2 does not hurt SAC’s performance. Could you explain why?

Other feedbacks
I put in a lot of effort in understanding this paper and re-summarizing it in this review. I’d hope the authors understand my confusion and restructure the paper correspondingly. 

review_title: OK submission, but confusing and lacking depth

Meta-review: I think there is a lot to commend in this paper: the general approach for training f_phi in this way is creative and interesting, the discussion of the amortization gap is thought-provoking, and the general idea is not something that I have seen in the literature before. That said, the reviewers raise a number of important concerns about the approach, chief of which is that the paper's explanation for why the method works is questionable. The proposed method can be viewed as simply a more expressive policy architecture. In fact, I suspect strongly that the modest increase in performance is likely explained by this alone. The discussion with Reviewer 2 in particular makes this issue very clear. I don't think the authors offered a very compelling response to this. Therefore, I think there are just too many question marks about this approach to accept the paper for publication at this time.

I do however think that this line of research is very promising, and I would encourage the authors to continue this work and flesh out the evaluation to be more rigorous and complete, understand whether the increase in performance comes down simply to increased expressivity or if the discussion of amortization is closer to reality, and also address other concerns raised by R2 and the other reviewers.
Decision: Reject