Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper proposes a method to learn how to optimize policy under the framework of Mirror Learning. Based on the learning result, they formulate a closed-form policy optimization method for learning the policy.
strengths_and_weaknesses: The idea of this paper is interesting. They propose a method that lies between human handcraft and learning discovery. The experimental result is interesting.
I'm not satisfied with the claim and some experimental results (see below for detail).

The author claim that TRPO and PPO are handcrafted by humans rather than discovered by learning. This is overclaimed to me. 
- The key idea of this paper also follows Mirror Learning, which is also discovered by humans.
- Furthermore, learning drift functions from scratch underperforms with respect to LPO. Instead, they learn with the PPO initialization, which is more handcrafted.

I'm not satisfied with the experimental result.
- As the authors claim that discovered by an algorithm is better, and there exits several variants following Mirror Learning, e.g., TRPO and variants of PPO. It's straightforward to compare the learned methods with these mirror-learning-based methods.
- The experimental results of learning drift functions from scratch underperform LPO and even PPO. Although the authors have also tried learning from PPO initialization, this implies it involves more human handcraft. 

questions: Learning discovery is interesting, and using the fixed learning mechanism is natural. Could that be true if the optimal learning mechanism is different across different environments?

Can you explain why you use the candidate arguments in the equation in Line 180? Are there any other candidates?
limitations:
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the learning of a “drift function” (a proximal regularization term in policy optimization) for reinforcement learning (RL). By parameterizing the drift function as a neural network and framing the objective as the expected return after K iterations of RL, the authors can find this drift function through black-box optimization. The authors analyze the structure of the learned drift function and compare it to PPO’s drift function. Inspired by insights on the learned function, the authors also present an analytic drift function that contains many of the same properties as the learned function. Experiments show that policy optimization with both novel regularizers regularly outperform PPO on several Brax continuous control problems.
strengths_and_weaknesses: The paper studies a highly practical problem in the design of surrogate objectives in policy optimization. The proposed approach is sensible, and the results do appear promising. The writing and presentation are also very clear, and it’s easy to read the paper. I especially like the analysis of the LPO function in Section 6, and the insights to its structure (rollback and cautious optimism) make sense for its role in policy optimization.

I think one major weakness of their work is that the evaluation environments are limited. All environments are continuous control problems in the Brax simulator. To assess how well the LPO/DPO drift function generalizes, I highly suggest evaluating the functions in other physics simulators (e.g., MuJoCo and Isaac) and entirely different domains (e.g., Atari and other discrete problems).

questions: What happens if you try to learn a drift function without the input $x$ from line 179? Does this drift function also give good RL performance?

Some other comments:\
Line 137: Is it supposed to be $\pi_k$ instead of $\bar\pi$?\
I would suggest moving Sec. 5.1 (Learning drift functions from scratch) to the Appendix since it performs worse than LPO and is not a significant part of the paper.\
Line 177: The variable $r$ is already used for the reward. It may make sense to use a different variable.\
Despite what the paper says, I did not find any code in the supplementary.

limitations: The authors acknowledge that they only study the drift function and hope to expand optimization to other parts of the mirror learning objective.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper is based on the concept of Mirror Learning which includes several RL algorithms like PPO, TRPO, MDPO, etc. The work empirically exploits the flexibility of choosing the "drift" function in the Mirror Learning domain by using meta-learning. Given a neural network drift function, the authors use Evolutionary Strategies to meta-learn a single drift function (generalizable across environments) where the objective is to maximize the return of the policy trained using an update rule composed of the drift function. This is termed the Learnt Policy Optimization (LPO) which is proposed in two variants: one uses PPO clip augmentation at the last hidden layer output and the other doesn't. Eventually, the authors draw meaningful observations from LPO's analysis and proposed Discovered Policy Optimization (DPO) using those observations. The authors verify the approach by showing that DPO's behavior is close to LPO's behavior. All experiments are conducted in brax-based environments.
strengths_and_weaknesses: The proposed method holds some strength by improving over a widely-used baseline using:
- parametrized drift functions to capture the importance of action distribution ratio and advantage in policy improvement.
- empirically observed and properly justified learned behavior based on the advantage being positive or negative.
- well-curated and mechanically optimized/fine-tuned algorithm (DPO) using the discovered drift function model.

However, there are the following weaknesses in the proposed framework:
- there are very less ablations on the choice of structure, for example, the non-linear formulation of $x_{r,A}$ and choice of constants for DPO (overall Appendix E is missing)
- there is no justification for why an initial drift function of PPO helps. While it is acknowledged that LPO-Zero underperforms ("possibly" due to less training), there is no evidence that much more training can help LPO-Zero converge to LPO.
- the experimental validation is not sufficient to prove the improvement claimed by the contribution. (there are no additional results in Appendix D as mentioned in the paper)

questions: It would be nice if the following can be justified and included in the paper:
- Since the overall objective for policy improvement aims to minimize $\mathbb{E}_{\pi_k}\[\mathcal{D}(\pi|s)\]$, I am curious to know where $\tilde{f}_\phi$ converges to and how the final drift function $f_\phi$ is different from the PPO one. This also points towards another interesting phenomenon of DPO (with constants) being an approximated version of PPO with smoothened contours (instead of strict clipping).
- If there is hyperparameter optimization of PPO (just clip factor) with evolutionary strategies to get the best performance (return) for a particular environment, what does the DPO performance comparison look like? 
- Can the authors add the corresponding heat map of PPO as well in Figure 5?
- The discovered drift function seems to be cherry-picked. It will be beneficial if the authors add justifications for their choice of the input vector to the drift function during LPO optimization and also in DPO. Why not $(r-1)^n$ in DPO as that would also satisfy the feasibility conditions? 
- For a single environment, can the authors show the variation of the final performance as a function of constants $\alpha$ and $\beta$ ?
- There should be more experiments on multiple benchmarks to justify the claim since this is an improvement of an algorithm. Can you add some analysis of Procgen experiments?
- Can you add a properly referenced appendix? The references in the paper do not correspond properly to some Appendix sections making it difficult to analyze.
- Also, add equation numbering.
limitations: The authors have acknowledged that the method shown in the presented work is a step towards discovering more robust and high-performing RL algorithms. In general, the work does not has significant limitations but lacks proper justification of choices which makes it unclear how to conduct algorithm discovery in a proper manner. Further, more empirical evidence and experiments are required to complete the contribution.
review_title: null

Meta-review: metareview: This paper proposes a policy-search approach in which a "drift" function is meta-learned in the framework of Mirror Learning.
After reading each other's reviews and the authors' feedback, the reviewers have solved most of their concerns and agree that this paper deserves publication.
The authors need to consider the reviewers' suggestions in preparing the final version of their paper.
confidence: Less certain
Decision: Accept