Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary:
Inspired by recent theoretical analysis of TRPO and PPO that use mirror descent, this paper proposes two new algorithms that directly minimize a mirror descent objective by taking multiple gradient steps. While similar to TRPO and PPO, MDPO is different in important ways, and happens to perform better in practice. The authors also propose a similar off-policy version of MDPO that happens to be closely related to SAC while also outperforming SAC experimentally. 



main_review: 
Main Review:
Overall I think this could be a good paper. I am really interested in the connections between PPO, TRPO and SAC and MDPO seems to be a more general version of all three of them. Experimentally the results are very sound and seem to outperform existing methods while also being a simpler algorithm with fewer hyperparameters. I could see this method becoming very popular as a standard RL baseline. 

With that being said, I think the off-policy section needs some work. Equation 11 is described in the appendix but it is not described when it is introduced. Can you include a description of how this equation works? At least give a reference to the appendix please. In general I thought the entire paragraph starting with “the main idea in algorithm 2…” was poorly written and confusing. Maybe this should be expanded into a couple of paragraphs. Can you explicitly write out the soft version of off-policy MDPO instead of just describing it in the text? Maybe include a new algorithm? “If we write the definition of KL and use the reparameterization trick in(16), we will rederive the loss function(11)used by our off-policy MDPO algorithm” I don’t think this is true, wouldn’t it be the same as (16) but without the current policy term? Maybe you can write out this equation. 

Also, the related works could be explained better. The authors did a great job describing the differences between MDPO and TRPO/PPO/SAC, but there are many other related methods out there that should be referenced and talked about, for example Trust-PCL.  

Minor points:
> “Since finding an optimal policy for an MDP involves solving a non-linear system of equations and the optimal policy may be deterministic (less explorative),”
 
Not sure if this explains why one might use an entropy regularizer. 

> “Finally, we ran a similar experiment for TRPO which shows that performing multiple gradient stepsat each iteration of TRPO does not lead to any improvement. In fact our results showed that in somecases, it even leads to worse performance than when performing a single-step update”

Where are these results? Can you reference them in the text? 

Could you put a vertical line in the results to separate on policy and off policy algorithms? 
I was surprised to see MDPO not learn at all in the freeway atari experiment. Could you add some more commentary on the atari experiments? PPO seems to do much better against MDPO than the mujoco experiments. 
Could you add hyperlinks to references and figures/algorithms?
summary_of_the_review: I really like this paper but think the off-policy section can be re-written to be more coherent.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work proposes an algorithm like TRPO and PPO inspired by Mirror Descent. The contribution of the paper is primarily algorithmic and also empirical; there is no supporting theory for the proposed algorithm but I believe this is fine for this type of paper.

main_review: The paper proposed an algorithm inspired by Mirror descent, much like TRPO and PPO, both of which are well known and used.
The method appears to work convincingly better than the state of the art in the proposed experiments.
For the on policy setting, an interesting difference with TRPO is the way the trust region subproblem is solved.
A great strength of this work is that it nicely connects with the available literature, clearly summarizing the state of the art as well as the algorithmic differences. A reference that could provide some value is 'On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift' by Agarwal et al. 
summary_of_the_review: I support the paper for publication. I come from more from the theory community, and therefore ask the metareviewer to discount my ( unfortunately brief) review accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on reinforcement learning with the tabular Markov decision process setting and proposes the mirror descent policy optimization for off-policy and on-policy situations. Furthermore, the experiment result shows that this new algorithm outperforms the previous algorithm in both off-policy and on-policy situations.
main_review: 1. The paper is well-written and easy to follow. More specifically, this work compared the novel algorithm with previous work (PPO, TRPO, SAC) carefully and clarified the experiment setting precisely.

2. The experiment parts contain some continuous control tasks, and this result highly supports the efficiency of the proposed algorithm compared with previous work (PPO, TRPO, SAC).

3. However, I still have some concerns about this work. It seems that this paper this work is a combination of mirror gradient descent with the previous TRPO or PPO structure, and the technical contribution is limited. Furthermore, the author compares the TRPO or PPO  algorithm, and it seems that none of the differences will improve the experiments. So it looks better if the author can make more discussion of the intuition compared with previous work.
summary_of_the_review: Based on the previous comment, it is marginally above the acceptance threshold due to the excellent experiment result.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper connects the optimization method, mirror descent, to the study of the policy optimization method. Based on the mirror descent principle, the paper proposes the MDPO algorithm, which updates the policy via approximately solving a trust-region problem. The paper proposes the on-policy and off-policy variants of MDPO. Furthermore, the paper connects the on-policy MDPO to PPO and TRPO and connects the off-policy MDPO to SAC. The contribution of the paper is to provide a unified viewpoint of several RL algorithms and shows that   MDPO performs equally or better than TRPO, PPO, and SAC in different tasks.

main_review: Strengths:
- The paper is well written, and the motivation is easy to follow. 
- Going beyond the tabular RL, the paper derives a scalable algorithm from the MD principle. Moreover, the algorithm is practical since it allows for function approximation.
- The result is well supported empirically over several different tasks, which shows the advantage of MDPO.

Weakness:
- The paper does not provide any theoretical analysis for the algorithm. While the convergence and optimality of mirror descent are well-known for the convex constrained problems, it is still not sure if reinforcement learning lies in such a framework.
- My primary concern is about the novelty of the paper. The connection between PPO, TRPO, and mirror descent is well-known (see, e.g. [11, 19, 20, 24, 28] and [A]). Extending the tabular setting to the function approximation setting is also well studied in [19, 26. 27] both theoretically and empirically. 

[A] Zhan, Wenhao, et al. "Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence." arXiv preprint arXiv:2105.11066 (2021).

summary_of_the_review: Overall, the paper is well written, and the empirical study well supports the result. However, the paper lacks the theoretical analysis, and the novelty of deriving MDPO from MD is limited. Given that, I recommend rejection for the paper.
review_title: null

Meta-review: This paper proposes and studies a variant of policy optimization---mirror descent policy optimization (MDPO)---which was inspired by the mirror descent algorithm in the optimization literature. The proposed algorithm attempts to find a policy parameter that maximizes the expected regularized advantage function,  where the regularization term is based on the KL divergence between the new policy iterate and the current policy iterate. The main contributions are algorithmic and empirical, with detailed discussions provided to illuminate the connection between MDPO and other existing policy optimization paradigms like TRPO, PPO, etc. The paper provides an interesting and useful contribution to the growing literature of policy optimization.
Decision: Accept (Poster)