Mirror Descent Policy Optimization
Authors: Manan Tomar
Lior Shani
Yonathan Efroni
Mohammad Ghavamzadeh
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement Learning
Policy Optimization

Abstract: Mirror descent (MD), a well-known first-order method in constrained convex optimization, has recently been shown as an important tool to analyze trust-region algorithms in reinforcement learning (RL). However, there remains a considerable gap between such theoretically analyzed algorithms and the ones used in practice. Inspired by this, we propose an efficient RL algorithm, called {\em mirror descent policy optimization} (MDPO). MDPO iteratively updates the policy by {\em approximately} solving a trust-region problem, whose objective function consists of two terms: a linearization of the standard RL objective and a proximity term that restricts two consecutive policies to be close to each other. Each update performs this approximation by taking multiple gradient steps on this objective function. We derive {\em on-policy} and {\em off-policy} variants of MDPO, while emphasizing important design choices motivated by the existing theory of MD in RL. We highlight the connections between on-policy MDPO and two popular trust-region RL algorithms: TRPO and PPO, and show that explicitly enforcing the trust-region constraint is in fact {\em not} a necessity for high performance gains in TRPO. We then show how the popular soft actor-critic (SAC) algorithm can be derived by slight modifications of off-policy MDPO. Overall, MDPO is derived from the MD principles, offers a unified approach to viewing a number of popular RL algorithms, and performs better than or on-par with TRPO, PPO, and SAC in a number of continuous and discrete control tasks.

Published as a conference paper at ICLR 2022
MIRROR DESCENT POLICY OPTIMIZATION
Manan Tomar"
Lior Shani Technion, Israel s shanl gm i com
Yonathan Efroni
University of Alberta, Amii manan. toma r @mmail com
Microsoft Research NYC ye f Froni @mi crosoft.com
Mohammad Ghavamzadch
Google Research
ghavamz a8 google. ghavamza@google.coomm
com
ABSTRACT
Mirror descent (MD), a well-known first-order method in constrained convex optimization, has recently been shown as important tool to analyze trust-region algorithms in reinforcement learning (RL). However, there remains a considerable gap between such theoretically analyzed algorithms and the ones used in practice. Inspired by this, we propose an efficient RL algorithm, called mirror descent policy optimization (MDPO). MDPO iteratively updates the policy by approximately solving trust-region problem, whose objective function consists of two terms: a linearization of the standard RI. objective and a proximity term that restricts two consecutive policies to be close to each other. Each update performs this approximation by taking multiple gradient steps on this objective function. We derive on on-policy and off-policy variants of MDPO, while emphasizing important design choices motivated by the existing theory of MD in RL. We highlight the connections between on- on-policy MDPO and two popular trust-region RL algorithms: TRPO and PPO, and show that explicitly enforcing the trust-region constraint in fact not necessity for high performance gains in TRPO. We then show how the popular soft actor-eritic (SAC) algorithm can be derived by slight modifications of off- MDPO. Overall, MDPO derived from the MD principles, offers a unified approach to viewing number of popular RL algorithms, and performs better than or on-par with TRPO, PPO, and SAC in a number of continuous and discrete control tasks.
INTRODUCTION
An important class of RI. algorithms consider an additional objective in their policy optimization that aims at constraining the consecutive policies to remain close to each other. These algorithms referred to as trust region or proximity-based, resonating the fact that they make the new policy to lie within a trust-region around the old one. This class includes the theoretically grounded conservative policy iteration (CPI) algorithm [15], as well as the state-of-the-arr deep RL algorithms, such as trust-region policy optimization (TRPO) [26] and proximal policy optimization (PPO) [28]. The main difference between these algorithms is in the way that they enforce the trust-region constraint. TRPO enforces explicitly through a line-search procedure that ensures the new policy is selected such that its KL-ddivergence with the old policy is below a certain threshold. PPO takes more relaxed approach and updates its policies by solving an unconstrained optimization problem in which the ratio of the new to old policies is clipped to remain bounded. It has been shown that this procedure does not prevent the policy ratios to go out of bound, and only reduces its probability [31, 9].
Mirror descent (MD) [6, 4] is first-order optimization method for solving constrained convex problems. Although MD is theoretically well- well-understood in optimization [3, 14], only recently, has been investigated for policy optimization in RL [25, 12, 20, 29, 1]. Despite the progress made by these results in establishing connections between MD and trust-region policy optimization, there are still considerable gaps between the trust-region RL algorithms that have been theoretically analyzed in their tabular form [29] and those that are used in practice, such as TRPO and PPO.
In this paper, motivated by the theory of MD in tabular RL, our goal is to derive scaleable and practical RL algorithms from the MD principles, and to use the MD theory to better understand and
Work done partially FAIR as part of AT Residency.

---

Published as a conference paper at ICLR 2022
explain the popular trust-region policy optimization methods. Going beyond the tabular case, when the policy belongs to parametric class, the trust-region problems for policy update in RL cannot be solved in form. We propose an algorithm, called mirror descent policy optimization (MDPO), that addresses this issue by approximately solving these trust-region problems via taking multiple gradient steps on their objective functions. We derive on-policy and off-policy variants of MDPO (Section 4). We highlight the connection between on on-policy MDPO and TRPO and PPO (Section 4.1), and empirically compare it against these algorithms on several continuous control tasks from OpenAI Gym [7] (Section 5.3). We then show that if we define the trust-region w.r.t. the uniform policy, instead of the old one, our off- oeflipllcy MDPO coincides with the popular soft actor-critic (SAC) algorithm [13]. We discuss this connection in detail (Section 4.2) and empirically compare these algorithms using the same set of continuous control problems (Section 5.4).
Our observations on the comparison between the MDPO algorithms and TRPO, PPO, and SAC are a result of extensive empirical studies on different versions of these algorithms (Sec- tion 5 and Appendices E and F). In particular, we first compare the vanilla versions of these algorithms in order better understand how the core of these methods work relative to each other. We then add number of code-level optimiza- tion techniques derived from the code-bases of TRPO, PPO, and SAC to these algorithms to compare their best form (those that obtain the best results reported in the literature) against each other, while also evaluating MDPO with PPO on 21 Atari games. We address the com- mon belief within the community that explicitly enforcing the trust-region constraint is a necessity for good performance in TRPO, by showing that MDPO, trust-region method based the MD principles, does not require enforcing a hard constraint and achieves strong performance by solely solving an unconstrained problem. We address another common belief that PPO is better performing algorithm than TRPO. By reporting results of both the vanilla version and the version loaded with code-level optimization techniques for all algorithms, we show that in both cases, TRPO consistently outperforms PPO. This is in line with some of the findings from a recent study on PPO and TRPO [9]. Finally, we provide an optimization perspective for SAC, instead of its initial motivation as an hatropy-regularized (soft) approximate dynamic programming algorithm. Through comprehensive experiments, we show that on-policy and off-policy MDPO achieve state-of-the-an performance across a number of benehmark tasks, and can be excellent alternatives to popular policy optimization algorithms, such TRPO, PPO, and SAC.
Clock Time
Performance
Num Hyperparams
1.00
2.0
1.0
0.75
1.5
0.50
1.0
0.5
0.25
0.5
MOPO PPO
TRPO
MDPO
PPO
TRPO
MDPO
PPO TRPO
Figure 1: Overall Comparison. Between MDPO, PPO, and TRPO, MDPO provides the best trade-off in terms of best average performance, less (normalized) wall clock times, and least number of algorithm specific hyper pa- rameters used.
PRELIMINARIES
In this paper, we assume that the agent's interaction with the enviromment is modeled as didsccuute Markov decision process (MDP), denoted by M M= (S,A, P, R, 7, 14), where S and A are the state and action spaces; P P(s' (s, a) is the transition kernel; R r(s,a) a) is the reward function; E (0, 1) is the discount factor; and μ is the initial state distribution. Let x S + A be stationary Markovian policy, where A is the set of probability distributions on A. The discounted frequency of
visiting state by following policy π is defined as (s()) = (1 (1 (20 I'I(s, s}|,, sr. The value function of policy π at state
S is defined as (T((s E E E120 ('r'", a)) so s, Λ]. Similarly, the nction-value function of π is defined as Q*((s a) = E E "r((s, a;)ss = s, an a, Φ]. The difference between the action-value Q and value V functions is referred to as the advantage function A (s a) Q* a) (s)..
Since finding an optimal policy for an MDP involves solving a non-linear system of equations and the optimal policy may be deterministic (less explorative), many researchers have proposed to add a regularizer in the form of an entropy term to the reward function, and then solve the entropy- regularized (or soft) MDP (e.g.. [16, 30, 25]). In this formulation, the reward function is modified as Tx(s, a) = r(s, a) + (+ where À is the regularization parameter and is an entropy-related term, such as Shannon entropy [10, 23], Tsallis entropy (17, 24], or relative entropy [2, 22]. Setting
2

---

Published as a conference paper at ICLR 2022
À 0, return to the original formulation, also referred to the hand MDP. In what follows, use the terms 'regularized" and 'soft' interchangeably.
2.1 MIRROR DESCENT IN CONVEX OPTIMIZATION
Mirror Descent (MD) [4] is a first-order trust-region optimization method for solving constrained convex problems, i.e., x² E arg minzee (xx), where is a convex function and the constraint set C is convex compact. In each iteration, MD minimizes a sum of two terms: 1) a linear approximation of the objective function distance between the updated Tą-1 and current 23 estimates. MD is considered a trust-region method, since the proximity term keeps the updates Fyk and x++1 close each other. We may write the MI) update as
the previous estimate The and 2) a proximity term that measures the
1
3k+1 arg migmin (Vi(xz), - xR) (2, LR
(1)
#EC
where (2, == (b2)) (xxx) (V 2x) the Bregman divergence associated with strongly convex potential function v. and l. is step-size determined by the MD analysis. When db = 111111 the Bergman divergence the Euclidean distance By (x, = || - x2113. and (1) becomes the projected gradient descent algorithm [3]. When vi is the negative Shannon entropy, the
Bregman divergence term takes the form of the KL divergence, i.c., BB x4) = KL(x, 22). In
this case, when the constraint set C' is the unit simplex, C = Ax, MD becomes the exponentiated gradient descent algorithm and (1) has the following closed form [4]:
2t. exp - 4 (=)))))
Σラ 2 exp
tu V,((m))
where o and vif are the 1hh coordinates 2g and f.
MIRROR DESCENT IN RL
The goal in RL to find optimal policy n². Two common notions of optimality, and result, two distinct ways to formulate RL as an optimization problem are as follows:
(a) n (-|s) E max (s), Vs E S,
(b) π E arg max E E
(3)
In (3a), the value function is optimized over the entire state space S. This formulation is mainly used in value function based RL algorithms. On the other hand, the formulation in (3b) is more common in policy optimization, where scalar that is the value function at the initial state (s (1) optimized.
Unlike the MD optimization problem, the objective function is not convex in in either of the above two RL optimization problems. Despite this issue, [12] and [29] have shown that we can still use the general MD update rule (1) and derive MD-style RL. algorithms with the update rules
1 (I(
(-18) +- arg max BnN (s, ts ai]
As
(4)
#Ell
Tk+1 arg max Espepng TEIl
BaNT [ATH
1 (l mx)
1
(5)
t
for the optimization problems (3a) and (3b), respectively. Note that while in (4), the policy is optimized uniformly over the state space S, in (5), it is optimized over the measure Pag. i.c. the state frequency induced by the current policy Tkk
4 MIRROR DESCENT POLICY OPTIMIZATION
In this section, we derive on on-policy and off-policy RL algorithms based on the MD-style update rules (4) and (5). We refer to our algorithms as mirror descent policy optimization (MDPO). Since the trust-region optimization problems in the update rules (4) and (5) cannot be solved in closed-form, we approximate these updates with multiple steps of stochastic gradient descent (SGD) on the objective functions of these optimization problems. In our on-policy MDPO algorithm, described in Section 4.1, we use the update rule (5) and compute the SGD updates using the Monte-Care (MC) estimate of the advantage function Th gathered by following the current policy т&- On the other hand,
3

---

Published as a conference paper at ICLR 2022
off-policy MDPO algorithm, described in Section 4.2, is based on the update rule (4) and calculates the SGD update by estimating 4 using samples from a replay buffer.
define the policy space, II, as class of smoothly parameterized stochastic polices, i..., Il = (=(-|s; : E S,0 E }). We refer to 0 as the policy parameter. We will use and 0 to represent policy, and IT and @ to represent the policy space, interchangeably.
In MDPO algorithms,
4.1 ON-POLICY MDPO
In this section, we derive an on-policy RL algorithm based on the MD-based update rule (5), whose pseudo-code is shown in Algorithm in Appendix A. We refer to this algorithm as on-policy MDPO. We may write the update rule (5) for the policy space & (defined above) as
0 arg argmax max (1, where (r Es-Pen [Ea-ry A a)] dee
KL ne. mor. (6)
Each policy update in (6) requires solving a constrained (over Q) optimization problem. In on-policy MDPO, instead of solving this problem, we update the policy by performing multiple SGD steps on the objective function (0, 0z). Interestingly, performing only a single SGD step on (r 02) is not sufficient as VoKL(-; To. To. |0=0, 0, and thus, if we perform single-step SGD, i.e.,
Vo (0, 10=0
Eumpa ar-Ty
Voo To, (als) (s,a))
the resulting algorithm would be equivalent vanilla policy gradient and misses the entire purpose of enforcing the trust-region constraint. As result, the policy update at each iteration k of on-policy MDPO involves m SGD steps as
m = Ok: for i 0
(i+11) + 0 (0, L
ak+1 mp(²)
where the gradient of the objective function
(o, 0k) 6= = BB
We
v log o (sls)
Esmpoy ta
VoKL(s To, Mo=of (7)
mex
can be estimated in an on-policy fashion using the data generated by the current policy Té. Since in practice, the policy space is often selected as Gaussian, we use the closed-form estimation. Our on-policy MDPO algorithm (Algorithm 1, Appendix A) has close connections to two popular on- on-policy trust-region RL algorithms: TRPO [26] and PPO [28]. We now discuss the similarities and differences between on-policy MDPO and these algorithms.
KL in this
Comparison with TRPO. At each iteration k; TRPO considers the constrained optimization problem
(a(a))) A (s, (s. Top (als)
E Sompay anTt
s.t.
E Egcpp,
[LL(s; To, no)] 8,
PEB
and updates its policy parameter by taking step in the direction of the natural gradient of the objective function (8) as 0x41 Ox + Pf [Vlog (als) 5* a)]. where F
Ex-ro, amtte
[V log (als) log mee [a is the Fisher information matrix for the current policy Te
then explicitly enforces the trust-region constraint in (8) by line-search: computing the KL-term for 0 = 0x+1 and checking if is larger than the threshold ö, in which case, the step size is reduced until the constraint is satisfied.
In comparison to TRPO, first, on-policy MDPO does not explicitly enforce the trust-region constraint, but approximately satisfies it by performing multiple steps of SGD on the objective function of the optimization problem in the MD-style update rule (6). We say "it approximately satisfies the constraint" because instead of fully solving (6), it takes multiple steps in the direction of the gradient of its objective function. Second, on-policy MDPO uses simple SGD instead of natural gradient, and thus, does not have to deal with the computational overhead of computing (or approximating) the inverse of the Fisher information matrix.' Third, the direction of KL in on-policy MDPO, KL( mT)..
'TRPO does not explicitly invert F. but instead, approximates the natural gradient update using conjugate gradient descent.
4

---

Published as a conference paper at ICLR 2022
is consistent with that in the MD update rule in convex optimization and is different than that in TRPO, (T(,,,, T). This does not cause any sampling problem for either algorithm, as both calculate the KL-term in closed-form (Gaussian policies). Fourth, while TRPO uses heuristies define the step-size and to reduce it in case the trust-region constraint is violated, on-policy MDPO uses a simple schedule, motivated by the theory of MD [4], and sets 4 = =1-k/ K, where K is the maximum numher of iterations. This way it anneals the step-size E3 from 1 to 0 over the iterations of the algorithm.
Comparison with PPO. At each iteration k. PPO performs multiple steps of SGD on the objective function of the following unconstrained optimization problem:
TT(aass) TTp, (als)
Ta(a]s) Tex (als)
max DE0
E
min
4 a), clip
e, -
in which the hyper-parameter determines how the policy ratio, <e/Tge- is clipped. It is easy to see that the gradient of the objective function in (9) is zero for the state-action pairs at which the policy ratio is clipped and is non-zero, otherwise. However, since the gradient is averaged over all the state-action pairs in the batch, the policy is updated even if its ratio is out of bound for some state-action pairs. This phenomenon, which has been reported in [3] and [9], shows that clipping in PPO does not prevent the policy ratios to go out of bound, but it only reduces its probability. This means that despite using clipping. PPO does not guarantee that the trust-region constraint always satisfied. In fact, recent results, including those in [9] and our experiments in Section 5.3, show that most of the improved performance exhibited by PPO is due to code-level optimization techniques, such as learning rate annealing, observation and reward normalization, and in particular, the use of generalized advantage estimation (GAE) [27]. Although both on-policy MDPO and PPO take multiple SGD steps on the objective function of unconstrained optimization problems (6) and (9), respectively, the way they handle the trust-region constraint is completely different.
Another interesting observation is that the adaptive and fixed KI. algorithms (we refer to as KL-PPOO here), proposed in the PPO paper [28], have policy update rules similar to on-policy MDPO. However, these algorithms have not been used much in practice, because it was shown in the same paper that they perform much worse than PPO. Despite the similarities, there are three main differences between the update rules of KL-PPO and on-policy MDPO. First, KL-PPO uses mini-batches whereas MDPO uses the entire data for their multiple (m) gradient updates at each round. Second, the scheduling scheme used for the tg parameter is quite different in KL-PPO and MDPO. In particular, KL-PPO either uses fixed ia or defines an adaptive scheme that updates (increase/decrease)) ls based on the KI. divergence magnitude at that time step. On the other hand, on-policy MDPO uses an annealed schedule update tg. starting from 1 and slowly bringing it down to near 0. Third, similar to TRPO, the direction of KL in KL-PPO, (L( TT. is different than that in on-policy MDPO, KL(TT, TT). Since in our experiments, on-policy MDPO performs significantly better than PPO (see Section 5.3), we conjecture that either any or a combination of the above differences, especially the first two, is the reason for the inferior performance of KL-PPO, compared PPO, reported in [28].
4.2 OFF-POLICY MDPO
In this section, we derive off-policy RL algorithm based on the MD update rule (4). We refer to this as off-policy MDPO and provide the pseudo-code in Algorithm in Appendix A. To emulate the uniform sampling over the state space required by (4), Algorithm 2 samples batch of states from replay buffer D (Line 4). While this sampling scheme is not truly uniform, it makes the update less dependent on the current policy. Similar to the on-policy case, we write the update rule (4) for the policy class θ as
0k41 + arg max (0 0x), where (0 (x) = EA-D Ea (s, a)] dge
1 KL(;; Re, m..) (10) to
-
The main idea in Algorithm 2 is to estimate the advantage or octiue-value function of the current policy, A or Qºx, in an off-policy fashion, using a batch of data randomly sampled from the replay buffer D. In similar manner to the policy update of our on-policy MDPO algorithm (Algorithm 1), described in Section 1, we then update the policy by taking multiple SGD steps on the objective function Vi 9x) of the optimization problem (10) (by keeping 6₁ fixed). A more presentable form of the policy loss in V can be written as follows:
L(0, 0z) B.nD log no (e,(e,)) s)|s) log To, (E,(s)))ss) t2 (s, (o,
(1I)
enN
5

---

Published as a conference paper at ICLR 2022
In particular, the first two terms here obtained just by opening the KL, whereas the advantage estimate is replaced by neural network estimate Qu, which is learned from off ooolicy data in a TD(O) fashion. Furthermore, solely in conjunction with Qv. which is fit to the Qu estimate of the current policy. Finally, the policy loss also uses the reparameterization trick where (o(c,)) is the action generated by sampling the noise from a zero-mean normal distribution N.
implementation detail, another neural network Va is used
We can easily modify Algorithm 2 to optimize seft (entropy regularized) MDPs. In this case, in the critic update (Line 12 of Algorithm 2), the Qu update remains unchanged, while in the Va update, the target changes from Ea (- [a] Eawno, [Q a) log (a]])) The loss function (11) used for the actor (policy) update (Lines 7-9 Algorithm 2) is also modified, Qy becomes the soft Q-function and term Xfą log TTy (a, [s) is added inside the expectation. We denote these changes explicitly in Algorithm 3.
Similarly on- in-policy MDPO that has close connection TRPO and PPO, discussed in Section 4.1, off-policy MDPO (Algorithm and 3) is related to the popular soft actor-critic (SAC) algorithm [13]. We now derive SAC by slight modifications in the derivation of off- off-policy MDPO. This gives an optimization interpretation to SAC, which we then use to show strong ties between the two algorithms.
Comparison with SAC. Soft actor-critic is approximate policy iteration algorithm in soft MDPs. At each iteration k it first estimates the (soft) Q-function of the current policy, Q and then sets the next policy to the (soft) greedy policy w.r.t. the estimated Q-function as
T (als) + exp (QTH (s,*(s,)) SAC (s),
(12)
where SSA = Eas exp a))l is a normalization term. However, since tractable policies are preferred in practice, SAC suggests to project the improved policy back into the policy space considered by the algorithm, using the following optimization problem:
exp (Q (8, (s.))) ZSAC(s)
OK-1 <- arg min CSAC (0, 0k);
S (0, 0z) EAADD KI, 8;
(13)
This update rule computes the next policy as the one with the minimum KL-divergence to the term on the RHS of (12)
Since the optimization problem in (13) is invariant to the normalization term, unlike (12), the policy update (13) does not need to compute ZSAC(s). By writing the KL definition and using the reparameterization trick in (13), SAC updates its policy by minimizing the following loss function:
SAC (0, 0x)
EE Allog no '~
(ān(e, s)]s) (āo
Q% (s, (E(
(14)
Comparing the loss in (14) with the one used in off-policy MDPO (Eq. 11), we notice that despite the similarities, the main difference is the absence of the current policy, Te, in the SAC loss function. To explain the relationship between off-policy MDPO and SAC, recall from Section 2.1 that if the constraint set is the unit simplex, i.e.. C Ax the MD update has the closed-form shown in (2). Thus, if the policy class (constraint set) IT in the update rule (4) is the entire space of stochastic policies, then we may write (4) in closed-form
(see e.g., [21, 29])
Ta+1(a)s) TA (als) exp (t+ (s, a)) / Z(s).
(15)
where Z(s) = Eaawr (aawm((1a)) (1la) exp (s,))( is normalization term. The closed-for form solution (15) equivalent to solving the constrained optimization problem (4) two phases (see [14]): 1) solving the unconstrained version of (4) that leads to the numerator of (15), followed by 2) projecting this (unconstrained) solution back into the constrained set (all stochastic policies) using the same choice of Bregman divergence (KL in our case), which accounts for the normalization term in (15). Hence, when we optimize over the parameterized policy space θ (instead of all stochastic policies), the MD update would be equivalent to finding policy 9 θ with minimum KL-divergence to the solution of the unconstrained optimization problem obtained in the first phase (the numerator of Eq. 15). This leads to the following policy update rule:
0x+1 +- arg min C(0, 02),
C(O, 02) = EEND KL no, ne. exp(te (16)
dge

---

Published as a conference paper at ICLR 2022
If write the definition of KL and use the reparameterizatioo trick in (16), we will rederive the loss function (11) used by our off- MDPO algorithm." Note that both SAC (13) and off ooflcy MDPO (16) use KL projection to project back to the set of policies. For SAC, the authors argue that any projection can be chosen arbitrarily. However, our derivation clearly shows that the selection of KI. projectin is dictated by the choice of the Bregman divergence.
As mentioned earlier. the main difference between the loss functions used in the policy updates of SAC (14) and off-policy MDPO (11) is the absence of the current policy, neg in the SAC's loss function." The current policy, T appears in the policy update of off-policy MDPO, because it is a trust-region algorithm, and thus, tries to keep the new policy close to the old one. On the other hand, following the original interpretation of SAC as an approximate dynamic programming algorithm, its policy update does not contain a term to keep the new and old policies close to each other. is interesting to note that SAC's loss function can be re-obtained by repeating the derivation which leads to off off-policy MDPO, and replacing the current policy, Tay. with the uniform policy in the objective (10) of off-policy MDPO. Therefore, SAC can be considered as trust-region algorithm w.r.t. the uniform policy (or an entropy regularized algorithm). This means its update encourages the new policy remain explorative, by keeping it close the uniform policy.
Trust-PCL [22] uses the path consistency idea along with entropy regularization and an additional term for remaining close to past policy. In principle, this resembles the pooicy MDPO algorithm. However, Trust-PCI. uses a multi-step consistency loss whereas off-policy MDPO uses single transitions. Moreover, besides different derivations, there remain implementation-level details between the two, as Trust-PCL only uses a V network while oof-py MDPO uses a Q function as well.
Due to space constraints, we defer a discussion on the forward and reverse KL directions (including the ECPO [21] algorithm) to Appendix D.
5 EXPERIMENTAL RESULTS
In this section, we empirically evaluate our on-policy and off-policy MDPO algorithms on a number of continuous control tasks from OpenAl Gym [7], and compare them with state-of-the-ar baselines: TRPO, PPO, and SAC. We report all experimental details, including the hyper-parameter values used by the algorithms, in Appendix B. In the tabular results, both in the main paper and in Appendices E and F, report the final training scores averaged over 5 runs and their 95% confidence intervals (CI). We bold-face the values with the best mean scores. We also compare on-policy MDPO and PPO on 21 Atari games from the ALE benchmark [5], showing averages over 5 random seeds. We strictly follow the
Vial
Meon urieooe devietre
2003
Timestops
1002
292
Willer
hyperparameters reported in the PPO paper, and use m
3500
Moon Epcods Rewerd
for all games.
3000
2506
For off- off-policy MDPO, we experiment with two potential functions &6 to define the Bregman divergence By: 1) Shan- non entropy, which results in the KL version (described in Section 4.2), and 2) Tsallis entropy, which results in the Tsallis version of off policy MDPO. We refer the reader to Appendix C for the complete description and detailed deriva- tion of the Tsallis version. Note that we did not pursue a similar bifurcation between Tsallis and KL induced Bregman divergences for the on-policy case since the exact derivations are more tedious there. Another important point to note is that the Tsallis entropy gives us range of entropies, con-
23.0
3500
3300
302
Timesteps
01-33
m-ldd
Figure 2: Performance of on-policy (top) and off-policy (bottom) MDPO (code level optimizations included) for dif- ferent values of m on the Walker2d task.
In soft MDPs, TTh is replaced by its soft version and a term - At& log no, (a|s) is added to the exponential (15). This will result the same changes in (16). Similar to the hard case, applying the reparameterization trick to the the soft version of (1 (16) gives us the soft version of the loss function (11). 'The same difference can also be seen in the policy updates (12) and (15) of SAC and off-policy MDPO.
7

---

Published as a conference paper at ICLR 2022
On PolPilcy
OIT Policy
Fav
MDPO
TRPO
PPO
MDPO-KT.
MDPO-Tsale
SAC
Hopper-v2
2361 S18)
1979 672)
2051 241)
2428 (d: 395)
2428 g
1870 404)
Walker2d-v2
4834 607)
4473 (= 558)
1490 (:: 292)
3591 (d: 366)
4028 (:: g
3738 (2: 312)
HallChoctah-v2
4172 (::: 1156)
3751 (=: 910)
2041 (#: 1319)
11823 (1:: 154)
11823 (d: 150), q
11928 (::: 342)
Ant-v2
5211 (22 43)
4682 (= 278)
59 (#: 155)
4434 (#: 749)
5486 (d: 737). 2-
4989 (:: 579)
Humanoid-v2
3234 (== 566)
44 132)
529 (t: 47)
5323 (1: 348)
5611 g
5191 (2: 312)
H. Standup-12
155261 3898)
149847 2632)
97223 (1:44179)
143955 4199)
165882 16600). q.4 154765 (d: 11721)
Table I: Comparisons on MuJoCo domains. Averaged (over runs) returns for Loaded+GAE version of
MDPO, TRPO, PPO, and SAC algorithms, together with their 95% confidence intervals. On-policy results are for 10M timesteps. The values with the best mean scores are hold-faced.
trolled by the parameter q E (0,2) (see Appendix C). Two special cases are 1) Shannon entropy for q = 1.0, and 2) sparse Tsallis for 2.0 [17, 18, 24].
5.1 ON MULTIPLE SGD STEPS
In on-policy MDPO, we implement the multi-step update at each MD iteration of the algorithm, by sampling M trajectories from the current policy, generating estimates of the advantage function, and performing gradient steps using the same set of trajectories. We evaluated on mn-oolicy MDPO for different values of m in all tasks. We show the results for Walker2d in Figure 2 (top). The results for all tasks show clear trade-off between m and the performance. Moreover, m 10 seems to be the best value across the tasks. This is why we use m = 10 in all our on-policy MDPO experiments. Our results clearly indicate that using m leads to inferior performance compared to m 10, reaffirming the theory that suggests solving the trust-region problem in RL requires taking several gradient steps at each MD iteration. Finally, in our preliminary experiments with TRPO, we observed that performing multiple gradient steps at each iteration of TRPO does not lead to any improvement, sometimes even leading to worse performance than when performing a single-step update.
For off-policy MDPO, performing multiple SGD steps at each MD iteration (Lines 6 to 10 in Algorithm 2) becomes increasingly time-consuming as the value of m grows. This is because off- policy algorithms perform substantially more gradient updates than their on-policy counterparts (a gradient step per environment step vs. a gradient step per almost 1, 000 environment steps). To address this issue, we resort to staying close to an m-step old copy of the current policy, while performing single gradient update at each iteration of the algorithm. This copy is updated every m iterations with the parameters of the current policy. Our results for the Hopper domain in Appendix G.1 show that the performance of MDPO can be improved by performing m gradient updates at each iteration, but omit from performing these experiments scale because of their unreasonably high wall-clock time. Finally, we evaluated poolyy MDPO for different values of m in all tasks and show the results for Walker2d in Figure 2 (bottom). We found hard to identify single best value of for all tasks. However, m 1000 had the most reasonable performance across the tasks, and thus, we use in all our off- policy MDPO experiments.
5.2 ON COVE-LEVEL OPTIMIZATIONS
There are certain "code-level optimization techniques" used in code-bases of TRPO, PPO, and SAC that result in enhanced performance. In [9]. the authors provided a case study of these techniques in TRPO and PPO. We provide a detailed description of these techniques in Appendix B, and report the performance of the algorithms without these techniques (vanilla or minimal version) and with these techniques (loaded and loaded+GAB versions) in Appendices E and F. Note that the loaded+GAB version of TRPO and PPO match their state-of-the-art results in the literature.
Overall, the key takeaway from our results that MDPO performs significantly better than PPO and on-par or better than TRPO and SAC, while being much simpler to implement, and more general as being derived from the theory of MD in RL. In the next two sections, we report our main observations from our on on-pol and off poo-cy experiments.
5.3 ON-POLICY RESULTT
We implemented three versions of on on-policy MDPO, TRPO, and PPO: 1) the vanilla or minimal version, 2) the loaded version in which add the code-level optimization techniques to these
8

---

Published as a conference paper at ICLR 2022
algorithms, and 3) the loaded version plus GAE, whose results are reported in Table 1. The results for all three versions are reported in Appendix E.
We clicit the following observations from our results. First, on-policy MDPO performs better than or on par with TRPO and better than PPO across all tasks. This contradicts the common belief that explicitly enforeing the constraint (e.g., through line-search) done in TRPO is necessary for achieving good performance. Second, on-policy MDPO can be implemented more efficiently than TRPO, because does not require the extra line-search step. Notably, TRPO suffers from scaling issues as it requires computing the correct step-size of the gradient update using a line search, which presents as an incompatible part of the computation graph popular auto-diff packages, such as TensorFlow. Moreover, MDPO performs significantly better than PPO, while remaining equally efficient in terms of implementation. Third, TRPO performs better than PPO consistently, both in the vanilla case and when the code-level optimizations (including GAE) are added to both algorithms. This is in contrast to the common belief that PPO is a better performing algorithm than TRPO. Our observation is in line with what noted in the empirical study of these two algorithms in [9]. and we believe it further reinforces it. Adding code-level optimizations and GAE improve the performance of PPO, but not enough to outperform TRPO, when it also benefits from these additions. Lastly, fourth, it was shown in [31] that PPO is prone to instability issues. Our experiments show that this is indeed the case PPO's performance improves until the standard time-step mark of 1M, and then decreases in some tasks. For example, in the Ant-v2 domain, both PPO and TRPO get to a similar score ( 1000) around the 1M mark but then PPO's performance decreases whereas TRPO continues increase, as can be seen in Table and Appendix E.
Atari results. To show that MDPO can be robustly used as an excellent substitute for PPO, we compare the two algorithms on 21 games from the ALE benchmark. Our results show that MDPO performs better or on par than PPO on 15 out of 21 games, while performing better than PPO on 6 out of 21 games. Due to space constraints, we report the full training plots in Appendix 10. Interestingly. both MDPO and PPO behave quite differently in a lot of games. Since we do not optimize any hyperparameters for MDPO, it might be possible to get more gains with further finetuning. Note that it is well known that TRPO leads to much inferior performance than PPO on the ALE benchmark. Indeed, comparing results with those in the TRPO paper, see that both MDPO and PPO win in out of the 6 games reported in the TRPO paper.
5.4 OFF-POLICY RESULTS
Similar to the on-policy case, we implemented both vanilla and loaded versions of off-policy MDPO and SAC. We report the results of the loaded version in this section (Table 1), and the complete results in Appendix F. We observe the following from these results. First, off-policy MDPO-KL performs on par with SAC across all tasks. Second, off- ofl-policy MDPO-Tsallis that has an extra hyper-parameter to tune can outperform SAC across all tasks. We observe that the best performing values of g are different for each domain but always lie in the interval [1.0, 2.0]. Third, off-policy MDPO results in performance increase in most tasks, both in terms of sample efficiency and final performance, in comparison to on-policy MDPO. This is consistent with the common belicf about the superiority of off- off-policy to on-policy algorithms.
Similar to off-policy MDPO, we can incorporate the Tsallis entropy in SAC. In [18], the authors showed performance improvement over SAC by properly tuning the value of in SAC-Tsallis However, in domains like Humanoid-v2 and Ant-v2, they only reported results for the 1M time-step mark, instead of the standard 3M. In our preliminary experiments with SAC-Tsallis in Appendix G.3, we did not see much improvement over SAC by tuning 4. unlike what we observed in our MDPO- Tsallis results. More experiments and further investigation are hence needed to better understand the effect of Tsallis entropy (and q) in these algorithms.
6 CONCLUSIONS
We derived on-policy and off-policy algorithms from the theory of MD in RL. Each policy update in our MDPO algorithms is formulated as a trust-region optimization problem. However, our algorithms do not update their policies by solving these problems, instead, update them by taking multiple gradient steps the objective function of these problems. We described in detail the relationship between on on-policy MDPO and TRPO and PPO. We also discussed how SAC can be derived by slight modifications of off off-policy MDPO. Finally, using comprehensive set of experiments, we showed

---

Published as a conference paper at ICLR 2022
that on-policy and off-policy MDPO can achieve performance better than equal to these three popular RL algorithms, and thus can be considered as excellent alternatives to them.
We can think of several future directions. In addition to evaluating MDPO algorithms in more complex and realistic problems, we would like to see their performance in discrete action problems in comparison with algorithms like DQN and PPO. Investigating the of Bregman divergences other than KL seems to be promising. Our work with Tsallis entropy is in this direction but more algorithmic and empirical work needs to be done. Finally, there are recent theoretical results incorporating exploration into the MD-based updates. Applying exploration to MDPO could prove most beneficial, especially complex environments.
REFERENCES
[I] Alekh Agarwal, Sham Kakade, Jason Lee, and Gaurav Mahajan. the theory policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
[2] M. Azar, V. Gómez, and H. Kappen. Dynamic policy programming. Journal of Machine Learning Research, 13:3207-3245. 2012.
[3] A. Beck. Finst-order methods optimization. SIAM, 25, 2017.
[4] A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
[5] Mare Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279 279, 2013.
[6] C. Blair. Problem complexity and method efficiency in optimization (A. Nemirovsky and D. Yudin). SIAM Review, 27(2):264, 1985.
[7] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. OpenAl Gym. Preprint arXiv: 160.00600.05550 2016.
[8] Dhariwal, Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor,
Y. Wu, and P. Zhokhov. OpenAl baselines, 2017.
[9] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Im- plementation matters in deep RL: A case study on PPO and TRPO. In Proceeding of the Sth International Conference Learning Representations, 2020.
[10] R. Fox, A. Pakman, and N. Tishby. Taming the noise reinforcement learning via soft updates. In Proceedings of the Thirly-Second Conference on Uncerluinty in Artificial Intelligence, pp. 202- 202-211, 2016.
[11] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning, pp. 1587-1596, 2018.
[12] M. Geist, B. Scherrer, and O. Pietquin. theory of regularized Markov decision processes. In Proceedings of the 36th International Conference Machine Learning, 2019.
[13] Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critie: Off-ppoicy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861-1870, 2018
[14] E. Hazan. Introduction to online convex optimization. arXiv preprint arXiv: 2019.
[15] S. Kakade and J. Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the 19th International Conference on Machine Learning, pp. 267-274, 2002.
[16] H. Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics, 11, 2005.
10

---

Published as a conference paper at ICLR 2022
[17] K. Lee, S. Choi, and S. Oh. Sparse Markov decision processes with causal sparse Tsallis entropy regularization for reinforcement learning. Robotics and Automation Letters, 2018.
[18] K. Lee, S. Kim, S. Lim, S. Choi, and S. Oh. Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning. Preprint arXiv:1902.00137. 2019.
[19] T. Lillicrap, Hunt, Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. Preprint arXiv: / 150.02997, 2015.
[20] B. Liu, Q. Cai, Z. Yang, and Z. Wang. Neural trust region/proximal policy optimization attains globally optimal policy. In Proceedings of Advances in Neural Information Processing Systems, 2019.
[21] Mei, C. Xiao, R. Huang, D. Schuurmans, and M. Muller. On principled entropy exploration in policy optimization. In Proceedings of the 28th Joint Conference on Artificial Intelligence, pp. 3130-3136, 2019.
[22] O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans. Trust-PCL: An off-policy trust region method for continuous control. Preprini 117xxv170..018)) 2017.
[23] O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Proceedings of the 31st Conference on Neural Information Processing Systems, pp. 2772-2782, 2017.
[24] O. Nachum, Y. Chow, and M. Ghavamzadeh. Path consistency learning in Tsallis entropy regularized mdps. In Proceedings of the 35th International Conference on Machine Learning, pp. 979 988, 2018.
[25] G. Neu, A. Jonsson, and V. Gómez. A unified view of retropy-reeularizee markov decision processes. Preprint arrXiv1705.0077 2017.
[26] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of the 32nd International conference on machine learning, pp. 1889-1897, 2015.
[27] Schulman, Moritz, S. Levine, M. Jordan, and Abbeel. High-dimensional continuous control using generalized advantage estimation. Preprint arXiv:1506.02438, 2015.
[28] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. Preprint arXiv: 1707 060000 2017.
[29] L. Shani, Y. Efroni, and S. Mannor. Adaptive trust region policy optimization: Global conver- gence and faster rates for regularized MDPs. In Proceedings of the 33rd AAAI Conference Artificial Intelligence, 2020.
[30] E. Todorov. Linearly-solvable Markov decision problems. In Proceedings of the 19th Advances in Neural Information Processing, pp. 1376-1376, 2006.
[31] Y. Wang, H. He, and X. Tan. Truly proximal policy optimization. In Proceeding of the 35th Conference Uncertainty in Artificial Intelligence, 2019.

---

Published as a conference paper at ICLR 2022
Appendix
A PSEUDOCODES
Below we provide the pseudocodes for the two MDPO algorithms, on on-policy and off-policy.
Algorithm 1 On-Policy MDPO
1: Initialize Value network Ve: Policy networks Nhew and Told'
2: for k = l...., K do
3: # +On-policy Data Generation
4: Simulate the current policy T0, for M steps;
for t = 1,.... M do
6: Calculate return R2 R(se, = ry: Estimate advantage a()),0))= R(ss, - Vo(e:)
7: end for
8:
# Policy Improvement (Actor Update)
9:
0 02:
10:
for O,
do
11:
((++1) - 60 + V 0x) 0=0
(Eq. 7)
12:
end for
0x+1
o(m))
14:
# Policy Evaluation (Critic Update)
15:
Update by minimizing N-minibatch (N S M) loss function Lv, = + Em [Vo(s)) RR]²:
16:
end for
12

---

Published as a conference paper at ICLR 2022
Algorithm 2 Off-Policy MDPO
1: Initialize Replay buffer D = 0: Value networks Va and Qui Policy networks Nnew and Tois:
2: for k= l,...,, K do
Take action aş -e, (-Iss), observe rh and Sk41, add (sk, de, Plk Sk41) to the replay buffer
4: Sample batch [[aj: "j. (544)))) 1) from D;
5: # Policy Improvement (Actor Update)
6: ( Oks
7:
for O, .,..., do
8:
((++1) +- 0 V/Va 0c) o
(Eq. 11)
9:
for 0k+1= = o(m))
10:
11:
# Policy Evaluation (Critic Update)
12:
Update and & by minimizing the loss functions
Lvs x/2 Σ [Vo(s)) Qu Te [8,))]
Lo, = 2 (r()),a)) + [Vo(s-=1) a,)
13: end for
L(0,0,) 0k) = E, Esp log no (āo(c, s)]s) log no, (Qo(c,))ls) to (s. (s,((css)) (Eq. revisited)
Algorithm 3 Off-Policy MDPO (Soft)
1: Initialize Replay buffer D 8: Value networks V4 and Qui Policy networks Nnew and Tota: for I...., K
Take action αş ~ Te, (-18x), observe Ta and 8k+1, and add (sk, ax, Tky 8k+1) to the replay buffer D;
4: Sample batch [[$]: aj, Ty, {1+1)] from D;
# Policy Improvement (Actor Update)
010) Ou:
7:
for O...., ......m-1... do
((++1)
( VoLi 0c) o=
(Eq. 11 soft)
9:
end for
0k+1 o(T))
10:
11:
Policy Evaluation (Critic Update)
Update and & by minimizing the loss functions
Lv, > [Vo(s)) - Q+ (3), menta (8z)) Alog To (s:))
Le, = Σn (r( + (Vo(ss-1) au));
end for
((0,0)) = -E, log (āo(s, - (1 -Atx) log
Te. (āo(e,))\s) t2 (s. (Eq. 11 soft)
eNN
13

---

Published as a conference paper at ICLR 2022
B EXPERIMENTAL DETAILS
B.1 SETUP
We evaluate all algorithms on OpenAl Gym [7] based continuous control tasks, including Hopper-v2. Walker2d-v2, HalfCheetah-v2, Ant-v2, Humanoid-v2 and HumanoidStandup-vV All experiments are run across 5 random seeds. Each plot shows the empirical mean of the random runs while the shaded region represents 95% confidence interval (empirical mean 1.966 empirical standard deviation / vn = 5). We report results in both figure and tabular forms. The tabular results denote the mean final training performance and the best values with overlapping confidence intervals are bolded.
For all off-policy experiments, we use À 0.2 across all tasks, which is known to be the best performing value for all tasks according [13] (In our experiments, value of 0.2 worked equally well for Humanoid as the reported 0.05 in the SAC paper). We report all details of our off-policy experiments including hyperparameter values in Table Moreover, since doing multiple gradient steps at each iteration becomes quite time consuming for the off pooicyy case, get around this issue by fixing the old policy (Teg) for m number of gradient steps, in order to mimic the effect from taking multiple gradients steps each iteration. This ensures that the total number of environment steps always equal to the total number of gradients steps, irrespective of the value of m. Finally, for all experiments, we use fixed Bregman stepsize (I/ty) opposed an annealed version like in the on-policy case.
B.2 CODE-LEVEL OPTIMIZATION TECHNIQUES
The widely available OpenAl Baselines [8] based PPO implementation uses the following five major modifications to the original algorithm presented in [28] value function clipping, reward normalization, observation normalization, orthogonal weight initialization and an annealed learning rate schedule for the Adam optimizer. These are referred to code level optimization techniques (as mentioned above sections) and are originally noted in [9]. Following the original notation, we refer the vanilla or minimal version of PPO, i.e. without these modifications as PPO-M. Then, consider two PPO versions which include all such code level optimizations, with the hyperparameters given [28]. One of them does not use GAE while the other version includes GAE. Therefore they are referred to as PPO-LOADED and PPO-LOADED+GAR respectively. These versions, although being far from the theory, have been shown be the best performing ones, and form as good baseline. We do a similar bifurcation for TRPO and on-policy MDPO. We report all details of our on-policy experiments including hyperparameter values in Table 2.
Similarly, for the off- MDPO versions, we again restrain from using the optimization tricks mentioned above. However we do employ three techniques that are common actor-critic based algorithms, namely: using separate Q and V functions as in [13], using two Q functions to reduce overestimation bias and using soft target updates for the value function. Prior work [11, 19] has shown these techniques help improve stability.
Similar to the on-policy experiments, we include a minimal and loaded version for the off-policy experiments well, which are described in Appendix D. In particular, this branching is done based on the neural network and batch sizes used. Since the standard values in all on on-policy algorithms is different from the standard values used by most off- policy approaches, show results for both set of values. This elicits a better comparison between on-policy and off-policy methods.
14

---

Published as a conference paper at ICLR 2022
Hyperpaaameterr
TRPO-M TRPO-LOADED
PPO-M
PPO-LOADED
MDPO-M MDPO-LOADED
Adam stepsize
3 X
10
Annealed from to
3 x
Annealed from to
minibatch size
128
128
number of gradient updates (m)
10
reward normalization
X
/
X
ohservation normalization
X
V
X
orthogonal weight initialization
value function clipping
X
X
GAE A
1.0
0.95
1.0
0.95
1.0
0,95
horizon (T)
2048
entropy coefficient
0.0
discount
0.99
total number timesteps
107
iruns used for plot averages contidence interval for plot runs
5
95%
Table 2: Hyper-parameters of all on-policy methods.
MDPO-M
MDPO-M
SAC-M
MDPO-LOADED
MDPO-LOADED SAC-LOADED
Hyperparameter
KL
Tsallis
KL
Tsallis
number hidden units per layer
64
256
minibatch size
64
64
256
256
256
entropy coefficient (X)
0.2
Adam slepsize
x 10
reward normalization
X
observation normalization
X
orthogonal weight initialization
X X 105
value function clipping
replay butfer size
target value function smoothing cocfficient number hidden layers
0.005
2
discount factor
0.99
#runs used for plot averages confidence interval for plot runs
5
95%
Table 3: Hyper-parameters of all off-policy methods.
Hopper-v2
Walker2d-v2
HalfCheetah-v2
Ant-v2
Humanoid-v2
HumanoidStandup-v2
Bregman stepsize (1/t))
0.8
0.4
0.3
0.5
0.5
0.3
Table Bregman stepsize for each domain, used by off-policy MDPO.
15

---

Published as a conference paper at ICLR 2022
C TSALLIS-BASED BREGMAN DIVERGENCE
As described in section 2.1, the MD update contains a Bregman divergence term. A Bregman divergence is measure of distance between two points, induced by a strongly convex function v. In the case where the potential function dà is the negative Shannon entropy, the resulting Bregman is the KL divergence. Similarly, when the negative Tsallis entropy, for real number q. i.e.
(T) = 1 q (1 (1-
Σ aaa |)))
(17)
we obtain the Tsallis Bregamn divergence, i.e.,
(T Tx) ạ Σ x(a ()TT((a 5)
Σ x(a s)8 + Σ TT(a s)
(18)
Note that the last term on the RHS of (18) is independent of the policy л being optimized. Also note that as q + 1, the Tsallis entropy collapses to the Shannon entropy - -. т (a 8) log ι(a 8). and thus, it generalizes the Shannon entropy. Moreover, for 2, the Tsallis entropy is called the sparse Tsallis entropy.
At first glance, the above expression is very different from the definition of the KL divergence. However, by defining the function log, as
q-1 4 * 1 and > 0, q = and x: 0,
logg
we may write the negative Tsallis entropy, defined by (17), as
(T) = Σ T(a 8) log, Ta 8);
(19)
and the Tsallis Bregman, defined by (18), in similar manner the KI. divergence
independent of the policy being optimized
(T Tk) = Σ T(a s) (log, T(a s) glog xs(a s)) - (1 - 4) Σ Ta s) log, a | s).
kL(T, for q=1
=0, for g=1
(20)
With this convenient definition, we can write the Tsallis-bast based version of the off off-policy MDPO objective defined in (11) as
I Tsallis (8, 0x) = E.-p e
log,, To (a((((,)))) s){s) q log,, Tu, (āole, s}[s) 22 (c, (21)
Note that the last term on the RHS of (20) is independent of the policy being optimized (i.e., π or 0), and thus, does not appear in the loss function Tuillss 62) in (21).
Note that on-policy MDPO uses a closed form version for the Bregman divergence (since both policies are Gaussian in implementation, a closed form of their KL exists). Such closed form version for the Tsallis based Bregman is quite cumbersome to handle in terms implementation, and thus did not pursue the Tsallis based version in the on-policy experiments. However, in principle, it is very much feasible and we leave this for future investigation.
16

---

Published as a conference paper at ICLR 2022
D REVERSE VS. FORWARD KL DIRECTION
Similar to the on-policy case, the mode-seeking or reverse direction the KL term in off-policy MDPO (Eq. 16) consistent with that in the MD update rule in convex optimization. With this direction of KL, the optimization problems for policy update in both off-policy MDPO and SAC are invariant to the normalization term (s).. Thus, these algorithms can update their policies without computing (s). In [21], the authors proposed an algorithm, called exploratory conservative policy optimization (ECPO), that resembles our soft off-policy MDPO, except in the direction of KI.. Switching the direction of KI. to mean-seeking forward has the extra overhead of estimating the normalization term for ECPO. However, [21], they argue that it results in better performance. They empirically show that ECPO performs better than several algorithms, including one that is close to off-policy MDPO, which they refer to as policy mirror descent (PMD), and report poor performance for it. We did not use their code-base and exact configuration, but did not observe such poor performance for our off-policy MDPO. In fact, experimental results of Section 5.4 show that off-policy MDPO performs better than on-par with SAC in six commonly used MuJoCo domains. More experiments and further investigation are definitely required to better understand the effect of the KL direction in MDPO algorithms.
17

---

Published as a conference paper at ICLR 2022
E ON-POLICY RESULTS
Here, we report the results for all on-policy algorithms, i.e. TRPO, PPO and MDPO. We have three variants here, 1) the minimal version, i.c. (TRPO, PPO, MDPO)-M, which makes use of code level optimizations, 2) the loaded version, i.e. (TRPO, PPO, JOOODDLLDDED, which includes all code level optimizations. and 3) the loaded+GAt GAE version, i.e. (TRPO, PPO. MDPOO-LOADEE+GA.E which includes all code level optimizations and also includes the use of GAE. We see that the overall performance increases in most cases as compared to the minimal versions. However, the trend in performance between these algorithms remains consistent to the main results.
MDPO
TRPO
PPO
Hopper-v2
1964 (1:277)
2382 (1:445)
1281 (d:35))
Walker2d-v2
2948 (#:29)
2454 (d:17))
424 (#92)
HalfCheetah-v2
2873 (4835)
1726 (1:60)
617 (#:35)
Ant-v2
1162 (1738)
1716 (2:33)
-40 (#:33
Humanoid-v2
635 (2:46)
449 (#:9)
448 (2:66
HumanoidSSandup-v22
127901 (4:6211)
100408
(#:112664)
96068 (1:11221)
Table 5: Performance of MDPO-M, compared against PPO-M. TRPO-M six MuJoCo tasks. The results averaged over runs, together with their 95% confidence intervals. values bolded.
mean scores are
Timestoo
Treestoo
(a) Hopper-v2
(b) Walker2d-v2
(c) HallCheettab
Ant-v2
(e) HumanoidStanddup-2
(f) Humanoid.v2
MDPO (Minimal)
(Minimal)
(Minimal)
Figure 3: Performance of MDPO-M, compared against PPO-M M. TRPO-M M on six MuJoCo tasks. The results are averaged over 5 runs, with their 95% confidence intervals shaded.
18

---

Published as a conference paper at ICLR 2022
Timestaps
Timesheeps
Timestapo
(a) Hopper-v2
(b) Walker2d-v2
(c) HallCheetah--22
Timostpss
Timestops
Timeteps
(d) Ant-v2
(c) HumanoidStandup-22
(1) Humanoid-v2
MOPO (Laaded)
TRPO (Loadar)
PPO (Loadod)
Figure 4: Performance of MDPO-LOADED, compared against loaded implementations (excluding GAE) of PPO and TRPO (PFO-LOADED, TRPO-LOADED) on six MuJoCo tasks. The results are averaged over runs, with their 95% confidence intervals shaded.
19

---

Published as a conference paper at ICLR 2022
MDPO
TRPO
PPO
Hopper-v2
2361 (2:518)
1979 (#672)
2051 (d:241)
Walker2d-v2
4834 (1:607)
4473 (d:558)
1490 (1:290)
HalfCheetah-v2
4172 (2:1156)
3751 (2910)
2041 (#:1319)
Ant-v2
5211 (2:43
4682 (d:278)
59 (3))
Humanoid-v2
3234 (4:566)
4414
(#-132)
529 (#-47)
HumanoidSStnduup-2
155261 (4:3898)
149847
(=2632)
97223 (4:4479)
Table 6: Performance of MDPO-LOADED++A.. compared against loaded implementations (including GAE) of PPO and TRPO PPO-LOADDD+GAA TRPO-LOADED+GAAD on six MuloCo tasks. The results are averaged over 5 runs, together with their 95% confidence intervals. The values with the best mean scores are bolded.
308
2.0
(a) Hopper-v2
(b) Walker2d-v2
(c) HallCheetah
300
280
Treeren
(d) Ant-v2
(e) HumanoidStandup--2
(1 Humanoid-v2
MDPD (Loaded++Ga GAE)
TRPO (Loarid++AAE)
PPO (Loaried+GAE)
Figure 5: Performance of MDPO-LOADED--AEE compared against loaded implementations (including GAE) of PPO and TRPO OPO-LOODED+GAE. TRPO-LOADED+GAE) on MuloCo tasks. The results are averaged over 5 runs, with their 95% confidence intervals shaded.
20

---

Published as a conference paper at ICLR 2022
F OFF-POLICY RESULTS
Here, we report the results for all off-policy algorithms, i.e. MDPO and SAC. We have two variants here, 1) the minimal version, i.c. MDPO-M and SAC-M, which uses the standard neural network and batch sizes (64) and 2) the loaded version, i.e. MDPO-LOADED and SAC-LOADED, which uses neural network and batch size of 256. We see that the overall performance increases in most cases as compared to the minimal versions, i.e. SAC-M, MDPO-M. However, the trend in performance between these algorithms remains consistent to the main results.
MDPO-KL
MDPO-Tsallis, Guest
SAC
H2pper-v2
1385 (1:488
1385 (2618), @ = 1.0
1501 (1:414)
Walker2d-v2
873 (1:)
1151 (2218) 9 1.8
635 (d:137)
HalfCheetah-v2
8098 (d:428)
8477 (#450, 9 = 1.4
9298 (4:31)
Ant-v2
1051 (d:244)
2348 (#330) 9 2.0
378 (4:3)
Humanoid-v2
2258 (2:77))
4426 (2:229, y = 1.6
3598 (2:772)
HumanoidStandup--2
131702 (27703)
138157 (288))) y= 1.2
142774 (::4864)
Table 7: Performance of KL and Tsallis based versions of MDPO-M, compared with SAC-M on six MuJoCo tasks. The results are averaged over 5 runs, together with their 95% confidence intervals. The values with the best mean scores bolded.
3030
(6) Hopper-v2
(b) Walker2d-y2
(c) HalfChootah--22
12000
290
03
13
03
00
02
Trestees
(d) Ant-v2
(c) Humanaid-v2
(f) HumanoidStandupp-2
SAC (Minimel)
MDPO-Taalis (Minimal)
MDPO (Minimnel)
Figure 6: Performance of KL and Tsallis based versions of MDPO-M, compared with SAC-M six MuJoCo tasks. X-axis represents time steps in millions. The results are averaged over 5 runs, with their 95% confidence intervals shaded.
21

---

Published as a conference paper at ICLR 2022
MDPO-KO KL
MDPO-Tsallis Qbest
SAC
Hopper-v2
2428 (::395)
2428 (#:39)), 9 = 1.0
1870 (d:404)
Walker2d-v2
3591 (2:36)
4028 (1:27), =.0
3738 (1:122
HalfCheetah--2
11823 (=154)
11823 (2154), q= 1.0
11928 (1:34))
Ant-v2
4434 (1749)
5486 (17377, = 2.0
4989 (2:59)
Humanoid-v2
5323 (#:48)
5611 (4-200), = 1.2
5191 (4:12)
HumanoidStandup-22
143955 (4-4499)
165882
(1:1664), q 1.4
154765 (411721)
Table 8: Performance of KL and Tsallis based versions of MDPO-LOADED, compared with SAC-LOADED on six MuJoCo tasks. The results are averaged over runs, together with their 95% confidence intervals. The values with the best mean scores are bolded.
380
1Rendags
(a) Hopper-v2
(b) Walker2d-v2
(c) HaliCheeethh
0.06
14000
180
6807
Tirastage
Timesrape
(d) Ant-v2
(e) Humanoid-v2
(f) HumanoidSiandup--2
SAC (Loared)
MDPO-Ttallss (Loaded]
MDPD [Loadar]
Figure 7: Performance of KL and Tsallis based versions of IDPO-LOADED. compared with SAC-LOADED on six MuloCo tasks. X-axis represents time steps millions. The results are averaged over runs, with their 95% confidence intervals shaded. Note that although there is overlap in the performance all methods, MDPO achieves a higher mean score in 5 out domains.
22

---

Published as a conference paper at ICLR 2022
G ADDITIONAL EXPERIMENTS
G.1 MULTI-STEP UPDATE
SAC
For off-policy MDPO, use a modified version of doing multi-step updates at each iteration (see section 5.1) due to computational reasons. In order to ensure fair com- parison, we used the single-step gradient updates for SAC our main experiments. Here, we resort to the original algorithm presented in Algorithm 2, wherein gradient steps at each iteration. We compare this version of MDPO-KL with similar multi-step version of SAC, as is originally reported in [13]. In Figure we see that doing multiple updates helps improve the score of both MDPO and SAC, as is expected.
2004
TH7. MOPO
1908
1208
do
Figure 8: Performance of off off-policc MDPO. compared with SAC on Hopper-v2. when do- ing both single and multiple gradient updates each iteration.
G.2 DIFFERENT TSALLIS ENTROPIES FOR BREGMAN AND MDP REGULARIZATION
So far in the paper, we have used the same Tsallis entropy (same value) for defining the Bregman divergence as well as the MDP regularizer. Here, we test the performance for when the two tsallis entropies are different. For this, sample from set of three q values (1.0, 1.5, 2.0} and report the results for every possible combination of q values used for defining the Bregman divergence and the MDP regularizer. We test this the Walker2d-v2, Humanoid-v2 and Ant-v2 domains (domains where we see the most improvement due to the addition of Tsallis entropy) and observe that sticking to the same values for both cases results in the best performance across all three domains (see Table 9).
Bregman q
MDP
1.0
9= 1.5
2.0
q=
3891 (1:36)
3268 (1:234)
1007 (1:422)
Walker2d-2
1=
2126 (1:456)
2805 (1:302)
1573 (1.326)
4=2.0
14 (::5)
2915 (4:391)
4028 (4:287)
9= 1.0
4434 (1:749)
3007 (1:572)
1913 (1:973)
Ant-v2
1=5.5
4119 (1:326)
5488 (1:233)
2781 (1:812)
4=2.0
(=951)
(8:184)
9.0
5323 (1:34)
4734 (1:341)
4561 (2:311)
Humanoid-v2
1=5
24 (22-)
5013 (1:274)
3766 (2:331)
4=2.0
(2
28(2)))
2751 (2:34)
Table 9: Different Tsallis Entropies. The results averaged over 5 runs, with 95% confidence intervals shaded.
G.3 TSALLIS-BASED SAC
We test performance of SAC-Tsallis while varying the q values. In our preliminary experiments with SAC-Tsallis in Table 9, we did not see much improvement over SAC by tuning 9. unlike what we observed in our MDPO-Tsallis results. More experiments and further investigation are definitely needed to better understand the effect of Tsallis entropy (and g) in these algorithms.
ABEV2
Molthccthh
Humhrr swnorrosstiddus 12
1300
112.00
Moan www.bdd www.adddtttaaaoo.... Rtasurd
4000
303000
400
0306
1000
4200
Trmentoga
Trasstopa
Trraslga
Figure 9: Performance of Tsallis SAC. The results averaged over 5 runs, with 95% confidence intervals shaded.
23

---

Published as a conference paper at ICLR 2022
H
ATARI RESULTS
Nomatbed Soore
Normalee Score
400
#000
3000
200
2000
1000
2503
4020
6200
12000
2005
4005
5005
2050
12000
6200
alles
extieix
midoo
mrdco
ppo
memmatteee fcoee
10000
10000
500
2000
1000
2015
2010
0003
2010
2000
2005
40000
iidpo
midgo
SPC
mdppo
Norma and Scor
Normel and Scoe
200
200
200
2000
2000
20010
6010
5000
cemtpede
msggo
Xpo
miloo
mdpp
1200
Norma and Score
100000
Nomn and Siore
60000
1000
50000
20002
Chapperommmandd
crazyclibeer
demonnateo
Homaleed Suote
4000
70
Nommatedd
20
2000
6503
9000
froeway
Seatbli
g00te"
20000
15000
Memaried Scart
#200
Nemaiied Scote
1000
2000
2000
2000
2000
0030
2002
4202
0000
2000
4030 pong
8000
8000
kruil
mitpo
opp
mitio
mitto
poo
Norma loed Score
15000
1500
Narma teee Score
1000
30200
1000
500
5000
Nomma
250
13
20000
3000
1000
1000
2010
20035
6000
10000
2000
4000
1010
1070
Figure 10: Comparison of MDPO with PPO 21 Atari games. Order of magnitude of x-axis 103, which roughly corresponds to 10M environment time steps or 40M game frames.
24


Reviews:
[{"review_id": "p9sZrVUFhjk", "reviewer": "Reviewer_qkZc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written, and the motivation is easy to follow. \n- Going beyond the tabular RL, the paper derives a scalable algorithm from the MD principle. Moreover, the algorithm is practical since it allows for function approximation.\n- The result is well supported empirically over several different tasks, which shows the advantage of MDPO.\n\nWeakness:\n- The paper does not provide any theoretical analysis for the algorithm. While the convergence and optimality of mirror descent are well-known for the convex constrained problems, it is still not sure if reinforcement learning lies in such a framework.\n- My primary concern is about the novelty of the paper. The connection between PPO, TRPO, and mirror descent is well-known (see, e.g. [11, 19, 20, 24, 28] and [A]). Extending the tabular setting to the function approximation setting is also well studied in [19, 26. 27] both theoretically and empirically. \n\n[A] Zhan, Wenhao, et al. \"Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence.\" arXiv preprint arXiv:2105.11066 (2021).\n", "summary_of_the_paper": "The paper connects the optimization method, mirror descent, to the study of the policy optimization method. Based on the mirror descent principle, the paper proposes the MDPO algorithm, which updates the policy via approximately solving a trust-region problem. The paper proposes the on-policy and off-policy variants of MDPO. Furthermore, the paper connects the on-policy MDPO to PPO and TRPO and connects the off-policy MDPO to SAC. The contribution of the paper is to provide a unified viewpoint of several RL algorithms and shows that   MDPO performs equally or better than TRPO, PPO, and SAC in different tasks.\n", "summary_of_the_review": "Overall, the paper is well written, and the empirical study well supports the result. However, the paper lacks the theoretical analysis, and the novelty of deriving MDPO from MD is limited. Given that, I recommend rejection for the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VBAytB8-BCk", "reviewer": "Reviewer_fuGH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposed an algorithm inspired by Mirror descent, much like TRPO and PPO, both of which are well known and used.\nThe method appears to work convincingly better than the state of the art in the proposed experiments.\nFor the on policy setting, an interesting difference with TRPO is the way the trust region subproblem is solved.\nA great strength of this work is that it nicely connects with the available literature, clearly summarizing the state of the art as well as the algorithmic differences. A reference that could provide some value is 'On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift' by Agarwal et al. ", "summary_of_the_paper": "The work proposes an algorithm like TRPO and PPO inspired by Mirror Descent. The contribution of the paper is primarily algorithmic and also empirical; there is no supporting theory for the proposed algorithm but I believe this is fine for this type of paper.\n", "summary_of_the_review": "I support the paper for publication. I come from more from the theory community, and therefore ask the metareviewer to discount my ( unfortunately brief) review accordingly.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RZEqC7QhEWJ", "reviewer": "Reviewer_wxw9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is well-written and easy to follow. More specifically, this work compared the novel algorithm with previous work (PPO, TRPO, SAC) carefully and clarified the experiment setting precisely.\n\n2. The experiment parts contain some continuous control tasks, and this result highly supports the efficiency of the proposed algorithm compared with previous work (PPO, TRPO, SAC).\n\n3. However, I still have some concerns about this work. It seems that this paper this work is a combination of mirror gradient descent with the previous TRPO or PPO structure, and the technical contribution is limited. Furthermore, the author compares the TRPO or PPO  algorithm, and it seems that none of the differences will improve the experiments. So it looks better if the author can make more discussion of the intuition compared with previous work.", "summary_of_the_paper": "This paper focuses on reinforcement learning with the tabular Markov decision process setting and proposes the mirror descent policy optimization for off-policy and on-policy situations. Furthermore, the experiment result shows that this new algorithm outperforms the previous algorithm in both off-policy and on-policy situations.", "summary_of_the_review": "Based on the previous comment, it is marginally above the acceptance threshold due to the excellent experiment result.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3JTkmnkbcG", "reviewer": "Reviewer_ELQd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nMain Review:\nOverall I think this could be a good paper. I am really interested in the connections between PPO, TRPO and SAC and MDPO seems to be a more general version of all three of them. Experimentally the results are very sound and seem to outperform existing methods while also being a simpler algorithm with fewer hyperparameters. I could see this method becoming very popular as a standard RL baseline. \n\nWith that being said, I think the off-policy section needs some work. Equation 11 is described in the appendix but it is not described when it is introduced. Can you include a description of how this equation works? At least give a reference to the appendix please. In general I thought the entire paragraph starting with “the main idea in algorithm 2…” was poorly written and confusing. Maybe this should be expanded into a couple of paragraphs. Can you explicitly write out the soft version of off-policy MDPO instead of just describing it in the text? Maybe include a new algorithm? “If we write the definition of KL and use the reparameterization trick in(16), we will rederive the loss function(11)used by our off-policy MDPO algorithm” I don’t think this is true, wouldn’t it be the same as (16) but without the current policy term? Maybe you can write out this equation. \n\nAlso, the related works could be explained better. The authors did a great job describing the differences between MDPO and TRPO/PPO/SAC, but there are many other related methods out there that should be referenced and talked about, for example Trust-PCL.  \n\nMinor points:\n> “Since finding an optimal policy for an MDP involves solving a non-linear system of equations and the optimal policy may be deterministic (less explorative),”\n \nNot sure if this explains why one might use an entropy regularizer. \n\n> “Finally, we ran a similar experiment for TRPO which shows that performing multiple gradient stepsat each iteration of TRPO does not lead to any improvement. In fact our results showed that in somecases, it even leads to worse performance than when performing a single-step update”\n\nWhere are these results? Can you reference them in the text? \n\nCould you put a vertical line in the results to separate on policy and off policy algorithms? \nI was surprised to see MDPO not learn at all in the freeway atari experiment. Could you add some more commentary on the atari experiments? PPO seems to do much better against MDPO than the mujoco experiments. \nCould you add hyperlinks to references and figures/algorithms?", "summary_of_the_paper": "Summary:\nInspired by recent theoretical analysis of TRPO and PPO that use mirror descent, this paper proposes two new algorithms that directly minimize a mirror descent objective by taking multiple gradient steps. While similar to TRPO and PPO, MDPO is different in important ways, and happens to perform better in practice. The authors also propose a similar off-policy version of MDPO that happens to be closely related to SAC while also outperforming SAC experimentally. \n\n\n", "summary_of_the_review": "I really like this paper but think the off-policy section can be re-written to be more coherent. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes and studies a variant of policy optimization---mirror descent policy optimization (MDPO)---which was inspired by the mirror descent algorithm in the optimization literature. The proposed algorithm attempts to find a policy parameter that maximizes the expected regularized advantage function,  where the regularization term is based on the KL divergence between the new policy iterate and the current policy iterate. The main contributions are algorithmic and empirical, with detailed discussions provided to illuminate the connection between MDPO and other existing policy optimization paradigms like TRPO, PPO, etc. The paper provides an interesting and useful contribution to the growing literature of policy optimization.
Decision: Accept (Poster)