Differentiable Trust Region Layers for Deep Reinforcement Learning
Authors: Fabian Otto
Philipp Becker
Vien Anh Ngo
Hanna Carolin Maria Ziesche
Gerhard Neumann
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
trust region
policy gradient
projection
Wasserstein distance
Kullback-Leibler divergence
Frobenius norm

Abstract: Trust region methods are a popular tool in reinforcement learning as they yield robust policy updates in continuous and discrete action spaces. However, enforcing such trust regions in deep reinforcement learning is difficult. Hence, many approaches, such as Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), are based on approximations. Due to those approximations, they violate the constraints or fail to find the optimal solution within the trust region. Moreover, they are difficult to implement, often lack sufficient exploration, and have been shown to depend on seemingly unrelated implementation choices. In this work, we propose differentiable neural network layers to enforce trust regions for deep Gaussian policies via closed-form projections. Unlike existing methods, those layers formalize trust regions for each state individually and can complement existing reinforcement learning algorithms. We derive trust region projections based on the Kullback-Leibler divergence, the Wasserstein L2 distance, and the Frobenius norm for Gaussian distributions. We empirically demonstrate that those projection layers achieve similar or better results than existing methods while being almost agnostic to specific implementation choices. The code is available at https://git.io/Jthb0.


Published as a conference paper at ICLR 2021
DIFFERENTIABLE TRUST REGION LAYERS FOR DEEP REINFORCEMENT LEARNING
Fabian Otto"
Philipp Becker
Bosch Center for Artificial Intelligence University of Tübingen
Karlsruhe Institute of Technology
Ngo Anh Vien & Hanna Carolin Ziesche Bosch Center for Artificial Intelligence
Gerhard Neumann
Karlsruhe Institute of Technology
ABSTRACT
Trust region methods are a popular tool in reinforcement learning as they yield ro- bust policy updates in continuous and discrete action spaces. However, enforcing such trust regions in deep reinforcement learning is difficult. Hence, many ap- proaches, such as Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), are based on approximations. Due to those approximations, they violate the constraints or fail to find the optimal solution within the trust re- gion. Moreover, they are difficult to implement, often lack sufficient exploration, and have been shown to depend seemingly unrelated implementation choices. In this work, we propose differentiable neural network layers to enforce trust re- gions for deep Gaussian policies via closed-fon projections. Unlike existing methods, those layers formalize trust regions for each state individually and can complement existing reinforcement learning algorithms. We derive trust region projections based on the Kullback-Leibler divergence, the Wasserstein L2 dis- tance, and the Frobenius norm for Gaussian distributions. We empirically demon- strate that those projection layers achieve similar or better results than existing methods while being almost agnostic to specific implementation choices. The code is available at pettps://qit.io//
INTRODUCTION
Deep reinforcement learning has shown considerable advances in recent years with prominent ap- plication areas such as games (Mnih et al., 2015, Silver et al.) 2017), robotics (Levine et al., 2015). and control (Duan et al. (0166) In policy search, policy gradient (PG) methods have been highly successful and have gained, among others, great popularity (Peters Schaal 20088. However, often it is difficult to tune learning rates for vanilla PG methods, because they tend to reduce the entropy of the policy too quickly. This results in a lack of exploration and, as a consequence, in premature or slow convergence. A common practice to mitigate these limitations is to impose constraint on the allowed change between two successive policies. Kakade & Langford (2002) provided a theoretical justification for this in the approximate policy iteration setting. Two of the arguably most favored policy search algorithms, Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a) and Proximal Policy Optimization (PPO) (Schulman et al.) 2017). follow this idea using the Kullback- Leibler divergence (KL) between successive policies as a constraint.
We propose closed-form projections for Gaussian policies, realized as differentiable neural network layers. These layers constrain the change successive policies by projecting the updated policy onto trust regions. First, this approach is more stable with respect to what Engstrom et al. (2020) refer to as code-level optimizations than other approaches. Second, it comes with the benefit of imposing constraints for individual states, allowing for the possibility of state-dependent trust regions. This allows us to constrain the state-wise maximum change of successive policies. In this we differ from previous works, that constrain only the expected change and thus cannot rely exact guarantees of monotonic improvement. Furthermore, we propose three different similarity measures, the KL
"Correspondence fabian.otto bosch.com

---

Published as a conference paper at ICLR 2021
divergence, the Wasserstein L2 distance, and the Frobenius norm, to base our trust region approach on. The last layer of the projected policy is now the the trust region layer which relies on the old policy input. This would result in ever-growing stack of policies, rendering this approach clearly infeasible. To circumvent this issue we introduce a penalty term into the reinforcement learning objective to ensure the input and output of the projection stay close together. While this still results in an approximation of the trust region update, we show that the trust regions are properly enforced. We also extend our approach to allow for controlled evolution of the entropy of the policy, which has been shown to increase the performance in difficult exploration problems (Pajarinen et al. 2019. Akrour et al. 2019).
We compare and discuss the effect of the different similarity measures as well as the entropy control on the optimization process. Additionally, we benchmark our algorithm against existing methods and demonstrate that we achieve similar or better performance.
2 RELATED WORK
Approximate Trust Regions. Bounding the size of the policy update in policy search is a com- mon approach. While Makade Langford (2002) originally focused on a method based on mixing policies, nowadays most approaches use KL trust regions to bound the updates. |Peters et al. (2010) proposed a first approach to such trust regions by formulating the problem as a constraint optimiza- tion and provided a solution based on the dual of that optimization problem. Still, this approach is not straightforwardly extendable to highly non-linear policies, such as neural networks. tempt to transfer those ideas to deep learning, TRPO (Schulman et al. 2015a) approximates the KL constraint using the Fisher information matrix and natural policy gradient updates (Peters & Schaal, 2008 Kakade, 200T), along with backtracking line search to enforce hard KL constraint. Yet, the resulting algorithm scales poorly. Thus, [Schulman et al. (2017) introduced PPO, which does not directly enforee the KL trust region, but clips the probability ratio in the importance sampling objec- tive. This allows using efficient first-order optimization methods while maintaining robust training. However, Engstrom et al. (2020) and Andrychhoicc al. (2020) recently showed that implementa- tion choices are essential for achieving state-of-the-an art results with PPO. Code-level optimizations, such as reward scaling as well as value function, observation, reward, and gradient clipping, can even compensate for removing core parts of the algorithm, e. g. the clipping of the probability ratio. Ad- ditionally, PPO heavily relies on its exploration behavior and might get stuck in local optima (Wang) jtt al., 2019). Tangkaratt et al. (2018) use a closed-form form solution for the constraint optimization based on the method of Lagrangian multipliers. They, however, require quadratic parametrization of the Q-Function, which can limit the performance. Pajarinen et al. (2019) introduced an approach based on compatible value function approximations to realize KL trust regions. Based on the rein- forcement learning as inference paradigm (Levine 2018). AAbdlmaleki et al. (2018) introduced an actor-critic approach using Expectation-Maxe Maximization based optimization with KL trust regions in both the E-step and M-step. [Song et al. (2020) proposed an on-policy version of this approach using a similar optimization scheme and constraints.
an at-
Projections for Trust Regions. Akrour et al. (2019) proposed Projected Approximate Policy It- eration (PAPI), a projection-basee solution to implement KL trust regions. Their method projects an intermediate policy, that already satisfies the trust region constraint, onto the constraint bounds. This maximizes the size of the update step. However, PAPI relies on other trust region methods to generate this intermediary policy and cannot operate in a stand-alone setting. Additionally, the projection is not directly part of the policy optimization but applied afterwards, which can result in sub-optimal policies. In context of computational complexity, both, TRPO and PAPI, simplify the constraint by leveraging the expected KI. divergence. Opposed to that, we implement the projec- tions as fully differentiable network layers and directly include them in the optimization process. Additionally, our projections enforce the constraints per state. This allows for better control of the change between subsequent policies and for state-dependent trust regions.
For the Kasbased projection layer need to resort to numerical optimization and implicit gradients for convex optimizations (Amos & Kolter 2017 Agrawal et al.] 2019). Thus, we investigate two alternative projections based the Wasserstein L2 and Frobenius norm, which allow for closed form solutions. Both, Wasserstein and Frobenius norm, have found only limited applications in reinforcement learning. Pacchiano al. (000)) the Wasserstein distance score behaviors of
2

---

Published as a conference paper at ICLR 2021
agents. Richemond & Maginnis (2017) proposed alternative algorithm for bandits with Wasser- stein based trust regions. Song & Zhao (2000) focus on solving the trust region problem for distri- butional policies using both KL and Wasserstein based trust regions for discrete action spaces. Our projections are applicable independently of the underlying algorithm and only assume a Gaussian policy, common assumption for continuous action spaces.
Several authors (Dalal et al. 2018 Chow et al. 2019 Yang et al. 2020) used projections as network layers to enforce limitations in the action or state space given environmental restrictions, such as robotic joint limits.
Entropy Control. Abdolmaleki et al. 2015 introduced the idea of explicitly controlling the de- crease entropy during the optimization process, which later was extended to deep reinforcement learning by Pajarinen et al. (2019) and Akrour et al. (2019). They use either an exponential or linear decay of the entropy during policy optimization to control the exploration process and escape local optima. To leverage those benefits, we embed this entropy control mechanism in our differentiable trust region layers.
PRELIMINARIES AND PROBLEM STATEMENT
We consider the general problem of policy search in a Markov Decision Process (MDP) defined by the tuple A, T. R, Po: 7). We assume the state space S and action space A are continuous and the transition probabilities T S x A S + [0, IJ describe the probability transitioning to state 8t+11 E S given the current state 8t E S and action at E A. We denote the initial state distributions as Po S [0, 1]. The reward returned by the environment is given by a function R S x A + I and 7 E 1) describes the discount factor. Our goal is to maximize the expected accumulated discounted reward R ET, Po.n 1E== /Ei=o'RR(s (s))., To find the optimal policy, traditional PG methods often make use of the likelihood ratio gradient and an importance sampling estimator. Moreover, instead of directly optimizing the returns, it has been shown to be more effective to optimize the advantage function as this results in an unbiased estimator of the gradient with less variance
Ta(als)
max (TT, TBas)) = max B (o Ta (als)
A Ta (s, (s.
(I)
where (s, E [1T 8, a0 a; E [ = 8; T| describes the advantage function, and the expectation is w.r.t Tha. i.e. ~ T-1ss, a), Tu (11),, 80 ~ Po((a)) 8 ~ Pra... where Prees stationary distribution of policy Tew.- The advantage function is commonly estimated bygeneralized advantage estimation (GAE) (Schulman et al. 2015b). Trust region methods use additional constraints for the given objective. Using constraint on the maximum KL over the states has been shown to guarantee monotonic improvement of the policy (Schulman et al.) 20) However, since all current approaches do not use maximum KL constraint but an expected KL constraint, the guarantee of monotonic improvement does not hold exactly either. We are not aware of such results for the W2 distance or the Frobenius norm.
For our projections we assume Gaussian policies Tea (aels)) (a (sr) = (a, ({(s),, (st)) represent the old as well as the current policy, respectively. We explore three trust regions on top of Equation I that employ different similarity measures between old and new distributions, more specifically the frequently used reverse KL divergence, the Wasser- stein L2 distance, and the Frobenius norm.
((ae|Poua(ss Es))(s:) and
Reverse KL Divergence. The KL divergence between two Gaussian distributions with means μ4 and #2 and covariances Σ and Eg can generally be written as
1 KL(Um, En} II (#z, ExH) 2 (w2 EEz 'w2 - m1) + log
Σg
1r d
where d is the dimensionality of ,l 112. The KL uses the Mahalanobis distance to measure the similarity between the two mean vectors. The difference of the covariances is measured by the difference in shape, i.e., the difference in scale, given by the log ratio of the determinants, plus the difference in rotation, given by the trace term. Given the KI. is non-symmetric, it is clearly not a distance, yet still frequently used divergence between distributions. We will use the more common reverse KI. for our trust region, where the first argument is the new policy and the second is the old policy.
3

---

Published as a conference paper at ICLR 2021
Wasserstein Distance. The Wasserstein distance is distance measure based optimal trans- port formulation, for more details see Villani (2008). The Wasserstein-2 distance for two Gaussian distributions generally be written
W2 (1u,, E1), (u2, Σ2)) = Im n212 + tr E1 -2 2 1/2
A key difference to the KL divergence is that the Wasserstein distance is a symmetric distance measure, c., W2(q. p) W2(p, q). Our experiments also revealed that it is beneficial measure the W2 distance in metric space defined by the covariance of the old policy distribution, denoted here as 2g, as the distance measure is then more sensitive the data-generating distribution. The W2 distance in this metric space reads
W2 (1/1 {12. Σ2}) =(12 p21² Dn (12 11))
+ tr Σ +I- ( " 1/)
Frobenius Norm. The Frobenius norm a matrix norm and can directly be applied the differ- ence of the covariance matrices of the Gaussian distributions. To measure the distance of the mean vectors, we will, similar the KL divergence, employ the Mahalanobis distance as this empirically leads to an improved performance in comparison to just taking the squared distance. Hence, we will denote the following metric as Frobenius norm between two Gaussian distributions
F( Σ1}. {02, E2)) = (N2 ET ' m) + tr((E2 "1)"(EE,)) -11)-
The Frobenius norm also constitutes symmetric distance measure.
4 DIFFERENTIABLE TRUST-REGION LAYERS FOR GAUSSIAN POLICIES
We present projections based on the three similarity measures, i.e., Frobenius norm, Wasserstein L2 distance, and KL divergence. These projections realize state-wise trust regions and can directly be integrated in the optimization process as differentiable neural network layers. Additionally, we extend the trust region layers include entropy constraint gain control over the evolution of the policy entropy during optimization. The trust regions are defined by a distance or divergence d[sr(-))) (ota(-(s)) between probability distributions. Complementing Equation I with the trust gion constraint leads to
max J (TT, Te..)
s.t. d( (-1s)), TT(-|s)) Se E.
(2)
While, in principle, we want enforce the constraint for every possible state, in practice, we can only enforce them for states sampled from rollouts of the current policy.
To solve the problem in Equation 2 standard neural network will output the parameters μt, E of Gaussian distribution Tp. ignoring the trust region bounds. These parameters are provided to the trust region layers, together with the mean and covariance of the old policy and parameter specifying the size of the trust region c. The new policy is then given by the output of the trust region layer. Since the old policy distribution is fixed, all distances divergences used in this paper can be decomposed into a mean and a covariance dependent part. This enables us to use separate trust regions as well as bounds for mean and covariance, allowing for more flexibility in the algorithm. The trust region layers aim to project my into the trust region by finding parameters j and Σ that closest to the original parameters y and while satisfying the trust region constraints. The projection is based on the same distance divergence which used to define the respective trust region. Formally, this corresponds to the following optimization problems for each s
arg min dmmea (Øs: ((s))s p.
s.t. dmean (Px: Mulaa (s)) (s)) S Gp: and
(3)
arg min doov (Le [{s) s.t. door (Se, Coi((s) S EY,
(4)
where A. and Σ, the optimization variables for state Here, dmean is the mean dependent part and doov is the covariance dependent part of the employed distance or divergence. For brevity of notation we will neglect all dependencies on the state in the following. We denote the projected policy (a[s) N(aai,, Ê))
4

---

Published as a conference paper at ICLR 2021
4.1 PROJECTION OF THE MEAN
For all three trust region objectives we make use of the same distance measure for the mean, the Mahalanobis distance. Hence, the optimization problem for the mean is given by
arg min (p+ - μi) Enl (μ - pi)
s.t. (Pels - pi)" Dold (Hold - (i) S cp.
(5)
By making use of the method of Lagrangian multipliers (see Appendix B.2). we can formulate the dual and solve it for the projected mean ji
1r + WiHold
(Hold 11) 2old (Hold - 14) 1.
i
with
(6)
+
6p
This equation can directly be used as mean for the Gaussian policy, while it easily allows to compute gradients. Note, that for the mean part of the KL we would need to use the E-1 instead of 2old in Yet, this objective still results in valid trust region problem which is
the objective of Equation 5 much easier to optimize.
4.2 PROJECTION OF THE COVARIANCE
Frobenius Projection. The Frobenius projection formalizes the trust region for the covariance with the squared Frobenius norm of the matrix difference, which yields
arg min tr (E-
s.t. tr tr(oouu Σ) (Eol Σ) c
We again use the method of Lagrangian multipliers (see Appendix B.3) and get the covariance E as
Σ + Eold
with tr (Eola E)T(E Σ)) -1
Σ
1+1
E
where 7) the corresponding Lagrangian multiplier.
Wasserstein Projection. Deriving the Wasserstein projection follows the same procedure. We obtain the following optimization problem
arg min tr En Σ 2 (EVAEEVA) )
(8)
s.t. tr (T+ 2
1/2 (///
1/2 old
Ex.
where 1 is the identity matrix. A closed form solution to this optimization problem can be found by using the methods outlined in Takatsu (201I). However, found the resulting solution for the projected covariance matrices to be numerically unstable. Therefore, we made the simplifying assumption that both the current Σ and the old covariance Eou commute with Ê. Under the common premise of diagonal covariances, this commutativity assumption always holds. For the more general case of arbitrary covariance matrices, we would need ensure the matrices are sufficiently close together, which is effectively ensured by Equation g Again, we introduce Lagrange multipliers and solve the dual problem to obtain the optimal primal and dual variables (see Appendix [.4]. Note however, that here chose the square root of the covariance matrix as primal variable. The corresponding projection for the square root covariance / is then
[ - 22 1/-1/22 1///////)
1//2 + E Σー
with
1+
where 77 is the corresponding Lagrangian multiplier. We see the same pattern emerging as for the Frobenius projection. The chosen similarity measure reappears in the expression for the Lagrangian multiplier and the primal variables are weighted averages of the corresponding parameters the old and the predicted Gaussian.
We assume the true matrix square root Σ //2 and not Cholesky factor Σ, I since it naturally appears in the expressions for the projected covariance from the original Wasserstein formulation.
5

---

Published as a conference paper at ICLR 2021
(a) Frobenius
(b) Wasserstein
(c) KL
(d) Entropy
Prileasa
Ki
3.00
1077 10711 1074
100
100
Figure 1: (a). (b), and (c): Interpolated covariances for the different projections for various values of For Frobenius and Wasserstein the intermediate distributions clearly have a larger entropy, while for the KL projection the intermediate entropy smaller. (d): Entropy of the interpolated distribu- tions. In this example π and Told have the same entropy. It can be seen that the entropy increases for the Frobenius and Wasserstein projections when transitioning between the distributions, while decreases for the KL. A more general statement regarding this can be found in Theorem !
KL Projection. Identically to the previous two projections, we reformulate Equation as
arg min (x-1S) + log
Σ
s.t. (Ead d log
E
ex: (10)
[
where d is the dimensionality of the action space. It is impossible to acquire a fully closed form solution for this problem. However, Abdolmalki et al. (2115), we can obtain the projected
precision A = Σ-1 by interpolation between the precision matrices of the old policy Vold and the current policy π
A 17 Aold + A "1" arg min ((1), s.t. 17 2 0,
(11)
1f"
where v is the corresponding Lagrangian multiplier and g(n)) the dual function. While this dual cannot be solved in closed form, an efficient solution exists using a standard numerical optimizer, such BFGS, since it is 1-D convex optimization. Regardless, we want a differentiable projection and thus also need to backpropagate the gradients through the numerical optimization. To this end, & Kolter (017) and compute those gradients by taking the differentials of the KKT
follow Amos follow[Amos conditions of the dual. We refer to Appendix B.5 for more details and derivations.
2019 Abdolmaleki al. 2015) have shown the benefits of introducing an entropy constraint H(TT) > 3 in addition to the trust region constraints. Such a constraint allows for more control over the exploration behavior of the policy. In order to endow our algorithm with this improved exploration behavior, we make use of the results from Akrour al. (2019) and scale the standard deviation of the Gaussian distribution with scalar factor exp [3 - (T)) /d}, which can also be individually computed per state.
Entropy Control. Previous works (Akrour et
4.3
ANALYSIS OF THE PROJECTIONS
instructive to compare the three projections. The covariance update is an interpolation for all three projections, but the quantities that are interpolated differ. For the Frobenius projection we directly interpolate between the old and current covariances (Equation 7 for the W2 projection between their respective matrix square roots (Equation 9 and for the KL projection between their inverses (Equation [T]. In other words, each projection suggests which parametrization to use for the covariance matrix. The different interpolations also have an interesting effect on the entropy of the resulting covariances which can observed in Figure Further, we can prove the following theorem about the entropy of the projected distributions
Theorem Let To and Te... be Gaussian and 17 2 0, then for the entropy of the projected distribution H(T) it holds that H()) 2 minimum (TT)) (T)))) for the Frobenius (Equation 7 and the Wasserstein projection (Equation 9). as well as, H(T) S. maximum H)(T)))) for the KI. projection (Equaation) 11.

---

Published as a conference paper at ICLR 2021
Table I: Mean return with 95% confidence interval of 20 epochs after completing 20% of the total training and for the last 20 epochs. We trained 40 different seeds for each experiment and com- puted five evaluation rollouts per epoch. The projections with (-E) and without entropy control are considered separately, therefore, each column may have up to two best runs (bold).
Hoppen-22
Walker2d v2
Hallcheetab x2
Amt-2
Humanoid v2
2015
Binal
20%
final
20%
dina
20%
final
21/5
final
FROB
1646 19
2578 17
2142 à
3411 1 19
2525 11
3552
1265
3035 4. 26
2176 05
5200 25
W2
+
2400
2284
33500
258G
3002
1962
87
5057
KL
1584
2470
2077
2300
4255
17
1400
3030
2923
0610
PAPI
1378 20
2540 12
1663 + 21
3032 30
1675 + 5
2380 6-6
645.+
3198 17
1824 + 74
5367 2:22
PPO-M M
1630 1:25
2321
1964
2771 40
1922 15
3272 18
1494 1.
2783 4 32
5172 4.25
2515
2400
2048
1637
2852
25
FROB-E
1567 1:30
2478 11
2007 1. 21
3370 1 27
2762 =: 10
4368 14
730 0
3475 :: 26
3692 1 44
5907 18
W2-E
1518 6-36
2437 13
2174 + 19
3303 + 25
2056 + 8
4213 13
855 + 27
3361 + 50
3658 + 56
5844 +
KL-E
1502 18
2407
2215
3171
28
2611 12
4584 18
955 16
3437 2:
3801
5430 16
The proof is based on the multiplicative version of the Minnkwwii inequality and can be found in Appendix [B.T Intuitively, this implies that the Frobenius and Wasserstein projections act more aggressively, e.e they rather yield a higher entropy, while the KL projection acts more conservatively, e., it rather yields smaller entropy. This could also explain why many KL based trust region methods lose entropy too quickly and converge prematurely. By introducing an explicit entropy control, those effects can be mitigated.
4.4 SUCCESSIVE POLICY UPDATES
The above projections can directly be implemented for training the current policy. Note, however, that each epoch the policy я, predicted by the network before the projection layer does not respect the constraints and thus relies on calling this layer. The policy of the projection layer 7; not only depends would result in an ever- goo--gronigg stack of policy networks becoming increasingly costly to evaluate. In other words, f, is computed using all stored networks of Ti Ti-1...... Tu- We now discuss the parametrization of स via amortized optimization.
the parameters of x; but also on the old policy network Ti,old #;-1- This
We need to encode the information of the projection layer into the parameters @ of the next policy, L.c. {aa s; 0) = po ( (als) is a composition function in which p denotes the projection layer. The output of Th is (μ,, Σ), and p computes (ji, Σ) according Equations 6 7 or 11 Formally, we aim to find set of parameters 0 = arg ming Eg-peas [d (-(ss),, To( (- I) where Pr.i is the state distribution of the old policy and d is the similarity measure used for the projection, such that minimize the expected distance or divergence between the projection and the current policy prediction.
The most intuitive way to solve this problem is to use the existing samples for additional regression steps after the policy optimization. Still, this adds a computational overhead. Therefore, we propose to concurrently optimize both objectives during training by penalizing the main objective, e...
((a [c]s; 0)
arg min E (s.a))=To
A saan (s,
a)
aBa--Pa [d ( 0), T=(-1s))
(12)
au (a|s)
Note that the importance sampling ratio is computed based on a Gaussian distribution generated by the trust region layer and not directly from the network output. Furthermore, the gradient for the regression penalty does not flow through the projection, it is solely acting as supervised learning signal. As appropriate similarity measures d for the penalty, we resort to the measures used in each projection. For a detailed algorithmic view see Appendix A
5 EXPERIMENTS
Mujoco Benchmarks We evaluate the performance of our trust region layers regarding sample complexity and final reward in comparison to PAPI and PPO on the OpenAl gym benchmark suite [Brockman et al.) 2016). We explicitly did not include TRPO in the evaluation, as |Engstro et al.) 2020) showed that it can can achieve similar performance to PPO. For our experiments, the PAPI projection and its conservative PPO version are executed in the setting sent to us by the author. The hyperparameters for all three projections and PPO have been selected with Optuna (Akiba et al., 2019). See Appendix D for a full listing of all hyperparameters. We use a shared set of hyper- parameters for all environments except for the Humanoid, which optimized separately. Next
7

---

Published as a conference paper at ICLR 2021
Amt V2
Astert- W2
RescherSpane-11
0.1
0,1
FROB
01688
W71
PAPI
0.00
PRO
FRO-M
Awalt (
100
Regiession
Rewoud
0.05
0.00
-00
FRUB
W2
KL
PAPI
PPO M
PPO
0.5
1.5
2.5
3
01
0.5
1.5
2.5
3
Sumples
-100
Sumples
110
Samples
Figure 2: (Left): Mean KL divergence for Ant-v2 standardizing measure to compare the policy changes among all methods. (Center): Mahalanobis distance between the mean values of the unpro- jected and old policy when using different in comparison to full regression. The mean bound for the W2 projection is set to 0.03 (dotted black line). (Right): Mean cumulative reward with 95% confidence interval based on 40 seeds for the semi-sparse 5-link Reacher task. For each method, besides PAPI, we train policies with (dashed) and without (solid) contextual covariances.
to the standard PPO implementation with all code-level optimizations we further evaluate PPO-M, which only leverages the core PPO algorithm. Our projections and PPO-M solely use the observa- tion normalization, network architecture, and initialization from the original PPO implementation. All algorithms parametrize the covariance as a non-contextual diagonal matrix. We refer to the Frobenius projection as FROB, the Wasserstein projection as W2, and the KL projection as KL.
Table D gives an overview of the final performance and convergence speed on the Mujoco bench- marks, Figure 4 in the appendix displays the full learning curves. After each epoch, we evaluate five episodes without applying exploration noise obtain the return values. Note that we initially do not include the entropy projection to provide a fair comparison to PPO. The results show that our trust region layers are able to perform similarly better than PPO and PAPI across all tasks. While the performance on the Hopper-v2 is comparable, the projections significantly outperform all baselines on the HalfCheetah-v2. The KL projection even demonstrates the best performance on the remaining three environments, Besides that, the experiments present a relatively balanced performance between projections, PPO, and PAPI. The differences are more apparent when compar- ing the projections to PPO-M. which uses the same implementation details as our projections. The asymptotic performance of PPO-M is on par for the Humanoid-v2, -22, but convergences much slower and is noticeably weaker on the remaining tasks. Consequently, the approximate trust region of PPO alone is not sufficient for good performance, only paired with certain implementation choices. Still, the original PPO cannot fully replace a mathematically sound trust region as ours, although it does not exhibit a strong performance difference. For this, Figure 2 visualizes the mean KL divergence at the end of each epoch for all methods. Despite the fact that neither W2 nor Frobenius projection use the KL, we leverage it here as standardizing measure to compare the change in the policy distributions. All projections are characterized by an almost constant change, whereas for PPO-M the changes are highly inconsistent. The code-level optimizations of PPO can mitigate this some extend but cannot properly enforce the desired constant change in the policy distribution. In partic- ular, we have found that primarily the learning rate decay contributes the relatively good behavior of PPO. Albeit, PAPI provides a similar principled trust region projection as we do, it still has some inconsistency by approaching the bound iteratively.
Entropy Control To demonstrate the effect of combining our projections with entropy control, as described in Section 4.2) we evaluate all Mujoco tasks again for this extended setting. The target entropy in each iteration i is computed by exponentially decaying the initial entropy Ho to k with temperature as K (Ho x)7 for where N is the total number of training steps. The bottom of Table [ the results for our projections with entropy control. Especially on the more complex tasks with more exploration, all three projections significantly benefit from the entropy control. Their asymptotic performance for the HalfCheetah--2, Ant-v2, and Humanoid-v2 increases and yields much faster convergence in the latter. For the other Mujoco tasks the performance remains largely constant since the complexity of these tasks is insufficient to benefit from an explicit entropy control, also noted by Pajarinen et al. (2019) and Abdolmalcki et al. (0155).
8

---

Published as a conference paper at ICLR 2021
Contextual Covariances. To emphasize the advantage of state-wise trust regions we consider the case of policies with state-dependent covariances. Existing methods, such as PPO and TRPO, are rarely used in this setting. In addition, PAPI cannot project the covariance in the contextual case. Further, Andrychowicz et al. (2200) demonstrated that for the standard Mujoco benchmarks, con- textual covariances are not beneficial in on-policy setting. Therefore, a task motivated from optimal control which benefits from a contextual covariance. We extend the Mujoco Reacher-v2 to 5-link planar robot, the distance penalty to the target is only provided in the last time step, t = 200, and the observation space also contains the current time step t. This semi- sparse reward specification imposes significantly harder exploration problem the agent is only provided with a feedback at the last time step. We again tuned all hyperparameters using Optuna Akiba et al. 2019) and did not include the entropy projection. All feasible approaches are compared with and without contextual covariances, the results therefor are presented in Figure (2 (right). All three projections significantly outperform the baseline methods with the non-contextual covariance. Additionally, both the W2 and KL projection improve their results in the contextual case. In con- trast, all baselines decrease in performance and are not able to leverage the advantage of contextual information. This poor performance mainly originates from incorrect exploitation. PPO reduces the covariance too quickly, whereas PAPI reduces it too slowly, leading to suboptimal performance for both. The Frobenius projection, however, does not benefit from contextual covariances either, since numerical instabilities arise from too small covariance values close to convergence. Those issues can be mitigated using a smaller covariance bound, but they cannot be entirely avoided. The KL projection, while yielding the best results throughout all experiments, relies numerical op- timization. Generally, this is computationally expensive, however, by leveraging an efficient C++ implementation this problem can be negated (see Appendix [..3.. As a bonus, the KL projection has all properties of existing Kl.-based trust region methods that have monotonic improvement guar- antees. Nevertheless, for quick benchmarks, the W2 is preferred, given it is slightly less prone to hyperparameter choices and does not require a dedicated custom implementation.
choose to evaluate on
Trust Region Regression Loss. Lastly, we investigate the main approximation of our approach, the trust region regression loss (Equation [2). In the following ablation, we evaluate how different choices of the regression weight affect the constraint satisfaction. Figure 2 (center) shows the Mahalanobis distance between the unprojected and the old policy means for different a values. In addition, for one run choose each epoch for several iterations. One key observation is that decreasing the penalty up to a certain threshold leads to larger changes in the policy and pushes the mean closer to its maximum bound. Intuitively, this can be explained by the construction of the bound. As the penalty is added only to the loss when the bound is violated, larger changes in the policy are punished while smaller steps do not directly affect the loss negatively. By selecting a larger a, this behavior is reinforced. Furthermore, can see that some smaller values of yield a behavior which is similar to the full regression setting. Consequently, it is justified to use a computationally simpler penalty instead of performing a full regression after each epoch.
0 and execute the trust region regression separately after
6 DISCUSSION AND FUTURE WORK
In this work we proposed differentiable projection layers to enforce trust region constraints for Gaussian policies in deep reinforcement learning. While being more stable than existing methods, they also offer the benefit of imposing the constraints on a state level. Unlike previous approaches that only constrain the expected change between successive policies and for whom monotonic im- provement guarantees thus only hold approximately, we can constrain the maximum change. Our results illustrate that trust regions similarity measures. Apart from the commonly used reverse KI,, we also leverage the Wasserstein distance and Frobenius norm. We demonstrated the subtle but important differences between those three different types of trust regions and showed our benchmark performance is on par or better than existing methods that use more code-level optimizations. For future work, plan to continue our research with more exploration-be -havy environments, in particular with contextual covariances. Ad- ditionally, more sophisticated heuristics or learning methods could be used to adapt the trust region bounds for better performance. Lastly, we are interested in using our trust region layers for other deep reinforcement learning approaches, such actor-critic methods.
effective tool in policy search for wide range of different

---

Published as a conference paper at ICLR 2021
REFERENCES
A. Abdolmaleki, R. Lioutikov, Peters, N. Lau, L. Reis, and G. Neumann. Model-based rel- ative entropy stochastic search. In Advances in Neural Information Processing Systems 28, pp. 3537-3545, 2015. URL http: //tw://wwwww.... ausy. tu darmstadt httpi//ddeeppaaddseeeeeeeee GerharddNeemann//ABBBo Iwa.eoooollleee NIPS2015 pdf
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posterioni policy optimisation. In International Conference on Learn- ing Representations, 2018. URL ht URtt /tppttppppp///pppiieeeeeeee
www.m/orrmmiooooo S1NNXQOUU
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable Convex Optimization Layers. Advances in Neural Information Processing Systems, 32, oct 2019.
http://aaxxxxv../aaa.oaoa/ http:///ggaaaaaa/a//a// 12430
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2623-2631, 2019. URL
http::/ ww..org//a. / abs////////// 1907. 10902
Riad Akrour, Joni Pajarinen, Jan Peters, and Gerhard Neumann. Projections for approximate policy iteration algorithms. In Proceedings of Machine Learning Research, pp. 181-190, 2019.
Brandon Amos and J. Zico Kolter. OptNet: Differentiable Optimization as a Layer in Neural Networks. In 34th International Conference Machine Learning, pp. 179-191, 2017. URL
http://arxiv.ore wtt.xxiiworg/bb//7073333
abs 11tp//1470004433
Marcin Andrychowiez. Anton Raichuk, Piotr Stañczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What Matters In On-Policy Reinforcement Learning? A Lcale-Scale Em- 2006 (5990)
pirical Study. In arXiv preprint, 2020.
Oleg Arenz, Mingjun Zhong. and Gerhard Neumann. Efficient Gradient-Free Variational Inference using Policy Search. In Proceedings of Machine Learning Research, pp. 234-243, 2018. URL
http::/ ngs ml press vv00 rren .t...am....mm.mmmmm htmll
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and org. /bs/
Wojciech Zaremba. OpenAl Gym. In arXiv preprint, 2016. URL
1666601544
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavamzadeh, and Edgar Duenez- Guzman. -yapunov-based Safe Policy Optimization for Continuous Control, In ICML Workshop RL4RealLife Submission, 2019.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yu- val Tassa. Safe Exploration in Continuous Action Spaces. In arXiv preprint, 2018. ISBN 1801.08757v1.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Reinforcement Learning for Continuous Control. 33rd International Conference on Machine Learning, pp. 2001-2014, 2016. URL/http:// //tp://aaavvvv/.a http://wxxw../////////// http//bb///11117////
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras. Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. International Conference on Learning Representations, 2020. URL
http:://arii: org www./rt.//////oo/////..//e..eeet /aab:22055.om 12729
Sham Kakade. A natural policy gradient. In Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS'01, pp. 1531-1538, Cambridge, MA, USA, 2001. MIT Press.
Sham M. Kakade and John C. Langford. Approximately Optimal Approximate Reinforcement Learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp- 267-274, 2002. URL htttss/:s:/// //dd acm. wwwp.//gg///o///ii 001111555555665 645531 656005
10

---

Published as a conference paper at ICLR 2021
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. CoRR, 0oos/1805.0090) 2018. URL http: aariivivv.......v org ha/pa/bs/1110. 00909
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-End Training of Deep Visuomotor Policies. In The Journal of Machine Learning Research, volume 17, pp. 1334-1373,
2015. URL /ttt:://aaat www.v.xxvvv....... org/ abs / 1504. C0702
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learn- ing. Nature, 518(7540):529-5335 2015. URL http:/ http:://wwwww. http://www..tt...eeeee.ee. nature........ com/ artmm/ss/tssss//// nature14236.
Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Anna Choromanska, Krzysztof Choroman- ski, and Michael I Jordan. Learning to Score Behaviors for Guided Policy Optimization. In Proceedings of the International Conference on Machine Learning, 2020. URL hetps:
// //arxiv.org abs 906.906600 04349
Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, and Gerhard Neumann. Compatible Natural Gradient Policy Search. Machine Learning, 10868-9);1443-146 2019. URL http:
/arxii.org/abb//.. 02220
Peters, Muelling, and Y. Altun. Relative entropy policy search. In Proceedings of the Twenty- Fourth National Conference on Artificial Intelligence (AAAI), Physically Grounded Al Track,
2010. URL http:://ww ias. informatik. uuddarnstatt, de/uploads Team/ SEPS.pdi.
JanFeters/Feters20..
Jan Peters and Stefan Schaal.
Reinforcement learning of motor skills with policy gradi- ents. Neural Networks, 21(4):682-697, 2008, URL httpps://ww www.:wwww.i.....a............... informatik
tu
K. B. Petersen and M. Pedersen. The matrix cookbook, 2012. UtL.htp::/wwwett. htttp::/www2.ccomptte. compute. dtu. http/pubdd.ppbbs//2222
Version 20121115.
Pierre H. Richemond and Brendan Maginnis. On Wasserstein Reinforcement Learning and the
Fokker-Planck equation. In arXiv preprint, 2017. URL httt:://axxiv..rr/abbb....
07185]
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. Proceedings of Machine Learning Reseurch, pp. 1889-1897, 2015a. URL
http:://prcceeaingg .wl. wtw.l..pes///3///eeee http////333/cuumaa13333 1ttmmmmmmnnm.mmm11...mm
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. In International Conference on Learning Representations, 2015b. URL http:://axixv org/ hagg/abs/1500
02438
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. In arXiv preprint. 2017. URL
06347
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering Chess and Shogi by Self-Play with a Gen- eral Reinforcement Learning Algorithm. In arXiv preprint, pp. 1-19, 2017. URL http:
7/ //arxiv.orr//as//1111 01815
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu. Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Ried- miller, and Matthew M. Botvinick. V-mpo: On-policy maximum posteriori policy optimization for discrete and continuous control. In International Conference on Learning Representations, 2020. URL httttss://pppereriiiii wtt.e//wwwwweeeeeeeeeeeeeeeeeeee htt.

---

Published as a conference paper at ICLR 2021
Jun Song and Chaoyue Zhao. Optimistic Distributionally Robust Policy Optimization. In arXiv preprint, pp. 15872-15882, 2020. [Rttpstts///att arxiv. htt.vvv.rggabb/2005555 07815
Asuka Takatsu. Wasserstein geometry of Gaussian measures. 1005 1026, 2011.
Osaka Journal of Mathematics, 48:
Voot Tangkaratt, Abbas Abdolmaleki, and Masashi Sugiyama. Guide actor-critic for continuous control. In Proceedings of the International Conference on Learning Representations, 2018.
URL /ttLittts///ccxi.oorgg wtwpg///////.......//./// 07606v2
Cédric Villami. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Yuhui Wang. Hao He, Xiaoyang Tan, and Yaozhong Gan. Trust region- guided proximal policy optimization. In Advances in Neural Information Processing Systems 32, pp. 626 636, 2019. URL /nttps://arxile org/ abs / 1901. 10314
Tsung-Yen Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-Based Constrained Policy Optimization. In International Conference on Learning Representations, 2020. htttt://s//// http:/sttts..oolllllee/eooooooee httpe//ccoo/eeeeeecc1000000----0
12

---

Published as a conference paper at ICLR 2021
A ALGORITHM
Algorithm Differentiable Trust Region Layer. The trust region layer acts as final layer after predicting Gaussian distribution. It projects this predicted Gaussian onto the trust region in case it violating the specified bounds. As output generates a projected mean and covariance that satisfy the respective trust region bound. The entropy control in the last step can be disabled.
Initialize bounds (uu <E, temperature 7 as well as target &s and initial entropy Ho.
1: procedure TRUSTREGIONLAVER(4. Σ, Hold. Eola) if diwcan (H,, Mala) > e then Compute j with Equation
2:
3:
4: else
5:
Ā== H if deov (,, Dola) >> ex, then
6:
7:
Compute Σ with Equations or 11
8:
else
9:
Σ =
== K (Ho x)T 101
(Optional) entropy control as described in Section 4.2
11:
if H=)) < 3 then
c exp {(B H(E)) /dim(a))
= return ju,
14:
Algorithm Algorithmic view the proposed Trust Region Projections. The trust region projec- tions itself do not require approximations, the old policy update in the last step is the only point where we introduce an approximation. This update would normally require additional supervised regression steps that minimize the distance between the network output and the projection. How- ever, by leveraging the regression penalty during policy optimization this optimization step can be omitted. Both approaches yield a policy, which is independent of the old policy distribution, i. e. can act without the projection while maintaining the trust region. However, the penalty does not require additional computation and the policy can directly generate new trajectories, equivalently to other trust region methods, such as PPO.
1: Initialize policy 0,.0
2: for i = 0, I,..., do
D epoch
3: Collect set of trajectories D; = {Tk} with ((0.o)
4: Compute advantage estimates A, with GAE
5: for j = 0, 1...,,,..... M do
6:
Use T (0s.g) predict Gaussian action distributions W(Js. Σis) for Dj
7:
= TRUST REGIONLAYER(H,,, Eijj-Pin Pi,0, Σi,))
8:
Update policy with Adam using the following policy gradient:
0i.j11 < Adam
Va (0,,))))
#[a|s;0)
Ag
aB aBs-pre...) (dd [d (), x-- x-1s:
loa..)
T(a|s;0..0)
9: Successive policy update: 0,+1,0 t- 0.,M
B DERIVATIONS
B. PROOF OF THEOREM
This section provides proof for Theorem 1. We mainly used the multiplicative version of the Brunn- Mink--Minkowski inequality
log(a)Ei1+ + > log((2i))* )
where E,, Eg are p.s.d, a, 3 positive, and 1.
13

---

Published as a conference paper at ICLR 2021
Frobenius Projection
H(
0.5 log 2
1 E
0.5 log
2e
77 Dold
11
0.5 0.5
(2meI) कोΙ det (2Te पों
1
0.5 log [ + 0.5 log 2
1/ + 1
17 H(mO) + 1 H( minimum (H(mo), H(To.)))
17 7
Wasserstein Projection
Let & denote the dimensionality of the distributions under consideration.
H(a) = 0.5 (og((2e))
l.g
(
0.5
77
0.3
2
n+
7+1
0.5 log
000
+
7
oold
+ log
50.5
"
old
+
17 +
77 +
+
(2T log
000
n+
2.5
-
+ log
0.5
n-
1old
0.5log(2Te)) log
[>]
Eold
log (2TE22) गमं
22re VFT
"
1) 1 0.5 log [2 +
0.5 log Eotll
n
+ H(TT) + 7
"7
H(TTe)) 2 minimum (H(T), H(TToa))
KL Projection
H(+)= 0.5 log 2
= 0.5 log
(2TeE)
-1
+
77
(2Te
1) 7
= 0.5 log
7 1 (2
1
+ "
(2 Eold
S -0.5 log
(1(TTe))
पेन (2
lo5
[2iTe\]]
n-1
12Te2Dold n711
(use the fact that: det(A-1) 1/ 1/det(A))
0.5 log 2TT + + 0.5 log Lata
0.5
+ 1
"7 + H(TO) + 17 H((Ta)) S maximum (H(TT), H(Taa))
1
17
B.2 MEAN PROJECTION
First, we consider only the mean objective
min
(μ- (i) Eodd (μ (i)
(Hols - pi) Iord (Holl - (i) S Cu.
s.t.
which give us the following dual
((j,w) (μ i End (μ (1) + w (Hood :T) Endd (Hold ()) -e))
(13)
14

---

Published as a conference paper at ICLR 2021
Differentiating w.r.t. yields
aC(p, w) 25n (it 10) 2wIirld (i - Mold).
Setting the derivative to and solving for it gives
1 + WHold
1 + w results in
Inserting the optimal mean fi in Equation
T
14 + wpold
-
11 + W/Hold
L(w) -
-1
Dold
11
1 + w
+ w
11 + wifoldd Mold
(A + WpHold
+ w
Dold
Hold cx
+
1 +
w² (p Mold) Endd (u Hold)
w (1s pota) T Endd (1r Mota) WE (1 + w)2
(1+
w)2 ww)
Thus, differentiating w.r.t w yields
al(w)
(μ Pola) T Dold (H - Mold) Ex.
dw
(1 w)2
Now solving (C(w) 1 0 for w, we arrive at
Wold) )z 2nd (1r (old)
S/
B.3 FROBENIUS COVARIANCE PROJECTION
We consider the following objective for the covariance part
min Σ
$ Σ F
[ Eoit 1 |r SCC
s.t.
with the corresponding Lagrangian
c(Σ.n)= > + (|E- Eoid F -cx
(14)
Differentiating w.r.t Σ yields
ac( 1)
2(E 17 (Eold Σ))
We again solve for Σ by setting the derivative 0, i.e.
$' Σ Eoodd
1 +
Inserting Σ into Equation yields the dual function
Σ + Eotd
(1
Σ + Entd
g(n) =
Σ +n
Dold
- -
1 ++
1 7
F
Differentiating w.r.t. 17 results in
Oc(m) Om
Σ Eold [
Ex.
(1 + 17)
ac(v) 1 0 yields
Hence, On
Σ Eotal | 1.
17"
15

---

Published as a conference paper at ICLR 2021
B.4 WASSERSTHIN COVARIANCE PROJECTION
As described the main text, the Gaussian distributions to have been rescaled by Eold to measure the distance in the metric space that is defined by the variance of the data. For notational simplicity, we show the derivation of the covariance projection only for the unscaled scenario. The scaled version be obtained by simple redefinition of the covariance matrices. For covariance projection we are interested in solving the following optimization problem
min tr Σ - - (2% Σ
tr Σ + Voldd 2 (2: 1/2
s.t.
I Ex,
which leads to the following Lagrangian function
E E -2 - 1/2
((Σ,1) = tr
E Eols -2 " old + tr
1/3) c
(15)
Assuming that Σ commutes with Σ as well as Eois. Equation 15 simplifies to
c(E.n) = tr -
1/2 -c) tr [ + Vold -252//2>
tr 52 - 28I + (tr (s Eou 25 -c),
(16)
where S is the unique positive semi-definite root of the positive semi-definite matrix E, i.e. S = 5222, Instead of optimizing the objective w.r.t E, we optimize w.r.t S in order, which greatly simplifies the caleulation. That is, we solve
Ac(S n) as
(1+1)) 1)25 - 2 2 + 1 1
for S, which leads to
/// + 1 old
1/2
+ 2 Eold + 2//5. o
S
Σ.
1 1+1
(1 + 17)
Inserting this into Equation
16 yields the dual function
g(n) tr E + Eold - /2'42""/ 1/22
2dd
1+ 1
The derivative of the dual w.r.t. 77 is given by
tr (E+ Eold - 2///3"//3
ac(n)
old
On
2)2
Now solving
aC(m) On
1 for 77, we arrive at
(r
E + Eold
2//55'///
Ex
B.5 KI--DIERGEECE PROJECTION
We derive the KI.-Divergence projection its general form, i.e., simultaneous projection of mean and covariance under an additional entropy constraint
म arg min KL (fr mo) KL (f Sc H H(#) 2 3.
16

---

account for the Published as a conference paper at ICLR 2021
Instead of working with this minimization problem consider the equivalent maximization prob- lem
arg max KL ())) s.t. (L S,
H iT 2 3
(17)
which is similar to the one considered in Model Based Relative Entropy Stochastic Search (MORE) (Abdolmaleki al. 2015), with few distinctions. To see those distinctions let 1) and w denote the Lagrangian multipliers corresponding to the KL. and entropy constraint respectively and consider the Lagrangian corresponding to the optimization problem in Equation 17
-F + n) (E (T + w (H( - 3)
= Ea [log mo] + +1 (I.(# mo..)) + (w + ()) wf.
Opposed to Abdolmaleki et al. 2015) we are not working with an unknown reward but using the log density of the target distribution instead. Thus we do not need to fit a surrogate and can directly read off the parameters of the squared reward. They are given by the natural parameters of æ,, i.e, A --1 and q = > p, Additionally, we need to add constant
additional entropy term in the original objective, similar to (Arenz et al. 2018
Following the derivations from Abdolmaleki al. (2015) and Arenz et al. (2018) we can obtain closed form solution for the natural parameters of v. given the Lagrangian multipliers and w
AaA + A 7 1+
1/ooM +
and i
(18)
17 + 1++
To obtain the optimal Lagrangian multipliers we can solve the following convex dual function using gradient descent
1
1
g(n, w) =MC wB + 19
Aold 9old + log dot (A) log((T))
2
2
+ +n+ + 1 ww
2
1
q
2
1
log det A log(27)
k
+ const,
0g(n.w) w)
and Ög(n, w)
( (Bau)
11 (T) 3.
On
Given the optimal Lagrangian multipliers, 17" and wt we obtain the parameters of the optimal distri- bution H* using Equation 18
Forward Pass For the forward pass compute the natural parameters of n, solve the optimiza- tion problem and compute mean and covariance of # from the optimal natural parameters. The corresponding compute graph is given in Figure
Analytical Computation
Numerical Optimization
17 Gold + q q"
μ
q 1=
1
^
g
11 w"
Ä 17" Aold + A
Σ
A A=E-1
w
D== = \+')
17" + w" +1
Figure 3: Compute graph of the KL projection layer. The layer first computes the natural parameters of π from the mean and covariance. Then it numerically optimizes the dual to obtain the optimal Lagrangian multipliers which are used get the optimal natural parameters. Ultimately, the optimal mean and covariance are computed from the optimal natural parameters. We omit the dependency on constants, i.e., the bound and as well the parameters of nold for clarity of the visualization.
17

---

Published as a conference paper at ICLR 2021
Backward Pass Given the computational graph in Figure gradients can be propagated back though the layer using standard back pack-propagation. All gradients for the analytical computations (black arrows in Figure }) are straight forward and can be found in (Petersen & Pedersen 2012). For the gradients of the numerical optimization of the dual (red arrows in Figure 3) we follow Amos & Kolter (2017) and differentiate the KKT conditions around the optimal Lagrangian multipliers computed during the forward pass. The KKT Conditions of the dual are given by
KL (f (T - mm m)) = 0, (Stationarity)
Vg(nn" ,w") + m m1(-7²) 0 and m2 m2(-w") (Complementary Slackness)
H - m2
here m = (m, m2)" denotes the Lagrangian multipliers for the box constraints of the dual (17 and need be non-negative). Taking the differentials of those conditions yields the equation system
ØKL (# (
0KL (f* (
On n
Ows OH(#*)
drj
0H(
du
Ory"
Ou*
dm
-m71
0
dm2/
-
ØKL (#* [T=[|rra
0KI.
(#*
(T
dg
dA
0q
0A
OH(#*
0H(#*)
dg
dA
0g
0A
0
which is (analytically) solved obtain the desired partial derivatives
On On
dw
Ow
and
0q Ox 0g
0A
Implementation We implemented the whole layer using C++, Armadillo, and OpenMP for par- allelization. The implementation saves all necessary quantities for the backward pass and thus numerical optimization only necessary during the forward pass. Before we perform a numerical optimization we check whether it is actually necessary. If the target distribution ã is within the trust region we immediately can set म* Tg. i.e., the forward and backward pass become the identity mapping. This check yield significant speed-ups, especially in early iterations, if the target is still close to the old distribution. convex dual, which is still fast. For example, for a 17-dimensional action space and batch size of 512, such as the Humanoid-v2 experiments, the layer takes roughly 170ms for the forward pass and 3.5ms for the backward pass if the all 512 Gaussians are actually projected If none of the Gaussians needs to be projected its less than Ims for forward and backward pass.
the projection necessary we use the L-BFGS to optimize the 2D
Simplifications If only diagonal covariances considered the implementation simplifies sig- nificantly, as computationally heavy operations (matrix inversions and cholesky decompositions) simplify to pointwise operations (divisions and square roots). If only the covariance part of the KL projected, we set Hold = f = jt* and du = 0 which is again a simplification for both the deriva- tions and implementation. If entropy equality constraint, instead of an inequality constraint, it is sufficient to remove the >> 0 constraint in the dual optimization.
2On & Core Intel Core 17-9700K CPU @ 3.60GHz
18

---

Published as a conference paper at ICLR 2021
C ADDITIONAL RESULTS
Figure shows the training curves for all Mujoco environments with a 95% confidence interval. Besides the projections also show the performance for PAPI and PPO. In Figure the projections also leverage the Entropy control based on the results from from Akrour et al. 2019
Hupper-22
Walker20-22
HalfChershh
5,000
2,500)
4,000
2,000
Bexenl
1,500
2,000
Rewand
2,000
1,000
1.000
1.000
PI0-M
1913
0.11
0.6
Sumples
Sample
-106
Samples
-106
Ant-2
Himonod-22
6,000
3,000
4,000
2,000
2,000
1,000
11
0.5
1.5
2.5
01.2
0.6
0.6
10"
Sumples
107
Figure 4: Training curves for the projection layer as well PPO and PAPI on the test environment. We trained 40 agents with different seeds for each environment using five evaluation episodes for every data point. The plot shows the total mean reward with 95% confidence interval.
x2
HalfCheeshh
2,500
5,000
3,000
2,000
1,000
2,000
Rowall
2,0000
1,000
1.000
1,000
PPO-M
Sumplo
Samplee
-10P
Sanplees
-10
Humeenid--22
Ant-2
4,000)
6,000
3,000
4,000
2,000
2,000
0
0.5
1.5
2.5
0
0.2
0.4
0.5
0.5
Samples
Samples
-107
Figure 5: Training curves for the projection layer with entropy control (-E) as well as PPO and PAPI on the test environment. We trained 40 agents with different seeds for each environment using five evaluation episodes for every data point. The plot shows the total mean reward with 95% confidence interval.
19

---

Published as a conference paper at ICLR 2021
D HYPERPARAMETERS
Tables and show the hyperparameters used for the experiments in Table [
Target entropy.
temperature, and entropy equality are only required when the entropy projection is included in the layer, otherwise those values are ignored.
Table 2: Hyperparameters for all three projections as well as PAPI, PPO. and PPO-M on the Mujoco benchmarks from Table
Frobenius
W2
KL
PAPI
PPO
PPO-M
rollouts
2048
GAE
0.95
discount factor
0.99
0.03
0.015
n.a.
ex: target entropy
0.001
n.a.
temperature
0.5
entropy equality
False
False
n.a.
optimizer epochs vf epochs Ir Ir minibatch size trust region loss weight a entropy loss penalty
adam 10
20
10
5e-5
3e-4
4.5e-4
2.5e-4
32
64
8.0
n.a.
0
normalized observations normalized rewards observation clip reward clip vf clip importance ratio clip
True
False n.a.
True
False
10
n.a.
10
n.a.
0.2
n.a.
0.2
contextual std hidden layers hidden layers vf hidden activation
False
[64, 64]
[64, 64]
tanh
20

---

Published as a conference paper at ICLR 2021
Table 3: Hyperparameters for all three projection well PAPI and PPO the Humanoid-v2
from Table
Frobenius
W2
KI.
PAPI
PPO
PPO-M
rollouts
16384
GAE X
0.95
discount factor
0.99
e
0.05
0.015
n.a.
le-4
0.01
le-4
n.a.
target entropy temperature
n.a.
0.2
n.a.
entropy equality
True
False
n.a.
optimizer epochs vf epochs
adam
10
10
Ir
le-4
le-4
Ir vf minibatch size
4.5e-4
le-4
256
512
entropy loss penalty trust region loss weight a
0
8.0
n.a.
normalized observations normalized rewards observation clip reward clip
True
False
True 10
False
n.a.
n.a.
10
n.a.
vf clip importance ratio clip
n.a.
0.2
n.a.
0.2
contextual std hidden layers hidden layers vf hidden activation
False
[64, 64]
[64, 64]
tanh
21

---

Published as a conference paper at ICLR 2021
Table Hyperparameters for three projection as well as PAPI and PPO on our ReacherSparse- vO task from Figure 2 contextual covariance.
The second value for eg of the KI. projection is the bound when using
Frobenius
W2
KI.
PAPI
PPO
PPO-M
rollouts
16384
GAE À discount factor
0.95
0.99
c
0.03
0.03
eg
5e-5
1c-3
5e-5/1e-3
n.a.
target entropy temperature
n.a.
n.a.
n.a.
entropy equality
False
optimizer
adam
epochs
10
epochs
20
3e-4
le-4
Ir vf minibatch size entropy loss penalty trust region penalty a
4.5c-4
le-4
256
512
0
8.0
n.a.
normalized observations normalized rewards observation clip reward clip
True
False
True 10
False
n.a.
n.a.
10
n.a.
vf clip importance ratio clip
n.a.
0.2
n.a.
0.2
contextual std hidden layers hidden layers vf hidden activation
False
[64, 64]
[64, 64]
tanh
22


Reviews:
[{"review_id": "p0pC5D-Txk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nIn trust-region-based policy optimization methods such as TRPO and PPO, it is difficult to tune and lots of approximations are required. The authors try to solve this issue by introducing the closed-form derivation of trust regions for Gaussian policies with three different types of divergence (or distance). Based on the theoretical derivation, the differentiable layer is proposed, where the layer is built upon “old” policy during the trust-region-based policy updates. The difference comes from the use of various divergences (or distances) are given in theoretical and empirical ways. \n\n### Quality\nThe proposed idea seems interesting and theoretical derivations seem mostly sound but empirical performance doesn’t support the authors’ claim, which makes me highly decrease the score. \n\n### Clarity\n- The Introduction is written mostly well although some sentences are too specific to be understandable at the first glance. \n- The Related Work is well-summarized and emphasizes the difference and advantages clearly. \n- Section 4 (which is about the main ideas) needs to be more clarified and reorganized for some parts. \n- The use of entropy projection is quite abrupt and I couldn’t understand its clear motivation. \n- Successive policy updates seem to improve the methods in a way that the mean is well-bounded during updates, but its explanation and justification are difficult to understand. \n- There are more comments in `Detalied Comments`\n\n### Originality\n- I really enjoyed reading projections of mean and covariance (~4.2) and ideas are somewhat novel. However, empirical results don’t support the authors’ claim in the sense that the differentiable trust-region layer doesn’t improve the performance significantly. \n\n### Significance\n- The idea is interesting, but its significance is low due to the empirical performances as well as loosely tuned baseline (PPO).\n- Additionally, some motivations/derivations/links among equations are unclear.\n\n### Detailed comments\n(p.1, Abstract) code-level optimization\n- I was curious about the definition at the first glance. \n\n(p.1, Introduction) our method comes with the benefit of imposing the constraints on\nthe level of individual states, allowing for the possibility of context dependent trust regions.\n- I couldn’t understand what this means at the first glance. \n\n(p.1, Introduction) Considering the trust region layers require\n- Considering the trust region layers requires\n\n(p.1, Introduction) the new policy without trust layers has to stay close to the output of the projection layer.\n- I don’t think such a specific methodology needs to appear in the introduction. \n\n(p.2, Related Work) However, Engstrom et al. (2020) and Andrychowicz et al. (2020) recently showed that code-level optimizations are essential for achieving state-of-the-art results with PPO\n- The meaning of code-level optimization needs to be clearer. \n\n(p.3, Related Work) They use either an exponential or linear decay of the entropy during policy optimization to control the exploration process and escape local optima. To leverage those benefits, we embed this entropy control mechanism in our differentiable trust region layers.\n- I’d rather add simple maths to describe what the previous works did for clarity. \n\n(p. 3, Preliminaries and Problem Statement) Eq. (1)\n- In the subscript of the expectation, I’d rather add either $t=1, …$ or trajectory distribution.\n- $A^\\pi$ -> $A^{\\pi_{\\mathrm{old}}}$?\n- I think using bold letters for all state-action pairs is a bit confusing. I’d use bold letter only for random variables and plain letters for non-random variables. \n\n(p.3, Preliminaries and Problem Statement) Using a constraint on the maximum KL over the states has been shown to guarantee monotonic improvement of the policy (Schulman et al., 2015a). However, as all current approaches do not use a maximum KL constraint but an expected KL constraint, thus the monotonic improvement guarantee also does not hold exactly.\n- I think this sentence should be emphasized. \n\n(p.3, Preliminaries and Problem Statement) Gaussian policies ~ as well as\n- as well as -> and?\n\n(p.3, Preliminaries and Problem Statement) the commonly used reverse KL\n- The expression `commonly used` here seems weird to me. \n\n(p.3, Preliminaries and Problem Statement) The similarity of the covariance is measured by the difference in entropy of both distributions\n- This statement seems incorrect since entropy is an averaged valued of negative log probability and KL is not exactly the difference between two entropies. \n\n(p.3, Preliminaries and Problem Statement) as it is always non-negative\n- since it satisfies the criteria of being a divergence.\n\n(p.3, Preliminaries and Problem Statement) as the distance measure is then more sensitive to the data-generating distribution.\n- If I understood correctly, the distance is defined by using the covariance of the old policy distribution (similar to the distance proposed in Dadashi et al., 2020 that only cares about diagonal covariance matrix), but how is this related to the sensitivity w.r.t. the data-generating distribution?\n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) Additionally, we extend the trust region layers to include an entropy constraint to gain control over the evolution of the policy entropy during optimization\n- I’d rather use this sentence where the formula for entropy constraint appears.\n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) The trust regions are defined by means of a distance or divergence ~\n- In my understanding, it’s not a mean over distance, but just a distance between new and old policies. \n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) Note that $\\mu$, $\\Sigma$ are state-dependent, which we will however neglect for ease of notation.\n- I’d rather keep the dependencies on states since it’s a bit confusing to understand the state dependencies of objectives (3) and (4).\n- If I understood correctly, (3) and (4) will be optimized for each $s\\in\\mathcal{S}$, and thus, we can use the solution of projection over all states the old policy can visit. \n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) The output of the trust region layer is then considered to be the new policy.\n- Since (3) and (4) are objectives, I’d rather state like “We desire the output of the trust region layer becomes the parameters of the new policy and formulate the trust region layer as follows.”\n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) As all distances or divergences used in this paper can be decomposed into a mean and a covariance dependent part\n- This may be since we don’t update the second distribution of all metrics -- therefore, $\\Sigma_2$ is fixed -- where the old policies will be plugged in. Such an explanation seems to be needed. \n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) as this gives the algorithm more flexibility\n- I understand the way the trust region will be used, but this statement is weird since joint optimization is much more general and flexible in my understanding. \n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) where $d_\\mu$ is the mean dependent part and $d_\\Sigma$ is the covariance dependent part of the employed similarity measure\n- The subscripts are confusing since $\\mu$ and $\\Sigma$ is used as input arguments of (3) and (4). $\\mu$ and $\\Sigma$ are the output parameters of Gaussian policy which seem to be fixed during the optimization of (3) and (4). This should be stated for clarity. \n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) All three trust region projections\n- “Three” indicates different distances, so it should be linked with the previous equations on distances. \n\n(p.4, Differentiable Trust-Region Layers for Gaussian Policies) By making use of the method of Lagrangian multipliers,~\n- Link Appendix A.2. for readers. \n\n(p.5, Wasserstein Projection) To find the projected covariance ~\n- I’d rather put this sentence after the sentence “However, in practice we found this approach to be numerically unstable.”\n\n(p.5, Wasserstein Projection)  For the more general case of arbitrary covariance matrices, we would need to ensure the matrices are sufficiently close together, which is effectively ensured by Equation 6.\n- I don’t fully understand what the authors intended to say. \n\n(p.5, Wasserstein Projection)  Note however, that here we chose the square root of the covariance matrix ~\n- Is there an advantage of using the square root of the covariance matrix? Also, it would be helpful if the definition of the square root of the matrix is given. \n\n(p.6, Figure 1) Entropy of the interpolated covariances\n- Covariance cannot define entropies. We should use “Entropy of the interpolated distributions”\n\n(p.6, Entropy Projection)\n- This is a bit abrupt to me since trust region w.r.t. old policy has been considered until page 5. At the first glance, I couldn’t understand why entropy projection is needed and how the scaling is related to the exploration. A more intuitive explanation is needed. \n\n(p.6, Analysis of the Projections) A paragraph with the sentence “It is instructive to compare the three projections.”\n- I’d rather use equations for a detailed explanation. \n\n(p.7, Successive Policy Updates) The above projections can directly be implemented for training the current policy. However, for successive policy updates the projections become rather impractical. Each projection would rely on calling the projection of the preceding policy.\n- I couldn’t understand this part. My best understanding was using stated projections for policy updates is impractical, but how each projection is related to preceding policy is unclear. \n\n\n\n(p.7, Successive Policy Updates) The most intuitive way to solve this problem is to use the existing samples for additional regression steps after the policy optimization. Yet, this adds a computational overhead. \n- I couldn’t understand this part. \n\n(p.7, Successive Policy Updates) Eq (10)\n- $\\tilde{\\pi}$ seems ”Differentiable trust-region layer” and needs to be mentioned. \n\n(p.7, Experiments) For our experiments, the PPO hyperparameters are based on the original publication (Schulman et al., 2017). The PAPI projection as well as its conservative PPO version are executed in the setting sent to us by the author. For our projections, all parameters are selected with Optuna (Akiba et al., 2019)\n- PPO is a baseline here but seems naively tuned.\n\n(p.7, Table 1) We trained ten agents on different seeds\n- It seems ten seeds, not ten agents?\n- PPO works much better in Hopper and Walker2d, which is different from the statement in the main context (“The results show that our trust region layers are able to perform similar or better than PPO and PAPI across all tasks”)\n\n(p.7, Figure 2) Left\n- The result doesn’t seem statistically significant. Means of proposed methods are within the confidence interval of PPO.\n\n### References\n- Dadashi et al., 2020, “Primal Wasserstein Imitation Learning”\n\n---\n### Response to Authors\nI'm satisfied with your responses, especially strengthened experimental results and the clarification of methods in the revision. As my concerns were on doubtful empirical results in the submission---although I thought the approach of the submission was sufficiently novel---I updated my score from 4 to 6 accordingly. I don't know if the authors will keep working on this direction, but I think it will be interesting to see the performance of *off-policy RL* with the proposed method. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mdNxXncSovO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors explore the use of KL, 2-Wasserstein, and Frobenius norm in order to derive trust region projections in DRL. The topic is relevant and novel - specially given the prevalence of TRPO and PPO in RL in recent years. \n\nThe paper is well-written and structured, with a nicely written related work section. I find the study very relevant and useful but somewhat incomplete. I would like the paper to better motivate\n\n- why these three (KL, Wasserstein, Frobenius) were chosen. By the way, be precise with the use of the word metric (beginning of Sec 4) when referring to KL. It would be nice to extend the same analysis for the wider families of metrics and divergences that these three are part of.\n\n- the rationale to select the right one for a given problem.\n\nI also find that the discussion on entropy control, although interesting on its own, somehow distracts from the main message of the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "jHHIW98v91c", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I’m terribly sorry but I noticed that somehow my review of this paper was not successfully submitted as I checked my submission tasks. After double checking with the area chair, we decide to add my review here. Notice that this is **only for the authors' reference and to provide some additional feedback for potential improvement of the paper in the future**, but it is **not considered as an official review in the final decision process**. Please find both the original review and the comments on the revised paper below. \n\n### [Original Review]\n\nThis paper considers trust region methods in deep reinforcement learning (DRL) and proposes differentiable trust region layers, a type of differentiable neural network layers to enforce state-wise trust regions exactly for deep Gaussian policies via closed-form projections. The proposed approach is flexible and general, and can in particular handle different trust regions (in KL, W2 and Frobenius norms, for example) and can be applied to existing RL algorithms as a complementary component. Empirical results show that the proposed trust region layers (together with entropy projection to encourage exploration) help PPO achieve similar or better results, and are less dependent on implementation/code-level optimization.\n\nIn general, the paper is relatively well-written and discusses about a novel and clean approach for solving the problem of enforcing trust region constraints in trust region DRL algorithms. However, the following issues should be noted and addressed:\n1. On page 2, the authors mention that “Additionally, the projection is not directly part of the policy optimization but applied afterwards, which can result in suboptimal policies”. But since the subproblems of TRPO/PPO are just (first-order) approximations to the original RL problem, the meaning of “suboptimal” here is unclear. \n2. On a related point, I’m wondering what would happen if the authors instead use the projection methods in this paper not as a layer but just as a post-processing step after each standard TRPO/PPO update. The authors should compare this approach with the proposed one (Section 4.4), as this is at least a natural and closely related benchmark (and may even perform better in practice, which is currently unclear without the numerical comparisons). Also, the authors mention that “for successive policy updates the projections become rather impractical”. However, it is not clear to me why it is impractical to use the projection as a post-processing step as mentioned above. \n3. Again, on a related point, in Section 4.4, it is unclear which policy is eventually adopted in execution. Is it the $\\pi_{\\theta}$ (before projection) or the projected policy $\\tilde{\\pi}$?\n4. Why is it that important to enforce the trust regions exactly? It seems that the major reason provided in this work for focusing on this problem is that exact trust region constraints will lead to some performance improvement with less code-level optimization. However, in general, the empirical performance improvement shown in this paper is not very significant. In fact, for Table 1, I don’t think the two criteria (“first” and “last”) are informative enough to characterize the overall performance, as it seems that the curves are crossing each other very frequently (in Figures 2 and 4), and so it would make a lot of difference to consider the last 20 epochs, 10 epochs, or just 5 epochs and so on. So it might be better to directly look at the curves. However, from Figures 2 and 4, it seems that with only the trust region layers, the performance improvement is not very obvious. It is only with the additional entropy projection that the performance becomes obviously better (Figure 2). Hence I think the authors should also include comparisons with the standard PPO/TRPO + entropy projection. Otherwise, it is not clear whether the entropy projection is central or the trust region layers proposed in this paper are central. \n5. On a related point, can the authors provide any reasoning about why it is important to enforce the trust region constraints exactly from a theoretical viewpoint?\n6. Why is it important to avoid code-level optimization? If I understand correctly, code-level optimization are just some tricks commonly adopted in TRPO/PPO methods, as pointed out in (Engstrom et al., 2020). Then why is it a big issue to need code-level optimization?\n\nThere are also some slightly more minor issues:\n1. In the abstract, the authors mention that existing trust region DRL methods “lack sufficient exploration”. However, as pointed out later in the paper, existing trust region DRL methods like PAPI have proposed to use entropy projection to encourage exploration, and so this claim is not very accurate. \n2. Again in the abstract, the authors claim that the proposed differentiable trust region layers can complement existing RL algorithms. However, it seems that the authors only applied these layers to the PPO algorithm. Can the proposed layers also be applied to other RL algorithms (beyond PPO/TRPO)? \n3. On page 3, in the definition of the Gaussian policies, the authors may want to make it clearer that $\\mu$ and  $\\Sigma$ are parametrized by $\\theta$ (if it’s the case). Otherwise, it may appear that $\\theta$ is simply the concatenation of $\\mu$ and $\\Sigma$, which would rule out the deep neural network parametrization. \n4. On page 4, at the end of Section 3, it would be better to explain why only the metric for $\\mu$ is scaled by $\\Sigma_2^{-1}$, while the metric for $\\Sigma$ is not. It may also be helpful to consider the alternative Frobenius norm with the second term replaced by ${\\rm tr}(\\Sigma_2-\\Sigma_1)^T\\Sigma_2^{-1}(\\Sigma_2-\\Sigma_1)$ and numerically test and compare the performance. \n5. What is the “entropy projection on its own” approach? Is it just adding entropy projection on top of (1) without trust region constraints?\n6. There seem to be some inconsistencies between the tables and the plots. In particular, for Humanoid-v2, Table 1 shows that KL performs the best in terms of the “last” criteria, but from the center plot of Figure 2, KL seems to be one of the worst in the last epochs. The authors should double check to make sure that there are no such kind of inconsistencies.\n\n### [Comments on the Revised Paper, Rating and Confidence]\n\nThe revised version now contains a much clearer description of how the layers are integrated into the algorithm, fixes several typos and reorganizes (and enriches some details of) the numerical experiments following comments of the other reviewers. \n\nHowever, most of my major concerns above still remain (which is expected as the authors didn’t get a chance to see my review, and I sincerely apologize for this). \n1. For example, although the authors now provide some more detailed explanations about “impractical projections” in Section 4.4, it is still unclear why one cannot use the projection as a post-processing step instead of a layer, and what the authors are trying to convey in the more detailed discussion about impractical projections with a growing storage of previous policy networks in the revised draft here. \n2. Also, the authors may want to clarify some new terminologies and notation introduced in the revision. For example, are “contextual policies” just policies with state-dependent covariances? And what is the index $t$ in the Adam updates in Algorithm 2, and should $a$ and $s$ also be $a_t$ and $s_t$ here? \n3. Another issue I noticed is that compared with the revised draft, the results (in terms of which method is optimal, and whether or not the proposed layers improve over PPO/PAPI) shown in the plots and the tables are not very stable, which indicate that different runs give pretty different results. Such kind of instability may also be relevant to the inconsistency between Table 1 and Figure 2 in the original draft mentioned in the original review above. \n\nOverall, I decide to maintain my original rating.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tpb9CKMCqsA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe paper proposes a way to impose trust region restrictions via projections when doing policy optimisation in Reinforcement Learning. The projections have a closed form and enforce a trust region for each state individually. The authors propose three types of projections based on Frobenius, Wasserstein distances and KL divergence. They compare them to the existing methods (PPO, PAPI) and provide some insights about their behaviour.\n\n### Pros\n- The paper is coherent and clearly written.\n- The paper has a clear motivation and the research question.\n- The paper has an extensive and detailed \"Related Work\" section.\n- I find the analysis of the projections and the result of Theorem 1 extremely interesting and insightful. However, I did not check the proof in the Appendix.\n- The Appendix has many details useful for further understanding of their approach and reproducing the results.\n\n### Cons\n- I find some claims not supported by enough evidence (see Questions).\n- I think the experimental section requires more analysis. It's fine not to beat the existing work with 2x better results, but there should be a thorough discussion of what the results mean (see Questions).\n- Related work comes before the Background, and sometimes it's quite difficult to understand the details of the prior work and their relation to the paper I am reviewing. \"Projections for Trust Regions\" subsections could have more details on positioning/comparing the proposed approach to prior work.\n\n### Reasoning behind the score\nI enjoyed reading this paper. The story flows coherently and logically. The problem the authors consider is important, and the proposed solution is reasonable theoretically and practically. However, the paper sometimes makes a claim without providing evidence to support it. The paper compares itself with the existing strong methods, however, I find the results section lacks analysis regarding this comparison.\n\n### Questions to the authors\n- Your method can impose per-state trust regions. How can you show that this is beneficial? Why is having this beneficial? Can you provide an ablation for this?\n- You claim that your method is \"more stable and less dependent on code-level optimizations\". I don't think you support this claim anywhere in the paper. How can you support that?\n- You mention projections in the introduction, but explain it only in the Related Work section. Can you somehow introduce it earlier?\n- In the last paragraph of 'Approximate Trust Regions', you mention RL as Inference, EM and Song et al. Can you explain the pros and cons of their approaches? Why is your approach still needed?\n- I think, there should be a clear description of what a 'differentiable projection layer' in more details.\n- What is the exact RL algorithm you use for optimisation? Can you provide the pseudocode in the appendix? Can you describe what you mean exactly by \"successive policy updates\"?\n- The plots shades overlap and it's really hard to say which method is better and if that's due to randomness or not. I would like to see more discussion on what the numbers tell us. If your method properly imposes trust regions, but the results are comparable to PPO, does it mean that approximate trust regions are okay? You say that \"Standard PPO is using a lot of code-level optimizations which are not used by our approach\". How can you interpret your results in the light of this? If your method is comparable to PPO without code-level optimisations, what does this tell about your method? Are there any other existing benchmarks which show the superiority of your method? Can you predict some settings, where you method will be significantly better than PPO?\n- In the introduction, you say that 'Due to the approximations, they [PPO, TRPO] violate the constraints or fail to find the optimal solution within the trust region'.  However, you also approximate the trust region in Section 4.4. Why is approximating okay for you, but not okay for PPO/TRPO?\n\n### Additional feedback not affecting the score\n- You do not include the initial state distribution to the definition of the MDP, without it, the transition function under the expectation in Equation 1 does not make sense ($\\mathcal{T}(\\cdot | s_{-1} a_{-1})$ for $t=0$. Same about the trajectory distribution.\n- typo \"covarinces\" on page 5\n- It took me a while to parse Equations 3 and 4 before I realised that parameters in the minimisation problem and in the constraint differ. Can you give a reference to such an optimisation problem in the existing literature (e.g. Boyd's book or somewhere else)?\n- In Section 4, when you describe the projections and Lagrangian multipliers, the results come a bit out of the blue (e.g. 4.1). You have more details in the Appendix, but you do not refer to them from the main text.\n- \"We trained ten agents on different seeds for each method\" in Table 1 caption sounds a bit confusing. Should it be 'for each environment'? Otherwise, it sounds as if you used 10 seeds for a method (2 per each of the five environments).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a differentiable trust region based on closed-form projects for deep reinforcement learning. The update is derived for three types of trust regions: KL divergence, Wasserstein L2 distance, and Frobenius norm, applied to PPO and PAPI, and shown to perform comparably to the original algorithms.

While empirically the proposed solutions does not bring clear benefits in terms of performance, as correctly acknowledged by the authors, it is rigorously derived and carefully described, bringing valuable insights and new tools to the deep RL toolbox. The authors improved the initial submission substantially based on the reviews during the discussion period, and the reviewers generally agree that the work is of sufficient quality that merits publication. To improve the paper and its impact, I would recommend applying the method to also off-policy algorithms for completeness. Overall, I recommend accepting this submission.
Decision: Accept (Poster)