Gradient Informed Proximal Policy Optimization
Authors: Sanghyun Son
Laura Yu Zheng
Ryan Sullivan
Yi-Ling Qiao
Ming Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Analytic Gradient-Based Policy Learning
Proximal Policy Optimization
Differentiable Programming
TL;DR: In this paper, we introduce a novel policy learning approach that leverages analytical gradients, which may exhibit high variance or bias, in order to improve the performance of the Proximal Policy Optimization algorithm.

Abstract: We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm. To incorporate analytical gradients into the PPO framework, we introduce the concept of an α-policy that stands as a locally superior policy. By adaptively modifying the α value, we can effectively manage the influence of analytical policy gradients during learning. To this end, we suggest metrics for assessing the variance and bias of analytical gradients, reducing dependence on these gradients when high variance or bias is detected. Our proposed approach outperforms baseline algorithms in various scenarios, such as function optimization, physics simulations, and traffic control environments. Our code can be found online: https://github.com/SonSang/gippo.

Gradient Informed Proximal Policy Optimization
Sanghyun Son
Laura Yu Zheng
Ryan Sullivan Yi-Ling Qiao
Ming C. Lin
Department of Computer Science
University of Maryland, College Park
{shh1295, hyyhhn2g sonnlli,ylinneee yilingq, lim aelllin@eeee Huudd edu
Abstract
We introduce novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm. To incorporate analytical gradients into the PPO framework, in- troduce the concept of an a-pollcy that stands as a locally superior policy. By adaptively modifying the value, we can effectively manage the influence of analytical policy gradients during learning. To this end, we suggest metrics for assessing the variance and bias of analytical gradients. reducing dependence on these gradients when high variance or bias is detected. Our proposed approach outperforms baseline algorithms various scenarios, such as function optimiza- tion, physics simulations, and traffic control environments. Our code can be found online: https///itthu..com/ hot.i/bb..ooo/SSSSn///ppip
Introduction
Reinforcement learning (RL) techniques often take gradient-frce experiences trajectories to collect information and learn policies-suitaleee for many applications such as video games and recommender systems. In contrast, differentiable programming offers analytical gradients that describe the action- result relationship (Equation 2}. In particular, recent developments in programming tools [Paszke et al. 2017] have enabled direct access to accurate analytical gradients of the environmental dynamics, as exemplified in physics simulations (de Avila Belbute- Peres et al. 2018 Degrave et al. 2019
Geilinger al.] 2020 Freeman al. 2 Qiao al. [202]] instead of using differentiable
models for approximation [Deisenoohh and Rasmussen 2011 Deisenroth et al., 2013]. Policy gradients estimated from the integration of these analytical gradients and the reparameterization trick (RP gradient) [Kingma and Welling, 2013) often display less variance than likelihood ratio policy gradients (LR gradient) [Williams and Peng 1989 Glynn 1990), derived using the log-derivative trick. Consequently, a subset of research has explored the utlizatiion of these analytical gradients for
RL applications, especially in physics domains [Qia et 2021 Mora al. 2021 Xu et al. 2022]
Nevertheless, in environments with inherent chaotic nature, the RP gradient found to exhibit high
variance [Parmas al., 2018) and even empirical bias (Suh al. 2022 Therefore, strategic
approach is necessary for the adoption of analytical gradients in policy learning [Metz et al. 2021] One such strategy involves the interpolation of the LR and RP gradients based their variance and bias [Parmas et al., 2018 Suh et al.) 2022]. made possible by the fact that both are different estimators of the same policy gradient. However, this approach requires estimating the LR and RP gradient for each sample trajectory to compute sample variance, a process that can be time-consuming (Appendix 7.7p. Additionally, for on-policy RL methods that do initially estimate the LR gradient, such as Trust Region Policy Optimization (TRPO) [Schulman et al., 2015a] or Proximal Policy Optimization (PPO) [Schulman et al., 2017) algorithms, remains uncertain how evaluate the variance and bias of analytical gradients. These gradients, however, continue to be desirable when they exhibit low variance and provide insightful information about environmental dynamics.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Given these challenges, our study explores how to incorporate analytical gradients into the PPO framework, taking their variance and bias into account. Our primary contributions are threefold: (1) we propose novel method to incorporate analytical gradients into the PPO framework without the necessity of estimating LR gradients, (2) we introduce an adaptive a-policy that allows us to dynamically adjust the influence of analytical gradients by evaluating the variance and bias of these gradients, and (3) we show that our proposed method, GI-PPO, strikes a balance between analytical -radient-based policy updates and PPO-based updates, yielding superior results compared to baseline algorithms in various environments.
We validate our approach across diverse environments, including classical numerical optimization problems, differentiable physics simulations, and traffic control environments. Among these, traf- fic environments typify complex real-world scenarios where analytical gradients are biased. We demonstrate that our method surpasses baseline algorithms, even under these challenging conditions.
2 Related Work
2.1 Policy Learning Without Analytical Gradients
The policy gradient is described as the gradient of the expected cumulative return in relation to policy parameters SSttoon et al. 1999]. For a stochastic policy, as examined in this paper, REIN- FORCE [Williams 1992 represents one of the initial methods to employ a statistical estimator for the policy gradient to facilitate policy learning. The policy gradient obtained with this estimator is commonly referred to likelihood-ratio (LR) gradient [Glynn| 1990 Fu et al. 2015). Despite the LR gradient's s ability to operate without analytical gradients of dynamics and its unbiased nature, it frequently exhibits high variance [Sutton al.) T998]. Furthermore, the nonstationarity of the policy
renders stable learning difficult to guarantee Konnda and Tsitsiklis 1999. [Schulman et al. 201.5).
Several strategies have been proposed to address the high variance issue inherent to the LR gradient, including the use of additive control variates [Konda and Tsitsiklis 1999 Greensmith et al. 2004). Among these. the actor-critic method [Barto et al. 1983] highly favored. However, the introduction of a value function may induce bias. To counteract this, the Generalized Advantage Estimator (GAE) (Schulman al. 20135) was developed derive more dependable advantage function. Concerning the issue of unstable learning, methods such as TRPO [Schulman et al.) 2015a] and PPO [Schulman al. 2017) propose optimizing a surrogate loss function that approximates random policy's 's expected return. By constraining the policy updates to remain close to the current policy, these methods significantly enhance stability in learning compared to pure policy gradient methods, which makes them more preferred nowadays.
2.2 Policy Learning With Analytical Gradients
In contrast to the LR gradient, the RP gradient is based on the reparameterization trick, which mandates analytical gradients of dynamics yet typically achieves lower variance [Kingma and Welling 2013 Murphy 2022]. The RP gradient can be directly estimated over the entire time horizon using Backpropagation Through Time (BPTT) [Mozer 1989] and subsequently used for policy updates. However, BPTT has been found to encounter multiple optimization challenges such as explosion or vanishing of analytical gradients over extended trajectories (Freeman et al., [02T]. This is because, surprisingly, the RP gradient, contrary to earlier beliefs, may experience exploding variance due to the chaotic nature of environments [Parmas et al., 20188 Metz et al.) 2021). Additionally, /Shh et al.)
2022 noted that the RP gradient could also be subject to empirical bias.
Several corrective measures exist for the issues concerning the RP gradient [Metz et al. 202 such as using a truncated time window bootstrapped with value function [Xu et al. 2022]. This approach has demonstrated its efficacy in addressing physies-related problems. Another class of solutions involves interpolating between LR and RP gradients by evaluating their sample variance and bias [Parmas et al. 2018 (Suh al., 2022]. Likewise, when we carefully address the problems of RP gradient, it can provide valuable information helpful for policy learning. However, when it comes to PPO, which is one of the most widely used RL methods nowadays, it unclear how we can leverage the analytical gradients that comprise the RP gradient, as it does not estimate LR gradient which can be used for comparison to the RP gradient. In this paper, we address this issue and show that we can use analytical gradients to enhance the performance of PPO.

---

3 Preliminaries
3.1 Goal
We formulate our problems as Markov decision process (MDP) defined by tuple (S, A, P.r. r. Po: Po:7),
where a set of states, A is a set actions, S x S + is a state transition model, 8 X A + IR is reward model, Po is the probability distribution of the initial states, and y is discount factor.
Under this MDP model, our goal is to train parameterized stochastic policy Te S x A + R where / represents parameters, that maximizes its expected sum of discounted rewards, (TT))
77( (T) = Eso Eso.a
Σ
7'r(s, ar)
(1)
In this paper, we denote the current policy that is used to collect experience as Te..
3.2 Analytical Gradients
We assume that S and A are continuous, differentiable spaces, and P and r are also differentiable models. Then, our differentiable environment provides us with following analytical gradients of observation reward a later timestep with respect to the action at the previous timestep:
0841+*
dritk
(2)
Da,
Oa.
where k is an integer. With these basic analytical gradients, compute the gradient of 04 In this paper, use Generalized Advantage 8n
certain advantage function, which we denote
Estimator (GAE) [Schulman et al. 2015b). See Appendix 7.2. 7.22. for details.
3.3 Policy Update
In this section, discuss how update policy in two different settings: the policy gradient method based on RP gradient and PPO.
3.3.1 RP Gradient
To compute the RP gradient, we rely on the reparameterization trick [Kingma and Welling, 2013). This trick requires us that we sample an action a from me by sampling a random variable from some other independent distribution q. that end, we assume that we use continuous differentiable bijective function (s.-s) R" + R that maps Then the following holds because of injectivity.
action a, where is the action dimension.
det (go(s,c) c) de
> 0, O.
(3)
and we can define the following relationship.
Definition 3.1 We define me 2 if following holds for arbitrary open set T. C R":
[ To(8, a)da = / q(s)de,
where T. go Te). Lemma 3.2 If π.
det 0go(s, (s)
To(s, a) = =(())
de
where go(s,e) <). The inverse is also true.
3

---

Proof: See Appendix 7.1.1
Specifically, we use (0, in this paper, and define go follows,
go(s, E) = po(s) + oo(s) e (llor())ll2 > 0),
which satísfies our assumption.
With gs e) and the analytical gradients from Equation 2
we obtain RP gradient by directly differ- which we use for gradient ascent. See Appendix 7.2.2
entiating the objective function in Equation [ for details.
3.3.2 PPO
PPO relies the observation that we can evaluate policy with our current policy wy. using its advantage function AT as
(TT))) = m((a)) + pro(s) / no(S, (s,
where Px. (s) is the discounted visitation frequencies. See Kakade and Langford 2002 Schulman
ctt al. 2013a] for the details.
As far as the difference between Th and is sufficiently small, we can approximate Pr, (s) with
Pro (8) [Schulman et 2015a), and get following surrogate loss function:
Lng (Te) n(TT) 1 pn)(()) / an)
Note that we can estimate this loss function with Monte Carlo sampling as described in Appendix 7.2.3 Since this is local approximation, the difference between गह and Tī must be small enough get an accurate estimate. TRPO [Schulman et al. 2013a] and PPO [Schulman et al.) 2017 constrain the difference be smaller than certain threshold, and maximize the right term of In particular, PPO restricts the ratio of probabilities To (s; a,) and (T(s;, a.) for every state-action pair (st, az) in the buffer as follows to attain the goal, using constant Eelipi
Cclip < To(si, a1)
Ediip
(5)
Tg(3;, ai)
4 Approach
In this section, we discuss how we can use analytical gradients in the PPO framework while consider- ing their variance and biases. We start our discussion with the definition of a-policy.
4.1 a-Policy
With the analytical gradient of advantage with respect an action (V (V. (s, a)) from our differen- tiable environments, we can define new class of policies Ta, parameterized by a, as follows.
Definition 4.1 Given current policy Tg. we define its a-policy as follows:
(a(s.a) det[[++a
if
Bas. s.l. = f(a) = a a (s
T. (8, =
else
(6)
where constant |a|] max (1
Here X1 (8, a) represents the minimum eigenvalue of Vi
Lemma 4.2 Mapping f is injective, and for an arbitrary open set of action T C A, Ta(s selects set of action T = f(A with the same probability that nT(s.)) selects A.

---

fi))
d-0
nix)
8-
4.5
430
26
-04
640
1255
40.3
3.00
5.5
1220 0.20
(a) De Jong's Function
(b) Ackley's Function
Figure I: a-policies for 1-dimensional (a) De Jong's function and (b) Ackley's function. The original policy rendered in blue, and alpha policies for different a values are rendered orange and red. Those a-policies are obtained from analytical gradients of the given function (x) rendered in black. Note that Ackley's a-policy becomes invalid when a = 2- 10-3, because of singularities near
Proof: See Appendix 7.1.2
In Figure I we provide renderings of a-policies for different as when we use 1-dimensional De Jong's s function and Ackley's s function [Molga and Smutnicki [2005] as our target functions. The illustration provides us an intuition that when selects slightly better action than Ty with the same probability. In fact, we can prove that Ta indeed gives us better estimated expected return in Equation [quation[[when
sufficiently small, Ta can be seen as policy that
is sufficiently small.
Proposition 4.3 If ||| <
Lig (Ta) T7( O(ja)) when a n((Tg) Lr (Ta) O(Ial) when 0,
where Lag denotes estimated expected return defined in Equation
Proof: See Appendix 7.1.3
This proposition tells that a-policy fits into PPO framework seamlessly, and thus if update policy towards a-policy, it does not contradict the PPO's objective. However, since there is a second order derivative V Va (s, a) in the definition, it is unelear how we can update our policy Ty towards In the next section, we show how we can do this and how it is related to RP gradient.
4.2 Relationship between RP gradient and a-policy
To show how we can approximate a-policy, for a deterministic function Ia such that vy 9ā. we define new function Ja follows, and provide Lemma about it.
(a(
Va (8,a), where = (s.
(8)
Lemma 4.4 When (8)
(g)(s,c) del
0go(s,c) del(I + (8,
del
(9)
de
de
Proof: See Appendix 7.1.4
Note that update go approximate Ja by minimizing the following loss.
L(0) Egg Eeo.c..... [100(s), (a) ga (st, (s,, c))l
(10)
In fact, it turns that minimizing Equation 10 leads our policy ma to approximate mas because of the following Proposition.
Proposition 4.5 If n e 98. for that satisfies the constraint in Definition 4.1 Ta 9a-

---

Proof: See Appendix 7.1.6
Note that different advantage function formulations for A in Equation 8 such as GAE [Schulman et al. 2015b] that we use this work. However, let us we consider the following advantage function A to see the relationship between RP gradient and a-policy.
Axp (sc, ar)
2 E. Es..,
Σ 7 r(sk ((((x.ax)) A=
(11)
Then, the following Lemma holds.
aL Lemma 4.6 When a 1, if we define Io with A, the RP gradient corresponds to 80
0 0.
Proof: See Appendix 7.1.5
This Lemma tells us that we can understand the RP gradient as the very first gradient we get when we update our policy toward Ta by minimizing Equation TO In other words, a-policy superior policy than the original one in the PPO's point of view (Proposition 4.3), but also target policy for RP gradient (Lemma 4.6)) Therefore, we conclude that a-policy a bridge that connects these two different policy update regimes.
only
4.3 Algorithm
Our algorithm is mainly comprised of 3 parts: (1) Update тTe to T. by minimizing Equation I0 (2) Adjust for next iteration, and (3) Update Ty by maximizing Equation In this section, we discuss details about Step 2 and 3, and provide the outline of algorithm.
4.3.1 Step 2: Adjusting &
we can observe that a controls the influence of analytical gradients in policy updates. If 0, it is trivial that n. Tg. which means that analytical gradients ignored. In contrast, as gets bigger, analytical gradients play bigger roles, and To becomes increasingly different from zg. Then, our mission is find well-balanced a value. Rather than setting it fixed hyperparameter, we opt to adjust it adaptively. We first present the following variance and bias criteria to control a, as it is commonly used assess the quality of gradients [aarma al. 2018 Suh et al. 2022
In Definition 4.1
Variance After we approximate Ta in Step 1), we can estimate det(! + a V2 state-action pair in buffer using Lemma 44 and adjust to bound the estimated values in certain range, [1 5, + 5). There are two reasons behind this strategy.
- (s, a)) for every
As shown in Definition 4.1
we have to keep a small enough that T does not become invalid policy. By keeping det( + a V22A, (s, (s,a)) larger than - to keep its positiveness, we guarantee that our policy is updated towards invalid policy.
Note that det 7 + A.A.A (s, a)) II (1 + ad:). where x; is the i-th eigenvalue of Va,Ang (s, a). By constraining this value near 1 by adjusting we can guarantee stable policy updates even when some eigenvalues of V A7 a) have huge magnitudes, which is related high variance RP gradient. In Appendix 1.3.1] we provide empirical evidence that this estimation is related to the sample variance of analytical gradients.
we already know that when estimate the right term of Equation 4 after Step 1), it should be positive value. However, if the analytical gradients were biased, this condition could be met. In runtime, if detect such cases, we decrease a.
Bias By Proposition 4.3
In addition, we use one more criterion to adjust o, which we call the out-of-range ratio. This criterion designed promise the minimum amount of policy update by PPO.
Oft-of-range-ratio define out-of-range-ratio as follows.
N
To To(8; a.)
oft-o--range-ratio
- > Cctip),
(12)
N
TT(3t, a4)

---

where N is the size of the buffer, II is the indicator function, and Cetsp is the constant from Equation We assess this ratio after the policy update in Step 1), allowing its value be non-zero when a A non-negligible value of this ratio indicates a potential violation of the PPO's restriction outlined in Equation 5) thereby compromising the effectiveness of PPO. Further elaboration can be found in Appendix 7.3.2 Therefore, when this ratio exceeds a predetermined threshold, we subsequently reduce the value a.
To sum it up, our strategy to control a aims at decreasing it when analytical gradients exhibit high variance or bias, or when there little room for PPO updates. This results greater dependence on PPO, which can be thought of as safeguard in our approach. Otherwise, we increase a, since we can expect higher expected return with greater a as shown in Proposition 4.3
4.3.2 Step 3: PPO Update
Since do policy updates based on PPO after the update based on Ta there is possibility that the PPO update can "revert" the update from Step 1). To avoid such situations, we propose to use the following Th in the place of Ty in Equation
(s a) =
2 (= + (s)
By using this virtual policy The we can restrict the PPO update to be done near both zy and Ta- See Appendix 7.3.2 for more details.
4.3.3 Pseudocode
In Algorithm|] [ we present pseudocode that illustrates the outline of our algorithm, GI-PPO. There five hyperparameters the algorithm, which are ag. 3, deer 8 and max(a:). We present specific values for these hyperparameters for our experiments in Appendix 7.4.2
Algorithm 1 GI-PPO
<- (lg. Initial value
< Constant multiplier larger than for a
ddet Soorr + Constant thresholds
B + Experience buffer
while Training ended do
Clear
while Not collected enough experience do
Collect experience (st, Ct, az, Tt, St+1] B
end
Estimate advantage for every (s;, ag) in using Eq. 14
Estimate advantage gradient da for every (s,, a4) in B using Eq.
For current a, approximate a-policy by minimizing loss in Equation 10
// Variance
For each 6; and its corresponding state-action pair (ss, a;), estimate det(T + a V2 12 (s, a)) by
Lemma and get its sample minimum (+mmi) and maximum (Hmaa)
// Bias
Evaluate expected additional return in Equation 16 with our current policy to get Ro // Out-off-angee Oft-of-range-ratio
Evaluate oft-of-range-ratio Equation 12 with our current policy to get Roorr
if tmmn 1 - ddet t'max 1 + ddet R. < 0 or Roory > doorr then
a = a/3 end else end
- B
clip(a. O, max(a))
Do PPO update by maximizing the surrogate loss in Equation 18
end

---

Table 1: Average maximum reward (f) for function optimization problems.
Problem
LR
RP
PPO
LR+RP
GI-PPO
Dejong (1)
-1.24+10-
-1.42=10-5
-5.21110-3
-6.36110-
-3.84+10 -10
Dejong (64)
-0.0007
-9.28+10-
-0.0011
-3.05=10-6
-1.04+10-6
Ackley (1)
-1,2772
-0.4821
-0.2489
-1,2255
-0.0005
Ackley (64)
-0.6378
-0.0089
-0.1376
-0.0326
-0.0036
LA+RP G1PPO
RR+RP GPP
G1 PPO
AP
(a) De Jong (1)
(b) De Jong (64)
(c) Ackley (1)
(d) Ackley (64)
Figure 2: Optimization curves for Dejong's and Ackley's function of dimension I and 64.
5 Experimental Results
In this section, we present experimental results that show our method's efficacy for various optimiza- tion and complex control problems. To validate our approach, tested various baseline methods on the environments that we use. The methods are as follows:
LR: Policy gradient method based on LR gradient.
RP: Policy gradient method based RP gradient. For physics and traffic problems, adopted a truncated time window of [x et al. 2022 to reduce variance.
PPO: Proximal Policy Optimization [Schulman et al. 2017].
LR+RP: Policy gradient method based on interpolation between LR and RP gradient using sample variance [Parmas et al.) 2018).
PE: Policy enhancement scheme of [Qiao al. 2021 for physics environments only.
GI-PPO: Our method based on Section
Please refer Appendix 7.4 to learn about implementation details these baseline methods, network architectures, and hyperparameters that we used for all of the experiments.
We have implemented our learning method using PyTorch 1.9 Paszke et al. 2219 As for hardware. all experiments were run with an Intel@ Xcon@ W-2255 CPU 3.70GHz, one NVIDIA RTX A5000 graphics card, and GB of memory. Experiment results are averaged across random seeds.
5.1 Function Optimization Problems
We start with how different learning methods find the maximum point for a given analytical function. We these experiments because RL algorithms be thought of as function optimizers through function sampling (Williams and Peng 1989). These problems are 1-step optimization problems, where agents guess optimal point
action and get the negative function value the reward.
As we used for rendering a-policies in Figure ! we use De Jong's 's function and Ackley's 's function for comparison, as they popular functions for testing numerical optimization algorithms [Molga and Smutnicki 2005). Please see Appendix 7.3.J for the details about the function definitions, In 0 for both of the problems. However, their loss
this setting, the optimum occurs at the point landscapes are quite different from one another De Jong's function has a very smooth landscape that only has one local (global) maximum, while Ackley's 's function has a very rugged landscape with multiple local maxima. Therefore, De Jong's 's function represents environments where RP gradients have lower variance, while Ackley's function represents the opposite.
Table shows that GI-PPO finds far better optimums than the other methods for most of the functions, except 64-dimensioona Dejong's function, where the RP method found slightly better

---

1800
1031
1031
4099
2000
G1 PP0
CI
2030
-4000
PPO
1839
1991
333
3.25
3.0
(a) Cartpole
(b) Ant
(c) Hopper
Figure 3: Learning graphs for 3 problems in differentiable physies simulation: Cartpole, Ant, and
Hopper [u et al. 2022]. In these environments, RP gradients already exhibit lower variance without much bias than LR gradients, because variance reduction scheme of [Xu et al. 2022
optimum than GI-PPO. Optimization curves in Figure 2 also prove that our method converges to the optimum faster than the other methods. Interestingly, the RP method achieved better results than the LR and LR+RP methods for all the functions, which contradicts our assumption that it would work well for Ackley's 's function. However, we found out that even though the RP method takes some time to find the optimum than the other methods as shown in Figure 2 when it approaches near-optimum, converges fast to the optimum based on analytic gradients. Since GI-PPO adaptively changes to favor RP gradient near optimum, it could achieve better results than the other methods. In Appendix 7.3.3 we provide visual renderings that trace the change of a value during optimization to help understanding, which also shows that higher is maintained for Dejong's function than Ackley's 's function, which aligns with our initial assumption. Also note that even though LR+RP also takes advantage of the RP method, and thus achieved better convergence rate than the two methods when it is far from optimum, it did not work well when it comes to near-optimum.
5.2 Differentiable Physics Simulation
Next, conducted experiments in differentiable physics simulations. We used Cartpole, Ant, and Hopper environments implemented by (Xu et al. 2022] for comparisons. However, since the current implementation does not support multiple backpropagations through the same computation graph for these environments, we could not test the L.+RPP method for them. Instead, we tried to faithfully implement another policy enhancement (PE) strategy of [Qiao al. [02T] showed its efficacy in differentiable physics simulations using analytical gradients.
comparison,
In Figure 3 can see that GI-PPO converged to better policy much faster than the baseline PPO for every environment, which validates our approach. However, the RP method achieved the best results for Ant and Hopper, while GI-PPO converged to the optimal policy slightly faster for Cartpole. To explain this result, we'd like to point out that a is upper bounded by oft-of-range-rait in our approach (Section 4.3. Therefore, because we have to use PPO to update the policy up to some extent as a safeguard. In the Ant environment, could observe that achieve better results than RP when we increased the ratio from 0.5 to 1.0 (Appendix) 77..44.. However, when the out-of-range ratio equals 1.0, it becomes more likely that we cannot use PPO safeguard, which contradicts our design intention. When it comes to the Hopper environment, we found that the variance of proper values over time large, so our strategy could not conform to it properly. Overcoming these limitations would be an interesting future work to extend our work.
the RP gradient is very effective, cannot fully utilize
the major bottleneck in training-in fact, GI-PPO could
5.3
Traffic Problems
Here demonstrate the effectiveness of GI-PPO an environment where continuous and discrete operations coexist, which leads to highly biased analytical gradients. We suggest a mixed-autononn traffic environment as a suitable benchmark, which has previously been explored gradient-free RL algorithms [Wu et al., 2017 Vinitsky et al.) 2018). In this paper, we use the pace car problem, where single autonomous pace car has to control the speed the other vehicles interference. The number of lanes, which represent the discontinuities in gradients, and the number of following human vehicles are different for each problem. Please see Appendix 7.5.2 for the details of this environment, and why the analytical gradients biased.
benchmark for

---

LA
RUP
IU
PPQ
PPO
LR
LR+
LA
GL-170
LA
GIAPO
RP
12
PPQ
PPO
LRt RP
LR
G1-PP1
G1-PP
SeYp
(a) Single Lane
(b) 2 Lanes
(c) 4 Lanes
(d) 10 Lanes
Figure Learning graphs for traffic pace car problem. Each problem has different number of
lanes and following human vehicles, which represent discontinuities and dimension of the problem. respectively.
In Figure 4 we can observe that GI-PPO exhibits a faster convergence rate and converges to better policy overall compared to all the other methods in 2, 4, and 10-lane environments. In single-lane environment, however, I-R+RP showed a better convergence rate than GI-PPO. Note that the method still performs well for 2 and 4-lane environments just as other baseline methods, even if there were multiple lanes. We assume this is because the RP method also leverages a state value function, which can provide unbiased analytical gradients in the presence of discontinuities. However, it does not perform well the 10-lane environment, where discontinuities are excessive compared to the other environments.
This result shows that GI-PPO can still leverage analytical gradients, even if they were highly biased for policy learning. Even though LP+RP, which is based on similar spirit as ours, also exhibited some improvements over I.P and RP, our method showed better results for more difficult problems because baseline PPO offers more stable learning than baseline LR, Also, the computa- tional cost GI-PPO was a little more than RP, while that LR+RP was much more expensive (Appendix 7.7
6
Conclusions
We introduced a novel policy learning method that adopts analytical gradients into the PPO frame- work. Based on the observation that the RP gradient leads our current policy to its a-policy, we suggested approximating explicitly, which allows us indirectly estimate variance and bias of analytical gradients. We suggested several criteria to detect such variance and bias and introduced an algorithm that manipulates a adaptively, which stands for the strength analytical gradients in policy updates. In our experiments of diverse RL environments, we successfully showed that our method achieves much better results than the baseline PPO by adopting analytical gradients. Even though the interpolated gradient of LR and RP gradient is based on similar motivations, our method performed much better, thanks to the baseline PPO's stable learning.
Limitations Despite GI-PPO's promising results, there are some remaining issues. First, even if we use a large a, we may not fully approximate the corresponding a-policy in limited number of optimization steps, which may give false signal for our strategy. We upper bound the maximum a as we did here, but a more rigorous optimization process may be helpful. This is also related to improving the strategy control adaptively, which would be promising future work.
Also, as discussed Section 5.2, our method is tightly bound to PPO PPO-that 1hh-thatiie is, even when analytical gradients are much more useful illustrated in differentiable physics problems, we cannot fully utilize them they exit the PPO's s bound. Even though it is more suitable for stable updates, could result in worse convergence rate. If could adjust PPO's clipping range dynamically, detect environments where RP gradients are much more reliable, we would be able to overcome this issue.
Finally, computational costs can be further reduced. To utilize analytical gradients, we need backprop- agation, which usually requires more time than forward steps. This is the reason why the learning methods based on analytical gradients require longer training time than the others (Appendix (.7). When the time window gets longer, the cost grows accordingly. However, as we have shown in our traffic experiments, our method works even when gradients are biased. Therefore, it would be an interesting research direction see using a very short time window for backpropagation, which produces more biased gradients, would be worthy of exploring for improved computational efficiency.
10

---

This research is supported in part by the ARO DURIP and LARPA HAYSTAC Grants, ARL Cooperative Agreement W911NF2120076. Dr. Barry Mersky and Capital One E- Nnovate Endowed Professorships. Yi-Ling Qiao would like to thank the support of Meta Fellowship.
Acknowledgements.
References
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846, 1983.
Filipe de Avila Belbute-Peres Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End- to-end differentiable physics for learning and control. Advances in neural information processing systems, 31, 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, page 6, 2019.
Mare Deisemroth and Carl Rasmussen. Pilco: A model- and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465-472, 2011.
Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends@ in Robotics, 2(1-2):1-142, 2013.
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physies engine for large scale rigid body simulation. arXiv preprint arXiv:2106. 1 13281, 2021.
C Fu, C Fu, and Michael Michael. Handbook of simulation optimization. Springer, 2015.
Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski. and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):55-88 84, 1990.
Evan Greensmith, Peter I. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for rein- forcement learning.
r1 rl_games May 2022.
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients not all you need. arXiv preprint arXiv:2111.058033 2021.
Marcin Molga and Czeslaw Smutnicki. Test functions for optimization needs. Test functions for optimization needs, 101:48, 2005.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference Machine Learning, pages 7805-7817. PMLR, 2021.
M Mozer. A focused backpropagation algorithm. Complex systems, 3:349-381, 1989.

---

Kevin P Murphy, Probabilistic machine learning: an introduction. MIT press, 2022.
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-based policy search robust the curse of chaos, In International Conference on Machine Learning. pages 1065-4077, PMLR, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Efficient differentiable simulation of articulated bodies. In International Conference on Machine Learning, pages 8661-8671. PMLR, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. mabll-baselinest Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-2 88, 2021. URL http://jmlr org/ pwppps/ 2appers/222/0--1212 1364. html
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint wrXhv:156.004:: 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.0077, 2017.
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pages 20668-20696 PMLR, 2022.
Richard S Sutton, Andrew G Barto, et al. Introduction reinforcement learning. 1998.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing, Congested traffic states in empirical observations and microscopic simulations. Physical Review E, 62(5): Aug 2000.
Eugene Vinitsky, Aboudy Kreidieh, Lue Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference on robot learning, pages 499.409. PMLR, 2018.
Ronald J Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Machine learning, 8(3):29 256, 1992.
Ronald Williams and Jing Peng. Reinforcement learning algorithms as function optimizers. In Proceedings of the International Joint Conference on Neural Networks, Washington DC, volume 2, pages 89 93, 1989.
Cathy Wu, Aboudy Kreidich, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv:1710.05465. 10, 2017.
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint 2771072004.07137, 2022.
12

---

7 APPENDIX
7.1 Proofs
7.1.1 Proof of Lemma 3.2
By Definition 3.1 Te a) should satisfy the following condition:
Jr ((c)de = / xp(s.a)da
Ta
J
(go(s, c)
To(s, go(s,go(s,e)) det(
de (by change of variable).
de
We denote the set of continuous distributions that satisfy this condition P. That is,
/ g(e)de = -f Po(s,)) |deu Ago(s, c) de.
P if JT.
Jr.
is element of P. and is well defined, because of Equation 3 In fact, cs, is the only element of P, the above condition should be met for any T. R" That is, if there was another distribution Eo(s,E) E.
Clearly, co(s, = q(c) /det( (ga(s.e))
J (PP(ssc)-Po(o.0))) Po(s, c))
/dct( )ga(s,e) de = 0, R".
de
Since Po is different from Po, there exists open set T, where Po Po. Then
- Po(s, dot(( (go(s,c) de > 0, Jr. (Po(s..)-
de
which contradiction. Therefore, ne(s,u)= = To(8, Po(ose). The inverse holds trivially.
7.1.2 Proof of Lemma 4.2
0f First, we show that f is an injective function by showing that the determinant of Jacobian always positive:
Ta is
0f det(I + (s,o)) Da
det
-1 ( 0 (a- <
max (s)
where \s(s, a) are the eigenvalues of V2A.,, (s, a) sorted in ascending order.
Then, for an arbitrary open set of action T C A, (s(s.)) selects ī F(T) c with the same probability that the original policy Ta(s,-) selects T.
J (s ā)dā
/ [det + +aV2 A (8)
Tp(8,a)
det( 05 da
) da
I |dot(T + aV A (8,
TT(s,a)
del(T + (2,r)(s,a))dda (s,a))
/
adda,
Therefore, (s, is valid probability distribution (T) -).
13

---

7.1.3 Proof of Proposition 4.3
For given state s, we estimate the (approximate) expected value of the state under To follows.
No (s. A) (s, dā
Tgls,00 a)
|et((T + aV A (8, a))l ATp as
(0, (s,a))]da
T TT(s,a) a) (s, allV A (8, ()) (8.))) da
Tols, a) (s,
(s, a)da + To(s, a)ls.a)|V a)ll
Therefore, can the following holds:
Lag (Ta) / Pro(s) (6) / An) (s, ā)dā
()(()) + a (r) ( (s,
Since Tp(s, a) and || An, (s, aa]l are positive, Ting ( is greater or equal to m)(T)) when a > 0. On the contrary, when a O, Lig (ma) is smaller or equal to (TTT)). Since this argument builds upon local approximation, it holds only when |a| << 1.
7.1.4 Proof of Lemma
When a = (8,
(s( g6((.e) +a As,
(8, @gg(s,e) + V2 An,
(gr(s,e) e)
de
de
de (ga(s,e)
( V2 An,
de
= det (gaa(s,) (dga(s.c) del (I a V2 Ar (8. del (gga(s,
de
det [ + a (s, det (āga(s,)) c)
de
Since det (00((e.e) Hr
(Equation 3), we can divide both sides the above equation with
det (Ama(s.e) and prove Lemma 4.4
7.1.5
Proof of Lemma 4.6
From Equation II
Or(sk. aKk)
V VaAro (St, 04) = R
2
Da,
#=
which can be rewritten using do as follows.
V Va (11, = B
1
(o(8k.90(88,.
(13)
kmt.
14

---

Then, by differentiating Equation 10 relationship:
with respect to & after plugging in A, we can get following
aT((0)
(go Et),
(go - 9a 9a (st, (s1,
06
0B
(gr (g0()))))) - (1) + Va (5,, ax)))]
Eg co....g..
00
1=0
(:: a 1)
2
(go c1)
Eep,co.....
2
(odno(s(.
00
1=0
x
dgs (gr($1,E)) EL)
E Eg 60
2
08
0gals;,
1=
( Equation
13
Ogo (8, ())
go 4))
E
00
(gr(s)
1=0 km
which equals to RP gradient in Appendix 7.2.2 but with different sign. However, they turn out to be the same because we minimize L(0), but maximize (TTT)). Therefore, we can say that we gain RP gradient as the first gradient when we minimize Equation 10 for particular advantage function A and 1.
7.1.6
Proof of Proposition 4.5
By Definition 4.] and Lemma 4.4
Ta (s, a)
( a)
det([ + a V2 (5,x,,),a))) (s,
)gg(3,c) }| det( Oga (s. c) 1-
= det(
de
de
where a = g)((,,),, and thus = yo (8, 6). Since Ty A gø. following holds:
Ta ā) = (s(s, a) det dga(s,c), c) )1- det dg. (g.(s. )|1
de
de
<(c) det Oga(s, E) 6), 1|11 det( (gg(s,t) )1 det( ())(3,))))1
de
de
de
(:: Lemma 3.2
c(e) det( (g, |1_
de
which implies Ta A Ja-
7.2
Formulations
7.2.1
Analytical Gradient of Generalized Advantage Estimator (GAE)
GAE
Schulman et al. 2015b) has been widely used in many RL implementations Schulman et al.
2017
Raffin al. [222 Makovicchuk and Makoviychuk] [2022] estimate advantages. GAE finds a balance between variance and bias of the advantage estimation by computing the exponentially- weighted average of the TD residual terms (6) Sutton et (1998), which are defined as follows:
8 T. + (V V(s.).
15

---

Then, GAE can be formulated as follows:
20
A,A Σ
(14)
k=0
We can compute the gradients for these terms as:
OAGAE
06
(7))
da.
da,
k=
0
(7^)² 1- 06
da.
k=
(7:) ++ 06
06%
(71)* da.
Dae
(15)
for < t)
da
06%
(*))*
da.
k=0
DAGAE
Da,
BACAE
Based on this relationship, can compute every rather than backpropagating for every A
with only one backpropagation of AGAE
da,
7.2.2 RP Gradient Formulation
To differentiate Equation I with respect to 0, we first rewrite Equation as follows.
7r(s,, a:)
(TTT = Eeg.go....
-Eegton..
r go (0))))))) (000000 Σ
Then, we can compute RP gradient as follows.
@m((Te)
Eso Sarra....
00
30
Eso.eg....
o rr cz)
00
1=
Ogo (B, E
go(kk
=Eaofo
30
Ör(Sk, go (ok, (x))
(::
* 0 only when k 2 t.)
0go(se, E1)
dgo(ser e4)
Ōr(sk go(sk E1) (go te)
Eao.fo
00
ago e))
ar(s,, ax)
00
Oa,
k=t
We can estimate this RP gradient using Monte Carlo sampling and terms in Equation and use it for gradient ascent to maximize Equation [
16

---

2
110
80
(a) Cartpole
(b) Ant
(c) Hopper
The first row shows the sample variance of analytical gradients ( (OA). and the second row shows the statistics of estimates of det(1 a Ar, (s, a)) for every state-action pair in the buffer after each epoch of training in different differentiable physics environments (Cartpole, Ant, and Hopper). We used fixed 10 Note that the minimum and maximum estimates in the second row exhibit clear tendency to deviate from more as the sample variance increases.
Figure 5:
7.2.3 Estimator for Equation
Since n((Tg) does not depend on 0, we can ignore the term and rewrite our loss function in Equation using expectation as follows:
T (TTO = ( ro(s, Aro (8,
Esmpigranaol Ar (s)
However, note that we did not collect trajectories, or experience buffer, using na. Therefore, we use another importance sampling function g(s, a) [Schulman et al. 2015a].
TB(ss a) Ar, (8,
Long (TO) = Eagupag anv((s.)
((s,a)
Then, we can estimate this value using Monte Carlo estimation as follows, where B is the experience buffer of size N, which stores state (s;), action (a,), and their corresponding advantage value (Ang (si, a:)) obtained by following policy ng:
To ai) Ang (8i, ag).
Ling (Te)
(16)
T8(8i, ai)
Note that we used Hy in the place of q. as the most natural importance sampling function we can (Schulman al. 2015a 2017].
7.3 Algorithm
Here we present the details of our algorithm in Section 4.3 To be specific, we present 1) an empirical evidence to validate our strategy control a, 2) the exact loss function that in PPO update, and 3) provide the change of a during real training to help understanding.
7.3.1 Empirical evidence
we argued that the estimate a a)) related the sample variance of analytical gradients, and we can get the estimate using Lemma 4.4 In Figure we empirically prove the validity of this argument in the differentiable physies environments used in Section 5.2 In the illustration, we can observe clear relationship between the sample variance of analytical gradients and the statisties of our estimates. Therefore, we can use these estimates cue control based on the variance of analytical gradients.
In Section 4.3
17

---

ed our partorr torrranee
16-3
50
20
500
1000
1500
2000
320
2000
15:00
2020
500
2000
1300
2002
520
1000
1500
2000
2.0
3.0
0.6
0.6
Detimee Dakerm
02.
Que bett
0.0
0.4
0.7
Moun
Mm
Max
1050
(a) De Jong's Function (64)
(b) Ackley's 's Function (64)
Figure 6: Change of a during solving function optimization problems Section 5.1 Since these problems one-step problems. and observations are all the same, there is no difference between mean, min, and max of estimated determinant.
7.3.2 Loss function for PPO Update
As discussed Section 3.3.2 PPO finds better policy in the proximity of the original policy by restricting the ratio of probabilities Ta(s;, a.) and (8t, a;), shown in Equation 5 In fact, PPO uses following surrogate loss function, which modified version of Equation T6, to mandate this:
N
To To(s a.) Arg (8,, a:), clip( To(s;, a;)
To (o a:) To (s, a))
(P)
min
1 Colipr + Colsp) (8i, a:)).
N
(17)
Note that if out-o--rane--ratio defined in Equation 12 already 1.0, which corresponds an extreme case, the gradient of (0) with respect to 0 is 0. Therefore, PPO does not contribute to the policy update at all in this case. It justifies our decision upper bound out-of-range-ratio by restricting a.
Also note that we proposed to use a virtual policy Ty instead of my in Section 4.3.2 to restrict PPO update to be done around Ta- This changes the importance sampling function in Equation 17 as follows, which corresponds to our final surrogate loss function to use in PPO update:
N
min( To(8i, as) Th(si, a;)
To (86,
Lairro (0) N
Azp (8i, az), clip( Th (n a;))
Celip. 1 + Gelip) Ang (8i, a:)).
(18)
7.3.3 Example: Function Optimization Problems (Section 5.1
Here we present how changes as we solve function optimization problems in Section 5.1 with Figure There are 4 plots for each of the problems.
Alpha: Shows change of over training epoch.
Estimated current performance: Shows expected additional return, which corresponds to Equation 16 (R, in Algorithm IT.
Estimate Determinant: Shows statistics of estimated det((T V (s,
Out of Range Ratio: Shows oft-oo-range--aun in Equation 12
For these problems, we have set agg = 10 6 = 0.4, and 6 = 0.5.
For De Jong's function (Figure 6a). we can observe that shows steady increase over time, but it is mainly upper bounded by variance criterion and out-of-range-ratioo In the end, a stabilizes around 10 10-3, In contrast, for Ackley's 's function (Figure 6b)
that increases rapidly at first, but
18

---

790
200
mng Pc
30
2008
1130
1250
1012
(a) Statistics of training in Ant environment when doorr 0.5.
1000
5640
Gn 5P0
(b) Statistics of training in Ant environment when d0orr 1.0.
Figure Change of a during training Ant environment in Section 5.2 In this environment, maximum out-of-range rafe----g--rtoo (foorr) plays major role in bounding from growing. Observe that GI-PPO achieves better results than RP when we set ó 1.0.
soon decreases due the variance and bias criteria. Compare these graphs with optimization curve in Figure 2d Then we would be able to observe that this large contributes to the faster convergence of GI-PPO compared PPO the early stage. However, after that, a decreases slowly due the out-of out-of-range-ratiie In the end, reaches approximately 10-6 which is much lower than 10-3 of De Jong's s function. These observations align well with our intuition that RP gradient would play bigger role for De Jong's function than Ackley's function, because De Jong's ' function exhibits RP gradients with much lower variance than those of Ackley's.
7.3.4 Example: Physics Simulation Problems (Section 5.2
In Section 5.2. GI-PPO could not achieve better results than RP in Ant and Hopper environments. For Ant environment, found out that the hyperparameter Soor GI-PPO's performance. In Figure 7 we illustrated the change of a during training along the change of -ut-o--rang---atio. When we set doorr 0.5. keeps decreasing over time keep the ratio around 0.5, which means the diminishing role of RP gradients in training. In contrast, when we set dourr 1.0, after 750 epochs, does not decrease anymore, and out-of-range ratio stays near This means that PPO rarely affects the training policy update is almost driven by RP gradients.
bottleneck that bounds
Likewise, there environments where RP gradients much more effective than the other policy gradients. In those situations, it would be more desirable to solely depend on RP gradients without considering PPO. Since method considers RP gradients in the PPO viewpoint, is hard to detect such cases yet.
7.4 Experimental Details
7.4.1 Baseline Methods
Here we explain implementation details of the baseline methods that were used for comparisons in Section [5] First, we'd like to point out that we used relatively a short time horizon to collect experience, and used the critic network for bootstrapping, instead of collecting experience for the
whole trajectory for all the methods. Therefore, the collected experience could be biased. However, we chose to implement in this way, because allows faster learning than collecting the whole trajectory in terms of number of simulation steps.
19

---

LR We can estimate original LR gradient as follows, using log- log-derivative trick [Wiliiams and Peng
1989
ynn
1990
@m(
B 6p Bg.......
[(50, 60
log (50, ao)
00
00
1=0
Beg 00.
Σ t=
L(30, 00, log no (50, ao)
00
where L(so. Ao.... - 7rr(st, ar) is the discounted accumulated return. However, this LR gradient often suffers from high variance, because L varies lot from trajectory trajectory. To reduce such variance and faithfully compare our method to LR gradient- method, we decided to use advantage function in the place of L, and particularly used GAE [Schulman al. (015b) as the advantage function as we used for PPO. After calculating this LR gradient, we take gradient ascent step with pre-defined learning rate.
RP By using RP gradient in Appendix 7.2.2 we can perform gradient ascent as we did in LR. Also, because we use short time horizon and critic network, this method is very similar to SHAC [Xu et al. 2022 Please refer to the paper for more details.
Our PPO implementation is based on that RL Games Makkvvichuk and Makoviychuk 2022). However, to be as fair as possible, used the same critic network as the other methods, instead of using that implemented in the RL library. Also, we omitted several other additional losses that are not vital PPO's formulation, such entropy loss.
PPO
LR+RP After we compute LR (V LR) and RP (V RP) gradient as shown above, we can interpolate them using their sample variance as follows Parmas et al. 2018).
LR+RP VL KLR + RP (1 KLR),
ShP o ORP + oR
SLR
where dR and d.n are sample standard deviation of RP and LR gradients. We gain these terms by computing trace covariance matrix of the sample gradients from different trajectories.
Since we have to compute this sample statistics, we have to do multiple different backpropagations for different trajectories, which incur lot of computation time. Also, we found out that computing covariance matrix is also time consuming when controller has large number of parameters. There- fore, decided and also truncate the gradient to smaller length (512) to facilitate computation.
only limited number of sample gradients (16) compute sample variance,
PE
We tried to faithfully re-implement policy enhancement scheme of Qiao al. 2021
7.4.2 Network architecture and Hy Hyperparraeters
In this section, we provide network architectures and hyperparameters that we used for experiments in S For each of the experiments, we used the same network architectures, the same length of time horizons before policy update, and the same optimization procedure for critic updates, etc. We present these common settings first for each of the problems.
For GI-PPO, there are hyperparameters for update towards a-policy, and those for PPO update. We denote the hyperparameters for a-policy update by appending (a), and those for PPO update by appending (PPO). For the definition of hyperparameters for a-policy update, please see Algorithm for details.
Function Optimization Problems (Section 5.1 follows.
For these problems, common settings are as
Actor Network: MLP with [32, 32] layers and ELU activation function
20

---

Critic Network: MLP with [32, 32] layers and ELU activation function
Critic Hyperparameters: Learning rate 10-3, Iterations 16, Batch Size
Number of parallel environments: 64
Horizon Length:
7 (Discount factor): 0.99
T (GAE): 0.95
Hyperparameters for LR follows.
Learning Rate: Dejong(1), Dejong(64)= 10-3, Ackley(II) 10-4, Ackley(64) = 30-
10
Learning Rate Scheduler: Linea
Hyperparameters for RP are as follows.
Learning Rate: Dejong(1). Dejong(64) 10-2, Ackleyy()), Ackley(64) 10-3
Learning Rate Scheduler: Linear
Hyperparameters for LR+RP are follows.
Learning Rate: Dejong(1), Dejong(64)= 10 Ackley(1) 10-A, Ackley(64) = 10-4
Learning Rate Scheduler: Linear
Hyperparameters for PPO are as follows.
Learning Rate: Dejong(1), (1), Ackley(1) 10 Dejong(64), Ackley(64) 10-2
Learning Rate Scheduler: Constant
Batch Size for Actor Update: 64
Number of Epochs for Actor Update: 5
Eclipi 0.2
Hyperparameters for GI-PPO are as follows.
(a) Learning Rate: 10
(o) Batch Size for Actor Update: 64
(a) Number Epochs for Actor Update:
(a) og: 10-5
(a) max(a): 1.0
(a) 8: 1.1
(a) 6 0.4
(a) 8 0.5
(PPO) Learning Rate: Dejong(1), Ackley(1) 10-4, Dejong(64), Ackley(64) 10-2
(PPO) Learning Rate Scheduler: Constant
(PPO) Batch Size for Actor Update: 64
(PPO) Number of Epochs for Actor Update:
(PPO) Ectipi 0.2
"Learning rate decreases linearly the minimum value learning progresses.
21

---

Differentiable Physics Problems (Section 5.2 For these problems, common settings are as follows.
Actor Network: MLP with [64, 64| (Cartpole), and [128, 62,32] (Ant, Hopper) layers and ELU activation function
Critic Network: MLP with [64, 64] layers and ELU activation function
Critic Hyperparameters: Learning rate 10 10-3 (Cartpole,, 2- 1-100- (-1t)) and 2- 1110000HHHHHH)) Iterations 16, Batch Size
Number of parallel environments: (A(Cttp,,,,, Ant), 256(Hopper)
Horizon Length: 32
(Discount factor); 0.99
T (GAB): 0.95
Hyperparameters for LR are as follows.
Learning Rate: 10-1
Learning Rate Scheduler: Linear
Hyperparameters for RP are as follows.
Learning Rate: Cartpole 10-2, 10-2,An,, Ant, Hopper 2 10-3
Learning Rate Scheduler: Linear
Hyperparameters for PPO are as follows.
Learning Rate: Cartpole = 3 10 Ant, Hopper = 10
Learning Rate Scheduler: Cartpole = Adaptive Ant, Hopper = Constant
Batch Size for Actor Update: 2048
Number of Epochs for Actor Update: 5
Eclip: 0.2
Hyperparameters for GI-PPO are as follows.
(a) Learning Rate: Cartpole 10-2, Ant 10-4, *.Hopper 5 -0
(a) Batch Size Actor Update: Cartpole, Ant 2048, Hopper 8192
(a) Number of Epochs for Actor Update: 16
(a) og: Cartpole, Ant 5 10-1 Hopper = 10-3
(a) max(a): 1.0
(a) 3: 1.02
(a) daei: 0.4
(a) 8 Cartpole, Hopper 0.75, Ant 0.5
(PPO) Learning Rate: Cartpole 10-4 Ant, Hopper -
(PPO) Learning Rate Scheduler: Cartpole Adaptive, Ant, Hopper Constant
(PPO) Batch Size for Actor Update: 2048
(PPO) Number Epochs for Actor Update:
(PPO) Ediip: 0.2
LLeannng rate is adaptively controlled, so that the KL divergence between the updated policy and the original policy is maintained certain value, 0.008 in this case.
22

---

Traffic Problems (Section 5.3
For these problems, common settings are as follows.
Actor Network: MLP with [512, 64, 64] layers and ELU activation function
Critic Network: MLP with [64, 64] layers and ELU activation function
Critic Hyperparameters: Learning rate = 10-3, Iterations = 16, Batch Size =
Number of parallel environments: 64
Horizon Length: 32
7 (Discount factor): 0.99
(GAE): 0.95
Hyperparameters for LR are follows.
Learning Rate: 3 10
Learning Rate Scheduler: Linear
Hyperparameters for RP are as follows.
Learning Rate: 10
Learning Rate Scheduler: Linear
Hyperparameters for LR+RP are as follows.
Learning Rate: 3 10-4
Learning Rate Scheduler: Linear
Hyperparameters for PPO are as follows.
Learning Rate: 3 10-4
Learning Rate Scheduler: Constant
Batch Size for Actor Update: 2048
Number of Epochs for Actor Update: 5
Celip: 0.2
Hyperparameters for GI-PPO are as follows.
(a) Learning Rate: 10
(a) Batch Size for Actor Update: 2048
(a) Number of Epochs for Actor Update: 16
(a) ag: 10
(a) max(a): 1.0
(a) 3: 1.1
(a) d 0.4
(a) 8 0.5
(PPO) Learning Rate: 3- 10
(PPO) Learning Rate Scheduler: Constant
(PPO) Batch Size for Actor Update: 2048
(PPO) Number of Epochs for Actor Update: 5
(PPO) Ediip: 0.2
7.5 Problem Definitions
Here present details about the problems suggested in Section
23

---

10
-4
160
125
10
De Jong's Function
Ackley's Function
Figure 8: Landscape of target functions in 2 dimensions.
7.5.1 Function Optimization Problems (Section 5.1
(N-dimensional) De Jong's s function (FD) and Ackley's function (FA) are defined for n-diimnnsional vector z and are formulated as follows [Molga and Smutnicki 2005
FD(x) Σ
2 x
Fa(x) = -20 exp(--0.2 -0.2
x3
n
i=1
expl
(2s(2 20 exp(1).
As mentioned in Section 5.] multiply to these functions to make these problems maximiza- tion problems. Also, even though these two functions have their optimum at = 0, they exhibit very different landscape as shown Figure g
When comes to the formal definition as RL environments, we can define these problems as follows.
Episode Length: 1
Observation: [0]
Action: n-dimensional vector x, all of which element is in 1-1,
Reward: De Jong's Function (5.1112x)) Ackley's Function F3(32.768x)
7.5.2 Traffic Problems (Section 5.3
Before introducing pace car problem, which we specifically discussed in this paper, we'd like to briefly point out the traffic model that used to simulate motions of individual vehicles in the traffic environment.
Traffic Model In our traffic simulation, the state R2N is defined as concatenation of all vehicle states, On E IR?, where S n S N stands for vehicle qa can be represented simply with the vehicle's 's position (æ,) and velocity (v,).
24

---

In
9n
Un
[
T ON
where n is the vehicle ID. Since this traffic state changes over time, we represent the traffic state at timestep st. Then our differentiable traffic simulator capable of providing the gradient of the simulation state at the next timestep t + 1 with respect to the state at the current timestep t. or dru1, dar This is possible because we use the Intelligent Driver Model (IDM) model car-following behavior in our simulator (Treiber el al. 2000), which is differentiable. IDM describes vehicle speed e and acceleration acceleration comfortable deceleration b, the minimum distance between vehicles in meters 6, vehicle length I, and difference in speed with the vehicle in front Av, as follows:
function of desired velocity vo. safe time headway in meters T, maximum
to dæa
Var
du
Un
6
88 (V= Awa) 2
0a dt
1
8a
Sa Ta-1
Fa la-1,
Va Av
s" (Va: Ava) = so + Uo T +
Va
Ava = va - aa-1
where a means the order of the vehicle in the lane. Therefore, (a 1 1)-th vehicle runs right in front of a-th vehicle, and this relationship plays an important role in IDM. Also note that the computed acceleration term (+) is differentiable with respect to the other IDM variables.
IDM is just one of many car-following models used in traffic simulation literature. We choose IDM for its prevalence in previous mixed autonomy literature, however, any ODE-based car-following model will also work in our simulator as far as it is differentiable. In our simulator, automatic differentiation governs the gradient computation.
Lane Under IDM, lane membership is one of the most vital variables for simulation. This is because IDM builds on the leader-follower relationship. When vehicle changes lane with lateral movement, as shown as red arrows in Figure ? such relationships could change, and our gradients do not convey any information about it, because lane membership a discrete variable in nature. Note that this lateral movement also affects the longitudinal movement, rendered in green arrows in Figure 9 vehicle, and gradient from it is valid as far as the lane membership does not change. However, when the lane membership changes, even if our simulator gives gradient, it tells only "partial" information in the sense that it only gives information about longitudinal behavior. Therefore, could say that analytical gradients get from this environment is "incomplete" (not useless at all, because it still gives us information about longitudinal movement), and thus biased.
Pace Car Problem In this problem, there is single autonomous vehicle that we have to control to regulate other human driven vehicles, which follow IDM, to run at the given target speed. Since human driven vehicles control their speeds based on the relationship to their leading vehicles, our autonomous vehicle has to control itself to run in front of those human driven vehicles and adjust their speeds to the target speed. The number of lanes and human driven vehicles varies across different environments as follows.
Single Lane: Number of lanes 1, Number of vehicles per lane
2 Lanes: Number of lanes 2. Number of vehicles per lane 2
4 Lanes: Number of lanes 4, Number of vehicles per lane 4
10 Lanes: Number of lanes 10. Number of vehicles per lane 1
25

---

See Figure TO for 10-lane environment.
Then formally define RI. environments follows, where N is the number of human driven vehicles in total, and Ugtt is the target speed.
Episode Length: 1000
Observation: E IR2N
Action: IR2 (ag = Acceleration, a Steering)
Reward:
1 Σ 1= min( N
1)
Vegt
However, to facilitate learning, additionally adopted following termination conditions to finish unpromising trials early.
Terminate when autonomous vehicle collides with human driven vehicle
Terminate when autonomous vehicle goes out of lane
Terminate when autonomous vehicle runs too far away from the other vehicles
Terminate when autonomous vehicle falls behind the other vehicles
When one of these conditions met, it gets reward of and the trial terminates.
7.6
Renderings
7.6.1
Traffic Simulation
Gradient flow As described in Appendix 7.5.2, analytical gradients are only valid along longitudinal direction, as illustrated with green arrows in Figure When a vehicle changes lane with lateral movement, as shown with red arrows, even though such change would incur different behaviors of following vehicles in multiple lanes, analytical gradients cannot convey such information. However, they still tell information about longitudinal behavior which is one of the motivations of our research, to use analytical gradients even in these biased environments.
Figure 9: Traflic Environment. In traffic flows, only partial gradients of forward simulation are known; acceleration along a (longitudinal) traffic lane is continuous and admits gradient flow (green), while (lateral) channing a discrete event and thus prohibits gradient flow (red). Analytical gradients convey only partial information about traffic dynamics.
Pace Car Problem We additionally provide renderings of the 10-lane pace car problem defined in Appendix 7.5.2. In Figure TO] we can observe 10 parallel lanes, 10 human driven vehicles rendered in yellow, and the autonomous vehicle rendered in blue that we have control. As shown there, human driven vehicles adjust their speeds based on IDM when the autonomous vehicle blocks their way. Therefore, the autonomous vehicle has to learn how to change their lanes, and also how to run in appropriate speed to regulate the following vehicles. Our experimental results in Section 5.3 show that our method achieves better score than the baseline methods by adopting biased analytical gradients in PPO.
26

---

Mean Speed: 18.16 m/s
Mean Speed: 16.70 m/s
(a) 0-th frame
(b) 150-th frame
Mean Speed: 19.58
Mean Speed: 19.84 m/s
300-th frame
450-th frame
Figure 10:
10-Lane pace car environment. The autonomous vehicle (Blue) has to interfere paths of the other human driven vehicles (Yellow) to limit their speeds to 10m/s. Even though a hard problem to achieve high score, our method achieved far better score than the baseline methods.
7.7 Computational Costs
We provide wall clock training times for each method in traffic environments to finish all the training epochs in Table 2 All of the trainings were done based on the settings in Appendix 7.5.2 Note that the longer training time does not always mean that the method is worse than the others the method could have achieved better results in shorter time, while requiring much longer time to complete all the training epochs.
In Table we can observe that GI-PPO requires little more time than RP, as computes analytical gradients as RP does, but requires additional time for controlling and doing PPO update. However, the additional cost is not so significant. In contrast, LR+RP consumes much more time than the other methods, even though it is based on the same spirit ours. This is because it has to backpropagate through each trajectory in the experience buffer to estimate the sample variance the gradients. Therefore, lower computational cost.
that method attains better results than LR+RP (Figure 4 with
Table 2: Average (wall clock) training time (sec) for traffic problems.
Problem
LR
RP
PPO
LR+RP
GI-PPO
Single Lane
120
282
181
1335
332
Lanes
142
330
218
1513
411
Lanes
304
646
366
2093
813
10 Lanes
244
496
294
2103
533
27


Reviews:
[{"review_id": "QZur4811Oz", "reviewer": "Reviewer_xDc8", "summary": "This paper introduces a policy gradient algorithm based on Proximal Policy Optimization (PPO) algorithm. This new algorithm utilizes the analytical gradients from differential environments and achieves competitive performance in various scenarios including function optimization, classic control environments, and traffic control environments. \nTo integrate the analytical gradient into the PPO framework, they use the reparameterization trick and introduce the concept of $\\alpha$-policy which is a locally superior policy. When $\\alpha$ is sufficiently small, the difference between the original policy and this $\\alpha$-policy would be small enough to satisfy the difference constraint in the PPO algorithm. Moreover, they propose metrics considering variance, bias, and out-of-range-ratio to dynamically adjust $\\alpha$ during the training.", "questions": "1. Line 419 \"Proof of Definition 5\" does not match the actual definition index. The author should separate definition 4.1 into a definition and a proposition. \n\n2. Can the author integrate the variance reduction techniques in [Xu et al., 2022] to achieve better performance? If not, what changes can be made to accommodate this?", "limitations": "I did not identify any potential negative societal impact from this work.\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of utilizing analytical gradients in physical control environments is powerful and can dramatically increase the efficiency of policy gradient algorithms when analytical gradients are available. This paper advances this idea by integrating it with the PPO algorithm and achieves competitive performance in various environments.\n\n2. This paper proposes the concept of $\\alpha$-policy and connect it with the reparameterization trick. $\\alpha$-policy is an adjustable policy such that when $\\alpha$ is small enough the difference between $\\alpha$-policy and the origin policy can be small enough to fit into PPO algorithm. \n\n3. This paper proposed metrics considering variance, bias, and out-of-range-ratio to dynamically adjust $\\alpha$ during training. ", "weaknesses": "1. When trying to fit the analytical gradients into the PPO algorithm, this paper didn't consider dynamically changing the PPO clip limit. Thus, the analytical gradients are not fully utilized.\n\n2. More theoretical analysis and better ways on how to adjust $\\alpha$ during training could be done and proposed.\n\n3. The experimental performance showed improved performance compared to PPO but does not achieve state-of-art performance in some environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and support, we really appreciate them. Here we address some of your concerns.\n\n**Q1. The analytical gradients are not fully utilized, because this paper didn't consider dynamically changing the PPO clip limit.**\n\nA1. Please refer to A1 of global rebuttal at the top.\n\n**Q2. More theoretical analysis and better ways on how to adjust $\\alpha$ during training could be done and proposed.**\n\nA2. Please refer to A3 of global rebuttal at the top.\n\n**Q3. The experimental performance showed improved performance compared to PPO but does not achieve state-of-art performance in some environments.**\n\nA3. Please refer to A2 of global rebuttal at the top.\n\n**Q4. Line 419 \"Proof of Definition 5\" does not match the actual definition index. The author should separate definition 4.1 into a definition and a proposition.**\n\nA4. Sorry for the confusion, and thank you for pointing it out. Definition 5 means Definition 4-1. We will fix it in the revised version, and separate the definition into definition and proposition as you commented for better understanding.\n\n**Q5. Can the author integrate the variance reduction techniques in [Xu et al., 2022] to achieve better performance? If not, what changes can be made to accommodate this?**\n\nA5. Thank you for the suggestion! In fact, **we already applied techniques in [Xu et al., 2022] to the analytical gradients that we used for GI-PPO in this work**. To be specific, we used the truncated time window suggested in the paper to estimate the analytical gradients and the following $\\alpha$ policy. We will add additioinal experimental results that show how the length of truncated time windows affect the training results in the supplementary material."}, {"review_id": "fIJWTCPoci", "reviewer": "Reviewer_WNVi", "summary": "In this manuscript, the author(s) combined the PPO framework with analytical gradients and proposed a policy learning method to learn better policy quickly. Through empirical experiments, the author(s) demonstrated that the proposed method was superior to baseline methods in several scenarios.\n\n\n", "questions": "\nDetailedly, I have the following comments/questions:\n\n1) For the description or empirical experiment details, some places in this manuscript seem unclear. For example,\n\n(1) Definition 5 was mentioned several times in the manuscript, such as, in Lines 179, 188, and even in the Supplementary Materials, Line 419, but where is Definition 5? \n\n(2) The proof is mainly for Proposition, Lemma, Theorem, etc. In Line 419 of Supplementary Materials, it mentioned \"Proof of Definition 5\", I am not clear. \n\n(3) Could you please explain/clarify the fluctuation of the blue curve (i.e., GI-PPO) in Figure 3 (c)? Why caused this phenomenon?\n\n(4) The order number (or citation) of Figures in this manuscript is messy. For instance, in Line 229, Figure 8 is in the Supplementary Materials, right?  Figure 1 seems to be cited until Line 245?\n\n2) If possible, some theoretical results, for example, Proposition 4.5, can be verified by empirical experiments?\n\n3) Some other tiny issues/typos:\n\n(1) The font size used for the coordinates in Figure is too small. \n\n(2) The format of mathematical formulas in this manuscript is inconsistent. Some end with punctuation, and some don't.\n\n(3) Many issues in References. For instance, Sometimes the conference name is the full name plus the abbreviation, and sometimes only the full name; Missed source of the cited references; and so on. Hope the author(s) will check carefully and correct them.\n\nIn addition, I did not scrutiny the proofs step-by-step, but I think the proofs should be Ok.\n\nThank the author(s) for submitting this interesting work. Except for the several possible issues/questions mentioned above. I think this is a basically good manuscript. I look forward to the response from the author(s). If the author feels any of my comments are inappropriate, please feel free to point them out :-) Thanks.\n\n\n\n", "limitations": "Yes. As the author(s) revealed, there might be several limitations, such as(1) How to approximate $\\alpha$-policy efficiently; (2) Adjusting PPO's clipping range; (3) Computational efficiency. Hope that the author(s) can further investigate these limitations and solve these potential problems in the near future. Good luck!\n\n\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Strong points:\n\n1) Utilizing $\\alpha$-policy to deal with the influence of analytical policy gradients when training;\n\n2) Adopting criteria to detect/calculate variance and bias;\n\n3) An algorithm was designed to handle the strength of analytical gradients during updating;\n\n4) Empirical results look good.\n\nBy the way, in the Supplementary Materials, the author(s) provided a mp4 video file, where the animation on the scenario of the traffic problem looks fantastic. Thanks for providing that. \n\n\n", "weaknesses": "### Weak points:\n\nAlthough the work of this manuscript looks interesting, some points in this manuscript are not so clear to me, I hope the author(s) can explain/detail them. Thanks.\n\n1) Some descriptions of empirical experiments are not so clear.\n\n2) Missing codes related to the proposed method in this manuscript. \n\n3) If possible, some theoretical results in this manuscript should be empirically verified. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and support, we really appreciate them. Here we address some of your concerns.\n\n**Q1. Where is Definition 5, and what does it mean by proof of Definition 5?** \n\nA1. Sorry for the confusion, Definition 5 means Definition 4-1. We will fix it in the revised version, and separate it into definition and proposition. Right now Definition 4-1 defines what $\\alpha$ policy is, and suggests some mathematical properties of it (which should be proposition), which we prove in the \"Proof of Definition 5\". We will clearly mention that the proof in line 419 is for the proposition part of Definition 4-1.\n\n**Q2. Could you please explain / clarify the fluctuation of the blue curve (i.e., GI-PPO) in Figure 3 (c)?**\n\nA2. Thank you for the question, the 1-d Ackley’s function that is dealt with in Figure 3 (c) has its optimum at $x = 0$. Therefore, the best policy would give us a mean of 0 and variance of 0 for this problem, as shown in Figure 1 (a). In Figure 1 (a), we can observe that the optimum is formed at mean = 0, var = 0, **but the optimum is very unstable** - that is, small perturbations in either mean or variance lead to a large degradation in the results. The fluctuation in the blue curve in Figure 3 (c) stems from this nature of the target function. However, since we use Adam optimizer, which adjusts its step size based on the optimization trajectory, we can observe that the fluctuation disappears in the end.\n\n**Q3. The order number (or citation) of Figures in this manuscript is messy.**\n\nA3. Thank you for pointing this out.   Due to the page limit, the figures are not always ordered and appear, as desired.  We will rearrange the figures in their placement within the latex file to better align figures with the page break in the manuscript, as suggested. \n\n**Q4. If possible, some theoretical results, for example, Proposition 4.5, can be verified by empirical experiments?**\n\nA4. Thank you for the suggestion, we can definitely verify it with additional experiments. We will incorporate those results in the final version.\n\n**Q5. There are miscellaneous typos / issues in the writing.**\n\nA5. Thank you for the detailed feedback!  We will further polish our exposition to incorporate your suggestions -- very much appreciated!"}, {"review_id": "2TlRFwn09g", "reviewer": "Reviewer_8kSu", "summary": "This paper integrates analytical gradients with PPO. They introduce an \\alpha-policy to control the bias and variance during the update of the policy. Results on some differentiable environments show the effectiveness of the proposed method.", "questions": "- What do you mean by inherent chaotic nature (Line 25)? Could you please give some examples?\n- I'm confused about how is eq. 2 used in your method.\nPlease also see my comments in the weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "This paper study how to incorporate analytical gradient into existing Rl algorithm, which is interesting.", "weaknesses": "The main weakness of the paper is that the writing of the paper is somewhat poor. \nThe presentation lacks motivation or necessary intuitive explanations. There are some imprecise statements and typos (see below).\nThese flaws make the paper hard to follow.\n\nMajor concerns:\n- This paper claims that it's a combination of analytical gradient and PPO. However, I didn't see where you use PPO (eq. 4 is not PPO!).\n- Some statements lack motivation. I don't understand why the algorithm is composed of three parts (Line 182). Could you give more explanation? And why the adjusting $\\alpha$ should be designed in that way? Besides, it would be good if there was more explanation of Def 4.1\n- The proposed method does not perform well on the physical simulation tasks. In Figure 4, the proposed method performs worse than the RP method on two high-dimensional tasks: Ant and Hopper.\n- In most of the practical tasks, it is hard to get the analytical gradient. This limits the potential of the proposed method. The paper only does experiments in simulation tasks. Could you give more examples of where we can get analytical gradients in practice?\n\nThere are some minor technical issues in the paper:\n- Line 106: The symbol $A$ is repeated with the action space $A$ (in Line 97).\n- Line 97: You assume the probability to be within $[0,1]$. This means that the action space is discrete. However, in Line 101, you assume action space is continuous.\n\n\nSome statements in the paper are wrong:\n- \"PPO relies on the observation that we can evaluate a policy πθ with our current policy πθ¯\". This is not right. Because the $\\rho_{\\pi_\\theta}(s)$ is not available.\n- Line 137: \"the difference between πθ¯ and πθ must be small enough to get an accurate estimate.\" In fact, in the theory of TRPO, they introduce a lower bound with KL divergence penalty term. They don't care whether that's an accurate estimate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.  We address some possible confusions below.\n\n**Q1. How is PPO used in this paper? Eq. 4 is not PPO, and description about PPO, especially that related to availability of $\\rho_{\\pi_{\\theta}}(s)$, is wrong.**\n\nA1. Though the exact formulation is different, the **main objective function of both TRPO and PPO is same as Eq. 4**. The exact formulation differs between these two algorithms because they opt for different ways to constrain the policy optimization space. To be specific, if we see Eq. 14 in [1] (TRPO), we can observe that the objective function is the MC estimate of our Eq. 4, and it imposes KL divergence constraint to limit optimization space. For PPO, as seen Eq. 7 in [2], we note that it also optimizes MC estimate of our Eq. 4, but clips the estimates for constraining optimization space. Therefore, we'd like to suggest that Eq. 4 explains the objective function of PPO well, and thus is a good theoretical basis to show how we can incorporate analytical gradients to PPO framework.\n\nAdditionally, we note that $\\rho_{\\pi_{\\theta}}(s)$ is not available, given collected experience with $\\pi_{\\bar{\\theta}}$. This is the **exactly the reason why we use $\\rho_{\\pi_{\\bar{\\theta}}}(s)$ instead of $\\rho_{\\pi_{\\theta}}(s)$, as shown in Eq. 4 and Eq. 3 in [1]**. If we replace $\\rho_{\\pi_{\\theta}}(s)$ with $\\rho_{\\pi_{\\bar{\\theta}}}(s)$, then it becomes the first order approximation of the expected return of $\\pi_{\\theta}$ [3]. In that sense, we stated that \"we can evaluate a policy $\\pi_{\\theta}$ with our current policy $\\pi_{\\bar{\\theta}}$\", even if we do not have access to $\\rho_{\\pi_{\\theta}}(s)$.\n\n**Q2. Why is the algorithm composed of three parts as shown in Line 182?**\n\nA2. We’d like to reemphasize that we use **variance and bias of analytical gradients** to control their role in policy update. It has been already used in several other literature [4], [5] as valid criteria to evaluate the validity of (analytical) RP gradients against LR gradients. However, **since we do not have LR gradients to compare, we need another measure to evaluate the variance and bias of analytical gradients**. \n\nTo that end, we suggested $\\alpha$ policy. Based on our propositions, **we can evaluate the variance and bias of analytical gradients if we have $\\alpha$ policy**. Therefore, we first need to update our policy toward $\\alpha$ policy - which is the first step of our algorithm. After approximating $\\alpha$ policy, we evaluate the variance and bias of analytical gradients using criteria suggested in the paper, which is the second step. In this step, if our criteria is not met, we then decrease $\\alpha$ so that analytical gradients do not contribute to the policy update much in the next iteration. Finally, we conduct policy updates based on PPO - in fact, as we have shown in the paper, this step is not necessary if the analytical gradients are correct, because $\\alpha$ policy is locally better than the current policy in PPO viewpoint (Proposition 4.2). However, if the analytical gradients are flawed and thus $\\alpha$ goes to 0, our policy update will be very slow without PPO. Therefore, we can **consider PPO as a safeguard, which guarantees certain amount of policy update even when the analytical gradients are not reliable**. \n\n**Q3. Why $\\alpha$ is adjusted in the way suggested in the paper?**\n\nA3. Please refer to A3 of global rebuttal at the top.\n\n**Q4. It would be good if there was more explanation of Def 4.1.**\n\nA4. Def 4.1 introduces the concept of $\\alpha$ policy, which can be considered as a **new policy that selects slightly better action than the old policy with the same probability (intuitively)**. We will add more explanation, and move Figure 8 in Appendix 7.6.1 to the main paper if possible to help understanding.\n\n**Q5. The proposed method does not perform well on the two physical simulation tasks.**\n\nA5. Please refer to A2 of global rebuttal at the top.\n\n**Q6. How can we apply this method to practical tasks?**\n\nA6. Please refer to A4 of global rebuttal at the top.\n\n**Q7. Minor technical issues in Line 97 and 106**\n\nA7. Thank you for pointing them out.  We will correct them in the revised version.\n\n**Q8. Line 137: Wrong description about $\\pi_{\\bar{\\theta}}$ and $\\pi_{\\theta}$.**\n\nA8. TRPO [1] suggests the lower bound based on the observation that they can use $L_{\\pi_{old}}(\\pi_{new})$ to approximate the desired $\\eta(\\pi_{new})$ up to the first order. Furthermore, in their practical algorithm, they constrain the optimization space by upper bounding KL divergence, instead of using it as a penalty term. Likewise, **their logic is grounded on the fact that we can locally approximate the expected return of the new policy using the old one**, and in this sense, we'd like to respectfully suggest that our description was indeed correct. \n\n**Q9. What is inherent chaotic nature?**\n\nA9. The term in this context mainly describes exploding variance of the analytical gradients. We'd like to introduce related papers that discuss this topic in detail: [4], [6].\n\n**Q10. How Eq. 2 is used is confusing.**\n\nA10. Eq. 2 shows the basic gradients that our differentiable environments provide us. With those gradients, we can compute gradients of advantage value w.r.t. the actions ($\\frac{dA}{da}$). If we use GAE for estimating advantage, we can compute the gradients using formulas provided in Appendix 7.2.1. In the formula, note that we use $\\frac{d \\delta}{da}$, which can be computed using the basic gradients in eq. 2, as $\\delta$ is defined using $r$ and $s$.\n\n[1] Schulman, John, et al. \"Trust region policy optimization.\"\n\n[2] Schulman, John, et al. \"Proximal policy optimization algorithms.\"\n\n[3] Kakade, Sham M. \"A natural policy gradient.\"\n\n[4] Parmas, Paavo, et al. \"PIPPS: Flexible model-based policy search robust to the curse of chaos.\"\n\n[5] Suh, Hyung Ju, et al. \"Do differentiable simulators give better policy gradients?.\"\n\n[6] Metz, Luke, et al. \"Gradients are not all you need.\""}, {"review_id": "21D0UdTDPa", "reviewer": "Reviewer_DV21", "summary": "This paper introduces a novel policy learning method that adopts analytical gradients into the PPO framework without the necessity of estimating LR gradients. The authors introduce an adaptive α-policy that allows us to dynamically adjust the influence of analytical gradients by evaluating the variance and bias of these gradients. The experiments show that the proposed method, GI-PPO, strikes a balance between analytical gradient-based policy updates and PPO-based updates, yielding solid results compared to baseline algorithms in various environments.\n", "questions": "This paper is well written in general. However, for the readers to better understand the algorithm, can we move Algorithm 1 GI-PPO from Appendix and add into main paper? To utilize analytic gradients, it seems GI-PPO needs more extra computations from back-propagation. Can the author provide the computational cost comparison to different baseline algorithms?\n", "limitations": "There is no code provided to reproduce the results.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces original ideas about how to incorporate analytical gradients into the PPO framework, taking their variance and bias into account. As a result, a locally superior policy, adaptive $\\alpha$-policy is introduced and fits into PPO framework seamlessly, by dynamically adjusting the influence of analytical gradients. This paper is of solid technical quality. It provides theoretical derivations and analysis to demonstrate how $\\alpha$-policy fits into PPO framework seamlessly, the relationship between reparameterization RP gradient and $\\alpha$-policy, and theoretical guidance to adjust $\\alpha$ based on variance and bias. The experiments are extensive in general, demonstrating the solid performance compared to baseline in different environments. Last, the paper is clearly written in general.\n\n\n", "weaknesses": "In Section 5.1 (page 7, line 229), Figure [8] is missing in main paper, although we may find the figure in Appendix. In Fig. 3, the plot is too small to distinguish the relative performance for different algorithms. In Section 5.2 (page 8), there is typo line 259,  it should be Figure [4] instead of Figure [5] for Differential Physics Simulation. In Fig. 4(b,c), GI-PPO is no better than RP for tasks Ant and Hopper in term of returning rewards. To strengthen the impact of the work, it might be better to add more challenging tasks, for example, Humanoid in Mujoco environment. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and support. Here we address your key concerns.\n\n**Q1. In Section 5.1, Figure 8 is missing in main paper, although we may find the figure in Appendix. Also, the plots in Figure 3 are too small to distinguish the relative performance of different algorithms.**\n\nA1. Thank you for the comment.  We will rearrange the figures to better align with the manuscript text, and enlarge them as well.  The current appearance is mainly due to the page limit and latex placement. \n\n**Q2. There is a typo in line 259, it should be Figure 4 instead of Figure 5 for Differential Physics Simulation.**\n\nA2. Thank you for finding the typo! We will fix it in the revised version.\n\n**Q3. In Figure 4, GI-PPO is no better than RP for tasks Ant and Hopper in term of returning rewards.**\n\nA3. Please refer to A2 of global rebuttal at the top.\n\n**Q4. To strengthen the impact of the work, it might be better to add more challenging tasks, for example, Humanoid in Mujoco environment.**\n\nA4. Thank you for the suggestions to further improve the exposition of this paper.  We will add those results in the revised paper.\n\n**Q5. Can we move Algorithm 1 GI-PPO from Appendix and add into main paper?**\n\nA5. We can definitely do that, but for now the page limit makes it hardly possible. If we can get additional pages, we will move it into main paper from the supplementary document.\n\n**Q6. Can the author provide the computational cost comparison to different baseline algorithms?**\n\nA6. Thank you for the question, we can add computational cost comparison in supplementary material for sure.  Briefly, GI-PPO only needs additional computational cost for PPO-based update than the RP method. Considering that the most of the computational cost comes from simulation steps and backpropagation for RP, such additional cost is not much."}, {"review_id": "eDcE7r0ppt", "reviewer": "Reviewer_39SH", "summary": "This paper studied the combined use of both the analytical policy gradient and the likelihood ratio policy gradient for training policy networks, based on the PPO algorithm. To make the combination feasible, a new alpha-policy is introduced and its approximation technique has been successfully developed. Besides some theoretical studies, empirical results further show the potential usefulness of the newly proposed algorithm in solving some benchmark problems.", "questions": "What are the possibilities and potential limitations of using the proposed policy gradient combination technique on other algorithms, particularly off-policy algorithms?\n\nWhat is the practical impact of the assumption regarding the prior knowledge of the environment dynamics that must be differentiable?\n\nWhat are the theoretical and practical advantages of the new algorithm design, compared to other model-based reinforcement learning methods?\n\nIncompleteness and bias present two separate dimensions regarding the quality of analytical policy gradients. Why should they be strongly correlated?", "limitations": "I do not have any concerns regarding this question.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The theoretical study in this paper regarding the analytical policy gradient and the alpha-policy sounds interesting and novel. The experiment results show that the new algorithm can be very useful on some benchmark problems.", "weaknesses": "While it is interesting to mix analytical policy gradients with learned policy gradients to enhance the reliability and performance of the policy network training algorithms, this idea is clearly not restricted to on-policy algorithms such as PPO. Although the authors made it clear that PPO is one of the most popularly used on-policy algorithms, it remains questionable why this paper only studies the effectiveness of using combined policy gradients in PPO. The possibility and potential limitations of using the proposed policy gradient combination technique on other algorithms, particularly off-policy algorithms, may need to be further justified and investigated.\n\nThis paper requires prior knowledge of the environment dynamics that must be differentiable (or partially differentiable) in nature. Many real-world reinforcement learning problems may not satisfy this requirement. Hence, the practical usefulness of the new algorithm remains a bit questionable. The authors are highly recommended to clearly evaluate and justify the practical value of the new algorithm.\n\nFurthermore, if the environment model is known in advance, it is possible to conduct effective model-based policy training without using analytical policy gradients. It remains unclear to me what the advantages of using analytic policy gradient techniques would be, compared to planning and other model-based reinforcement learning techniques. It is also desirable if the performance strength of using the analytic policy gradients, compared to other model-based reinforcement learning methods, can be experimentally evaluated and reported in this paper.\n\nThe authors stated several times in the paper that the analytical policy gradients would be highly biased when they are incomplete. I don't understand what this means and why it is true. Incompleteness and bias present two separate dimensions regarding the quality of analytical policy gradients. Why should they be strongly correlated?\n\nSome mathematical claims require more clarity. For example, Lemma 4.4 mentioned a particular advantage function A-hat. However, the actual definition of this function is not presented, making it hard to accurately understand this lemma and its applicability.\n\nSome critical details were missing regarding the new algorithm design in the main text. For example, the new algorithm requires you to adjust alpha from time to time. However, it is not clear how alpha is actually adjusted. The formula for adjusting alpha is not presented and clearly justified in the main text. Meanwhile, some claims in the algorithm design also need strong justifications. For example, it is stated on page 6 that by constraining the determinant of a matrix to be near 1, we can guarantee stable policy updates. However, the validity of this statement is not proved or properly explained. Moreover, if all eigenvalues are far greater than 1, alpha would be set to be very close to 0. In this case, would the analytical gradient become useless? Similarly, it is not clear why the ratio of the two determinants gives us the difference between the two policies.\n\nAccording to the experiment results presented in the main text, it seems that the new algorithm is only useful when the problem being solved has many poor local optima (hence the algorithm needs to inject some uncertainties or conducts more explorations) or when the complete analytical policy gradient cannot be obtained. On problems of differentiable physics simulations, using the analytical policy gradients alone appear to be able to achieve the best results. In view of this, the real practical value of the new algorithm may need to be further investigated on more benchmark problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments, we really appreciate them.\n\n**Q1. What are the possibilities and potential limitations of using the proposed policy gradient combination technique on other algorithms, particularly off-policy algorithms?**\n\nA1. We'd like to emphasize that **on-policy algorithms operate on a very different theoretical foundation from off-policy learning algorithms**. While on-policy algorithms estimate policy gradients that are only valid for the current policy, off-policy algorithms do not use them. Therefore, incorporating analytical gradients into off-policy algorithms requires a shift in theoretical perspective from ours. Moreover, there are already several works to which we can refer when it comes to off-policy algorithms [1]. While the idea of using analytical gradients to enhance off-policy learning algorithms is compelling, it is beyond the scope of this work and deserves a separate investigation.\n\n**Q2. What is the practical impact of the assumption regarding the prior knowledge of the environment dynamics that must be differentiable?**\n\nA2. Please refer to A4 of global rebuttal at the top.\n\n**Q3. What are the theoretical and practical advantages of the new algorithm design, compared to other model-based reinforcement learning methods?**\n\nA3. To the best of our knowledge, model-based RL methods approximate world dynamics from sample trajectories, which require another optimization procedure. Because of this, [2] reported that the SE-MBPO, which is one of the model-based RL methods augmented with analytical gradients, requires a lot more wall-clock time for training (8 hour) than the RP method (15 min) like ours in Ant environment. Therefore, our method is expected to be more efficient than other model-based RL methods.\n\n**Q4. Incompleteness and bias present two separate dimensions regarding the quality of analytical policy gradients. Why should they be strongly correlated?**\n\nA4. About the relationship between the two terms, we believe that \"bias\" is a broader term that includes \"incompleteness\". \"Bias\" in this context means that we cannot estimate the correct gradient even if we have infinite number of samples. In the traffic environments that we discussed, **gradients convey only partial information about world dynamics**. To be specific, they convey information along the lanes, but not across lanes. Please refer to Appendix 7.5.2. This is the reason why we called the gradients \"incomplete\", and under this circumstance we cannot estimate correct gradients even if there are infinite samples, which dictates their biases.\n\n**Q5. Lemma 4.4 mentioned a particular advantage function $\\hat{A}$, but its actual definition is not presented.**\n\nA5. The definition of $\\hat{A}$ is presented in Appendix 7.1.5. Please refer to it for details.\n\n**Q6. It is not clear how alpha is actually adjusted.**\n\nA6. There is a full pseudocode for our algorithm in Appendix 7.3.3, which illustrates how alpha is adjusted during training in detail. Please let us know if it is still unclear.\n\n**Q7. The claim that we can guarantee stable policy updates by constraining the determinant of a matrix to be near 1 is not proved or properly explained.**\n\nA7. Thank you for the comment, but we'd like to say that it is **one of necessary conditions that we have to satisfy to make policy updates more stable**, and we already suggested three reasonings for this claim. To briefly reiterate them, we proved that the determinant should stay near 1 to make $\\alpha$ policy not become an invalid policy. Also, as the determinant is related to variance of the RP gradients, (Figure 2), we have to keep it close to 1 to constrain the variance of analytical gradients used in policy updates. Finally, it helps us keep $\\alpha$ policy near the current policy - please refer to A9.\n\n**Q8. If all eigenvalues are far greater than 1, $\\alpha$ would be set to be very close to 0. In this case, would the analytical gradient become useless?**\n\nA8. Yes, this is our intention. If all eigenvalues are far greater than 1, it means that the variance of the analytical gradients  is very large, and thus it's very likely that the policy updates based on them would be unstable. In such cases, we decrease alpha to nearly 0, and rely mostly on PPO. This is the same spirit as others do when mixing LR and RP gradients for policy updates - if RP gradients have far higher variance than LR, they can be neglected [3].\n\n**Q9. It is not clear why the ratio of the two determinants gives us the difference between the two policies.**\n\nA9. When $\\alpha = 0$, it is trivial that the ratio equals to 1. Therefore, we can say that the deviation of the ratio from 1 is one of (indirect) sufficient conditions for detecting differences between the two policies, rather than necessary conditions.\n\n**Q10. It seems that the new algorithm is only useful when the problem being solved has many poor local optima or when the complete analytical policy gradient cannot be obtained.**\n\nA10.  In fact, our research is motivated by the observation that many environments we counter have these very properties you mentioned, thus this work is widely applicable. For instance, for rigid body physics simulation, there are a lot of discontinuities incurred due to collisions, which require special techniques to make them differentiable. Instead of doing that, we design a policy learning algorithm that can leverage (possibly biased) analytical gradients, because it is quite time-consuming to make every environment fully differentiable. We expect our algorithm could become a good baseline algorithm to test with only partial differentiability, which is much easier to obtain than full differentiability.\n\n[1] Qiao, Yi-Ling, et al. \"Efficient differentiable simulation of articulated bodies.\"\n\n[2] Xu, Jie, et al. \"Accelerated policy learning with parallel differentiable simulation.\"\n\n[3] Parmas, Paavo, et al. \"PIPPS: Flexible model-based policy search robust to the curse of chaos.\""}]

Author rebuttal:
We really appreciate all of the valuable comments from reviewers!

Here we address the major concerns, with a 1-page supplementary document.  Our code will be released publicly, when the paper is published.

**Q1. This paper does not fully utilize the analytical gradients, because it does not consider dynamically changing PPO clip limit.**

A1.  This is a limitation of the current implementation. To dynamically update the PPO's influence, another metric is required to **compare PPO against analytical gradients**.  Even if we could evaluate the quality of analytical gradients in PPO framework, we need an entirely new perspective to do it in the opposite direction.   This new exploration is an exciting future research directions worthy of a full investigation on its own, as many other issues (pros & cons) needs to be considered.

**Q2. The experimental results showed that GI-PPO outperforms PPO, but does not achieve state-of-art performance in some environments, especially in physics environments.**

A2.  Our method clearly outperforms PPO in EVERY environment, but it could not achieve the best performance in some of them, especially in physics simulations (Ant, Hopper).  We'll briefly discuss reasons for this from several aspects.

(1)  We'd like to reiterate that the **RP method is proven to be much more effective at solving those physics problems than other methods, especially PPO**. This is because we have already applied the variance reduction techniques suggested in [1] to these problems. However, because of the reason described in A1, even if it is more advantageous to use analytical gradients over PPO, the performance of our algorithm might be constrained by PPO.

To validate this claim, we conducted additional experiments for the Ant environment, which are displayed in **Figure 1** of the attached document. In the figure, **it's evident that GI-PPO performs slightly better than the RP method when $\delta_{oorr} = 1.0$**. Observing how $\alpha$ and out-of-range-ratio evolve over time in the plots, we see that $\alpha$ is clearly constrained by $\delta_{oorr}$, and this places a detrimental effect on the learning process. If analytical gradients are indeed more beneficial, we can prioritize them over PPO. However, systematizing this approach is currently beyond the scope of our paper.  It is one of the key future research directions currently under consideration.

(2) It requires a more effective strategy to adjust alpha during training. **At present, our approach finds balance between computational cost and performance**.  (Please see A3 for details.)  We discovered that our algorithm's performance in the physics environment, especially in the Hopper environment, is hindered by this issue.  We aim to devise a more systematic strategy and will incorporate it, if feasible, in the paper's revision.

**Q3. There is not so much explanation for the algorithm design to adjust $\alpha$. There could be more theoretical analysis for the algorithm, and even better ways to adjust $\alpha$ during training.**

A3. There are indeed multiple strategies available to adjust the $\alpha$ parameter during the training process.

In the current paper, our chosen strategy involves adjusting $\alpha$ contingent on the optimization results of each iteration. This is achieved by multiplying or dividing it by a predetermined (hyperparameter) constant. The intent behind this is to employ the modified $\alpha$ in the subsequent iteration to better satisfy our proposed conditions. Therefore, there is no guarantee to meet the conditions in every iteration, though works well in practice. As an alternative, we experimented with fine-tuning the $\alpha$ parameter during every iteration to more closely adhere to the constraints.  However, our findings suggest that this approach entails a significantly higher computational expense relative to our primary approach -- thus with a tighter bound but less efficient in practice.

We recognize that superior methods likely exist beyond these techniques.  We continue to investigate possible improved methods grounded in the same principles, as suggested in this paper, with more theoretical analysis *and* better empirical results.  In the meantime, our proposed method here already shows considerably better performance than most of known baselines in diverse environments consistently and we also demonstrated the validity of our approach.

**Q4. There could be limitations in applying this method to practical use cases, because there is no analytical gradients in real world.**

A4. We’d like to introduce two key insights on how our algorithm is beneficial for such cases.  First, we’d like to note that our algorithm can leverage analytical gradients even when they are biased, as shown in SECTION 5.3, **while the original RP method cannot**.  This means that we can **roughly estimate the gradients even if they are not very accurate (e.g. using finite difference method) and leverage them for real world problems**.  Second, nowadays there are many approaches that try to distill policies learned in simulation with access to excess information, including analytical gradients, to student models with access to less information, which are deployed in real world [2]. More generally, policies trained robustly in simulation with analytical gradients can also be used as expert policies for other applications, such as model compression [3] or domain generalization [4].  Likewise, there are many benefits we can take advantages of using our approach, even when it comes to real world applications.

[1] Xu, Jie, et al. "Accelerated policy learning with parallel differentiable simulation."

[2] D. Chen et al., “Learning by Cheating”.

[3] A. Ashok et al., “N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning”.

[4] D. Li et al., “Learning to generalize: Meta-learning for domain generalization”.


Meta-review: Weighting the pros and cons, overall, the reviewers believe this paper should be accepted.
Decision: Accept (poster)