Title: Gradient Informed Proximal Policy Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Analytic Gradient-Based Policy Learning

Proximal Policy Optimization

Differentiable Programming
Abstract: We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm. To incorporate analytical gradients into the PPO framework, we introduce the concept of an α-policy that stands as a locally superior policy. By adaptively modifying the α value, we can effectively manage the influence of analytical policy gradients during learning. To this end, we suggest metrics for assessing the variance and bias of analytical gradients, reducing dependence on these gradients when high variance or bias is detected. Our proposed approach outperforms baseline algorithms in various scenarios, such as function optimization, physics simulations, and traffic control environments. Our code can be found online: https://github.com/SonSang/gippo.

Gradient Informed Proximal Policy Optimization

Abstract

We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm. To incorporate analytical gradients into the PPO framework, we introduce the concept of an α -policy that stands as a locally superior policy. By adaptively modifying the α value, we can effectively manage the influence of analytical policy gradients during learning. To this end, we suggest metrics for assessing the variance and bias of analytical gradients, reducing dependence on these gradients when high variance or bias is detected. Our proposed approach outperforms baseline algorithms in various scenarios, such as function optimization, physics simulations, and traffic control environments. Our code can be found online: https://github.com/SonSang/gippo.

1 Introduction

Reinforcement learning (RL) techniques often take gradient-free experiences or trajectories to collect information and learn policies-suitable for many applications such as video games and recommender systems. In contrast, differentiable programming offers analytical gradients that describe the actionresult relationship (Equation 2). In particular, recent developments in programming tools [Paszke et al., 2017] have enabled direct access to accurate analytical gradients of the environmental dynamics, as exemplified in physics simulations [de Avila Belbute-Peres et al., 2018, Degrave et al., 2019, Geilinger et al., 2020, Freeman et al., 2021, Qiao et al., 2021], instead of using differentiable models for approximation [Deisenroth and Rasmussen, 2011, Deisenroth et al., 2013]. Policy gradients estimated from the integration of these analytical gradients and the reparameterization trick (RP gradient) [Kingma and Welling, 2013] often display less variance than likelihood ratio policy gradients (LR gradient) [Williams and Peng, 1989, Glynn, 1990], derived using the log-derivative trick. Consequently, a subset of research has explored the utilization of these analytical gradients for RL applications, especially in physics domains [Qiao et al., 2021, Mora et al., 2021, Xu et al., 2022].

Nevertheless, in environments with inherent chaotic nature, the RP gradient is found to exhibit high variance [Parmas et al., 2018] and even empirical bias [Suh et al., 2022]. Therefore, a strategic approach is necessary for the adoption of analytical gradients in policy learning [Metz et al., 2021]. One such strategy involves the interpolation of the LR and RP gradients based on their variance and bias [Parmas et al., 2018, Suh et al., 2022], made possible by the fact that both are different estimators of the same policy gradient. However, this approach requires estimating the LR and RP gradient for each sample trajectory to compute sample variance, a process that can be time-consuming (Appendix 7.7). Additionally, for on-policy RL methods that do not initially estimate the LR gradient, such as Trust Region Policy Optimization (TRPO) [Schulman et al., 2015a] or Proximal Policy Optimization (PPO) [Schulman et al., 2017] algorithms, it remains uncertain how to evaluate the variance and bias of analytical gradients. These gradients, however, continue to be desirable when they exhibit low variance and provide insightful information about environmental dynamics.

Given these challenges, our study explores how to incorporate analytical gradients into the PPO framework, taking their variance and bias into account. Our primary contributions are threefold: (1) we propose a novel method to incorporate analytical gradients into the PPO framework without the necessity of estimating LR gradients , (2) we introduce an adaptive α -policy that allows us to dynamically adjust the influence of analytical gradients by evaluating the variance and bias of these gradients, and (3) we show that our proposed method, GI-PPO, strikes a balance between analytical gradient-based policy updates and PPO-based updates, yielding superior results compared to baseline algorithms in various environments.

We validate our approach across diverse environments, including classical numerical optimization problems, differentiable physics simulations, and traffic control environments. Among these, traffic environments typify complex real-world scenarios where analytical gradients are biased. We demonstrate that our method surpasses baseline algorithms, even under these challenging conditions.

2 Related Work

2.1 Policy Learning Without Analytical Gradients

The policy gradient is described as the gradient of the expected cumulative return in relation to policy parameters [Sutton et al., 1999]. For a stochastic policy, as examined in this paper, REINFORCE [Williams, 1992] represents one of the initial methods to employ a statistical estimator for the policy gradient to facilitate policy learning. The policy gradient obtained with this estimator is commonly referred to as a likelihood-ratio (LR) gradient [Glynn, 1990, Fu et al., 2015]. Despite the LR gradient's ability to operate without analytical gradients of dynamics and its unbiased nature, it frequently exhibits high variance [Sutton et al., 1998]. Furthermore, the nonstationarity of the policy renders stable learning difficult to guarantee [Konda and Tsitsiklis, 1999, Schulman et al., 2015b].

Several strategies have been proposed to address the high variance issue inherent to the LR gradient, including the use of additive control variates [Konda and Tsitsiklis, 1999, Greensmith et al., 2004]. Among these, the actor-critic method [Barto et al., 1983] is highly favored. However, the introduction of a value function may induce bias. To counteract this, the Generalized Advantage Estimator (GAE) [Schulman et al., 2015b] was developed to derive a more dependable advantage function. Concerning the issue of unstable learning, methods such as TRPO [Schulman et al., 2015a] and PPO [Schulman et al., 2017] propose optimizing a surrogate loss function that approximates a random policy's expected return. By constraining the policy updates to remain close to the current policy, these methods significantly enhance stability in learning compared to pure policy gradient methods, which makes them more preferred nowadays.

2.2 Policy Learning With Analytical Gradients

In contrast to the LR gradient, the RP gradient is based on the reparameterization trick, which mandates analytical gradients of dynamics yet typically achieves lower variance [Kingma and Welling, 2013, Murphy, 2022]. The RP gradient can be directly estimated over the entire time horizon using Backpropagation Through Time (BPTT) [Mozer, 1989] and subsequently used for policy updates. However, BPTT has been found to encounter multiple optimization challenges such as explosion or vanishing of analytical gradients over extended trajectories [Freeman et al., 2021]. This is because, surprisingly, the RP gradient, contrary to earlier beliefs, may experience exploding variance due to the chaotic nature of environments [Parmas et al., 2018, Metz et al., 2021]. Additionally, Suh et al. [2022] noted that the RP gradient could also be subject to empirical bias.

Several corrective measures exist for the issues concerning the RP gradient [Metz et al., 2021], such as using a truncated time window bootstrapped with a value function [Xu et al., 2022]. This approach has demonstrated its efficacy in addressing physics-related problems. Another class of solutions involves interpolating between LR and RP gradients by evaluating their sample variance and bias [Parmas et al., 2018, Suh et al., 2022]. Likewise, when we carefully address the problems of RP gradient, it can provide valuable information helpful for policy learning. However, when it comes to PPO, which is one of the most widely used RL methods nowadays, it is unclear how we can leverage the analytical gradients that comprise the RP gradient, as it does not estimate LR gradient which can be used for comparison to the RP gradient. In this paper, we address this issue and show that we can use analytical gradients to enhance the performance of PPO.

3 Preliminaries

3.1 Goal

We formulate our problems as a Markov decision process (MDP) defined by a tuple ( S, A, P, r, ρ 0 , γ ) , where S is a set of states, A is a set of actions, P : S × A × S → R is a state transition model, r : S × A → R is a reward model, ρ 0 is the probability distribution of the initial states, and γ is a discount factor.

Under this MDP model, our goal is to train a parameterized stochastic policy π θ : S × A → R + , where θ represents parameters, that maximizes its expected sum of discounted rewards, η ( π θ ) :

In this paper, we denote the current policy that is used to collect experience as π ¯ θ .

3.2 Analytical Gradients

We assume that S and A are continuous, differentiable spaces, and P and r are also differentiable models. Then, our differentiable environment provides us with following analytical gradients of observation or reward at a later timestep with respect to the action at the previous timestep:

where k ≥ 0 is an integer. With these basic analytical gradients, we can compute the gradient of certain advantage function, which we denote as ∂A ∂a . In this paper, we use Generalized Advantage Estimator (GAE) [Schulman et al., 2015b]. See Appendix 7.2.1 for details.

3.3 Policy Update

In this section, we discuss how we update policy in two different settings: the policy gradient method based on RP gradient and PPO.

3.3.1 RP Gradient

To compute the RP gradient, we rely on the reparameterization trick [Kingma and Welling, 2013]. This trick requires us that we can sample an action a from π θ by sampling a random variable ϵ from some other independent distribution q . To that end, we assume that we use a continuous differentiable bijective function g θ ( s, · ) : R n → R n that maps ϵ to an action a , where n is the action dimension. Then the following holds because of injectivity,

and we can define the following relationship.

Definition 3.1 We define π θ ≜ g θ if following holds for an arbitrary open set T ϵ ⊂ R n :

where a = g θ ( s, ϵ ) . The inverse is also true.

where T a = g θ ( s, T ϵ ) .

Lemma 3.2 If π θ ≜ g θ ,

Proof:

See Appendix 7.1.1.

Specifically, we use q = N (0 , I ) in this paper, and define g θ as follows,

which satisfies our assumption.

With g θ ( s, ϵ ) and the analytical gradients from Equation 2, we obtain RP gradient by directly differentiating the objective function in Equation 1, which we use for gradient ascent. See Appendix 7.2.2 for details.

3.3.2 PPO

PPO relies on the observation that we can evaluate a policy π θ with our current policy π ¯ θ , using its advantage function A π ¯ θ as

where ρ π θ ( s ) is the discounted visitation frequencies. See [Kakade and Langford, 2002, Schulman et al., 2015a] for the details.

As far as the difference between π ¯ θ and π θ is sufficiently small, we can approximate ρ π θ ( s ) with ρ π ¯ θ ( s ) [Schulman et al., 2015a], and get following surrogate loss function:

Note that we can estimate this loss function with Monte Carlo sampling as described in Appendix 7.2.3. Since this is a local approximation, the difference between π ¯ θ and π θ must be small enough to get an accurate estimate. TRPO [Schulman et al., 2015a] and PPO [Schulman et al., 2017] constrain the difference to be smaller than a certain threshold, and maximize the right term of (4). In particular, PPO restricts the ratio of probabilities π θ ( s i , a i ) and π ¯ θ ( s i , a i ) for every state-action pair ( s i , a i ) in the buffer as follows to attain the goal, using a constant ϵ clip :

4 Approach

In this section, we discuss how we can use analytical gradients in the PPO framework while considering their variance and biases. We start our discussion with the definition of α -policy.

4.1 α -Policy

With the analytical gradient of advantage with respect to an action ( ∇ a A π ¯ θ ( s, a ) ) from our differentiable environments, we can define a new class of policies π α , parameterized by α , as follows.

Definition 4.1 Given current policy π ¯ θ , we define its α -policy as follows:

Here λ 1 ( s, a ) represents the minimum eigenvalue of ∇ 2 a A π ¯ θ ( s, a ) .

Lemma 4.2 Mapping f is injective, and for an arbitrary open set of action T ⊂ A , π α ( s, · ) selects a set of action ˜ T = f ( A ) with the same probability that π ¯ θ ( s, · ) selects A .

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

x

1.0

0.8

0.6

0.4

0.2

0.0

f

(

x

)

f

(

x

)

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

p

(

x

)

=0

=0.1

=0.2

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

x

4

3

2

1

0

f

(

x

)

f

(

x

)

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

p

(

x

)

=0

=1

e

3

=2

e

3

Figure 1: α -policies for 1-dimensional (a) De Jong's function and (b) Ackley's function. The original policy is rendered in blue, and alpha policies for different α values are rendered in orange and red. Those α -policies are obtained from analytical gradients of the given function f ( x ) rendered in black. Note that Ackley's α -policy becomes invalid when α = 2 · 10 -3 , because of singularities near 0.

Proof: See Appendix 7.1.2.

In Figure 1, we provide renderings of α -policies for different α s when we use 1-dimensional De Jong's function and Ackley's function [Molga and Smutnicki, 2005] as our target functions. The illustration provides us an intuition that when α is sufficiently small, π α can be seen as a policy that selects slightly better action than π ¯ θ with the same probability. In fact, we can prove that π α indeed gives us better estimated expected return in Equation 4 when α > 0 is sufficiently small.

Proposition 4.3 If | α |≪ 1 ,

where L π ¯ θ denotes estimated expected return defined in Equation 4.

Proof: See Appendix 7.1.3.

This proposition tells us that α -policy fits into PPO framework seamlessly, and thus if we update our policy towards α -policy, it does not contradict the PPO's objective. However, since there is a second order derivative ∇ 2 a A π ¯ θ ( s, a ) in the definition, it is unclear how we can update our policy π θ towards π α . In the next section, we show how we can do this and how it is related to RP gradient.

4.2 Relationship between RP gradient and α -policy

To show how we can approximate α -policy, for a deterministic function g ¯ θ such that π ¯ θ ≜ g ¯ θ , we define a new function g α as follows, and provide a Lemma about it.

Lemma 4.4 When a = g ¯ θ ( s, ϵ ) ,

Proof: See Appendix 7.1.4.

Note that we can update g θ to approximate g α by minimizing the following loss.

In fact, it turns out that minimizing Equation 10 leads our policy π θ to approximate π α , because of the following Proposition.

Proposition 4.5 If π ¯ θ ≜ g ¯ θ , for α that satisfies the constraint in Definition 4.1, π α ≜ g α .

Proof: See Appendix 7.1.6.

Note that we can use different advantage function formulations for A in Equation 8, such as GAE [Schulman et al., 2015b] that we use in this work. However, let us we consider the following advantage function ˆ A to see the relationship between RP gradient and α -policy.

Then, the following Lemma holds.

Lemma 4.6 When α = 1 , if we define g α with ˆ A , the RP gradient corresponds to ∂L ∂θ at θ = ¯ θ .

This Lemma tells us that we can understand the RP gradient as the very first gradient we get when we update our policy toward π α by minimizing Equation 10. In other words, α -policy is not only a superior policy than the original one in the PPO's point of view (Proposition 4.3), but also a target policy for RP gradient (Lemma 4.6). Therefore, we conclude that α -policy is a bridge that connects these two different policy update regimes.

4.3 Algorithm

Our algorithm is mainly comprised of 3 parts: (1) Update π θ to π α by minimizing Equation 10, (2) Adjust α for next iteration, and (3) Update π θ by maximizing Equation 4. In this section, we discuss details about Step 2 and 3, and provide the outline of our algorithm.

4.3.1 Step 2: Adjusting α

In Definition 4.1, we can observe that α controls the influence of analytical gradients in policy updates. If α = 0 , it is trivial that π α = π ¯ θ , which means that analytical gradients are ignored. In contrast, as α gets bigger, analytical gradients play bigger roles, and π α becomes increasingly different from π ¯ θ . Then, our mission is to find a well-balanced α value. Rather than setting it as a fixed hyperparameter, we opt to adjust it adaptively. We first present the following variance and bias criteria to control α , as it is commonly used to assess the quality of gradients [Parmas et al., 2018, Suh et al., 2022].

Variance After we approximate π α in Step 1), we can estimate det( I + α ∇ 2 a A π ¯ θ ( s, a )) for every state-action pair in our buffer using Lemma 4.4, and adjust α to bound the estimated values in a certain range, [1 -δ, 1 + δ ] . There are two reasons behind this strategy.

As shown in Definition 4.1, we have to keep α small enough so that π α does not become invalid policy. By keeping det( I + α ∇ 2 a A π ¯ θ ( s, a )) larger than 1 -δ to keep its positiveness, we guarantee that our policy is not updated towards an invalid policy.

Note that det( I + α ∇ 2 a A π ¯ θ ( s, a )) = Π n i =1 (1 + αλ i ) , where λ i is the i -th eigenvalue of ∇ 2 a A π ¯ θ ( s, a ) . By constraining this value near 1 by adjusting α , we can guarantee stable policy updates even when some eigenvalues of ∇ 2 a A π ¯ θ ( s, a ) have huge magnitudes, which is related to a high variance RP gradient. In Appendix 7.3.1, we provide empirical evidence that this estimation is related to the sample variance of analytical gradients.

Bias By Proposition 4.3, we already know that when we estimate the right term of Equation 4 after Step 1), it should be a positive value. However, if the analytical gradients were biased, this condition could not be met. In runtime, if we detect such cases, we decrease α .

In addition, we use one more criterion to adjust α , which we call the out-of-range ratio. This criterion was designed to promise the minimum amount of policy update by PPO.

Out-of-range-ratio We define out-of-range-ratio as follows,

where N is the size of the buffer, I is the indicator function, and ϵ clip is the constant from Equation 5. We assess this ratio after the policy update in Step 1), allowing its value to be non-zero when α > 0 . A non-negligible value of this ratio indicates a potential violation of the PPO's restriction outlined in Equation 5, thereby compromising the effectiveness of PPO. Further elaboration can be found in Appendix 7.3.2. Therefore, when this ratio exceeds a predetermined threshold, we subsequently reduce the value of α.

To sum it up, our strategy to control α aims at decreasing it when analytical gradients exhibit high variance or bias, or when there is little room for PPO updates. This results in greater dependence on PPO, which can be thought of as a safeguard in our approach. Otherwise, we increase α , since we can expect higher expected return with a greater α as shown in Proposition 4.3.

4.3.2 Step 3: PPO Update

Since we do policy updates based on PPO after the update based on π α , there is a possibility that the PPO update can 'revert' the update from Step 1). To avoid such situations, we propose to use the following π h in the place of π ¯ θ in Equation 5:

By using this virtual policy π h , we can restrict the PPO update to be done near both π ¯ θ and π α . See Appendix 7.3.2 for more details.

4.3.3 Pseudocode

In Algorithm 1, we present pseudocode that illustrates the outline of our algorithm, GI-PPO. There are five hyperparameters in the algorithm, which are α 0 , β, δ det , δ oorr , and max( α ) . We present specific values for these hyperparameters for our experiments in Appendix 7.4.2.

0

20000

40000

60000

80000

100000 120000

Step

0.200

0.175

0.150

0.125

0.100

0.075

0.050

0.025

0.000

0.025

Reward

LR

RP

PPO

LR+RP

GI-PPO

0

20000

40000

60000

80000

100000

120000

Step

10

8

6

4

2

0

Reward

LR

RP

PPO

LR+RP

GI-PPO

0

20000

40000

60000

80000

100000

120000

Step

20.0

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

Reward

LR

RP

PPO

LR+RP

GI-PPO

0

20000

40000

60000

80000

100000

120000

Step

20

15

10

5

0

Reward

LR

RP

PPO

LR+RP

GI-PPO

5 Experimental Results

In this section, we present experimental results that show our method's efficacy for various optimization and complex control problems. To validate our approach, we tested various baseline methods on the environments that we use. The methods are as follows:

LR: Policy gradient method based on LR gradient.

RP: Policy gradient method based on RP gradient. For physics and traffic problems, we adopted a truncated time window of [Xu et al., 2022] to reduce variance.

PPO: Proximal Policy Optimization [Schulman et al., 2017].

LR+RP: Policy gradient method based on interpolation between LR and RP gradient using sample variance [Parmas et al., 2018].

PE: Policy enhancement scheme of [Qiao et al., 2021], for physics environments only.

GI-PPO: Our method based on Section 4.3.

Please refer Appendix 7.4 to learn about implementation details of these baseline methods, network architectures, and hyperparameters that we used for all of the experiments.

We have implemented our learning method using PyTorch 1.9 Paszke et al. [2019]. As for hardware, all experiments were run with an Intel® Xeon® W-2255 CPU @ 3.70GHz, one NVIDIA RTX A5000 graphics card, and 16 GB of memory. Experiment results are averaged across 5 random seeds.

5.1 Function Optimization Problems

We start with how different learning methods find the maximum point for a given analytical function. We ran these experiments because RL algorithms can be thought of as function optimizers through function sampling [Williams and Peng, 1989]. These problems are 1 -step optimization problems, where agents guess an optimal point as an action and get the negative function value as the reward.

As we used for rendering α -policies in Figure 1, we use De Jong's function and Ackley's function for comparison, as they are popular functions for testing numerical optimization algorithms [Molga and Smutnicki, 2005]. Please see Appendix 7.5.1 for the details about the function definitions. In this setting, the optimum occurs at the point x = 0 for both of the problems. However, their loss landscapes are quite different from one another-De Jong's function has a very smooth landscape that only has one local (global) maximum, while Ackley's function has a very rugged landscape with multiple local maxima. Therefore, De Jong's function represents environments where RP gradients have lower variance, while Ackley's function represents the opposite.

Table 1 shows that GI-PPO finds far better optimums than the other methods for most of the functions , except 64-dimensional Dejong's function, where the RP method found slightly better

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

5000

4000

3000

2000

1000

Reward

LR

RP

PPO

PE

GI-PPO

(a) Cartpole

0

1

2

3

4

Step

1e6

0

1000

2000

3000

4000

5000

6000

7000

Reward

LR

RP

PPO

PE

GI-PPO

(b) Ant

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Step

1e7

0

1000

2000

3000

4000

5000

Reward

LR

RP

PPO

PE

GI-PPO

(c) Hopper

optimum than GI-PPO. Optimization curves in Figure 2 also prove that our method converges to the optimum faster than the other methods. Interestingly, the RP method achieved better results than the LR and LR+RP methods for all the functions, which contradicts our assumption that it would not work well for Ackley's function. However, we found out that even though the RP method takes some time to find the optimum than the other methods as shown in Figure 2, when it approaches near-optimum, it converges fast to the optimum based on analytic gradients. Since GI-PPO adaptively changes α to favor RP gradient near optimum, it could achieve better results than the other methods. In Appendix 7.3.3, we provide visual renderings that trace the change of α value during optimization to help understanding, which also shows that higher α is maintained for Dejong's function than Ackley's function, which aligns with our initial assumption. Also note that even though LR+RP also takes advantage of the RP method, and thus achieved a better convergence rate than the two methods when it is far from optimum, it did not work well when it comes to near-optimum.

5.2 Differentiable Physics Simulation

Next, we conducted experiments in differentiable physics simulations. We used Cartpole, Ant, and Hopper environments implemented by [Xu et al., 2022] for comparisons. However, since the current implementation does not support multiple backpropagations through the same computation graph for these environments, we could not test the LP+RP method for them. Instead, we tried to faithfully implement another policy enhancement (PE) strategy of [Qiao et al., 2021] as a comparison, as it showed its efficacy in differentiable physics simulations using analytical gradients.

In Figure 3, we can see that GI-PPO converged to a better policy much faster than the baseline PPO for every environment, which validates our approach. However, the RP method achieved the best results for Ant and Hopper, while GI-PPO converged to the optimal policy slightly faster for Cartpole. To explain this result, we'd like to point out that α is upper bounded by out-of-range-ratio in our approach (Section 4.3.1). Therefore, even if the RP gradient is very effective, we cannot fully utilize it because we have to use PPO to update the policy up to some extent as a safeguard. In the Ant environment, we could observe that it was the major bottleneck in training-in fact, GI-PPO could achieve better results than RP when we increased the ratio from 0.5 to 1.0 (Appendix 7.3.4). However, when the out-of-range ratio equals 1.0, it becomes more likely that we cannot use PPO as a safeguard, which contradicts our design intention. When it comes to the Hopper environment, we found that the variance of proper α values over time was large, so our strategy could not conform to it properly. Overcoming these limitations would be an interesting future work to extend our work.

5.3 Traffic Problems

Here we demonstrate the effectiveness of GI-PPO in an environment where continuous and discrete operations coexist, which leads to highly biased analytical gradients. We suggest a mixed-autonomy traffic environment as a suitable benchmark, which has previously been explored as a benchmark for gradient-free RL algorithms [Wu et al., 2017, Vinitsky et al., 2018]. In this paper, we use the pace car problem, where a single autonomous pace car has to control the speed of the other vehicles via interference. The number of lanes, which represent the discontinuities in gradients, and the number of following human vehicles are different for each problem. Please see Appendix 7.5.2 for the details of this environment, and why the analytical gradients are biased.

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

0

200

400

600

800

Reward

LR

RP

PPO

LR+RP

GI-PPO

(a) Single Lane

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

0

200

400

600

800

Reward

LR

RP

PPO

LR+RP

GI-PPO

(b) 2 Lanes

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

40

60

80

100

120

140

Reward

LR

RP

PPO

LR+RP

GI-PPO

(c) 4 Lanes

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

15

20

25

30

35

40

45

50

Reward

LR

RP

PPO

LR+RP

GI-PPO

(d) 10 Lanes

In Figure 4, we can observe that GI-PPO exhibits a faster convergence rate and converges to better policy overall compared to all the other methods in 2, 4, and 10-lane environments. In a single-lane environment, however, LR+RP showed a better convergence rate than GI-PPO. Note that the RP method still performs well for 2 and 4-lane environments just as other baseline methods, even if there were multiple lanes. We assume this is because the RP method also leverages a state value function, which can provide unbiased analytical gradients in the presence of discontinuities. However, it does not perform well in the 10-lane environment, where discontinuities are excessive compared to the other environments.

This result shows that GI-PPO can still leverage analytical gradients, even if they were highly biased for policy learning . Even though LP+RP, which is based on a similar spirit as ours, also exhibited some improvements over LP and RP, our method showed better results for more difficult problems because baseline PPO offers more stable learning than baseline LR . Also, the computational cost of GI-PPO was a little more than RP, while that of LR+RP was much more expensive (Appendix 7.7).

6 Conclusions

We introduced a novel policy learning method that adopts analytical gradients into the PPO framework. Based on the observation that the RP gradient leads our current policy to its α -policy, we suggested approximating it explicitly, which allows us to indirectly estimate variance and bias of analytical gradients. We suggested several criteria to detect such variance and bias and introduced an algorithm that manipulates α adaptively, which stands for the strength of analytical gradients in policy updates. In our experiments of diverse RL environments, we successfully showed that our method achieves much better results than the baseline PPO by adopting analytical gradients. Even though the interpolated gradient of LR and RP gradient is based on similar motivations, our method performed much better, thanks to the baseline PPO's stable learning.

Limitations Despite GI-PPO's promising results, there are some remaining issues. First, even if we use a large α , we may not fully approximate the corresponding α -policy in a limited number of optimization steps, which may give a false signal for our strategy. We can upper bound the maximum α as we did here, but a more rigorous optimization process may be helpful. This is also related to improving the strategy to control α adaptively, which would be a promising future work.

Also, as discussed in Section 5.2, our method is tightly bound to PPO-that is, even when analytical gradients are much more useful as illustrated in differentiable physics problems, we cannot fully utilize them if they exit the PPO's bound. Even though it is more suitable for stable updates, it could result in a worse convergence rate. If we could adjust PPO's clipping range dynamically, or detect environments where RP gradients are much more reliable, we would be able to overcome this issue.

Finally, computational costs can be further reduced. To utilize analytical gradients, we need backpropagation, which usually requires more time than forward steps. This is the reason why the learning methods based on analytical gradients require longer training time than the others (Appendix 7.7). When the time window gets longer, the cost grows accordingly. However, as we have shown in our traffic experiments, our method works even when gradients are biased. Therefore, it would be an interesting research direction to see if using a very short time window for backpropagation, which produces more biased gradients, would be worthy of exploring for improved computational efficiency.

Acknowledgements. This research is supported in part by the ARO DURIP and IARPA HAYSTAC Grants, ARL Cooperative Agreement W911NF2120076, Dr. Barry Mersky and Capital One ENnovate Endowed Professorships. Yi-Ling Qiao would like to thank the support of Meta Fellowship.

References

Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics , (5): 834-846, 1983.

Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. Endto-end differentiable physics for learning and control. Advances in neural information processing systems , 31, 2018.

Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics , page 6, 2019.

Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11) , pages 465-472, 2011.

Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends® in Robotics , 2(1-2):1-142, 2013.

C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281 , 2021.

C Fu, C Fu, and Michael Michael. Handbook of simulation optimization . Springer, 2015.

Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG) , 39(6):1-15, 2020.

Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM , 33(10):75-84, 1990.

Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research , 5(9), 2004.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning . Citeseer, 2002.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for reinforcement learning. https://github.com/Denys88/rl_games , May 2022.

Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients are not all you need. arXiv preprint arXiv:2111.05803 , 2021.

Marcin Molga and Czesław Smutnicki. Test functions for optimization needs. Test functions for optimization needs , 101:48, 2005.

Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning , pages 7805-7817. PMLR, 2021.

MMozer. A focused backpropagation algorithm. Complex systems , 3:349-381, 1989.

Kevin P Murphy. Probabilistic machine learning: an introduction . MIT press, 2022.

Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning , pages 4065-4074. PMLR, 2018.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Efficient differentiable simulation of articulated bodies. In International Conference on Machine Learning , pages 8661-8671. PMLR, 2021.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021. URL http://jmlr.org/papers/v22/20-1364.html .

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015a.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015b.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning , pages 20668-20696. PMLR, 2022.

Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning. 1998.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical Review E , 62(2):1805-1824, Aug 2000.

Eugene Vinitsky, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen. Benchmarks for reinforcement learning in mixed-autonomy traffic. In Conference on robot learning , pages 399-409. PMLR, 2018.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

Ronald J Williams and Jing Peng. Reinforcement learning algorithms as function optimizers. In Proceedings of the International Joint Conference on Neural Networks, Washington DC , volume 2, pages 89-95, 1989.

Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv:1710.05465 , 10, 2017.

Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint arXiv:2204.07137 , 2022.

7 APPENDIX

7.1 Proofs

7.1.1 Proof of Lemma 3.2

By Definition 3.1, π θ ( s, a ) should satisfy the following condition:

We denote the set of continuous distributions that satisfy this condition as P . That is,

Clearly, ¯ P θ ( s, ϵ ) = q ( ϵ ) · ∣ ∣ ∣ det( ∂g θ ( s,ϵ ) ∂ϵ ) ∣ ∣ ∣ -1 is an element of P , and is well defined, because of Equation 3. In fact, ¯ P θ ( s, ϵ ) is the only element of P , as the above condition should be met for any T ϵ ⊂ R n . That is, if there was another distribution ˜ P θ ( s, ϵ ) ∈ P ,

Since ˜ P θ is different from ¯ P θ , there exists an open set ˆ T ϵ where ˜ P θ < ¯ P θ . Then

which is a contradiction. Therefore, π θ ( s, a ) = π θ ( s, g θ ( s, ϵ )) = ¯ P θ ( s, ϵ ) . The inverse holds trivially.

7.1.2 Proof of Lemma 4.2

First, we show that f is an injective function by showing that the determinant of Jacobian ∂f ∂a is always positive:

where λ i ( s, a ) are the eigenvalues of ∇ 2 a A π ¯ θ ( s, a ) sorted in ascending order.

Then, for an arbitrary open set of action T ⊂ A , π α ( s, · ) selects ˜ T = f ( T ) ⊂ A with the same probability that the original policy π ¯ θ ( s, · ) selects T .

Therefore, π α ( s, · ) is a valid probability distribution as π ¯ θ ( s, · ) .

7.1.3 Proof of Proposition 4.3

For a given state s , we can estimate the (approximate) expected value of the state under π α as follows.

Therefore, we can see the following holds:

Since π ¯ θ ( s, a ) and ||∇ a A π ¯ θ ( s, a ) || 2 are positive, L π ¯ θ ( π α ) is greater or equal to η ( π ¯ θ ) when α > 0 . On the contrary, when α < 0 , L π ¯ θ ( π α ) is smaller or equal to η ( π ¯ θ ) . Since this argument builds upon local approximation, it holds only when | α |≪ 1 .

7.1.4 Proof of Lemma 4.4

When a = g ¯ θ ( s, ϵ ) ,

Since ∣ ∣ ∣ det ( ∂g ¯ θ ( s,ϵ ) ∂ϵ )∣ ∣ ∣ > 0 (Equation 3), we can divide both sides of the above equation with det ( ∂g ¯ θ ( s,ϵ ) ∂ϵ ) and prove Lemma 4.4.

7.1.5 Proof of Lemma 4.6

From Equation 11,

which can be rewritten using g θ as follows.

Then, by differentiating Equation 10 with respect to θ after plugging in ˆ A , we can get following relationship:

which equals to RP gradient in Appendix 7.2.2, but with different sign. However, they turn out to be the same because we minimize L ( θ ) , but maximize η ( π θ ) . Therefore, we can say that we gain RP gradient as the first gradient when we minimize Equation 10 for particular advantage function ˆ A and α = 1 .

7.1.6 Proof of Proposition 4.5

By Definition 4.1 and Lemma 4.4,

where a = g θ ( s, ϵ ) , and thus ˜ a = g α ( s, ϵ ) . Since π ¯ θ ≜ g ¯ θ , following holds:

which implies π α ≜ g α .

7.2 Formulations

7.2.1 Analytical Gradient of Generalized Advantage Estimator (GAE)

GAE [Schulman et al., 2015b] has been widely used in many RL implementations [Schulman et al., 2017, Raffin et al., 2021, Makoviichuk and Makoviychuk, 2022] to estimate advantages. GAE finds a balance between variance and bias of the advantage estimation by computing the exponentiallyweighted average of the TD residual terms ( δ V t ) Sutton et al. [1998], which are defined as follows:

Then, GAE can be formulated as follows:

We can compute the gradients for these terms as:

Based on this relationship, we can compute every ∂A GAE t ∂a t with only one backpropagation of A GAE 0 , rather than backpropagating for every A GAE t .

7.2.2 RP Gradient Formulation

To differentiate Equation 1 with respect to θ , we first rewrite Equation 1 as follows.

Then, we can compute RP gradient as follows.

̸

We can estimate this RP gradient using Monte Carlo sampling and terms in Equation 2, and use it for gradient ascent to maximize Equation 1.

0

25

50

75

100

125

150

175

200

0

5

10

15

20

25

Sample Variance

0

25

50

75

100

125

150

175

200

Epoch

0.96

0.98

1.00

1.02

Estimate Determinant

Mean

Min

(a) Cartpole

0

25

50

75

100

125

150

175

200

0

50

100

150

Sample Variance

0

25

50

75

100

125

150

175

200

Epoch

0.95

1.00

1.05

Estimate Determinant

Mean

Min

(b) Ant

0

25

50

75

100

125

150

175

200

0

100

200

300

400

500

Sample Variance

0

25

50

75

100

125

150

175

200

Epoch

0.95

1.00

1.05

1.10

Estimate Determinant

Mean

Min

(c) Hopper

7.2.3 Estimator for Equation 4

Since η ( π ¯ θ ) does not depend on θ , we can ignore the term and rewrite our loss function in Equation 4 using expectation as follows:

However, note that we did not collect trajectories, or experience buffer, using π θ . Therefore, we use another importance sampling function q ( s, a ) [Schulman et al., 2015a].

Then, we can estimate this value using Monte Carlo estimation as follows, where B is the experience buffer of size N , which stores state ( s i ), action ( a i ), and their corresponding advantage value ( A π ¯ θ ( s i , a i ) ) obtained by following policy π ¯ θ :

Note that we used π ¯ θ in the place of q , as it is the most natural importance sampling function we can use [Schulman et al., 2015a, 2017].

7.3 Algorithm

Here we present the details of our algorithm in Section 4.3. To be specific, we present 1) an empirical evidence to validate our strategy to control α , 2) the exact loss function that we use in PPO update, and 3) provide the change of α during real training to help understanding.

7.3.1 Empirical evidence

In Section 4.3, we argued that the estimate of det( I + α ∇ 2 a A π ¯ θ ( s, a )) is related to the sample variance of analytical gradients, and we can get the estimate using Lemma 4.4. In Figure 5, we empirically prove the validity of this argument in the differentiable physics environments used in Section 5.2. In the illustration, we can observe a clear relationship between the sample variance of analytical gradients and the statistics of our estimates. Therefore, we can use these estimates as a cue to control α based on the variance of analytical gradients.

0

500

1000

1500

2000

10

5

10

4

10

3

Alpha

0

500

1000

1500

2000

50

0

50

Estimated current performance

0

500

1000

1500

2000

0.6

0.7

0.8

0.9

1.0

Estimate Determinant

Mean

Min

0

500

1000

1500

2000

0.0

0.2

0.4

0.6

Out of Range Ratio

0

500

1000

1500

2000

10

6

10

5

10

4

Alpha

0

500

1000

1500

2000

2

0

2

4

6

Estimated current performance

0

500

1000

1500

2000

0.6

0.7

0.8

0.9

1.0

Estimate Determinant

Mean

Min

0

500

1000

1500

2000

0.0

0.2

0.4

0.6

Out of Range Ratio

(b) Ackley's Function (64)

7.3.2 Loss function for PPO Update

As discussed in Section 3.3.2, PPO finds a better policy in the proximity of the original policy by restricting the ratio of probabilities π θ ( s i , a i ) and π ¯ θ ( s i , a i ) , as shown in Equation 5. In fact, PPO uses following surrogate loss function, which is a modified version of Equation 16, to mandate this:

Note that if out-of-range-ratio defined in Equation 12 is already 1.0, which corresponds to an extreme case, the gradient of L PPO ( θ ) with respect to θ is 0. Therefore, PPO does not contribute to the policy update at all in this case. It justifies our decision to upper bound out-of-range-ratio by restricting α .

Also note that we proposed to use a virtual policy π h instead of π ¯ θ in Section 4.3.2 to restrict PPO update to be done around π α . This changes the importance sampling function in Equation 17 as follows, which corresponds to our final surrogate loss function to use in PPO update:

7.3.3 Example: Function Optimization Problems (Section 5.1)

Here we present how α changes as we solve function optimization problems in Section 5.1 with Figure 6. There are 4 plots for each of the problems.

Alpha: Shows change of α over training epoch.

Estimated current performance: Shows expected additional return, which corresponds to Equation 16 ( R α in Algorithm 1).

Estimate Determinant: Shows statistics of estimated det( I + α · ∇ 2 a A π ¯ θ ( s, a )) .

Out of Range Ratio: Shows out-of-range-ratio in Equation 12.

For these problems, we have set α 0 = 10 -5 , δ det = 0 . 4 , and δ oorr = 0 . 5 .

For De Jong's function (Figure 6a), we can observe that α shows steady increase over time, but it is mainly upper bounded by variance criterion and out-of-range-ratio. In the end, α stabilizes around 10 -3 . In contrast, for Ackley's function (Figure 6b) we can see that α increases rapidly at first, but it

0

1

2

3

4

Step

1e6

0

1000

2000

3000

4000

5000

6000

7000

Reward

LR

RP

PPO

PE

GI-PPO

0

250

500

750

1000

1250

1500

1750

2000

Epoch

10

2

10

1

10 0

Alpha

0

250

500

750

1000

1250

1500

1750

2000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Out of Range Ratio

0

1

2

3

4

Step

1e6

0

1000

2000

3000

4000

5000

6000

7000

Reward

LR

RP

PPO

PE

GI-PPO

0

250

500

750

1000

1250

1500

1750

2000

Epoch

10

2

10

1

10 0

Alpha

0

250

500

750

1000

1250

1500

1750

2000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Out of Range Ratio

(b) Statistics of training in

Ant environment when

δ

= 1

.

0

.

soon decreases due to the variance and bias criteria. Compare these graphs with optimization curve in Figure 2d. Then we would be able to observe that this large α contributes to the faster convergence of GI-PPO compared to PPO at the early stage. However, after that, α decreases slowly due to the out-of-range-ratio. In the end, α reaches approximately 10 -6 , which is much lower than 10 -3 of De Jong's function. These observations align well with our intuition that RP gradient would play bigger role for De Jong's function than Ackley's function, because De Jong's function exhibits RP gradients with much lower variance than those of Ackley's.

7.3.4 Example: Physics Simulation Problems (Section 5.2)

In Section 5.2, GI-PPO could not achieve better results than RP in Ant and Hopper environments. For Ant environment, we found out that the hyperparameter δ oorr was a bottleneck that bounds GI-PPO's performance. In Figure 7, we illustrated the change of α during training along the change of out-of-range-ratio. When we set δ oorr = 0 . 5 , α keeps decreasing over time to keep the ratio around 0 . 5 , which means the diminishing role of RP gradients in training. In contrast, when we set δ oorr = 1 . 0 , after 750 epochs, α does not decrease anymore, and out-of-range ratio stays near 1. This means that PPO rarely affects the training - policy update is almost driven by RP gradients.

Likewise, there are environments where RP gradients are much more effective than the other policy gradients. In those situations, it would be more desirable to solely depend on RP gradients without considering PPO. Since our method considers RP gradients in the PPO viewpoint, it is hard to detect such cases yet.

7.4 Experimental Details

7.4.1 Baseline Methods

Here we explain implementation details of the baseline methods that were used for comparisons in Section 5. First, we'd like to point out that we used relatively a short time horizon to collect experience, and used the critic network for bootstrapping, instead of collecting experience for the whole trajectory for all the methods. Therefore, the collected experience could be biased. However, we chose to implement in this way, because it allows faster learning than collecting the whole trajectory in terms of number of simulation steps.

LR We can estimate original LR gradient as follows, using log-derivative trick [Williams and Peng, 1989, Glynn, 1990]

where L ( s 0 , a 0 , ... ) = ∑ ∞ t =0 γ t r ( s t , a t ) is the discounted accumulated return. However, this LR gradient often suffers from high variance, because L varies a lot from trajectory to trajectory. To reduce such variance and faithfully compare our method to LR gradient-based method, we decided to use advantage function in the place of L , and particularly used GAE [Schulman et al., 2015b] as the advantage function as we used for PPO. After calculating this LR gradient, we take a gradient ascent step with pre-defined learning rate.

RP By using RP gradient in Appendix 7.2.2, we can perform gradient ascent as we did in LR. Also, because we use short time horizon and critic network, this method is very similar to SHAC [Xu et al., 2022]. Please refer to the paper for more details.

PPO Our PPO implementation is based on that of RL Games [Makoviichuk and Makoviychuk, 2022]. However, to be as fair as possible, we used the same critic network as the other methods, instead of using that implemented in the RL library. Also, we omitted several other additional losses that are not vital to PPO's formulation, such as entropy loss.

LR+RP After we compute LR ( ∇ LR ) and RP ( ∇ RP ) gradient as shown above, we can interpolate them using their sample variance as follows [Parmas et al., 2018].

where σ RP and σ LR are sample standard deviation of RP and LR gradients. We gain these terms by computing trace of covariance matrix of the sample gradients from different trajectories.

Since we have to compute this sample statistics, we have to do multiple different backpropagations for different trajectories, which incur a lot of computation time. Also, we found out that computing covariance matrix is also time consuming when controller has a large number of parameters. Therefore, we decided to use only limited number of sample gradients (16) to compute sample variance, and also truncate the gradient to smaller length (512) to facilitate computation.

PE We tried to faithfully re-implement policy enhancement scheme of [Qiao et al., 2021].

7.4.2 Network architecture and Hyperparameters

In this section, we provide network architectures and hyperparameters that we used for experiments in Section 5. For each of the experiments, we used the same network architectures, the same length of time horizons before policy update, and the same optimization procedure for critic updates, etc. We present these common settings first for each of the problems.

For GI-PPO, there are hyperparameters for update towards α -policy, and those for PPO update. We denote the hyperparameters for α -policy update by appending ( α ) , and those for PPO update by appending (PPO). For the definition of hyperparameters for α -policy update, please see Algorithm 1 for details.

Function Optimization Problems (Section 5.1) For these problems, common settings are as follows.

Actor Network: MLP with [32 , 32] layers and ELU activation function

Critic Network: MLP with [32 , 32] layers and ELU activation function · Critic Hyperparameters: Learning rate = 10 -3 , Iterations = 16, Batch Size = 4 · Number of parallel environments: 64 · Horizon Length: 1 · γ (Discount factor): 0.99 · τ (GAE): 0.95 Hyperparameters for LR are as follows. · Learning Rate: Dejong(1), Dejong(64) = 10 -3 , Ackley(1) = 10 -4 , Ackley(64) = 3 · 10 -4 · Learning Rate Scheduler: Linear 1 Hyperparameters for RP are as follows. · Learning Rate: Dejong(1), Dejong(64) = 10 -2 , Ackley(1), Ackley(64) = 10 -3 · Learning Rate Scheduler: Linear Hyperparameters for LR+RP are as follows. · Learning Rate: Dejong(1), Dejong(64) = 10 -3 , Ackley(1) = 10 -4 , Ackley(64) = 3 · 10 -4 · Learning Rate Scheduler: Linear Hyperparameters for PPO are as follows. · Learning Rate: Dejong(1), Ackley(1) = 10 -4 , Dejong(64), Ackley(64) = 10 -2 · Learning Rate Scheduler: Constant · Batch Size for Actor Update: 64 · Number of Epochs for Actor Update: 5 · ϵ clip : 0 . 2 Hyperparameters for GI-PPO are as follows. -3

(PPO) Learning Rate: Dejong(1), Ackley(1) = 10 -4 , Dejong(64), Ackley(64) = 10 -2

( α ) Learning Rate: 10 · ( α ) Batch Size for Actor Update: 64 · ( α ) Number of Epochs for Actor Update: 16 · ( α ) α 0 : 10 -5 · ( α ) max( α ) : 1 . 0 · ( α ) β : 1.1 · ( α ) δ det : 0 . 4 · ( α ) δ oorr : 0 . 5 · (PPO) Learning Rate Scheduler: Constant · (PPO) Batch Size for Actor Update: 64 · (PPO) Number of Epochs for Actor Update: 5 · (PPO) ϵ clip : 0 . 2

Differentiable Physics Problems (Section 5.2) For these problems, common settings are as follows.

Actor Network: MLP with [64 , 64] (Cartpole), and [128 , 64 , 32] (Ant, Hopper) layers and ELU activation function

Critic Network: MLP with [64 , 64] layers and ELU activation function

Critic Hyperparameters: Learning rate = 10 -3 (Cartpole), 2 · 10 -3 (Ant), and 2 · 10 -4 (Hopper), Iterations = 16, Batch Size = 4

Number of parallel environments: 64(Cartpole, Ant), 256(Hopper)

Horizon Length: 32

γ (Discount factor): 0.99

τ (GAE): 0.95

Hyperparameters for LR are as follows.

Learning Rate: 10 -4

Learning Rate Scheduler: Linear

Hyperparameters for RP are as follows.

Learning Rate: Cartpole = 10 -2 , Ant, Hopper = 2 · 10 -3

Learning Rate Scheduler: Linear

Hyperparameters for PPO are as follows.

Learning Rate: Cartpole = 3 · 10 -4 , Ant, Hopper = 10 -4

Learning Rate Scheduler: Cartpole = Adaptive 2 , Ant, Hopper = Constant

Batch Size for Actor Update: 2048

Number of Epochs for Actor Update: 5

ϵ clip : 0 . 2

Hyperparameters for GI-PPO are as follows.

( α ) Learning Rate: Cartpole = 10 -2 , Ant = 5 · 10 -4 , Hopper = 5 · 10 -3

( α ) Batch Size for Actor Update: Cartpole, Ant = 2048, Hopper = 8192 · ( α ) Number of Epochs for Actor Update: 16 · ( α ) α 0 : Cartpole, Ant = 5 · 10 -1 , Hopper = 5 · 10 -3 · ( α ) max( α ) : 1 . 0 · ( α ) β : 1.02 · ( α ) δ det : 0 . 4 · ( α ) δ oorr : Cartpole, Hopper = 0 . 75 , Ant = 0 . 5 · (PPO) Learning Rate: Cartpole = 3 · 10 -4 , Ant, Hopper = 10 -4 · (PPO) Learning Rate Scheduler: Cartpole = Adaptive, Ant, Hopper = Constant · (PPO) Batch Size for Actor Update: 2048 · (PPO) Number of Epochs for Actor Update: 5

(PPO) ϵ clip : 0 . 2

Traffic Problems (Section 5.3) For these problems, common settings are as follows.

Actor Network: MLP with [512 , 64 , 64] layers and ELU activation function

Critic Network: MLP with [64 , 64] layers and ELU activation function

Critic Hyperparameters: Learning rate = 10 -3 , Iterations = 16, Batch Size = 4

Number of parallel environments: 64

Horizon Length: 32

γ (Discount factor): 0.99

τ (GAE): 0.95

Hyperparameters for LR are as follows.

Learning Rate: 3 · 10 -4

Learning Rate Scheduler: Linear

Hyperparameters for RP are as follows.

Learning Rate: 10 -3

Learning Rate Scheduler: Linear

Hyperparameters for LR+RP are as follows.

Learning Rate: 3 · 10 -4

Learning Rate Scheduler: Linear

Hyperparameters for PPO are as follows.

Learning Rate: 3 · 10 -4

Learning Rate Scheduler: Constant

Batch Size for Actor Update: 2048

Number of Epochs for Actor Update: 5

ϵ clip : 0 . 2

Hyperparameters for GI-PPO are as follows.

( α ) Learning Rate: 10 -5

( α ) Batch Size for Actor Update: 2048

( α ) Number of Epochs for Actor Update: 16

( α ) α 0 : 10 -1

( α ) max( α ) : 1 . 0

( α ) β : 1.1

( α ) δ det : 0 . 4

( α ) δ oorr : 0 . 5

(PPO) Learning Rate: 3 · 10 -4

(PPO) Learning Rate Scheduler: Constant

(PPO) Batch Size for Actor Update: 2048

(PPO) Number of Epochs for Actor Update: 5

(PPO) ϵ clip : 0 . 2

7.5 Problem Definitions

Here we present details about the problems we suggested in Section 5.

(b) Ackley's Function

2.5

-10

5.0

7.5

20

-10.0

12.5

OE-

15.0

-40

17.5

-20.0

50

30

20

10

-30

0

-4

20

10

-10

-2

0

10

-20

20

OE-

30

7.5.1 Function Optimization Problems (Section 5.1)

(N-dimensional) De Jong's function ( F D ) and Ackley's function ( F A ) are defined for n -dimensional vector x and are formulated as follows [Molga and Smutnicki, 2005]:

As we mentioned in Section 5.1, we multiply -1 to these functions to make these problems maximization problems. Also, even though these two functions have their optimum at x = 0 , they exhibit very different landscape as shown in Figure 8.

When it comes to the formal definition as RL environments, we can define these problems as follows.

Episode Length: 1

Observation: [0]

Action: n -dimensional vector x , all of which element is in [ - 1 , 1] .

Reward: De Jong's Function = F D (5 . 12 x ) , Ackley's Function = F A (32 . 768 x )

7.5.2 Traffic Problems (Section 5.3)

Before introducing pace car problem, which we specifically discussed in this paper, we'd like to briefly point out the traffic model that we used to simulate motions of individual vehicles in the traffic environment.

Traffic Model In our traffic simulation, the state s ∈ R 2 N is defined as a concatenation of all vehicle states, q n ∈ R 2 , where 1 ≤ n ≤ N stands for vehicle ID. q n can be represented simply with the vehicle's position ( x n ) and velocity ( v n ).

where n is the vehicle ID. Since this traffic state s changes over time, we represent the traffic state at timestep t as s t . Then our differentiable traffic simulator is capable of providing the gradient of the simulation state at the next timestep t +1 with respect to the state at the current timestep t , or ds t +1 ds t .

This is possible because we use the Intelligent Driver Model (IDM) to model car-following behavior in our simulator [Treiber et al., 2000], which is differentiable. IDM describes vehicle speed ˙ x and acceleration ˙ v as a function of desired velocity v 0 , safe time headway in meters T , maximum acceleration a , comfortable deceleration b , the minimum distance between vehicles in meters δ , vehicle length l , and difference in speed with the vehicle in front ∆ v α as follows:

where α means the order of the vehicle in the lane. Therefore, ( α -1) -th vehicle runs right in front of α -th vehicle, and this relationship plays an important role in IDM. Also note that the computed acceleration term ( ˙ v ) is differentiable with respect to the other IDM variables.

IDM is just one of many car-following models used in traffic simulation literature. We choose IDM for its prevalence in previous mixed autonomy literature, however, any ODE-based car-following model will also work in our simulator as far as it is differentiable. In our simulator, automatic differentiation governs the gradient computation.

Lane Under IDM, lane membership is one of the most vital variables for simulation. This is because IDM builds on the leader-follower relationship. When a vehicle changes its lane with lateral movement, as shown as red arrows in Figure 9, such relationships could change, and our gradients do not convey any information about it, because lane membership is a discrete variable in nature. Note that this lateral movement also affects the longitudinal movement, rendered in green arrows in Figure 9, of a vehicle, and gradient from it is valid as far as the lane membership does not change. However, when the lane membership changes, even if our simulator gives gradient, it tells us only 'partial' information in the sense that it only gives information about longitudinal behavior. Therefore, we could say that analytical gradients we get from this environment is 'incomplete' (not useless at all, because it still gives us information about longitudinal movement), and thus biased.

Pace Car Problem In this problem, there is a single autonomous vehicle that we have to control to regulate other human driven vehicles, which follow IDM, to run at the given target speed. Since human driven vehicles control their speeds based on the relationship to their leading vehicles, our autonomous vehicle has to control itself to run in front of those human driven vehicles and adjust their speeds to the target speed. The number of lanes and human driven vehicles varies across different environments as follows.

Single Lane: Number of lanes = 1, Number of vehicles per lane = 1

2 Lanes: Number of lanes = 2, Number of vehicles per lane = 2

4 Lanes: Number of lanes = 4, Number of vehicles per lane = 4

10 Lanes: Number of lanes = 10, Number of vehicles per lane = 1

See Figure 10 for 10-lane environment.

Then we can formally define RL environments as follows, where N is the number of human driven vehicles in total, and v tgt is the target speed.

Episode Length: 1000

Observation: s ∈ R 2 N

Action: a ∈ R 2 ( a 0 = Acceleration, a 1 = Steering)

Reward: 1 -1 N ∑ N i =1 min( | v i -v tgt | v tgt , 1)

However, to facilitate learning, we additionally adopted following termination conditions to finish unpromising trials early.

Terminate when autonomous vehicle collides with human driven vehicle

Terminate when autonomous vehicle goes out of lane

Terminate when autonomous vehicle runs too far away from the other vehicles

Terminate when autonomous vehicle falls behind the other vehicles

When one of these conditions is met, it gets reward of -1 and the trial terminates.

7.6 Renderings

7.6.1 Traffic Simulation

Gradient flow As described in Appendix 7.5.2, analytical gradients are only valid along longitudinal direction, as illustrated with green arrows in Figure 9. When a vehicle changes lane with lateral movement, as shown with red arrows, even though such change would incur different behaviors of following vehicles in multiple lanes, analytical gradients cannot convey such information. However, they still tell us information about longitudinal behavior - which is one of the motivations of our research, to use analytical gradients even in these biased environments.

Pace Car Problem We additionally provide renderings of the 10-lane pace car problem defined in Appendix 7.5.2. In Figure 10, we can observe 10 parallel lanes, 10 human driven vehicles rendered in yellow, and the autonomous vehicle rendered in blue that we have to control. As shown there, human driven vehicles adjust their speeds based on IDM when the autonomous vehicle blocks their way. Therefore, the autonomous vehicle has to learn how to change their lanes, and also how to run in appropriate speed to regulate the following vehicles. Our experimental results in Section 5.3 show that our method achieves better score than the baseline methods by adopting biased analytical gradients in PPO.

(a)

0

-th frame

(b)

150

-th frame

MeanSpeed:18.16m/s

Mean Speed: 16.70 m/s

Mean Speed: 19.58 m/s

Mean Speed: 19.84 m/s

(c) 300 -th frame

(d) 450 -th frame

7.7 Computational Costs

We provide wall clock training times for each method in traffic environments to finish all the training epochs in Table 2. All of the trainings were done based on the settings in Appendix 7.5.2. Note that the longer training time does not always mean that the method is worse than the others - the method could have achieved better results in shorter time, while requiring much longer time to complete all the training epochs.

In Table 2, we can observe that GI-PPO requires a little more time than RP, as it computes analytical gradients as RP does, but requires an additional time for controlling α and doing PPO update. However, the additional cost is not so significant. In contrast, LR+RP consumes much more time than the other methods, even though it is based on the same spirit as ours. This is because it has to backpropagate through each trajectory in the experience buffer to estimate the sample variance of the gradients. Therefore, we can see that our method attains better results than LR+RP (Figure 4) with lower computational cost.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the combined use of both the analytical policy gradient and the likelihood ratio policy gradient for training policy networks, based on the PPO algorithm. To make the combination feasible, a new alpha-policy is introduced and its approximation technique has been successfully developed. Besides some theoretical studies, empirical results further show the potential usefulness of the newly proposed algorithm in solving some benchmark problems.\n\nSTRENGTHS:\nThe theoretical study in this paper regarding the analytical policy gradient and the alpha-policy sounds interesting and novel. The experiment results show that the new algorithm can be very useful on some benchmark problems.\n\nWEAKNESSES:\nWhile it is interesting to mix analytical policy gradients with learned policy gradients to enhance the reliability and performance of the policy network training algorithms, this idea is clearly not restricted to on-policy algorithms such as PPO. Although the authors made it clear that PPO is one of the most popularly used on-policy algorithms, it remains questionable why this paper only studies the effectiveness of using combined policy gradients in PPO. The possibility and potential limitations of using the proposed policy gradient combination technique on other algorithms, particularly off-policy algorithms, may need to be further justified and investigated.\n\nThis paper requires prior knowledge of the environment dynamics that must be differentiable (or partially differentiable) in nature. Many real-world reinforcement learning problems may not satisfy this requirement. Hence, the practical usefulness of the new algorithm remains a bit questionable. The authors are highly recommended to clearly evaluate and justify the practical value of the new algorithm.\n\nFurthermore, if the environment model is known in advance, it is possible to conduct effective model-based policy training without using analytical policy gradients. It remains unclear to me what the advantages of using analytic policy gradient techniques would be, compared to planning and other model-based reinforcement learning techniques. It is also desirable if the performance strength of using the analytic policy gradients, compared to other model-based reinforcement learning methods, can be experimentally evaluated and reported in this paper.\n\nThe authors stated several times in the paper that the analytical policy gradients would be highly biased when they are incomplete. I don't understand what this means and why it is true. Incompleteness and bias present two separate dimensions regarding the quality of analytical policy gradients. Why should they be strongly correlated?\n\nSome mathematical claims require more clarity. For example, Lemma 4.4 mentioned a particular advantage function A-hat. However, the actual definition of this function is not presented, making it hard to accurately understand this lemma and its applicability.\n\nSome critical details were missing regarding the new algorithm design in the main text. For example, the new algorithm requires you to adjust alpha from time to time. However, it is not clear how alpha is actually adjusted. The formula for adjusting alpha is not presented and clearly justified in the main text. Meanwhile, some claims in the algorithm design also need strong justifications. For example, it is stated on page 6 that by constraining the determinant of a matrix to be near 1, we can guarantee stable policy updates. However, the validity of this statement is not proved or properly explained. Moreover, if all eigenvalues are far greater than 1, alpha would be set to be very close to 0. In this case, would the analytical gradient become useless? Similarly, it is not clear why the ratio of the two determinants gives us the difference between the two policies.\n\nAccording to the experiment results presented in the main text, it seems that the new algorithm is only useful when the problem being solved has many poor local optima (hence the algorithm needs to inject some uncertainties or conducts more explorations) or when the complete analytical policy gradient cannot be obtained. On problems of differentiable physics simulations, using the analytical policy gradients alone appear to be able to achieve the best results. In view of this, the real practical value of the new algorithm may need to be further investigated on more benchmark problems.\n\nQUESTIONS:\nWhat are the possibilities and potential limitations of using the proposed policy gradient combination technique on other algorithms, particularly off-policy algorithms?\n\nWhat is the practical impact of the assumption regarding the prior knowledge of the environment dynamics that must be differentiable?\n\nWhat are the theoretical and practical advantages of the new algorithm design, compared to other model-based reinforcement learning methods?\n\nIncompleteness and bias present two separate dimensions regarding the quality of analytical policy gradients. Why should they be strongly correlated?\n\nLIMITATIONS:\nI do not have any concerns regarding this question.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel policy learning method that adopts analytical gradients into the PPO framework without the necessity of estimating LR gradients. The authors introduce an adaptive α-policy that allows us to dynamically adjust the influence of analytical gradients by evaluating the variance and bias of these gradients. The experiments show that the proposed method, GI-PPO, strikes a balance between analytical gradient-based policy updates and PPO-based updates, yielding solid results compared to baseline algorithms in various environments.\n\nSTRENGTHS:\nThis paper introduces original ideas about how to incorporate analytical gradients into the PPO framework, taking their variance and bias into account. As a result, a locally superior policy, adaptive $\\alpha$-policy is introduced and fits into PPO framework seamlessly, by dynamically adjusting the influence of analytical gradients. This paper is of solid technical quality. It provides theoretical derivations and analysis to demonstrate how $\\alpha$-policy fits into PPO framework seamlessly, the relationship between reparameterization RP gradient and $\\alpha$-policy, and theoretical guidance to adjust $\\alpha$ based on variance and bias. The experiments are extensive in general, demonstrating the solid performance compared to baseline in different environments. Last, the paper is clearly written in general.\n\nWEAKNESSES:\nIn Section 5.1 (page 7, line 229), Figure [8] is missing in main paper, although we may find the figure in Appendix. In Fig. 3, the plot is too small to distinguish the relative performance for different algorithms. In Section 5.2 (page 8), there is typo line 259,  it should be Figure [4] instead of Figure [5] for Differential Physics Simulation. In Fig. 4(b,c), GI-PPO is no better than RP for tasks Ant and Hopper in term of returning rewards. To strengthen the impact of the work, it might be better to add more challenging tasks, for example, Humanoid in Mujoco environment.\n\nQUESTIONS:\nThis paper is well written in general. However, for the readers to better understand the algorithm, can we move Algorithm 1 GI-PPO from Appendix and add into main paper? To utilize analytic gradients, it seems GI-PPO needs more extra computations from back-propagation. Can the author provide the computational cost comparison to different baseline algorithms?\n\nLIMITATIONS:\nThere is no code provided to reproduce the results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper integrates analytical gradients with PPO. They introduce an \\alpha-policy to control the bias and variance during the update of the policy. Results on some differentiable environments show the effectiveness of the proposed method.\n\nSTRENGTHS:\nThis paper study how to incorporate analytical gradient into existing Rl algorithm, which is interesting.\n\nWEAKNESSES:\nThe main weakness of the paper is that the writing of the paper is somewhat poor. \nThe presentation lacks motivation or necessary intuitive explanations. There are some imprecise statements and typos (see below).\nThese flaws make the paper hard to follow.\n\nMajor concerns:\n- This paper claims that it's a combination of analytical gradient and PPO. However, I didn't see where you use PPO (eq. 4 is not PPO!).\n- Some statements lack motivation. I don't understand why the algorithm is composed of three parts (Line 182). Could you give more explanation? And why the adjusting $\\alpha$ should be designed in that way? Besides, it would be good if there was more explanation of Def 4.1\n- The proposed method does not perform well on the physical simulation tasks. In Figure 4, the proposed method performs worse than the RP method on two high-dimensional tasks: Ant and Hopper.\n- In most of the practical tasks, it is hard to get the analytical gradient. This limits the potential of the proposed method. The paper only does experiments in simulation tasks. Could you give more examples of where we can get analytical gradients in practice?\n\nThere are some minor technical issues in the paper:\n- Line 106: The symbol $A$ is repeated with the action space $A$ (in Line 97).\n- Line 97: You assume the probability to be within $[0,1]$. This means that the action space is discrete. However, in Line 101, you assume action space is continuous.\n\n\nSome statements in the paper are wrong:\n- \"PPO relies on the observation that we can evaluate a policy πθ with our current policy πθ¯\". This is not right. Because the $\\rho_{\\pi_\\theta}(s)$ is not available.\n- Line 137: \"the difference between πθ¯ and πθ must be small enough to get an accurate estimate.\" In fact, in the theory of TRPO, they introduce a lower bound with KL divergence penalty term. They don't care whether that's an accurate estimate.\n\nQUESTIONS:\n- What do you mean by inherent chaotic nature (Line 25)? Could you please give some examples?\n- I'm confused about how is eq. 2 used in your method.\nPlease also see my comments in the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this manuscript, the author(s) combined the PPO framework with analytical gradients and proposed a policy learning method to learn better policy quickly. Through empirical experiments, the author(s) demonstrated that the proposed method was superior to baseline methods in several scenarios.\n\nSTRENGTHS:\n### Strong points:\n\n1) Utilizing $\\alpha$-policy to deal with the influence of analytical policy gradients when training;\n\n2) Adopting criteria to detect/calculate variance and bias;\n\n3) An algorithm was designed to handle the strength of analytical gradients during updating;\n\n4) Empirical results look good.\n\nBy the way, in the Supplementary Materials, the author(s) provided a mp4 video file, where the animation on the scenario of the traffic problem looks fantastic. Thanks for providing that.\n\nWEAKNESSES:\n### Weak points:\n\nAlthough the work of this manuscript looks interesting, some points in this manuscript are not so clear to me, I hope the author(s) can explain/detail them. Thanks.\n\n1) Some descriptions of empirical experiments are not so clear.\n\n2) Missing codes related to the proposed method in this manuscript. \n\n3) If possible, some theoretical results in this manuscript should be empirically verified.\n\nQUESTIONS:\nDetailedly, I have the following comments/questions:\n\n1) For the description or empirical experiment details, some places in this manuscript seem unclear. For example,\n\n(1) Definition 5 was mentioned several times in the manuscript, such as, in Lines 179, 188, and even in the Supplementary Materials, Line 419, but where is Definition 5? \n\n(2) The proof is mainly for Proposition, Lemma, Theorem, etc. In Line 419 of Supplementary Materials, it mentioned \"Proof of Definition 5\", I am not clear. \n\n(3) Could you please explain/clarify the fluctuation of the blue curve (i.e., GI-PPO) in Figure 3 (c)? Why caused this phenomenon?\n\n(4) The order number (or citation) of Figures in this manuscript is messy. For instance, in Line 229, Figure 8 is in the Supplementary Materials, right?  Figure 1 seems to be cited until Line 245?\n\n2) If possible, some theoretical results, for example, Proposition 4.5, can be verified by empirical experiments?\n\n3) Some other tiny issues/typos:\n\n(1) The font size used for the coordinates in Figure is too small. \n\n(2) The format of mathematical formulas in this manuscript is inconsistent. Some end with punctuation, and some don't.\n\n(3) Many issues in References. For instance, Sometimes the conference name is the full name plus the abbreviation, and sometimes only the full name; Missed source of the cited references; and so on. Hope the author(s) will check carefully and correct them.\n\nIn addition, I did not scrutiny the proofs step-by-step, but I think the proofs should be Ok.\n\nThank the author(s) for submitting this interesting work. Except for the several possible issues/questions mentioned above. I think this is a basically good manuscript. I look forward to the response from the author(s). If the author feels any of my comments are inappropriate, please feel free to point them out :-) Thanks.\n\nLIMITATIONS:\nYes. As the author(s) revealed, there might be several limitations, such as(1) How to approximate $\\alpha$-policy efficiently; (2) Adjusting PPO's clipping range; (3) Computational efficiency. Hope that the author(s) can further investigate these limitations and solve these potential problems in the near future. Good luck!", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a policy gradient algorithm based on Proximal Policy Optimization (PPO) algorithm. This new algorithm utilizes the analytical gradients from differential environments and achieves competitive performance in various scenarios including function optimization, classic control environments, and traffic control environments. \nTo integrate the analytical gradient into the PPO framework, they use the reparameterization trick and introduce the concept of $\\alpha$-policy which is a locally superior policy. When $\\alpha$ is sufficiently small, the difference between the original policy and this $\\alpha$-policy would be small enough to satisfy the difference constraint in the PPO algorithm. Moreover, they propose metrics considering variance, bias, and out-of-range-ratio to dynamically adjust $\\alpha$ during the training.\n\nSTRENGTHS:\n1. The idea of utilizing analytical gradients in physical control environments is powerful and can dramatically increase the efficiency of policy gradient algorithms when analytical gradients are available. This paper advances this idea by integrating it with the PPO algorithm and achieves competitive performance in various environments.\n\n2. This paper proposes the concept of $\\alpha$-policy and connect it with the reparameterization trick. $\\alpha$-policy is an adjustable policy such that when $\\alpha$ is small enough the difference between $\\alpha$-policy and the origin policy can be small enough to fit into PPO algorithm. \n\n3. This paper proposed metrics considering variance, bias, and out-of-range-ratio to dynamically adjust $\\alpha$ during training.\n\nWEAKNESSES:\n1. When trying to fit the analytical gradients into the PPO algorithm, this paper didn't consider dynamically changing the PPO clip limit. Thus, the analytical gradients are not fully utilized.\n\n2. More theoretical analysis and better ways on how to adjust $\\alpha$ during training could be done and proposed.\n\n3. The experimental performance showed improved performance compared to PPO but does not achieve state-of-art performance in some environments.\n\nQUESTIONS:\n1. Line 419 \"Proof of Definition 5\" does not match the actual definition index. The author should separate definition 4.1 into a definition and a proposition. \n\n2. Can the author integrate the variance reduction techniques in [Xu et al., 2022] to achieve better performance? If not, what changes can be made to accommodate this?\n\nLIMITATIONS:\nI did not identify any potential negative societal impact from this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}