SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sanghyun Son Laura Yu Zheng Ryan Sullivan Yi-Ling Qiao Ming C. Lin
{shh1295,lyzheng,rsulli,yilingq,lin}@umd.edu
We introduce a novel policy learning method that integrates analytical gradients from differentiable environments with the Proximal Policy Optimization (PPO) algorithm. To incorporate analytical gradients into the PPO framework, we introduce the concept of an  $\alpha$ -policy that stands as a locally superior policy. By adaptively modifying the  $\alpha$  value, we can effectively manage the influence of analytical policy gradients during learning. To this end, we suggest metrics for assessing the variance and bias of analytical gradients, reducing dependence on these gradients when high variance or bias is detected. Our proposed approach outperforms baseline algorithms in various scenarios, such as function optimization, physics simulations, and traffic control environments. Our code can be found online: https://github.com/SonSang/gippo.
Reinforcement learning (RL) techniques often take gradient-free experiences or trajectories to collect information and learn policies—suitable for many applications such as video games and recommender systems. In contrast, differentiable programming offers analytical gradients that describe the action-result relationship (Equation 2). In particular, recent developments in programming tools [Paszke et al., 2017] have enabled direct access to accurate analytical gradients of the environmental dynamics, as exemplified in physics simulations [de Avila Belbute-Peres et al., 2018, Degrave et al., 2019, Geilinger et al., 2020, Freeman et al., 2021, Qiao et al., 2021], instead of using differentiable models for approximation [Deisenroth and Rasmussen, 2011, Deisenroth et al., 2013]. Policy gradients estimated from the integration of these analytical gradients and the reparameterization trick (RP gradient) [Kingma and Welling, 2013] often display less variance than likelihood ratio policy gradients (LR gradient) [Williams and Peng, 1989, Glynn, 1990], derived using the log-derivative trick. Consequently, a subset of research has explored the utilization of these analytical gradients for RL applications, especially in physics domains [Qiao et al., 2021, Mora et al., 2021, Xu et al., 2022].
Nevertheless, in environments with inherent chaotic nature, the RP gradient is found to exhibit high variance [Parmas et al., 2018] and even empirical bias [Suh et al., 2022]. Therefore, a strategic approach is necessary for the adoption of analytical gradients in policy learning [Metz et al., 2021]. One such strategy involves the interpolation of the LR and RP gradients based on their variance and bias [Parmas et al., 2018, Suh et al., 2022], made possible by the fact that both are different estimators of the same policy gradient. However, this approach requires estimating the LR and RP gradient for each sample trajectory to compute sample variance, a process that can be time-consuming (Appendix 7.7). Additionally, for on-policy RL methods that do not initially estimate the LR gradient, such as Trust Region Policy Optimization (TRPO) [Schulman et al., 2015a] or Proximal Policy Optimization (PPO) [Schulman et al., 2017] algorithms, it remains uncertain how to evaluate the variance and bias of analytical gradients. These gradients, however, continue to be desirable when they exhibit low variance and provide insightful information about environmental dynamics.
Given these challenges, our study explores how to incorporate analytical gradients into the PPO framework, taking their variance and bias into account. Our primary contributions are threefold: (1) we propose a novel method to incorporate analytical gradients into the PPO framework without the necessity of estimating LR gradients, (2) we introduce an adaptive  $\alpha$ -policy that allows us to dynamically adjust the influence of analytical gradients by evaluating the variance and bias of these gradients, and (3) we show that our proposed method, GI-PPO, strikes a balance between analytical gradient-based policy updates and PPO-based updates, yielding superior results compared to baseline algorithms in various environments.
We validate our approach across diverse environments, including classical numerical optimization problems, differentiable physics simulations, and traffic control environments. Among these, traffic environments typify complex real-world scenarios where analytical gradients are biased. We demonstrate that our method surpasses baseline algorithms, even under these challenging conditions.
2.1 Policy Learning Without Analytical Gradients
The policy gradient is described as the gradient of the expected cumulative return in relation to policy parameters [Sutton et al., 1999]. For a stochastic policy, as examined in this paper, REINFORCE [Williams, 1992] represents one of the initial methods to employ a statistical estimator for the policy gradient to facilitate policy learning. The policy gradient obtained with this estimator is commonly referred to as a likelihood-ratio (LR) gradient [Glynn, 1990, Fu et al., 2015]. Despite the LR gradient's ability to operate without analytical gradients of dynamics and its unbiased nature, it frequently exhibits high variance [Sutton et al., 1998]. Furthermore, the nonstationarity of the policy renders stable learning difficult to guarantee [Konda and Tsitsiklis, 1999, Schulman et al., 2015b].
Several strategies have been proposed to address the high variance issue inherent to the LR gradient, including the use of additive control variates [Konda and Tsitsiklis, 1999, Greensmith et al., 2004]. Among these, the actor-critic method [Barto et al., 1983] is highly favored. However, the introduction of a value function may induce bias. To counteract this, the Generalized Advantage Estimator (GAE) [Schulman et al., 2015b] was developed to derive a more dependable advantage function. Concerning the issue of unstable learning, methods such as TRPO [Schulman et al., 2015a] and PPO [Schulman et al., 2017] propose optimizing a surrogate loss function that approximates a random policy's expected return. By constraining the policy updates to remain close to the current policy, these methods significantly enhance stability in learning compared to pure policy gradient methods, which makes them more preferred nowadays.
2.2 Policy Learning With Analytical Gradients
In contrast to the LR gradient, the RP gradient is based on the reparameterization trick, which mandates analytical gradients of dynamics yet typically achieves lower variance [Kingma and Welling, 2013, Murphy, 2022]. The RP gradient can be directly estimated over the entire time horizon using Backpropagation Through Time (BPTT) [Mozer, 1989] and subsequently used for policy updates. However, BPTT has been found to encounter multiple optimization challenges such as explosion or vanishing of analytical gradients over extended trajectories [Freeman et al., 2021]. This is because, surprisingly, the RP gradient, contrary to earlier beliefs, may experience exploding variance due to the chaotic nature of environments [Parmas et al., 2018, Metz et al., 2021]. Additionally, Suh et al. [2022] noted that the RP gradient could also be subject to empirical bias.
Several corrective measures exist for the issues concerning the RP gradient [Metz et al., 2021], such as using a truncated time window bootstrapped with a value function [Xu et al., 2022]. This approach has demonstrated its efficacy in addressing physics-related problems. Another class of solutions involves interpolating between LR and RP gradients by evaluating their sample variance and bias [Parmas et al., 2018, Suh et al., 2022]. Likewise, when we carefully address the problems of RP gradient, it can provide valuable information helpful for policy learning. However, when it comes to PPO, which is one of the most widely used RL methods nowadays, it is unclear how we can leverage the analytical gradients that comprise the RP gradient, as it does not estimate LR gradient which can be used for comparison to the RP gradient. In this paper, we address this issue and show that we can use analytical gradients to enhance the performance of PPO.
3.1 Goal
We formulate our problems as a Markov decision process (MDP) defined by a tuple  $(S, A, P, r, \rho_0, \gamma)$ , where  $S$  is a set of states,  $A$  is a set of actions,  $P: S \times A \times S \to \mathbb{R}$  is a state transition model,  $r: S \times A \to \mathbb{R}$  is a reward model,  $\rho_0$  is the probability distribution of the initial states, and  $\gamma$  is a discount factor.
Under this MDP model, our goal is to train a parameterized stochastic policy  $\pi_{\theta}:S\times A\to \mathbb{R}^{+}$  where  $\theta$  represents parameters, that maximizes its expected sum of discounted rewards,  $\eta (\pi_{\theta})$  ..
$$
\eta \left(\pi_ {\theta}\right) = \mathbb {E} _ {s _ {0}, a _ {0}, \dots \sim \pi_ {\theta}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \right]. \tag {1}
$$
In this paper, we denote the current policy that is used to collect experience as  $\pi_{\hat{\theta}}$
3.2 Analytical Gradients
We assume that  $S$  and  $A$  are continuous, differentiable spaces, and  $P$  and  $r$  are also differentiable models. Then, our differentiable environment provides us with following analytical gradients of observation or reward at a later timestep with respect to the action at the previous timestep:
$$
\frac {\partial s _ {t + 1 + k}}{\partial a _ {t}}, \frac {\partial r _ {t + k}}{\partial a _ {t}}, \tag {2}
$$
where  $k \geq 0$  is an integer. With these basic analytical gradients, we can compute the gradient of certain advantage function, which we denote as  $\frac{\partial A}{\partial a}$ . In this paper, we use Generalized Advantage Estimator (GAE) [Schulman et al., 2015b]. See Appendix 7.2.1 for details.
3.3 Policy Update
In this section, we discuss how we update policy in two different settings: the policy gradient method based on RP gradient and PPO.
3.3.1 RP Gradient
To compute the RP gradient, we rely on the reparameterization trick [Kingma and Welling, 2013]. This trick requires us that we can sample an action  $a$  from  $\pi_{\theta}$  by sampling a random variable  $\epsilon$  from some other independent distribution  $q$ . To that end, we assume that we use a continuous differentiable bijective function  $g_{\theta}(s,\cdot):\mathbb{R}^n\to \mathbb{R}^n$  that maps  $\epsilon$  to an action  $a$ , where  $n$  is the action dimension. Then the following holds because of injectivity,
$$
\left| \det  \left(\frac {\partial g _ {\theta} (s , \epsilon)}{\partial \epsilon}\right) \right| > 0, \forall \epsilon \in \mathbb {R} ^ {n}, \tag {3}
$$
and we can define the following relationship.
Definition 3.1 We define  $\pi_{\theta} \triangleq g_{\theta}$  if following holds for an arbitrary open set  $T_{\epsilon} \subset \mathbb{R}^{n}$ :
$$
\int_ {T _ {a}} \pi_ {\theta} (s, a) d a = \int_ {T _ {\epsilon}} q (\epsilon) d \epsilon ,
$$
where  $T_{a} = g_{\theta}(s,T_{\epsilon})$
Lemma 3.2 If  $\pi_{\theta} \triangleq g_{\theta}$ ,
$$
\pi_ {\theta} (s, a) = q (\epsilon) \cdot \left| \det \left(\frac {\partial g _ {\theta} (s , \epsilon)}{\partial \epsilon}\right) \right| ^ {- 1},
$$
where  $a = g_{\theta}(s,\epsilon)$ . The inverse is also true.
Proof: See Appendix 7.1.1.
Specifically, we use  $q = \mathcal{N}(0,I)$  in this paper, and define  $g_{\theta}$  as follows,
$$
g _ {\theta} (s, \epsilon) = \mu_ {\theta} (s) + \sigma_ {\theta} (s) \cdot \epsilon \quad \left(\left\| \sigma_ {\theta} (s) \right\| _ {2} > 0\right),
$$
which satisfies our assumption.
With  $g_{\theta}(s,\epsilon)$  and the analytical gradients from Equation 2, we obtain RP gradient by directly differentiating the objective function in Equation 1, which we use for gradient ascent. See Appendix 7.2.2 for details.
3.3.2 PPO
PPO relies on the observation that we can evaluate a policy  $\pi_{\theta}$  with our current policy  $\pi_{\bar{\theta}}$ , using its advantage function  $A_{\pi_{\bar{\theta}}}$  as
$$
\eta (\pi_ {\theta}) = \eta (\pi_ {\bar {\theta}}) + \int_ {s} \rho_ {\pi_ {\theta}} (s) \int_ {a} \pi_ {\theta} (s, a) A _ {\pi_ {\bar {\theta}}} (s, a),
$$
where  $\rho_{\pi_\theta}(s)$  is the discounted visitation frequencies. See [Kakade and Langford, 2002, Schulman et al., 2015a] for the details.
As far as the difference between  $\pi_{\bar{\theta}}$  and  $\pi_{\theta}$  is sufficiently small, we can approximate  $\rho_{\pi_{\theta}}(s)$  with  $\rho_{\pi_{\bar{\theta}}}(s)$  [Schulman et al., 2015a], and get following surrogate loss function:
$$
L _ {\pi_ {\bar {\theta}}} (\pi_ {\theta}) = \eta (\pi_ {\bar {\theta}}) + \int_ {s} \rho_ {\pi_ {\bar {\theta}}} (s) \int_ {a} \pi_ {\theta} (s, a) A _ {\pi_ {\bar {\theta}}} (s, a). \tag {4}
$$
Note that we can estimate this loss function with Monte Carlo sampling as described in Appendix 7.2.3. Since this is a local approximation, the difference between  $\pi_{\bar{\theta}}$  and  $\pi_{\theta}$  must be small enough to get an accurate estimate. TRPO [Schulman et al., 2015a] and PPO [Schulman et al., 2017] constrain the difference to be smaller than a certain threshold, and maximize the right term of (4). In particular, PPO restricts the ratio of probabilities  $\pi_{\theta}(s_i,a_i)$  and  $\pi_{\bar{\theta}}(s_i,a_i)$  for every state-action pair  $(s_i,a_i)$  in the buffer as follows to attain the goal, using a constant  $\epsilon_{clip}$ :
$$
1 - \epsilon_ {c l i p} <   \frac {\pi_ {\theta} \left(s _ {i} , a _ {i}\right)}{\pi_ {\bar {\theta}} \left(s _ {i} , a _ {i}\right)} <   1 + \epsilon_ {c l i p}. \tag {5}
$$
In this section, we discuss how we can use analytical gradients in the PPO framework while considering their variance and biases. We start our discussion with the definition of  $\alpha$ -policy.
4.1  $\alpha$ -Policy
With the analytical gradient of advantage with respect to an action  $(\nabla_{a}A_{\pi_{\bar{\theta}}}(\boldsymbol {s},\boldsymbol {a}))$  from our differentiable environments, we can define a new class of policies  $\pi_{\alpha}$ , parameterized by  $\alpha$ , as follows.
Definition 4.1 Given current policy  $\pi_{\bar{\theta}}$ , we define its  $\alpha$ -policy as follows:
$$
\pi_ {\alpha} (s, \tilde {a}) = \left\{ \begin{array}{c c c} \frac {\pi_ {\bar {\theta}} (s , a)}{\left| \det  \left(I + \alpha \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s , a)\right) \right|} & i f & \exists a s. t. \tilde {a} = f (a) = a + \alpha \cdot \nabla_ {a} A _ {\pi_ {\bar {\theta}}} (s, a) \\ 0 & e l s e \end{array} , \right. \tag {6}
$$
$$
w h e r e c o n s t a n t \left| \alpha \right| <   \frac {1}{\max  _ {(s , a)} \left| \lambda_ {1} (s , a) \right|}.
$$
Here  $\lambda_1(s,a)$  represents the minimum eigenvalue of  $\nabla_{a}^{2}A_{\pi_{\tilde{\theta}}}(\bar{s},a)$ .
Lemma 4.2 Mapping  $f$  is injective, and for an arbitrary open set of action  $T \subset A$ ,  $\pi_{\alpha}(s,\cdot)$  selects a set of action  $\tilde{T} = f(A)$  with the same probability that  $\pi_{\bar{\theta}}(s,\cdot)$  selects  $A$ .
(a) De Jong's Function
(b) Ackley's Function
Figure 1:  $\alpha$ -policies for 1-dimensional (a) De Jong's function and (b) Ackley's function. The original policy is rendered in blue, and alpha policies for different  $\alpha$  values are rendered in orange and red. Those  $\alpha$ -policies are obtained from analytical gradients of the given function  $f(x)$  rendered in black. Note that Ackley's  $\alpha$ -policy becomes invalid when  $\alpha = 2 \cdot 10^{-3}$ , because of singularities near 0.
Proof: See Appendix 7.1.2.
In Figure 1, we provide renderings of  $\alpha$ -policies for different  $\alpha$ s when we use 1-dimensional De Jong's function and Ackley's function [Molga and Smutnicki, 2005] as our target functions. The illustration provides us an intuition that when  $\alpha$  is sufficiently small,  $\pi_{\alpha}$  can be seen as a policy that selects slightly better action than  $\pi_{\bar{\theta}}$  with the same probability. In fact, we can prove that  $\pi_{\alpha}$  indeed gives us better estimated expected return in Equation 4 when  $\alpha > 0$  is sufficiently small.
Proposition 4.3 If  $|\alpha| \ll 1$ ,
$$
L _ {\pi_ {\bar {\theta}}} \left(\pi_ {\alpha}\right) - \eta \left(\pi_ {\bar {\theta}}\right) = O (| \alpha |) \text {w h e n} \alpha > 0, \tag {7}
$$
$$
\eta \left(\pi_ {\bar {\theta}}\right) - L _ {\pi_ {\bar {\theta}}} \left(\pi_ {\alpha}\right) = O (| \alpha |) \text {w h e n} \alpha <   0,
$$
where  $L_{\pi_{\tilde{\theta}}}$  denotes estimated expected return defined in Equation 4.
Proof: See Appendix 7.1.3.
This proposition tells us that  $\alpha$ -policy fits into PPO framework seamlessly, and thus if we update our policy towards  $\alpha$ -policy, it does not contradict the PPO's objective. However, since there is a second order derivative  $\nabla_{a}^{2}A_{\pi_{\bar{\theta}}} (s, a)$  in the definition, it is unclear how we can update our policy  $\pi_{\theta}$  towards  $\pi_{\alpha}$ . In the next section, we show how we can do this and how it is related to RP gradient.
4.2 Relationship between RP gradient and  $\alpha$ -policy
To show how we can approximate  $\alpha$ -policy, for a deterministic function  $g_{\bar{\theta}}$  such that  $\pi_{\bar{\theta}} \triangleq g_{\bar{\theta}}$ , we define a new function  $g_{\alpha}$  as follows, and provide a Lemma about it.
$$
g _ {\alpha} (s, \epsilon) = a + \alpha \cdot \nabla_ {a} A _ {\pi_ {\bar {\theta}}} (s, a), \text {w h e r e} a = g _ {\bar {\theta}} (s, \epsilon). \tag {8}
$$
Lemma 4.4 When  $a = g_{\bar{\theta}}(s,\epsilon)$
$$
\det  \left(\frac {\partial g _ {\alpha} (s , \epsilon)}{\partial \epsilon}\right) \cdot \det  \left(\frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon}\right) ^ {- 1} = \det  \left(I + \alpha \cdot \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, a)\right). \tag {9}
$$
Proof: See Appendix 7.1.4.
Note that we can update  $g_{\theta}$  to approximate  $g_{\alpha}$  by minimizing the following loss.
$$
L (\theta) = \mathbb {E} _ {s _ {0}, \epsilon_ {0},.. \sim q} \left[ \left\| g _ {\theta} \left(s _ {t}, \epsilon_ {t}\right) - g _ {\alpha} \left(s _ {t}, \epsilon_ {t}\right) \right\| ^ {2} \right]. \tag {10}
$$
In fact, it turns out that minimizing Equation 10 leads our policy  $\pi_{\theta}$  to approximate  $\pi_{\alpha}$ , because of the following Proposition.
Proposition 4.5 If  $\pi_{\bar{\theta}} \triangleq g_{\bar{\theta}}$ , for  $\alpha$  that satisfies the constraint in Definition 4.1,  $\pi_{\alpha} \triangleq g_{\alpha}$ .
Proof: See Appendix 7.1.6.
Note that we can use different advantage function formulations for  $A$  in Equation 8, such as GAE [Schulman et al., 2015b] that we use in this work. However, let us consider the following advantage function  $\hat{A}$  to see the relationship between RP gradient and  $\alpha$ -policy.
$$
\hat {A} _ {\pi_ {\theta}} \left(s _ {t}, a _ {t}\right) = \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t}, \dots \sim \pi_ {\theta}} \left[ \sum_ {k = t} ^ {\infty} \gamma^ {k} r \left(s _ {k}, a _ {k}\right) \right], \tag {11}
$$
Then, the following Lemma holds.
Lemma 4.6 When  $\alpha = 1$ , if we define  $g_{\alpha}$  with  $\hat{A}$ , the  $RP$  gradient corresponds to  $\frac{\partial L}{\partial \theta}$  at  $\theta = \bar{\theta}$ .
Proof: See Appendix 7.1.5.
This Lemma tells us that we can understand the RP gradient as the very first gradient we get when we update our policy toward  $\pi_{\alpha}$  by minimizing Equation 10. In other words,  $\alpha$ -policy is not only a superior policy than the original one in the PPO's point of view (Proposition 4.3), but also a target policy for RP gradient (Lemma 4.6). Therefore, we conclude that  $\alpha$ -policy is a bridge that connects these two different policy update regimes.
4.3 Algorithm
Our algorithm is mainly comprised of 3 parts: (1) Update  $\pi_{\theta}$  to  $\pi_{\alpha}$  by minimizing Equation 10, (2) Adjust  $\alpha$  for next iteration, and (3) Update  $\pi_{\theta}$  by maximizing Equation 4. In this section, we discuss details about Step 2 and 3, and provide the outline of our algorithm.
4.3.1 Step 2: Adjusting  $\alpha$
In Definition 4.1, we can observe that  $\alpha$  controls the influence of analytical gradients in policy updates. If  $\alpha = 0$ , it is trivial that  $\pi_{\alpha} = \pi_{\bar{\theta}}$ , which means that analytical gradients are ignored. In contrast, as  $\alpha$  gets bigger, analytical gradients play bigger roles, and  $\pi_{\alpha}$  becomes increasingly different from  $\pi_{\bar{\theta}}$ . Then, our mission is to find a well-balanced  $\alpha$  value. Rather than setting it as a fixed hyperparameter, we opt to adjust it adaptively. We first present the following variance and bias criteria to control  $\alpha$ , as it is commonly used to assess the quality of gradients [Parmas et al., 2018, Suh et al., 2022].
Variance After we approximate  $\pi_{\alpha}$  in Step 1), we can estimate  $\operatorname*{det}(I + \alpha \nabla_a^2 A_{\pi_\bar{\theta}}(s,a))$  for every state-action pair in our buffer using Lemma 4.4, and adjust  $\alpha$  to bound the estimated values in a certain range,  $[1 - \delta, 1 + \delta]$ . There are two reasons behind this strategy.
- As shown in Definition 4.1, we have to keep  $\alpha$  small enough so that  $\pi_{\alpha}$  does not become invalid policy. By keeping  $\operatorname*{det}(I + \alpha \nabla_a^2 A_{\pi_\delta}(s,a))$  larger than  $1 - \delta$  to keep its positiveness, we guarantee that our policy is not updated towards an invalid policy.
- Note that  $\operatorname*{det}(I + \alpha \nabla_{a}^{2}A_{\pi_{\bar{\theta}}} (s, a)) = \Pi_{i=1}^{n} (1 + \alpha \lambda_{i})$ , where  $\lambda_{i}$  is the  $i$ -th eigenvalue of  $\nabla_{a}^{2}A_{\pi_{\bar{\theta}}} (s, a)$ . By constraining this value near 1 by adjusting  $\alpha$ , we can guarantee stable policy updates even when some eigenvalues of  $\nabla_{a}^{2}A_{\pi_{\bar{\theta}}} (s, a)$  have huge magnitudes, which is related to a high variance RP gradient. In Appendix 7.3.1, we provide empirical evidence that this estimation is related to the sample variance of analytical gradients.
Bias By Proposition 4.3, we already know that when we estimate the right term of Equation 4 after Step 1), it should be a positive value. However, if the analytical gradients were biased, this condition could not be met. In runtime, if we detect such cases, we decrease  $\alpha$ .
In addition, we use one more criterion to adjust  $\alpha$ , which we call the out-of-range ratio. This criterion was designed to promise the minimum amount of policy update by PPO.
Out-of-range-ratio We define out-of-range-ratio as follows,
$$
\text {o u t - o f - r a n g e - r a t i o} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {I} \left(\left| \frac {\pi_ {\theta} \left(s _ {i} , a _ {i}\right)}{\pi_ {\bar {\theta}} \left(s _ {i} , a _ {i}\right)} - 1 \right| > \epsilon_ {\text {c l i p}}\right), \tag {12}
$$
where  $N$  is the size of the buffer,  $\mathbb{I}$  is the indicator function, and  $\epsilon_{clip}$  is the constant from Equation 5. We assess this ratio after the policy update in Step 1), allowing its value to be non-zero when  $\alpha >0$ . A non-negligible value of this ratio indicates a potential violation of the PPO's restriction outlined in Equation 5, thereby compromising the effectiveness of PPO. Further elaboration can be found in Appendix 7.3.2. Therefore, when this ratio exceeds a predetermined threshold, we subsequently reduce the value of  $\alpha$ .
To sum it up, our strategy to control  $\alpha$  aims at decreasing it when analytical gradients exhibit high variance or bias, or when there is little room for PPO updates. This results in greater dependence on PPO, which can be thought of as a safeguard in our approach. Otherwise, we increase  $\alpha$ , since we can expect higher expected return with a greater  $\alpha$  as shown in Proposition 4.3.
4.3.2 Step 3: PPO Update
Since we do policy updates based on PPO after the update based on  $\pi_{\alpha}$ , there is a possibility that the PPO update can "revert" the update from Step 1). To avoid such situations, we propose to use the following  $\pi_h$  in the place of  $\pi_{\bar{\theta}}$  in Equation 5: