SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\nabla_ {a} \hat {A} _ {\pi_ {\theta}} \left(s _ {t}, a _ {t}\right) = \frac {1}{2} \mathbb {E} _ {s _ {t}, \epsilon_ {t}, \dots \sim q} \left[ \sum_ {k = t} ^ {\infty} \gamma^ {k} \frac {\partial r \left(s _ {k} , g _ {\theta} \left(s _ {t} , \epsilon_ {t}\right)\right)}{\partial g _ {\theta} \left(s _ {t} , \epsilon_ {t}\right)} \right]. \tag {13}
$$
Then, by differentiating Equation 10 with respect to  $\theta$  after plugging in  $\hat{A}$ , we can get following relationship:
$$
\begin{array}{l} \frac {\partial L (\theta)}{\partial \theta} = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots , \sim q} \left[ \sum_ {t = 0} ^ {\infty} 2 \cdot \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \cdot \left(g _ {\theta} (s _ {t}, \epsilon_ {t}) - g _ {\alpha} (s _ {t}, \epsilon_ {t})\right) \right] \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots , \sim q} \left[ \sum_ {t = 0} ^ {\infty} 2 \cdot \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \cdot \left(g _ {\theta} (s _ {t}, \epsilon_ {t}) - \left(g _ {\theta} (s _ {t}, \epsilon_ {t}) + \nabla_ {a} \hat {A} _ {\pi_ {\theta}} (s _ {t}, a _ {t}))\right)\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} (\because \alpha = 1) \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots , \sim q} \left[ \sum_ {t = 0} ^ {\infty} - 2 \cdot \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \cdot \nabla_ {a} \hat {A} _ {\pi_ {\theta}} (s _ {t}, a _ {t}) \right] \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots , \sim q} \left[ \sum_ {t = 0} ^ {\infty} - 2 \cdot \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \cdot \frac {1}{2} \cdot \sum_ {k = t} ^ {\infty} \gamma^ {k} \frac {\partial r (s _ {k} , g _ {\theta} (s _ {t} , \epsilon_ {t}))}{\partial g _ {\theta} (s _ {t} , \epsilon_ {t})} \right] \\ \end{array}
$$
(  $\because$  Equation 13)
$$
= \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots , \sim q} \left[ \sum_ {t = 0} ^ {\infty} \sum_ {k = t} ^ {\infty} - \gamma^ {k} \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \cdot \frac {\partial r (s _ {k} , g _ {\theta} (s _ {t} , \epsilon_ {t}))}{\partial g _ {\theta} (s _ {t} , \epsilon_ {t})} \right],
$$
which equals to RP gradient in Appendix 7.2.2, but with different sign. However, they turn out to be the same because we minimize  $L(\theta)$ , but maximize  $\eta(\pi_{\theta})$ . Therefore, we can say that we gain RP gradient as the first gradient when we minimize Equation 10 for particular advantage function  $\hat{A}$  and  $\alpha = 1$ .
7.1.6 Proof of Proposition 4.5
By Definition 4.1 and Lemma 4.4,
$$
\begin{array}{l} \frac {\pi_ {\alpha} (s , \tilde {\alpha})}{\pi_ {\bar {\theta}} (s , a)} = \frac {1}{| \det  (I + \alpha \cdot \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s , a)) |} \\ = | \det  (\frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon}) | \cdot | \det  (\frac {\partial g _ {\alpha} (s , \epsilon)}{\partial \epsilon}) | ^ {- 1}, \\ \end{array}
$$
where  $a = g_{\bar{\theta}}(s,\epsilon)$ , and thus  $\tilde{a} = g_{\alpha}(s,\epsilon)$ . Since  $\pi_{\bar{\theta}} \triangleq g_{\bar{\theta}}$ , following holds:
$$
\pi_ {\alpha} (s, \tilde {\alpha}) = \pi_ {\bar {\theta}} (s, a) \cdot | \det (\frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon}) | \cdot | \det (\frac {\partial g _ {\alpha} (s , \epsilon)}{\partial \epsilon}) | ^ {- 1}
$$
$$
= q (\epsilon) \cdot | \det  (\frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon}) | ^ {- 1} \cdot | \det  (\frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon}) | \cdot | \det  (\frac {\partial g _ {\alpha} (s , \epsilon)}{\partial \epsilon}) | ^ {- 1}
$$
(·: Lemma 3.2)
$$
= q (\epsilon) \cdot | \det  (\frac {\partial g _ {\alpha} (s , \epsilon)}{\partial \epsilon}) | ^ {- 1},
$$
which implies  $\pi_{\alpha} \triangleq g_{\alpha}$ .
7.2 Formulations
7.2.1 Analytical Gradient of Generalized Advantage Estimator (GAE)
GAE [Schulman et al., 2015b] has been widely used in many RL implementations [Schulman et al., 2017, Raffin et al., 2021, Makoviichuk and Makoviychuk, 2022] to estimate advantages. GAE finds a balance between variance and bias of the advantage estimation by computing the exponentially-weighted average of the TD residual terms  $(\delta_t^V)$  Sutton et al. [1998], which are defined as follows:
$$
\delta_ {t} ^ {V} = r _ {t} + \gamma V (s _ {t + 1}) - V (s _ {t}).
$$
Then, GAE can be formulated as follows:
$$
A _ {t} ^ {G A E} (\gamma , \lambda) = \sum_ {k = 0} ^ {\infty} (\gamma \lambda) ^ {k} \delta_ {t + k} ^ {V}. \tag {14}
$$
We can compute the gradients for these terms as:
$$
\begin{array}{l} \frac {\partial A _ {t} ^ {G A E}}{\partial a _ {t}} = \sum_ {k = 0} ^ {\infty} (\gamma \lambda) ^ {k} \frac {\partial \delta_ {t + k} ^ {V}}{\partial a _ {t}} \\ = (\frac {1}{\gamma \lambda}) ^ {t} \sum_ {k = 0} ^ {\infty} (\gamma \lambda) ^ {t + k} \frac {\partial \delta_ {t + k} ^ {V}}{\partial a _ {t}} \\ = \left(\frac {1}{\gamma \lambda}\right) ^ {t} \left[ \sum_ {k = 0} ^ {\infty} (\gamma \lambda) ^ {t + k} \frac {\partial \delta_ {t + k} ^ {V}}{\partial a _ {t}} + \sum_ {k = 0} ^ {t - 1} (\gamma \lambda) ^ {k} \frac {\partial \delta_ {k} ^ {V}}{\partial a _ {t}} \right] \tag {15} \\ \end{array}
$$
$$
\begin{array}{l} \left(\because \frac {\partial \delta_ {k} ^ {V}}{\partial a _ {t}} = 0 \text {f o r} k <   t\right) \\ = \left(\frac {1}{\gamma \lambda}\right) ^ {t} \sum_ {k = 0} ^ {\infty} (\gamma \lambda) ^ {k} \frac {\partial \delta_ {k} ^ {V}}{\partial a _ {t}} \\ = (\frac {1}{\gamma \lambda}) ^ {t} \frac {\partial A _ {0} ^ {G A E}}{\partial a _ {t}}. \\ \end{array}
$$
Based on this relationship, we can compute every  $\frac{\partial A_t^{GAE}}{\partial a_t}$  with only one backpropagation of  $A_0^{GAE}$ , rather than backpropagating for every  $A_t^{GAE}$ .
7.2.2 RP Gradient Formulation
To differentiate Equation 1 with respect to  $\theta$ , we first rewrite Equation 1 as follows.
$$
\begin{array}{l} \eta (\pi_ {\theta}) = \mathbb {E} _ {s _ {0}, a _ {0}, \dots \sim \pi_ {\theta}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) \right] \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots \sim q} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, g _ {\theta} (s _ {t}, \epsilon_ {t})) \right]. \\ \end{array}
$$
Then, we can compute RP gradient as follows.
$$
\begin{array}{l} \frac {\partial \eta (\pi_ {\theta})}{\partial \theta} = \frac {\partial}{\partial \theta} \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots \sim q} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, g _ {\theta} (s _ {t}, \epsilon_ {t})) \right] \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots \sim q} \left[ \frac {\partial}{\partial \theta} \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, g _ {\theta} (s _ {t}, \epsilon_ {t})) \right] \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots \sim q} \left[ \sum_ {t = 0} ^ {\infty} \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \sum_ {k = t} ^ {\infty} \gamma^ {k} \frac {\partial r (s _ {k} , g _ {\theta} (s _ {k} , \epsilon_ {k}))}{\partial g _ {\theta} (s _ {t} , \epsilon_ {t})} \right] \\ \left(\because \frac {\partial r \left(s _ {k} , g _ {\theta} \left(s _ {k} , \epsilon_ {k}\right)\right)}{\partial g _ {\theta} \left(s _ {t} , \epsilon_ {t}\right)} \neq 0 \text {o n l y w h e n} k \geq t.\right) \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \dots \sim q} \left[ \sum_ {t = 0} ^ {\infty} \sum_ {k = t} ^ {\infty} \gamma^ {k} \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \frac {\partial r (s _ {k} , g _ {\theta} (s _ {k} , \epsilon_ {k}))}{\partial g _ {\theta} (s _ {t} , \epsilon_ {t})} \right] \\ = \mathbb {E} _ {s _ {0}, \epsilon_ {0}, \ldots \sim q} \left[ \sum_ {t = 0} ^ {\infty} \sum_ {k = t} ^ {\infty} \gamma^ {k} \frac {\partial g _ {\theta} (s _ {t} , \epsilon_ {t})}{\partial \theta} \frac {\partial r (s _ {k} , a _ {k})}{\partial a _ {t}} \right]. \\ \end{array}
$$
We can estimate this RP gradient using Monte Carlo sampling and terms in Equation 2, and use it for gradient ascent to maximize Equation 1.
(b) Ant
Figure 5: The first row shows the sample variance of analytical gradients  $(\frac{\partial A}{\partial a})$ , and the second row shows the statistics of estimates of  $\operatorname*{det}(I + \alpha \nabla_a^2 A_{\pi_\bar{\theta}}(s,a))$  for every state-action pair in the buffer after each epoch of training in 3 different differentiable physics environments (Cartpole, Ant, and Hopper). We used a fixed  $\alpha = 10^{-2}$ . Note that the minimum and maximum estimates in the second row exhibit a clear tendency to deviate from 1 more as the sample variance increases.
7.2.3 Estimator for Equation 4
Since  $\eta (\pi_{\bar{\theta}})$  does not depend on  $\theta$ , we can ignore the term and rewrite our loss function in Equation 4 using expectation as follows:
$$
\begin{array}{l} L _ {\pi_ {\bar {\theta}}} (\pi_ {\theta}) = \int_ {s} \rho_ {\pi_ {\bar {\theta}}} (s) \int_ {a} \pi_ {\theta} (s, a) A _ {\pi_ {\bar {\theta}}} (s, a) \\ = \mathbb {E} _ {s \sim \rho_ {\pi_ {\bar {\theta}}}, a \sim \pi_ {\theta} (s, \cdot)} \left[ A _ {\pi_ {\bar {\theta}}} (s, a) \right]. \\ \end{array}
$$
However, note that we did not collect trajectories, or experience buffer, using  $\pi_{\theta}$ . Therefore, we use another importance sampling function  $q(s,a)$  [Schulman et al., 2015a].
$$
L _ {\pi_ {\bar {\theta}}} (\pi_ {\theta}) = \mathbb {E} _ {s \sim \rho_ {\pi_ {\bar {\theta}}}, a \sim q (s, \cdot)} \Big [ \frac {\pi_ {\theta} (s , a)}{q (s , a)} A _ {\pi_ {\bar {\theta}}} (s, a) \Big ].
$$
Then, we can estimate this value using Monte Carlo estimation as follows, where  $B$  is the experience buffer of size  $N$ , which stores state  $(s_i)$ , action  $(a_i)$ , and their corresponding advantage value  $(A_{\pi_{\bar{\theta}}} (s_i, a_i))$  obtained by following policy  $\pi_{\bar{\theta}}$ :
$$
L _ {\pi_ {\bar {\theta}}} \left(\pi_ {\theta}\right) \approx \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\pi_ {\theta} \left(s _ {i} , a _ {i}\right)}{\pi_ {\bar {\theta}} \left(s _ {i} , a _ {i}\right)} A _ {\pi_ {\bar {\theta}}} \left(s _ {i}, a _ {i}\right). \tag {16}
$$
Note that we used  $\pi_{\bar{\theta}}$  in the place of  $q$ , as it is the most natural importance sampling function we can use [Schulman et al., 2015a, 2017].
7.3 Algorithm
Here we present the details of our algorithm in Section 4.3. To be specific, we present 1) an empirical evidence to validate our strategy to control  $\alpha$ , 2) the exact loss function that we use in PPO update, and 3) provide the change of  $\alpha$  during real training to help understanding.
7.3.1 Empirical evidence
In Section 4.3, we argued that the estimate of  $\operatorname*{det}(I + \alpha \nabla_a^2 A_{\pi_\bar{\theta}}(s,a))$  is related to the sample variance of analytical gradients, and we can get the estimate using Lemma 4.4. In Figure 5, we empirically prove the validity of this argument in the differentiable physics environments used in Section 5.2. In the illustration, we can observe a clear relationship between the sample variance of analytical gradients and the statistics of our estimates. Therefore, we can use these estimates as a cue to control  $\alpha$  based on the variance of analytical gradients.
(a) De Jong's Function (64)
Figure 6: Change of  $\alpha$  during solving function optimization problems in Section 5.1. Since these problems are one-step problems, and observations are all the same, there is no difference between mean, min, and max of estimated determinant.
(b) Ackley's Function (64)
7.3.2 Loss function for PPO Update
As discussed in Section 3.3.2, PPO finds a better policy in the proximity of the original policy by restricting the ratio of probabilities  $\pi_{\theta}(s_i,a_i)$  and  $\pi_{\bar{\theta}}(s_i,a_i)$ , as shown in Equation 5. In fact, PPO uses following surrogate loss function, which is a modified version of Equation 16, to mandate this:
$$
L _ {P P O} (\theta) = \frac {1}{N} \sum_ {i = 1} ^ {N} \min  \left(\frac {\pi_ {\theta} \left(s _ {i} , a _ {i}\right)}{\pi_ {\bar {\theta}} \left(s _ {i} , a _ {i}\right)} A _ {\pi_ {\bar {\theta}}} \left(s _ {i}, a _ {i}\right), c l i p \left(\frac {\pi_ {\theta} \left(s _ {i} , a _ {i}\right)}{\pi_ {\bar {\theta}} \left(s _ {i} , a _ {i}\right)}, 1 - \epsilon_ {c l i p}, 1 + \epsilon_ {c l i p}\right) A _ {\pi_ {\bar {\theta}}} \left(s _ {i}, a _ {i}\right)\right). \tag {17}
$$
Note that if out-of-range-ratio defined in Equation 12 is already 1.0, which corresponds to an extreme case, the gradient of  $L_{PPO}(\theta)$  with respect to  $\theta$  is 0. Therefore, PPO does not contribute to the policy update at all in this case. It justifies our decision to upper bound out-of-range-ratio by restricting  $\alpha$ .
Also note that we proposed to use a virtual policy  $\pi_h$  instead of  $\pi_{\bar{\theta}}$  in Section 4.3.2 to restrict PPO update to be done around  $\pi_{\alpha}$ . This changes the importance sampling function in Equation 17 as follows, which corresponds to our final surrogate loss function to use in PPO update:
$$
L _ {G I P P O} (\theta) = \frac {1}{N} \sum_ {i = 1} ^ {N} \min  \left(\frac {\pi_ {\theta} \left(s _ {i} , a _ {i}\right)}{\pi_ {h} \left(s _ {i} , a _ {i}\right)} A _ {\pi_ {\theta}} \left(s _ {i}, a _ {i}\right), c l i p \left(\frac {\pi_ {\theta} \left(s _ {i} , a _ {i}\right)}{\pi_ {h} \left(s _ {i} , a _ {i}\right)}, 1 - \epsilon_ {c l i p}, 1 + \epsilon_ {c l i p}\right) A _ {\pi_ {\theta}} \left(s _ {i}, a _ {i}\right)\right). \tag {18}
$$
7.3.3 Example: Function Optimization Problems (Section 5.1)
Here we present how  $\alpha$  changes as we solve function optimization problems in Section 5.1 with Figure 6. There are 4 plots for each of the problems.
- Alpha: Shows change of  $\alpha$  over training epoch.
- Estimated current performance: Shows expected additional return, which corresponds to Equation 16 ( $R_{\alpha}$  in Algorithm 1).
Estimate Determinant: Shows statistics of estimated  $\operatorname*{det}(I + \alpha \cdot \nabla_a^2 A_{\pi_\theta}(s,a))$
- Out of Range Ratio: Shows out-of-range-ratio in Equation 12.
For these problems, we have set  $\alpha_0 = 10^{-5}$ ,  $\delta_{det} = 0.4$ , and  $\delta_{oorr} = 0.5$ .
For De Jong's function (Figure 6a), we can observe that  $\alpha$  shows steady increase over time, but it is mainly upper bounded by variance criterion and out-of-range-ratio. In the end,  $\alpha$  stabilizes around  $10^{-3}$ . In contrast, for Ackley's function (Figure 6b) we can see that  $\alpha$  increases rapidly at first, but it
(a) Statistics of training in Ant environment when  $\delta_{oorr} = 0.5$
(b) Statistics of training in Ant environment when  $\delta_{oorr} = 1.0$
Figure 7: Change of  $\alpha$  during training in Ant environment in Section 5.2. In this environment, maximum out-of-range-ratio ( $\delta_{oorr}$ ) plays a major role in bounding  $\alpha$  from growing. Observe that GI-PPO achieves better results than RP when we set  $\delta_{oorr} = 1.0$ .
soon decreases due to the variance and bias criteria. Compare these graphs with optimization curve in Figure 2d. Then we would be able to observe that this large  $\alpha$  contributes to the faster convergence of GI-PPO compared to PPO at the early stage. However, after that,  $\alpha$  decreases slowly due to the out-of-range-ratio. In the end,  $\alpha$  reaches approximately  $10^{-6}$ , which is much lower than  $10^{-3}$  of De Jong's function. These observations align well with our intuition that RP gradient would play bigger role for De Jong's function than Ackley's function, because De Jong's function exhibits RP gradients with much lower variance than those of Ackley's.
7.3.4 Example: Physics Simulation Problems (Section 5.2)
In Section 5.2, GI-PPO could not achieve better results than RP in Ant and Hopper environments. For Ant environment, we found out that the hyperparameter  $\delta_{oorr}$  was a bottleneck that bounds GI-PPO's performance. In Figure 7, we illustrated the change of  $\alpha$  during training along the change of out-of-range-ratio. When we set  $\delta_{oorr} = 0.5$ ,  $\alpha$  keeps decreasing over time to keep the ratio around 0.5, which means the diminishing role of RP gradients in training. In contrast, when we set  $\delta_{oorr} = 1.0$ , after 750 epochs,  $\alpha$  does not decrease anymore, and out-of-range ratio stays near 1. This means that PPO rarely affects the training - policy update is almost driven by RP gradients.
Likewise, there are environments where RP gradients are much more effective than the other policy gradients. In those situations, it would be more desirable to solely depend on RP gradients without considering PPO. Since our method considers RP gradients in the PPO viewpoint, it is hard to detect such cases yet.
7.4 Experimental Details
7.4.1 Baseline Methods
Here we explain implementation details of the baseline methods that were used for comparisons in Section 5. First, we'd like to point out that we used relatively a short time horizon to collect experience, and used the critic network for bootstrapping, instead of collecting experience for the whole trajectory for all the methods. Therefore, the collected experience could be biased. However, we chose to implement in this way, because it allows faster learning than collecting the whole trajectory in terms of number of simulation steps.
LR We can estimate original LR gradient as follows, using log-derivative trick [Williams and Peng, 1989, Glynn, 1990]
$$
\begin{array}{l} \frac {\partial \eta (\pi_ {\theta})}{\partial \theta} = \frac {\partial}{\partial \theta} \mathbb {E} _ {s _ {0}, a _ {0}, \dots \sim \pi_ {\theta}} \left[ L (s _ {0}, a _ {0}, \dots) \sum_ {t = 0} ^ {\infty} \log \pi_ {\theta} (s _ {0}, a _ {0}) \right] \\ = \frac {\partial}{\partial \theta} \mathbb {E} _ {s _ {0}, a _ {0}, \dots \sim \pi_ {\theta}} \left[ \sum_ {t = 0} ^ {\infty} L (s _ {0}, a _ {0}, \dots) \log \pi_ {\theta} (s _ {0}, a _ {0}) \right], \\ \end{array}
$$
where  $L(s_0, a_0, \ldots) = \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t)$  is the discounted accumulated return. However, this LR gradient often suffers from high variance, because  $L$  varies a lot from trajectory to trajectory. To reduce such variance and faithfully compare our method to LR gradient-based method, we decided to use advantage function in the place of  $L$ , and particularly used GAE [Schulman et al., 2015b] as the advantage function as we used for PPO. After calculating this LR gradient, we take a gradient ascent step with pre-defined learning rate.
RP By using RP gradient in Appendix 7.2.2, we can perform gradient ascent as we did in LR. Also, because we use short time horizon and critic network, this method is very similar to SHAC [Xu et al., 2022]. Please refer to the paper for more details.
PPO Our PPO implementation is based on that of RL Games [Makoviichuk and Makoviychuk, 2022]. However, to be as fair as possible, we used the same critic network as the other methods, instead of using that implemented in the RL library. Also, we omitted several other additional losses that are not vital to PPO's formulation, such as entropy loss.
$\mathbf{LR} + \mathbf{RP}$  After we compute  $\mathrm{LR}(\nabla_{LR})$  and  $\mathrm{RP}(\nabla_{RP})$  gradient as shown above, we can interpolate them using their sample variance as follows [Parmas et al., 2018].
$$
\nabla_ {L R + R P} = \nabla_ {L R} \cdot \kappa_ {L R} + \nabla_ {R P} \cdot (1 - \kappa_ {L R}),
$$
$$
\kappa_ {L R} = \frac {\sigma_ {R P} ^ {2}}{\sigma_ {R P} ^ {2} + \sigma_ {L R} ^ {2}},
$$
where  $\sigma_{RP}$  and  $\sigma_{LR}$  are sample standard deviation of RP and LR gradients. We gain these terms by computing trace of covariance matrix of the sample gradients from different trajectories.
Since we have to compute this sample statistics, we have to do multiple different backpropagations for different trajectories, which incur a lot of computation time. Also, we found out that computing covariance matrix is also time consuming when controller has a large number of parameters. Therefore, we decided to use only limited number of sample gradients (16) to compute sample variance, and also truncate the gradient to smaller length (512) to facilitate computation.
PE We tried to faithfully re-implement policy enhancement scheme of [Qiao et al., 2021].
7.4.2 Network architecture and Hyperparameters
In this section, we provide network architectures and hyperparameters that we used for experiments in Section 5. For each of the experiments, we used the same network architectures, the same length of time horizons before policy update, and the same optimization procedure for critic updates, etc. We present these common settings first for each of the problems.
For GI-PPO, there are hyperparameters for update towards  $\alpha$ -policy, and those for PPO update. We denote the hyperparameters for  $\alpha$ -policy update by appending  $(\alpha)$ , and those for PPO update by appending (PPO). For the definition of hyperparameters for  $\alpha$ -policy update, please see Algorithm 1 for details.
Function Optimization Problems (Section 5.1) For these problems, common settings are as follows.
- Actor Network: MLP with [32, 32] layers and ELU activation function
- Critic Network: MLP with [32, 32] layers and ELU activation function
- Critic Hyperparameters: Learning rate  $= 10^{-3}$ , Iterations  $= 16$ , Batch Size  $= 4$
Number of parallel environments: 64
Horizon Length: 1
-  $\gamma$  (Discount factor): 0.99
$\tau$  (GAE): 0.95
Hyperparameters for LR are as follows.
- Learning Rate: Dejong(1), Dejong(64) = 10 $^{-3}$ , Ackley(1) = 10 $^{-4}$ , Ackley(64) = 3  $\cdot$  10 $^{-4}$
- Learning Rate Scheduler: Linear
Hyperparameters for RP are as follows.
- Learning Rate: Dejong(1), Dejong(64) = 10 $^{-2}$ , Ackley(1), Ackley(64) = 10 $^{-3}$
- Learning Rate Scheduler: Linear
Hyperparameters for  $\mathbf{LR} + \mathbf{RP}$  are as follows.
Hyperparameters for PPO are as follows.
- Learning Rate: Dejong(1), Ackley(1) = 10 $^{-4}$ , Dejong(64), Ackley(64) = 10 $^{-2}$
- Learning Rate Scheduler: Constant
- Batch Size for Actor Update: 64
Number of Epochs for Actor Update: 5
$\epsilon_{clip}$  : 0.2
Hyperparameters for GI-PPO are as follows.
$(\alpha)$  Learning Rate:  $10^{-3}$
$\bullet (\alpha)$  Batch Size for Actor Update: 64
$\bullet$ $(\alpha)$  Number of Epochs for Actor Update: 16
$(\alpha)\alpha_{0}\colon 10^{-5}$
$(\alpha)\max (\alpha)\colon 1.0$
$\bullet (\alpha)\beta$  :1.1
$\cdot (\alpha)\delta_{det}$  ..0.4
$\cdot (\alpha)\delta_{oorr}$  : 0.5
(PPO) Learning Rate: Dejong(1), Ackley(1) = 10 $^{-4}$ , Dejong(64), Ackley(64) = 10 $^{-2}$
(PPO) Learning Rate Scheduler: Constant
(PPO) Batch Size for Actor Update: 64
(PPO) Number of Epochs for Actor Update: 5
(PPO)  $\epsilon_{clip}$ : 0.2
Learning rate decreases linearly to the minimum value as learning progresses.
Differentiable Physics Problems (Section 5.2) For these problems, common settings are as follows.
- Actor Network: MLP with [64, 64] (Cartpole), and [128, 64, 32] (Ant, Hopper) layers and ELU activation function
- Critic Network: MLP with [64, 64] layers and ELU activation function
- Critic Hyperparameters: Learning rate  $= 10^{-3}$  (Cartpole),  $2 \cdot 10^{-3}$  (Ant), and  $2 \cdot 10^{-4}$  (Hopper), Iterations  $= 16$ , Batch Size  $= 4$
- Number of parallel environments: 64(Cartpole, Ant), 256(Hopper)
Horizon Length: 32
-  $\gamma$  (Discount factor): 0.99
$\tau$  (GAE): 0.95
- Learning Rate:  $10^{-4}$
- Learning Rate Scheduler: Linear
Hyperparameters for  $\mathbf{RP}$  are as follows.
- Learning Rate: Cartpole  $= {10}^{-2}$  ,Ant,Hopper  $= 2 \cdot  {10}^{-3}$
- Learning Rate Scheduler: Linear
- Learning Rate: Cartpole  $= 3 \cdot  {10}^{-4}$  ,Ant,Hopper  $= {10}^{-4}$
- Learning Rate Scheduler: Cartpole = Adaptive², Ant, Hopper = Constant
- Batch Size for Actor Update: 2048
Number of Epochs for Actor Update: 5
$\epsilon_{clip}$  : 0.2