SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\pi_ {h} (s, a) = \frac {1}{2} (\pi_ {\bar {\theta}} (s, a) + \pi_ {\alpha} (s, a)).
$$
By using this virtual policy  $\pi_h$ , we can restrict the PPO update to be done near both  $\pi_{\bar{\theta}}$  and  $\pi_{\alpha}$ . See Appendix 7.3.2 for more details.
4.3.3 Pseudocode
In Algorithm 1, we present pseudocode that illustrates the outline of our algorithm, GI-PPO. There are five hyperparameters in the algorithm, which are  $\alpha_0, \beta, \delta_{\text{det}}, \delta_{\text{oorr}}$ , and  $\max(\alpha)$ . We present specific values for these hyperparameters for our experiments in Appendix 7.4.2.
Algorithm 1 GI-PPO
$\alpha \leftarrow \alpha_{0}$  , Initial value   
 $\beta \leftarrow$  Constant multiplier larger than 1 for  $\alpha$ $\delta_{det},\delta_{oor}\gets$  Constant thresholds   
 $B\gets$  Experience buffer   
while Training not ended do Clear B while Not collected enough experience do Collect experience  $\{s_t,\epsilon_t,a_t,r_t,s_{t + 1}\} \to B$  end Estimate advantage  $A$  for every  $(s_i,a_i)$  in  $B$  using Eq. 14 Estimate advantage gradient  $\frac{\partial A}{\partial a}$  for every  $(s_i,a_i)$  in  $B$  using Eq. 15 For current  $\alpha$  , approximate  $\alpha$  -policy by minimizing loss in Equation 10 // Variance For each  $\epsilon_{i}$  and its corresponding state-action pair  $(s_i,a_i)$  , estimate det  $(I + \alpha \cdot \nabla_a^2 A_{\pi_\theta}(s,a))$  by Lemma 4.4 and get its sample minimum (psi min) and maximum (psi max) // Bias Evaluate expected additional return in Equation 16 with our current policy to get  $R_{\alpha}$  // Out-of-range-ratio Evaluate out-of-range-ratio in Equation 12 with our current policy to get  $R_{oorr}$  if  $\psi_{min} <   1 - \delta_{det}$  or  $\psi_{max} > 1 + \delta_{det}$  or  $R_{\alpha} <   0$  or  $R_{oorr} > \delta_{oorr}$  then  $\mid \alpha = \alpha /\beta$  end else  $\mid \alpha = \alpha \times \beta$  end  $\alpha = c l i p(\alpha ,0,\max (\alpha))$  Do PPO update by maximizing the surrogate loss in Equation 18
Table 1: Average maximum reward (↑) for function optimization problems.
Problem	LR	RP	PPO	LR+RP	GI-PPO
Dejong (1)	-1.24*10-6	-1.42*10-8	-5.21*10-5	-6.36*10-8	-3.84*10-10
Dejong (64)	-0.0007	-9.28*10-7	-0.0011	-3.05*10-6	-1.04*10-6
Ackley (1)	-1.2772	-0.4821	-0.2489	-1.2255	-0.0005
Ackley (64)	-0.6378	-0.0089	-0.1376	-0.0326	-0.0036
(a) De Jong (1)
(b) De Jong (64)
(c) Ackley (1)
Figure 2: Optimization curves for Dejong's and Ackley's function of dimension 1 and 64.
(d) Ackley (64)
In this section, we present experimental results that show our method's efficacy for various optimization and complex control problems. To validate our approach, we tested various baseline methods on the environments that we use. The methods are as follows:
- LR: Policy gradient method based on LR gradient.
- RP: Policy gradient method based on RP gradient. For physics and traffic problems, we adopted a truncated time window of [Xu et al., 2022] to reduce variance.
- PPO: Proximal Policy Optimization [Schulman et al., 2017].
- LR+RP: Policy gradient method based on interpolation between LR and RP gradient using sample variance [Parmas et al., 2018].
- PE: Policy enhancement scheme of [Qiao et al., 2021], for physics environments only.
- GI-PPO: Our method based on Section 4.3.
Please refer Appendix 7.4 to learn about implementation details of these baseline methods, network architectures, and hyperparameters that we used for all of the experiments.
We have implemented our learning method using PyTorch 1.9 Paszke et al. [2019]. As for hardware, all experiments were run with an Intel® Xeon® W-2255 CPU @ 3.70GHz, one NVIDIA RTX A5000 graphics card, and 16 GB of memory. Experiment results are averaged across 5 random seeds.
5.1 Function Optimization Problems
We start with how different learning methods find the maximum point for a given analytical function. We ran these experiments because RL algorithms can be thought of as function optimizers through function sampling [Williams and Peng, 1989]. These problems are 1-step optimization problems, where agents guess an optimal point as an action and get the negative function value as the reward.
As we used for rendering  $\alpha$ -policies in Figure 1, we use De Jong's function and Ackley's function for comparison, as they are popular functions for testing numerical optimization algorithms [Molga and Smutnicki, 2005]. Please see Appendix 7.5.1 for the details about the function definitions. In this setting, the optimum occurs at the point  $x = 0$  for both of the problems. However, their loss landscapes are quite different from one another—De Jong's function has a very smooth landscape that only has one local (global) maximum, while Ackley's function has a very rugged landscape with multiple local maxima. Therefore, De Jong's function represents environments where RP gradients have lower variance, while Ackley's function represents the opposite.
Table 1 shows that GI-PPO finds far better optimums than the other methods for most of the functions, except 64-dimensional Dejong's function, where the RP method found slightly better
(a) Cartpole
(b) Ant
Figure 3: Learning graphs for 3 problems in differentiable physics simulation: Cartpole, Ant, and Hopper [Xu et al., 2022]. In these environments, RP gradients already exhibit lower variance without much bias than LR gradients, because of variance reduction scheme of [Xu et al., 2022].
optimum than GI-PPO. Optimization curves in Figure 2 also prove that our method converges to the optimum faster than the other methods. Interestingly, the RP method achieved better results than the LR and  $\mathrm{LR + RP}$  methods for all the functions, which contradicts our assumption that it would not work well for Ackley's function. However, we found out that even though the RP method takes some time to find the optimum than the other methods as shown in Figure 2, when it approaches near-optimum, it converges fast to the optimum based on analytic gradients. Since GI-PPO adaptively changes  $\alpha$  to favor RP gradient near optimum, it could achieve better results than the other methods. In Appendix 7.3.3, we provide visual renderings that trace the change of  $\alpha$  value during optimization to help understanding, which also shows that higher  $\alpha$  is maintained for Dejong's function than Ackley's function, which aligns with our initial assumption. Also note that even though  $\mathrm{LR + RP}$  also takes advantage of the RP method, and thus achieved a better convergence rate than the two methods when it is far from optimum, it did not work well when it comes to near-optimum.
5.2 Differentiable Physics Simulation
Next, we conducted experiments in differentiable physics simulations. We used Cartpole, Ant, and Hopper environments implemented by [Xu et al., 2022] for comparisons. However, since the current implementation does not support multiple backpropagations through the same computation graph for these environments, we could not test the LP+RP method for them. Instead, we tried to faithfully implement another policy enhancement (PE) strategy of [Qiao et al., 2021] as a comparison, as it showed its efficacy in differentiable physics simulations using analytical gradients.
In Figure 3, we can see that GI-PPO converged to a better policy much faster than the baseline PPO for every environment, which validates our approach. However, the RP method achieved the best results for Ant and Hopper, while GI-PPO converged to the optimal policy slightly faster for Cartpole. To explain this result, we'd like to point out that  $\alpha$  is upper bounded by out-of-range-ratio in our approach (Section 4.3.1). Therefore, even if the RP gradient is very effective, we cannot fully utilize it because we have to use PPO to update the policy up to some extent as a safeguard. In the Ant environment, we could observe that it was the major bottleneck in training—in fact, GI-PPO could achieve better results than RP when we increased the ratio from 0.5 to 1.0 (Appendix 7.3.4). However, when the out-of-range ratio equals 1.0, it becomes more likely that we cannot use PPO as a safeguard, which contradicts our design intention. When it comes to the Hopper environment, we found that the variance of proper  $\alpha$  values over time was large, so our strategy could not conform to it properly. Overcoming these limitations would be an interesting future work to extend our work.
5.3 Traffic Problems
Here we demonstrate the effectiveness of GI-PPO in an environment where continuous and discrete operations coexist, which leads to highly biased analytical gradients. We suggest a mixed-autonomy traffic environment as a suitable benchmark, which has previously been explored as a benchmark for gradient-free RL algorithms [Wu et al., 2017, Vinitsky et al., 2018]. In this paper, we use the pace car problem, where a single autonomous pace car has to control the speed of the other vehicles via interference. The number of lanes, which represent the discontinuities in gradients, and the number of following human vehicles are different for each problem. Please see Appendix 7.5.2 for the details of this environment, and why the analytical gradients are biased.
(a) Single Lane
(b) 2 Lanes
Figure 4: Learning graphs for traffic pace car problem. Each problem has different number of lanes and following human vehicles, which represent discontinuities and dimension of the problem, respectively.
(c) 4 Lanes
(d) 10 Lanes
In Figure 4, we can observe that GI-PPO exhibits a faster convergence rate and converges to better policy overall compared to all the other methods in 2, 4, and 10-lane environments. In a single-lane environment, however,  $\mathrm{LR + RP}$  showed a better convergence rate than GI-PPO. Note that the RP method still performs well for 2 and 4-lane environments just as other baseline methods, even if there were multiple lanes. We assume this is because the RP method also leverages a state value function, which can provide unbiased analytical gradients in the presence of discontinuities. However, it does not perform well in the 10-lane environment, where discontinuities are excessive compared to the other environments.
This result shows that GI-PPO can still leverage analytical gradients, even if they were highly biased for policy learning. Even though  $\mathrm{LP + RP}$ , which is based on a similar spirit as ours, also exhibited some improvements over LP and RP, our method showed better results for more difficult problems because baseline PPO offers more stable learning than baseline LR. Also, the computational cost of GI-PPO was a little more than RP, while that of LR+RP was much more expensive (Appendix 7.7).
We introduced a novel policy learning method that adopts analytical gradients into the PPO framework. Based on the observation that the RP gradient leads our current policy to its  $\alpha$ -policy, we suggested approximating it explicitly, which allows us to indirectly estimate variance and bias of analytical gradients. We suggested several criteria to detect such variance and bias and introduced an algorithm that manipulates  $\alpha$  adaptively, which stands for the strength of analytical gradients in policy updates. In our experiments of diverse RL environments, we successfully showed that our method achieves much better results than the baseline PPO by adopting analytical gradients. Even though the interpolated gradient of LR and RP gradient is based on similar motivations, our method performed much better, thanks to the baseline PPO's stable learning.
Limitations Despite GI-PPO's promising results, there are some remaining issues. First, even if we use a large  $\alpha$ , we may not fully approximate the corresponding  $\alpha$ -policy in a limited number of optimization steps, which may give a false signal for our strategy. We can upper bound the maximum  $\alpha$  as we did here, but a more rigorous optimization process may be helpful. This is also related to improving the strategy to control  $\alpha$  adaptively, which would be a promising future work.
Also, as discussed in Section 5.2, our method is tightly bound to PPO—that is, even when analytical gradients are much more useful as illustrated in differentiable physics problems, we cannot fully utilize them if they exit the PPO's bound. Even though it is more suitable for stable updates, it could result in a worse convergence rate. If we could adjust PPO's clipping range dynamically, or detect environments where RP gradients are much more reliable, we would be able to overcome this issue.
Finally, computational costs can be further reduced. To utilize analytical gradients, we need backpropagation, which usually requires more time than forward steps. This is the reason why the learning methods based on analytical gradients require longer training time than the others (Appendix 7.7). When the time window gets longer, the cost grows accordingly. However, as we have shown in our traffic experiments, our method works even when gradients are biased. Therefore, it would be an interesting research direction to see if using a very short time window for backpropagation, which produces more biased gradients, would be worthy of exploring for improved computational efficiency.
Acknowledgements. This research is supported in part by the ARO DURIP and IARPA HAYSTAC Grants, ARL Cooperative Agreement W911NF2120076, Dr. Barry Mersky and Capital One E-NNovate Endowed Professorships. Yi-Ling Qiao would like to thank the support of Meta Fellowship.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846, 1983.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J Zico Kolter. End-to-end differentiable physics for learning and control. Advances in neural information processing systems, 31, 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, page 6, 2019.
Marc Deisenroth and Carl E Rasmussen. *Pilco: A model-based and data-efficient approach to policy search*. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465–472, 2011.
Marc Peter Deisenroth, Gerhard Neumann, Jan Peters, et al. A survey on policy search for robotics. Foundations and Trends® in Robotics, 2(1-2):1-142, 2013.
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281, 2021.
C Fu, C Fu, and Michael Michael. Handbook of simulation optimization. Springer, 2015.
Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-84, 1990.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. CiteSeer, 2002.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for reinforcement learning. https://github.com/Denys88/rl Games, May 2022.
Luke Metz, C Daniel Freeman, Samuel S Schoenholz, and Tal Kachman. Gradients are not all you need. arXiv preprint arXiv:2111.05803, 2021.
Marcin Molga and Czesław Smutnicki. Test functions for optimization needs. Test functions for optimization needs, 101:48, 2005.
Miguel Angel Zamora Mora, Momchil Peychev, Sehoon Ha, Martin Vechev, and Stelian Coros. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning, pages 7805-7817. PMLR, 2021.
M Mozer. A focused backpropagation algorithm. Complex systems, 3:349-381, 1989.
7 APPENDIX
7.1 Proofs
7.1.1 Proof of Lemma 3.2
By Definition 3.1,  $\pi_{\theta}(s,a)$  should satisfy the following condition:
$$
\begin{array}{l} \int_ {T _ {\epsilon}} q (\epsilon) d \epsilon = \int_ {T _ {a}} \pi_ {\theta} (s, a) d a \\ = \int_ {T _ {\epsilon}} \pi_ {\theta} (s, g _ {\theta} (s, \epsilon)) \cdot \left| \det  \left(\frac {\partial g _ {\theta} (s , \epsilon)}{\partial \epsilon}\right) \right| d \epsilon \quad (\text {b y}) \\ \end{array}
$$
We denote the set of continuous distributions that satisfy this condition as  $P$ . That is,
$$
P _ {\theta} (s, \epsilon) \in P \text {i f} \int_ {T _ {\epsilon}} q (\epsilon) d \epsilon = \int_ {T _ {\epsilon}} P _ {\theta} (s, \epsilon) \cdot \left| \det  \left(\frac {\partial g _ {\theta} (s , \epsilon)}{\partial \epsilon}\right) \right| d \epsilon .
$$
Clearly,  $\bar{P}_{\theta}(s,\epsilon) = q(\epsilon)\cdot \left|\operatorname *{det}(\frac{\partial g_{\theta}(s,\epsilon)}{\partial\epsilon})\right|^{-1}$  is an element of  $P$  , and is well defined, because of Equation 3. In fact,  $\bar{P}_{\theta}(s,\epsilon)$  is the only element of  $P$  , as the above condition should be met for any  $T_{\epsilon}\subset \mathbb{R}^{n}$  . That is, if there was another distribution  $\tilde{P}_{\theta}(s,\epsilon)\in P$
$$
\int_ {T _ {\epsilon}} (\bar {P} _ {\theta} (s, \epsilon) - \tilde {P} _ {\theta} (s, \epsilon)) \cdot \left| \det  \left(\frac {\partial g _ {\theta} (s , \epsilon)}{\partial \epsilon}\right) \right| d \epsilon = 0, \forall T _ {\epsilon} \subset \mathbb {R} ^ {n}.
$$
Since  $\tilde{P}_{\theta}$  is different from  $\bar{P}_{\theta}$ , there exists an open set  $\hat{T}_{\epsilon}$  where  $\tilde{P}_{\theta} < \bar{P}_{\theta}$ . Then
$$
\int_ {\hat {T} _ {\epsilon}} (\bar {P} _ {\theta} (s, \epsilon) - \tilde {P} _ {\theta} (s, \epsilon)) \cdot \left| \det (\frac {\partial g _ {\theta} (s , \epsilon)}{\partial \epsilon}) \right| d \epsilon > 0,
$$
which is a contradiction. Therefore,  $\pi_{\theta}(s,a) = \pi_{\theta}(s,g_{\theta}(s,\epsilon)) = \bar{P}_{\theta}(s,\epsilon)$ . The inverse holds trivially.
7.1.2 Proof of Lemma 4.2
First, we show that  $f$  is an injective function by showing that the determinant of Jacobian  $\frac{\partial f}{\partial a}$  is always positive:
$$
\begin{array}{l} \det  \left(\frac {\partial f}{\partial a}\right) = \det  (I + \alpha \cdot \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, a)) \\ = \Pi_ {i = 1} ^ {n} (1 + \alpha \cdot \lambda_ {i} (s, a)) > 0 \quad (\because | \alpha | <   \frac {1}{\max  _ {(s , a)} | \lambda_ {1} (s , a) |}), \\ \end{array}
$$
where  $\lambda_{i}(s,a)$  are the eigenvalues of  $\nabla_a^2 A_{\pi_\theta}(s,a)$  sorted in ascending order.
Then, for an arbitrary open set of action  $T \subset A$ ,  $\pi_{\alpha}(s,\cdot)$  selects  $\tilde{T} = f(T) \subset A$  with the same probability that the original policy  $\pi_{\tilde{\theta}}(s,\cdot)$  selects  $T$ .
$$
\begin{array}{l} \int_ {\tilde {T}} \pi_ {\alpha} (s, \tilde {a}) d \tilde {a} = \int_ {T} \frac {\pi_ {\bar {\theta}} (s , a)}{| \det  (I + \alpha \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s , a)) |} \cdot \left| \det  (\frac {\partial f}{\partial a}) \right| d a \\ = \int_ {T} \frac {\pi_ {\bar {\theta}} (s , a)}{\left| \det  \left(I + \alpha \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s , a)\right) \right|} \cdot \left| \det  \left(I + \alpha \cdot \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, a)\right) \right| d a \\ = \int_ {T} \pi_ {\bar {\theta}} (s, a) d a. \\ \end{array}
$$
Therefore,  $\pi_{\alpha}(s,\cdot)$  is a valid probability distribution as  $\pi_{\bar{\theta}}(s,\cdot)$ .
7.1.3 Proof of Proposition 4.3
For a given state  $s$ , we can estimate the (approximate) expected value of the state under  $\pi_{\alpha}$  as follows.
$$
\begin{array}{l} \int_ {\tilde {a}} \pi_ {\alpha} (s, \tilde {a}) A _ {\pi_ {\tilde {\theta}}} (s, \tilde {a}) d \tilde {a} \\ = \int_ {a} \frac {\pi_ {\bar {\theta}} (s , a)}{\left| \det  \left(I + \alpha \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s , a)\right) \right|} A _ {\pi_ {\bar {\theta}}} (s, a + \alpha \nabla_ {a} A _ {\pi_ {\bar {\theta}}} (s, a)) \left| \det  \left(I + \alpha \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, a)\right) \right| d a \\ \approx \int_ {a} \pi_ {\bar {\theta}} (s, a) \left[ A _ {\pi_ {\bar {\theta}}} (s, a) + \alpha | | \nabla_ {a} A _ {\pi_ {\bar {\theta}}} (s, a) | | ^ {2} \right] d a \\ = \int_ {a} \pi_ {\bar {\theta}} (s, a) A _ {\pi_ {\bar {\theta}}} (s, a) d a + \alpha \int_ {a} \pi_ {\bar {\theta}} (s, a) | | \nabla_ {a} A _ {\pi_ {\bar {\theta}}} (s, a) | | ^ {2} d a. \\ \end{array}
$$
Therefore, we can see the following holds:
$$
\begin{array}{l} L _ {\pi_ {\tilde {\theta}}} (\pi_ {\alpha}) = \int_ {s} \rho_ {\pi_ {\tilde {\theta}}} (s) \int_ {\tilde {a}} \pi_ {\alpha} (s, \tilde {a}) A _ {\pi_ {\tilde {\theta}}} (s, \tilde {a}) d \tilde {a} \\ \approx \eta (\pi_ {\bar {\theta}}) + \alpha \int_ {s} \rho_ {\pi_ {\bar {\theta}}} (s) \int_ {a} \pi_ {\bar {\theta}} (s, a) | | \nabla_ {a} A _ {\pi_ {\bar {\theta}}} (s, a) | | ^ {2} d a. \\ \end{array}
$$
Since  $\pi_{\bar{\theta}}(s,a)$  and  $||\nabla_{a}A_{\pi_{\bar{\theta}}}(\boldsymbol {s},a)||^{2}$  are positive,  $L_{\pi_{\bar{\theta}}}(\pi_{\alpha})$  is greater or equal to  $\eta (\pi_{\bar{\theta}})$  when  $\alpha >0$  On the contrary, when  $\alpha < 0$ $L_{\pi_{\bar{\theta}}}(\pi_{\alpha})$  is smaller or equal to  $\eta (\pi_{\bar{\theta}})$  . Since this argument builds upon local approximation, it holds only when  $|\alpha |\ll 1$
7.1.4 Proof of Lemma 4.4
When  $a = g_{\bar{\theta}}(s,\epsilon)$
$$
\begin{array}{l} g _ {\alpha} (s, \epsilon) = g _ {\bar {\theta}} (s, \epsilon) + \alpha \cdot \nabla_ {g _ {\bar {\theta}} (s, \epsilon)} A _ {\pi_ {\bar {\theta}}} (s, g _ {\bar {\theta}} (s, \epsilon)) \\ \Rightarrow \frac {\partial g _ {\alpha} (s , \epsilon)}{\partial \epsilon} = \frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon} + \alpha \cdot \nabla_ {g _ {\bar {\theta}} (s, \epsilon)} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, g _ {\bar {\theta}} (s, \epsilon)) \cdot \frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon} \\ = \left(I + \alpha \cdot \nabla_ {g _ {\bar {\theta}} (s, \epsilon)} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, g _ {\bar {\theta}} (s, \epsilon))\right) \cdot \frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon} \\ \Rightarrow \det  \left(\frac {\partial g _ {\alpha} (s , \epsilon)}{\partial \epsilon}\right) = \det  \left(I + \alpha \cdot \nabla_ {g _ {\bar {\theta}} (s, \epsilon)} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, g _ {\bar {\theta}} (s, \epsilon))\right) \cdot \det  \left(\frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon}\right) \\ = \det  \left(I + \alpha \cdot \nabla_ {a} ^ {2} A _ {\pi_ {\bar {\theta}}} (s, a)\right) \cdot \det  \left(\frac {\partial g _ {\bar {\theta}} (s , \epsilon)}{\partial \epsilon}\right). \\ \end{array}
$$
Since  $\left|\operatorname{det}\left(\frac{\partial g_{\bar{\theta}}(s,\epsilon)}{\partial\epsilon}\right)\right| > 0$  (Equation 3), we can divide both sides of the above equation with  $\operatorname*{det}\left(\frac{\partial g_{\bar{\theta}}(s,\epsilon)}{\partial\epsilon}\right)$  and prove Lemma 4.4.
7.1.5 Proof of Lemma 4.6
From Equation 11,
$$
\nabla_ {a} \hat {A} _ {\pi_ {\theta}} (s _ {t}, a _ {t}) = \frac {1}{2} \mathbb {E} _ {s _ {t}, a _ {t}, \dots \sim \pi_ {\theta}} \Big [ \sum_ {k = t} ^ {\infty} \gamma^ {k} \frac {\partial r (s _ {k} , a _ {k})}{\partial a _ {t}} \Big ],
$$
which can be rewritten using  $g_{\theta}$  as follows.