SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 2021. (Cited on pages 4, 5, 7, 8, 20, 22, 24, 28, 37, 38, 39, and 41.)
[2] Carlo Alfano and Patrick Rebeschini. Linear convergence for natural policy gradient with log-linear policy parametrization. arXiv preprint arXiv:2209.15382, 2022. (Cited on pages 2, 8, 20, 21, 22, 25, 28, 38, and 39.)
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in Neural Information Processing Systems, 2019. (Cited on pages 4, 9, 40, and 41.)
[4] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, 2019. (Cited on pages 9 and 43.)
[5] Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Computation, 1998. (Cited on page 21.)
[6] Jonathan Baxter and Peter L. Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 2001. (Cited on page 1.)
[7] Amir Beck. First-Order Methods in Optimization. SIAM-Society for Industrial and Applied Mathematics, 2017. (Cited on pages 4 and 23.)
[8] Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 2003. (Cited on pages 21 and 26.)
[9] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. (Cited on page 1.)
[10] Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786, 2019. (Cited on page 23.)
[11] Jalaj Bhandari and Daniel Russo. On the linear convergence of policy gradient methods for finite MDPs. In International Conference on Artificial Intelligence and Statistics, 2021. (Cited on pages 22, 23, and 25.)
[12] Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 2009. (Cited on pages 1 and 22.)
[13] Lev M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 1967. (Cited on page 3.)
[14] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 2015. (Cited on pages 3, 25, 31, and 32.)
[15] Semih Cayci, Niao He, and R Srikant. Linear convergence of entropy-regularized natural policy gradient with linear function approximation. arXiv preprint arXiv:2106.04096, 2021. (Cited on pages 2, 7, 8, 22, 25, 38, and 39.)
[16] Semih Cayci, Niao He, and R Srikant. Finite-time analysis of entropy-regularized neural natural actor-critic algorithm. arXiv preprint arXiv:2206.00833, 2022. (Cited on pages 8, 9, 20, 22, 24, 39, 43, and 44.)
[101] Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. In Advances in Neural Information Processing Systems, 2020. (Cited on pages 7 and 23.)
[102] Junyu Zhang, Chengzhuo Ni, Zheng Yu, Csaba Szepesvari, and Mengdi Wang. On the convergence and sample efficiency of variance-reduced policy gradient method. In Advances in Neural Information Processing Systems, 2021. (Cited on page 23.)
A Related work 18
B Equivalence of (9)-(10) and (11) in the tabular case 25
C AMPO for specific mirror maps 26
D Deferred proofs from Section 4.1 31
E Discussion of the first step (Line 1) of AMPO - the compatible function approximation framework 37
F Discussion of the second step (Line 2) of AMPO - the Bregman projection 37
G Discussion on Assumption (A1) - the approximation error 38
H Discussion on Assumption (A2) - the concentrability coefficients 39
I Discussion on Assumption (A3) - the distribution mismatch coefficients 40
J Sample complexity for neural network parameterization 40
Here we provide the related work discussion, the deferred proofs from the main paper and some additional noteworthy observations.
We provide an extended discussion for the context of our work, including a comparison of different PMD frameworks and a comparison of the convergence theories of PMD in the literature. Furthermore, we discuss future work, such as extending our analysis to the dual averaging updates and developing sample complexity analysis of AMPO.
A.1 Comparisons with other policy mirror descent frameworks
In this section, we give a comparison of AMPO with some of the most popular policy optimization algorithms in the literature. First, recall AMPO's update through (12), that is, for all  $s \in S$
$$
\pi_ {s} ^ {t + 1} \in \underset {\pi_ {s} \in \Delta (\mathcal {A})} {\operatorname {a r g m a x}} \left\langle \eta_ {t} f _ {s} ^ {t + 1} - \nabla h \left(\pi_ {s} ^ {t}\right), \pi_ {s} \right\rangle - \mathcal {D} _ {h} \left(\pi_ {s}, \pi_ {s} ^ {t}\right), \tag {20}
$$
where  $\eta_t f_s^{t+1} - \nabla h(\pi_s^t) \approx \eta_t Q_s^t$  following Line 1 of Algorithm 1. The proof of (20) can be found in Lemma F.1 in Appendix F.
Generalized Policy Iteration (GPI) [86]. The update consists of evaluating the Q-function of the policy and obtaining the new policy by acting greedily with respect to the estimated Q-function. That is, for all  $s \in S$ ,
$$
\pi_ {s} ^ {t + 1} \in \underset {\pi_ {s} \in \Delta (\mathcal {A})} {\operatorname {a r g m a x}} \left\langle Q _ {s} ^ {t}, \pi_ {s} \right\rangle . \tag {21}
$$
AMPO behaves like GPI when we perfectly approximate  $f_{s}^{t + 1}$  to the value of  $Q_{s}^{t}$  (e.g. when we consider the tabular case) and  $\eta_t \to +\infty$  (or  $\eta_t^{-1} \to 0$ ) which is the case with the use of geometrically increasing step-size schedule.
Mirror Descent Modified Policy Iteration (MD-MPI) [33]. Consider the full policy space  $\Pi = \Delta (\mathcal{A})^S$ . The MD-MPI's update is as follows:
$$
\pi_ {s} ^ {t + 1} \in \underset {\pi_ {s} \in \Delta (\mathcal {A})} {\operatorname {a r g m a x}} \left\langle Q _ {s} ^ {t}, \pi_ {s} \right\rangle - \mathcal {D} _ {h} \left(\pi_ {s}, \pi_ {s} ^ {t}\right), \quad \forall s \in \mathcal {S}. \tag {22}
$$
In this case, the PMD framework of Xiao [93], which is a special case of AMPO, recovers MD-MPI with the fixed step-size  $\eta_t = 1$ . Consequently, Assumption (A1) holds with  $\varepsilon_{\mathrm{approx}} = 0$ , and we obtain the sublinear convergence of MD-MPI through Theorem 4.3, which is
$$
V ^ {\star} (\mu) - \frac {1}{T} \sum_ {t <   T} \mathbb {E} \left[ V ^ {t} (\mu) \right] \leq \frac {1}{T} \left(\frac {\mathcal {D} _ {0} ^ {\star}}{(1 - \gamma)} + \frac {\nu_ {\mu}}{1 - \gamma}\right).
$$
As explained later in Appendix H that the distribution mismatch coefficient  $\nu_{\mu}$  is upper bounded by  $\mathcal{O}(\frac{1}{1 - \gamma})$ , we obtain an average regret of MD-MPI as  $\mathcal{O}\left(\frac{1}{(1 - \gamma)^2T}\right)$ , which matches the convergence results in Geist et al. [33, Corollary 3].
Trust Region Policy Optimization (TRPO) [78]. The TRPO's update is as follows:
$$
\pi^ {t + 1} \in \underset {\pi \in \Pi} {\operatorname {a r g m a x}} \mathbb {E} _ {s \sim d _ {\mu} ^ {t}} \left[ \left\langle A _ {s} ^ {t}, \pi_ {s} \right\rangle \right], \tag {23}
$$
$$
\text {s u c h} \mathbb {E} _ {s \sim d _ {\mu} ^ {t}} \left[ D _ {h} \left(\pi_ {s} ^ {t}, \pi_ {s}\right) \right] \leq \delta ,
$$
where  $A_{s}^{t} = Q_{s}^{t} - V^{t}$  represents the advantage function,  $h$  is the negative entropy and  $\delta > 0$ . Like GPI, TRPO is equivalent to AMPO when at each time  $t$ , the admissible policy class is  $\Pi^t = \{\pi \in \Delta(\mathcal{A})^S : \mathbb{E}_{s \sim d_\mu^t} D_h(\pi_s^t, \pi_s) \leq \delta\}$ , and we perfectly approximate  $Q_{s}^{t}$  with  $\eta_t \to +\infty$ .
Proximal Policy Optimization (PPO) [80]. The PPO's update consists of maximizing a surrogate function depending on the policy gradient with respect to the new policy. Namely,
$$
\pi^ {t + 1} \in \underset {\pi \in \Pi} {\operatorname {a r g m a x}} \mathbb {E} _ {s \sim d _ {\mu} ^ {t}} \left[ L \left(\pi_ {s}, \pi_ {s} ^ {t}\right) \right], \tag {24}
$$
$$
L \left(\pi_ {s}, \pi_ {s} ^ {t}\right) = \mathbb {E} _ {a \sim \pi^ {t}} \left[ \min  \left(r ^ {\pi} (s, a) A ^ {t} (s, a), \operatorname {c l i p} \left(r ^ {\pi} (s, a), 1 \pm \epsilon\right) A ^ {t} (s, a)\right) \right],
$$
where  $r^{\pi}(s,a) = \pi (s,a) / \pi^{t}(s,a)$  is the probability ratio between the current policy  $\pi^t$  and the new one, and the function  $\mathrm{clip}(r^{\pi}(s,a),1\pm \epsilon)$  clips the probability ratio  $r^{\pi}(s,a)$  to be no more than  $1 + \epsilon$  and no less than  $1 - \epsilon$ . PPO has also a KL variation [80, Section 4], where the objective function  $L$  is defined as
$$
L (\pi_ {s}, \pi_ {s} ^ {t}) = \eta_ {t} \langle A _ {s} ^ {t}, \pi_ {s} \rangle - D _ {h} (\pi_ {s} ^ {t}, \pi_ {s}),
$$
where  $h$  is the negative entropy. In an exact setting and when  $\Pi = \Delta (\mathcal{A})^S$ , the KL variation of PPO still differs from AMPO because it inverts the terms in the Bregman divergence penalty.
Mirror Descent Policy Optimization (MDPO.) [88]. The MDPO's update is as follows:
$$
\pi^ {t + 1} \in \underset {\pi \in \Pi} {\operatorname {a r g m a x}} \mathbb {E} _ {s \sim d _ {\mu} ^ {t}} [ \left\langle \eta_ {t} A _ {s} ^ {t}, \pi_ {s} \right\rangle - D _ {h} \left(\pi_ {s}, \pi_ {s} ^ {t}\right) ], \tag {25}
$$
where  $\Pi$  is a parameterized policy class. While it is equivalent to AMPO in an exact setting and when  $\Pi = \Delta (\mathcal{A})^S$ , as we show in Appendix B, the difference between the two algorithms lies in the approximation of the exact algorithm.
Functional Mirror Ascent Policy Gradient (FMA-PG) [89]. The FMA-PG's update is as follows:
$$
\begin{array}{l} \pi^ {t + 1} \in \underset {\pi^ {\theta}: \theta \in \Theta} {\operatorname {a r g m a x}} \mathbb {E} _ {s \sim d _ {\mu} ^ {t}} [ V ^ {t} (\mu) + \left. \left\langle \eta_ {t} \nabla_ {\pi_ {s}} V ^ {t} (\mu) \right| _ {\pi = \pi^ {t}}, \pi_ {s} ^ {\theta} - \pi_ {s} ^ {t} \right\rangle - D _ {h} \left(\pi_ {s} ^ {\theta}, \pi_ {s} ^ {t}\right) ] \tag {26} \\ \in \operatorname * {a r g m a x} _ {\pi^ {\theta}: \theta \in \Theta} \mathbb {E} _ {s \sim d _ {\mu} ^ {t}} [ \langle \eta_ {t} Q _ {s} ^ {t}, \pi_ {s} ^ {\theta} \rangle - D _ {h} (\pi_ {s} ^ {\theta}, \pi_ {s} ^ {t}) ], \\ \end{array}
$$
The second line is obtained by the definition of  $V^t$  and the policy gradient theorem (3). The discussion is the same as the previous algorithm.
Mirror Learning [51]. The on-policy version of the algorithm consists of the following update:
$$
\pi^ {t + 1} = \underset {\pi \in \Pi (\pi^ {t})} {\operatorname {a r g m a x}} \mathbb {E} _ {s \sim d _ {\mu} ^ {t}} [ \langle Q _ {s} ^ {t}, \pi_ {s} \rangle - D (\pi_ {s}, \pi_ {s} ^ {t}) ], \tag {27}
$$
where  $\Pi (\pi^t)$  is a policy class that depends on the current policy  $\pi^t$  and the drift functional  $D$  is defined as a map  $D:\Delta (\mathcal{A})\times \Delta (\mathcal{A})\to \mathbb{R}$  such that  $D(\pi_s,\bar{\pi}_s)\geq 0$  and  $\nabla_{\pi_s}D(\pi_s,\bar{\pi}_s)\big|_{\pi_s = \bar{\pi}_s} = 0$ . The drift functional  $D$  recovers the Bregman divergence as a particular case, in which case Mirror Learning is equivalent to AMPO in an exact setting and when  $\Pi = \Delta (\mathcal{A})^S$ . Again, the main difference between the two algorithms lies in the approximation of the exact algorithm.
A.2 Discussion on related work
Our Contributions. Our work provides a framework for policy optimization – AMPO. For AMPO, we establish in Theorem 4.3 both  $\mathcal{O}(1 / T)$  convergence guarantee by using a non-decreasing step-size and linear convergence guarantee by using a geometrically increasing step-size. Our contributions to the prior literature on sublinear and linear convergence of policy optimization methods can be summarized as follows.
- The generality of our framework allows Theorem 4.3 to unify previous results in the literature and generate new theoretically sound algorithms under one guise. Both the sublinear and the linear convergence analysis of natural policy gradient (NPG) with softmax tabular policies [93] or with log-linear policies [2, 98] are special cases of our general analysis. As mentioned in Appendix A.1, MD-MPI [33] in the tabular setting is also a special case of AMPO. Thus, Theorem 4.3 recovers the best-known convergence rates in both the tabular setting [33, 93] and the non-tabular setting [16, 2, 98]. AMPO also generates new algorithms by selecting mirror maps, such as the  $\epsilon$ -negative entropy mirror map in Appendix C.2 associated with Algorithm 2, and generalizes the projected Q-descent algorithm [93] from the tabular setting to a general parameterization class  $\mathcal{F}^{\Theta}$ .
- As discussed in Section 4.1, the results of Theorem 4.3 hold for a general setting with fewer restrictions than in previous work. The generality of the assumptions of Theorem 4.3 allows the application of our theory to specific settings, where existing sample complexity analyses could be improved thanks to the linear convergence of AMPO. For instance, since Theorem 4.3 holds for any structural MDP, AMPO could be applied directly to the linear MDP setting to derive a sample complexity analysis of AMPO which could improve that of Zanette et al. [99] and Hu et al. [36]. As we discuss in Appendix A.3, this is a promising direction for future work.
- From a technical point of view, our main contributions are: Definition 3.1 introduces a novel way of incorporating general parameterization into the policy; the update in Line 1 of Algorithm 1 simplifies the policy optimization step into a regression problem; and Lemma 4.1 establishes a key result for policies belonging to the class in Definition 3.1. Together, these innovations have allowed us to establish new state-of-the-art results in Theorem 4.3 by leveraging the modern PMD proof techniques of Xiao [93].
In particular, our technical novelty with respect to Xiao [93], Alfano and Rebeschini [2], and Yuan et al. [98] can be summarized as follows.
- In terms of algorithm design, AMPO is an innovation. The PMD algorithm proposed by Xiao [93] is strictly limited to the tabular setting and, although it is well defined for any mirror map, it cannot include general parameterization. Alfano and Rebeschini [2] and Yuan et al. [98] propose a first generalization of the PMD algorithm in the function approximation regime thanks to the linear compatible function approximation framework [1], but are limited to considering the log-linear policy parameterization and the entropy mirror map. On the contrary, AMPO solves the problem of incorporating general parameterizations in the policy thanks to Definition 3.1 and the extension of the compatible function approximation framework from linear to nonlinear, which corresponds to the parameter update in Line 1 of Algorithm 1. This innovation is key to the generality of the algorithm, as it allows AMPO to employ any mirror map and any parameterization class. Moreover, AMPO is computationally efficient for a large class of mirror maps (see Appendix C.2 and Algorithms 2 and 3). Our design is readily applied to deep RL, where the policy is usually parameterized by a neural network whose
last layer is a softmax transformation. Our policy definition can be implemented in this setting by replacing the softmax layer with a Bregman projection, as shown in Example 3.7.
- Regarding the assumptions necessary for convergence guarantees, we have weaker assumptions. Xiao [93] requires an  $\ell_{\infty}$ -norm on the approximation error of  $Q^t$ , i.e.,  $\| \widehat{Q}^t - Q^t \|_{\infty} \leq \varepsilon_{\mathrm{approx}}$ , for all  $t \leq T$ . Alfano and Rebeschini [2] and Yuan et al. [98] require an  $L_2$ -norm bound on the error of the linear approximation of  $Q^t$ , i.e.,  $\| w^\top \phi - Q^t \|_{L_2(v^t)}^2 \leq \varepsilon_{\mathrm{approx}}$  for some feature mapping  $\phi: S \times \mathcal{A} \to \mathbb{R}^d$  and vector  $w \in \mathbb{R}^d$ , for all  $t \leq T$ . Our approximation error  $\varepsilon_{\mathrm{approx}}$  in Assumption (A1) is an improvement since it does not require the bound to hold in  $\ell_{\infty}$ -norm, and allows any regression model instead of linear function approximation, especially neural networks, which greatly increases the representation power of  $\mathcal{F}^\Theta$  and expands the range of applications. We further relax Assumption (A1) in Appendix G and show that the approximation error bound can be larger for earlier iterations. In addition, we improve the concentrability coefficients of Yuan et al. [98] by defining the expectation under an arbitrary state-action distribution  $v^t$  instead of the state-action visitation distribution with a fixed initial state-action pair (see Yuan et al. [98, Equation (4)]).
- As for the analysis of the algorithm, while we borrow tools from Xiao [93], Alfano and Rebeschini [2], and Yuan et al. [98], our results are not simple extensions. In fact, without our work, it is not clear from Xiao [93], Alfano and Rebeschini [2], and Yuan et al. [98] whether PMD could have theoretical guarantees in a setting with general parameterization and an arbitrary mirror map. The two main problems on this front are the non-convexity of the policy class, which prevents the use of the three-point descent lemma by Chen and Teboulle [19, Lemma 3.2] (or by Xiao [93, Lemma 6]), and the fact that the three-point identity used by Alfano and Rebeschini [2, Equation 4] holds only for the negative entropy mirror map. Our Lemma 4.1 successfully addresses general policy parameterization and arbitrary mirror maps thanks to the design of AMPO. Additionally, we provide a sample complexity analysis of AMPO when employing shallow neural networks that improves upon previous state-of-the-art results in this setting. We further improve this sample complexity analysis in Appendix J, where we consider an approximation error assumption that is weaker than Assumption (A1) (see Appendix G).
We also include a comparison with Lan [55]. Our diffences can be outlined in two points.
- Lan [55] propose a PMD algorithm (Algorithm 2 in their paper) that can accommodate general parameterization and arbitrary mirror maps. As AMPO, it involves a two-step procedure where the first step is to find an approximation of  $Q^t - \eta_t^{-1}\nabla h(\pi^t)$  and the second step is to find the policy through a Bregman projection. However, it is unclear how to implement their algorithm in practice, as they do not propose a specific method to perform either step. We provide an explicit implementation of AMPO and identify a class of mirror maps that is computationally efficient for AMPO (see Appendix C.2 and Algorithms 2 and 3).
- In terms of theoretical analysis, they assume for their results that the approximation error is bounded in  $\ell_{\infty}$ -norm over the action space. Let
$$
\begin{array}{l} \varepsilon_ {\mathrm {d e t}} = \mathbb {E} _ {s \sim v ^ {\star}} \left[ \left\| \mathbb {E} \left[ f _ {s} ^ {t + 1} \right] - Q _ {s} ^ {t} - \eta_ {t} ^ {- 1} \nabla h \left(\pi_ {s} ^ {t}\right) \right\| _ {\infty} \right], \\ \varepsilon_ {s t o} = \mathbb {E} _ {s \sim v ^ {\star}} \left[ \left\| \mathbb {E} [ f _ {s} ^ {t + 1} ] - f _ {s} ^ {t + 1} \right\| _ {\infty} ^ {2} \right], \\ \end{array}
$$
where the expectation  $\mathbb{E}[f_s^{t + 1}]$  is taken w.r.t. the stochasticity of the algorithm employed to obtain  $f^{t + 1}$ . Lan [55] assume that both  $\varepsilon_{\mathrm{det}}$  and  $\varepsilon_{sto}$  are bounded for all iterations  $t$ . In contrast, our assumptions are weaker as they are required to hold for the  $L_{2}(v)$ -norm we define in Section 3. Additionally, Lan [55] establishes a  $\mathcal{O}(1 / \sqrt{T})$  convergence rate for their algorithm without regularization and a  $\mathcal{O}(1 / T)$  convergence rate in the regularized case, in both cases using bounded step-sizes. We improve upon these results by obtaining a  $\mathcal{O}(1 / T)$  convergence rate without regularization and a linear convergence rate.
Related literature. Recently, the impressive empirical success of policy gradient (PG)-based methods has catalyzed the development of theoretically sound algorithms for policy optimization. In particular, there has been a lot of attention around algorithms inspired by mirror descent (MD) [70, 8] and, more specifically, by natural gradient descent [5]. These two approaches led to policy mirror descent (PMD) methods [82, 54] and natural policy gradient (NPG) methods [42], which, as first shown by
Neu et al. [72], is a special case of PMD. For instance, PMD and NPG are the building blocks of the state-of-the-art policy optimization algorithms, TRPO [78] and PPO [80]. Leveraging various techniques from the MD literature, it has been established that PMD, NPG, and their variants converge to the global optimum in different settings. We refer to global optimum convergence as an analysis that guarantees that  $V^{\star}(\mu) - \mathbb{E}\left[V^{T}(\mu)\right] \leq \epsilon$  after  $T$  iterations with  $\epsilon > 0$ . As an important variant of NPG, we will also discuss the literature of the convergence analysis of natural actor-critic (NAC) [75, 12]. The comparison of AMPO with different methods will proceed from the tabular case to different function approximation regimes.
Sublinear convergence analyses of PMD, NPG and NAC. For softmax tabular policies, Shani et al. [82] establish a  $\mathcal{O}(1 / \sqrt{T})$  convergence rate for unregularized NPG and  $\mathcal{O}(1 / T)$  for regularized NPG. Agarwal et al. [1], Khodadadian et al. [45] and Xiao [93] improve the convergence rate for unregularized NPG and NAC to  $\mathcal{O}(1 / T)$  and Xiao [93] extends the same convergence rate to projected Q-descent. The same convergence rate is established by MD-MPI [33] through the PMD framework.
In the function approximation regime, Zanette et al. [99] and Hu et al. [36] achieve  $\mathcal{O}(1 / \sqrt{T})$  convergence rate by developing variants of PMD methods for the linear MDP [40] setting. The same  $\mathcal{O}(1 / \sqrt{T})$  convergence rate is obtained by Agarwal et al. [1] for both log-linear and smooth policies, while Yuan et al. [98] improve the convergence rate to  $\mathcal{O}(1 / T)$  for log-linear policies. For smooth policies, the convergence rate is later improved to  $\mathcal{O}(1 / T)$  either by adding an extra Fisher-non-degeneracy condition on the policies [61] or by analyzing NAC under Markovian sampling [94]. Yang et al. [95] and Huang et al. [37] consider Lipschitz and smooth policies [97], obtain  $\mathcal{O}(1 / \sqrt{T})$  convergence rates for PMD-type methods and faster  $\mathcal{O}(1 / T)$  convergence rates by applying the variance reduction techniques SARAH [73] and STORM [22], respectively. As for neural policy parameterization, Liu et al. [60] establish a  $\mathcal{O}(1 / \sqrt{T})$  convergence rate for two-layer neural PPO. The same  $\mathcal{O}(1 / \sqrt{T})$  convergence rate is established by Wang et al. [90] for two-layer neural NAC, which is later improved to  $\mathcal{O}(1 / T)$  by Cayci et al. [16], using entropy regularization.