SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Main Softmax PG result. We formalize the above intuition by proving the following main result, which establishes that reward order preservation with adequate representations is a sufficient condition for the global convergence of Softmax PG under log-linear function approximation.
Theorem 1 (Reward order preservation, non-domination features). Given any reward  $r \in \mathbb{R}^K$  and feature matrix  $X \in \mathbb{R}^{K \times d}$ . Denote  $x_i \in \mathbb{R}^d$  as the  $i$ -th row vector of  $X$ . If  $(i) x_i^\top x_i > x_i^\top x_j$  for all  $j \neq i$ , and  $(ii)$  there exists at least one  $w \in \mathbb{R}^d$ , s.t.,  $r' := Xw$  preserves the order of  $r$ , i.e., for all  $i, j \in [K]$ ,  $r(i) > r(j)$  if and only if  $r'(i) > r'(j)$ , then for any initialization  $\theta_1 \in \mathbb{R}^d$ , Algorithm 1 with a constant learning rate  $\eta > 0$  achieves global convergence of  $\pi_{\theta_i}^\top r \to r(a^*)$  as  $t \to \infty$ .
A few remarks about this theorem are in order.
Examples 1 to 3 all satisfy the non-domination condition (i) on  $X$ , and their differences lie in satisfying reward order preservation or not. However, the following example shows that if the condition (i) on  $X$  is removed, then global convergence is not always achievable for even linearly realizable rewards (with zero approximation error).
Proposition 2. Let  $K = 3$ ,  $d = 2$ ,  $X^{\top} = \left[ \begin{array}{ccc}0 & -10 & 0\\ -2 & 4 & 1 \end{array} \right] \in \mathbb{R}^{d\times K}$ , and  $r = Xw = (4,2, - 2)^{\top}$
where  $w = (-1, -2)^{\top} \in \mathbb{R}^{d}$ . With initialization  $\theta_{1} = (-\ln 2, \ln 2)^{\top}$ , Algorithm 1 does not achieve global convergence, i.e.,  $\pi_{\theta_t}(1) \nrightarrow 1$  as  $t \to \infty$ .
Generalization of tabular and linear realizability. When  $d = K$  and  $X = \mathbf{Id}$ , i.e., the softmax tabular parameterization  $\pi_{\theta} = \mathrm{softmax}(\theta)$ , it is always true that  $Xr = r$  preserves the order of  $r$ . Consequently, Theorem 1 recovers the global convergence result for PG in the softmax tabular setting [4, 22] as a special case. More generally, for non-domination features, when the reward is linearly realizable, such that  $Xw = r$  for some  $w \in \mathbb{R}^d$ , the global convergence of Softmax PG also follows from Theorem 1, since  $r$  preserves its own order when the approximation error is zero.
Corollary 1 (Linearly realizable rewards, non-domination features). Given any reward  $r \in \mathbb{R}^K$  and feature matrix  $X \in \mathbb{R}^{K \times d}$ . Denote  $x_i \in \mathbb{R}^d$  as  $i$ -th row vector of  $X$ . If (i)  $x_i^\top x_i > x_i^\top x_j$  for all  $j \neq i$ , and (ii) there exists  $w \in \mathbb{R}^d$ , s.t.,  $Xw = r$ , then for any initialization  $\theta_1 \in \mathbb{R}^d$ , Algorithm 1 with a constant learning rate  $\eta > 0$  achieves global convergence of  $\pi_{\theta_t}^\top r \to r(a^*)$  as  $t \to \infty$ .
It is worth mentioning that Proposition 2 and Corollary 1 together answer a question which still remain unsolved in PG literature [4]: with linearly realizable rewards (zero approximation error), whether standard Softmax PG achieves global convergence? Proposition 2 shows that linearly realizable reward on its own is not enough to guarantee global convergence, while Corollary 1 shows that with adequate features, linearly realizable reward implies global convergence. Note that the NPG global convergence result in [4], such as Eq. (6), does not apply to standard Softmax PG.
Ordering does not determine approximation. As already illustrated in Section 3, approximation error is not adequate for capturing the global convergence of Softmax PG. It is important to emphasize that the existence of an order-preserving reward  $r'$  is very different from having a small approximation error. When the approximation error is zero, then an order-preserving reward (equal to  $r$ ) always exists. However, in general,  $r'$  can take very different values than  $r$ , and hence have a very large approximation error, yet still enable global convergence as shown in Examples 1 and 3.
Proof idea. The idea behind the proof of the main theorem consists of three parts. We provide a sketch of the proof here; the full proof is given in Appendix A. First, starting from any initialization  $\theta_t \in \mathbb{R}^d$ , Algorithm 1 guarantees that  $\pi_{\theta_t}$  will approach a (generalized) one-hot policy as  $t \to \infty$ . To see why, first note that  $\pi_\theta^\top r$  is  $\beta$ -smooth over  $\theta \in \mathbb{R}^d$  with some  $\beta > 0$  (Lemma 3 in Appendix B), since the softmax transform is smooth [4, 22] and the feature matrix  $X$  has bounded values. This implies that using a sufficiently small constant learning rate  $0 < \eta \leq 2 / \beta$  we obtain,
$$
\pi_ {\theta_ {t + 1}} ^ {\top} r - \pi_ {\theta_ {t}} ^ {\top} r \geq \frac {1}{2 \beta} \cdot \left\| \frac {d \pi_ {\theta_ {t}} ^ {\top} r}{d \theta_ {t}} \right\| _ {2} ^ {2} \geq 0. \tag {10}
$$
Note that  $\pi_{\theta}^{\top}r$  is upper bounded by  $r(a^{*})$ . According to the monotone convergence,  $\pi_{\theta_t}^\top r\to c\leq r(a^*)$  as  $t\to \infty$ . This fact combined with Eq. (10) implies  $\left\| \frac{d\pi_{\theta_t}^\top r}{d\theta_t}\right\|_2\to 0$  as  $t\to \infty$ . Next, a special co-variance structure of softmax PG (Lemma 4) shows that  $\left\| \frac{d\pi_{\theta_t}^\top r}{d\theta_t}\right\|_2\to 0$  implies that  $\| \theta_t\| _2\to \infty$  and  $\pi_{\theta_t}$  approaches a (generalized) one-hot policy as  $t\to \infty$ .
Lemma 1. Under the same conditions as Theorem 1, and  $r(i)\neq r(j)$  for all  $i\neq j$  (unique action reward), Algorithm 1 assures  $\| \theta_t\| _2\to \infty$  and  $\pi_{\theta_t}(i)\rightarrow 1$  for an action  $i\in [K]$  as  $t\to \infty$
Remark 1. Removing the unique action reward condition in Lemma 1 makes  $\pi_{\theta_t}$  approach a generalized one-hot policy (rather than a strict one-hot in Lemma 1) as  $t \to \infty$  as a result.
According to Lemma 1,  $\theta_t$  grows unboundedly. Intuitively, this can be seen in Figure 1(a), where there are no stationary points in a finite region.
Second, for any vector  $r'$  that preserves the order of  $r$ , we establish the following key lemma.
Lemma 2 (Non-negative covariance of order preservation). If  $r' \in \mathbb{R}^K$  preserves the order of  $r \in \mathbb{R}^K$ , i.e., for all  $i, j \in [K]$ ,  $r(i) > r(j)$  iff  $r'(i) > r'(j)$ , then for any policy  $\pi \in \Delta(K)$ ,
$$
r ^ {\prime} ^ {\top} \left(d i a g (\pi) - \pi \pi^ {\top}\right) r = \operatorname {C o v} _ {\pi} \left(r ^ {\prime}, r\right) \geq 0. \tag {11}
$$
Now consider the direction  $w \in \mathbb{R}^d$  such that  $r' := Xw$  preserves the order of  $r$ . We have,
$$
\begin{array}{l} w ^ {\top} \theta_ {t + 1} = w ^ {\top} \theta_ {t} + \eta \cdot w ^ {\top} X ^ {\top} \left(\operatorname {d i a g} \left(\pi_ {\theta_ {t}}\right) - \pi_ {\theta_ {t}} \pi_ {\theta_ {t}} ^ {\top}\right) r \quad (\text {b y A l g o r i t h m 1}) (12) \\ = w ^ {\top} \theta_ {t} + \eta \cdot r ^ {\prime} ^ {\top} \left(\operatorname {d i a g} \left(\pi_ {\theta_ {t}}\right) - \pi_ {\theta_ {t}} \pi_ {\theta_ {t}} ^ {\top}\right) r \quad \left(r ^ {\prime} := X w\right) (13) \\ \geq w ^ {\top} \theta_ {t}. \quad (\text {b y L e m m a 2}) (14) \\ \end{array}
$$
Third, take a sub-optimal action  $i \in [K]$  with  $r(i) < r(a^{*})$ , and we show that the assumption  $\pi_{\theta_t}(i) \to 1$  as  $t \to \infty$  leads to a contradiction.
To that end, first observe that this assumption implies that for all large enough time  $t \geq 1$ ,
$$
\left[ \frac {X \theta_ {t}}{\| \theta_ {t} \| _ {2}} \right] (i) = \max  _ {a \in [ K ]} \left[ \frac {X \theta_ {t}}{\| \theta_ {t} \| _ {2}} \right] (a), \tag {15}
$$
which means that the sub-optimal action  $i \in [K]$  always has the largest score (since its probability  $\pi_{\theta_t}(i) \to 1$  is always the largest). Moreover, differences between actions' scores are unbounded, due to  $\frac{\pi_{\theta_t}(i)}{\pi_{\theta_t}(j)} = \exp \left\{[X\theta_t](i) - [X\theta_t](j)\right\} \to \infty$  for all other actions  $j \neq i$ .
Consider Example 1 for illustration. The top view of Figure 1(a) is shown in Figure 2(a). Take  $i = 2$  and  $r(i) = 8$ , and assume  $\frac{\theta_t}{\|\theta_t\|_2}$  stays in the green (sub-optimal) region of Figure 2(a), excluding its boundaries. This green region is partitioned in Figure 2(b), where the dark sub-region contains  $v_{2} \in \mathbb{R}^{d}$  such that  $[Xv_{2}](a^{*})$  is the second largest component among all  $a \neq 2$ , and the light sub-region is the remaining.
(a) Top view of Example 1.
Figure 2: Idea illustration.
(b) Illustration for the third part.
The argument is completed by addressing the two cases: (i) If  $\frac{\theta_t}{\|\theta_t\|_2}$  stays in the dark sub
region where  $v_{1} \in \mathbb{R}^{d}$  belongs to, then  $\pi_{\theta_t}^\top r > r(i) = 8$  must occur in finite  $t < \infty$ , implying  $\pi_{\theta_t}(i) \nrightarrow 1$ , contradicting the assumption. Intuitively, the contradiction occurs because the dark sub-region is closer to a higher plateau with reward 9, and scaling up  $\theta_t$ 's magnitude in this sub-region eventually ensures  $\pi_{\theta_t}^\top r > r(i) = 8$ . (ii) If  $\frac{\theta_t}{\|\theta_t\|_2}$  stays in the light sub-region which contains  $v_{2} \in \mathbb{R}^{d}$ , then  $w^\top \theta_t > u^\top \theta_t$  must occur in finite time  $t < \infty$ , implying that  $\frac{\theta_t}{\|\theta_t\|_2}$  will enter the dark sub-region, reducing to the first case. This argument depends on Eq. (12) and a key observation showing that  $u^\top \theta_{t + 1} < u^\top \theta_t$ , where  $u$  is a "worse" direction such that  $[Xu](a^{-}) = \max_{a\in [K]}[Xu](a)$  for some  $a^{-} \in [K]$  with  $r(a^{-}) < r(i)$ .
To summarize,  $\frac{\theta_t}{\|\theta_t\|_2}$  cannot always stay in the green sub-optimal region in Figure 2(a), which implies that  $\frac{\theta_t}{\|\theta_t\|_2}$  must eventually enter the optimal region that contains  $w$  and stay in that region. By Lemma 1 we then obtain  $\pi_{\theta_t}(a^*) \to 1$  and  $\pi_{\theta_t}^\top r \to r(a^*)$  as  $t \to \infty$  (see appendix).
4.2 Optimal Action Preservation is Necessary and Sufficient for NPG Convergence
Next, we investigate the global convergence conditions for NPG under log-linear policies. Unlike Softmax PG, the key property for determining global convergence of NPG is whether the projection of the rewards  $r$  onto the feature representation  $X$  preserves the top ranking of the optimal action.
Intuition and demonstration. First consider Example 1 where NPG successfully converges to a global maximum. From Algorithm 2, a simple calculation shows,
$$
X \theta_ {t + 1} = X \theta_ {t} + \eta \cdot X (X ^ {\top} X) ^ {- 1} X ^ {\top} r = X \theta_ {t} + \eta \cdot \frac {1}{5} \cdot (2 2, - 4, - 1 1, 8) ^ {\top}, \tag {16}
$$
which implies that the optimal action  $a^* = 1$  always receives the largest update to its score  $[X\theta_t](a^*)$  in each iteration. Next, take a sub-optimal action  $a = 2$ , as an example, and observe that,
$$
\frac {\pi_ {\theta_ {t + 1}} \left(a ^ {*}\right)}{\pi_ {\theta_ {t + 1}} (a)} = \frac {\pi_ {\theta_ {t}} \left(a ^ {*}\right)}{\pi_ {\theta_ {t}} (a)} \cdot \exp \left\{\eta \cdot \left(\hat {r} \left(a ^ {*}\right) - \hat {r} (a)\right) \right\} = \frac {\pi_ {\theta_ {t}} \left(a ^ {*}\right)}{\pi_ {\theta_ {t}} (a)} \cdot \exp \left\{\eta \cdot \frac {2 6}{5} \right\} \tag {17}
$$
by Eq. (3), where  $\hat{r} \coloneqq X(X^{\top}X)^{-1}X^{\top}r$ . Using a constant learning rate  $\eta > 0$  and applying Eq. (17), we have that  $\pi_{\theta_t}(a^*)$  grows exponentially with  $t$ , indicating that  $\pi_{\theta_t}(a^*) \to 1$  as  $t \to \infty$  since the same argument works for any sup-optimal action  $a \neq a^*$ . Moreover, the rate is  $O(e^{-c \cdot t})$ , since  $(\pi^* - \pi_{\theta_t})^\top r \leq 2 \cdot \|r\|_\infty \cdot (1 - \pi_{\theta_t}(a^*))$ . The  $O(e^{-c \cdot t})$  rate matches the results in softmax tabular settings [14, 20, 17, 27].
Second, consider Example 2 where NPG fails to converge to a global maximum. Using similar calculations to Eq. (16) we obtain,
$$
X \theta_ {t + 1} = X \theta_ {t} + \eta \cdot \hat {r} = X \theta_ {t} + \eta \cdot \frac {1}{5} \cdot (- 3, 1 8, - 9, - 6) ^ {\top} \tag {18}
$$
which implies that a sub-optimal action  $a = 2$  always receives the largest update on its score  $[X\theta_t](2)$  in each iteration. The failure in Figure 1(b) is then verified by similar arguments around Eq. (17).
Main NPG result. Based on these observations, it is evident that for NPG to converge globally, it is important for the optimal action to eventually always receive the largest update to its score, which makes it critical that the least square projection  $X(\bar{X}^{\top}X)^{-1}X^{\top}r$  preserves the top ranking of the optimal action. We formalize this intuition by establishing the following main result.
Theorem 2 (Optimal action preservation condition). For a constant learning rate  $\eta >0$ , a necessary and sufficient condition for Algorithm 2 to achieve global convergence  $\pi_{\theta_t}^\top r\to r(a^*)$  as  $t\to \infty$  from any initialization  $\theta_1\in \mathbb{R}^d$  is that  $\hat{r} (a^{*}) > \hat{r} (a)$  for all  $a\neq a^{*}$ , such that  $a^* := \arg \max_{a\in [K]}r(a)$  and  $\hat{r} \coloneqq X\left(X^{\top}X\right)^{-1}X^{\top}r$  is the least squares projection of  $r$  onto the column space of  $X$ . If the condition is satisfied, then the rate of convergence is  $(\pi^{*} - \pi_{\theta_{t}})^{\top}r\in O(e^{-c\cdot t})$  for some  $c > 0$ .
Proof idea. When the optimal action preservation is satisfied, similar arguments to Eqs. (16) and (17) guarantee that  $\pi_{\theta_t}(a^*)$  grows exponentially with  $t$ , indicating that  $\pi_{\theta_t}(a^*) \to 1$  as  $t \to \infty$ .
The constant  $c > 0$  in Theorem 2 depends on the gap of  $\hat{r}$ , i.e.,  $\hat{r}(a^*) - \max_{a \neq a^*} \hat{r}(a)$ , which finds similarities to NPG results in tabular settings [14, 15]. The main difference is that the gap of true reward  $r$  in tabular cases is replaced with the gap of least square projection  $\hat{r}$  in function approximation settings in Theorem 2. This similarity is an evidence for improving the rate to super-linear by using geometrically increasing step sizes, as in tabular settings [17, 27, 19, 28, 6].
One-sided Approximation Error. For NPG, [4, Lemma 6.2] introduces a "one-sided approximation error" quantity, which aims to overestimate the advantage of the optimal action  $a^*$ ,
$$
\epsilon_ {t} := r \left(a ^ {*}\right) - \pi_ {\theta_ {t}} ^ {\top} r - w ^ {\top} \left(x _ {a ^ {*}} - X ^ {\top} \pi_ {\theta_ {t}}\right) = r \left(a ^ {*}\right) - \pi_ {\theta_ {t}} ^ {\top} r - \left(\hat {r} \left(a ^ {*}\right) - \pi_ {\theta_ {t}} ^ {\top} \hat {r}\right). \tag {19}
$$
This quantity relaxes the notion of approximation error and still guarantees the global convergence of NPG, since if  $\sum_{t=1} \epsilon_t \in o(T)$ , then NPG with  $\eta \in O(1/\sqrt{T})$  achieves global convergence [4, Lemma 6.2]. We note however that Eq. (19) has two limitations: (i) Eq. (19) depends on the entire update trajectory  $\{\theta_t\}_{t \geq 1}$ , which is hard to verify. By contrast, the optimal action preservation condition in Theorem 2 only involves problem quantities  $X$  and  $r$ . (ii) It is not clear whether Eq. (19) is a necessary condition for global convergence, while optimal action preservation is proved above to be both necessary and sufficient.
We conducted additional simulations to check the theoretical results. First, we check whether the strict inequality of  $\hat{r}(a^{*}) > \hat{r}(a)$  for all  $a \neq a^{*}$  in Theorem 2 is required for NPG global convergence.
Example 4.  $K = 4$ ,  $d = 2$ ,  $X^{\top} = \left[ \begin{array}{ccc}0 & -1 & 0 & 1\\ -1 & 0 & 1 & 0 \end{array} \right] \in \mathbb{R}^{d\times K}$ , and  $r = (9,8,7,6)^{\top} \in \mathbb{R}^{K}$ . The best fit for  $r$  is  $\hat{r} = X(X^{\top}X)^{-1}X^{\top}r = (1,1, - 1, - 1)^{\top}$ .
Example 4 has  $\hat{r}(a^*) = \hat{r}(1) = \hat{r}(2)$ , which violates the strict inequality condition of  $\hat{r}(a^*) > \hat{r}(a)$  for all  $a \neq a^*$ . The consequence is that NPG guarantees  $\frac{\pi_{\theta_t}(a^*)}{\pi_{\theta_t}(2)} = \frac{\pi_{\theta_1}(a^*)}{\pi_{\theta_1}(2)}$  for all  $t \geq 1$ , which makes it impossible for  $\pi_{\theta_t}(a^*) \to 1$  as  $t \to \infty$ . This is observed in Figure 3(a), supporting that the strictly inequality condition in Theorem 2 is indeed necessary. The initialization is  $\theta_1 = (4,10)^\top$ , and  $\eta = 0.2$ . We run 150 iterations for NPG and  $1.5 \times 10^7$  iterations for Softmax PG.
Second, we run 150 iterations of NPG on Example 1. As shown in Figure 3(b), the quantity  $\log (\pi^{*} - \pi_{\theta_{t}})^{\top}r$  is a linear function of time  $t$ , implying that  $(\pi^{*} - \pi_{\theta_{t}})^{\top}r \in O(e^{-c\cdot t})$  with  $c > 0$ . This supports the convergence rate results in Theorem 2. Here  $\theta_{1}$  and  $\eta$  are the same as in Figure 1(a).
Third, we check whether the condition in Theorem 1 is required for Softmax PG global convergence.
(a) Example 4.
(b)  $\log (\pi^{*} - \pi_{\theta_t})^\top r$
Figure 3: Simulations for verifying theoretical results.
(c)  $\log \left(\pi^{*} - \pi_{\theta_t}\right)^{\top}r$
(d) Example 5.
Example 5.  $K = 6, d = 2, X^{\top} = \left[ \begin{array}{ccccc}0 & -1 & -1 & 0 & 1 & 1\\ -1 & 0 & 1 & 1 & 0 & -1 \end{array} \right],$  and  $r = (9,8,7,6,5,4)^{\top}$
Similar to Eq. (9), it is impossible to find any  $w \in \mathbb{R}^d$ , such that  $r' := Xw$  preserves the order of  $r$  in Example 5. However, as shown in Figure 3(d), Softmax PG achieves  $\pi_{\theta_t}^\top \to r(a^*) = 9$ , indicating that the reward order preservation condition in Theorem 1 is sufficient but not necessary for PG to achieve global convergence. The initialization is  $\theta_1 = (10, -2)^\top$ , and  $\eta = 0.2$ . We run 100 iterations for NPG and  $2 \times 10^6$  iterations for Softmax PG. Note that NPG behaves erratically on Example 5 (which does not satisfy its global convergence conditions), by first entering then leaving the optimal plateau, eventually approaching a sub-optimal solution.
Finally, we run  $10^{8}$  iterations of Softmax PG on Example 1, using the same  $\eta$  and  $\theta_{1}$  as in Figure 1(a). Figure 3(c) shows that the slope of  $\log (\pi^{*} - \pi_{\theta_{t}})^{\top}r$  over log  $t$  approaches  $-1$ , indicating that the global convergence rate is  $(\pi^{*} - \pi_{\theta_{t}})^{\top}r\in O(1 / t)$ , matching the softmax tabular setting results [22].
Checking ordering-based conditions. Checking the existence of  $w \in \mathbb{R}^d$  in Theorem 1 is known as linear feasibility in literature [11], i.e., determining whether a set of inequalities has a non-empty intersection. In particular, suppose  $X \in \mathbb{R}^{K \times d}$ , and  $r \in \mathbb{R}^K$  is sorted, i.e.,  $r(1) \geq r(2) \geq \dots \geq r(K)$ . Denote  $x_i \in \mathbb{R}^d$  as the  $i$ -th row vector of  $X$ . The linear feasibility problem in this case is to check if there exists  $w \in \mathbb{R}^d$ , such that for all  $i \in [K-1]$ ,  $x_i^\top w \geq x_{i+1}^\top w$ . Linear feasibility can be cast as linear programming (LP) using a dummy objective and keeping the constraints, hence any LP technique, such as the ellipsoid method, can be used to solve it [11]. On the other hand, checking the optimal action preservation in Theorem 2 requires the same information as in calculating approximation error  $\| \hat{r} - r \|_2 = \min_{w \in \mathbb{R}^d} \| Xw - r \|_2$ , since  $\arg \max_{a \in [K]} \hat{r}(a) = \arg \max_{a \in [K]} r(a)$  can be immediately verified after calculating the projection  $\hat{r} := X^\top (X^\top X)^{-1} X^\top r$ .
Generalization to Markov decision processes (MDPs). Our work provides some new and useful insights for understanding more complex settings, but it requires further investigation to resolve this highly non-trivial problem for general MDPs. See Appendix C for detailed discussions.
We believe this work opens new directions for understanding PG-based methods under function approximation, going well beyond the conventional approximation error based analysis. The major technical findings involve ordering-based conditions and relevant techniques (covariance and global convergence). Identifying exact necessary and sufficient conditions for the global convergence of Softmax PG remains future work. Extending the results and techniques to general MDPs is another important and challenging next step. Combining function approximation with recent results on stochastic on-policy sampling [20] is another interesting direction for agnostic learning. Investigating whether these new global convergence conditions might be used to achieve better representation learning is of great interest for algorithm design. Generalizing the proof techniques to other scenarios where non-linear transforms (activation functions) interact with low-dimensional features through gradient descent, such as in neural networks, is another lofty ambition.
The authors would like to thank anonymous reviewers for their valuable comments. CS, DS gratefully acknowledge funding from the Canada CIFAR AI Chairs Program, Amii and NSERC.
[22] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning, pages 6820-6829. PMLR, 2020.
[23] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897, 2015.
[24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[25] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, pages 1057-1063, 2000.
[26] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256, 1992.
[27] Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1-36, 2022.
[28] Rui Yuan, Simon S Du, Robert M Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. arXiv preprint arXiv:2210.01400, 2022.