SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which is the Euclidean projection on the probability simplex. In the tabular setting, where  $S$  and  $\mathcal{A}$  are finite and  $f^{\theta}(s,a) = \theta_{s,a}$ , (15) can be solved exactly with the minimum equal to zero, and Equations (15) and (16) recover the projected Q-descent algorithm [93]. As a by-product, we generalize the projected Q-descent algorithm from the tabular setting to a general parameterization class  $\mathcal{F}^{\Theta}$ , which is a novel algorithm in the RL literature.
Example 3.7 (Negative entropy). If  $\mathcal{Y} = \mathbb{R}_+^{|\mathcal{A}|}$  and  $\phi(x) = \exp(x - 1)$ , then  $h_\phi$  is the negative entropy mirror map from Example 3.2 and Line 1 in Algorithm 1 becomes
$$
\theta^ {t + 1} \in \operatorname {a r g m i n} _ {\theta \in \Theta} \left\| f ^ {t + 1} - Q ^ {t} - \frac {\eta_ {t - 1}}{\eta_ {t}} f ^ {t} \right\| _ {L _ {2} (v ^ {t})} ^ {2}. \tag {17}
$$
Consequently, based on Example 3.2, we have  $\pi_s^{t + 1}\propto \exp (\eta_t f_s^{t + 1})$  for all  $s\in S$ . In this example, AMPO recovers tabular NPG [82] when  $f^{\theta}(s,a) = \theta_{s,a}$ , and Q-NPG with log-linear policies [98] when  $f^{\theta}$  and  $Q^t$  are linear functions for all  $t\geq 0$ .
We refer to Appendix C.2 for detailed derivations of the examples in this section and an efficient implementation of the Bregman projection step. In addition to the  $\ell_2$ -norm and the negative entropy, several other mirror maps that have been studied in the optimization literature fall into the class of  $\omega$ -potential mirror maps, such as the Tsallis entropy [74, 57] and the hyperbolic entropy [34], as well as a generalization of the negative entropy [50]. These examples illustrate how the class of  $\omega$ -potential mirror maps recovers known methods and can be used to explore new algorithms in policy optimization. We leave the study of the application of these mirror maps in RL as future work, both from an empirical and theoretical point of view, and provide additional discussion and details in Appendix C.2.
In our upcoming theoretical analysis of AMPO, we rely on the following key lemma.
Lemma 4.1. For any policies  $\pi$  and  $\bar{\pi}$ , for any function  $f^{\theta} \in \mathcal{F}^{\Theta}$  and for  $\eta > 0$ , we have
$$
\langle \eta f _ {s} ^ {\theta} - \nabla h (\bar {\pi} _ {s}), \pi_ {s} - \tilde {\pi} _ {s} \rangle \leq \mathcal {D} _ {h} (\pi_ {s}, \bar {\pi} _ {s}) - \mathcal {D} _ {h} (\tilde {\pi} _ {s}, \bar {\pi} _ {s}) - \mathcal {D} _ {h} (\pi_ {s}, \tilde {\pi} _ {s}), \quad \forall s \in S,
$$
where  $\tilde{\pi}$  is the Bregman projected policy induced by  $f^{\theta}$  and  $h$  according to Definition 3.1, that is  $\tilde{\pi}_s = \mathrm{argmin}_{p\in \Delta (\mathcal{A})}\mathcal{D}_h(p,\nabla h^* (\eta f_s^\theta))$  for all  $s\in \mathcal{S}$ .
The proof of Lemma 4.1 is given in Appendix D.1. Lemma 4.1 describes a relation between any two policies and a policy belonging to the Bregman projected policy class associated with  $\mathcal{F}^{\Theta}$  and  $h$ . As mentioned in Remark 3.4, Lemma 4.1 is the direct consequence of (12) and can be interpreted as an application of the three-point descent lemma [19], while it cannot be applied to algorithms based on the update in (11) [88, 89] due to the non-convexity of the optimization problem (see also Appendix F). Notice that Lemma 4.1 accommodates naturally with general parameterization also thanks to (12). In contrast, similar results have been obtained and exploited for specific policy and mirror map classes [93, 60, 36, 98], while our result allows any parameterization class  $\mathcal{F}^{\Theta}$  and any choice of mirror map, thus greatly expanding the scope of applications of the lemma. A similar result for general parameterization has been obtained by Lan [55, Proposition 3.5] in the setting of strongly convex mirror maps.
Lemma 4.1 becomes useful when we set  $\bar{\pi} = \pi^t$ ,  $f^\theta = f^{t + 1}$ ,  $\eta = \eta_t$  and  $\pi = \pi^t$  or  $\pi = \pi^{\star}$ . In particular, when  $\eta_{t}f_{s}^{t + 1} - \nabla h(\pi_{s}^{t})\approx \eta_{t}Q_{s}^{\pi}$ , Lemma 4.1 allows us to obtain telescopic sums and recursive relations, and to handle error terms efficiently, as we show in Appendix D.
4.1 Convergence for general policy parameterization
In this section, we consider the parameterization class  $\mathcal{F}^{\Theta}$  and the fixed but arbitrary mirror map  $h$ . We show that AMPO enjoys quasi-monotonic improvement and sublinear or linear convergence, depending on the step-size schedule. The first step is to control the approximation error of AMPO.
(A1) (Approximation error). There exists  $\varepsilon_{\mathrm{approx}} \geq 0$  such that, for all times  $t \geq 0$ ,
$$
\mathbb {E} \big [ \left\| f ^ {t + 1} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h (\pi^ {t}) \right\| _ {L _ {2} (v ^ {t})} ^ {2} \big ] \leq \varepsilon_ {\mathrm {a p p r o x}},
$$
where  $(v^t)_{t\geq 0}$  is a sequence of distributions over states and actions and the expectation is taken over the randomness of the algorithm that obtains  $f^{t + 1}$ .
Assumption (A1) is common in the conventional compatible function approximation approach [1]. It characterizes the loss incurred by Algorithm 1 in solving the regression problem in Line 1. When the step-size  $\eta_t$  is sufficiently large, Assumption (A1) measures how well  $f^{t+1}$  approximates the current Q-function  $Q^t$ . Hence,  $\varepsilon_{\mathrm{approx}}$  depends on both the accuracy of the policy evaluation method used to obtain an estimate of  $Q^t$  [86, 79, 28] and the error incurred by the function  $f^\theta \in \mathcal{F}^\Theta$  that best approximates  $Q^t$ , that is the representation power of  $\mathcal{F}^\Theta$ . Later in Section 4.2, we show how to solve the minimization problem in Line 1 when  $\mathcal{F}^\Theta$  is a class of shallow neural networks so that Assumption (A1) holds. We highlight that Assumption (A1) is weaker than the conventional assumptions [1, 98], since we do not constrain the minimization problem to be linear in the parameters (see (61)). We refer to Appendix A.2 for a discussion on its technical novelty and Appendix G for a relaxed version of the assumption.
As mentioned in Remark 3.3, the distribution  $v^t$  does not depend on the current policy  $\pi^t$  for all times  $t \geq 0$ . Thus, Assumption (A1) allows for off-policy settings and the use of replay buffers [68]. We refer to Appendix A.3 for details. To quantify how the choice of these distributions affects the error terms in the convergence rates, we introduce the following coefficient.
(A2) (Concentrability coefficient). There exists  $C_v \geq 0$  such that, for all times  $t$ ,
$$
\mathbb {E} _ {(s, a) \sim v ^ {t}} \left[ \left(\frac {d _ {\mu} ^ {\pi} (s) \pi (a \mid s)}{v ^ {t} (s , a)}\right) ^ {2} \right] \leq C _ {v},
$$
whenever  $(d_{\mu}^{\pi},\pi)$  is either  $(d_{\mu}^{\star},\pi^{\star})$ $(d_{\mu}^{t + 1},\pi^{t + 1})$ $(d_{\mu}^{\star},\pi^{t})$  , or  $(d_{\mu}^{t + 1},\pi^{t})$
The concentrability coefficient  $C_v$  quantifies how much the distribution  $v^t$  overlaps with the distributions  $(d_\mu^\star, \pi^\star)$ ,  $(d_\mu^{t+1}, \pi^{t+1})$ ,  $(d_\mu^\star, \pi^t)$  and  $(d_\mu^{t+1}, \pi^t)$ . It highlights that the distribution  $v^t$  should have full support over the environment, in order to avoid large values of  $C_v$ . Assumption (A2) is weaker than the previous best-known concentrability coefficient [98, Assumption 9], in the sense that we have the full control over  $v^t$ . We refer to Appendix H for a more detailed discussion. We can now present our first result on the performance of Algorithm 1.
Proposition 4.2 (Quasi-monotonic updates). Let (A1), (A2) be true. We have, for all  $t \geq 0$
$$
\mathbb {E} \left[ V ^ {t + 1} (\mu) - V ^ {t} (\mu) \right] \geq \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \frac {\mathcal {D} _ {h} \left(\pi_ {s} ^ {t + 1} , \pi_ {s} ^ {t}\right) + \mathcal {D} _ {h} \left(\pi_ {s} ^ {t} , \pi_ {s} ^ {t + 1}\right)}{\eta_ {t} (1 - \gamma)} \right] \right] - \frac {2 \sqrt {C _ {v} \varepsilon_ {\mathrm {a p p r o x}}}}{1 - \gamma},
$$
where the expectation is taken over the randomness of AMPO.
We refer to Appendix D.3 for the proof. Proposition 4.2 ensures that an update of Algorithm 1 cannot lead to a performance degradation, up to an error term. The next assumption concerns the coverage of the state space for the agent at each time  $t$ .
(A3) (Distribution mismatch coefficient). Let  $d_{\mu}^{\star} \coloneqq d_{\mu}^{\pi^{\star}}$ . There exists  $\nu_{\mu} \geq 0$  such that
$$
\sup  _ {s \in \mathcal {S}} \frac {d _ {\mu} ^ {*} (s)}{d _ {\mu} ^ {t} (s)} \leq \nu_ {\mu}, \quad \text {f o r a l l t i m e s} t \geq 0.
$$
Since  $d_{\mu}^t (s)\geq (1 - \gamma)\mu (s)$  for all  $s\in S$  , obtained from the definition of  $d_{\mu}$  in (2), we have that
$$
\sup _ {s \in \mathcal {S}} \frac {d _ {\mu} ^ {\star} (s)}{d _ {\mu} ^ {t} (s)} \leq \frac {1}{1 - \gamma} \sup _ {s \in \mathcal {S}} \frac {d _ {\mu} ^ {\star} (s)}{\mu (s)},
$$
where assuming boundedness for the term on the right-hand side is standard in the literature on both the PG [e.g., 101, 90] and NPG convergence analysis [e.g., 1, 15, 93]. We refer to Appendix I for details. It is worth mentioning that the quasi-monotonic improvement in Proposition 4.2 holds without (A3).
We define the weighted Bregman divergence between the optimal policy  $\pi^{\star}$  and the initial policy  $\pi^0$  as  $\mathcal{D}_0^\star = \mathbb{E}_{s\sim d_\mu^\star}[\mathcal{D}_h(\pi_s^\star ,\pi_s^0)]$ . We then have our main results below.
An extended discussion of this approach is provided in Appendix G.
Theorem 4.3. Let (A1), (A2) and (A3) be true. If the step-size schedule is non-decreasing, i.e.,  $\eta_t \leq \eta_{t+1}$  for all  $t \geq 0$ , the iterates of Algorithm 1 satisfy: for every  $T \geq 0$
$$
V ^ {\star} (\mu) - \frac {1}{T} \sum_ {t <   T} \mathbb {E} \left[ V ^ {t} (\mu) \right] \leq \frac {1}{T} \left(\frac {\mathcal {D} _ {0} ^ {\star}}{(1 - \gamma) \eta_ {0}} + \frac {\nu_ {\mu}}{1 - \gamma}\right) + \frac {2 (1 + \nu_ {\mu}) \sqrt {C _ {v} \varepsilon_ {\mathrm {a p p r o x}}}}{1 - \gamma}.
$$
Furthermore, if the step-size schedule is geometrically increasing, i.e., satisfies
$$
\eta_ {t + 1} \geq \frac {\nu_ {\mu}}{\nu_ {\mu} - 1} \eta_ {t} \quad \forall t \geq 0, \tag {18}
$$
we have: for every  $T \geq 0$ ,
$$
V ^ {\star} (\mu) - \mathbb {E} \left[ V ^ {T} (\mu) \right] \leq \frac {1}{1 - \gamma} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {T} \left(1 + \frac {\mathcal {D} _ {0} ^ {\star}}{\eta_ {0} (\nu_ {\mu} - 1)}\right) + \frac {2 (1 + \nu_ {\mu}) \sqrt {C _ {v} \varepsilon_ {\mathrm {a p p r o x}}}}{1 - \gamma}.
$$
Theorem 4.3 is, to the best of our knowledge, the first result that establishes linear convergence for a PG-based method involving general policy parameterization. For the same setting, it also matches the previous best known  $O(1 / T)$  convergence [55], without requiring regularization. Lastly, Theorem 4.3 provides a convergence rate for a PMD-based algorithm that allows for arbitrary mirror maps and policy parameterization without requiring the assumption on the approximation error to hold in  $\ell_{\infty}$ -norm, in contrast to Lan [55]. We give here a brief discussion of Theorem 4.3 w.r.t. previous results and refer to Tables 1 and 2 in Appendix A.2 for a detailed comparison.
In terms of iteration complexity, Theorem 4.3 recovers the best-known convergence rates in the tabular setting [93], for both non-decreasing and exponentially increasing step-size schedules. While considering a more general setting, Theorem 4.3 matches or improves upon the convergence rate of previous work on policy gradient methods for non-tabular policy parameterizations that consider constant step-size schedules [60, 82, 61, 90, 1, 89, 16, 55], and matches the convergence speed of previous work that employ NPG, log-linear policies, and geometrically increasing step-size schedules [2, 98].
In terms of generality, the results in Theorem 4.3 hold without the need to implement regularization [17, 100, 15, 16, 54], to impose bounded updates or smoothness of the policy [1, 61], to restrict the analysis to the case where the mirror map  $h$  is the negative entropy [60, 36], or to make  $\ell_{\infty}$ -norm assumptions on the approximation error [55]. We improve upon the latest results for PMD with general policy parameterization by Vaswani et al. [89], which only allow bounded step-sizes, where the bound can be particularly small, e.g.,  $(1 - \gamma)^{3} / (2\gamma |\mathcal{A}|)$ , and can slow down the learning process.
When  $S$  is a finite state space, a sufficient condition for  $\nu_{\mu}$  in (A3) to be bounded is requiring  $\mu$  to have full support on  $S$ . If  $\mu$  does not have full support, one can still obtain linear convergence for  $V^{\star}(\mu') - V^{T}(\mu')$ , for an arbitrary state distribution  $\mu'$  with full support, and relate this quantity to  $V^{\star}(\mu) - V^{T}(\mu)$ . We refer to Appendix I for a detailed discussion on the distribution mismatch coefficient.
Intuition. An interpretation of our theory can be provided by connecting AMPO to the Policy Iteration algorithm (PI), which also enjoys linear convergence. To see this, first recall (12)
$$
\pi_ {s} ^ {t + 1} \in \operatorname * {a r g m i n} _ {p \in \Delta (\mathcal {A})} \langle - f _ {s} ^ {t + 1} + \eta_ {t} ^ {- 1} \nabla h (\pi_ {s} ^ {t}), p \rangle + \eta_ {t} ^ {- 1} \mathcal {D} _ {h} (p, \pi_ {s} ^ {t}), \quad \forall s \in \mathcal {S}.
$$
Secondly, solving Line 1 of Algorithm 1 leads to  $f_{s}^{t + 1} - \eta_{t}^{-1}\nabla h(\pi_{s}^{t})\approx Q_{s}^{t}$ . When the step-size  $\eta_t\to \infty$ , that is  $\eta_t^{-1}\rightarrow 0$ , the above viewpoint of the AMPO policy update becomes
$$
\pi_ {s} ^ {t + 1} \in \operatorname * {a r g m i n} _ {p \in \Delta (\mathcal {A})} \langle - Q _ {s} ^ {t}, p \rangle \iff \pi_ {s} ^ {t + 1} \in \operatorname * {a r g m a x} _ {p \in \Delta (\mathcal {A})} \langle Q _ {s} ^ {t}, p \rangle , \forall s \in \mathcal {S},
$$
which is the PI algorithm. Here we ignore the Bregman divergence term  $\mathcal{D}_h(\pi, \pi_s^t)$ , as it is multiplied by  $1 / \eta_t$ , which goes to 0. So AMPO behaves more and more like PI with a large enough step-size and thus is able to converge linearly like PI.
Proof idea. We provide a sketch of the proof here; the full proof is given in Appendix D. In a nutshell, the convergence rates of AMPO are obtained by building on Lemma 4.1 and leveraging modern PMD proof techniques [93]. Following the conventional compatible function approximation approach [1], the idea is to write the global optimum convergence results in an additive form, that is
sub-optimality gap  $\leq$  optimization error  $^+$  approximation error.
The separation between the two errors is allowed by Lemma 4.1, while the optimization error is bounded through the PMD proof techniques from Xiao [93] and the approximation error is characterized by Assumption (A1). Overall, the proof consists of three main steps.
Step 1. Using Lemma 4.1 with  $\bar{\pi} = \pi^t$ ,  $f^\theta = f^{t + 1}$ ,  $\eta = \eta_t$ ,  $\tilde{\pi} = \pi^{t + 1}$ , and  $\pi_s = \pi_s^t$ , we obtain
$$
\left\langle \eta_ {t} f _ {s} ^ {t + 1} - \nabla h \left(\pi_ {s} ^ {t}\right), \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \right\rangle \geq 0,
$$
which characterizes the improvement of the updated policy.
Step 2. Assumption (A1), Step 1, the performance difference lemma (Lemma D.4), and Lemma 4.1 with  $\bar{\pi} = \pi^t$ ,  $f^\theta = f^{t + 1}$ ,  $\eta = \eta_t$ ,  $\tilde{\pi} = \pi^{t + 1}$ , and  $\pi_s = \pi_s^\star$  permit us to obtain the following.
Proposition 4.4. Let  $\Delta_t \coloneqq V^\star(\mu) - V^t(\mu)$ . For all  $t \geq 0$ , we have
$$
\mathbb {E} \left[ \nu_ {\mu} \left(\Delta_ {t + 1} - \Delta_ {t}\right) + \Delta_ {t} \right] \leq \mathbb {E} \left[ \frac {\mathbb {E} _ {s \sim d _ {\mu} ^ {*}} \left[ \mathcal {D} _ {h} \left(\pi_ {s} ^ {\star} , \pi_ {s} ^ {t}\right) \right]}{(1 - \gamma) \eta_ {t}} - \frac {\mathbb {E} _ {s \sim d _ {\mu} ^ {*}} \left[ \mathcal {D} _ {h} \left(\pi_ {s} ^ {\star} , \pi_ {s} ^ {t + 1}\right) \right]}{(1 - \gamma) \eta_ {t}} \right] + (1 + \nu_ {\mu}) \frac {2 \sqrt {C _ {v} \varepsilon_ {\text {a p p r o x}}}}{1 - \gamma}.
$$
Step 3. Proposition 4.4 leads to sublinear convergence using a telescoping sum argument, and to linear convergence by properly defining step-sizes and by rearranging terms into the following contraction,
$$
\mathbb {E} \Big [ \Delta_ {t + 1} + \frac {\mathbb {E} _ {s \sim d _ {\mu} ^ {t}} \big [ \mathcal {D} _ {h} (\pi_ {s} ^ {\star} , \pi_ {s} ^ {t + 1}) \big ]}{(1 - \gamma) \eta_ {t + 1} (\nu_ {\mu} - 1)} \Big ] \leq \Big (1 - \frac {1}{\nu_ {\mu}} \Big) \mathbb {E} \Big [ \Delta_ {t} + \frac {\mathbb {E} _ {s \sim d _ {\mu} ^ {t}} \big [ \mathcal {D} _ {h} (\pi_ {s} ^ {\star} , \pi_ {s} ^ {t}) \big ]}{(1 - \gamma) \eta_ {t} (\nu_ {\mu} - 1)} \Big ] + \Big (1 + \frac {1}{\nu_ {\mu}} \Big) \frac {2 \sqrt {C _ {v} \varepsilon_ {\mathrm {a p p r o x}}}}{1 - \gamma}.
$$
4.2 Sample complexity for neural network parameterization
Neural networks are widely used in RL due to their empirical success in applications [67, 68, 84]. However, few theoretical guarantees exist for using this parameterization class in policy optimization [60, 90, 16]. Here, we show how we can use our framework and Theorem 4.3 to fill this gap by deriving a sample complexity result for AMPO when using neural network parameterization. We consider the case where the parameterization class  $\mathcal{F}^{\Theta}$  from Definition 3.1 belongs to the family of shallow ReLU networks, which have been shown to be universal approximators [38, 4, 27, 39]. That is, for  $(s,a)\in (\mathcal{S}\times \mathcal{A})\subseteq \mathbb{R}^{d}$ , define  $f^{\theta}(s,a) = c^{\top}\sigma (W(s,a) + b)$  with  $\theta = (c,W,b)$ , where  $\sigma (y) = \max (y,0)$  for all  $y\in \mathbb{R}$  is the ReLU activation function and is applied element-wisely,  $c\in \mathbb{R}^m$ ,  $W\in \mathbb{R}^{m\times d}$  and  $b\in \mathbb{R}^m$ .
At each iteration  $t$  of AMPO, we set  $v^{t} = d_{\mu}^{t}$  and solve the regression problem in Line 1 of Algorithm 1 through stochastic gradient descent (SGD). In particular, we initialize entry-wise  $W_{0}$  and  $b$  as i.i.d. random Gaussian variables from  $\mathcal{N}(0,1 / m)$ , and  $c$  as i.i.d. random Gaussian variables from  $\mathcal{N}(0,\epsilon_A)$  with  $\epsilon_{A}\in (0,1]$ . Assuming access to a simulator for the distribution  $v^{t}$ , we run SGD for  $K$  steps on the matrix  $W$ , that is, for  $k = 0,\dots ,K - 1$ ,
$$
W _ {k + 1} = W _ {k} - \alpha \left(f ^ {(k)} (s, a) - \widehat {Q} ^ {t} (s, a) - \eta_ {t} ^ {- 1} \nabla h \left(\pi_ {s} ^ {t}\right)\right) \nabla_ {W} f ^ {(k)} (s, a), \tag {19}
$$
where  $f^{(k)}(s,a) = c^{\top}\sigma ((W_0 + W_k)(s,a) + b), (s,a)\sim v^t$  and  $\widehat{Q}^t (s,a)$  is an unbiased estimate of  $Q^{t}(s,a)$  obtained through Algorithm 4. We can then present our result on the sample complexity of AMPO for neural network parameterization, which is based on our convergence Theorem 4.3 and an analysis of neural networks by Allen-Zhu et al. [3, Theorem 1].
Corollary 4.5. In the setting of Theorem 4.3, let the parameterization class  $\mathcal{F}^{\Theta}$  consist of sufficiently wide shallow ReLU neural networks. Using an exponentially increasing step-size and solving the minimization problem in Line 1 with SGD as in (19), the number of samples required by AMPO to find an  $\varepsilon$ -optimal policy with high probability is  $\widetilde{\mathcal{O}}(C_v^2\nu_\mu^5/\varepsilon^4(1-\gamma)^6)$ , where  $\varepsilon$  has to be larger than a fixed and non-vanishing error floor.
We provide a proof of Corollary 4.5 and an explicit expression for the error floor in Appendix J. Note that the sample complexity in Corollary 4.5 might be impacted by an additional  $\mathsf{poly}(\varepsilon^{-1})$  term. We refer to Appendix J for more details and an alternative result (Corollary J.4) which does not include an additional  $\mathsf{poly}(\varepsilon^{-1})$  term, enabling comparison with prior works.
We provide an empirical evaluation of AMPO in order to validate our theoretical findings. We note that the scope of this work is mainly theoretical and that we do not aim at establishing state-of-the-art results in the setting of deep RL. Our implementation is based upon the PPO implementation
from PureJaxRL [63], which obtains the estimates of the  $Q$ -function through generalized advantage estimation (GAE) [79] and performs the policy update using ADAM optimizer [48] and mini-batches. To implement AMPO, we (i) replaced the PPO loss with the expression to minimize in Equation (14), (ii) replaced the softmax projection with the Bregman projection, (iii) saved the constants  $\lambda$  along the sampled trajectories in order to compute Equation (14). The code is available here.
In Figure 1, we show the averaged performance over 100 runs of AMPO in two classic control environments, i.e. CartPole and Acrobot, in the setting of  $\omega$ -potential mirror maps. In particular, we choose:  $\phi(x) = e^x$ , which corresponds to the negative entropy (Example 3.7);  $\phi(x) = \sinh(x)$ , which corresponds to the hyperbolic entropy [34, see also (41)];  $\phi(x) = x$ , which corresponds to the Euclidean norm (Example 3.6); and the Tsallis entropy for two values of the entropic index  $q$  [74, 57, see also (40)]. We refer to Appendix C.2 for a detailed discussion on these mirror maps. We set the step-size to be constant and of value 1. For a comparison, we also plot the averaged performance over 100 runs of PPO.
The plots in Figure 1 confirm our results on the quasi-monotonicity of the updates of AMPO and on its convergence to the optimal policy. We observe that instances of AMPO with different mirror maps are very competitive as compared to PPO. We also note that, despite the convergence rates in Theorem 4.3 depend on the mirror map only in terms of a  $\mathcal{D}_0^\star$  term, different mirror maps may result in different convergence speeds and error floors in practice. In particular, our experiments suggest that the negative entropy mirror map may not be the best choice for AMPO, and that exploring different mirror maps is a promising direction of research.
We have introduced a novel framework for RL which, given a mirror map and any parameterization class, induces a policy class and an update rule. We have proven that this framework enjoys sublinear and linear convergence for non-decreasing and geometrically increasing step-size schedules, respectively. Future venues of investigation include studying the sample complexity of AMPO in on-policy and off-policy settings other than neural network parameterization, exploiting the properties of specific mirror maps to take advantage of the structure of the MDP and efficiently including representation learning in the algorithm. We refer to Appendix A.3 for a thorough discussion of future work. We believe that the main contribution of AMPO is to provide a general framework with theoretical guarantees that can help the analysis of specific algorithms and MDP structures. AMPO recovers and improves several convergence rate guarantees in the literature, but it is important to keep in consideration how previous works have exploited particular settings, while AMPO tackles the most general case. It will be interesting to see whether these previous works combined with our fast linear convergence result can derive new efficient sample complexity results.
We thank the anonymous reviewers for their helpful comments. Carlo Alfano was supported by the Engineering and Physical Sciences Research Council and thanks G-Research for partly funding attendance to NeurIPS 2023.