SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
h _ {\phi} (\pi_ {s}) = \sum_ {a \in \mathcal {A}} \int_ {1} ^ {\pi (a | s)} (\log (x + \epsilon) + 1) d x = \sum_ {a \in \mathcal {A}} \left[ (\pi (a | s) + \epsilon) \ln (\pi (a | s) + \epsilon) - (1 + \epsilon) \ln (1 + \epsilon) \right].
$$
Algorithm 3: Bregman projection for  $\omega$ -potential mirror maps
Input: vector to project  $x\in \mathbb{R}^{|\mathcal{A}|}$ $\omega$  -potential  $\phi$  ,precision  $\varepsilon$
1 Initialize
$$
\bar {\nu} = \phi^ {- 1} (1) - \max _ {a \in \mathcal {A}} x _ {a}
$$
$$
\underline {{\nu}} = \phi^ {- 1} (1 / | \mathcal {A} |) - \max _ {a \in \mathcal {A}} x _ {a}
$$
2 Define  $\tilde{x} (\nu) = \bigl (\sigma \bigl (\phi \bigl (x_a + \nu \bigr)\bigr)\bigr)_{a\in \mathcal{A}}$
while  $\| \tilde{x} (\bar{\nu}) - \tilde{x} (\nu)\| _1 > \varepsilon$  do 3 Let  $\nu^{+}\gets (\bar{\nu} +\nu) / 2$  4 if  $\sum_{a\in \mathcal{A}}\tilde{x}_a(\nu^+) > 1$  then 5  $\begin{array}{rl}{\bigtriangledown}\bar{\nu}\leftarrow \nu^{+} \end{array}$  6 else 7  $\begin{array}{r}\lfloor \underline{\nu}\leftarrow \nu^{+} \end{array}$
8 Return  $\tilde{x} (\bar{\nu})$
An exact solution to the associated projection can then be found in  $\widetilde{\mathcal{O}}(|\mathcal{A}|)$  computations using Algorithm 2, which has been proposed by Krichene et al. [50, Algorithm 4]. Additionally, following (14), the regression problem in Line 1 of Algorithm 1 becomes
$$
\theta^ {t + 1} \in \underset {\theta \in \Theta} {\operatorname {a r g m i n}} \left\| f ^ {\theta} - Q ^ {t} - \eta_ {t} ^ {- 1} \max (\eta_ {t - 1} f ^ {t}, 1 + \log (\epsilon) - \lambda_ {s} ^ {t}) \right\| _ {L _ {2} (v ^ {t})} ^ {2},
$$
where  $\lambda_s^t$  can be obtained through Algorithm 2.
The Bregman projection for generic mirror maps can be computed approximately in  $\widetilde{\mathcal{O}}(|\mathcal{A}|)$  computations through a bisection algorithm. Krichene et al. [50] propose one such algorithm, which we report in Algorithm 3 for completeness. We next provide two mirror maps that have appeared before in the optimization literature, but do not lead to an exact solution to the Bregman projection. We leave them as object for future work.
Negative Tsallis entropy [74, 57]. Let  $q > 0$  and define  $\phi$  as
$$
\phi_ {q} (x) = \left\{ \begin{array}{l l} \exp (x - 1) & \text {i f} q = 1, \\ \left[ \sigma \left(\frac {(q - 1) x}{q}\right) \right] ^ {\frac {1}{q - 1}} & \text {e l s e}. \end{array} \right.
$$
The mirror map  $h_{\phi_q}$  becomes the negative Tsallis entropy, that is
$$
h _ {\phi_ {q}} \left(\pi_ {s}\right) = \sum \pi (a \mid s) \log_ {q} (\pi (a \mid s)), \tag {40}
$$
where, for  $y > 0$
$$
\log_ {q} (y) = \left\{ \begin{array}{l l} \log (y) & \text {i f} q = 1, \\ \frac {- y ^ {q - 1}}{q - 1} & \text {e l s e .} \end{array} \right.
$$
If  $q \neq 1$  and following (14), the regression problem in Line 1 of Algorithm 1 becomes
$$
\theta^ {t + 1} \in \operatorname * {a r g m i n} _ {\theta \in \Theta} \left\| f ^ {\theta} - Q ^ {t} - \frac {\eta_ {t - 1}}{\eta_ {t}} f ^ {t} \right\| _ {L _ {2} (v ^ {t})} ^ {2},
$$
Hyperbolic entropy [34]. Let  $b > 0$  and define  $\phi$  as
$$
\phi_ {b} (x) = b \sinh (x)
$$
The mirror map  $h_{\phi_b}$  becomes the hyperbolic entropy, that is
$$
h _ {\phi_ {b}} (\pi_ {s}) = \sum_ {a \in \mathcal {A}} \pi (a \mid s) \operatorname {a r c s i n h} (\pi (a \mid s) / b) - \sqrt {\pi (a \mid s) ^ {2} + b ^ {2}}, \tag {41}
$$
and, following (14), the regression problem in Line 1 of Algorithm 1 becomes
$$
\theta^ {t + 1} \in \operatorname * {a r g m i n} _ {\theta \in \Theta} \left\| f ^ {\theta} - Q ^ {t} - \eta_ {t} ^ {- 1} \max  (\eta_ {t - 1} f ^ {t}, - \lambda_ {s} ^ {t}) \right\| _ {L _ {2} (v ^ {t})} ^ {2}.
$$
Hyperbolic-tangent entropy. Inspired by the hyperbolic entropy, we consider  $\phi$  as
$$
\phi (x) = \tanh (x) / 2 + 0. 5
$$
The mirror map  $h_\phi$  becomes
$$
h _ {\phi} \left(\pi_ {s}\right) = \frac {1}{2} \sum_ {a \in \mathcal {A}} (2 \pi (a \mid s) - 1) \operatorname {a r c t a n h} (2 \pi (a \mid s) - 1) + \frac {1}{2} \log \pi (a \mid s) (1 - \pi (a \mid s)), \tag {42}
$$
which, to the best of our knowledge, is not equivalent to any mirror map studied in the literature. Following (14), the regression problem in Line 1 of Algorithm 1 becomes
$$
\theta^ {t + 1} \in \operatorname * {a r g m i n} _ {\theta \in \Theta} \left\| f ^ {\theta} - Q ^ {t} - \frac {\eta_ {t - 1}}{\eta_ {t}} f ^ {t} \right\| _ {L _ {2} (v ^ {t})} ^ {2}.
$$
Regarding the limitations of the  $\omega$ -potential mirror map class, we are aware of two previously used mirror maps that cannot be recovered using  $\omega$ -potentials:  $h(x) = \frac{1}{2} x^{\top}\mathbf{A}x$  [32], for some positive-definite matrix  $\mathbf{A}$ , which generates the Mahalanobis distance, and  $p$ -norms, i.e.  $h(x) = \| x\| _p^2$  [74]. Note that the case where  $h(x) = \| x\| _p^p$  can be recovered by setting  $\phi (x) = (px)^{p / (1 - p)}$ .
We note that tuning the mirror map and the step-size can lead AMPO to encompass the case of deterministic policies, which can be obtained when using softmax policies by sending the step-size to infinity, effectively turning the softmax operator into a max operator. Another simple way of introducing deterministic policies in our framework is to choose the mirror map to be the Euclidean norm and to choose the step-size large enough. Doing so will cause the Bregman projection to put all the probability on the action that corresponds to the maximum value of  $f_{s}^{\theta}$ . Our results hold in this setting because our analysis does not use the policy gradient theorem (3), which has a different expression for deterministic policies [83].
D Deferred proofs from Section 4.1
D.1 Proof of Lemma 4.1
Here we provide the proof of Lemma 4.1, an application of the three-point descent lemma that accommodates arbitrary parameterized functions. Lemma 4.1 is the key tool for our analysis of AMPO. It is a generalization of both Xiao [93, Equation (44)] and Yuan et al. [98, Equation (50)] thanks to our two-step PMD framework. First, we recall some technical conditions of the mirror map [14, Chapter 4].
Suppose that  $\mathcal{Y} \subset \mathbb{R}^{|\mathcal{A}|}$  is a closed convex set, we say a function  $h: \mathcal{Y} \to \mathbb{R}$  is a mirror map if it satisfies the following properties:
(i)  $h$  is strictly convex and differentiable;
(ii)  $h$  is essentially smooth, i.e., the graident of  $h$  diverges on the boundary of  $\mathcal{V}$ , that is  $\lim_{x\to \partial \mathcal{V}}\| \nabla h(x)\| \to \infty$ ;
(iii) the gradient of  $h$  takes all possible values, that is  $\nabla h(\mathcal{V}) = \mathbb{R}^{|\mathcal{A}|}$ .
To prove Lemma 4.1, we also need the following rather simple properties, i.e., the three-point identity and the generalized Pythagorean theorem, satisfied by the Bregman divergence. We provide their proofs for self-containment.
Lemma D.1 (Three-point identity, Lemma 3.1 in Chen and Teboulle [19]). Let  $h$  be a mirror map. For any  $a, b$  in the relative interior of  $\mathcal{V}$  and  $c \in \mathcal{V}$ , we have that:
$$
\mathcal {D} _ {h} (c, a) + \mathcal {D} _ {h} (a, b) - \mathcal {D} _ {h} (c, b) = \left\langle \nabla h (b) - \nabla h (a), c - a \right\rangle . \tag {43}
$$
Proof. Using the definition of the Bregman divergence  $\mathcal{D}_h$ , we have
$$
\langle \nabla h (a), c - a \rangle = h (c) - h (a) - \mathcal {D} _ {h} (c, a), \tag {44}
$$
$$
\langle \nabla h (b), a - b \rangle = h (a) - h (b) - \mathcal {D} _ {h} (a, b), \tag {45}
$$
$$
\langle \nabla h (b), c - b \rangle = h (c) - h (b) - \mathcal {D} _ {h} (c, b). \tag {46}
$$
Subtracting (44) and (45) from (46) yields (43).
Lemma D.2 (Generalized Pythagorean Theorem of Bregman divergence, Lemma 4.1 in Bubeck [14]). Let  $\mathcal{X} \subseteq \mathcal{Y}$  be a closed convex set. Let  $h$  be a mirror map defined on  $\mathcal{Y}$ . Let  $x \in \mathcal{X}$ ,  $y \in \mathcal{Y}$  and  $y^{\star} = \operatorname{Proj}_{\mathcal{X}}^{h}(y)$ , then
$$
\langle \nabla h (y ^ {\star}) - \nabla h (y), y ^ {\star} - x \rangle \leq 0,
$$
which also implies
$$
\mathcal {D} _ {h} \left(x, y ^ {\star}\right) + \mathcal {D} _ {h} \left(y ^ {\star}, y\right) \leq \mathcal {D} _ {h} (x, y). \tag {47}
$$
Proof. From the definition of  $y^{\star}$ , which is
$$
y^{\star}\in \operatorname *{argmin}_{y^{\prime}\in \mathcal{X}}\mathcal{D}_{h}(y^{\prime},y),
$$
and from the first-order optimality condition [14, Proposition 1.3], with
$$
\nabla_ {y ^ {\prime}} \mathcal {D} _ {h} (y ^ {\prime}, y) = \nabla h (y ^ {\prime}) - \nabla h (y), \quad \text {f o r a l l} y ^ {\prime} \in \mathcal {Y},
$$
$$
\langle \nabla_ {y ^ {\prime}} \mathcal {D} _ {h} (y ^ {\prime}, y) | _ {y ^ {\prime} = y ^ {\star}}, y ^ {\star} - x \rangle \leq 0 \quad \Longrightarrow \quad \langle \nabla h (y ^ {\star}) - \nabla (y), y ^ {\star} - x \rangle \leq 0,
$$
which implies (47) by applying the definition of Bregman divergence and rearranging terms.
Now we are ready to prove Lemma 4.1.
Lemma D.3 (Lemma 4.1). Let  $\mathcal{V} \subset \mathbb{R}^{|\mathcal{A}|}$  be a closed convex set with  $\Delta(\mathcal{A}) \subseteq \mathcal{V}$ . For any policies  $\pi \in \Delta(\mathcal{A})^S$  and  $\bar{\pi}$  in the relative interior of  $\Delta(\mathcal{A})^S$ , any function  $f^\theta$  with  $\theta \in \Theta$ , any  $s \in S$  and for  $\eta > 0$ , we have that,
$$
\langle \eta f _ {s} ^ {\theta} - \nabla h (\bar {\pi} _ {s}), \pi_ {s} - \tilde {\pi} _ {s} \rangle \leq \mathcal {D} _ {h} (\pi_ {s}, \bar {\pi} _ {s}) - \mathcal {D} _ {h} (\tilde {\pi} _ {s}, \bar {\pi} _ {s}) - \mathcal {D} _ {h} (\pi , \tilde {\pi} _ {s}),
$$
where  $\tilde{\pi}$  is induced by  $f^{\theta}$  and  $\eta$  according to Definition 3.1, that is, for all  $s\in S$
$$
\tilde {\pi} _ {s} = \operatorname {P r o j} _ {\Delta (\mathcal {A})} ^ {h} \left(\nabla h ^ {*} \left(\eta f _ {s} ^ {\theta}\right)\right) = \underset {\pi_ {s} ^ {\prime} \in \Delta (\mathcal {A})} {\operatorname {a r g m i n}} \mathcal {D} _ {h} \left(\pi_ {s} ^ {\prime}, \nabla h ^ {*} \left(\eta f _ {s} ^ {\theta}\right)\right). \tag {48}
$$
Proof. For clarity of exposition, let  $p_s = \nabla h^*(\eta f_s^\theta)$ . Plugging  $a = \bar{\pi}_s$ ,  $b = p_s$  and  $c = \pi_s$  in the three-point identity lemma D.1, we obtain
$$
\mathcal {D} _ {h} \left(\pi_ {s}, \bar {\pi} _ {s}\right) - \mathcal {D} _ {h} \left(\pi_ {s}, p _ {s}\right) + \mathcal {D} _ {h} \left(\bar {\pi} _ {s}, p _ {s}\right) = \left\langle \nabla h \left(\bar {\pi} _ {s}\right) - \nabla h \left(p _ {s}\right), \bar {\pi} _ {s} - \pi_ {s} \right\rangle . \tag {49}
$$
Similarly, plugging  $a = \bar{\pi}_s, b = p_s$  and  $c = \tilde{\pi}_s$  in the three-point identity lemma D.1, we obtain
$$
\mathcal {D} _ {h} \left(\tilde {\pi} _ {s}, \bar {\pi} _ {s}\right) - \mathcal {D} _ {h} \left(\tilde {\pi} _ {s}, p _ {s}\right) + \mathcal {D} _ {h} \left(\bar {\pi} _ {s}, p _ {s}\right) = \left\langle \nabla h \left(\tilde {\pi} _ {s}\right) - \nabla h \left(p _ {s}\right), \tilde {\pi} _ {s} - \tilde {\pi} _ {s} \right\rangle . \tag {50}
$$
From (49), we have
$$
\begin{array}{l} \mathcal {D} _ {h} \left(\pi_ {s}, \bar {\pi} _ {s}\right) - \mathcal {D} _ {h} \left(\pi_ {s}, p _ {s}\right) + \mathcal {D} _ {h} \left(\bar {\pi} _ {s}, p _ {s}\right) \\ = \left\langle \nabla h \left(\bar {\pi} _ {s}\right) - \nabla h \left(p _ {s}\right), \bar {\pi} _ {s} - \pi_ {s} \right\rangle \\ = \langle \nabla h (\bar {\pi} _ {s}) - \nabla h (p _ {s}), \bar {\pi} _ {s} - \tilde {\pi} _ {s} \rangle + \langle \nabla h (\bar {\pi} _ {s}) - \nabla h (p _ {s}), \tilde {\pi} _ {s} - \pi_ {s} \rangle \\ \stackrel {(5 0)} {=} \mathcal {D} _ {h} (\tilde {\pi} _ {s}, \bar {\pi} _ {s}) - \mathcal {D} _ {h} (\tilde {\pi} _ {s}, p _ {s}) + \mathcal {D} _ {h} (\bar {\pi} _ {s}, p _ {s}) + \langle \nabla h (\bar {\pi} _ {s}) - \nabla h (p _ {s}), \tilde {\pi} _ {s} - \pi_ {s} \rangle . \\ \end{array}
$$
By rearranging terms, we have
$$
\left. \mathcal {D} _ {h} \left(\pi_ {s}, \bar {\pi} _ {s}\right) - \mathcal {D} _ {h} \left(\tilde {\pi} _ {s}, \bar {\pi} _ {s}\right) - \mathcal {D} _ {h} \left(\pi_ {s}, p _ {s}\right) + \mathcal {D} _ {h} \left(\tilde {\pi} _ {s}, p _ {s}\right) = \left\langle \nabla h \left(\bar {\pi} _ {s}\right) - \nabla h \left(p _ {s}\right), \tilde {\pi} _ {s} - \pi_ {s} \right\rangle . \right. \tag {51}
$$
From the Generalized Pythagorean Theorem of the Bregman divergence in Lemma D.2, also known as non-expansivity property, and from the fact that  $\tilde{\pi}_s = \mathrm{Proj}_{\Delta (\mathcal{A})}^h (p_s)$ , we have that
$$
\mathcal {D} _ {h} \left(\pi_ {s}, \tilde {\pi} _ {s}\right) + \mathcal {D} _ {h} \left(\tilde {\pi} _ {s}, p _ {s}\right) \leq \mathcal {D} _ {h} \left(\pi_ {s}, p _ {s}\right) \quad \Longleftrightarrow \quad - \mathcal {D} _ {h} \left(\pi_ {s}, p _ {s}\right) + \mathcal {D} _ {h} \left(\tilde {\pi} _ {s}, p _ {s}\right) \leq - \mathcal {D} _ {h} \left(\pi_ {s}, \tilde {\pi} _ {s}\right).
$$
Plugging the above inequality into the left hand side of (51) yields
$$
\left. \mathcal {D} _ {h} \left(\pi_ {s}, \bar {\pi} _ {s}\right) - \mathcal {D} _ {h} \left(\tilde {\pi} _ {s}, \bar {\pi} _ {s}\right) - \mathcal {D} _ {h} \left(\pi_ {s}, \tilde {\pi} _ {s}\right) \geq \langle \nabla h (\bar {\pi} _ {s}) - \nabla h (p _ {s}), \tilde {\pi} _ {s} - \pi_ {s} \rangle , \right.
$$
which concludes the proof with  $\nabla h(p_s) = \eta f_s^\theta$ .
We also provide an alternative proof of Lemma 4.1 later in Appendix F.
D.2 Bounding errors
In this section, we will bound error terms of the type
$$
\mathbb {E} _ {s \sim d _ {\mu} ^ {\pi}, a \sim \pi_ {s}} \left[ Q ^ {t} (s, a) + \eta_ {t} ^ {- 1} \left[ \nabla h \left(\pi_ {s} ^ {t}\right) \right] _ {a} - f ^ {t + 1} (s, a) \right], \tag {52}
$$
where  $(d_{\mu}^{\pi},\pi)\in \{(d_{\mu}^{\star},\pi^{\star}),(d_{\mu}^{t + 1},\pi^{t + 1}),(d_{\mu}^{\star},\pi^{t}),(d_{\mu}^{t + 1},\pi^{t})\}$ . These error terms appear in the forthcoming proofs of our results and directly induce the error floors in the convergence rates.
In the rest of Appendix D, let  $q^t: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  such that, for every  $s \in S$ ,
$$
d _ {s} ^ {t} := f _ {s} ^ {t + 1} - \eta_ {t} ^ {- 1} \nabla h \left(\pi_ {s} ^ {t}\right) \in \mathbb {R} ^ {| \mathcal {A} |}.
$$
So (52) can be rewritten as
$$
\mathbb {E} _ {s \sim d _ {\mu} ^ {\pi}, a \sim \pi_ {s}} \left[ Q ^ {t} (s, a) + \eta_ {t} ^ {- 1} \left[ \nabla h \left(\pi_ {s} ^ {t}\right) \right] _ {a} - f ^ {t + 1} (s, a) \right] = \mathbb {E} _ {s \sim d _ {\mu} ^ {\pi}, a \sim \pi_ {s}} \left[ Q ^ {t} (s, a) - q ^ {t} (s, a) \right]. \tag {53}
$$
To bound it, let  $(v^t)_{t\geq 0}$  be a sequence of distributions over states and actions. By using Cauchy-Schwartz's inequality, we have
$$
\begin{array}{l} \mathbb {E} _ {s \sim d _ {\mu} ^ {\pi}, a \sim \pi_ {s}} \left[ Q ^ {t} (s, a) - q ^ {t} (s, a) \right] \\ = \int_ {s \in \mathcal {S}, a \in \mathcal {A}} \frac {d _ {\mu} ^ {\pi} (s) \pi (a \mid s)}{\sqrt {v ^ {t} (s , a)}} \cdot \sqrt {v ^ {t} (s , a)} (Q ^ {t} (s, a) - q ^ {t} (s, a)) \\ \leq \sqrt {\int_ {s \in \mathcal {S} , a \in \mathcal {A}} \frac {\left(d _ {\mu} ^ {\pi} (s) \pi (a \mid s)\right) ^ {2}}{v ^ {t} (s , a)} \cdot \int_ {s \in \mathcal {S} , a \in \mathcal {A}} v ^ {t} (s , a) \left(Q ^ {t} (s , a) - q ^ {t} (s , a)\right) ^ {2}} \\ = \sqrt {\mathbb {E} _ {(s , a) \sim v ^ {t}} \left[ \left(\frac {d _ {\mu} ^ {\pi} (s) \pi (a \mid s)}{v ^ {t} (s , a)}\right) ^ {2} \right] \cdot \mathbb {E} _ {(s , a) \sim v ^ {t}} \left[ \left(Q ^ {t} (s , a) - q ^ {t} (s , a)\right) ^ {2} \right]} \\ \leq \sqrt {C _ {v} \mathbb {E} _ {(s , a) \sim v ^ {t}} \left[ (Q ^ {t} (s , a) - q ^ {t} (s , a)) ^ {2} \right]}, \\ \end{array}
$$
where the last line is obtained by Assumption (A2). Using the concavity of the square root and Assumption (A1), we have that
$$
\mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {\pi}, a \sim \pi_ {s}} \left[ Q ^ {t} (s, a) - q ^ {t} (s, a) \right] \right] \leq \sqrt {C _ {v} \varepsilon_ {\text {a p p r o x}}}. \tag {54}
$$
D.3 Quasi-monotonic updates - Proof of Proposition 4.2
In this section, we show Proposition 4.2 with its proof that the AMPO updates guarantee a quasi-monotonic property, i.e., a non-decreasing property up to a certain error floor due to the approximation error, which allows us to establish an important recursion about the AMPO iterates next. First, we recall the performance difference lemma [41] which is the second key tool for our analysis and a well known result in the RL literature. Here we use a particular form of the lemma presented by Xiao [93, Lemma 1].
Lemma D.4 (Performance difference lemma, Lemma 1 in [93]). For any policy  $\pi$ ,  $\pi' \in \Delta(\mathcal{A})^S$  and  $\mu \in \Delta(S)$ ,
$$
V ^ {\pi} (\mu) - V ^ {\pi^ {\prime}} (\mu) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\mu} ^ {\pi}} \left[ \left\langle Q _ {s} ^ {\pi^ {\prime}}, \pi_ {s} - \pi_ {s} ^ {\prime} \right\rangle \right].
$$
For clarity of exposition, we introduce the notation
$$
\tau := \frac {2 \sqrt {C _ {v} \varepsilon_ {\mathrm {a p p r o x}}}}{1 - \gamma}.
$$
Proposition 4.2 characterizes the non-decreasing property of AMPO. The error bound (54) in Appendix D.2 will be used to prove the result.
Proposition D.5 (Proposition 4.2). For the iterates of Algorithm 1, at each time  $t \geq 0$ , we have
$$
\mathbb {E} \left[ V ^ {t + 1} (\mu) - V ^ {t} (\mu) \right] \geq \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \frac {\mathcal {D} _ {h} \left(\pi_ {s} ^ {t + 1} , \pi_ {s} ^ {t}\right) + \mathcal {D} _ {h} \left(\pi_ {s} ^ {t} , \pi_ {s} ^ {t + 1}\right)}{\eta_ {t} (1 - \gamma)} \right] \right] - \tau .
$$
Proof. Using Lemma 4.1 with  $\bar{\pi} = \pi^t$ ,  $f^\theta = f^{t+1}$ ,  $\eta = \eta_t$ , thus  $\tilde{\pi} = \pi^{t+1}$  by Definition 3.1 and Algorithm 1, and  $\pi_s = \pi_s^t$ , we have
$$
\left\langle \eta_ {t} q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \right\rangle \leq \mathcal {D} _ {h} \left(\pi_ {s} ^ {t}, \pi_ {s} ^ {t}\right) - \mathcal {D} _ {h} \left(\pi_ {s} ^ {t + 1}, \pi_ {s} ^ {t}\right) - \mathcal {D} _ {h} \left(\pi_ {s} ^ {t}, \pi_ {s} ^ {t + 1}\right). \tag {55}
$$
By rearranging terms and noticing  $\mathcal{D}_h(\pi_s^t,\pi_s^t) = 0$ , we have
$$
\left\langle \eta_ {t} q _ {s} ^ {t}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \right\rangle \geq \mathcal {D} _ {h} \left(\pi_ {s} ^ {t + 1}, \pi_ {s} ^ {t}\right) + \mathcal {D} _ {h} \left(\pi_ {s} ^ {t}, \pi_ {s} ^ {t + 1}\right) \geq 0. \tag {56}
$$
Then, by the performance difference lemma D.4, we have
$$
\begin{array}{l} (1 - \gamma) \mathbb {E} \left[ V ^ {t + 1} (\mu) - V ^ {t} (\mu) \right] = \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \langle Q _ {s} ^ {t}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \rangle \right] \right] \\ { = } { \mathbb { E } \left[ \mathbb { E } _ { s \sim d _ { \mu } ^ { t + 1 } } \left[ \langle q _ { s } ^ { t } , \pi _ { s } ^ { t + 1 } - \pi _ { s } ^ { t } \rangle \right] \right] } \\ + \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \left\langle Q _ {s} ^ {t} - q _ {s} ^ {t}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \right\rangle \right] \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(5 5)} {\geq} \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \frac {\mathcal {D} _ {h} \left(\pi_ {s} ^ {t + 1} , \pi_ {s} ^ {t}\right) + \mathcal {D} _ {h} \left(\pi_ {s} ^ {t} , \pi_ {s} ^ {t + 1}\right)}{\eta_ {t}} \right] \right] \\ - \left| \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \left\langle Q _ {s} ^ {t} - q _ {s} ^ {t}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \right\rangle \right] \right] \right| \\ \geq \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \frac {\mathcal {D} _ {h} (\pi_ {s} ^ {t + 1} , \pi_ {s} ^ {t}) + \mathcal {D} _ {h} (\pi_ {s} ^ {t} , \pi_ {s} ^ {t + 1})}{\eta_ {t}} \right] \right] - \tau (1 - \gamma), \\ \end{array}
$$
which concludes the proof after dividing both sides by  $(1 - \gamma)$ . The last line follows from
$$
\begin{array}{l} \left. \right.\left| \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \langle Q _ {s} ^ {t} - q _ {s} ^ {t}, \pi_ {s} ^ {t + 1} - \pi_ {s} ^ {t} \rangle \right]\right]\right| \leq \left| \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}, a \sim \pi_ {s} ^ {t + 1}} \left[ Q ^ {t} (s, a) - q ^ {t} (s, a) \right]\right]\right| \tag {57} \\ + \left| \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}, a \sim \pi_ {s} ^ {t}} \left[ Q ^ {t} (s, a) - q ^ {t} (s, a) \right] \right] \right| \\ \end{array}
$$
$$
\stackrel {(5 4)} {\leq} \quad 2 \sqrt {C _ {1} \varepsilon_ {\text {e r r o r}}} = \tau (1 - \gamma), \tag {58}
$$
where both terms are upper bounded by  $\sqrt{C_v\varepsilon_{\mathrm{approx}}}$  through (54) with  $(d_{\mu}^{\pi},\pi) = (d_{\mu}^{t + 1},\pi^{t + 1})$  and  $(d_{\mu}^{\pi},\pi) = (d_{\mu}^{t + 1},\pi^{t})$ , respectively.
D.4 Main passage - An important recursion about the AMPO method
In this section, we show an important recursion result for the AMPO updates, which will be used for both the sublinear and the linear convergence analysis of AMPO.
For clarity of exposition in the rest of Appendix D, let
$$
\nu_ {t} := \left\| \frac {d _ {\mu} ^ {\star}}{d _ {\mu} ^ {t + 1}} \right\| _ {L _ {\infty}} := \sup _ {s \in \mathcal {S}} \frac {d _ {\mu} ^ {\star} (s)}{d _ {\mu} ^ {t + 1} (s)}.
$$
For two different time  $t, t' \geq 0$ , let  $\mathcal{D}_{t'}^t$  denote the expected Bregman divergence between the policy  $\pi^t$  and policy  $\pi^{t'}$ , where the expectation is taken over the discounted state visitation distribution of the optimal policy  $d_{\mu}^{\star}$ , that is,
$$
\mathcal {D} _ {t ^ {\prime}} ^ {t} := \mathbb {E} _ {s \sim d _ {\mu} ^ {*}} \left[ \mathcal {D} _ {h} \left(\pi_ {s} ^ {t}, \pi_ {s} ^ {t ^ {\prime}}\right) \right].
$$
Similarly, let  $\mathcal{D}_t^\star$  denote the expected Bregman divergence between the optimal policy  $\pi^{\star}$  and  $\pi^t$ , that is,
$$
\mathcal {D} _ {t} ^ {\star} := \mathbb {E} _ {s \sim d _ {\mu} ^ {\star}} \left[ \mathcal {D} _ {h} (\pi_ {s} ^ {\star}, \pi_ {s} ^ {t}) \right].
$$
Let  $\Delta_t \coloneqq V^\star(\mu) - V^t(\mu)$  be the optimality gap.
We can now state the following important recursion result for the AMPO method.
Proposition D.6 (Proposition 4.4). Consider the iterates of Algorithm 1, at each time  $t \geq 0$ , we have
$$
\mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {t + 1}}{(1 - \gamma) \eta_ {t}} + \nu_ {\mu} (\Delta_ {t + 1} - \Delta_ {t}) + \Delta_ {t} \right] \leq \mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {\star}}{(1 - \gamma) \eta_ {t}} - \frac {\mathcal {D} _ {t + 1} ^ {\star}}{(1 - \gamma) \eta_ {t}} \right] + (1 + \nu_ {\mu}) \tau .
$$
Proof. Using Lemma 4.1 with  $\bar{\pi} = \pi^t$ ,  $f^\theta = f^{t+1}$ ,  $\eta = \eta_t$ , and thus  $\tilde{\pi} = \pi^{t+1}$  by Definition 3.1 and Algorithm 1, and  $\pi_s = \pi_s^\star$ , we have that
$$
\langle \eta_ {t} q _ {s} ^ {t}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {t + 1} \rangle \leq \mathcal {D} _ {h} (\pi^ {\star}, \pi^ {t}) - \mathcal {D} _ {h} (\pi^ {\star}, \pi^ {t + 1}) - \mathcal {D} _ {h} (\pi^ {t + 1}, \pi^ {t}),
$$
which can be decomposed as
$$
\langle \eta_ {t} q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle + \langle \eta_ {t} q _ {s} ^ {t}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {t} \rangle \leq \mathcal {D} _ {h} (\pi^ {\star}, \pi^ {t}) - \mathcal {D} _ {h} (\pi^ {\star}, \pi^ {t + 1}) - \mathcal {D} _ {h} (\pi^ {t + 1}, \pi^ {t}).
$$
Taking expectation with respect to the distribution  $d_{\mu}^{\star}$  over states and with respect to the randomness of AMPO and dividing both sides by  $\eta_t$ , we have
$$
\mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {*}} \left[ \langle q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \right] \right] + \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {*}} \left[ \langle q _ {s} ^ {t}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {t} \rangle \right] \right] \leq \frac {1}{\eta_ {t}} \mathbb {E} [ \mathcal {D} _ {t} ^ {\star} - \mathcal {D} _ {t + 1} ^ {\star} - \mathcal {D} _ {t} ^ {t + 1} ]. \tag {59}
$$
We lower bound the two terms on the left hand side of (59) separately. For the first term, we have that