SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We highlight that all of the above sublinear convergence analyses, for both softmax tabular policies and the function approximation regime, are obtained either by using a decaying step-size or a constant step-size. Under these step-size schemes, our AMPO's  $\mathcal{O}(1 / T)$  sublinear convergence rate is the state of the art: it recovers the best-known convergence rates in the tabular setting [33, 93] without regularization; it improves the  $\mathcal{O}(1 / \sqrt{T})$  convergence rate of Zanette et al. [99] and Hu et al. [36] to  $\mathcal{O}(1 / T)$  for the linear MDP setting; it recovers the best-known convergence rates for the log-linear policies [98]; it matches the  $\mathcal{O}(1 / T)$  sublinear convergence rate for smooth and Fisher-non-degenerate policies [61] and the same convergence rate of Yang et al. [95] and Huang et al. [37] for Lipschitz and smooth policies without introducing variance reduction techniques; it matches the previous best-known convergence result in the neural network settings [16] without regularization; lastly, it goes beyond all these results by allowing general parameterization. We refer to Table 1 for an overview of recent sublinear convergence analyses of NPG/PMD.
Linear convergence analysis of PMD, NPG, NAC and other PG methods. In the softmax tabular policy settings, the linear convergence guarantees of NPG and PMD are achieved by either adding regularization [17, 100, 54, 58] or by varying the step-sizes [11, 46, 47, 93].
In the function approximation regime, the linear convergence guarantees are achieved for NPG with log-linear policies, either by adding entropy regularization [15] or by choosing geometrically increasing step-sizes [2, 98]. It can also be achieved for NAC with log-linear policy by using adaptive increasing step-sizes [20].
Again, our AMPO's linear convergence rate is the state of the art: not only it recovers the best-known convergence rates in both the tabular setting [93] and the log-linear policies [2, 98] without regularization [17, 100, 54, 58], nor adaptive step-sizes [11, 46, 47, 20], but also it achieves the new state-of-the-art linear convergence rate for PG-based methods with general parameterization, including the neural network parameterizations. We refer to Table 2 for an overview of recent linear convergence analyses of NPG/PMD.
Alternatively, by exploiting a Polyak-Lojasiewicz (PL) condition [76, 62], fast linear convergence results can be achieved for PG methods under different settings, such as linear quadratic control problems [31] and softmax tabular policies with entropy regularization [65, 97]. The PL condition
is extensively studied by Bhandari and Russo [10] to identify more general MDP settings. Like the cases of NPG and PMD, linear convergence of PG can also be obtained for the softmax tabular policy without regularization by choosing adaptive step sizes through exact line search [11] or by exploiting non-uniform smoothness [66]. When the PL condition is relaxed to other weaker conditions, PG methods combined with variance reduction methods such as SARAH [73] and PAGE [59] can also achieve linear convergence. This is shown by Fatkhullin et al. [29, 30] when the PL condition is replaced by the weak PL condition [97], which is satisfied by Fisher-non-degenerate policies [24]. It is also shown by Zhang et al. [102], where the MDP satisfies some hidden convexity property that contains a similar property to the weak PL condition studied by Zhang et al. [101]. Lastly, linear convergence is established for the cubic-regularized Newton method [71], a second-order method, applied to Fisher-non-degenerate policies combined with variance reduction [64].
Outside of the literature focusing on finite time convergence guarantees, Vaswani et al. [89] and Kuba et al. [51] provide a theoretical analysis for variations of PMD and show monotonic improvements for their frameworks. Additionally, Kuba et al. [51] give an infinite time convergence guarantee for their framework.
A.3 Future work
Our work opens several interesting research directions in both algorithmic and theoretical aspects.
From an algorithmic point of view, the updates in Lines 1 and 2 of AMPO are not explicit. This might be an issue in practice, especially for large scale RL problems. It would be interesting to design efficient regression solver for minimizing the approximation error in Line 1 of Algorithm 1. For instance, by using the dual averaging algorithm [7, Chapter 4], it could be possible to replace the term  $\nabla h(\pi_s^t)$  with  $f_{s}^{t}$  for all  $s\in S$ , to make the computation of the algorithm more efficient. That is, it could be interesting to consider the following variation of Line 1 in Algorithm 1:
$$
\left\| f ^ {t + 1} - Q ^ {t} - \frac {\eta_ {t - 1}}{\eta_ {t}} f ^ {t} \right\| _ {L _ {2} \left(v ^ {t}\right)} ^ {2} \leq \varepsilon_ {\text {a p p r o x}}. \tag {28}
$$
Notice that (28) has the same update as (17), however, (28) is not restricted to using the negative entropy mirror map. To efficiently solve the regression problem in Line 1 of Algorithm 1, one may want to apply modern variance reduction techniques [73, 22, 59]. This has been done by Liu et al. [61] for NPG method.
From a theoretical point of view, it would be interesting to derive a sample complexity analysis for AMPO in specific settings, by leveraging its linear convergence. As mentioned for the linear MDP [40] in Appendix A.2, one can apply the linear convergence theory of AMPO to other structural MDPs, e.g., block MDP [25], factored MDP [44, 85], RKHS linear MDP and RKHS linear mixture MDP [26], to build new sample complexity results for these settings, since the assumptions of Theorem 4.3 do not impose any constraint on the MDP. On the other hand, it would be interesting to explore the interaction between the Bregman projected policy class and the expected Lipschitz and smooth policies [97] and the Fish-non-degenerate policies [61] to establish new improved sample complexity results in these settings, again thanks to the linear convergence theory of AMPO.
Additionally, it would be interesting to study the application of AMPO to the offline setting. In the main text, we have discussed how to extend Algorithm 1 and Theorem 4.3 to the offline setting, where  $v^t$  can be set as the state-action distribution induced by an arbitrary behavior policy that generates the data. However, we believe that this direction requires further investigation. One of the major challenges of offline RL is dealing with the distribution shifts that stem from the mismatch between the trained policy  $\pi^t$  and the behaviour policy. Several methods have been introduced to deal with this issue, such as constraining the current policy to be close to the behavior policy [56]. We leave introducing offline RL techniques in AMPO as future work.
Another direction for future work is extending the policy update of AMPO to mirror descent algorithm based on value iteration and Bellman operators, such as MD-MPI [33], in order to extend existing results to the general parameterization setting. Other interesting settings that have been addressed using the PMD framework are mean-field games [96] and constrained MDPs [23]. We hope to build on the existing literature for these settings and see whether our results can bring any improvements.
Finally, this work theoretically indicates that, perhaps the most important future work of PMD-type algorithms is to design efficient policy evaluation algorithms to make the estimation of the  $Q$ -function
as accurate as possible, such as using offline data for training, and to construct adaptive representation learning for  $\mathcal{F}^{\Theta}$  to closely approximate  $Q$ -function, so that  $\epsilon_{\mathrm{approx}}$  is guaranteed to be small. This matches one of the most important research questions for deep Q-learning type algorithms for general policy optimization problems.
B Equivalence of (9)-(10) and (11) in the tabular case
To demonstrate the equivalence between the two-step update (9)-(10) and the one-step update (11) for policy mirror descent in the tabular case, it is sufficient to validate the following lemma, which comes from the optimization literature. The proof of this lemma can be found in Bubeck [14, Chapter 4.2]. However, for the sake of completeness, we present the proof here.
Lemma B.1 (Right after Theorem 4.2 in Bubeck [14]). Consider the mirror descent update in (5)-(6) for the minimization of a function  $V(\cdot)$ , that is,
$$
y ^ {t + 1} = \nabla h \left(x ^ {t}\right) - \eta_ {t} \nabla V (x) | _ {x = x ^ {t}}, \tag {29}
$$
$$
x ^ {t + 1} = \operatorname {P r o j} _ {\mathcal {X}} ^ {h} \left(\nabla h ^ {*} \left(y ^ {t + 1}\right)\right). \tag {30}
$$
Then the mirror descent update can be rewritten as
$$
x ^ {t + 1} \in \underset {x \in \mathcal {X}} {\operatorname {a r g m i n}} \eta_ {t} \left\langle x, \nabla V (x) \right| _ {x = x ^ {t}} \rangle + \mathcal {D} _ {h} \left(x, x ^ {t}\right). \tag {31}
$$
Proof. From definition of the Bregman projection step, starting from (29) we have
$$
\begin{array}{l} x ^ {t + 1} = \operatorname {P r o j} _ {\mathcal {X}} ^ {h} (\nabla h ^ {*} (y ^ {t + 1})) = \underset {x \in \mathcal {X}} {\operatorname {a r g m i n}} \mathcal {D} _ {h} (x, \nabla h ^ {*} (y ^ {t + 1})) \\ \in \underset {x \in \mathcal {X}} {\operatorname {a r g m i n}} \nabla h (x) - \nabla h (\nabla h ^ {*} (y ^ {t + 1})) - \left\langle \nabla h (\nabla h ^ {*} (y ^ {t + 1})), x - \nabla h ^ {*} (y ^ {t + 1}) \right\rangle \\ \begin{array}{l l} \stackrel {(4)} {\in} & \underset {x \in \mathcal {X}} {\operatorname * {a r g m i n}} \nabla h (x) - y ^ {t + 1} - \left\langle y ^ {t + 1}, x - \nabla h ^ {*} (y ^ {t + 1}) \right\rangle \end{array} \\ \in \underset {x \in \mathcal {X}} {\operatorname {a r g m i n}} \nabla h (x) - \langle x, y ^ {t + 1} \rangle \\ \begin{array}{l} \stackrel {(2 9)} {\in} \underset {x \in \mathcal {X}} {\operatorname {a r g m i n}} \nabla h (x) - \langle x, \nabla h (x ^ {t}) - \eta_ {t} \nabla V (x) | _ {x = x ^ {t}} \rangle \end{array} \\ \in \underset {x \in \mathcal {X}} {\operatorname {a r g m i n}} \eta_ {t} \left\langle x, \nabla V (x) | _ {x = x ^ {t}} \right\rangle + \nabla h (x) - \nabla h (x ^ {t}) - \left\langle \nabla h (x ^ {t}), x - x ^ {t} \right\rangle \\ \in \underset {x \in \mathcal {X}} {\operatorname {a r g m i n}} \eta_ {t} \langle x, \nabla V (x) | _ {x = x ^ {t}} \rangle + \mathcal {D} _ {h} (x, x ^ {t}), \\ \end{array}
$$
where the second and the last lines are both obtained by the definition of the Bregman divergence.
The one-step update in (31) is often taken as the definition of mirror descent [8], which provides a proximal view point of mirror descent, i.e., a gradient step in the primal space with a regularization of Bregman divergence.
C AMPO for specific mirror maps
In this section, we give the derivations for Example 3.2, which is based on the Karush-Kuhn-Tucker (KKT) conditions [43, 52], and then provide details about the  $\omega$ -potential mirror map class from Section 3.1.
C.1 Derivation of Example 3.2
We give here the derivation of Example 3.2. Let  $h$  be the negative entropy mirror map, that is
$$
h \left(\pi_ {s}\right) = \sum_ {a \in \mathcal {A}} \pi (a \mid s) \log \left(\pi (a \mid s)\right), \quad \forall \pi_ {s} \in \Delta (\mathcal {A}) \text {a n d} \forall s \in \mathcal {S}.
$$
For every state  $s \in S$ , we solve the minimization problem
$$
\pi_{s}^{\theta}\in \operatorname *{argmin}_{\pi_{s}\in \Delta (\mathcal{A})}\mathcal{D}_{h}(\pi_{s},\nabla h^{*}(\eta f_{s}^{\theta}))
$$
through the KKT conditions. We formalize it as
$$
\pi_ {s} ^ {\theta} \in \operatorname * {a r g m i n} _ {\pi_ {s} \in \mathbb {R} ^ {| A |}} \mathcal {D} _ {h} (\pi_ {s}, \nabla h ^ {*} (\eta f _ {s} ^ {\theta}))
$$
subject to  $\langle \pi_s, \mathbf{1} \rangle = 1$
$$
\pi (a \mid s) \geq 0, \quad \forall a \in \mathcal {A},
$$
where  $\mathbf{1}$  denotes a vector in  $\mathbb{R}^{|A|}$  with coordinates equal to 1 element-wisely. The conditions then become
$$
\text {(s t a t i o n a r i t y)} \quad \log \left(\pi_ {s} ^ {\theta}\right) - \eta f _ {s} ^ {\theta} + \left(\lambda_ {s} + 1\right) \mathbf {1} - c _ {s} = 0,
$$
(complementary slackness)  $c_{s}^{a}\pi^{\theta}(a\mid s) = 0,\quad \forall a\in \mathcal{A},$
$$
\text {(p r i m a l f e a s i b i l i t y)} \quad \langle \pi_ {s} ^ {\theta}, \mathbf {1} \rangle = 1, \pi^ {\theta} (a \mid s) \geq 0, \quad \forall a \in \mathcal {A},
$$
$$
\text {(d u a l f e a s i b i l i t y)} \quad c _ {s} ^ {a} \geq 0, \quad \forall a \in \mathcal {A},
$$
where  $\log (\pi_s)$  is applied element-wisely,  $\lambda_s\in \mathbb{R}$  and  $c_{s}^{a}\in \mathbb{R}$  are the dual variables, and  $c_{s}$  devotes the vector  $[c_s^a ]_{a\in \mathcal{A}}$  . It is easy to verify that the solution
$$
\pi_ {s} ^ {\theta} = \frac {\exp (\eta f _ {s} ^ {\theta})}{\| \exp (\eta f _ {s} ^ {\theta}) \| _ {1}},
$$
with  $\lambda_s = \log \left( \sum_{a \in \mathcal{A}} \exp (\eta f^\theta(s, a)) \right) - 1$  and  $c_s^a = 0$  for all  $a \in \mathcal{A}$ , satisfies all the conditions.
When  $f^{\theta}(s,a) = \theta_{s,a}$  we obtain the tabular softmax policy  $\pi^{\theta}(a\mid s)\propto \exp (\eta \theta_{s,a})$ . When  $f^{\theta}(s,a) = \theta^{\top}\phi (s,a)$  is a linear function, for  $\theta \in \mathbb{R}^d$  and for a feature function  $\phi :\mathcal{S}\times \mathcal{A}\to \mathbb{R}^{d}$ , we obtain the log-linear policy  $\pi^{\theta}(a\mid s)\propto \exp (\eta \theta^{\top}\phi (s,a))$ . When  $f^{\theta}:\mathcal{S}\times \mathcal{A}\rightarrow \mathbb{R}$  is a neural network, we obtain the softmax neural policy  $\pi^{\theta}(a\mid s)\propto \exp (\eta f^{\theta}(s,a))$ .
C.2 More on  $\omega$ -potential mirror maps
In this section, we provide details about the  $\omega$ -potential mirror map class from Section 3.1, including the derivation of (14), several instantiations of  $\omega$ -potential mirror map mentioned in Section 3.1 with their derivations, and an iterative algorithm to find approximately the Bregman projection induced by  $\omega$ -potential mirror map when an exact solution is not available.
We give a different but equivalent formulation of Proposition 2 of Krichene et al. [50].
Proposition C.1. For  $u \in (-\infty, +\infty]$  and  $\omega \leq 0$ , an increasing  $C^1$ -diffeomorphism  $\phi : (-\infty, u) \to (\omega, +\infty)$  is called an  $\omega$ -potential if
Let the mirror map  $h_{\phi}$  be defined as
We have that  $\pi_s^t$  is a solution to the Bregman projection
$$
\min _ {\pi \in \Delta_ {s}} P r o j _ {\Delta (\mathcal {A})} ^ {h} \bigl (\nabla h _ {\phi} ^ {*} (\eta_ {t - 1} f _ {s} ^ {t}) \bigr),
$$
if and only if there exist a normalization constant  $\lambda_s^t\in \mathbb{R}$  such that
$$
\pi^ {t} (a \mid s) = \sigma \left(\phi \left(\eta_ {t - 1} f ^ {t} (s, a) + \lambda_ {s} ^ {t}\right)\right), \quad \forall a \in \mathcal {A}, \tag {32}
$$
and  $\sum_{a\in \mathcal{A}}\pi^t (a\mid s) = 1$  , where for all  $s\in S$  and  $\sigma (z) = \max (z,0)$  for  $z\in \mathbb{R}$
We can now use Proposition C.1 to derive (14).
Consider an  $\omega$ -potential mirror map  $h_\phi$  associated with an  $\omega$ -potential  $\phi$ . By definition, we have
$$
\nabla h _ {\phi} \left(\pi_ {s} ^ {t}\right) = \left[ \phi^ {- 1} \left(\pi^ {t} (a \mid s)\right) \right] _ {a \in \mathcal {A}}. \tag {33}
$$
Plugging (32) into (33), we have
$$
\begin{array}{l} \nabla h _ {\phi} \left(\pi_ {s} ^ {t}\right) \stackrel {(3 3)} {=} \left[ \phi^ {- 1} \left(\pi^ {t} (a \mid s)\right) \right] _ {a \in \mathcal {A}} \\ \stackrel {(3 2)} {=} \left[ \phi^ {- 1} \left(\sigma \left(\phi \left(\eta_ {t - 1} f ^ {t} (s, a) + \lambda_ {s} ^ {t}\right)\right)\right) \right] _ {a \in \mathcal {A}} \\ = \left[ \max  \left(\phi^ {- 1} \left(\phi \left(\eta_ {t - 1} f ^ {t} (s, a) + \lambda_ {s} ^ {t}\right)\right), \phi^ {- 1} (0)\right) \right] _ {a \in A} \\ = \left[ \max  \left(\eta_ {t - 1} f ^ {t} (s, a) + \lambda_ {s} ^ {t}, \phi^ {- 1} (0)\right) \right] _ {a \in \mathcal {A}} \\ = \left[ \max  \left(\eta_ {t - 1} f ^ {t} (s, a), \phi^ {- 1} (0) - \lambda_ {s} ^ {t}\right) \right] _ {a \in \mathcal {A}} + \lambda_ {s} ^ {t}, \\ \end{array}
$$
where the third line is obtained by using the increasing property of  $\phi^{-1}$ , as  $\phi$  is increasing. Finally, plugging the above expression of  $\nabla h_{\phi}(\pi_s^t)$  into Line 1, we obtain (14), which is
$$
\theta^ {t + 1} \in \operatorname * {a r g m i n} _ {\theta \in \Theta} \mathbb {E} _ {(s, a) \sim v ^ {t}} \left[ (f ^ {\theta} (s, a) - Q ^ {t} (s, a) - \eta_ {t} ^ {- 1} \max  (\eta_ {t - 1} f ^ {t} (s, a), \phi^ {- 1} (0) - \lambda_ {s} ^ {t})) ^ {2} \right],
$$
where the term  $\lambda_s^t$  is dropped, as it is constant over actions and does not affect the resulting policy.
Once (14) is obtained, we can instantiate AMPO for mirror maps belonging to this class. We highlight that due to the definition of the Bregman divergence, two mirror maps that only differ for a constant term are equivalent and generate the same algorithm. We start with the negative entropy, which leads to a closed solution for  $\lambda_s^t$  and therefore for the Bregman projection.
Negative entropy. Let  $\phi(x) = \exp(x - 1)$ , which is an  $\omega$ -potential with  $\omega = 0$ ,  $u = +\infty$ , and
$$
\int_ {0} ^ {1} \phi^ {- 1} (x) d x = \int_ {0} ^ {1} \log (x) + 1 d x = [ x \log (x) ] _ {0} ^ {1} = 0 \leq + \infty .
$$
The mirror map  $h_{\phi}$  becomes the negative entropy, as
$$
h _ {\phi} (\pi_ {s}) = \sum_ {a \in \mathcal {A}} \int_ {1} ^ {\pi (a | s)} (\log (x) + 1) d x = \sum_ {a \in \mathcal {A}} \pi (a \mid s) \log (\pi (a \mid s)),
$$
and the associated Bregman divergence becomes the KL divergence, i.e.,  $D_{h_{\phi}}(\pi_s,\bar{\pi}_s) = \mathrm{KL}(\pi_s,\bar{\pi}_s)$ . Equation (17) follows from Equation (14) by the fact that
$$
\phi^ {- 1} (0) = \log (0) + 1 = - \infty ,
$$
which means that  $\max (\eta_{t - 1}f^t,\phi^{-1}(0) - \lambda_s^t) = \eta_{t - 1}f^t$  element-wisely.
As we showed in Appendix C.1, the Bregman projection (Line 2 of Algorithm 1) for the negative entropy has a closed form which is  $\pi_s^{t + 1}\propto \exp (\eta_t f_s^{t + 1})$
In NPG with softmax tabular policies [1, 93], at time  $t$ , the updates for the policies have
$$
\pi_ {s} ^ {t + 1} \propto \pi_ {s} ^ {t} \odot \exp \left(\eta_ {t} Q _ {s} ^ {t}\right), \quad \forall s \in \mathcal {S}. \tag {34}
$$
When considering AMPO with  $f^{\theta}(s,a) = \theta_{s,a} \in \mathbb{R}$ , from (17), we obtain that for all  $s, a \in S, \mathcal{A}$ , we have
$$
\begin{array}{l} \theta^ {t + 1} \in \operatorname * {a r g m i n} _ {\theta \in \mathbb {R} ^ {| S | \times | A |}} \left\| \theta - Q ^ {t} - \frac {\eta_ {t - 1}}{\eta_ {t}} \theta^ {t} \right\| _ {L _ {2} (v ^ {t})} ^ {2} \Longleftrightarrow f ^ {t + 1} (s, a) = Q ^ {t} (s, a) + \frac {\eta_ {t - 1}}{\eta_ {t}} f ^ {t} (s, a) \\ \Longleftrightarrow \eta_ {t} f ^ {t + 1} (s, a) = \eta_ {t} Q ^ {t} (s, a) + \eta_ {t - 1} f ^ {t} (s, a). \\ \end{array}
$$
With the above expression, we have the AMPO's updates for the policies rewritten as
$$
\begin{array}{l} \pi_ {s} ^ {t + 1} \propto \exp (\eta_ {t} f _ {s} ^ {t + 1}) \\ = \exp \left(\eta_ {t} Q _ {s} ^ {t} + \eta_ {t - 1} f _ {s} ^ {t}\right) \\ \propto \pi_ {s} ^ {t} \odot \exp (\eta_ {t} Q _ {s} ^ {t}), \quad \forall s \in \mathcal {S}, \\ \end{array}
$$
which recovers (34). In particular, the summation in the second line is element-wise and the third line is obtained because of  $\pi_s^t\propto \exp (\eta_{t - 1}f_s^t)$ , as shown in Appendix C.1.
In Q-NPG with log-linear policies [1, 98, 2], at time  $t$ , the updates for the policies have
$$
\pi^ {t + 1} (a \mid s) \propto \pi^ {t} (a \mid s) \exp \left(\eta_ {t} \phi (s, a) ^ {\top} w ^ {t}\right), \tag {35}
$$
where  $\phi :S\times \mathcal{A}\to \mathbb{R}^d$  is a feature map, and
$$
w ^ {t} \in \underset {w \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \mathbb {E} _ {(s, a) \sim d _ {\mu} ^ {t}} \left[ \left(Q ^ {t} (s, a) - \phi (s, a) ^ {\top} w\right) ^ {2} \right]. \tag {36}
$$
Like in the tabular case, when considering AMPO with  $\theta \in \mathbb{R}^d$ ,  $f^\theta(s,a) = \phi(s,a)^\top \theta$  and  $v^t = d_\mu^t$ , from (17), we obtain that for all  $s,a \in S, A$ , we have
$$
\begin{array}{l} \theta^ {t + 1} \in \operatorname * {a r g m i n} _ {\theta \in \mathbb {R} ^ {d}} \mathbb {E} _ {(s, a) \sim d _ {\mu} ^ {t}} \left[ \left(\phi (s, a) ^ {\top} \theta - Q ^ {t} (s, a) - \frac {\eta_ {t - 1}}{\eta_ {t}} \phi (s, a) ^ {\top} \theta^ {t}\right) ^ {2} \right] \\ \in \operatorname *{argmin}_{\theta \in \mathbb{R}^{d}}\mathbb{E}_{(s,a)\sim d_{\mu}^{t}}\Bigg[\Bigg(Q^{t}(s,a) - \phi (s,a)^{\top}\Big(\underbrace{\theta - \frac{\eta_{t - 1}}{\eta_{t}}\theta^{t}}_{w}\Big)\Bigg)^{2}\Bigg]. \\ \end{array}
$$
Compared the above form with (36), we obtain that
$$
w ^ {t} = \theta^ {t + 1} - \frac {\eta_ {t - 1}}{\eta_ {t}} \theta^ {t} \quad \Longleftrightarrow \quad \eta_ {t} \theta^ {t + 1} = \eta_ {t} w ^ {t} + \eta_ {t - 1} \theta^ {t}. \tag {37}
$$
Algorithm 2: Bregman projection for  $\epsilon$ -negative entropy
Input: vector to project  $x\in \mathbb{R}^{|A|}$  , parameter  $\epsilon$
1 Initialize  $y = \exp (x)$  element-wisely.
2 Let  $y^{(i)}$  be the  $i$ -th smallest element of  $y$ .
3 Let  $i^{\star}$  be the smallest index for which
$$
(1 + \epsilon (| \mathcal {A} | - i + 1)) y ^ {(i)} - \epsilon \sum_ {j \geq i} y ^ {(j)} > 0.
$$
Return: the projected vector  $(\sigma (-\epsilon +y_{a} / \lambda))_{a\in \mathcal{A}}$
So, the AMPO's updates for the policies can be rewritten as
$$
\begin{array}{l} \pi^ {t + 1} (a \mid s) \propto \exp (\eta_ {t} \phi (s, a) ^ {\top} \theta^ {t + 1}) \\ \stackrel {(3 7)} {=} \exp (\phi (s, a) ^ {\top} (\eta_ {t} w ^ {t} + \eta_ {t - 1} \theta^ {t})) \\ \propto \quad \pi^ {t} (a \mid s) \exp (\eta_ {t} \phi (s, a) ^ {\top} w ^ {t}), \\ \end{array}
$$
where the last line is obtained because of  $\pi_s^t\propto \exp (\eta_{t - 1}f_s^t)$ , as shown in Appendix C.1, and we recover (35).
We next present the squared  $\ell_2$ -norm and the  $\epsilon$ -negative entropy. For these two mirror maps, the Bregman projection can be computed exactly but has no closed form.
Squared  $\ell_2$ -norm. Let  $\phi$  be the identity function. The mirror map  $h_\phi$  becomes the squared  $\ell_2$ -norm, up to a constant term, as
$$
h _ {\phi} (\pi_ {s}) = \sum_ {a \in \mathcal {A}} \int_ {1} ^ {\pi (a | s)} x d x = \frac {1}{2} \sum_ {a \in \mathcal {A}} \left(\pi (a \mid s) ^ {2} - 1\right).
$$
The associated Bregman divergence becomes the squared Euclidean distance, i.e.,  $D_{h_{\phi}}(\pi_s,\bar{\pi}_s) = \frac{1}{2}\| \pi_s - \bar{\pi}_s\| _2^2$ , and  $\nabla h^{*}(\cdot)$  is the identity function. The update in (15) follows immediately and the Bregman projection step with the Euclidean distance becomes, for all  $s\in S$
$$
\pi_ {s} ^ {t + 1} = \operatorname {P r o j} _ {\Delta (\mathcal {A})} ^ {h _ {\phi}} \left(\nabla h ^ {*} \left(\eta_ {t} f _ {s} ^ {t + 1}\right)\right) = \operatorname {P r o j} _ {\Delta (\mathcal {A})} ^ {l _ {2}} \left(\eta_ {t} f _ {s} ^ {t + 1}\right) = \underset {p \in \Delta (\mathcal {A})} {\operatorname {a r g m i n}} \left\| p - \eta_ {t} f _ {s} ^ {t + 1} \right\| _ {2} ^ {2}. \tag {38}
$$
In the projected-Q descent for tabular policies developed by Xiao [93], at time  $t$ , the updates for the policies are
$$
\pi_ {s} ^ {t + 1} \in \underset {p \in \Delta (\mathcal {A})} {\operatorname {a r g m i n}} \left\| \pi_ {s} ^ {t} + \eta_ {t} Q _ {s} ^ {t} - p \right\| _ {2} ^ {2}, \quad \forall s \in \mathcal {S}. \tag {39}
$$
When considering AMPO with  $f^{\theta}(s,a) = \theta_{s,a}$  and  $\Theta = \mathbb{R}^{|S|\times |\mathcal{A}|}$ , (15) is solved with
$$
f ^ {t + 1} (s, a) = \theta_ {s, a} ^ {t + 1} = Q ^ {t} (s, a) + \eta_ {t} ^ {- 1} \pi^ {t} (a \mid s).
$$
Plugging the above expression into (38), we recover (39).
Notice that the Euclidean projection onto the probability simplex can be obtained exactly, as shown by Wang and Carreira-Perpinan [91].
$\epsilon$ -negative entropy [50]. Let  $\epsilon \geq 0$  and define the  $\epsilon$ -exponential potential as  $\phi(x) = \exp(x - 1) - \epsilon$ . The mirror map  $h_{\phi}$  becomes