SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where GradientAscent  $(\ell_t(\theta),\theta_t,m_a)$  means running GradientAscent for  $m_{a}$  iterations on  $\ell_t(\theta)$  with the initialization equal to  $\theta_{t}$ . Since we assume that  $\ell_t$  satisfies the  $PL$ -condition w.r.t.  $\theta$  for all  $t$ , based on the results from Karimi et al. [27], we get
$$
\ell_ {t} (\bar {\theta} _ {t + 1}) - \ell_ {t} (\theta_ {t + 1}) \leq \underbrace {c _ {1} \exp (- c _ {2} m _ {a}) \left(\ell_ {t} (\bar {\theta} _ {t + 1}) - \ell_ {t} (\theta_ {t})\right)} _ {:= e _ {t}}
$$
where  $c_{1}, c_{2}$  are problem-dependent constants related to the smoothness and curvature of  $\ell_t$ , and  $e_t$  is the approximation error diminishes as we increase the value of  $m_a$ . Following the same steps as before,
$$
\begin{array}{l} J \left(\pi_ {t + 1}\right) \geq J \left(\pi_ {t}\right) + \hat {g} \left(\pi_ {t}\right) ^ {\top} \left(\pi \left(\theta_ {t + 1}\right) - \pi_ {t}\right) - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} \left(\pi \left(\theta_ {t + 1}\right), \pi_ {t}\right) - \epsilon_ {t} \tag {UsingProp.1} \\ \geq J \left(\pi_ {t}\right) + \hat {g} \left(\pi_ {t}\right) ^ {\top} \left(\pi \left(\bar {\theta} _ {t + 1}\right) - \pi_ {t}\right) - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} \left(\pi \left(\bar {\theta} _ {t + 1}\right), \pi_ {t}\right) - \epsilon_ {t} \quad (\text {U s i n g t h e a b o v e b o u n d f o r G A}) \\ = J (\pi_ {t}) + \hat {g} (\pi_ {t}) ^ {\top} (\bar {\pi} _ {t + 1} - \pi_ {t}) - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} (\bar {\pi} _ {t + 1}, \pi_ {t}) - e _ {t} - \epsilon_ {t} \\ \geq J (\pi_ {t}) + \frac {1}{\eta_ {t} ^ {\prime}} \left\langle \nabla \Phi \left(\tilde {\pi} _ {t + 1}\right) - \nabla \Phi \left(\pi_ {t}\right), \bar {\pi} _ {t + 1} - \pi_ {t} \right\rangle - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \pi_ {t}\right) - \epsilon_ {t} ^ {c} - e _ {t} \quad (U s i n g t h e M A u p d a t e) \\ \geq J (\pi_ {t}) + \frac {1}{\eta_ {t} ^ {\prime}} \left\{D _ {\Phi} (\bar {\pi} _ {t + 1}, \pi_ {t}) + D _ {\Phi} (\pi_ {t}, \tilde {\pi} _ {t + 1}) - D _ {\Phi} (\bar {\pi} _ {t + 1}, \tilde {\pi} _ {t + 1}) \right\} - \left(\frac {1}{\eta} + \frac {1}{c}\right) D _ {\Phi} (\bar {\pi} _ {t + 1}, \pi_ {t}) - \epsilon_ {t} ^ {c} - e _ {t} \\ \end{array}
$$
(using Lemma 4)
$$
\begin{array}{l} = J(\pi_{t}) + \frac{1}{\eta^{\prime}_{t}}\underbrace{\left\{D_{\Phi}(\pi_{t},\tilde{\pi}_{t + 1}) - D_{\Phi}(\bar{\pi}_{t + 1},\tilde{\pi}_{t + 1})\right\}}_{:= A} + \left(\frac{1}{\eta^{\prime}_{t}} -\frac{1}{\eta} -\frac{1}{c}\right)D_{\Phi}(\bar{\pi}_{t + 1},\pi_{t}) - \epsilon^{c}_{t} - e_{t} \\ \geq J (\pi_ {t}) + \left(\frac {1}{\eta_ {t} ^ {\prime}} - \frac {1}{\eta} - \frac {1}{c}\right) D _ {\Phi} \left(\bar {\pi} _ {t + 1}, \pi_ {t}\right) - \epsilon_ {t} ^ {c} - e _ {t} (A \geq 0 \text {s i n c e} \bar {\pi} _ {t + 1} \text {i s t h e p r o j e c t i o n o f} \tilde {\pi} _ {t + 1} \text {i n t o t h e s i m p l e x}) \\ \geq J (\pi_ {t}) + \underbrace {\left(\frac {1}{\eta} + \frac {1}{c}\right)} _ {:= \frac {1}{\xi}} D _ {\Phi} (\bar {\pi} _ {t + 1}, \pi_ {t}) - \epsilon_ {t} ^ {c} - e _ {t} \quad \text {(s e t t i n g} \eta_ {t} ^ {\prime} \text {s . t .} 1 / \eta_ {t} ^ {\prime} \geq 2 / \eta + 2 / c) \\ \end{array}
$$
Recusing for  $T$  iterations and dividing by  $1 / \zeta$ , picking  $\mathcal{R}$  uniformly random from  $\{0, 1, 2, \ldots, T - 1\}$  and taking expectation we get
$$
\begin{array}{l} \mathbb {E} \frac {D _ {\Phi} (\bar {\pi} _ {\mathcal {R} + 1} , \pi_ {\mathcal {R}})}{\zeta^ {2}} = \frac {1}{\zeta^ {2} T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} D _ {\Phi} (\bar {\pi} _ {t + 1}, \pi_ {t}) \\ \leq \frac {\mathbb {E} [ J (\pi_ {T}) - J (\pi_ {0}) ]}{\zeta T} + \frac {\sum_ {t = 0} ^ {T - 1} \mathbb {E} \epsilon_ {t} ^ {c}}{\zeta T} + \frac {\sum_ {t = 0} ^ {T - 1} \mathbb {E} e _ {t}}{\zeta T} \\ \leq \frac {\left[ J \left(\pi^ {*}\right) - J \left(\pi_ {0}\right) \right]}{\gamma T} + \frac {\sum_ {t = 0} ^ {T - 1} \mathbb {E} \epsilon_ {t} ^ {c}}{\gamma T} + \frac {\sum_ {t = 0} ^ {T - 1} \mathbb {E} e _ {t}}{\gamma T} \\ \Longrightarrow \mathbb {E} \frac {D _ {\Phi} (\bar {\pi} _ {\mathcal {R} + 1} , \pi_ {\mathcal {R}})}{\zeta^ {2}} \leq \frac {1}{\zeta T} \left[ [ J (\pi^ {*}) - J (\pi_ {0}) ] + \frac {1}{c} \sum_ {t = 0} ^ {T - 1} \mathbb {E} D _ {\Phi^ {*}} \left(\nabla \Phi (\pi_ {t}) - c [ \nabla J (\pi_ {t}) - \hat {g} (\pi_ {t}) ], \nabla \Phi (\pi_ {t})\right) + \sum_ {t = 0} ^ {T - 1} \mathbb {E} e _ {t} \right] \\ \end{array}
$$
Note that it is possible to incorporate a sampling error (in the distribution  $d^{\pi}$  across states) for the actor update in Algorithm 1. This corresponds to an additional error in calculating  $D_{\Phi}$ , and we can use the techniques from Lavington et al. [31] to characterize the convergence in this case.
D.2 Exact setting with Lifting (Direct representation)
Recall the mirror ascent update in the functional space.
$$
\bar {\pi} _ {t + 1} = \underset {\pi \in \Pi} {\arg \max } \left[ J (\pi_ {t}) + \nabla J (\pi_ {t}) ^ {\top} (\pi - \pi_ {t}) - \frac {1}{\eta_ {t} ^ {\prime}} D _ {\Phi} (\pi , \pi_ {t}) \right],
$$
For the direct representation, we define  $\pi^s := p^\pi(\cdot|s)$ ,  $\hat{g}(\pi_t)(s,\cdot) = d^{\pi_t}(s)Q^{\pi_t}(s,\cdot)$  and  $D_{\Phi}(\pi,\pi_t) = \sum_s d^{\pi_t}(s)D_\phi(\pi^s,\pi_t^s)$ . Rewriting the MA update,
$$
\begin{array}{l} \bar {\pi} _ {t + 1} = \underset {\{\pi^ {s} \in \Delta_ {A} \} _ {s \in S}} {\arg \max } \left[ J (\pi_ {t}) + \sum_ {s} d ^ {\pi_ {t}} (s) \left[ \langle Q ^ {\pi_ {t}} (s, \cdot), \pi^ {s} - \pi_ {t} ^ {s} \rangle - \frac {1}{\eta_ {t} ^ {\prime}} D _ {\phi} (\pi^ {s}, \pi_ {t} ^ {s}) \right] \right] \\ \Rightarrow \bar {\pi} _ {t + 1} ^ {s} = \underset {\pi^ {s} \in \Delta_ {A}} {\arg \max } \left[ \left\langle Q ^ {\pi_ {t}} (s, \cdot), \pi^ {s} - \pi_ {t} ^ {s} \right\rangle - \frac {1}{\eta_ {t} ^ {\prime}} D _ {\phi} \left(\pi^ {s}, \pi_ {t} ^ {s}\right) \right] \quad (\text {C a n d e c o m p o s e a c r o s s s t a t e s s i n c e} d ^ {\pi_ {t}} (s) \geq 0) \\ \end{array}
$$
For each state  $s$  and  $Q^{\pi_t}(s,.)$  we define the set  $\Pi_t^s = \{\pi^s:\pi^s\in \arg \max_{p^s\in \Delta_A}\langle Q^{\pi_t}(s,.,),p^s\rangle \}$  i.e. a set of greedy policies w.r.t.  $Q^{\pi_t}(s,.)$ . Similar to Johnson et al. [24] we define  $\eta_t^\prime$  as follows
$$
\eta_ {t} ^ {\prime} \geq \frac {1}{c _ {t}} \max  _ {s} \left\{\min  _ {\pi \in \Pi_ {t} ^ {s}} D _ {\Phi} (\pi , \pi_ {t}) \right\} \tag {5}
$$
where  $c_{t} > 0$  is a constant. Now we consider the policy parameterization,  $\pi = \pi(\theta)$ . We assume that the mapping from  $\theta \rightarrow \pi$  is  $L_{\pi}$  Lipschitz continuous.
$$
\begin{array}{l} \ell_ {t} (\theta) := J \left(\pi_ {t}\right) + \sum_ {s} d ^ {\pi_ {t}} (s) \left[ \left\langle Q ^ {\pi_ {t}} (s, \cdot), \pi (\theta) ^ {s} - \pi_ {t} ^ {s} \right\rangle - \frac {1}{\eta_ {t} ^ {\prime}} D _ {\phi} \left(\pi (\theta) ^ {s}, \pi_ {t} ^ {s}\right) \right] \\ \tilde {\theta} _ {t + 1} := \underset {\theta} {\arg \max} \ell_ {t} (\theta) \\ \theta_ {t + 1} := \operatorname {G r a d i e n t A s c e n t} \left(\ell_ {t} (\theta), \theta_ {t}, m\right) \\ \tilde {\pi} _ {t + 1} := \pi (\tilde {\theta} _ {t + 1}) \\ \pi_ {t + 1} := \pi (\theta_ {t + 1}) \\ \end{array}
$$
GradientAscent  $(\ell_t(\theta),\theta_t,m)$  means that we run gradient ascent for  $m$  iterations to maximize  $\ell_t$  with  $\theta_{t}$  as the initial value. We assume that  $\ell_t$  satisfies Restricted Secant Inequality (RSI) and is smooth w.r.t.  $\theta$ . Based on the convergence property of Gradient Ascent for RSI and smooth functions [27], we have:
$$
\begin{array}{l} \left\| \tilde {\theta} _ {t + 1} - \theta_ {t + 1} \right\| _ {2} ^ {2} \leq \mathcal {O} (\exp (- m)) \\ \Rightarrow \| \tilde {\pi} _ {t + 1} - \pi_ {t + 1} \| _ {2} ^ {2} = \left\| \pi (\tilde {\theta} _ {t + 1}) - \pi (\theta_ {t + 1}) \right\| _ {2} ^ {2} \leq L _ {\pi} ^ {2} \left\| \tilde {\theta} _ {t + 1} - \theta_ {t + 1} \right\| _ {2} ^ {2} \leq \underbrace {e _ {t} ^ {2} := \mathcal {O} (\exp (- m))} _ {\text {a p p r o x i m a t i o n e r r o r}} \\ \end{array}
$$
(since  $\pi (\theta)$  is Lipschitz continuous)
Furthermore, we assume that  $\| \bar{\pi}_{t + 1} - \tilde{\pi}_{t + 1}\| _2^2\leq b_t^2$  for all  $t$  which represents the bias because of the function approximation. Before stating the main proposition of this section, we restate Johnson et al. [24, Lemma 2].
Lemma 3 (Lemma 2 of Johnson et al. [24]). For all  $(s, a) \in S \times \mathcal{A}$  we have
$$
Q ^ {\bar {\pi} _ {t + 1}} (s, a) \geq Q ^ {\pi_ {t}} (s, a).
$$
Now we state the main proposition of this part.
Proposition 10 (Convergence of tabular MDP with Lifting). Assume that (i)  $\ell_t(\theta)$  is smooth and satisfies RSI condition, (ii)  $\pi(\theta)$  is  $L_{\pi}$ -Lipschitz continuous, (iii) the bias is bounded for all  $t$  i.e.  $\|\bar{\pi}_{t+1} - \tilde{\pi}_{t+1}\|_2^2 \leq b_t^2$ , (iv)  $\|Q^\pi(s, \cdot)\| \leq q$  for all  $\pi$  and  $s$ . By setting  $\eta_t'$  as in Eq. (5) and running Gradient Ascent for  $m$  iterations to maximize  $\ell_t$  we have
$$
\left\| J (\pi^ {*}) - J (\pi_ {T}) \right\| _ {\infty} \leq \gamma^ {T} \bigg (\left\| J (\pi^ {*}) - J (\pi_ {0}) \right\| _ {\infty} + \sum_ {t = 1} ^ {T} \gamma^ {- t} \left(c _ {t} + \frac {q}{1 - \gamma} [ e _ {t} + b _ {t} ]\right) \bigg)
$$
where  $\pi^{*}$  is the optimal policy,  $\pi^{*s}$  refers to the optimal action in state  $s$ . Here,  $e_t = O(\exp(-m))$  is the approximation error.
Proof. This proof is mainly based on the proof of Theorem 3 of Johnson et al. [24]. Using Lemma 3 and the fact that  $\pi^s \geq 0$ , we have  $\langle Q^{\pi_t}(s,\cdot), \bar{\pi}_{t+1}^s \rangle \leq \langle Q^{\bar{\pi}_{t+1}}(s,\cdot), \bar{\pi}_{t+1}^s \rangle = J_s(\bar{\pi}_{t+1})$ . Using this inequality we get,
$$
\begin{array}{l} \left\langle Q ^ {\pi_ {t}} (s, \cdot), \pi^ {* s} - \bar {\pi} _ {t + 1} ^ {s} \right\rangle \geq \left\langle Q ^ {\pi_ {t}} (s, \cdot), \pi^ {* s} \right\rangle - J _ {s} (\bar {\pi} _ {t + 1}) \\ = \left\langle Q ^ {\pi_ {t}} (s, \cdot) - Q ^ {\pi^ {*}} (s, \cdot), \pi^ {* s} \right\rangle + \left\langle Q ^ {\pi^ {*}} (s, \cdot), \pi^ {* s} \right\rangle - J _ {s} (\bar {\pi} _ {t + 1}) \\ \geq - \left\| Q ^ {\pi_ {t}} (s, \cdot) - Q ^ {\pi^ {*}} (s, \cdot) \right\| _ {\infty} + J _ {s} (\pi^ {*}) - J _ {s} (\bar {\pi} _ {t + 1}) \quad \text {(H o l d e r ' s i n e q u a l i t y)} \\ \geq - \gamma \| J (\pi_ {t}) - J (\pi^ {*}) \| _ {\infty} + J _ {s} (\pi^ {*}) - J _ {s} (\bar {\pi} _ {t + 1}) \\ \end{array}
$$
The last inequality is from the definition of  $Q$  and  $J$  as follows. For any action  $a$ ,
$$
\begin{array}{l} Q ^ {\pi_ {t}} (s, a) - Q ^ {\pi^ {*}} (s, a) = \gamma \sum_ {s ^ {\prime}} P \left(s ^ {\prime} \mid s, a\right) \left[ J _ {s ^ {\prime}} \left(\pi_ {t}\right) - J _ {s ^ {\prime}} \left(\pi^ {*}\right) \right] \\ \leq \gamma \sum_ {s ^ {\prime}} P (s ^ {\prime} | s, a) \| J (\pi_ {t}) - J (\pi^ {*}) \| _ {\infty} \\ \leq \gamma \left\| J (\pi_ {t}) - J (\pi^ {*}) \right\| _ {\infty} \\ \end{array}
$$
From the above inequality,
$$
\begin{array}{l} - \gamma \| J (\pi_ {t}) - J (\pi^ {*}) \| _ {\infty} + J _ {s} (\pi^ {*}) - J _ {s} (\bar {\pi} _ {t + 1}) \leq \langle Q ^ {\pi_ {t}} (s, \cdot), \pi^ {* s} - \bar {\pi} _ {t + 1} ^ {s} \rangle \\ \leq \left\langle Q ^ {\pi_ {t}} (s, \cdot), p _ {t} ^ {s} - \bar {\pi} _ {t + 1} ^ {s} \right\rangle \quad (\text {F o r a n y} p _ {t} ^ {s} \in \Pi_ {t} ^ {s}) \\ \leq \frac {D _ {\phi} \left(p _ {t} ^ {s} , \pi_ {t} ^ {s}\right) - D _ {\phi} \left(p _ {t} ^ {s} , \bar {\pi} _ {t + 1} ^ {s}\right) - D _ {\phi} \left(\bar {\pi} _ {t + 1} ^ {s} , \pi_ {t} ^ {s}\right)}{\eta_ {t} ^ {\prime}} \\ \text {(U s i n g L e m m a 5 w i t h} d = Q ^ {\pi_ {t}} (s, \cdot), y = \bar {\pi} _ {t + 1} ^ {s}, x = p _ {t} ^ {s}) \\ \leq \frac {D _ {\phi} \left(p _ {t} ^ {s} , \pi_ {t} ^ {s}\right)}{\eta_ {t} ^ {\prime}} \\ \end{array}
$$
$$
\begin{array}{l} \Rightarrow - \gamma \| J \left(\pi_ {t}\right) - J \left(\pi^ {*}\right) \| _ {\infty} + J _ {s} \left(\pi^ {*}\right) - J _ {s} \left(\bar {\pi} _ {t + 1}\right) \leq \min  _ {p _ {t} ^ {s} \in \Pi_ {t} ^ {s}} \frac {D _ {\phi} \left(p _ {t} ^ {s} , \pi_ {t} ^ {s}\right)}{\eta_ {t} ^ {\prime}} \leq c _ {t} \quad (\text {B a s e d o n t h e d e f i n i t i o n o f} \eta^ {\prime} \text {i n E q . (5)}) \\ \Rightarrow - \gamma \| J (\pi_ {t}) - J (\pi^ {*}) \| _ {\infty} + J _ {s ^ {\prime}} (\pi^ {*}) - J _ {s ^ {\prime}} (\pi_ {t + 1}) \leq c _ {t} + J _ {s ^ {\prime}} (\bar {\pi} _ {t + 1}) - J _ {s ^ {\prime}} (\pi_ {t + 1}) \\ (S i n c e s i s a n a r r i b a t i r y s t a t e, c h a n g i n g s = s ^ {\prime} f o r \text {c o n v e n i e n c e}) \\ = c _ {t} + \frac {1}{1 - \gamma} \sum_ {s} d ^ {\bar {\pi} _ {t + 1}} (s) \left\langle Q ^ {\pi_ {t + 1}} (s, \cdot), \bar {\pi} _ {t + 1} ^ {s} - \pi_ {t + 1} ^ {s} \right\rangle \\ \end{array}
$$
(Using performance difference lemma 6 with the starting state equal to  $s'$ )
(Cauchy Schwartz)
$$
\begin{array}{l} \leq c _ {t} + \frac {1}{1 - \gamma} \sum_ {s} d ^ {\bar {\pi} _ {t + 1}} (s) \| Q ^ {\pi_ {t + 1}} (s, \cdot) \| \| \bar {\pi} _ {t + 1} ^ {s} - \pi_ {t + 1} ^ {s} \| \\ \leq c _ {t} + \frac {q}{1 - \gamma} \sum_ {s} d ^ {\bar {\pi} _ {t + 1}} (s) \left\| \bar {\pi} _ {t + 1} ^ {s} - \pi_ {t + 1} ^ {s} \right\| \\ \leq c _ {t} + \frac {q}{1 - \gamma} \sum_ {s} d ^ {\bar {\pi} _ {t + 1}} (s) \left[ \left\| \tilde {\pi} _ {t + 1} ^ {s} - \tilde {\pi} _ {t + 1} ^ {s} \right\| + \left\| \tilde {\pi} _ {t + 1} ^ {s} - \pi_ {t + 1} ^ {s} \right\| \right] \\ \leq c _ {t} + \frac {q}{1 - \gamma} (e _ {t} + b _ {t}) \\ \end{array}
$$
Since the above equation is true for all  $s'$  we have:
$$
\left\| J \left(\pi^ {*}\right) - J \left(\pi_ {t + 1}\right) \right\| _ {\infty} \leq \gamma \left\| J \left(\pi_ {t}\right) - J \left(\pi^ {*}\right) \right\| _ {\infty} + c _ {t} + \frac {q}{1 - \gamma} \left(e _ {t} + b _ {t}\right)
$$
Recursing for  $T$  iterations we get:
$$
\left. \left\| J \left(\pi^ {*}\right) - J \left(\pi_ {T}\right) \right\| _ {\infty} \leq \gamma^ {T} \left(\left\| J \left(\pi^ {*}\right) - J \left(\pi_ {0}\right) \right\| _ {\infty} + \sum_ {t = 1} ^ {T} \gamma^ {- t} \left(c _ {t} + \frac {q}{1 - \gamma} [ e _ {t} + b _ {t} ]\right)\right) \right.
$$
We can control the approximation error  $e_t$  by using a larger  $m$ . The bias term  $b_t$  can be small if our function approximation model is expressive enough.  $c_t$  is an arbitrary value and if we set  $c_t = \gamma^t c$  for some constant  $c > 0$ , then  $\sum_{t=1}^{T} \gamma^{-t}(c_t) = Tc$  and therefore  $\gamma^T Tc$  can diminish linearly. The above analysis relied on the knowledge of the true  $Q$  functions, but can be easily extended to using inexact estimates of  $Q^\pi$  by using the techniques developed in [61, 24].
D.3 Exact setting with lifting trick (Softmax representation)
In the softmax representation in the tabular MDP, we consider the case that  $\pi$  is parameterized with parameter  $\theta \in \mathcal{R}^n$ . In this setting  $\Phi$  is the Euclidean norm. Using Prop. 1, for  $\eta$  such that  $J + \frac{1}{\eta}\phi$  is convex we have for a given  $\pi_t$ ,
$$
\begin{array}{l} J (\pi) \geq J (\pi_ {t}) + \left\langle \nabla J (\pi_ {t}), \pi - \pi_ {t} \right\rangle - \frac {1}{\eta} D _ {\Phi} (\pi , \pi_ {t}) \\ = \underbrace {J (\pi_ {t}) + \left\langle \nabla J (\pi_ {t}) , \pi - \pi_ {t} \right\rangle - \frac {1}{2 \eta} \| \pi - \pi_ {t} \| _ {2} ^ {2}} _ {:= h (\pi)} \quad \text {(S i n c e} \phi (.) = \frac {1}{2} \|. \| _ {2} ^ {2}) \\ \end{array}
$$
If we maximize  $h(\pi)$  w.r.t.  $\pi$  we get
$$
\bar {\pi} _ {t + 1} = \underset {\pi} {\arg \max } \left\{h (\pi) \right\} \Longrightarrow \bar {\pi} _ {t + 1} = \pi_ {t} + \eta \nabla_ {\pi} J (\pi_ {t})
$$
Mei et al. [37, Lemma 8] proves that  $J(\pi)$  satisfies a gradient domination condition w.r.t the softmax representation. In particular, if  $a^*(s)$  is the optimal action in state  $s$  and  $\mu \coloneqq \min_{\pi} \frac{\min_s p^\pi(a^*(s)|s)}{\sqrt{S}\left\|\frac{d^\pi^*}{d^\pi}\right\|_\infty}$ , they prove that for all  $\pi$ ,
$$
\left\| \nabla_ {\pi} J (\pi) \right\| \geq \mu \left[ J \left(\pi^ {*}\right) - J (\pi) \right]
$$
Consider optimization in the parameter space where  $\ell_t(\theta) := J(\pi_t) + \langle \nabla J(\pi(\theta_t), \pi(\theta) - \pi(\theta_t)) - \frac{1}{\eta} D_\Phi(\pi(\theta), \pi(\theta_t)) \rangle$ .
$$
\begin{array}{l} \tilde {\theta} _ {t + 1} := \underset {\theta} {\arg \max } \ell_ {t} (\theta) \\ \tilde {\pi} _ {t + 1} = \pi (\tilde {\theta} _ {t + 1}) \\ \theta_ {t + 1} := \operatorname {G r a d i e n t A s c e n t} \left(\ell_ {t}, \theta_ {t}, m\right) \\ \pi_ {t + 1} = \pi \left(\theta_ {t + 1}\right) \\ \end{array}
$$
GradientAscent  $(\ell_t(\theta),\theta_t,m)$  means that we run gradient ascent for  $m$  iterations to maximize  $\ell_t$  with  $\theta_t$  as the initial value. Assuming that  $\ell_t$  is Lipschitz smooth w.r.t.  $\theta$  and satisfies the Polyak-Lojasiewicz (PL) condition, we use the gradient ascent property for PL functions [27] to obtain,
$$
h (\tilde {\pi} _ {t + 1}) - h (\pi_ {t + 1}) = \ell_ {t} (\tilde {\theta} _ {t + 1}) - \ell (\theta_ {t + 1}) \leq \underbrace {e _ {t} := O (\exp (- m))} _ {\text {a p p r o x i m a t i o n e r r o r}}
$$
Proposition 11 (Convergence of softmax+tabular setting with Lifting). Assume (i)  $J + \frac{1}{\eta}\phi$  is convex, (ii)  $J$  satisfies gradient domination property above with  $\mu >0$ , (iii)  $\ell_t(\theta)$  is Lipschitz smooth and satisfies PL condition, (iv)  $|h(\bar{\pi}_{t + 1}) - h(\tilde{\pi}_{t + 1})|\leq b_t$  for all  $t$ . Then after running Gradient Ascent for  $m$  iterations to maximize  $\ell_t$  we have
$$
\min  _ {t \in [ T - 1 ]} \left[ J \left(\pi^ {*}\right) - J \left(\pi_ {t}\right) \right] \leq \sqrt {\frac {J \left(\pi^ {*}\right) - J \left(\pi_ {0}\right) + \sum_ {t = 0} ^ {T - 1} \left[ e _ {t} + b _ {t} \right]}{\alpha T}}
$$
where  $\alpha := \frac{\eta\mu^2}{2}$  and  $e_t$  is the approximation error at iteration  $t$  and  $[T - 1] := \{0,1,2,\dots,T - 1\}$ .
Proof. Since  $J + \frac{1}{\eta}\phi$  is convex,
$$
\begin{array}{l} J (\pi_ {t + 1}) \geq h (\pi_ {t + 1}) = J (\pi_ {t}) + \langle \nabla J (\pi_ {t}), \pi_ {t + 1} - \pi_ {t} \rangle - \frac {1}{2 \eta} \| \pi_ {t + 1} - \pi_ {t} \| _ {2} ^ {2} \\ \geq h \left(\tilde {\pi} _ {t + 1}\right) - e _ {t} \quad (\text {U s i n g t h e G A b o u n d f r o m a b o v e}) \\ \geq h (\bar {\pi} _ {t + 1}) - e _ {t} - b _ {t} = J (\pi_ {t}) + \langle \nabla J (\pi_ {t}), \bar {\pi} _ {t + 1} - \pi_ {t} \rangle - \frac {1}{2 \eta} \| \bar {\pi} _ {t + 1} - \pi_ {t} \| _ {2} ^ {2} - e _ {t} - b _ {t} \\ \geq J \left(\pi_ {t}\right) + \frac {\eta}{2} \left\| \nabla_ {\pi} J \left(\pi_ {t}\right) \right\| _ {2} ^ {2} - e _ {t} - b _ {t} \quad (\text {S i n c e} \bar {\pi} _ {t + 1} = \pi_ {t} + \eta \nabla_ {\pi} J \left(\pi_ {t}\right)) \\ \geq J \left(\pi_ {t}\right) + \frac {\eta \mu^ {2}}{2} \left[ J \left(\pi^ {*}\right) - J \left(\pi_ {t}\right) \right] ^ {2} - e _ {t} - b _ {t} \quad (\text {U s i n g g r a d i e n t d o m i n a t i o n o f} J) \\ \Rightarrow J (\pi^ {*}) - J (\pi_ {t + 1}) \leq \underbrace {J (\pi^ {*}) - J (\pi_ {t})} _ {:= \delta_ {t}} - \underbrace {\frac {\eta \mu^ {2}}{2}} _ {:= \alpha} [ J (\pi^ {*}) - J (\pi_ {t}) ] ^ {2} + e _ {t} + b _ {t} \\ \Longrightarrow \delta_ {t + 1} \leq \delta_ {t} - \alpha \delta_ {t} ^ {2} + e _ {t} + b _ {t} \\ \Longrightarrow \alpha \delta_ {t} ^ {2} \leq \delta_ {t} - \delta_ {t + 1} + e _ {t} + b _ {t} \\ \end{array}
$$
Summing up for  $T$  iterations and dividing both sides by  $T$
$$
\begin{array}{l} \alpha \min  _ {t \in [ T - 1 ]} \delta_ {t} ^ {2} \leq \frac {1}{T} \alpha \sum_ {t = 0} ^ {T - 1} \delta_ {t} ^ {2} \\ \leq \frac {1}{T} [ \delta_ {0} - \delta_ {T + 1} ] + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} [ e _ {t} + b _ {t} ] \leq \frac {1}{T} [ \delta_ {0} ] + \frac {1}{T} \sum_ {t = 0} ^ {T - 1} [ e _ {t} + b _ {t} ] \\ \Rightarrow \min  _ {t \in [ T - 1 ]} \delta_ {t} \leq \sqrt {\frac {\delta_ {0} + \sum_ {t = 0} ^ {T - 1} \left[ e _ {t} + b _ {t} \right]}{\alpha T}} \\ \end{array}
$$
The above analysis relied on the knowledge of the exact gradient  $\nabla J(\pi)$ , but can be easily extended to using inexact estimates of the gradient by using the techniques developed in [63].
D.4 Helper Lemmas
Lemma 4 (3-Point Bregman Property). For  $x, y, z \in \mathcal{X}$ ,
$$
\langle \nabla \phi (z) - \nabla \phi (y), z - x \rangle = D _ {\Phi} (x, z) + D _ {\Phi} (z, y) - D _ {\Phi} (x, y)
$$
Lemma 5 (3-Point Descent Lemma for Mirror Ascent). For any  $z \in \text{rint} \, \text{dom} \, \phi$ , and a vector  $d$ , let
$$
y = \operatorname * {a r g   m a x} _ {x \in \mathcal {X}} \{\langle d, x \rangle - \frac {1}{\eta} D _ {\Phi} (x, z) \}.
$$
Then  $y \in \text{rint} \, \text{dom} \, \phi$  and for any  $x \in \mathcal{X}$
$$
\langle d, y - x \rangle \geq \frac {1}{\eta} \left[ D _ {\Phi} (y, z) + D _ {\Phi} (x, y) - D _ {\Phi} (x, z) \right]
$$
Lemma 6 (Performance Difference Lemma [26]). For any  $\pi, \pi' \in \Pi$ ,
$$
J (\pi) - J \left(\pi^ {\prime}\right) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d ^ {\pi}} \left[ \left\langle Q ^ {\pi^ {\prime}} (s, \cdot), p ^ {\pi} (\cdot | s) - p ^ {\pi^ {\prime}} (\cdot | s) \right\rangle \right]
$$
E Proofs for Sec. 5
Proposition 12 (State-wise lower bound). For (i) any representation  $\pi$  that is separable across states i.e. there exists  $\pi^s \in R^A$  such that  $\pi_{s,a} = [\pi^s]_a$ , (ii) any strictly convex mirror map  $\Phi$  that induces a Bregman divergence that is separable across states i.e.  $D_{\Phi}(\pi, \pi') = \sum_s d^\pi(s) D_{\phi}(\pi^s, \pi'^s)$ , (iii) any  $\eta$  such that  $J + \frac{1}{\eta}\Phi$  is convex, if (iv)  $\nabla J(\pi)$  is separable across states i.e.  $[\nabla J(\pi)]_{s,a} = d^\pi(s) [\nabla_{\pi^s}J(\pi)_a]$  where  $\nabla_{\pi^s}J(\pi) \in \mathbb{R}^A$ , then (v) for any separable (across states) gradient estimator  $\hat{g}$  i.e.  $[\hat{g}(\pi)]_{s,a} = d^\pi(s) [\hat{g}^s(\pi)]_a$  where  $\hat{g}^s(\pi) \in \mathbb{R}^A$ , and  $c \in (0,\infty)^S$ ,
$$
J (\pi) \geq J (\pi_ {t}) + \langle \hat {g} (\pi_ {t}), (\pi - \pi_ {t}) \rangle - \sum_ {s} d ^ {\pi_ {t}} (s) \left(\frac {1}{\eta} + \frac {1}{c _ {s}}\right) D _ {\phi} (\pi^ {s}, \pi_ {t} ^ {s}) - \sum_ {s} \frac {d ^ {\pi_ {t}} (s) D _ {\phi^ {*}} (\nabla \phi (\pi_ {t} ^ {s}) - c _ {s} \delta_ {t} ^ {s} , \nabla \phi (\pi_ {t} ^ {s}))}{c _ {s}}
$$
Proof. Using condition (iii) of the proposition with Lemma 2,
$$
\begin{array}{l} J (\pi) \geq J (\pi_ {t}) + \langle \nabla J (\pi_ {t}), \pi - \pi_ {t} \rangle - \frac {1}{\eta} D _ {\phi} (\pi , \pi_ {t}) \\ = J (\pi_ {t}) + \langle \hat {g} (\pi_ {t}), \pi - \pi_ {t} \rangle + \langle \nabla J (\pi_ {t}) - \hat {g} (\pi_ {t}), \pi - \pi_ {t} \rangle - \frac {1}{\eta} D _ {\phi} (\pi , \pi_ {t}) \\ \end{array}
$$
Using conditions (iv) and (v), we know that  $\left[\nabla J(\pi_t)\right]_{s,a} = d^{\pi_t}(s)\left[\nabla_{\pi^s}J(\pi_t)\right]_a$  and  $\left[\hat{g} (\pi_t)\right]_{s,a} = d^{\pi_t}(s)\left[\hat{g}^s (\pi_t)\right]_a$ . Defining  $\delta_t^s \coloneqq \nabla_{\pi^s}J(\pi_t) - \hat{g}^s (\pi_t) \in \mathbb{R}^A$ . Using conditions (i) and (ii), we can rewrite the lower-bound as follows,
$$
\begin{array}{l} J (\pi) \geq J (\pi_ {t}) + \langle \hat {g} (\pi_ {t}), (\pi - \pi_ {t}) \rangle + \sum_ {s} d ^ {\pi_ {t}} (s) \left\langle \delta_ {t} ^ {s}, \pi^ {s} - \pi_ {t} ^ {s} \right\rangle - \frac {1}{\eta} \sum_ {s} d ^ {\pi_ {t}} (s) D _ {\phi} (\pi^ {s}, \pi_ {t} ^ {s}) \\ = J (\pi_ {t}) + \langle \hat {g} (\pi_ {t}), \pi - \pi_ {t} \rangle + \sum_ {s} d ^ {\pi_ {t}} (s) \left[ \langle \delta_ {t} ^ {s}, \pi^ {s} - \pi_ {t} ^ {s} \rangle - \frac {1}{\eta} D _ {\phi} (\pi^ {s}, \pi_ {t} ^ {s}) \right] \\ \end{array}
$$
Using Lemma 1 with  $x = \delta_t^s$ ,  $y = \pi^s$  and  $y' = \pi_t^s$ ,