SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {\star}} \left[ \langle q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \right] \right] \stackrel {(5 6)} {\geq} \left\| \frac {d _ {\mu} ^ {\star}}{d _ {\mu} ^ {t + 1}} \right\| _ {L _ {\infty}} \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \langle q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \right] \right] \\ = \nu_ {t + 1} \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \left\langle Q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \right\rangle \right] \right] \\ + \nu_ {t + 1} \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \langle q _ {s} ^ {t} - Q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \right] \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {=} \quad \nu_ {t + 1} (1 - \gamma) \mathbb {E} \left[ V ^ {t} (\mu) - V ^ {t + 1} (\mu) \right] \\ + \nu_ {t + 1} \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {t + 1}} \left[ \langle q _ {s} ^ {t} - Q _ {s} ^ {t}, \pi_ {s} ^ {t} - \pi_ {s} ^ {t + 1} \rangle \right] \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(5 7)} {\geq} \nu_ {t + 1} (1 - \gamma) \mathbb {E} \left[ V ^ {t} (\mu) - V ^ {t + 1} (\mu) \right] - \nu_ {t + 1} \tau (1 - \gamma) \\ = \nu_ {t + 1} (1 - \gamma) \mathbb {E} [ \Delta_ {t + 1} - \Delta_ {t} ] - \nu_ {t + 1} \tau (1 - \gamma), \\ \end{array}
$$
where  $(a)$  follows from Lemma D.4. For the second term, we have that
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {\star}} \left[ \langle q _ {s} ^ {t}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {t} \rangle \right] \right] = \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {\star}} \left[ \langle Q _ {s} ^ {t}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {t} \rangle \right] \right] + \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {\star}} \left[ \langle q _ {s} ^ {t} - Q _ {s} ^ {t}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {t} \rangle \right] \right] \\ \stackrel {(b)} {=} \mathbb {E} [ \Delta_ {t} ] (1 - \gamma) + \mathbb {E} \left[ \mathbb {E} _ {s \sim d _ {\mu} ^ {*}} \left[ \langle q _ {s} ^ {t} - Q _ {s} ^ {t}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {t} \rangle \right] \right] \\ \begin{array}{l l} \stackrel {{(c)}} {{\geq}} & \mathbb {E} [ \Delta_ {t} ] (1 - \gamma) - \tau (1 - \gamma), \end{array} \\ \end{array}
$$
where  $(b)$  follows from Lemma D.4 and  $(c)$  follows similarly to (57), i.e., by applying (54) twice with  $(d_{\mu}^{\pi},\pi) = (d_{\mu}^{\star},\pi^{\star})$  and  $(d_{\mu}^{\pi},\pi) = (d_{\mu}^{\star},\pi^{t})$ .
Plugging the two bounds in (59), dividing both sides by  $(1 - \gamma)$  and rearranging, we obtain
$$
\mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {t + 1}}{(1 - \gamma) \eta_ {t}} + \nu_ {t + 1} (\Delta_ {t + 1} - \Delta_ {t} - \tau) + \Delta_ {t} \right] \leq \mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {\star}}{(1 - \gamma) \eta_ {t}} - \frac {\mathcal {D} _ {t + 1} ^ {\star}}{(1 - \gamma) \eta_ {t}} \right] + \tau .
$$
From Proposition 4.2, we have that  $\Delta_{t + 1} - \Delta_t - \tau \leq 0$ . Consequently, since  $\nu_{t + 1}\leq \nu_{\mu}$  by the definition of  $\nu_{\mu}$  in Assumption (A3), one can lower bound the left hand side of the above inequality by replacing  $\nu_{t + 1}$  by  $\nu_{\mu}$ , that is,
$$
\mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {t + 1}}{(1 - \gamma) \eta_ {t}} + \nu_ {\mu} (\Delta_ {t + 1} - \Delta_ {t} - \tau) + \Delta_ {t} \right] \leq \mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {\star}}{(1 - \gamma) \eta_ {t}} - \frac {\mathcal {D} _ {t + 1} ^ {\star}}{(1 - \gamma) \eta_ {t}} \right] + \tau ,
$$
D.5 Proof of the sublinear convergence analysis
In this section, we derive the sublinear convergence result of Theorem 4.3 with non-decreasing step-size.
Proof. Starting from Proposition D.6
If  $\eta_t\leq \eta_{t + 1}$
$$
\mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {t + 1}}{(1 - \gamma) \eta_ {t}} + \nu_ {\mu} (\Delta_ {t + 1} - \Delta_ {t}) + \Delta_ {t} \right] \leq \mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {\star}}{(1 - \gamma) \eta_ {t}} - \frac {\mathcal {D} _ {t + 1} ^ {\star}}{(1 - \gamma) \eta_ {t + 1}} \right] + (1 + \nu_ {\mu}) \tau . \tag {60}
$$
Summing up from 0 to  $T - 1$  and dropping some positive terms on the left hand side and some negative terms on the right hand side, we have
$$
\sum_ {t <   T} \mathbb {E} [ \Delta_ {t} ] \leq \frac {\mathcal {D} _ {0} ^ {\star}}{(1 - \gamma) \eta_ {0}} + \nu_ {\mu} \Delta_ {0} + T (1 + \nu_ {\mu}) \tau \leq \frac {\mathcal {D} _ {0} ^ {\star}}{(1 - \gamma) \eta_ {0}} + \frac {\nu_ {\mu}}{1 - \gamma} + T (1 + \nu_ {\mu}) \tau .
$$
Notice that  $\Delta_0\leq \frac{1}{1 - \gamma}$  as  $r(s,a)\in [0,1]$ . By dividing  $T$  on both side, we yield the proof of the sublinear convergence
$$
V ^ {\star} (\mu) - \frac {1}{T} \sum_ {t <   T} \mathbb {E} \left[ V ^ {t} (\mu) \right] \leq \frac {1}{T} \left(\frac {\mathcal {D} _ {0} ^ {\star}}{(1 - \gamma) \eta_ {0}} + \frac {\nu_ {\mu}}{1 - \gamma}\right) + (1 + \nu_ {\mu}) \tau .
$$
D.6 Proof of the linear convergence analysis
In this section, we derive the linear convergence result of Theorem 4.3 with exponentially increasing step-size.
Proof. Starting from Proposition D.6 by dropping  $\frac{\mathcal{D}_t^{t + 1}}{(1 - \gamma)\eta_t}$  on the left hand side, we have
$$
\mathbb {E} \left[ \nu_ {\mu} \left(\Delta_ {t + 1} - \Delta_ {t}\right) + \Delta_ {t} \right] \leq \mathbb {E} \left[ \frac {\mathcal {D} _ {t} ^ {\star}}{(1 - \gamma) \eta_ {t}} - \frac {\mathcal {D} _ {t + 1} ^ {\star}}{(1 - \gamma) \eta_ {t}} \right] + (1 + \nu_ {\mu}) \tau .
$$
Dividing  $\nu_{\mu}$  on both side and rearranging, we obtain
$$
\mathbb {E} \left[ \Delta_ {t + 1} + \frac {\mathcal {D} _ {t + 1} ^ {\star}}{(1 - \gamma) \nu_ {\mu} \eta_ {t}} \right] \leq \left(1 - \frac {1}{\nu_ {\mu}}\right) \mathbb {E} \left[ \Delta_ {t} + \frac {\mathcal {D} _ {t} ^ {\star}}{(1 - \gamma) \eta_ {t} (\nu_ {\mu} - 1)} \right] + \left(1 + \frac {1}{\nu_ {\mu}}\right) \tau .
$$
If the step-sizes satisfy  $\eta_{t + 1}(\nu_{\mu} - 1)\geq \eta_t\nu_\mu$  with  $\nu_{\mu}\geq 1$ , then
$$
\mathbb {E} \left[ \Delta_ {t + 1} + \frac {\mathcal {D} _ {t + 1} ^ {\star}}{(1 - \gamma) \eta_ {t + 1} (\nu_ {\mu} - 1)} \right] \leq \left(1 - \frac {1}{\nu_ {\mu}}\right) \mathbb {E} \left[ \Delta_ {t} + \frac {\mathcal {D} _ {t} ^ {\star}}{(1 - \gamma) \eta_ {t} (\nu_ {\mu} - 1)} \right] + \left(1 + \frac {1}{\nu_ {\mu}}\right) \tau .
$$
Now we need the following simple fact, whose proof is straightforward and thus omitted.
Suppose  $0 < \alpha < 1, b > 0$  and a nonnegative sequence  $\{a_t\}_{t \geq 0}$  satisfies
$$
a _ {t + 1} \leq \alpha a _ {t} + b \quad \forall t \geq 0.
$$
Then for all  $t\geq 0$
$$
a _ {t} \leq \alpha^ {t} a _ {0} + \frac {b}{1 - \alpha}.
$$
The proof of the linear convergence analysis follows by applying this fact with  $a_{t} =$
$$
\mathbb {E} \left[ \Delta_ {t} + \frac {\mathcal {D} _ {t} ^ {*}}{(1 - \gamma) \eta_ {t} (\nu_ {\mu} - 1)} \right], \alpha = 1 - \frac {1}{\nu_ {\mu}} \text {a n d} b = \left(1 + \frac {1}{\nu_ {\mu}}\right) \tau .
$$
E Discussion of the first step (Line 1) of AMPO – the compatible function approximation framework
Starting from this section, some additional remarks about AMPO are in order. In particular, we discuss in detail the novelty of the first step (Line 1) and the second step (Line 2) of AMPO in this and the next section, respectively. Afterwards, we provide an extensive justification of the assumptions used in Theorem 4.3 in Appendices G to I.
As mentioned in Remark 3.3, Agarwal et al. [1] study NPG with smooth policies through compatible function approximation and propose the following algorithm. Let  $\{\pi^{\theta}:\theta \in \Theta \}$  be a policy class such that  $\log \pi^{\theta}(a\mid s)$  is a  $\beta$ -smooth function of  $\theta$  for all  $s\in S$ ,  $a\in \mathcal{A}$ . At each iteration  $t$ , update
$$
\theta^ {t + 1} = \theta^ {t} + \eta w ^ {t},
$$
$$
w ^ {t} \in \operatorname * {a r g m i n} _ {\| w \| _ {2} \leq W} \left\| A ^ {t} - w ^ {\top} \nabla_ {\theta} \log \pi^ {t} \right\| _ {L _ {2} \left(d _ {\mu} ^ {t} \cdot \pi^ {t}\right)}, \tag {61}
$$
where  $W > 0$  and  $A^t(s, a) = Q^t(s, a) - V^t(s)$  represents the advantage function. While both the algorithm proposed by Agarwal et al. [1] and AMPO involve regression problems, the one in (61) is restricted to linearly approximate  $A^t$  with  $\nabla_\theta \log \pi^t$ , whereas the one in Line 1 of Algorithm 1 is relaxed to approximate  $A^t$  with an arbitrary class of functions  $\mathcal{F}^\Theta$ . Additionally, (61) depends on the distribution  $d_\mu^t$ , while Line 1 of Algorithm 1 does not and allows off-policy updates involving an arbitrary distribution  $v^t$ , as  $v^t$  is independent of the current policy  $\pi^t$ .
F Discussion of the second step (Line 2) of AMPO - the Bregman projection
As mentioned in Remark 3.4, we can rewrite the second step (Line 2) of AMPO through the following lemma.
Lemma F.1. For any policy  $\bar{\pi}$ , for any function  $f^{\theta} \in \mathcal{F}^{\Theta}$  and for  $\eta > 0$ , we have, for all  $s \in S$ ,
$$
\tilde{\pi}_{s}\in \operatorname *{argmin}_{p\in \Delta (\mathcal{A})}\mathcal{D}_{h}(p,\nabla h^{*}(\eta f_{s}^{\theta}))\iff \tilde{\pi}_{s}\in \operatorname *{argmin}_{p\in \Delta (\mathcal{A})}\langle -\eta f_{s}^{\theta} + \nabla h(\bar{\pi}_{s}),p\rangle +\mathcal{D}_{h}(p,\bar{\pi}_{s}).
$$
Equations (12) and (20) are obtained by choosing  $\tilde{\pi}_s = \pi_s^{t + 1}$  and  $\bar{\pi}_s = \pi_s^t$  for all  $s\in S,\eta = \eta_t$ $\theta = \theta^{t + 1}$ , and by changing the sign of the expression on the right in order to obtain an argmax.
Proof. Starting from the definition of  $\tilde{\pi}$ , we have
$$
\begin{array}{l} \tilde{\pi}_{s}\in \operatorname *{argmin}_{p\in \Delta (\mathcal{A})}\mathcal{D}_{h}(p,\nabla h^{*}(\eta f_{s}^{\theta})) \\ \in \operatorname *{argmin}_{p\in \Delta (\mathcal{A})}h(p) - h(\nabla h^{*}(\eta f_{s}^{\theta})) - \langle \nabla h(\nabla h^{*}(\eta f_{s}^{\theta})),p - \nabla h^{*}(\eta f_{s}^{\theta})\rangle \\ \in \operatorname *{argmin}_{p\in \Delta (\mathcal{A})}h(p) - \langle \eta f_{s}^{\theta},p\rangle \\ \in \operatorname *{argmin}_{p\in \Delta (\mathcal{A})}\langle -\eta f_{s}^{\theta} + \nabla h(\bar{\pi}_{s}),p\rangle +h(p) - h(\bar{\pi}_{s}) - \langle \nabla h(\bar{\pi}_{s}),p - \bar{\pi}_{s}\rangle \\ \in \operatorname * {a r g m i n} _ {p \in \Delta (\mathcal {A})} \langle - \eta f _ {s} ^ {\theta} + \nabla h (\bar {\pi} _ {s}), p \rangle + \mathcal {D} _ {h} (p, \bar {\pi} _ {s}), \tag {62} \\ \end{array}
$$
where the second and the last lines are obtained using the definition of the Bregman divergence, and the third line is obtained using (4)  $(\nabla h(\nabla h^{*}(x^{*})) = x^{*}$  for all  $x^{*} \in \mathbb{R}^{|A|}$ .
Lemmas F.1 and B.1 share a similar result, as they both rewrite the Bregman projection into the MD updates. However, the MD updates in Lemma B.1 are exact, while the MD updates in AMPO involve approximation (Line 1).
Next, we provide an alternative proof for Lemma 4.1 to show that it is the direct consequence of Lemma F.1. The proof will involve the application of the three-point descent lemma [19, Lemma 3.2]. Here we adopt its slight variation by following Lemma 6 in Xiao [93].
Lemma F.2 (Three-point decent lemma, Lemma 6 in Xiao [93]). Suppose that  $\mathcal{C} \subset \mathbb{R}^m$  is a closed convex set,  $f: \mathcal{C} \to \mathbb{R}$  is a proper, closed convex function,  $\mathcal{D}_h(\cdot, \cdot)$  is the Bregman divergence generated by a mirror map  $h$ . Denote  $\operatorname{rint} \operatorname{dom} h$  as the relative interior of  $\operatorname{dom} h$ . For any  $x \in \operatorname{rint} \operatorname{dom} h$ , let
$$
x ^ {+} \in \arg \min  _ {u \in \operatorname {d o m} h \cap \mathcal {C}} \{f (u) + \mathcal {D} _ {h} (u, x) \}.
$$
Then  $x^{+}\in \mathrm{rint}$  dom  $h\cap \mathcal{C}$  and for any  $u\in \mathrm{dom}h\cap \mathcal{C}$
$$
f \left(x ^ {+}\right) + \mathcal {D} _ {h} \left(x ^ {+}, x\right) \leq f (u) + \mathcal {D} _ {h} (u, x) - \mathcal {D} _ {h} \left(u, x ^ {+}\right).
$$
We refer to Yuan et al. [98, Lemma 11] for a proof of Lemma F.2.
Lemma 4.1 is obtained by simply applying the three-point descent lemma, Lemma F.2, to (62) with  $x^{+} = \tilde{\pi}_{s}$ ,  $f(u) = \langle -\eta f_{s}^{\theta} + \nabla h(\bar{\pi}_{s}), u \rangle$ ,  $u = \pi$  and  $x = \bar{\pi}_{s}$  and rearranging terms.
In contrast, it may not be possible to apply Lemma F.2 to (11), as  $\Pi (\Theta)$  is often non-convex.
G Discussion on Assumption (A1) - the approximation error
The compatible function approximation approach [1, 61, 15, 21, 2, 98] has been introduced to deal with large state and action spaces, in order to reduce the dimension of the problem and make the computation feasible. As mentioned in the proof idea in Page 8, this framework consists in upper-bounding the sub-optimality gap with an optimization error plus an approximation error. Consequently, it is important that both error terms converge to 0 in order to achieve convergence to a global optimum.
Assumptions similar to Assumption (A1) are common in the compatible function approximation literature. Assumption (A1) encodes a form of realizability assumption for the parameterization class  $\mathcal{F}^{\Theta}$ , that is, we assume that for all  $t\leq T$  there exists a function  $f^{\theta}\in \mathcal{F}^{\Theta}$  such that
$$
\left\| f ^ {\theta} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h (\pi^ {t}) \right\| _ {L _ {2} (v ^ {t})} ^ {2} \leq \varepsilon_ {\text {a p p r o x}}.
$$
When  $\mathcal{F}^{\Theta}$  is a class of sufficiently large shallow neural networks, this realizability assumption holds as it has been shown that shallow neural networks are universal approximators [39]. It is, however, possible to relax Assumption (A1). In particular, the condition
$$
\frac {1}{T} \sum_ {t <   T} \sqrt {\mathbb {E} \left[ \mathbb {E} \left\| f ^ {t + 1} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h \left(\pi^ {t}\right) \right\| _ {L _ {2} (v ^ {t})} ^ {2} \right]} \leq \sqrt {\varepsilon_ {\text {a p p r o x}}} \tag {63}
$$
can replace Assumption (A1) and is sufficient for the sublinear convergence rate in Theorem 4.3 to hold. Equation (63) shows that the realizability assumption does not need to hold for all  $t < T$ , but only needs to hold on average over  $T$  iterations. Similarly, the condition
$$
\sum_ {t \leq T} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {T - t} \frac {1}{\nu_ {\mu}} \sqrt {\mathbb {E} \left[ \mathbb {E} \left\| f ^ {t + 1} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h \left(\pi^ {t}\right) \right\| _ {L _ {2} \left(v ^ {t}\right)} ^ {2} \right]} \leq \sqrt {\varepsilon_ {\text {a p p r o x}}} \tag {64}
$$
can replace Assumption (A1) and is sufficient for the linear convergence rate in Theorem 4.3 to hold. Additionally, requiring, for all  $t < T$ ,
$$
\mathbb {E} \left[ \mathbb {E} \left\| f ^ {t + 1} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h \left(\pi^ {t}\right) \right\| _ {L _ {2} \left(v ^ {t}\right)} ^ {2} \right] \leq \frac {\nu_ {\mu} ^ {2}}{T ^ {2}} \left(1 - \frac {1}{\nu_ {\mu}}\right) ^ {- 2 (T - t)} \varepsilon_ {\text {a p p r o x}} \tag {65}
$$
is sufficient for Equation (64) to hold. Equation (65) shows that the error floor in the linear convergence rate is less influenced by approximation errors made in early iterations, which are discounted by the term  $\left(1 - \frac{1}{\nu_{\mu}}\right)$ . On the other hand, the realizability assumption becomes relevant once the algorithm approaches convergence, i.e., when  $t \simeq T$  and  $Q^t \simeq Q^\star$ , as the discount term  $\left(1 - \frac{1}{\nu_{\mu}}\right)$  is applied fewer times.
Finally, although Assumption (A1) holds for the softmax tabular policies and for the neural network parameterization, it remains an open question whether Assumption (A1) is necessary to achieve the global optimum convergence, especially when the representation power of  $\mathcal{F}^{\Theta}$  cannot guarantee a small approximation error.
6A convex function  $f$  is proper if  $\operatorname{dom} f$  is nonempty and for all  $x \in \operatorname{dom} f$ ,  $f(x) > -\infty$ . A convex function is closed, if it is lower semi-continuous.
H Discussion on Assumption (A2) - the concentrability coefficients
In our convergence analysis, Assumptions (A2) and (A3) involve the concentrability coefficient  $C_v$  and the distribution mismatch coefficient  $\nu_{\mu}$ , which are potentially large. We give extensive discussions on them in this and the next section, respectively.
As discussed in Yuan et al. [98, Appendix H], the issue of having (potentially large) concentrability coefficient (Assumptions (A2)) is unavoidable in all the fast linear convergence analysis of approximate PMD due to the approximation error  $\varepsilon_{\mathrm{approx}}$  of the  $Q$ -function [17, 100, 54, 16, 93, 20, 2, 98]. Indeed, in the fast linear convergence analysis of PMD, the concentrability coefficient is always along with the approximation error  $\varepsilon_{\mathrm{approx}}$  under the form of  $C_v\varepsilon_{\mathrm{approx}}$ , which is the case in Theorem 4.3. To not get the concentrability coefficient involved yet maintain the linear convergence of PMD, one needs to consider the exact PMD in the tabular setting [see 93, Theorem 10]. Consequently, the PMD update is deterministic and the full policy space  $\Delta (\mathcal{A})^S$  is considered. In this setting, at each time  $t$ , it exists  $\theta^{t + 1}$  such that, for any state-action distribution  $v^t$ ,
$$
\left\| f ^ {t + 1} - Q ^ {t} - \eta_ {t} ^ {- 1} \nabla h (\pi^ {t}) \right\| _ {L _ {2} (v ^ {t})} ^ {2} = 0 = \varepsilon_ {\mathrm {a p p r o x}},
$$
and  $C_v$  is ignored in the convergence analysis thanks to the vanishing of  $\varepsilon_{\mathrm{approx}}$ . We note that the PMD analysis in the seminal paper by Agarwal et al. [1] does not use such a coefficient, but a condition number instead. The condition number is controllable to be relatively small, so that the error term in their PMD analysis is smaller than ours. However, their PMD analysis has only a sublinear convergence rate, while ours enjoys a fast linear convergence rate. It remains an open question whether one can both avoid using the concentrability coefficient and maintain the linear convergence of PMD.
Now we compare our concentrability coefficient  $C_v$  with others used in the fast linear convergence analysis of approximate PMD [17, 100, 54, 15, 93, 20, 2, 98]. To the best of our knowledge, the previously best-known concentrability coefficient  $C_v$  was the one used by Yuan et al. [98, Appendix H]. As they discuss, their concentrability coefficient involved the weakest assumptions on errors among Lan [54], Xiao [93] and Chen and Theja Maguluri [20] by using the  $L_2$ -norm instead of the  $\ell_{\infty}$ -norm over the approximation error  $\varepsilon_{\mathrm{approx}}$ . Additionally, it did not impose any restrictions on the MDP dynamics compared to Cayci et al. [15], as the concentrability coefficient of Yuan et al. [98] was independent from the iterates.
Indeed, Yuan et al. [98] choose  $v^t$  such that, for all  $(s,a)\in S\times \mathcal{A}$ ,
$$
v ^ {t} (s, a) = (1 - \gamma) \mathbb {E} _ {(s _ {0}, a _ {0}) \sim \nu} \left[ \sum_ {t ^ {\prime} = 0} ^ {\infty} \gamma^ {t ^ {\prime}} P (s _ {t ^ {\prime}} = s, a _ {t ^ {\prime}} = a \mid \pi^ {t}, s _ {0}, a _ {0}) \right],
$$
where  $\nu$  is an initial state-action distribution chosen by the user. In this setting, we have
$$
v ^ {t} (s, a) \geq (1 - \gamma) \nu (s, a).
$$
From the above lower bound of  $v^t$ , we obtain that
$$
\begin{array}{l} \mathbb {E} _ {(s, a) \sim v ^ {t}} \left[ \left(\frac {d _ {\mu} ^ {\pi} (s) \pi (a \mid s)}{v ^ {t} (s , a)}\right) ^ {2} \right] = \int_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \frac {d _ {\mu} ^ {\pi} (s) ^ {2} \pi (a \mid s) ^ {2}}{v ^ {t} (s , a)} \\ \leq \int_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \frac {1}{v ^ {t} (s , a)} \leq \frac {1}{(1 - \gamma) \min _ {(s , a) \in \mathcal {S} \times \mathcal {A}} \nu (s , a)}, \\ \end{array}
$$
where the finite upper bound is independent to  $t$ .
As mentioned right after Assumption (A2), the assumption on our concentrability coefficient  $C_v$  is weaker than the one in Yuan et al. [98, Assumption 9], as we have the full control over  $v^t$  while Yuan et al. [98] only has the full control over the initial state-action distribution  $\nu$ . In particular, our concentrability coefficient  $C_v$  recovers the previous best-known one in Yuan et al. [98] as a special case. Consequently, our concentrability coefficient  $C_v$  becomes the "best" with the full control over  $v^t$  when other concentrability coefficients are infinite or require strong assumptions [77].
In general, for the ratio  $\mathbb{E}_{(s,a)}\sim v^t\left[\left(\frac{d_\mu^\pi(s)\pi(a|s)}{v^t(s,a)}\right)^2\right]$  to have a finite upper bound  $C_v$ , it is important that  $v^t$  covers well the state and action spaces so that the upper bound is independent of  $t$ . However, the
upper bound  $\frac{1}{(1 - \gamma)\min_{(s,a)\in S\times\mathcal{A}}\nu(s,a)}$  in Yuan et al. [98] is very pessimistic. Indeed, when  $\pi^t$  and  $\pi^{t + 1}$  converge to  $\pi^{\star}$ , one reasonable choice of  $v^{t}$  is to choose  $v^{t}\in \{d_{\mu}^{\star}\cdot \pi^{\star},d_{\mu}^{t + 1}\cdot \pi^{t + 1},d_{\mu}^{\star}\cdot \pi^{t},d_{\mu}^{t + 1}\cdot \pi^{t}\}$  such that  $C_v$  is close to 1.
We also refer to Yuan et al. [98, Appendix H] for more discussions on the concentrability coefficient.
I Discussion on Assumption (A3) - the distribution mismatch coefficients
In this section, we give further insights on the distribution mismatch coefficient  $\nu_{\mu}$  in Assumption (A3). As mentioned right after (A3), we have that
$$
\sup  _ {s \in \mathcal {S}} \frac {d _ {\mu} ^ {\star} (s)}{d _ {\mu} ^ {t} (s)} \leq \frac {1}{1 - \gamma} \sup  _ {s \in \mathcal {S}} \frac {d _ {\mu} ^ {\star} (s)}{\mu (s)} := \nu_ {\mu} ^ {\prime},
$$
which is a sufficient upper bound for  $\nu_{\mu}$ . As discussed in Yuan et al. [98, Appendix H],
$$
1 / (1 - \gamma) \leq \nu_ {\mu} ^ {\prime} \leq 1 / ((1 - \gamma) \min _ {s} \mu (s)).
$$
The upper bound  $1 / ((1 - \gamma)\min_s\mu (s))$  of  $\nu_{\mu}^{\prime}$  is very pessimistic and the lower bound  $\nu_{\mu}^{\prime} = 1 / (1 - \gamma)$  is often achieved by choosing  $\mu = d_{\mu}^{\star}$ .
Furthermore, if  $\mu$  does not have full support on the state space, i.e., the upper bound  $1 / ((1 - \gamma)\min_s\mu (s))$  might be infinite, one can always convert the convergence guarantees for some state distribution  $\mu^{\prime}\in \Delta (\mathcal{S})$  with full support such that
$$
\begin{array}{l} V ^ {\star} (\mu) - \mathbb {E} [ V ^ {T} (\mu) ] = \mathbb {E} \left[ \int_ {s \in \mathcal {S}} \frac {\mu (s)}{\mu^ {\prime} (s)} \mu^ {\prime} (s) \left(V ^ {\star} (s) - V ^ {T} (s)\right) \right] \\ \leq \sup  _ {s \in \mathcal {S}} \frac {\mu (s)}{\mu^ {\prime} (s)} \left(V ^ {\star} \left(\mu^ {\prime}\right) - \mathbb {E} \left[ V ^ {T} \left(\mu^ {\prime}\right) \right]\right). \\ \end{array}
$$
Then by the linear convergence result of Theorem 4.3, we only transfer the original convergence guarantee to  $V^{\star}(\mu') - \mathbb{E}[V^{T}(\mu')]$  up to a scaling factor  $\sup_{s \in S} \frac{\mu(s)}{\mu'(s)}$  with an arbitrary distribution  $\mu'$  such that  $\nu_{\mu}'$  is finite.
Finally, if  $d_{\mu}^{t}$  converges to  $d_{\mu}^{\star}$  which is the case of AMPO through the proof of our Theorem 4.3, then  $\sup_{s \in S} \frac{d_{\mu}^{\star}(s)}{d_{\mu}^{t}(s)}$  converges to 1. This might imply superlinear convergence results as discussed in Xiao [93, Section 4.3]. In this case, the notion of the distribution mismatch coefficients  $\nu_{\mu}$  no longer exists for the superlinear convergence analysis.
We also refer to Yuan et al. [98, Appendix H] for more discussions on the distribution mismatch coefficient.
J Sample complexity for neural network parameterization