SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} V ^ {\star} (s _ {1}) - V ^ {0} (s _ {1}) = \frac {\gamma}{1 - \gamma} - \alpha \frac {\gamma}{1 - \gamma} - (1 - \alpha) \frac {\gamma^ {2} + \delta}{1 - \gamma} \\ = (1 - \alpha) \frac {\gamma}{1 - \gamma} - (1 - \alpha) \frac {\gamma^ {2} + \delta}{1 - \gamma} \\ = \frac {1 - \alpha}{1 - \gamma} \left(\gamma - \gamma^ {2} - \delta\right) \\ \leq \frac {1 - \alpha}{1 - \gamma} \left(\gamma - \gamma^ {2}\right) \\ = \gamma \frac {1 - \alpha}{1 - \gamma} (1 - \gamma) \\ = \gamma (1 - \alpha) \\ \leq \gamma \\ \end{array}
$$
and by induction we can show this is the case for all states (above is base case), the inductive step is as follows (assuming  $V^{\star}(s_k) - V^0(s_k) \leq \gamma$ ),
$$
\begin{array}{l} V ^ {\star} (s _ {k + 1}) - V ^ {0} (s _ {k + 1}) = \frac {\gamma^ {k + 1}}{1 - \gamma} - (1 - \alpha) \frac {\gamma^ {k + 2} + \delta}{1 - \gamma} - \alpha \gamma V ^ {0} (s _ {k}) \\ = (1 - \alpha) \Big [ \frac {\gamma^ {k + 1} - \gamma^ {k + 2} - \delta}{1 - \gamma} \Big ] + \alpha \gamma \Big [ V ^ {\star} (s _ {k}) - V ^ {0} (s _ {k}) \Big ] \\ \leq (1 - \alpha) \gamma^ {k + 1} + \alpha \gamma^ {2} \\ \leq \gamma \\ \end{array}
$$
$$
\| V ^ {\star} - V ^ {0} \| _ {\infty} \leq \gamma ,
$$
which combining with (17) gives,
$$
\begin{array}{l} V ^ {\star} (s _ {k + 1}) - V ^ {k} (s _ {k + 1}) \geq \gamma^ {k} \| V ^ {\star} - V ^ {0} \| _ {\infty} - \frac {2 \delta}{1 - \gamma} \\ \Rightarrow \| V ^ {\star} - V ^ {k} \| _ {\infty} \geq \gamma^ {k} \| V ^ {\star} - V ^ {0} \| _ {\infty} - \frac {2 \delta}{1 - \gamma}. \\ \end{array}
$$
Choosing  $\delta = \frac{(1 - \gamma)}{4}\gamma^n\| V^\star -V^0\|_\infty \leq \frac{(1 - \gamma)}{4}\gamma^{n + 1}$  (which is in the valid range of  $\delta$ ) concludes the proof since  $\frac{2\delta}{1 - \gamma} \leq \frac{1}{2}\gamma^k\| V^\star -V^0\|_\infty$ .
F Proof of Theorem 4.3
Consider the same MDP as in the proof of Theorem 4.2 in Appendix E (see Figure 5). Denote  $c = \frac{1 - \gamma}{8}$  and note that  $c < \frac{\sqrt{\gamma}}{1 + \sqrt{\gamma}}\frac{1 - \gamma}{2}$  since  $\frac{1}{4} < \frac{\sqrt{\gamma}}{1 + \sqrt{\gamma}}$  for  $\gamma > 0.2$ .
Suppose you consider NPG updates with initial policy  $\pi^0 (a_1|s_i) = \alpha$ . Recall that NPG is the instance of PMD with relative entropy as the mirror map. It can be shown that NPG has the closed form update
$$
\pi^ {k + 1} (a | s) = \frac {\pi^ {k} (a | s) e ^ {\eta_ {k} Q ^ {k} (s , a)}}{\sum_ {a ^ {\prime}} \pi^ {k} (a ^ {\prime} | s) e ^ {\eta_ {k} Q ^ {k} (s , a ^ {\prime})}}.
$$
We know from the proof of Theorem E that for any step-size regime, for  $i > k + 1$
$$
Q ^ {k} (s _ {i}, a _ {1}) \leq Q ^ {k} (s _ {i}, a _ {2}).
$$
Now,  $\| V^{\star} - V^{0}\|_{\infty} = V^{\star}(s_1) - V^{0}(s_1)\leq \gamma -\frac{\delta}{1 - \gamma}$  (see Section F.1 below). The idea of the proof is to show that satisfying the bound given in the statement of the theorem will imply that a certain condition on the step-size.
Fix a state  $s_k$  and let  $k_0$  be the first iteration where  $Q^{k_0}(s_k, a_1) > Q^{k_0}(s_k, a_2)$ . By the above, we must have  $k \leq k_0 + 1$ , or  $k_0 \geq k - 1$ . By the proof of Theorem E, we also have  $\pi^{k_0}(a_1|s_k) \leq \alpha$  (before iteration  $k_0$ ,  $Q(s_k, \cdot)$  favors  $a_2$ , so  $\pi^{k_0}(a_1|s_k)$  has not increased compared to  $\pi^0(a_1|s_k) = \alpha$ ).
We want a  $\gamma$ -contraction at every iteration, i.e. we assume the following is satisfied:
$$
V ^ {\star} (s _ {k}) - V ^ {k _ {0} + 1} (s _ {k}) \leq \gamma^ {k _ {0} + 1} (\| V ^ {\star} - V ^ {0} \| _ {\infty} + c) \leq \gamma^ {k _ {0} + 1} (\gamma - \frac {\delta}{1 - \gamma} + c).
$$
Now, by direct computation,
$$
\begin{array}{l} V ^ {\star} (s _ {k}) - V ^ {k _ {0} + 1} (s _ {k}) = \pi^ {k _ {0} + 1} (a _ {1} | s _ {k}) \gamma (V ^ {\star} (s _ {k - 1}) - V ^ {k _ {0} + 1} (s _ {k - 1})) + \pi^ {k _ {0} + 1} (a _ {2} | s _ {2}) \frac {\gamma^ {k} - r _ {k}}{1 - \gamma^ {k}} \\ \geq \pi^ {k _ {0} + 1} (a _ {2} | s _ {2}) \frac {\gamma^ {k} - r _ {k}}{1 - \gamma} = \pi^ {k _ {0} + 1} (a _ {2} | s _ {2}) (\gamma^ {k} - \frac {\delta}{1 - \gamma}). \\ \end{array}
$$
Putting this together with the above (this is an implication as this is about the necessity rather than sufficiency), we must have:
$$
\begin{array}{l} \pi^ {k _ {0} + 1} \left(a _ {2} \mid s _ {2}\right) \left(\gamma^ {k} - \frac {\delta}{1 - \gamma}\right) \leq \gamma^ {k _ {0} + 1} \left(\gamma - \frac {\delta}{1 - \gamma} + c\right) \\ \Longrightarrow \pi^ {k _ {0} + 1} (a _ {2} | s _ {2}) \leq \frac {\gamma^ {k _ {0} + 1} (\gamma - \frac {\delta}{1 - \gamma} + c)}{(\gamma^ {k} - \frac {\delta}{1 - \gamma})} = \beta \\ \end{array}
$$
If we choose  $\delta < \frac{1}{2} (1 - \gamma)(1 - \sqrt{\gamma})\gamma^k$  then  $\beta < \sqrt{\gamma}$  and require
$$
\pi^ {k _ {0} + 1} \left(a _ {2} \mid s _ {2}\right) \leq \sqrt {\gamma}.
$$
To see this, start from  $\beta \leq \sqrt{\gamma}$ , this is equivalent to
$$
\begin{array}{l} \frac {\gamma^ {k _ {0} + 1} (\gamma - \frac {\delta}{1 - \gamma} + c)}{(\gamma^ {k} - \frac {\delta}{1 - \gamma})} \leq \sqrt {\gamma} \\ \Longleftarrow \frac {\gamma^ {k} (\gamma - \frac {\delta}{1 - \gamma} + c)}{(\gamma^ {k} - \frac {\delta}{1 - \gamma})} \leq \sqrt {\gamma} \quad \text {s i n c e} k _ {0} + 1 \geq k \\ \Longleftrightarrow \gamma^ {k} (\gamma - \frac {\delta}{1 - \gamma} + c) \leq \sqrt {\gamma} (\gamma^ {k} - \frac {\delta}{1 - \gamma}) \\ \Longleftarrow \gamma^ {k} (\gamma + c) \leq \sqrt {\gamma} \left(\gamma^ {k} - \frac {\delta}{1 - \gamma}\right) \\ \Longleftrightarrow \gamma^ {k - \frac {1}{2}} (\gamma + c) \leq \gamma^ {k} - \frac {\delta}{1 - \gamma} \\ \Longleftrightarrow \frac {\delta}{1 - \gamma} \leq \gamma^ {k - \frac {1}{2}} (\sqrt {\gamma} - \gamma - c) \\ \Longleftarrow \frac {\delta}{1 - \gamma} \leq \gamma^ {k - \frac {1}{2}} (\sqrt {\gamma} - \gamma - \frac {\sqrt {\gamma}}{1 + \sqrt {\gamma}} \frac {1 - \gamma}{2}) \quad \text {s i n c e} - c > - \frac {\sqrt {\gamma}}{1 + \sqrt {\gamma}} \frac {1 - \gamma}{2} \\ \Longleftrightarrow \frac {\delta}{1 - \gamma} \leq \gamma^ {k - \frac {1}{2}} (\sqrt {\gamma} - \gamma - \frac {\sqrt {\gamma}}{2} (1 - \sqrt {\gamma})) \\ \Longleftrightarrow \frac {\delta}{1 - \gamma} \leq \gamma^ {k - \frac {1}{2}} (\sqrt {\gamma} - \gamma - \frac {\sqrt {\gamma}}{2} + \frac {\gamma}{2}) \\ \Longleftrightarrow \frac {\delta}{1 - \gamma} \leq \gamma^ {k - \frac {1}{2}} (\frac {\sqrt {\gamma}}{2} - \frac {\gamma}{2}) \\ \Longleftrightarrow \delta \leq \frac {1}{2} \gamma^ {k} (1 - \sqrt {\gamma}) (1 - \gamma), \\ \end{array}
$$
which is the condition for  $\delta$  we imposed initially.
To achieve the above condition  $\pi^{k_0 + 1}(a_2|s_2) \leq \sqrt{\gamma}$ , recalling that  $\pi^{k_0}(a_2|s_2) \geq 1 - \alpha$ ,  $\eta_{k_0}$  has to satisfy
$$
\eta_ {k _ {0}} \geq \frac {1}{Q ^ {k _ {0}} (s _ {k} , a _ {1}) - Q ^ {k _ {0}} (s _ {k} , a _ {2})} \Big [ \log ((1 - \alpha) (1 - \sqrt {\gamma})) + K L (\tilde {\pi} _ {s _ {k}} ^ {k _ {0} + 1}, \pi_ {s _ {k}} ^ {k _ {0}}) \Big ]
$$
To see this, again start from  $\pi^{k_0 + 1}(a_2|s_2) \leq \sqrt{\gamma}$ , this is equivalent to (use  $k_0 = m$  for simplicity of notation) using the closed-form update of NPG:
$$
\begin{array}{l} \pi^ {m} \left(a _ {2} \mid s _ {2}\right) \exp \left(\eta_ {m} Q ^ {m} \left(s _ {k}, a _ {2}\right)\right) \leq \\ \sqrt {\gamma} \left(\pi^ {m} \left(a _ {2} \mid s _ {2}\right) \exp \left(\eta_ {m} Q ^ {m} \left(s _ {k}, a _ {2}\right)\right) + \pi^ {m} \left(a _ {1} \mid s _ {2}\right) \exp \left(\eta_ {m} Q ^ {m} \left(s _ {k}, a _ {1}\right)\right)\right) \\ \Longleftrightarrow \frac {1}{\sqrt {\gamma}} \leq 1 + \frac {\pi^ {m} \left(a _ {1} \mid s _ {2}\right)}{\pi^ {m} \left(a _ {2} \mid s _ {2}\right)} \exp \left(\eta_ {m} \left(Q ^ {m} \left(s _ {k}, a _ {1}\right) - Q ^ {m} \left(s _ {k}, a _ {2}\right)\right)\right) \\ \Longleftrightarrow \frac {1 - \sqrt {\gamma}}{\sqrt {\gamma}} \frac {\pi^ {m} \left(a _ {2} \mid s _ {2}\right)}{\pi^ {m} \left(a _ {1} \mid s _ {2}\right)} \leq \exp \left(\eta_ {m} \left(Q ^ {m} \left(s _ {k}, a _ {1}\right) - Q ^ {m} \left(s _ {k}, a _ {2}\right)\right)\right) \\ \Longleftrightarrow \eta_ {m} \left(Q ^ {m} \left(s _ {k}, a _ {1}\right) - Q ^ {m} \left(s _ {k}, a _ {2}\right)\right) \geq \log \left(\frac {1 - \sqrt {\gamma}}{\sqrt {\gamma}} \frac {\pi^ {m} \left(a _ {2} \mid s _ {2}\right)}{\pi^ {m} \left(a _ {1} \mid s _ {2}\right)}\right) \\ \Longleftrightarrow \eta_ {m} \geq \frac {1}{Q ^ {m} \left(s _ {k} , a _ {1}\right) - Q ^ {m} \left(s _ {k} , a _ {2}\right)} \left[ \log \left(\frac {1 - \sqrt {\gamma}}{\sqrt {\gamma}} \pi^ {m} \left(a _ {2} \mid s _ {2}\right)\right) + \log \left(\frac {1}{\pi^ {m} \left(a _ {1} \mid s _ {2}\right)}\right) \right] \\ \Longrightarrow \eta_ {m} \geq \frac {1}{Q ^ {m} \left(s _ {k} , a _ {1}\right) - Q ^ {m} \left(s _ {k} , a _ {2}\right)} \left[ \log \left((1 - \alpha) \frac {1 - \sqrt {\gamma}}{\sqrt {\gamma}}\right) + K L \left(\tilde {\pi} _ {s _ {k}} ^ {m + 1}, \pi_ {s _ {k}} ^ {m}\right) \right] \\ \Rightarrow \eta_ {m} \geq \frac {1}{Q ^ {m} \left(s _ {k} , a _ {1}\right) - Q ^ {m} \left(s _ {k} , a _ {2}\right)} \left[ \log \left((1 - \alpha) (1 - \sqrt {\gamma})\right) + K L \left(\tilde {\pi} _ {s _ {k}} ^ {m + 1}, \pi_ {s _ {k}} ^ {m}\right) \right]. \\ \end{array}
$$
As we take  $\alpha \to 0$ , the KL term will dominate. In particular, note  $\alpha < 1 - \gamma$  so  $1 - \alpha > \gamma$  and
$$
(1 - \alpha) (1 - \sqrt {\gamma}) > \gamma (1 - \sqrt {\gamma})
$$
and if we further impose the condition  $\alpha <  \gamma^{2}(1 - \sqrt{\gamma})^{2}$  then
$$
(1 - \alpha) (1 - \sqrt {\gamma}) > \sqrt {\alpha} > \sqrt {\pi^ {k _ {0}} \left(a _ {1} \mid s _ {2}\right)}
$$
and the step-size needs to satisfy the following condition:
$$
\begin{array}{l} \eta_ {k _ {0}} \geq \frac {1}{Q ^ {k _ {0}} (s _ {k} , a _ {1}) - Q ^ {k _ {0}} (s _ {k} , a _ {2})} \left[ \log \left(\sqrt {\pi^ {k _ {0}} (a _ {1} | s _ {2})}\right) + K L \left(\tilde {\pi} _ {s _ {k}} ^ {k _ {0} + 1}, \pi_ {s _ {k}} ^ {k _ {0}}\right) \right] \\ = \frac {1}{Q ^ {k _ {0}} \left(s _ {k} , a _ {1}\right) - Q ^ {k _ {0}} \left(s _ {k} , a _ {2}\right)} \left[ - \frac {1}{2} K L \left(\tilde {\pi} _ {s _ {k}} ^ {k _ {0} + 1}, \pi_ {s _ {k}} ^ {k _ {0}}\right) + K L \left(\tilde {\pi} _ {s _ {k}} ^ {k _ {0} + 1}, \pi_ {s _ {k}} ^ {k _ {0}}\right) \right] \\ = \frac {1}{2 \left(Q ^ {k _ {0}} \left(s _ {k} , a _ {1}\right) - Q ^ {k _ {0}} \left(s _ {k} , a _ {2}\right)\right)} K L \left(\tilde {\pi} _ {s _ {k}} ^ {k _ {0} + 1}, \pi_ {s _ {k}} ^ {k _ {0}}\right) \tag {18} \\ \end{array}
$$
Distinct Iterations: Note that the iteration  $k_{0}(s_{k})$  where  $Q(\cdot, s_{k})$  starts becoming bigger at  $a_{1}$  that  $a_{2}$  is distinct for each  $s_{k}$ . Fix any  $s_{k}$  and  $k_{0} = k_{0}(s_{k})$ . We have
$$
Q ^ {k _ {0}} \left(s _ {k}, a _ {1}\right) <   Q ^ {k _ {0}} \left(s _ {k}, a _ {2}\right)
$$
$$
Q ^ {k _ {0} + 1} \left(s _ {k}, a _ {2}\right) \leq Q ^ {k _ {0} + 1} \left(s _ {k}, a _ {1}\right)
$$
then  $\pi^{k_0 + 1}(a_1|s_k) \leq \pi^{k_0}(a_1|s_k) \leq \alpha$  (since  $Q^t$  points towards  $a_2$  in  $s_k$  for all  $t \leq k_0$ ). Then applying exactly the same steps as in the proof of Theorem 4.2, we have
$$
Q ^ {k _ {0} + 1} \left(s _ {k + 1}, a _ {1}\right) <   Q ^ {k _ {0} + 1} \left(s _ {k + 1}, a _ {2}\right),
$$
meaning that  $k_{0}(s_{k})$  is distinct to  $k_{0}(s_{k + 1})$ .
Upper Bounding Q-value difference: We want to upper-bound the Q-value difference appearing in the step-size condition above. We have,
$$
Q ^ {k _ {0}} (s _ {k}, a _ {2}) = \frac {r _ {k}}{1 - \gamma} = \frac {\gamma^ {k + 1} + \delta}{1 - \gamma}
$$
$$
Q ^ {k _ {0}} (s _ {k}, a _ {1}) = \gamma V ^ {k _ {0}} (s _ {k - 1}) \leq \frac {\gamma^ {k}}{1 - \gamma}.
$$
$$
\begin{array}{l} Q ^ {k _ {0}} \left(s _ {k}, a _ {1}\right) - Q ^ {k _ {0}} \left(s _ {k}, a _ {2}\right) \leq \frac {\gamma^ {k}}{1 - \gamma} - \frac {\gamma^ {k + 1} + \delta}{1 - \gamma} \\ = \gamma^ {k} - \frac {\delta}{1 - \gamma} \\ \leq \gamma^ {k}. \\ \end{array}
$$
Plugging this into the above bound (18), if the iterates of NPG are to satisfy the bound with the  $\gamma$ -rate in the statement of the theorem, the step-size must at least satisfy the following condition:
$$
\eta_ {k _ {0}} \geq \frac {1}{2 \gamma^ {k}} K L (\tilde {\pi} _ {s _ {k}} ^ {k _ {0} + 1}, \pi_ {s _ {k}} ^ {k _ {0}}),
$$
F.1 Largest sub-optimality gap at iteration 0
In this section, we prove the claim that
$$
\| V ^ {\star} - V ^ {0} \| _ {\infty} = V ^ {\star} (s _ {1}) - V ^ {0} (s _ {1}) \leq \gamma - \frac {\delta}{1 - \gamma}
$$
Proof: First of all,  $V^{\star}(s_1) - V^0 (s_1) = \pi^0 (a_2|s_1)\frac{\gamma - r_1}{1 - \gamma} = (1 - \alpha)(\gamma -\frac{\delta}{1 - \gamma})\leq \gamma -\frac{\delta}{1 - \gamma}$ . For the first part, we proceed by induction. We will use throughout that
$$
\frac {\gamma^ {k} - r _ {k}}{1 - \gamma} = \gamma^ {k} - \frac {\delta}{1 - \gamma} \leq V ^ {\star} (s _ {1}) - V ^ {0} (s _ {1}) = (1 - \alpha) (\gamma - \frac {\delta}{1 - \gamma}).
$$
This is true if (when LHS is the largest)
$$
\gamma^ {2} - \frac {\delta}{1 - \gamma} \leq (1 - \alpha) (\gamma - \frac {\delta}{1 - \gamma})
$$
which holds when
$$
\alpha \leq \frac {\gamma (1 - \gamma) ^ {2}}{\gamma (1 - \gamma) - \delta}
$$
$$
\Longleftarrow \alpha \leq 1 - \gamma
$$
Base Case:
$$
\begin{array}{l} V ^ {\star} (s _ {2}) - V ^ {0} (s _ {2}) = \alpha \gamma (V ^ {\star} (s _ {1}) - V ^ {0} (s _ {1})) + (1 - \alpha) \frac {\gamma^ {2} - r _ {2}}{1 - \gamma} \\ \leq \alpha \gamma \left(V ^ {\star} \left(s _ {1}\right) - V ^ {0} \left(s _ {1}\right)\right) + (1 - \alpha) \left(V ^ {\star} \left(s _ {1}\right) - V ^ {0} \left(s _ {1}\right)\right) \\ \leq V ^ {\star} (s _ {1}) - V ^ {0} (s _ {1}) \\ \end{array}
$$
Inductive Step: Assume true for  $k$ . Then,
$$
\begin{array}{l} V ^ {\star} (s _ {k + 1}) - V ^ {0} (s _ {k + 1}) = \alpha \gamma (V ^ {\star} (s _ {k}) - V ^ {0} (s _ {k})) + (1 - \alpha) \frac {\gamma^ {k + 1} - r _ {k + 1}}{1 - \gamma} \\ \leq \alpha \gamma \left(V ^ {\star} \left(s _ {1}\right) - V ^ {0} \left(s _ {1}\right)\right) + (1 - \alpha) \left(V ^ {\star} \left(s _ {1}\right) - V ^ {0} \left(s _ {1}\right)\right) \\ \leq V ^ {\star} (s _ {1}) - V ^ {0} (s _ {1}), \\ \end{array}
$$
G Inexact policy mirror descent and the generative model
The following Lemma from [38] controls the accuracy of the estimator  $\widehat{Q}_s^k$  specified in (10) of Section 5 with respect to  $H$  and  $M_{k}$ :
Lemma G.1 (Lemma 15 in [38]). Consider using (10) to estimate  $Q_{s}^{k}$  for all state-action pairs for  $K$  iterations of IPMD. Then for  $\delta \in (0,1)$ , if for all  $k \leq K$
$$
M _ {k} \geq \frac {\gamma^ {- 2 H}}{2} \log \left(\frac {2 K | \mathcal {S} | | \mathcal {A} |}{\delta}\right).
$$
Then with probability at least  $1 - \delta$ , we have for all  $k \leq K$
$$
\| \widehat {Q} _ {s} ^ {k} - Q _ {s} ^ {k} \| _ {\infty} \leq \frac {2 \gamma^ {H}}{1 - \gamma}.
$$
The proof of this result can be found in Lemma 15 of [38].
G.1 Proof of Theorem 5.1
This proof is similar to that of [38] (Theorem 14). It is also similar in structure to the proof of Theorem 4.1 in Section 6.
Fix a state  $s \in S$  and an integer  $k \geq 0$ . For now let's assume that our Q-estimates are  $\tau$ -accurate  $(\tau > 0)$ , i.e.
$$
\| Q ^ {k} - \widehat {Q} ^ {k} \| _ {\infty} \leq \tau
$$
for all  $k \geq 0$ . With this assumption, we have from Lemma A.5 in Appendix A.1,
$$
Q ^ {k + 1} (s, a) \geq Q ^ {k} (s, a) - \frac {2 \gamma \tau}{1 - \gamma}, \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
Now proceeding in a similar way to Section 6,
$$
\begin{array}{l} \langle \widehat {Q} _ {s} ^ {k}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {k + 1} \rangle = \langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {k + 1} \rangle + \langle \widehat {Q} _ {s} ^ {k} - Q _ {s} ^ {k}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {k + 1} \rangle \\ \geq \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} \right\rangle - \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} \right\rangle - \| \widehat {Q} _ {s} ^ {k} - Q _ {s} ^ {k} \| _ {\infty} \| \pi_ {s} ^ {\star} - \pi_ {s} ^ {k + 1} \| _ {1} \\ \geq \langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} \rangle - \langle Q _ {s} ^ {k + 1}, \pi_ {s} ^ {k + 1} \rangle - \frac {2 \gamma \tau}{1 - \gamma} - 2 \tau \\ \geq \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} \right\rangle - V ^ {k + 1} (s) - \frac {4 \gamma \tau}{1 - \gamma} \\ = \langle Q _ {s} ^ {k} - Q _ {s} ^ {\star}, \pi_ {s} ^ {\star} \rangle + V ^ {\star} (s) - V ^ {k + 1} (s) - \frac {4 \gamma \tau}{1 - \gamma} \\ \geq - \| Q _ {s} ^ {\star} - Q _ {s} ^ {k} \| _ {\infty} + V ^ {\star} (s) - V ^ {k + 1} (s) - \frac {4 \gamma \tau}{1 - \gamma} \\ \geq - \gamma \| V ^ {\star} - V ^ {k} \| _ {\infty} + V ^ {\star} (s) - V ^ {k + 1} (s) - \frac {4 \gamma \tau}{1 - \gamma}. \\ \end{array}
$$
Now again proceeding exactly as in Section 6 with this extra  $\tau$ -term using the step-size condition  $(c_k = \gamma^{2k + 1})$ , we end up with
$$
\left\| V ^ {\star} - V ^ {k + 1} \right\| _ {\infty} \leq \gamma \left\| V ^ {\star} - V ^ {k} \right\| _ {\infty} + \gamma^ {2 k + 1} + \frac {4 \gamma \tau}{1 - \gamma}.
$$
Unravelling this recursion yields
$$
\begin{array}{l} \| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \gamma^ {k} \left(\| V ^ {\star} - V ^ {0} \| _ {\infty} + \sum_ {i = 1} ^ {k} \gamma^ {- i} \gamma^ {2 (i - 1) + 1}\right) + \frac {4 \gamma \tau}{1 - \gamma} \sum_ {i = 0} ^ {k - 1} \gamma^ {i} \\ \leq \gamma^ {k} \Big (\| V ^ {\star} - V ^ {0} \| _ {\infty} + \frac {1}{1 - \gamma} \Big) + \frac {4 \gamma \tau}{(1 - \gamma) ^ {2}}. \\ \end{array}
$$
Now using the properties of the estimator (10) in Lemma G.1, we have with probability  $1 - \delta$  for all  $0 \leq k \leq K$ ,
$$
\tau = \frac {2 \gamma^ {H}}{1 - \gamma},
$$
$$
\begin{array}{l} \| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \gamma^ {k} \left(\| V ^ {\star} - V ^ {0} \| _ {\infty} + \frac {1}{1 - \gamma}\right) + \frac {8 \gamma^ {H}}{(1 - \gamma) ^ {3}} \\ \leq \frac {2}{1 - \gamma} \gamma^ {k} + \frac {8 \gamma^ {H}}{(1 - \gamma) ^ {3}}. \\ \end{array}
$$
This establishes the first bound. Now
$$
K > \frac {1}{1 - \gamma} \log \frac {4}{(1 - \gamma) \varepsilon} \Rightarrow \frac {2}{1 - \gamma} \gamma^ {k} \leq \varepsilon / 2,
$$
$$
H \geq \frac {1}{1 - \gamma} \mathrm {l o g} \frac {1 6}{(1 - \gamma) ^ {3} \varepsilon} \Rightarrow \frac {8 \gamma^ {H}}{(1 - \gamma) ^ {3}} \leq \varepsilon / 2
$$
$$
\left\| V ^ {\star} - V ^ {k} \right\| _ {\infty} \leq \varepsilon / 2 + \varepsilon / 2 = \varepsilon
$$
as required. In terms of M, we have
$$
\begin{array}{l} M \geq \frac {\gamma^ {- 2 H}}{2} \log \frac {2 K | \mathcal {S} | | \mathcal {A} |}{\delta} \\ \geq \frac {1}{2} \left(\frac {1 6}{(1 - \gamma) ^ {3} \varepsilon}\right) ^ {2} \log \frac {2 K | \mathcal {S} | | \mathcal {A} |}{\delta} \\ = \frac {1 6 ^ {2}}{2 (1 - \gamma) ^ {6} \varepsilon^ {2}} \mathrm {l o g} \frac {2 K | \mathcal {S} | | \mathcal {A} |}{\delta} \\ \end{array}
$$
and the corresponding number of calls to the sampling model, i.e. the sample complexity is (what we have shown above is actually a lower bound but can choose  $K, H, M$  so that it is of the following order),
$$
| \mathcal {S} | \cdot | \mathcal {A} | \cdot K \cdot H \cdot M = \tilde {O} \left(\frac {| \mathcal {S} | | \mathcal {A} |}{(1 - \gamma) ^ {8} \varepsilon^ {2}}\right),
$$
where the notation  $\tilde{O}()$  hides poly-logarithmic factors. This completes the proof.
H MDP examples
H.1 MDP on which distribution-mismatch coefficient scales with size of state space
We construct an MDP on which
$$
\theta_ {\rho} = \frac {1}{1 - \gamma} \bigg \| \frac {d _ {\rho} ^ {\star}}{\rho} \bigg \| _ {\infty},
$$
scales with  $|S|$ , and hence so does the iteration complexity of the bound of [38] for exact PMD.
Consider an MDP with state-space  $S = \{s_1, s_2, \dots, s_n\}$  of size  $n$  and arbitrary action space  $\mathcal{A}$ .  $s_1$  is an absorbing state giving out rewards of 1 at each time-step, regardless of the action taken, i.e
$$
p (s _ {1} | s _ {1}, a) = 1, \quad r (s _ {1}, a) = 1 \quad \forall a \in \mathcal {A}.
$$
All others states have an action, say  $a_1$ , that gives out a reward of 1 and with probability  $1 - \delta$  brings the agent to state  $s_1$  for some  $\delta > 0$  and spreads the remaining  $\delta$  probability arbitrarily amongst the other states. The other actions have arbitrary rewards strictly less than 1 associated to them, and arbitrary transition probabilities that place 0 mass on state  $s_1$ , i.e
$$
p (s _ {1} | s, a _ {1}) = 1 - \delta , \quad r (s, a _ {1}) = 1 \quad \forall s \neq s _ {1},
$$
$$
p (s _ {1} | s, a) = 0, \quad r (s, a) <   1 \quad \forall s \neq s _ {1}, \forall a \neq a _ {1}.
$$
Denote  $r_{\mathrm{max}} = \max_{s\neq s_1,a\neq a_1}r(s,a) < 1$ . The following condition ensures that  $a_1$  is the optimal action in all states,
$$
\delta \leq \frac {1 - \gamma}{\gamma} (1 - r _ {\max })
$$
so that  $\pi^{\star}(s) = a_{1}$  for all states  $s$ . To see this, consider  $s_i \neq s_1$ ,  $a_m \neq a_1$  and an arbitrary policy  $\pi$ ,
$$
\begin{array}{l} Q ^ {\pi} \left(s _ {i}, a _ {1}\right) = 1 + \gamma \left(\frac {1 - \delta}{1 - \gamma} + \sum_ {j = 2} ^ {n} p \left(s _ {j} \mid s _ {i}, a _ {1}\right) V ^ {\pi} \left(s _ {j}\right)\right) \\ \geq 1 + \gamma \frac {1 - \delta}{1 - \gamma} \\ \end{array}
$$
$$
\begin{array}{l} Q ^ {\pi} (s _ {i}, a _ {m}) = r (s _ {i}, a _ {m}) + \gamma \sum_ {j = 2} ^ {n} p (s _ {j} | s _ {i}, a _ {1}) V ^ {\pi} (s _ {j}) \\ \leq r _ {\max } + \gamma \frac {1}{1 - \gamma} \\ \end{array}
$$
and solving
$$
r _ {\max } + \gamma \frac {1}{1 - \gamma} \leq 1 + \gamma \frac {1 - \delta}{1 - \gamma}
$$
will yield the condition above.
Then for  $t \geq 1$  (abusing notation,  $s_t$  denotes the state at time  $t$ ),
$$
\begin{array}{l} \mathbb {P} ^ {\pi^ {*}} (s _ {t} = s _ {1} | s _ {0} = s) = \sum_ {s ^ {\prime}} \mathbb {P} ^ {\pi^ {*}} (s _ {t} = s _ {1}, s _ {t - 1} = s ^ {\prime} | s _ {0} = s) \\ = \sum_ {s ^ {\prime}} p \left(s _ {1} \mid s ^ {\prime}, a _ {1}\right) \mathbb {P} ^ {\pi^ {*}} \left(s _ {t - 1} = s ^ {\prime} \mid s _ {0} = s\right) \\ \geq \sum_ {s ^ {\prime}} (1 - \delta) \mathbb {P} ^ {\pi^ {*}} (s _ {t - 1} = s ^ {\prime} | s _ {0} = s) \\ = 1 - \delta \\ \end{array}
$$
$$
\begin{array}{l} d _ {\rho} ^ {\star} (s _ {1}) = (1 - \gamma) \sum_ {s} \rho (s) \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbb {P} ^ {\pi^ {\star}} \left(s _ {t} = s _ {1} \mid s _ {0} = s\right) \\ \geq (1 - \gamma) \sum_ {s} \rho (s) \sum_ {t = 1} ^ {\infty} \gamma^ {t} (1 - \delta) \\ \geq (1 - \gamma) \sum_ {s} \rho (s) \frac {\gamma}{1 - \gamma} (1 - \delta) \\ = \gamma (1 - \delta). \\ \end{array}
$$
$$
\left\| \frac {d _ {\rho} ^ {\star}}{\rho} \right\| _ {\infty} \geq \frac {d _ {\rho} ^ {\star} (s _ {1})}{\rho (s _ {1})} \geq \frac {\gamma (1 - \delta)}{\rho (s _ {1})}
$$
and depending on what  $\rho$  you consider,  $\theta_{\rho}$  can be arbitrarily large. In particular, the natural choice of the uniform starting-state distribution  $\rho(s) = 1/n$  leads to
$$
\theta_ {\rho} \geq n \frac {\gamma (1 - \delta)}{(1 - \gamma)}
$$
which gives an iteration complexity under the result of [38] for an  $\varepsilon$ -optimal policy that is
$$
n \frac {\gamma (1 - \delta)}{(1 - \gamma)} \log \frac {2}{(1 - \gamma) \varepsilon}.
$$
Recall that  $n = |\mathcal{S}|$ , so this iteration complexity scales linearly with the size of the state space.
H.2 Family of MDPs on which sub-optimality gaps can be made arbitrarily small
We present how to construct a family of MDPs on which  $\Delta^k (s)$  defined in Section 4 can be made arbitrarily small.
Consider an arbitrary MDP  $\mathcal{M}$  with state space  $S$  and action space  $\mathcal{A}$ . For each state-action pair  $(s, a) \in S \times \mathcal{A}$ , create a duplicate action  $a'$  s.t the transitions from that action in that state are the same as for the original pair, i.e
$$
p \left(s ^ {\prime} \mid s, a\right) = p \left(s ^ {\prime} \mid s, a ^ {\prime}\right) \quad \forall s ^ {\prime} \in \mathcal {S}
$$
and the reward is shifted down by  $\delta > 0$  from the original reward, i.e
$$
r (s, a ^ {\prime}) = r (s, a) - \delta .
$$