SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\eta_ {k} \geq \frac {1}{c _ {k}} \max  _ {s \in \mathcal {S}} \left\{\min  _ {\widetilde {\pi} _ {s} ^ {k + 1} \in \widetilde {\Pi} _ {s} ^ {k + 1}} D _ {h} \left(\widetilde {\pi} _ {s} ^ {k + 1}, \pi_ {s} ^ {k}\right) \right\}. \tag {5}
$$
Then we have for all  $k \geq 0$ ,
$$
\left\| V ^ {\star} - V ^ {k} \right\| _ {\infty} \leq \gamma^ {k} \left(\left\| V ^ {\star} - V ^ {0} \right\| _ {\infty} + \sum_ {i = 1} ^ {k} \gamma^ {- i} c _ {i - 1}\right). \tag {6}
$$
The sequence  $\{c_k\}_{k\in \mathbb{Z}_{\geq 0}}$  plays an important role in both the step-size constraint (5) and the bound (6). In particular, different choices will lead to different guarantees. We focus on  $c_{i} = \gamma^{2(i + 1)}c_{0}$  for some  $c_0 > 0$ , giving a step-size with a geometrically increasing component. The resulting bound is
$$
\| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \gamma^ {k} \Big (\| V ^ {\star} - V ^ {0} \| _ {\infty} + \frac {c _ {0}}{1 - \gamma} \Big),
$$
which converges linearly with the  $\gamma$ -rate, and matches the bounds of PI and VI as  $c_{0}$  goes to 0. PMD cannot do better as we will show in Theorem 4.2. We discuss other choices of  $\{c_k\}$  in Appendix C. We provide some simulations that demonstrate the behaviour of our step-size (5) and validate its benefits compared to the step-size of [38] in Appendix D.
Comparison to [38]: Linear convergence of unregularised PMD was first established by [38] under a geometrically increasing step-size  $\eta_{k} = \eta_{0} / \gamma^{k}$ . We discuss this step-size further and show the necessity of adaptivity to achieve the  $\gamma$ -rate in Section 4.2. Their rate of convergence is  $1 - \frac{1}{\theta_{\rho}}$  where  $\theta_{\rho}$  is an instance-dependent term defined as follows
$$
\theta_ {\rho} = \frac {1}{1 - \gamma} \left\| \frac {d _ {\rho} ^ {\star}}{\rho} \right\| _ {\infty},
$$
where  $d_{\rho}^{\star}$  is the visitation distribution defined in (1) under an optimal policy and  $\rho$  is the starting-state distribution to which the bound applies, i.e the bound is on  $V^{\star}(\rho) - V^{k}(\rho)$ . This  $\theta_{\rho}$  is at best  $\gamma$  when we use  $\rho$  to be the stationary distribution of the optimal policy. However, in this case, the guarantee only applies to states on the support of this stationary distribution and provides no guarantees for other states. In general, it is unclear how  $\theta_{\rho}$  may scale in a specific MDP. In particular, it is possible to construct an MDP where  $\theta_{\rho}$  scales linearly with the size of the state space  $|\mathcal{S}|$  (Appendix H.1). Though this MDP is somewhat trivial, it nonetheless illustrates how  $\theta_{\rho}$  can easily be large leading to slow rates of convergence. It is also not straightforward to obtain convergence in individual states from the bound in [38] due to the presence of  $\rho$  in the denominator of the mismatch coefficient in  $\theta_{\rho}$ . In contrast, we obtain the optimal  $\gamma$ -rate of convergence and our result holds in  $\ell_{\infty}$ -norm over all states so avoids having to deal with a starting-state distribution  $\rho$  altogether.
This distribution mismatch coefficient commonly appears in convergence bounds in the literature [19, 32, 10, 33, 3], both under exact and inexact policy evaluation. For many of these papers mentioned, removing it would be of great interest though often does not appear possible. Our results show that it is removable for the general family of PMD algorithms to obtain dimension-free linear convergence. The techniques we use may be of interest for removing this coefficient in other settings.
Comparison to [22]: The  $\gamma$ -rate was established by [22] for NPG, a specific instance of PMD for which the Bregman Divergence is the KL-divergence. The bound shown in their work is similar to the one implied by our result with  $c_{i} = \gamma^{2(i + 1)}c_{0}$ . Defining  $\Delta^k (s) = \max_{a\in \mathcal{A}}Q^k (s,a) - \max_{a\notin \mathcal{A}_s^k}Q^k (s,a)$ , the minimal sub-optimality gap in state  $s$  under  $\pi^k$ , then the step-size corresponding to their bound with the KL as Bregman Divergence is
$$
\eta_ {k} \geq \max  _ {s, \widetilde {\pi} _ {s} ^ {k + 1} \in \widetilde {\Pi} _ {s} ^ {k + 1}} \left\{\left(L _ {k} + \log | \mathcal {A} | + D (\widetilde {\pi} _ {s} ^ {k + 1}, \pi_ {s} ^ {k})\right) \frac {1}{\Delta_ {k} (s)}\right) \},
$$
where  $L_{k} = Lk$  for some constant  $L > 0$ . This highlights the connection with our step-size condition (5). In particular, they both have an adaptive component that depends linearly on the Bregman divergence between the current policy and the greedy policy and a non-adaptive component on which the bound depends. An important difference is that our step-size is independent of the sub-optimality gap  $\Delta_k(s)$ , and will be robust to situations where this gap is small. We can construct a general family of MDPs for which we can make  $\Delta_k(s)$  arbitrarily small and the step-size of [22] will correspondingly become arbitrarily large (Appendix H.2). Despite the apparent similarities with our results, their analysis is significantly different to ours as it exploits the specific closed-form update of NPG to bound the difference in value with an update of PI. Our analysis applies to PMD for a general mirror map (not just NPG) and as such does not utilize specific properties of the mirror map and does not require the analytic solution of the update to be known. Our analysis also easily extends to inexact PMD (see Section 5), which theirs does not.
Computing the step-size: The expression (5) for the step-size can be simplified. The minimum over  $\widetilde{\pi}_s^{k + 1}\in \widetilde{\Pi}_s^{k + 1}$  can be removed by taking any  $\widetilde{\pi}_s^{k + 1}\in \widetilde{\Pi}_s^{k + 1}$ . It was stated with the minimum to have the smallest condition but that is not necessary. In fact, there will often only be one greedy action, i.e.  $\mathcal{A}_s^k$  will contain a single action and there will just be one policy in  $\widetilde{\Pi}_s^{k + 1}$ . As for the maximum over  $s\in S$ , this condition is imposed because we are using the same step-size in all states for simplicity. If we allow different step-sizes in each state, then the step-size in state  $s$ , denoted  $\eta_k(s)$  would just have to satisfy (choosing any  $\widetilde{\pi}_s^{k + 1}\in \widetilde{\Pi}_s^{k + 1}$ )
$$
\eta_ {k} (s) \geq \frac {1}{c _ {k}} D _ {h} \left(\widetilde {\pi} _ {s} ^ {k + 1}, \pi_ {s} ^ {k}\right).
$$
The computational complexity of the step-size is then that of the Bregman divergence between two policies. Note that the step-size (5) is always finite. It can be unbounded when  $h$  is of Legendre type on the relative interior of  $\Delta(\mathcal{A})$  (as for the negative entropy) but then all iterates will belong to the relative interior of  $\Delta(\mathcal{A})$  and the Bregman divergences are well defined so the step-size is finite.
4.1 Optimality of PMD
We have established in Theorem 4.1 that PMD achieves a linear  $\gamma$ -rate. The following result shows that this rate is in fact optimal in a worst-case sense. The proof can be found in Appendix E.
Theorem 4.2. Fix  $n > 0$ . There exists a class of MDPs with  $|S| = 2n + 1$  and a policy  $\pi^0 \in \text{rint} \Pi$  such that running iterative updates of (4) for any positive step-size regime, we have for  $k < n$ :
$$
\left\| V ^ {\star} - V ^ {k} \right\| _ {\infty} \geq \frac {1}{2} \gamma^ {k} \left\| V ^ {\star} - V ^ {0} \right\| _ {\infty}. \tag {7}
$$
A key feature of this result is that the bound holds for  $k < n$ . For a fixed iteration budget, Theorem 4.2 implies that there exists an MDP on which PMD will not do better than the linear  $\gamma$ -rate for any step-size. The  $\gamma$ -rate for PMD that we prove in Theorem 4.1 is optimal in this sense.
Lower-bound beyond  $\mathbf{k} < \mathbf{n}$ ? For a fixed state-space size  $|\mathcal{S}| = 2n + 1$ , it is not known what lower-bound holds when  $k \geq n$ . However, PI is an instance of PMD (see Section 3.1) and thus a lower bound that scales with  $\gamma^k$  for all  $k > 0$  cannot hold since PI converges exactly in finite-iterations
(in fact with a number of iterations that scales linearly with  $|S|$  [31, Theorem 3]). However, an asymptotic lower-bound for PMD under arbitrary step-size will not hold even if we exclude PI from the class of considered algorithms. As the step-size tends to infinity, any PMD update recovers a PI update. This implies that general PMD can be arbitrarily close to PI's exact convergence for the same finite number of iterations. Thus, any lower-bound on the convergence of PMD must be limited to finite iterations. Since the finite iteration convergence of PI only scales linearly with  $|S|$  [31, Theorem 3], the number of iterations guaranteed by Theorem 4.2 has the same dependence on  $|S|$  as the number of iterations needed for exact convergence of PI. To the best of our knowledge, this lower bound on the value convergence of PMD scaling with  $\gamma^k$  is new. We expect this result may have been known for the special case of PI, though we could not find a proof of it in the literature. The works that establish a lower bound for PI do so in the setting of exact convergence to the optimal policy [18], not  $\varepsilon$ -accurate convergence, and for undiscounted MDPs [16].
Rate-Optimality Theorem 4.2 establishes that the  $\gamma$ -rate is optimal in the first  $|S|/2$  iterations. This non-asymptotic optimality is justified by the discussion above, which highlights that a rate for PMD under arbitrary step-size can only be optimal for a finite number of iterations, and specifically up until the exact convergence of PI.
There are some results on the super-linear convergence of NPG in the literature, though these apply once you have a policy within some neighbourhood of the optimal policy or value. [13] establish such a result for NPG in the regularised case, and [22] in the unregularised case under additional conditions. Theorem 4.2 does not contradict this latter result as for the MDP considered in the proof, the super-linear convergence would kick-in for iterations beyond the  $k < n$  considered here.
Lower-bound for general PG methods: The lower bound of Theorem 4.2 in fact applies to any algorithm that at each iteration increases the probability of the current greedy action. The greedy action is the action with highest action-value for the current policy. This covers algorithms more general than PMD and in particular, includes the vanilla PG algorithm.
4.2 Adaptive step-size necessity
We have established in Theorem 4.1 that PMD under an adaptive step-size achieves a linear  $\gamma$ -rate and in Theorem 4.2 that this rate is optimal for PMD. We now show the adaptivity is in fact necessary to achieve the  $\gamma$ -rate. This strengthens the notion of optimality from the previous section - both the rate and step-size are unimprovable. The proof can be found in Appendix F.
Theorem 4.3. Fix  $n > 0$  and  $\gamma > 0.2$ . There exists an MDP with state-space of size  $|\mathcal{S}| = 2n + 1$  and a policy  $\pi^0 \in \mathrm{rint} \Pi$  such that running iterative updates of NPG (PMD with  $h$  as the negative entropy) that satisfy  $\| V^{\star} - V^{k}\|_{\infty} \leq \gamma^{k}(\| V^{\star} - V^{0}\|_{\infty} + \frac{1 - \gamma}{8})$  requires
$$
\eta_ {k _ {i}} \geq K L \left(\tilde {\pi} _ {s _ {i}} ^ {k _ {i} + 1}, \pi_ {s _ {i}} ^ {k _ {i}}\right) / 2 \gamma^ {k _ {i}}, \quad \tilde {\pi} _ {s _ {i}} ^ {k _ {i} + 1} \in \widetilde {\Pi} _ {s} ^ {k + 1} \tag {8}
$$
for  $i = 1,\dots,n$  s.t  $k_{1} < k_{2} < \ldots < k_{n}$  where  $\{s_1,\dots,s_n\}$  are distinct states of the considered MDP.
This theorem states that there are at least  $n$  distinct iterations with  $n$  distinct states where the step-size has to be bigger than a quantity depending on the Bregman divergence between the current policy and its greedy policy in the considered state in order to achieve a linear  $\gamma$ -rate. This is precisely the notion of adaptivity that appears in the step-size condition of Theorem 4.1 and [22]. Theorem 4.3 shows we cannot improve on this in general and provides justification for using an adaptive step-size instead of the one from [38]. Beyond its necessity, the adaptivity of our step-size can be a strength: it is large when needed, small when not. This is illustrated by the simulations we provide in Appendix D.
Theorem 4.3 only holds for the particular case of the negative entropy (NPG). This limitation is due to technical reasons rather than fundamental ones. The proof relies on the closed form update of NPG and we don't have this for general mirror map (see Appendix F). However, this result for NPG is enough to establish the necessity of the step-size for PMD in general.
5 Sample complexity of inexact policy mirror descent under generative model
In the previous sections, we have assumed access to the state-action values  $Q_{s}^{k}$  to carry out the PMD update. In Inexact PMD (IPMD), we replace  $Q_{s}^{k}$  with an estimate  $\widehat{Q}_{s}^{k}$  giving the update
$$
\pi_ {s} ^ {k + 1} = \operatorname {a r g m i n} _ {p \in \Delta (\mathcal {A})} \left\{- \eta_ {k} \langle \widehat {Q} _ {s} ^ {k}, p \rangle + D _ {h} (p, \pi_ {s} ^ {k}) \right\}. \tag {9}
$$
Similarly to the exact case, IPMD iteratively applies update (9) for some sequence of  $\eta_{k} > 0$  and initial policy  $\pi^0\in \mathrm{rint}\Pi$ , this time only assuming access to an inexact estimator of  $Q^{k}$ .
We consider the setting of a generative model [21], which is a sampling model where we can draw samples from the transition probabilities  $p(\cdot |s,a)$  for any pair  $(s,a)$ . We borrow an estimator common in the literature (see e.g. [38], [25]): for all state-action pairs  $(s,a)$ , draw  $M_{k}$  trajectories of length or horizon  $H$ , i.e. samples of the form  $\left((s_0^{(i)},a_0^{(i)}),(s_1^{(i)},a_1^{(i)}),\dots,(s_{H - 1}^{(i)},a_{H - 1}^{(i)})\right)_{i = 1,\ldots ,M_k}$ , where  $a_{t}^{(i)}$  is drawn from  $\pi^k (\cdot |s_t^{(i)})$ ,  $s_{t + 1}^{(i)}$  is drawn from  $p(\cdot |s_t^{(i)},a_t^{(i)})$  and  $(s_0^{(i)},a_0^{(i)}) = (s,a)$ . Using these samples, we can define a truncated Monte-Carlo estimate of the values as follows,
$$
\widehat {Q} ^ {k} (s, a) = \frac {1}{M _ {k}} \sum_ {i = 1} ^ {M _ {k}} \widehat {Q} _ {(i)} ^ {k} (s, a), \quad \text {w h e r e} \quad \widehat {Q} _ {(i)} ^ {k} (s, a) = \sum_ {t = 0} ^ {H - 1} \gamma^ {t} r \left(s _ {t} ^ {(i)}, a _ {t} ^ {(i)}\right). \tag {10}
$$
We use these  $\widehat{Q}^k (s,a)$  to replace  $Q^{k}(s,a)$  in the PMD update step. [38] present a bound on the accuracy of this estimator which is restated in Appendix G. Following the same ideas as [38], we can extend Theorem 4.1 to the inexact setting. The following theorem establishes a sample complexity result, which is the sufficient number of calls to the generative model to obtain an  $\varepsilon$ -optimal policy. For simplicity, we focus on the step-size following from the choice  $c_{i} = \gamma^{2(i + 1)}$ .
Theorem 5.1. Consider applying iterative updates of (9) using the  $Q$ -estimator in (10) given access to a generative model with  $\pi^0 \in \text{rint } \Pi$  and step-sizes satisfying for all  $k \geq 0$  (with the definitions of  $\mathcal{A}_s^k$  and  $\widetilde{\Pi}_s^{k+1}$  suitably adjusted with  $\widehat{Q}_s^k$  instead of  $Q_s^k$ ),
$$
\eta_ {k} \geq \max  _ {s \in \mathcal {S}} \left\{\min  _ {\widetilde {\pi} _ {s} ^ {k + 1} \in \widetilde {\Pi} _ {s} ^ {k + 1}} \frac {D (\widetilde {\pi} _ {s} ^ {k + 1} , \pi_ {s} ^ {k})}{\gamma^ {2 k + 1}} \right\}.
$$
Fix  $\varepsilon > 0$ . For any  $\delta \in (0,1)$ , suppose the following are satisfied for all  $k \geq 0$ ,
$$
K > \frac {1}{1 - \gamma} \log {\frac {4}{(1 - \gamma) \varepsilon}}, \quad H \geq \frac {1}{1 - \gamma} \log {\frac {1 6}{(1 - \gamma) ^ {3} \varepsilon}} \quad a n d \quad M _ {k} = M \geq \frac {\gamma^ {- 2 H}}{2} \log {\frac {2 K | \mathcal {S} | | \mathcal {A} |}{\delta}}.
$$
Then with probability at least  $1 - \delta$ ,  $\|V^{\star} - V^{k}\|_{\infty} \leq \gamma^{k}\left(\|V^{\star} - V^{0}\|_{\infty} + \frac{1}{1 - \gamma}\right) + \frac{8}{(1 - \gamma)^{3}}\gamma^{H} < \varepsilon$ . Choosing  $K, H$  and  $M$  to be tight to their lower-bounds, the corresponding sample complexity is  $\tilde{O}\left(\frac{|S||\mathcal{A}|}{(1 - \gamma)^{8}\varepsilon^{2}}\right)$ , where the notation  $\tilde{O}()$  hides poly-logarithmic factors.
The proof can be found in Appendix G.1. The sample complexity established by [38] (Theorem 16) under a generative model and the same Q-estimator is  $\tilde{O}\Big(\frac{|S||\mathcal{A}|}{(1 - \gamma)^{8\varepsilon^2}}\left\| \frac{d_\rho^\star}{\rho}\right\|_\infty^3\Big)$ .
In their work, [38] stresses the interest in reducing the dependence on both  $1/(1 - \gamma)$  and the distribution mismatch coefficient in order to scale the PMD guarantees to more relevant settings such as function approximation. Theorem 5.1 partially resolves this matter by removing the dependence on the distribution mismatch coefficient, which may scale with the size of the state space (Appendix H.1). This makes the result dimension-optimal, which is crucial when scaling the results to large or infinite state or action spaces. The dependence on  $1/(1 - \gamma)$  remains distant from the  $1/(1 - \gamma)^{3}$  lower-bound of [6] (see Section 2). Whether this can be reached by PMD methods remains open, though using a more suitable Q-estimator than (10) with our step size regime and analysis, which extends to arbitrary Q-estimators, could bring the sample complexity closer to this.
In this section, we present the proof of Theorem 4.1. A key component in establishing the  $\gamma$ -rate is avoiding the performance difference lemma that we state in Appendix B. In prior works, the quantity
that we are looking to bound  $V^{\star}(\rho) - V^{k}(\rho)$  arises through the performance difference lemma. In particular, [38] use the lemma on  $\mathbb{E}_{s\sim d_{\rho}^{\star}}[\langle Q_s^k,\pi_s^\star -\pi_s^{k + 1}\rangle ]$ , which introduces a distribution mismatch coefficient in order to get a recursion. On the other hand, we extract the value sub-optimalitys  $V^{\star}(s) - V^{k}(s)$  and  $\| V^{\star} - V^{k + 1}\|_{\infty}$  directly from  $\langle Q_s^k,\pi_s^\star -\pi_s^{k + 1}\rangle$  in (12). This leads to an elegant analysis that may be of interest in the study of other methods, and ultimately allows us to remove distribution mismatch factors and obtain an exact  $\gamma$ -rate.
Proof of Theorem 4.1: Fix  $s \in S$  and  $k \geq 0$ . From Lemma A.2, we have that  $\langle Q_s^k, \pi_s^{k+1} \rangle \leq \langle Q_s^{k+1}, \pi_s^{k+1} \rangle = V^{k+1}(s)$ . This decouples the dependencies on  $\pi^k$  and  $\pi^{k+1}$  below and is one of the ingredients that allows us to bypass the performance difference lemma. Using this,
$$
\begin{array}{l} \langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {k + 1} \rangle \geq \langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} \rangle - V ^ {k + 1} (s) = \langle Q _ {s} ^ {k} - Q _ {s} ^ {\star}, \pi_ {s} ^ {\star} \rangle + \langle Q _ {s} ^ {\star}, \pi_ {s} ^ {\star} \rangle - V ^ {k + 1} (s) \\ \geq - \| Q _ {s} ^ {\star} - Q _ {s} ^ {k} \| _ {\infty} + V ^ {\star} (s) - V ^ {k + 1} (s), \tag {11} \\ \end{array}
$$
where the last step uses Hölder's inequality. Now we use that the difference in state-action values of different policies for the same state-action pair propagates the error to the next time-step, which is discounted by a factor of  $\gamma$ . Formally, for any state-action pair  $(s,a) \in S \times \mathcal{A}$ ,
$$
\begin{array}{l} Q ^ {\star} (s, a) - Q ^ {k} (s, a) = \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) \left(V ^ {\star} \left(s ^ {\prime}\right) - V ^ {k} \left(s ^ {\prime}\right)\right) \\ \leq \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) \| V ^ {\star} - V ^ {k} \| _ {\infty} = \gamma \| V ^ {\star} - V ^ {k} \| _ {\infty}, \\ \end{array}
$$
which is the same phenomenon that is responsible for the contraction of the Bellman operator. This gives  $\| Q_s^\star -Q_s^k\|_\infty \leq \gamma \| V^\star -V^k\|_\infty$ . Plugging into Equation (11),
$$
V ^ {\star} (s) - V ^ {k + 1} (s) - \gamma \| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {k + 1} \rangle .
$$
The rest of the proof relies on making the right-hand side of the above arbitrarily small by taking a large enough step size. Choose any greedy policy with respect to  $Q_{s}^{k}$ ,  $\widetilde{\pi}_{s}^{k + 1} \in \widetilde{\Pi}_{s}^{k + 1}$
$$
\begin{array}{l} V ^ {\star} (s) - V ^ {k + 1} (s) - \gamma \| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {\star} - \pi_ {s} ^ {k + 1} \right\rangle (12) \\ \leq \left\langle Q _ {s} ^ {k}, \widetilde {\pi} _ {s} ^ {k + 1} - \pi_ {s} ^ {k + 1} \right\rangle (13) \\ \end{array}
$$
where we use that  $\widetilde{\pi}_s^{k + 1}$  is greedy with respect to  $Q_{s}^{k}$ . We then apply Lemma A.1 or (16) to  $p = \widetilde{\pi}_s^{k + 1}$ ,
$$
\langle Q _ {s} ^ {k}, \widetilde {\pi} _ {s} ^ {k + 1} - \pi_ {s} ^ {k + 1} \rangle \leq \frac {D (\widetilde {\pi} _ {s} ^ {k + 1} , \pi_ {s} ^ {k}) - D (\widetilde {\pi} _ {s} ^ {k + 1} , \pi_ {s} ^ {k + 1}) - D (\pi_ {s} ^ {k + 1} , \pi_ {s} ^ {k})}{\eta_ {k}} \leq D (\widetilde {\pi} _ {s} ^ {k + 1}, \pi_ {s} ^ {k}) / \eta_ {k}.
$$
Combining with (13) and noting that this holds for any  $\widetilde{\pi}_s^{k + 1}\in \widetilde{\Pi}_s^{k + 1}$ , we have
$$
V ^ {\star} (s) - V ^ {k + 1} (s) - \gamma \| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \frac {1}{\eta_ {k}} \min  _ {\widetilde {\pi} _ {s} ^ {k + 1} \in \widetilde {\Pi} _ {s} ^ {k + 1}} D (\widetilde {\pi} _ {s} ^ {k + 1}, \pi_ {s} ^ {k}) \leq c _ {k}
$$
from the step-size condition in the statement of the theorem. Rearranging and recalling that  $s$  and  $k$  were arbitrary, we can choose  $s$  where  $V^{\star}(s) - V^{k + 1}(s)$  reaches its maximum value. We get
$$
\left\| V ^ {\star} - V ^ {k + 1} \right\| _ {\infty} \leq \gamma \left\| V ^ {\star} - V ^ {k} \right\| _ {\infty} + c _ {k},
$$
and unravelling this recursion completes the proof.
In this paper, we have shown that the general family of exact policy mirror descent algorithms in tabular MDPs under an adaptive step-size achieve the same dimension-free linear  $\gamma$ -rate of convergence of classical algorithms such as policy iteration. We provide matching lower-bounds that establish this rate as optimal for PMD and the adaptive step-size as necessary. We exploit a new approach to study the convergence of PMD, for which avoiding the performance difference lemma is a key element. Though the focus of our work is on the exact policy evaluation setting, the analysis naturally extends to the inexact setting, given access to an estimator of the action-value of a policy. We provide a result for a simple estimator under a generative model that improves upon the best-known sample complexity, although it still does not match the lower bound. Our method is general and applies to any estimator, meaning our result could be improved by a better estimator. Exploiting further algorithmic properties of PMD in the inexact setting may be needed to bridge the gap to the optimal sample complexity, and determine if PMD can match the lower bound in the inexact setting.
EJ is funded by EPSRC through the Modern Statistics and Statistical Machine Learning (StatML) CDT (grant no. EP/S023151/1).
We would like to thank the reviewers and meta-reviewers for their time and feedback, which led to a better version of this paper.