SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellert Weisz. POLITEX: Regret bounds for policy iteration using expert prediction. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3692-3702. PMLR, 2019.
[2] Alekh Agarwal, Sham Kakade, and Lin F. Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 67-83. PMLR, 2020.
[3] Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
[4] Carlo Alfano and Patrick Rebeschini. Linear convergence for natural policy gradient with log-linear policy parametrization. 2022. Preprint, arXiv: 2209.15382.
[5] Carlo Alfano, Rui Yuan, and Patrick Rebeschini. A novel framework for policy mirror descent with general parametrization and linear convergence. 2023. Preprint, arXiv: 2301.13139.
[6] Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert Kappen. Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Machine Learning, 91 (3):325-349, 2013.
[7] Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. Clustering with bregman divergences. Journal of Machine Learning Research, 6(58):1705-1749, 2005.
[8] Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
[9] Richard Bellman. Dynamic Programming. Dover Publications, 1957.
[10] Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. 2019. Preprint, arXiv: 1906.01786.
[11] Jalaj Bhandari and Daniel Russo. On the linear convergence of policy gradient methods for finite mdps. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2386-2394. PMLR, 2021.
[12] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3-4), 2015.
[13] Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization. Operations Research, 70(4): 2563-2578, 2021.
[14] Gong Chen and Marc Teboulle. Convergence Analysis of a Proximal-Like Minimization Algorithm Using Bregman Functions. SIAM Journal on Optimization, 3(3):538-543, 1993.
[15] Zaiwei Chen and Siva Theja Maguluri. Sample complexity of policy-based methods under off-policy sampling and linear function approximation. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 11195–11214. PMLR, 2022.
[35] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, second edition, 2018.
[36] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, 1999.
[37] M. J. Wainwright. Variance-reduced  $Q$ -learning is minimax optimal. Technical report, UC Berkeley, 2019. Preprint, arxiv:1906.04697.
[38] Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1-36, 2022.
[39] Rui Yuan, Robert M. Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 3332-3380. PMLR, 2022.
[40] Rui Yuan, Simon Shaolei Du, Robert M. Gower, Alessandro Lazaric, and Lin Xiao. Linear convergence of natural policy gradient methods with log-linear policies. In The Eleventh International Conference on Learning Representations, 2023.
[41] Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D. Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization, 33(2):1061-1091, 2023.
A Properties of PMD
We present lemmas relevant to the analysis of PMD. Key to the analysis is the Three-Point Descent Lemma, that relates the improvement of the proximal gradient update compared to an arbitrary point. It originally comes from [14] (Lemma 3.2) where a proof can be found, though we use a slightly modified version from [38] (Lemma 6).
Lemma A.1 (Three-Point Descent Lemma, Lemma 6 in [38]). Suppose that  $\mathcal{C} \subset \mathbb{R}^n$  is a closed convex set,  $\phi : \mathcal{C} \to \mathbb{R}$  is a proper, closed convex function,  $D_h(\cdot, \cdot)$  is the Bregman divergence generated by a function  $h$  of Legendre type and rint  $\text{dom} h \cap \mathcal{C} \neq \emptyset$ . For any  $x \in \text{rint} \text{dom} h$ , let
$$
x ^ {+} = \operatorname {a r g m i n} _ {u \in C} \left\{\phi (u) + D _ {h} (u, x) \right\}. \tag {14}
$$
Then  $x^{+}\in \mathsf{rint}$  dom  $h\cap C$  and  $\forall u\in C$
$$
\phi \left(x ^ {+}\right) + D _ {h} \left(x ^ {+}, x\right) \leq \phi (u) + D _ {h} (u, x) - D _ {h} \left(u, x ^ {+}\right) \tag {15}
$$
The update (4) of PMD is an instance of the proximal minimisation (14) with  $\mathcal{C} = \Delta (\mathcal{A})$ ,  $x = \pi_s^k$  and  $\phi (x) = -\eta_k\langle Q_s^k,x\rangle$ . Plugging these into (15), Lemma A.1 relates the decrease in the proximal objective of  $\pi_s^{k + 1}$  to any other policy, i.e.  $\forall p\in \Delta (\mathcal{A})$
$$
- \eta_ {k} \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} \right\rangle + D _ {h} \left(\pi_ {s} ^ {k + 1}, \pi_ {s} ^ {k}\right) \leq - \eta_ {k} \left\langle Q _ {s} ^ {k}, p \right\rangle + D _ {h} \left(p, \pi_ {s} ^ {k}\right) - D _ {h} \left(p, \pi_ {s} ^ {k + 1}\right). \tag {16}
$$
This equation is key to the analysis in Section 6. In particular, it allows us to prove the following lemma regarding the monotonic improvement in action-value of PMD iterates. This is an extension of Lemma 7 in [38].
Lemma A.2. Consider the policies produced by the iterative updates of PMD in (4). Then for any  $k > 0$ ,
$$
Q ^ {k + 1} (s, a) \geq Q ^ {k} (s, a), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
A.1 Proof of Lemma A.2
We first present Lemma 7 from [38], from which Lemma A.2 almost immediately follows.
Lemma A.3 (Descent Property of PMD, Lemma 7 in [38]). Consider the policies produced by the iterative updates of PMD in (4). Then for any  $k \geq 0$
$$
\left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \right\rangle \geq 0, \quad \forall s \in \mathcal {S},
$$
$$
V ^ {k + 1} (\rho) \geq V ^ {k} (\rho), \quad \forall \rho \in \Delta (\mathcal {S}).
$$
Proof. From [38]. Recall that the Three-Point Descent Lemma states that  $\forall p\in \Delta (\mathcal{A})$
$$
- \eta_ {k} \langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} \rangle + D _ {h} \left(\pi_ {s} ^ {k + 1}, \pi_ {s} ^ {k}\right) \leq - \eta_ {k} \langle Q _ {s} ^ {k}, p \rangle + D _ {h} \left(p, \pi_ {s} ^ {k}\right) - D _ {h} \left(p, \pi_ {s} ^ {k + 1}\right).
$$
Using this with  $p = \pi_s^k$
$$
D _ {h} \left(\pi_ {s} ^ {k}, \pi_ {s} ^ {k + 1}\right) + D _ {h} \left(\pi_ {s} ^ {k + 1}, \pi_ {s} ^ {k}\right) \leq \eta_ {k} \langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \rangle
$$
and since the Bregman divergences are none-negative and  $\eta_{k} > 0$
$$
0 \leq \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \right\rangle
$$
and the result follows by an application of the performance difference lemma (Appendix B)
$$
\begin{array}{l} V ^ {k + 1} (\rho) - V ^ {k} (\rho) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {k + 1}} \left[ \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \right\rangle \right] \\ \geq 0. \\ \end{array}
$$
Note that we use the performance difference lemma here because it gives a simple concise proof, but we do not actually need to. To maintain our claim that we avoid the use of the performance difference lemma, we can get the same result without it. We sketch how to do this as follows. From the first part of the lemma, we have
$$
\langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} \rangle \geq \langle Q _ {s} ^ {k}, \pi_ {s} ^ {k} \rangle = V ^ {k} (s),
$$
in all states  $s$ . Now note that the left hand side above is
$$
\begin{array}{l} \langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} \rangle = \sum_ {a} \pi^ {k + 1} (a | s) Q ^ {k} (s, a) \\ = \sum_ {a} \pi^ {k + 1} (a | s) \left(r (s, a) + \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) V ^ {k} \left(s ^ {\prime}\right)\right) \\ \end{array}
$$
and we can then apply  $\langle Q_{s'}^k,\pi_{s'}^{k + 1}\rangle \geq V^k (s')$  at state  $s^\prime$  ..
$$
\begin{array}{l} V ^ {k} (s) \leq \sum_ {a} \pi^ {k + 1} (a | s) \left(r (s, a) + \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) V ^ {k} \left(s ^ {\prime}\right)\right) \\ \leq \sum_ {a} \pi^ {k + 1} (a | s) \left(r (s, a) + \gamma \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, a\right) \sum_ {a ^ {\prime}} \pi^ {k + 1} \left(a ^ {\prime} \mid s ^ {\prime}\right) \left(r \left(s ^ {\prime}, a ^ {\prime}\right) + \gamma \sum_ {s ^ {\prime \prime}} p \left(s ^ {\prime \prime} \mid s ^ {\prime}, a ^ {\prime}\right) V ^ {k} \left(s ^ {\prime \prime}\right)\right)\right) \\ \end{array}
$$
and proceeding iteratively in the limit you get exactly  $V^{k + 1}(s)$ .
Since Lemma A.3 holds for any  $\rho \in \Delta(S)$ , it guarantees that the value in each state is non-decreasing for an update of PMD, i.e. for all  $s \in S$ ,
$$
V ^ {k + 1} (s) - V ^ {k} (s) \geq 0.
$$
Using this, we get
$$
Q ^ {k + 1} (s, a) - Q ^ {k} (s, a) = \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} p \left(s ^ {\prime} \mid s, a\right) \left(V ^ {k + 1} \left(s ^ {\prime}\right) - V ^ {k} \left(s ^ {\prime}\right)\right) \geq 0,
$$
A.2 Extension of Lemma A.2 to inexact setting:
As in the exact case, we first present Lemma 12 from [38] which is the extension of Lemma A.3 to the inexact case. We note that in the inexact case, we lose the monotonic increase of values due to the inaccuracy in our estimate  $\widehat{Q}^k$  of  $Q_s^k$ .
Lemma A.4. Consider the policies produced by the iterative updates of IPMD in (9). For any  $k \geq 0$ , if  $\| \widehat{Q}_s^k - Q_s^k \|_{\infty} \leq \tau$ , then
$$
\langle \widehat {Q} _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \rangle \geq 0, \quad \forall s \in \mathcal {S},
$$
$$
V ^ {k + 1} (\rho) \geq V ^ {k} (\rho) - \frac {2 \tau}{1 - \gamma}, \quad \forall \rho \in \Delta (\mathcal {S}).
$$
Proof. From [38]. The Three-Point Descent Lemma applied to the IPMD update (9) gives  $\forall p \in \Delta(\mathcal{A})$ ,
$$
- \eta_ {k} \langle \widehat {Q} _ {s} ^ {k}, \pi_ {s} ^ {k + 1} \rangle + D _ {h} \left(\pi_ {s} ^ {k + 1}, \pi_ {s} ^ {k}\right) \leq - \eta_ {k} \langle \widehat {Q} _ {s} ^ {k}, p \rangle + D _ {h} \left(p, \pi_ {s} ^ {k}\right) - D _ {h} \left(p, \pi_ {s} ^ {k + 1}\right).
$$
$$
D _ {h} \left(\pi_ {s} ^ {k}, \pi_ {s} ^ {k + 1}\right) + D _ {h} \left(\pi_ {s} ^ {k + 1}, \pi_ {s} ^ {k}\right) \leq \eta_ {k} \langle \widehat {Q} _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \rangle
$$
and since the Bregman divergences are none-negative and  $\eta_k > 0$
$$
0 \leq \langle \widehat {Q} _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \rangle ,
$$
which proves the first inequality. Now we cannot use the above inequality directly with the performance difference lemma since  $\widehat{\mathcal{Q}}_s^k$  is not the true action-value. Instead, we have
$$
\begin{array}{l} V ^ {k + 1} (\rho) - V ^ {k} (\rho) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {k + 1}} \left[ \left\langle Q _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \right\rangle \right] \\ = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {k + 1}} \left[ \langle Q _ {s} ^ {k} - \widehat {Q} _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \rangle + \langle \widehat {Q} _ {s} ^ {k}, \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \rangle \right] \\ \geq \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {k + 1}} \left[ - \| Q _ {s} ^ {k} - \widehat {Q} _ {s} ^ {k} \| _ {\infty} \| \pi_ {s} ^ {k + 1} - \pi_ {s} ^ {k} \| _ {1} \right] \\ \geq \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {k + 1}} \left[ - 2 \tau \right] \\ = - \frac {2 \tau}{1 - \gamma} \\ \end{array}
$$
Using the above lemma, we can state and prove the extension of Lemma A.2 to the inexact setting.
Lemma A.5. Consider the policies produced by the iterative updates of IPMD in (9). For any  $k \geq 0$ , if  $\| \widehat{Q}_s^k - Q_s^k \|_{\infty} \leq \tau$ , then
$$
\widehat {Q} ^ {k + 1} (s, a) \geq \widehat {Q} ^ {k} (s, a) - \frac {2 \tau \gamma}{1 - \gamma}, \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
Proof. As in the exact case, since Lemma A.4 holds for any  $\rho \in \Delta(S)$ , it applies to each state, i.e. for all  $s \in S$ ,
$$
V ^ {k + 1} (s) - V ^ {k} (s) \geq - \frac {2 \tau}{1 - \gamma}.
$$
Using this, we immediately have
$$
Q ^ {k + 1} (s, a) - Q ^ {k} (s, a) = \gamma \sum_ {s ^ {\prime} \in \mathcal {S}} p (s ^ {\prime} | s, a) \Big (V ^ {k + 1} (s ^ {\prime}) - V ^ {k} (s ^ {\prime}) \Big) \geq \frac {- 2 \tau \gamma}{1 - \gamma},
$$
B Performance difference lemma
Lemma B.1 (Performance Difference Lemma). For any  $\pi, \pi' \in \Pi$ , we have
$$
V ^ {\pi} (\rho) - V ^ {\pi^ {\prime}} (\rho) = \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\rho} ^ {\pi}} \Big [ \langle Q _ {s} ^ {\pi^ {\prime}}, \pi_ {s} - \pi_ {s} ^ {\prime} \rangle \Big ].
$$
The performance difference lemma [19] is a property that relates the difference in values of policies to the policies themselves. The proof can be found in their paper under Lemma 6.1.
C Guarantees of Theorem 4.1 for various step-size choices
We give here two more choices of  $\{c_k\}_{k\in \mathbb{Z}_{\geq 0}}$  for the step-size 5 of PMD and their corresponding guarantees from Theorem 4.1:
-  $c_{i} = c_{0}$  for some  $c_{0} > 0$  yields a step-size with a constant component. The resulting bound is
$$
\| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \gamma^ {k} \| V ^ {\star} - V ^ {0} \| _ {\infty} + \frac {c _ {0}}{1 - \gamma},
$$
which converges linearly up to some accuracy controlled by  $c_{0}$ .
-  $c_{i} = \gamma^{i + 1}c_{0}$  for some initial  $c_{0} > 0$  will yield a step-size with a component that is geometrically increasing as in [38], though at a slower rate than the one discussed in Section 4. The resulting bound is
$$
\| V ^ {\star} - V ^ {k} \| _ {\infty} \leq \gamma^ {k} \Big (\| V ^ {\star} - V ^ {0} \| _ {\infty} + k c _ {0} \Big),
$$
which converges linearly with the sought-for  $\gamma$ -rate, though in early iterations the  $k$  factor may dominate.
D Simulations
In this Section, we present simulations (see Figures 1-4 below) made by running NPG (PMD with negative-entropy mirror map, see [22]) on the hard MDP used to prove Theorem 4.2 - see Appendix E for the details of the construction. We use  $n = 25$  so  $|S| = 51$  and  $|\mathcal{A}| = 2$ . We use  $\delta = (1 - \gamma)\gamma^n / 100$ ,  $\gamma = 0.99$ ,  $\pi^0(a_1|s) = \alpha$  (note that  $a_1$  is the optimal action in each state). In each of the plots, we compare the performance of NPG using our step-size (5) from Theorem 4.1 (denoted Adaptive in the plots and by ADA in the discussion below) with  $c_k = \gamma^{2k}$  and  $c_0 = 1$  and the step-size of [38]  $\eta_k = \eta_0 / \gamma^k$  (denoted by INC in the discussion below) for a fixed  $\eta_0 = 1$ . In each of the plots, the left plot is  $\| V^\star - V^k \|_\infty$  against iterations and the right plot is  $\eta_k$  against iterations.
In all plots, the green curve represents the  $\gamma$ -rate. In particular it is the curve of  $y = \gamma^x$ .
With fixed initial step-size  $\eta_0 = 1$  for INC, the convergence of NPG with INC can be made arbitrarily slow in early iterations by choosing  $\alpha (= \pi^0 (a_1|s))$  closer to 0 (Figures 1-3). The convergence of NPG with ADA is unaffected by small values of  $\alpha$  because it adapts to them.
These observations agree with the theoretical results we established. Our adaptive step-size (5) achieves the  $\gamma$ -rate (Theorem 4.1) and this is optimal for iterations  $k$  up until  $n = 25$  (Theorem 4.2). The adaptivity in the step-size is necessary to achieve the  $\gamma$ -rate in these early iterations (Theorem 4.3).
In later iterations (Figure 1: iterations  $\approx 100$ ). Figure 2: iterations  $\approx 200$ ), INC has faster convergence than ADA. In these later iterations, acting greedily with respect to the action value of the current policy gives the optimal policy so INC converges faster because the step-size is larger (see Figure 1-2 right plot). This suggests combining ADA and INC by taking the maximum of both (Figure 4), yielding faster convergence than both. This is encouraging for future-work on super-linear convergence (as discussed in [38]) beyond the optimal  $\gamma$ -rate valid up to a finite iteration (see Section 4.1).
Figure 1:  $\alpha = 10^{-1}$ . Green curve:  $y = \gamma^x$ .
Figure 2:  $\alpha = 10^{-10}$ . Green curve:  $y = \gamma^x$ .
Figure 3:  $\alpha = 10^{-50}$ . Green curve:  $y = \gamma^{x}$
Figure 4:  $\alpha = 10^{-10}$ .  $\eta_{k} = \max (\text{Adaptive}, \eta_{0} / \gamma^{k})$  means the step-size is chosen as the largest between our "Adaptive" and the geometrically increasing of [38]. Green curve:  $y = \gamma^x$ .
Figure 5: Example MDP used in the proof of Theorem 4.2
E Proof of Theorem 4.2
Fix  $n > 0$  and  $\delta \in (0, (1 - \gamma)\gamma^n)$ . Consider the MDP shown in Figure 5. The state space is  $S = \{s_0, s_1, s_1', \dots, s_n, s_n'\}$  and the action space is  $A = \{a_1, a_2\}$ . There is a chain of states of length  $n + 1$  with the states indexed from 0 to  $n$ . The left-most state  $(s_0)$  is absorbing with reward  $+1$ . In the other states in the chain  $(s_i$  for  $i = 1, \dots, n$ ), the agent can take action  $a_1$  and move left (to  $s_{i-1}$ ) with reward of 0, or take action  $a_2$  and move to an additional absorbing state unique to the state it is currently in  $(s_i')$  with reward  $r_i = \gamma^{i+1} + \delta$  (that the agent also receives in that state for all future time-steps). Summarising, we have for  $1 \leq i \leq n$
$$
p \left(s _ {i - 1} \mid s _ {i}, a _ {1}\right) = 1, \quad r \left(s _ {i}, a _ {1}\right) = 0,
$$
$$
p \left(s _ {i} ^ {\prime} \mid s _ {i}, a _ {2}\right) = 1, \quad r \left(s _ {i}, a _ {2}\right) = r _ {i} = \gamma^ {i + 1} + \delta ,
$$
$$
p (s _ {i} ^ {\prime} | s _ {i} ^ {\prime}, a) = 1, \quad r (s _ {i} ^ {\prime}, a) = r _ {i} = \gamma^ {i + 1} + \delta \quad \forall a \in \mathcal {A}.
$$
The value of  $\delta$  is carefully restricted so that the optimal action in all the states of the chain is  $a_1$ . The proof will consist in showing that if the agent starts with an initial policy that places most probability mass on the sub-optimal action  $a_2$ , then it has to learn that  $a_1$  is the optimal action in the state directly to the left before it can start switching from action  $a_2$  to  $a_1$  in the current state. And this can at best happen one iteration at a time starting starting from the left-most state. In particular, we consider  $\pi^0$  s.t  $\pi^0 (a_1|s) = \alpha$ ,  $\pi^0 (a_2|s) = 1 - \alpha$  for all states and some  $\alpha$  s.t  $0 < \alpha \leq \delta (1 - \gamma)$ . We make the following claim from which the result will follow straightforwardly.
Claim: Fix  $k < n$ . The policies produced by PMD satisfy  $\pi^k (a_1|s_i) \leq \alpha$  for  $k < i \leq n$ .
We prove this claim by induction.
Base Case: We want to show that  $\pi^1(a_1|s_i) \leq \alpha$  for  $i > 1$ . We do this by showing that  $Q^0(s_i, a_1) \leq Q^0(s_i, a_2)$  for  $i > 1$  so that the probability of  $\pi^1(a_1|s_i)$  cannot increase w.r.t  $\pi^0(a_1|s_i)$ , which is  $\alpha$
(this follows from  $\langle Q_s^k,\pi_s^{k + 1} - \pi_s^k\rangle \geq 0$  for all iterations of PMD). We have:
$$
Q ^ {0} \left(s _ {i}, a _ {1}\right) = \gamma V ^ {0} \left(s _ {i - 1}\right)
$$
$$
\begin{array}{l} = \gamma \left(\alpha Q ^ {0} \left(s _ {i - 1}, a _ {1}\right) + (1 - \alpha) Q ^ {0} \left(s _ {i - 1}, a _ {2}\right)\right) \\ \leq \gamma \left(\alpha \frac {\gamma^ {i - 1}}{1 - \gamma} + \frac {r _ {i - 1}}{1 - \gamma}\right) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \gamma \Big (\delta (1 - \gamma) \frac {\gamma^ {i - 1}}{1 - \gamma} + \frac {\gamma^ {i} + \delta}{1 - \gamma} \Big) \\ = \frac {\gamma^ {i + 1}}{1 - \gamma} + \frac {\delta \gamma (1 + \gamma^ {i - 1} - \gamma^ {i})}{1 - \gamma} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {\leq} \frac {\gamma^ {i + 1}}{1 - \gamma} + \frac {\delta}{1 - \gamma} \\ = Q ^ {0} \left(s _ {i}, a _ {2}\right), \\ \end{array}
$$
where we used  $\alpha \leq \delta (1 - \gamma)$  in (a) and  $\gamma (1 + \gamma^{i - 1} - \gamma^i) < 1$  for  $\gamma \in [0,1)$  in (b). This concludes the base case.
Inductive Step: Now assume that the claim is true for  $k$  and we want to show that  $\pi^{k+1}(a_1|s_i) \leq \alpha$  for  $i > k + 1$ . We do this in the same way as the base case by showing that  $Q^k(s_i, a_1) \leq Q^k(s_i, a_2)$  for  $i > k + 1$  so that the probability of  $\pi^{k+1}(a_1|s_i)$  cannot increase w.r.t  $\pi^k(a_1|s_i)$ , which is less than or equal to  $\alpha$  by the inductive hypothesis. We have:
$$
\begin{array}{l} Q ^ {k} \left(s _ {i}, a _ {1}\right) = \gamma V ^ {k} \left(s _ {i - 1}\right) \\ = \gamma \left(\pi^ {k} \left(a _ {1} \mid s _ {i - 1}\right) Q ^ {k} \left(s _ {i - 1}, a _ {1}\right) + \pi^ {k} \left(a _ {2} \mid s _ {i - 1}\right) Q ^ {k} \left(s _ {i - 1}, a _ {2}\right)\right) \\ \stackrel {(a)} {\leq} \gamma \left(\alpha Q ^ {k} \left(s _ {i - 1}, a _ {1}\right) + Q ^ {k} \left(s _ {i - 1}, a _ {2}\right)\right) \\ \leq \gamma \left(\alpha \frac {\gamma^ {i - 1}}{1 - \gamma} + \frac {r _ {i - 1}}{1 - \gamma}\right) \\ \stackrel {(b)} {\leq} \gamma \left(\delta (1 - \gamma) \frac {\gamma^ {i - 1}}{1 - \gamma} + \frac {\gamma^ {i} + \delta}{1 - \gamma}\right) \\ = \frac {\gamma^ {i + 1}}{1 - \gamma} + \frac {\delta \gamma (1 + \gamma^ {i - 1} - \gamma^ {i})}{1 - \gamma} \\ \stackrel {(c)} {\leq} \frac {\gamma^ {i + 1}}{1 - \gamma} + \frac {\delta}{1 - \gamma} \\ = Q ^ {k} \left(s _ {i}, a _ {2}\right), \\ \end{array}
$$
where we used in (a) that  $\pi^k(a_1|s_{i-1}) \leq \alpha$  for  $i > k + 1$ , which is true by the inductive hypothesis since  $i - 1 > k$ , in (b) that  $\alpha \leq \delta(1 - \gamma)$  and in (c) that  $\gamma(1 + \gamma^{i-1} - \gamma^i) < 1$  for  $\gamma \in [0,1)$ . This concludes the proof of the claim.
Now using the claim
$$
\begin{array}{l} V ^ {k} \left(s _ {k + 1}\right) = \pi^ {k} \left(a _ {1} \mid s _ {k + 1}\right) Q ^ {k} \left(s _ {k + 1}, a _ {1}\right) + \pi^ {k} \left(a _ {2} \mid s _ {k + 1}\right) Q ^ {k} \left(s _ {k + 1}, a _ {2}\right) \\ \leq \alpha \frac {\gamma^ {k + 1}}{1 - \gamma} + \frac {r _ {k + 1}}{1 - \gamma} \\ = \alpha \frac {\gamma^ {k + 1}}{1 - \gamma} + \frac {\gamma^ {k + 2} + \delta}{1 - \gamma}, \\ \end{array}
$$
$$
\begin{array}{l} V ^ {\star} \left(s _ {k + 1}\right) - V ^ {k} \left(s _ {k + 1}\right) \geq \frac {\gamma^ {k + 1}}{1 - \gamma} - \alpha \frac {\gamma^ {k + 1}}{1 - \gamma} - \frac {\gamma^ {k + 2} + \delta}{1 - \gamma} \\ = \frac {\gamma^ {k + 1} (1 - \gamma)}{1 - \gamma} - \frac {\alpha \gamma^ {k + 1} + \delta}{1 - \gamma} \\ \geq \gamma^ {k + 1} - \frac {\alpha + \delta}{1 - \gamma} \\ \geq \gamma^ {k + 1} - \frac {2 \delta}{1 - \gamma}, \tag {17} \\ \end{array}
$$
where we used that  $\alpha \leq \delta$ . Now note that
$$
\begin{array}{l} V ^ {0} (s _ {1}) = \alpha Q ^ {0} (s _ {1}, a _ {1}) + (1 - \alpha) Q ^ {0} (s _ {1}, a _ {2}) \\ = \alpha \frac {\gamma}{1 - \gamma} + (1 - \alpha) \frac {\gamma^ {2} + \delta}{1 - \gamma}, \\ \end{array}
$$