SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This results in a new MDP  $\mathcal{M}'$  with an augmented action space  $\mathcal{A}'$ , that is twice the size of the action space of the original MDP  $\mathcal{M}$ . In terms of action-value of an arbitrary policy  $\pi$ , this results in
$$
Q _ {\mathcal {M} ^ {\prime}} ^ {\pi} (s, a) - Q _ {\mathcal {M} ^ {\prime}} ^ {\pi} (s, a ^ {\prime}) = \delta ,
$$
where the notation  $Q_{\mathcal{M}'}^{\pi}$  refers to action-values in the MDP  $\mathcal{M}'$ . In terms of sub-optimality gaps, this gives
$$
\Delta^ {\pi} (s) \leq \delta .
$$
Choosing  $\delta$  small enough, we can make the step-size of [22] arbitrarily large, at least in early iterations. The step-size condition (5) of Theorem 4.1 will be less affected by this issue as it does not depend directly on  $\Delta^k(s)$ , and not at all in the first iteration. Beyond its generality to PMD, this illustrates the benefit of our result restricted to NPG over the result of [22].