SlowGuess/NeurIPS_2023_OCR huggingface dataset:
J. Huang. Complexity Estimates and Reductions to Discounting for Total and Average-Reward Markov Decision Processes and Stochastic Games. PhD thesis, State University of New York at Stony Brook, 2016.
R. Iwaki and M. Asada. Implicit incremental natural actor critic algorithm. Neural Networks, 109: 103-112, 2019.
G. N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
Y. Jin and A. Sidford. Efficiently solving MDPs with stochastic mirror descent. In International Conference on Machine Learning, pages 4890-4900. PMLR, 2020.
Y. Jin and A. Sidford. Towards tight bounds on the sample complexity of average-reward MDPs. In International Conference on Machine Learning, pages 5055-5064. PMLR, 2021.
L. Kallenberg. Classification problems in mdps. Markov processes and controlled Markov chains, pages 151-165, 2002.
A. Maitra. Dynamic programming for countable state systems. Sankhya: The Indian Journal of Statistics, Series A, pages 241-248, 1965.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
P. J. Neumann, G. D. Sanders, L. B. Russell, J. E. Siegel, and T. G. Ganiats. Cost-effectiveness in health and medicine. Oxford University Press, 2016.
M. O'Sullivan and A. F. Veinott Jr. Polynomial-time computation of strong and n-present-value optimal policies in Markov decision chains. Mathematics of Operations Research, 42(3):577-598, 2017.
F. S. Perotto and L. Vercouter. Tuning the discount factor in order to reach average optimality on deterministic MDPs. In International Conference on Innovative Techniques and Applications of Artificial Intelligence, pages 92-105. Springer, 2018.
M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley and Sons, 2014.
S. M. Ross. Non-discounted denumerable Markovian decision models. The Annals of Mathematical Statistics, 39(2):412-423, 1968.
S. M. Rump. Polynomial minimum root separation. Mathematics of Computation, 33(145):327-336, 1979.
B. Scherrer. Improved and generalized upper bounds on the complexity of policy iteration. Advances in Neural Information Processing Systems, 26, 2013.
M. Schneckenreither. Average reward adjusted discounted reinforcement learning: Near-Blackwell-optimal policies for real-world applications. arXiv preprint arXiv:2004.00857, 2020.
L. N. Steimle, D. L. Kaufman, and B. T. Denton. Multi-model markov decision processes. IISE Transactions, 53(10):1124-1139, 2021.
Y. Tang, M. Rowland, R. Munos, and M. Valko. Taylor expansion of discount factors. In International Conference on Machine Learning, pages 10130-10140. PMLR, 2021.
A. Tewari and P. L. Bartlett. Bounded parameter Markov decision processes with average reward criterion. In International Conference on Computational Learning Theory, pages 263-277. Springer, 2007.
J. N. Tsitsiklis. Np-Hardness of checking the unichain condition in average cost MDPs. Operations research letters, 35(3):319-323, 2007.
J. Wang, M. Wang, and L. F. Yang. Near sample-optimal reduction-based policy learning for average reward MDP. arXiv preprint arXiv:2212.00603, 2022.
M. Wang. Primal-dual  $\backslash \pi$  learning: Sample complexity and sublinear run time for ergodic Markov decision problems. arXiv preprint arXiv:1710.06100, 2017.
Y. Wang, A. Velasquez, G. Atia, A. Prater-Bennette, and S. Zou. Robust average-reward Markov decision processes. arXiv preprint arXiv:2301.00858, 2023.
W. Wiesemann, D. Kuhn, and B. Rustem. Robust Markov decision processes. Mathematics of Operations Research, 38(1):153-183, 2013.
S. Yang, Y. Gao, B. An, H. Wang, and X. Chen. Efficient average reward reinforcement learning using constant shifting values. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Y. Ye. The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate. Mathematics of Operations Research, 36(4):593-603, 2011.
J. Zhang, B. O'Donoghue, and S. Boyd. Globally convergent type-I Anderson acceleration for nonsmooth fixed-point iterations. SIAM Journal on Optimization, 30(4):3170-3197, 2020.
U. Zwick and M. Paterson. The complexity of mean payoff games on graphs. Theoretical Computer Science, 158(1-2):343-359, 1996.
A Unichain instance for Proposition 3.5
We can extend Example 3.5 to a unichain MDP as follows: we add a transition from state 7 to state 0, with a reward of 0. We also add three intermediate states from 0 to 7 for action  $a_1$ , so that it takes as many periods to reach state 7 from state 0 for the three actions  $a_1, a_2, a_3$ . Note that this new MDP is unichain. We represent it in Figure 3a. Additionally, for this new MDP instance, we have  $v_{\gamma}^{a_1} = 1 / (1 - \gamma^5)$ ,  $v_{\gamma}^{a_2} = (r_1\gamma + r_2\gamma^2) / (1 - \gamma^5)$ ,  $v_{\gamma}^{a_3} = (r_4\gamma + r_5\gamma^2) / (1 - \gamma^5)$ , which are the same expressions as in Example 3.5, up to the common denominator  $(1 - \gamma^5)^{-1}$ . Therefore, we have proved that the same conclusion as Proposition 3.4 holds for unichain MDPs.
B Proof of Theorem 3.7
Proof. Consider the following MDP instance, represented in Figure 2a. The initial state is state 0, where there are two actions to be chosen,  $a_1$  or  $a_2$ . Action  $a_1$  yields an instantaneous reward of 1 and then the decision maker transitions to the absorbing state  $N$ , where there is a reward of 0. Otherwise, choosing action  $a_2$  yields an instantaneous reward  $r_0$  and takes the decision maker through a deterministic sequence of states 1, ...,  $N - 1$  with rewards  $r_1, \ldots, r_{N-1}$ , before transitioning to state  $N$ . For a given  $\gamma \in [0,1)$ , the closed-form expressions for the value functions  $v_\gamma^{a_1}, v_\gamma^{a_2}$  are  $v_\gamma^{a_1} = 1$  and  $v_\gamma^{a_2} = \sum_{t=0}^{N-1} r_t \gamma^t$ .
Note that  $\gamma \mapsto v_{\gamma}^{a_2}$  is a polynomial of degree  $N - 1$ . Using Lagrange interpolation polynomials (section 0.9.11, (Horn and Johnson, 2012)), we can find coefficients  $r_0,\dots,r_{N - 1}$  such that  $\gamma \mapsto v_{\gamma}^{a_1}$  is equal to 1 for all  $N - 1$  discount factors  $\gamma_{1},\ldots ,\gamma_{N - 1}$  and equal to 0.9 at  $\gamma_0 = 0$ . The value function  $v_{\gamma}^{a_2}$  resulting from this construction is highlighted in Figure 2b for  $N = 5$  and  $(\gamma_0,\gamma_1,\gamma_2,\gamma_3,\gamma_4,\gamma_5) = (0,0.2,0.4,0.6,0.8,1.0)$ . Let us note  $q\colon \gamma \mapsto v_{\gamma}^{a_1} - v_{\gamma}^{a_2}$ . Our choice of the rewards ensures that  $q$  is a polynomial of degree  $N - 1$ , with  $q(0) > 0$ , and  $q(\gamma) = 0$  for  $\gamma \in \{\gamma_1,\dots,\gamma_{N - 1}\}$ . Because  $\gamma \mapsto q(\gamma) - 1$  is a polynomial of degree  $N - 1$  with  $N - 1$  different real roots, it changes signs at every root. This shows that  $\gamma \mapsto v_{\gamma}^{a_1} - v_{\gamma}^{a_2}$  is positive on  $(\gamma_0,\gamma_1)$ , negative on  $(\gamma_1,\gamma_2)$ , then positive on  $(\gamma_2,\gamma_3)$ , etc.. Action  $a_1$  is optimal on  $(\gamma_{N - 1},\gamma_N) = (\gamma_{N - 1},1)$  because  $N$  is odd. This concludes the proof of Theorem 3.7.
Figure 2: MDP instance for our proof of Theorem 3.7 (Figure 2a) and the value functions for  $N = 5$  (Figure 2b).
C Weakly-communicating instances for Proposition 4.3
Consider the MDP instance from the proof of Proposition 4.3. We now add a deterministic transition from state  $s_2$  to state  $s_1$ , with a reward of 0 for action  $a_1$  and a reward of  $\epsilon$  for action  $a_2$ . The new MDP instance is represented in Figure 3b.
(a) Unichain instance for Proposition 3.5
(b) Weakly-communicating instance for Proposition 4.3
Figure 3: MDP instances to generalize Proposition 4.3 and Proposition 3.5.
First, this MDP instance is weakly-communicating since  $\{s_1,s_2\}$  is strongly connected under policy  $a_2$ . In this new MDP instance, we still have  $v_{\gamma}^{a_1} = 0$  but  $v_{\gamma}^{a_2} = (-1 + \epsilon \gamma) / (1 - \gamma)$ . Hence  $a_2$  is Blackwell optimal when  $\gamma \geq 1 / \epsilon$ . By choosing  $\epsilon$  larger than 1 and  $\epsilon \rightarrow 1$ , we obtain  $\gamma_{\mathrm{bw}}\rightarrow 1$ . This shows that we can extend Proposition 4.3 to weakly-communicating MDPs.
D Proof of Theorem 4.4
In this appendix, we provide the proof for Theorem 4.4. As noted in Section 4.1, to bound  $\gamma_{\mathrm{bw}}$ , it is enough to obtain an upper bound on  $\gamma (\pi ,\pi^{\prime},s)$  for any  $\pi ,\pi^{\prime}\in \Pi$  and  $s\in S$  such that  $\gamma \mapsto v_{\gamma ,s}^{\pi} - v_{\gamma ,s}^{\pi '}$  is not identically equal to 0, since  $\gamma_{\mathrm{bw}}\leq \max_{\pi ,\pi '\in \Pi ,s\in S}\gamma (\pi ,\pi ',s)$ . Since  $m$  is the maximum bit-size of the input data, we can write, for any  $(s,a,s')\in S\times A\times S$ ,  $P_{sa'} = n_{sa'} / m$ , for  $n_{sa'}\in \mathbb{N}$ ,  $n_{sa'}\leq m$ , and  $r_{sa} = q_{sa} / m$ ,  $|q_{sa}|\leq r_{\infty}$ . Examples of MDPs with finite bit-sizes include any real instances used for applications where the transition probabilities are estimated as empirical frequencies from some data, e.g. examining patients' transfers in hospitals as in (Hu et al., 2018) and (Grand-Clément et al., 2022), MDPs for hypertension treatment (Garcia et al., 2021), diabetes management (Stimle et al., 2021) and cancer detection (Goh et al., 2018), as well as the machine maintenance studied in (Wiesemann et al., 2013) and (Delage and Mannor, 2010). We now proceed to proving Theorem 4.4.
Step 1. We start by studying in more detail the properties of the value functions. The following lemma follows directly from Cramer's rule, as explained in Section 3.
Lemma D.1. We have
$$
v _ {\gamma , s} ^ {\pi} = \frac {\det  (M (\gamma , s , \pi))}{\det  (I - \gamma P _ {\pi})}, \tag {D.1}
$$
with  $M(\gamma, s, \pi)$  the matrix formed by replacing the  $s$ -th column of  $\mathbf{I} - \gamma \mathbf{P}_{\pi}$  by the vector  $\mathbf{r}_{\pi}$ .
From Lemma D.1, we have
$$
v _ {\gamma , s} ^ {\pi} = \frac {n (\gamma , s , \pi)}{d (\gamma , \pi)}
$$
for  $n(\gamma, s, \pi) = \det(M(\gamma, s, \pi))$  and  $d(\gamma, \pi) = \det(I - \gamma P_{\pi})$ . We choose the letter  $n$  for nominator and the letter  $d$  for denominator.
Note that  $\gamma \mapsto n(\gamma, s, \pi)$  is a polynomial of degree at most  $|\mathcal{S}| - 1$ , while  $\gamma \mapsto d(\gamma, \pi)$  is a polynomial of degree at most  $|\mathcal{S}|$ .
We have, by definition,
$$
\begin{array}{l} v _ {\gamma , s} ^ {\pi} - v _ {\gamma , s} ^ {\pi^ {\prime}} = \frac {n (\gamma , s , \pi)}{d (\gamma , \pi)} - \frac {n (\gamma , s , \pi^ {\prime})}{d (\gamma , \pi^ {\prime})} \\ = \frac {n (\gamma , s , \pi) d \left(\gamma , \pi^ {\prime}\right) - n (\gamma , s , \pi) d (\gamma , \pi)}{d (\gamma , \pi) d \left(\gamma , \pi^ {\prime}\right)} \\ \end{array}
$$
Therefore,  $v_{\gamma ,s}^{\pi} - v_{\gamma ,s}^{\pi^{\prime}} = 0$  for  $\gamma \in [0,1)$  implies that  $\gamma$  is a root of the following polynomial equation in  $\gamma$ :
$$
p (\gamma) = 0, \tag {D.2}
$$
for  $p$  the polynomial defined as
$$
p (\gamma) = n (\gamma , s, \pi) d \left(\gamma , \pi^ {\prime}\right) - n (\gamma , s, \pi^ {\prime}) d (\gamma , \pi). \tag {D.3}
$$
Step 2. We now study the properties of the polynomial  $p$ . Note that it is straightforward that  $p$  is a polynomial of degree  $N = 2|\mathcal{S}| - 1$ . We first study the properties of the polynomial  $\gamma \mapsto d(\pi, \gamma)$ . We have the following lemma.
Lemma D.2. We have
$$
d (\gamma , \pi) > 0, \forall \gamma \in [ 0, 1), \forall \pi \in \Pi ,
$$
and  $d(1,\pi) = 0, \forall \pi \in \Pi$ .
Proof of Lemma D.2. This lemma follows from the relation between the determinant of a matrix and its eigenvalues, through the characteristic polynomial:
$$
d (\gamma , \pi) = \det \left(\boldsymbol {I} - \gamma \boldsymbol {P} _ {\pi}\right) = \prod_ {\lambda \in S p (\boldsymbol {P} _ {\pi})} (1 - \gamma \lambda) ^ {\alpha_ {\lambda}},
$$
with  $\alpha_{\lambda}$  the algebraic multiplicity of the (potentially complex) eigenvalue  $\lambda$  in the spectrum  $Sp(\mathbf{P}_{\pi})$  of  $\mathbf{P}_{\pi}$ . Since  $\mathbf{P}_{\pi}$  is the transition matrix of a Markov chain, we know that the modulus of any eigenvalue  $\lambda$  of  $\mathbf{P}_{\pi}$  is smaller or equal to 1. This shows that  $d(\gamma, \pi) > 0, \forall \gamma \in [0,1), \forall \pi \in \Pi$ . To show  $d(1, \pi) = 0$ , we simply note that  $1 \in Sp(\mathbf{P}_{\pi})$  since  $\mathbf{P}_{\pi}$  is the transition matrix of a Markov chain.
From Lemma D.2 and the definition of  $p$  as in (D.3), it is straightforward that  $p(1) = 0$ .
Lemma D.3.  $\gamma = 1$  is a root of  $p$ .
We now bound the sum of the absolute values of the coefficients of  $p$ . We have the following theorem.
Theorem D.4. The polynomial  $m^{2|S|} \cdot p$  has integral coefficients, potentially negative. The sum of the absolute values of the coefficients of  $m^{2|S|}p$  is bounded by
$$
L = 2 \cdot | \mathcal {S} | \cdot r _ {\infty} \cdot m ^ {2 | \mathcal {S} |} \cdot 4 ^ {| \mathcal {S} |}.
$$
Theorem D.4 is based on the following three propositions. We note  $C_{\ell}^{k}$  the binomial coefficient defined as  $C_{\ell}^{k} = \ell! / k!(\ell - k)!$ .
Proposition D.5. For any  $\pi \in \Pi$ , the function  $\gamma \mapsto d(\pi, \gamma)$  is a polynomial of degree  $|S|$ . Moreover,  $\gamma \mapsto m^{|S|} \cdot d(\pi, \gamma)$  is a polynomial with integral coefficients (potentially negative), and the absolute value of its coefficient of degree  $k$  is bounded by  $m^{|S|} C_{|S|}^{k}$ .
Therefore, the sum of the absolute values of the coefficients of  $\gamma \mapsto m^{|S|} \cdot d(\pi, \gamma)$  is upper bounded by
$$
L _ {d} = m ^ {| \mathcal {S} |} \cdot 2 ^ {| \mathcal {S} |}.
$$
Proposition D.6. For any policy  $\pi \in \Pi$  and any state  $s \in S$ , the function  $\gamma \mapsto n(\gamma, s, \pi)$  is a polynomial of degree  $|\mathcal{S}| - 1$ . Moreover,  $\gamma \mapsto m^{|S|} \cdot n(\gamma, s, \pi)$  is a polynomial with integral coefficients (potentially negative), and the absolute value of its coefficient of degree  $k$  is bounded by  $m^{|S|} \cdot |\mathcal{S}| \cdot r_{\infty} \cdot C_{|\mathcal{S}| - 1}^{k} \cdot 2$ .
Therefore, the sum of the absolute values of the coefficients of  $\gamma \mapsto m^{|S|} \cdot n(\gamma, s, \pi)$  is upper bounded by
$$
L _ {n} = m ^ {| \mathcal {S} | - 1} \cdot | \mathcal {S} | \cdot r _ {\infty} \cdot 2 ^ {| \mathcal {S} |}.
$$
Proposition D.7. Let  $P = \sum_{i=0}^{n} a_i X^i$ ,  $Q = \sum_{j=0}^{m} b_j X^j$ . Then  $PQ = \sum_{k=0}^{n+m} c_k X^k$ ,  $c_k = \sum_{i,j;i+j=k} a_i b_j$ . Additionally, suppose that  $\sum_{i=0}^{n} |a_i| \leq L_P$ ,  $\sum_{j=0}^{m} |b_j| \leq L_Q$ . Then
$$
\sum_ {k = 0} ^ {n + m} \left| c _ {k} \right| \leq L _ {P} L _ {Q}.
$$
Combining Proposition D.5, Proposition D.6 and Proposition D.7 with the definition of the polynomial  $p$  in (D.3) yields Theorem D.4.
To conclude Step 2 of our proof, let us prove Proposition D.5 and Proposition D.6. Proposition D.7 simply follows from the multiplication rule for polynomials.
Proof of Proposition D.5. By definition,
$$
d (\gamma , \pi) = \det  \left(\boldsymbol {I} - \gamma \boldsymbol {P} _ {\pi}\right) = \sum_ {k = 0} ^ {| S |} a _ {k} \left(\gamma \boldsymbol {P} _ {\pi}\right),
$$
where  $M \mapsto a_k(M)$  is the  $(|S| - k)$ -th coefficient of the characteristic polynomial of a matrix  $M$ . By definition,  $a_k(M)$  is the sum of all the principal minors of size  $k$  of  $M$  (section 0.7.1, (Horn and Johnson, 2012)). This first shows that  $a_k(\gamma P_\pi) = \gamma^k a_k(P_\pi)$ , and therefore, that
$$
d (\gamma , \pi) = \sum_ {k = 0} ^ {| S |} \gamma^ {k} a _ {k} \left(\boldsymbol {P} _ {\pi}\right).
$$
$$
a _ {k} \left(\boldsymbol {P} _ {\pi}\right) \leq C _ {| \mathcal {S} |} ^ {k}, \forall k = 1, \dots , | \mathcal {S} |.
$$
Let  $g$  be a principal minor of  $P_{\pi}$  of size  $k$ . By definition,  $g$  is the determinant of a submatrix  $M$  of size  $k$  of  $P_{\pi}$ , obtained by deleting rows and columns with the same indices:  $g = \operatorname{det}(M)$ . For any matrix square  $M$ , we always have  $\operatorname{det}(M) = \operatorname{det}(M^{\top})$ . Now Hadamard's inequality shows that  $\operatorname{det}(M^{\top}) \leq \prod_{i=1}^{k} \| \operatorname{Col}_{i}(M^{\top}) \|_{2}$ , with  $\operatorname{Col}_{i}(M^{\top})$  the  $i$ -th column of  $M^{\top}$ , and therefore we have  $\operatorname{det}(M^{\top}) \leq \prod_{i=1}^{k} \| \operatorname{Col}_{i}(M^{\top}) \|_{1}$ . Note that the columns of  $M^{\top}$  have  $\ell_{1}$ -norm smaller than 1, since  $P_{\pi}$  is a stochastic matrix, and  $M$  is a submatrix of  $P_{\pi}$ . Therefore,  $g \leq 1$ . Because there are  $C_n^k$  possible principal minors of size  $k$  of  $P_{\pi}$ , we have  $a_k(P_{\pi}) \leq C_n^k, \forall k = 1, \dots, n$ .
Of course, we may have  $a_{k}(P_{\pi}) \notin \mathbb{Z}$ . However, for any principal minor  $g = \operatorname*{det}(M)$  of  $P_{\pi}$ , we have, by definition the determinant,
$$
\det  (\boldsymbol {M}) = \sum_ {\sigma \in \mathfrak {S} _ {k}} \varepsilon (\sigma) \prod_ {i = 1} ^ {k} M _ {\sigma (i) i}
$$
where  $\varepsilon(\sigma)$  is the signature of the permutation  $\sigma$  and  $\mathfrak{S}_k$  is the symmetric group, i.e., the group of all permutations of  $\{1, \dots, k\}$ . This shows, by definition  $m$  as the maximum bit-size of the input data, that  $m^{|S|} \det(M) \in \mathbb{Z}$ , and therefore that  $m^{|S|} a_k(P_\pi) \in \mathbb{Z}$  and that  $m^{|S|} a_k(P_\pi) \leq m^{|S|} C_{|S|}^k$ .
Proof of Proposition D.6. Using Laplace cofactor expansions (section 0.3.1, (Horn and Johnson, 2012)), we have that  $n(\gamma, s, \pi)$  is equal to
$$
\sum_ {s ^ {\prime} \in \mathcal {S}} (- 1) ^ {s + s ^ {\prime}} \cdot r _ {s ^ {\prime}, \pi \left(s ^ {\prime}\right)} \cdot \det  \left(\left(\boldsymbol {I} - \gamma \boldsymbol {P} _ {\pi}\right) _ {\mathcal {S} \backslash \{s ^ {\prime} \} \times \mathcal {S} \backslash \{s \}}\right), \tag {D.4}
$$
where  $(\boldsymbol{I} - \gamma \boldsymbol{P}_{\pi})_{\mathcal{S} \setminus \{s'\} \times \mathcal{S} \setminus \{s\}}$  is the matrix obtained from  $\boldsymbol{I} - \gamma \boldsymbol{P}_{\pi}$  by removing the  $s$ -th column and the  $s'$ -th row.
Note that  $\gamma \mapsto \operatorname*{det}\left((\boldsymbol{I} - \gamma \boldsymbol{P}_{\pi})_{\mathcal{S} \setminus \{s' \} \times \mathcal{S} \setminus \{s\}}\right)$  is a polynomial of degree  $|\mathcal{S}| - 1$  in  $\gamma$ . Similarly as for the proof of Proposition D.5,  $\gamma \mapsto m^{|S|} n(\gamma, s, \pi)$  is a polynomial of degree  $|\mathcal{S}| - 1$  with integral coefficients.
Let us consider  $I_{\backslash \{s', s\}}$  the matrix of dimension  $(|S| - 1) \times (|S| - 1)$ , obtained by removing the  $s$ -th column and the  $s'$ -th row from the identity matrix of dimension  $|S|$ , and let us call  $E_{s'}$  the matrix of dimension  $(|S| - 1) \times (|S| - 1)$ , where all rows are  $\mathbf{0}^\top$ , except the  $s$ -th row, equal to  $e_{s'}^\top$ .
Then  $\operatorname{det}\left((\boldsymbol{I} - \gamma \boldsymbol{P}_{\pi})_{\mathcal{S} \setminus \{s'\} \times \mathcal{S} \setminus \{s\}}\right)$  is equal to
$$
\det  \left(\left(I - \gamma P _ {\pi}\right) _ {S \backslash \{s ^ {\prime} \} \times S \backslash \{s \}} + E _ {s ^ {\prime}} - E _ {s ^ {\prime}}\right)
$$
and therefore is equal to
$$
\det  \left(I _ {\backslash \{s ^ {\prime}, s \}} + E _ {s ^ {\prime}} - (\gamma P _ {\pi}) _ {\mathcal {S} \backslash \{s ^ {\prime} \} \times \mathcal {S} \backslash \{s \}} - E _ {s ^ {\prime}}\right).
$$
We notice that  $I_{\backslash \{s', s\}} + E_{s'}$  is a matrix whose rows are exactly the rows of the identity matrix of  $\mathbb{R}^{|S|-1}$ , up to a certain permutation  $\sigma \in \mathfrak{S}_{|S|-1}$ . Let  $P^\sigma \in \mathbb{R}^{(|S|-1) \times (|S|-1)}$  the permutation matrix defined as  $P_{ij} = 1$  if  $\sigma(j) = i$  and 0 otherwise. Then for any matrix  $M$ , we have  $\operatorname{det}(P^\sigma M) = \operatorname{det}(P^\sigma) \operatorname{det}(M) = \varepsilon(\sigma) \operatorname{det}(M)$ , with  $\varepsilon(\sigma)$  the signature of the permutation  $\sigma$ . Since we always have  $\varepsilon(\sigma) \in \{-1, 1\}$ , this shows that  $\operatorname{det}\left((I - \gamma P_\pi)_{\mathcal{S} \setminus \{s'\} \times \mathcal{S} \setminus \{s\}}\right)$  is equal to
$$
\varepsilon (\sigma) \det  \left(\boldsymbol {I} - \left(\left(\gamma \boldsymbol {P} _ {\pi}\right) _ {\mathcal {S} \backslash \{s ^ {\prime} \} \times \mathcal {S} \backslash \{s \}} + \boldsymbol {E} _ {s ^ {\prime}}\right)\right).
$$
The map  $\gamma \mapsto \operatorname*{det}\left(\mathbf{I} - \left((\gamma \mathbf{P}_{\pi})_{\mathcal{S}\setminus \{s^{\prime}\} \times \mathcal{S}\setminus \{s\}} + \mathbf{E}_{s^{\prime}}\right)\right)$  is equal to
$$
\sum_ {k = 0} ^ {| \mathcal {S} | - 1} a _ {k} \left(\left(\gamma \boldsymbol {P} _ {\pi}\right) _ {\mathcal {S} \backslash \{s ^ {\prime} \} \times \mathcal {S} \backslash \{s \}} - \boldsymbol {E} _ {s ^ {\prime}}\right)
$$
where similarly as for the proof of Proposition D.5,  $a_{k}(M)$  is the  $k$ -th coefficient of the characteristic polynomial of a matrix  $M$ , i.e.,  $a_{k}(M)$  is equal to the sum of all the principal minors of  $M$  of dimension  $k \times k$ . Let
$$
M = \left(\gamma P _ {\pi}\right) _ {\mathcal {S} \backslash \{s ^ {\prime} \} \times \mathcal {S} \backslash \{s \}} - E _ {s ^ {\prime}}.
$$
Note that  $(P_{\pi})_{\mathcal{S} \setminus \{s'\} \times \mathcal{S} \setminus \{s\}}$  is a substochastic matrix, i.e., it has non-negative entries and the sum of the entries of each row is smaller or equal to 1. Note that  $M$  differs from  $(\gamma P_{\pi})_{\mathcal{S} \setminus \{s'\} \times \mathcal{S} \setminus \{s\}}$  only at the coefficient of index  $(s, s')$ . Using Hadamard's inequality, we find that
$$
a _ {k} (\boldsymbol {M}) \leq 2 \cdot C _ {| S | - 1} ^ {k}, m ^ {| S |} a _ {k} (\boldsymbol {M}) \in \mathbb {N}. \tag {D.5}
$$
We conclude by combining Equation (D.5) with Equation (D.4).
Step 3. We now lower bound the distance between any two roots of  $p$  by a scalar  $\eta > 0$ . Since we know that for  $\gamma(\pi, \pi', s) \in [0,1)$  and 1 are two roots of  $P$ , this will show that  $\gamma(\pi, \pi', s) < 1 - \eta$ . Our proof is based on the following theorem.
Theorem D.8 ((Rump, 1979)). Let  $p$  be a polynomial of degree  $N$  with integer coefficients, possibly with multiple roots. Let  $L$  be the sum of the absolute values of its coefficients. Then the distance between any two distinct roots of  $p$  is strictly larger
$$
\frac {1}{2 N ^ {N / 2 + 2} (L + 1) ^ {N}}.
$$
Recall that both  $\gamma (\pi ,\pi^{\prime},s)\in [0,1)$  and 1 are roots of the polynomial  $p$  . Therefore, we can combine Theorem D.8 with Theorem D.4 to obtain  $\gamma (\pi ,\pi ',s) <   1 - \eta (\mathcal{M})$  , with
$$
\eta (\mathcal {M}) = \frac {1}{2 N ^ {N / 2 + 2} (L + 1) ^ {N}}
$$
$$
N = 2 | \mathcal {S} | - 1,
$$
This concludes the proof of Theorem 4.4.
Remark D.9. Note that (Akian et al., 2019) use Theorem D.8 to obtain a lower bound on the average rewards of any two different policies, in the setting of two-player stochastic games.
Remark D.10. Theorem 1 in (Rump, 1979) provides a separation bound in the case where the polynomial  $p$  has complex coefficients. Unfortunately, the separation bound from Theorem 1 in (Rump, 1979) is not directly usable here, because it depends on the discriminant  $D(p)$  of the polynomial  $p$ , a quantity that is hard to lower-bound (in all generality). We decide to use the bound from Theorem 3 in (Rump, 1979) because it does not depend on  $D(p)$  but directly on the  $\ell_1$ -norm of  $p$  and of the degree of  $p$ , which can be computed in closed-form and can be bounded as in Proposition D.6 and Proposition D.5.
E Proof of Theorem 4.7