SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Theorem 4.7. Following table 4 in (Ye, 2011), we know that interior-point methods for the linear programming formulation of MDPs return an optimal policy in  $O\left(|S|^3 |\mathcal{A}|^2 (Q(\boldsymbol {r},\boldsymbol {P},\gamma))\right)$  arithmetic operations, with  $Q(\boldsymbol {r},\boldsymbol {P},\gamma)$  equal to the total bit-size of the MDP instance, i.e., the sum of the bit-sizes of all instantaneous rewards, transition probabilities, and the discount factor. By choosing  $\gamma = 1 - \eta (\mathcal{M})$  and noticing that  $\log (\eta (\mathcal{M})) = O\left(|S|\log (r_{\infty}) + |S|^2\log (m)\right) = O\left(|S|^2\log (m)\right)$ , we see that interior-point methods for the linear programming formulation of MDPs return an optimal policy in  $O\left(|S|^5 |\mathcal{A}|^2 (Q(\boldsymbol {r},\boldsymbol {P}))\right)$ , where  $Q(\boldsymbol {r},\boldsymbol {P})$  is the total bit-size of MDP instance.
F Proof of Section 4.2
Proof of the existence of  $\gamma_{\mathrm{bw},r}$ . Let
$$
\bar {\gamma} _ {r} = \max  _ {\pi , \pi^ {\prime} \in \Pi , s \in \mathcal {S}} \max  _ {\boldsymbol {P}, \boldsymbol {P} ^ {\prime} \in \mathcal {U} _ {\text {e x t}}} \gamma (\pi , \pi^ {\prime}, s, \boldsymbol {P}, \boldsymbol {P} ^ {\prime}),
$$
where  $\gamma (\pi ,\pi^{\prime},s,P,P^{\prime})$  is the largest zero of the function  $\gamma \mapsto v_{\gamma ,s}^{\pi ,P} - v_{\gamma ,s}^{\pi^{\prime},P^{\prime}}$  if it is not identically equal to zero, or  $\gamma (\pi ,\pi^{\prime},s,P,P^{\prime}) = 0$  otherwise. Recall that  $\mathcal{U}_{\mathrm{ext}}$  is the (finite) set of extreme points of  $\mathcal{U}$ . We will show that  $\Pi_{\gamma ,\mathrm{r}}^{\star} = \Pi_{\mathrm{bw},\mathrm{r}}^{\star},\forall \gamma >\bar{\gamma}_{\mathrm{r}}$ . Let  $\pi$  be a robust discount-optimal policy for some  $\gamma >\bar{\gamma}_{\mathrm{r}}$ . We will prove that  $\pi$  is a Blackwell-optimal policy. Since  $\pi$  is robust  $\gamma$ -discount-optimal, we have
$$
v _ {\gamma , s} ^ {\pi , \mathcal {U}} \geq v _ {\gamma , s} ^ {\pi^ {\prime}, \mathcal {U}}, \forall \pi^ {\prime} \in \Pi , \forall s \in \mathcal {S}.
$$
By definition  $v_{\gamma, s}^{\pi, \mathcal{U}} = \min_{P \in \mathcal{U}} v_{\gamma, s}^{\pi, P}, \forall s \in S$ . From (Iyengar, 2005), we know that the arg min in  $\min_{P \in \mathcal{U}} v_{\gamma, s}^{\pi, P}$  is attained at an extreme point of  $\mathcal{U}$ . Therefore, by definition of  $\bar{\gamma}_r$ , the function  $\gamma \mapsto v_{\gamma, s}^{\pi, \mathcal{U}} - v_{\gamma, s}^{\pi', \mathcal{U}}$  cannot be equal to 0 on  $(\bar{\gamma}_r, 1)$ , and therefore it does not change sign, since it is a continuous function. This shows that for all  $\gamma > \bar{\gamma}_r$ , we have
This shows the existence of the robust Blackwell discount factor  $\gamma_{\mathrm{bw,r}}$  and that  $\gamma_{\mathrm{bw,r}} < \bar{\gamma}_{\mathrm{r}}$ .
Proof of Theorem 4.11. We start by showing the following lemma.
Lemma F.1. Let  $\mathcal{M} = \left(S, \mathcal{A}, r, P^0\right)$  be an MDP instance with maximum bit-size  $m \in \mathbb{N}$ . Assume that  $\mathcal{U}$  is sa-rectangular, where for each  $(s, a) \in S \times \mathcal{A}$ ,  $\mathcal{U}_{sa}$  is constructed as in (4.1), with the scalars  $\left(\alpha_{sa}\right)_{s,a}$  of maximum bit-size  $m$ .
Then the maximum bit-size complexity to describe the transition probabilities associated with the extreme points of  $\mathcal{U}_{sa}$  is  $m'$  for  $p = \infty$  and  $2m'$  for  $p = 1$ .
Proof of Lemma F.1. In the proof of this lemma, we use the fact that the worst-case kernel  $\pmb{P}^{\star}$  of a policy  $\pi$  can be chosen as the arg min of the optimization problem  $\min_{\pmb{p} \in \mathcal{U}_{s\pi(s)}} \pmb{p}^\top \pmb{v}_\gamma^{\pi,\mathcal{U}}$ , where  $\pmb{v}_\gamma^{\pi,\mathcal{U}}$  is the worst-case value function of  $\pi$ . In particular, let  $\pmb{v} \in \mathbb{R}^S$ .
The case  $p = \infty$ . In this case, there exists a sorting solution to  $\min_{\boldsymbol{p} \in \mathcal{U}_{sa}} \boldsymbol{p}^{\top} \boldsymbol{v}$  for any  $(s, a) \in S \times \mathcal{A}$  and any  $\boldsymbol{v} \in \mathbb{R}^{S}$ , by sorting  $\boldsymbol{v}$ , see for instance proposition 3 in (Goh et al., 2018), equation (9) in (Givan et al., 1997), or appendix C in (Behzadian et al., 2021). In particular, let  $(s, a) \in S \times \mathcal{A}$  and define  $\sigma$  the permutation of  $S$  such that  $v_{\sigma(1)} \leq \ldots \leq v_{\sigma(|S|)}$ , and define  $i$  as the smaller integer in  $\{1, \ldots, |S|\}$  such that
$$
\sum_ {s ^ {\prime} = 1} ^ {i} \left(P _ {s a \sigma \left(s ^ {\prime}\right)} ^ {0} + \alpha_ {s a}\right) + \sum_ {s ^ {\prime} = i + 1} ^ {| S |} \left(P _ {s a \sigma \left(s ^ {\prime}\right)} ^ {0} - \alpha_ {s a}\right) \geq 1.
$$
Then a solution to  $\min_{\pmb{p} \in \mathcal{U}_{sa}} \pmb{p}^{\top} \pmb{v}$  is  $p_{\sigma(s')} = P_{sa\sigma(s')}^{0} + \alpha_{sa}$  if  $s' < i$ ,  $p_{\sigma(s')} = P_{sa\sigma(s')}^{0} - \alpha_{sa}$  if  $s' > i$ , and
$$
p _ {\sigma (i)} = 1 - \sum_ {s ^ {\prime} \in \mathcal {S} \backslash \{i \}} p _ {\sigma (s ^ {\prime})}.
$$
This closed-form shows that for any vector  $\pmb{v} \in \mathbb{R}^S$ , a solution of  $\min_{\pmb{p} \in \mathcal{U}_{sa}} \pmb{p}^\top \pmb{v}$  can be found as a vector with rational entries with a denominator of at most  $m$ .
The case  $p = 1$ . In this case, one can show that the optimization problem  $\min_{\pmb{p} \in \mathcal{U}_{sa}} \pmb{p}^{\top} \pmb{v}$  can be formulated as a linear program. Therefore, there exists an optimal basic feasible solution  $\pmb{p}$  which has the following form by lemma 5.4 and lemma 5.5 in (Ho et al., 2021). There exist  $j_1, j_2 \in S$  such that  $j_1 \neq j_2$  and for each  $i \in \mathcal{I} = S \setminus \{j_1, j_2\}$ :
$$
p _ {i} = 0 \quad \text {o r} \quad p _ {i} = P _ {s a i} ^ {0}
$$
$$
p _ {j _ {1}} \geq P _ {s a j _ {1}} ^ {0} \quad \text {a n d} \quad p _ {j _ {2}} \leq P _ {s a j _ {2}} ^ {0}.
$$
Then, in order for  $\pmb{p} \in \mathcal{U}_{sa}$  we need the following equalities to hold
$$
p _ {j _ {1}} + p _ {j _ {2}} = 1 - \sum_ {i \in \mathcal {I}} p _ {i}
$$
$$
\left(p _ {j _ {1}} - P _ {s a j _ {1}} ^ {0}\right) + \left(P _ {s a j _ {2}} ^ {0} - p _ {j _ {2}}\right) = \alpha_ {s a} - \sum_ {i \in \mathcal {I}} \left| p _ {i} - P _ {s a i} ^ {0} \right|.
$$
Combining the equalities above yields that
$$
\begin{array}{l} 2 p _ {j _ {1}} = \alpha_ {s a} - \sum_ {i \in \mathcal {I}} | p _ {i} - P _ {s a i} ^ {0} | + P _ {s a j _ {1}} ^ {0} - P _ {s a j _ {2}} ^ {0} \\ + 1 - \sum_ {i \in \mathcal {I}} p _ {i}. \\ \end{array}
$$
Because the right-hand side of the equation above is a sum of rational numbers with a denominator of at most  $m$ ,  $p_{j_1}$  is also rational with a denominator at most  $2m$ . Using an analogous argument for  $p_{j_2}$ , we get that there exists an optimal solution that is rational with a denominator of at most  $2m$ .
Theorem 4.11 then follows by applying Theorem 4.4 with on the MDP instance  $(S, \mathcal{A}, r, P')$  with  $P'$  an extreme point of  $\mathcal{U}$ . Lemma F.1 exactly describes the maximum bit-size of any transition  $P_{\text{sa}s'}'$  for  $(s, a, s') \in S \times \mathcal{A} \times S$  in the case of sa-rectangular uncertainty set based on  $\ell_1$ -distance or  $\ell_{\infty}$ -distance as in (4.1). This concludes the proof of Theorem 4.11.