SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Take a solution  $(z,\xi)$  to  $\mathrm{LP}_{h,s,\iota}(M)$ . This can be done in polynomial time as  $\mathrm{LP}_{h,s,\iota}(M)$  is a linear program with polynomial many variables and constraints. Then apply to the solution  $(z,\xi)$  the transformation used in Lemma 8 to obtain a feasible solution to  $\mathcal{R}_{h,s,\iota}(M)$ . This gives an optimal solution to  $\mathcal{R}_{h,s,\iota}(M)$ . To prove the last result, assume that there would exist a feasible solution  $(\kappa', \tilde{q}')$  to  $\mathcal{R}_{h,s,\iota}(M)$  such that  $\tilde{F}_{h,s,M}(\kappa', \tilde{q}') > \tilde{F}_{h,s,M}(\kappa, \tilde{q})$ , and apply the transformation defined in Lemma 7. This would find a solution  $(z',\xi')$  strictly better than  $(z,\xi)$  which contradicts the optimality of  $(z,\xi)$  for  $\mathrm{LP}_{h,s,\iota}(M)$ .
Theorem 5. For every  $h \in \mathcal{H}$ ,  $s \in S$ , and  $\iota \in \mathcal{D}_{\delta}$ , if Algorithm 4 is used for all  $h' > h$  as approximate oracle  $\mathcal{O}_{h', s, \iota}$  in Algorithm 3, then it implements an approximate oracle as in Definition 3.
Proof. Fix any  $\bar{h} \in \mathcal{H}$ ,  $\bar{s} \in S$  and  $\bar{\iota} \in \mathcal{D}_{\delta}$ . We show that for  $h = \bar{h} + 1$ ,  $s \in S$  and for any distribution  $\gamma \in \Delta(\mathcal{D}_{\delta})$ , it holds that:
$$
M _ {h} ^ {\delta} \left(s, \left\lfloor \sum_ {\iota \in \mathcal {D} _ {\delta} \left(s, M _ {h} ^ {\delta}\right)} \iota \gamma (\iota) \right\rfloor_ {\delta}\right) \geq \sum_ {\iota \in \mathcal {D} _ {\delta} \left(s, M _ {h} ^ {\delta}\right)} \gamma (\iota) M _ {h} ^ {\delta} (s, \iota), \tag {21}
$$
where the table  $M_h^\delta$  is the one built by Algorithm 3 with oracle of Algorithm 4.
Define  $\iota_{\mathbb{E}} := \sum_{\iota \in \mathcal{D}_{\delta}(s, M_h^{\delta})} \iota \cdot \gamma(\iota)$ . Then for every  $\iota, \iota' \in \mathcal{D}_{\delta}$  and  $\iota' \leq \iota$  we have that
$$
\Omega_ {h, s, \iota^ {\prime}} (M _ {h} ^ {\delta}) \geq \Omega_ {h, s, \iota} (M _ {h} ^ {\delta}),
$$
as  $\iota$  it only appears in the RHS of the constraint of the problem  $\mathcal{R}_{h,s,\iota}(M_h^\delta)$ .
By construction of Algorithm 3 we have that:
$$
M _ {h} ^ {\delta} (s, \iota) = \Omega_ {h, s, \iota} (M _ {h + 1} ^ {\delta}),
$$
and that by Lemma 7 and Lemma 8,  $\Omega_{h,s,\iota}(M_{h + 1}^{\delta})$  is equal to the value of  $\mathrm{LP}_{h,s,\iota}(M)$  described by Equations (19), for each  $\iota$ . This means that the function  $\iota \mapsto \Omega_{h,s,\iota}(M_{h + 1}^{\delta})$  is concave [Bertsekas, 1998, Theorem 5.1].
Thus for every distribution  $\gamma \in \Delta (\mathcal{D}_{\delta})$  we have:
$$
\Omega_ {h, s, (\sum_ {\iota \in \mathcal {D} _ {\delta}} \iota \cdot \gamma (\iota))} (M _ {h + 1} ^ {\delta}) \geq \sum_ {\iota \in \mathcal {D} _ {\delta}} \gamma (\iota) \cdot \Omega_ {h, s, \iota} (M _ {h + 1} ^ {\delta}).
$$
Observe that we cannot apply directly the above concavity property as, for every  $s \in S$ , in Equation (21) we are only selecting the components of  $\gamma$  such that  $M_h^\delta > -\infty$ .
To solve this problem we can, for any distribution  $\gamma \in \Delta(\mathcal{D}_{\delta})$ , table  $M$  and  $s \in S$ . define the new distribution  $\tilde{\gamma}$  on  $\mathcal{D}_{\delta}(s,M)$  that puts all the mass on the  $-\infty$  components of  $M$  on  $0$ . Formally  $\tilde{\gamma}(\iota) = \gamma(\iota)$  for all  $\iota \in \mathcal{D}_{\delta(s,M)}$ , and  $\tilde{\gamma}(0) = \gamma(0) + \sum_{\iota \in \mathcal{D}_{\delta} \setminus \mathcal{D}_{\delta}(s,M)} \gamma(\iota)$ . Note that with this definition we have  $\iota_{\mathbb{E}} = \sum_{\iota \in \mathcal{D}_{\delta}(s,M)} \iota \cdot \tilde{\gamma}(\iota)$  and  $\sum_{\iota \in \mathcal{D}_{\delta}(s,M)} \gamma(\iota) \Omega_{h,s,\iota}(M) \leq \sum_{\iota \in \mathcal{D}_{\delta}(s,M)} \tilde{\gamma}(\iota) \Omega_{h,s,\iota}(M)$ . Combining these inequalities we can conclude that:
$$
\begin{array}{l} M _ {h} ^ {\delta} (s, \lfloor \sum_ {\iota \in \mathcal {D} _ {\delta} (s, M _ {h} ^ {\delta})} \iota \gamma (\iota) \rfloor_ {\delta}) = \Omega_ {h, s, \lfloor \iota_ {\mathbb {E}} \rfloor_ {\delta}} (M _ {h + 1} ^ {\delta}) \\ \geq \Omega_ {h, s, \iota_ {\mathbb {E}}} (M _ {h + 1} ^ {\delta}) \\ \geq \sum_ {\iota \in \mathcal {D} _ {\delta} (s, M _ {h} ^ {\delta})} \tilde {\gamma} (\iota) \cdot \Omega_ {h, s, \iota} \left(M _ {h + 1} ^ {\delta}\right) \\ \geq \sum_ {\iota \in \mathcal {D} _ {\delta} (s, M _ {h} ^ {\delta})} \gamma (\iota) \cdot \Omega_ {h, s, \iota} \left(M _ {h + 1} ^ {\delta}\right) \\ = \sum_ {\iota \in \mathcal {D} _ {\delta} (s, M _ {h} ^ {\delta})} \gamma (\iota) \cdot M _ {h} ^ {\delta} (s, \iota), \\ \end{array}
$$
where the last inequality follows since it is easy to see that  $\Omega_{h,s,0}(M_{h + 1}^{\delta})\geq 0$  and the last equality follows from Lemma 7. This proves Equation (21).
Now assume that  $(\kappa, \tilde{q})$  is a the optimal solution to  $\mathcal{R}_{\bar{h}, \bar{s}, \bar{\iota}}(M_{\bar{h} + 1}^{\delta})$ . Clearly if  $(\kappa, \tilde{q}_{\mathbb{E}}) \in \Psi_{\bar{\iota}}^{\bar{h}, \bar{s}}$  then  $(\kappa, q) \in \Psi_{\bar{\iota} - \delta}^{\bar{h}, \bar{s}}$ , where  $q = \lfloor \tilde{q}_{\mathbb{E}} \rfloor_{\delta}$ .
Using the inequality of Equation (21) we can readily perform the following inequalities:
$$
\begin{array}{l} F _ {\bar {h}, \bar {s}, M _ {\bar {h} + 1} ^ {\delta}} (\kappa , q) \\ = \sum_ {\theta \in \Theta} \sum_ {a \in \mathcal {A}} \mu_ {h} (\theta | \bar {s}) \kappa (a | \theta) \left(r _ {h} ^ {\mathbf {S}} (\bar {s}, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | \bar {s}, a, \theta\right) M _ {h + 1} ^ {\delta} \left(s ^ {\prime}, q (a, s ^ {\prime})\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} \geq \sum_ {\theta \in \Theta} \sum_ {a \in \mathcal {A}} \mu_ {h} (\theta | \bar {s}) \kappa (a | \theta) \left(r _ {h} ^ {\mathsf {S}} (\bar {s}, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | \bar {s}, a, \theta) \sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} (\bar {s}, M _ {\bar {h} + 1} ^ {\delta})} \tilde {q} (\iota^ {\prime} | a, s ^ {\prime}) M _ {\bar {h} + 1} ^ {\delta} (s ^ {\prime}, \iota^ {\prime})\right) \\ = \tilde {F} _ {\bar {h}, \bar {s}, M _ {\bar {h} + 1} ^ {\delta}} (\kappa , \tilde {q}) \\ = v \\ = \Omega_ {\bar {h}, \bar {s}, \bar {\iota}} (M _ {\bar {h} + 1} ^ {\delta}) \\ \geq \Pi_ {\bar {h}, \bar {s}, \bar {\iota}} \left(M _ {h + 1} ^ {\delta}\right) \\ \end{array}
$$
and thus Algorithm 4 returns a tuple that satisfies the conditions required by Definition 3.