SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {V} _ {h} ^ {\mathbf {S}, \sigma} (s, \iota) = \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \varphi_ {h} (a | s, \iota , \theta) \left(r _ {h} ^ {\mathbf {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \mathcal {V} _ {h + 1} ^ {\mathbf {S}, \sigma} \left(s ^ {\prime}, g _ {h} (s, a, \iota , s ^ {\prime})\right)\right) \\ \geq \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \varphi_ {h} (a | s, \iota , \theta) \left(r _ {h} ^ {\mathrm {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) M _ {h + 1} ^ {\delta} \left(s ^ {\prime}, g _ {h} (s, a, \iota , s ^ {\prime})\right)\right) \\ = \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \kappa (a | \theta) \left(r _ {h} ^ {\mathbf {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) M _ {h + 1} ^ {\delta} \left(s ^ {\prime}, q \left(a, s ^ {\prime}\right)\right)\right) \\ = F _ {h, s, M _ {h + 1} ^ {\delta}} (\kappa , q) \\ \geq v = M _ {h} ^ {\delta} (s, \iota) \\ \end{array}
$$
where the first inequality holds by the inductive assumption, while the second inequality holds thanks to Definition 3.
Lemma 5. Let  $M_h^\delta$ ,  $I_h$  (for  $h \in \mathcal{H}$ ) be computed by Algorithm 3 instantiated with any oracle  $\mathcal{O}_{h,s,\iota}$  as in Definition 3, and let  $\sigma^\star = \{(I_h^\star, \varphi_h^\star, g_h^\star)\}_{h \in \mathcal{H}}$  be an optimal promise-form signaling scheme. For every  $h \in \mathcal{H}$ ,  $s \in S$ , and  $\iota \in I_h^\star(s)$ , it holds that  $\lceil \iota \rceil_\delta \in I_h(s)$  and  $M_h^\delta(s, \lceil \iota \rceil_\delta) \geq \mathcal{V}_h^{\mathbf{S},\sigma^\star}(s,\iota)$ .
Proof. We prove by induction that  $M_h^\delta(s, \lceil \iota \rceil_\delta) \geq \mathcal{V}_h^{\sigma^\star}(s, \iota)$  for all  $h \in \mathcal{H}, s \in S, \text{and} \iota \in I_h^\star(s)$ .
As a base for the induction, we consider  $h = H$ . Notice that  $M_H(s, \lceil \iota \rceil_\delta)$  is an upper bound on the solution of  $\mathcal{P}_{H,s,\lfloor \iota \rfloor_\delta}(M_{H + 1}^\delta)$  (this holds by Definition 3). Moreover,  $\hat{\kappa}(a|\theta) = \varphi_H^\star(a|s, \iota, \theta)$  and  $\hat{q}(a,s') = 0$  is a feasible solution to  $\mathcal{P}_{H,s,\lfloor \iota \rfloor_\delta}(M_{H + 1}^\delta)$  and so  $(\hat{\kappa},\hat{q}) \in \Psi_{\lfloor \iota \rfloor_\delta}^{h,s} \subseteq \Psi_{\iota -\delta}^{h,s}$ . This readily implies that  $M_H^\delta(s,\lceil \iota \rceil_\delta) \geq \mathcal{V}_H^{\mathsf{S},\sigma^\star}(s,\iota)$ , since:
$$
\begin{array}{l} M _ {H} ^ {\delta} (s, \lceil \iota \rceil_ {\delta}) = v \\ \geq \Pi_ {h, s, \iota - \delta} (M _ {H + 1} ^ {\delta}) \\ = \max  _ {(\kappa , q) \in \Psi_ {\iota - \delta}} F _ {h, s, M _ {H + 1} ^ {\delta}} (\kappa , q) \\ \geq F _ {h, s, M _ {H + 1} ^ {\delta}} (\hat {\kappa}, \hat {q}) = \mathcal {V} _ {H} ^ {\sigma^ {\star}} (s, \iota). \\ \end{array}
$$
As for the inductive step, consider any  $h < H$ . First, we show that  $\hat{\kappa}(a|\theta) = \varphi_h^\star(a|s,\iota,\theta)$  and  $\hat{q}(a,s') = \lceil g_h^\star(s,a,\iota,s') \rceil_\delta$  is a feasible solution to  $\mathcal{P}_{h,s,\lfloor \iota \rfloor_\delta}(M_{h+1}^\delta)$ . Then, we show that it gets expected rewards larger than  $\mathcal{V}_h^{\mathsf{S},\sigma^\star}(s,\iota)$ .
First we prove feasibility of  $(\hat{\kappa},\hat{q})$ . Consider first the constraint of Equation (4a).
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \hat {\kappa} (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} \mid s, a, \theta\right) \hat {q} (a, s ^ {\prime})\right) \\ = \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \varphi_ {h} ^ {\star} (a | s, \iota , \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s, a, \theta) \lceil g _ {h} ^ {\star} (s, a, \iota , s ^ {\prime}) \rceil_ {\delta}\right) \\ \end{array}
$$
The proof is completed by applying the definition of OPT.
$$
\begin{array}{l} \geq \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \varphi_ {h} ^ {\star} (a | s, \iota , \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s, a, \theta) g _ {h} ^ {\star} (s, a, \iota , s ^ {\prime})\right) \\ \geq \iota \geq \left\lfloor \iota \right\rfloor_ {\delta}, \\ \end{array}
$$
which proves that  $(\hat{\kappa},\hat{q})$  satisfies the constraint of Equation (4a).
Now, let us turn our attention to the constraint of Equation (4b):
$$
\begin{array}{l} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \hat {\kappa} (a | \theta) \left(r _ {h} ^ {R} (s, a, \theta) + \sum_ {s ^ {\prime} \in S} p _ {h} \left(s ^ {\prime} \mid s, a, \theta\right) \hat {q} (a, s ^ {\prime})\right) \\ = \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \varphi_ {h} ^ {\star} (a | s, \iota , \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \lceil g _ {h} ^ {\star} (s, a, \iota , s ^ {\prime}) \rceil_ {\delta}\right) \\ \geq \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \varphi_ {h} ^ {\star} (a | s, \iota , \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} \mid s, a, \theta\right) g _ {h} ^ {\star} (s, a, \iota , s ^ {\prime})\right) \\ \geq \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \varphi_ {h} ^ {\star} (a | s, \iota , \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} \mid s, a, \theta\right) \widehat {V} _ {h + 1} ^ {\mathsf {R}} \left(s ^ {\prime}\right)\right), \\ \end{array}
$$
which proves  $(\hat{\kappa},\hat{q})$  satisfies the constraint of Equation (4b) and thus  $\mathcal{P}_{h,s,\lfloor \iota \rfloor_{\delta}}(M_{h + 1}^{\delta})$  is feasible, i.e.,  $(\hat{\kappa},\hat{q})\in \Psi_{\lfloor \iota \rfloor_{\delta}}\subset \Psi_{\iota -\delta}$ .
Now, we prove that it gets expected rewards larger than  $\sigma^{\star}$ .
$$
\begin{array}{l} M _ {h} (s, \lceil \iota \rceil_ {\delta}) = v \\ \geq \Pi_ {H, s, t - \delta} \left(M _ {h + 1} ^ {\delta}\right) \\ = \max  _ {(\kappa , q) \in \Psi_ {\iota - \delta}} F _ {h, s, M _ {h + 1} ^ {\delta} (\kappa , q)} \\ \geq F _ {h, s, M _ {h + 1} ^ {s}} (\hat {\kappa}, \hat {q}) \\ = \sum_ {\theta \in \Theta} \sum_ {a \in \mathcal {A}} \mu_ {h} (\theta | s) \hat {\kappa} (a | \theta) \left(r _ {h} ^ {\mathsf {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) M _ {h + 1} ^ {\delta} \left(s ^ {\prime}, \hat {q} (a, s ^ {\prime})\right)\right) \\ = \sum_ {\theta \in \Theta} \sum_ {a \in \mathcal {A}} \mu_ {h} (\theta | s) \varphi_ {h} ^ {\star} (a | \theta , \iota , \theta) \left(r _ {h} ^ {\mathbf {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) M _ {h + 1} ^ {\delta} \left(s ^ {\prime}, \lceil g _ {h} ^ {\star} (s, a, \iota , s ^ {\prime}) \rceil_ {\delta}\right)\right) \\ \geq \sum_ {\theta \in \Theta} \sum_ {a \in \mathcal {A}} \mu_ {h} (\theta | s) \varphi_ {h} ^ {\star} (a | \theta , \iota , \theta) \left(r _ {h} ^ {\mathsf {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \mathcal {V} _ {h + 1} ^ {s, \sigma^ {\star}} \left(s ^ {\prime}, g _ {h} ^ {\star} (s, a, \iota , s ^ {\prime})\right)\right) \\ = \mathcal {V} ^ {\mathrm {S}, \sigma^ {*}} \left(s ^ {\prime}, \iota\right) \\ \end{array}
$$
where we used the induction assumption in the second inequality.
In conclusion, for every  $\iota \in I_h^\star (s)$ , it holds  $\mathcal{V}_h^{\sigma^*}(s,\iota)\geq 0$ , and, thus,  $M_h^\delta (s,\lceil \iota \rceil_\delta)\geq 0$ , implying that  $\mathcal{P}_{h,s,\iota -\delta}$  is feasible and thus  $\iota \in I_{h}(s)$
Proof. First of all we need to prove that the  $\sigma$  returned by Algorithm 3 is indeed a promise-form signaling scheme. All the properties are trivially satisfied except for i)  $g_{h}(s,a,\iota ,s^{\prime})\in I_{h + 1}(s^{\prime})$  for each  $s\in S,a\in \mathcal{A},s'\in S$ , and  $\iota \in I_h(s)$ , and ii)  $0\in I_1(s)$  for each  $s$ .
We start proving the first property. Let  $s \in S, a \in \mathcal{A}, s' \in S$ , and  $\iota \in I_h(s)$ . Since  $\iota \in I_h(s)$ , by the definition of Algorithm 3 it holds that  $v > -\infty$ , where  $(\kappa, q, v) \gets \mathcal{O}_{h,s,\iota - \delta}(M_{h + 1}^{\delta})$  is the solution returned by the call to the oracle. It is easy to see that this implies that  $M_{h + 1}^{\delta}(s', q(a, s')) > -\infty$ . We consider two cases.
- If  $h = H$ , then we have by construction that  $M_{h + 1}^{\delta}(s', q(a, s')) > -\infty$  if and only if  $q(a, s') = 0$ . Thus,  $g_h(s, a, \iota, s') = q(a, s') = 0 \in I_{H + 1}(s')$ .
- If  $h < H$ , then we  $M_{h + 1}^{\delta}(s', q(a, s')) > -\infty$  implying that  $v' > -\infty$ , where  $(\kappa', q', v') \gets \mathcal{O}_{h + 1, s', q(a, s')} (M_{h + 2}^{\delta})$  is the solution returned by the call to the oracle at step  $h + 1$ . By the definition of Algorithm 3, this implies that  $q(a, s') \in I_{h + 1}(s')$ . Thus,  $g_h(s, a, \iota, s') = q(a, s') \in I_{H + 1}(s')$ .
Now, we prove the second property. We only need to prove that for all  $s \in S$  we have  $0 \in I_1(s)$  but this is easily proved by Lemma 5 instantiated for  $h = 1$  and  $\iota = 0$ . Indeed we have that  $M_1^\delta (s,0) \geq \mathcal{V}_h^{\mathrm{S},\sigma^{\star}}(s,0)$ , where we recall that  $M_1^\delta (s,0) = v$  and  $(\kappa ,q,v)\gets \mathcal{O}_{h,s, - \delta}(M_2^\delta)$ . By the definition of Algorithm 3 this implies that  $0 \in I_{1}(s)$ .
Then, we prove the persuasiveness. By Lemma 3 we only need to prove that  $\sigma := \{(I_h,\varphi_h,g_h)\}_{h\in \mathcal{H}}$  satisfies the constraint of Equation 2 and that are  $\eta$ -honest for some  $\eta$ . This holds trivially for  $\eta = 2\delta$  as for all  $h\in \mathcal{H},s\in \mathcal{S}$  and  $\iota \in I_{h}(s)$  we have by construction that for all  $h\in \mathcal{H},s\in \mathcal{S}$  and  $\iota \in I_{h}(s)$ , and  $(\kappa ,q,v)\gets \mathcal{O}_{h,s,\iota -\delta}(M_{h + 1}^{\delta})$  we have
$$
\varphi_ {h} (a | s, \iota , \theta) = \kappa (a | \theta), \quad g _ {h} (s, a, \iota , s ^ {\prime}) = q (a, s ^ {\prime})
$$
and by construction  $(\kappa, q) \in \Psi_{\iota - 2\delta}^{h,s}$ . This implies that the constraints of Equation (4a) and Equation (4b) are verified and we can apply Lemma 3 which let us conclude that  $\sigma$  is  $(2\delta H)$ -persuasive.
We now prove the optimality of the signaling scheme  $\sigma$  returned by Algorithm 3. By combining Lemma 4 and Lemma 5 we have that for all  $s\in S,h\in \mathcal{H}$  and  $\iota \in I_h^\star (s)$
$$
\mathcal {V} _ {h} ^ {\mathbf {S}, \sigma} (s, \lceil \iota \rceil_ {\delta}) \geq \mathcal {V} _ {h} ^ {\mathbf {S}, \sigma^ {\star}} (s, \iota).
$$
$$
\begin{array}{l} \mathrm {O P T} \leq V ^ {\mathbf {S}, \phi^ {\alpha^ {*}}} \\ = \sum_ {s \in \mathcal {S}} \beta (s) V _ {1} ^ {\mathrm {S}, \phi^ {\sigma^ {\star}}} ((s)) \\ = \sum_ {s \in \mathcal {S}} \beta (s) \mathcal {V} _ {1} ^ {\mathrm {S}, \phi^ {\sigma^ {\star}}} (s, 0) \\ \leq \sum_ {s \in \mathcal {S}} \beta (s) \mathcal {V} _ {1} ^ {\mathbb {S}, \sigma} (s, 0) \\ = \sum_ {s \in \mathcal {S}} \beta (s) V _ {1} ^ {\mathbf {S}, \phi^ {\sigma}} (s, 0) \\ = V ^ {\mathsf {S}, \phi^ {\sigma}}, \\ \end{array}
$$
which proves our statement.
E Proofs omitted from Section 6
In this section we prove that the optimization problem  $\mathcal{R}_{h,s,\iota}(M)$  admits a polynomial time algorithm that finds an exact solution. We rewrite here for clarity the problem  $\mathcal{R}_{h,s,\iota}(M)$ :
$$
\max  _ {\substack {\kappa : \Theta \rightarrow \Delta (\mathcal {A})\\\tilde {q}: \mathcal {A} \times \mathcal {S} \rightarrow \Delta (\mathcal {D} _ {\delta})}} \sum_ {\theta \in \Theta} \sum_ {a \in \mathcal {A}} \mu_ {h} (\theta | s) \kappa (a | \theta) \left(r _ {h} ^ {\mathrm {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \left(\sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} \tilde {q} \left(\iota^ {\prime} | a, s ^ {\prime}\right) M \left(s ^ {\prime}, \iota^ {\prime}\right)\right)\right) \tag{18a}
$$
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \kappa (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \left(\sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} \iota^ {\prime} \cdot \tilde {q} \left(\iota^ {\prime} | a, s ^ {\prime}\right)\right)\right) \geq \iota \tag {18b} \\ \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \kappa (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s, a, \theta) \left(\sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} (s ^ {\prime}, M)} \iota^ {\prime} \cdot \tilde {q} (\iota^ {\prime} | a, s ^ {\prime})\right)\right) \geq \\ \end{array}
$$
$$
\sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \kappa (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a ^ {\prime}, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} \mid s, a ^ {\prime}, \theta\right) \widehat {V} _ {h + 1} ^ {\mathsf {R}} \left(s ^ {\prime}\right)\right) \quad \forall a, a ^ {\prime} \in \mathcal {A}. \tag {18c}
$$
Even if the optimization problem is defined over functions, clearly we can represent the functions  $\kappa$  and  $q$  with finite number of variables with linear constraints (to assure that the function's outputs are distributions). In order to handle the quadratic terms of  $\mathcal{R}_{h,s,\iota}(M)$  we define a "product" variable  $z_{a,s',\iota}$  for every  $a \in \mathcal{A}, s' \in S$ , and  $\iota \in \mathcal{D}_{\delta}(s', M)$ , which is used in place of
$$
\sum_ {\theta \in \Theta} \mu_ {h} (\theta , s) \kappa (a | \theta) p _ {h} \left(s ^ {\prime} \mid s, a, \theta\right) \tilde {q} \left(\iota^ {\prime} \mid a, s ^ {\prime}\right)
$$
in Program 18a and has to satisfy the constraint for all  $a \in \mathcal{A}$  and  $s' \in S$ :
$$
\sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} (s ^ {\prime}, M)} z _ {a, s ^ {\prime}, \iota^ {\prime}} := \sum_ {\theta \in \Theta} \mu_ {h} (\theta , s) \kappa (a | \theta) p _ {h} (s ^ {\prime} | s, a, \theta).
$$
For the linear variable we can introduce the non-negative variables  $\xi_{a,\theta} = \kappa (a|\theta)$  for each  $\theta \in \Theta$  and  $a\in \mathcal{A}$ , that need to satisfy the simplex constraint for all  $\theta \in \Theta$ , i.e.,  $\sum_{a\in \mathcal{A}}\xi_{a,\theta} = 1$ . By using these variables we can write the problem as an  $\mathrm{LP}_{h,s,\iota}(M)$ :
$$
\max  _ {\substack {\xi_ {a, \theta} \in \mathbb {R} _ {+}, \\ z _ {a, s ^ {\prime}, \iota^ {\prime}} \in \mathbb {R} _ {+}}} \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) r _ {h} ^ {\mathrm {S}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} \sum_ {a \in \mathcal {A}} \sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} z _ {a, s ^ {\prime}, \iota^ {\prime}} M \left(s ^ {\prime}, \iota^ {\prime}\right) \tag{19a}
$$
$$
\sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) r _ {h} ^ {\mathrm {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} \sum_ {a \in \mathcal {A}} \sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} \iota^ {\prime} \cdot z _ {a, s ^ {\prime}, \iota^ {\prime}} \geq \iota \tag {19b}
$$
$$
\sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} \sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} (s ^ {\prime}, M)} \iota^ {\prime} \cdot z _ {a, s ^ {\prime}, \iota^ {\prime}} \geq
$$
$$
\sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) \left(r _ {h} ^ {\mathsf {R}} (s, a ^ {\prime}, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} \mid s, a ^ {\prime}, \theta\right) \widehat {V} _ {h + 1} ^ {\mathsf {R}} \left(s ^ {\prime}\right)\right) \quad \forall a, a ^ {\prime} \in \mathcal {A} \tag {19c}
$$
$$
\sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} z _ {a, s ^ {\prime}, \iota^ {\prime}} = \sum_ {\theta \in \Theta} \mu_ {h} (\theta , s) \xi_ {a, \theta} p _ {h} \left(s ^ {\prime} \mid s, a, \theta\right) \quad \forall s ^ {\prime} \in \mathcal {S}, \forall a \in \mathcal {A} \tag {19d}
$$
$$
\sum_ {a \in \mathcal {A}} \xi_ {a, \theta} = 1 \quad \forall \theta \in \Theta . \tag {19e}
$$
Since  $\mathrm{LP}_{h,s,\iota}(M)$  has polynomial number of variable and constraints one can find a solution in polynomial time.
The next two lemmas show that one can use  $\mathrm{LP}_{h,s,\iota}(M)$ . The first shows that any solution to  $\mathcal{R}_{h,s,\iota}(M)$  can be used to find a solution to  $\mathrm{LP}_{h,s,\iota}(M)$ .
Lemma 7. Let  $(\kappa, q) \in \Psi_{\iota}^{h,s}$  be a feasible solution to  $\mathcal{R}_{h,s,\iota}(M)$ , then there exists a feasible solution to  $LP_{h,s,\iota}(M)$  with the same value.
Proof. Let  $(\kappa, \tilde{q})$  be a feasible solution to  $\mathcal{R}_{h,s,\iota}(M)$ . Then define:
$$
z _ {a, s ^ {\prime}, \iota^ {\prime}} := \sum_ {\theta \in \Theta} \mu_ {h} (\theta , s) \kappa (a | \theta) p _ {h} (s ^ {\prime} | s, a, \theta) \tilde {q} (\iota^ {\prime} | a, s ^ {\prime}), \forall a \in \mathcal {A}, s ^ {\prime} \in \mathcal {S}, \iota^ {\prime} \in \mathcal {D} _ {\delta} (s ^ {\prime}, M),
$$
$$
\xi_ {a, \theta} := \kappa (a | \theta), \forall a \in \mathcal {A}, \theta \in \Theta .
$$
Then one can show by direct calculation that all the constraints of  $\mathrm{LP}_{h,s,\iota}(M)$  are satisfied and that the objective value of  $\mathrm{LP}_{h,s,\iota}(M)$  is  $\Omega_{h,s,\iota}(M)$  (which is the value of program  $\mathcal{R}_{h,s,\iota}(M)$ ).
The next lemma shows a result which is "complementary" to the one above.
Lemma 8. Given a feasible solution of  $LP_{h,s,\iota}(M)$  one can find a solution to  $\mathcal{R}_{h,s,\iota}(M)$  with at least the same value.
Proof. Given a feasible solution  $z$  and  $\xi$  to  $\mathrm{LP}_{h,s,\iota}(M)$ . Construct a solution  $(\kappa, \tilde{q})$  to  $\mathcal{R}_{h,s,\iota}(M)$  as follows for each  $\iota' \in \mathcal{D}_{\delta}(s', M)$ ,  $a \in \mathcal{A}$  and  $s \in S$ :
$$
\tilde {q} (\iota^ {\prime} | a, s ^ {\prime}) := \left\{ \begin{array}{l l} \frac {z _ {a , s ^ {\prime} , \iota^ {\prime}}}{\sum_ {\theta \in \Theta} \mu_ {h} (\theta , s) \xi_ {a , \theta} p _ {h} (s ^ {\prime} | s , a , \theta)} & \text {i f} \sum_ {\theta \in \Theta} \mu_ {h} (\theta , s) \xi_ {a, \theta} p _ {h} (s ^ {\prime} | s, a, \theta) > 0 \\ \mathbb {I} (\iota^ {\prime} = 0) & \text {o t h e r w i s e} \end{array} \right.
$$
and for all  $a\in \mathcal{A}$  and  $\theta \in \Theta$
$$
\kappa (a | \theta) = \xi_ {a, \theta}.
$$
Notice that if for a specific  $h \in \mathcal{H}$ ,  $s \in \mathcal{S}$ ,  $s' \in \mathcal{S}$  and  $a \in \mathcal{A}$ , the constraint of Equation (19d) imposes that if  $\sum_{\theta \in \Theta} \mu_h(\theta, s) \xi_{a,\theta} p_h(s'|s, a, \theta) = 0$  then  $z_{a,s',\iota'} = 0$  for all  $\iota'$ . This means that in any case the following condition holds:
$$
z _ {a, s ^ {\prime}, \iota^ {\prime}} = \sum_ {\theta \in \Theta} \mu_ {h} (\theta , s) \xi_ {a, \theta} p _ {h} \left(s ^ {\prime} \mid s, a, \theta\right) \tilde {q} \left(\iota^ {\prime} \mid a, s ^ {\prime}\right). \tag {20}
$$
We now fix any  $h \in \mathcal{H}, s \in \mathcal{S}$  and  $\iota \in \mathcal{D}_{\delta}$  show that the constraints of  $\mathcal{R}_{h,s,\iota}(M)$  are satisfied, by using Equation (20). Let us start with the constraint of Equation (18b):
$$
\begin{array}{l} \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \kappa (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \left(\sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} \iota^ {\prime} \cdot \tilde {q} \left(\iota^ {\prime} | a, s ^ {\prime}\right)\right)\right) \\ = \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \sum_ {\iota \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} \sum_ {s ^ {\prime} \in \mathcal {S}} \iota^ {\prime} \xi_ {a, \theta} \mu_ {h} (\theta | s) p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \tilde {q} \left(\iota^ {\prime} | a, s ^ {\prime}\right) \\ = \sum_ {a \in \mathcal {A}} \sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {a \in \mathcal {A}} \sum_ {s \in \mathcal {S}} \sum_ {\iota \in \mathcal {D} _ {\delta} (s ^ {\prime}, M)} \iota^ {\prime} \cdot z _ {a, s ^ {\prime}, \iota^ {\prime}} \\ \geq \iota , \\ \end{array}
$$
where the last inequality holds as  $(z,\xi)$  is a feasible solution to  $\mathrm{LP}_{h,s,t}(M)$ . This proves that the constraint of Equation (18b) is satisfied.
For Equation (18c) similarly:
$$
\begin{array}{l} \sum_ {\theta \in \Theta} \mu_ {h} (\theta | s) \kappa (a | \theta) \left(r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \left(\sum_ {\iota^ {\prime} \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} \iota^ {\prime} \cdot \tilde {q} \left(\iota^ {\prime} | a, s ^ {\prime}\right)\right)\right) \\ = \sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {\theta \in \Theta} \sum_ {\iota \in \mathcal {D} _ {\delta} \left(s ^ {\prime}, M\right)} \sum_ {s ^ {\prime} \in \mathcal {S}} \iota^ {\prime} \xi_ {a, \theta} \mu_ {h} (\theta | s) p _ {h} \left(s ^ {\prime} | s, a, \theta\right) \tilde {q} \left(\iota^ {\prime} | a, s ^ {\prime}\right) \\ = \sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) r _ {h} ^ {\mathsf {R}} (s, a, \theta) + \sum_ {\theta \in \Theta} \sum_ {\iota \in \mathcal {D} _ {\delta} (s ^ {\prime}, M)} \sum_ {s ^ {\prime} \in \mathcal {S}} \iota^ {\prime} \cdot z _ {a, s ^ {\prime}, \iota^ {\prime}} \\ \geq \sum_ {\theta \in \Theta} \xi_ {a, \theta} \mu_ {h} (\theta | s) \left(r _ {h} ^ {\mathsf {R}} (s, a ^ {\prime}, \theta) + \sum_ {s ^ {\prime} \in \mathcal {S}} p _ {h} (s ^ {\prime} | s, a ^ {\prime}, \theta) \widehat {V} _ {h + 1} ^ {\mathsf {R}} (s ^ {\prime})\right), \\ \end{array}
$$
where in the last inequality we used that  $(z,\xi)$  is a feasible solution to  $\mathrm{LP}_{h,s,\iota}(M)$ . Thus the constraint of Equation (18c) is verified. This proves that  $(\kappa ,\tilde{q})$  is feasible for  $\mathcal{R}_{h,s,\iota}(M)$ . Then, by plugging Equation (20) into the objective of  $\mathcal{R}_{h,s,\iota}(M)$  one directly proves that the values of the two problems is the same.
Now we are ready to prove the main result of this section: