SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now take the expectation of the sum. We obtain by tower property of the conditional expectation, and using the i.i.d. assumption:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} [ \varphi (\lambda , c _ {t, h _ {t} (z _ {t})}, h _ {t} (z _ {t}), z _ {t}) ] = \sum_ {t = 1} ^ {T} \mathbb {E} [ \mathbb {E} [ \varphi (\lambda , c, h _ {t} (z _ {t}), z _ {t}) \mid z _ {t} ] ] \\ = \sum_ {t = 1} ^ {T} \sum_ {z \in \mathcal {Z}} \mathbb {E} [ \varphi (\lambda , c, h _ {t} (z), z) \mid z ] \mu (z) \\ = \sum_ {z \in \mathcal {Z}} \sum_ {t = 1} ^ {T} \mathbb {E} [ \varphi (\lambda , c, h _ {t} (z), z) \mid z ] \mu (z) \\ = \sum_ {z \in \mathcal {Z}} \sum_ {k \in [ K ]} \sum_ {\substack {t = 1 \\ h _ {t} (z) = k}} ^ {T} \mathbb {E} [ \varphi (\lambda , c, k, z) \mid z ] \mu (z) \\ = T \sum_ {z \in \mathcal {Z}} \sum_ {k \in [ K ]} \pi_ {k} (z) \mathbb {E} _ {c} [ \varphi (\lambda , c, k, z) \mid z ] \mu (z). \\ \end{array}
$$
Clearly  $\pi_k(z) \in \mathcal{P}_K$ , thus if we maximize over this space for the  $\pi_k(z)$  instead of over  $([K]^{\mathcal{Z}})^T$ , it will be higher. Therefore
$$
\operatorname {O P T} \leq T \sup  _ {\pi \in \mathcal {P} _ {K} ^ {\mathcal {Z}}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \left(R ^ {*} (\lambda) + \sum_ {z \in \mathcal {Z}} \sum_ {k \in [ K ]} \pi_ {k} (z) \mathbb {E} _ {c} [ \varphi (\lambda , c, k, z) \mid z ] \mu (z)\right). \tag {43}
$$
Notice that  $\mu (z)$  plays an important role in determining the optimal policy. If we had simply run the previous algorithm for each context, this would be akin to considering that the  $z_{t}$  are distributed uniformly according to  $\mu (z) = 1 / Z$ , which would have yielded a sub-optimal policy  $\pi$ .
Now let us take care of the lower bound. As done in Equation (25), we have that
$$
\mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \bar {R} \left(\gamma_ {t}\right) \right] \geq \mathbb {E} \left[ \varphi \left(\lambda_ {t}, k _ {t}, z _ {t}, c _ {t k _ {t}}\right) + R ^ {*} \left(\lambda_ {t}\right) + \left\langle \lambda_ {t}, \delta_ {t} - \gamma_ {t} \right\rangle \right],
$$
and as we did before we will use bandits algorithm on these rewards.
We would now like to bound the following adversarial contextual bandit problem regret:
$$
\mathrm{Reg} = \sum_{z\in \mathcal{Z}}\max_{k\in [K]}\sum_{\substack{t = 1\\ z_{t} = z}}^{T}\mathbb{E}[\varphi (\lambda_{t},c_{t},k,z)\mid \mathcal{H}_{t - 1},z_{t}] - \sum_{t = 1}^{T}\mathbb{E}[\varphi (\lambda_{t},c_{t},k_{t},z_{t})\mid \mathcal{H}_{t - 1},z_{t}].
$$
The weighted estimator  $\hat{\varphi}$  is unbiased as  $\varphi (\lambda_t,c_t,k_t,z_t)\mathbb{1}[k_t = k] / \pi_{tk}(z_t)$  is still unbiased. Indeed,  $c_{t}$  and  $k_{t}$  are conditionally independent given  $(z_{t},\mathcal{H}_{t - 1})$  , and  $\pi_{tk}(z_t)$  is  $(z_{t},\mathcal{H}_{t - 1})$  measurable. Thus
$$
\begin{array}{l} \mathbb {E} \left[ \frac {\varphi \left(\lambda_ {t} , c _ {t} , k _ {t} , z _ {t}\right) \mathbb {1} \left[ k _ {t} = k \right]}{\pi_ {t k} (z _ {t})} \mid \mathcal {H} _ {t - 1}, z _ {t} \right] = \frac {\mathbb {E} \left[ \varphi \left(\lambda_ {t} , c _ {t} , k _ {t} , z _ {t}\right) \mathbb {1} \left[ k _ {t} = k \right] \mid \mathcal {H} _ {t - 1} , z _ {t} \right]}{\pi_ {t k} (z _ {t})} \\ = \frac {\mathbb {E} [ \varphi (\lambda_ {t} , c _ {t} , k _ {t} , z _ {t}) \mid \mathcal {H} _ {t - 1} , z _ {t} ] \mathbb {E} [ \mathbb {1} [ k _ {t} = k ] \mid \mathcal {H} _ {t - 1} , z _ {t} ]}{\pi_ {t k} (z _ {t})} \\ = \mathbb {E} [ \varphi (\lambda_ {t}, c _ {t}, k _ {t}, z _ {t}) \mid \mathcal {H} _ {t - 1}, z _ {t} ]. \\ \end{array}
$$
As suggested section (18.1) of Lattimore & Szepesvári (2020) running one anytime version of  $EXP3$  (using a doubling trick for instance) on each of these public contexts  $z \in \mathcal{Z}$ , achieves a regret bound of order  $\mathcal{O}(\sqrt{K\log(K)}\sum_{z \in \mathcal{Z}}\sqrt{\sum_{t=1}^{T}\mathbb{1}[z_t = z]})$ .
Taking the expectation (for  $z_{t}$ ) of this regret bound, and using Jensen's inequality for the square root function we have a regret term of order
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {O} \left(\sqrt {K \log (K)} \sum_ {z \in \mathcal {Z}} \sqrt {\sum_ {t \in [ T ]} \mathbb {1} \left[ z _ {t} = z \right]}\right) \right] \leq \mathcal {O} \left(\sqrt {K \log (K)} \sum_ {z \in \mathcal {Z}} \sqrt {\mathbb {E} \left[ \sum_ {t \in [ T ]} \mathbb {1} \left[ z _ {t} = z \right] \right]}\right) \\ = \mathcal {O} \left(\sqrt {T K \log (K)} \sum_ {z \in \mathcal {Z}} \sqrt {\mu (z)}\right). \tag {44} \\ \end{array}
$$
Let  $(\pi^n) \in (\mathcal{P}_K^{\mathcal{Z}})^{\mathbb{N}}$  be the sequence of contextual source mixing that maximizes the upper bound for OPT in Equation (43). Then because the maximum of  $[K]$  points is greater than any convex combination,  $\pi^n(z)$  in particular, we have
$$
\begin{array}{l} \sum_{z\in \mathcal{Z}}\max_{k\in [K]}\sum_{\substack{t = 1\\ z_{t} = z}}^{T}\mathbb{E}[\varphi (\lambda_{t},c_{t},k,z)\mid \mathcal{H}_{t - 1},z]\geq \sum_{z\in \mathcal{Z}}\sum_{k\in [K]}\pi_{k}^{n}(z)\sum_{\substack{t = 1\\ z_{t} = z}}^{T}\mathbb{E}[\varphi (\lambda_{t},c,k,z)\mid \mathcal{H}_{t - 1},z] \\ = \sum_ {k \in [ K ]} \sum_ {t = 1} ^ {T} \pi_ {k} ^ {n} (z _ {t}) \mathbb {E} \left[ \varphi \left(\lambda_ {t}, c, k, z _ {t}\right) \mid \mathcal {H} _ {t - 1}, z _ {t} \right]. \\ \end{array}
$$
In expectation this yields:
$$
\begin{array}{l} \mathbb{E}[\sum_{z\in \mathcal{Z}}\max_{k\in [K]}\sum_{\substack{t = 1\\ z_{t} = z}}^{T}\mathbb{E}[\varphi (\lambda_{t},c_{t},k,z)\mid \mathcal{H}_{t - 1},z]]\geq \mathbb{E}[\sum_{k\in [K]}\sum_{t = 1}^{T}\pi_{k}^{n}(z_{t})\mathbb{E}[\varphi (\lambda_{t},c,k,z_{t})\mid \mathcal{H}_{t - 1},z_{t}]] \\ = \sum_ {k \in [ K ]} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathbb {E} \left[ \pi_ {k} ^ {n} \left(z _ {t}\right) \mathbb {E} \left[ \varphi \left(\lambda_ {t}, c, k, z _ {t}\right) \mid \mathcal {H} _ {t - 1}, z _ {t} \right] \mid \mathcal {H} _ {t - 1} \right] \right], \\ \end{array}
$$
where we used the linearity of the expectation, and the tower property of the expectation for the last equality.
Because  $z_{t}$  and  $\mathcal{H}_{t-1}$  are independent, we can first use Lemma D.3 and then the freezing lemma to express the conditional expectation given  $\mathcal{H}_{t-1}$  as
$$
\begin{array}{l} \mathbb {E} [ \mathbb {E} [ \pi_ {k} ^ {n} (z _ {t}) ] \mathbb {E} [ \varphi (\lambda_ {t}, c, k, z _ {t}) \mid \mathcal {H} _ {t - 1}, z _ {t} ] \mid \mathcal {H} _ {t - 1} ] ] = \mathbb {E} [ \mathbb {E} [ \pi_ {k} ^ {n} (z _ {t}) ] \mathbb {E} _ {c} [ \varphi (\lambda_ {t}, c, k, z) \mid z ] \mid \mathcal {H} _ {t - 1} ] ] \\ = \mathbb {E} \left[ \sum_ {z \in \mathcal {Z}} \mu (z) \pi_ {k} ^ {n} \left(z _ {t}\right) \mathbb {E} _ {c} \left[ \varphi \left(\lambda_ {t}, c, k, z\right) \mid z \right] \right] \\ \end{array}
$$
$$
\mathbb{E}[\sum_{z\in \mathcal{Z}}\max_{k\in [K]}\sum_{\substack{t = 1\\ z_{t} = z}}^{T}\mathbb{E}[\varphi (\lambda_{t},c_{t},k,z)\mid \mathcal{H}_{t - 1},z]]\geq \sum_{t = 1}^{T}\mathbb{E}[\sum_{k\in [K]}\sum_{z\in \mathcal{Z}}\mu (z)\pi_{k}^{n}(z)\mathbb{E}_{c}[\varphi (\lambda_{t},c,k,z)\mid z]].
$$
Grouping it together with the  $R^{*}(\lambda_{t})$  terms, and taking the inf over  $\lambda$  we derive that
$$
\begin{array}{l} \mathbb{E}\left[\sum_{t = 1}^{T}R^{*}(\lambda_{t}) + \sum_{z\in \mathcal{Z}}\max_{k\in [K]}\sum_{\substack{t = 1\\ z_{t} = z}}^{T}\mathbb{E}[\varphi (\lambda_{t},c_{t},k,z)\mid \mathcal{H}_{t - 1},z]\right] \\ \geq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \inf  _ {\lambda \in \mathbb {R} ^ {d}} \left(R ^ {*} (\lambda) + \sum_ {k \in [ K ]} \sum_ {z \in \mathcal {Z}} \mu (z) \pi_ {k} ^ {n} (z) \mathbb {E} _ {c} [ \varphi (\lambda , c, k, z) \mid z ]\right) \right] \\ \geq T \inf  _ {\lambda \in \mathbb {R} ^ {d}} \left(R ^ {*} (\lambda) + \sum_ {k \in [ K ]} \sum_ {z \in \mathcal {Z}} \mu (z) \pi_ {k} ^ {n} (z) \mathbb {E} _ {c} [ \varphi (\lambda , c, k, z) \mid z ]\right). \\ \end{array}
$$
As  $n\to \infty$  and by definition of  $\pi^n$  , we can conclude that
$$
\begin{array}{l} \mathbb{E}\left[\sum_{t = 1}^{T}R^{*}(\lambda_{t}) + \sum_{z\in \mathcal{Z}}\max_{k\in [K]}\sum_{\substack{t = 1\\ z_{t} = z}}^{T}\mathbb{E}[\varphi (\lambda_{t},c_{t},k,z)\mid \mathcal{H}_{t - 1},z]\right] \\ = T \sup  _ {\pi \in \mathcal {P} _ {K} ^ {\mathcal {Z}}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \left(R ^ {*} (\lambda) + \sum_ {z \in \mathcal {Z}} \sum_ {k \in [ K ]} \pi_ {k} (z) \mathbb {E} _ {c} [ \varphi (\lambda , c, k, z) \mid z ] \mu (z)\right) \\ \geq \mathrm {O P T}. \\ \end{array}
$$
Finally, using Equation (44) and the previous inequality we obtain
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \varphi \left(\lambda_ {t}, k _ {t}, z _ {t}, c _ {t}\right) + R ^ {*} \left(\lambda_ {t}\right) \right] \geq \operatorname {O P T} - \mathcal {O} \left(\sqrt {T K \log (K)} \sum_ {z \in \mathcal {Z}} \sqrt {\mu (z)}\right). \tag {45}
$$
The rest of the proof follows Appendix E.
Clearly, if  $\mu$  has a non-zero probability only for one  $z$ , we recover the previous regret bound without public information.
J.2 Infinite number of contexts - proof of Proposition 3.5
Lemma. If  $\mathbb{E}[a_t(x) \mid c_{tk}, z_t]$  and  $\mathbb{E}[f_t(x) \mid c_{tk}, z_t]$  are Lipschitz continuous in  $z$  with respective constants  $L_a$  and  $L_f$  for all  $x \in \mathcal{X}$  with respect to  $\|\cdot\|_2$ , then so is  $\varphi$  in  $z$  with Lipschitz constant  $L_a + L_f$ .
Proof. Let  $\lambda \in \mathbb{R}^d$ ,  $k \in [K]$ ,  $c_{tk}$  the additional information,  $z_1, z_2 \in \mathcal{Z}$ , and  $x_1, x_2 \in \mathcal{X}$  be the maximizers that respectively yields  $\varphi(\lambda, c_{tk}, k, z_1)$  and  $\varphi(\lambda, c_{tk}, k, z_2)$ . Because  $x_2$  is a maximizer, we have that
$$
\begin{array}{l} \varphi (\lambda , c _ {t k}, k, z _ {1}) - \varphi (\lambda , c _ {t k}, k, z _ {2}) \\ \leq \mathbb {E} [ f _ {t} (x _ {1}) \mid c _ {t k}, z _ {1} ] - \mathbb {E} [ f _ {t} (x _ {1}) \mid c _ {t k}, z _ {1} ] + \langle \lambda \mid \mathbb {E} [ a _ {t} (x _ {1}) \mid c _ {t k}, z _ {2} ] - \mathbb {E} [ a _ {t} (x _ {1}) \mid c _ {t k}, z _ {2} ] \rangle \\ \leq L _ {f} \| z _ {1} - z _ {2} \| _ {2} + L _ {a} \| a _ {t} (x _ {1}) \| _ {2} \| z _ {1} - z _ {2} \| _ {2} \\ \leq L _ {f} + L _ {a}. \\ \end{array}
$$
Using the same arguments because  $x_{1}$  is a maximizer, we obtain the symmetric inequality.
The main idea for the discretization (that averages the rewards over the discretized bins), is that the performance of the algorithm is close to the discretized optimal, which itself is close to the continuous optimal for smoothness reasons. However here the discretization is applied not to the original total utility  $\mathcal{U}$ , but instead to the bandits reward part. The number of discretized bins is then tuned depending on the lipschitz constants and the space dimension  $r$ .
The proof of the proposition Proposition 3.5 then directly follows from exercise 19.5 of Lattimore & Szepesvári (2020) applied to the contextual bandit problem with rewards  $\varphi$ .
Remark. In the case when  $\mathcal{Z}$  is continuous, there are no guarantees that there exists an optimal policy  $\pi^{*}$ . Indeed, if for the product topology the space  $\mathcal{P}_K^{\mathcal{Z}}$  is compact (by Tychonoff's theorem), the dual function used to derive OPT is not continuous. Vice versa, for a topology which makes this dual function continuous, it is unlikely for this space of functions to be compact. Hence why we need to take a maximizing sequence of  $\pi_{n}$  so that the dual function of these  $\pi_{n}$  converges to the upper bound of OPT.