SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We denote by  $\mu$  the distribution of  $(X,Z)$  and  $\nu$  the distribution of  $Y$ . By the Doob-Dynkin Lemma, because  $W$  is  $\sigma(Y,Z)$  measurable, we have that  $W = h(Y,Z)$  where  $h$  is measurable. First, by independence of  $Y$  and  $(X,Z)$ , we have that the distribution of  $(X,Y,Z)$  is  $d\nu \otimes d\mu$ . Thus
$$
\mathbb {E} [ g (X, Y, Z) W ] = \int \int g (x, y, z) h (y, z) d \nu (x, z) d \nu (y).
$$
Now let us compute the other expectation using Fubini theorem as both  $g$  and  $h$  are bounded:
$$
\begin{array}{l} \mathbb {E} [ V W ] = \int \int V (y, z) h (y, z) d \mu (z) d \nu (y) \\ = \int d \nu (y) \left(\int V (y, z) h (y, z) d \mu (z)\right) \\ = \int d \nu (y) \left(\mathbb {E} _ {Z} [ V (y, Z) h (y, Z) ]\right). \\ \end{array}
$$
The random variable  $h(y,Z)$  is  $\sigma (Z)$  measurable, thus because  $V(y,Z)$  is a conditional expectation given  $\sigma (Z)$ , we have
$$
\mathbb {E} _ {Z} [ V (y, Z) h (y, Z) ] = \mathbb {E} _ {X, Z} [ g (X, y, Z) h (y, Z) ] = \int g (x, y, z) h (y, z) d \mu (x, z).
$$
Using Fubini again we obtain
$$
\begin{array}{l} \mathbb {E} [ V W ] = \int d \nu (y) \left(\int g (x, y, z) h (y, z) d \mu (x, z)\right) \\ = \int \int g (x, y, z) h (y, z) d \mu (x, z) d \nu (y) \\ = \mathbb {E} [ g (X, Y, Z) W ], \\ \end{array}
$$
which proves that this is indeed the conditional expectation, as the lemma stated.
D.3 Boundedness of  $\lambda_{t}$  — proof of Lemma 3.2
We next show the proof of the boundedness of the  $\lambda_{t}$  for the sequence generated by our Algorithm 1. Lemma. Let  $\lambda_0\in \mathbb{R}^d$ $\eta >0$  , and an arbitrary sequences  $(\delta_1,\delta_2,\ldots)\in \Delta^{\mathbb{N}}$  . We define recursively  $\lambda_t$  as follows:
$$
\lambda_ {t + 1} = \lambda_ {t} + \eta (\delta_ {t} - \gamma_ {t}).
$$
Then for  $\Lambda = \cup_{\delta \in \Delta} \partial \bar{R}(\delta)$  we have the following upper bound:
$$
\left\| \lambda_ {t} \right\| _ {2} \leq L + 2 \eta \operatorname {D i a m} (\Delta) + \operatorname {D i s t} \left(\lambda_ {0}, \Lambda + B (0, 2 \eta \operatorname {D i a m} (\Delta))\right),
$$
where  $B(0,2\eta \mathrm{Diam}(\Delta))$  is the ball centered in 0 and of diameter  $2\eta$ $\mathrm{Diam}(\Delta)$ , with the diameter and the distance taken with respect to the euclidean  $L_{2}$  norm.
Proof. Recall that  $\bar{\Delta} = \cup_{\delta \in \Delta} \Delta_{\delta}$ , with  $\Delta_{\delta} = \{\gamma : \| \gamma - \delta \|_2 \leq \mathrm{Diam}(\Delta)\}$ . Hence, for any  $\delta \in \Delta$  and  $\gamma \in \bar{\Delta}$ , we have  $\| \gamma - \delta \|_2 \leq 2 \mathrm{Diam}(\Delta)$ . Let  $\Lambda = \cup_{\delta \in \bar{\Delta}} \partial \bar{R}(\delta)$ , and  $\mathcal{C} = \mathrm{Conv}(\Lambda) + B(0, \eta 2 \mathrm{Diam}(\Delta))$ . Using that  $\bar{R}$  is L-Lipschitz, we have that  $\Lambda$  is bounded by  $L$  for the norm  $\| \cdot \|_2$  and so does  $\mathrm{Conv}(\Lambda)$ . Thus  $\mathcal{C}$  is bounded by  $L + 2\eta \mathrm{Diam}(\Delta)$  for  $\| \cdot \|_2$ . This implies that, to prove the theorem, it suffices to show that  $\mathrm{Dist}(\lambda_{t+1}, \mathcal{C}) \leq \mathrm{Dist}(\lambda_t, \mathcal{C})$  is true for all  $t$ . Indeed, this would imply that
$$
\| \lambda_ {t} \| _ {2} \leq \sup  _ {c \in \mathcal {C}} \| c \| _ {2} + \operatorname {D i s t} (\lambda_ {t}, \mathcal {C}) \leq L + 2 \eta \operatorname {D i a m} \Delta + \operatorname {D i s t} (\lambda_ {0}, \mathcal {C}).
$$
In the remainder of the proof, we show that  $\mathrm{Dist}(\lambda_{t + 1},\mathcal{C})\leq \mathrm{Dist}(\lambda_t,\mathcal{C})$ . First, let us remark that  $\gamma_{t}$  is the solution of a convex optimization problem, with a constraint set  $\Delta_{\delta_t}$  that can be rewritten as a single inequality:  $h_t(\gamma)\leq 0$ , with  $h_t(\gamma) = \| \gamma -\delta_t\| _2^2 -\mathrm{Diam}(\Delta)^2$ . The derivative of this function  $h_t()$  is  $\nabla h_t(\gamma) = 2(\gamma -\delta_t)$ . As the point  $\delta_t$  is in the interior (assuming that  $\mathrm{Diam}(\Delta) > 0$ ) of  $\Delta_{\delta_t}$ , Slater's conditions hold, and we can apply the KKT conditions at the optimal point  $(\gamma_{t})$ . This implies that there exists  $\mu \geq 0$  such that  $0\in -\lambda_{t} + \partial \bar{R} (\gamma_{t}) + \mu 2(\gamma_{t} - \delta_{t})$ . This implies that there exists  $\lambda_{\delta_t}\in \partial \bar{R} (\gamma_t)$  such that
$$
\lambda_ {\delta_ {t}} - \lambda_ {t} = 2 \mu (\delta_ {t} - \gamma_ {t}).
$$
The norm of the left-hand side of the above equation must be equal to the norm of its right-hand side, which means that  $\| \lambda_{\delta_t} - \lambda_t\| _2 = 2\mu \| \delta_t - \gamma_t\| _2$ . This determines the value of  $\mu$ . Let  $\alpha_{t} = \eta \| \delta_{t} - \gamma_{t}\|_{2} / \| \lambda_{\delta_{t}} - \lambda_{t}\|_{2}$ , the value  $\lambda_{t + 1}$  is therefore equal to:
$$
\lambda_ {t + 1} = \lambda_ {t} + \eta (\delta_ {t} - \gamma_ {t}) = \lambda_ {t} + \alpha_ {t} \left(\lambda_ {\delta_ {t}} - \lambda_ {t}\right) = (1 - \alpha_ {t}) \lambda_ {t} + \alpha_ {t} \lambda_ {\delta_ {t}}.
$$
We now consider multiple cases:
Case 1. If  $\alpha_{t} > 1$ , then we rewrite  $\lambda_{t + 1}$  as:
$$
\begin{array}{l} \lambda_ {t + 1} = \lambda_ {\delta_ {t}} + (1 - \alpha_ {t}) (\lambda_ {t} - \lambda_ {\delta_ {t}}) \\ = \lambda_ {\delta_ {t}} + \eta (\| \delta_ {t} - \gamma_ {t} \| _ {2} - \| \lambda_ {\delta_ {t}} - \lambda_ {t} \| _ {2}) \frac {\lambda_ {t} - \lambda_ {\delta_ {t}}}{\| \lambda_ {t} - \lambda_ {\delta_ {t}} \| _ {2}}. \\ \end{array}
$$
By assumption, the  $\lambda_{\delta_t} \in \partial \bar{R}(\gamma_t) \subset \Lambda$ . Moreover, the norm of the second term is bounded by  $2\eta \mathrm{Diam}(\Delta)$  because
$$
\left| \left| \left| \delta_ {t} - \gamma_ {t} \right| \right| _ {2} - \left| \left| \lambda_ {\delta_ {t}} - \lambda_ {t} \right| \right| _ {2} \right| = \left| \left| \delta_ {t} - \gamma_ {t} \right| \right| _ {2} - \left| \left| \lambda_ {\delta_ {t}} - \lambda_ {t} \right| \right| _ {2} \leq 2 \operatorname {D i a m} (\Delta).
$$
This implies that  $\lambda_{t + 1}\in \mathrm{Conv}(\Lambda) + B(0,2\eta \operatorname {Diam}(\Delta)) = \mathcal{C}$ , and therefore that  $\mathrm{Dist}(\lambda_{t + 1},\mathcal{C}) = 0\leq \mathrm{Dist}(\lambda_t,\mathcal{C})$
Case 2. If  $\alpha_{t} \leq 1$  and  $\lambda_{t} \in \mathcal{C}$ . In this case,  $\lambda_{t+1}$  is a convex combination of  $\lambda_{\delta_{t}}$  and  $\lambda_{t}$ . As,  $\lambda_{\delta_{t}} \in \Lambda \subset \mathcal{C}$  and  $\lambda_{t} \in \mathcal{C}$ , and because  $\mathcal{C}$  is convex, we have that  $\lambda_{t+1} \in \mathcal{C}$ .
Case 3. If  $\alpha_{t} \leq 1$  and  $\lambda_{t} \notin \mathcal{C}$ , we define  $\pi_{t} = \lambda_{t}$  the projection of  $\lambda_{t}$  on  $\mathcal{C}$ . Let  $y_{t} = (1 - \alpha_{t})\pi_{t} + \alpha_{t}\lambda_{\delta_{t}}$ . Because  $\mathcal{C}$  is convex,  $\alpha_{t} \in [0,1]$ ,  $\lambda_{\delta_{t}} \in \mathcal{C}$ , and  $\pi_{t} \in \mathcal{C}$ , we have  $y_{t} \in \mathcal{C}$ . This implies that
$$
\begin{array}{l} \operatorname {D i s t} \left(\lambda_ {t + 1}, \mathcal {C}\right) \leq \left\| \lambda_ {t + 1} - y _ {t} \right\| _ {2} \\ = \left(1 - \alpha_ {t}\right) \| \lambda_ {t} - \pi_ {t} \| _ {2} \\ = \left(1 - \alpha_ {t}\right) \operatorname {D i s t} \left(\lambda_ {t}, \mathcal {C}\right) \\ \leq \operatorname {D i s t} (\lambda_ {t}, \mathcal {C}). \\ \end{array}
$$
This shows that for all cases,  $\mathrm{Dist}(\lambda_{t + 1},\mathcal{C})\leq \mathrm{Dist}(\lambda_t,\mathcal{C})$  , which concludes the proof.
Using this previous Lemma, we can simply bound the virtual reward functions.
Lemma D.4. For  $\lambda_0\in \partial R(0)$ , and for  $\lambda_{t}$  generated according to Algorithm 1, the virtual reward function
$$
\varphi (\lambda_ {t}, c _ {t k}, k) = \max  _ {x \in \mathcal {X}} \mathbb {E} [ f _ {t} (x) - \langle \lambda_ {t}, a _ {t} (x) \rangle | c _ {t k} ] - p _ {k},
$$
is bounded in absolute value by
$$
m := \bar {f} + L + 2 \eta \operatorname {D i a m} (\Delta) + \max  _ {k} | p _ {k} |.
$$
Proof. Because  $a_{t}(x)$  used for the update of  $\lambda_t$  is in  $\Delta$  for all  $t$ , we can apply Lemma 3.2 to bound the trajectory of the  $(\lambda_t)_{t \in [T]}$  generated by the algorithm. Using a triangle inequality first, then Cauchy-Schwartz inequality, and finally the boundedness assumptions on  $f_{t}$  and  $a_{t}$  with the Lemma, we have
$$
\begin{array}{l} \left| \max  _ {x \in \mathcal {X}} \mathbb {E} [ f _ {t} (x) - \langle \lambda_ {t}, a _ {t} (x) \rangle \mid c _ {t k} ] - p _ {k} \right| \leq \left| \max  _ {x \in \mathcal {X}} \mathbb {E} [ f _ {t} (x) \mid c _ {t k} ] \right| + \left| \max  _ {x \in \mathcal {X}} \mathbb {E} [ \langle \lambda_ {t}, a _ {t} (x) \rangle \mid c _ {t k} ] \right| + | p _ {k} | \\ \leq \left| \max  _ {x \in \mathcal {X}} \mathbb {E} \left[ f _ {t} (x) \mid c _ {t k} \right] \right| + \left| \max  _ {x \in \mathcal {X}} \mathbb {E} [ \| \lambda_ {t} \| _ {2} \| a _ {t} (x) \| _ {2} \mid c _ {t k} ] \right| + | p _ {k} | \\ \leq \bar {f} + L + 2 \eta \operatorname {D i a m} (\Delta) + \max  _ {k} | p _ {k} |. \\ \end{array}
$$
Lemma D.5. For  $\lambda_0\in \partial R(a)$  (for some  $a\in \mathcal{A}$ ), and for  $\lambda_{t}$  generated according to Algorithm 1, if the quantity  $a_{t}(x) / x$  is always well defined and in  $\Delta$  for all  $a_{t}$  and all  $x$ , then the virtual reward function
$$
\tilde {\varphi} \left(\lambda_ {t}, c _ {t k}, k\right) = \max  _ {x \in \mathcal {X}} \left(\mathbb {E} \left[ f _ {t} (x) - \left\langle \lambda_ {t}, a _ {t} (x) \right\rangle \mid c _ {t k} \right] + R ^ {*} \left(\lambda_ {t}\right)\right) - p _ {k},
$$
$$
\tilde {m} := \bar {f} + L + 2 \eta \operatorname {D i a m} (\Delta) + \max  \left\{R ^ {*} (\lambda) \mid \| \lambda \| _ {2} \leq L + 2 \eta \operatorname {D i a m} (\Delta) \right\} + \max  _ {k} | p _ {k} |.
$$
Proof. The proof of this Lemma is identical to the previous one, except that we also need to bound  $R^{*}$ . As a convex conjugate,  $R^{*}$  is convex. By Corollary 10.1.1 of Rockafellar (1970), every convex function from  $\mathbb{R}^d$  to  $\mathbb{R}$  is continuous. By Lemma 1 of Celli et al. (2022), we know that the domain of  $R^{*}$  is indeed  $\mathbb{R}^d$ , hence it is continuous.
Now let us look at the modified dual update  $\lambda_{t + 1} = \lambda_t + \eta (\gamma_tx_t - a_t(x_t))$ . We can factorize by  $x_{t}$  to recover  $\lambda_{t + 1} = \lambda_t + x_t\eta (\gamma_t - a_t(x_t) / x_t)$ . Because of our hypothesis,  $a_{t}(x_{t}) / x_{t}\in \Delta$ , therefore we can apply Lemma 3.2 with  $\eta_t = \eta x_t$ , which is upper bounded by  $\eta$ . Hence the  $\lambda_{t}$  are all in  $\{\lambda \in \mathbb{R}^d\mid \| \lambda \| _2\leq L + 2\eta \mathrm{Diam}(\Delta)\}$  which is compact. By continuity of  $R^{*}$ , it is continuous over this compact set hence bounded, and the virtual reward is bounded by  $\tilde{m}$  overall.
Remark. We made the technical assumption that  $a_{t}(x) / x \in \Delta$ . It is clear that when  $a_{t}$  is linear, such as  $a_{t}x$ , then  $a_{t}x / x = a_{t} \in \mathcal{A} \subset \Delta$  and the condition is satisfied. Regardless, if  $a_{t}(x) / x$  is always well defined and bounded, we can simply increase the size of  $\Delta$ , so that this quantity always remain in  $\Delta$ . Of course, this will degrade the performance of the algorithm as it depends on  $\mathrm{Diam}(\Delta)$ .
D.4 Expected difference between penalty and penalty of expectation — Proof of Lemma 3.3
We next prove the following lemma, which tells us that observing the conditional expectation of the  $a_{t}(x_{t})$  is enough to observe the expectation of the penalty  $R$ :
Lemma. For  $(x_{1},\ldots ,x_{T})$  and  $(k_{1},\ldots ,k_{T})$  generated according to Algorithm 1, we have the following upper bound:
$$
| \mathbb {E} [ R (\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} x _ {t}) - R (\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {a _ {t}} [ a _ {t} (x _ {t}) | c _ {t k _ {t}} ]) ] | \leq 2 L \sqrt {\frac {d}{T}}.
$$
Proof. Let us denote by  $\delta_t = \mathbb{E}_{a_t}[a_t(x_t) \mid c_{t k_t}]$  the expectation of  $a_t(x_t)$  with  $x_t$  fixed. Using first the triangle inequality for the expectation, and then the Lipschitzness of  $R$  we have that
$$
\begin{array}{l} | \mathbb {E} [ R (\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} (x _ {t})) - R (\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {a _ {t}} [ a _ {t} (x _ {t}) \mid c _ {t k _ {t}} ]) ] | \leq \mathbb {E} [ | R (\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} (x _ {t})) - R (\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}) | ] \\ \leq \frac {L}{T} \mathbb {E} [ \| \sum_ {t = 1} ^ {T} (a _ {t} (x _ {t}) - \delta_ {t}) \| _ {2} ] \\ \leq \frac {L}{T} \sqrt {\mathbb {E} [ \| \sum_ {t = 1} ^ {T} \left(a _ {t} \left(x _ {t}\right) - \delta_ {t}\right) \| _ {2} ^ {2} ]}, \tag {17} \\ \end{array}
$$
where we used Jensen's inequality with  $x \mapsto \sqrt{x}$  for the last inequality.
We denote by  $\delta_{t,l}$  the  $l$ -th coordinate of  $\delta_t$  for  $l\in [d]$ , and similarly for  $a_{t,l}$ . We have:
$$
\mathbb {E} [ \| \sum_ {t = 1} ^ {T} (a _ {t} (x _ {t}) - \delta_ {t}) \| _ {2} ^ {2} ] = \sum_ {l = 1} ^ {d} \mathbb {E} [ \left(\sum_ {t = 1} ^ {T} (a _ {t, l} (x _ {t}) - \delta_ {t, l})\right) ^ {2} ]
$$
By Equation (16), the sequence  $a_{t,l}(x_t) - \delta_{t,l}$  is a Martingale difference sequence for the filtration  $\sigma((a_\tau, c_{\tau k_\tau}, k_\tau)_{\tau \in [t-1]}, k_t, c_{tk_t})$ . Hence, using that the expectation is 0 by the law of total expectation, we obtain that
$$
\mathbb {E} \left[ \left(\sum_ {t = 1} ^ {T} \left(a _ {t, l} \left(x _ {t}\right) - \delta_ {t, l}\right)\right) ^ {2} \right] = \operatorname {V a r} \left[ \sum_ {t = 1} ^ {T} \left(a _ {t, l} \left(x _ {t}\right) - \delta_ {t, l}\right) \right] = \sum_ {t = 1} ^ {T} \operatorname {V a r} \left[ a _ {t, l} \left(x _ {t}\right) - \delta_ {t, l} \right]. \tag {18}
$$
The terms  $a_{t,l}(x_t) - \delta_{t,l}$  are bounded in  $[-2, 2]$  by the assumption that  $\| a_t(x) \|_2 \leq 1$ , therefore we can bound each of these variances by 4, for a total bound of  $4T$ . Plugging this into Equation (17) concludes the proof.
E Proof of the main Theorem 3.1
The proof builds upon the proof used in the full information case ( $a_{t}$  and  $f_{t}$  are given) from Balseiro et al. (2021); Celli et al. (2022).
In order to bound the regret, we upper bound the performance of OPT, and lower bound the performance of ALG. We proceed by first upper-bounding the optimum.
Recall that the convex conjugate  $R^*$  of  $R$  (which we consider as a function from  $\mathbb{R}^d \to \mathbb{R}$  that is  $+\infty$  outside of  $\Delta$ ) is a function that associates to a dual parameter  $\lambda$  the quantity  $R^*(\lambda)$ :
$$
R ^ {*} (\lambda) = \max  _ {\gamma \in \mathbb {R} ^ {d}} \{\langle \gamma , \lambda \rangle - R (\gamma) \} = \max  _ {\gamma \in \Delta} \{\langle \gamma , \lambda \rangle - R (\gamma) \}.
$$
The Fenchel-Moreau theorem, tells us that for a proper convex function, the biconjugate is equal to the original function:  $R^{**} = R$ .
Recall that  $\varphi$  is defined in Equation (5). We define the dual vector function  $\mathcal{D}(\lambda) = (\mathcal{D}(\lambda, 1), \dots, \mathcal{D}(\lambda, K))$  which is a vector representing the value of the dual conjugate problem. For coordinate  $k \in [K]$ , it is defined as
$$
\mathcal {D} (\lambda , k) = \mathbb {E} [ \varphi (\lambda , c _ {t k}, k) ] + R ^ {*} (\lambda).
$$
We denote the unit simplex of dimension  $K$  by  $\mathcal{P}_K = \{\pi \in \mathbb{R}^K\mid \pi_k\geq 0,\sum_{k = 1}^{K}\pi_k = 1\}$ .
Lemma E.1. We have the following upper-bound for the offline optimum:
$$
\mathrm {O P T} \leq T \sup  _ {\pi \in \mathcal {P} _ {K}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \langle \pi , \mathcal {D} (\lambda) \rangle ,
$$
Proof. We upper bound the performance of OPT through a Lagrangian relaxation using the convex conjugate of  $R$ , as was done similarly in Jenatton et al. (2016).
Let  $(k_{1},\ldots ,k_{T})\in [K]^{T}$  be the (deterministic) sequence of sources selected, and  $(x_{1},\dots,x_{T})\in \mathcal{X}^{T}$  the offline allocation. These  $x_{t}$  are all  $\sigma (c_{1k_1},\ldots ,c_{Tk_T})$  measurable. We define  $\tilde{f}_t(x) = \mathbb{E}_{f_t}[f_t(x)\mid c_{tk_t}]$  and  $\delta_t = \mathbb{E}_{a_t}[a_t(x_t)\mid c_{tk_t}]$ .
Using Jensen's inequality for  $R$  convex we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] - T \mathbb {E} \left[ R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right)\right) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] - T R \left(\mathbb {E} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right]\right). \\ \end{array}
$$
Because the  $x_{t}$  are  $\sigma(c_{1k1}, \ldots, c_{Tk_T})$  measurable, and by independence of the  $(f_t, a_t, c_{t1}, \ldots, c_{tK})$  we have that  $\mathbb{E}[a_t(x_t) \mid \sigma(c_{1k_1}, \ldots, c_{tk_T})] = \mathbb{E}_{a_t}[a_t(x_t) \mid c_{tk_t}] = \delta_t$ . With the same argument we obtain that  $\mathbb{E}[f_t(x_t) \mid \sigma(c_{1k_1}, \ldots, c_{tk_t})] = \tilde{f}_t(x_t)$ . Therefore
$$
\mathbb {E} \left[ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \leq \sum_ {t = 1} ^ {T} \tilde {f} _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right). \tag {19}
$$
We define the function  $\mathcal{L}:(\pmb {k},\mathbf{c},\mathbf{x},\lambda)\mapsto \sum_{t = 1}^{T}\tilde{f}_t(x_t) - p_{k_t} - \langle \lambda ,\delta_t\rangle +TR^* (\lambda).$
We now show that this function is always greater than the conditional expectation of  $\mathcal{U}$ :
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {k}, \mathbf {c}, \mathbf {x}, \lambda) \geq \inf  _ {\lambda \in \mathbb {R} ^ {d}} \mathcal {L} (\boldsymbol {k}, \mathbf {c}, \mathbf {x}, \lambda) \\ = \sum_ {t = 1} ^ {T} \tilde {f} _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - T \sup  _ {\lambda \in \mathbb {R} ^ {d}} \left\{\langle \lambda , \frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t} \rangle - R ^ {*} (\lambda) \right\} \\ = \sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x _ {t}) - p _ {k _ {t}} - T R ^ {* *} \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) \\ = \sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x _ {t}) - p _ {k _ {t}} - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) \\ \geq \mathbb {E} [ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid \mathbf {c} _ {\boldsymbol {k}} ], \tag {20} \\ \end{array}
$$
the last equality results from applying Fenchel-Moreau theorem, and the last inequality is obtained by using Equation (19).
Let us now compute the max in  $(x_{1},\ldots ,x_{T})$  of  $\mathcal{L}$ . Looking at the terms inside the sum, the only possible dependency of  $\tilde{f}_t(x_t) - p_{k_t} - \langle \lambda ,\delta_t\rangle$  to other contexts for  $t^\prime \neq t$ , is possible through the allocation  $x_{t}$ , which can depend on the other contexts. Therefore the maximum of the sum is the sum of the maximums. This yields that
$$
\begin{array}{l} \max  _ {\left(x _ {1}, \dots , x _ {T}\right)} \mathcal {L} (\boldsymbol {k}, \mathbf {c}, \mathbf {x}, \lambda) = \sum_ {t = 1} ^ {T} \max  _ {x _ {t} \in \mathcal {X}} \mathbb {E} _ {f _ {t}, a _ {t}} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \left\langle \lambda , a _ {t} \left(x _ {t}\right) \right\rangle \mid c _ {t k _ {t}} \right] + T R ^ {*} (\lambda) \\ = \sum_ {t = 1} ^ {T} \left(\varphi \left(\lambda , c _ {t k _ {t}}, k _ {t}\right) + R ^ {*} (\lambda)\right). \\ \end{array}
$$
Thus by taking the max in  $(x_{1},\ldots ,x_{T})$  on both sides of Equation (20) and then taking the expectation (which is over the contexts as everything is measurable with respect to them), we obtain
$$
\mathbb {E} \left[ \max  _ {\left(x _ {1}, \dots , x _ {T}\right)} \mathbb {E} \left[ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \right] \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \varphi (\lambda , c _ {t k _ {t}}, k _ {t}) + R ^ {*} (\lambda) \right] = \sum_ {t = 1} ^ {T} \mathcal {D} (\lambda , k _ {t}). \tag {21}
$$
What is nice here, is that by looking at the dual problem which was originally to address the non-separability in  $x$ , we also have the separability in  $k_{t}$ .
We denote by  $T_{k} = \sum_{t=1}^{T} \mathbb{1}[k_{t} = k]$  the number of times source  $k$  is selected, and by  $\hat{\pi} = (T_{1}/T, \dots, T_{K}/T) \in \mathcal{P}_{K}$  the selection proportions of each source  $k$  over the  $T$  rounds. This vector of proportions can be seen as the empirical distribution of selected sources. We can rewrite in the following way the dual objective
$$
\sum_ {t = 1} ^ {T} \mathcal {D} (\lambda , k _ {t}) = \sum_ {k = 1} ^ {K} T _ {k} \mathcal {D} (\lambda , k) = T \sum_ {k = 1} ^ {K} \hat {\pi} _ {k} \mathcal {D} (\lambda , k) = T \langle \hat {\pi}, \mathcal {D} (\lambda) \rangle .
$$
Finally, using the previous equation with Equation (21), taking first the inf in  $\lambda$  and then the max in  $k_{t}$  we can conclude:
$$
\mathrm {O P T} \leq T \max  _ {k _ {1}, \dots , k _ {T}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \langle \hat {\pi}, \mathcal {D} (\lambda) \rangle \leq T \sup  _ {\pi \in \mathcal {P} _ {K}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \langle \pi , \mathcal {D} (\lambda) \rangle .
$$
Remark. Note that the function  $\mathcal{D}(\lambda)$  is convex and lower semi-continuous (l.s.c.) (as the supremum of a family of l.s.c. functions). In addition,  $\langle \pi ,\mathcal{D}(\lambda)\rangle$  is concave and u.s.c. in  $\pi$  by linearity, and  $\mathcal{P}_K$  is compact convex. Hence Sion's minimax theorem can be applied, and the sup in  $\pi$  is actually a max.
We can now proceed to the rest of the proof of the main Theorem 3.1:
Theorem E.2. For  $\eta = L / (2\mathrm{Diam}(\Delta)\sqrt{T})$ ,  $m = \bar{f} +L + \max_k|p_k| + 2\eta \mathrm{Diam}(\Delta)$ ,  $\rho = \sqrt{\log(K) / (TKm^2)}$ , and  $\lambda_0\in \partial R(0)$ , algorithm Algorithm 1 has the following regret upper bound:
$$
\operatorname {R e g} \leq 2 \left(\left(L + \bar {f} + \max _ {k} | p _ {k} |\right) \sqrt {K \log (K)} + L \sqrt {d} + \operatorname {D i a m} (\Delta)\right) \sqrt {T} + 2 L \sqrt {K \log (K)}
$$
Proof. We first lower bound the performance of the algorithm. Then we simply need to apply Lemma E.1 to get the regret bound. Let  $\mathcal{H}_t$  be the natural filtration generated by the  $(c_{\tau k_{\tau}},k_{\tau})_{\tau \in [t]}$ . Let  $x_{t},k_{t},\lambda_{t}$  and  $\gamma_{t}$  defined as in 1. We denote by  $\delta_t = \mathbb{E}_{a_t}[a_t(x_t)\mid c_{tk_t}]$ . The quantity  $\delta_t$  is in  $\Delta$  because of the expectation and by convexity of  $\Delta$ . Note that here these are all random variables, with  $\gamma_{t}$  and  $\lambda_{t}$  being  $\mathcal{H}_{t - 1}$  measurable and  $x_{t}$  being  $\mathcal{H}_t$  measurable.
To first get a broad picture on how we are going to lower bound the utility of the algorithm, let us decompose it into different parts. It can be rewritten as