SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The last term  $\varphi$  is the one we want, we now need to lower bound the first two terms.
By Theorem 20.5 of Lattimore & Szepesvári (2020), we know with probability at least  $1 - \delta$  that  $\psi_{k}\in \mathcal{I}_{tk}$  for all  $t$ . By union bound, the probability that the  $\psi_{k}$  are simultaneously all in their confidence set is at least  $1 - K\delta$ . We denote this event by  $I$ .
Suppose that the event  $I$  is satisfied. Then by definition of  $\tilde{\psi}_{t_{k_t}}$ , we have
$$
x _ {t} ^ {*} \langle \tilde {\psi} _ {t k _ {t}} - \psi_ {k _ {t}}, c _ {t k _ {t}} \rangle \geq 0.
$$
Now let us look at the last term. We can first decompose it over the  $K$  sources:
$$
x _ {t} \left\langle \psi_ {k _ {t}} - \tilde {\psi} _ {t k _ {t}}, c _ {t k _ {t}} \right\rangle = \sum_ {k = 1} ^ {K} \theta_ {t k} x _ {t} \left\langle \psi_ {k} - \tilde {\psi} _ {t k}, c _ {t k} \right\rangle = \sum_ {k = 1} ^ {K} \left\langle \psi_ {k} - \tilde {\psi} _ {t k}, \theta_ {t k} x _ {t} c _ {t k} \right\rangle .
$$
Now, consider one source  $k$ , and sum the difference term for every  $t$ . By remarking that  $\theta_{tk} = \theta_{tk}^2$  we have that
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left| \left\langle \psi_ {k} - \tilde {\psi} _ {t k} \mid \theta_ {t k} x _ {t} c _ {t k} \right\rangle \right| = \sum_ {t = 1} ^ {T} \theta_ {t k} \left| \left\langle \psi_ {k} - \tilde {\psi} _ {t k} \mid \theta_ {t k} x _ {t} c _ {t k} \right\rangle \right| \\ \leq \sum_ {t = 1} ^ {T} \theta_ {t k} \| \psi_ {k} - \tilde {\psi} _ {t k} \| _ {V _ {t - 1, k}} \| \theta_ {t k} x _ {t} c _ {t k} \| _ {V _ {t - 1, k} ^ {- 1}} \quad (\text {d u a l - n o r m i n e q u a l i t y}) \\ \leq 2 \sum_ {t = 1} ^ {T} \theta_ {t k} \sqrt {\beta_ {t}} \| \theta_ {t k} x _ {t} c _ {t k} \| _ {V _ {t - 1, k} ^ {- 1}} \quad (\text {b y}) \\ \leq 2 \sum_ {t = 1} ^ {T} \theta_ {t k} \sqrt {\beta_ {T}} \max  \left\{1, \left\| \theta_ {t k} x _ {t} c _ {t k} \right\| _ {V _ {t - 1, k} ^ {- 1}} \right\} \quad (\beta_ {t} \text {a r e i n c r e a s i n g}) \\ \leq 2 \sqrt {\beta_ {T}} \sqrt {\sum_ {t = 1} ^ {T} \theta_ {t k} ^ {2}} \sqrt {\sum_ {t = 1} ^ {T} \max  \left\{1 , \left\| \theta_ {t k} x _ {t} c _ {t k} \right\| _ {V _ {t - 1 , k} ^ {- 1}} ^ {2} \right\}} (\mathrm {C . S . i n e q u a l i t y}) \\ = 2 \sqrt {\beta_ {T}} \sqrt {\sum_ {t = 1} ^ {T} \theta_ {t k}} \sqrt {\sum_ {t = 1} ^ {T} \max  \left\{1 , \| \theta_ {t k} x _ {t} c _ {t k} \| _ {V _ {t - 1 , k} ^ {- 1}} ^ {2} \right\}} \\ = 2 \sqrt {\beta_ {T}} \sqrt {T _ {k}} \sqrt {\sum_ {t = 1} ^ {T} \max  \left\{1 , \| \theta_ {t k} x _ {t} c _ {t k} \| _ {V _ {t - 1 , k} ^ {- 1}} ^ {2} \right\}}, \\ \end{array}
$$
where  $T_{k}$  is the number of times source  $k$  is selected.
By Lemma 19.4 of Lattimore & Szepesvári (2020), we have that
$$
\sum_ {t = 1} ^ {T} \max  \left\{1, \| \theta_ {t k} x _ {t} c _ {t k} \| _ {V _ {t - 1, k} ^ {- 1}} ^ {2} \right\} \leq 2 q _ {k} \log \left(1 + \frac {T \bar {c} ^ {2}}{q _ {k}}\right).
$$
Therefore, using Jensen inequality for the square root function which is concave over the  $T_{k}$ , and because  $\sum_{k} T_{k} = T$ , we obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {k = 1} ^ {K} | \langle \psi_ {k} - \tilde {\psi} _ {t k} | \theta_ {t k} x _ {t} c _ {t k} \rangle | \leq \sum_ {k = 1} ^ {K} 2 \sqrt {\beta_ {T}} \sqrt {T _ {k}} \sqrt {2 q _ {k} \log \left(1 + \frac {T \bar {c} ^ {2}}{q _ {k}}\right)} \\ \leq 2 \sqrt {\beta_ {T}} \sum_ {k = 1} ^ {K} \sqrt {T _ {k}} \sqrt {2 q _ {k} \log \left(1 + \frac {T \bar {c} ^ {2}}{q _ {k}}\right)} \\ \leq \sqrt {8 T \beta_ {T}} \sqrt {\sum_ {k = 1} ^ {K} q _ {k} \log \left(1 + \frac {T \bar {c} ^ {2}}{q _ {k}}\right)}. \\ \end{array}
$$
This yields a  $\mathcal{O}(\sqrt{KT}\log (T))$  error.
And with probability at most  $K\delta$  the concentration event  $I$  does not hold, but the error is bounded by 4 for each  $t$ . Thus overall in expectation, it is bounded by  $4TK\delta$ . Picking  $\delta = 1 / T$ , yields the desired sub-linear regret bound.
I Learning  $a_{t}$
In this section, we provide further intuitions on the existence of  $[K]$  different sources, and how it is possible to adapt the algorithm to learn  $\mathbb{E}[a_t\mid c_{tk}]$  on the fly. Let us consider the special case of our model, where the additional information given by each source is a noisy observation  $\hat{a}_{tk}$  of  $a_{t}$ , with different noise levels; in the main setting this can be described as  $c_{tk} = (z_t,\hat{a}_{tk})$ . Sources with lower noise levels will have higher prices (one directly pays for the level of accuracy). This setting can be used to give users the level of privacy they are willing to give up (as in Local Differential Privacy (Dwork & Roth, 2014)), by revealing some noisy information about their private data; of course, the more data is revealed (or the smaller the noise), the higher the compensation (the price).
Furthermore, if the source  $k$  corresponds to adding additive independent noise  $L_{tk}$  and to returning  $\hat{a}_{tk} = a_t + L_{tk}$ , then knowing the law of both  $L_{tk}$  and  $a_t$ , the decision maker can directly compute the conditional expectation of  $a_t$  having observed  $\hat{a}_{tk}$ . Nevertheless, if she only knows the noise distribution, but not the law of  $a_t$ , conditional expectation can still be learned in an online fashion in some cases.
Indeed, suppose for simplicity that  $a_{t} \in \{-1, 1\}$ ,  $\operatorname{Pr}(a_{t} = 1) = \alpha \in (0, 1)$ ,  $(z_{t}, u_{t})$  is independent of  $a_{t}$  (therefore we can focus only on dealing with  $a_{t}$ ), and that the  $L_{tk}$  are 0 mean  $\sigma_{k}$  sub-exponential random variables (see 2.7.5 Vershynin (2018)). We define the following running empirical estimate of  $\alpha$ :
$$
\hat {\alpha} _ {t} = \frac {1}{t} \sum_ {\tau = 1} ^ {t} \left(\hat {a} _ {\tau , k _ {\tau}} + 1\right) / 2.
$$
It is an unbiased estimator of  $\alpha$  as
$$
\mathbb {E} [ \hat {a} _ {t k} ] = \mathbb {E} [ a _ {t} ] + \mathbb {E} [ L _ {t k} ] = 2 \alpha - 1 + 0 = 2 \alpha - 1.
$$
Note that the  $\hat{a}_{t,\kappa_t}$  are not independent, and the conditional expectation is not Lipschitz-continuous with respect to the parameter  $\alpha$ . Still, we can derive an algorithm with a sub-linear regret bound.
First let us state a Lemma on the concentration of  $\hat{\alpha}_t$
Lemma I.1. We have the following concentration bound for  $t \gtrsim \log (T)$ :
$$
\Pr \left(| \hat {\alpha} _ {t} - \alpha | \geq \sqrt {\frac {\kappa^ {2} \log (T)}{c t}}\right) \leq \frac {2}{T} \tag {41}
$$
Proof. We refer to Chapter 2 of Vershynin (2018) for concentration results of sum of independent random variables. Let us recall some definitions, properties, and a concentration inequality.
Let us consider the following Orlicz space norm for random variables. For a random variable  $X$ , we define
$$
\left\| X \right\| _ {\psi^ {1}} = \inf  \{t > 0, \mathbb {E} [ \exp (| X | / t) ] \leq 2 \}.
$$
If  $\| X\|_{\psi^1}$  is finite, then  $X$  is called sub-exponential. It is a norm, and for a bounded variable  $X$  in  $[-1,1]$ , we have  $\| X\|_{\psi^1}\leq 1 / \log (2)$  (see exercise 2.5.7, example 2.5.8, and Lemma 2.7.6 of Vershynin (2018)). Hence because  $(a_t + 1) / 2 - \alpha \in [-\alpha ,1 - \alpha ]\subset [-1,1]$ , for  $\hat{\alpha}_{tk}$  the estimate obtained by always choosing source  $k$ , we have by triangle inequality that
$$
\left\| \hat {\alpha} _ {t k} - \alpha \right\| _ {\psi^ {1}} \leq \frac {1}{\log (2)} + \frac {\sigma_ {k}}{2}.
$$
We now recall Bernstein's inequality:
Theorem (Theorem 2.8.1 of Vershynin (2018)). Let  $X_{1},\ldots ,X_{N}$  be  $N$  independent, mean 0, subexponential random variables, with  $\kappa = \max_{i}\| X_{i}\|_{\psi^{1}}$ . Then for  $c > 0$  a constant, for every  $t > 0$  we have
$$
\Pr \left\{\left| \sum_ {i = 1} ^ {N} X _ {i} \right| \geq t \right\} \leq 2 \exp \left[ - c \min  \left(\frac {t ^ {2}}{N \kappa^ {2}}, \frac {t}{\kappa}\right) \right].
$$
We want to apply this Theorem to  $(\hat{a}_{t,k_t} + 1) / 2 - \alpha$  the centered  $\hat{\alpha}_t$  where we now change the source depending on  $k_{t}$ . Remark that now however the  $\hat{\alpha}_t$  is not a sum of independent variables anymore because of  $k_{t}$  which does depend on the previous realizations of the  $\hat{a}_{t,k_t}$ . Nevertheless it is a martingale difference for the filtration  $\mathcal{H}_t$ , and similar results will apply. We denote by  $\kappa = 1 / \log (2) + \max_k\sigma_k / 2$  an upper bound on the sub-exponential norm of the centered  $\hat{\alpha}_t$  conditional on  $\mathcal{H}_{t - 1}$ . Then the martingale  $\sum_{\tau = 1}^{t}\hat{\alpha}_{\tau} - \alpha$  is also a sub-exponential random variable.
This is a known result, which is stated here for completeness, but that can otherwise be skipped. Indeed from proposition 2.7.1 part  $e$  ) of Vershynin (2018), we know that finite sub-exponential norm for a centered random variable  $X$  means that the moment generating function of  $X$  at  $y$  is bounded by  $\exp (c_1\| X\|_{\psi^1}^2 y^2)$  for  $y\leq c_2 / \| X\|_{\psi^1}$ , with  $c_{1}$  and  $c_{2}$  some positive constants.
Therefore, for  $y \leq c_2 / \kappa$ , using the property of the conditional sub-exponentiality of  $\hat{\alpha}_t$ , we have
$$
\begin{array}{l} \mathbb {E} [ \exp (y \sum_ {\tau = 1} ^ {t} (\hat {\alpha} _ {\tau} - \alpha)) ] = \mathbb {E} [ \mathbb {E} [ \exp (y \sum_ {\tau = 1} ^ {t} (\hat {\alpha} _ {\tau} - \alpha)) \mid \mathcal {F} _ {t - 1} ] ] \\ = \mathbb {E} [ \exp (y \sum_ {\tau = 1} ^ {t - 1} (\hat {\alpha} _ {\tau} - \alpha)) \mathbb {E} [ \exp (y (\hat {\alpha} _ {t} - \alpha)) \mid \mathcal {F} _ {t - 1} ] ] \\ \leq \mathbb {E} [ \exp (y \sum_ {\tau = 1} ^ {t - 1} (\hat {\alpha} _ {\tau} - \alpha)) \mathbb {E} [ \exp (c _ {1} y ^ {2} \kappa^ {2}) \mid \mathcal {F} _ {t - 1} ] ]. \\ \end{array}
$$
Iterating this inequality we obtain for  $y \leq c_2 / \kappa$  that
$$
\mathbb {E} [ \exp (y \sum_ {\tau = 1} ^ {t} (\hat {\alpha} _ {\tau} - \alpha)) ] \leq \mathbb {E} [ \exp (c _ {1} t \kappa^ {2} y ^ {2}) ].
$$
This yields inequality (2.24) in Vershynin (2018) the main element needed for the proof of Bernstein's Inequality. Thus Bernstein's inequality still holds for centered martingale sequence which are conditionally sub-exponential. We stress that this is not a new concentration inequality result.
Let us now apply Bernstein's Inequality with a deviation of order  $\sqrt{\kappa^2\log(T) / (ct)}$ , for  $t\geq \log (T) / c$ :
$$
\begin{array}{l} \Pr \left(| \hat {\alpha} _ {t} - \alpha | \geq \sqrt {\frac {\kappa^ {2} \log (T)}{c t}}\right) = \Pr \left(\left| \sum_ {\tau = 1} ^ {t} (\hat {\alpha} _ {\tau} - \alpha) \right| \geq \sqrt {\frac {\kappa^ {2} t \log (T)}{c}}\right) \\ \leq 2 \exp \left[ - c \min  \left(\frac {\log (T)}{c}, \sqrt {\frac {\log (T) t}{c}}\right) \right] \\ = \frac {2}{T} \\ \end{array}
$$
Proposition I.2. Given these assumptions, using Algorithm 1 with  $\hat{\alpha}_t$  instead of  $\alpha$  to compute the conditional expectations yields an added regret of order  $\mathcal{O}((\max_k \sigma_k + 2 / \log(2)) \sqrt{T \log(T)})$ .
Proof. There are two parts of the proof of Theorem 3.1 that we need to modify, when the  $x_{t}$  is involved, and when we compare the penalty. We will deal with the first part (the more complicated), and the second one follows immediately by the same arguments. This will be done in 3 steps: we first highlight where the learning error occurs, then use the concentration result to decompose the error under a good and bad event, and finally show that under the good event the error in expectation is small enough.
Impact of learning error We suppose for simplicity that  $L_{tk}$  has a density  $g_{k}$  over the support  $\mathbb{R}$ , but this also holds for discrete random variables. We denote by  $\nu_{k}$  the density of  $\hat{a}_{tk}$ . First let us express  $\tilde{a}_t = \mathbb{E}[a_t\mid c_{tk_t}]$  as a function of  $\hat{a}_{t,k_t} = \hat{a}$  and  $\alpha$ :
$$
\begin{array}{l} \mathbb {E} \left[ a _ {t} \mid \hat {a} _ {t, k _ {t}} = \hat {a} \right] = \Pr \left(a _ {t} = 1 \mid \hat {a} _ {t, k _ {t}} = \hat {a}\right) - \Pr \left(a _ {t} = - 1 \mid \hat {a} _ {t, k _ {t}} = \hat {a}\right) \\ = \frac {\Pr (a _ {t} = 1) g _ {k _ {t}} (\hat {a} - 1)}{\nu_ {k _ {t}} (\hat {a})} - \frac {\Pr (a _ {t} = - 1) g _ {k _ {t}} (\hat {a} + 1)}{\nu_ {k _ {t}} (\hat {a})} \\ = \frac {\alpha g _ {k _ {t}} (\hat {a} - 1) - (1 - \alpha) g _ {k _ {t}} (\hat {a} + 1)}{\alpha g _ {k _ {t}} (\hat {a} - 1) + (1 - \alpha) g _ {k _ {t}} (\hat {a} + 1)}. \\ \end{array}
$$
Let us define the plug-in estimate of  $\tilde{a}_t$  using  $\bar{\alpha}_{t - 1}$  whenever  $\bar{\alpha}_{t - 1}$  is in [0, 1] by:
$$
s _ {t} (\hat {a}) = \frac {\bar {\alpha} _ {t - 1} g _ {k _ {t}} (\hat {a} - 1) - (1 - \bar {\alpha} _ {t - 1}) g _ {k _ {t}} (\hat {a} + 1)}{\bar {\alpha} _ {t - 1} g _ {k _ {t}} (\hat {a} - 1) + (1 - \bar {\alpha} _ {t - 1}) g _ {k _ {t}} (\hat {a} + 1)}. \tag {42}
$$
When the estimated parameter is not in [0, 1], we use  $s_t = 0$ .
We define by  $x_{t}$  the maximisation of the virtual value function  $\varphi$  when replacing  $\tilde{a}_{t}$  by  $s_t$ , and by  $x_{t}^{*}$  the allocation obtained if we were to actually use  $\tilde{a}_t$ . Because  $x_{t}$  is a maximizer of  $u_{t}x_{t} - \langle \lambda_{t},s_{t}\rangle x_{t}$  we have
$$
\begin{array}{l} u _ {t} x _ {t} = u _ {t} x _ {t} - \langle \lambda_ {t}, s _ {t} \rangle x _ {t} + \langle \lambda_ {t}, s _ {t} \rangle x _ {t} \\ \geq u _ {t} x _ {t} ^ {*} - \left\langle \lambda_ {t}, s _ {t} \right\rangle x _ {t} ^ {*} + \left\langle \lambda_ {t}, s _ {t} \right\rangle x _ {t} \\ = \varphi \left(\lambda_ {t}, \hat {a} _ {t, k _ {t}}, k _ {t}\right) + \left\langle \lambda_ {t}, \tilde {a} _ {t} - s _ {t} \right\rangle x _ {t} ^ {*} + \left\langle \lambda_ {t}, s _ {t} \right\rangle x _ {t}. \\ \end{array}
$$
The term  $\langle \lambda_t, s_t \rangle x_t$  will be tracked with the help of the OGD, it remains to take care of  $\langle \lambda_t, \tilde{a}_t - s_t \rangle x_t^*$ . By Cauchy Schwartz:
$$
| \langle \lambda_ {t}, \tilde {a} _ {t} - s _ {t} \rangle x _ {t} ^ {*} | \leq \| \lambda_ {t} \| _ {2} \| \tilde {a} _ {t} - s _ {t} \| _ {2}.
$$
We know that  $\lambda_{t}$  is already bounded, so we only need to bound the right-hand term. Decomposing depending on the values of  $k_{t}$ , we can rewrite
$$
s _ {t} - \tilde {a} _ {t} = \sum_ {k = 1} ^ {K} \mathbb {1} [ k _ {t} = k ] \left(s _ {t} - \mathbb {E} \left[ a _ {t} \mid \hat {a} _ {t k} \right]\right)
$$
Decomposition under good and bad event Denote by  $\mathcal{C}_t = [|\hat{\alpha}_t - \alpha |\leq \sqrt{(\kappa^2\log(T) / (ct)]}$  the event of  $\hat{\alpha}_t$  concentrating around its mean. For  $t$  large enough, that is to say  $t\geq (\kappa^2\log (T)) / (c\min (\alpha ,1 - \alpha))$ , the event  $\mathcal{C}_t$  is included in  $[\bar{\alpha}_{t - 1}\in (0,1)]$ . We can then decompose the analysis over these events using the triangle inequality:  $|s_t - \tilde{a}_t|\leq \mathbb{1}_{\mathcal{C}_{t - 1}}|s_t - \tilde{a}_t| + \mathbb{1}_{\bar{\mathcal{C}}_{t - 1}}|s_t - \tilde{a}_t|$  and we know that for the term multiplied by  $\mathbb{1}_{\mathcal{C}_{t - 1}}$  we have that either it is 0 or  $\bar{\alpha}_{t - 1}\in [0,1]$ . Let us consider such a  $t - 1$  big enough, and first analyze the left-hand term with the "good" event.
Small error under good event in expectation We consider the conditional expectation of this sum given  $k_{t}$ , and  $\bar{\alpha}_{t-1}$ . By independence between  $(k_{t}, \bar{\alpha}_{t-1}, E)$  and  $\hat{a}_{tk}$  for all  $k$ , we have:
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {1} _ {\mathcal {C} _ {t - 1}} \left| s _ {t} - \tilde {a} _ {t} \right| \mid k _ {t}, \bar {\alpha} _ {t - 1} \right] \\ \leq \mathbb {1} _ {\mathcal {C} _ {t - 1}} \sum_ {k = 1} ^ {K} \mathbb {1} [ k _ {t} = k ] \int_ {\mathbb {R}} \left| \frac {\bar {\alpha} _ {t - 1} g _ {k} (\hat {a} - 1) - (1 - \bar {\alpha} _ {t - 1}) g _ {k} (\hat {a} + 1)}{\bar {\alpha} _ {t - 1} g _ {k} (\hat {a} - 1) + (1 - \bar {\alpha} _ {t - 1}) g _ {k} (\hat {a} + 1)} - \frac {\alpha g _ {k} (\hat {a} - 1) - (1 - \alpha) g _ {k} (\hat {a} + 1)}{\alpha g _ {k} (\hat {a} - 1) + (1 - \alpha) g _ {k} (\hat {a} + 1)} \right| \nu_ {k} (\hat {a}) d \hat {a}. \\ \end{array}
$$
Because  $\nu_{k_t}(\hat{a}) = (\alpha g_k(\hat{a} -1) + (1 - \alpha)g_k(\hat{a} +1))$  we can simplify the integral:
$$
\mathbb {E} [ \mathbb {1} _ {\mathcal {C} _ {t - 1}} | s _ {t} - \tilde {a} _ {t} | \mid k _ {t}, \bar {\alpha} _ {t - 1} ] \leq \mathbb {1} _ {\mathcal {C} _ {t - 1}} \sum_ {k = 1} ^ {K} \mathbb {1} [ k _ {t} = k ] \int_ {\mathbb {R}} 2 | \bar {\alpha} _ {t - 1} - \alpha | \frac {g _ {k} (\hat {a} - 1) g _ {k} (\hat {a} + 1)}{\bar {\alpha} _ {t - 1} g _ {k} (\hat {a} - 1) + (1 - \bar {\alpha} _ {t - 1}) g _ {k} (\hat {a} + 1)} d \hat {a}.
$$
For  $\alpha \in [0,1]$ , we have the following inequality:
$$
\frac {x y}{\alpha x + (1 - \alpha) y} \leq \max  (x, y) \leq x + y.
$$
Thus using this inequality and the fact that  $g_{k}$  is a density (which sums to 1), we obtain that
$$
\begin{array}{l} \mathbb {E} \left[ \mathbb {1} _ {\mathcal {C} _ {t - 1}} \mid s _ {t} - \tilde {a} _ {t} \right| | k _ {t}, \bar {\alpha} _ {t - 1} ] \leq 2 \mathbb {1} _ {\mathcal {C} _ {t - 1}} \left| \bar {\alpha} _ {t - 1} - \alpha \right| \sum_ {k = 1} ^ {K} \mathbb {1} [ k _ {t} = k ] \left(\int_ {\mathbb {R}} g _ {k} (\hat {a} + 1) d \hat {a} + \int_ {\mathbb {R}} g _ {k} (\hat {a} - 1) d \hat {a}\right) \\ \leq 4 \mathbb {1} _ {\mathcal {C} _ {t - 1}} | \bar {\alpha} _ {t - 1} - \alpha | \sum_ {k = 1} ^ {K} \mathbb {1} [ k _ {t} = k ] \\ \leq 4 \sqrt {\frac {\kappa^ {2} \log (T)}{c (t - 1)}} \sum_ {k = 1} ^ {K} \mathbb {1} [ k _ {t} = k ]. \\ \end{array}
$$
Finally taking the full expectation yields
$$
\mathbb {E} [ \mathbb {1} _ {\mathcal {C} _ {t - 1}} | s _ {t} - \tilde {a} _ {t} | ] \leq 4 \sqrt {\frac {\kappa^ {2} \log (T)}{c (t - 1)}}.
$$
For the complementary event  $\bar{\mathcal{C}}_{t - 1}$
$$
\mathbb {E} \left[ \mathbb {1} _ {\bar {\mathcal {C}} _ {t - 1}} \left| s _ {t} - \tilde {a} _ {t} \right| \right] \leq 2 \mathbb {E} \left[ \mathbb {1} _ {\bar {\mathcal {C}} _ {t - 1}} \right] = \frac {4}{T}.
$$
Putting everything back together Now denoting by  $m_{\lambda}$  the upper bound over the  $\lambda_t$  obtained by Lemma 3.2 and summing over all  $t$ :
$$
\begin{array}{l} | \mathbb {E} [ \sum_ {t = 1} ^ {T} \langle \lambda_ {t}, \tilde {a} _ {t} - s _ {t} \rangle x _ {t} ^ {*} ] | \leq m _ {\lambda} \sum_ {t = 1} ^ {T} \mathbb {E} [ | s _ {t} - \tilde {a} _ {t} | ] \\ \leq 2m_{\lambda}\frac{\kappa^{2}\log(T)}{c\min(\alpha,1 - \alpha)} +m_{\lambda}\sum_{\substack{t = \frac{\kappa^{2}\log(T)}{c\min(\alpha,1 - \alpha)} +1}}^{T}\mathbb{E}[|s_{t} - \tilde{a}_{t}|] \\ \leq 2m_{\lambda}\frac{\kappa^{2}\log(T)}{c\min(\alpha,1 - \alpha)} +m_{\lambda}\sum_{t = \frac{\kappa^{2}\log(T)}{c\min(\alpha,1 - \alpha)} +1}^{T}(4\sqrt{\frac{\kappa^{2}\log(T)}{c(t - 1)}} +\frac{4}{T}) \\ \leq 2 m _ {\lambda} \frac {\kappa^ {2} \log (T)}{c \min (\alpha , 1 - \alpha)} + m _ {\lambda} \sum_ {t = 1} ^ {T} \left(4 \sqrt {\frac {\kappa^ {2} \log (T)}{c t}} + \frac {4}{T}\right) \\ \leq m _ {\lambda} \left(\frac {8}{\sqrt {c}} \sqrt {\kappa^ {2} T \log (T)} + 2 \frac {\kappa^ {2} \log (T)}{c \min (\alpha , 1 - \alpha)} + 4\right) = \mathcal {O} \left(\sqrt {T \log (T)}\right), \\ \end{array}
$$
where we used a series-integral comparison of  $x \mapsto 1 / \sqrt{x}$  for the second to last inequality.
There is a second place where replacing  $\tilde{a}_t$  by  $s_t$  impacts the performance of the algorithm. Indeed, through the gradient descent on  $\lambda_t$ , we will have a penalty converging towards  $R(\sum_{t=1}^{T} s_t x_t / T)$  and not  $R(\sum_{t=1}^{T} \tilde{a}_t x_t / T)$ . This is because the Online Gradient Descent for the  $\lambda_t$  uses  $s_t$  and not  $\tilde{a}_t$ . Nevertheless, because of the Lipschitz property of  $R$ :
$$
\mathbb {E} [ T | R (\frac {\sum_ {t = 1} ^ {T} \tilde {a} _ {t} x _ {t}}{T}) - R (\frac {\sum_ {t = 1} ^ {T} s _ {t} x _ {t}}{T}) | ] \leq L \sum_ {t = 1} ^ {T} \mathbb {E} [ | \tilde {a} _ {t} - s _ {t} | ] \leq \mathcal {O} (\sqrt {T \log (T)}),
$$
where the last inequality is obtained by similar manipulations on  $\mathcal{C}_t$  to what we just did.
Overall, the added regret when using  $s_t$  instead of  $\tilde{a}_t$  is of order  $\sqrt{T\log(T)}$  which still guarantees a sub-linear regret.
Another algorithm is possible: first sample the source with the lowest sub-exponential constant  $T^{2/3} \log(T)$  times, and then use the estimate obtained to compute  $s_t$  (without updating it with the newest noise received). The advantage is that even though we incur a  $T^{2/3}$  regret (this can be shown using basically the same methods), we can obtain a lower  $\kappa$ . Hence if there is a big difference between the sub-exponential constant of the different sources, or even one source with infinite sub-exponential constant, this may be a viable algorithm.
J Public Contexts
For this section and in order to make the dependence in the public context  $z_{t}$  appear more clearly, we suppose that  $c_{tk}$  represent only the additional information of the source  $k$ , and thus  $(c_{tk_t},z_t)$  represent the whole information available at time  $t$  after selecting a source.
J.1 Finite number of Contexts - proof of Proposition 3.4
We propose the following Algorithm 4.
Proposition. For  $\mu$  the probability distribution over  $\mathcal{Z}$  finite, we can derive an algorithm that has a modified regret of order  $\mathcal{O}(\sqrt{TK\log(K)}\sum_{z\in \mathcal{Z}}\sqrt{\mu(z)})$ , where  $\mu (z)$  is the probability that the public attribute is  $z$ .
Proof. Most of the proof consists is identical to the proof of Theorem 3.1, and only the different parts will be highlighted, which is the upper bound of OPT, and applying bandits algorithms.
Algorithm 4 Online Fair Allocation with Source Selection — With public contexts
Input: Initial dual parameter  $\lambda_0$ , initial source-selection-distribution  $\pi_0(z) = (1 / K, \dots, 1 / K)$ , dual gradient descent step size  $\eta$ , EXP3 step size  $\rho_t(z)$  that uses a doubling trick, cumulative estimated rewards  $S_0(z) = 0 \in \mathbb{R}^K$  for all  $z \in \mathcal{Z}$ .
Observe  $z_{t}\in \mathcal{Z}$
Draw a source  $k_{t} \sim \pi_{t}(z_{t})$ , where  $\pi_{tk}(z_t) \propto \exp (\rho_t(z_t)S_{(t - 1)k}(z_t))$  and observe  $c_{tk_t}$ .
$$
x_{t} = \operatorname *{arg  max}_{x\in \mathcal{X}}\mathbb{E}[f_{t}(x) - \langle \lambda_{t},a_{t}(x)\rangle   |  c_{tk_{t}},z_{t}].
$$
$$
S _ {t k} (z _ {t}) = S _ {t - 1, k} (z _ {t}) + \hat {\varphi} (\lambda_ {t}, c _ {t k}, k _ {t}, z _ {t}),
$$
$$
S _ {t k} (z) = S _ {t - 1, k} (z), \quad \forall z \in \mathcal {Z} \backslash \{z _ {t} \}.
$$
Compute the expected protected group allocation  $\delta_t = \mathbb{E}[a_t(x_t) \mid c_{tk_t}, z_t, x_t]$  and compute the dual protected groups allocation target and update the dual parameter:
We first upper bound the performance of OPT. For all  $t$  let  $h_t \in [K]^\mathcal{Z}$  be the deterministic source selection policies,  $k_t = h_t(z_t)$  the selected sources,  $x_t \in \mathcal{X}$  be  $(z_t, c_{t,k_t})$  measurable, and  $\lambda \in \mathbb{R}^d$  be a dual variable. In an identical manner to Equation (20) we can obtain the following inequality:
$$
\max  _ {\mathbf {x} \in \mathcal {X} ^ {T}} \mathbb {E} [ \mathcal {U} (\boldsymbol {x}, \boldsymbol {k}) \mid z _ {1}, c _ {1, k _ {1}}, \dots , z _ {T}, c _ {T, k _ {T}} ] \leq \sum_ {t = 1} ^ {T} \max  _ {\mathbf {x} \in \mathcal {X}} \mathbb {E} [ f _ {t} (x) - \langle \lambda , a _ {t} (x) \rangle \mid z _ {t}, c _ {t k _ {t}} ] - p _ {k _ {t}} + T R ^ {*} (\lambda).
$$
Let us define  $\varphi (\lambda ,c_{tk},k,z) = \max_{\mathbf{x}\in \mathcal{X}}\mathbb{E}[f_t(x) - \langle \lambda ,a_t(x)\rangle |z,c_{tk}] - p_k$  and  $\mu (z) = \operatorname *{Pr}(z_t = z)$ . We also define  $\pi_k(z) = \sum_{t = 1}^T\mathbb{1}[h_t(z) = k] / T$  the total number of times we would have selected source  $k$  for the public context  $z$ .