SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right)\right) \right] = \mathbb {E} \left[ \left(\sum_ {t = 1} ^ {T} f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \bar {R} \left(\gamma_ {t}\right)\right) \right. (22) \\ + \left(\sum_ {t = 1} ^ {T} \bar {R} \left(\gamma_ {t}\right) - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right)\right) (23) \\ \left. + \left(T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right)\right)\right) \right]. (24) \\ \end{array}
$$
The first line (Equation (22)) corresponds to some adjusted separable rewards, and we bound it by first reducing it to the virtual rewards  $\varphi$ , and then using bandits algorithms techniques. The second line (Equation (23)) corresponds to the difference between the unfairness of the adjusted rewards, and the true unfairness, which is not big because of the gradient descent on  $\lambda_{t}$ . Finally the last line (Equation (24)) corresponds to the difference between the penalty of the expected allocation and the expected penalty of the allocation, which we have already shown in Lemma 3.3 is bounded by  $\mathcal{O}(\sqrt{T})$ .
Reduction of Equation (22) to a bandit problem with  $\varphi$  Using first the tower property of the conditional expectation, and then Equation (15), we have that
$$
\mathbb {E} [ f _ {t} (x _ {t}) ] = \mathbb {E} \left[ \mathbb {E} [ f _ {t} (x _ {t}) \mid \mathcal {H} _ {t} ] \right] = \mathbb {E} \left[ \mathbb {E} _ {f _ {t}} \left[ f _ {t} (x _ {t}) \mid c _ {t k _ {t}} \right] \right].
$$
We add and remove  $\langle \lambda_t, \delta_t \rangle$ , and use that  $x_t$  is the allocation that yields the virtual reward function:
$$
\begin{array}{l} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} \right] = \mathbb {E} \left[ \mathbb {E} _ {f _ {t}} \left[ f _ {t} \left(x _ {t}\right) \mid c _ {t k _ {t}} \right] - p _ {k _ {t}} - \langle \lambda_ {t}, \delta_ {t} \rangle + \langle \lambda_ {t}, \delta_ {t} \rangle \right] \\ = \mathbb {E} [ \varphi (\lambda_ {t}, c _ {t k _ {t}}, k _ {t}) + \langle \lambda_ {t}, \delta_ {t} \rangle ]. \\ \end{array}
$$
Using the definition of  $\gamma_{t}$  as a maximizer, we have
$$
\begin{array}{l} - \bar {R} (\gamma_ {t}) = - \bar {R} (\gamma_ {t}) + \langle \lambda_ {t}, \gamma_ {t} \rangle - \langle \lambda_ {t}, \gamma_ {t} \rangle \\ = \max  _ {\gamma \in \Delta_ {\delta_ {t}}} \left\{\langle \lambda_ {t}, \gamma \rangle - \bar {R} (\gamma) \right\} - \langle \lambda_ {t}, \gamma_ {t} \rangle . \\ \end{array}
$$
Because  $\Delta \subset \Delta_{\delta_t}$  and, because  $\bar{R}$  and  $R$  are equal over  $\Delta$  we recover a lower bound with  $R^*$ :
$$
\begin{array}{l} \max  _ {\gamma \in \Delta_ {\delta_ {t}}} \left\{\langle \lambda_ {t}, \gamma \rangle - \bar {R} (\gamma) \right\} - \langle \lambda_ {t}, \gamma_ {t} \rangle \geq \max  _ {\gamma \in \Delta} \left\{\langle \lambda_ {t}, \gamma \rangle - \bar {R} (\gamma) \right\} - \langle \lambda_ {t}, \gamma_ {t} \rangle \\ = \max  _ {\gamma \in \Delta} \left\{\left\langle \lambda_ {t}, \gamma \right\rangle - R (\gamma) \right\} - \left\langle \lambda_ {t}, \gamma_ {t} \right\rangle \\ = R ^ {*} \left(\lambda_ {t}\right) - \langle \lambda_ {t}, \gamma_ {t} \rangle . \\ \end{array}
$$
$$
\mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \bar {R} \left(\gamma_ {t}\right) \right] \geq \mathbb {E} \left[ \varphi \left(\lambda_ {t}, c _ {t k _ {t}}, k _ {t}\right) + R ^ {*} \left(\lambda_ {t}\right) + \langle \lambda_ {t}, \delta_ {t} - \gamma_ {t} \rangle \right]. \tag {25}
$$
Application of Bandits Algorithm Let  $\pi_t$  be the distribution generated according to the algorithm. We actually have that  $\mathbb{E}[\varphi(\lambda_t, c_{t k_t}) + R^*(\lambda_t)]$  is just (through independence, measurability, and total expectation arguments)  $\mathbb{E}[\langle \pi_t, \mathcal{D}(\lambda_t) \rangle]$ . Hence the main idea is to apply Online Convex Optimization theorems to the linear rewards  $\mathcal{D}(\lambda_t)$  with the decision variable  $\pi_t$  (see Hazan (2022), Orabona (2019) or Lattimore & Szeptésvári (2020) for introductions to these types of problems). Here we don't have access either to  $\mathcal{D}(\lambda_t)$ , nor even to  $\mathcal{D}(\lambda_t, k_t)$ : we are in an adversarial bandit setting regarding the  $\lambda_t$ , and also in a stochastic setting regarding the expectation in  $\mathcal{D}$  taken with respect to  $c_t$ . Hence we use an unbiased estimator of the gradient  $\mathcal{D}(\lambda_t)$  for the linear gain, which we will describe further down.
Notice that it is crucial to be able to deal with any sequence  $\mathcal{D}(\lambda_t)$ , this is because these are neither independent nor related to martingales, as the  $\lambda_{t}$  are generated from a quite complicated Markov Chain with states  $(\lambda_{t},c_{t},\pi_{t},k_{t})$ . Hence the outside expectation does not help us much, and we will face as losses (or here rewards) the random arbitrary sequence of the  $\mathcal{D}(\lambda_t)$ . We don't care about  $R^{*}(\lambda_{t})$  as it is  $\mathcal{H}_{t - 1}$  measurable.
Here the different arms of a bandit problem correspond to different sources. Using the tower rule, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \varphi \left(\lambda_ {t}, c _ {t k _ {t}}, k _ {t}\right) \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \varphi \left(\lambda_ {t}, c _ {t k _ {t}}, k _ {t}\right) \mid \mathcal {H} _ {t - 1} \right] \right].
$$
To use results for adversarial bandits, the bandits rewards need to be bounded. By Lemma D.4, the virtual reward  $\varphi$ , and consequently its conditional expectation, is bounded by  $m := \bar{f} + L + 2\eta \mathrm{Diam}(\Delta) + \max_k |p_k|$ . Therefore  $\mathbb{E}[\varphi(\lambda_t, c_{tk}, k) \mid \mathcal{H}_{t-1}] / m \in [-1, 1]$ .
We now construct an unbiased estimator of  $\mathbb{E}[\varphi (\lambda_t,c_{tk},k)\mid \mathcal{H}_{t - 1}]$ , which is the following importance weighted estimator:
$$
\hat {\varphi} (\lambda_ {t}, c _ {t k}, k) = m - \mathbb {1} [ k _ {t} = k ] \frac {(m - \varphi (\lambda_ {t} , c _ {t k} , k))}{\pi_ {t k}}.
$$
This estimator is unbiased for the conditional expectation given  $\mathcal{H}_{t - 1}$ . Indeed because  $\pi_{tk}$  is  $\mathcal{H}_{t - 1}$  measurable, and because  $\mathbb{1}[k_t = k]$  and  $\varphi (\lambda_t,c_{tk},k)$  are independent conditionally on  $\mathcal{H}_{t - 1}$  we
have:
$$
\begin{array}{l} \mathbb {E} [ \hat {\varphi} (\lambda_ {t}, c _ {t k}, k) \mid \mathcal {H} _ {t - 1} ] = m - \frac {1}{\pi_ {t k}} \mathbb {E} [ (m - \varphi (\lambda_ {t}, c _ {t k}, k)) \mathbb {1} [ k _ {t} = k ] \mid \mathcal {H} _ {t - 1} ] \\ = m - \frac {1}{\pi_ {t k}} \mathbb {E} [ m - \varphi (\lambda_ {t}, c _ {t k}, k) \mid \mathcal {H} _ {t - 1} ] \mathbb {E} [ \mathbb {1} [ k _ {t} = k ] \mid \mathcal {H} _ {t - 1} ] \\ = m - \mathbb {E} [ m - \varphi (\lambda_ {t}, c _ {t k}, k) \mid \mathcal {H} _ {t - 1} ] \\ = \mathbb {E} [ \varphi (\lambda_ {t}, c _ {t k}, k) \mid \mathcal {H} _ {t - 1} ], \\ \end{array}
$$
which is exactly the reward needed.
We can now use the following Theorem 11.2 from Lattimore & Szepesvári (2020):
Theorem (EXP3 Regret Bound). Let  $X \in [0,1]^{T \times K}$  be the rewards of an adversarial bandit, then running EXP3 with learning rate  $\rho = \sqrt{2\log(K) / (TK)}$  and denoting  $k_{t}$  the arm chosen at time  $t$ , achieves the following regret bound:
$$
\max  _ {k \in K} \sum_ {t = 1} ^ {T} X _ {t, k} - \mathbb {E} [ \sum_ {t = 1} ^ {T} X _ {t, k _ {t}} ] \leq \sqrt {2 T K \log (K)}
$$
The fact that we can possibly have negative rewards instead of in [0, 1] like requested in the theorem, just changes that the loss estimator used as an intermediary is bounded by 2 instead of 1. Hence by selecting  $\rho = \sqrt{\log(K) / TK} /m$ , and using the importance weighted estimator  $\hat{\varphi} (\lambda_t,c_t)$  we can use the EXP3 Theorem an apply the expectation to obtain the following regret bound:
$$
\mathbb {E} \left[ \max  _ {k \in [ K ]} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \varphi \left(\lambda_ {t}, c _ {t k}, k\right) \mid \mathcal {H} _ {t - 1} \right] \right] - \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \varphi \left(\lambda_ {t}, c _ {t k _ {t}}, k _ {t}\right) \right] \leq 2 m \sqrt {T K \log (K)}.
$$
We know that any convex combination of the sum of the rewards of multiple sources is smaller than the sum of the rewards for the best source. Let us denote by  $(\pi^n)_{n\in \mathbb{N}}$  the sequence of distributions that maximizes the dual objective of Lemma E.1. We will use  $\pi^n$  as a convex combination, and taking the expectation over the regret bound we obtain:
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \varphi \left(\lambda_ {t}, c _ {t k _ {t}}, k _ {t}\right) \right] \geq \mathbb {E} \left[ \sum_ {k = 1} ^ {K} \pi_ {k} ^ {n} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \varphi \left(\lambda_ {t}, c _ {t k}, k\right) \mid \mathcal {H} _ {t - 1} \right] \right] - 2 m \sqrt {T K \log (K)}.
$$
Moreover using that  $c_{tk}$  is independent of  $\mathcal{H}_{t-1}$  and that  $\lambda_t$  is  $\mathcal{H}_{t-1}$  measurable we can deduce using the freezing Lemma that
$$
\mathbb {E} \left[ \varphi \left(\lambda_ {t}, c _ {t k}, k\right) \mid \mathcal {H} _ {t - 1} \right] + R ^ {*} (\lambda_ {t}) = \mathbb {E} _ {c _ {t k}} \left[ \varphi \left(\lambda_ {t}, c _ {t k}, k\right) \right] + R ^ {*} (\lambda_ {t}) = \mathcal {D} \left(\lambda_ {t}, k\right).
$$
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \varphi \left(\lambda_ {t}, c _ {t k _ {t}}\right) + R ^ {*} (\lambda_ {t}) \right] \geq \mathbb {E} \left[ \sum_ {k = 1} ^ {K} \pi_ {k} ^ {n} \sum_ {t = 1} ^ {T} \mathcal {D} \left(\lambda_ {t}, k\right) \right] - 2 m \sqrt {T K \log (K)} \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \pi^ {n}, \mathcal {D} (\lambda_ {t}) \rangle \right] - 2 m \sqrt {T K \log (K)} \\ \geq \sum_ {t = 1} ^ {T} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \left\langle \pi^ {n}, \mathcal {D} (\lambda) \right\rangle - 2 m \sqrt {T K \log (K)}. \\ \end{array}
$$
Taking the limit in  $n\to \infty$  and by definition of  $\pi^n$  we conclude that
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \varphi \left(\lambda_ {t}, c _ {t k _ {t}}\right) + R ^ {*} \left(\lambda_ {t}\right) \right] \geq T \sup  _ {\pi \in \mathcal {P} _ {K}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \left\langle \pi , \mathcal {D} (\lambda) \right\rangle - 2 m \sqrt {T K \log (K)}.
$$
Using the above regret bound and Equation (25) we obtain
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \bar {R} (\gamma_ {t}) \right] \geq T \sup  _ {\pi \in \mathcal {P} _ {K}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \left\langle \pi , \mathcal {D} (\lambda) \right\rangle - 2 m \sqrt {T K \log (K)} \\ + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \lambda_ {t}, \delta_ {t} - \gamma_ {t} \right\rangle \right]. \tag {26} \\ \end{array}
$$
Analysis of Equation (23) and dual gradient descent We will now compare  $TR(\sum_{t=1}^{T} \delta_t / T)$  and  $\sum_{t=1}^{T} \bar{R}(\gamma_t)$ . Here we consider a modified version of  $\bar{R}$ , which is equal to  $\bar{R}$  over  $\bar{\Delta}$  and is equal to  $+\infty$  outside of  $\bar{\Delta}$ , and we will use its convex conjugate  $\bar{R}^*$  (note that the convex conjugate of  $\bar{R}$  without these modifications would have been different!). Let
$$
\hat {\lambda} \in \underset {\lambda \in \mathbb {R} ^ {d}} {\arg \max } \{\langle \lambda , \frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t} \rangle - \bar {R} ^ {*} (\lambda) \}.
$$
We have by the Fenchel-Moreau theorem that
$$
\bar {R} \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) = \bar {R} ^ {* *} \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) = \langle \hat {\lambda}, \frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t} \rangle - \bar {R} ^ {*} (\hat {\lambda}).
$$
By definition of the convex conjugate of  $\bar{R}$ , for any  $\gamma \in \mathbb{R}^d$ , thus in particular for all  $\gamma_t$ , we have
$$
\bar {R} ^ {*} (\hat {\lambda}) \geq \langle \hat {\lambda}, \gamma_ {t} \rangle - \bar {R} (\gamma_ {t}).
$$
Summing for every  $t\in [T]$  we obtain
$$
T \bar {R} ^ {*} (\hat {\lambda}) \geq \sum_ {t = 1} ^ {T} \langle \hat {\lambda}, \gamma_ {t} \rangle - \bar {R} (\gamma_ {t})
$$
$$
\begin{array}{l} \langle \hat {\lambda}, \sum_ {t = 1} ^ {T} \delta_ {t} \rangle - T \bar {R} \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) \geq \sum_ {t = 1} ^ {T} \langle \hat {\lambda}, \gamma_ {t} \rangle - \bar {R} (\gamma_ {t}) \\ \Leftrightarrow \quad \langle \hat {\lambda}, \sum_ {t = 1} ^ {T} \delta_ {t} - \gamma_ {t} \rangle \geq T \bar {R} \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) - \sum_ {t = 1} ^ {T} \bar {R} (\gamma_ {t}). \\ \end{array}
$$
Finally because  $\bar{R}$  and  $R$  are equal over  $\Delta$ , and because  $\sum_{t=1}^{T} \delta_t / T \in \Delta$ , we derive the following inequality:
$$
\sum_ {t = 1} ^ {T} \bar {R} \left(\gamma_ {t}\right) - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) \geq \sum_ {t = 1} ^ {T} \langle \hat {\lambda}, \gamma_ {t} - \delta_ {t} \rangle . \tag {27}
$$
We now would like to track the left-hand sum evaluated at the dual parameter  $\hat{\lambda}$ . To do so we will use straightforward Online Gradient Descent on the linear function  $\lambda \mapsto \langle \lambda, \gamma_t - \delta_t \rangle$ , with sub-gradients  $\gamma_t - \delta_t$ .
We will use Lemma 6.17 from Orabona (2019):
Theorem (Online Gradient Descent). For  $g_{t} = \gamma_{t} - \delta_{t}$  a sub-gradient of a convex loss function  $l_{t}$  with  $\| g_t\| _2\leq G$ , for  $w_{t}$  generated according to a gradient descent update with parameter  $\eta$ , we have that for any  $u\in \mathbb{R}^d$ :
$$
\sum_ {t = 1} ^ {T} l _ {t} (w _ {t}) - l _ {t} (u) \leq \frac {\| u \| _ {2} ^ {2}}{2 \eta} + \frac {\eta}{2} T G ^ {2}.
$$
This is exactly our setting, and the way we update  $\lambda_{t}$ . Therefore
$$
\sum_ {t = 1} ^ {T} \langle \lambda_ {t}, \gamma_ {t} - \delta_ {t} \rangle - \langle \hat {\lambda}, \gamma_ {t} - \delta_ {t} \rangle \leq \frac {\| \hat {\lambda} \| _ {2} ^ {2}}{2 \eta} + \frac {\eta}{2} T G ^ {2}.
$$
Moreover, because  $\hat{\lambda}$  is the arg max which yields the convex conjugate of  $\bar{R}^*$ , we have  $\hat{\lambda} \in \partial \bar{R}^{**}(\sum \delta_t / T) = \partial R^{**}(\sum \delta_t / T)$ . Because  $R$  is  $L$ -Lipschitz for  $\| \cdot \|_2$ , we have  $\| \hat{\lambda} \|_2 \leq L$ . Hence
$$
\sum_ {t = 1} ^ {T} \langle \lambda_ {t}, \gamma_ {t} - \delta_ {t} \rangle - \langle \hat {\lambda}, \gamma_ {t} - \delta_ {t} \rangle \leq \frac {L ^ {2}}{2 \eta} + \frac {\eta}{2} T G ^ {2}.
$$
What do we lose by using  $\bar{R}$  instead of  $R$  to generate  $\gamma_{t}$  in our gradient descent? We have larger gradients bounds by a factor 2 in our case. Indeed instead of having  $G\leq \mathrm{Diam}(\Delta)$ , because  $\gamma_{t}$  can be outside of  $\Delta$ , we have  $\| \gamma_t - \delta_t\| _2\leq 2\operatorname {Diam}(\Delta)$ . In this case, choosing  $\eta = L / (2\operatorname {Diam}(\Delta)\sqrt{T})$  is optimal, and yields the following regret bound:
$$
\sum_ {t = 1} ^ {T} \left\langle \lambda_ {t}, \gamma_ {t} - \delta_ {t} \right\rangle - \left\langle \hat {\lambda}, \gamma_ {t} - \delta_ {t} \right\rangle \leq 2 L \operatorname {D i a m} (\Delta) \sqrt {T} \tag {28}
$$
Using both Equation (27) and Equation (28) yield:
$$
\sum_ {t = 1} ^ {T} \bar {R} \left(\gamma_ {t}\right) - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}\right) \geq \sum_ {t = 1} ^ {T} \left\langle \lambda_ {t}, \gamma_ {t} - \delta_ {t} \right\rangle - 2 L \operatorname {D i a m} (\Delta) \sqrt {T} \tag {29}
$$
We can finally put everything together. Through the decomposition with Equations (22) to (24), using respectively Equations (26) and (29) and Lemma 3.3 for each of those differences and summing the inequalities, the terms in  $\langle \lambda_t,\gamma_t - \delta_t\rangle$  cancel out, and we obtain that
$$
\mathrm {A L G} \geq T \sup _ {\pi \in \mathcal {P} _ {K}} \inf _ {\lambda \in \mathbb {R} ^ {d}} \langle \pi , \mathcal {D} (\lambda) \rangle - 2 m \sqrt {T K \log (K)} - 2 L \operatorname {D i a m} (\Delta) \sqrt {T} - 2 L \sqrt {d T}.
$$
By applying Lemma E.1 we conclude that
$$
\operatorname {R e g} = \operatorname {O P T} - \operatorname {A L G} \leq 2 \big ((L + \bar {f} + \max _ {k} | p _ {k} |) \sqrt {K \log (K)} + L \sqrt {d} + L \operatorname {D i a m} (\Delta) \big) \sqrt {T} + 2 L \sqrt {K \log (K)}.
$$
F The Importance of Randomizing Between Sources: OPT vs static-OPT
F.1 Proof of Proposition 2.1
We can directly derive from the proof of Lemma E.1 and Algorithm 1 the following proposition, which is a rewriting of Proposition 2.1.
Proposition. There are instances of the problem such that
$$
\lim  _ {T \rightarrow \infty} \frac {\text {s t a t i c - O P T}}{T} <   \lim  _ {T \rightarrow \infty} \frac {\text {O P T}}{T}.
$$
Proof. What we have shown indirectly in the proof of Theorem 3.1, is that
$$
| T \max  _ {\pi \in \mathcal {P} _ {K}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \langle \pi , \mathcal {D} (\lambda) \rangle - \mathrm {O P T} | \leq \mathcal {O} (\sqrt {T}),
$$
therefore  $\lim_{T\to \infty}\mathrm{OPT} / T = \lim_{T\to \infty}\max_{\pi \in \mathcal{P}_K}\inf_{\lambda \in \mathbb{R}^d}\langle \pi ,\mathcal{D}(\lambda)\rangle /T$  which also gives us a simple way to compute a close upper bound for OPT as a saddle point, instead of having to solve a combinatorial optimization problem.
Following the same arguments as in Lemma E.1 besides the last maximization step, we can also derive that
$$
\text{static - OPT}\leq T\max_{k\in [K]}\inf_{\lambda \in \mathbb{R}^{d}}\mathcal{D}(\lambda ,k).
$$
It remains to find an example such that
$$
\max  _ {k \in [ K ]} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \mathcal {D} (\lambda , k) <   \max  _ {\pi \in \mathcal {P} _ {K}} \inf  _ {\lambda \in \mathbb {R} ^ {d}} \langle \pi , \mathcal {D} (\lambda) \rangle .
$$
We used such an example in Appendix F.2, where  $\max_{k\in [K]}\inf_{\lambda \in \mathbb{R}^d}\mathcal{D}(\lambda ,k) = 0$  and  $\max_{\pi \in \mathcal{P}_K}\inf_{\lambda \in \mathbb{R}^d}\langle \pi ,\mathcal{D}(\lambda)\rangle = 0.25.$
F.2 Simple example and numerical illustration
We illustrate Proposition 2.1 and give an example that shows that randomizing over the different sources can outperform the best performance achievable by a single source.
We consider a case where the utility  $u_{t} \in \{-1, 1\}$  and the protected attribute of a user  $a_{t} \in \{-1, 1\}$ . All combination of  $u_{t}$  and  $a_{t}$  are equiprobable:  $\mathbb{P}(u_{t} = u, a_{t} = a) = 1/4$  for all  $u, a \in \{-1, 1\}$ . The penalty function is  $R(\delta) = 5|\delta|$ . There are  $K = 2$  sources of information. The first source is capable of identifying good individuals of group 1: it gives a context  $c_{t1} = 1$  when  $(a_{t}, u_{t}) = (1, 1)$  and  $c_{t2} = 0$  otherwise. The second source does the same for good individuals of group  $-1$ : it gives  $c_{t2} = 1$  when  $(a_{t}, u_{t}) = (-1, 1)$  and  $c_{t2} = 0$  otherwise. In terms of conditional expectations, this translates into  $\mathbb{E}[u_{t}|c_{t1} = 1] = \mathbb{E}[a_{t}|c_{t1} = 1] = 1$ ,  $\mathbb{E}[u_{t}|c_{t1} = 0] = \mathbb{E}[a_{t}|c_{t1} = 0] = -1/3$  for the first source; and  $\mathbb{E}[u_{t}|c_{t2} = 1] = 1$ ,  $\mathbb{E}[a_{t}|c_{t2} = 1] = -1$ ,  $\mathbb{E}[u_{t}|c_{t2} = 0] = -1/3$ ,  $\mathbb{E}[a_{t}|c_{t2} = 0] = 1/3$  for the second source.
We apply Algorithm 1 to this example, and we compare its performance to  $(i)$  the one of an algorithm that has only access to the first source (which is the best source since both sources are symmetric),  $(ii)$  a greedy algorithm that only use the first source and selects  $x_{t} = 1$  whenever  $\mathbb{E}[u_t\mid c_{t1}] > 0$ , and  $(iii)$  the offline optimal bounds OPT and static-OPT. The results are presented in Figure 1. We observe that our algorithm is close to OPT as shown in Theorem 3.1. It also vastly outperforms the algorithm that simply picks the best source, whose performance is close to static-OPT. The greedy has a largely negative utility because it is unfair. Our randomizing algorithm picks an allocation  $x_{t} = 1$  only when the context implies that  $u_{t} = 1$ . It obtains a fair allocation by switching sources. An algorithm that uses a single source cannot be fair unless it chooses  $x_{t} = 0$ , which is why static-OPT=0 for this example. The  $1^{st}$  and  $3^{rd}$  quartiles are reported for multiple random seeds, the solid lines are the means. The computations were done on a laptop with an i7-10510U and 16 Gb of ram.
F.3 Additional experiments on the effect on OPT of sources and fairness prices
In the main body of the paper, we study how one is able to use the online algorithm Algorithm 1 so as to achieve a good total utility asymptotically close to that of the offline optimal OPT. However we only consider  $R$  and the prices  $p_k$  of the different sources as inputs for the algorithm, and we did not discuss how their variation actually affects the performance of OPT. While the previous illustrating example shows how a mixture of sources may achieve strictly better performance, we now only focus on the static optimization problem and how various metrics may change as we vary  $R$  and  $p_k$ .
Data model: We consider a fairness penalty  $R(x) = rx^2$  with  $r \geq 0$ , and  $\mathcal{A} = \{-1,1\}$  so that  $a_t$  can only be either -1 or 1, encoding men and women for instance. We suppose that the protected attributes and the utility  $(u_t, a_t)$  are distributed according to  $\operatorname*{Pr}(a_t = 1) = \operatorname*{Pr}(a_t = -1) = 1/2$  and  $u_t \sim \mathcal{N}(a_t, 1)$ , hence the group for which  $a_t = 1$  is generally correlated to having better utilities than those for which  $a_t = -1$ . We suppose that there are no public contexts, and that there are  $K = 2$  information sources, with  $c_{t,1} = (a_t, u_t)$  for source 1 associated with a price of  $p_1 = p$ , and for source  $k = 2$  we can observe  $c_{t,2} = u_t$  with a price of 0. Here the sources are 'monotone' in that one contains strictly more information than the other, but is more expensive. Basically we are going to observe the utility  $u_t$  in all cases, but before observing  $u_t$  we can pay  $p$  to additionally observe  $a_t$ . Note that when  $r = 0$  then  $R = 0$  and there is no fairness penalty meaning that we don't actually care about  $a_t$  and will always select source  $k = 2$ , while when  $p = 0$  the information about  $a_t$  is free and there is no uncertainty.
We study the asymptotic regime with  $T \to \infty$ , and compute the optimal solution of the offline optimization problem through its dual problem. Figure 1 represents the quantities associated to the optimal solution for varying  $p$  and  $r$ : (top-left)  $\pi^*$  the percentage of times information is bought (i.e.,  $k = 1$  is chosen), (top-right) the probability of group 1 being selected  $\operatorname{Pr}(a_t = 1 \mid x_t = 1)$ , (bottom-left) the fairness cost relative to utility  $R(\mathbb{E}[a_tx_t]) / \mathbb{E}[u_tx_t]$ , and (bottom-right) the information cost relative to utility  $p\pi^* / \mathbb{E}[u_tx_t]$ .
Analysis and interpretation: