SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- (top-left) Looking at the optimal action in terms of information (to buy the observation of  $a_t$  or not), we see that there are three regimes:  $\pi^* = 1$ ,  $\pi^* = 0$  and a transitive regime  $\pi^* \in (0,1)$ . When  $r = 0$  there is no reason to care about  $a_t$ , thus we simply do not buy information  $(\pi^* = 0)$ , this remains true as long as the cost of  $p$  is large over  $r$ . When  $p = 0$  there are no costs to observe  $a_t$  and thus  $\pi^* = 1$  and this remains true as long as the fairness penalty  $r$  is large enough over  $p$ . In between, we may obtain optimal actions which are strictly between 0 and 1: in this case the best action is a strict convex combination between the two sources and there is some trade-off between buying costly information and selecting an unfair allocation.
- (top-right) Regarding the proportion of individuals of group 1 selected, we see that it is always higher than 0.5 as group  $a_{t} = 1$  correlates to higher utility compared to group  $a_{t} = -1$ . Therefore it is decreasing in  $r$  and goes towards the perfectly fair allocation 0.5 which would incur no penalty even for high  $r$ . It is also increasing in  $p$  as higher information costs make it less desirable to pay to observe  $a_{t}$ , and thus more difficult to accurately achieve a fair allocation: the support of the utility  $u_{t}$  conditioned on  $a_{t} = 1$  and conditioned on  $a_{t} = -1$  are not disjoint, which makes it difficult only by observing  $u_{t}$  to determine whether  $a_{t} = 1$  or  $-1$ .
- (bottom-left) For the relative fairness cost compared to utility, when the information cost  $p$  increases the allocation becomes more unfair as it becomes more difficult without access to  $a_t$  to choose a fair allocation with good utility as describe above. Moreover the average utility  $u_t$  of the selected groups tend to decrease as the individuals which are more likely to be  $a_t = -1$  based on  $u_t$  have low utility. Therefore the ratio fairness penalty over expected utility tends to increase. When  $r$  increases there are three conflicting effects: high  $r$  makes it more appealing to achieve a fair allocation at the cost of expected utility  $u_t$  which makes the latter decrease, the expected group allocation becomes more fair as seen previously, and the scaling of  $R$  increases. When  $r$  becomes high there is some balance between a fairer allocation and higher penalty, and when  $r = 0$  the fairness penalty is clearly 0. Overall for each fixed  $p$  there is some maximum in terms of fairness penalty relative to utility.
- (bottom-right) Finally for the information cost relative to utility, it is increasing in  $r$ . Indeed due to large fairness penalties it is better to buy information thus  $\pi^*$  increases while  $p$  remains fixed, hence the product  $p\pi^*$  increases. When  $p$  increases, we buy information less often ( $\pi^*$  decreases), and because  $p\pi^* = 0$  for  $p = 0$  or  $p$  large, there is some maximum for each  $r$  fixed.
Overall, we see that higher fairness penalties tend to make it more likely to buy information and select a fair allocation even at the cost of utility, while high information cost makes it unattractive to buy this information and can lead to unfairness due to the difficulty of identifying the protected attribute without this additional data.
Here we looked at a simple example to try to isolate how sensible is the optimal solution to some variations on  $r$  and  $p$ . Other parameters can lead to unfairness, such as an unbalance in the protected group population where one would still need a balanced allocation, or penalty functions with varying convexity strength.
G Other Fairness Penalty — proof of Theorem 4.1
In this section, we consider an other type of fairness penalty as described in Section 4.1.
We redefine  $\mathcal{U}(.)$  in the following way:
$$
\mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) = \sum_ {t = 1} ^ {T} \left(f _ {t} \left(x _ {t}\right) - p _ {k _ {t}}\right) - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R \left(\frac {\sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right)}{\sum_ {t = 1} ^ {T} x _ {t}}\right). \tag {30}
$$
When  $\mathcal{X} \subset \mathbb{R}_+^n$  with  $n > 1$ , there are two ways to intuitively generalize this penalty for higher dimension allocations. We can replace  $\sum_{t=1}^{T} x_t$  by  $\sum_{t=1}^{T} \sum_{i=1}^{n} x_{ti}$ , which is still a scalar and the proof follows directly. For instance, this corresponds for online bipartite matching with node arrival to a fairness condition on all online arriving nodes (it does not matter to which offline nodes they are matched). Otherwise we can instantiate  $n$  penalties  $R_1, \ldots, R_n$ , using  $\sum_{t} x_{ti}$  for each penalty  $R_i$  with  $i \in [n]$ . This corresponds to a fairness condition on the selected online nodes for each of the  $n$  offline nodes. This can be dealt with using one parameter  $\lambda_i$  for each penalty  $R_i$ . For this proof we will assume that  $n = 1$  without loss of generality.
We require the following technical assumption:  $\sum_{\tau = 1}^{t}a_{\tau}(x_{\tau}) / \sum_{\tau = 1}^{t}x_{\tau}\in \Delta$  for all  $x_{\tau},a_{\tau}$  and  $t$ . Of course, the size of  $\Delta$  can be increased to guarantee — if possible — that this quantity stays inside  $\Delta$ , but with the trade-off that  $\mathrm{Diam}(\Delta)$  increases. Clearly, this assumption is verified for the special case of  $a_{t}(x_{t}) = ax_{t}$  for some  $a\in \mathcal{A}$ , as the  $x_{\tau} / \sum_{t}x_{t}$  form a convex combination.
Let us redefine the virtual reward associated to this other fairness penalty:
$$
\tilde {\varphi} (\lambda , c _ {t k}, k) = \max  _ {x \in \mathcal {X}} \mathbb {E} \left([ f _ {t} (x) - \langle \lambda_ {t}, a _ {t} (x) \rangle | c _ {t k} ] + x R ^ {*} (\lambda_ {t})\right) - p _ {k}.
$$
We also redefine the unbiased estimator  $\hat{\varphi}$  with this new  $\tilde{\varphi}$ .
We now use Algorithm 3. There are 4 differences with Algorithm 2: the new function  $\varphi$  contains in addition  $xR^{*}(\lambda)$ ,  $x_{t}$  is the arg max with respect to this new virtual reward,  $\gamma_{t}$  is the arg max over  $\Delta_{\delta_t / x_t}$  instead of  $\Delta_{\delta_t}$ , and finally the update for  $\lambda_{t + 1}$  uses  $x_{t}\gamma_{t}$  instead of  $\gamma_{t}$ .
Algorithm 3 Online Fair Allocation with Source Selection — Other fairness penalty
Input: Initial dual parameter  $\lambda_0$ , initial source-selection-distribution  $\pi_0 = (1 / K, \dots, 1 / K)$ , dual gradient descent step size  $\eta$ , EXP3 step size  $\rho$ , cumulative estimated rewards  $S_0 = 0 \in \mathbb{R}^K$ .
for  $t\in [T]$  do
$$
x _ {t} = \underset {x \in \mathcal {X}} {\arg \max } \left(\mathbb {E} \left[ f _ {t} (x) - \left\langle \lambda_ {t}, a _ {t} (x) \right\rangle \mid c _ {t k _ {t}} \right] + x R ^ {*} (\lambda_ {t})\right). \tag {31}
$$
$$
S _ {t k} = S _ {t - 1, k} + \hat {\varphi} \left(\lambda_ {t}, c _ {t k}, k _ {t}\right), \tag {32}
$$
$$
\gamma_ {t} = \underset {\gamma \in \Delta_ {\delta_ {t} / x _ {t}}} {\arg \max } \{\langle \lambda_ {t}, \gamma \rangle - \bar {R} (\gamma) \},
$$
$$
\lambda_ {t + 1} = \lambda_ {t} - x _ {t} \eta \left(\gamma_ {t} - \delta_ {t} / x _ {t}\right).
$$
Let us first prove an upper bound on OPT, which actually guides the design of the modified virtual reward  $\tilde{\varphi}$ :
Lemma G.1. We have the following upper-bound for the offline optimum:
$$
\tilde {\mathrm {O P T}} \leq T \sup _ {\pi \in \mathcal {P} _ {K}} \inf _ {\lambda \in \mathbb {R} ^ {d}} \langle \pi , \tilde {\mathcal {D}} (\lambda) \rangle ,
$$
where  $\tilde{\mathcal{D}} (\lambda) = (\tilde{\mathcal{D}} (\lambda ,1),\ldots ,\tilde{\mathcal{D}} (\lambda ,K))$  is a vector representing the value of the dual conjugate problem, with for  $k\in [K]$  the coordinates
$$
\tilde {\mathcal {D}} (\lambda , k) = \mathbb {E} _ {c _ {t k}} [ \tilde {\varphi} (\lambda , c _ {t k}, k) ].
$$
Proof. The idea will be the same as in Lemma E.1, we only use a different lagrangian function.
We recall that the  $x_{t}$  are  $\sigma(c_{1k_1},\ldots,c_{tk_t})$  measurable (because the  $k_{t}$  are deterministic). Let  $\delta_{t} = \mathbb{E}_{a_{t}}[a_{t}(x_{t})|c_{tk}]$ , and  $\tilde{f}_t(x) = \mathbb{E}_{f_t}[a_t(x_t)|c_{tk}]$ . Using Jensen's inequality for  $R$  convex we have
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] - \mathbb {E} \left[ \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R \left(\frac {\sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right)}{\sum_ {t = 1} ^ {T} x _ {t}}\right) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) \mathbb {E} \left[ R \left(\frac {\sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right)}{\sum_ {t = 1} ^ {T} x _ {t}}\right) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R \left(\mathbb {E} \left[ \frac {\sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right)}{\sum_ {t = 1} ^ {T} x _ {t}} \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right]\right) \\ = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R \left(\frac {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} a _ {t} \left(x _ {t}\right) \mid c _ {1 k _ {1}} , \dots , c _ {T k _ {T}} \right]}{\sum_ {t = 1} ^ {T} x _ {t}}\right). \\ \end{array}
$$
Moreover by independence of the  $(f_{t},a_{t},c_{t1},\ldots ,c_{tK})$  we have that  $\mathbb{E}[a_t(x_t)\mid \sigma (c_{1k_1},\dots ,c_{tk_t})] = \mathbb{E}_{a_t}[a_t(x_t)\mid c_{tk_t}] = \delta_t$ . This is basically an application of Lemma D.3 where we state it more carefully. With the same argument we obtain that  $\mathbb{E}[f_t(x_t)\mid \sigma (c_{1k_1},\dots ,c_{Tk_T})] = \tilde{f}_t(x_t)$ . Therefore
$$
\mathbb {E} \left[ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \leq \sum_ {t = 1} ^ {T} \tilde {f} _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right). \tag {33}
$$
We define the function  $\mathcal{L}:(\pmb {k},\pmb {c},\pmb {x},\lambda)\mapsto \sum_{t = 1}^{T}\tilde{f}_t(x_t) - p_{k_t} - x_t\langle \lambda ,\tilde{a}_t\rangle +x_tR^* (\lambda).$
Using the Fenchel-Moreau theorem, and the previous inequality we have
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {k}, \mathbf {c}, \mathbf {x}, \lambda) \geq \inf  _ {\lambda \in \mathbb {R} ^ {d}} \mathcal {L} (\mathbf {c}, \mathbf {x}, \lambda) \\ = \sum_ {t = 1} ^ {T} \tilde {f} _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \sup  _ {\lambda \in \mathbb {R} ^ {d}} \left\{\langle \lambda , \sum_ {t = 1} ^ {T} \tilde {a} _ {t} x _ {t} \rangle - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R ^ {*} (\lambda) \right\} \\ = \sum_ {t = 1} ^ {T} \tilde {f} _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) \sup  _ {\lambda \in \mathbb {R} ^ {d}} \left\{\langle \lambda , \frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}} \rangle - R ^ {*} (\lambda) \right\} \\ = \sum_ {t = 1} ^ {T} \tilde {f} _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R ^ {* *} \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right) \\ = \sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x _ {t}) - p _ {k _ {t}} - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right) \\ \geq \mathbb {E} [ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid \mathbf {c} _ {\boldsymbol {k}} ]. \tag {34} \\ \end{array}
$$
We can then finish as we did for Lemma E.1.
We now proceed to the rest of the proof of Theorem 4.1:
Theorem. For  $\eta = L / (2\mathrm{Diam}(\Delta)\sqrt{T})$ ,  $m = \bar{f} +L + \max_k|p_k| + 2\eta \mathrm{Diam}(\Delta) + m^*$  ( $m^{*}$  is a constant defined in the proof and depends on  $\Delta$  and  $R$ ),  $\rho = \sqrt{\log(K) / (TKm^2)}$ , and  $\lambda_0\in \partial R(a)$  for some  $a\in \mathcal{A}$ , algorithm Algorithm 1 modified as suggested in Section 4.1 has the following regret upper bound:
$$
\operatorname {R e g} \leq 2 \left(\left(L + \bar {f} + \max  _ {k} \left| p _ {k} \right| + m ^ {*}\right) \sqrt {K \log (K)} + L \sqrt {d} + \operatorname {D i a m} (\Delta)\right) \sqrt {T} + 2 L \sqrt {K \log (K)}
$$
Proof. We present here only the parts that are different from the proof of Theorem 3.1, and we refer to the corresponding proof for more details.
Comparison of performance with  $\tilde{\varphi}$  By adding and removing  $-x_{t}\langle \lambda_{t},\tilde{a}_{t}\rangle +x_{t}R^{*}(\lambda_{t})$  , and using that  $x_{t}$  is the maximizer that yields  $\varphi$  we derive that
$$
\begin{array}{l} \mathbb {E} \left[ f _ {t} \left(x _ {t}\right) \right] = \mathbb {E} \left[ \mathbb {E} _ {f _ {t}} \left[ f _ {t} \left(x _ {t}\right) \mid c _ {t k _ {t}} \right] - \left\langle \lambda_ {t}, \delta_ {t} \right\rangle + \left\langle \lambda_ {t}, \delta_ {t} \right\rangle + x _ {t} R ^ {*} (\lambda_ {t}) - x _ {t} R ^ {*} (\lambda_ {t}) \right] \\ = \mathbb {E} \left[ \varphi \left(\lambda_ {t}, c _ {t k _ {t}}\right) + \left\langle \lambda_ {t}, \delta_ {t} \right\rangle - x _ {t} R ^ {*} (\lambda_ {t}) \right]. \\ \end{array}
$$
Because  $\Delta \subset \Delta_{\delta_t}$  and, because  $\bar{R}$  and  $R$  are equal over  $\Delta$  we have:
$$
\begin{array}{l} R ^ {*} (\lambda_ {t}) = \max  _ {\gamma \in \Delta} \{\langle \lambda_ {t}, \gamma \rangle - R (\gamma) \} \\ = \max  _ {\gamma \in \Delta} \{\langle \lambda_ {t}, \gamma \rangle - \bar {R} (\gamma) \} \\ \leq \max  _ {\gamma \in \Delta_ {\tilde {a} _ {t}}} \left\{\langle \lambda_ {t}, \gamma \rangle - \bar {R} (\gamma) \right\} \\ = \left\langle \lambda_ {t}, \gamma_ {t} \right\rangle - \bar {R} (\gamma_ {t}). \\ \end{array}
$$
$$
\mathbb {E} \left[ f _ {t} \left(x _ {t}\right) - p _ {k _ {t}} - x _ {t} \bar {R} \left(\gamma_ {t}\right) \right] \geq \mathbb {E} \left[ \tilde {\varphi} \left(\lambda_ {t}, c _ {t k _ {t}}\right) + \left\langle \lambda_ {t}, \delta_ {t} - x _ {t} \gamma_ {t} \right\rangle \right]. \tag {35}
$$
Application of bandit algorithms Compared to the previous setting, the only change is that the function  $R^{*}$  is included in  $\varphi$ , so we need to bound it as well. See Lemma D.5. This is where  $m^{*}$  comes from.
Dual gradient descent Let us now focus in the tracking of the fairness parameter. In the evaluation of our lower bound of the algorithm performance, we used  $-x_{t}\bar{R}(\gamma_{t})$  instead of the true penalty  $(\sum_{t=1}^{T} x_{t})R(\sum_{t=1}^{T} a_{t}(x_{t}) / \sum_{t=1}^{T} x_{t})$ . Because of a small modification of Lemma 3.3 we only need to compare it with the penalty over the conditional expectation of  $a_{t}x_{t}$ . Indeed, by the Lipschitz property of  $R$ , whether we re-scale by  $T$  or  $\sum x_{t}$  does not change anything.
Hence will now compare those two penalties. Here we will consider a modified version of  $\bar{R}$ , which is equal to  $\bar{R}$  over  $\bar{\Delta}$  and is equal to  $+\infty$  outside of  $\bar{\Delta}$ , and we will use its convex conjugate  $\bar{R}^*$  (note that the convex conjugate of  $\bar{R}$  without the modification would have been different!). Let
$$
\hat{\lambda}\in \operatorname *{arg  max}_{\lambda \in \mathbb{R}^{d}}\{\langle \lambda ,\frac{\sum_{t = 1}^{T}\delta_{t}}{\sum_{t = 1}^{T}x_{t}}\rangle -\bar{R}^{*}(\lambda)\} .
$$
We have by Fenchel-Moreau theorem that
$$
\begin{array}{l} \langle \hat {\lambda}, \frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}} \rangle - \bar {R} ^ {*} (\hat {\lambda}) = \bar {R} ^ {* *} \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right) \\ = \bar {R} \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right). \\ \end{array}
$$
$$
\langle \hat {\lambda}, \sum_ {t = 1} ^ {T} \delta_ {t} \rangle - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) \bar {R} \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right) = \left(\sum_ {t = 1} ^ {T} x _ {t}\right) \bar {R} ^ {*} (\hat {\lambda}).
$$
Multiplying by  $x_{t}$  and summing for every  $t \in [T]$  we obtain
$$
\left(\sum_ {t = 1} ^ {T} x _ {t}\right) \bar {R} ^ {*} (\hat {\lambda}) \geq \sum_ {t = 1} ^ {T} \langle \hat {\lambda}, x _ {t} \gamma_ {t} \rangle - x _ {t} \bar {R} (\gamma_ {t})
$$
$$
\begin{array}{l} \langle \hat {\lambda}, \sum_ {t = 1} ^ {T} \delta_ {t} \rangle - \left(\sum_ {t = 1} ^ {T} x _ {t}\right) \bar {R} \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right) \geq \sum_ {t = 1} ^ {T} \langle \hat {\lambda}, \gamma_ {t} x _ {t} \rangle - \bar {R} (\gamma_ {t}) \\ \Leftrightarrow \quad \langle \hat {\lambda}, \sum_ {t = 1} ^ {T} \delta_ {t} - x _ {t} \gamma_ {t} \rangle \geq (\sum_ {t = 1} ^ {T} x _ {t}) \bar {R} \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right) - \sum_ {t = 1} ^ {T} x _ {t} \bar {R} (\gamma_ {t}). \\ \end{array}
$$
Finally because  $\bar{R}$  and  $R$  are equal over  $\Delta$ , and because  $\sum_{t=1}^{T} \delta_t / \sum_{t=1}^{T} x_t \in \Delta$  by the assumption, we have that
$$
\langle \hat {\lambda}, \sum_ {t = 1} ^ {T} \delta_ {t} - x _ {t} \gamma_ {t} \rangle \geq \left(\sum_ {t = 1} ^ {T} x _ {t}\right) R \left(\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}\right) - \sum_ {t = 1} ^ {T} x _ {t} \bar {R} (\gamma_ {t}) \tag {36}
$$
As we did earlier, we apply Online Gradient Descent to track the term evaluated at  $\hat{\lambda}$  and obtain Equation (28). We just need to make sure that as before  $\hat{\lambda}$  is bounded. By the definition of  $\hat{\lambda}$ :
$$
\hat {\lambda} \in \partial \bar {R} ^ {* *} (\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}) = \partial \bar {R} (\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}) = \partial R (\frac {\sum_ {t = 1} ^ {T} \delta_ {t}}{\sum_ {t = 1} ^ {T} x _ {t}}).
$$
Because  $R$  is  $L$ -Lipschitz for  $\| \cdot \| _2$ , we have  $\| \hat{\lambda}\| _2\leq L$
The rest of the proof is identical.
H Learning  $u_{t}$  - proof of Proposition 4.2
We prove in this section the result regarding the added regret of learning  $u_{t}$ . In terms of utility functions  $f_{t}$ , protected attributes  $a_{t}$ , and allocations  $x_{t}$ , we go back to the setting of Section 2.1 with  $f_{t}(x) = u_{t}x$  and  $\mathcal{X} = \{0,1\}$ .
In addition to the assumptions made in Section 4.2, we give more precision regarding the structure of  $u_{t}$ . We define  $\theta_{tk} = \mathbb{1}[k_t = k]$  the indicator variable of selecting source  $k$  at time  $t$ . We define the following filtration for all  $k \in [K]$  and all  $t \in [T]$ :
$$
\mathcal {F} _ {t k} = \sigma \left(x _ {1}, c _ {1 k}, \theta_ {1 k}, u _ {1}, \dots , x _ {t}, c _ {t k}, \theta_ {t k}, u _ {t}, x _ {t + 1}, c _ {t + 1, k}, \theta_ {t + 1, k}\right).
$$
$$
\eta_ {t k} = u _ {t} - \left\langle \psi_ {k}, c _ {t k} \right\rangle , \tag {37}
$$
is a zero mean 1-subgaussian random variable conditionally on  $\mathcal{F}_{t - 1,k}$
Consider that each source  $k$  corresponds to one bandit problem, with an action set at time  $t$  composed of two contextual actions  $\{c_{tk}\theta_{tk}x,x\in \mathcal{X}\} = \{c_{tk}\theta_{tk},0\}$ , and a reward  $u_{t}x_{t}\theta_{tk}$ . If  $\theta_{tk} = 0$  then the action selected is 0, and the new reward is 0, which means both are "observed". If  $\theta_{tk} = 1$ , then the feedback is  $u_{t}$  if  $x_{t} = 1$ , and 0 otherwise. Regardless of the source selected, the feedback received and the action producing this feedback are observed for all arms in parallel. This is akin to artificially playing all the  $[K]$  bandits in simultaneously. This new reward still follows the conditional linearity assumption, and  $x_{t}\theta_{tk}\eta_{t}$  is still  $\mathcal{F}_{t - 1,k}$  conditionally 1-subgaussian.
With these rewards and actions, we can define for each source  $\hat{\psi}_{tk}$  and  $V_{tk}$  the least squares estimator and the design matrix as done in Equations (19.5) and (19.6) of Lattimore & Szepesvári (2020). Let  $\delta \in (0,1)$ . We define the confidence set parameter  $\beta_{t}$  as
$$
\sqrt {\beta_ {t}} = \bar {\psi} + \sqrt {2 \log \left(\frac {1}{\delta}\right) + \max  _ {k \in [ k ]} q _ {k} \log \left(1 + \frac {t \bar {c} ^ {2}}{\max  _ {k \in [ k ]} q _ {k}}\right)}. \tag {38}
$$
This parameter is independent of  $k$ , this will make the computations easier later on. We now define the ellipsoid confidence set of  $\psi_{k}$  as
$$
\mathcal {I} _ {t k} = \left\{\psi \in \mathbb {R} ^ {q _ {k}}, \| \psi - \hat {\psi} _ {t k} \| _ {V _ {t - 1, k}} ^ {2} \leq \beta_ {t} \right\}. \tag {39}
$$
Algorithm 1 is modified in the following way. For each time  $t$ , let  $\tilde{\psi}_{tk} = \arg \max_{\psi \in \mathcal{I}_{tk}}\langle \psi ,c_{tk}\rangle$ . We now select  $x_{t}$  using this optimistic estimate of  $\psi_{k}$ :
$$
x _ {t} = \underset {x \in \{0, 1 \}} {\arg \max } \left\{x \left(\left\langle \tilde {\psi} _ {t k}, c _ {t k} \right\rangle - \left\langle \lambda_ {t}, \tilde {a} _ {t} \right\rangle\right) \right\}. \tag {40}
$$
Proposition. Given the above assumptions, the added regret of having to learn  $\mathbb{E}[u_t\mid c_{tk}]$  is of order  $\mathcal{O}(\sqrt{KT}\log (T))$
The proof will proceed as follows: we first highlight where the learning will impact the algorithm, leverage the concentration results from Abbasi-yadkori et al. (2011) to decompose into a good event and bad event, and finally bound the loss over the good event.
Let us now analyze the main part of the proof in Appendix E that is modified, which is "Comparison of performance with  $\varphi$ ", because the only change is the quantity that  $x_{t}$  maximizes. Let  $x_{t}^{*}$  be the maximizer obtained by replacing the optimistic estimate with the true parameter  $\psi_{k}$ . Let  $\tilde{a}_{t} = \mathbb{E}[a_{t}|c_{tk_{t}}]$ . Because  $x_{t}$  maximizes the quantity with the optimistic estimate we have
$$
\begin{array}{l} x _ {t} \mathbb {E} \left[ u _ {t} \mid c _ {t k _ {t}} \right] - x _ {t} \left\langle \lambda_ {t}, \tilde {a} _ {t} \right\rangle = x _ {t} \left\langle \psi_ {k _ {t}}, c _ {t k _ {t}} \right\rangle - x _ {t} \left\langle \lambda_ {t}, \tilde {a} _ {t} \right\rangle \\ = x _ {t} \left(\left\langle \psi_ {k _ {t}}, c _ {t k _ {t}} \right\rangle - \left\langle \tilde {\psi} _ {t k _ {t}}, c _ {t k _ {t}} \right\rangle\right) + x _ {t} \left\langle \tilde {\psi} _ {t k _ {t}}, c _ {t k _ {t}} \right\rangle - x _ {t} \left\langle \lambda_ {t}, \tilde {a} _ {t} \right\rangle \\ \geq x _ {t} \left\langle \psi_ {k _ {t}} - \tilde {\psi} _ {t k _ {t}}, c _ {t k _ {t}} \right\rangle + x _ {t} ^ {*} \left\langle \tilde {\psi} _ {t k _ {t}}, c _ {t k _ {t}} \right\rangle - x _ {t} ^ {*} \left\langle \lambda_ {t}, \tilde {a} _ {t} \right\rangle \\ = x _ {t} \left\langle \psi_ {k _ {t}} - \tilde {\psi} _ {t k _ {t}}, c _ {t k _ {t}} \right\rangle + x _ {t} ^ {*} \left\langle \tilde {\psi} _ {t k _ {t}} - \psi_ {k _ {t}}, c _ {t k _ {t}} \right\rangle + x _ {t} ^ {*} \left\langle \psi_ {k _ {t}}, c _ {t k _ {t}} \right\rangle - x _ {t} ^ {*} \left\langle \lambda_ {t}, \tilde {a} _ {t} \right\rangle \\ = x _ {t} \left\langle \psi_ {k _ {t}} - \tilde {\psi} _ {t k _ {t}}, c _ {t k _ {t}} \right\rangle + x _ {t} ^ {*} \left\langle \tilde {\psi} _ {t k _ {t}} - \psi_ {k _ {t}}, c _ {t k _ {t}} \right\rangle + \varphi \left(\lambda_ {t}, c _ {t k _ {t}}, k _ {t}\right) + p _ {k _ {t}}. \\ \end{array}
$$