SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Using this unbiased estimator, we can apply the EXP3 algorithm to this virtual reward function.
Gradient descent part. Once the source  $k_{t}$  for user  $t$  is chosen and  $c_{tk_t}$  is observed, we can compute the decision  $x_{t}$  (see (6)). This  $x_{t}$  is then used in (8)-(9) to perform a dual descent step on the multiplier  $\lambda_{t}$ . Although using a dual descent step is classical, our implementation is different because we need to guarantee that the values of  $\lambda_{t}$  remain bounded for EXP3. To do so, we modify the geometry of the convex optimization sub-problem by considering a set of allocation targets larger than the original  $\Delta$ . For  $\delta \in \Delta$ , we define the set  $\Delta_{\delta}$  as the ball of center  $\delta$  and radius  $\mathrm{Diam}(\Delta)$ . This ball contains  $\Delta$ :  $\Delta \subset \Delta_{\delta}$ .
Algorithm 1 uses any extension of  $R$  to  $\bar{\Delta} = \cup_{\delta \in \Delta}\Delta_{\delta}$  that is convex and Lipschitz-continuous, for instance, the following one (see Lemma D.2):
$$
\bar {R} (\delta) = \inf  _ {\delta^ {\prime} \in \Delta} \left\{R \left(\delta^ {\prime}\right) + L \| \delta - \delta^ {\prime} \| _ {2} \right\},
$$
that has the same Lipschitz-constant  $L$  as  $R$  (which is the best Lipschitz-constant possible).
3.2 Algorithm and implementation
Combining these different ideas leads to Algorithm 1. This algorithm maintains a dual parameter  $\lambda_{t}$  that encodes the history of unfairness that ensures that the fairness penalty  $R(.)$  is taken into account. This dual parameter  $\lambda_{t}$  is used in (6) to compute the allocation  $x_{t}$  and in (7) to choose the source of information. The dual update (8)-(9) guarantees that we take  $R(.)$  into account.
Algorithm 1 Online Fair Allocation with Source Selection
Input: Initial dual parameter  $\lambda_0$ , step sizes  $\eta$  and  $\rho$ , cumulative estimated rewards  $S_0 = 0 \in \mathbb{R}^K$ . for  $t \in [T]$  do
Draw a source  $k_{t} \sim \pi_{t}$  where  $\pi_{tk} = \exp(\rho S_{t-1,k}) / \sum_{l=1}^{K} \exp(\rho S_{t-1,l})$ , and observe  $c_{tk_{t}}$ . Compute the allocation for user  $t$ :
$$
x _ {t} = \left\{ \begin{array}{l l} 1 & \text {i f} \mathbb {E} \left[ u _ {t} \mid c _ {t k _ {t}} \right] \geq \left\langle \lambda_ {t}, \mathbb {E} \left[ a _ {t} \mid c _ {t k _ {t}} \right] \right\rangle \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {6}
$$
Update the estimated rewards sum and sources distributions for all  $k \in [K]$ :
$$
S _ {t k} = S _ {(t - 1) k} + \hat {\varphi} \left(\lambda_ {t}, c _ {t k}, k _ {t}, k\right), \tag {7}
$$
Let  $\delta_t = x_t\mathbb{E}[a_t\mid c_{tk_t}]$ . Compute the dual fairness allocation target and update the dual parameter
$$
\gamma_ {t} = \underset {\gamma \in \Delta_ {\delta_ {t}}} {\arg \max } \left\{\langle \lambda_ {t}, \gamma \rangle - \bar {R} (\gamma) \right\}, \tag {8}
$$
$$
\lambda_ {t + 1} = \lambda_ {t} - \eta \left(\gamma_ {t} - \delta_ {t}\right). \tag {9}
$$
An interesting property of the dual gradient descent is that it manages to provide a good fair allocation while updating the source selection parameters simultaneously. Indeed if  $\lambda$  were fixed, then we could solve the source selection part through  $\varphi$  and a bandit algorithm. However, both the  $\lambda_{t}$  and the  $\pi_t$  change over time, which may hint at the necessity of a two-phased algorithm as the combination of these two problems generates non-stationarity for both the dual update and also for the bandit problem. The dual gradient descent manages to combine both updates in a single-phased algorithm.
The different assumptions imply that the decision maker has access to the  $\mathbb{E}[a_t|c_{tk}]$  and to  $\mathbb{E}[u_t\mid c_{tk}]$  for any possible context value  $c_{tk}$ . Such values could be indeed estimated from offline data. The knowledge of such values is sufficient to compute the allocation  $x_{t}$  in (6), the virtual value estimation  $\hat{\varphi}$  of (7), or to compute  $\delta_t$ . Once these values are computed, the only difficulty is to solve (8). In some cases, it might be solved analytically. Otherwise, it can also be solved numerically as it is an (a priori low-dimensional) convex optimization problem. Overall this is an efficient online algorithm which only uses the current algorithm parameters  $\lambda_t,\pi_t$  , and current context  $c_{tk_t}$
We emphasize that Algorithm 1 uses randomization among the different sources of information and does not aim to identify the best source. As shown in Proposition 2.1, this is important because using the best source of information can be strictly less good than using randomization. Moreover, it simplifies the analysis because it convexifies the set of strategies. This means that Sion's minimax theorem can be applied to some dual function, which allows for  $\lambda_{t}$  and  $\pi_t$  to be updated simultaneously. If one would try to target the best static source of information (static-OPT), one would need to determine the optimal dual parameter  $\lambda_{k}$  of each source. This would lead to an algorithm that is both more complicated and less efficient (because of Proposition 2.1).
The following theorem shows that Algorithm 1 has a sub-linear regret of order  $O(\sqrt{T})$ . This regret bound is comparable to those in Balseiro et al. (2021); Celli et al. (2022) but we handle the much more challenging case of having multiple sources of information, and imperfect information about  $a_{t}$  and  $u_{t}$ .
Theorem 3.1. Assume that Algorithm 1 is run with the parameters  $\eta = L / (2\mathrm{Diam}(\Delta)\sqrt{T})$ $m = \bar{u} +L + \max_k|p_k| + 2\eta \operatorname {Diam}(\Delta),\rho = \sqrt{\log(K) / (TKm^2)}$  , and that  $\lambda_0\in \partial R(0)$  , the subgradient of  $R$  at O. Then the expected regret of the algorithm is upper bounded by:
$$
\mathrm {R e g} \leq 2 ((L + \bar {u} + \max _ {k} | p _ {k} |) \sqrt {K \log (K)} + L \sqrt {d} + L \mathrm {D i a m} (\Delta)) \sqrt {T} + 2 L \sqrt {K \log (K)}.
$$
Note that this regret bound is tight: when  $R = 0$  the problem we consider reduces to a  $K$ -armed bandit with bounded rewards  $\mathbb{E}[\max(\mathbb{E}[u_t \mid c_{t,k}], 0) - p_k]$  for arm  $k \in [K]$ , which has a  $\Omega(\sqrt{T})$  regret lower bound (Lattimore & Szepesvári, 2020). It is of the same order in  $T$  as our regret upper bound. Remark that  $R = 0$  implies that  $L = 0$  and we do recover the regret bound for the EXP3 algorithm. Similarly if  $K = 1$  the bandit regret contribution disappears. In our analysis, the regret due to the interaction between the bandit and the online fair allocation is  $L\sqrt{K\log(K)T}$ .
The time-horizon dependent parameters used in Algorithm 1 can be adapted to obtain an anytime algorithm. While using a doubling trick directly for  $\eta$  is not possible as some protected attribute would already be selected when restarting the algorithm, we can use an adaptive learning rate of  $\eta_{t} = \mathcal{O}(1 / \sqrt{t})$ . Indeed due to the boundedness of the  $(\lambda_t)_{t\in T}$  (Lemma 3.2), we can act as if we had a finite diameter for the space of the  $\lambda_{t}$ . This results in a slight increase of regret, the constant term in Theorem 3.1 now scaling in  $\mathcal{O}(\sqrt{T})$ .
3.4 Sketch of proof
As mentioned above, when  $K = 1$  (resp.  $R = 0$ ) the problem reduces to fair online allocation as in Balseiro et al. (2021); Celli et al. (2022) (resp. to multi-armed bandits). The main technical difficulty thus lie in combining algorithms used in these problems. Ideally, we would have access to some optimal dual parameter  $\lambda_{k}^{*}$  before we run the algorithm so that we can simply run a bandit algorithm, which is obviously not possible as the selected source  $k_{t}$  affects the fairness, and the selected parameter  $\lambda_{t}$  affects the arms virtual rewards. In particular it is not clear how the  $\lambda_{t}$  evolve in the worst case while the algorithm is running. Instead we alternate between those primal and dual updates. We thus need to show that this alternation indeed achieves good performance.
We present two important lemmas used in the proof of Theorem 3.1. The first one guarantees that doing a gradient descent with  $\bar{R}$  implies that the dual values  $\lambda_{t}$  remain bounded. This is crucial for EXP3 as it implies that the virtual values  $\varphi$  remain bounded along the path of the  $\lambda_{t}$ .
Lemma 3.2. Let  $\lambda_0\in \mathbb{R}^d$ $\eta >0$  , and an arbitrary sequences  $(\delta_{1},\delta_{2},\ldots)\in \Delta^{\mathbb{N}}$  . Assume that  $\lambda_0\in \partial R(0)$  and define recursively  $\lambda_t$  by Equations (8) and (9). Then for all  $t$  , we have  $\| \lambda_t\| _2\leq L + 2\eta \mathrm{Diam}(\Delta)$
Sketch of Proof. The main idea is to show that the distance of  $\lambda_{t}$  to the reunion of subgradients of  $\bar{R}$  (which is bounded by Lipschitzness property) is a decreasing function in  $t$ . We can show this by using the  $KKT$  conditions of the optimization problem. The convex set over which we optimize is a simple Euclidean ball, because of our modification, centered around the appropriate point hence allowing us to redirect the gradient  $\lambda_{t+1} - \lambda_{t}$  towards this set. Moreover, we add some "security" around this set of the size of the gradient bound to make sure that the  $\lambda_{t}$  remains in this set. The full proof can be found in Appendix D.3.
The second lemma guarantees that having access to the conditional expectation  $\mathbb{E}[a_t|c_{tk}]$  is enough to derive a good algorithm when considering the conditional expectation of the total utility. This way we avoid the computation of the conditional expectation of  $R$ , which would be more difficult.
Lemma 3.3. For  $(x_{1},\ldots ,x_{T})$  and  $(k_{1},\dots ,k_{T})$  generated according to Algorithm 1, with  $\delta_t = x_t\mathbb{E}[a_t\mid c_{tk_t}]$ , we have the following upper bound:
$$
\left| \mathbb {E} [ R (\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} x _ {t}) - R (\frac {1}{T} \sum_ {t = 1} ^ {T} \delta_ {t}) ] \right| \leq 2 L \sqrt {\frac {d}{T}}.
$$
Sketch of proof. We use the Lipschitz property of  $R$  and some inequalities to directly compare the difference of the sums. The variable  $x_{t}$  depends on the past history and needs to be carefully taken into account, through proper conditioning. Finally, we compute the variance of a sum of martingale differences. See proof in Appendix D.4.
Using these two Lemmas, we now give the main ideas of the proof of Theorem 3.1. First, we upper-bound OPT through a dual function involving the convex conjugate  $R^{*}$ , with similar arguments as in Balseiro et al. (2021); Celli et al. (2022). Then we need to lower-bound ALG with this dual function minus the regret. The main difficulty is that the source virtual rewards distribution changes with  $\lambda_{t}$  and so does the average  $\lambda_{t}$  target through  $\pi_{t}$ . The performance of ALG can be decomposed at each step  $t$  into the sum of the virtual reward and a term in  $R^{*}(\lambda_{t})$  encoding the fairness penalty. We deal with the virtual rewards using an adversarial bandits algorithm able to handle any reward sequence using techniques for adversarial bandits algorithm from Hazan (2022); Lattimore & Szepesvári (2020), as the  $\lambda_{t}$  are generated by a quite complicated Markov-Chain. These rewards are bounded because of Lemma 3.2. This yields the two regret terms in  $\sqrt{K\log(K)}$ , the second one stems from the difference between  $\bar{\Delta}$  and  $\Delta$ . For the fairness penalty, using the online gradient descent on the  $\lambda_{t}$  we end up being close to the penalty of the conditional expectations up to the regret term in  $\mathrm{Diam}(\Delta)$ . This last term is close to the true penalty through Lemma 3.3. This provides us with a computable regret bound where all the parameters are known beforehand. The full proof can be found in Appendix E.
3.5 Public contexts
We now go back to the general case with  $|\mathcal{Z}| > 1$  finite. We would like to derive a good algorithm, which also takes into account the public information  $z_{t}$ . Reusing the analogy with bandit problems, this seems to be akin to the contextual bandit problem, where we would simply run the algorithm for each context in parallel. However, this would be incorrect: not only for a fixed public context does the non-separability of  $R$  couples the source selection with the fairness penalty, it also couples all of the public contexts  $z_{t}$  together. Hence the optimal policy is not to take the best policy for each context, which is once again different from the classical bandit setting.
The solution is to run an anytime version of the EXP3 algorithm for each public context in  $\mathcal{Z}$ , but to keep a common  $\lambda_{t}$  for the evaluation of the virtual value. While it is technically not much more difficult and uses similar ideas to what was done for different sources when  $|\mathcal{Z}| = 1$ , the fact that only one of the "block" of the algorithm (the bandit part) needs to be parallelized, is quite specific and surprisingly simple.
Proposition 3.4. For  $\mu$  the probability distribution over  $\mathcal{Z}$  finite, we can derive an algorithm that has a modified regret of order  $\mathcal{O}(\sqrt{TK\log(K)}\sum_{z\in \mathcal{Z}}\sqrt{\mu(z)})$ , where  $\mu (z)$  is the probability that the public attribute is  $z$ .
The algorithm and its analysis are provided in Appendix J.1. Note that in the worst case, when  $\mathcal{Z}$  is finite, one has  $\sum_{z\in \mathcal{Z}}\sqrt{\mu(z)}\leq \sqrt{|\mathcal{Z}|}$  by Jensen's inequality on the square root function.
If  $\mathcal{Z}$  is not finite but a bounded subset of a vector space set of dimension  $r$ , such as  $\mathcal{Z} = [0,1]^r$ , additional assumptions on the smoothness of the conditional expectations are sufficient to obtain a sub-linear regret algorithm:
Proposition 3.5. If the conditional expectations of  $a_{t}$  and  $u_{t}$  are both Lipschitz in  $z$ , then discretizing the space  $\mathcal{Z} = [0,1]^r$  through an  $\epsilon$ -cover and applying the previous algorithm considering that one public context corresponds to one of the discretized bins, we can obtain a regret bound of order  $\mathcal{O}(T^{(r + 1) / (r + 2)})$ .
The proof of this result uses standard discretization arguments under Lipschitz assumptions from Perchet & Rigollet (2013), which can also be found in Chapter 8.2 of Slivkins (2019) or in Exercise 19.5 of Lattimore & Szepesvári (2020). Specificities for this problem, such as these assumptions being enough to guarantee that  $\varphi$  is Lipschitz in  $z$ , can be found in Appendix J.2.
4 Extensions
4.1 Other types of fairness penalty
The fairness penalty term of Equation (1) is quantified as  $TR\left(\sum a_{t}x_{t} / T\right)$ . While this term is the same as the one used in Celli et al. (2022) and can encode various fairness definitions (see the discussion in the aforementioned paper), this does not encompass all possible fairness notions. For instance, one may want to express fairness as a function of  $\sum a_{t}x_{t} / \sum_{t}x_{t}$ , which is the conditional empirical distribution of the user's protected attributes given that they were selected. This would lead to replacing the original penalty term by  $(\sum_{t}x_{t})R(\sum a_{t}x_{t} / \sum_{t}x_{t})$ .
As pointed out in Celli et al. (2022), one possible issue is that  $R(\sum a_{t}x_{t} / \sum_{t}x_{t})$ , in general, is not convex in  $x$ , even if  $R$  is convex. However  $(\sum_{t}x_{t})R(\sum a_{t}x_{t} / \sum_{t}x_{t})$  is the perspective function of  $R(A(.))$  (with  $A$  the matrix with columns the  $a_{t}$ ), which is thus convex. Hence, Algorithm 1 can be adapted to handle this new fairness penalty with two modifications. First, for the bandit part, we run the algorithm with a new virtual reward function, expressed as:
$$
\tilde {\varphi} \left(\lambda_ {t}, c _ {t k}, k\right) = \max  \left(\mathbb {E} \left[ u _ {t} \mid c _ {t k} \right] - \left\langle \lambda_ {t}, \mathbb {E} \left[ a _ {t} \mid c _ {t k} \right] \right\rangle + R ^ {*} \left(\lambda_ {t}\right), 0\right) - p _ {k}.
$$
This leads to an allocation  $x_{t} = 1$  if  $\mathbb{E}[u_t\mid c_{tk_t}] + R^* (\lambda_t)\geq \langle \lambda_t,\mathbb{E}[a_t\mid c_{tk_t}]\rangle$  and  $x_{t} = 0$  otherwise.
Second, we modify the set on which the dual descent is done. The set  $\Delta$  now becomes  $\tilde{\Delta} = \operatorname{Conv}(\mathcal{A})$  (without the union with 0), and we now use  $\tilde{\delta}_t = \mathbb{E}[a_t \mid c_{tk_t}]$  instead of  $\delta_t = x_t\mathbb{E}[a_t \mid c_{tk_t}]$ . Line (8) remains unchanged up to replacing  $\delta$  and  $\Delta$  by  $\tilde{\delta}$  and  $\tilde{\Delta}$ . Finally the dual parameter update now becomes  $\lambda_{t+1} = \lambda_t - \eta x_t(\gamma_t - \tilde{\delta}_t)$ , which means that whenever  $x_t = 0$ ,  $\lambda_t$  does not change.
With these modifications, the following theorem (whose proof is very similar to the one of Theorem 3.1 and detailed in Appendix G), shows that we recover similar regret bounds as previously, with some modified constants due to the presence of  $R^{*}$  in the virtual reward, and the modified  $\Delta$ . This yields a new class of usable fairness penalties which was previously not known to work.
Theorem 4.1. Using Algorithm 1 with the modifications detailed above, the regret with respect to the objective with the modified penalty  $\left(\sum_{t} x_{t}\right) R\left(\sum_{t} a_{t} x_{t} / \sum_{t} x_{t}\right)$  is of order  $\mathcal{O}(\sqrt{T})$ .
4.2 Learning conditional utilities  $\mathbb{E}[u_t\mid c_{tk}]$
To compute the virtual values, the algorithm relies on the knowledge of  $\mathbb{E}[u_t\mid c_{tk}]$  for all possible context values. We now show how to relax this assumption even if the decision maker receives the feedback  $u_{t}$  only when the user is selected  $(x_{t} = 1)$ . We shall make the usual structural assumption of the classical stochastic linear bandit model. The analysis relies on Chapters 19 and 20 of Lattimore & Szeptesvári (2020), and only the main ideas are given here. For simplicity we will assume that there are no public contexts  $(|\mathcal{Z}| = 1)$ .
We assume that the contexts  $c_{tk}$  are now feature vectors of dimension  $q_k$ , and that for all  $k \in [K]$ , there exists some vector  $\psi_k \in \mathbb{R}^{q_k}$  so that
$$
u _ {t} - \left\langle \psi_ {k}, c _ {t k} \right\rangle , \tag {10}
$$
is a zero mean 1-subgaussian random variable conditioned on the previous observation (see a more precise definition in Appendix H). We make classical boundedness assumptions, that for all  $k$ :  $\| \psi_k\| _2\leq \bar{\psi}$ , that  $\| c_{tk}\| _2\leq \bar{c}$ , and that  $\langle \psi_k,c_{tk}\rangle \leq 1$  for all possible values of  $c_{tk}$ .
Intuitively, we aim at running a stochastic linear bandit algorithm on each of the different sources for  $T_{k}$  steps, where  $T_{k}$  is the number of times that source  $k$  is selected, but this breaks because of dependencies among the  $k$  sources. Hence, what we do is to slightly change the rewards and contextual actions, so that we do something akin to artificially running  $K$  bandits in parallel for  $T$  steps. We force a reward and action of 0 for source  $k$  whenever it is not selected, and consider that each source is run for  $T$  steps (even if it is actually selected less than  $T$  times). Thus we can directly leverage and modify the existing analysis for the stochastic linear bandit. The full algorithm is given in Appendix H.
Proposition 4.2. Given the above assumptions, the added regret of having to learn  $\mathbb{E}[u_t\mid c_{tk}]$  is of order  $\mathcal{O}(\sqrt{KT}\log (T))$
If the decision maker knows the optimal combination of sources, she can simply select sources proportionally to this combination, and actually learn the conditional expectation independently for each source. The worst case is to have to pull each arm  $T / K$  times, which then incurs an additional regret with the same  $\sqrt{K}$  constant. This shows that this bound is actually not too wasteful, as the cost of artificially running these bandits algorithm in parallel only impacts the logarithmic term.
We have shown how the problem of optimal data purchase for fair allocations can be tackled, using techniques from online convex optimization and bandits algorithms. We proposed a computationally efficient algorithm yielding a  $\mathcal{O}(\sqrt{T})$  regret algorithm in the most general case. Interestingly, because of the non-separable penalty  $R$ , the benchmark is different from a bandit algorithm, as randomization can strictly improve the performance for some instances, even though the setting is stochastic. We have also presented different types of fairness penalties that we can additionally tackle compared to previous works, in particular in the full information setting, and some instances where assumptions on the decision maker's knowledge can be relaxed.
Throughout the paper (even in Section 4.2 where we relax the assumption that  $\mathbb{E}[u_t\mid c_{tk}]$  is known), we assumed that the decision-maker knows  $\mathbb{E}[a_t\mid c_{tk}]$ . This assumption is reasonable as this is related to demographic data and not to a utility that may be specific to the decision maker. Yet, this assumption can also be relaxed in specific cases. As an example, suppose that the decision maker can pay different prices to receive more or less noisy versions of  $a_{t}$  that correspond to the data sources (the pricing could be related to different levels of Local Differential Privacy for the users, see Dwork & Roth (2014)). Then, under some assumptions,  $\mathbb{E}[a_t\mid c_{tk}]$  can also be learned online—we defer the details to Appendix I.
The works of Balseiro et al. (2021) and Celli et al. (2022) can include hard constraints on some budget consumption when making the allocation decision  $x_{t}$ , which we did not include in our work. If this budget cost is measurable with respect to  $c_{tk}$ , then our analysis does not preclude using similar stopping time arguments as was done in these two works. However if this budget consumption is completely random, our algorithm and analysis can not be directly applied as this budget consumption may give additional information on the  $a_{t}$  that was just observed. Regarding the i.i.d. assumption, in Balseiro et al. (2021) adversarial inputs are also considered, and the same could be done here for the contexts  $c_{t,k}$  with similar results. However some stochasticity is still needed so that  $\mathbb{E}[u_t\mid c_{t,k}]$  is well defined, which leads to an ambiguous stochastic-adversarial model. An open question would be to consider an intermediate case where  $\mathbb{E}[u_t\mid c_{t,k}]$  would depend on  $t$ , and take into account learning for non-stationary distributions.
This work has been partially supported by MIAI @ Grenoble Alpes (ANR-19-P3IA-0003), by the French National Research Agency (ANR) through grant ANR-20-CE23-0007, ANR-19-CE23-0015, and ANR-19-CE23-0026, and from the grant "Investissements d'Avenir" (LabEx Ecodec/ANR-11-LABX-0047).