Title: Trading-off price for data quality to achieve fair online allocation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Fairness

Online allocation

Bandits algorithms
Abstract: We consider the problem of online allocation subject to a long-term fairness penalty. Contrary to existing works, however, we do not assume that the decision-maker observes the protected attributes---which is often unrealistic in practice. Instead they can purchase data that help estimate them from sources of different quality; and hence reduce the fairness penalty at some cost. We model this problem as a multi-armed bandit problem where each arm corresponds to the choice of a data source, coupled with the fair online allocation problem. We propose an algorithm that jointly solves both problems and show that it has a regret bounded by $\mathcal{O}(\sqrt{T})$. A key difficulty is that the rewards received by selecting a source are correlated by the fairness penalty, which leads to a need for randomization (despite a stochastic setting). Our algorithm takes into account contextual information available before the source selection, and can adapt to many different fairness notions.

Trading-off price for data quality to achieve fair online allocation

Abstract

We consider the problem of online allocation subject to a long-term fairness penalty. Contrary to existing works, however, we do not assume that the decision-maker observes the protected attributes-which is often unrealistic in practice. Instead they can purchase data that help estimate them from sources of different quality; and hence reduce the fairness penalty at some cost. We model this problem as a multi-armed bandit problem where each arm corresponds to the choice of a data source, coupled with the online allocation problem. We propose an algorithm that jointly solves both problems and show that it has a regret bounded by O ( √ T ) . A key difficulty is that the rewards received by selecting a source are correlated by the fairness penalty, which leads to a need for randomization (despite a stochastic setting). Our algorithm takes into account contextual information available before the source selection, and can adapt to many different fairness notions. We also show that in some instances, the estimates used can be learned on the fly.

1 Introduction

We consider the problem of online allocation with a long-term fairness penalty: A decision maker interacts sequentially with different types of users with the global objective of maximizing some cumulative rewards (e.g., number of views, clicks, or conversions); but she also has a second objective of taking globally 'fair' decisions with respect to some protected/sensitive attributes such as gender or ethnicity (the exact concepts of fairness will be discussed later). This problem is important because it models a large number of practical online allocation situations where the additional fairness constraint might be crucial. For instance, in online advertising (where the decision maker chooses to which users an ad is shown), some ads are actually positive opportunities such as job offers, and targeted advertising has been shown to be prone to discrimination Lambrecht & Tucker (2019); Speicher et al. (2018); Ali et al. (2019). Those questions also arise in many different fields such as workforce hiring (Dickerson et al., 2018), recommendation systems (Burke, 2017), or placement in refugee settlements (Ahani et al., 2021).

There has been a flourishing trend of research addressing fairness constraints in machine learningsee e.g., (Chouldechova, 2017; Hardt et al., 2016; Kleinberg & Raghavan, 2018; Barocas et al., 2019; Emelianov et al., 2020; Molina & Loiseau, 2022)-and in sequential decision-making problems-see e.g., (Joseph et al., 2016; Jabbari et al., 2017; Heidari & Krause, 2018)-as algorithmic decisions have real consequences on the lives of individuals, with unfortunate observed discrimination (Buolamwini

&Gebru, 2018; Larson et al., 2016; Dastin, 2018). In online allocation problems, general long-term constraints have been studied for instance by Agrawal & Devanur (2015) who maximize the utility of the allocation while ensuring the feasibility of the average allocation, or by Balseiro et al. (2020) who show how to handle hard budget constraints for online allocation problems. More directly related to fairness, Nasr & Tschantz (2020) formulate the problem of fair repeated auctions with a hard constraint on the difference between the number of ads shown to each group. Finally, in recent works, Balseiro et al. (2021); Celli et al. (2022) consider the online allocation problem where a non-separable penalty related to fairness is suffered by the decision maker at the end of the decision-making process instead of a hard constraint-these works are the closest to ours.

All the aforementioned papers, unfortunately, assume that the protected attributes (which define the fairness constraints) are observed before taking decisions. In practice, it is often not the case-for instance to respect users' privacy (Guardian, 2023)-and this makes it challenging to satisfy fairness constraints (Lipton et al., 2018). In online advertising for example, the decision-maker typically has access to some public 'contexts' on each user, from which she could try to infer the value of the attribute; but it was shown that the amount of noise can be prohibitive and therefore ensuring that a campaign reaches a non-discriminatory audience is non-trivial Gelauff et al. (2020).

Our contribution. In this paper, we consider the online allocation problem under long-term fairness penalty, in the practical case where the protected attributes are not observed. Instead, we consider the case where the decision-maker can pay to acquire more precise information on the attributes (beyond the public context), either by directly compensating the user (the more precise the information on the attribute, the higher the price) or by buying additional data to some third parties data-broker. 1 Using this extra information, she should be able to estimate more precisely, and thus sequentially reduces, the unfairness of her decisions while keeping a high cumulative net reward. The main question we aim at answering is how should the decision maker decide when, and from which source (or at what level of precision), to buy additional data in order to make fair optimal allocations?

Compared to the closest existing works (Balseiro et al., 2021; Celli et al., 2022) which study online allocation with a long term fairness penalty, the main novelty in the setting we examine is two-fold: we allow for uncertainty on the attributes of each individual, and more importantly we consider jointly the fair online allocation problem with a source selection problem. Consequently, we present the efficient Algorithm 1 that tackles both of these challenges concurrently. This algorithm combines a dual gradient descent for the fair allocation aspect and a bandit algorithm for the source selection part. The final performance of an algorithm is its net cumulative utility (rewards minus costs of buying extra information) penalized by its long-term unfairness; that is quantified by the 'regret': the difference between this performance and the one of some benchmarking 'optimal' algorithm. We show that Algorithm 1 has a sub-linear regret bound under some stochastic assumptions. Notably, the performance achieved by Algorithm 1 using randomized source selection is strictly better than when using a single fixed source, because of the interaction through the fairness penalty-a key difference with standard bandit settings. On a more technical level we show how one can model the randomness and estimates for the protected attributes, how to bound the fair dual parameters which is crucial in order to use adversarial bandit techniques, and how to combine the analysis of the primal and dual steps of the algorithm.

There are many different definitions of group fairness that can be studied (e.g., demographic parity, equal opportunity, etc.). Instead of focusing on a specific one, we consider a generic formulation that can be instantiated to handle most of those different concepts (see Section 2.1 and Appendix A). We also discuss in Section 4.1 how to adapt our algorithm to different fairness penalties. This gives a higher level of generality to our results compared to existing approaches that can handle fewer fairness criteria (e.g., Celli et al. (2022)).

For the sake of clarity, we expose our key results in a simple setting. In particular, we assume binary decisions and a linear utility, and we assume that the expected utility conditional on the context is known. All these assumptions can be relaxed. In particular, we can learn the utilities (see Section 4.2). We can handle also more general decision variables and utility forms. This allows in particular to tackle problems of matching and auctions (albeit with some restrictions), we discuss that in Appendices B and C.

Related work. The problem of online fair allocation is closely related to online optimization problems with constraints, which is studied in a few papers. For instance, bandit problems with knapsack constraints where the algorithm stops once the budget has been depleted have been studied by Badanidiyuru et al. (2018); Agrawal & Devanur (2016). Li & Ye (2021); Li et al. (2022) consider online linear programs, with stochastic and adversarial inputs. Liu et al. (2021) deal with linear bandits and general anytime constraints, which can be instantiated as fairness constraints. More recently Castiglioni et al. (2022a,b) propose online algorithms for long-term constraints in the stochastic and adversarial cases with bandit or full information feedback. Some papers take into account soft long-term constraints (Agrawal & Devanur, 2015; Jenatton et al., 2016), and more recently in (Balseiro et al., 2021; Celli et al., 2022) where the long-term constraint can be instantiated as a fairness penalty-we also adopt a soft constraint. We depart from this literature by considering the case where the protected attributes (based on which fairness is defined) is not observed. We consider the case where the decision-maker can buy additional information to estimate it (which adds considerable technical complexity), but even in the case where no additional information can be bought our work extends that of (Balseiro et al., 2021; Celli et al., 2022).

As mentioned above, the fairness literature usually assumes perfect observation of the protected attributes yet noisy realizations of protected attributes or limited access to them to measure and compute fair machine learning models has also been considered (Lamy et al., 2019; Celis et al., 2021; Zhao et al., 2022). In some cases, the noise may come from privacy requirements, and the interaction between those two notions has been studied (Jagielski et al., 2019; Chang & Shokri, 2021), see (Fioretto et al., 2022) for a survey. There are also works on data acquisition, which is similar to purchasing information from different sources of information; e.g., Chen et al. (2018); Chen & Zheng (2019) study mechanisms to acquire data so as to estimate some of the population's statistics. They use a mechanism design approach where the cost of data is unknown and do not consider fairness (or protected attributes). However, none of these approaches can handle sequential decision problems with fairness constraints or penalties (and choosing information sources).

2 Preliminaries

2.1 Model and assumptions

We present here a simpler model, and later-on discuss possible extensions. Consider a decision maker making sequential allocation decisions for a known number of T users (or simply stages) in order to maximize her cumulative rewards. The user t has some protected attributes a t ∈ A ⊂ R d , that is not observed before taking a decision x t . On the other hand, the decision-maker first observes some public context z t ∈ Z , where Z is the finite set of all possible public contexts and she has the possibility to buy additional information. There are K different sources for additional information and choosing the source k t ∈ [ K ] has a cost of p k t , but it provides a new piece of information, which together with the public information is summarized in the random variable c tk t = ( z t , data from k t ). Based on this, the decision x t ∈ { 0 , 1 } can be made; this corresponds to include, or not, user t in the cohort (for instance, to display an ad or not). Including user t generates some reward/utility u t , which might be unknown to the decision maker (as it may depend on the private attribute), but can be estimated using the different contexts.

To fix ideas, we show how this model applies to two examples: Imagine an advertiser aiming to display ads to an user, able to see some bare-bone information through cookies, such as which website was previously visited ( z t ). Based on this information, they can decide whether or not to buy additional information ( c tk ) from different data brokers that collect user activity. For example, if they observe that the user has browsed clothing stores, they might opt to acquire data containing purchase details from this website. This enables them to estimate the user's gender ( a t ) based on the type of clothing bought. Now consider the problem of fairly relocating refugees to different cities. When the organization in charge of resettlement receives a resettlement case ( z t ), it can either decide to directly assign the refugee to a specific city, or to conduct an additional costly investigation (which might involve a third party watch-dog) to get more information ( c tk ) on some protected attributes of interest such as wealth or age ( a t ), which might have been intentionally misreported.

We assume that the global objective is to maximize the sum of three terms: the cumulative rewards of all selected users, minus the costs of the additional information bought, minus some unfairness penalty, represented by some function R ( . ) . Denoting by k = ( k 1 , · · · , k T ) and x = ( x 1 , · · · , x T )

the sources and allocations selected during the T rounds, the total utility of the decision maker is then

The penalty function R ( . ) is a convex penalty function that measures the fairness cost of the decisionmaking process, due to the unbalancedness of the selected users at the end of the T rounds. It can be used to represent statistical parity (Kamishima et al., 2011), as a measure of how far the allocation is from this fairness notion. This fairness penalty R ( . ) is also used in Balseiro et al. (2021); Celli et al. (2022). In fact, the objective (1) is equal the one used in these papers minus the cost of additional information bought, ∑ t p k t .

Knowns, unknowns and stochasticity We assume that users are independent and identically distributed (i.i.d.), in the sense that the whole vectors ( z t , u t , a t , c t 1 . . . c tK ) are i.i.d., drawn from some underlying unknown probability distribution. While this may be a strong assumption, some applications such as online advertising correspond to large T but to a short real time-frame, hence incurring very little variation in the underlying distribution. The prices p k and the penalty function R ( . ) are known beforehand. As mentioned several times, the only feedback received is c tk , after selecting source k , and this should be enough to estimate u t and a t . We therefore assume that the conditional expectations E [ u t | c tk t ] and E [ a t | c tk t ] are known. The rationale behind this assumption is that these conditional expectations have been learned from past data.

Penalty examples and generalizations A typical example for a t is the case of one-hot encoding: there are d protected categories of users and a t indicates the category of user t . For simplicity, assume that d = 2 , then the quantity ∑ t a ti x t is the number of users of type i ∈ { 1 , 2 } that have been selected. The choice of TR ( ∑ T t =1 a t x t /T ) = | ∑ t a t 1 x t -∑ t a t 2 x t | amounts to penalizing the decision maker proportional to the absolute difference of users in both groups. This generic setting can also model other notions of fairness, such as Equality of Opportunity (Hardt et al., 2016), by choosing other values for a t and R ( . ) , see examples and discussion in Appendix A.

Similarly, the choice of x t ∈ { 0 , 1 } can be immediately generalized to any finite decision set or even continuous compact one (say, the reward at stage t would then be u ⊤ t x t for some vector u t ), which makes it possible to handle problems such as bipartite matching. Instead of deriving a linear utility from selecting an user, general bounded upper semi-continuous (u.s.c.) utility functions can also be treated, and can be used to instantiate auctions mechanism (with some limitations detailed in Appendix C). We also explain how to relax the assumption that E [ u t | c tk t ] is known in Section 4.2, by deriving an algorithm that actually learns it in an online fashion, following linear contextual bandit techniques (Abbasi-yadkori et al., 2011).

Weshowin Section 3.5 that the assumption that Z is finite can be relaxed if all conditional distributions depend smoothly on z , following techniques from Perchet & Rigollet (2013).

Mathematical assumptions We shall assume | u t | ≤ ¯ u , for all t , and that ∥ a ∥ 2 ≤ 1 for all a ∈ A . We make, for now, no structural assumption on the variables c tk . We mention here that the decision maker has to choose a single source at each stage, but this is obviously without loss of generality (by adding void or combination of sources).

We define ∆ = Conv( A∪{ 0 } ) , where Conv is the closed convex hull of a set. Since A is compact, the set ∆ is also convex and compact. The penalty function R : ∆ → R is a proper closed convex function that is L -Lipschitz continuous for the Euclidean norm ∥·∥ 2 . While the convexity assumption is pretty usual, the Lipschitzness assumption is rather mild as ∆ is convex and compact. Nevertheless, interesting non-Lipschitz functions, such as the Kullback-Leibler divergence, can be modified to respect these assumptions (see Celli et al. (2022)).

2.2 Benchmark and regret

A usual measure of performance for online algorithms is the regret that compares the utility obtained by an online allocation to the one obtained by an oracle that knows all parameters of the problem, yet not the realized sequence of private attributes. We denote the performance of this oracle by OPT:

where the conditional expectation indicates that the oracle first chooses a contextual policy h t for all users that specifies which source k t = h t ( z t ) to select as a function of the variables z t . It then observes all contexts c tk t and makes for all t the decisions x t based on that.

Denoting ALG the expected penalized utility of an online algorithm, its expected regret is:

We remark that the benchmark of Equation (2) allows choosing different sources of information for different users with the same public information z t . As such, it differs from classical benchmarks in the contextual bandit literature that compare the performance of an algorithm to the best static choice of arm per context and whose performance would be

where the h ∈ [ K ] Z policy that maps the public information to a source selection is the same for all users.

The benchmark (4) is the typical benchmark in contextual multi-armed bandits, as the global impact of decisions at different epochs and for different contexts are independent. However, this is no longer the case with the unfairness penalty R ( . ) that requires coupling all decisions:

Proposition 2.1. There exist an instance of the problem and a constant b > 0 such that for all T :

This result shows that an algorithm that only tries to identify the best source will have a linear regret compared to OPT. This indicates that the problem of source selection and fairness are strongly coupled, even without public information available, and cannot be solved through some sort of two-phase algorithm where each problem is solved separately. The proof of this result is presented in Appendix F.1. In this paper, our primary emphasis lies in the examination of the performance disparity between an online algorithm and the offline optimum. Nevertheless, we provide supplementary experiments in Appendix F.3 that investigate how variations in the prices p k and the penalty R impact the solution of the offline optimum.

3 Algorithm and Regret Bounds

In this part, we present our online allocation algorithm and its regret bound. For clarity of exposition, we first present the algorithm in the case |Z| = 1 , i.e., without public information available (and thus we remove z t from the algorithm). The extension to |Z| > 1 uses similar arguments and is discussed in Section 3.5.

3.1 Overview of the algorithm

Algorithm 1 devised to solve this problem is composed of two parts: a bandit algorithm for the source selection, and a gradient descent to adjust the penalty regularization term. This requires a dual parameter λ t ∈ R d that is used to perform the source selection as well as the allocation decision x t .

The intuition is the following: the performance of each source is evaluated through some 'dual value' for a given dual parameter λ . The optimal primal performance is equal to the dual value when it is minimized in λ , because R is convex and randomized combinations of sources is allowed thus there is no duality gap. Hence the dual value of each source is iteratively evaluated in order to select the best source, and simultaneously minimize the dual value of the selected source through λ , so that the source selected is indeed optimal.

Bandit part. For the source selection, we use the EXP3 algorithm (see Chapter 11 of Lattimore & Szepesvári (2020)) on a virtual reward that depends on the dual parameter. Given a dual parameter λ ∈ R d and a context c tk , we define the virtual reward as

where ⟨ λ t , E [ a t | c tk ] ⟩ denotes the scalar product between λ t and E [ a t | c tk ] . To compute this expectation, one needs to know the quantities p k , E [ u t | c tk ] and E [ a t | c tk ] .

To apply EXP3, a key property is to ensure that the virtual rewards are bounded, which requires λ t to remain bounded. As we show in Lemma 3.2, this is actually guaranteed by the design of the gradient descent on λ . This lemma implies that there exists m ∈ R + such that | φ ( λ t , c tk ) | ≤ m for all t and k . Let us denote by π tk the probability that source k is chosen at time t and k t ∼ π t . We define the importance-weighted unbiased estimator vector ˆ φ ( λ, c t ) ∈ R K where each coordinate k ′ ∈ [ K ] is:

Using this unbiased estimator, we can apply the EXP 3 algorithm to this virtual reward function.

Gradient descent part. Once the source k t for user t is chosen and c tk t is observed, we can compute the decision x t (see (6)). This x t is then used in (8)-(9) to perform a dual descent step on the multiplier λ t . Although using a dual descent step is classical, our implementation is different because we need to guarantee that the values of λ t remain bounded for EXP3. To do so, we modify the geometry of the convex optimization sub-problem by considering a set of allocation targets larger than the original ∆ . For δ ∈ ∆ , we define the set ∆ δ as the ball of center δ and radius Diam(∆) . This ball contains ∆ : ∆ ⊂ ∆ δ .

Algorithm 1 uses any extension of R to ¯ ∆ = ∪ δ ∈ ∆ ∆ δ that is convex and Lipschitz-continuous, for instance, the following one (see Lemma D.2):

that has the same Lipschitz-constant L as R (which is the best Lipschitz-constant possible).

3.2 Algorithm and implementation

Combining these different ideas leads to Algorithm 1. This algorithm maintains a dual parameter λ t that encodes the history of unfairness that ensures that the fairness penalty R ( . ) is taken into account. This dual parameter λ t is used in (6) to compute the allocation x t and in (7) to choose the source of information. The dual update (8)-(9) guarantees that we take R ( . ) into account.

Algorithm 1 Online Fair Allocation with Source Selection

Input: Initial dual parameter λ 0 , step sizes η and ρ , cumulative estimated rewards S 0 = 0 ∈ R K . for t ∈ [ T ] do

Draw a source k t ∼ π t where π tk = exp( ρS t -1 ,k ) / ∑ K l =1 exp( ρS t -1 ,l ) , and observe c tk t . Compute the allocation for user t :

Update the estimated rewards sum and sources distributions for all k ∈ [ K ] :

Let δ t = x t E [ a t | c tk t ] . Compute the dual fairness allocation target and update the dual parameter

end for

An interesting property of the dual gradient descent is that it manages to provide a good fair allocation while updating the source selection parameters simultaneously. Indeed if λ were fixed, then we could solve the source selection part through φ and a bandit algorithm. However, both the λ t and the π t change over time, which may hint at the necessity of a two-phased algorithm as the combination of these two problems generates non-stationarity for both the dual update and also for the bandit problem. The dual gradient descent manages to combine both updates in a single-phased algorithm.

The different assumptions imply that the decision maker has access to the E [ a t | c tk ] and to E [ u t | c tk ] for any possible context value c tk . Such values could be indeed estimated from offline data. The knowledge of such values is sufficient to compute the allocation x t in (6), the virtual value estimation ˆ φ of (7), or to compute δ t . Once these values are computed, the only difficulty is to solve (8). In some cases, it might be solved analytically. Otherwise, it can also be solved numerically as it is an (a priori low-dimensional) convex optimization problem. Overall this is an efficient online algorithm which only uses the current algorithm parameters λ t , π t , and current context c tk t .

3.3 Regret bound

We emphasize that Algorithm 1 uses randomization among the different sources of information and does not aim to identify the best source. As shown in Proposition 2.1, this is important because using the best source of information can be strictly less good that using randomization. Moreover, it simplifies the analysis because it convexifies the set of strategies. This means that Sion's minimax theorem can be applied to some dual function, which allows for λ t and π t to be updated simultaneously. If one would try to target the best static source of information (static-OPT), one would need to determine the optimal dual parameter λ k of each source. This would lead to an algorithm that is both more complicated and less efficient (because of Proposition 2.1).

√

Theorem 3.1. Assume that Algorithm 1 is run with the parameters η = L/ (2 Diam(∆) T ) , m = ¯ u + L + max k | p k | + 2 η Diam(∆) , ρ = √ log( K ) / ( TKm 2 ) , and that λ 0 ∈ ∂R (0) , the subgradient of R at 0 . Then the expected regret of the algorithm is upper bounded by: √ √

The following theorem shows that Algorithm 1 has a sub-linear regret of order O ( T ) . This regret bound is comparable to those in Balseiro et al. (2021); Celli et al. (2022) but we handle the much more challenging case of having multiple sources of information, and imperfect information about a t and u t . √

Note that this regret bound is tight: when R = 0 the problem we consider reduces to a K -armed bandit with bounded rewards E [max( E [ u t | c t,k ] , 0) -p k ] for arm k ∈ [ K ] , which has a Ω( √ T ) regret lower bound (Lattimore & Szepesvári, 2020). It is of the same order in T as our regret upper bound. Remark that R = 0 implies that L = 0 and we do recover the regret bound for the EXP 3 algorithm. Similarly if K = 1 the bandit regret contribution disappears. In our analysis, the regret due to the interaction between the bandit and the online fair allocation is L √ K log( K ) T .

The time-horizon dependent parameters used in Algorithm 1 can be adapted to obtain an anytime algorithm. While using a doubling trick directly for η is not possible as some protected attribute would already be selected when restarting the algorithm, we can use an adaptive learning rate of η t = O (1 / √ t ) . Indeed due to the boundedness of the ( λ t ) t ∈ T (Lemma 3.2), we can act as if we had a finite diameter for the space of the λ t . This results in a slight increase of regret, the constant term in Theorem 3.1 now scaling in O ( √ T ) .

3.4 Sketch of proof

As mentioned above, when K = 1 (resp. R = 0 ) the problem reduces to fair online allocation as in Balseiro et al. (2021); Celli et al. (2022) (resp. to multi-armed bandits). The main technical difficulty thus lie in combining algorithms used in these problems. Ideally, we would have access to some optimal dual parameter λ ∗ k before we run the algorithm so that we can simply run a bandit algorithm, which is obviously not possible as the selected source k t affects the fairness, and the selected parameter λ t affects the arms virtual rewards. In particular it is not clear how the λ t evolve in the worst case while the algorithm is running. Instead we alternate between those primal and dual updates. We thus need to show that this alternation indeed achieves good performance.

We present two important lemmas used in the proof of Theorem 3.1. The first one guarantees that doing a gradient descent with ¯ R implies that the dual values λ t remain bounded. This is crucial for EXP3 as it implies that the virtual values φ remain bounded along the path of the λ t .

Lemma 3.2. Let λ 0 ∈ R d , η > 0 , and an arbitrary sequences ( δ 1 , δ 2 , . . . ) ∈ ∆ N . Assume that λ 0 ∈ ∂R (0) and define recursively λ t by Equations (8) and (9) . Then for all t , we have ∥ λ t ∥ 2 ≤ L +2 η Diam(∆) .

Sketch of Proof. The main idea is to show that the distance of λ t to the reunion of subgradients of ¯ R (which is bounded by Lipschitzness property) is a decreasing function in t . We can show this by using the KKT conditions of the optimization problem. The convex set over which we optimize is a simple Euclidean ball, because of our modification, centered around the appropriate point hence allowing us to redirect the gradient λ t +1 -λ t towards this set. Moreover, we add some 'security' around this set of the size of the gradient bound to make sure that the λ t remains in this set. The full proof can be found in Appendix D.3.

The second lemma guarantees that having access to the conditional expectation E [ a t | c tk ] is enough to derive a good algorithm when considering the conditional expectation of the total utility. This way we avoid the computation of the conditional expectation of R , which would be more difficult.

Lemma 3.3. For ( x 1 , . . . , x T ) and ( k 1 , . . . , k T ) generated according to Algorithm 1, with δ t = x t E [ a t | c tk t ] , we have the following upper bound:

Sketch of proof. We use the Lipschitz property of R and some inequalities to directly compare the difference of the sums. The variable x t depends on the past history and needs to be carefully taken into account, through proper conditioning. Finally, we compute the variance of a sum of martingale differences. See proof in Appendix D.4.

Using these two Lemmas, we now give the main ideas of the proof of Theorem 3.1. First, we upperbound OPT through a dual function involving the convex conjugate R ∗ , with similar arguments as in Balseiro et al. (2021); Celli et al. (2022). Then we need to lower-bound ALG with this dual function minus the regret. The main difficulty is that the source virtual rewards distribution changes with λ t , and so does the average λ t target through π t . The performance of ALG can be decomposed at each step t into the sum of the virtual reward and a term in R ∗ ( λ t ) encoding the fairness penalty. We deal with the virtual rewards using an adversarial bandits algorithm able to handle any reward sequence using techniques for adversarial bandits algorithm from Hazan (2022); Lattimore & Szepesvári (2020), as the λ t are generated by a quite complicated Markov-Chain. These rewards are bounded because of Lemma 3.2. This yields the two regret terms in √ K log( K ) , the second one stems from the difference between ¯ ∆ and ∆ . For the fairness penalty, using the online gradient descent on the λ t we end up being close to the penalty of the conditional expectations up to the regret term in Diam(∆) . This last term is close to the true penalty through Lemma 3.3. This provides us with a computable regret bound where all the parameters are known beforehand. The full proof can be found in Appendix E.

3.5 Public contexts

We now go back to the general case with |Z| > 1 finite. We would like to derive a good algorithm, which also takes into account the public information z t . Reusing the analogy with bandit problems, this seems to be akin to the contextual bandit problem, where we would simply run the algorithm for each context in parallel. However, this would be incorrect: not only for a fixed public context does the non-separability of R couples the source selection with the fairness penalty, it also couples all of the public contexts z t together. Hence the optimal policy is not to take the best policy for each context, which is once again different from the classical bandit setting.

The solution is to run an anytime version of the EXP3 algorithm for each public context in Z , but to keep a common λ t for the evaluation of the virtual value. While it is technically not much more difficult and uses similar ideas to what was done for different sources when |Z| = 1 , the fact that only one of the "block" of the algorithm (the bandit part) needs to be parallelized, is quite specific and surprisingly simple.

Proposition 3.4. For µ the probability distribution over Z finite, we can derive an algorithm that has a modified regret of order O ( √ TK log( K ) ∑ z ∈Z √ µ ( z )) , where µ ( z ) is the probability that the public attribute is z .

The algorithm and its analysis are provided in Appendix J.1. Note that in the worst case, when Z is finite, one has ∑ z ∈Z √ µ ( z ) ≤ √ |Z| by Jensen's inequality on the square root function.

If Z is not finite but a bounded subset of a vector space set of dimension r , such as Z = [0 , 1] r , additional assumptions on the smoothness of the conditional expectations are sufficient to obtain a sub-linear regret algorithm:

Proposition 3.5. If the conditional expectations of a t and u t are both Lipschitz in z , then discretizing the space Z = [0 , 1] r through an ϵ -cover and applying the previous algorithm considering that one public context corresponds to one of the discretized bins, we can obtain a regret bound of order O ( T ( r +1) / ( r +2) ) .

The proof of this result uses standard discretization arguments under Lipschitz assumptions from Perchet & Rigollet (2013), which can also be found in Chapter 8.2 of Slivkins (2019) or in Exercise 19.5 of Lattimore & Szepesvári (2020). Specificities for this problem, such as these assumptions being enough to guarantee that φ is Lipschitz in z , can be found in Appendix J.2.

4 Extensions

4.1 Other types of fairness penalty

The fairness penalty term of Equation (1) is quantified as TR ( ∑ a t x t /T ) . While this term is the same as the one used in Celli et al. (2022) and can encode various fairness definitions (see the discussion in the aforementioned paper), this does not encompass all possible fairness notions. For instance, one may want to express fairness as a function of ∑ a t x t / ∑ t x t , which is the conditional empirical distribution of the user's protected attributes given that they were selected. This would lead to replacing the original penalty term by ( ∑ t x t ) R ( ∑ a t x t / ∑ t x t ) .

As pointed out in Celli et al. (2022), one possible issue is that R ( ∑ a t x t / ∑ t x t ) , in general, is not convex in x , even if R is convex. However ( ∑ t x t ) R ( ∑ a t x t / ∑ t x t ) is the perspective function of R ( A ( . )) (with A the matrix with columns the a t ), which is thus convex. Hence, Algorithm 1 can be adapted to handle this new fairness penalty with two modifications. First, for the bandit part, we run the algorithm with a new virtual reward function, expressed as:

This leads to an allocation x t = 1 if E [ u t | c tk t ] + R ∗ ( λ t ) ≥ ⟨ λ t , E [ a t | c tk t ] ⟩ and x t = 0 otherwise.

Second, we modify the set on which the dual descent is done. The set ∆ now becomes ˜ ∆ = Conv( A ) (without the union with 0 ), and we now use ˜ δ t = E [ a t | c tk t ] instead of δ t = x t E [ a t | c tk t ] . Line (8) remains unchanged up to replacing δ and ∆ by ˜ δ and ˜ ∆ . Finally the dual parameter update now becomes λ t +1 = λ t -ηx t ( γ t -˜ δ t ) , which means that whenever x t = 0 , λ t does not change.

With these modifications, the following theorem (whose proof is very similar to the one of Theorem 3.1 and detailed in Appendix G), shows that we recover similar regret bounds as previously, with some modified constants due to the presence of R ∗ in the virtual reward, and the modified ∆ . This yields a new class of usable fairness penalties which was previously not known to work.

Theorem 4.1. Using Algorithm 1 with the modifications detailed above, the regret with respect to the objective with the modified penalty ( ∑ t x t ) R ( ∑ t a t x t / ∑ t x t ) is of order O ( √ T ) .

4.2 Learning conditional utilities E [ u t | c tk ]

To compute the virtual values, the algorithm relies on the knowledge of E [ u t | c tk ] for all possible context values. We now show how to relax this assumption even if the decision maker receives the feedback u t only when the user is selected ( x t = 1 ). We shall make the usual structural assumption of the classical stochastic linear bandit model. The analysis relies on Chapters 19 and 20 of Lattimore &Szepesvári (2020), and only the main ideas are given here. For simplicity we will assume that there are no public contexts ( |Z| = 1) .

We assume that the contexts c tk are now feature vectors of dimension q k , and that for all k ∈ [ K ] , there exists some vector ψ k ∈ R q k so that

is a zero mean 1 -subgaussian random variable conditioned on the previous observation (see a more precise definition in Appendix H). We make classical boundedness assumptions, that for all k : ∥ ψ k ∥ 2 ≤ ¯ ψ , that ∥ c tk ∥ 2 ≤ ¯ c , and that ⟨ ψ k , c tk ⟩ ≤ 1 for all possible values of c tk .

Intuitively, we aim at running a stochastic linear bandit algorithm on each of the different sources for T k steps, where T k is the number of times that source k is selected, but this breaks because of dependencies among the k sources. Hence, what we do is to slightly change the rewards and contextual actions, so that we do something akin to artificially running K bandits in parallel for T steps. We force a reward and action of 0 for source k whenever it is not selected, and consider that each source is run for T steps (even if it is actually selected less than T times). Thus we can directly leverage and modify the existing analysis for the stochastic linear bandit. The full algorithm is given in Appendix H.

Proposition 4.2. Given the above assumptions, the added regret of having to learn E [ u t | c tk ] is of order O ( √ KT log( T )) .

If the decision maker knows the optimal combination of sources, she can simply select sources proportionally to this combination, and actually learn the conditional expectation independently for each source. The worst case is to have to pull each arm T/K times, which then incurs an additional regret with the same √ K constant. This shows that this bound is actually not too wasteful, as the cost of artificially running these bandits algorithm in parallel only impacts the logarithmic term.

5 Conclusion

We have shown how the problem of optimal data purchase for fair allocations can be tackled, using techniques from online convex optimization and bandits algorithms. We proposed a computationally efficient algorithm yielding a O ( √ T ) regret algorithm in the most general case. Interestingly, because of the non separable penalty R , the benchmark is different from a bandit algorithm, as randomization can strictly improve the performance for some instances, even though the setting is stochastic. We have also presented different types of fairness penalties that we can additionally tackle compared to previous works, in particular in the full information setting, and some instances where assumptions on the decision maker's knowledge can be relaxed.

Throughout the paper (even in Section 4.2 where we relax the assumption that E [ u t | c tk ] is known), we assumed that the decision-maker knows E [ a t | c tk ] . This assumption is reasonable as this is related to demographic data and not to a utility that may be specific to the decision maker. Yet, this assumption can also be relaxed in specific cases. As an example, suppose that the decision maker can pay different prices to receive more or less noisy versions of a t that correspond to the data sources (the pricing could be related to different levels of Local Differential Privacy for the users, see Dwork &Roth (2014)). Then, under some assumptions, E [ a t | c tk ] can also be learned online-we defer the details to Appendix I.

The works of Balseiro et al. (2021) and Celli et al. (2022) can include hard constraints on some budget consumption when making the allocation decision x t , which we did not include in our work. If this budget cost is measurable with respect to c tk , then our analysis does not preclude using similar stopping time arguments as was done in these two works. However if this budget consumption is completely random, our algorithm and analysis can not be directly applied as this budget consumption may give additional information on the a t that was just observed. Regarding the i.i.d. assumption, in Balseiro et al. (2021) adversarial inputs are also considered, and the same could be done here for the contexts c t,k with similar results. However some stochasticity is still needed so that E [ u t | c t,k ] is well defined, which leads to an ambiguous stochastic-adversarial model. An open question would be to consider an intermediate case where E [ u t | c t,k ] would depend on t , and take into account learning for non-stationary distributions.

References

Abbasi-yadkori, Y., Pál, D., and Szepesvári, C. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , 2011.

Agrawal, S. and Devanur, N. Linear contextual bandits with knapsacks. In Advances in Neural Information Processing Systems , 2016.

Agrawal, S. and Devanur, N. R. Fast algorithms for online stochastic convex programming. In Proceedings of the 2015 Annual ACM-SIAM Symposium on Discrete Algorithms , 2015.

Ahani, N., Gölz, P., Procaccia, A. D., Teytelboym, A., and Trapp, A. C. Dynamic placement in refugee resettlement. Proceedings of the 22nd ACM Conference on Economics and Computation , 2021.

Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., and Rieke, A. Discrimination through Optimization: How Facebook's Ad Delivery can Lead to Biased Outcomes. In Proceedings of the ACM Conference on Computer Supported Cooperative Work , 2019.

Badanidiyuru, A., Kleinberg, R., and Slivkins, A. Bandits with knapsacks. J. ACM , 2018.

Balseiro, S., Lu, H., and Mirrokni, V. Dual mirror descent for online allocation problems. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Balseiro, S., Lu, H., and Mirrokni, V. Regularized online allocation problems: Fairness and beyond. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Balseiro, S. R., Besbes, O., and Weintraub, G. Y. Repeated auctions with budgets in ad exchanges: Approximations and design. Columbia Business School Research Paper Series , 2014.

Barocas, S., Hardt, M., and Narayanan, A. Fairness and Machine Learning . fairmlbook.org, 2019. URL http://www . fairmlbook . org .

Buolamwini, J. and Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency , 2018.

Burke, R. Multisided fairness for recommendation. In In Proceedings of the Workshop on Fairness, Accountability and Trans- parency in Machine Learning , 2017. Available as arXiv:1707.00093.

Castiglioni, M., Celli, A., and Kroer, C. Online learning with knapsacks: the best of both worlds. In Proceedings of the 39th International Conference on Machine Learning , 2022a.

Castiglioni, M., Celli, A., Marchesi, A., Romano, G., and Gatti, N. A unifying framework for online optimization with long-term constraints, 2022b. Available as Arxiv:2209.07454.

Celis, L. E., Huang, L., Keswani, V., and Vishnoi, N. K. Fair classification with noisy protected attributes: A framework with provable guarantees. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Celli, A., Colini-Baldeschi, R., Kroer, C., and Sodomka, E. The parity ray regularizer for pacing in auction markets. In Proceedings of the ACM Web Conference , 2022.

Cesari, T. and Colomboni, R. A nearest neighbor characterization of lebesgue points in metric measure spaces. Mathematical Statistics and Learning , 3, 2020.

Chang, H. and Shokri, R. On the privacy risks of algorithmic fairness. In IEEE European Symposium on Security and Privacy , 2021.

Chen, Y. and Zheng, S. Prior-free data acquisition for accurate statistical estimation. In Proceedings of the ACM Conference on Economics and Computation , 2019.

Chen, Y., Immorlica, N., Lucier, B., Syrgkanis, V., and Ziani, J. Optimal data acquisition for statistical estimation. In Proceedings of the 2018 ACM Conference on Economics and Computation , 2018.

Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data , 5(2), 2017.

Dastin, J. Amazon scraps secret ai recruiting tool that showed bias against women, 2018. URL https://www . reuters . com/article/us-amazon-com-jobs-automation-insight/ amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-womenidUSKCN1MK08G .

Dickerson, J. P., Sankararaman, K. A., Srinivasan, A., and Xu, P. Balancing relevance and diversity in online bipartite matching via submodularity. In AAAI Conference on Artificial Intelligence , 2018.

Dwork, C. and Roth, A. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. , 2014.

Emelianov, V., Gast, N., Gummadi, K. P., and Loiseau, P. On fair selection in the presence of implicit variance. In Proceedings of the 21st ACM Conference on Economics and Computation , 2020.

Fioretto, F., Tran, C., Van Hentenryck, P., and Zhu, K. Differential privacy and fairness in decisions and learning tasks: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence , 2022. Survey Track.

Flajolet, a. and Jaillet, P. Real-time bidding with side information. In Advances in Neural Information Processing Systems , 2017.

Gelauff, L. L., Goel, A., Munagala, K., and Yandamuri, S. Advertising for demographically fair outcomes. ArXiv , 2020. Available as arXiv:2006.03983.

Guardian. Facebook and instagram to restrict advertisers' access to teenagers' data. The Guardian , 2023. URL https://www . theguardian . com/technology/2023/jan/10/facebook-andinstagram-to-tighten-restrictions-around-teenagers-data .

Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems , 2016.

Hazan, E. Introduction to Online Convex Optimization . The MIT Press, 2nd edition, 2022.

Heidari, H. and Krause, A. Preventing disparate treatment in sequential decision making. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence , 2018.

Iyer, K., Johari, R., and Sundararajan, M. Mean field equilibria of dynamic auctions with learning: A dynamic revenue equivalence theorem. 49th Annual Allerton Conference on Communication, Control, and Computing , 2011.

Jabbari, S., Joseph, M., Kearns, M., Morgenstern, J., and Roth, A. Fairness in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning , 2017.

Jagielski, M., Kearns, M., Mao, J., Oprea, A., Roth, A., Malvajerdi, S. S., and Ullman, J. Differentially private fair learning. In Proceedings of the 36th International Conference on Machine Learning , 2019.

Jenatton, R., Huang, J., Csiba, D., and Archambeau, C. Online optimization and regret guarantees for non-additive long-term constraints, 2016. Available as arXiv:1602.05394.

Joseph, M., Kearns, M., Morgenstern, J., and Roth, A. Fairness in learning: Classic and contextual bandits. In Advances in Neural Information Processing Systems , 2016.

Kamishima, T., Akaho, S., and Sakuma, J. Fairness-aware learning through regularization approach. In IEEE 11th International Conference on Data Mining Workshops , 2011.

Kleinberg, J. and Raghavan, M. Selection problems in the presence of implicit bias. In 9th Innovations in Theoretical Computer Science Conference , 2018.

Lambrecht, A. and Tucker, Catherine, E. Algorithmic bias? an empirical study into apparent gender-based discrimination in the display of stem career ads. In Management Science , 2019.

Lamy, A., Zhong, Z., Menon, A. K., and Verma, N. Noise-tolerant fair classification. In Advances in Neural Information Processing Systems , 2019.

Larson, J., Mattu, S., Kirchner, L., and Angwin, J. How We Analyzed the COMPAS Recidivism Algorithm, 2016. URL https://www . propublica . org/article/how-we-analyzed-thecompas-recidivism-algorithm .

Lattimore, T. and Szepesvári, C. Bandit Algorithms . Cambridge University Press, 2020.

Li, X. and Ye, Y. Online linear programming: Dual convergence, new algorithms, and regret bounds. Operations Research , 2021.

Li, X., Sun, C., and Ye, Y . Simple and fast algorithm for binary integer and online linear programming. Mathematical Programming , 2022.

Lipton, Z., McAuley, J., and Chouldechova, A. Does mitigating ml's impact disparity require treatment disparity? In Advances in Neural Information Processing Systems , 2018.

Liu, X., Li, B., Shi, P., and Ying, L. An efficient pessimistic-optimistic algorithm for stochastic linear bandits with general constraints. In Advances in Neural Information Processing Systems , 2021.

Molina, M. and Loiseau, P. Bounding and approximating intersectional fairness through marginal fairness. In Advances in Neural Information Processing Systems , 2022.

Nasr, M. and Tschantz, M. C. Bidding strategies with gender nondiscrimination constraints for online ad auctions. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , 2020.

Orabona, F. A modern introduction to online learning, 2019. Available as arXiv:1912.13213.

Perchet, V. and Rigollet, P. The multi-armed bandit problem with covariates. The Annals of Statistics , 41(2), 2013.

Rockafellar, R. T. Convex Analysis . Princeton University Press, 1970.

Slivkins, A. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12 (1-2), 2019.

Speicher, T., Ali, M., Venkatadri, G., Ribeiro, F. N., Arvanitakis, G., Benevenuto, F., Gummadi, K. P., Loiseau, P., and Mislove, A. On the potential for discrimination in online targeted advertising. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency , 2018.

Vershynin, R. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.

Zhao, T., Dai, E., Shu, K., and Wang, S. Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , 2022.

A Fairness Penalty

In the main part of the paper, we have considered a penalty with a general form R ( ∑ t a t x t /T ) . Here we give examples of fairness notions that can be encoded through this general penalty.

Statistical parity Notions likes statistical parity are defined as constraints on the independence between protected attributes and selection decisions. More precisely, if X denotes a treatment variable, and A the protected attributes of an individual, we say that X and A are fair with respect to statistical parity if X and A are independent.

If X ∈ X and A ∈ A can take a finite number of values, this can be equivalently rewritten using probability distribution: X satisfies statistical parity if for all ( x, a ) ∈ X × A :

To encode statistical parity in our framework, we consider that the a t are one-hot encoding of the protected attributes ( i.e , a t is a vector with d = |A| dimensions that is equal to 0 everywhere except on the coordinate corresponding to the protected attribute of individual t , that equals 1 ). For the decision variables x t ∈ { 0 , 1 } , ∑ t x t is the number of individual selected and ∑ t ( a t ) i x t is the number of selected individual whose protected attributed is i ∈ [ d ] . Hence, (11) rewrites as: for all i ∈ [ d ] :

Recall that the variables a t are i.i.d. and let us denote by α ∈ P d the probability distribution vector of the a t . For large T , we have ∑ t a t /T ≈ α . Which means that we will say that an allocation vector x satisfies statistical parity if ∑ t a t x t /T = ∑ t αx t /T , which can be equivalently rewritten as

with a ′ t = a t -α .

Penalties R corresponding to statistical parity In practice, achieving strict statistical parity is very constraining, and most of the time some relaxation of this constraint is allowed, at the price of some penalty (Ahani et al., 2021). In our paper, we penalize the non-independence through the function R . There are different choices of penalty function that can be used:

A natural choice can be to penalize as a function of the distance between joint empirical distribution a ′ t x t /T and 0 . For instance, one could use R ( a ′ t x t /T ) = || a ′ t x t /T || β for any given norm || · || and any parameter β > 0 .

It is also possible to measure the non-independence with the empirical distribution of the x t conditioned on a t = a , which should be equal regardless of the protected group a ∈ A . In this case, the empirical distribution conditioned on the protected group i ∈ [ d ] is ∑ t a ti x t / ∑ t a ti ≈ ∑ t a ti x t / ( α i T ) . Similarly we the previous case, R can penalize how far is this vector from 0 .

In the two cases, the fairness penalty can be written as TR ( ∑ t a ′ t x t /T ) , for some modified a ′ t that are not necessarily one-hot encoder. This explains why we choose the set A to be more general than the set of one-hot encoders.

If we want to target another specific proportion for each protected group, we can replace α by any other distribution ν . Note that another possibility is to instead use the empirical distribution of the a t conditioned on the x t . This time the form of the vector is slightly different, it is ∑ t a t x t / ∑ t x t ; this is the type of penalty tackled and discussed in Section 4.1.

Equality of opportunity Another popular fairness notion is the one of equality of opportunity (Hardt et al., 2016), where the probability distribution is further conditioned on another finite random variable y t taking some discrete value in Y that indicates some fitness of the individual. For Y a

random variable with an outcome y ∈ Y considered as a positive opportunity, we say that X , A and Y are fair with respect to equality of opportunity, if

X and A are independent given [ Y = y ] .

In order to adapt this notion in the online allocation setting, is sufficient to consider the cartesian product A ′ = A×Y as an extended new set. Then if we would like to penalize with a metric related to equality of opportunity, we can for instance compose R with the projection over the coordinates corresponding to y t = 1 .

Overall, this shows that the choice of R , the type of fairness vectors, and the choice of A are all crucial when designing a fairness penalty.

B A More General Online Allocation Setting

In the main body of the paper, we choose to simplify the exposition and to restrict ourselves to x t ∈ { 0 , 1 } and to formulate utilities as scalar variables u t . In the following, the regret bounds of Algorithm 1 are obtained in a setting that is more general than the one previously introduced:

We now consider that the allocations decisions x t are in some set X ⊂ R n + . This makes it possible to tackle bipartite matching problems, by letting X = { x ∈ R n + | ∑ n i =1 x i ≤ 1 } be a matching polytope.

Instead of a random scalar u t being drawn, we suppose that a random function f t : X → R is drawn. The functions f t are supposed u.s.c. and are all bounded in absolute value by some ¯ f . This could simply be a deterministic concave function of u t for instance. Remark that when we assume that the conditional expectation can be computed, the decision maker's required knowledge is more demanding, as she needs to be able to compute it for all x ∈ X .

Instead of a random vector, a continuous random function a t : X → A can be drawn, with ∥ a t ( x ) ∥ 2 uniformly bounded for all a t and x , and we define ∆ = Conv( A ) . We suppose that ∥ a t ( x ) ∥ 2 ≤ 1 , which can be done without loss of generality.

The setting of Section 2.1 is recovered by using f t ( x ) = u t x , a t ( x ) = a t x and X = { 0 , 1 } . For the more general setting, we need to redefine the virtual value with f t and a t as:

This virtual reward is well defined by Lemma D.1. We rewrite Algorithm 1 using f t and this new virtual value to obtain Algorithm 2. There are two differences with Algorithm 1: First we use the argmax function in (13) instead of the explicit form of Equation (6); Second we use the virtual value function (12) in Equation (14) instead of the virtual value function of Equation (5).

C Reduction from auctions

In the rest of the paper, we assume that the decision variables are the values x t . In this section, we explain how our setting can be adapted the the case of targeted advertisement, that are usually allocated through auction mechanisms. In the later, the decision variables are the bids but not the x t : the variables x t are a consequence of the bids of the decision maker and of the bids of others.

Model The model that we consider is as follows: for user t , after choosing a source of information k t , the decision maker observes c tk t and decides to bid b t ∈ B . The allocation x t = 1 is given if her bid is higher than the maximum bid of the other bidders. If the bidder wins the auction, it then pays a price that depends on the auction format ( e.g. , b t for first price auctions and the second highest bid for second price auctions). Assuming that the bid of the others are not affected by the decision maker's strategy, the bids of the others are i.i.d. (see Iyer et al. (2011); Balseiro et al. (2014); Flajolet & Jaillet (2017) for a discussion about the stationarity assumption). In this case, one falls back to our model with i.i.d. users, except that the decision variables are the variables b t and we have x t ( b t ) . We can apply our algorithm to this setting, by replacing the set of strategies X by the set of possible bids B , which is also compact. For that, we need to consider a new utility function b ↦→ u t ( b ) x t ( b ) which is a bounded u.s.c. function, and the new protected attributes function b ↦→ a t x t ( b ) . If E [ a t x t ( b ) | c tk ] is continuous in b , this falls under the setting that we described in Appendix B.

Algorithm 2 Online Fair Allocation with Source Selection - General algorithm

Input: Initial dual parameter λ 0 , initial source-selection-distribution π 0 = (1 /K,... , 1 /K ) , dual gradient descent step size η , EXP3 step size ρ , cumulative estimated rewards S 0 = 0 ∈ R K .

Draw a source k t ∼ π t , where ( π t ) k ∝ exp( ρS ( t -1) k ) and observe c tk t . Compute the allocation for user t :

Update the estimated rewards sum and sources distributions for all k ∈ [ K ] :

Compute the expected protected group allocation δ t = E [ a t ( x t ) | c tk t , x t ] and compute the dual protected groups allocation target and update the dual parameter:

end for

Examples of utilities: first-price and second price auctions Let us denote by v t the value of the user t for the decision maker, and by d t the highest bid of the others for this user. The decision maker will win the auction if b t ≥ d t . Hence, the allocations will be x t ( b t ) = ✶ b t >d t . Moreover, the utility function will be:

f t ( b t ) = ( v t -b t ) x t ( b t ) for first price auctions.

f t ( b t ) = ( v t -d t ) x t ( b t ) for second price auctions.

These functions are u.s.c with respect to b t . Moreover, under mild assumption that the distribution of the highest bid of the others d t has a density, the function E [ a t x t ( b ) | c tk t ] is continuous. Hence, our algorithm can be applied.

Difficulty: Knowledge is power To apply the algorithm, the main difficulty is to be able to compute the value b t that maximizes Equation (12). For that it is sufficient for the decision maker to have access to E [ f t ( b t ) | c tk t ] and E [ a t x t ( b t ) | c t,k t ] . This presupposes a lot of distributional knowledge from her part. In the case of full information with a t and v t directly given, Balseiro et al. (2021); Celli et al. (2022) were able to obtain an efficient reduction from online allocations to second price auctions, using the incentive compatible nature of this kind of auction. Similarly, Flajolet & Jaillet (2017) were able to derive efficient bidding strategies using limited knowledge for repeated auctions with budget constraints under some independence assumptions. Translating what is done in these two lines of work to our general model is difficult because we would need to rely on the assumption that a t and d t are independent conditioned on c tk t . While we think that this assumption is not reasonable because different agents might have access to different information sources, and because as the a t is common to all bidders it is correlated with all the bids hence also with d t , we explain in the remark below how assuming independence can lead to a simple algorithm for second price auctions.

Some remarks:

For our model, the first price and second price auctions are equally difficult: The amount of knowledge needed to compute b t and π t is the same for both cases and first price auction is not more difficult than the second-price auction setting (contrary to what usually happens).

During this whole reduction from auctions, we have not assumed that the d t (or x t ) are observed as a feedback (whether the bidder wins the auction or not), and thus they are not assumed to be observed in the benchmark either. This is because observing d t or x t could give us access to information on a t that are not contained in c tk t .

If we do make the assumption that ( u t , a t ) is independent of d t conditionally on all c tk , then we do recover an easy algorithm for second price auctions, and we can allow for a

benchmark taking into account the feedback d t (simply because now this feedback does not give any additional information for the estimation of a t ). Indeed, in that case the optimal bid is b t = E [ v t -⟨ λ t , a t ⟩ | c tk ] . This follows from similar argument made in Flajolet & Jaillet (2017), as for b t some H t measurable bidding strategy we have:

and this last inequality is reached for b t = E [ v t -⟨ λ t , a t ⟩ | c tk ] .

D Technical & Useful Lemmas

In this section, we prove several lemmas that are useful for the proof of the main Theorem 3.1.

D.1 Preliminary lemmas

We first provide some simple lemmas. For completeness, we provide proofs of all results, even if they are almost direct consequences of the definitions.

Lemma D.1. The virtual value function φ is well defined by Equation (12) , as the arg max is non-empty and the maximum is reached.

Proof. We simply need to have that E [ f t ( x ) | c tk ] and -⟨ λ, E [ a t ( x ) | c tk ] ⟩ are u.s.c. in x . The continuity of a t is used to guarantee that -⟨ λ, a t ( x ) ⟩ is u.s.c. regardless of the values of λ . This is immediate from the definition of upper semi-continuity and the Reverse Fatou Lemma. We detail it here for f t for completeness. We say that f t is u.s.c. if and only if

For x ∈ X , let ( x n ) n ∈N be a sequence so that x n → x . Let P be a probability distribution over f t conditionally on c tk . The function | f t ( x n ) | ≤ ¯ f , f t ( x n ) is integrable, we can apply the Reverse Fatou Lemma for conditional expectation. If f t ( x n ) is not positive, we can apply it to f t ( x n ) + ¯ f which is positive. Using the Reverse Fatou Lemma and the upper semi-continuity of f t we obtain:

This means that x ↦→ E [ f t ( x ) | c tk ] is u.s.c. over X .

Because X is compact, and because the maximum of an u.s.c. function is reached over a compact set, the maximum is well defined.

Lemma D.2. Let ¯ R : δ ↦→ inf δ ′ ∈ ∆ { R ( δ ′ ) + L ∥ δ -δ ′ ∥ 2 } . Then ¯ R = R over ∆ , Lip ( R ) = Lip ( ¯ R ) , and ¯ R is convex.

Proof. The function ¯ R is obtained through the explicit formula of the Kirszbraun theorem, which states that we can extend functions over Hilbert spaces with the same Lipschitz constant. Hence the equality over ∆ and the same Lipschitz constant.

It remains to prove the convexity. Note that because R is proper, it is lower semi-continuous, ∥ δ -·∥ 2 as well, and ∆ is compact, hence the inf is reached and it is actually a minimum. Let δ 1 , δ 2 in R d and α ∈ [0 , 1] , and let u 1 and u 2 be the minimums of the optimization problem for respectively δ 1 and δ 2 . Let u = αu 1 +(1 -α ) u 2 . We have u ∈ ∆ because ∆ is convex, and using that R is convex and we have

¯

and taking the inf over ∆ on the left hand-side we obtain the convexity of R .

D.2 Conditional expectation: notation and abuse of notations

In the analysis of the algorithm, we will consider conditional expectation by fixing some of the random variables to specific values. To simplify notations, we will use the following notation: if X and Y are random variables and g is a measurable function, then we denote E X [ g ( X,Y )] the expectation with respect to X , that is:

where X ′ is a random variable independent of Y that has the same law as X . This is equivalent to the random variable obtained by fixing Y and taking the expectation with respect to the law of X only.

In Algorithm 2, we have assumed that the decision maker is able to compute E [ f t ( x t ) | c tk t ] and E [ a t ( x t ) | c tk t ] . While the law of c tk t is complicated (because it involves all past decision up to time t ), what the decision maker needs to compute is the value ζ ( c tk t , x, k t ) , where ζ ( c, x, k ) = E [ f t ( x ) | c tk = c ] . Those two quantities are equal because users are i.i.d. and k t and f t are independent (conditioned on z t and the history).

This manipulation is properly stated by Lemma D.3. Indeed by denoting H t -1 the whole history up to time t -1 , and applying this Lemma: for x t the decision generated by the algorithm which is ( H t -1 , k t , c tk t ) measurable, we have that

It is identical for a t ( x t ) : for ζ ′ ( c, x, k ) = E [ a t ( x ) | c tk = c ] we have the following equality

The following Lemma is a modification of the so-called "Freezing Lemma" (see Lemma 8 of Cesari &Colomboni (2020)) which helps simplifying conditional expectations by allowing you to "fix" the random variable if it is measurable with respect to the conditioning of the expectation:

Lemma D.3. Let X , Y and Z be random variables with ( X,Z ) independent of Y , and g some bounded measurable function. Let us denote by σ ( X ) the sigma-algebra generated by the random variable X . Then

Proof. Let V ( Y, Z ) = E X [ g ( X,Y,Z ) | σ ( Z )] . To show that W is the conditional expectation of g ( X,Y,Z ) given σ ( Y, Z ) , we need to show that V is σ ( Y, Z ) measurable and that for any bounded W that is σ ( Y, Z ) measurable we have

We denote by µ the distribution of ( X,Z ) and ν the distribution of Y . By the Doob-Dynkin Lemma, because W is σ ( Y, Z ) measurable, we have that W = h ( Y, Z ) where h is measurable. First, by independence of Y and ( X,Z ) , we have that the distribution of ( X,Y,Z ) is dν ⊗ dµ . Thus

Now let us compute the other expectation using Fubini theorem as both g and h are bounded:

The random variable h ( y, Z ) is σ ( Z ) measurable, thus because V ( y, Z ) is a conditional expectation given σ ( Z ) , we have

Using Fubini again we obtain

which proves that this is indeed the conditional expectation, as the lemma stated.

D.3 Boundedness of λ t -proof of Lemma 3.2

We next show the proof of the boundedness of the λ t for the sequence generated by our Algorithm 1. Lemma. Let λ 0 ∈ R d , η > 0 , and an arbitrary sequences ( δ 1 , δ 2 , . . . ) ∈ ∆ N . We define recursively λ t as follows:

Then for Λ = ∪ δ ∈ ¯ ∆ ∂ ¯ R ( δ ) we have the following upper bound:

where B (0 , 2 η Diam(∆)) is the ball centered in 0 and of diameter 2 η Diam(∆) , with the diameter and the distance taken with respect to the euclidean L 2 norm.

Proof. Recall that ¯ ∆ = ∪ δ ∈ ∆ ∆ δ , with ∆ δ = { γ : ∥ γ -δ ∥ 2 ≤ Diam(∆) } . Hence, for any δ ∈ ∆ and γ ∈ ¯ ∆ , we have ∥ γ -δ ∥ 2 ≤ 2 Diam(∆) . Let Λ = ∪ δ ∈ ¯ ∆ ∂ ¯ R ( δ ) , and C = Conv(Λ) + B (0 , η 2 Diam(∆)) . Using that ¯ R is L-Lipschitz, we have that Λ is bounded by L for the norm ∥ · ∥ 2 , and so does Conv(Λ) . Thus C is bounded by L +2 η Diam(∆) for ∥ · ∥ 2 . This implies that, to prove the theorem, it suffices to show that Dist( λ t +1 , C ) ≤ Dist( λ t , C ) is true for all t . Indeed, this would imply that

In the remainder of the proof, we show that Dist( λ t +1 , C ) ≤ Dist( λ t , C ) . First, let us remark that γ t is the solution of a convex optimization problem, wit a constraint set ∆ δ t that can be rewritten as a single inequality : h t ( γ ) ≤ 0 , with h t ( γ ) = ∥ γ -δ t ∥ 2 2 -Diam(∆) 2 . The derivative of this function h t () is ∇ h t ( γ ) = 2( γ -δ t ) . As the point δ t is in the interior (assuming that Diam(∆) > 0 ) of ∆ δ t , Slater's conditions hold, and we can apply the KKT conditions at the optimal point ( γ t ). This implies that there exists µ ≥ 0 such that 0 ∈ -λ t + ∂ ¯ R ( γ t ) + µ 2( γ t -δ t ) . This implies that there exists λ δ t ∈ ∂ ¯ R ( γ t ) such that

The norm of the left-hand side of the above equation must be equal to the norm of its right-hand side, which means that ∥ λ δ t -λ t ∥ 2 = 2 µ ∥ δ t -γ t ∥ 2 . This determines the value of µ . Let α t = η ∥ δ t -γ t ∥ 2 / ∥ λ δ t -λ t ∥ 2 , the value λ t +1 is therefore equal to:

We now consider multiple cases:

Case 1 . If α t > 1 , then we rewrite λ t +1 as:

By assumption, the λ δ t ∈ ∂ ¯ R ( γ t ) ⊂ Λ . Moreover, the norm of the second term is bounded by 2 η Diam(∆) because

This implies that λ t +1 ∈ Conv(Λ) + B (0 , 2 η Diam(∆)) = C , and therefore that Dist( λ t +1 , C ) = 0 ≤ Dist( λ t , C ) .

Case 2 . If α t ≤ 1 and λ t ∈ C . In this case, λ t +1 is a convex combination of λ δ t and λ t . As, λ δ t ∈ Λ ⊂ C and λ t ∈ C , and because C is convex, we have that λ t +1 ∈ C .

Case 3 . If α t ≤ 1 and λ t ̸∈ C , we define π t = λ t the projection of λ t on C . Let y t = (1 -α t ) π t + α t λ δ t . Because C is convex, α t ∈ [0 , 1] , λ δ t ∈ C , and π t ∈ C , we have y t ∈ C . This implies that

This shows that for all cases, Dist( λ t +1 , C ) ≤ Dist( λ t , C ) , which concludes the proof.

Using this previous Lemma, we can simply bound the virtual reward functions.

Lemma D.4. For λ 0 ∈ ∂R (0) , and for λ t generated according to Algorithm 1, the virtual reward function

is bounded in absolute value by

Proof. Because a t ( x ) used for the update of λ t is in ∆ for all t , we can apply Lemma 3.2 to bound the trajectory of the ( λ t ) t ∈ [ T ] generated by the algorithm. Using a triangle inequality first, then Cauchy-Schwartz inequality, and finally the boundedness assumptions on f t and a t with the Lemma, we have

Lemma D.5. For λ 0 ∈ ∂R ( a ) (for some a ∈ A ) , and for λ t generated according to Algorithm 1, if the quantity a t ( x ) /x is always well defined and in ∆ for all a t and all x , then the virtual reward function

is bounded in absolute value by

Proof. The proof of this Lemma is identical to the previous one, except that we also need to bound R ∗ . As a convex conjugate, R ∗ is convex. By Corollary 10.1.1 of Rockafellar (1970), every convex function from R d to R is continuous. By Lemma 1 of Celli et al. (2022), we know that the domain of R ∗ is indeed R d , hence it is continuous.

Now let us look at the modified dual update λ t +1 = λ t + η ( γ t x t -a t ( x t )) . We can factorize by x t to recover λ t +1 = λ t + x t η ( γ t -a t ( x t ) /x t ) . Because of our hypothesis, a t ( x t ) /x t ∈ ∆ , therefore we can apply Lemma 3.2 with η t = ηx t , which is upper bounded by η . Hence the λ t are all in { λ ∈ R d | ∥ λ ∥ 2 ≤ L +2 η Diam(∆) } which is compact. By continuity of R ∗ , it is continuous over this compact set hence bounded, and the virtual reward is bounded by ˜ m overall.

Remark. We made the technical assumption that a t ( x ) /x ∈ ∆ . It is clear that when a t is linear, such as a t x , then a t x/x = a t ∈ A ⊂ ∆ and the condition is satisfied. Regardless, if a t ( x ) /x is always well defined and bounded, we can simply increase the size of ∆ , so that this quantity always remain in ∆ . Of course, this will degrade the performance of the algorithm as it depends on Diam(∆) .

D.4 Expected difference between penalty and penalty of expectation - Proof of Lemma 3.3

We next prove the following lemma, which tells us that observing the conditional expectation of the a t ( x t ) is enough to observe the expectation of the penalty R :

Lemma. For ( x 1 , . . . , x T ) and ( k 1 , . . . , k T ) generated according to Algorithm 1, we have the following upper bound:

Proof. Let us denote by δ t = E a t [ a t ( x t ) | c tk t ] the expectation of a t ( x t ) with x t fixed. Using first the triangle inequality for the expectation, and then the Lipschitzness of R we have that

where we used Jensen's inequality with x ↦→ √ x for the last inequality.

We denote by δ t,l the l -th coordinate of δ t for l ∈ [ d ] , and similarly for a t,l . We have:

By Equation (16), the sequence a t,l ( x t ) -δ t,l is a Martingale difference sequence for the filtration σ (( a τ , c τk τ , k τ ) τ ∈ [ t -1] , k t , c tk t ) . Hence, using that the expectation is 0 by the law of total expectation, we obtain that

The terms a t,l ( x t ) -δ t,l are bounded in [ -2 , 2] by the assumption that ∥ a t ( x ) ∥ 2 ≤ 1 , therefore we can bound each of these variances by 4 , for a total bound of 4 T . Plugging this into Equation (17) concludes the proof.

E Proof of the main Theorem 3.1

The proof builds upon the proof used in the full information case ( a t and f t are given) from Balseiro et al. (2021); Celli et al. (2022).

In order to bound the regret, we upper bound the performance of OPT , and lower bound the performance of ALG . We proceed by first upper-bounding the optimum.

Recall that the convex conjugate R ∗ of R (which we consider as a function from R d → R that is + ∞ outside of ∆ ) is a function that associates to a dual parameter λ the quantity R ∗ ( λ ) :

The Fenchel-Moreau theorem, tells us that for a proper convex function, the biconjugate is equal to the original function: R ∗∗ = R .

Recall that φ is defined in Equation (5). We define the dual vector function D ( λ ) = ( D ( λ, 1) , . . . , D ( λ, K )) which is a vector representing the value of the dual conjugate problem. For coordinate k ∈ [ K ] , it is defined as

We denote the unit simplex of dimension K by P K = { π ∈ R K | π k ≥ 0 , ∑ K k =1 π k = 1 }.

Lemma E.1. We have the following upper-bound for the offline optimum:

Proof. We upper bound the performance of OPT through a Lagrangian relaxation using the convex conjugate of R , as was done similarly in Jenatton et al. (2016).

Let ( k 1 , . . . , k T ) ∈ [ K ] T be the (deterministic) sequence of sources selected, and ( x 1 , . . . , x T ) ∈ X T the offline allocation. These x t are all σ ( c 1 k 1 , . . . , c Tk T ) measurable. We define ˜ f t ( x ) = E f t [ f t ( x ) | c tk t ] and δ t = E a t [ a t ( x t ) | c tk t ] .

Using Jensen's inequality for R convex we have

Because the x t are σ ( c 1 k 1 , . . . , c Tk T ) measurable, and by independence of the ( f t , a t , c t 1 , . . . , c tK ) we have that E [ a t ( x t ) | σ ( c 1 k 1 , . . . , c tk T )] = E a t [ a t ( x t ) | c tk t ] = δ t . With the same argument we obtain that E [ f t ( x t ) | σ ( c 1 k 1 , . . . , c tk t )] = ˜ f t ( x t ) . Therefore

We define the function L : ( k , c , x , λ ) ↦→ ∑ T t =1 ˜ f t ( x t ) -p k t -⟨ λ, δ t ⟩ + TR ∗ ( λ ) .

We now show that this function is always greater than the conditional expectation of U :

the last equality results from applying Fenchel-Moreau theorem, and the last inequality is obtained by using Equation (19).

̸

Let us now compute the max in ( x 1 , . . . , x T ) of L . Looking at the terms inside the sum, the only possible dependency of ˜ f t ( x t ) -p k t -⟨ λ, δ t ⟩ to other contexts for t ′ = t , is possible through the allocation x t , which can depend on the other contexts. Therefore the maximum of the sum is the sum of the maximums. This yields that

Thus by taking the max in ( x 1 , . . . , x T ) on both sides of Equation (20) and then taking the expectation (which is over the contexts as everything is measurable with respect to them), we obtain

What is nice here, is that by looking at the dual problem which was originally to address the non-separability in x , we also have the separability in k t .

We denote by T k = ∑ T t =1 ✶ [ k t = k ] the number of times source k is selected, and by ˆ π = ( T 1 /T, . . . , T K /T ) ∈ P K the selection proportions of each source k over the T rounds. This vector of proportions can be seen as the empirical distribution of selected sources. We can rewrite in the following way the dual objective

Finally, using the previous equation with Equation (21), taking first the inf in λ and then the max in k t we can conclude:

Remark. Note that the function D ( λ ) is convex and lower semi-continuous (l.s.c.) (as the supremum of a family of l.s.c. functions). In addition, ⟨ π, D ( λ ) ⟩ is concave and u.s.c. in π by linearity, and P K is compact convex. Hence Sion's minimax theorem can be applied, and the sup in π is actually a max .

We can now proceed to the rest of the proof of the main Theorem 3.1:

Theorem E.2. For η = L/ (2 Diam(∆) √ T ) , m = ¯ f + L + max k | p k | + 2 η Diam(∆) , ρ = √ log( K ) / ( TKm 2 ) , and λ 0 ∈ ∂R (0) , algorithm Algorithm 1 has the following regret upper bound:

Proof. We first lower bound the performance of the algorithm. Then we simply need to apply Lemma E.1 to get the regret bound. Let H t be the natural filtration generated by the ( c τk τ , k τ ) τ ∈ [ t ] . Let x t , k t , λ t and γ t defined as in 1. We denote by δ t = E a t [ a t ( x t ) | c tk t ] . The quantity δ t is in ∆ because of the expectation and by convexity of ∆ . Note that here these are all random variables, with γ t and λ t being H t -1 measurable and x t being H t measurable.

To first get a broad picture on how we are going to lower bound the utility of the algorithm, let us decompose it into different parts. It can be rewritten as

The first line (Equation (22)) corresponds to some adjusted separable rewards, and we bound it by first reducing it to the virtual rewards φ , and then using bandits algorithms techniques. The second line (Equation (23)) corresponds to the difference between the unfairness of the adjusted rewards, and the true unfairness, which is not big because of the gradient descent on λ t . Finally the last line (Equation (24)) corresponds to the difference between the penalty of the expected allocation and the expected penalty of the allocation, which we have already shown in Lemma 3.3 is bounded by O ( √ T ) .

Reduction of Equation (22) to a bandit problem with φ Using first the tower property of the conditional expectation, and then Equation (15), we have that

We add and remove ⟨ λ t , δ t ⟩ , and use that x t is the allocation that yields the virtual reward function:

Using the definition of γ t as a maximizer, we have

Because ∆ ⊂ ∆ δ t and, because ¯ R and R are equal over ∆ we recover a lower bound with R ∗ :

Thus

Application of Bandits Algorithm Let π t be the distribution generated according to the algorithm. We actually have that E [ φ ( λ t , c tk t ) + R ∗ ( λ t )] is just (through independence, measurability, and total expectation arguments) E [ ⟨ π t , D ( λ t ) ⟩ ] . Hence the main idea is to apply Online Convex Optimization theorems to the linear rewards D ( λ t ) with the decision variable π t (see Hazan (2022), Orabona (2019) or Lattimore & Szepesvári (2020) for introductions to these types of problems). Here we don't have access either to D ( λ t ) , nor even to D ( λ t , k t ) : we are in an adversarial bandit setting regarding the λ t , and also in a stochastic setting regarding the expectation in D taken with respect to c t . Hence we use an unbiased estimator of the gradient D ( λ t ) for the linear gain, which we will describe further down.

Notice that it is crucial to be able to deal with any sequence D ( λ t ) , this is because these are neither independent nor related to martingales, as the λ t are generated from a quite complicated Markov Chain with states ( λ t , c t , π t , k t ) . Hence the outside expectation does not help us much, and we will face as losses (or here rewards) the random arbitrary sequence of the D ( λ t ) . We don't care about R ∗ ( λ t ) as it is H t -1 measurable.

Here the different arms of a bandit problem correspond to different sources. Using the tower rule, we have

To use results for adversarial bandits, the bandits rewards need to be bounded. By Lemma D.4, the virtual reward φ , and consequently its conditional expectation, is bounded by m := ¯ f + L + 2 η Diam(∆) + max k | p k | . Therefore E [ φ ( λ t , c tk , k ) |H t -1 ] /m ∈ [ -1 , 1] .

Wenowconstruct an unbiased estimator of E [ φ ( λ t , c tk , k ) |H t -1 ] , which is the following importance weighted estimator:

This estimator is unbiased for the conditional expectation given H t -1 . Indeed because π tk is H t -1 measurable, and because ✶ [ k t = k ] and φ ( λ t , c tk , k ) are independent conditionally on H t -1 we

have:

which is exactly the reward needed.

We can now use the following Theorem 11.2 from Lattimore & Szepesvári (2020):

Theorem (EXP3 Regret Bound) . Let X ∈ [0 , 1] T × K be the rewards of an adversarial bandit, then running EXP3 with with learning rate ρ = √ 2 log( K ) / ( TK ) and denoting k t the arm chosen at time t , achieves the following regret bound:

The fact that we can possibly have negative rewards instead of in [0 , 1] like requested in the theorem, just changes that the loss estimator used as an intermediary is bounded by 2 instead of 1 . Hence by selecting ρ = √ log( K ) /TK/m , and using the importance weighted estimator ˆ φ ( λ t , c t ) we can use the EXP3 Theorem an apply the expectation to obtain the following regret bound:

We know that any convex combination of the sum of the rewards of multiple sources is smaller than the sum of the rewards for the best source. Let us denote by ( π n ) n ∈ N the sequence of distributions that maximizes the dual objective of Lemma E.1. We will use π n as a convex combination, and taking the expectation over the regret bound we obtain:

Moreover using that c tk is independent of H t -1 and that λ t is H t -1 measurable we can deduce using the freezing Lemma that

Hence

Taking the limit in n →∞ and by definition of π n we conclude that

Using the above regret bound and Equation (25) we obtain

Analysis of Equation (23) and dual gradient descent We will now compare TR ( ∑ T t =1 δ t /T ) and ∑ T t =1 ¯ R ( γ t ) . Here we consider a modified version of ¯ R , which is equal to ¯ R over ¯ ∆ and is equal to + ∞ outside of ¯ ∆ , and we will use its convex conjugate ¯ R ∗ (note that the convex conjugate of ¯ R without these modification would have been different !). Let

We have by the Fenchel-Moreau theorem that

By definition of the convex conjugate of ¯ R , for any γ ∈ R d , thus in particular for all γ t , we have

Summing for every t ∈ [ T ] we obtain

Hence

Finally because ¯ R and R are equal over ∆ , and because ∑ T t =1 δ t /T ∈ ∆ , we derive the following inequality:

We now would like to track the left-hand sum evaluated at the dual parameter ˆ λ . To do so we will use straightforward Online Gradient Descent on the linear function λ ↦→⟨ λ, γ t -δ t ⟩ , with sub-gradients γ t -δ t .

We will use Lemma 6 . 17 from Orabona (2019):

Theorem (Online Gradient Descent) . For g t = γ t -δ t a sub-gradient of a convex loss function l t with ∥ g t ∥ 2 ≤ G , for w t generated according to a gradient descent update with parameter η , we have that for any u ∈ R d :

This is exactly our setting, and the way we update λ t . Therefore

Moreover, because ˆ λ is the arg max which yields the convex conjugate of ¯ R ∗ , we have ˆ λ ∈ ∂ ¯ R ∗∗ ( ∑ δ t /T ) = ∂R ∗∗ ( ∑ δ t /T ) . Because R is L -Lipschitz for ∥ · ∥ 2 , we have ∥ ˆ λ ∥ 2 ≤ L . Hence

What do we lose by using ¯ R instead of R to generate γ t in our gradient descent ? We have larger gradients bounds by a factor 2 in our case. Indeed instead of having G ≤ Diam(∆) , because γ t can be outside of ∆ , we have ∥ γ t -δ t ∥ 2 ≤ 2 Diam(∆) . In this case, choosing η = L/ (2 Diam(∆) √ T ) is optimal, and yields the following regret bound:

Using both Equation (27) and Equation (28) yield:

We can finally put everything together. Through the decomposition with Equations (22) to (24), using respectively Equations (26) and (29) and Lemma 3.3 for each of those differences and summing the inequalities, the terms in ⟨ λ t , γ t -δ t ⟩ cancel out, and we obtain that

By applying Lemma E.1 we conclude that

F The Importance of Randomizing Between Sources: OPT vs static-OPT

F.1 Proof of Proposition 2.1

We can directly derive from the proof of Lemma E.1 and Algorithm 1 the following proposition, which is a rewriting of Proposition 2.1.

Proposition. There are instances of the problem such that

Proof. What we have shown indirectly in the proof of Theorem 3.1, is that

therefore lim T →∞ OPT /T = lim T →∞ max π ∈P K inf λ ∈ R d ⟨ π, D ( λ ) ⟩ /T which also gives us a simple way to compute a close upper bound for OPT as a saddle point, instead of having to solve a combinatorial optimization problem.

Following the same arguments as in Lemma E.1 besides the last maximization step, we can also derive that

It remains to find an example such that

We used such an example in Appendix F.2, where max k ∈ [ K ] inf λ ∈ R d D ( λ, k ) = 0 and max π ∈P K inf λ ∈ R d ⟨ π, D ( λ ) ⟩ = 0 . 25 .

F.2 Simple example and numerical illustration

We illustrate Proposition 2.1 and give an example that shows that randomizing over the different sources can outperform the best performance achievable by a single source.

We consider a case where the utility u t ∈ {-1 , 1 } and the protected attribute of a user a t ∈ {-1 , 1 } . All combination of u t and a t are equiprobable: P ( u t = u, a t = a ) = 1 / 4 for all u, a ∈ {-1 , 1 } . The penalty function is R ( δ ) = 5 | δ | . There are K = 2 sources of information. The first source is capable of identifying good indivduals of group 1 : it gives a context c t 1 = 1 when ( a t , u t ) = (1 , 1) and c t 2 = 0 otherwise. The second source does the same for good individuals of group -1 : it gives c t 2 = 1 when ( a t , u t ) = ( -1 , 1) and c t 2 = 0 otherwise. In terms of conditional expectations, this translates into E [ u t | c t 1 =1] = E [ a t | c t 1 =1] = 1 , E [ u t | c t 1 =0] = E [ a t | c t 1 =0] = -1 / 3 for the first source; and E [ u t | c t 2 =1] = 1 , E [ a t | c t 2 =1] = -1 , E [ u t | c t 2 =0] = -1 / 3 , E [ a t | c t 2 =0] = 1 / 3 for the second source.

We apply Algorithm 1 to this example, and we compare its performance to ( i ) the one of an algorithm that has only access to the first source (which is the best source since both sources are symmetric), ( ii ) a greedy algorithm that only use the first source and selects x t = 1 whenever E [ u t | c t 1 ] > 0 , and ( iii ) the offline optimal bounds OPT and static-OPT. The results are presented in Figure 1. We observe that our algorithm is close to OPT as shown in Theorem 3.1. It also vastly outperforms the algorithm that simply picks the best source, whose performance is close to static-OPT. The greedy has a largely negative utility because it is unfair. Our randomizing algorithm picks an allocation x t = 1 only when the context implies that u t = 1 . It obtain a fair allocation by switching sources. An algorithm that uses a single source cannot be fair unless it chooses x t = 0 , which is why static-OPT=0 for this example. The 1 st and 3 rd quartiles are reported for multiple random seeds, the solid lines are the means. The computations were done on a laptop with an i7-10510U and 16 Gb of ram.

0

20000

40000

60000

80000

100000

5000

0

5000

10000

15000

20000

25000

30000

Total Utility

Algorithm 1

Algorithm 1 without source selection:

t

=1 or 2

OPT

static-OPT

k

T

F.3 Additional experiments on the effect on OPT of sources and fairness prices

In the main body of the paper, we study how one is able to use the online algorithm Algorithm 1 so as to achieve a good total utility asymptotically close to that of the offline optimal OPT . However we only consider R and the prices p k of the different sources as inputs for the algorithm, and we did not discuss how their variation actually affect the performance of OPT . While the previous illustrating example shows how a mixture of sources may achieve strictly better performance, we now only focus on the static optimization problem and how various metrics may change as we vary R and p k .

Data model: We consider a fairness penalty R ( x ) = rx 2 with r ≥ 0 , and A = {-1 , 1 } so that a t can only be either -1 or 1 , encoding men and women for instance. We suppose that the protected attributes and the utility ( u t , a t ) are distributed according to Pr( a t = 1) = Pr( a t = -1) = 1 / 2 and u t ∼ N ( a t , 1) , hence the group for which a t = 1 is generally correlated to having better utilities than those for which a t = -1 . We suppose that there are no public context, and that there are K = 2 information sources, with c t, 1 = ( a t , u t ) for source 1 associated with a price of p 1 = p , and for source k = 2 we can observe c t, 2 = u t with a price of 0 . Here the sources are 'monotone' in that one contains strictly more information than the other, but is more expensive. Basically we are going to observe the utility u t in all cases, but before observing u t we can pay p to additionally observe a t . Note that when r = 0 then R = 0 and there is no fairness penalty meaning that we don't actually care about a t and will always select source k = 2 , while when p = 0 the information about a t is free and there is no uncertainty.

We study the asymptotic regime with T → ∞ , and compute the optimal solution of the offline optimization problem through its dual problem. Figure 1 represent the quantities associated to the optimal solution for varying p and r : (top-left) π ∗ the percentage of times information is bought (i.e., k = 1 is chosen), (top-right) the probability of group 1 being selected Pr( a t = 1 | x t = 1) , (bottomleft) the fairness cost relative to utility R ( E [ a t x t ]) / E [ u t x t ] , and (bottom-right) the information cost relative to utility pπ ∗ / E [ u t x t ] .

Analysis and interpretation :

(top-left) Looking at the optimal action in terms of information (to buy the observation of a t or not), we see that there are three regimes: π ∗ = 1 , π ∗ = 0 and a transitive regime π ∗ ∈ (0 , 1) . When r = 0 there is no reason to care about a t , thus we simply do not buy information ( π ∗ = 0 ), this remains true as long as the cost of p is large over r . When p = 0 there are no costs to observe a t and thus π ∗ = 1 and this remains true as long as the fairness penalty r is large enough over p . In between, we may obtain optimal actions which are strictly between 0 and 1 : in this case the best action is a strict convex combination between the two sources and there is some trade-off between buying costly information and selecting an unfair allocation.

(top-right) Regarding the proportion of individuals of group 1 selected, we see that it is always higher than 0 . 5 as group a t = 1 correlates to higher utility compared to group a t = -1 . Therefore it is decreasing in r and goes towards the perfectly fair allocation 0 . 5 which would incur no penalty even for high r . It is also increasing in p as higher information costs make it less desirable to pay to observe a t , and thus more difficult to accurately achieve a fair allocation: the support of the utility u t conditioned on a t = 1 and conditioned on a t = -1 are not disjoint, which makes it difficult only by observing u t to determine whether a t = 1 or -1 .

(bottom-left) For the relative fairness cost compared to utility, when the information cost p increases the allocation becomes more unfair as it becomes more difficult without access to a t to choose a fair allocation with good utility as describe above. Moreover the average utility u t of the selected groups tend to decrease as the individuals which are more likely to be a t = -1 based on u t have low utility. Therefore the ratio fairness penalty over expected utility tend to increase. When r increases there are three conflicting effects: high r makes it more appealing to achieve a fair allocation at the cost of expected utility u t which makes the latter decrease, the expected group allocation becomes more fair as seen previously, and the scaling of R increases. When r becomes high there is some balance between a fairer allocation and higher penalty, and when r = 0 the fairness penalty is clearly 0 . Overall for each fixed p there is some maximum in terms of fairness penalty relative to utility.

(bottom-right) Finally for the information cost relative to utility, it is increasing in r . Indeed due to large fairness penalties it is better to buy information thus π ∗ increases while p remains fixed, hence the product pπ ∗ increases. When p increases, we buy information less often ( π ∗ decreases), and because pπ ∗ = 0 for p = 0 or p large, there is some maximum for each r fixed.

Overall, we see that higher fairness penalties tend to make it more likely to buy information and select a fair allocation even at the cost of utility, while high information cost makes it unattractive to buy this information and can lead to unfairness due to the difficulty of identifying the protected attribute without this additional data.

0.0

0.025 0.05 0.075

0.1

0.125 0.15 0.175

0.2

0.225 0.25 0.275

0.3

Information cost:

p

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Fairness penalty coefficient:

r

0

0

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

1

1

0.24

0

0

0

0

0

0

0

0

0

0

1

1

0.78

0.45

0.26

0.13

0.04

0

0

0

0

0

0

1

1

1

0.71

0.51

0.38

0.24

0.11

0

0

0

0

0

1

1

1

0.86

0.66

0.53

0.43

0.27

0.06

0

0

0

0

1

1

1

0.97

0.76

0.63

0.53

0.38

0.18

0

0

0

0

1

1

1

1

0.84

0.7

0.67

0.44

0.27

0.05

0

0

0

Information selection:

*

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.025 0.05 0.075

0.1

0.125 0.15 0.175

0.2

0.225 0.25 0.275

0.3

Information cost:

p

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Fairness penalty coefficient:

r

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.77

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.72

0.72

0.8

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.84

0.68

0.68

0.7

0.74

0.77

0.8

0.83

0.84

0.84

0.84

0.84

0.84

0.84

0.65

0.65

0.65

0.68

0.71

0.73

0.75

0.77

0.78

0.78

0.78

0.78

0.78

0.63

0.63

0.63

0.64

0.66

0.68

0.7

0.71

0.73

0.73

0.73

0.73

0.73

0.62

0.62

0.62

0.62

0.64

0.65

0.66

0.68

0.69

0.7

0.7

0.7

0.7

0.61

0.61

0.61

0.61

0.62

0.63

0.63

0.65

0.66

0.68

0.68

0.68

0.68

Protected group allocation: Pr(

at

=1

xt

=1)

0.65

0.70

0.75

0.80

0.0

0.025 0.05 0.075

0.1

0.125 0.15 0.175

0.2

0.225 0.25 0.275

0.3

Information cost:

p

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Fairness penalty coefficient:

r

0

0

0

0

0

0

0

0

0

0

0

0

0

0.06

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.1

0.08

0.08

0.16

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.2

0.09

0.09

0.11

0.16

0.2

0.24

0.28

0.3

0.3

0.3

0.3

0.3

0.3

0.09

0.09

0.09

0.12

0.16

0.2

0.24

0.27

0.3

0.3

0.3

0.3

0.3

0.09

0.09

0.09

0.1

0.13

0.16

0.19

0.23

0.28

0.29

0.29

0.29

0.29

0.08

0.08

0.08

0.09

0.11

0.14

0.17

0.2

0.24

0.29

0.29

0.29

0.29

0.08

0.08

0.08

0.08

0.1

0.12

0.13

0.18

0.21

0.27

0.28

0.28

0.28

Relative fairness cost:

R

(

[

atxt

])/

[

utxt

]

0.00

0.05

0.10

0.15

0.20

0.25

0.0

0.025 0.05 0.075

0.1

0.125 0.15 0.175

0.2

0.225 0.25 0.275

0.3

Information cost:

p

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Fairness penalty coefficient:

r

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0.05

0.02

0

0

0

0

0

0

0

0

0

0

0

0.05

0.07

0.06

0.05

0.03

0.01

0

0

0

0

0

0

0

0.05

0.1

0.1

0.1

0.09

0.07

0.04

0

0

0

0

0

0

0.05

0.1

0.13

0.13

0.13

0.13

0.1

0.02

0

0

0

0

0

0.05

0.1

0.15

0.16

0.16

0.16

0.14

0.08

0

0

0

0

0

0.05

0.1

0.15

0.17

0.18

0.21

0.17

0.12

0.03

0

0

0

Relative total information cost: (

p

*

)/

[

utxt

]

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Here we looked at a simple example to try to isolate how sensible is the optimal solution to some variations on r and p . Other parameters can lead to unfairness, such as an unbalance in the protected group population where one would still need a balanced allocation, or penalty functions with varying convexity strength.

G Other Fairness Penalty - proof of Theorem 4.1

In this section, we consider an other type of fairness penalty as described in Section 4.1. We redefine U ( . ) in the following way:

end for

Let us first prove an upper bound on OPT , which actually guides the design of the modified virtual reward ˜ φ :

Lemma G.1. We have the following upper-bound for the offline optimum:

where ˜ D ( λ ) = ( ˜ D ( λ, 1) , . . . , ˜ D ( λ, K )) is a vector representing the value of the dual conjugate problem, with for k ∈ [ K ] the coordinates

When X ⊂ R n + with n > 1 , there are two ways to intuitively generalize this penalty for higher dimension allocations. We can replace ∑ T t =1 x t by ∑ T t =1 ∑ n i =1 x ti , which is still a scalar and the proof follows directly. For instance, this corresponds for online bipartite matching with node arrival to a fairness condition on all online arriving nodes (it does not matter to which offline nodes they are matched). Otherwise we can instantiate n penalties R 1 , . . . , R n , using ∑ t x ti for each penalty R i with i ∈ [ n ] . This corresponds to a fairness condition on the selected online nodes for each of the n offline node. This can be dealt with using one parameter λ i for each penalty R i . For this proof we will assume that n = 1 without loss of generality.

We require the following technical assumption: ∑ t τ =1 a τ ( x τ ) / ∑ t τ =1 x τ ∈ ∆ for all x τ , a τ and t . Of course, the size of ∆ can be increased to guarantee - if possible - that this quantity stays inside ∆ , but with the trade-off that Diam(∆) increases. Clearly, this assumption is verified for the special case of a t ( x t ) = ax t for some a ∈ A , as the x τ / ∑ t x t form a convex combination.

Let us redefine the virtual reward associated to this other fairness penalty:

We also redefine the unbiased estimator ˆ φ with this new ˜ φ .

We now use Algorithm 3. There are 4 differences with Algorithm 2: the new function φ contains in addition xR ∗ ( λ ) , x t is the arg max with respect to this new virtual reward, γ t is the arg max over ∆ δ t /x t instead of ∆ δ t , and finally the update for λ t +1 uses x t γ t instead of γ t .

Algorithm 3 Online Fair Allocation with Source Selection - Other fairness penalty

Input: Initial dual parameter λ 0 , initial source-selection-distribution π 0 = (1 /K,... , 1 /K ) , dual gradient descent step size η , EXP3 step size ρ , cumulative estimated rewards S 0 = 0 ∈ R K . for t ∈ [ T ] do

Draw a source k t ∼ π t , where ( π t ) k ∝ exp( ρS ( t -1) k ) and observe c tk t Compute the allocation for user t :

.

Update the estimated rewards sum and sources distributions for all k ∈ [ K ] :

Compute the expected protected group allocation δ t = E [ a t ( x t ) | c tk t , x t ] and compute the dual protected groups allocation target and update the dual parameter:

Proof. The idea will be the same as in Lemma E.1, we only use a different lagrangian function.

We recall that the x t are σ ( c 1 k 1 , . . . , c tk t ) measurable (because the k t are deterministic). Let δ t = E a t [ a t ( x t ) | c tk ] , and ˜ f t ( x ) = E f t [ a t ( x t ) | c tk ] . Using Jensen's inequality for R convex we have

Moreover by independence of the ( f t , a t , c t 1 , . . . , c tK ) we have that E [ a t ( x t ) | σ ( c 1 k 1 , . . . , c tk t )] = E a t [ a t ( x t ) | c tk t ] = δ t . This is basically an application of Lemma D.3 where we state it more carefully. With the same argument we obtain that E [ f t ( x t ) | σ ( c 1 k 1 , . . . , c Tk T )] = ˜ f t ( x t ) . Therefore

We define the function L : ( k , c , x , λ ) ↦→ ∑ T t =1 ˜ f t ( x t ) -p k t -x t ⟨ λ, ˜ a t ⟩ + x t R ∗ ( λ ) .

Using the Fenchel-Moreau theorem, and the previous inequality we have

We can then finish as we did for Lemma E.1.

We now proceed to the rest of the proof of Theorem 4.1:

Theorem. For η = L/ (2 Diam(∆) √ T ) , m = ¯ f + L +max k | p k | +2 η Diam(∆) + m ∗ ( m ∗ is a constant defined in the proof and depends on ∆ and R ), ρ = √ log( K ) / ( TKm 2 ) , and λ 0 ∈ ∂R ( a ) for some a ∈ A , algorithm Algorithm 1 modified as suggested in Section 4.1 has the following regret upper bound:

Proof. We present here only the parts that are different from the proof of Theorem 3.1, and we refer to the corresponding proof for more details.

Comparison of performance with ˜ φ By adding and removing -x t ⟨ λ t , ˜ a t ⟩ + x t R ∗ ( λ t ) , and using that x t is the maximizer that yields φ we derive that

Because ∆ ⊂ ∆ δ t and, because ¯ R and R are equal over ∆ we have:

Thus

Application of bandit algorithms Compared to the previous setting, the only change is that the function R ∗ is included in φ , so we need to bound it as well. See Lemma D.5. This is where m ∗ comes from.

Dual gradient descent Let us now focus in the tracking of the fairness parameter. In the evaluation of our lower bound of the algorithm performance, we used -x t ¯ R ( γ t ) instead of the true penalty ( ∑ T t =1 x t ) R ( ∑ T t =1 a t ( x t ) / ∑ T t =1 x t ) . Because of a small modification of Lemma 3.3 we only need to compare it with the penalty over the conditional expectation of a t x t . Indeed, by the Lipschitz property of R , whether we re-scale by T or ∑ x t does not change anything.

Hence will now compare those two penalties. Here we will consider a modified version of ¯ R , which is equal to ¯ R over ¯ ∆ and is equal to + ∞ outside of ¯ ∆ , and we will use its convex conjugate ¯ R ∗ (note that the convex conjugate of ¯ R without the modification would have been different !). Let

Hence

By definition of the convex conjugate of ¯ R , for any γ ∈ R d , thus in particular for all γ t , we have

Multiplying by x t and summing for every t ∈ [ T ] we obtain

Hence

Finally because ¯ R and R are equal over ∆ , and because ∑ T t =1 δ t / ∑ T t =1 x t ∈ ∆ by the assumption, we have that

As we did earlier, we apply Online Gradient Descent to track the term evaluated at ˆ λ and obtain Equation (28). We just need to make sure that as before ˆ λ is bounded. By the definition of ˆ λ :

Because R is L -Lipschitz for ∥ · ∥ 2 , we have ∥ ˆ λ ∥ 2 ≤ L .

The rest of the proof is identical.

H Learning u t -proof of Proposition 4.2

We prove in this section the result regarding the added regret of learning u t . In terms of utility functions f t , protected attributes a t , and allocations x t , we go back to the setting of Section 2.1 with f t ( x ) = u t x and X = { 0 , 1 } .

In addition to the assumptions made in Section 4.2, we give more precision regarding the structure of u t . We define θ tk = ✶ [ k t = k ] the indicator variable of selecting source k at time t . We define the following filtration for all k ∈ [ K ] and all t ∈ [ T ] : F tk = σ ( x 1 , c 1 k , θ 1 k , u 1 , . . . , x t , c tk , θ tk , u t , x t +1 , c t +1 ,k , θ t +1 ,k ) .

We assume that the contexts c tk are now feature vectors of dimension q k , and that for all k ∈ [ K ] , there exists some vector ψ k ∈ R q k so that

is a zero mean 1 -subgaussian random variable conditionally on F t -1 ,k .

Consider that each source k corresponds to one bandit problem, with an action set at time t composed of two contextual actions { c tk θ tk x, x ∈ X} = { c tk θ tk , 0 } , and a reward u t x t θ tk . If θ tk = 0 then the action selected is 0 , and the new reward is 0 , which means both are 'observed'. If θ tk = 1 , then the feedback is u t if x t = 1 , and 0 otherwise. Regardless of the source selected, the feedback received and the action producing this feedback are observed for all arms in parallel. This is akin to artificially playing all the [ K ] bandits in simultaneously. This new reward still follows the conditional linearity assumption, and x t θ tk η t is still F t -1 ,k conditionally 1 -subgaussian.

With these rewards and actions, we can define for each source ˆ ψ tk and V tk the least squares estimator and the design matrix as done in Equations (19 . 5) and (19 . 6) of Lattimore & Szepesvári (2020). Let δ ∈ (0 , 1) . We define the confidence set parameter β t as

This parameter is independent of k , this will make the computations easier later on. We now define the ellipsoid confidence set of ψ k as

Algorithm 1 is modified in the following way. For each time t , let ˜ ψ tk = arg max ψ ∈I tk ⟨ ψ, c tk ⟩ . We now select x t using this optimistic estimate of ψ k :

Proposition. Given the above assumptions, the added regret of having to learn E [ u t | c tk ] is of order O ( √ KT log( T )) .

The proof will proceed as follows: we first highlight where the learning will impact the algorithm, leverage the concentration results from Abbasi-yadkori et al. (2011) to decompose into a good event and bad event, and finally bound the loss over the good event.

Let us now analyze the main part of the proof in Appendix E that is modified, which is 'Comparison of performance with φ ', because the only change is the quantity that x t maximizes. Let x ∗ t be the maximizer obtained by replacing the optimistic estimate with the true parameter ψ k . Let ˜ a t = E [ a t | c tk t ] . Because x t maximizes the quantity with the optimistic estimate we have

The last term φ is the one we want, we now need to lower bound the first two terms.

By Theorem 20 . 5 of Lattimore & Szepesvári (2020), we know with probability at least 1 -δ that ψ k ∈ I tk for all t . By union bound, the probability that the ψ k are simultaneously all in their confidence set is at least 1 -Kδ . We denote this event by I .

Suppose that the event I is satisfied. Then by definition of ˜ ψ tk t , we have

Now let us look at the last term. We can first decompose it over the K sources:

Now, consider one source k , and sum the difference term for every t . By remarking that θ tk = θ 2 tk , we have that

where T k is the number of times source k is selected.

By Lemma 19 . 4 of Lattimore & Szepesvári (2020), we have that

Therefore, using Jensen inequality for the square root function which is concave over the T k , and because ∑ k T k = T , we obtain

This yields a O ( √ KT log( T )) error.

And with probability at most Kδ the concentration event I does not hold, but the error is bounded by 4 for each t . Thus overall in expectation, it is bounded by 4 TKδ . Picking δ = 1 /T , yields the desired sub-linear regret bound.

I Learning a t

In this section, we provide further intuitions on the existence of [ K ] different sources, and how it is possible to adapt the algorithm to learn E [ a t | c tk ] on the fly. Let us consider the special case of our model, where the additional information given by each source is a noisy observation ˆ a tk of a t , with different noise levels; in the main setting this can be described as c tk = ( z t , ˆ a tk ) . Sources with lower noise levels will have higher prices (one directly pays for the level of accuracy). This setting can be used to give users the level of privacy they are willing to give up (as in Local Differential Privacy (Dwork & Roth, 2014)), by revealing some noisy information about their private data; of course, the more data is revealed (or the smaller the noise), the higher the compensation (the price).

Furthermore, if the source k corresponds to adding additive independent noise L tk and to returning ˆ a tk = a t + L tk , then knowing the law of both L tk and a t , the decision maker can directly compute the conditional expectation of a t having observed ˆ a tk . Nevertheless, if she only knows the noise distribution, but not the law of a t , conditional expectation can still be learned in an online fashion in some cases.

Indeed, suppose for simplicity that a t ∈ {-1 , 1 } , Pr( a t = 1) = α ∈ (0 , 1) , ( z t , u t ) is independent of a t (therefore we can focus only in dealing with a t ), and that the L tk are 0 mean σ k sub-exponential random variables (see 2 . 7 . 5 Vershynin (2018)). We define the following running empirical estimate of α :

It is an unbiased estimator of α as

Note that the ˆ a t,k t are not independent, and the conditional expectation is not Lispchitz-continuous with respect to the parameter α . Still, we can derive an algorithm with a sub-linear regret bound.

First let us state a Lemma on the concentration of ˆ α t

Lemma I.1. We have the following concentration bound for t ≳ log( T ) :

Proof. We refer to Chapter 2 of Vershynin (2018) for concentration results of sum of independent random variables. Let us recall some definitions, properties, and a concentration inequality.

Let us consider the following Orlicz space norm for random variables. For a random variable X , we define

If ∥ X ∥ ψ 1 is finite, then X is called sub-exponential. It is a norm, and for a bounded variable X in [ -1 , 1] , we have ∥ X ∥ ψ 1 ≤ 1 / log(2) (see exercise 2 . 5 . 7 , example 2 . 5 . 8 , and Lemma 2 . 7 . 6 of Vershynin (2018)). Hence because ( a t +1) / 2 -α ∈ [ -α, 1 -α ] ⊂ [ -1 , 1] , for ˆ α tk the estimate obtained by always choosing source k , we have by triangle inequality that

We now recall Bernstein's inequality:

Theorem (Theorem 2.8.1 of Vershynin (2018)) . Let X 1 , . . . , X N be N independent, mean 0 , subexponential random variables, with κ = max i ∥ X i ∥ ψ 1 . Then for c > 0 a constant, for every t > 0 we have

We want to apply this Theorem to (ˆ a t,k t + 1) / 2 -α the centered ˆ α t where we now change the source depending on k t . Remark that now however the ˆ α t is not a sum of independent variables anymore because of k t which does depend on the previous realizations of the ˆ a t,k t . Nevertheless it is a martingale difference for the filtration H t , and similar results will apply. We denote by κ = 1 / log(2) + max k σ k / 2 an upper bound on the sub-exponential norm of the centered ˆ α t conditional on H t -1 . Then the martingale ∑ t τ =1 ˆ α τ -α is also a sub-exponential random variable.

This is a known result, which is stated here for completeness, but that can otherwise be skipped. Indeed from proposition 2 . 7 . 1 part e ) of Vershynin (2018), we know that finite sub-exponential norm for a centered random variable X means that the moment generating function of X at y is bounded by exp( c 1 ∥ X ∥ 2 ψ 1 y 2 ) for y ≤ c 2 / ∥ X ∥ ψ 1 , with c 1 and c 2 some positive constants.

Therefore, for y ≤ c 2 /κ , using the property of the conditional sub-exponentiality of ˆ α t , we have

Iterating this inequality we obtain for y ≤ c 2 /κ that

This yields inequality (2 . 24) in Vershynin (2018) the main element needed for the proof of Bernstein's Inequality. Thus Bernstein's inequality still holds for centered martingale sequence which are conditionally sub-exponential. We stress that this is not a new concentration inequality result.

Let us now apply Bernstein's Inequality with a deviation of order √ κ 2 log( T ) / ( ct ) , for t ≥ log( T ) /c :

Proposition I.2. Given these assumptions, using Algorithm 1 with ˆ α t instead of α to compute the conditional expectations yields an added regret of order O ((max k σ k +2 / log(2)) √ T log( T )) .

Proof. There are two parts of the proof of Theorem 3.1 that we need to modify, when the x t is involved, and when we compare the penalty. We will deal with the first part (the more complicated), and the second one follows immediately by the same arguments. This will be done in 3 steps: we first highlight where the learning error occurs, then use the concentration result to decompose the error under a good and bad event, and finally show that under the good event the error in expectation is small enough.

Impact of learning error We suppose for simplicity that L tk has a density g k over the support R , but this also holds for discrete random variables. We denote by ν k the density of ˆ a tk . First let us express ˜ a t = E [ a t | c tk t ] as a function of ˆ a t,k t = ˆ a and α :

Let us define the plug-in estimate of ˜ a t using ¯ α t -1 whenever ¯ α t -1 is in [0 , 1] by:

When the estimated parameter is not in [0 , 1] , we use s t = 0 .

We define by x t the maximisation of the virtual value function φ when replacing ˜ a t by s t , and by x ∗ t the allocation obtained if we were to actually use ˜ a t . Because x t is a maximizer of u t x t -⟨ λ t , s t ⟩ x t , we have

The term ⟨ λ t , s t ⟩ x t will be tracked with the help of the OGD, it remains to take care of ⟨ λ t , ˜ a t -s t ⟩ x ∗ t . By Cauchy Schwartz:

We know that λ t is already bounded, so we only need to bound the right-hand term. Decomposing depending on the values of k t , we can rewrite

Decomposition under good and bad event Denote by C t = [ | ˆ α t -α | ≤ √ ( κ 2 log( T ) / ( ct )] the event of ˆ α t concentrating around its mean. For t large enough, that is to say t ≥ ( κ 2 log( T )) / ( c min( α, 1 -α )) , the event C t is included in [ ¯ α t -1 ∈ (0 , 1)] . We can then decompose the analysis over these events using the triangle inequality: | s t -˜ a t | ≤ ✶ C t -1 | s t -˜ a t | + ✶ ¯ C t -1 | s t -˜ a t | , and we know that for the term multiplied by ✶ C t -1 we have that either it is 0 or ¯ α t -1 ∈ [0 , 1] . Let us consider such a t -1 big enough, and first analyze the left-hand term with the 'good' event.

Small error under good event in expectation We consider the conditional expectation of this sum given k t , and ¯ α t -1 . By independence between ( k t , ¯ α t -1 , E ) and ˆ a tk for all k , we have:

Because ν k t (ˆ a ) = ( αg k (ˆ a -1) + (1 -α ) g k (ˆ a +1)) we can simplify the integral:

For α ∈ [0 , 1] , we have the following inequality:

Thus using this inequality and the fact that g k is a density (which sums to 1 ), we obtain that

Finally taking the full expectation yields

For the complementary event ¯ C t -1 :

Putting everything back together Now denoting by m λ the upper bound over the λ t obtained by Lemma 3.2 and summing over all t :

where we used a series-integral comparison of x ↦→ 1 / √ x for the second to last inequality.

There is a second place where replacing ˜ a t by s t impacts the performance of the algorithm. Indeed, through the gradient descent on λ t , we will have a penalty converging towards R ( ∑ T t =1 s t x t /T ) and not R ( ∑ T t =1 ˜ a t x t /T ) . This is because the Online Gradient Descent for the λ t uses s t and not ˜ a t . Nevertheless, because of the Lipschitz property of R :

where the last inequality is obtained by similar manipulations on C t to what we just did.

Overall, the added regret when using s t instead of ˜ a t is of order √ T log( T ) which still guarantees a sub-linear regret.

Another algorithm is possible: first sample the source with the lowest sub-exponential constant T 2 / 3 log( T ) times, and then use the estimate obtained to compute s t (without updating it with the newest noise received). The advantage is that even though we incur a T 2 / 3 regret (this can be shown using basically the same methods), we can obtain a lower κ . Hence if there is a big difference between the sub-exponential constant of the different sources, or even one source with infinite sub-exponential constant, this may be a viable algorithm.

J Public Contexts

For this section and in order to make the dependence in the public context z t appear more clearly, we suppose that c tk represent only the additional information of the source k , and thus ( c tk t , z t ) represent the whole information available at time t after selecting a source.

J.1 Finite number of Contexts - proof of Proposition 3.4

We propose the following Algorithm 4.

Proposition. For µ the probability distribution over Z finite, we can derive an algorithm that has a modified regret of order O ( √ TK log( K ) ∑ z ∈Z √ µ ( z )) , where µ ( z ) is the probability that the public attribute is z .

Proof. Most of the proof consists is identical to the proof of Theorem 3.1, and only the different parts will be highlighted, which is the upper bound of OPT , and applying bandits algorithms.

Algorithm 4 Online Fair Allocation with Source Selection - With public contexts

Input: Initial dual parameter λ 0 , initial source-selection-distribution π 0 ( z ) = (1 /K,... , 1 /K ) , dual gradient descent step size η , EXP3 step size ρ t ( z ) that uses a doubling trick, cumulative estimated rewards S 0 ( z ) = 0 ∈ R K for all z ∈ Z .

Observe z t ∈ Z

Draw a source k t ∼ π t ( z t ) , where π tk ( z t ) ∝ exp( ρ t ( z t ) S ( t -1) k ( z t )) and observe c tk t . Compute the allocation for user t :

Update the estimated rewards sum and sources distributions for all k ∈ [ K ] :

Compute the expected protected group allocation δ t = E [ a t ( x t ) | c tk t , z t , x t ] and compute the dual protected groups allocation target and update the dual parameter:

end for

We first upper bound the performance of OPT . For all t let h t ∈ [ K ] Z be the deterministic source selection policies, k t = h t ( z t ) the selected sources, x t ∈ X be ( z t , c t,k t ) measurable, and λ ∈ R d be a dual variable. In an identical manner to Equation (20) we can obtain the following inequality:

Let us define φ ( λ, c tk , k, z ) = max x ∈X E [ f t ( x ) -⟨ λ, a t ( x ) ⟩ | z, c tk ] -p k , and µ ( z ) = Pr( z t = z ) . We also define π k ( z ) = ∑ T t =1 ✶ [ h t ( z ) = k ] /T the total number of times we would have selected source k for the public context z .

We now take the expectation of the sum. We obtain by tower property of the conditional expectation, and using the i.i.d. assumption:

Clearly π k ( z ) ∈ P K , thus if we maximize over this space for the π k ( z ) instead of over ([ K ] Z ) T , it will be higher. Therefore

Notice that µ ( z ) plays an important role in determining the optimal policy. If we had simply run the previous algorithm for each context, this would be akin to considering that the z t are distributed uniformly according to µ ( z ) = 1 /Z , which would have yielded a sub-optimal policy π .

Now let us take care of the lower bound. As done in Equation (25), we have that

and as we did before we will use bandits algorithm on these rewards.

We would now like to bound the following adversarial contextual bandit problem regret:

The weighted estimator ˆ φ is unbiased as φ ( λ t , c t , k t , z t ) ✶ [ k t = k ] /π tk ( z t ) is still unbiased. Indeed, c t and k t are conditionally independent given ( z t , H t -1 ) , and π tk ( z t ) is ( z t , H t -1 ) measurable. Thus

As suggested section (18 . 1) of Lattimore & Szepesvári (2020) running one anytime version of EXP 3 (using a doubling trick for instance) on each of these public contexts z ∈ Z , achieves a regret bound of order O ( √ K log( K ) ∑ z ∈Z √ ∑ T t =1 [ z t = z ]) .

✶ Taking the expectation (for z t ) of this regret bound, and using Jensen's inequality for the square root function we have a regret term of order

Let ( π n ) ∈ ( P Z K ) N be the sequence of contextual source mixing that maximizes the upper bound for OPT in Equation (43). Then because the maximum of [ K ] points is greater than any convex combination, π n ( z ) in particular, we have

In expectation this yields:

where we used the linearity of the expectation, and the tower property of the expectation for the last equality.

Because z t and H t -1 are independent, we can first use Lemma D.3 and then the freezing lemma to express the conditional expectation given H t -1 as

Hence,

Grouping it together with the R ∗ ( λ t ) terms, and taking the inf over λ we derive that

As n →∞ and by definition of π n , we can conclude that

Finally, using Equation (44) and the previous inequality we obtain

The rest of the proof follows Appendix E.

Clearly, if µ has a non-zero probability only for one z , we recover the previous regret bound without public information.

J.2 Infinite number of contexts - proof of Proposition 3.5

Lemma. If E [ a t ( x ) | c tk , z t ] and E [ f t ( x ) | c tk , z t ] are Lipschitz continuous in z with respective constants L a and L f for all x ∈ X with respect to ∥ · ∥ 2 , then so is φ in z with Lipschitz constant L a + L f .

Proof. Let λ ∈ R d , k ∈ [ K ] , c tk the additional information, z 1 , z 2 ∈ Z , and x 1 , x 2 ∈ X be the maximizers that respectively yields φ ( λ, c tk , k, z 1 ) and φ ( λ, c tk , k, z 2 ) . Because x 2 is a maximizer, we have that

Using the same arguments because x 1 is a maximizer, we obtain the symmetric inequality.

The main idea for the discretization (that averages the rewards over the discretized bins), is that the performance of the algorithm is close to the discretized optimal, which itself is close to the continuous optimal for smoothness reasons. However here the discretization is applied not to the original total utility U , but instead to the bandits reward part. The number of discretized bins is then tuned depending on the lipschitz constants and the space dimension r .

The proof of the proposition Proposition 3.5 then directly follows from exercise 19 . 5 of Lattimore & Szepesvári (2020) applied to the contextual bandit problem with rewards φ .

Remark. In the case when Z is continuous, there are no guarantees that there exists an optimal policy π ∗ . Indeed, if for the product topology the space P Z K is compact (by Tychonoff's theorem), the dual function used to derive OPT is not continuous. Vice versa, for a topology which makes this dual function continuous, it is unlikely for this space of functions to be compact. Hence why we need to take a maximizing sequence of π n so that the dual function of these π n converges to the upper bound of OPT .

Peer reviews:
comments: SUMMARY:
Existing works in the fair online allocation space assume that the decision-maker has access to protected attributes. In contrast, this paper studies the choice of a data source as a multi-armed bandit problem in conjunction with the primary allocation problem. This paper introduces a computationally efficient algorithm with sublinear regret in the general case. The contributions of this paper are exclusively theory-based, including regret bounds for various types of regret penalties.

STRENGTHS:
This is the first paper to consider optimizing over data sources in the problem space of fair online allocation. This paper contributes an algorithm with proven regret bounds, which is significant. I believe that other researchers are likely to build off of this work, and that it advances research in a demonstrable way. 

This paper is excellently written. It is clear, well organized, and adequately informs the reader. In particular, I am happy to see good explanations of the reasoning, limitations, and assumptions made in this work.

WEAKNESSES:
This paper does not contribute code or empirical results; both of which would be of interest to the research community.

QUESTIONS:
N/A

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

comments: SUMMARY:
The authors formulate the online fair allocation problem, where a decision maker may choose an information source $k_t$ based on public context $z_t$ and generate an allocation decision $x_t$ with a reward $1$ if $x_t=1$. The fairness penalty is some convex function that is minimized when the decision $x_t$ is uncoupled with the private type $a_t$. The authors developed a virtual-value-based algorithm that achieves $O(\sqrt{T})$ online regret.

STRENGTHS:
The fairness problem is of interest to the community, especially when the private attributes are unseen.

WEAKNESSES:
It seems that the source selection part is simply the exponential weight algorithm with some virtual values which can be updated by the EXP3 algorithm. The allocation is based on the dual value $\lambda_t$ which can then be updated by the dual gradient descent. In my opinion, the authors did not make enough effort to explain the intuitions and justified the use of these algorithms, let alone the novelty of the work. Also, I'm not sure about the technical involvement, since it seems that the only challenge here is having multiple sources of information.

QUESTIONS:
What is the theoretical lower bound for this problem?

LIMITATIONS:
No concerns here.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an online allocation problem with long-term fairness constraints where, unlike prior works, the protected attributes of online users (e.g., gender, ethnicity) are not observed, but they can be purchased from $K$ different sources for some additional cost. The goal is to maximize the overall utility of the users minus the cost of additional information that was bought minus a fairness penalty that is characterized by a convex function $R(\\cdot)$. The authors study this problem under an i.i.d. assumption on the users and propose Algorithm 1 with sublinear regret for the problem.\n\nSTRENGTHS:\n- Addressing fairness constraints in online allocation is a very well-motivated problem and the authors' assumption that protected attributes are not observed is a more realistic model of the applications. Therefore, the contributions of this work are of potential interest to a wide audience.\n- Simultaneous source selection and dual gradient descent in Algorithm 1 is an important contribution that shows the naive approach of performing these two tasks separately is not good enough for this problem.\n- The proposed algorithm is compatible with various notions of group fairness.\n\nWEAKNESSES:\n- The problem setup is a bit confusing. I'm not sure how the public context $z_t$, protected attributes $a_t$, and the purchased piece of information $c_{tk_t}$ are related to each other. On line 152, the authors make the assumption that $c_{tk_t}$ contains $z_t$ and I'm not quite sure if I understand what that means.\n- The authors have distinguished their paper from prior works based on the fact that protected attributes are not observed and could only be purchased. However, a comparison of the paper with prior works in terms of techniques and proof ideas is missing. It appears that both primal and dual updates in Algorithm 1 are quite standard and the main challenge is to combine these two steps. This point needs to be highlighted and clarified.\n- Given the vagueness of the problem framework, explaining the setting in the context of a few motivating applications is very helpful. In particular, the process of purchasing data from data sources needs clarification.\n\nQUESTIONS:\n- Can you provide a few motivating applications that could be cast into your framework and explain what data sources are and how they relate to each other?\n- Can you explain the connection between $a_t$, $z_t$, and $c_{tk_t}$ in the context of a motivating application? How do these quantities relate to each other? What is the dimension of each quantity? How is $\\mathbb{E}[a_t|c_{tk_t}]$ estimated?\n- Assuming that $\\sum_{t}x_t>0$, the function $(\\sum_t x_t)R(\\frac{\\sum_t a_tx_t}{\\sum_t x_t})$ is simply the perspective of the convex function $R(\\cdot)$ and therefore, it should be convex in $x=(x_1,\\dots,x_T)$. However, on lines 329-330, it is claimed that the opposite is true. Can you explain why convexity doesn't hold?\n- Is it possible to use Algorithm 1 to obtain results for the adversarial setting? What would be the main challenges?\n- On line 131, why is $c_{tk}$ enough to estimate $u_t$ and $a_t$? How about the public context information $z_t$, is it not helpful for estimation?\n\n\n--------------------------------\nI've read the authors' rebuttal, thanks for addressing my questions and concerns.\n\nLIMITATIONS:\nDiscussion of potential negative societal impact is not necessary.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of online resource allocation with long-term fairness constraints. Different from mainstream literature, the authors assume that the fairness context is not known to the player in advance. However, the player does have the option of paying to buy fairness context information from different sources to make more fair decisions. The algorithms consists of two parts, the first part focuses on source selection for buying protected fairness information and the second part focuses on the allocation decision. The regret of the proposed algorithm is analyzed and it is shown that it is $O(\\sqrt(T))$. Last, there are some additional discussions and results regarding some other notions of fairness.\n\nSTRENGTHS:\n++ It is interesting that the authors consider the setting where protected attributes (such as demographics) are unknown to the online decision maker, but the decision maker should still be fair. It makes sense that prior works on fair allocation would only consider the case where these are known, but the reviewer is not familiar enough with the literature to know whether that’s true. \n\n++ The assumption that a user’s protected attributes can be estimated from the online context and/or additional data which is purchased is an interesting addition to the setting. \n\n++   It’s nice that it can model multiple notions of fairness as discussed in the last section of the paper.\n\nWEAKNESSES:\n-- The fairness in their setting is a penalty rather than a hard constraint. This should be clarified since the claim of long-term hard fairness constraints is mentioned multiple times in the paper. \n\n-- I was expecting to see more discussion about the trade-off between paying for additional information and paying for fairness — i.e., when fairness doesn’t cost much, how does that impact the allocation? Maybe the regret bounds still hold in either case, but it doesn’t tell us qualitatively how the algorithm actually creates fair allocations. Especially since this “trade-off” is explicitly mentioned in the title of the paper…\n\n\n-- The assumption that users are drawn i.i.d. from a probability distribution enables this second assumption that the conditional expectation of protected attributes given context is known (learned from past data). However, in practice, past data is often unfair, which means I think these estimates may be inaccurate, particularly for underrepresented groups, and that intuitively should impact fairness here. It seems that there is no discussion of distribution shift or, in general an acknowledgment that the past data might exhibit this property. They do mention in the conclusion that the estimate of protected attributes could be learned online.\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}