SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mathieu Molina  
Inria, FairPlay Team, Palaiseau, France  
mathieu.molina@inria.fr
Nicolas Gast  
Univ. Grenoble Alpes, Inria, CNRS  
Grenoble INP, LIG, 38000 Grenoble, France  
nicolas.gast@inria.fr
Patrick Loiseau  
Inria, FairPlay Team, Palaiseau, France  
patrick.loiseau@inria.fr
Vianney Perchet  
CREST, ENSAE, Palaiseau, France  
Criteo AI Lab, Paris, France  
vianney.perchet@normalesup.org
We consider the problem of online allocation subject to a long-term fairness penalty. Contrary to existing works, however, we do not assume that the decision-maker observes the protected attributes—which is often unrealistic in practice. Instead they can purchase data that help estimate them from sources of different quality; and hence reduce the fairness penalty at some cost. We model this problem as a multi-armed bandit problem where each arm corresponds to the choice of a data source, coupled with the online allocation problem. We propose an algorithm that jointly solves both problems and show that it has a regret bounded by  $\mathcal{O}(\sqrt{T})$ . A key difficulty is that the rewards received by selecting a source are correlated by the fairness penalty, which leads to a need for randomization (despite a stochastic setting). Our algorithm takes into account contextual information available before the source selection, and can adapt to many different fairness notions. We also show that in some instances, the estimates used can be learned on the fly.
We consider the problem of online allocation with a long-term fairness penalty: A decision maker interacts sequentially with different types of users with the global objective of maximizing some cumulative rewards (e.g., number of views, clicks, or conversions); but she also has a second objective of taking globally "fair" decisions with respect to some protected/sensitive attributes such as gender or ethnicity (the exact concepts of fairness will be discussed later). This problem is important because it models a large number of practical online allocation situations where the additional fairness constraint might be crucial. For instance, in online advertising (where the decision maker chooses to which users an ad is shown), some ads are actually positive opportunities such as job offers, and targeted advertising has been shown to be prone to discrimination Lambrecht & Tucker (2019); Speicher et al. (2018); Ali et al. (2019). Those questions also arise in many different fields such as workforce hiring (Dickerson et al., 2018), recommendation systems (Burke, 2017), or placement in refugee settlements (Ahani et al., 2021).
There has been a flourishing trend of research addressing fairness constraints in machine learning—see e.g., (Chouldechova, 2017; Hardt et al., 2016; Kleinberg & Raghavan, 2018; Barocas et al., 2019; Emelianov et al., 2020; Molina & Loiseau, 2022)—and in sequential decision-making problems—see e.g., (Joseph et al., 2016; Jabbari et al., 2017; Heidari & Krause, 2018)—as algorithmic decisions have real consequences on the lives of individuals, with unfortunate observed discrimination (Buolamwini
& Gebru, 2018; Larson et al., 2016; Dastin, 2018). In online allocation problems, general long-term constraints have been studied for instance by Agrawal & Devanur (2015) who maximize the utility of the allocation while ensuring the feasibility of the average allocation, or by Balseiro et al. (2020) who show how to handle hard budget constraints for online allocation problems. More directly related to fairness, Nasr & Tschantz (2020) formulate the problem of fair repeated auctions with a hard constraint on the difference between the number of ads shown to each group. Finally, in recent works, Balseiro et al. (2021); Celli et al. (2022) consider the online allocation problem where a non-separable penalty related to fairness is suffered by the decision maker at the end of the decision-making process instead of a hard constraint—these works are the closest to ours.
All the aforementioned papers, unfortunately, assume that the protected attributes (which define the fairness constraints) are observed before taking decisions. In practice, it is often not the case—for instance to respect users' privacy (Guardian, 2023)—and this makes it challenging to satisfy fairness constraints (Lipton et al., 2018). In online advertising for example, the decision-maker typically has access to some public "contexts" on each user, from which she could try to infer the value of the attribute; but it was shown that the amount of noise can be prohibitive and therefore ensuring that a campaign reaches a non-discriminatory audience is non-trivial Gelauff et al. (2020).
Our contribution. In this paper, we consider the online allocation problem under long-term fairness penalty, in the practical case where the protected attributes are not observed. Instead, we consider the case where the decision-maker can pay to acquire more precise information on the attributes (beyond the public context), either by directly compensating the user (the more precise the information on the attribute, the higher the price) or by buying additional data to some third parties data-broker.<sup>1</sup> Using this extra information, she should be able to estimate more precisely, and thus sequentially reduces, the unfairness of her decisions while keeping a high cumulative net reward. The main question we aim at answering is how should the decision maker decide when, and from which source (or at what level of precision), to buy additional data in order to make fair optimal allocations?
Compared to the closest existing works (Balseiro et al., 2021; Celli et al., 2022) which study online allocation with a long term fairness penalty, the main novelty in the setting we examine is two-fold: we allow for uncertainty on the attributes of each individual, and more importantly we consider jointly the fair online allocation problem with a source selection problem. Consequently, we present the efficient Algorithm 1 that tackles both of these challenges concurrently. This algorithm combines a dual gradient descent for the fair allocation aspect and a bandit algorithm for the source selection part. The final performance of an algorithm is its net cumulative utility (rewards minus costs of buying extra information) penalized by its long-term unfairness; that is quantified by the "regret": the difference between this performance and the one of some benchmarking "optimal" algorithm. We show that Algorithm 1 has a sub-linear regret bound under some stochastic assumptions. Notably, the performance achieved by Algorithm 1 using randomized source selection is strictly better than when using a single fixed source, because of the interaction through the fairness penalty—a key difference with standard bandit settings. On a more technical level we show how one can model the randomness and estimates for the protected attributes, how to bound the fair dual parameters which is crucial in order to use adversarial bandit techniques, and how to combine the analysis of the primal and dual steps of the algorithm.
There are many different definitions of group fairness that can be studied (e.g., demographic parity, equal opportunity, etc.). Instead of focusing on a specific one, we consider a generic formulation that can be instantiated to handle most of those different concepts (see Section 2.1 and Appendix A). We also discuss in Section 4.1 how to adapt our algorithm to different fairness penalties. This gives a higher level of generality to our results compared to existing approaches that can handle fewer fairness criteria (e.g., Celli et al. (2022)).
For the sake of clarity, we expose our key results in a simple setting. In particular, we assume binary decisions and a linear utility, and we assume that the expected utility conditional on the context is known. All these assumptions can be relaxed. In particular, we can learn the utilities (see Section 4.2). We can handle also more general decision variables and utility forms. This allows in particular to tackle problems of matching and auctions (albeit with some restrictions), we discuss that in Appendices B and C.
This setting includes as special cases the extremes where no additional information can be bought and where the full information is available.
Related work. The problem of online fair allocation is closely related to online optimization problems with constraints, which is studied in a few papers. For instance, bandit problems with knapsack constraints where the algorithm stops once the budget has been depleted have been studied by Badanidiyuru et al. (2018); Agrawal & Devanur (2016). Li & Ye (2021); Li et al. (2022) consider online linear programs, with stochastic and adversarial inputs. Liu et al. (2021) deal with linear bandits and general anytime constraints, which can be instantiated as fairness constraints. More recently Castiglioni et al. (2022a,b) propose online algorithms for long-term constraints in the stochastic and adversarial cases with bandit or full information feedback. Some papers take into account soft long-term constraints (Agrawal & Devanur, 2015; Jenatton et al., 2016), and more recently in (Balseiro et al., 2021; Celli et al., 2022) where the long-term constraint can be instantiated as a fairness penalty—we also adopt a soft constraint. We depart from this literature by considering the case where the protected attributes (based on which fairness is defined) is not observed. We consider the case where the decision-maker can buy additional information to estimate it (which adds considerable technical complexity), but even in the case where no additional information can be bought our work extends that of (Balseiro et al., 2021; Celli et al., 2022).
As mentioned above, the fairness literature usually assumes perfect observation of the protected attributes yet noisy realizations of protected attributes or limited access to them to measure and compute fair machine learning models has also been considered (Lamy et al., 2019; Celis et al., 2021; Zhao et al., 2022). In some cases, the noise may come from privacy requirements, and the interaction between those two notions has been studied (Jagielski et al., 2019; Chang & Shokri, 2021), see (Fioretto et al., 2022) for a survey. There are also works on data acquisition, which is similar to purchasing information from different sources of information; e.g., Chen et al. (2018); Chen & Zheng (2019) study mechanisms to acquire data so as to estimate some of the population's statistics. They use a mechanism design approach where the cost of data is unknown and do not consider fairness (or protected attributes). However, none of these approaches can handle sequential decision problems with fairness constraints or penalties (and choosing information sources).
2.1 Model and assumptions
We present here a simpler model, and later-on discuss possible extensions. Consider a decision maker making sequential allocation decisions for a known number of  $T$  users (or simply stages) in order to maximize her cumulative rewards. The user  $t$  has some protected attributes  $a_{t} \in \mathcal{A} \subset \mathbb{R}^{d}$ , that is not observed before taking a decision  $x_{t}$ . On the other hand, the decision-maker first observes some public context  $z_{t} \in \mathcal{Z}$ , where  $\mathcal{Z}$  is the finite set of all possible public contexts and she has the possibility to buy additional information. There are  $K$  different sources for additional information and choosing the source  $k_{t} \in [K]$  has a cost of  $p_{k_t}$ , but it provides a new piece of information, which together with the public information is summarized in the random variable  $c_{tk_t} = (z_t, \text{data from } k_t)$ . Based on this, the decision  $x_{t} \in \{0,1\}$  can be made; this corresponds to include, or not, user  $t$  in the cohort (for instance, to display an ad or not). Including user  $t$  generates some reward/utility  $u_{t}$  which might be unknown to the decision maker (as it may depend on the private attribute), but can be estimated using the different contexts.
To fix ideas, we show how this model applies to two examples: Imagine an advertiser aiming to display ads to an user, able to see some bare-bone information through cookies, such as which website was previously visited  $(z_{t})$ . Based on this information, they can decide whether or not to buy additional information  $(c_{tk})$  from different data brokers that collect user activity. For example, if they observe that the user has browsed clothing stores, they might opt to acquire data containing purchase details from this website. This enables them to estimate the user's gender  $(a_{t})$  based on the type of clothing bought. Now consider the problem of fairly relocating refugees to different cities. When the organization in charge of resettlement receives a resettlement case  $(z_{t})$ , it can either decide to directly assign the refugee to a specific city, or to conduct an additional costly investigation (which might involve a third party watch-dog) to get more information  $(c_{tk})$  on some protected attributes of interest such as wealth or age  $(a_{t})$ , which might have been intentionally misreported.
We assume that the global objective is to maximize the sum of three terms: the cumulative rewards of all selected users, minus the costs of the additional information bought, minus some unfairness penalty, represented by some function  $R(\cdot)$ . Denoting by  $\pmb{k} = (k_{1},\dots ,k_{T})$  and  $\pmb{x} = (x_{1},\dots ,x_{T})$
the sources and allocations selected during the  $T$  rounds, the total utility of the decision maker is then
$$
\mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) = \sum_ {t = 1} ^ {T} u _ {t} x _ {t} - \sum_ {t = 1} ^ {T} p _ {k _ {t}} - T R \left(\frac {1}{T} \sum_ {t = 1} ^ {T} a _ {t} x _ {t}\right). \tag {1}
$$
The penalty function  $R(.)$  is a convex penalty function that measures the fairness cost of the decision-making process, due to the unbalancedness of the selected users at the end of the  $T$  rounds. It can be used to represent statistical parity (Kamishima et al., 2011), as a measure of how far the allocation is from this fairness notion. This fairness penalty  $R(.)$  is also used in Balseiro et al. (2021); Celli et al. (2022). In fact, the objective (1) is equal the one used in these papers minus the cost of additional information bought,  $\sum_{t} p_{k_{t}}$ .
Knows, unknowns and stochasticity We assume that users are independent and identically distributed (i.i.d.), in the sense that the whole vectors  $(z_{t},u_{t},a_{t},c_{t1}\dots c_{tK})$  are i.i.d., drawn from some underlying unknown probability distribution. While this may be a strong assumption, some applications such as online advertising correspond to large  $T$  but to a short real time-frame, hence incurring very little variation in the underlying distribution. The prices  $p_k$  and the penalty function  $R(.)$  are known beforehand. As mentioned several times, the only feedback received is  $c_{tk}$ , after selecting source  $k$ , and this should be enough to estimate  $u_{t}$  and  $a_{t}$ . We therefore assume that the conditional expectations  $\mathbb{E}[u_t\mid c_{tk_t}]$  and  $\mathbb{E}[a_t\mid c_{tk_t}]$  are known. The rationale behind this assumption is that these conditional expectations have been learned from past data.
Penalty examples and generalizations A typical example for  $a_{t}$  is the case of one-hot encoding: there are  $d$  protected categories of users and  $a_{t}$  indicates the category of user  $t$ . For simplicity, assume that  $d = 2$ , then the quantity  $\sum_{t} a_{ti} x_{t}$  is the number of users of type  $i \in \{1, 2\}$  that have been selected. The choice of  $TR(\sum_{t=1}^{T} a_{t} x_{t} / T) = |\sum_{t} a_{t1} x_{t} - \sum_{t} a_{t2} x_{t}|$  amounts to penalizing the decision maker proportional to the absolute difference of users in both groups. This generic setting can also model other notions of fairness, such as Equality of Opportunity (Hardt et al., 2016), by choosing other values for  $a_{t}$  and  $R(\cdot)$ , see examples and discussion in Appendix A.
Similarly, the choice of  $x_{t} \in \{0,1\}$  can be immediately generalized to any finite decision set or even continuous compact one (say, the reward at stage  $t$  would then be  $\mathbf{u}_t^\top x_t$  for some vector  $\mathbf{u}_t$ ), which makes it possible to handle problems such as bipartite matching. Instead of deriving a linear utility from selecting an user, general bounded upper semi-continuous (u.s.c.) utility functions can also be treated, and can be used to instantiate auctions mechanism (with some limitations detailed in Appendix C). We also explain how to relax the assumption that  $\mathbb{E}[u_t \mid c_{tk_t}]$  is known in Section 4.2, by deriving an algorithm that actually learns it in an online fashion, following linear contextual bandit techniques (Abbasi-yadkori et al., 2011).
We show in Section 3.5 that the assumption that  $\mathcal{Z}$  is finite can be relaxed if all conditional distributions depend smoothly on  $z$ , following techniques from Perchet & Rigollet (2013).
Mathematical assumptions We shall assume  $|u_t| \leq \bar{u}$ , for all  $t$ , and that  $\| a \|_2 \leq 1$  for all  $a \in \mathcal{A}$ . We make, for now, no structural assumption on the variables  $c_{tk}$ . We mention here that the decision maker has to choose a single source at each stage, but this is obviously without loss of generality (by adding void or combination of sources).
We define  $\Delta = \mathrm{Conv}(\mathcal{A}\cup \{0\})$ , where  $\mathrm{Conv}$  is the closed convex hull of a set. Since  $\mathcal{A}$  is compact, the set  $\Delta$  is also convex and compact. The penalty function  $R:\Delta \to \mathbb{R}$  is a proper closed convex function that is  $L$ -Lipschitz continuous for the Euclidean norm  $\| \cdot \| _2$ . While the convexity assumption is pretty usual, the Lipschitzness assumption is rather mild as  $\Delta$  is convex and compact. Nevertheless, interesting non-Lipschitz functions, such as the Kullback-Leibler divergence, can be modified to respect these assumptions (see Celli et al. (2022)).
2.2 Benchmark and regret
A usual measure of performance for online algorithms is the regret that compares the utility obtained by an online allocation to the one obtained by an oracle that knows all parameters of the problem, yet not the realized sequence of private attributes. We denote the performance of this oracle by OPT:
$$
\operatorname {O P T} = \max  _ {\boldsymbol {h} \in ([ K ] ^ {\mathcal {Z}}) ^ {T}} \mathbb {E} \left[ \max  _ {\boldsymbol {x} \in \{0, 1 \} ^ {T}} \mathbb {E} \left[ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \right], \tag {2}
$$
where the conditional expectation indicates that the oracle first chooses a contextual policy  $h_t$  for all users that specifies which source  $k_t = h_t(z_t)$  to select as a function of the variables  $z_t$ . It then observes all contexts  $c_{tk_t}$  and makes for all  $t$  the decisions  $x_t$  based on that.
Denoting ALG the expected penalized utility of an online algorithm, its expected regret is:
$$
\operatorname {R e g} = \operatorname {O P T} - \mathbb {E} [ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) ] = \operatorname {O P T} - \operatorname {A L G}. \tag {3}
$$
We remark that the benchmark of Equation (2) allows choosing different sources of information for different users with the same public information  $z_{t}$ . As such, it differs from classical benchmarks in the contextual bandit literature that compare the performance of an algorithm to the best static choice of arm per context and whose performance would be
$$
\text {s t a t i c - O P T} := \max  _ {h \in [ K ] ^ {\mathcal {Z}}} \mathbb {E} \left[ \max  _ {\boldsymbol {x} \in \{0, 1 \} ^ {T}} \mathbb {E} \left[ \mathcal {U} (\boldsymbol {k}, \boldsymbol {x}) \mid c _ {1 k _ {1}}, \dots , c _ {T k _ {T}} \right] \right], \tag {4}
$$
where the  $h \in [K]^{\mathcal{Z}}$  policy that maps the public information to a source selection is the same for all users.
The benchmark (4) is the typical benchmark in contextual multi-armed bandits, as the global impact of decisions at different epochs and for different contexts are independent. However, this is no longer the case with the unfairness penalty  $R(\cdot)$  that requires coupling all decisions:
Proposition 2.1. There exist an instance of the problem and a constant  $b > 0$  such that for all  $T$ :
$$
\text {s t a t i c} - \mathrm {O P T} + b T <   \mathrm {O P T}.
$$
This result shows that an algorithm that only tries to identify the best source will have a linear regret compared to OPT. This indicates that the problem of source selection and fairness are strongly coupled, even without public information available, and cannot be solved through some sort of two-phase algorithm where each problem is solved separately. The proof of this result is presented in Appendix F.1. In this paper, our primary emphasis lies in the examination of the performance disparity between an online algorithm and the offline optimum. Nevertheless, we provide supplementary experiments in Appendix F.3 that investigate how variations in the prices  $p_k$  and the penalty  $R$  impact the solution of the offline optimum.
3 Algorithm and Regret Bounds
In this part, we present our online allocation algorithm and its regret bound. For clarity of exposition, we first present the algorithm in the case  $|\mathcal{Z}| = 1$ , i.e., without public information available (and thus we remove  $z_{t}$  from the algorithm). The extension to  $|\mathcal{Z}| > 1$  uses similar arguments and is discussed in Section 3.5.
3.1 Overview of the algorithm
Algorithm 1 devised to solve this problem is composed of two parts: a bandit algorithm for the source selection, and a gradient descent to adjust the penalty regularization term. This requires a dual parameter  $\lambda_t \in \mathbb{R}^d$  that is used to perform the source selection as well as the allocation decision  $x_t$ .
The intuition is the following: the performance of each source is evaluated through some "dual value" for a given dual parameter  $\lambda$ . The optimal primal performance is equal to the dual value when it is minimized in  $\lambda$ , because  $R$  is convex and randomized combinations of sources is allowed thus there is no duality gap. Hence the dual value of each source is iteratively evaluated in order to select the best source, and simultaneously minimize the dual value of the selected source through  $\lambda$ , so that the source selected is indeed optimal.
Bandit part. For the source selection, we use the EXP3 algorithm (see Chapter 11 of Lattimore & Szepesvári (2020)) on a virtual reward that depends on the dual parameter. Given a dual parameter  $\lambda \in \mathbb{R}^d$  and a context  $c_{tk}$ , we define the virtual reward as
$$
\varphi \left(\lambda_ {t}, c _ {t k}, k\right) = \max  \left(\mathbb {E} \left[ u _ {t} \mid c _ {t k} \right] - \langle \lambda_ {t}, \mathbb {E} \left[ a _ {t} \mid c _ {t k} \right] \rangle , 0\right) - p _ {k}, \tag {5}
$$
where  $\langle \lambda_t, \mathbb{E}[a_t \mid c_{tk}] \rangle$  denotes the scalar product between  $\lambda_t$  and  $\mathbb{E}[a_t \mid c_{tk}]$ . To compute this expectation, one needs to know the quantities  $p_k$ ,  $\mathbb{E}[u_t | c_{tk}]$  and  $\mathbb{E}[a_t | c_{tk}]$ .
To apply EXP3, a key property is to ensure that the virtual rewards are bounded, which requires  $\lambda_{t}$  to remain bounded. As we show in Lemma 3.2, this is actually guaranteed by the design of the gradient descent on  $\lambda$ . This lemma implies that there exists  $m \in \mathbb{R}^{+}$  such that  $|\varphi(\lambda_{t}, c_{tk})| \leq m$  for all  $t$  and  $k$ . Let us denote by  $\pi_{tk}$  the probability that source  $k$  is chosen at time  $t$  and  $k_{t} \sim \pi_{t}$ . We define the importance-weighted unbiased estimator vector  $\hat{\varphi}(\lambda, c_{t}) \in \mathbb{R}^{K}$  where each coordinate  $k' \in [K]$  is:
$$
\hat {\varphi} (\lambda , c _ {t k}, k, k ^ {\prime}) = m - \mathbb {1} [ k ^ {\prime} = k ] \frac {m - \varphi (\lambda , c _ {t k} , k)}{\pi_ {t k}}.
$$