SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abhishek Sinha, Ativ Joshi
School of Technology and Computer Science
Mumbai 400005, India
abhishek.sinha@tifr.res.in
ativ@cmi.ac.in
Rajarshi Bhattacharjee, Cameron Musco, Mohammad Hajiesmaili
Manning College of Information
and Computer Sciences
{rbhattacharj, cmusco,
hajiesmaili}@cs.umass.edu
We consider a fair resource allocation problem in the no-regret setting against an unrestricted adversary. The objective is to allocate resources equitably among several agents in an online fashion so that the difference of the aggregate  $\alpha$ -fair utilities of the agents achieved by an optimal static clairvoyant allocation and the online policy grows sublinearly with time. The problem inherits its difficulty from the non-separable nature of the global  $\alpha$ -fairness function. Previously, it was shown that no online policy could achieve a sublinear standard regret in this problem. In this paper, we propose an efficient online resource allocation policy, called Online Fair Allocation (OFA), that achieves sublinear  $c_{\alpha}$ -approximate regret with approximation factor  $c_{\alpha} = (1 - \alpha)^{-(1 - \alpha)} \leq 1.445$ , for  $0 \leq \alpha < 1$ . The upper bound on the  $c_{\alpha}$ -regret for this problem exhibits a surprising phase transition phenomenon - transitioning from a power-law to a constant at the critical exponent  $\alpha = 1/2$ . Our result also resolves an open problem in designing an efficient no-regret policy for the online job scheduling problem in certain parameter regimes. Along the way, we introduce new algorithmic and analytical techniques, including greedy estimation of the future gradients for non-additive global reward functions and bootstrapping second-order regret bounds, which may be of independent interest.
The notion of algorithmic fairness refers to learning algorithms that guarantee fair predictions even when subjected to adversarially biased training data [Dwork et al., 2012]. Fairness has become a major criterion for designing and deploying large-scale learning algorithms that affect a diverse user base. Since the training data could be highly skewed in practice, it is essential to make minimal assumptions about the data-generating process and design provably robust and fair learning policies. Guaranteeing fairness becomes even more challenging in the online learning setup as there is no distinction between the training and test data, and no assumption is made on the input data sequence. As an example, consider an online recruitment campaign where a learning algorithm decides the target group (identified by, say, the tuple (race, gender, age)) to which an ad for a job vacancy is to be displayed. Suppose a revenue-maximizing recommendation algorithm concludes from past data that more revenue is generated by showing the ad to Group A compared to Group B. In that case, the ad-serving algorithm will eventually end up showing that ad exclusively to Group A while discriminating against Group B users of a potential job opportunity [Hao, 2019]. One of the overarching goals of this paper is to design efficient online learning policies that provably mitigate this algorithmic bias while maintaining efficiency irrespective of the past data seen so far (refer to Example B.1 in the Appendix for a closely-related scheduling problem considered in this paper).
Towards this goal, we consider a generic online fair resource allocation problem, which we call NOFRA (No-Regret Fair Resource Allocation). In this problem, a fixed set of resources needs to be equitably shared among  $m$  agents over multiple rounds. Note that fairness is a complex, multidimensional, and essentially subjective concept. Several metrics have been introduced in the literature to quantify the degree of fairness in resource allocation, including  $\alpha$ -fairness [Lan et al., 2010], proportional fairness [Kelly, 1997, Mo and Walrand, 2000], max-min fairness [Radunovic and Le Boudec, 2007, Nace and Pioro, 2008], and Jain's fairness index [Jain et al., 1984]. In this paper, we consider the problem of maximizing the  $\alpha$ -fairness function, in which the utility of each agent when allocated  $R$  units grows as  $R^{1 - \alpha}$ . The parameter  $\alpha$  is restricted to the interval  $[0,1)$ . This same range of  $\alpha$  has been studied earlier in a game-theoretic set up by Altman et al. [2010].
On the hardness of the NOFRA problem: In the standard online learning setting, the cumulative reward accrued over a given time horizon is the sum of the rewards obtained at each round [Hazan, 2019]. In contrast, the objective in NOFRA is to maximize the global  $\alpha$ -fairness function, which is equal to the sum of cumulative rewards of each agent raised to the power  $1 - \alpha, 0 \leq \alpha < 1$ . Due to the power-law non-linearity, the objective function of NOFRA is non-separable with respect to time. Note that the non-separability of the objective function is essential to induce fairness in sequential allocations by incorporating the diminishing return property. However, this renders the NOFRA problem fundamentally different from standard online learning problems. In particular, Theorem 2 proves a non-trivial lower bound to the approximation ratio achievable by any online learning policy for this problem. This lower bound does not hold in classic settings with additive rewards.
Our contributions: We show that despite its non-separable structure, the NOFRA problem can be approximately reduced to an online linear optimization problem with a greedily defined sequence of reward vectors. The resulting problem can then be solved with an online convex optimization (OCO) policy with a second-order regret bound. In particular, we make the following contributions:
- In Algorithm 1, we present an efficient online resource allocation policy, Online Fair Allocation (OFA), that approximately maximizes the aggregate  $\alpha$ -fairness function of the agents. We show that OFA achieves  $(1 - \alpha)^{-(1 - \alpha)} \leq 1.445$ -approximate sublinear regret (Theorem 1). To the best of our knowledge, OFA is the first online policy that approximately maximizes the  $\alpha$ -fairness against any adversary, which could even be adaptive.
- In Theorem 2, we establish a lower bound to the approximation factor  $c_{\alpha}$  achievable by any online learning policy for the  $\alpha$ -fair reward function. Our lower bound improves upon the prior best-known lower bound of Si Salem et al. [2022], which established only that  $c_{\alpha} > 1$ .
- Technically, our proposed OFA policy optimizes the non-additive global reward function by greedily estimating the future gradients and using these estimates to construct an online linear optimization problem such that sublinear regret for this linear problem translates into sublinear approximate-regret for the global optimization problem. The linear optimization problem is then solved with an online gradient ascent policy with adaptive step sizes. The resulting algorithm is simple and intuitive: we have weights on each user that decay as the user's cumulative reward increases.
- On the analytical side, a key challenge is that the online linear optimization problem has a demand sequence (and hence, a gradient sequence) that depends on our past actions. We introduce a new proof technique that controls the norm of the gradients in this setting so that a tight second-order regret bound can be established. This technique applies generally to problems with states where the future gradients depend on past actions.
- We provide numerical simulation results supporting our theoretical conclusions and demonstrating the effectiveness of our approach in several applications of the NOFRA problem.
Related work: Several closely related works with non-additive reward functions appear in the online learning literature [Si Salem et al., 2022, Even-Dar et al., 2009, Rakhlin et al., 2011]. In the following, we explain why the NOFRA problem is fundamentally different from the existing studies. An extended literature review is given in Appendix A.
Closest to our work, Si Salem et al. [2022] considered the problem of designing fair online resource allocation policies. The authors showed that it is impossible to design a no-regret policy without restricting the set of admissible adversarial demand sequences. Given this negative result, the authors
proposed a no-regret policy using a primal-dual framework under the assumption of a restricted adversary, which exhibits an essentially i.i.d. stochastic-like fluctuation. In contrast, we design a robust policy with an approximate sublinear regret without restricting the adversary. Due to the significantly weaker assumption, our proposed policy and its analysis are drastically different from that of Si Salem et al. [2022]. Even-Dar et al. [2009] considered maximizing a global concave objective function in the no-regret setup under the further assumption that the optimal offline reward is convex. They presented an approachability-based policy in this setting and also proved the impossibility of achieving sublinear regret when the convexity condition is violated. Although the objective function in NOFRA is concave, we show that the optimal static offline reward function fails to be convex (see Appendix C.8). Hence, the policy of Even-Dar et al. [2009] does not apply to the NOFRA problem. Rakhlin et al. [2011] considered the problem of no-regret learnability for a wide class of non-additive global functions. However, their results also do not apply to our setting as our problem is not no-regret learnable (see Theorem 2). Agrawal et al. [2016] considered the problem of maximizing a concave utility function with the bandit information feedback and derived an efficient policy. However, their  $\tilde{O}(\sqrt{T})$  regret bound holds only in the stochastic setting. Offline fair caching algorithms using the  $\alpha$ -fair utility function were proposed in Liu et al. [2020]. However, their policy cannot be used in the online setting due to the casually available information structure. Finally, the work by Paria and Sinha [2021], Bhattacharjee et al. [2020], Mukhopadhyay et al. [2022] considered the online caching problem with the linear utility function. However, due to the non-linearity, extending their work to the  $\alpha$ -fair utility function entails several technical challenges, which we resolve in this paper.
In this section, we give the general formulation of the NOFRA problem, along with examples of concrete resource allocation problems that fit into this framework.
Consider a set of agents among which a limited resource is to be fairly divided. Assume that the resource allocated to the  $i^{\text{th}}$  agent on the  $t^{\text{th}}$  round is represented by an  $N$ -dimensional non-negative vector  $y_{i}(t)$ , where  $N$  is arbitrary. On every round  $t$ , the  $i^{\text{th}}$  agent requests an  $N$ -dimensional non-negative demand (or, reward) vector  $x_{i}(t)$ . The demand vectors are revealed to an online allocation policy  $\pi$  at the end of each round. We make no assumption on the regularity of the demand vector sequence, which could be adversarial (c.f., Si Salem et al. [2022]). Before the  $N \times m$  dimensional aggregate demand matrix  $\pmb{x}(t) \equiv (x_{1}(t), x_{2}(t), \ldots, x_{m}(t))$  for round  $t$  is revealed, the online resource allocation policy  $\pi$  chooses a non-negative  $N \times m$  dimensional allocation matrix  $\pmb{y}(t) = (y_{1}(t), y_{2}(t), \ldots, y_{m}(t))$  from the set of all feasible allocations  $\Delta$ . The set of all feasible allocations is assumed to be convex (see Remark 2 below). The reward accrued by the agent  $i$  on round  $t$  is given by the inner-product  $\langle x_{i}(t), y_{i}(t) \rangle$ , which, without loss of generality, is assumed to be upper-bounded by one. The cumulative reward accrued by agent  $i$  by the end of round  $t$  is given by
$$
R _ {i} (t) = R _ {i} (t - 1) + \left\langle x _ {i} (t), y _ {i} (t) \right\rangle , \forall i, t, \tag {1}
$$
where we set the initial condition  $R_{i}(0) \equiv 1, \forall i$ . By iterating the above recursion, the cumulative reward  $R_{i}(t)$  can be alternatively expressed as follows:
$$
R _ {i} (t) = 1 + \sum_ {\tau = 1} ^ {t} \left\langle x _ {i} (\tau), y _ {i} (\tau) \right\rangle . \tag {2}
$$
We make two mild technical assumptions on the structure of the demand and allocation vectors.
Assumption 1.  $\delta \leq \| x_{i}(t)\|_{1}\leq 1,\forall i,t,$  for some constant  $\delta >0$
Assumption 2. Let  $\mathbf{1}_{N\times m}$  denote the  $N\times m$  all-1 matrix. Then  $\mu \mathbf{1}_{N\times m}\in \Delta$  for some constant  $\mu >0$ .
The above assumptions imply that it is possible to ensure a non-zero reward for all agents on all rounds. The assumptions are used in the regret analysis only (see Eq. (25)). Our proposed online policy is oblivious to the value of the parameters  $\delta$  and  $\mu$ .
The utility of any user for a cumulative reward of  $R$  is given by the concave  $\alpha$ -fair utility function  $\phi: \mathbb{R}_+ \to \mathbb{R}_+$ , defined as follows:
$$
\phi (R) = \phi_ {\alpha} (R) \equiv \frac {R ^ {1 - \alpha}}{1 - \alpha}, R \geq 0, \tag {3}
$$
<sup>1</sup>The rationale behind setting the initial cumulative rewards to non-zero values is to ensure that the utility function  $\phi$  remains continuously differentiable at any round  $t \geq 1$ .
for some constant  $0 \leq \alpha < 1$ . The fairness parameter  $\alpha$  induces a trade-off between the desired efficiency and fairness by incorporating a notion of diminishing return property in the global objective function. The static offline optimal allocation with larger  $\alpha$  leads to more equitable cumulative rewards [Bertsimas et al., 2012]. Setting  $\alpha = 0$  reduces the problem to the "unfair" online linear optimization problem. Our objective is to design an online resource allocation policy  $\pi \equiv \{\pmb{y}(t)\}_{t \geq 1}$  that minimizes the regret for maximizing the aggregate utilities of all users compared to any fixed offline resource allocation strategy  $\pmb{y}^{*} \in \Delta$ . To be precise, assume that the offline, fixed resource allocation  $\pmb{y}^{*}$  yields a cumulative reward of  $\pmb{R}^{*}(T)$  for a horizon of length  $T \geq 1$ . Then, our objective is to design a resource allocation policy which minimizes the  $c$ -approximate regret (which we refer to as  $c$ -regret for short) defined as:
$$
\operatorname {R e g r e t} _ {T} (c) \equiv \sum_ {i = 1} ^ {m} \phi \left(R _ {i} ^ {*} (T)\right) - c \sum_ {i = 1} ^ {m} \phi \left(R _ {i} (T)\right), \tag {4}
$$
for some small constant  $c \geq 1$ . In the case of standard regret ( $c = 1$ ), we drop the argument in the parenthesis. Note that, unlike the standard online convex optimization problem, in the NOFRA problem, the reward function is global, in the sense that it is non-separable across time [Even-Dar et al., 2009]. For this reason, it is necessary to consider the  $c$ -regret with  $c > 1$ , rather than the standard regret (i.e., with  $c = 1$ ). This will become clear from Theorem 2, where we prove an explicit lower bound to the approximation factor achievable by any online policy for the global  $\alpha$ -fair reward function. This lower bound implies a concrete lower bound on the achievable  $c$ . Our lower bound improves upon [Si Salem et al., 2022, Theorem 1], where it was shown that no online policy can achieve a sublinear standard regret for the NOFRA problem under an unrestricted adversary.
Remark 1: When  $\alpha > 1$ , the offline benchmark  $\phi(R_i^*(T))$  itself becomes  $O(1), \forall i$ . Hence, in this regime, a sublinear regret bound (4) becomes vacuous. Consequently, we restrict the fairness parameter  $\alpha$  to the interval  $[0,1)$ .
Remark 2: In the above problem definition, we assumed that the set of feasible allocations  $\Delta$  is convex and thus that there are no integrality constraints on the allocation, i.e., the components of the allocation matrix  $y(t)$  are allowed to be fractional. However, in many combinatorial resource allocation problems, the allocation vector is required to be integral. In this case, the feasible action set  $\Delta$  is naturally defined to be the convex hull of the integral actions. In Appendix C.6, we consider the setting with integrality constraints and extend our algorithm and analysis to design a randomized integral allocation policy with a sublinear regret bound.
Motivating examples: The statement of NOFRA is fairly general, and by suitably choosing the reward and allocation vectors, many standard resource allocation problems can be reduced to NOFRA. In what follows, we highlight the example of fair shared caching [Bhattacharjee et al., 2020, Paria and Sinha, 2021]. We refer the readers to Appendix B for additional examples of online job scheduling and online matching.
Figure 1: The online shared caching problem
<sup>2</sup>Some authors define the  $\alpha$ -fairness function with an extra additive term, i.e.,  $\phi_{\alpha}^{\prime}(R) = \frac{R^{1 - \alpha} - 1}{1 - \alpha}$  [Si Salem et al., 2022]. Note that, in the range  $0 \leq \alpha < 1$ , this alternative definition changes the regret metric (4) only by an additive constant. We use the fairness function  $\phi_{\alpha}(R)$  as it is positively homogeneous of degree  $(1 - \alpha)$  - a property that we exploit in our analysis.
3Our theoretical results and algorithms trivially extend to the  $\pmb{w}$  -weighted fairness function  $\sum_{i}w_{i}\phi (R_{i}(T))$  for some non-negative weight vector  $w_{i}\geq 0,\forall i$
In the Online Shared Caching problem with a single shared cache,  $m$  users are connected to a single cache of capacity  $k$  (see Figure 1 for a schematic). At each round, each user requests a file from a library of size  $N$ . The file request sequence may be adversarial. At the beginning of each round  $t$ , an online caching policy prefetches at most  $k$  files on the cache, denoted by the vector  $y(t)$  such that
$$
\sum_ {i = 1} ^ {N} y _ {i} (t) = k, \quad 0 \leq y _ {i} (t) \leq 1, \forall i \in [ N ]. \tag {5}
$$
The set of all feasible caching configurations is denoted by  $\Delta_k^N$ . Immediately after the prefetching at round  $t$ , each user  $i$  reveals its file request, represented by the one-hot encoded demand vector  $x_i(t)$ ,  $1 \leq i \leq m$ . A special case of the above caching model for a single user ( $m = 1$ ) has been investigated in previous work [Bhattacharjee et al., 2020, Mhaisen et al., 2022, Joshi and Sinha, 2022].
Let  $R_{i}(t)$  denote the cumulative (fractional) hits obtained by the  $i^{\mathrm{th}}$  user up to round  $t \geq 1$ . We have
$$
R _ {i} (T) = 1 + \sum_ {t = 1} ^ {T} \left\langle x _ {i} (t), y (t) \right\rangle , \quad 1 \leq i \leq m. \tag {6}
$$
The online shared caching problem can be easily reduced to an instance of the NOFRA problem by taking the demand matrix to be  $\pmb{x}(t) = (x_1(t), x_2(t), \dots, x_m(t))$ . Since the allocation vector  $y(t)$  is common to all users, the allocation matrix can be taken to be  $\pmb{y}(t) = (y(t), y(t), \dots, y(t))$ . It can be observed that Assumption 2 holds in this case with  $\mu = k / N$  by noting that  $k / N \mathbf{1}_N \in \Delta_k^N$ .
3 Designing an approximately-no-regret policy for NOFRA
In view of the difficulty outlined in Section 1, the design and analysis of the Online Fair Allocation (OFA) policy consists of two parts. First, in Lemma 1, we show that by greedily estimating the terminal gradient with the current gradients at each round, the  $(1 - \alpha)^{-(1 - \alpha)}$ -approximate regret of the NOFRA problem can be upper bounded by the standard regret of a surrogate online linear optimization problem with a policy-dependent gradient sequence. Note that the regret of the surrogate problem depends on the norms of the gradients, which, in turn, depends on the policy. We use the online gradient ascent policy with adaptive step sizes to solve the surrogate problem while simultaneously controlling the norm of the gradients. The following section details our technique.
3.1 Reducing the NOFRA problem to an Online Linear Optimization Problem with policy-dependent subgradients
Since the  $\alpha$ -fairness function  $\phi : \mathbb{R}_{>0} \to \mathbb{R}_{>0}$  is concave and continuously differentiable, we have that for any  $x, y > 0$ :
$$
\phi (x) - \phi (y) \leq \phi^ {\prime} (y) (x - y). \tag {7}
$$
Let  $\beta \geq 1$  be a constant, which will be fixed later. Taking  $x = R_{i}^{*}(T)$  and  $y = \beta R_{i}(T)$  in the above inequality, we have
$$
\begin{array}{l} \phi \big (R _ {i} ^ {*} (T) \big) - \beta^ {1 - \alpha} \phi \big (R _ {i} (T) \big) \stackrel {{(a)}} {{=}} \phi \big (R _ {i} ^ {*} (T) \big) - \phi \big (\beta R _ {i} (T) \big), \\ \stackrel {(b)} {\leq} \phi^ {\prime} (\beta R _ {i} (T)) (R _ {i} ^ {*} (T) - \beta R _ {i} (T)) \\ \stackrel {(c)} {\leq} \quad \beta^ {- \alpha} \phi^ {\prime} (R _ {i} (T)) \sum_ {t = 1} ^ {T} \left\langle x _ {i} (t), y _ {i} ^ {*} - \beta y _ {i} (t) \right\rangle , \tag {8} \\ \end{array}
$$
where in (a), we have used the positive homogeneity property of the fairness function  $\phi (\beta x) = \beta^{1 - \alpha}\phi (x)$ , in (b), we have used the concavity of the fairness function  $\phi (\cdot)$  from Eqn. (7), and, finally, in (c), we have used the definition of cumulative rewards (2), the fact that  $\beta \geq 1$ , and the homogeneity of the function  $\phi (\cdot)$ , which gives  $\phi^{\prime}(\beta x) = \beta^{-\alpha}\phi^{\prime}(x)$ .
<sup>4</sup>In the above, we allow fractional caching, which can be easily converted to a randomized integral caching policy (where  $y_{i}(t) \in \{0,1\}$ ,  $\forall i,t$ ) via sampling. See Section C.6 for details.
Summing up the bound (8) over all agents  $i \in [m]$ , we obtain the following upper bound to the  $\beta^{1 - \alpha}$ -Regret of any online policy for the NOFRA problem:
$$
\operatorname {R e g r e t} _ {T} \left(\beta^ {1 - \alpha}\right) \leq \beta^ {- \alpha} \sum_ {t = 1} ^ {T} \left(\sum_ {i} \left\langle \phi^ {\prime} \left(R _ {i} (T)\right) x _ {i} (t), y _ {i} ^ {*} - \beta y _ {i} (t) \right\rangle\right). \tag {9}
$$
It is critical to note that each of the quantities  $\phi^{\prime}(R_i(T))$  depends on the entire sequence of past actions  $\{y_t\}_{t = 1}^T$ . Hence,  $\phi^{\prime}(R_i(T))$  in (9) is not known to the online policy when it takes its actions. More importantly, the value of the coefficient  $\phi^{\prime}(R_i(T))$  depends on the future actions and requests through (1). Hence, it is impossible to know the value of this coefficient in an online fashion. To get around this fundamental difficulty, we now consider a surrogate regret-minimization problem by replacing the term  $\phi^{\prime}(R_i(T))$  by its causal counterpart  $\phi^{\prime}(R_i(t - 1))$  at the beginning of round  $t$ . In other words, we now design an online learning policy that minimizes the regret of the following online linear optimization problem:
$$
\operatorname {L} - \operatorname {R e g r e t} _ {T} \equiv \sum_ {t = 1} ^ {T} \left(\sum_ {i} \left\langle \phi^ {\prime} \left(R _ {i} (t - 1)\right) x _ {i} (t), y _ {i} ^ {*} - y _ {i} (t) \right\rangle\right). \tag {10}
$$
To recapitulate the information structure, recall that at the beginning of round  $t$ , the cumulative reward vector  $R(t - 1)$ , and hence the coefficient  $\phi'(R_i(t - 1))$ , is known. Still, the current request vector  $x(t)$  is unknown to the online learner before it makes its allocation decision  $y_t$  on the  $t^{\text{th}}$  round. We now establish the following key lemma that relates the original regret bound (9) to the regret bound (10) for the surrogate problem.
Lemma 1. Consider any arbitrary online resource allocation policy and assume that the utility function of each user is given by the  $\alpha$ -fair utility function (3). Then, for any horizon of length  $T \geq 1$ , we have
$$
\operatorname {R e g r e t} _ {T} \left(c _ {\alpha}\right) \leq (1 - \alpha) ^ {\alpha} \cdot L - \operatorname {R e g r e t} _ {T} + c _ {\alpha} m, \tag {11}
$$
where  $c_{\alpha}\equiv (1 - \alpha)^{-(1 - \alpha)}\leq e^{1 / e} <   1.445$