A new look at fairness in stochastic multi-armed bandit problems

1 INTRODUCTION . The multi-armed bandit ( MAB ) problem is a classical framework for sequential decision-making in uncertain environments . Starting with the seminal work of Robbins ( 1952 ) , over the years , a significant body of work has been developed to address both theoretical aspects and practical applications of this problem . In a traditional stochastic multi-armed bandit ( MAB ) problem ( Lai & Robbins , 1985 ; Auer et al. , 2002 ; Vermorel & Mohri , 2005 ; Bubeck & Cesa-Bianchi , 2012 ) , a learner has access to K arms and pulling arm k generates a stochastic reward for the principal from an unknown distribution Fk with an unknown expected reward µk . If the mean rewards were known as prior information , the learner could just repeatedly pull the best arm given by k∗ = arg maxk µk . However , the learner has no such knowledge of the reward of each arm . Hence , one should use some learning algorithm π which operates in rounds , pulls arm πt ∈ { 1 , . . . , K } in round t , observes the stochastic reward generated from reward distribution Fπt , and uses that information to learn the best arm over time . The performance of learning algorithm π is evaluated based on its cumulative regret over time horizon T , defined as R̄π ( T ) = µk∗T − T∑ t=1 Eµπt . ( 1 ) To achieve the minimum regret , a good learner should make a balance between exploration ( pulling different arms to get more information of reward distribution of each arm ) and exploitation ( pulling the arm currently believed to have the highest reward ) . In addition to the above classical MAB problem , many variations of the MAB framework have been extensively studied in the literature recently . Various papers study MAB problems with additional constraints which include bandits with knapsack constraints ( Badanidiyuru et al. , 2013 ) , bandits with budget constraints ( Xia et al. , 2015 ) , sleeping bandits ( Kleinberg et al. , 2010 ; Chatterjee et al. , 2017 ) , etc . Except these , there is a huge research interest in fairness within machine learning field . Fairness has been a hot topic of many recent application tasks , including classification ( Zafar et al. , 2017a ; b ; Agarwal et al. , 2018 ; Roh et al. , 2021 ) , regression ( Berk et al. , 2017 ; Rezaei et al. , 2019 ) , recommendation ( Celis et al. , 2017 ; Singh & Joachims , 2018 ; Beutel et al. , 2019 ; Wang et al. , 2021 ) , resource allocation ( Baruah et al. , 1996 ; Talebi & Proutiere , 2018 ; Li et al. , 2020 ) , Markov decision process ( Khan & Goodridge , 2019 ) , etc . There are two popular definitions of fairness in the MAB literature . 1 ) . The fairness is introduced into the bandit learning framework by saying that it is unfair to preferentially choose one arm over another if the chosen arm has lower expected reward than the unchosen arm ( Joseph et al. , 2016 ) . In other words , the learning algorithm can not favor low-reward arms . 2 ) . The fairness is introduced such that the algorithm needs to ensure that uniformly ( i.e. , at the end of every round ) each arm is pulled at least a pre-specified fraction of times ( Patil et al. , 2020 ) . In other words , it imposes an additional constraint to prevent the algorithm from playing low-reward arms too few times . In this paper , we adopt a new perspective , e.g. , in addition to maximizing the cumulative expected reward , it also allows the user to specify how “ hard ” or how “ soft ” the fairness requirement on each arm is . In this view , it is not always easy even to formulate the problem and to introduce an appropriate notion of regret . We thus propose a new formulation of fairness MAB by introducing penalty termAk max ( τkT−Nk ( T ) , 0 ) , whereAk , τk are the penalty rate and fairness fraction for arm k and Nk ( T ) is the number of times pulling arm k. Hence it gives penalization when the algorithm fails to meet the fairness constraint and penalty term is proportional to the gap between pulling number and its required level . To solve this regularized MAB problem , we also propose a hard-threshold upper confidence bound ( UCB ) algorithm . It is similar to the classical UCB algorithm but adds an additional term to encourage the learner to favor those arms whose pulling numbers are below the required level at each round . The advantage of our approach is that it allows the user to distinguish , if desired , between arms for which is more important to sample an arm with required frequency and those arms for which it is less important to do so . To the best of our knowledge , there is no work on mathematical framework of fairness MAB with regularization term in the literature . In this paper , we provide a relatively complete theory for the fairness MAB . We rigorously formalize the penalized regret which can be used for evaluating the performance of learning algorithm under fairness constraints . On theoretical side , the hard-threshold UCB algorithm is proved to achieve asymptotic fairness when a large penalty rate is chosen . The algorithm is shown to obtain O ( log T ) regret when the sub-optimality gap is assumed to be fixed . Additionally , the characterization of fluctuation of non-fairness level , max1≤t≤T max ( τkt−Nk ( t ) , 0 ) is also given . Its magnitude is also shown to be O ( log T ) . Moreover , we establish a sub-optimal gap-free regret bound of proposed method and provide insights on how hard-threshold based UCB index works . We also point out that the analysis of proposed hard-threshold UCB algorithm is much harder than the classical UCB due to the existence of interventions between different sub-optimal arms . On numerical side , the experimental results confirm our theory and show that the performance of the proposed algorithm is better than other popular methods . Our method achieves a better trade-off between reward and fairness . Notations . For real number x , ( x ) + stands for max { 0 , x } ; bxc is the largest integer smaller or equal to x . For integer n , we use [ n ] to represent the set { 1 , . . . , n } . We say a = O ( b ) ; a = Ω ( b ) if there exists a constant C such that a ≤ Cb ; a ≥ b/C . The symbols E and P ( · ) denote generic expectation and probability under a probability measure that may be determined from the context . We let π be a generic policy / learning algorithm . 2 ACHIEVING FAIRNESS VIA PENALIZATION . Consider a stochastic multi-armed bandit problem with K arms and unknown expected rewards µ1 , . . . , µK associated with these arms . The notion of fairness we introduce consists of proportions τk ≥ 0 , k = 1 , . . . , K with τ1 + · · ·+ τK < 1 . We use T ∈ { 1 , 2 , . . . , } to denote the time horizon and Nk , π ( t ) to denote the number of times that arm k has been pulled by time t ∈ [ T ] using policy π . For notational simplicity , we may write Nk , π ( t ) as Nk ( t ) . It is desired to pull arm k at least at the uniform rate of τk , k = 1 , . . . , K. In other words , the learner should obey the constraint that Nk ( t ) ≥ τkt for any t ∈ [ T ] . Thus a good policy aims to solve the following optimization problem , arg max π E ∑ k µkNk , π ( T ) , subject to Nk , π ( t ) ≥ τkt for all k and t. ( 2 ) Instead of directly working with such a constrained bandit problem , we consider a penalization problem . That is , one gets penalized if the arm is not pulled sufficiently often . To reflect this , we introduce the following design problem . Let Sπ ( T ) be the sum of the rewards obtained by time t under policy π , i.e. , Sπ ( T ) = ∑T t=1 rπt where πt is the arm index chosen by policy π at time t ∈ [ T ] and rπt is the corresponding reward . Then the penalized total reward is defined as Spen , π ( T ) = Sπ ( T ) − K∑ k=1 Ak ( τkT −Nk , π ( T ) ) + , ( 3 ) where A1 , . . . , AK are known nonnegative penalty rates . Our goal is to design a learning algorithm to make the expectation of Spen , π ( T ) as large as possible . By taking the expectation , we have E [ Spen , π ( T ) ] = K∑ k=1 µkE [ Nk , π ( t ) ] − K∑ k=1 AkE [ ( τkT −Nk , π ( T ) ) + ] , ( 4 ) which is the penalized reward achieved by policy π and we would like to maximize it over π . Now we are ready to introduce the penalized regret function , which is the core for the regret analysis . To derive the new regret , we first note that maximizing E [ Spen , π ( T ) ] is the same as minimizing the following loss function , L ( T ) = µ∗T − E [ Spen , π ( T ) ] = K∑ k=1 [ ∆kE [ Nk ( t ) ] +AkE [ ( τkT −Nk ( T ) ) + ] ] , ( 5 ) where we denote µ∗ = max k=1 , ... , K µk , ∆k = µ ∗ − µk , k = 1 , . . . , K. In order to find the minimum possible value of L ( T ) , let us understand what a prophet ( who knows the expected rewards µ1 , . . . , µK ) would do . Clearly , a prophet ( who , in addition , is not constrained by integer value ) would solve the following optimization problem , min x1 , ... , xK K∑ k=1 [ ∆kxk +AkE ( τkT − xk ) + ] subject to K∑ k=1 xk = T , xk ≥ 0 , k = 1 , . . . , K , and pull arm k for xk times ( k = 1 , . . . , K ) . By denoting yk = xk/T , k ∈ [ K ] , we transform this problem into min y1 , ... , yK K∑ k=1 [ ∆kyk +Ak ( τk − yk ) + ] subject to K∑ k=1 yk = 1 , yk ≥ 0 , k = 1 , . . . , K. ( 6 ) We will solve the problem ( 6 ) by finding y1 , . . . , yK that satisfy the constraints and that minimize simultaneously each term in the objective function . It is not hard to observe the following facts . 1 . For A ≥ 0 , function y 7→ A ( τ − y ) + achieves its minimum value of 0 for y ≥ τ . 2 . For A ≥ ∆ > 0 , function y 7→ ∆y +A ( τ − y ) + achieves its minimum of ∆τ at y = τ . 3 . For ∆ > A ≥ 0 , function y 7→ ∆y +A ( τ − y ) + achieves its minimum of Aτ at y = 0 . As a result , we introduce the following three sets Aopt = { k ∈ [ K ] : µk = µ∗ } , Acr = { k ∈ [ K ] : Ak ≥ ∆k > 0 } , Anon−cr = { k ∈ [ K ] : ∆k > Ak } , where Aopt consists of all optimal arms , Acr consists of sub-optimal arms with penalty rate larger than ( or equal to ) the sub-optimal gap and Anon−cr includes sub-optimal arms with penalty rate smaller than the sub-optimal gap . Therefore an optimal solution to the problem ( 6 ) can be constructed as follows . Let k∗ be an arbitrary arm in Aopt , and choose yk = 1− ∑ j∈Acr∪ ( Aopt\ { k∗ } ) τj , k = k ∗ , τk , k ∈ Acr ∪ ( Aopt \ { k∗ } ) , 0 , k ∈ Anon−cr . ( 7 ) Therefore , a prophet would choose ( modulo rounding ) in ( 5 ) Nk ( T ) = ( 1− ∑ j∈Acr∪ ( Aopt\ { k∗ } ) τj ) T , k = k∗ , τkT , k ∈ Acr ∪ ( Aopt \ { k∗ } ) , 0 , k ∈ Anon−cr , ( 8 ) leading to the following optimal value of L ( T ) , L∗ ( T ) = ∑ k∈Acr ∆kτkT + ∑ k∈Anon−cr AkτkT = ( K∑ k=1 min ( ∆k , Ak ) τk ) T. ( 9 ) Given an arbitrary algorithm π , we can therefore define the penalized regret as Rπ ( T ) = Lπ ( T ) − L∗ ( T ) = ∑ k∈Aopt AkE ( τkT −Nk , π ( T ) ) + ( 10 ) + ∑ k∈Acr [ ∆kE ( Nk , π ( T ) − τkT ) +AkE ( τkT −Nk , π ( T ) ) + ] + ∑ k∈Anon−cr [ ∆kENk , π ( T ) +Ak ( E ( τkT −Nk , π ( T ) ) + − τkT ) ] .

Summary: This paper examines the problem of reward maximization in a Multi-Armed Bandit, with the additional fairness constraint that each arm should be pulled for at least a fraction of time. The authors propose and analyze an algorithm where the UCB index of an arm $i$   is increased by a specified amount (which is a parameter for the algorithm) $A_i$ for the time instances where the arm is lagging behind the fairness constraint. Results about the asymptotic satisfaction (or not) of the fairness constraints depending on the relationship between $A_i$ and the suboptimality gap of each arm are given, as well as results about the gap-dependent and gap-independent regret of the studied algorithm under a modified reward with the threshold (that is, a cost $A_i$ is given to all arms $i$ that do not satisfy the fairness constraints at time $t$). This algorithm is also compared with two recent state of the art algorithms for this problem in synthetic examples and illustrates superior performance.