Fair Exploration via Axiomatic Bargaining

Motivated by the consideration of fairly sharing the cost of exploration between multiple groups in learning problems , we develop the Nash bargaining solution in the context of multi-armed bandits . Specifically , the ‘ grouped ’ bandit associated with any multi-armed bandit problem associates , with each time step , a single group from some finite set of groups . The utility gained by a given group under some learning policy is naturally viewed as the reduction in that group ’ s regret relative to the regret that group would have incurred ‘ on its own ’ . We derive policies that yield the Nash bargaining solution relative to the set of incremental utilities possible under any policy . We show that on the one hand , the ‘ price of fairness ’ under such policies is limited , while on the other hand , regret optimal policies are arbitrarily unfair under generic conditions . Our theoretical development is complemented by a case study on contextual bandits for warfarin dosing where we are concerned with the cost of exploration across multiple races and age groups . 1 Introduction . Exploration in learning problems has an implicit cost , insomuch that exploring actions that are eventually revealed to be sub-optimal incurs regret . We study how this cost of exploration is shared in a system with multiple stakeholders . At the outset , we present two motivating examples . Personalized Medicine and Adaptive Trials : Multi-stage , adaptive designs [ 1 , 2 , 3 , 4 ] , are widely viewed as a frontier in clinical trials . More generally , the ability to collect detailed patient level data , and real time monitoring ( eg . glucose monitoring for diabetes [ 5 , 6 ] ) has raised the specter of learning personalized treatments . Among other formulations , such problems may be viewed as contextual bandits . For instance , for the problem of optimal warfarin dosing [ 7 ] , the context at each time step corresponds to a patient ’ s covariates , arms correspond to different dosages of warfarin , and the reward is the observed efficacy of the assigned dose . In examining such a study in retrospect , it is natural to measure the regret incurred by distinct groups of patients ( eg . by race or age ) . What makes a profile of regret across such groups fair or unfair ? Revenue Management for Search Advertising : Ad platforms enjoy a tremendous amount of flexibility in the the choice of ads served against search queries . Specifically , this flexibility exists both in selecting a slate of advertisers to compete for a specific search , and then in picking a winner from this slate . Now a key goal for the platform is learning the affinity of any given ad for a given search . In solving such a learning problem – for which many variants have been proposed [ 8 , 9 ] – we may again ask the question of who bears the cost of exploration , and whether the profile of such costs across various groups of advertisers is fair . ∗The full version of the paper can be found at https : //arxiv.org/abs/2106.02553 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Bandits , Groups and Axiomatic Bargaining . Delaying a formal development to later , any bandit problem has an associated ‘ grouped ’ variant . Specifically , we are given a finite set of groups ( eg . races or age groups in the warfarin example ) , and each group is associated with an arrival probability and a distribution over action sets . At each time step , a group and an action set is drawn from this distribution from which the learning algorithm must pick an action . Heterogeneity in groups is thus driven by differences in their respective distributions over feasible action sets . In addition to measuring overall regret , we also care about the regret incurred by specific groups , which we can view as the cost of exploration borne by that group . In reasoning about ‘ fair ’ regret profiles we turn to the theory of axiomatic bargaining . There , a central decision maker is concerned with the incremental utility earned by each group from collaborating , relative to the utility the group would earn on its own . Here this incremental utility is precisely the reduction in regret for any given group relative to the optimal regret that group would have incurred ‘ on its own ’ . A bargaining solution maximizes some objective function over the set of achievable incremental utilities . The utilitarian solution , for instance , maximizes the sum of incremental utilities which would reduce here to the usual objective of minimizing total regret . The Nash bargaining solution maximizes an alternative objective , the Nash Social Welfare ( SW ) function . This latter solution is the unique solution to satisfy a set of axioms any ‘ fair ’ solution would reasonably satisfy . This paper develops the Nash bargaining solution to the ( grouped ) bandit problem . 1.2 Contributions . In developing the Nash bargaining solution , we focus primarily on what is arguably the simplest non-trivial grouped bandit setting . Specifically , we consider the ‘ grouped ’ K-armed bandit model , wherein each group corresponds to a subset of the K arms . We make the following contributions relative to this problem : Regret Optimal Policies are Unfair ( Theorem 3.1 ) : We show that all regret optimal policies for the grouped K-armed bandit share a structural property that make them ‘ arbitrarily unfair ’ – in the sense that the Nash SW is−∞ for these solutions – under a broad set of conditions on the problem instance . Achievable Fairness ( Theorem 3.3 ) : We derive an instance-dependent upper bound on the Nash SW for the grouped K-armed bandit . This can be viewed as a ‘ fair ’ analogue to a regret lower bound ( e.g . [ 10 ] ) for the problem , since a lower bound on achievable regret ( forgoing any fairness concerns ) would in effect correspond to an upper bound on the utilitarian SW for the problem . Nash Solution ( Theorem 4.1 ) : We produce a policy that achieves the Nash solution . Specifically , the Nash SW under this policy achieves the upper bound we derive on the Nash SW for all instances of the grouped K-armed bandit . Price of Fairness for the Nash Solution ( Theorem 4.2 ) : We show that the ‘ price of fairness ’ for the Nash solution is small : if G is the number of groups , the Nash solution achieves at least O ( 1/ √ G ) of the reduction in regret achieved under a regret optimal solution relative to the regret incurred when groups operate separately . Taken together , these results establish a rigorous framework for the design of bandit algorithms that yield fair outcomes across groups at a low cost to total regret . As a final contribution , we extend our framework beyond the grouped K-armed bandit and undertake an empirical study : Linear Contextual Bandits and Warfarin Dosing : We extend our framework to grouped linear contextual bandits , yielding a candidate Nash solution there . Applied to a real-world dataset on warfarin dosing using race and age groups , we show ( a ) a regret optimal solution that ignores groups is dramatically unfair , and ( b ) the Nash solution balances out reductions in regret across groups at the cost of a small increase in total regret . 1.3 Related Literature . Two pieces of prior work have a motivation similar to our own . [ 11 ] studies a setting with multiple agents with a common bandit problem , where each agent can decide which action to take at each time . They show that ‘ free-riding ’ is possible — an agent that can access information from other agents can incur only O ( 1 ) regret in several classes of problems . This is consistent with our motivation . [ 12 ] studies a very similar grouped bandit model to ours , and provides a ‘ counterexample ’ in which a group can have a negative externality on another group . This example is somewhat pathological and stems from considering an instance-specific fixed time horizon ; instead , if T →∞ , all externalities become non-negative ( details in Appendix A.1 ) . Our grouped bandit model is also similar to sleeping bandits [ 13 ] , in which the set of available arms is adversarially chosen in each round . The known , fixed group structure in our model allows us to achieve tighter regret bounds than [ 13 ] . There have also been a handful of papers [ 14 , 15 , 16 , 17 ] that study ‘ fairness in bandits ’ in a completely different context . These works enforce a fairness criterion between arms , which is relevant in settings where a ‘ pull ’ represents some resource that is allocated to that arm , and these pulls should be distributed between arms in a fair manner . In these models , the decision maker ’ s objective ( maximize reward ) is distinct from that of a group ( obtain ‘ pulls ’ ) , unlike our setting ( and motivating examples ) where the groups and decision maker are aligned in their eventual objective . Our upper bound on Nash SW borrows classic techniques from the regret lower bound results of [ 10 ] and [ 18 ] . Our policy follows a similar pattern to recent work on regret-optimal , optimization-based policies for structured bandits [ 19 , 20 , 21 , 22 ] . Unlike those policies , our policy has no forced exploration . Further the optimization problem defining the Nash solution can generically have multiple solutions whereas the aforementioned approaches would require this solution to be unique ; our approach does not require a unique solution . Nonetheless , we believe that the framework in the aforementioned works can be fruitfully leveraged to construct Nash solutions for general grouped bandits , and we provide such a candidate solution as an extension . Our fairness framework is inspired by the literature on fairness in welfare economics — see [ 23 , 24 ] . Specifically , we study fairness in exploration through the lens of the axiomatic bargaining framework , first studied by [ 25 ] , who showed that enforcing four desirable axioms induces a unique fair solution . [ 26 ] is an excellent textbook reference for this topic . 2 The Axiomatic Bargaining Framework for Bandits . Let θ ∈ Θ be an unknown parameter and let A be the action set . For every arm a ∈ A , ( Yn ( a ) ) n≥1 is an i.i.d . sequence of rewards drawn from a distribution F ( θ , a ) parameterized by θ and a . We let µ ( a ) = E [ Y1 ( a ) ] be the expected reward of arm a . In defining a grouped bandit problem , we let G be a set of G groups . Each group g ∈ G is associated with a probability distribution P g over 2A , and a probability of arrival pg ; ∑ g pg = 1 . The identity of the group arriving at time t , gt , is chosen independently according to this latter distribution ; At is then drawn according to P gt . An instance of the grouped bandit problem is specified by I = ( A , G , p , P , F , θ ) , where all quantities except for θ are known . At each time t , a central decision maker observes gt and At , chooses an arm At ∈ At to pull and observes the reward YNt ( At ) +1 ( At ) , where Nt ( a ) is the total number of times arm a was pulled up to but not including time t. Let A∗t ∈ argmaxa∈At µ ( a ) be an optimal arm at time t. Given an instance I and a policy π , the total regret , and the group regret for group g ∈ G are respectively RT ( π , I ) = E [ T∑ t=1 ( µ ( A∗t ) − µ ( At ) ) ] and RgT ( π , I ) = E [ T∑ t=1 1 ( gt = g ) ( µ ( A ∗ t ) − µ ( At ) ) ] , where the expectation is over randomness in arrivals ( gt , At ) , rewards Yn ( a ) , and the policy π . Finally , so that the notion of an optimal policy for some class of instances , I , is well defined , we restrict attention to consistent policies which yield sub-polynomial regret for any instance in that class : Ψ = { π : RT ( π , I ) = o ( T b ) ∀I ∈ I , ∀b > 0 } .

Summary: This paper considers an asymptotic (in time) framework for analyzing fairness of grouped bandit policies through Nash social welfare. Focusing mainly on the grouped k-armed bandit setting, they show that regret-optimal policies are arbitrarily unfair. Following this, they show an upper bound to the nash social welfare, design a policy, PF-UCB, which achieves the social welfare upper bound, and prove bounds on the “price of fairness” for this policy, relative to the regret-optimal policy. Finally, they consider an extension to contextual linear bandits, proposing a possible solution in this setting, and evaluate their methods empirically.