Fair Exploration via Axiomatic Bargaining

Motivated by the consideration of fairly sharing the cost of exploration between multiple groups in learning problems , we develop the Nash bargaining solution in the context of multi-armed bandits . Specifically , the ‘ grouped ’ bandit associated with any multi-armed bandit problem associates , with each time step , a single group from some finite set of groups . The utility gained by a given group under some learning policy is naturally viewed as the reduction in that group ’ s regret relative to the regret that group would have incurred ‘ on its own ’ . We derive policies that yield the Nash bargaining solution relative to the set of incremental utilities possible under any policy . We show that on the one hand , the ‘ price of fairness ’ under such policies is limited , while on the other hand , regret optimal policies are arbitrarily unfair under generic conditions . Our theoretical development is complemented by a case study on contextual bandits for warfarin dosing where we are concerned with the cost of exploration across multiple races and age groups . 1 Introduction . Exploration in learning problems has an implicit cost , insomuch that exploring actions that are eventually revealed to be sub-optimal incurs regret . We study how this cost of exploration is shared in a system with multiple stakeholders . At the outset , we present two motivating examples . Personalized Medicine and Adaptive Trials : Multi-stage , adaptive designs [ 1 , 2 , 3 , 4 ] , are widely viewed as a frontier in clinical trials . More generally , the ability to collect detailed patient level data , and real time monitoring ( eg . glucose monitoring for diabetes [ 5 , 6 ] ) has raised the specter of learning personalized treatments . Among other formulations , such problems may be viewed as contextual bandits . For instance , for the problem of optimal warfarin dosing [ 7 ] , the context at each time step corresponds to a patient ’ s covariates , arms correspond to different dosages of warfarin , and the reward is the observed efficacy of the assigned dose . In examining such a study in retrospect , it is natural to measure the regret incurred by distinct groups of patients ( eg . by race or age ) . What makes a profile of regret across such groups fair or unfair ? Revenue Management for Search Advertising : Ad platforms enjoy a tremendous amount of flexibility in the the choice of ads served against search queries . Specifically , this flexibility exists both in selecting a slate of advertisers to compete for a specific search , and then in picking a winner from this slate . Now a key goal for the platform is learning the affinity of any given ad for a given search . In solving such a learning problem – for which many variants have been proposed [ 8 , 9 ] – we may again ask the question of who bears the cost of exploration , and whether the profile of such costs across various groups of advertisers is fair . ∗The full version of the paper can be found at https : //arxiv.org/abs/2106.02553 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1.1 Bandits , Groups and Axiomatic Bargaining . Delaying a formal development to later , any bandit problem has an associated ‘ grouped ’ variant . Specifically , we are given a finite set of groups ( eg . races or age groups in the warfarin example ) , and each group is associated with an arrival probability and a distribution over action sets . At each time step , a group and an action set is drawn from this distribution from which the learning algorithm must pick an action . Heterogeneity in groups is thus driven by differences in their respective distributions over feasible action sets . In addition to measuring overall regret , we also care about the regret incurred by specific groups , which we can view as the cost of exploration borne by that group . In reasoning about ‘ fair ’ regret profiles we turn to the theory of axiomatic bargaining . There , a central decision maker is concerned with the incremental utility earned by each group from collaborating , relative to the utility the group would earn on its own . Here this incremental utility is precisely the reduction in regret for any given group relative to the optimal regret that group would have incurred ‘ on its own ’ . A bargaining solution maximizes some objective function over the set of achievable incremental utilities . The utilitarian solution , for instance , maximizes the sum of incremental utilities which would reduce here to the usual objective of minimizing total regret . The Nash bargaining solution maximizes an alternative objective , the Nash Social Welfare ( SW ) function . This latter solution is the unique solution to satisfy a set of axioms any ‘ fair ’ solution would reasonably satisfy . This paper develops the Nash bargaining solution to the ( grouped ) bandit problem . 1.2 Contributions . In developing the Nash bargaining solution , we focus primarily on what is arguably the simplest non-trivial grouped bandit setting . Specifically , we consider the ‘ grouped ’ K-armed bandit model , wherein each group corresponds to a subset of the K arms . We make the following contributions relative to this problem : Regret Optimal Policies are Unfair ( Theorem 3.1 ) : We show that all regret optimal policies for the grouped K-armed bandit share a structural property that make them ‘ arbitrarily unfair ’ – in the sense that the Nash SW is−∞ for these solutions – under a broad set of conditions on the problem instance . Achievable Fairness ( Theorem 3.3 ) : We derive an instance-dependent upper bound on the Nash SW for the grouped K-armed bandit . This can be viewed as a ‘ fair ’ analogue to a regret lower bound ( e.g . [ 10 ] ) for the problem , since a lower bound on achievable regret ( forgoing any fairness concerns ) would in effect correspond to an upper bound on the utilitarian SW for the problem . Nash Solution ( Theorem 4.1 ) : We produce a policy that achieves the Nash solution . Specifically , the Nash SW under this policy achieves the upper bound we derive on the Nash SW for all instances of the grouped K-armed bandit . Price of Fairness for the Nash Solution ( Theorem 4.2 ) : We show that the ‘ price of fairness ’ for the Nash solution is small : if G is the number of groups , the Nash solution achieves at least O ( 1/ √ G ) of the reduction in regret achieved under a regret optimal solution relative to the regret incurred when groups operate separately . Taken together , these results establish a rigorous framework for the design of bandit algorithms that yield fair outcomes across groups at a low cost to total regret . As a final contribution , we extend our framework beyond the grouped K-armed bandit and undertake an empirical study : Linear Contextual Bandits and Warfarin Dosing : We extend our framework to grouped linear contextual bandits , yielding a candidate Nash solution there . Applied to a real-world dataset on warfarin dosing using race and age groups , we show ( a ) a regret optimal solution that ignores groups is dramatically unfair , and ( b ) the Nash solution balances out reductions in regret across groups at the cost of a small increase in total regret . 1.3 Related Literature . Two pieces of prior work have a motivation similar to our own . [ 11 ] studies a setting with multiple agents with a common bandit problem , where each agent can decide which action to take at each time . They show that ‘ free-riding ’ is possible — an agent that can access information from other agents can incur only O ( 1 ) regret in several classes of problems . This is consistent with our motivation . [ 12 ] studies a very similar grouped bandit model to ours , and provides a ‘ counterexample ’ in which a group can have a negative externality on another group . This example is somewhat pathological and stems from considering an instance-specific fixed time horizon ; instead , if T →∞ , all externalities become non-negative ( details in Appendix A.1 ) . Our grouped bandit model is also similar to sleeping bandits [ 13 ] , in which the set of available arms is adversarially chosen in each round . The known , fixed group structure in our model allows us to achieve tighter regret bounds than [ 13 ] . There have also been a handful of papers [ 14 , 15 , 16 , 17 ] that study ‘ fairness in bandits ’ in a completely different context . These works enforce a fairness criterion between arms , which is relevant in settings where a ‘ pull ’ represents some resource that is allocated to that arm , and these pulls should be distributed between arms in a fair manner . In these models , the decision maker ’ s objective ( maximize reward ) is distinct from that of a group ( obtain ‘ pulls ’ ) , unlike our setting ( and motivating examples ) where the groups and decision maker are aligned in their eventual objective . Our upper bound on Nash SW borrows classic techniques from the regret lower bound results of [ 10 ] and [ 18 ] . Our policy follows a similar pattern to recent work on regret-optimal , optimization-based policies for structured bandits [ 19 , 20 , 21 , 22 ] . Unlike those policies , our policy has no forced exploration . Further the optimization problem defining the Nash solution can generically have multiple solutions whereas the aforementioned approaches would require this solution to be unique ; our approach does not require a unique solution . Nonetheless , we believe that the framework in the aforementioned works can be fruitfully leveraged to construct Nash solutions for general grouped bandits , and we provide such a candidate solution as an extension . Our fairness framework is inspired by the literature on fairness in welfare economics — see [ 23 , 24 ] . Specifically , we study fairness in exploration through the lens of the axiomatic bargaining framework , first studied by [ 25 ] , who showed that enforcing four desirable axioms induces a unique fair solution . [ 26 ] is an excellent textbook reference for this topic . 2 The Axiomatic Bargaining Framework for Bandits . Let θ ∈ Θ be an unknown parameter and let A be the action set . For every arm a ∈ A , ( Yn ( a ) ) n≥1 is an i.i.d . sequence of rewards drawn from a distribution F ( θ , a ) parameterized by θ and a . We let µ ( a ) = E [ Y1 ( a ) ] be the expected reward of arm a . In defining a grouped bandit problem , we let G be a set of G groups . Each group g ∈ G is associated with a probability distribution P g over 2A , and a probability of arrival pg ; ∑ g pg = 1 . The identity of the group arriving at time t , gt , is chosen independently according to this latter distribution ; At is then drawn according to P gt . An instance of the grouped bandit problem is specified by I = ( A , G , p , P , F , θ ) , where all quantities except for θ are known . At each time t , a central decision maker observes gt and At , chooses an arm At ∈ At to pull and observes the reward YNt ( At ) +1 ( At ) , where Nt ( a ) is the total number of times arm a was pulled up to but not including time t. Let A∗t ∈ argmaxa∈At µ ( a ) be an optimal arm at time t. Given an instance I and a policy π , the total regret , and the group regret for group g ∈ G are respectively RT ( π , I ) = E [ T∑ t=1 ( µ ( A∗t ) − µ ( At ) ) ] and RgT ( π , I ) = E [ T∑ t=1 1 ( gt = g ) ( µ ( A ∗ t ) − µ ( At ) ) ] , where the expectation is over randomness in arrivals ( gt , At ) , rewards Yn ( a ) , and the policy π . Finally , so that the notion of an optimal policy for some class of instances , I , is well defined , we restrict attention to consistent policies which yield sub-polynomial regret for any instance in that class : Ψ = { π : RT ( π , I ) = o ( T b ) ∀I ∈ I , ∀b > 0 } .

Summary: This paper studies the potential ‘unfairness’ due to exploration in the ‘grouped’ bandit setting. In this setting, the utility gained by a given group under some learning policy is considered as the reduction in that group’s regret relative to the regret that group would have incurred ‘on its own. This paper derives policies that yield the Nash bargaining solution relative to the set of incremental utilities possible under other policies. Corresponding theoretical development is also provided, including an analysis on the unfairness of regret optimal policies, the optimality of the new solution ‘proportionally fair’ UCB, and interesting analysis on the price of fairness. An extension to grouped contextual linear bandits is also provided.