(Almost) Free Incentivized Exploration from Decentralized Learning Agents

1 Introduction . Multi-armed bandits ( MAB ) is a simple yet powerful model for sequential decision making with an exploration-exploitation tradeoff ( Bubeck and Cesa-Bianchi , 2012 ; Lattimore and Szepesvári , 2020 ) . In standard MAB settings , one principal , who has a long-term system-level objective , takes charge of selecting and playing arms . However , such assumption does not always hold in reality . It is often the case that arm pulls are performed by multiple different agents whose individual goals are not aligned with the system , and the principal can only observe agents ’ actions . One typical example is the individual buyers ( agents ) and the online shopping platform ( the principal ) . Such scenarios complicate decision making and introduce significant difficulties to optimize the system performance . Incentivized exploration has been proposed to address this problem ( Frazier et al. , 2014 ; Mansour et al. , 2015 ) . Specifically , bonuses can be offered by the principal to incentivize agents to perform specific actions , e.g. , to explore their originally underrated arms . This framework provides an opportunity to reconcile different interests between the principal and agents . As a concrete example , the online shopping platform can offer discounts on certain items so that individual buyers would buy them and provide feedbacks , which can be used to optimize future strategies of the platform . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . While incentivized exploration has been investigated in the existing literature , we recognize two major limitations . First , almost all of the existing works assume the participating agents to be myopic , i.e. , they always choose the empirically best arm . Second , it is always assumed that at each time slot , one new agent participates in the system , i.e. , agents never stay or return . In other words , prior research mainly considers how to incentivize one single temporary myopic agent . These two assumptions largely limit the applicability of incentivized exploration . In this work , we extend the study of incentivized exploration beyond the aforementioned barriers , and investigate situations with multiple long-term strategic agents . In particular , we focus on the scenarios ( see Section 3.1 ) where the principal wants to identify the ( overall ) best arm whereas the heterogeneously involved agents only care about their different individual cumulative rewards . For such scenarios , the “ Observe-then-Incentivize ” ( OTI ) mechanism is proposed and several interesting observations are obtained . First , we find that strategic agents ’ intrinsic needs of learning can actually benefit principal ’ s exploration by providing “ free pulls ” . In other words , as opposed to myopic agents , the self interests of strategic agents can be exploited by the principal . Second , it turns out that increasing the number of participating agents can significantly mitigate the principal ’ s burden on incentivizing , which highlights the importance of increasing the population of agents . A crux of our findings is the following intriguing conceptual message : when there are sufficiently many learning agents involved , the exploration process of the principal could be ( almost ) free . Behind these findings , three novel technical components play critical roles in the design and analysis of OTI , all of which may have independent values . • First , a simple yet provably effective incentive-provision strategy is developed , which can efficiently regulate strategic agents ’ behaviors and serves as the foundation of the algorithm analysis . • Second , a best arm identification algorithm is carefully crafted to tackle the varying amounts of local information from heterogeneous agents . This setting itself is novel in best arm identification . • A high-probability finite-time lower bound of UCB algorithms ( Auer et al. , 2002 ) is proved , which contributes to a better understanding of the celebrated UCB . These insights and techniques are unique in incentivizing multiple long-term strategic agents , which may find applications in related problems , and encourage future research in this direction . 2 Related Works . Incentivized exploration . Since proposed by Frazier et al . ( 2014 ) ; Kremer et al . ( 2014 ) , many progresses have been made in incentivized exploration in MAB . Especially , there exist two lines of studies . The first one ( Kremer et al. , 2014 ; Mansour et al. , 2015 , 2016 ; Immorlica et al. , 2020 ; Sellke and Slivkins , 2020 ) assumes the principal can observe the full history while the agents can not , and the principal leverage such information advantage to perform incentivizing . The second line , which our setting follows , considers a publicly available history while the incentives are done through compensations . This idea is first introduced by Frazier et al . ( 2014 ) and generalized by Han et al . ( 2015 ) , both on Bayesian settings . The non-Bayesian case , as adopted in this work , is first studied by Wang and Huang ( 2018 ) , and recently extended by Liu et al . ( 2020 ) ; Wang et al . ( 2021 ) . However , the aforementioned works mainly consider that one new myopic agent enters the system at each time slot and leaves afterward . The only exception is Mansour et al . ( 2016 ) , where multiple but still temporary and myopic agents are considered . This work differs from them in considering multiple long-term strategic agents . In addition , another notable difference is that almost all prior works focus on regret minimization for the principal , instead of best arm identification . One important related work is Chen et al . ( 2018 ) , which studies temporary myopic agents with heterogeneous preferences . Free explorations are also observed there because heterogeneous preferences result in agents exploiting all arms . However , the “ free pulls ” of OTI is provided by strategic agents ’ intrinsic needs for explorations , which is fundamentally different . Regardless of the differences , both results show the value of further investigating agents ’ behaviors in incentivized exploration . Federated MAB . This work is related to and can potentially contribute to the studies of federated MAB ( FMAB ) ( Shi and Shen , 2021 ; Shi et al. , 2021 ; Zhu et al. , 2021 ) , which considers a similar framework of multiple heterogeneous agents and a global principal . These studies assume agents would unconditionally give up learning their own local ones and naively follow global instructions . which seldom holds in practice and makes those approaches unrobust . However , the proposed incentivizing exploration scheme achieves the “ best ” of both worlds , i.e. , agents learn local models with additional compensations and the principal learn the global model via a small amount of cost . 3 Incentivized Exploration from Decentralized Learning Agents . 3.1 Motivation . Consider the following scenario : one company ( the principal ) can manufacture several products , and it would like to identify one product that best suits the market . A natural strategy is to perform a market survey by having a group of users to try these products for some time and observing their feedbacks . However , different users have different preferences over these products and they are also learning in this process . Once they identify their preferred products , there is little incentive for them to explore others , which limits the information gathering by the company . Such a scenario is common in real life . For example , telecommunications companies such as AT & T ( the principal ) try to find the optimal channel to serve the clients ( agents ) of an area through a period of trials ; content-providing websites such as YouTube ( the principal ) test the proposed features with prospective users ( agents ) . The idea of incentivized exploration fits in these practical scenarios perfectly since it can be utilized to provide extra bonuses for users to try different products . However , the principal now faces the problem of how to incentivize multiple long-term strategic agents simultaneously , which has not been investigated in the prior research to the best of our knowledge . In addition , similar settings of applications are also considered in the recently proposed federated MAB ( FMAB ) ( Shi and Shen , 2021 ; Zhu et al. , 2021 ) . However , current FMAB studies only consider naive users who always follow the principal ’ s instructions , and can not be applied to strategic users that have individual learning abilities and objectives . 3.2 Agents and the Principal . Following the motivation example in Section 3.1 , we consider a total of M available decentralized agents , each of which interacts with a local bandit environment equipped with the same set of K arms ( referred to as local arms ) but with agent-dependent arm utilities . Namely , at time step t , a reward Xk , m ( t ) ∈ [ 0 , 1 ] is associated with agent m ’ s action of pulling arm k , which is sampled independently from an unknown distribution whose expectation is µk , m : = E [ Xk , m ( t ) ] . In general , µk , m 6= µk , n for m 6= n. These agents are ( naturally ) assumed to be decentralized and self-interested , i.e. , their goal is to collect as much ( of their own ) reward as possible during a certain time horizon . Besides these heterogeneous agents , there is also a principal . The principal faces a global bandit game , which also has the same set of K arms ( referred to as global arms for distinction ) . The expected rewards of global arm k are the exact average of corresponding local arms , i.e. , µk : = 1 M ∑ m∈ [ M ] µk , m . The principal ’ s goal is to identify the optimal global arm with a certain confidence 1− δ within a time horizon T , where δ ∈ ( 0 , 1 ) is a pre-fixed constant of the failure probability . Without loss of generality , each local bandit game is assumed to have one unique optimal local arm , which is denoted as k∗ , m : = arg maxk∈ [ K ] µk , m for agent m and its expected reward is µ∗ , m : = µk∗ , m , m . Correspondingly , the sub-optimal gap for arm k 6= k∗ , m is defined as ∆k , m : = µ∗ , m−µk , m and for arm k∗ , m , we denote ∆k∗ , m , m = ∆min , m : = mink 6=k∗ , m ∆k . Similarly , we also assume there is one unique optimal global arm k∗ : = arg maxk∈ [ K ] µk in the global bandit game , whose expected reward is µ∗ : = µk∗ . The global sub-optimal gap for arm k 6= k∗ is defined as ∆k : = µ∗ − µk and for arm k∗ , we denote ∆k∗ = ∆min : = mink 6=k∗ ∆k . In addition , the time horizon T is assumed to be known to the principal but not to agents , as it is often the principal who determines such a horizon . 3.3 Interaction and Observation Model . While the agents can directly pull their local arms and observe rewards , the principal can not interact with the global game . Instead , she can only observe local actions and rewards . In other words , with agent m pulling arm k at time t and getting reward Xk , m ( t ) , the principal can also observe the action k and the reward Xk , m ( t ) , which may be used to estimate expected rewards of global arms . It is helpful to interpret the global game as a virtual global characterization of all the local games ( although it does not necessarily align with any specific one ) , which results in the challenge that it can not be directly interacted with but can only be inferred indirectly . This indirect information gathering introduces significant difficulties for the principal . In particular , it is challenging to obtain sufficient local information from self-interested agents to aggregate precise global information . For example , an arm k can be exactly , or very close to , the optimal one in the global model , but also be highly sub-optimal on agentm ’ s local model . Thus , the principal needs large amount of local explorations on arm k to estimate it , which contradicts with agent m ’ s willingness as arm k may not provide high local rewards . Thus , the task of best ( global ) arm identification is likely to fail only with this passive gathering of information , which is numerically verified in Section 6 . 3.4 Incentivized Exploration To address these challenges , we resort to the paradigm of incentivized exploration , which provides a means for the principal to influence local actions . At time step t , the principal can provide extra bonuses for agents to explore arms , which will be announced to the agents before their decision making . Specifically , at time t , the bonus on arm k for agent m is denoted as Ik , m ( t ) ≥ 0 . If agent m chooses arm k , she has an observation of Xk , m ( t ) but obtains a reward X ′k , m ( t ) = Xk , m ( t ) + Ik , m ( t ) . Intuitively , if the principal wants to incentivize agentm to explore a certain arm k against the agent ’ s original willingness , she should give a high bonus , i.e. , Ik , m ( t ) > 0 ; otherwise , she should spare no bonus , i.e. , Ik , m ( t ) = 0 . In this way , the principal can leverage the extra bonuses to have agents gather her desired local information . The interaction model with incentivized exploration between the principal and one agent ( among overall M agents ) is illustrated in Fig 1 . Under the basic framework , we now formally define the learning objectives of agents and the principal . Agents ’ Objectives . First , the self-interested agents want to collect as many rewards as possible , but note that the rewards now consist of two parts : the original rewards generated by pulled arms and additional bonuses from principal ’ s incentives . Thus , the cumulative rewards of agent m is defined as Rm ( T ) : = ∑T t=1 ( Xπm ( t ) , m ( t ) + Iπm ( t ) , m ( t ) ) where πm ( t ) denotes the arm pulled by agent m at time t. We remark that such rational learning agents differ fundamentally from myopic ones assumed in most research of incentivized exploration . Principal ’ Objectives . On the principal ’ s side , her first goal is to identify the best global arm with a confidence higher than 1− δ. Rigorously , this goal can be stated as P [ k̂∗ ( T ) = k∗ ] ≥ 1− δ , where k̂∗ ( T ) denotes the identified arm at horizon T . Given that the identification is correct , the principal also aims at spending as few cumulative incentives as possible , which is defined as C ( T ) : = ∑T t=1 ∑ m∈ [ M ] Iπm ( t ) , m ( t ) . Note that best arm identification is the principal ’ s major task whereas minimizing cost C ( T ) is only meaningful given k∗ ( T ) = k∗ is achieved . Though not central , our mechanism will also satisfy other desirable properties such as storage efficiency . In this work , we adopt the perspective of the principal , and try to optimize the principal ’ s performance w.r.t . her learning objectives specified above . In other words , agents are ignorant of principal ’ s goals and only focus on their individual rewards , and we only incorporate their “ selfishness ” to design an effective and efficient strategy for the principal . Such choice is natural in common applications where the behaviors of principal ( e.g. , company , commercial platform ) can be designed , while agents ( e.g. , users , customers ) normally perform their own decision making which can not be specified . Remark . Note that agents and the principal actually represent two kinds of bandit learning objectives . Namely , the agents target regret minimization ( Auer et al. , 2002 ; Garivier and Cappé , 2011 ) ( although we here use the equivalent notation of cumulative rewards instead of regret ) , while the principal aims at best arm identification ( Audibert et al. , 2010 ; Jamieson et al. , 2014 ; Garivier and Kaufmann , 2016 ) . This formulation is reasonable as agents often care more about their own cumulative benefits while the principal aims at the final result , which can then be used in the future . However , as noted in Bubeck et al . ( 2011 ) , even with the same game instance , these two objectives do not necessarily align with each other , not to mention the additional global-local heterogeneity considered in our model .

Summary: This paper studies incentivized exploration in multi-armed bandits with long-term strategic agents. The question of how to incentivize agents to explore underrated actions/arms has become an increasingly popular topic in the recent few years. Yet, it is still a relatively underexplored area for studying how to incentivize UCB agents. Therefore, I think the proposed method contributes to a significant problem.