(Almost) Free Incentivized Exploration from Decentralized Learning Agents

1 Introduction . Multi-armed bandits ( MAB ) is a simple yet powerful model for sequential decision making with an exploration-exploitation tradeoff ( Bubeck and Cesa-Bianchi , 2012 ; Lattimore and Szepesvári , 2020 ) . In standard MAB settings , one principal , who has a long-term system-level objective , takes charge of selecting and playing arms . However , such assumption does not always hold in reality . It is often the case that arm pulls are performed by multiple different agents whose individual goals are not aligned with the system , and the principal can only observe agents ’ actions . One typical example is the individual buyers ( agents ) and the online shopping platform ( the principal ) . Such scenarios complicate decision making and introduce significant difficulties to optimize the system performance . Incentivized exploration has been proposed to address this problem ( Frazier et al. , 2014 ; Mansour et al. , 2015 ) . Specifically , bonuses can be offered by the principal to incentivize agents to perform specific actions , e.g. , to explore their originally underrated arms . This framework provides an opportunity to reconcile different interests between the principal and agents . As a concrete example , the online shopping platform can offer discounts on certain items so that individual buyers would buy them and provide feedbacks , which can be used to optimize future strategies of the platform . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . While incentivized exploration has been investigated in the existing literature , we recognize two major limitations . First , almost all of the existing works assume the participating agents to be myopic , i.e. , they always choose the empirically best arm . Second , it is always assumed that at each time slot , one new agent participates in the system , i.e. , agents never stay or return . In other words , prior research mainly considers how to incentivize one single temporary myopic agent . These two assumptions largely limit the applicability of incentivized exploration . In this work , we extend the study of incentivized exploration beyond the aforementioned barriers , and investigate situations with multiple long-term strategic agents . In particular , we focus on the scenarios ( see Section 3.1 ) where the principal wants to identify the ( overall ) best arm whereas the heterogeneously involved agents only care about their different individual cumulative rewards . For such scenarios , the “ Observe-then-Incentivize ” ( OTI ) mechanism is proposed and several interesting observations are obtained . First , we find that strategic agents ’ intrinsic needs of learning can actually benefit principal ’ s exploration by providing “ free pulls ” . In other words , as opposed to myopic agents , the self interests of strategic agents can be exploited by the principal . Second , it turns out that increasing the number of participating agents can significantly mitigate the principal ’ s burden on incentivizing , which highlights the importance of increasing the population of agents . A crux of our findings is the following intriguing conceptual message : when there are sufficiently many learning agents involved , the exploration process of the principal could be ( almost ) free . Behind these findings , three novel technical components play critical roles in the design and analysis of OTI , all of which may have independent values . • First , a simple yet provably effective incentive-provision strategy is developed , which can efficiently regulate strategic agents ’ behaviors and serves as the foundation of the algorithm analysis . • Second , a best arm identification algorithm is carefully crafted to tackle the varying amounts of local information from heterogeneous agents . This setting itself is novel in best arm identification . • A high-probability finite-time lower bound of UCB algorithms ( Auer et al. , 2002 ) is proved , which contributes to a better understanding of the celebrated UCB . These insights and techniques are unique in incentivizing multiple long-term strategic agents , which may find applications in related problems , and encourage future research in this direction . 2 Related Works . Incentivized exploration . Since proposed by Frazier et al . ( 2014 ) ; Kremer et al . ( 2014 ) , many progresses have been made in incentivized exploration in MAB . Especially , there exist two lines of studies . The first one ( Kremer et al. , 2014 ; Mansour et al. , 2015 , 2016 ; Immorlica et al. , 2020 ; Sellke and Slivkins , 2020 ) assumes the principal can observe the full history while the agents can not , and the principal leverage such information advantage to perform incentivizing . The second line , which our setting follows , considers a publicly available history while the incentives are done through compensations . This idea is first introduced by Frazier et al . ( 2014 ) and generalized by Han et al . ( 2015 ) , both on Bayesian settings . The non-Bayesian case , as adopted in this work , is first studied by Wang and Huang ( 2018 ) , and recently extended by Liu et al . ( 2020 ) ; Wang et al . ( 2021 ) . However , the aforementioned works mainly consider that one new myopic agent enters the system at each time slot and leaves afterward . The only exception is Mansour et al . ( 2016 ) , where multiple but still temporary and myopic agents are considered . This work differs from them in considering multiple long-term strategic agents . In addition , another notable difference is that almost all prior works focus on regret minimization for the principal , instead of best arm identification . One important related work is Chen et al . ( 2018 ) , which studies temporary myopic agents with heterogeneous preferences . Free explorations are also observed there because heterogeneous preferences result in agents exploiting all arms . However , the “ free pulls ” of OTI is provided by strategic agents ’ intrinsic needs for explorations , which is fundamentally different . Regardless of the differences , both results show the value of further investigating agents ’ behaviors in incentivized exploration . Federated MAB . This work is related to and can potentially contribute to the studies of federated MAB ( FMAB ) ( Shi and Shen , 2021 ; Shi et al. , 2021 ; Zhu et al. , 2021 ) , which considers a similar framework of multiple heterogeneous agents and a global principal . These studies assume agents would unconditionally give up learning their own local ones and naively follow global instructions . which seldom holds in practice and makes those approaches unrobust . However , the proposed incentivizing exploration scheme achieves the “ best ” of both worlds , i.e. , agents learn local models with additional compensations and the principal learn the global model via a small amount of cost . 3 Incentivized Exploration from Decentralized Learning Agents . 3.1 Motivation . Consider the following scenario : one company ( the principal ) can manufacture several products , and it would like to identify one product that best suits the market . A natural strategy is to perform a market survey by having a group of users to try these products for some time and observing their feedbacks . However , different users have different preferences over these products and they are also learning in this process . Once they identify their preferred products , there is little incentive for them to explore others , which limits the information gathering by the company . Such a scenario is common in real life . For example , telecommunications companies such as AT & T ( the principal ) try to find the optimal channel to serve the clients ( agents ) of an area through a period of trials ; content-providing websites such as YouTube ( the principal ) test the proposed features with prospective users ( agents ) . The idea of incentivized exploration fits in these practical scenarios perfectly since it can be utilized to provide extra bonuses for users to try different products . However , the principal now faces the problem of how to incentivize multiple long-term strategic agents simultaneously , which has not been investigated in the prior research to the best of our knowledge . In addition , similar settings of applications are also considered in the recently proposed federated MAB ( FMAB ) ( Shi and Shen , 2021 ; Zhu et al. , 2021 ) . However , current FMAB studies only consider naive users who always follow the principal ’ s instructions , and can not be applied to strategic users that have individual learning abilities and objectives . 3.2 Agents and the Principal . Following the motivation example in Section 3.1 , we consider a total of M available decentralized agents , each of which interacts with a local bandit environment equipped with the same set of K arms ( referred to as local arms ) but with agent-dependent arm utilities . Namely , at time step t , a reward Xk , m ( t ) ∈ [ 0 , 1 ] is associated with agent m ’ s action of pulling arm k , which is sampled independently from an unknown distribution whose expectation is µk , m : = E [ Xk , m ( t ) ] . In general , µk , m 6= µk , n for m 6= n. These agents are ( naturally ) assumed to be decentralized and self-interested , i.e. , their goal is to collect as much ( of their own ) reward as possible during a certain time horizon . Besides these heterogeneous agents , there is also a principal . The principal faces a global bandit game , which also has the same set of K arms ( referred to as global arms for distinction ) . The expected rewards of global arm k are the exact average of corresponding local arms , i.e. , µk : = 1 M ∑ m∈ [ M ] µk , m . The principal ’ s goal is to identify the optimal global arm with a certain confidence 1− δ within a time horizon T , where δ ∈ ( 0 , 1 ) is a pre-fixed constant of the failure probability . Without loss of generality , each local bandit game is assumed to have one unique optimal local arm , which is denoted as k∗ , m : = arg maxk∈ [ K ] µk , m for agent m and its expected reward is µ∗ , m : = µk∗ , m , m . Correspondingly , the sub-optimal gap for arm k 6= k∗ , m is defined as ∆k , m : = µ∗ , m−µk , m and for arm k∗ , m , we denote ∆k∗ , m , m = ∆min , m : = mink 6=k∗ , m ∆k . Similarly , we also assume there is one unique optimal global arm k∗ : = arg maxk∈ [ K ] µk in the global bandit game , whose expected reward is µ∗ : = µk∗ . The global sub-optimal gap for arm k 6= k∗ is defined as ∆k : = µ∗ − µk and for arm k∗ , we denote ∆k∗ = ∆min : = mink 6=k∗ ∆k . In addition , the time horizon T is assumed to be known to the principal but not to agents , as it is often the principal who determines such a horizon . 3.3 Interaction and Observation Model . While the agents can directly pull their local arms and observe rewards , the principal can not interact with the global game . Instead , she can only observe local actions and rewards . In other words , with agent m pulling arm k at time t and getting reward Xk , m ( t ) , the principal can also observe the action k and the reward Xk , m ( t ) , which may be used to estimate expected rewards of global arms . It is helpful to interpret the global game as a virtual global characterization of all the local games ( although it does not necessarily align with any specific one ) , which results in the challenge that it can not be directly interacted with but can only be inferred indirectly . This indirect information gathering introduces significant difficulties for the principal . In particular , it is challenging to obtain sufficient local information from self-interested agents to aggregate precise global information . For example , an arm k can be exactly , or very close to , the optimal one in the global model , but also be highly sub-optimal on agentm ’ s local model . Thus , the principal needs large amount of local explorations on arm k to estimate it , which contradicts with agent m ’ s willingness as arm k may not provide high local rewards . Thus , the task of best ( global ) arm identification is likely to fail only with this passive gathering of information , which is numerically verified in Section 6 . 3.4 Incentivized Exploration To address these challenges , we resort to the paradigm of incentivized exploration , which provides a means for the principal to influence local actions . At time step t , the principal can provide extra bonuses for agents to explore arms , which will be announced to the agents before their decision making . Specifically , at time t , the bonus on arm k for agent m is denoted as Ik , m ( t ) ≥ 0 . If agent m chooses arm k , she has an observation of Xk , m ( t ) but obtains a reward X ′k , m ( t ) = Xk , m ( t ) + Ik , m ( t ) . Intuitively , if the principal wants to incentivize agentm to explore a certain arm k against the agent ’ s original willingness , she should give a high bonus , i.e. , Ik , m ( t ) > 0 ; otherwise , she should spare no bonus , i.e. , Ik , m ( t ) = 0 . In this way , the principal can leverage the extra bonuses to have agents gather her desired local information . The interaction model with incentivized exploration between the principal and one agent ( among overall M agents ) is illustrated in Fig 1 . Under the basic framework , we now formally define the learning objectives of agents and the principal . Agents ’ Objectives . First , the self-interested agents want to collect as many rewards as possible , but note that the rewards now consist of two parts : the original rewards generated by pulled arms and additional bonuses from principal ’ s incentives . Thus , the cumulative rewards of agent m is defined as Rm ( T ) : = ∑T t=1 ( Xπm ( t ) , m ( t ) + Iπm ( t ) , m ( t ) ) where πm ( t ) denotes the arm pulled by agent m at time t. We remark that such rational learning agents differ fundamentally from myopic ones assumed in most research of incentivized exploration . Principal ’ Objectives . On the principal ’ s side , her first goal is to identify the best global arm with a confidence higher than 1− δ. Rigorously , this goal can be stated as P [ k̂∗ ( T ) = k∗ ] ≥ 1− δ , where k̂∗ ( T ) denotes the identified arm at horizon T . Given that the identification is correct , the principal also aims at spending as few cumulative incentives as possible , which is defined as C ( T ) : = ∑T t=1 ∑ m∈ [ M ] Iπm ( t ) , m ( t ) . Note that best arm identification is the principal ’ s major task whereas minimizing cost C ( T ) is only meaningful given k∗ ( T ) = k∗ is achieved . Though not central , our mechanism will also satisfy other desirable properties such as storage efficiency . In this work , we adopt the perspective of the principal , and try to optimize the principal ’ s performance w.r.t . her learning objectives specified above . In other words , agents are ignorant of principal ’ s goals and only focus on their individual rewards , and we only incorporate their “ selfishness ” to design an effective and efficient strategy for the principal . Such choice is natural in common applications where the behaviors of principal ( e.g. , company , commercial platform ) can be designed , while agents ( e.g. , users , customers ) normally perform their own decision making which can not be specified . Remark . Note that agents and the principal actually represent two kinds of bandit learning objectives . Namely , the agents target regret minimization ( Auer et al. , 2002 ; Garivier and Cappé , 2011 ) ( although we here use the equivalent notation of cumulative rewards instead of regret ) , while the principal aims at best arm identification ( Audibert et al. , 2010 ; Jamieson et al. , 2014 ; Garivier and Kaufmann , 2016 ) . This formulation is reasonable as agents often care more about their own cumulative benefits while the principal aims at the final result , which can then be used in the future . However , as noted in Bubeck et al . ( 2011 ) , even with the same game instance , these two objectives do not necessarily align with each other , not to mention the additional global-local heterogeneity considered in our model .

Summary: The paper proposes incentivized exploration in multi-armed bandits (MAB) that includes multiple and long-term strategic agents that have individual learning abilities and objectives. They find that the intrinsic needs of the agents to learn benefits the principal and that increasing the number of these agents can result in (almost) free exploration process for the principal. The contribution of the paper is that it goes beyond previous researches in the field that dealt with temporary myopic agents and studied long-term strategic agents for MAB, proposing Observe-then-Incentivize (OTI) algorithm in the process.