Contextual Multi-Armed Bandit with Communication Constraints

1 INTRODUCTION . In the last few years , synergies between Machine Learning ( ML ) and communication networks have attracted a lot of interest in the research community , thanks to the fruitful interplay of the two fields in emerging applications from Internet of things ( IoT ) to autonomous vehicles and other edge services . In most of these applications , both the generated data and the processing power is distributed across a network of physically distant devices , thus a reliable communication infrastructure is pivotal to run ML algorithms that can leverage the collected distributed knowledge ( Park et al. , 2019 ) . To this end , a lot of recent works have tried to redesign networks and to efficiently represent information to support distributed ML applications , where the activities of data collection , processing , learning and inference are performed in different geographical locations , and should consider limited communication , memory , or processing resources , as well as addressing privacy issues . In contrast to the insatiable growth in our desire to gather more data and intelligence , available communication resources ( bandwidth and power , in particular ) are highly limited , and must be shared among many different devices and applications . This requires the design of highly communicationefficient distributed learning algorithms , particularly for edge applications . Information theory , in particular the rate-distortion theory , have laid the fundamental limits of efficient data compression with the aim to reconstructing the source signal with the highest fidelity ( Cover & Thomas , 2006b ) . However , in the aforementioned applications , the goal is often not to reconstruct the source signal , but to make some inference based on that . This requires task-oriented compression , filtering out the unnecessary information for the target application , and thus decreasing the number of bits that have to be transmitted over the communication networks . This approach should target the questions of what is the most useful information that has to be sent , and how to represent it , in order to meet the application requirements consuming the minimum amount of network resources . Our goal in this paper is to investigate a theoretically grounded method to efficiently transmit data in a Contextual Multi-Armed Bandit ( CMAB ) problem , in which the context information is available to a decision-maker , whereas the actions can be taken by a remote entity , called controller , controlling a multitude of agents . We assume that a limited communication link is available between the decisionmaker and the controller to communicate at each round the intended actions . The controller must decide on the actions to take based on the message received over the channel , while the decisionmaker observes the rewards at each round , and updates its policy accordingly . This scenario can model , for example , a personalized ad placement application , where the content owner observes the individual visitors to its website ; and hence , has the context information , but must convey the ads that must be shown to each visitor to a separate entity that manages the marketing content . This will require communicating hundreds or thousands of adds to be placed at each round , from among a large set of possible adds , within the communication resource and delay constraints of the underlying communication channel , which is quantified as the number of bits available per agent . This problem may arise in other similar applications of CMABs with communication constraints between the decision-maker and the controller ( Bouneffouf & Rish , 2019 ) . 1.1 RELATED WORK . Given the amount of data that is generated by machines , sensors and mobile devices , the design of distributed learning algorithms is a hot topic in the ML literature . These algorithms often impose communication constraints among agents , requiring the design of methods that would allow efficient representation of messages to be exchanged . While rate-distortion theory deals with efficient lossy transmission of signals ( Cover & Thomas , 2006b ) , in ML applications , we typically do not need to reconstruct the underlying signal , but make some inference based on that . These applications can be modeled through distributed hypothesis testing ( Berger , Sep. 1979 ; Ahlswede & Csiszár , 1986 ) and estimation ( Zhang et al. , 2013 ; Xu & Raginsky , 2017 ) problems under rate constraints . In parallel to the theoretical rate-distortion analysis , significant research efforts have been invested in the design of practical data compression algorithms , focusing on specific information sources , such as JPEG and BPG for image compression , or MPEG and H.264 for video compression . While adapting these tools to specific inference tasks is difficult , recently deep learning techniques have been employed to learn task-specific compression algorithms ( Torfason et al. , 2018 ; Jankowski et al. , 2021 ) , which achieve significant efficiency by bypassing image reconstruction . While the above mainly focus on the inference task through supervised learning , here we consider the CMAB problem . There is a growing literature on multi-agent Reinforcement Learning ( RL ) problems with communication links ( Foerster et al. , 2016 ; Sukhbaatar et al. , 2016 ; Havrylov & Titov , 2017 ; Lazaridou et al. , 2017 ) . These papers consider a multi-agent partially observable Markov decision process ( POMDP ) , where the agents collaborate to resolve a specific task . In addition to the usual reward signals , agents can also benefit from the available communication links to better cooperate and coordinate their actions . It is shown that communication can help overcome the inherent non-stationarity of the multi-agent environment . Our problem can be considered as a special case of this general RL formulation , where the state at each time is independent of the past states and actions . Moreover , we focus on a particular setting in which the communication is one-way , from the decision-maker that observes the state and the reward , towards the controller that takes the actions . This formulation is different from the existing results in the literature involving multi-agent Multi-Armed Bandit ( MAB ) . In Agarwal et al . ( 2021 ) , each agent can pull an arm and communicate with others . They do not consider the contextual case , and focus on a particular communication scheme , where each agent shares the index of the best arm according to its experience . Another related formulation is proposed in Hanna et al . ( 2021 ) , where a pool of agents collaborate to solve a common MAB problem with a rate-constrained communication channel from the agents to the server . In this case , agents observe their rewards and upload them to the server , which in turn updates the policy used to instruct them . In Park & Faradonbeh ( 2021 ) , the authors consider a partially observable CMAB scenario , where the agent has only partial information about the context . However , this paper does not consider any communication constraint , and the partial/ noisy view of the context is generated by nature . Differently from the existing literature , our goal here is to identify the fundamental information theoretic limits of learning with communication constraints in this particular scenario . 2 PROBLEM FORMULATION . 2.1 CONTEXTUAL MULTI-ARMED BANDIT ( CMAB ) PROBLEM . We considerN agents , which experience independent realizations of the same CMAB problem . The CMAB is a sequential decision game in which the environment imposes a probability distribution PS over a set of contexts , or states , S , which is finite in our case . The game proceeds in rounds , and at each round t = 1 , . . . , T , a realization of the state st , i ∈ S is sampled from the distribution PS for each agent i ∈ { 1 , . . . , N } . At each time step t , and for each agent i , states are sampled iid according to PS . In the usual CMAB setting , the decision-maker would observe the states { st , i } Ni=1 of the agents , and choose an action ( or arm ) at , i ∈ { 1 , . . . , K } = A , for each agent , where K is the total number of available actions , with probability πt , i ( at , i|st , i ) given by a ( possibly ) stochastic policy πt : S → ∆K . Here ∆K is the K-dimensional simplex , containing all possible distributions over the set of actions . Once the actions have been taken by all the agents , at each round t the environment returns rewards for all the agents following independent realizations of the same reward process , rt , i = r ( st , i , at , i ) ∼ PR ( r|st , i , at , i ) , ∀i ∈ { 1 , . . . , N } , which depends on the state and the action of the corresponding agent . Based on the history of returns up to round t − 1 , the decision-maker can optimize its policy to maximize the sum of the rewards G = ∑T t=1 ∑N i=1 rt , i . 2.2 RATE-CONSTRAINED CMAB . In our modified version , the process of observing the system states is spatially separated from the process of taking the actions . The environment states , { st , i } Ni=1 , are observed by a central entity , called the decision-maker , that has to communicate to the controller over a rateconstrained communication channel , at each round t , the information about the actions { at , i } Ni=1 the agents should take . The decision-maker can exploit the knowledge accumulated from the states and rewards observed by all the agents up to round t − 1 , denoted by H ( t − 1 ) = { ( { s1 , i } Ni=1 , { a1 , i } N i=1 , { r1 , i } N i=1 ) , . . . , ( { st−1 , i } N i=1 , { at−1 , i } N i=1 , { rt−1 , i } N i=1 ) , } ∈ H ( t−1 ) , to optimize the policy to be used at round t. Consequently , the problem is to communicate the action distribution , i.e. , the policy πt ( a|st ) , which depends on the specific state realizations observed in round t , to the controller within the available communication resources while inciting the minimal impact on the performance of the learning algorithm . Specifically , the decision-maker employs function f ( N ) t : H ( t−1 ) ×SN → { 1 , 2 , . . . , B } to map the history up to time t and the states of the agents at time t to a message index to be transmitted over the channel . The controller , on the other hand , employs a function g ( N ) t : { 1 , 2 , . . . , B } → AN to map the received message to a set of actions for the agents . In general , both functions f ( N ) t and g ( N ) t can be stochastic . Average per period regret achieved by sequences { f ( N ) t , g ( N ) t } T t=1 is given by ρ ( N ) ( T ) = 1 T E [ T∑ t=1 N∑ i=1 r ( st , i , a ∗ ( st , i ) − r ( st , i , at ) ] , ( 1 ) where at , i = gt , i ( m ( t ) ) is the action taken by agent i based on message m ( t ) = f ( N ) t ( H ( t− 1 ) , { st , i } Ni=1 ) transmitted in round t , a∗ ( st , i ) is the action with maximum mean reward in state st , i , i.e. , the optimal action , and the expectation is taken with respect to the state distribution PS and reward distribution PR . We say that , for a given time period T and N agents , an av- erage regret- communication rate pair ( ρ , R ) is achievable if there exist functions { f ( N ) t , g ( N ) t } T t=1 as defined above with rate 1N log2B ≤ R and regret ρ ( N ) ( T ) ≤ ρ . If a sufficiently large rate is available for communication , i.e. , R ≥ logK , then the intended action for each agent can be reliably conveyed to the controller . Otherwise , to achieve the learning goal while satisfying the rate constraint , the decision-maker must apply a lossy compression scheme , such that the action distribution adopted by the pool of agents resembles the intended policy as much as possible .

Summary: This paper studies a CMAB problem where the actions for multiple agents are sent from the decision-maker over a rate-limited communication channel. The authors developed information-theoretic performance bound for Thompson sampling based policies, which reduce the problem to transmitting conditional probability distributions over a communication channel. A practical coding method was also developed. Numerical experiments were carried out to validate the proposed design.