Contextual Multi-Armed Bandit with Communication Constraints

1 INTRODUCTION . In the last few years , synergies between Machine Learning ( ML ) and communication networks have attracted a lot of interest in the research community , thanks to the fruitful interplay of the two fields in emerging applications from Internet of things ( IoT ) to autonomous vehicles and other edge services . In most of these applications , both the generated data and the processing power is distributed across a network of physically distant devices , thus a reliable communication infrastructure is pivotal to run ML algorithms that can leverage the collected distributed knowledge ( Park et al. , 2019 ) . To this end , a lot of recent works have tried to redesign networks and to efficiently represent information to support distributed ML applications , where the activities of data collection , processing , learning and inference are performed in different geographical locations , and should consider limited communication , memory , or processing resources , as well as addressing privacy issues . In contrast to the insatiable growth in our desire to gather more data and intelligence , available communication resources ( bandwidth and power , in particular ) are highly limited , and must be shared among many different devices and applications . This requires the design of highly communicationefficient distributed learning algorithms , particularly for edge applications . Information theory , in particular the rate-distortion theory , have laid the fundamental limits of efficient data compression with the aim to reconstructing the source signal with the highest fidelity ( Cover & Thomas , 2006b ) . However , in the aforementioned applications , the goal is often not to reconstruct the source signal , but to make some inference based on that . This requires task-oriented compression , filtering out the unnecessary information for the target application , and thus decreasing the number of bits that have to be transmitted over the communication networks . This approach should target the questions of what is the most useful information that has to be sent , and how to represent it , in order to meet the application requirements consuming the minimum amount of network resources . Our goal in this paper is to investigate a theoretically grounded method to efficiently transmit data in a Contextual Multi-Armed Bandit ( CMAB ) problem , in which the context information is available to a decision-maker , whereas the actions can be taken by a remote entity , called controller , controlling a multitude of agents . We assume that a limited communication link is available between the decisionmaker and the controller to communicate at each round the intended actions . The controller must decide on the actions to take based on the message received over the channel , while the decisionmaker observes the rewards at each round , and updates its policy accordingly . This scenario can model , for example , a personalized ad placement application , where the content owner observes the individual visitors to its website ; and hence , has the context information , but must convey the ads that must be shown to each visitor to a separate entity that manages the marketing content . This will require communicating hundreds or thousands of adds to be placed at each round , from among a large set of possible adds , within the communication resource and delay constraints of the underlying communication channel , which is quantified as the number of bits available per agent . This problem may arise in other similar applications of CMABs with communication constraints between the decision-maker and the controller ( Bouneffouf & Rish , 2019 ) . 1.1 RELATED WORK . Given the amount of data that is generated by machines , sensors and mobile devices , the design of distributed learning algorithms is a hot topic in the ML literature . These algorithms often impose communication constraints among agents , requiring the design of methods that would allow efficient representation of messages to be exchanged . While rate-distortion theory deals with efficient lossy transmission of signals ( Cover & Thomas , 2006b ) , in ML applications , we typically do not need to reconstruct the underlying signal , but make some inference based on that . These applications can be modeled through distributed hypothesis testing ( Berger , Sep. 1979 ; Ahlswede & Csiszár , 1986 ) and estimation ( Zhang et al. , 2013 ; Xu & Raginsky , 2017 ) problems under rate constraints . In parallel to the theoretical rate-distortion analysis , significant research efforts have been invested in the design of practical data compression algorithms , focusing on specific information sources , such as JPEG and BPG for image compression , or MPEG and H.264 for video compression . While adapting these tools to specific inference tasks is difficult , recently deep learning techniques have been employed to learn task-specific compression algorithms ( Torfason et al. , 2018 ; Jankowski et al. , 2021 ) , which achieve significant efficiency by bypassing image reconstruction . While the above mainly focus on the inference task through supervised learning , here we consider the CMAB problem . There is a growing literature on multi-agent Reinforcement Learning ( RL ) problems with communication links ( Foerster et al. , 2016 ; Sukhbaatar et al. , 2016 ; Havrylov & Titov , 2017 ; Lazaridou et al. , 2017 ) . These papers consider a multi-agent partially observable Markov decision process ( POMDP ) , where the agents collaborate to resolve a specific task . In addition to the usual reward signals , agents can also benefit from the available communication links to better cooperate and coordinate their actions . It is shown that communication can help overcome the inherent non-stationarity of the multi-agent environment . Our problem can be considered as a special case of this general RL formulation , where the state at each time is independent of the past states and actions . Moreover , we focus on a particular setting in which the communication is one-way , from the decision-maker that observes the state and the reward , towards the controller that takes the actions . This formulation is different from the existing results in the literature involving multi-agent Multi-Armed Bandit ( MAB ) . In Agarwal et al . ( 2021 ) , each agent can pull an arm and communicate with others . They do not consider the contextual case , and focus on a particular communication scheme , where each agent shares the index of the best arm according to its experience . Another related formulation is proposed in Hanna et al . ( 2021 ) , where a pool of agents collaborate to solve a common MAB problem with a rate-constrained communication channel from the agents to the server . In this case , agents observe their rewards and upload them to the server , which in turn updates the policy used to instruct them . In Park & Faradonbeh ( 2021 ) , the authors consider a partially observable CMAB scenario , where the agent has only partial information about the context . However , this paper does not consider any communication constraint , and the partial/ noisy view of the context is generated by nature . Differently from the existing literature , our goal here is to identify the fundamental information theoretic limits of learning with communication constraints in this particular scenario . 2 PROBLEM FORMULATION . 2.1 CONTEXTUAL MULTI-ARMED BANDIT ( CMAB ) PROBLEM . We considerN agents , which experience independent realizations of the same CMAB problem . The CMAB is a sequential decision game in which the environment imposes a probability distribution PS over a set of contexts , or states , S , which is finite in our case . The game proceeds in rounds , and at each round t = 1 , . . . , T , a realization of the state st , i ∈ S is sampled from the distribution PS for each agent i ∈ { 1 , . . . , N } . At each time step t , and for each agent i , states are sampled iid according to PS . In the usual CMAB setting , the decision-maker would observe the states { st , i } Ni=1 of the agents , and choose an action ( or arm ) at , i ∈ { 1 , . . . , K } = A , for each agent , where K is the total number of available actions , with probability πt , i ( at , i|st , i ) given by a ( possibly ) stochastic policy πt : S → ∆K . Here ∆K is the K-dimensional simplex , containing all possible distributions over the set of actions . Once the actions have been taken by all the agents , at each round t the environment returns rewards for all the agents following independent realizations of the same reward process , rt , i = r ( st , i , at , i ) ∼ PR ( r|st , i , at , i ) , ∀i ∈ { 1 , . . . , N } , which depends on the state and the action of the corresponding agent . Based on the history of returns up to round t − 1 , the decision-maker can optimize its policy to maximize the sum of the rewards G = ∑T t=1 ∑N i=1 rt , i . 2.2 RATE-CONSTRAINED CMAB . In our modified version , the process of observing the system states is spatially separated from the process of taking the actions . The environment states , { st , i } Ni=1 , are observed by a central entity , called the decision-maker , that has to communicate to the controller over a rateconstrained communication channel , at each round t , the information about the actions { at , i } Ni=1 the agents should take . The decision-maker can exploit the knowledge accumulated from the states and rewards observed by all the agents up to round t − 1 , denoted by H ( t − 1 ) = { ( { s1 , i } Ni=1 , { a1 , i } N i=1 , { r1 , i } N i=1 ) , . . . , ( { st−1 , i } N i=1 , { at−1 , i } N i=1 , { rt−1 , i } N i=1 ) , } ∈ H ( t−1 ) , to optimize the policy to be used at round t. Consequently , the problem is to communicate the action distribution , i.e. , the policy πt ( a|st ) , which depends on the specific state realizations observed in round t , to the controller within the available communication resources while inciting the minimal impact on the performance of the learning algorithm . Specifically , the decision-maker employs function f ( N ) t : H ( t−1 ) ×SN → { 1 , 2 , . . . , B } to map the history up to time t and the states of the agents at time t to a message index to be transmitted over the channel . The controller , on the other hand , employs a function g ( N ) t : { 1 , 2 , . . . , B } → AN to map the received message to a set of actions for the agents . In general , both functions f ( N ) t and g ( N ) t can be stochastic . Average per period regret achieved by sequences { f ( N ) t , g ( N ) t } T t=1 is given by ρ ( N ) ( T ) = 1 T E [ T∑ t=1 N∑ i=1 r ( st , i , a ∗ ( st , i ) − r ( st , i , at ) ] , ( 1 ) where at , i = gt , i ( m ( t ) ) is the action taken by agent i based on message m ( t ) = f ( N ) t ( H ( t− 1 ) , { st , i } Ni=1 ) transmitted in round t , a∗ ( st , i ) is the action with maximum mean reward in state st , i , i.e. , the optimal action , and the expectation is taken with respect to the state distribution PS and reward distribution PR . We say that , for a given time period T and N agents , an av- erage regret- communication rate pair ( ρ , R ) is achievable if there exist functions { f ( N ) t , g ( N ) t } T t=1 as defined above with rate 1N log2B ≤ R and regret ρ ( N ) ( T ) ≤ ρ . If a sufficiently large rate is available for communication , i.e. , R ≥ logK , then the intended action for each agent can be reliably conveyed to the controller . Otherwise , to achieve the learning goal while satisfying the rate constraint , the decision-maker must apply a lossy compression scheme , such that the action distribution adopted by the pool of agents resembles the intended policy as much as possible .

Summary: This work studies a rate-constrained contextual multi arm bandit (RC-CMAB) problem: the decision maker has to make action decisions for multiple parallel (independent and identical) CMAB problems (i.e. agents), but can only communicate the actions for each CMAB problem to a controller through a rate-constrained communication channel, from which the controller receives and decodes the actions of the decision maker, and applies such decoded decisions to each CMAB.  The paper first formulates the problem as a policy compression problem under an information theoretic framework, and then characterizes the optimal compression scheme for infinite agents. Next, the paper presents a practical coding scheme to communicate actions to finite agents under rate constraints, and finally benchmarks its performance against the compression scheme in the asymptotic regime.