Title: Multi-Agent Learning with Heterogeneous Linear Contextual Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-agent

Bandits

Cooperative
Abstract: As trained intelligent systems become increasingly pervasive, multiagent learning has emerged as a popular framework for studying complex interactions between autonomous agents. Yet, a formal understanding of how and when learners in heterogeneous environments benefit from sharing their respective experiences is far from complete. In this paper, we seek answers to these questions in the context of linear contextual bandits. We present a novel distributed learning algorithm based on the upper confidence bound (UCB) algorithm, which we refer to as H-LINUCB, wherein agents cooperatively minimize the group regret under the coordination of a central server. In the setting where the level of heterogeneity or dissimilarity across the environments is known to the agents, we show that H-LINUCB is provably optimal in regimes where the tasks are highly similar or highly dissimilar.

Multi-Agent Learning with Heterogeneous Linear Contextual Bandits

Abstract

As trained intelligent systems become increasingly pervasive, multi-agent learning has emerged as a popular framework for studying complex interactions between autonomous agents. Yet, a formal understanding of how and when learners in heterogeneous environments benefit from sharing their respective experiences is still in its infancy. In this paper, we seek answers to these questions in the context of linear contextual bandits. We present a novel distributed learning algorithm based on the upper confidence bound (UCB) algorithm, which we refer to as H-LINUCB, wherein agents cooperatively minimize the group regret under the coordination of a central server. In the setting where the level of heterogeneity or dissimilarity across the environments is known to the agents, we show that H-LINUCB is provably optimal in regimes where the tasks are highly similar or highly dissimilar.

1 Introduction

Heterogeneous multi-agent systems enable agents to work together and coordinate their actions to solve complex problems. These systems are inherently scalable, as they can distribute the computational load across multiple agents. This scalability allows the system to handle large and sophisticated tasks beyond the capabilities of a single agent. Despite the potential of multi-agent systems, it poses the following fundamental challenges.

Statistical challenge. Each agent's reward distribution may vary, meaning that different agents receive different rewards for the same action. This heterogeneity in reward distributions introduces complexity and makes coordination among agents more difficult. Furthermore, Wang et al. [2021] point out that an ineffective use of shared data could lead to a significant negative impact on the overall performance. In particular, sharing experiences amongst agents may hinder the system's performance if the tasks are too dissimilar [Rosenstein, 2005, Brunskill and Li, 2013].

Computational complexity. Coordinating the decisions of numerous agents and performing complex computations pose challenges in terms of computational resources, time constraints, and algorithmic scalability.

Communication cost. Efficient communication is also a fundamental challenge for a largescale multi-agent system. As the number of agents in a system increases, the complexity of interactions between agents grows exponentially. Managing the interaction of numerous agents and making decisions in a timely manner becomes increasingly difficult.

Several works address these challenges for heterogeneous multi-agent systems, including federated linear bandits [Huang et al., 2021, Li and Wang, 2022], clustering bandits [Gentile et al., 2014, Ghosh et al., 2023], multi-task linear bandits [Hu et al., 2021, Yang et al., 2021]. However, these works either rely on some special structure of the parameter [Li and Wang, 2022], e.g., a low-rank structure [Hu

et al., 2021] or make different assumptions, e.g., stochastic contexts, finite decision set [Huang et al., 2021, Ghosh et al., 2023], etc.

In this work, we provide a general notion of heterogeneous multi-agent linear contextual bandits ( ε -MALCB) and give analytical results under different regimes of heterogeneity. Specifically, we study a model that consists of M agents. Each agent i ∈ [ M ] plays a d -dimensional linear contextual bandit, parametrized by θ i (see Section 3 for more details), for T rounds. We capture the heterogeneity by a dissimilarity parameter ε > 0 such that ∥ θ i -θ j ∥ 2 ≤ ε , for all i, j ∈ [ M ] . Notably, we do not assume any special structure on the linear parameter; and we allow the decision set to be infinite and possibly chosen adversarially.

Motivating application. Consider a personalized recommendation system for online advertisements [Li et al., 2010, Bouneffouf et al., 2020, Ghosh et al., 2023]. Here, the platform needs to be adaptive to user preferences and maximize total user clicks based on user-click feedback. Each ad can be represented as a context vector, encoding information such as publisher, topic, ad content, etc. The inner product of the ad's context vector and user preference represents the alignment. A higher inner product value indicates greater relevance of the ad. Furthermore, the recommended ads must be personalized to accommodate user preference differences. One naive approach would involve solving a separate linear contextual bandit problem for each user. However, we can pose the following question: Can we enhance the system's performance by pooling data from other users? If so, to what extent of user heterogeneity can we achieve that?

Contributions. We make the following contributions in this paper.

First, we formulate the heterogeneous multi-agent linear contextual bandits as ε -MALCB problem, building on the classic notion of heterogeneity in multiarmed bandits (MABs) Wang et al. [2021]. Our notion of heterogeneity is natural and captures many settings in real-world applications.

Second, when the level of dissimilarity is known, we propose a distributed algorithm, namely H-LINUCB which achieves a regret of ˜ O ( d √ MT +min { εdMT,dM √ T } ) under the coordination of a central server. We discuss in detail how to handle the dissimilarity and introduce a criterion for stopping collaboration when the level of dissimilarity is high. We show that under the regime of low dissimilarity, we can still achieve a regret of ˜ O ( d √ MT ) , which is the same regret rate as if M agents collaborate to solve the same task. In this regime, H-LINUCB outperforms independent learners improving by a factor of √ M , from ˜ O ( dM √ T ) to ˜ O ( d √ MT ) . This is significant when we have a large number of agents.

Third, we complement the upper bound with a lower bound of Ω( d MT + min { εMT,dM √ T } ) . This suggests that our theoretical guarantees are tight in settings where tasks are highly similar or highly dissimilar.

√

Finally, we validate our theoretical results with numerical simulations on synthetic data. When the level of dissimilarity is small, H-LINUCB outperforms independent learning. When the level of dissimilarity is high, our simulation shows that blindly using shared data can lead to linear regret, emphasizing the importance of the criterion we propose for when to stop the collaboration.

2 Related work

The classic linear bandits have a rich literature in both theory and application; see, for example, [Abbasi-Yadkori et al., 2011, Li et al., 2010, Chu et al., 2011, Chatterji et al., 2020, Rusmevichientong and Tsitsiklis, 2010, Bouneffouf et al., 2020, Mahadik et al., 2020], to name a few. With a surge in distributed computing, multi-agent systems have shown their potential and gained more attention in recent years. A large body of works dedicated to studying the homogeneous setting of multiple collaborating agents solve a global linear bandits problem [Wang et al., 2020, Dubey and Pentland, 2020, Moradipari et al., 2022, Mitra et al., 2022, Martínez-Rubio et al., 2019, Chawla et al., 2022].

The problem of multi-agent linear bandits in heterogeneous environments, on the other hand, has received limited attention. Soare et al. [2014] were amongst the first to study heterogeneous linear

bandits; however, the focus in that work is not on group regret, and the authors only consider the setting where tasks are similar. More recently, Huang et al. [2021] proposed an algorithm with a novel multi-agent G-Optimal design. They assume that the heterogeneity comes from the contexts associated with each agent, but agents can still collaborate since they share the same arm parameters. Li and Wang [2022] consider an extension where they assume that each agent's parameter has two components - a shared global component and an individual local component. This formalization requires agents to work on their respective tasks (the local component) but still allows agents to collaborate on the common task (the global component). Wang et al. [2021] study heterogeneity of the Bernoulli MABs problem and provide guarantees for both cases when the level of heterogeneity is known and unknown.

A related line of work studies heterogeneous linear bandits through clustering [Gentile et al., 2014, Li et al., 2016, 2019, Korda et al., 2016, Ghosh et al., 2023]. These works give a guarantee based on the clustering structure of the different linear bandit problems - agents belonging to the same cluster will likely achieve the highest collaboration gain. We do not make any assumption about the 'clusterability' of different bandit problems we may encounter. A yet another approach focuses on multi-task linear bandits, wherein we solve multiple different but closely related linear bandits tasks [Yang et al., 2021, 2022, Hu et al., 2021, Cella et al., 2023, Du et al., 2023]. In particular, these works rely on the assumption that all tasks share a common k -dimensional representation, where k ≪ d . Then, pooling data from different bandits helps learn a good representation and reduces the statistical burden of learning by reducing the linear bandit problem in d dimensions to a k -dimensional setting. We do not consider multi-task learning here.

Our formulation of heterogeneous contextual linear bandits is similar to that of misspecified and corrupted bandits setting (Remark 3.1 for more details) [Ghosh et al., 2017, Lattimore and Csaba, 2020, Takemura et al., 2021, Foster et al., 2020, He et al., 2022]. It is then natural to ask if we can apply the techniques from that part of the literature to deal with the dissimilarity between different bandits in a heterogeneous setting. However, there is a fundamental difference in how the two problems manifest themselves. While misspecification may be typically unavoidable in many settings, in a heterogeneous bandit setting, an agent can always choose to rely solely on its own data if it finds that the data from other agents are too dissimilar.

3 Preliminaries

Multi-Agent Linear Contextual Bandits. We consider a multi-agent learning setting with M agents. At each round t ∈ [ T ] , each agent m ∈ [ M ] picks an action (context) 1 x m,t ∈ D m,t , where D m,t ⊆ R d is a given decision set. The agent m receives reward y m,t = x ⊤ m,t θ m + η m,t , where θ m ∈ R d is an unknown but fixed parameter and η m,t is sub-Gaussian noise. Let F t denote the filtration, i.e., the σ -algebra, induced by σ ( { x m,k } m ∈ [ M ] ,k ≤ t +1 , { η m,k } m ∈ [ M ] ,k ≤ t ) .

Regret. Our goal is to design algorithms for multi-agent linear contextual bandits that achieve a small group regret defined as

Assumption 3.1. Without loss of generality, we assume that,

Bounded parameters : ∥ θ m ∥ 2 ≤ 1 , ∥ x ∥ 2 ≤ 1 , ∀ x ∈ D m,t , m ∈ [ M ] , t ∈ [ T ] .

Sub-Gaussian noise : η m,t is conditionally zero-mean 1-sub-Gaussian random variable with respect to F t -1 .

We note that the assumptions above are standard in linear bandits literature [Abbasi-Yadkori et al., 2011, Hu et al., 2021, Huang et al., 2021]. Further, it is straightforward to let ∥ θ m ∥ ≤ B , for some constant B , by appropriately scaling the rewards. We make no additional assumptions on the context. The decision set could be infinite, and given to each agent possibly adversarially.

Definition 3.1. ( ε -MALCB Problem) A multi-agent linear contextual bandits problem is said to be an ε -MALCB problem instance, if for any two agents i, j ∈ [ M ] , ∥ θ i -θ j ∥ 2 ≤ ε , for an ε ≥ 0 . We call ε the dissimilarity parameter.

Definition 3.2. (Homogeneous setting) A multi-agent linear contextual bandits problem is homogeneous, if it is an ε -MALCB with ε = 0 , i.e., θ i = θ j , for all i, j ∈ [ M ] .

Given the bound on the parameters, we have that ∥ θ i -θ j ∥ 2 ≤ 2 for any i, j ∈ [ M ] . Therefore, it suffices to only consider the case where ε ∈ [0 , 2] .

̸

Remark 3.1. (Misspecified structure) Under the Assumption 3.1, for any two agents i, j we have that θ ⊤ i x -ε ≤ θ ⊤ j x ≤ θ ⊤ i x + ε . Then, E [ y j,x ] = θ ⊤ i x +∆( x ) , for ∆( x ) ∈ [ -ε, ε ] . This represents a misspecified structure wherein agent i receives the reward y j,x from agent j = i .

Remark 3.2. (Recover the ε -MPMAB of Wang et al. [2021]). We note that the ε -MPMAB is a special case of ε -MALCB. Define the mean reward of K arms for agent m as θ m = [ µ m 1 , . . . , µ m K ] . Then, the reward for arm k at round t is y m t,k = θ ⊤ i e k + η t . The decision set D = { e 1 , . . . , e K } are the standard basis vectors. This is a fixed set of arms, given to all agents at each round. The dissimilarity parameter ε is defined as: ∥ θ i -θ j ∥ ∞ ≤ ε for all i, j ∈ [ M ] .

Nonetheless, the results in Wang et al. [2021] are not directly comparable to ours since the dissimilarity parameter ε hides inside the size of the set of subpar arms |I ε | . 2 Furthermore, Wang et al. [2021] give guarantees in a full-communication setting, in which each agent has full access to the past data of all other agents at every round.

Remark 3.3. There are other formulations that also capture the heterogeneity in multi-agent linear bandits. Huang et al. [2021] consider a multi-agent linear bandits setting with a fixed size decision set, containing K actions, { θ a } K a =1 , which is unknown to the agents. Each agent i is associated with K different contexts { x i,a } K a =1 . At reach round, each agent i picks an action a ∈ [ K ] , and receives reward r i,a = x ⊤ i,a θ a + η i,a . Since x i,a can vary for different agents, this captures the heterogeneity across agents. It also allows for collaboration across agents since they share the same decision set.

Li and Wang [2022] assume that each agent parameter θ has a special structure that consists of a shared global component and a unique local component. The reward of agent i can be given as,

The notion of ε -MALCB is in the worst-case sense. One can imagine an ε -MALCB instance of M agents, such that M -1 agents have identical linear parameter, i.e. θ i = θ j , ∀ i, j ∈ [ M -1] , and the parameter of the last agent ∥ θ M -θ M -1 ∥ 2 = ε . With this type of instance, for a large M , we can simply use DISLINUCB of Wang et al. [2021] and achieve nearly optimal regret. Clustering bandits framework could also be used for this ε -MALCB instance since it presents a strong cluster structure. Even though our formulation takes a pessimistic approach but it can handle the case that bandits are largely unclustered.

Our goal is to design a system such that its performance is no worse than running M independent bandit algorithms for each user (zero collaboration). The system should also be adaptive to the heterogeneity in the problem instance, i.e., it should automatically leverage any structure in the problem parameters { θ m } M m =1 to collaboratively solve all bandit problems at a 'faster' rate. To benchmark the performance of such a system, we consider the following baseline.

Independent Learners (IND-OFUL). We establish a baseline algorithm in which each agent independently runs an optimal linear contextual bandits algorithm (OFUL, [Abbasi-Yadkori et al., 2011]) without any communication. Each agent incurs ˜ O ( d √ T ) regret, and ˜ O ( dM √ T ) group regret.

Notation. We denote the weighted norm of vector x w.r.t. matrix A (Mahalanobis norm) as ∥ x ∥ A = √ x ⊤ Ax . We write A ≽ B iff A -B is a positive semi-definite matrix. We use ˜ O ( · ) to hide the polylogarithmic factors in standard Big O notation.

4 Main Results

In this section, we present H-LINUCB, a UCB-style algorithm for ε -MALCB problem, and give guarantees for the case when the dissimilarity ε is known to the agents. In Section 4.2, we present a lower bound and discuss the implication of our results in different regimes of dissimilarity. We defer the detailed proof to the Appendix.

Algorithm 1 H-LINUCB

4.1 H-LINUCB Algorithm

H-LINUCB is a distributed UCB-style algorithm (see Algorithm 1 for pseudocode), in which agents work cooperatively under the coordination of a central server.

H-LINUCB has two learning phases: the collaboration phase (for rounds t ∈ { 1 , . . . , τ -1 } ) and the independent learning phase (for rounds t ∈ { τ, . . . , T } ), where τ ≤ T is the collaboration budget. Intuitively, our two-phase learning framework ensures that the agents stop collaboration after τ rounds lest they incur a linear regret in bandit environments with large dissimilarity. Naturally, then, the parameter τ should depend on the dissimilarity parameter, ε . We give an optimal choice of τ in Theorem 4.1.

At each round t < τ (the collaboration phase), each agent's data is weighted to adapt to the dissimilarity across different agents (Line 15). Then, each agent uses the weighted data to construct

its Confidence Ellipsoid (Line 12) and makes a decision following the optimism principle (Line 13). When a certain condition is met (Line 18), data is pooled and synchronized across the agents. Starting from round τ , all collaboration ceases and each agent enters the independent learning mode and runs an independent copy of the OFUL algorithm [Abbasi-Yadkori et al., 2011] locally for the last T -τ +1 rounds.

We note that H-LINUCB builds upon DISLINUCB of Wang et al. [2020, Protocol 8] with the following modifications:

We scale each agent's data using the weight min(1 , α/ ∥ x m,t ∥ V -1 m,t ) , which we adopt from He et al. [2022], to handle the dissimilarity across different agents (Line 15).

We only allow collaboration until round τ (Line 18). The value of τ depends on the dissimilarity parameter, which we assume is given.

We reset the variables V syn , b syn , V epoch,m , b epoch,m at round τ (Lines 6-9), where each agent switches to the independent learning mode. Here, epoch refers to the time period between two consecutive synchronization rounds.

Each agent uses all of the data available to them at each round to construct the Confidence Ellipsoid C m,t using the result in Lemma 4.1. Given the confidence ellipsoid, the agent chooses the action optimistically: ( x m,t , ˜ θ m,t ) = arg max ( x,θ ) ∈D m,t ×C m,t ⟨ θ, x ⟩ . During the collaboration phase, if the variation in the volume of the ellipsoid exceeds a certain synchronization threshold, D , it triggers a synchronization condition (Lines 18-20). Subsequently, the central server commences the synchronization procedure to update V syn , b syn across all participating agents (Lines 21-27. The optimal value of D depends on the number of agents M , dimension d , and the collaboration budget τ .

The weight min(1 , α/ ∥ x m,t ∥ V -1 m,t ) is a truncation of the inverse bonus, where α > 0 is a threshold parameter that shall be optimized later. When x m,t is not explored much, we have a large exploration bonus ∥ x m,t ∥ V -1 m,t (low confidence). Hence, the algorithm will put a small weight on it to avoid a large regret due to stochastic noise and misspecification. When ∥ x m,t ∥ V -1 m,t is small (high confidence),

H-LINUCB puts a large weight on it, and it can be as large as one [He et al., 2022]. 3 We note that using this weighting could have a significant negative impact on the performance if we are not careful. Several recent studies show that we can incur a regret of ˜ O ( d √ T + ε √ dT ) and ˜ O ( d √ T + εdT ) for misspecified and corrupted linear bandits, respectively [Lattimore and Csaba, 2020, Takemura et al., 2021, Foster et al., 2020, He et al., 2022]. 4 However, a direct application of an algorithm designed for misspecified/corrupted linear bandits to our setting can lead to linear regret when ε = Θ(1) . This is significantly worse as compared to naive independent learning, which always achieves sub-linear ˜ O ( dM √ T ) regret.

We emphasize that the condition ( t < τ ) in Line 18 is crucial to avoid linear regret for H-LINUCB in the regime of large ε . For example, for ε = Θ(1) , τ = T , Algorithm 1 incurs ˜ O ( dMT ) regret, which is linear in term of MT . Furthermore, Theorem 4.3 indicates that there exists an instance of ε -MALCB such that any algorithm incurs at least Ω( dM √ T ) regret. Then, each agent playing OFUL independently would be enough to achieve an optimal ˜ O ( dM √ T ) regret. This suggests that we get a tighter upper bound if we cease collaboration; we discuss the stopping criterion and the choice of τ in Theorem 4.2.

Communication protocol. We use a star-shaped communication network where M agents can interact with a central server [Wang et al., 2020, Dubey and Pentland, 2020]. Each agent communicates with the server by uploading and downloading its data but does not communicate directly with each other. The communication will be triggered only if any agent has enough new data since the last synchronization round. Finally, we assume no latency, or error in the communication between the central server and agents.

Remark 4.1. Wang et al. [2020] show that DISLINUCB can rely on old data and produce nearly optimal policy without much communication, only incurring logarithmic factors in the final regret. H-LINUCB has the same communication cost of O ( M 1 . 5 d 3 ) , as DISLINUCB, which does not depend on the horizon T .

When an agent uses data from other agents, due to the dissimilarity, we need to adjust our confidence bound to make sure the true linear parameter θ m lies in the defined ellipsoid with high probability. The next result shows how to construct such a Confidence Ellipsoid .

Lemma 4.1. (Confidence Ellipsoid). With probability at least 1 -Mδ 1 -Mδ 2 , for each agent m ∈ [ M ] , θ m lies in the confidence set,

where

Note that the result above provides two separate confidence bounds for θ m , one for the period before round τ and the other one for after round τ . Before round τ , agent m will use all the data from other agents to construct C m,t . The proof follows by first bounding ∥ ∥ ∥ ˆ θ m,t -θ m ∥ ∥ ∥ as

̸

V

m,t

(

λ

)

Here, I 1 is a bounded regularization term, I 3 is a noise term that can be bounded by self-normalization lemma (Lemma C.4). Finally, the dissimilarity term I 2 is bounded from above by αεMt using the definition of dissimilarity ∥ θ i -θ m ∥ ≤ ε , and applying a similar argument as He et al. [2022] and the choice of the weight w m,t = min(1 , α/ ∥ x m,t ∥ V -1 m,t ) . For the phase after round τ , we use the same argument as Abbasi-Yadkori et al. [2011] to construct the confidence bound.

Next, we present the group regret upper bound of Algorithm 1 up to round τ .

Theorem 4.1. Given Assumption 3.1, T ≥ 1 , any τ ≤ T , and δ 1 > 0 , setting λ = 1 and α = √ d εMτ in the upper bound of β t on the confidence interval according to Lemma 4.1 ∀ t ∈ [ T ] , and setting the synchronization threshold D = τ log( Mτ ) / ( dM ) , we have that with probability at least 1 -Mδ 1 , the group regret of Algorithm 1 up to round τ , is bounded as

Theorem 4.1 shows that Algorithm 1 incurs ˜ O ( d Mτ ) regret in the first term (which is the same order as a single agent playing for Mτ rounds) plus a penalty of using the data from other agents in the order of ˜ O ( εdMτ ) . The regret of ˜ O ( d √ Mτ ) is unavoidable for any regime of ε , and this rate is known to be optimal in the case of homogeneous multi-agent. Since we use the technique from He et al. [2022] to handle the dissimilarity, the corruption amount of each round is ε , and a total corruption of εdMτ when the central server allows to collaborate up to round τ . To the best of our knowledge, we are not aware of any UCB-based algorithm for misspecified linear bandits in the setting of infinite arms. We expect that employing a misspecified linear bandits algorithm would achieve a regret of ˜ O ( d √ MT + ε √ dMT ) , which is tighter by a factor of √ d . It is worth noting that the CW-OFUL algorithm in He et al. [2022] is designed for handling corruption, whether it can achieve ˜ O ( d √ T + ε √ dT ) in a misspecified setting remains an open question.

√

We now present our main result giving an upper bound on the group regret of H-LINUCB.

Theorem 4.2. Given Assumption 3.1, T ≥ 1 , let λ = 1 , α = √ d εMτ , δ 1 = δ 2 = 1 M 2 T in the upper bound of β t on the confidence interval (see Lemma 4.1) ∀ t ∈ [ T ] . Let τ = min( ⌊ 1 2 ε 2 ⌋ , T ) , and let the synchronization threshold D = τ log( Mτ ) / ( dM ) . Then, the expected group regret of Algorithm 1 is bounded as

Here, τ is the maximum round that the central server allows communication. After that, all agents switch to independent learning. By choosing τ = min( ⌊ 1 2 ε 2 ⌋ , T ) , agents fully cooperate in the regime ε ∈ [0 , 1 √ 2 T ] if T ≤ 1 2 ε 2 , and gradually reduce τ as ε increases from 1 √ 2 T to + ∞ . This is important for avoiding a linear regret since when ε is large, most of the regret comes from the εdMT term and dominates the d √ MT term. The condition on Line 6 also discards all of the synchronized data. In the extreme case, when ε > 1 , there is no collaboration happening due to the condition in Line 18 failing at every round. In other words, H-LINUCB behaves like IND-OFUL.

In the other extreme case, when ε = 0 , all agents solve identical linear bandits. The weight in Line 15 always evaluates to its minimum value of 1 for all t ∈ [ T ] since α = √ d εMT → + ∞ . We have t < ε 2 for all rounds. Therefore, the reset condition in Line 6 is never triggered and H-LINUCB behaves exactly like DISLINUCB, achieving a regret of ˜ O ( d √ MT ) , which is optimal up to some logarithmic factors.

Theorem 4.2 suggests that the upper bound of H-LINUCB is tighter than IND-OFUL for all ε .

Remark 4.2. Ghosh et al. [2023] also propose a personalized algorithm (PMLB) for the heterogeneous multi-agent linear bandits; however, our problem setting is fundamentally different than that of Ghosh et al. [2023]. They consider a finite action set and impose a strong distributional assumption on how contexts are generated, i.e., the stochastic context x i,t for each action i and at each round t is zero-mean and forms a positive-definite covariance matrix. In stark contrast, we consider the adversarial setting where the context set is adversarially generated at each round (and thus, the associated action set can be infinite and arbitrary). This renders the algorithm and guarantees of Ghosh et al. [2023] inapplicable in our setting and, thus, requires a completely different treatment. Those assumptions of Ghosh et al. [2023] are crucial for them to obtain the ˜ O ( T 1 / 4 ) bound (for ε = 0 ). We note that this bound is not information-theoretically possible in our adversarial setting; the minimax lower bound in such settings is Ω( √ T ) (Theorem 4.3).

4.2 Lower bound

In this section, we present a lower bound result for the ε -MALCB problem. We denote R A , I ( M,T ) as a regret of algorithm A on a problem instance I of M agents run for T rounds.

Theorem 4.3. Let I ( ε ) denote the class of ε -MALCB problem instances that satisfy the Assumption 3.1. Then for any d, M, T ∈ Z + with d 2 ≤ T, d 2 48 ≤ T , ∀ ε ≥ 0 , we have the following,

The first term is a straightforward observation that solving an ε -MALCB is at least as hard as solving a single linear bandits for MT rounds, or M agents solving identical bandits for T rounds. The second term suggests that we pay an additional regret of εMT for a small ε ∈ [0 , d √ T ] , and Ω( dM √ T ) for a large ε ≥ d √ T . We note that Ω( dM √ T ) is also the lower bound of IND-OFUL when each agent incurs a regret of at least Ω( d √ T ) . We believe that the analysis of the lower bound could be tightened by using the arguments from misspecified bandits literature, achieving a lower bound of Ω( d √ MT +min { ε √ dMT,dM √ T } ) .

The lower bound suggests that our upper bound is tight up to logarithmic factors in the following extreme regimes, (i) ε ∈ [0 , 1 √ MT ] , where R ( M,T ) = Θ( d √ MT ) ; (ii) ε ∈ [ d √ T , + ∞ ] , where R ( M,T ) = Θ( dM √ T ) . In regime (i), all agents solve tasks that are similar to one another, yielding the highest collaborative gain. In regime (ii), tasks are highly dissimilar, H-LINUCB turns off the collaboration and lets agents solve their own tasks individually.

Ind-OFUL

DisLinUCB

H-LinUCB

40000

30000

Group Regret

20000

10000

0

2500

5000

7500

10000

0

2500

5000

7500

10000

0

2500

5000

7500

10000

(a) ε = 0.0

(b)ε= 0.00067

(c)ε=0.00268

60000

50000

Group Regret

40000

30000

20000

10000

0

2500

5000

7500

10000

0

2500

5000

7500

10000

0

2500

5000

7500

10000

(d) ε = 0.27

(e)ε =0.68

(f)ε = 0.76

Finally, in the regime that ε ∈ ( 1 √ MT , d √ T ) , our results illustrate the interpolation between two extremes. In this regime, our upper bound presents a gap of d in the dissimilarity term εdMT compared to εMT in the lower bound.

The key idea in the proof of Theorem 4.3 is based on an information-theoretic lower bound of Lemma C.1, wherein we extend the result from (single-agent) linear bandits to heterogeneous multi-agent linear bandits. Here, we give the lower bound result without any constraints on the communication, hence, this is also the lower bound of the H-LINUCB algorithm.

5 Numerical Simulations

In this section, we provide some numerical simulations to support our theory. Our goal is to address the following question: how does H-LINUCB perform in three different regimes of dissimilarity: (i) ε ∈ [0 , 1 √ ] , (ii) ε ∈ ( 1 √ , d √ ) , (iii) ε ∈ [ d √ , + ∞ ] ?

We compare the performance of H-LINUCB with that of the following two algorithms: (a) Independent Learners (IND-OFUL), wherein each agent independently runs OFUL algorithm of Abbasi-Yadkori et al. [2011], and there is no communication between agents (zero collaboration), and (b) DISLINUCB, for which we use the implementation of Wang et al. [2020] without any modification.

Simulation setup. We generate the ε -MALCB problem for M = 60 , d = 30 , T = 10000 via the following procedure. We first choose a value of ε in each of the three dissimilarity regimes. Then we create the linear parameters { θ m } M m =1 as follows. Let u, { v m } M m =1 be random vectors with unit norm. We set θ m = c · u + ε 2 v m , where c is a constant in the range [0 , 1 -ε ] . This guarantees ∥ θ m ∥ ≤ 1 and ∥ θ i -θ j ∥ ≤ ε for any two agents i, j . At each round, for each agent, we create a new decision set with a size of 50 , each action is random and normalized to 1. The random noise is sampled from the standard normal distribution, η ∼ N (0 , 1) . We run each experiment 10 times, then report the

group regret averaged over the runs and the confidence intervals in Figure 1. Our code is available here: https://github.com/anhddo/hlinUCB .

Results and discussions. In regime (i), where the level of dissimilarity is small, plots (a) and (b) show that H-LINUCB retains a regret comparable with DISLINUCB.

In regime (ii), plots (c) and (d) illustrate the interpolation between the two extreme regimes.

In regime (iii), plots (e) and (f), DISLINUCB incurs linear regret, H-LINUCB has the same rate with IND-OFUL. This illustrates that collaboration brings no benefit when the dissimilarity is high.

6 Conclusions

In this paper, we studied the heterogeneous multi-agent linear contextual bandit problem. We formulated the problem under the notion of ε -MALCB, and provided the upper and lower bounds when ε is known. We showed that our results are provably optimal in the regime where tasks are highly similar or highly dissimilar. Finally, we validated our theoretical results with numerical simulations on synthetic data.

A natural avenue for future work would be to close the gap in the regime ε ∈ ( 1 √ MT , d √ T ) . Another research direction pertains to designing an adaptive algorithm when ε is unknown. Such an algorithm would be practical and flexible enough to apply to a wide range of heterogeneous multi-agent bandit problems. We are also interested in extending this work to a more challenging setting such as Reinforcement Learning.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011. 2, 3, 4, 6, 7, 9, 14, 22, 23

Djallel Bouneffouf, Irina Rish, and Charu Aggarwal. Survey on applications of multi-armed and contextual bandits. In 2020 IEEE Congress on Evolutionary Computation (CEC) , pages 1-8. IEEE, 2020. 2

Emma Brunskill and Lihong Li. Sample complexity of multi-task reinforcement learning. In Uncertainty in Artificial Intelligence , page 122. Citeseer, 2013. 1

Leonardo Cella, Karim Lounici, Grégoire Pacreau, and Massimiliano Pontil. Multi-task representation learning with stochastic linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 4822-4847. PMLR, 2023. 3

Niladri Chatterji, Vidya Muthukumar, and Peter Bartlett. Osom: A simultaneously optimal algorithm for multi-armed and linear contextual bandits. In International Conference on Artificial Intelligence and Statistics , pages 1844-1854. PMLR, 2020. 2

Ronshee Chawla, Abishek Sankararaman, and Sanjay Shakkottai. Multi-agent low-dimensional linear bandits. IEEE Transactions on Automatic Control , 2022. 2

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011. 2

Yihan Du, Longbo Huang, and Wen Sun. Multi-task representation learning for pure exploration in linear bandits. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on

Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 8511-8564. PMLR, 23-29 Jul 2023. 3

Abhimanyu Dubey and Alex ` Sandy'Pentland. Differentially-private federated linear bandits. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6003-6014. Curran Associates, Inc., 2020. 2, 6, 14

Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33:11478-11489, 2020. 3, 6

Claudio Gentile, Shuai Li, and Giovanni Zappella. Online clustering of bandits. In International Conference on Machine Learning , pages 757-765. PMLR, 2014. 1, 3

Avishek Ghosh, Sayak Ray Chowdhury, and Aditya Gopalan. Misspecified linear bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017. 3

Avishek Ghosh, Abishek Sankararaman, and Kannan Ramchandran. Multi-agent heterogeneous stochastic linear bandits. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19-23, 2022, Proceedings, Part IV , pages 300-316. Springer, 2023. 1, 2, 3, 8

Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 3, 6, 7, 15

Jiachen Hu, Xiaoyu Chen, Chi Jin, Lihong Li, and Liwei Wang. Near-optimal representation learning for linear bandits and linear rl. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4349-4358. PMLR, 18-24 Jul 2021. 1, 3

Ruiquan Huang, Weiqiang Wu, Jing Yang, and Cong Shen. Federated linear contextual bandits. Advances in Neural Information Processing Systems , 34:27057-27068, 2021. 1, 2, 3, 4

Nathan Korda, Balazs Szorenyi, and Shuai Li. Distributed clustering of linear bandits in peer to peer networks. In International conference on machine learning , pages 1301-1309. PMLR, 2016. 3

Tor Lattimore and Szepesvári Csaba. Bandit algorithms . Cambridge University Press, 2020. 3, 6, 22, 23

Chuanhao Li and Hongning Wang. Asynchronous upper confidence bound algorithms for federated linear bandits. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 6529-6553. PMLR, 28-30 Mar 2022. 1, 3, 4

Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. Proceedings of the 19th international conference on World wide web - WWW '10 , 2010. doi: 10.1145/1772690.1772758. 2

Shuai Li, Claudio Gentile, and Alexandros Karatzoglou. Graph clustering bandits for recommendation. arXiv preprint arXiv:1605.00596 , 2016. 3

Shuai Li, Wei Chen, Shuai Li, and Kwong-Sak Leung. Improved algorithm on online clustering of bandits. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , IJCAI'19, page 2923-2929. AAAI Press, 2019. ISBN 9780999241141. 3

Kanak Mahadik, Qingyun Wu, Shuai Li, and Amit Sabne. Fast distributed bandits for online recommendation systems. In Proceedings of the 34th ACM international conference on supercomputing , pages 1-13, 2020. 2

David Martínez-Rubio, Varun Kanade, and Patrick Rebeschini. Decentralized cooperative stochastic bandits. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. 2

Aritra Mitra, Arman Adibi, George J. Pappas, and Hamed Hassani. Collaborative linear bandits with adversarial agents: Near-optimal regret bounds. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 2

Ahmadreza Moradipari, Mohammad Ghavamzadeh, and Mahnoosh Alizadeh. Collaborative multiagent stochastic linear bandits. In 2022 American Control Conference (ACC) , pages 2761-2766. IEEE, 2022. 2

Michael T. Rosenstein. To transfer or not to transfer. In NIPS 2005 , 2005. 1

Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411, 2010. 2

Marta Soare, Ouais Alsharif, Alessandro Lazaric, and Joelle Pineau. Multi-task linear bandits. In NIPS2014 workshop on transfer and multi-task learning: theory meets practice , 2014. 2

Kei Takemura, Shinji Ito, Daisuke Hatano, Hanna Sumita, Takuro Fukunaga, Naonori Kakimura, and Ken-ichi Kawarabayashi. A parameter-free algorithm for misspecified linear contextual bandits. In International Conference on Artificial Intelligence and Statistics , pages 3367-3375. PMLR, 2021. 3, 6

Yuanhao Wang, Jiachen Hu, Xiaoyu Chen, and Liwei Wang. Distributed bandit learning: Near-optimal regret with efficient communication. In International Conference on Learning Representations , 2020. 2, 6, 7, 9, 14, 15

Zhi Wang, Chicheng Zhang, Manish Kumar Singh, Laurel Riek, and Kamalika Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 1531-1539. PMLR, 13-15 Apr 2021. 1, 2, 3, 4

Jiaqi Yang, Wei Hu, Jason D. Lee, and Simon Shaolei Du. Impact of representation learning in linear bandits. In International Conference on Learning Representations , 2021. 1, 3

Jiaqi Yang, Qi Lei, Jason D Lee, and Simon S Du. Nearly minimax algorithms for linear bandits with shared representation. arXiv preprint arXiv:2203.15664 , 2022. 3

Weitong Zhang, Jiafan He, Zhiyuan Fan, and Quanquan Gu. On the interplay between misspecification and sub-optimality gap in linear contextual bandits. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 41111-41132. PMLR, 23-29 Jul 2023. 6

A Proof of Upper Bound

A.1 Proof of Lemma 4.1

Proof. We prove that the true parameter of each agent lies in the Confidence Ellipsoid with high probability. In particular, we prove such results for two periods, for round t < τ and t ≥ τ . We denote δ 1 , δ 2 as the probabilities such that true linear parameters lie outside the confidence ellipsoid for the period of t < τ , and t ≥ τ , respectively.

For the round t < τ , let t s be the last round that synchronization occurs, where t s < t . Then each agent's statistics can be divided into two parts, its own data and the pooled data from other agents since round t s . Specifically, for an agent m , at round t , we have the following for V m,t , b m,t ,

̸

We then have the following for ˆ θ m,t ,

̸

̸

Notice that the summation in the first term is the covariance matrix V m,t but missing the regularization term λI ; thus, the first term can be simplified as θ m -λV -1 m,t θ m . Applying triangle inequality, we have the following for ∥ ∥ ∥ ˆ θ m,t -θ m ∥ ∥ ∥ V m,t ,

̸

For the regularization term I 1 , it is upper bounded by,

̸

̸

̸

̸

̸

For the dissimilarity term I 2 , we have,

̸

where we use triangle inequality in the first inequality, the second inequality holds since w m,t ≤ α/ ∥ x m,t ∥ V -1 m,t by the definition of w m,t , the last inequality holds due to the bounded parameters assumption and the ε -MALCB definition (see Assumption 3.1 and Definition 3.1).

̸

To bound term I 3 , we denote vector ¯ x m,t = √ w m,t x m,t , and the random noise ¯ η m,t = √ w m,t η m,t . We have w m,t ≤ 1 , therefore, ¯ η m,t is 1 -subGaussian. Furthermore, we rewrite the covariance matrix as V m,t = λI + ∑ t s =1 ¯ x m,s ¯ x ⊤ m,s + ∑ i = m ∑ t s k =1 ¯ x i,k ¯ x ⊤ i,k . We then have the following for the noise term I 3 ,

̸

where we apply Lemma C.4 in the first inequality. Now, putting these three terms together, we have the following Confidence Ellipsoid bound for round t < τ ,

We now turn our attention to the round t ≥ τ . Since agents switch to independent learning , we follow the same argument for confidence ellipsoid as in Theorem 2 of Abbasi-Yadkori et al. [2011], and get

Finally, we complete the proof by taking a union bound over all agents.

A.2 Proof of Theorem 4.1

Before proving Theorem 4.1, we give an upper bound on the pseudo-regret, r m,t , in the next proposition. The proof follows standard arguments for linear bandits; most of the arguments can be extracted from Wang et al. [2020], Dubey and Pentland [2020].

Proposition A.1. The pseudo-regret r t obtained by any agent m at round t , is upper bounded as

Proof. Recall that an agent makes decisions optimistically: ( x m,t , ˜ θ m,t ) = arg max ( x,θ ) ∈D m,t ×C m,t x ⊤ θ.

Let x ⋆ m,t denote the optimal action at round t of agent m , i.e., x ⋆ m,t = arg max x ∈D m,t x ⊤ θ m .

We then have the following for the pseudo-regret

where the last inequality is due to the fact that β t ≥ 1 and that suboptimality is no more than 2.

We now proceed to prove Theorem 4.1. The proof technique follows the analysis of group regret in Wang et al. [2020] and applies the arguments from Theorem 4.2 of He et al. [2022] to handle the dissimilarity.

Theorem A.1 (Theorem 4.1 restated) . For a given T , and τ ≤ T , if we set D = τ log( Mτ ) / ( dM ) , and β t , ∀ t ∈ [ τ ] according to Lemma 4.1, then, given Assumption 3.1, with probability at least 1 -Mδ 1 , the group regret of Algorithm 1 up to round τ , is upper bounded as

where ξ t = log ( 1+ Mt/ ( λd ) δ 1 ) . Furthermore, if we choose λ = 1 , α = √ d εMτ , then the group regret is upper bounded as

We refer to this group regret as the 'collaboration' regret, and denote it as R collab ( M,T ) .

Proof. Let P be the total number of synchronization rounds, then, we can index the synchronization matrix V syn for P epoch as { V syn,p } P p =1 , and define V p = λI + V syn,p . Observe that det( V 0 ) = det( λI ) = λ d and det( V P ) ≤ (trace( V P ) /d ) d ≤ ( λ + Mτ/d ) d . Therefore,

By telescoping, we have that log det( V P ) det( V 0 ) = ∑ P p =1 log det( V p ) det( V p -1 ) . Therefore, we have at most R = ⌈ d log(1 + Mτ λd ) ⌉ epochs such that log det( V p ) det( V p -1 ) ≥ 1 ; otherwise, it violates the condition

in Equation (1). WLOG, we use logarithm base 2 for the determinant ratio. This implies that for all but R epochs,

We call the epochs satisfying Equation (2) as good epochs. We imagine a single agent playing M ( τ -1) actions x 1 , 1 , x 2 , 1 , . . . , x M -1 ,τ -1 , x M,τ -1 sequentially. Let W m,t = λI + ∑ M i =1 ∑ t -1 s =1 w i,s x i,s x ⊤ i,s + ∑ m -1 j =1 w j,t x j,t x ⊤ j,t be the covariance matrix of this imaginary agent when it gets to x m,t . If x m,t belongs to a good epoch, say the j -th epoch, we have the following:

where the first inequality is due to the fact that W m,t ≽ V m,t , and the second inequality follows from Lemma C.2.

Now, applying the Proposition A.1, we bound the pseudo-regret r m,t of these good epochs as follows:

where we use Lemma C.2 in the second inequality, and Equation (3) in the third inequality.

Let R good ( M,τ ) be the group regret of these good epochs up to round τ . Suppose P good contains all the good epochs, and B p contains all the pairs ( m,t ) belonging to epoch p . We have

where in the last equality we split the summation to two cases: w m,t = 1 , and w m,t < 1 .

For term I 1 , we consider all the pair ( m,t ) in good epochs up to round τ such that w m,t = 1 ; we assume that we have a total of such K pairs, and these pairs can be sequentially listed as { ( ¯ m 1 , ¯ t 1 ) , ( ¯ m 2 , ¯ t 2 ) , · · · , ( ¯ m K , ¯ t K ) } . With this notation, for i ≤ K , we construct the auxiliary covariance matrix A ¯ m i , ¯ t i = λI + ∑ i -1 k =1 x ¯ m k , ¯ t k x ⊤ ¯ m k , ¯ t k . Thus, we have W ¯ m i , ¯ t i ≽ A ¯ m i , ¯ t i , which implies ∥ x ¯ m i , ¯ t i ∥ W -1 ¯ mi , ¯ t i ≤ ∥ x ¯ m i , ¯ t i ∥ A -1 ¯ mi , ¯ t i .

We now have the following for term I 1 :

where in the second inequality we use Cauchy Schwarz's inequality and the fact that K ≤ Mτ , and use Lemma C.3 in the second inequality.

For term I 2 , since we consider the case such that w m,t < 1 , thus, by the definition of w m,t we have 1 α w m,t · ∥ x m,t ∥ V -1 m,t = 1 . Therefore, we have,

where the last inequality follows from Equation (3).

Similarly to what we have done with I 1 , we consider all the pair ( m,t ) in good epochs up to round τ such that w m,t < 1 ; we assume that we have a total of such K pairs, and these pairs can be sequentially listed as { ( ¯ m 1 , ¯ t 1 ) , ( ¯ m 2 , ¯ t 2 ) , · · · , ( ¯ m K , ¯ t K ) } . Furthermore, we define ¯ x m,t = √ w m,t x m,t , and construct the auxiliary covariance matrix,

Thus, we have W ¯ m i , ¯ t i ≽ ¯ W ¯ m i , ¯ t i , which implies ∥ x ¯ m i , ¯ t i ∥ W -1 ¯ mi , ¯ t i ≤ ∥ x ¯ m i , ¯ t i ∥ ¯ W -1 ¯ mi , ¯ t i .

Therefore,

where we use Lemma C.3 in the fourth inequality. Putting the bounds for I 1 and I 2 together, we have

Plugging β τ -1 = √ λ + αεMτ + √ d log( 1+ Mτ/λ δ 1 ) , the regret of the good epochs is bounded as

Now, we focus on the epochs that are not good , which do not satisfy Equation (2). Let p be one such epoch, and let t 0 be the first round and n be the length, respectively, of the epoch p . Recall that at the beginning of each epoch, agents' covariance matrices are synchronized, i.e., V m,t 0 = λI + V syn , ∀ m ∈ [ M ] . We can then bound the regret of the (bad) epoch p as follows.

where we use Cauchy-Schwarz's inequality and Lemma C.3 in the second inequality.

Lets say the synchronization is triggered on round t 0 + n +1 , i.e., for one of the agents we have that ( n +1)log det( V m,t 0 + n +1 ) det( V m,t 0 ) > D . Since n log det( V m,t 0 + n ) det( V m,t 0 ) < D for all m ∈ [ M ] , for this bad epoch (i.e., from round t 0 to round t 0 + n ), we can bound the group regret as

As we argued earlier, since det V p ≤ ( λ + MT d ) d , the bad epochs are rare, i.e., there are at most R = ⌈ d log(1 + Mτ λd ) ⌉ (otherwise, if det( V p ) / det( V p -1 ) > 2 for more than R rounds, then the condition of Equation (1) is violated). Setting D = τ log( Mτ ) / ( dM ) and plugging β τ -1 into the bound above,

We finish the proof by putting the regret of the good and the bad epochs together,

A.3 Proof of Theorem 4.2

Proof. Setting δ 1 = δ 2 = 1 / ( M 2 T ) , then the group regret caused by failure event, which does not satisfy Lemma 4.1, is at most MT · ( Mδ 1 + Mδ 2 ) = O (1) . Thus, we mainly consider the group regret when Lemma 4.1 holds.

From round τ onward, agents switch to independent learning mode, this is the group regret from round τ to the end, we denote it as R ind ( M,T ) . Furthermore, for round t ≥ τ , we have V m,t = λI + ∑ t -1 s = τ x m,s x ⊤ m,s as the gram matrix which only contains agent's data.

Applying Proposition A.1, we have the following

where we use Cauchy-Schwarz's inequality in the first inequality, and Lemma 4.1 in the second inequality. Combining with the regret from the beginning up to round τ from Theorem 4.1, we have

We notice that Equation (4) has the second term that is linear in term of Mτ . To avoid linear regret, the choice of τ needs to adapt to the dissimilarity level ε .

For the last two terms, by Cauchy-Schwarz's inequality, we have εdMτ + dM √ T -τ ≤ dM √ 2( ε 2 τ 2 + T -τ )) , this term can be optimized by setting τ = 1 2 ε 2 . We need to restrict τ = min( ⌊ 1 2 ε -2 ⌋ , T ) since τ can not exceed T . In other words, all agents can collaborate up to round min ( ⌊ 1 2 ε -2 ⌋ , T ) .

Next, we consider the following two cases:

Case 1: For 1 2 ε -2 ≤ T , we have that dM √ T -τ ≤ dM √ T , and εdMτ = dM 2 ε ≤ dM √ T . Thus, εdMτ + dM √ T -τ ≤ 2 dM √ T .

√

Case 2: For 1 2 ε -2 > T , we have τ = T . This implies that εdMτ + dM √ T -τ = εdMT . We also have εdMT < 1 √ 2 dM √ T due to 1 2 ε 2 > T .

Notice that, in both cases, εdMτ + dM T -τ is evaluated as small as εdMT and always be upper bounded by 2 dM √ T . Therefore, in Equation (4), the summation of the last two terms of the group regret is upper bounded by 2 min { εdMT,dM √ T } .

By the choice of δ 1 , δ 2 , λ , and assume that d ≥ 1 , we can upper bound ξ T ≤ 4 log( MT ) . Therefore, we have the following for the expected group regret

B Proof of Lower Bound

B.1 Proof of Theorem 4.3

Before proving the theorem, we give a formal definition of ε -MALCB problem for p -norm,

Definition B.1. Given an instance of multi-agent linear bandits, it belongs to the class of p -norm ε -MALCB, which we denote as I p ( ε ) , if for all i, j ∈ [ M ] , ∥ θ i -θ j ∥ p ≤ ε .

To prove Theorem 4.3, we use the results of the two following lemmas on max-norm ε -MALCB instances.

LemmaB.1. Let I ∞ ( ε ) be the class of max-norm ε -MALCB instances that satisfy the Assumption 3.1. For ε ≥ 0 , we have the following,

LemmaB.2. Let I ∞ ( ε ) be the class of max-norm ε -MALCB instances that satisfy the Assumption 3.1. Assume d ≤ 2 T, d 2 48 ≤ T . For ε ≥ 0 , we have the following,

Proof of Theorem 4.3. Combining the results of Lemma B.1 and Lemma B.2, we have the following results for the class of max-norm ε -MALCB, for ε ≥ 0 ,

In addition, ∥ x ∥ ∞ ≤ ε implies ∥ x ∥ 2 ≤ ε √ d , therefore, I ∞ ( ε ) ⊆ I 2 ( ε √ d ) . In other words, for ε ≥ 0 ,

Proof of Lemma B.1. In this lemma, we prove that solving any ε -MALCB instance for T rounds is at least as hard as solving a (single-agent) linear bandits for MT rounds. We prove the lemma by contradiction, which is based on linear bandits lower bound of Lemma C.1. We have that I ∞ ( ε ′ ) ⊆ I ∞ ( ε ) , for 0 ≤ ε ′ ≤ ε . This implies, for ε ≥ 0 ,

We completes the proof by proving inf A sup I∈I ∞ (0) R A , I ( M,T ) = Ω ( d √ MT ) .

Now, we assume there exists an algorithm A which achieves sup I∈I ∞ (0) R A , I ( M,T ) < d √ MT 16 √ 3 .

We observe that I ∞ (0) is the class of multi-agent solving exactly the same linear bandits problem. We simulate the algorithm B on a single agent linear bandits for MT rounds by the protocol of M agents solving an identical linear bandits for T rounds. Therefore, if we have R A , I ( M,T ) < d √ MT 16 √ 3 then we also achieve R B , I ( MT ) < d √ MT 16 √ 3 , which contradicts Lemma C.1. Thus, we have sup I∈I ∞ (0) R A , I ( M,T ) ≥ d √ MT √ , which completes the proof.

Proof of Lemma B.2. We extend Lemma C.1 to multi-agent linear contextual bandit by constructing the following max-norm ε -MALCB instance.

Max-norm ε -MALCB instance. The set of linear parameter of M agents { θ m } M m =1 belong to a d -dimensional hypercube θ m ∈ {± ε } d , where ε ∈ [0 , 1 √ d ] . Let D = { x ∈ R d : ∥ x ∥ 2 ≤ 1 } be the action set given to agents at every round. The reward when agent m picks action x is defined as r m,x = θ ⊤ m x + η m,x , where the noise samples from a standard normal distribution, η m,x ∼ N (0 , 1) .

We first verify if the instance satisfies the Assumption 3.1. It satisfies the context assumption since we also restrict the context vector to lie in the unit ball. It also satisfies the ∥ θ m ∥ ≤ 1 assumption because we restrict ε ∈ [0 , 1 √ d ] . Finally, η m,x ∼ N (0 , 1) is 1-subGaussian distribution. This instance belongs to I ∞ (2 ε ) since ∥ θ i -θ j ∥ ∞ ≤ 2 ε for all i, j ∈ [ M ] .

Now, we proceed to prove Lemma B.2. Let { θ m } M m =1 be a set of parameters of the max-norm ε -MALCB instance. For brevity, we omit the commas in the subscripts when it is clear from the context, e.g., θ mi = θ m,i or x tmi = x t,m,i , ∀ t ∈ [ T ] , m ∈ [ M ] , i ∈ [ d ] . Given m ∈ [ M ] , i ∈ [ d ] , we define the stopping time τ mi = T ∧ min { t : ∑ t s =1 x 2 smi ≥ t/d } , and the function U mi ( z ) = ∑ τ mi t =1 ( 1 √ d -x tmi · z ) 2 . We then have following result for U mi (1) :

Let x ⋆ m be the optimal action of agent m . We then have the following for the group regret

where we use the fact that optimal action x ⋆ m = [ θ m 1 ∥ θ m ∥ , · · · , θ md ∥ θ m ∥ ] ⊤ , which is a unit vector and has the same direction with θ m , the first inequality holds due to ∥ x tm ∥ 2 ≤ 1 .

̸

Now, let { θ ′ m } M m =1 be another set of linear parameters that are different at only one coordinate of the linear parameter of only one agent compared to { θ m } M m =1 . Specifically, fix g ∈ [ M ] , i ∈ [ d ] , we have θ g,i = -θ ′ g,i ; otherwise, θ k,j = θ ′ k,j , for ( k, j ) = ( g, i ) , ∀ k ∈ [ M ] , j ∈ [ d ] . To simplify the notion, we define Φ , Φ ′ as Md -dimensional vectors, where Φ , Φ ′ ∈ {± ε } Md . That is, Φ , Φ ′ represent { θ m } M m =1 , { θ ′ m } M m =1 , respectively. We let P and P ′ be the law of U gi ( z ) w.r.t. the interaction of the multi-agent linear bandits induced by Φ , Φ ′ , respectively. We then get

where the first inequality holds due to Lemma C.5 and Equation (5), the second inequality follows the stopping time arguments from Lattimore and Csaba [2020], the last inequality holds due to the assumption that d ≤ 2 T . Then,

where the last inequality holds for ε ∈ [0 , 1 4 √ 3 √ d T ] , this satisfies the requirement ε ∈ [0 , 1 √ d ] of the instance construction due to the assumption that d 2 48 ≤ T . Furthermore, we denote Φ -mi as a ( Md -1) -dimensional vector, which is obtained by excluding Φ mi from vector Φ . Let R Φ ( M,T ) be the regret w.r.t. Φ , and applying randomization hammer , we have the following,

Therefore, we conclude that there exists an instance with the parameter set of { θ m } M m =1 such that R ( M,T ) ≥ εMT √ d 4 , for ε ∈ [0 , 1 4 √ 3 √ d T ] . Notice that this proof holds for I ∞ (2 ε ) class. Scaling down by 2, we have the following for I ∞ ( ε ) class,

Observe that εMT √ d 8 is a strictly increasing function w.r.t. ε . It is at most dM √ T 16 √ 3 , when ε = 1 2 √ 3 √ d T . In other words, R I ∞ ( ε ) ( M,T ) ≥ min( εMT √ d 8 , dM √ T 16 √ 3 ) for ε ∈ [0 , 1 2 √ 3 √ d T ] .

Recall that inf A sup I∈I ∞ ( ε ) R ( M,T ) ≥ inf A sup I∈I ∞ ( ε ′ ) R ( M,T ) , for any ε ≥ ε ′ ≥ 0 . Thus, we conclude, ∀ ε ≥ 0 , the following holds,

which completes the proof.

C Supporting Lemmas

Lemma C.1. ([Lattimore and Csaba, 2020, Theorem 24.2]). Assume d ≤ 2 n and let D = { x ∈ R d : ∥ x ∥ 2 ≤ 1 } . Then there exists a parameter vector θ ∈ R d with ∥ θ ∥ 2 2 = d 2 / (48 n ) such that R n ( D , θ ) ≥ d √ n/ (16 √ 3) .

Lemma C.2. ([Abbasi-Yadkori et al., 2011, Lemma 12]). Let A,B and C be positive semi-definite matrices such that A = B + C . Then, we have that

̸

Lemma C.3. ([Abbasi-Yadkori et al., 2011, Lemma 11]). Let { X t } ∞ t =1 be a sequence in R d , V a d × d positive definite matrix and define ¯ V t = V + ∑ t s =1 X s X ⊤ s . Then, we have that

Further, if ∥ X t ∥ 2 ≤ L for all t , then

Lemma C.4. (Self-Normalized Bound for Vector-Valued Martingales, [Abbasi-Yadkori et al., 2011, Theorem 1]). Let { F t } ∞ t =0 be a filtration. Let { η t } ∞ t =1 be a real-valued stochastic process such that η t is F t -measurable and η t is conditionally R -sub-Gaussian for some R ≥ 0 i.e.

Let { X t } ∞ t =1 be an R d -valued stochastic process such that X t is F t -1 -measurable. Assume that V is a d × d positive definite matrix. For any t ≥ 0 , define ¯ V t = V + ∑ t s =1 X s X ⊤ s . Then, for any δ > 0 , with probability at least 1 -δ , for all t ≥ 0 ,

Lemma C.5. (Pinsker's inequality [Lattimore and Csaba, 2020, Equation 14.12]) For measures P and Q on the same probability space (Ω , F ) , we have the following,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the heterogeneous multi-agent contextual bandit problems. They propose the H-LinUCB algorithm, where agents coordinate at the beginning stage by pooling and synchronizing the data, until a certain time determined by the dissimilarity. They show the algorithm is optimal when the tasks are highly similar or highly dissimilar.\n\nSTRENGTHS:\nThe paper is well-written and organized in general. Multi-agent learning, especially in the heterogeneous setting, is of great interest in the literature. The scenario considered is hard, as they don't add special structures for the parameters other than assuming the dissimilarity is known. Yet the results are strong and seem to be theoretically rigorous. Moreover, this work improves DISLINUCB by discarding coordination when the dissimilarity level is too high and the coordination hurts the regret.\n\nWEAKNESSES:\nThe experiment part is relatively weak to me. Currently, the authors consider three settings of different levels of dissimilarity to highlight the advantage of the proposed H-LINUCB in achieving good regret in all settings. However, there isn't a setting where H-LINUCB dominates the other two algorithms. This makes H-LINUCB more likely an interpolation of the fully communicating and fully independent methods. I hope the authors could at least show some experiments where H-LINUCB uniquely succeeds or achieves the best regret, or discuss why this is impossible or hard.\n\nQUESTIONS:\nAlthough both plots (c) and (d) correspond to regime (ii), H-LINUCB performs differently in the two settings. It resembles Ind-OFUL in (c) but not in (d). What's the key component that leads to the different behavior? Is there a threshold for $\\epsilon$ above which H-LINUCB could be shown to copy Ind-OFUL?\n\nLIMITATIONS:\nI don't see any limitations or potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a multi-agent linear contextual bandit (with central server coordination) setting where $M$ agents each has (possibly different) unknown but fixed $d$-dimensional bandit parameter $\\theta_m$, and the $\\ell_2$-norm bound on the bandit parameters $\\epsilon$ (i.e., $||\\theta_i - \\theta_j||_2 \\leq \\epsilon, \\forall i, j \\in [M]$) is known to the agents. The authors propose a UCB-based algorithm that collaborates for the first $\\epsilon^{-2}$ time slots and then learns individually. The authors also present the related regret upper bound and lower bound and run experiments on synthetic data.\n\nSTRENGTHS:\n- Heterogeneous collaborative multi-agent setting is an upcoming area of interest, so this paper is likely to be relevant to the community.\n- The idea of adjusting the length of collaboration by the dissimilarity (i.e., $\\tau = \\min(\\lfloor\\epsilon^{-2}\\rfloor, T)$) is neat and intuitive.\n- The code used for the experiments is also included and the results should be easily replicable.\n\nWEAKNESSES:\n- The introduction of the proposed algorithm (Section 4.1) is not easy to follow and understand, many new variables are used without explanations\n- Notion of heterogeneity and the part of algorithm for handling collaboration with dissimilarity seem to be straightforward extensions of prior works, the new technical challenges for analyzing the proposed model and algorithm are not clearly discussed\n\nQUESTIONS:\n- How does the algorithm perform if the given $\\epsilon$ is off from the real $\\epsilon$ of the environment? \n- How does the result in this paper generalize to linear (non-contextual) bandit?\n- How important is the dissimilarity defined on $\\ell_2$-norm? Why didn't the authors consider $\\ell_\\infty$ norm as prior work?\n- (minor point) It is not clear what kind of norm the one in line 42 is.\n\nLIMITATIONS:\nThe authors discuss some limitations (that $\\epsilon$ is assumed known and that the proposed algorithm is not optimal for some values of $\\epsilon$) in the conclusion section. In my opinion, the primary limitation of this work is the practicality of the proposed algorithm as $\\epsilon$ is often not precisely known in real application.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multi-agent linear stochastic bandit under a model of heterogeneity of the agents. Concretely, the model assumes a centralized controller who can communicate information to and from all of the M different agents in the network. At each time, every agent in the network, will play an arm from a set of possible arms (that are adversarially chosen). The choice of which arm an agent plays is a function of only the past information at that agent -- namely the agent's past arms pulled and rewards observed and the information communicated by the central controller to the agent. Subsequently, the agent observes a noisy stochastic reward where the noise is independent across agents and time. \n\nTo facilitate group learning, the paper assumes structure among the heterogeneity. Concretely, the paper assumes that the unknown parameter for any pair of agents is at-most \\epsilon away in L2 distance. This modeling assumes that although the agents are different, nonetheless have a latent structure which can enable faster learning.\n\nUnder this structure, the paper proposes an algorithm such that the sum regret of all agents scale sub-linearly in M and T. The paper also gives lower bound evidencing that their upper bounds are order wise tight.\n\nSTRENGTHS:\nWell articulated problem, algorithm, intuition and the key steps of the analysis.\n\nWEAKNESSES:\nThe main weakness is the positioning and contributions of the paper in comparison to \"Multi-agent heterogeneous stochastic linear bandits, Ghosh et.al. in ECML-PKDD 2022\" which the submission cites. \n\nSpecifically, the submission claims that the model of heterogeneity studied is different from Ghosh et.al. (Line 280 of the submission). However, a cursory examination indicates that the assumption of the submission is stronger than Ghosh et.al's in the sense that Defn 3.1 of the submission implies that the average parameter is close to any agent's parameter by \\epsilon, which is the assumption in Ghosh et.al. Does this imply that the assumptions in the present paper with regard to heterogeneity are stronger than Ghosh et.al.'s? \n\nA somewhat less satisfying feature of the algorithm is also the known \\epsilon which seems to be non needed by Ghosh et.al. ?\n\nQUESTIONS:\nThe main questions for the authors is to elucidate their contributions in comparision to Ghosh et.al.? \n\nSpecifically, \n\n1)The authors make a stronger assumption on the heterogeneity (Defn 3.1) compared to Ghosh et.al. Does this stronger assumption on heterogeneity is what enables the authors to allow for adversarially generated set of arms? \n\n2) The algorithm of Ghosh et.al. does not need \\epsilon while the present paper assumes knowledge of \\epsilon. Can the authors comment on the necessity of this information?\n\n3) In the case of stochastic arms, is the  weaker result of the submission compared to Ghosh et.al. the price needed to pay for having an adversarial arm setup? \n\n***Post Rebuttal: *** Thank you for your response and I have updated my score.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers a multi-agent linear contextual bandit model with heterogeneity among the agents. The authors propose a novel algorithm called H-LinUCB to minimize the group cumulative regret when agents communicate through a central server. When the level of heterogeneity is known to the agents, they show that H-LinUCB is provably optimal in regimes when the agents are highly similar or dissimilar.\n\nSTRENGTHS:\nOriginality:\n------------\n- The authors introduced the notion of heterogeneity as a $\\epsilon$-MALCB problem, inspired by (Wang et al. (2021))\n- The model considered in this paper is well-motivated  \n- Related work covered in detail\n\nQuality:\n-----------\n- The theoretical analysis seems to be concrete\n- Numerical simulations are provided to validate the theoretical results\n\nClarity:\n-----------\n- The paper is well-written and easy to follow, with the exception of some proofs in the supplementary material\n\nSignificance:\n-----------\n- The proposed notion of heterogeneity could be of interest to researchers, particularly for the case when the level of heterogeneity is unknown\n\nWEAKNESSES:\n- The algorithmic ideas and the regret analysis are known extensions of the work in (He et al. (2022), Wang et al. (2020), Dubey et al. (2020))\n- The lower bound proof is extended from the ideas in (Lattimore and Csaba (2020))\n- The algorithm depends on the assumption that the level of heterogeneity $\\epsilon$ is known, which need not be true in realistic scenarios\n- The proof of Theorem 4.1 in Appendix A.2 is not easy to follow, because the steps aren't explained in detail. Even though the regret analysis is extended from (He et al. (2022), Wang et al. (2020), Dubey et al. (2020)), I believe a paper should be self-contained.\n- The lower bound seems to be loose, because it is off from the regret upper bound by a factor of $d$ in one of the terms. Furthermore, it is not applicable in the regime $\\epsilon \\in (1/\\sqrt{MT}, d/\\sqrt{T})$\n\nQUESTIONS:\n- Some of the notation isn't defined in Appendix A.2. What is $V_p$ in line 498? How is the condition on the ratio of determinants obtained below line 499? I was hoping if the authors could re-write the proof of Theorem 4.1 so that it is easy to follow\n\n----\n\nPost rebuttal: I have raised the score from 5 to 6.\n\nLIMITATIONS:\nI don't see any potential negative societal impact of their work", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

comments: SUMMARY:
This paper studies multi-agent linear contextual bandits problem where each agent faces different bandits model. The heterogeneity is captured by the $l_2$ distance between the environment parameters. An upper confidence bound (UCB) algorithm, termed as H-LinUCB is proposed. The regret upper bound nearly-matches the lower bound proved by this paper in both small and large heterogeneity regime. The experimental results also valid the effectiveness of the proposed algorithm.

STRENGTHS:
+ The paper provides solid theoretical results including both regret upper bound and regret lower bound. The discussion of the small and large $\epsilon$ cases are insightful.

+ The paper provides experimental results to validate their theoretical results.

WEAKNESSES:
The major weaknesses is the design of the stopping criterion. Please see the question part for detailed discussion.

QUESTIONS:
It seems like we can have a simpler criterion when $\epsilon$ is known. For example, if $\epsilon\leq 1/\sqrt{T}$, we deploy a H-LinUCB with $\tau=T$ and when $\epsilon\geq 1/\sqrt{T}$, we deploy individual OFUL algorithm for each agent. The regret upper bound is $O(\epsilon MT + d\sqrt{MT})$ and $dM\sqrt{T}$ respectively, which scales the same as that provided in the paper. In my opinion, this rule is simple and may have lower communication and computation cost since there is no communication when $\epsilon$ is large.

Can author discuss whether this method works?

Another question is that is it able to detect the magnitude of $\epsilon$ when $\epsilon$ is unknown so that the algorithm can automatically “stop” to aggregate?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26