SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anh Do Johns Hopkins University ado8@jhu.edu
Thanh Nguyen-Tang Johns Hopkins University nguyent@cs.jhu.edu
Raman Arora Johns Hopkins University arora@cs.jhu.edu
As trained intelligent systems become increasingly pervasive, multi-agent learning has emerged as a popular framework for studying complex interactions between autonomous agents. Yet, a formal understanding of how and when learners in heterogeneous environments benefit from sharing their respective experiences is still in its infancy. In this paper, we seek answers to these questions in the context of linear contextual bandits. We present a novel distributed learning algorithm based on the upper confidence bound (UCB) algorithm, which we refer to as H-LINUCB, wherein agents cooperatively minimize the group regret under the coordination of a central server. In the setting where the level of heterogeneity or dissimilarity across the environments is known to the agents, we show that H-LINUCB is provably optimal in regimes where the tasks are highly similar or highly dissimilar.
Heterogeneous multi-agent systems enable agents to work together and coordinate their actions to solve complex problems. These systems are inherently scalable, as they can distribute the computational load across multiple agents. This scalability allows the system to handle large and sophisticated tasks beyond the capabilities of a single agent. Despite the potential of multi-agent systems, it poses the following fundamental challenges.
- Statistical challenge. Each agent's reward distribution may vary, meaning that different agents receive different rewards for the same action. This heterogeneity in reward distributions introduces complexity and makes coordination among agents more difficult. Furthermore, Wang et al. [2021] point out that an ineffective use of shared data could lead to a significant negative impact on the overall performance. In particular, sharing experiences amongst agents may hinder the system's performance if the tasks are too dissimilar [Rosenstein, 2005, Brunskill and Li, 2013].
- Computational complexity. Coordinating the decisions of numerous agents and performing complex computations pose challenges in terms of computational resources, time constraints, and algorithmic scalability.
- Communication cost. Efficient communication is also a fundamental challenge for a large-scale multi-agent system. As the number of agents in a system increases, the complexity of interactions between agents grows exponentially. Managing the interaction of numerous agents and making decisions in a timely manner becomes increasingly difficult.
Several works address these challenges for heterogeneous multi-agent systems, including federated linear bandits [Huang et al., 2021, Li and Wang, 2022], clustering bandits [Gentile et al., 2014, Ghosh et al., 2023], multi-task linear bandits [Hu et al., 2021, Yang et al., 2021]. However, these works either rely on some special structure of the parameter [Li and Wang, 2022], e.g., a low-rank structure [Hu
et al., 2021] or make different assumptions, e.g., stochastic contexts, finite decision set [Huang et al., 2021, Ghosh et al., 2023], etc.
In this work, we provide a general notion of heterogeneous multi-agent linear contextual bandits  $(\varepsilon$ -MALCB) and give analytical results under different regimes of heterogeneity. Specifically, we study a model that consists of  $M$  agents. Each agent  $i \in [M]$  plays a  $d$ -dimensional linear contextual bandit, parametrized by  $\theta_{i}$  (see Section 3 for more details), for  $T$  rounds. We capture the heterogeneity by a dissimilarity parameter  $\varepsilon > 0$  such that  $\| \theta_{i} - \theta_{j} \|_{2} \leq \varepsilon$ , for all  $i, j \in [M]$ . Notably, we do not assume any special structure on the linear parameter; and we allow the decision set to be infinite and possibly chosen adversarily.
Motivating application. Consider a personalized recommendation system for online advertisements [Li et al., 2010, Bouneffouf et al., 2020, Ghosh et al., 2023]. Here, the platform needs to be adaptive to user preferences and maximize total user clicks based on user-click feedback. Each ad can be represented as a context vector, encoding information such as publisher, topic, ad content, etc. The inner product of the ad's context vector and user preference represents the alignment. A higher inner product value indicates greater relevance of the ad. Furthermore, the recommended ads must be personalized to accommodate user preference differences. One naive approach would involve solving a separate linear contextual bandit problem for each user. However, we can pose the following question: Can we enhance the system's performance by pooling data from other users? If so, to what extent of user heterogeneity can we achieve that?
Contributions. We make the following contributions in this paper.
- First, we formulate the heterogeneous multi-agent linear contextual bandits as  $\varepsilon$ -MALCB problem, building on the classic notion of heterogeneity in multiarmed bandits (MABs) Wang et al. [2021]. Our notion of heterogeneity is natural and captures many settings in real-world applications.
- Second, when the level of dissimilarity is known, we propose a distributed algorithm, namely H-LINUCB which achieves a regret of  $\tilde{\mathcal{O}}(d\sqrt{MT} + \min\{\varepsilon dMT, dM\sqrt{T}\})$  under the coordination of a central server. We discuss in detail how to handle the dissimilarity and introduce a criterion for stopping collaboration when the level of dissimilarity is high. We show that under the regime of low dissimilarity, we can still achieve a regret of  $\tilde{\mathcal{O}}(d\sqrt{MT})$ , which is the same regret rate as if  $M$  agents collaborate to solve the same task. In this regime, H-LINUCB outperforms independent learners improving by a factor of  $\sqrt{M}$ , from  $\tilde{\mathcal{O}}(dM\sqrt{T})$  to  $\tilde{\mathcal{O}}(d\sqrt{MT})$ . This is significant when we have a large number of agents.
- Third, we complement the upper bound with a lower bound of  $\Omega(d\sqrt{MT} + \min\{\varepsilon MT, dM\sqrt{T}\})$ . This suggests that our theoretical guarantees are tight in settings where tasks are highly similar or highly dissimilar.
- Finally, we validate our theoretical results with numerical simulations on synthetic data. When the level of dissimilarity is small, H-LINUCB outperforms independent learning. When the level of dissimilarity is high, our simulation shows that blindly using shared data can lead to linear regret, emphasizing the importance of the criterion we propose for when to stop the collaboration.
The classic linear bandits have a rich literature in both theory and application; see, for example, [Abbasi-Yadmori et al., 2011, Li et al., 2010, Chu et al., 2011, Chatterji et al., 2020, Rusmevichientong and Tsitsiklis, 2010, Bouneffouf et al., 2020, Mahadik et al., 2020], to name a few. With a surge in distributed computing, multi-agent systems have shown their potential and gained more attention in recent years. A large body of works dedicated to studying the homogeneous setting of multiple collaborating agents solve a global linear bandits problem [Wang et al., 2020, Dubey and Pentland, 2020, Moradipari et al., 2022, Mitra et al., 2022, Martínez-Rubio et al., 2019, Chawla et al., 2022].
The problem of multi-agent linear bandits in heterogeneous environments, on the other hand, has received limited attention. Soare et al. [2014] were amongst the first to study heterogeneous linear
bandits; however, the focus in that work is not on group regret, and the authors only consider the setting where tasks are similar. More recently, Huang et al. [2021] proposed an algorithm with a novel multi-agent G-Optimal design. They assume that the heterogeneity comes from the contexts associated with each agent, but agents can still collaborate since they share the same arm parameters. Li and Wang [2022] consider an extension where they assume that each agent's parameter has two components - a shared global component and an individual local component. This formalization requires agents to work on their respective tasks (the local component) but still allows agents to collaborate on the common task (the global component). Wang et al. [2021] study heterogeneity of the Bernoulli MABs problem and provide guarantees for both cases when the level of heterogeneity is known and unknown.
A related line of work studies heterogeneous linear bandits through clustering [Gentile et al., 2014, Li et al., 2016, 2019, Korda et al., 2016, Ghosh et al., 2023]. These works give a guarantee based on the clustering structure of the different linear bandit problems - agents belonging to the same cluster will likely achieve the highest collaboration gain. We do not make any assumption about the "clusterability" of different bandit problems we may encounter. A yet another approach focuses on multi-task linear bandits, wherein we solve multiple different but closely related linear bandits tasks [Yang et al., 2021, 2022, Hu et al., 2021, Cella et al., 2023, Du et al., 2023]. In particular, these works rely on the assumption that all tasks share a common  $k$ -dimensional representation, where  $k \ll d$ . Then, pooling data from different bandits helps learn a good representation and reduces the statistical burden of learning by reducing the linear bandit problem in  $d$  dimensions to a  $k$ -dimensional setting. We do not consider multi-task learning here.
Our formulation of heterogeneous contextual linear bandits is similar to that of misspecified and corrupted bandits setting (Remark 3.1 for more details) [Ghosh et al., 2017, Lattimore and Csaba, 2020, Takemura et al., 2021, Foster et al., 2020, He et al., 2022]. It is then natural to ask if we can apply the techniques from that part of the literature to deal with the dissimilarity between different bandits in a heterogeneous setting. However, there is a fundamental difference in how the two problems manifest themselves. While misspecification may be typically unavoidable in many settings, in a heterogeneous bandit setting, an agent can always choose to rely solely on its own data if it finds that the data from other agents are too dissimilar.
Multi-Agent Linear Contextual Bandits. We consider a multi-agent learning setting with  $M$  agents. At each round  $t \in [T]$ , each agent  $m \in [M]$  picks an action (context) $^1$ $x_{m,t} \in \mathcal{D}_{m,t}$ , where  $\mathcal{D}_{m,t} \subseteq \mathbb{R}^d$  is a given decision set. The agent  $m$  receives reward  $y_{m,t} = x_{m,t}^\top \theta_m + \eta_{m,t}$ , where  $\theta_m \in \mathbb{R}^d$  is an unknown but fixed parameter and  $\eta_{m,t}$  is sub-Gaussian noise. Let  $\mathcal{F}_t$  denote the filtration, i.e., the  $\sigma$ -algebra, induced by  $\sigma(\{x_{m,k}\}_{m \in [M], k \leq t+1}, \{\eta_{m,k}\}_{m \in [M], k \leq t})$ .
Regret. Our goal is to design algorithms for multi-agent linear contextual bandits that achieve a small group regret defined as
$$
\mathcal {R} (M, T) = \sum_ {t = 1} ^ {T} \sum_ {m = 1} ^ {M} \left(\max  _ {x \in \mathcal {D} _ {m, t}} \langle x, \theta_ {m} \rangle - \langle x _ {m, t}, \theta_ {m} \rangle\right).
$$
Assumption 3.1. Without loss of generality, we assume that,
1. Bounded parameters:  $\| \theta_{m} \|_{2} \leq 1, \| x \|_{2} \leq 1, \forall x \in \mathcal{D}_{m,t}, m \in [M], t \in [T]$ .
2. Sub-Gaussian noise:  $\eta_{m,t}$  is conditionally zero-mean 1-sub-Gaussian random variable with respect to  $\mathcal{F}_{t - 1}$ .
We note that the assumptions above are standard in linear bandits literature [Abbasi-Yadkori et al., 2011, Hu et al., 2021, Huang et al., 2021]. Further, it is straightforward to let  $\| \theta_{m} \| \leq B$ , for some constant  $B$ , by appropriately scaling the rewards. We make no additional assumptions on the context. The decision set could be infinite, and given to each agent possibly adversarially.
1 Throughout the paper, we use the terms action and context interchangeably.
Definition 3.1. ( $\varepsilon$ -MALCB Problem) A multi-agent linear contextual bandits problem is said to be an  $\varepsilon$ -MALCB problem instance, if for any two agents  $i,j \in [M]$ ,  $\|\theta_i - \theta_j\|_2 \leq \varepsilon$ , for an  $\varepsilon \geq 0$ . We call  $\varepsilon$  the dissimilarity parameter.
Definition 3.2. (Homogeneous setting) A multi-agent linear contextual bandits problem is homogeneous, if it is an  $\varepsilon$ -MALCB with  $\varepsilon = 0$ , i.e.,  $\theta_{i} = \theta_{j}$ , for all  $i,j\in [M]$ .
Given the bound on the parameters, we have that  $\| \theta_{i} - \theta_{j}\|_{2}\leq 2$  for any  $i,j\in [M]$ . Therefore, it suffices to only consider the case where  $\varepsilon \in [0,2]$ .
Remark 3.1. (Misspecified structure) Under the Assumption 3.1, for any two agents  $i,j$  we have that  $\theta_i^\top x - \varepsilon \leq \theta_j^\top x \leq \theta_i^\top x + \varepsilon$ . Then,  $\mathbb{E}[y_{j,x}] = \theta_i^\top x + \Delta(x)$ , for  $\Delta(x) \in [-\varepsilon, \varepsilon]$ . This represents a misspecified structure wherein agent  $i$  receives the reward  $y_{j,x}$  from agent  $j \neq i$ .
Remark 3.2. (Recover the  $\varepsilon$ -MPMAB of Wang et al. [2021]). We note that the  $\varepsilon$ -MPMAB is a special case of  $\varepsilon$ -MALCB. Define the mean reward of  $K$  arms for agent  $m$  as  $\theta_{m} = [\mu_{1}^{m}, \dots, \mu_{K}^{m}]$ . Then, the reward for arm  $k$  at round  $t$  is  $y_{t,k}^{m} = \theta_{i}^{\top} e_{k} + \eta_{t}$ . The decision set  $\mathcal{D} = \{e_{1}, \dots, e_{K}\}$  are the standard basis vectors. This is a fixed set of arms, given to all agents at each round. The dissimilarity parameter  $\varepsilon$  is defined as:  $\| \theta_{i} - \theta_{j} \|_{\infty} \leq \varepsilon$  for all  $i, j \in [M]$ .
Nonetheless, the results in Wang et al. [2021] are not directly comparable to ours since the dissimilarity parameter  $\varepsilon$  hides inside the size of the set of subpar arms  $|\mathcal{I}_{\varepsilon}|^2$ . Furthermore, Wang et al. [2021] give guarantees in a full-communication setting, in which each agent has full access to the past data of all other agents at every round.
Remark 3.3. There are other formulations that also capture the heterogeneity in multi-agent linear bandits. Huang et al. [2021] consider a multi-agent linear bandits setting with a fixed size decision set, containing  $K$  actions,  $\{\theta_{a}\}_{a = 1}^{K}$ , which is unknown to the agents. Each agent  $i$  is associated with  $K$  different contexts  $\{x_{i,a}\}_{a = 1}^{K}$ . At reach round, each agent  $i$  picks an action  $a\in [K]$ , and receives reward  $r_{i,a} = x_{i,a}^{\top}\theta_{a} + \eta_{i,a}$ . Since  $x_{i,a}$  can vary for different agents, this captures the heterogeneity across agents. It also allows for collaboration across agents since they share the same decision set.
Li and Wang [2022] assume that each agent parameter  $\theta$  has a special structure that consists of a shared global component and a unique local component. The reward of agent  $i$  can be given as,  $r_{i,x} = \left[ \begin{array}{c}\theta^{(g)}\\ \theta^{(i)} \end{array} \right]^{\top}\left[ \begin{array}{c}\mathbf{x}^{(g)}\\ \mathbf{x}^{(l)} \end{array} \right]$ . We do not assume such special structure of the linear parameter.
The notion of  $\varepsilon$ -MALCB is in the worst-case sense. One can imagine an  $\varepsilon$ -MALCB instance of  $M$  agents, such that  $M - 1$  agents have identical linear parameter, i.e.  $\theta_{i} = \theta_{j}, \forall i, j \in [M - 1]$ , and the parameter of the last agent  $\| \theta_{M} - \theta_{M - 1} \|_{2} = \varepsilon$ . With this type of instance, for a large  $M$ , we can simply use DisLINUCB of Wang et al. [2021] and achieve nearly optimal regret. Clustering bandits framework could also be used for this  $\varepsilon$ -MALCB instance since it presents a strong cluster structure. Even though our formulation takes a pessimistic approach but it can handle the case that bandits are largely unclustered.
Our goal is to design a system such that its performance is no worse than running  $M$  independent bandit algorithms for each user (zero collaboration). The system should also be adaptive to the heterogeneity in the problem instance, i.e., it should automatically leverage any structure in the problem parameters  $\{\theta_m\}_{m=1}^M$  to collaboratively solve all bandit problems at a "faster" rate. To benchmark the performance of such a system, we consider the following baseline.
Independent Learners (IND-OFUL). We establish a baseline algorithm in which each agent independently runs an optimal linear contextual bandits algorithm (OFUL, [Abbasi-Yadkori et al., 2011]) without any communication. Each agent incurs  $\tilde{\mathcal{O}}(d\sqrt{T})$  regret, and  $\tilde{\mathcal{O}}(dM\sqrt{T})$  group regret.
Notation. We denote the weighted norm of vector  $x$  w.r.t. matrix  $A$  (Mahalanobis norm) as  $\| x \|_A = \sqrt{x^\top Ax}$ . We write  $A \succcurlyeq B$  iff  $A - B$  is a positive semi-definite matrix. We use  $\tilde{\mathcal{O}}(\cdot)$  to hide the polylogarithmic factors in standard Big O notation.
$^{2}$ ROBUSTAGG( $\varepsilon$ ) algorithm achieves  $\tilde{\mathcal{O}}(\sqrt{\mathcal{I}_{\varepsilon}MT} + M\sqrt{(|\mathcal{I}_{\varepsilon}| - 1)T} + M|\mathcal{I}_{\varepsilon}|)$  regret when  $\varepsilon$  is known. The subpar arms  $\mathcal{I}_{\varepsilon}$  is defined in Wang et al. [2021, Section 3.2].
In this section, we present H-LINUCB, a UCB-style algorithm for  $\varepsilon$ -MALCB problem, and give guarantees for the case when the dissimilarity  $\varepsilon$  is known to the agents. In Section 4.2, we present a lower bound and discuss the implication of our results in different regimes of dissimilarity. We defer the detailed proof to the Appendix.
Algorithm 1 H-LINUCB
Input: Dissimilarity parameter  $\varepsilon$ , number of agents  $M$ , number of rounds  $T$ , regularization parameter  $\lambda$ , dimension  $d$ , confidence parameters  $\beta_{t}, \forall t \in [T]$ , weight threshold parameter  $\alpha$ , collaboration budget  $\tau$ , synchronization threshold  $D$   
1:  $t_{syn} \gets 1$  ▷  $t_{syn}$  is the index of the last synchronized round  
2:  $V_{syn} \gets 0, b_{syn} \gets 0 \triangleright V_{syn}, b_{syn}$  store the relevant statistics of all agents after synchronization  
3:  $V_{epoch,m} \gets 0, b_{epoch,m} \gets 0, \forall m \in [M]$  ▷  $V_{epoch,m}, b_{epoch,m}$  store the relevant statistics of agent  $m$  in the current epoch  
4: for  $t = 1, \dots, T$  do  
5: for Agent  $m = 1, \dots, M$  do  
6: if  $t = \tau$  then  
7:  $V_{syn} \gets 0, b_{syn} \gets 0$   
8:  $V_{epoch,m} \gets 0, b_{epoch,m} \gets 0$   
9: end if  
10:  $V_{m,t} \gets \lambda I + V_{syn} + V_{epoch,m}$   
11:  $\hat{\theta}_{m,t} \gets V_{m,t}^{-1}(b_{syn} + b_{epoch,m})$   
12: Construct the confidence ellipsoid  $\mathcal{C}_{m,t} = \left\{\theta \in \mathbb{R}^{d}: \left\| \hat{\theta}_{m,t} - \theta \right\|_{V_{m,t}} \leq \beta_{t}\right\}$   
13:  $(x_{m,t}, \tilde{\theta}_{m,t}) = \arg \max_{(x,\theta) \in \mathcal{D}_{m,t} \times \mathcal{C}_{m,t}}\langle \theta, x\rangle$   
14: Play  $x_{m,t}$  and get reward  $y_{m,t}$   
15:  $w_{m,t} \gets 11[t < \tau] \min\left(1, \alpha/\|x_{m,t}\|_{V_{m,t}^{-1}}\right) + 11[t \geq \tau]$   
16:  $V_{epoch,m} \gets V_{epoch,m} + w_{m,t}x_{m,t}x_{m,t}^{\top}$   
17:  $b_{epoch,m} \gets b_{epoch,m} + w_{m,t}x_{m,t}y_{m,t}$   
18: if log  $[\det(V_{m,t} + w_{m,t}x_{m,t}x_{m,t}^{\top}) / \det(\lambda I + V_{syn})] \cdot (t - t_{syn}) \geq D$  and  $t < \tau$  then  
19: Send a synchronization signal to the server to start a communication round  
20: end if  
21: if A communication round is started then  
22: Agent  $i$  sends  $V_{epoch,i}, b_{epoch,i}$  to the server,  $\forall i \in [M]$   
23: Server computes  $V_{syn} \gets V_{syn} + V_{epoch,i}, b_{syn} \gets b_{syn} + b_{epoch,i}, \forall i \in [M]$   
24: Server sends  $V_{syn}, b_{syn}$  back to all agents  
25:  $V_{epoch,i} \gets 0; b_{epoch,i} \gets 0; \forall i \in [M] \triangleright$  Reset  $V_{epoch,i}, b_{epoch,i}$  for the new epoch  
26:  $t_{syn} \gets t$   
27: end if  
28: end for  
29: end for
4.1 H-LINUCB Algorithm
H-LINUCB is a distributed UCB-style algorithm (see Algorithm 1 for pseudocode), in which agents work cooperatively under the coordination of a central server.
H-LINUCB has two learning phases: the collaboration phase (for rounds  $t \in \{1, \dots, \tau - 1\}$ ) and the independent learning phase (for rounds  $t \in \{\tau, \dots, T\}$ ), where  $\tau \leq T$  is the collaboration budget. Intuitively, our two-phase learning framework ensures that the agents stop collaboration after  $\tau$  rounds lest they incur a linear regret in bandit environments with large dissimilarity. Naturally, then, the parameter  $\tau$  should depend on the dissimilarity parameter,  $\varepsilon$ . We give an optimal choice of  $\tau$  in Theorem 4.1.
At each round  $t < \tau$  (the collaboration phase), each agent's data is weighted to adapt to the dissimilarity across different agents (Line 15). Then, each agent uses the weighted data to construct
its Confidence Ellipsoid (Line 12) and makes a decision following the optimism principle (Line 13). When a certain condition is met (Line 18), data is pooled and synchronized across the agents. Starting from round  $\tau$ , all collaboration ceases and each agent enters the independent learning mode and runs an independent copy of the OFUL algorithm [Abbasi-Yadkori et al., 2011] locally for the last  $T - \tau + 1$  rounds.
We note that H-LINUCB builds upon DisLINUCB of Wang et al. [2020, Protocol 8] with the following modifications:
- We scale each agent's data using the weight  $\min(1, \alpha / \|x_{m,t}\|_{V_{m,t}^{-1}})$ , which we adopt from He et al. [2022], to handle the dissimilarity across different agents (Line 15).
- We only allow collaboration until round  $\tau$  (Line 18). The value of  $\tau$  depends on the dissimilarity parameter, which we assume is given.
- We reset the variables  $V_{syn}, b_{syn}, V_{epoch,m}, b_{epoch,m}$  at round  $\tau$  (Lines 6-9), where each agent switches to the independent learning mode. Here, epoch refers to the time period between two consecutive synchronization rounds.
Each agent uses all of the data available to them at each round to construct the Confidence Ellipsoid  $\mathcal{C}_{m,t}$  using the result in Lemma 4.1. Given the confidence ellipsoid, the agent chooses the action optimistically:  $(x_{m,t},\tilde{\theta}_{m,t}) = \arg \max_{(x,\theta)\in \mathcal{D}_{m,t}\times \mathcal{C}_{m,t}}\langle \theta ,x\rangle$  . During the collaboration phase, if the variation in the volume of the ellipsoid exceeds a certain synchronization threshold,  $D$  , it triggers a synchronization condition (Lines 18-20). Subsequently, the central server commences the synchronization procedure to update  $V_{syn},b_{syn}$  across all participating agents (Lines 21-27. The optimal value of  $D$  depends on the number of agents  $M$  , dimension  $d$  , and the collaboration budget  $\tau$