Title: Decentralized Randomly Distributed Multi-agent Multi-armed Bandit with Heterogeneous Rewards
Venue: NeurIPS 2023
Accepted: yes
Keywords: decentralized multi-agent MAB

heterogeneous light-tailed and heavy-tailed rewards

time dependent random graphs
Abstract: We study a decentralized multi-agent multi-armed bandit problem in which multiple clients are connected by time dependent random graphs provided by an environment. The reward distributions of each arm vary across clients and rewards are generated independently over time by an environment based on distributions that include both sub-exponential and sub-gaussian distributions. Each client pulls an arm and communicates with neighbors based on the graph provided by the environment. The goal is to minimize the overall regret of the entire system through collaborations. To this end, we introduce a novel algorithmic framework, which first provides robust simulation methods for generating random graphs using rapidly mixing markov chains or the random graph model, and then combines an averaging-based consensus approach with a newly proposed weighting technique and the upper confidence bound to deliver a UCB-type solution. Our algorithms account for the randomness in the graphs, removing the conventional doubly stochasticity assumption, and only require the knowledge of the number of clients at initialization. We derive optimal instance-dependent regret upper bounds of order $\log{T}$ in both sub-gaussian and sub-exponential environments, and a nearly optimal instance-free regret upper bound of order $\sqrt{T}\log T$ up to a $\log T$ factor. Importantly, our regret bounds hold with high probability and capture graph randomness, whereas prior works consider expected regret under assumptions and require more stringent reward distributions.

Decentralized Randomly Distributed Multi-agent Multi-armed Bandit with Heterogeneous Rewards

Abstract

We study a decentralized multi-agent multi-armed bandit problem in which multiple clients are connected by time dependent random graphs provided by an environment. The reward distributions of each arm vary across clients and rewards are generated independently over time by an environment based on distributions that include both sub-exponential and sub-Gaussian distributions. Each client pulls an arm and communicates with neighbors based on the graph provided by the environment. The goal is to minimize the overall regret of the entire system through collaborations. To this end, we introduce a novel algorithmic framework, which first provides robust simulation methods for generating random graphs using rapidly mixing Markov chains or the random graph model, and then combines an averaging-based consensus approach with a newly proposed weighting technique and the upper confidence bound to deliver a UCB-type solution. Our algorithms account for the randomness in the graphs, removing the conventional doubly stochasticity assumption, and only require the knowledge of the number of clients at initialization. We derive optimal instance-dependent regret upper bounds of order log T in both sub-Gaussian and sub-exponential environments, and a nearly optimal mean-gap independent regret upper bound of order √ T log T up to a log T factor. Importantly, our regret bounds hold with high probability and capture graph randomness, whereas prior works consider expected regret under assumptions and require more stringent reward distributions.

1 Introduction

Multi-armed Bandit (MAB) [Auer et al., 2002a,b] is an online sequential decision-making process that balances exploration and exploitation while given partial information. In this process, a single player (agent, client) aims to maximize a cumulative reward or, equivalently, minimize the cumulative loss, known as regret, by pulling an arm and observing the reward of that arm at each time step. The two variants of MAB are adversarial and stochastic MAB, depending on whether rewards are chosen arbitrarily or follow a time-invariant distribution, respectively. Recently, motivated by the development of federated learning [McMahan et al., 2017], multi-agent stochastic multi-armed bandit has been drawing increasing attention (commonly referred to as multi-agent MAB). In this variant, multiple clients collaboratively work with multiple stochastic MABs to maximize the overall performance of the entire system. Likewise, regret is an important performance measure, which is the difference between the cumulative reward of always pulling the global optimal arm by all clients and the actual cumulative reward gained by the clients at the end of the game, where global optimality is defined with respect to the average expected reward values of arms across clients. Thereafter, the question for each client to answer is essentially how to guarantee an optimal regret with limited observations of arms and insufficient information of other clients. Assuming the existence of a central server, also known as the controller, [Bistritz and Leshem, 2018, Zhu et al., 2021b, Huang et al., 2021, Mitra et al., 2021, Réda et al., 2022, Yan et al., 2022], allow a controller-client framework where the controller integrates and distributes the inputs from and to clients, adequately addressing the challenge

posed by the lack of information of other clients. However, this centralization implicitly requires all clients to communicate with one another through the central server and may fail to include common networks with graph structures where clients perform only pair-wise communications within the neighborhoods on the given graphs. Non-complete graphs capture the reality of failed communication links. Removing the centralization assumption leads to a decentralized multi-agent MAB problem, which is a challenging but attracting direction as it connects the bandit problem and graph theory, and precludes traditional centralized processing.

In the field of decentralized multi-agent MAB, it is commonly assumed that the mean reward value of an arm for different clients is the same, or equivalently, homogeneous. This assumption is encountered in [Landgren et al., 2016a,b, 2021, Zhu et al., 2020, Martínez-Rubio et al., 2019, Agarwal et al., 2022, Wang et al., 2022, 2020, Li and Song, 2022, Sankararaman et al., 2019, Chawla et al., 2020]. However, this assumption may not always hold in practical scenarios. In recent years, there has been an increasing emphasis on heterogeneous reward settings, where clients can retain different mean values for the rewards of the same arm. The transition to heterogeneous reward settings presents additional technical challenges. Clients are unable to infer the global optimal arms without sequential communications regarding the rewards of the same arm at other clients. Such communications, however, are limited by the partially observed rewards, as other clients may not pull the same arm, and constrained by the underlying graph structure. We study the heterogeneous setting with time varying graphs.

Traditionally, rewards are assumed to be sub-Gaussian distributed. However, there has been a recent focus on MAB with heavy-tailed reward distributions. This presents a non-trivial challenge as it is harder to concentrate reward observations in sublinear time compared to the light-tailed counterpart [Tao et al., 2022]. In the work of [Jia et al., 2021], sub-exponential rewards are considered and analyzed in the single-agent MAB setting with newly proposed upper confidence bounds. Meanwhile, for multi-agent MAB, heavy-tailed distributions are examined in a homogeneous setting in [Dubey et al., 2020]. However, the heterogeneous setting studied herein has not yet been formulated or analyzed, posing more challenges compared to the homogeneous setting, as discussed earlier.

Besides rewards, the underlying graph assumptions are essential to the decentralized multi-agent MAB problem, as increased communication among clients leads to better identification of global optimal arms and smaller regret. The existing works [Sankararaman et al., 2019, Chawla et al., 2020] relate regret with graph structures and characterize the dependency of regret on the graph complexity with respect to conductance. When considering two special cases, [Chawla et al., 2020] demonstrates the theoretical improvement achieved by circular ring graphs compared to complete graphs, and [Li and Song, 2022] numerically shows that the circular ring graph presents the most challenging scenario with the largest regret, while the complete graph is the simplest. There are two types of graphs from a time perspective: time-invariant graphs, which remain constant over time, and time-varying graphs, which depend on time steps and are more challenging but more general. Assumptions on time-invariant graphs include complete graphs [Wang et al., 2021] where all clients can communicate, regular graphs [Jiang and Cheng, 2023] where each client has the same number of neighbors, and connected graphs under the doubly stochasticity assumption [Zhu et al., 2020, 2021a,b]. Independently from our work, recent work [Zhu and Liu, 2023] has focused on time-varying B -connected graphs, where the composition of every l consecutive graphs is a strongly connected graph. However, their doubly stochasticity assumption, where all elements of edge probability also called weight matrices are uniformly bounded by a positive constant, can be violated in several cases. Additionally, their graphs may be strongly correlated to meet the connectivity condition when l > 1 , which may not always hold in practice. No research has been conducted on time-varying graphs with only connectivity constraints or without constraints on connectivity. Additionally, current time-varying graphs do not provide insight into how the graphs change over time. As the graphs are generated by the environment, similar to the generation of rewards, it remained unexplored considering random graphs in an i.i.d manner, such as random edge failures or random realizations as pointed out for future research in [Martínez-Rubio et al., 2019]. We also address this situation.

Traditionally, random graphs have often been formulated using the Erd˝ os-Rényi (E-R) model, which has been widely adopted in various research domains. The model, described by G ( M,c ) , consists of M vertices with each pair of vertices being connected with probability c . Notably, the E-R model is 1) not necessarily connected and 2) stochastic that allows for random edge failures, and has found applications in mean-field game [Delarue, 2017] and majority vote settings [Lima et al., 2008]. Though it has only been used in numerical experiments for the decentralized multi-agent MAB

setting with homogeneous rewards [Dubey et al., 2020], the theoretical study of this model in this context remained unexplored until this work, let alone with heterogeneous rewards and time-varying graphs. Alternatively, one can consider all connected graphs (there are exponentially many of them), and the environment can randomly sample a connected graph and produce i.i.d. samples of such random connected graphs. This approach mimics the behavior of stochastic rewards and allows the environment to exhaust the sample space of connected graphs independently, without the doubly stochasticity assumption, which, however, has not yet been studied and it is also addressed herein.

For the multi-agent MAB framework, methods in MAB are a natural extension. [Zhu et al., 2021b] adapt the UCB algorithm to the multi-agent setting. This algorithm uses weighted averages to achieve consensus among clients and is shown to have a regret of order log T for time-invariant graphs. A follow-up study in [Zhu and Liu, 2023] re-analyzes this algorithm for time-varying B -connected graphs under the aforementioned assumptions under the doubly stochasticity assumption by adding an additional term compared to UCB. An effective UCB-based method for random graphs without doubly stochasticity assumption and for sub-exponential distributed rewards remained unexplored.

This paper presents a novel contribution to the decentralized multi-agent MAB problem by studying both heterogeneous rewards and time-varying random graphs, where the distributions of rewards and graphs are independent of time. To the best of our knowledge, this is the first work to consider this problem and to investigate it with heavy-tailed reward distributions. Specifically, the paper investigates 1) heterogeneous sub-exponential and sub-Gaussian distributed rewards and 2) random graphs including the possibly disconnected E-R model and random connected graphs, and applies them to the decentralized multi-agent MAB framework. This work bridges the gap between largedeviation theories for sub-exponential distributions and multi-agent MAB with heterogeneous rewards, and the gap between random graphs and decentralized multi-agent MAB.

To this end, we propose a brand new algorithmic framework consisting of three main components: graph generation, DrFed-UCB: burn-in period, and DrFed-UCB: learning period. For the learning period, we modify the algorithm by [Zhu et al., 2021b] by introducing new UCB quantities that are consistent with the conventional UCB algorithm and generalize to sub-exponential settings. We also introduce a newly proposed stopping time and a new weight matrix without the doubly stochasticity assumption to leverage more information in random graphs. A burn-in period is crucial in estimating the graph distribution and initializing the weight matrix. We embed and analyze techniques from random graphs since the number of connected graphs is exponentially large in the number of vertices, and directly sampling such a graph is an NP-hard problem. In particular, we use the Metropolis-Hastings method with rapidly mixing Markov chains, as proposed in [Gray et al., 2019], to approximately generate random connected graphs in polynomial time. We additionally demonstrate its theoretical convergence rate, making it feasible to consider random connected graphs in the era of large-scale inference.

We present comprehensive analyses of the regret of the proposed algorithm, using the same regret definition as in existing literature. Firstly, we show that algorithm DrFed-UCB achieves optimal instance-dependent regret upper bounds of order log T with high probability, in both sub-Gaussian and sub-exponential settings, consistent with prior works. We add that although both [Zhu et al., 2020] and our analyses use the UCB framework, the important algorithmic steps are different and thus also the analyses. Secondly, we demonstrate that with high probability, the regret is universally upper bounded by O ( √ T log T ) in sub-exponential settings, including sub-Gaussian settings. This upper bound matches the upper and lower bounds in single-agent settings up to a log T factor, establishing its tightness.

The paper is organized as follows. We first introduce the notations used throughout the paper, present the problem formulation, and propose algorithms for solving the problem. Following that, we provide theoretical results on the regret of the proposed algorithm in various settings.

2 Problem Formulation and Methodologies

2.1 Problem Formulation

Throughout, we consider a decentralized system with M clients that are labeled as nodes 1 , 2 , . . . , M on a time-varying network, which is described by an undirected graph G t for 1 ≤ t ≤ T where parameter T denotes the time horizon of the problem. Formally, at time step t , G t = ( V, E t ) where V = { 1 , 2 , . . . , M } and E t denotes the edge set consisting of pair-wise nodes and representing the

̸

neighborhood information in G t . The neighbor set N m ( t ) include all neighbors of client m based on G t . Equivalently, the graph G t can be represented by the adjacency matrix ( X t i,j ) 1 ≤ i,j ≤ M where the element X t i,j = 1 if there is an edge between clients i and j and X t i,j = 0 otherwise. We let X i,i = 1 for any 1 ≤ i ≤ M . With this notation at hand, we define the empirical graph (adjacency matrix) P t as P t = ( ∑ t s =1 X s i,j ) 1 ≤ i,j ≤ M t . It is worth emphasizing that 1) the matrix P t is not necessarily doubly stochastic, 2) the matrix captures more information about G t than the prior works based on |N m ( t ) | , and 3) each client m only knows the m -th row of P t without knowledge of G t , i.e. { P t ( m,j ) } j are known to client m , while { P t ( k, j ) } j for k = m are always unknown. Let us denote the set of all possible connected graphs on M nodes as G M .

̸

We next consider the bandit problems faced by the clients. In the MAB setting, the environment generates rewards. Likewise, we again use the term, the environment, to represent the source of graphs G t and rewards r i m ( t ) in the decentralized multi-agent MAB setting. Formally, there are K arms faced by each client. At each time step t , for each client 1 ≤ m ≤ M , let the reward of arm 1 ≤ i ≤ K be r i m ( t ) , which is i.i.d. distributed across time with the mean value µ i m , and is drawn independently across the clients. Here we consider a heterogeneous setting where µ i m is not necessarily the same as µ j i for m = j . At each time step t , client m pulls an arm a t m , only observes the reward of that arm r m a t m ( t ) from the environment, and exchanges information with neighbors in G t given by the environment. In other words, two clients communicate only when there is an edge between them.

̸

By taking the average over clients as in the existing literature, we define the global reward of arm i at each time step t as r i ( t ) = 1 M ∑ M m =1 r i m ( t ) and the subsequent expected value of the global reward as µ i = 1 M ∑ M m =1 µ i m . We define the global optimal arm as i ∗ = arg max i µ i and arm i = i ∗ is called global sub-optimal. Let ∆ i = µ i ∗ -µ i be the sub-optimality gap. This enables us to quantify the regret of the action sequence (policy) { a t m } 1 ≤ t ≤ T 1 ≤ m ≤ M as follows. Ideally, clients would like to pull arm i ∗ if knowledge of { µ i } i were available. Given the partially observed rewards due to bandits (dimension i ) and limited accesses to information from other clients (dimension m ), the regret is defined as R T = Tµ i ∗ -1 M ∑ T t =1 ∑ M m =1 µ m a t m which measures the difference of the cumulative expected reward between the global optimal arm and the action sequence. The main objective of this paper is to develop theoretically robust solutions to minimize R T for clients operating on time-varying random graphs that are vulnerable to random communication failures, which only require knowledge of M .

2.2 Algorithms

In this section, we introduce a new algorithmic framework that incorporates two graph generation algorithms, one for the E-R model and the other for uniformly distributed connected graphs. More importantly, the framework includes a UCB-variant algorithm that runs a learning period after a burn-in period, which is commonly referred to as a warm-up phase in statistical procedures.

2.2.1 Graph Generation

We investigate two types of graph dynamics as follows, for which we propose simulation methods that enable us to generate and analyze the resulting random graphs.

E-R random graphs At each time step t , the adjacency matrix of graph G t is generated by the environment by element-wise sampling X t i,j according to a Bernoulli distribution. Specifically, X t i,j follows a Bernoulli distribution with parameter c .

̸

Uniformly distributed connected graphs At each time step t , the random graph G t is generated by the environment by uniformly sampling a graph from the sample space of all connected graphs G M , which yields the adjacency matrix ( X t i,j ) 1 ≤ i = j ≤ M corresponding to G t . Generating uniformly distributed connected graphs is presented in Algorithm 3 as in Appendix. It is computationally infeasible to exhaust the sample space G M since the number of connected graphs is exponentially large. To this end, we import the Metropolis-Hastings method in [Gray et al., 2019] and leverage rapidly mixing Markov chains. Remarkably, by adapting the algorithm into our setting which yields a new algorithm, we construct a Markov chain that converges to the target distribution after a finite number of burn-in steps. This essentially traverses graphs in G M through step-wise transitions, with a complexity of O ( M 2 ) from previous states. More precisely, at time step s with connected graph G s , we randomly choose a pair of nodes and check whether it exists in the edge set. If this is the case, we

remove the edge from G s and check whether the remaining graph is connected, and only accept the graph as G s +1 in the connected case. If the edge does not exist in the edge set, we add it to G s and get G s +1 . In this setting, let c = c ( M ) be the number of times an edge is present among all connected graphs divided by the total number of connected graphs. It is known that c = 2 log M M -1 (1) [Trevisan]. The distribution of G s eventually converges to the uniform distribution in G M . The formal statements are in Appendix.

2.2.2 Main Algorithm

In the following, we present the proposed algorithm, DrFed-UCB, which comprises of a burn-in period and a learning period described in Algorithm 1 and Algorithm 2, respectively.

We start by introducing the variables used in the algorithm with respect to client m . We use ¯ µ i m ( t ) , n m,i ( t ) to denote reward estimators and counters based on client m 's own pulls of arm i , respectively, and use ˜ µ i m , N m,i ( t ) to denote reward estimators and counters based on the networkwide pulls of arm i , respectively. By network-wide, we refer to the clients in N m ( t ) . Denote the stopping time for the filtration { G s } t s =1 as h t m,j = max s ≤ t { ( m,j ) ∈ E s } ; it represents the most recent communication between clients m and j . The weights for the network-wide and local estimators are P ′ t ( m,j ) and d m,t defined later, respectively.

There are two stages in Algorithm 1 where t ≤ L as follows. For the first τ 1 steps, the environment generates graphs based on one of the aforementioned graph generation algorithms to arrive at the steady state, while client m pulls arms randomly and updates local estimators { ¯ µ i m , n i m } i . Afterwards, the environment generates the graph G t that follows the distribution of interest, while client m updates { ¯ µ i m , n i m } i and row m of P t by exchanging information with its neighbors in the graph G t . Note that client m does not have any global information about G t . At the end of the burn-in period, client m computes the network-wide estimator ˜ µ i m by taking the weighted average of local estimators of other clients (including itself), where the weights are given by the m -th row of weight matrix P ′ and d , which depend on P and knowledge of M and satisfy ∑ M j =1 P ′ t ( m,j ) + d m,t M = 1 .

Subsequently, we describe Algorithm 2 where t ≥ L +1 . There are four phases in one iteration enumerated below in the order indicated. A flowchart of this algorithm is provided in Appendix.

UCB Given the estimators ˜ µ i m ( t ) , n m,i ( t ) , N m,i ( t ) , ¯ µ i m ( t ) , client m either randomly samples an arm or pulls the arm that maximizes the upper confidence bound using ˜ µ i m ( t ) , n m,i ( t ) , depending on whether n m,i ( t ) ≤ N m,i ( t ) -K holds for some arm i . This additional condition ensures consensus among clients regarding which arm to pull. The upper confidence bound F ( m,i, t ) is specified as F ( m,i, t ) = √ C 1 ln t n m,i ( t ) and F ( m,i, t ) = √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) in settings with sub-Gaussian and sub-exponential rewards, respectively. Constants C 1 and C 2 are determined in the analyses and they depend on σ which is an upper bound of standard deviations of the reward values (it is formally defined later).

Environment and client interaction After client m pulls arm a t m , the environment sends the reward r t m,a t m and the neighbor set N m ( t ) in G t to client m . Client m does not know the whole G t and obtains only the neighbor set N m ( t ) .

Transmission Client m sends the maintained local and network-wide estimators { ¯ µ i m ( t ) , ˜ µ i m ( t ) , n m,i ( t ) , N m,i ( t ) } i to all clients in N m ( t ) and receives the ones from them.

Update estimators At the end of an iteration, client m first updates the m -th row of matrix P t . Subsequently, client m updates the quantities { ¯ µ i m ( t ) , ˜ µ i m ( t ) , n m,i ( t ) , N m,i ( t ) } 1 ≤ i ≤ K adhereing to: t m,j = max s ≥ τ 1 { ( m,j ) ∈ E s ) } and 0 if such an s does not exist

Similar to [Zhu et al., 2021b, Zhu and Liu, 2023], the algorithm balances between exploration and exploitation by the upper confidence bound and a criterion on n m,i ( t ) and N m,i ( t ) that ensures that all clients explore arms at similar rates and thereby 'staying on the same page.' After interacting

with the environment, clients move to the transmission stage, where they share information with the neighbors on G t , as a preparation for the update stage.

Different from the upper confidence bound approach in [Zhu and Liu, 2023], which has an extra term of 1 t in the UCB criterion, our proposal is aligned with the conventional UCB algorithm. Meanwhile, our update rule differs from that in [Zhu et al., 2021b, Zhu and Liu, 2023] in three key aspects: (1) maintaining a stopping time t m,j that tracks the most recent communication to client j , and (2) updating ˜ µ i m based on both ˜ µ j i and ¯ µ j i for j ∈ N m ( t ) , and (3) using a weight matrix based on P ′ t and P t computed from the trajectory { G s } s ≤ t in the previous steps. The first point ensures that the latest information from other clients is leveraged, in case there is no communication at the current time step. The second point ensures proper integration of both network-wide and local information, smoothing out biases from local estimators and reducing variances through averaging. The third point distills the information carried by the time-varying graphs and determines the weights of the available local and network-wide estimators, removing the need for the doubly stochasticity assumption. The algorithm assumes that the clients know M and σ 2 .

We note that t m,j is the stopping time by definition and that ¯ µ i m is an unbiased estimator for µ i m with a decaying variance proxy. Meanwhile, the matrices P ′ t and P t are not doubly stochastic and keep track of the history of the random graphs. By introducing t m,j and P t , we can show that the global estimator ˜ µ i m ( t ) behaves similarly to a sub-Gaussian/sub-exponential random variable with an expectation of µ i and a time-decaying variance proxy proportional to 1 min j n j,i ( t ) . This ensures that the concentration inequality holds for ˜ µ i m ( t ) with respect to µ i and that client m tends to identify the

Algorithm 2: DrFed-UCB: Learning period

global optimal arms with high probability, which plays an important role in minimizing regret. The formal statements are presented in the next section.

3 Regret Analyses

In this section, we show the theoretical guarantees of the proposed algorithm, assuming mild conditions on the environment. Specifically, we consider various settings with different model assumptions. We prove that the regret of Algorithm 2 has different instance-dependent upper bounds of order log T for settings with sub-Gaussian and sub-exponential distributed rewards, and a meangap independent upper bound of order √ T log T across settings. Many researchers call such a bound instance independent but we believe such a terminology is misleading and thus we prefer to call it man-gap independent, given that it still has dependency on parameters pertaining to the problem instance. The results are consistent with the regret bounds in prior works.

3.1 Model Assumptions

By definition, the environment is determined by how the graphs (E-R or uniform) and rewards are generated. For reward we consider two cases.

Sub-g At time step t , the reward of arm i at client m has bounded support [0 , 1] , and is drawn from a sub-Gaussian distribution with mean 0 ≤ µ i m ≤ 1 and variance proxy 0 ≤ ( σ i m ) 2 ≤ σ 2 .

Sub-e At time step t , the reward of arm i at client m has bounded support [0 , 1] , and follows a sub-exponential distribution with mean 0 ≤ µ i m ≤ 1 and parameters 0 ≤ ( σ i m ) 2 ≤ σ 2 , 0 ≤ α i m ≤ α .

With these considerations, we investigate four different environments (settings) based on the two graph assumptions and the two reward assumptions: Setting 1.1 corresponds to E-R and Sub-g, Setting 1.2 to Uniform and Sub-g, Setting 2.1 to E-R and Sub-e, and Setting 2.2 to Uniform and Sub-e. For each setting, we derive upper bounds on the regret in the next section.

3.2 Regret Analyses

In this section, we establish the regret bounds formally when clients adhere to Algorithm 2 in various settings. We denote Setting 1.1, Setting 1.2 with M < 11 , and Setting 1.2 with M ≥ 11 as s 1 , s 2 and s 3 , respectively. Likewise, we denote Setting 2.1, Setting 2.2 with M < 11 , and Setting 2.2 with M ≥ 11 as S 1 , S 2 and S 3 , respectively. See Table 1 in Appendix for a tabular view of the various settings.

̸

Note that the randomness of R T arises from both the reward and graph observations. Considering S 1 , S 2 , S 3 differ in the reward assumptions compared to s 1 , s 2 , s 3 , we define an event A that preserves the properties of the variables with respect to the random graphs. Given the length of the burnin period L i for i ∈ { s 1 , s 2 , s 3 } and the fact that L s i = L S i since it only relies on the graph assumptions, we use L to denote max i L s i . Parameters 0 < δ, ϵ < 1 are any constants, and the parameter c = c ( M ) represents the mean value of the Bernoulli distribution in s 1 , S 1 and the probability of an edge in s 2 , S 2 , s 3 , and S 3 among all connected graphs (see (1)). We define events A 1 = {∀ t ≥ L, || P t -cE || ∞ ≤ δ } , A 2 = {∃ t 0 , ∀ t ≥ L, ∀ j, ∀ m,t + 1 -min j t m,j ≤ t 0 ≤ c 0 min l n l,i ( t + 1) } , and A 3 = {∀ t ≥ L, G t is connected } . Here E is the matrix with all values of 1 . Constant c 0 = c 0 ( K, min i = i ∗ ∆ i , M, ϵ, δ ) is defined later. Since c = c ( M ) this implies that G t depends on M . We define A = A ϵ,δ = A 1 ∩ A 2 ∩ A 3 , which yields A ∈ Σ with Σ being the subσ -algebra formed by { Ω , ∅ , A, A c } . This implies E [ ·| A ϵ,δ ] and P [ ·| A ϵ,δ ] are well-defined, since A only relies on the graphs and removes the differences among s 1 , s 2 , s 3 ( S 1 , S 2 , S 3 ), enabling universal regret upper bounds.

Next, we demonstrate that event A holds with high probability.

Proof Sketch. The complete proof is deferred to Appendix; we discuss the main logic here. The proof relies on bounding the probabilities of A 1 , A 2 , A 3 separately. For A 1 , its upper bound holds by the analysis of the mixing time of the Markov chain underlying G t and on the matrix-form Hoeffding inequality. We obtain an upper bound on P ( A 2 ) by analyzing the stopping time t m,j and the counter n m,i ( t ) . For the last term P ( A 3 ) , we show that the minimum degree of G t has a high probability lower bound that is sufficient for claiming the connectivity of G t . To put all together, we use the Bonferroni's inequality and reach the lower bound of P ( A ϵ,δ ) .

Subsequently, we have the following general upper bound on the regret R T of Algorithm 2 in the high probability sense, which holds on A in any of the settings s 1 , s 2 , s 3 with sub-Gaussian rewards.

̸

Theorem 2. Let f be a function specific to a setting and detailed later. For every 0 < ϵ < 1 and 0 < δ < f ( ϵ, M, T ) , in setting s 1 with c ≥ 1 2 + 1 2 √ 1 -( ϵ MT ) 2 M -1 , s 2 and s 3 , with the time horizon T satisfying T ≥ L , the regret of Algorithm 2 with F ( m,i, t ) = √ C 1 ln t n m,i ( t ) satisfies that E [ R T | A ϵ,δ ] ≤ L + ∑ i = i ∗ (max { [ 4 C 1 log T ∆ 2 i ] , 2( K 2 + MK ) } + 2 π 2 3 P ( A ϵ,δ ) + K 2 + (2 M -1) K ) = O (ln T ) where the length of the burn-in period is explicitly L =

MT

with λ being the spectral gap of the Markov chain in s 2 , s 3 that satisfies 1 -λ ≥ 1 2 ln 2 ln 2 p ∗ ln 4+1 ,

p

∗

C

1

=

p

= 8

∗

σ

M

)

<

{

1

12

and

M

(

M

c

0

+2)

M

4

}

=

.

Proof Sketch. The proof is carried out in Appendix; here we describe the main ideas as follows. We note that the regret is proportional to the total number of pulling global sub-optimal arms by the end of round T . We fix client m for illustration without loss of generality. We tackle all the possible cases when clients pull such a sub-optimal arm - (i) the condition n m,i ( t ) ≤ N m,i ( t ) -K is met, (ii) the upper confidence bounds of global sub-optimal arms deviate from the true means, (iii) the upper confidence bounds of global optimal arms deviate from the true means, and (iv) the mean values of global sub-optimal arms are greater than the mean values of global optimal arms. The technical novelty of our proof is in that 1) we deduce that the total number of events (ii) and (iii) occurring can

(

2

c

0

(

K, min

i

̸

=

i

∗

∆

i

, M, ϵ, δ

)

, and the instance-dependent constant

be bounded by some constants using newly derived conditional concentration inequalities that hold by our upper bounds on the conditional moment generating functions and by the unbiasedness of the network-wide estimators and 2) we control (i) by analyzing the scenarios where the criteria are met, which do not occur frequently.

Remark ( Specification of the parameters ) . Note that the choice of f depends on the problem settings. Specifically, in setting s 1 , we set f ( ϵ, M, T ) = 1 2 + 1 4 √ 1 -( ϵ MT ) 2 M -1 . By the definition of c , we have f ( ϵ, M, T ) < c . In setting s 2 with M < 11 , we specify f ( ϵ, M, T ) = 1 2 which meets f < c due to (1). Lastly, in setting s 3 with M ≥ 11 , we choose f ( ϵ, M, T ) = 1 2 2 log M M -1 and again we have f < c due to (1). We observe that the regret bound is dependent on the transition kernel π and the spectral gap λ of the underlying Markov chain associated with π . This indicates the significance of graph complexities and distributions within the framework of the random graph model when deriving the regret bounds, in a similar manner as the role of graph conductance in the regret bounds established in [Sankararaman et al., 2019, Chawla et al., 2020] for time-invariant graphs.

To proceed, we show a high probability upper bound on the regret E [ R T | A ϵ,δ ] of Algorithm 2 for settings S 1 , S 2 , S 3 with sub-exponential rewards.

̸

Theorem 3. Let f be a function specific to a setting and defined in the above remark. For every 0 < ϵ < 1 and 0 < δ < f ( ϵ, M, T ) , in settings S 1 with c ≥ 1 2 + 1 2 √ 1 -( ϵ MT ) 2 M -1 , S 2 , S 3 with the time horizon T satisfying T ≥ L , the regret of Algorithm 2 with F ( m,i, t ) = √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) satisfies E [ R T | A ϵ,δ ] ≤ L + ∑ i = i ∗ (∆ i +1) · (max([ 16 C 1 log T ∆ 2 i ] , [ 4 C 2 log T ∆ i ] , 2( K 2 + MK )) + 4 P ( A ϵ,δ ) T 3 + K 2 +(2 M -1) K ) = O (ln T ) where L, C 1 are specified as in Theorem 2 and C 2 C 1 ≥ 3 2 .

Proof Sketch. The proof is detailed in Appendix. The proof logic is similar to that of Theorem 2. However, the main differences lie in the upper confidence bounds, which require proving new concentration inequalities and moment generating functions for the network-wide estimators.

In addition to the instance-dependent regret bounds of order O ( log T ∆ i ) that depend on the suboptimality gap ∆ i which may be arbitrarily small and thereby leading to large regret, we also establish a universal, mean-gap independent regret bound that applies to settings with sub-exponential and sub-Gaussian rewards. A formal proof is deferred to Appendix.

Theorem 4. Assume the same conditions as in Theorems 2 and 3. The regret of Algorithm 2 satisfies that E [ R T | A ϵ,δ ] ≤ L 1 + 4 P ( A ϵ,δ ) T 3 +( √ max( C 1 , C 2 ) ln T +1) 4 M P ( A ϵ,δ ) T 3 + K ( C 2 (ln T ) 2 + C 2 ln T + √ C 1 ln T √ T (ln T +1)) = O ( √ T ln T ) . where L 1 = max( L, K (2( K 2 + MK ))) , L, C 1 is specified as in Theorem 2, and C 2 C 1 ≥ 3 2 . The involved constants depend on σ 2 but not on ∆ i .

3.3 Other Performance Measures

Communication cost Assuming a constant cost of establishing a communication link, as defined in [Wang et al., 2020, Li and Song, 2022], denoted as c 1 , the communication cost C T can be calculated as C T = c 1 · ∑ T t =1 | E t | , which is proportional to ∑ T t =1 | E t | . Alternatively, following the framework proposed in [Wang et al., 2022, Li and Wang, 2022, Sankararaman et al., 2019, Chawla et al., 2020], the communication cost can be defined as the total number of communications among clients, which can be represented as C T = ∑ T t =1 | E t | , similar to the previous definition. Regarding the quantity C T = ∑ T t =1 | E t | , the number of edges E t could be O ( M ) for sparse graphs, and at most O ( M 2 ) . In the random graph model, the expected number of edges is M ( M -1) 2 c , which implies O ( M 2 ) in the worst case scenario and the total communication cost, in a worst-case scenario, is of order O ( TM 2 ) . This analysis holds also for the random connected graph case where c represents the probability of having an edge. This cost aligns with the existing work of research on decentralized distributed multi-agent MAB problems without a focus on the communication cost, where edge-wise communication is a standard practice. Optimizing communication costs to achieve sublinearity, as discussed in [Sankararaman et al., 2019, Chawla et al., 2020], is a subject for future research.

Complexity and privacy guarantee At each time step, the time complexity of the graph generation algorithm is O ( M 2 + M + | E t | ) ≤ O ( M 2 ) , where the first term accounts for edge selection, and the second and third terms are for graph connectivity verification using Algorithm 1 (benefitting from the use of Markov chains for graph generation). The main algorithm comprises various stages involving

multiple agents. The overall time complexity is calculated as O ( MK + M 2 + MK + M 2 ) = O ( M 2 + MK ) , consistent with that of [Zhu et al., 2021b, Dubey et al., 2020]. It is noteworthy that most procedures can operate in parallel (synchronously), such as arm pulling, broadcasting, estimator updating, and E-R model generation, with the exception being random connected graph generation due to the Markovian property. Privacy guarantees are also important. Here, we note that clients only communicate aggregated values of raw rewards. Differential privacy is not the focus of this work but may be considered in future research.

4 Numerical Results

In this section, we present a numerical study of the proposed algorithm. Specifically, we first demonstrate the regret performance of Algorithms 2 and 3, in comparison with existing benchmark methods from the literature, in a setting with time-invariant graphs. Moreover, we conduct a numerical experiment with respect to time-varying graphs, comparing the proposed algorithm with the most recent work [Zhu and Liu, 2023]. Furthermore, the theoretical regret bounds of the proposed algorithm, as discussed in the previous section, exhibit different dependencies on the parameters that determine the underlying problem settings. Therefore, we examine these dependencies through simulations to gain insights into the exact regret incurred by the algorithm in practice. The benchmark algorithms include GoSInE [Chawla et al., 2020], Gossip_UCB [Zhu et al., 2021b], and Dist_UCB [Zhu and Liu, 2023]. Notably, GoSInE and Gossip_UCB are designed for time-invariant graphs, while Dist_UCB and the proposed algorithm (referred to as DrFed-UCB) are tailored for time-varying graphs. Details about numerical experiments are refered to Appendix.

Benchmark comparison results The visualizations for both time-invariant and time-varying graphs are in Appendix. We evaluated regret by averaging over 50 runs, along with the corresponding 95% confidence intervals. In time-invariant graphs, DrFed-UCB consistently demonstrates the lowest average regret, showcasing significant improvements. Dist_UCB and DrFed-UCB exhibit larger variances (Dist_UCB having the largest variances), which might have resulted from the communication mechanisms designed for time-varying graphs. In time-varying graphs, our regret is substantially lower compared to that of Dist_UCB. In terms of time complexity, DrFed-UCB and GoSInE are approximately six times faster than Dist_UCB.

Regret dependency results Additionally, we illustrate how the regret of DrFed-UCB depends on several factors, including the number of clients M , the number of arms K , the Bernoulli parameter c for the E-R model, and heterogeneity measured by max i,j,k | µ k i -µ k j | . The visualizations are available in Appendix. We observe that regret monotonically increases with the level of heterogeneity and the number of arms, while it monotonically decreases with connectivity, which is equivalent to an increase in graph complexity. However, this monotonic dependency does not hold with respect to M due to the accumulation of the information gain.

5 Conclusions

In this paper, we consider a decentralized multi-agent multi-armed bandit problem in a fully stochastic environment that generates time-varying random graphs and heterogeneous rewards following subgaussian and sub-exponential distributions, which has not yet been studied in existing works. To the best of our knowledge, this is the first theoretical work on random graphs including the E-R model and random connected graphs, and the first work on heterogeneous rewards with heavy tailed rewards. To tackle this problem, we develop a series of new algorithms, which first simulate graphs of interest, then run a warm-up phase to handle graph dynamics and initialization, and proceed to the learning period using a combination of upper confidence bounds (UCB) with a consensus mechanism that relies on newly proposed weight matrices and updates, and using a stopping time to handle randomly delayed feedback. Our technical novelties in the results and the analyses are as follows. We prove high probability instance-dependent regret bounds of the order of log T in both sub-gaussian and sub-exponential cases, consistent with the regret bound in the existing works that only consider the expected regret. Moreover, we establish a nearly tight instance-free regret bound of order √ T log T for both sub-exponential and sub-gaussian distributions, up to a log T factor. We leverage probabilistic graphical methods on random graphs and draw on theories related to rapidly mixing Markov chains, which allows us to eliminate the doubly stochasticity assumption through new weight matrices and a stopping time. We construct new network-wide estimators and invent new concentration inequalities for them, and subsequently incorporate the seminal UCB algorithm into this distributed setting. A discussion on future work is refered to Appendix.

References

Agarwal, V. Aggarwal, and K. Azizzadenesheli. Multi-agent multi-armed bandits with limited communication. The Journal of Machine Learning Research , 23(1):9529-9552, 2022.

Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2-3):235-256, 2002a.

Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002b.

Bistritz and A. Leshem. Distributed multi-player bandits-a game of thrones approach. Advances in Neural Information Processing Systems , 31, 2018.

Chawla, A. Sankararaman, A. Ganesh, and S. Shakkottai. The gossiping insert-eliminate algorithm for multi-agent bandits. In International conference on artificial intelligence and statistics , pages 3471-3481. PMLR, 2020.

Chib and E. Greenberg. Understanding the metropolis-hastings algorithm. The american statistician , 49(4):327-335, 1995.

Delarue. Mean field games: A toy model on an erdös-renyi graph. ESAIM: Proceedings and Surveys , 60:1-26, 2017.

Dubey et al. Cooperative multi-agent bandits with heavy tails. In International conference on machine learning , pages 2730-2739. PMLR, 2020.

C. Duchi, A. Agarwal, and M. J. Wainwright. Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Transactions on Automatic control , 57(3): 592-606, 2011.

Fan, B. Jiang, and Q. Sun. Hoeffding's inequality for general markov chains and its applications to statistical learning. The Journal of Machine Learning Research , 22(1):6185-6219, 2021.

Gray, L. Mitchell, and M. Roughan. Generating connected random graphs. Journal of Complex Networks , 7(6):896-912, 2019.

Huang, W. Wu, J. Yang, and C. Shen. Federated linear contextual bandits. Advances in neural information processing systems , 34:27057-27068, 2021.

Jia, C. Shi, and S. Shen. Multi-armed bandit with sub-exponential rewards. Operations Research Letters , 49(5):728-733, 2021.

Jiang and H. Cheng. Multi-agent bandit with agent-dependent expected rewards. Swarm Intelligence , pages 1-33, 2023.

Landgren, V. Srivastava, and N. E. Leonard. On distributed cooperative decision-making in multiarmed bandits. In 2016 European Control Conference (ECC) , pages 243-248. IEEE, 2016a.

Landgren, V. Srivastava, and N. E. Leonard. Distributed cooperative decision-making in multiarmed bandits: Frequentist and bayesian algorithms. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 167-172. IEEE, 2016b.

Landgren, V. Srivastava, and N. E. Leonard. Distributed cooperative decision making in multi-agent multi-armed bandits. Automatica , 125:109445, 2021.

Li and H. Wang. Communication efficient federated learning for generalized linear bandits. Advances in Neural Information Processing Systems , 35:38411-38423, 2022.

Li and L. Song. Privacy-preserving communication-efficient federated multi-armed bandits. IEEE Journal on Selected Areas in Communications , 40(3):773-787, 2022.

W. Lima, A. O. Sousa, and M. Sumuor. Majority-vote on directed erd˝ os-rényi random graphs. Physica A: Statistical Mechanics and its Applications , 387(14):3503-3510, 2008.

Martínez-Rubio, V. Kanade, and P. Rebeschini. Decentralized cooperative stochastic bandits. Advances in Neural Information Processing Systems , 32, 2019.

McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

McNew. The eigenvalue gap and mixing time. 2011.

Mitra, H. Hassani, and G. Pappas. Exploiting heterogeneity in robust federated best-arm identification. arXiv preprint arXiv:2109.05700 , 2021.

Nedic and A. Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control , 54(1):48-61, 2009.

Perchet, P. Rigollet, S. Chassang, and E. Snowberg. Batched bandit problems. The Annals of Statistics , pages 660-681, 2016.

Réda, S. Vakili, and E. Kaufmann. Near-optimal collaborative learning in bandits. In NeurIPS 2022-36th Conference on Neural Information Processing System , 2022.

Sankararaman, A. Ganesh, and S. Shakkottai. Social learning in multi agent multi armed bandits. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 3(3):1-35, 2019.

Scaman, F. Bach, S. Bubeck, Y. T. Lee, and L. Massoulié. Optimal algorithms for smooth and strongly convex distributed optimization in networks. In international conference on machine learning , pages 3027-3036. PMLR, 2017.

Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

Tao, Y. Wu, P. Zhao, and D. Wang. Optimal rates of (locally) differentially private heavy-tailed multi-armed bandits. In International Conference on Artificial Intelligence and Statistics , pages 1546-1574. PMLR, 2022.

Trevisan. Cs 174 randomized algorithms. URL http://theory.stanford.edu/~trevisan/ cs174/ .

P.-A. Wang, A. Proutiere, K. Ariu, Y. Jedra, and A. Russo. Optimal algorithms for multiplayer multi-armed bandits. In International Conference on Artificial Intelligence and Statistics , pages 4120-4129. PMLR, 2020.

Wang, L. Yang, Y.-Z. J. Chen, X. Liu, M. Hajiesmaili, D. Towsley, and J. C. Lui. Achieving near-optimal individual regret & low communications in multi-agent bandits. In The Eleventh International Conference on Learning Representations , 2022.

Wang, C. Zhang, M. K. Singh, L. Riek, and K. Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In International Conference on Artificial Intelligence and Statistics , pages 1531-1539. PMLR, 2021.

Yan, Q. Xiao, T. Chen, and A. Tajer. Federated multi-armed bandit via uncoordinated exploration. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5248-5252. IEEE, 2022.

Zhu and J. Liu. Distributed multi-armed bandits. IEEE Transactions on Automatic Control , 2023.

Zhu, R. Sandhu, and J. Liu. A distributed algorithm for sequential decision making in multi-armed bandit with homogeneous rewards. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 3078-3083. IEEE, 2020.

Zhu, E. Mulle, C. S. Smith, and J. Liu. Decentralized multi-armed bandit can outperform classic upper confidence bound. arXiv preprint arXiv:2111.10933 , 2021a.

Zhu, J. Zhu, J. Liu, and Y . Liu. Federated bandit: A gossiping approach. In Abstract Proceedings of the 2021 ACM SIGMETRICS/International Conference on Measurement and Modeling of Computer Systems , pages 3-4, 2021b.

A Additional Background Work

Developing efficient algorithms for decentralized systems has been a popular research area in recent years. Among them, gossiping algorithms have been proven to be successful [Scaman et al., 2017, Duchi et al., 2011, Nedic and Ozdaglar, 2009]. In this approach, each client computes iteratively a weighted average of local estimators and network-wide estimators obtained from neighbors. The goal is to derive an estimator that converges to the average of the true values across the entire system. The weights are represented by a matrix that respects the graph structure under certain conditions. The gossiping-based averaging approach enables the incorporation of MAB methods in decentralized settings. In particular, motivated by the success of the UCB algorithm [Auer et al., 2002a] in stochastic MAB, [Landgren et al., 2016a,b, 2021, Zhu et al., 2020, 2021a,b, Martínez-Rubio et al., 2019, Chawla et al., 2020, Wang et al., 2021] import it to various decentralized settings under the assumption of sub-Gaussianity, including homogeneous or heterogeneous rewards, different graph assumptions, and various levels of global information. The regret bounds obtained are typically of order log T . However, most existing works assume that the graph is time-invariant under further conditions, which is often not the case. For example, [Wang et al., 2021] provide a optimal regret guarantee for complete graphs which are essentially a centralized batched bandit problem [Perchet et al., 2016]. Connected graphs are also considered, but [Zhu et al., 2020] assume that the rewards are homogeneous and graphs are time-invariant related to doubly stochastic matrices. In addition, [Martínez-Rubio et al., 2019] propose the DDUCB algorithm for settings with time-invariant graphs and homogeneous rewards, dealing with deterministically delayed feedback and assuming knowing the number of vertices and the spectral gap of the given graph. Meanwhile, [Jiang and Cheng, 2023] propose an algorithm C-CBGE that is robust to Gaussian noises and deals with client-dependent MAB, but requires time-invariant regular graphs. [Zhu et al., 2021b] propose a gossiping-based UCB-variant algorithm for time-invariant graphs. In this approach, each client maintains a weighted averaged estimator by gossiping, uses doubly stochastic weight matrices depending on global information of the graph, and adopts a UCB-based decision rule by constructing upper confidence bounds. Recently, [Zhu and Liu, 2023], revisit the algorithm and add an additional term to the UCB rule for time-varying repeatedly strongly connected graphs, assuming no global information. However, the doubly stochasticity assumption excludes many graphs from consideration. Our algorithm builds on the approach proposed by [Zhu et al., 2021b] with new weight matrices that do not require the doubly stochasticity assumption. Our weight matrices leverage more local graph information, rather than just the size of the vertex set as in [Zhu and Liu, 2023, Zhu et al., 2021b]. We introduce the terminology of the stopping time for randomly delayed feedback, along with new upper confidence bounds that consider random graphs and sub-exponentiality. This leads to smaller high probability regret bounds, and the algorithm only requires knowledge of the number of vertices that can be obtained at initialization or estimated as in [Martínez-Rubio et al., 2019].

In the context of bandits with heavy-tailed distributed rewards, the UCB algorithm continues to play a significant role. [Dubey et al., 2020] are the first to consider the multi-agent MAB setting with homogeneous heavy-tailed rewards. They develop a UCB-based algorithm with an instancedependent regret bound of order log T . They achieve this by adopting larger upper confidence bounds and finding cliques of vertices, even though the graphs are time-invariant and known to clients. In a separate line of work, [Jia et al., 2021] consider the single-agent MAB setting with sub-exponential rewards, and propose a UCB-based algorithm that enlarges or pretrains the upper confidence bounds, achieving a mean-gap independent regret bound of order √ T log T . We extend this technique to the decentralized multi-agent MAB setting with heterogeneous sub-exponential rewards, using a gossiping approach, and establish both an optimal instance-dependent regret bound of O (log T ) and a nearly optimal mean-gap independent regret bound of O ( √ T log T ) , up to a log T factor.

Our work draws on the classical literature on random graphs. From the perspective of generating random connected graphs, we build upon a numerically efficient algorithm introduced in [Gray et al., 2019], which is based on the Metropolis-Hastings algorithm [Chib and Greenberg, 1995], despite its lack of finite convergence rate for non-sparse graphs. We follow their algorithm and, in addition, provide a new analysis on the convergence rate and mixing time of the underlying Markov chain. In terms of the E-R model, it has been thoroughly examined in various areas, such as mean-field games [Delarue, 2017] and majority vote settings [Lima et al., 2008]. However, these random graphs have not yet been applied to the decentralized multi-agent MAB setting that is partially determined by the underlying graphs. Our formulation and analyses bridge this gap, providing insights into the dynamics of decentralized multi-agent MAB in the context of random graphs.

B Future work

Recent advancements have been made in reducing communication costs with respect to the dependency in multi-agent MAB with homogeneous rewards (in the generalized linear bandit setting [Li and Wang, 2022], the ground truth of the unknown parameter is the same for all clients), such as achieving O ( √ TM 2 ) in [Li and Wang, 2022] for centralized settings or O ( M 3 log T ) through Global Information Synchronization (GIS) communication protocols assuming time-invariant graphs in [Li and Song, 2022]. Likewise, [Sankararaman et al., 2019, Chawla et al., 2020] improve the communication cost of order log T or o ( T ) through asynchronous communication protocols and balancing the trade-off between regret and communication cost. More recently, [Wang et al., 2022] establish a novel communication protocol, TCOM, which is of order log log T by means of concentrating communication around sub-optimal arms and performing aggregation of estimators across time steps. Furthermore, [Wang et al., 2020] develops a new leader-follower communication protocol, which selects a leader that communicates to the followers. Here the communication cost is independent of T which is much smaller. The incorporation of random graph structures and heterogeneous rewards introduces its own complexities, which poses challenges to reductions in communication costs. These great advancements introduce a promising direction for communication efficiency as a next step within the context herein.

C Details on numerical experiments in Section 4

We report the experimental details in Section 4, including both benchmarking and regret properties of the algorithms. The implementation details of the experiments are as follows, including the data generation, benchmarks, and the evaluation metrics.

The process of data generation involves both reward generation and graph generation. First we generate different numbers of arms and clients, denoted as K and M , respectively. Specifically, we generate rewards using the Bernoulli distribution in the sub-Gaussian distribution family, varying the mean values µ i m by introducing multiple levels of heterogeneity denoted as h = max i,j,m | µ i m -µ j | and then for each arm k , partitioning the range [0 . 1 , 0 . 1 + ( k + 1) · h/K ] into M intervals. In terms of graph generation, we generate E-R models with varying values of c , to capture graph complexity. Specifically, for the benchmarking experiment with time-invariant graphs, we set K = 2 , M = 5 , h = 0 . 1 , c = 1 , i.e. complete graphs. For the benchmarking experiment with time-varying graphs, we set K = 2 , M = 5 , h = 0 . 1 , c = 0 . 9 . For the regret experiments, the parameters are h ∈ { 0 . 1 , 0 . 2 , 0 . 3 } , M ∈ { 5 , 8 , 12 } , c ∈ { 0 . 2 , 0 . 5 , 0 . 9 , 1 } , and K ∈ { 2 , 3 , 4 } . We selected the least positive number of arms K = 2 to keep computational times low and M = 5 to have small graphs but still a variety of them.

We compare the new method DrFed-UCB with the classical methods, such as the Gossiping InsertEliminate algorithm (GoSInE) in [Chawla et al., 2020] which focuses on deterministic graphs and sub-Gaussian rewards and motivated our work. We also include the Gossip UCB algorithm (Gossip_UCB) [Zhu et al., 2021b] as a benchmark. Meanwhile, in terms of time-varying graphs, we implement the algorithm, Distributed UCB (Dist_UCB) in [Zhu and Liu, 2023] that has been developed for time-varying graphs, and compare our algorithm to this benchmark.

Evaluation The evaluation metric is the regret measure as defined in Section 4. More specifically, for the experiments, we use the average regret over 50 runs for each benchmark and also report the 95% confidence intervals across the 50 runs. With respect to the communication cost as another performance measure, it is computed explicitly. Additionally, the runtime can provide insights into the time complexity of the models.

Benchmark comparison results The results for time-invariant and time-varying graphs are presented in Figure 1 (a) and Figure 1 (b), respectively. The x-axis represents time steps, while the y-axis shows the average regret up to that time step. Figure 1 (a) demonstrates that DrFed-UCB exhibits the smallest average regret among all methods in time-invariant graphs, with significant improvements. More precisely, with respect to the Area Under the Curve (AUC) of the regret curve, the improvements of DrFed-UCB over GoSInE, Gossip_UCB, and Dist_UCB are 132%, 158%, and 128%, respectively, showcasing the regret improvement of the newly proposed algorithm compared to the benchmarks. Notably, both Dist_UCB and DrFed-UCB result in larger variances, primarily observed in Dist_UCB. This phenomenon may be attributed to the communication mechanisms designed for time-varying graphs. In Figure 1 (b), we observe that our regret is notably smaller compared to Dist_UCB in settings with time-varying graphs. Specificaly, the AUC of Dist_UCB is

50

GoSInE

Dist_UCB

Gossip_UCB

40

Cumulative Regret

DrFed-UCB

30

20

10

0

0

250

500

750

1000

1250

1500

1750

2000

Time Horizon

Dist_UCB

DrFed-UCB

20

Cumulative Regret

15

10

5

0

0

250

500

750

1000

1250

1500

1750

2000

Time Horizon

(a) time-invariant graphs

(b) time-varying graphs

(a)

h

(b)

M

6

h = 0.1

5

+...

h = 0.3

h = 0.2

CumulativeRegret

4

3

1

0

0

250

500

750

10001250150017502000

Time Horizon

C =0.2

8

C = 0.5

C = 0.9

CumulativeRegret

C=1

6

4

2

0

0

250

500

750

1000

1250

1500

1750

2000

Time Horizon

14

M = 5

M = 8

12

.....

M = 12

CumulativeRegret

10

8

6

4

2

0

0

250

500

750

10001250150017502000

Time Horizon

K =2

5

K = 3

K = 4

CumulativeRegret

4

3

2

1

0

0

250

500

750

1000

1250

150017502000

Time Horizon

(c)

c

(d)

K

96.6% larger than that of our regret curve, which implies the significant improvement in this setting with time-varying graphs. Furthermore, we perform a time complexity comparison, revealing that DrFed-UCB and GoSInE are approximately six times faster than Dist_UCB. Lastly, communication cost is directly computed by the total number of communication rounds and follows an explicit formula. Specifically, the communication costs of DrFed-UCB, Gossip_UCB, and Dist_UCB are of order T , whereas GoSInE exhibits only o ( T ) , suggesting a potential direction for optimizing communication costs.

Regret dependency results Meanwhile, we illustrate how DrFed-UCB's regret depends on several factors: the number of clients ( M ), the number of arms ( K ), the Bernoulli parameter ( c ) for the E-R model, and heterogeneity measured by h . The regret metrics are presented as (a), (b), (c), and (d) in Figure 2, respectively. We observe that regret monotonically increases with the level of heterogeneity and the number of arms, while decreasing with connectivity, which is equivalent to an increase in graph complexity. However, this monotonic trend does not apply to the number of clients. This is due to the following considerations. On one hand, a large M implies a greater number of incident edges

Start

Environment

determine

which arm

pull an arm

recievereward and

Ind 01

neighbor set

anedgewithother

exchange

Skip

clients

YES

estimators

NO

Update

estimators

of each client, providing more global information access and potentially leading to smaller regret. On the other hand, a large M also weakens the Chernoff-Hoeffding inequality for clients transmitting information, which might result in larger regret.

D Algorithms and Tables in Section 3

The algorithm for generating random connected graphs is presented in Algorithm 3 as follows.

The flowchart of Algorithm 2 is presented in Figure 3 to illustrate the information flow in the algorithm. The table below displays the various settings we consider for the regret analysis.

̸

̸

E Remarks on the theoretical results in Section 3.2

E.1 Remarks on Theorem 2

Remark ( The condition on the time horizon ) . Although the above regret bound holds for any T > L , the same bound applies to T ≤ L as follows. Assuming T ≤ L , we obtain E [ R T | A ϵ,δ ] ≤ T ≤ L where the first inequality is by noting that the rewards are within the range of [0 , 1] .

Remark ( The upper bound on the expected regret ) . Theorem 2 states a high probability regret bound, while the expected regret is often considered in the existing literature. As a corollary of Theorem 2, we establish the upper bound on E [ R T ] if ϵ = log T MT as follows. Note that

̸

where the first inequality uses E [ R T | A ϵ,δ ] ≤ T and the second inequality follows by Theorem 1. Here l 1 and l 2 are constants depending on K,M,δ, min i = i ∗ ∆ i , and λ .

Remark ( Comparison with previous work ) . A comparison to the regret bounds in the existing literature considering sub-Gaussian rewards is as follows. Our regret bounds are consistent with the prior works where the expected regret bounds are of order log T . Note that the regret bounds in [Zhu and Liu, 2023] cannot be used here since the update rule and the settings are different. Their update rule and analyses cannot carry over to our settings, which explains why we invent these modifications and proofs. On the one hand, the time-varying graphs they consider do not include the E-R model, and we can find counter-examples where their doubly stochastic weight matrices W t result in the divergence of W 1 · W 2 . . . W T . This makes the key proof step invalid in our framework. On the other hand, their time-varying graphs include the connected graphs when l = 1 , but they also make an additional assumption of doubly stochastic weight matrices, which is not applicable to regular graphs. Furthermore, they study an expected regret upper bound, while we prove a high probability regret bound that captures the dynamics in the random graphs. The graph assumptions in other works, however, are stronger, such as [Zhu et al., 2021b] consider time-invariant graphs and [Wang et al., 2021] assume graphs are complete [Perchet et al., 2016]. In contrast to some work that focuses on homogeneous rewards in decentralized multi-agent MAB, we derive regret bounds of the same order log T in a heterogeneous setting. If we take a closer look at the coefficients in terms of K,M,λ, ∆ i , our regret bound is determined by O (max( K, 1+ λ 1 -λ , 1 M 2 ∆ i ) log T ) . The work of [Zhu and Liu, 2023] arrives at O (max { log T ∆ i , K 1 , K 2 } ) where K 1 , K 2 are related to T without explicit formulas. Our regret is smaller when K ∆ i ≤ 1 and 1+ λ 1 -λ ∆ i ≤ 1 , which can always hold by rescaling ∆ i , i.e. for many cases we get substantial improvement.

E.2 Remarks on Theorem 4

Remark. Based on the expression of L 1 , we obtain that L 1 is independent of the sub-optimality gap ∆ i . Meanwhile, we have C 1 = 8 σ 2 · 12 M ( M +2) M 4 and C 2 = 3 2 C 1 = 12 σ 2 · 12 M ( M +2) M 4 . This implies that the established regret bound in Theorem 4 does not rely on ∆ i but does depend on σ 2 . To this end, we use the terminology, mean-gap independent bounds, to only represent bounds having no dependency on ∆ i , rather than instance independent that seems to be an overclaim in this case.

Remark ( Comparison with previous work ) . For decentralized multi-agent MAB with homogeneous heavy-tailed rewards and time-invariant graphs, [Dubey et al., 2020] provide an instance-dependent regret bound of order log T . In contrast, our regret bound has the same order for heterogeneous settings with random graphs, as shown in Theorem 3. Additionally, we provide a mean-gap independent regret bound as in Theorem 4. In the single-agent MAB setting, [Jia et al., 2021] consider sub-exponential rewards and derive a mean-gap independent regret upper bound of order √ T log T . Our regret bound of √ T log T is consistent with theirs, up to a logarithmic factor. Furthermore, our result is consistent with the regret lower bound as proposed in [Slivkins et al., 2019], up to a log T factor, indicating the tightness of our regret bound.

Remark. The discussion regarding the conditions on T , the expected regret E [ R T ] , and the parameter specifications follow the same logic as those in Theorem 2. We omit the details here.

F Proof of results in Section 3.2

F.1 Lemmas and Propositions

Lemma 1. For any m,i, t > L , we have

Proof of Lemma 1. The proof is referred to [Zhu and Liu, 2023].

Lemma 2. For any m,i, t > L , if n m,i ( t ) ≥ 2( K 2 + KM + M ) and graph G t is connected, then we have

where the min is taken over all clients, not just the neighbors.

Proof of Lemma 2. The proof is referred to [Zhu and Liu, 2023].

Lemma 3 (Generalized Holder's inequality) . For any r > 0 and measurable functions h i for i = 1 , . . . , n , if ∑ n i =1 1 p i = 1 r , then

The proof follows from the Young's inequality for products.

Lemma 4. Suppose that random variables X 1 , X 2 , . . . , X n are such that Y i = E [( X 1 , . . . , X n ) | ( X 1 , . . . , X i -1 , X i +1 , . . . , X n )] are sub-Gaussian distributed with variance proxy σ 1 , σ 2 , . . . , σ n , respectively. Then the sum of these sub-Gaussian random variables, ∑ n i =1 X i , is again sub-Gaussian with variance proxy ( ∑ n i =1 σ i ) 2 .

Proof. First, without loss of generality, let us assume E [ X i ] = 0 . Otherwise, we can always construct a random variable X i -E [ X i ] which has the same variance proxy with a difference up to a constant.

Defining p i = ∑ n k =1 σ k σ i gives ∑ n i =1 1 p i = 1 . Let µ be the distribution function of random vector ( X 1 , . . . , X n ) . By specifying h i ( x ) = exp ( λx ) and r = 1 , we obtain that for any λ > 0 we have

where the first inequality is by Lemma 3 and the second inequality follows the definition of subGaussian random variables.

Lemma 5. Suppose that random variables X 1 , X 2 , . . . , X n are independent sub-Gaussian distributed with variance proxy σ 1 , σ 2 , . . . , σ n , respectively. Then we have that the sum of these sub-Gaussian random variables, ∑ n i =1 X i , is again sub-Gaussian with variance proxy ∑ n i =1 σ 2 i .

Proof. For any λ > 0 note that

Since X 1 , X 2 , . . . , X n are independent random variables, we further have

where the inequality is by the definition of sub-Gaussian random variables.

This concludes the proof.

Proposition 1. Under E-R, for any 1 > δ, ϵ > 0 , and any fixed t , t ≥ L s 1 , the maintained matrix P t satisfies

with probability 1 -ϵ T . This implies that with probability at least 1 -ϵ for any t ≥ L s 1 , we have

Proof. We start with the convergence rate of matrix P t for fixed t .

We recall that in E-R, the indicator function X s i,j for edge ( i, j ) at time step s follows a Bernoulli distribution with mean value c . This implies that { X s i,j } s are i.i.d. random variables which allows us to use the Chernoff-Hoeffding inequality

For the probability term, we note that for any t ≥ L s 1 ,

since t ≥ L s 1 ≥ ln T 2 ϵ 2 δ 2 by the choice L s 1 of the burn-in period in setting 1.

As a result, the maintained matrix P t satisfies with probability at least 1 -ϵ T that

which concludes the first part of the statement.

Subsequently, consider the probability P ( || P t -cE || ∞ < δ, ∀ t > L s 1 ) . We obtain

where the first inequality uses the Bonferroni's inequality.

This completes the second part of the statement.

We next pin down the Markov chain governing Algorithm 1. Its states compound to all connected graphs if G and G ′ are connected, then the transition probability is defined by

Here ∆ denotes the symmetric difference and α ( G ) is the number of all connected graph that differ with G by at most one edge. Algorithm 1 is a random walk in the Markov chain denoted as CG -MC . The intriguing question is if the stationary distribution corresponds to the uniform distribution on all connected graphs on M nodes and if it is rapidly mixing. The next paragraph gives affirmative answers.

Proposition 2. In CG -MC , for any time step n ≥ 1 and initial connected graph G init , we have

where p ∗ = p ∗ ( M ) < 1 and π ∗ is the uniform distribution on all connected graphs.

Proof. Based on the definition of π ∗ , we have

Therefore, there exists a constant 0 < C f < 1 such that for any two connected graphs G , G ′ with | E ( G )∆ E ( G ′ ) | = 1 we have

In essence C f = 1 π ∗ min G,G ′ π ( G | G ′ ) < 1 .

If G = G ′ , then there are two possible cases. First, if α ( G ) < ( M ( M -1) 2 , then π ( G | G ) > 2 M ( M -1) > π ∗ and π ( G | G ) > 0 . Otherwise, we have π ( G | G ) = 0 . In other words, the set G ̸∈ { G ′ : π ( G ′ | G ) ≤ π ∗ ( G ′ ) , π ( G ′ | G ) > 0 } .

This implies that for G ′ ∈ { G ′ : π ( G ′ | G ) ≤ π ∗ ( G ′ ) , π ( G ′ | G ) > 0 } , we have | E ( G )∆ E ( G ′ ) | = 1 and subsequently π ( G | G ′ ) ≥ C f π ∗ .

We start with the one-step transition and obtain

where we denote the term 1 -1 # { connected graphs } and the term p ′ +(1 -C f )(1 -p ′ ) by p ′ and p ∗ , respectively. It is worth noting that p ∗ = p ∗ ( M ) and p ∗ < 1 since p ′ , C f < 1 . Here the third inequality uses the above argument on the graphs in the set { G ′ : π ( G ′ | G ) ≤ π ∗ ( G ′ ) , π ( G ′ | G ) > 0 } , and the last inequality uses the following result. By definition,

where the last inequality uses α ( G ) ≥ 1 by the definition of α ( G ) .

Suppose at time step n , the result holds, i.e. for any G

Then we consider the transition kernel at the n +1 step. Note that

where the second inequality is by using the definition of || · || TV and the last inequality holds by the results in the basis step and the induction step, respectively. The first inequality requires more arguments as follows. Consider the integral

where the results hold by exchanging the orders of the integrals as a result of Funibi's Theorem and the inequality uses the fact that ∫ A π n ( G ′ | S ) d G ′ ≤ 1 .

This completes the proof by concluding the mathematical induction.

Proposition 3. For any 1 > δ, ϵ > 0 , we obtain that for setting 2, for any fixed t ≥ L s 2 , the maintained matrix P t satisfies with probability 1 -2 ϵ T

Meanwhile, the graph generated by Algorithm 3 converges to the stationary distribution with

where π ∗ is the uniform distribution on all connected graphs.

Proof. Suppose we run the rapidly mixing markov chain for a time period of length τ 1 = ln ζ 2 ln p ∗ where ζ < δ 5 . By applying Proposition 2, we obtain that for any time step t > τ 1 ,

i.e.

̸

For the second phase, we reset the counter of time step and denote the starting point t = τ 1 + 1 as t = 1 . Based on the definition of P t , we have P t = ( ∑ t s =1 X s i,j t ) 1 ≤ i = j ≤ M where X s i,j is the indicator function for edge ( i, j ) on Graph G s that follows the distribution π τ 1 + s ( G, · ) . Let us denote Y s i,j the indicator function for edge ( i, j ) on Graph G obj s following the distribution π ∗ ( · ) and P obj t = ( ∑ t s =1 Y s i,j t ) 1 ≤ i = j ≤ M .

By the Chernoff-Hoeffding inequality and specifying ζ = 2( p ∗ ) τ 1 , we derive

holds with probability 1 -2 exp {-2 tζ 2 } .

Consider the difference between P t and P obj t and we obtain that

̸

where the last term is bounded by (3).

For the second quantity E [( X s i,j -Y s i,j )] , we have that for any s, i, j

̸

̸

and subsequently by the law of total expectation, we further obtain

In like manner, we achieve

We now proceed to the analysis on the first term. Though { X s i,j } s are neither independent or identically distributed random variables, the difference ∑ t s =1 X s i,j t -E [ X 1 i,j ] can be upper bounded by the convergence property of π n . Note that X s i,j is only different from X s +1 i,j when edge ( i, j ) is sampled at time step s and the generated graph is accepted.

We observe that

Meanwhile, we can write X s +1 i,j = 1 X s +1 i,j =1 = 1 G s +1 contains edge ( i,j ) and similarly, X s i,j = 1 X s i,j =1 = 1 G s contains edge ( i,j ) . Denote event E as E = { connected graph G contains edge ( i, j ) } . This gives us

Furthermore, the quantity ∫ A 1 S ∈ E π ∗ ( S ) dS can be simplified as

since G obj s follows a distribution with density π ∗ ( · ) .

A new Hoeffding lemma for markov chains has been recently shown as follows in [Fan et al., 2021]. Let a ( λ ) = 1+ λ 1 -λ where λ is the spectrum of the Markov chain CG -MC and by the Theorem 2.1 in [Fan et al., 2021], we obtain that

since X s i,j = 1 G s ∈ E satisfies 0 ≤ 1 G s ∈ E ≤ 1 , i.e. the values are within the range of [0 , 1] .

By the result E [ X 1 i,j ] -ζ ≤ E [ Y 1 i,j ] ≤ E [ X 1 i,j ] + ζ , we obtain

Putting the results (3), (4) and (6) together, we derive

which holds with probability at least 1 -2 exp {-2 a ( λ ) -1 tζ 2 } -2 exp {-2 tζ 2 } For the probability term 1 -2 exp {-2 a ( λ ) -1 tζ 2 } -2 exp {-2 tζ 2 } , we have

Therefore, the distance between the empirical matrix and the constant matrix reads as with probability at least 1 -2 ϵ T ,

where ζ = 2( p ∗ ) τ 1 < 1 5 δ by the choice of parameter δ and ζ . This completes the proof of Proposition 3.

Next, we proceed to explicitly characterize the spectrum of CG -MC which plays a role in the length of burning period L s 2 and L s 3 .

.

which holds by

Proposition 4. In setting 2, the spectral gap 1 -λ of CG -MC satisfies that for θ > 1 ,

Proof. It is worth noting that for G = G ′ and | E ( G )∆ E ( G ′ ) | = 1 , we have

̸

by the fact that π ∗ ( G ) = π ∗ ( G ′ ) and π ( G ′ | G ) = π ( G | G ′ ) = 2 M ( M -1) .

̸

For G = G ′ and | E ( G )∆ E ( G ′ ) | > 1 or | E ( G )∆ E ( G ′ ) | = 0 , we have π ∗ ( G ) π ( G ′ | G ) = π ∗ ( G ′ ) π ( G | G ′ ) since π ( G ′ | G ) = π ( G | G ′ ) = 0 .

For G = G ′ , we have π ∗ ( G ) π ( G ′ | G ) = π ∗ ( G ′ ) π ( G | G ′ ) by the expression.

As a result, CG -MC is reversible. Meanwhile, it is ergodic since it has a stationary distribution π ∗ as stated in Proposition 2.

Henceforth, by the result of Theorem 1 in [McNew, 2011] that holds for any ergodic and reversible Markov chain, we have

where τ ( e ) is the mixing time for an error tolerance e and λ 2 is the second largest eigenvalue of CG -MC . Choosing e > 1 2 immediately gives us

Again by Proposition 2, we have

Consequently, we arrive at and subsequently

by plugging τ ( e ) into (7).

This completes the proof of the lower bound on the spectral gap 1 -λ 2 .

In the following proposition, we show the sufficient condition for graphs generated by the E-R model being connected.

Proposition 5. Assume c in setting 1 meets the condition

where 0 < ϵ < 1 . Then, with probability 1 -ϵ , for any t > 0 , G t following the E-R model is connected.

Proof. For 1 ≤ j ≤ M , we denote the degree of client j as d j .

It is straightforward to have 1) ∑ M j =1 d j = 2 · total number of edegs, 2) E [ total number of edges ] = c · M ( M -1) 2 and 3) random variables d 1 , d 2 , . . . , d M are dependent but follow the same distribution. Note that d j follows a binomial distribution with E [ d j ] = c · ( M -1) where c is the probability of an edge. Then by the Chernoff Bound inequality, we have

where KL (0 . 5 || c ) denotes the KL divergence between Bernoulli( 0 . 5 ) and Bernoulli( c ). For the term KL (0 . 5 || c ) , we can further show that

which leads to P ( d j < M -1 2 ) ≤ exp { ( M -1) · 1 2 log 4 c (1 -c ) } .

Meanwhile, we have specified the choice of c as

which guarantees exp { ( M -1) · 1 2 log 4 c (1 -c ) } ≤ ϵ MT as follow. We observe that

This is summarized as for any j

Meanwhile, it is known as if δ ( G t ) ≥ M -1 2 , then we have that graph G t is connected where δ ( G t ) = min m d m .

As a result, consider the probability and we obtain that

P ( graph G t is connected )

where the second inequality holds by the Bonferroni's inequality and the third inequality uses (8). Consequently, we obtain

where the first inequality holds again by the Bonferroni's inequality and the second inequality results from the above derivation.

This completes the proof.

On graphs with the established properties, we next show the results on the transmission gap between two consecutive rounds of communication for any two clients and the number of arm pulls for all clients.

Proposition 6. We have that with probability 1 -ϵ , for any t > L and any m , there exists t 0 such that

where c 0 = c 0 ( K, min i = i ∗ ∆ i , M, ϵ, δ ) .

̸

Proof. The edges in setting 1 follow a Bernoulli distribution with a given parameter c by definition. Though setting 2 does not explicitly define the edge distribution, the probability of an edge existing in a connected graph, denoted as c , is deterministic, independent of time since graphs are i.i.d. over time and homogeneous among edges.

Henceforth, it is straightforward that c satisfies

and equivalently c = 2 E ( N ) M ( M -1) where N denotes the number of edges in a random connected graph.

We observe that 0 ≤ N ≤ M ( M -1) 2 . Furthermore, the existing result in [Trevisan] yields

Consequently, the probability term c has an explicit expressions c = 2 E [ N ] M ( M -1) = 2 log M M -1 .

For setting s 2 , S 2 we have c = 2 log M M -1 ≥ 1 2 since M < 10 , while in setting s 1 , S 1 , the condition on c guarantees c > 1 2 . Note that t +1 -t m,j follows a geometric distribution since each edge follows a Bernoulli distribution, which holds by

and

= P ( there is an edge between m and j at time step t+k and t m,j , no edge at time step t +1 , . . . , t + k -1) P ( there is an edge between m and j at time step t m,j )

Note that P ( t +1 -min j t m,j ≥ t 0 ) which denotes the tail of a geometric distribution depends on the choice of c . More precisely, the tail probability P 0 is monotone decreasing in c .

When c = 1 2 , we obtain that

Choosing t 0 = ln M 2 T ϵ ln 2 leads to P 0 = 1 -( 1 2 ) t 0 = 1 -ϵ M 2 T and

where the last inequality holds by the choice of L . This implies min l n l,i ( t + 1) -t 0 ≥ (1 -c 0 ) min l n l,i ( t +1) , i.e. t 0 ≤ c 0 min l n l,i ( t +1) .

Therefore, with probability 1 -ϵ M 2 T ,

where the first inequality results from the fact t m,l ≥ min j t m,j , the second inequality uses the fact from (9), the third inequality applies the definition of n , and the last inequality holds by the choice of t 0 .

Consider setting s 3 , S 3 where M > 10 . Generally, for a given parameter c , we obtain

and subsequently

For the probability term P 0 , we further arrive at

by the choice of t 0 ≥ ln( ϵ M 2 T ) ln(1 -c ) .

Meanwhile, we claim that the choice of t 0 satisfies

since ln( ϵ M 2 T ) ln(1 -c ) ≤ c 0 min l n l,i ( t +1) holds by noting n l,i ( t +1) ≥ n l,i ( L ) ≥ L K and

To summarize, in all the settings, we have that with probability at least 1 -ϵ M 2 T ,

Therefore, we obtain that in setting s 1 , S 1

where the inequality is a result of the Bernoulli's inequality.

In setting s 2 , S 2 , s 3 , S 3 , { t +1 -min j t 1 ,j , . . . , t +1 -min j t M,j } follow the same distribution, but are dependent since they construct a connected graph. However, we have the following result

As a consequence, we arrive at that in setting s 1 , S 1 , s 2 , S 2 , s 3 , S 3 ,

where the first inequality again uses the Bonferroni's inequality and the second inequality holds by applying (11) and (12).

After establishing the transmissions among clients, we next proceed to show the concentration properties of the network-wide estimators maintained by the clients.

The first is to demonstrate the unbiasedness of these estimators with respect to the global expected rewards.

Proposition 7. Assume the parameter δ satisfies that 0 < δ < c = f ( ϵ, M, T ) . For any arm i and any client m , at every time step t , we have

Proof. The result can be shown by induction as follows. We start with the basis step by considering any time step t ≤ L +1 . By the definition of ˜ µ i m ( t ) = ˜ µ i m ( L +1) , we arrive at

where P ′ m,j ( L ) = { 1 M , if P L ( m,j ) > 0 0 , else . The definition of A ϵ,δ and the choice of δ guarantee that | P L -cE | < δ < c on event A ϵ,δ , i.e. we have for any t ≥ L , P t > 0 and thereby obtaining

Therefore, we continue with (13) and have

where the last equality uses the law of total expectation.

With the derivations, we further have

where the second equality (15) uses the fact that { s : n j,i ( s ) -n j,i ( s -1) = 1 } is contained in σ ( n j,i ( l )) l ≤ L m,j and the third equality (16) results from that r j i ( s ) is independent of everything else given s and E [ r j i ( s )] = µ j i .

The induction step follows a similar analysis as follows. Suppose that for any s ≤ t we have E [˜ µ i m ( s ) | A ϵ,δ ] = µ i .

For time step t +1 , we first write it as

where P ′ t ( m,j ) and d are constants since P t ( m,j ) > 0 for t ≥ L on event A ϵ,δ and the last equality is again by the law of total expectation.

This gives us that by the law of total expectation

where the first equality uses P ′ t ( m,j ) and d are constants on event A ϵ,δ , the second equality is derived by re-organizing the terms, the third equality again uses the law of total expectation and integrates the second term by t m,j , the fourth equality elaborate the second term and the equality in the last line follows from the induction and (15, 16 17).

This completes the induction step and thus shows the unbiasedness of the network-wide estimators conditional on event A ϵ,δ .

Then we characterize the moment generating functions of the network-wide estimators and conclude that they have similar properties as their local rewards.

Proposition 8. Assume the parameter δ satisfies that 0 < δ < c = f ( ϵ, M, T ) . In setting s 1 , s 2 , s 3 where rewards follow sub-gaussian distributions, for any m,i, λ and t > L where L is the length of

the burn-in period, the global estimator ˜ µ i m ( t ) is sub-Gaussian distributed. Moreover, the conditional moment generating function satisfies that with P ( A ϵ,δ ) = 1 -7 ϵ ,

where σ 2 = max j,i (˜ σ j i ) 2 and C = max { 4( M +2)(1 -1 -c 0 2( M +2) ) 2 3 M (1 -c 0 ) , ( M +2)(1 + 4 Md 2 m,t ) } .

Proof. We prove the statement on the conditional moment generating functions by induction. Let us start with the basis step.

Note that the definition of A ϵ,δ and the choice of δ again guarantee that for t ≥ L , | P t -cE | < δ < c on event A ϵ,δ . This implies that for any t ≥ L , m and j , P t ( m,j ) > 0 , and if t = L

Consider the time step t ≤ L +1 . The quantity satisfies that

where the third equality holds by (19), the fourth equality uses the definition µ i = 1 M ∑ M i =1 µ j i , and the last inequality results from the generalized hoeffding inequality as in Lemma 3 and the fact that ˆ ¯ µ m i,j ( h L m,j ) = ¯ µ j i ( h L m,j ) .

Note that for any client j , we have

and if t > L

It is worth noting that given s , r j i ( s ) is independent of everything else, which gives us

where the first inequality holds by the definition of sub-Gaussian random variables r j i ( s ) -µ j i with an mean value 0 , the second inequality results from 1 A ϵ,δ ≤ 1 , and the last inequality uses n j,i ( h L m,j ) ≥ min j n j,i ( h L m,j ) for any j .

Therefore, we obtain that by plugging (23) into (21)

which completes the basis step.

Now we proceed to the induction step. Suppose that for any s < t +1 where t ≥ L , we have

The update rule of ˜ µ i m implies that

where the first inequality uses Lemma 3 and the second inequality applies (24) as the assumption for the induction step and holds by exchanging the expectations with the multiplication since again given s the reward ( r j i ( s ) -µ j i ) is independent of other random variables.

We continue bounding the last two terms by using the definition of sub-Gaussian random variables ( r j i ( s ) -µ j i ) and obtain

Building on that, we establish

where the first inequality uses the fact that for any j , n j,i ( t ) ≥ min j n j,i ( t ) and n j,i ( t m,j ) ≥ min j n j,i ( t m,j ) . For the second inequality, the first term is a result of min j n j,i ( t ) min j n j,i ( t +1) ≥ min j n j,i ( t ) min j n j,i ( t )+1 ≥ L/K L/K +1 since n j,i ( t ) > n j,i ( L ) = L/K and the ratio is monotone increasing in n , and the second term is bounded based on the following derivations

where the last inequality holds by applying Proposition 6 that holds on event A ϵ,δ .

Therefore, we can rewrite the above expression as

where the first inequality holds by the choice of P ′ t ( m,j ) , d m,t , L, c 0 and C and the second inequality uses the fact that 1 A ϵ,δ ≤ 1 and min j n j,i ( t +1) ∈ σ ( { n m,i ( t ) } t,i,m ) .

This completes the induction step and subsequently concludes the proof.

Proposition 9. Assume the parameter δ satisfies that 0 < δ < c = f ( ϵ, M, T ) . In setting s 1 , s 2 , and s 3 , for any m,i and t > L where L is the length of the burn-in period, ˜ µ m,i ( t ) satisfies that if if

n m,i ( t ) ≥ 2( K 2 + KM + M ) , then with P ( A ϵ,δ ) = 1 -7 ϵ ,

Proof. By Proposition 7, we have E [˜ µ m,i ( t ) -µ i | A ϵ,δ ] = 0 , which allows us to consider the tail bound of the global estimator ˜ µ i m ( t ) conditional on event A ϵ,δ as follows.

Note that

where the last inequality is by the fact that 1 exp { λ ( n )(˜ µ m,i ( t ) -µ i ) }≥ exp { λ ( n ) √ C 1 log t n m,i ( t ) } ≤

By the assumption that δ < c , we have Proposition 8 holds. Subsequently, by Proposition 8 and Lemma 2 which holds since n m,i ( t ) ≥ 2( K 2 + KM + M ) , we have for any λ

Again, we utilize the law of total expectation and further obtain

where the first inequality holds by (44) and the second inequality holds by choosing λ ( n ) = √ C 1 log t n m,i ( t ) 2 Cσ 2 n m,i ( t ) and by the choice of parameter C 1 such that C 1 4 Cσ 2 ≥ 2 or equivalently C 1 ≥ 8 Cσ 2 .

In like manner, we obtain that by repeating the above steps with µ i -˜ µ m,i ( t ) , we have

which complete the proof.

̸

Proposition 10. Assume the parameter δ satisfies that 0 < δ < c = f ( ϵ, M, T ) . An arm k is said to be sub-optimal if k = i ∗ where i ∗ is the unique optimal arm in terms of the global reward, i.e. i ∗ = arg max 1 M ∑ M j =1 µ j i . Then in setting s 1 , s 2 and s 3 , when the game ends, for every client m , 0 < ϵ < 1 and T > L , the expected numbers of pulling sub-optimal arm k after the burn-in period satisfies with P ( A ϵ,δ ) = 1 -7 ϵ

Proof of Proposition 10. We claim that what lead to pulling an sub-optimal arm i are explicit by the decision rule of Algorithm 2, meaning that the result a t m = i holds when any of the following conditions is met:

Case 1: n m,i ( t ) ≤ N m,i ( t ) -K ,

Case 2: ˜ µ m,i -µ i > √ C 1 log t n m,i ( t -1) ,

Case 3: -˜ µ m,i ∗ + µ i ∗ > √ C 1 log t n m,i ∗ ( t -1) ,

Case 4: µ i ∗ -µ i < 2 √ C 1 log t n m,i ( t -1) .

Then we formally consider the number of pulling arms n m,i ( T ) starting from L +1 . For any l > 1 , we have that based on the above listed conditions

Consequently, the expected value of n m,i ( t ) conditional on A ϵ,δ reads as

where l = max { [ 4 C 1 log T ∆ 2 i ] , 2( K 2 + MK + M ) } .

For the last term in (30), we have

since the choice of l satisfies l ≥ [ 4 C 1 log T ∆ 2 i ] with ∆ i = µ i ∗ -µ i .

For the first two terms, we have on event A ϵ,δ

where the first inequality holds by the property of the probability measure when removing the event n m,i ( t -1) ≥ l and the second inequality holds by (47) and (29) as stated in Proposition 9, which holds by the assumption that δ < c .

For Case 1, we note that Lemma 1 implies that

with the definition of N m,i ( t +1) = max { n m,i ( t +1) , N j,i ( t ) , j ∈ N m ( t ) } .

Departing from the result that the difference between N m,i ( t ) and n m,i ( t ) is at most K ( K +2 M ) , we then present the following analysis on how long it takes for the value -n m,i ( t ) + N m,i ( t ) to be smaller than K .

At time step t , if Case 1 holds for client m , then n m,i ( t +1) is increasing by 1 on the basis of n m,i ( t ) . What follows characterizes the change of N m,i ( t +1) . Client m satisfying n m,i ( t ) ≤ N m,i ( t ) -K will not change the value of N m,i ( t +1) by the definition N m,i ( t +1) = max { n m,i ( t +1) , N j,i ( t ) , j ∈

N m ( t ) } . Moreover, for client j ∈ N m ( t ) with n j,i ( t ) < N j,i ( t ) -K , i.e. N j,i ( t + 1) will not be affected by n j,i ( t + 1) ≤ n j,i ( t ) + 1 . Thus, the value of N m,i ( t + 1) = max { n m,i ( t + 1) , N j,i ( t ) , j ∈ N m ( t ) } is independent of such clients. We observe that for client j ∈ N m ( t ) with n j,i ( t ) > N j,i ( t ) -K , the value N j,i ( t ) will be the same if the client does not sample arm i , which leads to a decrease of 1 in the difference -n m,i ( t ) + N m,i ( t ) . Otherwise, if such a client samples arm i which brings an increment of 1 to N m,i ( t ) , the difference between n m,i ( t ) and N m,i ( t ) will remain the same. However, the latter has just been discussed and must be the cases as in Case 2 and Case 3, the total length of which has already been upper bounded by π 2 3 as shown in (32).

Therefore, the gap is at most K ( K +2 M ) -K + π 2 3 , i.e.

Subsequently, we derive that

where the inequality results from (30), (31), (32), and (33).

This completes the proof steps.

Next, we establish the concentration inequalities of the network-wide estimators when the rewards follow sub-exponential distributions, i.e. in setting S 1 , S 2 , and S 3 .

Proposition 11. Assume the parameter δ satisfies that 0 < δ < c = f ( ϵ, M, T ) . In setting S 1 , S 2 , and S 3 , for any m,i, λ and t > L where L is the length of the burn-in period, the global estimator ˜ µ i m ( t ) is sub-exponentially distributed. Moreover, the conditional moment generating function satisfies that with P ( A ϵ,δ ) = 1 -7 ϵ , for | λ | < 1 α

where σ 2 = max j,i (˜ σ j i ) 2 and C = max { 4( M +2)(1 -1 -c 0 2( M +2) ) 2 3 M (1 -c 0 ) , ( M +2)(1 + 4 Md 2 m,t ) } .

Proof. Assume that parameter | λ | < 1 α . Weprove the statement on the conditional moment generating function by induction. Let us start with the basis step.

Note that the definition of A and the choice of δ again guarantee that for t ≥ L , | P t -cE | < δ < c on event A . This implies that for any t ≥ L , P t > 0 and thereby obtaining that if t = L

and if t > L

Consider the time step t ≤ L +1 . The quantity

where the third equality holds by (34), the fourth equality uses the definition µ i = 1 M ∑ M i =1 µ j i and the last inequality results from the generalized hoeffding inequality as in Lemma 3.

Note that for any client j , by the definition of ¯ µ j i ( h L m,j ) we have

It is worth noting that given s , r j i ( s ) is independent of everything else, which gives us

where the first inequality holds by the definition of sub-exponential random variables r j i ( s ) -µ j i with mean 0 , the second inequality again uses 1 A ϵ,δ ≤ 1 , and the last inequality is by the fact that n j,i ( h L m,j ) ≥ min j n j,i ( h L m,j ) .

Therefore, we obtain that by plugging (38) into (36)

which completes the basis step.

Now we proceed to the induction step. Suppose that for any s < t +1 where t +1 > L +1 , we have

The update rule of ˜ µ i m again and (25) implies that

We continue bounding the last two terms by using the definition of sub-exponential random variables ( r j i ( s ) -µ j i ) and obtain

Building on that, we establish

where the first inequality uses the fact that n j,i ( t ) ≥ min j n j,i ( t ) and n j,i ( t m,j ) ≥ min j n j,i ( t m,j ) . For the second inequality, the first term is a result of min j n j,i ( t ) min j n j,i ( t +1) ≥ min j n j,i ( t ) min j n j,i ( t )+1 ≥ L/K L/K +1 since n j,i ( t ) > n j,i ( L ) = L/K and the ratio is monotone increasing in n , and the second term is bounded through applying Proposition 6, which holds on event A ϵ,δ and leads to

Therefore, we can rewrite the above expression as

where the first inequality holds again by the choice of P ′ t ( m,j ) , d m,t , L and c 0 and the second inequality uses the fact that 1 A ϵ,δ ≤ 1 .

This completes the induction step and subsequently concludes the proof.

Proposition 12. Assume the parameter δ satisfies that 0 < δ < c = f ( ϵ, M, T ) . In setting S 1 , S 2 , and S 3 , for any m,i and t > L where L is the length of the burn-in period, the deviation of ˜ µ m,i ( t )

satisfies that if n m,i ( t ) ≥ 2( K 2 + KM + M ) , then with P ( A ϵ,δ ) = 1 -7 ϵ ,

Proof. By Proposition 7, we have E [˜ µ m,i ( t ) -µ i | A ϵ,δ ] = 0 , which allows us to consider the tail bound of the global estimator ˜ µ i m ( t ) conditional on event A ϵ,δ . It is worth mentioning that by the choice of C 1 and C 2 , we have

where ˜ σ 2 is 2 Cσ 2 n m,i ( t ) .

Note that since we set Rad = √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) , we obtain

On the one hand, if √ C 1 log T

where the last inequality is by the fact that 1 exp { λ ( n )( | ˜ µ m,i ( t ) -µ i | ) }≥ exp { λ ( n ) C 2 ln T n m,i ( t ) } ≤ exp { λ ( n )( | ˜ µ m,i ( t ) -µ i | ) } exp { λ ( n ) C 2 ln T ( t ) } .

By the assumption that δ < c , we have Proposition 11 holds. Subsequently, by Proposition 11 and Lemma 2 which holds since n m,i ( t ) ≥ 2( K 2 + KM + M ) , we have for any | λ | < 1 α

Likewise, we obtain that by taking λ = -λ ,

With (44) and (45), we arrive at for any | λ | < 1 α that

Again, we utilize the law of total expectation and further obtain that | λ ( n ) | < 1 α

where the first inequality holds by (46).

Note that the condition √ C 1 log T n m,i ( t ) ˜ σ 2 > 1 α implies that n m,i ( t ) < C 1 ln T ˜ σ 2 α which is the global optima of the function in (47). This is true since n m,i ( t ) ≤ C 1 log T α 2 (˜ σ ) 4 ≤ ( C 2 log T ) 2 C 1 . Henceforth, (47) is monotone decreasing in λ ( n ) ∈ (0 , 1 α ) and we obtain a minima when choosing λ ( n ) = 1 α and using the continuity of (47).

Formally, it yields that

where the last inequality uses the choice of C 2 and the condition that 1 2 α 2 ˜ σ 2 -1 α C 2 ln T n m,i ( t ) ≤ -4 ln T which holds by the following derivation. Notably, we have

where the first inequality uses n m,i ( t ) < C 1 ln T ˜ σ 2 α and the last inequality is by the choices of parameters ( 1 2 -C 2 C 1 ) C 1 2 Cσ 2 ≤ -4 .

On the other hand, if √ C 1 log T n m,i ( t ) ˜ σ 2 < 1 α , i.e. n m,i ( t ) ≥ C 1 log T α 2 (˜ σ ) 4 , we observe for | λ ( n ) | < 1 α

by a same argument from (43) to (47) replacing C 2 ln T n m,i ( t ) with √ C 1 ln T n m,i ( t ) . When choosing λ ( n ) = √ C 1 log T n m,i ( t ) ˜ σ 2 that meets the condition λ < 1 α under the assumption √ C 1 log T n m,i ( t ) ˜ σ 2 < 1 α and noting that

C 1 2 Cσ 2 ≥ 4 , we obtain

To conclude, by (41) and (42), we have

which completes the proof.

̸

Proposition 13. Assume the parameter δ satisfies that 0 < δ < c = f ( ϵ, M, T ) . An arm k is said to be sub-optimal if k = i ∗ where i ∗ is the unique optimal arm in terms of the global reward, i.e. i ∗ = arg max 1 M ∑ M j =1 µ j i . Then in setting S 1 , S 2 and S 3 , when the game ends, for every client m , 0 < ϵ < 1 and T > L , the expected numbers of pulling sub-optimal arm k after the burn-in period satisfies with P ( A ϵ,δ ) = 1 -7 ϵ

Proof. Recall that Rad = √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) . We again have a t m = i holds when any of the following conditions is met: Case 1: n m,i ( t ) ≤ N m,i ( t ) -K , Case 2: ˜ µ m,i -µ i > √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) , Case 3: -˜ µ m,i ∗ + µ i ∗ > √ C 1 ln T n m,i ∗ ( t ) + C 2 ln T n m,i ∗ ( t ) , and Case 4: µ i ∗ -µ i < 2( √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) ) .

By (30), the expected value of n m,i ( t ) conditional on A ϵ,δ reads as

where l is specified as l = max { [ 4 C 1 log T ∆ 2 i ] , 2( K 2 + MK + M ) } with ∆ i = µ i ∗ -µ i .

For the last term in the above upper bound, we have

since the choice of l satisfies l ≥ max([ 16 C 1 log T ∆ 2 i ] , [ 4 C 2 log T ∆ i ] , 2( K 2 + MK )) .

For the first two terms, we have on event A ϵ,δ

where the first inequality holds by the property of the probability measure when removing the event n m,i ( t -1) ≥ l and the second inequality holds by Proposition 12, which holds by the assumption that δ < c .

For Case 1, we note that Lemma 1 implies that

with the definition of N m,i ( t +1) = max { n m,i ( t +1) , N j,i ( t ) , j ∈ N m ( t ) } .

Departing from the result that the difference between N m,i ( t ) and n m,i ( t ) is at most K ( K +2 M ) , we then present the following analysis on how long it takes for the value -n m,i ( t ) + N m,i ( t ) to be smaller than K .

At time step t , if Case 1 holds for client m , then n m,i ( t +1) is increasing by 1 on the basis of n m,i ( t ) . What follows characterizes the change of N m,i ( t +1) . Client m satisfying n m,i ( t ) ≤ N m,i ( t ) -K will not change the value of N m,i ( t +1) by the definition N m,i ( t +1) = max { n m,i ( t +1) , N j,i ( t ) , j ∈ N m ( t ) } . Moreover, for client j ∈ N m ( t ) with n j,i ( t ) < N j,i ( t ) -K , i.e. N j,i ( t + 1) will not be affected by n j,i ( t + 1) ≤ n j,i ( t ) + 1 . Thus, the value of N m,i ( t + 1) = max { n m,i ( t + 1) , N j,i ( t ) , j ∈ N m ( t ) } is independent of such clients. We observe that for client j ∈ N m ( t ) with n j,i ( t ) > N j,i ( t ) -K , the value N j,i ( t ) will be the same if the client does not sample arm i , which leads to a decrease of 1 in the difference -n m,i ( t ) + N m,i ( t ) . Otherwise, if such a client samples arm i which brings an increment of 1 to N m,i ( t ) , the difference between n m,i ( t ) and N m,i ( t ) will remain the same. However, the latter has just been discussed and must be the cases as in Case 2 and Case 3, the total length of which has already been upper bounded by 2 P ( A ϵ,δ ) T 3 as shown in (52).

Therefore, the gap is at most K ( K +2 M ) -K + 2 P ( A ϵ,δ ) T 3 , i.e.

Subsequently, we derive that

where the inequality results from (50), (51), (52) and (53).

F.2 Proof of Theorems

Theorem 1. For event A ϵ,δ and any 1 > ϵ, δ > 0 , we have P ( A ϵ,δ ) ≥ 1 -7 ϵ .

Proof. Recall that we define events

where A 1 , A 2 , A 3 belong to the σ -algebra in the probability space since the time horizon is countable, i.e. for the probability space (Ω , Σ , P ) , A 1 , A 2 , A 3 ∈ Σ .

Meanwhile, we obtain

where the first inequality includes all settings and L ≥ L s i , the second inequality results from the Bonferroni's inequality and the third inequality holds by Proposition 1 and Proposition 3.

At the same time, note that

where the first inequality is by the definition of L , the second inequality again uses the Bonferroni's inequality, and the third inequality results from Proposition 6.

Moreover, we observer that

where the first inequality uses the definition of L , the second inequality is by the Bonferroni's inequality and the third inequality holds by Proposition 5 and the definition of s 2 , s 3 where all graphs are guaranteed to be connected.

Consequently, we arrive at

where the first inequality utilizes the Bonferroni's inequality, the second inequality results from (54), (55), and (56).

This concludes the proof and shows the validness of the statement.

Theorem 2. Let f be a function specific to a setting and detailed later. For every 0 < ϵ < 1 and 0 < δ < f ( ϵ, M, T ) , in setting s 1 with c ≥ 1 2 + 1 2 √ 1 -( ϵ MT ) 2 M -1 , s 2 and s 3 , with the time horizon T satisfying T ≥ L , the regret of Algorithm 2 with F ( m,i, t ) = √ C 1 ln t n m,i ( t ) satisfies that

̸

where the length of the burn-in period is explicitly

̸

with λ being the spectral gap of the Markov chain in s 2 , s 3 that satisfies 1 -λ ≥ 1 2 ln 2 ln 2 p ∗ ln 4+1 , p ∗ = p ∗ ( M ) < 1 and c 0 = c 0 ( K, min i = i ∗ ∆ i , M, ϵ, δ ) , and the instance-dependent constant C 1 = 8 σ 2 max { 12 M ( M +2) M 4 } .

Proof. The optimal arm is denoted as i ∗ satisfying

For the proposed regret, we have that for any constant L ,

where the first inequality is by taking the absolute value and the second inequality results from the assumption that 0 < µ j i < 1 for any arm i and client j .

Note that ∑ K i =1 ∑ M m =1 n m,i ( T ) = M ( T -L ) where by definition n m,i ( T ) is the number of pulls of arm i at client m from time step L +1 to time step T , which yields that

̸

where the second inequality uses the fact that ∑ m : µ i m ∗ -µ i m ≤ 0 n m,i ( T )( µ i m ∗ -µ i m ) ≤ 0 holds for any arm i and the last equality is true since n m,i ( T )( µ i m ∗ -µ i m ) = 0 for i = i ∗ and any m .

̸

Meanwhile, by the choices of δ such that δ < c = f ( ϵ, M, T ) , we apply Proposition 10 which leads to for any client m and arm i = i ∗ ,

As a result, the upper bound on R T can be derived as by taking the conditional expectation over R T on A ϵ,δ

̸

and

̸

̸

where the second inequality holds by plugging in (57).

̸

Meanwhile, we note that for any i = i ∗ ,

which gives us that

̸

Hence, the regret can be upper bounded by

(59)

where the inequality is derived from (60) and L is the same constant as in the definition of A ϵ,δ . This completes the proof.

Theorem 3. Let f be a function specific to a setting and defined in the above remark. For every 0 < ϵ < 1 and 0 < δ < f ( ϵ, M, T ) , in settings S 1 with c ≥ 1 2 + 1 2 √ 1 -( ϵ MT ) 2 M -1 , S 2 , S 3 with the time horizon T satisfying T ≥ L , the regret of Algorithm 2 with F ( m,i, t ) = √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) satisfies

̸

where L, C 1 are specified as in Theorem 2 and C 2 C 1 ≥ 3 2 .

Proof. By the regret decomposition as in (58), we obtain that

̸

By Proposition 13, we have that with probability at least 1 -7 ϵ

Following (60) gives us that

Therefore, we derive that with probability at least P ( A ϵ,δ ) = 1 -7 ϵ

̸

which completes the proof.

Theorem 4. Assume the same conditions as in Theorems 2 and 3. The regret of Algorithm 2 satisfies that

where L 1 = max( L, K (2( K 2 + MK + M ))) , L, C 1 is specified as in Theorem 2, and C 2 C 1 ≥ 3 2 . The involved constants depend on σ 2 but not on ∆ i .

Proof. Define U t m ( i ) and L t m ( i ) as ˜ µ i m ( t ) + Rad ( i, m, t ) and ˜ µ i m ( t ) -Rad ( i, m, t ) , respectively, where Rad is previously defined as Rad ( i, m, t ) = √ C 1 ln T n m,i ( t ) + C 2 ln T n m,i ( t ) . Weobserve that by definition, the regret R T can be written as

Subsequently, the conditional expectation of R T has the following decomposition

where L 1 = max( L, 2( K + MK + M )) .

For the first term, we derive its upper bound as follows.

Note that

where the first inequality uses the monotone property of E [ · ] , the second inequality omits the latter negative quantity, the third inequality holds by the fact that 0 ≤ µ ∗ i ≤ 1 , and the last inequality is by Proposition 12.

In like manner, we have that the last term satisfies that

by the same logic as the above and substituting i ∗ with a t m , and thus we omit the details here.

We then proceed to bound the second term. Based on the decision rule in Algorithm 2, we have either E [ U t m ( i ∗ ) -U t m ( a t m ) | A ϵ,δ ] < 0 or n m,i ( t ) < N m,i ( t ) -K . This is equivalent to

By definition, U t m ( i ∗ ) = ˜ µ i ∗ + Rad ( i ∗ , m, t ) implies that

which leads to and subsequently

Following (53) that only depends on whether clients stay on the same page that relies on the transmission, we obtain

which immediately leads to

Afterwards, we consider the third term and have

Putting (65, 66, 68, 69) all together, we deduce that

Meanwhile, we observe that by definition

By the sum of the Harmonic series, we have

Meanwhile, by the Cauchy-Schwartz inequality, we obtain

where the last inequality again uses the result on the Harmonic series as in (71).

Therefore, the cumulative value can be bounded as

Using the result of (70), we have

which completes the proof.

G Choices of parameter c 0 in Theorem 2

Parameter c 0 We note that c 0 is a pre-specified parameter which are different in different settings. The choices of c 0 are as follows. Meanwhile, we need to study whether the possible choices of c 0 explode in terms of the order of T .

Remark (2) . The regret reads

̸

with L denoted as L = max { L 1 , L 2 , L 3 } = max { a 1 , a 2 , a 3 , b 1 c 0 , b 2 c 0 , b 3 c 0 } and C 1 denoted as max { e 1 -c 0 , f } , where parameters a 1 , a 2 , a 3 , b 1 , b 2 , b 3 , e, f are specified as

This function of c 0 is non-differentiable which brings additional challenges and requires a case-bycase analysis.

Let a = { a 1 , a 2 , a 3 } and b = { b 1 , b 2 , b 3 } . Then continue with the decision rule as in the previous discussion.

Case 1: there exists c 0 such that a ≥ b c 0 ,i.e. c ≥ b a and b a ≤ 1 Then R T is monotone increasing in c due to C 1 and c 0 = b a gives us the optimal regret R 1 T .

Case 2: if a ≤ b c 0 ,i.e. c 0 ≤ b a

-else we have c 0 ≥ 1 -e f

* if 1 -e f > b a , it leads to contradiction and this can not be the case.

else 1 -e f ≤ b a , we obtain

̸

̸

which implies that the optimal choice of c 0 is √ b √ b + √ e ∑ i = i ∗ ([ 4 log T ∆2 i ])

̸

if 1 -e f ≤ √ b √ b + √ e ∑ i = i ∗ ([ 4 log T ∆2 i ]) ≤ b a , this gives us the final choice of c 0 and

the subsequent local optimal regret R 2 T .

̸

subsequent local optimal regret is R T

elif √ b √ b + √ e ∑ i = i ∗ ([ 4 log T ∆2 i ]) < 1 -e f , the optimal choice of c 0 is 1 -e f and the 2

else the optimal choice of c 0 is b a and the subsequent local optimal regret is R 2 T

Compare R 1 T and R 2 T and choose the c 0 associated with the smaller value.

̸

̸

̸

̸

̸

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of multi-agent multi-armed bandits where the arm rewards generated for each agent is heterogeneous across agents, and the goal is to find the global optimal arm, where the reward of the global optimal arm is the mean of the mean reward observed by agents. In addition, this work considers that the agents are connected through an incomplete and time-varying topology. Both heterogeneous agent-specific reward and time-varying underlying topology make sense from a practical perspective and also add intellectually interesting technical challenges to the multi-agent MAB problem. The authors propose a series of algorithms for graph generation and bandit learning and then analyze the regret of the proposed algorithms under multiple different assumptions and in both problem-dependent and problem-independent settings. The regret bounds are interesting and (near) optimal in most cases.\n\nSTRENGTHS:\n++ This paper addresses a very challenging setting of multi-agent MAB problem where reward generation is heterogeneous and the graph is time-varying. \n\n++ The presentation of the paper is clear and solid and it clearly explains the challenges and missing points in the existing literature. \n\n++ The proposed algorithms make sense intuitively.\n\n++ The statements of the results sound solid (even though, I cannot confirm the details of the proof) and significant, with sufficient explanations in comparison with the literature.\n\nWEAKNESSES:\n-- Overall, this paper is a solid work with novel and non-trivial contributions. Unfortunately, the technical part of this paper is beyond my core expertise, and I can't note on the correctness of the proofs. \n\n-- In the literature of multi-agent bandits, there is a substantial thread of the work that considers communication cost as a metric to improve upon. This work, however, does not touch at all the communication complexity in their study. Can authors comment on whether or not it makes sense to add communication cost analysis to their work? \n\n-- In this work, the authors, by default, assume that the goal of the game is to find the global optimal arm. However, one may suggest that in a multi-agent setting, each agent may care more about their local optimal arm. Can the authors comment on how the algorithms may or may not work when the objective is to find local optimal arms?\n\nQUESTIONS:\nsee two questions above.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUpdate: I've read the authors' rebuttal as well as all the reviews from other reviewers. The authors' clarification are helpful and mitigate my concerns. I've raised the score from 6 to 7 and I think this manuscript is acceptable for publication.\n\nThis paper aims at making contributions to decentralized multi-agent MAB problem research by studying both heterogeneous rewards (with heavy-tailed reward distributions) and time-varying random graphs, where the distributions of rewards and graphs are independent of time. It is claimed that this is the first work to consider this problem and to investigate it with heavy-tailed reward distributions. To be more specific, the paper investigates 1) heterogeneous sub-exponential and sub-gaussian distributed rewards and 2) random graphs including the possibly unconnected E-R model and random connected graphs, and applies them to the decentralized multi-agent MAB framework using a proposed UCB-type solution. It is considered as an extension of relevant existing literature (especially [Dubey et al.,2020] and and [Zhu et al., 2021b]) on decentralized multi-agent MAB  research area.\n\nSTRENGTHS:\n1. Paper presentation is clean and clear, and overall speaking, the manuscript is reader friendly and straightforward to follow.\n2. There are quite nontrivial novel theoretical analysis results included in the paper, which seems to be solid.\n3. There are quite a couple of helpful remarks/explanations comparing with existing literature.\n\nWEAKNESSES:\n1. I'd like to see more more explanations/elaborations about the motivation/practical backgrounds why those types graphs are considered and what are some example real world applications that might have those types of graphs. For example, this paper consider the uniformly distributed connected graph generated using Algorithm 1, but could the author provide any example real world applications that have that kind of graphs? Especially about the part of adding extra minimum number of edges to make the randomly sampled graph to be connected . That seems to be a purely artificial/simulation based approach, and I don't obviously see much real world applications that demonstrate such behavior. The paper introduction briefly mention random failure graph in real world applications as motivation, but obviously random failed graphs have essential difference compared with such formulated graphs. More elaborations/explanations about the motivations/applications would be helpful to convince that such an artificial formulation is not just for convenient mathematical analysis, but also very beneficial to real world applications, rather than just a mathematical game.\n\n2. In algorithm 1, I saw a constant probability \"1/2\" in the graph sampling part. Why this specific \"1/2\" is used here? Would any constant probability 0<p<1 work the same here for the analysis and theoretical results (as well as obtained bounds when comparing with literature)? It would be great if the authors could help clarify/confirm this in the manuscript.\n\n3. I saw undirected graph is considered/assumed in this paper. How hard/difficult to consider the more generic scenario of directed graphs and what are the main challenges? It would be valuable to see some discussions on this for future research directions.\n\n4. I'd like to see some scaling and complexity discussions. Is the proposed method naturally scalable to large multi-agent systems? It would be good to see some experiments and analysis on this, ideally comparing with relevant literature.\n\nQUESTIONS:\nPlease see my questions in my above comment when taking about the weakness.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents a novel algorithm for solving decentralized multi-agent multi-armed bandit problems for two graph dynamics under both sub-exponential and sub-gaussian reward distributions. Their whole algorithm framework can be divided into a Burn-in period and a Learning period, ensuring a process warm-up phase before getting UCB-type estimators. The paper includes detailed theorems and proofs in both the main manuscript and appendix with regret analysis.

STRENGTHS:
1. Authors fill in the gap in the multi-agent multi-armed bandit problem with the intersection between heavy-tailed reward distributions and heterogeneous settings with time-varying graphs.
2. The paper is clearly written and easy to follow.

WEAKNESSES:
1. The paper is full of details but without a conclusion to re-emphasize the main statement and contribution of this paper.
2. Please refer to the questions

QUESTIONS:
1. In line 368, is there any typo mentioning the same conditions as in Theorem 2 and $\textbf{4}$?
2. In line 312, the 3rd and 4th terms are exactly the same. Is there any typo?
3. Could you please elaborate more about how you get L1 in theorem 4?

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies stochastic multi-armed bandit over a network of agents communicating locally. Specifically, for each of the K arms, each agent has an associated underlying arm-mean. The agents can only communicate with their neighbors in each round to exchange information. The goal of the agents is to pull arms so that the regret with respect to the average arm-mean averaged over the agents is minimized. This is a challenging problem since the agents can only infer their arm mean from the observed rewards and must rely on information sharing to learn the average of the arm means averaged over the network.\n\nIn order to model information sharing, the paper assumes that the communication graph over agents is a stochastic process that possibly changes over time. This allows for the information about an arm collected by an agent to percolate over the network so that all agents can potentially learn of the mean of an arm averaged over agents. However, the agents never have full information on the graph connectivity and at each time, can only observe who their neighbors are. \n\nIn this scenario, the paper proposes an algorithm based on initial coordinated exploration and information sharing followed by UCB based on estimated network sample average over arms. The proposed algorithm is elegant and practical.\n\nSTRENGTHS:\nA clear articulation of the problem and an elegant algorithm to solve it. Good analysis of the algorithm so that a reader can follow along. High probability bounds that do not average over the randomness in the graph generation process that is typical of results in this area.\n\nWEAKNESSES:\nLack of experimental evidence. Given one of the core contributions of the paper is an elegant algorithm, presenting synthetic simulations will substantiate the contributions of the paper.\n\nQUESTIONS:\nCan the authors substantiate the insights using synthetic simulations? For example, it will be nice if the authors can\n\n-  \"visualize\" the flow of information in the algorithm, \n-  the impact of heterogeneity to the resulting regret\n-  Impact of the graph generation process on regret. For example complete graphs is the easiest while a ring graph is the hardest? Perhaps the graph spectrum plays a role in the regret? \n\nOf-course these are a lot of changes, but will appreciate intuition/insights on these questions and will make the paper more complete.\n\n***EDIT Post Rebuttal:***\n--\n I thank the authors for the thorough response and would encourage them to add it to the camera ready. I would further propose to add two related references to the additional list of references. \n\nSankararaman et.al. Social Learning in Multi Armed Bandits, SIGMETRICS 2019\n\nChawla et.al. The Gossiping Insert-Eliminate Algorithm for multi-agent Bandits, AISTATS 2020\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a multiplayer graph bandit setting. In particular, they consider  heterogenous rewards in which the mean rewards for each agent are different, and are allowed to be both sub-gaussian and sub-exponential. They also consider random graphs between agents, which dictates which agents are allowed to communicate their rewards with each other. This graph structure is allowed to vary with time. They consider two types of graph generations as well as two types fo reward families for a total of 4 settings. \n\nThey propose an algorithm with the following components (1) graph generation, (2) DrFed-UCB (burn in period), and (3) DrFed-UCB (learning period). \n\nTheir regret analysis achieves optimal instance dependent regret on the order of log T with high probability.\n\nSTRENGTHS:\nThis paper considers a setting that is very general and interesting as well. For instance rewards can vary as well as the communication graphs which are generally assumed to be fixed. Furthermore, they allow for heavy tailed reward dsitributions where as standard bandit literature only considers subgaussian rewards. Their results seem to be very strong for this rather general setting.\n\nWEAKNESSES:\nI think the organization of the settings as well as the key results can be better (Maybe each setting gets its own theorem latex environment). Moreover, I'd use section 3.2 to list the regret bounds, and outline the analysis in a section following that (theorem 1 can go in that section). Kind of related to this point I think the statements of theorem 2 , theorem 3, theorem 4 are too detailed. Maybe just state the regret bound and define the constants in paragraph below it or in the proof.\n\nQUESTIONS:\nI'm not sure I understand S_1, S_2, and S_3. This is described in the first paragraph of sction 3.2 and it's worded quite confusingly. for isntance I'm not sure what M is referring to in describing Seettings 1.1, 1.2, 2.1, and 2.2. Also ,whcih setting is theorem 4 for? S_1, S_2 or S_3?\n\nLIMITATIONS:\nThe graph generation is limited to two distributions (ER and uniform). Furthermore, their algorithm can be costly in communication.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-24"}