SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper, we consider a decentralized multi-agent multi-armed bandit problem in a fully stochastic environment that generates time-varying random graphs and heterogeneous rewards following subgaussian and sub-exponential distributions, which has not yet been studied in existing works. To the best of our knowledge, this is the first theoretical work on random graphs including the E-R model and random connected graphs, and the first work on heterogeneous rewards with heavy tailed rewards. To tackle this problem, we develop a series of new algorithms, which first simulate graphs of interest, then run a warm-up phase to handle graph dynamics and initialization, and proceed to the learning period using a combination of upper confidence bounds (UCB) with a consensus mechanism that relies on newly proposed weight matrices and updates, and using a stopping time to handle randomly delayed feedback. Our technical novelties in the results and the analyses are as follows. We prove high probability instance-dependent regret bounds of the order of  $\log T$  in both sub-gaussian and sub-exponential cases, consistent with the regret bound in the existing works that only consider the expected regret. Moreover, we establish a nearly tight instance-free regret bound of order  $\sqrt{T}\log T$  for both sub-exponential and sub-gaussian distributions, up to a  $\log T$  factor. We leverage probabilistic graphical methods on random graphs and draw on theories related to rapidly mixing Markov chains, which allows us to eliminate the doubly stochasticity assumption through new weight matrices and a stopping time. We construct new network-wide estimators and invent new concentration inequalities for them, and subsequently incorporate the seminal UCB algorithm into this distributed setting. A discussion on future work is referred to Appendix.
We deeply appreciate the NeurIPS anonymous reviewers and the meta-reviewer for their valuable and insightful suggestions and discussions. This final version of the paper has been made possible through their great help. Additionally, we are much obliged to the authors of the papers [Chawla et al., 2020, Zhu et al., 2021b], especially Ronshee Chawla from [Chawla et al., 2020] and Zhaowei Zhu from [Zhu et al., 2021b], for promptly sharing the code of their algorithms, which has helped us to run the benchmarks presented in this work. Their gesture definitely improves the advancement of the field.
M. Agarwal, V. Aggarwal, and K. Azizzadenesheli. Multi-agent multi-armed bandits with limited communication. The Journal of Machine Learning Research, 23(1):9529–9552, 2022.
P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2-3):235-256, 2002a.
P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002b.
I. Bistritz and A. Leshem. Distributed multi-player bandits-a game of thrones approach. Advances in Neural Information Processing Systems, 31, 2018.
R. Chawla, A. Sankararaman, A. Ganesh, and S. Shakkottai. The gossiping insert-eliminate algorithm for multi-agent bandits. In International conference on artificial intelligence and statistics, pages 3471-3481. PMLR, 2020.
S. Chib and E. Greenberg. Understanding the metropolis-hastings algorithm. The american statistician, 49(4):327-335, 1995.
F. Delarue. Mean field games: A toy model on an ergős-renyi graph. ESAIM: Proceedings and Surveys, 60:1-26, 2017.
A. Dubey et al. Cooperative multi-agent bandits with heavy tails. In International conference on machine learning, pages 2730-2739. PMLR, 2020.
J. C. Duchi, A. Agarwal, and M. J. Wainwright. Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Transactions on Automatic control, 57(3): 592-606, 2011.
J. Fan, B. Jiang, and Q. Sun. Hoeffding's inequality for general markov chains and its applications to statistical learning. The Journal of Machine Learning Research, 22(1):6185-6219, 2021.
C. Gray, L. Mitchell, and M. Roughan. Generating connected random graphs. Journal of Complex Networks, 7(6):896-912, 2019.
R. Huang, W. Wu, J. Yang, and C. Shen. Federated linear contextual bandits. Advances in neural information processing systems, 34:27057-27068, 2021.
H. Jia, C. Shi, and S. Shen. Multi-armed bandit with sub-exponential rewards. Operations Research Letters, 49(5):728-733, 2021.
F. Jiang and H. Cheng. Multi-agent bandit with agent-dependent expected rewards. Swarm Intelligence, pages 1-33, 2023.
P. Landgren, V. Srivastava, and N. E. Leonard. On distributed cooperative decision-making in multiarmed bandits. In 2016 European Control Conference (ECC), pages 243-248. IEEE, 2016a.
P. Landgren, V. Srivastava, and N. E. Leonard. Distributed cooperative decision-making in multiarmed bandits: Frequentist and bayesian algorithms. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 167-172. IEEE, 2016b.
P. Landgren, V. Srivastava, and N. E. Leonard. Distributed cooperative decision making in multi-agent multi-armed bandits. Automatica, 125:109445, 2021.
J. Zhu, R. Sandhu, and J. Liu. A distributed algorithm for sequential decision making in multi-armed bandit with homogeneous rewards. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 3078–3083. IEEE, 2020.
J. Zhu, E. Mulle, C. S. Smith, and J. Liu. Decentralized multi-armed bandit can outperform classic upper confidence bound. arXiv preprint arXiv:2111.10933, 2021a.
Z. Zhu, J. Zhu, J. Liu, and Y. Liu. Federated bandit: A gossiping approach. In Abstract Proceedings of the 2021 ACM SIGMETRICS/International Conference on Measurement and Modeling of Computer Systems, pages 3-4, 2021b.
A Additional Background Work
In the context of bandits with heavy-tailed distributed rewards, the UCB algorithm continues to play a significant role. [Dubey et al., 2020] are the first to consider the multi-agent MAB setting with homogeneous heavy-tailed rewards. They develop a UCB-based algorithm with an instance-dependent regret bound of order  $\log T$ . They achieve this by adopting larger upper confidence bounds and finding cliques of vertices, even though the graphs are time-invariant and known to clients. In a separate line of work, [Jia et al., 2021] consider the single-agent MAB setting with sub-exponential rewards, and propose a UCB-based algorithm that enlarges or pretrains the upper confidence bounds, achieving a mean-gap independent regret bound of order  $\sqrt{T\log T}$ . We extend this technique to the decentralized multi-agent MAB setting with heterogeneous sub-exponential rewards, using a gossiping approach, and establish both an optimal instance-dependent regret bound of  $O(\log T)$  and a nearly optimal mean-gap independent regret bound of  $O(\sqrt{T}\log T)$ , up to a  $\log T$  factor.
Our work draws on the classical literature on random graphs. From the perspective of generating random connected graphs, we build upon a numerically efficient algorithm introduced in [Gray et al., 2019], which is based on the Metropolis-Hastings algorithm [Chib and Greenberg, 1995], despite its lack of finite convergence rate for non-sparse graphs. We follow their algorithm and, in addition, provide a new analysis on the convergence rate and mixing time of the underlying Markov chain. In terms of the E-R model, it has been thoroughly examined in various areas, such as mean-field games [Delarue, 2017] and majority vote settings [Lima et al., 2008]. However, these random graphs have not yet been applied to the decentralized multi-agent MAB setting that is partially determined by the underlying graphs. Our formulation and analyses bridge this gap, providing insights into the dynamics of decentralized multi-agent MAB in the context of random graphs.
B Future work
Recent advancements have been made in reducing communication costs with respect to the dependency in multi-agent MAB with homogeneous rewards (in the generalized linear bandit setting [Li and Wang, 2022], the ground truth of the unknown parameter is the same for all clients), such as achieving  $O(\sqrt{T} M^2)$  in [Li and Wang, 2022] for centralized settings or  $O(M^3 \log T)$  through Global Information Synchronization (GIS) communication protocols assuming time-invariant graphs in [Li and Song, 2022]. Likewise, [Sankararaman et al., 2019, Chawla et al., 2020] improve the communication cost of order  $\log T$  or  $o(T)$  through asynchronous communication protocols and balancing the trade-off between regret and communication cost. More recently, [Wang et al., 2022] establish a novel communication protocol, TCOM, which is of order  $\log \log T$  by means of concentrating communication around sub-optimal arms and performing aggregation of estimators across time steps. Furthermore, [Wang et al., 2020] develops a new leader-follower communication protocol, which selects a leader that communicates to the followers. Here the communication cost is independent of  $T$  which is much smaller. The incorporation of random graph structures and heterogeneous rewards introduces its own complexities, which poses challenges to reductions in communication costs. These great advancements introduce a promising direction for communication efficiency as a next step within the context herein.
C Details on numerical experiments in Section 4
We report the experimental details in Section 4, including both benchmarking and regret properties of the algorithms. The implementation details of the experiments are as follows, including the data generation, benchmarks, and the evaluation metrics.
The process of data generation involves both reward generation and graph generation. First we generate different numbers of arms and clients, denoted as  $K$  and  $M$ , respectively. Specifically, we generate rewards using the Bernoulli distribution in the sub-Gaussian distribution family, varying the mean values  $\mu_i^m$  by introducing multiple levels of heterogeneity denoted as  $h = \max_{i,j,m} |\mu_i^m - \mu_j|$  and then for each arm  $k$ , partitioning the range  $[0.1, 0.1 + (k + 1) \cdot h / K]$  into  $M$  intervals. In terms of graph generation, we generate E-R models with varying values of  $c$ , to capture graph complexity. Specifically, for the benchmarking experiment with time-invariant graphs, we set  $K = 2$ ,  $M = 5$ ,  $h = 0.1$ ,  $c = 1$ , i.e. complete graphs. For the benchmarking experiment with time-varying graphs, we set  $K = 2$ ,  $M = 5$ ,  $h = 0.1$ ,  $c = 0.9$ . For the regret experiments, the parameters are  $h \in \{0.1, 0.2, 0.3\}$ ,  $M \in \{5, 8, 12\}$ ,  $c \in \{0.2, 0.5, 0.9, 1\}$ , and  $K \in \{2, 3, 4\}$ . We selected the least positive number of arms  $K = 2$  to keep computational times low and  $M = 5$  to have small graphs but still a variety of them.
We compare the new method DrFed-UCB with the classical methods, such as the Gossiping InsertEliminate algorithm (GoSInE) in [Chawla et al., 2020] which focuses on deterministic graphs and sub-Gaussian rewards and motivated our work. We also include the Gossip UCB algorithm (Gossip_UCB) [Zhu et al., 2021b] as a benchmark. Meanwhile, in terms of time-varying graphs, we implement the algorithm, Distributed UCB (Dist_UCB) in [Zhu and Liu, 2023] that has been developed for time-varying graphs, and compare our algorithm to this benchmark.
Evaluation The evaluation metric is the regret measure as defined in Section 4. More specifically, for the experiments, we use the average regret over 50 runs for each benchmark and also report the  $95\%$  confidence intervals across the 50 runs. With respect to the communication cost as another performance measure, it is computed explicitly. Additionally, the runtime can provide insights into the time complexity of the models.
Benchmark comparison results The results for time-invariant and time-varying graphs are presented in Figure 1 (a) and Figure 1 (b), respectively. The x-axis represents time steps, while the y-axis shows the average regret up to that time step. Figure 1 (a) demonstrates that DrFed-UCB exhibits the smallest average regret among all methods in time-invariant graphs, with significant improvements. More precisely, with respect to the Area Under the Curve (AUC) of the regret curve, the improvements of DrFed-UCB over GoSInE, Gossip_UCB, and Dist_UCB are  $132\%$ ,  $158\%$ , and  $128\%$ , respectively, showcasing the regret improvement of the newly proposed algorithm compared to the benchmarks. Notably, both Dist_UCB and DrFed-UCB result in larger variances, primarily observed in Dist_UCB. This phenomenon may be attributed to the communication mechanisms designed for time-varying graphs. In Figure 1 (b), we observe that our regret is notably smaller compared to Dist_UCB in settings with time-varying graphs. Specifically, the AUC of Dist_UCB is
(a) time-invariant graphs
(b) time-varying graphs
Figure 1: The regret of different methods in settings with both time-invariant and time-varying graphs
(a)  $h$
(b)  $M$
(c)  $c$
Figure 2: The regret of the proposed algorithm in problem settings with different parameters
(d)  $K$
96.6% larger than that of our regret curve, which implies the significant improvement in this setting with time-varying graphs. Furthermore, we perform a time complexity comparison, revealing that DrFed-UCB and GoSInE are approximately six times faster than Dist_UCB. Lastly, communication cost is directly computed by the total number of communication rounds and follows an explicit formula. Specifically, the communication costs of DrFed-UCB, Gossip_UCB, and Dist_UCB are of order  $T$ , whereas GoSInE exhibits only  $o(T)$ , suggesting a potential direction for optimizing communication costs.
Regret dependency results Meanwhile, we illustrate how DrFed-UCB's regret depends on several factors: the number of clients  $(M)$ , the number of arms  $(K)$ , the Bernoulli parameter  $(c)$  for the E-R model, and heterogeneity measured by  $h$ . The regret metrics are presented as (a), (b), (c), and (d) in Figure 2, respectively. We observe that regret monotonically increases with the level of heterogeneity and the number of arms, while decreasing with connectivity, which is equivalent to an increase in graph complexity. However, this monotonic trend does not apply to the number of clients. This is due to the following considerations. On one hand, a large  $M$  implies a greater number of incident edges
Figure 3: Flowchart of Algorithm 2
of each client, providing more global information access and potentially leading to smaller regret. On the other hand, a large  $M$  also weakens the Chernoff-Hoeffding inequality for clients transmitting information, which might result in larger regret.
D Algorithms and Tables in Section 3
The algorithm for generating random connected graphs is presented in Algorithm 3 as follows.
Algorithm 3: Generate a uniformly distributed connected graph
Initialization: Let  $\tau_{1}$  be given; Generate a random graph  $G^{init}$  by selecting each edge with probability  $\frac{1}{2}$ ; Connectivity: make  $G^{init}$  connected by adding the least many edges to get  $G_{0}$ ; for  $t = 0,1,2,\ldots ,\tau_{1}$  do Randomly sample an edge pair  $e = (i,j)$ ; Denote the edge set of  $G_{s}$  as  $E_{s}$ ; if  $e\in E_s$  then Remove  $e$  from  $E_{s}$  to get  $G_{s}^{\prime} = (V,E_{s}\backslash \{e\})$ ; if  $G_{s}^{\prime}$  is connected then  $G_{s + 1} = G_s^{\prime}$ ; else reject  $G_{s}^{\prime}$  and set  $G_{s + 1} = G_{s}$ ; end else  $G_{s + 1} = (V,E_s\cup \{e\})$ ; end
The flowchart of Algorithm 2 is presented in Figure 3 to illustrate the information flow in the algorithm. The table below displays the various settings we consider for the regret analysis.
Table 1: Settings
E-R	uniform	M	reward
s1	✓	any	sub-g
s2	✓	[1, 10]	sub-g
s3	✓	[11,∞)	sub-g
S1	✓	any	sub-e
S2	✓	[1, 10]	sub-e
S3	✓	[11,∞)	sub-e
E Remarks on the theoretical results in Section 3.2
E.1 Remarks on Theorem 2
Remark (The condition on the time horizon). Although the above regret bound holds for any  $T > L$ , the same bound applies to  $T \leq L$  as follows. Assuming  $T \leq L$ , we obtain  $E[R_T|A_{\epsilon, \delta}] \leq T \leq L$  where the first inequality is by noting that the rewards are within the range of [0, 1].
Remark (The upper bound on the expected regret). Theorem 2 states a high probability regret bound, while the expected regret is often considered in the existing literature. As a corollary of Theorem 2, we establish the upper bound on  $E[R_T]$  if  $\epsilon = \frac{\log T}{MT}$  as follows. Note that
$$
\begin{array}{l} E \left[ R _ {T} \right] = E \left[ R _ {T} \mid A _ {\epsilon , \delta} \right] P \left(A _ {\epsilon , \delta}\right) + E \left[ R _ {T} \mid A _ {\epsilon , \delta} ^ {c} \right] P \left(A _ {\epsilon , \delta} ^ {c}\right) \leq P \left(A _ {\epsilon , \delta}\right) \cdot E \left[ R _ {T} \mid A _ {\epsilon , \delta} \right] + T \cdot \left(1 - P \left(A _ {\epsilon , \delta}\right)\right) \\ \leq (1 - 7 \epsilon) (L + \sum_ {i \neq i ^ {*}} (\max  \left\{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 (K ^ {2} + M K) \right\} + \frac {2 \pi^ {2}}{3 P (A _ {\epsilon , \delta})} + K ^ {2} + (2 M - 1) K)) + 7 \epsilon T \\ \leq l _ {1} + l _ {2} \log T + \sum_ {i \neq i ^ {*}} (\max  \left\{\left[ \frac {4 C _ {1} \log T}{\Delta_ {i} ^ {2}} \right], 2 (K ^ {2} + M K) \right\} + \frac {2 \pi^ {2}}{3 (1 - 7 \epsilon)} + K ^ {2} + (2 M - 1) K) + 7 \frac {\log T}{M} \\ \end{array}
$$
where the first inequality uses  $E[R_T|A_{\epsilon, \delta}] \leq T$  and the second inequality follows by Theorem 1. Here  $l_1$  and  $l_2$  are constants depending on  $K, M, \delta, \min_{i \neq i^*} \Delta_i$ , and  $\lambda$ .
Remark (Comparison with previous work). A comparison to the regret bounds in the existing literature considering sub-Gaussian rewards is as follows. Our regret bounds are consistent with the prior works where the expected regret bounds are of order  $\log T$ . Note that the regret bounds in [Zhu and Liu, 2023] cannot be used here since the update rule and the settings are different. Their update rule and analyses cannot carry over to our settings, which explains why we invent these modifications and proofs. On the one hand, the time-varying graphs they consider do not include the  $E-R$  model, and we can find counter-examples where their doubly stochastic weight matrices  $W_{t}$  result in the divergence of  $W_{1} \cdot W_{2} \ldots W_{T}$ . This makes the key proof step invalid in our framework. On the other hand, their time-varying graphs include the connected graphs when  $l = 1$ , but they also make an additional assumption of doubly stochastic weight matrices, which is not applicable to regular graphs. Furthermore, they study an expected regret upper bound, while we prove a high probability regret bound that captures the dynamics in the random graphs. The graph assumptions in other works, however, are stronger, such as [Zhu et al., 2021b] consider time-invariant graphs and [Wang et al., 2021] assume graphs are complete [Perchet et al., 2016]. In contrast to some work that focuses on homogeneous rewards in decentralized multi-agent MAB, we derive regret bounds of the same order  $\log T$  in a heterogeneous setting. If we take a closer look at the coefficients in terms of  $K, M, \lambda, \Delta_{i}$ , our regret bound is determined by  $O(\max(K, \frac{1+\lambda}{1-\lambda}, \frac{1}{M^{2} \Delta_{i}}) \log T)$ . The work of [Zhu and Liu, 2023] arrives at  $O(\max\{\frac{\log T}{\Delta_{i}}, K_{1}, K_{2}\})$  where  $K_{1}, K_{2}$  are related to  $T$  without explicit formulas. Our regret is smaller when  $K \Delta_{i} \leq 1$  and  $\frac{1+\lambda}{1-\lambda} \Delta_{i} \leq 1$ , which can always hold by rescaling  $\Delta_{i}$ , i.e. for many cases we get substantial improvement.
E.2 Remarks on Theorem 4
Remark. Based on the expression of  $L_{1}$ , we obtain that  $L_{1}$  is independent of the sub-optimality gap  $\Delta_{i}$ . Meanwhile, we have  $C_1 = 8\sigma^2 \cdot 12\frac{M(M + 2)}{M^4}$  and  $C_2 = \frac{3}{2} C_1 = 12\sigma^2 \cdot 12\frac{M(M + 2)}{M^4}$ . This implies that the established regret bound in Theorem 4 does not rely on  $\Delta_{i}$  but does depend on  $\sigma^2$ . To this end, we use the terminology, mean-gap independent bounds, to only represent bounds having no dependency on  $\Delta_{i}$ , rather than instance independent that seems to be an overclaim in this case.
Remark (Comparison with previous work). For decentralized multi-agent MAB with homogeneous heavy-tailed rewards and time-invariant graphs, [Dubey et al., 2020] provide an instance-dependent regret bound of order  $\log T$ . In contrast, our regret bound has the same order for heterogeneous settings with random graphs, as shown in Theorem 3. Additionally, we provide a mean-gap independent regret bound as in Theorem 4. In the single-agent MAB setting, [Jia et al., 2021] consider sub-exponential rewards and derive a mean-gap independent regret upper bound of order  $\sqrt{T\log T}$ . Our regret bound of  $\sqrt{T}\log T$  is consistent with theirs, up to a logarithmic factor. Furthermore, our result is consistent with the regret lower bound as proposed in [Slivkins et al., 2019], up to a log  $T$  factor, indicating the tightness of our regret bound.
Remark. The discussion regarding the conditions on  $T$ , the expected regret  $E[R_T]$ , and the parameter specifications follow the same logic as those in Theorem 2. We omit the details here.
F Proof of results in Section 3.2
F.1 Lemmas and Propositions
Lemma 1. For any  $m, i, t > L$ , we have
$$
n _ {m, i} (t) \geq N _ {m, i} (t) - K (K + 2 M)
$$
Proof of Lemma 1. The proof is referred to [Zhu and Liu, 2023].
Lemma 2. For any  $m, i, t > L$ , if  $n_{m,i}(t) \geq 2(K^2 + KM + M)$  and graph  $G_t$  is connected, then we have
$$
n_{m,i}(t)\leq 2\min_{j}n_{j,i}(t).
$$
where the min is taken over all clients, not just the neighbors.
Proof of Lemma 2. The proof is referred to [Zhu and Liu, 2023].
Lemma 3 (Generalized Holder's inequality). For any  $r > 0$  and measurable functions  $h_i$  for  $i = 1, \ldots, n$ , if  $\sum_{i=1}^{n} \frac{1}{p_i} = \frac{1}{r}$ , then
$$
\left| \left| \Pi_ {i = 1} ^ {n} h _ {i} \right| \right| _ {r} \leq \Pi_ {i = 1} ^ {n} \left| \left| h _ {i} \right| \right| _ {p _ {k}}.
$$
The proof follows from the Young's inequality for products.
Lemma 4. Suppose that random variables  $X_{1}, X_{2}, \ldots, X_{n}$  are such that  $Y_{i} = E[(X_{1}, \ldots, X_{n}) | (X_{1}, \ldots, X_{i-1}, X_{i+1}, \ldots, X_{n})]$  are sub-Gaussian distributed with variance proxy  $\sigma_{1}, \sigma_{2}, \ldots, \sigma_{n}$ , respectively. Then the sum of these sub-Gaussian random variables,  $\sum_{i=1}^{n} X_{i}$ , is again sub-Gaussian with variance proxy  $(\sum_{i=1}^{n} \sigma_{i})^{2}$ .
Proof. First, without loss of generality, let us assume  $E[X_i] = 0$ . Otherwise, we can always construct a random variable  $X_{i} - E[X_{i}]$  which has the same variance proxy with a difference up to a constant.
Defining  $p_i = \frac{\sum_{k=1}^n \sigma_k}{\sigma_i}$  gives  $\sum_{i=1}^n \frac{1}{p_i} = 1$ . Let  $\mu$  be the distribution function of random vector  $(X_1, \ldots, X_n)$ . By specifying  $h_i(x) = \exp(\lambda x)$  and  $r = 1$ , we obtain that for any  $\lambda > 0$  we have