SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mengfan Xu<sup>1</sup> Diego Klabjan<sup>1</sup>
$^{1}$ Department of Industrial Engineering and Management Sciences, Northwestern University MengfanXu2023@u.northwestern.edu, d-klabjan@northwestern.edu
We study a decentralized multi-agent multi-armed bandit problem in which multiple clients are connected by time dependent random graphs provided by an environment. The reward distributions of each arm vary across clients and rewards are generated independently over time by an environment based on distributions that include both sub-exponential and sub-Gaussian distributions. Each client pulls an arm and communicates with neighbors based on the graph provided by the environment. The goal is to minimize the overall regret of the entire system through collaborations. To this end, we introduce a novel algorithmic framework, which first provides robust simulation methods for generating random graphs using rapidly mixing Markov chains or the random graph model, and then combines an averaging-based consensus approach with a newly proposed weighting technique and the upper confidence bound to deliver a UCB-type solution. Our algorithms account for the randomness in the graphs, removing the conventional doubly stochasticity assumption, and only require the knowledge of the number of clients at initialization. We derive optimal instance-dependent regret upper bounds of order  $\log T$  in both sub-Gaussian and sub-exponential environments, and a nearly optimal mean-gap independent regret upper bound of order  $\sqrt{T}\log T$  up to a  $\log T$  factor. Importantly, our regret bounds hold with high probability and capture graph randomness, whereas prior works consider expected regret under assumptions and require more stringent reward distributions.
Multi-armed Bandit (MAB) [Auer et al., 2002a,b] is an online sequential decision-making process that balances exploration and exploitation while given partial information. In this process, a single player (agent, client) aims to maximize a cumulative reward or, equivalently, minimize the cumulative loss, known as regret, by pulling an arm and observing the reward of that arm at each time step. The two variants of MAB are adversarial and stochastic MAB, depending on whether rewards are chosen arbitrarily or follow a time-invariant distribution, respectively. Recently, motivated by the development of federated learning [McMahan et al., 2017], multi-agent stochastic multi-armed bandit has been drawing increasing attention (commonly referred to as multi-agent MAB). In this variant, multiple clients collaboratively work with multiple stochastic MABs to maximize the overall performance of the entire system. Likewise, regret is an important performance measure, which is the difference between the cumulative reward of always pulling the global optimal arm by all clients and the actual cumulative reward gained by the clients at the end of the game, where global optimality is defined with respect to the average expected reward values of arms across clients. Thereafter, the question for each client to answer is essentially how to guarantee an optimal regret with limited observations of arms and insufficient information of other clients. Assuming the existence of a central server, also known as the controller, [Bistritz and Leshem, 2018, Zhu et al., 2021b, Huang et al., 2021, Mitra et al., 2021, Reda et al., 2022, Yan et al., 2022], allow a controller-client framework where the controller integrates and distributes the inputs from and to clients, adequately addressing the challenge
posed by the lack of information of other clients. However, this centralization implicitly requires all clients to communicate with one another through the central server and may fail to include common networks with graph structures where clients perform only pair-wise communications within the neighborhoods on the given graphs. Non-complete graphs capture the reality of failed communication links. Removing the centralization assumption leads to a decentralized multi-agent MAB problem, which is a challenging but attracting direction as it connects the bandit problem and graph theory, and precludes traditional centralized processing.
In the field of decentralized multi-agent MAB, it is commonly assumed that the mean reward value of an arm for different clients is the same, or equivalently, homogeneous. This assumption is encountered in [Landgren et al., 2016a,b, 2021, Zhu et al., 2020, Martinez-Rubio et al., 2019, Agarwal et al., 2022, Wang et al., 2022, 2020, Li and Song, 2022, Sankararaman et al., 2019, Chawla et al., 2020]. However, this assumption may not always hold in practical scenarios. In recent years, there has been an increasing emphasis on heterogeneous reward settings, where clients can retain different mean values for the rewards of the same arm. The transition to heterogeneous reward settings presents additional technical challenges. Clients are unable to infer the global optimal arms without sequential communications regarding the rewards of the same arm at other clients. Such communications, however, are limited by the partially observed rewards, as other clients may not pull the same arm, and constrained by the underlying graph structure. We study the heterogeneous setting with time varying graphs.
Traditionally, rewards are assumed to be sub-Gaussian distributed. However, there has been a recent focus on MAB with heavy-tailed reward distributions. This presents a non-trivial challenge as it is harder to concentrate reward observations in sublinear time compared to the light-tailed counterpart [Tao et al., 2022]. In the work of [Jia et al., 2021], sub-exponential rewards are considered and analyzed in the single-agent MAB setting with newly proposed upper confidence bounds. Meanwhile, for multi-agent MAB, heavy-tailed distributions are examined in a homogeneous setting in [Dubey et al., 2020]. However, the heterogeneous setting studied herein has not yet been formulated or analyzed, posing more challenges compared to the homogeneous setting, as discussed earlier.
Besides rewards, the underlying graph assumptions are essential to the decentralized multi-agent MAB problem, as increased communication among clients leads to better identification of global optimal arms and smaller regret. The existing works [Sankararaman et al., 2019, Chawla et al., 2020] relate regret with graph structures and characterize the dependency of regret on the graph complexity with respect to conductance. When considering two special cases, [Chawla et al., 2020] demonstrates the theoretical improvement achieved by circular ring graphs compared to complete graphs, and [Li and Song, 2022] numerically shows that the circular ring graph presents the most challenging scenario with the largest regret, while the complete graph is the simplest. There are two types of graphs from a time perspective: time-invariant graphs, which remain constant over time, and time-varying graphs, which depend on time steps and are more challenging but more general. Assumptions on time-invariant graphs include complete graphs [Wang et al., 2021] where all clients can communicate, regular graphs [Jiang and Cheng, 2023] where each client has the same number of neighbors, and connected graphs under the doubly stochasticity assumption [Zhu et al., 2020, 2021a,b]. Independently from our work, recent work [Zhu and Liu, 2023] has focused on time-varying  $B$ -connected graphs, where the composition of every  $l$  consecutive graphs is a strongly connected graph. However, their doubly stochasticity assumption, where all elements of edge probability also called weight matrices are uniformly bounded by a positive constant, can be violated in several cases. Additionally, their graphs may be strongly correlated to meet the connectivity condition when  $l > 1$ , which may not always hold in practice. No research has been conducted on time-varying graphs with only connectivity constraints or without constraints on connectivity. Additionally, current time-varying graphs do not provide insight into how the graphs change over time. As the graphs are generated by the environment, similar to the generation of rewards, it remained unexplored considering random graphs in an i.i.d manner, such as random edge failures or random realizations as pointed out for future research in [Martínez-Rubio et al., 2019]. We also address this situation.
Traditionally, random graphs have often been formulated using the Erdős–Rényi (E-R) model, which has been widely adopted in various research domains. The model, described by  $G(M, c)$ , consists of  $M$  vertices with each pair of vertices being connected with probability  $c$ . Notably, the E-R model is 1) not necessarily connected and 2) stochastic that allows for random edge failures, and has found applications in mean-field game [Delarue, 2017] and majority vote settings [Lima et al., 2008]. Though it has only been used in numerical experiments for the decentralized multi-agent MAB
setting with homogeneous rewards [Dubey et al., 2020], the theoretical study of this model in this context remained unexplored until this work, let alone with heterogeneous rewards and time-varying graphs. Alternatively, one can consider all connected graphs (there are exponentially many of them), and the environment can randomly sample a connected graph and produce i.i.d. samples of such random connected graphs. This approach mimics the behavior of stochastic rewards and allows the environment to exhaust the sample space of connected graphs independently, without the doubly stochasticity assumption, which, however, has not yet been studied and it is also addressed herein.
For the multi-agent MAB framework, methods in MAB are a natural extension. [Zhu et al., 2021b] adapt the UCB algorithm to the multi-agent setting. This algorithm uses weighted averages to achieve consensus among clients and is shown to have a regret of order  $\log T$  for time-invariant graphs. A follow-up study in [Zhu and Liu, 2023] re-analyzes this algorithm for time-varying  $B$ -connected graphs under the aforementioned assumptions under the doubly stochasticity assumption by adding an additional term compared to UCB. An effective UCB-based method for random graphs without doubly stochasticity assumption and for sub-exponential distributed rewards remained unexplored.
This paper presents a novel contribution to the decentralized multi-agent MAB problem by studying both heterogeneous rewards and time-varying random graphs, where the distributions of rewards and graphs are independent of time. To the best of our knowledge, this is the first work to consider this problem and to investigate it with heavy-tailed reward distributions. Specifically, the paper investigates 1) heterogeneous sub-exponential and sub-Gaussian distributed rewards and 2) random graphs including the possibly disconnected E-R model and random connected graphs, and applies them to the decentralized multi-agent MAB framework. This work bridges the gap between large-deviation theories for sub-exponential distributions and multi-agent MAB with heterogeneous rewards, and the gap between random graphs and decentralized multi-agent MAB.
To this end, we propose a brand new algorithmic framework consisting of three main components: graph generation, DrFed-UCB: burn-in period, and DrFed-UCB: learning period. For the learning period, we modify the algorithm by [Zhu et al., 2021b] by introducing new UCB quantities that are consistent with the conventional UCB algorithm and generalize to sub-exponential settings. We also introduce a newly proposed stopping time and a new weight matrix without the doubly stochasticity assumption to leverage more information in random graphs. A burn-in period is crucial in estimating the graph distribution and initializing the weight matrix. We embed and analyze techniques from random graphs since the number of connected graphs is exponentially large in the number of vertices, and directly sampling such a graph is an NP-hard problem. In particular, we use the Metropolis-Hastings method with rapidly mixing Markov chains, as proposed in [Gray et al., 2019], to approximately generate random connected graphs in polynomial time. We additionally demonstrate its theoretical convergence rate, making it feasible to consider random connected graphs in the era of large-scale inference.
We present comprehensive analyses of the regret of the proposed algorithm, using the same regret definition as in existing literature. Firstly, we show that algorithm DrFed-UCB achieves optimal instance-dependent regret upper bounds of order  $\log T$  with high probability, in both sub-Gaussian and sub-exponential settings, consistent with prior works. We add that although both [Zhu et al., 2020] and our analyses use the UCB framework, the important algorithmic steps are different and thus also the analyses. Secondly, we demonstrate that with high probability, the regret is universally upper bounded by  $O(\sqrt{T}\log T)$  in sub-exponential settings, including sub-Gaussian settings. This upper bound matches the upper and lower bounds in single-agent settings up to a  $\log T$  factor, establishing its tightness.
The paper is organized as follows. We first introduce the notations used throughout the paper, present the problem formulation, and propose algorithms for solving the problem. Following that, we provide theoretical results on the regret of the proposed algorithm in various settings.
2 Problem Formulation and Methodologies
Throughout, we consider a decentralized system with  $M$  clients that are labeled as nodes  $1,2,\ldots ,M$  on a time-varying network, which is described by an undirected graph  $G_{t}$  for  $1\leq t\leq T$  where parameter  $T$  denotes the time horizon of the problem. Formally, at time step  $t$ ,  $G_{t} = (V,E_{t})$  where  $V = \{1,2,\dots ,M\}$  and  $E_{t}$  denotes the edge set consisting of pair-wise nodes and representing the
neighborhood information in  $G_{t}$ . The neighbor set  $\mathcal{N}_m(t)$  include all neighbors of client  $m$  based on  $G_{t}$ . Equivalently, the graph  $G_{t}$  can be represented by the adjacency matrix  $(X_{i,j}^{t})_{1\leq i,j\leq M}$  where the element  $X_{i,j}^{t} = 1$  if there is an edge between clients  $i$  and  $j$  and  $X_{i,j}^{t} = 0$  otherwise. We let  $X_{i,i} = 1$  for any  $1\leq i\leq M$ . With this notation at hand, we define the empirical graph (adjacency matrix)  $P_{t}$  as  $P_{t} = \frac{(\sum_{s = 1}^{t}X_{i,j}^{s})_{1\leq i,j\leq M}}{t}$ . It is worth emphasizing that 1) the matrix  $P_{t}$  is not necessarily doubly stochastic, 2) the matrix captures more information about  $G_{t}$  than the prior works based on  $|\mathcal{N}_m(t)|$ , and 3) each client  $m$  only knows the  $m$ -th row of  $P_{t}$  without knowledge of  $G_{t}$ , i.e.  $\{P_t(m,j)\}_{j}$  are known to client  $m$ , while  $\{P_t(k,j)\}_{j}$  for  $k\neq m$  are always unknown. Let us denote the set of all possible connected graphs on  $M$  nodes as  $\mathcal{G}_{\mathcal{M}}$ .
We next consider the bandit problems faced by the clients. In the MAB setting, the environment generates rewards. Likewise, we again use the term, the environment, to represent the source of graphs  $G_{t}$  and rewards  $r_{i}^{m}(t)$  in the decentralized multi-agent MAB setting. Formally, there are  $K$  arms faced by each client. At each time step  $t$ , for each client  $1 \leq m \leq M$ , let the reward of arm  $1 \leq i \leq K$  be  $r_{i}^{m}(t)$ , which is i.i.d. distributed across time with the mean value  $\mu_{i}^{m}$ , and is drawn independently across the clients. Here we consider a heterogeneous setting where  $\mu_{i}^{m}$  is not necessarily the same as  $\mu_{i}^{j}$  for  $m \neq j$ . At each time step  $t$ , client  $m$  pulls an arm  $a_{m}^{t}$ , only observes the reward of that arm  $r_{a_{m}^{t}}^{m}(t)$  from the environment, and exchanges information with neighbors in  $G_{t}$  given by the environment. In other words, two clients communicate only when there is an edge between them.
By taking the average over clients as in the existing literature, we define the global reward of arm  $i$  at each time step  $t$  as  $r_i(t) = \frac{1}{M}\sum_{m=1}^{M}r_i^m(t)$  and the subsequent expected value of the global reward as  $\mu_i = \frac{1}{M}\sum_{m=1}^{M}\mu_i^m$ . We define the global optimal arm as  $i^* = \arg \max_i\mu_i$  and arm  $i \neq i^*$  is called global sub-optimal. Let  $\Delta_i = \mu_{i^*} - \mu_i$  be the sub-optimality gap. This enables us to quantify the regret of the action sequence (policy)  $\{a_m^t\}_{1 \leq t \leq M}^{1 \leq t \leq T}$  as follows. Ideally, clients would like to pull arm  $i^*$  if knowledge of  $\{\mu_i\}_i$  were available. Given the partially observed rewards due to bandits (dimension  $i$ ) and limited accesses to information from other clients (dimension  $m$ ), the regret is defined as  $R_T = T\mu_{i^*} - \frac{1}{M}\sum_{t=1}^{T}\sum_{m=1}^{M}\mu_{a_m^t}^m$  which measures the difference of the cumulative expected reward between the global optimal arm and the action sequence. The main objective of this paper is to develop theoretically robust solutions to minimize  $R_T$  for clients operating on time-varying random graphs that are vulnerable to random communication failures, which only require knowledge of  $M$ .
2.2 Algorithms
In this section, we introduce a new algorithmic framework that incorporates two graph generation algorithms, one for the E-R model and the other for uniformly distributed connected graphs. More importantly, the framework includes a UCB-variant algorithm that runs a learning period after a burn-in period, which is commonly referred to as a warm-up phase in statistical procedures.
2.2.1 Graph Generation
We investigate two types of graph dynamics as follows, for which we propose simulation methods that enable us to generate and analyze the resulting random graphs.
E-R random graphs At each time step  $t$ , the adjacency matrix of graph  $G_{t}$  is generated by the environment by element-wise sampling  $X_{i,j}^{t}$  according to a Bernoulli distribution. Specifically,  $X_{i,j}^{t}$  follows a Bernoulli distribution with parameter  $c$ .
Uniformly distributed connected graphs At each time step  $t$ , the random graph  $G_{t}$  is generated by the environment by uniformly sampling a graph from the sample space of all connected graphs  $\mathcal{G}_M$ , which yields the adjacency matrix  $(X_{i,j}^{t})_{1\leq i\neq j\leq M}$  corresponding to  $G_{t}$ . Generating uniformly distributed connected graphs is presented in Algorithm 3 as in Appendix. It is computationally infeasible to exhaust the sample space  $\mathcal{G}_M$  since the number of connected graphs is exponentially large. To this end, we import the Metropolis-Hastings method in [Gray et al., 2019] and leverage rapidly mixing Markov chains. Remarkably, by adapting the algorithm into our setting which yields a new algorithm, we construct a Markov chain that converges to the target distribution after a finite number of burn-in steps. This essentially traverses graphs in  $\mathcal{G}_M$  through step-wise transitions, with a complexity of  $O(M^2)$  from previous states. More precisely, at time step  $s$  with connected graph  $G_{s}$ , we randomly choose a pair of nodes and check whether it exists in the edge set. If this is the case, we
remove the edge from  $G_{s}$  and check whether the remaining graph is connected, and only accept the graph as  $G_{s+1}$  in the connected case. If the edge does not exist in the edge set, we add it to  $G_{s}$  and get  $G_{s+1}$ . In this setting, let  $c = c(M)$  be the number of times an edge is present among all connected graphs divided by the total number of connected graphs. It is known that  $c = 2\frac{\log M}{M-1}$  (1) [Trevisan]. The distribution of  $G_{s}$  eventually converges to the uniform distribution in  $\mathcal{G}_{M}$ . The formal statements are in Appendix.
2.2.2 Main Algorithm
In the following, we present the proposed algorithm, DrFed-UCB, which comprises of a burn-in period and a learning period described in Algorithm 1 and Algorithm 2, respectively.
We start by introducing the variables used in the algorithm with respect to client  $m$ . We use  $\tilde{\mu}_i^m(t), n_{m,i}(t)$  to denote reward estimators and counters based on client  $m$ 's own pulls of arm  $i$ , respectively, and use  $\tilde{\mu}_i^m, N_{m,i}(t)$  to denote reward estimators and counters based on the network-wide pulls of arm  $i$ , respectively. By network-wide, we refer to the clients in  $\mathcal{N}_m(t)$ . Denote the stopping time for the filtration  $\{G_s\}_{s=1}^t$  as  $h_{m,j}^t = \max_{s \leq t} \{(m,j) \in E_s\}$ ; it represents the most recent communication between clients  $m$  and  $j$ . The weights for the network-wide and local estimators are  $P_t'(m,j)$  and  $d_{m,t}$  defined later, respectively.
There are two stages in Algorithm 1 where  $t \leq L$  as follows. For the first  $\tau_{1}$  steps, the environment generates graphs based on one of the aforementioned graph generation algorithms to arrive at the steady state, while client  $m$  pulls arms randomly and updates local estimators  $\{\bar{\mu}_i^m, n_i^m\}_i$ . Afterwards, the environment generates the graph  $G_t$  that follows the distribution of interest, while client  $m$  updates  $\{\bar{\mu}_i^m, n_i^m\}_i$  and row  $m$  of  $P_t$  by exchanging information with its neighbors in the graph  $G_t$ . Note that client  $m$  does not have any global information about  $G_t$ . At the end of the burn-in period, client  $m$  computes the network-wide estimator  $\tilde{\mu}_i^m$  by taking the weighted average of local estimators of other clients (including itself), where the weights are given by the  $m$ -th row of weight matrix  $P'$  and  $d$ , which depend on  $P$  and knowledge of  $M$  and satisfy  $\sum_{j=1}^{M} P_t'(m, j) + d_{m,t}M = 1$ .
Subsequently, we describe Algorithm 2 where  $t \geq L + 1$ . There are four phases in one iteration enumerated below in the order indicated. A flowchart of this algorithm is provided in Appendix.
UCB Given the estimators  $\tilde{\mu}_i^m (t),n_{m,i}(t),N_{m,i}(t),\bar{\mu}_i^m (t)$  , client  $m$  either randomly samples an arm or pulls the arm that maximizes the upper confidence bound using  $\tilde{\mu}_i^m (t),n_{m,i}(t)$  , depending on whether  $n_{m,i}(t)\leq N_{m,i}(t) - K$  holds for some arm  $i$  . This additional condition ensures consensus among clients regarding which arm to pull. The upper confidence bound  $F(m,i,t)$  is specified as  $F(m,i,t) = \sqrt{\frac{C_1\ln t}{n_{m,i}(t)}}$  and  $F(m,i,t) = \sqrt{\frac{C_1\ln T}{n_{m,i}(t)} + \frac{C_2\ln T}{n_{m,i}(t)}}$  in settings with sub-Gaussian and sub-exponential rewards, respectively. Constants  $C_1$  and  $C_2$  are determined in the analyses and they depend on  $\sigma$  which is an upper bound of standard deviations of the reward values (it is formally defined later).
Environment and client interaction After client  $m$  pulls arm  $a_{m}^{t}$ , the environment sends the reward  $r_{m, a_{m}^{t}}^{t}$  and the neighbor set  $\mathcal{N}_{m}(t)$  in  $G_{t}$  to client  $m$ . Client  $m$  does not know the whole  $G_{t}$  and obtains only the neighbor set  $\mathcal{N}_{m}(t)$ .
Transmission Client  $m$  sends the maintained local and network-wide estimators  $\{\tilde{\mu}_i^m (t),\tilde{\mu}_i^m (t),n_{m,i}(t),N_{m,i}(t)\}_{i}$  to all clients in  $\mathcal{N}_m(t)$  and receives the ones from them.
Update estimators At the end of an iteration, client  $m$  first updates the  $m$ -th row of matrix  $P_{t}$ . Subsequently, client  $m$  updates the quantities  $\{\tilde{\mu}_i^m (t),\tilde{\mu}_i^m (t),n_{m,i}(t),N_{m,i}(t)\}_{1\leq i\leq K}$  adhering to:  $t_{m,j} = \max_{s\geq \tau_1}\{(m,j)\in E_s\}$  and 0 if such an  $s$  does not exist
$$
n _ {m, i} (t + 1) = n _ {m, i} (t) + \mathbb {1} _ {a _ {m} ^ {t} = i}, N _ {m, i} (t + 1) = \max  \{n _ {m, i} (t + 1), \hat {\tilde {N}} _ {i, j} ^ {m} (t), j \in \mathcal {N} _ {m} (t) \} \tag {2}
$$