Title: Distributed Multi-Agent Bandits Over Erdős-Rényi Random Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: distributed stochastic bandit
Abstract: We study the distributed multi-agent multi-armed bandit problem with heterogeneous rewards over random communication graphs. Uniquely, at each time step $t$ agents communicate over a time-varying random graph $\mathcal{G}\_t$ generated by applying the Erdős–Rényi model to a fixed connected base graph $\mathcal{G}$ (for classical Erdos-Rényi graphs, $\mathcal{G}$ is a complete graph), where each potential edge in $\mathcal{G}$ is randomly and independently present with the link probability $p$. Notably, the resulting random graph is not necessarily connected at each time step. Each agent's arm rewards follow time-invariant distributions, and the reward distribution for the same arm may differ across agents. The goal is to minimize the cumulative expected regret relative to the global mean reward of each arm, defined as the average of that arm’s mean rewards across all agents. To this end, we propose a fully distributed algorithm that integrates the arm elimination strategy with the random gossip algorithm. We theoretically show that the regret upper bound is of order $\log T$ and is highly interpretable, where $T$ is the time horizon. It includes the optimal centralized regret $\mathcal O\left(\sum_{k: \Delta_k>0} \frac{\log T}{\Delta_k}\right)$ and an additional term $\mathcal O\left(\frac{N^2 \log T}{p \lambda_{N-1}(\operatorname{Lap}(\mathcal{G}))} + \frac{KN^2 \log T}{p}\right)$ where $N$ and $K$ denote the total number of agents and arms, respectively. This term reflects the impact of $\mathcal G$'s algebraic connectivity $\lambda_{N-1}(\operatorname{Lap}(\mathcal{G}))$ and the link probability $p$, and thus highlights a fundamental trade-off between communication efficiency and regret. As a by-product, we show a nearly optimal regret lower bound. Finally, our numerical experiments not only show the superiority of our algorithm over existing benchmarks, but also validate the theoretical regret scaling with problem complexity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a random graph setting where the nodes are agents each of whom can select several arms for rewards. The stochasticity in the graph drives whether the agents communicate with each other based on the existence of a sampled edge. They then use gossip based reward estimation algorithm for agents to get both a personal and a global idea of the rewards that the arms provide and then which arms to keep and which to do away with. For this stochastic and challenging setting, they first upper bound the agent-wise and global community expected regret, then they interpret those results based on the obtained expression terms, ultimately relating it to graph parameters of p and graph connectivity (using Laplace matrix of the base graph). They discuss some specific upper boundds on known graph structures as examples. Then they proceed with lower bound definition on such class of problems. For these assumptions they have a LB estimate that is again a reinterpreted term from the UB. Then their experiments follow to validate their the bounds on the algorithm used on this problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Consideration of a practical yet less investigated problem in contemporary literature using random graphs with Multi-Arm Bandits for communication.\n2. Work translates to stricter bounds on simplified problems already studied in the literature.\n3. Actionable interpretation of the regret bound terms: global regret, $delta_k$, connectivity parameters $p$ and eigen values of the graph.\n4. A sound initiative to define minimum requirements on the class of problem and prepare a lower bound on the same.\n5. While the originality is not with the algorithm itself, its application to a defined problem and its performance bounds is novel. \n \nWeakness:\n1. A bit mild on the multi-agent side of problem formulation: getting information from agents, even ones you are neighbors with can be noisy/adversarial from the agents' perspective!\n2. In section 3 lines 146-166, authors could be upfront about why they define both local and global reward definitions and how would it be helpful in the algorithm later! Specifically, line 151 (However...) could be made more concrete. Specifically in 173-175 they define the global objective that the policy should optimize. But what is individual agent trying to optimize? Local estimate of sampled arms? Global estiamte of sampled arms? A combination of them with a confidence bound? This only gets concrete by 201-204 when the actual algorithm details appear. An equivalent of equation (1) for individual agents can be made more explicit, just as they did so for the regret bounds later.\n\nQUESTIONS:\n1. Its mentioned in the experimental settings that regret graphs show confidence intervals too. But the plots (Figure 2) doesnt clearly illustrate the same. Can this be made clearer?\n2. Line 337 (...we sample each <> independently...) seems to have a missing term. Is it \"edge\"? \n3. In line 338, what is this new term $q_i$ used to define means? They dont seem to have a definition either above or in the appendix table symbology! Is this expected?\n4. In line 348 authors briefly discuss the algorithm's performance based on the rate at which agents discard their dominated arms. A graph on these dynamics, something like %arms discarded over total arms discarded along with the regret in a temporal phase, like in Figure 1 would be very informative, right? This could help clearly define for the agents' exploitation phase of farming rewards from the \"trusted\" arms!\n\nLIMITATIONS:\n1. A bit mild on the multi-agent side of problem formulation: getting information from agents, even ones you are neighbors with can be noisy/adversarial from the agents' perspective! Basically, assuming agents can always get a noiseless global estimate is a milder study on behalf of multi agent interaction. This enforces their \"distributed\" nature of their algorithm on a practical domain.\n2. While authors already clearly acknowledge their algorithm suffers from a tradeoff between regret minimization and communication cost reduction, they could probably benefit from analyzing the communication in a broadcasted sense that emerges from the connected components in the $\\mathcal{G}_t$, rather than a one-to-one communication between edges several times.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a distributed multi-agent multi-armed bandit (MA-MAB) problem in a heterogeneous reward setting where agents communicate over time-varying random graphs derived from a fixed base graph via the Erdős–Rényi model. The authors propose a fully distributed algorithm that combines successive arm elimination with a gossip-based communication protocol, enabling agents to estimate global rewards despite potentially disconnected communication at each round. Theoretical analysis shows that the algorithm achieves a regret bound of order $O(\\log T)$, which is highly interpretable in terms of the graph properties and connectivity. The paper also establishes a matching lower bound up to constant factors, and provides empirical results validating the regret–communication trade-off and superiority over prior work.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- **Quality:** The paper’s main claims are supported by rigorous and well-structured proofs, demonstrating a solid theoretical foundation. Additionally, the paper includes a basic experimental evaluation that demonstrates the superiority of the proposed method.\n- **Clarity:** The proposed algorithm is well-motivated and clearly explained, with intuitive justifications provided in the main text. The authors carefully analyze the theoretical results, offering meaningful comparisons with prior work. Finally, the effectiveness of the method is supported by a set of illustrative experimental evaluations.\n- **Significance:** This work advances the state-of-the-art by providing an algorithm with provable regret bounds that operates in a distributed and heterogeneous environment with more limited connectivity than considered in prior works [1,2]. Additionally, the regret bounds improve the state-of-the-art in a relaxed setting [1]. \n- **Originality:** Although the authors build on established techniques such as running-consensus-based estimation [3] and successive elimination-based implicit synchronization [4], they employ them in a novel manner in the random connectivity heterogeneous setting to achieve an algorithm with improved performance. \n\n**Weaknesses:**\nRather than separating into broad quality, clarity, significance and originality categories, I will outline my main concerns in a more detailed manner below.\n- My main concern lies in the tightness of the proposed regret bounds. While the first term in the upper bound of Theorem 5.3 matches the lower bound and scales as $O(\\sum \\log T /\\Delta_k)$, the remaining terms—particularly $O(KN^2\\log T/p)$—dominate the overall regret (up to the reward gaps). This discrepancy is not adequately addressed in the proposed lower bound or in the accompanying discussion in Remark 5.10, leaving a notable gap between the upper and lower bounds. Furthermore, it appears that even in homogeneous settings with time-invariant graphs, the regret of the proposed algorithm still scales as $O(KN^2\\log T)$, which is significantly worse than the bounds achieved in prior work, such as [3]. That said, this limitation seems to be common in the literature, and as acknowledged in Remark 5.5, the authors’ results nonetheless represent a meaningful improvement over those in [1,2].\n- Given the aforementioned limitations, this work represents an incremental advancement in the state of the art for heterogeneous MAMAB under time-varying communication graphs. While it offers meaningful improvements, several known challenges in this setting remain unresolved.\n- The techniques employed in this work—such as successive elimination [4], running consensus estimation [3], and the lower bound proof strategy [5]—are based on established methods, which limits the overall novelty of the contribution.\n\nOther issues – \n- Implementing the proposed algorithm requires agents to have global knowledge of certain properties of the communication graph, such as the eigenvalues of its Laplacian. Although this assumption has been adopted in prior work [3], it remains unrealistic in many practical settings.\n- Although the edge communication probability $p$ can be estimated during a burn-in phase, this introduces an additional term in the regret upper bound—$O(N^2\\log T/p^2)$ (Theorem D.2)—which can be larger and further impacts the algorithm’s performance.\n- In the proposed algorithm, agents exchange their estimates of the global mean for each arm in every round, resulting in high communication overhead—linear in $T$—and complex message structures. Unfortunately, the communication cost is neither optimized nor analyzed at the bit level in this work.\n- The main text does not include proof outlines for Lemma 5.2 and Theorem 5.3. Given that this is primarily a theoretical contribution, providing such outlines is important for guiding readers through the core ideas of the analysis. I strongly recommend including them in a revised version of the paper.\n- The authors claim that the numerical experiments validate the theoretical regret scaling with problem complexity; however, Figure 2 only illustrates the general trend of regret as the communication probability $p$ and base graph connectivity vary. It does not empirically validate the precise dependence on these factors as specified in the regret upper bound of Theorem 5.3, which I suspect may be loose.\n- Additional proofreading would improve the paper. For example, in line 81 “the” should be removed, and the sentence in lines 190-191 has been stated before in the same paragraph. \n\n\n[1] Xu, Mengfan, and Diego Klabjan. \"Decentralized randomly distributed multi-agent multi-armed bandit with heterogeneous rewards.\" Advances in Neural Information Processing Systems 36 (2023): 74799-74855.\\\n[2] Zhu, Zhaowei, et al. \"Federated bandit: A gossiping approach.\" Proceedings of the ACM on Measurement and Analysis of Computing Systems 5.1 (2021): 1-29.\\\n[3] Martínez-Rubio, David, Varun Kanade, and Patrick Rebeschini. \"Decentralized cooperative stochastic bandits.\" Advances in Neural Information Processing Systems 32 (2019).\\\n[4] Yang, Lin, et al. \"Cooperative multi-agent bandits: Distributed algorithms with optimal individual regret and constant communication costs.\" arXiv preprint arXiv:2308.04314 (2023).\\\n[5] Lattimore, Tor, and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.\n\nQUESTIONS:\nAs noted earlier, my primary concerns relate to the level of novelty and the tightness of the proposed regret bounds.\n1. Could you clarify the main technical challenges in establishing the regret upper bounds in Lemma 5.2 and Theorem 5.3? In particular, what are the key obstacles in reducing the second and third terms in Theorem 5.3? Do you believe it is possible to improve these terms to at least match known bounds in the time-invariant, homogeneous setting [1]?\n2. Additionally, in Theorem 5.9, what distinguishes your proof of the regret lower bound from the arguments of a similar lower bound for the single-agent case in [1], which your approach seems to closely follow? Is there potential to strengthen the lower bound by incorporating graph-dependent terms, such as the communication probability $p$?\n3. For a revised version, I recommend moving the Conclusion and Future Work section into the main text, while optionally relocating part of the Related Work section to the appendix if space is limited.\n\n[1] Martínez-Rubio, David, Varun Kanade, and Patrick Rebeschini. \"Decentralized cooperative stochastic bandits.\" Advances in Neural Information Processing Systems 32 (2019).\\\n[2] Lattimore, Tor, and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of multi-agent multi-armed bandit. In particular, it accounts for a scenario where the rewards are heterogeneous across the $N$ agents and the communication graph is stochastic and sampled from an Erdos-Renyi model at every round.  The paper's main contribution is analyzing this setting, which generalizes the models that have been proposed in the literature so far: the authors propose an algorithm that achieves no-regret and highlight the components of the regret upper bound, that are interpretable; moreover, they provide a lower bound that proves that at least one component of the upper bound is tight. Finally, some experimental evaluations on both synthetic and real-world data are provided.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is overall well written, the narration can be followed without any trouble, and the message of the paper is clear.\n- The comparison with existing results is fair and highlights the additional difficulties of this setting. \n- The experimental campaign goes beyond the simple comparison with the competing algorithms and explores the links between the regret and the problem parameters.\n\nWeaknesses:\n\n- I don't like how the conclusions and future work are presented as an appendix. There are two cases: either this section is important (and I believe it is, as the future work is one of the most interesting parts of a paper) and you find a spot for that in the main paper; or this section is not crucial for the paper and you just avoid it. In both cases, having an appendix for this makes no sense in my opinion. This is not a major point, but I wanted to report it.\n\n- The regret-communication trade-off that appears in the regret upper bound does not appear in the lower bound, and thus there is no way to be sure that this trade-off actually exists and is not a byproduct of the regret analysis. **This is a major point**: the narration of the paper strongly depends on this \"instance-dependent\" type of guarantee, and the meaning of the problem parameters appearing in it. The authors provide a lower bound, which basically resorts to the standard MAB lower bound but does not explore the presence of $p$ or other graph-dependent quantities. There is some algorithmic contribution, but since the overall approach is based on existing families of algorithms, the technical contribution is what can make this paper accepted or not, in my opinion. \n\n- The methodological contribution is not a breakthrough if compared to previous works in this literature. The authors provide algorithmic routines to deal with the stochasticity of the graph and the heterogeneity of rewards simultaneously, but overall the main underlying ideas to tackle the MA-MAB problem have already been outlined in previous works.\n\nQUESTIONS:\nCan the authors argue whether the upper bound is tight or not about the problem-dependent quantities?\n\nThis is a major point; if I missed some parts of the submission, and the authors can answer this positively, I would reconsider my score positively. However, note that this is not my only evaluation criterion, because I also think that the methodological contribution is not a breakthrough if compared to previous works in this literature.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the heterogeneous multi-agent bandit setting, where the goal of each bandit is to minimize regret with respect to the globally best arm. The agents are assumed to exist on a fixed graph $\\mathcal{G}$, and communicate with neighbors with probability $p$.\n\nThe authors propose an arm elimination algorithm that uses gossip to reach a consensus. The paper provides an upper bound on the regret of the algorithm, $\\mathcal{O}\\left(\\sum_{k:\\Delta_k>0}\\Delta_k^{-1}\\log(T)+p^{-1}\\lambda^{-1}N^2\\log(T)+p^{-1}N^2\\log(T)\\right)$, where $\\lambda$ is the spectral graph of the graph $\\mathcal{G}$, that neatly decomposes into a centralized regret bound and a consensus/decentralization term. \n\nThe authors provide a lower bound for the centralized regret that matches their upper bound. Lastly, experiments to demonstrate the algorithm are conducted.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The authors provide a stronger regret bound compared to many papers analyzing similar settings. Previous works have obtained regret bounds such as $\\mathcal{O}\\left(\\sum_{k:\\Delta_k>0} \\Delta_k^{-1}N\\log(T)\\right)$ in [1]. The algorithm presented in this paper has a superior bound of $\\mathcal{O}\\left(\\sum_{k:\\Delta_k>0} \\Delta_k^{-1}\\log(T) + KN\\log(T)\\right)$.\n\n2. The algorithm is simpler than many similar algorithms and uses fewer hyperparameters. Previous algorithms require a burn-in time $L$, with $L$ being a complicated function of other hyperparameters. These algorithms also require $p>1/2$. Furthermore, the gossip mechanism is simplified with each bandit maintaining a local record of the values of the arms, which is communicated over the graph. \n\n3. Previous works do not capture the consensus/communication term of the regret as neatly as the regret bound presented. In [1], the communication cost is largely captured by the burn-in period $L$, which does not depend directly on the graph structure.\n\n[1] Xu, Mengfan, and Diego Klabjan. \"Decentralized randomly distributed multi-agent multi-armed bandit with heterogeneous rewards.\" Advances in Neural Information Processing Systems 36 (2023): 74799-74855.\n\nWeaknesses:\n\nThe motivation for seeking global optimality rather than local is lacking. It would be best to include a statement, i.e., that the local bandits might have biases, to clarify this.\n\nThe experiments are lacking. In particular:\n\n1. There should be more experiments with varied time horizons $T$, with comparisons to the upper bound. Specifically, Figure $1$ records the running regret given a fixed horizon $T$. Another graph showing the terminal regret for different horizons $T$ may better capture the dependence of the regret on $T$.\n\n2. Figure 2 would benefit from more data points, particularly with $p$ close to $0$. Figure $2$ may also benefit from a log-scale, and linear fits to verify the $p^{-1}$ relation of the regret.\n\n3. The experiments do not sufficiently capture the dependence on the spectral gaps of the base graph. A graph of the regret v/s the spectral gaps for a family of graphs, e.g., expander graphs, might be useful to verify the dependence on the graph structure.\n\nFurther, the conclusion and future work section of the paper has been included in the appendix rather than the main body.\n\nQUESTIONS:\n1. How tight is the consensus/gossip upper bound? Would it be possible to come up with a lower bound that includes the communication cost for some specific graph, like a tree or a cycle?\n\n2. In Lemma B.4, is the $K$ in the upper bound necessary? The \"waves\" for each of the arms should happen in \"parallel\", which should render the argument in lines 512 to 515 unnecessary.\n\n3. How is the second equality in equation (14) (line 531-532, the equality after the definition) obtained?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}