Title: Individual Regret in Cooperative Stochastic Multi-Armed Bandits
Venue: NeurIPS 2025
Accepted: yes
Keywords: MAB

Multi-armed bandit

stochastic MAB

cooperation
Abstract: We study the regret in stochastic Multi-Armed Bandits (MAB) with multiple agents that communicate over an arbitrary connected communication graph.
We analyzed a variant of Cooperative Successive Elimination algorithm, $\texttt{Coop-SE}$, and show an individual regret bound of ${O}(\mathcal{R} / m + A^2   + A \sqrt{\log T})$ and a nearly matching lower bound.
Here $A$ is the number of actions, $T$ the time horizon, $m$ the number of agents, and $\mathcal{R} = \sum_{\Delta_i > 0}\log(T)/\Delta_i$ is the optimal single agent regret, where $\Delta_i$ is the sub-optimality gap of action $i$.
Our work is the first to show an individual regret bound in cooperative stochastic MAB that is independent of the graph's diameter.

When considering communication networks there are additional considerations beyond regret, such as message size and number of communication rounds.
First, we show that our regret bound holds even if we restrict the messages to be of logarithmic size.
Second, for logarithmic number of
communication rounds, we obtain a regret bound of ${O}(\mathcal{R} / m+A \log T)$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem in which multiple agents (vertices of a graph) face the same stochastic bandit problem and can communicate over the edges of the graph. The authors give algorithms for this problem with strong individual regret guarantees. Importantly, these bounds are independent of the diameter of the graph. The first algorithm requires much information to be passed across each edge on each trial. However, the later algorithms require much less, and one algorithm only requires a logarithmic number of trials in which to communicate. The authors also give an almost matching lower bound on the individual regret.\n\nSTRENGTHS AND WEAKNESSES:\nI think that the results of this paper are quite remarkable. At first I didn’t believe that the results could be possible (especially the O(log(m)) message size results) but after thinking about it I now believe that they could be true. Unfortunately, I haven’t had the time to read through the proof and verify the results.\n\nThe lower bound is also a nice addition to the paper.\n\nThe results of the paper are clearly communicated.\n\nAlthough this is not a weakness of the result itself, I feel that, in order to make the paper self contained, there should be some explanation of how SE works before we jump into the description of Coop-SE as the explanation of the latter assumes familiarity with the former.\n\nQUESTIONS:\nI have no questions for the authors as I feel I understand the results. Since there is no section for comments I will point out some typos here:\n\nIn Line 149 you say that the agent observes a reward but you don’t say that the reward is drawn from the SMAB distribution.\n\nYou have some small brackets in Theorem 2 which should be big brackets. Same for Line 194.\n\nTypo in definition 10 - begging instead of beginning\n\nLIMITATIONS:\nNo limitations noticed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the cooperative stochastic multi-agent multi-armed bandits problem, where multiple agents are connected by a undirected connected graph. In this model, multiple agents aggregate information via communication to speedup the learning process. Importantly, this paper proposes a cooperative algorithm with a regret bound, independent of any graph parameter. Authors also provide a lower bound and discusses the regret achievability under two communication constraints, including communication-round constraint and communication-bit constraint.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n- This paper proposes a simple cooperative algorithm, ensuring a low individual regret for each agent. Importantly, the regret bound is independent of any graph parameter. This result implies that if the number of agents goes to infinity, the regret bound is entirely independent of arm gap.\n\n- This paper presents a lower bound for the individual regret nearly matching the upper bound in the minimax form.\n\n- Further, authors consider the constrained communication setting. For both logarithmic communication bit and logarithmic communication  rounds, two algorithms are proposed to achieve regret bounds, independent of graph parameter.\n\n\nWeakness: I don't see any major weakness of this paper, and authors have acknowledged that there exists a gap between the upper and lower bounds. Though the design of algorithm is simple, ie., extending elimination based algorithms to multi-agent setup, it is indeed effective, and authors do a careful analysis to get a good regret bound. \n\nThe reason why I don't give a higher score is that this paper does not give any algorithmic contribution and the analysis, though careful, seem not to be applicable to broader cases (i.e., applying to UCB or TS algorithm, but the current analysis only works for elimination algorithms).\n\n\nSome minor suggestions on presentation:\n\n- For Table 1, I'd suggest adding the definitions of $f(G)$ and $h(G)$ in either the table caption or the footnote to remind readers.\n- In warmup section, authors do not provide any pseudocode, even no informal version and informative description, which makes the section less informative. After quick passing section 3 (warmup section), I don't feel any connection between section 3 and section 4. This means I can directly read section 4 without reading section 3, which makes section 3 redundant. I would suggest removing section 3 if it does not give any warmup functionality.\n- In line 12, algorithm 6, it should take union over all neighbors $v'$.\n\nQUESTIONS:\n- Can authors discuss more why the current method does not work for UCB?\n\n- I am a bit worried about the significance of individual regret in the stochastic multi-agent MAB setting, though it is indeed interesting in non-stationary setting, Many existing papers in stochastic setting use group regret (or equivalently average regret) as the metric, but those algorithms also have a low individual regret, and they just not give a formal proof on that. This is because in stochastic setting, the high-level behind speedup is to aggregate samples across agents, in light of the Hoeffding bound. Can author mention a paper in the stochastic setting that have low group regret, but individual regret is large?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies individual regret in cooperative stochastic multi-armed bandit (MAB) problems, where multiple agents interact over a communication graph. The authors propose Coop-SE, a fully decentralized extension of the Successive Elimination algorithm, and prove that it achieves individual regret bounds of $O(\\mathcal{R}/m+A^2+A\\sqrt{\\log T})$, which is importantly independent of the graph's diameter. They also introduce two variants that handle communication constraints: Coop-SE-CONGEST (for small message sizes) and Coop-SE-Comm-Cost (for few communication rounds), both maintaining diameter independent regret guarantees. A lower bound is also presented, supporting the near-optimality of their approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- **Quality:** The paper’s main claims are backed by thorough and well-structured proofs, demonstrating a strong theoretical foundation.\n- **Clarity:** The authors effectively explain the motivation behind removing diameter dependence, using intuitive examples. The algorithms and proof techniques are clearly described and mostly self-contained within the main paper.\n- **Significance:** Although cooperative multi-agent MAB has been studied extensively, this is the first work to achieve individual regret bounds that are independent of both the graph diameter and the number of agents.\n- **Originality:** While the Successive Elimination and message passing ideas are not new in this setting, the paper introduces original contributions—such as implicit synchronization across agents and cluster-based strategies—to obtain diameter-independent bounds.\n\nOverall, I am convinced that achieving diameter-independent bounds is important, and appreciate the authors’ effort to also discuss limited communication settings. \n\n**Weaknesses:**\nRather than separating into broad quality, clarity, significance and originality categories, I will outline my main concerns in a more detailed manner below.\n- While the regret bound in Theorem 2 (As well as Theorems 4 and 5) is claimed to be significant due to its independence from the graph diameter, I am not fully convinced of its practical value because of the presence of a $O(\\sqrt{\\log T})$-dependent term. As $T$ grows, this term can dominate, potentially nullifying the advantage of removing diameter dependence. In fact, as noted in line 194, the *actual* bound is $O(A\\min (A+\\sqrt{\\log T},D))$, meaning that for sufficiently large horizons, the regret essentially matches the bound of [1]. This raises concerns about the tightness and novelty of the result. The absence of experimental validation to support the claimed improvements further amplifies this concern, as it leaves open the question of whether the proposed algorithm offers tangible advantages in realistic settings.\n- This concern is further reinforced by the significant gap between the $O(A\\sqrt{\\log T})$ additive term in the upper bound and the $\\Omega(\\sqrt{A})$ term in the lower bound. It remains unclear whether the $T$-dependence in the additive term is truly necessary, or whether a tighter, $T$-independent bound might be achievable.\n- While the treatment of limited communication settings in Sections 7 and 8 is valuable and introduces novel ideas, I believe addressing the substantial gap in all of the regret bounds is a more pressing issue. This gap makes the overall theoretical contribution feel somewhat incomplete. Furthermore, the regret bound for Coop-SE-Comm-Cost includes a larger additive term of $O(A\\log T)$, which—aside from being independent of the gaps—effectively offsets the cooperative benefit of the $O(1/m)$ term by dominating it for large $T$.\n- As previously noted, this work does not include any experimental evaluation. While the primary contributions are theoretical, it would still be valuable to empirically demonstrate that the proposed algorithms outperform baseline methods in practice—at least in representative scenarios. This is particularly important given the noticeable gap between the upper and lower bounds on regret, which raises questions about the practical tightness of the theoretical results.\n\n[1] Wang, Xuchuang, and Lin Yang. \"Achieving near-optimal individual regret low communications in multi-agent bandits.\" The Eleventh International Conference on Learning Representations (ICLR). 2023.\n\nSome other concerns – \n- The overall structure of the paper could benefit from refinement. The Introduction and Related Work sections extend to page 4, which feels overly long. Consider moving parts of the Related Work—particularly the final paragraph, which discusses broad extensions of cooperative bandits not directly relevant to the paper’s main contributions—to the appendix. This would free up space to include a more detailed explanation of the Coop-SE algorithm in Section 4, and to add a proper Conclusion and Future Work section to the main paper.\n- Additional proofreading would improve the paper. For instance, there appears to be a typo in line 73 regarding the regret bound for Coop-SE-Comm-Cost, which is inconsistent with the other results presented. The algorithms proposed in the paper should also be mentioned explicitly in the abstract. In line 184, the word “is” should be removed. Moreover, Algorithm 2—a simplified version of Coop-SE—is included in the main text but is never referenced or discussed, which may confuse readers. There is also a mismatch in the presentation of regret bounds: the lower bound is given in minimax form, while the upper bounds are instance-dependent. This transition should be explicitly addressed in the main paper. Lastly, the spanning tree notation $\\mathcal{T}$, introduced in line 335, is used without prior definition in the main text.\n\nQUESTIONS:\n1. As noted earlier, the primary reason for my current score is the $T$-dependent gap between the regret upper bounds and the lower bound. That said, I do recognize the motivation outlined in Section 1.1 for pursuing diameter-independent bounds, particularly through the use of $T$-dependent terms to illustrate worst-case scenarios. I also acknowledge that the bounds presented in this work are strictly stronger than those in prior studies such as [1]. Given this, could the authors clarify how their algorithms are expected to perform on a fixed problem instance as a function of $T$? Specifically, do the proposed methods offer practical advantages over existing algorithms in such settings?\n2. Could the authors provide an intuitive explanation for the emergence of the $O(\\sqrt{\\log T})$ and $O(\\log T)$ terms in the regret upper bounds? What underlying factors contribute to this dependence? Additionally, have any attempts been made to reduce or eliminate this $T$-dependence, and if so, what challenges made it difficult to achieve?\n3. In Sections 7.1 and 7.2, the authors describe preprocessing phases required to construct a spanning tree or clustering for the limited communication algorithms. How much global knowledge about the communication graph is assumed or required from the agents during these phases? Additionally, could the authors clarify how regret is accounted for during these preprocessing steps? Since each round contributes to the overall regret, it would be helpful to understand how these initial phases impact the total regret accumulation.\n\n[1] Wang, Xuchuang, and Lin Yang. \"Achieving near-optimal individual regret low communications in multi-agent bandits.\" The Eleventh International Conference on Learning Representations (ICLR). 2023.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines individual regret in multi-agent stochastic MABs residing on a communication graph. This paper explicitly investigates how to remove the dependency on the graph diameter from the regret upper bound, which is achieved through a more sophisticated analysis of the Coop-SE algorithm. Then, the paper extends the Coop-SE algorithm to two types of low-communication variants, which have reasonably worse regret upper bounds.\n\nSTRENGTHS AND WEAKNESSES:\n- S1: The analysis for Coop-SE (especially Lemma 1) is more sophisticated than prior ones, enabling the removal of dependence on $D$.  The reviewer thinks that it is an interesting new approach.\n- S2: The writing of this paper is clear and motivating. S1.1 did a good job of motivating the study, and S5 provides an easy-to-follow proof sketch.\n---\n- W1: Although concretely new progress on the direction of multi-agent MAB, the reviewer thinks that the result in this paper is not very significant (even the authors did a good job in the discussion in S1.1). The main improvement of this paper is from $D$ to $\\min\\{A+\\sqrt{\\log T}, D\\}$, where the latter can remove the dependence of $D$ when the graph has a large diameter. However, \n    - (1) the $O(\\sqrt{\\log T})$ term is still relatively large since the time horizon $T$ in the bandit setting is typically the largest constant. \n    - (2) The improvement only happens when the graph has a large diameter (larger than $\\sqrt{\\log T}$!).\n- W2: The current results are not tight. The upper bound $O(A^2)$ has a large gap in comparison with the $\\Omega(\\sqrt A)$ lower bound.\n- W3: Although the analysis is new, the Coop-SE algorithm has been proposed multiple times in various prior works (as the authors are also aware). It would be better to posit the contribution of this work on the analysis side and attribute the merit of the Coop-SE algorithm design to these prior works (e.g., remove “ours” after Coop-SE in Table 1).\n\nQUESTIONS:\n- Line 37: why omit the $\\log T$ regret term here? It dominates the $\\sqrt{\\log T}$ term.\n- Line 65: the $\\mathcal R$ still dependent on the gaps.\n- Line 67: It would be better to present the upper bound in minimax form here as well.\n- Line 74—79: the open problem can be addressed by other prior works mentioned in the related work section, why emphasizes it here? Is there a special insight of the current work on the problem?\n- The communication round of TCOM should be additive $D$ instead of multiplicative $D$.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the cooperative stochastic multi-armed bandit problem, where a number of agents ($m$), connected via a fixed graph, face the same bandit instance and can exchange information with their neighbours. The goal here is to guarantee small regret for every individual agent in the network. The authors noted that previous works achieve regret bounds of the form $\\mathcal{R}/m + D$, where $\\mathcal{R}$ is the optimal single-player regret and $D$ denotes the diameter (or some other function) of the graph that can be as large as $m$ in some cases. This additive dependence on $m$ means that the contribution of $\\mathcal{R}$ to the regret bound cannot be made arbitrarily small in general as $m$ increases. The authors address this issue by proposing an algorithm that enjoys a similar regret bound but replaces $D$ with $A \\cdot \\min(D, A + \\sqrt{\\log T})$, where $A$ is the number of actions and $T$ the time horizon. A worst-case lower bound of $\\sqrt{AT / m} + \\sqrt{A}$ is also provided, matching the first term of the upper bound in a worst-case sense and showing that some additive dependence on $A$ can be unavoidable for some graphs. The adopted algorithm, Coop-SE, is an adaptation of the successive elimination algorithm to the cooperative setting that is completely decentralized and uses only local graph information. Similar regret guarantees are also provided for two extensions of the algorithm that respect constraints on the message size or the number of communication rounds.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper seems to provide a solid contribution to the literature on this problem. Unlike the cited regret bounds previous works, the dependence of the new regret bound on the reciprocals of the sub-optimality gaps can get arbitrarily small as the number of agents increase, regardless of the structure of the graph, so long as it is connected. The adopted algorithm is simple and intuitive, and the explanations are fairly clear.\n\nOn the other hand, the presented results still do not provide a clear idea as to what extent the structure of the graph can be exploited, particularly concerning the second (additive term) in the regret. This term seems to be at least $A$ in the upper bound for any graph (including, e.g., the complete graph), while the opposing $\\sqrt{A}$ term in the lower bound only concerns a particular structure (the line graph) and does not seem to apply in a graph-specific sense. Perhaps this should be emphasized more. Aside from this, I see no major weaknesses in the paper. Some minor points concerning the introduction; it does not seem that the additive terms in Theorems 4 and 5 are exactly as reported in Section 1.2 and Table 1; moreover, in lines 91-93, it seems that the argument should be that $h(\\mathcal{G})$ can increase (potentially up to $m$) if $\\lambda_2$ is close to $1$, not $0$.\n\nQUESTIONS:\nConcerning the additive term, do you believe that the shortcoming is in the upper bound? If so, could this be a limitation of the successive elimination algorithm? Could it be a limitation of only using local graph information?\n\nLIMITATIONS:\nYes, a brief discussion on the limitations of this work are discussed in the future work section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}