One More Step Towards Reality: Cooperative Bandits with Imperfect Communication

Abstract: The cooperative bandit problem is increasingly becoming relevant due to its applications in large-scale decision-making. However, most research for this problem focuses exclusively on the setting with perfect communication, whereas in most real-world distributed settings, communication is often over stochastic networks, with arbitrary corruptions and delays. In this paper, we study cooperative bandit learning under three typical real-world communication scenarios, namely, (a) message-passing over stochastic time-varying networks, (b) instantaneous reward-sharing over a network with random delays, and (c) message-passing with adversarially corrupted rewards, including byzantine communication. For each of these environments, we propose decentralized algorithms that achieve competitive performance, along with near-optimal guarantees on the incurred group regret as well. Furthermore, in the setting with perfect  communication, we present an improved delayed-update algorithm that outperforms the existing state-of-the-art on various network topologies. Finally, we present tight network-dependent minimax lower bounds on the group regret. Our proposed algorithms are straightforward to implement and obtain competitive empirical performance.


Reviews and author responses:

 We thank the reviewer for the response.

---**Results are novel and not incremental:** Here we consider three types of communication imperfections and we discuss the novelty and non incremental nature of results in each imperfection as follows. Despite the simplicity of our ideas, theoretical analysis poses significant challenges. 

Random link failures: Our work is the first work to provide general theoretical results for the case with probabilistic communication over general communication networks. Providing theoretical results for general network graphs with probabilistic communication poses two significant theoretical challenges. 1.) Providing a tail probability bound for the case where agents communicate probabilistically (see Lemma 3 and Lemma 5) and 2.) decomposing the regret into high probability events that can be bounded depending on the network structure and  low probability event that can be upper bounded using tail probabilities (see Lemma 2 and Lemma 6). 

Random delays: To the best of our knowledge our work is the first work to study random communication delays in cooperative stochastic bandits. We provide novel theoretical analysis for this setting (see Lemma 7) by proving that the group regret can be decomposed in to the summation of regret under perfect communication and a term that is proportional to the expected number of outstanding messages in the network. 

Corruptions: We present a novel combination of arm elimination and UCB exploration for the adversarially corrupted communication setting. Our analysis involves 2 novel components: first, we provide a new analysis of epoch-based arm elimination under communication. This requires a new argument to bound the probability of the bad arm being picked (Lemma 10). Next, we provide a new algorithm to handle delays in the communication of arms: to our knowledge, arm elimination in a delayed feedback setting has not been explored before. We demonstrate that by running local estimation algorithms in between arm elimination epochs, one can obtain only a $\mathcal O(N\log\log T)$ additional increase in regret

Lower bound: Our lower bounds for minimax rates are the first of their kind in this setting and for many graph families (linear, circular, regular, etc.) are within a constant factor of the upper bound presented in our algorithms (see Remark 8).

---**Clarification on centralized/ partition among agents:** The reviewer is correct that we require a partition (and hence we put "centralized" in quotes), however, in contrast to the remark, the algorithm is indeed decentralized in the sense that each agent can operate entirely independently of the others. We would also like to clarify that simply requiring a partition does not preclude the algorithm from being decentralized: if the graph G is known to all agents then the dominating set can be found independently by each agent. If there are $k$ unique dominating sets, the agents can simply partition the horizon into $k$ interleaved chunks of size $T/k$ each, and subsequently run $k$ copies of RCL-RC, one for each chunk in a completely decentralized manner. If the graph G is unknown, the agents only require $\mathcal O(N \cdot \gamma)$ rounds to recover the graph structure (up to $\gamma$ hops), in a decentralized manner, via the message-passing protocol, increasing the regret by a constant factor.

Furthermore, the algorithm operates differently for each agent: the agents switch between a personal UCB and a group arm selection mechanism, both steps of which can be run entirely independently of other agents.




The paper studies cooperative multi-armed bandit in a few scenarios including stochastic time-varying networks, random communication delays, and byzantine data corruption. For each scenario, a decentralized algorithm is proposed and analyzed for its network regret bound. For the special case with perfect communication, an algorithm is proposed and shown to outperforms the existing state-of-the-art algorithms.   The paper considers an interesting and important problem with different "realistic" communication settings. 

My biggest concern is about the setting of cooperative multi-armed bandit which is the basis for all four algorithms in the paper: A uniform upper bound for all arms' (unknown) variance proxy is known to each agent. In the problem formulation, each agent faces the same set of arms and there is no difference in reward observation distribution among the agents. That means each agent can learn the best arm by itself without any information from other agents, which can be done by any state-of-the-art UCB algorithms. However, now with collaboration and additional information from others, each agent even need more arm information -- the upper bound. The basic setting itself is not convincing. Moreover, in the proposed algorithms, each agent makes use of this uniform upper bound in the confidence bound function, which is a critical component determining regret bounds. Considering different arms may have large difference in variance proxy, such a design generally may not lead to optimal regret. 

The algorithm RCL_LF is mainly described by words. To me it is not precise enough. 

In Remark 1, the claim that "Theorem 1 quantifies the benefit of communication" needs more explanation/validation. The benefit of communication (i.e. collaboration) also needs to compare with the case when each agent learns itself by state-of-the-art UCB algorithms. 

I have trouble understanding Remark 2. There is no d in the theorem. The setting of p with d will needs further network information. Similarly, the claim in Remark 3 is also hard to follow, which needs further details/explanation.

The ideas and algorithm in Section 5 are not clearly explained. If the algorithm requires a partition of exploring and imitating agents, then it requires a centralized design procedure. Does the partition arbitrary? Can any agent be corrupted? 

Update on Sep 2:  Thanks the authors for detailed response. I am not fully convinced by the authors' response. The authors' model is based on the assumption that each agent can directly transmit reward values. To me this simplifies the algorithm design and analysis. Also because of this, the claim that cooperative bandit here can improve network regret is straightforward, since for each clique with M agents, it is essentially equivalent to the case where each agent can pull M arms at one time. Thus the constant before log T term, when p=1, is then the number of covering cliques instead of the number of total agents.

After reading the references in the authors' response, I feel the random link failure (Theorem 1) and stochastic communication delays (Theorem 3) are incremental results, particularly the latter as also reflected by the proof sketch of Theorem 3.

As for Section 5, the authors admitted in the response that the proposed algorithm requires a centralized design/partition among the agents, which is not feasible for the decentralized/cooperative setting. 

With the above, I decide to keep the score. 
 Yes

 Thank you for the response and for increasing the score. 

 Thank you very much for the response and for increasing the score. Due to the complexity of the joint probability of agent-action space, it is difficult to provide a tight upper bound for regret of individual agents that depend on the local structure of the graph. However, we will explore this direction in future work. 

This paper considers the stochastic cooperative bandit setting. In this setting a network of agents works together to control the regret, which is the expected difference between the cumulative rewards of the agents and the cumulative reward of the best arm. While this setting has been studied in prior work, this paper addresses some real worlds issues that may be encountered. These issues include communication between agents failing, delay in messages over the network, and adversarial corruptions of the messages. All these variations of the standard stochastic cooperative bandit setting lead to problems for standard algorithms, which the paper addresses.   The problem that is studied is quite well motivated and I agree that the results in this paper are a step towards applications of the setting in more realistic scenarios. 

While the influence of delay and adversarial corruptions have been studied in isolation, it appears that it has not been studied in the stochastic cooperative bandit setting. Therefore, the results surrounding these problems are novel, but not entirely unexpected given prior work. Nevertheless, the solutions to these problems appear to be novel and are very well explained. 

However, as far as I am aware, the problem of communication failing has not been studied before. The challenges this brings to algorithms are clearly explained and the solution, ignoring messages to control bias, is elegant. The experiments are also quite helpful in explaining the challenges that are posed to the problem of failing communication. There is one caveat, which is that the probability of communicating a message is assumed to be known (line 45). This appears to be a weak spot in the argument that this problem is a step towards reality, as I cannot imagine a scenario in which this is known. I think that a discussion on why it is realistic that this probability is known and why it needs to be known would be a good addition to the paper. 

The results in the paper are well discussed. Where possible the results are related to prior work and lower bounds. The experiments are a good addition  to the paper. 

Minor comments,
- Figure 3: the bottom plot of the message passing experiment does not appear to be correct: the y-label and the letter match the bottom plot of the instantaneous reward sharing experiment. Also, instantaneous contains a typo. 
- line 220: extra (
- Algorithm 1, initialize:  before $\lambda$ a comma is missing. 
 The authors adequately addressed the limitations and potential negative societal impact of their work.

 Thank you for the clarification of the probability of communicating messages. I have updated my score. 

The paper studies the multi-agent cooperative bandit model and presents three scenarios where "imperfections" in the communications between agents. The first is when communication links fail, second is when communications are delayed and third is where the communication messages are corrupted. Each of these are important problems by themselves and are crucial to understand to deploy multi-agent bandit algorithms in practice. For these three cases, the paper proposes UCB based algorithms that exploit the graph structure, and show that, despite communication imperfection, the group regret is smaller than the sum of individual regrets had agents chosen not to collaborate at all. The results thus provide a smooth trade-off where if the communications are good, the regret is small, while in the worst case, the regret is no worse off than no communication.  The paper is quite comprehensive and contains many results. The attractive feature of the paper is that the analysis for the various cases of imperfection can be handled using similar ideas. 

One place off improvement is to give an algorithm for RCL-LF, which I did not find even in the supplementary materials. Overall, I like the paper and its attempts. Nevertheless, I had a few questions on the model with random link failure, which is the reason for my low score. I am happy to increase my score if the following items can be explained.

1. In the noiseless case, the reference (missing in the submission) "The gossiping insert-eliminate algorithm for multi-agent bandits" by Chawla et.al. in AISTATS 2020 shows that for any graph, every agent incurs a regret reduction of order N. So, this is an improvement over the Dubey, Pentland result quoted in Line 188 of the submission. Therefore the results of Chawla et.al. is also an improvement over Theorem 5 of the submission ?

2. I am a bit confused on lines 148-152 in the submission. I am quoting the last sentence (152): "This improved performance has a consequence, however: agents that are poorly connected only observe exploitative arm pulls, which requires them to explore for longer in order to obtain similar estimates for sub-optimal arms, increasing their regret." This seems contradictory to results, of Chawla et.al. above, which show that for any graph, every agent incurs a regret reduction of order N ?

3. I am confused about the algorithmic methodology, as to why must agents discard information. The generated messages are of the form  (Arm Pulled, Reward, Agent, time) that are floating around in the network. Second, the reward noted down in the message is independent of the decision to play the arm, as the decision of arm-pulls are based on historical samples. Thus, very message is "useful" and "tractable" (since they are iid) and can potentially be used to improve regret ? Provide pseudo-code for RCL-LF. 

 Thank you for the explanation on the need to discard the information. Indeed, the broadcast nature of the communication model necessitates regulation. As a follow-up to this question is can one prove regret bounds that depend on the "local structure" of the graph at an agent ? That will be interesting, although not sure how easy that will be :)

I have updated my score to 7. 

 We would like to sincerely thank the reviewer for their feedback and address the questions below.

**Psuedo code for ${\tt RCL-LF}$:** We have provided the pseudo-code for  ${\tt RCL-LF}$ in response to all reviewers. We will add this in the revised version of the paper.

**Comparison of results with Chawla et.al. in AISTATS 2020:**
Thank you for pointing out this work! Indeed, Chawla et al is a relevant work, and we provide a detailed comparison as follows, which we will include in the updated version. While the paper provides similar results, there are several differences in the setup considered in Chawla et al compared to our setup.

First, we can see that Chawla et al do not provide a uniform $\mathcal O(\frac{1}{N})$ speedup, but in fact, their regret depends on the difficulty of the first $\frac{K}{N}$ arms, which is a $\mathcal O(\frac{1}{N})$  speed up *only* when all arms are “uniformly” suboptimal, i.e., $\Delta_i \approx \Delta_j \forall i, j \in [K]$. In reality, the speed up shown by Chawla et al can be much lower than $\mathcal O(\frac{1}{N})$  for real-world problems, as shown below.

For example, consider a bandit problem with 3 types of arms: 1 optimal arm, $d = \frac{K}{N} + 2$ “hard’’ arms with gap $\Delta_h$, and $K-d-1$ “easy’’ arms with gap $\Delta_e$. One can see that our algorithms obtain *uniformly better regret* whenever $\frac{\Delta_h}{\Delta_e} \geq \frac{K-d-1}{d\left(\frac{N}{\bar{\chi}(G_\gamma)}-1\right)}$, regardless of $G$ or $\gamma$, i.e, when the arms are less “similar”, which is the case for real-world problems (if all suboptimal arms are similar then it doesn’t matter which suboptimal arms get picked anyway).

Furthermore, observe that our algorithm will always provide a speed up of order $\frac{\bar{\chi}(G_\gamma)}{N}$ regardless of the arms themselves, and when we run our algorithm by setting the delay parameter $\gamma = d_\star(G)$ (diameter of the graph $G$), we obtain an $\mathcal O(\frac{1}{N})$ speedup regardless of the sparsity of $G$. We can see that, in the above example, if we set $\Delta_e = \infty$, Chawla et al obtain no speed up, whereas our algorithm obtains the same speed up.

Next, we remark that the regret bounds should not only be compared in terms of the leading terms since the constants differ greatly between the bounds. Our constants (per-agent) scale as $\mathcal O(K)$ in the worst case, whereas Chawla et al obtain a constant between $\mathcal O(K+(\log N)^\beta)$ and $\mathcal O(K+N^\beta)$ for some $\beta \gg 1$, based on the graph structure, which can dominate the $\log T$ term when we have a large number of agents present.

Finally, Chawla et al consider a gossip-based protocol. While this protocol involves sparsity, it does not incorporate delays in communication (each message is obtained instantaneously between agents), whereas our communication protocol involves delays in messages (each hop adds a delay of 1-time step when forwarding messages. Line 200), making the protocols a little different with respect to the key contributions. Our key contributions are to provide bounds on the utilization of information in the network, and Chawla et al operate by eliminating arms in a distributed fashion without tracking delays explicitly.

**Explanation for 148-152:** Lines 148-152 conveys two important points and we explain these as follows. \
1.) The amount of information received by each agent depends on how connected they are to the agents. Well-connected agents receive information from more agents than poorly connected agents. (This leads to an information disparity across the network.) Thus well-connected agents receive more information and obtain better performance. This is due to the fact that with more information well-connected agents are able to quickly reduce the uncertainty associated with optimal arm and sub-optimal arms. Thus they are able to do more exploiting and less exploring which leads to pulling the optimal arm more frequently. \
2.) This improved performance of well-connected agents has a direct consequence on poorly connected agents who are connected to well-connected agents. As explained in the previous point, well-connected agents do less exploring. Thus information received by poorly connected agents from well-connected agents is mostly about the optimal arm. Thus this information does not provide much help to the receiving agent  (poorly connected agent) with reducing the uncertainty associated with the sub-optimal arms. Thus they are required to pull suboptimal arms by themselves to reduce the uncertainty associated with sub-optimal arms. This leads to more exploration and poor performance for poorly connected agents.

This causes group performance to decrease in graphs with irregular degree distributions

**Lines 148-152 do not contradict Chawla et al:** This is due to the differences in the setups. In Chawla et al they consider that at every communication round each agent $i$ samples an agent according to the probability distribution $P(i,.)$ and receives information from that agent. Since $P$ is irreducible the graph $G$ is connected. Thus regardless of their degree (number of neighbors), each agent receives information from only one other agent at any given communication round. Thus all the agents in the network receive the same amount of information. This leads to agents obtaining similar performance. In our setting, in any given communication round each agent can receive information from all its neighbors. Thus in our setting agents with a high degree receive more information and agents with a low degree receive less information. This leads to a high information disparity in our setting. Thus in our setting well-connected agents obtain low regret and poorly connected agents obtain high regret. 
 
**Explanation for discarding information:** We agree with the reviewer that every message is useful and tractable. However, due to the degree disparity among agents in a general network $G$, every message does not reach every agent. 

First, consider the instantaneous reward sharing case. In this case, each agent is sharing the instantaneous action and reward with its neighbors and those messages are not forwarded to other agents. In this case, it is clear that the number of messages (amount of information) received by each agent is proportional to the degree (i.e., number of neighbors) of the agent. As explained in the first two paragraphs of Section 3, this creates an information disparity among agents. Well-connected agents (agents with a high degree) receive more information and perform better than poorly connected agents. As a result of this disparity group performance decreases. The effect of information disparity is exacerbated in the presence of link failures. Thus in order to regulate the information flow in the network in our algorithm each agent, $i$ incorporates received information with a probability inversely proportional to their degree (i.e. $p_i=\frac{d_{\min}}{d_{i}}$). This is equivalent to discarding messages with $1-p_i$. Thus all the agents approximately receive the same amount of information regardless of their degree (number of neighbors), leading to improved group performance. 

Now consider the message passing case. In this paper, we consider the case where each message is forwarded at most $\gamma$ times through $G$. This is equivalent to the case where each agent can send messages only up to $\gamma$ hop neighbor. When  $\gamma = d_\star(G)$ (diameter of the graph $G$) we have that $G_{\gamma}$ is a complete graph. And we have $p_i=\frac{d_{\min}(G_{\gamma})}{d_i(G_{\gamma})}=1$ and agents do not discard any message. However, when $\gamma < d_\star(G)$ the graph  $G_{\gamma}$ is not complete. Therefore agents receive different amounts of information which are approximately proportional to the degree distribution of $G_{\gamma}.$ As explained in the previous paragraph this information disparity leads to a performance disparity among agents. As a result group performance decreases.  In this case we design the algorithm such that each agent $i$ discards messages with $1-p_i$ where $p_i=\frac{d_{\min}(G_{\gamma})}{d_i(G_{\gamma})}.$ This regulates the information flow mitigating the bias introduced by information disparity. As a result, the group obtains near-optimal performance.

In summary, information disparity created by different agents receiving different amounts of information leads to poor group performance. We use message discarding to reduce the information disparity introduced by the degree distribution of the network and communication parameter $\gamma.$ Hence our algorithms obtain improved performance. 



 We would like to sincerely thank the reviewer for their time and answer questions below.

**Justification for basic setting - cooperative multi-armed bandits:** It is true that in the setting we consider in this paper each agent can learn the best arm by itself by using any state-of-the-art ${\tt UCB}$ algorithm without any communication. However in this case a group of $N$ agents suffer a regret equivalent to $N$ times the regret suffered by any individual agent. For single agent case ${\tt UCB}$ type algorithms obtain a dominant regret term $O(\sum_{k>1}\frac{\log T}{\Delta_k}).$ Then without communication $N$ agents obtain a regret of $O(N\sum_{k>1}\frac{\log T}{\Delta_k}).$ However with communication agents obtain more information and obtain significantly improved performance. When agents communicate over a complete graph ($\bar{\chi}(G)=1$) with perfect communication ,i.e. $p=p_i=1$ group obtains a regret of $O(\sum_{k>1}\frac{\log T}{\Delta_k}).$ Thus we see that with communication, the group suffers a significantly smaller regret compared to the no communication case, and obtaining optimal regret in the networked case is a well established and extensively studied problem in multi-agent and network optimization, please see the references below:

Cesa-Bianchi, N., Cesari, T. R., and Monteleoni, C. Cooperative online learning: Keeping your neighbors updated. arXiv preprint arXiv:1901.08082, 2019a. \
Cesa-Bianchi, N., Gentile, C., and Mansour, Y. Delay and cooperation in nonstochastic bandits. The Journal of Machine Learning Research, 20(1):613–650, 2019b.\
Landgren, P., Srivastava, V., and Leonard, N. E. On distributed cooperative decision-making in multiarmed bandits. In 2016 European Control Conference (ECC), pp. 243–248. IEEE, 2016a. \
Landgren, P., Srivastava, V., and Leonard, N. E. Distributed cooperative decision-making in multiarmed bandits: Frequentist and bayesian algorithms. In 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 167– 172. IEEE, 2016b. \
Landgren, P., Srivastava, V., and Leonard, N. E. Social imitation in cooperative multiarmed bandits: Partition Based algorithms with strictly local information. In 2018 IEEE Conference on Decision and Control (CDC), pp. 5239–5244. IEEE, 2018.\
Dubey, A. and Pentland A. "Kernel methods for cooperative multi-agent contextual bandits." International Conference on Machine Learning. PMLR, 2020.\
Dubey, A. and Pentland A. "Cooperative multi-agent bandits with heavy tails." International Conference on Machine Learning. PMLR, 2020.\
Lalitha, Anusha, and Andrea Goldsmith. "Bayesian Algorithms for Decentralized Stochastic Bandits." IEEE Journal on Selected Areas in Information Theory 2.2 (2021): 564-583.\
Madhushani, Udari, and Naomi Ehrich Leonard. "Heterogeneous stochastic interactions for multiple agents in a multi-armed bandit problem." 2019 18th European Control Conference (ECC). IEEE, 2019.\
Madhushani, Udari, and Naomi Ehrich Leonard. "Heterogeneous explore-exploit strategies on multi-star networks." 2021 American Control Conference (ACC). IEEE, 2021.

**Unknown variance proxy:**  It is typical to assume that an upper bound for variance proxy is known in cooperative bandits, please see the references presented above, which all assume an upper bound on the variance proxy of each arm. However, we agree with the reviewer that having arms with a large difference in variance proxy will lead to poor regret bounds. This can be avoided by simply assuming that variance proxy of all the arms are known to agents. The results for this can be obtained by simply replacing $\sigma$ with $\sigma_k$ in our bounds (Remove $\sigma$ and use $\sum_{k>1}\frac{\sigma_k}{\Delta_k}$ instead of $\sum_{k>1}\frac{1}{\Delta_k}$). Further, extending this to the case with unknown variance proxy values is a direct extension of our current results. This directly follows from combining our proof techniques with variance proxy estimation methods present in single-agent bandit algorithms such as [1] and [2], see below. Since the focus of our work is studying communication imperfections, we refrained from using unknown variances to avoid algebraic complications.  

[1] Audibert, Jean-Yves, Rémi Munos, and Csaba Szepesvári. "Exploration–exploitation tradeoff using variance estimates in multi-armed bandits." Theoretical Computer Science 410.19 (2009): 1876-1902.\
[2] Cowan, Wesley, Junya Honda, and Michael N. Katehakis. "Normal Bandits of Unknown Means and Variances." J. Mach. Learn. Res. 18 (2017): 154-1.

**Psuedo code for ${\tt RCL-LF}$:** We have provided the pseudo-code for ${\tt RCL-LF}$ in response to all reviewers. We will add this in the revised version of the paper.

**Explanation/validation of Remark 1:** Substituting $p=p_i=0$ to Theorem 1 reduces the dominant regret term to $O(N\sum_{k>1}\frac{\log T}{\Delta_k}).$ Substituting $p=p_i=1$ to  Theorem 1 reduces the dominant regret term to $O(\bar{\chi}(G)\sum_{k>1}\frac{\log T}{\Delta_k}).$ Thus result given in Theorem 1 recovers the regret bounds for $p=p_i=0$ (no communication) and $p=p_i=1$ (perfect communication over a complete graph, i.e. $\bar{\chi}(G)=1$.) respectively. Further the result given in Theorem 1 can be slightly improved by replacing $\bar{\chi}(G)(\max_{i\leq N}p_i)$ with $\sum_{\mathcal{C}\in \mathfrak{C}}(\max_{i\in \mathcal{C}}p_i),$ where $\mathfrak{C}$ is a non overlapping clique covering of communication graph $G$. (This result follows from equation 17.) Then we obtain $\sum_{i=1}^N(1-p_ip)+\sum_{\mathcal{C}\in \mathfrak{C}}(\max_{i\in \mathcal{C}}p_i)p=N-p(\sum_{i=1}^Np_i-\sum_{\mathcal{C}\in \mathfrak{C}}(\max_{i\in \mathcal{C}}p_i)).$ Since the clique covering is non overlapping  the results show that agents obtain improved group performance for any communication probability $p>0$ for any non trivial graph compared to no communication case where each agent learns by itself. In the final version we will revise Remark 1 to improve clarity. 

**Explanation/validation of Remark 2:** Theorem 1 provides a general result which is valid for any $p_i\in [0,1].$ Here we choose an optimal set of $p_i$ values, i.e. $p_i=\frac{d_{\min}}{d_{i}}$ in order to optimize group performance. As stated in the first two paragraphs of Section 3 in the main paper the amount of information received by agents varies according to the degree distribution of the network $G,$ which defines the number of neighbors. Well-connected agents (agents with a high degree) receive more information and perform better than poorly connected agents. As a result of this disparity group performance decreases. The effect of information disparity is exacerbated in the presence of link failures. Thus in order to regulate the information flow in the network in our algorithm each agent, $i$ incorporates received information with a probability inversely proportional to their degree (i.e. $p_i=\frac{d_{\min}}{d_{i}}$). This is equivalent to discarding messages with $1-p_i$. Thus all the agents approximately incorporate the same amount of information regardless of their degree (number of neighbors), leading to improved group performance. It is reasonable to assume that each agent knows their own degree. In addition, we only assume that each agent knows the minimum degree of the network. In the final version, we will revise Remark 2 to improve clarity. 

**Explanation/validation of Remark 3:** Similar to Remark 2, in Remark 3 also, we aim to choose an optimal set of $p_i$ values in order to optimize group performance by reducing information disparity among agents. In ${\tt RCL-LF}$ with message passing, each hop in the network adds a delay of 1-time step when forwarding messages. This makes it difficult to find an optimal set of $p_i$ values. However when messages are forwarded without a delay we can obtain near-optimal performance by choosing $p_i=\frac{d_{\min}(G_{\gamma})}{d_i(G_{\gamma})}.$ Note that when agents pass messages without a delay this is equivalent to agents communicating over the network $G_{\gamma}.$ Thus performance improvement follows from the same reasoning given in Remark 2. In the final version, we will revise Remark 3 to improve clarity. 

**Section 5**: The algorithm ${\tt RCL-RC}$ can be considered to be “centralized” as some agents mimic other agents. Indeed, the set of exploring agents can be any arbitrary vertex cover of the graph G. However, if we select a large exploring set $\mathcal I$ of size $n$, the regret incurred will scale proportionately, and hence we select the smallest vertex cover, i.e., the minimal dominating set. Please see remarks 5 and 6 for more detail on how the regret is affected by the size of the exploring set of agents. In all cases, *any reward can be corrupted*, regardless of the agent obtaining the reward.


 Please find the pseudocode for RCL-LF below.

Input: arms $k\in [K],$ variance proxy upper bound $\sigma^2,$ parameter $\xi,$ message discarding probability $p\_i,\forall k\in [K]$
Initialize: $N\_k^{i}(0)= \widehat{\mu}\_k^{i}(0)=C\_k^{i}(0)=0, \forall k,i$

for each iteration $t \in [T]$ do\
&nbsp;&nbsp;&nbsp;		for each agent $i \in [N]$ do\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		If $t=1$ then\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;			$A\_{t}^{i}\gets {RandomArm} \left([K]\right)$\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		else\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;			$A\_{t}^{i}\gets\arg\max\_{k}\widehat{\mu}\_k^{i}(t-1)+C\_k^{i}(t-1)$\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		end if \
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		$Create(\mathbf{m}\_t^{i}:=\langle A\_t^{i},r\_t^{i},i,t\rangle)$ \
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		$Send(\mathbf{M}^{i}\_t \gets \mathbf{M}\_{t-1}^{i} \cup \mathbf{m}\_t^{i})$ \
&nbsp;&nbsp;&nbsp; 		end for\
&nbsp;&nbsp;&nbsp;		for each agent $i \in [N]$ do\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		for each neighbor $j \in \mathcal{N}_{i} (G_\gamma)$ do \
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;			Discard messages with probability $1-p_i$\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;			for each message $\mathbf{m} \in \mathbf{M}_t^j$ do\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;				with probability $p\_i,$ $\mathbf{M}\_t^{i}\gets \mathbf{M}\_t^{i} \cup \mathbf{m}$\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;				with probability $1-p\_i,$ $\mathbf{M}\_t^{i}\gets \mathbf{M}\_t^{i}$\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;			end for \
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		end for \
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		for each arm $k \in [K]$ do\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		$Calculate   \left(N\_k^{i}(t), \widehat{\mu}\_k^{i}(t),C\_k^{i}(t)\right)$\
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;		end for\
&nbsp;&nbsp;&nbsp; end for\
end for



 Thank you for the constructive feedback! In the following, we address your concerns individually.

**Probability of communicating messages $p$ is unknown:** There is a typo in line 45. It should be corrected as “each edge being present in $G$ with an unknown probability $p$.” In ${\tt RCL-LF}$ each agent $i$ discards incoming messages with probability $1-p_i$ and probability of communicating messages $p$ is unknown to agents. We have provided the pseudo-code for RCL-LF in response to all reviewers. 

**Addressing minor comments:** We will fix all the typos in the revised version.


Meta-review: The paper investigates 3 scenarios in the multi-agent cooperative bandit model where the communications between agents is not perfect. 
In particular, the paper covers the cases when: (i) communication links fail; (ii) communications are delayed; and (iii) the communication messages are corrupted. For teach of these cases, the author propose a UCB based algorithm that exploits the graph structure of communication channels, and show that group regret is smaller than the sum of individual regrets (when agents chosen not to collaborate at all), even in the case of imperfect communication. That is, it's better to form a grand coalition rather than acting independently.

While the majority of the reviewers found the responses of the authors to be sufficient and have clarified most of their concerns, the 3rd reviewer still argues that the results (especially Theorems 1 and 3) are rather incremental. Let me express my own opinion here that I respectfully disagree with this reviewer, as I think each of these problems themselves is quite essential and therefore worth investigating. The proof techniques might not be too difficult, but I also agree with one of the reviewers that this should be a positive thing, rather than a reason for rejection (if the motivation + positioning of the problem is well justified). Therefore, I would like to recommend the opinion of the majority of the reviewers, that is, to accept this paper (although as a poster).

Decision: accept