Hijacking Robot Teams Through Adversarial Communication
Authors: Zixuan Wu
Sean Charles Ye
Byeolyi Han
Matthew Gombolay
Venue: CoRL 2023 Oral (2023)
Keywords: Adversarial Attacks
Multi-Agent Reinforcement Learning

Abstract: Communication is often necessary for robot teams to collaborate and complete a decentralized task. Multi-agent reinforcement learning (MARL) systems allow agents to learn how to collaborate and communicate to complete a task. These domains are ubiquitous and include safety-critical domains such as wildfire fighting, traffic control, or search and rescue missions. However, critical vulnerabilities may arise in communication systems as jamming the signals can interrupt the robot team. This work presents a framework for applying black-box adversarial attacks to learned MARL policies by manipulating only the communication signals between agents. Our system only requires observations of MARL policies after training is complete, as this is more realistic than attacking the training process. To this end, we imitate a learned policy of the targeted agents without direct interaction with the environment or ground truth rewards. Instead, we infer the rewards by only observing the behavior of the targeted agents. Our framework reduces reward by 201% compared to an equivalent baseline method and also shows favorable results when deployed in real swarm robots. Our novel attack methodology within MARL systems contributes to the field by enhancing our understanding on the reliability of multi-agent systems.

Hijacking Robot Teams Through Adversarial Communication
Zixuan Wu
Byeolyi Han Georgia Institute of Technology, Atlanta, GA, USA {zwu380, seancye, bhan67, Soambbllayy@ggtecchh edu
Sean Ye
Matthew Gombolay
Abstract:
Communication is often necessary for robot teams to collaborate and complete a decentralized task. Multi-agent reinforcement learning (MARL) systems allow agents to learn how to collaborate and communicate to complete task. These domains are ubiquitous and include safety-critical domains such as wildfire fight- ing, traffic control, or search and rescue missions. However, critical vulnerabil- ities may arise in communication systems as jamming the signals can interrupt the robot team. This work presents framework for applying black-box adver- sarial attacks to learned MARL policies by manipulating only the communication signals between agents. Our system only requires observations of MARI. poli- cies after training is complete, as this is more realistic than attacking the training process. To this end, we imitate a learned policy of the targeted agents without direct interaction with the environment or ground truth rewards. Instead, we infer the rewards by only observing the behavior of the targeted agents. Our frame- work reduces reward by 201 % compared also shows favorable results when deployed in real swarm robots. Our novel at- tack methodology within systems contributes to the field by enhancing our understanding on the reliability of multi-agent systems.
equivalent baseline method and
Keywords: Adversarial Attacks, Multi-Agent Reinforcement Learning
Introduction
Effective communication among robots essential for information exchange, collaboration, and collective decision-making It plays a vital role in various roboties domains, including collaborative manipulation [1] and multi-robot navigation [2]. Ensuring reliable and secure communication is crucial for maintaining the overall system's 's performance, safety, and integrity.
Multi-agent reinforcement learning (MARL) has been a powerful tool to train agents in complex domains but the literature is lacking in studies about the vulnerabilities and defenses of these sys- tems. MARL techniques that utilize communication have been widely applied to scenarios where multiple agents need collaborate for shared goal in robotics tasks such as autonomous driving [3, 4, 5] and path planning [2. 7]. Researchers examined various aspects of communication in MARL, including when to communicate [8], who communicate with [9], and different types of graph-structured communication [10, 11]. Some of these frameworks use binarized communication to pursue Low-Size, -Weight, and -Power (Low-SWAP! systems [11] which causes agents to com- municate in a highly efficient manner. Malicious actors can compromise these systems and endanger the lives of many [12].
There are only few works that assume the communication channel can be imperfect [13] can be attacked [14] paralleling with its computer vision counterpart [15, 16. 17, 18, 19], which makes it crucial to understand these weaknesses such that we can create appropriate defense mechanisms. In this work, we learn to attack the communication signals within a multi-robot team discretely
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

010010101
Attacked
001110101
000100110
000101010
101010101
Expected
011010100
Message Senders
Adversarial Communication Channel
Message Receivers
Adversarial Communication Pipeline: Multi-agent team (left) communicates information for decentralized coordination; an adversarial system (middle) learns model of the teams' activities and communication patterns and (right) broadcasts counterfeit team messages to trick team members towards pursuing low-prion low-priority activities.
Figure I:
without any trace of the training process on the target robots. First, we learn surrogate policies from the observation, messages, and actions of the target robots which are accessible from malware or insecure networks [20, 21, 22, 23, 24]. Second, we estimate the agent rewards from their behaviors instead using rewards from the environment. Finally, we use an actor-critic framework completely offline to learn how to hijack the targeted system without environment interactions. Our method requires the least prior knowledge compared to prior work and results in robots traversing the wrong location and drastically hindering team performance.
Contributions:
1. We propose actor-critic framework that enables adversarial policy to learn without direct interaction with the target agents the environment nor the ground truth agent ward. Our framework manipulates the behaviors of target robots with the communication attacking strategy learned through surrogate target policies and transferable to real ones. Additionally, we introduce a differentiable framework for training adversarial communica- tion policies that can modify digital communication signals [10, 11].
2. We demonstrate the effectiveness of our algorithm three distinct multi-agent domains: predator-prey. prey. and ppeaker-listener. Across these domains, method surpasses the baseline approach by reducing the reward of the target agents by 465% compared to a baseline approach.
partially observable predator
3. We validate the applicability of our algorithm physical swarm robots in the Robotarium [25]. By acting as a strong adversary, our method reduces the reward achieved by the target agents by an average of 201 % across all three environments compared to a baseline strategy, which employs an equivalent random flipping approach.
Related Work
In this section, describe how multi-agent reinforcement learning can be used control robots in Dec-POMDP setting. We also provide an overview of the role of communication in MARL frame- work and highlight the vulnerability of communication to adversarial attacks, leading to potential system failures and safety risks.
2.1 Multi-Agent Reinforcement Learning
In recent years, communication has played a crucial role enhancing coordination and collaboration
among robots in multi-agent reinforcement learning (MARL) frameworks [8, 9, 10, 26, 11, 27, 28).
Compared to previous works in learning policies [29, 30], recent MARL frameworks have enabled learning in more complex environments and train multiple agents. These MARL communication frameworks have been used in several robotics applications such as multi-robot path planning [2] and cooperative driving [31]. Various approaches have been proposed, including trainable differen- tiable communication channels [26, 27]. partially observable environments [28], and soft-attention networks for selective communication 9]. However, the effectiveness of communication is threat- ened by adversarial attacks, which can lead system failures and safety risks, particularly in do- mains like self-driving vehicles [32]. This paper aims to evaluate the robustness of communication

---

in MARL systems, building upon prior advancements in communication techniques of the binarized communication approach [11] that improves bandwidth efficiency. Key to note is that our adversar- ial policy does not make any assumptions on targets other than inter-agent binarized communication. Our adversarial attack can hypothetically utilized any framework as long as we can learn good surrogate policies of target agents. We aim show the generality of our approach in future work.
2.2 Adversarial Attacks in MARL and Communication
Adversarial attacks were first studied within the context of computer vision, where small perturba- tions to the input could induce faulty outputs [18]. Adversarial attacks aim to deteriorate model
performance in tasks like classification [16, 17, 19], segmentation [33, 34, 35], or object detec-
tion [36, 37]. These ideas were later extended reinforcement learning [38, 39], altering agent actions through perturbations in enviromment observations [40].
In adversarial attacks, two common categories are white-box attacks, which assume knowledge of neural network weights, and black-box attacks, which assume limited model parameter information. White-box attacks typically optimize objectives using methods such as Fast Gradient Sign Method (FGSM) [16, 17] Projected Gradient Descent (PGD) [19]. Meanwhile, black-box attacks rely on surrogate models that approximate decision boundaries such that the adversarial attacking targeting on it could be transferred to the original models. Input-output pairs in black-box scenarios can also be augmented by FGSM [18] or PGD [19] to generate a synthetic dataset that induces similar decision boundaries. We utilize black box attacks with augmentation similar to FGSM for first-order approximation of our surrogate models.
Prior work [41] has trained adversarial communication protocol in a multi-agent setting by using reinforcement learning agent to optimize the adversarial policy. However, this approach requires direct interaction with the environment and is impractical as the training process would be easily detected by observers of the system. To address this limitation, we propose a black box setting and employ transfer attacks [42], training a surrogate model to mimic the target model. Furthermore, our work distinguishes itself by assuming a "man-in-the-midlll"" attack, where an interceptor subtly flips the aggregated binarized communication vectors. This is in contrast to previous assumptions of single target victim agent with limited number (s N of potentially malicious messages [14].
3 Problem Formulation
We ground our problem in a Decentralized Partially Observable Markov Decision Process (Dec- POMDP) formalism which is a 10-tuple of (S, M, A, P, R. Y, O, II, 7). S is the state set of the environment and M is the message state set. For each agent í E == J,..., N tte agent chooses an action a; E A at state s; E S. The transition function is denoted by S( A). Each agent has its own reward based on global state-actions r; = R₂ A) and y E [0,1] as the discount factor. Since the environment is partially observable, each agent also has its Va E T which is produced by the observation function r ( Agents can have two policies: an action policy πद (a.|Tz) and a message policy πद (m? TT). These are both conditioned on their own partial observations and the messages received from other communicative agents {2r, M m,
individual partial observation
Adversarial policy I2 ady!
Environment
Observation Y
Madv = 8 O Mout
Message policy x
-mmg My m4)
Action policy x
partial observable agents detection ranges
A (ug Ag ... aq) Figure 2: Adversarial Policy Test-Time: Our adversarial attacking policy addv changes the mes- sages from the message policy x" such that the receiving agents are maximally disrupted.
3

---

We model single adversarial policy T, Moo) that has access to the partial observations Y, message outputs Mout of all agents and produces adversarial signal The partial observations could be gathered by the adversary by shadowing the relevant targeted agents. We generate a mali- cious revised communication, denoted as Made, by combining the perturbation with the original messages M sent to each agent, formulated as Madv 8 @ Mout, Figure 2 shows how the adversarial policy godv influences the target policies. Here consider the transition and policies as deterministic ones: S" P(S,A), (T,), ma (T.)) and d rodv Mout).
4 Methodology
In this section, we define the design choices for training adversarial communication policy. The goal of the policy is to minimize the reward of the victim agents while minimizing the difference between the original and tampered communication vector. Algorithm I provides an overview of our overall training procedure: 1) Learning a surrogate policy, 2) Learning an actor-critic for the ad- versarial policy, and 3) Updating the actor with differentiable binary communication. We presume access to the observations, messages, and actions of the target agents, assuming that we have suc- cessfully intercepted the communication protocol the multi-agent system. However, we do not assume access to ground truth rewards. We also assume binary communication channel of 16-bits but further studies could extend our work to remove this assumption.
Algorithm 1: Adversarial Communication Pseudocode
Input: ((T(Observ)iions) M/ (Messages), A(Actions))
for i = 0, 1, 2.. do for a= N do
Sample batch of observation, message, actions: (uy, eg) ~ Y, (m). ~ M. ~ A Update the surrogate policy " (aylus,
end
Compute targets
Radv Σ log (aauaamaaaaaaa (aylus, (agl
(1)
y == Radv
+ Qu (T', A''l A' Alt
(2)
Update the adversarial critic with step of gradient descent
c ő (o(Qu - y) Update the adversarial policy with one step of gradient descent
(3)
- 0 + Vo Asde Cstip)
end
First, we train surrogate policies to imitate the real policies (Section 4.1). We can then learn Q- function for the adversarial policy by assigning rewards based upon the surrogate policies (Eq. 1)
and using the Bellman equation to update the critic (Eq. 2. 3). Details are described in Section 4.3.
Finally, the adversarial policy is updated with the differentiable binary flipping mechanism (Eq. 4) and described in Section 4.2. We include hyperparameters Appendix B.2 for more details.
4.1 Learning a Surrogate Policy
As our method black-box attack, we assume we do have access the ground truth policies of the agents attacking. To this end, learn surrogate policy (л ) for each agent attacking, where assume access to dataset A) consisting of observations,

---

communication, action pairs from victim agents. The surrogate policies are used in two ways as described in the next section: 1) as a reward signal to learn a critic and 2) as a mechanism to simulate the adversarial output into real agent actions. To obtain a similar first-order approximation of our surrogate policies to the real policies, we augment our dataset with neighboring data by adding small Gaussian noise each input to produce augmented input-output pairs. We find that this augmentation is enough to obtain a useful surrogate policy and thus there no need to achieve higher precision using much more complicated methods like FSGM augmenting [18]. We use behavioral cloning methods to minimize the log-likes ee-llikeihhod between actions given the agent observations and messages from other agents (log p(a m)).
4.2 Differentiable Targeting of Binary Communication Channels
In binary communication, each bit has two states: O and 1. Modifying bit involves flipping it to the other state. To make this process differentiable, we need to parameterize it similarly to its continuous counterpart. We define the adversarial modified message Made the composition of the original communication vector Mout and the parameterized modification 8 [41].
Two methods are available to parameterize the adversarial policy. One method is to directly output the adversarially revised communication vector as Made = 8 gadv (Mout, v)) which we call the "direct" form. The second method is called "flipping" whose adversarial policy output is used to indicate which digit to flip such that we can write it in an XOR form using boolean algebra in Equation 5 where means pointwise multiplication.
Made = 5- Mout + 3 Mout 8. 5- (1-1 + (1 (1-6) M
(5)
4.3 Learning an Actor-Critic for Adversarial Communication
Our goal for the adversarial policy is to secretly train itself without any interaction with the vironment in black-box setting such that the adversarial training process does not induce any abnormalities and cannot be detected. This requirement raises higher standard than recent work [41], where the adversarial policy is trained with reinforcement learning and requires environment interactions of the adversarial policy's ss actions. We adapt the actor-critic framework to learn I) a critic (T, A)) that learned within the observation-action (T, A) space of the target agents and 2) an actor policy that distorts the communication messages. We use the actor-critic framework allows us to utilize the original Dec-POMDP of the target agents for the critic rather than building new separate MDP within the space of the observations and messages actions. We train our Q-function o (T, A) to use the observation-actions of all agents in the environment based on the Bellman equation and TD eгror as shown in Equation 6.
((0) = EY,A, Y [(Q
1) y = Rade + 7Qs.torg A')A A' (T()
(6)
A question naturally arises: how do we get the reward Radv to train this Q-function? Because we do not assume access to ground-truth rewards as previous literature does, we cannot utilize the negative mean of all agent reward Radv - - Ei= T4 to optimally degrade performance on their own metrics. Instead, assign the reward for certain state-action pair of all agents to be the inverse of the log probability of the optimal action from the surrogate policy (Eq. 7).
1 Σ log(=;)" suT (a (a,lu;.m))
pado
Intuitively, we are driving the critic to punish state-action pairs visited by the target policies. Another interpretation is that the probability of the samples' appearance is proportional to the exponential of the reward in inverse reinforcement learning (IRL) theory [43, 44].
Given a well-trained critic, we can learn the adversarial policy adde adde (8) Mouut to modify the com- munication messages to maximize the Q-function minus bit flipping penalty Cytip = L, (8). We
utilize the surrogate policies again to produce hypothetical actions given the modified communica- tion vectors.
8 adu (T,Mout Mouu,, ad 5. 5-(1 (1 (1-Mouy Mutt + (1 - 6) out and 11 (v,, my medu)
(8)
5

---

The adversarial policy can then be updated through automatic differentiation to produce messages that disturb the surrogate policies. With our method, we can train the adversarial communication policy completely offline with the only assumption being that we have intercepted some observation, message, and action pairs from the target policies.
Results and Discussion
5.1 Environments
We utilize three domains originally proposed by the MADDPG [28] paper and modified for our use: Predator-Capture-PPee Partially Observable Predator-Prey, and Speaker-Listens ments, the goal is to maximize the number of collisions between the agents and the target. Further details of these environments included in Appendix B.I.
all environ-
Predator-Capture-PPee (PCP) In this environment, team of agents must capture an adversary prey opponent. To emphasize the role of communication in this domain, capture agents cannot see any other agents and have to make decisions based on the received messages from all observing agents.
Partially Observable Predator-Prey (PO-PP) We modify the predator-pon prey environment such that all predator agents can only receive the location of the prey when they are within distance d of the prey. The agents must communicate with each other to locate the position of to the prey. We remove the capture agents from this environment.
Speaker--istener (SL) In speaker-listener, team of two agents, consisting of a speaker and listener must work together for the listener reach target color destination. The speaker must communicate the target color to the listener and the listener must then go to the color destination.
5.2 Adversarial Communication Validation
In this section, we validate our secret adversarial communication channel performance by com- paring with a random flipping method where we flip the same number of bits as the adversarial communication. We ensure that these two methods are always flipping the same number of bits in the communication and compare the results across various numbers of bits flipped. The random flipping baseline has been used in prior work [14] and represents a non-adaptive black-box attacker. Therefore, we modify the adversarial policy loss function to L = -Q(u, a, m) + a 1/N, (0), which regularizes the policy loss by the average sum bits flipped. The coefficient a is used to balance the adversarial policy performance with the number of bits flipped and we control the regu- larization speed by annealing a = an max(n, 3, 0)/e (where ne is the training episode number, is the regularizer intercept, and is the regularizer slope) to fine-tune the bits flipped. We evaluate the reward and collision statistics by rumning 50 episodes for each adversarial policy checkpoint. Training curves with the regularized loss and hyperparameteres can be found in Appendix B.2.
Figure shows the rewards and collisions versus the episodes from which we get our adversarial policy in the three environments. Agents' reward and collisions from the adversarial communication increase as the number of bits flipped decreases but always less than those from random flipping. This result validates the adversary property of method and distinguishes it from random noise with the same magnitude. We also find that the gap between the two methods is lower when the number of bits flipped is lower, which is reasonable since it is difficult for adversarial policy to attack the critical combinations of the communication digits in such a limitation. Interestingly, the random flipping curve nearly remains horizontal in the PO-PP enviromment, which means that random attack does not work all regardless of the number of bits flipped. This is because the predators do not only rely on communication but also on their own observations take actions, therefore, irregularly changing the communication will not confuse agents much, compared to the adversarial policy which flips crucial bits and guides the agents to low-reward regions.

---

Com
Predaton-Capture-Pe
Partial Observable Pradetor-Prey (PO-PP)
Speaker-Listenal
0.2
Average Reward
-0.3
Average Reward
-0.2
Average Reward
-0.3
-0.4
-0.4
-0.5
-0.5
-0.6
-0.8
-0.7
-0.9
10
15
20
-1.0
10
15
20
25
-0.0
10
12
14
Bita Flipped
Bita Fllpped
Bits Flliped
Average Rewards in PCP, PO-PP and SL respectively
Predntor-Capture-ee Precrre--rryyree (PCP)
Predator-Prey
Number of Collisions
SpstkerrListener (SL)
0,150
Collisionn
0.25
0.125
0.125
0.20
0.100
0.15
0,075
0.375
0.10
0.050
Avorage
0,050
0.025
0,05
0.000
0.000
Bite Fllpped
Bitts
Bits Filpped
(b) Average Collisions in PCP, PO-PP and SL respectively
Figure 3: The figure illustrates the comparison between the adversarial policy (blue) and random flipping (red) in terms of average reward and number of collisions. Consistently, the adversarial communication policy degrades team performance more effectively than the random policy across all bit flip counts, leading lower rewards and lower number of collisions for agents.
5.3 Comparison of Adversarial Message Parameterization
In this section, we compare our adversarial policy, between the "flipping" regularization strategy (as described in Section 4.2), and the "direct" strategy in terms of the normalized adversarial policy performance score (Sc) which denotes how much worse an agent performs per flipping a single digit (Appendix B.3). The training and testing settings are the same as section 5.2 and we record the maximum normalized score of the flipping method (ours) the direct method (Table 1).
Table 1: Reward and Collision Normalized Scores
PCP
PO-PP
SL
Reward Sc
Collision Se
Reward Se
Collision Se
Reward Se
Collision Se
Adv[Ours] Adv[Direct]
0.11
4.58=2.23
0.04:00.2
1.45
0.13:00.06 0.11+00.05
31.38:6.51 30.72++6.19 13.68
0.01+-0.01
1.07++0.14
0.02+-0.01
0.80+00077
Random
0.01
0.96++0.29
0.00
0.30++0.88
0.04++0.02
In the PCP and PO-PP environments, the direct strategy exhibits significantly worse reward and collision scores, performing at 90.9% and 76.64% lower, respectively, compared to approach.
This discrepancy arises from the failure of the direct strategy to effectively balance lowering agent performance and reducing flipped bits. In these environments, the direct strategy results in 38.7% and 65% bits flipped, whereas our approach achieves 7.5% and 28.3% of bits flipped. Interest- ingly, in the SL environment, where there is a single message sender, one receiver, and 16 bits of communication, the direct strategy performs relatively better due the simpler balancing of perfor- mance and regularization terms. Nevertheless, our approach still outperforms the direct strategy by 18.18 18188 and 2.15% in terms of reward and collision, with 8. 3.15% and 8.29% of bits flipped, respec- tively. These findings highlight the critical importance of the flipping representation in facilitating backward gradient computation.
5.4 Robotarium Physical Robot Demonstration
We demonstrate results physical swarm robotics system (details in Appendix A). We utilize based position control to drive each robot according to their policies both with the adversarial
7

---

communication intervention and random flipping intervention. We show that our adversarial com- munication policy drastically reduces reward in all three environment settings (Table 2) with the same number of bits flipped per episode and averaged over three episodes.
Table Attacked Agents Reward and Collision
PCP
PO-PP
SL
Reward
Collision
Reward
Collision
Reward
Collision
Adv[Ours]
0.31
0.05:-0.23 0.29110.5
0..1
0.05+0.23
-0.43-+0.14
0.00
Random
0..1311.33
0.02-10.56
0,20:0.40
-0.21
0.34100.7
Prey
Prey
Predator
Predator
Predator
Predator
Capturer
Capturer
Capturer
Capturer
Prey
Prey
Pray
Predator
Prey
Predstor
Predator
Casturer
Capturer
Capturer
Capturer
Figure 4: PCP Demonstration: Adversarial Communication (top) vs Random Flipping (bottom)
6 Limitations and Future Work
Our work relies on the assumption that the surrogate policies can accurately model the behaviors of the true policies and depends on using the agent policies estimate the ground-truth reward function. A limitation of this approach is the amount of training data required imitate the surrogate policy, where the target agents may detect that information is being collected. Additionally, IRL methods [44] could be used to learn a reward function that better reflects the ground truth reward.
Our work has important ethical implications as could be used to attack important systems but can improve the community's s ability to help improve the robustness these systems by characterizing the vulnerabilities. This work does assume any strategies for the target agents defend against adversarial attacks. A defender may include parity bit indicating whether the total number of 1-bits is even or odd defend against attacks the communication. Additionally, defenders could learn response strategy by adjusting its communication scheme if they had access to previous experiences with the attackers. Finally, if the adversarial attack is conducted on multi-agent system with interpretable communication vectors, it may be easy to identify that messages have been altered. We leave adversarial attacks in this setting for future work.
7 Conclusion
We introduce a practical adversarial communication policy that does not need direct environment in- teraction, enhancing the feasibility of adaversarial attacks. Our method also utilizes robust approach for estimating agent rewards from observing behaviors only, without reward information. Lastly, we pioneer a differentiable method for adversarial communication discrete binary channels, flipping bits for improved attack efficacy. Our algorithm validated on real swarm robots in the Robotarium platform. This showcases the versatility and real-world applicability of our approach. Our frame- work opens avenues for enhanced security and robustness in multi-agent systems, with potential implications across various domains.

---

Acknowledgments
We wish thank our reviewers for their valuable feedback in revising our manuscript. We also thank Letian Chen for his expertise in inverse reinforcement learning, which inspired many of our ideas. Additionally, both Rohan Paleja and Esi Seraj provided invaluable insights on multi-agent reinforcement learning. This work was sponsored by the Naval Research Laboratory under grant number N00173-11-1-G009
References
[1] L. E. Parker, D. Rus, and G. S. Sukhatme. Multiple mobile robot systems. Springer handbook of robotics, pages 1335-1333 1384, 2016.
[2] Q. Li, W. Lin, Z. Liu, and A. Prorok. Message-aware graph attention networks for large-scale multi-robot path planning. IEEE Roboties and Automation Letters, 6(3):5533-5540, 2021. doi: 10::10.1109/LRA 20000099
[3] Q. Jiang, M. Qin, S. Shi, W. Sun, and B. Zheng. Multi-agent reinforcement learning for traffic signal control through universal communication method. L. D. Raedt, editor, Proceedings of the Thirty-Fine First International Joint Conference on Artificial Intelligence, IJCAI 2022, Viemna, Austria, 23-29 July 2022, pages 3854-3860. ijcai.org, 2022. doi:10. (0..2446/jjaaii0222222
[4] E. Van der Pol and F. A. Oliehoek. Coordinated deep reinforcement learners for traffic light control. Proceedings of learning, inference and control of multi-agent systems (at NIPS 2016), 8:21-38, 2016.
[5] B. Liu and Z. Ding. A distributed deep reinforcement learning method for traffic light control. Neurocomputing. 490:390-399, 2022. ISSN 0925-2312. doi.https://ooi.org/II wtt.///////.............. 10110166))) neucom.
2021.11.106.
[6] H. Qie, D. Shi, T. Shen, X. Xu, Y. Li, and L. Wang. Joint optimization of multi-uav target assignment and path planning based on multi-agent reinforcement learning. IEEE access, 7: 146264-146272, 2019.
[7] M. Zolfpour-Arokhlo A. Selamat, Z. Mohd Hashim, and H. Afkhami. Modeling route planning system based on value-based dynamic programming with multi-agent reinforce- ment learning algorithms. Engineering Applications of Artificial Intelligence, 29:163-177,
2014. ISSN 0952-1976. doi:https://doi.or/11... 10t.:/..1116/...e..... www.pjaaaaapppa........m. 200ppp........00101 2014.01 .001.
[8] A. Singh, T. Jain, and Sukhbaatar. Learning when to communicate scale in multiagent cooperative and competitive tasks. In Proceedings of International Conference on Learning Representations, 2019.
[9] A. Das, T. Gervet, J. Romoff, Batra, D. Parikh, M. Rabbat, and J. Pineau. TarMAC: Targeted multi-agent communication. In K. Chaudhuri and Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1538-1546. PMLR, 09-15 Jun 2019.
[10] Y. Niu, Paleja, and M. Gombolay. Multi-agent graph-attention communication and teaming. In Proceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, pages 963-973, 2021.
[11] E. Seraj. Wang. R. Paleja, D. Martin, M. Sklar, A. Patel, and M. Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21 international conference autonomous agents and multiagent systems, pages 1173- 1182, 2022.
[12] S. Strunsky. N.j. man fined $32k for illegal gps device that disrupted newark airport system, 2013.

---

[13] S. G. Konan, E. Seraj, and M. Gombolay. Iterated reasoning with mutual information in coop- erative and byzantine decentralized teaming. In Proceedings of International Conference on Learning Representations, 2022.
[14] Y. Sun, R. Zheng, P. Hassanzadeh, Y. Liang, S. Feizi, S. Ganesh, and F. Huang. Certifiably robust policy learning against adversarial multi-agent communication. In The Eleventh Inter- national Conference on Learning Representations, 2023.
[15] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde- Farde. S. Ozair, A. Courville, and Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139- 144, 2020.
[16] C. Szegedy, W. Zaremba, I. Sutskever, Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. Y. Bengio and LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[17] I. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples. In Proceedings of International Conference on Learning Representations, 2015.
[18] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami. Practical black- box attacks against machine learning. In Proceedings of the 2017 ACM Asia conference computer and communicattons security, pages 506-519, 2017.
[19] A. Madry, Makelov, L.. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning mod- els resistant to adversarial attacks. In Proceedings of International Conference on Learning Representations, 2018.
[20] S. Ornes. How to hack a self-driving car. Physics World, 33(8):37, 2020.
[21] S. Jafarnejad, L. Codeca, W. Bronzi, R. Frank, and T. Engel. A hacking experiment: When connectivity meets vulnerability. In 2015 IEEE globecom workshops (GC Wkshps), pages 1-6. IEEE, 2015.
[22] C. Miller. Lessons learned from hacking a car. IEEE Design Test, 36(6):7-9, 2019. doi: 10.11000AAA.2.01.288565
[23] T. Roccia. Today's connected cars vulnerable to hacking, malware, 2018.
[24] Pisarov and Mester. The future of autonomous vehicles. FME Transactions, 49(1):29-35,
2021.
[25] S. Wilson, P. Glotfelter, L. Wang, S. Mayya, G. Notomista, M. Mote, and M. Egerstedt. The robotarium: Globally impactful opportunities, challenges, and lessons learned in remote- access, distributed control of multirobot systems. IEEE Control Systems Magazine, 40(1): 26-44, 2020. 10 100.1109/MSS.200 2019.2949973.
[26] J. Foerster, I. A. Assael N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 29, 2016.
[27] S. Sukhbaatar, A. Szlam, and R. Fergus. Learning multiagent communication with backpropa- gation. In Proceedings of the 30th International Conference Neural Information Processing Systems, NIPS' 16, page 2252-2261 Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819,
[28] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed eeeeraatvee-oopetttvvye environments. Advances in neural information pro- cessing systems, 30, 2017.
10

---

[29] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial Intelligence, 101(1):99-134, 1998. ISSN 0004-3702.
[30] C. Amato, D. S. Bernstein, and S. Zilberstein. Solving pomdps using quadratically constrained linear programs. In Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS '06, page 341-343, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933034. doi:10. 145/1 160633. 166636
[31] N. Hyldmar, Y. He, and A. Prorok. fleet of miniature cars for experiments in cooperative driving. In 2019 International Conference on Robotics and Automation (ICRA), pages 3238- 3244, 2019. doi: N0..109//CRA2221..87990
[32] A. Chowdhury, G. Karmakar, J. Kamruzzaman, A. Jolfaei, and R. Das. Attacks on self- cars and their countermeasures: A survey. IEEE Access, 1227300--273445 2020.
[33] A. Arnab, O. Miksik, and H. S. Torr. On the robustness of semantic segmentation models to adversarial attacks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[34] Z. Zhang, S. Huang, X. Liu, B. Zhang, and D. Dong. Adversarial attacks on yolact instance segmentation. Computers & Security, 11::62 2022. ISSN 0167-4048.
[35] J. Gu, H. Zhao, V. Tresp, and P. H. Torr. Segpgd: An effective and efficient adversarial attack for evaluating and boosting segmentation robustness. In ECCV, pages 308-325. Springer, 2022.
[36] C. Xie, Wang. Z. Zhang, Y. Zhou, L. Xie, and A. Yuille. Adversarial examples for seman- tic segmentation and object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1387, 2017. 1001000119//CCC22011..0
[37] M. Yin, S. Li, C. Song, M. S. Asif, A. K. Roy-Chowdhury, and S. V. Krishnamurthy. Ade: Adversarial attacks against object detection that evade context consistency checks. In 2022 JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.
[38] T. Chen, J. Liu, Y. Xiang, W. Niu, E. Tong, and Z. Han. Adversarial attack and defense in reinforcement Sarning-from ai security view. Cybersecurity, 2:1-22, 2019.
[39] H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C.-J. Hsieh. Robust deep rein- forcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 13:21024-210331 2020.
[40] R. Jiao, H. Liang. Sato, Shen, Q. A. Chen, and Q. Zhu. End-to-end uncertainty-baat based mitigation of adversarial attacks automated lane centering. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 266-273, 2021. doi: 10. 109110/4338333320222 1.977599..
[41] W. Xue, W. Qiu, B. An, Z. Rabinovich, S. Obraztsova, and C. K. Yeo. Mis-spoke mis-lead: Achieving robustness in multi-agent communicative reinforcement learning. In Proceedings of the 2 st International Conference on Autonomous Agents and Multiagent Systems, pages 1418-1426, 2022.
[42] A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru, and Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poi- soning attacks. 28th USENIX Security Symposium (USENIX Security 19), pages 321-338, Santa Clara, CA, Aug. 2019. USENIX Association. ISBN 978- 178-1-999133-06-99
[43] B. D. Ziebart, A. L. Maas, A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, volume 8, pages 1433-1438, 2008
[44] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference Learning Representations, 2018.
11

---

Appendix A Real-World Demonstrations: Robotarium
We use Robotarium [20], a free remotely accessible swarm robotics research platform, to do real- world demonstrations. is equipped with a group miniature differential drive robots 'GRITSBots' on testhed measuring 130x90x1800 em, with a projector and an automatic overhead tracking sys- tem. The GRITSBot's main board has WiFienabled 160 MHz ESP8266 chip as the controller and communication (54 MBit/s WiFi) and the stepper motors droven by Atmega 168 microcontroller [20]. The global position is tracked using overhead camera and then used down-stream for safety checking and feedback control. Features of the robot environment are displayed by the projector for visualization.
First, we need to run our algorithm in the robotarium simulator before implementing it on the real platform. However, physical collisions strictly prohibited when using actual robots. To overcome this limitation, record the trajectories of each agent in the real environment and perform post- analysis to determine if there are any instances where two robots collide. This analysis is based on the relative distance between the robots, following our predefined criteria. A collision between two robots defined as when the circles centered on each robot intersect. The radius of each circle is defined according to the environment specifications [23]. The reward for environments is defined as the 1.2 distance between the robot and target destination. In PP and PCP, the target is the prey robot. In SL, the target the designated goal location. We show the trajectories we collected in each environment Fig 5.
We also include the average reward and collision numbers of each robot Tables 3-8. shows that our adversarial method universally outperforms the random flipping one for each agent since makes the attacked agents receive less reward and has fewer collisions with their targets. Moreover, find that our method is even more stable than the random flipping one, with standard deviation only decreasing by 22.97%, 53.33%, and 40.89% average in the three environments.
Table 3: PCP Reward
Table 5: SL Reward
Capture Agent I Capture Agent -0.76+H..22
Average
Listener
Adv[Ours] Random
-0.6
00.1
Adv[Ours]
-0.43:1..14
00,2710.29
0.090-10.36
-0.1810.33
Random
00.11:0.15
Table 4: PCP Collisions
Table 6: SL Collisions
Capture Agent
Capture Agent 2 Average
Listener
Adv Random
0.02 0.17+0.37
0.09 0.40
0.05
Adv[Ours Random
0.00
0.29+10.45
0.34+00.7
Table 7: PO-PP Reward
PO Agent
PO Agent
PO Agent 3
Agent
Average
Adv[Ours] Random
-0.81 1..34
-0.80-10.27 -0.06
-0.80-00.00 -0.03
-0.71 0.02+
0.01
0.60
Table 8: PO-PP Collisions
Agent 1
PO Agent
PO Agent
PO Agent
Average
Adv[Ours]
0.00:10.06
0.00110.00
0,00=0.00
0.00 -0.05
0.05
Random
0.26.10.44
0.06.10.33
0.33=0.47
0.15
0.2010.40
12

---

(a) Predator Capture Prey
4m40
(b) Partially Observable PP
1-
(c) Speaker Listener
Figure 5: Comparison of Environment Trajectories: All three environments are shown, where the left images the adversarial communication policy rollouts and the right images the random flipping rollouts.
13

---

(a) Predator-Capturee Prey Adversarial Communication
(b) Predator Predator-Capurre-- Prey Random Flipping
(c) Partial Observability Predator-Pres Adversarial Communication
(d) Partial Observability Predator Prey Random Flipping
(c) Speaker-Listener Adversarial Communication
(1) Speaker- Listener Random Flipping
Figure 6: These image series show the performance of agents when applying our adversarial com- munication and random flipping strategy in three environment: Predator-Capttrr-Pre Peey (a, b), Partial Observability Predator-Prey (c, d) and Speaker-Listener (e,
14

---

Appendix B
Simulation Experiment Details
Appendix B.1 Domains
Here we show the hyperparameters used in each environment training (Table 9) and qualitative results (Figure 6).
In the PCP environment (a, b), the predators (also called perception agents) are shown red which can observe all other agents, however, the yellow capture agent (also called action agents) are blind and can only know where the prey (green) is by receiving the messages from the predators. There- fore, communication is the only useful information based on which the capturers can make decisions. Each capture agent will receive 16-bit communication from each of the three predators so we in- tercept 48 bits and modify them with our adversarial policy. Compare with Figure 6(a) and 6(b), we find that our adversarial policy can successfully push the captures agents away from the prey but the random flipping one cannot stop the capture agents from pursuing the prey with the same number of bits flipped.
We observe similar behaviors PO-PP when we compare Figure 6(c) and 6(d), in which the preda- tors and prey are shown with red and green. The difference between PO-PP and PCP environments that we remove the capture agents but change the predators to be partially observable agents which only the prey within certain distance. Predators change color from red to grey if they ob- serve the prey. If one predator observes the prey, it can broadcast this information to others with its 16-bit communication so that the team can cooperate with each other to achieve higher rewards. When we apply the adversarial policy (see Figure 6), we find that the predators just ignore the prey even though they see it and never collaborate to collide with the prey compared with the random flipping one in Figure
In the speaker-listener environment (Figure 6(e, f)), the speaker knows the colored goal the listener should go to but the listener does not. However, the listener knows the position of the three colored goals. Therefore, the speaker needs to learn to communicate the correct color within its 16-bit communication and the listener should learn which color it needs to go to from the message. Our adversarial method (6) can make the listener go to a completely wrong colored destination, while the random flipping method cannot because cannot attack the crucial bits of the communication.
Appendix B.2 Training Details
We show the hyperparameters for all environments in Table 9.
Table 9: Hyperparameters for training and testing PCP, PO-PP and SL
Hyperparameter
Environment
Value
Buffer Length
PCP, PO-PP, SL
1048576
Episode Number
PCP. PO-PP. SL
50001
Episode Length
PCP. PO-PP. SL
100
Batch Size
PCP, PO-PP, SI.
1024
Discount Factor >
PCP. PO-PP, SI.
0.9
Learning Rate
PCP. PO-PP, SL.
0.0001
Regularizer Coefficient do
PCP, SL
0.1
Regularizer Coefficient do
PO-PP
0.004
Regularizer Intercept B
PCP, PO-PP, SL
3000
Regularizer Slope
PCP. PO-PP,
20000
Perception Threshold n
PCP, PO-PP, SL
3
In Figure 7, we show the reward and collision numbers over the training procedure. We start the training without the bit- st-flipping regularizer term Cytsp Then, at episode / (3000), we begin regu- larize the adversarial policy to flip fewer and fewer bits. As training procedes, the regularization term
15

---

dominates the training process and the agents rewards increase as fewer bits are flipped. However, we see that our adversarial agent policy outperforms the random policy at every training iteration.
Appendix B.3 Normalized Score
The normalized score is defined as:
S RCnogdo RCadv
max(Ny, 7)
where RCede and Rona_dde represent the reward or the collision number with and without apply- ing the adversarial policy and their difference represents how much the adversarial communication channel degrades the agent performance. is then normalized by the the number of bits flipped with perception threshold which increases the numeric stability in case of extremely small flipping number y. A higher score signifies that, for each bit flipped, the adversarial approach has a greater detrimental impact on the team's 's performance.
Plopec
Predatoe-Capture-ee (PCP)
Partial Observable Predator-Prey (PO-PP)
Speaker-LLitenne
-02
Plypped
cage Noward
0.2
Filppod
Rewwrd
Bits Pippped
-0.5
20
-044
-0.4
10.8
-0.8
-0.5
10020
20000
31000
02002
5000
10000
15000
20020 23000 30000
10000
20000 Teaco 400ED 50020 00000 Episede
Episede
Episode
(a) Average Rewards in PCP, PO-PP and SL respectively
Predater-Captuue--eee (PCP)
Predetor-Pvey (PO-P9)
Spea (SL)
Nomber Collistens
0.14
Number Colliteene
0.25
0.12
0.20
Bits Fitopedd
0.10
Bit Prtopeed
0.20
Diba Filroeed
0.08
0.08
0.15
0.06
0.3
0.04
0.0
0.02
0.02
0:00
Epleode
Epluade
Eplaade
(b) Average Collisions for PCP, PO-PP and SL respectively
Figure 7: The average reward and number of collisions during training are displayed for the ad- versarial policy (blue) and random flipping (red), respectively (using left y-axis). The dotted-brown line represents the episode where the bit-flipping regularization term begins. The regularization term Cipp pushes the adversarial policy to flip fewer bits as training progresses such that adversarial ef- feet becomes weaker. The resulting number of bits flipped is represented the black curve (using right y-axis). Our adversarial communication policy consistently outperforms the random policy.
16

---

Appendix C Normalized Score Tables for Attacked Agents
We show detailed tables that quantify the reward and number of collisions for each individual agent here for our adversarial communication with flipping mode, direct mode and the random flipping. Our proposed method uniformly better than all other strategies across all attacked agents.
Table 10: PCP Reward Scores
Table 12: SL Reward Scores
Capture Agent
Capture Agent 2
Average
Listener
Adv[Ours] Ady[Direct]
0.10=
0.12::005
0.11=0.05
Adv[Ours] 0.1310.06
0.01
0.01+00.01 0.01
0.01=0.01
Adv[Direct] 0.11 0.04
Random
0.01
0.01=00.11
Random
Table 11: PCP Collision Scores
Table 13: SL Collision Scores
Capture Agent
Capture Agent 2
Average
Listener
Adv[Ours] Ady[Direct]
4.53 2.23
4.63+42.23 1.07:00114 0.99:10.29
4.58+-2.23
Ady[Ours] Adv[Direct]
61.38
1.07=0.14
1.07:10.14
30.72.16.19
Random
0.93
0.96110229
Random
13.6813.119
Table 14: PO-PP Reward Scores
PO Agent
Agent 2
Agent 3
PO Agent
Average
Adv[Ours] Adv[Direct]
0.02
0.04#0.02
0.04:0.02
0.04
0.04:0.02
0.02++0.01
0.02=+0.01
0.02=0.01
0.02=#0.00
0.02+:0.01
Random
0.00.10.01
0.00.10.01
0.00
0.01
0.00110.01
Table 15: PO-PP Collision Scores
PO Agent I
PO Agent 2
PO Agent 3
PO Agent 4
Average
Adv[Ours] Adv[Direct]
1.41-:0.55
1.44:10.51
0.52
1.42+:0.65
1.45:+0.55
0.76:10.07
0.78110.07
0.8210.08
0.8410.08 0.43:10.88
0.80110.07
Random
0.10:1087
0.3410.86
0.3311092
0.30::0.88
Appendix D Whitebox Analysis
In this section, we compare our methods with a whitebox version of our algorithm (Table 16), where do not utilize surrogate policies and instead use the true agent policies and reward to learn the adversarial policy. The results show that our adversarial policy achieves comparable collision and reward scores across all domains. The performance between the whitebox method and our method is similar in PCP and PO-PP but has a larger difference in SL. This shows that our surrogate policy can successfully approximate the ground truth agent policies and aid training an adversarial policy.
Table 16: Reward and Collision Normalized Scores
PCP
PO-PP
SL Collision Sc
Reward Se
Collision Sc
Reward Se
Collision Se
Reward Sc
Whitebox
0.1250.05
4.60-12.30
0.04+0002
1.37-+0.43
0.17::0.09
41.56-4.98
Adv[Ours]
0.11-
4.58++2.23
0.04+-0.02
1.45++0.55
0.13++0.06
31.38-66 6.51
17

---

Appendix E Surrogate Policy Losses
In this section, we show the surrogate policy loss curves. We see that the surrogate policy loss converges relatively quickly before episode 5000. This indicates that while we train for up 60,000 episodes, much less data could be used to train stable surrogate policy that can be used for the adversarial communication policy.
Predator-Capptur-Pre
Partial Observable Predator-Pray (PO-99)
Speakker
Surrogete Pollcy Lo45
0.3
0.3
0.2-
Sumogaat
0.4
0.2
0.0
10000
20000 Episode
30000
5000
10000
15000
20000
30000
20000
20000
40000
Episode
Episode
Figure 8: Surrogate Policy Loss in PCP, PO-PP and SL respectively
Appendix F
Robustness of Analysis Adversarial Policy
We evaluate whether the adversarial policy maintains performance under minor modifications to the target agent policies. our experiment, we extend the training of the target team agents by an ad- ditional 10,000 episodes, subtly changing their policies. Every 1,000 episodes during this extended training. gauge the effectiveness of the adversarial attacking policy, where the adversarial policy is frozen and acts without any further training. Our findings indicate that the adversarial policy's 's performance remains largely consistent, with only a slight decrease in effectiveness and increase in variance (see Figure 9).
Flipped
Flipped
Ceedator-Capture Prey (PCP)
Predator- (PCP)
-02
0.14
Reward Standard Deviation
0.13
Average Reward
-0.3
Bits Fllpped
Bits Flipped
-0.4
-0.3
0.09
-0.6
0.08
2000
6000
10000
8R00
10000
Episode
Episode
Figure 9: Robustness of Adversarial Policy: Reward Mean (left) and Standard Deviation (right)
18


Reviews:
[{"review_id": "efSQGCc34Al", "reviewer": "Reviewer_3YTc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on the problem of tricking a neural network that has been trained for the purposes of multi-agent interaction. The idea is for an adversary to learn how to manipulate the data communicated between robots so as to manipulate the outputs of the model that has been learned by the multi-robot team. The paper looks at such hijacking/hacking in the context of three different multi-agent tasks, and compares the proposed method to randomly manipulating bits in the messages that are communicated between robots.  ", "summary_of_recommendation": "The reviewer has recommended strong accept given that this is the first work that this reviewer is aware of in which a black box hacker is trained to perform a man-in-the-middle attack on a multi-robot learning policy. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "\nThe figure captions and title are way to small... if accepted, can you please make them larger in the final version?\n\nIn the figures the use of the word \"episode\" is somewhat confusing. Is this a training episode?\n\nAn additional limitation may be that gathering this much training data (20,000 + training iterations) may be difficult in real life, since there is an implicit assumption that either (a) the adversary has access to a simulation of the team that it can use for learning, or (b) the adversary has access to 20,000 actual interactions in which the team does not detect it is being hacked. Note: this does not detract from the novelty of the work, but it should be mentioned. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality: While the idea of hacking/hijacking in a swarm is not new, the specific way that this hijacking is accomplished is (in the opinion of this reviewer) new. The most important difference versus previous work appears to be the limited amount of interaction, e.g., for training purposes, that the black-box adversary model needs with the robot team it is attempting to manipulate. \n\nQuality: The submission is technically sound and the claims well supported by the experimental results. The fact that the method is tested on three different types of multi-robot missions adds to the significance of the work. The paper makes a decent case that the most closely related work is different enough (requiring different training conditions that are non-trivial in the context of an adversarial setting) that comparison should not be required. \n\nClarity: Is the submission is clearly written, well organized, and easy to understand. \n\nSignificance: The results are important in that this shows how the hijacking problem can be addressed in a setting that is arguably new and less strict than that which previous work has addressed. \n\n\nRelevance: The paper addresses a problem in robot learning, experiments were performed in the robotarium (that is, using real robots).\n\nLimitations: The limitations section exists and is adequate for the purposes of the CoRL conference. \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "YQWtEobHpJ", "reviewer": "Reviewer_ooVv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors of the paper present their approach to a black-box attack multi-agent reinforcement learning system. They draw attention to the importance of secure communication between agents during solving tasks and create their malicious model to alternate these communication messages. In opposite to other articles in this field, authors do not interact with the environment while training their malicious policy; they neither have access to ground truth agents’ policies or rewards. The authors consider binary communication channels and compare their approach with the random bit-flipping policy. The proposed method shows better metrics than baseline on three different Dec-POMDPP environments in simulation and Robotarium physical robots.", "summary_of_recommendation": "It is an important paper that could improve the safeness of MARL real-world systems, providing an effective black-box attack on such systems without interaction with the environment or access to the reward. However, the experimental section is unclear and, in my opinion, requires more explanations and information. \n\nAfter author response: I appreciate the author's efforts to improve the clarity of the paper. Also, all of my concerns were well-addressed. Considering this I am upgrading my score to strong accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Explain or better represent the results of experiments, as was stated in the review\n\n* Add information about hyperparameters, and the amount of data to train adversarial policy to encourage reproducibility \n\n* On lines 211 and 214 presented formulas without specific values of hyperparameters, variables beta, and epsilon are not explained at all\n\n* Include the amount of acquired training data to the limitations of your approach", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality) Even though the idea of an attack on communication in the MARL system is not original [36, Mis-spoke or mislead: Achieving Robustness in Multi-Agent Communicative Reinforcement Learning], the authors increase the applicability of such an attack by introducing a successful black-box algorithm.\n\nQuality ) The theoretical background of the paper is convincing, however, the experimental part raises many questions for me. At 219 the authors state that collision numbers increase when we decrease the number of bits flips, which sounds counterintuitive. It is not clear to me the meaning of the episode axis in Figure. 3: If it illustrates the training process of malicious policy then why does it perform better than random flip at point zero? In Table 1 the best algorithm has the most number of collisions, however, in Table 2, it is the opposite.\n\nClarity ) The authors do not provide any details on the learning process i.e. hyper parameters, number of bits in communication messages, enough number of observations to successfully perform such attack. It will be hard to reproduce the results.\n\nSignificance) I think the results of this paper could be useful because they propose a real-world applicable attack on MARL systems and researchers should be aware of such algorithms, thinking about effective defenses.\n\nRelevance) I think it is a relevant paper because the authors present an algorithm that could compromise real-world systems. In addition, they demonstrate the effectiveness of their approach on physical swarm robots.\n\nLimitations) The authors do not say about the time requirement for their agent to perform such an attack in terms of observed interactions or real-time, it is important because a man-in-the-middle attack will be detected if it requires a lot of time. Otherwise, limitations are addressed well.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Rdyp0mtHI_p", "reviewer": "Reviewer_fyCD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a communication attack method to compromise multi-robot coordination.  The method requires knowledge of the robots' observations as well as of their message policies given the observations. The proposed method is tested on the Robotarium platform in target tracking tasks.", "summary_of_recommendation": "The paper proposes an interesting method for compromising multi-robot direction.  In my opinion, the current version of the paper would benefit from clarifying the practicality of the assumptions and the degree of detectability of the designed attacks.\n\nUPDATE: I update my score to strong accept since the authors addressed all my during with their rebuttal.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "Please see all my comments in the weakness section above.  I would also elaborate on the robustness of the learned attack policy.  How effective a learned attack policy remain when the robots alter their message policy?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n-  The proposed method improves upon previous works by not requiring interaction with the environment, thus, increasing its potential to not be detectable.\n-  The proposed method is tested on hardware, using the Robotarium multi-robot test bed.\n\nWeaknesses\n\n-  I would elaborate on how the attacker can have access to the observations of the agents.  If these observations are, for instance, camera images, how the attacker can get access to these images or to a processed form of those? \n\n-  I would discuss why the proposed attack scheme is not detectable.  For example, why it cannot be the case that the altered messages don't make sense? In such a case ---altered message doesn't make sense--- the attack is immediately detected.   This can happen when, for instance, the altered messages prescribe locations that are outside the boundary of operation.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "ENq-XwiZ4H", "reviewer": "Reviewer_N9UF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel security context for adversarial attacks on multi-agent systems. The proposal is that, through some mechanism, an adversary is able to perform a man-in-the-middle attack on a multi-agent robot team. This allows the attacker to modify communication signals and influence behavior. They consider a setting where the attacker acts in a one-shot fashion to execute the attack but observes the victim's behavior beforehand.", "summary_of_recommendation": "I'm fairly positive about this paper and I think it studies a problem that is quite interesting. I think the approach is sound and has some promise. I have a few reservations about the evaluation baselines and thoroughness which hold me back from a strong accept. If the authors can address those in the rebuttal I am likely to increase my score. \n\nUpdate after rebuttal: I am satisfied with the response and have updated my score accordingly. I recommend that the authors update the paper to make sure that the justification of the baselines is clear to readers. Additionally, I think that a discussion of driving the system to arbitrary goals may help open up interesting avenues for future work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "## Evaluation Baselines\n\n* Can you justify your choice of baselines? Why did you pick the bit-flipping method? Why did you control the bit flips in the way you did? Are there alternative random baselines you considered?\n* Can you compare against the setting without surrogate policies? How much harder is the setting where you have to learn surrogate policies for victims?\n\n## Discussion questions\n\n* It seems like this attack works to prevent a system from accomplishing its goal. If, instead, you had an, alternative, desired goal for the system, could you modify the attack to steer the system towards arbitrary goals?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths\n\n## Originality/Significance\n\nThe paper considers a novel setting for adversarial attacks that is relevant and interesting. The attack setting is plausible for several applications and there is a good potential for follow-on work. The setting, where attackers can not experiment directly on victim policies, is challenging and potentially valuable. \n\n## Quality\n\nThe approaches in the paper are sound and the evaluation results are good.\n\n# Weaknesses\n\n## Clarity\n\nThere are a few places where the clarity can be improved:\n\n* Can you make the inputs to Algorithm 1 more clear? I realize they are defined elsewhere, but it would help to be redundant here, IMO\n* Figure 3: can you add more to the caption to help readers interpret this figure? I had a hard time figuring out what is going on (still not sure I got it tbh...)\n* Can you clarify in 5.4 if this is in simulation or on a physical system? This was hard for me to figure out.\n\n## Quality\n\nI also have some questions about the evaluation methodology, please see that portion of the review.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the case of a multi-agent scenario where the agents communicate with each other. An adversarial policy is used for tricking the agents to perform actions which are not originally desired. The method is evaluated in simulated benchmark tasks and using physical swarm robots in the Robotarium. The paper studies an interesting problem setting and the proposed approach gets high performance in the experiments.
Decision: Accept (Oral)