Robust Multi-Agent Reinforcement Learning against Adversaries on Observation
Authors: Chenghe Wang
Yuhang Ran
Lei Yuan
Yang Yu
Zongzhang Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-agent reinforcement learning
robust reinforcement learning
cooperative multi-agent systems
adversarial training

Abstract: With the broad applications of deep learning, such as image classification, it is becoming increasingly essential to tackle the vulnerability of neural networks when facing adversarial attacks, which have been widely studied recently. In the cooperative multi-agent reinforcement learning field, which has also shown potential in real-life domains, little work focuses on the problem of adversarial attacks. However, adversarial attacks on observations that can undermine the coordination among agents are likely to occur in actual deployment. This paper proposes a training framework that progressively generates adversarial attacks on agents' observations to help agents learn a robust cooperative policy. One attacker makes decisions on a hybrid action space that it first chooses an agent to attack and then outputs the perturbation vector. The victim policy is then trained against the attackers. Experimental results show that our generated adversarial attacks are diverse enough to improve the agents' robustness against possible disturbances. 

Under review as a conference paper at ICLR 2023
ROBUST MULTI-AGENT REINFORCEMENT LEARNING AGAINST ADVERSARIES ON OBSERVATION
Anonymous authors Paper under double-blind review
ABSTRACT
With the broad applications of deep learning, such as image classification, it is be- coming increasingly essential to tackle the vulnerability of neural networks when facing adversarial attacks, which have been widely studied recently. In the coop- erative multi-agent reinforcement learning field, which has also shown potential in real-life domains, little work focuses on the problem of adversarial attacks. However, adversarial attacks on observations that can undermine the coordination among agents are likely to occur in actual deployment. This paper proposes a training framework that progressively generates adversarial attacks on agents' ob- servations to help agents learn robust cooperative policy. One attacker makes decisions on a hybrid action space that it first chooses an agent to attack and then outputs the perturbation vector. The victim policy is then trained against the attackers. Experimental results show that our generated adversarial attacks are diverse enough improve the agents' robustness against possible disturbances.
INTRODUCTION
The target of reinforcement learning (RL) is to learn the policies in complex environments to get long-term rewards. The technique of MARL introduced adapt RL algorithms multi-agent systems. In recent years, Multi-Agent Reinforcement Leaming (MARL) has attracted widespread attention (Du Dingl 2021) and has been applied in numerous domains, including sensor net- works (Zhang & Lesser 2011). autonomous vehicle teams (Zhou et al. (020), and traffic signal control (Du al. (021). However, neural networks are proven to be vulnerable to adversarial per- turbations (Huang et al. (017), and some small perturbations may cause the deep RL. policy to fail.
Therefore, it is of great significance to train a robust policy to help deploy current RL algorithms to real-life applications.
In single-agent reinforcement learning, some research studies enhance policy robustness by using adversarial learning and achieve good results. Pinto al. (201)) propose method that jointly trains a pair of agents, including a protagonist and an adversary, and the protagonist learns to fulfill the original task goals while being robust to the disruptions generated by its adversary. Pattanaik jet al. (2018) show that deep RI. can be fooled easily and train an RI. agent under naive attacks to improve its robustness. Zhang et al./ (2021) propose framework of alternating training with learned adversaries, which trains an adversary online with the agent using a policy gradient following the optimal adversarial attack framework. However, such studies are rare in cooperative MARL, and current works mainly focus on the setting where teammates may betray or agents' actions may be maliciously modified (Li et al., 2019 Phan al.) 202T 2020 Hu & Zhang 2022). However, in real-life applications of cooperative MARL,, the most vulnerable parts of the agents are the sensors that can be disturbed by noise or jamming attacks. Agents are closely related to each other when cooperating to accomplish tasks, and even small perturbation on agent's observation from the sensors can make it deflect from coordination and cause the whole multi-agent system to fail. Therefore, how to design algorithm that can obtain policy that is robust observations in cooperative MARL is noteworthy.
This paper proposes robust MARL training framework for observation perturbations, RObust Multi-agent reinforcement learning against Adversaries on Observation (ROMAO). Our contribu- tions can be summarized follows:

---

Under review as a conference paper at ICLR 2023
We propose a hybrid action attacker that attacks based on the global state. Its output sists of a discrete action and a continuous action. The discrete action refers to which agent to disturb, and the continuous action refers to the disturbance added to the observation of that agent.
We propose a robust training framework that generates attackers gradually and helps the victim team learn robust cooperative policy that is resistant to perturbations observa- tions.
We additionally propose an optional defense module that can further enhance agents' ability to defend against attacks on observations.
Experimental results show that the proposed attacker can pinpoint the weakness of the cooperative policies, and our robust training framework along with the defense module can effectively improve the robustness of the policies against possible attacks.
2
RELATED WORK
2.1 COOPERATIVE MULTI-AGENT REINFORCHMENT LEARNING
Cooperative MARL has made prominent progress these years. Research on it aims to help agents learn policies to coordinate and complete cooperative tasks. Many methods have emerged under the CTDE paradigm, most of which can be roughly divided into policy-based and value-based methods. MADDPG (Lowe al. 2017), COMA (Foerster et al. 2018), and MAAC (Igbal & Sha 2019) are typical policy gradient- based methods that explore the optimization of multi-agent policy gradient methods, while MADDPG can also be employed in competitive scenarios. Another category of co- operative MARL. approaches, value-based methods, mainly focus on factorizing the value function. VDN (Sunehag et al.) 2018) aims decompose the team value function into agent-wise value func- tions by a simple additive factorization. Following the Individual-Global
(IGM) principle (Son) al. 2019). QMIX (Rashid al., 2018) improves the way of value function decomposition by learning mixing network, which approximates monotonic function value decomposition.
2.2 ADVERSARIAL ATTACK
The adversarial attack has been explored in many areas. In image classification, the adversarial attack means generating adversarial examples. The adversarial example is deceptive input model that is purposely designed to cause a model to make a mistake in its predictions but makes no difference humans. [Goodfellw al. (2015) propose simple and fast gradient-based method that is used to generate adversarial examples to make the model classify incorrectly while minimiz- ing the amount of perturbation added to the pixels of the image. Loison et al. (2020) use feature selection to minimize the number of features modified while causing the wrong classification, and flat perturbations are added features iteratively according to saliency value by decreasing order.
2.3 ADVERSARIAL ROBUSTNESS OF RI. AGENTS
Based on the effectiveness of adversarial attacks images, Huang et al. (2017) propose method to inject adversarial perturbation into the input to confuse the RL policy. Some researchers (Gleave al. 2019 Zhao et al. 2020) focus on black box attacks in RL, which more challenging be- cause of the lack of information about the parameters of the target model. Adversarial training is empirically shown improve agents' robustness make the policies experience possible adversar- ial attacks. Piinto et al. (2017) propose method to train an agent in the presence of disturbance and obtain more robust policies. Zhang et al. (202T) propose method that involves the concurrent training of an attacker and the victim agent using policy gradient following the optimal adversarial attack framework. [Sun al. (20T)) decouple the problem of finding state perturbations into finding the best policy perturbation directions and crafting correspondent state perturbations.
2.4 ADVERSARIAL ATTACKS IN COOPERATIVE MARL
There could be various types of adversarial attacks cooperative multi multi-agent systems. Some re- searchers focus the setting where some teammates may betray and minimize their shared
2

---

Under review as a conference paper at ICLR 2023
turn (Phan et al. 202 2020 Li al., 2019). Meanwhile, some researchers prefer the setting where components of a Markov Decision Process (MDP), such as states, actions, or observations, are per- turbed. (Hu & Zhang (2022) propose a sparse adversarial attack actions of cooperative multi-agent systems and can make the victim team perform poorly when only a few agents are attacked at a few timesteps. /Zhou & Liu (2022) propose robust training framework for the state-of-the-an rein- forcement learning method MFAC (Yang et al. 2018) when the state is perturbed. [in et al. (2020) propose a method attack one agent's observation in team. It is achieved by an indirect way that the attacker first tries to find wrong action it should encourage the victim agent to take. Then, the attacker uses adversarial examples to mislead the victim into choosing the action. This work is most relevant to our work because it considers the indirect attacks on observations, and the attacker only chooses one agent to attack. While in our work, we consider the setting that every agent is at risk of attack, and we cannot access the parameters of the Q functions of the victims. We focus on black box attacks on observations and how to defend against them, which is more reasonable and realistic.
PROBLEM FORMULATION
This paper considers a fully cooperative multi-agent task where agents only have access to partial observations for the victim side. When the attacker's policy is fixed, the model is defined as M = (N,8, {A') );=1 P. {0') 1t R, 7), where N = {l...., n is the set of agents, s E S is the true global state from which agent can get local observation O² E O². In order to relieve the partial observability problem, we add an RNN module, GRU (Cho et al. 2014), called agent trajectory encoder, to encode the history (of, .i.,,.,,,, ,/ a;-,,, of) into T", with oj E A', ; E O' stand for the action and observation of agent i time I. At each timestep, each agent selects an action aj E A', forming a joint action a E A = II==, A', leading to the next state 81-1 ~ P(-|s), a) and getting a shared reward R(s;, a). The formal objective for the multi-agent policy (the victim side) is to find a joint policy (T, a) to maximize the global value function:
Qune (T. a) = E,,a
ERsR(se at) So = s, ag = a, π
where y indicates the discount factor. Perturbations are integrated into {0')= and the agents cannot access the oracle information about whether they are under attack. Meanwhile, when the victim team's policy is fixed, the Markov Decision Process for the attacker is defined as A = (S, A, P, R, The attacker shares the same state space with the victim team and A: N (O+) Ti=1. which means that the attacker first chooses one agent and then generates a perturbation with the same dimension as the agent's 's observation. The transition function defined as P:
Å {4th - S. Since π is fixed when training the attackers, a E A is only determined by the state E S. The reward function is î - R and the objective of the attacker's policy R is reduce the return of the victim team which is equivalent maximizing the following objective (ā, E Å):
J(+) = E
Σ r R' az) 8841 ~ P (-18k, åz, at) ae ~ f (86,))
(2)
Attacks on multi-agent systems may be diverse. In some scenarios, random noise can be seen as a potential implicit attacker. While in others, the attacker can add perturbations to a specific observation dimension of a specific agent, which is small enough and will not make difference to people but can seriously degrade the performance of multi-agent systems. We mainly consider the following kinds of possible attacks multi-agent in our work:
Attacking random agent with random noise: agent system may contain random noise, which may be physically caused (e.g., sensor errors). We assume that the noise in each observation dimension each agent is indepen- dently and identically distributed.
The observation of each agent in a multi-
Attacking a specific agent with random noise: some scenarios, only one specific agent will contain random noise. We also assume the noise is independently and identically distributed in this attack setting.
Attacking specific observation dimension of specific agent: In some scenarios, specific observation dimension of a specific agent may be under attack (c.g.. in soccer, a player injures one eye resulting in inaccurate observations of himself).
3

---

Under review as a conference paper at ICLR 2023
Section 4.1
Attackers Pool
State
Latent
[πî n22 m3
Action Space
Observations
Environment
Defense Module
Defender
Perturbation
Actions
MARL Algorithms
Detector
Observation
Section 4.3
Reconstructor
Section 4.4
Figure I: Structure of our robust training framework RAMAO. The attackers' pool stores history ., m, where m is hyperparameter we set about the maximum number
attacker models j. 1, 2,2, ..., of models the pool can store.
To be more consistent with the real-life problem setting, assume that the é norm of all attacks less than or equal to constant In general, performance of a policy falls off cliff when the policy is exposed to the above attacks. We want to obtain a robust policy that can prevent the performance from dropping too much under attacks with modes including but not limited to the above three.
4 METHOD
The overall structure of our proposed framework is shown in Figure L and we train the attacker and the defender (the victim team) alternatively. When training the attackers, we fix the defender and maintain the attackers' pool to store all the history attackers have trained so far. Every time the attacker starts turn for training, it stores its last attack model îty in the pool and resets its exploration and replay buffer eliminate the bias introduced by previous attacks. During interactions, the HyAR attacker receives the global state, chooses an action in a latent space, and decodes the latent action to perturb the observation of specific agent (Details are listed in Section 4.1). When training the defender, as stated in Section 4.3 we randomly sample an attacker from the attackers' pool and train the defender against it with only access the perturbed observations. The defense module
optional because it needs communication and will be introduced in detail in Section 4.4
4.1 HYBRID ACTION SPACE
In our method, the attacker first selects an agent to attack, then gives the offset for each dimension of the observation, which can be considered a hybrid action space problem because selecting an agent is a discrete action and giving the offset for each dimension of the observation is continuous action. We will consider Parameterized Action MDP (PAMDP) (S, H. P, 7, 7) (Masson & Konidaris 2015), which is an extension MDP to consider the discrete-comtinuous discrete-comtimnnn hybrid action space H [kk Zxx) 2 E Xr, Vk e K),, where I.C. (1. K} is the set of the discrete action and X' the continuous parameter set for each E ( al. 2021).
Typical policy representations like Gaussian distribution and Multinomial distribution also cannot model the heterogeneous components of the hybrid action space. A naive approach to solving the hybrid action problem is convert the hybrid action space into discrete action space by discretizing the continuous action space. However, this ignores the implicit connection between continuous and discrete actions. Meanwhile, it reduces scalability and brings approximation difficulties.
4

---

Under review as a conference paper at ICLR 2023
Li al. (2021) propose novel method named HyAR for solving hybrid action space. HyAR con- structs a unified and decodable latent representation space (Rdi+0a, dj is the latent space dimension for discrete action & and d2 is for continuous action zg) for original hybrid actions, and the agent learns a latent policy. Then, with the decoder, we can map the selected latent action back to the original hybrid action space to interact with the environment. In more detail, HyAR establishes an embedding table Eg E Rx xxxxi to represent the K discrete actions. Each row ec.& = (k( is d1- dimensional embedding vector for the discrete action k. Then, with state s, HYAR uses an encoder (= Ek, $, (ç.b) parameterized by ø to map 2& to the latent vector 2x E Rdd RL algorithms for continuous actions like DDPG (Lillicrap et al. 2016) or TD3 (Fujimoto et al., 2018) can then be applied to train on this latent action space. For inference, each time HyAR policy receives a state
will output latent actions e, R.. then use the decoder and embedding table to reconstruct the latent actions back to the origin hybrid actions T.
HR takes full advantage of environmental dynamics and considers the connection between dis- crete and continuous actions. In our work, we employ HyAR to help the attacker learn the hybrid action to inflict perturbations on agents' observations. Practically, we employ TD3 as the policy to decide the latent space. Based state 8 at time t, our attacker policy first outputs discrete action & to indicate the agent's 's ID that we will attack. Then the attacker policy outputs a continuous action Ft, which has the same dimension as Ok,- The victim agent does not have oracle informa- tion about whether it is under attack and would make decisions based on the perturbed observation Ok: Ike
4.2 ATTACKER OPTIMIZATION METHOD
The goal of the adversarial attacker is minimize the victim team's expected return by perturbing the observations of certain agent in multi-agent system. We do not introduce any other intuitive rewards that are possible to influence the possible attack modes. During the alternative training between the attacker and the victim team, we reset the exploration of the attacker to reduce the bias from the last round of attack learning. While for the victim team, we save the attacker's 's model every round, and the victim team train their cooperation policy against all historical attackers, which would further enhance the robustness of our victim policy.
4.3 ROBUST TRAINING
As formulated above, the attacker can inflict perturbations of amount C' in terms of the & norm on one agent's observation. However, we do not assume what kind of attacker pattern the attacker uses. In practice, an attacker may apply random noise to attack the observation, while others can also concentrate on specific dimensions of the observation based on the current situation. The attack may even have long-term effect that we should not only focus one timestep when the agent's s observation is perturbed. To discover the weakness of the victim team's policy and make agents get used to different kinds of perturbations, we employ the above attackers and alternate training between the attackers and the victim team. The advantage of alternate training is that when the defender has a high win rate, it forces the attacker to find some tricky attacks, and the defender can learn to deal with these tricky attacks, thus greatly increasing the ability to defend against both general attacks certain extreme attacks. During the training of the victim team, we sample an attacker for an episode from a history pool of all attackers trained so far so that the victim team can defend against not only the current version of the attacker but also all the history attackers to improve its general robustness. Besides, our training framework is agnostic to specific MARL methods and can be applied to improve any MARL method's robustness.
4.4 DEFENSE MODULE
Apart from the adversarial training, we propose a multi-agent system defense module to defend against the above perturbations. In multi-agent scenarios, agents interact in the same environment, so agents close to each other may share part of their observations. Inspired by this fact, we came up with a defense module that agent can reconstruct its observations based on the observations of its teammates in its field of view. However, an agent cannot access the information about whether it is perturbed not, so we need to train perturbation detector to judge whether an agent is under
5

---

Under review as a conference paper at ICLR 2023
attack. The mentioned training be achieved during the centralized training phase since both the original and the perturbed observations are available.
EXPERIMENTS
In this section, first display the scenarios involved in experiments in Section 5.1 including StarCraft II unit micromanagement benchmark (SMAC) (Samvelyan et al. 2019) and predator-pree
(PP) (Boehmer et al. 2020). We show the main results and the effectiveness of robust training
framework in Section 5.2 Then we present part of our training curve on QMIX and employ Principal
Component Analysis (PCA) (Pearson [90I] illustrate the diversity of our generated attackers in
Section 5.3 Furthermore, we demonstrate the effects of our defense module in Section 5.4 and the generality our trained robust cooperative policy.
We choose QMIX (Rashid et al. 2018), which is popular baseline in cooperative MARL methods and approximates monotonic function value decomposition for joint value function, as our victim team policy, while our robust training framework ROMAO is agnostic to specific MARI algorithms or attacker policies. ROMAO is general framework that can employed to improve the robust- ness of any given MARL method against perturbations on observations. More details about our experiments would be listed the appendix.
5.1 ENVIRONMENTS
Inpproper Ceasefive
Knprogerr
Weddiing
(a) Improper Wandering
(b) Improper Ceasefire
Figure Agents' improper behavior patterns caused by observation perturbations.
We evaluate our robust training framework in environments including StarCraft II unit microman- agement benchmark (SMAC) and predator-prey (PP). More detailed descriptions of these environ- ments can be found in the appendix.
5.1.1 STARCRAFT II MICROMANAGEMENT BENCHMARK (SMAC)
SMAC (Figure is popularly used combat scenario StarCraft II unit micromanagement tasks, where train the ally units to beat enemy units controlled by the built-in AI with an unknown strategy. At each timestep, agents can move or attack any enemies and get a global reward equal the total damage done enemy units. Figure shows the map of 5m_vs_6n in SMAC which indicates that we control 5 marines to defcat 6 enemy marines. It is hard map in SMAC that the marines we control must work closely win the battle but if one of the teammates is perturbed and does not cooperate normally as shown in Figure 2(a) (bb) the other agents would be short of fire and in great danger.
5.1.2 PREDATOR PREY (PP)
PP (Boehmer et al. 2020 is partially observable grid-world task where predators (agents) are trained to capture n moving preys. Agents get rewards only when there are no empty grids around the prey and at least two predators adjacent to it execute "catch" action concurrently. Miscatching would lead to a punishment of -2, this punishment makes it challenging benchmark for current MARL methods.

---

Under review as a conference paper at ICLR 2023
Maps
2s3z
3m
3s vs 5z
5m vs 6m
Methods
Vanilla QMIX
95.3++3.4
91.1+1.5
81.2++4.4
58.3++3.7
Random QMIX
98.9++0.9
97.2+0.9
97.6++099
1..4
Attack Mode 1
One-agent QMIX
81.442.3
8..0
77.943.0
59.2:66.8
ROMAO
99.0-10.7
93.911.7
98.1+-0.8
68.8
Vanilla QMIX
92.7:1.99
91.7+33.2
7..6::
57.3:13,2
Random QMIX
99.0+10.7
97.9+1.00
98.111..0
67.8:14.0
Attack Mode 2
One-agent QMIX
82.942.77
86.1
82,242.77
60.142..7
ROMAO
96.3+1.0
94.922.1
97.8+0.77
66,0+2.33
Vanilla QMIX
72.6++5.9
5.73+22.0
45.8+4.11
13.5++0.7
Random QMIX
83.1
90.8+2.0
56.0+3.5
3.142.4
Attack Mode 3
One-agent QMIX
69.3+4.22
92
54.6114.8
13.642.3
ROMAO
94.6++2.8
94.4+1.99
75.3+-2.6
23.3+
Table I: Average test win rates of different methods under various attack modes with limited per- turbation range of amount terms of the l norm. We consider three different and general attack modes. Vanilla QMI means the QMIX policy trained without perturbations. Random QMIX means the policy trained under random perturbations similar to Attack Mode 2. One-agent QMIX means the policy trained with only agent 0 is under attack and the attack is trained with TD3. ROMAO means the QMIX policy enhanced with our robust training framework.
5.2 RESULTS AND ANALYSIS
Thole] the main results of our experiments about the average test win rates and correspondent standard deviations of different methods under various attack modes with limited perturbation range of amount in terms of the & norm. We use perturbation range 5 for testing to evaluate the policies' generality trained under perturbations of range 10. Attack mode denotes the attack that at every timestep, the attacker randomly chooses an agent and generates random perturbations. Attack mode 2 denotes the attack that at the start of an episode, the attacker randomly chooses an agent and generates random perturbations on that agent's s observation throughout the entire episode. Attack mode denotes the attack that, the start of an episode, the attacker randomly chooses an agent and generates perturbations on only one random dimension of the agent's ' observation, which means an offset of is added to a random dimension of the observation. Vanilla QMIX denotes the QMIX policies trained without perturbations. Random QMIX denotes the QMIX policies trained under the attack that randomly chooses an agent at each timestep and inflict random perturbations of amount C' in terms of the E norm on the agent's 's observation. One-agent QMIX denotes the QMIX policies trained under the attack that always chooses agent 0 and inflict random perturbations of amount C. In practice, we set C 10. All the means and standard deviations are the results of five random seeds for each method.
As shown in Table T after training with ROMAO, the ROMAO-trained QMIX policy is robust enough to defend against possible attacks and has a good and stable performance compared with QMIX policies that trained in other ways. Generally, the vanilla QMIX performs worst since it has not seen any attack patterns before the evaluation, but it is still robust to some extent. It may be attributed to the fact that in multi-agent system, even if one agent shuts down, other agents can still complete some simple tasks that are not demanding coordination. Training the MARL algorithms under random perturbations can also improve the policies' robustness, but they cannot deal with all possible situations. Random QMIX is trained under random perturbations like Attack Mode 2 so that it performs best on it, but it cannot generalize to other attacks and there a significant drop in performance in Attack Mode 3, which is an extreme type of attack, while HAMAO-trained policy performs well on all these attacks. Attacking only one agent is also insufficient to generate all possi- ble situations under attack because the attacking target can be switched among different agents. Due to the alternate training of ROMAO, can cover more types of attacks and has better performance for extreme attacks like Attack Mode 3. The results demonstrate the necessity to employ hybrid action space to generate diverse and influential attackers and the effectiveness of our proposed robust training framework.
7

---

Under review as a conference paper at ICLR 2023
5.2.1 RESULTS ON PP
We also carry out some experiments on Predator-Prey (PP). RO- MAO denotes the QMIX policy after the robust training in our pro- posed framework where attackers generate perturbations of amount C' in terms of the { norm. RanPert denotes the QMIX policy trained under random perturbations of amount C. QMIX denotes
Results on PP
40
-30
Avarage Mean
the QMIX policy that is trained without any perturbations. The policies are evaluated under perturbations of different range. practice, set C' 10 for training and C' = 5 for this evaluation. As illustrated in Figure 3 ROMAO policies perform best, showing proposed framework effectiveness. During the training phase of our robust training framework, attackers generate diverse attack modes that uncover different weaknesses of the current policy, while random perturbation cannot cover all the possible situations.
-8
-37
36
ROMAO
RanPert
QMIX
Figure 3: Results on Predator- Prey.
5.3
TRAINING PROCEDURE VISUALIZATION
Policy Embedding Projection
1.0
ROMAO
0.2
Test Win Rate
0.8
0.6
0.4
0.0
0.2
3
0.0
0.0
0.6
1.2
1.8
2.4
3.0
Timesteps
167
-0.2
-0.1
-0.2
0.0
0.2
0.4
(a)
(b)
Figure 4: (a) The alternating training curve of the victim policy's performance. (b) The distribution of the representations of the attackers' policies. The darker the color of the dot, the earlier the attacker policy is generated.
Figure ((a) shows the alternating training curve for QMIX on the map of 5m_vs_6m in SMAC under framework ROMAO. We first train QMIX on 5m vs 6m for 2M timesteps and get an average win rate around 80%, and then load the trained model into the ROMAO framework to start alternating training. The curve shows the win rate of the current QMIX policy under the attack of the current attackers when testing. The attacker is trained in the first turn so that the win rate gets reduced first and fluctuates during the alternating training. Finally, the robust policy can maintain win rate of over 60% however the attacker attacks.
We also illustrate the diversity of attacker policies in Figure 4(b) As stated in Section 4 we maintain an attackers' pool to store the previously trained attackers. As the defender will learn to adapt to the attackers' perturbations, the current attack mode may expire, and the attacker must seek new attack mode to perturb the victim team's observations. In practice, we reset the attacker's exploration and replay buffer eliminate the bias introduced by previous attacks. order to achieve high rewards, the attackers should spontaneously find diverse attack modes to perturb the observations of the constantly updated victim agents.
Inspired by the recent success of transformers in deep learning (Vaswani et al. 2017), we utilize a transformer to encode a perturbed trajectory to represent an attack policy. We first sample n trajectories of states {s (I' is the episode length) when the trained QMIX interacts with the environment under random perturbations. With the instinctive ability transformer to deal with variable-length input sequences,
concurrently train transformer encoder that inputs trajectory (st (To) and outputs embedding h and Multilayer Perceptron (MLP) decoder to reconstruct sr from h. After training, we fix the transformer encoder and get an attacker's policy embedding by first sampling 16 trajectories of vanilla QMIX under the attack of this attacker and then calculating the
8

---

Under review as a conference paper at ICLR 2023
mean of the outcomes after feeding the trajectories into the encoder. Finally, PCA project the policies' embeddings on a 2-dimensional plane. As illustrated in Figure 4bb we show the embeddings of the first 25 attacker policies. The darker the color, the earlier the policy is generated.
We can see that the attackers can generate diverse attack policies spontaneously without explicit encouragement like intrinsic rewards to continuously lower the victim team's 's win rate.
Defense Effectiveness
Perturbation Range Generalization
0.8
Average Win Rate
Average Win Rate
0.6
0.6
0.4
0.4
ROMAO
0.2
ROMAO
0.2
RanPert
RanPert
+
QMIX
QMIX
0.0
2
10
100
w/o defense
defense
perturbation range
(a)
(b)
Figure 5: (a) Policies' performance without and with the defense module. (b) Policies' performance under different perturbation ranges. ROMAO denotes the QMIX policy enhanced by our proposed framework. RanPert denotes the policy trained under random perturbations of amount in terms of the l norm. QMIX denotes the QMIX policy that is trained with no perturbations.
5.4 DEFENSE MODULE
Figure ((a) shows the effectiveness of our proposed defense module QMIX in the map of 5m_vs_6m in Section 4.4 The win rate in the figure is tested under the random perturbation of amount 10 in terms of the & norm, and the left three bars show the win rates of the trained policies without the defense module, and the right three show the improved performance with the defense module. RanPert achieves the best performance among the three policies because, in practice, we train RanPert under random perturbations of amount C 10. It means that the training and testing environments are the same for RanPert so that it can achieve excellent performance. However, in reality, we cannot know in priority what kinds of perturbations in store policies would be facing. The performances all these policies are improved with the defense module without any further learning, demonstrating the effectiveness of proposed defense module.
5.5 POLICY GENERALITY
As illustrated in Figure 5()) we show the performances of different policies under different ranges of random perturbations the map of 5m_vs_6m ROMAO, RanPert, and QMIX are trained in the same ways as in Section 5.4] We can see that QMIX and Ranpert perform best as the testing settings are the same as their training settings (perturbations of amount and 10). Under other circumstances, ROMAO, the QMIX policy after the training of our robust framework, achieves the best performance even under the perturbations of 100.
CONCLUSION
This paper proposes new robust MARL training framework, RAMAO, for observation perturba- tions, which any MARL algorithm can employ to improve its robustness against possible attacks on agents' observations, The experimental results show that the attacker can generate diverse attack patterns to excavate the weaknesses of the victim MARL policy. After training, the MARL policy can defend against various kinds perturbations, and the defense can be further enhanced with our proposed defense module. Future work can concentrate how deal with concurrent attacks multiple agents efficiently as the combination blast cannot be avoided to attack multiple agents in ROMAO when the number of agents increases.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Wendelin Boehmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In ICMI,, pp- 980-991, 2020.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pp. 1724-1734, 2014.
Wei Du and Shifei Ding. A survey on multi-agent deep reinforcement learning: From the perspective of challenges and applications. Artificial Intelligence Review, 5465):3255-3238, 2021.
Xin Du, Jiahai Wang. Siyuan Chen, and Zhiyue Liu. Multi-agent deep reinforcement learning with spatio-temporal feature fusion for traffic signal control. In ECML, pp. 470-485, 2021.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, pp. 2974-2982, 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation enror in actor-critic methods. arXiv preprint arXiv: 180..0007. 2018.
Adam Gleave, Michael Demnis, Neel Kant, Cody Wild, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905. 10615, 2019.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Yizheng Hu and Zhihua Zhang. Sparse adversarial attack in multi-agent reinforcement learning. arXiv preprint NXNiv:2205.0936222 2022.
Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv: 022770400222,,.... 2017.
Shariq Iqbal and Fei Sha. Actor-attention-crittc for multi-agent reinforcement learning. In ICML, pp. 2961-2970, 2019.
Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation. arXiv preprint a5X5v:2109.05490, 2021.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In AAAI, pp. 4213-4220, 2019.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcie and Nicolas Papernot. On the robustness cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2003.03722, 2020.
António Loison, Théo Combey, and Hatem Hajri. Probabilistic jacobian- saliency maps at- tacks. arXiv preprint 2XXi:20077000032 2020.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettiee environments. In NIPS, pp. 6379-6390, 2017.
Warwick Masson and George Dimitri Konidaris. Reinforcement learning with parameterized ac- tions. arXiv preprint
2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPs, 2019.
10

---

Under review as a conference paper at ICLR 2023
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In AAMAS, pp. 2040-2042, 2018.
Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. Philosophical Magazine and Journal of Science, ((11):559557, 1901.
Thomy Phan, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Bernhard Kempter, Cornel Klein, Horst Sauer, Reiner Schmid, Jan Wieghardt, Mare Zeller, and Claudia Linnhoff-Pope Poohoff-PPopin. Learning and testing resilience in cooperative multi-agent systems. In AAMAS, pp. 1055-1063, 2020.
Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia Linnhoff- Popien. Resilient multi-agent reinforcement learning with adversarial value decomposition. In AAAI, pp. 11308-11316, 2021.
Lemrll Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In ICML, pp. 2817-2826, 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foer- ster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1485, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Phillip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-ager challenge. In AAMAS, pp. 2186 2288, 2019.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In ICMI., pp. 5887-5896, 2019.
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks deep RL. arXiv preprint a0Xiv:2106.088 2021.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-deccomoosition networks for cooperative multi-agent learning based on team re- ward. In AAMAS, pp. 2085 2087, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, arXiv preprint 1rxXiv:706.03760 2017.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In ICML, pp. 3571155880 2018.
Chongjie Zhang and Victor R. Lesser. Coordinated multi-agent reinforcement learning in networked distributed pomdps. In AAAI, pp. 764-770, 2011.
Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. ICLR, 2021.
Yiren Zhao, Ilia Shumailov, Han Cui, Xitong Gao, Robert Mullins, and Ross Anderson. Blackbox attacks on reinforcement learning agents using approximated temporal information. In DSN-W, pp. 16-24, 2020.
Ming Zhou, Jun Luo, Julian Villela, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, Aurora Chongxi Huang, Ying Wen, Kimia Hassan- zadch, Daniel Graves, Dong Chen, Zhengbang Zhu, Nhat M. Nguyen, Mohamed Elsayed, Kun Shao, Sanjeevan Ahilan, Baokuan Zhang, Jiannan Wu, Zhengang Fu, Kasra Rezaee, Peyman Yadmellat, Mohsen Rohani, Nicolas Perez Nieves, Yihan Ni, Seyedershad Banijamali, Alexan- der Imani Cowen-Rivers, Zheng Tian, Daniel Palenicek, Haitham Bou-Ammar, Hongbo Zhang, Wulong Liu, Jianye Hao, and Jun Wang. SMARTS: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09976 2020.
Ziyuan Zhou and Guanjun Liu. Romfac: robust mean-field actor-critic reinforcement learning against adversarial perturbations on states. arXiv preprint arXXv:2205.07222
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MORE RELATED WORK
A.1.1 COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING
We introduce more about the value function factorization methods like QMIX in MARL here. VDN (Sunchag et al., 2018) and QMIX (Rashid et al. (2018) are two value-based factorization meth- (IGM) (Son et al. 2019) principle,
ods in cooperative MARL and follow the Individual-Globan which asserts the consistency between joint and local greedy action selections by the joint value function (Toe(T a) and individual value functions (Q((r'aa)) i=
VT T, arg maxQ tot (T, acA
arg maxQ1 (r),a") a'EA
arg maxQn (T" aa*) E
VDN (Sunehag et al., 2018) factorizes the global value function Q.V (T, a) as the sum of all the agents' local value funtions [Q' (Q', =
a VDN (T. (r
(4)
QMIX (Rashid et al. 2018) extends VDN by factorizing the global value function Qiot a monotonic combination of the agents' local value funtions
QQII
(T, a) as
QMIX (T, ar.). > O.
VI W,W, 0Q (T',a'))))
(5)
We mainly test framework ROMAO QMIX for its proven performance in various papers. QMIX uses hyper-net conditioned on the global state to generate the weights and biases of the local Q-values and uses the absolute value operation to keep the weights positive to guarantee mono- tonicity.
A.1.2 ADVERSARIAL ATTACKS IN COOPERATIVE MARL
Some researchers focus on the setting where some teammates may betray and minimize their shared return. Phan et al. (2021) and Phan et al. (2020) propose to train competing teams of protagonist and antagonist agents of varying sizes to improve resilience against arbitrary agent changes. Li al. 2019) extend MADDPG with a minimax objective to make the learned policy robust and behave well even with strategies not seen during training. Agents update policies considering a worst-case scenario; assuming that all other agents act adversarially.
A.2
ENVIRONMENTS
A.2.1 STARCREFT MICROMANAGEMENT BENCHMARK (SMAC)
SMAC (Samvelyan et al. 2019 is a combat scenario of StarCraft II unit micromanagement tasks. We consider partial observation setting, where agent can only see circular area around it with radius equal to the sight range, which is set to 9. We train the ally units with reinforcement learning algorithms beat enemy units controlled by the built-in AI. At the beginning of each episode, allies and enemies are generated at specific regions on the map. Every agent takes action from the discrete action space at each timestep, including the following actions: no-op, move [direction], attack [enemy id), and stop. Under the control of these actions, agents can move and attack in continuous maps. MARL agents will get a global reward equal to the total damage done to enemy units at each timestep. Killing each enemy unit and winning the combat (killing all the enemies) will bring additional bonuses of 10 and 200, respectively. After training, the agents need to learn a feasible policy to win the battle even when there is imbalance between the allies and the enemies. For example, in map 5m vs 6m, agents need to learn to focus fire, i.e., jointly attack and kill enemy possible.
units one after another, to reduce the enemy force
12

---

Under review as a conference paper at ICLR 2023
A.2.2 PREDATOR-PRED (PP)
At the beginning of each episode, predators and prey spawn at random positions in a grid world. Agents can move in one of the four compass directions, remain immobile, or ortry to catch any adjacent prey. The prey can move and can be caught only if there are no empty grids around the prey and at least two predators adjacent to it execute the "catch" action concurrently. Once some agents successfully capture prey, the agents will receive a reward 10, and both the agents and the prey will be removed from the grid. However, if agent exccutes "catch" action and no prey is captured, all agents would receive a punishment complete. In this paper, eight predators and eight prey generated at random positions in a 10 x 10 grid world at the start of an episode. We consider all agents have restricted field of vision with a range of 2 (5 5 grids around the agent).
-2. These features further complicate the overall task that
the agents required
A.3 MORE EXPERIMENTAL RESULTS
Maps
2s3z
3m
3s_vs_5z
5m_vs_6m
Methods
Vanilla QMIX
97.7.1..3
97.910.2
89.5+22.0
6.,5
Random QMIX
99.2:10.8
98.3
98.3+0.7
73.14:5.4
Attack Mode 1
One-agent QMIX
84.2++2.8
92.9+1.7
96.3+0099
65.0+t4.1
ROMAO
98.1
96.8+10.3
98.6
75.6.13.1
Vanilla QMIX
95.3
98.3+0.5
88.8+2.11
72.9+7.8
Random QMIX
98,8:10.9
98.9-1.6
99.2+0.77
72.6+44.3
Attack Mode 2
One-agent QMIX
83.3+-2.2
94.3+11.4
95.0++1.6
67,4+-.2
ROMAO
97.9++9.8
95.3++2.1
99.3+
77.1+2.1
Vanilla QMIX
95.4++1.
0.0+0.00
39.3+
21.2++2.2
Random QMIX
98.1+
98.9+0.9
77.5+-2.9
63.1+334
Attack Mode 3
One-agent QMIX
96.9 98.6-10.6
75.6
91.8+1..5
55.6
ROMAO
98.5
92.411.6
63.7
Table Average test win rates of different methods under various attack modes with limited pertur- bation range of amount 2 in terms of the & norm.
Maps
2s3z
3m
3s_v8_52
5m_vs-6m
Methods
Vanilla QMIX
89.1
76.7+1.3
21.1+3.88
20.8
Random QMIX
96.9+-0.6
96.5
96.7++0.7
12.2+0088
Attack Mode
One-agent QMIX
80.8+22.4
92.3+22.3
28.3+33.6
14.4+
ROMAO
97.1+1.2
91.811..
81.242.3
41.7+11.9
Vanilla QMIX
91.7+11.4
7.87+
20.3+:2.8
26.2++2.3
Random QMIX
98.8++1.0
98.3+10.9
97.6++0.7
65.1
Attack Mode 2
One-agent QMIX
80.1
92.4112.7
29.643..8
28.3+...0
ROMAO
96.8
93.3
80.612.22
51.1433
Vanilla QMIX
1.4+1.0
0.3
48.0+13.4
0.0+00.0
Random QMIX
12,2+11.2
91.2-+2.5
44.04-2.0
0.0+00.0
Attack Mode 3
One-agent QMIX
32.5+4..1
88.742.6 87.0
83.5+4.00 86.0-E.00
0,0
ROMAO
79.6+2.33
4.2+11.2
Table 3: Average test win rates of different methods under various attack modes with limited pertur- bation range of amount 10 in terms of the É norm.
Table shows the results of our experiments about the average test win rates and correspondent standard deviations of different methods under various attack modes with a limited perturbation range of amount 2 in terms of the e norm. As the perturbation is relatively subtle, ROMAO-
13

---

Under review as a conference paper at ICLR 2023
Maps
2s3z
3m
3s vs 5z
5m 6m
Methods
Vanilla QMIX
73.6++2.4
57.7+-0.5
0.1+0..2
0.8++0.8
Random QMIX
86.8++2.6
9..3++08
90.0++1.5
9.7-t2.77
Attack Mode
One-agent QMIX
71.0+2..
62.2+3.2
0.1
0,8:00.8
ROMAO
93.8++2.0
8..4
73.34:.2
26.7.1399
Vanilla QMIX
83.1
59.6+1.9
0.0:0..0
3.8::0.5
Random QMIX
95.8:11.3
96.3+.09
90.44:2.7
55.6
Attack Mode 2
One-agent QMIX
76.7:44.4
62.642.88
0.0+0.0
11.34188
ROMAO
95.6+2.6
83.3+1.7
76.443.0
29.7
Vanilla QMIX
0.5++0.2
0.0+0.0
40.5+3.22
0.04-000
Random QMIX
0,4
90.4
24.042..
0,0
Attack Mode 3
One-agent QMIX
0,0
47.243.0
76.441.6 89.2+1.0
0.0
ROMAO
29.2+
61.1+2.77
3.1+1.1
Table 4: Average test win rates of different methods under various attack modes with limited pertur- bation range of amount 20 terms of the & norm.
trained policy does not show a great advantage over other methods while still dominating in most scenarios.
and the results of our experiments about the average test win rates and correspondent standard deviations of different methods under various attack modes with limited perturbation range of amount 10 and 20 in terms of the & norm. From the experimental results, we can see that ROMAO can defend against different attacks. Although Random QMIX and One-agent QMIX perform better than ROMAO some environments (c.g., 3s_vs_52 they all experience significant drop in performance some other environments(c.gg. 5m 6m). On the contrary, the performance of ROMAO is relatively stable.
A.4 SOFTWARR
We use the following software versions:
Python 3.8
SMAC 1.0 (Samvelyan et al.
2019)
Py'Torch 1,12.1
(Paszke al. 2019
A.5
HARDWARE
We use the following hardware:
NVIDIA RTX A4000
12th Gen Intel(R) Core(TM) 19-12900K
A.6 IMPLEMENTATION AND PAPPER-PRAAMTERRS
We implement ROMAO based on the puthor-provided implementation HyAR (Li al. 2021 and QMIX (Rashid et al. 2018). We list the hyper-parameters of the attacker's network, HyAR (Li al. 2021) and ROMAO in table the other parameters remain the same as the default for the pymarl framework (Samvelyan et al. 2019).
14

---

Under review as a conference paper at ICLR 2023
Hyper-parameters
Value
Actor learning rate
3 10
Critic learning rate
3 x 10
Batch size
256
Attacker's Network
Optimizer
Adam
Q-network
layers ReLU activated MIPs with 256 units
Policy Network
3 layers ReLU activated MLPs with 256 units
VAE training steps
10a
VAE batch size
128
HyAR
HyAR batch size
256
Discrete action dim
8
Parameter action dim
64
Number of iterations
3 x 10
Test interval
10
Test episodes
20
ROMAO
Perturbation C
10
Buffer size
5000
Target update interval
200
Table 5: The dyper-parameee setting of ROMAO.
15


Reviews:
[{"review_id": "u76s8fhtbQ", "reviewer": "Reviewer_G76o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new training framework for multi-agent reinforcement learning (MARL) to improve the robustness of MARL agents by generating state/observation perturbation on strategically selected agent during training. They additionally propose a defense module to  prevent the MARL agent from being attacked by observation perturbation. Experiments on SMAC and predator-prey (PP) environmenst are presented to illustrate the effectiveness of the adversarial training framework.", "summary_of_the_review": "I think the current paper is more like a technical report on what the authors did to perform adversarial training on a MARL setting. I do not see a systematic study into why the authors select such framework or any discussion on the convergence guarantee of the adversarial training framework. Some details are missing such as the training of attacker and the defense system.", "strength_and_weaknesses": "Strengths:\n- The new method employs a hybrid action space to select agent to attack along with the perturbation. The authors also incorporate encoder-decoder structure so that the learning is conducted on a latent repsentation space.\n- The approach is shown to make the MARL agent robust to various attacks in SMAC and PP environments.\n\nWeaknesses:\n- My major concern is that the approach presented here is somewhat heuristic. For example, there is no discussion on the convergence guarantee when performing adversarial training. The authors presents an alternating training approach between the attacker and victim agents. I wonder if they happen to select this or the authors have tried other approach and this seems to work well. More details about this are welcome.\n- Another concern is about how the attacker is trained. The authors provide 3 attack mode but do not clearly describe how attackers are trained under those modes. More details should be provided.\n- The defense module is vaguely described and it is not easy to understand the detailed steps without going through the code. It would be great to have more details of this procedure.\n- In a multi-agent setting, there is possibility that two or more agents get attacked simultaneously which is not considered in the paper. I wonder what the authors think about extending the current approach to this setting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m5r3fyQe8P", "reviewer": "Reviewer_gLvj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies observation perturbation attacks in cooperative multi-agent reinforcement learning (MARL), where the attacker can choose one agent to attack and inject noise with a bounded norm into its observations to minimize the long-term return of the system. The paper considers a test stage attack where a MARL policy has already been trained, and the attacker can observe both the underlying model and the MARL policy. The paper adopts an alternate training approach to train a robust defense policy and evaluates the approach in two MARL environments. ", "summary_of_the_review": "The paper considers a state perturbation attack that targets a single agent in a cooperative multi-agent reinforcement learning (MARL) system and adopts alternative training to obtain a robust policy. Due to the simplification of the threat model considered, the problem is similar to state perturbation against single RL. The proposed attack and defense methods thus follow known techniques with little adaptation and do not provide new insights into MARL security. ", "strength_and_weaknesses": "Recent work has focused on state perturbation attacks against a single RL agent. It is interesting to study the multi-agent setting. The alternate training approach for robust RL was proposed by Pinto et al. (2017) and has recently been applied to train a robust policy for single-agent state perturbation by Zhang et al. (2021). Due to its generality, the application of this approach to the multi-agent setting considered in the paper is straightforward. That being said, I am still surprised that the paper only provides a short description of the approach without providing any details.\n\nThere are two problems with the alternate training approach, none of which has been addressed in the paper. First, it has a very high training complexity due to the alternate training of the MARL policy and the attack policy, and it is unclear if the approach will converge. Second, the policy thus trained can be conservative. As shown in the experiments, the robust policy obtained this way is often weaker than the policy trained using adversarial training against random attacks. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8yv9TGpMn2", "reviewer": "Reviewer_8Gkv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the adversarial robustness of MARL, with a focus on observation attacks. The proposed attacker can decide both which agent to attack and how to perturb its observation, via a hybrid-action attacker. Then the paper proposes a robust training approach by alternately training the attacker and the agents. An additional defense module is proposed which reconstructs the attacked observation based on the observations of teammates. Experiments in StarCraftII and Predator-Prey have shown the effectiveness of the proposed attacks and defenses.", "summary_of_the_review": "This paper studies an interesting and important problem, but the proposed method is of limited novelty and may suffer from efficiency issues. The presentation of the paper is good, while the completeness of experiments can be further improved.\n\n\n---- After rebuttal ---\n\nThank the authors for the responses. I think this line of research is indeed interesting. But I think the current work has relatively limited contribution, so I will maintain my score.", "strength_and_weaknesses": "Strengths:\n\n1. The problem of observation attack in MARL environments is relatively new and interesting.\n2. The paper provides attack and defense results in various scenarios, and the visualizations are good for understanding. \n3. The paper is in general well-writte and easy-to-follow.\n4. The idea of the defense module is interesting.\n\nWeaknesses:\n\n1. My major concern is on the novelty of the paper. \n- The MARL attack problem is a straightforward extension of the single-agent attack problem, as the attack just need to learn an agent id in addition to the observation perturbation. Such an additional action output is easy to achieve with DNN parametrization. It would be more interesting and less trivial if the attacker attacks more than one agents simultaneously (e.g. perturb the observations of up to K agents). \n- The alternate training idea is commonly used by many prior works [1,2,3] to improve the agent robustness, which makes the novelty of the paper limited. What is more, prior works focus on the robustness of a single victim agent, whereas this paper aims to improve the robustness of an MARL system. In this case, I would be worried about the efficiency of alternate training, given that training multi-agent policies is usually more difficult and less stable than training a single agent. \n\n2. The experiment section looks interesting, but not adequate to justify the significance of the work. For example, what is the clean performance of the proposed training method (i.e., the reward of the ROMAO models when there is no attack)? This is an important metric for practical use of the robust training approaches [1,2]. It is possible that the model overly adapt to attacks while not achieving good natural performance.\n\n3. The defense module is interesting, but the details are not illustrated much in the paper. This method is reasonable when the observations have redundant information. For examples, agents look at the same object from different angles. However, if the observation information of each agent does not overlap, such a reconstruction may not be achievable. In addition, since the agent does not know who is attacked, it is possible that the other agents' observations are corrupted. How do they deal with this case? More details of this defense would be appreciated.\n\n\n\n\n[1] Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. ICLR, 2021.\n[2] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. ICLR 2022.\n[3] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. ICML 2017.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject