Title: Robust Multi-Agent Reinforcement Learning against Adversaries on Observation
Authors: Chenghe Wang
Yuhang Ran
Lei Yuan
Yang Yu
Zongzhang Zhang
Venue: ICLR 2023
Keywords: multi-agent reinforcement learning
robust reinforcement learning
cooperative multi-agent systems
adversarial training

Abstract: With the broad applications of deep learning, such as image classification, it is becoming increasingly essential to tackle the vulnerability of neural networks when facing adversarial attacks, which have been widely studied recently. In the cooperative multi-agent reinforcement learning field, which has also shown potential in real-life domains, little work focuses on the problem of adversarial attacks. However, adversarial attacks on observations that can undermine the coordination among agents are likely to occur in actual deployment. This paper proposes a training framework that progressively generates adversarial attacks on agents' observations to help agents learn a robust cooperative policy. One attacker makes decisions on a hybrid action space that it first chooses an agent to attack and then outputs the perturbation vector. The victim policy is then trained against the attackers. Experimental results show that our generated adversarial attacks are diverse enough to improve the agents' robustness against possible disturbances. 

TL;DR: We propose a training framework that progressively generates adversarial attacks on agents' observations to help agents learn a robust cooperative policy. 

Review discussion:
Strengths:
1. The problem of observation attack in MARL environments is relatively new and interesting.
2. The paper provides attack and defense results in various scenarios, and the visualizations are good for understanding.
3. The paper is in general well-writte and easy-to-follow.
4. The idea of the defense module is interesting.
Weaknesses:
1. My major concern is on the novelty of the paper.
- The MARL attack problem is a straightforward extension of the single-agent attack problem, as the attack just need to learn an agent id in addition to the observation perturbation. Such an additional action output is easy to achieve with DNN parametrization. It would be more interesting and less trivial if the attacker attacks more than one agents simultaneously (e.g. perturb the observations of up to K agents).
- The alternate training idea is commonly used by many prior works [1,2,3] to improve the agent robustness, which makes the novelty of the paper limited.
What is more, prior works focus on the robustness of a single victim agent, whereas this paper aims to improve the robustness of an MARL system. In this case, I would be worried about the efficiency of alternate training, given that training multi-agent policies is usually more difficult and less stable than training a single agent.
2. The experiment section looks interesting, but not adequate to justify the significance of the work.
For example, what is the clean performance of the proposed training method (i.e., the reward of the ROMAO models when there is no attack)? This is an important metric for practical use of the robust training approaches [1,2]. It is possible that the model overly adapt to attacks while not achieving good natural performance.
3. The defense module is interesting, but the details are not illustrated much in the paper. This method is reasonable when the observations have redundant information. For examples, agents look at the same object from different angles.
However, if the observation information of each agent does not overlap, such a reconstruction may not be achievable.
In addition, since the agent does not know who is attacked, it is possible that the other agents' observations are corrupted. How do they deal with this case? More details of this defense would be appreciated.
[1] Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. ICLR, 2021.
[2] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. ICLR 2022.
[3] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. ICML 2017.
Clarity: this paper is clearly presented.
Quality: the quality of this paper is okay but limited.
Novelty: the novelty of this paper is relatively limited.
Reproducibility: code and instructions are provided, although I did not run it.
NO.
Thanks for your thoughtful comments! We offer some clarification to your questions here and different parts we would try to improve in our future work.
**Q1:** The novelty of this paper is limited and the situations under which two or more agents are attacked should be considered.
**A1:** That is indeed an interesting line of research and we would focus on situations in which two or more agents are under attack in our future work.
In addition, we will further improve our defense module since currently it only supports the reconstruction of a single agent's observation.
**Q2:** About the metrics used to demonstrate the method's performance.
**A2:** We use win rates as our metric of policy performance as this is commonly used when it comes to SMAC benchmarks. We did not show many results where there is no attack but we can see from Fig. 5(b) that ROMAO does not lose much performance when the perturbation range is 0. Illustrated in Tab. 2 in the appendix, ROMAO still keeps a good performance when the perturbation is small. We would include more results under no attack in our main text in our next revision to show that ROMAO can still achieve good natural performance.
**Q3:** More details about the defense module need to be introduced.
**A3:** The defense module needs communications between agents. As agents' observations can overlap, it is possible to reconstruct one agent's observation from its neighbors' observations. We train such a reconstructor during the centralized training phase. We will include more details about this in our next version.
Recent work has focused on state perturbation attacks against a single RL agent. It is interesting to study the multi-agent setting. The alternate training approach for robust RL was proposed by Pinto et al. (2017) and has recently been applied to train a robust policy for single-agent state perturbation by Zhang et al. (2021). Due to its generality, the application of this approach to the multi-agent setting considered in the paper is straightforward. That being said, I am still surprised that the paper only provides a short description of the approach without providing any details.
There are two problems with the alternate training approach, none of which has been addressed in the paper.
First, it has a very high training complexity due to the alternate training of the MARL policy and the attack policy, and it is unclear if the approach will converge.
Second, the policy thus trained can be conservative. As shown in the experiments, the robust policy obtained this way is often weaker than the policy trained using adversarial training against random attacks.
The problem studied is almost identical to the single-agent setting since (1) a fully cooperative MARL environment is considered; (2) the attacker is external to the system and can only target a single agent; (3) the attacker has full knowledge about the MDP model and the joint MARL policy. With these simplifications, the attacker's problem (when the MARL policy is fixed) is again single-agent MDP. The only difference compared with previous work on single-agent state perturbation is that now the attacker also needs to decide which agent to attack in addition to the perturbation action. The hybrid action space does not introduce significant new challenges, and the paper simply adopts the HyRL approach of Li et al. (2021) to address that.
The paper mentions very briefly a defense component where an agent reconstructs its observation from the observations of its neighboring teammates, which sounds interesting.
However, no details are provided.
It looks like the paper only considers random attacks in the experiments according to the three attack modes defined in Section 5.2, which is inconsistent with the three attack types mentioned in Section 3.
In particular, it is unclear why the paper does not consider the worst-case attack that responds optimally to the fixed MARL policy.
NO.
Thanks for your inspiring comments! We offer some clarification to your questions here and hope they could be helpful.
**Q1:** More detailed description of the alternating training approach should be added.
**A1:** It is truly a critical part that we should emphasize in our main text. We will introduce the alternating training approach in more detail in our next version.
**Q2:** What about the training complexity of this approach?
**A2:** As the attacker adapts hybrid action space and a single-agent policy to perturb the observations of the multi-agent system, the training complexity for the attacker is not a burden.
In addition, the multi-agent policy converges fast after a switch of training and we would include a more detailed training curve in our next revision.
**Q3:** Why the results of worst-case attacks are not included?
**A3:** Since we consider the situation of black-box attacks, we want to examine the performance of our robust policy under some general attacks like random perturbations.
However, worst-case attacks are also critical to show the lower bound of our policies' performance and we would include that in our next revision.
**Q4:** More details about the defense module need to be introduced.
**A4:** The defense module needs communications between agents. As agents' observations can overlap, it is possible to reconstruct one agent's observation from its neighbors' observations. We train such a reconstructor during the centralized training phase. We will put more effort into this module in our future work and include more details about this in our next version.
Strengths:
- The new method employs a hybrid action space to select agent to attack along with the perturbation. The authors also incorporate encoder-decoder structure so that the learning is conducted on a latent repsentation space.
- The approach is shown to make the MARL agent robust to various attacks in SMAC and PP environments.
Weaknesses:
- My major concern is that the approach presented here is somewhat heuristic.
For example, there is no discussion on the convergence guarantee when performing adversarial training. The authors presents an alternating training approach between the attacker and victim agents. I wonder if they happen to select this or the authors have tried other approach and this seems to work well. More details about this are welcome.
- Another concern is about how the attacker is trained. The authors provide 3 attack mode but do not clearly describe how attackers are trained under those modes. More details should be provided.
- The defense module is vaguely described and it is not easy to understand the detailed steps without going through the code. It would be great to have more details of this procedure.
- In a multi-agent setting, there is possibility that two or more agents get attacked simultaneously which is not considered in the paper. I wonder what the authors think about extending the current approach to this setting.
The clarity of the paper is decent but there are still key details missing. There are not many adversarial training framework for MARL and this paper aims to advance this research direction which is good. T
NO.
Thanks for your inspiring comments! We offer some clarification to your questions here and hope they could be helpful. We also include some particular parts that we would try to improve in our future work.
**Q1:** The approach is somewhat heuristic.
**A1:** Adversarial training has been used in many fields, like image classification and reinforcement learning, and its effectiveness in improving agents’ robustness has been empirically shown in [1, 2]. The alternating training approach is chosen in consideration of the fact that we need to incessantly find the vulnerability of our multi-agent cooperative policies, as one round of training cannot ensure that our robust policy can deal with all different attacks.
Concerning the convergence guarantee, we will put more effort into this part in our future work.
**Q2:** How the attackers are trained listed in the table?
**A2:** To show the robustness of our method against general attacks, these attacks tested are not trained and are randomly generated. Attack mode 1 denotes the attack in which the attacker randomly chooses an agent at every timestep and generates random perturbations. Attack mode 2 denotes the attack that at the start of an episode, the attacker randomly chooses an agent and generates random perturbations on that agent’s observation throughout the entire episode. Attack mode 3 denotes the attack in which the attacker randomly chooses an agent at the start of an episode and generates perturbations on only one random dimension of the agent’s observation.
**Q3:** More details about the defense module need to be introduced.
**A3:** The defense module needs communications between agents. As agents’ observations can overlap, it is possible to reconstruct one agent’s observation from its neighbors’ observations. We train such a reconstructor during the centralized training phase. We will include more details about this in our next version.
**Q4:** What about situations in which two or more agents are attacked?
**A4:** That is an interesting research direction and we plan to focus on this part in our future research to enrich our work.
[1] Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In *ICLR*, 2021.
[2] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In *ICML*, pp. 2817–2826, 2017.

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: The paper presents a new Multi Agent RL (MARL) method to improve robustness by generating adversarial perturbations on the observation space during training. The reviewers find some of the ideas in hybrid action space and perturbation quite interesting. However, there are several concerns, from motivation (theoretical or descriptive), lack of training details, and an unclear presentation of the roles of different agents. Detailed comments are present in the reviewer's feedback. At this point, the reviewers do not find this work ready for publication at ICLR.
justification_for_why_not_higher_score: Major concerns are a lack of motivation for the algorithm and unclear description of the framework.
justification_for_why_not_lower_score: N/A

Decision: Reject