Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new training framework for multi-agent reinforcement learning (MARL) to improve the robustness of MARL agents by generating state/observation perturbation on strategically selected agent during training. They additionally propose a defense module to  prevent the MARL agent from being attacked by observation perturbation. Experiments on SMAC and predator-prey (PP) environmenst are presented to illustrate the effectiveness of the adversarial training framework.
strength_and_weaknesses: Strengths:
- The new method employs a hybrid action space to select agent to attack along with the perturbation. The authors also incorporate encoder-decoder structure so that the learning is conducted on a latent repsentation space.
- The approach is shown to make the MARL agent robust to various attacks in SMAC and PP environments.

Weaknesses:
- My major concern is that the approach presented here is somewhat heuristic. For example, there is no discussion on the convergence guarantee when performing adversarial training. The authors presents an alternating training approach between the attacker and victim agents. I wonder if they happen to select this or the authors have tried other approach and this seems to work well. More details about this are welcome.
- Another concern is about how the attacker is trained. The authors provide 3 attack mode but do not clearly describe how attackers are trained under those modes. More details should be provided.
- The defense module is vaguely described and it is not easy to understand the detailed steps without going through the code. It would be great to have more details of this procedure.
- In a multi-agent setting, there is possibility that two or more agents get attacked simultaneously which is not considered in the paper. I wonder what the authors think about extending the current approach to this setting.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is decent but there are still key details missing. There are not many adversarial training framework for MARL and this paper aims to advance this research direction which is good. T
summary_of_the_review: I think the current paper is more like a technical report on what the authors did to perform adversarial training on a MARL setting. I do not see a systematic study into why the authors select such framework or any discussion on the convergence guarantee of the adversarial training framework. Some details are missing such as the training of attacker and the defense system.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies observation perturbation attacks in cooperative multi-agent reinforcement learning (MARL), where the attacker can choose one agent to attack and inject noise with a bounded norm into its observations to minimize the long-term return of the system. The paper considers a test stage attack where a MARL policy has already been trained, and the attacker can observe both the underlying model and the MARL policy. The paper adopts an alternate training approach to train a robust defense policy and evaluates the approach in two MARL environments. 
strength_and_weaknesses: Recent work has focused on state perturbation attacks against a single RL agent. It is interesting to study the multi-agent setting. The alternate training approach for robust RL was proposed by Pinto et al. (2017) and has recently been applied to train a robust policy for single-agent state perturbation by Zhang et al. (2021). Due to its generality, the application of this approach to the multi-agent setting considered in the paper is straightforward. That being said, I am still surprised that the paper only provides a short description of the approach without providing any details.

There are two problems with the alternate training approach, none of which has been addressed in the paper. First, it has a very high training complexity due to the alternate training of the MARL policy and the attack policy, and it is unclear if the approach will converge. Second, the policy thus trained can be conservative. As shown in the experiments, the robust policy obtained this way is often weaker than the policy trained using adversarial training against random attacks. 


clarity,_quality,_novelty_and_reproducibility: The problem studied is almost identical to the single-agent setting since (1) a fully cooperative MARL environment is considered; (2) the attacker is external to the system and can only target a single agent; (3) the attacker has full knowledge about the MDP model and the joint MARL policy. With these simplifications, the attacker's problem (when the MARL policy is fixed) is again single-agent MDP. The only difference compared with previous work on single-agent state perturbation is that now the attacker also needs to decide which agent to attack in addition to the perturbation action. The hybrid action space does not introduce significant new challenges, and the paper simply adopts the HyRL approach of Li et al. (2021) to address that. 

The paper mentions very briefly a defense component where an agent reconstructs its observation from the observations of its neighboring teammates, which sounds interesting. However, no details are provided. 

It looks like the paper only considers random attacks in the experiments according to the three attack modes defined in Section 5.2, which is inconsistent with the three attack types mentioned in Section 3. In particular, it is unclear why the paper does not consider the worst-case attack that responds optimally to the fixed MARL policy. 
summary_of_the_review: The paper considers a state perturbation attack that targets a single agent in a cooperative multi-agent reinforcement learning (MARL) system and adopts alternative training to obtain a robust policy. Due to the simplification of the threat model considered, the problem is similar to state perturbation against single RL. The proposed attack and defense methods thus follow known techniques with little adaptation and do not provide new insights into MARL security.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the adversarial robustness of MARL, with a focus on observation attacks. The proposed attacker can decide both which agent to attack and how to perturb its observation, via a hybrid-action attacker. Then the paper proposes a robust training approach by alternately training the attacker and the agents. An additional defense module is proposed which reconstructs the attacked observation based on the observations of teammates. Experiments in StarCraftII and Predator-Prey have shown the effectiveness of the proposed attacks and defenses.
strength_and_weaknesses: Strengths:

1. The problem of observation attack in MARL environments is relatively new and interesting.
2. The paper provides attack and defense results in various scenarios, and the visualizations are good for understanding. 
3. The paper is in general well-writte and easy-to-follow.
4. The idea of the defense module is interesting.

Weaknesses:

1. My major concern is on the novelty of the paper. 
- The MARL attack problem is a straightforward extension of the single-agent attack problem, as the attack just need to learn an agent id in addition to the observation perturbation. Such an additional action output is easy to achieve with DNN parametrization. It would be more interesting and less trivial if the attacker attacks more than one agents simultaneously (e.g. perturb the observations of up to K agents). 
- The alternate training idea is commonly used by many prior works [1,2,3] to improve the agent robustness, which makes the novelty of the paper limited. What is more, prior works focus on the robustness of a single victim agent, whereas this paper aims to improve the robustness of an MARL system. In this case, I would be worried about the efficiency of alternate training, given that training multi-agent policies is usually more difficult and less stable than training a single agent. 

2. The experiment section looks interesting, but not adequate to justify the significance of the work. For example, what is the clean performance of the proposed training method (i.e., the reward of the ROMAO models when there is no attack)? This is an important metric for practical use of the robust training approaches [1,2]. It is possible that the model overly adapt to attacks while not achieving good natural performance.

3. The defense module is interesting, but the details are not illustrated much in the paper. This method is reasonable when the observations have redundant information. For examples, agents look at the same object from different angles. However, if the observation information of each agent does not overlap, such a reconstruction may not be achievable. In addition, since the agent does not know who is attacked, it is possible that the other agents' observations are corrupted. How do they deal with this case? More details of this defense would be appreciated.




[1] Huan Zhang, Hongge Chen, Duane S. Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. ICLR, 2021.
[2] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. ICLR 2022.
[3] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. ICML 2017.
clarity,_quality,_novelty_and_reproducibility: Clarity: this paper is clearly presented. 

Quality: the quality of this paper is okay but limited.

Novelty: the novelty of this paper is relatively limited.

Reproducibility: code and instructions are provided, although I did not run it.
summary_of_the_review: This paper studies an interesting and important problem, but the proposed method is of limited novelty and may suffer from efficiency issues. The presentation of the paper is good, while the completeness of experiments can be further improved.


---- After rebuttal ---

Thank the authors for the responses. I think this line of research is indeed interesting. But I think the current work has relatively limited contribution, so I will maintain my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new Multi Agent RL (MARL) method to improve robustness by generating adversarial perturbations on the observation space during training. The reviewers find some of the ideas in hybrid action space and perturbation quite interesting. However, there are several concerns, from motivation (theoretical or descriptive), lack of training details, and an unclear presentation of the roles of different agents. Detailed comments are present in the reviewer's feedback. At this point, the reviewers do not find this work ready for publication at ICLR.
justification_for_why_not_higher_score: Major concerns are a lack of motivation for the algorithm and unclear description of the framework.
Decision: Reject