Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Needs Clarification (NC): a submission which does not meet the reviewing criteria and needs clarification for its described problem or solution
review_content: summary_of_contributions: The work experimentally investigates the robustness of a benchmark multi-agent reinforcement learning algorithm and presents some observations under different noise levels.
strengths_and_weaknesses: I appreciate the authors’ aim of comprehensively studying the robustness of MARL under attacks. However, the current work is too preliminary to draw any meaningful conclusion. Experiments with different platforms, MARL algorithms, and attack methods should be conducted. The authors observe that a large $\sigma$ improves the agents’ reward in some environments. Can the authors come up with an explanation for this and conduct experiments accordingly to verify their hypothesis?
suggested_changes: Please see the comments above.
review_title: null

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: This work investigates the robustness of MADDPG towards gaussian noise attacks, specifically noise injected in the observation and environments. The experiments were performed on a multi-agent benchmark environment and the results shows MADDPG reacts differently towards the gaussian noise depending on the type of noise injection and the type of environment. Sometimes, the noise injection resulted in better performance, and sometimes it degrades the MADDPG performance.
strengths_and_weaknesses: Strengths:

1. The paper is relatively well written and easily understandable in terms of clarity and results.
2. The claims made by the authors also reflects the results shown, which I truly appreciate, especially since no overall trends were observed.
3. Basic submission requirements were met.

Weakness:

1. Certain parts of the paper in terms of discussion of relevant literature and description of the experiments to aid reproducibility could be improved upon.
2. The title is slightly misleading, as it seems to be more general than the actual experiments performed. 


suggested_changes: 1. The title of the paper is slightly misleading as it suggests that multiple MARL algorithms were evaluated. I suggest the authors change the title to be more concise since the experiments were only performed on MADDPG

2. In terms of discussion of related works, I understand that the authors may be restricted in terms of space to cite and discuss the many related works in literature. As such, perhaps the authors could cite more literature survey type papers focusing on robustness, rather than citing 1 or 2 specific papers related to adversarial attacks.

3. From the paper, it wasn't very clear what execution-based noise means. I could understand how the observation noise perturbations were performed, but it wasn't clear how the execution perturbations were achieved. It would greatly improve the reproducibility of the paper if the authors could add an additional paragraph on it.

4. I'm also curious if the results shown in the Figures were results averaged over multiple runs. As RL is famous for its sensitivity towards random initializations, it would make the results more convincing if the results were averaged over multiple runs. If the results were indeed already averaged, it would be helpful to show the standard deviations of the rewards as well.

All the best!
review_title: null

Meta-review: metareview: This work explore the robustness of MADDPG towards gaussian noise attacks for MARL. This work is interesting and novel. Also, the performance seems to be nice. The presentation is also nice. However, the authors should conduct extensive experiments. Also, the English expression should be improved to make it more clear.
confidence: 5: The area chair is absolutely certain
Decision: Invite to present