Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game
Authors: Simin Li
Jun Guo
Jingqiao Xiu
Ruixiao Xu
Xin Yu
Jiakai Wang
Aishan Liu
Yaodong Yang
Xianglong Liu
Venue: ICLR 2024 poster (2024)
Keywords: Multi-agent reinforcement learning
Robustness
Game Theory
Adversarial Attack
Primary area: reinforcement learning
TL;DR: We study robust cooperative MARL against Byzantine adversary using a Bayesian game approach

Abstract: In this study, we explore the robustness of cooperative multi-agent reinforcement learning (c-MARL) against Byzantine failures, where any agent can enact arbitrary, worst-case actions due to malfunction or adversarial attack. To address the uncertainty that any agent can be adversarial, we propose a Bayesian Adversarial Robust Dec-POMDP (BARDec-POMDP) framework, which views Byzantine adversaries as nature-dictated types, represented by a separate transition. This allows agents to learn policies grounded on their posterior beliefs about the type of other agents, fostering collaboration with identified allies and minimizing vulnerability to adversarial manipulation. We define the optimal solution to the BARDec-POMDP as an ex interim robust Markov perfect Bayesian equilibrium, which we proof to exist and the corresponding policy weakly dominates previous approaches as time goes to infinity. To realize this equilibrium, we put forward a two-timescale actor-critic algorithm with almost sure convergence under specific conditions. Experiments on matrix game, Level-based Foraging and StarCraft II indicate that, our method successfully acquires intricate micromanagement skills and adaptively aligns with allies under worst-case perturbations, showing resilience against non-oblivious adversaries, random allies, observation-based attacks, and transfer-based attacks.

Published as a conference paper at ICLR 2024
BYZANTINE ROBUST COOPERATIVE MULTI-AGENT RE- INMERTEMENT LEARNING AS A BAYESIAN GAME
Simin Li', Jun Guo', Jingqiao Xiu', Ruixiao Xu', Xin Jiakai Wang", Aishan Liu!,4, Yaodong Yang', Xianglong Liu 'KKLSEE Lab, Beihang University, China 2Zhongguancun Laboratory, China
*Institute of Artificial Intelligence, Peking University & BigAI, China "Institute of data space, Hefei Comprehensive National Science Center, China
ABSTRACT
In this study, we explore the robustness of cooperative multi-agent reinforcement learning (c-MARL) against Byzantine failures, where any agent can enact arbitrary, worst-case actions due to malfunction or adversarial attack. To address the uncer- tainty that any agent can be adversarial, propose Bayesian Adversarial Robust Dec-POMDP (BARDec-POMDP) framework, which views Byzantine adversaries nature-dictated types, represented by separate transition. This allows agents learn policies grounded on their posterior beliefs about the type of other agents, fostering collaboration with identified allies and minimizing vulnerability adver- sarial manipulation. We define the optimal solution to the BARDec-POMMD as an ex interim robust Markov perfect Bayesian equilibrium, which we proof exist and the corresponding policy weakly dominates previous approaches as time goes to infinity. To realize this equilibrium, we put forward two-timescale actor-critic algorithm with almost sure convergence under specific conditions. Experiments on matrix game, Level-based Foraging and StarCraft indicate that, our method successfully acquires intricate micromanagement skills and adaptively aligns with allies under worst-case perturbations, showing resilience against non-oblivious adversaries, random allies, observation-bast based attacks, and ransfer-based attacks.
INTRODUCTION
Cooperative multi-agent reinforcement learning (c-MARL) RRahidd al., 2018 Yu et al., 2021 Kuba ct al.] 202T) has shown remarkable efficacy in managing groups of agents with aligned interests in complex tasks (Vinyals et al. 2019 Berner al. 2019.. Nevertheless, real-world applications often deviates from the presumption of full cooperation. In rohot swarm control (Hüttenrauch et al. 2019). individual robots may act unpredictably due to hardware or software malfunctions, or even display worst-case adversarial actions if compromised by a non-oblivions adversan GGleave et al. 2019 Lin
al. 2020 Dinh et al. 2023 Liu et al. 2019 2020a b 2023 Wang al., 2021). Such uncertainty of allies undermine the cooperative premise of c-MARL, rendering the learned policy non-robust.
In single-agent reinforcement learning (RL), robustness under uncertainty is addressed through maximin optimization between an uncertainty set and robust agent within the framework of robust Markov Decision Processes (MDPs) (Nilim& & El Ghaoui 2005 Tyengar 2005; Wiesemann et al.. et al. [2077] Tessler et al. 2019 Zhang et al. 2020a). However, ensuring robustness in c-MARL when dealing with uncertain allies presents a greater challenge. This is largely due to the potential for Byzantine failure (Yin et al., 2018 Xue et al., 2021), situations where defenders are left in the dark regarding which ally may be compromised and what their resulting actions might be.
To address Byzantine failures. employ Bayesian game approach, which treats Byzantine adversarics as types assigned by nature, with each agent operating unaware of others' type. We formalize robust c-MARL Bayesian Adversarial Robust Dec-POMDD (BARDec-POMDP)
where existing robust MARL researches (Li et al.) 2019; Sun et al. 2022) Phan et al. 2020 2021 can be reinterpreted pursuing an ex ante equilibrium (Shoham & Leyton- 2008), viewing
"oorreponniigg Authors. E-mails: yaodong.yang @ @ku.edu..nn x buaa.edu.cn.

---

Published as a conference paper at ICLR 2024
all other agents as potential adversaries. However, these methods might not yield optimal outcomes as they can mask the trade-offs between the equilibria that cooperative and craustness-focuseee agents
aim for. Moreover, this approach can result in overly conservative strategies (Li et al. 2019 Sun et al. 2022), given the low likelihood of adversaries taking control of all agents.
Instead, seek ex interim mixed -treteyt robust Markov perfect Bayesian equilibrium, which weakly dominates the policy of ex ante equilibrium in previous robust MARL studies as time goes to infinity. Agents in our ex interim equilibrium makes decisions based on its inferred posterior belief over other agents, enhancing cooperation with allies and defense against adversaries concurrently. To realize this equilibrium, we derive a robust Harsanyi-Bellman Bellmam equation for value function update and introduce a two-timescale actor-critic algorithm, with almost sure convergence under certain assump- tions. Experiments in matrix game, Level- Foraging and StarCraft II shows our defense exhibits intricate micromanagement skills and adaptively aligns with allies under worst-case perturbations. Consequently, our defense outperforms existing baselines under non-oblivious adversaries, random allies, observation- attacks and transfer-based attacks by large margins.
Contribution. Our contributions are two-fold: first, we theoretically formulate Byzantine adver- saries in c-MARL as a CARDec-POMDP, and concurrently pursues robustness and cooperation by targeting an ex interim equilibrium. Secondly, to achieve this equilibrium, we devise an actor-critic algorithm that ensures almost sure convergence under certain conditions. Empirically, our method exhibits greater resilience against a broad spectrum of adversaries on three c-MARL. environments.
Related Work. Our research belongs to the field of robust RL, theoretically framed as robust MDPs Nilim & El Ghaoui 2005) Iyengar 2005: Tamar et al. 2013 Wiesemann et al.) 2013). This framework trains a defender to counteract worst-case adversary amid uncertainty, which can stem from environment transitions (Pinto et al.] 2017: Mankowitz et al., (001)), actions (Tessler et al.) 2019). states (Zhang et al.) 2020a] 2021) and rewards (Wang et al. (2020. In robust MARL, action uncertainty has been a central focus. M3DDPG (Li et al. 2019) enhances robustness in MARL through agents taking jointly worst-case actions under small perturbation budget. Evaluation was done via one agent consistently introducing worst-case perturbations. This is later known as adversarial policy (Gleave et al., 2019) or non-oblivious adversary (Dinh et al. | (2023), practical and detrimental form of attack. Follow-up works either enhanced M3DDPG (Sun et al. 2022) or defended against uncertain adversaries by presupposing each agent as potentially adversarial (Nisioti
al. 202 Phan et al. 2020 2021). which our LARDec-POMDP formulation interprets seeking a conservative ex ante equilibrium. Another approach by Kalogiannis et al. (2022) studies a special case that the adversary is known. Besides action perturbation, studies have also explored robust MARL under uncertainties in reward (Zhang et al. 2020c), environmental dynamics (Zhao et al..
2020), and observations (Han al. 2022 He et al.
2023 Zhou & Liu 2023).
Bayesian games and their MARL applications represents another relevant field. With roots in Harsanyi's pioneering work (Harsanyi 1967), Bayesian games have been used analyze games with incomplete information by transforming them into complete information games featuring chance moves made by nature. Within MARL, Bayesian games have been utilized coordinate varying agent types, a concept known as ad hoc coordination (Albrecht & Ramamoorthy 2015] Albrecht al. 2016; Stone et al. 2010; |Barrett et al., 2017]. This problem was theoretically framed as a stochastic Bayesian game and solved using the Harsanyi-Bellman
ad hoc coordination algorithm (Albrecht & Ramamoorthy 2015). Subsequent research has concentrated on agents with varying types [ 2019). open ad hoc teamwork (Rahman et al. 2022). and human coordination (Tylkin ct al. 2021: Strouse et al. 202I). Our work differs from these works by assuming a worst-case, oon-oblivious adversary with conflicting goals, whereas in ad hoc coordination, agents have common goals and non-confticting secondary objectives (Grosz & Kraus 1999, Mirsky et al. 2022).
2
PROBLEM FORMULATION
2.1
COOPERATIVE MARL AND ITS FORMULATION
The problem of c-MARL can be formulated as Decentralized Partially Observable Markov Decision Process (Dec-POMDP) (Olihhoek & Amato 2016). defined as a tuple:
G = SW,,0,,,AA, R,7),
2

---

Published as a conference paper at ICLR 2024
where V = {1, (I,...,.) .) is the set of N agents, S is the global state space, O observation space, with O the observation emission function. = XiEN is the joint action space, P S x A + ((S) is the state transition probability, mapping from current state and joint actions to probability distribution over the state space. R R::. $ x A + IR is the shared reward function for cooperative agents and 7 [0, 1) is the discount factor.
XieNO' is the
At time t and global state & E S, each agent adds current observation oj to its history and gets Ii - [o). ab.... ...].. Then, each agent i selects its action oj E A'' using its policy x (-IHH)), which maps current history to its action space. The global state then transitions to 81+-1 according to transition probability P(st-1 [St at), with a, = {o,,..... a, the joint actions. Each agent receives a shared reward = R(s,, at). The goal of all agents is to learn a joint policy π = IIiEv '' that maximize the long-term return [Tx E 7 a T(. H.)].
2.2 BAYESIAN ADVERSARIAL ROBUST DEC-POMDP
In numerous real-world situations, some allies may experience Byzantine failure (Yin et al., 2018 Xue et al., (021) and thus, not perform cooperative actions expected. This includes random actions due to hardware/softwarc error and adversarial actions if being controlled by an adversary, which violates the fully cooperative assumption in Dec-POMDP. We propose Boyesian Adversarial Robust Dec-POMDP (BARDec- POMDD) to cope with uncertainties in agent actions, defined as follows:
: = ,,0,S,,,,,,,,,P R.
where N, S, O, O, A. y represent the number of agents, global state space, observation space, observation emission function, joint action space and discount factor, following Dec-POMDP.
As depicted in Fig. [ BARDec-POMDP views the Byzantine adversary sition characterized by type 0 and adversarial policy म. At the start of each episode, type is selected from the type space + = X indicates the agent is cooperative and O' = 1 signifies adversaries. At time f, if agent is assigned 0 1, the ac- tion at taken by cooperative agent with policy (-(()))) is replaced by action af sampled from an adversary with policy (-H) 0). The attack process is characterized by action perturbation probability pa (ā, a, ft 0) Ile +' (H; (). 00 + +5(a; a:)aa(11--0)) (1-0') that maps joint actions a, to joint actions with perturbations āc. where 8(.) is the Dirac delta function. Note that the actions of cooperative agents and adversaries are taken simultancously. Finally, the state transition probability P(s1+1 (8; az) takes the perturbed actions and output the state of next timestep. The shared reward rs = " ā,) for (cooperative) agents is defined over perturbed actions. Given type 6, the value function can be defined as (s) E 1> y'relso 8, a, ~ (-|H,,, å, H-,HH,, 0)]. We leave the goal of adversary and robust agents to Section. 2.3 and 2.4 below.
uncertain tran-
Environment (((t)1))...
ag
Og
with 8¹ {0, 1}. 8¹
Action Perturbation pp (a,la,, ft
Agents a-rT( (H₂)
Figure I: Framework of c-MARL with Byzantine adversaries. The action taken by agents with 8* = 1 are replaced by the adversary policy #².
Our MARDec-POMID formulation is flexible and draws close connection with current literature. Regarding type space, Dec-POMDP can be viewed as a BARDec DecPPODPP with θ = ON. Robust MARL approaches. such as M3DDPG (Li et al. 2019) and ROMAX (Sun et al. 2022 assumes agents are entirely ad adversaiial, which refers to type space θ = 1y- Subsequent robust MARL researchs (Phan et al., 2020 2021 Nisioti et 2021), though not explicitly defining the type space, can be integrated in our BARDec- POMDP. Our formulation also draws inspiration from state-adversart MDP (Zhang et al.) 2020a) which considers adversary as a part of decision making process, and probabilistic action robust MDP (Tessler et al. 2019) by their formulation of action perturbation.
2.3 THREAT MODEL
The robustness towards action perturbations in both single and multi-agent RL has gained prominence since the pioneering works of (Tessler et al., 2019 Li et al. 2019). Action uncertainties, formulated as a type of adversarial attack known as adversarial policy Gleave et al., 2019 Wu et al. 2021 Guo
et al. 2021), or non-oblivious adversary Dinn al. 2023 represent pragmatic and destructive
3

---

Published as a conference paper at ICLR 2024
form of attack that is challenging counter. In line with these works, propose practical threat model with certain assumptions on attackers and defenders.
Assumption 2.1 (Attacker's 's capability and limitations). At the onset of an episode, the attacker can select and arbitrarily manipulate the actions of agents with type Hr = 1. Within episode, the type cannot be altered and we assume there is only one attacker.
Our main focus in this paper is to model action uncertainties of unknown agents in c-MARL as rypes shaped by nature and to advance corresponding solution concept. In line with (Li al. 2019), we assume one agent is vulnerable to action perturbations in each episode. In real-world, the type space be more complicated, potentially involving adversaries controlling multiple agents (Nisioti et al., 2021) perturbing actions intermittently (Lin et al. 2017), or featuring a non-binary type space (Xie) al. 2022). These variations can be viewed as straightforward extensions of our work.
Proposition 2.1 (Existence of worst-case adversary). For any robust c-MARL with fixed agent policy, a worst-case (i.e., most harmful) adversary exists.
Proof sketch. Since defender policies are fixed, they can be considered part of the environment transitions for attackers. Thus, the attackers solve RL problem. See full proof in Appendix, A.
Assumption 2.2 (Defender's capability and limitations). The defender tion during training stage, including global state and information of other agents. However, during testing, the defender relies solely on partial observations and is agnostic of the type of other agents. The defender's policy is fixed during an attack and must resist the worst-case adversary R..
all available informa-
2.4 SOLUTION CONCEPT
In this section, we first introduce the non-optimal solution concept seek by existing robust c-MARL methods, then pose our optimal solution con- cept for BARDec-PONI POMDP. Specifi- cally, existing robust c-MARI. meth- ods blindly maximize reward without considering the type of others. This akin to an ex ante equilibrium in Bayesian game (Shoham & Leyton- Brown 2008), where agents make de- cisions based on the prior belief about the types of other agents.
Cooperative Equilbrium
Robuar Equilbrium
Expectation
Adaptivv Adjustment
Adversarial Allack
Atfack
Attsck
Uncertaintyy Set
(a) ante
(b) interim
Figure 2: ex ante RMPBE obseures differences between each type by taking expectation, while interim RMPBE adapts to current type.
JEE and adversarial policy #.A (# (AA,i JiEN forms ante robust Markov perfect Bayesian equilibrium (RMPBE), for all p(0), E S, (0 x A)",
Definition 2.1 (ex ante robustness). A joint cooperative policy TEA = (M. EAi :.)EA
(T EA (-HHA,)))),)))))))))))) (. (H, E arg max Ep()
min =(1H)) A(-(HH0)
Vo(s)]
T with Vo(s)= E Pa (ala म, (d'))) + EvES
By maximizing the expected value under prior p(0) over types, as illustrated Fig. 2 the policy might struggle to balance the different equilibrium corresponding to cooperation and robustness against different agents as (Byzantine) adversaries. In contrast, we propose a more refined ex interim robustness concept, such that under current history, each agent make optimal decisions to maximize their expected value from their posterior belief of current type, with following assumptions:
Assumption 2.3. Assume belief and policy are updated under following conditions: (1) Consistency. At each timestep each agent updates its belief b; = p(0) H;) of the current type by Bayes' rule. (2) Sequential rationality. Each policy maximizes the expected value function under belief b3.
Here we use F' (-I b²) to denote the explicit dependence on belief b¹. The two conditions are common in dynamic games with incomplete information (Shoham & Leyton-Brown 2008).
4

---

Published as a conference paper at ICLR 2024
Definition 2.2 (ex interim robustness). Under Assumption policy n. ( DiEN and adversarial policy a ( 'IEN forms an ex interim robust Markov perfect Bayesian equilibrium, if Vs E S, E (O x A)*
2.3 and let (b") a joint cooperative
nT 21 (-1H,b), b), # (-1,,0) E arg max Epce H) x(|HH6b
min vo(+))
(2)
((11800)
Note that both ex ante and ex interim RMPBE requires optimality of each individual agent i. To reduce notation complexity, we use joint policy and belief instead.
Proposition 2.2 (Existence of RMPBE). Assume RARDec-POMDP of finite agents, finite set of state, observation and action space, agents use stationary policies, the type space compact set, then ex ante and ex interim mixed strategy robust Markov perfect Bayesian equilibrium exists.
Proof sketch. sketch. The proof is done by first showing the policy and its corresponding value function, with uncertainties of the current type and presence of the adversaries, satisfy the requirements of Kakutani's fixed point theorem. Next, by Kakutani's fixed point theorem, there always exists an optimal fixed point corresponding to a mixed strategy RMPBE. See full proof in Appendix. A.2
Unlike c-MARL with optimal deterministic policies (Olichoek al. 2008), in robust c-MARL, a pure-strategy equilibrium is not guaranteed to exist. This is intuitive since zero-sum sum games do not always have a pure-strategy equilibrium. The finding suggests the optimal policies for robust c-MARI. are stochastic. Next, we show the relation between ante and ex interim equilibrium.
Proposition 2.3. Under Assumption 2.3 given finite type space and the prior of each type is not zero, as - 0o.
π E1 (-HH,,by by) weakly dominates (- H2) under the worst-case adversary.
Proof sketch. As f + oc, by the consistency of Bayes' rule, the belief converges to the true type. Thus, ex interim policies that maximize the value function under the true type are guaranteed to weakly dominate (i.e., have value higher equal to) ex ante policies. See full proof in Appendix A.3
3 ALGORITHM
In this section, we explain how to find the optimal solution in Definition 2.2 We start by defining the robust Harsanyi-Bellman equation, update rule of value function which converges a fixed point. Then, we develop a two-timescale actor-critic algorithm that considers belief of others' type, which ensures almost sure convergence under assumptions stochastic approximation theory.
3.1 ROBUST HARSANYI-BELLMAN EQUATION
We first define the Bellman type update of value functions for our ex interim equilibrium, Considering the function before and after action perturbation, we can formulate the Q function via cumulative reward, with posterior belief b' p(0) H') over type:
Q'ss, a, b²) -Ep(o111)
E
r = an a. ~ (-1 bz), A(-HH,, 0)
(3)
('(,, = =
E
7 = ap = =a 2 be,, ae ~ a H 0)
The two Q functions are defined with different purpose. Q' b¹) is the expected Q function before action perturbation, suitable for decision making of defenders, such fictitious self-play (Heinrich Silver 2016) and soft actor-critic (Haarnoja et al. 2018). In this way, the action perturbation can be viewed as part of the environment transition, resulting in P (s'' a, 4, 0) ((s' s, pa (ala, ft 0).
On the other hand, ('(s, a. b') is the Q function with actions taken by the adversary, suitable for policy gradients (Sutton Barto 2018) and decision making of the adversary. For Q'(s, a, H*), the action perturbation is integrated into the policy of robust agents, resulting in mixed policy ((a b, 0) = (a a, म. 0) T TTa H, = (1 8) (aaHH, () b) (()HH, 0). The relationship between the two functions is follows:
Q' a. b')
Σ ((0
pd a, (s. b..
(5)
aEA
5

---

Published as a conference paper at ICLR 2024
Next, formulate the Bellman-type equation for the two Q functions, which we call the robust Harsanyi-Bellnn Bellman equation. This update differs from the conventional approach by considering the posterior belief over other agents and the worst-case adversary.
Definition 3.1. We define the robust Harsanyi-Bellne equation for Q function as:
(')
max min Σp(O)H)) क( s(-|H.b) R(- H.0) BEG
ΣΣ
Σ
P(s'|s, as 01
EEES aEA
a(a'IH' (f',, (6)
R(s, a) + Σ
a'EA
(s,
max min R(s, a) + Σ
P(s'|s, |s a)
Σ p(0
n(. (.HH)
'ES
DE0
Σ . b. 0)0 ('' 8'EAA
This Q function can be estimated via Temporal Difference (TD) loss.
Proposition 3.1 (Convergence). Assume the belief is updated via Bayes' rule, the space of state, actions and belief are finite, updating value functions by robust Barsanyi-Bellmer equation converge to the optimal value Q* (s, a, b') and (2 a, b*).
Proof sketch. sketch. adversaries and Bayesian belief update, and showing our Q function forms contraction mapping, Next, applying Banach's 's fixed point theorem completes the proof. See full proof in Appendix. A.4
The proof is done by combining the standard convergence proof of Q function with
3.2 ACTOR-CRITIC ALGORITHM FOR ex interim ROBUST EQUILIBRIUM
Armed with the robust Harsanyi- -arsann Bellman equation, we propose actor-critic algorithm to achieve our proposed ex interim equilibrium with almost sure convergence under certain assumptions. We first derive the policy gradient theorem for robust c-MARL. Assume policies of robust agents and adversaries are parameterized by F == )ie and F == (# kee respectively, forming a mixed
policy म J.. (1- 0) Th + +-- t... Define the performance for a robust agent in the episodic case as (Ø) (a) (s, b'] and (zero-sum) adversary J (d) Eap (' b'), where (s((s) is the state visitation frequency. The policy gradients for n and #3 are then defined as:
Theorem 3.1. The policy gradient theorem for robust agent and adversary i is:
y. J' (s) (.)
[11 0') To (a' H Q') (s,
(8)
3. J (80) Es-p (1
0' log t Há' (),
(9)
The policy gradient naturally depends on ('ss,, a, b²), which related to policy gradient and decision of adversaries. Specifically, 0 cuts off the gradient of robust agents T&. with 0¹ = 1 and cut off the gradient of adversary at.. with 0 0. The detailed derivation is deferred to Appendix. A.5
Convergence. In zero-sum Markov games, achieving convergence through policy gradients remains challenging, with current theoretical results being dependent on specific conditions (Daskalakis ct al.) 2020: Zhang et al. 2020b Kalogiannis et al.) 2022). In this paper, prove that, under
certain assumptions in stochastic approximation theory (Borkar 1997 Borkar & Meyn 2000 Borkar 2009), applying a two-timescale update for both adversaries and defenders, stated in Theorem 3.1 guarantees almost sure convergence (i.e., converge with probability 1) to an ex interim RMPBE. detailed proof is provided in Appendix [A.6aa an application of stochastic approximation. With this two-timescale update, the adversary's 's policy is updated on a faster timescale and is essentially cquilibrated, while the defender's policy is updated on slower timescale and remains quasi-static. Despite these ad advances, establishing finite sample, global convergence guarantees without restrictive assumptions remains an open problem, warranting future research.
Finally, we suggest update rules for critic and belief networks. Assuming the critic Q' (s, a, b²) is parameterized by sh. As for belief by, the calculation of b' via Bayes' rule requires assess to the policy

---

Published as a conference paper at ICLR 2024
of other agents, which is not possible during deployment. To remedy this, we approximate the belief = maxę (0) H)) using neural network parameterized by &. The objectives to update critic and belief network are:
min (R(ss - (s', a', )-- + Q as))
(10)
min - log (0:
- 8) log Pe Pe (0)
(11)
with critic trained via TD loss and belief network trained by binary cross entropy loss. See the pseudo-code for our algorithm in Appendix.
EXPERIMENTS
at m a
a
d' m of
(a) Toy
(b) LBF
(c) SMAC
Figure 3: Environments used in our experiments. The toy iterative matrix game proposed by Han et al. (2022). We use map 12x/2-4p-3f-c for LBF and map 4m 3m for SMAC.
Environments. To validate the efficacy of proposed approach, we conducted experiments three benchmark cooperative MARL environments, as shown Fig. Environments include a toy iterative matrix game proposed by (Han et al., (202),, rewarding XNOR XOR actions at different state, 12x/2-4p-3f-c
Level- -asee Foraging (LBF) (Papoudakis et al., 2020) and 4m vs 3m of the StarCraft Multi-agent Challenge (SMAC) (Samvelyan al. 2019). which reduce to 3m in the presence of an adversary.
Baselines. Our comparative study includes MADDPG (Lowe al. 2017), M3DDPG (Li et al. 2019). MAPPO YYu et al. (02II, RMAAC He et al. (2023 ex ante robust MAPPO (EAR-MAPPO), MAPPO variant of (Phan et al. 2020 Zhang et al. 2020c) that considers ante equilibrium, which is also an ablation of our approach without belief. We dubbed our method ex interim robust MАPPO PEIR-MAPPO and add an ideal case which grants access to true type, labelled "True Type". It's worth noting that we couldn't directly adapt M3DDPG onto the MAPPO framework due to its reliance on Q(s, a), a component not compatible with MAPPO's use of V(s) as a critic. More experiment details are given in Appendix. C For fair comparison, all methods use the same codebase, network structure and hyperparameters. //github. comm DIG- G BwweaeeihheBeee EIR MAPPO
Code and demo videos available at https
Evaluation protocol. each environment with N cooperative agents, the robust policy was trained using five random seeds. Attack results were compiled by launching attacks each of the N agents using the same five seeds, yielding a total of x attacks per environment. We plot all results with 95% confidence interval.
Evaluated attacks. We consider four types of threats. (1) Non-oblivious adversaries (Gleave et al. 2019): fix the trained policy and deployed zero-sum, worst-case adversarial policy attack each agents separately, (2) Random agents: an agent perform random actions from a uniform distribution, possibly via hardware software failure (labelled as "random"), (3) Noisy observations: we add to bounded adversarial noise (Lin et al. 2020) with perturbation budgets E {0.2, 0.5, 1.0) to the observation of an agent (denoted as ="). (4) Transferred adversaries: attackers initially train
policy on a surrogate algorithm, then directly transfer the attack to target other algorithms.
4.1 ROBUSTNESS OVER NON-OBLIVIOUS ATTACKS
We first evaluate our performance under the most arduous non-oblivious attack, where an adversary manipulate any agent in cooperative tasks and execute arbitrary learned worst-case policy. The cooperation and attack performance on three environments are given in Fig. 4 Across all environments, FIR-MAPPO consistently delivers robust performance close to the maximum
7

---

Published as a conference paper at ICLR 2024
MADDPG
M3DDPG
MAPPO
RMAAC
EAR-MAPPO
EIR-MAPPO
True Type
Cooperalive, Toy
Cooperalive, LBF 12x12-4p-3Fe
Cooperalive, SMAC 4m vs 3m
1.0
20
Reward
60
90
Reverd
0.8
0.6
Roward
15
70
0.4
10
60
0.2
50
0.0
0.0 0.5
2.0 3.0 3.5
0.0
0.5
1.5
0.5 1.0
2.0 Timestaps
3.5 107
Timesteps
167
Timesteps
167
Adversary, Toy
Adversarial, LBF 12x12-4p-3f-0
Adversary. SMAC 3m
50
Reward
40
30
Reward
0.8
0.6
15
20
0.4
10
0.2
0.0
1
2
2
3
5
5
Timesleps
1e6
Timesteps
Timesteps
Figure Cooperative and robust performance on three c-MARL environments. EIR-MAPPO achieves higher robust performance against non-oblivious adversaries and have cooperative perfor- mance on par with baselines. Reported on seeds for cooperation and x N attacks.
reward achievable in each environment under attack (50 for Toy, 1.0 for LBF, 20 for SMAC), outperforming baselines by large margins and displays robustness equalling the ideal True Type defense. Concurrently, EIR-MAPPO maintains cooperative performance on par with MAPPO.
Unfocused Fire
Unfocused
Swayed
Allies.
Kiting
Kiting
Focus
Fire
(a) MADDPG/MSDDPG
(b) MAPPO/RMAAC
(c) SEAR-MAPHR
(d) EIR-MAPPO
Figure 5: Agent behaviors under attack. Red square indicates the adversary agent. Existing methods are either swayed, having unfocused fire or perform bad kiting. contrast, our EIR-MAPPO learns kiting and focused fire simultaneously, under the presence of worst-case adversary.
As illustrated in Fig. [] a detailed examination of each method's behaviour under attack enriches our comprehension of robustness, with adversaries marked by red square. First, MADDPG and M3DDPG can be easily swayed by adversaries. Fig- 5a a downward-moving adversary easily diverts two victims from the battle, resulting in single victim facing three opponents. As for MAPPO and RMAAC, agents fail to master useful micromanagement strategies, such as kiting or focused fire during comba Consequently, the agent do not exihibit any cooperation skills under attack and not skillful enough to win the game. As for EAR-MAPPO, agents occasionally demonstrate kiting but fall short in executing focused fire. They spread fire two enemies instead of concentrating fire on one. Furthermore, even successful kiting can be compromised. In Fig. |c| the adversary advances, causing two half-health agents mistakenly believe that ally is coming its aid, and thus retreats to kite the enemy. This. however, leaves another low-health ally vulnerable to enemy fire and immediately being eliminated. Finally, we find that both EIR-MAPPO and True Type demonstrate focused fire and kiting, proving resistant to adversarial agents. Illustrated in Fig. [dd] two low-health agents retreat avoid being eliminated, while an agent with high health advances to shield its allies, showcasing classic kiting behaviour. Moreover, allies coordinate to eliminate enemies, leaving one enemy nearly unscathed and another at half health.
4.2 ROBUSTNESS OVER VARIOUS TYPE OF ATTACKS
Apart from the worst-case oblivious adversary, c-MARL encounter various uncertainties in real world, ranging from allies taking random actions, having uncertainties in observations, or transferred
Micromanagements are granular control strategies for agents to win in StarCraft Il (Samvelyan al. 2019 Kiting enables agents to evade enemy fire by stepping outside the enemy's attack range, thereby compelling the enemy to give chase rather than attack; focused fire requires taking enemies down one after another.
8

---

Published as a conference paper at ICLR 2024
adversary trained on alternate algorithms. We examine the robust performance of all methods under such diverse uncertainties in Fig. 6 The rows signify uncertainties while the columns represent the evaluated methods. Diagonal entries (i.e., blocks with the same uncertainty and evaluated method) denote non-oblivious attacks. For each uncertainty (column), the method of highest reward was marked by red square. Furthermore, since the True Type represents ideal scenario, if it secures the highest reward, we mark the method that gained the second -highest reward as well.
Toy
LBF 12x12-4p-3f-
SMAC 3m
20
MADDING
72.76
05.05 51
4.8
5.02
10.63
10.7
01
19:35
18.52
M3DDPG-550
5.58 0.01
22.13
63
69
4853
55.115552
1415552.66
0.00
9.03
0.78
0.05
0.80
0.39
0.64
5.13
11:25
16.56
MAPPO 26.51
74:65
4.75
39.67
25
01
0.09
0:13
0.53
0.64
0.80
0.51
0.50
48
11
7.85
10. 72
64
19
18.54
42.0
0.09
0:
0:11
91
0.81
6.0
8.67
9:43
15:35
19
10.24
Uncertaintees
EAR MAPPO
43.80
77.16-
0.61
0.62
6.99
10.32
EIR MAPPO
5621
55.3
50
57.11
69:3
46
-0.04
0.11
0.73
0.61
0.81
0.32
0.62
7.22
4.73
10.8110.7
15.69
18
12.88.68
Type
42
52.01
37.88
47.07
-0.07
0:13
0.77
0.8
0.85
0.67
0.52
7.11
0.74
9:09
10.98
06
18
16.9
Random 00.0350
50
60.3
8.84
6.63
18.29
12.02
19.6
19.4
(.0.2-09.7
57:18
59.4
57
04
73.0480011
0.81
0:95
0.05
0.00
18.01
E-0.5-50
33.50
14.00
61-1
26.71
85
0.64
0.55
0:75
0.8
3.8
1.96
0.54
14.010
1.555
9:26
16.
PM
07
17.21
6 50.04
34.22
5.88 61.50 21.58
44.18 45.6
00,00
0.78
0.82
0.93
0.50
11
8:19
19:22
17.52
Average 48 78
41.4..01000 57
51 59.5
0.22
0.77
0.65
0.85
0.80
18.42
MADDPO
MAPPO
RMAAC
EAR AARDD
ERAMPPO
Tepe Tyeo
MADIIO
MAPPO
RMMAC
EAR MAPPD
ERR ER-M
MADOPG
MAPPO
RMAAC
EAR MAR-MAPP
EIR ERRMAPP
The Type
True
Figure 6: Evaluating robustness over diverse attacks, row indicates uncertainties and column indicates evaluated methods. EIR-MAPPO achieves improved robustness almost all uncertainties in LBF and SMAC, while having higher robustness on average in Toy. Results averaged over 5 x attacks.
We first emphasize the effectiieness of our EIR-MAPPO approach. Considering the average reward gained under a broad range of uncertainties, our EIR-MA EIR-MAPPO surpasses baselines by 5.8 % on Toy, 5.88% on LBF, and 25.45% SMAC. Notably, EIR-MAPPO and True Type yields highest reward in almost all LBF and SMAC environments. In toy environment, owing to the existence of two pure-strategy equilibria, algorithms and attacks deploying deterministic strategies can occasionally yield higher rewards by chance. However, given its superior worst-case robustness, our EIR-MAPPO consistently delivers commendable results under all uncertainties.
Our second observation focuses on the relationship between action and observation perturbations. As an algorithm designed to counteract observation uncertainties, RMAAC is robust against observation perturbations, but fails to counter unseen action perturbations. In contrast, our EIR-MAPPO maintains its robustness against observation perturbations, even though it has not encountered the attack before. This resilience is due to the fact that observational attacks ultimately affect agents' choices of actions, which reduces observation uncertainty to a form of action uncertainty.
5 CONCLUSIONS
In this paper, we study robust c-MARL against Byzantine threat, where agents in this system to fail, or being compromised by an adversary. We frame the problem as Dec-POMDP and define its solution as an ex interim RMPBE, such that the equilibrated policy weakly dominates ex ante solutions in prior research, when time goes to infinity. To actualize this equilibrium, we introduce Harsanyi-Bellman equation for value function updates, and an actor-critic algorithm with almost sure convergence under specific conditions. Experimental results show that under worst-case adversarial perturbation, our method can learn intricate and adaptive cooperation skills, and can withstand non-oblivious, random, observation-bast based, and transferred adversaries. As future work, plan to apply our method to c-MARL applications, including robot swarm control (Hüttenrauch et al. 2019). traffic light management (Chu et al. 2019), and power grid maintenance (Xi et al. 2018).
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the National Key Research and Development Plan of China (2022ZD01 16405) and National Natural Science Foundation of China (62306025, 62022009, 62132010 and 62206009).
REFERENCES
Stefano V Albrecht and Subramanian Ramamoorthy. A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. arXiv preprint 2015.
Stefano V Albrecht, Jacob W Crandall, and Subramanian Ramamoorthy, Belief and truth hypothe- sised behaviours. Artificial Intelligence, 235:63-94, 2016.
Karl Astrom al. Optimal control of markov processes with incomplete state information. Journal of mathematical analysis and applications, 10(1):174-2) 205, 1965.
Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone. Making friends the fly: Cooperating with new teammates. Artificial Intelligence, 242: 132:112-7 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0660 2019.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
Vivek S Borkar. Stochastic approximation with two time scales. Systems & Control Letters, 29(5):
291-294, 1997.
Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 48. Springer, 2009.
Vivek Borkar and Sean P Meyn. The ode method for convergence of stochastic approximation and reinforcement learning. SIAM Journal Control and Optimization, 38(2):447-469, 2000.
Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 21))): 1086-1095, 2019.
Junyoung Chung. Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 4rXiv:1412.3555, 2014.
Constantinos
Daskalakis, Dylan Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems, 33: 5527-5540, 2020.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020.
Persi Diaconis and David Freedman. On the consistency bayes estimates. The Annals of Statistics, pp. 1-26, 1986.
Le Cong Dinh, David Henry Mguni, Long Tran-Thanh, Jun Wang, and Yaodong Yang. Online markov decision processes with non-oblivious strategic adversary. Autonomous Agents and Multi-Agent Systems, 37(1):15, 2023.
Arlington M Fink. Equilibrium in a stochastic n-person game. Journal of science of the hiroshima university, series ai (mathematics), 28(1):89-93, 1964.
10

---

Published as a conference paper at ICLR 2024
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint 19Xi::1905.1015, 2019.
Barbara J Grosz and Sarit Kraus. The evolution of sharedplans. Foundations of rational agency, pp. 227-262, 1999.
Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial policy learning in two-player competitive games. International Conference on Machine Learning, pp. 3910-3919, PMLR, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pp. 1861-1870. PMLR, 2018.
Songyang Han, Sanbao Su, Sihong He, Shuo Han, Haizhao Yang, and Fei Miao. What is the solution for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02275, 2022.
John Harsanyi. Games with incomplete information played by "bayesian" players, i-iii part i. the basic model. Management science, 14(3):159-182, 1967.
Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.
Johannes Heinrich and David Silver. Deep reinforcement learning from self-play in imperfect- information games. arXiv preprint arXiv:1603.01121, 2016.
Maximilian Hüttenrauch, Sosic Adrian, Gerhard Neumann, et al. Deep reinforcement learning for swarm systems. Journal of Machine Learning Research, 20(54):1-31, 2019.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Fivos Kalogiannis, loannis Anagnostides, Toannis Panageas, Emmanouil- vaimanouil-Vasileioss Vlatakis- Gkaragkounis, Vaggos Chatziafratis, and Stelios Stavroulakis. Efficiently computing nash equilibria in adversarial team markov games. arXiv preprint arXiv:2208.0220 02204, 2022.
Erim Kardeş, Fernando Ordóñezz, and Randolph W Hall. Discounted robust stochastic games and an application to queueing control. Operations research, 59(2):365-382 2011.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint vrXiv:2109.1125: 2021.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang. and Stuart Russell. Robust agett reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4213-4220, 2019.
Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW), pp. 62-68. IEEE, 2020.
Yen-Chen Lin, Zhang-War Wei Hong, Yuan-Hong Liao, Meng-Li Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint 2017.
Aishan Liu, Xianglong Liu, Jiaxin Fan, Yuqing Ma, Anlan Zhang, Huiyuan Xie, and Dacheng Tao. Perceptual-sensitive gan for generating adversarial patches. AAAI, 2019.
Aishan Liu, Tairan Huang, Xianglong Liu, Yitao Xu, Yuging Ma. Xinyun Chen, Stephen Maybank, and Dacheng Tao. Spatiotemporal attacks for embodied agents. In ECCV, 2020a.
Aishan Liu, Jiakai Wang, Xianglong Liu, Bowen Cao, Chongzhi Zhang, and Hang Yu. Bias-based universal adversarial patch attack for automatic check-out. In ECCV, 2020b.
11

---

Published as a conference paper at ICLR 2024
Aishan Liu, Jun Guo, Jiakai Wang, Siyuan Liang, Renshuai Tao, Wenbo Zhou, Cong Liu, Xianglong Liu, and Dacheng Tao. X-adv: Physical adversarial object attacks against x-ray prohibited item detection. In USENIX Security Symposium, 2023.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiive environments. Advances in neural information processing systems, 30, 2017.
Daniel Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmalcki, Jost To- bias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:19066 07516, 2019.
Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork research. In Multi-Agent Systems: 19th European Conference, EUMAS 2022, Düsseldorf, Germany, September 14-16, 2022, Proceedings, pp. 275-293, Springer, 2022.
Arnab Nilim and Laurent El Ghaoui. Robust control markov decision processes with uncertain transition matrices. Operations Research, 53(5):880-799 2005.
Eleni Nisioti, Daan Bloembergen, and Michael Kaisers. Robust multi-agent q- g-learning in cooperative games with adversaries. In Proceedings of the AAAI Conference on Artificial Intellience, 2021.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentrulized POMDPs. Springer, 2016.
Frans Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353, 2008.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.00 07869, 2020.
Bei Peng. Tabish Rashid, Christian Schroeder de Witt, Plerre-Alexanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34: 12221, 2021.
Thomy Phan, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Bernhard Kempter, Cornel Klein, Horst Sauer, Reiner Schmid, Jan Wieghardt, Mare Zeller, et al. Learning and testing resilience in cooperative multi-agent systems. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, pp. 1055-1063, 2020.
Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia Linnhoff- Popien. Resilient multi-agent reinforcement learning with adversarial value decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1 11308-11 1316, 2021.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017.
Arrasy Rahman, Ignacio Carlucho, Niklas Höpner, and Stefano V Albrecht. A general learn- ing framework for open ad hoc teamwork using graph- policy learning. arXiv preprint arXiv:2210.05444 15.04488, 2022.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference machine learning, pp. 4295-4304. PMLR, 2018.
Manish Chandra Reddy Ravula. Ad-hoc teamwork with behavior-switching agents. PhD thesis, 2019.
Mikayel Samvelyan. Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 0902 04043, 2019.
12

---

Published as a conference paper at ICLR 2024
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 1rXiv:150600244: 02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:I707.000 2017.
Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008.
Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1504-1509, 2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021.
Chuangchuang Sun, Dong-Ki Ki Kim, and Jonathan P How. Romax: Certifiably robust deep multiagent reinforcement learning via convex relaxation. In 2022 International Conference on Robotics and Automation (ICRA), pp. 5503-5510. IEEE, 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Aviv Tamar, Huan Xu, and Shie Mannor. Scaling up robust mdps by reinforcement learning. arXiv preprint 1rX6vv1306.66 6189, 2013.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In International Conference on Machine Learning, pp. 2215-6224.
PMLR, 2019.
Paul Tylkin, Goran Radanovic, and David C Parkes. Learning robust helpful behaviors two- player cooperative atari environments. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1686-1688, 2021.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-33 2019.
Jiakai Wang, Aishan Liu, Zixin Yin, Shunchang Liu, Shiyu Tang, and Xianglong Liu. Dual attention suppression attack: Generate adversarial camouflage physical world. In CVPR, 2021.
Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 6202-6209, 2020.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi- agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35:16509-16521, 2022.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 38(1):153-183, 2013.
Xian Wu, Wenbo Guo, Hua Wei, and Xinyu Xing. Adversarial policy training against deep rein- forcement learning. In 30th USENIX Security Symposium (USENIX Security 21), pp. 1883-1900, 2021.
Lei Xi, Jianfeng Chen, Yuehua Huang, Yanchun Xu, Lang Liu, Yimin Zhou, and Yudan Li. Smart generation control based multi-agent reinforcement learning with the idea of the time tunnel. Energy, 153:977-987, 2018.
Annie Xie, Shagun Sodhani, Chelsea Finn, Joelle Pineau, and Amy Zhang. Robust policy learning over multiple uncertainty sets. International Conference on Machine Learning, pp. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
Wanqi Xue, Wei Qiu, Bo An, Zinovi Rabinovich, Svetlana Obraztsova, and Chai Kiat Yeo. Mis-spoke or mis-lead: Achieving robustness multi-agent communicative reinforcement learning. arXiv preprint arXiv:2108.038800 2021.
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantinerrobust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.0195, 2021.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 21210024-1333 2020a.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on
state observations with learned optimal adversary. arXiv preprint (rXiv:2101.08452 2021.
Kaiqing Zhang, Bin Hu, and Tamer Basar. On the stability and convergence of robust adversarial
reinforcement learning: case study on linear quadratic systems. Advances Neural Information Processing Systems, 33:22056-2222 22068, 2020b.
Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Gene, Sunil Mallya, and Tamer Basar. Robust multi- agent reinforcement learning with model uncertainty. Advances in neural information processing systems, 33:10571- 2020c.
Eric Zhao, Alexander R Trott, Caiming Xiong, and Stephan Zheng. Ermas: Learning policies robust to reality gaps in multi-agent simulations. 2020.
Yifan Zhong, Jakub Grudzien Kuba, Siyi Hu, Jiaming Ji. and Yaodong Yang. Heterogeneeou--agent reinforcement learning. arXiv preprint arXiv:2304.0990 09870, 2023.
Ziyuan Zhou and Guanjun Liu. Robustness testing for multi-agent reinforcement learning: State perturbations on critical agents. arXiv preprint arXiv:2306.00 06/666 2023.
14

---

Published as a conference paper at ICLR 2024
APPENDIX FOR "BYZANTINE ROBUST COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING AS A BAYESIAN GAME"
A PROOFS AND DERIVATIONS
A.1 PROOF OF PROPOSITION 2.1
In this section, we proof worst-case adversary always exist for any BARDec-POMD. with fixed defenders. Considering different numbers of adversaries (i.e., Eiex)) with fixed defenders, the attackers can be seen solving POMDP or Dec-POMDP, where optimal solution exists.
First, EiEN' 0g O, no adversary exists and the environment is reduced a fully cooperative setting. Since adversary do not even exist, the problem becomes vacuous, and any achieves the same (optimal) result.
Second, if EiEN 0; 1, the problem is reduced to POMDP G == (8, O,,, adversary, where S the global state space, O the observation space for adversary, O is the observation emission function, A is the action space of adversary, R SS x A - R is the reward function for adversary. Actions taken by adversary are sampled via adversarial policy at #'( HH; 0). The environment transition for adversary is defined as (( (s, ai) = P($s+11 a,) p po I., (2,0))) az).
N 7) for
where a, is the mixed joint actions after perturbation in BARDec-POMDP. Here, P(s ā.) is the environment transition
which combines with the transition p (a, TT 0,0,0)))
Tie ó(σ) - at) 0 T (-IH)) (1 0') represents the decision of which is fixed and treated as a part of environment transition. Following the proof Astrom et al. (T965), there always exists an optimal policy for POMDP. Thus, there exists optimal adversary.
Third, with EIEN 0, 1, the problem is reduced to Dec-POMDP with S the global state space, Oa XiE(B'=1} O is the observation space of adversaries, O is the observation emission function, R is the reward function for adversary. Actions taken by adversary are sampled via adversarial policy â ~ (- HH, 0). The environment transition for adversary is defined as P( 8t, ā,) P(s++1|Se a.) p (a. |T, á.). Here, P(s 1 âe) is the environment transition for BARDec- POMDP, with the effect of the policy of merged in transition pa (a I, 0, â.) ILiE 5( at) 8' + (-H,)
Rt 7),
(x=(("=1)= action space of adversaries, R S X A
8¹). which is fixed and treated part of transition function. Following the proof of Bernstein et al. (2002), there always exists an optimal policy for Dec-POMDP. Thus, there exists an optimal adversary.
A.2 PROOF OF PROPOSITION 2.2
We proof this by showing our BARDec-POMDP satisfies the requirement of Kakutani's fixed point theorem. Our proof is extension of Kardeş et al. (2011), which shows equilibrium exists in robust stochastic games. In the following proof, we will first state existing results, then shows the existence of our interim RMPBE as generalization of the proof of (Kardey et al., 2011) considering Bayesian update and transforming action uncertainty as a kind of environment uncertainty. The existence proof of ex ante RMPBE rise corollary.
A.2.1 PRELIMINARIES
We first introduce existing results.
Theorem A.1 (Kakutani's fixed point theorem). X is a closed, bounded, and convex set a Euclidean space, and & is an upper semicontinuous correspondence mapping X into the family of closed, convex subsets of X, then 3x E X, s. x E ((x).
Next, we brief the definition of robust stochastic game and its equilibrium concept before we introduce the main result of Kardeş et al. (001)).
15

---

Published as a conference paper at ICLR 2024
In stochastic game G G
with finite state and actions, where / is the indicator of agents, S is the state space, A = XieN is the joint action space, P : S x A + ((S) is the state transition probability, R: : S x A x X R. is the reward function that maps state and actions to reward of each agent, 7 is the discount factor. Let 8 E S, a² E A'', with a the joint actions, S ((A) is the policy for agent i with ss))s, rd R(s,a, is the reward for agent i. A robust stochastic game is a stochastic game with perturbed reward fr E R and environment transition P(s'| (s.a) pr, with R" and p are the bounded perturbation range.
Define the expected value function V (s) under perturbed reward and environment transition T.
recursively through the following Bellman equation:
V (.
= Σ (a'ls) II x (a2)s)))
+ Σ
aEA
J#i
SEES
the worst-case value function V T.R.,, P. (8) is defined Vi (0) min V m. D. (8). The equi- librium policy, if exists, is thus Vi E NN 8 E S,a E A. (+1s) E arg arg max Vi(s), with corresponding value as Vi
(xt,a-"), (5) = maxxi Vi
a.A.P. (s).
Next, let (1-1s) = IIIEN (11s be the joint policy, and the set of value functions for each agent as V ( = (V T (5), V2 (+)) VN (s)). Define mappings / and & by B(x = {V'IV' = max;: min;, & Vi the set of value functions and optimal policies.
p.
( (s)} and (( = ( = v (mi (5)} to be
Theorem A.2 (Kardey et al. (20IT). Assume uncertainties in transition probabilities and payoffs belongs to compact sets, all agents use stationary strategies, then the set of functions fV (s).
Ra, P E pa} } is equicontinuous and V V.,.., (s) continuous all its variables. (( is an upper semicontinuous correspondence mapping ((A) in convex and closed subsets of (AA), which satisfies the assumptions of Kakutani's 's fixed point theorem.
We are now ready to construct our proof.
A.2.2 EXISTENCE OF EX INTERIM RMPBE
We first state the existence of ex interim RMPBE. The proof is conducted by transforming BARDec- POMDP to Dec-POMDI with environmental uncertainties, and applying Bayesian update to value functions. Note that we include some results of Kardey et al. (2011) for a self-contained proof.
First, by definition of our BARDec-POMDP, it can be transformed to a robust Dec-POMDP with adversary environmental dynamics. This is done by combining environment transition P((s41 a) with action perturbation probability pa (a la, म., O), resulting in P(*++1|$ Ā. O) P((++1 (8, a) Po (ála, म. B). Thus, the robust Dec-POMDP can be seen as particular case of stochastic game with shared reward, partial observations and uncertainties in perturbations. Thus, some results of Kardeş et al. (201T) can be taken for our proof. Before that, let us redefine some notations for clarity of our proof.
Let us redefine the expected value function (s(s) using the following Bellman equation:
Vi) = Σ (a H bO)
II n' (a' H b' + 37/2
ΣΣ a, VVV,, S'ESNEA
DEA
Note that Vi (s) assumes the type 0 is known, thus there is uncertainty over 0 and the function is not updated by robust Harsanyi-Bellman equation. The expected value function with The worst-case value func- tion Vi A.A. o (s) and Vi + (s) are defined as V V= = mins Vi and V ()) min+ V/ (s). With optimal (equilibrium) policy defined as m. the corresponding value is Vi (m=t.=-),At..b00 (s) = max,' Vi (. (8). The mappings B and & are similarly defined ((T-i) (V max,, min, Virt, (mrta-').A. (s)) and Φ(π-i) = [Λ(]((( = ((=:a-))) (s)) to be the set of value functions and optimal policies. By Proposition. 3.1 the optimal value at each e S is unique, which denote it vz.
belief, Vi ar (s), is defined by V (s)
16

---

Published as a conference paper at ICLR 2024
Lemma A.1 (Equicontinuity of {Vi R (6), E ((A))) For every e > 0, 38 > 0, for any (+2)
(TT, a, of (s4), (6)) and (T22 V R, (52), 12), V - + 1 (s)))
6, then, VR E (AA), (. - V (#2) <.
Proof. By Lemma of Kardeş et al. 201 the equicontinuity of {V (V)) E A(A) holds, if
we consider as uncertainties ((x' 8, a, n, 0). All we need now is to extend the proof considering belief b²
V ($1) - V V (82)
p(O) H!) (o ()(0)1)) (,2
Σ (P(())))) VA Vain pp(0) - ((2))
Σ [((()))) (11 (V,,) - (())) + - p(0)H))) V(222,((52)
[I(())[] (V,1),((1)) - (3))) p(0)11))) (
[P(() (Vm.)).)) [
p(0)H))) [
Since p(0 p((HH)) is probability function, have p(0 Hj) S 1. Since reward is finite, and E 1,2 defined by discount factor % IV (V,,5,(8;;) is also bounded, iee. IV (..,,,((s)) K.
Now, let
min 1}
V ( (x2) < <&
2 |
minfe,
(1) p(01115)) <
2 K
and let 62). we have:
& (51) V= (82)
>> [Ip(0) H)) (V))) - V
+((({)HH) +(x000))2)+((((00)1))) p(0)HH)) (
BEe
=Σ [Ip(0) Hil 61] + Σ (82 [
9EB
Sc/2 + 6/2=6.
Thus, the set of functions {Vi H, (s), E A(A)} is equicontinuous.
Lemma A.2. (i) a convex set.
Proof. The proof follows Theorem of Kardey al. (201T). By Lemma 2 of Kardey et al. (201T). the
pointwise minimum of an equicontinuous set of function is continuous, V , (s) min; w. s (s)
continuous on all its variables. Besides, V + (s) is defined by discounted factor and is bounded.
Thus, the maxima of V * b (s) exists.
Second, in Proposition 3.1 we have proof that updating Q function by robust Harsanyi-Behh equation yields optimal robust Q value. It rise as simple corollary that the optimal value, Viria (ximm-) A. exists. By equality Vi "1
(ai) min;
EaEA (a (0) Hi (2)111,, 63) Ir' Evess Eaea P(, V a1, (8"). ((T-+) # 00
17

---

Published as a conference paper at ICLR 2024
Third, by Lemma 3 of Kardey et al. (201I) the value function considering worst- adversary
x, bi = min Laed xt (a H ,b) II x (a² H. (H,,)) 3/2
case
Eves Easa
Eoco P(C) (s, a n. O, #. (o (5)] is concave in I' with fixed x and V R (8")
Finally, we show the convexity of ő(T "). Let i 22, E o( o(TT)), with fixed x x By definition
of Vi (mi (,,,,-)),,,,000 VT, E S, E A(O), i
N, V
F. (5) (1-A)V (m. x ,. 2
J,,0,00 = Vi
V "... (.).))))) (9). Thus, AA E [0, 1J. we also have AV
(x a. (8)
Vi R.A.,b (s). By concavity of V (i (5), V (mf ( D... (8) ( (-) T... (s) + (1
X)V( (ns ",, I + +(1 A)m - J. (s). Since by Proposition 3.1 V n bi (s) is
optimal and unique. Thus, Ari,, + (1 - 2 E क( "), क( () is convex set.
Next, we first introduce several lemmas, then show (xx) is an upper semicontinuous correspondence.
Lemma A.3. Let T be the robust Harsanyi-Bellman operator defined in Appendix. TV far 3,.. or (8) is continuous in --.. The set (7V1r) Vis (eA,,-"hA,, is bounded) is equicontinuous.
A.6
Proof. By Lemma 4 of Kardeş et al. (0011).
is continuous and the set (mi, f).... (5) is bounded} is equicontinuous. Since TV (ri.)-)) = TEpoo (a (m (o (s)], the expectation of a continuous function is still continuous, and the expectation over a equicontinuous set is still equicontinuous. This completes the proof.
(nina 9 (s)] (mi,
Lemma A.4. Define the optimal value as vf = [wt, u/.....'g). If If + n', न. + - ,
v² and TH E "(T,"",, then T' <()-),, i.e., p( ") is an upper semicontinuous correspondence.
Proof. The proof is by Lemma 5 of Fink (164). We re-write it here using our notation to make the proof self-contained. Specifically, Vs E S.H E (O x A)" b E (()). De- fine function f(-) as fVV=,t. (s)) min (. T'
II M1 (a') (r)
Eves EieA Evce )P s, at VV) (s'). Recall v is the fixed point. Let m. E
Ó(T [f(")) - -v < ff( - f(((TT + 1f((B(TTTTT (, (s)) D = If(e)) f(B(T, |s)) + 1B(T
(8) 02 0 a xc.
Now we need to show when म. - π -anndd ( + v., 3i (s) = vi. We have l0 - T S
and TH E ((T as + 0c, Iu - _[TT [T (T, (T -u By our assumption, ((TT," : 1²
Jog 3(m, [|s] 13(==="1 (s) -T
13(m,'' [s) (3(mm + 0. Thus, 10 - Tul + 0 as + oc. As (( = ( A.3
+ 0, [T s) - To + O. By Lemma
m min+ Vi (,<- (8) v 3(T (8)=
As we have 3(T 11) = ug. we have of Tog and is fixed point. Thus, of = ( = B( 10) (s). As consequence, n' E (( ((π is upper semicontinuous
min,' Vin ( ,,
correspondence.
Now, we have proofed ø is an upper semicontinuous correspondence (Lemma. A.4]. mapping ((A)
into the family of convex subsets of A(A) (Lemma. A.2). Since o( is an upper semicontinuous correspondence, it is also a closed set for any т. Thus, the result satisfies the requirement of Kakutani's fixed point theorem, with equilibrium policy ((
A.2.3 EXISTENCE OF EX INTERIM RMPBE
The existence of ex ante RMPBE follows the proof of ex interim RMPBE, but having prior belief p(0) that is never updated. Since the expectation over p(0) is linear combination and p(0) is bounded, the addition of p(0) do not violate the convergence and continuity of value functions. The proof thus follows the result of interim RMPBE.
Note that I Lemma of Kardey et al. 2011 considers cost, which is the negative of reward. While in their proof, the function considering cost convex. Taking the negative of convex function thus concave.
18

---

Published as a conference paper at ICLR 2024
A.3 PROOF OF PROPOSITION 3.2
For convenience of notations, Let us redefine the value function under worst-case adversary and type 0 at time but additionally adding n, into the notation of (s(s) for clarity, resulting in
Voth R.
min H(-
Σ pa (ala, 0) II क <' (H) +7 Σ P(s'ls av (8))). iEN
P(ss'
s'ES
We can then redefine the value function for ex ante RMPBE as V (0) pp( IV," 1V, the value function for interim RMPBE as V.T.a. (8) Ep(oLH) 1/
(s) and
Next, for each 0 E O, with time + oo, we have be = H() + 0 by consistency of Bayes' rule Diaconis & Freedman 1986), if 40 p,)()) and with finite type space O The resulting value
function at f - OC is thus:
2
(s):
where nt is always the optimal adversarial policy for current æ. The rest follows, As for the expected return of ex ante robustness for 0, we get:
(8) = Ep(e) V0
(5)
((0)
During evaluation, Vp(0), at t + oc, since current type 0 is known and do not change, the return consequence, the value function ex ante and (s) and V.r. ,a (s), respectively. At l oo and vO O,
conditions on 0, instead of expectations on
ex interim equilibrium is ya A*E. -
have be = Hz) + 0, and following inequality holds:
El EEI
2 Vo
(8),
which use the fact Vgg (s) a sufficient condition for this equality to hold is when the type space θ contains one type only. Note
Vg (s). Considering to the gap between V Vz
(s) and V &
p(ù) (s),
that even at t + oo, the relation between expected value function V BA 5 (s) of ex ante equilibrium
and V.g ,51 (s) of ex interim equilibrium is still not known. This is because the ex ante equilibrium can get high expected values by simply "believing" in some prior p(0) that yields high value.
However, since the belief is not correct, the resulting policy is non-optimal in any type at t + co. M
A.4 PROOF OF PROPOSITION 3.3
Overview. We first proof the contraction mapping of Q(,, b²) by combining standard proof of the contraction mapping of Q function and Bayesian belief update. Afterwards, applying Banach's fixed point theorem completes the proof. The convergence of Q(s, a, b¹) follows the same vein.
We first show the proof for Q-function for Q (s, a, b¹). The proof incorporates our robust Harsanyi- Bellman equation in the contraction mapping proof of robust MDP (Iyengar 2005). Based on Bellman equation in Definition 3.1 let A(O) be a probability over θ, the optimal Q-function of contraction operator T, defined from a generic function Q S x A A()) + R can be defined as:
(s, = max min R(s, a) + =((o.))) #(-lo.0)
P Σ p(0 Hi)
L'ES
BEB
Σ
A'
Next, we show T forms a contraction operator, such that for any two Q function Q and Q2 assuming TQ (s, a, b') 2 (s) b'), the following holds:
19

---

Published as a conference paper at ICLR 2024
ITQI
TQ2ll> <
221
Specifically, for e > O and with fixed a E A...ES,6e A(0), H (0 x A)" for Q1 and Q2.
min R(s, a) + #1-1H,0)
Σ P() a) L''ES
Σ (()
Σ K'EA
('00",,,,))((6 2 TTQ
BE0
Note that updating belief by Bayes' rule required for consistency of 6 We require fixed H as well since the calculation of belief and Q function depends on H', Now can also choose conditional policy measure of the adversary n. such that:
R(s, a) +7
Σ
() Σ
H')
Σ
S
L'ES
8E0
min R(s, a) + *(-10,0)
Σ P(b'ls,))
Σ ((O(H*)
Σ
7( H' b',' 0'' te.
LESS
AC0
R'EA
Then,
0 0 TQi - -TQ2
min (((s,a) +7
Σ s(
Σp(O(H)
(()HH Σ
H''ll''' (1 te)
((-1H,0)
LEESS
BEB
a'EAA
min R(s,ā) +7
Σ s(
Σ )(0
H,)H
L'ES
AEO
Ex,
R(s,a)+ +
Σ P(8'(8,
Σ
p(0
Σ
H
(6 +e)
LES
BE8
REA
Ex,
R(s, a) +7
Σ ((s'|s, Σ H') Σ H'
,, 83000 -c)
LESS
BE8
WEA
>E,, [Qi Q1 2c S yE,, |Q8 Q
2e S EE 1 1211oc 2c.
Thus, we have:
I[TQ' - To 5 - + 2c,
and since by definition, is arbitrary, then we have ||TQ; - TQ I - 1i
Finally, since T is a contraction operator on a Banach space, by Banach's 's fixed point theorem, updating Q' (s. a. b¹) by Bellman operator T will converge to the optimal value function (s b').
In the same way, the convergence of Q'(s, a, b²) is again done by robust Harsanyi-Bellmm equation, (s, a, b') ((-HHbb) ((-HH.0)
max min Evco p(0)H()) Evess ΣaEA P(s'|s, a, म., [M( a)
7 E''E H HH))" a', 8.)] Expanding the function in the same way above completes the proof.
20

---

Published as a conference paper at ICLR 2024
A.5 PROOF OF THEOREM 4.1
We first discuss the policy gradient with Th (a' b"):
Vo
=V psi
Σ
a
H,,, (s, (s,
LAEA
Σ (1 (11-0)mm (0 -0.0 ('(s.=,11)
qi
LEAA
Σ
(1 0') o (1
BEA
(1
8: [R(s,a)
REA
P(s'|s. s. a)
Σ
[ (s' T.
WES&EA
OE
ΣE SEA
(1 -
- (7(1 0')V
Tas (d')H' 8) +7 Σ ((51
s'ES
SE0
Pr(s ( , π
(1 - (o -
SEES 1=0
aeA
Considering V ((bb,, we have
V ci ((0) =V Φe ('(0 b¹)
ΣΣ
Pr(su - s,t,
(a .3). (,)
SES t=
REA
Σ (1s) Σ (1
8 V
SES
REA
n(s)
Σ n(s')
Va- b.) Q'
WES EvEST(s') n(s')
EES
#ES
ReA
Σ ((s')
Σ pT (s) - [d-0maldd11;0)- (,,
EES
SES
REA
(s)
(1 - 0' (a (1)
SES
REAA
Using the lor-derivative trick, we have:
V Vo' x
E1 (s) Σ (1 - 0') (6 ((,,
8E8
aEA
Q (s. a, 6*) Va b)
Σ P (s)
(1 0² (a
Te. (a H ( b)
KES
BEA
(n (6) (3(5s,,b6)) a
21

---

Published as a conference paper at ICLR 2024
The proof of Tz. (ā'|o', basically follows the same method.
V 33 (s, b²) =V Be
Σ
(ā
L
Σ LaEA
(1 ( +0- -0, ()))
Σ
0c
x 3 04 01V (',,,,,,)
3EA
Σ REA
0' V Si &e (a
(10 0 [)s,a)
P(s s. a
Σ ,. (' (5' 8666)
J'S
OE
Σ 0¹. V 0). Q' + ( ( b. 0) T 0² V.
SEA
Rs. (a" ,0) +r ΣΣ P(s'|s, a)
Σ
(() -ー o. (8' H U, 0) ((6,
e'ESa'EA
8E8
Pr(s π
(a u1 0) Q' (,)
SESS 1=0
aEA
Considering V J J ((b') and use the log-derivative trick same as above, we get:
V ()))
Σ
8
(a' (a'l 0) - (s,
x
p
gs
SES
TEA
A (a H 9) a V (a 0)
p (s)
8
(a' 0)
SES
RE
-B
B (s), (- (- 1 Eo In a (a' H 0) Q' bb)).
This completes the proof.
A.6 CONVERGENCE PROOF OF THEOREM 3.1
We proof this via stochastic approximation theory of Borkar (Borkar 1997 Borkar & Meyn 2000 Borkar| 2009), where the robust agent is quasi-static and the adversary is essentially equilibrated (Borkar & Meyn 2000). One notable difference is, since the type in our work was sampled from prior distribution ~ p(0), we take the expectation with respect to p(0) to follow the notation of stochastic approximation theory.
Proposition A.1 (Convergence). Theorem 3.1 in main paper converge robust Bayesian Markoy Perfect equilibrium a.s. if the following assumption holds.
Assumption A.1. Given step n, learning rate of म and मे as an)) and ((n) with ann), 3(n) e (0, 1). denote the probability of having an adversary as pd p(0 = 1), such that Vi E N, pm - 3(n)p' 0o, -0, (="((m)((1-'))+++((
< oo, a(n)(1pp')) 0.
s(m)p())
Note that the assumption slightly differs from standard stochastic approximation, since adversaries and robust agents are not uniformly explored.
A E O, (' a, 0) is Lipshitz continuous. As a corollary, Q' a, b²) = Epto QQ' (s. 0)] is Lipshitz continuous.
Assumption A.2. Vi E N
Assumption A.3. Let v(ss, a, 6) denote the number of visit to state s and action a under 8¹. Vs, a, O. (18, a, 0) xo.
22

---

Published as a conference paper at ICLR 2024
Assumption A.4. The enror in stochastic approximation (i.e., inaccuracy in critic value, environment noise, beliefetc.) constitutes martingale difference sequences with respect to the increasing a o-fields.
Assumption A.S. v0 E O a global asymptotically stable ex interim equilibrium (n.",, A E! exists.
Assumption A.6. v0 E @ sup, (1 | || || co.
proof. We can write the update rule of т and in their Ordinary Differential Equation (ODE) form:
Egi (000)00)) = -Ep -p(o)
[Tr] + ae((ss a))(1
p² BBii /V log of (R(s. Tn Tn H 8. 0)Q a a',
Σ (( a) Σ p(0
Σ
EE
BEO
REA
~ [ + B(( a)
Eg
Eggnp(o |V log _ [[ a)
(( a)
Σ p(0
Tn (a
b, 0) 0)0 (s, ('','''")))) ()))
O20
where assume (s', (²) is the learned critic, updated faster timescale than a(n) and ((n), following Assumption 1.1-1.5, such that Q' is essentially equilibrated. The error terms are embedded in Q (8', a', ). Thus, the update rule of x² and #² follows the general update rule of stochastic optimization (Borkar 2009):
In+1 =T, a(n))(((an Un) M
Un+1 yyn +a(n))g(2 vn) + M
Thus, by Theorem 4.1 in (Borkar & Meyn 2000) or Theorem 2 in (Borkar 2009). nn and म, converge
to equilibrium.
B ADDITIONAL DETAILS ON ALGORITHM
We implement algorithm top of MAPPO (Yu et al. 2021 MAPPO is widely used multi-agent extension of PPO and consistently achieves strong performance on many benchmark environments. Note that our method do not include algorithm-specifie structures, which means it can easily be applied to other actor-critic based algorithms, such as IPPO (de Witt et al., 2020), HATRPO (Kuba al.] (022), MAT (Wen et al.) 2022) etc. easily. However, while technically possible, we do not suggest a MADDPG implementation (Lowe et al. 2017) of our algorithm. This is because MADDPG provide deterministic output, but pure-strategy robust Markov perfect Perfect Bayesian equilibrium is not guaranteed to exist, as we have shown in Appedix. ?? by a counterexample.
One important thing to notice is that we empirically find using larger learning rate for adversaries during training of EIR-MAPPO do not always work well in all environments. For example, the learning dynamics of adversary in our toy environment can be unstable even with learning rate 5e 4, and get worse if the learning rate further increase. which is much smaller than the convergence rate suggested by previous papers (Daskalakis et al. 2020). We empirically find adversaries using slightly higher learning rates than robust agents works well, but requires extensive tuning. To tackle this problem, we maintain the central assumption of two-timescale updates (i.e., updating the adversary faster and the robust agent slower), but instead update the adversary for more rollouts (denoted by interval 1. in our algorithm), while update the victim for less rollouts. This do not violate our proof in Appendix A.6 since taking expectations to policy update brings the same result.
We closely follow the implementation details of MAPPO and PPO (Schulman et al. 2017). including parameter sharing, generalized advantage estimation (GAE) (Schulman al., 2015). and other tricks in the codebase of MAPPO, available at hntps::s///.cmm.m / com. / mar .benchmark/ oo-policy Note that we use fixed learning rate for both adversary and robust agents. Pe H") GRU (Chung et al., 2014) with input pe (b* ,o"),, where h; is the hidden state that summarize observations in previous timesteps and of observation of current timestep.
23

---

Published as a conference paper at ICLR 2024
Algorithm 1 interim robust c-MARL (EIR-MAPPO),
Input: Policy network of robust agents T.. adversary म a value function Vi, belief network Pę- Output: Trained policy network of robust agents Tá.
1: for k= 0, 1, 2, K do
2: Sample ~ p(0). Initialize
3: for 0, 1, 2, do
4:
Vi NN, perform rollout under bj pe (O) a. To(-|Hle,bb)) b,), â, # ( 0) and
St+1 P(st+ 1s,, ar, R), receive Hi41 and
5:
T +- r U (b,, a, ā, S141, H4t1, Tt, H1, 8).
6:
end for
7:
for 0, 1, ... do
8:
k % interval == then
9:
Using r, calculate A as, b²) by GAE; calculate b' by Pe-
10:
- Ø a Vogg H A') as, b'). // Shared parameters Φ - Øt a o (0)
11:
サ 인 + a V (r ('', Q
13:
+ { V log (pe (1 + (1 - log(1 pe (0
14:
else
15:
+ o² a. log (a H 8)A (s, b.). // Update adversaries for
more rollouts, with others fixed.
Empirically stabilize
training.
16:
end if
end for
18:
end for
C ADDITIONAL DETAILS ON EXPERIMENTS
C.I ENVIRONMENT DETAILS
a # a²
a' # a² == 1
S2
1
r
(a) Toy
(b) LBB
(c) SMAC
Figure 7: Environments used in experiments. The toy game is proposed by Han al. 2022 We use map 3m in SMAC (Samvelyan et al.) 2019) and map 12x12-4p-4f LBF Papoudakis et al. 2020).
In this section, we introduce more details environment. Again, we add the figure of environments Fig. Next, we introduce the tasks, actions and reward of each environments as follows.
Toy environment. The toy environment was first proposed by (Han et al., 2022) to study the effect of state-based attacks on c-MARL. In this game, two agents play simultaneously for 100 iterations to achieve maximum coordination. Specifically, in state 81, two agents seeks same actions (XNOR gate), while state sy seeks two agents seeks different actions (XOR gate). During attack, the adversary can take over each agent, and perform actions to maximally attack another agent. The attack requires the robust agent simultaneously identify other agents adversary allies, while taking cooperative actions other agent is an ally, and take randomized action other agent an adversary.
Level-Based Foraging environment. Level-Based Foraging environment (Papoudakis et al. 2020) aims set of agents cooperatively achieve maximum reward in food collection process. Each
24

---

Published as a conference paper at ICLR 2024
agents are assigned different "levels", while the food can only be collected via agents with level higher than the food. Note that we use the cooperative setting in LBF, which majority agents (except the adversary) have to collaborate jointly to achieve the goal. While there do not exist a commonly used testhed in LBF, we use 2f in our experiment.
We also need to notice that, in LBF environment, an adversary is capable of physically interfere with other agents, such as blocking the way of others or intentionally colliding with other agents, resulting in deadlock ever since. While Gleave al. (2019) cited an important aspect of adversarial policy as "not to physically interfere with others", we turn the collision in LBF off to represent this.
StarCraft Multi-Agent Challenge environment. StarCraft Multi-Agent Challenge environment (Samvelyan al.) (019) is the most commly used testbed for e-MARL, which agents control team of red agents and seeks to win a team of blue agents with built-in AIs. We adapt the map 3m proposed in SMAC testbed to 4m 3m, with one agent as adversary. Thus, the map can still be viewed as 3m, albeit with one adversary agents trying to fool its teammates. Note that the adversary cannot attack its allies by design of SMAC environment.
C.2 IMPLEMENTATION DETAILS
The implementation of MAPPO, RMAAC, EAR-MAPPO and EIR-MAPPO are based on the orig-
inal codebase of MAPPO (https: / huub. com/ wtt.ecommmmmm/aeeeeeeeee waw..beeccmarkoooooooooo pcyccy) The implementation of MADDPG and M3DDPG resembles the code of FACMAC (Peng et al.) 2021) wwwooowwiirrrr rll wwwtwhirr/facaaaae facmac) and feeeerogeneousAgeee
Reinforcement Learn- ing (HARL) codebase (https: /ttgttttttttttt tht.....hbu......g. com PKUU WARL/HARI, Our code are available in supplementary files, and will be open sourced after this paper is accepted.
com /
For all environments, we set p(0 0x) 0.5 and p(0 = 1.) 0.5/N, where 1, denotes the one-hot vector with 8 = and others 0. This probability of selecting p(0) remains fixed throughout training process. During training, we store the model of robust agents with improved robustness without decreasing cooperation reward, evaluated on the adversary during training. While testing, we held all parameters in robust agents fixed, including policy and belief network. Then, black-box adversary was trained following the approach of adversarial policy. The adversary also follows CTDE approach, assuming assess to state, reward and local observation during training, and local observation only in testing. For fair comparison, we attack all baselines by PPO (Schulman et al. 2017).
As for M3DDPG, note that the original version of M3DDPG (Li et al. 2019) are designed for continuous control only, where actions are continuous and can be perturbed by small value, while in discrete control, one will have to completely change the action, or not changing the action at all. To solve that, we add the noise perturbation to the action probability of MADDPG and send it to Q function instead. We also find using large for M3DDPG will make the policy impossible to converge in fully cooperative settings: since M3DDPG add perturbations directly to each agents, resulting in overly challenging setting. As such, we select the largest which enables maximum cooperation result in each setting.
As for RMAAC, the perturbation in training is set 0.5 following their original paper, except for c = 0.05 in toy environment since otherwise the policy will not converge in normal training.
Next, present all hyperparameters of each environment in the table below. These hyparameters follows the default in previous papers, including MAPPO (Yu et al. 202T). HARL (Zhong et al. 2023) and FACMAC (Peng al. 2021
25

---

Published as a conference paper at ICLR 2024
Table Hyperparameters for MAPPO, RMAAC, EAR-MAPPO, EIR-MAPPO toy environment.
Hyperparameter
Value
Hyperparameter mini-batee num
Value
Hyperparameter
Value
rollouts
10
1
PPO epoch
5
gamma
0.99
max grad norm
10
PPO clip
0.05
gain
0.01
max episode len
200
entropy coef
0.01
actor network
MLP
actor Ir critic Ir 5c-5
5e-5
eval episode
hidden dim
128
optimizer
Adam
belief network
GRU
adversary Ir
5e-5
Huber loss
True
use PopArt
True
belief Ir
5e-5
Huber delta
10
adversary interval
10
GAE lambda
0.95
RMAAC
0.05
Table 2: Hyperparameters for MADDPG and M3DDPG in toy environment.
Hyperparameter
Value
Hyperparameter mini-batch num
Value
H Hyperparameter
Value
rollouts
10
1
gamma
0.99
actor network
MLP
actor
5e-5
eval episode
32
hidden dim
256
critic Ir
5e-5
optimizer
Adam
buffer size
1000000
batch size
1000
epsilon
0.1
Table 3: Hyperparameters for the PPO adversary in toy environment.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
rollouts
10
mini-batch num
PPO epoch
5
gamma
0.99
max grad norm
10
PPO clip
0,05
gain
0.01
max episode len
200
entropy coef
0.01
actor network
MLP
adversary Ir
5e-5
eval episode
32
hidden dim
128
critic Ir
5e-5
optimizer
Adam
use PopArt
True 0.95
Huber loss
True
Huber delta
10
GAE lambda
Table Hyperparameters for MAPPO, RMAAC, EAR-MAPPO, EIR-MAPPO in SMAC environ- ment.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
rollouts
20
mini-batch num
PPO epoch
gamma
0.95
max grad norm
10
PPO clip
0.05
gain 0.01
max episode len
200
entropy coef
0.01
actor network
MLP
actor Ir
5e-4
eval episode
32
hidden dim
128
critic Ir
5e-4
optimizer
Adam
belief network
GRU
adversary Ir
5e-4
Huber loss
True
use PopArt
True
belief
5e-4
Huber delta
10
adversary interval
5
GAR lambda
0.95
RMAAC
0.05
Table 5: Hyperparameters for MADDPG and M3DDPG in SMAC environment.
Hyperparameter
Value
Hyperparameter mini-batch num
Value
Hyperparameter
Value
rollouts
20
1
gamma
0.99
actor network
MLP
actor Ir
5e-4
eval episode
32
hidden dim
256
critic Ir
5e-4
optimizer
Adam
buffer size
1000000
batch size
1000
epsilon
0.01
26

---

Published as a conference paper at ICLR 2024
Table Hyperparameters for the PPO adversary in SMAC environment.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
rollouts
20
mini-batch num
1
PPO epoch
5
gamma
0.99
max grad norm
10
PPO clip
0.05
gain
0.01
max episode len
200
entropy coef
0.01
actor network
MLP
adversary Ir
5e-4
eval episode
32
hidden dim
128
critic Ir
5c-4
optimizer
Adam
use PopArt
True
Huber loss
True
Huber delta
GAE lambda
0.95
Table Hyperparameters for MAPPO, RMAAC, EAR-MAPPO, EIR-MAPPO in LBF environment.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
rollouts
20
mini-batch num
1
PPO epoch
5
gamma
0.99
max grad norm
10
PPO clip
0.05
gain
0.01
max episode len
200
entropy coef
0.01
actor network
MLP
actor Ir
5e-4
eval episode
32
hidden dim
128
critic Ir
5e-4
optimizer
Adam
belief network
GRU
adversary Ir
5e-4
Huber loss
True
use PopArt
True
belief Ir
5e-4
Huber delta
10
adversary interval
5
GAE lambda
0.95
RMAAC
0,05
Table Hyperparameters for MADDPG and M3DDPG in LBF environment.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
rollouts
20
mini-bated Mini-batch num
1
gamma
0.99
network
MLP
actor Ir
5e-5
eval episode
32
hidden dim
256
critic Ir
5e-5
optimizer Adam
buffer size
1000000
batch size
1000
epsilon
0.1
Table 9: Hyperparameters for the PPO adversary in LBF environment.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
rollouts
20
mini-batch num
PPO epoch
gamma
0,99
max grad norm
10
PPO clip
0.05
gain
0.01
max episode len
200
entropy coef
0.01
actor network
MLP
adversary Ir
5e-4
eval episode
32
hidden dim
128
critic Ir
5e-4
optimizer
Adam
use PopArt
True
Huber loss
True
Huber delta
10
GAE lambda
0.95
27


Reviews:
[{"review_id": "8Z5SZbUvjd", "reviewer": "Reviewer_FinH", "summary": "The research explores the stability of cooperative multi-agent reinforcement learning when faced with adversarial strategies and disruptions. It introduces the Bayesian Adversarial Robust Decentralized Partially Observable Markov Decision Process (BARDec-POMDP) as a framework for addressing these challenges. The study demonstrates that conventional robust cooperative MARL approaches only achieve an ex interim equilibrium and advances the concept of an ex post equilibrium for BARDec-POMDP, which it argues is superior under worst-case scenarios. To attain this ex post equilibrium, the study presents a novel actor-critic method named EPR-MAPPO. Testing on simple games, the LBF environment, and the StarCraft Multi-Agent Challenge (SMAC) illustrates that EPR-MAPPO outperforms standard models, showcasing resilience against four distinct threat categories.", "questions": "How can the proposed method benefit real-world applications? Any concrete examples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript is easy to follow. Addressing the robustness of Multi-Agent Reinforcement Learning (MARL) is crucial, particularly for its practical deployment in real-life situations. The introduction of the ex post equilibrium within the BARDec-POMDP framework by the authors offers a thoughtful contribution to the field. The paper includes an in-depth examination of the algorithms it introduces. Furthermore, the empirical findings presented are coherent and compelling.", "weaknesses": "The document frequently refers to its appendix, suggesting that it does not stand alone as effectively as it could. To enhance the feeling of a complete and self-contained work, the authors should further take simplification, such as using simpler terms within the equations for greater clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rz0BidXag7", "reviewer": "Reviewer_EssV", "summary": "The paper addresses the robustness of multi-agent reinforcement learning against Byzantine failures. A Bayesian Adversarial Robust\nDec-POMDP (BARDec-POMDP) framework is proposed.  The theoretical formulation of the problem is one of the contributions of the\npaper and an actor-critic algorithm that produces convergence under some conditions.  Convergence is not ensured but experimentally\nthe approach shows great resilience against a spectrum of adversaries.\n\nThe paper makes some strong assumptions, like the fact that in each episode there is only one attacker. Other assumptions are consistency and sequential rationality, which are used to form an ex interim robust Markov perfect Bayesian equilibrium (RMPBE). With some additional assumptions, the existence of ex ante and ex interim RMPBE are guaranteed. For the ex interim equilibrium, the Q function can be written using the Bellman-type equation for the two Q functions, which the paper calls \"the robust Harsanyi-Bellman equation.\" With a few more assumptions, updating the value function via the Bayes rule guarantees convergence to the optimal value of Q.\n\nThe algorithm proposed is applied to different problems, a toy metrics game and some map problems. For the attacks, four types of threats are considered: non-oblivious adversaries, random agents, noisy observations, and transferred adversaries. The experimental results cover robustness over non-oblivious attacks and various types of attacks.", "questions": "If you were to write a paper that fits in the page limits, would you include some of the material from the Appendices and drop some of the material currently in the paper or leave the paper unchanged?  I am wondering what you think are the most important parts needed to present the work without exceeding the page limits.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The novelty of the paper is in the theoretical formulation of the problem and in the actor-critic algorithm that produces convergence\nunder some conditions.  Convergence is not ensured but experimentally the approach shows great resilience against a spectrum of adversaries.", "weaknesses": "- The fact that convergence cannot be guaranteed despite the assumptions made is a weakness of the paper. The method proposed is complex, and the assumptions are strong, yet there is no guarantee of convergence\n- The proofs are all in the Appendix, which makes the paper quite long, too long for a conference paper, and that makes it appear incomplete.  Even the analysis of two of the environments used for the experiments is in the Appendix. Expecting the reviewers to read a paper of 36 pages is too much for a conference.  Without the Appendices the paper is incomplete.\n- The paper is hard to read, it assumes significant knowledge of the field and is full of acronyms and citations that break the flow of the text.\n\nAfter the discussion and the changes made to the paper, I reduced my criticism about convergence and increased the rating I gave for contribution. However, the presentation needs more work.  Many of the corrections made to the paper have grammar errors, like mismatches of singular/plural. I understand the rush of making the changes, but the writing is not where it should be.  I also still object to the length of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sTB1o0eY5X", "reviewer": "Reviewer_UzHb", "summary": "The paper addresses Byzantine failures in multi-agent reinforcement learning (MARL), where agents can adopt malicious behavior due to malfunction caused by hardware/software faults. First, a new problem setting, called Bayesian Adversarial Robust Dec-POMDP (BARDec-POMDP), is proposed to model adversarial intervention as a transition, where originally cooperative joint actions are modified before being executed in the actual environment. The setting is inspired by Bayesian games, where agents have types that need to be inferred by other agents through their beliefs. The goal is to learn a more fine-grained solution than prior work, where policies are able to collaborate with functional agents, while being robust against adversaries therefore finding a better cooperation-robustness trade-off. Ex post mixed-strategy robust Bayesian Markov perfect equilibrium is proposed as a solution concept for BARDec-POMDPs, which is claimed to be weakly dominant over equilibria that were pursued in previous works.\n\nBased on this setting, a two-timescale actor-critic is proposed based on the robust Harsanyi-Bellman equation, defining the maximin value according to the ideal case (before any adversarial modification) and the adversarial case. The policy gradient uses the value of the adversarial case as critic. The approach is evaluated in a variety of games, including a matrix game, a gridworld game, and a map of SMAC. It is also evaluated against different attacker types.", "questions": "1. Why do adversaries condition on observations despite them not being Markovian (in contrast to the non-adversarial policies)? Is this intentional or a mistake?\n2. Do you have any intuition on why M3DDPG performs so poorly in the presence of an adversary compared to the standard MAPPO without any robustness mechanism in Figure 4?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses a very important problem, which is often neglected in the MARL community.\n\nThe proposed setting (BARDec-POMDP) can model realistic scenarios, where agents can behave adversarially due to malfunctioning.\n\nWhile most works on robust MARL focus on defenses against any kind of adversarial attack, which leads to conservative policies, the paper aims to find a trade-off between collaborating with actual functional agents and robustness against actual adversaries, which is a more desirable goal for real-world applications. Bayesian games provide a neat framework to discern between these types of agents, which is a convincing proposal.\n\nThe solution concept seems valid.\n\nI like the evaluation, which first starts with a small toy problem before scaling up to larger benchmarks like SMAC. I also appreciate the evaluation with different adversarial strategies.", "weaknesses": "My main concern with the paper is the lack of self-containment: The paper excessively references the appendix, which hurts readability (it is not convenient to peek into the appendix for almost every paragraph of the main paper).\n\nThe paper needs a clear prioritization of what is really important and what can be safely left in the appendix. For example, important proofs can be sketched informally with little space, but delegating them to the appendix completely is not acceptable.\n\nBefore checking, if I can raise my score, I need a revised version of the paper that is more self-contained, e.g., where the number of appendix references is significantly reduced and proofs are at least sketched such that the reader is not forced to switch documents all the time.\n\n**Minor Comments**\n- In Dec-POMDPs (Section 2.1), policies select their actions by the history of past observations and actions. Conditioning only on observations is insufficient since they are not Markovian. The definition in Section 2.2 is correct, though.\n- Line 119: “was replaced” - “**is** replaced”\n- Line 277: “four type of threats” - “four type**s** of threats”\n- The text size in Figure 2 is too small for printed versions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)