Bayesian Robust Cooperative Multi-Agent Reinforcement Learning Against Unknown Adversaries
Authors: Kiarash Kazari
György Dán
Venue: ICLR 2026 Poster (2026)
Keywords: multi-agent reinforcement learning
adversarial attacks
Bayesian games
robust RL
Primary area: reinforcement learning
TL;DR: A Bayesian approach for coping with unseen attacks in c-MARL

Abstract: We consider the problem of robustness against adversarial attacks in cooperative multi-agent reinforcement learning (c-MARL) at deployment time, where agents can face an adversary with an unknown objective. We address the uncertainty about the adversarial objective by proposing a Bayesian Dec-POMDP game model with a continuum of adversarial types, corresponding to distinct attack objectives. To compute a perfect Bayesian equilibrium (PBE) of the game, we introduce a novel partitioning scheme of  adversarial policies based on their performance against a reference c-MARL policy. This allows us to cast the problem as finding a PBE in a finite-type Bayesian game. To compute the adversarial policies, we introduce the concept of an externally constrained reinforcement learning problem and present a provably convergent algorithm for solving it. Building on this, we propose to use a simultaneous gradient update scheme to obtain robust Bayesian c-MARL policies. Experiments on diverse benchmarks show that our approach, called BATPAL, outperforms state-of-the-art baselines under a wide variety of attack strategies, highlighting its robustness and adaptiveness.

Published as a conference paper at ICLR 2026
BAYESIAN ROBUST COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING AGAINST UNKNOWN
ADVERSARIES
Kiarash Kazari & György Dán
KTH Royal Institute of Technology, Stockholm, Sweden (kkazari, gyuriyyrrr seuurijekthss.
ABSTRACT
We consider the problem of robustness against adversarial attacks in cooperative multi-agent reinforcement learning (c-MARL) at deployment time, where agents can face an adversary with an unknown objective. We address the uncertainty about the adversarial objective by proposing Bayesian Dec-POMDP game model with a continuum of adversarial types, corresponding to distinct attack objectives. To compute a perfect Bayesian equilibrium (PBE) of the game, we introduce a novel partitioning scheme of adversarial policies based on their performance against reference c-MARL policy. This allows to cast the problem finding a PBE in a finite-type Bayesian game. To compute the adversarial policies, we in- troduce the concept of externlly constrained reinforcement learning problem and present a provably convergent algorithm for solving it. Building on this, we propose use simultaneous gradient update scheme to obtain robust Bayesian c-MARL policies. Experiments on diverse benchmarks show that our approach, called BATPAL, outperforms otate-of-the-arr baselines under a wide variety of at- tack strategies, highlighting its robustness and adaptiveness.
INTRODUCTION
Cooperative multi-agent reinforcement learning (c-MARL) has achieved remarkable performance in areas such autonomous driving, 5G networks, robotics, and smart grids (Canese et al.] [021). as it allows agents to learn distributed policies for complex sequential tasks. Nonetheless, the failure or the compromise of even a single agent, either through direct manipulation of its actions or by corrupting its observations, can degrade the overall team performance (Lin et al. 2020), calling for policies that are robust against faults and adversarial attacks.
Existing approaches for obtaining robust policies rely on dataset augmentation or on adversarial
training (Gleave et al., 2019 Pattanaik et al. 2017: Havens al.) 2018 Pinto et al. 2017 Phan
et al. 2021 Liu et al. 2024a] Li et al.) (024). Dataset augmentation involves introducing one
or more adversarial perturbations during training, allowing agents to learn under adversarial and
nominal conditions simultaneously (Gleave et al. 2019 Pattanaik et al. 2017 Havens et al. 2018).
The alternative approach is based on jointly training the benign and the adversarial agents, typically formulated as zero-sum Stackelberg game, and saddle-point equilibrium in policies is sought after (Pinto et al. 2017 Phan et al. 2021] Liu et al., 2024a). These approaches typically yield a single policy optimized for adversarial conditions and thus they typically suboptimal when all agents are cooperative. Even if the trained policy can maintain a belief about the presence of an adversary, in Li al. (2024), robust learning based on saddle-point equilibria against worst case adversary found using gradient descent has three fundamental limitations.
First, it relies the assumption of a worst case adversary, which fails capture adversaries with objective other than minimizing the team reward as well as non-cooperative behavior due to failure. These may deviate substantially from worst-case attacks (Liu al. 2024b Kokolakis et al. 2020). and thus the defender's 's max-min policy may be far from optimal considering the actual adversarial strategy, resulting in poor team performance.
"Code available http://gittbccomkiisssaaBBB...B

---

Published as a conference paper at ICLR 2026
Second, the optimization problem solved is inherently non-convex, and learning algorithms prone to converge to local stationary points, which may not be globally optimal (Kalogiannis et al.) 2022 Fiez et al. 2020; Reddi et al. 2024). As a result, the saddle-pond point policies are local Stackelberg equilibria (Loftin et al. 2024), potentially far from the equilibrium sought after.
Third, exposure perturbed versions of single adversarial policy during training can cause the agents' representation of adversarial dynamics to overfit. Consequently, when faced with a different type of adversarial behavior at deployment, the agents may fail to adapt their policies to the previ- ously learned max-min strategy (Liu et al. 2024a). In such cases, they may not even achieve the minimum performance guarantee that the max min strategy theoretically expected to provide.
To address these limitations, we introduce a novel approach for training robust MARL policies that can adapt to diverse set of adversarial behaviors. Instead of learning a single max-min policy, our approach partitions the set of adversarial policies into disjoint subsets, defined by the range of team reward they would impose, and then computes a policy that is robust to a representative adversarial policy from each subset. Although our approach cannot completely eliminate the problem of lo- cal stationary points described above, it mitigates the problem by restricting the search to smaller, isolated feasible sets. Moreover, the subsets are constructed so that adversarial policies in different subsets exhibit distinct behaviors. The defender's MARL policy is then trained to adapt based on its follows.
belief of adversarial behavior. Our main contributions
(1) We introduce a Bayesian Dec-POMDP game model with a continuum of adversarial types and propose novel criterion for discretizing the type space ensure exposure diverse set of adver- sarial policies during training. Based on the perfect Bayesian equilibrium of the game, we formulate the Bayesian regret as the objective to characterize the robustness of policy.
(2) To compute an equilibrium, we introduce the concept of an externally-constrrinnee RL to find the adversarial policies of different types. We propose both a provably convergent algorithm and a practically efficient variant to solve this problem. Building on these, we design an end-to-end adversarial learning framework, termed BATPAL, derive Bayesian robust c-MARL policies.
(3) Through extensive simulations, we demonstrate the effectiveness of BATPAL in adapting to unseen adversarial policies across various benchmark MARL environments, and show that it con- sistently outperforms state-of-the-ane robust MARL algorithms.
interaction modeled as a game, and the agents seek max-min policy for execution temtttioo-timeeeee robustness. RARL (Pinto et al.) (017) and RARAL (Pan al. 2019) focus on adversarial disturbances with alternating optimization, while Tessler al. (019) and RAP (Vinitsky et al.] (0200) study adversarial manipulation of actions. Although effective against worst-case attacks, such approaches can be overly conservative; recent work (Liu et al.) 2024b) addresses this by considering non-worst-case adversaries, but in a lifelong learning context.
Related Work: In robust learning the
adopts a max-min value func- tion. while RAT (Phan et al. 2020) and RADAR (Phan et al. 2021 consider environments with subset of adversarial agents. Yuan et al. (2023) and Lee et al. 2025) model budget-limited attacks, and Liu et al. (2024a) studies adjustable, non-worst-case adversaries in two-agent scenarios. Most recently, Li et al. (2024) propose to maintain belief states about what teammates are compromised, but considers a worst case adversary only, leaving agents undefended against unseen adversaries.
For execution-time robustness in MARL, M3DDPG (Li et al.
2019)
MODEL AND PROBLEM FORMULATION
2.1 C-MARL MODEL
We consider a Dec-POMDP M (W,S, (A') R, P, (.N'),,,,,,,. (,,,7), 7), where N {1, 2I., N} is the set of agents, is the set of states, and A' and O¹ are the set of actions and the set of observations of agent i, respectively. We assume that S and A' are finite sets. Furthermore, R(s,, a.). P(s $t, a.), and O(o, s1) denote the reward function, the state transition probability, and the conditional observation probabilities, respectively. Finally, a and y 1 denote the ini- tial state distribution and the discount factor, respectively. We denote the history of observations, rewards and own actions of agent i up time by T/-
2

---

Published as a conference paper at ICLR 2026
We assume that the reward is bounded such that, without loss of generality, [R(s, a)l < ((s, a) x A. The value function when the agents follow a joint policy श = (Λ ..... ...,.T)) V) is defined EI 2 R(s,, ar) So s]. We define the expected initial state value as V E V (so)]. Throughout the paper, we use the game-theoretic notation x to denote the collection of x for all agents o i, where can be actions, observations, any other quantity.
as: (s)
2.2 BAYESIAN DEC-POMDP AS A MODEL OF ADVERSARIAL ROBUSTNESS
During deployment, agents may deviate from their pre-trained policies due to hardware or software emoor and due adversarial activity (Lin et al. 2020 Kazari et al. 2023 2025). The identity and the objective of non-cooperative agents is, however, unknown to the cooperative agents. Yet, most of the literature on robust single agent and multi agent RL focuses on worst case adversaries, i.c., one that minimizes the team reward (Gleave et al.) 2019 Tessler et al. 2019 Li et al. 2019)22024 Only a few recent works considered robustness e.g., assuming the adversary may not fully control the victim (Liu et al. 2024a). via population of adversaries (Vinitsky et al. 2020), or via repeated encounters in a bandit setting (Liu et al. 2024b).
non-worst case adversarie in single agent setting,
To capture diversity of adversarial agents' objectives and the resulting uncertainty, we propose the Bayesian Dec-POMDP defined as
MB = (W, S, (O")eN; (A')ie,, R, (2') JEN, 1,,,7),
where O' is the type space of agent i, extending the Dec-POMDP formulation. The type O' cap- tures the uncertainty about the reward function of agent i, and a subset of || in general. Nonetheless, every compact subset of Euclidean space is equinumerous to [0, 1]. sider O' = [0, 1].
The type 0¹ O' of agent is drawn at the beginning of each episode. We denote by by the agents' prior about the types and the initial system state, obtained hased on μ.. The type 0 -0 corresponds to agent aiming to maximize the team reward, each 0 > O corresponds to agent that aims to maximize some other reward function. For notational convenience, we use 0i = as the type of an adversarial agent that aims to minimize the team reward. The policy n' (ailT]: 0³) of agent is thus a function of its type 8', and the joint action of the agents has distribution and it governs the state transitions. Importantly, even if states are fully observable, the policies of the agents need not be stationary due to incomplete information; they depend on their beliefs b' (7/) about the types of the other agents and their policies, maintained based on the observation history.
~ IT (aj)
(a] mf 0*)
Our Bayesian Dec-POMDP formulation generalizes existing formulations in the literature (Li et al.) 2024| 2019 Yuan et al. 2023). A Dec-POMDP with only cooperative agents corresponds to On. mixed cooperative-compeiiii
settings with NA victim agents correspond to ||0||o Na Li
ct al. 2024), and a single worst case adversary corresponds to |Ollu = ||O||1
2.3 THREAT MODEL AND PROBLEM STATEMENT
Aligned with the game model MB we consider that the identity of the victims and the adversarial objective are unknown to benign agents, and the types of the agents do not change during an episode. For case of notation we consider that the adversary takes control of single victim agent o E N, and we denote the adversarial policy by p" (-1r"" "t//"..00 ("). Game MM is Bayesian game with imperfect information, its solution thus a perfect Bayesian equilibrium (PBE), defined as follows.
Definition 2.1. A perfect Bayesian equilibrium (PBE) is a profile of cooperative policies (T Jee and of adversarial policies (0 "e,,,,,,N... e,,... and a belief system (b' (rT))) and (b"(r", 8") D'EN,, that satisfies (i) each policy is optimal in expectation at every history given the beliefs (sequential rationality) (ūi beliefs are updated using Bayes rule based on the equilibrium policies for on-path histories, as well as for off- histories whenever possible.
To evaluate agent policies, with a slight abuse of notation, let us denote the expected initial state value when non-victim agents follow the joint policy л - and the victim follows policy pa.0- by v Intuitively, for any victim agent v and policy p".0" the non-victim agents should perform optimally, i.e., as close as possible the optimal policy against p" p=.8", We can thus evaluate the
3

---

Published as a conference paper at ICLR 2026
policies in terms of the Bayesian regret defined
R( (0,,0)) p (T)] = )- fmax( V p0.0 vmpp""
(I)
"
where expectation is taken over the prior by- Observe that a PBE minimizes by definition, and our objective is to learn such equilibrium policy profile (T")ee..
3 BAYESIAN TYPE-PARTIIIOND ADVERSARIAL LEARNING (BATPAL)
Ideally, the defender would learn a policy that minimizes the Bayesian regret. However, since the attacker can choose from (possibly infinitely) many adversarial policies, finding policy that corre- sponds to a PBE is computationally infeasible. To overcome this issue, we propose novel approach that partitions the adversarial type space into finite number of subsets, resulting in Bayesian Dec- POMDP Mg with type space é' {0, 1, ...., K} for agent i. Assuming most one victim agent, the support of p the set of all 0 such that ||0||o S 1. This set can be equivalently represented by
Z = {(v, k) v E N k E 21. K}} (0),
(2)
where 0 represents the non-adversarial type for all agents. We denote by p(0) the common prior over 0 = (0) ; .... 0% With a slight abuse of notation, we use both (0())=) and b((2(r)) for E E
to denote the beliefs.
While partitioning itself is conceptually simple, it is not straightforward how to map adversarial types O' to θτ and how to choose adversarial policies that would be representative for each discrete adversarial type 0'_ Our proposed solution is partition adversarial types based their severity, defined appropriately, and to use the worst-case policy in each partition as representative. The core idea then train a single policy that performs optimally against the worst-case adversarial policies in all partitions, i.e., PBE of game MB. This approach allows us to explore rich set of adversarial policies during training, which is essential for obtaining a PBE policy profile.
3.1 REFERENCE- VALUE BASED PARTITIONING
The main issue in partitioning the type space is that the reward function for each type θ" is private to agent and other agent cannot know it. If the other agents were to distinguish between two different types O", they only could it by playing against these two types with a fixed policy and observing the rewards they get. This motivates us to define our partitioning based on how well different adversarial types perform against a reference baseline policy.
Let To arg max, V be cooperative policy profile. We refer to wo as the reference policy and denote Vmax = mo Given a victim agent v of type 0", let us denote the minimum expected initial state value by Vie minp" / Note that Ve is the lowest value an adversarial policy played by v can impose while the other players use Th. Thus, under the reference policy of the non-victim agents, the expected initial state value induced by an attack on lies in [V min: Vmna]. We can thus define the severity of an adversarial policy p" as
V Vmax V
7
(3)
V Vmaa
Ve
The severity of every adversarial policy satisfies n,- [0, 1] by definition. Essentially, assuming that the adversary plays optimally with respect to its private reward function, provides mapping from O [o, 1]. by which the type -overaarrsarral) remains unchanged.
We use the above to partition adversarial policies according to their severity and the victim agent. A policy p" belongs to adversarial type : = (v. k)) if n,- (_x 1 k. and we denote the set of all such policies by IT.. Note that any adversarial policy belongs to exactly one of the sets II for E 2. The following proposition shows that such a partitioning is possible.
Proposition 3.1. If states are observable, then II, is a nonempty set for all 2 E Z.
Given this partitioning, the discrete adversarial types 0" correspond to sets II,, and it can be shown that the PBE in MB corresponds a policy म² (T*1 ..... x N such that
x (.(T, 8¹ = 0) E arg max Ex(())))) min V "0 W Vi p
(4)
x
4

---

Published as a conference paper at ICLR 2026
That is, each adversarial policy is worst case policy in the corresponding partition, and x mini- mizes the Bayesian regret (see Appendix B.I for proof).
prooff).
Before presenting our solution to we first claborate on how such categorization enhances the robustness of MARL compared to learning a single max-min policy. First, as empirically showed by Kazari et al. 2023), , there is general trade-off between the impact of an attack and the abnormality of the victim agent's 's behavior as perceived by non-victim agents. Here, the abnormality refers to a difference between what the non-victim agents expect to observe based on the reference policy and what the victim actually does. Thus, one would expect that if two adversarial policies have a large difference in VTo:"", and accordingly belong to very distinct severity levels, their behavior would be easy to distinguish from the non-victim agents' perspective. This would help MARL training to encounter diverse set of adversarial policies. The next two propositions provide theoretical support for this reasoning. Proposition 3.2 establishes a bound on the KI. divergence between two arbitrary policies in terms of their reference initial state values. The KI. divergence quantifies the discrepancy between two probability distributions and is commonly employed as metric for evaluating policy
diversity in regularized reinforcement learning tasks (Yuan et al. 2023 Derek & Isola 2021).
Proposition 3.2. Consider victim agent and any two adversarial policies p²:0: and po.02. If states are observable then we have
71-7)² and 20 [DLL 9 (s) 2 (1
2 | mo Mo
(5)
where doopp is the discounted state visitation distribution under (p", To
Following Cohen et al. (2019). we define the diversity of a set of policies (p 3 K
Jk== as
Div( 02 Jk=1) K(K-T) Eizy [E, (s) (s))]. Proposition 3.3 provides lower
bound on the diversity of adversarial policies generated by our partitioning through solving 4
Proposition 3.3. If p denotes the best response of adversarial rype ls against TO in (4) then
to (REL) (1 (1-7)2(Vmar Vmmin P 2)
Div(La
(6)
12K
Recall that one of the issues with learning a single max-min policy over the entire set of adversarial policies is its sub-optimality when the c-MARL team faces an arbitrary non-worst case attack. The next proposition demonstrates how the proposed partitioning mitigates this issue.
Proposition 3.4. Let g" E II be an arbitrary adversarial policy for some 2 = (v. k) and n e
arg max, min V=.p", Then, assuming fully observable states, we have
k<(mmax
V
RB (TT') S
K
The proof is provided in Appendix C.4 To interpret this result, let us compare the case of with K > 1. Note the case with K is equivalent to learning max-min policy over the set of all adversarial policies with v as the victim. When K = 1, the bound on the regret for any arbitrary adversarial policy can get large Vmax V In contrast, when K 1, gives severity-depender bound. In particular, for attacks belonging to lower severity levels, i.e., small k; the optimality gap becomes smaller, as the ratio & becomes smaller.
Finally, although the issue of getting stuck in local optima remains, our partitioning- based approach improves the likelihood of finding better solutions by restricting the search collection of smaller, non-overlapping feasible subsets that together cover the entire feasible space of adversarial policies.
4
ROBUST LEARNING
4.1 LEARNING ADVERSARIAL POLICIES VIA EXTERNALLY CONSTRAINED RL
To solve first focus on solving the inner minimization problem, i.e., for given (v. k), non-victim policy and reference policy No. find a policy ~² arg min,"en, V=,"" Since throughout the subsection we focus on a single adversarial policy, we drop superscript e for nota- tional simplicity. Observe that from the adversary's perspective, fixing the non-victim policies to any
5

---

Published as a conference paper at ICLR 2026
given policy induces a specific POMDP (see 21 and (22 in the Appendix for the corresponding reward and transition kernel). Thus, the inner minimization a constrained problem of the form min IE E P
s.t. E.- (b) h,
(8)
where (() and (0) denote the initial state value function of policy p when applied to two dif- ferent POMDPs, namely POMDP, and POMDPo- respectively, and I and h are some real numbers. Here, p o and POMDP, refer to POMDPs induced by fixing To and æ, respectively, and and h are the boundaries of severity type k. For notational simplicity. in the rest of the subsection we consider that states are observable, hence we refer to these MDP1 and MDP
Observe that MDP; and MDP, share the same action and state spaces, but differ in the reward func- tion and the transition dynamics. Thus, although problem (8) resembles the constrained RL problem in the context of safe learning (Paternain et al. 2019 Liu et al. 2020), there is a fundamental dif- ference. In constrained RL, the costs that define the constraints are essentially obtained through the same trajectory as the rewards in the objective function. On the contrary, in our problem, the objective and the constraints correspond to different MDPs, and consequently, different trajectories. To highlight this difference, henceforth we refer to (8) as the externally constrained RL problem.
We propose to use the log barrier method to approximate
(8) via an unconstrained problem. That is, define Vi = Eewu G) (s)] for 0, 1, and obtain
min
V log(VV - 1) - Algg(h - (0)
where À is hyperparameter controlling the -caasbbiyyy trade off. We propose to solve (9) using a gradient descent approach on policy Pe parametrized by parameter vector v/. The gradient of the objective function can then be expressed as follows. Proposition 4.1. The policy gradient of the objective function is
De 7 Benday (11o) VV ve log P (als) A (8)
}B. Bendoys ampo(is) log (als)) (s, a)]
ạ Egny [6j (s) -1 h - E.wg V80 (s)
(10)
where Ag) and d(y) denote the advantage function and the discounted state visitation distribution under pi corresponding to MDP,, respectively. (Proof in Appendix C.5
Then, the stochastic update rule for the policy parameters would be
Un+1 = thn anhen.
(11)
where R.. is an estimate of gv. and or, the learning rate. To estimate the gradient Pv. let V. and
(G)
Vi denote some unbiased estimators of vPen and V VG) respectively. where j = 0, Then, (i)
()
our proposed estimate is
Iv.
V (1)
A
v (0)
(12)
vun (0)
h V.un (0)
To obtain the estimates, we collect trajectories of M episodes in the form (St,m (t),,()) Tr.m.(j))?=o" 1 /== by executing Pv on MDP, for E {0,1). In prac-
tice, Tm could be the time to reach terminal state or the episodic time limit. We propose to maintain two parametrized functions, namely Voran and Vony, as the critics to estimate V (s)
and Vll (s). Then, V is obtained in the same way as standard actor-critic algorithm Sutton (f) & Barto 2018), using the empirical average of V log Pe (als) (8,a), where A is the advantage function calculated based on Voo). Moreover, we can obtain Vrvn as
M
Σ
Σ 0=
Y' TTrem, + Volon (8 (.Tm.
(13)
M

---

Published as a conference paper at ICLR 2026
A major difference between the proposed stochastic update and standard policy gradient methods is that our estimate of the gradient is not unbiased, as E( J # Iv., even if Vit estimators. Thus, our algorithm is not guaranteed converge using standard arguments (Robbins & Monro 1951). Yet, it does converge with a proper selection of the step sizes, as we show next.
(0) and Vh are unbiased
(j)
Proposition 4.2. Assume that parameterization bb, the following conditions hold:
(1) (0) and V are unbiased estimators.(2) For any E S and E A, the function log Pa(als) is twice differentiable with respect to v/, and both its first and second derivatives are bounded. (3) There exists a strictly feasible starting point tho, i.e., 1 < Vili < h. (4) There exists constant
VV Ç > O, such that V VPCn is nonzero when h - h -V S h or S (0) S1+5.
Then, for any e, > O, there exists sequence of adaptive step sizes {an) and some values Niter and M, such that after Niter iterations of (77) using 122 we have millns Neter l19 |l S e with probability at least 1-8. Moreover, as X + 0 the obtained point approaches a KKT point of the
constrained problem g with probability at least 1-ô. (Proof and detailed expressions of an Nater
and M available in Appendix C.6
Despite the above convergence result, using 12 poses two practical challenges. First, computing the adaptive step size o, is computationally expensive and requires estimating bounds on the gradient of log Pv. which is difficult in general. Second, estimating log-barrier gradients near the boundary of the feasible region is sensitive to noise (Usmanova et al. 2024). Mitigating the sensitivity requires large number of episodic samples M, which infeasible in practical RL settings.
To address these issues, we propose to incorporate the PPO loss function (Schulman et al. 2017) into the policy updates. The key intuition is that the clipping mechanism in PPO constrains policy updates by preventing large deviations from the current policy. This implicitly mitigates the risk of crossing into the infeasible region due to high-variance gradient estimates, while also eliminating the need to compute adaptive step sizes in practice. Moreover, when the initial policy lies outside the feasible region, the update direction is reversed such that the gradient step encourages convergence toward the feasible set. Then the gradient calculation of our proposed algorithm, which we refer to as externally-conse constraieed PPO (EC-PPO), can be summarized as
PPO (1)
X( -I h (0)
(0) if I++S V. (0)
<h-
EC-PPO
(14)
Vign
11 h)) (0):
otherwise,
where PPO. is the gradient of the PPO objective function (Eq.(7) in Schulman et al. 2017 ) and (1)
<> 0 is small value to prevent gradient explosion.
4.2 BAYESIAN ADVERSARIAL MARL TRAINING
To find the perfect Bayesian equilibrium policies in (4) recall that PBE policies are optimal in expec- tation given the beliefs. Thus, we incorporate b as an input to म represented as x <'-17", ; bi, 0 0).
can be shown that MB B is equivalent to a partially observable stochastic game g with N + players, where player N + 1 plays adversarially against the others (details in Appendix B). This interpretation allows us to employ the framework of adversarial training with min-oracle (Kalogian- nis al. 2022 Liu al. 2024a) for policy updates. Let F (-IT;, be policy parametrized by w² that represents ा.' -'(-""","" 05, = 0). Also, let Pvp be the adversarial policy parametrized by db = =v=) -v=)>==z00 such that 2 corresponds to (.]s,, 0t = k) for z (v, k).. Note that if fw.. represents the expected initial state value function of non-adversarial agents in G, their objective is to find arg max,, min& fw.1. Then, assuming that there is an oracle that for any given policy n.
returns a best response policy (w) arg ming V.wAp, the MARL policy is updated as
Wn+1 w. + 3n V (/ (ww).
(15)
where B, is a step size. is straightforward to verify that minimizing V equivalent to minimizing V. the expected initial state value function of the original game Mn. Consequently. our externaly constrained RI. algorithm can serve as oracle compute "f** for each E 2, since fixing w
7

---

Published as a conference paper at ICLR 2026
Time Att
324
lime 2013
Time 2904
Tree Stepe
(a) LBF
(b) MPE-Spread
(c) SMAC-2s3z
(d) SMAC-MMM
Figure I: Average episodic return of the proposed adversarial training, evaluated over 5 runs.
reduces the problem to g Moreover, note that for a fixed va and assuming updated beliefs, the problem reduces to the standard c-MARI. setting.
Such policy optimization is theoretically guaranteed converge to Nash equilibrium of Markov games under simplified settings, such direct parameterization and fully observable states (Kalo-) giannis et al. 2022; Daskalakis et al., 2020). However, these guarantees rely on performing exact minimization at each policy update and having access to exact gradients, both of which are infea- sible in practice. To address this, we employ simultaneous gradient updates, also known as two- timescale stochastic simultaneous gradient descent-ascent (Daskalakis et al. (020). Then assuming that (w, 2 is an unbiased stochastic estimate of V V (w.), and that g gEC-Pr (w, (b) is the adversarial gradient derived by (14), our policy updates can be summarized
x/++1 = 4/n Onlw (...ee-PPO(.,,.. (wn, 1b)))
(16)
Wn+1 = Wn + Bn.w((w 2b,)
(17)
The intuition that by selecting on 2 Pn. the adversary's policy serves as an approximate min- oracle, while from the adversary's perspective the c-MARL policy appears nearly quasi-static. To compute (wo, v.,,, we first need to obtain the agents' beliefs. For this purpose, we employ a parametrized function approximator bx, (0-1 implemented using a Recurrent Neural Network (RNN) that takes x' as input. The belief model is trained against the true type 0-1 using a cross- entropy loss. Then, by feeding (b',7*) into the policy network and using the value estimate provided by the critic Var1) (3), we compute gw (wn, 2bn) in the same way as in standard actor-critic algo- rithm. We refer to our algorithm as Bayesian Type-Partitioned Adversarial Learning (BATPAL), and provide its pseudo-code in the Appendix.
5 EVALUATION
We evaluate BATPAL against various attack types in four c-MARL environments. We consider the 2s3z and MMM scenarios from the StarCraft II Multi-Agent Challenge (SMAC) (Samvelyan al. 2019) with five and ten agents, respectively. We use scenario (00---5--)0f-c in Level-Based Foraging (LBF) (Papoudakis et al. (02T) and the Spread scenario from Multi-Partich Environments (MPE) (Mordatch & Abbeel 2017). involving five and three agents, respectively. Additionally, we include results for SMAC scenarios lc3s5z and 1 1m in the Appendix.
In all environments, we applied our algorithm to train a robust c-MARL policy and set of adver- sarial policies with different severity indices. We used MAPPO (Yu al. 2022) both for updating the c-MARL policy in adversarial learning and to obtain the reference policy wo in the pre-training phase. We assumed a uniform prior over all possible types in the training. Moreover, for a low- complexity implementation, we used parameter-sharing across all agents. Accordingly, we main- tained a single neural network for c-MARL policy and K networks for different adversarial types. Moreover, during training, we update adversarial networks in randomized way, i.e., only one of K networks is updated per each update of the c-MARL policy (see the pscudo code in Appendix E. For more details on implementation, we refer to Appendix E.2
Baselines We compare our proposed method with state-of-the-an oh-the-rrt baselines including EIR- MAPPO (Li et al. 2024), Generalized Maxmin (Gen-Maxmin) (Liu et al. 2024a). and RAP (Vinit-) sky et al. 2020). We also include the evaluation of the vanilla MAPPO algorithm against the consid- ered attacks. Moreover, provide a comprehensive assessment of the results, we include for each
8

---

Published as a conference paper at ICLR 2026
our
EIR-0
GenMi
RAP
MAPPO
KT
Outs
BIR-M
GenH
RAP
MAPPO
KT
DurS
EIR-H
64nM
RAP
MAPPO
KT
OuFS
BMN-M
Gen/M
RAP
MAPPO
KT
Attack
1.00
0.59
0.79
0.71
1.00
0.97
0.99
0.98
0.96
0.90
0.94
1.00
1.00
0.98
1.00
1.00
1.00
Severity
0.68
0.41
0.47
0.26
0.38
0.85
0.87
0.78
0.53
0.58
0.85
0.84
0.66
0.49
0.84
0.79
0.58
0.96
0.96
0.86
0.97
0.95
0.79
1.00
Severity
0.18
0.29
0.68
0.81
0.17
0.18
0.94
0.88
0.59
0.29
0.59
0.62
0.67
0.00
0.83
Severity
0.35
0.00
0.38
0.21
0.06
0.62
0.73
0.49
0.66
0.02
0.58
0.82
0.70
0.07
0.05
3.
0.00
0.73
0.78
0.63
0.79
0.74
0.55
0.84
A-EIR-MAPPO
0.26
0.06
0.03
0.15
0.00
0.35
0.664
0.87
0.30
0.54
0.20
0.04
0.65
0.83
0.83
0.80
0.43
A- A-Gen Maxmin
0.68
0.29
0.68
0.80
0.49
0.01
0.64
0:69
0.93
0.02
0.53
A-RAP
0.32
0.35
0.25
0.15
0.24
0.53
0.03
0.65
0.65
0.54
0.24
0.55
0.66
0,53
0.19
0.01
0.58
0.00
0.64
0.93
0.82
0.49
0.64
0.37
0.93
ACT
0.35
0.15
0.24
0.26
0.50
0.81
0:49
0.00
0.00
0.53
0.83
0.72
0.35
0.22
0.20
0.00
0.75
0.09
0.75
0.79
0.78
0.46
0.87
DYN-1
0.44
0.73
0.59
0.83
0.62
0.47
DYN-2
0.59
0.24
0,50
0.24
0:35
0.53
0.74
0.62
0.05
0.63
0.73
0.71
0.56
0.57
0.74
0.38
0.90
0.92
0.90
0.86
0.63
0.42
0.42
0.96
LBF
MPE-Spread
5MAC-253z
SMAC-MMM
Figure 2: Performance in four environments against 10 adversarial policies (Best, 2nd best not considering KT). The episode rewards for MPE-Spread and LBF are in [-189, -47] and [0.4, 0.74).
attack the results obtained using an oracle defender that is aware of the type of the adversary and is trained against it. This baseline, referred Known Type (KT), serves empirical upper bound. Finally, we include comparison with ROMANCE (Yuan et al. (023) in Appendix D.2
Attacks We use 10 adversarial policies for the evaluation. On the one hand, the policies trained in the adversarial training process of BATPAL, these are indexed by their severity level. addi- tion, we use the adversarial policies trained against EIR-MAPPO, Gen-Maxmin and RAP, these marked as "A-X," where X corresponds to the name of the baseline. To assess generalization, we further evaluate all methods against three dynamic adversaries, unseen by all methods. These adver- saries are trained by fixing non-victim policies and training an RL agent with a reward function that balances adversarial impact on c-MARL performance with detectability (Kazari et al. (002). We consider three such adversaries: ACT, which minimizes team reward, and DYN-1 and DYN-2, with DYN-2 placing greater emphasis on close to normal behavior (low detectability). For all attacks, we apply the policy to any victim agent for 50 episodes and report the averages across all episodes.
5.1
RESULTS
Figure shows the learning curves of BATPAL with K 4 severity levels. The curves show the episodic rewards of the learned policy when evaluated in both non-adversarial settings and against simultaneously trained adversarial policies. The results demonstrate the convergence of the proposed training scheme across all scenarios and adversarial types.
comparec Figure compares the performance BATPAL with baselines. For SMAC environments, we use the team win rate of the c-MARL agents as the performance metric, while for the other environments use the mean episodic total reward, normalized to enable unified comparison across environments. We can make several key observations based the results. First, in terms of non-adversarial per- formance, the BATPAL performs at least as well as vanilla MAPPO, indicating that robustification does compromise optimality under normal conditions. Second, although operative policy profile, it almost always outperforms the robust baselines policies when they face diverse
train a single
the attack they are trained against. This highlights the importance of exposing agents set of adversarial policies in order to obtain robust policies. Third, from the adversary's 's perspective, the worst performance of other baselines in many cases occurs when they face one of the attacks generated for training BATPAL, rather than their own adversarial policies. This can be attributed to adversarial training getting stuck in local stationary points, which further justifies our proposed method for adversarial search over disjoint sets. Fourth, although the upper bound represented by KT is obtained empirically and may not correspond to the true upper bound on the performance of robust policy, the performance gap to KT provides an indication of the regret associated with each policy. In many cases, our algorithm achieves near no-regret, even against unseen attacks. is also worth noting that, for previously encountered attacks, uncertainty regarding both the

---

Published as a conference paper at ICLR 2026
Table 1: Results of ablation study on the MPE-Spread and SMAC-2s3z envirommets..
(a) MPE-Spreat
(b) SMAC-2s3z
Variation
No Attack
ACT
DYN
A-Gea-Maxmin
Variation
Attack
ACT
DYN
A-Gen-Maxmin
BATPAL
1,00
0.81
0.67
0.80
BATPAL
0.98
0.72
0.52
0.50
No Belief
0.81
0.70
0.67
0.61
No Belief
0.96
0.33
0.30
0.36
Perfect Beliel
1,02
0.75
0.78
0.68
Perfect Beliel
1.00
0.34
0.42
0.39
BC PG
0.97
0.75
0.65
0.74
EC PG
0.96
0.41
0.38
0.41
Fixed Types
0.95
0.78
0.59
0.67
Fixed Types
0.94
0.42
0.14
0.19
adversary's 's type and the identity of the victim agent (if any) prevents the defender from consistently executing an optimal policy, We provide more results in the Appendix.
5.2 ABLATION STUDY
In this section evaluate the importance of various components of BATPAL by replacing each of them with some alternatives. Table shows the results of evaluating different variations of our method against three representative unseen attacks and also in the non-adversarial setting in MPE- Spread and SMAC-2s37.
Beliefs We considered two variants of BATPAL to evaluate the effect of the belief network. The first variant is trained without belief, while in the second variant the type of the adversary is provided directly to the policy network instead of a belief estimate. In Table these two variants are labeled No Belief and Perfect Belief, respectively. The results show that without the belief, the robustness of the c-MARL policy trained using BATPAL is significantly worse than when using the belief. In this case, even the performance in the non-adversarial setting can be affected as the results in MPE-Spread show. Interestingly, when the perfect belief is fed to the policies, robustness still suffers when facing attacks unseen during training. This due to that the belief captures the severity level of the attack, but not the actual attack type. These results clearly show the importance of using the belief in BATPAL.
evaluate the effect of PPO in externally constrained updates of adver- sarial policies, we consider a variant in which the adversarial updates are performed using simple policy gradient method without the clipping mechanism. We refer to this approach as EC PG. As shown in Table I although this method achieves performance close BATPAL in non-adversarial settings, it is not generally robust against attacks with different adversarial types. This can be ex- plained by the fact that, without the clipping mechanism, the policies easily move far outside the feasible region. Consequently, the adversarial policies encountered during training are not rep- resentative their designated types, which undermines overall robustness.
EC RL Algorithm
Ensemble of Adversarial Types Finally, evaluate the effect of the proposed partitioning method, we consider an alternative approach in which the c-MARL policies are trained against an ensemble of fixed adversarial policies. For this approach, we use four dynamic adversaries (Kazari et al. 2023) trained with different trade-offs between severity and detectability to ensure diversity among the adversaries used during training. Table shows that this approach, called Fixed Types, can lead to significantly lower performance compared to BATPAL. This observation highlights that while diversity necessary, it is not sufficient to achieve generalizable robustness against unknown adversaries.
CONCLUSION
We showed that reference-value-bon partitioning of adversarial types enhances the adaptability of c-MARL agents to unseen adversaries by exposing the c-MARL team to a diverse set of adversarial policies, demonstrated both theoretically and empirically. We proposed EC-PPO to learn adversarial policies of different types and demonstrated that it can be effectively integrated into Bayesian adversarial learning framework BATPAL. Our results show that BATPAL outperforms the state-of- the-art by achieving almost no-regret performance against various unseen attacks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the Swedish Research Council under grants 2020-03860, 2024- 05269 and 2025-06310 and by the Swedish Civil Defence and Resilience Agency (Project MAD- VAMCHS). The computations were enabled by resources provided by the National Academic In- frastructure for Supercomputing in Sweden (NAISS) at Linköping University partially funded by the Swedish Research Council through grant agreement no. 2022-06725.
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Alexander Bukharin, Yan Li, Yue Yu, Qingru Zhang, Zhehui Chen, Simiao Zuo, Chao Zhang, Songan Zhang, and Tuo Zhao. Robust multi-agent reinforcement learning via adversarial reg- ularization: Theoretical foundation and stable algorithms. Advances Neural Information Pro- cessing Systems, 36:68121-683 2023.
Lorenzo Canese, Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Marco Re, and Sergio Spanò. Multi- Multi- reinforcement learning: A review of challenges and applica- tions. Applied Sciences, 11(11). 2021.
Andrew Cohen, Xingye Qiao, Lei Yu, Elliot Way, and Xiangrong Tong. Diverse exploration via con- jugate policies for policy gradient methods. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3404-3411, 2019.
Imre Csiszár and János Körner. Information theory: coding theorems for discrete memoryless sys- tems. Cambridge University Press, 2011.
Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in neural information processing systems, 33: 5527-5540, 2020.
Kenneth Derek and Phillip Isola. Adaptable agent populations via a generative model of policies. In Advances in Neural Information Processing Systems, volume 34, pp. 3902-3913. Curran As- sociates, Inc., 2021.
Tanner Fiez, Benjamin Chasnov, and Lillian Ratliff. Implicit learning dynamics in stackelberg games: Equilibria characterization, convergence analysis, and empirical study. In International conference on machine learning, pp. 3133-3144. PMLR, 2020.
A. M. Fink. Equilibrium in stochastic n-person game. Journal of Science of the Hiroshima
University, 18,89-93, 1964.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv: 10615, 2019.
Aaron Havens, Zhanhong Jiang, and Soumik Sarkar. Online robust policy learning in the presence of unknown adversaries. Advances in neural information processing systems, 31, 2018.
Fivos Kalogiannis, Ioannis Anagnostides. Ioannis Panageas, Emmanouil- Vasileios Vlatakis- Gkaragkounis, Vaggos Chatziafratis, and Stelios Stavroulakis. Efficiently computing nash equi- libria adversarial team markov games. arXiv preprint 220: 02204, 2022.
Kiarash Kazari, Ezzeldin Shereen, and György Dán. Decentralized anomaly detection in cooperative multi-agent reinforeement learning. In Proc. of IJCAI, pp. 162-170, 2023.
Kiarash Kazari, Ezzeldin Shereen, and György Dán. Distributed detection of adversarial at- tacks in multi-agent reinforcement learning with continuous action space. arXiv preprint arXiv:2508. 15764, 2025.

---

Published as a conference paper at ICLR 2026
Nick-Marios T Kokolakis, Aris Kanellopoulos, and Kyriakos G Vamvoudakis. Bounded rational unmanned aerial vehicle coordination for adversarial target tracking. In 2020 American control conference (ACC). pp.
IEEE, 2020.
Sunwoo Lee, Jaebak Hwang, Yonghyeon Jo, and Seungyul Han. Wolfpack adversarial attack for robust multi-agent reinforcement learning. In Proc. of International Conference on Machine Learning, pp. 33025-33056 PMLR, 2025.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang. and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.
2019.
Simin Li, Jun Guo, Jingqiao Xiu, Ruixiao Xu, Xin Yu, Jiakai Wang, Aishan Liu, Yaodong Yang, and Xianglong Liu. Byzantine robust cooperative multi-agent reinforcement learning as a bayesian game. In ICLR, 2024.
Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In Proc. of IEEE Security and Privacy Workshops (SPW), 2020.
Xiangyu Liu, Souradip Chakraborty, Yanchao Sun, and Furong Huang. Rethinking adversarial policies: generalized attack formulation and provable defense in rl. In The Twelfth International Conference on Learning Representations (ICLR), 2024a.
Xiangyu Liu, Chenghao Deng, Yanchao Sun, Yongyuan Liang, and Furong Huang. Beyond worst- case attacks: Robust rl with adaptive defense via non-dominated policies. In The Twelfth Interna- tional Conference on Learning Representations (ICLR), 2024b.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4940-4947, 2020.
Robert Loftin, Mustafa Mert Çelikok, Herke van Hoof, Samuel Kaski, and Frans A. Oliehoek. Un- coupled learning of differential stackelberg equilibria with commitments. In Proc. of Internatonaa Conf. on Autonomous Agents and Multiagent Systems (AAMAS), pp. 1265-1273. International Foundation for Autonomous Agents and Multiagent Systems, 2024.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv: 770.00900,. 2017
Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA), pp. 8522-8528. IEEE, 2019.
Matteo Papini, Matteo Pirotta, and Marcello Restelli. Smoothing policies and safe policy gradients. Machine Learning, 111(11):4811-4333 2022.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. Proceedings of NeurlPS, 2021. URL
07869
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained re- inforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32, 2019.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017.
Thomy Phan, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, Bernhard Kempter, Cornel Klein, Horst Sauer, Reiner Schmid, Jan Wieghardt, Mare Zeller, et al. Learning and testing resilience in cooperative multi agent systems. In Proceedings of the 19th International Conference on Au- tonomous A Agents and MultiAgent Systems, pp. 1055-1063, 2020.
12

---

Published as a conference paper at ICLR 2026
Thomy Phan, Lenz Belzner, Thomas Gabor, Andreas Sedlmeier, Fabian Ritz, and Claudia Linnhoff- Popien. Resilient multi-agent reinforcement learning with adversarial value decomposition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11308-11316, 2021.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wi- ley Series in Probability and Statistics. Wiley, 1994.
Aryaman Reddi, Maximilian Tölle, Jan Peters, Georgia Chalvatzaki, and Carlo D D'Eramo. Robust adversarial reinforcement learning via bounded rationality curricula. In ICLR, 2024.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1992.04444 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. In International Conference on Machine Learning, pp. 6215-6224.
PMLR, 2019.
Ilnura Usmanova, Yarden As, Maryam Kamgarpour, and Andreas Krause. Log barriers for safe black-box optimization with application to safe reinforcement learning. Journal of Machine Learning Research, 25(171):1-54 2024.
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.0188 01825, 2020.
Lintao Ye, Martin Figura, Yixuan Lin, Mainak Pal, Pranoy Das, Ji Liu, and Vijay Gupta. Resilient multi-agent reinforcement learning with function approximation. IEEE Transactions Auto- matic Control, 2024.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems, 35:24611-24622 2022.
Lei Yuan, Ziqian Zhang, Ke Xue, Hao Yin, Feng Chen, Cong Guan, Lihe Li, Chao Qian, and Yang Yu. Robust multi-agent coordination via evolutionary generation of auxiliary adversar- ial attackers. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp- 11753-11762, 2023.
Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. Heterogeneous-ager reinforcement learning. Journal of Machine Learning Research, 25(32): 1-67, 2024. http:: htl.httt:////mr.rrg/////// http://arrrpppppprr./////mmp v/sss//22//2333 22255/23-488.... html
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A RELATED WORK
Adversarial robustness in reinforcement learning has been studied mainly through two main ap- proaches: adversarial training and robust learning. In adversarial training, known form of adver- sarial perturbation is introduced during the training phase, allowing the agent to learn both adversar- ial and nominal transitions simultaneously. For example. Gleave al. (2019); Pattanaik et al. (2017) employ this approach to defend against various types of manipulations. A closely related method (Havens et al.) 2018) which applies adversarial training within a meta-learning framework to en- able adaptation to attacks. These methods primarily target training-time timm attacks and rely on prior knowledge of the adversary.
Robust learning instead models the adversayy interaction game, often zero-sum, where the agent seeks max-min policy for execution-time robustness. Such works are often categorized under robustness to uncertainty, but the uncertainty is explicitly modeled as being induced by an adversary. For instance, RARL. (Pinto et al.] (017) and RARAL (Pan et al.] (019) consider adver- saries capable of applying model disturbances and propose alternating optimization methods to find a robust policy. On the other hand, [essler et al. (2019) and RAP (Vinitsky et al.] 2020) study ad- versarial manipulation of actions. Although effective against worst-case attacks, such approaches can be overly conservative; recent work (Liu et al.] (024b) addresses this by considering arbitrary non-worst-case adversaries in a lifelong learning context.
In MARI,, robustness has been explored both at training and at execution. Training-ting defenses include adversarial regularization for smooth policies (Bukharin et al. 2023) and consensus-basee
learning robust to Byzantine agents (Ye et al.. 2024). Fxecution-time resilience has been studied
through robust learning. M3DDPG (Li al. 2019) adopts a max-min value function with the idea that each agent assumes all other agents to be adversarial. RAT (Phan et al., (020) and RADAR Phan et al. (202T) consider environments with a subset of worst-case adversarial agents. ROMANCE Yuaan et al. 2023) and WALL (Lee et al. (202)) consider adversaries capable of targeting all agents but with a limited budget of attack numbers. Recently, Li et al. (2024) propose adversarial belief states that allow agents to adapt online when teammates are compromised. While this approach addresses the challenge of reacting to attacks on different agents, it remains focused on worst-case robustness and does not capture the diversity of adversarial strategies. Finally, Liu et al. (2024a) studies adaptation to non-worst-cas adversaries in two-agent scenarios. However, the authors con- sider fixed adversarial, which limits the generalization of the robustness to unseen attacks.
B BAYESIAN GAME FORMULATION
B. DISCRETE TYPE FORMULATION
As explained in Section Mr Bayesian game with type spaces é³. In Bayesian games the strategy space is assumed to be type-independent. while the utility is assumed to be type dependent. To align our problem with this model we define the utility function
V
if 0t =0 0 Ō= = k, n' E (i,k) E E
( i 2)
VT
2
-00, i = k, 11,, (i, (tak) Z
The last line is to restrict the set of adversarial policies of each type to the corresponding set IIg, and the second line is based on our assumption that the representative of each discrete type is the worst case adversarial strategy in the corresponding partition.
Now assume that (T" with pmv = . ) is a PBE of MB. With the utilities defined above, it can be immediately seen that it corresponds to a solution of ( and vise-versa. This is because when player v's type is 0 # it knows its own type and also the type of the other players. So for fixed strategy profile n², it plays strategy pr that minimizes V " within the corresponding [l.. On the other hand, when a player i's type is 0 O. given fixed
14

---

Published as a conference paper at ICLR 2026
profile (x="',p")) it plays the strategy that maximizes its expected payoff based on its belief, and the payoff is defined as V(ri. mn "),r"e""" for the type corresponding to 6"
Moreover, PBE minimizes the regret in MB for the given adversarial profile p². To show that, note that we can write
arg min R(x)) = arg min (v, pr (T]
arg min (i [max V sprrion) Vm.pa"
arg min (e,00 Jud max ( wo
(e Vnone
(18)
Notice that max (VT por is independent of thus the minimizer above is equivalent
arg min (o,00)-ho V "000" arg max E (1, V '"".0"
(19)
which is satisfied by πt by definition.
B.2 EQUIVALENT DEC-POMDP FORMULATION
The Bayesian game My g ca equivalently formulated as a partially observable stochastic game G with + 1 players, where player N - 1 denotes the adversary. and with states & = (s, 0). Each agent& i E A only observes its own type O (as part of its observation in G), while the adversary has full observability of the types 0. The reward function (5, a) is the same as R(s, a) for E NN and set to - [ss a) for player N + 1.
The initial state distribution is is based on p(0), i.e.,
s (su. 0)) ((so)p(0),
(20)
and the state transition probabilities are defined as
P(5lss,))
if =
P((d) 0 ((s,
0,
otherwise
Additionally, when = k > O, the strategy space of player N + is restricted to policies Il with (v, k). In this case, player v's actions become ineffective, which can be modeled using a singleton action set. This model can considered as partially observable "Adversarial Team Markov Game" proposed by Kalogiannis al. (2022).
15

---

Published as a conference paper at ICLR 2026
C PROOFS
C.I PROOF OF PROPOSITION 3.1
First we need to argue that Vim and V are well-defined. Notice that with full observability as- sumption, the DEC-MDP model is equivalent to an stochastic game, which always has Nash equilibrium (Fink 1964). Since the rewards of all players are identical, the Nash equilibrium corre- sponds maximizing V over the set of all policies. Thus, To, and accordingly, Vimx exist.
Now if we fix the policies of all non-victim agents to To then the adversary faces a single-agent MDP with the reward function and the state transition probability defined as follows:
R(s,a")= = Σ R(s, (a",
") "(a"""a (s))
(21)
P(s'|s,a')= Σ P(d's, (a", a )T) (a lss)
(22)
Such MDP always has optimal solution (Puterman 1994). thus V also exists.
Now, we show that all sets IT2 are non-empty. For a given victim e, consider 7" E aar min VRo.P= Define P ap" + (1 a)z for E [0, 1]. Fix To as the policy of non-victim agents. For any policy, T,, the Bellman equation matrix form is
(I- -I-2
(23)
where vx is the vectorized state value function, p=" is matrix with elements PT En P(s"|s, (a")ss, rr is vector with elements r -a R(s, "",,a")""" (a"ls), and P and R are defined above. Accordingly, have VT, 1T vn For ph. it is easy to verify that
p.. aP# + (1-aa) a(PF" - -pri) pro r ard + - +(1-a)r"s a(r -r + ro.
(24)
(25)
Thus, have
Vo = T (I- - -P7o))
(a(r"" -r rm)).
(26)
Note that, since pF" and p are both row-stochastic matrices, PP is also a row-stochastic matrix and thus, the matrix inverse in (26) always exists, and the result a continuous function of a. Accordingly, V in (6) is continuous function of When is 0 and 1, V equals V and V respectively. Thus, as a sweeps between and 1, VT sweeps between Vmax and V min
continuously. Thus, IT, for all 2 is non-empty.
C.2 PROOF OF PROPOSITION 3.2
For notational simplicity, we omit the superscript 9. Note that when मे is fixed, under the full ob- servability assumption, the DEC-MDP can be viewed MDP for agent Thus, the performance difference lemma (Agarwal et al. 2021) implies that
To (so) Fo (so)) E. Banddiin Bas (TT.P((s.
(27)
Taking the expectation with respect to so, we get
vo - P Egnd E
( ( aa)
(28)
By taking the absolute value of both sides and applying the Jensen inequality, we can write
VTo / To.p3 < Eg Earmpi((b) [ATOrP
(29)
16

---

Published as a conference paper at ICLR 2026
Now, can write
(atcl)()e)) AATo
Σ plllasss) (sToni [q (
Σ [((0a(ss)(000000se00)) B Bauwpallss) (s,
Σ (2((""))) @mo
(ar(al|s) p2(a"1 (
(30)
Note that the reward function is bounded in [-1, 1]. thus the Q-function is bounded by E 17x Thus, conclude that
I-y"
Ba-np((11)) [A,0 [s p2())]1
(31)
1 1-7
and accordingly,
V VRaiii $ 1-7
E Ea [|p((s) (s))))
(32)
On the other hand, the Pinsker inequality (Csiszár & Körner 2011 implies that
[| P2(s)]11 (PDis(Pi(s)) ()())), E
(33)
Thus, by taking the expectation and applying Jensen's 's inequality to the concave square-root function, one can obtain
VAO.PiS 1o S
7) V
2F s (pra))(())))
(34)
and therefore,
(1 - 7)2 no.pi m00 2 Bandgin (Pkc())(())))((6))
(35)
2
C.3 PROOF OF PROPOSITION 3.3
Note that by definition, we have vr E II, for 2 = (v, k), Accordingly,
k K
k
S Apink S
(36)
K
We can write mo.pp" Vmax 7 & (Vmax V Thus. applying Proposition 3.2 leads
(1 (1-7) 7)2 (Viax Vre p1pppoo 70
(s)) 2
(37)
Snd tea
2
Therefore, the diversity can be lower bounded as
(1 (1- 2K(K 1) (Vnnx V
Div(fp ty ) 2
Mp
nor 12
(38)
{
Thus, it is sufficient to find the minimum of Sx==xs Eitey MMp.os Up conditioned on 36). For
the ease of notation we denote Mptn, by 7/s in the rest of the proof. We claim the following: TF K is
even, the minimum of Sx is attained when n x for < K and ni
i-1
K for í 2 K 1. If K
odd, the minimum of Sg is attained when n & for < 2 m d-1 for > K+1 and
K-1,
K
7841 }.. Moreover, in both cases the minimum is (K-IXK-
17

---

Published as a conference paper at ICLR 2026
We prove this claim by induction. We prove that if it holds for some K, it will hold for K + 2 as well. The bases for our induction are K' = 2, and K = The case with K = 2 is trivial. For
K 3, we write
1 2((1)) (12 3 3
S3 2((7s- -m)² (122 m)² + (7s 72)²)) 2 2(1,
3 -
72) -
(39)
The minimizer of the right hand side of (39 is 7/2 1.2 and the left and right hand sides are equal when 1/1 _/4 and 7/3 = Thus, these values correspond to the minimum of S3, which is $.4 This proves our claim for K 3.
Now prove that if thee claim holds for K K1, it holds for K = K1 + 2. We can write
SK,+2 = =2 1 (7)++
K
K1+1 Σ (mKL+2-m)*) (40)
Σ 28KjSKK+1
(n-n))²²
(m - Σ
Note that SK is invariant with respect to any constant shift of all elements, accordingly the min- imum of E2 n;)² is achieved by the same relative configuration the mini- mizer of Sk,, Based on the assumption of induction, the minimum of Sk, is (K(-1)(1-2)
Then, according to the scaling from K1 to K1 + 2, the minimum of 2 1 n,) S' (kit2) 2
Accordingly, given (36) have:
K1+1
K1+1
K1 K1 +
1
K1 + 1
Sk 22
S' -
mh
2
+
K1 + Thh²)
(41)
K1 +2
2=2
k=
The equality holds when 771 claim (due to the assumption of induction and the fact that K and K + have the same parity). K1 is even, the first summation in (41) in this configuration is equal to
K1 7K =
K1+1 and the rest of 7/ 's are as specified in our K
K1/2
k
k ソー
K
K1-1
K1
+
2
K1+2 (K1 + 2 2
Σ
1²).
(42)
1
K1/2
km
The second summation is also equal the same value. Thus, the value of Sk this configura- tion becomes
K
(K1 1-(11) - 2)
Sk.+2 = (Ki + 2)
K? +
+ 2
3 +
k2
12
k=1
1
3 K? + (K1 - 1)(K: -2) Ki(Ki - 1) 1)(2 -1)
= (K1 2)2
12
KiKK3 + 5K7 + 8K1 + 1
Ki(Kı + 1)(Ks + 2)
2 (K1 + 2)2
2
)
12
(K1 + 2)2
12
(43)
which is simplified (1) Similar caleulations for odd K₁ leads the same result. This is the same expression as we claimed for K = K₁ + 2, which concludes the proof.
C.4 PROOF OF PROPOSITION 3.4
We have
R (TT") )= = (max V /=
(44)
Note that max V < max, V Vmux-
Also, we can write
min V=
(45)
E'Ell,
However, based on the definition of ng, we know that min V max, min V=.e". Thus, follows that
k
V
2 min VTa.""
V Vimx
(Vmux L
(46)
p² Ell,
K
Thus, conclude that Rax (x+)
R (Viax Ve )
18

---

Published as a conference paper at ICLR 2026
C.5 PROOF OF PROPOSITION 4.1
The policy gradient theorem (Sutton & Barto 2018) implies that
V Vellesmp (s)] Boy a V log (p(als)) j 0,
(47)
Thus, we can write
E (0) (s)]
Egndoy (-)
V log Pas (a)s) (s. [.))]
V. 10g Bsny V() [s) -1
E Esnyl 16 [s)] -
V. (0) (s)
Bon log (als) (s, a)]
V log(lo Bog( (1000())) (s)])
Eanu V(0 (s)
h (0)
pe (s)
This proves the result.
C.6 PROOF OF PROPOSITION 4.2
First let us establish the following preliminaries.
Lemma C.1. let G and H be the upper bounds on || log (als)) and |[ log re- spectively. Then,
The variance of Vi is bounded by o² M(--))²
The variance of the gradient estimates of the value function, i.e., 1 I-y (jy is bounded by
3²(M) M(1-3)
AG²
VPPe is Lipschitz continuous with respect sb with constant (i)
2G (1-7)2. also B-
smooth with smoothness constant B (1-2 (+ + H1.
Proof. First notice that if a is the variance bound per one sample (one episode), then (M) = vM 0. 1 Vi
To bound a, note that ||ra|| s El 7'rell
I- - 1- and cannot be
larger than
Regarding the variance of the gradient estimation, note that for any st,
V log (a:lss) (Be, az)ll S (1-7)
G max A (s,
(48)
8.0
Moreover, we have
2
(AP (QP VP* < !Q (s. + (VP) _
(49)
1-
Thus, each per sample (per episode) estimate of the gradient is norm-bounded by implies that
2G (1-7)a+
which
4G
V
(50)
M(1
The bound the norm range of V v VPo will also implies Lipschitz continuity with the same bound
26 [I-
Finally, the smoothness constant is the direct consequence of Lemma 6 in PPapin al. 2022 by setting the bounds E|| log (alss() < G, E[[| log ps (a]s) G2, and
[ogs)())s)]] < H.
19

---

Published as a conference paper at ICLR 2026
In analysis use the results of Usmanova al. (2024). however, given the structure of our problem we are able to derive simpler step-sizes. Moreover, we base our proof on a required error e on the true gradient Iy given fixed À instead of fixed required emoor the noisy gradients.
To use these results of Usmanovaa et al. (202)), first we have to confirm that the required assump- tions hold. The Lipschtiz continuity and smoothness of V are already established using the above lemma, and assumption 3 in Proposition 4.2 ensures feasible starting point. Moreover, since the gradient of our constraints differ only on their signs, the extended Mangasarian- Frroniiiii-rrtizzzz constraint qualification (MFCQ) requirements proposed by [Usmanova al. (2024) equivalent to the require- ment that "there are positive constants Ç and q, such that V V.
(0) 2 g when - S V S
(0)
or V I Assumption 4 in Proposition 4.2 guarantees this condition. This is because V v Vel is differentiable in [ + Ç) and (h S. hi and hence V Vo) is continuous over these sets. Accordingly, the Extreme Value Theorem guarantees the existence of minimum in each of these sets. We define q as the minimum among these two minimums, and is indeed positive.
(0)
Let us define 0.5( z0r )² and
C=
(51)
21 () max
SBcA L2
1+
Be 41.
where B and L are defined in Lemma function in ( Now define
C.
We denote by Fx the log-barrier regularized objective
((F) + 1-y + 22 1))
Niter
(52)
32еCA
and let 8
5
"Nuer"
To find the local smoothness constant, we further require to define x1 VP88a (0)
1, x h VP
z v and x = h V. Moreover, assume ra = z o (M) In
Now
we are ready to introduce the adaptive step-size and the local smoothness constant Let B2 = + 10B z + ) + SL2N o + x Moreover, let Dn &
Fx.
2
min
We define the step size
21+ Bz1 21+ Bz2
{
Dn 1 1 [196..11
On = min
(53)
Lemma C.2. The function Fx is locally smooth and its smoothness constant is less B2 with proba-
bility at least 1 -
Proof. Let Un (V.) smooth with constant
Vi (0)
Lemma 2 in Usmanova et al.
2024 implies that F3 is locally
(2( = B 10A
10X
B +
8A((
Un 2
+
Un 2) I
(54)
x_d
long on & min
}. and J +
2 for 1,2.
/ 1
/
Now, note that we can consider 23 is lower bound on zy with probability 1 1- 8, and we have
PPn
Yn = (V
Peon (0)
Ioin Sein = |
peen (0)
I
Pen S L.
(55)
Thus, the step size a, satisfies the requirement with probability least 8. Moreover, Lemma in Usmanova al. 2024) implies that w
which concludes the proof.
n
2.
20

---

Published as a conference paper at ICLR 2026
Based on the smoothness of Fx, for any with probability 1 - 3 we have
(x(e)) - Fx(n-1) on (gv., Iw.)) / Bra 19 112 2 2 zan 1 1| an Ge Illlon. - - l|.
1
2
(56)
The last inequality holds because of the selection of the step sizes, which implies that an B3 S with probability 1 8.
Note that with this selection of step sizes, Theorem 4 in Usmanova et al. (0224) implies the feasi-
bility of all vn for all = 1,2,... .... Niter with probability at least 1 - 2N 2N = 1 - 6. Then, by summing up the above inequality for n < Niter, we obtain
Niter11
Niter min an anllls.. 11 |ل 119.. - D0..11)
x
Σ ==0
anll9e.
|1( |122le9..11 19 11 11 - Dvu.ll)
I Fx(1)) min Fx(1)
1
$ (1 + 1 1- 7
+ 22 +2Alog(h -1)
(57)
w.p Accordingly, given the definition of Niter. we obtain
3
min an Il $ 11 1 |19.. ll - llav. - Iv.. I1)
$ CC.
(58)
32
To bound llg.. - Iv. 1l. assume that M is large enough such that
(M) S
min ( (z+h), (..")) (z.
20AL log
1, In T. 2 T.
(M) s
min
(59)
20 log 1
Then, Lemma in Usmanova al. 2024 implies that w.p. least 1- 8 we have
1 + -/2 x2 I
llev. - Dv.ll1 S (MM log 3
+ X0 log
)
(M)
log
+
xx .=
xr x²
$ 5c
(60)
20
4
Thus, w.p. 8, Jehn Ivall s 1 for all < n Nater. On the other hand, by Lemma 5 in we know 2 CA for all 0 s < Neter w.p. 1 - 6. Thus we can write
Usmanova et al. (2024)
1
CCx 2 min an vallle..11 11 |1 - -19e.. - -vn.l)) 11)
2 ,M 9
(61)
32
where g - mina ll positivity) we obtain
Therefore, we have 2 -g s 3 Solving for g (and given its
2
2
3e
9
+
(62)
16
16
Finally, we obtain that w.p 8:
6
3E
min 9o.. < min Ilav.. -v.. 11)
(63)
which concludes the proof for the first part. Moreover, approaching the solution to KKT point
of the constrained problem is the direct consequence of Lemma 7 in Usmanova et al. 2024
when
X + 0.
21

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RESULTS
D.1 LEARNING CURVES OF THE BASELINES
Figure shows the learning curves of the baselines in their training phase against their adver- sarial policy.
Ecnsac Pabut
sodec sccur Over
Lancht satar
200
01
03
Tanfenn
(a) LBF
(b) SPF-Spread
SMAC-2s3z
(d) SMAC-MMM
Figure 3: Average episodic return of the basclines during adversarial learning.
D.2 EVALUATION OF ROMANCE
ROMANCE (Yuan al. 2023) is c-MARL framework designed for adversaries with limited budget of action manipulations. Thus its comparison in our setting is not fair, however, for the sake of completeness we report its performance against attacks trained in BATPAL and also dynamic adversaries in SMAC-2s3z. We used the already trained models in the original implementation.
Table Win rate of ROMANCE against different attacks SMAC-2s3z
Attack
Win Rate
No Attack
0.97
Severity
0.23
Severity
0.08
Severity
0.07
Severity 4
0.05
ACT
0.05
DYN-1
0.07
DYN-2
0.06
D.3 REFERENCE- VALUE EVALUATION
Figure44shen shows the normalized initial state value function of the different attacks trained for BATPAL against the reference policy Ty in all environments. This figure shows how the type of the attacks changed during the training.
D.4 COMPARISON OF DIFFERENT VALUES OF K
Table shows the evaluation of policies learned by BATPAL with different number of severity types K. It can be observed that, as expected, increasing the number of adversarial types generally enhances the robustness the c-MARL policy. However, each additional severity type requires the introduction of additional network, which increases the overall training time. Nonetheless, the results indicate that even with a relatively small number of severity levels (c.g.. = 4), satisfactory performance can be achieved across most scenarios
D.5 MULTIPLE VICTIMS
BATPAL can be adapted for robustness to attacks against multiple victims. Consider a powerful adversary that launches coordinated attack on N, victim agents v1, V2, -.. UN.- We can group the
22

---

Published as a conference paper at ICLR 2026
(a) LBF
(b) MPE-Spread
(c) SMAC-2s3z
(d) SMAC-MMM
Figure 4: Normalized average initial state value of the BATPAL attacks against the reference policy.
Table 3: BATPAL performance for different values of K in four environments.
Environment
Attack
= 3
K
K 5
No attack
0.99
1.0
1.0
EIR-MAPPO
0.72
0.64
0.73
MPE-Spread
ACT
0.79
0.81
0.89
DYN-1
0.64
0.67
0.71
DYN-2
0.70
0.72
0.70
No attack
1.0
1.0
1.01
A-EIR-MAPPO
0.20
0.26
0.38
LBF
ACT
0.18
0.35
0.35
DYN-1
0.44
0,52
0.53
DYN-2
0.52
0.58
0.60
No attack
0.94
0.98
0.98
M-EIR-MAPPO
0.1
0.38
0.29
SMAC-2s3z
ACT
0.16
0.72
0.79
DYN-1
0.23
0.52
0.53
DYN-2
0.55
0.71
0.76
No attack
1
1
A-EIR-MAPPE
0.83
0.85
0.87
SMAC-MMM
ACT
0.87
0.89
0.88
DYN-1
0.83
0.82
0.88
DYN-2
0.89
0.92
0.92
victim agents into a single "compound" agent whose action and observation spaces correspond to the joint actions and observations of the victim agents. Doing results in system with N -N, benign agents, one adversary and corresponding severity scores. Our analysis applies in this setting, and BATPAL can be used for training. The main shortcoming this approach the combinatorial explosion of the sets of victim agents with increasing N.. as the prior distribution would have to be defined on all possible combinations of N, agents, which would also have to be explored during training.
Importantly, this formulation eliminates the need to define separate severity level for each victim agent; instead, the severity is defined for the entire group as whole. Moreover, in principle, it can provide robustness to attacks against less than N, agents, since, for example, an attack on N, agents can be viewed as attack on N., agents with lower severity.
We evaluated this approach by training a model with N, = 3 victims for SMAC-MMM, and testing it when two or three agents are manipulated using different attacks. Table 4 presents the results of this evaluation. Since EIR-MAPPO designed for single victim, we include comparisons only with the other two baselines. The results show that BATPAL is superior to the baselines in the multiple-victee setting as well. However, for a severe attack such as ACT, when three agents are attacked, the task becomes challenging that none of the methods is able produce good policy.
23

---

Published as a conference paper at ICLR 2026
Table Win of different methods in SMAC-MMM against attacks multiple victims.
Attack
BATPAL
Gen-Maxmin
RAP
Severity
0.64
0.68
0.40
DYN I
0.44
0.16
0.27
2 Victims
ACT
0.32
0.06
0.24
Severity
0.28
0.18
0.14
DYN 1
0.10
0.00
0.00
3 Victims
ACT
0.00
0.00
0.00
D.6 ADDITIONAL ENVIRONMENTS
In addition the results shown in Figure we provide here results for additional SMAC scenarios, namely lc3s5z and 11m. It can be observed that BATPAL remains superior to the other baselines providing consistent robustness, especially in SMAC-Ic3s5Z, where the performance gap between BATPAL and the other methods is substantial.
Table 5: Win rate of different methods in
and SMAC-11m.
Environment
Method
No Attack
Severity
Severity
Severity
Severity
A-EIR-MAPPI
A-Gen-Maxmin
A-RAP
BIR-MAPPO
0.90
0.53
0.43
0.52
0.52
0.56
0.41
0.47
SMAC-Ic355
Gen-Maxmin
0,90
0.45
0.35
0.43
0.42
0.38
0.79
0.37
RAP
0.98
0.64
0.77
BATPAL
0.96
0,82
0,86
0.63
0.54
0.29
0.69
0.44
0.86
SMAC-IIm
Gen-Maxmin
1,00
0.90
0.83
0,32
RAP
0.98
0.86
0,86
0.46
0.40
0.49
0.64
0.44
24

---

Published as a conference paper at ICLR 2026
E IMPLEMENTATION
E. PSEUDO CODE
The pseudo-code of our algorithm shown in Algorithm One important point is that when updating the policy networks, we sample the type according to the prior distribution and update only the corresponding network. Consequently, each update, at most one adversarial network is updated.
Algorithm Adversarial Learning in BATPAL
The reference policy networks wy. the policy networks w the critic (11), the reference critic ((o). the belief networks x", and the adversarial policies v
Input Networks:
Pretraining:
2: Train the c-MARL team in a non-adversarial environment and obtain wp. Vmax, and V
3: Adversarial Training:
4: for each iteration do
5: Sample 0 ~ p. and set 2 (k, u) 0.
6: for m 1, ..2.... M do
D Storing experiences corresponding to POMDP,
7: Sample the initial state and observations (0,(1))
8: for f == 0, 1, Tm do
9:
Sample non-victim actions (1 T (11 bi), where bj bx (Té, Sample adversary action if z # 0, ot ~ Pip
Set the joint action profile a, = (a ") " if
12:
Environment transitions and (1+1, (1,(1) obtained
13:
Store the transition history H2,(1)
14:
end for end for
15:
16:
if * 0 then
Storing experiences corresponding to POMDP
17:
for m = 1,2...., M do
18:
Sample the initial state and observations 80.(0), (o,(0)
19:
for t = 2,........ T do
Sample non-victim actions di,00) Tug (-ITé,(0) Sample adversary action a, ~ Pap
Set the joint action profile a, (a a,
23:
Environment transitions and 8)+1,(0)= (r,10) are obtained
24:
Store the transition history H4,(()
25:
end for end for end if
26:
27:
28:
Update the critics ((0)- Φ(1) using the advantages obtained by He,00) and He,))
29:
Update x" using true types in Hz,(1)
Update w using (17)
31:
Update g²* using (16
end for
E.2 IMPLEMENTATION DETAILS
We used MAPPO as the backbone algorithm in updating c-MARL policies. Our implementation has been built on the implementation of HARL (Zhong et al. 2024) and EIR-MAPPO (Li et al.) (024)) We used parameter sharing across the agents, thus maintained single belief network, single c-MARL policy network, and one network per each adversarial type (in total K).
As baselines, used EIR-MAPPO (Li et al. 2024), Generalized Maxmin (Gen-Maxmin) Liu et al., 2024a), and RAP (Vinitsky et al. 2020). EIR-MAPPO can be regarded as a special case of BATPAL with only single adversarial type, and used their original implementation in
25

---

Published as a conference paper at ICLR 2026
Table Hyperparameters used in all environments
Hyperparameter
Value / Description
Discount factor (?)
0.99
Actor network
MLP
Belief network
GRU
Belief hidden layer
single layer with 128 units
Policy learning rate (3)
0.0005
Adversary learning rate (a)
0.0005
Critic learning rate
0.0005
Entropy coefficient
0.01
Table 7: The values used as Vnaz and V min'
Environment
Vmaa
V
min
SMAC-2s3z
15.3
4
SMAC-MMM
16
10
SMAC-11m
16
10
SMAC-1c3s52
16
7
LBF
0.55
0.3
MPE-Spread
-50
-130
comparisons. Gen-Maxmin models adversary that at each time step in two-agent setting, plays a worst-case attack (trained using adversarial learning) with probability q and a cooperative policy with probability q. We adapted their algorithm to the multi-agent setting and set g 0.5. Moreover, based on the results reported by Liu et al. (2024a) we selected a learning rate of 0.0001 for non-victim agents and 0.0005 for the adversarial agents. RAP, on the other hand, considers population of adversarial policies; however, unlike our method, these policies are not differentiated by behavioral diversity and are all trained under the max-min principle. RAP is originally designed for single-agent RI,, and we adapted it to the c-MARL setting.
In our implementations, the value of log barrier coefficient À is 0.1 in SMAC environments and 0.2 in the other two environments. The rest of the hyperparameters are the same for all environments and are reported in Table 6 Moreover, the values used as Vmna and Vmin environments are shown in Table
E.3 COMPLEXITY CONSIDERATIONS
As mentioned earlier, we keep the complexity of BATPAL low by using parameter sharing and randomized updates of the adversarial network. In terms of the number of policy updates, this makes BATPAL comparable a reference policy, but this stage uses far fewer samples than the full adversarial training, since the reference policy does not need to reach complete convergence. Another difference is that BATPAL requires two sets of experience samples (corresponding to the two POMDPs). However, sample collection is performed in parallel across multiple rollouts, and we handle this efficiently by splitting the rollouts into two groups and assigning one half to the POMDP associated with the reference policy experiences. To assess the overall complexity, compares the execution time training BATPAL with that of the baselines for the same number of environment time steps, corresponding to the results reported in Figure 2 The execution times are measured terms of wall-clock time, and for unified comparison, we normalize all times by the execution time required for the full convergence of vanilla MAPPO in the non-adversarial setting on the same hardware. By comparing observe that although BATPAL has higher complexity than the
standard adversarial learning. Our method does require pretraining
the execution times in Table 2 baselines, the resulting increase in computation time is not significant, and is essentially the cost paid for achieving unified robustness.
From another perspective, with respect to K, one would expect that more samples and more policy model updates are required to train policy that performs equally well against all adversarial types
26

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of the exccution times. The execution times are normalized to the execution time of a vanilla MAPPO until convergence.
Environment
BATPAL
EIR-MAPPO
Gen-Maxmin
RAP
SMAC-2s32
2.83
2.52
2.61
2.36
SMAC-MMM
3.53
3.21
3.26
3.16
LBF
2.44
2.10
2.27
1.98
MPE-Spread
4.21
3.54
3.90
3.28
Training Steps vs K (dashed exoected linear scaling)
SMAC, MMM
LSP
MPE-Spread
16
Trakking Stopo (naltors)
12
Figure 5: Training time-steps as a function of K. The dashed lines show the expected linear growth.
as K increases. Given our randomized update the networks, which corresponds to sampling different types during training, it is reasonable to anticipate that the number of samples needed to reach a given performance level would grow proportionally. For example, one may expect an increase by factor of 5/4 when K increases from type). To assess this, we evaluated the number of environment time steps required to reach an average reward over all types equal to 90% of the optimal non-adversarial policy. Figure shows the results of this evaluation, and shows that except for SMAC-2s3Z, which shows close to linear growth, the other environments remain well below the expected linear growth. As a conclusion, if more complex environment requires finer severity levels, the increase in complexity with respect to K can be expected be sublinear.
(note that there is also non-adversarial
27


Reviews:
[{"review_id": "q4ZBI9ULop", "reviewer": "Reviewer_PTA9", "summary": "This paper proposes BATPAL, a Bayesian framework for making c-MARL robust against unknown adversarial behaviors. Instead of assuming a single worst-case adversary, the authors model uncertainty through a Bayesian Dec-POMDP with a range of adversarial types. The authors introduce an externally constrained RL formulation with the EC-PPO algorithm to learn representative adversarial policies, and use simultaneous gradient updates to train robust cooperative agents toward a Perfect Bayesian Equilibrium.", "questions": "- The paper uses 4 levels of severity. How does it affect the performance of the proposed method if we use a different number of severity levels?\n- How does the EC PPO compare to other constrained RL algorithms? Are they any evaluation specifically on the model?\n- What is the real-world applicability of the proposed method and what are the real-world use cases for this type of method where you can assume", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is extending standard MARL robustness beyond worst-case adversaries, which could potentially provide a new problem setup.\n- The paper provides a theoretical analysis, such as regret bounds and a convergence proof for EC-PPO.", "weaknesses": "- Even though the method is theoretically sound,  the computational cost of training multiple adversarial policies (one per severity level and victim) could potentially be prohibitively large for with increasing agent team size.\n- There seems to missing ablation such as the performance of the proposed method with and without the belief network, and how does the method performs with different under different accuracy of the blief network?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5OvV8U1NUl", "reviewer": "Reviewer_oTvg", "summary": "This paper addresses the critical problem of robustness against unknown adversaries in cooperative Multi-Agent Reinforcement Learning (c-MARL). The authors propose BATPAL, a novel method that models the problem as a Bayesian Dec-POMDP with a continuum of adversarial types. To make the problem tractable, they introduce a partitioning scheme for the adversarial type space based on the severity of an attack, measured by its performance against a reference policy. This discretization allows them to frame the problem as finding a Perfect Bayesian Equilibrium in a finite-type game. Empirical results on several benchmarks demonstrate that BATPAL outperforms state-of-the-art baselines against a variety of attack strategies.", "questions": "Please refer to the weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Novel Problem Formulation: The Bayesian formulation to capture uncertainty over adversarial objectives is a relevant and timely direction for robust MARL, moving beyond the standard worst-case adversary assumption.\n\n- Comprehensive Evaluation: The experimental section is thorough, using multiple diverse environments (SMAC, MPE, LBF) and a wide array of adversarial policies, including unseen dynamic adversaries, to evaluate the proposed method.", "weaknesses": "- Limited Theoretical Novelty and Justification for Partitioning: While the Bayesian game model is a good framing, the core partitioning mechanism, while intuitive, lacks a strong theoretical justification. Proposition 3.2 provides a bound on KL divergence, but this bound can be very loose in practice, and it is not conclusively demonstrated that this partitioning is the optimal or most efficient way to discretize the type space. The choice feels more heuristic than principled, and the theoretical benefits (mitigating local optima, ensuring diversity) are stated but not rigorously proven.\n\n- High Complexity and Insufficient Ablation: The proposed method is significantly more complex than the baselines, requiring a pre-trained reference policy, K separate adversarial networks, a belief network, and a two-timescale update. This complexity raises concerns about scalability and computational cost. A major weakness is the lack of a proper ablation study. It is unclear how much each component (e.g., the Bayesian belief, the specific partitioning, the EC-PPO component) contributes to the final performance. For instance, would a simpler ensemble of adversaries trained with different fixed rewards perform similarly? The gains could be attributed more to exposure to a diverse set of adversaries during training rather than the specific Bayesian partitioning scheme.\n\n- Technical Presentation and Proofs: The presentation of the externally-constrained RL algorithm and its convergence proof is highly technical and complex. While Proposition 4.2 claims convergence, the assumptions (e.g., perfect critics, bounded derivatives, a strictly feasible starting point) are very strong and often not met in deep RL practice. The subsequent switch to a more practical EC-PPO variant, while understandable, undermines the theoretical guarantees provided earlier. This creates a gap between the theory and the practical implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yr3pw4LDi3", "reviewer": "Reviewer_GewT", "summary": "This paper studies execution-time robustness in cooperative multi-agent reinforcement learning under unknown adversarial objectives. The authors formulate the problem as a Bayesian Dec-POMDP with a continuous adversarial type space and make it tractable by discretizing adversarial types based on performance degradation (severity) relative to a reference policy. The resulting framework seeks a Perfect Bayesian Equilibrium (PBE), where the cooperative policy conditions on beliefs over adversarial types. To generate representative adversaries, the paper proposes an Externally Constrained RL formulation solved via a log-barrier gradient method with convergence guarantees and introduces a practical EC-PPO variant for stability. A severity-dependent regret bound theoretically shows that finer partitions reduce regret and improve adaptivity. Empirical evaluations on SMAC, LBF, and MPE-Spread demonstrate consistent performance gains over baselines across both seen and unseen adversarial conditions.", "questions": "Minor notation and typographical issues:\n\n1.\tIn Equation (12), it is unclear what the subscript T in S_T specifically denotes. Given the surrounding context, it appears that this might be a typographical error and that the intended notation is S_t, representing the state at timestep t. Clarifying whether T refers to a specific terminal time or if it is indeed a typo would help prevent confusion.\n\n2.\tIn Appendix Table 1, the entry labeled “SAMC-2s3z” seems to contain a typographical error; it should likely read “SMAC-2s3z.” Correcting this minor issue would improve the overall consistency of the presentation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Conceptual novelty and clarity of the discretized Bayesian game:\nThe paper presents a novel Bayesian framework that models adversarial uncertainty as a continuous type space and discretizes it based on the degree of performance degradation (severity). Here, severity is defined as the reduction in performance relative to a fixed cooperative reference policy, which allows the continuous adversarial type space to be transformed into a computationally tractable finite set of types, formulated as a Bayesian Dec-POMDP $\\hat{M}_B$ (Eq. 2). The equilibrium concept of this discretized game is defined as a Perfect Bayesian Equilibrium (PBE), where the cooperative policy is conditioned on beliefs over adversarial types and adaptively responds to different levels of attack severity (Eq. 4). This formulation clearly captures cooperative–adversarial interactions under adversarial uncertainty, enabling tractable reasoning over continuous uncertainty without relying on manually defined attacker classes.\n\n2. Technical contributions for adversary synthesis:\nThe paper introduces an Externally Constrained Reinforcement Learning formulation for training adversarial policies. Each adversary aims to minimize the cooperative team’s return while satisfying the severity constraint defined over the partitioned type space. To address this constrained optimization problem, the authors propose a log-barrier policy-gradient method and theoretically prove convergence to a KKT point under mild assumptions. Furthermore, to ensure practical stability, they propose the EC-PPO algorithm, which incorporates PPO-style clipping to alleviate instability near constraint boundaries. The algorithm performs simultaneous gradient updates between the cooperative agents and adversaries, allowing adversaries to act as approximate minimizing oracles while the cooperative policy progressively adapts to them.\n\n3. Theoretical and Empirical Insights into Severity-Based Partitioning:\nThe paper proposes a severity-based partitioning of the adversarial type space to enhance the diversity of attack behaviors and provides both theoretical and empirical justifications for this design. This approach demonstrates that considering multiple severity partitions, rather than training a single worst-case (max–min) adversarial policy, leads to lower regret and higher adaptability, as mathematically established in the theoretical analysis. Such partitioning enables the model to learn representative adversarial policies for each severity level, thereby maintaining training stability and convergence while capturing a wider spectrum of adversarial strategies. Furthermore, experimental results presented in Section 5 (Figure 2) reveal that the most severe attacks do not always cause the greatest performance degradation. This finding suggests that the impact of adversarial behavior cannot be fully explained by attack strength alone, as attacks of different severity levels may exert distinct strategic effects. In particular, in the LBF, MPE-Spread, and SMAC-2s3z environments, moderate-intensity attacks were observed to disrupt cooperative balance more severely than the strongest attacks, which is consistent with the theoretical analysis. Overall, the proposed severity-based partitioning effectively handles continuous uncertainty in adversarial intensity and provides a principled means to achieve quantitative robustness across a diverse range of attack strengths.", "weaknesses": "1. Limited threat model and multi-agent realism:\nThe paper assumes a restricted threat model in which at most one victim agent is attacked per episode. While this setting enables analytical simplification (Sec. 3), it may not fully capture the diversity of adversarial scenarios that can arise in realistic multi-agent environments. For instance, the EGA mechanism in ROMANCE [1] generates a wide range of attack policies through evolutionary processes, and the Wolfpack attack in WALL [2] dynamically targets multiple agents across timesteps, reflecting more complex and coordinated adversarial behaviors inherent to multi-agent systems. To better account for such multi-agent adversarial settings, interactive and cooperative attack characteristics beyond those of single-agent scenarios may need to be further considered. The proposed method provides meaningful diversity by learning various single-victim attacks with different severities; however, this diversity appears to focus mainly on the single-agent perspective and might not fully capture the broader robustness challenges that could emerge from simultaneous or cooperative multi-agent adversarial interactions. Including a discussion or empirical extension on how the proposed Bayesian partitioning framework could be adapted or evaluated under these richer multi-agent adversarial scenarios would help clarify the scope and enhance the practical relevance of the work.\n\n2. Limited diversity of experimental scenarios:\nThe experimental evaluation appears to be conducted on a relatively limited set of scenarios — one environment each for LBF and MPE, and two scenarios for SMAC. Compared to recent studies on robust MARL, which often consider a broader range of tasks or larger-scale multi-agent systems, the current evaluation setup may provide only a partial view of the proposed method’s generality. Expanding the experiments to include additional and more diverse cooperative environments (e.g., larger SMAC maps, alternative LBF configurations, or other benchmark multi-agent settings) would further enhance the empirical validation and offer a more comprehensive assessment of the method’s robustness and scalability.\n\n3. Computational cost and scalability with respect to severity partitioning:\nThe proposed approach maintains a separate adversarial policy for each severity level K, suggesting that computational complexity may increase as K grows. In the appendix, results are reported only for K = 3, 4, 5, and the authors also note that training becomes more demanding for larger K. In more complex environments, finer partitioning could require learning multiple diverse adversarial policies, which may introduce scalability challenges in terms of training time and computational resources. Providing quantitative measurements of the computational cost analysis would enhance the empirical transparency and credibility of the study.\n\n\n[1] Yuan, L., Zhang, Z., Xue, K., Yin, H., Chen, F., Guan, C., Li, L., Qian, C., and Yu, Y. Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11753–11762, 2023.\n[2] Lee, Sunwoo, et al. \"Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learning.\" International conference on machine learning. PMLR, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2CCoJELen", "reviewer": "Reviewer_JzM1", "summary": "This paper provides a method to optimize against Bayesian adverrsaries with several unknown types that corresponds to different reward reductions. The authors initially assume a continuous type space, then discretize it to some representitive type spaces, thus learning a more flexible policy against attackers. Experiments show higher robustness of their methods.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper solves the problem of continuous type in MARL with Bayesian attackers. While previous works solves the problem with 0-1 type, the paper expands the problem to continuous space and give solid theoretical derivations.\n\n2. Technically, the authors propose BATPAL to gain robustness against varying attackers. While experiments are not highly extensive, it sufficiently verifies the claim of users.", "weaknesses": "1. only one adversary is considered. This is probably inherited from EIR-MAPPO, but this remains a limitation nonetheless. Since the attackers are less severe, maybe we can consider multiple attackers?\n\n2. A very simple method of controlling the severity of attack is by a linear combination of a benign policy and an adversarial policy, as shown in [1]. Why we need to learn a constrained policy as shown in Section 4? I'm not asking for additional experiments, just curious about the design since current method is quite complex both theoretically and empirically, and I wonder if a simple interpolation of policies can solve most problems.\n\n[1] Action Robust Reinforcement Learning and Applications in Continuous Control.\n\n3. I do not fully understand Eqn. 7. Since it is proposed to solve (4), and in (4) the problem is formulated as a min-max game over a shared value function, then why we have two POMDPs? My understanding is we need to minimize reward while constraining it in a  certain range, but in this way it is a single constrained POMDP.\n\n4. Do there exist an exact threat model of adversaries and a clear definition of adversaries? I see some definitions in Sec 4.1 but am curious of how  to formally determine a \"ground truth\" type given an attack policy. As also claimed by authors, two attack policies can vary in parameter and behaviors, while a single defense policy may not be effective for both.\n\n5. In Proposition 3.2 the importance of generating diverse adversaries are proposed. However, in Eqn. 7, 8 and 9, I wonder which term corresponds to diversity. I assume it is related to \\theta_1 and \\theta_2 in Proposition 3.2? but numbers are different. What refers to 0 and 1 in subscripts with bracket?\n\n6. Clarity. This is not a big issue, but I would recommend using bullet points or explicit subsections to improve the organization of the paper. Currently some important informations and settings are hard to find in a glance.\n\n7.  Proposition 3.2 seems to be the difference between (1) and (2)? Guess this is a typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)