Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms
Authors: Alexander Bukharin
Yan Li
Yue Yu
Qingru Zhang
Zhehui Chen
Simiao Zuo
Chao Zhang
Songan Zhang
Tuo Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-Agent Reinforcement Learning
Theory of Robust Reinforcement Learning
Adversarial Regularization
TL;DR: We theoretically show we can control multi-agent policies' robustness through smoothness and provide a computationally stable algorithm, ERNIE, to do so.

Abstract: Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy’s Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Motivated by these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE’s adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE.

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms
Alexander Bukharin
Yan Li
Yue Yu!
Qingru Zhang
Zhehui Chen
Simiao Zuo
Chao Zhang
Songan Zhang
Tuo Zhao
Abstract
Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes their environment. This presents serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. this work we show that we can gain robustness by controlling policy's Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Based these insights, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE's adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. addition, we extend ERNIE mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code available https: /tops::/giiie ghtt com/ wwwaaaaaiun/ea////..
propose
robust MARL framework,
1
Introduction
In the past decade advances in deep neural networks and greater computational power have led to great successes for Multi-Agent Reinforcement Learning (MARL), which has achieved success wide variety of multi-agent decision-making tasks ranging from traffic light control [1] to StarCraft [2]. However, while much effort has been devoted applying MARL new problems, there has
been limited work regarding the robustness of MARL policies.
Despite the limited attention paid robustness, it is essential for MARL policies to be robust. Most MARL policies are trained in a fixed environment. Since these policies are trained solely to perform well in that environment, they may perform poorly in environment with slightly different transition dynamics than the training environment. In addition, while agents are fed with exact state information in training, MARL policies deployed in the real world can receive inaccurate state information (e.g., due to sensor error). Finally, even a single agent acting maliciously or differently than expected can cause chain reaction that destabilizes the whole system. These phenomena cause significant concern for the real-world deployment of MARL algorithms. where the environment dynamics and observation noise can change over time. We observe that even when the change the environment's dynamics is small, the performance of MARL algorithms can deteriorate severely (See an example
Georgia Institute of Technology
Google
Microsoft
Ford Motor Company
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in Section 5). Thus there is an emerging need for MARL algorithms that are robust to changing transition dynamics, observation noise, and changing behavior of agents.
Although many robust RL methods have been proposed for the single agent case, three major barriers prevent their for MARI.. Theoretically, it is not clear if or when such methods can work for MARL. Methooloogiccally, it is not straightforward to apply single agent robust RL methods to MARL, as single agent methods may not consider the interactions between several agents. Algorithmically, single agent robust RL algorithms are often unstable, and may not perform well when applied to inherently unstable MARL training. Therefore to learn robust MARL policies, we provide theoretical, methodological, and algorithmic contributions.
Theory. Theoretically, we first show that when the transition and reward function are smooth, policy's s value function is also smooth. In our experiments, we show that this assumption can serve as useful prior knowledge, even if the transition function is not smooth in every state. Second, prove that a smooth and close-to-optimal policy exists in any such environment. Third, we show that a policy's 's robustness is proportional its Lipchitz constant with no smoothness assumption on the environment' 's smoothness. These observations advocate for using smoothness as an inductive bias to not only reduce the policy search space, but simultaneously improve the robustness of the learned policy. Finally, we prove that large neural networks are capable of approximating the target policy or Q functions with smoothness guarantees. These findings give us the key insight that in order to learn robust and high- tigh-performing deep MARL policies, we should enforce the policies' smoothness.
Method. Based on these findings, we propose new training framework adv Ersarially Regularized multiageNt refnrerceeent |Earning (ERNIE), that applies adversarial training to learn smooth and robust MARL policies in a principled manner. In particular, to minimize the discrepancy between each policy's 's output given a perturbed observation and non- perturbed observation. This adversarial regularization gives two main benefits: Lipschitz continuity and rich data augmentation with adversarial examples. The adversarial regularization encourages the learned policies to be Lipschitz continuous, improving robustness. Augmenting the data with adversarial examples further provides robustness against environment changes. Specifically, new scenarios emerge when the environment changes, and data augmentation with adversarial examples provides a large coverage of these scenarios as long as the environment change is small. Adapting to adversarial examples during training ensures that the agents will perform reasonably even the worst case.
develop an adversarial regularizer
To further provide robustness against the changing behaviors of a few malicious agents, we propose an extension of ERNIE that minimizes the discrepancy between the global Q-function with maliciously perturbed joint actions and non-perturbed joint actions. This regularizer encourages the policies to produce stable outputs even when a subset of agents acts sub-optimally, therefore granting robustness. Such robustness has not been considered in previous works.
Algorithm. We find that adversarial regularization can improve robust performance [3]. However, ad- versarial regularization can also be unstable. More concretely, conventional adversarial regularization can be formulated as a zero-sum game where the defender (the policy) and attacker (the perturbation) hold equal positions and play against each other. In this case, a small change in the attacker's strategy may result in a large change for the defender, rendering the problem ill-conditioned. Coupled with the already existing stability issues that come with training MARL algorithms, this instability issue greatly reduces the power of adversarial regularization methods for MARL.
To address this issue, we reformulate adversarial training game, the leader (defender) has the advantage as it knows how the follower (attacker) will react to its actions and can act accordingly. This advantage essentially makes the optimization problem smoother for the defender, leading to a more stable training process.
Stackelberg game. In Stackelberg
Extension to Mean-field MARL. We further demonstrate the general applicability of ERNIE by developing its extension to robustify mean-field MARL algorithms. The mean-field approximation has been widely received as a practical strategy to scale up MARL algorithms while avoiding the curse of many agents [4]. However, as mean-field algorithms are applied to real-world problems, is essential to develop robust versions. To facilitate policy learning that is more robust, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization [5: 6].
To demonstrate the effectiveness of the proposed framework, we conduct extensive experiments that evaluate the robustness of ERNIE on traffic light control and particle environment tasks. Specifically,
2

---

we evaluate the robustness of MARL policies when the evaluation environment deviates from the training environment. These deviations include observation noise, changing transition dynamics, and malicious agent actions. The results show that while state-of-the-ar arrt MARL algorithms are sensitive to small changes in their environment, the ERNIE framework enhances the robustness of these algorithms without sacrificing efficiency.
Contributions. We remark that adversarial regularization has been developed for single-agent RL, but never for MARL [3]. Our contribution in this paper has four aspects: (1) advances in theoretical understanding (2) development of new regularizers for MARL (3) new algorithms for stable adversarial regularization in MARL (4) comprehensive experiments in number of environments.
2 Background
In this section, we introduce the necessary background for MARL problems together with related literature. We consider the setting of cooperative MARL, where agents work together to maximize a global reward.
Cooperative Markov Games We consider a partially observable Markov game (S, ON AAW, PP R. 7) in which set of agents interact within a common environment. We let S C I denote the global state space, O C Ro denote the observation space for each of the N agents, A S A denote the action space, P S x A + S denote the transition kernel, 7 denotes the discount factor, and R : S x A + IR denotes the reward function. At every time step t, each of the N agents selects an action according to its policy, which can be stochastic or deterministic. Then, the system transitions to the next state according to the transition kernel and each agent receives a reward Fi..- We denote the global reward at time as r.. The goal of each agent is to find a policy that maximizes the discounted sum of its own reward, Σ120 7 7rri,--
Robust RL. In recent years many single agent robust RL techniques have been proposed. Most of these methods use information about the underlying simulator to train agents over a variety of relevant environment settings [7; 8; 9; 10; 11]. Although these methods can provide robustness against a wide range of environment changes, they suffer from long training times and require expert knowledge of the underlying simulator, which is not practical. Another direction of research focuses on perturbation based methods [3: 12). Perturbation based methods train the policy to be robust to input perturbations, encouraging the policy to act reasonably in perturbed or previously unseen states. [13] certify robustness by adding smoothing noise to the state; is not clear how this affects the learned policy's 's optimality. Another related line of work [14; 15; 16: 17] studies robust markov decision processes and provides a principled way to learn robust policies. However, such methods Inspiring our work, [3] proposes to learn a smooth policy in single agent RL, but they do so to reduce training complexity rather than increase robustness and provide no theoretical justification for their method. Instead, we theoretically connect smoothness to robustness, extend perturbation based methods to MARL, and develop a more stable perturbation computation technique, and develop an extension to mean-field MARL.
often require strict assumptions on the perturbation/uncerree
Robust MARL. Recently, some works have studied the robustness of MARL systems. Lin et al. [18] studies how to attack MARL systems and finds that MARL systems are vulnerable to attacks on even a single agent. Zhang et al. [19] develop a framework handle MARL with model uncertainty by formulating MARL as a robust Markov game. However, their proposed method only considers uncertainty in the reward function, while this article focuses on robustness to observation noise and changing transition dynamics. Li et al. [20] modify the MADDPG algorithm to consider the worst-case actions of the other agents in continuous action spaces with the M3DDPG algorithm. M3DDPG aims to grant robustness against the actions of other agents, which is less general than the robustness against observation noise, changing transition dynamics, and malicious agents that our method aims for. Wang et al. [21] consider robustness against uncertain transition dynamics, but their algorithm is not applied to deep MARI. More recently, He et al. [22]; Han et al. [23] introduces the concept of robust equilibrium and proposes to learn adversarial policy to perturb each agent's 's observations. Finally Zhou et al. [24] propose to learn robust policies by minimizing the cross-entropy loss between agent's 's actions in non-perturbed states and perturbed states.
The ERNIE framework is also related to several existing works which use similar adversarial training methods but target different domains such as trajectory optimization [25]. semi- supervisd learn- ing [26; 27: 28], fine-tuning language models [29; 30], and generalization in supervised learning [31].
3

---

From Lipschitz Continuity to Robustness
This section presents the theoretical motivation for our algorithm by showing that Lipschitzness (smoothness) serves as a natural way to gain robustness, while reducing the policy search space. We start by observing that certain natural environments exhibit smooth transition and reward functions, especially when the transition dynamics are governed by physical laws (e.g., MuJuCo environment [32], Pendulum [33).* Formally, this stated as the following.
Definition 3.1. Let S _ IR We say the environment is (L,, Lp)-smooth, if the reward function r: S x A + R, and the transition kernel IP: S x S x R satisfy
|r(s, a) r(s',a)) < L, ||s s'1l and [IP(-|s, a) P(-ss', (()ss',))l1+ S Le ||s s'1,,
for (s, 8)) x 8 A. ||-ll denotes metric on Rd We say policy π is Lm-soohth if [|<(+ 7 (-1s')l L || s'll.
Without loss of generality, we assume (r(s, a)1 - for any (s. S A. We then present our theory. Due to the space limit, we defer all technical details to the appendix.
From smooth environments to smooth values. We proceed show that if the environment is smooth, then the value functions for smooth policies are also smooth.
Theorem 3.1. Suppose the environment is (Iey, Ism-shooth. Then the Q-function of any policy n defined as
as, D 0s) s, ao - a], V(s,a,,
Lipschitz continuous in the first argument. That is,
a) - Q('', a)l S LQ |s s'll
(1)
= L, + LLp 7). Suppose in addition the policy is smooth.. Then the value function, defined as
where LQ=
v (8) = E. 1E. 711( az) = =] vs.
Lipschitz continuous.
That is,
(V* - V* 1ss-
7) where Lv = (1) Lq.
In view of Theorem 3.1, it is clear that whenever the environment and the policy are smooth, then the value functions are also smooth. A natural and important follow-up question to ask is whether this claim holds in the reverse direction. More concretely, we ask whether it reasonable seek policy that is also smooth with respect to the state while maximizing the reward. If the claim holds true, then seeking a smooth policy can serve as an efficiett and unbiased prior knowledge, that can help us reduce the policy search space significantly, while still guaranteeing that we are searching for pegforeeffrming policies.
Existence of smooth and nearly-optimal policies. The following result shows that for any >> 0, there exists an e-optimal policy that is O (LQ/E)) smooth, where LQ defined in Theorem 3.1 only depends on the smoothness of reward and transition. This structural observation naturally suggests seeking a smooth policy for smooth environments.
Theorem 3.2. Suppose the environment is (Lr, -mmoohh Then for any 0, there exists e-optimal policy π. that is also smooth, i.e.,
2r V* I Ws E S and [|((|()) - S [Allog|A - s'1l/e,
where Lo is defined as in Theorem 3.1.
Notably, the proof of Theorem 3.2 relies on the key observation that any smooth Q-function satisfying (I) can be fed into the softmax operator, which induces a smooth policy. This observation also provides way for value-based methods (e.g.. Q-learning) learn a smooth policy. Namely, one can first learn a smooth surrogate of the optimal Q-function, and then feed the learned surrogate into the softmax operator to induce close-to-optinee oo-tmtl policy that also smooth.
See Remark 3.2 for discussions when the smoothness property holds approximately.

---

Robustness against observation noise with smooth policies. We have so far established that smooth policies naturally exist in smooth environment as close-to-optimal policies, and thus smoothness serves as strong prior for policy search. We will further demonstrate that the benefits of smooth policy go beyond boosting learning efficiency, by bringing the additional advantage of robustness against observation uncertainty.
Theorem 3.3. Let ((as)) ) be -mmooth policy. For any perturbation sequence {6%) define perturbed policy (non- mon-stttionary) के = (me) by
{a = [[ass- - 8%),
with |5:|| S for all t 2 0. Accordingly, define the value function of the non-stationary policy म
v* (8) =E =E -i 7 az)1 = ar ~ Te(.18t 18c + 8% 0%,),
st+1 ~ P( ae)]
Then have (s) (6))-17 V3(())
2het (I-7)z Similarly, we have
(5 a) - (s, (1-7)²,
2h.e
where Of is defined similarly as V*
Theorem 3.3 establishes the following fact: for discounted MDP with finite state and finite action space, the value of the policy when providing the perturbed state is close to the value of the policy when given the non-perturbed state, provided the policy Lipschitz continuous in its state. As an important implication, the learned smooth policy will be robust in the state observation, in the sense that the accumulated reward will not deteriorate much when noisy, or even adversarially constructed state observations are given to the policy upon decision making.
We emphasize that Theorem 3.3 holds without any smoothness assumption the transition or the reward function. It should also be noted that there are various notions of robustness in MDP, e.g., robustness against changes in transition kernel [34; 17], which we defer as future investigations.
Before we conclude this section, we briefly remark on certain generality of our discussion. Remark 3.1 (Applicability to MARL). The discussions in this section do not depend on the size of the state space, and apply to the multi-agent setting without any change. To see this, note that our discussion holds for any discrete acttinaction space. Setting S as the joint state space and A as the joint action space, then the obtained results trivially carry over to the cooperative MARL setting. Remark 3.2 (Environments with approximate smoothness). Many environments are partially smooth, in the sense that the transition or the reward is non-smooth only on small fraction of the state space. Typical examples include the Box2D environment [33], where the agent receives smooth reward when in non-termin ao-termmiall states (airborne for Lunar Lander), and receives a lump-sum reward in the terminal state (land/crash) - a vanishing fraction of the entire state space. Given the environment being largely smooth, it should be expected that for most states the optimal policy is locally smooth. Consequently, inducing a smoothness prior serves as a natural regularization to constrain the search space when solving these environments, without incurring large bias.
Remark 3.3 (Non-smooth environments). From the perspective of robust statistics, achieving robustness often necessitates a certain level of smoothness in the learned policy, regardless of the smoothness of the optimal policy. scenarios where the environment itself non-smooth, the optimal policy can also be non-smooth. However, it is important to note that such non-smooth optimal policies are typically not robust. This means that by trading-off between the approximation bias and robustness, the smooth policy learnt by out method has the potential to outperform non-smooth policies perturbed environments.
4 Method
In this section, we propose our robust MARL framework, advErsarially Regularized multiageNt reInforcement |Earning (ERNIE).
4.1 Learning Robust Policy with ERNIE
Section 3 shows that the robustness of policy depends on its Lipschitz constant. Therefore, in ERNIE we propose to control the Lipschitz constant of each policy with adversarial regularization.
5

---

Given policy nB2, where l is the agent index, the ERNIE regularizer is defined by
Rx 0x) = max (TTO, (ox + 6), To, (ox)).
||
Here 8(0g, 0k) is perturbation adversarially chosen to maximize the difference between the policy's 's output for the perturbed observation оą + 8(0,, 0x) and the original observation og- In this case e controls the perturbation strength and || usually taken be the lg or loo norm. Note that (og; 0&) essentially measures the local Lipschitz smoothness of policy function around the observation og, defined in metric r). Therefore minimizing R, (og: 0k) will encourage the policy to be smooth.
Regularization (2) allows straightforward incorporation into MARL algorithms that directly perform policy search. For actor-critic based policy gradient methods, the regularizer (2) can be directly included into the objective for updating the actor (policy) networks. When optimizing stochastic policies (e.g.. MAPPO [35]). D can be taken to be the KL divergence and for deterministic policies (e.g., MADDPG [36] or Q-learning [37]), we set D be the lp norm.
More concretely, let C() denote the policy optimization objective, i.c., the negative weighted value function of the policy. We then augment (()) with (2), and minimize the regularized objective
min F(0) = (()) + > m= ET,,, [R (on; 0n)]
9
where À is hyperparameter. We remark that Shen et al. [3] has explored similar regularization for single-agent RL (with a goal of improving sample efficiency), but as we explain in sections 4.2, 4.3, and 4.4, successful application to MARL robustness highly non-trivial.
4.2 Stackelbeg Training with Differentiable Adversary
Although accurately solving (3) will result in high-performing and robust policy, we note that (3) nonconvex-nonconee minimax problem. In practice, we can use multiple steps of projected gradient ascent to approximate the worse-case state perturbation 8(ox, 02), followed by one-step gradient descent for updating the policies/Q-funen funnctinn.. Even though this optimization method already significantly improves robustness over the baseline algorithmss, we found that the training process could be quite unstable. We hypothesize that the intrinsic instability of MARL algorithms due to simultaneous updates of multiple agents is greatly amplified by the non-smooth landscape of adversarial regularization.
To promote more stable straining process, we propose to reformulate adversarial training in ERNIE a Stackelberg game. The reformulation defines adversarial regularization leader-follower game [38]:
R 8 (o): = (Too(o (o (o,0)),
s.t. 8 (o, = Up o Up o (o,
K-fold composition
Here denotes the operator composition (i.e o f(g(-))) and
8k+1 0) (6 0)) 8 + vVD (To (To (o (o,0)) to(o))
is a one-step gradient ascent for maximizing the divergence of the perturbed and original observation.
Compared to the vanilla adversarial regularizer in (2), the perturbation 8 is treated the model parameter This formulation allows the leader (0) to anticipate the action of the follower (8), since the follower's 's response given observation is fully specified by 8 (o, 0). This structural anticipation effectively produces an easier and smoother optimization problem for the leader (0), whose gradient, termed Stackelberg gradient, can be readily computed by
function of
rR (o, 8% (0):0) 2R (o, 8K 0)
ŌRa (o, 8K (0), 0) 8 (0) 05K (0)
00
00
30
leader
lender-follower interaction
Note that the gradient used in (3) only contains the "leader" term, such that interaction between the model 0 and the perturbation is ignored. The computation of the Stackelberg gradient can be reduced to Hessian vector multiplication using finite difference method [39], which only requires two

---

backpropogations and extra (dd) complexity operation. Thus no significant computational overhead introduced for solving (4).
The benefit of Stackelberg training for MARL is twofold. First, a smoother optimization problem results in more stable training process. This extra stability is essential given the inherent instability of MARL training. Second, giving the policy 0 priority over the attack during the training process allows for a better training data fit than normal adversarial training allows. This better fit allows the MARL policies trained with Stackelberg training to perform better in lightly perturbed environments than those trained with normal adversarial regularization.
4.3 Robustness against Malicious Actions
Given the complex interactions of agents within of a multi-agent system, a robust policy for any given agent should meet the criterion that the action made is not overly dependent on any small subset of agents. This is particularly the case when the agents are homogeneous in nature [4; 40], and thus there should be no notion of coreset agents in the decision- decision-making process that could heavily influence the actions of other agents. We proceed to show how ERNIE could be adopted to induce such notion of robustness.
The core idea of ERNIE for this scenario is to encourage policy/Q-function smoothness with respect to joint actions. Similar to our treatment in (2), we now seek to promote learning a Q-function that yields consistent value when perturbing the actions for any small subset of agents. Specifically, for discrete action space, we define regularizer the global Q-function as
(s.a)
max [1Q(ss, Q(s, wB D(a.a')
(5)
where D(a, a') E. (aa; e a'). The regularizer (5) seeks to compute the worst subset of changed actions with cardinality less than K. For continuous action spaces, one could replace the metric D in (5) by a differentiable metric defined over the action space (e.g.. 11-1|2-nore -nomm) and then evaluate the regularizer with projected gradient ascent.
To evaluate the adversarial regularizer for the discrete action space, we propose to solve (5) in greedy manner by finding the worst-case change in the action of a single agent at time, until the action of K agents is changed. Specifically, at each training step, we search through all the agents/actions and then pick the actions that produce the top-K changes in the Q-function, resulting in O[IA] N K) computation. Our complete algorithm can be found in Appendix H, and we find that in our numerical study, perturbing the action of single agent (K 1) is sufficient for increased robustness.
Similar to the regularizer in (2), the regularizer in (5) provides the benefits of Lipschitz smoothness (with respect to the Hamming distance) and data augmentation with adversarial examples. If the behavior of a few agents changes (either maliciously or randomly), the behavior of policies trained by conventional methods may change drastically. On the other hand, policies trained by our method will continue to make reasonable decisions, resulting in more stable performance (see section 5).
4.4 Extension to Mean-field MARL
MARL algorithms have been known to suffer from the curse of many agents [4), as the search space of policies and value functions grows exponentially w.r.t. the number of agents. A practical approach to tackle this challenge of scale is to adopt the field approximation, which views each agent as realizations from distribution of agents. This distributional perspective requires distinct treatment of ERNIE applied to the mean-field setting.
Mean-field MARL avoids the curse of many agents by approximating the interaction between each agent and the global population of agents with that of an agent and the average agent from the population. In particular,
approximate the action-value function of agent j as Q' a)
Q' (s}, de, aj. a,), where a is the global joint action, s is the global state, a, is the average action of agent j's neighbors, and d, is the empirical distribution of states over the population. Such an approximation has found widespread applications in practical MARL algorithms [41: 42; 40; 43], and can be motivated in principled fashion for agents of homogeneous nature [4; 40].
To learn robust and scalable policies, we extend ERNIE to the mean-field setting by applying adversarial regularization to the approximation terms d, and ă². It is important to note that as the terms de and d', represent distributions over states, we bound the attack by the Wasserstein distance [44]. In what follows we only apply the regularizer to d, for simplicity. This leads to new regularizer
7

---

defined over the mean field state
R (s; 0) =
max E 1100(s, Q0(8, Wid d. aEA
(8, ds, a) a)l
where W denotes the Wasserstein distance metric. Since the explicit Wasserstein constraint may be difficult optimize in practice, we can instead enforce the constraint through regularization, as displayed in Appendix C.
5 Experiments
We conduct extensive experiments to demonstrate the effectiveness of our proposed framework. In each environment, we evaluate MARL algorithms trained by the ERNIE framework against baseline robust MARL algorithms. To evaluate robustness, we train MARI policies in a non-perturbed environment and evaluate these policies in a perturbed environment. The reported results are gathered over five runs for each algorithm. Given the space limit, we put additional results in Appendix E.
Traffic Light Control. In this scenario, cooperative agents learn to minimize the total travel time of cars in a traffic network. We use QCOMBO [45] and COMA [46] (results in appendix E) as our baseline algorithms and conduct experiments using the Flow framework [47]. We train the MARL policies on a two-by-two grid (four agents). We then evaluate the policies on a variety of realistic environment changes, including different car speeds, traffic flows, network topologies, and observation noise. In each setting, we plot the reward for policies trained with ERNIE, the baseline algorithm (QCOMBO), and another baseline where the attack 5 is generated by Gaussian random variable -aaussa,, see Appendix 1). Implementation details can found in Appendix G.
Figure 1, 2a, and 2b show that the baseline algorithm is vulnerable to small changes in the training environment (higher reward is better). On the other hand, ERNIE achieves more stable reward on each environment change. This observation confirms that the ERNIE framework can improve robustness against observation noise and changing transition dynamics. The Gaussian baseline performs well on some environment changes, like when the observations are perturbed by Gaussian noise, but performs poorly on other environment changes, like when the car speed is changed. We hypothesize that while some environment changes may be covered by Gaussian perturbations, other environment changes are unlike Gaussian perturbations, resulting a poor performance from this bascline.
Robustness Against Malicious Actions. We also evaluate the extension of ERNIE to robustness against changing agent behavior, which refer as ERNIE-A. To change agent behavior, adversarially change the action of randomly selected agent a small percentage of the time, i.e. 5% or 3% of the time. As can be seen in Figures 2c and 2d, the two baseline algorithms perform poorly when some agent's ' behavior changes. In contrast, ERNIE-A is able to maintain a higher reward.
ERNIE
Passiin Gaisisian
-200
X0E THE 2000 200 so1 200 1200
Lid0 LM0) 2010 2006 X00 200 2000
1 7500 2000 2000
(a) Gaussian Noise (0.01)
(b) Different Speed
(c) Traffic Flow (Flow-3)
Figure I: Evaluation curves on different environment changes for traffic light control.
Particle Environments.
We evaluate ERNIE on the cooperative navigation, predator-prey, tag, and cooperative communication tasks. In each setting, we investigate the performance of the baseline algorithm (MADDPG), ERNIE, M3DDPG, and the baseline- gaussian in environments with varying levels of observation noise. We also compare ERNIE to the RMA3C algorithm proposed in [23]. In Figure 3, we find ERNIE performs better or equivalently than MADDPG in all settings. Surprisingly, M3DDPG can provide some robustmess against observation noise, even though it is designed to provide robustness against malicious actions.
baseline-gaussian
Mean-field MARL. We evaluate the performance of the mean-field ERNIE extension on the co- operative navigation task [36] with different numbers of agents. We compare the performance of the baseline algorithm, ERNIE, and M3DDPG under various levels of observation noise. We use
8

---

ERNIE
ERNIE
Crantinn-Gaussiann
Eascline
Baseline
soe noe
1800 180 2000 2000 3013
1M0 200 2003 2000
(a) Larger Network
(b) Different Topology
(c) Actions (3%)
(d) Actions (5%)
Figure 2: Performance on changed traffic network topologies and with malicious agents. Figures (c) and (d) we perturb the actions according to the specified percentages.
Newaad
Baseline-Gen
RMAII
-320
-34
0.8
(a) Covert Comm.
(b) Tag
(c) Navigation
(d) Predator Prey
Figure 3: Training reward versus noise level (c) in the evaluation environment for the particle games.
mean-field MADDPG as baseline and follow the implementation of [40]. As can be seen in Figure 4, ERNIE displays a higher reward and slower decrease in performance across noise levels.
110
CRNIC-MF Mean Field N3DDPG
-1.0
-200
-2300
-200
-20
-240
04
-290
125
-805
-90
40.0
01
43.00
lgg
l0g
log
(a) N=3
(b) 9
(c) N=15
(d) Model Width
Figure 4: (c)-(c) Training reward versus noise level (mean standard deviation over runs) with various number of agents (N) (d) Network width and robustness.
LRNIE
Parcorsl
Pacork
200
3011
1000 4.0
200
201
40
(a) Different Speed
(b) Observation Noise
(c) Sensitivity to
(d) Sensitivity to K
Figure 5: Sensitivity and ablation experiments.
Hyperparameter Study. We investigate the sensitivity of ERNIE to the hyperparameters K (the number of attack steps) and (the perturbation strength). We plot the performance of different hyperparameter settings in the traffic light control task with perturbed seeds. From Figures and 5d we can see that adversarial training (K 0) outperforms the baseline (K 0) for all K. Similarly, indicates that ERNIE is robust to different hyperparameter settings and K.
speeds three random
can that all values of outperform the baseline (e 0). This
Sensitivity and Ablation Study. The advantage of the ERNIE framework goes beyond improving mean reward. show this, we evaluate 10 different initializations of each algorithm in two traffic environments: one with different speeds and another environment with observation noise. We then sort the cumulative rewards of the learned policies and plot the percentiles Figures Sa and 5b. Although the best-case performance is the same for all algorithms, ERNIE significantly improves the

---

robustness to failure. As an ablation, we evaluate the effectiveness of ERNIE with and without the Stackelberg formulation of adversarial regularization (ST). ERNIE displays better performance in both settings, indicating that Stackelberg training can lead to more stable training process. Ablation experiments in other environments can be found Appendix E.4.
Robustness and Network Width. In Appendix B, we show that in order to learn a robust policy with ERNIE, we should use sufficiently wide neural network. Therefore, we evaluate the robust performance of ERNIE using policy networks with 32, 64, 128, and 256 hidden units. We carefully tune their regularization parameters such that all networks perform similarly in the lightly perturbed environment. As seen in Figure 4d, when the perturbed testing environment deviates more from the training environment, the performance of the narrower policy networks (32/64 hidden units) significantly drops, while the wider networks (128/256 hidden units) are more stable. We also observe that when the policy networks are sufficiently wide (128/256), their robustness similar.
Robotics Experiments. Additional experiments multi-agent drone control environments can be found in Appendix E. which further verify the enhanced robustness that ERNIE provides.
6 Discussion
ERNIE is motivated by smoothness, but real-world environments are not always smooth. In section we hypothesize that most environments are least partially smooth, implying that smoothness can serve as useful prior knowledge while providing robustness (our experiments validate this). To increase ERNIE's flexibility, future work could adaptively select based on the current state allow for state-dependent smoothness.
References
[1] Marco A Wiering. Multi-agent reinforcement learning for traffic light control. In Proceedings of the Seventeenth International Conference Machine Learning, pages 1151-1158, 2000.
[2] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
[3] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning. pages 8707-8716 88. PMLR, 2020.
[4] Lingxiao Wang, Zhuoran Yang. and Zhaoran Wang. Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning. In International Conference Machine Learning, pages 10092-10103. PMIR, 2020.
[5] Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):995-62,, 2010.
[6] Joel Goh and Melvyn Sim. Distributionally robust optimization and its tractable approximations. Operations research, 58(4-part-1)9902-91 1):9019917, 2010.
[7] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2): 335-359, 2005.
[8] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference Machine Learning, pages 2817-2826.
PMLR, 2017.
[9] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning, arXiv preprint arXiv:1907.1s. 13 3166, 2019.
[10] Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforce- ment learning. In 2019 International Conference Robotics and Automation (ICRA), pages 8522-8528. IEEE, 2019.
[11] Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International Conference Machine Learning, pages 23484-23526. PMLR, 2022.
10

---

[12] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:210244 2020.
[13] Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openrevier http://pperreeeee..eefrrtmmm http:/orrmmmmmmdddaiiaeaae iddmdddai@NBBB:
[14] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
[15] Arnab Nilim and Laurent EI Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-779) 2005.
[16] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust rein- forcement learning using offline data. Advances in neural information processing systems, 35: 32211-32224, 2022.
[17] Yan Li, Tuo Zhao, and Guanghui Lan. First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2209.10579. 2022.
[18] Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW), pages 62-68. IEEE, 2020.
[19] Huan Zhang, Hongge Chen, Duane $ Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference Learning Representations, 2021. URL
[20] Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4213-4220, 2019.
[21] Yudan Wang, Yue Wang, Yi Zhou, Alvaro Velasquez, and Shaofeng Zou. Data-driven robust multi-agent reinforcement learning. In 2022 IEEE 32nd International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6, 2022.
[22] Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.
[23] Songyang Han, Sanbao Su, Sihong He, Shuo Han, Haizhao Yang. and Fei Miao. What is the solu- tion for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02270: 2022.
[24] Ziyuan Zhou, Guanjun Liu, and Mengchu Zhou. robust mean-field actor-critic reinforce- ment learning against adversarial perturbations on agent states. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[25) Zhigen Zhao, Simiao Zuo, Tuo Zhao, and Ye Zhao. Adversarially regularized policy learning guided by trajectory optimization. In Proceedings of The 4th Annual Learning for Dynamics and Control Conference, pages 844-857. PMLR, 23-24 Jun 2022. URL http:: https::/proceedingss html.
mlr.preste wwtppp/ssss///aaepeeeeeee
[26] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 1189):197-11993, 2018.
[27] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances neural information processing systems, 32, 2019.

---

[28] Simiao Zuo, Yue Yu, Chen Liang, Haoming Jiang, Siawpeng Er. Chao Zhang, Tuo Zhao, and Hongyuan Zha. Self Self-training with differentiable teacher. In Findings of the Association for Computational Linguistics: NAACI. 2022, pages 933-949, Seattle, United States, July 2022.
Association for Computational Linguistics.
[29] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao, SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguisties, pages 2177 2177-2190, Online, July 2020. Association for Computational Linguistics.
[30] Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pre- trained language model with weak supervision: A contrastive-regulaiza self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1063-1077, Online, June 2021. Association for Computational Linguistics.
[31] Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, and Tuo Zhao. Adversarial regularization as stackelberg game: An unrolled optimization approach. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6562-6577, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[32] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 JEEF/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEFE, 2012.
[33] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. 2016.
[34] Andrzej Ruszczyíski. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125(2):235-266 2010.
[35] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems,
[36] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiive environments. Advances in neural information processing systems, 30, 2017.
[37] John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation. Advances neural information processing systems, 9, 1996.
[38] Heinrich Von Stackelberg. Market structure and equilibrium. Springer Science Business Media, 2010.
[39] Barak A Pearlmutter and Jeffrey Mark Siskind. Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Transactions Programming Languages and Systems (TOPLAS), 30(2):1-36, 2008
[40] Yan Li, Lingxiao Wang. Jiachen Yang, Ethan Wang, Zhaoran Wang, Tuo Zhao, and Hongyuan Zha. Permutation invariant policy optimization for mean-field multi-agent reinforcement learning: A principled approach. arXiv preprint arXiv:2105.08260 2021.
[41] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5571-5580. PMLR, 2018
[42] Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang. Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In The World Wide Web Conference, pages 983-994, 2019.
12

---

[43] Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Mean-field multi- reinforcement learning: A decentralized network approach. arXiv preprint arXiv:2108.0273: 2021.
[44] Ludger Rüschendorf. The wasserstein distance and approximation theorems. Probability Theory and Related Fields, 70(1):117-122) 1985.
[45] Zhi Zhang, Jiachen Yang, and Hongyuan Zha. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv preprint http:/999.1051,, 2019.
[46] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[47] Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Hugene Vinitsky, and Alexandre Bayen. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint 17X:5:1710.05465, 10, 2017.
[48] Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep relu neural networks in w s. p norms. Analysis and Applications, 18(05):803-88 859, 2020.
[49] Haim Brezis and Haim Brézis. Functional analysis, Sobolev spuces and partial differential equations, volume 2. Springer, 2011.
[50] Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact wasserstein distance. Uncertainty in Artificial Intelligence, pages 433 PMLR, 2020.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:142..6980, 2014.
[52] Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela Schoellig. Learning to fly-a gym environment with pybullet physics for reinforcement learning of multi- agent quadcopter control. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7512-7519. IEEE, 2021.
[53] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[55] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, al. [TensorFlow): ): system for (Cale) machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265-283, 2016.
13


Reviews:
[{"review_id": "V9LKkTb72V", "reviewer": "Reviewer_mUHD", "summary": "This paper proposes a method to train robust policies in a multi-agent reinforcement learning setting via adversarial regularization. The authors showed that robust policies can be achieved by enforcing Lipschitz continuity with respect to states for robustness to noisy states and with respect to actions for noisy action by other potential agents. To enforce the Lipschitz continuity, the authors proposes an adversarial regularization term that can be added to conventional RL training objectives, which minimizes the maximum distance between the policy under a perturbed state/action and the policy under the original state/action. The authors also reformulated the minimax problem into a Stackelberg game, which further stabilize the training process. Finally, the authors also extended their adversarial regularization to the setting of mean-field MARL, which enables the training of large-scale robust marl agents. Experiments and results were provided for several MARL benchmarks. ", "questions": "I do not have any major questions that requires clarification.\n\nMinor comments/questions:\n\"Wang et al. [16] consider robustness against uncertain transition dynamics, but their\nalgorithm is not applied to deep MARL.\". This sentence really fall under the subsection of robust RL rather than robust MARL then.\n\nIn the supplementary material:\nFigure 6: What does STELLA represent? I am assuming its a typo.\n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a well-written study on a novel approach to train robust multi-agent RL policies. While the core idea of using adversarial regularization may not be entirely novel, its application to multi-agent systems and its extension to mean-field RL is an interesting, albeit expected direction of development. The authors also provided relatively comprehensive experimental results, and the discussions and remarks effectively addressed most of my major questions that I thought of when reviewing the paper. In terms of significance, I believe that this work will further the development of robust MARL systems. \n", "weaknesses": "Overall, I believe the paper could benefit further if the authors could also provide additional experiments in the extension to mean-field MARL where the authors also apply the regularizer to the action to validate their claims for the sake of completeness of experiments. In addition to that, it will also be beneficial if the authors would also provide the run time comparisons for the traffic flow environments on top of the MPE environments for completeness. Finally, if I am understand correctly, both of the experiments are employed a reward structure that is continuous and smooth. It will be more convincing if the authors could also demonstrate the effectiveness of their proposed framework on an environment with an approximately smooth vs non-smooth reward structure. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mUHD, \n\nThank you for the constructive comments! We provide responses to your critiques and questions below.\n\n- **Weakness 1: ERNIE on mean-field actions.**\nThank you for the suggestion. The results can be seen in Figure 1 of the rebuttal material. We find that applying ERNIE to  mean-field actions does not significantly improve performance. We believe this is the case since we evaluate the algorithm in a cooperative environment, so robustness against other agent actions is not very necessary. As future work, we would like to evaluate ERNIE applied to mean-field actions in competitive (instead of cooperative) environments.\n\n- **Weakness 2: Time comparison on traffic light control.**\nHere are the results: ERNIE takes 0.022 seconds per iteration and the baseline takes 0.023 seconds per iteration.\n\n- **Weakness 3: Evaluation in approximately-smooth environments.**\nWe remark that in traffic light control, the reward function is only partially smooth. This is because the reward is the sum of various categorical variables (i.e. if there was a sudden car stop and the light changed or not). Despite this, ERNIE can still boost performance. This supports our claims that ERNIE can perform well in approximately smooth environments.\n\n- **Minor Remark: Discussion of [16].**\nThank you, we will move the sentence to the appropriate section.\n\n- **Stella typo.**\nSorry about that, it is a typo! We will fix this in the future versions.\n\n"}, {"review_id": "kRuMZcm3FB", "reviewer": "Reviewer_NyS9", "summary": "This paper proposes to develop a robust MARL framework based on Lipschitz smoothness of the learned policy (smooth policies). The authors prove the existence of smooth and nearly-optimal policies and propose a framework (ERNIE) to learn robust MARL policy. Experiments are conducted on the two environment: Traffic light control and particle environment. \n\n", "questions": "1. Is the description of Line 48 correct? Is the robustness proprotional to Lipschitz constant or inverse proportional? \n\n2. It is proved that a smooth and close-to-optimal policy exists in smooth enviornment (Line 47-48), but is it guaranteed to obtain such a close-to-optimal policy? and how? \n\n3. Line 215-216 - the boldface and captilaized words does not correspond to ERNIE. \n\n4. Fig 3: the ERNIE's result in Fig 3 seem to be not good, only (a) seem to be outperforming baselines. \n\n5. It's still not clear to me how can the proposed framework be extended beyond smooth environment. Please provide a few example environments that is practically non-smooth and how this framework might be extended?\n\n=================== \nincrease rating from 4 to 5 in the post-rebuttal. ", "limitations": "The authors have a paragraph discussing potential future work of the method, but not necessarily the limitation. \n", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Robust MARL is an important problem to be addressed\n", "weaknesses": "There seems to be many contents squeezed in this paper: theory, framework, connection to game theory (stacklebeg training) and evaluations. It will be nice to have an overview and outline at the end of Intro. \n\n1. This paper also aims to tackle a wide range of threat models if I understand correctly: state perturbation, transition perturbations etc. But why some experiments are evaluating action perturbations? \n\n2. As discussed in Lin 140-141, transition dynamics governed by physical laws (e.g. MuJoCo environment) are of interest. But why in the Experiments, the authors did not show the result of MARL MujoCo Benchmark? \n\n3. Is the ERNIE framework primiarily designed to tackle the threat model of state-perturbation? See Eq 2 of ERNIE regularizer. How can it be used to tackle the transition perturbation threat model? \n\n4. The result in Theorem 3.1 is natural when everything is smooth. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NyS9, \n\nThank you for constructive comments! We believe the weaknesses you pointed out are addressed by our rebuttal. In particular, we make the following changes: (1) Clarify the experimental setting for action robustness (2) Adding new experiments in robotic simulation tasks (3) Clarifying how ERNIE can address the transition perturbation threat. \n\n- **Weakness 0: Include overview and outline at end of introduction.**\nThank you for this suggestion. We will add an outline/overview to the end of the introduction.\n\n- **Weakness 1: Why are some experiments evaluating action perturbation?**\nIn section 4.3 (Line 267), we introduce a version of ERNIE designed to induce robustness against the changing behavior/actions of other agents. To evaluate this version of ERNIE, we conduct experiments where some of the agent's actions are perturbed. We will further clarify the experimental setting and algorithm used in line 336.\n\n- **Weakness 2: Why did we not use the MARL MuJoCo environments?**\nThank you for the suggestion. To alleviate your concern, we have evaluated ERNIE in the PyBullet (an open-source implementation of MuJoCo) multi-agent drones environment, which also has transition dynamics governed by physical laws. We use this environment over MuJoCo due to compatibility with our hardware. The results can be seen in Figure 3 of the rebuttal material. In this environment ERNIE is shown to be much more robust than the baseline, further confirming the efficacy of ERNIE.\n\n- **Weakness 3: How does ERNIE tackle the transition perturbation threat model?**\nIndeed, ERNIE is primarily designed to tackle the threat model of state-perturbation. However, as we show empirically, ERNIE can also help to tackle the transition perturbation threat model (See Figures 1 and 2). We believe ERNIE provides robustness against transition perturbations due to the data-augmentation that happens when we add adversarially perturbed examples to the training set. Specifically, new scenarios emerge when the environment changes, and data augmentation with adversarial examples provides a large coverage of these scenarios as long as the environment change is small. Adapting to adversarial examples during training ensures that the agents will perform reasonably even in the worst case. Such reasoning can be found in line 61.\n\n- **Weakness 4: Theorem 3.1 holds in smooth environments.**\nIt is true that Theorem 3.1 only holds in smooth environments. However, we remark that smoothness is often a good approximation. This can be observed by the good performance of ERNIE in the traffic light control setting, where the reward is not smooth (it is the composition of several categorical functions).\nFurthermore, we remark that theorem 3.3 does not have any smoothness requirement! This suggests that ERNIE can add robustness even in non-smooth environments (see line 191). \n\n- **Question 1: Is lipschitzness proportional to robustness?**\nYou are correct here, it should be inversely proportional. We will change it in the next version.\n\n- **Question 2: Are we guaranteed to achieve a close-to-optimal and smooth policy?**\nCurrently, we cannot prove convergence to the close to optimal policy. This would be quite difficult due to the difficulty of neural network training and non-stationarity of RL training. However in practice, we find that ERNIE is able to find a policy that is high-performing and fairly smooth (as evidenced by robustness to input perturbations). This indicates ERNIE is an effective way to search for smooth and close to optimal policies.\n\n- **Question 3: ERNIE spelling error.**\nWe apologize for this mistake, it will be fixed in the next version.\n\n- **Question 4: Bad performance in Figure 3.**\nNote that ERNIE shows good performance in 3(a) and 3(c). However for 3(b) and 3(d), the baseline policies are already fairly robust (performance doesn’t decrease with increasing perturbation), possibly due to structure of task. Therefore ERNIE cannot show superior performance than the baseline, but performs no worse than the baseline. In essence, we believe that robustness does not matter much on this task, and therefore ERNIE does not show much improvement.\n\n- **Question 5: Extension beyond a smooth environment.**\nThere are two ways ERNIE could be extended beyond smooth environments (1) First, ERNIE can straightforwardly be applied to smooth environments with no modification. Although this will result in some approximation errors, the gains in robustness may be worth the added approximation error. An example of this is traffic light control, where the reward function is only partially smooth. This is because the reward is the sum of various categorical variables (i.e. if there was a sudden car stop and the light changed or not). Despite this, ERNIE can still boost performance. This supports our claims that ERNIE can perform well in approximately smooth environments. (2) In addition, we could extend ERNIE to non-smooth environments by using state-adaptive regularization. In particular, we could choose a smaller $\\lambda$ in states where the environment is not smooth. We can identify these states by looking at the scale of the ERNIE regularizer (if it is large, then the state is likely non-smooth). We look forward to investigating such a strategy in future work.\n\n- **Limitations:**\t\nWe will add a paragraph on the limitations. In particular, we will discuss the requirement of a smooth or approximately smooth environment, the additional train time of ERNIE, and the fact that ERNIE does not necessarily solve all robustness issues.\n\n\n\n\n"}, {"review_id": "DQWPj6YRmc", "reviewer": "Reviewer_J2jx", "summary": "Robust RL is attracting great attention, and lately there has been great advancements both in theory and empirically. However, there is only limited work in robust multi-agent reinforcement learning. Under some assumptions, the paper shows the existence of policies with nice properties. Building upon this, the authors propose a training framework that includes adversarial perturbation on observation in order to increase robustness of the learned policy. ", "questions": "1. I find that calling lipschitzness smoothness quite confusing. In particular, in Definition 3.1, the \"smoothness\" is actually defined as the reward and dynamics being $L_r,L_{\\mathbb{P}}$-Lipschitz. Subsequently, in the writing, there are many places where the authors interchange the smoothness and lipschitzness, e.g., line 187-188. As far as I know, more commonly smoothness is about the gradient of a function being Lipschitz. Is there a strong reason for this particular choice of words?\n1. This recent work [1] considers a Markov game with state perturbation adversaries. How does it compare to your work?  \n[1] Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.\n1. What will be the main difficulty in extending ERNIE to continuous state and action setting?\n1. The main idea of making the learned policy Lipschitz is somewhat similar to the idea of smoothening the decision boundary. I would like to think this as the robustness in the \"average\" case rather than the worst case. Is there any work in the MARL literature that considers the worst case robustness?", "limitations": "1. In the paper survey section regarding the single-agent robust RL, I think the authors should include another important line of work: robust Markov decision process (RMDP). This distributionally robust reinforcement learning (DR-RL) framework provides a principled way to achieve robustness against perturbations in models. For example, [1,2] are considered to be the seminal theoretical works that established the framework. In this line of work, some works do not require expert knowledge of the underlying simulator. [3] is an offline DR-RL work that can even handle continous state and action empirically. [4,5] are policy gradient methods for DR-RL, and they also do not need prior expert knowledge.  On line 191-193, two DR-RL papers are included as reference but I think the authors should include some in the paper survey section as well.  \n[1] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280, 2005.  \n[2] Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005.  \n[3] Panaganti, K., Xu, Z., Kalathil, D., & Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. Advances in neural information processing systems, 35, 32211-32224.  \n[4] Wang, Yue, and Shaofeng Zou. \"Policy gradient method for robust reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.  \n[5] Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Levy, and Shie Mannor. Policy gradient for s-rectangular robust Markov decision processes. ArXiv preprint, abs/2301.13589, 2023. URL https://arxiv.org/abs/2301.13589.  \n1. Although the authors hint that the algorithm can tackle perturbations in dynamics/transition models, this is mainly an empirical claim with no theory to corroborate.\n1. Lipschitz assumptions on reward and model are quite strong in my opinion. This is very unlikely to be true or verifiable in larger systems. Hence, the generalizability of the proposed method is limited.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main ideas behind this formulation are quite intuitive. The paper also contains insightful theoretical component to support such intuition. The algorithm is shown to be promising in simulations. ", "weaknesses": "See limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer J2jx,\n\nThank you for your very insightful review! We provide our responses below. We believe that by including discussions on the papers and works you mentioned, the quality of our paper is greatly improved.\n\n- **Question 1: Smoothness terminology.** Indeed, the smoothness in the optimization literature refers to the gradient being Lipschitz continuous, but this convention is perhaps made mainly within the optimization community. Here we choose the word smoothness as it conveys the intuition that the action made by the policy changes gradually as we vary the state. Kindly note that we do state the precise meaning in Definition 3.1. \n\n- **Question 2: How does [1] compare to our work?**\nThank you for bringing this work to our attention, we were previously unaware of it. Our work and that of [1] are quite different and focus on different aspects of the robust MARL problem. Theoretically, [1] focuses on defining the robust equilibrium, and then showing that their proposed algorithm can converge. On the other hand, our work focuses on examining how smoothness affects robustness, and when smoothness can be used to boost the policies’ robustness. On the algorithmic side, [1] proposes to learn a separate adversarial policy for each agent, which learns to perturb that agent’s observation. This is quite different from our algorithm, which employs a Stackelberg version of adversarial regularization. Rather than learning a separate adversarial agent to perturb each policy as in [1], we can simply take a few steps of gradient ascent for each agent. We also remark that the algorithm developed in [1] does not put as large an emphasis on addressing the instability of robust MARL training as our work does. In addition, we provide extensions to mean-field MARL and action robustness, which [1] does not.\n\n- **Question 3: Extending ERNIE to continuous state and action.** \nAlgorithmically, there is no difficulty in applying ERNIE to continuous state and action space. More specifically, ERNIE is already applied with continuous state spaces, and the regularizer defined in equation (2) can stay the same for continuous action spaces as well. Theoretically, it may be difficult to prove Theorem 3.2 in the case of continuous action spaces, as our current proof relies on feeding the optimal Q-function through the softmax operator.\n\n- **Question 4: Smoothing the decision boundary.** We agree with the smooth policy being related to smooth decision boundary, as a smooth decision boundary is typically a result of smooth decision function. To the best of our understanding there is no prior discussion in MARL literature that considers robustness in the worst-case setting, but we are happy to discuss this issue further if you know of any such works.\n\n- **Limitation 1:** Thank you for these suggestions! We agree that the provided works can help contextualize our work further. We will add discussion of the works [1-5] that you suggested.\n\n- **Limitation 2:** Indeed, for now this claim is mostly empirical and we do not have theory to back it up. However, we remark that this is quite an interesting and important finding, since it potentially indicates a cheap way to provide robustness against changing transition dynamics (which is a very difficult problem). In future work, we would like to develop theory regarding this empirical phenomena we observe. For now, we believe ERNIE provides robustness against transition perturbations due to the data-augmentation that happens when we add adversarially perturbed examples to the training set. Specifically, new scenarios emerge when the environment changes, and data augmentation with adversarial examples provides a large coverage of these scenarios as long as the environment change is small. Adapting to adversarial examples during training ensures that the agents will perform reasonably even in the worst case. Such reasoning can be found in line 61. \n\n\n- **Limitation 3:** Indeed, the smoothness assumption is unlikely to be completely satisfied in large-scale, complex MARL environments. However, we remark that smoothness is often a good approximation. This can be observed by the good performance of ERNIE in the traffic light control setting (a complex environment), where the reward is not smooth (it is the composition of several categorical functions) yet ERNIE still performs quite well.\n"}, {"review_id": "1HwN96fpCh", "reviewer": "Reviewer_fpJY", "summary": "The paper introduces a robust Multi-Agent Reinforcement Learning (MARL) framework called ERNIE (adversarially Regularized multi-ageNt reInforcement lEarning). The goal of ERNIE is to address the issue of policy robustness in MARL algorithms. The key contributions of the paper are as follows:\n\n1. Robust MARL Framework: The authors propose ERNIE, which promotes the Lipschitz continuity of policies through adversarial regularization. This regularization enhances robustness against noisy observations, changing transition dynamics, and malicious actions of agents.\n2. Lipschitz Control: ERNIE controls the Lipschitz constant of policies, allowing them to be more resilient to variations in the environment. This control is achieved by leveraging adversarial regularization, which encourages Lipschitz continuity with respect to state observations and actions.\n3. Training Stability: The paper addresses the training instability that adversarial regularization can introduce by reformulating it as a Stackelberg game. This reformulation provides a more stable training process, making the optimization problem smoother for the defender (policy).\n4. Experimental Evaluation: The authors extensively evaluate the effectiveness of ERNIE in traffic light control and particle environments. The experiments demonstrate that ERNIE improves robustness and performance compared to non-robust MARL approaches.\n5. Extension to Mean-field MARL: ERNIE is extended to mean-field MARL, a strategy to scale up MARL algorithms. \n6. Theoretical Insights: The paper provides theoretical insights into the smoothness and robustness of policies in MARL. It shows that in environments with smooth transition and reward functions, policies' value functions are also smooth. The authors prove the existence of smooth and close-to-optimal policies in such environments and establish that policy robustness is proportional to its Lipschitz constant.", "questions": "It would be great if the authors can resolve my concerns in the weakness box and the Originality concerns in the strength box. I will decide my final score after the rebuttal phase according to the authors' responses.", "limitations": "NAN", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Originality**: The theoretical insights into the smoothness of environments and robustness of policies look interesting to me. By assuming the smoothness of environments, we can get some good properties of robustness. This idea is very interesting to me. However, this idea is very similar to the paper [1]“****Deep Reinforcement Learning with Robust and Smooth Policy****”. \n\n[1] Shen, Q., Li, Y., Jiang, H., Wang, Z., & Zhao, T. (2020, November). Deep reinforcement learning with robust and smooth policy. In *International Conference on Machine Learning* (pp. 8707-8718). PMLR.\n\n**Quality**: The paper is very informative. I can see the authors have a lot to show readers in the paper: the theoretical results, the proposed framework, how to solve the problem efficiently, and how to adopt the proposed framework to large-scale multi-agent systems.\n\n**Clarity**: The paper appears to be clear in its presentation of the problem, the proposed solution, and the results of the experiments. The authors provide a theoretical motivation for their algorithm, explaining how Lipschitzness (smoothness) serves as a natural way to gain robustness while reducing the policy search space. The extension of ERNIE to mean-field MARL is also clearly explained.\n\n**Significance**: I believe this paper's theoretical analysis of how smoothness helps in training robust policies would help future researchers in the field of robust MARL to develop better solutions to the robustness issues.", "weaknesses": "**Confusing Writing**: I found the paper sometimes inconsistent in motivation, methods, and experiments, which makes me confused about what problem this paper tries to solve. In the abstract, the authors point out that we need to consider real-to-sim gaps in the deployment of MARL algorithms, I quote the original sentence there “This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment.” But later, the authors claim that their proposed method is robust to perturbations on observations and actions. I also quote the original sentences as follows: “Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization.” and “In particular, we develop an adversarial regularizer to minimize the discrepancy between each policy’s output given a perturbed observation and a non- perturbed observation.” When I dive into the methodology of this paper, I found most of the contexts are about **Robustness against observation noise with smooth policies** and **Robustness against Malicious Actions.** We can see that the abstract and some texts in the introduction mislead readers about what kind of robustness this paper tries to achieve in MARL. \n\n**Evaluations on Simple Games**: The evaluation of the proposed methods is conducted on specific tasks such as traffic light control and particle environment tasks. While these tasks are complex, they may not fully reveal insights into how the smoothness of environments affects the robustness of policies.  I would suggest running experiments on some simple matrix games or small-scale multi-agent toy examples [1] to better study how smoothness is connected to robustness. \n\n[1] Hu J, Wellman MP. Nash Q-learning for general-sum stochastic games. Journal of machine learning research. 2003;4(Nov):1039-69.\n\n**Lack of Comparison with Robust MARL Methods:** This paper claims that the proposed method is robust to perturbed actions and observations. Considering there are some existing works about robust MARL considering adversarial-purebred observations [1, 2, 3] and perturbed actions [4], should the authors compare the proposed method with at least one of these robust MARL methods?  \n\n[1] Han S, Su S, He S, Han S, Yang H, Miao F. What is the solution for state adversarial multi-agent reinforcement learning?. arXiv preprint arXiv:2212.02705. 2022 Dec 6.\n\n[2] He S, Han S, Su S, Han S, Zou S, Miao F. Robust Multi-Agent Reinforcement Learning with State Uncertainty. Transactions on Machine Learning Research. 2023 Mar 10.\n\n[3] Zhou Z, Liu G, Zhou M. A Robust Mean-Field Actor-Critic Reinforcement Learning Against Adversarial Perturbations on Agent States. IEEE Transactions on Neural Networks and Learning Systems. 2023 Jun 5.\n\n[4] Li, S., Guo, J., Xiu, J., Yu, X., Wang, J., Liu, A., ... & Liu, X. (2023). Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game. *arXiv preprint arXiv:2305.12872*.\n\n**Writing Suggestions:**\n\nThe mean-field MARL part does not tightly connect to the main text of this paper. Would it be better to make it a remark instead of a subsection? Besides, the organization of this paper makes me confused, for example, Robustness against observation noise with smooth policies (a paragraph) and Robustness against Malicious Actions (a subsection) are placed on different layers of this paper. Should they be on similar layers? As a paper studying robust MARL, in the related work, this paper uses more text to discuss robust RL works than robust MARL. The literature review of robust MARL is not comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fpJY,\n\nThank you for your very detailed and helpful review! We believe that by addressing your critiques, we have significantly improved the paper. In particular, we clarify the originality of our method, add comparisons with robust MARL methods, and provide ways to clear up the writing.\n\n- **Weakness 0: Difference from Shen et al., 2020 [1] (Clarification on Originality).**\nThe work of [1] is one of the motivating works from which we base our work. However, we believe we have provided several significant contributions over [1]. We list them as follows: (1) [1] provides no theoretical results to support their work. In constrast, we develop theory as to why smoothness provides robustness and can boost sample efficiency. (2) [1] is developed for the single-agent case and does not consider the instability of MARL training. To improve upon this, we develop a Stackelberg extension to adversarial regularization which can significantly boost performance. (3) We develop extensions to mean-field MARL and action robustness, which [1] does not consider.\n\n- **Weakness 1: Confusing Writing.**\nWe apologize for any confusion in the introduction. By bridging the sim-to-real gap, we meant to refer to the cases where the simulator does not contain any observation noise but the real world environment does (or contains noise in the actions). We remark that this is a realistic part of the sim-to-real gap (i.e. cameras in simulation may not pick up any dust particle, but in the real world they may). To avoid confusion, we will state precisely what type of robustness we are attempting to achieve, including the specific sim-to-real robustness we seek.\n\n- **Weakness 2: Evaluations on Simple Games.**\nOne of the main benefits of our algorithm over previous works such as [1] is that we improve the stability of the deep MARL training and develop adversarial regularization for MARL. In matrix environments, it may be very difficult to evaluate these gains as deep MARL is not needed. However, we do conduct experiments on the simple spread environment, which has a very simple objective (for the agents to cover landmarks). In this environment (Figure 3c), we can observe that smoothness not only improves robustness, but also clean performance.\n\n- **Weakness 3: Lack of Comparison with Robust MARL Methods.**\nThis is a great suggestion. We apply the method RMA3C from [1] to our particle environment experiments. We remark that this algorithm is quite similar to the RMAAC algorithm in [2], which we discuss in response to question 2 of J2jx. The experimental results of RMA3C can be seen in Figure 2 of the rebuttal material. We notice that RMA3C can provide some robustness, but cannot outperform ERNIE. We hypothesize this is because RMA3C trains on perturbed observations which may result in underfitting. On the other hand, ERNIE trains with clean observations and applies regularizations to the perturbed observations, allowing better fitting to the non-perturbed environment while still providing robustness. With the addition of this baseline, we have compared our algorithm with three robust MARL baselines.\n\n- **Weakness 4: Writing Suggestions.**\nWe apologize for the confusion. These suggestions make sense to us. We can make the mean-field MARL section a remark, as no sections build upon it, and it is more of an extension of ERNIE to a special case. In addition, we agree that robustness against observation noise can be put in similar layers of the paper (subsections). Finally, we believe that by adding the discussion on the related literature you suggested in weakness 3 ([1-4]), our robust MARL literature review will be more comprehensive.\n\nPlease let us know if we can add further experiments or discussion to help resolve your concerns about our work.\n"}, {"review_id": "U7aEcTmPqD", "reviewer": "Reviewer_YAu4", "summary": "The paper proposes a training objective Lipschitz regularization for obtaining adversarial robustness in cooperative multi-agent RL. To overcome training instablities, the paper reframes the regularization as a Stackleberg game. The paper also shows how the method can be applied to Mean-field approximations of some MARL training algorithms and carries out two experiments on the traffic and the multi-agent Particle Environments.", "questions": "The method seems to be focused on Cooperative Markov Games. Can you elaborate on whether, how, or why the method would apply to other multi-agent games (competitive or other objectives, etc.)?\n\nHow does ERNIE compare to direct applications on Eq (2) and (3) in the training process? ", "limitations": "Readability of the figures could be improved", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Nice background on connecting Lipschitz continuity with robustness with respect to different variables of game and policy.\n- Overall solid method aiming to overcome the instability of adversarial training in the context of MARL.", "weaknesses": "I believe the clarity of the experiments section could be significantly improved. The figures are barely legible due to the small (font) size. Moreover, the figure caption could be improved, e.g., does \"Evaluation curves\" refer to \"Learning curves\" as in the episode reward over the training? Moreover, what does \"Baseline-Gaussian\" refer to? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer YAu4,\n\n- **Weakness 1: Clarity of figures.**\nThank you for pointing this out! We will increase the font size on the figures to improve the legibility. As for \"Evaluation Curves,\" this refers to the reward achieved when we evaluate the policy in the perturbed environment. We also just realized \"training reward\" in the captions of Figures 3 and 4 should be replaced with \"evaluation reward.\" Thank you for catching this! Finally, Baseline-Gaussian refers to a Baseline where the perturbation in ERNIE is sampled from a Gaussian distribution.\n\n- **Question 1: Could the method be applied to competitive multi-agent games?**\nYes, ERNIE can straightforwardly be applied to non-cooperative environments. However, the theoretical analysis may become significantly more complicated in such settings. Therefore we leave applications to non-cooperative settings for future work.\n\n- **Question 2: How does ERNIE compare to direct application of equations (2) or (3)?**\nPlease note that we evaluate the direct application of (2) and (3) into training in line 358-374 and Figures 5a-b (this baseline is denoted as ERNIE w/o ST). As we can see, ERNIE significantly outperforms this baseline, showing the benefit of our proposed Stackelberg training in ERNIE.\n\n"}]

Author rebuttal:
Dear Reviewers,

Thank you for all your helpful comments and critiques! We believe addressing them has significantly improved our work. In particular, we have made several additions to our work: 

1. New experiments in multi-agent robotic control.
2. Comparison with new robust MARL algorithms (for both the related work section and the experiments section). 
3. Adding new ablation studies (ERNIE for mean-field actions, more time comparisons).
4. Proposing new ways to organize the paper for clarity. 

You may find our new experimental results in the attached document. We also provide individual responses to your reviews.

Meta-review: This paper makes a solid contribution to an interesting problem. Some concerns remain, particularly regarding comparisons with existing literature and clarity of the presentation, which the authors are encouraged to address in the final version.
Decision: Accept (poster)