Title: Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Theory of Robust Reinforcement Learning

Adversarial Regularization
Abstract: Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy’s Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Motivated by these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE’s adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE.

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

Abstract

Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy's Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE's adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE .

1 Introduction

In the past decade advances in deep neural networks and greater computational power have led to great successes for Multi-Agent Reinforcement Learning (MARL), which has achieved success on a wide variety of multi-agent decision-making tasks ranging from traffic light control [1] to StarCraft [2]. However, while much effort has been devoted to applying MARL to new problems, there has been limited work regarding the robustness of MARL policies.

Despite the limited attention paid to robustness, it is essential for MARL policies to be robust. Most MARL policies are trained in a fixed environment. Since these policies are trained solely to perform well in that environment, they may perform poorly in an environment with slightly different transition dynamics than the training environment. In addition, while agents are fed with exact state information in training, MARL policies deployed in the real world can receive inaccurate state information (e.g., due to sensor error). Finally, even a single agent acting maliciously or differently than expected can cause a chain reaction that destabilizes the whole system. These phenomena cause significant concern for the real-world deployment of MARL algorithms, where the environment dynamics and observation noise can change over time. We observe that even when the change in the environment's dynamics is small, the performance of MARL algorithms can deteriorate severely (See an example

in Section 5). Thus there is an emerging need for MARL algorithms that are robust to changing transition dynamics, observation noise, and changing behavior of agents.

Although many robust RL methods have been proposed for the single agent case, three major barriers prevent their use for MARL. Theoretically, it is not clear if or when such methods can work for MARL. Methodologically, it is not straightforward to apply single agent robust RL methods to MARL, as single agent methods may not consider the interactions between several agents. Algorithmically, single agent robust RL algorithms are often unstable, and may not perform well when applied to inherently unstable MARL training. Therefore to learn robust MARL policies, we provide theoretical, methodological, and algorithmic contributions.

Theory. Theoretically, we first show that when the transition and reward function are smooth, a policy's value function is also smooth. In our experiments, we show that this assumption can serve as a useful prior knowledge, even if the transition function is not smooth in every state. Second, we prove that a smooth and close-to-optimal policy exists in any such environment. Third, we show that a policy's robustness is proportional to its Lipchitz constant with no smoothness assumption on the environment's smoothness . These observations advocate for using smoothness as an inductive bias to not only reduce the policy search space, but simultaneously improve the robustness of the learned policy. Finally, we prove that large neural networks are capable of approximating the target policy or Q functions with smoothness guarantees. These findings give us the key insight that in order to learn robust and high-performing deep MARL policies, we should enforce the policies' smoothness.

Method. Based on these findings, we propose a new training framework - adv E rsarially R egularized multiage N t re I nforcement l E arning (ERNIE), that applies adversarial training to learn smooth and robust MARL policies in a principled manner. In particular, we develop an adversarial regularizer to minimize the discrepancy between each policy's output given a perturbed observation and a nonperturbed observation. This adversarial regularization gives two main benefits: Lipschitz continuity and rich data augmentation with adversarial examples. The adversarial regularization encourages the learned policies to be Lipschitz continuous, improving robustness. Augmenting the data with adversarial examples further provides robustness against environment changes. Specifically, new scenarios emerge when the environment changes, and data augmentation with adversarial examples provides a large coverage of these scenarios as long as the environment change is small. Adapting to adversarial examples during training ensures that the agents will perform reasonably even in the worst case.

To further provide robustness against the changing behaviors of a few malicious agents, we propose an extension of ERNIE that minimizes the discrepancy between the global Q-function with maliciously perturbed joint actions and non-perturbed joint actions. This regularizer encourages the policies to produce stable outputs even when a subset of agents acts sub-optimally, therefore granting robustness. Such robustness has not been considered in previous works.

Algorithm. We find that adversarial regularization can improve robust performance [3]. However, adversarial regularization can also be unstable. More concretely, conventional adversarial regularization can be formulated as a zero-sum game where the defender (the policy) and attacker (the perturbation) hold equal positions and play against each other. In this case, a small change in the attacker's strategy may result in a large change for the defender, rendering the problem ill-conditioned. Coupled with the already existing stability issues that come with training MARL algorithms, this instability issue greatly reduces the power of adversarial regularization methods for MARL.

To address this issue, we reformulate adversarial training as a Stackelberg game. In a Stackelberg game, the leader (defender) has the advantage as it knows how the follower (attacker) will react to its actions and can act accordingly. This advantage essentially makes the optimization problem smoother for the defender, leading to a more stable training process.

Extension to Mean-field MARL. We further demonstrate the general applicability of ERNIE by developing its extension to robustify mean-field MARL algorithms. The mean-field approximation has been widely received as a practical strategy to scale up MARL algorithms while avoiding the curse of many agents [4]. However, as mean-field algorithms are applied to real-world problems, it is essential to develop robust versions. To facilitate policy learning that is more robust, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization [5; 6].

To demonstrate the effectiveness of the proposed framework, we conduct extensive experiments that evaluate the robustness of ERNIE on traffic light control and particle environment tasks. Specifically,

we evaluate the robustness of MARL policies when the evaluation environment deviates from the training environment. These deviations include observation noise, changing transition dynamics, and malicious agent actions. The results show that while state-of-the-art MARL algorithms are sensitive to small changes in their environment, the ERNIE framework enhances the robustness of these algorithms without sacrificing efficiency.

Contributions. We remark that adversarial regularization has been developed for single-agent RL, but never for MARL [3]. Our contribution in this paper has four aspects: (1) advances in theoretical understanding (2) development of new regularizers for MARL (3) new algorithms for stable adversarial regularization in MARL (4) comprehensive experiments in a number of environments.

2 Background

In this section, we introduce the necessary background for MARL problems together with related literature. We consider the setting of cooperative MARL , where agents work together to maximize a global reward.

· Cooperative Markov Games . We consider a partially observable Markov game 〈 S , O N , A N , P , R , N, γ 〉 in which a set of agents interact within a common environment. We let S ⊆ R S denote the global state space, O ⊆ R O denote the observation space for each of the N agents, A ⊆ R A denote the action space, P : S × A ↦→ S denote the transition kernel, γ denotes the discount factor, and R : S × A ↦→ R N denotes the reward function. At every time step t , each of the N agents selects an action according to its policy, which can be stochastic or deterministic. Then, the system transitions to the next state according to the transition kernel and each agent receives a reward r i,t . We denote the global reward at time t as r g t . The goal of each agent is to find a policy that maximizes the discounted sum of its own reward, ∑ t ≥ 0 γ t r i,t .

· Robust RL . In recent years many single agent robust RL techniques have been proposed. Most of these methods use information about the underlying simulator to train agents over a variety of relevant environment settings [7; 8; 9; 10; 11]. Although these methods can provide robustness against a wide range of environment changes, they suffer from long training times and require expert knowledge of the underlying simulator, which is not practical. Another direction of research focuses on perturbation based methods [3; 12]. Perturbation based methods train the policy to be robust to input perturbations, encouraging the policy to act reasonably in perturbed or previously unseen states. [13] certify robustness by adding smoothing noise to the state; it is not clear how this affects the learned policy's optimality. Another related line of work [14; 15; 16; 17] studies robust markov decision processes and provides a principled way to learn robust policies. However, such methods often require strict assumptions on the perturbation/uncertainty. Inspiring our work, [3] proposes to learn a smooth policy in single agent RL, but they do so to reduce training complexity rather than increase robustness and provide no theoretical justification for their method. Instead, we theoretically connect smoothness to robustness, extend perturbation based methods to MARL, and develop a more stable perturbation computation technique, and develop an extension to mean-field MARL.

· Robust MARL . Recently, some works have studied the robustness of MARL systems. Lin et al. [18] studies how to attack MARL systems and finds that MARL systems are vulnerable to attacks on even a single agent. Zhang et al. [19] develop a framework to handle MARL with model uncertainty by formulating MARL as a robust Markov game. However, their proposed method only considers uncertainty in the reward function, while this article focuses on robustness to observation noise and changing transition dynamics. Li et al. [20] modify the MADDPG algorithm to consider the worst-case actions of the other agents in continuous action spaces with the M3DDPG algorithm. M3DDPG aims to grant robustness against the actions of other agents, which is less general than the robustness against observation noise, changing transition dynamics, and malicious agents that our method aims for. Wang et al. [21] consider robustness against uncertain transition dynamics, but their algorithm is not applied to deep MARL. More recently, He et al. [22]; Han et al. [23] introduces the concept of robust equilibrium and proposes to learn an adversarial policy to perturb each agent's observations. Finally Zhou et al. [24] propose to learn robust policies by minimizing the cross-entropy loss between agent's actions in non-perturbed states and perturbed states.

The ERNIE framework is also related to several existing works which use similar adversarial training methods but target different domains such as trajectory optimization [25], semi-supervised learning [26; 27; 28], fine-tuning language models [29; 30], and generalization in supervised learning [31].

3 From Lipschitz Continuity to Robustness

This section presents the theoretical motivation for our algorithm by showing that Lipschitzness (smoothness) serves as a natural way to gain robustness, while reducing the policy search space. We start by observing that certain natural environments exhibit smooth transition and reward functions, especially when the transition dynamics are governed by physical laws (e.g., MuJuCo environment [32], Pendulum [33]). * Formally, this is stated as the following.

Definition 3.1. Let S ⊆ R d . We say the environment is ( L r , L P ) -smooth, if the reward function r : S × A → R , and the transition kernel P : S × S × R satisfy

for ( s, s ′ , a ) ∈ S × S × A . ‖ · ‖ denotes a metric on R d . We say a policy π is L π -smooth if

Without loss of generality, we assume | r ( s, a ) | ≤ 1 for any ( s, a ) ∈ S × A . We then present our theory. Due to the space limit, we defer all technical details to the appendix.

From smooth environments to smooth values. We proceed to show that if the environment is smooth, then the value functions for smooth policies are also smooth.

Theorem 3.1. Suppose the environment is ( L r , L P ) -smooth. Then the Q-function of any policy π , defined as is Lipschitz continuous in the first argument. That is,

where L Q := L r + γ L P / (1 -γ ) . Suppose in addition the policy is L π -smooth. Then the value function, defined as is Lipschitz continuous. That is,

In view of Theorem 3.1, it is clear that whenever the environment and the policy are smooth, then the value functions are also smooth. A natural and important follow-up question to ask is whether this claim holds in the reverse direction. More concretely, we ask whether it is reasonable to seek a policy that is also smooth with respect to the state while maximizing the reward. If the claim holds true, then seeking a smooth policy can serve as an efficient and unbiased prior knowledge, that can help us reduce the policy search space significantly, while still guaranteeing that we are searching for high-performing policies.

Existence of smooth and nearly-optimal policies. The following result shows that for any /epsilon1 > 0 , there exists an /epsilon1 -optimal policy that is O ( L Q / /epsilon1 ) smooth, where L Q defined in Theorem 3.1 only depends on the smoothness of reward and transition. This structural observation naturally suggests seeking a smooth policy for smooth environments.

Theorem 3.2. Suppose the environment is ( L r , L P ) -smooth. Then for any /epsilon1 > 0 , there exists an /epsilon1 -optimal policy π that is also smooth, i.e.,

where L Q is defined as in Theorem 3.1.

Notably, the proof of Theorem 3.2 relies on the key observation that any smooth Q -function satisfying (1) can be fed into the softmax operator, which induces a smooth policy. This observation also provides a way for value-based methods (e.g., Q-learning) to learn a smooth policy. Namely, one can first learn a smooth surrogate of the optimal Q -function, and then feed the learned surrogate into the softmax operator to induce a close-to-optimal policy that is also smooth.

· Robustness against observation noise with smooth policies. We have so far established that smooth policies naturally exist in a smooth environment as close-to-optimal policies, and thus smoothness serves as a strong prior for policy search. We will further demonstrate that the benefits of smooth policy go beyond boosting learning efficiency, by bringing in the additional advantage of robustness against observation uncertainty.

Theorem 3.3. Let π ( a | s ) be L π -smooth policy. For any perturbation sequence { δ t s } t ≥ 0 ,s ∈ S , define a perturbed policy (non-stationary) ˜ π = { ˜ π t } t ≥ 0 by π t ( a | s ) = π ( a | s + δ t s ) ,

˜ with ‖ δ t s ‖ ≤ /epsilon1 for all t ≥ 0 . Accordingly, define the value function of the non-stationary policy ˜ π

Then we have | V π ( s ) -V ˜ π ( s ) | ≤ 2 L π /epsilon1 (1 -γ ) 2 , Similarly, we have where Q ˜ π is defined similarly as V ˜ π .

Theorem 3.3 establishes the following fact: for a discounted MDP with finite state and finite action space, the value of the policy when providing the perturbed state is close to the value of the policy when given the non-perturbed state, provided the policy is Lipschitz continuous in its state. As an important implication, the learned smooth policy will be robust in the state observation, in the sense that the accumulated reward will not deteriorate much when noisy, or even adversarially constructed state observations are given to the policy upon decision making.

We emphasize that Theorem 3.3 holds without any smoothness assumption on the transition or the reward function. It should also be noted that there are various notions of robustness in MDP, e.g., robustness against changes in transition kernel [34; 17], which we defer as future investigations.

Before we conclude this section, we briefly remark on certain generality of our discussion.

Remark 3.1 ( Applicability to MARL) . The discussions in this section do not depend on the size of the state space, and apply to the multi-agent setting without any change. To see this, note that our discussion holds for any discrete state-action space. Setting S as the joint state space and A as the joint action space, then the obtained results trivially carry over to the cooperative MARL setting.

Remark 3.2 ( Environments with approximate smoothness) . Many environments are partially smooth, in the sense that the transition or the reward is non-smooth only on a small fraction of the state space. Typical examples include the Box2D environment [33], where the agent receives smooth reward when in non-terminal states (airborne for Lunar Lander), and receives a lump-sum reward in the terminal state (land/crash) - a vanishing fraction of the entire state space. Given the environment being largely smooth, it should be expected that for most states the optimal policy is locally smooth. Consequently, inducing a smoothness prior serves as a natural regularization to constrain the search space when solving these environments, without incurring a large bias.

Remark 3.3 ( Non-smooth environments) . From the perspective of robust statistics, achieving robustness often necessitates a certain level of smoothness in the learned policy, regardless of the smoothness of the optimal policy. In scenarios where the environment itself is non-smooth, the optimal policy can also be non-smooth. However, it is important to note that such non-smooth optimal policies are typically not robust. This means that by trading-off between the approximation bias and robustness, the smooth policy learnt by out method has the potential to outperform non-smooth policies in perturbed environments.

4 Method

In this section, we propose our robust MARL framework, adv E rsarially R egularized multiage N t re I nforcement l E arning (ERNIE).

4.1 Learning Robust Policy with ERNIE

Section 3 shows that the robustness of a policy depends on its Lipschitz constant. Therefore, in ERNIE we propose to control the Lipschitz constant of each policy with adversarial regularization.

Given a policy π θ k , where k is the agent index, the ERNIE regularizer is defined by

Here δ ( o k , θ k ) is a perturbation adversarially chosen to maximize the difference between the policy's output for the perturbed observation o k + δ ( o k , θ k ) and the original observation o k . In this case /epsilon1 controls the perturbation strength and || · || is usually taken to be the /lscript 2 or /lscript ∞ norm. Note that R π ( o k ; θ k ) essentially measures the local Lipschitz smoothness of policy function π θ around the observation o k , defined in metric D ( · , · ) . Therefore minimizing R π ( o k ; θ k ) will encourage the policy to be smooth.

Regularization (2) allows straightforward incorporation into MARL algorithms that directly perform policy search. For actor-critic based policy gradient methods, the regularizer (2) can be directly included into the objective for updating the actor (policy) networks. When optimizing stochastic policies (e.g., MAPPO [35]), D can be taken to be the KL divergence and for deterministic policies (e.g., MADDPG [36] or Q-learning [37]), we set D to be the /lscript p norm.

More concretely, let L ( θ ) denote the policy optimization objective, i.e., the negative weighted value function of the policy. We then augment L ( θ ) with (2), and minimize the regularized objective

where λ is a hyperparameter. We remark that Shen et al. [3] has explored similar regularization for single-agent RL (with a goal of improving sample efficiency), but as we explain in sections 4.2, 4.3, and 4.4, successful application to MARL robustness is highly non-trivial.

4.2 Stackelbeg Training with Differentiable Adversary

Although accurately solving (3) will result in a high-performing and robust policy, we note that (3) is a nonconvex-nonconcave minimax problem. In practice, we can use multiple steps of projected gradient ascent to approximate the worse-case state perturbation δ ( o k , θ k ) , followed by one-step gradient descent for updating the policies/Q-function. Even though this optimization method already significantly improves robustness over the baseline algorithms, we found that the training process could be quite unstable. We hypothesize that the intrinsic instability of MARL algorithms due to simultaneous updates of multiple agents is greatly amplified by the non-smooth landscape of adversarial regularization.

To promote a more stable straining process, we propose to reformulate adversarial training in ERNIE as a Stackelberg game. The reformulation defines adversarial regularization as a leader-follower game [38]:

Here ◦ denotes the operator composition (i.e f ◦ g = f ( g ( · )) ), and is a one-step gradient ascent for maximizing the divergence of the perturbed and original observation.

Compared to the vanilla adversarial regularizer in (2), the perturbation δ is treated as a function of the model parameter θ . This formulation allows the leader ( θ ) to anticipate the action of the follower ( δ ), since the follower's response given observation o is fully specified by δ K ( o, θ ) . This structural anticipation effectively produces an easier and smoother optimization problem for the leader ( θ ), whose gradient, termed Stackelberg gradient, can be readily computed by

Note that the gradient used in (3) only contains the 'leader' term, such that interaction between the model θ and the perturbation δ is ignored. The computation of the Stackelberg gradient can be reduced to Hessian vector multiplication using finite difference method [39], which only requires two

backpropogations and extra O ( d ) complexity operation. Thus no significant computational overhead is introduced for solving (4).

The benefit of Stackelberg training for MARL is twofold. First, a smoother optimization problem results in a more stable training process. This extra stability is essential given the inherent instability of MARL training. Second, giving the policy θ priority over the attack δ during the training process allows for a better training data fit than normal adversarial training allows. This better fit allows the MARL policies trained with Stackelberg training to perform better in lightly perturbed environments than those trained with normal adversarial regularization.

4.3 Robustness against Malicious Actions

Given the complex interactions of agents within of a multi-agent system, a robust policy for any given agent should meet the criterion that the action made is not overly dependent on any small subset of agents. This is particularly the case when the agents are homogeneous in nature [4; 40], and thus there should be no notion of coreset agents in the decision-making process that could heavily influence the actions of other agents. We proceed to show how ERNIE could be adopted to induce such a notion of robustness.

The core idea of ERNIE for this scenario is to encourage policy/Q-function smoothness with respect to joint actions . Similar to our treatment in (2), we now seek to promote learning a Q-function that yields a consistent value when perturbing the actions for any small subset of agents. Specifically, for discrete action space, we define a regularizer on the global Q-function as

/negationslash where D ( a , a ′ ) = ∑ i I ( a i = a ′ i ) . The regularizer (5) seeks to compute the worst subset of changed actions with cardinality less than K . For continuous action spaces, one could replace the metric D in (5) by a differentiable metric defined over the action space (e.g., ‖ · ‖ 2 -norm), and then evaluate the regularizer with projected gradient ascent.

To evaluate the adversarial regularizer for the discrete action space, we propose to solve (5) in a greedy manner by finding the worst-case change in the action of a single agent at a time, until the action of K agents is changed. Specifically, at each training step, we search through all the agents/actions and then pick the actions that produce the topK changes in the Q-function, resulting in a O ( |A| ∗ N ∗ K ) computation. Our complete algorithm can be found in Appendix H, and we find that in our numerical study, perturbing the action of a single agent ( K = 1 ) is sufficient for increased robustness.

Similar to the regularizer in (2), the regularizer in (5) provides the benefits of Lipschitz smoothness (with respect to the Hamming distance) and data augmentation with adversarial examples. If the behavior of a few agents changes (either maliciously or randomly), the behavior of policies trained by conventional methods may change drastically. On the other hand, policies trained by our method will continue to make reasonable decisions, resulting in more stable performance (see section 5).

4.4 Extension to Mean-field MARL

MARL algorithms have been known to suffer from the curse of many agents [4], as the search space of policies and value functions grows exponentially w.r.t. the number of agents. A practical approach to tackle this challenge of scale is to adopt the mean-field approximation, which views each agent as realizations from a distribution of agents. This distributional perspective requires a distinct treatment of ERNIE applied to the mean-field setting.

Mean-field MARL avoids the curse of many agents by approximating the interaction between each agent and the global population of agents with that of an agent and the average agent from the population. In particular, we can approximate the action-value function of agent j as Q j ( s , a ) = Q j ( s j , d s , a j , ¯ a j ) , where a is the global joint action, s is the global state, ¯ a j is the average action of agent j 's neighbors, and d s is the empirical distribution of states over the population. Such an approximation has found widespread applications in practical MARL algorithms [41; 42; 40; 43], and can be motivated in a principled fashion for agents of homogeneous nature [4; 40].

To learn robust and scalable policies, we extend ERNIE to the mean-field setting by applying adversarial regularization to the approximation terms d s and ¯ a j . It is important to note that as the terms d s and d ′ s represent distributions over states, we bound the attack by the Wasserstein distance [44]. In what follows we only apply the regularizer to d s for simplicity. This leads to a new regularizer

defined over the mean field state

where W denotes the Wasserstein distance metric. Since the explicit Wasserstein constraint may be difficult to optimize in practice, we can instead enforce the constraint through regularization, as displayed in Appendix C.

5 Experiments

We conduct extensive experiments to demonstrate the effectiveness of our proposed framework. In each environment, we evaluate MARL algorithms trained by the ERNIE framework against baseline robust MARL algorithms. To evaluate robustness, we train MARL policies in a non-perturbed environment and evaluate these policies in a perturbed environment. The reported results are gathered over five runs for each algorithm. Given the space limit, we put additional results in Appendix E.

Traffic Light Control. In this scenario, cooperative agents learn to minimize the total travel time of cars in a traffic network. We use QCOMBO [45] and COMA [46] (results in appendix E) as our baseline algorithms and conduct experiments using the Flow framework [47]. We train the MARL policies on a two-by-two grid (four agents). We then evaluate the policies on a variety of realistic environment changes, including different car speeds, traffic flows, network topologies, and observation noise. In each setting, we plot the reward for policies trained with ERNIE, the baseline algorithm (QCOMBO), and another baseline where the attack δ is generated by a Gaussian random variable (Baseline-Gaussian, see Appendix I). Implementation details can be found in Appendix G.

Figure 1, 2a, and 2b show that the baseline algorithm is vulnerable to small changes in the training environment (higher reward is better). On the other hand, ERNIE achieves more stable reward on each environment change. This observation confirms that the ERNIE framework can improve robustness against observation noise and changing transition dynamics. The Gaussian baseline performs well on some environment changes, like when the observations are perturbed by Gaussian noise, but performs poorly on other environment changes, like when the car speed is changed. We hypothesize that while some environment changes may be covered by Gaussian perturbations, other environment changes are unlike Gaussian perturbations, resulting in a poor performance from this baseline.

Robustness Against Malicious Actions. We also evaluate the extension of ERNIE to robustness against changing agent behavior, which we refer to as ERNIE-A. To change agent behavior, we adversarially change the action of a randomly selected agent a small percentage of the time, i.e. 5% or 3% of the time. As can be seen in Figures 2c and 2d, the two baseline algorithms perform poorly when some agent's behavior changes. In contrast, ERNIE-A is able to maintain a higher reward.

(b) Different Speed

(c) Traffic Flow (Flow-3)

20

40

Reward

60

08-

ERNIE

-100

Baseline

120

0

5001000150020002500300035004000

Iteration

-40

50

Reward

60

-70

80

0

5001000 15002000250030003500 4000

Iteration

-40

-50

Reward

-60

70

-80

06-

-100

0

700140021002800350042004900

Iteration

Particle Environments. We evaluate ERNIE on the cooperative navigation, predator-prey, tag, and cooperative communication tasks. In each setting, we investigate the performance of the baseline algorithm (MADDPG), ERNIE, M3DDPG, and the baseline-gaussian in environments with varying levels of observation noise. We also compare ERNIE to the RMA3C algorithm proposed in [23]. In Figure 3, we find ERNIE performs better or equivalently than MADDPG in all settings. Surprisingly, M3DDPG can provide some robustness against observation noise, even though it is designed to provide robustness against malicious actions.

Mean-field MARL. We evaluate the performance of the mean-field ERNIE extension on the cooperative navigation task [36] with different numbers of agents. We compare the performance of the baseline algorithm, ERNIE, and M3DDPG under various levels of observation noise. We use

(b) Different Topology

(c) Actions (3%)

(d) Actions (5%)

20

60

40

70

70

60

40

80

08-

Reward

100

120

140

08

ERNIE

Baseline

Reward

100

09-

80

Reward

R

-110

120

130

140

100

90

Baseline

Reware

100

110

120

130

140

90

0

500 1000 1500 2000 2500 3000 3500 4000

Iteration

0

500 1000 1500 2000 2500 3000 3500 4000

Iteration

0

500 1000 1500 2000 2500 3000 3500 4000

Iteration

0

500 1000 1500 2000 2500 3000 3500 4000

Iteration

(b) Tag

(c) Navigation

(d) Predator Prey

80

80

-180

Reward

60

40

20

0

RMA3C

ERNIE

MADDPG

M3DDPG

Reward

-20

60

20

40

0

Reward

-280

-200

220

-240

000-

-260

Reward

30

10

20

0

-20

-40

-320

-10

-40

-60

-340

-20

-1

-0.8

-0.6

-0.4

-0.2

0.0

-1

-0.8

-0.6

-0.4

-0.2

0.0

0.0

0.3

0.6

0.9

1.2

1.5

1.8

-1

-0.8

-0.6

-0.4

-0.2

0.0

log ε

log ε

log ε

log ε

mean-field MADDPG as our baseline and follow the implementation of [40]. As can be seen in Figure 4, ERNIE displays a higher reward and a slower decrease in performance across noise levels.

(a) N=3

(b) N=6

(c) N=15

-310

-160

-320

-340

MeanField

M3DDPG

ERNIE-MF

Reward

-4000

-4100

vard

3475

-3500

3525

Reward

-180

-200

-220

em

-360

OSE-

4300

-4200

Rew

3550

3575

-3600

240

-260

-280

32

64

128

-370

-380

-1-0.9 -0.8 -0.7 -0.6 -0.5 -0.4-0.3 -0.2 -0.1 0.0

4400

-1-0.9 -0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0.0

3625

3650

-1-0.9 -0.8 -0.7 -0.6 -0.5-0.4 -0.3 -0.2 -0.1 0.0

-300

-320

0.5

256

1.0

2.0

4.0

log ε

log ε

log ε

E

(a) Different Speed

(b) Observation Noise

(c) Sensitivity to

/epsilon1

(d) Sensitivity to

K

150

200

ERNIE

200

20

0=）

=0.01

=0.005

=0.003

一

=0.007

20

K=0

K=1

K =2

K=

4

K=6

250

ERNIE w/o ST

30

30

Reward

300

350

Baseline

Reward

-300

40

40

R

400

400

50

50

450

500

500

60

60

550

600

70

70

0

20

Percentile

40

60

80

0

20

Percentile

40

60

80

0)

1000

2000

3000

4000

0

1000

2000

3000

4000

Hyperparameter Study. We investigate the sensitivity of ERNIE to the hyperparameters K (the number of attack steps) and /epsilon1 (the perturbation strength). We plot the performance of different hyperparameter settings in the traffic light control task with perturbed car speeds on three random seeds. From Figures 5c and 5d we can see that adversarial training ( K > 0 ) outperforms the baseline ( K = 0) for all K . Similarly, we can see that all values of /epsilon1 outperform the baseline ( /epsilon1 = 0) . This indicates that ERNIE is robust to different hyperparameter settings of /epsilon1 and K .

Sensitivity and Ablation Study. The advantage of the ERNIE framework goes beyond improving the mean reward. To show this, we evaluate 10 different initializations of each algorithm in two traffic environments: one with different speeds and another environment with observation noise. We then sort the cumulative rewards of the learned policies and plot the percentiles in Figures 5a and 5b. Although the best-case performance is the same for all algorithms, ERNIE significantly improves the

robustness to failure. As an ablation, we evaluate the effectiveness of ERNIE with and without the Stackelberg formulation of adversarial regularization (ST). ERNIE displays better performance in both settings, indicating that Stackelberg training can lead to a more stable training process. Ablation experiments in other environments can be found in Appendix E.4.

Robustness and Network Width. In Appendix B, we show that in order to learn a robust policy with ERNIE, we should use a sufficiently wide neural network. Therefore, we evaluate the robust performance of ERNIE using policy networks with 32, 64, 128, and 256 hidden units. We carefully tune their regularization parameters such that all networks perform similarly in the lightly perturbed environment. As seen in Figure 4d, when the perturbed testing environment deviates more from the training environment, the performance of the narrower policy networks (32/64 hidden units) significantly drops, while the wider networks (128/256 hidden units) are more stable. We also observe that when the policy networks are sufficiently wide (128/256), their robustness is similar.

Robotics Experiments. Additional experiments in multi-agent drone control environments can be found in Appendix E.1, which further verify the enhanced robustness that ERNIE provides.

6 Discussion

ERNIE is motivated by smoothness, but real-world environments are not always smooth. In section 3, we hypothesize that most environments are at least partially smooth, implying that smoothness can serve as useful prior knowledge while providing robustness (our experiments validate this). To increase ERNIE's flexibility, future work could adaptively select λ based on the current state to allow for state-dependent smoothness.

References

Marco A Wiering. Multi-agent reinforcement learning for traffic light control. In Proceedings of the Seventeenth International Conference on Machine Learning , pages 1151-1158, 2000.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning , pages 8707-8718. PMLR, 2020.

Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning. In International Conference on Machine Learning , pages 10092-10103. PMLR, 2020.

Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research , 58(3):595-612, 2010.

Joel Goh and Melvyn Sim. Distributionally robust optimization and its tractable approximations. Operations research , 58(4-part-1):902-917, 2010.

Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation , 17(2): 335-359, 2005.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning , pages 2817-2826. PMLR, 2017.

Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196 , 2019.

Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8522-8528. IEEE, 2019.

Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International Conference on Machine Learning , pages 23484-23526. PMLR, 2022.

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and ChoJui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems , 33:21024-21037, 2020.

Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust reinforcement learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=mwdfai8NBrJ .

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2): 257-280, 2005.

Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. Advances in neural information processing systems , 35: 32211-32224, 2022.

Yan Li, Tuo Zhao, and Guanghui Lan. First-order policy optimization for robust markov decision process. arXiv preprint arXiv:2209.10579 , 2022.

Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Leon-Garcia, and Nicolas Papernot. On the robustness of cooperative multi-agent reinforcement learning. In 2020 IEEE Security and Privacy Workshops (SPW) , pages 62-68. IEEE, 2020.

Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=sCZbhBvqQaU .

Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4213-4220, 2019.

Yudan Wang, Yue Wang, Yi Zhou, Alvaro Velasquez, and Shaofeng Zou. Data-driven robust multi-agent reinforcement learning. In 2022 IEEE 32nd International Workshop on Machine Learning for Signal Processing (MLSP) , pages 1-6, 2022.

Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research , 2023.

Songyang Han, Sanbao Su, Sihong He, Shuo Han, Haizhao Yang, and Fei Miao. What is the solution for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02705 , 2022.

Ziyuan Zhou, Guanjun Liu, and Mengchu Zhou. A robust mean-field actor-critic reinforcement learning against adversarial perturbations on agent states. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Zhigen Zhao, Simiao Zuo, Tuo Zhao, and Ye Zhao. Adversarially regularized policy learning guided by trajectory optimization. In Proceedings of The 4th Annual Learning for Dynamics and Control Conference , pages 844-857. PMLR, 23-24 Jun 2022. URL https://proceedings. mlr.press/v168/zhao22b.html .

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence , 41(8):1979-1993, 2018.

Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems , 32, 2019.

Simiao Zuo, Yue Yu, Chen Liang, Haoming Jiang, Siawpeng Er, Chao Zhang, Tuo Zhao, and Hongyuan Zha. Self-training with differentiable teacher. In Findings of the Association for Computational Linguistics: NAACL 2022 , pages 933-949, Seattle, United States, July 2022. Association for Computational Linguistics.

Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 2177-2190, Online, July 2020. Association for Computational Linguistics.

Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pretrained language model with weak supervision: A contrastive-regularized self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1063-1077, Online, June 2021. Association for Computational Linguistics.

Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, and Tuo Zhao. Adversarial regularization as stackelberg game: An unrolled optimization approach. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 6562-6577, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

Andrzej Ruszczy´ nski. Risk-averse dynamic programming for markov decision processes. Mathematical programming , 125(2):235-261, 2010.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems , 35:24611-24624, 2022.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems , 30, 2017.

John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation. Advances in neural information processing systems , 9, 1996.

Heinrich Von Stackelberg. Market structure and equilibrium . Springer Science & Business Media, 2010.

Barak A Pearlmutter and Jeffrey Mark Siskind. Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Transactions on Programming Languages and Systems (TOPLAS) , 30(2):1-36, 2008.

Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang, Zhaoran Wang, Tuo Zhao, and Hongyuan Zha. Permutation invariant policy optimization for mean-field multi-agent reinforcement learning: A principled approach. arXiv preprint arXiv:2105.08268 , 2021.

Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning , pages 5571-5580. PMLR, 2018.

Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang, Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In The World Wide Web Conference , pages 983-994, 2019.

Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Mean-field multi-agent reinforcement learning: A decentralized network approach. arXiv preprint arXiv:2108.02731 , 2021.

Ludger Rüschendorf. The wasserstein distance and approximation theorems. Probability Theory and Related Fields , 70(1):117-129, 1985.

Zhi Zhang, Jiachen Yang, and Hongyuan Zha. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv preprint arXiv:1909.10651 , 2019.

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv:1710.05465 , 10, 2017.

Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep relu neural networks in w s, p norms. Analysis and Applications , 18(05):803-859, 2020.

Haim Brezis and Haim Brézis. Functional analysis, Sobolev spaces and partial differential equations , volume 2. Springer, 2011.

Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact wasserstein distance. In Uncertainty in Artificial Intelligence , pages 433-453. PMLR, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P Schoellig. Learning to fly-a gym environment with pybullet physics for reinforcement learning of multiagent quadcopter control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 7512-7519. IEEE, 2021.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. { TensorFlow } : a system for { Large-Scale } machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16) , pages 265-283, 2016.

Peer reviews:
comments: SUMMARY:
The paper proposes a training objective Lipschitz regularization for obtaining adversarial robustness in cooperative multi-agent RL. To overcome training instablities, the paper reframes the regularization as a Stackleberg game. The paper also shows how the method can be applied to Mean-field approximations of some MARL training algorithms and carries out two experiments on the traffic and the multi-agent Particle Environments.

STRENGTHS:
- Nice background on connecting Lipschitz continuity with robustness with respect to different variables of game and policy.
- Overall solid method aiming to overcome the instability of adversarial training in the context of MARL.

WEAKNESSES:
I believe the clarity of the experiments section could be significantly improved. The figures are barely legible due to the small (font) size. Moreover, the figure caption could be improved, e.g., does "Evaluation curves" refer to "Learning curves" as in the episode reward over the training? Moreover, what does "Baseline-Gaussian" refer to?

QUESTIONS:
The method seems to be focused on Cooperative Markov Games. Can you elaborate on whether, how, or why the method would apply to other multi-agent games (competitive or other objectives, etc.)?

How does ERNIE compare to direct applications on Eq (2) and (3) in the training process?

LIMITATIONS:
Readability of the figures could be improved
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a robust Multi-Agent Reinforcement Learning (MARL) framework called ERNIE (adversarially Regularized multi-ageNt reInforcement lEarning). The goal of ERNIE is to address the issue of policy robustness in MARL algorithms. The key contributions of the paper are as follows:\n\n1. Robust MARL Framework: The authors propose ERNIE, which promotes the Lipschitz continuity of policies through adversarial regularization. This regularization enhances robustness against noisy observations, changing transition dynamics, and malicious actions of agents.\n2. Lipschitz Control: ERNIE controls the Lipschitz constant of policies, allowing them to be more resilient to variations in the environment. This control is achieved by leveraging adversarial regularization, which encourages Lipschitz continuity with respect to state observations and actions.\n3. Training Stability: The paper addresses the training instability that adversarial regularization can introduce by reformulating it as a Stackelberg game. This reformulation provides a more stable training process, making the optimization problem smoother for the defender (policy).\n4. Experimental Evaluation: The authors extensively evaluate the effectiveness of ERNIE in traffic light control and particle environments. The experiments demonstrate that ERNIE improves robustness and performance compared to non-robust MARL approaches.\n5. Extension to Mean-field MARL: ERNIE is extended to mean-field MARL, a strategy to scale up MARL algorithms. \n6. Theoretical Insights: The paper provides theoretical insights into the smoothness and robustness of policies in MARL. It shows that in environments with smooth transition and reward functions, policies' value functions are also smooth. The authors prove the existence of smooth and close-to-optimal policies in such environments and establish that policy robustness is proportional to its Lipschitz constant.\n\nSTRENGTHS:\n**Originality**: The theoretical insights into the smoothness of environments and robustness of policies look interesting to me. By assuming the smoothness of environments, we can get some good properties of robustness. This idea is very interesting to me. However, this idea is very similar to the paper [1]“****Deep Reinforcement Learning with Robust and Smooth Policy****”. \n\n[1] Shen, Q., Li, Y., Jiang, H., Wang, Z., & Zhao, T. (2020, November). Deep reinforcement learning with robust and smooth policy. In *International Conference on Machine Learning* (pp. 8707-8718). PMLR.\n\n**Quality**: The paper is very informative. I can see the authors have a lot to show readers in the paper: the theoretical results, the proposed framework, how to solve the problem efficiently, and how to adopt the proposed framework to large-scale multi-agent systems.\n\n**Clarity**: The paper appears to be clear in its presentation of the problem, the proposed solution, and the results of the experiments. The authors provide a theoretical motivation for their algorithm, explaining how Lipschitzness (smoothness) serves as a natural way to gain robustness while reducing the policy search space. The extension of ERNIE to mean-field MARL is also clearly explained.\n\n**Significance**: I believe this paper's theoretical analysis of how smoothness helps in training robust policies would help future researchers in the field of robust MARL to develop better solutions to the robustness issues.\n\nWEAKNESSES:\n**Confusing Writing**: I found the paper sometimes inconsistent in motivation, methods, and experiments, which makes me confused about what problem this paper tries to solve. In the abstract, the authors point out that we need to consider real-to-sim gaps in the deployment of MARL algorithms, I quote the original sentence there “This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment.” But later, the authors claim that their proposed method is robust to perturbations on observations and actions. I also quote the original sentences as follows: “Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization.” and “In particular, we develop an adversarial regularizer to minimize the discrepancy between each policy’s output given a perturbed observation and a non- perturbed observation.” When I dive into the methodology of this paper, I found most of the contexts are about **Robustness against observation noise with smooth policies** and **Robustness against Malicious Actions.** We can see that the abstract and some texts in the introduction mislead readers about what kind of robustness this paper tries to achieve in MARL. \n\n**Evaluations on Simple Games**: The evaluation of the proposed methods is conducted on specific tasks such as traffic light control and particle environment tasks. While these tasks are complex, they may not fully reveal insights into how the smoothness of environments affects the robustness of policies.  I would suggest running experiments on some simple matrix games or small-scale multi-agent toy examples [1] to better study how smoothness is connected to robustness. \n\n[1] Hu J, Wellman MP. Nash Q-learning for general-sum stochastic games. Journal of machine learning research. 2003;4(Nov):1039-69.\n\n**Lack of Comparison with Robust MARL Methods:** This paper claims that the proposed method is robust to perturbed actions and observations. Considering there are some existing works about robust MARL considering adversarial-purebred observations [1, 2, 3] and perturbed actions [4], should the authors compare the proposed method with at least one of these robust MARL methods?  \n\n[1] Han S, Su S, He S, Han S, Yang H, Miao F. What is the solution for state adversarial multi-agent reinforcement learning?. arXiv preprint arXiv:2212.02705. 2022 Dec 6.\n\n[2] He S, Han S, Su S, Han S, Zou S, Miao F. Robust Multi-Agent Reinforcement Learning with State Uncertainty. Transactions on Machine Learning Research. 2023 Mar 10.\n\n[3] Zhou Z, Liu G, Zhou M. A Robust Mean-Field Actor-Critic Reinforcement Learning Against Adversarial Perturbations on Agent States. IEEE Transactions on Neural Networks and Learning Systems. 2023 Jun 5.\n\n[4] Li, S., Guo, J., Xiu, J., Yu, X., Wang, J., Liu, A., ... & Liu, X. (2023). Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game. *arXiv preprint arXiv:2305.12872*.\n\n**Writing Suggestions:**\n\nThe mean-field MARL part does not tightly connect to the main text of this paper. Would it be better to make it a remark instead of a subsection? Besides, the organization of this paper makes me confused, for example, Robustness against observation noise with smooth policies (a paragraph) and Robustness against Malicious Actions (a subsection) are placed on different layers of this paper. Should they be on similar layers? As a paper studying robust MARL, in the related work, this paper uses more text to discuss robust RL works than robust MARL. The literature review of robust MARL is not comprehensive.\n\nQUESTIONS:\nIt would be great if the authors can resolve my concerns in the weakness box and the Originality concerns in the strength box. I will decide my final score after the rebuttal phase according to the authors' responses.\n\nLIMITATIONS:\nNAN", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRobust RL is attracting great attention, and lately there has been great advancements both in theory and empirically. However, there is only limited work in robust multi-agent reinforcement learning. Under some assumptions, the paper shows the existence of policies with nice properties. Building upon this, the authors propose a training framework that includes adversarial perturbation on observation in order to increase robustness of the learned policy.\n\nSTRENGTHS:\nThe main ideas behind this formulation are quite intuitive. The paper also contains insightful theoretical component to support such intuition. The algorithm is shown to be promising in simulations.\n\nWEAKNESSES:\nSee limitations.\n\nQUESTIONS:\n1. I find that calling lipschitzness smoothness quite confusing. In particular, in Definition 3.1, the \"smoothness\" is actually defined as the reward and dynamics being $L_r,L_{\\mathbb{P}}$-Lipschitz. Subsequently, in the writing, there are many places where the authors interchange the smoothness and lipschitzness, e.g., line 187-188. As far as I know, more commonly smoothness is about the gradient of a function being Lipschitz. Is there a strong reason for this particular choice of words?\n1. This recent work [1] considers a Markov game with state perturbation adversaries. How does it compare to your work?  \n[1] Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Robust multi-agent reinforcement learning with state uncertainty. Transactions on Machine Learning Research, 2023.\n1. What will be the main difficulty in extending ERNIE to continuous state and action setting?\n1. The main idea of making the learned policy Lipschitz is somewhat similar to the idea of smoothening the decision boundary. I would like to think this as the robustness in the \"average\" case rather than the worst case. Is there any work in the MARL literature that considers the worst case robustness?\n\nLIMITATIONS:\n1. In the paper survey section regarding the single-agent robust RL, I think the authors should include another important line of work: robust Markov decision process (RMDP). This distributionally robust reinforcement learning (DR-RL) framework provides a principled way to achieve robustness against perturbations in models. For example, [1,2] are considered to be the seminal theoretical works that established the framework. In this line of work, some works do not require expert knowledge of the underlying simulator. [3] is an offline DR-RL work that can even handle continous state and action empirically. [4,5] are policy gradient methods for DR-RL, and they also do not need prior expert knowledge.  On line 191-193, two DR-RL papers are included as reference but I think the authors should include some in the paper survey section as well.  \n[1] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280, 2005.  \n[2] Arnab Nilim and Laurent El Ghaoui. Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005.  \n[3] Panaganti, K., Xu, Z., Kalathil, D., & Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. Advances in neural information processing systems, 35, 32211-32224.  \n[4] Wang, Yue, and Shaofeng Zou. \"Policy gradient method for robust reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.  \n[5] Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Levy, and Shie Mannor. Policy gradient for s-rectangular robust Markov decision processes. ArXiv preprint, abs/2301.13589, 2023. URL https://arxiv.org/abs/2301.13589.  \n1. Although the authors hint that the algorithm can tackle perturbations in dynamics/transition models, this is mainly an empirical claim with no theory to corroborate.\n1. Lipschitz assumptions on reward and model are quite strong in my opinion. This is very unlikely to be true or verifiable in larger systems. Hence, the generalizability of the proposed method is limited.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to develop a robust MARL framework based on Lipschitz smoothness of the learned policy (smooth policies). The authors prove the existence of smooth and nearly-optimal policies and propose a framework (ERNIE) to learn robust MARL policy. Experiments are conducted on the two environment: Traffic light control and particle environment.\n\nSTRENGTHS:\nRobust MARL is an important problem to be addressed\n\nWEAKNESSES:\nThere seems to be many contents squeezed in this paper: theory, framework, connection to game theory (stacklebeg training) and evaluations. It will be nice to have an overview and outline at the end of Intro. \n\n1. This paper also aims to tackle a wide range of threat models if I understand correctly: state perturbation, transition perturbations etc. But why some experiments are evaluating action perturbations? \n\n2. As discussed in Lin 140-141, transition dynamics governed by physical laws (e.g. MuJoCo environment) are of interest. But why in the Experiments, the authors did not show the result of MARL MujoCo Benchmark? \n\n3. Is the ERNIE framework primiarily designed to tackle the threat model of state-perturbation? See Eq 2 of ERNIE regularizer. How can it be used to tackle the transition perturbation threat model? \n\n4. The result in Theorem 3.1 is natural when everything is smooth.\n\nQUESTIONS:\n1. Is the description of Line 48 correct? Is the robustness proprotional to Lipschitz constant or inverse proportional? \n\n2. It is proved that a smooth and close-to-optimal policy exists in smooth enviornment (Line 47-48), but is it guaranteed to obtain such a close-to-optimal policy? and how? \n\n3. Line 215-216 - the boldface and captilaized words does not correspond to ERNIE. \n\n4. Fig 3: the ERNIE's result in Fig 3 seem to be not good, only (a) seem to be outperforming baselines. \n\n5. It's still not clear to me how can the proposed framework be extended beyond smooth environment. Please provide a few example environments that is practically non-smooth and how this framework might be extended?\n\n=================== \nincrease rating from 4 to 5 in the post-rebuttal.\n\nLIMITATIONS:\nThe authors have a paragraph discussing potential future work of the method, but not necessarily the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to train robust policies in a multi-agent reinforcement learning setting via adversarial regularization. The authors showed that robust policies can be achieved by enforcing Lipschitz continuity with respect to states for robustness to noisy states and with respect to actions for noisy action by other potential agents. To enforce the Lipschitz continuity, the authors proposes an adversarial regularization term that can be added to conventional RL training objectives, which minimizes the maximum distance between the policy under a perturbed state/action and the policy under the original state/action. The authors also reformulated the minimax problem into a Stackelberg game, which further stabilize the training process. Finally, the authors also extended their adversarial regularization to the setting of mean-field MARL, which enables the training of large-scale robust marl agents. Experiments and results were provided for several MARL benchmarks.\n\nSTRENGTHS:\nThe paper presents a well-written study on a novel approach to train robust multi-agent RL policies. While the core idea of using adversarial regularization may not be entirely novel, its application to multi-agent systems and its extension to mean-field RL is an interesting, albeit expected direction of development. The authors also provided relatively comprehensive experimental results, and the discussions and remarks effectively addressed most of my major questions that I thought of when reviewing the paper. In terms of significance, I believe that this work will further the development of robust MARL systems.\n\nWEAKNESSES:\nOverall, I believe the paper could benefit further if the authors could also provide additional experiments in the extension to mean-field MARL where the authors also apply the regularizer to the action to validate their claims for the sake of completeness of experiments. In addition to that, it will also be beneficial if the authors would also provide the run time comparisons for the traffic flow environments on top of the MPE environments for completeness. Finally, if I am understand correctly, both of the experiments are employed a reward structure that is continuous and smooth. It will be more convincing if the authors could also demonstrate the effectiveness of their proposed framework on an environment with an approximately smooth vs non-smooth reward structure.\n\nQUESTIONS:\nI do not have any major questions that requires clarification.\n\nMinor comments/questions:\n\"Wang et al. [16] consider robustness against uncertain transition dynamics, but their\nalgorithm is not applied to deep MARL.\". This sentence really fall under the subsection of robust RL rather than robust MARL then.\n\nIn the supplementary material:\nFigure 6: What does STELLA represent? I am assuming its a typo.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}