SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alexander Bukharin† Yan Li† Yue Yu† Qingru Zhang† Zhehui Chen‡ Simiao Zuo§
Chao Zhang†
Songan Zhang
Tuo Zhao†
Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy's Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE's adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE.
In the past decade advances in deep neural networks and greater computational power have led to great successes for Multi-Agent Reinforcement Learning (MARL), which has achieved success on a wide variety of multi-agent decision-making tasks ranging from traffic light control [1] to StarCraft [2]. However, while much effort has been devoted to applying MARL to new problems, there has been limited work regarding the robustness of MARL policies.
Despite the limited attention paid to robustness, it is essential for MARL policies to be robust. Most MARL policies are trained in a fixed environment. Since these policies are trained solely to perform well in that environment, they may perform poorly in an environment with slightly different transition dynamics than the training environment. In addition, while agents are fed with exact state information in training, MARL policies deployed in the real world can receive inaccurate state information (e.g., due to sensor error). Finally, even a single agent acting maliciously or differently than expected can cause a chain reaction that destabilizes the whole system. These phenomena cause significant concern for the real-world deployment of MARL algorithms, where the environment dynamics and observation noise can change over time. We observe that even when the change in the environment's dynamics is small, the performance of MARL algorithms can deteriorate severely (See an example
† Georgia Institute of Technology ‡ Google § Microsoft ¶ Ford Motor Company
in Section 5). Thus there is an emerging need for MARL algorithms that are robust to changing transition dynamics, observation noise, and changing behavior of agents.
Although many robust RL methods have been proposed for the single agent case, three major barriers prevent their use for MARL. Theoretically, it is not clear if or when such methods can work for MARL. Methodologically, it is not straightforward to apply single agent robust RL methods to MARL, as single agent methods may not consider the interactions between several agents. Algorithmically, single agent robust RL algorithms are often unstable, and may not perform well when applied to inherently unstable MARL training. Therefore to learn robust MARL policies, we provide theoretical, methodological, and algorithmic contributions.
Theory. Theoretically, we first show that when the transition and reward function are smooth, a policy's value function is also smooth. In our experiments, we show that this assumption can serve as a useful prior knowledge, even if the transition function is not smooth in every state. Second, we prove that a smooth and close-to-optimal policy exists in any such environment. Third, we show that a policy's robustness is proportional to its Lipchitz constant with no smoothness assumption on the environment's smoothness. These observations advocate for using smoothness as an inductive bias to not only reduce the policy search space, but simultaneously improve the robustness of the learned policy. Finally, we prove that large neural networks are capable of approximating the target policy or Q functions with smoothness guarantees. These findings give us the key insight that in order to learn robust and high-performing deep MARL policies, we should enforce the policies' smoothness.
Method. Based on these findings, we propose a new training framework - advErsarily Regularized multiageNt reInforcement lLearning (ERNIE), that applies adversarial training to learn smooth and robust MARL policies in a principled manner. In particular, we develop an adversarial regularizer to minimize the discrepancy between each policy's output given a perturbed observation and a non-perturbed observation. This adversarial regularization gives two main benefits: Lipschitz continuity and rich data augmentation with adversarial examples. The adversarial regularization encourages the learned policies to be Lipschitz continuous, improving robustness. Augmenting the data with adversarial examples further provides robustness against environment changes. Specifically, new scenarios emerge when the environment changes, and data augmentation with adversarial examples provides a large coverage of these scenarios as long as the environment change is small. Adapting to adversarial examples during training ensures that the agents will perform reasonably even in the worst case.
To further provide robustness against the changing behaviors of a few malicious agents, we propose an extension of ERNIE that minimizes the discrepancy between the global Q-function with maliciously perturbed joint actions and non-perturbed joint actions. This regularizer encourages the policies to produce stable outputs even when a subset of agents acts sub-optimally, therefore granting robustness. Such robustness has not been considered in previous works.
Algorithm. We find that adversarial regularization can improve robust performance [3]. However, adversarial regularization can also be unstable. More concretely, conventional adversarial regularization can be formulated as a zero-sum game where the defender (the policy) and attacker (the perturbation) hold equal positions and play against each other. In this case, a small change in the attacker's strategy may result in a large change for the defender, rendering the problem ill-conditioned. Coupled with the already existing stability issues that come with training MARL algorithms, this instability issue greatly reduces the power of adversarial regularization methods for MARL.
To address this issue, we reformulate adversarial training as a Stackelberg game. In a Stackelberg game, the leader (defender) has the advantage as it knows how the follower (attacker) will react to its actions and can act accordingly. This advantage essentially makes the optimization problem smoother for the defender, leading to a more stable training process.
Extension to Mean-field MARL. We further demonstrate the general applicability of ERNIE by developing its extension to robustify mean-field MARL algorithms. The mean-field approximation has been widely received as a practical strategy to scale up MARL algorithms while avoiding the curse of many agents [4]. However, as mean-field algorithms are applied to real-world problems, it is essential to develop robust versions. To facilitate policy learning that is more robust, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization [5; 6].
To demonstrate the effectiveness of the proposed framework, we conduct extensive experiments that evaluate the robustness of ERNIE on traffic light control and particle environment tasks. Specifically,
we evaluate the robustness of MARL policies when the evaluation environment deviates from the training environment. These deviations include observation noise, changing transition dynamics, and malicious agent actions. The results show that while state-of-the-art MARL algorithms are sensitive to small changes in their environment, the ERNIE framework enhances the robustness of these algorithms without sacrificing efficiency.
Contributions. We remark that adversarial regularization has been developed for single-agent RL, but never for MARL [3]. Our contribution in this paper has four aspects: (1) advances in theoretical understanding (2) development of new regularizers for MARL (3) new algorithms for stable adversarial regularization in MARL (4) comprehensive experiments in a number of environments.
In this section, we introduce the necessary background for MARL problems together with related literature. We consider the setting of cooperative MARL, where agents work together to maximize a global reward.
- Cooperative Markov Games. We consider a partially observable Markov game  $\langle S, \mathcal{O}^N, \mathcal{A}^N, \mathcal{P}, \mathcal{R}, N, \gamma \rangle$  in which a set of agents interact within a common environment. We let  $S \subseteq \mathbb{R}^S$  denote the global state space,  $\mathcal{O} \subseteq \mathbb{R}^O$  denote the observation space for each of the  $N$  agents,  $\mathcal{A} \subseteq \mathbb{R}^A$  denote the action space,  $\mathcal{P}: S \times \mathcal{A} \mapsto S$  denote the transition kernel,  $\gamma$  denotes the discount factor, and  $\mathcal{R}: S \times \mathcal{A} \mapsto \mathbb{R}^N$  denotes the reward function. At every time step  $t$ , each of the  $N$  agents selects an action according to its policy, which can be stochastic or deterministic. Then, the system transitions to the next state according to the transition kernel and each agent receives a reward  $r_{i,t}$ . We denote the global reward at time  $t$  as  $r_t^g$ . The goal of each agent is to find a policy that maximizes the discounted sum of its own reward,  $\sum_{t \geq 0} \gamma^t r_{i,t}$ .
- Robust RL. In recent years many single agent robust RL techniques have been proposed. Most of these methods use information about the underlying simulator to train agents over a variety of relevant environment settings [7; 8; 9; 10; 11]. Although these methods can provide robustness against a wide range of environment changes, they suffer from long training times and require expert knowledge of the underlying simulator, which is not practical. Another direction of research focuses on perturbation based methods [3; 12]. Perturbation based methods train the policy to be robust to input perturbations, encouraging the policy to act reasonably in perturbed or previously unseen states. [13] certify robustness by adding smoothing noise to the state; it is not clear how this affects the learned policy's optimality. Another related line of work [14; 15; 16; 17] studies robust markov decision processes and provides a principled way to learn robust policies. However, such methods often require strict assumptions on the perturbation/uncertainty. Inspiring our work, [3] proposes to learn a smooth policy in single agent RL, but they do so to reduce training complexity rather than increase robustness and provide no theoretical justification for their method. Instead, we theoretically connect smoothness to robustness, extend perturbation based methods to MARL, and develop a more stable perturbation computation technique, and develop an extension to mean-field MARL.
- Robust MARL. Recently, some works have studied the robustness of MARL systems. Lin et al. [18] studies how to attack MARL systems and finds that MARL systems are vulnerable to attacks on even a single agent. Zhang et al. [19] develop a framework to handle MARL with model uncertainty by formulating MARL as a robust Markov game. However, their proposed method only considers uncertainty in the reward function, while this article focuses on robustness to observation noise and changing transition dynamics. Li et al. [20] modify the MADDPG algorithm to consider the worst-case actions of the other agents in continuous action spaces with the M3DDPG algorithm. M3DDPG aims to grant robustness against the actions of other agents, which is less general than the robustness against observation noise, changing transition dynamics, and malicious agents that our method aims for. Wang et al. [21] consider robustness against uncertain transition dynamics, but their algorithm is not applied to deep MARL. More recently, He et al. [22]; Han et al. [23] introduces the concept of robust equilibrium and proposes to learn an adversarial policy to perturb each agent's observations. Finally Zhou et al. [24] propose to learn robust policies by minimizing the cross-entropy loss between agent's actions in non-perturbed states and perturbed states.
The ERNIE framework is also related to several existing works which use similar adversarial training methods but target different domains such as trajectory optimization [25], semi-supervised learning [26; 27; 28], fine-tuning language models [29; 30], and generalization in supervised learning [31].
3 From Lipschitz Continuity to Robustness
This section presents the theoretical motivation for our algorithm by showing that Lipschitzness (smoothness) serves as a natural way to gain robustness, while reducing the policy search space. We start by observing that certain natural environments exhibit smooth transition and reward functions, especially when the transition dynamics are governed by physical laws (e.g., MuJuCo environment [32], Pendulum [33]).* Formally, this is stated as the following.
Definition 3.1. Let  $\mathcal{S} \subseteq \mathbb{R}^d$ . We say the environment is  $(L_r, L_{\mathbb{P}})$ -smooth, if the reward function  $r: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ , and the transition kernel  $\mathbb{P}: \mathcal{S} \times \mathcal{S} \times \mathcal{R}$  satisfy
$$
\left| r (s, a) - r (s ^ {\prime}, a) \right| \leq L _ {r} \left\| s - s ^ {\prime} \right\| \mathrm {a n d} \left\| \mathbb {P} (\cdot | s, a) - \mathbb {P} (\cdot | s ^ {\prime}, a) \right\| _ {1} \leq L _ {\mathbb {P}} \left\| s - s ^ {\prime} \right\|,
$$
for  $(s, s', a) \in S \times S \times \mathcal{A}$ .  $\|\cdot\|$  denotes a metric on  $\mathbb{R}^d$ . We say a policy  $\pi$  is  $L_{\pi}$ -smooth if
$$
\left\| \pi (\cdot | s) - \pi (\cdot | s ^ {\prime}) \right\| _ {1} \leq L _ {\pi} \left\| s - s ^ {\prime} \right\|.
$$
Without loss of generality, we assume  $|r(s,a)| \leq 1$  for any  $(s,a) \in S \times \mathcal{A}$ . We then present our theory. Due to the space limit, we defer all technical details to the appendix.
- From smooth environments to smooth values. We proceed to show that if the environment is smooth, then the value functions for smooth policies are also smooth.
Theorem 3.1. Suppose the environment is  $(L_r, L_{\mathbb{P}})$ -smooth. Then the  $Q$ -function of any policy  $\pi$ , defined as
$$
Q ^ {\pi} (s, a) = \mathbb {E} ^ {\pi} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) | s _ {0} = s, a _ {0} = a \right], \forall (s, a),
$$
is Lipschitz continuous in the first argument. That is,
$$
\left| Q (s, a) - Q \left(s ^ {\prime}, a\right) \right| \leq L _ {Q} \| s - s ^ {\prime} \|, \tag {1}
$$
where  $L_{Q} \coloneqq L_{r} + \gamma L_{\mathbb{P}} / (1 - \gamma)$ . Suppose in addition the policy is  $L_{\pi}$ -smooth. Then the value function, defined as
$$
V ^ {\pi} (s) = \mathbb {E} ^ {\pi} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r (s _ {t}, a _ {t}) | s _ {0} = s \right], \forall s,
$$
is Lipschitz continuous. That is,
$$
\left| V ^ {\pi} (s) - V ^ {\pi} \left(s ^ {\prime}\right) \right| \leq L _ {V} \| s - s ^ {\prime} \|,
$$
where  $L_{V}\coloneqq L_{\pi} / (1 - \gamma) + L_{Q}$
In view of Theorem 3.1, it is clear that whenever the environment and the policy are smooth, then the value functions are also smooth. A natural and important follow-up question to ask is whether this claim holds in the reverse direction. More concretely, we ask whether it is reasonable to seek a policy that is also smooth with respect to the state while maximizing the reward. If the claim holds true, then seeking a smooth policy can serve as an efficient and unbiased prior knowledge, that can help us reduce the policy search space significantly, while still guaranteeing that we are searching for high-performing policies.
- Existence of smooth and nearly-optimal policies. The following result shows that for any  $\epsilon > 0$ , there exists an  $\epsilon$ -optimal policy that is  $\mathcal{O}(L_Q / \epsilon)$  smooth, where  $L_{Q}$  defined in Theorem 3.1 only depends on the smoothness of reward and transition. This structural observation naturally suggests seeking a smooth policy for smooth environments.
Theorem 3.2. Suppose the environment is  $(L_r, L_{\mathbb{P}})$ -smooth. Then for any  $\epsilon > 0$ , there exists an  $\epsilon$ -optimal policy  $\pi$  that is also smooth, i.e.,
$$
V ^ {*} (s) - V ^ {\pi} (s) \leq \frac {2 \epsilon}{1 - \gamma}, \forall s \in \mathcal {S} a n d \| \pi (\cdot | s) - \pi (\cdot | s ^ {\prime}) \| _ {1} \leq | \mathcal {A} | \log | \mathcal {A} | L _ {Q} \| s - s ^ {\prime} \| / \epsilon ,
$$
where  $L_{Q}$  is defined as in Theorem 3.1.
Notably, the proof of Theorem 3.2 relies on the key observation that any smooth  $Q$ -function satisfying (1) can be fed into the softmax operator, which induces a smooth policy. This observation also provides a way for value-based methods (e.g., Q-learning) to learn a smooth policy. Namely, one can first learn a smooth surrogate of the optimal  $Q$ -function, and then feed the learned surrogate into the softmax operator to induce a close-to-optimal policy that is also smooth.
* See Remark 3.2 for discussions when the smoothness property holds approximately.
- Robustness against observation noise with smooth policies. We have so far established that smooth policies naturally exist in a smooth environment as close-to-optimal policies, and thus smoothness serves as a strong prior for policy search. We will further demonstrate that the benefits of smooth policy go beyond boosting learning efficiency, by bringing in the additional advantage of robustness against observation uncertainty.
Theorem 3.3. Let  $\pi(a|s)$  be  $L_{\pi}$ -smooth policy. For any perturbation sequence  $\{\delta_s^t\}_{t \geq 0, s \in S}$ , define a perturbed policy (non-stationary)  $\widetilde{\pi} = \{\widetilde{\pi}_t\}_{t \geq 0}$  by
$$
\widetilde {\pi} _ {t} (a | s) = \pi (a | s + \delta_ {s} ^ {t}),
$$
with  $\| \delta_s^t\| \leq \epsilon$  for all  $t\geq 0$ . Accordingly, define the value function of the non-stationary policy  $\widetilde{\pi}$
$$
V ^ {\widetilde {\pi}} (s) = \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \mid s _ {0} = s, a _ {t} \sim \widetilde {\pi} _ {t} (\cdot \mid s _ {t} + \delta_ {s _ {t}} ^ {t}), \right.
$$
$$
s _ {t + 1} \sim \mathbb {P} (\cdot | s _ {t}, a _ {t}) ].
$$
Then we have  $|V^{\pi}(s) - V^{\widetilde{\pi}}(s)| \leq \frac{2L_{\pi}\epsilon}{(1 - \gamma)^2}$ , Similarly, we have
$$
\left| Q ^ {\pi} (s, a) - Q ^ {\widetilde {\pi}} (s, a) \right| \leq \frac {2 L _ {\pi} \epsilon}{(1 - \gamma) ^ {2}},
$$
where  $Q^{\tilde{\pi}}$  is defined similarly as  $V^{\tilde{\pi}}$ .
Theorem 3.3 establishes the following fact: for a discounted MDP with finite state and finite action space, the value of the policy when providing the perturbed state is close to the value of the policy when given the non-perturbed state, provided the policy is Lipschitz continuous in its state. As an important implication, the learned smooth policy will be robust in the state observation, in the sense that the accumulated reward will not deteriorate much when noisy, or even adversarially constructed state observations are given to the policy upon decision making.
We emphasize that Theorem 3.3 holds without any smoothness assumption on the transition or the reward function. It should also be noted that there are various notions of robustness in MDP, e.g., robustness against changes in transition kernel [34; 17], which we defer as future investigations.
Before we conclude this section, we briefly remark on certain generality of our discussion.
Remark 3.1 (Applicability to MARL). The discussions in this section do not depend on the size of the state space, and apply to the multi-agent setting without any change. To see this, note that our discussion holds for any discrete state-action space. Setting  $S$  as the joint state space and  $\mathcal{A}$  as the joint action space, then the obtained results trivially carry over to the cooperative MARL setting.
Remark 3.2 (Environments with approximate smoothness). Many environments are partially smooth, in the sense that the transition or the reward is non-smooth only on a small fraction of the state space. Typical examples include the Box2D environment [33], where the agent receives smooth reward when in non-terminal states (airborne for Lunar Lander), and receives a lump-sum reward in the terminal state (land/crash) – a vanishing fraction of the entire state space. Given the environment being largely smooth, it should be expected that for most states the optimal policy is locally smooth. Consequently, inducing a smoothness prior serves as a natural regularization to constrain the search space when solving these environments, without incurring a large bias.
Remark 3.3 (Non-smooth environments). From the perspective of robust statistics, achieving robustness often necessitates a certain level of smoothness in the learned policy, regardless of the smoothness of the optimal policy. In scenarios where the environment itself is non-smooth, the optimal policy can also be non-smooth. However, it is important to note that such non-smooth optimal policies are typically not robust. This means that by trading-off between the approximation bias and robustness, the smooth policy learnt by out method has the potential to outperform non-smooth policies in perturbed environments.
In this section, we propose our robust MARL framework, advErsarily Regularized multiageNt reInforcement lEarning (ERNIE).
4.1 Learning Robust Policy with ERNIE
Section 3 shows that the robustness of a policy depends on its Lipschitz constant. Therefore, in ERNIE we propose to control the Lipschitz constant of each policy with adversarial regularization.
Given a policy  $\pi_{\theta_k}$ , where  $k$  is the agent index, the ERNIE regularizer is defined by
$$
R _ {\pi} \left(o _ {k}; \theta_ {k}\right) = \max  _ {| | \delta | | \leq \epsilon} D \left(\pi_ {\theta_ {k}} \left(o _ {k} + \delta\right), \pi_ {\theta_ {k}} \left(o _ {k}\right)\right). \tag {2}
$$
Here  $\delta(o_k, \theta_k)$  is a perturbation adversarially chosen to maximize the difference between the policy's output for the perturbed observation  $o_k + \delta(o_k, \theta_k)$  and the original observation  $o_k$ . In this case  $\epsilon$  controls the perturbation strength and  $||\cdot||$  is usually taken to be the  $\ell_2$  or  $\ell_{\infty}$  norm. Note that  $R_{\pi}(o_k; \theta_k)$  essentially measures the local Lipschitz smoothness of policy function  $\pi_{\theta}$  around the observation  $o_k$ , defined in metric  $D(\cdot, \cdot)$ . Therefore minimizing  $R_{\pi}(o_k; \theta_k)$  will encourage the policy to be smooth.
Regularization (2) allows straightforward incorporation into MARL algorithms that directly perform policy search. For actor-critic based policy gradient methods, the regularizer (2) can be directly included into the objective for updating the actor (policy) networks. When optimizing stochastic policies (e.g., MAPPO [35]),  $D$  can be taken to be the KL divergence and for deterministic policies (e.g., MADDPG [36] or Q-learning [37]), we set  $D$  to be the  $\ell_p$  norm.