SlowGuess/NeurIPS_2023_OCR huggingface dataset:
More concretely, let  $\mathcal{L}(\theta)$  denote the policy optimization objective, i.e., the negative weighted value function of the policy. We then augment  $\mathcal{L}(\theta)$  with (2), and minimize the regularized objective
$$
\min  _ {\theta} \mathcal {F} (\theta) = \mathcal {L} (\theta) + \lambda \sum_ {n = 1} ^ {N} \mathbb {E} _ {\pi_ {n}} \left[ R _ {\pi} \left(o _ {n}; \theta_ {n}\right) \right], \tag {3}
$$
where  $\lambda$  is a hyperparameter. We remark that Shen et al. [3] has explored similar regularization for single-agent RL (with a goal of improving sample efficiency), but as we explain in sections 4.2, 4.3, and 4.4, successful application to MARL robustness is highly non-trivial.
4.2 Stackelbeg Training with Differentiable Adversary
Although accurately solving (3) will result in a high-performing and robust policy, we note that (3) is a nonconvex-nonconcave minimax problem. In practice, we can use multiple steps of projected gradient ascent to approximate the worse-case state perturbation  $\delta(o_k, \theta_k)$ , followed by one-step gradient descent for updating the policies/Q-function. Even though this optimization method already significantly improves robustness over the baseline algorithms, we found that the training process could be quite unstable. We hypothesize that the intrinsic instability of MARL algorithms due to simultaneous updates of multiple agents is greatly amplified by the non-smooth landscape of adversarial regularization.
To promote a more stable straining process, we propose to reformulate adversarial training in ERNIE as a Stackelberg game. The reformulation defines adversarial regularization as a leader-follower game [38]:
$$
R _ {\pi} (o, \delta_ {\theta} ^ {K} (o); \theta) = D \left(\pi_ {\theta} (o + \delta^ {K} (o, \theta)), \pi_ {\theta} (o)\right) \tag {4}
$$
$$
\text {s . t .} \delta^ {K} (o, \theta) = \underbrace {U _ {\theta} \circ U _ {\theta} \circ \cdots \circ U _ {\theta}} _ {\text {K - f o l d c o m p o s i t i o n}} (\delta^ {0} (o, \theta)).
$$
Here  $\circ$  denotes the operator composition (i.e  $f\circ g = f(g(\cdot))$ ), and
$$
\delta^ {k + 1} (o, \theta) = U _ {\theta} \left(\delta^ {k} (o, \theta)\right) = \delta^ {k} (o, \theta) + \eta \nabla_ {\delta} D \left(\pi_ {\theta} \left(o + \delta^ {k} (o, \theta)\right), \pi_ {\theta} (o)\right)
$$
is a one-step gradient ascent for maximizing the divergence of the perturbed and original observation.
Compared to the vanilla adversarial regularizer in (2), the perturbation  $\delta$  is treated as a function of the model parameter  $\theta$ . This formulation allows the leader  $(\theta)$  to anticipate the action of the follower  $(\delta)$ , since the follower's response given observation  $o$  is fully specified by  $\delta^K(o, \theta)$ . This structural anticipation effectively produces an easier and smoother optimization problem for the leader  $(\theta)$ , whose gradient, termed Stackelberg gradient, can be readily computed by
$$
\frac {\partial R _ {\pi} (o , \delta_ {\theta} ^ {K} (o) ; \theta)}{\partial \theta} = \underbrace {\frac {\partial R _ {\pi} (o , \delta^ {K} , \theta)}{\partial \theta}} _ {\text {l e a d e r}} \quad + \underbrace {\frac {\partial R _ {\pi} (o , \delta^ {K} (\theta) , \theta)}{\partial \delta^ {K} (\theta)} \frac {\delta^ {K} (\theta)}{\partial \theta}} _ {\text {l e a d e r - f o l l o w e r i n t e r a c t i o n}}
$$
Note that the gradient used in (3) only contains the "leader" term, such that interaction between the model  $\theta$  and the perturbation  $\delta$  is ignored. The computation of the Stackelberg gradient can be reduced to Hessian vector multiplication using finite difference method [39], which only requires two
backpropogations and extra  $\mathcal{O}(d)$  complexity operation. Thus no significant computational overhead is introduced for solving (4).
The benefit of Stackelberg training for MARL is twofold. First, a smoother optimization problem results in a more stable training process. This extra stability is essential given the inherent instability of MARL training. Second, giving the policy  $\theta$  priority over the attack  $\delta$  during the training process allows for a better training data fit than normal adversarial training allows. This better fit allows the MARL policies trained with Stackelberg training to perform better in lightly perturbed environments than those trained with normal adversarial regularization.
4.3 Robustness against Malicious Actions
Given the complex interactions of agents within of a multi-agent system, a robust policy for any given agent should meet the criterion that the action made is not overly dependent on any small subset of agents. This is particularly the case when the agents are homogeneous in nature [4; 40], and thus there should be no notion of coreset agents in the decision-making process that could heavily influence the actions of other agents. We proceed to show how ERNIE could be adopted to induce such a notion of robustness.
The core idea of ERNIE for this scenario is to encourage policy/Q-function smoothness with respect to joint actions. Similar to our treatment in (2), we now seek to promote learning a Q-function that yields a consistent value when perturbing the actions for any small subset of agents. Specifically, for discrete action space, we define a regularizer on the global Q-function as
$$
R _ {\omega} ^ {A} (s, \mathbf {a}) = \max  _ {D (\mathbf {a}, \mathbf {a} ^ {\prime}) \leq K} \| Q (s, \mathbf {a}; \omega) - Q (s, \mathbf {a} ^ {\prime}; \omega) \| _ {2} ^ {2}, \tag {5}
$$
where  $D(\mathbf{a}, \mathbf{a}') = \sum_{i} I(\mathbf{a}_i \neq \mathbf{a}'_i)$ . The regularizer (5) seeks to compute the worst subset of changed actions with cardinality less than  $K$ . For continuous action spaces, one could replace the metric  $\bar{D}$  in (5) by a differentiable metric defined over the action space (e.g.,  $\| \cdot \|_2$ -norm), and then evaluate the regularizer with projected gradient ascent.
To evaluate the adversarial regularizer for the discrete action space, we propose to solve (5) in a greedy manner by finding the worst-case change in the action of a single agent at a time, until the action of  $K$  agents is changed. Specifically, at each training step, we search through all the agents/actions and then pick the actions that produce the top-  $K$  changes in the Q-function, resulting in a  $\mathcal{O}(|\mathcal{A}| * N * K)$  computation. Our complete algorithm can be found in Appendix H, and we find that in our numerical study, perturbing the action of a single agent ( $K = 1$ ) is sufficient for increased robustness.
Similar to the regularizer in (2), the regularizer in (5) provides the benefits of Lipschitz smoothness (with respect to the Hamming distance) and data augmentation with adversarial examples. If the behavior of a few agents changes (either maliciously or randomly), the behavior of policies trained by conventional methods may change drastically. On the other hand, policies trained by our method will continue to make reasonable decisions, resulting in more stable performance (see section 5).
4.4 Extension to Mean-field MARL
MARL algorithms have been known to suffer from the curse of many agents [4], as the search space of policies and value functions grows exponentially w.r.t. the number of agents. A practical approach to tackle this challenge of scale is to adopt the mean-field approximation, which views each agent as realizations from a distribution of agents. This distributional perspective requires a distinct treatment of ERNIE applied to the mean-field setting.
Mean-field MARL avoids the curse of many agents by approximating the interaction between each agent and the global population of agents with that of an agent and the average agent from the population. In particular, we can approximate the action-value function of agent  $j$  as  $Q^{j}(\mathbf{s},\mathbf{a}) = Q^{j}(s_{j},d_{s},a_{j},\bar{a}_{j})$ , where  $\mathbf{a}$  is the global joint action,  $\mathbf{s}$  is the global state,  $\bar{a}_j$  is the average action of agent  $j$ 's neighbors, and  $d_{s}$  is the empirical distribution of states over the population. Such an approximation has found widespread applications in practical MARL algorithms [41; 42; 40; 43], and can be motivated in a principled fashion for agents of homogeneous nature [4; 40].
To learn robust and scalable policies, we extend ERNIE to the mean-field setting by applying adversarial regularization to the approximation terms  $d_{s}$  and  $\bar{a}^j$ . It is important to note that as the terms  $d_{s}$  and  $\bar{d}_s^\prime$  represent distributions over states, we bound the attack by the Wasserstein distance [44]. In what follows we only apply the regularizer to  $d_{s}$  for simplicity. This leads to a new regularizer
defined over the mean field state
$$
R _ {\mathcal {W}} ^ {Q} (s; \theta) = \max _ {\mathcal {W} (d _ {s} ^ {\prime}, d _ {s}) \leq \epsilon} \sum_ {a \in \mathcal {A}} \| Q _ {\theta} (s, d _ {s} ^ {\prime}, a) - Q _ {\theta} (s, d _ {s}, a) \| _ {2} ^ {2},
$$
where  $\mathcal{W}$  denotes the Wasserstein distance metric. Since the explicit Wasserstein constraint may be difficult to optimize in practice, we can instead enforce the constraint through regularization, as displayed in Appendix C.
We conduct extensive experiments to demonstrate the effectiveness of our proposed framework. In each environment, we evaluate MARL algorithms trained by the ERNIE framework against baseline robust MARL algorithms. To evaluate robustness, we train MARL policies in a non-perturbed environment and evaluate these policies in a perturbed environment. The reported results are gathered over five runs for each algorithm. Given the space limit, we put additional results in Appendix E.
Traffic Light Control. In this scenario, cooperative agents learn to minimize the total travel time of cars in a traffic network. We use QCOMBO [45] and COMA [46] (results in appendix E) as our baseline algorithms and conduct experiments using the Flow framework [47]. We train the MARL policies on a two-by-two grid (four agents). We then evaluate the policies on a variety of realistic environment changes, including different car speeds, traffic flows, network topologies, and observation noise. In each setting, we plot the reward for policies trained with ERNIE, the baseline algorithm (QCOMBO), and another baseline where the attack  $\delta$  is generated by a Gaussian random variable (Baseline-Gaussian, see Appendix I). Implementation details can be found in Appendix G.
Figure 1, 2a, and 2b show that the baseline algorithm is vulnerable to small changes in the training environment (higher reward is better). On the other hand, ERNIE achieves more stable reward on each environment change. This observation confirms that the ERNIE framework can improve robustness against observation noise and changing transition dynamics. The Gaussian baseline performs well on some environment changes, like when the observations are perturbed by Gaussian noise, but performs poorly on other environment changes, like when the car speed is changed. We hypothesize that while some environment changes may be covered by Gaussian perturbations, other environment changes are unlike Gaussian perturbations, resulting in a poor performance from this baseline.
Robustness Against Malicious Actions. We also evaluate the extension of ERNIE to robustness against changing agent behavior, which we refer to as ERNIE-A. To change agent behavior, we adversarially change the action of a randomly selected agent a small percentage of the time, i.e.  $5\%$  or  $3\%$  of the time. As can be seen in Figures 2c and 2d, the two baseline algorithms perform poorly when some agent's behavior changes. In contrast, ERNIE-A is able to maintain a higher reward.
Particle Environments. We evaluate ERNIE on the cooperative navigation, predator-prey, tag, and cooperative communication tasks. In each setting, we investigate the performance of the baseline algorithm (MADDPG), ERNIE, M3DDPG, and the baseline-gaussian in environments with varying levels of observation noise. We also compare ERNIE to the RMA3C algorithm proposed in [23]. In Figure 3, we find ERNIE performs better or equivalently than MADDPG in all settings. Surprisingly, M3DDPG can provide some robustness against observation noise, even though it is designed to provide robustness against malicious actions.
Mean-field MARL. We evaluate the performance of the mean-field ERNIE extension on the cooperative navigation task [36] with different numbers of agents. We compare the performance of the baseline algorithm, ERNIE, and M3DDPG under various levels of observation noise. We use
mean-field MADDPG as our baseline and follow the implementation of [40]. As can be seen in Figure 4, ERNIE displays a higher reward and a slower decrease in performance across noise levels.
Hyperparameter Study. We investigate the sensitivity of ERNIE to the hyperparameters  $K$  (the number of attack steps) and  $\epsilon$  (the perturbation strength). We plot the performance of different hyperparameter settings in the traffic light control task with perturbed car speeds on three random seeds. From Figures 5c and 5d we can see that adversarial training  $(K > 0)$  outperforms the baseline  $(K = 0)$  for all  $K$ . Similarly, we can see that all values of  $\epsilon$  outperform the baseline  $(\epsilon = 0)$ . This indicates that ERNIE is robust to different hyperparameter settings of  $\epsilon$  and  $K$ .
Sensitivity and Ablation Study. The advantage of the ERNIE framework goes beyond improving the mean reward. To show this, we evaluate 10 different initializations of each algorithm in two traffic environments: one with different speeds and another environment with observation noise. We then sort the cumulative rewards of the learned policies and plot the percentiles in Figures 5a and 5b. Although the best-case performance is the same for all algorithms, ERNIE significantly improves the
robustness to failure. As an ablation, we evaluate the effectiveness of ERNIE with and without the Stackelberg formulation of adversarial regularization (ST). ERNIE displays better performance in both settings, indicating that Stackelberg training can lead to a more stable training process. Ablation experiments in other environments can be found in Appendix E.4.
Robustness and Network Width. In Appendix B, we show that in order to learn a robust policy with ERNIE, we should use a sufficiently wide neural network. Therefore, we evaluate the robust performance of ERNIE using policy networks with 32, 64, 128, and 256 hidden units. We carefully tune their regularization parameters such that all networks perform similarly in the lightly perturbed environment. As seen in Figure 4d, when the perturbed testing environment deviates more from the training environment, the performance of the narrower policy networks (32/64 hidden units) significantly drops, while the wider networks (128/256 hidden units) are more stable. We also observe that when the policy networks are sufficiently wide (128/256), their robustness is similar.
Robotics Experiments. Additional experiments in multi-agent drone control environments can be found in Appendix E.1, which further verify the enhanced robustness that ERNIE provides.
ERNIE is motivated by smoothness, but real-world environments are not always smooth. In section 3, we hypothesize that most environments are at least partially smooth, implying that smoothness can serve as useful prior knowledge while providing robustness (our experiments validate this). To increase ERNIE's flexibility, future work could adaptively select  $\lambda$  based on the current state to allow for state-dependent smoothness.
[1] Marco A Wiering. Multi-agent reinforcement learning for traffic light control. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 1151-1158, 2000.
[2] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
[3] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pages 8707-8718. PMLR, 2020.
[4] Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning. In International Conference on Machine Learning, pages 10092–10103. PMLR, 2020.
[5] Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595-612, 2010.
[6] Joel Goh and Melvyn Sim. Distributionally robust optimization and its tractable approximations. Operations research, 58(4-part-1):902–917, 2010.
[7] Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2): 335-359, 2005.
[8] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826. PMLR, 2017.
[9] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196, 2019.
[10] Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 8522-8528. IEEE, 2019.
[11] Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International Conference on Machine Learning, pages 23484-23526. PMLR, 2022.
[43] Haotian Gu, Xin Guo, Xiaoli Wei, and Renyuan Xu. Mean-field multi-agent reinforcement learning: A decentralized network approach. arXiv preprint arXiv:2108.02731, 2021.
[44] Ludger Ruschendorf. The Wasserstein distance and approximation theorems. Probability Theory and Related Fields, 70(1):117-129, 1985.
[45] Zhi Zhang, Jiachen Yang, and Hongyuan Zha. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv preprint arXiv:1909.10651, 2019.
[46] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[47] Cathy Wu, Aboudy Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv:1710.05465, 10, 2017.
[48] Ingo Guhring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep relu neural networks in w s, p norms. Analysis and Applications, 18(05):803-859, 2020.
[49] Haim Brezis and Haim Brezis. Functional analysis, Sobolev spaces and partial differential equations, volume 2. Springer, 2011.
[50] Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact Wasserstein distance. In Uncertainty in Artificial Intelligence, pages 433-453. PMLR, 2020.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[52] Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P Schoellig. Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7512-7519. IEEE, 2021.
[53] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[54] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[55] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. {TensorFlow}: a system for {Large-Scale} machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265-283, 2016.