Probe Into Multi-agent Adversarial Reinforcement Learning through Mean-Field Optimal Control
Authors: Ziming Wang
Fengxiang He
Bohan Wang
Die Gan
Dacheng Tao
Venue: Submitted to ICLR 2023 (2023)

Abstract: Multi-agent adversarial reinforcement learning (MaARL) has shown promise in solving adversarial games. However, the theoretical tools for MaARL's analysis is still elusive. In this paper, we take the first step to theoretically understanding MaARL through mean-field optimal control. Specifically, we model MaARL as a mean-field quantitative differential game between two dynamical systems with implicit terminal constraints. Based on the game, we respectively study the optimal solution and the generalization of the fore-mentioned game. First of all, a two-sided extremism principle (TSEP) is then established as a necessary condition for the optimal solution of the game. We further show that TSEP is also sufficient given that the terminal time is sufficiently small. Secondly, based on the TSEP, a generalization bound for MaARL is proposed. The bound does not explicitly rely on the dimensions, norms, or other capacity measures of the model, which are usually prohibitively large in deep learning.

Under review as a conference paper at ICLR 2023
PROBE INTO MULTI-AGENT ADVERSARIAL REINFORCEMENT LEARNING THROUGH MEAN-FIELD OPTIMAL CONTROL
Anonymous authors Paper under double-bline review
ABSTRACT
Multi-agent adversarial reinforcement learning (MaARL) has shown promise in solving adversarial games. However, the theoretical tools for MaARL's analysis is still elusive. In this paper, take the first step to theoretically understand MaARL through mean-field optimal control. Specifically, we model MaARL as mean-field quantitative differential game between two dynamical systems with implicit terminal constraints. Based on the game, we respectively study the opti- mal solution and the generalization of the fore-mentioned game. We first establish a two-sided extremism principle (TSEP) as necessary condition for the opti- mal solution of the game. We then show that this TSEP also sufficient given that the terminal time is sufficiently small. Based on the TSEP, a generalization bound for MaARL is further proposed. This bound does not explicitly rely on the dimensions, norms, or other capacity measures of the model, which are usually prohibitively large in deep learning. To our best knowledge, this the first work the theory of MaARL.
INTRODUCTION
Reinforcement learning (RL) (Sutton 1988: Sutton and Barto 1998 Barto et al. 1991). aimed at single-agent environments, has been successfully deployed in many application areas, including ethology (Dayan and Daw| 2008), economics (Jasmin al., 201 I), psychology (Leibo al. 2018), and system control (Arel et al. 2010)). It studies how artificial systems learn to predict the optimal action according to the current state. The agent in the system determines the best course of action to maximize its rewards, which also moves its state to the next state.
However, real-world RL agents inhabit natural environments also populated by other agents. Agents in these environments interact with each other and modify each other's ' rewards via their actions. Based on this point, multi-agent adversarial reinforcement learning (MaARL) is proposed by |tther and Veloso (2003), in which adversarial neural networks employed for solving games in adver- sarial environments (Mandlekar et al.] (017). MaARL is well suited for multi-party game problems such autonomous driving (Behzadan and Munir 2019: Pan al. 2019), Al gaming (Mandlekar cet al. 2017 Pinto et al., 2017 Zhang et al. 2020), and auction games (Bichler et al. [2021). More- over, adversarial neural networks can also improve feature robustness and sample efficiency (Ma)
et al. 2018).
Despite the empirical popularity of MaARL, its theoretical understanding remains blank. We tribute such gap between theory and practice to the lack of analysis tool: the optimization objective defined by dynamical system, which too complex to analyze directly.
In this paper, we aim to provide new theoretical tools for the analysis of MaARL. We probe into MaARL from the view of mean-field optimal control. Specifically, our contributions can be summa- rized as follows:
1. We propose to model MaARL as a mean-field quantitative differential game (Pontryagin 1985); and thus, its corresponding training process is regarded as how to achieve the op- timal control of this game. The mean-field two-sided extremism principle (TSEP) (Guo et al. 2005) is then presented, which relies on the loss function and terminal constraints.

---

Under review as a conference paper at ICLR 2023
This mean-field TSEP serves the necessary conditions of the convergence (or equiva- lently, the optimality) of the mean-fich. quantitative differential game; when the terminal time small enough, this mean- ficld TSEP is also unique solution, and thus serves the sufficient conditions of the convergence.
2. The optimal objective function value is characterized by the viscosity solution (F et al. 2019) of a mean-field Hamiltonn-acobi-Issas (HJI) equation (Guo et al. (005). We then prove that this viscosity solution is unique. The HJI equation gives a global characterization of adversarial reinforcement learning, while the previously given mean-field TSEP a local special case.
3. Based on the TSEP, generalization enror bound for MaARL is proved. The bound is of the order (L)/VN), where N is the number of samples. They do not explicitly rely on the dimensions, norms, or other capacity measures of the network parameter, which are usually prohibitively large in deep learning.
To the best of our knowledge, this is the first work on developing theoretical foundations for ad- versarial reinforcement learning. Our work may inspire novel designs of optimization methods for adversarial reinforcement learning. Moreover, the techniques may be of independent interest in modeling other adversarial learning algorithms, including generative adversarial networks (Good-
fellow al. 2020: Liu and Tuzel 2016 Mao et al. 2017). and solving partial differential equations
Zang et al.
2020
2 RELATED WORKS.
Mean-field optimal control. Since the work of Fornasier and Solombrino (2014) which introduces the concept of the mean-field optimal control and descríbes it as rigorous limiting process, vari- ous applications of mean-field optimal control in different scenarios were proposed. |ornasier et al. (2019) focus on the role of govermment of large population of interacting agents as a mean mean-field optimal control problem derived from deterministic finite agent dynamics, Burger et al. (2021) de- rive a framework to compute optimal controls for problems with states in the space of probability measures, and Albi et al. (0222) studied the problem of mean-field selective optimal control for multi-population dynamics based on transient leadership. In terms of the development of mathemat- ical tools for mean-field optimal control. [onnet and Frankowska (2022) investigate some of the fine properties of the value function associated with an optimal control problem the Wasserstein space of probability measures, and Bonnet and Rossi (202I) provide sufficient conditions under which the controlled vector fields solution of optimal control problems formulated on continuity equations are
Lipschitz regular in space. See (Bonnet al. 2022 Zhou and Xu 2020 Carrillo et al. 2020 for more reference.
Deep learning theory based on dynamics. Previous works have been devoted establish the the- oretical foundations of deep learning by the dynamical system viewpoint since E (2017). Based on the PMP and the method of successive approximation (Kantorovitch) 1939). new optimization methods are developed by Li et al. (2018) [Li and Hao (2018). [Sonoda and Murata (2017) study the continuum limit of training neural networks, and Chang et al. (201 ab; Haber and Ruthotto (2017) contribute to the design of network architecture based on dynamical systems and differential equa- tions. et al. (001)) propose to employ mean-field optimal control formulation for explaining deep learning. They prove the mean-field optimality conditions of both the Hamilton-Jacobi Jww.-a-obbb---laaannn type and the Pontryagin type (Pontryagin] 1987). Similar results are given by Persio and Garbelli (202T) through associating deep learning with stochastic optimal control (Guo et al. (2055) from the perspective of mean-field games (Lasry and Lions] 2007). These mean-field results reflect the probabilistic nature of deep learning. Compared with above works, this paper models an MaARL algorithm as a mean-field quantitative differential game between two dynamical systems, rather than a single dynamical system.
3 PRELIMINARIES
One-agent reinforcement learning, One-agent reinforcement learning aims to solve a K K-step deci- sion problem. Specificallyy, the agent named D. starts from state 20 r= At step k; the agent
2

---

Under review as a conference paper at ICLR 2023
able to take action at based its current state z and moves state xf+1 x6 fa(a" , a²), where fK is the transition function. The error of action a is penalized by Lx(x", a²) from the environment.
K A final penalty ((**, y) tion, e.g., the engine power of vehicle in autonomous driving.As summary, the overall penalties during the agent course as follows:
applies at the last step K, where y represents some known prior informa-
K-1 Σ
(( 0(2 .8)
La(a at) K=U
In deep reinforcement learning, the action at is represented by a deep neural network parameterized by 0h, i.e., a² = a* (x*, 0²), and we can view the agent D2 as mapping D2(x; 02 {0) k==) from the initial state z = x to the last-step state x Also, in practice, the terminal state is usually subject to constraint represented by function g(* such vehicle (the agent) is controlled to reach a certain area (the constraint). In this way, one-agent reinforcement learning can be modelled as a mean-field optimal control problem with trainable parameters {0h
[ K- as follows,
k=
K-1
inf (z.g)=m D(x .v)+ + 8
Σ k=0
Lx( 06)
s.t. 1tt = x² + fa(z" )²),2²² = I IR"1, ,9(x*) =0 O.
(1)
Multi-agent adversarial reinforcement learning. In this setting, besides the original deep- learning-based agent (x 02 (0=]K=)) RR R. (Eq. 1. an adversarial deep-learning- based agent Da(æa; 0a = [0]) : TR + Rd exists, where 0₂ E O₂ and 0u E θ₂ are parameters of D2 and Da respectively, and x; and Fd initial states of D= and Da respectively. We use (a 8h) and (=²), 8 to denote the action of the original and the adversarial agents at step k. respectively. The penalty at step k now relies on the states and actions both of the original agent D2 and of the adversarial agent Da. represented as (x a s: 0%). Similarly, the terminal cost function can be represented as ( 0z), Da( 0z), 3).
In MaARL, 0g is trained to maximize the loss, while 0, is trained minimize the loss. We then formulate the adversarial reinforcement learning problem as
inf sup E(zz.ma.)) (( K v) 0. e. bae
Σ
L xt oh 02)
k=0
8.1.
xx k+1 = =x + fz(1 f2 = 2z I2 (2 = 0,
k+1 x. + fa 02), o...., K 1, x2
(x 0.
d
The goal MaARL to find the optimal parameters 8 and Oa satisfying Eq. 2 such that two
agents reach a Nash equilibrium (Maskin 1999). In this paper, we consider a currently popular offline setting (Agarwal et al. 2020 of MaARL, where the model is learned on N data points (x Ld, U1) ) 1,",, sampled from the distribution μ.
4 MAARL AS A MEAN-FIELD DIFFERENTIAL GAME
Given the current formulation of MaARL (Eq. it is not easy to provide theoretical analysis due to its discrete iterations. To faciliate analysis, we consider the dynamical systems viewpoint and translate problem ( into the following continuous form.
inf sup J(0,, 02) inf sup (... ((((()).)) 1." L.000.00000000))))
(3)
8s
0.
dx(t)
s.t. dt
= f(x()) (.tt),,2.00 0d), z00) = (xT, xT,, x(t/) E S: (x|g(2) = 0).
3

---

Under review as a conference paper at ICLR 2023
(o=(tt)))),, (xattt))) [0, 10.ty] + R" za(+) [0, + R 0 (x)+ : [0,ty] - R", 0a(+): [0,ty] + g
where f(x, 02 0u) (z, 02), (x, 0a) g(x(tt))
RT + R", (a(+): _2 + R Φ, L and f are all functions of appropriate input and output dimensions. Thus, x: [0, [y] + R², g _ - RP, = n1 + n2 and p pi + P2. We define Uz (Ma) the of admissable strategy 0₂ (0a) that satisfies the terminal constraint g=(xx(t))) (ga(Ta(ty)) = 0).
We note that the above problem (Eq. (3) special case of the mean-field differential games, and name it as the mean-field quantitative differential game. We believe that Eq. (3) is a reasonable mod- eling of MaARL, since most dynamic systems in MaARL scenarios are naturally described in terms of continuous time because of physical laws (e.g., the trajectory of the vehicle in autonomous driv- ing). Furthermore, as the mean-field quantitative differential game is special case of the mean-field differential games, methodology from this area can be borrowed and can offer theoretical insight into this problem.
Our goal is to characterize the optimal strategy (0", 02) of Eq. 33 and the corresponding optimal trajectory x² (1) for any (0z, Oa) E U2 x Ua that satisfies
J(0 0a) S J( 02) S 02)
called the saddle point condition. Furthermore, as Eq. (3) is characterization of the expected penalty while empirically only the penalty from the sample is available, another goal of us to characterize the gap between the sampled penalty and the expected penalty.
where Eq.
The rest of the paper is organized as follows: in Section 5 we characterize the optimal solution of Eq. 3) through the mean-field two-sided extremism principle (TSEP). In Section we characterize the optimal objective function value of Eq. (3) through the mean-field HJI function. Finally, in Section we derive the generalization bound between the sampled penalty and the expected penalty.
5 MODELING OPTIMAL SOLUTION USING MEAN-FIELD TSEP
In this section, we characterize the optimality of the mean-field quantitative differential game (3) through a two-sided extremism principle (TSEP) with terminal constraints. We prove that satisfying such a TSEP is a necessary condition for being the optimal solution of Eq. assumptions, we will show the TSEP is also sufficient condition.
B
With additional mild
We first introduce the Hamilton function of Eq. (3) as follows,
Hxx(t),0, (z) (t))), (t)) = ((x( (a(()) + (0)
Intuitively, H : IR" x O₂ x O& x R" + R is the total energy of the dynamical system and va IR"
represents the momentum.
We are now ready to prove derive the necessary condition of being the optimal solution of Eq. (3)
Theorem 5.1 Under the assumptions,
If is bounded and f. L are continuous w.r.t. 0z, Oz:
ti) f. and & are continuously differentiable w.r.t x, and the distribution μ has bounded support.
Let (02,0% 0%) Ug x Uu be the optimal strategy of problem 3.. xt) be the corresponding optimal trajectory, then there exists v0" :00, tr] + TR. and & E RDP such that for E [0,t,].
1) 8" (t)))= = f(z 02, 03).
x0) To. 3- (t) = -V (t), (2(), (t)))))))))))
(ty), vo) 5" VVVag(z" (ttr))
(5)
2) B( (1), (1),, 30 (1) sup inf Be 0z, Oz, (1))
o, BeER,
= in sup ( (x (1), Oz Oa, 1/= (1)), a.e,
(6)
-. 0,E0,
where f()), T(.-) and ((.) defined in Eq. 3 and H()) is the Hamilton function.
4

---

Under review as a conference paper at ICLR 2023
Theorem 5.1 introduces the necessary conditions for the convergence of the unique global solution to the mean-field TSEP, relying on the loss function and terminal constraints.
Since the necessary conditions for optimality have been provided by the TSEP, natural question is to understand when sufficient conditions for optimality can be also provided. This part presents one simple case where it is sufficient, i.e., optimal solution exists, when does the mean-field TSEP admit a unique solution?
Theorem 5.2 Suppose that
i) J is bounded, g is continuously differentiable warl with bounded and Lipschitz partial deriva- tives, p has bounded support IR x "R
ûi J. I and & are twice continuously disferentiable w.r. 0₂ and 02 with bounded and Lipschitz partial derivatives, and Of _00_00,,. OT, 00.0000,000 0000000000000.0000 = 0;
tii) H (zz Oz. 0s, 3) is strongly concave in 22, strongly convex in Oa and uniform in E R", 1/ IR".
Then for sufficiently small iy, if (0=, 02) and (02. 02) are solutions of the mean-field TSEP derived 02)
in Theorem 5.1 and are continuously wir time t, then (02,0)
Theorem 5.2 shows that small ty roughly corresponds to the regime where the reachable set of the forward dynamics is small. Hence, the solution is unique. We then assume the continuity of 81, fd 82, f3 with respect in Theorem 5.2 In fact, when 01, fd. 02. 02 are discontinuous at most a set with zero measure, we can also conclude for a.e. E [0, tyl that (0) (t) (E(1),0(()))
MODELING OPTIMAL OBJECTIVE FUNCTION VALUE VIA MEAN-FIELD HJI EQUATION
In this section, we study mean-field HJI equation from another perspective. This section presents (1) the mean-field HJI equation for MaARL; and (2) the relationship between the HJI equation and the TSEP.
6.1 OPTIMAL OBJECTIVE FUNCTION VALUE OBEYS MEAN-FIELD HJI EQUATION
To simplify the notations, we define
:= 0z, u)),
(7)
where J, 0% and 02 are defined in Section 4 One can easily observe that v (1, (u) corresponds to the optimal objective function value with sample distribution μ and time t. we then have following theorem characterizing v.
Theorem 6.1 Under the assumptions
i) f. L and & are bounded. and the distribution μ E P2(
ii) f, and & are Lipschitz continuous w.r.t and the Lipschitz constant of and L are independent of 0a
Suppose the optimal value function v² (1, p)) of Eq. 7 exists, then it is the unique viscosity solution (see the definition in Appendix B to the following mean-field HJI equation
{
(,v μ)) + inf sup 0, B.EB.
2m+m 10,v(t, u) v) 0. 0.) 0] + L(x, 0z, 0x) 18) O,
o(t I Jautm ((x,))d((.VV)
(8)
where f()), T(.-) and +(.) defined in (3).
The optimal value function e" (t, ()) is the solution to Eq. 8 in Theorem 6.1 revealing the dynamic
programming principle, which shows that for any optimal trajectory, starting from any intermediate
5

---

Under review as a conference paper at ICLR 2023
state in the trajectory, the remaining trajectory is also optimal. Theorem 6.1 also establishes the uniqueness of the HJI equation with regards to viscosity, and identifies the value function for the mean-field optimal control problem the unique solution of the HJI equation.
6.2 CONNECTION BETWERN HJI AND TSEP
In Theorem 5.1 we prove the necessary condition of being the optimal solution of Eq. (3) is char- acterized by the TSEP, while 6. shows the optimal objective value is the unique viscosity solution of the mean-fiell HJI equation. One may wonder what is the connection between the TSEP and the mean-field HJI equation. In this section, we will show that the TSEP can be understood as a local result compared to the global characterization the HJI equation. To see this, we will first introduce some basic knowledge on the Wasserstein space and its derivation rules.
6.2.1 DERIVATIVE WASSERSTEIN SPACE
Let D represent the Fréchet derivative on Banach spaces. Namely, if F.F: : U + V a mapping between two Banach spaces (U, (Iu) and (V, Ilv), then (F + V is linear operator satisfies
F(( + v) F(x) (F(x)
as llyle 0.
1
Denote X E Ru+m random variable, we use the shorthand (2(2, Ratm) for (( Rn+m)) to represent set [R**m--vluee square integrable random variables with respect to probability measure IP. Then we equip this Hilbert space with the norm ||X||>> == (E) x|23)1/2 As assumed in the previous section, 2o R"" y/ E Rn random variables and (zo, y0) μ (2(**tm), where P2(IR*t")) denotes the integrable probability measure defined on the Euclidean space Ru+m, The space (2(R**m can be equipped with metric by 2- 2-Wasserttein distance
Wo(u,)) = inf |X E (82, with Px = 1, = m
For μ E (2( define ||l|| = (fgutm || [0ll'pp(da ((dw)) Now the variable X (22, TR++m) if and only if its law Px E (2(Ttmmm For any function P2 - IR, we can lift it into its "extension" U E LL(, Batm) Cardaliaguet 2012) by U(X) ((PP)). VX E (S2, R" In particular, we have that u is C (P3((T ]). the lifted function U Fréchet differentiable with continuous derivatives. Since ( I=+*) can be identified with its dual, if the Fréchet derivative (XX exists, by Riesz' theorem, can identified with an element of 1,2 B=+m), )
DU))) E(DU(X) Y], VY E (+
One may check that the law of (X does not depend on X but only on the law of X, thus the derivative of at p Px is defined DU O,,((P)((X)) for some function (F)) Ra+mm m
6.3 DERIVE THE CHARACTERIZATION
In what follows, provide the connection between the HJI equation and TSEP. We will show that the TSEP can be understood as a local result compared to the global characterization of the HJI equation. For the value function vtt, μ)) in deduced HJI (equation 8). consider the lifted function (t,, X), where X (xxy) ~ μ. We define the Hamiltonian for the lifted HJI equation as
inf sup H(X, DxV 0, BLE
[tx (f.. 0z). 0,, 0.)]
(10)
Suppose 01(X, Dx (1,, X)) and 0x V X)) are the corresponding optimal strategies and define Dx V (t, X), we have
P))], p "o., PP P))] = =0,
H(X, P) 1PC.::/-BBPPUUCL.1.1..:.0.1....0 E [o, (xx,)______((x)))))__(,)),__
(1I)

---

Under review as a conference paper at ICLR 2023
where the last equation follows from the first order optimality condition. Define X; (Iz, v), Pt = X,), we can apply the characteristic evolution equations (Subbotina) 2006
X = - Pt). P = -
Dx (X,,,
(12)
into equation first n components Pt, have
Plugging equation|
11
12 and let (1 01 (X,, Pt), (t( = (X,, Pt) and is the
a = f(r, (t), 02()), = = (1))), (7())p: VaL(Tr, (1),), (7))).
(13)
If let t/ pp. the first equalities of equation in Theorem 5.1 is converted equation [3 The Hamilton equation in TSEP can be regarded as the characteristic equations for the HJI equa- tion originating from μo. which justifies the claim that the TSEP constitutes local condition as compared to the HJI equation.
7 GENERALIZATION BOUND
In this section, we establish generalization bounds for MaARL. in the offline setting both from the perspective of the global minimum of the loss function and from the perspective of algorithmic stability.
7.1
GENERALIZATION BOUND FROM TSEP
We define the loss function of each training sample
X; = (Zz, Ld. Vi),
i= 1=1,...,A
(0 (.) = (xz (t (t)) x xd(t))) 1"' L(as()),aa(t),,00))0)
where (0) 222 T.) Idd Now
J(0,, 02) = E Xo (0, 0s; Xo),
(14)
and we define
JN.0,. 0a) N
1
Σ (0,
(15)
We then estimate the generalization bounds for offline MaARL based on the TSEP. The necessary condition of Hamiltonian for the sampled version is expressed as
H( NN (t),
O2 (t), Go (070I+(())
inf
sup
H oz (t), 0z. Oi 2.
e e e
N
a.c.
(16)
where [0,ty] ty), 8 and 0g are the solution of sampled TSEP. Note that O and O₂ are suffi- ciently large, e.g.
+ = R 0,7100a= = R2,
the solution 02,00 02 of TSEP satisfies
F(0 (t) := E,, Vo. H (x
0000 (z))) = 0, a.e. E [0,ty].
while the solution AN BN of sampled TSEP satisfies
N
OO ON) = N
0% oN Ve. 0. (x.
0% 0
(t), (N)))
(t)) = 0, a.e. fe E [0,ty].
(17)

---

Under review as a conference paper at ICLR 2023
Now, FN is a random approximation of F and
(F (t) F()..0.)(t)
for all 2z, Ou and a.e. t E ty,
Let (U, Il ). [V. -Iv) be Banach spaces and F U V. We first provide the definition of stability, which is the primary condition that ensures the approximation of FN to F.
Definition 7.1 For p > and x E U. S,(x) == ty E E | - yluv < p). The mapping F is stable on (z)x) if there exists constant K, such that,
ly - - S K (y))F(y) F(z)llv, Vy,z (
Notice that in this case, we are only concerned about whether 02 and 02 follow the first-order op- timality condition. We define = (0T, 0² and redefine F(0", (-)()) and (x 03 (.) as F(0)(). (.). respectively. Then we obtain the following Theorem 7.1 which describes the convergence of the sampled solution the mean-field solution the number of samples increases.
Theorem 7.1 Assuming that L. and & are bounded and Lipschitz continuous with respect to a and the Lipschitz constants of I and L are independent of Oz, Oa- Let (0 07) be a solution of = 0 (Eq- 171),, which is stable on Sp(((02)" (O)) for some > 0. Then there ex- (
ists positive constants Su C, K1, K2. Pr < and a random variable ON = ((ON)" (0%) (o((00))((02))))) such that for 8 E (0, so). the following holds.
Ns²
P P[100 - QN = 2 Cs] S 4exp
K1 + K25
P[[00 - oN || 2 Cs] S 4exp
Ki + K28
N
([J(0),01) JION, 0N0 2 s] < 4xexp
K1 + K25
P[ + of s 4exp {
N
(18)
Ki K280
The loss function 14 is uniformly bounded under the given assumptions, then we can apply the Hoeffding's 's inequality (Corollary 2 in Pinelis and Sakhanenko (1986). Using Theorem En[E et al. (2019) and rewriting 8 as (0T )T) this theorem be proved. Let < 1, set the right-hand side of Eq. 18) to be less than 8. Solving for e immediately yields the following bound.
Corollary 7.1 Under the assumptions and notations of Theorem 7.1 for any
2
0<85
max . (0.
exp
(19)
K1 + K28
the following inequality holds with probability at least 1
Ki +
K
10: - 0.²1 11,00 <CV
log
N
8'
K1 +
K₂
10g 1N <OV
log
(20)
N
8.
Ki
+
K2
4 5 log
JOOO 07) LOONNO0000
N
basically shows that the difference between the optimizer over the whole distribution and the optimizer over finite samples is bounded, and has order O(1/VN) with total of N samples. This bound is independent of the training algorithm.
Corollary 7.1
8

---

Under review as a conference paper at ICLR 2023
7.2 GENERALIZATION BOUND VIA ALGORITHMIC STABILITY
In the end, we estimate the generalization bounds for offline ARL from the view of algorithmic stability. In the rest of this section, we redefine the integral form in J, Jv. J as the discrete sum form ( and redefine V1, 72 as the total dimension of 0z. 0s- We define the generalization error by taking the expectation with respect the randomized algorithm
er(O,, 02) == [A 0a) - (N,0,, 0.)]
We update 02 and OL2 alternately, i.e. from the initial value (0z.0, 04,0), update 02 by M2, steps to get (fs...... (d,o). then update 0d by Ma,1 steps get (fs.Ms.. Od.Ma,.) Keep going until the algorithm converges, we can get aa,: Od.,a..). (Oz.M., Ma.) ... (O..Me..: Md.Ma..).
Consider Stochastic Gradient Langevin Dynamies (SGLD), which is popular variant of stochastic gradient methods adding isotropic Gaussian noise in each iteration, e.g.
0,,6+1 02, 7h Vo, (N,,, 0a,0) + 22 I.).
We have the following generalization bound in expectation of random draw of training data.
Theorem 7.2 Suppose that (0z, 0a: X) is uniformly bounded by C, and
v (0,, (2; X) Vo. Ox: S Les Jo (0,, 0a; X) Vo, 0g: X')l| S Tad VX,X',
then have the following generalization bound
E EeerO2,.M... 0d,Ma,
1/2
Ma Ma.111 Σ
min (K1, M2,1 M2.-1) VBLCC
Σ
N
j=k++1
(21)
1/2
Maa
Ma
2 min (k2, Ma Ma.--1 +
VBLaC
fli
N
j=k++1
where M2,0 Ma O. h and k2 are chosen satisfy 17,, - 2/BLL Th2 1m 2/313.
Theorem 7.2 obtains a bound of O(INN), which matches the generalization bounds of stochastic gradient descent ascent (SGDA) for minimax problems in Lei et al. (022). This bound relies on the aggregated step sizes and does not explicitly depend on the dimensions, norms, or other capacity measures of the parameter, which are usually excessively large in deep learning.
CONCLUSION
Adversarial reinforcement learning (MaARL) has shown superior performance in solving adversar- ial games, but the theoretical understanding of MaARL is still premature. This paper studies the convergence and generalization of MaARL under the mean-field optimal control framework. We first model MaARL as a mean-field quantitative differential game problem. We prove the necessary conditions for the convergence of MaARL from two perspectives, two-sided extremism principle (TSEP) and Hamilton-Jacoii-Be field TSEP and the HJI equation are also established. Further, we present the connection between mean-field TSEP and a mean-field HJI equation. In this way, we show that the TSEP is actually local special case compared to the global characterization of the HJI equation. We also prove two generalization bounds of orders (1I/VN) and (LL/N) from two aspects, global minimum of the loss function and algorithmic stability, respectively, where N is the number of initial states used in training. Both bounds do not explicitly rely on the dimensions, norms, or other capacity measures of the network parameter, which usually prohibitively large in deep learning, The bounds illus- trate how the algorithmic randomness facilitates the generalization of MaARL. To the best of our knowledge, this is the first theoretical work the convergence and generalization of MaARI..
(HJI) equation. The uniqueness of the solutions to mean-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on of- fline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020. http://proceedings..lr. wwr.smml..s.rrss..o 19/agarwa2200..
Giacomo Albi, Stefano Almi, Marco Morandotti, and Francesco Solombrino. Mean-field selective optimal control via transient leadership. Applied Mathematics & Optiiizaaiion, 85(2):1-44, 2022.
Itamar Arel, Cong Liu, Tom Urbanik, and Airton G Kohls. Reinforcement learning-based multi- agent system for network traffic signal control. IET Intelligent Transport Systems, 2010. https://doi.orr/ I0. tts.
Andrew Gehret Barto, Steven Bradtke, and Satinder Singh. Real-time learning and control using asynchronous dynamic programming. University of Massachusetts at Amherst, Department of Computer and Information Science, 1991.
V. Behzadan and A. Munir. Adversarial reinforcement learning framework for benchmarking col- lision avoidance mechanisms in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine, 2019. httpss:/doo..rr/// 10w./II///ISSSS. STT.TTS..TT.2998998
Martin Bichler, Maximilian Fichtl, Stefan Heidekrüger, Nils Kohring, and Paul Sutterer. Learning equilibria in symmetric auction games using artificial neural networks. Nature Machine Intelli- gence, 3(8):687-669) 695, 2021. http:///wwwww.....yyooooooooetta
Benoît Bonnet and Hélène Frankowska. Semiconcavity and sensitivity analysis in mean-field op- timal control and applications. Journal de Mathémaliques Pures et Appliquées, 157:282-345, 2022.
Benoît Bonnet and Francesco Rossi. Intrinsic lipschitz regularity of mean-field optimal controls. SIAM Journal on Control and Optimization, 59(3):201 1-22046,,,,, 2021.
Benoit Bonnet, Cristina Cipriani, Massimo Fornasier, and Hui Huang. A mean-field optimal control approach to the training neurodes. Algorithmic Optimization and Data Science, 20, 2022.
Martin Burger, René Pinnau, Claudia Totzeck, and Oliver Tse. Mean-fiedd optimal control and optimality conditions in the space of probability measures. SIAM Journal Control and Opti- mization, 59(2):977-1066, 2021.
Cardaliaguet. Notes on mean field games. Technical report, from P.-L. Lions' lectures at Collège de France, 2012. www.ceremade.deeee wwwp:crrrmmddddduiiieee.eee http:r///FF22010000.p0.
José A Carrillo, Edgard A Pimentel, and Vardan K Voskanyan. On mean field optimal control problem. Nonlinear Analysis, 199:112039 2020.
B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, and E. Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018a. A.t.aaaaoorAA.AAtatcccoooollllll (6te////ooooooao/11666611/111121 1668/11527
B. Chang, L. Meng, Haber, F. Tung, and D. Begert. Multi-level residual networks from dynamical systems view. In International Conference Learning Representations, 2018b.
https:/openeeviee.ee///f//iii SddSdds-dd--SSS---S
Peter Dayan and Nathaniel Daw. the brain. Cognitive. Affective, & Behavioral Neuroscience, 8(4):429 2008.
Decision theory, reinforcement learning, and
W. E. A proposal on machine learning via dynamical systems. Communications Mathematics and Statistics, 5(1):1-11, 2017, https://bbl.or/11.0007////111810
W. E, J. Han, and Q. Li. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 6(1): 2019. http://ooooooo..ooo///////////11
Massimo Forasier and Francesco Solombrino. Mean-field optimal control. ESAIM: Control, Opti-
misation and Calculus of Variations, 20(4): 123-1152, 2014.
10

---

Under review as a conference paper at ICLR 2023
Massimo Fornasier, Stefano Lisini, Carlo Orrieri, and Giuseppe Savaré. Mean-field optimal control as gamma-lime limi of finite agent controls. European Journal of Applied Mathematics, 30(6):1153- 1186, 2019.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63(11411,,,444 http:://rrro///..111444442200000
2020.
L. Guo, D. Z. Cheng, and X. Feng. Introduction control theory: from basic concepts research frontiers. Science Press, 2005.
E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse problems, 34(1): 014004, 2017.
EA Jasmin, TP Imthias Ahamed, and VP Jagathy Raj. Reinforcement learning approaches to eco- nomic dispatch problem. International Journal of Electrical Power & Energy Systems, 33(4): 836-845, 2011. https://doi.orr//100 wtw.or////....1.11 oije.. 2010. 12.008
L. Kantorovitch. The method of successive approximation for functional equations. Acta Mathe- matica, 71()):63-97, 1939. https///doo.org/000000///////
Lasry and Lions. Mean field games. Japanese journal of mathematics, 2(1):229 260, 2007. hups://doi.orr/100 1070000005537700700--665555
Y. Lei, Z. Yang, T. Yang, and Y. Ying. Stability and generalization of stochastic gradient methods for minimax problems. arXiv preprint 1rXiv:2005.03793, 2021. https//arrwv.or/aa///05550177771
Joel Z Leibo, Cyprien de Masson d' Autume, Daniel Zoran, David Amos, Charles Beattie, Keith Anderson, Antonio García Castañeda, Manuel Sanchez, Simon Green, Audrunas Gruslys, et al. Psychlab: a psychology laboratory for deep reinforcement learning agents. arXiv preprint 1801:0800..0116 2018. https://axivvoog//////810..8811
Q. Li and Hao. An optimal control approach to deep learning and applications weightt neural networks. In International Conference on Machine Learning. pages 2985-2994, PMLR, 2018. http://prooceedings.mmt mlr, press/v80/1i pre.s/////.m....s///880/0 8b.html
Q. Li, L. Chen, and C. Tai. Maximum principle based algorithms for deep learning. Journal of Machine Learming Research, 18:1-29, 2018. http://mml..org/apperrs/// http://///pppppe//////111113..10 15176
M. Liu and O. Tuzel. Coupled generative adversarial networks. Advances in neural information processing systems, 29:469-477, 2016.
X. Ma, K. tonomous vehicle control with adversarial reinforcement learning. In 2018 IEEE Intelligent Vehi- cles Symposium (IV), pages 1665-1671. IEEE, 2018. https::/doi.org// 10. TT..999.VS..01. 8500450
and M. J. Kochenderfer. Improved robustness and safety for au-
A. Mandlekar, Y. Zhu, A. Garg. L. Fei-Fei, and S. Savarese. Adversarially robust policy learn- ing: Active construction of physically-plausibe perturbations. 2017 IEEE/RSJ Interna- tional Conference Intelligent Robots and Systems (IROS), pages 3932 3939. IEEE, 2017.
https://doi.org/10 1...g//11111111 (0tp//RRS.201782206665
X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794- 2802, 2017. https://doi.org/ 10.1 109//CCV.0017.000
Eric Maskin. Nash equilibrium and welfare optimality, The Review of Economic Studies, 66(1): 23-38, 1999. https///co..og/1..1111//////
X. Pan, D. Seita, Y. Gao, and J. Canny. Risk averse robust adversarial reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 8522-8528. IEEE, 2019. https://doi.org// 10.
L. D. Persio and M. Garbelli. Deep learning and mean-field games: stochastic optimal control perspective. Symmetry, 13(1):14, 2021. https://oi..ng/0033305mmmmmmmmm/

---

1. F. Pinelis and A. I. Sakhanenko. Remarks on inequalities for large deviation probabilities. Theory of Probability & Its Applications, 30(1):143-148, 1986. http:://www..iiiooooooo///////// Under review as a conference paper at ICLR 2023
L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learn- ing. In International Conference on Machine Learning, pages 2817-2826. PMLR, 2017.
L. S. Pontryagin. The mathematical theory of optimal processes and differential games. Trudy Matematicheskogo Instituta imeni VA Steklova, 169:1 19-158, 1985. http:/mi.mattneet.r/mm2222
L. S. Pontryagin. Mathematical theory of optimal processes. pttps///sc.panda3222 321 coom///doomaanssooosksssoooooo google..om
CRC press, 1987.
S. Sonoda and N. Murata. Double continuum limit of deep neural networks. In
ICML. Workshop Principled Approuches to Deep Learning, volume 1740, 2017. https///wwwppadl..oe wwwpa/aaeea//p/ppa//2......33332
N. Subbotina. The method of characteristics for hamilton-jacobi equations and applica- tions to dynamical optimization. Jourmal of mathematical sciences, 135(3):295533091 2006.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988. htt.s//aw....oooommmmmm/////////
Richard S Sutton and Andrew G Barto. Reinforcement learning: introduction, ser. adaptive computation and machine learning. Cambridge, Massachusetts: MIT Press, 6:15-17, 1998. http://ww..0001111/0022225177777
W. Uther and M. Veloso. Adversarial reinforcement learning. Technical report, Tech. rep., Camegie Mellon University. Unpublished, 2003. http://www.cin.ufpe.br cavmi/CMU-CSS-3-10070
Y. Zang, G. Bao, X. Ye, and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020. https://doi.org/I00 wttp://rr//...1111......... jepp./1...
K. Zhang, B. Hu, and T. Basar. On the stability and convergence of robust adversarial reinforcement learning: case study on linear quadratic systems. Advances in Neural Information Processing Systems, 33, 2020.
Zejian Zhou and Hao Xu. A novel mean-field-game--t optimal control for very large-scale mul- tiagent systems. IEEE Transactions on Cybernetics, 2020.
A PROOF OF RESULTS IN SECTION 5
Before proving Theorem 5.2 we write the express in Theorem 5.1 more compactly. For each control
process 82 E - ( t O.) and Os _
L ([0,ty], Qa). we denote by z²=.0a := {-] 0- As
0 5 t
ty} and vi = { 00 <tstty) } the solutions of Hamilton's 's Equation equation i.e.
B. f(xt (1), Oz, 0a),
Ps Ba
To
00..0a -V, 0z (l), (1),), v/ 01)
6,,0,
V. (
02.0a 20)
<Vag(s (n
We have the following lemma, which provides an estimate of the difference between r0100) 1500 and 2.00 o2 v/02..0.
Lemma A. Let 02 0 E - ((0,t)),0,) and 02 02 L (10, Qa). Then there exists a con- stant To such that for all l E To), it holds that:
1 02
0.08 I1 + 0/.0 o0. Iea I C(t)) 821l2~ + [10] (%1l
where C(ty) > O satisfies C(ty) - 0 1 + O.
12

---

Under review as a conference paper at ICLR 2023
A.1 Denote 60 := 0. --- -0g 60g 8 fg. óx x02.03 x02.03
Proof A.1 (Proof of Lemma
and bip 1 3022.03 The first two assumptions of T'hoormm 5.2 leads to
|1020 ! 1 0 (2(s),0(()) f(x." 2 , (}(s))]dss
S K SS¹ 11or,llds + K ! |100(())lds + K I. [|60a())]dss,
and so
16 1000 S Kt, Ktyl1öz|11a + *Kty1180=11e + Kit K
I S == =/K, have
Kly (1102111> + 11802l10>>
|100:11000 S 1 Kt,
(22)
Similarly,
|1 & K|1rx1, 11 + K /.¹ [0āraall 11 |100%.11 + [1)0=(0)] |160x())]lds,
11 & (K +
|- - + 1160gll + KKtt(160)1l=> + 1160
hence
| K(1 + t1)
Kts (160=111.> [180all<=),
|8 + 1 Kiş
Ki,
which combined with equation 22 proves the lemma.
We can now prove Theorem 5.2
Proof A.2 (Proof of Theorem 5.2) By uniform strong concavity and the second assumption The- orem 3.2 there exists Xn such that
xt) (tt()))((tt)(+ < [En Vo, HI, 0%
(.((), (2)))
Epp Vo, H( (1) (101, }] (0: - (2()).
< Epo Vo, Hr,
En, Vo, (( 8/00 (t), (1(() 02.01
(0((t) - (2))).
(1),), (t()) of o En Ve, 0, H. 0. (t))))(()))) of Note that Euo, Ve..0, (x,, 0000 o2
Hx
0, Vt E [0, tyl due to the optimality and continuity, then combining the two inequalities above we have
No (o( (2(() +100(t)) (+)))
END Vo, H(x, 0. 01 (2()), (3))) vi o1
ENo Vo, (xt 0f (1)1)
(0:(() (((1))
EPD Vo, Hz, (t), (3()) ((t))
ENo V
(03(1) 02(0))
E po Vo, H 0100 (10,
Vo, H
00((1) 02(0
E P. Vo, (1, 0 (t())
Vo, (1), (1 v [1110)(()) 03(1)
<K([]o<[1>> + || 1.00 (||50 11 + 1100all.=)
Combining the above with Lemma A.7 we have
KC(l))
2KC(4s) (1,00112,-. + 1100al13.. Ao
||00|| 12,- + ||50x|1 < Au
(1150=11<> + |100 |
C(ty) + as t, + 0, by taking t, sufficiently small, so that 2KC(tz) < do which implies
||60=|| 1-=- 1100||10 1. = O.
13

---

Under review as a conference paper at ICLR 2023
B PROOF OF THEOREM 6..
Now we introduce the definition of viscosity solution. Consider a function v(t, Px) [0, Ey x
(2(RRtm + R. the Hamiltonian [X,, Opx v(t, (xt,)x)(X)) : (2, (") x (S2, R Rutm) + R
and V L (22, (22 + R, where satisfies
dv
ÿt H Orx ott, (x)(()) = on (0,tg) (22,
(23)
5 Fx) (X),
on 152, Rutm).
Then the lifted function (I,, X) v((1, satisfies
Ov
a1 + [XX, Dx VX, O,
[0, (t) x L (22, R=+m), on L (12, RR ))
(24)
XX (X),
We say that a bounded, uniformly continuous function u [0, L P2(RR*mm + viscosity solution to equation 23 if its lifted function U [0,ty] (22, 1"**") + R defined by
(1 X) ((,, Px),
is a viscosity solution to the lifted equation equation 24
namely:
i) Ulty, X) v (X) and for any test function E Cl. ty] x (2, ((+)) such that the map U 7 has a local maximum (lo, Xo) E (0,4,) x (2, ("),) one has
(ta Xo) HXo: (4o, Xo)) 2 0.
ii) X) 2 (X) and for any test function C ([0, ty] x (2), R***)) such that the map U 7 has local minimum at (fo, Xo) [0,ty) L L p=+*"),, one has
acy(to: Xo) + (X,, (t Xn)) S..
For further details we refer the interested readers (E et al. 2019).
Proof B.1 (Proof of Theorem 6.1 Suppose (ttt, v)) is a viscosity solution to equation g and (0',, (o) is the corresponding optimal strategy.
We first fix 8',, consider
( (Qu(tt (f,, + L(x, 8' 02)d (0) = 0,
(t, + sup e
vn (L).
(F,,y)d((,,))
Jn+m
(25)
By Theorem I and Theorem in EE et al. 2019). of (1, is the unique viscosity solution to equa- tion 25 satisfies
J (1,1))) sup Eeuaswm [ ((x(t)))))) + ((a((), Asella
(26)
Then fix 0', similarly we have
inf (( [ ((z(t)),)) LL0(0),0,0))
d' d( Now for (0', 02), equation is satisfied, thus v',t, n) (t,
(27)
0.e14.
14


Reviews:
[{"review_id": "tkw40HqJr2", "reviewer": "Reviewer_NCJm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper models Multi-agent adversarial reinforcement learning (MaARL) as a as a mean-field quantitative differential game. A generalization error bound for MaARL is derived in terms of number of samples. ", "summary_of_the_review": "This paper models Multi-agent adversarial reinforcement learning (MaARL) as a as a mean-field quantitative differential game. A generalization error bound for MaARL is derived in terms of number of samples. The key results of the paper need to have error between MaARL and mean field game as the number of agents grow. Further, aspects of uniqueness of equilibrium needs discussion. ", "strength_and_weaknesses": "+: The paper says that \"this is the first work on developing theoretical foundations for adversarial reinforcement learning\". \n\n-: The paper is not easy to read, or fully get the novelty. \n\n-: Even though the results are around multiple agents, none of the results take the number of agents into account. How are the gaps in terms of scaling users? Note that similar results exist for convergence of MARL to MFC.\n\n-: The paper talks about NE, while we need to have assumptions for uniqueness of such equilibrium for convergence or generalization guarantees. Such discussion will help the paper.  \n\n-: Mean field aspects will require information sharing, and that is not realizable in the adversarial setups. This aspect limits the practicality of the setup. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "p_Zehhs6xh", "reviewer": "Reviewer_fRuc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies a two-player game from the point of view of mean field control, recently introduced to study optimal control problems in which the goal is to control a distribution. Here, the distribution is the joint distribution of the state at the current time and some information provided at initial time that can be used to compute the terminal total. The authors formulate the problem and then derive optimality conditions. Last, they provide some bounds between the mean-field solution and the solution obtained using an N-sample approximation, as well as some convergence result for a gradient-based algorithm.", "summary_of_the_review": "Overall, I believe that the comparison with the existing literature (particularly on mean-field type games) should be more detailed, and the main contributions should be more clearly connected to reinforcement learning. ", "strength_and_weaknesses": "Strength: Applying mean field control to multi-agent problems seem interesting and not well studied.\n\nWeaknesses: The literature review and the link with machine learning should be strengthened, particularly for this type of conference. I provide below more detailed comments:\n\n(1) One part of the literature is not mentioned: multi-player games with mean-field players have already been studied. See for example \n- Tembine, H., 2017. Mean-field-type games. AIMS Mathematics, 2(4), pp.706-735\n- Cosso, A. and Pham, H., 2019. Zero-sum stochastic differential games of generalized McKean–Vlasov type. Journal de Mathématiques Pures et Appliquées, 129, pp.180-212.\n- Carmona, R., Hamidouche, K., Laurière, M. and Tan, Z., 2021. Linear-quadratic zero-sum mean-field type games: Optimality conditions and policy optimization. Journal of Dynamics & Games, 8(4).\nSome of these references and the ones cited therein provide optimality conditions. I would recommend clarifying the connection between your results and existing ones.\n\n(2) In the title, and then throughout the text, the authors repeatedly mention that their work contributes to (multi-agent adversarial) reinforcement learning (RL). However, unless I missed something, the work is not directly connected RL. Even the connection with machine learning is not clear to me. There is a gradient-descent type algorithm mentioned in Section 7.2 (but only for one of the two players), for which a bound is provided in Theorem 7.2. But I did not find the proof of this theorem in the appendix. Perhaps it is a direct consequence of an existing result. As a conclusion, I do not clearly see how this paper contributes to the machine learning community. I would appreciate more clarification on this point.\n\n(2) In Theorem 7.2 it seems that some assumptions are not clearly stated. For instance, I imagine that we need to at least ensure that J is differentiable. Please provide a complete set of assumptions.\n\nTypo:\nPage 4: “In Section 5” → “In Section 6”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "bGcLHW0ScC", "reviewer": "Reviewer_p9RX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyses the solution to the multi-agent adversarial RL (MaARL) problem via mean-field optimal control viewpoint. The authors characterizes the solution using the two-sided extremism principle (TSEP) and HJI equation then provides a connection between them. At the end, the authors provide a generalization bound for MaARL.", "summary_of_the_review": "My major concerns for this paper is the similarity with the results in E. et al (2018). I find the results presented in the paper is a direct adaptation from E. et al (2018). I hope to see more clarification about the non-triviality and novelty of their results in this paper. Moreover, part of the theoretical results presented in the paper do not have proof.", "strength_and_weaknesses": "Strength:\n- The paper focuses on advancing theoretical results in multi-agent adversarial reinforcement learning which has has limited theory.\n\nWeaknesses:\n- My major concerns is the similarity between the results presented in the paper and the one in [1]. Although this paper and [1] deals with a different objective function, once the objective function in MaARL is reformulated as a mean-field optimal control problem, I find the techniques to arrive at the results are pretty similar.\n- The structure of the results in the paper is also similar to [1]: theorem 5.1 vs theorem 3 in [1], theorem 5.2 vs theorem 4 in [1], section 6.2 vs section 6.1 in [1].\n- I do not find the proof of the results in section 7 in the appendix while I find the results in theorem 7.1 is just the application of theorem 6 in [1].\n\n[1] E. et al. A mean-field optimal control formulation of deep learning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "OI5_E2eyip", "reviewer": "Reviewer_HHzm", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to analyze multi-agent adversarial reinforcement learning theoretically through the lens of a mean-field quantitative differentiable game. The authors derive an error bound under their assumptions.", "summary_of_the_review": "Given the issues with the paper described above, I do not think the paper should be accepted until these issues are addressed.", "strength_and_weaknesses": "The major weakness of this paper is that it appears to ignore the vast literature on theory for multi-agent environments. The authors state \"Despite the empirical popularity of MaARL, its theoretical understanding remains blank\" and \"To the best of our knowledge, this is the first work on developing theoretical foundations for adversarial reinforcement learning.\" In fact, the whole field of noncooperative game theory is dedicated to a theoretical understanding of multi-agent adversarial environments, and within that field there is a lot of literature on developing a theoretical understanding of reinforcement learning algorithms in multi-agent environments specifically, including convergence bounds.\n\nConsider, for example:\n\"From Poincaré recurrence to convergence in imperfect information games: Finding equilibrium via regularization\" by Perolat et al.\n\"DREAM: Deep regret minimization with advantage baselines and model-free learning\" by Steinberger et al.\n\nMoreover, basing the theoretical analysis on mean-field quantitative differentiable games seems like a strange choice. Why not limit the analysis to a more specific subset of MaARL that fits your assumption?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject