Provably convergent quasistatic dynamics for mean-field two-player zero-sum games
Authors: Chao Ma
Lexing Ying
Venue: ICLR 2022 Poster (2022)
Keywords: quasistatic
minimax optimization
mixed Nash equilibrium
mean-field formulation

Abstract: In this paper, we study the problem of finding mixed Nash equilibrium for mean-field two-player zero-sum games. Solving this problem requires optimizing over two probability distributions. We consider a quasistatic Wasserstein gradient flow dynamics in which one probability distribution follows the Wasserstein gradient flow, while the other one is always at the equilibrium. Theoretical analysis are conducted on this dynamics, showing its convergence to the mixed Nash equilibrium under mild conditions. Inspired by the continuous dynamics of probability distributions, we derive a quasistatic Langevin gradient descent method with inner-outer iterations, and test the method on different problems, including training mixture of GANs. 

Published as a conference paper at ICLR 2022
PROVABLY CONVERGENT QUASISTATIC DYNAMICS FOR MEAN-FIELD TWO-PLAYER ZERO-SUM GAMES
Chao Ma, Lexing Ying Department of Mathematics Stanford University Stanford, CA 94305, USA {chaoma, Iireommn,exx_ngeesttatooo.....
ABSTRACT
In this paper, study the problem of finding mixed Nash equilibrium for mean- field two-player zero-sum games. Solving this problem requires optimizing over two probability distributions. We consider a quasistatic Wasserstein gradient flow dynamics which one probability distribution follows the Wasserstein gradient flow, while the other one is always the equilibrium. Theoretical analysis are conducted on this dynamics, showing its convergence to the mixed Nash equilib- rium under mild conditions. Inspired by the continuous dynamics of probabil- ity distributions, we derive a quasistatic Langevin gradient descent method with inner-outer iterations, and test the method on different problems, including train- ing mixture of GANs.
INTRODUCTION
Finding Nash equilibrium has seen many important applications in machine learning, such as gener- ative adversarial networks (GANs) (Goodfellow et al. 2014a) and reinforcement learning (Busoniu et al. 2008). In these problems, pure Nash equilibria the problems arising from machine learning are usually nonconvex in x and nonconcave in y. in which case pure Nash equilibrium may not exist. And even it exists, there is no guarantee for any optimization algorithm to find it efficiently. This difficulty is reflected in practice, that compared with simple minimization, machine learning applications involving Nash equilibria usually have more complicated behaviors and more subtle dependence on hyper-parameters. For example, stable and efficient training of GANs requires number of carefully designed tricks (Gao et al.) 2018).
usually search for function f(x y)- Yet,
On the other hand, the mixed Nash equilibrium (MNE) is known to exist in much more general set- tings, e.g. when the strategy spaces are compact and the payoff function continuous (Gllicksherg) 1952). In the mixed Nash equilibrium problem, instead of taking "pure strategies" and y, two "mixed strategies" for æ and y, in the form of probability distributions, are considered, resulting in the following functional,
J [(x u)p(2) Jaly)dady,
where and are density functions of probability distributions of æ and y, respectively. Efforts are invested develop theoretically endorsed algorithms that can efficiently find MNE for high dimensional problems, with applications on the training of mixture of GANs. In Hsieh et al. 22019) mirror-descent algorithm is proposed and its convergence is proven. In Domingo-Ennin Enrich et al. (2020), theoretical analysis and empirical experiments are conducted for gradient descent-ascent flow under a Wasserstein-Fister
metric and its particle discretization.
In this paper, we also consider the mixed Nash equilibrium problem, and propose simple Qua- siStatic Wasserstein Gradient Flow (QSWGF) for solving the problem. In our dynamics, we treat q component with much faster speed than p, hence is always equilibrium as p moves. With entropy regularization for both and (without requirement on the strength the regularizations), prove that the QSWGF converges to the unique mixed Nash equilibrium from any initialization (under mild conditions). Furthermore. we show there is a simple way to discretize the QSWGF re- gardless of the complexity of the Wasserstein gradient flow of p induced by the fact that y is always

---

Published as a conference paper at ICLR 2022
at equilibrium. Concretely, partition function related with p appears in the QSWGF dynamies, and we find an efficient way to approximate the partition function. By discretizing the QSWGF, derive particle dynamics with inner-outer structure, named the QuasiStatic Langevin Gra- dient Descent algorithm (QSLGD). In QSLGD, after each iteration of the outer problem (for the x particles), the inner loop conducts sufficient iterations to bring the y particles to equilibrium. Nu- merical experiments show the effectiveness of QSLGD on synthetic examples and training mixture of GANs. Our method outperforms the vanilla Langevin gradient descent-ascent method when the entropy regularization is weak.
As summary, our two major contributions are:
1. We propose the quasistatic Wasserstein gradient flow dynamics for mixed Nash equilibrium problems, and show its convergence to the unique Nash equilibrium under weak assump- tions. Our result neither requires the entropy regularization be sufficiently strong, nor assumes the dynamics to converge a priori.
2. We derive a simple while practical quasistatic Langevin gradient descent algorithm by dis- cretizing the quasistatic Wasserstein gradient flow, by finding efficient way to approx- imate the partition function appearing in the dynamics of p. The proposed algorithm is applied on several problems including training mixtures of GANs.
RELATED WORK
The mixed Nash equilibrium problem has long history, with the proof of its existence dates back to Morgenstern & Von Neumann (1953). draws new attention in recent years, especially in the machine learning community, due the development of GANs (Goodfellow et al. 2014a) and ad- versarial training (Goodfellow et al. 2014b). Training mixture GANs is already discussed in paper (Goodfellow al.] 2014a). Some numerical experiments were conducted in (Arora et al.) 2017]. In'Grnaoova et al. 2017). the authors proposed an online learning approach for training mix- ture of GANs, and proved its effectiveness for semi-shallow GANs (GANs whose discriminator is shallow neural network). Yet, rigorous theoretical treatment to an algorithm started from (Hsich) al.] 2019), which mirror descent method was studied and proven converge. The imple- mentation of the mirror descent method involves big computational cost that asks for heuristics to alleviate. Later, (Domingo-Hnrfe
et al.) 2020) studied more efficient algorithms under mixture of Wasserstein and Fisher-Rao metrics. Theoretically, the time average of the dynamics' trajectories is shown to converge to the mixed Nash equilibrium. As a comparison, in this work we show the global convergence of the quasistatic Wasserstein gradient flow without the need of taking time average. Meanwhile, the Wasserstein nature of our dynamics makes it easy to implement as well.
The Wasserstein gradient flow in the density space has been explored in previous works. For exam- plc, (Wang& & Li 2219) studied the Nesterov's ' accelerated gradient flows for probability distributions under the Wasserstein metric, and (Arbel et al. 2019)) studied practical implementations of the natu- ral gradient method for the Wasserstein metric. Both works focus on minimization problems instead of min-max problems considered in this work. more related work is Lin et al. (202Ib), where natural gradient based algorithm is proposed for training GANs. Yet, the method still optimizes one generator and one discriminator, searching for pure Nash equilibrium. Another work that derives algorithms for GANs from Wasserstein perspective is (Lin et al. 2021a).
Another volume of works that studies the Wasserstein gradient flow in the machine learning con- text is the mean-field analysis of neural networks. This line of works started from two-layer neural networks (Mei al. 2018: Rotskoff & Vanden-Fijnden 2018 Chizat Bach 2018 (Siiinnano & Spiliopoulos] 2020), to deep fully cully-connected networks (Araújo et al. 2019, Sirignaano & Spiliopou- los| 2021 Nguyen 2019; Wojtowytsch et al. 2020), and residual networks (Lu et al. 2020 al. 2020). The mean-field formulations treat parameters as probability distributions, and the training dynamics are usually the gradient flow under Wasserstein metric. Attempts to prove convergence of the dynamics to global minima are made (Mei et al. 2018 Chizat & Bach 2018 Rotskoff et al. 2019). though in the case without entropy regularization a convergence assumption should usually be made priori.
2

---

Published as a conference paper at ICLR 2022
3 THE QUASISTATIC DYNAMICS
We consider the entropy regularized mixed Nash equilibrium problem, which in our case is equiva- lent with solving the following minimax problem:
Jaxa K (x, ())(())))) + 8-1 / p log pdæ 8-1 glog qdy.
/
min max PEP(O) P(P(
(1)
In ① Ω is compact Riemannian manifold without boundary, and P( is the set of probability distributions on S2. Since S2 is compact, any probability distribution in P(S2) naturally has finite mo- ments. Let Epp. g) Saxa K (x, 1))((x)))))))) (dd)) and S(p) - Jn p log pdæ and S(g) Sa glog qdy be the (negative) entropy of p and q, respectively. Then, the minimax problem ① can be written in
short as
min max E(p, 9) (()) PEP(O) PEP(S))
Remark 1. Strictly speaking, in ① we should distinguish probability distributions and their den- sity function (if exist), and the entropy should also be defined using the Radon-Nikk Nikoddm derivative with canonical measure. In this paper, since p and indeed have density functions because of the entropy regularization, we shall abuse the notation by using p and q to represent both probability distributions and their density functions.
The entropy regularizations in ① and make the problem strongly convex in p and strongly concave in q. Hence, there exists unique Nash equilibrium for the problem. Such results shown for example by the following theorem from (Domingo-Entee
et al. 2020).
Theorem 1. (Theorem of (Domingo-Enrice et al. (0200) Assume is compact Polish metric space equipped with canonical Borel measure, and that K is continuous function Q x Ω. Then, problem (2) has a unique Nash equilibrium given by the solution of the following fixed-point problem:
- exp(-BUU g(z)= 1
p(x) Zp
exp(3V(g.P)).
Dn
where Zp and Zg are normalization constants to make sure and q are probability distributions, and U and V are defined as
U(x,9) 5E(p.9) 6p
(x) [(( V(v.p) 5E(r,9) (1) / K K(x, (l,,))(()dd.. ōg
Considering the efficiency in high-dimensional cases, natural dynamics of interest find the Nash equilibrium for (2) the gradient descent-ascent flow under the Wasserstein metric,
d.p: V (PU 3 log pt))
0.g. V V(-V + 8 1og 4:)).
because can be easily discretized into a Langevin gradient descent-ascent method by treating the PDEs Fokker-Planck equations of SDEs. When 3 is sufficiently large, (4) be proven converge linearly to the unique MNE of (2) (Eberle et al. 2019). However, when 3 is small, whether converges remains open. This hinders the application of (4) because in practice the entropy terms are usually used as regularization and are kept small. (We realize that it is proven Domingo-Enrich Bruna (2022) when our work is under review.)
In (4. the dynamics of and have the same speed. In this work, instead, we study a quasistatic limiting dynamics when the speed of g becomes faster and faster compared with that of p. In this case, at any time we assume q: reaches at the equilibrium of the maximizing problem instantaneously by fixing = Pt ? That is to say, at any time f, l is determined by
Wasserstein gradient descent dynamics, which can be understood
at g(p.] == arg max ((p..)) -(p..))----(())) (EP(2)
(5)
On the other hand, Pt follows the Wasserstein gradient descent flow with qe g/pb] the equilib- rium:
a, vv. vn. 8(E(p), g(p:)) ---'S(((pi))) + B "'loogg ))
(6)
öp:
3

---

Published as a conference paper at ICLR 2022
The following theorem shows Gt q[pe] can be explicitly written Gibbs distribution depending on Pt. and thus the free energy in (6) can be simplified to depend on partition function related with Pt-
Theorem 2. Assume K is contimuous on the compact set Q and B > 0. Then, for fixed pt the
maximization problem 5 has unique solution
(pe(y) = (p(Pt)
(xp(3V((y.P))
where 7g(()) is a normalization factor, 7p(p) = (xxp(3V(y.)))er Moreover, the dynamics (6) for Pt can be written as
P.V (
83 log (pr)) 8
+ log ))
O.p: V
ôp:
Let 8 log (p) + 8 S(p).. By Theorem 2 the dynamics (8 of pt is the Wasserstein gradient descent flow for minimizing Fp.s(p). (p). By the Proposition 3 below, Fp.3 is strongly convex with respect p. Therefore, is possible prove global convergence for the dynamics (8), and thus the convergence for the quasistatic Wasserstein gradient flow for the minimax problem 2
Proposition For any probability distributioos pv, p2 P(2), and any
[0, 1}, we have (xpe(Ap1 + (1 A)p2) < (p + A)Fp,e(P2).
practice the partition function log Zg (px) in high dimensional spaces. However, show in the following proposition that the variation of the partition function with respect to can be written as a simple form involving q.. This property will be used to derive particle method in Section
seems hard to approximate, especially when It
Proposition 4. For any P(t), we have
83 log (p) (.,9(p)),
(9)
6p
where qpp) is defined in 7 Therefore, the dynamics
( is equivalent with
O.p: (p.V (U.z 8-1 logg:))
(10)
CONVERGENCE ANALYSIS
In this section, analyze the convergence of the quasistatic dynamics following assumptions on K.
First, make the
Assumption 1. Assume K E (200((2 x 2), which means has continuous derivatives of any order (with respect to both and y).
Since is compact, assumption Timplies boundedness and Lipschitz continuity of any derivatives of K.
Now, we state our main theorem, which shows the convergence of QSWGF to the Nash equilibrium.
Theorem 5. (main theorem) Assume Assumption hoolds for K. Then, starting from any initial Po. 40 E P()), the dynamics (7). (8) has unique solution (pt, q:)e2o. and the solution converges weakly to the unique Nash equilibrium of (2), (p" ,q"), which satisfies the fixed point problem 3
Theorem 5 guarantees convergence of the quasistatic Wasserstein gradient flow for any B, giving theoretical endorsement to the discretized algorithm that we will introduce in the next section. Note that the initialization Yo in the theorem is not important, because we assume q achieves equilibrium immediately after the initialization.
Remark 2. The assumption on K's smoothness can be made weaker. For example, during the proof. up 4-th order derivatives of K is enough give sufficient regularity to the solution of the dynamics. We make the strong assumption partly to prevent tedious technical analysis so as to focus the idea and insights.

---

Published as a conference paper at ICLR 2022
Proof sketch We provide some main steps and ideas of the proof of the main theorem in this section. The detailed proof is put in the appendix.
By the last section, since d: is always at equilibrium, only need to considering Wasserstein gradient descent flow for F Therefore, we can build our analysis based on the theories in (Mei al. 2018) and (Jordan et al. 1998). However, compared with the analysis therein, our theory deals with a new energy term 3 log (p),, which has not been studied by previous works. From now on, let (p) = 3- log (p),, and ((.p)) (F) By simple calculation we have op
v(r,p)= = U(r,)))((())))))) (p) / ( y) exp (L, (K(a, dy.
(11)
First, we study the free energy (p)) and show that it has a unique minimizer which satisfies a fixed point condition. This the result of the convexity of Fp.3. We have the following lemma. Lemma 1. Assume Assumption Thold for K. Then, F2.S has unique minimizer p" that satisfies Fp,s(v").
Fn,s(p")=
inf PEP(S)
Moreover, p" is the unique solution of the following fixed point problem,
p= exp (-3V0(x,
(12)
where Z is the normalization factor.
Next, we want to show that any trajectory given by dynamics 10)) will converge to the unique minimizer of Fp.s- To achieve this, we first study the existence, uniqueness, and regularity of the solution to (8). i.e. the trajectory indeed exists and is well behaved. Related results are given by the following lemma.
Lemma 2. Assume Assumption I holds for K. Then, starting from any initial po E P(2). the weak solution (pr).o to g exists and is unique. Moreover, (p:) is smooth on (0, 00) x Ω.
The proof of Lemma based on Proposition 5.1 of (Jordan et al. 1998). Especially, the existence part is proven using the JKO scheme proposed in (Jordan et al.) 1998). We consider a sequence of probability distributions given by the following discrete iteration schemes with time step h.
1 (p. +hF.)))))))p))))))) hPPhoto) {
Po = po- ph arg min
k> 0,
PEP(S))
where W2(p, g) means the 2-Waseerstein distance between probability distributions p and q. Let (of )20 the piecewise constant interpolations of (pi) A20 on time. We show (p)) converges weakly (after taking subsequence) to weak solution of as h tends to O. Details are given in the appendix.
Finally, noting that Fp.s is Lyapunov function of the dynamics 8 we have the following lemma showing the convergence of (Pt)zz to the solution of the Boltzmann fixed point problem 12. This finishes the proof of the main theorem.
Lemma 3. Let (pr)ezo be the solution of 8 from any initial Po E P())). Let p" be the unique 0o.
minimizer of Fp.9 given [22.. Then, Pt converges to p" weakly
As byproduct, since our convergence results does not impose requirement on 3, if one is interested in the minimax problem without entropy regularization,
min pE (EP(
max E(p,
(13)
then, Theorem 5 in (Domingo-Enrire Enrich et al.
2020) ensures that the quasistatic dynamics converges
to approximate Nash equilibrium of (13) long 8 is small enough. Specifically, pair of probability distributions (p, q) is called c- Nash equilibrium (3)
sup E(p, q') inf ('EP(0)
E(p'.q) P'P((O)
Then, we have the following theorem as a direct results of Theorem 5 in (Domingo-Enee Ennichh et al. 2020
5

---

Published as a conference paper at ICLR 2022
Theorem 6. Let CK be the bound of K that satisfies (x, y) CK for any I. Q. and let Kip(K) be the Lipschitz constant of K. For any >> 0, let 8 = e/(2ip(KK), and let Vg be the volume of ball with radius 6 in Ω2. Then, long
4 log
2(1 Vi)
(1C-1))
(ACK
B 3>
V8
there exists T > which depends on e, such that for any > T, the solution Pt, qt of the dynamics and 1 t satisfies
sup E(p,, d'E((N)
Elp', qr) S. TEP(()
inf
5 THE QUASISTATIC LANGEVIN GRADIENT DESCENT-ASCENT METHOD
is well known that PDEs with the form
@ep(t, = V (p((,x))((tt x:) + AAp(t,a)
Pokker-Planc' equations for SDEs dx,, -u(t, Xt)dt V2XdW, and the solution for the PDE characterizes the law of X, solution of the SDE at any time. This result connects the Wasserstein gradient flow with SDE, and gives natural particle discretization approximate the continuous Wasserstein gradient flow. For example, the Wasserstein gradient descent-ascent flow dynamics is the Fokker-Planck equation of the SDES
dX.- = - (XU(X, dt 1/23-1
dY, VyV(Y., Pt /28-IdW;
where and are the laws of X, and Y,, respectively, and W, and W/ are two Brownian motions. Note that we have
V.U(z.9) S V=K(z.y)(v)) vvV(u.P) / (y((x.))p()ddd.
Therefore, i..d. picking x po and y for = 1.2,... the particle update scheme, named Langevin Gradient Descent-Ascent (LGDA),
x k+1
(i) h
K((( ( + 211/2/11- 2
(0) k+1
y + (( y()) 22.3-
(14)
approximately solves the SDEs, and thus the empirical distributions of x() and y approximate the solutions of when n is large. Here, <00 and (6) are i.i.d. samples from the standard Gaussian.
Quasistatic Langevin gradient descent method Similarly, the dynamics (8) for p the Fokker- Planck equation for the SDE
dx. VV Pe dt /23-IdW;
(15)
where is the law of X,. By proposition we have 4 (z, Pt) = Uzx g(pe)). Hence, 15 can be written
dx, = - (Xx, + 2,3 dW.
(16)
with qfre) at the equilibrium of the maximization problem 5) which can be attained by solving the SDE
dY, V VY,,, Pt)dt 2
(17)
for sufficiently long time. This motivates to design quasistatic particle method as discretization for the quasistatic Wasserstein gradient flow. Specifically, the method consists of an inner loop and an outer loop. The method starts from some particles x( and 1 from Pn and qu. respectively. Then, at the k-th step, the inner loop conducts enough iterations on the
2, ..., sampled i.i.d.
Y particles to solve (17) with pe fixed (i.e. with the X particles fixed), which drives the empirical distribution of (9) in near equilibrium before each update the outer loop. Next, the outer loop updates x(()) according the SDE 16). The algorithm is summarized in Algorithm

---

Published as a conference paper at ICLR 2022
Algorithm 1: Quasistatic Langevin gradient descent method (QSLGD)
input Rez Bay ko, k1, h2. T E N4, h2, hy 0, Po, 90 E P(2)) output: Final partictes (X Y )
Sample (Xx*)) y i.i.d. from Po. and (Z)))) ) i..d. from go:
Yo.0 +- 1/ for = 1,,,, R2,...7..) Ryj
for +- to ky do
Y( - 1
x-(() by Σ VyK(X
1 xx,0-1) 22
0, +
/* remark 3.1 */
j=l
end
())) -- y(i for = 1,, 2..,
0,kg
for t <- 1 to T do
y( + --,(1 for 1.2,
for 8 <- 1 do
y(i) t (-1)
+ hy Σ
v VKK(
1 -1 + 2
10
1-1,8-1
5=1
end
12
for f- 1 to k2 do
y(1) 111,
(i)
The by
VyK(X, --1tthh- / 22
13
1.1
nz
1((m-1)m
((0
14
t-1
11tktt for i 1, 2..,, Ny:
15
end
keny
x(9) x((1 +
Kenly Σ N V p( + 22e
16
/* remark 3.2
((i) + (li)
17
+k2 for i = 1,2,.., nyy
end
Remark 3. Generally speaking. Algorithm 7 consists of two nested loops. The inner loop solves Y particles equilibrium in each step the outer loop, while the outer loop makes one iteration every time, using the equilibrium Y particles. In the the following are some additional explanation for the Algorithm:
line 4: at the beginning of the algorithm, we conduct ky additional inner iterations for Y, where ko may be large number. This is because the beginning the Y particles far from equilibrium. In later outer iterations, since each time the X particles only move for small distance, the Y particles close to the equilibrium. Therefore, kn and lo need not to be large.
line 17:
In each inner loop, we conduct kn + K2 inner iterations for the Y particles, and collect those from the last ka iterations. We use these knn particles the update of X particles to approximate the distribution q(p). We assume during the last 22 inner iterations the Y particles are equilibrium. One can take k2 to be 1 if ny is large enough, while taking large k allows smaller number of Y particles.
5.1 EXAMPLES
In this section, we apply the quasistatic Langevin gradient descent method to several problems.
I-dimensional game on torus torus. Specifically, we consider
We first consider problem with r and on the 1-dimensional
K(x, y) = (2r..) sin()
where x, E R/Z. is easy to show that, with this K and a positive 3, at the Nash equilibrium of the problem (T) and g both uniform distributions. We take initial distributions Po and go

---

Published as a conference paper at ICLR 2022
LGDA
IGDM
LGDA 05
06
05 155kF33
Q6 f
QS -3
QS ka
0S
0S A) 10
KL dive divergence gence
number of particles
Figure 1: Experiment results with K (xx, y) sin sin (2ry). The three figures show the KL divergence of the empirical particle distribution to the uniform distribution of LGDA and QSLGD different 8. 7) and number of particles. Each point an average of 5 experiments.
to be the uniform distribution on [0, 1/4]. Figure I shows the comparison of the quasistatic particle method with LGDA for different 3, step length, and number of particles. In the experiments, all quasistatic methods take kn 1000 and k2 1, with different k; shown in the legends. For each experiment, we conduct 300000, 150000, 60000, 30000 outer iterations for LGDA, QS2, QS5, and QS10, respectively. We take different different numbers of iterations for different methods in the consideration of different number of inner iterations. The error is then computed after the last iteration, measured by the KL divergence of the empirical distribution given by particles and the uniform distribution (both in the forms of histograms with 10 equi-length bins). Each point in the figures is an average of 5 experiments.
Seen from the left figure, the QSLGD has comparable performance than LGDA when | is small, which case diffusion dominates the dynamics, while it performs much better than LGDA when 3 is large. We can also see better tolerance to large / when more inner iterations are conducted. This shows the advantage of the QSLGD over LGDA when the regularization stength is weak. The middle figures shows slightly better performance of the QSLGD when the step length (both Te and 7y) is small. However, when 7) is big, LGDA tends to give smaller error. The results may be caused by the instability of the inner loop when 7) is big. It also guides to pick small step length when applying the proposed method. Finally, the right figure compares the influence of the number of particles when 100 and yj = 0.01, in which case the two methods perform similarly. We can see that the errors for both methods scale in a 1/n rate as the number of particles n changes.
Polynomial games on spheres In the second example, consider polynomial games on sphere similar to that studied in (Doningo-Enrich et al. 2020).
K(x, v) x² Agæ: + x" A 8" A2v + Ag(=))
(18)
where 2; y E Sd-1 and (x²) the element-wise square of æ. In this problem, consider the Nash equilibrium of minp max, E(p. q). Hence, we take big 3 (small 3-1) and compare the Nikaido and Isoda (NT) error of the solutions found by different methods (Nikaidō & Isoda 1955). is defined by
The NI error
N1(p.)) sup E(p,, d'EP(())
E(p'.g)) .q). P'EP(N)
inf
which is also used in Theorem The left panel of Figure the NI errors of the solutions found by different methods with different dimensions, see comparable performance of the QSLGD with LGDA.
GANs Finally, test our methods the training of GANs. We train GANs to learn Gaussian mixtures. The results after training are shown in the middle and right panels of Figure 2 where Gaussian mixtures with 4 and 8 modes learned, respectively. We train GANs with 5 generators and 5 discriminators, and take kn = 100, k = 5, h2 1. The results show that the mixture of GANs trained by QSLGD can learn Gaussian mixtures successfully.
In the right panel of Figure 2 we show the results of learning high dimensional Gaussian mixtures. In the d-dimensional experiment, the Gaussian mixture has d modes centered at e,, e2 - ed with
8

---

Published as a conference paper at ICLR 2022
0:50
d-100
Figure 2: (Left) The NI error of the solutions found by different algorithms for the polynomial game 18). at different dimensions. Each point is an average of 10 experiments. (Middle left, Middle right) Generation results of mixture of GANs. The blue points are sampled from groundtruth dis- tribution, while the green points are generated by the mixture of generators. (Right) The average squared distance generated data closest mode center for learning high dimensional Gaussian mixtures.
standard deviation 0.1. Here, is the i-th unit vector in the standard basis of Rd. Model and algorithm with same eyper-parameters as above are used. In the figure, we measure the average squared distance of the generated data to the closest mode center along the training process. The figure shows that the average squared distance can be reduced to 0.3 0.5 after 10000 iterations. While the ideal value is 0.1, the current results still show that the learnt distribution concentrates the mode centers. Better results may be obtained after longer training or careful hyper-parameter tuning.
6 DISCUSSION
In this paper, we study the quasistatic Wasserstein gradient flow for the mixed Nash equilibrium problem. We theoretically show the convergence the continuous dynamics to the unique Nash equilibrium. Then, quasistatic particle method is proposed by discretizing the continuous dynam- ies. The particle method consists of two nested loops, and conduct sufficient inner loop in each step of the outer loop. Numerical experiments show the effectiveness of the method. Comparison with LGDA shows the proposed method has advantage over LGDA when ,3 is large (which is usually the case of interest), and performs good LGDA in most other cases.
Theoretical extensions are possible. For example, strong convergence results may be established by similar approaches taken (Feng & Li 2020p. We leave this as future work.
In practice, the idea of nested loops is not new for minimax optimization problems. It is already discussed and utilized in the earliest works for GANs (Goodfellow al.] 2014a). and Wasserstein GANs (Arjovsky et al., (017). In those works, the discriminator is updated for several steps each time the generator is updated. Our work is different from these works because we consider mixed Nash equilibrium and hence our method is particle based, while their method searches for pure Nash cquilibrium.
Finally, though particle methods finding mixed Nash equilibria have stronger theoretical guarantees, applying these methods to the training GANs faces the problem computational cost. With both the generator and discriminator being large neural networks, training mixture of GANs with many generators and discriminators imposes formidable computational cost. Developing more efficient particle methods for GANs is important future work.
REFERENCES
Dyego Araújo, Roberto I Oliveira, and Daniel Yukimura. A mean-field limit for certain deep neural networks. arXiv preprint arXiv:1906.00/93 2019.
Michael Arbel, Arthur Gretton, Wuchen Li, and Guido Montúfar. Kernelized wasserstein natural gradient. arXiv preprint 11X1v:1910.09652, 2019.

---

Published as a conference paper at ICLR 2022
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. PMLR, 2017.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference Machine Learning, pp. 224 232. PMLR, 2017.
Lucian Busoniu, Robert Babuska, and Bart De Schutter. comprehensive survey of multiagent rein- forcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156-172, 2008
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. arXiv preprint arXiv:/805.000 2018.
Carles Domingo-Enrich and Joan Bruma. Simultaneous transport evolution for minimax equilibria on measures. arXiv preprint,
2022.
Carles Domingo-Enriche Samy Jelassi, Arthur Mensch, Grant Rotskoff, and Joan Bruna. A mean- field analysis of two-player zero-sum games. arXiv preprint arXiv:2002.06227, 2020.
Weinan E, Chao Ma, and Lei Wu. Machine learning from continuous viewpoint, i. Science China Mathematics, 63(11):2220 02-233222666 2020.
Andreas Eberle, Arnaud Guillin, and Raphael Zimmer. Quantitative harris-type theorems for dif- fusions and mckean- vlasov processes. Transactions of the American Mathematical Society, 371
(10):7135-7173, 2019.
Qi Feng and Wuchen Li. Entropy dissipation via information gamma calculus: Non-reversible stochastic differential equations. arXiv preprint arXiv:2011.00058 2020.
Fei Gao, Yue Yang, Jun Wang, Jinping Sun, Erfu Yang, and Huiyu Zhou. A deep convolutional generative adversarial networks (degans)-based semi-supervised method for object recognition in synthetic aperture radar (sar) images. Remote Sensing, 10(6):846, 2018.
Irving L Glicksberg. A further generalization of the kakutani fixed point theorem, with application to nash equilibrium points. Proceedings of the American Mathematical Society, 311:1100-774 1952.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 201 4a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
Paulina Grnarova, Kfir Y Levy, Aurelien Lucchi, Thomas Hofmann, and Andreas Krause. An online learning approach to generative adversarial networks. arXiv preprint (0X.:/7006.0266) 2017.
Ya-Ping Hsich, Chen Liu, and Volkan Cevher. Finding mixed nash equilibria of generative ad- versarial networks. In International Conference on Machine Learning, pp. 2810-2819 PMLR, 2019.
Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokker- planck equation. SIAM journal on mathematical analysis, 19(1):1-17, 1998.
Alex Tong Lin, Samy Wu Fung, Wuchen Li, Levon Nurbekyan, and Stanley Osher. Alternating the population and control neural networks to solve high-dimensional stochastic mean-field games. Proceedings of the National Academy of Sciences, 11((1)), 2021a.
Alex Tong Lin, Wuchen Li, Stanley Osher, and Guido Montúfar, Wasserstein proximal of gans. arXiv preprint arXiv: 20t.:220020082222 b021b.
Yiping Lu, Chao Ma. Yulong Lu, Jianfeng Lu, and Lexing Ying. A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterization from depth. In In- fernational Conference on Machine Learning, pp. 6426-6436. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2022
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 116667, 2018.
Marc Mezard and Andrea Montanari. Information, physics, and computation. Oxford University Press, 2009.
Oskar Morgenstern and John Von Neumann. Theory of games and economic behavior. Princeton university press, 1953.
Phan-Minh Nguyen. Mean field limit the learning dynamics of multilayer neural networks. arXiv preprint arXiv: 0rXiv:/92.002880 2019.
Hukukane Nikaidô and Kazuo Isoda. Note non-cooperative convex games. Pacific Journal of Mathematics, 5(S1):807-815, 1955.
Grant Rotskoff, Samy Jelassi, Joan Bruna, and Eric Vanden Vanden-Eiinden Eijnden. Global convergence of neuron birth-death dynamies. arXiv preprint arXiv: 1902.01834, 2019.
Grant Rotskoff and Eric Vanden-Eijnden. Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation emror. star, 1050:22, 2018.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 1800)1222-- 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of deep neural networks. Math- ematics of Operations Research, 2021.
Yifci Wang and Wuchen Li. Accelerated information gradient flow. arXiv:1909.02102. 2019.
arXiv preprint
Stephan Wojtowytsch et al. On the banach spaces associated with multi-layer relu networks: Function representation, approximation theory and gradient descent dynamics. arXiv preprint arXiv:2007.15623, 2020.

---

Published as a conference paper at ICLR 2022
A PROOFS FOR SECTION 3
A.1 PROOF OF THEOREM 2
Note that the free energy in be written
J. (( (0((())))
(19)
in which the first term is linear with respect to 9. Hence, calculation with Lagrange multiplier shows (19) has unique minimizer q(pe] with the form of Gibbs distribution (e.g. see Chapter 4 of (Mezard & Montanari )009):
alpr]()) = (p(p))
(5V(5V (x..P)))))))
(20)
Next, we consider the free energy for Pt when q is at the equilibrium. By
19 we have
E(pt, g(p)) / V V ((w.pi)gl
(g(re) I
p.)
(21)
On the other hand, we have
B- 3-'S(g(Di)) = 3-1 I exp(3V(y,P.)) log Z,(p,)
1
1 exp(3V p.)) dy
3-1
(g(pt)
=3 S Zg(Di)
log Z (p()) dy
(p.) / (VV. exp(3V(y) - 8-1 log (p(p:)
(22)
Combining 21 and 22 we obtain
Epe q pc]) + 3 S(p,) - 3- (((p=)) = 3- log 7p(pt) 3 5
Therefore, the dynamics of pt is the Wasserstein gradient descent flow minimizing the free energy 8-1 log (p:) + 3 (px)). given by
88 log (y(Pt)
+ 9 log log Pt ))
O.P: = V
P
ōp:
This finishes the proof.
A.2 PROOF OF PROPOSITION
Since S(p) strongly convex, suffices to show log (p) is convex. Recall that
log Zy(p) log ( eVp(SV(y. (y. p))
Note that V(.p) = f (xx ((()dd is linear with respect to p. we have
V(, + (1- A)p2) AV + (1 A)V(p.2).
Hence,
log 7gpAp1 + (1- 2)p2)) = log (/ exp(3V(V, Apr + - 2)p2))dy
log (/ exp(3AV(U.PP)) -xpp:pi)))ppp((((11111 -
(000(3(1))V((uPP)) 2)V((.
(1/ exp(3VV(V.P)) ( exp(3V(v.PP)) (v. 1-1
< log
Alg (p1) + (1 - X) log (P2)
(23)
The second last line is given by the Hölder inequality.
12

---

Published as a conference paper at ICLR 2022
A.3 PROOF OF PROPOSITION 4
The proposition follows from the following derivations.
83-1 log (z(p)
=3- 1
(P)
öp
(r)(D)
8p
=3- (p)
exp(3V
J. Kxx,V) v)
exp( dy
Zg(p)
( K(x, ((cx))ylll Jn
(24)
B PROOF OF THEOREM 5
In this section, prove main theorem. The proof will follow the sketch given in Section 4
Given the assumptions on and the conclusions of Lemma] and Lemmal) 2 Lemma B is direct result of Lemma 10.12 in (Mei al., (018), which will ignore the proof. In the following, Some techniques in the proof come from (Jordan et al. 1998)
we show Lemma and Lemma 2
and (Mei et 2018).
B. PROOF OF LEMMA
Our proof follows the proof Proposition 4.1 in (Jordan et al., 1998). First, we show the existence of the minimizer for pp To see this, note that K is bounded Q. Assume Cx is constant such that [<x,,)) Ck for any x, y E Then, we have
Ep))(p) = / Uwx (lpl))(z)dr==
J K(x, ((r.))alpl()))))ddddd 2 -CK-
and
S(p) 1 p(z) log p(x)dr 2
/
1
-
dr
50
e
This means (p) is lower bounded, i.c. inf, Fp)e(p) -00. Hence, we can find a sequence (pu) 2 such that
lim Fp,0(pk) = inf Fp.s(p).
Similar to (oordan et al. 1998). we can show boundedness of {J max log Pk. 0}dr) and (fd)), which implies that (pk) is uniformly integrable, and thus there exists a weakly convergent subsequence of (pk).
Without loss of generality, assume Pk - p" L (22). Then we need show p* is a minimizer of Fp.3. By Jordan et al., 1998), the entropy term satisfies
S(p") < inm S(pw). k0000
Hence, the conclusion follows if Ep.a is continuous in the weak topology. To show this, first note that for any p P(22), we have
/
(xp()V(y, 2 e BBx
Because the function log(#) 1/c-Lipschie for (c, oc), for any Ph we have
3- 1 log 7,(x") & B- 1 6Cx /
(cVVlus)) - - esr(no') dy
| log (pa) 1-3---1-11
13

---

Published as a conference paper at ICLR 2022
By similar boundedness argument, we have
1/ (cOV(uPP)) eVV(vx')) dy
/ cev(vpe) coVcuiD") dy
53
I S
BlV(y.pk) - ((y,p"))
I SK [ K( ((x()))x(()) -p" p=)))) dy
Totally we have
8 log (p(p)) 8- log (p'p') < 290k S. K(x, ((x, (x))ddz dy.
Since K' is bounded and Lipschitz, it is easy to show that
/ 1/1 K(I, p dy = =0.
lim k-0o
Therefore, have
Fe,)(")) < lim inf Pp.s(ps) = inf Fp.a(p) k-+00
and thus Fp")(p") = inf, (p))(p).
Next, we show p' satisfies the fixed point condition
p" exp (- (x,
(25)
This follows the proof Lemma 10.3 in Mei et al. 2018), by first showing p" has full support on Ω, and then showing
((, + 8 log (xx)
is a constant.
Finally, we show p* is unique following Lemma 10.4 of (Mei al. 2018). Specifically, we show the Boltzmann fixed point problem (25) only has one solution by the convexity of Fp.8. Assume 25 has two different solutions P1 and P2. i.c.
P1 7(m)) exp (-30 P2 = 7(p2) exp (x,
Then, we have
log Z(pi) - (3x(x, log (x),
log 4(p2) - (xx P2) log P2 Taking difference of the above two equations, and integrating with p1 P2, have
0 = / (log 7(p1) - log ((p)))(p -P2(x))dr
- 1 - (p)()))
((z,,22)(p p2(x)) / (logpi())
logp2()))(( (1ogp(())-1gpp()))))))
- p2(x))dx.
(26)
By the monotonicity of log the second term of (26) is non non-negative, and takes zero only if p1 P2- For the first term, recall that we have proven Proposition 3 that Ep.s convex, we then have
/ Va (<x, (2(x)))dr,
Ep.)(p)) 2 Ep.s(p2)
and (p2)(22) 2 Epp((P)) ((Z,p1)(s(z)) - -0---Pii (x))ddr
Taking difference of the two equations gives
[ (V((E) (zz (r) 2 0.
Therefore, (26) holds if and only if p1 P2. This finishes the proof of uniqueness of p", and also completes the proof of Lemmal]
14

---

Published as a conference paper at ICLR 2022
B.2 PROOF OF LEMMA
2
We show the existence of the weak solution using the JKO scheme used in (Jordan et al. 1998). Let .ist(-.)) be the distance metric on f2. Then, the 2- Wasserstein distance is on P(S2) is defined
1/2
inf W2(p1,P2) P2) (EET(p. Jax. (2)
where I'(p1. P2) contains all couplings pi and p2, i.c. probability distributions on S2 x S with first and second marginals being Pl and p2, respectively, Then, for any h consider the sequence of probability distributions (p obtained by the following iteration scheme:
{
P6 Po: ph arg min
W3 hFF.s(o)). h 0,
(27)
(2
PEP(S))
By similar argument of Proposition 4.1 in (Jordan et al. 1998 each minimization problem in 27 has a unique solution. Hence, (p uniquely defined. Let p be the piecewise constant inter- polation of (pt) on I, i.e. p = fort E [kh, (k 1)h),
for k 0, We now show that there exists subsequence of hn and pt such that Pt ha - Pr (O, Ω for any T > 0 and pæ is a weak solution of (8). This is proven in two steps:
1. The existence of weakly convergence subsequence, and
2. b approximately satisfies the equation (8).
For the first point, we show uniform integrability by showing
[ Ph sc
(28)
and
[ max log pê, Oldx SC
(29)
for any and 0, and an absolute constant C. Equation (28) follows directly from the compact- ness of Ω. For (29 note that for any h and k: 2 we have
W2 (ph, x .) + h.pp,(ph)) & hF hrppe(pkh1)) 2
which implies
Fr.((1k) S Fn.((k--1)
Therefore,
[ max log logr O)do S(ptt mmin(rk log ph: 03 dx
s((t)) / dr
< Pr.((tt) - (ps(2 + / -oe
S Fp.o((hi)) Ep.o (pt) + 1 dr.
Since K is bounded, Ep.s is bounded, and thus the above expression is also bounded, which gives With 28 and (29). there exists pz(z) and sequence (h,) with h, -> 0, such that pit- pe P(L)) for almost every T'. By changing Pt
29
((, Q) for any T > O. Moreover, a zero measure set t, we can assume pt E P(2)) for any è E (0, oo). With the same analysis
as Jordan et al. 1998), the weak convergence can happen for any i.. p. Pt in L (()) for any
1 (0, oe).
15

---

Published as a conference paper at ICLR 2022
For the second point, similar to Jordan et al. 1998), consider any vector field & C (2, and the corresponding flux 2 given by
0, = {(2+), Bo(x) x,
and let dr = #. then have
1 (( W2 (2 hFF..(()))))
( W pk) h
(30)
for any
> We need to study the limit when 0+. By the calculation (Jordan et
1998)
we have
lim sup +-+0+
( W W3
W3
(1k- $ / (y (y- ()==))())d((z,,)))
(31)
Jaxa
and
d
=- - p Edx.
S(g+)
(32)
dr
r=0
where the y in (31 is the optimal transport between ph-1 and p For the Ep.s term, we have
1
La exp(BV or 4-))
(g:))(g)) Eppe T-++
lim
lim +--
log
BT
La exp (v. pb)
1
Jo (y. 4-)
lim
T-+0
BT
L.aa exp dy
1
(/ (exp( exp(BV(x. (o0g:000000r))))-ooooooo0)0))/1)
lim
--0++
EVV (1, (/VaK(o,)) - (x) dy
32g
V. (., E(x)p) (r)de.
(33)
Combining the above result with (31 and taking both { and -. we get from (30 that
h
Jax (y =)((())d(( v) V. V.. - ((x)p
p
Edr 0
(34)
B g
for any & E C (2, Ω). Then, following the derivation in (Jodaan et al. 5.1), as well as the following control
1998 (proof of Proposition
(r S Ch
for any N that satisfies Nh < T for some fixed T, we can integrate 34 over by viewing pt as rf at appropriate t, and take the limit hn - 0 and show that is weak solution. During the limit, we need pay special attention the second term, i.c the following limit, which is not dealt with in the reference:
J. /.V.(P))))))) E(x)p!" (x) = ('' [ (())p)))))(())))))d))))
lim 10000
(35)
We prove this by showing V. (xr converges to Va (x. uniformly. For any fixed f, recall that we have
pt pr.
Therefore, for any Ω we have
/ K(x, vipi" (x)dxx - / K(x,, (x)
16

---

Published as a conference paper at ICLR 2022
Note that Sa (x, (2)pp((dd is uniformly continuous with respect y for any p P(2), we conclude that So K (z, y)p!" hi (x)dx converges to Jn (xx ())((a) uniformly over y. Hence, we have (x - 7 (pt). ), and glvl" ((v) - (v) uniformly. This further implies that
V. Va P( = / Va (o,Kx,,)oor "l
converges to a (x. Pr) for all æ E S2 This finishes the proof of 33 and also completes the proof that Pt weak solution of of equation (8)
Now we have proven the existence of the weak solution. The regularity and uniqueness of the solution follows the same analysis of Proposition 5.1 in (Jordan al. 1998).
17


Reviews:
[{"review_id": "inuDzlPVmbz", "reviewer": "Reviewer_6QW7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall well written with clear mathematics. \n\nCons: The authors present the gradient flow formulation for zero-sum games. Here they complete the sup step and only consider the inf steps. \n\nWeakness: \n\n1. For the functional  F(p)=max_{q} E(p, q)-beta S(q). Did the author provide some geodesic convexity property of F in Wasserstein space?\n\n2. Some motivations on Energy functional (1) are needed. The method seems to work for a general type of energy or loss function. \n\n3. The authors miss important literature on Wasserstein dynamics in GANs, and mean-field games. I suggest the authors carefully compare them with the results in this paper. \n\nArbel, Gretton, Li, Montufar, Kernelized Wasserstein natural gradient, ICLR, 2019\n\nLin, Li, Montufar, Wasserstein proximal of GANs, GSI 2021. \n\nLin, et.al. Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games,  arXiv:2002.10113.\n\nWang, Li, Accelerated information gradient flow, arXiv:1909.02102. ", "summary_of_the_paper": "In this paper, the authors study the minimax problem for finding the mixed Nash equilibrium for mean-field two-player zero-sum games. This is an inf-sup problem in the Wasserstein space. In this paper, the authors consider the case where the sup problem is solved and apply the gradient descent flow for the result inf sup functional. Some theoretical analyses are conducted on these dynamics. They show its convergence to the mixed Nash equilibrium under some conditions. Some numerical schemes for the quasistatic Langevin gradient descent method with inner-outer iterations are presented. They show that the method works on training the mixture of GANs. ", "summary_of_the_review": "The authors study a gradient flow of a particular type functional, which comes from an inf-sup problem in zero-sum games. The paper is overall well written. Some key issues on geodesic convexity and relation to literature are missed. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fOJxziWQ-Qk", "reviewer": "Reviewer_izUn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The idea of using the different rate for the two parameter density space is implemented to solve the original min-max problem. Although the idea of having slow and fast rate is not new in the literature, the authors manage to covert the problem into a simplified problem. The reduced problem can be solved nicely. \n\n2. The Wasserstein gradient flow is indeed a powerful tool. It is mentioned that since $F_{p,\\beta}$ is strongly convex and the global convergence is expected.  Indeed, a perturbed Wasserstein Gradient flow on the density manifold is studied in [1]\"Entropy dissipation via Information Gamma calculus: Non-reversible stochastic differential equations, Feng Q, Li W\". It will be interesting to see that if the non-gradient flow on the density space could find its corresponding QuasiStatic Wasserstein flow story. In particular, [1] provides a faster convergence rate with perturbation added on the drift term. Is there a potential acceleration of the convergence in solving the min-max problem here? \n\n3. The main theorem (Theorem 5) provides a weak convergence to the unique Nash equilibrium. Is it possible to achieve a stronger convergence in this QuasiStatic Wasserstein flow case? Note that the convergence of the density to the equilibrium in [1] is proved in $L^1$ norm. Instead of using the Wasserstein distance, the K-L is used instead. Can the author improve the convergence from this perspective? \n\n4. The empirical study is only carried for 1-d torus and d-Sphere. Of course, higher dimension will be challenging. Is there anything potentially applicable or limitation of the proposed algorithms to be used in high dimension problems. \n\n5. The analysis is conducted in compact manifolds/domains. For general global in space problems, is there a way to select the domain and then apply the algorithms in this paper to solve those general problems? ", "summary_of_the_paper": "The paper under review studies the QuasiStatic Wasserstein Gradient Flow and its application to solve two-player zero-sum games. The corresponding Langevin SDE is derived for the Fokker-Planck equation, and the discretized algorithm (GD) is proposed in this new setting.  Example of the min-max problem on 1-d torus and spheres are provided. ", "summary_of_the_review": "The proposed algorithms show its powerfulness in solving the min-max problem. Empirical study is supportive. The theoretical analysis is nicely presented with the QuasiStatic Wasserstein flow. The idea and the theoretical part both show the novelty. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YC-29CqWFjo", "reviewer": "Reviewer_4gr6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The intuition and novelty of this paper is good, the authors proposes on analyzing a special setting of finding a mixed Nash equilibrium, by first observing the convergent behavior of the limiting dynamics, and then design a new algorithm via a reasonable discretization. However, the presentation and organization of this paper largely hinders the understanding of the primary contributions.\n\nIn the Introductory part of this paper, when the authors argues that the main contribution of their paper includes a convergence result of quasi static Wasserstein gradient flow dynamics under mild assumptions, and a practical discretization of the dynamics, it is insufficient to illustrate the significance of this paper. The readers lack information on whether similar results have already been obtained by previous works. Is this paper the first approach of this kind? How to compare the convergence result of this paper with [Hsieh et al. 2019], [Domingo-Enrich et al.], etc.? What are the main technical difficulties of designing such an algorithm? The related works in this paper also seems incomplete, the connection of the mean-field studies with the theoretical studies in GANs is missing. All these drawbacks have made this work hard to evaluate.\n\nI searched for the proof of Theorems, Propositions and Lemmas in the Appendix, it seems that the proof of Theorem 5, which is the main theorem, is missing? Most of the results in the main text are direct applications of previous results, it might be better to explain the intuition of the derivation, rather than listing every steps as Lemmas, Theorems, Propositions. ", "summary_of_the_paper": "This paper focuses on the methodology of finding mixed Nash equilibrium. The authors first introduces the corresponding quasi static Wasserstein gradient flow dynamics for solving the problem, which is a limiting dynamics with $q$ seen as infinitely faster than $p$. Then the continuous dynamics is proved to be convergent under mild assumptions. Furthermore, by discretizing the above gradient flow, the authors proposes a practical algorithm for solving min-max optimization problems with GAN as a special example. Experimental results shows the superiority over the vanilla LGDA algorithm especially with large $\\beta$’s.\n", "summary_of_the_review": "Overall, this paper provides a novel algorithm towards finding the mixed Nash equilibrium. However, the presentation of this paper does not support their result well. As a result, the paper is not ready for acceptance at this time.\n\n\n----------\nPost Rebuttal Comments:\n\nThe authors have addressed my concerns. Therefore I'm increasing my score from 5 to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MpBtA_cHOu1", "reviewer": "Reviewer_TF7Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and have clear objectives. It might be a good idea to add more experiments illustrating the power of the authors' method.", "summary_of_the_paper": "This paper studies a minimax problem arising from finding the mixed Nash equilibrium for mean-field two player games. A quasi-static Wasserstein gradient flow is proposed to solve such optimization problems. By discretizing this flow, a simple while practical Langevin gradient descent algorithm is proposed.", "summary_of_the_review": "Good paper. Clear writing. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors first consider a mean field two player zero sum game and consider quasistatic Wasserstein gradient flow dynamics for solving the problem. The dynamics is proved to be convergent under some assumptions. Finally, the authors provide a discretization of the gradient flow and using this proposes an algorithm for solving min-max optimization problems. They use this algorithm for GAN's as the main example. Experimental results claim that the algorithm outperforms langevin gradient descent especially in high dimensionas.

This paper sits right at the border. But subsequent to the author response, one of the reviewers has updated the score and seems more positive about the paper. In view of this, I am leaning towards  an accept.
Decision: Accept (Poster)