Statistical Mechanics of Min-Max Problems
Authors: Yuma Ichikawa
Koji Hukushima
Venue: Accepted by TMLR ()

Abstract: Min-max optimization problems, also known as saddle point problems, have attracted significant attention due to their applications in various fields, such as fair beamforming, generative adversarial networks (GANs), and adversarial learning. However, understanding the properties of these min-max problems has remained a substantial challenge. This study introduces a statistical mechanical formalism for analyzing the equilibrium values of min-max problems in the high-dimensional limit, while appropriately addressing the order of operations for min and max. As a first step, we apply this formalism to bilinear min-max games and simple GANs, deriving the relationship between the amount of training data and generalization error and indicating the optimal ratio of fake to real data for effective learning. This formalism provides a groundwork for a deeper theoretical analysis of the equilibrium properties in various machine learning methods based on min-max problems and encourages the development of new algorithms and architectures.

Published in Transactions on Machine Learning Research (1/2025)
Statistical Mechanics of Min-Max Problems
Yuma Ichikawa
ichskuwa-yumal dc..cc.. www....oooo..c...oooooooo ichikawa. yuma Manaaafujjslscco
Deportment of Basic Science, University of Tokyo
Fujitsu Limited
Koji Hukushima
b-hubushhima@g.ccc.c ecc. woo.://iiw.aeeoooooooooo.oo.oo. ac.jp
Department of Basic Science, University of Tokyo
Reviewed OpenReview: https: https://openrevier openreview. net/ forum? id= qZqUFeTtuu
Abstract
Min-max optimization problems, also known as saddle point problems, have attracted significant attention due to their applications in various fields, such fair beamforming, generative adversarial networks (GANs), and adversarial learning. However, understanding the properties of these min-max problems has remained a substantial challenge. This study introduces a statistical mechanical formalism for analyzing the equilibrium values of min- problems in the high-dimensional limit, while appropriately addressing the order of operations for min and max. As first step, apply this formalism to bilincar min-max games and simple GANs, deriving the relationship between the amount of training data and generalization error and indicating the optimal ratio of fake to real data for effective learning. This formalism provides a groundwork for a deeper thcoretical analysis of the equilibrium properties various machine learning moth based on min-max problems and encourages the development of new algorithms and architectures.
1 Introduction
Min-max optimization problems, also known saddle point problems, are well-known classical optimization problems extensively studied in the context of zero-sum games (Wald 1945 Von Neumann & Morgenstern, 1947). These problems have diverse applications across various fields, such as game theory, machine learning, and signal processing. In game theory, min-max problems arise in zero-sum games where one player's gain corresponds to another's loss. Several methods have been proposed find the min-max value or equilibrium points in these games (Dem'yanov yanov Pevnyi 1972 Maistroskii 1977 Bruck 1977 Lions 1978 Nemhauser
& Wolscy 1988 Freund & Schapire 1999.. In machine learning. min-max games relevant for training
generative adversarial networks (GANs) [Goodfellow et al.) 2020 Arjovsky et al. 2017 Additionally, in
adversarial learning, these problems are employed to train models that are robust to adversarial attacks by
optimizing a worst-case perturbed loss function (Szegedy et al. 2013 Goodfellow et al. 2014b Papernot
ct al. 2016 Madry et al. 2017),
Despite the widespread application of min-max optimization problems, several challenges still need to be addressed, including understanding the usefulness of these min-max formulations, evaluating the convergence properties of the algorithms, and conducting sensitivity analyses of min-max values. A promising approach to addressing these issues is to analyze the typical-case behavior of min-max problems by examining the min-max value averaged over random instances drawn from distributions that capture realistic scttings, referred to as randomized instance ensembles. Stadfssiccal-meccaniiall approaches, which have demonsttraed their effectiveness in analyzing the typical-case behavior of randomized instance ensembles of optimization and contrrann-satiisaactio problems (Mózard & Parisi 1986 Fontanari 1995), provide powerful formalism for such analyses. Extending this formalism to analyze the typical-case behavior of min-max values thus presents potential direction for further research, although this has not yet been fully explored.

---

Published in Transactions on Machine Learning Research (1/2025)
This study applies the statistical mechanical formalism to min-max problems, modeling them virtual two-temperature system. This formalism enables a sensitivity analysis of the typical-case min-max values in the high-dimensional limit. Notably, this formalism properly addresses the order of min-max operations, critical in non-conver sconarios where interchanging the order of min and max can lead to incorrect results Razaviyayn et al., 2020). Using this formalism, we analyze lypical-case min-max values of bilinear min-max games and simple GANs. In particular, we derive the relationship between the amount of training data and generalization error and indicate the optimal ratio of fake data to real data for effective learning.
Our main contributions are as follows:
We introduce a celiisticalmmehaniiee formalism developed for sensitivity analysis of equilibrium values in dige-dimenlional min-max problems.
Applying this approach, we conduct a detailed sensitivity analysis on a bilinear min-man max game to verify the theoretical validity of our approach.
Building on this formalism, we analyze the generalization performance of GANs and determine the optimal ratio between fake and real data for practical training.
2
Related Work
The replica method, which is employed in this study, is non-rigorous but powerful heuristic approach in statistical physics (Edwards & Anderson 1975 Mézard et al. 1987 Mezard & Montanari 2009). It has
been proven to be a valuable method for high-dimensional investigated the relationship between dataset size and generalization error in supervised learning, including
senchhnn--eerning problemss. Previous studies have
single-layer (Gardner Derrida 1988 Opper Haussler 1991 Barbier et al. 2019 Aubin al.
2020 and
multi-layer Aubin et al. 2018 neural networks, as well as kernel methods(Diete (Dietrich et al.)
1999: Bordelon
et al. 2020 Gerace al. 2020 In unsupervised learning, the replica method has also been applied to
dimensionality reduction techniques such as the principal component analysis (Biehl & Mietzner 1993 Hoyle
& Rattray 2004
2007), and to generative models such as energy-based models (Decelle et al.] 2018 Ichikawa
& Hukushima, 2022) and denoising autoencoders (Cui & Zdeborová 2023). However, the dataset-size
dependence of GANs has not been previously analyzed, which this study aims to address.
Related to our work, statistical mechanical formalism for addressing min-max problems has been proposod (Varga 1998). However, the treatment of the inverse temperature limit differs from our approach, and it has limitations in accurately handling the order of the min and max operations. In the context of adversarial learning, which involves a non-convex and concave min-max problem, Tanner et al. (2024) analyzes a tractable setting where the internal maximization can be solved. By reducing such cases to standard optimization problems, they apply the replica method and approximate message passing to explore the core phenomenology obscrved in the adversarial robustness. Even cases where the internal maximization cannot be explicitly solved, the formalism discussed here provides a basis for further analysis and potential extensions to more complex scenarios.
Notation Here, we summarize the notations used in this study. We use the shorthand expression [N]- = [1, .2............... N), where N E N. la Radd denotes d x d identity matrix, and 1a denotes the vector (1,...,1)² 1)T E nd and O₂ denotes the vector (0, (O....,0) T) E Rdd For matrix A (Ayy E dd and vector a = (a:) Rdd, we use the shorthand expressions d.A - II I
I dAi and da _ II=_ da,, respectively. The notation Oz (1) describes the asymptotic order of a function with respect to the parameters de and dy- Specifically, function dyy is said to be Od,d, (1) if it remains bounded as d, and dy grow large (or tend toward some specified limit), independently of d, and dy The standard Gaussian measure is defined as Dz e d1=111=11 /2 (2m)m//2 The notation extr, (a:) represents the evaluation of a function f(x) at its extremum with respect to the variable æ. Specifically, this shorthand implics locating and evaluating fx) points where its gradient Va = 0.
2

---

Published in Transactions on Machine Learning Research (1/2025)
3 Statistical Physics Formalism for Min-Max Optimization Problems
This section introduces statistical-mocchniial formalism that models min-max problems as a virtual two- lemperature system from a statistical mechanics perspective. Min-max problems are formally expressed as
(A) min max Væ y: A), s.t. ZEXCRT, " yEYER", #EX yEy
where ((+)) RRd x R + IR is bivariate function; E R& and y E Rdy are the optimization variables; X and Y are the feasible sets; A is a parameter characterizing the problem, e.g., graph G. We introduce the following Boltzmann distribution to analyze min-max probloms for a given bivariate function V(x, y; A) in Eq. [T, with virtual inverse temperatures Amin E IR and Smax E I:
(8min; Praxx A) e -umin Biax In Sy dye Pmace (
1
Panaa (2;; A)
where Z(3min Smax, A) is the normalization constant, also known the partition function. Hereafter, we refer to it as the partition function. In this context, two-temperature system is particularly important because it allows us to distinguish between the opposing optimization objectives inherent in min-max problems, similar to the different thermal behaviors in statistical mechanics.
By taking the limit 8max + +oo followed by Amin - +oc, the distribution lim lim Pamnn-maas Dmax (x; A) Auin + Smax +
concentrates on uniform distribution over the min-max values, assuming that well-defined min-max values exist for -),,-) and the min- max value is bounded over feasible sets X' and y. Note that the order of these limits is crucial because the min and max operations canmot. be interchanged in non-convex and non-concave min-max problems (Razaviyayn et al., 2020), i.e., mingex maxyey V(x, y A) A maxyey min V(x, y; A). While a similar formulation has been used in the previous work (Varga 1998 0, they simultancously take the
limits of both Amin and 9max with a fixed ratio Phin/Pmae -00m..mmmm (1), which does not fully capture the distinct effects of min and max operations in non-convex settings. Such approach generally does not yield accurate results when the function V(x, y; A) is non-convex with respect to and y.
ceaiistcal-mechaaiiee approaches have demonstrated their effectiveness in analyzing the typical-case behavior specifically, the properties of the optimal value averaged over the instances that follow a distribution p(A)- for optimization and consstriin--aatiffacion problems (Mézard & Parisi 1986 Fontanari, 1995). These analyses have succeeded providing insights inlo different aspects of combinatorial optimization, unlike worst-case analysis. This work also focuses on evaluating the typical cases of min-max problems characterized by a random parameter A. Our main objective is to calculate the logarithm of ((Quiny Pinxx A) averaged over the random variables A in the limit Bmax co followed by Bmin - 0o:
Ω lim lim f(Smiu Bmax),
Amin +000 Bunux mi
where
f(mii Buaxx)
EA (log Z(Buin Punxx A)],
Puind
which is referred to as the free energy density. This free energy generating function with variables æ and y. Appendix shows how to calculate the function of the optimal value æ through this free energy.
Setting the ratio of the inverse temperatures as -miia//msee this can be rewritten as
Blind EA log f dace Baiia -Buine Coniaaainn (mmar In dyee voonii))) (
f(mming Brnxx)
Bmind EA log / dæ (Sdgeeraavann Caneaaaaaasiiie )7
(2)
Although calculating the expectation value of the logarithm is generally difficult, we begin by using the identity
Eog f(A) lim 1 log (A((()))
(3)
70-0
3

---

Published in Transactions on Machine Learning Research (1/2025)
to expand the logarithmic form follows:
Ω lim
lim
lim
log (y(,inn Prnax);
(4)
Bain Smax +00
4-++0 T
where
[Uxx dac
P m7 ( (yayeeTua V (( )")
Z, Bainn Praxx)
- EAA
(5)
At this point, note that these transformations are purcly algebraic identities without assuming the parameters or to be integers.
Following the idea of the replica method [Ewards & Anderson 1975
1983 Zdeborová & Parisi 1979
Krzakala 2016; Gabrié 2020), we then proceed under the assumption that y and p are natural numbers. Specifically, y, rather than addressing Eq (4) directly real values 7 and p, one caleulates the average of the 7-th and p-th powers for 7.p N, then performs an analytic continuation to Y,P= R for this expression, and finally takes the limits 7 +0, 9min + too and fmax +oo. Based on this replica "trick", the calculation simplifies to the replicated partition function Zy(Buin Pmax) as an approximation of Zn (pmin Pmax):
Z, (Pmins Bumxx) (Pmin: Frnxx) 2 EA II dx II 1=1 y
e Junex En., V(x" ((t"","""" v²"
(6)
up to the first order of y to lake the y + +0 limit on the right- hand side of Eq. 4 This computation is a standard procedure in the statistical physics of interaction systems including random variables, and is generally accepted exact, although rigorous proof has not yet been provided. Specifically, the mathematical rigor of the method remains limited due to the unproven uniqueness of the analytic continuation, issue noted for the moment problem (Tanaka 2007). As noted in Section 2 the replica method has provided various results in high dimensional statistics and machine learning as well.
Additionally, before taking the limits, Amin - oo and 9max + 0o, the concept of finite inverse temperatures Amin and Bmax corresponds to scenarios where neither the minimum nor the maximum fully achieved, a common situation in the min-max algorithms. This approach provides valuable insights into cases where neither extreme is fully realized or both only partially optimized. Exploring novel algorithms based on this fiiite-tmpeeratuee generalization of min-max problems represents an intriguing direction for future work. Furthermore, in game theory, this formalism can be interpreted as a framework for modeling games under relaxed assumptions of complete rationality, where players æ and are assumed to behave with bounded rationality rather than adhering strictly to classical models of fully rational behavior (Von Neumann & Morgenstern 1947).
In the following sections, we apply this formalism fundamental and significant bilinear min-max game, demonstrating that the analytic continuation of p in the replica method is rigorous operation. We then analyze the minimal model of GANs more practical example.
4 Bilinear Min-max Games
This min-max formalism introduces two replica parameters: associated with the randomness of A, and p, related to the dual structure of min-max problems. The analytic contimution with respect to the replica parameter y is widely recognized as effective and frequently employed the statistical mechanics of optimization. However, the analytic continuation of the replica parameter p has not yet boen explored. While establishing its matherattical validity presents challenges, this study eliminates the influence of the replica parameter 7 associated with the randomness of A and rigorously demonstrates that the analytic continuation with respect to holds for fundamental bilincar min-max games. Specifically, we show that the free energy density derived using the replica trick in Eq. (6) as explained in Section 3 is equivalent to the exact expression in Eq. 5 derived without analytic continuation of y and p for bilinear min-max games Daskalakis et al. 2017).
Tseng 1995
Bilinear games are regarded as a fundamental example for studying new min-max optimization algorithms
and techniques (Daskalakis al. 2017 Gidel et al. 2019 2018 Liang & Stokes 2019). Mathematically,
4

---

Published in Transactions on Machine Learning Research (1/2025)
bilinear zero-sum games can be formulated the following min-max problem:
min max V(x, y; W), (0, we (0.11)
where .) given by
1 x W W
1
V( y: W) 2dz
y W Wyy Vdzd, x. T W Way' y + xx bb, +y byyy 2dy
where W (W zz, W yy. W Wxy b by). For simplicity, Way Ld xd, E Rdy We, by Wyy Lay xd, E idd, = bg E Rd,, and by - byld, E Rdy, The following results can be readily extended to the matrices W W up. and Way with a limited number of eigenvalues of Od,,d, (1). For a detailed discussion, refer to Appendix B
assume W Wzrld, Rd=xd= W
Wzs
In this setting, the analytically contimued free energy density f (8min mmax W) calculated using replicated partition function Z, (Smin: (manx Eq. (6) coincides with the exact free energy density f(mmin 9max W) from the partition function Z, (8min Bmax) in Eq. (5).
Theorem 4.1 For anyy Buins Pmax R and Wans Ways Way: bus by R, the following equality holds:
f(8mia Pmaxi W) f(Bmin Smaxi W).
where
f8min Pmaxi W)
extr m .
Wyr 2
) +
K00yy (m²)2 + Way m2 m
2
m² + kbym
H(m") Bmax
H(m²)
Bunn
where K === dufde, (xx) - log(x)) - (1 - x)) log(1 x) denotes binary cross entropy, and extr denotes the extremum operation.
This theorem establishes the validity of the analytic continuation for the replica parameter p using Eq. ( for bilinear min-max games. The detailed proof of this theorem is provided in Appendix A
5 Generative Adversarial Networks
Generative adversarial networks (GANs) (Goodfellow et al.) 2020) aim to model high- digh-dimensional probability distributions based on training datasets. Despite significant progress practical applications (Arjovsky ltt al.) 2017 Lucic et al.] 2018 Ledig al. 2017 Isola al., 2017 Reed et al.] 2016), several issues are yet to be resolved, including how the amount of training data influences generalization performance and how sensitive GANs are to specific hyperparameters. This section analyzes the relationship between the amount of training data and generalization error. Additionally, we conduct a sensitivity analysis on the ratio of fake data generated by the generator to the amount of training data, which is critical for the training of GANs. Our analysis employs a minimal setup that captures the intrinsic structure and learning dynamics of GANs (Wang et al. 2019). We consider the high-dimensional limit, where the number of real and fake samples, n and n, respectively, and the dimension are large while remaining comparable. Specifically, we analyze the regime in which n, n.ñ.d ă n/4 (dd),, commonly referred to sample complexity.
while maintaining comparable ratio, i.e., n/a Odd") and
5.1 Settings
Generative model for the dataset We consider that training dataset D (==)"= where each x Rd drawn from the following distribution:
1 w dt + Vmm"
2h
(7)
Vd
5

---

Published in Transactions on Machine Learning Research (1/2025)
where w" Rd is deterministic feature vector, cx E R is random scalar drawn from standard normal distribution p(c) = -M 1), nx is background noise vector whose components are i.i.d. from the standard normal distribution (n; 0a, Ia), and 17 E R is a scalar parameter to control the strength of the noise. We also assume that | 112 = 1. This generative model, known as the spiked covariance model (Wishart) 1928
Potters & Bouchaud 2020 has been studied in statistics to analyze the performance of unsupervised learning
methods such as PCA (psen Hansen 2019 Bichl & Mietzzer 1993: Hoyle & Rattray 2004); sparse PCA
Lesieur al., 2015), deterministic autoencoders (Refinetti & Goldt 2022). and variational autoencoder
Ichikawa & Hukushima 2024
2023
GAN model Following Wang et al. 2019), we assume that the generator has the same lincar structure as the dataset generative model described in Eq. 7
g w)
Vd w2 + Viñ.
(8)
where w E Rd is learnable parameter, IR is latent variable drawn from a standard normal distribution p(z) = Oz 0),, 1), á is a noise vector whose components are i.i.d. from the standard normal distribution [nn; 02, La), and ij E R is a scalar parameter to control the strength of the noise.
We also define the lincar discriminator
(x; v) = (
1 x)
(9)
where x is an input vector, which can be either the real data xn from Eq. or the fake one g(>"; w) from The vector E Rd is a learnable parameter, and IR + R can be any function.
Eq. S
Training algorithm
min-max The GAN is trained by solving the following min- max optimization problem:
min max V(w, w; D),
(10)
wERd VER
where
n2 V(w, v: D.
>
3 1/11 1u0 ||/11/2 2
Σ Φ (w(a":)))
(11)
Fu=i are the latent values of the fake data. The last two terms are regularization terms, where A and Å control the regularization strength. This value function defined in Eq. 11] is a general form that includes various types of GANs. Specifically, when & = ō and || < 1, it represents a Wasserstein GANs (WGANs) (Arjovsky et al.) 2017) and, when Φ(x) = log (x) and ø(z) = log(1 - (xx) with a being the sigmoid function, it corresponds to the Vanilla GANs, which minimize the JS-divergence (Goodfellow let al. 20) As we assumes linear discriminator, Vww, D, D) can be expressed a function of linear combinations T uTa" /Va and TT "ls":w/// va follows:
and D = {=
((,,;,D,D)
Σ Via
Φ v x²)) #=1
1
A
N
(12)
where, for clarity in the subsequent analysis, we redefined & and & as functions of the lincar combinations TTan //a and TT "=2="m))/ Va.
Generalization error In the ideal case where the generator perfectly learns the underlying true probability distribution, we have w² w. Therefore, define the generalization error Eg as
Eg (1, w0) A Ep
[[|@ w=1 ==113]
(13)
where w denotes the min-max optimal value Eq. The generalization error, Eg, quantifies the accuracy of signal recovery from the training data.

---

Published in Transactions on Machine Learning Research (1/2025)
5.2 Replica Calculation
We apply the replica formalism sketched in Section 3 to derive set of deterministic equations characterizing the typical behavior of GANs.
In this problem sctting, the replicated partition function Z, in Eq. 1 can be expressed as
7
P dw II [ doal e En of (((()))) w'cc / ((o0!)T "
Zy (Amin 3max)
II
Rd
Ege Banax En. "(t(n"))" w==z (((())))))))))))))
Amax Eu 101l= (11011))))))
To take the average over n and n, we notice that since n and ni follow a multivariate normal distribution (i;; Oa. Ia), the quantities u = ((va)) T n/ Vd)a, and i = ((val) nz/ Vd)aa also follow Gaussian multivariate distribution as
p(u) p(ū) NN Q):
where
(Qt)) E BR Q j (
To conduct further computations, we introduce auxiliary variables through the following identities:
1 1 II I=I[ a ff f S(dQF) - ( T JdQ III d 8(dmf - (va1) T "ww" "ww"ddd dmm III
/ 8(dbf (val) T Tw=" dDG.
abls
The replicated partition function then be expressed
/ dQdmdbe (Q
Z, (3mins Smax)
where the entropic term S(Q, m, b) and energetic term T(Q, m) are defined as follows:
/ II dwo at
II d f 5(dQ)) (val) Toe))
S(Q, m. b) A
dBmin
abls
f 5(dmf - (Dal) "n'dd f 8(db8 - (v (ol))T=)e Amax Ens (A112001 2 A1lll00'111
x II
al
(fDc) dup(u)e Bmax E.r o (1(1)))) w0 w / ((uul)TT))
T(Q, m, m,b) b) In
A
Bmin
(JD=) dup(ū)e Banx Eas (3((o))) w" / (()))))
+
In
Buin
Using the Fourier representation of the delta function, S(Q, m, b) is further expressed as
1
/ [Qudmdbee (( (t-QQ--- mm
S(Q,m,b)
log
d.3min
/
II
dw d E.n, prlpal E., a0 Enn by u0 Eapu E., (A(m -x (14)
al
Replica symmetric ansatz
Here, we assume the following symmetric structure:
ba E [7] Vl, E [p], Q +
Sab +
X
Öisdan
(15)
Bmin
Bmax
B Adab - &:
Va,be E [7]. Vl E [p]. Qaa = fisx
(16)
Buin
Va e 11 [p]. mi = m, mf 3uaxth.
(17)
Wa E b)) [p], bf b, Bq = Bmaxb.
(18)

---

Published in Transactions on Machine Learning Research (1/2025)
This replica symmetric (RS) structure restricts the integration of the replicated weight parameters {w"), )
{va)} across the entire (((x)))) x R(dxxp) to subspace that satisfies the constraints in Eg. 15 18). This structure, along with scaling by the maximum and minimum beta values, is similar to the standard one-step
replica symmetry breaking (IRSB) (Mézard et al. 1987 Takahashi & Kabashima, 2022).
We now turn to the entropic term S(Q, m, b). The terms that exclude the integrals with respect to {val) and {w") can be expressed as
1 tQQQ in m b
2
= 7mimi
(-2 á 9+
g+
x Buax
x
8
Â
RA +
åq
+
+
mm + 86)
(19)
Bmin
Smin
Bmin
The term that includes the integrals with respect to {val and {wr) can be expressed as
-2 (v + En (Voann à 5t Viste" Vioter+tu" ont samin E. (w=0)
1x
II al
dw DC d
Amin
P V 5"t +x+Vxxto m+ (tttum+t"b)))
/ II dw DS DÇ"
sdose - +fmmxll+22))
(Vi
Admin
2 E (w0
Exg
/ II dw e
Aggan Σ. (w= (2==)² -2-
Aran
E. (V
Ex
2(0
Pain
Ez (ddnakse Buin (-2(2- 20 (VACI m6)?
2(0+x)
This can be derived using the identity, for any a E RR and any x E R, e#n" Dzevāse Summarizing these results, the entropic term can be written as
S(Q, mQ b, Q m. b) = (-2 a
A
X
x+
Â Bmin
+ + + Zq
Ad
+ mm + 86
Suin
Busax
Bmin
-2==-50² sw² wacc t mnt wai?
1
/ Dz
/
Bmnn
dwddee
210
Buin
By laking the limit as Bmax > oo followed by Pmin + oo, we obtain
S(Q. m. m.m.b.Q.mbb ? 50000 A) -)) (x (xx 2mm 21 + 8 - (4 + â + A)A
i(n² + x)
We next turn to the energetic term T(Q, m,,), b). Under the RS ansatz, u follows
al
X
al
+
A
1fth
VOC.
Buax
Buin
8

---

Published in Transactions on Machine Learning Research (1/2025)
where gat, xd1, 8d yd ç, and & follow the standard normal distribution WV 0, 1). Then, the energetic term T(Q. m. b) can be expand
In (f De dup(u)e Amax E., (2(o01)) w' -o-c-- ct ===2("")
T(Q, m)
mia
(fDz / dūp(üj} -max Ens of (1 "0=* / In
-mmax
3min
(b)
The term (a) can be simplified as
(a)
a Bmin
In Ee,e
/ II
Dyy Dy Da Swnex &
Ene @ (me+
Paaax
Pminn v" V
+
al
a Bmin a Aiin
EEGE
[J DJ (
(Dare Drremmann mot mil
Pmas
Proin v4t5)
log f Dy ( Doce Mareemanneenn io V
Mmin 1/ vTL
+ O2(7))
a
"Bei
log
dye
Emjuy?
(feeee Amaax tmma me+ MV +0,(1))
+0,(73)
Pmna
Taking the limit as Amax oo followed by Buuin Do, obtain:
{
12² max { x2 Φ (mc+ vn (VXc+ VDy+ vot)
(a) = ayyag max
Similarly, the term (b) is also expressed as
ā E, In
dğe Eugiu? y² (ʃ dive Sugur 2agpxg2-mmmae -mmn 0(2 S +0,
+
(b) mmnu
Taking the same limits, we find:
(b) = a. max
2 12 max
1 x2 Φ bb vi (ix++ VAy + voc
Putting the entropic term and energetic term together, the free energy density is given by
f f= extr
1
qå (x- A)X 2(mm + 66) + 82 + (1 Λ)A
i(m² + +x)
A.x.(q + aá x. m. m.)))
6 2 m.
0.m.m,
where
1
1
(gg. A, m, = Engg
max
max
Φ me + vū (xx + vAy + v
(20)
2
1 x²2 (bz + vi Vxx (vx3+ VDJ+ VV
(g. A, m,
E,,
max
2 y
max
(21)
2
Note that the min and max operations are involved in the two-level optimization described Eqs. 20 and
21
5.3 Results: Application Simple GANs
In this subsection, following Wang al. 2019 we apply the formulation derived above to the simple WGAN to demonstrate its generalization properties and conduct a sensitivity analysis of the ratio fake to real data.

---

Published in Transactions on Machine Learning Research (1/2025)
3.0
r-0.25
r- 1.00
2.5
0.50
1.25
1.5
2.0
lim Eg (a)
1.0
1.5
1.0
0.5
0.5
0.0
0..00
10
20
30
40
50
0.25 0.50
0.75
1.00 1.25
1.50
a
r
Figure 1: (Teft) Generalization error as a function of sample complexity a for different values of the ratio r. (Right) Asymptotic generalization error lim E(a) as function the the ratio r.
Self-consistent Equations We consider the case where the functions ((x) and ((x) are both quadratic, defined as x(x)= = ((x) = x² z²/2. This setting allows for an explicit calculation of the free energy density, which is given by
A(m² + +
a(n + m²)
ā(ñg + bD) 7 +
]
f= extr m. &
q9 x²
2mm 268
(22)
2
82 + (6 A)Ā
nx -
To find the extremum in Eq.
22 require that the gradient with respect to each order parameter equals zero. This results in the following set of self-consistent equations:
1 +))
mi
bā(m2 + x)
q ( \(q + 2))2²
=
82 + \(y + A)
82 + \(4 + A)
b b
(62 + X(q + A))
q = an
7x +1 8 an(an + m p)
arj
āñ((qn + &PP))
am
b āb
+
+
m
(nx 1)2
(ix + 1)2
7X
7x +1
Our analysis focuses how the generalization error depends on a while varying the ratio r, as generating fake data from the generator is generally much easier than collecting real data. Fig. [ (Left) shows the dependence of generalization error on sample complexity a for various values of the ratio r. The results demonstrate sharp decline in the generalization error as the ratio r increases. However, when r becomes large, the generalization error increases in the region where is large, eventually leading to phase where no lcarning occurs, and the generalization equals 1. This implies that as increases, the learning becomes dominated by only fake data.
Learning Curve For simplicity, set à and x= =
In contrast, for smaller r, real data consistently dominates the objective function V(w, D)D), resulting in steady decrease in generalization error. However, the reduced influence of the fake data component in the objective function, which drives the learning of the generator, requires a significantly larger amount of real data for effective generator training.
Asymptotic Generalization Error We next analyze the asymptotic behavior of the generalization error when the sample complexity becomes sufficiently large. The asymptotic behavior of the generalization error function of is given by
2 E LEVE
2v (V[(F + Oa(a-1) r rs 1
Eg
(r-1)ra!/2
1 1+Oala
>>
10

---

Published in Transactions on Machine Learning Research (1/2025)
The results for data approximately equal to half of the real data is effective when the dataset approaches infinity. At r= 1, a phase transition occurs, suggesting that the model changes from a phase of effective learning phase to one where fake data becomes dominant. Beyond this point, for r 2 1, the model fails to learn any meaningful signal w", and the generalization error 1.
shown in Fig. 1 Right). The optimal ratio is r r= 1/2, indicating that using fake
Furthermore, when r 1/2, the generalization error scales as Ey a which represents the optimal asymptotic behavior for model-matched scenario. These results demonstrate the critical role of the ratio in determining learning performance. Therefore, tuning the ratio r according to the available real data is crucial for achieving optimal performance. In practice, it is known that in training GANs, the stability of learning can deteriorate depending on the ratio of fake to real data. This theoretical analysis provides insights into the importance of the ratio r and expected to contribute to improving learning algorithms.
6 Conclusion
This study introduces statistical mechanical formalism to analyze high-dimensional min-max optimization problems, focusing on the critical order of min and max operalions in non-convex scenarios. Our goal was to perform sensitivity analysis of equilibrium values, providing new insights into their properties and generalization performance.
We applied this approach to a simple min-max game, evaluated the generalization performance of GANs, and derived the optimal ratio of fake to real data for effective learning. This successful application not only validates the approach but also opens the way for extending this formalism more complex min-max problems and broader applications, suggesting a promising direction for significant advancements in machine learning and optimization.
Acknowledgments
We thank T. Takahashi, K. Okajima, Y. Nagano, and K. Nakaishi for useful discussions and suggestions. This work was supported by JST Grant Number JPMJPF2221 and JPSJ Grant-in-Aid for Scientific Research Number 23H01095. Additionally, YI was supported by the WINGS-FMSP program at the University of Tokyo.
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214 223. PMLR, 2017.
Benjamin Aubin, Antoine Maillard, Florent Krzakala, Nicolas Macris, and Lenka Zdeborová. The committec machine: Computational to statistical gaps in learning a two-layers neural network. Advonces in Newral Information Processing Systems, 31, 2018.
Benjamin Aubin, Florent Krzakala, Yue Lu, and Lenka Zdeborová. Generalization error in high-dimensional perceptrons: Approaching bayes error with convex optimization. Advances in Newral Informalion Processing Systems, 33:12199 12219,, 2020.
Jean Barbier, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová. Optimal errors and phase transitions in high-dimensional generalized lincar models. Proceedings of the National Academy of Sciences, 116(12):5451 5460, 2019.
Michael Biehl and Andreas Mietzner. Statistical mechanics of unsupervised learning. Europhysics Letders, 24 (5):421 1993.
Blake Bordelon, Abdulkadir Canatar, and Congiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pp. 1024-1034. PMI.R, 2020.
11

---

Published in Transactions on Machine Learning Research (1/2025)
Ronald E Bruck. On the weak convergence of ergodic iteration for the solution of variational inequalities for monotone operators in hilbert space. J. Math. Anal. Appl, 61(1):159 (1),119,, 1977.
Hugo Cui and Lenka Zeleborová. High-dimensional asymptotics of denoising autoencoders. ar preprinl arXiv:2305.11441, 2023.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis, and Haoyang Zeng. Training gans with optimism. arXiv preprint (Tf:::771100144. 2017.
Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. Thermodynamics of restricted boltzmann machines and related learning dynamics. Journal of Statistical Physics, 172:1576 1608, 2018.
Vladimir Fedorovich Dem Con'yanov and Aleksandr Borisovich Pevnyi. Numerical methods for finding saddle points. USSR Compulational Mathemalics and Mathematical Physics, 12(5):11-555 52, 1972.
Rainer Dietrich, Manfred Opper, and Haim Sompolinsky. Statistical mechanics of support vector works Physical Review Letters, 82(14)22975, 1999.
Samuel Frederick Edwards and Phil W Anderson. Theory of spin glasses. Journal of Physics F; Metal Physics, 5(5):965, 1975.
José Fernando Fontanari. A statistical analysis of the knapsack problem. Journal of Physics A: Mathemationl and General, (8(1))477 1995.
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicative weights. Games ond Economic Behavior, 29(1-2):79 103, 1999.
Marylou Gabrié. Mean-field inference methods for neural networks. Journal of Physics A: Mothematical and Theoretical, 53(22)) 2020.
Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general, 21 (1(1)277 1988.
Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. In International Conference Machine Learning, pp. 3452 3462. PMLR, 2020.
Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. variational inequality perspective on generative adversarial networks. arXiv preprint omXiv:1802.1055/ 2018
Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Rémi Le Priol, Gabriel Huang, Simon Tacoste-Julien, and Toannis Mitliagkas. Negative momentum for improved game dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1802 1811. PMLR, 2019.
lan Goodfellow, Jean Poudet-Abadie, Mehdi Mirza, Bing Xu, David Farle, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Newral Information Processing Systems, 27, 2014a.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiu preprint ar 1rxii:1412 22..1.2.666 6572, 2014b.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
David C Hoyle and Magnus Rattray. Princhal-componnntaanlly. cigenvaluc spectra from data with bymmetuy-breaking structure. Physical Review E, 692)):226124, 2004.
David C Hoyle and Magnus Rattray. Statistical mechanics of learning multiple orthogonal signals: asymptotic theory and fluctuation effects. Physical Review F,, 75(1) 7506611,,,,,, 2007.
12

---

Published in Transactions on Machine Learning Research (1/2025)
Yuma Ichikawa and Koji Hukushima. Statistical-mechhniiee study of deep boltzmann machine given weight parameters after training by singular value decomposition. Journal of the Physical Sociely of Japan, 91 (11):114001, 2022.
Yuma Tchikawa. and Koji Hukushima. Dataset size dependence of rate-distortion curve and threshold of posterior collapse in linear vac. arXiv preprint arXiv:2309.077 07663, 2023.
Yuma Ichikawa and Koji Hukushima. Learning dynamics in linear VAE: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with KL anncaling. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp. 1936-1944. PMLR, 02 04 May 2024.
Niels Ipsen and Lars Kai Hanson. Phase transition in pea with missing data: Reduced signal-to-noise ratio, not sample size! In International Conference on Machine Learning, pp. 2951 2960. PMLR, 2019.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125- 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and poltern recognilion, pp. 1891- 2017.
Thibault Lesieur, Florent Krzakala, and Lenka Zdeborová. Phase transitions in sparse pea. In 2015 IEEE International Symposium on Information Theory (ISIT), pp. 1835-1639. IEEE, 2015.
Tengyuan Liang and James Stokes. Interaction matters: A note on non-asymptotic local convergence of generative adversarial networks. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 907 915. PMLR, 2019.
Picrro-Louis Lions. Une méthode itérative de résolution d' une inéquation variationnelle. Israel Journal of Mathemalics, 31:204 208, 1978.
Mario Lucie, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. Advances in Neural Information Processing Systems, 31, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXie: 17X.::7766.0683, 2017.
D Maistroskii. Gradient methods for finding saddle points. Matekon, 13(3):22, 1977.
Mare Mezard and Andrea Montanari. Information, physics, and computation. Oxford University Press, 2009.
Mare Mézard and Giorgio Parisi. A replica analysis of the travelling salesman problem. Journal de physique, 47(8):1285-1225 1986.
Mare Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987.
George Nemhauser and Laurence Wolsey, intescicne series in discrete mathematics and optimization. Integer and Combinatorial Optimization, pp. 765-766, 1988.
Manfred Opper and David Haussler. Generalization performance of bayes optimal classification algorithm for learning a perceptron. Physical Review Letters, 66(20):2677, 1991.
Nicolas Papernot, Patrick McDanicl, Somesh Jha, Matt Fredríkson, Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE Europeon symposium on security and privacy (FuroS&P)) pp. 787. IEFE, 2016.
13

---

Published in Transactions on Machine Learning Research (1/2025)
Giorgio Parisi. Toward a mean field theory for spin glasses. Physics Letters A, 73(3)200 205, 1979.
Giorgio Parisi. Order parameter for spin-glasses. Plysical Review Letters, 50(24):1946, 1983.
Mare Potters and Jean-Philippe Bouchaud. A First Course in Random Matrir Theory: For Physicists, Engineers and Dala Scientists. Cambridge University Press, 2020.
Meisam Razaviyayn, Tiajjian Huang, Songtao Lu, Maher Nouiehed, Maziar Sanjabi, and Mingyi Hong. Nonconvex min-max optimization: Applications, challenges, and recent theoretical advances. IEEE Signal Processing Magazine, 37(5):55-66, 2020.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pp. 1060 1069. PMLR, 2016.
Maria Refinetti and Sebastian Goldt. The dynamics representation learning in shallow, non-linear autoencoders. In International Conference on Machine Learning, pp. 18499-185199 PMLR, 2022.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprinl Xiv:13312 6199, 2013.
Takashi Takahashi and Yoshiyuki Kabashima. Macroscopic analysis of vector approximate message passing in mismathmdd setting. IEEE Transactions Information Theory, 68(8):55579 5600, 2022.
Toshiyuki Tanaka. Moment problem replica method. Interdisciplinary information sciences, 13(1):17 23, 2007.
Kasimir Tanner, Matteo Vilucchio, Brumo Loureiro, and Florent Krzakala. A high dimensional model for adversarial training: Geometry and tradc-offs. arXiv preprint arXiv:2402.0667, 2024.
Paul Tseng, On linear convergence of iterative methods for the variational inequality problem. Journal of Computational and Applied Mathematics, 60(1-2):277-55,, 1995.
Poter Varga. Minimax games, spin glasses, and the polynomial-ting hicrarchy of complexity classes. Physical Review E, 57(6))64887, 1998.
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior, 2nd rev. 1947.
Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Mathematics, 46 (2):265 280, 1945.
Chuang Wang, Hong Hu, and Yue Lu. A solvable high-dimensionall model of gan. Advances in Neural Information Processing Systems, 32, 2019.
John Wishart. The generalised product moment distribution in samples from a normal multivariate population. Biometrika, pp. 32-52 1928.
Lenka Zelebrrvá and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms. Advances in Physics, 65(5):453 552, 2016.
14

---

Published in Transactions on Machine Learning Research (1/2025)
A Derivation of Theorem 4.1 proof
In this section, provide the derivation proof of Theorem 4.1 The derivation begins with the calculation of the free energy density without the analytic continuation of p Smin/ Pmax to p E N. The free energy density in Eq. (2) is connected to the effective Hamiltonian, Hott (x; W), which is defined through the relationship:
13min Smaxi W)
1 dn
Ew log Σ exp (-3min Hott( W)).
The effective Hamiltonian is given by
(x; W)
log Σ
Smac V (w.y:WW)
3max
Amax Wrnda Telles 1dn )a= dxwax (u 4
1y )² twoy / dddy ( 1da
log
Σe
dx
3max
y
dadel ao Idde ) +bydu v tode)) dx
Amnx
e
Smaxdy Waze
dee )² tbo 1dz
log
dx
Smax
tworo / de T Tdi ) ut (Tidg) th 1T
dy
x
y
Wardy
x ldze (x]ee) + bads x 1dz dx
Bmaxxa
dx
log [
Smandy
""ge (m -aay---ayy
" " Bmax Softplus(1te)) +o(d,) mn byy
/dnddme
x
Bmax
where Softplus(x) = log(1 + e.). To evaluate the intogral with respect to mh and m² we take the limit as dy + oo and apply the saddle point approximation. The effective Hamiltonian can be expressed as follows:
Hett (x:; W) d2
Wax
x T1de Id
bx
x Idz dx
2
d₂z
Wyy (m²)² 32-2 + Wayk - 1/2 (
x Id,
1
1 3
Softplus(m)) ))
+ extr
m + bym"
mm
dz
Bmax
15

---

Published in Transactions on Machine Learning Research (1/2025)
where K dy/ds. By summing over æ, the free energy density can be calculated follows: f(3min 9maxi W)
a T -a
Amind
tdo
log
&
"xx (m + 1/2( Tdte )
-mmnndd
k exer m² vay
m bym
Imax m "in" Pmax
Softplus (1ic")
x
de 2T
/
dm" drnn -maadd (gga (m=")²++ +bam Pensn
Softplus(1ii"
log
-aiiado
i extr 1 (m²)² - x "n" thym" Amax mU.mu"
m mn Softplus(mn))
Waz (m=)² bym" "mmm +
1 Softplus(m")
extr m= ris" m shev
m m²
2
Brin
Bunin
Killy (m²)) + Wayk' //2 m2mm² m² + robym"
m²m²
(oftplus(mn))
Bmax
Bmax
The final equality obtained by applying the saddle point method evaluate the integral. From the saddle point equations, the following expressions
m ( m ar (
Further transformation of the equation yields the following expression:
Wrz 2
(m )² + kWWy (m²)² + Wayk 1/2 mmmm² m2 + bym = + m
[Bmin fmaxi W)=
extr m .
2
H(me)] (23)
H(m") +
Buin
Smax
where (xx) - log - (1 x)) log(1 x)) represents the binary cross-entropy.
Next, we proceed to evaluate the free energy density under analytic continuation in the replica method, which is expressed as
Smax de
"igel
1d
K
Σ
M/ Jdy dy
[Bmin Bmaxi W)
log
Σ
de
Buin d,
Vi....lyy
Amacdda
(Wyye 1/2
ao Idz
M Jily
tb.p
Jide de
txcyy E,
E
We introduce the order parameter through the Fourier transform representation of the delta function:
f {Bmins Smaxi W)
1 ( dr (2T))
) dm" drn II dm dri Smorda
"Zguz (m²) - E (mif) way
t m2 E, m?)
log
Bm
x c Smordd (bapm "oomm+*ne treby E, mi. Smax ris" mn=
Smiwe E, i mmm?))
Σ
L m E,
dz (2
dm" dmn II dmy d e mm ""get (m²)² sway Σ, (m²) (2 + x m² E, m))
log
Bminde
X e Imoardo (Dapm" troy byyy E, m.. Denase
the" m""
Trause E, m.7m mm Tumnx Softplus(m - Σ, Softplus(rr,))
Jumns
16

---

Published in Transactions on Machine Learning Research (1/2025)
Under the assumption of replica symmetry, where 47 [p], my m² my m², can further reformulate the expression as follows:
f(3min Smaxi W)
log I dm" drfn" dm" e Aminds (=qa (m²) ") muge (m")² +wayn // 1/ "mm²m)))
x e Puiind (b + t m Amaxp ii =m= Amax in mn Amaxy Softplus(m1 )) smiex (oftphu(("")-
extr mm.. M
Uaa
(m²)
Kw (2 + Way A 1/2 12 mm + bam + kby
2
2
fn m
k
m²m
1 Softplus(m))
x Softplus
maxx
Bmax
fmaxP
Buax
The final equality is derived by handling the integral using the saddle point method. Consequently, the following saddle point equation is obtained:
m (m)", m" (
Substituting these results, the following expression for the free energy density derived as
fBminn Amaxi W)
extr m .MV
W22
(m )2 2 +
KW (m + wayk 1/2 mmm m2 + bamr + mD
2
2
H(m") )+m")++ + Bmax Hmme)) (24)
Buni
This result coincides with the exact free energy density f'8min, 9max; W), derived without the need for analytic continuation.
B Generalization of Theorem 4.1 to Finite Eigenmodes
In this section, generalize Theorem number of eigenmodes as follows:
4.1 to cases where WaE, W and Wy, Wy decomposed into finite
W Z2 aaa W Σ
Bmbmb Way
Σ
n.C.CT as am, Ins M,N N Od,,d, (L) (1).
In this formulation, V(x, y, W) can be expanded as follows:
V(a, v: W)
d, Σ
01
1
a:
d Σ Bm
1 y bm
+ v dady
Σ Yn
1 x
TTc,,
(
c
dz
dy
ds
dy
t=1
m=1
+ byd x
1dt
+ bydy
y 11 1db
dn
dly
This expansion constitutes direct extension of the caleulations in Appendix A with the terms x Las/ds and y 1u,/d, augmented by overlaps with the cigenvectors, {= az/d, ht ty"bm//a 1m and {=T TeTe//d, y² en/d, N By performing analogous calculations, we find that the free energy density, without assuming analytic contimuation, aligns with the analytically continued free energy density. Furthermore, this free energy density is characterized by the saddle point condition involving M + = Oa, a, (1) variables, as Eq. (23). If we assume L, M, Od,.d, (dz), (dz), then as d, 0o, the limit becomes trivial; Without appropriate scaling, the free energy density diverges.
17

---

Published in Transactions on Machine Learning Research (1/2025)
C Evaluation of Functions of the Optimal Value of Min-Mar Problems
In this section, we present method for evaluating the expected value Ex over set of problem instances A, where ((A) argmin EEx maxyey Væ, y; A) represents the min-max optimal solution. This analysis provides insights into how Eq. (2) functions as a generating function. We also demonstrate that this
approach can be directly applied to evaluate the generalization error in Section particularly Eq. 13
The key idea is expand EA (a(G) as follows:
/ Pmnn (z)
AA (G(((())) EA
lim
lim
Buiin + Smax +
E.
lim
lim
ow f(3miu Pmaxi (x()))
Ami - 00 Smax
where we extend the free energy from Eq. (2) by introducing a parameter w as follows:
(-Amin log
/ dyy exp (Smax yi A)) + (((())
f(8mnn imaxi (G(x))
Amindy A log dx exp
Bmax
We employ this technique to derive the generalization error in iention for GANs can be expressed as:
Specifically, the generalization error
1 (g w²) ED (D) w" w=1l" 112
compute this, we augment the free energy calculation by adding the term w[||w||2 2w T w²). This adjustment is incorporated into the calculation by modifying the term X(u2)² in the exponent of the d-th power expression in Eq. 144 to (w )² - 2w w")). Since these terms quadratic, the Gaussian integration remains straightforward. The remaining calculation follows the same procedure in the main text and is thus omitted for brevity. Eventually, we obtain the following form:
lim Eg --A-2 & m, + (2.
where A(q, R, m, 8) is determined by Eq. 22
explicitly given as
1b+ (((++)) 2)
A(g. & m. b) b)
V V
83 + x({+1)
18


Reviews:
[{"review_id": "0g6io4C3tN", "reviewer": "Reviewer_oYGi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Main review\n--------\n- I agree that, in principle, the papers main idea can be useful. However, at least to me, the mathematical derivations are not clear, which is why I recommend only accepting this paper if substantial revision have been made. To be more precise: pretending that parameters $p$ and $\\gamma$ are integers and that the entries of the vectors in question are iid (that is, we replicate a 1d-particle d times) one can pull the exponent out of the expectation and together with the limit formula for the natural logarithm exchange expectation and logarithm. But, then taking $\\gamma \\to 0$ conflicts with the assumption that $\\gamma$ is a integer. However, no bounds / estimation errors or any of the like are presented, so for an non-expert like me it is very difficult to gauge if this heuristic is meaningful. The only concrete evidence for the efficacy of this idea is provided in the proof of Thm. 3.1, however it seems to me that the calculations performed there also hinge on the fact that the parameters are integers, so that, in effect, nothing is proven.\n- Furthermore, if I understand correctly, mathematical equality and heuristic are mixed, e.g. (6) should not be an equality, but the right hand side is just an heuristic to the left side, right?\n- Equations (3) and (1) are contradictory since $\\Omega$ depends on $A$, however, the right hand side of (3) does not.\n- It would be nice if there were so hints given, e.g. in the appendix, on how the claimed extension of \"scalar\" $W_{xx}$ to matrices with limited number of eigenvalues (what does \"of $\\mathcal{O}(1)$ mean here?) can be accomplished.\n- Furthermore, I do not understand, why in going from (14) to (15), there is no $f$ in the latter formula. Naively plugging in (12) into (14) gives a different result than (15).\n- Am I assuming correctly that in subsection 4.2 we always consider $\\gamma, p \\in \\mathbb{N}$ and not use implicitly somehow \"analytic continuation\". If yes, this should be clearly stated at the beginning of the subsection.\n- What are the $Dc$ and $Dz$ in the equation defining $\\mathcal{T}$ and is the integration over $\\mathbb{R}^{p \\gamma}$?\n- I think that the self-consistent (what does this mean, exactly?) equations after eq. (27) are not quite correct: the signs of b and \\hat{b} should be reversed and we should instead have $\\hat{\\chi} = - \\frac{\\alpha \\eta (q \\eta + m^2)}{(\\eta \\chi - 1)^2} - \\frac{\\tilde{\\alpha} \\tilde{\\eta} (q \\tilde{\\eta} + b^2)}{(\\tilde{\\eta} \\chi + 1)^2}$.\n- To be honest, I do not know how $\\varepsilon_g$ was derived and would appreciate a bit more detail on that.\n- For people not too familiar with statistical mechanics I recommend providing more background, maybe in form of a short and concise subsection (clearly defining the terms \"$\\extr$, \"(virtual) two-temperature system\", \"randomized instance ensembles\", \"dataset ensembles\", \"complete rationality\", \"handling the integral using the saddle point method\") and adding more references, e.g. for backing up the statement that \"Such an approach generally does not yield accurate results when the function $V$ is non-convex\" or the imprecise claim that \"the distribution $\\lim_{\\beta_{\\max} \\to \\infty} \\lim_{\\beta_\\min} ...$ concentrates on a uniform distribution over the min-max values\" (this statement is not precise regarding the type of convergence and probably needs some assumptions on $V$).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose using a heuristic from statistical mechanics, namely the replica trick, to study min-max problems. They use the Boltzmann distribution in the limit to describe the min-max-value and the replica trick in essence allows for exchanging expectation and logarithm to simplify computation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Typos\n------\n- In the notation section it should be d a^i instead of d a_i, because that is how it is used in the sequel if I am not mistaken.\n- In the sentence after eq. (1) it should say \"e.g. *a* graph $G$\".\n- At the beginning of the first line of p. 3 there is an \"is\" missing.\n- In eq. (6) in the exponent the \"; A\" should be included in the round brackets.\n- At the beginning of p. 4 it says \"$V(\\cdot, \\cdot)$ is given by $V(x, y; W)$ ...\"\n- Section 4.1: it should say \"drawn *from* the following distribution\" and \"*are\" the latent values of the fake data\".\n- Section 4.2: the expression Z_gamma is missing the $(\\beta_\\min, \\beta_\\max)$ (at least that is how it is written in other parts of the paper).\n- Line before eq. (23): it should say \"*we* obtain\".\n- Figure 1. \"*Generalization* error\"\n- In the first formula in appendix A the expectation should be over W instead of over A.\n- \"non-convex to the variables\" sounds off, I instead suggest \"with respect to $x, y$\" or something similar.\n- In (7) the dependence of $V$ on $A$ is missing.\n- In the last line of the big equality chain on the first page of the appendix, should it be $+$ instead of $\\times$ separating both terms?\n\nBibliography\n-------\n- Leaving out the third author, Jean Barbier, in the second reference and then using \"et al\" seems strange. This reference is also missing pages numbers.\n- Names of conferences and journals, like ICML, NeuRIPS, Physical Review Letters, or Princeton University Press should be capitalized.\n- In the fifth reference, the first names of the authors are abbreviated by their first letter, which is inconsistent with the other references. Something similar is true for the paper by Nemirovsky et al. and by Papernot et al. Similarly, Ian Goodfellow is once listed like that and once as Ian J Goodfellow.\n- In the seventh reference, the name of the journal is abbreviated, which is inconsistent with the other references.\n- Mathematician's names or common abbreviations like GNNs, PCA, VAE, Boltzmann or Hilbert should be capitalized, like they are in the titles of these articles.\n- The reference \"Learning dynamics in linear VAE: ...\" is the only one with a URL attached, which is inconsistent.\n\nFormatting\n-------\n- I suggest replacing $e^{...}$ by $\\exp\\left(...\\right)$ in almost all instances to enhance readability, because the ... will be too small otherwise.\n- I also think it would benefit to have brackets indicate where the arguments of e.g. the natural logarithm or an expectation end, and it is clear if exponents are outside or inside of these operators.\n- The notation $p(c) = \\mathcal{N}(0, 1)$ and its cousins used in section 4.1. are not clear to me, since the right hand side does not depend on $c$. If I see this correctly, $p(c)$ should be the value of the density of a N(0, 1)-distributed random variable at the point $c \\in \\mathbb{R}$.\n- Omitting indices and regions of integration declutters the notation but makes it harder to see what is going on, especially when reading through this submission for the first time. For example, in the first big equation in subsection 4.2, is the integration over $\\mathcal{X} \\times \\mathcal{Y} = \\{ 0, 1 \\}^{d_x + d_y}$, as suggested by equation (7), or over $\\mathbb{R}^d \\times \\mathbb{R}^d$?\n- Using $d$ for both the discriminator, which takes input variables, and the dimension (see notation section) can cause confusion.\n- I am assuming that equation (4) is a definition, which is usually marked with $\\overset{\\triangle}{=}$ in the submission, but since this is not carried out consistently - as far as I can tell - I am not sure, but this matters. I have the same issue with equation (7).\n- I do not understand why equation (2) is phrased more complicated than necessary, why can the simplification that is preformed in eq. (5) be applied to cancel the exponential and the logarithm?", "broader_impact_concerns": "I do not have any concerns on the ethical implications of the work that would require adding a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LzaVzaLGaf", "reviewer": "Reviewer_W6Mb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n> Novelty\n\nThe technique presented is indeed new compared to the existing ones. Also, this work inspires a more detailed study of the equilibrium properties of min-max problems.\n\n> Mathematical calculations\n\nI found no errors in the deductions made by the authors.\n\n**Weaknesses**\n\n> Intepretability\n\nThere are some parts of the work that I find difficult to interpret. This makes it noticeably more difficult to understand the contribution of the paper. For more details see **Requested Changes**.\n\n> Сleanliness of text writing\n\nI noticed some marks that affect the cleanliness of the text of the paper. For more details see **Requested Changes**.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a statistical mechanical formalism for analyzing equilibrium points of min-max problems in a high-dimensional perspective. Moreover, the authors apply the above formalism to the study of bilinear min-max problems and GANs, for which numerical experiments have also been conducted.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In this paper, I can emphasize the following shortcomings:\n\n> **Intepretabilty**\n\n$\\bullet$ I would suggest to relocate section **Related work** from the end to the beginning of the paper; to be more precise, after **Introduction**. The main reason for this is that the **Related work** section allows you to understand what existing results the authors are relying on in their paper, and in my opinion, it should be before the main results.\n\n$\\bullet$ The authors mentioned that the presented formalism is investigated under high-dimensional consideration. It may be not quite clear to the reader why the formalism is given exactly for high-dimensional problems. It may be worth adding a comment on this matter in the text of the paper.\n\n$\\bullet$ As for me, it is not clear how replica trick allows to move from Eq. (5) to Eq. (6). As I know, the replica trick consists in the following expression: \n$$\\ln Z = \\lim_{n \\to 0}\\frac{Z^n - 1}{n}.$$\nCould you explain this transition, please?\n\n> **Сleanliness of text writing**\n\n$\\bullet$ Comma instead of dot on page 1:  \"$\\ldots$loss function (Szegedy et al., 2013; Goodfellow et al., 2014b; Papernot et al., 2016; Madry et al., 2017)$\\textcolor{red}{,}$\"\n\n$\\bullet$ Missing comma in formula on page 2.\n\n$\\bullet$ Superfluous numbering of formulas: I have not found any references on Eq. (12) and Eq. (24). There may be other formulas with redundant enumeration.\n\n$\\bullet$ There are different designations for definitions. For example, sometimes the authors use $\\stackrel{\\triangle}{=}$, and sometimes just $=$, for example in Eq. (12). It would be better to unify the style of definitions.\n\n$\\bullet$ Extra dot on page 4: \"The detailed proof of this theorem is provided in Appendix$\\textcolor{red}{.}$ A.\"\n\n$\\bullet$ Missing comma on page 4 in Theorem 3.1. \n\n$\\bullet$ Overflow of formulas on pages 14-16.\n\nThere may be other shortcomings, so I'd suggest to authors to double-check the entire paper. \n\nIt is worth noting that the **Interpretability** section is more important in terms of understanding the work, while the **Cleanliness of text writing** is a review for minor blunders.", "broader_impact_concerns": "I do not have any concerns on the ethical implications of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rt5jRvCYPa", "reviewer": "Reviewer_rYy9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper introduces a novel way of looking at min-max problems by framing them within a statistical mechanics formalism. The use of a two-temperature system to model the interaction between min-max operations is interesting. The paper’s formalism opens the door for broader applications beyond bilinear games and simple GANs. he paper’s formalism opens the door for broader applications beyond bilinear games and simple GANs.\n- The mathematical analysis is rigorous and well presented. \n- The analysis of GAN generalization performance is a particularly strong aspect of the paper. The authors provide a clear theoretical basis for understanding how the amount of training data and the ratio of fake to real data impact GAN learning.\n\nWeaknesses:\n- The dense mathematical exposition, combined with the lack of intuitive explanations or visualizations might make the results less accessible to readers from applied fields.\n- While the mathematical findings are solid, the paper would greatly benefit from empirical experiments—especially in the context of GANs. Testing the predicted optimal fake-to-real data ratio and evaluating the framework’s performance on real-world problems would significantly strengthen its impact.\n- The GAN model used is quite simplified, and applying the formalism to more realistic GAN architectures would make the results more relevant to current research trends. This would be helpful to showcase the full potential of the framework in real-world machine learning applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel statistical mechanics formalism for analyzing equilibrium values in high-dimensional min-max optimization problems, particularly addressing non-convex scenarios where the order of operations between min and max cannot be interchanged. This framework is applied to two specific problems. First, the authors provide a sensitivity analysis on bilinear min-max games to validate the model. Second, they analyze the relationship between training data and generalization error in GANs, and deriving the optimal ratio of fake to real data for training efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Adding experimental validation, particularly in the context of GANs, would be important to make a convincing case for the applicability of the framework. One example could be to test the predicted optimal fake-to-real data ratio in a GAN for a real world task, and to show how the theoretical results translate to real-world outcomes.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers acknowledged that the paper provides a novel technique for analyzing min-max problems using statistical mechanical formalism. There is also a consensus among the reviewers that the paper should be accepted. The authors also improved the readability during the discussion and
Decision: Accept (poster)