High-probability complexity bounds for stochastic non-convex minimax optimization
Authors: Yassine Laguel
Yasa Syed
Necdet Aybat
Mert Gurbuzbalaban
Venue: NeurIPS 2024 poster (2024)
Keywords: nonconvex minimax optimization
high-probability guarantees
stochastic gradient descent ascent methods
Primary area: optimization
TL;DR: We provide the first high-probability complexity guarantees for nonconvex/PL minimax problems that satisfy the PL-condition in the dual variable.

Abstract: Stochastic smooth nonconvex minimax problems are prevalent in machine learning, e.g., GAN training, fair classification, and distributionally robust learning. Stochastic gradient descent ascent (GDA)-type methods are popular in practice due to their simplicity and single-loop nature. However, there is a significant gap between the theory and practice regarding high-probability complexity guarantees for these methods on stochastic nonconvex minimax problems. Existing high-probability bounds for GDA-type single-loop methods only apply to convex/concave minimax problems and to particular non-monotone variational inequality problems under some restrictive assumptions. In this work, we address this gap by providing the first high-probability complexity guarantees for nonconvex/PL minimax problems corresponding to a smooth function that satisfies the PL-condition in the dual variable. Specifically, we show that when the stochastic gradients are light-tailed, the smoothed alternating GDA method can compute an $\varepsilon$-stationary point within $\mathcal{O}(\frac{\ell \kappa^2 \delta^2}{\varepsilon^4} + \frac{\kappa}{\varepsilon^2}(\ell+\delta^2\log({1}/{\bar{q}})))$ stochastic gradient calls with probability at least $1-\bar{q}$ for any $\bar{q}\in(0,1)$, where $\mu$ is the PL constant, $\ell$ is the Lipschitz constant of the gradient, $\kappa=\ell/\mu$ is the condition number, and $\delta^2$ denotes a bound on the variance of stochastic gradients. We also present numerical results on a nonconvex/PL problem with synthetic data and on distributionally robust optimization problems with real data, illustrating our theoretical findings.

High-probability complexity guarantees for nonconvex minimax problems
Yassine Laguel*
Yasa Syed Department of Statistics Rutgers University
Laboratoire Jean Alexandre Dieudonné Université Côte Azur Nice, France
Piscataway, New Jersey, USA
yassine waageell@ui--coteeeeeeeee
yasa. wyeedrutgere......oo.. edu
Needet Serhat bybat
Mert Gürbüzbalaban
Department of Industrial Engineering
Rutgers Business School
Penn State University University Park, PA, USA
Rutgers University
Piscataway, New Jersey, USA mg1366erutgers. edu
Saa100psu.edu
Abstract
Stochastic smooth nonconvex minimax problems are prevalent in machine learning, e.g., GAN training, fair classification, and distributionally robust learning. Stochas- tic gradient descent ascent (GDA)-type methods are popular in practice due to their simplicity and single-loop nature. However, there is significant gap between the theory and practice regarding high-probaa probability complexity guarantees for these methods on stochastic nonconvex minimax problems. Existing phgo-bbbbybilityy bounds for GDA-type single-loop methods only apply to convex/concave minimax problems and to particular non-monotone variational inequality problems under some restrictive assumptions. In this work, we address this gap by providing the first sighbprrbbbiliity complexity guarantees for nonconvex/PL. minimax problems corresponding to smooth function that satisfies the PL-condition in the dual variable. Specifically, we show that when stochastic gradients are light-tailed, the smoothed alternating GDA method can compute e-stationary point within 0 5 (e + 8² log(1/q))) stochastic gradient calls with probability least 1 - for any ạ E (0, 1), where μ is the PL constant, is the Lipschitz constant of the gradient, и. = l/μ is the condition number, and 8² denotes a bound on the variance of stochastic gradients. We also present numerical results on a nonconvex/PL problem with synthetic data and on distributionally robust optimization problems with real data, illustrating our theoretical findings.
1 Introduction
Minimax optimization problems arise frequently in machine learning (ML) applications; indeed, constrained optimization problems such deep learning with model constraints [24], dictionary learning [10, 57] or matrix completion [28] can recast as a minimax optimization problem through Lagrangian duality. Other applications include but are not limited to the training of GANs [59], fair learning [72], supervised learning [61, 49, 51, 74], adversarial deep learning [75], game theory [54, 58], robust optimization [4, 3]. distributionally robust learning [46, 75, 24], meta-learning [70]
and multi-agent reinforcement learning [15]. Many of these applications can be reformulated in
"Coresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the following minimax form:
min max f(z,y), y), rER VER
where f R x Rd -> R is smooth function, i.e. differentiable with Lipschitz gradient; f can possibly be nonconvex in x and nonconcave in y. First-order primal-dual (FOPD) methods have been the leading computational approach for computing low-to-medium medium-accuracyy stationary points for these problems because of their cheap iterations and mild dependence of their overall complexities on the problem dimension and data size [9, 8, 36]. In the context of FOPD methods, there are two key settings for (1):
(i) the deterministic setting, where the partial gradients Vzf and ygff are exactly available, (ii) the stochastic setting, where we have only access to (inexact) stochastic estimates of the partial gradients, in which case the problem in (1) is called a stochastic minimax problem.
can be argued that the stochastic setting is more relevant to modern machine learning applications where gradients are typically estimated randomly from mini-batches of data, or sometimes intentionally perturbed with random noise to ensure data privacy [14, 34, 1J.
Convex and nonconvex minimax optimization. In the convex case (when is convex² in and concave in y), several approaches have been considered including Variational Inequalities (VIs) and primal-dual algorithms, see. e.g. [29, 20, 5, 67, 12, 73, 60, 11] and the references therein. One disadvantage of using the VI approach for solving minimax problems (by identifying the signed gradient map Gzz, y) a [ y) T - v fa. y) T as the corresponding operator in the VI) is that one needs to set the primal and dual stepsize to be the same. This can be restrictive in applications where f exhibits different smoothness properties in the primal (x) and dual (ÿ) block coordinates -this is often the case in distributionally robust learning [73], adversarial learning [43] and in the Lagrangian reformulations of constrained optimization problems that involve many constraints [47]. The gap function (I M&e) - supz.9 (Zxx. y) - f (Iz yu), and the squared distance to the set of saddle points D( yk) A minfllax x=1 1|2 + 1lykk v" 1|2 v=" is saddle point) are standard metrics for assessing the quality of the output 2y = (Ix. Vk. generated by an FOPD algorithm after iterations among many others [35, 73, 20].
In the nonconvex setting, i.c., when f is nonconvex in æ. the aim is to compute a stationary point. Let M(*K, yı)) denote measure for the stationarity of iterates (Zt, 81. J; a common metric is the norm of the gradient, i.e., M(KR, M&) A [ [IT: Ms)|l and its variants such as [[ )I when f is strongly concave in y, where (.) = maxy, l'(-2) 2) denotes the primal function -for other metrics and relation between them, see [62]. There are several algorithms that admit (gradient) complexity guarantees for computing a stationary point of nonconvex minimax problems under various strong concavity, concavity or weak concavity-type assumptions in the y variable -see the references in [62].
In this paper, we consider smooth nonconvex-PL (NCPL) problems where f is smooth function such that it is possibly nonconvex in x and it satisfies the Polyak-Lojasiewicz (PL. condition in y. The PL condition is weaker assumption (milder condition) than strong concavity in y -in fact, PL condition in the dual does not even require quasi-concavity. NCPL problems constitute a rich class of problems arising in many ML applications including but not limited to fair classification [48], robust neural network training with dual regularization [48, eqn. (14)), overparametrized systems and neural networks [42], linear quadratic regulators [17], smoothed Lasso problems [23] subject to constraints, distributionally robust learning with lg regularization in the dual [72], deep AUC maximization [68] and covariance matrix learning with Wasserstein GANs [52]. For deterministic NCPL problems, the alternating gradient descent ascent (AGDA) method and its smoothed version (smoothed AGDA) have the complexity of O(x² /e2) and (kx/²²), respectively, for finding a point (z. y) satisfying | f 7) I e as shown in [70, 66]. Here k e/p is the condition number, where é is the Lipschitz constant of the gradient, and p is the PL constant. For Catalyst-AGDA, [66] shows also the rate (ms/²)) for deterministic NCPL. problems.
In expectation and high-probability bounds. Most of the existing guarantees in the literature for stochastic FOPD algorithms are provided in expectation, i.c., a bound on the number of iterations (or the stochastic gradient evaluations) is provided for (ZG 3k) S E or ([ yk)] S & to hold (see, e.g., [73, 72, 29] and the references therein). Yet having such guarantees on average does
3f Rd + R U ( (+0o) called (merely) convex. / (Iz) (1 ()22) S if (a.) + (1 1)f (az) for every æ1, .2 Rd and t [0, 1] with the convention that +00 for all a E I.

---

not allow to control tail events, i.e., even if [[G Bee) is small, ([2,, 3k) can still be arbitrarily large with non-zero probability. To this end, probabbbbyilltty guarantees have been considered in the literature [35, 64, 20, 32, 22]. These results allow to control the risk associated with the worst-case tail events as they specify how many iterations would be sufficient to ensure (a ys) is sufficiently small for any given failure probability q E (0, 1). To derive high-pron nigh-probabilityy bounds, one common approach involves running the algorithm in parallel multiple times and strategically selecting an optimal output to convert in-expectation bounds into high-probability guarantees [62, 37]. Alternatively, advanced concentration inequalities can be employed under light-tail assumptions to control noise accumulation across iterates without requiring multiple runs [53, 27]. For saddle point problems. we note that existing high-probability bounds mostly apply to the monotone VI setting, to strongly convex/strongly concave (SCSC) minimax problems. To our knowledge, guarantees for nonconvex minimax problems are non-existent in the literature, even for nonconvev/troongly concave (NCSC) problems with the exception of trivial loose bounds one can obtain by a standard application of Markov's inequality (see Remark 13 for details). In particular, we note that the existing VI literature with high-probability bounds non-monotone operators such as star-co-coereive operators [20, 55), do not apply to NCSC problems."
New probabbbbbbiitty bounds for NCPL optimization. To address these shortcomings, we focus on developing high-pro ceetbbbbbbbblitty guarantees for NCPL problems. Among the existing algorithms in the stochastic NCPL setting [64], stochastic gradient descent ascent (SGDA) methods and their variants are quite popular for ML applications, e.g., training GANs and adversarial learning, as SGDA is easy to implement due to its single-loop structure. Guarantees in expectation for stochastic NCSC problems are well supported by the literature- see [72, 63, 40, 6, 31, 30, 44, 66, 33, 39] and the references therein. To our knowledge, among single-loop methods for NCPL problems, the best guarantees in expectation are given by the smoothed alternating gradient descent ascent (Sm-AGDA) method [66], which can compute an almost stationary point (Ø, f) satisfying E|ll f 7)ll S E in O(En / + Ex/=²) stochastic gradient calls, where 8² is upper bound on the variance of the stochastic gradients. In this work, we consider the sm AGDAADA algorithm, and to our knowledge we provide the first-time high-probabiliyy bounds (using single-loop method that does not resort to restarts and parallel runs) for the minimax problem (1) in the NCSC and NCPL settings. More precisely, we focus purely stochastic regime in which data streams over time which renders the use of mini-bater schemes or running the method in parallel impractical; therefore, approaches based on Markov's inequality [62] are no longer applicable (see also Remark 13).
Contributions. Our contributions are threefold:
We present the first high-probubility complexity result for the am- Am-AGDA algorithm in the NCPL setting by building upon a Lyapunov function first introduced in [70] for nonconvex-conce problems. Later, for the same Lyapunov function, state-of- thr--rrttt--tttt complexity bounds in expectation are provided for the NCPL setting in [66]. In this paper, we derive a novel descent property for this Lyapunov function in the almost sure sense (Theorem 7 and Corollary 8), allowing us to develop useful concentration arguments for it to derive high-probability bounds. Our Lyapunov analysis not only sheds light on the convergence properties of sm-AGDA, but also guides the parameter selection for Sm-ACDA. Specifically, we show that Sm-A can compute an almost stationary point (Ø. ÿ) satisfying f 0)1l S with probability 1 q E (0, 1) within
1x2882 + - (e 52 log(1/g)) stochastic gradient calls. The lower complexity bound Te.g -0 - of 2(2 + 1)) for NCSC problems [38, 71] in expectation (see also [63, 72]) suggests that our high-rrobbillity bound for sm- AG-AGDA is tight in terms of its dependence on E. Furthermore, to our knowledge, these are the first high-probability guarantees for any algorithm in the NCPL setting.
Under light-tail (sub-Gaussian) assumption on the gradient noise (Assumption 3), which is common in the literature [35, 32, 19], we develop a new concentration result (Theorem 9) that can be of independent interest. From this concentration inequality, we observe that the cost of strengthening the existing complexity result in expectation to high-probability one is relatively low, i.e., in the final complexity, the probability parameter q only appears in an additive term that scales with 2 Consequently, this represents non mon-dominant overhead compared to the e term already present in state-of -f------thheeeee the-art expectation bounds [66].
'For example, when f smooth (Lipschitz and continuously differentiable) and signed gradient map G(æ,y) y) is star cocoerciee with constant E >> 0 around a stationary point (æ., y-). then by [21, Lemma C.6], the operator ld - GGx y) is non expansive around (I., y=); thus, the Jacobian of G at (I., y-) has non negaavvee eigenvalues implying f is merely convex/merely concave around (I., y-). ). ie., I cannot be NCSC.
3

---

Algorithm
Complexity
Problem
Metric
NC.
Epoch-GDA (64]*
2 log(1/G)
SCSC
9(5x)
Clipped-SGDA [20]
max
62
log (2) log (x/(0)
SCSC
D(ax)
X
Clipped-SEG [20]*
① mux
log ()) log (15/s)
SCSC
D(aa)
Stochastic APD [35]
0 clos
(1+llgg') (ooloo(////1000 (/1))/ top(l/le)
SCSC
D(sa)
Miror Prox [32]"
0
max
CD2
قوا log (1/43
MCMC
((3a)
x
Clipped-SGDA 120
[max{]
2= log (1/4)
MCMC
(n(()))
Clippod-SEG (20)
max
cm
log (1/4)
MCMC
Gn(5x)
Sn-AGDA [Our Paper, Coro. 14)1
(e+ log(1/9))
NCPL
8+1
Table I: Summary the high-prohhiility hounds minimax problem classes when the gradient f is Lipschitz (with parameter and
stochastic gradient variance is bounded by 52, The second column reports the complexity (number of calls stochastic gradient oracle)
required to achieve the (stationarity) metric reported in the fourth column to be at most with prohahility 1 q E (0, 1); (()) igmores some legarithmic terms, Here, (t, is the strong convexity constant, is the comstaatn and n. f/j. Let (=) [V,)(=) - v()))"] "
with (z, 9) and Ag. RFI Σ 33. G[x) & max (EC (G(=), E =4), where Z the domain of the problem with diameter
E (0, +ool, and (R(Z) _ max 2:l|x-z. is my (G(+), 5 sa) where 2. = (a. .v. ) is stationary point. The third column reports the minimux problem class. The lifth column indicales whether the results supports nonconvexity, ie., whether f can be smooth function nonconvex in [64] is two-loop methhod. Applicable quasi-strongly monotone G that is star-co-coercive around s. and supports heavy- miled gradients. Applicable quaci-strongly momotone G and supports heavy-tailed gradients. P Supports proximal steps handle non smooth ceavex penalty." Applies to menotone G that stur-co-coerivve around =.. Makes light-tail assumption (Ass. 3).
Third, we provide experiments that illustrate our theoretical results. We first provide an example of an NCPL-game with synthetic data and then focus on distributionally robust optimization problems with real data, illustrating the performance of the sm- AGDA in terms of high-probability guarantees.
2 Preliminaries and Technical Background
Stationarity metric. We consider the minimax problem in (1) for f : Rd x Rdd + such that f is smooth (Assumption 1) and f(z, -) satisfies the PL property for all x E Rdh (Assumption 2); moreover, we also assume that we only have access to unbiased stochastic estimates of f such that the stochastic error G(æz, y. {) ff y) has a light tail (Assumption 3) for any (z, v). where G(x, y, {) denote the stochastic estimate of f y) and denotes the randomness in the estimator.
Our aim is to compute a (Ear, Ey) ( -stationnary point (ž, 5) for (1) such that [ 7) S Ez and V (ā, r)ll I Ey. We also call (œ., 5) e-stationary point if [V, 5) S e. Clearly, whenever (ā, 77 is sy --ttttioonnn then it is also e-stationary for E (=) + 2/11/2
Smoothed alternating gradient descent ascent (em-AGDA:: The method can be considered as an inexact proximal point method and was introduced in [70]. More specifically, in each iteration of sm-AGDA, given a proximal center 2 and the current iterate (æe, yt), the method computes the next using a stochastic gradient descent ascent step on regularized function f:
iterate (I++1, Mt+1)
f(z 21) A f(x, v) +
P - 20l1
(2)
Following the stochastic alternating gradient descent ascent (stochastic AGDA) steps, the proximal center at iteration f, i.e., 2t. is updated as shown in Algorithm where Gr Vt. (f41) and Gy(x144 Me, 1)+1) denote conditionally unbiased stochastic estimators of the gradients [zf(z ve) and Vy (Z 1t). Throughout the analysis we assume that V is Lipschitz, which is standard in the study first-order optimization algorithms for smooth minimax problems; see, e.g., [72, 73, 75, 63].
Assumption 1. (Lipschitz gradient) For all (x1, (2) E ddi x Rdd, there exists E >>> 0
[ 21) V. (22, 22)l < [[|== x2|l + || y2ll)
/ M1) v v2 < [[|[=] - x2ll + Ily1 - v2ll).
(4)
The following condition, known as Polyak-t.ojcciewiz (PL.) condition is weaker than assuming strong concavity in y. and does not even necessitate ff to be even quasi-concave in the variable. It holds in many ML applications including those in [48, 48, 42, 17, 23, 72, 68, 52, 66].
Assumption 2. (PL condition in y) For every x E Rd,, max,ER22 fxx,)) y) has a non-empty solution set and finite optimal value. Moreover, there exists μ O such that:
(x, 2 2n[ max f(x, v) - fxx v)l. yERd2
Vx TeeR' Rd1
(5)

---

We assume that we have only access to stochastic estimates (2t, Ut, 17+1) and Gy (1 Mr, 16+1) 1) of the partial gra- dients fI Ms) and fzf(x We), where S141 and S141 are random vari- ables defined
Algorithm 1 sm-AGDA
Input: (Io, yo: 20), TI, 12 > 0, 0, [0, 11, 0
for t = 0, 2..,2.....3
do
T++1
Ti Gn Wt: St:1) p(at =t)
probability space (2, P), i.e., the source of randomness in the gradient estimates. Note that Sm-AGDA has Gauss-Seidel updates, i.c., the stochastic estimate of the partial gra- dient Gy(Ze+1, (x.5++1) 1 is evaluated at the updated point (Ie+1, Br) instead of (æ,, Vr). To capture the sequential information flow, we next introduce the natural filtrations that represent all the information available before an update: Let & and &/ be revealed sequentially in the natural order of the sm- AGDA updates, i.c., çi + < + E + Eg + E3
Mt+1 Yt + T2 (1+11, Ut. St
2r+1 21 + 3(x++1 zz)
end for
Output: choose (i; () uniformly from { yt Himo
and let (FF) and (F()) denote
the associated filtration", i.e., let Fd A (0, and
= (Ei+1))
F441 oFH V1Z0. 0.
(6)
Introducing multiple filtrations to represent the sequential information flow is common the study of stochastic algorithms with Gauss-Seidel updates -see, e.g., papers stochastic ADMM, and [7, 69]; and we follow the same approach. Consider the gradient noise (errors) at time N:
A Gz( ye 1 Vzf(zt, vo), A Gy(Ee+1 (t (t+
Vyf(Ee+1, yr).
Finally, we also assume that the gradient noise is unbiased conditionally the past information and that it admits a light Sub--Guussian tail.
Assumption 3. (Light tail) For any t 2 O, there exists scalars 8g, 6g > O such that
A
F O,
PIATT 2 8
2e 25
2
A
O,
PP [ 2
F i+1
2e 225
(8)
For developing high-probability bounds in the learning context, is common to assume that gradient estimates are sub- Sob-Guussian [32, 35, 18]. While this assumption may not always hold (see e.g. [25, 56]), it often holds when gradients are estimated via mini-batching, as a consequence of the central limit theorem. It will also hold when the gradient noise is bounded. Additionally, adoption of differential privacy mechanisms within gradient-based schemes [14, 34, 1], to enhance data privacy, results frequently in sub- Gaussian gradient errors.
3 High-probability bounds for sm sm-AGDA
For analyzing sm AGDA, similar to [66, 70], we consider the following Lyapunov function:
V A Vi Me: =x) = (a yri 21) ) + 2xx))) 2 #x);
where P(2) and ((;;)) denote the saddle point value and the dual function value, respectively, of the auxiliary problem ming max, fx. 2) for any fixed z and f defined in (2), i.e..
V Vyy 2) A min f(r y; and P(z) A min max f(x, v) zER
(10)
ZER4 yER
Next, we introduce natural assumption, commonly made in the literature [66, 65). Without this assumption, there are pathological cases where primal function ((x)) may be unbounded leading to divergence of gradient-based methods; example would be f(x -x² - y² in dimension one.
Assumption 4. Consider the primal function & Rdd + RR i.e., ((x) max f(z, y). There exists x e d such that x 2 +xx*)) min (xx).
'Given a random variable ç, (5) denotes the a alagebra generated by E: moreover, given two alalgebras, Vn and Σe, abusing the notation, a (Ei, 2n) denotes the o-aggebra generated En U E2.
5

---

Under Assumption 4, it immediately follows that Vi 2 d for all t E N -since P(z) V 2) 2 0, f(z, yi 2)= - v (y;z) 2 0 and P(z) 2 Φ* for all v, We will next study the change V2 Vi+1 in the Lyapunov function and show that an approximate descent property holds. First, we need two key lemmas that characterize the evolution of (at, yt; z)) and (iy =e) over the iterations.
Lemma 5. Suppose Assumptions 1, 2, 3 and 4 hold. Consider sm-AGDA given in Alg. I with E (0, pعبر and B E (0, 1]. For any ( E N, we have:
((1411111112441 - [(,,, (i 2x) S
V Me + (1+ 2T) 11 [7
2
+ tn((p + f)n 1) V. =)) + p+ E
2
+ T2 + Fm) (v A/) 2B
# 11 ||
||
Proof. The proof is provided in Appendix B.I.
From Assumption 1, when p > L, the auxilliary function f(- y; 2) is (p - E) stronnlyy convex for any fixed y, 2; hence, there is a unique minimizer for every y, fixed, denoted by
(y, 2) - argmin (1,,112)
(11)
Le., 2v, f(z" (y. 2). 2). In the rest of the paper, we will take p > E and exploit this property, The following lemma characterizes the change in the dual function V.
Lemma 6. Suppose Assumptions 1, 2, 3 and 4 hold. Consider the sm-AGDA iterate sequence (1e, Dt, heN for p > f. For any L E N, it holds that
V (yc-11 2(11)
(yre 2x) 2 T2 v (x" (ye, 21). off v()) + T2 (V of(x" (16, 21), Ue), A
L 2
2 22 (I of 4e) )= 2(V v. ( M6), A + 1|2?11
p
2 (z+1 24, 11+1 + 2 2x (y++1, 10))
where Lv A el ((1+
+ and the map x² (s,-) is defined by (11).
Proof. The proof is provided in Appendix B.2.
The next result provides an approximate descent property on the Lyapunov function. Its proof builds on Lemmas 5 and 6 and descent property on the function P (given in Lemma 15 of the Appendix); and leverages smoothness properties of the functions of and Vo and the map (u. 2) + x² (v. 2) as well as the strong convexity of f with respect to x.
Theorem 7. Suppose Assumptions 1, 2, 3 and 4 hold. Consider the sm-AGDA algorithm with purameters p E, E (0, 1), TI E (0, p+tl and T2 >> chosen such that
L T 20, d 2 P
2p²
nd 48, (p- - --)2
Co 2 -2 + T2
2² 1-
2 0,
38
p-
for some constant v > 0, where L = (1 (1+ 11+210 p+v
p- Then,
- Ve-V++ 201l VillV...( i/ 2e )1( + 02l y fx (ye, 21) w:)l ca callxr - zel12
+ ca V V Wei (x), Ai) + (es V y m.) + ca V (2"(/x="")) 2x), we). At) (12)
+ A 12 + ?
for some constants {c C R. that are explicitly given in Appendix C, which may depend on v, as
well as the problem and sm- A CDA parameters that can be chosen such that c1, C2, Cs > 0.
Proof. The proof is given in Appendix C.
With some specific choice of parameters in sm-AGDA, we can obtain simplifications to the coefficients {o 8 from Theorem 7 (explicitly given in Appendix C). As such, this yiclds the following corollary.

---

Corollary 8. Under the premise of Theorem 7, let p- = 2f, 1I E (0, 3el 72 = 18. = aut2 for
E (0, Juc) Then, A I -B, + C++1 + D all 1 E N. where
12 and
A: _ ViVi,
B, _ -
71
T2 v2 (ye, 21), vill + Sp
1| 21
r.t
5
8
8
C441 A [1923pp p E
48
+ 4enl + 2c(3²) गं + ( ( 1) [aI 2), Ai)
+
p-
V
+r2((1 + LT2 + 2 (af( Ui) - of (ve, 2). 31).
Det1 A 277 + 8673 3 A!I?
Proof. The proof is given in Appendix D.
Next, we provide concentration inequality which will be key obtain high-probability bounds.
Theorem 9. Let {Fe be a filtration on (R,,,,)) Let At, Br, Ct. Dr be four stochastic processes
adapted to the filtration such that there exist ac, a > and Ti > 0 such that for all E N: (i)
B₁ 0, (ii) E[e2C(+1
Fe] S BB for all X > 0, (iti) BleADe+1 | Fil S e for all
E
0, and (iv)
Act1-Ae S BB, C-1 + D++1. Then, for any q E (0,1), we have
Bi $ (Au - Ar) + hooo + 2n max (20 b log (2) 2 1 1
P
Proof. The proof is provided in Appendix E.
Remark 10. While the above concentration inequality seems tailored to the analysis of sm-AGDA can also aid in deriving high probability bounds for many other first-order methods for nonconvex minimax problems that outputs a randomized iterate; indeed, the majority of existing Lyapunov argu- ments in the nonconvex setting are built upon constructing telescoping sums in line with Theorem 9, e.g., stochastic alternating GDA [66] for NCPL minimax problems and optimistic GDA [45] for strongly convex-strongly concave problems.
We next present our main result which provides potbbbbrbbbbitty bound on the sm- AGDA iterates. The main idea of the proof is to apply Theorem 9 to the processes introduced in Corollary 8
Theorem 11. In the premise of Corollary 8, sm-AGDA iterates (Iz, y1) for TI S tे satisfy
T-1
ΣΣ
[V. w:)l + x|| V.l wc) S Qq.r. 2 21-4. q, VTEN, (0,1)
T
for some 24.r O m((otbb) nO0² + 67) me log )1) explicitly stated in Appendix F, where Ao 2 ((2)) - -0, bo 2sup [ vi 20) - f(z, ((Ix,000000 3. Ct, Proof Sketch. Let the stochastic processes Ar, Ct, D, in Theorem 9 be chosen as A, = d
n.
B, = B,, C. = C. D. = D, where A., B,, Cr D; are defined in Corollary 8 and T. > 0 be the primal stepsize in sm-AGDA; according to Corollary 8, we have 11+1-4, < BB + C++1 + D for & E N. Since Aj and A admit sub-Gaussian tails, it can be shown that the conditions of Theorem 9 are satisfied for some appropriate constants of and o.. Therefore, Theorem 9 implies a tail bound on Et= B. Using the relation between f and f. one can also show that / m/) )= (iB),, for all E This last inequality allows to translate the tail bound for =t= B, to a tail bound for ΣT- [ 2t)l + x|| k/|V V yf 201l. The details of the proof is
71
k|| v ye 222 = =
T-1
provided in Appendix F of the supplementary material.
Remark 12. Suppose 3m-AGDA, given in Alg. 1, is run for T' iterations, and it outputs a randomly selected iterate (IV, YU), where the random iteration index U is chosen uniformly at random from Theorem 11 implies that
the set (0, 1, ,,...,, 13, Le., IP(U =) 1/T for i 0, 0,.....,
P [Vaa )l? + V
No wo)1 S Qq.r 2 21-
Furthermore, in comparison with existing complexity bounds in expectation for sm- AGDA [66], our quantile bound requires only an overhead of order O( log(1 /q)). Unless q is very small, this is typically negligible in comparison the Oc=-1) already present in rates in expectation.
7

---

Remark 13. In contrast to probbbbbyyyylity bounds derived from standard Markov-type arguments, our approach achieves significantly better scaling with respect to both q and e. Specifically, consider an oracle that can generate a sample (5. i) with FA[1 S (#. 1) II] S after G(()) iterations/stochusiic samples. In particular, [66] shows that one can take (()) = O Rx.2 =
=
for the sm-AGDA
algorithm assuming the variance of the stochastic gradient is bounded 8 A naive higb-proba bound could be constructed by ensuring Eill V f(ē. 9) S q E and applying Markov's 's inequality to yield an e-stationary point with probability at least 1 - q. However, this approach results in
+ q'= q²(² ) leuding to a significuntly worse dependence on q than ours. Alternativeyy, following the rationale in [19, 62], generate can run the sm- AGDA algorithm m = (2(1o((1/)) times in parallel; where in each run we generate an e/2-solution and among the solutions, we select the one with the smallest estimated gradient norm. This would require mG(()= = (log(1/0) 2x262 c
complexity bound of ((E)) = O
6x266
mf
q'c'
probabiibyy bound, one
iterations/stochasiic samples. log((1/)) uE
In this approach, the logarithmic term log ()) multiplies the high-order ((22) term, whereas in our approach it only affects the second-order (++) term. Therefore, our results scale better with respect to q and s. In addition, such (multiple) parallel run approach, is often impractical in streaming/online settings, where data arrives sequentially, and real-time processing is essential.
Corollary 14. Under the premise of Theorem 11, consider running the sm-AGDA method for some fixed number of iterations T E N with parameters chosen as Ti min
/
32
VTE
and T2 -1/48 where 8² & 82 + bg. Then, for any q E (0, 1), Sm-AGDA can compute an (E,E/V stationarity point with probability at least - q when the number of iterations T is fixed to ton g²
8 log (3)) d' to
Te.4 = 0
which requires Te.q stochastic gradient calls.
Proof. This is direct consequence of Theorem 11, proof is provided in Appendix G.
4 Numerical Illustrations
In this section, illustrate the performance of AGDA We consider NCPL problem with synthetic data, as well as a nonconvex DRO problem using real datasets. For synthetic experiments, we used an ASUS Laptop model Q540VJ with 13th Generation Intel Core 19-13900H using 16GB RAM and ITB SSD hard drive. For the DRO experiments, we used a high-performance computing cluster with automatic GPU selection (NVIDIA RTX 3050, RTX 3090, A A100, or Tesla P100) based on GPU availability, ensuring optimal use of computational resources.
Synthetic experiments on an NCPL game. We consider the following NCPL problem:
mín max mi ||0|| + sin (3//2=// + +]] + x Ky- m2 [|lv| 2 + ((yn)] TER
(13)
which can be interpreted as a game between two players [48, 35] where m1, m2 > are constants and the symmetric matrix K is set randomly, similar to the standard bilinear game setting considered in set K 10K KI K (M + M )/2 where M is d d matrix with
[35]. More specifically, entries being i.i.d centered Gaussian having variance 0². This problem is nonconvex in æ (without satisfying the PL condition in æ²). Though the exact gradient is known, we consider stochastic gradient oracle, which returns noisy gradients similar to the setting of [35, 11, 16, 2], i.e., for each iteration 1 (0...., 1}. Gz (Ire; yei 5 = VaI( v.) + Et+1 and (3 6 (b+1)
(yf( V:) + CM 5t+1. with 1120 iid 0(0,62la) ) and (424 sid (N La,) where Ia is the d x d identity matrix and 82 some constant variance. This setting satisfies all our assumptions, and our high- ceobbbbbbbbblity results (Theorem 11 and Coro. 14) are applicable. In this experiment, we fix
di = d2 = 30, and mi = m2 = 82 = o² = 1. The solution to this problem is (x*, , )²*) = (0, 0).
Experimental results. The parameters of the problem are explicitly available as = 2m/1 and E max{12m, 8mg. ||]]] To illustrate Theorem 11, we set 3
1600 T2 3 28
Tap
and we considered two cases: Ti = उे (long step) and 7 = I (short step) to explore the behavior of sm-AGDA for different stepsizes. We generated N 25 sample paths for T = 10, 000
Iae
iterations, and on the left panel of Fig. 1, for each iteration i, we report the average of Mz) ع over N = 25 realizations corresponding to different sample paths, and the shaded region depicts the range statistic, i.c., for every fixed iteration t, we shade the
[ )12 + x|| (x,(
8

---

Average path across gradients
Average path across iterates
log 17//,7 yoll
t: -1/(34)
T. 1/(31)
1/(12))
1/(120)
log(1x
2.5
0.-
2000
4000
5000
8200
30002
2000 4000
0000
0200
10200
Iterations
Iterations
Figure 1: NCPL game with T. = de (long stop) and Y1 12c (short stop). (Left) Average of M. (t) over 25 sample paths vs. iterations f. (Right) Average of (t) 25 sample paths iterations 1. both plots, shaded regions depict the range statistic over 25 sample paths.
vertical line between the maximum and minimum values of MM,(() for the 25 paths. On the right panel of Fig. we also report the squared distance, I(t) & [z, x" 1|2 + 11 - y" 11 in a similar manner. We observe that the range statistic for (t) diminishes to value inversely proportional to T as t > T; this inline with our theoretical results in Theorem 11. The existence of sinusoidal terms in the minimax objective is a source for oscillatory behavior in these figures. As t - T, both M,,(t) and [(t) exhibit oscillations, of which amplitude are smaller for the small step size compared to the large step-size -at the expense of slower convergence; the iterate paths are not as oscillatory.
For T = 10, 000 and Ti we next consider the path averages XT 2 Mu((). Indeed, based on 1000 sample paths, each for 7 10, 000 it- erations, we compare the empirical quantiles of XT with the theoretical upper bound 2g.1 on its quantiles (implied by Theorem 11). Figure 2 shows the cumulative distribution function (CDF) of the empirical distribution alongside the theoretical explicit upper bound 2y.r for the stationarity measure [V yv) + xV,, vVuf( yc')l with U
32 fixed,
1.0
Theorctical CDF Empriial
3.6
0.2
0.0
78.8
Figure 2: Comparison of our thcoretical upper bound in Theorem 11 and the empirical stationarity measure an for Problem (1.3). Results are reported cumulative distribution functions.
uniform over {I....,T) T). The details of the empirical quantile estimation are provided Appendix I due to space considerations. We observe that the empirical CDF has a sigmoid-like shape, while the theoretical quantiles that lie above display concave form. This difference may arise because the theoretical quantile bounds Qq.r are designed to capture the worst-case behavior across the class of NCPL problems, whereas this specific NCPL example may not represent the worst-case scenario. Distributionally Robust nonconvex Logistic Regression. We consider the DRO problem
a9a epoch=20
gisette epach=20
sido0 epoch=20
0.4
0.5
0.3
0.3
0.1
0.1
0.0
0.0
3
893 epoch=250
gisette epoch=550
sido0 epoch=250
0.8
0.5
0.8
0.4
0.6
0.4
0.2
6.2
0.0
22
2
-2
-2
SAPD+
SMDAVR
SMAGDA
Figure 3: Histograms for the stationarity measure log /IV (x,, IIll for en-AGDA and baseline algorithms (SAPD+, SMDA, and SMDAYR) on a9a, gisette, and sido0 datasets. Each algorithm is run 200 times, First row reports histograms after 20 epochs, and second reports histograms the end of training.

---

min max
(
Σ
de
yj (3;;),,))) r(x) s(v)),
(14)
#ER DEY d2 j=1
where (,(x; 0,0;;,,,00 (..,bbbb)) b,) = log + exp(-b,a)
denotes the logistic loss tied an input-output pair primal regularization for the learning model wxy
(ag, by) E Rd x x(-1,1). and r(x) = A1 Ex=
i= I-wai
E IRd We allow the distribution Y {y RR y 2 0, 1 }} to deviate from the uniform distribution u _ dg where 1 denotes the vector of ones, and penalize the distance between y and u through the regularization map g - 2 parameters as = 10, A = 10e and A2 = 1. Since r is nonconvex with Lipschitz gradient and g is strongly convex, this am NCSC problem.
Anda lly - ull?. We set the regularization
Datasets, Algorithms and Hyperparameters. We consider three standard datasets for this problem, which are summarized as follows: The s sido0 dataset [50] has dj 4932 and d2 = 12678. The gisette dataset [26] has dj 5000 and d2 6000. Finally, the a9a dataset [13] has d1 123 and d2 = 32561. We compare the performances of AGDA against two other baselines that achieve state-of-the-art performance in expectation for these datasets [72]. Specifically, we evaluate SAPD+, which a two-loop method where the subproblems are solved by the SAPD algorithm [73], and SMDAVR, variance reduced extension of SMDA algorithm [31]. Since (14) is constrained, we augment sm-AGDA with a projection step in the update of the variable onto the dz-dimensional simplex and adopt the analogous stationarity metric [ y/ )[2 + V y1) for constrained problems where Py is a projection to the dual domain Y. For all datasets, the primal stepsize 7 of sm-AGDA is tuned via grid-search over (10-*, s k < 43. The dual stepsize 72 is set 72 n Similarly, B3 is estimated through grid- over {10 ko-k,_ S S 5). The parameter p is also tuned similarly on grid, our code is provided as supplementary document for the details. For other methods, our hyperparameters are tuned in accordance with [72].
Experimental results. In Figure 2, we plot histograms of our stationarity metric, across 200 runs in logarithmic scale. We report the stationarity measure both in early phase of the training (i.c. = 20 epochs), and in later phases (i.e. é 550 epochs for gisette and 250 epochs for a9a and sido0). Our theoretical results are presented for unconstrained problems the dual, therefore they not directly applicable to the DRO problem where the dual domain is constrained. That being said, we observe that they are still predictive of performance in the DRO setting. More specifically, Figure 2 is supportive of our high-probability complexity bounds for am-AGDA, in the sense that the distribution of the stationarity metric for sm- AGDA tends to concentrate. Notably, it outperforms observe that histograms for all
the concentration behaviour of the other baselines. Furthermore, baselines hardly evolve after 20 epochs. This is consistent with previous experiments carried on these datasets [72] where performance was measured terms of the decay of the average loss and its standard deviation. As such, we conclude that sm- sm-AGDA performs better both in the early phase and the later stage. In our experience, we observed Sm-AGDA could accomodate larger stepsizes compared to the other algorithms, which may have contributed to its good performance.
Conclusion
Existing high-probability bounds only apply to convex/concave minimax problems or non-monotone variational inequality problems under restrictive assumptions to knowledge. We close this gap by providing the first high-probability complexity guarantees for nonconvex/PL minimax problems satisfying the PL-condition in the dual variable for the sm- AGDA method. We also provide numerical results for an NCPL example and for nonconvex distributionally robust logistic regression.
Acknowledgements
Yassine Laguel, Yasa Syed and Mert Gürbüzbalahan's research are supported part by the grants Office of Naval Research Award Numbers N00014-21-1-221 and N00014-24-1-262.. National Science Foundation (NSF) CCF-18 11F-1814888 814888 and NSF DMS-2053485. Necdet Serhat Aybat's work was supported in part by the Office of Naval Research Awards N00014-21-1-227 and N00014-24-1-266.
10

---

References
[I] Jason Altschuler and Kunal Talwar. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. Advances in Neural Information Processing Systems, 35:3788-3800 2022.
[2] Necdet Serhat Aybat, Alireza Fallah, Mert Gurbuzbalaban, and Asuman Ozdaglar. Robust accel- erated gradient methods for smooth strongly convex functions. SIAM Journal on Optimization, 30(1):717-751, 2020.
[3] Amir Beck and Aharon Ben-Tal. Duality in robust optimization: primal worst equals dual best. Operations Research Letters, 37(1):1-6, 2009.
[4] Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust Optimization, volume 28. Princeton University Press, 2009.
[5] Alcksandr Beznosikov, Boris Polyak, Eduard Gorbunov, Dmitry Kovalev, and Alexander Gasnikov. Smooth monotone stochastic variational inequalities and saddle point problems: A survey. European Mathematical Society Magazine, (127):15-28, 15, 2023.
[6] Radu loan Bot and Axel Böhm. Alternating proximal-gradient steps for (stochastic) nonconvex- concave minimax problems. arXiv preprint arXiv:2007. 2020.
[7] Radu Ioan Bot, Panayotis Mertikopoulos, Mathias Staudigl, and Phan Tu Vuong. Minibatch wrwward-bcckward-fooee
methods for solving stochastic variational inequalities. Stochastic Systems, 1()): (22):11222)) 12-139, 2021.
[8] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 pages 177-186. Springer, 2010.
[9] Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 1,
[10] Nicolas Boumal and Pierre-antoine Absil. Rtrme: A Riemannian trust-region method for low-rank matrix completion. Advances in Neural Information Processing Systems, 24, 2011.
[11] Bugra Can, Mert Gurbuzbalaban, and Lingjiong Zhu. Accelerated linear convergence of stochastic momentum methods in Wasserstein distances. In International Conference on Machine Learning, pages 191- PMLR, 2019.
[12] Antonin Chambolle and Thomas Pock. A first-order primal-dual algorithm for convex problems with applications imaging. Journal of mathematical imaging and vision, 40:120-145. 2011. 1.
[13] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):1-27, 2011.
[14] Rishav Chourasia, Jiayuan Ye, and Reza Shokri. Differential privacy dynamics of Langevin diffusion and noisy gradient descent. Advances in Neural Information Processing Systems, 34:14771-14781, 2021.
[15] Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in Neural Information Processing Systems, 33:5527-555 -5400, 2020.
[16] Alireza Fallah, Asuman Ozdaglar, and Sarath Pattathil. An optimal multistage stochastic gradient method for minimax problems. 2020 59th IEEE Conference on Decision and Control (CDC), pages 3573-3579. IEEE, 2020.
[17) Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. International Conference on Machine Learning, pages 1467-1476. PMLR, 2018.
[18] Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization I: A generic algorithmic framework. SIAM Journal on Optimization,
2012.

---

[19] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-236) 2013.
[20] Eduard Gorbunov, Marina Danilova, David Dobre, Pavel Dvurechenskii, Alexander Gasnikov, and Gauthier Gidel. Clipped stochastic methods for variational inequalities with heavy-tailed noise. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 11311--1332, 2022.
[21] Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: (1/kk) last- iterate convergence for monotone variational inequalities and connections with cocoercivity, In International Conference on Artificial Intelligence and Statistics, pages 366 402, 2022.
[22] Eduard Gorbunov, Abdurakhmon Sadiev, Marina Danilova, Samuel Horváth, Gauthier Gidel, Pavel Dvurechensky, Alexander Gasnikov, and Peter Richtárik. probabbbbayylity convergence for composite and distributed stochastic minimization and variational inequalities with heavy- tailed noise. arXiv preprint arXiv:2310.0188 2023.
[23] Mert Gürbützaalaaan, Yuanhan Hu, Umut Simsekli, and Lingjiong Zhu. Cyclic and randomized stepsizes invoke heavier tails in SGD than constant stepsize. Transactions on Machine Learning Research, August 2023.
[24] Mert Gürbüübalaban, Andrzej Ruszczyúski, and Landi Zhu. A stochastic subgradient method for distributionally robust non-convex and non-smooth learning. Journal of Optimization Theory and Applications, 194(3):1014-1411 2022.
[25] Mert Gürbüüzalaaaan, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in SGD. In International Conference on Machine Learning, pages 3964-3975. PMLR, 2021.
[26] Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the NIPS2003 feature selection challenge. Advances in Neural Information Processing Systems, 17, 2004.
[27] Nicholas JA Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory, pages 1579-1613. PMLR, 2019.
[28] Mark Herbster, Stephen Pasteris, and Lisa Tse. Online matrix completion with side information. Advances in Neural Information Processing Systems, 33:20402-20414, 2020.
[29] Yu-Guan Hsich, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the con- vergence of single-call stochastic extra-gradient methods. Advances in Neural Information Processing Systems, 32, 2019.
[30] Feihu Huang. Shangqian Gao, Jian Pei, and Heng Huang. Accelerated zeroth-order and first- order momentum methods from mini to minimax optimization. Journal of Machine Learning Research, 23(36):1-70, 2022.
[31] Feihu Huang, Xidong Wu, and Heng Huang. Efficient mirror descent ascent methods for nonsmooth minimax problems. Advances in Neural Information Processing Systems, 34, 2021.
[32] Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems, 1(1::17-58, 2011.
[33] Yang Junchi, Xiang Li, and Niao He. Nest your adaptive algorithm for parameter-agnostee nonconvex minimax optimization. In Advances in Neural Information Processing Systems, 2022.
[34] Nurdan Kuru, S. ilker Birbil, Mert Gürbüzzalaban, and Sinan Yildirim. Differentially private accelerated optimization algorithms. SIAM Journal on Optimization, 22(2):795-821) 2022.
[35] Yassine Laguel, Necdet Serhat Aybat, and Mert Gürbüzbalaban, High probability and risk-everse guarantees for stochastic saddle point problems. accepted to Journal of Machine Learning (JMLR), arXiv preprint S.iv:2304.00-44 2023.
[36] Guanghui Lan, First-order and stochastic optimization methods for machine learning, volume 1. Springer, 2020.
12

---

[37] Dongyang Li, Haobin Li, and Junyu Zhang. General procedure to provide high-probabiliy guarantees for stochastic saddle point problems. arXiv preprint WxXiv:2405.0321, 2024.
[38] Haochuan Li, Yi Tian, Jingzhao Zhang, and Ali Jadbabaie. Complexity lower bounds for serconvex-strongly-oooe concave min-max optimization. Advances in Neural Information Process- ing Systems, 34:1792-1800 1804, 2021.
[39] Xiang Li, Junchi Yang, and Niao He. Tiada: A time-scale adaptive algorithm for nonconvex minimax optimization. In The Eleventh International Conference on Learning Representations, 2023.
https:// www.eeeeiieeeeeeeooorree.eeeeeee
[40] Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning, pages 6083-6093. PMLR, 2020.
[41]
Tianyi Lin, Chi Jin, and Michael I Jordan. Near-optimal algorithms for minimax optimization. In Conference on Learning Theory, pages 2738-2779. PMLR, 2020.
[42] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-11855 1168811-622261111111 2022. Special Issue on Harmonic Analysis and Machine Learning.
[43] Aleksander Madry, Alcksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0660 06083, 2017.
[44] Gabriel Maneino- Ball and Yangyang Xu. Variance-reduced accelerated methods for decen- tralized stochastic double-regularizeed nonconvex strongly minimax problems. arXiv preprint arXiv:2307.07113. 2023.
[45] Aryan Mokhtari, Asuman Ozdaglar, and Sarath Pattathil. Convergence rate of o(1/k) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization, 30(4):320-33511, 2020.
[46] Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In Advances in Neural Information Processing Systems, pages 1208-2216 2016.
[47] Angelia Nedich and Tatiana Tatarenko. Huber loss-based penalty approach problems with linear constraints. arXiv preprint 1.0Vv27711000844, 2023.
[48] Maher Nouiched, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems, 32, 2019.
[49] Balamurugan Palaniappan and Francis Bach. Stochastic variance reduction methods for saddle- point problems. In Advances in Neural Information Processing Systems, pages 1416-1424, 2016.
[50] DTP AIDS Antiviral Screen program. SidoO: a phamacology dataset. https://wwi causality. inf ethz. chhhz.ch/dae wwwchhtt/aa/tpooo///aaapoo html, 2008
[51] H Rafique, M Liu, Q Lin, and T Yang. Non-convex min-mar optimization: provable algorithms and applications in machine learning (2018). arXiv preprint 1811 1810.
[52] Harish Rajagopal. Multistage step size scheduling for minimax problems. Master's the- sis, ETH Zurich. URL: 11850/572991, 2022.
ethz. http/hand1e/22.5000
[53] Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.56477 2011.
[54] Meisam Razaviyayn, Tianjian Huang, Songtao Lu, Maher Nouiehed, Maziar Sanjabi, and Mingyi Hong. Nonconvex min-max optimization: Applications, challenges, and recent theoreti- cal advances. IEEE Signal Processing Magazine, 37(5):55-66, 2020.
13

---

[55] Abdurakhmon Sadiev, Marina Danilova, Eduard Gorbunov, Samuel Horváth, Gauthier Gidel, Pavel Dvurechensky. Alexander Gasnikov, and Peter Richtárik. High-probabel bounds for stochastic optimization and variational inequalities: the case of unbounded variance. In International Conference on Machine Learning, pages 29563-29644 PMLR, 2023.
[56] Umut Simsekli, Levent Sagun, and Mert Gürbüzhalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827-5837. PMLR, 2019.
[57] Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere it: Recovery by riemannian trust-region method. IEEE Transactions on Information Theory, 63(2):885-914, 2017.
[58] J v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen, 100(1):295-320, 1928.
[59] Emmanouil- Vlatakis- Gkaragkounis, Lampros Flokas, and Georgios Piliouras. Solving min-max optimization with hidden structure via gradient descent ascent. Advances Neural Information Processing Systems, 34:2373-25 2386, 2021.
[60] Yuanhao Wang and Jian Li. Improved algorithms for convex-concan minimax optimization. Advances in Neural Information Processing Systems, 33:4800 18100-8811 4810, 2020.
[61] Linli Xu, James Neufeld, Bryce Larson, and Dale Schuurmans. Maximum margin clustering. In Advances in Neural Information Processing systems, pages 1537-1544, 2005.
[62] Qiushui Xu, Xuan Zhang, Needet Serhat Aybat, and Mert Gürbüzbalaban. stochastic gda method with backtracking for solving nonconvex (strongly) concave minimax problems. arXiv preprint arXiv:2403.0r 07806, 2024.
[63] Qiushui Xu, Xuan Zhang, Necdet Serhat Aybat, and Mert Gürbüzbalaban. A Stochastic GDA Method With Backtracking For Solving Nonconvex (Strongly) Concave Minimax Problems. arXiv e-prints, page 2i0::::0330080666 March 2024.
[64] Yan Yan, Yi Xu, Qihang Lin, Wei Liu, and Tianbao Yang. Optimal epoch stochastic gradient de- scent ascent methods for min-max optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5789-5800, 2020.
[65] Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for class of nonconvex-nonccoccave minimax problems. Advances in Neural Information Processing Systems, 13:1153-1165, 2020.
[66] Junchi Yang, Antonio Orvieto, Aurelien Lucchi, and Niao He. Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference on Artificial Intelligence and Statistics, pages 5485-5517. PMLR, 2022.
[67] TaeHo Yoon and Ernest Ryu. Accelerated minimax algorithms flock together. arXiv preprint Ar0iv:2205.1 1093, 2022.
[68] Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 3040-3049, 2021.
[69] Yuxuan Zeng, Zhiguo Wang, Jianchao Bai, and Xiaojing Shen. An accelerated stochas- tie ADMM for nonconvex and nonsmooth finite-sum optimization. arXiv preprint arXiv:2306..m 05899, 2023.
[70] Jiawei Zhang, Peijun Xiao, Ruoyu Sun, and Zhiquan Luo. A single-loop smoothed gradient descent-ascent algorithm for nonconvex-concat min-max problems. Advances in Neural Information Processing Systems, 33:7377-7389, 2020.
14

---

[71] Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of sonconvex-stronglyycomm
minimax optimization. In Uncertainty in Artificial Intelligence. pages 482-492. PMLR, 2021.
[72] Xuan Zhang, Necdet Serhat Aybat, and Mert Gürbüzzalaban. SAPD+: An accelerated stochastic method for nonconvex-coneat minimax problems. Advances in Neural Information Processing Systems, 02221666-2688) 2022.
[73] Xuan Zhang, Necdet Serhat Aybat, and Mert Gürbüzzalaban. Robust accelerated primal-dual methods for computing saddle points. SIAM Journal on Optimization, 34(1):1097-1130, 2024.
[74] Yuchen Zhang and Lin Xiao. Stochastic primal-dual coordinate method for regularized empirical risk minimization. The Journal of Machine Learning Research, 18(1):2939-2980, 2017.
[75] Landi Zhu, Mert Gürbüüzallaaa,, and Andrzej Ruszezyúski. Distributionally robust learning with weakly convex losses: Convergence rates and finite-sample guarantees. arXiv preprint arXiv:2301.00 00.6696999,9.,, 19, 2023.
15

---

High-probability complexity guarantees for nonconvex minimax problems
APPENDIX
The organization of the Appendix is as follows:
In Section we summarize the key notations that used throughout the main paper and the Appendix.
In Section B, we provide the proofs of Lemmas from Section 3 that were used for deriving the approximate Lyapumov descent property.
In Section C, we present the proof of Theorem 7 which provides an approximate descent property in terms of the Lyapunov function for the sm-AGDA iterates.
In Section D, we provide the proof of Corollary 8, which studies particular choice of parameters within Theorem 7.
In Section E we present a proof of Theorem 9 which provides a concentration inequality.
In Section F, we provide the proof of Theorem which yields high- behh-prbbbbility results for the sm-AGDA algorithm.
In Section G, we provide proof of Corollary 14 that provides a high-probability (iteration) complexity bound for the sm-AGDA iterates.
In Section we present supplementary lemmas that were used derive the results.
In Section I, provide further details about numerical experiments.
A
Notation
The key notations that will be used throughout the Appendix is follows:
f(x,y: [(x, v) + E[1 =112 denotes the auxiliary problem.
=) = min.. f(x, ys is the dual function of the auxiliary problem.
((x;2) = max, fxx y: 2) is the primal function of the auxiliary problem.
P(=) min,, max, (1r, y: is the optimal value of the primal problem min, o 2)
x² (v, 2) = argmin, (z, 2) for given y, in the auxiliary function.
(2) = argmin,, (xx; 2) is the unique optimal solution to the auxiliary primal problem.
Y=(=) argmax,, (g 2) is the set of optimal solutions to the auxiliary dual problem.
(2) = + (x= (y. 2), y) denotes an update in y in the direction of the gradient of the dual function, i.e., along the direction ( 2) V y (x (y, =), y).
(I,,,,Eiz) Gz E) -(x-2)
A = G2 1 V (az. yr) and A (z (h+1) v ( Br) de- note the gradient error, i.c., the difference between the stochastic estimates of the partial gradients and the exact partial gradients.
B Proofs of Lemmas from Section 3
B.1 Proof of Lemma
Lemma 5. Suppose Assumptions 2 and hold. Consider sm-AGDA, stated in Algorithm 1, with Ti (0, and and B E (0, 1). For any t M, we have:
((111) f(r, 2x)
16

---

s " IV f =1]l 2z V ( P |2 2111l
l
2
2
+ + p)J 1) V.f(2,, M(i + (2(1 + (AY) (=((F(1)3)))
p+
2 +||/ t72 ||!!!
Proof. Since, 2) is (p + () -)smoothh we have
f(11+1. - f(z v)
- (2++1 - Tt, V (x) + p p+b
| - xyl1
2
p l. -/11G (2 Me 2.1?
[T (2 Mbi 21), V f Uei z)) +
2
=(Ptt 1 71) V [1 (Ti + (p f)), Val(ne
24
p p ||
where last equality follows from Go (1))) = V.l(e,, + Af. Hence, for S have
L ptf we
f(1++1, y13 - f(res yei 21)
S T 1 Mii - 1)A Val(e. p+c
2 ti ?112
(15)
Similarly, we observe that for all, V,I((,);))) = v,l(a, w), for all x, 2, which together with the smoothness of fzx, implies
f(1 M141i 2)) - fxxe+1 Mii zt)
I (V (V 2)), Met1 y1) + l
+=5 21l1141 -ell yell
(V (V Gy U1, (b+1)) + L - (,I,G,,(4,11066 16.41,81.5411 S1+1111
(16)
LT2
=72 1 ク (ff(( 1= + 72(1 + 2T ( ff u)) A/) +
|
2
where we used again the identity Gy(Ie+1, yr, {441 V,, ( Mii 22) + A
Finally, we observe from the sm-AGDA update rule 21+1 2t = 3( =) that 1 ( - zx)
Tt-1 2t and (1 ()(x+11 - 2)) = (I1+1 zt) (2t+1 2) xt+1 -1-1. This gives
f(5441 244122+11 ((Ie41) p [||2++1 1+1111 112 || 2z [=]
2
P
21)11 -2,) - (11 - (3)
|| 2+
2
g2
(17)
p
(1 3)2
|| z0|1
1
1l zel?
2
B22
P | - 11
<
28
where we used 0 < B < 1. Therefore, summing up (15), (16), (17) yields the claim.
B.2 Proof of Lemma 6
Lemma 6. Suppose Assumptions and hold, and p 2 E. Then, for any N,
V ((41)12441) (y( 2t) 72 (2vV (a1a"""", (11,2) v.), (yf(1 To T2 (v ()), vc). A7)
17

---

Lo
+ + (1e41) At) (147119
2
P
(2441 26, 2++1 + 22x (1(+1, 2-1)),
2
where Lu (1+ ptl p-8) and the map x² (c,-) is defined in (11).
Proof. By Lemma 19, V is smooth in y for any given E Rdd, Then, using V, (v):)) = Vyf(2" (yt: 21),
2t), yt), we obtain
L
Le (8(-1. 21) (Br, 2)) 2 (Vyf(z' (ye, 21).0 86 2)2) yt+1 - ve) 2
|1 - y1l1
2 T2 of (x (ve, 2.) Mt), Vuf(E ve)) + T2 (V (ve. 21), A?)
L
2 - (I V ( mil 2(V ( 21) A2) + ||
(18)
Furthermore, by definition of V, we also have
V (y6+1) 2441) (1++1;2
( 21+1) Ve+1:
((="((t+, 2x), :6+1:
2 fx" (y++1, 24+1): Me+1i 2+11 - f(x² (16+1, 2++1); (111:2)
P [|lze+1 x² (1/+1: 2011)1 1ize - x (y++1 1+13
2
2 (24+1 - 21) [2++1 + 2t 2x (1/++ 2++1]]
(19)
Summing (18) and (19), we conclude.
C Proof of Theorem 7
Before we move on to the proof of Theorem 7, we first provide result from [70, 64] that quantifies the change the function over the iterations. We also provide proof for the sake of completeness.
Lemma 15 (Lemma B.7 in [70]). Suppose Assumptions 1, and 3 hold. Consider the sm-AGDA iterate sequence (It, St: =E)EN for p > E, and let (=))) 2 argmax, (yr, 2) denote the set of maximizers of (:.2) given For any t N and y" (2++1) E Y (2141), it holds that
P(a-1) P(x)) -
P (241 - 2t, 2+11 + - 2x" (2441,, 2)).
2
Proof. Let y2(41) Y (2++1) and (ze) Y*(x)) be two arbitrary maximizers. We have
P(a+1) P(z) min max far, 2441) min max f(z ys 2)
max min f(r, max min f(x, v3
(x()((++)): 241) ) - (x)))))
(x(+1); 2411)
(x++1):2)
f(x" (v" (24+1), 2441), (2(+1))))))))))))) 2411 - f(x" (y" (2++1)))) (2(+1); 2)) f(x" ((0(((ee)))) 2t), v" (2++1); =41) - f(x* (v" (2++1); (#++1): 21)
P (2441 20) [2441 - 2x 2v"(2" (x=(),)), 2
The first and the third equality above hold by the definition of P(z) and (y; functions; on the other hand, for the second equality, one needs strong duality to hold. This interchange is valid and can be justified by the simple fact that iss strongly convex in (therefore, it satisfies the PL. condition in has been established in
2), and it also satisfies the PI. condition y according to our assumption [65, Lemma 2.1] that the double-sided PL property allows one for the min-max switch.
18

---

Equipped with this lemma, the stage is set to prove Theorem 7 from Section 3. We first restate an extended version of this theorem, where the constants {cili=1 are provided explicitly.
Theorem Suppose Assumptions 1. 3 and 4 hold. Consider the sm-AGDA algorithm with parameters > E, B E (0, 1). TI E (0; p4tl and T2 >> 0 chosen such that
2p² p l
pd
Co - 2 Ev +
1 2² LØT2 2 0,
% A
+
48,
2 0
33
(p
for some constant v > 0, where by (1+ ) Then,
1 - V4+1 201 V Sx Wr: 2x) 12 + 02 V y (x" (yt, z1); y1 + caller - 2.12
(x
+ + V V yes =)) A (cs V ( ve) + co V (ve, U.),,
+ c 1 2 csll
(20)
where the coefficients eı to cs have the following forms:
C1
11
(p-
8)2
+ +
col + + 483pp
0
P 22-1 )
(ch3 + ( ET + (11+72)2 n.
243p
2pl
2
c2 2 (p f)j
1
c= 16
p
1)
44 = 1928p
P ++ ! P 1
T2
48 tool² +26 -2688) nी
G = (1 + lr - T2),
06 = 2T2
96 p+ 2
2 +
p p+c 200 L (1 + 2hurr)) + 666²² nt.
2
6
G8 4838 (15p)))) + 2
p2222 2 p-
+ La +3(11 la + 2l 72.
Furthermore, the sm- AGDA parameters can be chosen such that C1. 02 C3 > 0.
Proof. Combining the inequalities in Lemmas 5, and 15 gives us a lower bound of the form:
V/2 V4+1 2 Ai + A2 + As + A4 + As + Ac.
(21)
for any y (2++1) E (=++1) appearing in A3, where
1 = T1 | v [2, Mii 2x) 12 1 1 - l 2
2 LuT2) IV 17 + 2B
11 2+11l1
A = 272 (V v (vr, 2), M1) - (2f(11+1,1) Wt), (x((2++1181))
A3 2p(#++1 2, x² (y" (Ec+1), x² (1441))(+11)
A4 = -n ((p + mm - 1) ( ye: 21, A7),
A5 = ( (2r ff "yr,, v1) - T2 (1 + ETo + 2 Vyf(ee AT).
Ag = p p+6 77 ?1
Lr3 -
2
Next, we provide lower bounds for several terms in the above inequality, including A2, Ag. As || (x 32) and || 22 At the end, using these bounds within (21), we will be able to establish a descent property for (V.), which will allow to apply our concentration result (Theorem 9) for deriving the desired high probability bounds.
Lower bound for Az. Using Cauchy-Schward inequality and 2-smoothness of f. we have
As 2 -722 Vyff (ye, 21). 91)
2 -122[[lee+1 - x' (vr, 2z) (o
2 - (etul/Vya
lv | x (v,, 22
Ve >>>0 0.
19

---

where last line follows from Young's inequality. Thus, by Lemma 20, we obtain for any v > 0
2
A 2 -- V (T (+)+
1 Tf(p 8)3
t 2 Ve: 2=
4 ( a. (at, ( Mhi
28 i 2
21), ; A7)
(22)
Lower bound for A3. By Bennhy--Shwarz
and Lemma 21,
A3 = 2p(2141 2x, x² x' (244 2x) - x² (y" (v((4+))) (441) 2p/21+1 2t, x= =0("""""n000 (2++1); =+11 - x² (yt+1: 26+1)) 2 2c+1 - ==1ll<=* (z(+1); zt) x² (v" (2141),
+ 2p(z++1 - =t, x² (1) (24+1), ze+1) - (yt+ #++1))
2p²
2 p- E | - 2112 + 2p(21-1 - 24 x' (2)1(++)): 2+11) x² (9041,
Hence, using Young's ss inequality, for all B 0, obtain
( 1/20 + p p-e) || 2e12 6,pp|=² (y (=++a, 2411) x² (1++11)++111( (23)
2p² "r²)
Ag 2
63
We now lower bound the second term on the right-hand side of (23). First, note that we have x² zy"" (2++1), =+11) = x (2++1), which follows from the fact that min, max, f(z, v; = maxy min, (2r, z) for all since f is strongly convex in and satisfies the condition in As- | wi
sumption 2. Hence, using the inequality {w;) E Id: and N Z 1, get
s N Em ||0 which holds for any
(1="(((*++)) 24+1) x" (yt+1, 2++1)12 S A||="(=44)))) xx (2+)11 + 4||00 (2+) - (2t (2+), 2111
+4||=² (vT (z),, x' (1/+11, 2 Il? + +41x"(y(+11 21) x (21+1,
By Lemma 21 and Lemma 23, observe that
8p2
4||x" x² (x+)) +
- x= (V1+1, 211+111 - (p 0)2 111241 -zell.
Using Lemma 24, we also have
4
2pl 2 1 T2 T2 2 11Vyf("((),)) 2)), v.))1.
41|="(2)) - x= (vt (2t), (p-1)p
Finally, using Lemma 18, we get
4||0" (yi (zt), 21) - x¹ (Ut+1, 21
+ {
4 p 11 (z+) -
p )+
V, Vyl(x ( 21), v1) (, Ve, S
p-6
p+/22
4 p- 22 v ff ( 22). we) (V 17)
<4 p + l
1/
(26² |x (yr, 2)) - x1+1l1 +2)
where the last inequality stems from the é-smoothness of f. In view of Lemma 20, we may further upper bound the above quantity as follows:
4||x= (ve" (zx), ze) x= (vt+1, 2t)
20

---

I 16 (+1) p p- () l शंं +8 ) ||ل!!
-ppp- 8)3 + +i) +
ui: + A3) + [|A71]]]
Summing up the three intermediate inequalities we established above, we obtain
(1x (2+11, 2++1) x² (yc+1, 1+11
8p²
(1 (1+ =2 2 VVV (3),2))
1,1+1 - 211 (p-
(p- 6)2
p-
+ 1) 82
+ 16
(pppp f)2 +1
V Va
p+62
+ 32
@ ( Ve: 21)
p p-
+ 16 p 1
p- p-
0) 00- 1A1
1 47 +8
In conclusion for Ag, using (23) and the above inequality, we obtain after some rearrangement
P
2/²
1d
|| - =el12 (06637(( (00))((=t) ETp ( ||4;11
A3 2
+
+ 48,3
63
p -
(p - 8)2
24
2pl 2
[ (ve, 31)1
72
(p - f)p
(24)
00.3p p 1g l2
7p(p- - 8)2 +1) 7/12 |V= 21]1
p 1)
192,3p
p=1)
p- p- e
(Vaf(xe we: 2), A7) (888pl
P (2+))² p- 72 |4
Lower bound for As. Below we first bound (V U1). A i.e.
(VyJ( v.), A.) = v (. ,). + (V (1((1)))))) - (xI(
VVl((m, ()), + 12 V Vul(((41, - V V.s((r, 3
t
(Vu((r We), ,),, AT) + G (2+,
ye G+122)1 + 3311111A11
12f2
G
(uff(r, vc), AT)
| IV f(,
Me 24) )I
||ح|| + 3T
672
where the first incquality follows from Young's incquality, the second incquality follows from Vyf being f-smooth and the update rule #++1 = It T2G2 (ut, ye. 21). and in the third inequality we use A Gz. Ut, Ef.1; 2.) fal Ve: a.). Thus, since - (1 + Erz + 2b < O. for any 72 >> O, we obtain
eri
As 2 - r2((1 + LT + 2L m2)
V (m,, ye =1)
| + 3=111171
(25)
Grz
6
+ +2r (z" (ye, 21. ve) - (2 + LT 21
V ve), A
fl
Lower bound for V (1. w. 2 Using Lemma 20 we can lower bound V (x++1, woll as follows:
V y
2
v fx (w) 24) w1)1 / - Vyl(2" (2r, v.)l
M
V (x 2). w:)11 Al - (v1.
2 VV fa (2=(00)000 (y/, 21)22
2 [
77(p ej² + +1) 1 [eller,
1
21) AT) +[|47]?
21

---

We observe that v 30ll appears in both An and the lower bound given in (22) for A2; hence, grouping the two terms together and using the definition of 2 0, get
72
l 1 2 - -LOTT)
-220)
(V | = coll V vyd (1141,
1V (G Vos(2 (wi 20), 2.). - -2cne² (1o 10= tr)
(26)
2
200l (2(V=f(as Ai) + ||A;11)..
Lower bound for ||=+1 - zel1². Since 2++1 22 3[I(++1 - 2e) for some 3 > 0. we observe that:
|| - 20l1? = 3 1 - -z,11
= 331121 20 - 21 2-1 2 Wo. 2.)11
ye 2 8² (plece |1:e - zill - W. (611:211))
and since (1,,
V. far, yos z1) Ai obtain
g2 1|+ 2
2-11
=e v ( -V U4: 2). A7) - -
V.
Note that ||
zell appears in both A₁ and the lower bound given in (24) for As; hence, grouping the terms together and using definition of da 2 Using (27), we obtain
P
p
2p²
483
p
||
zell² d61lzc11 2011
28
63
p
(
E)2
82 1|= ==11² 22h 22 (V ff Ve: 2e), Ai)
2 -6, 7 1 Ve =1
+ co
We conclude by combining all these lower bounds, i... the claimed Lyapunov descent inequality follows directly from summing (21). (22). (24), (25), (26) and (27). Finally, follows after straightforward computations that there exist choice of am-AGDA parameters which yield c1, 02. ca >> 0 while satisfying the conditions cu 2 0 and da 2 0; in fact Corollary 8 provides such em-AGDA parameters explicitly. This completes the proof.
c
D Proof of Corollary
The lower bound provided in Theorem resembles the descent property we require for our concentra- tion result in Theorem 9. To allow for its proper application, develop stepsize policy inspired by [66].
Corollary 8. Under the premise of Theorem 7, consider the purameters p = =26, E (0, 3
T2 Zh B aptz for any (0, J.l. Then, Â < -B, Cu1 Det1 for all N,
where v = 7
and
T2 [V (yr, 21). 2 3r 5p11 21 yi 21) TI [x)
À 2 Vi,
B.
5
8
8
Ö.+1 1 [11928p p 8 8 + 18 + 4 l + 2c) ) गी
(1 ( (1=+ -1) (Vaf(tte 2x), Ai)
1
+r2((1 L72 + 2z) VzI( - 2V, (x) 21) A7).
Din & 207111111 + 8073 ?
Proof. In view of Theorem 7, it suffices to prove that setting p = 26, 71 E (0, 11 - T. ajr2 for for some positive < 406 and 7,6 need to show that this choice of parameters implics the following lower bounds:
12 leads to both co 2: 0 and c/ 0: furthermore, also
22

---

0a2B ( (ii) cs
2
po 8 (10) 2E7 (0)cs -8(12.
First, we show that our parameter choice implies that co, c 2 0. Noting that Lag = 46, using 71 S 32" we may bound from above and below as follows:
l
72 2 cD - -2 + T2
21
1 2 - LI0T2
(27)
=72 (1- (5
+
L + (v)
T1
2 T2
9
2 オオ 20. 0.
48
2 144
4
Moreover, since 3 = 01172 and T2 S
T448
1
we get
3 <
144
using 2 1. Hence, for S
400
1
2p²
pd
1 88(1 + 483) (1 + //3)
0 0 p p-6 l (p- 8)3
+ 48/3
(28)
3 123
where the last incquality follows from a S 406 (V5 - 1); therefore, ce Next, we prove bounds on c1, C22 C32 C7 Cg separately. Proof of part (i). Since p= 26, v
2 P 13 33
12 7
and TI
S
32 we first observe that
28
+ -| = こ/ ( +
7
(1+1)
1
5
V (p f)2
6
1) 1
6
9
27
(29)
Furthermore, using T2 =
I5. and 33 = a1112, we obtain
963p
p
+
1
+ नミ
= 9636 (p + 8)2
p
( + 1p 1)3
12
8
(p 1p- 2/2
(p- f)2
1 1 T02
<
96 aTM 18
(1+
T20
1920
Ti
TI
5a
2592 T1
(30)
where last line follows from the fact that u/l- s 1 and
S
Tu in which we used
183
T1
37
Since T2 2 co 2 0 and -2--222 20 (6--)z + 2
20
using T2
we obtain
48:
1
207²
5
-20
(p- 8)3
+ o
108
71
(31)
Using Lu 4e, Ti S 37 and T2 S
1
we get
E (1 + 27 + 2T 12) oo
1
8
1
1
8
17
+
71
71.
(32)
6
6
144
144
18
144
144
288
Using T1 S 37 T2 S 1447 this implies
-6 >
-3"
-B
P
2
3
144f
ajl
38
1
71
>
648 T1-
(33)
Combining equations (29), (30), (31), (32) and (33), we obtain
5
5a
5
17
25
17 + a
G
2
T1
2
27
2592
108
288
648
2
108
288
(34)
181
3a
60
a
Ti
2
864
288
5
23

---

for a S
1
< 2.4, which completes the proof of part (1).
106
Proof of part (ii. Due to our parameter choice, we first note that
248p (1+
2pl )
72 48a (1 12 0-66 S 96072 S
2pl )
2
(35)
(p Elyz p p- (
8
where last line follows from T2 1447 and S 406 Finally, (27) implies that
hence,
4:
C2 T2
Proof of part (iii). We have shown that c 2 hence, C2 2
ps 8
Proof of part (ie). According to (27), T2 2 co: hence, we deduce that
-2a 2 2T 2 72
(36)
where the last inequality follows from 52 S 1472
Furthermore, note that
-663pp
p p
2
e -7728a73
123 l
(37)
p p-
with the last inequality following also from
& S and
T2 S
1 144t
We also note that
2
p +
[T]
3
14 l
8
2
Finally, since 6 s 3/ according to (28), we get
6/ 2 -3² P 2
ou1z2 2
0
f
(38)
33
3
216
where we used 72 S Tuue Thus, since
(0, 1), it follows from (36), (37), (38) and (32) that
07 2 -
14
a
17
2 2
72
123
9
216
96
Proof of part (v). For cg, we may simply observe that
4883p p 1+6 L.=0 )²
+
+ Lo + 38(1 + Lr + 22(TTT))
2
864apu2 +
4 + 3 (1 + LT2 + 2L
E
2 6a " + + +4+ 1+ + 8
1
2 2 -81T2
p 2
144
144
where we used n/l 1, Lo 4l, T2 5 1447
and < 20
E
Proof of Theorem 9
Theorem 9. Let FFhee be a filtration on (Ω, P). Let Ar, B,, Ct. D, be four stochastic processes
adapted to the filtration such that there exist oc, DD > 0 and Ti > 0 such that for all N: (i)
B, 0, (ii) Ele201+1
Ft] S ex B' for all X > 0, (iii BleADet1 | Fi] S doo for all
0, and (iv)
Act1-Ae < -B + C-1 D441. Then, for any q E (0, we have
TI
B, s (Ao Ar) + TOōT + 2n max (20 (202 of) log 2
IP
t=D
24

---

Proof. For some fixed y > 0, let
T-1
Sr _
B. - (Ao - Ar)
t=0
with the convention that So - 0. For any T E N, we have
SI+1 = 7
Σ
B, (Ao Ar+1)
T-1
7
Σ
Bt - (Au -Ar) + yBr - (Ar - Ar+1)
1=0
ST + tBT (AT AT+1)
ST + (7 1i By + ni Br (AT - Ar+1)
+ Br + TCCT+ + Ti DT+1,
where the inequality follows from condition (iv) of the hypothesis. Hence, for any 0 < x S 2n and any T N:
Eis1 Frl S EleAST B DM Frl
S e ASr e M Br Ble E Ef Frl
AST e A(7- Br (12 ( )² (22Tr133)
A8r e M +2 )a By e Xnn
where the first inequality follows from Cauchy- Schwarz and in the second one we use (ii) and (iii) of the hypothesis. Fixing 7 = TI / yields for all 0 < X I am
- 1 + 2X02m S..
- 2 2oo =1
[ [وmaz (, }. using Br 2 by (i) of the hypothesis, we get
Therefore, for 0 <>S min
Ele Frl S e 2r
and rolling this recursion backwards and noting So = 0 yields:
EleAsy] < ATO
thus, using a Chernoff bound, we get
P(St t) EleAsrJe- N (()
Since for q (0, 1}.
(( < + 12 TOAT x
log(())
we have
T-1 P
1
B, < (Ao -Ar) n A log(ÿ) 2 21- q.
t=0
The claim follows by taking
min
2n
201
25

---

F
Proof of Theorem 11
Theorem 11. In the premise of Corollary 8, sm-AGDA iterates (x1, for S satisfy
1 37
P
T [v. will Kll
S 24.T, 2 21-4, WTEN, (011),
for some Qq. 0 m()otbb) m6 + 8 Tie + / log ( )1) explicitly stated in Appendix F, where D. - ((2)) $6, bo 2supa.y { (ro. y; 20) f(x yo: 20)).
As a first step, we provide a helper lemma that shows that our concentration result (Theorem 9) is applicable to the sm- -eDe---erlltedd processes Ar. Br, Ct, D. introduced in Corollary 8.
Lemma 16. Let A, = A. B, = Br, C. = Cr. Dr = D.. where Ae, Be, C₁, D, are defined in Corollary 8; moreover, let TI > O be the primal stepsize in sm-AGDA. Then, the processes A,, B,, Ct, Dt are adapted to the filtration F, - FP, where FK is defined in (6), and they satisfy the conditions of T'heorem 9 with the following constants:
(1) 00)0 = (2406 + 326)),
(1i) o = 1667,5 + 64e7262
Proof. The fact that A,, Br, Ct2 Dr are measurable with respect to F, F. for any & E N follows directly from the definition of Ft. Note that xy and yt are also F--measurable for all i E N. We prove part (i) and part (ii) separately.
Proof of part (i). First, recall that C++1 Cc+1 (V yei y1) Ai) (cs vf y1) eo vf (ye, z1), st), A!). We would like to show that CCet Fi] S o for all X > 0. From Assumption 3, we note that for any À 2 0:
E
exp
A( X(-ca (1e 24)
| S exp
Ve: 23)1 )
where we used [35, Lemma 3]. Now, given the value of ca in Theorem 7 and the convexity of t + t?,
we have
2 (Tzत7 5ाी (p+ 11) 2 + 5 ()) + 5 (4cp8 + (2c))02
c 5 19233 (p#t)
Now, leveraging the stepsize policy specified in Corollary 8, since E (0, 1). using (37) we get
((1 1923p
P p l
le 1
< 20
( 123
2 20
a
2
2
3 123
4
Similarly, as Ti S 30
1 have 1(p + l)TT - 1) E [0, 1). which implies
5ाी (p + 4n - 1) - 5ही.
2 T, we have (#T7)²
< g / < o.. Furthermore, using (27), we have 5 (2) S
Since v=
If 80T1-22 < 4
Finally, (38) and a E (0, 1) imply that
x2) s 20
a
(n)
-
1 $
216
Thus, of Gr2. which implies that
E exp
Wei 2x), Ar) | Fr s exp 48A²77' V.. ( Mei (()))
(39)
Moreover, noting A is revealed after Aú, using [35, Lemma 3] again along with the inequality |u + all $ 2|| + 2| we have:
exp
V -N(csVyflo + c Vuf(2' (v., 2)) A7) 1 Fe, A/
(40)
26

---

}
(1+ + + yf (V, 2.). wr). Ai) I Fes A?
exp
Sexp (8A2T3 [1 + + + 2 V (
2V / (v) 21) 11 11)1 62)
$ exp (641²*2 Vyf(2 (yt 21 16 (1 fr2 + T2) of 6e) exp (64X2+3 y (+ (yt, 2). y:)) + | y (z), 30 57)
(41)
where the last line follows from 0 < 1 + ET + 22 v 1+ + 144 + 141 2
Thus, since A is revealed after Ai, using (39) and (41) together with the tower property of the conditional expectations, we get
E exp -A -N(a(Vv.(u..i:))... +cs (or ]
Sexp
+
62 (1V (M) /f/.
(6) og
(42)
Finally, observe that
[V Vf (11,))),
S [ 11)
vuf (v).
I f||21
x² (Vt, 21)
v | Vef( 21)
where in the last inequality we used the (p f)-strong convexity of f(, yti ze) and the fact that we
have V la (x" (ye, 2x), Mei zi) O. Therefore, since p 26, we obtain
( < v (v,, 2). 11) + V (x),
(43)
Plugging this inequality into (42) yields
E exp
-> (Ve 9o: 2)) (f 21)
S exp
x² (48-768 1287753)
=ll 102 Vw
< exp
X'max ( (1877 + 128-388) 1530A9-263) R.
S exp
X' (24065 320))..)
where the last inequality follows from r2 58
Proof of part (i. Recall that D(+1 A Di41 = A + 1 We would like to show
that Det1 Fi] S d for all X 0, 1 for some oD > First, observe that for any
such that À & min { 162 64e7z0z }. we have
E ADe+1 Fil = E
( 2 Af AFIP]
ANO P $ IE 1
EE
( (2
(128A01763)
e (1607788 +6467226)
where we used [35, Lemma 2] in the second inequality.
27

---

Before completing the proof of Theorem 11, we provide another lemma that gives a lower bound on B, in terms of the squared norm the partial gradients, based on our choice of sm-AGDA parameters. Lemma 17. For any & E N. we have
20k
16k.
32
[ xll yff y0)l I max
Be.
(44)
T1
72
3
Proof. Since V. m1) = vaf(Ir, Me + p - z1), using laa + 612 - 2||a|1 9 + 2|66/ we get
[V, )= < 2/ [ yr; =)]l + 2p²|| - zz 1|
Hence, together with (43) and the definition of Br, we obtain
V ( )11 V (ct, 1=)
$ (2 + 2k) V f (Tr, ye:
21
2k|
V Vuf(x" (2"/xx"" (ve, 21).)))))))) 31)1 + 2pp - =el1
(2 - 2x) 5 2x.8 8
2p² 8
< max
Bt
T2
Bp
20n
16k
32
< max
B2.
Ti 72
B
with the last inequality following from 2 1.
We may now provide our proof for Theorem 11.
Proof of Theorem 11. According to Lemma 16, the processes Ar, Br, Ct, and D, defined in the statement of Lemma 16 satisfy the conditions of Theorem 9 if we set Ti as the primal stepsize of sm-AGDA. Therefore, for any q E 10,
T-1 B.STA
( (1) )))² 2
P
{22 D L=
B, S (Vo - Vt) + nooT + 2T, max (202 ob) (206.0b) log
--1..
Thus, dividing by T and using Lemma 17, we can conclude that with the probability least 7. the following event holds:
V fxx Vill + m/
T
(45)
2 max
20kc
16k
32
- (Vo V7) +ob
1 T max (102, 20%) log (1))
Finally, we can relate the potential gap Vo Vr to the primal suboptimality, i.e., Ao. and to the duality gap, i.e., ba/2 = sup2,8 _u (z ys zo) fsx, yos z0), at the initialization, following the same arguments provided in [66]. More precisely, for V( y, 2) - f(z,0:2) - 21 2) + 2P(z), we first observe that
Vu V1 S Vo- min V(x, y.2) Vo min f(x (v. 2.212
3.y.2
(46)
< Vo min P(=).
where last line follows from f(x, yi 2) (y 2) for all y. and P(z) - ( 2) 2 0 for all y, Since p = 26, we also note that
P(zo) = min max f(x v) + f11 =0ll2 & max f v) = ((20)-
9
Finally, since P is the Moreau envelope of L. we have min, P(z) min, 4()); therefore,
Vo min P(z) = f( vo: =0) ( z0) + 2P(2n) min +(=)
<((2)) min (x)) + f(ro, No: 20) - 1 (1 20) + P(zo) V 20) (47)
< ((20) min (2) + by bo Do + by-
28

---

Note that 72 implies 16x/T2 2 20k/TI, and 328/3 Therefore,
32
KK/T22 > 16k/T2 since E (0, 1).
20k
16k
32
64
max
K/T2.
(48)
T2
B
Therefore, combining (45), (46), (47) and (48), we conclude that 2q.7 has the following explicit form:
24 = = A + bo
r3
log }
r2 +
(49)
T
T
where the constants T1, and 13 are defined as
64
12 o 1667: 6 + 12
T=03) r3 m 20 = 40 4n (2406² 326)).
72
where the equalities follow from the expressions of of and of provided in Lemma 16.
G Proof of Corollary 14
Setting T2
T1/48 for any Ti > 0 implies that Qq.7 defined in (49) satisfies Qq.r 18//nttb ensures that 32 VTES
O
+(gg+ 7
+ Ti End + 8k log (1)
Hence, setting min
Qx o (Ao (15
do T
(Ao + bo)e T
62 ())
+
ōn +
log
T
Finally, obtain an e/ e(e,e/V//)ssstee -tta-stationary point, it suffices to have the above bound smaller than (ce2), which is guaranteed when the following three conditions are met up to a constant factor: (i)
(Ao+do)eN 3, (ii) / (bo+ 8x 3. and (iii)
6x log (1) This is directly implied by the
T
value Te,q given in our corollary statement.
H Supplementary Lemmas
In this section, present sequence of supplementary lemmas essential for deriving our main result, Theorem (11). Some of these results are well-known and are directly referenced. For others, we have improved specific algebraic constants. Additionally, some lemmas are extensions of existing bounds provided in expectation.
Lemma 18. For any a E Rd and y1, U2 E Rd,, [yx, - x (12.
1 p--
11 - -y2ll-
Proof. This result is provided in [66, Lemma C.I], which immediately follows from [41, Lemma B.2, part (c)).
Lemma 19. For any & E RR,,, the map > (y 2) = min f(z, y, 2) is (1+ p-1) smooth.
Proof. This result immediately follows from [41, Lemma B.2, part (d)). Indeed, for any 31, 92 E Rdd,
we have
IV (g1, 2) - V, I|
|V (v1, 2).))) v1) v,f( (v2, 2), y2)1l
< <(l="(yn.2) (wz. =)1l + Elly - wall - [+ p p Ellyn vell.
p
where the first equality follows from Danskin's theorem, the first inequality follos from é-smoothness of f. and the last inequality follows from Lemma 18.
Lemma 20. Consider the iterates (zit, Ut, ze) of the sm- AGDA algorithm. For any t E N,
| - x" xV,, 21 $ 7 8)3
V Vc: 21
(V. Mi 2)) AF) + 27 1/12
29

---

Proof. For any t E N, using the fact that f is strongly convex in with modulus (p E) together with x² (ye, z4) argmin, fa, yii ze), and I++1 I2 ( Mr, (44112)). we get
| x² (yr, 2)l s 2|| x (yt, =))l + 2|| xz 21
$ (p- 2) V f (Te, yri 2x )|( + 27 G (zt, Vt, E7 S4+1:2)11
Using the identity (z Vi, Vts 1 2)) = a + V f 21)24) within the above inequality yields the desired result.
Lemma 21. For any y E R2 21, 22 E Rd we have:
|1" m1) x (v. 22)ll 1 - --11 |1
p
=2ll.
Proof. This result follows from [70, Lemma B.2]. Indeed, since f is strongly convex in æ with modulus p- - L, we get
f(x" (y. (y. 21), ys 22) f(x" 22) 2 p | (g. (g. 31) -2, (2,
(50)
Then swapping 21, 22 leads to f(x (y, 22,, yi 21) fxx (y, 21) 2: 21)
2
p- -
120 (y.21) 21) - x² x21,
(51)
2
Furthermore, from the definition f. follows that
f(" (y, 22). a1 f(z" [xx"((' (y, 22), vi 22)
p
(1|=" (v, 72) - =1 112 | (2) - 22 221l?)
2
(1121 -112211 + 2(xx (v. 22), 22 - 21))
2
similarly, swapping 21, 22 in the above inequality leads to
f(x (v. (1.2)).r z2) f(x (y, 21), yi =1)
p
(11=0 (y, 21) - -2212 12- (v, - 21 =11
2
(1122112 ||21122 2(x" (y, 21), 22 - 21))
2
Thus, summing the above two identities and applying Cauchy- Schwartz gives us
f(x (2,22), yr 21) f(x (y, 22), yi 22) f(x" (y, 21), y 22) f(x (y, 21) y
S pl|0" (v, 21) - xx (v, 22
22ll-
We can lower bound the left hand side of the above inequality using (50) and (51), which leads to
(p - |= (y. 21) (y, 22 s p||æ* (y. 21) x x"(, +2)]||[ 22|l-
(52)
Rearranging this inequality yields the desired result.
Lemma 22. For any x E Rdd,, - (x;; 2) is (p- - -t-strongly convex.
Proof. For any y E gdd, , E Rd x + f(z, yiz) 2) is strongly convex with modulus l, i.c., + v; || ||= is convex. Then, x + SuPyER v; - | is convex as is the pointwise supremum of convex functions. Therefore, I + (2;22 - | is convex, and this implies x + (xx, =) is strongly convex with modulus p
Lemma 23. For all 21, 22 E Rd,, we have
|x x² (22))
S
|| 221l- p p-
P
Proof. Using the result of Lemma 22, one can show this result following exactly the same arguments in the proof of Lemma 21.
Lemma 24. For any y E TRd,, E RRd, holds that
2p
(z) (z), 2| (p f)ız
1 T2l
(1,,)((((,,
Proof. The proof is the same as [66, Lemma C.2].
30

---

I Further Details about Figure?
In this section, we provide further details about how the empirical and theoretical quantiles are estimated in Figure 2 and are compared.
Generation of the theoretical quantiles Qq. For any given 9 (0, 1), estimating an upper bound on Q.. requires estimating an upper bound on the quantity Do + bo based on Theorem 11. Other constants such as T1, 12, r3. l. and μ are explicitly known the setting of this experiment based on the NCPL game where T 10,000 is fixed.
First, set the initial point (Io, yo) randomly, where each component of zo and yo is sampled and we set 20 Ii. We then estimate an upper bound on the quantity Ao + numerically based on a grid search, resulting in A. + bo 12. Second, we generate a linear mesh mm with a grid size m = 0.0002 over the interval [0, 1J. For E I. we calculate 2q. based on Theorem 11. Third, we generate sequence of quantiles Q1... T. These quantiles are used to create CDF via linear interpolation using the scipy interpolate package's 's interpld function Python. Note that this quantile sequence generates CDF over the values QIm.T.
uniformly from the interval [-20, 20J.
Generation of the empirical quantiles of the random variable X7. We generate 1,000 samples
(X-+) 11000 from the sample paths corresponding the NCPL. game with T 10,000, where XT T El= [V==f<z, y0? + of 3/)l represents the path averages of the gradients. generated over Im using NumPy's quantile generator in Python,
Quantiles for this sequence ensuring alignment with the mesh over which the theoretical quantiles were generated. Evidently, our theoretical quantiles dominate the empirical quantiles pointwise, demonstrating that in the challenging NCPL regime, our theory provides empirically verifiable guarantees on the tail behavior of the random variable Xr.
Comparison of quantiles. We plotted the CDF corresponding to the theoretical quantiles 2q. over the values of the empirical quantiles using common mesh grid over the range of the empirical averages the sample paths. In other words, we scaled the quantiles Qg.r with an affine transforma- tion that their range matches the range of the empirical quantiles. This affine scaling preserves the shape of the distribution corresponding to the theoretical quantiles and allows for better visualization.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes] The abstract and the introduction reflect the paper's contributions and scope accurately. In fact, we provide adequate references to the results from our paper for the claims in the introduction so that the paper's contributions and scope are easier to understand.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
[Yes] Although we provide the first high-probability bounds for nonconvex/PL minimax problems the best of our knowledge, there may still be some room to improve the condition number A dependency based on the existing lower complexity bounds [38, 71] in expectation, unless the lower complexity bounds are loose. Also, acknowledged that for the distributionally robust optimization experiment our assumptions do not all hold (which is limitation); but that our results are still predictive of the practical performance.
Answer:
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes] All assumptions are clearly stated or referenced in the statement of any theorem, we provide a sketch of the proof of our main theorem.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
We uploaded our code as supplementary material, and explained in detail how experiments performed. All the datasets datasets and we provided the adequate references to them.
Answer: [es]]
well-known benchmark
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes] We explain the experimental setup detail and provide the code, the datasets are well-known benchmark sets that are publicly available and we provided the adequate references.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes] Our numerical experiments section has all the details.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
[Yes] In the experiments, we compare the histogram of the solutions found by multiple algorithms. This has more resolution than standard approaches that has the average performance (as an average over the algorithm runs) and the standard deviation over the runs; because can visualize the behavior (histogram) of all runs.
Answer:
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
32

---

Answer: [Yes] We explained the details of the operating system/computing resources.
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps://neurippp wttppclicp/eeaeeeeececcccc csGuidelinest
Answer: [Yes] We preserve anonymity and we absolutely confirm with the code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA] Our paper is mainly a theoretical paper with convergence guarantees for
stochastic mini-max algorithms, we aware of any direct impacts the society.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA] Our paper has theoretical nature an we are not aware of any potential misuse risk for our results.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
We use public datasets, and implemented the algorithms on our own in Python; no licensed material is used. We use public well-known benchmark datasets which are properly referenced.
Answer: [NA]
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA] We do not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA] The paper does not involve crowdsoureing nor research with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA] No human subjects or crowdsourcing were involved.
33


Reviews:
[{"review_id": "dD7smVP9cq", "reviewer": "Reviewer_NU5h", "summary": "The authors derive a high probability bound for convergence of a stochastic gradient descent-ascent method over a nonconvex (PL class) of functions. Specifically, they analyze the sm-AGDA algorithm, which previously only had a bound in expectation, and show a similarly tight high probability bound when gradient noise is assumed subgaussian. The authors provide two experimental settings demonstrating that the distribution of these iterates over multiple runs converge as expected for the algorithm.", "questions": "* As mentioned in remark 11 this bound appears to be pretty tight compared to the bound in expectation. Do you think if this type of analysis was applied to other existing algorithms you would get similarly tight bounds?", "limitations": "The authors clearly state their assumptions.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Technically solid paper that thoroughly analyzes the sm-AGDA algorithm and understandably interprets the results (e.g. remark 11). I don't think the assumptions are too strict, or at least not outside the norm for this type of analysis. \n\n* Good comparison to other related works, especially Table 1 is useful to gauge the current state of theoretical results for this problem.\n\n* Helps to move analysis closer to realistic nonconvex min-max optimization settings.", "weaknesses": "* While I understand that the bound in Thm 10 over the average gradient norms motivates the random sampling of iterates in the sm-AGDA algorithm, it still seems like practically the best thing to do in Fig. 1 is to take the last iterate (or at least sample over some last window of iterates). Is there a straightforward way to come up with a similar high probability bound on the last iterate? \n\n* I'd maybe like to see more discussion about how the analysis in this paper differs from that in the original sm-AGDA work [63]. At least for some previous work I've seen with high probability bounds for convex optimization, oftentimes the actual modification required to go from expectation to high probability is fairly minor (i.e. just boils down to bounding the summation of Subgaussian noise terms).\n\n* I like the first set of experiments but a bit confused at the purpose of the second. I don't quite understand comparing concentration properties of other methods if the analysis only applies to one algorithm. Maybe if the authors highlighted what unique components of the sm-AGDA algorithm might give it better quantile convergence properties than others this would make more sense.\n\nIf the authors can address some of these concerns I am willing to raise my score to a more solid accept\n\nOther Minor Notes:\n* DRO abbreviation in line 238 not defined initially\n* [63] and [64] appear to be referencing the same paper\n* Line 306 and 311 appear to be referencing the wrong figure", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the feedback. Below we provide a point-by-point response to each of the weakness/question raised in order.\n\n**Weaknesses**\n\n**1-** The only known last-iterate results for nonmonotone VI problems require more restrictive conditions like local *quadratic growth* around critical points or 2nd-order sufficient conditions [28, 74]. Some existing worst-case analysis in convex-concave cases shows that complexity with averaging can be strictly better than the last iterate [76]. In conclusion, establishing a last-iterate convergence result in our nonconvex setting appears to be difficult and unlikely to offer better guarantees than our averaging approach.\n\n**2-** We concur with the reviewer that noise from subGaussian gradient estimates are likely to produce a light-tailed distribution for the final iterate (or its average), but the main challenge remains achieving a precise balance within the bias-quantile trade-off of our complexity. Our bounds are tight as when $\\delta=0$, we recover the deterministic $O(\\kappa/\\epsilon^2)$ bound for sm-AGDA; and the quantile part (i.e. terms involving $\\delta$) of our bounds scales favorably with $\\epsilon$ and $q$. Turning expectation estimates to high-probability (HP) estimates via Markov inequality would result in much worse estimates than ours, see our **General response: Tightness of our high-probability bounds**.\n\nOur bounds achieve good complexity by striking the right balance between their bias and quantile terms. This is done by defining stochastic processes ($\\tilde{A}_t, \\tilde{B}_t, \\tilde{C}_t$, and $\\tilde{D}_t$ in Cor. 8) that show the desired concentration properties (Thm. 9). Constructing these processes (e.g., $\\tilde{C}_t$) is non-trivial, requiring careful measurability considerations and deriving various inequalities (exploiting smoothness and PL properties) to extend the existing analysis for deriving expectation guarantees [63] and for the deterministic setting [68] of sm-AGDA. For a discussion on measurability challenges in the HP analysis of momentum-averaged algorithms, see [34] which obtained HP bounds in the SCSC setting. Similarly, sm-AGDA employs momentum averaging through its parameter $\\beta$ in the nonconvex setting and faces similar measurability issues. These challenges are specific to HP analyses and do not occur in the less sensitive expectation bounds from [63], where noise can be mitigated using the properties of conditional expectation.\n\nObtaining HP guarantees in the nonconvex min-max problems is significantly harder than for nonconvex unconstrained minimization due to the necessary time-scale separation between the primal and dual, where primal updates leading to a descent in the primal function can amplify errors in the dual domain if parameters and descent properties are not carefully designed/analyzed [75,6,39].\n\n**3-** In our experiments, both SAPD+ and SMDAVR needed smaller primal and dual step sizes and more tuning of them. Generally, for GDA methods, $\\tau_1$ and $\\tau_2$ must be chosen such that $\\tau_1/\\tau_2=\\Omega(\\kappa)$. For stoc. GDA [39] and stoc. AGDA [6], $\\tau_1/\\tau_2=\\Theta(\\kappa^2)$, but sm-AGDA does not require this thanks to its primal regularization which further allows working with larger primal and dual step sizes of the same order. For example, in sm-AGDA, one can use a primal step size $\\tau_1 = O\\left(\\min\\left(\\frac{1}{\\ell}, \\frac{1}{\\sqrt{T}}\\right)\\right)$ and a dual step size $\\tau_2=\\Theta(\\tau_1)$. These factors contribute to sm-AGDA's practical success and to why it can admit better quantile convergence properties.\n\nReviewer is correct that our quantile bound analysis only applies to sm-AGDA. However, we compare sm-AGDA's concentration properties with SAPD+ and SMDAVR to provide a baseline for sm-AGDA results. SAPD+ and SMDAVR have state-of-the-art complexity bounds in expectation (SAPD+ [70] has better condition number dependency than sm-AGDA [63]). Yet, their quantile bounds have not been studied. We wanted thus to see sm-AGDA's practical performance against these methods.\n\nsm-AGDA is a state-of-the-art method among single-loop algorithms for nonconvex-PL minmax problems in both deterministic [68] and in-expectation performance metrics [63]. For example, sm-AGDA offers better in-expectation guarantees than stochastic (alternating) GDA in terms of the complexity bound's dependence on the condition number $\\kappa$. Single-loop methods like sm-AGDA are favored for their simplicity when compared to multi-loop algorithms, e.g., [70]; hence, they are easy to tune. These factors contribute to why sm-AGDA may achieve better quantile properties than other methods like stoc. GDA or multiple-loop algorithms.\n\n**4-** We addressed the minor notes, thanks for the good catch.\n\n**Questions**\n\n**1-** Indeed, our approach can adapt to other algorithms for nonconvex-PL problems once one defines new stochastic processes $\\tilde A_t, \\tilde B_t, \\tilde C_t, \\tilde D_t$ (as in Cor. 8) and adjusts the potential function. For example, our analysis applies to stoc. AGDA, but results in worse HP bounds compared to sm-AGDA in terms of their dependence to the condition number $\\kappa$, i.e., $\\kappa$ factor in the sm-AGDA complexity becomes $\\kappa^2$ for stoc. AGDA.\n\nWe hope these responses satisfy the reviewer and lead to an improvement in our score in terms of a solid accept.\n\n## References:\n[74] W. Azizian, F. Iutzeler, J. Malick, and P. Mertikopoulos. The rate of convergence of bregman proximal methods: Local geometry versus regularity versus sharpness. SIAM Journal on Optimization, 34(3):2440–2471, 2024.\n\n[75] H. Li, F. Farnia, S. Das, and A. Jadbabaie. On convergence of gradient descent ascent: A tight local analysis. ICML, pages 12717–12740. 2022.\n\n[76] N. Golowich, S. Pattathil, C. Daskalakis, and A. Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. COLT, pp. 1758–1784. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2S3VbJnEv0", "reviewer": "Reviewer_H241", "summary": "This paper considers the important open problem of stochastic smooth nonconvex minimax optimization. This paper proposes single-loop stochastic GDA method, which was known to be practically desirable but had no theoretical complexity compared to other non-single-loop methods with better complexies on nonconvex minimax problems. The analysis in this work fills some of above gap, provides the first high-probability complexity for nonconvex minimax while assuming a PL condition on dual variable, and proves assuming light tailed stochastic gradients GDA converges to a near stationary point with a certain complexity. Numerical results on NCPL game with synthetic data and distributionally-robust optimization with real data shows the proposed sm-AGDA outperforms existing algorithms SAPD+ and SMDAVR.", "questions": "1. What are the existing best-known complexities of (possibly non-single-loop) algorithms on nonconvex minimax problem?\n\n2. Can you compare in more details (overall problem class, overall algorithm, assumptions, complexity order dependencies) of your complexity result compared with [64]? \n\n3. Does each of your assumption (PL condition, etc.) hold for both of your numerical experiments?\n\n**Updates: \n\nThe authors' rebuttal addressed every question very well. I have increased my score from 6 to 7.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper solves the important open problem of stochastic smooth nonconvex minimax optimization, proposes a single-loop GDA method called sm-AGDA, which is constructed by Lyapunov function for nonconvex-concave problems.\n\n2. Assuming PL condition, this paper proved the first high-probability complexity bound on such single-loop algorithm on nonconvex minimax problem. The order of complexity is reasonably good.\n\n3. Numerical experiments show the proposed method is practically superior to existing methods.", "weaknesses": "1. Font size at the end of Page 8, and entire Page 9 are smaller than the required size. \n\n2. There is not much algorithm novelty from [68].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the referee for their time invested in reviewing our paper. Below are our responses to the questions raised.\n\n**Weaknesses**:\n\n**1-** The smaller font size is a typo due to misplaced bracket after a displayed equation. We will fix this issue; thanks for the good catch.\n\n**2-** Our work extends sm-AGDA from [68] by allowing light-tail stochastic estimates $\\widetilde{\\nabla}_x f, \\widetilde{\\nabla}_y f$ in place of the exact partial gradients. This is relevant for large-scale optimization and machine learning where gradients are often estimated from streaming or random data samples. sm-AGDA's convergence in the stochastic setting was first established in expectation in [64]. Our work provides its **first** high-probability (HP) bounds for stochastic nonconvex-PL problems.\n\nA substantial body of work has been dedicated to establishing HP guarantees for first-order methods. These works are often theoretical and involve complex arguments to achieve tight bounds, even in the minimization setting for methods like stochastic gradient descent (SGD) [77,78]. Our paper falls within this line of research.\n\nOur main result (Thm 10) is a tight sampling complexity bound for finding an $\\epsilon$-stationary solution with probability at least $1-q$ on nonconvex-PL problems. Our bounds are tight as when $\\delta=0$ (i.e. gradients estimates are exact), we recover the deterministic complexity of sm-AGDA [68]; and the quantile part of our bound (involving $\\delta^2$) scales favorably with $\\epsilon$ and $\\kappa$. Turning expectation estimates to HP estimates via standard Markov approaches would result in much worse estimates than ours, as we explain in our **General response: Tightness of our high-probability bounds**.\n\nOur key technical contribution lies in identifying stochastic processes ($\\tilde{A}_t, \\tilde{B}_t, \\tilde{C}_t, \\tilde{D}_t$ in Cor 8) with desired concentration properties (Thm 9). Constructing these processes (e.g., $\\tilde{C}_t$) is non-trivial, requiring careful measurability and deriving various inequalities (exploiting smoothness and PL properties) to extend the existing analysis for deriving expectation guarantees [63] and for the deterministic setting [68] of sm-AGDA. For a discussion on related measurability challenges in Lyapunov analysis of momentum-averaged methods, see [34] which obtained HP bounds in the SCSC setting.\n\n**Questions**\n\n**1-** In this paper, we considered stochastic smooth nonconvex-PL (NCPL) problems. For deterministic NCPL problems, AGDA and sm-AGDA have the complexity of $O(\\kappa^2/\\epsilon^2)$ and $O(\\kappa/\\epsilon^2)$ respectively for finding a point $(x,y)$ satisfying $\\\\|\\nabla f(x,y)\\\\|\\leq \\epsilon$ as shown in [68]. Here $\\kappa = L/\\mu$ is the condition number, where $\\ell$ is the Lipschitz constant of the gradient, and $\\mu$ is the PL constant. For Catalyst-AGDA, [64] shows also the rate $O(\\kappa/\\epsilon^2)$ for deterministic NCPL problems. Regarding stochastic NCPL problems, the only existing result holds in expectation by [64]; the authors show that stochastic AGDA and stochastic sm-AGDA have the complexity of $O(\\kappa^4/\\epsilon^4)$ and $O(\\kappa^2/\\epsilon^4)$ respectively for computing a point $(x,y)$ which satisfies $\\mathbb{E}\\\\|\\nabla f(x,y)\\\\|\\leq \\epsilon$.\n\nOur paper provides the first HP results for stochastic NCPL problems, showing that the stochastic sm-AGDA method can compute a point $(x,y)$ that satisfies $\\|\\|\\nabla f(x,y)\\|\\| \\leq \\epsilon$ w.p. at least $1-q$ within $O\\left(\\ell\\kappa^2\\delta^2 \\epsilon^{-4} + \\kappa \\epsilon^{-2} (\\ell + \\delta^2 \\log(1/q))\\right)$ stochastic gradient calls for any $q \\in (0, 1)$.\n\n**2-** In our paper, we consider the same algorithm (sm-AGDA) under the same assumptions (smooth NCPL problems) considered in [64]. Assuming that the variance of the stochastic gradient is bounded by a constant $\\delta^2$, [64] shows a complexity of  $O\\left(\\ell\\kappa^2\\delta^2 \\epsilon^{-4} + \\kappa \\epsilon^{-2} \\ell \\right)$ stochastic gradient calls for computing $(x,y)$ that satisfies $\\mathbb{E}\\\\|\\nabla f(x,y)\\\\|\\leq \\epsilon$. In our work, we make an additional light-tail (subGaussian) assumption on the noise, and obtain a HP result showing that $O\\left(\\ell\\kappa^2\\delta^2\\epsilon^{-4} + \\kappa \\epsilon^{-2} (\\ell + \\delta^2 \\log(1/q))\\right)$ stochastic gradient calls are sufficient to calculate a point $(x,y)$ satisfying $\\|\\|\\nabla f(x,y)\\|\\| \\leq \\epsilon$ w.p. at least $1-q$. Such light-tail assumptions are commonly made for HP results even in convex minimax problems as we discuss in the introduction.\n\nTo our knowledge, this is the first HP result for a nonconvex minimax problem as we highlight in the introduction/Table 1. In particular, one should note that HP guarantees provide a much finer resolution than in-expectation ones, since gradients can be small on average (in expectation), while still being arbitrarily large with some positive probability. Finally, our HP bounds prove to be tight as we recover the in expectation one from [64] using the fact that the expectation of any random variable can be written as the integral of its quantile function from $p=0$ to $p=1$ : $\\mathbb{E}[U] = \\int_{p=0}^1 Q_p(U) dp$.\n\n**3-** Yes. The NCPL assumption holds for the first experiment directly; here the primal problem is indeed non-convex and the dual is PL (and not strongly concave). The second experiment is in the non-convex/strongly concave (NCSC) setting as the regularizer $g(y)$ is strongly convex. Since SC implies the PL property; the second experiment in the NCSC setting is a special case of the NCPL setting. As such, our assumption of NCPL objectives hold for both of our experiments.\n\nWe hope these responses satisfy the reviewer and result in an improved score.\n\n**References** \n\n[77] A.R., O.S., K.S. Making gradient descent optimal for strongly convex stochastic optimization. ICML 2012.\n\n[78] N. H., et al. \"Tight analyses for non-smooth stochastic gradient descent.\" COLT 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4S3u8d9jC1", "reviewer": "Reviewer_Zwn2", "summary": "The authors build upon earlier works to prove a high probability upper bound, that is linear in variance of the stochastic gradients, on the number of stochastic gradient calls of smoothed alternating GDA method.", "questions": "see weaknesses", "limitations": "Theory paper, no concerns", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written with clear theorems, proofs and numerical results.", "weaknesses": "Focusing only on their concentration bound, which they claim is of independent interest and is the crux of their high probability bounds: the bound in Theorem 9 (which is fairly customized and not easy to state) follows like any other chernoff/chebychev type bound - using Markov type inequality after exponentiation; and I don't see any novelty in their theoretical ideas/techniques here. Perhaps I have missed something, if so the authors can clarify it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the referee for their time invested in our paper. Below are our responses to the weaknesses raised.\n\nWhile our concentration inequality (Thm 9) seems tailored to the analysis of sm-AGDA, we can argue that it can also aid in deriving high probability bounds for many other nonconvex first-order methods that outputs a randomized iterate. Indeed, most Lyapunov arguments in the nonconvex setting are built upon telescoping quantities in line with our Thm 9. This includes stochastic alternating GDA for NCPL minimax problems and optimistic GDA [34] for strongly convex-strongly concave problems. The challenge lies in finding the appropriate stochastic processes $(\\tilde{A}_t, \\tilde{B}_t, \\tilde{C}_t, \\tilde{D}_t)$, as done in Cor 8 for sm-AGDA. This is akin to prime factorization—easy to verify but hard to come up with. Thus, our significant technical novelty is devising these processes for sm-AGDA such that the concentration result in Thm 9 applies. We mentioned Thm 9 to be of independent interest, not because it is particularly hard to prove but because it acts as a useful inequality potentially applicable to other algorithms like stochastic AGDA and stochastic gradient descent (SGD). \n\nFor example, stochastic AGDA is a special case of the sm-AGDA algorithm we analyzed, obtained by setting the parameters $\\beta=p=0$. Thm 9 leads to a new complexity result of  $O(\\ell \\kappa^2 / \\epsilon^2 + \\delta^2 \\log(1/q) \\kappa^2 / \\epsilon^2 + \\ell \\kappa^4 \\sigma^2 / \\epsilon^4)$ for stochastic AGDA, if we were to apply Thm 9 to processes $\\tilde A_t = \\tau_1 V_t, \\tilde B_t = \\frac{\\tau_1}{32} \\|\\| \\nabla \\Phi(x_t) \\|\\|^2 + \\frac{\\tau_2}{64} \\|\\| \\nabla f(x_t, y_t)\\|\\|, \\tilde D_t = \\kappa \\ell\\tau_1^2\\|\\| \\Delta_t^x \\|\\|^2 + \\ell\\tau_2^2 \\|\\| \\Delta_t^y \\|\\|^2/8$ together with an appropriately modified Lyapunov function $V_t$ and $\\tilde C_t$ term. Similarly, by adapting these processes and by removing the dual terms that relate to $y$ variable, we can obtain complexity results for SGD.\n\nThanks to the novelty of our approach, we can also obtain tight bounds as we discuss next. Specifically, our complexity bound for sm-AGDA for finding an $\\epsilon$-stationary solution w.p. at least $1-q$ is $O \\left(\\ell \\kappa/\\epsilon^2 + \\delta^2 \\log\\left(1/q\\right) \\kappa/\\epsilon^2 + \\delta^2 \\ell \\kappa^2/\\epsilon^4\\right),$ where $\\kappa = \\ell/\\mu$ is the condition number, and $\\delta^2$ is the subGaussian proxy tied to the stochastic gradient estimates $\\widetilde{\\nabla}_x f, \\widetilde{\\nabla}_y f$. These bounds are tight in the sense that when $\\delta=0$, we recover the deterministic $O(\\kappa/\\epsilon^2)$ complexity for sm-AGDA; and the quantile term scales favorably with $\\epsilon$ and $q$. One may expect the iterates to have light-tail properties when the gradient noise is light-tailed, but quantifying the proxy/variance of the iterates is not straightforward since it depends on various algorithm and problem parameters such as $\\mu,\\ell,\\tau_1,\\tau_2,\\beta$, etc. Furthermore, turning in-expectation estimates to high-probability (HP) estimates via Markov inequality would result in much worse estimates than ours, as we explain in the window titled **General response: Tightness of our high-probability bounds**. As we discuss in this window, a surprising result of our analysis, contrary to advanced approaches leveraging Markov's inequality by running the method several times in parallel, is that the cost of getting an HP bound only affects negligible terms of the initial expectation bounds giving $O(\\epsilon^{-4}+\\log(1/q)\\epsilon^{-2})$ complexity, instead of multiplying the whole in-expectation bound by $\\log(1/q)$ in which case one gets a much worse $O(\\log(1/q)\\epsilon^{-4})$ complexity. This feature departs from standard guarantees developed for the last iterate in the SCSC setting [79,34]. \n\nOur bounds achieve good complexity by striking the right balance between the bias and variance/quantile terms in the complexity. This is accomplished by carefully defining the stochastic processes $\\tilde{A}_t, \\tilde{B}_t, \\tilde{C}_t, \\tilde{D}_t$ in our Cor 8 that exhibit the desired concentration properties as stated in our generic concentration result (Thm 9). In particular, these processes depend on the parameters $\\tau_1, \\tau_2,$ and $\\beta$ in non-trivial ways, requiring parameter choice/design optimization to achieve our rate results. Naive choices would result in a much worse complexity bound in terms of its dependence on $\\epsilon, \\kappa, $ and $q$. In addition, constructing the processes $\\tilde{A}_t, \\tilde{B}_t, \\tilde{C}_t, \\tilde{D}_t$ requires careful measurability considerations along with deriving various inequalities (that carefully exploit the smoothness and PL properties of the objective) to extend the existing analysis for deriving expectation guarantees [63] and for the deterministic setting [68] of sm-AGDA. For a discussion on the challenges related to measurability in the Lyapunov analysis of momentum-averaged algorithms, we refer to [34], which obtained HP bounds in the strongly convex-strongly concave (SCSC) setting. Similarly, sm-AGDA employs momentum averaging using the parameter $\\beta$ in the nonconvex setting and encounters comparable measurability issues. These challenges are specific to HP analyses and do not occur in the less sensitive expectation bounds from [64], where noise can be mitigated using the properties of conditional expectation. Therefore, substantial amount of new work is in fact needed to obtain our results. \n\nTo summarize, our bounds are highly non-trivial requiring significant novelty, where we get the state-of-the-art complexity for nonconvex/PL minimax problems by far. We would appreciate if the referee would consider raising their score.\n\n**References:**\n\n[79] Cutler, Drusvyatskiy, and Harchaoui. Stochastic optimization under time drift: iterate averaging, step-decay schedules, and high probability guarantees. NeurIPS, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JgSYljZz1P", "reviewer": "Reviewer_vFP9", "summary": "This paper investigates the stochastic non-convex minimax optimization problem $\\min_x \\max_y f(x,y)$ where the function $f$ is smooth, then nonconvex in $x$ and PL in $y$. The authors presented a high probability analysis for the smoothed alternating gradient descent ascent (sm-AGDA) method. This analysis is based on a Lyapunov function from prior work for non-convex concave setting. Also, under a light-tail assumption on the gradient noise, the authors developed a concentration result. Finally, they include experiments of nonconvex-PL setting with synthetic data and then distributionally robust optimization problems with real data.", "questions": "Why in the experiments the authors compare with SAPD+ and SMDAVR, not the methods in Table 1?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors provided theoretical proofs and experiments to support their claims. The analysis seems to be solid. The authors compared their complexity with other high probability bounds results.", "weaknesses": "1. While the authors cited some results in expectation for the similar setting, I think it is important to compare the (Markov) high probability bounds obtained by these expectation results with the results obtained in this paper. If the results are not better, then the contribution is limited. \n\nAlso I don't get what the authors meant by \"More precisely, we focus on a purely stochastic regime in which data streams over time which renders the use of mini-batch schemes or running the method in parallel impractical; therefore, approaches based on Markov’s inequality [59] are no longer applicable.\" (line 99-101). \n\n2. In the theorems and lemma, the values $\\tau_1, \\tau_2, \\tau, \\bar{\\tau}$ is really confusing. Do you need to pick specific values for the step sizes, if so, what is $\\tau_2$? Are $\\tau$ and $\\bar{\\tau}, \\tau_1$ the same?\n\nUpdate: I thank the authors for the rebuttal. I keep my recommendation the same, as changing to high probability bound from expectation results is somewhat expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the referee for their time invested in reviewing our paper. Below are our responses to the questions raised.\n\n**Weaknesses**\n\n**1-** Please see our general response to the reviewers above in the window titled **General response: Tightness of our high-probability bounds,** where we explain why our bounds are significantly tighter than any naive bound based on applying Markov's inequality to the existing results in expectation. As we discuss in detail there, applying Markov's inequality yields significantly loose bounds in terms of their dependence on the quantile parameter $q \\in (0,1)$. \n\nA more advanced approach - see e.g. [19,59] - involves running $O(\\log(1/q))$ copies of the sm-AGDA algorithm together with a postprocessing step requiring additional $\\mathcal{O}(1/\\epsilon)$ samples for each copy to identify an $\\epsilon$-stationary point with high probability. \nThe overall cost of this approach is of order $O(\\log(1/q)) \\cdot \\mathcal G(\\epsilon) + O(\\log(1/q)) \\cdot O(1/\\epsilon) $, where $\\mathcal G(\\epsilon)$ denotes the complexity in expectation, assuming stochastic gradients have a variance bounded by $\\sigma^2$. This results in an unavoidable dependence in the failure probability $q$ of order $\\log(1/q) \\epsilon^{-4}$ which is significantly worse than our $\\log(1/q) \\epsilon^{-2}$ dependance.\n\nMoreover, approaches like [19,59] can remain interesting in practice when one is allowed to have the $O(\\log(1/q))$ runs of sm-AGDA processed in parallel, but loses all interest with respect to our approach when parallelization is impractical or impossible.  Indeed, the approaches that require $O(\\log(1/q))$ parallel runs can be impractical for the streaming data setting we consider where the data arrives one by one in a streaming fashion; the aforementioned impracticality is mainly because of the fact that the parallel runs would have to wait for the arrival of  new data points to be able take one step. This is what we meant to convey in our sentence \"*More precisely, we focus on a purely stochastic regime in which data streams over time, rendering the use of mini-batch schemes or running the method in parallel impractical, therefore ... [59] are no longer applicable*\". In the revised version, we clarified this sentence and added comparisons with naive approaches based on the Markov inequality.\n\n**2-** While $\\tau_1$ and $\\tau_2$ are primal and dual stepsizes, $\\bar{\\tau}$ corresponds to a particular choice of the primal stepsize where we set $\\tau_1=\\bar{\\tau}$ in our main complexity result. To avoid confusion, we will remove $\\bar{\\tau}$ and use only $\\tau_1$ instead. The symbol $\\tau$ is used in our concentration inequality, to emphasize that it can in principle apply to any positive scalar (not necessarily the primal stepsize); however in practice we choose $\\tau=\\tau_1$. Therefore, to simplify the notation we will also replace $\\tau$ with $\\tau_1$ in the revised version.\n\n**Questions**\n\n**1-** The methods provided in the table are mostly for convex/concave problems and do not provide any guarantees for the stochastic nonconvex/strongly convex (NCSC) problem we consider in this experiment. There are also some VI results in the table, but they do not apply to NCSC problems either. The point of the table was to summarize the existing HP results and emphasize that no HP bounds were known for NCSC problems prior to our paper. SAPD+ and SMDAVR algorithms have in expectation guarantees for NCSC problems, but they do not admit HP guarantees. SAPD+ and SMDAVR were also among the most competitive ones tested on this example in [70]. This is the main reason why we compared sm-AGDA with these two algorithms even if they do not admit HP guarantees.\n\nWe hope these responses satisfy the reviewer and lead to an improvement in our score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l7tD38M5IE", "reviewer": "Reviewer_pQzv", "summary": "This paper proves a high probability convergence bound and almost sure convergence for stochastic smoothed AGDA method for minimax problems in nonconvex-PL setting. This is the first high probability guarantee and almost sure convergence guarantee in this nonconvex setting.", "questions": "1. The algorithm and analysis are built upon the approaches in 63 and 68. Could the authors provide more insights on the novelty of the proof compared to 63 and 68?\n2. Is it possible to extend this analysis to other settings like the nonconvex-concave setting?\n3. Can this analysis be extended to other algorithms such as stochastic GDA?\n4. Missing references: \n[1]. Shen et al   Stochastic gradient descent ascent for federated minimax optimization\n[2]  Zheng et al Universal gradient descent ascent method for nonconvex-nonconcave minimax optimization \n[3].  Ozdaglar et al What is a good metric to study the generalization of minimax learners\n[4] Li et al Nonsmooth-nonconvex-nonconcave minimax optimization: primal-dual balancing and iteration complexity analysis", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is the first high probability convergence guarantee and almost sure convergence guarantee in this nonconvex setting, going beyond convex setting and convergence in expectation.\nThe paper also presents a concentration bound (Theorem 9) which is of independent interests and this bound allows us to obtain high probability convergence results from expectation convergence results for minimax problems.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all the insightful comments. Below are our responses:\n\n**1-** [68] analyzes sm-AGDA in the deterministic nonconvex-concave setting, and introduces a Lyapunov function to establish a complexity result. In a follow-up work, [63] leverages the same Lyapunov function to adapt the analysis to the stochastic nonconvex-PL setting, and establishes $O(\\ell\\kappa / \\epsilon^2 + \\ell\\kappa^2 \\sigma^2 / \\epsilon^4)$ complexity; $\\kappa$ is the condition number, and $\\sigma^2$ bounds the variance.\n\nSome  Lyapunov arguments in our proof are built upon these analyses to establish our high-probability (HP) bound. In fact, it is a common strategy in the literature on HP bounds to build upon existing deterministic/expectation analyses [81,82]. Yet the key difficulty remains in controlling how the noise from gradients accumulates along iterations. While one may intuitively expect that the gradient estimates being subGaussian should produce a light-tail distribution for sm-AGDA’s output, achieving a precise balance between bias and quantile trade-off in the complexity bound usually turns out to be highly nontrivial – even in the strongly convex-strongly concave (SCSC) setting [34].\n\nIn the nonconvex-PL case, we show that for finding an $\\epsilon$-solution w.p. at least $1 - q$, the complexity of sm-AGDA is $O(\\ell\\kappa / \\epsilon^2 + \\delta^2 \\log(1/q) \\kappa / \\epsilon^2 + \\ell\\kappa^2 \\delta^2 / \\epsilon^4)$ under the subGaussian assumption on the gradients. This bound is tight: for the deterministic case, i.e., when $\\delta = 0$, we recover the existing $O(\\kappa / \\epsilon^2)$ complexity bound for sm-AGDA; and the quantile dependence of our bounds scales favorably with $\\epsilon$ and $q$. Our bound is tighter than those obtained by the standard expectation to HP conversions, as we explain in our general response to the reviewers above.\n\nOur main technical novelty behind this result is the identification of stochastic processes (see $ \\tilde{A}_t, \\tilde{B}_t, \\tilde{C}_t,$ and $\\tilde{D}_t$ in Cor. 8) that exhibit the desired concentration properties as stated in our generic concentration result (Thm. 9). Constructing these processes, e.g., $\\tilde{C}_t$, is non-trivial, requiring careful measurability considerations along with deriving some new convex inequalities to extend the existing analysis for deriving expectation guarantees [63] and for the deterministic setting [68] of sm-AGDA. These challenges are specific to HP analyses and do not occur in the less sensitive expectation bounds from [63], where noise can be mitigated using the properties of conditional expectation. For a discussion on the challenges related to measurability in the Lyapunov analysis of momentum-averaged algorithms, we refer to [34], which obtained HP bounds in the SCSC setting. Similarly, sm-AGDA employs momentum averaging in the non-convex setting whenever $\\beta > 0$, which complicates the analysis with measurability issues comparable to those arising in [34].\n\nFinally, a surprising result of our analysis, contrary to naive approaches leveraging Markov’s inequality by running the method several times in parallel, is that the cost of getting a HP bound only affects negligible terms of the initial expectation bounds giving $ O(\\epsilon^{-4} + \\log(1/q) \\epsilon^{-2}) $ complexity, instead of multiplying the whole bound by $\\log(1/q)$ to get a worse bound of $O(\\log(1/q) \\epsilon^{-4})$ - see our general response in that regard. This feature departs from standard guarantees in the SCSC setting [34].\n\n**2-** In our proof, we use the PL property of the dual to achieve an appropriate Lyapunov descent using stoc. gradients. It is not clear whether our proof directly extends to the nonconvex-concave setting. The only convergence analysis we know for sm-AGDA in the nonconvex-concave setting is given in [68]; but, this analysis is given only for the deterministic case and requires the compactness of the dual domain. Yet, we believe that combining our analysis techniques with some deterministic inequalities from [68] may allow to extend our results to the nonconvex-concave setting. We will consider this as part of future work as it would require a detailed analysis with some fundamental differences from the current work.\n\n**3-** In stoc. GDA, the primal and dual updates can be implemented either in an alternating fashion or in a Jacobi style where both updates are performed simultaneously. The alternating case is referred to as stoc. alternating GDA (AGDA). By slightly modifying our proof, we can show a similar HP result holds for stoc. AGDA, with a degraded complexity of $O(\\ell \\kappa^2 / \\epsilon^2 + \\delta^2 \\log(1/q) \\kappa^2 / \\epsilon^2 + \\ell \\kappa^4 \\delta^2 / \\epsilon^4)$. Our stoc. AGDA’s complexity has extra $\\kappa$ factors on the bias and variance/quantile error parts, respectively, when compared to sm-AGDA bound we derive in our paper. To obtain the stoc. AGDA bound, we apply Thm. 9 to the modified stochastic processes: $\\tilde A_t = \\tau_1 V_t, \\tilde B_t = \\frac{\\tau_1}{32} \\|\\| \\nabla \\Phi(x_t) \\|\\|^2 + \\frac{\\tau_2}{64} \\|\\| \\nabla f(x_t, y_t)\\|\\|, \\tilde D_t = \\kappa \\ell\\tau_1^2\\|\\| \\Delta_t^x \\|\\|^2 + \\ell\\tau_2^2 \\|\\| \\Delta_t^y \\|\\|^2/8$,\n$$\\tilde C_t = \\left\\langle \\left( (1 + \\alpha) \\frac{2\\ell + \\ell \\kappa}{2} \\tau_1^2 - \\alpha \\left(\\tau_1 + l \\tau_1^2 - \\tau_2 \\ell^2 \\tau_1^2 \\right) \\nabla_x f(x_t, y_t) - (1 + \\alpha) \\tau_1 \\nabla \\Phi(x_t) \\right), \\Delta_t^x \\right\\rangle - \\left\\langle \\alpha (\\tau_2 - \\ell \\tau_2^2) \\nabla_y f(x_t, y_t), \\Delta_t^y \\right\\rangle, \n$$\nwhere $V_t = (1 + \\alpha) \\Phi(x_t) - \\alpha f(x_t, y_t)$ is a Lyapunov function, $\\alpha$ is a particular constant. Jacobi-style stoc. GDA is known to be divergent even in the convex-concave setting [80], so we did not consider it. We also added the missing references.\n\n**References**\n\n[80] Zhang, Wang, Lessard, Grosse. AISTATS 2022.\n\n[81] Harvey, Liaw, Plan, Randhava, COLT 2019.\n\n[82] Rakhlin, Shamir, Sridharan, ICML 2012.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### General response: Tightness of our high-probability bounds

We thank the referees for their time invested in reviewing our paper. Following the request from several referees, we discuss below the benefit of our approach against naive expectation-to-high-probability conversions using Markov's inequality.

Our bounds are significantly better than the simple high-probability bounds one can obtain using the existing results in terms of expectation; this point is explained in detail below. For the nonconvex-PL setting considered in our paper, the only expectation result we are aware of is by [64]. Assuming the variance of the stochastic gradient is bounded by $\delta^2$, for sm-AGDA the authors show a complexity of $\mathcal{G}(\epsilon):=O\left(\frac{\ell\kappa^2\delta^2}{\epsilon^4} + \frac{\kappa}{\epsilon^2}\ell \right)$ iterations/stochastic samples for computing an $\epsilon$-stationary solution in expectation, i.e., for computing $(x,y)$ that satisfies $\mathbb{E}\\|\nabla f(x,y)\\|\leq \epsilon$. Here, $\mu$ is the PL constant, $\ell$ is the gradient Lipschitz constant and $\kappa=\ell/\mu$. This result basically acts as an oracle that generates a sample $(\hat x,\hat y)$ satisfying $\mathbb{E}\\|\nabla f(\hat x,\hat y)\\| \leq \epsilon$ after $\mathcal{G}(\epsilon)$ iterations. We next present some naive approaches using the Markov inequality to obtain trivial high-probability bounds that hold w.p. at least $1-q$ for any given $q\in (0,1)$:

- The first approach involves applying the oracle to generate a solution $(\hat x,\hat y)$ such that $\mathbb{E}[\\|\nabla f(\hat{x}, \hat{y})\\|] \leq q \cdot \epsilon$. Then by naively using Markov's inequality, we get an $\epsilon$-stationary point w.p. at least $1-q$. The complexity bound for this approach is $\mathcal{G}(q\epsilon) = O\left(\frac{\ell\kappa^2\delta^2}{q^4\epsilon^4} + \frac{\kappa}{\epsilon^2q^2}\ell \right)$ and it scales badly with $q$.
- The second approach involves calling the oracle $m$ times to generate a group of $\epsilon/2$-stationary solutions $\\{(\hat x^{(i)}, \hat y^{(i)})\\}_{i=1}^m$. If the performance metric $\\| \nabla f (\hat x^{(i)}, \hat y^{(i)})\\|$ can be evaluated to $\epsilon$-accuracy for all $i\in\{1,\ldots,m\}$, i.e., for each $i$, we get a stochastic estimate $\tilde\nabla{f (\hat x^{(i)}, \hat y^{(i)})}$ such that $\\|\tilde\nabla{f (\hat x^{(i)}, \hat y^{(i)})}-\nabla{f (\hat x^{(i)}, \hat y^{(i)})}\\| \leq \epsilon$ with high probability, then the point $(\hat x^{(i^*)}, \hat y^{(i^*)})$ such that $i^* = \arg\min\\{\\|\tilde\nabla{f (\hat x^{(i)}, \hat y^{(i)})}\\| : i=1,\ldots, m\\}$ will satisfy the desired high-probability bound as long as $m = \Omega \left( \log \frac{1}{q} \right)$. This approach resolves the unfavorable dependence on $q$ present in the first approach. However, evaluating the stochastic estimate $\tilde\nabla{f (\hat x^{(i)}, \hat y^{(i)})}$ with the aforementioned properties for each $i$ typically requires $O(1/\epsilon^2)$ samples if light-tail assumptions are not made for the stochastic gradients [36, 59], [Thm 2.4,19] (using Markov inequality-type bounds or standard concentration inequalities such as [Lem 2.3,19]); on the other hand, the number of samples required for each $i$ is $O(\delta^2/\epsilon)$ if light-tail (subGaussian) assumptions are made for the stochastic gradients ignoring some logarithmic factors [Cor. 2.5,19]. In our paper, we consider general problems *without* assuming a finite-sum form for the objective and the data can be arriving in a streaming fashion. Therefore, the total complexity with the second approach when applied to the problems we consider becomes at least $\log\left(\frac{1}{q}\right) \mathcal{G}(\epsilon) + \log\left(\frac{1}{q}\right) O\left(\frac{\delta^2}{\epsilon}\right) = O\left(\log\left(\frac{1}{q}\right)\frac{\ell\kappa^2\delta^2}{\epsilon^4} + \log\left(\frac{1}{q}\right)\frac{\kappa\ell}{\epsilon^2} + \log\left(\frac{1}{q}\right)\frac{\delta^2}{\epsilon}\right)$.
- For unconstrained strongly convex-strongly concave (SCSC) problems, one can use the robust distance estimation technique together with also $m = \Omega \left( \log \frac{1}{q} \right)$ parallel runs; however, this approach does not apply to nonconvex minimax problems we consider in this paper [Sec 2,36].

Our high-probability result shows a complexity of $O\left(\frac{\ell\kappa^2\delta^2}{\epsilon^4} + \frac{\kappa}{\epsilon^2}\Big(\ell + \delta^2 \log(\frac{1}{q})\Big)\right)$ under the light-tail (subGaussian) assumption. This result is significantly better than the approaches mentioned above; indeed, it has not only the best (logarithmic) scaling with respect to $q$ but also with respect to $\epsilon$. In the second approach, the logarithmic term $\log(1/q)$ multiplies the high-order $O(1/\epsilon^4)$ term, whereas in our approach it only affects the second-order $O(1/\epsilon^2)$ term. More importantly, the approaches that require $\log(1/q)$ parallel runs can be impractical for the streaming data setting we consider where the data arrives one by one in a streaming fashion; the aforementioned impracticality is mainly because of the fact that the parallel runs would have to wait for the arrival of $\Omega(\log(1/q))$ new data points to be able take one step.

Meta-review: This paper addresses the high probability convergence of the Gradient Descent-Ascent-type method for stochastic nonconvex minimax optimization. The problem studied is PL in the max variable and hence the inner problem is computationally "easy" to solve. Having easy inner problem and working with the dual function makes the problem easier than some other settings to analyze (in min-max optimization). By deriving high probability bounds, the paper extend previous results that only provided bounds in expectation. The analysis also introduces a new concentration bound  that might be of independent interest.  The paper includes numerical experiments comparing their studied algorithms with existing methods. 

Some reviewers found the theoretical techniques (particularly the concentration bounds) not significantly novel, as it follows existing bounds/techniques. Some reviewers believe that the paper could benefit from a deeper comparison with related works, especially regarding the differences between the current and prior analyses.  There were also some concern that the algorithmic contributions have limited insight into new theoretical developments beyond those already present in the literature.

Overall, the paper is borderline. While the paper makes contributions to the theoretical understanding of GDA-type algorithms for min-max optimization, addressing the above concerns could strengthen its impact.
Decision: Accept (poster)