Title: High-probability complexity bounds for stochastic non-convex minimax optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: nonconvex minimax optimization

high-probability guarantees

stochastic gradient descent ascent methods
Abstract: Stochastic smooth nonconvex minimax problems are prevalent in machine learning, e.g., GAN training, fair classification, and distributionally robust learning. Stochastic gradient descent ascent (GDA)-type methods are popular in practice due to their simplicity and single-loop nature. However, there is a significant gap between the theory and practice regarding high-probability complexity guarantees for these methods on stochastic nonconvex minimax problems. Existing high-probability bounds for GDA-type single-loop methods only apply to convex/concave minimax problems and to particular non-monotone variational inequality problems under some restrictive assumptions. In this work, we address this gap by providing the first high-probability complexity guarantees for nonconvex/PL minimax problems corresponding to a smooth function that satisfies the PL-condition in the dual variable. Specifically, we show that when the stochastic gradients are light-tailed, the smoothed alternating GDA method can compute an $\varepsilon$-stationary point within $\mathcal{O}(\frac{\ell \kappa^2 \delta^2}{\varepsilon^4} + \frac{\kappa}{\varepsilon^2}(\ell+\delta^2\log({1}/{\bar{q}})))$ stochastic gradient calls with probability at least $1-\bar{q}$ for any $\bar{q}\in(0,1)$, where $\mu$ is the PL constant, $\ell$ is the Lipschitz constant of the gradient, $\kappa=\ell/\mu$ is the condition number, and $\delta^2$ denotes a bound on the variance of stochastic gradients. We also present numerical results on a nonconvex/PL problem with synthetic data and on distributionally robust optimization problems with real data, illustrating our theoretical findings.

High-probability complexity guarantees for nonconvex minimax problems

Abstract

Stochastic smooth nonconvex minimax problems are prevalent in machine learning, e.g., GAN training, fair classification, and distributionally robust learning. Stochastic gradient descent ascent (GDA)-type methods are popular in practice due to their simplicity and single-loop nature. However, there is a significant gap between the theory and practice regarding high-probability complexity guarantees for these methods on stochastic nonconvex minimax problems. Existing high-probability bounds for GDA-type single-loop methods only apply to convex/concave minimax problems and to particular non-monotone variational inequality problems under some restrictive assumptions. In this work, we address this gap by providing the first high-probability complexity guarantees for nonconvex/PL minimax problems corresponding to a smooth function that satisfies the PL-condition in the dual variable. Specifically, we show that when the stochastic gradients are light-tailed, the smoothed alternating GDA method can compute an ε -stationary point within O ( ℓκ 2 δ 2 ε 4 + κ ε 2 ( ℓ + δ 2 log(1 / ¯ q ))) stochastic gradient calls with probability at least 1 -¯ q for any ¯ q ∈ (0 , 1) , where µ is the PL constant, ℓ is the Lipschitz constant of the gradient, κ = ℓ/µ is the condition number, and δ 2 denotes a bound on the variance of stochastic gradients. We also present numerical results on a nonconvex/PL problem with synthetic data and on distributionally robust optimization problems with real data, illustrating our theoretical findings.

1 Introduction

Minimax optimization problems arise frequently in machine learning (ML) applications; indeed, constrained optimization problems such as deep learning with model constraints [24], dictionary learning [10, 57] or matrix completion [28] can be recast as a minimax optimization problem through Lagrangian duality. Other applications include but are not limited to the training of GANs [59], fair learning [72], supervised learning [61, 49, 51, 74], adversarial deep learning [75], game theory [54, 58], robust optimization [4, 3], distributionally robust learning [46, 75, 24], meta-learning [70] and multi-agent reinforcement learning [15]. Many of these applications can be reformulated in

Yasa Syed

Mert Gürbüzbalaban

the following minimax form:

where f : R d 1 × R d 2 → R is a smooth function, i.e., differentiable with a Lipschitz gradient; f can possibly be nonconvex in x and nonconcave in y . First-order primal-dual (FOPD) methods have been the leading computational approach for computing low-to-medium-accuracy stationary points for these problems because of their cheap iterations and mild dependence of their overall complexities on the problem dimension and data size [9, 8, 36]. In the context of FOPD methods, there are two key settings for (1):

(i) the deterministic setting, where the partial gradients ∇ x f and ∇ y f are exactly available,

(ii) the stochastic setting, where we have only access to (inexact) stochastic estimates of the partial gradients, in which case the problem in (1) is called a stochastic minimax problem .

It can be argued that the stochastic setting is more relevant to modern machine learning applications where gradients are typically estimated randomly from mini-batches of data, or sometimes intentionally perturbed with random noise to ensure data privacy [14, 34, 1].

Convex and nonconvex minimax optimization. In the convex case (when f is convex 2 in x and concave in y ), several approaches have been considered including Variational Inequalities (VIs) and primal-dual algorithms, see. e.g. [29, 20, 5, 67, 12, 73, 60, 11] and the references therein. One disadvantage of using the VI approach for solving minimax problems (by identifying the signed gradient map G ( x, y ) ≜ [ ∇ x f ( x, y ) ⊤ , -∇ y f ( x, y ) ⊤ ] ⊤ as the corresponding operator in the VI) is that one needs to set the primal and dual stepsize to be the same. This can be restrictive in applications where f exhibits different smoothness properties in the primal ( x ) and dual ( y ) block coordinates -this is often the case in distributionally robust learning [73], adversarial learning [43] and in the Lagrangian reformulations of constrained optimization problems that involve many constraints [47]. The gap function G ( x k , y k ) ≜ sup x,y f ( x k , y ) -f ( x, y k ) , and the squared distance to the set of saddle points D ( x k , y k ) ≜ min {∥ x k -x ⋆ ∥ 2 + ∥ y k -y ⋆ ∥ 2 | ( x ⋆ , y ⋆ ) is a saddle point } are standard metrics for assessing the quality of the output z k = ( x k , y k ) generated by an FOPD algorithm after k iterations among many others [35, 73, 20].

In the nonconvex setting, i.e., when f is nonconvex in x , the aim is to compute a stationary point. Let M ( x k , y k ) denote a measure for the stationarity of iterates ( x k , y k ) ; a common metric is the norm of the gradient, i.e., M ( x k , y k ) ≜ ∥∇ f ( x k , y k ) ∥ and its variants such as ∥ ∇ Φ( x k ) ∥ when f is strongly concave in y , where Φ( · ) = max y f ( · , y ) denotes the primal function -for other metrics and relation between them, see [62]. There are several algorithms that admit (gradient) complexity guarantees for computing a stationary point of nonconvex minimax problems under various strong concavity, concavity or weak concavity-type assumptions in the y variable -see the references in [62].

In this paper, we consider smooth nonconvex-PL (NCPL) problems where f is a smooth function such that it is possibly nonconvex in x and it satisfies the Polyak-Lojasiewicz (PL) condition in y . The PL condition is a weaker assumption (milder condition) than strong concavity in y -in fact, PL condition in the dual does not even require quasi-concavity. NCPL problems constitute a rich class of problems arising in many ML applications including but not limited to fair classification [48], robust neural network training with dual regularization [48, eqn. (14)], overparametrized systems and neural networks [42], linear quadratic regulators [17], smoothed Lasso problems [23] subject to constraints, distributionally robust learning with ℓ 2 regularization in the dual [72], deep AUC maximization [68] and covariance matrix learning with Wasserstein GANs [52]. For deterministic NCPL problems, the alternating gradient descent ascent (AGDA) method and its smoothed version (smoothed AGDA) have the complexity of O ( κ 2 /ϵ 2 ) and O ( κ/ϵ 2 ) , respectively, for finding a point (˜ x, ˜ y ) satisfying ∥∇ f (˜ x, ˜ y ) ∥ ≤ ϵ as shown in [70, 66]. Here κ ≜ ℓ/µ is the condition number, where ℓ is the Lipschitz constant of the gradient, and µ is the PL constant. For Catalyst-AGDA, [66] shows also the rate O ( κ/ϵ 2 ) for deterministic NCPL problems.

In expectation and high-probability bounds. Most of the existing guarantees in the literature for stochastic FOPD algorithms are provided in expectation, i.e., a bound on the number of iterations k (or the stochastic gradient evaluations) is provided for E [ G ( x k , y k )] ≤ ε or E [ D ( x k , y k )] ≤ ε to hold (see, e.g., [73, 72, 29] and the references therein). Yet having such guarantees on average does

not allow to control tail events, i.e., even if E [ G ( x k , y k )] is small, G ( x k , y k ) can still be arbitrarily large with a non-zero probability. To this end, high-probability guarantees have been considered in the literature [35, 64, 20, 32, 22]. These results allow to control the risk associated with the worst-case tail events as they specify how many iterations would be sufficient to ensure G ( x k , y k ) is sufficiently small for any given failure probability ¯ q ∈ (0 , 1) . To derive high-probability bounds, one common approach involves running the algorithm in parallel multiple times and strategically selecting an optimal output to convert in-expectation bounds into high-probability guarantees [62, 37]. Alternatively, advanced concentration inequalities can be employed under light-tail assumptions to control noise accumulation across iterates without requiring multiple runs [53, 27]. For saddle point problems, we note that existing high-probability bounds mostly apply to the monotone VI setting, or to strongly convex/strongly concave (SCSC) minimax problems. To our knowledge, high-probability guarantees for nonconvex minimax problems are non-existent in the literature, even for nonconvex/strongly concave (NCSC) problems with the exception of trivial loose bounds one can obtain by a standard application of Markov's inequality (see Remark 13 for details). In particular, we note that the existing VI literature with high-probability bounds on non-monotone operators such as star-co-coercive operators [20, 55], do not apply to NCSC problems. 3

New high-probability bounds for NCPL optimization. To address these shortcomings, we focus on developing high-probability guarantees for NCPL problems. Among the existing algorithms in the stochastic NCPL setting [64], stochastic gradient descent ascent (SGDA) methods and their variants are quite popular for ML applications, e.g., training GANs and adversarial learning, as SGDA is easy to implement due to its single-loop structure. Guarantees in expectation for stochastic NCSC problems are well supported by the literature - see [72, 63, 40, 6, 31, 30, 44, 66, 33, 39] and the references therein. To our knowledge, among single-loop methods for NCPL problems, the best guarantees in expectation are given by the smoothed alternating gradient descent ascent ( sm-AGDA ) method [66], which can compute an almost stationary point (˜ x, ˜ y ) satisfying E [ ∥∇ f (˜ x, ˜ y )] ∥ ≤ ε in O ( ℓκ 2 δ 2 /ε 4 + ℓκ/ε 2 ) stochastic gradient calls, where δ 2 is an upper bound on the variance of the stochastic gradients. In this work, we consider the sm-AGDA algorithm, and to our knowledge we provide the first-time high-probability bounds (using a single-loop method that does not resort to restarts and parallel runs) for the minimax problem (1) in the NCSC and NCPL settings. More precisely, we focus on a purely stochastic regime in which data streams over time which renders the use of mini-batch schemes or running the method in parallel impractical; therefore, approaches based on Markov's inequality [62] are no longer applicable (see also Remark 13).

Contributions. Our contributions are threefold:

We present the first high-probability complexity result for the sm-AGDA algorithm in the NCPL setting by building upon a Lyapunov function first introduced in [70] for nonconvex-concave problems. Later, for the same Lyapunov function, state-of-the-art complexity bounds in expectation are provided for the NCPL setting in [66]. In this paper, we derive a novel descent property for this Lyapunov function in the almost sure sense (Theorem 7 and Corollary 8), allowing us to develop useful concentration arguments for it to derive high-probability bounds. Our Lyapunov analysis not only sheds light on the convergence properties of sm-AGDA , but also guides the parameter selection for sm-AGDA . Specifically, we show that sm-AGDA can compute an almost stationary point (˜ x, ˜ y ) satisfying ∥∇ f (˜ x, ˜ y ) ∥ ≤ ε with probability 1 -¯ q ∈ (0 , 1) within T ε, ¯ q = O ( ℓκ 2 δ 2 ε 4 + κ ε 2 ( ℓ + δ 2 log(1 / ¯ q ) ) ) stochastic gradient calls. The lower complexity bound of Ω( 1 ε 2 + 1 ε 4 ) for NCSC problems [38, 71] in expectation (see also [63, 72]) suggests that our high-probability bound for sm-AGDA is tight in terms of its dependence on ε . Furthermore, to our knowledge, these are the first high-probability guarantees for any algorithm in the NCPL setting.

Under light-tail (sub-Gaussian) assumption on the gradient noise (Assumption 3), which is common in the literature [35, 32, 19], we develop a new concentration result (Theorem 9) that can be of independent interest. From this concentration inequality, we observe that the cost of strengthening the existing complexity result in expectation to a high-probability one is relatively low, i.e., in the final complexity, the probability parameter ¯ q only appears in an additive term that scales with ε -2 . Consequently, this represents a non-dominant overhead compared to the ε -4 term already present in state-of-the-art expectation bounds [66].

Third, we provide experiments that illustrate our theoretical results. We first provide an example of an NCPL-game with synthetic data and then focus on distributionally robust optimization problems with real data, illustrating the performance of the sm-AGDA in terms of high-probability guarantees.

2 Preliminaries and Technical Background

Stationarity metric . We consider the minimax problem in (1) for f : R d 1 × R d 2 → R such that f is smooth (Assumption 1) and f ( x, · ) satisfies the PL property for all x ∈ R d 1 (Assumption 2); moreover, we also assume that we only have access to unbiased stochastic estimates of ∇ f such that the stochastic error G ( x, y, ξ ) -∇ f ( x, y ) has a light tail (Assumption 3) for any ( x, y ) , where G ( x, y, ξ ) denote the stochastic estimate of ∇ f ( x, y ) and ξ denotes the randomness in the estimator.

Our aim is to compute a ( ε x , ε y ) -stationary point (˜ x, ˜ y ) for (1) such that ∥∇ x f (˜ x, ˜ y ) ∥ ≤ ε x and ∥∇ y f (˜ x, ˜ y ) ∥ ≤ ε y . We also call (˜ x, ˜ y ) an ε -stationary point if ∥∇ f (˜ x, ˜ y ) ∥ ≤ ε . Clearly, whenever (˜ x, ˜ y ) is ( ε x , ε y ) -stationary, then it is also ε -stationary for ε = ( ε 2 x + ε 2 y ) 1 / 2 .

Smoothed alternating gradient descent ascent ( sm-AGDA ): The method can be considered as an inexact proximal point method and was introduced in [70]. More specifically, in each iteration of sm-AGDA , given a proximal center z t and the current iterate ( x t , y t ) , the method computes the next iterate ( x t +1 , y t +1 ) using a stochastic gradient descent ascent step on a regularized function ˆ f :

Following the stochastic alternating gradient descent ascent (stochastic AGDA) steps, the proximal center at iteration t , i.e., z t , is updated as shown in Algorithm 1, where G x ( x t , y t , ξ x t +1 ) and G y ( x t +1 , y t , ξ y t +1 ) denote conditionally unbiased stochastic estimators of the gradients ∇ x f ( x t , y t ) and ∇ y f ( x t +1 , y t ) . Throughout the analysis we assume that ∇ f is Lipschitz, which is standard in the study of first-order optimization algorithms for smooth minimax problems; see, e.g., [72, 73, 75, 63].

Assumption 1. (Lipschitz gradient) For all ( x 1 , y 1 ) , ( x 2 , y 2 ) ∈ R d 1 × R d 2 , there exists ℓ > 0

The following condition, known as Polyak-Łojaciewicz (PL) condition is weaker than assuming strong concavity in y , and does not even necessitate f to be even quasi-concave in the y variable. It holds in many ML applications including those in [48, 48, 42, 17, 23, 72, 68, 52, 66].

Assumption 2. (PL condition in y ) For every x ∈ R d 1 , max y ∈ R d 2 f ( x, y ) has a non-empty solution set and a finite optimal value. Moreover, there exists µ > 0 such that:

We assume that we have only access to stochastic estimates G x ( x t , y t , ξ x t +1 ) and G y ( x t +1 , y t , ξ y t +1 ) of the partial gradients ∇ y f ( x k , y k ) and ∇ x f ( x t +1 , y t ) , where ξ x t +1 and ξ y t +1 are random variables defined on a probability space (Ω , P ) , i.e., the source of randomness in the gradient estimates. Note that sm-AGDA has Gauss-Seidel updates, i.e., the stochastic estimate of the partial gra-

Algorithm 1 sm-AGDA

Input:

( x 0 , y 0 , z 0 ) , τ 1 , τ 2 > 0 , β ∈ [0 , 1] , p ≥ 0

for

t

= 0

,

1

,

2

, . . . , T

-

1

do

x t +1 = x t - τ 1 [ G x ( x t , y t , ξ x t +1 ) + p ( x t - z t )]

y

t

+1

=

y

t

+

τ

2

G

y

(

x

t

+1

, y

t

, ξ

y

t

+1

)

z t +1 = z t + β ( x t +1 - z t )

end for

Output:

choose (˜ x, ˜ y ) uniformly from { ( x t , y t ) } T - t =0

1

dient G y ( x t +1 , y t , ξ y t +1 ) is evaluated at the updated point ( x t +1 , y t ) instead of ( x t , y t ) . To capture the sequential information flow, we next introduce the natural filtrations that represent all the information available before an update: Let ξ x t and ξ y t be revealed sequentially in the natural order of the sm-AGDA updates, i.e., ξ x 1 → ξ y 1 → ξ x 2 → ξ y 2 → ξ x 3 →··· , and let ( F x t ) t ≥ 1 and ( F y t ) t ≥ 1 denote the associated filtration 4 , i.e., let F y 0 ≜ {∅ , Ω } , and

Introducing multiple filtrations to represent the sequential information flow is common in the study of stochastic algorithms with Gauss-Seidel updates -see, e.g., papers on stochastic ADMM, and [7, 69]; and we follow the same approach. Consider the gradient noise (errors) at time t ∈ N :

Finally, we also assume that the gradient noise is unbiased conditionally on the past information and that it admits a light (sub-Gaussian) tail.

Assumption 3. (Light tail) For any t ≥ 0 , there exists scalars δ x , δ y > 0 such that

For developing high-probability bounds in the learning context, it is common to assume that gradient estimates are sub-Gaussian [32, 35, 18]. While this assumption may not always hold (see e.g. [25, 56]), it often holds when gradients are estimated via mini-batching, as a consequence of the central limit theorem. It will also hold when the gradient noise is bounded. Additionally, adoption of differential privacy mechanisms within gradient-based schemes [14, 34, 1], to enhance data privacy, results frequently in sub-Gaussian gradient errors.

3 High-probability bounds for sm-AGDA

For analyzing sm-AGDA , similar to [66, 70], we consider the following Lyapunov function:

where P ( z ) and Ψ( · ; z ) denote the saddle point value and the dual function value, respectively, of the auxiliary problem min x max y ˆ f ( x, y ; z ) for any fixed z and ˆ f defined in (2), i.e.,

Next, we introduce a natural assumption, commonly made in the literature [66, 65]. Without this assumption, there are pathological cases where primal function Φ( x ) may be unbounded leading to divergence of gradient-based methods; an example would be f ( x, y ) = -x 2 -y 2 in dimension one.

Assumption 4. Consider the primal function Φ : R d 1 → R , i.e., Φ( x ) = max y ∈ R d 2 f ( x, y ) . There exists x ∗ ∈ R d 1 such that Φ ∗ ≜ Φ( x ∗ ) = min x ∈ R d 1 Φ( x ) .

Under Assumption 4, it immediately follows that V t ≥ Φ ∗ for all t ∈ N -since P ( z ) -Ψ( y, z ) ≥ 0 , ˆ f ( x, y ; z ) -Ψ( y ; z ) ≥ 0 and P ( z ) ≥ Φ ∗ for all x, y, z . We will next study the change V t -V t +1 in the Lyapunov function and show that an approximate descent property holds. First, we need two key lemmas that characterize the evolution of ˆ f ( x t , y t ; z t ) and Ψ( y t ; z t ) over the iterations.

Lemma 5. Suppose Assumptions 1, 2, 3 and 4 hold. Consider sm-AGDA given in Alg. 1 with τ 1 ∈ (0 , 1 p + ℓ ] and β ∈ (0 , 1] . For any t ∈ N , we have:

Proof. The proof is provided in Appendix B.1.

From Assumption 1, when p > ℓ , the auxilliary function ˆ f ( · , y ; z ) is ( p -ℓ ) -strongly convex for any fixed y, z ; hence, there is a unique minimizer for every y, z fixed, denoted by

i.e., Ψ( y, z ) = ˆ f ( x ∗ ( y, z ) , y ; z ) . In the rest of the paper, we will take p > ℓ and exploit this property. The following lemma characterizes the change in the dual function Ψ .

Lemma 6. Suppose Assumptions 1, 2, 3 and 4 hold. Consider the sm-AGDA iterate sequence { ( x t , y t , z t ) } t ∈ N for p > ℓ . For any t ∈ N , it holds that

where L Ψ ≜ ℓ ( 1 + p + ℓ p -ℓ ) and the map x ∗ ( · , · ) is defined by (11) .

Proof.

The next result provides an approximate descent property on the Lyapunov function. Its proof builds on Lemmas 5 and 6 and a descent property on the function P (given in Lemma 15 of the Appendix); and leverages smoothness properties of the functions ˆ f and Ψ and the map ( y, z ) ↦→ x ∗ ( y, z ) as well as the strong convexity of ˆ f with respect to x .

Theorem 7. Suppose Assumptions 1, 2, 3 and 4 hold. Consider the sm-AGDA algorithm with parameters p > ℓ , β ∈ (0 , 1] , τ 1 ∈ (0 , 1 p + ℓ ] and τ 2 > 0 chosen such that

for some constant ν > 0 , where L Ψ = ℓ ( 1 + p + ℓ p -ℓ ) . Then,

for some constants { c i } 8 i =1 ⊂ R that are explicitly given in Appendix C, which may depend on ν , as well as the problem and sm-AGDA parameters that can be chosen such that c 1 , c 2 , c 3 > 0 .

With some specific choice of parameters in sm-AGDA , we can obtain simplifications to the coefficients { c i } 8 i =1 from Theorem 7 (explicitly given in Appendix C). As such, this yields the following corollary.

Proof. The proof is given in Appendix D.

Next, we provide a concentration inequality which will be key to obtain our high-probability bounds. Theorem 9. Let {F t } t ∈ N be a filtration on (Ω , F , P ) . Let A t , B t , C t , D t be four stochastic processes adapted to the filtration such that there exist σ C , σ D > 0 and τ 1 > 0 such that for all t ∈ N : ( i ) B t ≥ 0 , ( ii ) E [ e λC t +1 | F t ] ≤ e λ 2 σ 2 C B t for all λ > 0 , ( iii ) E [ e λD t +1 | F t ] ≤ e λσ 2 D for all λ ∈ [ 0 , 1 σ 2 D ] and ( iv ) A t +1 -A t τ 1 ≤ -B t + C t +1 + D t +1 . Then, for any ¯ q ∈ (0 , 1] , we have P ( τ 1 2 T -1 ∑ t =0 B t ≤ ( A 0 -A T ) + τ 1 σ 2 D T +2 τ 1 max { 2 σ 2 C , σ 2 D } log ( 1 ¯ q ) ) ≥ 1 -¯ q.

Proof. The proof is provided in Appendix E.

Remark 10. While the above concentration inequality seems tailored to the analysis of sm-AGDA, it can also aid in deriving high probability bounds for many other first-order methods for nonconvex minimax problems that outputs a randomized iterate; indeed, the majority of existing Lyapunov arguments in the nonconvex setting are built upon constructing telescoping sums in line with Theorem 9, e.g., stochastic alternating GDA [66] for NCPL minimax problems and optimistic GDA [45] for strongly convex-strongly concave problems.

We next present our main result which provides a high-probability bound on the sm-AGDA iterates. The main idea of the proof is to apply Theorem 9 to the processes introduced in Corollary 8.

Theorem 11. In the premise of Corollary 8, sm-AGDA iterates ( x t , y t ) for τ 1 ≤ 1 3 ℓ satisfy

for some Q ¯ q,T = O ( κ (∆ 0 + b 0 ) τ 1 T + κ ( δ 2 x + δ 2 y ) ( τ 1 ℓ + 1 T log ( 1 ¯ q ))) explicitly stated in Appendix F, where ∆ 0 ≜ Φ( z 0 ) -Φ ∗ , b 0 ≜ 2 sup x,y { ˆ f ( x 0 , y ; z 0 ) -ˆ f ( x, y 0 ; z 0 ) } .

Proof Sketch. Let the stochastic processes A t , B t , C t , D t in Theorem 9 be chosen as A t = ˜ A t , B t = ˜ B t , C t = ˜ C t , D t = ˜ D t where ˜ A t , ˜ B t , ˜ C t , ˜ D t are defined in Corollary 8 and τ 1 > 0 be the primal stepsize in sm-AGDA ; according to Corollary 8, we have A t +1 -A t τ 1 ≤ -B t + C t +1 + D t +1 for t ∈ N . Since ∆ x t and ∆ y t admit sub-Gaussian tails, it can be shown that the conditions of Theorem 9 are satisfied for some appropriate constants σ 2 C and σ 2 D . Therefore, Theorem 9 implies a tail bound on ∑ T -1 t =0 ˜ B t . Using the relation between f and ˆ f , one can also show that ∥∇ x f ( x t , y t ) ∥ 2 + κ ∥∇ y f ( x t , y t ) ∥ 2 = O ( ˜ B t ) , for all t ∈ N . This last inequality allows to translate the tail bound for ∑ T -1 t =0 ˜ B t to a tail bound for ∑ T -1 t =0 ∥∇ x f ( x t , y t ) ∥ 2 + κ ∥∇ y f ( x t , y t ) ∥ 2 . The details of the proof is provided in Appendix F of the supplementary material.

Remark 12. Suppose sm-AGDA , given in Alg. 1, is run for T iterations, and it outputs a randomly selected iterate ( x U , y U ) , where the random iteration index U is chosen uniformly at random from the set { 0 , 1 , . . . , T -1 } , i.e., P ( U = t ) = 1 /T for t = 0 , 1 , . . . , T -1 . Theorem 11 implies that

Furthermore, in comparison with existing complexity bounds in expectation for sm-AGDA [66], our quantile bound requires only an overhead of order O ( ε -2 log(1 / ¯ q )) . Unless ¯ q is very small, this is typically negligible in comparison to the O ( ε -4 ) already present in rates in expectation.

Remark 13. In contrast to high-probability bounds derived from standard Markov-type arguments, our approach achieves significantly better scaling with respect to both ¯ q and ϵ . Specifically, consider an oracle that can generate a sample (ˆ x, ˆ y ) with E [ ∥∇ f (ˆ x, ˆ y ) ∥ ] ≤ ϵ after G ( ϵ ) iterations/stochastic samples. In particular, [66] shows that one can take G ( ϵ ) = O ( ℓκ 2 δ 2 ϵ 4 + κℓ ϵ 2 ) for the sm-AGDA algorithm assuming the variance of the stochastic gradient is bounded by δ 2 . A naive high-probability bound could be constructed by ensuring E [ ∥∇ f (ˆ x, ˆ y ) ∥ ] ≤ ¯ q · ϵ and applying Markov's inequality to yield an ϵ -stationary point with probability at least 1 -¯ q . However, this approach results in a complexity bound of G (¯ qϵ ) = O ( ℓκ 2 δ 2 ¯ q 4 ϵ 4 + κℓ ¯ q 2 ϵ 2 ) , leading to a significantly worse dependence on ¯ q than ours. Alternatively, following the rationale in [19, 62], to generate a high-probability bound, one can run the sm-AGDA algorithm m = Ω(log(1 / ¯ q )) times in parallel; where in each run we generate an ε/ 2 -solution and among the solutions, we select the one with the smallest estimated gradient norm. This would require m G ( ϵ ) = O ( log(1 / ¯ q ) ℓκ 2 δ 2 ϵ 4 +log(1 / ¯ q ) κℓ ϵ 2 ) iterations/stochastic samples.

In this approach, the logarithmic term log ( 1 ¯ q ) multiplies the high-order O ( 1 ε 4 ) term, whereas in our approach it only affects the second-order O ( 1 ε 2 ) term. Therefore, our results scale better with respect to ¯ q and ε . In addition, such a (multiple) parallel run approach, is often impractical in streaming/online settings, where data arrives sequentially, and real-time processing is essential.

Corollary 14. Under the premise of Theorem 11, consider running the sm-AGDA method for some fixed number of iterations T ∈ N with parameters chosen as τ 1 = min ( 1 3 ℓ , 48 √ ∆ 0 + b 0 √ Tℓδ 2 ) and τ 2 = τ 1 / 48 where δ 2 ≜ δ 2 x + δ 2 y . Then, for any ¯ q ∈ (0 , 1) , sm-AGDA can compute an ( ε, ε/ √ κ ) stationarity point with probability at least 1 -¯ q when the number of iterations T is fixed to T ε, ¯ q = O ( (∆ 0 + b 0 ) ℓκ ε 2 + δ 2 log ( 1 ¯ q ) κ ε 2 + δ 2 (∆ 0 + b 0 ) ℓκ 2 ε 4 ) which requires T ε, ¯ q stochastic gradient calls.

Proof. This is a direct consequence of Theorem 11, a proof is provided in Appendix G.

4 Numerical Illustrations

In this section, we illustrate the performance of sm-AGDA . We consider an NCPL problem with synthetic data, as well as a nonconvex DRO problem using real datasets. For synthetic experiments, we used an ASUS Laptop model Q540VJ with 13th Generation Intel Core i9-13900H using 16GB RAMand 1TB SSD hard drive. For the DRO experiments, we used a high-performance computing cluster with automatic GPU selection (NVIDIA RTX 3050, RTX 3090, A100, or Tesla P100) based on GPU availability, ensuring optimal use of computational resources.

Synthetic experiments on an NCPL game. We consider the following NCPL problem:

which can be interpreted as a game between two players [48, 35] where m 1 , m 2 > 0 are constants and the symmetric matrix K is set randomly, similar to the standard bilinear game setting considered in [35]. More specifically, we set K = 10 ˜ K/ ∥ ˜ K ∥ , ˜ K = ( M + M ⊤ ) / 2 where M is a d × d matrix with entries being i.i.d centered Gaussian having variance σ 2 . This problem is nonconvex in x (without satisfying the PL condition in x ). Though the exact gradient is known, we consider a stochastic gradient oracle, which returns noisy gradients similar to the setting of [35, 11, 16, 2], i.e., for each iteration t ∈ { 0 , ..., T -1 } , G x ( x t , y t ; ξ x ) = ∇ x f ( x t , y t ) + ξ x and G y ( x t +1 , y t , ξ y ) =

t +1 t +1 t +1 ∇ y f ( x t +1 , y t ) + ξ y t +1 , with ( ξ x t +1 ) t ≥ 0 iid ∼ N ( 0 , δ 2 I d 1 ) and ( ξ y t +1 ) t ≥ 0 iid ∼ N ( 0 , δ 2 I d 2 ) where I d is the d × d identity matrix and δ 2 is some constant variance. This setting satisfies all our assumptions, and our high-probability results (Theorem 11 and Coro. 14) are applicable. In this experiment, we fix d 1 = d 2 = 30 , and m 1 = m 2 = δ 2 = σ 2 = 1 . The solution to this problem is ( x ∗ , y ∗ ) = ( 0 , 0 ) .

Experimental results. The parameters of the problem are explicitly available as µ = 2 m 2 , and ℓ = max { 12 m 1 , 8 m 2 , ∥ K ∥} . To illustrate Theorem 11, we set β = τ 2 µ 1600 , τ 2 = τ 1 48 , p = 2 ℓ and we considered two cases: τ 1 = 1 3 ℓ (long step) and τ 1 = 1 12 ℓ (short step) to explore the behavior of sm-AGDA for different stepsizes. We generated N = 25 sample paths for T = 10 , 000 iterations, and on the left panel of Fig. 1, for each iteration t , we report the average of M κ ( t ) ≜ ∥∇ x f ( x t , y t ) ∥ 2 + κ ∥∇ y f ( x t , y t ) ∥ 2 over N = 25 realizations corresponding to different sample paths, and the shaded region depicts the range statistic, i.e., for every fixed iteration t , we shade the

Averagepathacrossgradients

Averagepathacrossiterates

(z(*X×),4)601

T1 =1/(3)

log(I1xI12 + IIyk12)

T1 =1/(3)

10

7.5

T1 =1/(12e)

T1 =1/(12e)

8

5.0

6

2.5

4

0.0

0

2000

4000

6000

8000

10000

0

2000

4000

6000

8000

10000

Iterations

Iterations

vertical line between the maximum and minimum values of M κ ( t ) for the 25 paths. On the right panel of Fig. 1, we also report the squared distance, I ( t ) ≜ ∥ x t -x ∗ ∥ 2 + ∥ y t -y ∗ ∥ 2 , in a similar manner. We observe that the range statistic for M κ ( t ) diminishes to a value inversely proportional to T as t → T ; this is inline with our theoretical results in Theorem 11. The existence of sinusoidal terms in the minimax objective is a source for oscillatory behavior in these figures. As t → T , both M κ ( t ) and I ( t ) exhibit oscillations, of which amplitude are smaller for the small step size compared to the large step-size -at the expense of a slower convergence; the iterate paths are not as oscillatory.

For T = 10 , 000 and τ 1 = 1 3 ℓ fixed, we next consider the path averages X T ≜ 1 T ∑ T -1 t =0 M κ ( t ) . Indeed, based on 1000 sample paths, each for T = 10 , 000 iterations, we compare the empirical quantiles of X T with the theoretical upper bound Q q,T on its quantiles (implied by Theorem 11). Figure 2 shows the cumulative distribution function (CDF) of the empirical distribution alongside the theoretical explicit upper bound Q q,T for the stationarity measure ∥∇ x f ( x U , y U ) + κ ∇ y f ( x U , y U ) ∥ with U

1.0

TheoreticalCDF

EmpiricalCDF

0.8

0.6

0.4

0.2

0.0

78.4

78.6

78.8

79.0

79.2

79.4

uniform over { 1 , . . . , T } . The details of the empirical quantile estimation are provided Appendix I due to space considerations. We observe that the empirical CDF has a sigmoid-like shape, while the theoretical quantiles that lie above display a concave form. This difference may arise because the theoretical quantile bounds Q q,T are designed to capture the worst-case behavior across the class of NCPL problems, whereas this specific NCPL example may not represent the worst-case scenario.

Distributionally Robust nonconvex Logistic Regression. We consider the DRO problem

3

2

1

0

1

2

3

0.0

0.2

0.4

0.6

0.8

1.0

a9a - epoch=20

3

2

1

0

1

2

3

0.0

0.1

0.2

0.3

0.4

3

2

1

0

1

2

0.0

0.1

0.2

0.3

0.4

0.5

sido0 - epoch=20

3

2

1

0

1

2

3

0.0

0.2

0.4

0.6

0.8

a9a - epoch=250

3

2

1

0

1

2

3

0.0

0.2

0.4

0.6

0.8

gisette - epoch=550

3

2

1

0

1

2

0.0

0.1

0.2

0.3

0.4

0.5

sido0 - epoch=250

SAPD+

SMDAVR

smAGDA

where ℓ j ( x ; a j , b j ) = log ( 1 + exp ( -b j a ⊤ j x )) denotes the logistic loss tied to an input-output pair ( a j , b j ) ∈ R d 1 ×{-1 , 1 } , and r ( x ) = λ 1 ∑ d 1 i =1 ωx 2 i 1+ ωx 2 i a primal regularization for the learning model x ∈ R d 1 . We allow the distribution y ∈ Y ≜ { y ∈ R d 2 : y ≥ 0 , 1 ⊤ y = 1 } to deviate from the uniform distribution u ≜ 1 d 2 1 where 1 denotes the vector of ones, and we penalize the distance between y and u through the regularization map g : y ↦→ λ 2 d 2 2 ∥ y -u ∥ 2 . We set the regularization parameters as ω = 10 , λ 1 = 10 e -4 , and λ 2 = 1 . Since r is nonconvex with a Lipschitz gradient and g is strongly convex, this is an NCSC problem.

Datasets, Algorithms and Hyperparameters. We consider three standard datasets for this problem, which are summarized as follows: The sido0 dataset [50] has d 1 = 4932 and d 2 = 12678 . The gisette dataset [26] has d 1 = 5000 and d 2 = 6000 . Finally, the a9a dataset [13] has d 1 = 123 and d 2 = 32561 . We compare the performances of sm-AGDA against two other baselines that achieve state-of-the-art performance in expectation for these datasets [72]. Specifically, we evaluate SAPD+ , which is a two-loop method where the subproblems are solved by the SAPD algorithm [73], and SMDAVR , a variance reduced extension of SMDA algorithm [31]. Since (14) is constrained, we augment sm-AGDA with a projection step in the update of the y variable onto the d 2 -dimensional simplex and adopt the analogous stationarity metric ∥∇ x f ( x t , y t ) ∥ 2 + ∥ P Y ∇ y f ( x t , y t ) ∥ 2 for constrained problems where P Y is a projection to the dual domain Y . For all datasets, the primal stepsize τ 1 of sm-AGDA is tuned via a grid-search over { 10 -k , 1 ≤ k ≤ 4 } . The dual stepsize τ 2 is set as τ 2 = τ 1 48 . Similarly, β is estimated through a grid-search over { 10 -k , 3 ≤ k ≤ 5 } . The parameter p is also tuned similarly on a grid, our code is provided as a supplementary document for the details. For other methods, our hyperparameters are tuned in accordance with [72].

Experimental results. In Figure 2, we plot histograms of our stationarity metric, across 200 runs in a logarithmic scale. We report the stationarity measure both in early phase of the training (i.e. t = 20 epochs), and in later phases (i.e. t = 550 epochs for gisette and t = 250 epochs for a9a and sido0 ). Our theoretical results are presented for unconstrained problems in the dual, therefore they are not directly applicable to the DRO problem where the dual domain is constrained. That being said, we observe that they are still predictive of performance in the DRO setting. More specifically, Figure 2 is supportive of our high-probability complexity bounds for sm-AGDA , in the sense that the distribution of the stationarity metric for sm-AGDA tends to concentrate. Notably, it outperforms the concentration behaviour of the other baselines. Furthermore, we observe that histograms for all baselines hardly evolve after 20 epochs. This is consistent with previous experiments carried on these datasets [72] where performance was measured in terms of the decay of the average loss and its standard deviation. As such, we conclude that sm-AGDA performs better both in the early phase and the later stage. In our experience, we observed sm-AGDA could accomodate larger stepsizes compared to the other algorithms, which may have contributed to its good performance.

5 Conclusion

Existing high-probability bounds only apply to convex/concave minimax problems or non-monotone variational inequality problems under restrictive assumptions to our knowledge. We close this gap by providing the first high-probability complexity guarantees for nonconvex/PL minimax problems satisfying the PL-condition in the dual variable for the sm-AGDA method. We also provide numerical results for an NCPL example and for nonconvex distributionally robust logistic regression.

References

Jason Altschuler and Kunal Talwar. Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. Advances in Neural Information Processing Systems , 35:3788-3800, 2022.

Necdet Serhat Aybat, Alireza Fallah, Mert Gurbuzbalaban, and Asuman Ozdaglar. Robust accelerated gradient methods for smooth strongly convex functions. SIAM Journal on Optimization , 30(1):717-751, 2020.

Amir Beck and Aharon Ben-Tal. Duality in robust optimization: primal worst equals dual best. Operations Research Letters , 37(1):1-6, 2009.

Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust Optimization , volume 28. Princeton University Press, 2009.

Aleksandr Beznosikov, Boris Polyak, Eduard Gorbunov, Dmitry Kovalev, and Alexander Gasnikov. Smooth monotone stochastic variational inequalities and saddle point problems: A survey. European Mathematical Society Magazine , (127):15-28, 2023.

Radu Ioan Bo¸ t and Axel Böhm. Alternating proximal-gradient steps for (stochastic) nonconvexconcave minimax problems. arXiv preprint arXiv:2007.13605 , 2020.

Radu Ioan Bo¸ t, Panayotis Mertikopoulos, Mathias Staudigl, and Phan Tu Vuong. Minibatch forward-backward-forward methods for solving stochastic variational inequalities. Stochastic Systems , 11(2):112-139, 2021.

Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 , pages 177-186. Springer, 2010.

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018.

Nicolas Boumal and Pierre-antoine Absil. Rtrmc: A Riemannian trust-region method for low-rank matrix completion. Advances in Neural Information Processing Systems , 24, 2011.

Bugra Can, Mert Gurbuzbalaban, and Lingjiong Zhu. Accelerated linear convergence of stochastic momentum methods in Wasserstein distances. In International Conference on Machine Learning , pages 891-901. PMLR, 2019.

Antonin Chambolle and Thomas Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of mathematical imaging and vision , 40:120-145, 2011.

Chih-Chung Chang and Chih-Jen Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):1-27, 2011.

Rishav Chourasia, Jiayuan Ye, and Reza Shokri. Differential privacy dynamics of Langevin diffusion and noisy gradient descent. Advances in Neural Information Processing Systems , 34:14771-14781, 2021.

Constantinos Daskalakis, Dylan J Foster, and Noah Golowich. Independent policy gradient methods for competitive reinforcement learning. Advances in Neural Information Processing Systems , 33:5527-5540, 2020.

Alireza Fallah, Asuman Ozdaglar, and Sarath Pattathil. An optimal multistage stochastic gradient method for minimax problems. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 3573-3579. IEEE, 2020.

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467-1476. PMLR, 2018.

Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization I: A generic algorithmic framework. SIAM Journal on Optimization , 22(4):1469-1492, 2012.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Eduard Gorbunov, Marina Danilova, David Dobre, Pavel Dvurechenskii, Alexander Gasnikov, and Gauthier Gidel. Clipped stochastic methods for variational inequalities with heavy-tailed noise. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 31319-31332, 2022.

Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: O (1 /k ) lastiterate convergence for monotone variational inequalities and connections with cocoercivity. In International Conference on Artificial Intelligence and Statistics , pages 366-402, 2022.

Eduard Gorbunov, Abdurakhmon Sadiev, Marina Danilova, Samuel Horváth, Gauthier Gidel, Pavel Dvurechensky, Alexander Gasnikov, and Peter Richtárik. High-probability convergence for composite and distributed stochastic minimization and variational inequalities with heavytailed noise. arXiv preprint arXiv:2310.01860 , 2023.

Mert Gürbüzbalaban, Yuanhan Hu, Umut Simsekli, and Lingjiong Zhu. Cyclic and randomized stepsizes invoke heavier tails in SGD than constant stepsize. Transactions on Machine Learning Research , August 2023.

Mert Gürbüzbalaban, Andrzej Ruszczy´ nski, and Landi Zhu. A stochastic subgradient method for distributionally robust non-convex and non-smooth learning. Journal of Optimization Theory and Applications , 194(3):1014-1041, 2022.

Mert Gürbüzbalaban, Umut Simsekli, and Lingjiong Zhu. The heavy-tail phenomenon in SGD. In International Conference on Machine Learning , pages 3964-3975. PMLR, 2021.

Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the NIPS2003 feature selection challenge. Advances in Neural Information Processing Systems , 17, 2004.

Nicholas JA Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Conference on Learning Theory , pages 1579-1613. PMLR, 2019.

Mark Herbster, Stephen Pasteris, and Lisa Tse. Online matrix completion with side information. Advances in Neural Information Processing Systems , 33:20402-20414, 2020.

Yu-Guan Hsieh, Franck Iutzeler, Jérôme Malick, and Panayotis Mertikopoulos. On the convergence of single-call stochastic extra-gradient methods. Advances in Neural Information Processing Systems , 32, 2019.

Feihu Huang, Shangqian Gao, Jian Pei, and Heng Huang. Accelerated zeroth-order and firstorder momentum methods from mini to minimax optimization. Journal of Machine Learning Research , 23(36):1-70, 2022.

Feihu Huang, Xidong Wu, and Heng Huang. Efficient mirror descent ascent methods for nonsmooth minimax problems. Advances in Neural Information Processing Systems , 34, 2021.

Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems , 1(1):17-58, 2011.

Yang Junchi, Xiang Li, and Niao He. Nest your adaptive algorithm for parameter-agnostic nonconvex minimax optimization. In Advances in Neural Information Processing Systems , 2022.

Nurdan Kuru, ¸ S. ˙ Ilker Birbil, Mert Gürbüzbalaban, and Sinan Yildirim. Differentially private accelerated optimization algorithms. SIAM Journal on Optimization , 32(2):795-821, 2022.

Yassine Laguel, Necdet Serhat Aybat, and Mert Gürbüzbalaban. High probability and risk-averse guarantees for stochastic saddle point problems. accepted to Journal of Machine Learning (JMLR), arXiv preprint arXiv:2304.00444 , 2023.

Guanghui Lan. First-order and stochastic optimization methods for machine learning , volume 1. Springer, 2020.

Dongyang Li, Haobin Li, and Junyu Zhang. General procedure to provide high-probability guarantees for stochastic saddle point problems. arXiv preprint arXiv:2405.03219 , 2024.

Haochuan Li, Yi Tian, Jingzhao Zhang, and Ali Jadbabaie. Complexity lower bounds for nonconvex-strongly-concave min-max optimization. Advances in Neural Information Processing Systems , 34:1792-1804, 2021.

Xiang Li, Junchi Yang, and Niao He. Tiada: A time-scale adaptive algorithm for nonconvex minimax optimization. In The Eleventh International Conference on Learning Representations , https://openreview.net/forum?id=zClyiZ5V6sL , 2023.

Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning , pages 6083-6093. PMLR, 2020.

Tianyi Lin, Chi Jin, and Michael I Jordan. Near-optimal algorithms for minimax optimization. In Conference on Learning Theory , pages 2738-2779. PMLR, 2020.

Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in overparameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis , 59:85-116, 2022. Special Issue on Harmonic Analysis and Machine Learning.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Gabriel Mancino-Ball and Yangyang Xu. Variance-reduced accelerated methods for decentralized stochastic double-regularized nonconvex strongly-concave minimax problems. arXiv preprint arXiv:2307.07113 , 2023.

Aryan Mokhtari, Asuman E Ozdaglar, and Sarath Pattathil. Convergence rate of o(1/k) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM Journal on Optimization , 30(4):3230-3251, 2020.

Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In Advances in Neural Information Processing Systems , pages 2208-2216, 2016.

Angelia Nedich and Tatiana Tatarenko. Huber loss-based penalty approach to problems with linear constraints. arXiv preprint arXiv:2311.00874 , 2023.

Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class of non-convex min-max games using iterative first order methods. Advances in Neural Information Processing Systems , 32, 2019.

Balamurugan Palaniappan and Francis Bach. Stochastic variance reduction methods for saddlepoint problems. In Advances in Neural Information Processing Systems , pages 1416-1424, 2016.

DTP AIDS Antiviral Screen program. Sido0: a phamacology dataset. https://www. causality.inf.ethz.ch/data/SIDO.html , 2008.

H Rafique, M Liu, Q Lin, and T Yang. Non-convex min-max optimization: provable algorithms and applications in machine learning (2018). arXiv preprint arXiv:1810.02060 , 1810.

Harish Rajagopal. Multistage step size scheduling for minimax problems. Master's thesis, ETH Zurich. URL: https://www.research-collection.ethz.ch/handle/20.500. 11850/572991 , 2022.

Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.5647 , 2011.

Meisam Razaviyayn, Tianjian Huang, Songtao Lu, Maher Nouiehed, Maziar Sanjabi, and Mingyi Hong. Nonconvex min-max optimization: Applications, challenges, and recent theoretical advances. IEEE Signal Processing Magazine , 37(5):55-66, 2020.

Abdurakhmon Sadiev, Marina Danilova, Eduard Gorbunov, Samuel Horváth, Gauthier Gidel, Pavel Dvurechensky, Alexander Gasnikov, and Peter Richtárik. High-probability bounds for stochastic optimization and variational inequalities: the case of unbounded variance. In International Conference on Machine Learning , pages 29563-29648. PMLR, 2023.

Umut Simsekli, Levent Sagun, and Mert Gürbüzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5827-5837. PMLR, 2019.

Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere ii: Recovery by riemannian trust-region method. IEEE Transactions on Information Theory , 63(2):885-914, 2017.

J v. Neumann. Zur theorie der gesellschaftsspiele. Mathematische Annalen , 100(1):295-320, 1928.

Emmanouil-Vasileios Vlatakis-Gkaragkounis, Lampros Flokas, and Georgios Piliouras. Solving min-max optimization with hidden structure via gradient descent ascent. Advances in Neural Information Processing Systems , 34:2373-2386, 2021.

Yuanhao Wang and Jian Li. Improved algorithms for convex-concave minimax optimization. Advances in Neural Information Processing Systems , 33:4800-4810, 2020.

Linli Xu, James Neufeld, Bryce Larson, and Dale Schuurmans. Maximum margin clustering. In Advances in Neural Information Processing systems , pages 1537-1544, 2005.

Qiushui Xu, Xuan Zhang, Necdet Serhat Aybat, and Mert Gürbüzbalaban. A stochastic gda method with backtracking for solving nonconvex (strongly) concave minimax problems. arXiv preprint arXiv:2403.07806 , 2024.

Qiushui Xu, Xuan Zhang, Necdet Serhat Aybat, and Mert Gürbüzbalaban. A Stochastic GDA Method With Backtracking For Solving Nonconvex (Strongly) Concave Minimax Problems. arXiv e-prints , page arXiv:2403.07806, March 2024.

Yan Yan, Yi Xu, Qihang Lin, Wei Liu, and Tianbao Yang. Optimal epoch stochastic gradient descent ascent methods for min-max optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 5789-5800, 2020.

Junchi Yang, Negar Kiyavash, and Niao He. Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. Advances in Neural Information Processing Systems , 33:1153-1165, 2020.

Junchi Yang, Antonio Orvieto, Aurelien Lucchi, and Niao He. Faster single-loop algorithms for minimax optimization without strong concavity. In International Conference on Artificial Intelligence and Statistics , pages 5485-5517. PMLR, 2022.

TaeHo Yoon and Ernest K Ryu. Accelerated minimax algorithms flock together. arXiv preprint arXiv:2205.11093 , 2022.

Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3040-3049, 2021.

Yuxuan Zeng, Zhiguo Wang, Jianchao Bai, and Xiaojing Shen. An accelerated stochastic ADMM for nonconvex and nonsmooth finite-sum optimization. arXiv preprint arXiv:2306.05899 , 2023.

Jiawei Zhang, Peijun Xiao, Ruoyu Sun, and Zhiquan Luo. A single-loop smoothed gradient descent-ascent algorithm for nonconvex-concave min-max problems. Advances in Neural Information Processing Systems , 33:7377-7389, 2020.

Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of nonconvex-strongly-concave minimax optimization. In Uncertainty in Artificial Intelligence , pages 482-492. PMLR, 2021.

Xuan Zhang, Necdet Serhat Aybat, and Mert Gürbüzbalaban. SAPD+: An accelerated stochastic method for nonconvex-concave minimax problems. Advances in Neural Information Processing Systems , 35:21668-21681, 2022.

Xuan Zhang, Necdet Serhat Aybat, and Mert Gürbüzbalaban. Robust accelerated primal-dual methods for computing saddle points. SIAM Journal on Optimization , 34(1):1097-1130, 2024.

Yuchen Zhang and Lin Xiao. Stochastic primal-dual coordinate method for regularized empirical risk minimization. The Journal of Machine Learning Research , 18(1):2939-2980, 2017.

Landi Zhu, Mert Gürbüzbalaban, and Andrzej Ruszczy´ nski. Distributionally robust learning with weakly convex losses: Convergence rates and finite-sample guarantees. arXiv preprint arXiv:2301.06619 , 2023.

High-probability complexity guarantees for nonconvex minimax problems APPENDIX

The organization of the Appendix is as follows:

In Section A, we summarize the key notations that are used throughout the main paper and the Appendix.

In Section B, we provide the proofs of Lemmas from Section 3 that were used for deriving the approximate Lyapunov descent property.

In Section C, we present the proof of Theorem 7 which provides an approximate descent property in terms of the Lyapunov function for the sm-AGDA iterates.

In Section D, we provide the proof of Corollary 8, which studies particular choice of parameters within Theorem 7.

In Section E we present a proof of Theorem 9 which provides a concentration inequality.

In Section F, we provide the proof of Theorem 11 which yields high-probability results for the sm-AGDA algorithm.

In Section G, we provide a proof of Corollary 14 that provides a high-probability (iteration) complexity bound for the sm-AGDA iterates.

In Section H, we present supplementary lemmas that were used to derive the results.

In Section I, we provide further details about numerical experiments.

A Notation

The key notations that will be used throughout the Appendix is as follows:

ˆ f ( x, y ; z ) = f ( x, y ) + p 2 ∥ x -z ∥ 2 denotes the auxiliary problem.

Ψ( y ; z ) = min x ˆ f ( x, y ; z ) is the dual function of the auxiliary problem.

Φ( x ; z ) = max y ˆ f ( x, y ; z ) is the primal function of the auxiliary problem.

P ( z ) = min x max y ˆ f ( x, y ; z ) is the optimal value of the primal problem min x Φ( x ; z )

x ∗ ( y, z ) = argmin x ˆ f ( x, y ; z ) for given y, z in the auxiliary function.

x ∗ ( z ) = argmin x Φ( x ; z ) is the unique optimal solution to the auxiliary primal problem.

Y ∗ ( z ) = argmax y Ψ( y ; z ) is the set of optimal solutions to the auxiliary dual problem.

y + ( z ) = y + τ 2 ∇ y f ( x ∗ ( y, z ) , y ) denotes an update in y in the direction of the gradient of the dual function, i.e., along the direction ∇ Ψ( y ; z ) = ∇ y f ( x ∗ ( y, z ) , y ) .

ˆ G x ( x, y, ξ ; z ) ≜ G x ( x, y, ξ ) + p ( x -z )

∆ x t = G x ( x t , y t , ξ x t +1 ) -∇ x f ( x t , y t ) and ∆ y t = G y ( x t +1 , y t , ξ y t +1 ) -∇ y f ( x t +1 , y t ) denote the gradient error, i.e., the difference between the stochastic estimates of the partial gradients and the exact partial gradients.

B Proofs of Lemmas from Section 3

B.1 Proof of Lemma 5

Lemma 5. Suppose Assumptions 1, 2 and 3 hold. Consider sm-AGDA , stated in Algorithm 1, with τ 1 ∈ (0 , 1 p + ℓ ] and β ∈ (0 , 1] . For any t ∈ N , we have:

Proof. Since, ˆ f ( · , y ; z ) is ( p + ℓ ) -smooth, we have

where last equality follows from ˆ G x ( x t , y t ; z t ) = ∇ x ˆ f ( x t , y t ; z t ) + ∆ x t . Hence, for τ 1 ≤ 1 p + ℓ , we have

Similarly, we observe that for all, ∇ y ˆ f ( x, y ; z ) = ∇ y f ( x, y ) , for all x, y, z , which together with the smoothness of f ( x, · ) implies

where we used again the identity G y ( x t +1 , y t , ξ y t +1 ) = ∇ y ˆ f ( x t +1 , y t ; z t ) + ∆ y t .

where we used 0 < β ≤ 1 . Therefore, summing up (15), (16), (17) yields the claim.

B.2 Proof of Lemma 6

Lemma 6. Suppose Assumptions 1, 2 and 3 hold, and p ≥ ℓ . Then, for any t ∈ N ,

where L Ψ ≜ ℓ ( 1 + p + ℓ p -ℓ ) and the map x ∗ ( · , · ) is defined in (11) .

Proof. By Lemma 19, Ψ is L Ψ -smooth in y for any given z ∈ R d 1 . Then, using ∇ y Ψ( y t ; z t ) = ∇ y f ( x ∗ ( y t , z t ) , y t ) , we obtain

Furthermore, by definition of Ψ , we also have

Summing (18) and (19), we conclude.

C Proof of Theorem 7

Before we move on to the proof of Theorem 7, we first provide a result from [70, 64] that quantifies the change in the function P over the iterations. We also provide its proof for the sake of completeness.

Lemma 15 (Lemma B.7 in [70]) . Suppose Assumptions 1, 2 and 3 hold. Consider the sm-AGDA iterate sequence ( x t , y t , z t ) t ∈ N for p > ℓ , and let Y ∗ ( z ) ≜ argmax y Ψ( y ; z ) denote the set of maximizers of Ψ( · , z ) for given z . For any t ∈ N and y ∗ ( z t +1 ) ∈ Y ∗ ( z t +1 ) , it holds that

Proof. Let y ∗ ( z t +1 ) ∈ Y ∗ ( z t +1 ) and y ∗ ( z t ) ∈ Y ∗ ( z t ) be two arbitrary maximizers. We have

The first and the third equality above hold by the definition of P ( z ) and Ψ( y ; z ) functions; on the other hand, for the second equality, one needs strong duality to hold. This interchange is valid and can be justified by the simple fact that ˆ f is strongly convex in x (therefore, it satisfies the PL condition in x ), and it also satisfies the PL condition in y according to our assumption 2. It has been established in [65, Lemma 2.1] that the double-sided PL property allows one for the min-max switch.

Equipped with this lemma, the stage is set to prove Theorem 7 from Section 3. We first restate an extended version of this theorem, where the constants { c i } 8 i =1 are provided explicitly.

Theorem 7. Suppose Assumptions 1, 2, 3 and 4 hold. Consider the sm-AGDA algorithm with parameters p > ℓ , β ∈ (0 , 1] , τ 1 ∈ (0 , 1 p + ℓ ] and τ 2 > 0 chosen such that

for some constant ν > 0 , where L Ψ = ℓ ( 1 + p + ℓ p -ℓ ) . Then,

where the coefficients c 1 to c 8 have the following forms:

Furthermore, the sm-AGDA parameters can be chosen such that c 1 , c 2 , c 3 > 0 .

Proof. Combining the inequalities in Lemmas 5, 6 and 15 gives us a lower bound of the form:

for any y ∗ ( z t +1 ) ∈ Y ∗ ( z t +1 ) appearing in A 3 , where

Next, we provide lower bounds for several terms in the above inequality, including A 2 , A 3 , A 5 , ∥∇ y f ( x t +1 , y t ) ∥ 2 and ∥ z t +1 -z t ∥ 2 . At the end, using these bounds within (21), we will be able to establish a descent property for { V t } , which will allow to apply our concentration result (Theorem 9) for deriving the desired high probability bounds.

Lower bound for A 2 . Using Cauchy-Schwarz inequality and ℓ -smoothness of f , we have

where last line follows from Young's inequality. Thus, by Lemma 20, we obtain for any ν > 0

Lower bound for A 3 . By Cauchy-Schwarz inequality and Lemma 21,

Hence, using Young's inequality, for all β > 0 , we obtain

We now lower bound the second term on the right-hand side of (23). First, note that we have x ∗ ( y ∗ ( z t +1 ) , z t +1 ) = x ∗ ( z t +1 ) , which follows from the fact that min x max y ˆ f ( x, y ; z ) = max y min x ˆ f ( x, y ; z ) for all z since ˆ f is strongly convex in x and satisfies the PL condition in Assumption 2. Hence, using the inequality ∥ ∥ ∥ ∑ N i =1 w i ∥ ∥ ∥ 2 ≤ N ∑ N i =1 ∥ w i ∥ 2 , which holds for any { w i } ∈ R d 1 and N ≥ 1 , we get

By Lemma 21 and Lemma 23, we observe that

Using Lemma 24, we also have

Finally, using Lemma 18, we get

where the last inequality stems from the ℓ -smoothness of f . In view of Lemma 20, we may further upper bound the above quantity as follows:

Summing up the three intermediate inequalities we established above, we obtain

In conclusion for A 3 , using (23) and the above inequality, we obtain after some rearrangement

where the first inequality follows from Young's inequality, the second inequality follows from ∇ y f being ℓ -smooth and the update rule x t +1 = x t -τ 1 ˆ G x ( x t , y t , ξ x t +1 ; z t ) , and in the third inequality we use ∆ x t = ˆ G x ( x t , y t , ξ x t +1 ; z t ) -∇ x ˆ f ( x t , y t ; z t ) . Thus, since -τ 2 (1 + ℓτ 2 +2 L Ψ τ 2 ) < 0 , for any τ 2 > 0 , we obtain

We observe that ∥∇ y f ( x t +1 , y t ) ∥ 2 appears in both A 1 and the lower bound given in (22) for A 2 ; hence, grouping the two terms together and using the definition of c 0 ≥ 0 , we get

Lower bound for ∥ z t +1 -z t ∥ 2 . Since z t +1 = z t + β ( x t +1 -z t ) for some β > 0 , we observe that:

and since ˆ G x ( x t , y t , ξ x t +1 ; z t ) = ∇ x ˆ f ( x t , y t ; z t ) + ∆ x t , we obtain

Note that ∥ z t +1 -z t ∥ 2 appears in both A 1 and the lower bound given in (24) for A 3 ; hence, grouping the two terms together and using the definition of c ′ 0 ≥ 0 Using (27), we obtain

We conclude by combining all these lower bounds, i.e., the claimed Lyapunov descent inequality follows directly from summing (21), (22), (24), (25), (26) and (27). Finally, it follows after straightforward computations that there exist choice of sm-AGDA parameters which yield c 1 , c 2 , c 3 > 0 while satisfying the conditions c 0 ≥ 0 and c ′ 0 ≥ 0 ; in fact Corollary 8 provides such sm-AGDA parameters explicitly. This completes the proof.

D Proof of Corollary 8

The lower bound provided in Theorem 7 resembles the descent property we require for our concentration result in Theorem 9. To allow for its proper application, we develop a stepsize policy inspired by [66].

Corollary 8. Under the premise of Theorem 7, consider the parameters p = 2 ℓ , τ 1 ∈ (0 , 1 3 ℓ ] , τ 2 = τ 1 48 , β = αµτ 2 for any α ∈ (0 , 1 406 ] . Then, ˜ A t +1 -˜ A t τ 1 ≤ -˜ B t + ˜ C t +1 + ˜ D t +1 for all t ∈ N , where ν = 12 τ 1 ℓ and

Proof. In view of Theorem 7, it suffices to prove that setting p = 2 ℓ , τ 1 ∈ (0 , 1 3 ℓ ] , τ 2 = τ 1 48 , β = αµτ 2 for for some positive α ≤ 1 406 and ν = 12 τ 1 ℓ leads to both c 0 ≥ 0 and c ′ 0 ≥ 0 ; furthermore, we also need to show that this choice of parameters implies the following lower bounds:

First, we show that our parameter choice implies that c 0 , c ′ 0 ≥ 0 . Noting that L Ψ = 4 ℓ , using τ 1 ≤ 1 3 ℓ , we may bound c 0 from above and below as follows:

Moreover, since β = αµτ 2 and τ 2 ≤ 1 144 ℓ , we get β ≤ α 144 using κ ≥ 1 . Hence, for α ≤ 1 406 ,

where the last inequality follows from α ≤ 1 406 ≤ 3 2 ( √ 5 -1) ; therefore, c ′ 0 ∈ [ p 4 β , p 3 β ] . Next, we prove bounds on c 1 , c 2 , c 3 , c 7 , c 8 separately.

Proof of part ( i ) . Since p = 2 ℓ , ν = 12 τ 1 ℓ and τ 1 ≤ 1 3 ℓ , we first observe that

Furthermore, using τ 2 = τ 1 48 , and β = αµτ 2 , we obtain

where last line follows from the fact that µ/ℓ ≤ 1 and τ 3 2 ℓ 2 τ 1 = τ 2 1 ℓ 2 48 3 ≤ 1 144 2 · 48 , in which we used τ 1 ≤ 1 3 ℓ .

Since τ 2 ≥ c 0 ≥ 0 and -2 ℓ 2 ( 1 ( p -ℓ ) 2 + τ 2 1 ) ≥ -20 9 , using τ 2 = τ 1 48 , we obtain

Using L Ψ = 4 ℓ , τ 1 ≤ 1 3 ℓ and τ 2 ≤ 1 144 ℓ , we get

Using τ 1 ≤ 1 3 ℓ , τ 2 ≤ 1 144 ℓ , this implies

Combining equations (29), (30), (31), (32) and (33), we obtain

for α ≤ 1 406 < 2 . 4 , which completes the proof of part ( i ) .

Proof of part ( ii ) . Due to our parameter choice, we first note that

where last line follows from τ 2 ≤ 1 144 ℓ and α ≤ 1 406 . Finally, (27) implies that c 0 2 ≥ τ 2 4 ; hence, c 2 ≥ ( 1 4 -1 8 ) τ 2 = τ 2 8 .

Proof of part ( iii ) . We have shown that c ′ 0 ≥ p 4 β ; hence, c 3 ≥ pβ 8 .

Proof of part ( iv ) . According to (27), τ 2 ≥ c 0 ; hence, we deduce that

where the last inequality follows from τ 2 ≤ 1 144 ℓ . Furthermore, we note that

with the last inequality following also from µ ℓ ≤ 1 and τ 2 ≤ 1 144 ℓ . We also note that

Finally, since c ′ 0 ≤ p 3 β according to (28), we get

where we used τ 2 ≤ 1 144 ℓ . Thus, since α ∈ (0 , 1) , it follows from (36), (37), (38) and (32) that

Proof of part ( v ) . For c 8 , we may simply observe that

where we used µ/ℓ ≤ 1 , L Ψ = 4 ℓ , τ 2 ≤ 1 144 ℓ , and α < 1 20 .

E Proof of Theorem 9

Theorem 9. Let {F t } t ∈ N be a filtration on (Ω , F , P ) . Let A t , B t , C t , D t be four stochastic processes adapted to the filtration such that there exist σ C , σ D > 0 and τ 1 > 0 such that for all t ∈ N : ( i ) B t ≥ 0 , ( ii ) E [ e λC t +1 | F t ] ≤ e λ 2 σ 2 C B t for all λ > 0 , ( iii ) E [ e λD t +1 | F t ] ≤ e λσ 2 D for all λ ∈ [ 0 , 1 σ 2 D ] and ( iv ) A t +1 -A t τ 1 ≤ -B t + C t +1 + D t +1 . Then, for any ¯ q ∈ (0 , 1] , we have

Proof. For some fixed γ > 0 , let

with the convention that S 0 ≜ 0 . For any T ∈ N , we have

where the inequality follows from condition (iv) of the hypothesis. Hence, for any 0 < λ ≤ 1 2 τ 1 σ 2 D and any T ∈ N :

where the first inequality follows from Cauchy-Schwarz and in the second one we use (ii) and (iii) of the hypothesis. Fixing γ = τ 1 / 2 yields for all 0 < λ ≤ 1 4 τ 1 σ 2 C :

Therefore, for 0 < λ ≤ min { 1 4 τ 1 σ 2 C , 1 2 τ 1 σ 2 D } , using B T ≥ 0 by (i) of the hypothesis, we get

and rolling this recursion backwards and noting S 0 = 0 yields:

thus, using a Chernoff bound, we get

Since for ¯ q ∈ (0 , 1] ,

we have

The claim follows by taking λ = 1 2 τ 1 min { 1 2 σ 2 C , 1 σ 2 D } .

F Proof of Theorem 11

Theorem 11. In the premise of Corollary 8, sm-AGDA iterates ( x t , y t ) for τ 1 ≤ 1 3 ℓ satisfy

for some Q ¯ q,T = O ( κ (∆ 0 + b 0 ) τ 1 T + κ ( δ 2 x + δ 2 y ) ( τ 1 ℓ + 1 T log ( 1 ¯ q ))) explicitly stated in Appendix F, where ∆ 0 ≜ Φ( z 0 ) -Φ ∗ , b 0 ≜ 2 sup x,y { ˆ f ( x 0 , y ; z 0 ) -ˆ f ( x, y 0 ; z 0 ) } .

As a first step, we provide a helper lemma that shows that our concentration result (Theorem 9) is applicable to the sm-AGDA -related processes ˜ A t , ˜ B t , ˜ C t , ˜ D t introduced in Corollary 8.

Lemma 16. Let A t = ˜ A t , B t = ˜ B t , C t = ˜ C t , D t = ˜ D t , where ˜ A t , ˜ B t , ˜ C t , ˜ D t are defined in Corollary 8; moreover, let τ 1 > 0 be the primal stepsize in sm-AGDA . Then, the processes A t , B t , C t , D t are adapted to the filtration F t ≜ F y t , where F y t is defined in (6) , and they satisfy the conditions of Theorem 9 with the following constants:

Proof. The fact that A t , B t , C t , D t are measurable with respect to F t = F y t for any t ∈ N follows directly from the definition of F t . Note that x t and y t are also F t -measurable for all t ∈ N . We prove part ( i ) and part ( ii ) separately.

Proof of part ( i ) . First, recall that C t +1 = ˜ C t +1 = -c 4 ⟨∇ x ˆ f ( x t , y t ; z t ) , ∆ x t ⟩ - ⟨ c 5 ∇ y f ( x t , y t ) + c 6 ∇ y f ( x ∗ ( y t , z t ) , y t ) , ∆ y t ⟩ . We would like to show that E [ e λC t +1 | F t ] ≤ e λ 2 σ 2 C B t for all λ > 0 . From Assumption 3, we note that for any λ ≥ 0 :

where we used [35, Lemma 3]. Now, given the value of c 4 in Theorem 7 and the convexity of t ↦→ t 2 , we have

Now, leveraging the stepsize policy specified in Corollary 8, since α ∈ (0 , 1) , using (37) we get

Similarly, as τ 1 ≤ 1 3 ℓ , have | ( p + ℓ ) τ 1 -1) | ∈ [0 , 1] , which implies

Since ν = 12 τ 1 ℓ , we have 5 ( 4 ℓ ν τ 2 1 ) 2 ≤ 5 9 ℓ 4 τ 6 1 ≤ τ 2 1 4 . Furthermore, using (27), we have 5 ( 4 c 0 ℓ 2 τ 2 1 ) 2 ≤ 80 τ 4 1 τ 2 2 ℓ 4 ≤ τ 2 1 4 . Finally, (38) and α ∈ (0 , 1) imply that

Thus, c 2 4 ≤ 6 τ 2 1 , which implies that

Moreover, noting ∆ y t is revealed after ∆ x t , using [35, Lemma 3] again along with the inequality ∥ u + v ∥ ≤ 2 ∥ u ∥ 2 +2 ∥ v ∥ 2 , we have:

where the last line follows from 144 144

Thus, since ∆ y t is revealed after ∆ x t , using (39) and (41) together with the tower property of the conditional expectations, we get

Finally, observe that

where in the last inequality we used the ( p -ℓ ) -strong convexity of ˆ f ( · , y t ; z t ) and the fact that we have ∇ ˆ f x ( x ∗ ( y t , z t ) , y t ; z t ) = 0 . Therefore, since p = 2 ℓ , we obtain

Plugging this inequality into (42) yields

where the last inequality follows from τ 2 = τ 1 48 .

Proof of part ( ii ) . Recall that D t +1 ≜ ˜ D t +1 = 2 ℓτ 2 1 ∥ ∆ x t ∥ 2 +8 ℓτ 2 2 ∥ ∆ y t ∥ 2 . We would like to show that E [ e λD t +1 | F t ] ≤ e λσ 2 D for all λ ∈ [ 0 , 1 σ 2 D ] for some σ D > 0 . First, observe that for any λ > 0 such that λ ≤ min { 1 16 ℓτ 2 1 δ 2 x , 1 64 ℓτ 2 2 δ 2 y } , we have

where we used [35, Lemma 2] in the second inequality.

Before completing the proof of Theorem 11, we provide another lemma that gives a lower bound on B t in terms of the squared norm of the partial gradients, based on our choice of sm-AGDA parameters. Lemma 17. For any t ∈ N , we have

Proof. Since ∇ x ˆ f ( x t , y t ; z t ) = ∇ x f ( x t , y t ) + p ( x t -z t ) , using ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 , we get ∥∇ x f ( x t , y t ) ∥ 2 ≤ 2 ∥∇ x ˆ f ( x t , y t ; z t ) ∥ 2 +2 p 2 ∥ x t -z t ∥ 2 .

Hence, together with (43) and the definition of B t , we obtain

with the last inequality following from κ ≥ 1 .

We may now provide our proof for Theorem 11.

Proof of Theorem 11. According to Lemma 16, the processes A t , B t , C t , and D t defined in the statement of Lemma 16 satisfy the conditions of Theorem 9 if we set τ 1 as the primal stepsize of sm-AGDA . Therefore, for any ¯ q ∈ [0 , 1) ,

Thus, dividing by τ 1 2 T and using Lemma 17, we can conclude that with the probability at least 1 -¯ q , the following event holds:

Finally, we can relate the potential gap V 0 -V T to the primal suboptimality, i.e., ∆ 0 , and to the duality gap, i.e., b 0 / 2 = sup x,y { ˆ f ( x 0 , y ; z 0 ) -ˆ f ( x, y 0 ; z 0 ) , at the initialization, following the same arguments provided in [66]. More precisely, for V ( x, y, z ) ≜ ˆ f ( x, y ; z ) -2Ψ( y, z ) + 2 P ( z ) , we first observe that

where last line follows from ˆ f ( x, y ; z ) -Ψ( y, z ) ≥ 0 for all y, z and P ( z ) -Ψ( y, z ) ≥ 0 for all y, z . Since p = 2 ℓ , we also note that

Finally, since P is the Moreau envelope of Φ , we have min z P ( z ) = min z Φ( z ) ; therefore,

Note that τ 2 = τ 1 48 implies 16 κ/τ 2 ≥ 20 κ/τ 1 , and 32 ℓ/β = 32 α · κ/τ 2 > 16 κ/τ 2 since α ∈ (0 , 1) . Therefore,

Therefore, combining (45), (46), (47) and (48), we conclude that Q q,T has the following explicit form:

where the constants r 1 , r 2 and r 3 are defined as

where the equalities follow from the expressions of σ 2 C and σ 2 D provided in Lemma 16.

G Proof of Corollary 14

Setting τ 2 = τ 1 / 48 for any τ 1 > 0 implies that Q ¯ q,T defined in (49) satisfies Q ¯ q,T = O ( κ (∆ 0 + b ) τ 1 T + τ 1 ℓκδ 2 + δ 2 κ T log ( 1 ¯ q )) . Hence, setting τ 1 = min ( 1 3 ℓ , 48 √ ∆ 0 + b 0 √ Tℓδ 2 ) ensures that

Finally, to obtain an O ( ε, ε/ √ κ ) -stationary point, it suffices to have the above bound smaller than O ( ε 2 ) , which is guaranteed when the following three conditions are met up to a constant factor: (i) (∆ 0 + b 0 ) ℓκ T ≤ ε 2 3 , (ii) √ (∆ 0 + b 0 ) ℓ T δκ ≤ ε 2 3 , and (iii) δ 2 κ T log ( 1 ¯ q ) ≤ ε 2 3 . This is directly implied by the value T ε, ¯ q given in our corollary statement.

H Supplementary Lemmas

In this section, we present a sequence of supplementary lemmas essential for deriving our main result, Theorem (11). Some of these results are well-known and are directly referenced. For others, we have improved specific algebraic constants. Additionally, some lemmas are extensions of existing bounds provided in expectation.

Proof. This result is provided in [66, Lemma C.1], which immediately follows from [41, Lemma B.2, part (c)].

Proof. This result immediately follows from [41, Lemma B.2, part (d)]. Indeed, for any y 1 , y 2 ∈ R d 2 , we have

where the first equality follows from Danskin's theorem, the first inequality follos from ℓ -smoothness of f , and the last inequality follows from Lemma 18.

Lemma 20. Consider the iterates ( x t , y t , z t ) of the sm-AGDA algorithm. For any t ∈ N ,

Proof. For any t ∈ N , using the fact that ˆ f is strongly convex in x with modulus ( p -ℓ ) together with x ∗ ( y t , z t ) = argmin x ˆ f ( x, y t ; z t ) , and x t +1 = x t -τ 1 ˆ G x ( x t , y t , ξ x t +1 ; z t ) , we get

Using the identity ˆ G x ( x t , y t , ξ x t +1 ; z t ) = ∆ x t + ∇ x ˆ f ( x t , y t ; z t ) within the above inequality yields the desired result.

Lemma 21. For any y ∈ R d 2 , z 1 , z 2 ∈ R d 1 , we have:

Proof. This result follows from [70, Lemma B.2]. Indeed, since ˆ f is strongly convex in x with modulus p -ℓ , we get

Then swapping z 1 , z 2 leads to

Furthermore, from the definition ˆ f , it follows that

similarly, swapping z 1 , z 2 in the above inequality leads to

Thus, summing the above two identities and applying Cauchy-Schwartz gives us

We can lower bound the left hand side of the above inequality using (50) and (51), which leads to

Rearranging this inequality yields the desired result.

Lemma 22. For any x ∈ R d 1 , x ↦→ Φ( x ; z ) is ( p -ℓ ) -strongly convex.

Proof. For any y ∈ R d 2 , z ∈ R d 1 , x ↦→ ˆ f ( x, y ; z ) is strongly convex with modulus p -ℓ , i.e., x ↦→ ˆ f ( x, y ; z ) -p -ℓ 2 ∥ x ∥ 2 is convex. Then, x ↦→ sup y ∈ R d 2 ˆ f ( x, y ; z ) -p -ℓ 2 ∥ x ∥ 2 is convex as it is the pointwise supremum of convex functions. Therefore, x ↦→ Φ( x ; z ) -p -ℓ 2 ∥ x ∥ 2 is convex, and this implies x ↦→ Φ( x, z ) is strongly convex with modulus p -ℓ .

Lemma 23. For all z 1 , z 2 ∈ R d 1 , we have

Proof. Using the result of Lemma 22, one can show this result following exactly the same arguments in the proof of Lemma 21.

Lemma 24. For any y ∈ R d 2 , z ∈ R d 1 , it holds that

Proof. The proof is the same as [66, Lemma C.2].

I Further Details about Figure 2

In this section, we provide further details about how the empirical and theoretical quantiles are estimated in Figure 2 and are compared.

Generation of the theoretical quantiles Q q,T . For any given q ∈ (0 , 1) , estimating an upper bound on Q q,T requires estimating an upper bound on the quantity ∆ 0 + b 0 based on Theorem 11. Other constants such as r 1 , r 2 , r 3 , ℓ , and µ are explicitly known in the setting of this experiment based on the NCPL game where T = 10 , 000 is fixed.

First, we set the initial point ( x 0 , y 0 ) randomly, where each component of x 0 and y 0 is sampled uniformly from the interval [ -20 , 20] , and we set z 0 = x 0 . We then estimate an upper bound on the quantity ∆ 0 + b 0 numerically based on a grid search, resulting in ∆ 0 + b 0 = 12 . Second, we generate a linear mesh I m with a grid size m = 0 . 0002 over the interval [0 , 1] . For q ∈ I m , we calculate Q q,T based on Theorem 11. Third, we generate a sequence of quantiles Q I m ,T . These quantiles are used to create a CDF via linear interpolation using the scipy.interpolate package's interp1d function in Python. Note that this quantile sequence generates a CDF over the values Q I m ,T .

Generation of the empirical quantiles of the random variable X T . We generate 1,000 samples { X ( i ) T } 1000 i =1 from the sample paths corresponding to the NCPL game with T = 10 , 000 , where X T = 1 T ∑ T t =1 ∥∇ x f ( x t , y t ) ∥ 2 + κ ∥∇ y f ( x t , y t ) ∥ 2 represents the path averages of the gradients. Quantiles for this sequence were generated over I m using NumPy's quantile generator in Python, ensuring alignment with the mesh over which the theoretical quantiles were generated. Evidently, our theoretical quantiles dominate the empirical quantiles pointwise, demonstrating that in the challenging NCPL regime, our theory provides empirically verifiable guarantees on the tail behavior of the random variable X T .

Comparison of quantiles. We plotted the CDF corresponding to the theoretical quantiles Q q,T over the values of the empirical quantiles using a common mesh grid over the range of the empirical averages of the sample paths. In other words, we scaled the quantiles Q q,T with an affine transformation so that their range matches the range of the empirical quantiles. This affine scaling preserves the shape of the distribution corresponding to the theoretical quantiles and allows for better visualization.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] The abstract and the introduction reflect the paper's contributions and scope accurately. In fact, we provide adequate references to the results from our paper for the claims in the introduction so that the paper's contributions and scope are easier to understand.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] Although we provide the first high-probability bounds for nonconvex/PL minimax problems to the best of our knowledge, there may still be some room to improve the condition number κ dependency based on the existing lower complexity bounds [38, 71] in expectation, unless the lower complexity bounds are loose. Also, we acknowledged that for the distributionally robust optimization experiment our assumptions do not all hold (which is a limitation); but that our results are still predictive of the practical performance.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] All assumptions are clearly stated or referenced in the statement of any theorem, we provide a sketch of the proof of our main theorem.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] We uploaded our code as a supplementary material, and explained in detail how our experiments are performed. All the datasets we use are well-known benchmark datasets and we provided the adequate references to them.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] We explain the experimental setup in detail and provide the code, the datasets are well-known benchmark sets that are publicly available and we provided the adequate references.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] Our numerical experiments section has all the details.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] In the experiments, we compare the histogram of the solutions found by multiple algorithms. This has more resolution than standard approaches that has the average performance (as an average over the algorithm runs) and the standard deviation over the runs; because one can visualize the behavior (histogram) of all runs.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] We explained the details of the operating system/computing resources.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] We preserve anonymity and we absolutely confirm with the code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] Our paper is mainly a theoretical paper with convergence guarantees for stochastic mini-max algorithms, so we are not aware of any direct impacts to the society.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] Our paper has theoretical nature an we are not aware of any potential misuse risk for our results.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] We use public datasets, and implemented the algorithms on our own in Python; no licensed material is used. We use public well-known benchmark datasets which are properly referenced.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] We do not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] No human subjects or crowdsourcing were involved.

Peer reviews:
comments: SUMMARY:
This paper proves a high probability convergence bound and almost sure convergence for stochastic smoothed AGDA method for minimax problems in nonconvex-PL setting. This is the first high probability guarantee and almost sure convergence guarantee in this nonconvex setting.

STRENGTHS:
This is the first high probability convergence guarantee and almost sure convergence guarantee in this nonconvex setting, going beyond convex setting and convergence in expectation.
The paper also presents a concentration bound (Theorem 9) which is of independent interests and this bound allows us to obtain high probability convergence results from expectation convergence results for minimax problems.

WEAKNESSES:
See Questions.

QUESTIONS:
1. The algorithm and analysis are built upon the approaches in 63 and 68. Could the authors provide more insights on the novelty of the proof compared to 63 and 68?
2. Is it possible to extend this analysis to other settings like the nonconvex-concave setting?
3. Can this analysis be extended to other algorithms such as stochastic GDA?
4. Missing references: 
[1]. Shen et al   Stochastic gradient descent ascent for federated minimax optimization
[2]  Zheng et al Universal gradient descent ascent method for nonconvex-nonconcave minimax optimization 
[3].  Ozdaglar et al What is a good metric to study the generalization of minimax learners
[4] Li et al Nonsmooth-nonconvex-nonconcave minimax optimization: primal-dual balancing and iteration complexity analysis

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the stochastic non-convex minimax optimization problem $\\min_x \\max_y f(x,y)$ where the function $f$ is smooth, then nonconvex in $x$ and PL in $y$. The authors presented a high probability analysis for the smoothed alternating gradient descent ascent (sm-AGDA) method. This analysis is based on a Lyapunov function from prior work for non-convex concave setting. Also, under a light-tail assumption on the gradient noise, the authors developed a concentration result. Finally, they include experiments of nonconvex-PL setting with synthetic data and then distributionally robust optimization problems with real data.\n\nSTRENGTHS:\nThe authors provided theoretical proofs and experiments to support their claims. The analysis seems to be solid. The authors compared their complexity with other high probability bounds results.\n\nWEAKNESSES:\n1. While the authors cited some results in expectation for the similar setting, I think it is important to compare the (Markov) high probability bounds obtained by these expectation results with the results obtained in this paper. If the results are not better, then the contribution is limited. \n\nAlso I don't get what the authors meant by \"More precisely, we focus on a purely stochastic regime in which data streams over time which renders the use of mini-batch schemes or running the method in parallel impractical; therefore, approaches based on Markov’s inequality [59] are no longer applicable.\" (line 99-101). \n\n2. In the theorems and lemma, the values $\\tau_1, \\tau_2, \\tau, \\bar{\\tau}$ is really confusing. Do you need to pick specific values for the step sizes, if so, what is $\\tau_2$? Are $\\tau$ and $\\bar{\\tau}, \\tau_1$ the same?\n\nUpdate: I thank the authors for the rebuttal. I keep my recommendation the same, as changing to high probability bound from expectation results is somewhat expected.\n\nQUESTIONS:\nWhy in the experiments the authors compare with SAPD+ and SMDAVR, not the methods in Table 1?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors build upon earlier works to prove a high probability upper bound, that is linear in variance of the stochastic gradients, on the number of stochastic gradient calls of smoothed alternating GDA method.

STRENGTHS:
The paper is well written with clear theorems, proofs and numerical results.

WEAKNESSES:
Focusing only on their concentration bound, which they claim is of independent interest and is the crux of their high probability bounds: the bound in Theorem 9 (which is fairly customized and not easy to state) follows like any other chernoff/chebychev type bound - using Markov type inequality after exponentiation; and I don't see any novelty in their theoretical ideas/techniques here. Perhaps I have missed something, if so the authors can clarify it.

QUESTIONS:
see weaknesses

LIMITATIONS:
Theory paper, no concerns
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the important open problem of stochastic smooth nonconvex minimax optimization. This paper proposes single-loop stochastic GDA method, which was known to be practically desirable but had no theoretical complexity compared to other non-single-loop methods with better complexies on nonconvex minimax problems. The analysis in this work fills some of above gap, provides the first high-probability complexity for nonconvex minimax while assuming a PL condition on dual variable, and proves assuming light tailed stochastic gradients GDA converges to a near stationary point with a certain complexity. Numerical results on NCPL game with synthetic data and distributionally-robust optimization with real data shows the proposed sm-AGDA outperforms existing algorithms SAPD+ and SMDAVR.\n\nSTRENGTHS:\n1. This paper solves the important open problem of stochastic smooth nonconvex minimax optimization, proposes a single-loop GDA method called sm-AGDA, which is constructed by Lyapunov function for nonconvex-concave problems.\n\n2. Assuming PL condition, this paper proved the first high-probability complexity bound on such single-loop algorithm on nonconvex minimax problem. The order of complexity is reasonably good.\n\n3. Numerical experiments show the proposed method is practically superior to existing methods.\n\nWEAKNESSES:\n1. Font size at the end of Page 8, and entire Page 9 are smaller than the required size. \n\n2. There is not much algorithm novelty from [68].\n\nQUESTIONS:\n1. What are the existing best-known complexities of (possibly non-single-loop) algorithms on nonconvex minimax problem?\n\n2. Can you compare in more details (overall problem class, overall algorithm, assumptions, complexity order dependencies) of your complexity result compared with [64]? \n\n3. Does each of your assumption (PL condition, etc.) hold for both of your numerical experiments?\n\n**Updates: \n\nThe authors' rebuttal addressed every question very well. I have increased my score from 6 to 7.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors derive a high probability bound for convergence of a stochastic gradient descent-ascent method over a nonconvex (PL class) of functions. Specifically, they analyze the sm-AGDA algorithm, which previously only had a bound in expectation, and show a similarly tight high probability bound when gradient noise is assumed subgaussian. The authors provide two experimental settings demonstrating that the distribution of these iterates over multiple runs converge as expected for the algorithm.\n\nSTRENGTHS:\n* Technically solid paper that thoroughly analyzes the sm-AGDA algorithm and understandably interprets the results (e.g. remark 11). I don't think the assumptions are too strict, or at least not outside the norm for this type of analysis. \n\n* Good comparison to other related works, especially Table 1 is useful to gauge the current state of theoretical results for this problem.\n\n* Helps to move analysis closer to realistic nonconvex min-max optimization settings.\n\nWEAKNESSES:\n* While I understand that the bound in Thm 10 over the average gradient norms motivates the random sampling of iterates in the sm-AGDA algorithm, it still seems like practically the best thing to do in Fig. 1 is to take the last iterate (or at least sample over some last window of iterates). Is there a straightforward way to come up with a similar high probability bound on the last iterate? \n\n* I'd maybe like to see more discussion about how the analysis in this paper differs from that in the original sm-AGDA work [63]. At least for some previous work I've seen with high probability bounds for convex optimization, oftentimes the actual modification required to go from expectation to high probability is fairly minor (i.e. just boils down to bounding the summation of Subgaussian noise terms).\n\n* I like the first set of experiments but a bit confused at the purpose of the second. I don't quite understand comparing concentration properties of other methods if the analysis only applies to one algorithm. Maybe if the authors highlighted what unique components of the sm-AGDA algorithm might give it better quantile convergence properties than others this would make more sense.\n\nIf the authors can address some of these concerns I am willing to raise my score to a more solid accept\n\nOther Minor Notes:\n* DRO abbreviation in line 238 not defined initially\n* [63] and [64] appear to be referencing the same paper\n* Line 306 and 311 appear to be referencing the wrong figure\n\nQUESTIONS:\n* As mentioned in remark 11 this bound appears to be pretty tight compared to the bound in expectation. Do you think if this type of analysis was applied to other existing algorithms you would get similarly tight bounds?\n\nLIMITATIONS:\nThe authors clearly state their assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}