A Cubic Regularization Approach for Finding Local Minimax Points in Nonconvex Minimax Optimization
Authors: Ziyi Chen
Zhengyang Hu
Qunwei Li
Zhe Wang
Yi Zhou
Venue: Accepted by TMLR ()

Abstract: Gradient descent-ascent (GDA) is a widely used algorithm for minimax optimization. However, GDA has been proved to converge to stationary points for nonconvex minimax optimization, which are suboptimal compared with local minimax points. In this work, we develop cubic regularization (CR) type algorithms that globally converge to local minimax points in nonconvex-strongly-concave minimax optimization. We first show that local minimax points are equivalent to second-order stationary points of a certain envelope function. Then, inspired by the classic cubic regularization algorithm, we propose an algorithm named Cubic-LocalMinimax for finding local minimax points, and provide a comprehensive convergence analysis by leveraging its intrinsic potential function. Specifically, we establish the global convergence of Cubic-LocalMinimax to a local minimax point at a sublinear convergence rate and characterize its iteration complexity. Also, we propose a GDA-based solver for solving the cubic subproblem involved in Cubic-LocalMinimax up to certain pre-defined accuracy, and analyze the overall gradient and Hessian-vector product computation complexities of such an inexact Cubic-LocalMinimax algorithm. Moreover, we propose a stochastic variant of Cubic-LocalMinimax for large-scale minimax optimization, and characterize its sample complexity under stochastic sub-sampling. Experimental results demonstrate faster or comparable convergence speed of our stochastic Cubic-LocalMinimax than the state-of-the-art algorithms such as GDA and Minimax Cubic-Newton. In particular, our stochastic Cubic-LocalMinimax was also faster as compared to several other algorithms for minimax optimization on a particular adversarial loss for training a convolutional neural network on MNIST. 

Published in Transactions on Machine Learning Research (03/2023)
A Cubic Regularization Approach for Finding Local Minimax Points in Nonconvex Minimax Optimization
Ziyi Chen
u1276972@@utah.cm edu
Deportment of Electrical and Computer Engineering University of Utah
Zhengyang Hu
daton.30@maiil.sstc.ccom ustc. edu.cr
School of Mathematical Sciences University of Science and Technology of China
Qunwei Li Ant Group
ganuet..qui@antgropp..oom
Zhe Wang
wwr.g. www.g.009220su..comm
JD company
Yi Zhou
yi.hhowuutah edu
Department of Electrical and Computer Engineering University of Utah
Reviewed on OpenReview: /t opwreevee... net/ per/forum? id- JWMdg331e
Abstract
Gradient descent-ascent (GDA) is a widely used algorithu for minimax optimization. How- ever, GDA has been proved to converge stationary points for nonconvex minimax optimiza- tion, which are suboptimal compared with local minimax points. In this work, we develop cubic regularization (CR) type algorithms that globally converge to local minimax points in nonconvex--ttronglyycoott
minimax optimization. We first show that local minimax points are equivalent to second-order stationary points of a certain envelope function. Then, inspired by the classic cubic regularization algorithm, we propose an algorithm named Cubic- LocalMinimax for finding local minimax points, and provide a comprehensive convergence analysis by leveraging its intrinsic potential function. Specifically, we establish the global convergence of MomlMi-ocaaliimmaxx and characterize its iteration complexity. Also, propose GDA-based solver for solving the cubic subproblem involved in Cubie-Local and analyze the overall gradient and Hessian-vector product computation complexities of algorithm. Moreover, we propose a stochastic variant
local minimax point at sublinear convergence rate
up to certain pre-defined accuracy,
such an inexact Cubie-LocalMinima of Cubie-LocalMinimax for large-scale minimax optimization, and characterize its sample complexity under stochastic sub-sampling. Experimental results demonstrate faster or com- parable convergence speed of our stochastic Lubie-LocalMinimax than the state-of-the-ar algorithms such GDA (Lin al. 2020) and Minimax Cubic-Newton (Luo al.] 2022). In particular, our stochastic Cubic- LocalMinimas was also faster compared to several other algorithms for minimax optimization on a particular adversarial loss for training convolutional neural network on MNIST.
1 Introduction
Minimax optimization (a.k.a. two-player sequential zero-sum games) popular modeling framework that has broad applications in modern machine learning, including game theory (Ferreira al. 2012). generative

---

Published in Transactions on Machine Learning Research (03/2023)
2014], adversarial training (Sinha et al. 2017 reinforcement learning 'Qiu et al. 2020 Ho and Ermon 2016 Song et al. 2018). etc. A standard minimax optimization problem is shown below, where f is a smooth bivariate function.
adversarial networks (Goodfellow et al.
min max fax y). 2ER" WER"
(P)
In the existing literature, many optimization algorithms have been developed to solve different types of minimax problems. Among them, a simple and popular algorithm is the gradient descent-ascent (GDA), which alternates between a gradient descent. update on x and a gradient ascent update on y in each iteration. Specifically, the global convergence of GDA has been established for minimax problems under various types of global geometries, such as convex-concave-lype geometry (f is convex x and concave in y) (Nediē and Ozdaglar 2009 Du and Hu, 2019 Mokhtari et al. 2020 Zhang and Wang, 2002)) bi-linear geometry
NNeumann 1928 Robinson 1951 and Polyal--.ojasiewicz geometry (Nouiched et al. 2019 Yang et al. 2020).
yet these geometries are not satisfied by general nonconvex minimax problems in modern machine learning applications. Recently, many studies proved the convergence of GDA in nonconvex minimax optimizatiion ion for both conconvex-coneane problems (Lin et al.) 2020; Nouiehed et al.) 2019 Xu et al.) 2023) and nonconvex- strongly-concave problems (Lin et al. 2020 Xu et al. 2023: Chen et al. 2021]. In these studics, it has beon
shown that GDA converges sublinearly to stationary point where the gradient of an envelope-type function ((x) == maxy (x, y) vanishes.
Although GDA can find stationary points in nonconvex minimax optimization, the stationary points may include candidate solutions that are far more sub-optimal than global minimax points, (e.g. saddle points of the envelope function +) which are known to be the major challenge for training high-dimensional machine learning models (Daphhin et al., 2014; Jin et al. 2017 Zhou and Liang, 2018). However, finding global minimax points is in general NP-hard (Jin et al.j (0200). Recently, Jiin et al. (2200) proposed a notion of local minimax point that computationally tractable and is close to global minimax point (see Definition 2.1 for
the formal definition).
In the existing literature, several studies have proposed Newton-type GDA algorithms for finding such local minimax points. Specifically, Wang et al. (2020) proposed a Follow-the- Ridge (FR) algorithm, which is a variant of GDA that applics a sccond-order correction term to the gradient ascent update. In particular, the authors showed that any strictly stable fixed point of FR is a local minimax point, and vice versa. In another the authors proposed two Newton-type GDA algorithms that are proven to locally
work Zhang et al. 2021). ; converge to local minimax point at a lincar and super-lincar convergence rate, respectively. However, these second-order-type GDA algorithms only have asymptotic convergence guarantees that require initializing sufficiently close to local minimax point, and they do not have any global convergence guarantees. Therefore, we are motivated to ask the following fundamental question.
Q: Can we develop globally convergent algorithms that can efficiently find local mánimax points in nonconver minimar optimization? What are their convergence rates and complexities?
In this work, we provide comprehensive answers to these questions. We develop deterministic and stochastic cubic regularization type algorithms that globally converge to local minimax points in nonconvex minimax optimization, and study their convergence rates, computation complexities and sample complexities.
1.1 Our Contributions
We consider the minimax optimization problem (P), where f twice-difforentiable with Lipschitz continuous gradient and Hessian and is nouromvee--torrlyyyomeeee In this setting, we first show that local minimax points of f are equivalent to second-order stationary points of the envelope function 4(x) := maX f(z, y). Then, inspired by the classic cubic regularization algorithm, we propose an algorithm named Cubic-LocalMinmmat to find local minimax points. The algorithm uses gradient ascent to update which is then used to estimate the gradient and Hessian involved in the cubic regularization update for x (see Algorithm T for more details).
Global convergence. We show that Cubic-LocalMinine admits an intrinsic potential function H, (scc Proposition 4.1) that monotonically decreases over the iterations. Based on this property, we prove that every limit point of ( generated by Cubic-LocclMinnaax is local minimax point. Moreover, to achieve
2

---

Published in Transactions on Machine Learning Research (03/2023)
e-accurate local minimax point, Cubie-LocalMnimmx requires (L mumber of cubic updates and o (L2gk2i number of gradient ascent updates, where & > denotes the problem condition number.
GDA-Cubic solver. The updates of Cubic-TaocalMninaax involve a cubic subproblem that has a very special Hessian structure. To solve this subproblem, we reformulate it a minimax optimization problem, for which develop GDA-type solver (sce Algorithm 2 We name such variant of Cubie-LocalMinmmax as Inexact Cubic-LocalMininen By bounding the approximation error of the cubic solver carefully, we establish monotonically decreasing potential function and establish the same iteration complexity as that of Cubie-LocalMininee Moreover, the total number of Hessian-vector product computations involved in the cubic solver of the order Ö(L)nEE=4).
Sample complexity. We further develop a stochastic variant of Cubic-LocalMininaa named as Stochastic Cubie-LocalMinnmar, which applies stochastic sub-sampling to improve the sample complexity in large-scale minimax optimization. In particular, we adopt time-varying batch sizes in a way such that the induced gradient inexactness and Hessian inexactness are adapted to the optimization increment | Iz-1ll in the previous iteration. Consoquently, to achieve an c-accurate local minimax point, stochastic Cubic-Lock Yubic-LocalMiniaa requires querying O(N3.5--T) number of gradient samples and (1x2-5,-5) number of Hessian samples.
Reference
Algorithm
Setting Deterministic f(:.)
Metric"
# Graddients
Nouiched et al.
2019
Multi-step GDA
PL
c-Nash
0 Innc
Nouiched et al.
2019
Multi-step GDA
Deterministic
Concave
e-Nash
3.
Lin et al.
2020
GDA
Deterministic
Strongly-concave c-stationary
O(Nee
Xu et al.
2023
AGP
Deterministic
Strongly-coneave
e-stationary
Olss 2)
Lin
2020
GDA
Deterministic
Concave
e-stationary
Ole ")
Xu et al.
[0233
AGP
Deterministic
Concave
e-stationary
Ole ")
Lin et al.
(2020
GDA
Stochastic
Strongly-concave
e-stationary
O(x
Lin
(2020)
GDA
Stochastic
Concave
e-stationary
O(e
Xu et al.
2020b
SREDA
Stochastic
Strongly-concave
e-stationary
O
Qiu et al.
(2020)
VR-STS
Stochastic
Strongly-concave
e-stationary
Ole
Table Comparison of first-order algorithms for nonconvex minimax optimization.
"Metric" denotes the point which the algorithm will converge, with the following choices:
Nash: [V, y)ll S 6, fz, y)ll < e.
e-stationary: [[V where (xx)) == maxyeR" f(x,v).
2 *# Gradients" denotes the required number of gradient evaluations.
Reference
Algorithm
Setting
# Graddients
#Hv products'
Luo et al. (0022)
Inexact MCN
Deterministic
((x6--10)
0(x
Our Theorem
Algorithm
Deterministic
Ō(x25-11.5
Ö(k2e-2)
Accelerated Algorithm 4
Deterministic
① ((x2c-1.5)
((x
Our Theorem
Algorithm
Stochastic
ō ( -3.))
5 (x22 2.226-2.0) -2.0)
Table 2: Comparison second-order algorithms for finding local minimax points in nonconvex-stronggyy-oo optimization.
The required number of Hessian-vector product evaluations.
1.2 Other Related Works
Deterministic first-order algorithms for minimax optimization Under seroonly-convve--strrree concave sctting, Zhang and Wang (2021) obtained itcration complexity ( In(c=1)) for optimistic gradient descent-ascent (OGDA) algorithm, and Mokhtari et al. (2020) obtained iteration complexity O In(e")) for both OGDA and extra-gradient (EG) algorithm which applics two-step GDA in each itcration. YYag et al. (2020) studied an alternating gradient descent-ascent (AGDA) algorithm in which the gradient ascent step uses the current variable r++1 instead of 2; and achieved iteration complexity ( In(e-))) for two-sided Polyak
3

---

Published in Transactions on Machine Learning Research (03/2023)
Lojasiewiez (PL) minimax optimization. These works require require strong geometric assumptions such as strong convexity and two-sided PT,, while our work focuses on general nonconvex minimax optimization. Nonconvex minimax optimization is also widely studied in existing literature. For example, Xu et al. (2023) studied an alternating gradient projection (AGP) algorithm which applics &p regularizer to the local objective function of GDA followed by projection onlo the constraint sets and achieves iteration complexities O(e -1 and (x)5c= under nonconvex-coneawe and nonconvex-strongly-come
settings, respectively. Nouiched et al.
(2019) studied multi-step GDA with multiple gradient ascent steps per iteration and iteration complexity II((e")) under nonconvex-PPi PL setting and they also studied the momentum-accelleateer version which achieves iteration complexity O 3.5 In(=))) under nonconvex-concave setting. However, these works under nonconvex setting only have convergence guarantee to stationary points, whereas all the variants of our Cubic-LocalMininas algorithm converge to local minimax points. More detailed comparison of the convergence results of the above deterministic first-order algorithms are shown in Section 1.1
Deterministic second-order algorithms for minimax optimization : Adolphs et al. 2019 analyzed the stability of a second-order variant of the GDA algorithm. Huang et al. (2022b) also proposed a cubic
regularized Newton algorithm which solves convex-concave minimax optimization problem with O (In(c="))
and (((1--))(69) iterations respectively under Lipschitz-type and Hölder-type (with parameter 0) error bound conditions, but the computation complexity for solving the cubic subproblem is lacking. In (Luo ee al. 2022), the authors proposed a Minimax Cubie-Newlon (MCN) algorithm that is different from our Lobie-LocalMinima in various aspects, as we elaborate with more details in Sections || and 5 We list three major differences as follows. First, we develop a GDA-basod solver for solving the cubic subproblem which does not require cubic objective evaluation, whereas they use gradient-based solver with Chebyshev
polynomials which requires additional computation. Second, our algorithm requires much fewer gradient. ascent steps in practice since we adopt more adaptive approximation error bounds (see eqs. (3p & 4 ). Third, we develop stochastic version of Cubic-LocaMMinmaa and analyze its sample complexity, which to our knowledge has not been studied in the existing literature. More detailed comparison of the convergence results between (Luo et al., 2022) and our work are shown in Section
1.1
Stochastic algorithms for minimax optimization Lin et al. (0000): Yang et al. (2020) analyzed stochastic GDA and stochastic AGDA, respectively, which are direct extensions of GDA and AGDA to the stochastic setting. Specifically, stochastic GDA algorithm (Lin et al.) 2020) achieves sample complexities
O and O)E-*) under nonconvex--trongly concave and nonconvex-coneave settings respectively. (Yang et al. 2020 proposed stochastic AGDA algorithm and its accelerated version via SVRG variance reduction technique, which achieve sample complexities (x-c-1) and O(N + In(("l))) respectively for finite-sum two-sided PL minimax optimization with N samples. First-order variance reduction reduction techniques have stochastic minimax optimization. For example, SREDA (Xu) lt al.] 2020b) and STORM (Qiu et al., 2020) achieve sample complexities ((x3E=3) and O(E-²) respectively. More detailed comparison of the convergence results of the above stochastic first-order algorithms are shown in Section 1.1 Gradient-free versions of these variance reduction techniques have also been applied to minimax optimization, including SPIDER (Xu et al. 2020c) and STORM (Huang et al. 2022a Compared with our work which has convergence guarantee to local minimax points of nonconvex minimax optimization, the above stochastic algorithms either require strong geometric assumptions such as strong convexity and two-sided PT., or only converge to stationary points of nonconvex minimax optimization.
also been applied to nonconvex strongly-concnt
The CR algorithm dates back to Grieeank 1981), , where global convergence is established. In (Nesterov and Polyak 2006), the author analyzed the convergence rate of CR to second-order stationary points in nonconvex optimization. In (Nesterov) 2008), the authors established the sub-linear convergence of CR for solving convex smooth problems, and they further proposed an accelcrated version of CR with improved sub-linear convergence. (Yue et al. (2019) studied the asymptotic convergence properties of CR under the error bound condition, and established the quadratic convergence of the iterates. Recently, [Hallak and Teboulle (2020) proposed a framework of two-directional method for finding sccond-order stationary points in general smooth nonconvex optimization. The main idea to search for a feasible direction toward the solution and is not based on cubic regularization. Several other works proposed different. methods to solve the cubic subproblem of CR, e.g., (Agarwal et al. 2017 Carmon and Duchi 2019 Cartis et al.) 2011b). Another line of work aimed at improving the computation efficiency of CR by solving the cubic subproblem
Cubic regularization (CR):
4

---

Published in Transactions on Machine Learning Research (03/2023)
with inexact gradient and Hessian information. In particular, Ghadimi et al. (017 proposed an inexact CR for solving convex problem. Also. Cartis et al. (2011a) proposed an adaptive inexact CR for nonconvex optimization, whereas Jiang et al. (2017) further studied the accelerated version for convex optimization.
Several studies explored subsampling schemes to implement incxact CR algorithms, c.g, (Kohler and Lucchi
2017
Xu el. al. 2020a Zhou and Liang 2018: Wang el. al. 2018b).
2
Problem Formulation and Preliminaries
We consider the following standard minimax optimization problem (P), where f is a nonconvex-strongly concave bivariate function and is twice-differentiable. Throughout the paper, we define the envelope function ((x) = maxyeR" f(x,y).
min max [(x v)-
(P)
EER yEB"
Our goal is to develop algorithms that converge to local minimax point of (P), which is defined as follows. Definition 2.1 (Local minimax point Jin et al. (2020)). A point (x",y") .y") a local minimar point of (P) if y² is local marimum of function [(x", ; and there erists a constant ōn > O such that x² is a local minimoun of function ga(z)) = maxy:ll--"1 sá f(x, y) for any å E (0, 6o].
Local minimax points are different from global minimax points, which require xt and y² to be the global minimizer and global maximizer of the functions Φ(.) and f(, simultancously. In general minimax optimization, it has been shown that global minimax points can be neither local minimax points nor even stationary points (Jin et al. 2020). However, global minimax point necessarily implies local minimax point in optimization. Morcover, under mild conditions, many machine lcarning problems have been shown to possess local minimax points, e.g., generalive adversarial networks (GANs) (Nagarajan and Kolter 2017: Zhang et al. 2021 distributional robust machine learning (Sinha al. 2018 etc.
nonconvex-ttrongly-cooe
Local minimax point is also an extension (nccessary condition) of local Nash equilibrium (Jin ct al.) 2020 A point (x', is defined as local Nash equilibrium if f(x", .v) S ffxx",y) fz, v²) for any x, that satisfies fIr = ligó fx. is achieved at (x","") Such zero-duality gap condition is required for simultaneous games where the min-player x and the max-player y act simultancously (Jin et al. 2020). In fact, Jin et al. (2020) pointed out that most machine learning applications are sequential games where min-player and max- max-player act scquentially and the sequence (i.c., who plays first) is crucial and pre-spocified. For example, in adversarial training, classifier acts first and then the adversary generates an adversarial sample. In GAN training, generator acts first followed by discriminator. Moreover, in sequential games, zero duality gap does not necossarily hold, i.c., local Nash equilibrium may not exist, so local minimax solution is more commonly used
-| 8 and Ily y1 S 8. In other words, local zero-duality gap min le=="1|5o maxy:lyy--11l maxx:llo-v'Ilss 6
ming:]]s-=* (
(Jin et al. 2020).
In (Jin et al. 2020 the following set of second-order conditions have been proved to be sufficient conditions for local minimax points. Moreover, as we show later, our algorithm design is inspired by these conditions.
Definition 2.2 (Sufficient conditions for local minimax). point (I. v) is a local minimaz point of (P) if the following conditions hold.
1. Stationary; V1f(z,)) - V2f(2,)) = 0;
2. Non-degeneracy: V22f(2,)) 3) 0, and [Vnf Vief(Ve2)) VVu v) 0.
Throughout the paper, we adopt the following standard assumptions on the minimax optimization problem
(P). These conditions have been widely adopted in the related works (Lin et al. 2020 Jin al. 2020
Zhang
et al. 2021
Assumption The minimaz problem (P) satisfies:
1. Function f(.-)) is Ia-smooth and function f(x,,) is p-strongly concave for any fixed x;;
2. The Hessian mappings Vus, Vi2f, V21,, V22l are Toz Lipphiccl continvuuus;
5

---

Published in Transactions on Machine Learning Research (03/2023)
3. Function ((x) maX f(z y) is bounded below and has bounded sub-level sets.
To elaborate, item considers the class of nonconvex-strongly-oom
functions thatt has been widely
studied in the minimax optimization literature (Lin et al. 2020 Jin et al. 2020 Xu et al. 2023 Lu al., 2020), and it is also satisfied by many machine learning applications. Item 2 assumes that the block Hessian matrices of f are Lipschitz continuous, which is a standard assumption for analyzing second-order optimization algorithms (Nesterov and Polyak 2006 Agarwal et al. 2017). Moreover, item 3 guarantees that the minimax problem has at least one solution.
3 A Cubic Regularization Approach for Finding Local Minimax Points
In this section, we propose a cubic regularization type algorithm that leverages the cubic regularization technique to find local minimax points of the nonconvex minimax problem (P). We first relate local minimax points to certain second-order stationary points in Section 3.1 based on which we further develop the algorithm in Section 3.2
3.1 On Local Minimax and Second-Order Stationary Condition
Regarding the conditions of local minimax points listed in Definition 2.2 note that the stationary conditions in item 1 are easy to achieve, e.g., by performing standard gradient updates. For the non-degeneracy conditions listed item 2, the first condition is guaranteed as f(x, -) is strongly concave. Therefore, the major challenge is achieve the other non-degeneracy condition [Vnf Vi2f(V22f)
(z, 3) > Interestingly, minimax optimization, such non-degeneracy condition has close connections to certain second-order stationary condition on the envelope function ((x), as formally stated in the following proposition. Throughout, we denole LI/1 as the condition number.
nonconvex-strongly-comm
Proposition 3.1. Let Assumption
hold. Then, the following statements hold.
1. The mapping y" (x)) == arg maX f(x, v) is unique and k-Lipschitz continuous for every fined el al. 2020;;
Lin
2. (x) is (1 + smooth and ( = vxf(zx [Lin et al. 2020
3. Define mapping G( v) Vuff- V 112f -Va Then, G is Lipschitz continuous mapping
with Lipschitz constant La = L2 x)²;
4- The Hessian of + satisfies <2 G(r,y (I:)), and it is Lipschitz continnous with Lipschitz constant
L+ Lo (1+ = +))3
The above proposition points out that the non-degeneracy condition G(x, y) > actually corresponds to second order stationary condition of the envelop function ((x). To explain more specifically, consider pair of points (e.v"(=)), in which y (x) == arg max, f(x v). Since f(x,-) is strongly concave and y (x) is the maximizer, we know that y (x) must satisfy the stationary condition V2/(I, y (x)) = 0 and the non-degeneracy condition 222f(1, y (z)) x O. Therefore, in order to be local minimax point, x must satisfy the stationary condition [If(x, y² (x)) = and the non-degeneracy condition G(z, y"(x) > O, which, by 3.1 are oquivalent to the of second-order stationary conditions stated in the
items 2 and 4 of Proposition
following fact.
Fact (Eotushenko 1974) Let Assumption 7 hold. Then, (a, y² (x:) is a local minimax point of (P) if satisfies the following set of second-order stationary conditions.
(Second-order stationary): ( 0, V²0(2)>
To summarize, to find a local minimax point in nonconvex--trrogyyyoommm find a sccond-order stationary point of the smooth nonconvex envelope function ((x). Such a key
minimax optimization, it suffices
observation the basis for developing our proposed algorithm in the next subsection. We also note that the proof of Proposition 3.1 is not trivial. Specifically, we need to first develop bounds for the spectrum

---

Published in Transactions on Machine Learning Research (03/2023)
norm of the block Hessian matrices in Lemma Lipschitz continuity of the G mapping in item 1. Moreover, we leverage the optimality condition of f(x,, :) to derive an expression for the maximizer mapping y (x) (see eq. (46) in the appendix), which is used to further prove item 2.
A.1 (see the first page of the appendix), which helps prove the
3.2 The Cubic-LocalMinimax Algorithm
In the existing literature, many second-order optimization algorithms have been developed for finding second- order stationary points of nonconvex minimization problems (Nesterov and Polyak 2006 Agarwal et al., 2017: Yue al., 2019. Zhou al., 2018). Hence, one may want to apply these algorithms to minimize the nonconvex function ((x) and find local minimax points of the minimax problem (P). However, these algorithms are not directly applicable, as the function ((x) involves a special mazimizalion structure and hence its specific function form & well as the gradient V and Hessian 22 are implicit. Instead, our algorithm design can only leverage information of the bi-variate function f.
Our algorithm design is inspired by the classic cubic regularization algorithm (Nsterrvv and Polyak 2006 Specifically, to find a second-order stationary point of the envelope function 2(x), the conventional cubic regularization algorithm would perform the following iterative update.
1 8++1 arg min Vo(r:) s+ 2² V2)((x))
1 61
||&|13
Te+1 It + St+1>
(1)
where The > 0 is a proper learning rate. However, due to the special maximization structure of 4, its gradient and Hessian have complex formulas (sce Proposition 3.1 that involve the mapping y* (z), which cannot be computed exactly in practice. Hence, we aim develop algorithm that efficiently computes approximations of (T)x), (z),,, and use them to perform the cubic regularization update.
To perform the cubic regularization update in eq. ①1. we need to compute V = VIf(z,, y (xx)) and (2)( = G(v,, y" (x+)) (by Proposition 3.1 1 both of which depend on the maximizer y" (ax) of the function f) Since .x),,) is strongly-concave, we can run N₂ iterations of gradient ascent to obtain an approximated maximizer UN, & y (xre, and then approximate (*));,) V²((I;) using Vis(ze, WN.) and G( UN.), respectively. Intuitively, these are good approximations due to two reasons: (i) ÜN. converges to y" (axa at fast linear convergence rate; and (ii) both Vif and G are shown to be Lipschitz continuous in their second argument. We refer to this algorithm and summarize its update rule in Algorithm | below, which terminales whenever the maximum of the previous two increments [|s:-]]] V [|sæ|] is below a certain threshold e'. Such an output rule helps characterize the computation complexity of the algorithm. In Scction 5 we provide a comprchensive discussion on how to solve the cubic subproblem with the special Hessian matrix G(ar, M6+1) using first-order GDA- type algorithms.
Cubic Regularization for Local Minimax
(Cubic-LocalMinnng)
4 Convergence and Iteration Complexity of Cubic-LocalMinine
In this section, study the global convergence properties and the iteration complexity of Cubie-LocallMnnmax The key to convergence analysis is characterizing an intrinsic potential function of Cubie-LocalMinmmx in nonconvex minimax optimization. We formally present this result in the following proposition.
Proposition 4.1 (Potential function). Let Assumption hold. For any a > O, choose e' S aLi
3L0 .
Th S (9L + 18a + 28 and My = 2_ Define the potential function H, == ((x;) + (Lg + 2a 3,
Then, when N₂ 2 (k
, the output of Lubic-LocalMimmmar satisfies the following
LoA022
potential function decrease properly for all E.
H - H. < - (Lv +a ()) [|sx
Proposition 4.1 reveals that Cubic-LocaLMinimas admits an intrinsic potential function He, which takes the form of the envelope function ((x)) plus the cubic increment. term ||s:||3. Moreover, the potential function H.
7

---

Published in Transactions on Machine Learning Research (03/2023)
Algorithm 1 Cubie-LocalMinines
Input: Initialize zo: yo, learning rates T/ fly threshold d', numbers of iterations T, No Define | =
for -= 0, 1.1.2.... ...,.
do
Initialize =
for k 0, 0, ..,, N₁- do
ūk+1 = Un + 17 V2f (x)
end
Set M/41 VN., Solve the cubic problem for St+1:
argmin Vif( + 2.h G(Fe.Ne+1)8
||
61/
Update Tz-1 It. 81+1
end
Output: ,T","', T min ||-11 V
s S c
is monotonically decreasing along the optimization path of Cubie-LocalMinnna,, implying that the algorithm continuously makes optimization progress.
The key for establishing such a potential function that, by running a sufficient number of inner gradient ascent iterations, we can obtain sulficiently accurate approximated maximizer y++1 (xxx) Consequently, the f vt+1) and G(Ia,314)) involved in the cubic subproblem good approximations of ( and <x;((r;; respectively. In fact, the approximation errors are proven to satisfy the following bounds.
[V - Vif(z,, M(+1) S 12 [ M(1)) (1|ss + +()).
(3)
(4)
On one hand, the above bounds are tight. enough to establish the decreasing potential function. On the other hand, they are flexible and are adapted to the increment || = || xt- produced by the previous cubic update. Therefore, when the increment is large (i.e., || = 0(1) > O(e)), which usually occurs in the early iterations, our algorithm requires only O(1) gradient ascent steps. As a comparison, the Minimax Cubic-Newton (MCN) algorithm proposed in (Luo et al. 2022) adopts cotstant-level approximation errors, i.e., Mt+1) || S O which requires much more gradient
( (Z Vt+1)ll O(e²) and [V² ascent steps (Oln(1/ee)) in every iteration. In the converging phase where ||sı|| < O(c), algorithm requires ((I((/ee)) gradient ascent steps, which is of the same order that of MCN. Combining the two cases, our algorithm is more practical. Such an idea of adapting the inexactness to the previous increment in eqs. (3) and 4 is further leveraged to develop a scalable stochastic variant of Cubie-Local in Section
we obtain the following global convergence rate of Cubie-LocalMinmmax a second-order stationary point of &. Throughout, we adopt the following standard measure of second-order stationary introduced in (Nesterov and Polyak 2006).
Based on Proposition
mxx) = / [ Amiin (V2((x)
/33L
Intuitively, smaller plz) means that the point x is closer to being second-order stationary.
Theorem (Convergence and complexity of Cubie-LocallMmnmass) Lel the conditions of Proposition 4.1
hold with a = B = L$. For any 0 < < LIv33Le
choose e' = V33Lx and T 2 4(zo)- 48Lae"
Then, the
La
3Lae
output of Cubic-Locallinimaaz satisfies
u(Tr) S
(5)
1Qur d' corresponds O(vP)
2022
8

---

Published in Transactions on Machine Learning Research (03/2023)
Consequently, the total number of required cubic iterations satisfies T S OVVLxk and the total number of required gradient ascent ilerations salisfies Et N₂ S (VVTnn2
Remark 1:
The convergence rate result. in Theorem 1 is about the terminated iteration T", which is specified by stopping criterion and is upper bounded by T. This is different from most of the existing last-iterate convergence results where the last iterate T is prespecified.
We note that the gradient ascent steps for updating Vk+1 in Algorithm | can be accelerated by using the standard Nesterov's momentum. In this way, the total number of requirod gradient ascent iterations will reduce to the order O(VIane which matches that of the Minimax Cubic-Newton algorithm proposed in (Luo et al., 2022).
Remark 2:
The above theorem shows that the gradient norm [+)())) vanishes at sublincar rate 0), and the second-order stationary measure -mmin (z)) converges at sublinear rate O(T-$). Both results match the convergence rates of the cubic regularization algorithm for nonconvex minimization (Nesterov and Polyak, 2006). As a comparison, the standard GDA does not guarantee the convergence of -Amin (v² and its convergence rate of [[V](=))]] is of the order O(T- (Lin et al. 2020), which is orderwise slower than that of Cubie-LocalMinimaat Therefore, by leveraging the curvature of the approximated Hessian matrix G( Cubic-Locke Lubie-LocalMininas. is able to find second-order stationary points of & at fast rale.
We note that the proof of the global convergence results in Theorem ! is critically based on the intrinsic potential function He that we charactorized in Proposition 4.1 Spocifically, note that the cubic subproblem in Cubie-Local involves an approximated gradient V (ir,, M/++1) and Hessian matrix G( Dt+ Such inexactness of the gradient and Hessian introduces non-negligible noise to the cubic regularization update of cannot make monotonic progress decreasing the function value C as opposed to the standard cubic regularization algorithm in nonconvex minimization (which uses exact gradient and Hessian). Instead, we take different approach and show that as long as the gradient and Hessian approximations are sufficiently accurate, one can construct a monotonically decreasing potential function H, that leads to the desired global convergence guarantee.
Cubie-LocalMmimax, Consequently, Cubic-Locallininee
5
How to Solve the Cubic Subproblem of Cubic-LocalMinimax?
The Cubie-LocalMinimax presented in Algorithm involves a cubic subproblem that takes the following form.
1 2 arg min ((8) ;= gT 1 + 8141 arg
1
As
|s||3,
(6)
where g V VIs(I., A H11 H H12l1 H21 with Her V Viel (
To solve the above cubic subproblem, one standard approach is to apply the existing gradient-based solvers (Carmon and Duchi 2019 Tripuraneni al. 2018). which requires computing the Hessian-vector product A 8. However, in Cubie-LocalMinimas the Hessian matrix takes the complex form A H HH2H Ha that involves product of block Hessian matrices as well matrix inverse. Hence, directly computing the product of such a Hessian matrix with any vector can be highly inefficient. On the other hand, in (Luo et al., 2022), the authors proposed two-timescale update rules for computing such Hessian-vector product, where they approximate the matrix inverse Hzy 1 via Chebyshev polynomial?? ? To further simplify these update rules and reduce computation, we next propose an efficient GDA-type algorithm to solve this cubic subproblem with the special Hcssian matrix A.
Our main idea is to reformulate the cubic subproblem in order to avoid the matrix inverse H involved in the Hessian matrix A. Specifically, we observe that the above cubic subproblem can be rewritten as the following minimax optimization problem.
1 min max ( = g s 20 H + T H1 + 20 Hyzu +
1 ||s||3.
(7)
67
28ee eqs. (32)-(33) of Laoo al. 2022
9

---

Published in Transactions on Machine Learning Research (03/2023)
To explain, note that the above bi-variate function & is strongly concave in with the unique maximizer given by 0*())= == arg max, ((8, 1) = H218. Substituting this maximizer into the function ((8,) yields the original cubic subproblem, i... ((5, (s)) = o(s). Moreover, since @(s, u) is a nonconvex-strongyyycom function (because Hz < -,T), we are motivated to develop a GDA-type solver to solve it. Specifically, our solver, named as GDA-Cubic Solver, partially inspired by the existing gradient-based cubic solvers (Tripuraneni et al.) 2018) and is summarized in Algorithms 2 and 3
To elaborate, the solver performs updates
based on the following two cases.
Large gradient ||||| 2 4L3x"na: In this case, the first-order gradient is far from being stationary, and it is more preferable to constrain the solution of the cubic subproblem in eq. (6) to s= T 9 for some
y >>> 0 (Tripuraneni et al. 2018). In particular, the optimal choice y", named as Cauchy radius, has been shown in (Conn et al. 2000 to take the following form.
7" arg min @ (-7 -
9 =
=-(( (V"11/// A Ag)² + 2
"a9" Ag
(8)
||g||
720
T Ag with A = H1I H12ll 22 H21, we propose to rewrite it as [gll
Here, to compute the quantity
g T Ag
9 H1g
(H219) T=" where w" == H2 |g]||
H2ı
(9)
||/|2
||al2
Note that both Higg and H2ng are Hessian-vector products that can be efficiently computed by the popular machine learning platforms such TensorFlow (Abadi 2015) and PyTorch (Paszke 2019) To compute w" note that it can be viewed as the unique maximizer of the j-strongly concave problem max fwo T H22 (H210) [a w. We can solve this problem by performing K gradient descent steps (see eq. )10)) and obtain an approximated minimizer R w² with high accuracy.
Small gradient II9ll < 4L3 In this case, propose to solve the equivalent cubic subproblem in eq. via a nested-loop GDA-type algorithm, as it is nonconvex-strongly-oneee maximize (ss,-) via gradient ascent for multiple iterations to estimate the maximizer (*(s) (see eq. 11
Specifically, first. fix and
Then, we fix and minimize e)) via one step of perturbed gradient descent (see eq. 12
We note that all the steps of GDA-Cubic Solver are based on computing Hessian-vector products, which can be efficiently computed and does not require storing the Hessian matrix. Equipped with this GDA-Cubic Solver, we propose the following Inexact Cubie-LocalMinimax algorithm summarized in Algorithm
A similar algorithm named as Inexact Minimax Cubic-Newton (IMCN) with inexact cubic solver was first. proposed by (Luo et al. 2022). Our Algorithm differs from IMCN in the following aspects.
First, as mentioned in Section 4 adopt the more relaxed adaptive gradient and Hessian approximations in eqs. & (4p for the gradient ascent steps, whereas they adopt constant approximation errors.
(Algorithm 4 and our GDA-based cubic solver (Algorithm 2) adopt simple termination rules that are purely based on tracking the norm of the increments || [|se]], which are directly accessible in each iteration. As a comparison, the termination rules of their Inexact Minimax Cubic-Newtor Algorithm and its cubic solver need to additionally track the approximate objective function value of the cubic subproblem, which requires additional computation.
Second, both our Inexact
We obtain the following overall computation complexity result of Algorithm
Theorem 2 (Computation complexity of Inexact Cubie-LocallMnnnaaa) Let Assumplion hold. For any 0 L 1/2 2//2 Lan?) and 8 (0,1), choose 106VL. T =e(-== (#(z)) x- + 3))
S min 228.L4 6
Ln
2 and N = = Lia Lal T. = -(L"'), ny trth implementing Algorithm 2 at the t-th iteration, use hyperparameters in Lemma G. with 8' [ 3111)l < x Tr and use those in Lemma G.2 otherwise. When implementing Algorithm 3 use the hyperpnrameter choices in Lemma G.S. Then, with probability at least 1 - 5, the output of Ineract
(see eq. (45) in Algorithm 4 When
Lor2
8/T if
10

---

Published in Transactions on Machine Learning Research (03/2023)
Algorithm GDA-Cubic Solver
Input: Gradient g. Hessians H11, H12, H22, perturbation magnitude 0, learning rates fe Bls Ta. numbers of
iterations K, (=)
if |لel| KL 7/2 then
we for l 0, K do
H219
Wk+1 wk + 1/0
H22
(10)
llg1l
end
(H21g) WK
8K |1
111 [
7K (11,3k) )2
2m |1
6'k NK9
else
Obtain Uniform( (I E R"
|||||| 1}).
for k = O...., K - 1 do
0k,0 = 0 for L = o,..., No4 - do
0k U2,e + n, + H22
(11)
end
0/ = U
|1 5/
H12 + 1/1 56 (1t + H11
(12)
2n
end end Output: sk-
Algorithm GDA-Cubic FinalSolver
Input: Gradient 9, Hessians H1, H12, H2n, learning rates Ya Jus Us numbers of itcrations K, [N
for 0,..., K do
Uk.0 0 for f 0, ..,,, No do
Obtain 04,++1 using eq. end
11 with learning rate 7/--
= Uk.NK
|1 s/
9k -9 + H1S
H12 +
(13)
27z
1441 =//6 7.9k
(14)
end
Output: 8'2, K" min(k: ||gs|] & T
11

---

Published in Transactions on Machine Learning Research (03/2023)
Algorithm 4 Inexact Cubic-LocalMininax
Input: Initialize To: yo, learning rates "/s fly threshold d', numbers of iterations T, No Define | = for -= 0, 1.1.2.... ...,.
do
Initialize =
for k 0, 1, ...,, No-
do
ūk+1 = Un + 17 (2f
end
Set M++1 V/N
Approximately solve the cubic problem argmin, (I. 1/ G(r 6n. ||ss1 for 3441 using
2 with g : VIs(Fettt1) and Hyr == Vkes(Tt, (k, l (1,2)) Update It-1 It + 3++1 if | V
Algorithm
| S é' then
Obtain using Algorithm 3 with g Vi VIs( and HEr = VEel (k, E (1,2))
T' == min(t || | S Se) +
Tr = 2r -1
Output: ,
end
end
Cubic-LocalMtnimar satisfies
((Tr)) S..
(15)
Consequently, the total number of required cubic iterations satisfies T S 00 (V(Ixk)))))) the total number of required gradient ascent iterations sotisfies Σt- N, O(vLzx223 and the total number of required Hessian-vector product computations (in Algorithms
&
30 is of the order H(L)n²E
We can apply the standard Nesterov's ' momentum to further accelerate the convergence rate of the gradient ascent steps of Algorithms 2 and The resulting total number of gradient ascent iterations 4),
Remark:
and total mumber of Hessian-vector products will then be improved to ÓVLLzk2 L e-) and respectively, which match those of the Inexact Minimax Cubic-Newton algorithm in (T.uo et al. 2022
Compared with Theorem] 1 it can be seen from the above Theorem 2 that the total number of cubic iterations and that of gradient ascent iterations remain the same, demonstraing the effectiveness of our proposed CDA-based cubic solver. Moreover, since algorithm design and cubic solver design are different from those of (Luo et al. 2022). our convergence proof of Theorem is therefore substantially different from that of (Luo et al. 2022 in the following aspects.
First, our Algorithm adopts the more relaxed adaptive approximation criteria (3 & 4 to save computation in practice, and thus cannot guarantee monotonic decrease of ((x)). Instead, we establishod ((x++1) (xxe) -1111 Sax + 11,3) Se+ || + (9L4 6a + 9.3) | (see eq. (88),, which implies that
our constructed potential function H, == (xx)) (10L + 7a + 103) | |3 is monotonically decreasing
shown in Proposition G.4
Second, our Inexact Cubic- Lobbi-LocalMiimmax (Algorithm 4 uses the termination rule ||ا-1||| V | Sé' that only relies on the norm of the increments, we need to prove || second-order stationary condition μхx)) S e. In particular, we have proved eq. (76 in Lemma G.2 which implies that || < é' cannot hold under large gradient, so we conclude that || Vif((rr gT++ S In this small gradient case, eq. (64) we proved
|I, ||ك||| s O(c') in order to ensure the
4L7 Lemma G.1 implics that the exact CR solution ST satisfies [|sr || < 3e', which combined with the final cubic solver (Algorithm 3 yields that the final CR solution must satisfy eq. (82) (i.e., ||ح]|| S 7e') in Lemma G.3 As a comparison, the Inexact Minimax
12

---

Published in Transactions on Machine Learning Research (03/2023)
Algorithm in (Luo et al. 2022 terminates based tracking the objective function value of the cubic subproblem, which requires additional Hessian-vector product computation in practice and does not involve these technical developments.
6 Stochastic Cubic-LocalMinimax
In this section, we apply stochastic sub-sampling to further improve the performance and complexity of Cubie-LocalMinimas in large-scale nonconvex minimax oplimization with big data. Specifically, we consider the following stochastic finite-sum minimax optimization problem (Q).
N
min max f(r,y) == #ER YER"
Σ f( v), N
1
(Q)
where N denotes the total number of training samples and fi corresponds to the loss function on the i-th sample. We adopt the following assumptions.
Assumption 2. The stochastic minimaz optimization problem (Q) satisfies:
For any sample i, function Ii(:s)) is IA -m-sooth, function fe(:,)) Ta-Tisschitz for anly fixed y, and function fi( p-strongly concave for any fixed x;
2. For any sample i, the Hession mappings Vi1fs, V12fi, V21fi, V22fi are L2- Lipschitt continuous;
3. Function $(x) == maxyeR= f(x, y) bounded below and has bounded sub-level sets.
solve the above problem (Q) would require querying full partial gradients and full Hessian matrices that involve all the training samples. Instead, it is more efficient to approximate these quantitics via stochastic sub-sampling. Therefore, we propose stochastic Cubie-LocalMinimax algorithm, which replaces the exact quantities Vn and G involved in the cubic update by their corresponding stochastic approximations
Applying Cubie-LocalMnnmaa
Specifically, to approximate the partial gradient Vif, we sub-sample a mini-batch of samples B with replacement from the training set and construct the following sample average approximation.
vif(z.v)= = B1
Vifi(1,9..
(16)
iEB
On the other hand, to approximate the matrix G, sub-sample mini-batches of samples B11 B12, B21 B2 with replacement and construct approximated Hessian matrices Vn1f, Vi2f, Vanf V22 in the same way above. Then, we construct the following approximation of G.
G(z,v)
(17)
We summarize the update rule of stochastic Cubic-Local in Algorithm 5 below. In particular, we run stochastic gradient ascent (SGA) in the inner iterations to obtain the approximated maximizer yt+1s and its high-probability convergence rate has been established in the existing stochastic optimization literature Harvey et al. 2019) as shown Theorem below.
Theorem 3. Let Assumption 2 hold. For all t, k, assume that 2/ Wt)l Lo and (2 7k) (ef Telll < 1 almost surely. The inner stochastic gradient ascent steps in Algorithm converge at the sollowing rale with probabilily at least --.
11941 vxooll (
(0(
Lo ln(1/8) + L p²N,
The following lemma characterizes the sample complexities of all the stochastic approximators for achieving a certain approximation accuracy.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Algorithm 5 Stochastic Cubic-LocalMininax
Input: Initialize T0: yo, learning rates !/e, threshold c', numbers iterations T, N₂ Define | =
for -= 0, 1,2...... ...
do
Initialize =
for k = 0, 2,.,..., .N. do
Query sample to compute V2fe (2t, Dkc)
Mx+1 = Dk +
1lk +
end
Set Mo+1 Lk=0 N1
N111
24
Sample minibatch Bi (4), (n),), (1) B2i (1), Br2(() to compute eq.
16 and eq.
Then, solve the
following cubic problem for St11
1 5
argmin ( 8
G(s
||
61/
Update Ft-1 Ft 8141
end
Output: ,T with T = min | V | S St}
Lemma 6.1. Fir any 0 <1 s 2Lo, E25 4h and choose the following batch sizes
m
B 2
In
(18)
2
[Bul. B12), BBal, BBal 2 0
In
m
+ 11
(19)
Then, the slochastic approzimalors salisfy the following error bounds with probubility at least 1 6.
(v.f(x,)) Vif(2,8)1 S1, (a./(,,,)VVi/lee w)ll Se,, vk, E {1,2). [[G(z,)) G(z, v)ll -G(,,
(20)
(21)
s (x+ 1)
(22)
Therefore, by choosing proper batch sizes, the inexactness of the stochastic gradient, Hessian and Hessian estimators can be controlled within desired range. From this perspective, stochastic Cubie-LocalMnnnmat can be viewed as an inexact version of the Cubie-LoocalMinimax algorithm.
To characterize the convergence and sample complexity of stochastic Cubic-LocalMinnmas,, we adopt an adaptive inexactness criterion for the sub-sampling scheme. Specifically, we choose sime-varying batch sizes in way such that the gradient and Hessian inexactness iteration are proportional the previous increment, i.e,, (t) x 1| (t) ||s:||- Such an adaptive inexact criterion has been justified in the cubic regularization literature (Wang et al. 2018a 2019) with the following advantages: 1) it is adapted to the optimization increment and hence leads to reduced batch sizes when the increment is large in the early iterations; 2) it makes the batch size scheduling scheme in Lemma 6.1 practical, as the batch sizes in iteration now depend on the increment ||se|| obtained in the previous iteration / -- 1. We also note that since the sub-sampling scheme is adapted to the previous increment, the output rule Algorithm 5 is designed to control the value of both the current and the previous increments. This termination rule critical to bound the adapted gradient and Hessian inexactness the analysis.
We obtain the following global convergence and sample complexity result of stochastic Cubie-LocalMinmmas Theorem 4 (Convergence and sample complexity). Let Assumption and Theorem hold. For any 0 <
La /s3La. choose e' V33L0' S 55 T
((me) 48 and N 2 O Lm
Lg
3Loe3
14

---

Published in Transactions on Machine Learning Research (03/2023)
Morcover, in iteration choose the batch sizes according to eqs.
18
and
19
with the ineractness given by
(1)) L
|1s2112
33I. 3aimm)
e² 2
22
L 2(K + 1)²
(1sull
A4T11
v33L
Then, the output of Stochastic Cubie-Localliinmma satisfies
p(or)) c.
(23)
Consequently, the total numbeer of cubic iterations satisfies T S O gradient samples salisfies E (Ne + |B1 (1)) S 0
(12 the total number of queried and the total number of queried Hessian m
Lox VL2
samples satisfies ET- E E (t)e(t) & <0(
LiNa. Vlge&
m
Therefore, under adaptive sub-sampling, the induced gradient and Hcssian inexactness (1(t), c2(t) are properly controlled so that the ileration complexity T" of stochastic Cubie-LocalMinmma remains the same level as that of Cubie-LocalMinnmaa. Moreover, as opposed to the sample complexity of Cubie-LocalMimmmas that scales linearly with regard the data size N, the sample complexity of stochastic Cubie-LocalMininas is independent of N. For example, by comparing the more expensive Hessian sample complexity between and stochastic
Cubic-LocallMngg
we conclude that stochastic sub-sampling helps
reduce the sample complexity so long as the data size is large, i..., N 2 Ō L3
L
7 Experiments
In this section, we test the numerical performance of Cubie-LocalMinimax and compare it with existing algorithms in solving synthetic minimax optimization problem and an adversarial deep learning problem. All the experiments are implemented on Google Colab with Intel(R) Xeon(R) CPU (12 cores, 2.20GHz), A100 GPU of cuda 11.2, and 83.48 GB memory. The code can be downloaded from https: gi github....... com/ http:r/e--...mmmmmmmmm-eeeeeeeee wewpmmaxx-xxxemmtttt...
7.1 Synthetic Minimax Optimization Problem
We consider the following finitesum minimax optimization problem with parameters [21, E R and y = [2] E 2
1 N min max f(x, E1(,, with f((x,y) w(ss)
5y2 vी + A,31 40
+ B
(24)
FEB DERa
2
where Ar, B, are independently drawn from a uniform distribution over the interval 1.55, N 1000 is the total number of samples, and the function w(.) is W-shaped nonconvex function whose exact form is presented in Appendix H.T In this setting, each function Sa is nonconvex-strongly-comm
We apply our stochastic Cubie-LocalMinimax algorithh to solve the above synthetic problem. We choose the initialization point to be x = [0.1; (2) = [2) to be 20, 100, 1000, respectively. We choose the number of gradient ascent steps N, = for cach outer loop, the strong concavity constant μ 1, and choose the learning rate 7/2 0.01. To solve the cubic subproblem, we use the standard gradient descent with learning rate 0.01, as the gradient and Hessian of (zx) can be analytically computed for this synthetic problem.
[1;1] and set the batch size [t) = (nt(t) = [B2((t) =
Figure shows the performance at each epoch (cach update of x is considered as an epoch) (left figure) and time complexity (right figure) of stochastic Cubie-LocalMinnmax under different batch sizes. Here, the y-axis denotes the function value of ((x)= = max, f (a, y), which we aim to minimize. It can be seen that stochastic Cubie-LocalMinmaa takes more epochs but much less time to converge under smaller batch size. This is because the synthetic minimax problem involves relatively small noise so that the stochastic gradients computed over small batch of samples sufficiently accurate.
15

---

Published in Transactions on Machine Learning Research (03/2023)
Bacchsoe B
Datchazee
100
0.1
e
10
20
0.2
0.4
epochs
Time(s)
Figure 1: Performance at each epoch (left) and time complexity (right) of stochastic Cubie-LocalMinimaa under different batch sizes. The y-axis denotes the function value of ((x) maxy (x, y) that we aim to minimize.
We further compare the performance of stochastic Cubie-LocalMinimax with that of the standard stochastic GDA and second order algorithms including GDN /FR////////AA (Zhang al. 2021 and Minimax Cubic- Newton (MCN) algorithm (Luo al 2022]. All the algorithms use batch size 100 and the fixed initialization point x [0; 0; 1), [1;1]. For both stochastic gradient ascent steps N, = 10 and == 1. We do one step of cubic descent and one step of gradient descent in each algorithm, respectively. The difference between the two algorithms that the cubic solver in Algorithm is replaced with one gradient descent step, and both cubic solver and gradient descent step have the same learning rate 0.01. The implomentation details of GDN/TGDA/ CC/CN CN algorithms are a bit involved and refer them to Appendix H.. For the MCN algorithm (Algorithm 2 of (Luo et al. 2022 we apply 10 Nesterov's accelerated gradient ascent steps y with learning rate r7 0.5 and momentum coefficient 0 0.5.
and stochastic GDA
choose the
Figure 2 shows the comparison of the performance al each epoch and time complexity for both algorithms. It can be seen that our stochastic Lubie-LocalMinimat has comparable performance to CN and MCN, but converges faster than all the other algorithms under both number of epochs and time complexity measures, demonstrating the effectiveness of both the cubic regularization approach and our proposed GDA-Cubic Solver.
0.15
0.15
GDA
GDA
Qubic LocalMinimax
Cubic LocalMinimax
TGDA
GDN
GDN
0.1
CN
0.1
CN
MCN
MCN
0.05
0.05
20
40
60
80
100
120
0.2
0.4
epochs
Time(s)
Figure 2: Comparison of all the algorithms for synthetic experiment.
16

---

Published in Transactions on Machine Learning Research (03/2023)
7.2 Adversarial Deep Learning
We further compare the stochastic Cubic-Locall Minimax with the classical GDA and second order algorithms including GDN/FR CDN/FR/TGDA RR/DAAA///G (Zhang et al. 2021 and Inexact Minimax Cubic-Newton (IMCN) algorithm (Luo et al. 2022) in the application of adversarial deep learning, which aims to train an adversarially-roban convolutional neural network model (see Section H.2 for the detail of the neural network structure) by solving the following minimax optimization problem using the MNIST dataset with 50k training samples and 10k
test samples.
1
min max (((ho(Ee.us)) Alls: - -x:ll3]
(25)
(E
where 50k is the number of training samples, 0 is the parameter of the neural network ho, (xi Vi) corresponds to the i-th image and label respectively, & refers to the adversarial sample corresponding to and we choose cross-entropy loss function as l and penalty coefficient À = 2.0.
When implementing stochastic Cubie-LocalMmninx (Algorithm 5), we choose batchsize [Bi (n( [Ba = [Bz (4) [2) = 512 and implement N, = 20 gradient ascent steps with learning rate 0.1. For the cubic solver, we implement A Algorithm with 7/ = when || > 10 untill = iterations reached or [[ Hz12 || < 10-3 in the update rule 100; Otherwise, we choose
0 (i.e. no random perturbation), K 30, 72. = 0.01, = 0.002, 7/- = 0.1 and Ng = until either = 30 iterations is reached or the gradient terms of both 5/2 and u are sufficiently small (i.e., max (2ss + wk. ||g H11s4 + H12 +
lg
||= (
2n1
10-3 in the update rules 11 & 12 When implementing the classical GDA algorithm, we replace the cubic with one stochastic gradient descent step to update 0 using also
solver of stochastic Labic-LocalMingg batchsize 512 and learning rate 0.01, while the rest hyperparameters are not changed. The implementation details of GDN/TTO TGGAA/F//C algorithms are bit involved and we refer them to Appendix H.3 For IMCN algorithm (Algorithm 3 of (Luo et al. 2022)). ), we use 1, = 0.4 and the small gradient case of cubic-solver with learning rate 0.002 and early termination rule ||Vc(>)] 0.001 where the cubie-regullariztion objective function. The Chebyshev polynomial used to compute Hessian-vector product is truncated to K" = 5 terms.
Cubic-LocalMinimax
2-
Robust test accuracy
0.8
GDA
GDN
CN
M
LocalMoaca/i
0.6
TGDA
GDA
IMCN
GDN
0.4
CN
TGDA
0.5
0.2
IMCN
20
40
60
80
100
20
40
60
80
100
epochs
epochs
Figure 3: Comparison of (xx) at each epoch for adversarial deep learning. GDA, TGDA and FR conincide.
Figure 4: Comparison of robust test accuracy al each epoch for adversarial deep learning. GDA, TGDA and FR conincide.
Figure compares the objective function value (xx) at each epoch of both algorithms, which is estimated via 40 gradient ascent steps with learning rate 0.1 to obtain the approximate maximizer (c,) can be seen that our stochastic Cubie-LocalMinmmax algorithm and IMCN are comparable and both algorithms have significantly faster convergence in terms of epoch than the other algorithms. Figure 4 further demonstrates
17

---

Published in Transactions on Machine Learning Research (03/2023)
Qubic-LocalMinimax
Cubic-LocalMinimax
2.0
GDA
Robust test accuracy
0.8
GDN
CN
CN
1.5
TGDA
TGDA
FR
FR
IMCN
1.0
0.2
1M
2M
6M
7M
8M
2M
3M
4M
5M
7M
8M
# of oracle calls
# of oracle calls
Figure 5: Comparison of ((x) per oracle call for adversarial deep learning.
Figure 6: Comparison of robust lest accuracy per oracle call for adversarial deep learning.
Cubic-LocalMininng
Cubic-LocalMinimax
2.0
GDA
Robust test accuracy
GDA
GDN
GDN
CN
1.5
TGDA
0.6
TGDA
FR
IMCN
IMCN
1.0
0.4
0.5
40
10
20
30
40
Time (s)
(s)
Figure 7: Comparison of (x)) on time complex- ity for adversarial deep learning.
Figure 8: Comparison of robust test accuracy on time complexity for adversarial deep learning.
the advantage of stochastic Cubic- Lobbi-LocalMinimax algorithm and IMCN in robust test accuracy which is estimated on test samples. can be seen that the robust models trained by stochastic Cubie-LocalMinimax and IMCN are also more robust in generalization.
Figure 5 and i compare the objective function value (xx) and robust test accuracy respectively per oracle call (i.e., evaluation of gradient Hessiann-vector product) of all the algorithms. Similarly, Figures and compare (zz) and robust test accuracy respectively on time complexity of all the algorithms. It can be seen from these figures that our stochastic Dabbie-LocaaMininege algorithm takes significantly fewer oracle calls and less time than all the other algorithms to converge in both objective function and robust test accuracy.
8 Conclusion
We developed cubic regularization type algorithms that globally converge to local minimax points in nonconvex- strongly-concave minimax optimization. These algorithms include the basic Cubic-LocalMinnnes,, Inexact. Cubic-LocalMinimax with our proposed GDA-based cubic solver and stochastic Cubie-LocalMinnmax for large- scale minimax optimization. By designing and leveraging an intrinsie potential function that monotonically decreases over the iterations, have obtained the computation sample complexities required by each algorithm to achieve 6-approximate local-mininax point. Experimental results demonstrate faster convergence of our stochastic Lubic-LocalMiniaa than the standard stochastic GDA algorithm. A future direction is to extend the proposed algorithms to bilevel optimization, which is a generalization of minimax optimization.
18

---

Published in Transactions on Machine Learning Research (03/2023)
References
Abadi, M. (2015). TensorFlow: Large-scale machine learning heterogeneous systems. Software available from tensorflow.org
Adolphs, L., Dancshmand, H., Lucchi, A., and Hofmann, T. (2019). Local saddle point optimization: A
curvature exploitation approach. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), pages 486-495.
Agarwal, N., Allen-Zhu, 2.,, Bullins, B., Hazan, E.,, and Ma, T. (2017). Finding approximate local minima faster than gradient descent. In Proc. Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 1195 1199.
Carmon, Y. and Duchi, J. (2019). Gradient descent finds the cubic-regularized nonconvex newton step. SIAM Journal on Optimization, 29(3):2146 2146-22888.
Cartis, C., Gould, N. I. M., and Toint, P. (2011a). Adaptive cubic regularization methods for unconstrained optimization. part ii: worst-case function- and derivative-evaalation complexity. Mathematical Programming, 2300
Cartis, C., Gould, N. M., and Toint, L. (2011b). Adaptive cubic regularization methods for unconstrained optimization. part i: motivation, convergence and numerical results. Mothematical Programming.
Chen, Z., Zhou, Y., Xu, T., and Liang, Y. (2021). Proximal gradient descent-ascent: Variable convergence under kt geometry. In Proc. International Conference Learning Represenlations (ICLR).
Conn, A. R., Gould, N. I., and Toint, P. L. (2000). Trust region melhods. SIAM.
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Proc. Advances in Neural Information Processing Systems (NewrIPS), page 2933 2941.
Du, S. S. and Hu, W. (2019). Linear convergence of the primal-dual gradient method for convex-concave saddle point problems without strong convexity. In Proc. International Conference Artificial Intelligence and Statistics (AISTATS), pages 196-205.
Evtushenko, Y. G. (1974). Some local properties of minimax problems. USSR Computational Mothematics and Mathematical Physics, 14(3):129-1 138.
Ferreira, M. A. M., Andrade, M., Matos, M. C. P., Filipe, J. A., and Coelho, M. P. (2012). Minimax theorem and nash equilibrium. International Journal of Latest Trends in Finance & Economic Sciences.
Ghadimi, S., Liu, H., and Zhang, T. (2017). Second-order methods with cubic regularization under inexact. information. ArXiv: 1710.05782.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Proc. Advances in Neurul Information Processing Systems (NeurIPS), pages 2672-2680
Griewank, A. (1981). The modification of newton's method for unconstrained optimization by bounding cubic terms. Technical Report
Hallak, N. and Teboulle, M. (2020). Finding second-order stationary points constrained minimization: A feasible direction approach. Journal of Optimization Theory and Applications, 186(2):480-003.
Harvey, N. J., Liaw, C., and Randhawa, S. (2019). Simple and optimal high-probability bounds for strongly- convex stochastic gradient descent. M.X9.099090008
Ho, and Ermon, S. (2016). Generative adversarial imitation learning. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 4565 -4733.
19

---

Published in Transactions on Machine Learning Research (03/2023)
Huang, F., Gao, S., Pei, J., and Huang, H. (2022a). Accelerated zeroth-order and first-order momentum methods from mini to minimax optimization. J. Mach. Learn. Res., 23:36-1.
Huang, K., Zhang, J., and Zhang, S. (2022b). Cubic regularized newton method for the saddle point models: A global and local convergence analysis. Journal of Scientific Computing, 91(2):1-31
Jiang, B., Lin, T., and Zhang, S. (2017). A unified scheme to accelerate adaptive cubic regularization and gradient methods for convex optimization. 1xX0::1110.00
Jin, C., Ge, R., Netrapalli, P., Kakade, S. M., and Jordan, M. I. (2017). How to escape saddle points efficiently.
In Proc. International Conference on Machine Learning (ICML), volume 70, pages 1724 1732.
Jin, C., Netrapalli, P., and Jordan, M. I. (2020). What is local optimality nonconvex-noncoonee minimax optimization? In Proc. International Conference on Machine Learning (ICML).
Kohler, J. M. and Lucchi, A. (2017). Sub-sampled cubic regularization for non-convex optimization. In Proc.
International Conference on Machine Learning (ICML), volume 70, pages
Lin, T., Jin, C., and Jordan, M. (2020). On gradient descent ascent. for nonconvex-coneave minimax problems. In Proc. International Conference on Machine Learning (ICML).
Lu, S., Tsaknakis, I., Hong, M., and Chen, Y. (2020). Hybrid block successive approximation for one-sided non-convex min-max problems: algorithms and applications. IEEE Transactions on Signal Processing.
Luo, L.,, Li, Y., and Chen, C. (2022). Finding second-order stationary points in noncomves-strongyy-ooommm minimax optimization. In Proc. Advances in Neural Information Processing Systems (Neurips).
Mokhtari, A., Ozdaglar, A., and Pattathil, S. (2020). A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1497-1507.
Nagarajan, V. and Kolter, J. Z. (2017). Gradient descent gan optimization is locally stable. In Proc. Advances in Newral Information Processing Systems (Neurips), page 5591-5600.
Nedié, A. and Ozdaglar, A. (2009). Subgradient methods for saddle-point problems. Journal of optimization theory and opplications, 142(5):205-225 -288.
Nesterov, Y. (2008). Accelerating the cubic regularization of newton's method on convex problems. Mathe- matical Programméng, 112(1):159-18)
Nesterov, Y. and Polyak, (2006). Cubic regularization of newton's method and its global performance. Mathemalical Programmiing.
Neumaun, v. (1928). Zur theorie der gesellschaftsspiele. Matheraalichee annalen, 100(1):295 320.
Nouiehed, M., Sanjabi, M., Huang, T., Lee, J. D., and Razaviyayn, M. (2019). Solving class of non-convex min-max games using iterative first order methods. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 14934-14942.
Paszke, A. (2019). Pytorch: An imperative style, high-performance deep learning library. In Proc. Advances in Neural Information Processing Systems (Newríps), pages
Qiu, S., Yang, Z., Wei, X., Ye, J., and Wang, Z. (2020). tingle-timescale stochastic nonconvex-concave optimization for smooth nonlinear td learning. ArXin:2008. 10103.
Robinson, J. (1951). An ilerative method of solving game. Annals of mathemalics, 54(2):296-301
Sinha, A., Namkoong, H., and Duchi, J. (2018). Certifiable distributional robustness with principled adversarial training. In Proc. International Conference on Learning Representations (ICLR).
Sinha, A., Namkoong, H., and Duchi, J. C. (2017). Certifying some distributional robustness with principled adversarial training. In Proc. International Conference Learning Representations (ICLR).
20

---

Published in Transactions on Machine Learning Research (03/2023)
Song, J., Ren, H., Sadigh, D., and Ermon, S. (2018). Multi-agent generative adversarial imitation learning. In Proc. Advances in Newral Information Processing Systems (NewrIPS), pages 7461-7472.
Tripuraneni, N., Stern, M., Jin, C., Regier, J., and Jordan, M. I. (2018). Stochastic cubic regularization for fast. nonconvex optimization. In Proc. Advances in Neural Information Processing Systems (Neurips), NIPS'18, page 2904-2913.
Wang, Y., Zhang, G., and Ba, J. (2020). On solving minimax optimization locally: A follow-the-ridge approach. In Proc. International Conference on Learning Representations (ICLR).
Wang, Z., Zhou, Y., Liang, Y., and Lan, G. (2018a). A note on inexact condition for cubic regularized newlon's ' method. ArXiv:1806 188.00888.7784
Wang, Z., Zhou, Y., Liang, Y., and Lan, G. (2018b). Sample Complexity of Stochastic Varianco-Reduced Cubic Regularization for Nonconvex Optimization. ArXi::1802.0 X80:11802077772211
Wang, Z. Zhou, Y., Liang, Y., and Lan, G. (2019). Stochastic variance-reduced cubic regularization for nonconvex optimization. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS), pages 2731-2740.
Xu, P., Roosta, F., and Mahoney, M. W. (2020a). Newton-type methods for non-convex optimization under inexact hessian information. Mathematical Programming, 184(1):35 70.
Xu, T., Wang, Z., Liang, Y., and Poor, H. V. (2020b). Enhanced first and zeroth order variance reduced algorithms for min-max optimization. ArXiw:2006.0930
Xu, T., Wang, Z., Liang, Y., and Poor, H. V. (2020c). Gradient free minimax optimization: Variance reduction and faster convergence. ArXim:2006.099:
Xu, Z., Zhang, H., Xu, Y., and Lan, G. (2023). unified single-loop alternating gradient projection algorithm for nonconvex concave and convex nonconcave minimax problems. Mathematical Programming, pages 1-72.
Yang, J., Kiyavash, N., and He, N. (2020). Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. In Proc. Advances in Neural Information Processing Systems (NeurIPS).
Yue, M.-C., Zhou, Z., and Man-Cho So, A. (2019). On the quadratic convergence of the cubic regularization method under a local error bound condition. STAM Journal on Optimization, 29(1):904-99)) 9322.
Zhang, G. and Wang, Y. (2021). On the suboptimality of negative momentum for minimax optimization. In Proc. International Conference Artificial Intelligence and Statistics (AISTATS), pages 2098-2106
Zhang, G., Wu, K. Poupart, P., and Yu, Y. (2021). Newton-type methods for minimax optimization. arX4v:200..14592.
Zhou, Y. and Liang, Y. (2018). Critical points of linear neural networks: Analytical forms and landscape properties. In Proc. International Conference on Learning Representations (ICLR).
Zhou, Y., Wang, Z., and Liang, Y. (2018). Convergence of cubic regularization for nonconvex optimization under kl property. In Proc. Advonces in Neural Information Processing Systems (NeurIPS), pages 3760-3763
21

---

Published in Transactions on Machine Learning Research (03/2023)
A Supporting Lemmas
We first prove the following auxiliary lemma that bounds the spectral norm of the Hessian matrices.
Lemma A.1. Let Assumption hold. Then, for any a E Rm and y E RR, the Hessian malrices of f(x y)
and (z y) = [vnf- V12I(V22 221 (x. satisfy the following bounds.
[V22[(,,))] S V22f(r,)) = [21f v)ll s Lr,
(26)
(27)
V2,((z _ L1,
(28)
|G( s (1)+k...
(29)
The same bounds also hold for Vuf, Viaf. 2if Anf and & defined in Section 0 under Assumption 2
Proof. We first prove eq. 266 Consider any R and y E R". By Assumption | we know that f(z,) is p--tronny concave, which implies that -222 v) ž pl. Thus, further conclude that
[[ -|| Amax (1-
(Amin( (12((x,)))) S pp
Next, we prove eq.
27
Consider any x, u E m and y E Ik", we have
V21 yuul
V. fxr + tu, v)
01
lim
V2f(( + tu,y) 22f 8)
100
lim
V22((x tu, (2f(z,
1-0
[t]
L1 [
lim
||tt|||
L1 |||l
(30)
which implics that V2f y) < L, Since is twice differentiable and has continuous socond-order
derivative, have V12 (I, V12((1,))
(21 y), and hence eq. (2) follows. The proof of eq. (28) is similar.
Finally, eq. 29 can be proved as follows using eqs.
26 & 27
[|G(r, vl1l < Vuf v)1l + V12f v) V22 (I. v) V f(r v)ll S L1 + = L1(1
The proof is similar for the stochastic minimax optimization problem (Q) in Section G under Assumption 2
The following lemma restates Lemma. 3 of (Wang et al. 2018a), which states the necessary conditions of exact CR solution.
Lemma A.2. The solution 8841 of the cubic regularization problem in Algorithm satisfies the following conditions,
Vif(ze +
+
|| 0,
(31)
2n
1
G( (et1) +
||8 | 1== O, 2n
(32)
ViS(ee 8141 + 2
8844
T
Uc+1 ((t,, S 4n
||
(33)
22

---

Published in Transactions on Machine Learning Research (03/2023)
Lemma A.3. Suppose the gradient Vif Mo+1) and Hession G Mo+1) involved in the cubic-regulorization step in Algorithm 7 salisfy the following bounds for all t < T -1 with T = minnt 2 0: || V ||x:|| - c'):
[[v( - VII(Ir, 1)l - 3 || 1| ) [ - ((( - a + +
(34)
(35)
Then, choosing 77= S (9L + 18a + 288)-1, the sequence {oe)e generated by Algorithm _ satisfies that for all S T -2 2,
H4t1 H₂ S (I. + + 1) 11 + ||sx|11),
(36)
where H, = (xxt) + (I. + 2a + 3/ 11 The same conclusion holds for Algorithm 5 by replacing Vif(ze, Mt+1), G( Vt+1) with their stochastic estimators ( M(41), ((11) respectively.
This is a key lemmn which provides per-iteration decrease of the potential function given that we can access sufficiently accurate gradient VIf( 3(+1) 2 (x;)) and Hessian matrix G(r,, M(+1) s (2)( In this case, the CR objective formed by Vif( 1t+ and G(I₁, Mt+1) is sufficiently close to the target CR objoctive formed by VZ(z)) and (z)( so we can basically follow the standard CR convergence analysis with these additional approximation errors.
Proof. By the Lipschitz continuity of the Hessian of 4, we obtain that
((1+1)) - (xx))
L$ | - xze 1|
< ( - xz) + 2 (01111 - 2n) (2 - xx) +
2
6
+ T
= Vif((z, T 8441 + 2 8444
||setal1
6
+ (V)(x;)- -
8111 + 25 5 (V²((())) - G(Izr ((12))
(0)
L
1
||st1||1 11 + +| + +
+ a 2
|| (1|sell +
6
4nz
(8)
L
1
||8441
1| + (2 + B [2) |141 || + || + )
a
6
4n
((a)
L$
1
|8141
||
+
aa
+ 3 (3 | 11 + +)
6
4m/
2
L
3a
38
|| || (a + 2/
4n
6
2
(60)
x (2L + 3a
+ 4,8
s641/| 3| + (a + +1,)
where (i) uses eqs. 33
34 & 35
(ii) uses the inequality that at? S a v 63 S ,b²+b2 2 0,
(iii) uses e < ||s,|| V | < s
3 + |1 12 v4 < LL T' -2 2 based on the termination criterion of
T', and (iv) uses 7s < (9L$ + 18a
+ 28,3) 1 Eq. 336 follows from the above inequality by defining
H. = (Ix)) + (Lo + 2a + 33) 2 12
Note that the cubie-regularization step in Algorithm simply replaces V Vi 11), G(a,, Mo+1) in Algorithm
with their stochastic estimators f f( pr. 1)-1), ((z,, respectively. Hence, eq. 36 holds for Algorithm
after such replacement in eqs. 34 & 35
Lemma A.4. Suppose all the conditions of Lemma A.3 hold. IJTZ D(2a) [(za)-0' 60 (Le 33++
(Ls+a++) +B) in Algoritlem
then T' min(t 2 1 : [|s-1|l v || S e'} < ( o +3o+++344)4) < T. Consequently, the output of (L)+at
Algorithm has the following convergence rate
+ L.g + 2a + 23)e2
(37)
27
23

---

Published in Transactions on Machine Learning Research (03/2023)
((T(T))) 1 -1- Laz + 2a) 2ea)e'1.
(38)
27
The same conclusion holds for Algorithm 5 by replacing (iI Meti), G(F1,Be+)) in the conditions 34
35 with their stochastic estimators (1,, Gla,, vo+1) respectively.
This Lemma provides two nice properties of the last-iterate T", i.e. T has a finite upper bound T, and ØTv is close to the second-order stationary condition ( = O, <² > given by Fact [ These properties can be proved respectively by two facts implied by the termination rule || V [|s S e'. First, before termination (i.c., < T 1), have ||s V /|sll
d', which results in sufficient potential function decrease He+1 - H1, < (L4 + + I) given by eq. 36 The number T" of such sufficient decreases has to be finitely bounded since H, 2 min, D(x) > -00. Second, at the last iteration t = T", Vif(rz, Mr++) is very close to 0 and G( Mt-1) ) is very close to positive semi-definite based on eqs. [31 & 32 Also, the gradient Vif M++1) = ( and Hessian G(I,, Mt+1) = V2((nz) have c'-lovel small approximation error based on eqs. (34 & 35 respectively. Hence, Iy is approximately a second-order stationary point of $ i.e, eqs.
37
hold.
Proof. Suppose T < T does not hold, i.e. | V ||sell > e' SIST, which implies that eq. for all 0 st t
36 holds
based on Lemma A.3
On one hand, telescoping eq.
36 over t = 0, 1..... T - 1 yield that
T-1
Ho HT 2 (Ing + a + ()
(1|s++1 11 + [|sell
T-1
(Lo + a + 3)
Σ
([]sell v ||8++1))*
T(L + G)
(39)
On the other hand, recalling the definition of H4 in Lemma A.3 we have
Ho HT (2x0) 4 (xT) (L4 2a (1 - ||
2 (xo) - क- + (L+ + 3a + +38)
(40)
where (i) uses || = and $ (xT) 2 4- min (zx). Note that eqs.
39
& 40
contradict.
Therefore, we must have 1 T < T for any T > 4(2o) - :- (Le 4.))
which implies that T" <
(Le to+ B)
(x0) 4" +Lo+ T. taetat
Finally, conclude that
1
(2(TT))
|| ||
-
21/x
(
V V2 2 ST + / V
211/0 1|
(60)
1)
Lo
||ST 12 + 3(1( 12 (2) + a([[sT--11 e (Ts
2/p/
(116)
1
L+ + 2a + 2/3 23)
(41)
211
where (i) uses eq.
331.
(ii) uses eqs. 34 & (35g and the item 4 of Proposition 3.1 that 2 is La Lipschitz,
and (iii) uses -1 V |T E Also,
(8)
( 2 G(TTT-11,99 yr)) [[((T--1,))) 2-- (xTr))
24

---

Published in Transactions on Machine Learning Research (03/2023)
(10)
([((TT-1,, - >
=
|| ||
2mu
(100)
=
+ 1,0) ||T |1 -1l E
27
(50)
1
+ La + 22 )a)el,
(42)
21/
where (i) uses Weyl's incquality, (ii) uses cq.
32 (iii) uses eq.
(35 and the item 4 of Proposition 3.1 that
4h is Læ-Lipschitz, and (iv) uses || ST S e'
For the stochastic minimax optimization problem (Q) in Section 6 on the error of the stochastic estimators.
prove the following supporting lemma
Lemma A.5. Fir any 0 < & 2Lo. 0 << E2 4L1 and choose the following batch sizes
Lg
B1
2 0
In
(18)
c
8
L2
+
n2
BBuh B12), B21 B22 2
In
(19)
Then, the stochastic approzimators satisfy the following error bounds with probability at least 1 6.
[Vil(E,)) vif(z,)) S1,
(20)
[V - - Veel w)ll s2, Vk, E {1,2),
(21)
([((z,)) G( w)ll (k 1)
(22)
Proof. Based on Lemmas 6 & 8 of (Kohler and Lucchi 2017 we obtain that with probability least 1 - 6, the following bounds hold. (We replaced 5 in (Kohler and Lucchi 2017) with 6/5 by applying union bound to the following 5 events.)
In ln(10m/5) + 1/4
(V. - vif(z, v)ll s 4v2LoV
S E1
(43)
BB
(10(m V n))8)
(..fl()) - vz, S 4LLV
S ez k2 l E {1,2}.
(44)
[k.el
Note that here we only consider the cases that B11, BR,el < N for all k, E (1,2). Otherwise, |B = yields [,,f(x,))) - (1f(x,8) 0 and |Bx N yields [ Vi,f(z, v) = = eg. Hence, in both cases, the above high probability bounds hold, which further implies eq. (22) following the argument. below.
(Slo,v)-G(x,p) G(a,))l
$ //VVnl(z,v) - vn http:///ww.ni/t//i//////a/aaaaa/
Se2 + (
H1tp -
1
2 62 + tz - 11 + V (z. Vaa
'1 Li te2
(1)
2 62 2n6 + 1 22 & (x + 1)262-
(45)
where (i) and (ii) use Lemma
A.1
Regarding the high-probability convergence rate of the inner stochastic gradient ascent (SGA) in Algorithm the following result is direct application of Theorem 3.1 in (Harvey et al. 2019).
25

---

Published in Transactions on Machine Learning Research (03/2023)
B Proof of Proposition 3.1
Proposition B.1. Let Assumption 1 hold. Then, the following statements hold.
1. The mapping y (x) == arg maXyER" f(x, y) is unique and k-Lipschitz continuous for every fixed
Lin
et al. 2020
2. (x) is L1(1 + smooth and v (zf(z,y((z) Lin et al. 2020
3. Define mapping G(a, v) =
Then, G is Lipschitz continuous mapping
with Lipschitz constant LG = (1 + ;
4. The Hessian of & satisfies (x) G(z,y' (r)), and it is Lipschilz continuous with Lipschitz constant
L = La ( k))
Proof. The iteems 1 & have been proved in (Lin et al 2020
We first prove the item Consider any x, E Ih" and v.v' R For convenience we denote = (xr, v) and
z' = (2',3// Then, by Assumption and using the bounds of Lemma A.1 we have that
|G(x', ./) G(a,,))1
[2> w')ll
+
(2,(
= (
+ ==2"0|V=a//(d,,//""11111v11 V22f(2''
(L2llz² L1 L =ll)
Lz(1+ + 2x)]|2 = -3 (L2ll2 =11)x=
< L2(1 + 1
Next, we prove the item Consider any fixed x E IR"", we know that f(x,)) :) achieves its maximum at y (z), where the gradient vanishes, i.e., V2f(z,)((x)) = O. Thus, we further obtain that
0 = V2/2 v + (z))
which implies that
= (
(46)
With the above equation, take derivative of vΦ(z) V (z)) and obtain that
(ina://aa))-nT//aaaaa))aaa))aaa -G)z,"((z))
(47)
Moreover, we have that
(00) v (x') -(x,x(()))
V
LG [||a' - Il| + [11"(x))~
<La + +*) k)| =|1,
(48)
where the last step uses the item [ of Proposition 3.1 This proves the item
26

---

Published in Transactions on Machine Learning Research (03/2023)
C Proof of Proposition 4.1
Proposition C.1 (Potential function). Let Assumption hold. For any o, 0, choose c' I ali
aho
1/ S (9T. + 18a + 28/3)-1 and fly = Lit= Define the potential function H, == ((x)) + (I + 2a + 3/3)| Then, when N₂ 2 (n Lail I the output of Lubic-LocalMimmmaz satisfies the following potential function decrease property for all E N.
Loae
H41 H. (LLb + () + Ilsell
2
Note that Lemma A.3 already proves eq. under the condition that the approximate gradient (1)(1)) are sufficiently accurate, as given by egs. V (2r,3t++)) (((z)) and Hessian G( Me++ as
34 & (35) respectively. Hence, it remains to prove eqs. (34 & 35 by showing that the gradient ascent
steps on the p-strongly convex function f(zr.y) yields sufficiently small error || - y" (x+)] as shown in eq.
50) below.
Proof. The required number of inner gradient ascent steps is shown below
No 2 Iln (Li 11100 - v/' (To) (1o))l
2L1a||s1-111 + L1(a + [
N₂ 2 kln
La,3602
[0(x L1al|s1111 || + L2(( L :ISIST.
(49)
LoBe2
To prove eq.
2 we first prove by induction that for any t t2 0.
11 - (x+)) S = |[1 + c2)
= -1
(50)
LG
L1
Note that yt+1 obtained by applying N₂ gradient ascent steps starting from yt- Hence, by the convergence rate of gradient ascent algorithm under strong concavity, we conclude that with learning rate By L+p
11u0+1 vxx))) S(1 - k 2) 11 - o (xx))l
(51)
When 0, eq.
51 implies that
llv1 y" (xv)ll I 11yo 3 (zo))) exp (No In(1 - x)
(4)
L 111 y (xoo))
I1vo - (2o)) exp
In
23
2
(i)
all|soll + C)
B(1s 1 + (2)
(52)
L1
LG
L1
where (i) uses eq. (9) and In(1 -x) - 0 for = =x (0,1) and (ii) uses || = S ala
Hence, eq.
BLo
50 holds when
If eq.
50 holds for = k 11-16 E 10,T" - 2],, then
| 1/" (2x))l
< x 1 MNellykk 3² (Ix-1)ll + (1 No (2==(x==)== (I*))l
2 exp (No In(1 - x)) ( (allss-1ll+ +E)) ^ B(11sk-11 +(() + x8l nol(s|l)
La
(0)
S Sexp -Inn (2L1all (0- 1ll + Li(a + Low)looll) 1)
(alllox- (0(1lo--ll + 6') + xi x||ssk]
LGBBe2
LGG
(42) x
LogE
a(2 sk 1ll [|sxl)) + LGK||sK||
2 |---1| BLaaa|ss-1
Li(a Lox)]|sa
LG
27

---

Published in Transactions on Machine Learning Research (03/2023)
Be (t)) ad'
3e
a[[|sall +c)
B[1s2]1 c²)
^
S
L1 LG
L1
LG
L1
where (i) uses eq. 50 for t k and the fact that y is k-Lipschitz mapping (see
(Lin et al. 2020
Chen et al.
2021) for the proof), (ii) uses In(1 - k kー S -x and eq-
149) (iii) uses Sk-1 V | S |
+ 8k
for k < T" 1, and (iv) uses the condition that e' S ala This proves eq. (50) holds for k and thus for
8Lo
t e 00, T" 1]. which further implies eqs. 34 & 33
Hence, by Lemma we prove that eq. 2 holds
for all 0 stts T"
D Proof of Theorem
Theorem (Convergence and complexity of Cubic- Subie-LocalMimmmax)) Let the conditions of Proposition 4.1
hold with a = 3 = La. For any 0 <
Liv S3La
choose = /33L4 and T 2 ((zo) +8Laee"
Then, the 3Lae
La
output of Cubic-LocalMinmmaz satisfies
p(rr) Sc
(5)
Consequently, the total ramber of required cubic iterations satisfies T S O(VL2k and the total number of required gradient ascent iterations satisfies EiDo N. OVL_n2255
Eq. 5 directly implied by the second-order stationary conditions & Hence, it remains to compute the itcration numbers T" and Σt" N₂ by substituting the hyperparameters.
Proof. Substituting a 3 Lg = Lg + k)3 and é' v33Le into Lemma
A.4 yields that when T >
(m))) 4" +8c and fr= = (55Ta) -1, we have T" S v33T. v and moreover,
33(0(z))
4"
O ((TTi2h 6 S
32
3e
V ( <
+ Lo + 2xx + 23 2²2 S
21/2
Amnn (V2 2 -
+ Lo 2a 2a)2z
-33LL
27/
This proves eq.
Note that the number of gradient ascent iterations N, should satisfy eq. L2(1 x)3 and e' VaLL. into eq. 49 yields that
49
Substituting B L.
No xln (L1ll - y "² (100))) wln (331, - y" ('(oo))l(((e²)
(Liaa|85111 + Li(a 1(ell
N xln
666T1(1 + +x) 18 327, k)]lsell : SSEST
kn
2
where (i) uses LG = L2 + x)² and La = L2 + x) in Proposition 3.1
When we select N₂ such that the above holds with equality, then the total mumber of gradient ascent iterations is upper bounded as follows
In (1]]|1 +33, (2++))(ss))) (3)l)
N x (33L) 1110 (xxo)ll/(2e²) +
22
1=
7"
<x (33L, 11:0 y (zxo))1
+
In (1|s--Ill + [|sæl]) + KI In (66Lie -2 + x))
28

---

Published in Transactions on Machine Learning Research (03/2023)
RT
S Klln (33L) 1110 (=(oo))l/
In (41|=-1ll* + +11
3 7
KT (66I,c (1++))
(0)
x N In (33L - (zo))l/ + KT
-'-1
In
||ا 3 + 11)ll())
3
T
1=
+ KT in (66L1E ( + +x)
(00)
S wIn (33L, l1v0 '/" (2o) (zo))l/
KT" In
31.07 T ("
(4(Ho 1!)
TT In (66T4 (1 + x))
((TT) - VIn
where (i) uses (a b)3 < 4(a² + 13) for any a,b 2 0, (ii) applies Jensen's inequality to the concave function In(-), and (iii) telescopes eq. 2 over = 0,1, , T
E Proof of Theorem 3
Theorem 3. Let Assumption hold. For all t, k,k, assume that 2f ūc)ll S Lo and 2f ūk) (2f( ui-)ll I almost surely. The inner stochastic gradient ascent steps in Algorithm converge at the following rate with probability at least 1 8.
|1++1 (xx))) =0(
Lo In(1/6) + LB N2N,
Proof. Based on Theorem 3.1 of (Harvey et al. 2019 the inner SGA steps in Algorithm has the following convergence rate with probability at least 6.
f(ruy" (xx)) f(a, v1+1) 0 Lu ln(1/)) + L3
(53)
M.N. which by p-strong concavity of f(x,,)) proves the following convergence rate.
Lo (n1//1 + L3 112 Ne
- - ((())) - f(a,, ((ee))))))//0000
F Proof of Theorem 4
Theorem 4 (Convergence and sample complexity). Let Assumption and Theorem
hold. For any 0 < E S
33Le.
choose e' VisT The S 5ol.o T 44'))=" and N, O - +e
Lo 1
Lo
31.00
Morcover, in iteration choose the batch sizes according to eqs.
18 and 19
with the inezactness given
(t) Ls (1sa11
2 2Lo.
L
e2(t)
[|seel +
AL1.
33L
2(k + 1)2
V33L.
Then, the output of Stochastic Cubic-LocalMinnima satisfies
p(Tr) Se
(23)
Consequently, the total number of cubic iterations satisfies T" s O (o the total number of queried gradient samples satisfies Er=O (Ne + (t))) < O 18x3. In
m &
and the total number of queried Hessian
samples salisfies -' -2 Σ [B) < 0 LRx2.
VLzez In mitn
29

---

Published in Transactions on Machine Learning Research (03/2023)
Compared with deterministic optimization, the stochastic gradient error [| Mo+1) Vz)))) results
from not only the gradient ascent induced error ( Vt+1) (=) but also stochastic approximation
error if vc) - f Bt]ll- The stochastic Hessian error |G( M/+1) (2,(),) is similar. Hence,
the main idea is to show that after incorporating the stochastic approximation errors from Lemma 6.1 the
stochastic gradient and Hessian errors still satisfy the conditions 34 & 335 which by Lemma A.4 shows the
desired convergence result.
Proof. In Lemma 6.1 replace z, y with Fe. yr, Bi with (i(t), and Bule with Bx,e(t) for any &&& E (1,2}, and substitute the following hyperparameters.
el L7
e²
[|+a|12+
A 2Lo
OLLo +
331.0
(2 2(x + 1)2
L
[1ss.1 +
^ 4L1
( + v Loc)).
/33T.
Then, we obtain that using the following batchsizes
[B 2 0
L
m
Et) In
se +
(54)
T.7
[Ba,e(t]] 0
m + 1 6
In
(Lo sell
2)
the stochastic approximators satisfy the following error bounds with probability at least 8.
L
| I Mh) V f will S Ei(()) <
|| +
(55)
2
33L
L
|ā( u)) ((xx),80) s c2(+) S
||
(56)
/331.g
Based on Theorem 3 using the following number of stochastic gradient ascent steps
Lo In(1/6) + L3
N₂ 0
22
2² (33[(()b))P+++333333)))1111111 ([]]=]ll + /ellle///////// /3LL
@ 0
Lg ln(1/5) L
(((Lol (1el2 + +))//)) L1 (1l +e/
= 0
Lo + LA
(57)
2 [| + + A s1 +2/LLa)
where (i) uses K L,/H and Lag [1 + +x) (Proposition 3.1 we have
Iluet1 - 3² (xx)) I (V]
To (n(1(1))) + 12
NºN,
< L min
L (1seel12 +
2 ) 1 33L$
(|se|l
V33L0 v
(58)
Lo
Therefore,
[4(2)) Vif(z,, B(+1)l
ViS((e) (xx)) - Tis(2e I Vif(e.
Tillett1 - y (xx))l 2
Lo
|1s
2
6²
33L
(w8)
2
5 Lo (1seel22 33L
(59)
30

---

Published in Transactions on Machine Learning Research (03/2023)
where (i) uses VΦ(x)= v=f(z.y((x)) in Proposition 3.1 (ii) uses eq.
(55 and Assumption |
and (iit) uses
eq.
58
(z,) ((22)3++1) vc+1)ll
[G(z (xx)) - G( Be+1)ll + [|G Mo+1) - G(
(6)
La Lally441 - Vy (2xx))l
Lo
(| +
2
v33LØ
(00)
Lo (1ssell
(60)
V33Lg
where (i) uses (x) G( y (rr) in Proposition 3.1 (ii) uses eq. (56)
and the item of Proposition 3.1 and
(ii) uses eq. 158 Eqs. 159
(60
imply that the conditions (34 & 35 hold with a = B = Lg L (1 x)
and é' /33L.
In Lemma
A.4 by substituting these values of at, B, and T/e = (55T.)) we obtain that.
when T 2 ((x))
- +8Lge"
we have 7 S v33Lg (
0 L 5 s T' and moreover,
31.0
3rd
[
+ La + 2a + +28) se:
21/0
Ami (V2D(IT))) )) 2 -
(1) 1 + La + 20 c' - -33LO,,
27
This proves that eq.
23 holds with probability at least 1 1- 8.
Choosing both N, in eq. 57 and the batchsizes in eq. (54 with equality, the number of gradient computations has the following upper bound.
(N, + (B))
T-11
Lo ln(66/)) + L3
I
In m
+
2 ||: 1 ++) A [1 + 2/LL)
L + 8
Lo Ln(1/)) 43
43 -)
プィー ^ L/
_ STO
Loht
13 In m)
(In(1/8) + Lo)
(10
13k2
O(V 1.5 ))
In
Linihs v VL2
S 0
In
7
m
(61)
where (i) uses d' = V33Le
S
L
(1+x)
which implies that Ax-2 s O(172/144) (1i) uses T <
La
La
O(VIxk -3)) we proved above. The number of Hessian computations has the following upper bound.
T-1 2 2
k=
1=1
T,
In m + n
Li L 1|221
8
E=0
Lits*
m + n
In
Tooe2
0=0
31

---

Published in Transactions on Machine Learning Research (03/2023)
50
L
m + n
S 0
in
L
(9)
1,
m + m
0
In
(62)
VLze
8
where (i) uses Lo = L2(1 + x)² O(L2n²), (ii) uses I v33L
(x(g) 4- 1-82
3x
(vIn
we
proved above.
G Solving Cubic- Cubic-Regulln Resiic-eguurrzzttonnn problem
In this section, we obtain the properties the GDA-Cubic Solver (Algorithm 2 and GDA-Cubic FinalSolver (Algorithm 3) in Lemmas G.I...3. and show in Proposition G.4 that Algorithm using these cubic solvers admits intrinsic potential function H, (see Proposition 4.1) that monotonically decreases the iterations. Finally, using these lemmas and Proposition G.4 A Algorithm 4
we will prove Theorem 2 on the computation complexity of
Lemma G.1. Por any 0 < 8' < 1, when ||9|| S 4l2n2 Mly implement Algorithm 2 with inilializalion
% = O and hyperparameters K 1 Li In
mml mmilinnm Lin"n min"no+ + In In(L 1)) N2, N'
Loc
θ 1u[1----1-1]
Then, the output s'x satisfies the following inequalities with probability at least 1 - 8' where 8" ;= arg min, d(s).
" The = L, 13 22
Lae
108Lin0=
x(s'x) - x0s*) S e'Ils*11?
+ 22
(63)
2407..
Lac
|| s 2|| + 20
+
(64)
|
Lemma G.1 provides convergence properties of GDA-Cubic Solver (Algorithm 2 in the small gradient case ggll < 4L me- The key to prove the convergence rate 63 is to equivalently write the gradient descent step 12) as gradient descent. the CR objective function ON defined by eq. (67) below, which is e'-close to the target CR objective function & with sufficiently large number N' of gradient ascent steps. Therefore, we can leverage the existing convergence result of gradient descent on Φ' from (Carmon and Duchi 2019) and extend the result to σ. The CR function & is also close to convex, so small ((5x) o(s*) means s" not far from sxx which implies eq. 64).
Proof. The gradient ascent step can be rewritien as 0,,6-1 = (I + 7h H22)UK, + mn H By iterating it over = 0, I,..,,,,, N' and using Uk O, we obtain that
UK = The
Σ
(I + no H22) 11 = -11 (1 - - + + 122 H
(65)
t=
By substituting the above equality, the gradient descent step
12 can be rewritten as
||s||| 26 where Ax = Hn H12 (I (I t. H)))))HHHH
1/4 =/ - - (0+ ANish
(66)
21
It can be easily verified that the above update rule 666 can be seen gradient descent steps with random perturbation of on the following cubie-regularization problem
1 T A + N = arg min ON = Ts 3
1 ||1113
=9
(67)
6/
Note that
|Ax H1| + 11 H
|||T
(T no
H22) (111
32

---

Published in Transactions on Machine Learning Research (03/2023)
< | H11l || + | H22 ||| H12l
(6) S L1 + Lipe" L1 S Amax == 2L18
(68)
where (i) uses I 3
H22 3 pl (Assumption 1 and fe =1/L, which imply that 0 3 I
1/ H2 =
(1 - -- and thus /|| (I + yn H22) [ || 1, and (ii) uses || s Li, ||H S L1, ||
S /
(Assumption Similarly, we obtain that
|ll Hnll + |1 H ||| H12 S Amax == 2L1x,
(69)
and that
|| All (1(1 + H22 M1 (1) (1 - x1-x (1- 1140 All S
(1) Lae
(70)
(7L
where (i) uses -hا- 3 H22 1 - pl (Assumption and To = 1/L1 which imply that O 3 + no H2 1 (1 - 1),, and (ii) uses N' In 1m(1982 2 /
Ao = O and |||||| s 2L1
[n((1
Hence, the optimal solutions 8" arg min, ((8) satisfies
|| S ||A||m (IIAAl)m.) + 21
(6) 2||A||m + 2/ l
7L == Smax:
(71)
where the proof of (i) is in eq- (7a) of (Carmon and Duchi 2019 (ii) uses the inequality that Va+bs Va+ vb
for any a, 0, and (iii) uses ||||| s 2L and [6|| S 4L2 no- Similarly, sty- == arg min, &v (s) satisfies
| 1l S Smax-
(72)
Then, based on Lemma 2.6 and Theorem 3.2 of CCrmmon and Duchi 2019). the gradient descent step of the cubie-regularization problem (67 yields that || < | S 8max for all & and that dov-
66
"1 + L with probability at least 1 1-6 8', using the hyperparameter choices below which dur(( (5xr) 240n.
can be easily verified to be satisfied by those used in this Lemma.
% 0
4( (Amax + Smax/ / (2n=))
22L1K
[e' 1/sxx11/((240m) ) Loo /30 /2111
(2 ( /
Loe
(73)
Amax + /1
12
12(2L 9max/ n1x))
108L1 KTT
6ln (1
3/mm Naain
+ 1444
21 ((240m=)+Lor
3 (Amas + +.man/n
10/ 1|2
K >
(74)
(1/2)n=
dl 11 1220040m L
211200L NM 3ln (1+ 3. 3.1m 2
nu 244
+ 7In 742407, Loe
2160L, M71
é' + 240m Lag e3 1 |
where eq. 73)
corresponds to a = Amax +le 11 Amax Smax/ e' el 1 [/0p0 E [mmin 1] (Amin = 5L 22
2 1
10
since < 8 Nv
< 8max km)) defined in Theorem 3.2 of (Carmon and Duchi 2019) which yields amin
and (1/2) eq. 74
Then, eq.
can be proved as follows
((sx)) d(s") (Ø(sk)) dn (ok)) + (gx: (nx(())))) + (syr) (b(s))
33

---

Published in Transactions on Machine Learning Research (03/2023)
(
1 (Ō((k) (sx)) + </115/r11?
+ Lye + (sN) -0(s))
2407z
2 (A - AN-))'K + <'110
1 22 - ANT)S*
+ L
240mz
(10) s (7LIATT)2
+ <'11s 112
+ Laze
240
<'1ss 2
(75)
2401,
where (i) uses Φx ( S ΦN (5 + <'Ile 112 + L and oN (s)r) S (s)) (ii) uses the definitions of (c))
240y
and (-) in eqs. 6 & 67 respectively, (iii) uses eq. 70 and max([I [s/ S 8max == 7L KJ]a-
Eq. 644 can proved as follows.
é' e'Iss || 1|
(1)
+ 2T
(0((kk) -(s")
2407z
(21)
(s'x - -) 8²)T (7(AA (A+ ||5*1| 1) (8'x -s") x*)+ +
12m= (1s511 - (1* +11xx(ll
27
(686) || (1|s (1|5*112 | + ||s'x 112 - 21| 2 I11
12
|| (||**| - 2 kl)),
2
12
where (1) uses eq. 63
proved above, (ii) uses eq. (6) of Carmoon and Duchi 2019 and (iii) uses Proposition
2.1 of Carmon and Duchi 2019) which states that A +
|s O.
2n.
Lemma G.2. When II > 4L2x² implement Algorithm with hyperporameters K 2
2lm(100/(7))
Im[ x 1)
The =1=1/ 1/h and initialize so = Then, the outpul 8'x salisfies ((*x) S To / La with probabilily at least 5. Correspondingly, the approzimate CR solution 8N in Algorithm satisfies
||s/x || L1
(76)
1 11
<(5x) v 4
-19
x3 12
(77)
Lemma G.2 provides the properties of the solution given by GDA-Cubic Solver (Algorithm) in the large gradient case ||9|| > 4L7x3ne. The main idea of proof is to show that s'x T close to T 9,
the properties about s'x can be approximately obtained by studying x g. To show that уk = 7 we only need to show that WK obtained by gradient ascent step of p-strongly concave function converges to w" exponentially fast as shown in eq. (80), and that there exists Lipschitz continuous function such that (Hag)) ww and >-' E f Hu
7KK = (( 11 e I9
# (Ha1)) =²
1911
Ti
191
Proof. 7" has the following lower bound.
Nng' Ag 2 l|al|2
A
7n9" T Ag ||
+ 2nxllal1
ns V
( (T ||
TAg [|a]|2
+ 8L2
LiKnz
(78)
34

---

Published in Transactions on Machine Learning Research (03/2023)
where (i) uses |||| 2 4L3K2m nu. and (ii) uses the monotonically decreasing property of the function v x2 + 8L4
x and ||||| I |nll ||H H2 |ll I (1 + k) s 2 based on Lemma. A.1
> = arg m20 ((~99/119l)) also satisfies the stationary condition below
aØ(~19)
y=2
By "-y - +
2 Ag +
(79)
21/
Note that the gradient steps arg max w w0 H22 gHa
10 aim the Lı-smooth, μ-strongly concave maximization problem w" = all Therefore, these gradient steps with learning rate 1/L1 and initialization Wy 0 have the following convergence rate
(0) ||wx - w'll S (1 - 11) =01
KM/a
(80)
100
where (i) uses we 0, ||w*|| S H22
Hz19 T9ll
< H22 H21 Sp L x and K
2 (m ) [n x
Denote the function E(u)=
/u²
21
|gl|
u(uu R). Then have
hI >-1 -
T
g
(H21g) T wx)
(5 g H1
(2219) T "w" 1191
w²))
H1
||9||
|gll
[19||
1191
9
9
(H21g) Twou (1 w)w']]
(H21g)
T(wK w²)
5'
H11
|g|||
|g9l|
|19|
|1g||
(i)
(iHt)
Lyssu
I
0.17"
50
where (i) applies the Lagrange Mean Value Theorem to the function & with E [0, (ii) uses [<'())] 1 s 2, Halll s L1 and eq. (00) and (iii) uses eq. 178). The above inequality implies that
222+20.llo Iaal
0.9y" S SKK- $ 1.17"
(81)
Therefore, eq-
76 can be proved follows.
(88) 7(0.9)
|
7K
0.9h
KT]a 2 LIKTI
where (i) uses 8'x
7K 9. (ii) uses eq. Ioll
81 and (iii) uses eq. 78
Eq.
77 can be proved as follows.
T
o(sk) = -->kllo 11ll +
Ag +
2||g||2
6m
97*
|| + (1.17*)²
T Ag + (1.17*)3
10
2||g|
67
(1))
7"
(7*)3
|1l1|
20m
(H1)
(1.4L (1.4L1877)²
20
2500mz
Li 1
4
where (i) uses eq.
81 (ii) uses eq.
79 and (iii) uses eq.
78 and || 2
4L7x2 nw.
35

---

Published in Transactions on Machine Learning Research (03/2023)
Lemma G.3. Implement Algorithm with initialization 6 = and hyperparameters K = 9
1.3n²
Liget
max L/o))| Th = i. 7 = 2 22|1K² Then, if ||>*1| S 3e' and |i S 4L2 the algorithm will terminate with K' (k: lIaell S l K and the output s'y. satisfies
N₂ = N' In|2 1n[[1=-11 111
| $ 77'-
(82)
||5 11 skx| S 2L 2n
[| = + Askr. +
(83)
Lemma G.3 provides convergence properties of GDA-Cubic FinalSolver (anelveerr-lggeae (Algorithm 3) in the small gradient case |6 < 4L2k2nz n. Note that Algorithm 3 is noiseless version of Algorithm 2 Hence, the proof logic is similar to Lemma G.T To elaborate, Algorithm 8 is equivalent. to apply gradient descent to the approximate CR objective function Φx defined by eq. (67), which c'-close to the target CR objective function ø. Then based the nice geometry of the two close CR objective functions, their optimiiers s/y- = arg min, Φx
and s* arg min, ((s) also have close norms such that ||s*|| S 3e' implies || Té', as shown by eqs. (87
& 88
Then eq. (82) follows from ||sx*|| I || which has been proved by Lemma 2.6 of (Carmon and
Duchi 2019 for gradient descent steps on CR objective function Φ. K' < K follows from the nonconvex
convergence rate of minoskgk-1 || where gk = Vol Finally, eq. 88 follows from | S T
and Vo(sk) s (s = 9K.
Proof. Since s" argmin, min, (s),, 0= Vp(s") = g As + ||>"|| s*, which implics that 2n
|
S ||||||| |||||| + || (6)
27/ S 2L)x(3ee) + (3e') (3e')
|| = Ass +
s*
I 13L
(84)
2n
2n
where (i) uses ||||| < 2h,1 Kr ||*
s 3e' and ||**|| S 8max = 7h1
Following the proof of eq-
70), we can prove that when N' In/2Line' max (24 Ln.)
In[(
I (1 - 1- 1A0 - All < 2L - -n I min ( T (85)
|AN - All = [1(T + n. Hzg)
24m=" 7
Substituting eqs.
65
&
66
into eq.
13
we obtain that
| 21/
gk 9 s
5/6 (())
(86)
Hence, the update rule 14] can be seen gradient descent step solving the cubic-regularization problem
67 without random perturbation. Therefore, based on Leminas 2.3 and eq. (11) of (Carmon and Duchi we have
2019
|| &
|| S Smax and that when Vs MLLN S A[] (2na3)
7
|| soN((6) (x)(8")
0=
(
<|1a|1|="|1 1
||s*||
61
x 36' 33'(11Iiee' LIx(3E') + (3e (7L
115
$ 2 Like"
where (i) uses o66 and the definition of function ox in eq. 67 ; (ii) uses eq.
884
||s*|| S max(3e', 7[1kz=)
and ||Ni|| || < 2L Rearranging the above inequality, we obtain that
1 K
115L,K0 (0)
min 11 12 < 0SksK-1
11xx =
Kn, $ L2,
K
0=
36

---

Published in Transactions on Machine Learning Research (03/2023)
where (i) uses K
2 e
and 1/s 22L16' Hence, K" min Il S } KK-1
Next, will prove eq- proof of Lemmaa G.1
82
On one hand, using the same proof logic as that of eq-
64 (see the end of the
obtain that
( (s" six') " |s 1) (s* -5x0) + 12n
@N.( - (s
(1| - (I + 1)
27/00
(2) |sNvell |I (11s%x112 1/2 -|s-"111- - 111
12
(124)
(11s - 6é'),
(87)
121a
where (i) uses eq- (6) of Carmon and Duchi 2019 (ii) uses Proposition 2.1 of (Carmon and Duchi 2019 which states that A +
|
I = O and (iii) uses ||*|| s 3e', On the other hand,
2n.
(s) (sx))
(s) o(s")) (Ø(s*) (sx))
(ox)(s") - -((s)) + =(((xx)) (N.()))))) 1 T T(AN: -Ass" 1 -5 (A (A
AN:)sN
2
(11)
(96 + +113,.1133,,
(88)
241
where (i) uses d(s*)= min, ((s) < (( (ii) uses the definitions of o(.) and (.)(-) in eqs.
6 & 67
respectively, and (iii) uses eq. 85 and | < 3e'. Combining eqs. 87 & 188 yields that
96 13
| S Ge's + 2/ (9e22 11 = 6.5c' + 2
2/
Suppose || 7d and substitute it into the right side of the above incquality. Then we obtain the contradiction that || < 6.8e'. Therefore, || || < || S 7e', i.e., eq. 82 is proved. Finally, eq. can be proved as follows
Vo(sk)) V gK' || + |
< II AN LAE
(2) Lap''
(7e') + Lape = 2L
(89)
where (i) uses |1 < L and the definitions of Φ(-) and (.) in cqs. [| S 76'.
6 & 67
respectively, (ii) uses
eq. 85 and
Proposition G.4 (Potential decrease for Inexact Cubie-LLcallinnmaax). Let Assumption hold. For any a, 0, < e' < ale and 8 E (0, 1), choose The (168L$ + 120a + 168,3 Ty Tita and BLG
N₂ 2 θ in Lolls 1ll (1+
(see eg. (99).. When implementing Algorithm at the t-th iteration, use hyperparameters in Lemma G.1 with 8' o/T if VV M(+1)ll S 4L3x2 and use those in Lemma G.2 otherwise. Define the potential function H. == ((x)) (10Ls 7a: + 10 Then, the output of Cubic-LocalMinnee satisfies the following potential decrease property with probability at least - 8.
Lase
He11 H, < - (L4 + + 3)+ ).
(90)
Compared with Proposition 4.1 which also provides potential function decrease, Proposition G.4 only has access to the inexact CR solution obtained from GDA-Cubic Solver (Algorithm) 2 In small and large gradient.
37

---

Published in Transactions on Machine Learning Research (03/2023)
cases respectively, this inexact CR solution has properties given by Lemmas G.1 & G.2 and the Taylor expansion of 4 (x++1) - (are) can be upper bounded by eqs. (3) & 77 as shown in egs. (66 & 998 Both cases yield eq. (90).
Proof. Following the proof of Proposition 4.1 it can be seen that when é'
and N₂ >
3LG
θ k In Lall sell) ) the following bounds always hold, which are analogous to eqs.
(34 &
Loa0"
35 with exact solutions 8t-1 and 8t replaced by 8;-1 and 8 respectively.
[x))))) - VIf(Iz, - (( |2 + ((;) M++ill a1| +E').
(91)
(92)
Then we consider the following two cases.
(Case 1) When [V Bt+1)l 4T2 the output sxx satisfies eqs. 63 & 94 with probability at least
1- 8'. Also note that the imput variables of Algorithm 2 are := Vif yt-1) and A == G(z,, vt+1)
H1 1 H22 H21. the output s'x of Algorithm 2 is assigned to 8441 which is later used for the update
2++1 = Iz + $1+1. and the optimal CR solution s* in Algorithm 2 corresponds to 8t+1 in Algorithm [
Therefore, eqs. 63 & 644 transform to the following inequalities at the t-th iteration of Algorithm which
by applying union bound hold simultaneously for all 0 S with probability at least 1 T8 6.
(rf( 11641-114777 s(+1)+ 1_ 51 Glrree 1
|8+1 -118+11 ||3
2 $ G(e
61/
<11s2t211
+ 2
(93)
2407z
24m La
|sel|| < x
+ 20
+
(94)
||se|1
Based eq. 944, if || é', then || & 2/ +' since The (1681.))-1 Otherwise, ||sell S c'. Combining the two cases yiclds the following incquality.
||+||| s 2|l + + |s,11? 8| + 2e
(95)
Therefore,
((E++1) - (xxz)
S 141 V8(x)) + 1TT (2) 8041 +
La
5
| 13
6
Vif(z. (
- (v0(z))
5 V
fz yo+1)
1
1 TT
4 G(x,, Me+ 3.41 + La
2 5/1
(V2 - G( M/+
3 +
|8++1 1|3
2
(9)
1 T
5
3|
3
11 |1(111111
+ (2) + |8 [[ || + + + s V VIS(
+
2 7
G(res 18841
|8++1|1
3
Li
e' e'I|st+1ll
| ||
+ 2L
6f/
6
6n
240
(00)
α
22(11
13 | ||
+ 23)
|| 113
+
1 (-_11) 1-1) | 13+
La
4e' (5
+ e3
+ 2
2
12
6
6m
120m.
(im)
Lg
1
+ 3)
(31
1 ||
||5
2
6
67
4
4
il |12 (1 || + 13, + | 3
+
||,,11
+
2L
(11 11+ 11%
120m
a
+ [3] 8.4111 + 13
L$
1
|| 1|3
2
6
6m
38

---

Published in Transactions on Machine Learning Research (03/2023)
4 1++1| 113 +
+(111,,11 +
(82+1119)
||
|3,113 + 2L (1152_1113 (1%)l1))
120mz
30
+ +33+ + 13L4
1
||(+1|| + aa + 23 +
2Lx +
1
||5e|1
12
24m
(vd)
(111. 80 + 11,3)| 1 + (9I.g + 6a 9/3]
(96)
where (i) uses Z(+1 = Zt + 8++1 and the fact that (2)( is Lo-Lipschitz continuous (see the item 4
of Proposition 3.1, (ii) uses eqs. 99h. 92 & 93 (iii) uses eqs.
33
& 195 and the inequality that
ab² < vb a +b", Va, b 2 0, (iv) uses S s | 3 1I" = ||=l| 3
40 It kt< T" -2- n E {1,3) a
based on the termination criterion of T" in Algorithm uses T/e = (168L4. 120a + 16
(v) uses ab a b
+ Ya, 2 0, and (vi)
(Case 2) When [[V M++1)ll > 4L2n2n n2 similar to case eq.
77
transforms follows in Algorithm
vif(2u0)+1 s441 + 2 G(rr..44))
1
+
s141 ||$ - -
-
(97)
67
Then, obtain that
4 (x(+1) - (21)
$ 141 VQ(x;) + 1 + (2 + ||
=8 (VO((z) V V.J M1)))))++1))))))) + 3 V (xt; Mt+1)
1 T (xx) - G( 36++ s1+1 +
1 T
La
2 s41
s1+1 G(z1,)641))))) +
|st+1ll 113
2
6
2 B1| 11l 2 + sc+1 (1|ssll + +t')
+ (Le.
L
-
-
1
1
-i
1ix031
6m
(0)
x (11Lg + Sa 11 5 3 (9Lg + +6 + |3,
(98)
where (i) uses eqs. 91
92
97
and (ii) follows the same proof logic as that of eq. 196
Eq. (66 in case and eq.
98
in case 2 are the same. By rearranging them and using He = ((xx)) + (10Lg +
7 + 10,3) & | we can prove eq-
190
Theorem 2 (Computation complexity Inexact Cubie-LocalMinine Let Assumption hold. For any 0 < S. min
53L18_ LLL,
11/221//// 1/2 22/2 Ian") and d (0, choose é' 106VI.0' T ====0vl O+ (2)0) x- + 3))
228./1
(see eq. (49) in Algorithm 4 When implementing Algorithm g at the t-th iteration, use hyperparameters in Lemma G. with 8'
Ta = O ny = LitH and N₂ = = Lice 1a
2
Loe
5/T of
(Ir, 3(41)ll S xL2 ma, and use those in Lemma G.2 otherwise. When èmplementing Algorithm 3 use the lyperporameter choices in Lemma G.3. Then, with probabilily al least 1 6, the oulput of Ineract Cubic-LocalMiningg satisfies
p(Tr) Sc
(15)
Consequently, the total number of required cubic iterations satisfies & O0( <O(TInn))5+ the total number of required gradient ascent iterations satisfies TT' N, x(VI2k23 and the total number of required Hessian-vecton product computations (in Algorithms & is of the order Ō(LinZe
The proof logic is very similar that of Theorem 1 with the major difference that we leverage the propertics of inexact CR solution given by Lemmas G.I G.3 and the potential decrease given by Proposition G.4 instead
of Proposition 4.1
39

---

Published in Transactions on Machine Learning Research (03/2023)
Proof. First, it can be easily verified that the hyperparameter choices of this Theorem fits those in Propo-
sition G.4 with = = T.p- In particular, é' 100//. with 0 < S 106L1 and E (0, 1) satisfies
vLa
0 0<E'
داع required by Proposition G.4 since The (456Lx)
B = La and LG L1 + k) S L
BLc
(see Proposition 3.1
Suppose T T does not hold, i.e, ||-- Ill ||0a|> ><= é'= 106vLe
lA SLST. Then, one hand, telescoping
oq. (90) over 0, 1, ...., T yield that
Ho HT (Lx + B)
(1 11 )
T-1
3L$
s| V ||s4111)
c
106VLI
(
2 (x0) 40 +
(99)
where (1) uses T 397006//Lo(0(z0)) x" c On the other hand, recalling the definition of H₃ in
Proposition G.4. we have
Ho HT = =x(x)) (xT)) + 27 (1 (1:601 - | 122) (1) - (xxo) $ +
e2
(100)
137'
where (i) uses ||so|| e' = 106/L. and 4 > 4' mil (xx). Note that eqs.
199 &
100
contradict.
Therefore, we must have 1 S T <T = 397006 La ( - x- + =.
Since || < é', we have Vif 41)+ - 4L Otherwise, eq. 76 directly implies the ccontradic-
tion that ||T 2 LΙкЛи = 256L*
L1 > = 106/L.
(since e < 2 ). Hence, [[Vif(ers (orr)) & 4L7 ne
which implies eq- (95] and thus implies || < 2
3T> +e' 36'. Therefore, the conditions of Lemma G.3
are met, so eqs.
(82) & 133
hold. Rewriting eqs.
82 & (83) with t- Vif(2T-11, vT. A & G(IT VT
and 3 + s'xx yields that
| S 7d
(101)
|
V VI (TT) + G(IT-1111))
2L
(102)
2115
Therefore,
| FT ||
||3
[T(
(1 - G(F (
+ 22
21,-
17 (V
V (
|
+ +
( G )3|
+ 2L
21/
| 31] +
Tad 7 +0) + 27 + 2
(16)
< 11191 (io) 2
(103)
where (i) uses eq. 102 (ii) uses eqs.
91 92 & 101 _. = -T 3 and the item of Proposition 3.1
that 2 is Lipschic, (iii) uses (168L. + 120g + 188,3)-1-(11
|| 8
S e', a B == T.b
and eq. 101h, and (iv) uses e' 10B/T.0. Also,
2 ( 7 G(xT-11, yr.) |[G(TT-1, -3
40

---

Published in Transactions on Machine Learning Research (03/2023)
(() 1
= 21/2
ST 1- a(133r1111111 E)I L
((i) -237 T -2V-- LocI,
(104)
where uses Weyl's inequality, (ii) uses TT 2T + eqs.
32 & 99 and the item 4 of Proposition 3.1 that T2p is Lø-Lipschitz, (iii) uses eq. 101 1/2 (456L.)
La, |5 d' and (iv) uses
é' = 106/T.6 Combining eqs. (103) & (104) proves eq.
15 where p(r) = v (+)=) V Amin (x))
v331.
Finally, we compute the computation complexities. We have proved that the total number of cubic iterations satisfies 7 < T 397006 (oxx))) - $6 + + (VTT2xx-663) (T.g = (2 + x)3 = ( based on Proposition 3.1b We can also prove that the total number of gradient ascent iterations have the same bound ΣT-1 N₂ S O(
as that of Theorem 1 following the proof logic the end of Appendix Then we compute the total number of Hessian-vector product computations in cubic solvers (Algorithms 2 When implementing Algorithm 2 at the t-th iteration, if V f Bt+1)ll I 4L3x2 may then based the number of Hessian-vest vector product computations is proportional to
D
&
Lemma
G.1
KN (98( n/)//L/e" O(LiKTZe
In[(1 x 1-1)-11
= 0 1n L
VLB
-0 Li 1/2 x2-11
0/ 1/222
1 1/2
where (i) uses Lg. = L2(1 x)3 = ((2nn) If V f ve++ I|| > nL Tee based on Lemma G.2 the number of Hossian-vector product computations proportional to = (x), whose order is not larger than the
1/2 above 1,2, -1) since s 1/ 2/2 Based Lemma G.3 Algorithm implemented once
with the total number of Hessian-vector product computations proportional to
k2 (kk) 0 (
(2((//TT)))) - 5
Li
13 -(12
KN =0
La (e)2
As a result, the total number of Hessian-vector product computations in Algorithm is
= 1,
0LL O
= =((1)h6
where (i) uses S Lyn to absorb the term 0(1
H Experiment Details
In this section, we present the details of both synthetic minimax problem and the neural network simulation.
H.1 Details of Synthetic Minimax Problem
In this section aim to solve the problem below:
v A
5y2
min max x YER N
w
w(xx)=
+
B,
(105)
40
where A, and B, are independent random variables from uniform distribution range from 0.5 to 1.5 and u w(zz) has the exact form below
41

---

Published in Transactions on Machine Learning Research (03/2023)
Ve(o + (L (1+1) - (L (L+ 133 x LLVE
6x + 2/2
- < 8 -ve:
3
Vcx²
-v
w(x)
(106)
VEX² + J
0<ISVE < Wes
2/2
Ex + 3
Velx (I + 1) + f0 (L+ 1)ve) + (3
and we set = 0.01 and L = 5 in our experiment. Through simple computing we can calculate that:
N
((x) = (2a) +
Σ
A.)
22
2
(107)
1=1
0.000
0.025
0.055
0010
0.000
-0.005
-1.00
-0.75
-0.50
-0.25
0.00
0.5
0.75 100
Figure 9: Figure of the w-shapod function w(x).
H.2 Details of Neural Network Simulation
The network model use is a convolutional neural network that consists of two convolution blocks followed by two fully connected layers. Specifically, each convolution block contains convolution layer, max-pooling layer with stride step 2, and a ReLUU activation layer. The convolution layers in the two blocks have 1, 10 input channels and 10, 20 output channels, respectively, and both of them have kernel size 5, stride step and no padding. The two fully connected layers have input dimensions 320, 50 and output dimensions 50, 10, respectively.
H.3 Details of GDN/TGDA/FR/CE
We apply a slight variant of GDN [Zhang et al. 2021 to the equivalent minimax optimization problem max, min, - f(r, 3) with 20 gradient ascent steps Ut+ = y + +..01 ve) followed by Newton descent step 2t+1 = 2;
(VIif) -1 (VIf) (++1))+1) for the update of has analytical solution for synthetic simulation. For adversarial deep learning, optimization problem max, (s) ss Vinf
Mo+1
obtain by solving the equivalent Vif Bt+1) s via 30 gradient ascent steps with learning rale 0.002 and early lermination rule (s)]l < 0.001. Batchsizes 512 and 100 are used to compute all the stochastic gradients and Hessians for synthetic simulation and adversarial deep learning respectively.
For TGDA, we apply 10, 20 gradient ascent stops on y with learning rates 0.01,0.1 for synthetic simulation and adversarial deep learning respectively, followed by Newton-type update step 2++1 =
Compared with the original GDN (hang al. which gradient ascent is sufficiently fast.
switch the roles between and y, since fz, -) is strongly concave
42

---

Published in Transactions on Machine Learning Research (03/2023)
(Vzzf)"' Me+1) has analytical solution for synthetic simulation. For adversarial deep learning, we obtain by solving the equivalent optimization problem max, =2(s)) := 1 V22f(=2,, Dt-1)s (2 M1+1)s s via 30 gradient ascent steps with learning rate 0.1. Batchsizes 512 and 100 are used to compute all the stochastic gradients and Hessians for synthetic simulation and adversarial deep learning respectively.
2, -.0 V ((t
For FR, we apply 20 gradient ascent steps on y with learning rates 0.01, 0.1 for synthetic simulation and adversarial deep learning respectively, followed by Newton-type update step 2 2, 0.01 i Bc++)) - 12 (V12)) M++1) on
a (VV
M(+1) has analytical solution for syuthetic simulation. For adversarial deep learning, we obtain by solving the equivalent optimization problem max, (s) :- f V ())) (V2)f)V 11 via 30 gradient ascent steps with early termi- nation rule [s)l < 0.001 and learning rate 0.002. Batchsizes 512 and 100 are used to compute all the stochastic gradients and Hessians for synthetic simulation and adversarial deep learning respectively.
CN algorithm follows the original update rule in (Zhang et al. 2021 that Tt+1 2x -e (t) (1),, (rr, yc), M++1 8:
V y). For synthetic exper- iment, = -) Vi Bt) has analytical solution and we compute = (G( W1) - 0.57) -1 V [ Wt) instead to avoid singularity. For adversarial deep learning, to compute = G( Vr - ( Vt), we apply GDA with 50 iterations to the minimax optimization problem min, max, (f M.) + 1s Vuf(((((t.)))) + s V12f(2141.3e)) ) + 1u Voe Ye, where each iteration has gradient ascent step on v with learning rate 0.1 followed by 1 gradient descent step on with learning rate 0.002. Batchsizes 512 and 100 are used to compute all the stochastic gradients and Hessians for synthetic simulation and adversarial deep learning respectively.
where G(z,, y) [V= Vinf-
V22
43


Reviews:
[{"review_id": "fzZfhp269V", "reviewer": "Reviewer_jvV1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Even though I like the basic premise of the paper, I cannot recommend its acceptance at this point.\n\nMy main reasons for this are:\n\n1) Incremental extension: The current work is a slight extension of an accepted NeurIPS submission \n(see arxiv https://arxiv.org/pdf/2110.04814.pdf ) this year. Since TMLR is a new approach for journal submission, I will let to the (higher-order) Editors' committee to decide about this double submission issue. I just comment this fact because NeurIPS provides its own Proceedings.\n\n2)  Lack of clarity: The main ideas of the paper are expected given the previous literature  but the presentation is fairly confusing. The solution concept of mini-max optimization in non-convex non-concave structure is certainly well-defined. However, it is important to notice the lack of duality gap. It would be important to explain why this absence is not crucial for the important applications of this concept in ML. More importantly, from the way that the results are written using the parameters $T$ and $T'$, it is impossible to understand if the authors refer to a best-iterate result (which is the expected one given the non-convexity) or last-iterate convergence to a point which satisfies a specific stopping criterion (which is computational intractable in general case). Since the readership of TMLR cannot be assumed to be necessarily well-versed in non-convex optimization, the authors should do an effort to clearly present the basic notions needed from convergence guarantees in continuous optimization, so that readers can follow clearly the intuition behind their results.\n\n3) \n\na) Having said (2), it would be more interesting given the incrementally of the current submission and the accepted NeurIPS result to provide multiple additional experimental work which would verify the practicality of the result. Especially given the best iterate approach of the result, why such kind of results are important and interesting in ML community? \n\nb) An additional issue that I would like to raise for the experiments is the suitability of number of epochs as the correct metric. It would be important to provide system-wise information about the execution of the experiment. \n\nStrengths:\nClear scope of focus: besides of the paper’s length (due in part to many ancillary\nresults), it is very easy to filter which results comprise the authors’ main contributions and which ones are secondary. I strongly appreciate the well-positioned papers and the easy flow writing style of this paper.\nHowever,  I had to read both parts (main and appendix) two times until I could safely pin-\npoint the authors’ main results proof ideas – a somewhat disappointing trait for a\npaper which purports to provide an intuitive optimization idea.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this two-part paper, the authors provide the analysis of a Cubic regularization approach for finding local minimax points in the general bifurcation min-max optimization problem. More precisely the starting point of the authors is the observation that standard gradient descent-ascent approach typically converges to stationary points for nonconvex minimax optimization, which are suboptimal compared with local minimax points. In order to tackle this gap, the authors turn their attention inspired by the convex-concave analogue on Cubic Regularization techniques. Their first contribution is to provide an equivalent characterization of the local minimax points as the second-order stationary points of a certain envelope function. This second-order envelope function actually is based on the standard approach of Shur's Complement of the objective bifurcation. Having established this equivalence, initially they provide a deterministic and stochastic best-iterate convergence for value/gradient/hessian oracle and finite-sum case. Following the analogue of non-convex single agent optimization, the authors describe a non-exact hessian vector approximation scheme to avoid the complete computation of Hessian matrix.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Specific suggestions for revision\nBelow, I am providing some specific suggestions that the authors might want to\ntake into account in revising the paper. In order to increase the readability of the appendices,I would strongly suggest that the authors will provide a ''foreshadowing'' paragraph before every proof. In this short period they could explain the intuition behind the proofs in order to increase the readability of them.  \nMy expectation for a journal publication is  the manuscript which is self-contained, with correct grammar and syntax, and it offers the opportunity not only to proof-read but obtain important intuition behind the techniques. If these basics are not explained to readers unfamiliar with technique, the paper intuition is almost impossible to follow.\n\nIn the current form, experiments are more confusing than  convincing for the contribution.\nIn practice single loop optimization schemes work faster (ceteris paribus any system details).\nSo it would be beneficial to understand the applicability of such method to real practice scenario. \nSuch kind of experiments are more valuable than standard simulations with \"almost artificial data\".\n\nBasics of convergence guarantees. Given that the basic aim of the paper is to\nprovide a min-max CR method for non-convex optimization, the basic notions for convergence should appear early on in the main text (not in an appendix) and be clearly explained to the reader. Also, I would suggest that the authors provide a more intuitive discussion of the characterization lemma. This is the most interesting element in the draft - in my humble opinion - and it is less developed in the current version.\n\nFinally, a couple of comments regarding the equivalence of the two\nnotions of the notion of min-max optimization and the existence of duality gap \nwould be helpful. The authors should thoroughly proofread the manuscript for\ngrammatical and spelling errors. It is impossible to provide an exhaustive list\nhere, but one mistake/issue that occurs at almost every page of the manuscript is the usage of min-max/minimax etc alternatively without drafting the differences.", "broader_impact_concerns": "Not-applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OTALCKaEII", "reviewer": "Reviewer_VJjz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- Authors show reformulation that local minimax points are equivalent to a second-order stationarity points of an envelope function $\\Phi$. \nReformulation simplifies bivariate minimax optimization to an univariate minimization problem with specific structure. Envelope function $\\Phi$ is designed to preserve Lipschitz continuity and shows insights into minimax optimization.\n\n- Convergence rates and proofs are sound. I checked main paper and whole Appendix except Appendix G.\n\n- Even though there are some confusing parts in the paper (see weaknesses), I liked reading the paper. I encourage authors to revise those, and submit again.   \n\n------\n\n**Weaknesses:**\n- **High probability bounds** in the stochastic setup (from Harvey et al., 2019) have an important restriction on heterogeneity of functions $f_i$; they needs variance of stochastic gradients to be bounded. With restriction on functions $f_i$ to be almost homogeneous, results in the stochastic setup are much weaker.\nThis restriction is correctly reported in the Appendix F, but it is not mentioned at all in the main paper.\n\n- **Literature comparison** is lacking. \n    - Literature section (1.2 Other Related Work): While literature comparison lists multiple papers and their objectives, it is unclear how different results do they present.\\\n E.g., ``Yang et al. (2020) studied an alternating gradient descent-ascent (AGDA) algorithm in which the gradient ascent step uses the current variable $x_{t+1}$ instead of $x_t$.” What is their resulting convergence rate? How did usage of $x_{t+1}$ influence their analysis?\n    - While theoretical comparison to other algorithms is in paper, this information is scattered over paper and is not complete. I'd appreciate table (or section) with concise technical comparison. Possible columns: method name, reference, assumptions, oracle type, convergence rate. \n    - Paper proposes algorithm and compares it to GDA. While this comparison gives some insight, GDA doesn’t seem to be most relevant baseline (first-order method vs second-order method comparison). There is no experimental comparison to any other similar (second-order) methods.\n    - Some relevant methods to compare to: Follow-the-Ridge (Wang et al., 2020), Minimax Cubic-Newton (Luo et al., 2021), Inexact Minimax Cubic-Newton (Luo et al., 2021), Cubic-Solver (Luo et al., 2021), Final-Cubic-Solver (Luo et al., 2021), Cubic-Solver (Tripuraneni et al., 2018).\n\n----\n **Presentation issues:**\n\n- Paper narrative seems discontinuous.\\\nE.g., Propositions 1,2 and Fact 1 include very similar results, yet they are (unnecessarily) separated by section break and paragraphs of text.\\\nE.g., first two paragraphs of Section 3.2 are literature comparison, and can be included with the rest of literature comparison (section 1.2). \n\n- On page 9, there is detailed comparison to work (Luo et al., 2021), but it lacks technical details:\n    - Does replacing constant precision approximations by adaptive ones, lead to improvement theory/practice? \n    - `` [their solver] requires more Hessian-vector product computation per iteration!\" How much more? Is it significant?\n    - `` [their algorithm MCN and their solver] need to additionally track the approximate objective function value of the cubic subproblem, which requires additional computation.\" How much more computation is needed? Is it significant?\n\n- Experiments section: \n    - Left and right plots in Figures 1, 2 are very similar (time is almost proportional to samples). What may be more interesting is to plot performance per-iteration.\n    - What is performance compared to other second-order minmax algorithms? \n    - I’d appreciate if stochastic versions would have been compared to a full batch case -- to highlight how much time/samples were actually saved by introducing stochasticity.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article investigates the minimax optimization problem, for smooth non-convex-strongly-concave function.\nIts goal is to find a local minimax point that satisfies second-order non-degeneracy conditions.\n\nAuthors reformulate the objective as a minimization problem of an envelope function $\\Phi$ and show the properties of the reformulation mapping.\n\nDue to the structure of the reformulated envelope function, known methods are not applicable. Consequently, authors propose an algorithm Cubic-LocalMinimax for solving the problem. \nCubic-LocalMinimax tackles the structure of the problem by using (implicit) second-order cubically regularized Newton method. Authors explain how to solve implicit subproblem, they propose to utilize (first-order) Gradient Ascent method for solving the subroutine.\n\nAuthors extend their method to allow using stochastic estimates of gradients and Hessians (for empirical risk minimization) and provide high-probability bounds for the convergence as (Harvey et al., 2019).\n\nAuthors show global convergence of the proposed method in theory and experimentally show practical advantages compared to a Gradient Descent Ascent.\n\n**Questions:**\n\n- Regarding GDA-Cubic Solver: When gradients are small, GDA-Cubic Solver uses first-order GDA? For small gradient case, wouldn’t it make more sense to use a second-order methods instead? ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical adjustments:**\n- Add concise technical comparison of the literature (in theory).\n- Experimentally compare proposed algorithms to similar algorithms (not just first-order GDA).\n- Clarify assumptions in the stochastic setup (main paper) to obtain high-probability bounds.\n- Revise paper to improve presentation. \n\n**Details/suggestions:**\n- Footnote 2 & citation:  (Luo et al., 2021) had revision on arXiv on November 9, 2022\n    - it doesn’t have eq. (36)-(37)\n    - it has 3 authors\n- Name of algorithm _Inexact Cubic-LocalMinimax_ has inconsistent delimiters: [space], \"-\", [no space]\n", "broader_impact_concerns": "Work is of theoretical nature. I don't see any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lRdFNankGb", "reviewer": "Reviewer_YWBv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It has a lot of references regarding recent work and compares in detail the differences with (Luo and Chen 2021).\n2. It analyzes the exact gradient computations of solving cubic regularization.\n3. Both deterministic and stochastic cases are considered with detailed analysis.\n4. The authors conduct experiments to verify the practicability of CR methods in robust classification.\n5. The paper is clearly written with detailed proof steps.\n6. A new potential function is proposed to analyze the convergence.\n\nWeaknesses:\n1. One of my main concerns is the comparison with existing literature. My suggestion would be to list all related methods in a table, including algorithm, setting, reference, and convergence rates.\n* a) with (Luo and Chen 2021): both papers analyze the CR methods for nonconvex-strongly-concave (NCSC) functions. What is the main advantage of the current paper? Does this work achieve faster convergence? Or is it the stochastic part? Or is the difference in the algorithmic design?\n* b) with (HZZ 2020 [1]): HZZ 2020 also studies CR method for saddle points with strongly-convex-strongly-concave (SCSC) functions, which are special cases of NCSC. Could the authors provide a comparison wrt the convergence rate? \n* c) with other methods (TGDA/FR/GDN/CN, Zhang et al 2021): although these methods study local convergence, at least some experimental results should be shown to compare the convergence.\n2. The main solutions this paper aims to find are strict local minimax points. With strong concavity of y given x, finding the best response can be efficient. Therefore, the main problem reduces to finding stationary points that have positive curvatures. There have already been such work in the literature, such as [2]. In my opinion, any such work could be applied to the given problem with guaranteed convergence. The current work seems more like a simple combination of Gradient Ascent of y and CR for the envelope function.\n3. Minor issues:\n* a) Definition 1 is not quite intuitive. The authors are suggested to use the clearer equivalent version. See Lemma 16 of https://arxiv.org/pdf/1902.00618.pdf and Prop 3.6 from [3]. In fact, the case when $f(x, y)$ is *not* strongly concave in $y$ is more common and more interesting for practical use.\n* b) Fact 1 is known, for example, in [4]. \n* c) It's better to add section numbers to equations.\n* d) In Figure 3/4, the comparison with GDA is not fair. The $x$-axis should be the number of gradient calls rather than the number of epochs because CR takes more gradient steps.\n* e) Theorem 1 only finds stationary points with at most small negative curvatures, but this condition is not Fact 1.\n* f) Algorithms 4 and 5 should be brought to the main text for easy reference.\n\n\n\n[1] Cubic Regularized Newton Method for Saddle Point Models: a Global and Local Convergence Analysis, https://arxiv.org/pdf/2008.09919.pdf.\n\n[2] Escape saddle points by a simple gradient-descent based algorithm, NeurIPS 2021, https://openreview.net/pdf?id=lEf52hTHq0Q.\n\n[3] Optimality and Stability in Non-Convex Smooth Games, https://jmlr.org/papers/volume23/20-918/20-918.pdf\n\n[4] Some local properties of minimax problems, https://www.sciencedirect.com/science/article/abs/pii/0041555374901074\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work analyzes the global convergence of cubic-regularized (CR) methods to strict local minimax points with deterministic and finite-sum stochastic settings. It also conducts experiments on synthetic data and robust classification to verify the convergence of CR methods.\n\n\n--post rebuttal--\n\nThank you for your effort in the revision and for addressing my comments carefully. After checking with our reviews, it is clear that the current paper does not improve Luo et al 2022 in terms of the theoretical convergence rate, in the deterministic setting. For the stochastic setting, it provides a new convergence bound, which is not available in Luo et al. I like the new experiments added and the comprehensive comparison with existing methods. The proposed method seems to be comparable with (I)MCN and CN/GDN but is not always better. \n\nI agree that the comparison table should be moved to the main paper. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See strengths and weaknesses.", "broader_impact_concerns": "This is theoretical work, and I don't have broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new method and analysis for  the minimax optimization problem, for smooth non-convex-strongly-concave function. Their goal is to find a local minimax point that satisfies second-order non-degeneracy conditions. To achieve their goal, they 
reformulate the objective as a minimization problem of an envelope function, and then develop a second-order cubically regularized Newton method method that leverages the structure of this envelope function.  

There is some closely related, and almost concurrent work by Luo, L., Li, Y., and Chen, C, which does take away from some of the novelty of the paper. But the authors do a good job of comparing their work to Luo et. al. and amply cite Luo et. al, and thus I find this acceptable. 

The major issue brought up by the reviewers was a lack of clarity and insufficient comparison to the literature. I can see that the authors have done much to address this, and many other questions and recommendations of the reviewers. But authors also posted their substantial revision quite late in the process (5 days after discussion phase), making it difficult for the reviewers to carefully check this new version. Furthermore, all three reviewers still see some issues with clarity, and thus am accepting *conditioned* on improving clarity and to give time for the reviewers to check again the final version.

In particular, reviewer VJjz requested:

1.  What was formally Section I in the appendix (now in the main paper), should be re-written and in it's current state was very hard to read
2. "Paper suffers from insufficient literature comparison and presentation issues. I consider claims related to paper's novelty unclear/unsupported." I see the authors have made a great effort on this, but bare in mind I will ask the reviewers to check the literature comparison again after the next re-submission.

Reviewer YWBv pointed out that 

1. Table 2, some typos (additional space) in e.g. $X=N$
2. Table 3, Variance assumption should be $\sigma^2$
3. Def 1, the two parentheses should be combined

Finally, may I also request that the authors adjust the statement in the abstract "Experimental results demonstrate faster convergence of our stochastic Cubic-LocalMinimax than some existing algorithms" to qualify exactly in which experimental setting is Cubic-LocalMinimax faster. 

Decision: Accept (poster)