Minimax Optimization with Smooth Algorithmic Adversaries

1 INTRODUCTION . This paper considers minimax optimization minx maxy f ( x , y ) in the context of two-player zero-sum games , where the min-player ( controlling x ) tries to minimize objective f assuming a worst-case opponent ( controlling y ) that acts so as to maximize it . Minimax optimization naturally arises in a variety of important machine learning paradigms , with the most prominent examples being the training of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and adversarially robust models ( Madry et al. , 2018 ) . These applications commonly engage deep neural networks with various techniques such as convolution , recurrent layers , and batch normalization . As a result , the objective function f is highly nonconvex in x and nonconcave in y. Theoretically , minimax optimization has been extensively studied starting from the seminal work of von Neumann ( Neumann , 1928 ) , with many efficient algorithms proposed for solving it ( Robinson , 1951 ; Korpelevich , 1976 ; Nemirovski , 2004 ) . A majority of these classical results have been focused on convex-concave functions , and heavily rely on the minimax theorem , i.e. , minx maxy f ( x , y ) = maxy minx f ( x , y ) , which no longer holds beyond the convex-concave setting . Recent line of works ( Lin et al. , 2020a ; Nouiehed et al. , 2019 ; Thekumparampil et al. , 2019 ; Lin et al. , 2020b ; Ostrovskii et al. , 2020 ) address the nonconvex-concave setting where f is nonconvex in x but concave in y by proposing meaningful optimality notions and designing algorithms to find such points . A key property heavily exploited in this setting is that the inner maximization over y given a fixed x can be computed efficiently , which unfortunately does not extend to the nonconvex-nonconcave setting . Consequently , nonconvex-nonconcave optimization remains challenging , and two of the most fundamental theoretical issues are still unresolved : ( i ) what is an appropriate notion of optimality that can be computed efficiently ? and ( ii ) can we design algorithms that do not suffer from cycling or diverging behavior ? Practitioners often use simple and popular algorithms such as gradient descent ascent ( GDA ) and other variants for solving these challenging optimization problems . While these algorithms seem to perform well in terms of producing high quality images in GANs and robust models in adversarial training , they are also highly unstable , particularly in training GANs . Indeed the instability of GDA and other empirically popular methods is not surprising since they are known to not converge even in very simple settings ( Daskalakis & Panageas , 2018a ; Bailey et al. , 2020 ) . This current state of affairs strongly motivates the need to develop a strong theoretical foundation for nonconvex-nonconcave minimax optimization and to design better algorithms for solving them . This work considers the challenging nonconvex-nonconcave setting . Our framework sprouts from the practical consideration that under a computational budget , the max-player can not fully maximize f ( x , · ) since nonconcave maximization is NP-hard in general . In fact , in both the settings of GAN and adversarial training , in practice , the max-player employs simple gradient based algorithms such as gradient ascent , run for a few steps – on the order of 5 for GAN training and 40 for adversarial training ( see , e.g. , Arjovsky et al . 2017 ; Madry et al . 2018 ) – to estimate argmaxy f ( x , y ) . To capture this aspect , we assume that the max-player has a toolkit of multiple ( potentially randomized ) algorithmsA1 , A2 , · · · , Ak in an attempt to solve the maximization problem given fixed x , and picks the best solution among these algorithms . This motivates us to study the surrogate of the minimax optimization problem as minx maxi∈ [ k ] f ( x , Ai ( x ) ) = minx maxλ∈∆k ∑k i=1 λif ( x , Ai ( x ) ) , ( 1 ) where ∆k denotes the k-dimensional simplex , and Ai ( x ) denotes the output of algorithm Ai for a given x . When both the objective function f and the algorithms { Ai } ki=1 are smooth ( defined formally in Section 3 ) , we can show that ( 1 ) becomes a smooth nonconvex-concave minimax optimization problem , where recent advances can be leveraged in solving such problems . In particular , given the smooth algorithms deployed by the adversary ( i.e . the max-player ) , this paper proposes two algorithms for solving problems in ( 1 ) . The first algorithm is based on stochastic gradient descent ( SGD ) , which is guaranteed to find an appropriate notion of “ -approximate stationary point ” in O ( −4 ) gradient computations . The second algorithm is based on proximal algorithm , in the case of deterministic adversarial algorithms { Ai } ki=1 , this algorithm has an improved gradient complexity O ( −3 ) or Õ ( poly ( k ) / 2 ) depending on the choice of subroutine within the algorithm . All our algorithms are guaranteed to make monotonic progress , thus having no limit cycles . Our second set of results show that , many popular algorithms deployed by the adversary such as multi-step stochastic gradient ascent , and multi-step stochastic Nesterov ’ s accelerated gradient ascent are in fact smooth . Therefore , our framework readily applies to those settings in practice . We also present complementing experimental results showing that our theoretical framework and algorithm succeed on simple , conceptual examples of GAN and adversarial training . While more work is needed to scale our approach to large scale benchmarks in GAN and adversarial training , our experimental results serve as a proof of concept demonstration that our algorithm converges to desirable points in practice , at least on simple conceptual examples . 2 RELATED WORK . Due to lack of space , we focus our discussion here on directly related works and present a more detailed overview of related work in Appendix A . Many existing works that propose local optimality notions in nonconvex-nonconcave minimax optimization suffer from nonexistence of such points in general ( Ratliff et al. , 2013 ; Ratliff et al. , 2016 ; Jin et al. , 2020 ; Fiez et al. , 2020 ; Zhang et al. , 2020a ; Farnia & Ozdaglar , 2020 ) . To the best of our knowledge , the only works that propose a relaxed local optimality notion that is shown to exist and be computable in polynomial time is due to Keswani et al . ( 2020 ) ; Mangoubi & Vishnoi ( 2021 ) . The aforementioned works are similar to this paper in the sense that the min-player faces the max-player with computational restrictions , but are different from ours in terms of the model of the max-player and the algorithms to solve the problem . Concretely , Keswani et al . ( 2020 ) ; Mangoubi & Vishnoi ( 2021 ) consider the stationary points of a “ smoothed '' greedy-max function , which is computed by maximizing along a locally ascending path when fixing the min-player . In contrast , our paper considers the stationary points of a “ max '' function computed by particular smooth algorithms given their initializations , which is a more faithful surrogate to the objective that the min-player wishes to minimize . Depending on the choice of smooth algorithms and their initializations , the optimal points of Keswani et al . ( 2020 ) ; Mangoubi & Vishnoi ( 2021 ) and the optimal points in our paper can be very different , making them incomparable . 3 PRELIMINARIES . In this section , we present problem formulation and preliminaries . We consider function f satisfying Assumption 1 . We denote w = ( x , y ) , and assume f : Rd1 × Rd2 → R is : ( a ) B-bounded i.e. , |f ( w ) | ≤ B , ( b ) G-Lipschitz i.e. , |f ( w1 ) − f ( w2 ) | ≤ G‖w1 − w2‖ , ( c ) L-gradient Lipschitz i.e. , ‖∇f ( w1 ) −∇f ( w2 ) ‖ ≤ L‖w1 − w2‖ , ( d ) ρ-Hessian Lipschitz i.e. , ‖∇2f ( w1 ) −∇2f ( w2 ) ‖ ≤ ρ‖w1 − w2‖ . where ‖·‖ denotes Euclidean norm for vectors and operator norm for matrices . We aim to solve minx∈Rd1 maxy∈Rd2 f ( x , y ) . Since maxy∈Rd2 f ( x , y ) involves non-concave maximization and hence is NP-hard in the worst case , we intend to play against algorithm ( s ) that y-player uses to compute her strategy . Concretely , given x ∈ Rd1 , we assume that the y-player chooses her ( potentially random ) strategy ŷz ( x ) = Ai∗ ( x ) ( x , zi∗ ( x ) ) , where we use shorthand z : = ( z1 , · · · , zk ) , as i∗ ( x ) = argmaxi∈ [ k ] f ( x , Ai ( x , zi ) ) , where A1 , · · · , Ak are k deterministic algorithms that take as input x and a random seed zi ∈ R ` , where zi are all independent . Note that the framework captures randomized algorithms e.g. , A could be stochastic gradient ascent on f ( x , · ) , with initialization , minibatching etc . determined by the random seed z . This also incorporates running the same algorithm multiple times , with different seeds and then choosing the best strategy . We now reformulate the minimax objective function to : minx∈Rd1 g ( x ) where g ( x ) : = Ez [ f ( x , ŷz ( x ) ) ] . ( 2 ) For general algorithms Ai , the functions f ( x , Ai ( x , zi ) ) need not be continuous even when f satisfies Assumption 1 . However , if the algorithms Ai are smooth as defined below , the functions f ( x , Ai ( x , zi ) ) behave much more nicely . Definition 1 ( Algorithm Smoothness ) . A randomized algorithm A : Rd1 × R ` → Rd2 is : ( a ) G-Lipschitz , if ‖A ( x1 , z ) −A ( x2 , z ) ‖ ≤ G‖x1 − x2‖ for any z . ( b ) L-gradient Lipschitz , if ‖DA ( x1 , z ) −DA ( x2 , z ) ‖ ≤ L‖x1 − x2‖ for any z . Here DA ( x , z ) ∈ Rd1 × Rd2 is the Jacobian of the function A ( · , z ) for a fixed z . The following lemma tells us that f ( x , A ( x , z ) ) behaves nicely whenever A is a Lipschitz and gradient Lipschitz algorithm . For deterministic algorithms , we also use the shortened notation A ( x ) and DA ( x ) . Lemma 1 . Suppose A is G′-Lipschitz and L′-gradient Lipschitz and f satisfies Assumption 1 . Then , for a fixed z , function f ( · , A ( · , z ) ) is G ( 1 +G′ ) -Lipschitz and L ( 1 +G′ ) 2 +GL′-gradient Lipschitz . While g ( x ) defined in ( 2 ) is not necessarily gradient Lipschitz , it can be shown to be weakly-convex as defined below . Note that an L-gradient Lipschitz function is L-weakly convex . Definition 2 . A function g : Rd1 → R is L-weakly convex if ∀ x , there exists a vector ux satisfying : g ( x′ ) ≥ g ( x ) + 〈ux , x′ − x〉 − L2 ‖x ′ − x‖2 ∀ x′ . ( 3 ) Any vector ux satisfying this property is called the subgradient of g at x and is denoted by ∇g ( x ) . An important property of weakly convex function is that the maximum over a finite number of weakly convex function is still a weakly convex function . Lemma 2 . Given L-weakly convex functions g1 , · · · , gk : Rd → R , the maximum function g ( · ) : = maxi∈ [ k ] gi ( · ) is also L-weakly convex and the set of subgradients of g ( · ) at x is given by : ∂g ( x ) = { ∑ j∈S ( x ) λj∇gj ( x ) : λj ≥ 0 , ∑ j∈S ( x ) λj = 1 } , where S ( x ) : = argmaxi∈ [ k ] gi ( x ) . Thus , under Assumption 1 and the assumptions Ai are all G′-Lipschitz and L′-gradient Lipschitz , then g ( · ) defined in ( 2 ) is L ( 1 + G′ ) 2 + GL′-weakly convex . The usual optimality notion for weakly-convex functions is approximate first order stationary points ( Davis & Drusvyatskiy , 2018 ) . Approximate first-order stationary point for weakly convex functions : In order to define approximate stationary points , we also need the notion of Moreau envelope . Definition 3 . The Moreau envelope of a function g : Rd1 → R and parameter λ is : gλ ( x ) = minx′∈Rd1 g ( x ′ ) + ( 2λ ) −1 ‖x− x′‖2 . ( 4 ) The following lemma provides useful properties of the Moreau envelope . Lemma 3 . For an L-weakly convex function g : Rd1 → R and λ < 1/L , we have : ( a ) The minimizer x̂λ ( x ) = arg minx′∈Rd1 g ( x′ ) + ( 2λ ) −1‖x−x′‖2 is unique and g ( x̂λ ( x ) ) ≤ gλ ( x ) ≤ g ( x ) . Furthermore , arg minx g ( x ) = arg minx gλ ( x ) . ( b ) gλ is λ−1 ( 1 + ( 1− λL ) −1 ) -smooth and thus differentiable , and ( c ) minu∈∂g ( x̂λ ( x ) ) ‖u‖ ≤ λ−1‖x̂λ ( x ) − x‖ = ‖∇gλ ( x ) ‖ . First order stationary points ( FOSP ) of a non-smooth nonconvex function are well-defined , i.e. , x∗ is a FOSP of a function g ( x ) if , 0 ∈ ∂f ( x∗ ) . However , unlike smooth functions , it is nontrivial to define an approximate FOSP . For example , if we define an ε-FOSP as the point x with minu∈∂g ( x ) ‖u‖ ≤ ε , where ∂g ( x ) denotes the subgradients of g at x , there may never exist such a point for sufficiently small ε , unless x is exactly a FOSP . In contrast , by using above properties of the Moreau envelope of a weakly convex function , it ’ s approximate FOSP can be defined as ( Davis & Drusvyatskiy , 2018 ) : Definition 4 . Given an L-weakly convex function g , we say that x∗ is an ε-first order stationary point ( ε-FOSP ) if , ‖∇g1/2L ( x∗ ) ‖ ≤ ε , where g1/2L is the Moreau envelope with parameter 1/2L . Using Lemma 3 , we can show that for any ε-FOSP x∗ , there exists x̂ such that ‖x̂− x∗‖ ≤ ε/2L and minu∈∂g ( x̂ ) ‖u‖ ≤ ε . In other words , an ε-FOSP is O ( ε ) close to a point x̂ which has a subgradient smaller than ε . Other notions of FOSP proposed recently such as in Nouiehed et al . ( 2019 ) can be shown to be a strict generalization of the above definition .

Summary: The paper considers a min-max optimization of general nonconvex functions. The paper proposes methods to approximately find stationary points by "smoothing" the max part. Experimental results show the superiority of the proposed approach to previous work.