Minimax Optimization with Smooth Algorithmic Adversaries

1 INTRODUCTION . This paper considers minimax optimization minx maxy f ( x , y ) in the context of two-player zero-sum games , where the min-player ( controlling x ) tries to minimize objective f assuming a worst-case opponent ( controlling y ) that acts so as to maximize it . Minimax optimization naturally arises in a variety of important machine learning paradigms , with the most prominent examples being the training of generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and adversarially robust models ( Madry et al. , 2018 ) . These applications commonly engage deep neural networks with various techniques such as convolution , recurrent layers , and batch normalization . As a result , the objective function f is highly nonconvex in x and nonconcave in y. Theoretically , minimax optimization has been extensively studied starting from the seminal work of von Neumann ( Neumann , 1928 ) , with many efficient algorithms proposed for solving it ( Robinson , 1951 ; Korpelevich , 1976 ; Nemirovski , 2004 ) . A majority of these classical results have been focused on convex-concave functions , and heavily rely on the minimax theorem , i.e. , minx maxy f ( x , y ) = maxy minx f ( x , y ) , which no longer holds beyond the convex-concave setting . Recent line of works ( Lin et al. , 2020a ; Nouiehed et al. , 2019 ; Thekumparampil et al. , 2019 ; Lin et al. , 2020b ; Ostrovskii et al. , 2020 ) address the nonconvex-concave setting where f is nonconvex in x but concave in y by proposing meaningful optimality notions and designing algorithms to find such points . A key property heavily exploited in this setting is that the inner maximization over y given a fixed x can be computed efficiently , which unfortunately does not extend to the nonconvex-nonconcave setting . Consequently , nonconvex-nonconcave optimization remains challenging , and two of the most fundamental theoretical issues are still unresolved : ( i ) what is an appropriate notion of optimality that can be computed efficiently ? and ( ii ) can we design algorithms that do not suffer from cycling or diverging behavior ? Practitioners often use simple and popular algorithms such as gradient descent ascent ( GDA ) and other variants for solving these challenging optimization problems . While these algorithms seem to perform well in terms of producing high quality images in GANs and robust models in adversarial training , they are also highly unstable , particularly in training GANs . Indeed the instability of GDA and other empirically popular methods is not surprising since they are known to not converge even in very simple settings ( Daskalakis & Panageas , 2018a ; Bailey et al. , 2020 ) . This current state of affairs strongly motivates the need to develop a strong theoretical foundation for nonconvex-nonconcave minimax optimization and to design better algorithms for solving them . This work considers the challenging nonconvex-nonconcave setting . Our framework sprouts from the practical consideration that under a computational budget , the max-player can not fully maximize f ( x , · ) since nonconcave maximization is NP-hard in general . In fact , in both the settings of GAN and adversarial training , in practice , the max-player employs simple gradient based algorithms such as gradient ascent , run for a few steps – on the order of 5 for GAN training and 40 for adversarial training ( see , e.g. , Arjovsky et al . 2017 ; Madry et al . 2018 ) – to estimate argmaxy f ( x , y ) . To capture this aspect , we assume that the max-player has a toolkit of multiple ( potentially randomized ) algorithmsA1 , A2 , · · · , Ak in an attempt to solve the maximization problem given fixed x , and picks the best solution among these algorithms . This motivates us to study the surrogate of the minimax optimization problem as minx maxi∈ [ k ] f ( x , Ai ( x ) ) = minx maxλ∈∆k ∑k i=1 λif ( x , Ai ( x ) ) , ( 1 ) where ∆k denotes the k-dimensional simplex , and Ai ( x ) denotes the output of algorithm Ai for a given x . When both the objective function f and the algorithms { Ai } ki=1 are smooth ( defined formally in Section 3 ) , we can show that ( 1 ) becomes a smooth nonconvex-concave minimax optimization problem , where recent advances can be leveraged in solving such problems . In particular , given the smooth algorithms deployed by the adversary ( i.e . the max-player ) , this paper proposes two algorithms for solving problems in ( 1 ) . The first algorithm is based on stochastic gradient descent ( SGD ) , which is guaranteed to find an appropriate notion of “ -approximate stationary point ” in O ( −4 ) gradient computations . The second algorithm is based on proximal algorithm , in the case of deterministic adversarial algorithms { Ai } ki=1 , this algorithm has an improved gradient complexity O ( −3 ) or Õ ( poly ( k ) / 2 ) depending on the choice of subroutine within the algorithm . All our algorithms are guaranteed to make monotonic progress , thus having no limit cycles . Our second set of results show that , many popular algorithms deployed by the adversary such as multi-step stochastic gradient ascent , and multi-step stochastic Nesterov ’ s accelerated gradient ascent are in fact smooth . Therefore , our framework readily applies to those settings in practice . We also present complementing experimental results showing that our theoretical framework and algorithm succeed on simple , conceptual examples of GAN and adversarial training . While more work is needed to scale our approach to large scale benchmarks in GAN and adversarial training , our experimental results serve as a proof of concept demonstration that our algorithm converges to desirable points in practice , at least on simple conceptual examples . 2 RELATED WORK . Due to lack of space , we focus our discussion here on directly related works and present a more detailed overview of related work in Appendix A . Many existing works that propose local optimality notions in nonconvex-nonconcave minimax optimization suffer from nonexistence of such points in general ( Ratliff et al. , 2013 ; Ratliff et al. , 2016 ; Jin et al. , 2020 ; Fiez et al. , 2020 ; Zhang et al. , 2020a ; Farnia & Ozdaglar , 2020 ) . To the best of our knowledge , the only works that propose a relaxed local optimality notion that is shown to exist and be computable in polynomial time is due to Keswani et al . ( 2020 ) ; Mangoubi & Vishnoi ( 2021 ) . The aforementioned works are similar to this paper in the sense that the min-player faces the max-player with computational restrictions , but are different from ours in terms of the model of the max-player and the algorithms to solve the problem . Concretely , Keswani et al . ( 2020 ) ; Mangoubi & Vishnoi ( 2021 ) consider the stationary points of a “ smoothed '' greedy-max function , which is computed by maximizing along a locally ascending path when fixing the min-player . In contrast , our paper considers the stationary points of a “ max '' function computed by particular smooth algorithms given their initializations , which is a more faithful surrogate to the objective that the min-player wishes to minimize . Depending on the choice of smooth algorithms and their initializations , the optimal points of Keswani et al . ( 2020 ) ; Mangoubi & Vishnoi ( 2021 ) and the optimal points in our paper can be very different , making them incomparable . 3 PRELIMINARIES . In this section , we present problem formulation and preliminaries . We consider function f satisfying Assumption 1 . We denote w = ( x , y ) , and assume f : Rd1 × Rd2 → R is : ( a ) B-bounded i.e. , |f ( w ) | ≤ B , ( b ) G-Lipschitz i.e. , |f ( w1 ) − f ( w2 ) | ≤ G‖w1 − w2‖ , ( c ) L-gradient Lipschitz i.e. , ‖∇f ( w1 ) −∇f ( w2 ) ‖ ≤ L‖w1 − w2‖ , ( d ) ρ-Hessian Lipschitz i.e. , ‖∇2f ( w1 ) −∇2f ( w2 ) ‖ ≤ ρ‖w1 − w2‖ . where ‖·‖ denotes Euclidean norm for vectors and operator norm for matrices . We aim to solve minx∈Rd1 maxy∈Rd2 f ( x , y ) . Since maxy∈Rd2 f ( x , y ) involves non-concave maximization and hence is NP-hard in the worst case , we intend to play against algorithm ( s ) that y-player uses to compute her strategy . Concretely , given x ∈ Rd1 , we assume that the y-player chooses her ( potentially random ) strategy ŷz ( x ) = Ai∗ ( x ) ( x , zi∗ ( x ) ) , where we use shorthand z : = ( z1 , · · · , zk ) , as i∗ ( x ) = argmaxi∈ [ k ] f ( x , Ai ( x , zi ) ) , where A1 , · · · , Ak are k deterministic algorithms that take as input x and a random seed zi ∈ R ` , where zi are all independent . Note that the framework captures randomized algorithms e.g. , A could be stochastic gradient ascent on f ( x , · ) , with initialization , minibatching etc . determined by the random seed z . This also incorporates running the same algorithm multiple times , with different seeds and then choosing the best strategy . We now reformulate the minimax objective function to : minx∈Rd1 g ( x ) where g ( x ) : = Ez [ f ( x , ŷz ( x ) ) ] . ( 2 ) For general algorithms Ai , the functions f ( x , Ai ( x , zi ) ) need not be continuous even when f satisfies Assumption 1 . However , if the algorithms Ai are smooth as defined below , the functions f ( x , Ai ( x , zi ) ) behave much more nicely . Definition 1 ( Algorithm Smoothness ) . A randomized algorithm A : Rd1 × R ` → Rd2 is : ( a ) G-Lipschitz , if ‖A ( x1 , z ) −A ( x2 , z ) ‖ ≤ G‖x1 − x2‖ for any z . ( b ) L-gradient Lipschitz , if ‖DA ( x1 , z ) −DA ( x2 , z ) ‖ ≤ L‖x1 − x2‖ for any z . Here DA ( x , z ) ∈ Rd1 × Rd2 is the Jacobian of the function A ( · , z ) for a fixed z . The following lemma tells us that f ( x , A ( x , z ) ) behaves nicely whenever A is a Lipschitz and gradient Lipschitz algorithm . For deterministic algorithms , we also use the shortened notation A ( x ) and DA ( x ) . Lemma 1 . Suppose A is G′-Lipschitz and L′-gradient Lipschitz and f satisfies Assumption 1 . Then , for a fixed z , function f ( · , A ( · , z ) ) is G ( 1 +G′ ) -Lipschitz and L ( 1 +G′ ) 2 +GL′-gradient Lipschitz . While g ( x ) defined in ( 2 ) is not necessarily gradient Lipschitz , it can be shown to be weakly-convex as defined below . Note that an L-gradient Lipschitz function is L-weakly convex . Definition 2 . A function g : Rd1 → R is L-weakly convex if ∀ x , there exists a vector ux satisfying : g ( x′ ) ≥ g ( x ) + 〈ux , x′ − x〉 − L2 ‖x ′ − x‖2 ∀ x′ . ( 3 ) Any vector ux satisfying this property is called the subgradient of g at x and is denoted by ∇g ( x ) . An important property of weakly convex function is that the maximum over a finite number of weakly convex function is still a weakly convex function . Lemma 2 . Given L-weakly convex functions g1 , · · · , gk : Rd → R , the maximum function g ( · ) : = maxi∈ [ k ] gi ( · ) is also L-weakly convex and the set of subgradients of g ( · ) at x is given by : ∂g ( x ) = { ∑ j∈S ( x ) λj∇gj ( x ) : λj ≥ 0 , ∑ j∈S ( x ) λj = 1 } , where S ( x ) : = argmaxi∈ [ k ] gi ( x ) . Thus , under Assumption 1 and the assumptions Ai are all G′-Lipschitz and L′-gradient Lipschitz , then g ( · ) defined in ( 2 ) is L ( 1 + G′ ) 2 + GL′-weakly convex . The usual optimality notion for weakly-convex functions is approximate first order stationary points ( Davis & Drusvyatskiy , 2018 ) . Approximate first-order stationary point for weakly convex functions : In order to define approximate stationary points , we also need the notion of Moreau envelope . Definition 3 . The Moreau envelope of a function g : Rd1 → R and parameter λ is : gλ ( x ) = minx′∈Rd1 g ( x ′ ) + ( 2λ ) −1 ‖x− x′‖2 . ( 4 ) The following lemma provides useful properties of the Moreau envelope . Lemma 3 . For an L-weakly convex function g : Rd1 → R and λ < 1/L , we have : ( a ) The minimizer x̂λ ( x ) = arg minx′∈Rd1 g ( x′ ) + ( 2λ ) −1‖x−x′‖2 is unique and g ( x̂λ ( x ) ) ≤ gλ ( x ) ≤ g ( x ) . Furthermore , arg minx g ( x ) = arg minx gλ ( x ) . ( b ) gλ is λ−1 ( 1 + ( 1− λL ) −1 ) -smooth and thus differentiable , and ( c ) minu∈∂g ( x̂λ ( x ) ) ‖u‖ ≤ λ−1‖x̂λ ( x ) − x‖ = ‖∇gλ ( x ) ‖ . First order stationary points ( FOSP ) of a non-smooth nonconvex function are well-defined , i.e. , x∗ is a FOSP of a function g ( x ) if , 0 ∈ ∂f ( x∗ ) . However , unlike smooth functions , it is nontrivial to define an approximate FOSP . For example , if we define an ε-FOSP as the point x with minu∈∂g ( x ) ‖u‖ ≤ ε , where ∂g ( x ) denotes the subgradients of g at x , there may never exist such a point for sufficiently small ε , unless x is exactly a FOSP . In contrast , by using above properties of the Moreau envelope of a weakly convex function , it ’ s approximate FOSP can be defined as ( Davis & Drusvyatskiy , 2018 ) : Definition 4 . Given an L-weakly convex function g , we say that x∗ is an ε-first order stationary point ( ε-FOSP ) if , ‖∇g1/2L ( x∗ ) ‖ ≤ ε , where g1/2L is the Moreau envelope with parameter 1/2L . Using Lemma 3 , we can show that for any ε-FOSP x∗ , there exists x̂ such that ‖x̂− x∗‖ ≤ ε/2L and minu∈∂g ( x̂ ) ‖u‖ ≤ ε . In other words , an ε-FOSP is O ( ε ) close to a point x̂ which has a subgradient smaller than ε . Other notions of FOSP proposed recently such as in Nouiehed et al . ( 2019 ) can be shown to be a strict generalization of the above definition .

Summary: The paper is trying to address the problem of non-convex non-concave min-max optimization under the perspective of application of smoothed algorithms between the two opponents. More precisely, the paper examine a model where the max-player applied a zero-memory smooth (from differential perspective) algorithm and min-player SGD/SNAG or proximal methods providing results similar with the state of art. The convergence is reassured by "potential" arguments but the fixed point is not an equally game-theoretic notion like Nash Equilibrium.