Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity

1 Introduction . Motivated from the recent interest in solving adversarial formulations in machine learning such as generative adversarial networks ( GANs ) [ Goodfellow et al. , 2014 ] , we consider in this paper a more abstract formulation of the problem and focus on solving the following unconstrained stochastic variational inequality ( VI ) problem:2 Find x∗ ∈ Rd such that ξ ( x∗ ) = 1 n n∑ i=1 ξi ( x ∗ ) = 0 , ( 1 ) where each ξi : Rd → Rd is Lipschitz continuous . Further , we assume that the problem ( 1 ) has a unique3 solution x∗ and that the operator ξ is µ-quasi-strongly monotone : there is a µ ≥ 0 such that : 〈ξ ( x ) , x− x∗〉 ≥ µ‖x− x∗‖2 ∀x ∈ Rd ( 2 ) ∗Corresponding author : nicolasloizou1 @ gmail.com . †Canada CIFAR AI Chair . 2While our presentation focuses on this finite-sum structure , most of our convergence results can easily be adapted to the general stochastic setting ( see App . D ) . Also , we do not use the full power of variational inequalities that usually have constraints [ Harker and Pang , 1990 ] , but standard algorithms for ( 1 ) are coming from this literature [ Gidel et al. , 2018 ] . 3This assumption can be relaxed ; but for simplicity of exposition we enforce it . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . If µ = 0 , then we say that ξ satisfies the variational stability condition : 〈ξ ( x ) , x− x∗〉 ≥ 0 [ Hsieh et al. , 2020 ] . In the variational inequality literature , condition ( 2 ) is also known as strong stability condition [ Mertikopoulos and Zhou , 2019 ] or as strong Minty variational inequality ( MVI ) [ Diakonikolas et al. , 2021 , Song et al. , 2020 ] . Problem ( 1 ) generalizes the solution of several types of stochastic smooth games [ Facchinei and Kanzow , 2007 , Scutari et al. , 2010 , Mertikopoulos and Zhou , 2019 ] . The simplest example is the unconstrained min-max optimization problem ( also called a zero-sum game ) : min x1∈Rd1 max x2∈Rd2 1 n n∑ i=1 gi ( x1 , x2 ) , ( 3 ) where each component function gi : Rd1×Rd2 → R is assumed to be smooth . Here , ξi represents the appropriate concatenation of the block-gradients of gi : ξi ( x ) : = ( ∇x1gi ( x1 , x2 ) ; −∇x2gi ( x1 , x2 ) ) , where x : = ( x1 ; x2 ) . Solving ( 1 ) then amounts to finding a stationary point x∗ = ( x∗1 ; x ∗ 2 ) for ( 3 ) , which under a convex-concavity assumption for gi for example , implies that it is a global solution for the min-max problem . More generally , we might seek the pure Nash equilibrium of a k-players game , where each player j is simultaneously trying to find the action x∗j which minimizes with respect to xj ∈ Rdj their own cost function 1n ∑n i=1 f ( j ) i ( xj , x−j ) , while the other players are playing x−j , which represents x = ( x1 , . . . , xk ) with the component j removed . Here , ξi ( x ) is the concatenation over all possible j ’ s of∇xjf ( j ) i ( xj , x−j ) . Such finite sum formulations appear in several machine learning applications such as generative adversarial networks ( GANs ) [ Goodfellow et al. , 2014 ] , robust learning [ Wen et al. , 2014 ] or even some formulations of reinforcement learning [ Pfau and Vinyals , 2016 ] . A standard algorithm that has been used to solve ( 1 ) is the stochastic version of the classical gradient method [ Dem ’ yanov and Pevnyi , 1972 , Nemirovski et al. , 2009 ] or its variance reduced version [ Balamurugan and Bach , 2016 ] , that we call stochastic gradient descent-ascent ( SGDA ) in this paper.4 More recently , Mescheder et al . [ 2017 ] analyzed some limitations of the gradient method in the context of GAN training and proposed an alternative efficient algorithm which could be used to solve ( 1 ) that they called consensus optimization ( CO ) , which combines gradient updates with the minimization of ‖ξ ( x ) ‖2 . While the practical version of their algorithm for large n is stochastic ( SCO , that randomly samples i ’ s ) and displayed good performance [ Mescheder et al. , 2017 ] , the only global convergence rate guarantees existing in the literature so far is only for the deterministic variant [ Azizian et al. , 2020 , Abernethy et al. , 2021 ] . The classical results from the stochastic VI literature are inappropriate for several reasons . First , a uniform bound over x on the variance E [ ‖ξi ( x ) − ξ ( x ) ‖2 ] is typically assumed to get convergence guarantees ( see e.g . Nemirovski et al . [ 2009 ] , Gidel et al . [ 2018 ] , Mertikopoulos and Zhou [ 2019 ] , Yang et al . [ 2020 ] , Lin et al . [ 2020b ] ) , but this is not compatible with the unconstrained aspect of ( 1 ) . For example , suppose gi is a quadratic function , then the variance typically goes to infinity as x→∞ . More appropriate relaxed assumptions have been considered to prove the convergence of other algorithms for ( 1 ) such as the stochastic extragradient method [ Hsieh et al. , 2020 , Mishchenko et al. , 2020 ] and its variance-reduced version [ Chavdarova et al. , 2019 ] , or the stochastic Hamiltonian gradient method [ Loizou et al. , 2020 ] , but not yet to the best of our knowledge for SGDA nor SCO . Second , the classical analysis for SGDA [ Nemirovski et al. , 2009 ] typically considers the convergence of the average of the iterates rather than for the last-iterate . However , as pointed out among others by Daskalakis et al . [ 2018 ] and Chavdarova et al . [ 2019 ] , getting last-iterate convergence is important to apply the methods on potentially non-monotone problems such as GANs , where averaging is not appropriate . The only non-asymptotic last iterate convergence result for SGDA that we are aware of is Lin et al . [ 2020b ] , which focuses on a different class of problems ( not assuming quasi-strong monotonicity ) but it relies on strong assumptions on E [ ‖ξi ( x ) ‖2 ] ( see Section 4 ) . In this paper , we address both of these issues . We generalize the recent improved analysis of SGD [ Gower et al. , 2019 ] to the case of unconstrained stochastic variational inequality ( 1 ) , and prove the last-iterate convergence for both SGDA and SCO without requiring any bounded variance assumption . We focus on quasi-strongly monotone VI problems , a class of structured non-monotone operators for which we are able to provide tight convergence guarantees and avoid the standard issues ( cycling and divergence of the methods ) appearing in the more general non-monotone regime . 4We use this suggestive name motivated from the min-max formulation ( 3 ) , though we also call SGDA the simple update xk+1 = xk − αξik ( x k ) to solve ( 1 ) in the more general non-zero sum game scenario . Main Contributions . The key contributions of this work are summarized as follows : • We propose the expected co-coercivity ( EC ) assumption , which is the appropriate generalization of the expected smoothness assumption from Gower et al . [ 2019 ] to Problem ( 1 ) . We explain the benefits of EC and show that is strictly weaker than the bounded variance assumption and “ growth conditions ” previously used for the analysis of stochastic algorithms for ( 1 ) . • Using the EC assumption , we prove the first last-iterate convergence guarantees for stochastic gradient descent-ascent ( SGDA ) on ( 1 ) without any unrealistic noise assumption . We show a linear convergence rate to a neighborhood of x∗ when constant step-size is used , and a O ( 1/k ) rate to the exact solution when using a decreasing step-size rule . For the latter , we propose a theoretically motivated switching rule from a constant to a decreasing step-size to get faster convergence . • Using the EC assumption , we provide the first convergence analysis of a stochastic variant ( SCO ) of the consensus optimization ( CO ) algorithm proposed by Mescheder et al . [ 2017 ] and previously used to trained GANs . In particular , we prove last-iterate convergence for SCO , for both constant and decreasing step-sizes . As a corollary of our results , we obtain an improved convergence analysis for the deterministic CO . Furthermore , we explain how the update rule of the stochastic Hamiltonian gradient descent [ Loizou et al. , 2020 ] is a special case of the SCO and show that in this scenario , our analysis matches the theoretical guarantees presented in Loizou et al . [ 2020 ] . • Inspired by recent results from the optimization literature [ Gower et al. , 2019 ] , we give the first stochastic reformulation of the variational inequality problem ( 1 ) which enables us to provide convergence guarantees of SGDA and SCO under the arbitrary sampling paradigm [ Richtárik and Takáč , 2016 ] . This allows us to give insights into the complexity of minibatching . 2 Arbitrary Sampling : Stochastic Reformulation of Problem ( 1 ) In this work , we provide theorems through which we can analyze all minibatch variants of the two algorithms under study , SGDA and SCO . To do this , we construct a so-called “ stochastic reformulation ” of the variational inequality problem ( 1 ) . Our approach is inspired by recently proposed stochastic reformulations of standard optimization problems , like the empirical risk minimization in Gower et al . [ 2019 ] and linear systems in Richtárik and Takác [ 2020 ] , Loizou and Richtárik [ 2020a , b ] . In each step of our algorithms , we assume we are given access to unbiased estimates g ( x ) ∈ Rd of the operator such that E [ g ( x ) ] = ξ ( x ) . For example , we can use a minibatch to form an estimate of the operator such as g ( x ) = 1b ∑ i∈S ξi ( x ) , where S ⊂ { 1 , . . . , n } will be chosen uniformly at random and |S| = b . To allow for any form of minibatching , we use the arbitrary sampling notation g ( x ) = ξv ( x ) : = 1n ∑n i=1 viξi ( x ) , where v ∈ Rn+ is a random sampling vector drawn from some distribution D such that ED [ vi ] = 1 , for i = 1 , . . . , n. Note that the unbiasedness follows immediately from this definition of the sampling vector : ED [ ξv ( x ) ] = 1n ∑n i=1 ED [ vi ] ξi ( x ) = ξ ( x ) . Thus , with each user-defined distribution D , we are able to introduce a stochastic reformulation of problem ( 1 ) as follows : Find x∗ ∈ Rd such that ED [ ξv ( x ∗ ) : = 1 n n∑ i=1 viξi ( x ∗ ) ] = 0 . ( 4 ) Since ξv ( x ) as an unbiased estimate of the operator ξ ( x ) , we can now use stochastic ( simultaneous ) gradient descent-ascent ( SGDA ) to solve ( 4 ) as follows : xk+1 = xk − αkξvk ( xk ) , ( 5 ) where vk ∼ D is sampled i.i.d at each iteration and αk > 0 is a stepsize . We highlight that in our analysis , we allow to select any distribution D that satisfies ED [ vi ] = 1 ∀i , and for different selection of D , ( 5 ) yields different interpretation as an SGDA method for solving the original problem ( 1 ) . In this work , we mostly focus on the b–minibatch sampling , however note that our analysis holds for every form of minibatching and for several choices of sampling vectors v. Definition 2.1 ( Minibatch sampling ) . Let b ∈ [ n ] . We say that v ∈ Rn is a b–minibatch sampling if for every subset S ∈ [ n ] with |S| = b , we have that P [ v = nb ∑ i∈S ei ] = 1/ ( n b ) : = b ! ( n−b ) ! n ! By using a double counting argument , one can show that if v is a b–minibatch sampling , it is also a valid sampling vector ( ED [ vi ] = 1 ) [ Gower et al. , 2019 ] . See Gower et al . [ 2019 ] for other choices of sampling vectors v .

Summary: This paper proposes convergence analysis on the optimization algorithms of smooth games, where the problem is formulated as a stochastic variational inequality, under the novel assumptions called expected co-coercivity and quasi-strong monotonicity. These assumptions serve the role of smoothness and strong convexity in the classical analyses of SGD for convex minimization. The authors show the last-iterate convergence rates for stochastic gradient descent-ascent (SGDA) and stochastic consensus optimization (SCO). The authors also relax the assumptions on the coordinate sampling schemes and on the noise.