Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity

1 Introduction . Motivated from the recent interest in solving adversarial formulations in machine learning such as generative adversarial networks ( GANs ) [ Goodfellow et al. , 2014 ] , we consider in this paper a more abstract formulation of the problem and focus on solving the following unconstrained stochastic variational inequality ( VI ) problem:2 Find x∗ ∈ Rd such that ξ ( x∗ ) = 1 n n∑ i=1 ξi ( x ∗ ) = 0 , ( 1 ) where each ξi : Rd → Rd is Lipschitz continuous . Further , we assume that the problem ( 1 ) has a unique3 solution x∗ and that the operator ξ is µ-quasi-strongly monotone : there is a µ ≥ 0 such that : 〈ξ ( x ) , x− x∗〉 ≥ µ‖x− x∗‖2 ∀x ∈ Rd ( 2 ) ∗Corresponding author : nicolasloizou1 @ gmail.com . †Canada CIFAR AI Chair . 2While our presentation focuses on this finite-sum structure , most of our convergence results can easily be adapted to the general stochastic setting ( see App . D ) . Also , we do not use the full power of variational inequalities that usually have constraints [ Harker and Pang , 1990 ] , but standard algorithms for ( 1 ) are coming from this literature [ Gidel et al. , 2018 ] . 3This assumption can be relaxed ; but for simplicity of exposition we enforce it . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . If µ = 0 , then we say that ξ satisfies the variational stability condition : 〈ξ ( x ) , x− x∗〉 ≥ 0 [ Hsieh et al. , 2020 ] . In the variational inequality literature , condition ( 2 ) is also known as strong stability condition [ Mertikopoulos and Zhou , 2019 ] or as strong Minty variational inequality ( MVI ) [ Diakonikolas et al. , 2021 , Song et al. , 2020 ] . Problem ( 1 ) generalizes the solution of several types of stochastic smooth games [ Facchinei and Kanzow , 2007 , Scutari et al. , 2010 , Mertikopoulos and Zhou , 2019 ] . The simplest example is the unconstrained min-max optimization problem ( also called a zero-sum game ) : min x1∈Rd1 max x2∈Rd2 1 n n∑ i=1 gi ( x1 , x2 ) , ( 3 ) where each component function gi : Rd1×Rd2 → R is assumed to be smooth . Here , ξi represents the appropriate concatenation of the block-gradients of gi : ξi ( x ) : = ( ∇x1gi ( x1 , x2 ) ; −∇x2gi ( x1 , x2 ) ) , where x : = ( x1 ; x2 ) . Solving ( 1 ) then amounts to finding a stationary point x∗ = ( x∗1 ; x ∗ 2 ) for ( 3 ) , which under a convex-concavity assumption for gi for example , implies that it is a global solution for the min-max problem . More generally , we might seek the pure Nash equilibrium of a k-players game , where each player j is simultaneously trying to find the action x∗j which minimizes with respect to xj ∈ Rdj their own cost function 1n ∑n i=1 f ( j ) i ( xj , x−j ) , while the other players are playing x−j , which represents x = ( x1 , . . . , xk ) with the component j removed . Here , ξi ( x ) is the concatenation over all possible j ’ s of∇xjf ( j ) i ( xj , x−j ) . Such finite sum formulations appear in several machine learning applications such as generative adversarial networks ( GANs ) [ Goodfellow et al. , 2014 ] , robust learning [ Wen et al. , 2014 ] or even some formulations of reinforcement learning [ Pfau and Vinyals , 2016 ] . A standard algorithm that has been used to solve ( 1 ) is the stochastic version of the classical gradient method [ Dem ’ yanov and Pevnyi , 1972 , Nemirovski et al. , 2009 ] or its variance reduced version [ Balamurugan and Bach , 2016 ] , that we call stochastic gradient descent-ascent ( SGDA ) in this paper.4 More recently , Mescheder et al . [ 2017 ] analyzed some limitations of the gradient method in the context of GAN training and proposed an alternative efficient algorithm which could be used to solve ( 1 ) that they called consensus optimization ( CO ) , which combines gradient updates with the minimization of ‖ξ ( x ) ‖2 . While the practical version of their algorithm for large n is stochastic ( SCO , that randomly samples i ’ s ) and displayed good performance [ Mescheder et al. , 2017 ] , the only global convergence rate guarantees existing in the literature so far is only for the deterministic variant [ Azizian et al. , 2020 , Abernethy et al. , 2021 ] . The classical results from the stochastic VI literature are inappropriate for several reasons . First , a uniform bound over x on the variance E [ ‖ξi ( x ) − ξ ( x ) ‖2 ] is typically assumed to get convergence guarantees ( see e.g . Nemirovski et al . [ 2009 ] , Gidel et al . [ 2018 ] , Mertikopoulos and Zhou [ 2019 ] , Yang et al . [ 2020 ] , Lin et al . [ 2020b ] ) , but this is not compatible with the unconstrained aspect of ( 1 ) . For example , suppose gi is a quadratic function , then the variance typically goes to infinity as x→∞ . More appropriate relaxed assumptions have been considered to prove the convergence of other algorithms for ( 1 ) such as the stochastic extragradient method [ Hsieh et al. , 2020 , Mishchenko et al. , 2020 ] and its variance-reduced version [ Chavdarova et al. , 2019 ] , or the stochastic Hamiltonian gradient method [ Loizou et al. , 2020 ] , but not yet to the best of our knowledge for SGDA nor SCO . Second , the classical analysis for SGDA [ Nemirovski et al. , 2009 ] typically considers the convergence of the average of the iterates rather than for the last-iterate . However , as pointed out among others by Daskalakis et al . [ 2018 ] and Chavdarova et al . [ 2019 ] , getting last-iterate convergence is important to apply the methods on potentially non-monotone problems such as GANs , where averaging is not appropriate . The only non-asymptotic last iterate convergence result for SGDA that we are aware of is Lin et al . [ 2020b ] , which focuses on a different class of problems ( not assuming quasi-strong monotonicity ) but it relies on strong assumptions on E [ ‖ξi ( x ) ‖2 ] ( see Section 4 ) . In this paper , we address both of these issues . We generalize the recent improved analysis of SGD [ Gower et al. , 2019 ] to the case of unconstrained stochastic variational inequality ( 1 ) , and prove the last-iterate convergence for both SGDA and SCO without requiring any bounded variance assumption . We focus on quasi-strongly monotone VI problems , a class of structured non-monotone operators for which we are able to provide tight convergence guarantees and avoid the standard issues ( cycling and divergence of the methods ) appearing in the more general non-monotone regime . 4We use this suggestive name motivated from the min-max formulation ( 3 ) , though we also call SGDA the simple update xk+1 = xk − αξik ( x k ) to solve ( 1 ) in the more general non-zero sum game scenario . Main Contributions . The key contributions of this work are summarized as follows : • We propose the expected co-coercivity ( EC ) assumption , which is the appropriate generalization of the expected smoothness assumption from Gower et al . [ 2019 ] to Problem ( 1 ) . We explain the benefits of EC and show that is strictly weaker than the bounded variance assumption and “ growth conditions ” previously used for the analysis of stochastic algorithms for ( 1 ) . • Using the EC assumption , we prove the first last-iterate convergence guarantees for stochastic gradient descent-ascent ( SGDA ) on ( 1 ) without any unrealistic noise assumption . We show a linear convergence rate to a neighborhood of x∗ when constant step-size is used , and a O ( 1/k ) rate to the exact solution when using a decreasing step-size rule . For the latter , we propose a theoretically motivated switching rule from a constant to a decreasing step-size to get faster convergence . • Using the EC assumption , we provide the first convergence analysis of a stochastic variant ( SCO ) of the consensus optimization ( CO ) algorithm proposed by Mescheder et al . [ 2017 ] and previously used to trained GANs . In particular , we prove last-iterate convergence for SCO , for both constant and decreasing step-sizes . As a corollary of our results , we obtain an improved convergence analysis for the deterministic CO . Furthermore , we explain how the update rule of the stochastic Hamiltonian gradient descent [ Loizou et al. , 2020 ] is a special case of the SCO and show that in this scenario , our analysis matches the theoretical guarantees presented in Loizou et al . [ 2020 ] . • Inspired by recent results from the optimization literature [ Gower et al. , 2019 ] , we give the first stochastic reformulation of the variational inequality problem ( 1 ) which enables us to provide convergence guarantees of SGDA and SCO under the arbitrary sampling paradigm [ Richtárik and Takáč , 2016 ] . This allows us to give insights into the complexity of minibatching . 2 Arbitrary Sampling : Stochastic Reformulation of Problem ( 1 ) In this work , we provide theorems through which we can analyze all minibatch variants of the two algorithms under study , SGDA and SCO . To do this , we construct a so-called “ stochastic reformulation ” of the variational inequality problem ( 1 ) . Our approach is inspired by recently proposed stochastic reformulations of standard optimization problems , like the empirical risk minimization in Gower et al . [ 2019 ] and linear systems in Richtárik and Takác [ 2020 ] , Loizou and Richtárik [ 2020a , b ] . In each step of our algorithms , we assume we are given access to unbiased estimates g ( x ) ∈ Rd of the operator such that E [ g ( x ) ] = ξ ( x ) . For example , we can use a minibatch to form an estimate of the operator such as g ( x ) = 1b ∑ i∈S ξi ( x ) , where S ⊂ { 1 , . . . , n } will be chosen uniformly at random and |S| = b . To allow for any form of minibatching , we use the arbitrary sampling notation g ( x ) = ξv ( x ) : = 1n ∑n i=1 viξi ( x ) , where v ∈ Rn+ is a random sampling vector drawn from some distribution D such that ED [ vi ] = 1 , for i = 1 , . . . , n. Note that the unbiasedness follows immediately from this definition of the sampling vector : ED [ ξv ( x ) ] = 1n ∑n i=1 ED [ vi ] ξi ( x ) = ξ ( x ) . Thus , with each user-defined distribution D , we are able to introduce a stochastic reformulation of problem ( 1 ) as follows : Find x∗ ∈ Rd such that ED [ ξv ( x ∗ ) : = 1 n n∑ i=1 viξi ( x ∗ ) ] = 0 . ( 4 ) Since ξv ( x ) as an unbiased estimate of the operator ξ ( x ) , we can now use stochastic ( simultaneous ) gradient descent-ascent ( SGDA ) to solve ( 4 ) as follows : xk+1 = xk − αkξvk ( xk ) , ( 5 ) where vk ∼ D is sampled i.i.d at each iteration and αk > 0 is a stepsize . We highlight that in our analysis , we allow to select any distribution D that satisfies ED [ vi ] = 1 ∀i , and for different selection of D , ( 5 ) yields different interpretation as an SGDA method for solving the original problem ( 1 ) . In this work , we mostly focus on the b–minibatch sampling , however note that our analysis holds for every form of minibatching and for several choices of sampling vectors v. Definition 2.1 ( Minibatch sampling ) . Let b ∈ [ n ] . We say that v ∈ Rn is a b–minibatch sampling if for every subset S ∈ [ n ] with |S| = b , we have that P [ v = nb ∑ i∈S ei ] = 1/ ( n b ) : = b ! ( n−b ) ! n ! By using a double counting argument , one can show that if v is a b–minibatch sampling , it is also a valid sampling vector ( ED [ vi ] = 1 ) [ Gower et al. , 2019 ] . See Gower et al . [ 2019 ] for other choices of sampling vectors v .

Summary: This paper makes multiple contributions to the convergence analysis of unconstrained stochastic variational inequality problem:   (1) This paper proposes the expected co-coercivity assumption for operators that is equivalent to expected smoothness for functions, and weaker than many prevalent assumptions for variational inequality problem such as bounded gradient, bounded variance and growth condition.  (2) This paper generalizes the unbiased stochastic gradient with mini-batch into unbiased stochastic gradient with continuous spectrum, and applies it to generalize the two prevalent algorithms for unconstrained stochastic variational inequality problem, namely, stochastic gradient descent ascent analysis (SGDA) and Consensus Optimization (CO). The latter is further generalized into Stochastic Consensus Optimization (SCO) by applying to variational inequality problem of finite-sum structure, which contains Combines and generalizes several related algorithms including SGDA, deterministic CO and stochastic Hamiltonian gradient descent (SHGD) as special cases.   (3) This paper gives the first last-iterate convergence analysis of this generalized SGDA that only relies on the expected co-coercivity assumption and the quasi-strongly monotone assumption that are weaker than the previous analysis of SGDA, and recovers the results of deterministic GDA. Effects of hyper-parameters are well studied including constant step-size, diminishing step-size and the optimal batch-size.  (4) This paper gives the first last-iterate convergence analysis of SCO that recovers the results of SGDA, deterministic CO and stochastic Hamiltonian gradient descent (SHGD), and the convergence is even faster than an existing analysis of deterministic CO.