Enhanced First and Zeroth Order Variance Reduced Algorithms for Min-Max Optimization

1 INTRODUCTION . Min-max optimization has attracted significant growth of attention in machine learning as it captures several important machine learning models and problems including generative adversarial networks ( GANs ) Goodfellow et al . ( 2014 ) , robust adversarial machine learning Madry et al . ( 2018 ) , imitation learning Ho & Ermon ( 2016 ) , etc . Min-max optimization typically takes the following form min x∈Rd1 max y∈Rd2 f ( x , y ) , where f ( x , y ) , { E [ F ( x , y ; ξ ) ] ( online case ) 1 n ∑n i=1 F ( x , y ; ξi ) ( finite-sum case ) ( 1 ) where f ( x , y ) takes the expectation form if data samples ξ are taken in an online fashion , and f ( x , y ) takes the finite-sum form if a dataset of training samples ξi for i = 1 , . . . , n are given in advance . This paper focuses on the nonconvex-strongly-concave min-max problem , in which f ( x , y ) is nonconvex with respect to x for all y ∈ Rd2 , and f ( x , y ) is µ-strongly concave with respect to y for all x ∈ Rd1 . The problem then takes the following equivalent form : min x∈Rd1 { Φ ( x ) , max y∈Rd2 f ( x , y ) } . ( 2 ) The objective function Φ ( · ) in eq . ( 2 ) is nonconvex in general , and hence algorithms for solving eq . ( 2 ) are expected to attain an approximate ( i.e. , -accurate ) first-order stationary point . The convergence of deterministic algorithms for solving eq . ( 2 ) has been established in Jin et al . ( 2019 ) ; Nouiehed et al . ( 2019 ) ; Thekumparampil et al . ( 2019 ) ; Lu et al . ( 2020 ) . SGD-type of stochastic algorithms have also been proposed to solve such a problem more efficiently , including SGDmax Jin et al . ( 2019 ) , PGSMD Rafique et al . ( 2018 ) , and SGDA Lin et al . ( 2019 ) , which respectively achieve the overall complexity of O ( κ3 −4 log ( 1/ ) ) 1 , O ( κ3 −4 ) , and O ( κ3 −4 ) . Furthermore , several variance reduction methods have been proposed for solving eq . ( 2 ) for the nonconvex-strongly-concave case . PGSVRG Rafique et al . ( 2018 ) adopts a proximally guided SVRG method and achieves the overall complexity of O ( κ3 −4 ) for the online case and O ( κ2n −2 ) for the finite-sum case . Wai et al . ( 2019 ) converted the value function evaluation problem to a specific minmax problem and applied SAGA to achieve the overall complexity of O ( κn −2 ) for the finite-sum case . More recently , Luo et al . ( 2020 ) proposed a novel nested-loop algorithm named Stochastic Recursive Gradient Descent Ascent ( SREDA ) , which adopts SARAH/SPIDER-type Nguyen et al . ( 2017a ) ; Fang et al . ( 2018 ) of recursive variance reduction method ( originally designed for solving the minimization problem ) for designing gradient estimators to update both x and y . Specifically , x takes the normalized gradient update in the outer-loop and each update of x is followed by an entire innerloop updates of y. Luo et al . ( 2020 ) showed that SREDA achieves an overall complexity ofO ( κ3 −3 ) for the online case in eq . ( 1 ) , which attains the optimal dependence on Arjevani et al . ( 2019 ) . For the finite-sum case , SREDA achieves the complexity of O ( κ2 √ n −2 + n+ ( n+ k ) log ( κ/ ) ) for n ≥ κ2 , and O ( ( κ2 + κn ) −2 ) for n ≤ κ2 . Although SREDA achieves the optimal complexity performance in theory , two issues may substantially degrade its practice performance . ( 1 ) SREDA has a stringent requirement on the initialization accuracy ζ = κ−2 2 , which hence requires O ( κ2 −2 log ( κ/ ) ) gradient estimations in the initialization , and is rather costly in the high accuracy regime ( i.e. , for small ) . ( 2 ) The convergence of SREDA requires the stepsize to be substantially small , i.e. , at the -level with αt = O ( min { / ( κ ` ‖vt‖2 ) , 1/ ( κ ` ) } ) , which restricts each iteration to make only -level progress with ‖xt+1 − xt‖2 = O ( /κ ` ) . Consequently , SREDA can run very slowly in practice . • Thus , a vital question arising here is whether we can guarantee the same optimal complexity performance of SREDA even if we considerably relax its initialization ( i.e. , much bigger than O ( 2 ) ) and enlarge its stepsize ( i.e. , much bigger than O ( ) ) . The answer is highly nontrivial because the original analysis framework for SREDA in Luo et al . ( 2020 ) critically relies on these restrictions . The first focus of this paper is on developing a novel analytical framework to guarantee that such enhanced SREDA continues to hold the SREDA ’ s optimal complexity performance . Furthermore , in many machine learning scenarios , min-max optimization problems need to be solved without the access of the gradient information , but only the function values , e.g. , in multi-agent reinforcement learning with bandit feedback Wei et al . ( 2017 ) ; Zhang et al . ( 2019 ) and robotics Wang & Jegelka ( 2017 ) ; Bogunovic et al . ( 2018 ) . This motivates the design of zeroth-order ( i.e. , gradient-free ) algorithms . For nonconvex-strongly-concave min-max optimization , Liu et al . ( 2019 ) studied a constrained problem and proposed ZO-min-max algorithm that achieves the computational complexity of O ( ( d1 + d2 ) −6 ) . Wang et al . ( 2020 ) designed ZO-SGDA and ZO-SGDMSA , where ZO-SGDMA achieves the best known query complexity of O ( ( d1 + d2 ) κ2 −4 log ( 1/ ) ) among the zeroth-order algorithms for this problem . All of the above studies are of SGD-type , and no efforts have been made on developing variance reduction zeroth-order algorithms for nonconvex-strongly-concave min-max optimization to further improve the query complexity . • The second focus of this paper is on applying the aforementioned enhanced SREDA algorithm to design a zeroth-order variance reduced algorithm for nonconvex-strongly-concave min-max problems , and further characterizing its complexity guarantee which we anticipate to be orderwisely better than that of the existing stochastic algorithms . 1.1 MAIN CONTRIBUTIONS . This paper first studies an enhanced SREDA algorithm , called SREDA-Boost , which improves SREDA from two aspects . ( 1 ) For the initialization , SREDA-Boost requires only an accuracy of ζ = κ−1 , which is much less stringent than that of ζ = κ−2 2 required by SREDA . ( 2 ) SREDABoost allows an accuracy-independent stepsize α = O ( 1/ ( κ ` ) ) , which is much larger than the -level stepsize α = O ( min { / ( κ ` ‖vt‖2 ) , 1/ ( κ ` ) } ) adopted by SREDA . Hence , SREDA-Boost can run much faster than SREDA . 1The constant κ = ` /µ , where µ is the strong concavity parameter of f ( x , · ) , and ` is the Lipschitz constant of the gradient of f ( x , y ) as defined in Assumption 2 . Typically , κ is much larger than one . The first contribution of this paper lies in developing a new analysis technique to provide the computational complexity guarantee for SREDA-Boost , establishing that even with considerably relaxed conditions on the initialization and stepsize , SREDA-Boost achieves the same optimal complexity performance as SREDA . The analysis technique of SREDA in Luo et al . ( 2020 ) does not handle such a case , because the proof highly relies on the stringent initialization and stepsize requirements . Central to our new analysis framework is the novel approach for bounding two interconnected stochastic error processes : tracking error and gradient estimation error ( see Section 4 for their formal definitions ) , which take three steps : bounding the two error processes accumulatively over the entire algorithm execution , decoupling these two inter-related stochastic error processes , and establishing each of their relationships with the accumulative gradient estimators . The second contribution of this paper lies in proposing the zeroth-order variance reduced algorithm ZO-SREDA-Boost for nonconvex-strongly-conconve min-max optimization when the gradient information is not accessible . For the online case , we show that ZO-SREDA-Boost achieves an overall query complexity of O ( ( d1 + d2 ) κ3 −3 ) , which outperforms the best known complexity ( achieved by ZO-SGDMSA Wang et al . ( 2020 ) ) in the case with ≤ κ−1 . For the finite-sum case , we show that ZO-SREDA-Boost achieves an overall query complexity of O ( ( d1 +d2 ) ( κ2 √ n −2 +n ) +d2 ( κ 2 +κn ) log ( κ ) ) when n ≥ κ2 , andO ( ( d1 +d2 ) ( κ2 +κn ) κ −2 ) when n ≤ κ2 . This is the first study that applies the variance reduction method to zeroth-order nonconvex-stronlgy-concave min-max optimization . 1.2 RELATED WORK . Due to the vast amount of studies on min-max optimization and the variance reduced algorithms , we include below only the studies that are highly relevant to this work . Variance reduction methods for min-max optimization are highly inspired by those for conventional minimization problems , including SAGA Defazio et al . ( 2014 ) ; Reddi et al . ( 2016 ) , SVRG Johnson & Zhang ( 2013 ) ; Allen-Zhu & Hazan ( 2016 ) ; Allen-Zhu ( 2017 ) , SARAH Nguyen et al . ( 2017a ; b ; 2018 ) , SPIDER Fang et al . ( 2018 ) , SpiderBoost Wang et al . ( 2019 ) , etc . But the convergence analysis for min-max optimization is much more challenging , and is typically quite different from their counterparts in minimization problems . For strongly-convex-strongly-concave min-max optimization , Palaniappan & Bach ( 2016 ) applied SVRG and SAGA to the finite-sum case and established a linear convergence rate , and Chavdarova et al . ( 2019 ) proposed SVRE later to obtain a better bound . When the condition number of the problem is very large , Luo et al . ( 2019 ) proposed a proximal point iteration algorithm to improve the performance of SAGA . For some special cases , Du et al . ( 2017 ) ; Du & Hu ( 2019 ) showed that the linear convergence rate of SVRG can be maintained without the strongly-convex or strongly concave assumption . Yang et al . ( 2020 ) applied SVRG to study the min-max optimization under the two-sided Polyak-Lojasiewicz condition . Nonconvex-strongly-concave min-max optimization is the focus of this paper . As we discuss at the beginning of the introduction , the SGD-type algorithms have been developed and studied , including SGDmax Jin et al . ( 2019 ) , PGSMD Rafique et al . ( 2018 ) , and SGDA Lin et al . ( 2019 ) . Several variance reduction methods have also been proposed to further improve the performance , including PGSVRG Rafique et al . ( 2018 ) , the SAGA-type algorithm for min-max optimization Wai et al . ( 2019 ) , and SREDA Luo et al . ( 2020 ) . Particularly , SREDA has been shown in Luo et al . ( 2020 ) to achieve the optimal complexity dependence on . This paper further provides the convergence guarantee for SREDA-Boost ( which enhances SREDA with relaxed initialization and much larger stepsize ) by developing a new analysis technique . While SGD-type zeroth-order algorithms have been studied for min-max optimization , such as Menickelly & Wild ( 2020 ) ; Roy et al . ( 2019 ) for convex-concave min-max problems and Liu et al . ( 2019 ) ; Wang et al . ( 2020 ) for nonconvex-strongly-concave min-max problems , variance reduced algorithms have not been developed for zeroth-order min-max optimization so far . This paper proposes the first such an algorithm named ZO-SREDA-Boost for nonconvex-strongly-concave min-max optimization , and established its complexity performance that outperforms the existing comparable algorithms ( see Table 1 ) .

Summary: The paper proposes a SREDA-Boost, which builds upon SREDA for nonconvex-strongly-concave minimax problem. The SREDA-Boost algorithm is less restrictive to initialization and has an accuracy-independent and larger step size. Thus it can run substantially faster than SREDA. The main contribution to the first-order optimization story is a new analytical framework that builds upon the previous analysis in SREDA and overcomes the dependence of highly accurate initialization via bounding the tracking error and gradient estimation error separately. It also proposes a zeroth-order variance reduction algorithm for the same optimization problem, which has the largest possible step size so far and also improves the complexity of the state-of-the-art in some cases. Various experiments have validated the superiority. The theoretical analysis and empirical results look good to me.