Provably Faster Algorithms for Bilevel Optimization

1 Introduction . Bilevel optimization has become a timely and important topic recently due to its great effectiveness in a wide range of applications including hyperparameter optimization [ 7 , 5 ] , meta-learning [ 33 , 16 , 1 ] , reinforcement learning [ 14 , 24 ] . Bilevel optimization can be generally formulated as the following minimization problem : min x∈Rp Φ ( x ) : = f ( x , y∗ ( x ) ) s.t . y∗ ( x ) = arg min y∈Rq g ( x , y ) . ( 1 ) Since the outer function Φ ( x ) : = f ( x , y∗ ( x ) ) depends on the variable x also via the optimizer y∗ ( x ) of the inner-loop function g ( x , y ) , the algorithm design for bilevel optimization is much more complicated and challenging than minimization and minimax optimization . For example , if the gradient-based approach is applied , then the gradient of the outer-loop function ( also called hypergradient ) will necessarily involve Jacobian and Hessian matrices of the inner-loop function g ( x , y ) , which require more careful design to avoid high computational complexity . This paper focuses on the nonconvex-strongly-convex setting , where the outer function f ( x , y∗ ( x ) ) is nonconvex with respect to ( w.r.t . ) x and the inner function g ( x , y ) is strongly convex w.r.t . y for any x . Such a case often occurs in practical applications . For example , in hyperparameter optimization [ 7 ] , f ( x , y∗ ( x ) ) is often nonconvex with x representing neural network hyperparameters , but the inner function g ( x , · ) can be strongly convex w.r.t . y by including a strongly-convex regularizer on y . In few-shot meta-learning [ 1 ] , the inner function g ( x , · ) often takes a quadratic form together with a strongly-convex regularizer . To efficiently solve the deterministic problem in eq . ( 1 ) , various bilevel optimization algorithms have been proposed , which include two popular classes of deterministic gradient-based methods respectively based on approximate implicit differentiation ( AID ) [ 31 , 9 , 8 ] and iterative differentiation ( ITD ) [ 28 , 6 , 7 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Recently , stochastic bilevel opitimizers [ 8 , 20 ] have been proposed , in order to achieve better efficiency than deterministic methods for large-scale scenarios where the data size is large or vast fresh data needs to be sampled as the algorithm runs . In particular , such a class of problems adopt functions by : Φ ( x ) : = f ( x , y∗ ( x ) ) : = Eξ [ F ( x , y∗ ( x ) ; ξ ) ] , g ( x , y ) : = Eζ [ G ( x , y ; ζ ) ] where the outer and inner functions take the expected values w.r.t . samples ξ and ζ , respectively . Along this direction , [ 20 ] proposed a stochastic gradient descent ( SGD ) type optimizer ( stocBiO ) , and showed that stocBiO attains a computational complexity of Õ ( −2 ) in order to reach an - accurate stationary point . More recently , several studies [ 2 , 11 , 22 ] have tried to accelerate SGD-type bilevel optimizers via momentum-based techniques , e.g. , by introducing a momentum ( historical information ) term into the gradient estimation . All of these optimizers follow a single-loop design , i.e. , updating x and y simultaneously . Specifically , [ 22 ] proposed an algorithm MSTSA by updating x via a momentum-based recursive technique introduced by [ 3 , 36 ] . [ 11 ] proposed an optimizer SEMA similarly to MSTSA but using the momentum recursive technique for updating both x and y . [ 2 ] proposed an algorithm STABLE , which applies the momentum strategy for updating the Hessian matrix , but the algorithm involves expensive Hessian inverse computation rather than hypergradient approximation loop . However , as shown in Table 1 , SEMA , MSTSA and STABLE achieve the same complexity order of Õ ( −2 ) as the SGD-type stocBiO algorithm , where the momentum technique in these algorithms does not exhibit the theoretical advantage . Such a comparison is not consistent with those in minimization [ 3 ] and minimax optimization [ 15 ] , where the single-loop momentum-based recursive technique achieves provable performance improvements over SGD-type methods . This motivates the following natural but important question : • Can we design a faster single-loop momentum-based recursive bilevel optimizer , which achieves order-wisely lower computational complexity than SGD-type stocBiO ( and all other momentumbased algorithms ) , and is also easy to implement with efficient matrix-vector products ? Although the existing theoretical efforts on accelerating bilevel optimization algorithms have been exclusively focused on single-loop design1 , empirical studies in [ 20 ] suggested that double-loop bilevel algorithms such as BSA [ 8 ] and stocBiO [ 20 ] achieve much better performances than single-loop algorithms such as TTSA [ 14 ] . A good candidate suitable for accelerating double-loop algorithms can be the popular variance reduction method , such as SVRG [ 21 ] , SARAH [ 30 ] and SPIDER [ 4 ] , which typically yield provably lower complexity . The basic idea is to construct low-variance gradient estimators using periodic high-accurate large-batch gradient evaluations . So far , there has not been any study on using variance reduction to accelerate double-loop bilevel optimization algorithms . This motivates the second question that we address in this paper : • Can we develop a double-loop variance-reduced bilevel optimizer with improved computational complexity over SGD-type stocBiO ( and all other existing algorithms ) ? If so , when such a double-loop algorithm holds advantage over the single-loop algorithms in bilevel optimization ? 1.1 Main Contributions . This paper proposes two algorithms for bilevel optimization , both outperforming all existing algorithms in terms of complexity order . We first propose a single-loop momentum-based recursive bilevel optimizer ( MRBO ) . MRBO updates variables x and y simultaneously , and uses the momentum recursive technique for constructing lowvariance mini-batch estimators for both the gradient ∇g ( x , · ) and the hypergradient ∇Φ ( · ) ; in contrast to previous momentum-based algorithms that accelerate only one gradient or neither . Further , MRBO is easy to implement , and allows efficient computations of Jacobian- and Hessian-vector products via automatic differentiation . Theoretically , we show that MRBO achieves a computational complexity ( w.r.t . computations of gradient , Jacobian- and Hessian-vector product ) of Õ ( −1.5 ) , 1In the literature of bilevel optimization , although many hypergradient-based algorithms include an iteration loop of Hessian inverse estimation , such a loop is typically not counted when these algorithms are classified by the number of loops . This paper follows such a convention to be consistent with the existing literature . Namely , the single- and double-loop algorithms mentioned here can include an additional loop of Hessian inverse estimation in the hypergradient approximation . which outperforms all existing algorithms by an order of −0.5 . Technically , our analysis needs to first characterize the estimation property for the momentum-based recursive estimator for the Hessian-vector type hypergradient and then uses such a property to further bound the per-iteration error due to momentum updates for both inner and outer loops . We then propose a double-loop variance-reduced bilevel optimizer ( VRBO ) , which is the first algorithm that adopts the recursive variance reduction for bilevel optimization . In VRBO , each inner loop constructs a variance-reduced gradient ( w.r.t . y ) and hypergradient ( w.r.t . x ) estimators through the use of large-batch gradient estimations computed periodically at each outer loop . Similarly to MRBO , VRBO involves the computations of Jacobian- and Hessian-vector products rather than Hessians or Hessian inverse . Theoretically , we show that VRBO achieves the same near-optimal complexity of Õ ( −1.5 ) as MRBO and outperforms all existing algorithms . Technically , differently from the use of variance reduction in minimization and minimax optimization , our analysis for VRBO needs to characterize the variance reduction property for the Hessian-vector type of hypergradient estimators , which only involves Hessian vector computation rather than Hessian . Such estimator introduces additional errors to handle in the telescoping and convergence analysis . Our experiments2 show that VRBO achieves the highest accuracy among all comparison algorithms , and MRBO converges fastest among its same type of single-loop momentum-based algorithms . In particular , we find that our double-loop VRBO algorithm converges much faster than other singlrloop algorithms including our MRBO , which is in contrast to the existing efforts exclusively on accelerating the single-loop algorithms [ 2 , 11 , 22 ] . Such a result also differs from those phenomenons observed in minimization and minimax optimization , where single-loop algorithms often outperform double-loop algorithms . 1.2 Related Works . Bilevel optimization approaches : At the early stage of bilevel optimization studies , a class of constraint-based algorithms [ 13 , 35 , 29 ] were proposed , which tried to penalize the outer function with the optimality conditions of the inner problem . To further simplify the implementation of constraint-based bilevel methods , gradient-based bilevel algorithms were then proposed , which include but not limited to AID-based [ 33 , 7 , 34 , 17 ] , ITD-based [ 9 , 31 , 8 , 16 , 19 ] methods , and stochastic bilevel optimizers such as BSA [ 8 ] , stocBiO [ 20 ] , and TTSA [ 14 ] . The finite-time ( i.e. , non-asymptotic ) convergence analysis for bilevel optimization has been recently studied in several works [ 8 , 20 , 14 ] . In this paper , we propose two novel stochastic bilevel algorithms using momentum recursive and variance reduction techniques , and show that they order-wise improve the computational complexity over existing stochastic bilevel optimizers . Momentum-based recursive approaches : The momentum recursive technique was first introduced by [ 3 , 36 ] for minimization problems , and has been shown to achieve improved computational complexity over SGD-based updates in theory and in practice . Several works [ 22 , 2 , 11 ] applied the 2Our codes are available online at https : //github.com/JunjieYang97/MRVRBO similar single-loop momentum-based strategy to bilevel optimization to accelerate the SGD-based bilevel algorithms such as BSA [ 8 ] and stocBiO [ 20 ] . However , the computational complexities of these momentum-based algorithms are not shown to outperform that of stocBiO . In this paper , we propose a new single-loop momentum-based recursive bilevel optimizer ( MRBO ) , which we show achieves order-wisely lower complexity than existing stochastic bilevel optimizers . Variance reduction approaches : Variance reduction has been studied extensively for conventional minimization problems , and many algorithms have been designed along this line , including but not limited to SVRG [ 21 , 26 ] , SARAH [ 30 ] , SPIDER [ 4 ] , SpiderBoost [ 37 , 38 , 18 ] and SNVRG [ 41 ] . Several works [ 27 , 39 , 40 , 32 ] recently employed such techniques for minimax optimization to achieve better complexities . In this paper , we propose the first-known variance reduction-based bilevel optimizer ( VRBO ) , which achieves a near-optimal computational complexity and outperforms existing stochastic bilevel algorithms . Two concurrent works : As we were finalizing this submission , two concurrent studies were posted on arXiv recently ( [ 23 ] was posted on May 8 and [ 12 ] was posted on May 5 ) . Both studies overlap only with our MRBO algorithm , nothing similar to our VRBO . Specifically , [ 23 ] and [ 12 ] respectively proposed the SUSTAIN and RSVRB algorithms for bilevel optimization , both using momentumbased design as our MRBO . Although SUSTAIN and RSVRB have been shown to achieve the same theoretical complexity of O ( −1.5 ) as our MRBO ( and VRBO ) , both algorithms have major drawbacks in their design , so that their empirical performance ( as we demonstrate in our experiments ) is much worse that our MRBO ( and even worse than our VRBO ) . SUSTAIN adopts only singlesample for each update ( whereas MRBO uses minibatch for stability ) ; and RSVRB requires to compute Hessian inverse at each iteration ( whereas MRBO uses Hessian-vector products for fast computation ) . As an additional note , our experiments demonstrate that our VRBO significantly outperforms all these single-loop momentum-based algorithms SUSTAIN and RSVRB as well as our MRBO .

Summary: This paper presents two algorithms (one single-loop and one double-loop) which exploit variance reduction techniques to improve the best known sample complexity for stochastic bilevel optimization problems with strongly convex lower-level from $O(\epsilon^{-2})$ to  $O(\epsilon^{-1.5})$.    Bilevel optimization consists in two nested optimization problems: the upper-level aims at minimizing an objective function where some of the variables are a solution of the lower-level problem. In the stochastic setting, upper and lower-level objectives are not accessible and are instead replaced by unbiased estimators. In addition to several smoothness assumptions, this work and several recent others consider the case when the lower-level problem is strongly-convex, which guarantees that the lower-level solution is unique, the existence of the upper-level gradient, and allows to provide convergence guarantees to stationary points even when the lower-level solution is not computed exactly.   This work studies two algorithms (MRBO and VRBO) which both achieve a sample complexity of $O(\epsilon^{-1.5})$, which improves upon the previous optimal complexity of $O(\epsilon^{-2})$. Both algorithms exploit variance reduction  to achieve the desired complexity. MRBO is single loop and combines variance reduction and momentum while VRBO is double loop and uses large-batch gradients to reduce the variance. Recent works have already applied momentum and variance reduction to solve stochastic bilevel problems but this one is the first (together with two concurrent works) to achieve a sample complexity this low. The authors present experiments on the data hyperclearner setting which show that VRBO performs the best and double-loop algorithms seem to outperform single-loop ones.