Provably Faster Algorithms for Bilevel Optimization and Applications to Meta-Learning

1 INTRODUCTION . Bilevel optimization has received significant attention recently and become an influential framework in various machine learning applications including meta-learning ( Franceschi et al. , 2018 ; Bertinetto et al. , 2018 ; Rajeswaran et al. , 2019 ; Ji et al. , 2020a ) , hyperparameter optimization ( Franceschi et al. , 2018 ; Shaban et al. , 2019 ; Feurer & Hutter , 2019 ) , reinforcement learning ( Konda & Tsitsiklis , 2000 ; Hong et al. , 2020 ) , and signal processing ( Kunapuli et al. , 2008 ; Flamary et al. , 2014 ) . A general bilevel optimization takes the following formulation . min x∈Rp Φ ( x ) : = f ( x , y∗ ( x ) ) s.t . y∗ ( x ) = arg min y∈Rq g ( x , y ) , ( 1 ) where the upper- and inner-level functions f and g are both jointly continuously differentiable . The goal of eq . ( 1 ) is to minimize the objective function Φ ( x ) w.r.t . x , where y∗ ( x ) is obtained by solving the lower-level minimization problem . In this paper , we focus on the setting where the lower-level function g is strongly convex with respect to ( w.r.t . ) y , and the upper-level objective function Φ ( x ) is nonconvex w.r.t . x . Such types of geometrics commonly exist in many applications including meta-learning and hyperparameter optimization , where g corresponds to an empirical loss with a strongly-convex regularizer and x are parameters of neural networks . A broad collection of algorithms have been proposed to solve such types of bilevel optimization problems . For example , Hansen et al . ( 1992 ) ; Shi et al . ( 2005 ) ; Moore ( 2010 ) reformulated the bilevel problem in eq . ( 1 ) into a single-level constrained problem based on the optimality conditions of the lower-level problem . However , such type of methods often involve a large number of constraints , and are hard to implement in machine learning applications . Recently , more efficient gradient-based bilevel optimization algorithms have been proposed , which can be generally categorized into the approximate implicit differentiation ( AID ) based approach ( Domke , 2012 ; Pedregosa , 2016 ; Gould et al. , 2016 ; Liao et al. , 2018 ; Ghadimi & Wang , 2018 ; Grazzi et al. , 2020 ; Lorraine et al. , 2020 ) and the iterative differentiation ( ITD ) based approach ( Domke , 2012 ; Maclaurin et al. , 2015 ; Franceschi et al. , 2017 ; 2018 ; Shaban et al. , 2019 ; Grazzi et al. , 2020 ) . However , most of these studies have focused on the asymptotic convergence analysis , and the finite-time analysis ( that characterizes how fast an algorithm converges ) has not been well explored except a few attempts recently . Ghadimi & Wang ( 2018 ) provided the finite-time analysis for the ITD-based approach . Grazzi et al . ( 2020 ) provided the iteration complexity for the hypergradient computation via ITD and AID , but did not characterize the finite-time convergence for the entire execution of algorithms . • Thus , the first focus of this paper is to develop a comprehensive and enhanced theory , which covers a broader class of bilevel optimizers via ITD and AID based techniques , and more importantly , to improve the exiting analysis with a more practical parameter selection and order-level lower computational complexity . The stochastic bilevel optimization often occurs in applications where fresh data need to be sampled as the algorithms run ( e.g. , reinforcement learning ( Hong et al. , 2020 ) ) or the sample size of training data is large ( e.g. , hyperparameter optimization ( Franceschi et al. , 2018 ) , Stackelberg game ( Roth et al. , 2016 ) ) . Typically , the corresponding objective function is given by min x∈Rp Φ ( x ) = f ( x , y∗ ( x ) ) : = { Eξ [ F ( x , y∗ ( x ) ; ξ ) ] 1 n ∑n i=1 F ( x , y ∗ ( x ) ; ξi ) s.t . y∗ ( x ) = arg min y∈Rq g ( x , y ) : = { Eζ [ G ( x , y∗ ( x ) ; ζ ) ] 1 m ∑m i=1G ( x , y ∗ ( x ) ; ζi ) , ( 2 ) where f ( x , y ) and g ( x , y ) take either the expectation form w.r.t . the random variables ξ and ζ or the finite-sum form over given data Dn , m = { ξi , ζj , i = 1 , ... , n ; j = 1 , ... , m } often with large sizes n and m. During the optimization process , the algorithms sample data batch via the distributions of ξ and ζ or from the setDn , m . For such a stochastic setting , Ghadimi & Wang ( 2018 ) proposed a bilevel stochastic approximation ( BSA ) method via single-sample gradient and Hessian estimates . Based on such a method , Hong et al . ( 2020 ) further proposed a two-timescale stochastic approximation ( TTSA ) , and showed that TTSA achieves a better trade-off between the complexities of inner- and outer-loop optimization stages than BSA . • The second focus of this paper is to design a more sample-efficient algorithm for bilevel stochastic optimization , which achieves an order-level lower computational complexity over BSA and TTSA . 1.1 MAIN CONTRIBUTIONS . Our main contributions lie in developing enhanced theory and provably faster algorithms for the nonconvex-strongly-convex bilevel deterministic and stochastic optimization problems , respectively . Our analysis involves several new developments , which can be of independent interest . We first provide a unified finite-time convergence and complexity analysis for both ITD and AID based bilevel optimizers , which we call as ITD-BiO and AID-BiO . Compared to existing analysis in Ghadimi & Wang ( 2018 ) for AID-BiO that requires a continuously increasing number of innerloop steps to achieve the guarantee , our analysis allows a constant number of inner-loop steps as often used in practice . In addition , we introduce a warm start initialization for the inner-loop updates and the outer-loop hypergradient estimation , which allows us to backpropagate the tracking errors to previous loops , and results in an improved computational complexity . As shown in Table 1 , the gradient complexities Gc ( f , ) , Gc ( g , ) , and Jacobian- and Hessian-vector product complexities JV ( g , ) and HV ( g , ) of AID-BiO to attain an -accurate stationary point improve those of Ghadimi & Wang ( 2018 ) by the order of κ , κ −1/4 , κ , and κ , respectively , where κ is the condition number . In addition , our analysis shows that AID-BiO requires less computations of Jacobian- and Hessianvector products than ITD-BiO by an order of κ and κ1/2 , which provides a justification for the observation in Grazzi et al . ( 2020 ) that ITD often has a larger memory cost than AID . We then propose a stochastic bilevel optimizer ( stocBiO ) to solve the stochastic bilevel optimization problem in eq . ( 2 ) . Our algorithm features a mini-batch hyper-gradient estimation via implicit differentiation , where the core design involves a sample-efficient Hypergradient estimator via the Neumann series . As shown in Table 2 , the gradient complexities of our proposed algorithm w.r.t . F and G improve upon those of BSA ( Ghadimi & Wang , 2018 ) by an order of κ and −1 , respectively . In addition , the Jacobian-vector product complexity JV ( G , ) of our algorithm improves that of BSA by an order of κ . In terms of the target accuracy , our computational complexities improve those of TTSA ( Hong et al. , 2020 ) by an order of −1/2 . We further provide the theoretical complexity guarantee of ITD-BiO , AID-BiO and stocBiO in metalearning and hyperparameter optimization . The experiments validate our theoretical results for determinisitic bilevel optimization , and demonstrate the superior efficiency of stocBiO for stochastic bilevel optimization . Due to the space limitations , we present all theoretical and empirical results on hyperparameter optimization in the supplementary materials . 1.2 RELATED WORK . Bilevel optimization approaches : Bilevel optimization was first introduced by Bracken & McGill ( 1973 ) . Since then , a number of bilevel optimization algorithms have been proposed , which include but not limited to constraint-based methods ( Shi et al. , 2005 ; Moore , 2010 ) and gradient-based methods ( Domke , 2012 ; Pedregosa , 2016 ; Gould et al. , 2016 ; Maclaurin et al. , 2015 ; Franceschi et al. , 2018 ; Ghadimi & Wang , 2018 ; Liao et al. , 2018 ; Shaban et al. , 2019 ; Hong et al. , 2020 ; Liu et al. , 2020 ; Li et al. , 2020 ; Grazzi et al. , 2020 ; Lorraine et al. , 2020 ) . Among them , Ghadimi & Wang ( 2018 ) ; Hong et al . ( 2020 ) provided the finite-time complexity analysis for their proposed methods for the nonconvex-strongly-convex bilevel optimization problem . For such a problem , this paper develops a general and enhanced finite-time analysis for gradient-based bilevel optimizers for the deterministic setting , and proposes a novel algorithm for the stochastic setting with order-level lower computational complexity than the existing results . Some works have studied other types of loss geometries . For example , Liu et al . ( 2020 ) ; Li et al . ( 2020 ) assumed that the lower- and upper-level functions g ( x , · ) and f ( x , · ) are convex and stronglyconvex , and provided an asymptotic analysis for their methods . Ghadimi & Wang ( 2018 ) ; Hong et al . ( 2020 ) studied the setting where Φ ( · ) is strongly-convex or convex , and g ( x , · ) is strongly-convex . Bilevel optimization in meta-learning : Bilevel optimization framework has been successfully employed in meta-learning recently ( Snell et al. , 2017 ; Franceschi et al. , 2018 ; Rajeswaran et al. , 2019 ; Zügner & Günnemann , 2019 ; Ji et al. , 2020a ; b ) . For example , Snell et al . ( 2017 ) proposed a bilevel optimization procedure for meta-learning to learn a common embedding model for all tasks . Rajeswaran et al . ( 2019 ) reformulated the model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) as a bilevel optimization problem , and proposed iMAML via implicit gradient . The paper provides a theoretical guarantee for two popular types of bilevel optimization algorithms , i.e. , AID-BiO and ITD-BiO , for meta-learning . Bilevel optimization in hyperparameter optimization : Hyperparameter optimization has become increasingly important as a powerful tool in the automatic machine learning ( autoML ) ( Okuno et al. , Algorithm 1 Deterministic bilevel optimization via AID or ITD 1 : Input : Stepsizes α , β > 0 , initializations x0 , y0 , v0 . 2 : for k = 0 , 1 , 2 , ... , K do 3 : Set y0k = y T k−1 if k > 0 and y0 otherwise 4 : for t = 1 , .... , T do 5 : Update ytk = y t−1 k − α∇yg ( xk , y t−1 k ) 6 : end for 7 : Hypergradient estimation via • AID : 1 ) set v0k = vNk−1 if k > 0 and v0 otherwise 2 ) solve vNk from∇2yg ( xk , yTk ) v = ∇yf ( xk , yTk ) via N steps of CG starting from v0k 3 ) compute Jacobian-vector product∇x∇yg ( xk , yTk ) vNk via automatic differentiation 4 ) compute ∇̂Φ ( xk ) = ∇xf ( xk , yTk ) −∇x∇yg ( xk , yTk ) vNk • ITD : compute ∇̂Φ ( xk ) = ∂f ( xk , y T k ) xk via backpropagation w.r.t . xk 8 : Update xk+1 = xk − β ∂f ( xk , y T k ) ∂xk 9 : end for 2018 ; Yu & Zhu , 2020 ) . Recently , various bilevel optimization algorithms have been proposed in the context of hyperparameter optimization , which include implicit differentiation based methods ( Pedregosa , 2016 ) , dynamical system based methods via reverse or forward gradient computation ( Franceschi et al. , 2017 ; 2018 ; Shaban et al. , 2019 ) , etc . This paper demonstrates the superior efficiency of the proposed stocBiO algorithm in hyperparameter optimization .

Summary: The paper presents two algorithms - one for the deterministic and one for stochastic bilevel optimization. The paper claims the methods are lower cost in computational complexity for various terms and easy to implement. A finite-time convergence proof is provided for the algorithms.  Empirical results are presented for meta-learning, and (in the appendix) hyperparameter optimization.