Multi-Objective Meta Learning

1 Introduction . In the past few years , deep learning has achieved great success in various fields [ 43 ] because it can effectively and efficiently process massive and high-dimensional data . However , training a deep learning model from scratch often requires a large amount of labeled data to learn a large number of model parameters and needs to choose hyperparameters by hand . As a way to address those problems by enabling models to learn how to learn , meta learning has attracted considerable attention recently [ 19 , 20 ] . Meta learning gains knowledge from multiple meta training tasks so that the knowledge can be reused in new tasks or new environments rapidly with a few training examples . Taken broadly , objective functions of meta learning models are usually formulated as a bi-level optimization problem where the lower-level subproblem represents the adaptation to a given task with learned meta parameters and the upper-level subproblem tries to optimize these meta parameters via a meta objective [ 19 ] . Hence , from this view , meta learning has a wide range of applications such as hyperparameter optimization [ 13 ] , Neural Architecture Search ( NAS ) [ 29 ] , and Reinforcement Learning ( RL ) [ 68 ] . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In many studies on conventional meta learning methods and applications , there is only a single meta objective in the upper-level subproblem . For example , the Model-Agnostic Meta-Learning ( MAML ) method [ 12 ] only measures the performance on a validation dataset in the upper-level subproblem to evaluate the learned initialization of parameters . DARTS [ 29 ] , a differentiable method for NAS , evaluates the performance of the searched architecture on the validation dataset . However , in real-world applications , there are usually more than one objective to be considered . For example , for MAML , we may need to consider not only the performance but also the robustness which can help adapt to new tasks with the learned initialization . Similarly , the network size and performance should be balanced in NAS , especially when the searched architecture will be deployed to devices with limited resources such as mobile phones . In those applications , we can see that there is a need to balance multiple possibly conflicting objectives in meta learning . Meta learning with multiple objectives thus has drawn much attention in recent studies . Specifically , some works study specific meta learning problems in the multi-objective case , such as multi-objective NAS [ 66 , 57 , 1 , 34 ] , multi-objective RL [ 4 ] , and so on . However , those works either linearly combine multiple objectives into a single objective for the upper-level subproblem [ 65 , 63 , 11 ] or utilize multi-objective bi-level evolutionary algorithms [ 5 , 54 , 47 ] to handle it . The former approach needs to tune weights associated with all the objectives , which is time consuming , and its performance depends on the set of candidate weights in , for example , the cross validation method . The latter approach , whose computational complexity is even higher , has no convergence guarantee in the optimization process and is not easy to be integrated into gradient-based learning models such as deep neural networks , which limits its use in many learning models . To alleviate those limitations in existing works , in this paper we propose a unified gradient-based Multi-Objective Meta Learning ( MOML ) framework with a convergence guarantee . The MOML framework formulates objective functions in meta learning with multiple objectives as a MultiObjective Bi-Level optimization Problem ( MOBLP ) , where the lower-level subproblem is to learn the adaptation to a task similar to vanilla meta learning and the upper-level subproblem minimizes a vector-valued function corresponding to multiple objectives for the meta learner . To solve MOBLP , we devise the first gradient-based optimization algorithm by alternatively solving the lower-level and upper-level subproblems via a gradient descent method and a gradient-based multi-objective optimization method such as [ 7 ] , respectively . We theoretically prove the convergence properties of the proposed gradient-based optimization method . To show the effectiveness of the MOML framework , we apply it to several meta learning problems , including few-shot learning , domain adaptation , multi-task learning , and NAS , where multi-task learning is firstly formulated from the perspective of meta learning . In summary , the main contributions of this paper are four-fold . • We propose a unified MOML framework based on the MOBLP and devise a gradient-based optimization algorithm for the MOML framework . • We prove the convergence property of the proposed optimization algorithm . • We formulate several learning problems as instances of the MOML framework . • Experiments show that MOML achieves state-of-the-art performance on those learning problems . 2 Related Work . Meta Learning . Meta learning ( a.k.a . learning to learn ) learns knowledge from multiple tasks and then adapts it to new tasks with a few samples quickly . Many studies in meta learning mainly focus on solving the few-shot learning problem . From this view , meta learning can be divided into three main categories , including metric-based approach [ 55 , 56 ] , model-based approach [ 28 ] , and optimization-based approach [ 12 , 39 ] . For example , as an optimization-based method , MAML learns an initialization of model parameters so that a new task can be learned with a few training samples by fine-tuning the learned initialization . A widely-used formulation in meta learning can be cast as a bi-level optimization problem , where the upper-level subproblem is to learn meta parameters by minimizing a meta objective and the lower-level subproblem is to quickly adapt to new tasks with meta parameters [ 45 , 19 ] . For example , MAML adapts to a new task by using the associated training dataset and the learned initialization in the lower level , and then updates the initialization according to the validation performance in the upper level . From this perspective , meta learning is a general learning paradigm and has more general applications [ 19 ] . In this paper , we study meta learning from the perspective of the bi-level optimization . Multi-Objective Optimization . Multi-objective optimization is to address the problem of simultaneously minimizing multiple objectives , which may conflict with each other . The simplest way to handle multi-objective optimization is to convert to a single-objective optimization problem via the linear scalarization approach which minimizes the weighted sum of all the objectives . Actually , many machine learning algorithms adopt this approach . For example , minimizing the regularization term to control the model complexity and minimizing the training loss are two conflicting objectives , and a learning model is usually to minimize a linear combination of those two terms . Recently , many kinds of advanced multi-objective optimization algorithms are proposed , such as evolutionary algorithms [ 73 ] , population-based algorithms [ 15 ] , gradient-based algorithms [ 7 , 37 ] , and so on . Some of them have been successfully applied to solve machine learning problems [ 22 ] . In this work , we focus on gradient-based algorithms because this approach can be easily integrated into gradient-based machine learning models such as deep learning models . 3 The MOML Framework . In this paper , the proposed MOML framework has a unified objective function , which is formulated as a Multi-Objective Bi-Level optimization Problem ( MOBLP ) , as min α∈A , ω∈Rp F ( ω , α ) = ( F1 ( ω , α ) , F2 ( ω , α ) , ... , Fm ( ω , α ) ) T s.t . ω ∈ S ( α ) , ( 1 ) where function F : Rp × Rn → Rm is a vector-valued jointly continuous function for the m desired meta objectives and A is a nonempty compact subset of Rn . In problem ( 1 ) , S ( α ) is defined as the set of optimal solutions to minimize f ( ω , α ) w.r.t . ω , i.e. , S ( α ) = arg min ω f ( ω , α ) . ( 2 ) When m equals 1 , problem ( 1 ) reduces to a standard Bi-Level optimization Problem ( BLP ) , which is a widely-used formulation in meta learning [ 19 ] , and hence from this perspective , the MOML framework is a generalization of meta learning . In problems ( 1 ) and ( 2 ) , F is called the Upper-Level ( UL ) subproblem and f : Rp × Rn → R is the Lower-Level ( LL ) subproblem . The LL subproblem can be considered as a constraint to the UL subproblem . In MOML , F contains multiple meta objectives to be achieved for the meta learner and f defines the objective function for current task such as the training loss . In Section 5 , we will see the application of MOML in different learning problems , including few-shot learning , domain adaptation , multi-task learning , and neural architecture search . To solve problem ( 1 ) , there exist a type of works [ 5 , 54 , 47 ] which adopt multi-objective evolutionary algorithms . However , such methods have a high complexity without convergence guarantee and are not easy to be integrated with gradient-based models such as deep neural networks . Hence , we do not include them in experiments . To the best of our knowledge , there is no gradient-based optimization algorithm with convergence guarantee to solve an MOBLP , which is what we will do in the next section . 4 Optimization . In this section , we devise a general algorithm to solve the MOBLP ( i.e. , problem ( 1 ) ) and provide convergence analyses under certain assumptions . 4.1 Lower-Level Singleton Condition . Due to the complicated dependency between UL and LL variables , solving the MOBLP is challenging , especially when optimal solutions of the LL subproblem are not unique . For a BLP with a single objective in the UL subproblem , many studies [ 8 , 13 , 52 ] potentially require that the LL subproblem only admits a unique minimizer for every α ∈ A . This condition can simplify both the optimization process and convergence analyses and is first fomally introduced as Lower-Level Singleton ( LLS ) condition in [ 30 ] . For the MOBLP , the LLS condition is necessary . If the LLS condition does not hold , the MOBLP is even ill-defined [ 9 ] . To see this , suppose for a fixed α0 , we get a set of solutions S ( α0 ) for the LL subproblem . Since F is vector-valued in problem ( 1 ) , it is unclear that at which ω ∈ S ( α0 ) the UL subproblem F should be evaluated . With the LLS condition , problem ( 1 ) can be simplified as min α∈A ϕ ( α ) = F ( ω∗ ( α ) , α ) s.t . ω∗ ( α ) = arg min ω f ( ω , α ) . ( 3 ) 4.2 Gradient-based Optimization Algorithm . Here we design a gradient-based optimization algorithm to solve problem ( 3 ) . Usually , there is no closed form for the solution ω∗ ( α ) of the LL subproblem and so it is difficult to optimize the UL subproblem directly . Another approach is to use the optimality condition of the LL subproblem ( i.e. , ∇ωf ( ω , α ) = 0 ) as equality constraints for the UL subproblem in a way similar to [ 41 ] . However , this approach only works for LL subproblems with simple forms and can not work for general learning models . To solve problem ( 3 ) , we take a strategy similar to the alternating optimization . In the first part of each iteration ( corresponding to steps 3-6 in Algorithm 1 ) , we solve the LL subproblem via gradient descent methods . Specifically , with an initialization ω0 for the LL variable where the index t of the iteration is omitted for notation simplicity , the solution of the LL subproblem can be updated for K steps as ωk+1 ( α ) = Tk ( ωk ( α ) , α ) , k = 1 , . . . , K − 1 , where Tk represents an operator to update ω . Here we consider a first-order gradient descent method for Tk such as the Stochastic Gradient Descent ( SGD ) method and Tk can be formulated explicitly as Tk ( ωk ( α ) , α ) = ωk ( α ) − µ∇ωf ( ωk ( α ) , α ) , where µ > 0 denotes the step size and ∇ωf ( ωk ( α ) , α ) denotes the derivative of f w.r.t . ω at ω = ωk ( α ) . In the second part of each iteration ( corresponding to steps 7-9 in Algorithm 1 ) , by fixing the value of ω as the current solution ωK ( α ) obtained in the first part , we solve the UL subproblem as min α ϕK ( α ) = F ( ωK ( α ) , α ) . ( 4 ) Algorithm 1 Optimization Algorithm for MOML Input : numbers of iterations ( T , K ) , step size ( µ , ν ) 1 : Randomly initialized α0 ; 2 : for t = 1 to T do 3 : Initialize ωt0 ( αt ) ; 4 : for j = 1 to K do 5 : ωtj ( αt ) ← ωtj−1 ( αt ) − µ∇ωf ( ωtj−1 ( αt ) , αt ) ; 6 : end for 7 : Compute gradients∇αFi ( ωtK ( αt ) , αt ) for all the i ’ s ; 8 : Compute the gradient d ( ωtK ( αt ) , αt ) = MOPSolver ( { ∇αFi ( ωtK ( αt ) , αt ) } ) ; 9 : αt+1 = αt − νd ( ωtK ( αt ) , αt ) with a step size ν ; 10 : end for Problem ( 4 ) is an unconstrained Multi-Objective optimization Problem ( MOP ) and we can use any multiobjective optimization algorithms to solve it . Here we choose gradientbased multi-objective optimization methods as they can be seamlessly integrated into any gradient-based learning framework . There are several gradientbased multi-objective optimization algorithms [ 44 , 7 , 58 ] and they commonly find an appropriate descent direction d for all the objectives in F by aggregating their gradients w.r.t . α . So such process is denoted by d = MOPSolver ( { ∇αFi ( ωK ( α ) , α ) } mi=1 ) in Algorithm 1 . In this paper , we adopt a simple gradient-based MOP method called Multiple Gradient Descent Algorithm ( MGDA ) [ 7 ] , whose details are introduced in Appendix A.2 due to page limit . The entire algorithm to solve problem ( 3 ) is shown in Algorithm 1 , which to the best of our knowledge is the first gradient-based optimization algorithm for MOBLPs . In Algorithm 1 , we obtain only one solution for MOBLPs , which is different from multi-objective evolutionary algorithms that can find a population of solutions . How to obtain multiple nearly optimal solutions for MOBLPs is beyond the scope of this paper and we will study it in our future work .

Summary: This work leverages multi-objective optimization to augment meta learning algorithms. In particular, the inner optimization proceeds as normal; while the outer optimization uses Desideri's multiple-gradient descent algorithm to dynamically scale the per-task inner gradients. The paper concludes with a theoretical proof of convergence and empirical analysis on several applications of meta learning.