Multi-Objective Meta Learning

1 Introduction . In the past few years , deep learning has achieved great success in various fields [ 43 ] because it can effectively and efficiently process massive and high-dimensional data . However , training a deep learning model from scratch often requires a large amount of labeled data to learn a large number of model parameters and needs to choose hyperparameters by hand . As a way to address those problems by enabling models to learn how to learn , meta learning has attracted considerable attention recently [ 19 , 20 ] . Meta learning gains knowledge from multiple meta training tasks so that the knowledge can be reused in new tasks or new environments rapidly with a few training examples . Taken broadly , objective functions of meta learning models are usually formulated as a bi-level optimization problem where the lower-level subproblem represents the adaptation to a given task with learned meta parameters and the upper-level subproblem tries to optimize these meta parameters via a meta objective [ 19 ] . Hence , from this view , meta learning has a wide range of applications such as hyperparameter optimization [ 13 ] , Neural Architecture Search ( NAS ) [ 29 ] , and Reinforcement Learning ( RL ) [ 68 ] . ∗Equal contribution . †Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In many studies on conventional meta learning methods and applications , there is only a single meta objective in the upper-level subproblem . For example , the Model-Agnostic Meta-Learning ( MAML ) method [ 12 ] only measures the performance on a validation dataset in the upper-level subproblem to evaluate the learned initialization of parameters . DARTS [ 29 ] , a differentiable method for NAS , evaluates the performance of the searched architecture on the validation dataset . However , in real-world applications , there are usually more than one objective to be considered . For example , for MAML , we may need to consider not only the performance but also the robustness which can help adapt to new tasks with the learned initialization . Similarly , the network size and performance should be balanced in NAS , especially when the searched architecture will be deployed to devices with limited resources such as mobile phones . In those applications , we can see that there is a need to balance multiple possibly conflicting objectives in meta learning . Meta learning with multiple objectives thus has drawn much attention in recent studies . Specifically , some works study specific meta learning problems in the multi-objective case , such as multi-objective NAS [ 66 , 57 , 1 , 34 ] , multi-objective RL [ 4 ] , and so on . However , those works either linearly combine multiple objectives into a single objective for the upper-level subproblem [ 65 , 63 , 11 ] or utilize multi-objective bi-level evolutionary algorithms [ 5 , 54 , 47 ] to handle it . The former approach needs to tune weights associated with all the objectives , which is time consuming , and its performance depends on the set of candidate weights in , for example , the cross validation method . The latter approach , whose computational complexity is even higher , has no convergence guarantee in the optimization process and is not easy to be integrated into gradient-based learning models such as deep neural networks , which limits its use in many learning models . To alleviate those limitations in existing works , in this paper we propose a unified gradient-based Multi-Objective Meta Learning ( MOML ) framework with a convergence guarantee . The MOML framework formulates objective functions in meta learning with multiple objectives as a MultiObjective Bi-Level optimization Problem ( MOBLP ) , where the lower-level subproblem is to learn the adaptation to a task similar to vanilla meta learning and the upper-level subproblem minimizes a vector-valued function corresponding to multiple objectives for the meta learner . To solve MOBLP , we devise the first gradient-based optimization algorithm by alternatively solving the lower-level and upper-level subproblems via a gradient descent method and a gradient-based multi-objective optimization method such as [ 7 ] , respectively . We theoretically prove the convergence properties of the proposed gradient-based optimization method . To show the effectiveness of the MOML framework , we apply it to several meta learning problems , including few-shot learning , domain adaptation , multi-task learning , and NAS , where multi-task learning is firstly formulated from the perspective of meta learning . In summary , the main contributions of this paper are four-fold . • We propose a unified MOML framework based on the MOBLP and devise a gradient-based optimization algorithm for the MOML framework . • We prove the convergence property of the proposed optimization algorithm . • We formulate several learning problems as instances of the MOML framework . • Experiments show that MOML achieves state-of-the-art performance on those learning problems . 2 Related Work . Meta Learning . Meta learning ( a.k.a . learning to learn ) learns knowledge from multiple tasks and then adapts it to new tasks with a few samples quickly . Many studies in meta learning mainly focus on solving the few-shot learning problem . From this view , meta learning can be divided into three main categories , including metric-based approach [ 55 , 56 ] , model-based approach [ 28 ] , and optimization-based approach [ 12 , 39 ] . For example , as an optimization-based method , MAML learns an initialization of model parameters so that a new task can be learned with a few training samples by fine-tuning the learned initialization . A widely-used formulation in meta learning can be cast as a bi-level optimization problem , where the upper-level subproblem is to learn meta parameters by minimizing a meta objective and the lower-level subproblem is to quickly adapt to new tasks with meta parameters [ 45 , 19 ] . For example , MAML adapts to a new task by using the associated training dataset and the learned initialization in the lower level , and then updates the initialization according to the validation performance in the upper level . From this perspective , meta learning is a general learning paradigm and has more general applications [ 19 ] . In this paper , we study meta learning from the perspective of the bi-level optimization . Multi-Objective Optimization . Multi-objective optimization is to address the problem of simultaneously minimizing multiple objectives , which may conflict with each other . The simplest way to handle multi-objective optimization is to convert to a single-objective optimization problem via the linear scalarization approach which minimizes the weighted sum of all the objectives . Actually , many machine learning algorithms adopt this approach . For example , minimizing the regularization term to control the model complexity and minimizing the training loss are two conflicting objectives , and a learning model is usually to minimize a linear combination of those two terms . Recently , many kinds of advanced multi-objective optimization algorithms are proposed , such as evolutionary algorithms [ 73 ] , population-based algorithms [ 15 ] , gradient-based algorithms [ 7 , 37 ] , and so on . Some of them have been successfully applied to solve machine learning problems [ 22 ] . In this work , we focus on gradient-based algorithms because this approach can be easily integrated into gradient-based machine learning models such as deep learning models . 3 The MOML Framework . In this paper , the proposed MOML framework has a unified objective function , which is formulated as a Multi-Objective Bi-Level optimization Problem ( MOBLP ) , as min α∈A , ω∈Rp F ( ω , α ) = ( F1 ( ω , α ) , F2 ( ω , α ) , ... , Fm ( ω , α ) ) T s.t . ω ∈ S ( α ) , ( 1 ) where function F : Rp × Rn → Rm is a vector-valued jointly continuous function for the m desired meta objectives and A is a nonempty compact subset of Rn . In problem ( 1 ) , S ( α ) is defined as the set of optimal solutions to minimize f ( ω , α ) w.r.t . ω , i.e. , S ( α ) = arg min ω f ( ω , α ) . ( 2 ) When m equals 1 , problem ( 1 ) reduces to a standard Bi-Level optimization Problem ( BLP ) , which is a widely-used formulation in meta learning [ 19 ] , and hence from this perspective , the MOML framework is a generalization of meta learning . In problems ( 1 ) and ( 2 ) , F is called the Upper-Level ( UL ) subproblem and f : Rp × Rn → R is the Lower-Level ( LL ) subproblem . The LL subproblem can be considered as a constraint to the UL subproblem . In MOML , F contains multiple meta objectives to be achieved for the meta learner and f defines the objective function for current task such as the training loss . In Section 5 , we will see the application of MOML in different learning problems , including few-shot learning , domain adaptation , multi-task learning , and neural architecture search . To solve problem ( 1 ) , there exist a type of works [ 5 , 54 , 47 ] which adopt multi-objective evolutionary algorithms . However , such methods have a high complexity without convergence guarantee and are not easy to be integrated with gradient-based models such as deep neural networks . Hence , we do not include them in experiments . To the best of our knowledge , there is no gradient-based optimization algorithm with convergence guarantee to solve an MOBLP , which is what we will do in the next section . 4 Optimization . In this section , we devise a general algorithm to solve the MOBLP ( i.e. , problem ( 1 ) ) and provide convergence analyses under certain assumptions . 4.1 Lower-Level Singleton Condition . Due to the complicated dependency between UL and LL variables , solving the MOBLP is challenging , especially when optimal solutions of the LL subproblem are not unique . For a BLP with a single objective in the UL subproblem , many studies [ 8 , 13 , 52 ] potentially require that the LL subproblem only admits a unique minimizer for every α ∈ A . This condition can simplify both the optimization process and convergence analyses and is first fomally introduced as Lower-Level Singleton ( LLS ) condition in [ 30 ] . For the MOBLP , the LLS condition is necessary . If the LLS condition does not hold , the MOBLP is even ill-defined [ 9 ] . To see this , suppose for a fixed α0 , we get a set of solutions S ( α0 ) for the LL subproblem . Since F is vector-valued in problem ( 1 ) , it is unclear that at which ω ∈ S ( α0 ) the UL subproblem F should be evaluated . With the LLS condition , problem ( 1 ) can be simplified as min α∈A ϕ ( α ) = F ( ω∗ ( α ) , α ) s.t . ω∗ ( α ) = arg min ω f ( ω , α ) . ( 3 ) 4.2 Gradient-based Optimization Algorithm . Here we design a gradient-based optimization algorithm to solve problem ( 3 ) . Usually , there is no closed form for the solution ω∗ ( α ) of the LL subproblem and so it is difficult to optimize the UL subproblem directly . Another approach is to use the optimality condition of the LL subproblem ( i.e. , ∇ωf ( ω , α ) = 0 ) as equality constraints for the UL subproblem in a way similar to [ 41 ] . However , this approach only works for LL subproblems with simple forms and can not work for general learning models . To solve problem ( 3 ) , we take a strategy similar to the alternating optimization . In the first part of each iteration ( corresponding to steps 3-6 in Algorithm 1 ) , we solve the LL subproblem via gradient descent methods . Specifically , with an initialization ω0 for the LL variable where the index t of the iteration is omitted for notation simplicity , the solution of the LL subproblem can be updated for K steps as ωk+1 ( α ) = Tk ( ωk ( α ) , α ) , k = 1 , . . . , K − 1 , where Tk represents an operator to update ω . Here we consider a first-order gradient descent method for Tk such as the Stochastic Gradient Descent ( SGD ) method and Tk can be formulated explicitly as Tk ( ωk ( α ) , α ) = ωk ( α ) − µ∇ωf ( ωk ( α ) , α ) , where µ > 0 denotes the step size and ∇ωf ( ωk ( α ) , α ) denotes the derivative of f w.r.t . ω at ω = ωk ( α ) . In the second part of each iteration ( corresponding to steps 7-9 in Algorithm 1 ) , by fixing the value of ω as the current solution ωK ( α ) obtained in the first part , we solve the UL subproblem as min α ϕK ( α ) = F ( ωK ( α ) , α ) . ( 4 ) Algorithm 1 Optimization Algorithm for MOML Input : numbers of iterations ( T , K ) , step size ( µ , ν ) 1 : Randomly initialized α0 ; 2 : for t = 1 to T do 3 : Initialize ωt0 ( αt ) ; 4 : for j = 1 to K do 5 : ωtj ( αt ) ← ωtj−1 ( αt ) − µ∇ωf ( ωtj−1 ( αt ) , αt ) ; 6 : end for 7 : Compute gradients∇αFi ( ωtK ( αt ) , αt ) for all the i ’ s ; 8 : Compute the gradient d ( ωtK ( αt ) , αt ) = MOPSolver ( { ∇αFi ( ωtK ( αt ) , αt ) } ) ; 9 : αt+1 = αt − νd ( ωtK ( αt ) , αt ) with a step size ν ; 10 : end for Problem ( 4 ) is an unconstrained Multi-Objective optimization Problem ( MOP ) and we can use any multiobjective optimization algorithms to solve it . Here we choose gradientbased multi-objective optimization methods as they can be seamlessly integrated into any gradient-based learning framework . There are several gradientbased multi-objective optimization algorithms [ 44 , 7 , 58 ] and they commonly find an appropriate descent direction d for all the objectives in F by aggregating their gradients w.r.t . α . So such process is denoted by d = MOPSolver ( { ∇αFi ( ωK ( α ) , α ) } mi=1 ) in Algorithm 1 . In this paper , we adopt a simple gradient-based MOP method called Multiple Gradient Descent Algorithm ( MGDA ) [ 7 ] , whose details are introduced in Appendix A.2 due to page limit . The entire algorithm to solve problem ( 3 ) is shown in Algorithm 1 , which to the best of our knowledge is the first gradient-based optimization algorithm for MOBLPs . In Algorithm 1 , we obtain only one solution for MOBLPs , which is different from multi-objective evolutionary algorithms that can find a population of solutions . How to obtain multiple nearly optimal solutions for MOBLPs is beyond the scope of this paper and we will study it in our future work .

Summary: The authors consider and propose a multiple objectives perspective for Meta-Learning. More specifically, they formulate the problem as a bilevel-optimization problem in which the lower-level problem is the within-task problem and the upper-level problem is a vector-valued problem aiming at minimizing a multi-objective meta-learning problem. Existing state-of-the-art methods for multiple objectives meta-learning mainly combine multiple objectives into a single objective in a weighted sum manner and they tune the weights associated with all the objectives, which could be time consuming. Others, use new evolutionary algorithms not relying on convergence guarantees and not easily applicable to deep learning methods. Differently from those works, the authors propose a gradient-based Multi-Objective-Meta-Learning (MOML) gradient-based method not requiring tuning weights. Their method alternatively solves the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Under specific assumptions, the authors guarantees an asymptotic convergence result for their method, assuming to be able to compute the meta-subgradients of the meta-objective. Finally, they show the advantage of applying their multi-objective meta-learning method in comparison to state-of-the-art methods on problems of few-shot learning, semi-supervised domain adaptation, multi-task learning, and neural architecture search.