Multi-Objective Meta Learning
Year: 2022

Abstract:
Meta learning with multiple objectives has been attracted much attention recently since many applications need to consider multiple factors when designing learning models. Existing gradient-based works on meta learning with multiple objectives mainly combine multiple objectives into a single objective in a weighted sum manner. This simple strategy usually works but it requires to tune the weights associated with all the objectives, which could be time consuming. Different from those works, in this paper, we propose a gradient-based Multi-Objective Meta Learning (MOML) framework without manually tuning weights. Specifically, MOML formulates the objective function of meta learning with multiple objectives as a Multi-Objective Bi-Level optimization Problem (MOBLP) where the upper-level subproblem is to solve several possibly conflicting objectives for the meta learner. To solve the MOBLP, we devise the first gradient-based optimization algorithm by alternatively solving the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Theoretically, we prove the convergence properties of the proposed gradient-based optimization algorithm. Empirically, we show the effectiveness of the proposed MOML framework in several meta learning problems, including few-shot learning, domain adaptation, multi-task learning, and neural architecture search. The source code of MOML is available at https://github.com/Baijiong-Lin/MOML.

heading: 1 Introduction
text: In the past few years, deep learning has achieved great success in various fields [43] because it can effectively and efficiently process massive and high-dimensional data. However, training a deep learning model from scratch often requires a large amount of labeled data to learn a large number of model parameters and needs to choose hyperparameters by hand.
As a way to address those problems by enabling models to learn how to learn, meta learning has attracted considerable attention recently [19, 20]. Meta learning gains knowledge from multiple meta training tasks so that the knowledge can be reused in new tasks or new environments rapidly with a few training examples. Taken broadly, objective functions of meta learning models are usually formulated as a bi-level optimization problem where the lower-level subproblem represents the adaptation to a given task with learned meta parameters and the upper-level subproblem tries to optimize these meta parameters via a meta objective [19]. Hence, from this view, meta learning has a wide range of applications such as hyperparameter optimization [13], Neural Architecture Search (NAS) [29], and Reinforcement Learning (RL) [68]. ∗Equal contribution. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In many studies on conventional meta learning methods and applications, there is only a single meta objective in the upper-level subproblem. For example, the Model-Agnostic Meta-Learning (MAML) method [12] only measures the performance on a validation dataset in the upper-level subproblem to evaluate the learned initialization of parameters. DARTS [29], a differentiable method for NAS, evaluates the performance of the searched architecture on the validation dataset. However, in real-world applications, there are usually more than one objective to be considered. For example, for MAML, we may need to consider not only the performance but also the robustness which can help adapt to new tasks with the learned initialization. Similarly, the network size and performance should be balanced in NAS, especially when the searched architecture will be deployed to devices with limited resources such as mobile phones. In those applications, we can see that there is a need to balance multiple possibly conflicting objectives in meta learning.
Meta learning with multiple objectives thus has drawn much attention in recent studies. Specifically, some works study specific meta learning problems in the multi-objective case, such as multi-objective NAS [66, 57, 1, 34], multi-objective RL [4], and so on. However, those works either linearly combine multiple objectives into a single objective for the upper-level subproblem [65, 63, 11] or utilize multi-objective bi-level evolutionary algorithms [5, 54, 47] to handle it. The former approach needs to tune weights associated with all the objectives, which is time consuming, and its performance depends on the set of candidate weights in, for example, the cross validation method. The latter approach, whose computational complexity is even higher, has no convergence guarantee in the optimization process and is not easy to be integrated into gradient-based learning models such as deep neural networks, which limits its use in many learning models.
To alleviate those limitations in existing works, in this paper we propose a unified gradient-based Multi-Objective Meta Learning (MOML) framework with a convergence guarantee. The MOML framework formulates objective functions in meta learning with multiple objectives as a MultiObjective Bi-Level optimization Problem (MOBLP), where the lower-level subproblem is to learn the adaptation to a task similar to vanilla meta learning and the upper-level subproblem minimizes a vector-valued function corresponding to multiple objectives for the meta learner. To solve MOBLP, we devise the first gradient-based optimization algorithm by alternatively solving the lower-level and upper-level subproblems via a gradient descent method and a gradient-based multi-objective optimization method such as [7], respectively. We theoretically prove the convergence properties of the proposed gradient-based optimization method. To show the effectiveness of the MOML framework, we apply it to several meta learning problems, including few-shot learning, domain adaptation, multi-task learning, and NAS, where multi-task learning is firstly formulated from the perspective of meta learning. In summary, the main contributions of this paper are four-fold.
• We propose a unified MOML framework based on the MOBLP and devise a gradient-based optimization algorithm for the MOML framework.
• We prove the convergence property of the proposed optimization algorithm.
• We formulate several learning problems as instances of the MOML framework.
• Experiments show that MOML achieves state-of-the-art performance on those learning problems.

heading: 2 Related Work
text: Meta Learning. Meta learning (a.k.a. learning to learn) learns knowledge from multiple tasks and then adapts it to new tasks with a few samples quickly. Many studies in meta learning mainly focus on solving the few-shot learning problem. From this view, meta learning can be divided into three main categories, including metric-based approach [55, 56], model-based approach [28], and optimization-based approach [12, 39]. For example, as an optimization-based method, MAML learns an initialization of model parameters so that a new task can be learned with a few training samples by fine-tuning the learned initialization. A widely-used formulation in meta learning can be cast as a bi-level optimization problem, where the upper-level subproblem is to learn meta parameters by minimizing a meta objective and the lower-level subproblem is to quickly adapt to new tasks with meta parameters [45, 19]. For example, MAML adapts to a new task by using the associated training dataset and the learned initialization in the lower level, and then updates the initialization according to the validation performance in the upper level. From this perspective, meta learning is a general
learning paradigm and has more general applications [19]. In this paper, we study meta learning from the perspective of the bi-level optimization.
Multi-Objective Optimization. Multi-objective optimization is to address the problem of simultaneously minimizing multiple objectives, which may conflict with each other. The simplest way to handle multi-objective optimization is to convert to a single-objective optimization problem via the linear scalarization approach which minimizes the weighted sum of all the objectives. Actually, many machine learning algorithms adopt this approach. For example, minimizing the regularization term to control the model complexity and minimizing the training loss are two conflicting objectives, and a learning model is usually to minimize a linear combination of those two terms. Recently, many kinds of advanced multi-objective optimization algorithms are proposed, such as evolutionary algorithms [73], population-based algorithms [15], gradient-based algorithms [7, 37], and so on. Some of them have been successfully applied to solve machine learning problems [22]. In this work, we focus on gradient-based algorithms because this approach can be easily integrated into gradient-based machine learning models such as deep learning models.

heading: 3 The MOML Framework
text: In this paper, the proposed MOML framework has a unified objective function, which is formulated as a Multi-Objective Bi-Level optimization Problem (MOBLP), as
min α∈A,ω∈Rp
F (ω, α) = (F1(ω, α), F2(ω, α), ..., Fm(ω, α)) T s.t. ω ∈ S(α), (1)
where function F : Rp × Rn → Rm is a vector-valued jointly continuous function for the m desired meta objectives and A is a nonempty compact subset of Rn. In problem (1), S(α) is defined as the set of optimal solutions to minimize f(ω, α) w.r.t. ω, i.e.,
S(α) = arg min ω f(ω, α). (2)
When m equals 1, problem (1) reduces to a standard Bi-Level optimization Problem (BLP), which is a widely-used formulation in meta learning [19], and hence from this perspective, the MOML framework is a generalization of meta learning. In problems (1) and (2), F is called the Upper-Level (UL) subproblem and f : Rp × Rn → R is the Lower-Level (LL) subproblem. The LL subproblem can be considered as a constraint to the UL subproblem. In MOML, F contains multiple meta objectives to be achieved for the meta learner and f defines the objective function for current task such as the training loss. In Section 5, we will see the application of MOML in different learning problems, including few-shot learning, domain adaptation, multi-task learning, and neural architecture search.
To solve problem (1), there exist a type of works [5, 54, 47] which adopt multi-objective evolutionary algorithms. However, such methods have a high complexity without convergence guarantee and are not easy to be integrated with gradient-based models such as deep neural networks. Hence, we do not include them in experiments. To the best of our knowledge, there is no gradient-based optimization algorithm with convergence guarantee to solve an MOBLP, which is what we will do in the next section.

heading: 4 Optimization
text: In this section, we devise a general algorithm to solve the MOBLP (i.e., problem (1)) and provide convergence analyses under certain assumptions.

heading: 4.1 Lower-Level Singleton Condition
text: Due to the complicated dependency between UL and LL variables, solving the MOBLP is challenging, especially when optimal solutions of the LL subproblem are not unique.
For a BLP with a single objective in the UL subproblem, many studies [8, 13, 52] potentially require that the LL subproblem only admits a unique minimizer for every α ∈ A. This condition can simplify both the optimization process and convergence analyses and is first fomally introduced as Lower-Level Singleton (LLS) condition in [30].
For the MOBLP, the LLS condition is necessary. If the LLS condition does not hold, the MOBLP is even ill-defined [9]. To see this, suppose for a fixed α0, we get a set of solutions S(α0) for the LL subproblem. Since F is vector-valued in problem (1), it is unclear that at which ω ∈ S(α0) the UL subproblem F should be evaluated. With the LLS condition, problem (1) can be simplified as
min α∈A ϕ(α) = F (ω∗(α), α) s.t. ω∗(α) = arg min ω f(ω, α). (3)

heading: 4.2 Gradient-based Optimization Algorithm
text: Here we design a gradient-based optimization algorithm to solve problem (3). Usually, there is no closed form for the solution ω∗(α) of the LL subproblem and so it is difficult to optimize the UL subproblem directly. Another approach is to use the optimality condition of the LL subproblem (i.e., ∇ωf(ω, α) = 0) as equality constraints for the UL subproblem in a way similar to [41]. However, this approach only works for LL subproblems with simple forms and cannot work for general learning models.
To solve problem (3), we take a strategy similar to the alternating optimization. In the first part of each iteration (corresponding to steps 3-6 in Algorithm 1), we solve the LL subproblem via gradient descent methods. Specifically, with an initialization ω0 for the LL variable where the index t of the iteration is omitted for notation simplicity, the solution of the LL subproblem can be updated for K steps as ωk+1(α) = Tk(ωk(α), α), k = 1, . . . ,K − 1, where Tk represents an operator to update ω. Here we consider a first-order gradient descent method for Tk such as the Stochastic Gradient Descent (SGD) method and Tk can be formulated explicitly as Tk(ωk(α), α) = ωk(α)− µ∇ωf(ωk(α), α), where µ > 0 denotes the step size and ∇ωf(ωk(α), α) denotes the derivative of f w.r.t. ω at ω = ωk(α). In the second part of each iteration (corresponding to steps 7-9 in Algorithm 1), by fixing the value of ω as the current solution ωK(α) obtained in the first part, we solve the UL subproblem as
min α ϕK(α) = F (ωK(α), α). (4)
Algorithm 1 Optimization Algorithm for MOML Input: numbers of iterations (T , K), step size (µ, ν) 1: Randomly initialized α0; 2: for t = 1 to T do 3: Initialize ωt0(αt); 4: for j = 1 to K do 5: ωtj(αt)← ωtj−1(αt)− µ∇ωf(ωtj−1(αt), αt); 6: end for 7: Compute gradients∇αFi(ωtK(αt), αt) for all the i’s; 8: Compute the gradient
d(ωtK(αt), αt) = MOPSolver({∇αFi(ωtK(αt), αt)}); 9: αt+1 = αt − νd(ωtK(αt), αt) with a step size ν;
10: end for Problem (4) is an unconstrained Multi-Objective optimization Problem (MOP) and we can use any multiobjective optimization algorithms to solve it. Here we choose gradientbased multi-objective optimization methods as they can be seamlessly integrated into any gradient-based learning framework. There are several gradientbased multi-objective optimization algorithms [44, 7, 58] and they commonly find an appropriate descent direction d for all the objectives in F by aggregating their gradients w.r.t. α. So such process is denoted by d = MOPSolver({∇αFi(ωK(α), α)}mi=1) in Algorithm 1. In this paper, we adopt a simple gradient-based MOP method called Multiple Gradient Descent Algorithm (MGDA) [7], whose details are introduced in Appendix A.2 due to page limit.
The entire algorithm to solve problem (3) is shown in Algorithm 1, which to the best of our knowledge is the first gradient-based optimization algorithm for MOBLPs. In Algorithm 1, we obtain only one solution for MOBLPs, which is different from multi-objective evolutionary algorithms that can find a population of solutions. How to obtain multiple nearly optimal solutions for MOBLPs is beyond the scope of this paper and we will study it in our future work.

heading: 4.3 Convergence Analysis
text: Algorithm 1 is simple and intuitive, but its convergence properties are unknown, which is what we aim to study in this section. As an MOBLP, problem (3) cannot reduce to SOML with a scalar-valued objective function in the upper-level subproblem when using gradient-based Algorithm 1 to solve it. Therefore, it has different theoretical properties from SOML as we need to focus on the convergence properties of a minimal point set instead of a minimum scalar in the UL subproblem.
We first recall some notions about vector-valued functions. Consider a vector-valued function g(z) : Rn → Rm (m,n ∈ N,m ≥ 2). We denote by Min g(z) the set of all the minimal points of function g(z). Min g(z) is also called the Pareto frontier or Pareto-optimal set. The corresponding efficient solution or Pareto-optimal solution set of g(z) is denoted by Eff(g(z)). The convexity of vector-valued functions is called the P-convex. The details of these definitions can be found in Appendix A.
To help analyze convergence properties of Algorithm 1, with a basic assumption about the LLS condition which is mentioned in Section 4.1 and is widely adopted in BLPs [52, 10], we can obtain the following result. Theorem 1. If f(ω, α) is jointly continuous, arg minω f(ω, α) is a singleton for every α ∈ A, and ω∗(α) is uniformly bounded on A, then the function F (ω∗(α), α) is continuous w.r.t. α.
BecauseA is a compact set, Theorem 1 implies the existence of solutions. Theorem 1 and the uniform convergence of ωK(α) can further imply the convergence for the solution of the LL subproblem, which is similar to that of the standard BLP problem [13].
For the convergence of Algorithm 1, we need to analyze minimal point sets of the images of perturbed functions ϕK(α) and ϕ(α) which are defined in problems (4) and (3), respectively. We consider the most natural set convergence under this setting, i.e., the Kuratowski-Painlevé set-convergence. Due to page limit, please refer to these definitions in Appendix A. Under certain assumptions which are used in analyses of BLP and MOP [13, 35], we have the following convergence results. Theorem 2. In addition to Theorem 1, it is assumed that (i) F (·, α) is uniformly Lipschitz continuous; (ii) The iterative sequence {ωk(α)}Kk=1 converges uniformly to ω∗(α) on A as K → +∞; (iii) A is a convex set; (iv) ϕK is P-convex and ϕ is strictly P-convex. Then, the Kuratowski-Painlevé set-convergence of both the minimal point set and efficient solution set in Algorithm 1 holds, i.e.,
Min ϕK(α)→ Min ϕ(α), Eff ϕK(α)→ Eff ϕ(α).
Thus, under some specific assumptions, Theorem 2 provides a theoretical justification for the convergence of Algorithm 1 to solve MOBLPs.

heading: 5 Applications of MOML
text: In this section, we introduce several use cases of the MOML framework in several learning problems, including few-shot learning, semi-supervised domain adaptation, multi-task learning, and neural architecture search.

heading: 5.1 Few-Shot Learning
text: Few-Shot Learning (FSL) aims to tackle the problem of training a model with only a few training samples [64]. Recently, FSL is widely studied from the perspective of meta learning by using prior knowledge in the meta training process. Most studies in FSL only consider the classification performance. However, in real-world applications, the performance is not the only important focus. For example, we expect FSL models to not only have good performance but also be robust to adversarial attacks [26], which may improve the generalization of FSL models. In the following, we can see that this setting can naturally be modeled by the proposed MOML framework.
Problem Formulation. Suppose there is a base dataset Dbase with a category set Cbase and a novel dataset Dnovel with a category set Cnovel, where Cbase ∩ Cnovel = ∅. The goal of FSL is to adapt the knowledge learned from Dbase to help the learning of Dnovel. In the ith meta training episode, we generate from Dbase an N -way k-shot classification task, which consists of a support set Ds(i)base and a query set Dq(i)base. For the robustness, we add perturbations generated by the Projected Gradient Descent (PGD) method [26] into each data point in Dq(i)base to generate a perturbed query set D q(i),adv base . The objective function of the FSL model that considers both the performance and the robustness can be formulated as
min α
( LF (ω∗(i)(α), α,Dq(i)base),LF (ω ∗(i)(α), α,Dq(i),advbase ) )
s.t. ω∗(i)(α) = arg min ω LF (ω, α,Ds(i)base), (5)
where ω represents model parameters, α denotes the meta parameters to encode common knowledge that can be transferred to novel tasks, and LF (ω, α,D) denotes the average classification loss of a model with model parameters α and meta parameters ω on a dataset D. In the UL subproblem of problem (5), the first objective measures the classification loss on the query set based on ω∗(i)(α) obtained by solving the LL subproblem and the second objective measures the robustness via the classification performance on the perturbed query set. Problem (5) provides a general formulation, which depends on what α represents, for FSL. We adapt problem (5) to three FSL methods (i.e., MAML [12], ProtoNet [55] and BOIL [40]) and the detailed formulations are put in Appendix C. It is easy to see that problem (5) fits the MOML framework and we can use Algorithm 1 to solve it.
For an MOBLP such as problem (1), a common approach is to transform the UL subproblem into a single objective problem via the linear scalarization approach [65, 63, 11]. In contrast with MOML, this approach is named as Single-Objective Meta Learning (SOML) in this paper and used as an extra baseline method in our experiments. Specifically, with fixed weights {γi}mi=1, which satisfy∑m i=1 γi = 1 and γi ≥ 0, problem (1) is transformed to a standard bi-level optimization problem as
minα,ω∈S(α) ∑m i=1 γiFi(ω, α). This approach heavily depends on the selection of weights {γi}mi=1 and inappropriate weights usually lead to poor performance.
Experiments. Experiments are conducted on two FSL benchmark datasets, mini-ImageNet [61] and CUB-200-2011 (referred to as CUB) [62]. Due to page limit, experimental settings and experimental results on the CUB dataset are put in Appendix C. The original baseline methods (i.e., MAML, ProtoNet, and BOIL) are trained in a conventional way and we evaluate the clean accuracy and PGD accuracy of all the methods, where the clean accuracy is measured on the original test query set and the PGD accuracy is tested on the query set perturbed by the PGD attack. As the clean accuracy and the PGD accuracy are reported to be conflicting to each other [70], inspired by the widelyused F1-score based on the precision and recall, we propose a new metric called Balance score (B-score), which is defined as B-score = 2× (CA× PA)/(CA + PA) with CA and PA denoting the clean accuracy and PGD accuracy, respectively, to
fully measure the performance in terms of the clean accuracy and PGD accuracy.
The average results over 600 testing tasks on the mini-ImageNet dataset are presented in Table 1. From the results, we can see that the adversarial robustness of the original FSL methods (i.e., MAML, ProtoNet, and BOIL) is poor, while SOML and the proposed MOML can significantly improve the PGD accuracy. For example, MOML can improve the PGD accuracy of ProtoNet by about 40.91% under the 5-shot setting. Since the classification accuracy and adversarial robustness are conflicting [70], the clean accuracy of SOML and MOML slightly drops when compared with the orignal FSL baselines. The B-score of MOML is higher than SOML, which indicates the multi-objective formulation in the UL subproblem of MOML is better than the single-objective formulation in SOML.

heading: 5.2 Semi-Supervised Domain Adaptation
text: Semi-Supervised Domain Adaptation (SSDA) aims to address the domain shift between two domains so that the model trained in a label-rich source domain can be adapted to a target domain with limited labeled samples and abundant unlabeled samples [69]. A widely used approach for SSDA is to align the distributions of two domains by finding some domain-invariant components. There are usually three objectives to be considered, including two classification losses on two domains and an alignment loss to measure the domain discrepancy. While existing works such as [49, 74] optimize
all the objectives by simply computing a weighted sum of them, we formulate the SSDA problem as an MOBLP under the MOML framework.
Problem Formulation. Given a source domain S and a target domain T , the source domain has a large labeled dataset DS and the target domain has a limited labeled dataset DlT as well as a large unlabeled dataset DuT , where DT = DlT ⋃ DuT denotes the entire dataset for the target domain. The average classification losses in the source and target domains are represented by LD(ω,DS) and LD(ω,DlT ), respectively, where ω is the model parameter. The alignment loss denoted by LA(ω, α,DS ,DuT ) aims to learn domain-invariant components such as a domain-invariant projection space by minimizing the local maximum mean discrepancy in DSAN [74] or domain-invariant prototypes by maximizing the entropy in MME [49], where α is the initialization of ω. Then we can formulate the SSDA problem under the MOML framework as
min α
(LD(ω∗(α),DS),LD(ω∗(α),DlT ),LA(ω, α,DS ,DuT ))
s.t. ω∗(α) = arg min ω LA(ω, α,DS ,DuT ). (6)
In the LL subproblem of problem (6), we aim to learn a model to find a domain-invariant component between two domains via the alignment loss LA by optimizing ω with an initialization α, and in the UL subproblem, we expect to improve the model further by updating α via minimizing the two classification losses together with the alignment loss. Here α acts similar to the parameter initialization in MAML (i.e., α in problem (5)) and helps learn ω in the LL subproblem, but it does not require any adaptation on the testing process. We can follow two state-of-the-art domain adaptation models (i.e., DSAN and MME) to design the alignment loss and due to page limit, the detailed formulation of the alignment loss LA is put in Appendix D. Although a bi-level objective function is also formulated in Meta-MME [27], there exist two significant differences between Meta-MME and the proposed MOML method. Firstly, Meta-MME aims to learn the initialization of the network parameters by minimizing the source classification loss and the alignment loss in its LL subproblem and then validate it on a few target labeled samples in the UL subproblem, which is different from the proposed MOML formulation (i.e., problem (6)). Secondly, the proposed MOML method considers a multi-objective optimization problem in the UL subproblem, which is different from Meta-MME. Empirically, MOML outperforms the Meta-MME method as shown in Table 2.
Experiments. Experiments are conducted on the Office-31 dataset [48], which has 3 domains: Amazon (A), Webcam (W) and DSLR (D). By following [59, 33], we construct all six transfer
tasks. Each class in the target domain has three labeled images in the training process by following [49]. Baseline models in comparison include a deep neural network (denoted by ‘S+T’) that is trained on DS ⋃ DlT and other eight state-of-the-arts domain adaptation methods: ENT [16], DANN [14], ADR [50], CDAN [32], MME [49], Meta-MME [27], APE [23] and DSAN [74]. As DSAN and MME are two state-of-the-art domain adaptation models, to improve their performance further, SOML and MOML adopt the alignment loss in DSAN and MME, respectively, leading to two groups of models, i.e., (DSAN, DSAN+SOML, DSAN+MOML) and (MME, Meta-MME, MME+SOML, MME+MOML). All the methods except S+T are trained on DS ⋃ DT . Due to page limit, details of baselines and experimental settings are put in Appendix D.
Experimental results on the Office-31 dataset are shown in Table 2. The incorporation of SOML into DSAN and MME can consistently improve the performance of two baseline models and the proposed MOML method further improves the classification accuracy. For example, MME+MOML significantly improves the performance of the first five tasks and achieves the perfect performance (i.e., the 100% accuracy) on the last task as most baselines did. Moreover, compared with state-ofthe-art baselines, DSAN+MOML achieves the best result (i.e., 94.59%) on transfer task A→W and MME+MOML has the best accuracy (i.e., 78.30% and 77.71%) on transfer tasks D→A and W→A. Besides, MME+MOML achieves the best average classification accuracy of 90.53% and significantly outperforms all of the baselines, which indicates the effectiveness of the proposed MOML method.

heading: 5.3 Multi-Task Learning
text: Multi-Task Learning (MTL) [2, 72] aims to improve the performance of multiple tasks simultaneously by leveraging useful information contained in these tasks. Learning the loss weighting is a challenge in MTL and there are some works [51, 31, 71] to solve this problem. Among those works, Sener and Koltun [51] formulate multi-task learning problems from the perspective of multi-objective optimization and implicitly learn the task weights via MGDA, Liu et al. [31] estimate the task weight of each task as the ratio of the training losses in the last two iterations for the corresponding task, and Yu et al. [71] project each task’s gradient onto the normal plane of the other. Different from those works which are all based on single-level optimization problems on the entire training set, for the first time we formulate this problem as an MOBLP based on the split of the entire training dataset and solve this problem based on the MOML framework.
Problem Formulation. Suppose there are m tasks. The ith task has a dataset Di for model training. Here each Di is partitioned into two subsets: the training dataset Dtri and the validation dataset Dvali , where Dtri is used to train a multi-task model and Dvali is to measure the performance of the multi-task model on the ith task. f(·;ω), the learning function of the multi-task model parameterized by ω, receives data points from the m tasks and outputs predictions. Let αi ∈ [0, 1] denotes the loss weight for the ith task. The goal is to jointly learn the loss weights α = (α1, . . . , αm)T and the model parameter ω. The objective function of the proposed method under the MOML framework is formulated as
min α
( LMTL(ω∗(α),Dval1 ), . . . ,LMTL(ω∗(α),Dvalm ) ) s.t. ω∗(α) = arg min
ω m∑ i=1 αiLMTL(ω,Dtri ), 0 ≤ αi ≤ 1 ∀i, m∑ i αi = 1, (7)
where LMTL(ω,D) = 1|D| ∑
(x,y)∈D `(f(x;ω), y) denotes the average loss of f(·;ω) on a dataset D with |D| denoting the size of D and `(·, ·) denoting a loss function. In the LL subproblem of problem (7), when given weights in α, we aim to learn a MTL model to get optimal parameters ω∗ on the training dataset and in the UL subproblem, we expect to update α via minimizing the loss of the trained MTL model with parameters ω∗ on the validation dataset of each task.
Experiments. Experiments are conducted on the NYUv2 [53], Office-31 and Office-Home [60] datasets. Baseline methods in the comparison include different loss weighting strategies such as Equal Weights (EW), Dynamic Weight Average (DWA) [31] with the temperature parameter T as 2, MGDA [51], PCGrad [71] and SOML, and they are built on two multi-task architectures, including Deep Multi-Task Learning (DMTL) that adopts the hard-sharing structure to share the first several layers and Multi-Task Attention Network (MTAN) [31]. The ResNet-50 is used as the backbone and we do not use data augmentation. Due to page limit, the introduction of datasets, details on
experimental settings, and experimental results on the Office-31 and Office-Home datasets are put in Appendix E.
Experimental results on the NYUv2 dataset are shown in Table 3. Firstly, SOML outperforms the EW, DWA, and MGDA strategies when using the DMTL architecture and achieves comparable performance with the four baselines with the MTAN architecture. It indicates the proposed bi-level formulation (i.e., problem (7) with weighted combined objectives in the UL subproblem used in SOML) can achieve very good performance when compared with the state-of-the-art baselines. Secondly, the MOML method outperforms the SOML method under both DMTL and MTAN architectures, which means the multi-objective formulation in the UL subproblem is better than the single-objective formulation. Finally, it is noticeable that the proposed MOML method achieves state-of-the-art results in many metrics. For example, when training with the DMTL architecture, MOML can achieve 54.98% in terms of the mIoU, which is significantly higher than other baselines even with the advanced MTAN architecture. MOML with the MTAN architecture achieves the best results on six metrics. Experimental results on the NYUv2, Office-31, and Office-Home datasets, the latter two of which have their results in Appendix E, show that the proposed MOML framework achieves state-of-the-art performance for learning loss weights in MTL.
The loss weights learned by the proposed MOML are (0.3120, 0.3804, 0.3076) and (0.2541, 0.4860, 0.2599) for the three tasks (i.e., semantic segmentation, depth estimation, and surface normal prediction) when using the DMTL and MTAN architectures, respectively. It is interesting to find that when using different architectures, the weight of the depth estimation task is commonly the highest, while the other two tasks have similar weights.

heading: 5.4 Neural Architecture Search
text: NAS aims to design the architecture of neural networks in an automated way. In architecture design, we usually need to consider multiple factors. For example, we expect that the searched architecture has good performance, behaves robustly to noises, and consumes low resources. In this case, we formulate the NAS with multiple objectives as a case of MOML. Due to page limit, the introduction of datasets, details on experimental settings, and experimental results are put in Appendix F.
Problem Formulation. By following the DARTS method [29], in an operation space denoted by O, each element is an operation function o(·) and each cell is a directed acyclic graph with N nodes, where each node represents a hidden representation and each edge (i, j) denotes a candidate operation o(·) with a probability α(i,j)o . Therefore, α = {α(i,j)o }(i,j)∈E,o∈O is a representation of the neural architecture, where E denotes the set of all the edges in all the cells. The entire dataset is split into a training dataset denoted by Dtr and a validation dataset denoted by Dval. The multi-objective NAS considers three objectives: the classification accuracy, adversarial robustness and the number of parameters. We formulate three corresponding losses as LN (ω,α,Dval),LN (ω,α,Dadvval ), and Lnop(α), where ω denotes all the model parameters in the
neural network and Dadvval denotes the perturbed validation dataset by adding perturbations on each data point. LN (ω,α,D) denotes the average classification loss on a dataset D of a neural network with parameters ω and an architecture α,
LN (ω,α,D) = 1
n ∑ (x,y)∈D `(ω,x, y),
where `(ω,x, y) denotes the loss function for each sample and n is the size of D. To formulate Lnop(α), we denote by no the number of parameters associated with an operation o and by Nnop(α) the number of parameters in a searched architecture α. Then Nnop(α) can be computed by Nnop(α) = ∑ (i,j)∈E n
(i,j), where n(i,j) is the number of parameters of the searched operation on the edge (i, j). As we determine the operation of each edge by selecting the one with the largest probability, hence we have n(i,j) = n
argmaxo∈O α (i,j) o
. As the arg max
operation is non-differentiable, we use the softmax function to approximate Nnop(α) as N̂nop(α) =∑ (i,j)∈E ∑ o∈O
exp(α(i,j)o )∑ o′∈O exp(α (i,j) o′ ) no. Therefore, to search a network architecture with an expected
size L, Lnop(α) can be formulated as Lnop(α) = |N̂nop(α)− L|. (8)
Finally, the overall formulation for the multi-objective NAS is formulated as min α (LN (ω∗(α),α,Dval),LN (ω∗(α),α,Dadvval ),Lnop(α))
s.t. ω∗(α) = arg min ω LN (ω,α,Dtr). (9)
In the LL subproblem of problem (9), when given the architecture α, we can train a model with optimal parameters ω on the training dataset and in the UL subproblem, we expect to update the architecture α by making a trade-off among the validation loss, the adversarial robustness and the number of parameters. Obviously problem (9) matches the MOML framework. It is easy to see that the DARTS method is a special case of problem (9) when its UL subproblem contains the first objective only and hence problem (9) generalizes the DARTS method by considering two more objectives.
Compared with the NSGANetV2 method [34] that utilizes a multi-objective bi-level evolutionary algorithm, MOML is more efficient and has a convergence guarantee. Moreover, NSGANetV2 uses ensembled surrogate models to predict the accuracy of an architecture, which may incur a performance gap between the UL and LL subproblems. The LL subproblem of NSGANetV2 only chooses over 300 candidate architectures for evaluation with a supernet constructed for weight sharing, which may lead to suboptimal solutions.

heading: 6 Conclusions
text: As a generalization of meta learning based on the bi-level formulation, a simple MOML framework based on the multi-objective bi-level optimization is proposed in this paper. In the MOML framework, the upper-level subproblem takes multiple objectives of a learning problem into consideration. To solve the objective function of the MOML framework, a gradient-based optimization algorithm is proposed and the convergence property of this algorithm is studied. Moreover, several use cases of the MOML framework are investigated to demonstrate the effectiveness of the MOML framework. In our future work, we will apply the MOML framework to more learning problems.

heading: Acknowledgements
text: This work is supported by NSFC grant 62076118. We thank Dr. Jin Zhang for helpful discussion.

Review summaries:
Multi-objective bi-level optimization problem is formulated to efficiently do meta learning with multiple objectives. By properly using Multiple Gradient Descent Algorithm (MGDA), the authors propose a unified gradient-based multi-objective meta learning (MOML) framework with showing convergence guarantee. Moreover, it has been appropriately applied to several meta learning problems with more than one objective. 

The authors consider and propose a multiple objectives perspective for Meta-Learning. More specifically, they formulate the problem as a bilevel-optimization problem in which the lower-level problem is the within-task problem and the upper-level problem is a vector-valued problem aiming at minimizing a multi-objective meta-learning problem. Existing state-of-the-art methods for multiple objectives meta-learning mainly combine multiple objectives into a single objective in a weighted sum manner and they tune the weights associated with all the objectives, which could be time consuming. Others, use new evolutionary algorithms not relying on convergence guarantees and not easily applicable to deep learning methods. Differently from those works, the authors propose a gradient-based Multi-Objective-Meta-Learning (MOML) gradient-based method not requiring tuning weights. Their method alternatively solves the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Under specific assumptions, the authors guarantees an asymptotic convergence result for their method, assuming to be able to compute the meta-subgradients of the meta-objective. Finally, they show the advantage of applying their multi-objective meta-learning method in comparison to state-of-the-art methods on problems of few-shot learning, semi-supervised domain adaptation, multi-task learning, and neural architecture search.

This work leverages multi-objective optimization to augment meta learning algorithms. In particular, the inner optimization proceeds as normal; while the outer optimization uses Desideri's multiple-gradient descent algorithm to dynamically scale the per-task inner gradients. The paper concludes with a theoretical proof of convergence and empirical analysis on several applications of meta learning.