Multi-Objective Meta Learning

Abstract: Meta learning with multiple objectives has been attracted much attention recently since many applications need to consider multiple factors when designing learning models. Existing gradient-based works on meta learning with multiple objectives mainly combine multiple objectives into a single objective in a weighted sum manner. This simple strategy usually works but it requires to tune the weights associated with all the objectives, which could be time consuming. Different from those works, in this paper, we propose a gradient-based Multi-Objective Meta Learning (MOML) framework without manually tuning weights. Specifically, MOML formulates the objective function of meta learning with multiple objectives as a Multi-Objective Bi-Level optimization Problem (MOBLP) where the upper-level subproblem is to solve several possibly conflicting objectives for the meta learner. To solve the MOBLP, we devise the first gradient-based optimization algorithm by alternatively solving the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Theoretically, we prove the convergence properties of the proposed gradient-based optimization algorithm. Empirically, we show the effectiveness of the proposed MOML framework in several meta learning problems, including few-shot learning, domain adaptation, multi-task learning, and neural architecture search. The source code of MOML is available at https://github.com/Baijiong-Lin/MOML.


Reviews and author responses:

 We thank you for the detailed and constructive comments. We have finished the non-asymptotic analysis of our algorithm and we give the proof sketch in the following. 

We make the following assumptions: (i) $f(\omega,\alpha)$ is $c_1$-strongly-convex w.r.t. $\omega$. (ii) The functions $F_i$, $\nabla_\alpha F_i$, $\nabla_\alpha f$, $\nabla_{\alpha} \nabla_{\omega} f$ and $ \nabla^2_{\omega} f$ are $V$-, $L_1$-, $L_2$-, $\chi$-, and $\psi$-Lipschitz w.r.t. $\omega$, respectively. (iii) $|| D(\nabla \varphi,\gamma)|| \le M$, where the weight $\gamma$ is computed by MGDA and $D(\varphi,\gamma) = \sum_{i=1}^m \gamma_i\varphi_i$ denotes the convex hull of $\varphi_i$. (iv) $\varphi(\alpha)$ is $c_2$-strongly-convex and $L$-Lipschitz. (v)  $\gamma$ is a $\beta$-Lipschitz continuous function of the parameters $\varphi_i(\alpha)$, $1\le i \le m$. Here the first assumption is to satisfy the lower-level singleton condition.

We first analyze the gradient estimation error of $\nabla\varphi(\alpha)$. For the $i$-th entry of $\varphi(\alpha)$, we have $\nabla \varphi_i(\alpha_k) = \nabla_{\alpha}F_i(\omega^*(\alpha_k),\alpha_k) + \frac{\partial \omega^*(\alpha_k)}{\partial \alpha_k} \nabla_{\omega}F_i(\omega^*(\alpha_k),\alpha_k)$. Then by using assumptions (i) and (ii), we get the gradient estimation error for the $i$-th entry of $F$ with $K$ inner-loop steps as
$$|| \frac{\partial F_i(\omega_K(\alpha_k),\alpha_k)}{\partial \alpha_k} - \nabla \varphi_i(\alpha_k) || \le (1-\mu c_1)^{K/2} H_1 ||\omega^0-\omega^*(\alpha_k) || + H_2 (1-\mu c_1)^{K},$$ 
where $H_1$ and $H_2$ are two functions in terms of $V$, $L_1$, $L_2$, $\chi$, and $\psi$, and can be considered as two constants.  To simplify this result, we can prove that $||\omega^0-\omega^*(\alpha_k)||$ is bounded by a constant since the set of $\omega$ is bounded. By denoting by $\Gamma(K)$ the right-hand side of the above inequality, we can see that the gradient estimation error $\Gamma(K)$ decays exponentially w.r.t. $K$ if the step size $\mu < 1/c_1$.

We now consider the convergence of the UL subproblem. Since the weight $\gamma$ is determined by MGDA. we will get different $\gamma$'s for the true gradient $\nabla \varphi_i(\alpha_k)$ and the approximate gradient $\frac{\partial F_i(\omega_K(\alpha_k),\alpha_k)}{\partial \alpha_k}$. Thus, to further analyze the error caused by $\gamma$ in each iteration, we need assumption (v).  Then by using assumptions (iii) and (iv), we can get following convergence result as

$$\min_{t=1...T}D(\varphi(\alpha_t),\gamma_t) - D(\varphi(\alpha^*),\bar{\gamma_t}) \le \frac{2 L^2}{c_2(t+1)} + 2(1+\beta M)\delta \cdot \Gamma(K),$$

where $ \bar{\gamma_t} = (\sum_{t=1}^{T}t\gamma_t)/\sum_{t=1}^{T}t$,  $\delta$ is the upper-bound of the distance between any two points in the bounded set $\mathcal{A}$, and the step size of the UL subproblem $\nu$ is set to be $\frac{2}{c_2(t+1)}$. The sequence $\{\gamma_t\}$ is bounded and converges to $\gamma^*$. Let $\alpha^*$ be the unique solution corresponding to $\gamma^*$. Then $\bar{\gamma_t}$ converges to $\gamma^*$, which implies that $\min_{t=1...T}D(\varphi(\alpha_t),\gamma_t)$ converges to the Pareto optimal solution. So we get the non-asymptotic result as this convergence depends on both numbers of the inner-loop steps and outer-loop steps (i.e, $K$ and $t$).

We will add this non-asymptotic analysis and give a detailed proof in the revision.

 Thank you for your suggestion. We will reorganize the paper in the revision.

**About the Novelty**

The contributions of our paper are three-fold, i.e., the MOML framework, the optimization algorithm with analysis, and the applications to several areas of machine learning.

The first contribution of our paper is to propose the MOML framework based on the multi-objective bi-level problem, including a clear problem definition, a unified formulation, and a general optimization algorithm with theoretical guarantee. We think that the proposed framework is very important since  it studies a certain problem in depth and unifies some seemingly unrelated machine learning problems from a novel perspective, which will give readers more inspirations. 

A similar example is the graph embedding framework (refer to the following reference [1] published in IEEE TPAMI with more than 3000 citations in Google scholar) in the area of Dimensionality Reduction (DR), which gives a unified formulation for various DR techniques. Although the graph embedding framework in [1] seems "limited" novelty as this framework only gives a unified formulation without any other "new'' thing, this framework is considered to be one of the most important works in the DR area since this framework can provide insights for DR. Similarly, we think that the proposed MOML framework in our paper is a contribution to inspire more advanced models in different areas of machine learning, let alone the proposed gradient-based optimization algorithm  with the convergence guarantee for the first time.

The second contribution in our paper is to provide an optimization algorithm to solve the multi-objective bi-level optimization problem in the MOML framework with convergence analysis. Currently, the state-of-the-art methods to solve the multi-objective bi-level optimization problem are evolutionary algorithms. It is well known that evolutionary algorithms have a very high computational complexity as they converge slowly according to empirical observations and currently it is hard or even impossible to analyze their convergence. Due to those reasons, evolutionary algorithms are seldom used in machine learning. Hence, we propose a gradient-based optimization algorithm with a much lower complexity to solve the multi-objective bi-level optimization problem for the first time, which we think is a contribution to the machine learning community. Moreover, it is well known that the convergence analysis of multi-objective optimization problems is much harder than that of single-objective optimization problems, which are just conventional optimization problems, according to the following references [2,3]. Similarly, the convergence analysis of multi-objective bi-level optimization problems is much harder than that of conventional bi-level optimization problems. Hence, we think that the convergence analysis in this paper is worth a contribution.

The third contribution is that the MOML framework is applied to various areas of machine learning and achieves superior performance against state-of-the-art baseline models in respective areas, which demonstrates the usefulness and effectiveness of the proposed MOML framework.

[1] Graph Embedding and Extensions: A General Framework for Dimensionality Reduction. Yan et. al. IEEE TPAMI, 2007.

[2] A Stochastic Multiple Gradient Descent Algorithm. Mercier et.al. European Journal of Operational Research, 2018.

[3] Variational Convergence for Vector-valued Functions and its Applications to Convex Multiobjective Optimization. López. Mathematical Methods of Operations Research, 2013.

 Yes indeed, the scalarization is called SOML in this draft. However, I think the author should reorganize the presentation to move SOML to the background. You chose a moo version of gradient descent to optimize the problem.

However, my biggest concern still exists. I don't see any new individual components or good insight out of this paper, I still believe this paper is a recombination of all existing techniques. Therefore I revise my score to weak reject. 

This paper claims to present a "new" framework for multi-objective meta-learning. Their methodology is straightforward and easy to understand, and they have empirically evaluated their methods on several tasks and showed good results.  The title of this paper excites me in the first place, but the idea of this method quickly disappoints me as the method is a simple learning extension to the current method. Unfortunately, I did not learn any new insight from this paper, so I decide to reject it.
Pros:
easy to understand
many experiments with good results
Cons:
I suggest authors taking a broader investigation of literature in Multi-objective Optimization. There are many approaches to consolidate multiple objectives into one object to transform the problem into single-objective optimization. Here you only studied the scalarization, largely ignoring other methods such as hyper-volume, and etc.. The authors should conduct an in-depth literature review in this field. Frankly, even Wikipedia covers a few methods. https://en.wikipedia.org/wiki/Multi-objective_optimization
The idea is boring. Correct me if I'm wrong; I believe the method only parameterizes the weights for each objective at the upper level and learns these weights. That's the main contribution of this method. If this is the case, I can easily come up with thousands of variants of proposed methods, e.g., by learning the scalarization, by learning the construction of hyper-volume. I understand you have a good result, but why is this better than others? The idea should be a few lines (< 5) of changes to DARTS.
The evaluation seems to cover quite different tasks, but the improvement seems very limited (e.g. Table.2). Overall it cannot convince me that this method solves some problems that other methods cannot solve. I understand a few points of improvement in these tables, but these numbers are not significantly different.

======to other reviewers======

To other reviewers who are not familiar with DARTS, I highly recommend taking a look at it here (https://arxiv.org/abs/1806.09055). The proposed method simply combines multiple objectives of DARTS, and performs the experiments on different downstream tasks. Therefore, this novelty and contribution of this paper is very limited. 

I read the authors' comments, but the authors did not address my novelty concerns, especially on its difference to DARTS. This work is clearly a trivial version of DARTS applied to a different domain. Here the trivial part is that
1) the formulation of bi-level optimization is exactly same as DARTS.
2) Darts goes to single objective optimization, which optimizes 

min o1

The proposed method in this paper is simply:

min o1*w1+o2*w2+o3*w3.....

where w1, w2, w3 are learnable weights.

The inner loop of both methods is exactly same. So I really don't think this is a valid contribution to the research community, especially to a top conference like ICLR. See my comments above. 

I don't foresee any negative social impact.

 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

**Q1**: Isn't the equation below eq.2 transforms the objective function into the same formulation I listed above (yes my description is a simplified version)?

**A1**: Definitely NO! As mentioned in our paper, the equation below Eq. 2 (Line 117-118) is the formulation of Single-Objective Meta Learning (abbreviated as SOML in our paper) **rather than the proposed MOML method**. This method is only used as a baseline method to further show the effectiveness of our proposed MOML method. We have clearly state this in our paper. Please read **Line 121-123** in our paper carefully.

**Q2**:  It seems MOML scalarizes multiple objectives into 1 objective and then performs the exactly same optimization routine like DARTS except for the inner level used MOP-Solver.

**A2**: As mentioned in our response before, our proposed MOML method does not simply combine multiple objectives into single objective via linear scalarization but optimize directly a vector-valued function by multi-objective optimization algorithms. In addition, the MOPSolver is used in the upper-level optimization rather than the lower (or inner) level optimization, which is also mentioned in our response before. Finally, multi-objective bi-level optimization is totally different from the bi-level optimization in terms of both optimization and convergence analysis.  

**Q3**: It seems the results in table.3 has no apparent advantages.

**A3**: Firstly, we have compared the proposed MOML method with the state-of-the-art baselines in all cases. Secondly, to the best of our knowledge, although the improvements seem not so large, these have been a large progress especially in semi-supervised domain adaptation and multi-task learning problems.

 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

To clearly discuss the connections and differences between [1] and our work, we use the notations in [1]. In fact, the formulation of the Online-Within-Online (OWO) Meta-Learning problem in [1] can also be considered as a bi-level optimization problem (i.e., Eqs. (2) and (3) in [1]). The predictor vector $\omega$ is solved by minimizing the LL subproblem $L_{Z_t}(\omega,\theta) $ .Then the meta-parameter $\theta$ is solved in the UL subproblem, i.e., $\theta^* = \arg\min_{\theta} \sum_{i=1}^{T}L_{Z_i}(\omega,\theta) + F(\theta)$. This bi-level optimization problem also needs to satisfy the lower-level singleton condition (i.e., Assumption 1 in [1]). However, in [1], the UL subproblem is the sum of the losses in $T$ tasks, which is a scalar-valued optimization problem. Our work focuses more on the vector-valued optimization problem on the upper level. Moreover, [1] mainly focuses on online meta-learning, which means that the datasets are observed sequentially. Therefore, in the OWO meta-learning, the online mirror descent method is adopted to solve the within-task problem. This online meta-learning setting is not considered in our paper. Our work pays more attention to the application of our framework to several learning problems such as few-shot learning, semi-supervised domain adaptation, multi-task learning, and neural architecture search.

One main contribution of [1] is to derive a cumulative error bound for OWO meta-learning. However, such an error analysis method in OWO meta-learning setting is not easy to be applied to our framework. The main reason is that the structure of UL and LL variables in our framework is more general than that in [1].  In [1], the approximation error of the LL subproblem can be accumulated into the upper level, because the UL subproblem can be regarded as the accumulation of the LL objective function in $T$ tasks plus a meta-regularizer $F(\theta)$ (See Eq. (3) in [1]). However, in MOML, there is no direct dependency between the UL and LL subproblems since we only assume that they are two different functions. Due to the lack of such a direct dependency, it is difficult to use the analysis method in [1] to analyze the error of MOML. We will further explore the cumulative error bound in our future research.

We can take a similar analysis to the reference `On the Iteration Complexity of Hypergradient Computation' to estimate the approximation error of the meta-gradient.

For the non-asymptotic convergence rate, we will first study that for multi-objective optimization and then we can analyze for the proposed MOML framework. 

We sincerely thank you for the suggestions. We will add the above discussion between references [1,2,3] you suggested and MOML in the revision.

 Q2: It seems this work is clearly a trivial version of DARTS applied to a different domain. The formulation of bi-level optimization is exactly the same as DARTS.

A2: Firstly, our main contribution is to propose a.......


===> Follow up response:
MOML scalarizes multiple objectives into 1 objective and then performs the exactly same optimization routine like DARTS except for the inner level used MOP-Solver. 

MOP Solver is not new, scalarizing multiple objective is not new, and bi-level optimization routine is not new. Therefore, MOML is a recombination of existing techniques. Unless the authors proposes some novel piece, please clarify here. I get it you tried to claim MOML is new, but a recombination of these techniques does not seems new to me. 

Q3: About the novelty

A3: The reviews of the reviewer on the contribution of this paper are one-sided. What we want to emphasize is that the proposed framework has wide applications in machine learning, so we use four cases to illustrate this. The reviewers simply ignored our work on semi-supervised domain adaptation and multi-task learning. We transform these two problems into a multi-objective bi-level optimization problem, which has not been studied in previous works. On these two cases, we achieved state-of-the-art performance and this shows the effectiveness of our framework. Moreover, note that in this paper we propose the FIRST gradient-based method with convergence guarantee.

===> Follow up response:

It is a bit tricky to evaluate the results in table.3, no apparent advantages are observed. If you claims this is better than others, can you elaborate why your are better than others?

 Thank you. I understand the objective is a vector not a value, but isn't the equation below eq.2 transforms the objective function into the same formulation I listed above (yes my description is a simplified version). 


 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

We have mentioned that our proposed method is not simply to minimize o1*w1+o2*w2+o3*w3....., with learnable w1, w2, w3, ... There are two misunderstandings in your insight. Firstly, what we want to solve is to minimize a vector-valued function (i.e., problem (1) in our paper) **rather than a weighted sum function**. Secondly, the MGDA approach we adopt in our paper finds the Pareto-stationary solution by solving a quadratic programming  problem (i.e., problem (8) in our paper) rather than learning the weights. Note that the weights are **not learnable** and is determined by  minimizing the optimality condition of the multi-objective optimization in each iteration.

There is another mistake in your comments. We use MOPSolver in the upper level optimization **rather than lower level optimization**. 

We emphasize once again that our main contribution is to propose a unified multi-objective meta learning framework with convergence guarantee. The convergence analysis of a multi-objective meta learning problem is much challenging than a meta learning problem or a multi-objective problem. Hence, we think our work is meaningful rather than a trivial extension.

 My biggest concern of this paper is whether it introduces a new knowledge to the readers? As I have pointed in my review above, the contribution of this paper is trivial to me. Here is my rational:

The paper does not introduce any new methods, it is almost same as DARTS that formulates the problem as "bi-level optimizations". The only key difference is that DARTS solves 

min o1

The proposed method in this paper is simply:

min o1w1+o2w2+o3*w3.....

where w1, w2, w3 are learnable weights.

You mentioned you used MOPSolver in lower level optimization, but this is just a trivial modification. As I said,  you can estimate/calculate the gradient using many other techniques, gradient descent, evolutionary algorithm and etc..

However, the general framework of this method is same as the published works except for that some minor details have been changed. From this perspective, the novelty of this paper is very limited, and nearly contribute no new knowledge to the community. This is okay if the authors solve a very difficult problem with known techniques, but the experiments in this paper does not seem to be so.

If you think I miss something, please further clarify your "main" contribution, rather than trivial extension e.g. replacing sgd with other solvers. My current stance is to argue for rejection.


 Thanks to the authors for the reply. I think that the paper could benefit if the authors addressed more in deep my Q2, Q3 and Q4. In particular, regarding Q2, could you please clarify which are the main differences you noticed between the proposed framework and the framework in [1]? 


 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

**Q1**: It seems the proposed method is simply min o1*w1+o2*w2+o3*w3....., where w1, w2, w3 are learnable weights.

**A1**: We think the reviewer has strongly misunderstood our work. In our proposed MOML framework, there is a multi-objective problem in the upper-level subproblem, which means what we want to solve is to minimize a vector-valued function (i.e. problem (1) in our paper) rather than a weighted sum function. We adopt the MGDA method to solve this multi-objective problem in our paper. The MGDA approach finds the Pareto-stationary solution by solving a quadratic programming problem (i.e. problem (8) in our paper) rather than learning the weights. Note that the weights are not learnable and is determined by minimizing the optimality condition in each iteration.

**Q2**: It seems this work is clearly a trivial version of DARTS applied to a different domain. The formulation of bi-level optimization is exactly the same as DARTS.

**A2**: Firstly, our main contribution is to propose a unified framework which can model various practical application problems in different fields. NAS (or DARTS) is one of the practical application examples to show our proposed MOML framework meaningful. In addition, any DARTS-based or other gradient-based NAS methods with multiple objectives to consider can be fitted and solved by our proposed framework, which is one of the advantages of our work. Secondly, MOML is to solve a multi-objective bi-level optimization problem, which is much more general than the conventional bi-level formulation in DARTS. So in this paper we propose the FIRST gradient-based method with convergence guarantee. Note that the optimization method used in DARTS CANNOT be used to solve the multi-objective bi-level formulation in MOML.  

**Q3**: About the novelty

**A3**: The reviews of the reviewer on the contribution of this paper are one-sided. What we want to emphasize is that the proposed framework has wide applications in machine learning, so we use four cases to illustrate this. The reviewers simply ignored our work on semi-supervised domain adaptation and multi-task learning. We transform these two problems into a multi-objective bi-level optimization problem, which has not been studied in previous works. On these two cases, we achieved state-of-the-art performance and this shows the effectiveness of our framework. Moreover, note that in this paper we propose the FIRST gradient-based method with convergence guarantee.

 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

**Q1**: It is recommended to further elaboration in Section 3 and 4 to elucidate the proposed method.

**A1**: We will further clarify our proposed method in the revision, including defining $\omega$ and $\alpha$ and giving details of the MOPSolver.

**Q2**: It is recommended to give a runtime comparison of MOML to SOML.

**A2**: Using multi-objective optimization methods to solve the UL subproblem does run slower than SOML, because we need an extra step to determine the common descent direction for multiple objectives. We will discuss this limitation and compare the running time in the revision.

 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

**Q1**: It is recommended to give an example of the multi-meta-objective $F$, immediately after Eq. 1.

**A1**: We will give an example to motivate the multi-meta-objective $F$ in the revision.

**Q2**:
Is it possible to formally quantify such an approximation error as done in [1] below? 

**A2**:
We notice that the proposed framework is different from that in the given reference. We think that the cumulative error bound may not be directly calculated by the lower-level subproblem in our bi-level optimization problem. We will add disucssions with [1] in the revision and further explore this error bound problem in our future research.

**Q3**: What about the step in which the authors compute the meta-subgradients by a routine? Can you estimate the approximation error? 

**A3**: In each iteration, we calculate the gradient of $\alpha$ in the upper level by automatic differentiation techniques. Thus, we need to store the intermediate variables {$\omega_t$}$_{t=1}^K$ and evaluate the Jacobian term. The error of this full back-propagate approach is zero. Therefore, if we assume that the optimal solution of the LL subproblem can be obtained by the $K$-step gradient descent, the error of the meta-gradient computation is only caused by the stochastic gradient descent, which is commonly assumed to have an error bounded by a constant. However, in practice, we can only find an approximated solution $\omega_K$, which is close to the optimal solution $\omega^*$. This means we need to evaluate the error between $\frac{\partial F(\omega_K,\alpha)}{\partial \alpha}$ and $\frac{\partial F(\omega^*,\alpha)}{\partial \alpha}$. Following the analysis in the following reference [1], we can prove that, under certain assumptions, the error of this approximation decays exponentially w.r.t. $K$ in the LL subproblem. We will add this analysis in the revision.

[1] On the Iteration Complexity of Hypergradient Computation. Grazzi et. al. ICML, 2020.

**Q4**: It would be nice to have a non-asymptotic convergence rate in Thm. 2. 

**A4**: The overall convergence rate in this framework relies on the gradient-based optimization methods used in both lower-level and upper-level subproblems. Following the answer to the last question, we can give a convergence rate analysis of the lower-level subproblem. However, the uniform convergence rate is still hard to prove because of the complexity of multi-objective optimization on the upper level. We will explore the non-asymptotic analysis for the proposed MOML in our future research.

**Q5**:
What about the connection to the conditional Meta-Learning approach introduced in [2,3] below?

**A5**: 
We think that the connection between such motivation and conditional meta-learning methods can be established by considering the robustness as a condition. Conditional meta-learning can handle complex task distribution via adaptive initialization by establishing a mapping from a dataset to meta-parameters. Therefore, our motivation may be achieved by adding an additional robustness loss function to establish such mapping in conditional meta-learning methods. 

**Q6**:
Clarify f in Eq. 2 and w in Eq. 1. 

**A6**:
We will clarify f to be the within-task objective and explicitly define the dependency between $\omega$ and $\alpha$ in the revision. 

**Q7**:“How to obtain multiple nearly optimal solutions for MOBLPs is beyond the scope of this paper and we will study it in our future work.” What do you mean? 

**A7**:
This means that the proposed gradient-based optimization method can find one Pareto optimal solution. This is different from other types of multi-objective optimization methods such as evolutionary algorithms which can output multiple solutions. Moreover, if we use some gradient-based multi-objective methods such as [2,3] to solve the UL subproblem, the proposed MOML method can also output multiple solutions and the convergence properties studied in this paper still hold.

[2] Pareto Multi-Task Learning. Lin et. al. NeurIPS, 2019.

[3] Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization. Mahapatra et. al. ICML, 2020.

 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

**Q1**: How the SOML was trained?

**A1**: In the case of semi-supervised domain adaptation, we directly apply the optimal weights using in the original method as the weights of SOML. In the other three cases (i.e., few-shot learning, multi-task learning, and neural architecture search), we first conduct a grid search on the weights of SOML and then select the best performance of SOML to report.

**Q2**: Can the proposed MOML handle with the situation of emphasizing particular objective function?

**A2**: The proposed MOML is a general framework and can also handle such situations you mentioned. In this paper, we use the MGDA method, which has no preference on any objective function, to solve the upper-level subproblem. If we want to emphasize particular objective function, we can use other gradient-based methods such as [1] to solve the problem (4). This will be a variant of the proposed MOML framework and our theoretical results still hold in this case. 

[1] Multi-Task Learning with User Preferences: Gradient Descent with Controlled Ascent in Pareto Optimization. Mahapatra et. al. ICML, 2020.

**Q3**: It is recommended to contain NAS results in the main paper.

**A3**: In the revision, we will appropriately adjust the main body and put the NAS results in the main paper.

 We thank you for the detailed and constructive comments. In the following we will make a response. If you think we have not addressed your concerns, we look forward to further discussions with you.

In this paper, we compare with the scalarization approach (i.e., SOML) as it is the most widely used approach in machine learning literature to handle multiple objectives. Other alternatives such as hyper-volume will be studied in our future work.

For the upper-level problem, we adopt the MGDA method, which is a gradient-based multi-objective optimization method. The MGDA method can find the Pareto-stationary solution by solving a quadratic programming problem whose solution is intended to find the common decent direction for multiple objectives, though it can be viewed as loss weights in some sense. Moreover, any gradient-based multi-objective optimization method can be used to solve the upper-level problem, which is an advantage of the proposed MOML method. 

In experiments, we have compared with the state-of-the-art baseline models. Experimental results show that the proposed MOML method outperforms those state-of-the-art models, which demonstrates the effectiveness of the proposed MOML method.

Multi-objective bi-level optimization problem is formulated to efficiently do meta learning with multiple objectives. By properly using Multiple Gradient Descent Algorithm (MGDA), the authors propose a unified gradient-based multi-objective meta learning (MOML) framework with showing convergence guarantee. Moreover, it has been appropriately applied to several meta learning problems with more than one objective.   - There have been many meta learning problems that optimize multiple objective functions, and the proposed multi-objective meta learning (MOML) framework is trying to make it easier by avoiding the hyper-parameter tuning for combing those objective functions.
- Multiple Gradient Descent Algorithm (MGDA) seamlessly fits into gradient-based meta-learning frameworks, and this proposed method is straight forward. Moreover, it is good to have convergence analysis to properly understand the method.
- The right applications have been shown in the experimental section, and it is interesting to see how this can be applied.
- The single-objective meta learning (SOML) is used as a baseline, where the weighting parameters for objective functions have to be tuned. However, it is unclear, how this SOML was trained or how the performance varies by randomly choosing the weighting parameters?
- When we have multiple objective functions, we would like to emphasize particular objective funcion rather than others. For SOML, we have chance to manually emphasize it, but could this kind of situation can be handled with the poposed MOML? 
 
 - it would be better to contain NAS results in the main paper by shortening other experiment parts or moving partially to appendix. 

The authors consider and propose a multiple objectives perspective for Meta-Learning. More specifically, they formulate the problem as a bilevel-optimization problem in which the lower-level problem is the within-task problem and the upper-level problem is a vector-valued problem aiming at minimizing a multi-objective meta-learning problem. Existing state-of-the-art methods for multiple objectives meta-learning mainly combine multiple objectives into a single objective in a weighted sum manner and they tune the weights associated with all the objectives, which could be time consuming. Others, use new evolutionary algorithms not relying on convergence guarantees and not easily applicable to deep learning methods. Differently from those works, the authors propose a gradient-based Multi-Objective-Meta-Learning (MOML) gradient-based method not requiring tuning weights. Their method alternatively solves the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Under specific assumptions, the authors guarantees an asymptotic convergence result for their method, assuming to be able to compute the meta-subgradients of the meta-objective. Finally, they show the advantage of applying their multi-objective meta-learning method in comparison to state-of-the-art methods on problems of few-shot learning, semi-supervised domain adaptation, multi-task learning, and neural architecture search.  The authors consider a multi-objective perspective for meta-learning in which one desires optimizing multiple objectives simultaneously at the meta-level. Such a problem seems to be interesting and the paper is quite well written.

Maybe, the necessity of considering a multiple-objective perspective in Meta-Learning is not enough motivated. It would be nice to give an example of the multi-meta-objective F, immediately after Eq. 1.

Since usually there is no closed-form solution to the lower-level problem, the authors solve such a problem approximately by applying Stochastic Gradient Descent (see Alg. 1). After this, they apply standard multi-objective optimization algorithms on the approximation of the upper-level problem obtained by replacing the lower-level solution with the last iteration of Stochastic Gradient Descent. The approximation error introduced by this substitution in not enough analyzed in my opinion. Is it possible to formally quantify such an approximation error as done in [1] below?

In order to study the convergence properties of their meta-algorithm in Alg. 1, the authors need to consider optimization tools for vector-valued functions and to make standard assumptions in literature, such as the existence and the uniqueness of the solution of the lower-level problem.  About the convergence result presented in Thm. 2: it would be nice to have a non-asymptotic convergence rate. 

What about the step in which the authors compute the meta-subgradients by a routine? This is not an exact computation, so there will be an approximation error. Can you estimate it as done in [1] below?

The authors say that, for MAML, one may need to consider not only the performance but also the robustness which can help adapt to new tasks with the learned initialization. If one motivation supporting the necessity of considering multiple-objective is controlling the robustness of the learned initialization, what about the connection to the conditional Meta-Learning approach introduced in [2,3] below?

f in Eq. 2 is the within-task objective. Clarify this. In Eq. 1, w depends on \alpha: indicate this dependency by denoting it with w_\alpha.

Lines 181-182: “How to obtain multiple nearly optimal solutions for MOBLPs is beyond the scope of this paper and we will study it in our future work.” What do you mean? Why?

The experiments seem to be exhaustive and show the advantage of applying their multi-objective meta-learning method in comparison to state-of-the-art methods on problems of few-shot learning, semi-supervised domain adaptation, multi-task learning, and neural architecture search.

1.   Online-Within-Online Meta-Learning, Denevi et al. 2019
2.   A structured prediction approach for conditional meta-learning, Wang et al. 2020
3.   The Advantage of Conditional Meta-Learning for Biased Regularization and Fine Tuning, Denevi et al. 2020. yes

This work leverages multi-objective optimization to augment meta learning algorithms. In particular, the inner optimization proceeds as normal; while the outer optimization uses Desideri's multiple-gradient descent algorithm to dynamically scale the per-task inner gradients. The paper concludes with a theoretical proof of convergence and empirical analysis on several applications of meta learning.   This work is well-written, and for the most part, lucid. Certain sections are quite dense, which I attribute to the author's desire to include a comprehensive explanation of their methods/proofs/experiments, but I would recommend further elaboration in Section 3 and 4 to elucidate the proposed method. For instance, clearly defining $\alpha$, clearly defining $w$, and elaborating on how the MOPSolver combines LL gradients would likely help readers better grasp the scope of the proposed method, especially if they are unfamiliar with multi-objective optimization. 

The theoretical analysis on convergence supports the authors' proposal, and strong empirical results on a diverse range of applications further bolster the utility of MOML. Given this is a general purpose augmentation that can be applied to many Meta Learning problems to improve performance, I believe it is a significant contribution to the field. However, one well-known detriment associated with Multi-Objective Optimization methods relates to its slow runtime. It would be beneficial for the authors to provide comparison to SOML (single objective, weighted combination in the outer optimization) to be transparent on the limitations of Desideri's algorithm. 

In summary, I believe this paper is well-written, incorporates prior work to make a novel contribution, provides theoretical analysis, offers comprehensive empirical results, and makes a meaningful contribution to the meta learning domain. In many ways, it feels like the Meta Learning analog of Sener and Koltun's Multi-Task Learning as Multi-Objective Optimization, which itself was a seminal work. I would recommend a runtime comparison of MOML to SOML should be conducted and added to the paper.

Meta-review: This paper proposes a method for multi-objective meta-learning which can be used for various settings such as semi-supervised and multi-task learning. The paper is well written. The authors provide the theoretical analysis on convergence, and the experimental results demonstrate the effectiveness of the proposed method. More discussion on the relation with the existing methods is needed to clarify the technical contribution of the proposed method.

Decision: accept