Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to formulate the meta-learning as a multi-objective optimization (MOO) problem instead of the traditional single objective problem where the sum of each task’s loss is optimized. By doing do the compromising phenomenon can be avoided, but it also incurs a new problem — the current gradient-based MOO solvers cannot scale to a large number of objectives. Then the authors proposes a scalable gradient-based solver via the use of improvement functions. Besides, this algorithm is guaranteed to have theoretical convergence. Experimental results on few-shot regression, few-shot classification and reinforcement learning show that the proposed method, called SIMOL, outperforms several existing algorithms in terms of either efficiency or generalization. 
strength_and_weaknesses: Strength:

1. This work is well motivated that reformulating the original problem to a MOO can avoid the compromising phenomenon.
2. The overall logic of deriving the solution along the journey is quite clear: 1) The improvement function can provide a closed-form gradient for the original problem; 2) The closed-form requires  uniform sampling of the tasks, which can be parameterized as a neural network; 3) Optimizing the network requires another sampling of tasks to compute the gradient of the NN; 4) Theoretically, the gap between the full batch and mini-batch of tasks can be bounded, so we can safely adopt the SGD to optimize the NN and finally solve the MOO problem with pareto optimality.
3. Empirical studies looks very promising that the proposed method works quite well on multiple tasks and is better than the competitors.


Weaknesses:

1. There are tons of math equations. The author may need to simplify or condense them a bit, and give timely interpretation for them. It is important to clearly extract and convey the message behind the equations to the readers.
2. The fonts of Figure 1 and 2 are quite small that it is hard to read.
3. Typos: e.g., “the converged solution may be comprised” —> “the converged solution may be compromised”
4. Immediately after Equation (7), it is logically not correct to say “We first rewrite the above as:”, since Lemma 3.1 is to prove that (7) and (8) are equivalent, but before the proof, we don’t know if that’s true.
5. Definition 2.1. and 2.2. may need to cite some classical literatures rather than those new ones, as these definitions should exist long time ago.
6. In Table 3, SIMOL has much large variance than others in the MAML category. Would that be a concern?
clarity,_quality,_novelty_and_reproducibility: The overall idea is clearly conveyed, though the massive equations hurts the readability a bit.

The idea of converting the mete-learning problem as a multi-objective optimization sounds new, and the derived algorithm with theoretical guarantee is a solid contribution too.

Algorithm 1 seems easy to reimplement, so the results could be reproducible with some effort. 
summary_of_the_review: The paper seems to be the first to reformulate the meta-learning problem as a MOO to avoid the compromising problem. Then a gradient-based algorithm guaranteed to converge to pareto optimal solution is proposed. Overall, this is a solid contribution.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers solving meta-learning (formulated as bilevel optimization with the inner level being tasks) using techniques from multi-objective optimization (MOO). Previous work along this line required computing gradients for each task at each iteration, which is expensive. This paper proposes an algorithm to find an $\epsilon$-Pareto stationary point using only mini-batches of tasks, built on the concept of improvement functions from MOO. Experiments are done on linear regression, few-shot classification, and meta-reinforcement learning. The proposed algorithm SIMOL generally improves over MAML and Prototypical Networks without significantly increasing computational burden, unlike previous work.
strength_and_weaknesses: # Strengths
- The worst task performance of SIMOL improves over those of vanilla MAML and Prototypical Networks without increasing the computational burden drastically. The overall performance with a PN backbone is also better.
- The algorithm is well mathematically motivated, and theoretical analysis of convergence is provided along with experiments.
- The paper is well written and the arguments are straightforward to understand. 

# Weaknesses
- There are no ablation studies, e.g. on the architecture of the network
- For the MAML backbone, the improvement of SIMOL is often not statistically significant
- The convergence theory may not be tight; the rate is slower than indicated from experiments

# Questions
- Are gradient signs missing from line 5 of Alg. 1?
- How are the hyperparameters chosen?
- Why is PCGard not included as a baseline?

clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper is written pretty clearly. The text is not difficult to understand, 

# Quality
The algorithm seems mathematically sound, although I did not closely check the proofs. The experiments are fairly complete in terms of evaluation metrics and diversity of environments, although no ablations are included. SIMOL provides improvements that are often statistically significant.

# Novelty
I am not completely up-to-date with the literature on multi-task optimization, but as far as I know the use of improvement functions for meta-learning is novel and presents a new class of algorithms.

# Reproducibility
The authors do not provide code, but provide pseudocode and hyperparameters. I don't think reproducing the results would be an issue.
summary_of_the_review: This paper is a novel application of approaches from MOO to meta-learning, with the intuition that a Pareto optimality would also lead to faster adaptation at test time. The algorithm is mathematically motivated and analyzed, and shown to improve upon MAML with similar computational complexity. However, SIMOL does not always lead to statistically significant improvements and no ablation studies are carried out.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes SIMOL, a soft improvement based multi-objective optimization algorithm, for efficient meta learning. The proposed method first formulates meta learning as a multi-objective optimization problem, where each task is an objective to optimize. Then, to handle the huge number of tasks (e.g., can be up to 7 x 10^6) in meta-learning, it further leverages the soft improvement algorithm with mini-batch update as its scalable gradient-based optimizer for model training. Theoretical analyses and experimental studies have been conducted to show the good properties and practical performance of the proposed method.   

strength_and_weaknesses: **Strengths:**

+ This paper is generally well-organized and easy to follow.

+ To my understanding, this is the first work to consider meta learning as a multi-objective optimization problem that treats each task (the number could be huge) as a single objective to optimize. But I also have some concerns on this formulation (see weaknesses below). 

+ The proposed SIMOL method is simple and easy to implement. 

**Weaknesses:**

**1. Novelty**

This work proposes to formulate meta learning as a multi-objective optimization problem. However, many similar works have already been done to leverage multi-objective optimization for multi-task learning [1,2,3]. Given the close relation between meta learning and multi-task learning [4], this formulation is straightforward.

The second contribution to dealing with the huge number of tasks is indeed interesting. However, the proposed scalable optimization method directly uses the improvement function [5] with the mini-batch update, which makes it like yet another multi-objective optimization method for MTL/meta learning. I also have some concerns on the proposed method's ability to deal with the huge number of objectives (see below).

**2. Multi-Objective Optimization Formulation for Better Generalization**

It is unclear why the multi-objective optimization formulation could lead to a better generalization performance for meta learning.

*i) Why Pareto solution on the training set is good for meta learning?* 

In multi-task learning, the goal is to find a solution to maximize the performance for all tasks (objectives). Since the tasks could be conflicted with each other, it is reasonable to find Pareto solution(s) for the given problem. In meta-learning, however, the goal is to improve the fast adaption performance for new tasks that are not seen for the training (optimization) process. Why could a Pareto solution on the training set be good for better generalization performance for unseen tasks?

Is it meaningful to formulate meta-learning as a multi-objective optimization problem with huge objectives, especially when the tasks we actually care about (e.g., new unseen tasks) are not in this huge set of objectives? A clear discussion on this motivation could be very helpful.

*ii) Why could the simple linear scalarization approach be worse than other methods for meta learning?*

To my understanding, the simple linear scalarization approach can still find (stationary) Pareto solutions, although it is true that it cannot find any Pareto solution on the non-convex part of the Pareto front. Since the goal here is to find a single Pareto solution (but not to find the whole Pareto front), it is unclear why the solution found by linear scalarization will perform worse than the solutions found by other multi-objective optimization methods. Different Pareto solutions should be non-dominated with each other, and it is hard to tell which one should be better than the others, especailly for the performane of unseen tasks.

*iii) Why the improvement function method could be better to handle meta learning with huge tasks?*

The improvement method used in this work is a non-scalarization multi-objective optimization algorithm, which aims to find a common descent direction for **all** objective functions. Many other multi-objective optimization algorithms also have the same goal but with different mechanisms. Given its min-max structure, the improvement method is closely relative to the Chebyshev method for multi-objective optimization [6] (now with equal preferences and adaptive reference point), which has been used for MTL [7,8]. Why could it have better performance for meta learning?

*iv) Why a single Pareto solution could be good for all tasks?*

Closely related to the previous points, since there is no single best solution for all objectives, most multi-objective optimization methods aim to find a good approximation (e.g., a set of representative solutions with different optimal trade-offs) for a given problem [6]. See the related approaches for multi-task learning [9, 10, 11] and also meta multi-objective reinforcement learning [12]. 

The motivation for finding only one single Pareto solution to have the best overall performance for all objectives (and even the new objectives that the algorithm has never seen) is not solid. 

**3. Algorithm Performance**

The closely related work that takes meta-learning as multi-task learning [4] should be included as a baseline. Since this work is "one step further" from [4], why not directly develop the SIMOL method based on [4] which could be more efficient and straightforward than the bilevel formulation?

**Much more importantly**, due to the opposite findings in some closely related work, a solid analysis could be needed to support the proposed algorithm's promising performance. The key motivation of this work is to use a multi-objective optimization algorithm to solve the meta-learning problem, and some promising performances have been reported for different problems. However, the current work on multi-task learning leads to an opposite finding. According to [13,14,15], a simple linear scalarization method can perform comparably or even better than all those multi-objective optimization methods for both supervised learning and reinforcement learning. The effect of fine-tuned hyper-parameters (rather than the multi-objective optimization formulation) could lead to significantly different algorithm performances. Due to the close relation between multi-task learning and meta-learning, strong and solid evidence and discussion should be provided to support the finding in this work. 


**4. Theory**

This work leverage the theoretical analysis from the improvement function method [5] to give the (Pareto stationary) convergence analysis, and Lin et al. [16] to give the convergence analysis for mini-batch update. However, the more important issues, such as the connection from the Pareto solution to better generalization performance for meta-learning, and the reason why the Pareto solution found by simple linear scalarization could have poorer generalization performance, are not discussed.

**Other Comments**

1. In Figure 1, if batch size =1, both mini-batch MGDA and SIMOL will only have one objective value (either f1 or f2) and its gradient at each step. There should be only a single term in (2) for SIMOL without any maximization comparison. Why did they perform quite differently in this case? 

2. It is also interesting to know the performance of the best-10% for different problems.

3. It seems that a ")" is missing in the definition of \bar d^* two lines below (16).

[1] Multi-task learning as multi-objective optimization. NeurIPS 2018.

[2] Gradient surgery for multi-task learning. NeurIPS 2020.

[3] Conflict-averse gradient descent for multi-task learning. NeurIPS 2021.

[4] Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation. ICML 2021.

[5] Interactive bundle-based method for nondifferentiable multiobjeective optimization: NIMBUS. Optimization 1995.

[6] Nonlinear multiobjective optimization. Kluwer 1998.

[7] Tchebycheff Procedure for Multi-task Text Classification. ACL 2020.

[8] A Multi-objective / Multi-task Learning Framework Induced by Pareto Stationarity. ICML 2022.

[9] Pareto multi-task learning. NeurIPS 2019.

[10] Efficient Continuous Pareto Exploration in Multi-Task Learning. ICML 2020.

[11] Learning the Pareto Front with Hypernetworks. ICLR 2021.

[12] Meta-Learning for Multi-objective Reinforcement Learning. IROS 2019.

[13] A closer look at loss weighting in multi-task learning. arXiv:2111.10603, 2021.

[14] In Defense of the Unitary Scalarization for Deep Multi-Task Learning. arXiv:2201.04122, 2022.

[15] Do Current Multi-Task Optimization Methods in Deep Learning Even Help? arXiv:2209.11379, 2022.

[16] On gradient descent ascent for nonconvex-concave minimax problems. ICML 2020.

clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is generally well-written and well-organized.

**Quality:** The quality of the overall paper is good. But the lack of discussion for some crucial concerns listed in the weaknesses makes the contribution unclear.

**Novelty:** The multi-objective optimization formulation is a reasonable extension from closely related work, and the novel contribution on dealing with huge tasks for meta learning is not well supported by the current analysis. 

**Reproducibility:** Given the recently opposite findings on multi-objective optimization for MTL, there is a concern that the proposed method could not be robust for the meta learning problem.
summary_of_the_review: To my understanding, this is the first work to consider meta learning as a multi-objective optimization problem that treats each task (the number could be huge) as a single objective to optimize. However, due to the major concerns on the multi-objective optimization formulation, algorithm performance, and theoretical analysis, I cannot vote to accept the current manuscript.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper develops a new method, called SIMOL, for meta learning through using multi-objective optimization.  The authors argue that optimizing the average loss across tasks can cause the meta learning method to focus on a small set of tasks.  Thus instead they propose treating each task as a separate objective in a multi-objective optimization framework.  On a variety of experiments the authors demonstrate that SIMOL outperforms at least in the worst case on the popular MAML and Prototypical network methods. 

The reviewers found the work clearly written, well motivated and technically sound.  One reviewer initially raised a number of questions, including why the multi-task optimization approach would necessarily lead to better generalization performance (as opposed to training set loss).  It appears that the authors convincingly addressed these concerns and the reviewer raised their score to an accept accordingly.  Another reviewer indicated they were willing to "champion the acceptance" after reading the author feedback.  Other weaknesses identified by the reviewers included questions about statistical significance regarding a subset of the experiments comparing to MAML and a lack of ablation studies.  

Overall the reviewers were in agreement on acceptance (8, 6, 6).  Overall, this work seems broadly interesting to the community in meta learning and the fact that it's a novel approach that should be applicable on top of other approaches could make it quite useful. 
 Therefore the recommendation is to accept the paper.  Hopefully the feedback from the reviewers and the responses from the discussion period will be useful in further refining the manuscript before publication.  
justification_for_why_not_higher_score: While this paper was above borderline, two reviewers thought it was a marginal accept (although one did say they would champion acceptance).  The work seems broadly interesting, but the reviewers did raise some questions including regarding the statistical significance of the results and ablation studies.
justification_for_why_not_lower_score: Meta learning is broadly interesting to the community, and to that sub-field and this is a novel approach that seems to improve things.
Decision: Accept: poster