Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores the generalization of overparameterized meta-learning models with a nested structure, specifically model-agnostic meta-learning (MAML) and implicit MAML (iMAML). Under some theoretical assumptions, the authors derive an upper bound for the excess risk of MAML and iMAML in the overparameterized setting. Further, they derive conditions for benign overfitting in nested meta-learning and provide hyperparameters for MAML and iMAML that preserve the benign overfitting.
strengths_and_weaknesses: The paper does a good job of explaining the motivations for studying the problem. In addition, the authors present their contributions clearly and provide a proof outline of their main result, which contributes to the clarity of the paper.

The main weakness of the paper is in the discussion of the experimental results. I believe that the significance of the theoretical results could be further emphasized by explaining the experimental results in more depth (and adding some numerical results in a tabular format). 
questions: Is there an intuitive way to explain why the cross-task variance upper bound primarily depends on 1/M asymptotically?
limitations: The authors adequately address the limitations of their work. Specifically, they acknowledge that their current analysis is limited to the non-Bayesian setting and the linear model assumption.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper analyzes the “benign overfitting” phenomenon of nested meta-learning, where an over-parameterized model can still work well in the meta linear regression setting. Both MAML and iMAML are covered by the analyses. Numerical simulations validate the results of the theoretical analysis.

strengths_and_weaknesses: The paper analyzes an interesting phenomenon in meta-learning in the meta linear regression setting. The authors need to clarify the paper only deals with the meta linear regression, which simplifies the analysis with a different thread of tools. BTW, the "nested meta-learning" is just the bi-level optimization in meta-learning. Since most meta-learning methods adopt the bi-level form, it is not necessary to introduce a new term.

In addition, the authors need to discuss the gap between the analysis and the practical MAML/iMAML, since the latter methods work on few-shot learning with novel classes. Some discussions on how the analysis can extend to practical cases or which intuitions it provides are welcome.

The difference between the tools used for the analysis in benign overfitting and [1] should be discussed.

[1] Kong et al. Meta-learning for Mixed Linear Regression. ICML 2020.
questions: Please refer to the weakness in the previous parts. 

To better show what it the “benign overfitting" in meta-learning, the authors may consider showing some empirical results of various meta-learning methods at first (even not fully consistent with the assumptions in the paper). Based on the final results of the paper, we can see the benign overfitting in the meta linear regression, but we are not sure whether the same phenomenon will occur in more meta-learning tasks.

Since the analysis is based on the case where the total number of meta-training data is smaller than the number of parameters in the model, which is not always the case in meta-learning, the authors may provide more real examples where we will meet the case. 
limitations: The authors have adequately addressed the limitations and potential negative social impact of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims to answer whether overparameterized nested meta learning models would lead to overfitting or not. As a first step, the paper theoretically analyzes the generalization performance of relatively tractable linear models in the meta linear regression problem with the minimum norm solution. By decomposing excess risk into three individual terms and bounding them separately, the bound for the excess risk can be obtained. Based on the bound, condition for benign overfitting in meta learning is derived. Numerical experiments validated the theoretical findings.
strengths_and_weaknesses: Pros:

+ The paper demonstrated how the analysis of benign overfitting in linear regression can be extended to meta linear regression problems. The analysis provides understanding on how data data heterogeneity, model adaptation affect benign overfitting in meta-learning. The derived bound of the excess risk is verified by showing the same trend as the simulated excess risk in numerical simulations.

+ The paper is well organized. It clearly shows the assumption of the derivations and highlights the key difference between different methods, e.g. Figure 1 and Table 2. The contents in the paper are easy to follow. 

Cons:

- It is not clear how the benign overfitting condition in (13) can be used to determine whether certain settings are benign or not. For example, the parameters M, d in the settings of Example 1 and Example 2 are finite while (13) expresses the condition in limit form. More explanations are needed to demonstrate the significance of the derived benign overfitting condition. 

- The use of "nested meta-learning" is confusing since meta-learning itself indicates a nested structure. Using "nested" before meta-learning may refer to a new method where multiple (more than two) levels of optimization are used. However, the paper actually analyzed MAML and iMAML, which are gradient-based meta-learning methods using two-level optimization.

- There is a large gap between the theoretical analysis and and the meta-learning methods used in practice in terms of the data, model architecture, and prediction task. Although the setting in this paper facilitates theoretical analysis, it would be better to show the possible connection between derived results and real meta-learning applications via discussion or experiments.


--------------------- Post rebuttal --------------------- 
I have read the rebuttal and all my concerns have been addressed. 
questions: - The MAML analyzed in the paper only uses one gradient step in adaptation, while a more realistic scenario is to use multiple gradient descents in the adaptation phase. How does this change affect your analysis? 

- Example 1 and Example 2 show the settings that satisfy the benign overfitting condition. What setting can violate such condition?

- Why larger data heterogeneity makes it more difficult for the benign overfitting condition to be satisfied for both MAML and iMAML?

- What's the connection between the derived results and real meta-learning applications? How can the derived results help understand more practical meta-learning settings?
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provides generalization bounds for overparamterized linear model in meta learning, for three different methods: empirical risk minimization (ERM), model agnostic meta learning (MAML) and implicit model agnostic meta learning (iMAML).

Model. In a meta learning problem, there are M tasks and labeled data $\{(x_{i, m}, y_{i,m})\}_{i \in [N], m \in [M]}$ are drawn i.i.d. from each distribution. The paper considers linear model, where the goal is to come up a share $\theta^{*} \in \mathbb{R}^{d}$ that fits M tasks: For ERM, it just minimizes the square loss; while for MAML and iMAML, it reduces to minimize certain regularized square loss. The paper considers the ``overparameterized'' regime where $d\geq MN$, and the generalization bound explain the the benign overfitting phenomena in certain angle.

Results. It is hard to summarize the generalization in a few words. But it follows the seminal work of [Barlett et al. PNAS 2020], but with an extra term that characterize the cross-task data heterogeneity, which is defined as the maximum discrepancy between data covariance matrix of each task, which is unique to meta-learning or multi-task learning.

Methods. The method generally follows the work of [Barlett et al. PNAS 2020] with extra effort handling the cross-task data heterogeneity, i.e., data coming from different task could have different eigenvector. I feel it is non-trivial to deal with this factor.
strengths_and_weaknesses: Strength. The paper provides the first generalization bound for meta learning under over-parameterized linear model, the proof handles data coming from different source with different covariance matrix (eigenvector especially) is non-trivial to me. The paper is very well-written.

Weakness. There is no major weakness of the paper. Here is one minor question. 

The data assumes overparameterized model for multi-task learning and the benign overfitting somewhat motivates from empirical observation of double descent. Are these valid? E.g. does double descent happen for meta learning, and does the neural network for MAML very over-parameterized when data comes from multi-task are available? It would be good to have some discussion later.

--------------------- Post rebuttal ---------------------
I have read the rebuttal and keep my evaluation.
questions: The paper outline some technical difficulty in the second, and resolves nicely in the proof. Personally I would recommend explicit state how they resolve these issues in their proof outline. One technical question that appears to be non-trivial to me is that how you handle different eigen-vectors from different tasks? It seems the covariance matrix could be quite different (though the eigenvalue is assumed to aligned well). Can you provide some intuition.


Another minor comments is to state that all figures are numerical calculation of the generalization bound in some place, it is bit confusing to me.



limitations: No
review_title: null

Meta-review: metareview: This paper explores the generalization of minimum norm optima for various meta-learning objectives, including basic ERM, model-agnostic meta-learning (MAML) and implicit MAML (iMAML). The generative model considered is "mixed linear regression", in which each of tasks follows a linear + Gaussian noise data model (a different direction per task). The main conceptual takeaway is to tease out how the task heterogeneity affects the generalization bounds (through a "cross variance" quantity), and to compare how much "overfitting" happens for the different objectives. The proof techniques largely follow [Bartlett et al. PNAS 2020] --- due to the linearity/Gaussianity, it boils down to a series of concentration bounds.  Since data coming from different tasks has a different SVD, this makes the proofs non-trivial to extend. 
confidence: Certain
Decision: Accept