Understanding Benign Overfitting in Gradient-Based Meta Learning
Authors: Lisha Chen
Songtao Lu
Tianyi Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: meta learning
benign overfitting
excess risk

Abstract: Meta learning has demonstrated tremendous success in few-shot learning with  limited supervised data. In those settings, the meta model is usually overparameterized. While the conventional statistical learning theory suggests that overparameterized models tend to overfit, empirical evidence reveals that overparameterized meta learning methods still work well -- a phenomenon often called ``benign overfitting.'' To understand this phenomenon, we focus on the meta learning settings with a challenging bilevel structure that we term the gradient-based meta learning, and analyze its generalization performance under an overparameterized meta linear regression model. While our analysis uses the relatively tractable linear models, our theory contributes to understanding the delicate interplay among data heterogeneity, model adaptation and benign overfitting in gradient-based meta learning tasks. We corroborate our theoretical claims through numerical simulations. 

Understanding Benign Overfitting in Gradient-based Meta Learning
Lisha Chen
Songtao Lu IBM Research Yorktown Heights, NY, USA songtao@ibm com
Tianyi Chen
Rensselaer Polytechnic Institute
Rensselaer Polytechnic Institute Troy, NY. USA
Troy, NY, USA edu
chentianyy19@gmai.cco
Abstract
Meta learning has demonstrated tremendous success in few-shot learning with lim- ited supervised data. In those settings, the meta model usually overparameterized. While the conventional statistical learning theory suggests that overparameterized models tend overfit, empirical evidence reveals that overparameterized meta learning methods still work well a phenomenon often called "benign overfitting." To understand this phenomenon, we focus on meta learning settings with a challenging bilevel structure that we term the gradient-based meta learning, and analyze generalization performance under an overparameterized meta linear regression model. While our analysis uses the relatively tractable linear models, our theory contributes to understanding the delicate interplay among data heterogeneity, model adaptation and benign overfitting in gradient- meta learning tasks. We corroborate our theoretical claims through numerical simulations.
Introduction
Meta learning, also referred to as "learning to learn", usually learns a prior model from multiple
tasks so that the learned model able to quickly adapt to unseen tasks 43 26 Meta learning has been successfully applied to few-shot learning learning 2 13,, image recognition 52 federated learning 22], reinforcement learning [2 and communication systems [1O]. While there are many exciting meta learning methods today, in this paper, we will study representative meta learning setting where the goal is to learn a shared initial model that can quickly adapt to task-specific models. This adaptation may take explicit form such the output of one gradient descent step, which is referred to as the model agnostic meta learning (MAML) method [21]. Alternatively, the adaptation step may take an implicit form such the solution of another optimization problem, which is referred to as the implicit MAML (iMAML) method (39]. Since both MAML and iMAML will solve bilevel optimization problem, term them the gradient-base based meta learning thereafter. In many cases, overparameterized models are used as the initial models in meta learning for quick adaptation. For example, Resnet-based based MAML models typically have around 6 million parameters, but trained on 1-3 million meta-training data [12]. Training such initial models is often difficult in meta learning because the number of training data is much smaller than the dimension of the model parameter.
Previous works on meta learning mainly focus on addressing optimization challenges or analyzing the generalization performance with sufficient data [8 19 [4]. Different from these works, are particularly interested in the generalization performance of the sought initial model in practical scenarios where the total number of data from all tasks is smaller than the dimension of the initial model, which we term overparameterized meta learning. In those overparameterized regimes, the generalization error of meta learning models is not fully understood. Motivated by this, ask:
If and when overparameterized meta learning models would lead to overfitting, provably?
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Empirical studies have demonstrated that the MAML with overparameterized model generally performs better than MAML with underparameterized model [ a phenomenon often called "benign overfitting." To show this, we plot in Figure I the empirical results from Table A5 in [12] Complementing this, we take an initial step by answering this theoretical question in the meta linear regression setting.
Prior art 1.1
review prior art that we group in the following three categories.
45
CUB T-bet
Dun S-chet
miillaggphe Lddat
minilmaechic 5-thot
Benign overfitting analysis. The empirical success of overparameterized deep neural networks has inspired theoretical studies of overparameterized learning. The most closest line of work is benign overfitting in linear regression [5] which provides excess risk that mea- sures the difference between expected population risk of the empirical solution and the optimal population risk. Analysis of overparameterizee linear regression model with the minimum- norm solution. concludes that certain data covariance matrices lead to benign overfitting, explaining why overparameterized models that perfectly fit the noisy training data can work well during testing. The analysis has been extended to ridge
Accurscy
66
Com-4
Cono-
Reasze-1
Resse-11
Resve-31
Networks
Figure I: Accuracy vs networks with increas- ing dimensions for MAML on few-shot image classification with different datasets [12
regression 46). multi-class classification [50), and adversarial learning with linear models g While previous theoretical efforts on benign overfitting largely focused on linear models, most recently, the
analysis of benign overfitting has been extended two-layer neural networks 7 33 22 However,
existing works mainly study benign overfitting for empirical risk minimization problems, rather than bilevel problems such gradient-based meta learning, which the focus of this work.
Meta learning. Early works of meta learning build black-box recurrent models that can make pre- dictions based few examples from new tasks 44 26 2 13), or learn shared feature representation among multiple tasks [44] [48] More recently, meta learning approaches aim to find the initialization of model parameters that quickly adapt new tasks with few number of optimization steps such as MAMI. [21
38 41 The empirical success of meta learning has also stimulated recent interests building the theoretical foundation of meta learning methods.
Generalization of meta learning. The excess risk, as metric of generalization ability gradient- based meta learning has been analyzed recently [15] 9 49 19). The generalization of meta learning has been studied in [32) in the context of mixed linear regression, where the focus is on investigating when abundant tasks with small data compensate for lack of tasks with big data. Generalization performance has also been studied in relevant but different setting representation based meta learning [14] [17]. Information theoretical bounds have been proposed in (30 [1], which bound the generalization error in terms of mutual information between the input training data and the output of the meta-learning algorithms. The PAC-Bayes framework has been extended to meta learning to provide a PAC-Bayes meta-population risk bound [1 40 16, 20 These works mostly focus on the where the meta learning model is underparameterized; that is, the total number of meta training data from all tasks is larger than the dimension of the model parameter. Recently, overparameterized meta learning has attracted much attention. Bernacchia (6] suggests that in overparameterized MAML, negative learning rate in the inner loop is optimal during meta training for linear models with Gaussian data. Sun et [455 shows that the optimal representation in representation-bant method of moment estimator. Besides our work, a concurrent work (7 also studies a common setting where the meta learning models incur overparameterization in the meta level, and we both cover the nested MAML method. However, the two studies differ in terms of how the empirical solution of the meta parameter is obtained. In our case, we consider the minimum é-2 norm solution, while [27] consider the solution trained with T-step stochastic gradient descent (SGD). Furthermore, analysis covers both MAML and iMAML, while (27] only considers MAML.
meta learning is overparameterized and provides sample complexity for the

---

Table I: A comparison with closely related prior work on meta learning with linear models. "Reps." and "Gradient" refer to representation based methods and gradient-based methods; "Per-task" refers to the per-task level overparameterization and "Meta" refers to the meta level overparaneterization.
Prior work
Type of meta learning Overparameterization Reps.
Methods
Focus of analysis
Gradient
Per-task Meta
Bai et al.
iMAML
valid-vaiiattion split Optimal step size
Bernacchia 6
MAML
Chen et al. 9
v
V
MAML. BMAML
Test risk comparison
Huang et al. 28
V
MAML
SGD solution
Kong al. 32
Effect of small data tasks
Saunshi et al.
42
Train-validation split
Sun ct al. 45
Optimal representation
Ours
MAML, iMAML
Benign overfitting
Compared the most relevant works, our work is different in the following aspects. Compared the works that also analyze generalization error or sample complexity in linear meta learning models such as [15 we focus on the overparameterized case when the total number of training data is smaller than the dimension of the model parameter. Compared to the work that focus on representation -asee meta learning with a bilinear structure [45], we consider initialization- based meta learning methods with bilevel structure such as MAML and iMAML. Furthermore, we provide tight analysis of the excess risk with explicit consideration of the benign overfitting condition.
summary of key differences compared to prior art is provided in Table [] We distinguish two differ- ent overparameteriiztion settings: the per-task level overparameterizzaion where the dimension of model parameter is larger than the number of training data per task, but smaller than the total number of data across all tasks; and, ii) the meta level overparameterization where the dimension of model parameter is larger than the total number of training data from all tasks.
1.2
This work
This paper provides a unifying analysis of the generalization performance for meta learning problems with overparameterized meta linear models. To our best knowledge, this is the first work that provides the condition for benign overfitting gradient-based meta learning including MAML and iMAML.
Technical challenges. Before we introduce the key result of our paper, we first highlight the challenges of analyzing the generalization of gradient-based meta learning and characterizing its benign overfitting condition, compared to the non-bilevel setting such as in [5] 46 45).
T1) Due to the bilevel structure of gradient-based meta learning, the solution to the meta training objective involves high order terms of data covariance. As a result, the dominating term the excess risk propagated from the label noise contains higher order terms, which is harder to quantify and can potentially lead to orders of magnitude higher excess risk than the linear regression case [5] 46 45).
T2) The existing analysis of benign overfitting in single-level problems 5 46 has a solution that is directly related the data covariance matrix. However, due to the nested structure of gradient-based meta learning and thus the solution matrix, the solution matrix is function of both the data covariance matrix and the hyperparameters such as the step size. Therefore, what kind of data matrices can satisfy the benign overfitting condition cannot be directly implied.
T3) Due to the multi-task learning nature of meta learning, the excess risk of MAML depends on the heterogeneity across different tasks in terms of both the task data covariance and the ground truth task parameter. As a result, the data covariance matrices from different tasks have different eigenvectors. This is in contrast to the linear regression case where all the data follow the same distribution.
Contributions. In view of challenges, our contributions can be summarized follows.
CI) Focusing on the relatively tractable linear models, we derive the excess risk for the minimum- norm solution to overparameterized gradient-based meta learning including MAML and iMAML. Specifically, the excess risk upper bound adopts the following form
Cross-task variance + Per-task variance + Bias
where the cross-task variance quantifies the error caused by finite task number and the variation of the ground truth task specific parameter, which is a unique term compared to
3

---

single task learning. The bias quantifies the bias resulting from the minimum-none norm solution. And the per-task variance quantifies the error caused by noise in the training data.
C2)
We compare the benign overfitting condition for the overparameterizee gradient- based meta learning models and that for the empirical risk minimization (ERM) which learns a single shared parameter for all tasks. We show that overfitting more likely to happen MAML and its variants such as implicit MAML than in ERM. In addition, larger data heterogeneity across tasks will make overfitting more likely happen.
C3)
We discuss the choice of hyperparameter, c.g., the step size in MAML and the weight of the regularizer in iMAML, such that if the data leads to benign overfitting in ERM, it also leads to benign overfitting in MAML and iMAML. We show that negative step size can preserve benign overfitting in MAML. This is complementary to the recent discovery that the optimal step size of overparameterized MAML during training is negative [6]
Problem Formulation and Methods
In this section, we will introduce the problem setup and the considered meta learning methods.
Problem setup. In the meta-learning setting, assume task m is drawn from task distribution, i.e. m ~ M. For each task m., we observe N samples with input feature Zm E X C IR and target label ym. E Vm C R drawn i.idd from a task-specific data distribution Pn. These samples are collected in the dataset Dm {Im,m Vm.n) ) which is divided into the train and validation datasets, denoted as Dir and D m And [Der N1r and [Dwal = Na with N NNr + NNv.. We use the empirical loss (o... Dm) of per-task parameter O.. E +,, as measure of the performance. In this paper, we consider regression problems, where en is defined as the mean squared error.
J
The goal for gradient- based meta learning methods, such as MAML 21 and iMAML (39 is to learn an initial parameter On E Oo. which, with an adaptation method A O X (Xm x )nm) Nex Om. can generate a per-task parameter Om that performs well on the validation data for task Given M tasks, our meta-learning objective is computed as the average the per-task objective, given by
M
Meta training objective
(o D) = M
Σ m=
em (A(Oo, Din), D
(1)
Obtaining the empirical solution 0 by minimizing under a meta learning method A, in the meta testing stage, we evaluate 0g on the population risk, given by
Meta testing objective
R RA(0,)) = Em ED (A(0)) ) D
(2)
Methods. We focus on understanding the generalization of two representative madient-bbased meta learning meth- ods MAML [2I] and iMAML (39 in the overparameterized
Vo,
7
regime. MAML obtains the task-specific parameter én (9)) by taking one step gradient descent with step size a of the per-task loss function f, from the initial parameter Og that is
0g
0₂
MAML
A(Oo, m) = 00 a Va, fm (Oo, D
(3)
On the other hand, iMAML obtains the task-specific param- eter 6, from the initial parameter &g by optimizing the task- specific loss regularized by the distance between Om and to. that is
0
05
02
iMAML
A(lo. D = arg min fy (0, Dtr + 2 - 00l1
(4)
Figure 2: Two types of meta learning.
where 7 > 0 is the weight of the regularizer. As summarized in Figure 2 MAMI. has smaller computation complexity than iMAML since iMAML requires solving an inner problem during adaptation, while iMAMI. may achieve smaller test error since it explicitly minimize the loss.

---

3 Main Results: Benign Overfitting for Gradient-based Meta Learning
In this section, introduce the meta linear regression model and some necessary assumptions for the analysis. We present the main results, highlight the key steps of the proof and conduct simulations to verify our results. Due space limitations, will defer the proofs to the supplementary document.
3.1 Meta linear regression setting
To make a precise analysis, we will assume the following linear data model. Denoting the ground truth parameter on task m 0 E IRd, and the noise as ém. we assume the data model for task m is
Vm
6 Fm + Cm-
(5)
Given the linear model (5. the meta training problem ① with adaptation method 33) or 4 generally have unique solutions when d < NM. However, when the meta model Gg and thus the per-task model fm overparameterized, i.e. d > NM, the training problem (I may have multiple solutions. In the subsequent analysis, we will analyze the performance of the minimum norm solution because recent advances in training overparameterized models reveal that gradient descent-based methods converge to the minimum norm solution [24] 35). We provide a formal definition below.
Definition 1 (Minimum norm solution). Denote X 1m ..... y N.al I R ig.Naxd, yo.
1y..1..... Nmm.N.al Naa)" IT R..... With A(0, Dm) being either (3) or 44, the minimum norm solution to the meta training problem under the linear regression loss is expressed by
min |0ol 80
s.t. Oo Largmin min
X A(0,
(6)
0
m=1
In our analysis, we make the following basic assumptions.
Assumption (Overparameterizzed model). The total number of meta training data is smaller than the dimension of the model parameter; i.e. N M < d.
Assumption (SubGaussian data). The noise em is subGaussian with E[em] and 18(22 I a². For the m-th task, data I'm Vn A Zans where Zm has centered, independent, d.-subGaussian entries; [Z = 0, 10,E,,,2, [m La. with Ia being a d x d identity matrix.
Assumption (Data covariance matrix). 1) Assume for all m E [M], [d] Am.i >> O. Tr(Am), Tr(A) are bounded, iee. for all m E [M), Tr(Am) S Cx- 2) Cross-task data hetero- geneity V[[Q... JMM {M_))= = maxi, [Ax; >mma)//s) is bounded above and below.
Assumption 4 (Task parameter). The ground truth parameter 0*, is independent of Xm and satisfies Cov[Or.] = (R2 /d)), where R is a constant, and the entries of are i.i.d. O(R/Vd) subbaassssss
Assumption N defines the setting that the meta level is overparameterizedd which has also been used
in 45 Note that Assumptions 2 are common in the analysis of meta learning in [5 3
With the linear data model 5 the (minimum norm) solutions to the meta training objective ① and
the meta testing objective (2) can computed analytically which we will summarize next.
Proposition 1. (Empirical and population level solutions) Under the meta linear regression model the meta testing objective of method A (2) can be equivalently written as
(A((8)) = Em [1100 - 8% [[]
where the matrix A. and its empirical version W are given in Table with Qu!l The optimal solutions to the meta-test risk and the minimum-none norm solutions to the empirical meta training loss given below respectively
* x
xal
00- = arg min R RA(0n) = -E,, [WA] m
-1 Em W m
(8a)
0g
0= = arg min L (00, D) (E m=
wA)
M (Emm
W
A
(8b)
An
where denotes the Moore-Penrose pseudo inverse; A is an error term that depends on Xms and specified the supplementary document.
5

---

To study overfitting in the meta learn- ing model, we quantify its generaliza- tion ability via the widely used metric- excess risk. The excess risk of method A (which can be "ma" for MAML and "im" for iMAML), with an empir- ical solution &g and population solu- tion 0g', is defined as
Table 2: Weight matrices under different method A.
Method ERM
Weight matrices
W
Qm
W
Qm
MAML
Wma (I
a
Qm
(I
aQm
m
Wms
m = (I
tr
Q (I
m
iMAML
W mm
(7 Qm
I
Qm Qm
Qm + 1)
E EA(6)) = ( -
- (06))
m
vim M
( Qu
+
Qui
m
In (9, the excess risk measures the difference between the population risk of the empirical solution, d and the optimal population risk. Given total number of training samples MN, if d co, the classic learning theory implies that the excess risk (66") also grows, which leads to overfitting [25].
The larger the excess risk, the further the empirical solution d is from the optimal population solution B,.. indicating more severe overfitting.
3.2
Main results
With the closed-form solutions given in Proposition 1 are ready bound the excess risk of MAML and iMAML in the overparameterized linear regime. For notation brevity, we first introduce some universal constants such as co; C1 C2.-.., and only present the dominating terms in the subsequent results. The precise presentation of remaining terms are deferred to the supplementary document.
We first decompose the excess risk into three terms in Proposition 2
Proposition 2. Define W. = Em [W]]- The excess risk of a meta learning method A can be bounded by
(A(0A) S Eo.. + & + &
(10)
where the first term Eo:, is a function of 0r 0 W.A, W. which quantifies the weighted variance of the ground truth task specific parameters tin the second term &em, as function of Cm. is the weighted noise variance; and the third term Es. function of 0 W., W.A. m. is the bias of the minimum--ree norm solution in overparameterized MAML or iMAML.
Based on this decomposition, as we will show in Section 4 the bound of the excess risk can be derived from the bound of these three terms Eog, EEg, Es, respectively, which gives Theorem [
Theorem (Excess risk bound). Suppose Assumptions 1-4 hold. Let (1(-) 2 j2²(-).. denote the eigenvalues of a matrix in the descending order. For the meta linear regression problem with the minimum-nores norm solution 6. for 0 S l S d, define the effective ranks as
(W-)= Eise (W-).
(ISA
Rs (W = EESHRt (W4)
(1I)
P841 (W4)
With the cross-task data heterogeneity V defined in Assumption 3 if there exist universal con- stants ct C2, 03 >> such that the effective dimension K- = min z O rr.(WA) ' ca NM,, Cg log(1/5) < NM and he*
< NM/c3. then with probability at least 1 - 5, the excess risk satisfies
(WA) MN
k-
MN Ra- (E-(W-)
(1+ 1 ( (((Wabm)))
8 (0()) S I 27
0²
(12)
MN
Theorem Daaeeemmprroo provides the excess risk bound via the effective ranks. In (TT). the effective ranks r and Rs of matrix capture the distribution of the eigenvalues of this matrix, and the effective dimension k* determines the above upper bound by considering the asymmetry of the eigenvalues of the solution matrix. The idea is to choose k- that makes R3- large enough and keeps k. small enough compared to M N so that the variance term the excess risk controlled. For example, ro is the trace normalized by the largest eigenvalue, which is bounded above by Ro- And both ro and Ro are no larger than the rank of the matrix, and they are equal to the rank only when non-zero eigenvalues are equal. If the eigenvalues distribute more uniformly, the effective rank will be larger, otherwise smaller.

---

3,5
BRM
ERM
5
0.01
0.01
2,5
E(
0.2
8(())
10
0.5
1000
0.5
10
20
30
40
50
60
70
80
2V
(a) MAML with different az.
(b) iMAML with different 7
Figure 3: Excess risk number of samples (N) with different hyperparameters (M 10, d 200).
Remark 1. 1) The definition of effective rank has been also given in [S] but only the data matrix
Q. And our setting reduces to the single task ERM learning, or the linear regression case in (5]. when
M 1, 0n =og- W Q, which implies that the cross-task variance in (TO) as well as the data heterogeneity v(-) reduces to zero. Accordingly. Theorem reduces to Theorem 4 in [5].
2) Given Theorem in order to control the excess risk of solution 0z,, we want ro (W-)) to be small compared to the total number of training samples N. but rı- (W-) and Rg- (W-4) to large
compared to M N. In addition, the cross-task heterogeneity V should be small. Since for matrix T.Tx(W) S R2 ()) < d, this suggests the model benefits from overparameterization
Building upon Theorem we now discuss the conditions for "benign overfitting", which refers the situation that overparameterization does not "harm" the excess risk, or the excess risk still vanishes when d > M N and N, M, increase.
Definition (Condition for benign overfitting in meta learning). The weight matrices W- for method A satisfy the benign overfitting condition in gradient- based meta learning, if and only if
To WA) NM
k lim M
NM RE. (WA)
lim N
lim
0.
(13)
NM
This guarantees the excess risk 112) goes to zero in overparameterizze meta learning models with sufficient training data from all tasks. To provide an intuitive explanation, Plot the population risk versus the number of the training data, which demonstrates the "double descent" curve. Namely, as N increases, E(6u) first decreases, then increases and then decreases again, is discovered in overparameterized neural networks [36]. The trend in Figure similar the trend observed in [37] When (M/MM >> 1, the model is overparameterized, which can overfit the training data, leading to larger excess risk as N decreases. However, FigureBsshon the excess risk does not become too large as N decreases, indicating that overfitting does not severely harm the population risk in this case.
3.3 Examples and discussion
In this section, we discuss how the benign overfitting condition (13) in gradient-based meta learning reduces to that in single task linear regression; e.g., in [5] 46). We also provide examples show
Q1) how certain properties of meta training data affect the excess risk; and,
Q2) how choose the hyperparameters that preserve benign overfitting.
Data covariance and cross-task heterogeneity. Theorem D reveals that the excess risk depends on both the cigenvalues of the data covariance matrix Qm. and the cross-task data heterogeneity, measured by V([Q.... { { We give an example below better demonstrate how these two properties of gradient- meta training data affect the excess risk.
TI-d)))))))) Vm. Set M 10, d = 200, d 20, 0.1 for MAML and q 10 for iMAML Then the benign overfitting condition 13 satisfied by MAML and iMAML. We plot the excess risk under different in Figure
Example 1 (Data covariance), Suppose Qm
7

---

0
0.1
0.03
((h))
0.03
3 0.01
0.01
(a) MAML
(b) iMAML
Figure 4: Excess risks number of samples (N) for Qm diag(Ta, (Td, 3T d d; with different B.
9,
E(0)
20
60
70
10
(a) MAML.
iMAML
Figure 5: Excess risks of MAML and iMAML vs number of training samples (N) for Qm [1 + Wm diag(Ia, 3Ia-d, Wm N(0,o with different ow.
From Figure 4 we can observe that given a fixed number of training data N, the population risk increases with 3 for both MAML and iMAML. This observation verifies our theory since larger B results in smaller Re (W-),, leading to larger upper bound on the variance term in (12).
demonstrates how the per-task data matrix Qm affects the excess risk. We consider another example that demonstrates how the data heterogeneity across tasks affects the excess risk.
Example I
Example 2 (Data hetcrogeneity). Suppose Qm |wim + 1/diag (La with Wi N/ for all m. Set M = 10, d = 200, dj = 20, 8 0.3, a = for MAML and 7 = 0.1 for iMAML. Then it satisfies the benign overfitring condition for MAML and iMAML. Figure 5 plots the excess risk with different choices of aw.
Observing from Figure 5 that the larger o the higher the excess risk, and the more difficult for the benign overfitting condition to be satisfied for both MAML and iMAML. Therefore, compared to ERM with single task, the benign overfitting condition for MAML more restrictive as imposes constraints for both the expected data covariance Qm. and the data heterogeneity VAW m M=))
M
Connection to multi-task ERM. To compare benign overfitting in the gradient- based meta learning with that in the conventional ERM, where 8, = 6g, we can set the step size a = in MAML, or 7 DO in iMAMI, and Nm N, which reduces conventional ERM without adaptation.
Compared to that of MAMI. and iMAMI. 13). the benign overfitting condition is less restrictive 7y. Intuitively, benign overfitting is more
for ERM since it does not impose constraints likely to happen in MAML or iMAML than in ERM. The hyperparameters a and will affect the eigenvalues of W With respectively, thus affecting their corresponding excess risk. Here provide a sufficient condition where the benign overfitting condition ERM is preserved MAML
iMAML. We summarize the results in the corollary below.
Corollary (Hyperparameters that preserve benign overfitting). Recall X1 is the largest eigenvalue of Q2 For MAML, when 0 < a < 1. Ix; and for iMAML, when 7 2 X1. then the effective ranks of Wms and Win are bounded above and below by positive constant times the effective rank of Q, and therefore the benign overfitting condition holds for MAML and iMAMI. if it holds for ERM. To summarize, there constants Cı, O2 C32 such that for k* min 2 0 (Q) 2 N NM For

---

ERM
BRM
MAML
MAML
Log
10
20
30
40
60
d/N (M 5,6
M (N 10,d 200,
(a) Cross-task variance ratio model dimension
(b) Bias vs task number
Figure 6: Cross-task variance and bias versus task number to elaborate Lemma [ and Lemma
< 1, 02 (1/6) < NM and k < KNM/c Ca. with probability at least 7 2N22/ it follows
MN R&- )
(A() x
E E
ro
o²
k-
MN
1+ (1+V((Q m)mm)))))))))))m)m))
(14)
MN
Remark 2. For MAML, let the unordered eigenvalues jú, (W"") = A, - a. One challenge to control f.:( (Wis)) is that jū,( (Wma) are not necessarily monotonic w.r.t. X; that is, does not necessarily hold that ji 2 f02 2 the order of the eigenvalues preserved. For this to hold, requires (A,(A, to be a monotonically non-decreasing function of A, which yields S 3X Similar results can be obtained for iMAML by
jād. For any X; 2 Az. if fı; (W-)- 2 (i,((' (W~), then we say
controlling the value of y. And the bound on a or y further ensures that (u: (W-) is bounded above and below by positive constant times the effective rank of Q.
4 Proof Outline
In this section, we highlight the key steps of the proof for Theorem 1 We achieve so by analyzing the three terms in Proposition @respectively
The first terms in (00) can be bounded based on the concentration inequalities on subGaussian
variables, given in Lemmas [ and
Lemma (Bound on cross-task variance). With probability least 5, it follows
(
M
M
WO - 0.9)
S Ô 10(1) Eem
Egpt.
W m
(15)
m=11
WA
where Ō(-) hides the log polynomial dependence on N, M,
The cross-task variance term analzyed Lemma is unique in meta learning, which captures the data heterogeneity across different tasks. To claborate Lemma I we plot the cross-task variance versus the task number in Figure 6a with task number M 5, training validation split parameter
= N 0.5, per-task data number N = 10. This figure demonstrates that the ratio of cross-task variance and per-task variance decreases with d/N, which is consistent with Lemma
Lemma 2 (Bound bias). For any log( (1)) < MN with probability at least 1 1 8, we have
ro (WA)
ro (W
log(1/8)
& < 1 max
(16)
M
MN
M Nv
This term similar to the bias term in the linear regression case, but directly depending on the solution matrix W instead of the data matrix Q. To elaborate Lemma 2 Figure 6b demonstrates that the bias term decays with until reaches zero when the model is underparameterizee. These two terms in Lemma and Lemma do go infinity as N, M,, d increase.
Note that, the key step the bound on E which is the dominating term the decomposition of excess risk 10 in the overparameterized regime. We will bound it below.

---

Lemma 3 (Bound on per-task variance). There exist constants e1, C2. such that for 0 S 2/M/a, Tu( (WA) 2 e, and ko S k, with probability at least 1 - Te T2 it follows
ko
MN
) (1+ V([ mJm=1))
e
(17)
Ma
RRo(W-4
Note that, in the single task linear regression case, the there is cross-task data heterogeneity, i.c., = 0. This term is unique in the meta learning setting with multiple tasks. Plugging the results of Lemmas and into (10), we will reach Theorem
Conclusions and Limitations
This paper studies the generalization performance of the gradient-based meta learning with an overparameterized model. For a precise analysis, we focus on linear models where the total number of data from all tasks is smaller than the dimension of the model parameter. We show that when the data heterogeneity across tasks is relatively small, the per-task data covariance matrices with certain properties lead benign overfitting for gradient-base solution. This explains why overparameterized meta learning models can generalize well in new data and new tasks. Furthermore, our theory shows that overfitting is more likely to happen in meta learning than in ERM, especially when the data heterogeneity across tasks is relatively high.
meta learning with the minimum-none norm
One limitation of this work is that the analysis focuses on the meta línear regression case. While this analysis can capture practical cases where we reuse the feature extractor from pre-trained models and only meta-train the parameters in the last linear layer, it also promising to extend our analysis to nonlinear cases via means of random features and neural tangent kernels in the future work.
Acknowledgments
This work was partially supported by National Science Foundation MoDL-SCALE Grant 2134168 and the Rensselaer- IBM Al Research Collaboration (http://airc.ry) edu), part of the IBM AI Horizons Network (http://ibm. biz/AlHorizons)
References
[1] Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended pac-bayes theory. In Proc. International Conference on Machine Learning, pages 205 204- Stockholm, Sweden, 2018.
[2] Marein Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Proc. Advances in Neural Information Processing Systems, pages 3981-3998 Barcelona, Spain, 2016.
[3] Yu Bai, Minshuo Chen, Pan Zhou, Tuo Zhao, Jason Lee, Sham Kakade, Huan Wang, and Caiming Xiong- How important the validaaiiot split in meta-learning? In Proc. International Conference on Machine Learning, pages 543 553, virtual, 2021.
Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradient-based based [4] meta-lcaming. In Proc. International Conference Machine Learning, pages 424-433 Long Beach, CA,, 2019.
[5] Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117745)300033330070 2020.
[6] Alberto Bemacchia. Meta-lcarning with negative learning rates. Proc. International Conference on Learning Representations, virtual, 2020.
[7] Yuan Cao, Zixiang Chen, Mikhail Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. arXiv preprint w:Xiv:2202.06522 2022.
[8] Jinghui Chen, Yuan Cao, and Quanquan Gu. Benign overfitting in adversarially robust linear classification. arXiv preprint arXiv:2112.15250 2021.
10

---

[9] Lisha Chen and Tianyi Chen. Is bayesian model agnostic meta learning better than model agnostic meta learning, provably? In Proceedings of The 25th International Conference on Artificial Intelligence and Statisticx, 2022.
[10] Lisha Chen, Sharu Theresa Jose, Ivana Nikoloska, Sangwoo Park, Tianyi Chen, and Osvaldo Sime- one. Learning with limited samples-metta-eamingg and applications to communication systems. arXiv preprint:2210.025555 October 2022.
[11] Qi Chen, Changjian Shui, and Mario Marchand. Generalization bounds for meta-lcaming: An information- theoretic analysis. In Proc. Advances in Neural Information Processing Systems, virtual, 2021.
[12] Wei-Yu Yu Chen, Yen-Cheng Liu,, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. Proc. International Conference on Learning Representations, Vancouver, Canada, 2018.
[13] Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matt Botvinick, and Nando Freitas. Learning to learn without gradient descent by gradient descent. In Proc. International Conference on Machine Learning, pages 748-756, Sydney, Australia, 2017.
[14] Kurtland Chua, Lei, and Jason Lec. How fine-tuning allows effective meta- neta-learning, Proc. Advances in Neural Information Processing Systems, volume 34, 2021.
[15] Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learn around a common mean. In Pruc. Advances in Neural Information Processing Systems, volume 31, Montreal, Canada, 2018.
[16] Nan Ding, Xi Chen, Tomer Levinboim, Sebastian Goodman, and Radu Soricut. Bridging the gap between practice and pac-bayes theory few-shot meta-learning. Proc. Advances in Neural Information Processing Systems, virtual, 2021.
[17] Simon Shaolei Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2020.
[18] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- model-agnostic meta-learning algorithms. In Proc. International Conference on Artificial Intelligence and Statistics, pages 1082-1092, virtual, 2020.
[19] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Generalization of model-agnostic heta-lcarming algorithms: Recurring and unseen tasks. In Proc. Advances in Neural Information Processing Systems, virtual, 2021.
[20] Alec Farid and Anirudha Majumdar. Generalization bounds for meta-learning via pac-bap pac-bayes and uniform stability. In Proc. Advances in Neural Information Processing Systems, virtual, 2021.
[21] Chelsea Finn. Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proc. International Conference on Machine Learning, page 1136-1135, Sydney, Australia, 2017.
[22] Spenoer Frei, Niladri S Chatterji, and Peter L Bartlett. Benign overfitting without lincarity: Neural network classifiers trained by gradient descent for noisy linear data. arXiv preprint arXiv:2202.0522 2022.
[23] Katelyn Gao and Ozan Sener. Modeling and optimization trade-off in meta-learning. In Proc. Advances in Neural Information Processing Systems, volume 33, virtual, 2020.
[24] Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Proc. International Conference on Machine Learning, volume 80, pages 1832-1841, Stockholm, Sweden, 10-15 Jul 2018.
[25] Trevor Hastie, Robert Tihshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[26] Sepp Hochreiter, Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In Proc. International Conference on Artificial Neural Networks, pages 87-94, Vienna, Austria, 2001.
[27] Yu Huang, Yingbin Liang, and Longbo Huang. Provable generalization of overparameterized meta- learning trained with SGD. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA, December 2022.

---

[28] Yu Huang, Yingbin Liang, and Longbo Huang. Provable generalization of overparameterized meta-learning trained with sgd. In Proc. Advances in Nearal Information Processing Systems, New Orleans, LA, 2022.
[29] Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. arXiv 0veprint:100..1244 12488, September 2019.
[30] Sharu Theresa Jose and Osvaldo Simeone. Information-theeortiic generalization bounds for meta-learning and applications. Entropy, 23(1))126, 2021.
[31] Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, 23(1):110-13,, 2017.
[32] Weihao Kong, Raghav Somani, Zhao Song, Sham Kakade, and Sewoong Oh. Meta-learning for mixed linear regression. In Proc. International Conference on Machine Learning, pages 5394-5404, virtual, 2020.
[33] Zhu Li, Zhi-Hua Zhou, and Arthur Gretton. Towards an understanding of benign overfitting in neural networks. arXiv preprint urXiv:2106.03211 12.222200.22222222 2021.
[34] L. Mirsky. A trace incquality of john von neumann. Monatshefte für Mathemarik, 79:303-306, 1975.
[35] Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data regression. IEEE Journal on Selected Areas in Information Theory, 1(1):67 83, 2020.
[36] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Iya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, (021(12):12400)) 2021.
[37] Preetum Nakkiran, Prayaag Venkat, Sham Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. In Proc. International Conference on Learning Representations, virtual, 2020.
[38] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-lcaming algorithms. arXiv preprint A8Xiv:8803.02800 2018.
[39] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Proc. Advances in Neural Information Processing Systems, pages 113-124, Vancouver, Canada, 2019.
[40] Jonas Rothfuss, Vincent Fortuin, Martin Josifoski, and Andreas Krause. PACOH: optimmll meta- learning with muc-guarantees. In Proc. International Conference on Machine Learning, pages 91 1116-1 1926, virtual, 2021.
[41] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Picter Abbeel. Promp: policy search. In Proc. International Conference Learning Representations, Vancouver, Canada, 2018.
Proximal meta-
[42] Nikunj Saunshi, Arushi Gupta, and Wei Hu. A representation learning perspective the importance of train-validation splitting in meta-leariing In International Conference on Machine Learning, pages 9333-9343, 2021.
[43] Schmidhuber. A neural network that embeds own meta-levels. Proc. IEEE International Conference on Neural Nenworks, pages 407-412 vol.1, 1993.
[44] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Proc. Advances in Neural Information Processing Systems, pages 4080-4090, Long Beach, CA, 2017.
[45] Yue Sun. Adhyyan Narang. Halil Ibrahim Gulluk, Samet Oymak, and Maryam Fazel. Towards sample- efficient overparameterized meta-learning. In A. Beygelzimer. Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Proc. Advances in Neural Information Processing Systems, virtual, 2021.
[46] Alexander Tsigler and Peter L. Bartlett, Benign overfitting in ridge regression. arXiv: Statistics Theory, 2020.
[47] Roman Vershynin. Diggessnims Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018
[48] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In Proc. Advances in Neural Information Processing Systems, volume 29, pages 3630-3638, Barcelona, Spain, 2016.
12

---

[49] Haoxiang Wang. Ruoyu Sun, and Bo Li. Global convergence and generalization bound of gradient-based meta-learning with deep neural nets. arXiv preprini arXiv:200.. 14606, 2020.
[50] Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign overfitting in multiclass classification: All roads lead to interpolation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Proc. Advances in Neural Information Processing Systems, virtual, 2021.
[51] Hermann Weyl. Das asymptotische verteilungsgesetr der eigenwerte linearer partieller differentialgleichun- gen (mit einer anwendung auf die theorie der hohlraumstrahlung). Mathematische Annalen, 71(4):441-473 1912.
[52] Barret Zoph. Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. Proc. IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8697-8710, Salt Lake City, UT, June 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Work of theoretical nature, no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Supplementary material for the complete proofs.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URLY? [N/A] Work of theoretical nature.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A] Work of theoretical nature.
cite the authors who propose the baseline models.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vNAv6jjZVN7", "reviewer": "Reviewer_TRZn", "summary": "This paper explores the generalization of overparameterized meta-learning models with a nested structure, specifically model-agnostic meta-learning (MAML) and implicit MAML (iMAML). Under some theoretical assumptions, the authors derive an upper bound for the excess risk of MAML and iMAML in the overparameterized setting. Further, they derive conditions for benign overfitting in nested meta-learning and provide hyperparameters for MAML and iMAML that preserve the benign overfitting.", "questions": "Is there an intuitive way to explain why the cross-task variance upper bound primarily depends on 1/M asymptotically?", "limitations": "The authors adequately address the limitations of their work. Specifically, they acknowledge that their current analysis is limited to the non-Bayesian setting and the linear model assumption.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper does a good job of explaining the motivations for studying the problem. In addition, the authors present their contributions clearly and provide a proof outline of their main result, which contributes to the clarity of the paper.\n\nThe main weakness of the paper is in the discussion of the experimental results. I believe that the significance of the theoretical results could be further emphasized by explaining the experimental results in more depth (and adding some numerical results in a tabular format). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QDx8QlCs77A", "reviewer": "Reviewer_FrXq", "summary": "The paper provides generalization bounds for overparamterized linear model in meta learning, for three different methods: empirical risk minimization (ERM), model agnostic meta learning (MAML) and implicit model agnostic meta learning (iMAML).\n\nModel. In a meta learning problem, there are M tasks and labeled data $\\{(x_{i, m}, y_{i,m})\\}_{i \\in [N], m \\in [M]}$ are drawn i.i.d. from each distribution. The paper considers linear model, where the goal is to come up a share $\\theta^{*} \\in \\mathbb{R}^{d}$ that fits M tasks: For ERM, it just minimizes the square loss; while for MAML and iMAML, it reduces to minimize certain regularized square loss. The paper considers the ``overparameterized'' regime where $d\\geq MN$, and the generalization bound explain the the benign overfitting phenomena in certain angle.\n\nResults. It is hard to summarize the generalization in a few words. But it follows the seminal work of [Barlett et al. PNAS 2020], but with an extra term that characterize the cross-task data heterogeneity, which is defined as the maximum discrepancy between data covariance matrix of each task, which is unique to meta-learning or multi-task learning.\n\nMethods. The method generally follows the work of [Barlett et al. PNAS 2020] with extra effort handling the cross-task data heterogeneity, i.e., data coming from different task could have different eigenvector. I feel it is non-trivial to deal with this factor.", "questions": "The paper outline some technical difficulty in the second, and resolves nicely in the proof. Personally I would recommend explicit state how they resolve these issues in their proof outline. One technical question that appears to be non-trivial to me is that how you handle different eigen-vectors from different tasks? It seems the covariance matrix could be quite different (though the eigenvalue is assumed to aligned well). Can you provide some intuition.\n\n\nAnother minor comments is to state that all figures are numerical calculation of the generalization bound in some place, it is bit confusing to me.\n\n\n", "limitations": "No", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength. The paper provides the first generalization bound for meta learning under over-parameterized linear model, the proof handles data coming from different source with different covariance matrix (eigenvector especially) is non-trivial to me. The paper is very well-written.\n\nWeakness. There is no major weakness of the paper. Here is one minor question. \n\nThe data assumes overparameterized model for multi-task learning and the benign overfitting somewhat motivates from empirical observation of double descent. Are these valid? E.g. does double descent happen for meta learning, and does the neural network for MAML very over-parameterized when data comes from multi-task are available? It would be good to have some discussion later.\n\n--------------------- Post rebuttal ---------------------\nI have read the rebuttal and keep my evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IbhBUT8m8F1", "reviewer": "Reviewer_WJZn", "summary": "The paper aims to answer whether overparameterized nested meta learning models would lead to overfitting or not. As a first step, the paper theoretically analyzes the generalization performance of relatively tractable linear models in the meta linear regression problem with the minimum norm solution. By decomposing excess risk into three individual terms and bounding them separately, the bound for the excess risk can be obtained. Based on the bound, condition for benign overfitting in meta learning is derived. Numerical experiments validated the theoretical findings.", "questions": "- The MAML analyzed in the paper only uses one gradient step in adaptation, while a more realistic scenario is to use multiple gradient descents in the adaptation phase. How does this change affect your analysis? \n\n- Example 1 and Example 2 show the settings that satisfy the benign overfitting condition. What setting can violate such condition?\n\n- Why larger data heterogeneity makes it more difficult for the benign overfitting condition to be satisfied for both MAML and iMAML?\n\n- What's the connection between the derived results and real meta-learning applications? How can the derived results help understand more practical meta-learning settings?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n+ The paper demonstrated how the analysis of benign overfitting in linear regression can be extended to meta linear regression problems. The analysis provides understanding on how data data heterogeneity, model adaptation affect benign overfitting in meta-learning. The derived bound of the excess risk is verified by showing the same trend as the simulated excess risk in numerical simulations.\n\n+ The paper is well organized. It clearly shows the assumption of the derivations and highlights the key difference between different methods, e.g. Figure 1 and Table 2. The contents in the paper are easy to follow. \n\nCons:\n\n- It is not clear how the benign overfitting condition in (13) can be used to determine whether certain settings are benign or not. For example, the parameters M, d in the settings of Example 1 and Example 2 are finite while (13) expresses the condition in limit form. More explanations are needed to demonstrate the significance of the derived benign overfitting condition. \n\n- The use of \"nested meta-learning\" is confusing since meta-learning itself indicates a nested structure. Using \"nested\" before meta-learning may refer to a new method where multiple (more than two) levels of optimization are used. However, the paper actually analyzed MAML and iMAML, which are gradient-based meta-learning methods using two-level optimization.\n\n- There is a large gap between the theoretical analysis and and the meta-learning methods used in practice in terms of the data, model architecture, and prediction task. Although the setting in this paper facilitates theoretical analysis, it would be better to show the possible connection between derived results and real meta-learning applications via discussion or experiments.\n\n\n--------------------- Post rebuttal --------------------- \nI have read the rebuttal and all my concerns have been addressed. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GColPEnaGjq", "reviewer": "Reviewer_sqGF", "summary": "This paper analyzes the “benign overfitting” phenomenon of nested meta-learning, where an over-parameterized model can still work well in the meta linear regression setting. Both MAML and iMAML are covered by the analyses. Numerical simulations validate the results of the theoretical analysis.\n", "questions": "Please refer to the weakness in the previous parts. \n\nTo better show what it the “benign overfitting\" in meta-learning, the authors may consider showing some empirical results of various meta-learning methods at first (even not fully consistent with the assumptions in the paper). Based on the final results of the paper, we can see the benign overfitting in the meta linear regression, but we are not sure whether the same phenomenon will occur in more meta-learning tasks.\n\nSince the analysis is based on the case where the total number of meta-training data is smaller than the number of parameters in the model, which is not always the case in meta-learning, the authors may provide more real examples where we will meet the case. ", "limitations": "The authors have adequately addressed the limitations and potential negative social impact of the work.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper analyzes an interesting phenomenon in meta-learning in the meta linear regression setting. The authors need to clarify the paper only deals with the meta linear regression, which simplifies the analysis with a different thread of tools. BTW, the \"nested meta-learning\" is just the bi-level optimization in meta-learning. Since most meta-learning methods adopt the bi-level form, it is not necessary to introduce a new term.\n\nIn addition, the authors need to discuss the gap between the analysis and the practical MAML/iMAML, since the latter methods work on few-shot learning with novel classes. Some discussions on how the analysis can extend to practical cases or which intuitions it provides are welcome.\n\nThe difference between the tools used for the analysis in benign overfitting and [1] should be discussed.\n\n[1] Kong et al. Meta-learning for Mixed Linear Regression. ICML 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept