Meta-learning with negative learning rates
Authors: Alberto Bernacchia
Venue: ICLR 2021 Poster (2021)
Keywords: Meta-learning

Abstract: Deep learning models require a large amount of data to perform well. When data is scarce for a target task, we can transfer the knowledge gained by training on similar tasks to quickly learn the target. A successful approach is meta-learning, or "learning to learn" a distribution of tasks, where "learning" is represented by an outer loop, and "to learn" by an inner loop of gradient descent. However, a number of recent empirical studies argue that the inner loop is unnecessary and more simple models work equally well or even better. We study the performance of MAML as a function of the learning rate of the inner loop, where zero learning rate implies that there is no inner loop. Using random matrix theory and exact solutions of linear models, we calculate an algebraic expression for the test loss of MAML applied to mixed linear regression and nonlinear regression with overparameterized models. Surprisingly, while the optimal learning rate for adaptation is positive, we find that the optimal learning rate for training is always negative, a setting that has never been considered before. Therefore, not only does the performance increase by decreasing the learning rate to zero, as suggested by recent work, but it can be increased even further by decreasing the learning rate to negative
values. These results help clarify under what circumstances meta-learning performs best.

Published as a conference paper at ICLR 2021
META-LEARNING WITH NEGATIVE LEARNING RATES
Alberto Bernacchia
MediaTTek Research
a alberto... www.eaachh..acctkeareeeaaaa.a...
ABSTRACT
Deep learning models require large amount of data to perform well. When data is scarce for a target task, we can transfer the knowledge gained by training on similar tasks quickly learn the target. A successful approach is meta-learning, or learning to learn a distribution of tasks, where learning is represented by an outer loop, and to learn by an inner loop of gradient descent. However, num- ber of recent empirical studies argue that the inner loop is unnecessary and more simple models work equally well even better. We study the performance of MAML as function of the learning rate of the inner loop, where zero learning rate implies that there is no inner loop. Using random matrix theory and exact solutions of linear models, we calculate an algebraic expression for the test loss of MAML applied to mixed linear regression and nonlinear regression with overpa- rameterized models. Surprisingly, while the optimal learning rate for adaptation is positive, we find that the optimal learning rate for training is always negative, setting that has never been considered before. Therefore, not only does the per- formance increase by decreasing the learning rate to zero, as suggested by recent work, but it can be increased even further by decreasing the learning rate to neg- ative values. These results help clarify under what circumstances meta-learning performs best.
INTRODUCTION
Deep Learning models represent the state-of-the-art in several machine learning benchmarks (Le- Cun et al. (2015), and their performance does not seem to stop improving when adding more data and computing resources (Rosenfeld et al.] (2020), Kaplann al. (2020)). However, they require large amount of data and compute to start with, which are often not available to practitioners. The approach of fine-tuning has proved very effective address this limitation: pre-train model source task, for which a large dataset is available, and use this model as the starting point for a quick additional training (fine-tuning) on the small dataset of the target task (Pan & Yang (2010), Donahuel et al. (2014). Yosinski et al. (2014)) This approach is popular because pre-trained models are often made available by institutions that have the resources to train them.
In some circumstances, multiple source tasks are available, all of which have scarce data, as opposed single source task with abundant data. This case is addressed by rett-learnnng, in which a model gains experience over multiple source tasks and uses it to improve its learning of future target tasks. The idea of meta-learning inspired by the ability of humans to generalize across tasks, without having to train on any single task for long time. A meta veea-leernnngg problem is solved by a bi-level optimization procedure: an outer loop optimizes meta- veeta-parameters across tasks, while an inner loop optimizes parameters within each task (Hospedales et al. (2020))
The idea of meta-learning has gained some popularity, but a few recent papers argue that a simple alternative to meta-learning is just good enough, in which the inner loop is removed entirely (Chen) cet al. (2020a). Tian et al. (2020), Dhillon et al. (2020), Chen et al. (2020b). Raghu et al. 2222 ). Other studies find the opposite (Goldblum et al. 2020). Collins et al. (2020). Gao & Sener 2020 is hard to resolve the debate because there is little theory available to explain these findings.
In this work, using random matrix theory and exact solutions of linear models, we derive algebraic expression of the average test loss of MAML, a simple and successful meta-learning algorithm (Finn) et al. (0077), ) function of its hyperparameters. In particular, study its performance

---

Published as a conference paper at ICLR 2021
function of the inner loop learning rate during meta-training. Setting this learning rate to zero is equivalent to removing the inner loop, as advocated by recent work (Chen et al. (2020a). Tian et al. (2020), Dhillon et al. (2020). Chen et al. (2020b), Raghu et al. (2020)) Surprisingly, we find that the optimal learning rate is negative, thus performance can be increased by reducing the learning rate below zero. In particular, we find the following:
In the problem of mixed linear regression, we prove that the optimal learning rate is always negative in overparameterized models. The same result holds in underparameterized mod- els provided that the optimal learning rate is small in absolute value. We validate the theory by running extensive experiments.
We extend these results to the case of nonlinear regression and wide neural networks, in which the output can be approximated by a linear function of the parameters (Jacot et al. (2018), Lee et al. (2019)). While in this case we cannot prove that the optimal learning rate always negative, preliminary experiments suggest that the result holds in this case as well.
2 RELATED WORK
The field of meta-learning includes broad range of problems and solutions, see Hoospdalles et al. (2020) for a recent review focusing on neural networks and deep learning. In this context, meta- learning received increased attention in the past few years, several new benchmarks have been in- troduced, and a large number of algorithms and models have been proposed to solve them (Vinyals) al. (2017). Bertinetto et al. (2019), Triantafillou al. (2020)). Despite the surge in empirical work, theoretical work is still lagging behind.
Similar to our work, a few other studies used random matrix theory and exact solutions to calculate the average test loss for the problem of linear regression (Advani & Saxe (2017), Hastie et al. (2019), Nakkiran ).. To our knowledge, our study the first to apply this technique the problem of meta-lcarning with multiple tasks. Our results reduce to those of linear regression in the case of one single task. Furthermore, we are among the first to apply the framework of Neural Tangent Kernel (Jacot et al. (2018). Lee et al. (2019)) to the problem of meta-learning (a few papers appeared after our submission: Yang Hu 2020), Wang et al. (2020a), Zhou et al. 202T)).
Similar to few theoretical studies looked at the problem of mixed linear regression in the context of meta-learning. In Denevi et al. (2018), Bai et al. (202T), meta-parameter is used to bias the task- specific parameters through a regularization term. Kong et al. (2020) looks whether many tasks with small data can compensate for lack of tasks with big data. Tripuraneni et al. (2020), Du et al. (020)) study the sample complexity of representation learning. However, none of these studies look into the effect learning rate on performance, which is our main focus.
In this work, focus MAML, simple and successful meta-learning algorithm (Finn et al. (017)). A few theoretical studies have investigated MAML, looking at: universality of the opti- mization algorithm (Finn & Levine (208)), bayesian inference interpretation (Grant et al. (2088) ) proof of convergence (Ji et al.] (2020). difference between convex and non-convex losses (Saunshi al. (2020)), global optimality (Wang et al. (2020b)), effect of the inner loop (Collins et (2020),
Gao & Sener (020 Again, none of these studies look at the effect of the learning rate, the main subject of our work. The theoretical work Khodak et al. (2019) connects the learning rate to task similarity, while the work of et al. (2017) meta-learns the learning rate.
3 META-LEARNING AND MAML
In this work, we follow the notation of Hospedales al. (2020) and we MAML [Finn et al. (017 as the meta- leaa-in algorithm. We assume the existence of distribution of tasks 7 and, for each task, loss function L and distribution of data points D {Ir y²' with input x= and label y". We assume that the loss function is the same for all tasks, L = L, but each task is characterized by different distribution of the data. The empirical meta-learning loss is evaluated on sample of
2

---

Published as a conference paper at ICLR 2021
tasks, and sample of n, validation data points for each task:
Emeta (w;
c (0(w: )z m()) )0)
(I)
1=1
The training set D
{x,) 1()) (())
and validation set D.S") = {0) (1 (1)
(1) .
are
f=1:me
j=1m
drawn independently from the same distribution each task i. The function represents the adap- tation of the meta- peramparreeter w. which is evaluated the training set. Different meta-learning algorithms correspond to a different choice of 0, we describe below the choice of MAML (Eq(). the subject of this study. During meta-training, the loss of [q is optimized with respect to the meta-parameter w, usually by stochastic gradient descent, starting from an initial point wi- The optimum is denoted w (Dr, D.). This optimization referred to the outer loop, while com- putation of is referred to as the inner loop of meta-learning. During meta-testing, a new (target) task given and 0 adapts on set D, of n, target data points. The final performance of the model is computed on test data D, of the target task. Therefore, the test loss is equal to
grest = m (w> D.):
(2)
In MAML, the inner loop corresponds to few steps of gradient descent, with given learning rate Or-- In this work we consider the simple case of a single gradient step:
(ww; D(
at
ac
T4
Waa 1( (())
./3
If the learning rate ar is zero, then parameters are not adapted during meta-training and ((w) = w. In that case, single set of parameters in learned across all data and there no inner loop. However, it is important to note that a distinct learning rate a, is used during meta-testing. A setting similar this has been advocated in a few recent studies (Chen et al. (2020a). Tian et (2020). Dhillon et al. (2020), Chen et al. (2020)) Raghu et al. (020))
We show that, intuitively. the optimal learning rate meta-testing (adaptation) time a, is always positive. Surprisingly, the family of problems considered in this study, we find that the optimal learning rate during meta-training ag is instead negative. We note that the setting a 0 effectively does not the ny training data points, therefore we could in princi- ple add this data to the validation set, but do not consider this option here since we are interested in a wide range of possible values of a, as opposed the specific case a₁ 0.
W
y
4 MIXED LINEAR REGRESSION
M
P(
x
training
validation 1.....y
target
test
We study MAML applied to the problem of mixed linear regression. Note that the goal here is not to solve the problem of mixed linear regression, but probe the performance of MAML function of its hyperparameters.
= 1.....
1......
meta-training 1....
meta-testing
Figure 1: Graphical model of data generation in mixed linear regression
In mixed linear regression, each task is character- ized by a different linear function, and a model evaluated by the mean squared error loss function. We assume a generative model the form of y x"'w
w + 2, where x is the input vector (of dimension
p). is the output (scalar), 2 is noise (scalar), and w vector of generating parameters (of dimension p), therefore p represents both the number of parameters and the input dimension. All distributions are assumed Gaussian:
22 Wo: - [T
w ~ N
x~ C~ (p)
v|x,w~
w ~ (X.W~((XTw,o wx))
(4)
p
3

---

Published as a conference paper at ICLR 2021
where I, is the pxp p identity matrix, is the label noise, Wo is the task mean and represents the task variability. Different meta-training tasks i correspond to different draws of generating parameters w(1, while the parameters for the meta-testing task are denoted by w'. We denote by superscripts v, r, the training, validation, target and test data, respectively. A graphical model of data generation is shown in Figure
Using random matrix theory and exact solutions of linear models, we calculate the test loss func- tion of the following hyperparameters: the number of training tasks m, number of data points per task for training (nz), validation (nz)) and target (n,,), learning rate for training Qt and for adaptation to target o Furthermore, we have the hyperparameters specific to the mixed linear regression prob- lem: p. v, Wo. Since we exact solutions to the linear problem, our approach is equivalent to running the outer loop optimization until convergence (see section 77.1inn the Appendix for details). We derive results in two cases: overparameterized n.m and underparameterized p < Nom..
RESULTS
5.1
OVERPARAMETERIIZE CASE
In the overparameterized case, the number of parameters p is larger than the total number of valida- tion data across tasks m. In this case, since the data does not fully constrain the parameters, the optimal value of w found during meta-training depends the initial condition used for optimiza- tion, which we call Wo-
Theorem 1. Consider the algorithm of section (MAML one-step), and the data generating model of section 4 (mixed linear regression). Let p m0 Let p(E) and ne(E) be any function of order OEE) as & oc. Let Wwo wal be of order Ogg-1/4). Then the test loss of Eq averaged over the entire data distribution (see Eq 27 in the Appendix) is equal to
a²
jest
1
+
2
14
oip 1 1
+ h"
v2
1
+
1
Wo -ol
wol?
0² n.n
+
(5558/2)
(5)
2
2p
ht
where we define the following expressions
ht (1 (1-a))2+ aj
2p -PP1
(6)
1ht
h" = (1 (1-a,)) )²)²
2 + ar
2
nr
Proof. The proof of this Theorem can be found in the Appendix, sections
7.3
7.3.1
The loss always increases with the output noise and task variability v. Overfitting expressed in 5 by the term (w0 wol. the distance between the initial condition for the optimization of wo and the ground truth mean of the generating model wa- Adding more validation data n, and tasks m may increase or decrease the loss depending on the size of this term relative to the noise (Nakkiran (2019), as does reducing the number of parameters p. However, the loss always decreases with the number of data points for the target task 7),, as that data only affects the adaptation step.
Our main focus is studying how the loss affected by the learning rates, during training o and adaptation O,, The loss is quadratic and convex function of a,, therefore it has a unique minimum. While it is possible to compute the optimal value of a, from 5q here we just note that the loss is sum of two quadratic functions, one has minimum at ar, 0 and another has a minimum at a, = (1+ (p + 1)/n,).. therefore the optimal learning rate is in between the two values and is always positive. This is intuitive, since positive learning rate for adaptation implies that the parameters get closer to the optimum for the target task. An example of the loss as a function of the adaptation learning rate a, is shown in Figure 2h. where also show the results of experiments
4

---

Published as a conference paper at ICLR 2021
in which run MAML empirically. The good agreement between theory and experiment suggest that Eq is accurate.
However, the training learning rate a shows the opposite: by taking the derivative of Eq with respect to ag, it is possible to show that it has a unique absolute minimum for negative value of aq- This can be proved by noting that this function has the same finite value for large positive negative Or, its derivative is always positive at aa = 0, and it has one minimum (-) and one maximum (+)
values
# 10
2p + V
+
2 +
T4e
(8)
2p
p
Note that the argmax or is always positive, while the argmin o, is always negative. This result counter-intuitive, since a negative learning rate pushes parameters towards higher values of the loss. However, learning of the meta-parameter is performed by the outer loop (minimize Eq[], for which there no learning rate since we are using the exact solution to the linear problem and thus are effectively training to convergence. Therefore, it remains unclear whether the inner loop (3q(3) should push parameters towards higher or lower values of the loss. An example of the loss function of the training learning rate a, is shown in Figure Pp where also show the results of experiments in which we run MAML empirically. Here the theory slightly underestimate the experimental loss, but the overall shapes of the curves in good agreement, suggesting that Eq is accurate. Additional experiments are shown in the Appendix, Figure 6
a
b
1.1
theory experiment
0.75
theory oxperment
0.74
10
Bell a
072
0.1
0.-
069
0. 0.1
-0. 00
0.2
0.3
04
-1.
-1.0
-0.5
0.0
0.5
1.0
leaming rate (adaptation)
learning (raining)
Figure 2: Average test loss of MAML as function of the learning rate, overparameterizeed mixed linear regression, as predicted by our theory and confirmed in experiments. a) Effect of learning rate or, during adaptation. b) Effect of learning rate or during training. The optimal learning rate during adaptation is positive, while that during training is negative. Values of parameters: n; 30, n. 2, n, 20, m 3,p 60, a 1., V/ 0.5, wo a, = 0.2, in panel b) we set 0.2. In the experiments, each run is evaluated on 100 test tasks of 50 data points each, and each point an average over 100 runs (a) or 1000 runs (b).
wo 0. In panel a) we set
5.2 UNDERPARAMETERIZED CASE
In the underparameterizze case, the number of parameters p is smaller than the total number of validation data across tasks n.m. In this case, since the data fully constrains the parameters, the optimal value of found during meta-training is unique. We prove the following result.
Theorem 2. Consider the algorithm of section (MAML steps and the data generating model of section 4 (mixed linear regression). Let M. Let n,(E) and ne(E) be any function of order (E). For E, m + oo, the test loss of Eq averaged over the entire data distribution (see Eq 27 in the Appendix) is equal
5

---

Published as a conference paper at ICLR 2021
I
eest
0²
(1
1
+
hrve
2 +
2
h
a²
h'
at
[nn, + 1) + p92]
u² [no 1) ppul +0
O
((mE)
2h T
nt
p
where h", hh are defined as in previous section, Eqs 6 see Eqs 98 107 in the Appendix.
and gi are order O(1) polynomials in Oct
Proof. The proof of this Theorem can be found in the Appendix, sections 7.3
7.3.2
Again, the loss always increases with the output noise a and task variability v. Furthermore, in this case the loss always decreases with the number of data points nyy fyy and tasks m. Note that, for a very large number of tasks m. the loss does not depend on meta-training hyperparameters og, nes nz. When the number of tasks is infinite, doesn't 't matter whether we run the inner loop, and how much data we have for each task.
As the overparameterized case, the loss quadratic and convex function of the adaptation learning rate a,,, and there is a unique minimum. While the value of the argmin is different, in this case as well the loss is a sum of two quadratic functions, one with minimum at a, 0 and another with minimum at a, 1/ (1 + (p + 1)/n,), therefore the optimal learning rate is again in between the same two values and is always positive. Similar comments applies to this case: a positive learning rate for adaptation implies that the parameters get closer to the optimum for the target task. An example of the loss as a function of the adaptation learning rate a,, is shown in Figure 3h where we also show the results of experiments in which we run MAML empirically. The good agreement between theory and experiment suggest that Eq 9 is accurate.
As function of the training learning rate or, the loss Eq is the ratio of two fourth order polyno- mials, therefore it is not straightforward to determine its behaviour. However, it is possible to show that the following holds
σ² p 2 0
(10)
dai
mm
suggesting that performance is always better for negative values of a around zero. Even if counter- intuitive, this finding aligns with that of previous section, and similar comments apply. An example of the loss as a function of the training learning rate a, is shown in Figure Bp. where we also show the results of experiments in which we run MAML empirically. A good agreement observed between theory and experiment, again suggesting that Eq accurate. Additional experiments are shown in the Appendix, Figure 6
5.3 NON-GAUSSIAN THEORY IN OVERPARAMETERIIZE MODELS
In previous sections we studied the performance of MAML applied to the problem of mixed linear regression. It remains unclear whether the results in the linear case are relevant for the more in- teresting case of nonlinear problems. Inspired by recent theoretical work, we consider the case of nonlinear regression with squared loss
C. (w) =
E
E x
Iy
/ (x, 2
(11)
2
where y is target output and / (x, w) the output of a neural network with input x and parameters w. The introduction of the Neural Tangent Kernel showed that, in the limit of infinitely wide neural networks, the output is linear function of its parameters during the entire course of training (Jacot et al. (2018), Lee et al. (2019)) This is expressed by first order Taylor expansion
f (x, w) ~ f (x, wa) + k (x, w (w wo)
(12)
k (x, wo) V. I (x, w) lxwwo
(13)

---

carry over Published as a conference paper at ICLR 2021
a
b
1.06
heary experment
theory
14
experiment
Test loss
1.02
12
100
10-
0:8
-0.1 0.0
0.2
0.3
04
0.5
-0.4
0.0
04
leaming rate (adeptation)
learning rate (triiinn))
Figure 3: Average test loss as a function of the learning rate, on underparameterized mixed linear regression, as predicted by our theory and confirmed experiments. a) Effect of learning rate or during testing. b) Effect of learning rate (t during training. The optimal learning rate during testing is always positive, while that during training is negative. Values parameters: n, = 5, ny = 40, p 30,
25, n2 10, ar = 0.2. In the experiments. the model is evaluated on 100 tasks of 50 data points each, and each point is average over 100 (a) or 1000 (b) runs.
0.2, = 0.2. In panel
set &e 0.2, in panel b) we set
The parameters w remain close the initial condition wo during the entire course of training, phenomenon referred to as lazy training (Chizat et al. (2020)), and therefore the output can be linearized around wo. Intuitively, in model that heavily overparameterized. the data does not constrain the parameters, and parameter that minimizes the loss in [ can be found in the vicinity of any initial condition wo. Note that, while the output of the neural network is linear in the parameters, it remains nonlinear function of its input, through the vector of nonlinear functions k 13
By substituting Eq 12 into Eq. 11 the nonlinear regression becomes effectively linear, in the sense that the loss is a quadratic function of the parameters w, and all nonlinearities are contained in the functions k in Eq 13 that are fixed by the initial condition wo. This suggests that the theory developed the previous section to this problem. However, in this case the input to the linear regression problem is effectively k (x), and some of the assumptions made in the previous section are not acceptable. In particular, even if we assume that x Gaussian, k (x) is a nonlinear function of and cannot be assumed Gaussian. We prove the following result, where generalize the result of section 5. to non-Gaussian inputs and weights.
Theorem 3. Consider the algorithm of section 3 (MAML one-step), with Wo = O. and the data generating model of section { where the inpul x and the weights w are not necessarily Gaussian, and have zero mean and covariances, respectively, E FExx" and En, Eww"" Let F be the matrix of fourth order moments F = E (x" Ex) xx_. Let p m. Let p(E) and ny (E) be any Let Tr (2%) be of order O (5-1), and let the variances of matrix
function of order O(E) as products of the rescaled imputs x/ /p. up to sixth order, be of order O (g-1) (see Eqs 134 136 in the Appendix). Then the test loss of 2 averaged over the entire data distribution (see Eq in the Appendix) is equal
I test
- (r.
1
1+ - (
2
2
nr
Tr (H'
(Em 0² 1+
+
n
mi
+ +0 0 3/2))
(14)
2
(H')
where define the following matrices
a (F 2
H²=
[l 0,21)²
(15)
144
H¹
E(T - 2)
a F Σ33
(16)
7

---

Published as a conference paper at ICLR 2021
Proof. The proof of this Theorem can be found in Appendix, section 7.4
I(p + 2). wa 0, Note that this result reduces Eqs 5 6 when Σ = I, Σ. = F=
w w- =0 0. This expression for the loss is more difficult to analyze than those given in the previous sections, because it involves traces of nonlinear functions of matrices, all elements of which are free hyperparameters. Nevertheless, it is possible to show that, as function of the adaptation learning rate ay, the loss in Eq [4] is still quadratic function. As a function of the adaptation learning rate ar, the loss in Eq 14 is the ratio of two fourth order polynomials, but it is difficult to draw any conclusions since their coefficients do not appear to have simple relationships.
Even if the influence of the hyperparameters is not easy to predict, the expression in 14 can still be used to quickly probe the behavior of the loss empirically, by using example values for the Σ, Em, F, since computing the expression is very fast. Here we choose values of Σ, En, by a single random draw from a Wishart distribution
22
Σ~ (L.
En (1,p)) P
(17)
Note that the number of degrees of freedom of the distribution is equal to the size of the ma- trices, p, therefore this covariances display significant correlations. Furthermore, we choose F = 2 + ΣTr (>²),, which is the value taken when x follows a Gaussian distribution. There- fore, we effectively test the loss Eq [4 for Gaussian distribution, as in previous section, but we stress that the expression is valid for any distribution of x within the assumptions of Theorem 3 We also run experiments of MAML, applied again to mixed linear regression, but now using the covari- ance matrices drawn in Eq 17 Figure S shows the loss in Eq [4] as function of the learning rates, during adaptation (panel a) and training (panel b). Qualitatively, we observe a similar behaviour as in section [.1] the adaptation learning rate has unique minimum for a positive value of a,, while the training learning rate shows better performance for negative values of a,, Again, there is good agreement between theory and experiment, suggesting that Eq 14 is a good approximation.
b
Beeory experiment
theory
244
22-
1.00
1.8
16-
1.04
1.02
12-
10
-01
0.0
022
0.3
0.4
0.5
-2.0
-1.5
-2.0
-05
00
0.5
1.0
2.0
leaming rate (adeptatio))
leaming rate (traning)
Figure 4: Average test loss of MAML as function of the learning rate, on overparameterized mixed linear regression with Wishart covariances, as predicted by our theory and confirmed in experiments. a) Effect of learning rate during adaptation. b) Effect of learning rate a; during training. The optimal learning rate during adaptation positive, while that during training appears to be negative. Values of parameters: N4 30, n, = 2, n4 20,
1., // 0.5, wp = O, Wo = O. In panel a) we set a₁ = 0.2, in panel b) we set a, = 0.2. In the experiments, each run is evaluated on 100 tasks of 50 data points each, and each point is average over 100 runs (a) 500 runs (b).
3, 60,
5.4 NONLINEAR REGRESSION
To investigate whether negative learning rates improve performance on non-linear regression in practice, we studied the simple case of MAML with a neural network applied to a quadratic function. Specifically, the target output is generated according The data x, 2 and generating parameters w are sampled as described in section bias b was drawn from Gaussian distribution of zero mean and unit variance.). We use 2-layer
(w" x + b)² + 2, where b is bias term.
4 (in addition, the
8

---

Published as a conference paper at ICLR 2021
feed-forwarts neural network with ReLU activation functions. Weights initialized following Gaussian distribution of zero mean and variance equal to the inverse number of inputs. We report results with a network width of 400 in both layers; results were similar with larger network widths. We use the square loss function and we train the neural network in the outer loop with stochastic gradient descent with a learning rate of 0.001 for 5000 epochs (until convergence). We used most parameters identical to section [. 72 30; f2 2; n2 20; m = p = 60; = 1, 0.5, Wo = 0. The learning rate for adaptation was set to ar 0.01. Note that in section 5.1 the model was initialized at the ground truth of the generative model (wp = wo). while here the neural network parameters are initialized at random. Figure the test loss as a function of the learning rate oz. The best performance is obtained for negative learning rate of -0.0075.
6 DISCUSSION
We calculated algebraic expressions for the average test loss of MAML applied to a simple family of lin- ear models, a function of the hyperparameters. Surprisingly, we showed that the optimal value of the learning rate of the inner loop during training negative. This finding seems to carry over to more interesting nonlinear models in the overparameter- ized case. However, additional work is necessary to establish the conditions under which the optimal learning rate may be positive, for example by prob- ing more extensively Eq 14
0081
0.00
0.09
0078
0076
0075
A negative optimal learning rate is surprising and counter-intuitive, since negative learning rates push parameters towards higher values of the loss. How- ever, the meta-training loss is minimized by the outer loop, therefore it is not immediately obvious whether the learning rate of the inner loop should be positive, and we show that in some circumstances it should not. However, perhaps obviously, we also show that the learning rate during adaptation at test time should always be positive, otherwise the target task cannot be learned.
-0.055
-0 010
-0 005
8.000
6.055
leoming rate (traning)
Figure 5: Average test loss MAML as a function of the learning rate, on nonlinear (quadratic) regression using 2-layer feed- forward neural network. Optimal learning rate is negative, consistent with results on the linear case. Each run is evaluated on 1000 test tasks, and each point is an average over 10 runs. Error bars show standard errors. Note the qualitative similarity with Figures 2 and
In this work, we considered the case of nonlinear models in the overparameterized case. However, typical applications of MAML (and meta-learning in general) implement relatively small models due to the heavy computational load of running bi-level optimization, including both outer and inner loop. Our theory applies regression problems, and assumes a limited number of tasks where data is independently drawn in each task, while some ap- plications use a large number of tasks with correlated draws (for example, images may be shared across tasks in few-shot image classification, see Bertinetto et al. (201)). Our theory is valid at the exact optimum of the outer loop, which is equivalent to training the outer loop to convergence, therefore overfitting may occur in the outer loop of our model. Another limitation of our theory represented by the assumptions on the input and task covariance, which have no correlations in Theorems 2 and are subject to some technical assumptions in Theorem 3
To the best of our knowledge, nobody has considered training meta-learning models with negative learning rates in the inner loop. Given that some studies advocate removing the inner loop altogether, which is similar to setting the learning rate to zero, it would be interesting to try a negative one. On the other hand, it is possible that negative learning rate does not work in classification problems, in nonlinear models, or using input or tasks with a complex structure, settings that are outside the theory presented in this work.
We would like to thank Paolo Grazieschi for helping with formalizing the theorems, and Ritwik Niyogi for helping with nonlinear regression experiments.

---

Published as a conference paper at ICLR 2021
REFERENCES
Madhu S. Advani and Andrew M. Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv:1710.036 03666777..0677 [physics, q-bio, stat], October 2017. URL http:
03667
arXiv: 1710.03667.
Yu Bai, Minshuo Chen, Pan Zhou, Tuo Zhao, Jason D. Lee, Sham Kakade, Huan Wang, and Caiming Xiong. How Important is the Train- Validation Split in Meta-Learning? arXiv:2010.0584 [cs, stat], February 2021. URL http:: /ttp///axxvvvv/// org/ abs 2010.00000000000 05843,
arXiv: 2010.05843.
Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable flom solvers. 18X5: 06 [cs, stat], July 2019. URL http://
arxiv.org/abs/
08136
arXiv: 1805.08155 066.08136.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A Closer Look at Few-shot Classification. arXiv: 040.22244.4222 [cs], January 2020a. URL http:
//aaxiv.orggaaaaa abs 1904. 04232
arXiv: 1904.04232.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A New Meta-Baseling for Few-Shot Learning. arXiv:2003.,m 04390 [cs]. March 2020b. URL http://arxiv.org
fbs/2003. 04390
arXiv: 2003.04390.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On Lazy Training in Differentiable Program- ming. 1812 07956 [cs, math). January 2020. URL htttt:://axxvv.ooo htttt:::/axxiv.org//aaaao
07956 arXiv: 1812.07956.
Liam Collins, Aryan Mokhtari, and Sanjay Shakkottai. Why Does MAML Outperform ERM? An Optimization Perspective. 0rXiv:2010.1467 [cs, math, stat), December 2020. URL http:
//arxiv.org/ab:: /2tpr///aa////00... 14672
arXiv: 2010.14672.
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning To Learn Around A Common Mean. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp- 10169-10179. Curran Associates, Inc., 2018. URL
n nips. wwwpeerss..ss...cpppepeeeee cc/ppprr/ around- a- common mean. pdf
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A Baseline for Few-Shot Image Classification. (50v:099000022 [cs, stat), March 2020. URL http:
/ //a.xix.rrgaas////.3...
02729
arXiv: 1909.02729.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Dar- rell. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. ICML, pp. 2014.
Simon S. Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Lei. Few-Shot Learning via Learning the Representation, Provably. arXiv:2002.0944: [cs, math, stat], February 2020. URL http:
7 //arxiv.org/abs//////
09434
arXiv: 2002.09434.
Chelsea Finn and Sergey Levine. Meta-Learning and Universality: Deep Representations and Gra- dient Descent can Approximate any Learning Algorithm. arXiv: 1622 [cs], February 2018. URL
arXiv: 1710.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostie Meta-Learning for Fast Adap- tation of Deep Networks. arXiv: 1703.03400 [cs], March 2017. URL http:://axxv..rrr
abs/1703. 03400
arXiv: 1703.03400.
Katelyn Gao and Ozan Sener. Modeling and Optimization Trade-off in Meta-learning.
arXiv:2010.1 12916 [cs, math, stat], October 2020. URL http:://ardiv.orr//bb///00 12916 arXiv: 2010.12916.
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriaa Cherepanova, and Tom Gold- stein. Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks. arXiv:2002.06755 06750002..673.... [cs, stat), March 2020. URL http::ttpp///.
arXiv: 2002.06753.
10

---

Published as a conference paper at ICLR 2021
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. RECASTING GRADIENT-BASED META-LEARNING AS HIERARCHICAL BAYES. ICLR, pp. 13, 2018.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in High- Dimensional Ridgeless Least Squares Interpolation. arXiv: 08560 [cs, math, stat], Novem- 0000//50000088000000 08560 arXiv: 1903.08560.
ber 2019. URL httttp://axxi...om/
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey, Meta-Learning in Neural Networks: A Survey. wwXiv:2004005499 [cs, stat], April 2020. URL hatt:://axxi..oxe
abs/2004. 05439
arXiv: 2004.05439.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and
Generalization in Neural Networks. arXiv:/806.07511 [cs, math, stat], June 2018. URLhttp: wlarxii..rrgabs//IBB00 07572
arXiv: 1806.07572.
Kaiyi Ji, Junjie Yang. and Yingbin Liang. Multi-Step Model-Agnostic Meta-Lerrning: Convergence and Improved Algorithms. arXi::2002.0788 07830 [cs, math, stat], February 2020. URL http:
0ttp:/2222.00888338 07836 arXiv: 2002.07836.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv:2001 2000:66600006 [cs, stat), January 2020. http:::::/aav...og/gggo./ooo///p
08361
arXiv: 2001.08361.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Adaptive Gradient-Based Meta- Learning Methods. arXiv:1906.022. [cs, stat), December 2019. URLaatp::/arxii.orgg
abs/1906 02717
arXiv: 1906.02717.
Weihao Kong, Raghav Somani, Zhao Song, Sham Kakade, and Sewoong Oh. Meta-learning for mixed linear regression. arXiv:2002.089) [cs, stat], February 2020. URL [tttp://arxiv. org/abs/2002. 08936
arXiv: 2002.08936.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):466444,
May 2015, ISSN 0028-0836, 1476-4687. doi: 10. 1a.1008/natrre455500 URL http://www.
nature. htt.://eccccoorrrrecs//aaatteaae
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve Linear Models Under Gradi- ent Descent. arXiv:1902.0677 [cs, stat), February 2019. URL httt:://axiv..orrt//////// arXiv: 1902.06720.
19C2.06722
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to Learn Quickly for Few-Shot Learning. arXiv:1707.0908 [cs], September 2017. URL http://tp/aat http///xxxvv//o///
abs/1707. 09835
arXiv: 1707.09835.
Preetum Nakkiran. More Data Can Hurt for Linear Regression: Sample-wise Double Descent.
191.v:191..07244 [es, math, stat], December 2019. URL http::///aawwww.ooo/////////////
07242 arXiv: 1912.07242.
Sinno Jialin Pan and Qiang Yang. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering. 12110):1345-133 1359, October 2010. ISSN 1041-4347. doi: 10. 1099/09TT 2909.191. URL
ore. ieee org/ http://dommeettt22526///
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid Learning Feature Reuse? Towards Understanding the Effectiveness of MAML. Xiv:vv/999900. 9999099177777 [es, stat], Febru- ary 2020. RRLLhtttp://arxx...rrr abt////s//99/ 909. 09157
arXiv: 1909.09157.
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A CONSTRUCTIVE PREDICTION OF THE GENERALIZATION ERROR ACROSS SCALES. ICLR, pp. 30, 2020.
Nikunj Saunshi. Yi Zhang, Mikhail Khodak, and Sanjeev Arora. A Sample Complexity Separation between Non- Con-Convex and Convex Meta- Meaninnn arXiv:2002. [cs, math, stat], February 2020. htt.hhtt::/// http:/awxiv...rra/a// www./a/g///////e////e///eeeeee// /202 11172
arXiv: 2002.11177

---

Published as a conference paper at ICLR 2021
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? arXiv:2003. [cs], arXiv: 2003.11539.
June 2020. URL
11539
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- Dataset: Dataset of Datasets for Learning Learn from Few Examples. 0rXiv:1903.03000 [cs, stat], February 2020. URL htt.: arxiv org abs 1903.
000x3//0abb9990003333 03096
arXiv: 1903.03096.
Nilesh Tripuraneni, Chi Jin, and Michael I. Jordan. Provable Meta-Learning of Linear Representa- tions. arXiv:2002. 1684 fcs, stat], February 2020. URL http:://arxivvoorg/a/////000 168
arXiv: 2002.11684
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing Networks for One Shot Learning. arXiv: 1606,04000 [cs, stat), December 2017. URL
http://arxiv..or//aae www.org//aa/.iioo 04080
arXiv: 1606.04080.
Haoxiang Wang, Ruoyu Sun, and Bo Li. Global Convergence and Generalization Bound of
Gradient-Batd Meta-Learning with Deep Neural Nets. arXiv:2006. 14606 [cs, stat], November 2020a.
14600 arXiv: 14606.
4606
Lingxiao Wang. Qi Cai, Zhuoran Yang, and Zhaoran Wang. On the Global Optimality Model- Agnostic Meta-Learning. arXiv:2006.1316 /3182 [cs, stat], June 2020b. URL http://arxiv. bxq/abs/2005..o 13182
arXiv: 2006.13182.
Greg Yang and Edward J. Hu. Feature Learning in Infinite-Width Neural Networks. arXiv:2011.14522 [cond-mat], November 2020. URL http://aaxxxv.orr//ab///2111 14522 arXiv: 2011.14522.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? arXiv:1411.1792 [cs], November 2014. URL http::tpt/xii..o///////
1411. 1792
arXiv: 1411.1792.
Yufan Zhou, Zhenyi Wang, Jiayi Xian, Changyou Chen, and Jinhui Xu. Meta-Learning with Neural Tangent Kernels. arXiv:2 (a:::2202209999 [cs], February 2021. URL fttt:://arxiv.orr//bbb.
2102. 03909
arXiv: 2102.03909.
12

---

Published as a conference paper at ICLR 2021
7 APPENDIX
b
theory overparam
theory overparam theory underparam experient
7-
theory underparam
experiment
0.7
test 00533055
0.6
0.0
02
-0.4
-0.2
0.0
learning rate (trining)
learning rate (training)
C
theery
theory overparam theory underparam
theory underparam
0.50
experiment
operiment
0035
040
test
0.3
02
0.25
01
0. 0.2 0. 0.0
-0.4
leamiing (training)
Figure 6: Average test loss of MAML as a function of the learning rate ar (training) on mixed linear regression, showing the transition from strongly overparameterized (a), to weakly overparameterized (b), weakly underparameterized (c) and strongly underparameterized (d). As expected, predictions of theory accurate only in panels (a) and (d). The amount of validation data increases from panels (a) to (d), with the following values: m = 1, f2 = 2 (a), m = 5, nh = 5 (b), m = 10, n, = 10 (c), equal n, 40, n, 40, 50, a 0.5.,
10, n. 40. Other parameters
0.5,
a, = 0.2, W/ = O, wo (0.1, 0.1,..., 0.1) (note that overfitting occurs since Wh * wo). In the experiments, each run evaluated 100 tasks of 50 data points each, and each point is an average over 100 runs.
7.1 DEFINITION OF THE LOSS FUNCTION
We consider the problem of mixed linear regression y = Xw + z with squared loss, where X is n X p matrix of input data, each row is one of n data vectors of dimension p. n X 1 noise vector, w is a X vector of generating parameters and y is a X I output vector. Data is collected for m tasks, each with a different value of the parameters w and different realization of the input X and noise z. We denote by (()) the parameters for task i, for i l,..., m. For given task i, denote by x (()) the input data for, respectively, the training and validation sets, by z (i) (i) the corresponding noise vectors and by ((),, vr()) the output vectors. We denote by ne, nh the data sample size for training and validations sets, respectively.
For given task i, the training output is equal to
y ((()) X (()) (+)) + ((0)
(18)
Similarly, the validation output is equal to
v(t) x w (1) + 2()
(19)
We consider MAMI. as a model for meta-learning (Finn 2017). The meta-training loss equal
to
meta
o() - x g
(20)
2n
13

---

Published as a conference paper at ICLR 2021
where vertical brackets denote euclidean norm, and the estimated parameters ( (w) are equal to
the one-step gradient update the single-task training loss L ly X g(0) (2/ /2m,, with initial condition given by the meta-parameter w. The single gradient update equal
(w) = Ip
- n2
ar x x(6 w x())
at nhe
yf
(21)
where I, is the p x p identity matrix and ag is the learning rate. training loss with respect to the meta-
We seek to minimize the meta-
meta- parameter w, namely
arg min C
(22)
We evaluate the solution w/ by calculating the meta-test loss
Cleat
2ns ly x=0
(23)
Note that the test loss is calculated over test data X z², and test parameters w', namely
y
(24)
Furthermore, the estimated parameters 0* are calculated on separate set target data X", z", namely
0*
110 1p
Or X xrr w²
X
Or x
(25)
ny
nby
y" = X'ww"" +2"
(26)
Note that the learning rate and sample size can different testing, denoted by ar, Rl, ng We are interested in calculating the average test loss, that is the test loss of Eq 23 averaged over the entire data distribution, equal to
test
1 ly" - -x00112
(27)
W
2n,
7.2 DEFINITION PROBABILITY DISTRIBUTIONS
We assume that all random variables are Gaussian. particular, we assume that the rows of the matrix X are independent, and each row, denoted by x, is distributed according to multivariate Gaussian with zero mean and unit covariance
N (0.4p)
(28)
where 4p the p p identity matrix. Similarly, the noise is distributed following multivariate Gaussian with zero mean and variance equal to o², namely
~ N N0, 1l))
(29)
Finally, the generating parameters are also distributed according to multivariate Gaussian of vari- ance /²/ /p, namely
2 wo:
N
11 p
(30)
The generating parameter w drawn once and kept fixed within a task, and drawn independently for different tasks. The values of and drawn independently in all tasks and datasets (train- ing, validation, target, test). In order to perform the calculations in the next section, we need the following results.
14

---

Published as a conference paper at ICLR 2021
Lemma 1. Let X be a Gaussian X p random matrix with independent rows, and each row has covariance equal to Ig. the p x p identity matrix. Then:
B X x] = nlp
(31)
B
(xxx)² = = (n + 1 = 121p
(32)
B [x*x))² = (n² + + 3mp + 3n + 3p + 4) Lp = ni H31p
(33)
B xxTxx)" =n
=xxTx)"]==<1> +
6n 6np²+
(34)
+6n + + 20) 1p n uslp
(35)
E
x (xxxrxX"))) x)] (n'p+2n) 4, pn v1,1
(36)
E
(x=x)² Tr x)= = #
+ np + 4n 4p 4) I, pn u2, Ip
(37)
R
XTX (x*x)²) = (n np + +
4) Ip pmm 11,27p
(38)
xx 2 Tr (xxT x) (n + np + 2n2pp + 2n²p + 2mp²+ +
(39)
+8n² + Sp² + 21mp + 20n + 20p + 20) I = pm 1221
(40)
where the last equality in each of these expressions defines the variables p. Furthermore, for any n x n symmetric matrix and any p x symmetric matrix D, independent of X:
E [x" cx] (r(C)) Lp
(41)
X n 1n RII()
(42)
Proof. The Lemma follows by direct computations the above expectations, using Isserlis' theo- rem. Particularly, for higher order exponents, combinatorics plays crucial role in counting products of different Gaussian variables in an effective way.
Lemma 2. Let x be Gaussian random matrices, of size respectively nhe X p and n. X p, with independent rows, and each row has covariance equal to Lp. the x identity matrix. Let p(E) and (E) be any function of order O(g) as { - co. Then:
x=00 =phn +0
(43)
x0 x0) (()) xx)))²² = pn In, + +0 ((3/2)
(44)
x"00 x40 x(10 x(() =p + +p+ 1)In, + +0 (5312)
(45)
Note that the order O (g) applies to all elements of the matrix in each expression. For i #
x²10xx0)² =0
=0((112)
(46)
x(10 x((1) x((xx(0)7 =0(2312)
(47)
x"(0) x(()) x(6)xx01) =0 (55522
(48)
15

---

Published as a conference paper at ICLR 2021
Furthermore, for any positive real number å and for any p x p symmetric matrix D independent of X, where Tr(D) and Tr(D²) are both of order O(g*)
x -D) Lnd + +0(5012
(49)
X=() X DXx(1) =Tr(D) ntn. O (5Lt6/2)
(50)
x-000 x DX x(()) =Tr =D) ni(re + +p+ 1)Im, +0 +0 (52-0/2)
(51)
x- = = (5112)
(52)
x000 X DXx =0 (414001)
(53)
x=00 x D x x =0 -
(54)
Proof. The Lemma follows by direct computations of the expectations and variances of each term.
Lemma 3. Let X x be Gaussian random matrices, of size respectively ny x p and n4 X p. with independent rows, and each row has covariance equal to Ip the p identity matrix. Let (E) and (E) be any function of order for cc. Then:
X x" no Lp + +0 0
(55)
X x2 nn Ip + O (53/2)
(56)
X x x"x x = n + +p+ 1)I, + O (53/2)
(57)
Note that the order O (E) applies to all elements of the matrix in each expression.
Proof. The Lemma follows by direct computations of the expectations and variances of each term.
7.3 PROOF OF THEOREMS 1 AND 2
We calculate the average test loss as a function of the hyperparameters 7h, Thy 1ly, p. 11, Olty Ory 0, for the test output, rewrite the test loss in Eq 27 as
v, Wa. Using the expression in Eq 24
Z Flest
1 2n.
X (w' 00)++++** 2212
(58)
We start by averaging this expression with respect to X zz, noting that 0* does not depend on test data. We further average with respect to w', but note that A depends on test parameters, so we average only terms that do not depend on 0*. Using Eq 31 the result is
teat
2
Wa
E
2
(wo 8w') 0-
(59)
2
2
2
where we define 5w w/ Wg. The second term in the expectation is linear 0* and can be averaged over X", z", using Eq 25 and noting that w" does not depend on target data. The result is
E 0* = (1 (r) or (wo + 5w')
(60)
x
Using Eq 60 we average over w' the second term in the expectation of 59.59 19.5aam and find
tast
+
1
0r)
(v² (wol)) - (1 - a,) w/ ~ + 0
(61)
2
2
2
16

---

Published as a conference paper at ICLR 2021
We average the last term of this expression over z", w', using Eq 25 and noting that w does not depend on target data and test parameters. The result is
a (w' (w- wwo) (x-Tx*) 2 (w" (wo-wo) n
1 == E w/ x"
|
(62)
20, X X wf (w
az wo) ww)-
azv² nEP
(xxxxx)))
n? Tr
Tr Tr
(63)
We now average over X", again noting that wi does not depend on target data. Using Eqs 31 we find
32
1+ p p nr
(22 (22 loot lwo-=ol 20, (w+ -wo) + 070²p
E E X w'
EEEGH
| + az
ny
(64)
We can now rewrite the average test loss 61 as
C Jest
0²
1
aP Thy
(1 a²) + t + 22p+1 ny
(2 ( wol*)
(65)
In order to average the last term, we need an expression for wt We note that the loss in Eq 20 quadratic in w, therefore the solution of Eq [2 can be found using standard linear algebra. In
particular, the loss in 20 can be rewritten as
meto
by Bwl
(66)
2n
where y is vector of shape n. x 1, and B is matrix of shape n, x p. The vector is stack of m vectors
x²(1) (Iz- ar x ((())(11) x w(l) m; (xxx)((11)) z00) +z(
n.
7
(67)
X((mm) (I~ ar (m) x((m) w(m) a (x((m (m)) z((m) + (m))
ne
Similarly, the matrix B is a stack of m matrices
X²(1) (Io- as X1()² x((1) x(()xx(())
m
B
(68)
Xu(m) (Iz- x((m)? ? X((m)
We denote by Lp the p p identity matrix. The expression for w that minimizes 6 on whether the problem is overparameterized (p > n,m) or underparameterizeed (p < m)) therefore we distinguish these two cases in the following sections.
7.3.1 OVERPARAMETERTAA CASE (THROREM T
In the overparameterized case (p n,m)), under the assumption that the inverse of BB" exists, the value of w that minimizes Eq 66 is equal to
T (BB") 7 7 ( (BB"B"BBB) Wo
(69)
The vector We is interpreted as the initial condition of the parameter optimization of the outer loop. when optimized by gradient descent. Note that the matrix B does not depend on w, z', 1z,"", and Ew Exy Exx 7 Bwo. We denote by dny the deviation from the average, and we have
wo Bd (BB") 8y + [ Bd (BB") -1 B) (wo - wo)
(70)
We square this expression and average over w, z' z". We use the eyclic property of the trace and the fact that BT (BBT) -1 B is a projection. The result is
- - wol2 Tr (BB") - + (wD wo)" [-- Bd (BB") -1 B] (wo wo) (71)
17

---

In order to calculate IE - wo 2 in 6 we need to average this expression over training and validation data. These averages are hard compute since they involve nonlinear functions of the data. However, we can approximate these terms by assuming that and n, are large, both of order O(E), where & is large number. Furthermore, we assume that wW Wa is of order O(g-1/4). prove that Published as a conference paper at ICLR 2021
The matrix I' is defined as
T
0
r = EEEE EERoy 6"
0
0
(72)
Wa
0
[mm]
Where matrix blocks are given by the following expression
r
2² (() at
Ip
x
x(i)) x
+o²
In,
+
a x())
xxx0xx00
x x
P
Thg
n
(73)
convenient to rewrite the scalar product of Eq 71 terms of trace of outer products
w wol =Tr [BBT) (r (wo wo) (wo wo)" B²)] WD wol 2
(74)
Using Lemma together with the expressions of B (Eq 68) and 1 (Eqs 72 73).
- BB"
1
(1 at) 2
2 + + o
Inom 0 0((5 1/2 (5-112
(75)
P
n
(1 az)² + aa x
aip Ium + +0 + (5-2/2)
T= v²
t 1+
(76)
n1
T4
B (wo wo) (wo wo)" BT= ww - wol? wol? (1-a)) az)²
p + + ag
Inom + +0 05-55-70 (5-1/2)
(77)
n1
Using Eq 75 and Taylor expansion, the inverse (BBT)* -1 is equal
(BBT)- _ p
+ 1 (1 az)² + at
Inom + 0 (5-312).
(78)
T4e
Substituting the three expressions above in Eq
74| and ignoring terms of lower order, we find
1+ oir
E [| wol 20 = -(1 7, wo 2
Wi
nm p
22 0²
0 ( 3/3)
p
(1 (1- ov)) o p+1
(79)
Substituting this expression into in Eq
65
we find the value of average test loss
test
0²
app nr
+
+
(80)
ofp +
+h-
y2
1
+
((1 7om
) Wo
wol 2
o² n
ne
+ +0
(5 (5-3/2
p
2p
he
(81)
where we define the following expressions
= (1- - 0z 2z)2 + n4
P +
and h" = (1 a,,) 2 + at
x
(82)
7.3.2 UNDERPARAMETERRIZR CASE (THEOREM 2
In the underparameterizeed case (p n.m. under the assumption that the inverse of BT B exists, the value of w that minimizes Eq 66 equal to
3- =(B² B) B²a
(83)
18

---

Published as a conference paper at ICLR 2021
Note that the matrix B does not depend on w, zz, z", and Ew Eg Ez 7 = Bwo- We denote by dry the deviation from the average, and we have
w wol 2 = Tr
8y
(84)
We need to average this expression in order calculate E - wol in Eq 6 We start by aver- aging dry 8yy over w, z', z", since B does not depend on those variables. Note that w, z z" are independent on each other and across tasks. As in previous section, we denote by I the result of this operation, given by Eq. 72 73 Finally, we need to average over the training and validation data
E 3
wol
E
E Tr
(85)
hard to average this expression because it includes nonlinear functions of the data. However, approximate these terms by assuming that either & is defined by assuming that both ne and ny are of order ((E). Using Lemma 3 together with the
& (or both) is large number, where
expression of B (Eq (8). and noting that each factor in Eq 85 has a sum over independent terms, we can prove that
1 B (1 2a + az Ip + O ((mc)-
(86)
m
The expression for Aa is given in Eq Using this result and a Taylor expansion, the inverse is equal
n.m B BB B) (1 2a + p2) Ip O ((mE)-1/2
(87)
Similarly, the term TTB
B is equal to its average plus term of smaller order
TB LB(BIBB) +0 ((mE)
(88)
mm
n.
We substitute these expressions in 85 and neglect lower orders. Here we show how to cal- culate explicitly the expectation of TB.. For ease of notation, we define the matrix 11 I 2i (0) x²0) Using the expressions of B (Eg68) and I (Eqs 72 73), the expression for
B is given by
A x² A000 + v² AAO0)" xx()" xx(0) xx(()A1))
BTBB = 02
+
0.0² O T X "0 x x x x (1())
m
(89)
We use Eqs
31
32
calculate the average of the first term in Eq 89
E
E
A
X
x A = n, (1 20, + a²H²) Ip
(90)
We use Eqs
31
33
36
37
38
39
to calculate the average of the second term
m
A1(0) T xo(i) T xv(i) )100)
2
7
no (no + 1) !(10)* + 160) Tr (AMG))]
E
1=1
(91)
= mn, (n, + 1) (1 da, + 6a2 U2 - 4a us + a 11 lp+
+ mn,P (1 4a, + 2a #2 + 40, - 40 12,1 + a (222222) I,
(92)
Finally, we compute the average of the third term, using Eqs 31 32
33
34
41
36
37
E E X" T x x T x110 ( x=(() (100
(93)
ny (n. + A007 x x00 100) + n ( 1 (x(()x((()
E
(94)
mnn (nl + 1) ne (1 201 + 22) 4p + mn (1-201H1,,
20041,11 + af 4p
(95)
19

---

Published as a conference paper at ICLR 2021
Putting everything together in Eq 85 and applying the trace operator, find the following expres- sion for the mnea--pramette variance
E w
-wol
wol
P m
(1 - 20, + aîpz) -2 { (1-20) 2a, + +a.22) +
a70²
[nay 1) (1-22 + +ayus) + 201 + a9H2.1)
74
1
+
(nu + 1) (1-414x + 6ag - 40² + artl()) +
p
+ p (1 4ay + 2a, R2 + 40, - 4a + af } +0
(96)
We rewrite this expression
at [(n + 1) p92]
v²
E W wol==
p
02
he
[In + 1) g3 + pgs]
n
nt
+ (mm))
(97)
where we defined the following expressions for 9:
20,112 + a/Pa
(98)
92 201/1,11 + a2
(99)
f3 4a, + 60i μ22 40 us + a/1²²
(100)
91 la, + 2a 407 40/ 4/
(101)
and μи are equal
1
P22
(n+ + +p+ 1pp11 1)
(102)
T4
1
#1
(nj + p² + 3np + 3n, +p3+p 4)
(103)
n2
1
144 =
(ng + p² + bnLp Gnup² + 6ng + 6pp 17mp + 21m, + 21p + 20)
(104)
n
1
11,1
(nēp + 2nx)
(105)
p
12.1
(n:p + mzp² + n&p + 4n, + 4p +
(106)
p
1
(māp + n&p + 2n²p² + 2nzp + 2n,p² + 8n + 8p² + 21m.p + 20n, + 20p + 20)
12,
nip
(107)
Substituting this expression back into Eq 65 returns the final expression for the average test loss, equal to
C test
0²
app
h²
2
7r
h"
p
0²
ht
+
af
[[n 1) 91 + pgol
+ 2² [nn, + 1) go + rou) + O ((mE)- -3/2)
2ht²
m.
n4
(108)
7.4 PROOF OF THEOREM
In this section, we release some assumption on the distributions of data and parameters. In particular, do not assume specific distribution for input data vectors and generating parameter vector
20

---

1 Published as a conference paper at ICLR 2021
w, besides that different data vectors independent, and are data and parameters for different tasks. We further assume that those vectors have zero mean, and denote their covariance as
y Exx"
(109)
Fx: = Eww
(110)
We will also use the following matrix, including fourth order moments
F= (x"
(111)
We do not make any assumption about the distribution of but note that, if x is Gaussian, then F 2Σ + ΣTr (22).. We keep the assumption that the output noise Gaussian and independent for different data points and tasks, with variance o². Using the same notation as in previous sections, will also use the following expressions (for any p matrix A)
E [ X] = nΣ
(112)
Tr
= (A + ( (F-
(113)
We proceed to derive the same formula under these less restrictive assumptions, the overparam- eterized case only, following is the same derivation of section 773 We further assume wh O. Wn = 0. Again we start from the expression in Eq f4 the test output, and we rewrite the test loss
in Eq 27 as
test
E
X (w' - -0) +2"2 z"1
(114)
2n.
We average this expression with respect to X*, x", noting that 0" does not depend on test data. We further average with respect w', but note that 0* depends on test parameters, so we average only terms that do not depend on 0* Using Eq 112 the result
o² + Tr(x2,)) + E clest
-0 - 0r)
(115)
2
The second term in the expectation is linear in 0" and can be averaged over X z", using Eq 25 and noting that w" does not depend on target data. The result
E X
0" = (I- + aw
(116)
Furthermore, we show below (Eq 128) that the following average holds
EE IE w =
(117)
Combining Eqs! 16
117
we can calculate the second term in the expectation Eq 115 and find
festt
2 + Tr (TEEw) a, Tr (Σ> =k O 0"
(118)
We start by averaging the third term of this expression does not depend on target data and test parameters. The result is
z", w', using Eq 25 and noting that w
E E
E
T
0
= Tr
-rarxe ar x x x (1-
Σ
I
w w T (1 ar x
( xx_xxxx)) xT
(119)
nh
n2
2
X'EXIT]
T
EXIT X X"E X
Tr
+
(120)
n?
n?
We now average over X", again noting that wt does not depend on target data. Using Eqs we find
112
13
EEE 0 Σ 0- = Tr -Tr w
- nr
)
E(I - -(I-a,)))² axE)2
(r -
(121)
w'
22
1 (F 239)
Tr (I²2
of
En
33
(122)
nr
nr
21

---

Published as a conference paper at ICLR 2021
We can rewrite the average test loss in Eq 118 as
Jest
0²
1 ++ ct
(3))
Tr
(Z B w.ww ( Hr
(123)
nl
2
where we define the following matrix
a (F (F -وو
H1 -1-
(124)
Br
In order average the last term, need expression for wt We note that the loss in Eq 20
quadratic in w, therefore the solution in Eq 22 can be found using standard linear algebra. In
particular, the loss in Eq(20can can be rewritten as
metoa
by Bwl
(125)
2n
where y is vector of shape n. 1, and B matrix of shape n.m x p. The vector y stack of m vectors
X (I-
a x(1)) xxx))x w()) an (xxx(( z((1) z1)
n
7
(126)
X(((m) (1- a
x ((m) ((mm) w(m)) X X((mm T 2 (m))) zm(m)
Similarly, the matrix B is a stack of m matrices
X (1- x x((1)
B =
(127)
X²(mm) (I- as x(mm) X((m)
In the overparameterized case (p m.). under the assumption that the inverse of BBT exists, the value w that minimizes Eq 125 and that also has minimum norm, equal to
w = B" (BB")-²
(128)
Note that the matrix B does not depend on w, z' z", and Ew Ex Ez = O, therefore Eq 117 holds. order finish calculating Eq 123 we need to average the following term
Tr H (TTw"++) T [BB" - rh (BB") -1 (BH'B))
(129)
where we used the cyclic property of the trace. We start by averaging "Y² over w, z', zz, since B does not depend on those variables. Note that w, z are independent on each other and across tasks. We denote by I' the result of this operation, which is equal to a block diagonal matrix
r =
EEEE " x
0
0
(130)
0
p(m)
Where matrix blocks are given by the following expression
(()) = xv(()
(1
at
(')
(((()
En
at
x(())² x((1) x
(131)
74
o In, 3
a
x
(() x)())))
(132)
ni
Finally, we need to average over the training and validation data
ET
H [H'''""'T]
EEE Tr (BB*) -1 '.'''r'o (BBB") (BHBB*
(133)
X2
22

---

Published as a conference paper at ICLR 2021
These averages hard to compute since they involve nonlinear functions of the data. However, can approximate these terms by assuming that p and ny are large, both order O(E), where g is a large number. Furthermore, we assume that Tr (x2) of order O ((-1), and that the variances of matrix products of the rescaled inputs x/ /p. up to sixth order, are all of order O (5-1), in particular
1 xm(() xr(()xx)?)) (5-
Var
(134)
X X X (x(x)()))) = =0
Var
(135)
10) x xx()i)))) (E-))
Var
X
(136)
Then, using Eqs
112
113 and the expressions of B (Eq 127 and (Eqs) 130 131), we can prove that
BBT = Tr (1') mnom + +0 +0(
(137)
Tr H²) o² 1+ a T (29) (xx))
from + +0 (E(1
r
(138)
nte
BH Bd = Tr TH (H*H²) Iamm + +0 (51/2)
(139)
where, similar to Eq 124 we define
H - (I--aT)) + of (F Σ3)
(140)
Note that all these terms are of order O (E). The inverse of BBT can be found by Taylor expansion
(BB*)~+= Tr (11*) mmom + (g-33/12
(141)
Substituting these expressions in Eq 133 we find
Tr (E H') + 0²
1+ od (Σ²)
E Tr (H'''' (H''"
+0 (5-312
3/2) (142)
Tr (H')
Substituting this expression into in Eq
123
we find the value of average test loss
o²
Z Tr (Ew H
+
1 +
Tr (Σ²)
(143)
2
2
Tr(''
Tr
(In H²)
+
[1+ of [Σ))
}
+ +0 +0(5 (3/3))
nmm
(144)
Tr (Hi))
23


Reviews:
[{"review_id": "h3wpoZbNSZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies meta-learning in the mixed linear regression setting, focusing on the effect of the within-task step-size on performance. For over-parameterized, under-parameterized, and NTK regimes they derive expressions for test-time loss that suggest that negative or close-to-zero learning rates are optimal, and provide experiments that closely match these results. However, some aspects of the mathematical approach are unclear, and the work's impact is limited without an investigation of the consequences of the analysis.\n\nStrengths:\n1. Understanding meta-learning in simple settings and focusing on the effect of learning rate are worthwhile goals.\n2. The authors provide experimental evidence closely following the outlined theoretical results.\n\nWeaknesses:\n1. The mathematical results in the paper are difficult to follow carefully. It is still unclear to me what objective is being solved by the meta-training procedure, and various derivations in the appendix seem non-rigorous, such as replacing denominator terms by expectations. Formal statements and some proof sketches would be helpful. See also Questions 1-4.\n2. No experimental investigation beyond the limited settings studied theoretically. The authors’ investigation leads to a clear prescriptive suggestion—use of negative within-task step-sizes during meta-training—that should be investigated on actual data to get a sense of whether the analysis extends to practical settings. This would be useful to do in both linear settings (see e.g. the experiments in Denevi et al. (2019)) and for standard meta-learning tasks. See also Questions 5-6.\n3. The analysis is limited in not being able to handle representation learning or the case of label shuffling among tasks. Motivating empirical work such as the paper of Raghu et al. (2020) suggests that representation learning is a key component of being able to meta-learn without inner loop updates, while theoretical work (e.g. Saunshi et al. (2020)) suggests that the linear setting studied here cannot account for successful meta representation learning.\n\nQuestions:\n1. The paper seems to suggest the objective ignores task structure completely, but in this case how does the within-task step-size affect meta-training at all?\n2. How many steps of MAML are used in the analysis?\n3. The line “All of the above distributions apply independently to each task and dataset (training, validation, target, test).” seems to suggest that data for the same task is drawn from different distributions - is this the case and if so how it this at all justifiable in the meta-learning setting?\n4. In what references can the Gaussian moment results (36-46) be found?\n5. How closely do experiments follow the analysis in non-asymptotic settings?\n6. In the NTK results, why not investigate agreement between theory and experiments using kernel matrices obtained from actual networks rather than probability distributions on symmetric matrices?\n\nNotes:\n1. “their performance doesn’t seem to stop improving when adding more data and computing resources” - why would we expect it to?\n2. “A meta-learning problem is solved by a bi-level optimization procedure: an outer loop optimizes meta-parameters across tasks, while an inner loop optimizes parameters within each task (Hospedales et al. (2020)).” - not all algorithms do this, c.f. Reptile (Nichol et al., 2018).\n3. “recent papers argue that a simple alternative to meta-learning is just good enough, in which the inner loop is removed entirely (Chen et al. (2020a), Tian et al. (2020), Dhillon et al. (2020), Chen et al. (2020b), Raghu et al. (2020)). This is surprising, because this approach merges all source tasks into a single big training set, and it does not even distinguish between different source tasks during training.” - the paper of Raghu et al. (2020) does not remove the inner loop during training and still distinguishes between source tasks, since gradient updates still use data from the same task. Actually merging all data into a single big training set was shown to perform poorly by Finn et al. (2017).\n4. “In the problem of mixed linear regression, we prove that the optimal learning rate is always negative in overparameterized models. The same result holds in underparameterized models for small values of the learning rate.” - the second sentence is unclear. The learning rate is negative when it is small?\n5. “none of these studies look at the effect of the learning rate” - in the convex setting, Khodak et al. (2019) connect the within-task learning rate to the task similarity.\n6. “Note that each task is characterized by a different distribution of the data, and we separate the training Dt and validation data Dv.”  - it is unclear whether these distributions are empirical distributions or population distributions. If they are empirical distributions then minimizing (1) is not the “goal of meta-learning” but an objective used to achieve the goal. If they are population distributions, are Dt and Dv different? Why and how?\n7. What does the equality-like symbol in (5) mean? “Asymptotically equal to”?\n8. Numerous equation reference numbers are missing parentheses.\n9. “learning of the meta-parameter ω is performed by the outer loop” - still unclear which objective is being used for this. (1)?\n\nReferences:\n- Denevi, Stamos, Ciliberto, Pontil. “Online-within-online meta-learning.” NeurIPS 2019.\n- Khodak, Balcan, Talwalkar. “Adaptive gradient-based meta-learning methods”. NeurIPS 2019.\n- Nichol, Achiam, Schulman. “On first-order meta-learning algorithms.” arXiv 2018.  \n\n# Update after rebuttal phase\n\nThank you to the authors for engaging with reviewer comments. I think the paper is much clearer now, and the additional results in Figures 5 and 6 indicate that the analysis may be relevant for practical meta-learning settings. I am not sure of the necessity of the new data-generating plot for mixed linear regression in Figure 1 (my uncertainty here was resolved with words); the authors might consider using the space for putting Figure 6 (currently in the appendix) in the main paper, or for additional experiments. Two more notes:  \n1. It may make the paper easier to read if the appendix were part of the same PDF as the main paper and not in the supplementary material.\n2. While the experiments are perhaps not difficult to reproduce, code would be helpful to the community.\n\nI am increasing my rating to a 6, as I believe the paper presents an interesting result with sufficient evidence. I am not giving a higher rating as I think the paper's impact would increase substantially with experiments on actual data, either in the mixed linear or deep net setting. For future versions of the paper, I encourage the authors to consider adding such results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gymjZGCgL--", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper gives much needed attention to the theoretical underpinnings of modern meta-learning algorithms such as MAML; it introduces a novel formal argument, discovers surprising implications and follows through to show that such predictions hold experimentally, despite being counterintuitive.\n\n\nStrong points: \n- Novel analysis of a practically important aspect of MAML-like meta-learning algorithms: setting the learning rates for training and adaptation; surprising theoretical result (of negative learning rates being optimal in some cases during meta-training) is well evaluated in controlled conditions.\n- Honest discussion of limitations and good intuition is provided for applicability of the work. This is not too hard to do, but so many papers don’t provide it. Great job!\n- Writing is clear enough, although the paper is dense.\n- The authors don’t discuss previous empirical works (e.g. Meta-SGD/LEO) where meta-learning of the learning rate leads to negative inner-loop learning rates for some parameters, but such experiments actually provide further evidence to back up their claims, this time in SOTA deep models.\n\n\nWeak points:\n- Unfortunately, the paper includes only toy-task experiments, even by the standards of  meta-learning research.\n\n\nRecommendation and Rationale:\nI strongly support acceptance because this paper contains much needed fundamental work on theoretical underpinnings of modern meta-learning.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZyVHL24PrT9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe authors of this paper prove that the optimal learning rate for MAML is negative under mixed linear regression and nonlinear regression with overparametrized models. They verify that theoretic bounds align with numerical experiments.\n\n### Comments\n* As the authors note, \"theoretical work is still lagging behind,\" so theoretical research to explain advances in meta-learning is useful.\n* I think the clarity could be improved, especially in section 3. It was a bit difficult to understand notation and the exact setting. For instance, I think it'd be preferable to use a different Greek letter for the learning rates, rather than $\\alpha_t$ and $\\alpha_r$. \n* I performed a surface level check of the proofs, and the results look correct to me.\n* It would be nice to discuss more the practical implications of the results or whether there still exists a divide between theory and practice.\n\n### Recommendation / Justification\nI vote that the paper is marginally above the acceptance threshold. I think the results presented are interesting, but it is not clear to me what the implications are. It would be nice if the authors explored whether the setting they analyze aligns with more commonly used meta-learning tasks.\n\n### Questions\n* In the last paragraph of Section 3, are you referring to the learning rates at meta-training and meta-testing time?\n\n### Minor feedback \n* not only the performance -> not only [does] the performance\n* These results help clarifying -> These results help clarify\n* In general, I think it is better to reference equations as Equation 13 rather than 13.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "U22Azq1o6bG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper uses random matrix theory to study meta-learning in mixed linear regression and finds that the optimal learning rate for the inner loop/training is negative in order to minimize the test/adaption error. The results are interesting and novel. However, there are some concerns regarding the practical relevance and presentation of the results. \n\nMajor comments:\n\n1. Implementation of negative learning rates in practice: This paper provides an interesting perspective that a negative learning rate could reduce the test error. My concern is with a negative learning rate, the training loss $\\mathcal{L}^{(i)}$ in Eq. (3)  increases and the algorithm may not converge (at least on the training sets). In practice, for example, on deep learning models motivated by this paper in the first paragraph, how do you decide when to stop training parameters $\\theta^{(i)}$ and $\\omega$? How could you use the results in this paper to provide some guidance? \n2. Presentation of the main results: I would suggest the authors formally state the results in theorems or propositions. Currently,  the main results are presented in Eq. (5), (9), (10), (15), and (16), that seem to be informal and need clarification. First, how is $\\bar{\\mathcal{L}}^{test}$ defined? Second, what does $\\simeq$ mean in this context? If it means ````\" approximately equal to,\" then what is the order of the estimation error? Third, the results are derived using mean-squared loss as shown in Eq. (25) and (28) in the appendix. It is helpful to be explicit about the loss function in the main text. Fourth, the loss function does not seem to have a regularization term. In the overparameterized regime, would the model suffer from overfitting? \n3. Experiments: Could you elaborate more on why the theory matches the experiment pretty well in Figures 1.a, 2.a, and 3.a, while not the case in Figures 1.b, 2.b, and 3.b (especially Figure 3.b)? If I understand correctly, the data generating process in the simulation follows the assumption in the main results. Is it because the estimation error in $\\bar{\\mathcal{L}}^{test}$ (the terms omitted in the RHS of $\\simeq$) is not negligible in finite samples? Also, is the curve in Figures 1.b and 3.b robust to the choice of parameters? It may be helpful to include a few other simulation setups in the appendix. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While this paper would be significantly improved with experiments on real data, the reviewers all agreed that there is value in the ideas and simple experiments in this paper and all voted for acceptance after the discussion period.

We encourage the authors to consider adding an experimental evaluation in more realistic settings (e.g. with real data) in the final version of the paper.
Decision: Accept (Poster)