First-order ANIL provably learns representations despite overparametrisation
Authors: Oğuz Kaan Yüksel
Etienne Boursier
Nicolas Flammarion
Venue: ICLR 2024 poster (2024)
Keywords: meta-learning
misspecification
overparametrization
pretraining
optimization
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: In the limit of an infinite number of tasks, first-order ANIL with a linear two-layer network architecture provably learns linear shared representations despite having a width larger than the dimension of the shared representations.

Abstract: Due to its empirical success in few-shot classification and reinforcement learning, meta-learning has recently received significant interest. Meta-learning methods leverage data from previous tasks to learn a new task in a sample-efficient manner. In particular, model-agnostic methods look for initialization points from which gradient descent quickly adapts to any new task. Although it has been empirically suggested that such methods perform well by learning shared representations during pretraining, there is limited theoretical evidence of such behavior. More importantly, it has not been shown that these methods still learn a shared structure, despite architectural misspecifications. In this direction, this work shows, in the limit of an infinite number of tasks, that first-order ANIL with a linear two-layer network architecture successfully learns linear shared representations. This result even holds with _overparametrization_; having a width larger than the dimension of the shared representations results in an asymptotically low-rank solution. The learned solution then yields a good adaptation performance on any new task after a single gradient step. Overall, this illustrates how well model-agnostic methods such as first-order ANIL can learn shared representations.

Published as a conference paper at ICLR 2024
FIRST-ORDER ANIL PROVABLY LEARNS
REPRESENTATIONS DESPITE OVERPARAMETRIZZATON
Oguz Kaan Yüksel
Etienne Boursier"
TML. Lab, EPFL.
INRIA, Université Paris Saclay, LMQ et enne wererrsieriiiiiii fr
oguz. @epfl cc
Nicolas Flammarion
TML Lab, EPFL
nicolas, walllas.flammann flammarion@epfl.comm
ABSTRACT
Due to its empirical success in few-shot classification and reinforcement learning, meta-learning has recently received significant interest. Meta-learning methods leverage data from previous tasks to learn new task in sample-efficient manner. particular, model-agnostic methods look for initialization points from which gradient descent quickly adapts any task. Although it has been empirically suggested that such methods perform well by learning shared representations dur- ing pretraining, there is limited theoretical evidence of such behavior. More im- portantly, it has not been shown that these methods still learn a shared structure, despite architectural misspecifications. In this direction, this work shows, in the limit of an infinite number of tasks, that first-order ANIL with a linear two-layer network architecture successfully learns linear shared representations. This result even holds with overparametrization, having a width larger than the dimension of the shared representations results in an asymptotically low-rank solution. The learned solution then yields good adaptation performance on any new task after a single gradient step. Overall, this illustrates how well model-agnostic methods such as first-order ANIL can learn shared representations.
INTRODUCTION
Supervised learning usually requires a large amount of data. To overcome the limited number of available training samples for a single task, multi-task learning estimates a model across multiple tasks (Ando & Zhang 2005: 2005) Cheng et al. 2011). The global performance can then be improved for individual tasks once structural similarities between these tasks are correctly learned and leveraged. Closely related, meta-learning aims to quickly adapt to any new task, by leveraging the knowledge gained from previous tasks, e.g., by learning shared representation that enables fast adaptation.
Meta-learnin has been mostly popularized by the success of the Model-Agnostic (MAML) algorithm for few-shot image classification and reinforcement learning (Finn al.) 2017). MAML searches for an initialization point such that only a few task-specific gradient descent iter- ations yield good performance any new task. It is model-agnostic in the sense that the objective is readily applicable to any architecture that is trained with a gradient descent procedure, without any modifications. Subsequently, many model-agnostic methods have been proposed (Nichol al.) 2018 Antoniou et al., 2019; Raghu et al. [2200] Hospedales et al. (022). Raghu et al. (2020) em- pirically support that MAML implicitly learns shared representation across the tasks, since its intermediate layers do not significantly change during task-specific finetuning. Consequently, they propose the Nomost-No-Inner-LLovv task-specific updates and performs similarly to MAML. However, to avoid heavy computations for second-order derivatives, practitioners generally use first-order approximations such as FO-MAML or FO-ANIL that achieve comparable performances at cheaper cost (Nichol et al. 2018).
Meta-Learning
(ANIL) algorithm, which only updates the last layer during
Despite the empirical success of model- methods, little known about their behaviors in theory. To this end, our work considers the following question on the pretraining of FO-ANIL:
"This work completed while F. Boursier member of TMI. Lab, EPFL..

---

Published as a conference paper at ICLR 2024
Do model- methods learn shared representations in sew-shot settings?
Proving positive optimization results on the pretraining of meta-learning models is out of reach in general, complex settings that may be encountered in practice. Indeed, research beyond linear mod- els has mostly been confined to the finetuning phase (Ju et al.) 2022 Chua et al. 2021). Hence, to allow a tractable analysis, we study FO-ANIL the canomical multi-task model of a linear shared representation; and consider a linear two-layer network, which is the minimal architecture achieving non-trivial performance. Traditional multi-task learning methods such as Burer- Montr-Monteiro factor- ization (or matrix factorization) (Tripuraneni et al.) 2021 Du et al. 2021; Thekumparampil et al.) 202T] and nuclear norm regularization (Rohde Tsybakov 201 1 Boursier et al. 2022) are known for correctly learning the shared representation. Besides being specific to this linear model, they rely on prior knowledge of the hidden dimension of the common structure that is unknown in practice.
For meta-learning in this canonical multi-task model, /Saunshi al. (2020) has shown the first result under overparametrization by considering a unidimensional shared representation, infinite samples per task, and an idealized algorithm. More recently, (Collins et al. 2022) has provided multi- dimensional analysis for MAML and ANIL in which the hidden layer recovers the ground-truth low-dimensional subspace at an exponential rate. Similar to multi- task methods, the latter result relies on well-specification of the network width, i.e., it has to coincide with the hidden dimension of the shared structure. Moreover, requires weak alignment between the hidden layer and the ground truth at initialization, which is not satisfied in high-dimensional settings.
The power of MAML and ANIL, however, comes from their good performance despite mismatches between the architecture and the problem; and in few-shof settings, where the number of samples per task is limited but the number of tasks is not. In this direction, we prove a learning result under a framework that reflects the meta-learning regime. Specifically, we show that FO-ANIL successfully learns multidimensional linear shared structures with an overparametrised network width and with- out initial weak alignment. Our setting of finite samples and infinite tasks is better suited for prac- tical scenarios and admits novel behaviors unobserved in previous works. In particular, FO-ANIL not only learns the low-dimesioonal subspace, but it also unlearns its orthogonal complement. This unlearning does not happen with infinite samples and is crucial during task-specific finetuning. In addition, we reveal a slowdown due to overparametrization, which has been also observed in su- pervised learning (Xu & Du, 2023). Overall, our result provides the first learning guarantee under misspecifications, and shows the benefits of model-agnostic veta-learning over multi-task learning.
Contributions. We study FO-ANIL a linear shared representation model introduced in Sec- tion 2 In order to allow tractable yet non-trivial analysis, we consider infinite tasks idealisation, which is more representative of meta-learning than the infinite samples idealisation considered in previous works. Section presents our main result, stating that FO-ANIL asymptotically learns an accurate representation of the hidden problem structure despite a misspecification in the network width. When adapting this representation comparable to linear regression on the hidden low-dimensional subspace. Section 4 then discusses these results, their limitations, and compares them with the literature. Finally, Section emppirially illustrates the success of model- methods in learned representation and at test time.
a new task, FO-ANII. quickly achieves test loss
2
PROBLEM SETTING
2.1 DATA DISTRIBUTION
In the following, tasks are indexed by i E N. Each task corresponds to d-dimensional linear regression task with parameter 0_,; E Rd and m observation samples. Mathematically, we have for each task observations (X,, 91) E mx x R" such that
y/ X,O.,s + 2j where z; IR" is some random noise.
Some shared structure is required between the tasks to meta-learn, i.c., to be able to speed up the learning of a new task. Similarly to the multi-task linear representation learning setting, we assume that the regression parameters 0,,; all lie in the same small k-dimensional linear subspace, with k < d. Equivalently, there is an orthogonal matrix B. E Rdxk and representation parameters W..s E R such that 0.,, B. w., for any task i. To derive proper analysis of this setting, we assume a random design of the different quantities of interest, summarized in Assumption [ This assumption and how it could be relaxed is discussed in Section 4
2

---

Published as a conference paper at ICLR 2024
Input (4)
Taak-peottic outper layer
Chrpprsmee hiison
0,3
Buniaa Ra
Bytivmm fus
B. geis
B gol
eggitiiennt Vblayer netwockls with a slared boddon layer
serves the seeelive the minegeeffeed stullent
mattookatrted FO-ANIL
(or lase Link-spiiili sdlaptation)
Figure 1: Left: Regression tasks with parameters 0,, E Rd are confined in a lower dimensional subspace, equivalent to the column space of matrix B, E RRdkk Center: This is equivalent to hav- ing two-layer, linear, teacher networks where B, is the shared hidden layer and the outputs layers Wa,i E R are task-specific. The meta-learning task is finding an initialization that allows fast adap- tation to any such task with a few samples. Right: The student network has the same architecture but it is agnostic to the problem hidden dimension, i.e., k' 2 k, which is the main difficulty in our theoretical setting. On the contrary, previous works on model-agnostic and representation learning methods assume k' = k, i.e., the hidden dimension is priori known to the learner.
Assumption 1 (random design). Each row of X; is drawn i..dd according to (0, ((OLL La) and the coordinates of 2; are i.i.d, centered random variables of variance 0². Moreover the task parameters W..i are drawn i.id with E[w.,) 0 and covariance matrix L. = E[w,, W.W [ cl, with O.
2.2 FO-ANIL ALGORITHM
This section introduces FO-ANIL in the setting described above for N E N tasks, as well as in the idealized setting of infinite tasks (N x.). The goal of model-agnostic methods is learn parameters, for a given neural network architecture, that quickly adapt to a new task. This work focuses on a linear two-layer network architecture, parametrised by B = (B, w) E Bdd R with & S k < d. The estimated function is then given by fo x + Bu.
The ANIL algorithm aims at minimizing the test loss on new task, after small number of gradient steps on the last layer of the neural network. For the sake simplicity, we here consider a single gradient step. ANII. then aims at minimizing over the quantity
CANIL (f) E..... [c. (0 aV (0) v1)
where L; is the (expected) test loss on the task i, which depends on w. L; X;, v/) is the empir- ical loss on the observations (X,, y;); and a the gradient step size. When the whole parameter updated at test time, i..., V.. is replaced by Ve. this instead corresponds to the MAML algorithm.
For model-agnostic methods, it important to split the data two for inner and outer loops during training. Otherwise, the model would indeed overfit the training set and would learn poor, full rank representation of the task parameters (Saunshi et al. 2021 For min + Mout = m with min < m, split the observations of each task as (X," v(") T x " the min first rows of (X,, (4); and (Xputt, yf"h) E I.olout Xd
I Mous the Mout last rows of (X,,
While training, ANIL alternates at each step E N between an inner and an outer loop to update the parameter 0,, In the inner loop, the last layer of the network is adapted to each task following
Wt. f- Wr - a Z, (0,; v!")
Again, updating the whole parameter Or,: with Va would correspond to MAML algorithm. In the outer loop, ANIL then takes a gradient step (with learning rate B) the validation loss obtained for the observations (Xout, yfutt) after this inner loop. With 0₁, == (B,, I),, it updates
d t 6: EN (O: oo (i Xoout p
where the matrix He accounts for the derivative of the function 9, + dr Computing the second- order derivatives appearing in fl,s is often very costly. Practitioners instead prefer to use first-order approximations, since they cheaper in computation and yield similar performances (Nichol et al. 2018). FO-ANII. then replaces He. by the identity matrix in Equation
3

---

Published as a conference paper at ICLR 2024
2.2.1 DETAILED ITERATIONS
In our regression setting, the empirical squared error is given by L, x),
2m 3/+
BuB 2. In that case, the FO-ANIL inner loop of Equation 2 gives in the setting Section 2.1
W1, wi
( T1" (B,w, B,w,,)
(X!"))
(4)
min
min
The multi-task learning literature often considers a large number of tasks (Thekumparampil et al, 2021 Boursier et al. (022) to allow a tractable analysis. Similarly, we study FO-ANIL. in the limit of an infinite number tasks N = oo to simplify the outer loop updates. this limit, iterations are given by the exact gradient of the ANIL loss defined in Equation ( when ignoring the second-order derivatives. The first-order outer loop updates Equation (3) then simplify with Assumption to
W1+1 = Wi - B(1, - aB, B,) Baw
(5)
B++1 = B, -BB,El,, + aBB.E. B,.
(6)
where wt still given by Equation 44. Moreover, Lemma 15 in the Appendix allows with As- sumption I to compute an exact expression of Blw,, s
E E/w,, w1 (Ig- -a,, B.) (IN- aB, Bi) + a² BB B,E,, B.
+ a² B. (Beww, B, + B.E.B. + (1 Tr(2.) + + B..
Mhn
The first line is the covariance obtained for an infinite number of samples. The second line comes from errors due to the finite number of samples and the label noise. As a comparison, MAML also updates matrices B in the inner loop. which then intervene in the updates of W/ and B, Because of this entanglement, the iterates of first-order MAML (and hence its analysis) very cumbersome.
LEARNING GOOD REPRESENTATION
Given the complexity of its iterates, FO-ANIL is very intricate analyze even in the simplified set- ting of infinite tasks. The objective function is non-convex in its arguments and the iterations involve high-order terms in both wt and Bt, as seen in Equations (5) to 77) Theorem [ yet characterizes convergence towards some fixed point (of the iterates) satisfying a number of conditions.
Theorem 1. Let Bo and wo be initialized such that B. Bo is full rank,
| 0
[_] Min dz )
|| = (c.Amin (E.)),
Min
where (m.) is the smallest eigenvalue of E.,, σ² Tr(E.) + o² and the O notation hides universal constants. Let also the step sizes satisfy 2 B and O (1/6).
Then under Assumption 7 FO-ANIL (given by Equations (5) and 66 with initial parameters Bo: wa. asymptotically satisfies the following
lim B 1 B, 500
lim Br O, t1200
lim B. B,B. B. = a mia +
1 Min I T
("in
Min + 1 Σ. +Iss) )1))
(8)
g2
where B.d. (ddx(dk) is an orthogonal matrix spanning the orthogonal of col( i.e.,
B I and B. B. =0.
An extended version of Theorem I and its proof are postponed to Appendix C We conjecture that Theorem[]ho hoolds with arbitrary task covariances Σ, beyond the identity covariance in Assump- tion I Before discussing the implications of Theorem we provide details on the proof strategy.
The proof is based on the monotonic decay of [[ B1 |Bawell and the monotonic increase of B_ B, B, in the Loewner order sense. As these three quantities interrelated, simultaneously controlling them challenging. The initialization given in Theorem Taciievss this by conditioning the dynamics to be bounded and well-behaved. The choice of is also erucial as it guarantees the
4

---

Published as a conference paper at ICLR 2024
decay of [BBwellk after ||BT, Bellz decays. While these two quantities decay, A, B BBB; B. follows a recursion where B__ B, and B,w, act as noise terms. The proof utilizes two associ- ated recursions to respectively upper and lower bound A, (in Loewner order) and then show their monotonic convergence to A.. A more detailed sketch of the proof could be found in Appendix B
Theorem T states that under mild assumptions on initialization and step sizes, the parameters learned by FO-ANIL. verify three key properties after convergence:
1. Be is -eficief,,,,,,, i.e., FO-AFIILLLLLLI learns to ignore the entire d dimensional orthogonal subspace given by B,, I. as expressed by the first limit in Equation (8)
2. The learned initialization yields the zero function, as given by the second limit in Equa- tion (8). Note that wy does not necessarily converge to O; however, it converges to the null space of B., thanks to the third property. Although intuitive, showing that B, converges to the mean task parameter (assumed 0 here) is very challenging when starting away from This property is crucial for fast adaptation on a new task.
it, as discussed in Section 4
3. BJ Bxx - B. is proportional identity. Along with the first property, this fact implies that the learned matrix Bx exactly spans col (B.)). Moreover, its squared singular values scale
allowing to perform rapid learning with single gradient step of size
These three properties allow to ohtain a good performance on a new task after a single gradient descent step, intended by the training objective of ANIL, The generalization error at test time is precisely quantified by Proposition [ in Section [.1] In addition, the limit points characterized by Theorem are shown be global minima of the ANIL objective in Equation (I in Appendix
Interestingly, Theorem | holds for quite large step sizes a, 3 and the limit points only depend on these parameters by the a -1 scaling of A.. Also note that A. - L when min + co. Yet, there is some shrinkage of A, for finite number of samples, that is significant when mi is of order of the inverse eigenvalues of 52 X., This shrinkage mitigates the variance of the estimator returned after a single gradient step, while this estimator is unbiased with no shrinkage (min oc).
Although the limiting behavior of FO-ANIL holds for any finite Min, the convergence rate can be arbitrarily slow for large min- In particular, FO-ANIL becomes very slow to unlearn the orthogonal complement of col [,) when min is large, as highlighted by Equation (14) in Appendix ( At the limit of infinite samples min = oo, FO-ANII. thus does not unlearn the orthogonal complement and the first limit of Equation (8) in Theorem does not hold anymore. This unlearning is yet crucial at test time, since reduces the dependency of the excess risk from k to (see Proposition I].
3.1 FAST ADAPTATION TO A NEW TASK
Thanks to Theorem I FO-ANIL learns the shared representation during pretraining. It is yet unclear how this result enhances the learning of new tasks, often referred as finetuning in the literature.
Consider having learned parameters (B,ú) pdxk'' IR. following Theorem
B B 0: Bi 0:
B B, =A.
We then observe a new regression task with miett observations (X,v) 1. Me xd x gitiout and parameter w. E & such that
y= xB +
(10)
where the entries of = are i.i.d. centered a sub-Gaussian random variables and the entries of X are i..dd standard Gaussian variables following Assumption [ The learner then estimates the regression parameter of the new task doing one step of gradient descent:
Wiest ② - aV u ((B X.y) = á + aB Eton B.w. +
B
(11)
Mcest
X As in the inner loop of ANIL, single gradient step is processed here. Note that it is unclear whether single or more gradient steps should be run at test time. No- tably, (B, w) has not exactly converged in practice, since we consider a finite training time: B is thus full rank. The least squares estimator of the linear regression with data (XB,v) y) might then lead to overfitting. Rumning just a few gradient steps can be helpful by preventing overfitting since it implicitly regularizes the norm of the estimated parameters (ao et al. 2007 Neu & Rosasco,
with Etest == Moncs
5

---

Published as a conference paper at ICLR 2024
2018). The best strategy (e.g. the number of gradient steps) to while finetuning is intricate
problem, independently studied in the literature (see e.g. Chua et al. 2021 Ren et al. 2023) and is out of the scope of this work. Additional details are provided in Appendix
When estimating the regression parameter with Bwtest, the excess risk on this task is exactly below allows to bound the risk on any new observed task.
| B2 B, Proposition |
Proposition 1. Let B, Weest satisfy Equations (9) and (11 for a new task defined by Equation 10 If Mhest 2 k; then with probability at least 4e 2,
|BWrrest B.w+l12 -0
+ /2mmman (x..)
100,122 122 + wll
k o
+
k Maest
Min
Meest
A more general version of Proposition T and its proof are postponed Appendix [ The proof relies on the exact expression of Wrest after a single gradient update. The idea is to decompose the difference Bwrest w, in three terms, which are then bounded using concentration inequalities.
The first two terms come from the error due to proceeding a single gradient step, instead of converg- ing towards the ERM weights: the first one is the bias of this error, while the second one is due to its variance. The last term the typical error of linear regression on lt dimensional space. Note this bound does not depend on the feature dimension d (nor k'), but only on the hidden dimension k.
When learning a new task without prior knowledge, e-g., with simple linear regression on the d-dimensional space of the features, the error instead scales as 0 Must (Bartlett al. 2020
FO-ANII. thus leads to improved estimations on new tasks, when it beforehand learned the shared representation. Such a learning is guaranteed thanks to Theorem [ Surprisingly, FO-ANII. might only need single gradient step to outperform linear regression on the d-dimensional feature space, as empirically confirmed in Section 5 As explained, this quick adaptation is made possible by the a scaling of B, which leads to considerable updates after a single gradient step.
DISCUSSION
No prior structure knowledge. Previous works model-agnostic methods and matrix fac- torization consider a well-specified learning architecture, i.e., = k (Tripuraneni et al. 2021 Thekumparampil al. 2021 Collins al. 2022). In practical settings, the true dimension k hid- den, and estimating it is part of learning the representation. Theorem iistead states that FO-ANIL recovers this hidden true dimension It asymptotically when misspecified (k' k) and still learns good shared representation despite overparametrization (c.g., k' = d). Theorem I thus illustrates the adaptivity of model-agnostic methods, which we believe contributes to their empirical success.
Proving good convergence of FO-ANIL despite misspecification in network width is the main tech- nical challenge of this work. When correctly specified, it sufficient to prove that FO-ANIL learns the subspace spanned by B., which is simply measured by the principal angle distance by Coolins et al. (2022). When largely misspecified (k' = d), this measure is always and poorly reflects how good is the learned representation. Instead of a single measure, two phenomena are quantified here. FO-ANIL indeed not only learns the -iw-eeennioonal subspace. but it also unlearns its orthogonal complement More precisely, misspecification sets additional difficulties in controlling simultane- ously the variables Wi and B, through iterations. When k'' = k; this control is possible by lower bounding the singular values of B,- A similar argument is however not possible when k' > k; as the matrix B, is now rank deficient (at least asymptotically). To overcome this challenge, we use a different initialization regime and analysis techniques with respect Saunshi et al. (2020):(Coliin et al. (2022). These advanced techniques allow to prove convergence of FO-ANIL with different assumptions on both the model and the initialization regime, as explained below.
Superiority of agnostic methods. When correctly specified (k' k),, model-agnostic methods do not outperform traditional multi-task learning methods. For example, the Burer-Monteiro factoriza- tion minimizes the non-convex problem
min BERaXk WER 2N EM (,(
(12)
A Ithough Saunshi et al. (2020) consider misspecified setting, the orthogonal complement is not unlearned in their case, since they assume an infinite number of samples per task (see Infinite tasks model paragraph).

---

Published as a conference paper at ICLR 2024
where W stands for the i-th column of the matrix W. Tripuraneni et al. (021) show that any local minimum of Equation (12) correctly learns the shared representation when K' = k. However when misspecified (e.g., taking k' d), there is such guarantee. In that case, the optimal B need to be full rank (e.g., B = Iæ) to perfectly fit the training data of all tasks, when there is label noise. This setting then resembles running independent d-dimensional linear regressions for each task and directly leads to a suboptimal performance of Burer- Monteiro factorizations, as illustrated in Section 5 This is another argument in favor of model-agnostic methods in practice: while they provably work despite overparametrization. traditional multi-task methods priori do not.
Although Burer-Monteiro performs worse than FO-ANIL in the experiments of Section 5 it still largely outperforms the single-task baseline. We believe this good performance despite over- parametrization might be due to the implicit bias of matrix factorization towards low-rank solu- tions. This phenomenon remains largely misunderstood in theory, even after being extensively stud- ied (Gunasekar et al. 2017 Arora et al., 2019: Razin & Cohen [020 Li et al., 2021). Explaining the surprisingly good performance of Burer-Monteing thus remains a major open problem.
Infinite tasks model. A main assumption in Theorem T is the infinite tasks model, where updates are given by the exact (first-order) gradient of the objective function in Equation (I). Theoretical works often assume a large number of tasks to allow a tractable analysis (Thekumparampil et al. 2021 BBuursier et al., 2022). The infinite tasks model idealises this type of assumption and leads to simplified parameters updates. Note these updates, given by Equations (5) and (6), remain intricate to analyze. [Saunshi aSansze. et al. Collins et al. (2022) instead consider an infinite number of samples per task, i.e., min po. This assumption leads to even simpler updates, and their analysis extends to the misspecified setting with some extra work, as explained in Appendix G Collins et al. (2022) also extend their result to a finite number of samples in finite-time horizon, using concentration bounds on the updates to their infinite samples counterparts when sufficiently many samples are available.
More importantly, the infinite samples idealisation is not representative of few-shot settings and some phenomena are not observed in this setting, First, the superiority of model-agnostic methods is not apparent with an infinite number of samples per task. In that case, matrices B only spanning col( (.) also minimise the problem of Equation (12). potentially making Burer-Monteiro optimal despite misspecification. Second, a finite number of samples is required to unlearn the orthogonal of col (B.). When Min = oc, FO-ANIL does not unlearn this subspace, which hurts the performance at test time for large k', as observed in Section 5 Indeed, there is no risk of overfitting (and hence no need to unlearn the orthogonal space) with an infinite number of samples. On the contrary with a finite number of samples. FO-ANIL tends to overfit during its inner loop. This overfitting is yet penalized by the outer loss and leads to unlearning the orthogonal space.
Extending Theorem to finite number of tasks is left open for future work. Section 5 Jempirically supports that similar result holds. A finite tasks and sample analysis similar to Collins et al. (2022) is not desirable, as mimicking the infinite samples case through concentration would omit the unlearning part, as explained above. With misspecification, we believe that extending Theorem T to a finite number of tasks is directly linked to relaxing Assumption T] Indeed, the empirical task mean and covariance are not exactly 0 and the identity matrix in that case. Obtaining a convergence result with general task mean and covariance would then help in understanding the finite tasks case.
assumes zero mean task parameters, μu = E[w,, . = O. Considering non-zero task mean adds two difficulties to the existing analysis. First, controlling the dynamics of w; is much harder, as there is an extra term μe, in its update, but also B,W; converges to B.p., # 0 instead. Moreover, updates of B, have an extra asymmetric rank 1 term depending on n.. Experi- ments in Appendix yeet support that both FO-ANIL and FO-MAML succeed when μ., * 0.
Limitations. Assumption I
In addition, we assume that the task covariance Σ, is identity. The condition number of Σ, is related to the task diversity and the problem hardness (Tripuraneni et al.) 2020 [Tekkmparampil et al. 2021: Collins et al., (202)). Under Assumption [ the task diversity is perfect (i.e., the condi- tion number is 1), which simplifies the problem. The main challenge in dealing with general task covariances is that the updates involve non-commutative terms. Consequently, the main update rule of B. B, B. no longer preserves the monotonicity used to derive upper and lower bounds on its iterates. However, experimental results in Section 5 suggest that Theorem [ still holds with any di- agonal covariance. Hence, we believe our analysis can be extended to any diagonal task covariance. The matrix E,, being diagonal is not restrictive, it is always the case for a properly chosen B..
7

---

Published as a conference paper at ICLR 2024
Lastly, the features X follow standard Gaussian distribution here. It is needed to derive exact expression of (wr, with Lemma 15 which can be easily extended to any spherically symmet- ric distribution. Whether Theorem [ holds for general feature distributions yet remains open.
Additional technical discussion. For space reasons, we leave the technical details on Theorem to Appendix A In particular, we remark that our initialization only requires full-rank initialization without any initial alignment and describe how derive a rate for the first limit in Theorem [ which shows a slowdown due to overparametrization similar to the previous work by Xu & Du 2023)
EXPERIMENTS
This section empirically studies the behavior of model-agnostic methods on a toy example. We consider setup with a large but finite number of tasks limited number of samples per task
5000, feature dimension d = 50, 30, small hidden dimension &o 5 and Gaussian label noise with variance o² 4. We study largely misspecified problem where d. To demonstrate that Theorem [ holds more generally, we consider a non-identity covariance Σ, proportional to diag(1, ;.k,,, k)) Further experimental details, along with additional experiments involving two-layer and three-layer ReLU networks, can be found in Appendix
FD ANIL (inf tasks
Lo ANIL (inf samples)
FO ANIL
FO MAML
Burer Monteiro
Xua BB 19
10
100000
10004
Iterations
Evolution of smallest (left) and largest (right) squared singular value of B. B, during training. The shaded area represents the standard deviation observed over 10 runs.
Figure 2:
To observe the differences between the idealized models and the true algorithm, FO-ANIL with finite samples and tasks is compared with both its infinite tasks and infinite samples versions. is also compared with FO-MAML and Burer-Monteiro factorization.
Figure 2 first illustrates how the different methods Iearn the ground truth subspace given by B.. More precisely, shows the evolution of the largest and smallest squared singular value of B_ B, On the other hand, jilustates how different methods unlearn the orthogonal complement of col(B.), by showing the evolution of the largest and averaged squared singular value of B Br.
ANIL (inf tasks)
FC ANIL samples)
FO-ANIL
FO MAML
Burer Monteiro
B (b))
A [ [r)
5-
5000
10000
15000
20000
25000
5000
10000
15000
20000
25000
Iterations
Iterations
Figure 3: Evolution of average (leff) and largest (right) squared singular value BT B, during training. The shaded area represents the standard deviation observed over runs.
Finally, Table comppress the excess risks achieved by these methods on a new task with both 20 and 30 samples. The parameter is estimated by a ridge regression on (X B, y). where 2 is the represen- tation learned while training. Additionally, after a single gradient descent update. These methods are also compared with the single-task base- line that performs ridge regression the d-dimensional feature space, and the oracle baseline that
report the loss obtained for model-agnostic methods
8

---

Published as a conference paper at ICLR 2024
directly performs ridge regression the ground truth k-dimensional parameter space. Ridge gression is used for all methods, since regularizing the objective largely improves the test loss here. For each method, the regularization parameter is tuned using grid grid-search over multiple values.
Table I: Excess risk evaluated on 1000 testing tasks. The number after + is the standard deviation over 10 independent training runs. For model-agnostic methods, 1-GD refers single gradient descent step at test time; Ri Ridge refers to ridge estimator with respect to the learned representation.
Must = 20
Must 30
Single-task ridge
1.84 + 0.03
1.63 + 0.02
Oracle ridge Burer-Monteiro
0.50 + 0.01
0.34 0.01
1.23 + 0.03
1.03 + 0.02
1-GD
Ridge
1-GD
Ridge
FO-ANIL FO-MAMI.
0.81 + 0.01
0.73
0.64 0.01
0.57 + 0.02
0.81 0.01
0.04
0.63 0.01
0.58 + 0.01
FO-ANIL infinite tasks
0.67 0.77 0.01
0.60 0.60
0.52
FO-ANIL infinite samples
1.78 + 0.02
+
1.19 = 0.01
0.84 0.02
As predicted by Theorem 1 FO-ANIL with infinite tasks exactly converges to A.. More precisely, quickly learns the ground truth subspace and unlearns its orthogonal complement as the singular values of B_ B, decrease to O, at the slow rate given in Appendix H FO-ANIL and FO-MAML with a finite number of tasks, almost coincide. Although very close to infinite tasks FO-ANIL, they seem unlearn the orthogonal space of col(B,) even more slowly. In particular, there are few directions (given by the maximal singular value) that are unlearned either very slowly or up to a small error. However on average, the unlearning happens at a comparable rate, and the effect of the few extreme directions is negligible. These methods thus learn a good representation and reach an excess risk approaching the oracle baseline with either ridge regression or just single gradient step.
On the other hand, as predicted in Section 4 FO-ANIL with an infinite number of samples quickly learns col(B.). but does not unlearn the orthogonal complement. The singular values along the orthogonal complement stay constant. A similar behavior is observed for Burer-Monteiro factoriza- tion: the ground truth subspace quickly learned, but the orthogonal complement is not unlearned. Actually, the singular values along the orthogonal complement even increase during the first steps of training. For both methods, the inability of unlearning the orthogonal complement significantly hurts the performance at test time. Note however that they still outperform the single-task baseline. The singular values along col( B.) are indeed larger than along its orthogonal complement. More weight is then put on the ground truth subspace when estimating a new task.
These experiments confirm the phenomena described in Sections 3 and Model-agnostic methods not only learn the good subspace, but also unlearn its orthogonal complement. This unlearning yet happens slowly and many iterations are required to completely ignore the orthogonal space.
6 CONCLUSION
This work studies first-order ANIL in the shared linear representation model with a linear two- layer architecture. Under infinite tasks idealisation, FO-ANIL successfully learns the shared, low-dimensional representation despite overparametrization in the hidden layer. More crucially for performance during task specifci finetuning, the iterates of FO-ANIL not only learn the low- dimensional subspace but also forget its orthogonal complement. Consequently, single-step gra- dient descent initialized on the learned parameters achieves a small excess risk on any given new task. Numerical experiments confirm these results and suggest they hold in more general setups, e.g., with uncentered, anisotropic task parameters and a finite number tasks. As consequence, work suggests that model-agnostic methods are also model-agnostic in the sense that they suc- cessfully learn the shared representation, although their architecture is not adapted to the problem parameters. Extending our theoretical results these ods, such as MAML, remains open for future work. Lastly, our work presents a provable shared representation learning result for the pretraining of meta-learning algorithms. Thus, it connects to the literature on representation learning with pretraining; in particular, it demonstrates a slowdown due to overparametrization that has been recently demonstrated in supervised learning.
general settings more intricate meth-

---

Published as a conference paper at ICLR 2024
REFERENCES
Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(61):1817-1853, 2005. URL
http:/ /ttp:///mlr www.or/g/pggga pwwpsspppps/paaa/ae /vw h115//ndo05a.ht.. html
v6
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to train your MAML. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreviet net.///wweeeeeeeeeeeeeeeeeeeeeee forum?id=
HJGven05y7
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence A'clc Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Anmual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp- 7411-7422, 2019. URL htttptss//////eeeeeeii
new..eeurgie ips. ssc//paper/221/haatii C0c783 60C7855focraa011111aa6662222222 Abst html
Peter L. Bartlett, Philip M. Long. Gábor Lugosi, and Alexander Tsigler. Benign overfitting in
linear regression. Proceedings the National Academy of Sciences, 1 117748):30033-0077 2020.
ISSN 0027-8424, 1091-099199999 doi: 10. 20.1073/pnas.19977 1907378117. URL (tt.ht:::: /ttp:::::///ooo./1/////
1073/pnas. 1attaass19930 1907376117
Etienne Boursier, Mikhail Konobeev, and Nicolas Flammarion. Trace norm regularization for multi- task learning with scarce data. In Po-Ling Loh and Maxim Raginsky (eds.), Conference on Learning Theory, 2-5 July 2022, London, UK, volume 178 of Proceedings of Machine Learn- ing Research, pp. 1303 1327. PMLR, 2022. URL 0178/boursier2220 html
ml press/
Bin Cheng, Guangean Liu, Jingdong Wang, Zhong Huang, and Shuicheng Yan. Multi-task low- rank affinity pursuit for image segmentation. In Dimitris N. Metaxas, Long Quan, Alberto San- feliu, and Lue Van Gool (eds.), IEEE International Conference on Computer Vision ICCV 201 I, Barcelona, Spain, November 6-13, 2011, pp. 2439-2446. IEEE Computer Society, 2011. doi: 10. /10. 1109/CCC.2011 6116528. URL hitps: //iiiiii....// org 10. 1ww.11199/ICCC 2011. 6126528
Kurtland Chua, Qi Lei, and Jason D. Lee. How fine-tuning allows for effective meta-learning, In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. URL
8871-8884 2021.
neurii.........................p CC wwwpcc/pppp...cc.cc.. 202 202//ha21/hhss /hash/ -bt.:999--bbtttttta .html
Liam Collins, Aryan Mokhtari. Sewoong Oh, and Sanjay Shakkottai. MAML and ANIL provably learn representations. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learn- ing Research, pp. 4238 4310. PMLR, 2022. URL v162/col1 http:/oooinnaa....ooooooo html
mmr. .press/
Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL net www.mooiidoooo.................e F
https://openrev/
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-learning algorithms. In Silvia Chiappa and Roberto Calandra (eds.), ), The 23rd International Conference on Artificial Intelligence and Statisties, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pp. 1082-1092. PMLR, 2020. URLhttp: / http:////occeeeengg.aeee mlr pttss//100/ fa 1ah20a html
10

---

Published as a conference paper at ICLR 2024
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. Generalization of model-agnostic meta-learning algorithms: Recurring and unseen tasks. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural In- formation Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp- 5469-5480, 2021, URL 2b763288faed3 777700748aaee15aaaeae
Abstract.
Chelsea Finn. Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 I August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017. URL ngs mlr.. wrw.s/esssee.
html
Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Sre- bro. Implicit regularization in matrix factorization. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 581 2017. URL
neur. wwtpeeewwpp...pppppeeeeeeeaeee wwtppppppe///////aaaaaea -bsttsttattAA html
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. IEFE Transactions on Puttern Analysis and Machine Intelligence, 4499)5544--169, 2022. doi: 10. 10.109/TPAMM222211377922 11C9/TPAMI 2021. 3079209
URLhttps: /ttttps://do..org//333
Kaiyi Ji, Junjie Yang, and Yingbin Liang. Theoretical convergence of multi-step model-agnostic
meta-learning. Journal of Machine Learning Research, 23:29:1-29:41, 2022. URL http://
wwwl..rrg/paper
202222222220 221/20072222 html
Haotian Ju, Dongyue Li, and Hongyang R. Zhang. Robust fine-tuning of deep neural net- works with bassed generalization guarantees. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 10431-10461. PMLR, 2022. URL https: 7/proceedings mlr. press/vill / 62 ju22a. html
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In 9th International Conference on Learning Representattons, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https::/openreewe
net/ forum?idiioooooo ? 7 5m5H7R
Gergely Neu and Lorenzo Rosasco. Iterate averaging as regularization for stochastic gradient de- scent. Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet (eds.), Conference On Learn- ing Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018, volume 75 of Proceedings of Ma- chine Learning Research, pp. 3222-3242. PMLR, 2018 URL utLhttp::/ pecttp:/proce /ttp::/d//occeeinggs.. saapodiiinssmm.......mm.mm.. mlrr. ppress/v ww/55//nu8l80 8tmm
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. CoRR, abs/1803.02999, 2018 IRtlhhttp///arxix www.rriww...rrooeooooooo abs 1803. 000.8......0999
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of MAML. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. pattts::/oppnne
ew. www.wweeeeeeee..e...ee
Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ties Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, / F21 e255 8 0 f 2.5 0aw.eaacccc444e 9ew. 4 http:/iis...pppppp/////////aaeee

---

Published as a conference paper at ICLR 2024
Yi Ren, Shangmin Guo, Wonho Bae, and Danica J. Sutherland. How to prepare your task head for finetuning. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL /tLLittpp///peeeeeeeeee ?df? 2iiiiidvvXX/pree----
Philippe Rigollet and Jan-Christian Hütter. High-dimensional statistics, 2023. URL https:/ //
19244
Angelika Rohde and Alexandre B. Tsybakov. Estimation of high-dimensional low-rank matrices.
The Annals of Statistics, 39(2), April 2011. ISSN 0090-5364. doi: 10. 10.1214/10-a08860. URL
httpp://at doi. www.oogg./......... 1214/121/1 -108660-1-88600
Nikunj Saunshi, Yi Zhang, Mikhail Khodak, and Sanjeev Arora. A sample complexity sepa- ration between non-convex and convex meta-learning. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 8512-8521. PMLR, 2020. URL .ma.ediiiin.m..................m mlr. arzessvvilgn wwt.s//aannssaaaaaaaa htm.
Nikunj Saunshi, Arushi Gupta, and Wei Hu. A representation learning perspective on the importance of (trainvvaldatte splitting in meta-learning. Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9333-93333 9343. PMLR, 2021. URL http://proceedingg mlr.
htt.://nnw.......ps/s/s......... wwwps//wss///epppppp. wttpss/113//asssss//ees htt.1///aunshhiie... html
Kiran Koshy Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. Statisti- cally and computationally efficient linear meta-representation learning. In Marc' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 18487-18500, 2021. URL httptss:pppocceeinngggg/ mew.eeeeiidnnn....e.......... wwtprrippp.ccppppp/////2 html]
Nilesh Tripuraneni, Michael I. Jordan, and Chi Jin. On the theory of transfer learning: The impor- tance of task diversity. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL 5958 htt.b/fcccc/4466ff3e32
neurips cc.ppaper//22//hast// Abstraat html
Nilesh Tripuraneni, Chi Jin, and Michael I. Jordan. Provable meta-learning of linear representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10434-10445 PMLR, 2021. html
mlr. wttppprs///11//rpprrrreeeeeeeeee
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. In Yon- ina C. Eldar and Gitta Kutyniok (eds.), Compressed Sensing, pp. 210-268 Cambridge University Press, 2012. doi: 10.1017/CB0978001
URL hittps://don http://cc...gg///////// orr.:/../g11001111101111
cbo9780511 077800117943000
606
Weihang Xu and Simon S. Du. Over-paaamettrrizatoon exponentially slows down gradient descent for learning a single neuron. In Gergely Neu and Lorenzo Rosasco (eds.), The Thirty Sixth Anmual Conference on Learning Theory, COLT 2023, 12-15 July 2023, Bangalore, India, volume 195 of Proceedings of Machine Learning Research, pp. 1155-1198. PMLR, 2023. URL https: //proceedings. mmlr wrwps/pepp....... V195 xu23a.
wttp:///...///1.1.1eee html
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constr: Approx., 26(2):289-315, 2007. ISSN 0426 500365-006-0663-2. URL
doi: 10.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL DISCUSSION
Initialization regime. Theorem D requires a bounded initialization to ensure the dynamics of FO- ANIL stay bounded. Roughly, we need the squared norm of B,, Bo to be O ((am)) antee |B S a for any t. We believe the Min dependency is an artifact of the analysis and is empirically not needed. Additionally, we bound Wp to control the scale of E El s that appears in the update of B,. A similar inductive condition is used by Collins et al. (022).
guar-
More importantly.
analysis only needs full rank B Bu, which holds almost surely for usual initializations. Collins et al. (2022) instead require that the smallest eigenvalue of B. Bo is bounded strictly away from which does hold when d > K. This indicates that their analysis covers only the tail end of training and not the initial alignment phase.
contrast with the convergence result Collins et al. (2022). Theorem does not provide any convergence rate for FO-ANIL but only states asymptotic results. Appendix H
Rate of convergence.
provides an analogous rate for the first limit of Theorem | B Bell
2 = O ) Due to mis-
specification, this rate is slower than the one by Collins et al. 2022 (exponential vs. polynomial).
similar slow down due to overparametrization has been recently shown when learning a single ReLU neuron (Xu & Du, 2023). In our setting, rates are more difficult obtain for the second and third limits, as the decay of quantities of interest depends on other terms in complex ways. Remark that rates for these two limits
studied by Collins et al. (2022). In the infinite samples limit, rate for the third limit can yet be derived when k: =' K'
Relaxation to finite number of tasks. In the limit of infinite tasks, Theorem proves that the asymptotic solution is low-rank and the complement B.,, is unlearned. Figure shows this behavior with a relatively big initialization. For pretraining with a finite number of tasks, the decay is not towards 0 but to small residual value. The scale of this residual depends on the number of tasks (decreasing to when the number of tasks goes to infinity) and possibly on other problem parameters such as k and Most notably, when a very small initialization is chosen, the singular values in the complement space can increase until this small scale.
This indicates that Theorem relaxed to finite number of tasks will include non-zero but small component in the orthogonal space. In our experiments, these residuals do not hurt the finetuning performance. Note that these residuals also present for simulations of FO-ANIL with infinite tasks due to the finite time horizon.
Theoretical analyzes of model-agnostic meta-learning. Other theoretical works on model- agnostic meta-learning have focused on convergence guarentees (Fallah et al. 2020 Ji et al.) 2022) or generalization (Fallah al. 202T). contrast, this work focuses on pretraining of model-agnostic meta-learning and learning of shared representations under the canonical model of multi-task learn- ing.
13

---

Published as a conference paper at ICLR 2024
B SKETCH OF PROOF
The challenging part of Theorem is that B, E Rdxk' involves two separate components with different dynamics:
B, - B.B. B. B. B B.
The first term B, B, eventually scales in a 1/2 whereas the second term B Br converges to 0, resulting in nearly rank-deficient B. The dynamics of these two terms and W/ are interdependent, which makes it challenging to bound any them.
Regularity conditions. The first part of the proof consists in bounding all the quantities of interest. Precisely, we show by induction that the three following properties hold for any t,
1. I1,"" B.1 = ||,,12,
2. ||wall S ||
3. | Bell2 S Bolla (13)
Importantly, the first and third conditions, along with the initialization conditions, imply || < -1 The monotonicity of the function نe described below leads to [| B++1 S || 2l. Also, using the inductive assumptions with the update equations for B i B. and w, allows show that
both the second and third properties hold at time 1.
Now that the three different quantities of interest have been properly bounded, we can show the three limiting results of Theorem
Unlearning the orthogonal complement. We first show that lim B B, = 0. Equa- tion O directly yields BT, Be+1 BT, B, (Ix ww.l). The previous bounding con- ditions guarantee for a well chosen /3 that [[E[w,,,w; ",, I B-1. Moreover thanks to Equation (7) w., 2/2 0² B. B,B.T B.,,, which finally yields
| B 12 S a²g o² min
B 1.13 B.
(14)
Learning the task mean. We can now proceed to the second limit in Theorem I B,w, can be decomposed into two parts, giving B,wwll I [| Bew || [[ Bewelle. As /elll bounded and ||_T, Bella converges to O, the second term vanishes. A detailed analysis on the updates of B. B,w gives
[ B (1
8
+ a:31E..112
||B_ B, ||2 + (1
4a
which implies that lim B,w, for properly chosen a; B.
Feature learning. We now focus on the limit of the matrix A, = B. B,B, B. E Raxk, The recursion on A, induced by Equations (5) and (6) is as follows,
A4+1 (I, (A.) A (I + (A))(
(15)
288ym (II + a&R.( B B,U, B.) + B.U B..
(a² B I13) and U, is some noise term defined Appendix C From there, we can define functions St and J
where Sym(A) (A+ (A,) = [I
amm
A.)
min
min
approximating the updates given in Equation 15 such that
(A, A (A.).
Moreover, these functions preserve the Loewner matrix order for commuting matrices of interest. Thanks to that, we can construct bounding sequences of matrices (AF). (A!") such that
3. A A Using the first two points, can then show that both sequences Af, A are non-decreasing and converge to A, under the conditions of Theorem The third point then concludes the proof.
A (A
2. 1 = 8 (AA))
14

---

Published as a conference paper at ICLR 2024
C PROOF OF THEOREM
The full version of Theorem is given by Theorem 2 In particular, it gives more precise conditions on the required initialization and step sizes.
Theorem 2. Assume that e₁ < 1, 02 are small enough positive constants verifying
Min + C
C1 (c2 +2))
<)min(E.);
Min
2m (Min + +1)
and B are selected such that the following conditions hold:
1. 3 S a.
2. 1
2 4 E1,
a²
3.
C2
min +2
C102
2
Min +
[
+
4
Min (min
aB
Mini
2m (Min + 1)
min
Min
+ 1)3
4.
2
6
(11,112
22
aB
min
+1
Furthermore, suppose that parameters Bn and are initialized such that the following three con- ditions hold:
1. B. Bo is full rank,
2. BBlll < a min +
3. 1woll3 < aC2-
Then, FO-ANIL (given by Equations (5) and (6) with initial parameters Bo, we inner step size a, outer step size B, asymptotically satisfies the following
lim B BT, =
(16)
lim B,w, = 0,
(17)
1 min
Min +
lim B, B,B, B. = 1-00
I
E. + I&a
(18)
a Min
The main tools for the proof are presented and discussed in the following subsections. Section C.1 proves monotonic decay in noise terms provided that B, bounded by above. Section C.2 pro- vides bounds for iterates and describes the monotonicity between updates. Section C.3 constructs sequences that bound the iterates from above and below. Section C.4 presents the full proof using the tools developed in previous sections. In the following, common recursions relevant objects are derived.
15

---

Published as a conference paper at ICLR 2024
The recursion on B₁ defined in Equation (6 leads to the following recursions on Ct B, Be Rkxk' and D2 == B, B. E ((ddk)xk'
C411= = 11 + aa IN a(min + + C.C.)) E.
a²B
(11B,r 112 Tr(E.) + 02) C,C.T Ce
Min
Min
aBB aB, Bi) wews (Iv - aB; Bi) + a² B₁ Bewwe B, B.
- 3Ce
(IN (dw-
min
a² [[Breor|l++rriee ) 0²) D. Dz
+
(19)
min
D41 Dr IN - -3 -aB, Bi) wwsm (Ix - aB, B.) a2B B, Baww, B, Br
Min
a² a²mi +1) C. E,C-
a²B
B1 Tr(E.) + + B, Bi
(20)
min
Min
For ease of notation, let 0² = Tr((E.,)) 02 8c
B 11 + 0² and define the following objects.
R(A,T): = I a(Min + 1) A E.
a (ō² +T) A, R&(A) = R(A, B.ww)
Min
Miln
a² B. Bawww T B. Bi.
W, = (Iv - aB, BB) www!
T (I
aB, Bi) +
Min
U. = W + a² DD De.
mim
a² (m + C.T.C. + a² a S,B. B..
V/ W. +
(21)
Min
min
Then, the recursion for A, C,C,T is
A (In + (B)) Ar (I + a3R((A))) + &CUUCCT
(22)
- 3 (Ix + a.BR(AA)) C.U.C - BCUU,CT (T+ aBB((A)))
C.1 REGULARITY CONDITIONS
Lemmas and 2 control ||w/|| and |eeell across iterations, respectively. Lemma shows that [[Cwwell2 decaying with noise term that vanishes as Dtlla gets small. Corollary combines all three results and yields the first two claims of Theorem 1
lim B., B, 0, lim Ba -
under the assumption that conditions of Lemmas 2 and 3 are satisfied for all t. Lemmas 4 and bound ||U]|22 and [|W] ensuring that the recursions of A, are well-behaved in later sections.
Lemma 1. Assume that
TO 3 B, B, 3 Min C1 I
a min +
for constants 0 S co. <i < 1 such that Boo (1 c1)) S Min + 1. Then,
||we+1|12 I (1 11- co(1
000 - -11
|| well2
Min + 1
Proof. From the assumption,
Min + Is 3 IN aBT B, (1 acc)
and
B. B. aB. Bi) z co(1 c1 I
min
16

---

Published as a conference paper at ICLR 2024
Recalling the recursion for wy defined in Equation
(5).
(1
3 c0o(1 (1)
||w++ll2 S
Min +
Lemma 2. Assume that
1
|
| S co.
for a constant c 2 and a, 3 satisfy
1
Min
+2
c
c+
((mmn + 1)|2=112 + o²).
(23)
a
Min
min
1
28
(24)
aB3
Minn
Then,
[D D2 S (1
a²3 D1 |D.D,, ll.
Min
Proof. The recursion on D is given by
D4t1 De(T. - DV
where we recall V, is defined in Equation ||Villa S t. By the definition V,
21 First step is to show I BV, ł 0 by proving
(min + 1)
Vi S [Well +
IITTT E,C,
5. l| B, B1ll2.
min
min
(A)
(B)
(0)
Term A is bounded by Lemma
For the term
(B)
using ||C||12 B Bell2 S |/l|1
I.
1.
12-
a
Term C is bounded as
| Billa = |BBl 1
& B + 22
+ + sc+
a
Combining three bounds and using the condition in Equation 23
[IVo < Min 1
a
((mnn + 1)]| σ²) + ac S 3
min
min
Therefore, is possible to upper bound D241 D as follows,
DIt = D. DV
DI -V.)D.
a²B DD Dr
Dr
Is
D.T a23 o² D.D. D.D.T
min
I
Mia
Let D.D.T = =Q,S be the SVD decomposition of D.D,T in this proof. Then,
a² a² 52 Ω
D141 1 22
St
min
Note that
Min 38
by Equation and for any 81 S 82 a
Min 20
a² 2 2 (1
a² 3 o2 21))
S2
Min
min
17

---

Published as a conference paper at ICLR 2024
by monotonicity of + x(1 022) Hence, the largest eigenvalue of St, s(1 the largest eigenvalue of (St
a²8 g2s)
min
mij
and
mmp
a²3 (1-220 - I (IID.DTTl))
1 I
|!,,D] 112.
Min
Lemma 3. Suppose that 3 S and the following conditions hold,
|B.ll
- ||
1
12
S
Min
1 S acc
a
a
Min
where O is a constant such that
B
do
π²
Min +
Min
a
Min
Min +1
(min + 1)3"
Then,
[|C++wee1lle (1
B
+ a. 1311121112)
|[Cwell2+ MI)
4a
for constant M depending only
Proof. Let Q2 = -7 Ct. Expanding the recursion for wt+1-
C411W141 = C++1 (I - -:24 + a,,52)) we
(N)
+ aBC++1 (D. Dr 20 D: - a² D - a22, D. D D.D. D.) we
Since || < 1. there is some constant Mn depending only such that
MBl D 2
B
Expanding term (A)
min + A1
C++1 (Ix 320 + a,S2)) wi aB I-a
E.Co (Ix
b0, + aBl?) we
min
(0)
a²8 6,54 - -3 (Iv all;) wwim (Ix aal))
a²8
-c,
Ip
S S2.
(I+ - 300 aBS?)) we
Min
min
(
+ aBC (D. Dw (I+ aa0) (I& ash waw, D. D. aD, Dewne DJ Di).
where a = a mur+1
Similarly to term
BB) there is a constant Mg depending only on a such that
MF D.
wy l|2 N
E
Bounding term
Mia + +11,) No
Min T
Ex, (I+ BA, a,337) C.u.
2
Min + 1
T
Aelle| 3A, + a;31 CC
Min
||al|a
[ a
min
Sa(Ae) (1 (1- (A,)))( + aBAa(A.))) [[Cowlll..
min
18

---

Published as a conference paper at ICLR 2024
Re- Re-writing term (D)
a²B B
a²B min
D
L
dede))
(Is - 3A, + aBAA) Bdi (INr aAAe)
A2Ae
Crwe
Min
where d1 and d2 are defined as
d ( (Ix alt wt: (I& B2, + a:B(()) wx), d2 (a (I+ 1 + a,522) wi).
As all eigenvalues of 22 are in 0o mia
mint1
[
1
8
It I - -32 202++ + a,32 3 I
4a
and
1
8
3 (I al2; (I - 8822 + a352,) Is
Min +
4a
1 Mi
0 / Ω (T, --2 + o,352,) = min +
Therefore, di and are non-negative and bounded from above follows,
ac (1- in
em
Sd ac, 05 d2 S
Min +1
min +
By assumptions,
a²3 222
aB
a9 (c o²)
de dell (I& BA, + +BB1
11
5, <
Min
min
+ 1
Min +
and combining all the negative terms in (F)
a² d,Ac II
2² d2 ag
c 0² min +
3A, + aBA))+ (Iv aAe)
+c+
I
min
min
(min + 1)2
Hence,
(F) is bounded by below and above,
< (I& - BA, + cBBA7)
Thus, the norm of (F) is bounded by above,
3
<
4a
Combining all the bounds,
B 4a
|[Ce41ee11le
+ a
|[C,uoll 1111.1221w.l...
where M is a constant depending only a. Corollary 1. Assume that conditions of Lemma are satisfied for a fixed e O for all times t.
Then, Lemma directly implies that
lim B. B lim D, = 0. {-200
1-00
Further, assume that conditions of Lemma 3 is satisfied for all times and
2 4 2/122
(25)
a
Then, Lemmas 2 and together imply that
lim 100
B
19

---

Published as a conference paper at ICLR 2024
Proof. The first result directly follows by Lemma 2
lim 1000
D = 0.
Hence, for any 6 > 0, there exist a le such that
> te: |Pollla < Vea
Observe that for any
1B441 S B, C+41 ll2 + [|B., D [[e+1 + Dr 12412241
Therefore, by Lemma 3 for any le:
B aB1122+112
[1Cu e² M
[B41we+1lk 2
te Vea
4a
B
|Bewelle e M
S
a.11E.1l2)
tc.
4a
Vea
By Equation 25
1
+ a: 0131|En |=ala
<
4xx
and [B 22 is decaying for t > te as long as
M V/o
B,w,ll2 2 aB
|>
10
Hence, for any O, it possible to find to > te such that for all > te,
c1
Bewelll S
Vea
+e..
a3
- l2)
la
As and c' are arbitrary,
BrWell2 = O. tox
lim
Lemma 4. Assume that D 2(m +1j Then,
|| < 1 ||l1 112 I ac2 for constants cı 02 E R4.
Min +
ci (02 + +
Ul/l12 < a
02
min
2mi (Min + 1)
Proof. By definition Us.
a min
Uelll S ||ill2+
D. Dellz-
Term A is bounded by Lemma
For the term
B
bounding òN by conditions on B, and wis
6, B 12 + S C2 x2
one has the following bound
a 8, Drll2
αC (c2 σ²) 2min (Min + 1)
Min
Combining the two bounds yields the result,
Min +1 +
e (02 + 02)
C2
Min
2mmin (Min +
20

---

Published as a conference paper at ICLR 2024
Lemma 5. Assume that |Bill I 1 and || v ac for constant R4. Then,
Min +
|W&ll - ac
Min
Proof. By using 3 Be B, 1 I
1(Tx - aB, Be) (IN - aB B,)ll2 11 -aB; B₁) we)well < |00, 12 S ac,
|B, Bewrer B, Brll2 ||B B S 2 11
and the result follows
Will & I[ - aB, B,),w,, (I - aB, B,) + a²
min + Min
[BB w, B, Ball2 S ac
min
C.2 BOUNDS ON ITERATES AND MONOTONICITY
The recursion for A, given in Equation 22 has the following main term:
(I& +
+ (A)))
Lemma 6 bounds A4+1 from above by this term, i.e., terms involving U. negative. On the other hand, Lemma bounds A4+1 from below with the expression
(I, + ( - a, +
- a
(26)
where 7t. E RR is scalar such that |UA||2 S a7 Lastly, Lemma 9 shows that updates of the form of Equation (26 enjoy monotonicity property which allows the control of A, over time from above and below by constructing sequences matrices, described in Appendix C.3
Lemma 6. Suppose that ||Ue|12 S 3. Then,
A+11 m (I, + aBR((Ad) A (L. aAR((A.))
Proof. As |1tl12 < 4
Q.U.T SC.UUCT BUF)CT 20
Using Appendix
A++1 = (I. + (AR((A)) (A, BC,U,C7) (I. + a3R((A)) SCUUCT + PCUUCTT
3 (I, + (A.) (A, -3,,U,,,, (I++ (A))
1 (I, (AiR((A)) Ar (I& (ABR((A)))
Lemma 7. Let "i be a scular such that ||U|||2 S an S I. Then,
23.
(I, + aBB,((A) a + taBR((A)) aBy(la 1 AA+11
Proof. By using |Uel12 S artt.
anArr C.U.C. Ci - 0.
Moreover, as
x x x Bx² is an increasing function in [0, l. the maximal eigenvalue of
U, - OU?
3s² < art - a² 0²332 where the maximal eigenvalue U.. Hence,
(o7: a² 33? IN (U+- BUŁ) 0.
21

---

Published as a conference paper at ICLR 2024
Therefore, the following expression is positive semi-definite,
2Sym (I& a.RR.(A)) (onAA
- C.U(TT)
(I+ + (A)) -CU (
- ((07:A--CU;CC; - - -QU?(())
= C. -(a - -2²37))) IN - (U3)) CT.
The result follows by
A++1 = (I& + a. A (I+ + (AR( - 205ym (LL + (()((A) - -ROUCTT
- -22 = + (.3R((A)) +I (A)) -20 ((I, + a A4)
= (I + a. - A, + +a)R((A)) - aB7T1a)
Lemma 8. Let C. = -V,S.T be the (thin) SVD decomposition of Ct and let be scalar such
that [[T U 2 12 S o7 5 20 Then,
(Is + a (Ae) - aB + a:3R((As)) - a 3 Ac+1.
Proof. is sufficient to observe that anA, C,U.C, V. - -T, S.V. = 0,
and use the same argument as in the proof of Lemma
Lemma For non-negative scalars T,% let f(sT,n) Sym, (R) + Sym, be defined as follows,
f(A:T,7) (I, + aBR((,, aB-II) + -
Then, ((5T,7) preserves the partial order between any A, N' that commutes with each other and E,, i.e.
min 1 A = f(AsT,n) f(A:T,)))
a min +1 when the the following condition holds,
σ² +
- a- 50
Mia
Proof. The result follows if and only if - (A- ž -18(1 J)A' R(A,
A
+ - (80 .) AR(A,T)
(B)
+
R(A, TAR(A))
(27)
By Lemma 16
1
1
A - 1 (A A') + -1) 7 A') A
3 JAA+ A' (2(A A²) = 21 2I2 A¹)
Bounding term A by using commutativity of A, A' with Σ. and A' 1 m1
min
R(N,)) R(A, r)A a(min + +1
-2 I
Σ +
Min
Min + 1
ammin + +1
σ² +
Σ, +
I
1' E.(A - A')
Min
Min +1
a(ma + 1)
8² + r
(12 A²2.
Min
min +1
22

---

Published as a conference paper at ICLR 2024
The term (B) is equal to the term (A) and thus bounded by the same expression. By Lemma 17
A3 A = O.
Bounding term (C) using the commutativity of A, A' with E, and A, A' 3 min
min
R(N,T) .7) R(A,F) a(min + +1)
T² 7
D
(12
A12)
min
Min +
aamiu + +1
2
σ² +
E.(A (LA--V9 -AYE,
Σ, +
I+
113
Mia
min +
J² +
3 E
|E1||2
Min + A)
Therefore, Equation
27 is satisfied if
x² + +
x² +
(1 1--38))² 2 40, - -137)) |2:.1l2
+ 4a 33 12
|E +
+1
+
which holds by the given condition.
Remark 1. Let T,7 be scalars such that O < T and 0 < 7 < Amin (E.). Define A, 7) as follows,
A (T,7) a min
min
I
T² +T
+ + Ex
J² + Is
(28)
+1
min + 1
min +
(A,,T, is fixed point of the function f as
(A. = -
Corollary 2. Let A be symmetric p.s.d. matrix which commutes with Σ, and satisfy
3 (T,7))
for some sealars 0 < and 0 < y Amia (E.). Then, assuming that conditions of Lemma are satisfied,
A I<; (,(7,))
Proof. For the left-hand side, note that
T(),T,)) T A (T,7))
Hence, by the given assumption and commutativity,
A (I, + a,R(A, - a A (I+ + a TBPL))= = f(A:T,n). For the right-hand hand side, note that by Lemma 9
f(A:T,7) = =) =
Lemma 10. Let тe and "tt be non-negative, non-increasing scalar sequences such that Yo Amin (E.), and A be a symmetric p.s.d. matrix that commutes with E, such that
A 2 ATO, ())
where Ax(T, 7) is defined in Equation
28 Furthermore, suppose that a and 3 satisfy
Tu 8,
aB
Min +1
Then, the sequence of matrices that defined recursively
A (0) = A. (++1) = fAA ;T)
satisfy
lim (1) A, lim Te, lim m)
1-00
1000 t-000
23

---

Published as a conference paper at ICLR 2024
Proof. By the monotone convergence theorem, 77 and 7/2 are convergent. Let Too and Yoo denote the limits, i.e.,
Too lim Tte Yeo lim Ti-
As A ( and Σ. are commuting normal matrices, they are simultaneuusly diagonalizable, i.e., there exists orthogonal matrix E b and diagonal matrices with positive entries D(0), D, such that
A QD Σ. QD.Q". Then, applying f to any matrix of from A = QDQ", where is diagonal matrix with positive entries, yields
f(A:T,)) - Q [I + - -a²3 + min +
D D
Min + Ib
8² +
+1 - -as)) DQ
min
Observe that f operates entry-wise on diagonal elements of D, i.e., for any diagonal element s of D, the output the corresponding entry of I given by the following map g(:: 8,, IR + R,
g(s; s 7. = (1
-1+ Lass a²3 min
s(s.
σ²
aB²) 8,
min
Min
where s, is the corresponding diagonal entry of D.. Hence, Lemma TO holds if
lim 8, S (Too Yoo):
where s; defined recursively from an initial value so for any t follows,
St+1 g(8) 8x; Tt. 7t);
and so0 7) is defined as
1 Min
1 (1-(7+
0² +
0² +
800 (T,
7
8x
a min +
min +
min +
Observe that
800 (T; 10+
0² +
1
Min
(sx 7),
Min + 1
a
min +
and
-) 8t.
(s.
g(3 SA1 Tc.7)= =
1+ aB (s00 -
min +
Hence,
Min + -1)
σ² + r
8oc (Te, 76) 91+1 (so0 (T,, 72) st)
se) 1 1 aB
and in each iteration 8t takes a step towards 8 (Te(Tee (Te. By assumptions 80 S 8
800 (To, 70) and as
1
8² +
03
min +
for all f, 81+1 never overshoots 8 (TE, 71), i.e.,
S 3t+1 So0 (Te, 71) < so0 so0 (T++1, 7--1).
Therefore, &t is an increasing sequence bounded above by (To, 7oo) and by invoking the mono- tone convergence theorem, 8; convergent. Assume that s; convergences to 80x0 < 800 (Too Yoo)- Then, there exist l, such that (To, Tt.) s'cc tE. By analyzing the sequence.
s =S,,
$1,4 = g( Te, Mt
it is easy to show that
St.ss t and lim tt = 8 (Te,, Th.) > syc:
8-000
which leads to contradiction. Hence, lim 81 8o0 (Tos )oo)-
Remark 2. Assume the setup of Lemma 10 and that the sequences T₁ and i converge to Then, as f + oo, A₃ convergences A.,
lim A, = A. 1-000
24

---

Published as a conference paper at ICLR 2024
C.3 SEQUENCE OF BOUNDS
Lemma 11 constructs a sequence of matrices A that upper bounds iterates of A. The idea is to use the monotonicity property described in Lemma 9
together with the upper bound in Lemma 6 to control A, from above. Lemma 10 with Remark 2 then allow to conclude lim A = A.. For this purpose, Lemma I1 assume a sufficiently small initialization that leads dynamics where
||B S a -12 and well2. |elll2 are monotonically decreasing.
In a similar spirit, Lemma 12 construct a sequence of lower bound matrices A given that it is possible to select two scalar sequences Tr and "r. At each step, the lower bounds Af takes step towards A, (T) described by Remark [ For ensuring that A, does not decay, the sequences To and ?/2 are chosen to be non-increasing, which results increasing A, (Te, 7e) and A/. In the limit t + 0C, A convergences to the fixed-point A, (lim,+>o lim 7r), which serves as the asymptotic lower bound. Finally, Corollary shows that possible to construct these sequences with the limit 0 under some conditions.
Lemma 11. Assume that Bg and wo are initialized such that
B
C1
1
||wo S ae2,
a min +
for constants 0 < 4 < 11, < 02 and or, B satisfy the following conditions:
min +
1 ((min + 1)|12, 1l2 + +0²) 27
1.
max
02
033
Min
Min
min
(02 + σ²)
Min
2.
2
C2
aB
Min
2mm Minn 1)
3.
2 5( E(11E+11++
oB
Min
+
4. 3 < a.
The series N. defined recursively as
Ao = ||
No I1(Ts aBR(A (Ie + a3R(A )
upper bounds the iterates AG, i.e. all t, Ao A, Moreover, ł AL for all
Proof. The result follows by induction. It is easy to check that the given assumptions satisfy the conditions of Lemmas and 9 for all time steps. Assume that for time the following assumptions hold.
[D, ll2 a non-increasing sequence for $
2. [wsll2 is non-increasing sequence for < t.
3. A, 3 A A. for all
Then, for time + 1, the following conditions holds:
1. By using A, I A. 1
min (min
+1) and D.D.T = BoBB 1 a mI+
B,B.T 3 Mmn + C1
||elle - -
min -1
Therefore, by Lemma I and Lemma 2
| 112 < ||w.l2,
|PP N |D.D, 1l.
2. By applying Lemma
min +
a (c2 02)
|1r|12 <a
C2
5
min
2m (Min + +1)
23
25

---

Published as a conference paper at ICLR 2024
Therefore, by Lemma
A+11 3 (TA + +..RR.)),(T: + +o,RR.)
3. By applying Lemma 9 with A = Ap and A := An.
(I& + o: ( (I. + aBR,) 0,A,;0,0) = f(A7 00,0) 3 A/41-
4. By applying Lemma 9 with A = A, and A' = A..
;(A7: 3 0, A.
Therefore, Λl4 3 ||,,1121a A.
5. Combining all the results,
A411 3 141 3 A..
Lemma 12. Ti and "it non-increasing scalar sequences such that
| 1 123 ST.
|Ul| 5 an
S 2
and S C22 7o < >mia( Assume that all the assumptions of Lemma 11 hold with constants C1 and Oz. and a, 3 satisfy the following extra conditions
JB 5
+ σ²
+ Amin (E.).
min
Then, the series At defined as follows
N/6 = min (Amin (Ao) Amin (Aa (To, 7o))) I,,
A Amin (IT a:R(AA Te) a a33TII)) (A aBR(AN Te) a3 Ix,
lower bounds the iterates A,, for all I, At A.. Moreover At 3 A441 for all t.
Proof. The result follows by induction. It is easy to check that given assumptions satisfy the 9 for all time steps. Suppose that for all time S t,
conditions of Lemmas 2 and
! A, 3 A., A = A, 1t).
Then, for time + 1, the following conditions hold:
1. By Lemma 7
A 2 (I, + aBR((A)) -alA))/
2. By Lemma
9
(I +a9R(()))) (A a,3R(AA) - -B
(I, + a.3R (aS - a.
3. Using commutativity of E. and At.
(I& + a:BR((A)) aB I2 ( -
(Ix -B3 a3R((A) a
M
4. By Corollary 2
A < Mi11 3 AA(TT. T)).
As T+4 S Ti and 7441 S %t.
A Aft 3 A.(T.. 3 (e
26

---

Published as a conference paper at ICLR 2024
5. Combining all the results,
A 3 A4t1
Aft1 3 (T 3 (T441 7+-1).
Remark 3.
The condition "/t in Lemma 12 can be relaxed by the condition used in Lemma
Corollary 3. Assume that Lemmal 12 holds with constants C1, C22 and constant sequences
T= C
7
C2
Min
+
2m (min +1)
C1 (c2 +
< Amia (x+).
Min
Furthermore, suppose a,, 3 satisfy the following extra properties,
1
4 112
4
8
40x
02 Min 2 +
Mmn
ox3
Min
+1
Min
(min + 1)2
Let C. = V be the (thin) SVD decomposition of Cr. Then, there exist non-increasing scalar sequences Tr and 7/. such that
B,w cz
[rU l2 S am - 23
with the limit
lim Te 0, lim 7 0.
Proof. All the assumptions of Corollary D are satisfied with constant c == c2. Hence,
||Dell = 0, lim [|Baeella = 0. t+00
lim
(29)
Moreover, the sequence |Bwwel is upper bounded above,
|Bw/ll2 s ||Blll||walla S c2.
Take any sequence 0 < T/ S that monotonically decays to O. Set To = T6 and 8: 0. Recursively define T; as follows: for each > 0, find the smallest & such that
[B,u w. S T/C
for all s 2 st. Then, set T., = T/ and for all 9t-1 S s < St. T2 T/-1. is easy to check that this procedure yields non-increasing scalar sequence with the desired limit.
By Lemma 12 with % = 7, A₁ non-decaying, and its lowest eigenvalue is bounded from below. Using the limits in Equation (29).
lim C.U,C. = 0, 100
which implies that lim [ sequence 7t with the desired limit.
12 = 0. A similar argument yields a non- non-increasing scalar
C.4 PROOF THEOREM 2
By Lemma 11
A J A/
A, and DD4ill2 & [Della for all Using the initialization condition,
B 2 3 - D 1 < ||rl| + || < ||+]|2 ||Boll2 < a.
Now, the conditions of Corollary 1 are satisfied with C2- By Corollary | lim D. 0, lim Bow. 1-
0.
Moreover, by Corollary there exist non non-increasing sequences To and % that are decaying. By Lemma 12 with these sequences yield Af- 3 A,, for all t. Finally, by Lemma 10
lim A + A, and lim Al - A.; (-220
t0000
which concludes Theorem
27

---

Published as a conference paper at ICLR 2024
D PROOF OF PROPOSITION 1
2 below gives a more complete version of Proposition 1 stating an upper bound holding with probability at least ó for any > O.
Proposition
Proposition 2. Let B, Werst satisfy Equations and 11 for a new task defined by Equation 10 For any 5 > O with probability at least
BWttst - B. B.w,l12=0 1 1+40%/amm (2.)
vk+ log(
log(#)
|wall + max
1|w-
Min
Mitcstt
Macst
k
log( (
log( ()
+
V
1+
1+
Mreet
k
Mitest
where recall 5² = =Tr(..)) + o².
Using Equation 11 it comes
Bwrest B.W. (aBB E, B.) + a
BB X
Mess
(as, I&) w, aB.A, (B. Etestest B. I&)
B.A.B... 2.
w.
Miest
The rest of the proof aims individually bounding the norms of the terms note that by definition of A.,
(A)
B
and
First
Min 72
2²
aA.- Is
min + 1 Is
E. +
I
(min + 1)2
Min +
This directly implies that
min
|1a., Is
min +
(min + 1)3
Amiu (E,) +
0² min +1
0²
Amin (E.)+0 /min.
(30)
Min +1 1
Moreover, the concentration inequalities of 5:
I Lemmas 13 and claim that with probability at least 1
2 2lg((5) (Vk- + 2 log( 2
Vk +
[ Enest B, Ixll2 max
v Mitest
Mtest
log(
log( +))
| x 2/22 S 160 Mustk (1
(1 +
2k
2meest
These two bounds along with Equation 30 then allow to bound the terms follows
(A)
B and
(C)
as
[[A)]]22 5 Amiua ( /mij
Min 1
Vic + v 2lg(
(Vh + v 2og(())
1. 1 7.1mm
[l
3
1
max
||w-lll
Min
Mirest
Mrest
1
E. 12
k
log
log 4 2meest
[| < 16
1
(1
(1 +
Min +
Maestt
2k
28

---

Published as a conference paper at ICLR 2024
TT
where we used in the two last bounds that a alIA,,l2 l12 bounds finally yields Proposition and Proposition Lemma 13. For any 5 > O, with probability at least 2
12 12+00 /min. Summing these three
min +1
with the particular choice ó 4e
Vk
2 log()))
(Vi+ + / log
Etent B, Ixll2 S max
Mitest
Mitett
vk+ logí m
and | 112 I Miest 1
Mises
Proof. Note that B X is a matrix in Rx whose entries are independent standard Gaussian variables. From there, applying Corollary 5.35 and Lemma 5.36 from Vershynin (2012) with t
V 2 48 directly leads to Lemma 13
Lemma 14.
log( 2k
log( # 2m
[B_ X #|12 2 16 / Mestt (1 +
(1 +
2
Proof. Let = B. in this proof. Recall that A has independent entries following standard normal distribution. A and z are independent, which implies that A |=||| ~ (O, Is). Typical bounds on Gaussian variables then give (see e.g. Riigolltt & Hütter 2023
Theorem 1.19)
| |=||
log( #
2 24 (1+
2k
4
similar bound holds on the a sub-Gaussian vector 2, which is of dimension Must
log( 4
|||||
Messtt (1 +
2marst
Combining these two bounds then yields Lemma 14
29

---

Published as a conference paper at ICLR 2024
E TECHNICAL LEMMAS
Lemma 15. Let Σ = x X where X E Raxd is such that each row is composed of i.i.d. samples
(O, For any unit vector v,
n a+1
E[Evv' Σ Ia
DU
Proof. Let Let x' ~ (O), By expanding covariance Σ and i..d. assumption.
1 2))xx''
E T
xx (xx,
E
(z, (ar o)
72
n
(A)
(B)
For the term
(A)
E
(x,
x.
1kk
E
1y ) 2y
1=1
Any term with an odd-order power cancels out as the data is symmetric around the origin, and
E (sx
= 2vv T
+ La,
by the following computations,
E (ar, 2xx "Is = 1 Bag
E[1 = 330 +
Σ
01 20 +
iデj
E (x, v) xxx 1gx 20 [ 20yuk-
For the term B by i.i.d. assumption,
E [xx, (a', u)arr" 1 Ea,,,, ((ar
With a similar argument, is easy to see [( u)ar) E
[x vi, and [x
Combining the two terms yields Lemma 15
Lemma 16. Let A and B be positive semi-definite symmetric matrices of shape lo and AB
B.A. Then,
AB < |A]|2B.
Proof. As A and B are normal matrices that commute, there exist orthogonal Q such that A = QA and = QA&Q where AA and Ag are diagonal. Then,
AB = QA N [AAIQAnQT.
for any vector R
K T ABU Σ (As)) ||Al22 Σ (AB) ||A
Lemma 17. Let A and B be positive semi semi-definit symmetric matrices of shape k k such that AB BA and A3 B. Then, for any k E N,
4 x Bk
(31)
Proof. As and B are normal matrices that commute, there exist orthogonal Q such that A = QA and B = QA&Q where AA and Ag are diagonal. Then,
B² - A = Q(A - A 2 0,
as Z A implies An 2 AA
30

---

Published as a conference paper at ICLR 2024
F FIXED POINTS CHARACTERIZED BY THEOREM ARE GLOBAL MINIMA
The ANII loss with m samples in the inner loop reads,
CAIIL (B, m) W......... [B X,; B. B. :.l., where is the updated head after a step of gradient descent, i.e.,
(32)
x (X, vi)
w w: X5 Ui) = w
(33)
m
Whenever the context is clear, we will write v or vo (w)) instead of w (w; X yi) for brevity. Theo-
rem | proves that minimizing objective in Equation 32 with FO-ANIL algorithm asymptotically
convergences to a set of fixed points, under some conditions. In Proposition 3 we show that these points are global minima of the Equation (32).
Proposition 3. Fix any (B,0) () that satisfy the three limiting conditions of Theorem
B__ J =0,
Bi: 0,
B. BB B. A.
Then, (B, u)) is the minimizer of the Equation
32 ie.,
(B, w) argmin CANIL w; mmn)
B,
The strategy of proof is iteratively show that modifying points satisfy these three limits reduce the ANIL loss. Lemmas 18 to 20 demonstrates how to modify each point such that the resulting point obeys a particular limit and has better generalization.
Proof.
For any (B, w), define the following points,
(B1, w1) = (B - B. B B, B.
(B2, w22) (B1, w1 - BT (BıBT) -1 B101).
Then, Lemmas 18 to 20 show that
CANIL (B, w; mm) 2 CANIL(B1, wi: mi)
2 CANIL w22 ma) 2 CANIL(B, ma))
Since (B, w) is arbitrary.
(ß, 1) E argmin CANIL wi man)
B,
Lemma 18. Consider any parameters (B, w) E Baxk' RR Let B' B B., B B. Then, for any m have
LANIL w; m) 2 LANIL (B', m).
Proof. Decomposing the loss into two orthogonal terms yields the desired result,
LANIL (B, wi m.)
1
E N...
B. Büü w., 1
+ Ex...
[|| B
2
2
[.T Bw - w..1 1²]
< Ew.... N.x.v
2
= CANIL (B',w;,) m)
Lemma 19. Consider any parameters (B,w) Baxk x BK such that BT,, B 0. Let w'
T (BBT)- Bw. Then, for any > O, have
LANIL ( w; m) LANUL (B, (B:'; w';m),
31

---

Published as a conference paper at ICLR 2024
Proof. Expanding the square,
CANIL (B, w; m)) CANIL(B w'; = Bw....... || Bū(w) w. ||B_ Bū((") - wr,ll"]
E2 E.., [B || E.,, W(,, Bú((w) Bú((w)M
(B)
First, expanding (ww) and vi(w) by Equation (33
Bū(w)= [L m
BB X2 x.) Bw
BB X. yos
Bū(w") BB x, yt.
m
m
For the first term,
A
Ex,
|1( [a BB X, X,) Bw
2
+
20
i
a ((To-
BB X, x:) Bw, BB X X, X,B.w.).
m
Ex,
[|(1.
BB - x.)
Bw
N 0,
where we have used that the tasks and the noise are centered around O. For the second term,
Em..: [B, BB,
(1.
[ -
a BB X, x.
Bw O,
Min
where we have again used that the tasks are centered around 0. Putting two results together yields Lemma 19
Lemma 20. Consider any parameters (B, w) E Baxk' x RK such that B,,, B = 0, Bw = O. Let (B', w') E Raxk' x Rk such that B _ B' = 0, B'w = and B B B. = A.. Then, we have
CANIL (B, w; ma)) 2 LANIL(B', w/ mm).
Proof. Let = B._ BBT B. in this proof. Using B,,, B = O, we have
1 By, N,.. BB B w..i
LANIL (B, w; min)
Plugging in the definition of w.
a² 1
LANIL (B, wi min
Ew. X1.3 ||B_ BBT x,T vill?
2 m
a Ew....... [[ B. BBT x3] wi)l + er
17lin
(B)
Using that the label noise is centered,
A
E E
[|| B. BB B E.B.w,
12] + +xx.
B. BB x7
where 2; == 1 X. X,. By the independence of W.,s, X₁ and Lemma 15
min
tr (B. BB Em.,,., [E,B , B. Ed] BB B.)
tr (B. BB" Ex. E.B. Σx] BBT B.)
1 Min + tr [B BB B.E.B. BBT B.) + min
tr (B. BB
BB BB B.)
Min
Min
+
tr (AE,,)) + 1
tr (r.)
min
Min
32

---

Published as a conference paper at ICLR 2024
For the term
(D)
we have
tr B.,BBBBx [X] 2,2, Xi] BB* B.)
min
-r²t (BBBBTT Ex, [2.]] (UBBBBBBB. B.)
BB
o¹tr (B. BB BB B.)
o'tr (A?).
Lastly, for the term
B we have
Eu,, X; [(w,,i B, BB X X,B.w,,))
Min
Bw., [( B,, BB B.w.,)) tr (AE.).
Putting everything together using M, is scaled identity,
a
1
CANIL wi; min) =
(mmin + 1) Lr (AEEA) + (r.( (A²)) - atr crtr (AE.) +
2 (E.)
2mn
a²
(mman + 1) - 2 + o²) tr (A²) al al + tr (E.).
2mn
Hence, the loss depends on B only through == B. BBT B, for all (B, w) such that B x 0, Bw = 0. Taking the derivative w.r.t yields that a minimizer if and only
(imin 1) ||_ x²) A Xmax (E.)== 0.
Min
This quantity is minimized for A, as
min + A, E, +
Ø² A, = E..
min
Min
33

---

Published as a conference paper at ICLR 2024
G EXTENDING COLLINS ET AL. (2022) ANALYSIS TO THE MISSPECIFIED SETTING
We show that the dynamics for infinite samples in the misspecified setting k k' d is reducible to a well-specified case studied in [Colinn et al.] (2022). The idea to show that the dynamics is restricted to k-dimensional subspace via time-independent bijection between misspecified and well-specified iterates.
In the infinite samples limit, Min 00, Mout oC, the outer loop updates of Equation ( simplify with Assumption to
W++1 = We - -B (Brw - BB
B = -BB,BA (Arwr + aB, BLll)))
(34)
+ -I,
aB,B,)) B. ((,, (Arwr)) + &EE B.).
where μ., and E. respectively are the empirical task mean and covariance, and A, B This leads to following updates on C₁ = B, Bd,
C++1 (I+ + aB (I. C.C, (.C.)E.)C. - (Arr + aC, ua)
+ 3 - (I=C0 Me (Arw))
A key observation of this recursion is that all the terms end with C. or A.. This observation is sufficient to deduce that C is fixed in its row space.
Assume that Bg is initialized such that
ker(Co) S ker(Ao))
This condition is always satisfiable by a choice of B, that guarantees B3 Ba - alg, similarly to Collins et al. (2022). With this assumption, there no dynamics in the kernel space of Co. More precisely, we show that for all time t, ker(Cn) S ker(C,) n ker(A,). Then, it is easy to conclude that B, has simplified rank-deficient dynamics.
Assume the following inductive hypothesis at time f,
ker(Co) S ker(C,) ker(A.))
For time step + 1, we have for all v E ker (Ct) n ker Ar), C++1V 0. As result, the next step contains the kernel space of the previous step, i.e., ker(Cn) C ker(C,) () n ker( S ker(Cr+1) Similarly, inspecting the expression for A,+1, we have for all v E ker(C,) n ker(Ar), A++14 and ker (Co) S ker(Cr) n ker(A:) S ker( (+))). Therefore, the induction hypothesis time step ++ holds.
Now, using that ker( (Ct col(C.T) row spaces of Ct confined in the same k-dimensional subspace, col(C,T) 2 col(CT) ). Let R E Bo and Ry (KK--k) be orthogonal matrices that span col(C, and (ol(C)) ) respectively. That is, R and Rı satisfy HR I col (R) = (Cl(C) and R_R -k col(R1) coI(C]])+ It is easy show that updates to B, and we are orthogonal to i,,
B,R B.R and Ri Ryw
With this result, we can prove that there is a k-dimensional parametrization of the misspecified dynamics. Let wo E Re, By dd defined as
Bo = B Up Ruv-
Running FO-ANIL in the infinite samples limit, initialized with By and wo mirrors the dynamics of the original misspecified iterations, i.c., B, and wi satisfy,
B, B.R.. d; Rws, Baü B,w; Bo R. R.
This given bijection proves that iterates are fixed throughout training on the k - k-dimensional subspace R..). Hence, as argued in Section 4 the infinite samples dynamics do not capture unlearning behavior observed in Section In contrast, the infinite tasks idealisation exhibits both learning and unlearning dynamics.
34

---

Published as a conference paper at ICLR 2024
H CONVERGENCE RATE FOR UNLEARNING
In Proposition 4 we derive the rate | B, = 0
Min n² 80
Proposition 4. Under the conditions Theorem 2
| Bell -
(35)
a² gz t
BT_ Boll2
for any time 2
Proof. Recall that Lemma 2 holds for all time steps by Theorem That is, for all t > 0,
| B & (1 (1-x) II Bill)) | l
(36)
where = 2² 2 for brevity. Now, assume the inductive hypothesis in Equation 35
holds for
Min
time t. Observe that the function x (1 kT) increasing on [0, 1 and 2x
| B,
B Boll a min + 1 2k
by the assumptions of Theorem
Then, by Equation (36) and monotonicity of x + (1 kT)z
(t - 1) + [B Boll
|| |B__ B++1 112 -
66
kt JlB. Boll kt + [|B.. Boll
<t + B Boll3
Using the inequality of arithmetic and geometric means,
l(t - 1) + BB Boll
A. (t + 1) +
[B, 1 Bo 1
B B
# (t + 1) +
kol B., Boll
BB B
K B. 1
Hence, the induction hypothesis at time step t + 1 holds.
35

---

Published as a conference paper at ICLR 2024
I ADDITIONAL MATERIAL ON EXPERIMENTS
I.1 EXPERIMENTAL DETAILS
In the experiments considered in Section samples are split into two subsets with min 20 and Mout 10 for model-agnostic methods. The task parameters Wa,s drawn ii.d. from (O, 2.). where Σ. odiag(1, ...,, k) and is constant chosen so that |[ || vk. Moreover, the features are drawn i.i.d. following a standard Gaussian distribution. All the curves are averaged over 10 training runs.
Model-agnostic methods are all trained using step sizes 8 = 0.025. For the infinite tasks model, the iterates are computed using the close form formulas given by Equations (5) and (6) for min 20. For the infinite samples model, it is computed using the closed form formula of Collins et al. (2022)
Equation (3)) with N 5000 tasks. The matrix By is initialized randomly as an orthogonal matrix such that By Bo Iee Is. The vector wo is initialized uniformly at random the k' o'-dimensional sphere with squared radius 0.01k'a.
For training Burer-Montein method, we initialize Bo is initialized randomly as an orthogonal matrix such that By Bo = Ioo Ix and each column of W initialized uniformly at random on the k'- dimensional sphere with squared radius 0.01k' a. P Also, similarly to Tripuraneni et al. (2021). we add B B, W.W,T 11 regularizing term the training loss ensure training stability. The matrices B, and W, are simultaneously trained with LBFGS using the default parameters of scipy.
For Table 1 consider ridge regression for each learned representation. For example, if we learned the representation given by the matrix B RR the Ridge estimator is given by
argmin Luest (Bw; X, 3) + Al| l|
wERk'
The regularization parameter À is tuned for each method using grid search over multiple values.
1.2 GENERAL TASK DISTRIBUTIONS
In this section, run similar experiments to Section 5 but with a more difficult task distribution and 3 training runs per method. particular the task parameters are now generated as We,i ~ N (i,, E.), where μa, is chosen uniformly at random on the k-sphere of radius vk. Also, E, is chosen proportional to diag(e) ...., )²), so that its Frobenius-norm is 2 and its condition number et1
FO-ANIL (inl tasks)
FO-ANIL
FO-MAML
AQuan LO0 (b))
20
10m
2.00
4000
30:00
9000
10000
2000
1000
(200
8000
10003
Itorations
Iterations
Figure 4: Evolution of largest (left) and smallest (right) squared singular values of B, B, during training. The shaded area represents the standard deviation observed over 3 runs.
Similarly to Section Figures and show the evolution of the squared singular values on the good subspace and its orthogonal component during the training. Similarly the well-behaved case of Section 5 model-agnostic methods seem to correctly learn the good subspace and unlearn its orthogonal complement, still at a very slow rate. The main difference that the matrix towards which B. B, B, converges does not exactly correspond to the A, matrix defined in Theorem | We believe this due to an additional term that should appear in the presence of a non-zero task mean. We yet do not fully understand what this term should be.
We choose a small initialization regime Monteroo be in the good implicit bias regime. Note that Surer-Monteint yields worse performance when using a larger initialization scale.
36

---

Published as a conference paper at ICLR 2024
FO-ANIL (idl Lasks)
FO-ANIL
FO-MAMIL
10
X2 10_180 (8)
X (40)) (30)
10.00
Iterations
Figure 5:
Evolution of average (lef)) and largest (right) squared singular value of B Bg during training. The shaded area represents the standard deviation observed over 3 runs,
FO ANIL (inf todks)
FO-ANIL
FC MAML
(1,m)
100
302
Iterations
Figure 6: Evolution of B,w; B. during training. The shaded area represents the standard
deviation observed over 3 runs.
Figure the other hand shows the evolution of B,w, B.p+. |I while training. This value quickly decreases to O. This decay implies that model-agnostic methods learn not only the low-dimensional space on which the task parameters lic, but also their mean value. then chooses this mean value as the initial point, and consequentially, the task adaptation happens quickly at test time. Overall, the experiments in this section suggest that model-agnostic methods still learn good representation when facing more general task distributions.
1.3 NUMBER OF GRADIENT STEPS AT TEST TIME
This section studies what should be done at test time for the different methods. Figure iillustrates how the excess risk evolves when running gradient descent over the head parameters w, for the methods trained in Section 5 For all results, gradient descent is run with step size 0.01, which is actually smaller than the a used while training FO-ANIL.
Keeping the step size equal to a leads to optimization complications when rumning gradient descent: the objective loss diverges, since the step size is chosen too large. This divergence is due to the fact that FO-ANIL chooses large scale B, while training: this ensures a quick adaptation after single gradient step but also leads to divergence of gradient descent after many steps.
The excess risk first decreases for all the methods while running gradient descent. However, after some critical threshold, it increases again for all methods except the Oracle. It is due to the fact that at some point in the task adaptation, the methods start overfitting the noise using components along the orthogonal complement of the ground-truth space. Even though the representation learned by FO-ANIL is nearly rank-deficient, it is still full rank. As can be in the difference between FO-ANIL and Oracle, this tiny difference between rank-deficient and full rank actually leads to a huge performance gap when running gradient descent until convergence.
Additionally, Figure niielyy illustrates how early stopping plays some regularizing role here. Over- all, this suggests it is far from obvious how the methods should adapt test time, despite having learned good representation.
37

---

Published as a conference paper at ICLR 2024
Single task
Oradle
ro ANIL
.... Burer Montero
2,5-
2.0-
0.5-
0,5
40
50
Number steps
Figure 7: Evolution of the excess risk (evaluated on 1000 tasks with Maset 30) with respect to the number of gradient descent steps processed, averaged over 10 training runs.
1.4 IMPACT OF NOISE AND NUMBER OF SAMPLES INNER UPDATES
additional experiments to illustrate the impact of label noise and the number of samples on the decay of the orthogonal complement of the ground-trun truth subspace. The experimental setup is the same Section | for FO-ANIL with finite tasks, except for the changes in the number of samples per task and the variance of label noise.
In this section,
Figure 8 illustrates the decay of squared singular value of B__ B, during training. As predicted by Appendix H the unlearning is fastest when Min 10 and slowest when min = 30. Figure 9 plots the decay with respect to different noise levels. The rate derived for the infinite tasks model suggests that the decay is faster for larger noise. However, experimental evidence with finite number tasks is more nuanced. The decay is indeed fastest for 0² = 4 and slowest for o² = on average. However, the decay of the largest singular value slows down for a² 4 in second time, while the decay still goes on with /² 0, and the largest singular value eventually becomes smaller than in the o² case. This observation might indicate the intricate dynamics of FO-ANIL with finite tasks.
(A5T (8))
Kaad A2T, (ic))
5000
(0014)
(7000)
2000
20000
G000
10,00
10.00
20.00
25000
Iterations
Iterations
Figure 8: Evolution of average (leff) and largest (right) squared singular value of B, B, during
training. The shaded area represents the standard deviation observed over runs.
10
(1F (8)
19.00
20000
25.00
5000
25.00
Figure 9: Evolution of average (left) and largest (right) squared singular value of B, B, during training. The shaded area represents the standard deviation observed over runs.
38

---

Published as a conference paper at ICLR 2024
1.5 SCALING LAWS IN PROPOSITION I
In this subsection, we study the scaling laws predicted by the upper bound in Proposition We com- pute excess risk and estimation errors and compare them with the predictions from Proposition T All errors are computed by sampling 1000 test-time tasks with 1000 test samples each.
In order to show that there is no dependency on after pretraining with FO-ANIL, we run experi- ments with varying d = k' and k, in the same experimental setup as described in Appendix [.I To mimic fhot and high-sample regimes, we select Meet 20 and Miestt 1000. Our results are shown in Figure 10 The excess risk does not scale with the ambient dimension d but with the hidden low-rank dimension k.
3,0-
1.50
2.5
Eaccoss risk
1.25
2.0
2.00
0.50
0.25
Figure 10: Excess risk for varying d and for FO-ANIL with test samples miest = 10 (left) and Mhest 1000 (right).
Next, we run a series of experiments evaluate scaling laws predicted by Proposition 3 Similarly, we follow the experimental setting detailed in Appendix I.] with an identity Σ.. In order to provide a clean comparison, Σ, scaled in test time such that E & ||] for all h values. This allows us to isolate the impact of k. min and Must. on the generalization error after adaptation. We also set
k'' 25 and N 25000 for the rest of this subsection.
Estimation error
10
Estimation 107
xx 10-1
Figure 11: Estimation error of learned FO-ANIL representations for k = 5, 10, 15 and min = 20 (left) and min 40 (right) with o² 2. Horizontal lines bounds used in Proposition | for the first term that scales independently from Mitstt i.e., bounds for Marst 0o.
Figure II shows the scaling of the loss with respect to k and different choices of miest for Min 20 and min 40, together with predictions made from Proposition [ Black horizontal lines are bounds for the first term that does not scale with Mest i.e., bound to the generalization error when Myest- = oc. We observe that the bound used in Proposition [ is tight. The dependency on is through the term d² 22/amin (E.) which is equal to tr(2)/ min (E.)
when o² = O.
In order to evaluate the other two terms in Proposition 1 we subtract the dashed black lines from [Bttsst B. W. 2l and plot with respect to Vk/meett- Figure 12 shows that this excess estimation error is linear in / Vk/Meest Black horizontal lines are (1+ + o²) / Vk/meest that serves as upper bound to the two last terms in Proposition [ Overall, our results indicate the scaling given by Proposition 1 is tight.
39

---

Published as a conference paper at ICLR 2024
5
R-
R-
Escess edimation er1
estimation
0.3
166
Figure 12: Excess estimation enror of learned FO-ANIL representations for k: 10, 15, 20, 25 40 (right) with o² = 2. Horizontal lines are bounds used in Proposition D foor the last two terms that scale with meestt
and Min 20 (left) and
1.6 IMPACT OF NONLINEARITY AND MULTIPLE LAYERS
We train two-layer and three-layer ReLU networks and study the scaling of test loss with the di- mension d and hidden dimension k. All the hidden layers have units. The experimental setting is the same as in Appendix I.1 except that tasks are processed in batches of size 100 out pool of 25000 for faster training. In the of two-layer ReL.U networks, for three-layer ReLU networks, we adjust the values to a = 3 = 0.01.
0.025, while
Figure 13: Excess risk for two-layer and three-layer ReLU networks pretrained with FO-ANIL and varying d, k and o² = (left) and o² 4 (right). The shaded area represents the standard deviation observed over 3 runs.
Figure 13 shows that the excess risk does not scale with the ambient data dimensionality d but with the hidden problem dimension k. This is evidence that suggests the adaptability of model-agnostic general networks.
meta-learning pretraining extends to
Next, we study representation learning in ReLUU networks. Let (.) Rdd - Rd represent the network function that sends data to intermediate representations before the last layer. Then, compute the best linear approximation of ff as follows: Let X R be a matrix with each row is sampled from the d-dimensional isotropic Gaussian distribution. Solve the following minimization problem over all B E Rdxd,
argmin BeRaxa
||XB-f(X)]33-
where f(X) E R is the output of the network applied each separately. Applying this to each time step & with N = 1000, we obtain B, that approximates the ReLU network throughout its trajectory. Finally, we repeat the experiments on singular values check learning in the good feature space and unlearning in the complement space with the sequence B..
The feature learning behavior in two-layer and three-layer ReLU networks, illustrated in Fig- ures 14 and 16 closely mirrors that of the linear case. Notably, both ANIL, MAML, and their first-order counterparts exhibit increasing singular values in the good feature space. We observe swifter learning with second-order methods. Moreover, there is difference in the scale of singular values between first-order and second-order approaches in two-layer networks. In the context of thyee-layer networks, ANIL and MAML exhibit distinct scales.
40

---

Published as a conference paper at ICLR 2024
FO ANIL
ANIL
FO MAML
MAMIL
12.
1000
2.5
15000
Iterations
Figure 14: Evolution of average (left) and largest (right) squared singular value of B B, during FO- ANIL pretraining with two-layer ReLU networks. The shaded area represents the standard deviation observed over 3 runs.
FO-ANIL
ANIL
FC-MAML
MAMIL
X2 (8) M8)
22 (87
Figure 15: Evolution of average (left) and largest (right) squared singular value of B Br during FO-ANIL pretraining with two-layer ReLU networks. The shaded area represents the standard deviation observed over runs.
In Figures 15 and 17 the dynamics in the complement feature space for two-layer and three-layer ReLU networks depicted. While the average singular value exhibits decaying trend, contrary to our experiments with two-layer linear networks, the maximal singular value does not show a similar decay. Note, however, that the scale of the initialization smaller than in the linear case and singular values in all complement directions remain small when compared to good feature directions in the pretraining phase. This smaller initialization due nonlinearities in ReLU networks; the linear approximation yield smaller singular values than the weight matrices of the ReLU network which initialized the same scale as experiments with linear networks (for three-layer networks, their product is of the same scale). As elaborated further in Appendix A| this behavior in the complement space is also observable in two-layer linear networks under small initializations and is influenced by the finite number of tasks opposed to the infinite tasks considered in Theorem
[8) 88)
10-
14000
10000)
35000
20000
20000
10000
100000
12000
200000
25000
Iterations
Iterations
Figure 16: Evolution of average (Left) and largest (right) squared singular value of B. B, during FO-ANIL pretraining with three-layer ReLU networks. The shaded area represents the standard deviation observed over 3 runs.
Overall, experiments with two-layer and three-layer ReLU networks show that they learn the k- dimensional shared structure with higher magnitude than the rest of the complement directions. This implies the learning of shared task structure and good generalization under adaptation with
41

---

Published as a conference paper at ICLR 2024
FO ANIL
ANIL
FO MAML
MAMI
X2 11.11 (8)
Ag. (10) (b)
2-
Figure 17: Evolution of average (leff) and largest (right) squared singular value of B B, during FO-ANIL pretraining with three-layer ReLU networks. The shaded area represents the standard deviation observed over runs.
few samples, and might indicate that Theorem ! and Proposition ! could be extended to nonlin- ear networks. The regularization effect of model-agnostic meta-learning on complement directions could be better seen with initializations that result in a linear map with high singular values in every direction. We leave detailed exploration of the unlearning process to future work.
42


Reviews:
[{"review_id": "LRT7e4Mnyz", "reviewer": "Reviewer_4g2e", "summary": "This paper studies the ANIL meta-learning algorithm, and proves that the meta-learned feature extractor converges towards a fixed point.\nThis result is obtained under the following assumptions:\n\n- The learned architecture is a two-layer linear network, possibly over-parameterized.\n- The algorithm is ANIL with first-order gradient (ie, FO-ANIL where the adaptation loop is not differentiated).\n- There are infinite tasks but finite samples per tasks.\n- Each task is generated by with a ground-truth linear regressor and additive noise.\n\nAdditionally, it shows that such fixed point yields a good performance after only a single step of gradient descent adaptation. The authors provide an extensive (and insightful!) discussion of their results, and validate them empirically under similar assumptions as above.", "questions": "- While ANIL is typically motivated as an approximation to MAML, it much more closely resembles metric-based few-shot learning such as prototypical networks, metaoptnet, or R2D2 where the main difference is the algorithm to solve the linear head. How difficult would it be to extend your results for ANIL to these methods?\n- In section 4, you discuss overfitting at the end of the “infinite task models” paragraphs. It isn’t obvious to me how unlearning the orthogonal space reduces overfitting and why there’s no need to unlearn it with infinite samples — could you expand on what is meant? I’m trying to understand how this observation may translate to the other methods mentioned above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Strong and correct theoretical analysis: the authors obtain non-trivial results for FO-ANIL and — to the best of my assessment — their analysis is correct. I especially like the two-steps approach, which first shows the existence of a fixed-point representation (Theorem 1), and then showing that this fixed-point achieves low excess risk (Proposition 1, Appendix C).\n- In-depth discussion of results: my favorite part of the paper is Section 4, where the authors discuss the implications of their results. For example, they clearly contrast their result with prior work and argue that, in the over-parameterized regime (large width), model-agnostic methods such as ANIL can outperform traditional multi-task methods. (See “Superiority of agnostic methods” in Section 4.) They also show that this advantage does not appear with infinite samples thus motivating future work to pay attention to this regime. They also bring attention to the importance of a full-rank initializations, which is a generally understudied area in gradient-based meta-learning, potentially responsible for the many instabilities we observe in practice.", "weaknesses": "- Significance: while the analysis is strong, I wonder if it is relevant. FO-ANIL is almost never used in practice (ANIL is cheap enough), and definitely not with (two) linear layers. Thus, could the authors bring forward the insights that may apply to more realistic settings? I know others also consider this setting (eg, Tripuraneni et al., 2020) but they tend to include insights beyond parameter convergence (eg, effect of task diversity).\n- Limited scope of empirical results: since this paper studies a highly idealized setting, I would hope the authors could provide results echoing their theorems in settings where the assumptions are relaxed. For example, what about multiple non-linear layers? Or 2nd-order ANIL (or even MAML)? Or a cross-entropy loss, which would be relevant to classification problems? Or, better, could we validate these insights on benchmarks widely used in the literature for real-world algorithms? ANIL is easy and cheap enough to implement that these asks are not unrealistic and would strongly support the authors’ results.\n- Novelty: the authors emphasize that their analysis is in the finite-sample regime, which they successfully argue is significantly more relevant than infinite-sample one. However they are not the first ones to study this setting: as they mention, Collins et al., 2022 already consider it, but Ji et al., 2022 (Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning) and Fallah et al., 2021 (Generalization of Model-Agnostic Meta-Learning Algorithms: Recurring and Unseen Tasks) also do, and provide convergence guarantees and generalization bounds for MAML, a more generic algorithm than ANIL. Neither of these works are mentioned in the paper, so I’d encourage the authors to discuss the novel insights they bring to the meta-learning community which are not already available in these works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eL48P4CZTr", "reviewer": "Reviewer_FVcv", "summary": "The paper studies the shared representation learned by first-order ANIL in a linear multi-task model. It theoretically shows that in the case of infinite tasks and overparameterized width, FO-ANIL succeeds in extracting the low-rank ground-truth shared structure while learning to ignore orthogonal directions allowing for fast adaptation. It empirically validates these theoretical findings contrasting the low-rank FO-ANIL solutions with those found via multi-task learning.", "questions": "1. In your theoretical analysis you assume matching depth in the model architecture. Since this is a linear model, is it possible to say something about the case where we have a mismatch in terms of the number of layers? (given that in the linear case this does not change the expressiveness of the model)\n2. Have you tried adding a nonlinearity to your model in the empirical experiments? I wonder to what extent this affects the main conclusion of learning the right subspace and unlearning the orthogonal one.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well motivated, presents its results clearly and is well embedded in existing literature.\nThe findings it presents are interesting and novel.", "weaknesses": "The paper clearly motivates the need to study a simplified linear, first-order, single gradient-step version of ANIL. Nevertheless, a small-scale empirical verification to the extent by which violating some of these assumptions affect the results could have further strengthened the paper.\n\nWhile proofs for all theoretical results are provided in the appendix, the code for reproducing the small empirical section was missing upon submission to the best of my knowledge. Providing it would increase my confidence in the reproducibility of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTDih38GuW", "reviewer": "Reviewer_aoqC", "summary": "This work theoretically studies the ability of first-order ANIL to find useful generalisation points for a network trained on one subsequent gradient descent step. Firstly the authors show that ANIL is able to align to the same feature space as the ground-truth data generating model and also completely ignores the orthogonal complement of this feature space. This implies that ANIL is learning an appropriate representation space for the set of possible tasks, essentially affording the model an initialisation with useful features. The benefit due to this useful initialisation is then characterised by bounding the network's risk for a new task after a single gradient step. This is shown to have better scaling when the network hidden layer is smaller than the original input feature space. Experiments demonstrating the alignment of the features space learned by  ANIL to ground-truth are shown and some results showing the benefit of ANIL over ridge regression are also shown.", "questions": "1. Can Equation 7 not be simplified further since $\\Sigma_*$ is proportional to the identity matrix and $B_*$ is orthogonal?\n2. The scaling law on the generalisation error of FO-ANIL is based on $k$ while linear regression is based on $d$. This is used to justify that FO-ANIL outperforms regression, but does this only hold when ground truth representation space is smaller than the input space $k < d$? What if $k>d$?\n3. On the first line of the subsection titled \"Initialisation Regime\" the bounded initialisation for Theorem 1 is mentioned. This seems different to what is stated in the theorem. Where exactly does this need for bounded initialisation appear in the theorem or its assumptions?\n\n## Minor Points\n1. \"The goal of meta agnostic methods\" is a typo.\n2. Why is the highlighting misaligned for the pdf? There may be a rendering issue.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "## Originality\nThis work uses a teacher-student style of analysis where a ground-truth model generates data for the subsequent trained model. This is a well-established method of analysis but I have not seen it applied to ANIL or meta-learning. Thus, this work presents a sufficiently original setting for theory. It is also made clear how this work departs from other similar analysis in recent literature such as [1].\n\n## Quality\nAssumptions of this work are clearly stated and the limitations made explicit. The setup is also appropriate for studying the desired phenomena. The results and conclusions drawn from the theory are accurate and clear. Similarly, the interpretation of experimental results is also clear and direct from what is shown. The experimental design showing the approach of the meta-learned feature space singular values is also appropriate and supports and complements the theoretical results. The logical progression of the arguments made in Sections 2 and 3 is also intuitive and follows a clear structure which is appreciated.\n\n## Clarity\nThe paper is well written and figures are clear and neat. Notation is used appropriately and is intuitive. Of particular note is how the mathematics is discussed. The language is clear and precise. This results in the mathematics and notation supporting the written discussion very well. I don't feel like it is always the case that both forms of exposition are of a high quality and it does help this work greatly. The Introduction is also particularly clear and establishes the setting of the paper well.\n\n## Significance\nThe significance of this work is a particular strong points. The results shown here are practical and insightful. The fact that ANIL provably learns features is an important result and I can a lot of future work tying into these results - even just by analogy or inspiration, for example in work on the lottery ticket hypothesis and continual learning. Additionally, the result of a convergence rate provides a helpful degree of nuance to the results and one which I can see guiding practical intuition and future work which aims to converge quicker.\n\n[1] Liam Collins, Aryan Mokhtari, Sewoong Oh, and Sanjay Shakkottai. Maml and anil provably learn representations. arXiv preprint arXiv:2202.03483, 2022.", "weaknesses": "## Quality\nThere are unfortunately some weaker points to the quality of this work as well. I do not think enough is done to thoroughly guide a reader through the actual theory of this work. While I appreciate that presenting theory in a page limit is difficult - and I do not propose for full derivations or anything of the sort be moved out of the appendix - there is in general little discussion or proof sketches presented. As a result the theorems and propositions are stated and it is not clear how the statement is derived from the premise. This is better for Proposition 1 than Theorem 1, where Proposition 1 at least has some discussion in the paragraph following immediately where it is stated that the error is decomposed into three terms and then bounded by concentration inequalities. What is two lines of writing then lead to deeper insight and more of this is needed. I see in Appendix A that a proof sketch of Theorem 1 is given and it is quite long. But as far as I can tell sufficient space could be made for a condensed version in the main paper and this should be prioritised as Theorem is the main result of this work which even Proposition 2 relies on. As a result, some points which I think could be very impactful fall flat, such as those at the top of page 5.\n\nOn the point of making space for a proof sketch, I appreciate the in-depth discussion of this work. However I am not certain that it should be prioritised over more technical detail. Particularly the long comparison to [1] seems excessive for the main paper and could be deferred to the appendix along with much of this section. Another example would be the statement \"Theorem 1 answers the conjecture of Saunshi et al. (2020).\" but the conjecture is not stated and so this does not benefit the reader. I would also caution against phrases like \"More importantly, the infinite samples idealisation does not reflect the initial motivation of meta-learning, which is to learn tasks with a few samples. Interesting phenomena are thus not observed in this simplified setting.\". Once again, this is quite conversational and speculative for the main body of work. It also comes across as fairly combative and seems to belittle prior work. Thus, I think it would be best to compress the discussion, keeping mainly the limitations component and the component which introduces Burer-Monteiro since it is used in the experiments. This would make significance space for Appendix A in the main paper.\n\nFinally, I am concerned that the experiment summarised in Table 1 does not quite assess or demonstrate the truth of Proposition 1. Since a scaling law is given in Proposition 1, is it not possible to experimentally show how tight of a bound this is? Rather than just demonstrating that meta-learning can out-perform ridge regression on the input features but not ridge regression on the ground-truth representation?\n\n## Clarity\nI re-iterate that I think clarity is a strong point of this work. If possible, I would suggest a figure summarising the setup be presented. It would certainly be helpful and could significantly aid clarity. Once again space would need to be made, but I think the Introduction could also be compressed to make this possible. This is a minor point and I do think the setting is clear just with the notation. If such a figure could not fit in the main paper then one in the appendix would still be helpful.\n\nGiven the above. I would be inclined to increase my score to a 6 or 7 if the restructuring of Appendix A and Section 4 occurs and is done so clearly. I would increase my score further if experiments supporting Proposition 1 are shown and the figure summarising the setup included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1uBQsTB93", "reviewer": "Reviewer_pdse", "summary": "This paper analyzes the representations learned by first-order ANIL (Almost-No-Inner-Loop), a representative gradient-based meta-learning algorithm. The analysis is performed in the context of two-layer linear networks in an overparameterized multi-task regression setting, where the width of the network is larger than the dimension of the ground-truth parameter subspace. The results generalize prior results from Collins et al. where it is assumed that the width of the network is well-specified and the number of samples in each task is sufficiently large.", "questions": "-\tWill it change the learned representations if we add l2 weight decay regularization to the meta-training objective to avoid overfitting (with finite/infinite samples per task)?\n-\tIs there any empirical evidence suggesting that having more samples in each task can be harmful to gradient-based meta-learning approaches in practice, beyond your numerical experiments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "-\tThe analysis is technically solid and in general well-presented.\n-\tThe discussion and comparisons with prior results are clear.", "weaknesses": "-\tAs suggested by the paper, the main conceptual message, as well as the key distinction induced by the network overparameterization, is that meta-learning not only learns the desired subspace spanned by the columns of $B_*$, but also “unlearns” the orthogonal complement of this subspace. However, the analysis is based on the fact that the number of inner-loop training samples $m_\\mathrm{in}$ is insufficient so the inner-loop gradient descent is prone to overfitting, which is then penalized by the outer-loop meta-training process. What if we use other techniques to mitigate overfitting such as weight decay as commonly used in practice?\n-\tIn the high level, the theoretical results in this paper show that overparameterization/more task-specific data provably _hurts_ generalization in their simplified meta-learning setting. I am not sure if this is consistent with the growing empirical evidence that overparameterization/more data usually _helps_ generalization in practical deep learning. Does this mismatch suggest that the linear model considered in this paper is somewhat over-simplified?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)