Provable Meta-Learning with Low-Rank Adaptations
Authors: Jacob L. Block
Sundararajan Srinivasan
Liam Collins
Aryan Mokhtari
Sanjay Shakkottai
Venue: NeurIPS 2025 poster ()
Keywords: Meta-Learning
Low-Rank Adaptation
Parameter-Efficient Fine-Tuning
Non-convex Optimization
Primary area: theory

Abstract: The power of foundation models (FMs) lies in their capacity to learn highly expressive representations that can be adapted to a broad spectrum of tasks. However, these pretrained models require additional training stages to become effective for downstream applications. In the multi-task setting, prior works have shown empirically that specific meta-learning approaches for preparing a model for future adaptation through parameter-efficient fine-tuning (PEFT) can outperform standard retraining methods, but the mechanism of the benefits of meta-learning has been largely unexplored. We introduce a framework for generic PEFT-based meta-learning to learn a model that can easily adapt to unseen tasks. For linear models using LoRA, we show that standard retraining is provably suboptimal for finding an adaptable set of parameters and provide strict performance guarantees for our proposed method. We verify these theoretical insights through experiments on synthetic data as well as real-data vision and language tasks. We observe significant performance benefits using a simple implementation of our proposed meta-learning scheme during retraining relative to the conventional approach.

Provable Meta-Learning with Low-Rank Adaptations
Jacob L. Block
Sundararajan Srinivasan
Liam Collins Snap, Inc.
UT Austin bloockutteas.eei edu
UT Austin
sundararajansSuteexa..c edu
1collnns@@sapchat.ccm
Aryan Mokhtari
Sanjay Shakkottai
UT Austin & Google Research mokhtari Castariiiaiiinnima utexas.e..... edu
UT Austin
sanjay www.sskttttaiitteaa..e.oo
Abstract
The power of foundation models (FMs) lies in their capacity to learn highly expres- sive representations that can be adapted to broad spectrum of tasks. However, these pretrained models require additional training stages to become effective for downstream applications. In the multi-task setting, prior works have shown em- pirically that specific meta-learning approaches for preparing model for future adaptation through parameter-efficin retraining methods, but the mechanism of the benefits of meta-learning has been largely unexplored. We introduce a framework for generic PEFT-based meta- learning learn model that can easily adapt to unseen tasks. For linear models using LoRA, we show that standard retraining is provably suboptimal for finding an adaptable set of parameters and provide strict performance guarantees for our proposed method. We verify these theoretical insights through experiments on synthetic data as well as real-data vision and language tasks. We observe significant performance benefits using simple implementation of our proposed meta-learning scheme during retraining relative the conventional approach.
fine-tuning (PEFT) can outperform standard
Introduction
Foundation Models (FMs) learn rich representations that are useful for a variety of downstream tasks. The first stage of FM training referred to as pretraining, where combination massive public, proprietary, and synthetic sources of data is used to learn general-purposs purpose model from scratch DA. However, due to the enormous cost of training sute-of-thee-at models on such large datasets, pretraining is largely infeasible for most. Thus, the most popular and viable way to utilize FMs for adapt an existing pretrained model.
specific applications
We consider this problem of adapting a pretrained FM to set of related tasks. We refer to this as retraining, where given a number of tasks with many samples, our goal is recover model that learns the task structure and can be quickly adapted to future tasks with limited samples. In other works this has been referred we adapt the model to new task with few samples in what we denote the fine-tuning stage. In this last stage, we typically employ parameter-efficien fine-tuning (PEFT) methods training heuristics which sacrifice learning expressiveness for improved computational efficiency 7 Ultimately. the purpose of retraining to prepare the model efficient future adaptation, and the effectiveness of retraining method is measured by the model's performance on the fine-tuning task.
pre-finetuning [5] or supervised fine-tuning [6]. After retraining,
Standard approaches retraining involve fitting the model the aggregation of the different retraining tasks. While this seems reasonable and has been empirically successful [ [0] it does not leverage knowledge of the downstream fine-tuning procedure to tailor the retrained model perform well after
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

such adaptation. Rather, it retrains the model to minimize the average loss across the retraining tasks regardless of the PEFT method to be employed later. Thus, there is no assurance that the recovered solution is indeed adaptable to future unseen tasks relative to other possible retraining solutions.
Meta-learning is a natural framework to address this issue, as it explicitly aims to learn adaptable models, typically in low-resource, few-shot settings using tradient-basee adaptations [ [2]. The success of meta-learning algorithms largely attributed to their ability learn useful representations, and Reptile [T3 Recent works have shown empirical benefits to retraining using specific PEFT-based meta-learning methods [16 20).
as even model-agnosti gradient-based meta-learning algorithms like MAML M
have been shown to implicitly learn representations in linear settings 44
15
but theoretical guarantees showcasing these gains have not been established.
Contributions. We propose a framework for PEFT-based meta-learning during retraining which explicitly accounts for the specific PEFT method that will be used for future fine-tuning. While our framework is general to any PEFT method, we focus on Low-Rank Adaptation (LoRA) [ for our theoretical results. Specifically, we consider multiple linear regression tasks where each ground truth regressor is a rank-k perturbation of a common matrix A E RRd where d' is the ambient dimension. Given set of T tasks, our goal is recover A so that we can easily fine-tune to new, unseen task by learning low-rank perturbation using LoRA. With this in mind, we show the following:
We prove that standard retraining, which does not incorporate any knowledge of the PEFT method for fine-tuning, fails to recover parameters that are low-rank adaptable, as the recov- ered model is not low-rank away from A and consequently the test model (Proposition [ As a result, applying low-rank adaptations cannot account for the large rank discrepancy to the test task (Proposition 2) Further, fine-tuning with very large rank account for this discrepancy defcats the purpose of PEFT and results in squared prediction error which grows as O (iTT(AT.a)) with high probability, where n is the number of test samples (Remark 2p. Thus, standard retraining performs worse when given access more tasks.
For our meta-learning framework, we guarantee that any minimizer of the meta-learning loss in the infinite sample case is low-rank adaptable unseen tasks (Theorem (2. Further, we show that if there are at least three retraining tasks, the ground truth parameters are the unique global minima up to orthogonal symmetry (Theorem 33.. As result, LoRA fine-tuning is effective in adapting to the test task and with high probability achieves squared prediction error which grows as O ( (kd) (Corollary 4). In contrast to standard retraining, we achieve the optimal rate which does not include any dependence on T.
We prove in the infinite sample case, every second-order stationary point of our meta- learning loss when applied to two retraining tasks in fact globally optimal (Theorem 4 In this case there are no spurious local minima of our meta-lcarning loss and optimality is completely determined by second-order information. Thus, local optimization methods like perturbed gradient descent can efficiently find global minima.
To the author's knowledge, these are the first results proving that PEFT-based meta-lcarning outper- forms standard retraining methods in any setting. Proofs are presented in Appendix B To verify our theoretical insights, we compare the performance of the standard retraining and LoRA-based meta- learning objectives in a synthetic multi-output linear regression setting. We show clear improvements using LoRA-based meta-learning for all data parameter settings. Then, we apply an implementation of our general PEFT- based meta-learning framework to a vision task on CIFAR- 10 [21 as well as a natural language experiment using the RoBERTa (22 language model on the ConvAI2 (33 dataset with two PEFT schemes: LoRA and last layer fine-tuning. In both cases, we observe that PEFT-based meta-learning outperforms standard retraining.
1.1 Related Work
Meta-learning is a technique for learning models that can be rapidly adapted to unseen tasks by leveraging access to prior tasks during training. For example, Model-Agnostic Meta-Learning (MAML) [T] and Reptile [3] are popular methods that aim to find a model that can be adapted to a new task after a small number of steps of gradient descent the new task's loss function.
Prior analyses of gradient-based meta-learning for linear models consider learning shared feature subspace across tasks [4] 24 [5]. This reduces to subspace recovery problem, as each task reveals

---

a linear measurement of this shared space. In contrast, we study tasks that are low-rank perturbations of shared arbitrary matrix a novel setting with distinct challenges. For instance, in our setting just three retraining tasks suffice to exactly recover the ground-truth parameters (Theorem 3]. whereas subspace-baad based approaches require the number of tasks to exceed the subspace dimension. These differences motivate new analytical tools and lead to qualitatively different results.
Many meta-learning approaches specific to FMs for incorporating PEFT-based adaptation have been proposed. Hong and Jang 17.. Bansal et al. 18, and Gheini et al. [19] applied meta-learning with architecture adaptations that inject task-specific trainable layers within the FM architecture. Hou et al. [66] combined architecture adaptations with parameter perturbations similar to LoRA. They considered a complicated loss that updates the adapters and FM weights over different splits of the data and showed empirical gains over standard retraining and other gradient- based MAML-style algorithms. Aghajanyan et al. [5] proposed multi-task objective that trains an FM on different tasks simultaneously to encourage learning universally applicable representation. It forces the FM to learn shared data representation but allows for task-specific prediction heads. Overall, these works multi-task objective and showed empirical gains over standard retraining strategies. However, their experimental results motivate a deeper theoretical exploration of when standard retraining is insufficient relative to meta-learning approaches, how many tasks are needed to learn a rich representation. and how to best adapt to tasks unseen in the training stage.
each proposed a meta-learning
Lastly, although we focus on LoRA, different PEFT methods have been proposed, including variants of LoRA (25 and architecture adaptations (28 among others. Further, recent works have analyzed the theoretical aspects of LoRA in the fine-tuning stage (29] 3D), but they explored orthogonal directions to the analysis of LoRA-based meta-learning during retraining. Extended discussion of
these prior works is in Appendix A
Notation. We use bold capital letters for matrices and bold lowercase letters for vectors. N(u, Σ) refers to the multivariate Gaussian distribution with mean and covariance matrix Σ. Ia refers to the d x d identity matrix. |1-Il refers to the Frobenius norm. Sa refers the set of d d symmetric matrices, and S is the set of d x d positive semi-definite matrices. Og refers to the set of d x d orthogonal matrices. [n] refers to the set {1..... n). For a matrix X E R=xx im(X) and ker(X) refer to the image and kernel of X, while (e) E mm denotes the column-wise vectorization of X. For subspaces M, N, dim (M) refers the dimension of M and M + N {ax + y x M, y E NJ. If M n N = {0). we write the direct sum M + N.
2 Retraining and Fine-Tuning Schemes
We briefly recap the optimization process for standard retraining of an FM across multiple tasks followed by fine-tuning on a downstream task. We then introduce a general framework for PEFT-based meta-lcarning which adjusts the retraining phase to incorporate insights from fine-tuning.
2.1 Standard Retraining Then Fine-Tuning
Consider a collection of T' tasks of interest T [T) where each task Tr is drawn from task distribution D and consists of n, labeled examples T, = (x yt Di Without loss of generality, we assume consistent dimensions across tasks, so æt.. E Rdd, Vt.j E RRyy for all tasks and sample indices j. Let X, TRd2 xx, and Y, Rd, denote the concatenation of the respective input samples and labels from task t, and consider a model Φ W) Rd - Rdd parameterized by weights W that maps feature vectors predicted labels. We abuse notation and write &(X, W) to denote the concatenation of (( W) for j E [me]. Typically W = (W,,... Wm.) is a list of matrices where W; E Rdxd parameterizes the 7th layer of a neural network. We assume each W, is square for convenience.
Retraining Phase. Given a loss function C, standard retraining attempts to minimize the aggregated loss over collection of training tasks 22 2]. This amounts to solving
WSR = min W
L L W), Y.),
(1)
3

---

where SR stands for Standard Retraining. The above optimization problem seeks a set of universal parameters that define unique mapping function capable of translating inputs to outputs across all tasks involved in the retraining phase. We denote the corresponding model as Φ -; WSR).
Fine-Tuning Phase. In the subsequent fine-tuning, PEFT is used to refine the model to fit downstream task with fewer labeled samples. Formally, consider an unseen task Tr+1 drawn from the same task distribution D. We define PEFT as any method which fits the model to task Tr+1 by fixing W = WSR in the original parameterization and fine-tuning the mapping क6 WSR) using additional parameters O. For example, I could parameterize trainable perturbations of WSR
trainable layers inserted into the architecture of the retrained model 7 25 51. We denote the fine-tuned model as PFr : WsR, 0) _ + d and again abuse notation by writing @ WSR, 0) to denote the concatenation of ( WSR, 0) for each j E [NT+I]- During the fine-tuning stage, the goal is to find the optimal additional parameters, 0, that minimize the loss for the downstream task TT+,, solving
min
WR. 0), Yr+1).
In particular, when LoRA is used for fine-tuning, the model is adapted to task TT+1 by fixing the ar- chitecture and the retrained weights WSR and only training low-rank perturbations for each of the ma- trices W I. I..... WsR.m. For rank-r adaptations, we parameterize O = ((Q1, Vi),... (Qm. Vn)), where Q; V, Rdxr are the factors of the low-rank adaptation of the 7th matrix in WSR- The fine-tuned model is just the original model where the 7hh weight matrix W; is now perturbed to be W; + Q.V.". For Q, V. E (Rdxr)" define the LoRA loss
LIORA (Q, ; W) = -c))) (x((TH): (W: Q.V. m =.)
YT-1).
(3)
The LoRA fine-tuning optimization problem is then
min (Q, V WSR). Q.
(4)
This pipeline seems reasonable as we first fit the model to the aggregation of the retraining tasks which we hope will promote learning the general structure of the tasks drawn from D. However, nothing about standard retraining promotes learning an adaptable solution relative to other candidate solutions that fit the retraining tasks. Next, we introduce general meta-learning framework which explicitly incorporates the adaptation mechanism during retraining.
2.2 PEFT-Based Meta-I.earning
Since the ultimate goal of retraining is to perform well on an unseen downstream task, we study a general PEFT-based meta-learning (PEFT-ML. objective that explicitly fits weights and adapter parameters to the training tasks. Rather than training single model on the aggregation of the retraining tasks, we instead incorporate the adapters during the retraining process and learn adapted models for each task. Let (()) be the set of adapter parameters for the th training task Te The PEFT-MI. objective searches for a single set of base weights Meu such that for all E [T]. the th adapted model &rr( : W Meeta ((!)) minimizes the loss over the training task Te. More precisely, we define the proposed PEFT-ML objective
T
W Meta
min
Σ C.WW,,
(5)
where (W) denotes the optimal loss on task & after fine-tuning:
L. (W) min L ( Xx W 0(0) v.)
((
When we use LoRA as the udaptation method, we define Qtt), ((t) (RRxx))" as the list of factors of the low-rank adapter Q (1)))) applied to the 2h weight matrix for the 4t task. Then the inner objective (W) reduces
C (W):
min FLARA v()) W). V(9
4

---

In this case, we refer to the objective function in 5 as LORA-ML. This proposed optimization problem is designed replace the standard retraining objective in T After solving we recover base parameters Weta that are explicitly designed to be adaptable to downstream tasks drawn from the same distribution those seen in retraining. To perform finetuning, we then run the exact same minimization in but using retrained weights WMet instead of WSR.
3
Main Results
To establish our theoretical results, we consider T 2 1 multii-uttput linear regression retraining tasks (T+))=1 and one downstream test task TT+I. where the ground-truth regressor for each task is low-rank perturbation of a common shared matrix. Precisely, for each & E [T we assume task Ti independently drawn from distribution A- which is associated with some arbitrary fixed matrix A' E Rdxd and intrinsic adaptation rank & < d. Each task T. D is parameterized by the shared matrix A and a task-specific rank-r matrix Rí such that the samples (Zt.j yt.) E Rd R from T, are related by the noisy linear transformation
yt. = (A* R2 ) 6t,j
where Et independently generated noise terms Et,j (( We assume the inputs æ are i.i.d with zero-mean E [ = 0 and covariance E [ La Lastly, we generate the ground truth rank-r adaptations R2 as the symmetric outer product of random Gaussian factors U
= s.t. vec(U;) ~ (N
The above generative model defines the input-output relationships for each task similar linear models, differing from each other only by a low-rank perturbation R," We construct the adapters R; symmetric for convenience of analysis, but note that the limitations of standard retraining which we demonstrate in Section 3.1 also hold for general adapters (Appendix(C)
Remark For convenience, we require a mild sense of task diversity and assume that any collection ofr S number of columns chosen from the set of all columns of [U; 1-+1 are linearly independent. We note that our generative model for each Uł ensures this assumption holds with probubility 1.
The learner uses the linear model (ax; A) Aæ for A E Rdxd and retrains on tasks Ti...., with the ultimate goal of efficient adaptation to Tr+1 using LoRA. The aim to recover the parameter value Á = A" in the retraining phase so that the fine-tuned model Pyr A, Q, V) = (á+QVT) > fits the data distribution of any downstream task also drawn from D for proper rank-k adapter QV?
. Tr
We define the finite-sample loss function for task t as
R2
C (A)
Am
(7)
2n
and we define (A) the shifted and scaled infinite sample loss:
(A)
Rr..y ITT (A))
o² 2 - A -
UUU A
(8)
o²
We consider the setting where for the retraining tasks S T we have large ny, but for the test task NT+1 is small. This reflects practical scenarios where we have access to large retraining datasets compared the low-resource fine-tuning task. Thus, we assume access to the infinite sample loss functions LJ for the retraining tasks / < T. Then, for ease of notation, define number of test task samples. We ultimately aim to use LoRA to fit the finite-sample test task loss CT efficiently in n. Given a learned representation A E R from retraining, the fine-tuning problem using LoRA with rank reduces to
N7+1 1 as the
min Q. VeR
C7 (A QVT)
(9)
5

---

Since QVT can parameterize any rank-r matrix. 9 is specific parametrization for what is commonly known as reduced rank regression [31. is clear that even realize the optimal regressor A' Á + UT+ UT+1 for QVT, we need r 2 rank(A' Á + + Fur- ther, results in reduced rank regression and matrix sensing in general reveal the importance of rank(A" + T+1 - A) in terms of the hardness of minimizing +7 (A
QV
2.11 Consider A Bddd and let rank A + U1+1U -)), Let Q", V E Ráxr minimize + (A + QVT) over all rank-r factors Q, V E Raxr and let XT+1 21+111..... ,TT..,, denote the matrix of test task inputs. Denote the matrix of prediction
Lemma 1 Bunea al.
A).
errors = (A* + U7 UT XT+1 (A+ (A+Q'V Q*V* )XT+1- )
Then V/y > O,
E 24(1 + orrd n
1
X1+1) 2 21-
P
(10)
n
The squared prediction error scales linearly with rd. This matches the information- lower bound to learn rd number of parameters and is minimax optimal over all rank-r matrices when the singular values of X1+1 are uniformly bounded 33 Thus, a larger rank of A - Á 1 inflates the fine-tuning prediction error, as hope to recover Á A that rank(A' A U ( k. We next compare the standard retraining
U
| and LoRA-ML 6 objectives.
3.1 Negative Results for Standard Retraining then Fine-Tuning
Consider standard retraining then fine-tuning as a candidate for ultimately minimizing 9 The learner first finds single matrix ASR that minimizes the sum of losses Ei= L7:
Ask = arg min
Σ
A' + U,U - -
(11)
Then when given test task TI+I, the learner solves mill VERaxt CTy1 + QVT). However, this strategy suffers substantial loss in both the retraining and fine-tuning stages. Notice the loss in m convex and quadratic in A, so the first-order optimality condition shows that
T Σ U ASR = A + T t=1
(12)
Thus, ASR recovers A added to the average of the retraining ground truth adaptations U.U;T. However, ASR performs poorly on all of the retraining tasks, standard retraining is unable to disentangle the common structure A from the task-specific adapters U.U.".
Theorem 1. Let U* = .U?....,U;) Then,
U.).
Eu Σ
(Ass)) -T- (1-1)kd(d+1l 52 (Tkd))
Thus, ASr suffers significant loss on the retraining tasks when averaged over the generation process of ground truth parameters U* Further, ASR is not low-rank adaptable to the test task. Crucially, the intrinsic dimension of the test task is rank(A" + U U++1 ÁSR) min d, k( 1)}, so an adaptation rank of min(d. k(T 1)} is required even achieve the ground truth test task parameters.
Proposition If test fine-tuning rank r < min (d, K(T + 1)}, then (Q, V A > 0 for all
rank-r adapter factors Q V E Rdxr
Even though the test task parameters are only rank-k away from A", standard retraining fails to exploit this structure and inflates the necessary rank to min(d, k(T + 1)). Thus, standard retraining actually recovers worse representations as the number of tasks T grows. this case, failing to fine-tune with large enough rank causes significant loss.

---

Proposition 2. For large number of retraining tasks T - oo and test fine-tuning rank (Q, V: Asu) = Ω r)k2) for all Q, V E Igdxr.
As the number of retraining tasks grows to infinity, the squared error between the test task recovered parameter and the ground truth is determined by the under-specification of the fine-tuning rank relative to the ambient dimension
The above propositions show the cost of under-specifyee the fine-tuning rank relative to the large intrinsic dimension of the test task which results from standard retraining. Conversely, applying the necessarily large fine-tuning rank min(d, d, k(T + 1)} both defeats the purpose of low-rank adaptation and still incurs large prediction error when fine-tuning with limited samples.
using Ase adapted with LoRA using rank = min(d, k(T + +. This can achieve optimal population risk but suffers in the fiite-ssmpll setting. Using Lemma I we can only hope achieve squared prediction error of order O min(kT,a) n
Remark 2. Consider the sample loss (
when wine-fineturr fine-tuning, which is much larger than the optimal rate ( (kd) if we had in fact recovered the ground truth A during retraining.
Thus, standard retraining recovers parameters that cannot be efficiently low-rank adapted to new tasks. In contrast, our analysis of LoRA-ML for retraining shows much improved performance.
3.2 Results for LoRA-Meta-Learme
Consider applying (6) to this problem instance. We introduce low-rank adapters during the retraining phase to model the different training tasks and search for value of A such that for all Tr, the loss C after running LoRA on Ti is minimized. This promotes values of A that can be easily adapted to unseen tasks downstream. We use the LoRA-ML loss but with symmetric low-rank adapters U,U," for the t task Te in retraining. We still use asymmetric adapters for fine-tuning on the test task with loss 4T+ The LoRA-ML loss given access to infinite sample task losses L3 is then
T
M (A) = Σ
min (A
(13)
U,
t=1
Define the concatenation of each U, as U = (U UT) E (RRdx))². Then minimizing 13 is
equivalent to solving mİn L" U) where
T
(* U) 1
Σ
(14)
We have seen that standard retraining does not recover an optimal solution, but it is unclear what the global minima of this new objective function are and if they can be easily found. Note that by fixing A. (14 is T independent symmetric matrix factorization problems, and by fixing U. (14) a convex quadratic problem over A. Despite these well-understood sub-prot sub-problems, joint minimization over A and U presents challenging variable interactions that complicate the analysis. Nevertheless, we employ careful landscape analysis of ([4) to address these questions.
3.2.1 Landscape of Global Minima of 14
We first show that the objective well- cell-posed, i.e., minimization of L leads to an adaptable solution.
Theorem 2. L (Á, = 0, then Á = A + C where rank(C) < 2k
Any point is global minimum of (4) and only if achieves zero loss. Theorem 2 guarantees that the values of A that achieve global minimization of (14) are at most rank-2k away from the ground truth parameter A" Then, the remaining intrinsic dimension of the test task is just 3k < d.
Corollary 1. Cf (A, Ū) = 0, there exists a rank-3k adapter QVT such that Ci+ V: A) = 0.
Since the sufficient LoRA rank for fine-tuning is just 3k: we realize a much improved fine-sample prediction error.
7

---

Corollary 2. Let (A, = 0 and let Q E Rdx3k minimize L (A+ QVT) over all Q. E Rdx3k Then, A Q'V*T satisfies Lemma with r 3k.
V
Thus, retraining with LoRA-ML leads to squared prediction error on the task task which grows asymptotically as O
( kd), Although the unnecessary factor of 'T' incurred by standard retraining is avoided when using LORA-ML, the rate still contains an additional factor of 3 over the ideal case when r = k since A' is not guaranteed to be recovered exactly. However, this minor discrepancy is mitigated when the number of tasks satisfies T 2 3. In this case, exact recovery of the ground truth parameter A possible.
Theorem 3. For any number of tasks 2 3 and ambient dimension d 2 3k, if (A,Ú) 0 then Á A and U,U,"" = U, for all t E [T]
This guarantees that the ground truth parameters are the unique global minimum up to orthogonal symmetry when there are three or more tasks, as long as the ground truth adaptation rank k is relatively small compared to the ambient dimension d. This result is surprising, as most theoretical results for multi-task learning require stricter conditions on the number of tasks T, typically where T is required to be larger than the effective task dimension [34 [4]. However, we establish this uniqueness result for the absolute condition T 3. As a result, we only need a rank-k adaptation to realize the test task.
Corollary 3. For T if (A, 0, then 3Q, V E Bgd such that 1 VQ. A) O.
We then achieve the desired fine-sample prediction emror.
Corollary 4. For T 2 let L (Á, Ú) and let Q", V E Raxk minimize C++1 (A QQT) over all Q V E Rd Then, Å + Q satisfies Lemma 1 with k.
Note that the condition 7 3 is necessary to establish Theorem if there only two tasks we can construct ground truth parameters such that the induced loss L has infinite solutions. See Appendix F. for example.
Summary. These results show that all global minima of the LoRA-ML objective are low-rank adaptable to the downstream task and achieve finite- inite-sample test task prediction error which grows as O kd Crucially, this avoids the factor of I'' incurred by standard retraining. Further, if 2 3, minimizing the LoRA-ML objective guarantees recovery of the ground truth parameters.
3.2.2 Algorithms for Minimizing T4
As shown above, minimizing the LoRA-ML objective (14) leads to recovery of the ground truth parameters, with small rank-2k error term when T 2. We prove that this minimization problem can always be solved by local optimization methods when there are two retraining tasks.
Theorem 4. IfT = and the ambient dimension 2 2k, then L (A, Ú) = if and only if (A, Ú) is second order stationary point (SOSP) of L*
When 7 = 2, local optimization algorithms for finding SOSPs, such as perturhed gradient descent and cubic-regularized Newton method, can efficiently minimize the meta-learning objective. Surprisingly, when there are three or more tasks, numerical experiments (see Appendix F.2) show that adversarially picking U. can result in specific instantiations of (I4) with spurious local minima. In the next section, we perform extensive numerical experiments for various values of T' which show that these spurious minima are almost never found in practice and vanilla gradient descent is sufficient to minimize (14).
Experiments
We test our framework across three settings. We first consider synthetic data for linear models using LoRA fine-tuning to validate our theory in Section 31 Next, we conduct real-data experiments vision and language tasks, comparing general PEFT-ML retraining approach to standard retraining and Reptile [13], a gradient-base based meta-learning method. To highlight the flexibility of our framework, we consider
different fine-tuning schemes: LoRA and fine-tuning just the last layer.
Synthetic Data. We first test our model on synthetic regression tasks, relaxing the assumptions from our theoretical results. We consider data Yt. (A* + R jais Eij for task t and [ne], where
8

---

10a
M.
Setting L to be n=100, LoRA-ML
n=10000,
T:2, SR
n= 5R
T-3, SR
Test Task Loss
10
n-1000, SR
SR
n=10000,
Test
109
10a
20-2
0
2000
4000
6030
B200
10000
200
400
600
800
1000
lteration
lteration
Figure 1: Linear model fine-tuning performance varying number of retraining tasks T (left) and number fine-tuning samples n (right) for LoRA-MI. (ours) and standard retraining (SR).
Table 1:
Mean test accuracies and standard errors for PEFT-ML and standard retraining methods using LoRA-based and last-layer-based fine-tuning for adapting subset of CIFAR-10 classes.
LoRA Fine- Tuning
Last-Layer Fine- Tunnng
Method
Mean
Std. Err.
Method
Mean
Std. Err.
LoRA-MI.
86.09
0.35
Last-Layer-Mi
83.90
0.17
SR+LoRA
85,29
0.17
SR+Last-Layer
81.16
0.22
Reptile+LoRA
81.30
0.42
ReptileLLast-Laeer
72.55
0.39
Fr.j. 61j, and R2 are generated just as in Section 3 A is constructed by sampling each entry as an i.i.d. (0, 1) random variable. We define parameters N, n such that the number of samples per retraining task n4 N for all S T, and the number of test task samples #T+1 the mean squared error loss, we rum gradient descent on the standard retraining (T) and the LoRA-ML (5) objectives. After recovering Á during retraining, we apply the low-rank adaptation QVT when fine-tuning the test task. When T 2, we rank-3k adaptation during fine-tuning account for the inexact recovery explained in Theorem 2 and otherwise use rank-k adaptation.
In each experiment we vary one hyperparameter from fixed set of values and plot the prediction enror between the recovered model and the ground truth model 1 ES || (Á QV ) Results are averaged over 5 trials, with the shaded region showing the full range of values. Figure shows that LoRA-MI. retraining significantly outperforms standard retraining (SR) for all data settings. For T' 2, achieve global minimization and recover an adaptable solution. Thus, even though there may exist spurious local minimizers, do not encounter them in practice. Further, the fine-tuning performance after standard retraining worsens with larger T. This is supported by our theory in Section 3.1 which shows that T increases, standard retraining recovers worse solutions that leave larger intrinsic dimension for the fine-tuning stage. See Appendix [a hyperparameter details and further ablations.
that applying gradient descent to the LoRA-ML objective is sufficient to
Vision Experiments. We use CIFAR-10 (21). and define T = 4 binary retraining tasks involving classification between consecutive class labels. Specifically, task classifies between classes 1 and 2, task 2 between classes 3 and 4, etc. The test task is binary classification between classes 9 and 10. We evaluate our PEFT-ML framework using a model based on the MLP-Mixer architecture (35). We compare two different adaptation methods: (i) LoRA with rank adapters, and (ii) fine-tuning only the last layer. For both PEFT strategies, we evaluate three retraining strategies: (a) PEFT-ML for each PEFT method (LoRA-ML, Last-Layer-ML). (b) standard retraining, and (c) Reptile, popular gradient- based meta-learning method. We report mean results along with standard errors across 5 trials in Table [ Across both fine-tuning methods, PEFT-ML retraining consistently yields the best performance. Interestingly, standard retraining outperforms Reptile, as Reptile assumes gradient-based adaptation which is misaligned with the PEFT methods used adapt at test-time.
Language Experiments. We test our PEFT-MI. framework using the ConvAl dataset with the RoBERTa-base model (22). C12vAI2 comprises conversations between two personas, where each

---

Table Mean accuracies + standard error for 10 tasks using different retraining and fine-tuning method combinations on ConvA12.
Algorithm
T1
T3
T4
TS
T7
T8
T9
T10
Avg
1,9
SR+LoRA
59 2
31
4
50 17
40 2
24 1
20 = 2
41 = 10
36 1.2
23 1:5
26 1
35.0 + 4,1
ReptileHLoRA
10
Last-Layer-Mi ML
55.12
51
44 2
3614
55 1511
4
33 14
34.14
40.2
1 1.5
SR+Las--Laer
41 41
9
33
36
=5
28
31
24
15 6
6
24.5 = 4,2
Reptile+Last Layer
35
4
18
1813
29
19
4
95.19
17
16.14
19.4 1
Table 3: Mean accuracies (averaged over all test tasks) = standard error for different retraining methods at varying LoRA ranks during fine-tuning.
Algorithm
Rank 1
Rank 4
Rank
Rank 16
LoRA-ML
44.7 1 0.8
48.6 + 1.6
47.4 + 1,9
48.2 + 1.5
SR+LoRA
36.3 1 4.1
35.5 + 4.3
35.0
+ 4.1
37.1 + 4.1
Reptile+LoRA
26.6 + 5.8
27.7 5.3
28.4 = 5.0
27.8 = 5.9
persona is associated with a list facts that informs their responses. We model learning the dialogue continuations of each persona as different task, where aim select the correct continuation retrain with T = 10 tasks and then fine- fine-tune
from a set of choices given the conversation history. the model from the best-performing epoch to each of the 10 test tasks. We run 5 trials and report the mean accuracy and standard error on the held-out data for each test task in Table 2 same setup and method naming conventions in the vision experiments. basTe meta- meta-learning performs best for both adaptation methods.
using the
Table 2 reports results where LoRA-MI. retraining and all fine-tuning uses LoRA rank of 8. To assess robustness to the choice of rank, Table performance across different values. LoRA- ML consistently outperforms the baselines, and while choosing rank larger than clearly improves the performance LoRA-MIL ranks 4, 8, and 16 yield similar results. We note that LoRA-ML is always retrained with the same rank used for fine-tuning. Additional details for both real-data experiments are provided in Appendix
Conclusion
We presented PEFT-ML for retraining FM collection of tasks to prepare the model for subsequent downstream fine-tuning, We theoretically demonstrated strict performance gaps between standard retraining and the PEFT-ML objective using LoRA (LoRA-ML). Empirically, retraining with PEFT-ML outperformed standard retraining for adapting to unseen downstream tasks using LoRA and last-layer fine-tuning. Future research includes extending our theoretical analysis to more general adapters and more complex model architectures, such as transformers.
Acknowledgments
This work was supported part by NSF Grants 2019844 2107037, and 2112471, ONR Grant N00014-19-1-2566, the Machine Learning Lab (MLL) at UT Austin, the NSF AT Institute for Foundations of Machine Learning (IFML), and Qualcomm through the Wireless Networking and Communications Group (WNCG) Industrial Affiliates Program. We are grateful for computing support on the Vista GPU Cluster through the Center for Generative AI (CGAl) and the Texas Advanced Computing Center (TACC) at the University of Texas at Austin.
10

---

References
[1] Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training
of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computationnl linguistics: human language technologies, volume (long and short papers), pages 4171-41888 2019.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1900 2020.
[3] Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Hassan Awadalla, Nguyen Bach, et al. Phi-3 technical report: A highly capable language model locally on your phone. Technical Report ASR-TR-20244, 12, Microsoft, 2024.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748-878 1748-8763. PMLR, 2021.
[5] Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with rrefinntuning. In Proceedings the 202/ Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
[6] Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 177-198, 2024.
[7] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[8] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers). Association for Computational Linguistics, 2021.
[9] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unitiedqa: Crossing format boundaries with a single qa system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, 2020.
[10] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-lo-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[11] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[12) Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International conference on machine learning, pages 2927-2936. PMLR, 2018.
[13] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 029999999999...... 2018
[14] Liam Collins, Aryan Mokhtari, Sewoong Oh, and Sanjay Shakkottai. MAML and ANIL provably learn representations. International conference on machine learning, pages 4238- 4310. PMLR, 2022.
11

---

[15] Nikunj Saunshi, Yi Zhang, Mikhail Khodak, and Sanjeev Arora. A sample complexity separation between non-convex and convex meta-learning. In International Conference on Machine Learning, pages 8512-8521. PMLR, 2020.
[16] Zejiang Hou, Julian Salazar, and George Polovets. Meta-learning the difference: Preparing large language models for efficient adaptation. Transactions of the Association for Computational Linguistics, 10:1249-120 1265, 2022.
[17] SK Hong and Tae Young Jang. AMAL: Meta knowledge-driven few-shot adapter learning, In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 10381-10389, 2022.
[18] Trapit Bansal, Salaheddin Alzubi, Tong Wang, Jay-Yoon Lee, and Andrew McCallum. Meta- adapters: Parameter efficient few-shot fine-tuning through meta-learning. In International Conference on Automated Machine Learning, pages 19-1. PMLR, 2022.
[19] Mozhdeh Gheini, Xuezhe Ma, and Jonathan May. Know where you're going: Meta-learning for parameter-efficient fine-tuning, In Findings of the Association for Computational Linguisties: ACL 2023, pages 1602-11612, 2023.
[20] Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. Meta-learning online adaptation of language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4418-4432, 2023.
[21] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[22] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
[23] Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander H. Miller, Kurt Shuster, Jack Urbanek, Douwe Kicla, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander I. Rudnicky, Jason Williams, Joelle Pineau, Mikhail S. Burtsev, and Jason Weston. The second conversational intelligence challenge (ConvA12). ArXiv, abs/1902.00098, 2019.
[24] Kiran Thekumparampil, Prateek Jain, Pranceth Netrapalli, and Sewoong Oh. Statistically and computationally efficient linear mete-repreesnntttion learning. Advances in Neural Information Processing Systems, 34:1 18587- 2021.
[25] Shih- Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference Machine Learning, 2024.
[26] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLORA: Efficient finetuning of quantized LLMs. Advances in neural information processing systems, 36: 10088- 10115, 2023.
[27] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. The Eleventh International Conference Learning Representutions, 2023.
[28] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2790 PMLR, 6 2019.
[29] Uijeong Jang, Jason D Lee, and Ernest K Ryu. LoRA training in the NTK regime has spurious local minima. In Proceedings of the 41st International Conference on Machine Learning, pages 21306-21329 2024.
12

---

[30] Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. In The Twelfth International Conference on Learning Representations, 2024.
[31] Alan Julian Izenman. Reduced-rank regression for the multivariate linear model. Journal of Multivariate Analysis, 5(2):248-264, 1975. ISSN 0047-259X. doi: https://oi.orr//100 1016/ 0047- 25777/555X77)90422111 1.
[32] Florentina Bunea, Yiyuan She, and Marten H. Wegkamp. Optimal selection of reduced rank estimators of high-dimensional matrices. The Annals of Statistics, (012)):22221300))) 2011.
ISSN 00905364, 21 21889666.
[33]
Angelika Rohde and Alexandre B. Tsybakov. Estimation of high-dimensional low-rank matrices. The Annals of Statistics, 39(2):887-930, 2011. ISSN 00905364, 21688966.
[34] Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In International Conference on Learning Representations, 2021.
[35] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 24261-24272, 2021.
[36] Yuqian Zhang, Qing Qu, and John Wright. From symmetry geometry: Tractable nonconvex problems. arXiv preprint 2rX75 2020.
13

---

A Related Work on LoRA-Style PEFT
There is a vast amount of work in developing PEFT methods for FMs. The LoRA algorithm has established itself as a popular and successful PEFT strategy and has inspired various extensions such as QLoRA, DoRA, and others 226 25 27). These algorithms are heuristics for mimicking the full finetuning of FM specific downstream task and have proven to be empirically successful various settings. However, there lack of theoretical analysis on the adaptability of PFMs under LoRA-style adaptations, the ability to efficiently optimize LoRA-style objectives, and the kinds solutions they recover. Some recent works have attempted to analyze different parts of these thcoretical questions.
29 analyzes the optimization landscape for LoRA for the Neural Tangent Kemel regime. The authors show that LoRA finetuning converges in this setting as they prove that the objective function satisfies strict saddle property, ensuring that there are no spurious local minima. However, this focuses on the actual ability of LoRA to converge to the optimal low-rank adapter given an FM, and does not consider the adaptability of the in the first place.
Convergence of LoRA.
Expressivity of LoRA. [30] derives the expressive power of LoRA as function of model depth. This work shows that under some mild conditions, fully connected and transformer networks when respectively adapted with LoRA can closely approximate arbitrary smaller networks. They quantify the required LoRA rank achieve this approximation as well as the resulting approximation error.
B Proofs
B.1 Proof of Theorem 1
By Equation [12 we have that ASR A + + + U In the following the expectation is always taken over U (Ui.... U7), where U/ Bddk satisfies vec(U,) ~ NN Iax) Then,
T
T
Ey.
Ci(
(Ask)
=
A' ,'+ A U
T
==1
Σ
UNU T
U, T
2=1
Σ
U.U -KII
U -kI
Σ E TUU, -
UU:: - kI
T
Σ T
(U:U:-- -ki) -ki)
U. -bx)
TE
-x1l -
+ 1
Σ UUU. - -k1)
T
F.
(UJUTT - ( U.U: - 1
- 2E tr
where the last equality follows from the fact that each U,U," U, are i.i.d.
14

---

Note that the second term pe
(U:U;T
-ki)
|/ is the total variance of the sum of i.idd
FF
matrices U KI, so it is equal to the sum of the individual total variances:
Σ
T
U,
kt)
-T
2 E [V:U:
T
- kI) -
11 = | - [
trr (U; - kI) (E== U,U," - k1 } the sum of total
Further, the third term
2R
covariances between zero mean random matrices UJU+T kI and each U KI. Since each
U is drawn independently. we only pick up the first term of Σ U. kI:
Σ U;u.T kI )} E tr (UTUi - kf) (UTU? - KI)
E tr (UTUTT- kI)
E ||UTUTT -
Combining the above simplifications gives that
T Eu- Σ (ASR)) = 1)B ||iiu - 101l"]]
Then using the fact that each U, is an independent sample of d d Wishart distribution with scale matrix I and k degrees of freedom, we have that
E |NUUT kl |1/2 = kd(d +
B.2
Proof of Propositions
Recall ASR A + E/= UüU;
T
Then for any Q, V E Rdxr
(Q, V: AsR)
1
A' + Uj U-
U
ASR QVT
2
T Σ [=1
1
U
U7+1
U.U; -QV
2
By Remark
we have that rank (Ui-IUTH, - SLUUU Proposition follows from that fact that rank(Q) (QV Sr.
min(d, + 1)). Then
Further, as T oc, the strong law of large numbers implies that 4E U.U:
U U;
kI. Thus for large
T Σ
U U T
U.U.T - QU U U - kI - -I
QV }
(15)
Using classic low-rank matrix factorization results, the Q'V-T that minimizes U - kI - QVTI? will exactly capture the r eigenvectors of U7+1U - with largest magnitude eigenvalue. But, U++JU-T kI has d - k eigenvalues of magnitude k.
so V* can only capture r of them. Thus, | U kI - Q. 2 (d k - r)h
Since Q' minimized this quantity, we have that
(1+ V; ASR) 2 (d - k - k)2
VQ, E Rdxr
Thus, (T+1(Q, V ASR) scales as (d - ks - r z (d k) since & < d.
15

---

B.3 Proof of Theorem 2
Since (* 0 and L 2 we must have that VaC*
Thus, A= A' - 7 - (ū,ū,² -U;U;T) Plugging this into L gives
T
0= --
_
A' + UUUU - A' T Σ 0.U. - -v;;
T
2
Σ
U.U;" - -UU,
1 Σ (U.U.' T -U
2
T ==1
Thus each term of the summation is zero, so for all 1, E [T).
ú,07 - u.u. = Ü, - U
Combining these results gives that
1 T
Á = A
Σ
(U.Ú," -U.Ú,-U;UU:"
T
s=1
A" - -
Let C = -Ū, + T
Then Å A= A" + and rank(C)) S rank((,ūT)) + rank
s 2k.
Note the the effective remaining test-task dimension
rank (A* + +U-- A C) = rank (U7+1UTH -)
< rank (U++1UTT1) + rank (C)
< 3k:
B.4 Proof of Theorem 3
Proof. Since (',AÚ)) 0, we have that for all f, [T].
Ú,Ú, - UU- - -U""
(16)
Applying this to the first three tasks and rearranging gives that
U.U = Ü, + +ü - ي,Ú,
(17)
= Ú, +UJUS - ÜZÚT.
(18)
We first show that im(ú,) imfU)..
Since U > O, we must have that ims()s C im(()) + im and im(ūs) C im((ū))
im(Ug) otherwise there would exist vector ker (Ú,Úī +UJU;T nerr(U ) whose existence contradicts the positive cemi-defnnttneess of U. Thus,
im(Uf) S im(ūr) + im?)
(19)
im(U;) C im(ú,) im;)
(20)
16

---

Using that fact that for subspaces and im3) both sides of 19 and 20 respectively. This gives:
Y.Y. Z, CY = X Y + Z, we can add im (?;)
im (TT @ im(U5) S im(U)) + im(U
(21)
im im(Uf) im((U)) C im((i) im(Ug).
(22)
For t (2,3), we clearly have that dim (im(Üi) im(U;)) - dim (0m)) + (um))m(?)) 2k.
and dim (im(UF) im((U)) 2k. Thus,
(im(UT) @ im(U))) (im(Un) @ im(U))
(23)
(im(Uf) @ im(U3)) - (im(()) + im(U3)
(24)
Lemma 2. (ii((ú)) @ im( (ū;) @ im(UU)) = im(ú1)
Proof. Clearly, im(ūr) C ((im(Ü)) @ im(U:)) n [im((1) @ im(UU))). To show the converse, consider æ E (fim(U)) @ im();) n (im(U,) @ im(Ug)))
By assumption there exists some a, b, c, d E R such that
Üia b
(25)
Ü Ujd
(26)
Thus,
-) - Uü - =
(27)
By Equation 23
can write
im(U)) (lim(U)) @ im(U)) n [im( e (m))
(im((1)) @ im(U)) n (U,) @ img)))
Thus, i (Um) img))]] C im (m;))))) (0), so
im(ūr) n (im(U)) @ im(U5)) = {0)
(28)
implies that a and b d = 0. Thus x Ú E im(ū;), so (im(Ui) @ im (U2)) n (im(U)) (Ū, @ im(U7)) C im(ū?).
Applying Equation 28 to Equation
27
Then Equations 19 and 20 combined with Lemma 2 implies that im(U;) C im(U)) but
dim(im(Uf)) dim(im (U,)) so im(U;) im Lm())..
Since the initial assumptions about U, and Uf analogously hold for the corresponding matrices for tasks and 3, by the exact same argument we can show that
im((;) im, Vt [T]
(29)
Then by equation (6). im() 2 im (ū,ŪT - im (Ü,ÚJ U;U:T) iim(U;)) Thus,
im (ū,Ūō U.u C im(Uf) n (mm)))
(0).
17

---

Thus Ú U Then by Equation [6 Ü,ū, U for all t E [T]. Lastly, since
L* (A. = 0, we have that V = 0, so
1
A A
E T
U,U; U,U, = A
B.5 Proof of Theorem 4
Proof. Clearly if L* (A, 0, then (A, Ú) is SOSP. The reverse direction is the challenging part of the proof. We equivalently prove that if (A,Ú) is critical point and L (A, Ú) # 0. then L² U) has negative eigenvalue.
Assume for the sake of contradiction that (Á, Ú) critical point and L* Ú) # 0. Then,
T
(AC T(Á - A") + Σ ÜUU U,U;")
(30)
(A,)" = (A A + 0,0, - U
(31)
Thus,
T
Σ (0,0) T
- -,U;T).
(32)
T
Define -.( U.U,T - +Σ= - -UU;7) Despite being a slight abuse of notation, we refer to (Ú) as just B, for the remainder of the proof.
Then 31) cquivalently states:
B,Ú, =0
(33)
Note that by construction, ΣT B, 0.
Considering C function of the flattened vector (ec) and let U
[z] 1x U2 = [y yxh, we compute the Hessian
V2C VU,
VU,
VAC
VAL
-ー==
VAC)
V 0
U L
O 22,4
(34)
(VU,
VAC
18

---

where
V 21
VU V VAL = (01 Ox1) ... (ox 2 x+ E Rd xdk
VUS VAC = [19 1 91) ... (yx 1 yv)l R xdk
VE, L' 2(A + UU - A' U I
ক + 20122
x x2 + x20 12Ø |
x xz I + x
22 x 21
x x + 25
+2
0 x1 I + 1
xa + ||
T2 -* 2(A
2 A'
U 1) )@ I
yi 99 11
yi y2 + y2v1
yi + y:
3/2 3 1 + 9192
y232 + 1
y yv 1 + yx.
+
y4 y1 1 + y19k
y + | 1 [
Note that @ denotes the Kronecker sum defined as X + Y = I 1 X + Y Ø I where @ is the Kronecker product.
Lemma 3. (") O and only B: 0 for each & [7].
Since (Á,Ū) is critical point, then plugging Equation 32) that
Proof.
into the definition of & gives
1 T
(AAAú))= = Σ
B
Thus L*,A, if and only B,
Vt.
Lemma 4. If (A, > O, then the eigenvectors corresponding to the non-zero eigenvalues Ú, are the leading non non-negative eigenvectors of A + +UūU, for all t e [T].
Proof. Consider the function J.(U,; A) - A' - A U.U, F. Je is simply the
tth summand in L where A Á is fixed and we only consider the variable U.. Minimizing fo is identical the problem of symmetric matrix factorization.
Using well-known properties of symmetric matrix factorization, since (. = O. we must have that U. V.T where the columns of V, are the properly scaled eigenvectors of A + U,U;T Á with non-negative eigenvalues where each column has norm equal the square root of its corresponding eigenvalue, and r E O₂ is some orthogonal matrix. Further, if the eigenvectors
corresponding to the non-zero eigenvalues of Ú,ú, are not the leading non-negative eigenvectors, then 2 (Ú)) * 0 by (36). Since ((,,) is a diagonal block of V (Á, V² f( y would imply (AL * 0.
Remark 3. Without loss of generality, we can assume that the eigenvectors corresponding to the non-zero eigenvalues of Ü,Ü, are the leading non-negative eigenvectors of A + U.U? -A for all i.
Lemma 5. (Ü,ÚJ -Ū,Ū) x (U:U3T - -UiUT x for all im((ū) im((U)
19

---

Proof. Recall B1 (Ü,Ü UTU - 02ú9 + +U;U:")
Then applying first-order sta-
tionarity and the fact that B2 = B1, we have
U,2ūg -ūÚū) U₁ U1= U U.UIT) Ún
(U2ú. - -,ūū,) U2 U2= (UZUZT SSU;T-U;UY -Tuu U2. )
Corollary 5. Ü,ÚI Ü, and - - share an eigenbasis.
Proof. Using the lemma, any non-zero eigenvector-eigenvalue pair of U2Ú2 - Ü, is also an cigenvector-eigenvalue pair of U: U Denote the space defined by the span of these eigenvectors as S. Then other eigenvectors of U - -UTU are orthogonal to S, so they are also O-eigenvectors of ÜZÜ, Ü,Ü,T Thus the two matrices share an eigenbasis.
Corollary 6. dim (imUs + im imúú) - 2k 1. i.e., the set of columns Ún and U2 are not linearly independent.
Proof. Assume for contradiction that the vectors the set S i ,l,,,,,ih) k are linearly independent, where e, is the ith standard basis vector in IR*
= (Ūie; i U (Ū2e,
Then note that (Ü,ÚT -Zúū + 0 and (U;U,T U,U;") + 0 for all S. By
Lemma 5 Ü,0T Ü2ú and U - - agree for each vector on the 21-dimensional space span( (). But, both rank(U,ū, -Ú,ū, rank(UU;U -U3UZ S 2k by construction. Then by dimension counting, U,Ú, - Ü2Ü, and UTU UN must send span (S)- to O.
Thus, Ü,Ú," - -Uú and UTU UUŞ T agree the entire basis formed by concatenating
basis vectors of span(S) with those of span(S),, (S). This implies that Ü₁Ú, U2 U
U and thus B = Ü,ÚT Ü²Ú, U + U 0. Then B = B = 0 so by
Lemma
L' Ú) = 0 which is a contradiction.
Lemma 6. U
U T has exactly k positive and k negative eigenvalues.
Proof. First, note that U has exactly k positive eigenvalues and eigenvalues of Then UgU;" (Ufe))(Ufei) has rank - because of the linear independence of the columns of the combined set of columns U and Už. Further, since we subtract (Uie) e, (U){) we must be accumulating an additional negative eigenvalue relative to UüU²? Continuing this process (Ufe)) contributes
shows that subtracting exactly one more negative eigenvalue, since Uf e1 can never be written linear combination of e eke U,,
from UNU;" U -
Ujeú} for 0 < j < k. The result then follows from induction.
Lemma 7, rank(Üi) = rank (Uz) =
Proof. Assume for contradiction that rank(Ü;) = < & without loss of generality. Since by
Remark (3) assume the columns of Un are the leading k non-negative eigenvectors of A
U - Å = Ú,Ú, -B,, this must imply that A + +UUU Á --U = B1 x 0.
=-B
Plugging in the definition of B, gives that & (0,0T U.U. - Ú,OJ + U:U > 0. Thus,
Ú,ÚT > U + ~,Ü U > U.U.T U:U;-. This contradicts the fact from Lemma (6) that U UNU has i positive eigenvalues.
With this lemma, we will prove the existence of direction of 2 with negative curvature. Instead of directly working with this matrix, we instead use the Schur complement to work with a different form.
20

---

Theorem 5. (Schur Complement) Since - > if and only if
( - (VAVU((AA (V24 ( (
(VoVAC(AAU) =0.
Define M = Vi -VA(IAA))(AAVGAAAA) ( (VACIAA)
(VoVaL
For example, when k: 2 and letting U1 (0) 02). U2 ly1 y2l. have
Mu M12
M
M
M2
where
2B + x + ||0
0j 22 + x
M
32 2 I + x2
2B + x2202 + a22 2
- x 11 y
x 12 y2
M
-02 y)
yn 2
2. y2 y2
2BB + vi9i + 11 y/2 y2 + y1
y1 y2 + y2y1
M22 =
2B, + y2y2 + 11
For brevity, we do not include the full form of M for general k. However, we can make an easy simplification that will allow for much cleaner expression.
Using Corollaries (5) and (6), there is an eigenvector z of UNU - U with eigenvalue
* 0 such that z E kker (Ú2Ü - Ú, Assume without loss of generality that À > 0. and consider 122k Define the function g(- ;=)= 2 R parameterized by such that g(a; =) (a 2) M (ar 2), where we partition a 02), on a2 R* Then after some algebra,
g (o:; 2) = 0,00 + U2a0 || + (1a11 [|022
(35)
We prove the existence of E 22 x Rd such that g (a:; x) considering two different cases. Define N S2 + E as the function that returns the number of strictly negative eigenvalues of its input.
Case 1: N (Ü₂Úā -Ūū < k.
Using Corollary (6). can pick such that Ú + U2a2 an 0.
Because N (Ú,Ú. - ú,ú,)) k. N (U;U;" T UFU = k. and Ú₂ú,J - Ú, and USU;T U share an eigenbasis by Corollary there exists = E nd that is À -eigenvector of UG -UU < O, where & E ker Üū - Ú₁ú,)
Then for the same choice of a,
sign (g (or; z)) sign [|ال||| - ||
sign (g (a: x sign ||02112 2 - ||<||| 11
Then if ||a1ll2 # || one of the above expressions is negative and thus M has a negative cigenvalue. This then implies 2 Ú) * 0.
21

---

Otherwise || 2| |122112-- Then g (a: z) 0, but Va,g(a;)) U, (Uia)+ + U2a2)
2Aoz -2xx2 # O. Thus g(a; 2) 0 and Vg((x;;) so there exists in infinitesimal
neighborhood around a where g(o: z) < 0. Thus M has negative eigenvalue so V (A, Ú) x 0.
Case 2: N (ÜzÚN- - - k.
Define m = dim (im((i) n im(( imm(ū))) By Corollary 6. m 2 L, so we can select orthogonal matrix
T On such that Üzrer E (im(()) nim(())). Define y Ü2rer.
Clearly for any B E Sa and R E S N (B) 2 N (B + R). Then since N (-U;ūT) k by Lemma
7 we have that
N (-Ū;Ú N - = N- ((Urre)) (UsTee))UsSei))) - -U,Ü0
-Ū,Ü;)=
2 NN ((sr)) (UAT) - -000)== = N (Ü(ūūū - -,ūū)) = k,
Thus, Nu0'"0 - - k. But, since y E im im(U1), rank (uuT - -Ū,ÜT) k. Thus,
yy - -Ú $0.
(36)
Take such that Ú, = y and o2 = ree. Then
uirn U,ú, = (Uja) (tia)² T - -
(37)
Ú, (ajaī T U] 0.
(38)
Therefore ||aı 1l2 1.
Then g (a: z) = ||oai++ + 1 + (1a1112 [|02))) = => -11a112-11 s O.
g (o;; 2) < 0, then we are done. Otherwise, g (a; z) O. Then the same analysis from Case I will show that Vg(or; z) o 0, so there exists à in an infinitesimal neighborhood around a where g(ā; 2)) strictly negative. This then implies our desired result.
B.6 Derivation of Equation
Recall our generative model where each input sample x E IRd satisfies E [ae] = 0 and E [x 1 = of each noise sample is generated independently of æ as e NN Ia), and y (A* R x + e. Then,
C (A)] = E
y Az 1
= E
(A' + R A)x + c12
= E [(AA + R - - + || + 2E (A' + R₁ - A
E [tr (x (A* - -R₁ A.) (A + R At)x)] +o²
+ [c] (A' + R - A [x]
= E [tr ((A + R - A,) (A' + R - A xZ }]] + +02
tr {(A" + R A.) (A' TAA +R - A.) [ }} +
03 ((A' + R T(A *RRR; -R:-A -a)) +
11' + R - A.l - +0²
22

---

Thus, (A.)) 1 (020021 0f + R of expectation, so
Aclle + oZ) Then E (A)) [[C)
by linearity
1 (A,)
- | A' + U,U; -
C Limitations of Standard Retraining for Asymmetric Adapters
In this section we show that even if the ground truth rank-k adapters Ri are asymmetric in our theoretical model defined Section 3 standard retraining still fails to recover a retraining solution ASR that guarantees that either rank ÅSR A')
ASR
A
is small.
Consider task task-specific adapters Rj E BRdx drawn independently from general distribution DR such that rank( R2; = k and DR is absolutely continuous with respect to the Lebesgue measure. Then since ASR A Σi= R,, immediately have that ASa and A are far apart in terms of rank:
rank (ASR A*) min (ni kT) w.p.
As result, the necessary adaptation rank during fine-tuning to realize the test task parameters A' + Ri+1 is min(d, k[T + 1) just as in Proposition
Additionally, we show that we can never guarantee that ASR A+ small for general adapter distributions DR. Let mean of the adapter distribution be MR = E [Ri]. Since the error between ASR and A is the average of retraining adapters + Σ Rí then by the law of large numbers must have that the number of retraining tasks T xo, then ASR A" MR- Thus, can only guarantee that standard retraining will approach reasonable solution in the special and impractical case where both (i) the number of tasks approaches infinity, and (ii) the mean of the adapter distribution M = O. If either of these conditions do not hold, then standard retraining will provably fail to recover A"
We lastly note that even though we model each task by the additive parameterization A + R,,, we cannot absorb the mean MR into A" to reduce to the case where MR O without destroying the low-rank structure of R Specifically, given a problem instance defined by tasks T. parameterized by A R where MR E [R we could equivalently consider the tasks parameterized by (A' + MR) + [R MR))- Then the transformed adapters R MR are zero-mean, but rank R MR) may be as large as the ambient dimension d, eliminating the low-rank structure needed for the computational and samplc-efficiency gains of low-rank adaptation during test-time fine-tuning.
D Synthetic Experiments
D.1 Linear Model
We first test our meta-learning framework on synthetic regression tasks. We consider data yr.j
(A* + R2 wr, + Et.j for task t and j [me]. where we generate Tt,js Et,j, and R2 as in Section with æy. ~N I). WN .01 x I), and R = UUU," with each element of U, sampled as (O, 1) random variable. We similarly generate the entries of A* as N(, 1) random variables. We set the number of samples per retraining task for each task as nh N, so each retraining task equipped with N samples. Further, we denote the number of test task samples 1 n. We set L to be the mean squared error loss, and we run gradient descent on the standard retraining (I) and the LORA-ML (5) objectives. For the LoRA-ML objective, we alternate between the outer step, updating the shared parameter A, and the inner steps, where we update the task specific parameter U, for each task independently. We run 3000 outer steps, where after each outer step we run 10 inner steps. We use a learning rate of 3 x 10-2 for the outer steps and 3 x 10-3 for the inner steps. For the standard retraining objective, we simply run gradient descent using the learning rate 10-2 over 3000 epochs.
23

---

After recovering A during retraining, we apply the low-rank adaptation QVT when fine-tuning to the task. When T = we use rank-3k adaptation during fine-tuning account for the inexact recovery explained in Theorem 2 and otherwise use rank-k adaptation. To perform the adaptation, we run gradient descent on the LoRA objective
(9) using a learning rate of 10-3
In each experiment we vary one hyperparameter from a fixed set of values and plot the prediction error between the recovered model and the ground truth model 1 Σ, |/ (A Results are averaged over 5 trials, with the shaded region showing the full range of values across trials. For each experiment, we fine-tuned for different number of epochs, as some required more epochs to converge. The performance by epoch is displayed on the x-axis of each plot.
"VV+)r+, ll
104
T=2, LORA-ML
T-3, LORA-ML
T=5, LORA-ML
T-2,
Test Task Loss
T=3,
102
T=5,
101
109
10-1
10-2
10-3
2000
4000
6000
8000
10000
Iteration
Figure 2: Linear model fine-tuning performance varying the number of retraining tasks T, This is an enlargement the left subfigure of Figure
n=100, LORA-ML
106
n=1000, LORA-ML
n=10000, LORA-ML
n=100, SR
Test Task Loss
10
n=1000, SR
n=10000, SR
102
10-2
200
400
600
800
1000
Iteration
Figure 3: Linear model fine-tuning performance varying the number of samples for the test task n. This is an enlargement of the right subfigure of Figure
24

---

N=100, LoRA-ML
10
N=1000, LORA-ML
N=10000, LoRA-ML
N=100,
Test Task Loss
102
N=1000, SR
N-10000,
10a
100
10-1
10-2
10-3
200
400
600
800
1000
Iteration
Figure 4: Linear model fine-tuning performance varying the number samples per retraining task N.
k=1, LORA-ML
10
k=2, LoRA-ML
k=3, LoRA-ML
103
k=1, SR
Test Task Loss
k=2, SR
102
k=3, SR
101
100
10-1
10-2
10-3
200
400
600
800
1000
Iteration
Figure 5: Linear model fine-tuning performance varying the ground truth adaptation rank k.
25

---

10
d=5, LORA-ML
d=10, LORA-ML
104
d=20, LORA-ML
d-5,
10a
Test Task Loss
d-10, SR
102
d-20,
101
10
10-2
10-3
400
600
800
1000
1200
1400
Iteration
Figure 6: Linear model fine-tuning performance varying the ambient dimension d.
D.2 Shallow Network
synthetic data experiment using data generated using shallow network architecture. We first generate inputs æt E IRd distributed as (0, Ia). We then generate ground truth parameters A E Rdxd c² E Rd and U, E dd for each task E [T] by sampling the entries as (0, 1) random variables. We denote R = U,U; Lastly, the outputs Ve, E for task é are generated as jt.j c² s((A" + Rt )=t,)) et.. where s(-) is the element-wise sigmoid function and etj
We additionally
noise. We again define parameters N, such that n, N for all t < T and f1+1 = n. Setting L to be the mean squared error loss, we apply the AdamW optimizer with learning rate of 1 x 10-3 both the standard retraining (T] and the LoRA-ML (5) objectives. After recovering Á and è during retraining, we apply the low-rank adaptation QVT only to A when fine-tuning to the test task. In each experiment we vary single hyperparameter from fixed set of values and plot the prediction error between the recovered model and the ground truth model
(O0,001
1 - @ s((Á(( + QV )TT+1s)) - e¹ s((A+ 11 - To mitigate the effects of outliers, we plot the median over 10 trials, where the tables that accompany each figure report the range of the central 6 values of the last epoch prediction error, clipping the two best and worst trials.
Figures 7 8 10 and again show that retraining using the LoRA-ML objective leads to much better fine-tuning performance relative to standard retraining. Figure shows that the LoRA-ML objective effectively exploits the number of tasks, since fine-tuning performance improves as T increases. Further, we observe in Figure that fine-tuning with any number of samples after standard retraining cannot even recover the performance of fine-tuning with just 100 samples after LoRA-MI. retraining. We note that the range of the results varied much more in the shallow network setting relative to the linear model experiments. Although LoRA-ML performed much better than standard retraining when looking at the median result, few individual trials showed outlier results.
The data parameters for both synthetic experiments are summarized in Table Both were performed using a single NVIDIA A40 GPU.
26

---

T LORA-ML LORA-ML T LORA-ML
T-2, SR
Test Task Loss
T
5.
10
8000
12000
Iteration
Figure 7:
Shallow network fine-tuning performance while varying number retraining tasks
Table 4: Median and central range of the test task loss after fine-tuning (smaller is better) at the last epoch for the shallow network experiment across 10 trials while varying T.
Method
T= 2
T =
T= 5
LoRA-MI
0.047 (0.042, 0.127)
0.018 (0.017, 0.028)
0.005 (0.004, 0.024) 0.076 (0.046, 0.107)
SR
0.119 (0.104, 0.161)
0.092 (0.086, 0.225)
+ == 100, LORA-ML
n 1000, LORA-ML LORA-ML n- 100, SA
Test Task Loss
n 1000, SR
10-9
5000
12000
Iteration
Figure 8: Shallow network fine-tuning performance while varying number of test task samples n.
Table Median and central range of the test task loss after fine-tuning (smaller is better) the last epoch for the shallow network experiment across 10 trials while varying n.
Method
100
1000
10000
LoRA-ML
0,006 (0.005, 0.034)
0.003 (0.002, 0.061)
0.004 (0.004, 0.017) 0.115 (0.112, 0.165)
SR
0.132 (0.114, 0.171) 0.147 (0.109, 0.221)
27

---

100, LORA-ML
1000, LORA-ML
N 10000, LORA-ML
N 100, SR
Test Task Loss
10P
1000,
N 10000, SR
10-1
10-9
8000
12000
Iteration
Figure 9:
Shallow network linettuning performance while varying number of samples per retraining
task N.
Table Median and central range of the test task loss after fine-tuning (smaller is better) at the last epoch for the shallow network experiment across 10 trials while varying N.
Method
N 100
1000
10000
LoRA-ML
0.138 100, 0.262)
0.010 (0.006, 0.055)
0.023 (0. (0.009, 0.048) 0.085 (0.074, 0.108)
SR
0.354 (0.319, 0.646)
0.075 (0.073, 0.092)
LoRA-ML
<- LORA-ML
< LORA-ML 1, 3, SR
Test Task Loss
< 5, SR
10-1
4600
8000
12000
Iteration
Figure 10: Shallow network fine-tuning performance while varying ground truth adaptation rank k.
Table 7: Median and central range of the test task loss after fine-tuning (smaller is better) at the last epoch for the shallow network experiment across 10 trials while varying A.
Method
k = 1
k 3
k = 5
LoRA-ML
0.138 10.100, 0.262)
0.093 (0.077, 0.117)
0.044 (0.043, 0.067) 0.151 (0.140, 0.299)
SR
0.354 (0.319, 0.646)
0.283 (0.261, 0.359)
28

---

d 5, LoRA-ML
d LoRA-ML
10P
d 20, LoRA-ML
d SR
Test Task Loss
d 10, SR
d 20, SR
10-1
10-2
10-3
4000
8000
12000
Iteration
Figure 11: Shallow network fine-tumin fine-tuning performance while varying ambient dimension d.
Table 8: Median and central range of the test task loss after fine-tuning (smaller is better) at the last epoch for the shallow network experiment across 10 trials while varying d.
Method
d=
d 10
d 20
LORA-MI.
0.002 (0.001, 0.003)
0.005 (0.004, 0.010) 0.082 (0.082, 0. 0.125)
0.069 (0.050, 0.121) 0.483 (0.438, 0.690)
SR
0.004 (0.003, 0.021)
Table 9: Synthetic Data Parameters
Experiment
T
11
N
1
C,
Linear, varying 2"
(2.3,5)
100
5000
10
Linear, varying n
3
100.1000,100000
5000
10
1
Linear, varying
3
100
1100, 1000]
10
Linear, varying &
3
100
1000
(12,3)
10
Linear, varying d
3
100
5000
5,10020
Shallow Network, varying
(23355)
100
1000
10
Shallow Network, varying n
3
[ [100,1000,1000
1000
10
Shallow Network, varying N
3
100
[100, 10000]
10
Shallow Network, varying
3
100
1000
(1335]
10
Shallow Network, varying d
3
100
1000
(5,10,20)
E Real Data Experiments
In this section, we describe the real data experiments in complete detail. All of the real data experiments were performed on a single NVIDIA H200 GPU.
E.1 Vision Experiments
We use CIFAR- 10 21 and define T = 4 binary classification retraining tasks. Each task requires classification between consecutive CIFAR-10 10 class labels. Specifically, task 1 classifies between classes 1 and 2, task 2 between classes 3 and 4, etc. The test task is binary classification between classes and 10. We evaluate our PEFT-ML framework using model based on the MLP-Mixer architecture 35 We use depth of 1, and an embedding dimension of 512 for the patches. We compare two different adaptation methods: (i) LoRA with rank adapters, and (ii) fine-tuning only the last layer. For both PEFT strategies, we evaluate three retraining strategies: (a) PEFT-ML for
29

---

Table 10: Mean test accuracies and standard errors for PEFT-ML and standard retraining methods using LoRA-basee and last-layer- based fine-tuning for adapting to subset of CIFAR- 10 classes.
LoRA Fine-Tuning
Last-Layer Fine-Tuming
Method
Mean
Std. Err.
Method
Mean
Std. Err.
LoRA-ML
86.09
0.35
Last-Layer-ML
83,90
0.17
SR+LoRA
85.29
0.17
SR+Last-Layer
81.16
0.22
Reptile+LoRA
81,30
0,42
Reptile+Last-Lan
72.55
0.39
each PEFT method (LoRA-ML, Las--Layer-ML.. (b) standard retraining (SR), and (c) Reptile, popular gradient-basee meta-learning method. We report mean results along with standard errors across 5 trials in Table 10
For each retraining method, we take the model from the epoch with the best average performance on the validation samples for the retraining tasks then be fine-tuned. We also apply basic trans- formations of the retraining data like random cropping and flipping to reduce overfitting. During fine-tuning, we take the best accuracy on the heldout samples for the test task across epochs for each We summarize the hyperparameter
trial, and we report the average of these values in the Table 10 choices in Tables [ and 12
Table 11: Retraining hyperparameters for vision experiments. Standard Retraining
Hyperparameter
LoRA-ML
Last-Layer-ME
Reptile
Learning Rate
10
10
10-1
10-2
Outer Learning Rate
N/A
N/A
N/A
5
Weight Decay
10
10
10
10-5
Epochs
100
100
100
5
Inner Epochs
N/A
N/A
N/A
20
Optimizer
Adam
Adam
Adam
Adam
Batch Size
256
256
256
256
LoRA Rank
N/A
N/A
N/A
Table 12: Fine- Tuning hyperparameters for vision experiments
plpeepaarametrr
LoRA Fine- Tuning Last-Layer Fine- Tunn-Tinng
Learning Rate
5 10-4
5 10-1
Weight Decay
10
10-
Epochs
100
100
Optimizer
Adam
Adam
LoRA Rank
NA
30

---

Table 13: Mean accuracies + standard error for 10 test tasks using different retraining and fine-tuning method combinations on ConvA12.
Algorithm
T1
T2
13
T4
155
T6
17
T8
T9
T10
Avg
LoRA--MI
57422
41:24
512-
50:14
51:24
30:22
66:3
47::4
43
38:22
47.4
SR-LoRA
59 4.5
31:4
50:7
40:24
24:25
20:12
41::10
36:22
23:55
26:66
35.02:4.1
RepilleLLRAA
28.4
Lay--Lyee--ML.
51
44
Mat 4
25 4
58
4
33
40.2 +:
SR-Last-Layer
41
9::.5
29 1
15
31
24
15
24.5 4.2
LaptiletLas-LLaee
35 1:6
13 13:44
18
29 ::
19
18 1:4
15
17:
1
16
19.4 4.43
Table 14: Mean accuracies (averaged over all tasks) = standard error for different retraining methods varying LoRA ranks during fine-tuning.
Algorithm
Rank 1
Rank 4
Rank
Rank 16
LoRA-ML
44.7 + 0.8
48.6 + 1.6
47.4 1,9
48.2 + 1.5
SR+LORA
36.3 1: 4.1
35.5 + 4.3
35.0 +: 4.1
37.1 +: 4.1
Reptilc+LoRA
26.6 + 5.8
27.7 5.3
28.4 1.445. 5.0
27.8 + 5.9
E.2 Language Experiments
We use the ConvA12 23 dataset for the language tasks. ConvA12 comprises conversations between two personas, where each persona associated with list of facts that informs their responses. We model learning the dialogue continuations each persona as a different task. For each continuation are given 20 candidate continuations, with one option being the correct continuation.
We retrain starting from RoBERTa-base model 22 along with the RoBERTa tokenizer. We add sequence classification head and use cross-entropy loss to predict whether given continuation is in fact the correct response. We label the incorrect continuations as class 0 and the correct continuation class for training, and to perform inference probability of being from class 1. We retrain on the ten tasks (personas) with the most data, and fine-tune the next ten tasks (personas) with the most data. We perform 5 runs for each experiment (both retraining and fine-tuning) and report the means and the standard errors of the accuracies the fine-tuned model the validation sets of the fine-tuning tasks. During retraining, from the epoch which demonstrates the best average performance on the validation samples for the training tasks. We then fine-tune this retrained model and report the test task accuracies from the last fine-tuning epoch.
select the continuation with the highest predicted
save the model
The hyperparameters used are listed below in Tables 15 and 16
Table 15: Retraining Hyperparameters for language experiments
Hyperparameter
Standard Retraining LoRA-ML Last-Layer-Mi
Reptile
Learning Rate
5 10-5
10-5
10-5
5. 10-5
Learning Rate Schedule
Linear
Linear
Linear
Linear
Batch Size
8
8
8
Epochs
10
10
10
Inner Epochs
N/A
N/A
N/A
20
Optimizer
AdamW
AdamW W
Adam W
Adam W
LoRA Rank
N/A
8
N/A
N/A
LoRA Dropout
N/A
0.1
N/A
N/A
LoRA Alpha
N/A
16
N/A
N/A
Outer Learning Rate
N/A
N/A
N/A
10
E.3
Asset Information
We use the CIFAR-10 10 21 and ConvA12 23 datasets our experiments. CIFAR-10 is publicly
available but does not specify explicit license. ConvA12 is distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. We also use the MI.--Mxxe (35 and RoBERTa- Base [22] model architectures, including the pretrained weights for RoBERTa. The official MLP.
31

---

Table 16: Fine- Tuning Hyperparameters for language experiments
Hyperparameter
LoRA Fine- Tuming
Last-Layer Fine-tuning
Learning Rate
10
10
Learning Rate Schedule
Linear
Linear
Batch Size
8
8
Epochs
10
10
Optimizer
AdamW
AdamW
LoRA Rank
8
N/A
LoRA Dropout
.1
N/A
LoRA Alpha
16
N/A
Mixer implementation is licensed under the Apache License 2.0, while RoBERTa-Base, as accessed via Hugging Face, is licensed under the MIT License.
Theory Notes
F.1
Non-Uniqueness Global Min for 2
Consider T = 2, k = 1, = 2, A' O, and z for t = 1,2, where is the tth standard basis vector. Clearly the ground truth perturbations u, are orthonormal and thus linearly independent. The set of global minima of L are (A,U) such that A + Σ (u T -uuu and u.u
" u -
7 -. u u 0. It not hard to see that a global minimum follows from
When properly parameterized. this system of equations defines hyperbola where each point corresponds to global minimum of L
any set values of u U2 such that u I - u = -1
F.2 Spurious Local Minima
We observe that for T 2 for certain tasks U UU, Už), is possible find points U that are local minima, but not global minima. To find these points, we sample true tasks U from normal distribution and use a numerical solver to find zeros of the gradient of the reduced loss
- 1 EV.U -UU:-U;;;] -..U--U;U:J] ((U) - Σ D. - U,u -
Through the Schur complement argument used to prove Theorem local minimum only L has a spurious local minimum.
see that L has spurious
Typically, these zeros are close to the global minimum. Occasionally, it possible to find point Ú with gradients close to 0 and with positive definite Hessians. We then confirm that these are close to the spurious local minimum through the following argument.
Consider the function
r(U) vec(U ( Ú) veo (V()))
Clearly, there is a minimum of È in the 5-ball of if r(U): > 0 for all U on the boundary of the 8-ball. As continuous, if for some small enough e, > 0f rU)) y for all U on the e-net of the boundary of the 8-ball, then there exists spurious local minimum the 8-ball around Ú. Numerically, such points and 8, and
be found which would imply that spurious local minima exist, barring any errors due to numerical computation. To confirm, we run gradient descent from this point and observe that the loss stays constant in Figure [2
32

---

2064
0.000
Figure 12: Loss does not decrease near these spurious local minima
G Example Pseudocode for Minimizing 5
Algorithm Meta-Adapter Training
1: Input: Tasks Tú, t E [T). learning rate n, number of epochs Ne, batches per epoch No
2: Initialize: Model parameters Wo, ((t) for all I....,
3: for epoch e = to N₂ do
4: for b ...... N₁, do T I..... do 6: Load next batch 3rD from T
5: for t
7:
Compute gradient g (*)
Vw w. 00
(Eeen)E [((BFr (x: W. 0(0)).)))11 v)
8:
Update adapter parameters: 0
9: end for
10:
Update base parameters: W We+
W. 7) Σi
9w
(1)
11: end for
end for
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims focus on theoretical guarantees for meta-leaming infused with LoRA on linear models. We explicitly state our contributions, as well as their scope relating to LoRA adaptations on linear models, both the abstract and the introduction. Further, our claims that PEFT-based meta-learning during retraining improves performance over standard retraining supported through our experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: For all results, we state the scope in which they apply. Our theory applies to linear models using LoRA fine-tuning, and each theoretical statement only applies within our stated assumptions. In the conclusion, we mention future work which includes extending our analysis more complex model classes and different kinds of PEFT methods.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
34

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For all of theoretical statements, we state the assumptions along with the result statements. The proofs are in Appendix
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all training details in the main body and in Appendices and We further release all code in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility.
35

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We release all code, and we the publicly available datasets CIFAR-10 and A1nvA12.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ mublic/guidess owt.:/oeuuumiiiiiioooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. htte/uuliccggguuiie Ctteee//doSSSSebiseeeeee ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We explain the experimental methodologics in the paper body along with key hyperparameter choices, like the LoRA rank. The full experimental details are in Appendices
D and B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Each set of results accompanied with measure of uncertainty. For the synthetic data experiments, we plot average results along with a shaded region showing the full range of values for each setting. For the real data experiments, we report mean results along with standard errors across trials.
36

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: These are reported Appendices and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttge/e//eerrge..c///llll/teeie
Answer: [Yes]
Justification:
This work conforms to each criterion.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
37

---

This work focuses on the theoretical understanding of model preparation for adaptation using parameter-effiee fine-tuning methods such LoRA. It does not introduce new models, datasets, or applications with direct societal implications.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any new data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Each dataset and model architecture we use is properly cited in the main paper body, and the license information is included in the appendices.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
38

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com/ comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release our code which includes documentation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use experiments of this kind.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do experiments with human subjects.
Guidelines:
39

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did use LLMs for any original or core component of this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hutpss//maarrpss.cc//oofeeeeeeee for what should or should not be described.
40


Reviews:
[{"review_id": "ZMlytD93b0", "reviewer": "Reviewer_bDJi", "summary": "This paper proposes a new meta-learning framework tailored for parameter-efficient fine-tuning. Even though they claim it can work with any PEFT method, the main focus is on LoRA. By providing theoretical results in a linear regression setting, the author shows that standard retraining cannot produce representations that are easily adaptable to new tasks. Their proposed framework on the other hand guarantees low-rank adaptability and lower prediction error. Experiments on synthetic data and small-scaled vision and language tasks support the theoretical claims.", "questions": "Suggestions:\n\n-\tThis paper is not easy to read for practical and general ML audiences. A short paragraph bridging the theory and experiments and an overview figure of the pipeline can improve it.\n-\tAdd computational complexity analysis\n-\tShowing large-scale benchmark experiments", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The theoretical contribution is novel and strong. Showing that standard retraining is suboptimal and proposing a framework with provably better adaptation.\n-\tThe problem formulation is well-defined with clear assumptions.\n-\tThe theoretical analysis of the effect of the number of tasks on the proposed model is another strength of this paper. Especially unique recovery of the ground truth for three or more tasks.\n-\tFigure 1 comprehensively supports the theoretical results in the controlled linear setting.\n\nMajor issues:\n\n-\tThe theoretical results are driven under linear regression setting with Gaussian noise. However, classification tasks on CIFAR-10 using an MLP-Mixer and language modeling tasks with RoBERTa involve nonlinear architectures and classification losses. The theoretical guarantees do not extend to these more complex settings. \n-\tThe proposed framework requires solving a nested optimization problem. For each task, adapter parameters must be learned in the inner loop. This introduces computational overhead compared to standard retraining. However, the paper does not provide any analysis or empirical comparison of the computational cost, memory usage, or convergence time between the proposed method and the baselines. \n-\tSome results (for example, Table 1 - left) show that the performance of standard retraining is very close to the proposed method. Even in some cases better, if we consider the standard deviations. The empirical result contradicts the strong theoretical claims which predict that standard retraining should become significantly worse as the number of tasks increases. This suggests that the theoretical benefits may not transfer effectively to nonlinear and classification settings and authors should discuss it.\n-\tMore ablation studies can add great value to the paper, such as more diverse parameter-efficient fine-tuning methods or empirical results on real-world large-scale benchmarks.\n\nMinor issues:\n\n-\tLine 282, “task task”", "was_revised": true, "final_justification": "The paper presents a novel idea supported by a solid theoretical foundation. The authors have addressed my concerns in their responses and plan to provide a revised version that includes additional results and discussion.", "rebuttal": "Dear Reviewer bDJi, thank you for highlighting the novelty and strength of our theoretical contribution, as well as the clarity of our formulation. We answer your remaining questions below.\n\n**(Q1) The theoretical results are derived under a linear regression setting with Gaussian noise...The theoretical guarantees do not extend to these more complex settings.**\n\n(A1): Thank you for raising this important point regarding the applicability of our theoretical guarantees to more complex model classes and a broader range of loss functions. Our goal is to develop a theoretically principled framework for preparing a model for efficient test-time adaptation using PEFT, and as you correctly note, our theoretical analysis focuses on model adaptation in the setting of linear models with squared $\\ell_2$ error. This setting enabled strong theoretical guarantees, yet still gave rise to several counterintuitive phenomena and posed non-trivial technical challenges (please see our response (A4) to Reviewer qeBR for further discussion of these challenges). We agree that extending this framework to encompass more expressive architectures and diverse objectives, particularly those beyond regression, is a valuable direction for future work, especially as our empirical results demonstrate that the principles extend effectively to non-linear models and classification tasks.\n\n**(Q2) The proposed framework requires solving a nested optimization problem...the paper does not provide any analysis or empirical comparison of the computational cost, memory usage, or convergence time between the proposed method and the baselines.**\n\n(A2) Thank you for the thoughtful question regarding the computational overhead, memory usage, and convergence time of our method compared to standard retraining (SR).\n\nWe first address the memory cost. We discuss the memory overhead of LoRA-ML in Appendix D.3, which arises from the additional adapter parameters. If we assume a model has $m$ layers, each parameterized by a $d \\times d$ matrix, and uses rank-$k$ LoRA adapters, then SR uses $md^2$ parameters, while LoRA-ML retraining uses $m(d^2 + 2kdT)$ parameters (where $T$ is the number of tasks). Since $k(T+1) \\ll d$ in our setting, the total parameter count of LoRA-ML remains $O(md^2)$ during retraining, so LoRA-ML does not incur any additional asymptotic memory complexity relative to SR. At the fine-tuning stage, both LoRA-ML and SR models require the same number of trainable parameters since standard LoRA is applied.\n\nIn terms of the computational cost and nested optimization, we note that while our objective is naturally expressed in a nested form (Eqn. (5)), we emphasize that LoRA-ML differs from methods like MAML in a key way: the inner optimization is over lightweight adapter parameters, not full shared weights. This allows us to implement optimization methods for the LoRA-ML objective either via explicit inner/outer loops or through joint optimization over both the shared and task-specific parameters. In fact, we utilize this joint form in our theoretical setting as we start with the nested formulation in Eqn. (13) and instead analyze the equivalent joint formulation in Eqn. (14).\n\nAs you mention, the LoRA-ML objective can be empirically optimized using either an outer/inner loop strategy or simple joint minimization, each incurring different computational cost. In our experiments, we explored both strategies. In the linear experiments, we used a structured inner/outer loop: 10 inner steps on task-specific adapters followed by 1 outer step on shared parameters, for 3000 outer epochs. We compared this to SR trained for 3000 steps, since the SR objective is convex and converges quickly. In the larger vision and language experiments, we adopted a joint optimization strategy for LoRA-ML and used the same number of epochs as SR, without explicit inner steps, achieving stronger performance with no additional training loops. We agree that this is an important design choice to highlight and will update the revised version of the paper to reflect this. Further exploration of the tradeoffs between joint optimization and inner/outer-loop strategies is an interesting direction for future work, particularly for improving the efficiency of LoRA-ML at scale.\n\nIn terms of convergence time, while we do not provide a theoretical convergence rate comparison, our empirical results show that LoRA-ML achieves better performance within the same epoch budget as SR in large-scale settings. This suggests the convergence of LoRA-ML is competitive in practice.\n\nWe will incorporate the discussion of memory usage, computational cost of different optimization strategies, and convergence time into the revised paper.\n\n**(Q3) Some results (for example, Table 1 - left) show that the performance...This suggests that the theoretical benefits may not transfer effectively to nonlinear and classification settings and authors should discuss it.**\n\n(A3) Thank you for highlighting this point. As you mention, even though our proposed method demonstrates stronger results than standard retraining (SR) across tasks and experiments as a whole, for some specific tasks, the performance gap can vary, especially when the performance variance is taken into account. This behavior is a function of two key characteristics of each experiment: (i) the complexity of the test task relative to the expressivity of the adaptation mechanism, and (ii) the number of available samples per task.\n\nIn Table 1 (left), we test our framework through a vision task on CIFAR-10, where each task involves binary classification between two distinct classes. We employ an MLP-Mixer architecture and adapt to the test task after retraining using LoRA. We observe that the performance gain using LoRA-ML relative to SR is not drastic, and the performance variances of both methods are small. LoRA induces a fairly expressive adaptation mechanism for the relatively easier task of binary classification between CIFAR-10 classes, so it is able to recover from potentially suboptimal models recovered by SR during retraining. Further, since we have access to many samples per task, the performance variance of each method is small.\n\nIn contrast, in Table 1 (right), we consider the same vision experiment except we adapt using just the last layer fine-tuning, which is significantly less expressive than LoRA. As such, we see a much wider gap between the performance of LoRA-ML and SR while still achieving small performance variance due to the large number of samples in CIFAR-10.\n\nLastly, in Table 2, we present the performance of our framework through a language task on ConvAI2, a dataset of conversations between personas. Each task corresponds to predicting the dialogue continuation of a specific persona. Relative to CIFAR-10, this experiment is much more complex and is equipped with many fewer samples per task. This leads to a larger variance in per-task performance. However, when we compare the average performance when adapting to all 10 test tasks, we observe that LoRA-ML significantly outperforms SR, even with variances taken into account.\n\n**(Q4) More ablation studies can add great value to the paper...**\n\n(A4) Thank you for this suggestion. While we did not have time to incorporate additional PEFT methods or larger datasets during the rebuttal period, we include an ablation for the LoRA-ML method demonstrating how the choice of the adapter rank $k$ during retraining affects performance, as this is the sole additional hyperparameter of LoRA-ML relative to the standard retraining procedure. We test using our language modeling experiment, where we aim to learn the responses of different personas in the ConvAI2 dataset. In Table 2 of the paper, we show results using rank-8 adaptations for LoRA-ML along with other methods. Below, we show the effect of different ranks and include standard retraining followed by rank-$k$ LoRA (SR+LoRA) as a baseline.\n\n**Test Task Prediction Acc. (↑)**\n|Rank|1|4|8|12|16|\n|-|-|-|-|-|-|\n|LoRA-ML|0.37|0.36|0.39|0.31|0.37|\n|SR+LoRA|0.20|0.20|0.26|0.19|0.20|\n\nWe report the mean accuracy over the 10 test tasks using 5 random trials. The results demonstrate that LoRA-ML outperforms SR across a variety of ranks, demonstrating the effectiveness of our framework and its insensitivity to the adapter rank hyperparameter. We will include these results in the revised version of the paper.\n\n**(Q5) This paper is not easy to read for practical and general ML audiences. A short paragraph bridging the theory and experiments, and an overview figure of the pipeline can improve it.**\n\n(A5) Thank you for this helpful suggestion. Readability is an important concern, and we have aimed to keep our exposition and theoretical statements as accessible as possible while maintaining technical precision. While we cannot upload new figures during the rebuttal phase, we appreciate your feedback and will add a pipeline diagram in the revised version of the paper, as well as the following text to better connect the theory and the experiments:\n\n``In our theoretical analysis, we showed that minimizing a meta-learning-based retraining objective infused with LoRA (Eqn. 13) leads to provably stronger test-time adaptation compared to standard retraining for a class of linear models. Building on this, we evaluate our general framework for PEFT-based meta-learning (Eqn. 5) in more practical settings, using deeper and more complex architectures, both regression and classification tasks, and various PEFT methods beyond LoRA. Across these experiments, we find that incorporating the PEFT method directly into the retraining objective via our framework consistently improves downstream adaptation performance.\"\n\n**Concluding Remark:**\n\nWe hope that our responses have adequately addressed your concerns. If so, we kindly request that you consider revising your score. We are committed to addressing any additional concerns you may have during the discussion phase."}, {"review_id": "cTTKMtqrQa", "reviewer": "Reviewer_eo3S", "summary": "This paper introduces a framework for meta-learning with low-rank adaptations, aiming to enhance the adaptability of foundation models to unseen tasks. ​ The authors argue (and present proofs) that standard retraining methods aggregate task data to optimize model parameters, but they fail to account for the downstream fine-tuning, often resulting in suboptimal fine-tuning. ​ The proposed framework explicitly incorporates the fine-tuning mechanism during retraining to prepare the model for efficient adaptation to new tasks. ​The paper offers theoretical insights and guarantees for the proposed method, and tests the framework in three settings, demonstrating performance improvements over standard retraining. ​", "questions": "1. Do you have insights for why Reptile outperforms on some tasks, given the theoretical guarantees?\n2. How did you conclude that \"these spurious minima are almost never found in practice?\" I do not think the presented experiments suffice for this claim IMO.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The new framework combines PEFT with meta-learning to address the limitations of standard retraining methods. ​\n2. The paper provides rigorous theoretical guarantees and insights for the proposed framework, demonstrating that the proposed method outperforms standard retraining in terms of adaptability to unseen tasks. These results are supported by proofs and detailed analysis (under some theoretical constraints).\n3. Empirical results show that the proposed framework outperforms standard retraining and other meta-learning methods (e.g., Reptile) across synthetic, vision, and language tasks. ​\n4. The framework is tested with two PEFT strategies (LoRA and last-layer fine-tuning), demonstrating flexibility across different adaptation methods.\n\nWeakness:\n1. The results are not tested on strong datasets (CIFAR is a small dataset, and ConvAI2 does not properly capture task diversity IMO). Additionally, the paper assumes \"task diversity\" and \"linear independence\" of task-specific adaptations, which, of course, do not hold in real-world scenarios where tasks can be highly correlated; therefore, I think the method should be tested on more realistic datasets.\n2. Lack of ablation for the impact of different ranks or architectures.\n3. This can be subjective, while I appreciate the formal notations and explicit definitions, I do believe many parts of the write-up can be simplified (or written in simpler notation).", "was_revised": true, "final_justification": "The rebuttal acknowledges dataset scale, adds a simple rank ablation, and clarifies the claim about spurious minima with synthetic evidence and planned plots. A broader evaluation would strengthen the case; however, the gains across settings are convincing, so I will maintain the original rating.", "rebuttal": "Dear Reviewer eo3s, thank you for highlighting our rigorous guarantees as well as our practical experiments. We address your remaining questions below.\n\n**(Q1) The results are not tested on strong datasets (CIFAR is a small dataset, and ConvAI2 does not properly capture task diversity IMO). Additionally, the paper assumes \"task diversity\" and \"linear independence\" of task-specific adaptations, which, of course, do not hold in real-world scenarios where tasks can be highly correlated; therefore, I think the method should be tested on more realistic datasets.**\n\nThank you for raising this important point regarding the connection between our experimental setup and the assumptions made in our theoretical analysis. Regarding our theoretical framework, we note that assuming some level of task diversity is standard for meta-learning analyses [1-4], as if the tasks are extremely similar, modeling them individually provides little benefit. In our setting, the linear independence among the column spaces of the adapters forms a natural way to model this notion of task diversity.\n\nWe acknowledge that this may not hold in practice for real data, and we agree that evaluating our method on additional large-scale and realistic datasets would further strengthen the empirical evidence for its effectiveness. While our goal is to develop a theoretically principled framework for preparing a model for efficient test-time adaptation using PEFT, we recognize that larger-scale evaluation is a natural next step for future work.\n\n**(Q2) Lack of ablation for the impact of different ranks or architectures.**\n\nThank you for this suggestion. While we did not have time to incorporate additional architectures during the rebuttal period, we include an ablation for different ranks during the LoRA-ML retraining phase below for the language modeling task, where we aim to learn the responses of different personas in the ConvAI2 dataset. In Table 2 of the paper, we show results using rank-8 adaptations for LoRA-ML along with various other methods. Below, we show how changing the rank affects performance. As a baseline, we include standard retraining followed by rank-$k$ LoRA (SR+LoRA), where we use the same rank for both methods in each column. \n\n**Test Task Prediction Acc. (↑)**\n|Rank|1|4|8|12|16|\n|-|-|-|-|-|-|\n|LoRA-ML|0.37|0.36|0.39|0.31|0.37|\n|SR+LoRA|0.20|0.20|0.26|0.19|0.20|\n\nWe report the mean accuracy over the 10 test tasks using 5 random trials. The results demonstrate that LoRA-ML outperforms SR across a variety of ranks, demonstrating the effectiveness of our framework and its insensitivity to the adapter rank hyperparameter. We will include these results in the revised version of the paper.\n\n**(Q3) This can be subjective, while I appreciate the formal notations and explicit definitions, I do believe many parts of the write-up can be simplified (or written in simpler notation).**\n\nThank you for this comment. We note that Reviewer bDJi raised a similar point regarding the readability of the paper, which we agree is an important aspect of our presentation. To address this, we will include a new figure illustrating our proposed meta-learning pipeline in an accessible way, along with additional text bridging the theoretical and empirical sections. Please see our response (A5) to Reviewer bDJi for the exact planned wording.\n\n**(Q4) Do you have insights for why Reptile outperforms on some tasks, given the theoretical guarantees?**\n\nThank you for this question. Across all of our experiments, Reptile achieves the highest average performance only on the language modeling experiment for Task 1 using both adaptation mechanisms and for Task 6 using just last-layer adaptations. For our language modeling experiment, the test tasks are equipped with relatively few samples, leading to large performance variations across all methods. This leads to larger variation in the per-task performances, but when averaged over all test tasks, we observe that LoRA-ML comprehensively outperforms both Reptile and standard retraining (SR). Thus, these isolated instances where Reptile achieves better performance do not indicate a deeper trend. \n\n**(Q5) How did you conclude that \"these spurious minima are almost never found in practice?\" I do not think the presented experiments suffice for this claim IMO.**\n\nThank you for bringing this point up. This claim is supported by our extensive synthetic data experiments described in Section 4, which test the LoRA-ML objective for linear models, the same setting as our theoretical analysis. We observe, over all experiments, including the additional ablations in Appendix C.1, that minimizing the LoRA-ML objective with vanilla gradient descent and then fine-tuning the recovered shared parameter estimate $ \\\\hat{A} $ consistently minimizes the test-task loss up to the noise floor. If the LoRA-ML objective converged to a spurious local minimum which did not correspond to the ground truth shared parameter $ A^\\\\ast $, then the test task performance would significantly suffer. \n\nWe appreciate the reviewer's concern about this claim and will add plots that track the convergence of $ \\\\hat{A} $ to $ A^\\\\ast $ within our synthetic data experiments to further justify this claim. While we cannot upload new figures during the discussion period, we will include them in the revised version of the paper. \n\n**Concluding Remark**:\nWe hope that our responses have adequately addressed your questions and are committed to addressing any additional questions you may have during the discussion phase.\n\n[1] Saunshi, N., et al. ``A sample complexity separation between non-convex and convex meta-learning.\" ICML 2020.\n\n[2] Collins, L., et al. ``Maml and anil provably learn representations.\" ICML 2022.\n\n[3] Thekumparampil, K., et al. ``Statistically and computationally efficient linear meta-representation learning.\" NeurIPS 2021.\n\n[4] Du, S., et al. ``Few-shot learning via learning the representation, provably.\" ICLR 2021."}, {"review_id": "ECl5hUSMVr", "reviewer": "Reviewer_JxEW", "summary": "This paper provides a theoretical framework based on a linear regression model to analyze why meta-learning is more suitable than standard retraining during the retraining stage for preparing a model for future low-rank adaptation.", "questions": "See Strengths And Weaknesses above.", "limitations": "The authors acknowledge that the primary limitation of their work is that the theoretical analysis applies only to linear models with LoRA fine-tuning.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper provides a principled theoretical framework to explain why meta-learning is more suitable than multi-task learning (standard retraining) for retraining stage to prepare for future low-rank adaptation.\n2. The theoretical analysis looks solid.\n3. The paper is well-organized.\n\n**Weaknesses:**\n\n1. **Unfair theoretical setting between standard retraining and meta-learning:** The comparison between meta-learning and standard retraining appears to be unfair. In the standard retraining setup, only the base model \\$W\\$ is trained (as in Equation 1), while in the meta-learning setup, both the base model \\$W\\$ and the adapter \\$\\theta\\$ are trained (as in Equations 5 and 6). A fair comparison should involve the same model architecture and trainable parameters, with only the training algorithm differing.\n\n2. **Unclear definition and distinction between meta-learning and multi-task learning objectives, which could lead to unreliable conclusions:** Equation 14 is presented as the meta-learning objective. However, it is unclear how this differs from multi-task learning when both the base model and the adapter are trained. In traditional meta-learning, a key characteristic is the bi-level optimization on the same set of parameters, whereas the current formulation appears to reduce to a multi-task learning objective. If my interpretation is correct, the paper may actually be comparing multi-task learning on both the base model and adapter vs. multi-task learning on the base model only, rather than demonstrating a true meta-learning strategy.\n\n3. **Limited theoretical scope:** The theoretical analysis is confined to linear regression tasks. It remains unclear how well the conclusions generalize to more complex models.\n\n4. **Counterintuitive result in Theorem 3:** Theorem 3 suggests that there is an absolute condition on the number of tasks needed for generalization in meta-learning. However, empirical studies in the literature [1,2,3] often highlight the importance of *task diversity* for generalization performance, rather than merely the number of tasks. The paper should provide a discussion clarifying this discrepancy. Additionally, it would be valuable to explain the practical implications of the theorem for task construction in real meta-learning scenarios.\n\n\n[1] Meta-Learning with Fewer Tasks through Task Interpolation, ICLR 2022.\n\n[2] Data Augmentation for Meta-Learning, ICML 2021.\n\n[3] Meta-Learning Requires Meta-Augmentation, NeurIPS 2020.", "was_revised": true, "final_justification": "My primary concern was the distinction between the proposed PEFT-Meta-Learning (PEFT-ML) and standard Multi-Task Learning (MTL). This concern was effectively addressed through discussions with the authors, particularly their clarification of the formulation.\n\nWhile the theoretical contributions are currently limited to linear models, I acknowledge the value of the work in providing a principled perspective on the connection between meta-learning and foundation models. This perspective, in my view, is meaningful for the meta-learning community.", "rebuttal": "Dear Reviewer JxEW, thank you for recognizing the strength of our analysis as well as the organization of our presentation. We address your questions below.\n\n**(Q1) Unfair theoretical setting between standard retraining and meta-learning...**\n\n(A1) Thank you for this question which touches on a central difference in the way meta-learning models the different tasks relative to learning them in aggregate, which we refer to as standard retraining (SR).\n\nWe consider the problem of preparing a model for efficient adaptation to an unseen, downstream task, using a set of accessible representative tasks drawn from the same task distribution. Meta-learning is a well-established framework for this problem, as it explicitly aims to learn model parameters that can be \"easily\" adapted. A key aspect of any meta-learning method is defining this \"ease\" of adaptability. In MAML, for example, the adaptation mechanism is chosen to be a few steps of gradient descent on the task-specific loss. As you mention, this adaptation does not introduce additional parameters, as MAML simply uses a different training algorithm on the same parameter vector compared to standard retraining, which minimizes the average task loss directly.\n\nWe focus on parameter-efficient fine-tuning (PEFT) adaptation methods, as these are standard for fine-tuning large-scale models. A common PEFT paradigm is to freeze the given model weights and introduce a small set of additional trainable parameters, like LoRA adapters. To incorporate PEFT with meta-learning, we must include these additional parameters to form the inner meta-learning loss, just as MAML devises its inner loss as the result of applying a fixed number of gradient steps on the task-specific loss. Therefore, when we compare our PEFT-based meta-learning framework to SR, we are comparing two fundamentally different modeling strategies: (i) our approach, which models each task as a PEFT adaptation of a shared model, and (ii) SR, which learns a single parameter by aggregating all task data. Because PEFT methods can rely on additional task-specific parameters for fine-tuning, the fine-tuning adaptation procedure and the parameterization are inseparable. \n\nHowever, to compare our framework to a method which models all tasks in the aggregate but also includes the same number of trainable parameters as our proposed method, we analyze the following method, which we denote SR+. Consider our theoretical setting where we are given $T$ tasks, where each task $t$ is equipped with the infinite-sample loss function $\\mathcal{L}_t^\\ast$ defined by the shared parameter $A^\\ast \\in \\mathbb{R}^{d \\times d}$ along with the task-specific rank-$k$ adapter with factor $U_t^\\ast \\in \\mathbb{R}^{d \\times k}$:\n\n\\begin{equation*}\n    \\mathcal{L}_t^\\ast (A) = \\|| A - A^\\ast - U_t^\\ast U_t^{\\ast \\top} \\||_F^2\n\\end{equation*}\n\nWe define the SR+ method to minimize $ \\sum_t \\mathcal{L}_t^\\ast $ just like SR, except it uses the parameterization $A + U U^\\top$, where $A \\in \\mathbb{R}^{d \\times d}$ aims to recover the ground truth matrix $A^\\ast$, while $U \\in \\mathbb{R}^{d \\times kT}$ represents a rank $kT$ adaptation applied to each task. Formally, SR+ minimizes:\n\n$$\n\\mathcal{L}_{SR+} (A,U) =  \\sum _{t=1}  ^T  \\|| A + U U^\\top - A^\\ast - U_t^\\ast U_t^{\\ast \\top} \\||_F^2\n$$\n\nThus, SR+ uses the same number of parameters as LoRA-ML but still models the tasks in aggregate like SR. We then directly show that it is impossible to recover $A^\\ast$ by minimizing this SR+ objective.\n\n**Theorem:** Let $\\hat{U}$ be any $d \\times kT$ matrix. Then for the matrix $\\hat{A} = -\\hat{U} \\hat{U}^\\top + A^\\ast + \\frac{1}{T} \\sum_{t=1}^T U_t^\\ast U_t^{\\ast \\top}$, we have that $(\\hat{A}, \\hat{U})$ globally minimizes $\\mathcal{L}_{\\text{SR+}}$\n\nThus $A^\\ast$ is completely unidentifiable by the SR+ loss, as any value of $\\hat{U}$ corresponds to a global minimizer, whereas the only value that leads to recovery $\\hat{A} = A^\\ast$ is when $\\hat{U}$ contains the properly scaled columns of the ground truth factors $U_t^\\ast$. However, this is impossible to recover, since any of the infinite values of $\\hat{U} \\in \\mathbb{R}^{d \\times kT}$ can result in global minimization. This is in contrast to our LoRA-ML objective, where we show that global minimization always leads to exact recovery of the ground truth parameter $A^\\ast$, with a rank $2k$ error only when $T=2$.\n\nWe hope this clarifies the rationale behind our setup and addresses your concern about a fair comparison between our method and standard retraining. We are happy to provide further details if helpful.\n\n**(Q2) Unclear definition and distinction between meta-learning and multi-task learning...rather than demonstrating a true meta-learning strategy.**\n\n(A2) Thank you for this comment. We clarify the interpretation of our method meta-learning rather than standard multi-task learning.\n\nA key feature of meta-learning is the bi-level optimization structure, where the outer objective minimizes a sum of task-specific losses over shared parameters, where each task-specific loss is itself a minimization problem over a task-specific adaptation. This precisely underlies our derivation: in the linear case in Equation (13), the inner problem minimizes the loss for each task after fine-tuning with the task-specific adapter $U_t$, and the outer problem optimizes the shared base model parameter $A$ to improve post-adaptation performance.\n\nBecause the LoRA adaptation introduces new parameters to update, the entire meta loss can be rewritten in Eqn. (14) as the joint minimization over $A$ and $\\\\{ U_t\\\\}_{t=1}^T$. While this resembles a multi-task learning objective, it is the outcome of a meta-learning formulation with a specific adaptation rather than simply a multi-task loss.\n\nAs you mention, in prior model agnostic meta-learning methods such as MAML or Reptile, the inner adaptation runs full gradient steps over all model parameters. Our method aligns with model-based meta-learning, where the focus is on structuring the model (e.g., via PEFT modules) to enable fast adaptation. This explains why our meta-objective can be equivalently expressed in the form  of Eqn. (14), since the inner problem introduces new parameters to update with respect to the outer problem.\n\n**(Q3) Limited theoretical scope..**\n\n(A3) We recognize that our theoretical contributions only directly apply to adapting linear models with LoRA. While this setting allows us to derive precise and interpretable guarantees within the new paradigm of meta-learning via PEFT-based adaptation, we acknowledge that these results do not directly extend to the non-linear or deep networks where we ultimately aim to apply these ideas.\n\nTo address the practical setting, we include a set of empirical experiments across vision and language tasks using deep networks, demonstrating that the key principles established theoretically, such as improved adaptation via retraining with our proposed framework, continue to hold in practice. We agree that extending our framework to more complex models is a natural and interesting next step for future work. \n\n**(Q4) Counterintuitive result in Theorem 3...**\n\n(A4) Thank you for this comment.\n\nTheorem 3 states that, given infinite samples per task, access to 3 independent tasks guarantees exact recovery of the ground truth parameters, no matter the rank of the ground truth adapters. This contrasts with prior meta-learning analyses [1-4] that consider linear representation learning settings, where the number of tasks needed for ground truth parameter recovery depends on the intrinsic task dimension. This difference arises since these previously analyzed settings correspond to subspace recovery problems, but our setting requires estimating a matrix $A^\\ast$ from low-rank perturbations $A^\\ast + U_t^\\ast U_t^{\\ast \\top}$. For more details on the technical challenges of our setting, please see our response (A4) to Reviewer qeBR.\n\nIn terms of the empirical observations that highlight the importance of task-diversity rather than just the number of tasks, we note that this is in fact supported by our result in Theorem 3 which shows that recovery fails with many dependent tasks but succeeds with as few as three independent ones.\n\nRegarding practical task construction, we first note that the result of Theorem 3 is specific to LoRA-based meta-learning for linear models. For other PEFT strategies or model architectures, 3 tasks may not be enough to guarantee exact parameter recovery. Further, in practical settings, we only have access to tasks with a finite number of samples, whereas Theorem 3 considers the infinite sample case. With only finite samples, each task does not give an exact measurement of the true parameters, so increasing the number of tasks can effectively denoise the parameter estimate by providing additional measurements of the true parameters. Thus, it is likely beneficial to construct a larger set of diverse, high-quality tasks in practice. \n\nWe will add this discussion relating the result in Theorem 3 to the empirical studies you mention as well as its implications for practical task construction to the revised paper.\n\n**Concluding Remark**\n\nWe hope that our responses have adequately addressed your concerns regarding the theoretical characterization of our framework and the guarantees we provide, along with how they relate to practical insights. If so, we kindly request you to consider revising your score. We are committed to addressing any additional questions you may have during the discussion phase.\n\n[1] Saunshi, N., et al. ``A sample complexity separation between non-convex and convex meta-learning.\" ICML 2020.\n\n[2] Collins, L., et al. ``Maml and anil provably learn representations.\" ICML 2022.\n\n[3] Thekumparampil, K., et al. ``Statistically and computationally efficient linear meta-representation learning.\" NeurIPS 2021.\n\n[4] Du, S., et al. ``Few-shot learning via learning the representation, provably.\" ICLR 2021."}, {"review_id": "ZamMILCqxp", "reviewer": "Reviewer_qeBR", "summary": "This paper provides an analysis of the retraining and fine-tuning stages that foundation models require for downstream applications. The authors argue that the retraining phase should be tailored to the nature of the fine-tuning stage, which usually employs parameter-efficient fine-tuning methods. Meta-learning is proposed as the overall framework to achieve this within the retraining phase. To derive theoretical results, the authors propose a multiple linear regression setting where each ground truth is a low-rank perturbation away from a ground truth matrix $A^\\star$. Given many tasks, the authors show that standard retraining (i.e., minimizing the sum of the losses for all tasks) results in a high-rank estimate and does not lead to efficient fine-tuning. In contrast, they show that the meta-learning approach in the high-sample limit is low-rank adaptable, proving that approaches such as Low-Rank Adaptation lead to fast fine-tuning.", "questions": "1. Could you explain why have you chosen $R_t^\\star = U_t^\\star (U_t^\\star)^\\top$. I know that it is difficult justify the theoretical setting as it is unclear what is the structure of the real-world data. However, since this choice is a bit unusual, it requires some commentary from the authors.\n\n2. Is it possible to derive the same results with $R_t^\\star \\in \\mathbb{R}^{d \\times d}$ that is of low-rank but not necessarily PSD? I believe then the minimizer (analog of Equation 12) converges to $A^\\star$ with a decaying rate in $T$. This means that standard retraining might actually work without any issues if there are many tasks. Could you comment on this claim of mine?\n\n3. Remark 1 is not suited for many task regimes. Could you comment on how reasonable Remark 1 is in practice? In conjunction with the previous question, wouldn't your results nullify then as standard retraining benefits from a high number of tasks?\n\n4. Can you comment a bit more in detail about the technical challenges in the paper?", "limitations": "The work is theoretical in nature, and its applicability is limited to the setting introduced by the authors. This is evident from the text, and I consider this limitation to be acknowledged. The authors also chose to run experiments to verify if their intuitions extend to real-world settings, which is a plus. The only suggestion I have is that the authors could discuss their technical assumptions (see questions section) so that any limitations are more clearly presented.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nThe authors introduce an original framework to study retraining and fine-tuning. This is well-suited to theoretical insights and intuition. In terms of significance, it can lead to follow-up works on the area. Overall, the paper is written clearly.\n\n**Weaknesses:**\n\nThe design choices in the theoretical setting are not well-discussed. This hinders the overall quality of the theoretical work. Examples include the positive-definite structure of $R_t^\\star$ and if Remark 1 holds in practice. In addition, I believe the results are technically non-challenging.", "was_revised": true, "final_justification": "I keep my original score as the main results of the paper are due to choices that are made to simplify the analysis. With a more forthcoming presentation, my recommendation would be \"borderline accept\" as it has results that are technical correct and interesting.", "rebuttal": "Dear Reviewer qeBR, thank you for your positive feedback regarding the clarity of our presentation and the originality of our contributions. We address your remaining questions below.\n\n**(Q1-2) Could you explain...commentary from the authors... Is it possible... claim of mine**\n\n(A1-2) Thank you for raising this important point.\n\nAs you note, the structure of real-world adaptations is not well understood. We analyze the symmetric case to avoid complications in the analysis that do not primarily relate to our goal of studying meta-learning with low-rank adaptation. The LoRA fine-tuning approach we analyze is closely related to problems in matrix sensing, where assuming the ground truth matrix is symmetric can lead to a cleaner analysis. While it is not immediately clear how to extend the guarantees of our proposed LoRA-ML objective to the asymmetric case (please see our response (A4) for details on technical challenges), we emphasize that **even in the case of asymmetric adapters, standard retraining provably fails**.\n\nIf the ground truth adapters $R_t^\\ast = U_t^\\ast V_t^\\ast$ are asymmetric, the analog of Eqn. (12) shows that standard retraining (SR) recovers $ \\hat{A}_{SR} = A^\\ast + \\frac{1}{T} \\sum _{t=1} ^T R_t^\\ast $. For any $T$, this estimate will deviate from $A^\\ast$ by an error of large rank under a mild condition:\n\n**Theorem**: For finite $T$, assume the distribution over adapters $R_t^\\ast$ is absolutely continuous with respect to the Lebesgue measure and that the ambient dimension $d$ is large enough so $kT \\leq d$. Then with probability 1, \n$ rank(\\hat{A}_{\\text{SR}} - A^\\ast) = kT $\n\nThus for finite $T$, an analog of Proposition 1 still holds: a rank-$k(T+1)$ adapter is required to exactly fit to a test task, even though recovering $A^\\ast$ would have allowed us to adapt using only a rank-$k$ adapter.\n\nFor large $T$, we emphasize that in general **$\\hat{A}_{SR}$ does not converge to $A^*$**. The error term $\\frac{1}{T} \\sum_{t=1}^T R_t^\\ast$ is the average of the adapters. If we assume that $R_t^\\*$ are drawn i.i.d. from a fixed distribution with mean $M_R$, then this term converges almost surely to $M_R$ as $T \\to \\infty$:\n\n**Theorem:** Consider ground truth adapters $R_t^*$ which are drawn from some arbitrary distribution with mean $M_R$. Then, $\\lim_{T \\rightarrow \\infty} \\hat{A}_{SR} = A^\\ast +  M_R$\n\nEven this limit, SR cannot recover $A^\\ast$ unless $M_R = 0$. We emphasize that this $0$-mean scenario is highly restrictive. To be effective in realistic applications, an algorithm must succeed under general task distributions.\n\nWe now clarify our assumption that $U_t^\\ast$ is generated according to a $0$-mean distribution in our symmetric setting, even though we have just argued against requiring such assumptions. This choice was for notational and analytical convenience, as **all of our guarantees for LoRA-ML in Section 3.2 hold deterministically for any $\\\\{U_t^*\\\\}_{t=1}^T$ as long as the diversity condition in Remark 1 is satisfied**. We use the assumed distribution over $U_t^*$ only to quantify the error of SR in an average sense in Theorem 1 and Proposition 2. These assumptions do not restrict the generality of our results on the performance of LoRA-ML, as long as the diversity condition holds. \n\nWe will update the revised paper to include the above discussion and clarify (i) why we assumed symmetric adapters, (ii) why standard retraining still fails in the asymmetric case, and (iii) why the distributional parameters over $U_t^\\ast$ do not affect our performance guarantees for LoRA-ML. \n\n**(Q3) Remark 1 is not suited ... number of tasks?**\n\n(A3) Thank you for this comment. To clarify, we make the technical assumption of independent adapters in Remark 1 to naturally formalize the notion of task diversity for our theoretical setting. This is analogous to prior works [1-4], which consider the problem of recovering a shared linear representation across tasks. To guarantee recovery, they require that the tasks represent a set of independent vectors on this shared subspace.\n\nWe note that it is difficult to make claims about the structure of idealized adapters for real data. When adapting deep networks via LoRA across multiple layers, it is also unclear which properties of these adapters meaningfully capture task diversity. Linear independence was natural for our setting, but a more complex notion is likely required for deep networks. Further, in low diversity settings, the diminishing marginal benefit of meta-learning relative to standard retraining is expected. If the tasks are entirely unrelated, or conversely nearly identical, then meta-learning is not the appropriate tool: in the former case, no shared structure exists to exploit, and in the latter, modeling task-specific variation is unnecessary.\n\nWe agree with the reviewer that the characterization of task diversity deserves further discussion, particularly in relation to the relative performance of meta-learning versus standard retraining. We will update the paper to include these additional insights, clarifying both the importance of our task diversity assumptions and their implications for performance.\n\n**(Q4) Can you comment a bit more in detail about the technical challenges in the paper?**\n\n(A4) Thank you for this comment. We are happy to discuss the technical challenges of our analysis and theoretical framework as a whole.\n\nFirstly, we note that our framework differs fundamentally from standard meta-learning analyses focused on recovering a shared linear representation across tasks [1-4]. In those settings, parameter recovery reduces to a subspace recovery problem. In contrast, our framework involves recovering $A^\\ast$ from low-rank perturbations $A^\\ast + U_t^\\ast U_t^{\\ast \\top}$. Here, the measurements are additive and non-linear (algebraically) in $A^\\ast$, making the recovery problem fundamentally different.\n\nThe first technical challenge we faced was understanding the solution set of our LoRA-ML objective in Eqn. (14). Remarkably, we show in Theorem 3 that if (i) $T \\geq 3$ and (ii) there exist distinct adapters $U_s^\\ast$, $U_t^\\ast$, $U_r^\\ast$ with independent column spaces, then the *only* solutions minima of Eqn. (14) are the exact ground truth parameters. This is counterintuitive to prior meta-learning analyses, which require that the number of tasks scale with the intrinsic dimension of each task to ensure recovery. However, in our case, no matter the ground truth adapter rank $k$, we can always guarantee recovery given the above assumptions.\n\nThe crux of the proof relies on using the set of equalities $U_1^\\ast U_1^{\\ast \\top} = U_1 U_1^{ \\top} + U_2^\\ast U_2^{\\ast \\top} - U_2 U_2^{\\top} = U_1 U_1^{ \\top} + U_3^\\ast U_3^{\\ast \\top} - U_3 U_3^{\\top}$ along with the assumption that the combined columns of $U_1^\\ast$, $U_2^\\ast$, and $U_3^\\ast$ are linearly independent to show that $im(U_1) = im(U_1^\\ast)$, where $im$ denotes the matrix image. We show that for any $\\\\{U_i,U_i^\\ast\\\\}_{i=1}^3$ which satisfy both of the above conditions, we must have $im(U_1) = im(U_1^\\ast)$. The rest of the proof then quickly follows.\n\nThe second challenge we faced related to the landscape of our proposed objective in Eqn. (14). In Theorem 4, we show with $T=2$ tasks, our objective Eqn. (14) has a strict saddle property, meaning that even though it is non-convex, every second-order stationary point (SOSP) is in fact a global minimizer.\n\nThis proof is more involved, but we discuss the key steps along with the barrier we face going beyond $T=2$. We note that for a fixed $A$, the objective in Eqn. (14) reduces to separate matrix factorization objectives, but joint minimization leads to challenging interactions between the variables, which form the technical complications. When $T=2$, the analysis reduces to relating the matrix $U_1 U_1^\\top-U_2U_2^\\top$ to its ground truth counterpart $U_1^\\ast(U_1^\\ast)^\\top-U_2^\\ast(U_2^\\ast)^\\top$. For larger $T $ we cannot make this reduction. The proof follows by contradiction, where we assume the existence of an SOSP $(A,U)$ of Eqn. (14), which does not achieve the minimum function value. For such an $(A,U)$, we prove that $U_1U_1^\\top-U_2U_2^\\top$ shares an eigenbasis with $U_1^\\ast(U_1^\\ast)^\\top-U_2^\\ast(U_2^\\ast)^\\top$, but $U_1U_1^\\top-U_2U_2^\\top$ is rank deficient whereas $U_1^\\ast(U_1^\\ast)^\\top-U_2^\\ast(U_2^\\ast)^\\top$ has rank-$2k$ due to the diversity assumption. Thus, there exists a non-zero eigenvector of $U_1^\\ast(U_1^\\ast)^\\top-U_2^\\ast(U_2^\\ast)^\\top$ which belongs to the nullspace of $U_1U_1^\\top-U_2U_2^\\top$. We use this eigenvector to construct a negative direction of the Hessian, contradicting the fact that $(A,U)$ was an SOSP.\n\nWe note that our analysis relied on many subtle properties of the problem, which followed from the assumption of only two tasks. Through numerical experiments, we found strong evidence that our objective in Eqn. (14) can exhibit spurious local minima in some degenerate cases when $T>2$ (please see Appendix E). This sharp transition from $T=2$ to $T=3$ underscores the unique complexity of the meta-learning landscape we consider and highlights the broader challenges of our analysis.\n\n**Concluding Remark:**\n\nWe hope that our responses have adequately addressed your concerns regarding our theoretical assumptions and their practicality, as well as the technical challenges we encountered. If so, we kindly request you to consider revising your score. We are committed to addressing any additional questions you may have during the discussion phase.\n\n[1] Saunshi, N., et al. \"A sample complexity separation between non-convex and convex meta-learning.\" ICML 2020.\n\n[2] Collins, L., et al. \"Maml and anil provably learn representations.\" ICML 2022.\n\n[3] Thekumparampil, K., et al. \"Statistically and computationally efficient linear meta-representation learning.\" NeurIPS 2021.\n\n[4] Du, S., et al. \"Few-shot learning via learning the representation, provably.\" ICLR 2021."}]

Author rebuttal:

Meta-review: This paper theoretically analyzes a meta-learning problem setup based on low-rank adaptation and investigates conditions under which it outperforms simple multitask learning. While the scope of the theoretical analysis is limited to linear models, the claims are solid and the experimental results including beyond-linear setting support them well. The findings offer a principled perspective that spans meta-learning and foundation models.

Reviewer opinions are split around the borderline. In particular, qeBR gave a score of 3 and argued that the problem setup is overly simplified for theory. However, the authors’ response to this point appears sufficiently reasonable. No other major concerns are evident, and I judge that the paper’s theoretical contribution is substantial for the meta-learning community, so I recommend acceptance.
Decision: Accept (poster)