Title: Provable Meta-Learning with Low-Rank Adaptations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Meta-Learning

Low-Rank Adaptation

Parameter-Efficient Fine-Tuning

Non-convex Optimization
Abstract: The power of foundation models (FMs) lies in their capacity to learn highly expressive representations that can be adapted to a broad spectrum of tasks. However, these pretrained models require additional training stages to become effective for downstream applications. In the multi-task setting, prior works have shown empirically that specific meta-learning approaches for preparing a model for future adaptation through parameter-efficient fine-tuning (PEFT) can outperform standard retraining methods, but the mechanism of the benefits of meta-learning has been largely unexplored. We introduce a framework for generic PEFT-based meta-learning to learn a model that can easily adapt to unseen tasks. For linear models using LoRA, we show that standard retraining is provably suboptimal for finding an adaptable set of parameters and provide strict performance guarantees for our proposed method. We verify these theoretical insights through experiments on synthetic data as well as real-data vision and language tasks. We observe significant performance benefits using a simple implementation of our proposed meta-learning scheme during retraining relative to the conventional approach.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an analysis of the retraining and fine-tuning stages that foundation models require for downstream applications. The authors argue that the retraining phase should be tailored to the nature of the fine-tuning stage, which usually employs parameter-efficient fine-tuning methods. Meta-learning is proposed as the overall framework to achieve this within the retraining phase. To derive theoretical results, the authors propose a multiple linear regression setting where each ground truth is a low-rank perturbation away from a ground truth matrix $A^\\star$. Given many tasks, the authors show that standard retraining (i.e., minimizing the sum of the losses for all tasks) results in a high-rank estimate and does not lead to efficient fine-tuning. In contrast, they show that the meta-learning approach in the high-sample limit is low-rank adaptable, proving that approaches such as Low-Rank Adaptation lead to fast fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe authors introduce an original framework to study retraining and fine-tuning. This is well-suited to theoretical insights and intuition. In terms of significance, it can lead to follow-up works on the area. Overall, the paper is written clearly.\n\n**Weaknesses:**\n\nThe design choices in the theoretical setting are not well-discussed. This hinders the overall quality of the theoretical work. Examples include the positive-definite structure of $R_t^\\star$ and if Remark 1 holds in practice. In addition, I believe the results are technically non-challenging.\n\nQUESTIONS:\n1. Could you explain why have you chosen $R_t^\\star = U_t^\\star (U_t^\\star)^\\top$. I know that it is difficult justify the theoretical setting as it is unclear what is the structure of the real-world data. However, since this choice is a bit unusual, it requires some commentary from the authors.\n\n2. Is it possible to derive the same results with $R_t^\\star \\in \\mathbb{R}^{d \\times d}$ that is of low-rank but not necessarily PSD? I believe then the minimizer (analog of Equation 12) converges to $A^\\star$ with a decaying rate in $T$. This means that standard retraining might actually work without any issues if there are many tasks. Could you comment on this claim of mine?\n\n3. Remark 1 is not suited for many task regimes. Could you comment on how reasonable Remark 1 is in practice? In conjunction with the previous question, wouldn't your results nullify then as standard retraining benefits from a high number of tasks?\n\n4. Can you comment a bit more in detail about the technical challenges in the paper?\n\nLIMITATIONS:\nThe work is theoretical in nature, and its applicability is limited to the setting introduced by the authors. This is evident from the text, and I consider this limitation to be acknowledged. The authors also chose to run experiments to verify if their intuitions extend to real-world settings, which is a plus. The only suggestion I have is that the authors could discuss their technical assumptions (see questions section) so that any limitations are more clearly presented.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical framework based on a linear regression model to analyze why meta-learning is more suitable than standard retraining during the retraining stage for preparing a model for future low-rank adaptation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper provides a principled theoretical framework to explain why meta-learning is more suitable than multi-task learning (standard retraining) for retraining stage to prepare for future low-rank adaptation.\n2. The theoretical analysis looks solid.\n3. The paper is well-organized.\n\n**Weaknesses:**\n\n1. **Unfair theoretical setting between standard retraining and meta-learning:** The comparison between meta-learning and standard retraining appears to be unfair. In the standard retraining setup, only the base model \\$W\\$ is trained (as in Equation 1), while in the meta-learning setup, both the base model \\$W\\$ and the adapter \\$\\theta\\$ are trained (as in Equations 5 and 6). A fair comparison should involve the same model architecture and trainable parameters, with only the training algorithm differing.\n\n2. **Unclear definition and distinction between meta-learning and multi-task learning objectives, which could lead to unreliable conclusions:** Equation 14 is presented as the meta-learning objective. However, it is unclear how this differs from multi-task learning when both the base model and the adapter are trained. In traditional meta-learning, a key characteristic is the bi-level optimization on the same set of parameters, whereas the current formulation appears to reduce to a multi-task learning objective. If my interpretation is correct, the paper may actually be comparing multi-task learning on both the base model and adapter vs. multi-task learning on the base model only, rather than demonstrating a true meta-learning strategy.\n\n3. **Limited theoretical scope:** The theoretical analysis is confined to linear regression tasks. It remains unclear how well the conclusions generalize to more complex models.\n\n4. **Counterintuitive result in Theorem 3:** Theorem 3 suggests that there is an absolute condition on the number of tasks needed for generalization in meta-learning. However, empirical studies in the literature [1,2,3] often highlight the importance of *task diversity* for generalization performance, rather than merely the number of tasks. The paper should provide a discussion clarifying this discrepancy. Additionally, it would be valuable to explain the practical implications of the theorem for task construction in real meta-learning scenarios.\n\n\n[1] Meta-Learning with Fewer Tasks through Task Interpolation, ICLR 2022.\n\n[2] Data Augmentation for Meta-Learning, ICML 2021.\n\n[3] Meta-Learning Requires Meta-Augmentation, NeurIPS 2020.\n\nQUESTIONS:\nSee Strengths And Weaknesses above.\n\nLIMITATIONS:\nThe authors acknowledge that the primary limitation of their work is that the theoretical analysis applies only to linear models with LoRA fine-tuning.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for meta-learning with low-rank adaptations, aiming to enhance the adaptability of foundation models to unseen tasks. ​ The authors argue (and present proofs) that standard retraining methods aggregate task data to optimize model parameters, but they fail to account for the downstream fine-tuning, often resulting in suboptimal fine-tuning. ​ The proposed framework explicitly incorporates the fine-tuning mechanism during retraining to prepare the model for efficient adaptation to new tasks. ​The paper offers theoretical insights and guarantees for the proposed method, and tests the framework in three settings, demonstrating performance improvements over standard retraining. ​\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The new framework combines PEFT with meta-learning to address the limitations of standard retraining methods. ​\n2. The paper provides rigorous theoretical guarantees and insights for the proposed framework, demonstrating that the proposed method outperforms standard retraining in terms of adaptability to unseen tasks. These results are supported by proofs and detailed analysis (under some theoretical constraints).\n3. Empirical results show that the proposed framework outperforms standard retraining and other meta-learning methods (e.g., Reptile) across synthetic, vision, and language tasks. ​\n4. The framework is tested with two PEFT strategies (LoRA and last-layer fine-tuning), demonstrating flexibility across different adaptation methods.\n\nWeakness:\n1. The results are not tested on strong datasets (CIFAR is a small dataset, and ConvAI2 does not properly capture task diversity IMO). Additionally, the paper assumes \"task diversity\" and \"linear independence\" of task-specific adaptations, which, of course, do not hold in real-world scenarios where tasks can be highly correlated; therefore, I think the method should be tested on more realistic datasets.\n2. Lack of ablation for the impact of different ranks or architectures.\n3. This can be subjective, while I appreciate the formal notations and explicit definitions, I do believe many parts of the write-up can be simplified (or written in simpler notation).\n\nQUESTIONS:\n1. Do you have insights for why Reptile outperforms on some tasks, given the theoretical guarantees?\n2. How did you conclude that \"these spurious minima are almost never found in practice?\" I do not think the presented experiments suffice for this claim IMO.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new meta-learning framework tailored for parameter-efficient fine-tuning. Even though they claim it can work with any PEFT method, the main focus is on LoRA. By providing theoretical results in a linear regression setting, the author shows that standard retraining cannot produce representations that are easily adaptable to new tasks. Their proposed framework on the other hand guarantees low-rank adaptability and lower prediction error. Experiments on synthetic data and small-scaled vision and language tasks support the theoretical claims.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The theoretical contribution is novel and strong. Showing that standard retraining is suboptimal and proposing a framework with provably better adaptation.\n-\tThe problem formulation is well-defined with clear assumptions.\n-\tThe theoretical analysis of the effect of the number of tasks on the proposed model is another strength of this paper. Especially unique recovery of the ground truth for three or more tasks.\n-\tFigure 1 comprehensively supports the theoretical results in the controlled linear setting.\n\nMajor issues:\n\n-\tThe theoretical results are driven under linear regression setting with Gaussian noise. However, classification tasks on CIFAR-10 using an MLP-Mixer and language modeling tasks with RoBERTa involve nonlinear architectures and classification losses. The theoretical guarantees do not extend to these more complex settings. \n-\tThe proposed framework requires solving a nested optimization problem. For each task, adapter parameters must be learned in the inner loop. This introduces computational overhead compared to standard retraining. However, the paper does not provide any analysis or empirical comparison of the computational cost, memory usage, or convergence time between the proposed method and the baselines. \n-\tSome results (for example, Table 1 - left) show that the performance of standard retraining is very close to the proposed method. Even in some cases better, if we consider the standard deviations. The empirical result contradicts the strong theoretical claims which predict that standard retraining should become significantly worse as the number of tasks increases. This suggests that the theoretical benefits may not transfer effectively to nonlinear and classification settings and authors should discuss it.\n-\tMore ablation studies can add great value to the paper, such as more diverse parameter-efficient fine-tuning methods or empirical results on real-world large-scale benchmarks.\n\nMinor issues:\n\n-\tLine 282, “task task”\n\nQUESTIONS:\nSuggestions:\n\n-\tThis paper is not easy to read for practical and general ML audiences. A short paragraph bridging the theory and experiments and an overview figure of the pipeline can improve it.\n-\tAdd computational complexity analysis\n-\tShowing large-scale benchmark experiments\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}