NeurIPS 2025

Provable Meta-Learning with Low-Rank Adaptations
Jacob L. Block
UT Austin
jblock@utexas.edu
Sundararajan Srinivasan
UT Austin
sundararajans@utexas.edu
Liam Collins
Snap, Inc.
lcollins2@snapchat.com
Aryan Mokhtari
UT Austin & Google Research
mokhtari@austin.utexas.edu
Sanjay Shakkottai
UT Austin
sanjay.shakkottai@utexas.edu
Abstract
The power of foundation models (FMs) lies in their capacity to learn highly expres-
sive representations that can be adapted to a broad spectrum of tasks. However,
these pretrained models require additional training stages to become effective for
downstream applications. In the multi-task setting, prior works have shown em-
pirically that specific meta-learning approaches for preparing a model for future
adaptation through parameter-efficient fine-tuning (PEFT) can outperform standard
retraining methods, but the mechanism of the benefits of meta-learning has been
largely unexplored. We introduce a framework for generic PEFT-based meta-
learning to learn a model that can easily adapt to unseen tasks. For linear models
using LoRA, we show that standard retraining is provably suboptimal for finding
an adaptable set of parameters and provide strict performance guarantees for our
proposed method. We verify these theoretical insights through experiments on
synthetic data as well as real-data vision and language tasks. We observe significant
performance benefits using a simple implementation of our proposed meta-learning
scheme during retraining relative to the conventional approach.
1
Introduction
Foundation Models (FMs) learn rich representations that are useful for a variety of downstream tasks.
The first stage of FM training is referred to as pretraining, where a combination of massive public,
proprietary, and synthetic sources of data is used to learn a general-purpose model from scratch
[1–4]. However, due to the enormous cost of training state-of-the-art models on such large datasets,
pretraining is largely infeasible for most. Thus, the ...