LiFT: Learning to Fine-Tune via Bayesian Parameter Efficient Meta Fine-Tuning
Authors: Minyoung Kim
Timothy Hospedales
Venue: ICLR 2025 Spotlight (2025)
Keywords: Bayesian methods
Parameter efficient fine-tuning
meta learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a novel hierarchical Bayesian model for meta PEFT that can be superior to both existing general meta learning algorithms like MAML and recent LoRA zoo mixing approaches such as Retrievers and model-based clustering.

Abstract: We tackle the problem of parameter-efficient fine-tuning (PEFT) of a pre-trained large deep model on many different but related tasks. Instead of the simple but strong baseline strategy of task-wise independent fine-tuning, we aim to meta-learn the core shared information that can be used for unseen test tasks to improve the prediction performance further. That is, we propose a method for {\em learning-to-fine-tune} (LiFT). LiFT introduces a novel hierarchical Bayesian model that can be superior to both existing general meta learning algorithms like MAML and recent LoRA zoo mixing approaches such as LoRA-Retriever and model-based clustering. In our Bayesian model, the parameters of the task-specific LoRA modules are regarded as random variables where these task-wise LoRA modules are governed/regularized by higher-level latent random variables, which represents the prior of the LoRA modules that capture the shared information across all training tasks. To make the posterior inference feasible, we propose a novel SGLD-Gibbs sampling algorithm that is computationally efficient. To represent the posterior samples from the SGLD-Gibbs, we propose an online EM algorithm that maintains a Gaussian mixture representation for the posterior in an online manner in the course of iterative posterior sampling. We demonstrate the effectiveness of LiFT on NLP and vision multi-task meta learning benchmarks.

(Zhao et al. 2024b) proposes to solve new tasks as Published as a conference paper at ICLR 2025
LIFT: LEARNING TO FINE-TUNE VIA BAYESIAN PA- RAMETER EFFICIENT META FINE-TUNING
Minyoung Kim' & Timothy M. Hospedales 1.2
Samsung Al Center Cambridge, UK mi mikkm220gmmm 1@gmaimalll com
University of Edinburgh, UK hoospedaleseed. ac. uk
ABSTRACT
We tackle the problem of parameter-efficient fine-tuning (PEFT) of a pre-trained large deep model on many different but related tasks. Instead of the simple but strong baseline strategy of task-wise independent fine-tuning, we aim meta-leam the core shared information that can be used for unseen test tasks to improve the prediction performance further. That is, we propose method for learning- to-fine-tuxe tune (LiFT). LiFT introduces novel hierarchical Bayesian model that can be superior to both existing general meta learning algorithms like MAML and recent LoRA zoo mixing approaches such as based clustering. our Bayesian model, the parameters of the task-specific LoRA modules regarded as random variables where these task-wise LoRA modules are governed/regularizzed by higher-level latent random variables, which represents the prior of the LoRA modules that capture the shared information across all training tasks. To make the posterior inference feasible, we propose a novel SGLD- sampling algorithm that is computationally efficient. To represent the posterior samples from the SGLD-Gibbs, we propose an online EM algorithm that maintains Gaussian mixture representation for the posterior in an online manner in the course of iterative posterior sampling. We demonstrate the effectiveness of LiFT on NLP and vision multi-task meta learning benchmarks.
and model-
INTRODUCTION
Applied machine learning in most domains is increasingly dominated by paradigm of fine-tuning models pre-trained on large-scale unlabelled data with self-supervised objectives (Bommasani et al., 2021). The consolidation of the community around this workflow has led to flourishing of research in parameter-efficient fine-tuning (PEFT) methods (Ding et al. 2023 Hu et al. 2022). PEFT methods insert, or select, small set of learnable parameters with the pre-trained model and update only these on the task of interest, while keeping the rest the model frozen. The aim is to benefit from the representation provided from large are-trained model, while only learning small number of parameters to avoid overfitting and keep the marginal parameter storage cost per task low. Popular PEFT methods such as LoRA (Hu al. 2022), nowadays often underpin flagship new capabilities in NLP Ding et al. 2023). vision and multi- (Liu et al., 2024) AI.
The PEFT paradigm's 's efficacy due effective knowledge transfer from the upstream pre-trained model to the downstream task of interest. However, in most workflows each downstream task is solved independently, and there is no exploitation of knowledge transfer between the growing array of supervised tasks that are tackled downstream. This has led some recent studies to ask whether PEFT modules can be re-used across multiple downstream tasks in order to further improve performance? For example L-RA-HUB (Huang et al. (023) proposes to learn new tasks linear combinations of old tasks' LoRA modules, and mixtures of old tasks' LoRAs
In this paper we propose to tackle the cross-task PEFT knowledge transfer problem from the perspec- tive of Bayesian meta-learning, Meta-learning (Hospedales et al. 2021) aims for "learning how to learn" by jointly learning multiple-source tasks, and extracting some knowledge of "how to learn" that enables improved learning of the target tasks. Bayesian meta-learners extract such task-agnostic knowledge in the form of Bayesian priors. Rather than independently learning each source tasks'
PEFT module and recombining them for new target task (Huang et al. 2023 Zhao et al. 2024b),

---

Published as a conference paper at ICLR 2025
jointly learn the source tasks with hierarchical Bayesian model, and extract prior over PEFT's to improve target task learning. More specifically, we learn a mixture prior over PEFT modules, which can be considered This results in a more compact set of transferred knowledge compared to the exhaustive library-based methods. Meanwhile, each target task is effectively regularised by the most relevant latent tasks, which effectively boosts target task performance while combating overfitting.
extracting small set of prototypical latent task templates from the source tasks.
Our approach, termed "Learning to Fine-Tune" (LiFT) can be efficiently implemented using the SGLD-Gibbs sampler for both source and target task learning, making it both theoretically elegant and practically effective. The novel SGLD- DiGib sampling algorithm is one of our main technical contributions, which is computationally efficient, making the posterior inference feasible with many training tasks. Another technical contribution is that we propose an online EM algorithm that efficiently maintains a Gaussian mixture representation of the posterior samples from the SGLD- Gibbs posterior in an online manner in the course of iterative posterior sampling.
On range of NLP and Vision tasks, our experimental results show that LiFT outperforms both classic meta-learning methods (MAML (Finn et al. 2017a), Reptile (Nichol et al. 2018), etc.) and previous Bayesian meta-learning algorithms (e.g., BMAML (Yoon et al. 2018) and ABML (Ravi & Beatson 2019a)) by large margin. Compared to the latter, we have also observed improved uncertainty quantification thanks to the hierarchical Bayesian nature of the proposed method. Our meta learning approach also exhibits comparable or often superior performance to recent library-based approaches, including LoRA-Retriever (Zhao et al., (024b) and the model-based PEFT clustering method (Ostapenko et al. 2024) on cross-task benchmarks.
2
METHODOLOGGY
2.1 PROBLEM SETUP
We have many training tasks T, .2...., assume availability of a strong pre-trained deep (foundation) model parameters 6 which is powerful and useful for novel downstream tasks. We follow the parameter-efficient fine-tur fine-tuning (PEFT) strategy: freezing 0 and introducing/learming the extra adapter modules (e.g., LoRA (Hu et al. 2022 ). The parameters of these extra modules for the task Ti are denoted by Of. That is. (0% 00) forms model for the task T/ where only 8 is to be learned with D,, and 6 frozen. Although our approach is not restricted the type of the PEFT adapter architecture employed, our experiments focus on the LoRA PEFT for its popularity and strong downstream performance. At the downstream test time, we are given training examples D. sampled from an unseen task T. so that we can do test-time adaptation/fine of the meta-trained model with D..
.., TN where each task T, only reveals training data D, ~T> TE We
2.2 A HIERARCHICAL BAYESIAN MODEL FOR META-PEFT
In our Bayesian model design, we assume that each task (data) Tr (D:) is generated/govemmed by the random variables 0, def
Recall that 0f is the LoRA parameters (random variables) specific to the task i, and 80 is the base model parameters (constant). We introduce higher-level random variable & that regularizes all task-wisc variables (0#) Ji=1 (N is the number of training tasks). We can think of & as some core LoRA parameter values that may be useful for all different types of tasks (shared across tasks). Also, since a random variable, it can take multiple different values. presumably one of the several key prototypes or representative LoRA values a posteriori (i.e., if conditioned on the meta training set). See Fig. for the graphical models.
Assuming i.i.d. generation of 0f given d, the prior distribution becomes:
N
p ( = p(o) II ((0))
We specifically consider the Gaussian priors: p(0 (6) is assumed to be centered at Φ. and p(Ø) to be O-centered, considering that 0 LoRA modules essentially amounts to the pre-trained model itself with no adapter module added. That is,
p()) = A1(600
p(0 0,
(2)
2

---

Published as a conference paper at ICLR 2025
(b)
Figure 1: Our graphical models for (a) meta training and (b) meta test. Gray- nodes (data) indicate evidences, wrange/red-filles nodes are latent random variables, and the blue node (base model parameters) is constant. In meta test (b), we have the latent LoRA variable 8 associated with the observed test data D, (indicated by red).
where o² and 3 are fixed (user-chosen) variances.
For the likelihood, each LoRA parameters 8f, together with the frozen pre- pre-trained parameters 8 is associated with the data D, through the underlying loss function (6,,, 80; D,). More specifically,
(() x exp(-I((0", 80: D:))
Note that the loss function 4 can be task specific (e.g., in the NLP cases, I can be the cross-entropy loss for classification tasks, while can be the next token prediction cross cross-entrppy loss for the text generation tasks such as text summarization).
2.3 POSTERIOR INFERENCE BY SGLD-GIBBS SAMPLING
From our model definition, the posterior distribution becomes:
p(d (1 x p()) II P(00))))(
Among several options for the approximate posterior inference (e.g., variational inference, MCMC), we adopt the stochastic-gradient MCMC sampling (Welling & Teh (20IT), and combine it with the Gibbs sampling (Gelfand & Smith 1990; MacKay 2003) for the computational efficiency, which
will be clear in our explanation below.
First, applying the stochastic- stochastic-gradient Langevin dynamics (SGLD) sampler, leads to the following state transition where the state here being comprised of all random variables á and { 1m
10,0 ... 0% 10, 0f
0%
" "V (o logp(c, 2
(5)
where is the step size, NO0, I), and the derivative is computed with respect to and all ( main benefit of adopting SGLD that we can avoid the difficult normalizing constant in (4) since the derivative of the normalizing constant with respect to (Ø, {09 ) in (5) becomes 0. This simplifies the SGLD recurrence as:
/==1
N (o, 09 ... ... ( 01 ... 0% + " log Σ
(1ogp((1l0) + +V (6)
This can be written equivalently as:
N
log pl (09 Φ + Φ + 11 V. 1ogg(0)+ Σ
+ 77
2
0: - 0: + Vn. logg(00 +og Ho
1i= N)
(8)
According to the SGLD theorem WWlling & Teh jtt al.] 2021). under mild conditions, the recurrence (6) is guaranteed to converge to the samples from the posterior p(ó, ( (
2011
Raginsky al. 2017 Xu al. 2018 Zou
However, running each recurrence step is computationally very demanding since one has to retain the states and gradients for all [ y in the (GPU) memory even if each adapter module 0f takes up
3

---

Published as a conference paper at ICLR 2025
relatively small memory footprint. To avoid this issue, we propose a novel SGLD- sampling which combines the Gibbs sampler (Gelfand & Smith 1990 MacKay 2003) and SGLD sampling.
SGLD-Gibbs Sampling, Following the Gibbs sampling practice in MCMC, chain by randomly picking a subset of the state at each step, and do MCMC progression. As long as all variables in the state are visited sufficiently many times and frequently, it is known that the new chain also converges to the target posterior distribution (Gelfand & Smith 1990). In our case, we pick single 0g in each SGLD step 6 sequentially in the round-robin manner. Since is the ultimate shared information we aim to learn/infer during meta training, we always choose e in the Gibbs sampling step, which turns out to lead to faster convergence practice. Another computational bottleneck is the sum of the gradients V Σ, p(@, (o) over all i = 1,..., .N for each & update in 77) To remedy this issue, we introduce an auxiliary variable J that maintains this sum of the gradients, and we let it asynchronously updated (Eq. (II): at each iteration for task i, subtracting the old gradient for from J to approximate the cavity sum E of V p(0g (o) and adding a new (6) to J. More specifically, each SGLD- Debbs step consists of:
reform the
gradient V log (()
+ + V p(d) + log log log p( (D)18)))
? N
+ VT-zo
(9)
0. + 0. +
11
V
vn-200
(10)
2
J + J - V log p( (old) (6) + V s log p(6, (new) (6)
(11)
for i = N, .............. N....
The variable J maintains V Σ
= log p(p (6) in approxi-
mately, which remains asynchronously up-to-date due to [1].. We initialize J to O. The justificatior
of the SGL.D-Gibbs sampling empirically verified on some toy experiment in Appendix A where we show the convergence of the proposed SGLD-Gibbs sampling to the true posterior.
As will be seen in Sec. 2.4 and Fig. b)), we only need the posterior of the higher-level p(o) ( (
during the downstream meta test stage. Hence, we only maintain the posterior samples (()(m) LM from the SGLD-Gibbs recurrence I after some burn-in period. The next issue is how we represent the posterior samples succinetly. One popular choice is the Gaussian approximation (i.c., summarizing { Jmm by their first- and second-order moments). However, we aim enrich by mixture of Gaussians to better approximate the true posterior that is inherently multi-modal distribution. In the implementation, we save the latest parameter samples 0%s either the GPU memory if space is available in the disk otherwise. In the latter case load from the disk the latest 6% for the current Gibbs subset i in the SGLD-Gibbs iteration.
m==1
Online-EM Mixture for Posterior Approximation. A main challenge in representing the M pos- terior samples (m(m) } M 1 as a mixture of Gaussians (e.g.. via the famous Expectation- (EM) algorithm (Dempster et al.) 1977)), is that we may not store all those M samples in one place in batch due to the memory overhead As a remedy, we propose novel online-EM mixture maintenance algorithm We only access of ((()) one at time without revisit, and update the mixture in an online fashion. We aim to approximate the posterior by an order- K Gaussian mixture:
((o R Σ NajN(ouug.E) 5=1
(12)
Upon observing a posterior sample (((m), our online EM algorithm updates the parameters of the mixture by the following equations (Detailed derivations can be found in Appendix B):
(E-step) Compute the following component assignment probabilities the current mixture:
(o(V( M4 Σn) fj Σx= (olm))) py, Ex))
= Ij K)
(13)
'Our SGLD-Gibbs without the asynchronous update with J. has a transition kernel in MCMC defined to be a composition of the SGLD kernel and the Gibhs kemel that are hoth known to converge to the target distribution. Hence SGLDDGibbs should converge to the target distribution in thcory.
For instance, the number posterior samples (M) can be as large as the number of whole training steps.
"There exist several onlinc/sequential FM algorithms that exploit the recursive structure of the EM in manners similar ours (Appendix for details). These prior methods may be employed in place of our proposed onc.
4

---

Published as a conference paper at ICLR 2025
Online EM Learning Curve
7.5
5.0
-10-
loge bestrood (numning
2.5
-20
0.0
2.5
5.0
Samples
Figure 2: (Toy experiment for online EM) (Left) Data generated from true mixture distribution.
(Right) Online EM learning curve.
(M-step) Update the mixture parameters as follows (for jo ,.....; K):
1- E..nnn ny Py Rg + tq
My 74
S + ngS, tq (m))(m))mm ((()T
0j
= S, S----p
11 +
(followed by the update: n2 n, + 93)
(14)
One can interpret the formulation above as one EM step applied the current MoG estimate. Due to the high dimensionality of we often restrict E (and S) to be diagonal spherical in practice. Also, the initial {n, } can seen as the prior strength in the mixture estimation, and we find that initial n, = 100 often works well in practice.
To justify our online mixture estimation method, we have conducted a proof-of-concept toy ex- periment. For the data we devise true mixture distribution with three Gaussian components in 2D. From this true model, we generate 3000 samples (1000 samples per component) as shown in Fig. L[ftt). The offline EM which accesses all 3000 samples at once converges to its (local) minimum (data log-likelihood objective) -6.15 after 16 EM iterations (i.e., 16 epochs). Our online EM algorithm converges even faster to its minimum objective -6.07 (the running averaged objective -6.17) after seeing only about 2000 samples, which corresponds to 2/3 epoch. The results are shown in Fig. Right). We also compute the KL divergences between the true mixture and the EM-learned mixtures: (L.(""=|[pp" [[polline) 0.0781 while KL(p"" | " [ponline) 0.0686. This signifies the effectiveness of the proposed online EM algorithm.
2.4 META TEST AS BAYESIAN INFERENCE
At test time, for the given training data D. from the unseen test task T., we would ultimately like to infer the posterior distribution of the LoRA module for the target task, denoted by 0%. That is, p(0") {D+ J.W. where the evidence now encompasses all the training data {D. and the test
data D.. This posterior can be derived as follows:
/ pl00e, D., (D:) dø
D.. (D.) p(0,, D.ID..
(15)
-f ((D) 10 ( do = P(D.109) BECC
(16)
From 15) to 16). use the conditional independence from model (Fig. [b)), namely (
i= jo. that is, 0r and {D:} 1 are conditionally independent given Φ. Using our Gaussian
mixture approximation of ( 1 in (2). the expectation in (16) can be written as:
K
K
E SCOKDAAN))P ~ Σ a (N x)1 = Σ
(17)
where in the latter equality of 17 we use the Gaussian identity with our Gaussian prior p(@.(o)
(000; &, B²T) from (2).
Since we have derived the posterior p(02) D., (D.) up to constant, we can take samples from by running the SGLD sampler. (Note that this is test-time SGLD sampling apart from the one for training.) More specifically, we repeat the following recurrence:
K
8% t- 8 + 2
Voz logg((D(00.) + log 1,000,,,,,,,,,88) + v
(18)
j=1
5

---

Published as a conference paper at ICLR 2025
Intuitively, each step of (88) can be an SGD step (with the added SGLD noise) for the penalized objective function: ! loss on D2 + ② penalty of deviating from one of the ssale-adjusted)
protorypes {1z) } scaled by { where these prototypes were learned from meta training.
RELATED WORK
General multi-task meta learning. The gradient-based adaptation technique MAMI. (Finn et al. 2017b) is recognized one of the most popular episodic meta learning algorithms due to its empirical success and known theoretical support (Chen & Chen (022). Several variants have been proposed to enhance MAML'S performance well as addressing its computational overhead, including the L2
regularization technique (Rajeswaran et al., 2019). the first-order approximation (Nichol et al. 2018 and probabilistic extension (Finn et al. 2018 Ravi & Beatson 2019b Nguyen et al.) 2020) are highly related to our work, however, they are weak in several aspects. For instance, BMAML (Yoon al., 2018) not a hierarchical Bayesian model, but turns MAML's gradient-based deterministic adaptation steps to stochastic counterpart using the particle- based stochastic ensemble-bare based adaptation steps. ABML (Ravi & Beatson 2019a) relies on the variational inference, and resorts to a Gaussian posterior approximation.
2018). Bayesian extensions MAML (Finn al. 2018 Yoon
PEFT/LoRA adapter mixture/ensemble techniques. Especially in the NLP domain with emphasis on cross- cross-task adaptation (Ye et al. 2021), large number of techniques have been proposed. Consid- ering the large scale of the backbone networks (e.g., LLMs), many recent approaches aim to focus on adapting relatively smaller PEFT modules instead of the full backbone adaptation in Ye et al. (202T). As it is overwhelming to carry out all of the literature reviews in this paper, we just highlight here few works that the most relevant to our method, including Meta-ICI. (Min al. 2022). mixtures-of-LoRR (Feng et al. 2024 Asadi et al. 2024), LoRA Retriever (Zhao et al. (0224)). and model-based clustering method (Ostapenko al.
2024) We leave the discussions on these works
and relation to our approach in Appendix D
Modulation-baser meta learning & Online EM algorithms. There have been several works that consider meta- meta-learning by updating relatively few parameters (modulation). There have been several prior works on the online EM methods the literature. Although we find that none of them are not identical to the one proposed in this paper, most are similar in nature to ours in that the recursive structure of the EM is exploited. The discussions on these works can be found in Appendix D
EXPERIMENTS
We test our LiFT algorithm three benchmark datasets from NLP and vision for the cross-task PEFT transfer learning problem: i) Cross-Fit text-to-text generation NLP problem; ii) VTAB image classficcatoon/preditiion vision problem; and iii) Shakespeare next word prediction NLP problem. The details of the experimental settings are discussed in the subsequent sections, Sec. 4.1 Sec. 4.2 and Sec.
4.3
We also conduct uncertainty quantification in Sec. 4.4 and ablation study Sec. 4.5
4.1 CROSS-FIT NLP BENCHMARK
Following the seminal work from Ye et (021 on the cross-task NLP benchmark, we mostly follow their experimental design and task grouping ontology. There are 160 NLP tasks") 41 in the text-to-text format. Each task consists of train/dev/tsst sets, where train and dev datasets contain 32 examples, and test datasets have the rest. Inspired by the task ontology introduced in (Ye et al.) 2021), we consider 7 task splits for cross-task meta learning: CLS-45, CLS-23, CLS-0, Mult MMwlillle-hhhaeeee plee-hooiiieeeeee ce-O/ A. MRC-Q/A, A, ILT and Pa raapraasii ng. The first three contains all classification tasks for meta test where the meta train set consists of either all classification tasks (CI.5-45,, all non-classification tasks O), or half-and- -hlff (CLS- - 23). For the last four splits, we use the held-out tasks as a meta train set, e.g., in MCQA, all Mon- tasks are used in the meta training stage, while all MCQA tasks form the meta task set. Please Fig. 5 in Appendix for details of the task splits.
'Since some datasets are outdated or license protected, not all tasks were available at the time of our experimental study. There 23 tasks failed to download, and only manage 137 tasks in this study.

---

Published as a conference paper at ICLR 2025
Due to the diversity of the tasks, the related metrics are different (e.g., accuracy, F1 score, ROUGE-L, or EM scores). Even though a heuristic solution of reporting the average relative gain (ARG) was introduced in (Ye et al. 202I). we find that this metric is sensitive to the score ranges, and biased to those tasks for which the baseline model has near-O) scores. Hence we stick to the simple average of the metries over all test tasks regardless of the metric types.
The results are summarized in Table especially our proposed LiFT at the last rows against the first group of competing methods titled "Baselines / Meta learners". We use the BART-base encoder- decoder model (Lewis et al., 2020) as the main backbone with rank-4 LoRA for all competing methods. In our LiFT we vary the posterior mixture order K K= 1, 3, 5. See Fig. for the impact of the mixture order compared to the competing counterparts. The two baselines are: "No Meta Train" that simply ignores the meta train set, and only fine-tune the LoRAs on the training set of each meta test task; "Union Train" is a standard multi-task baseline where it combines all meta training tasks into a big single union dataset, and trains the LoRAs on the union set. At meta test time it performs PEFT starting from the trained model as initial model. The four meta learners (MAML (Finn et al. 2017b), First-Order-MAML, i-MAML (Rajeswaran et al. 2019) and Reptile (Nichol et al. 2018)) the most popular meta learning algorithms, for instance, MAML learns the best initial model with a one-step inner loop iteration in its roverse-mode hyper- arper-gradient computation with the meta train tasks. It evident that our LiFT significantly outperforms all these baselines and meta learners.
Comparison with Bayesian Meta Learning Methods. In the second group of Table T titled "Bayesian", we compare previous Bayesian meta learning algorithms including Bayesian MAML (BMAML) (Yoon et al. 2018) and Amortized Bayesian Meta Learning (ABML) (Ravi & Beatson 2019ab with our LiFT. Unlike our method that is based on the mixture posterior estimate, ABML's Gaussian posterior and the ensemble-based (with M particles) BMAML fall short.
Comparison with Mixture of MAMLs. mixture version of MAMLS amounts to maintaining a set of K initial parameters to be optimized, instead of single initial parameters in the original MAML. For the iMAML mixture, we maintain the K L2 regularizers in the inner optimization. The main difference of our LiFT from iMAML/MAML mixtures is that we use the Bayesian inference online EM) and testing (SGLD), while the latter approaches use numer-
for training ISGLDD-Gibe ically less stable implicit function theorem (Lorraine et al. 2020) or roverse-mode differentiation. Experimental results in the third group of Table (titled "Mixtures") support our claim.
Comparison to LoRA Mixture Strategies. Several recent multi-task LoRA mixture methods adopt the strategy of: i) task-wise LoRA training to form a LoRA zoo; and ii) at test time either use routing mechanism (MoE) or mixing the LoRA parameters, either all LoRAs in the zoo or selectively most relevant ones. LoRA Retriever (Zhao et al., 2024a) selects top- K relevant LoRAs using the cosine similarity, and combine those K LoRAs by either mixture or fusion operations. Another recent strategy is the model-based clustering (MBC-ju) (Ostapenko et al. 2024), where they perform the k-means clustering on the task-wise trained LoRAs (after some PCA dimensionality reduction). In the fourth group of Table titlled "LoRA Mixtures", we contrast with the performance of these methods. They usually underperform our LiFT on this cross-task dataset/experiment.
4.2 VTAB VISION BENCHMARK
Next we test our LiFT on the cross-task vision problem formed using the VTAB-IK benchmark (Zhai et al. 2019), which is comprised of 19 different image datasets for image classification and prediction. These datasets exhibit highly diverse domains/appettscondiiion including: different image acquisi- tion (e.g., from standard imaging to special purpose remote sensing or medical imaging), different objects/concepts (c.g.. generic. fine- -raignd or abstract concepts). and different prediction types (object recognition, counting, or pose estimation). Each of the 19 datasets consists of K training examples, and we use the conventional splits of train 80% and validation 20%.
To form cross-task problem for testing our meta fine-tuning model, we partition the 19 datasets into three groups along their domains. They are: natural (7 datasets), special (4), and st struttured (8). For instance, "Cifar100" and "Caltech101" belong to the natural group since their images are from natural scenes, while "Clevr-count" and "Dsprites-loc" are grouped in the structured group as they mainly contain shaped/strrutured objects. We then construct train/test task splits for each group (c.g., natural) by having datasets that do not belong to the group set as train tasks and those in the group as test tasks. The splits of tasks are shown at the head of Table
7

---

Published as a conference paper at ICLR 2025
Table 1: Cross-Fit cross-task transfer learning results with different task splits. For each task split, we report the metric scores averaged over test tasks (1). See texts for details.
Classification
Non-classification
Avg
CLS-45
CLS-23
CLS-0
MCQA
MRC
NLI
Para.
Baselines Meta learners
No Meta Train
57.34
57.34
57.34
28.61
21.87
53.08
51.49
46.72
Union Train
57.25
53.59
50.87
28.55
26.21
52.84
53.13
46.06
MAML
52.68
58.60
52.73
29.38
25.13
54.08
55.30
46.84
FO-MAML
59.38
55.25
51.84
25.70
27.21
54.90
51.20
46.50
i-MAML
59.61
57.91
58.51
28.53
26.
49.79
55.65
48.02
Reptile
57.59
57.84
54.62
29.03
25.92
55.07
55.42
47.93
Bayesian
BMAMI.
= 5
56.89
57.78
53.30
30.47
27.87
54,45
54.45
47.89
ABML
56.02
55.71
50.95
30.33
28.44
53.49
55.99
47.28
Mixures
MAML- Mix
K-5 5
54.02
60.13
54.16
29.97
26.57
55.08
55.97
47.99
i-MAML-MiX
=5
60.13
59.36
59.24
29.03
26.47
53.48
56.90
49.23
Retriever
K=
55.08
53.04
54.09
28.31
21,31
51.67
52.51
45.14
LoRA Mixtures
Mixture
==all
51.68
54,57
55.32
27.92
21.33
51.68
53.03
45.08
Retriever
K K
57.91
54.93
55.51
28.31
21,43
51.54
54.72
46.34
Fusion
K =all
55,30
56.01
54.74
28.36
21.42
51.08
$3.85
45.82
K K=
55.03
55.25
55.44
28.30
21.36
52.10
53.79
45.90
MBC-
K 10
55.39
55.95
57.13
28.40
21.39
51.36
52.05
45.95
Oues
LiFT
=
64.12
63.87
61.90
30.37
28.69
56.93
57.39
51.90
CLS-45
CLS-23
CLS-0
MCQA
50
60
Score
55
Model order
Mudel order
Mode order
Model order
MRC
MANIL Mix i-MAMLLMXX
Score
Score
Score
25.0
22.5
50
RetrieverFusion
LIFT (Ours)
Model
Model
Mode:
Figure Impact of the model orders on the Cross-Fi benchmark. We vary the model order K or M to 1, 3, 5 in the X-axis. It is the mixture order in our LiFT, MAML-MiX, and i-MAML-Mix; the number of particles in BMAML; the number of relevant LoRAs in Retriever-Mixture and Retriever-Fusion.
The results summarized in Table For all competing methods, we the ViT-B/16 model (Doso- vitskiy et al., (022)) pre-trained with ImageNet (Deng et al., 22009). For the PEFT module, the rank-4 LoRÅ adapters added the VIT where we put adapters the Q/K/V embedding matrices as well as the output matrices. For our LiFT, we vary the posterior mixture order K = 1, 2, 3. For the compet- ing meta learning methods (MAML and its variants), we use a single inner step for computational tractability. The hyperparameters and experimental details can be found in Appendix C The result indicates that our method, across all task groups, consistently outperforms the competing meta learn- ing methods, the multi-task baselines, and the ProtoNet (Snell et al. (017) and its memory-efficient full-batch version ProtoNet-LITE (Bronskill et al. 2021). ). This signifies the efficacy of the proposed hierarchical Bayesian meta PEFT algorithm on the vision cross-task transfer learning setting.
4.3 SHAKESPEARE DATA NEXT WORD PREDICTION
From the LEAF benchmark Caldas et al. 2019), we collect Shakespeare play lines where we treat each character personalized task. Then the lines played by each character are regarded data for
8

---

Published as a conference paper at ICLR 2025
Table VTAB-IK results.
Non-natura Natural
Non-special Special
Non-structured Structured
SNORS Azim
Clfar 100
Callech101
DID
Flowert
Pass
SWHN
Sum3997
AVERAGE
Camelyon
EuroAA
Rosiee4
Retinoputhy
AVERAGE
Cleve-Comnt Con-CConn
Cleve Dist
DMLab
KITTI
d5pr- Loc
d5 ON
SNORB-BR Ele
AVERAGE
Method
No Meta Train
63.0 91.1
67.8
98.5
90.3
83.9
51.4
78.0
84.5
94.6
83.8
74.4
84.3
70.0
66.2
45.0
748
71.5
46.7
25.7
35.2
54.4
Union Tmin
65.3 92.0
68.7
99.4
90.2
83.9
53.8
79.0
84.0
95.0
84.3
74.9
84.5
71.4
67.5
45.8
75.4
72.2
46.7
25.6
36.7
53.2
MAML.
64.1 92.0
69.7
99.4
90.7
84.3
56.9
79.6
84.9
94.6
86.0
74.9
85.1
69.7
68.4
46.7
75.5
74.5
47.1
25.6
36.8
53.5
FO-MAML
64.0 92.0 69.4
99.4
90.7
84.7 53.4
79.1
84.9
95.0
84,3
74.9
84.8
69.5
69.2
46.7
75.5
73.6
46.7
25.3
36.7
53.4
1-MAML
64.6 91.5 69.5
99.4 90.9
84.4 53.8
79.2
85.8
94.6
84.6
74.9
85.0
72.2
68.8
46.4
75.5
75.3
46.7
25.6
36.6
53.9
Reptile
64.2 92.0 69.2
99.4
90.7
84.8
54.0
79.2
82.7
94.6
84.8
74.9
84.2
72.5
68.9
45.6
76.1
72.6
46.2
25.6
35.8
53.4
ProtoNet
66.5 92.0
69.7
99.4
91.3
85.2
54.7
79.8
84.0
93.7
86.0
73.5
84.3
71.6
64.8
45.8
76.1
70.8
46.3
27.7
38.6
53.2
ProtoNet-L.TT
64.5 91.5
68.3
99.4
91.9
84.8
53.8
79.2
81.8
94.6
86.3
74.9
84.4
71.2
64.8
42.0
78.4
69.0
46.3
27.5
42,9
55.3
K-
66.9
54.9
79.7
LIFT
K =2
67.1
92.9
70.6
99,4
91.9
87.2
54.8
80.6
85.1
95.0
86.4
76.3
85.7
73.8
70.0
50.4
80.2
72.5
48.1
30.8
46,9
59.1
67.4
92.9
70.6
99.4
91.9
87.1
$4.8
80.6
85.5
95.9
86.4
77.6
86.3
77.4
70.6
51.0
79.6
74.7
48.5
30.9
46.4
59.9
Table 3: Shakexpeare dataset with play characters tasks. Next word prediction perplexity (.).
Random task split
Play-wise task split
Avg
tr/va/te-6010/30
tr/va/e=:0/10010
No Meta Trin
12.14
12.12
12.15
12.09
12.13
Union Train
11,30
11.22
11.35
11.19
11,27
MAML
11.87
11.71
11.87
11.68
11.78
FO-MAML
11.80
11,55
11.80
11.50
11.66
i-MAML Reptile
12.25
12.87
12.31
12.19
12.41
11.31
11.24
11.33
11.22
11.28
K K=
11.01
10.97
11.08
11.00
11.02
LIFT
K
11.02
10.98
11.08
11.00
11.02
K 3
11.02
10.97
11.09
10.99
11.02
the task. We deal with 152 tasks/characters from 36 plays that contain a considerably large amount of text lines, and the goal predict the next word given the current and all previous tokens. For the task split, we consider both random split and play-wise split where the latter ensures that no play is overlapped between meta train and tasks. For each split, we randomly have 10%%%%%%3%% and 10%/10%%/100 train/dev/test task split. We use the BART-base and rank-4 LoRA as the backbone and PEFT adapters. Tablee]ssoww perplexity scores averaged over test tasks. Overall all methods yield high perplexity implying that next word prediction is a challenging prediction task. However, our LiFT consistently outperforms the multi-task baselines and the meta learning algorithms.
4.4 UNCERTAINTY QUANTIFICATION
As Bayesian approach, model can capture uncertainty effectively, and lead to better error calibration than deterministic methods. The error calibration metrics measure how well the prediction accuracy and the prediction confidence are aligned. For instance, the ECE (Guo et al. (017) is defined
as: ECE D Nr lace(b)- - con where we partition test instances into B bins along the model's prediction confidence scores, and cb)) acc(b) are the average confidence and accuracy for the b-th bin, respectively. Another metric widely used is the test negative log-likelihood (NI.L.). For further details, please refer to (Kim & Hospedales, 2023).
N
On the Cross-fit benchmark, we perform uncertainty quantification on two test tasks for the CLS-45 meta-trained models. Our LiFT is compared to the (deterministic) MAML and two Bayesian meta learning methods: B.MAML (Yonn et al. 2018) and ABML (Ravi & Beatson 2019a). In BMAML, vary the number of samples M (called particles) in their stochastic ensemble adaptation steps. The results Table 3 show that our hierarchical Bayesian method leads to considerably better enror calibration than previous Bayesian meta learning algorithms both ECE and NLL metrics.

---

Published as a conference paper at ICLR 2025
Table 4: Uncertainty quantification on two test tasks for CLS-45 incta-trained models.
ethos-race
superglue-eb
ECE (L)
(L)
BCE ())
NLL (4)
MAML
0.1797
0.7077
0.1679
0.7485
M
0.1727
0.6895
0.1644
0,8406
M =3
0.1507
0.6582
0,1555
0.8069
BMAML
M =
0.1147
0.6423
0.1435
0,7825
ABML
0.1238
0.6775
0.1572
K
0.0900
0.3330
0,1269
0,7118
LIFT (Ours)
K 3
0.0701
0.3021
0.1368
0.7721
K 55
0.0937
0.3986
0.1186
0.6680
Table 5: Ablation study on the Cross-Fit benchmark. From our model, we remove either the mixture deviation regularization term at test time (denoted by "Reg (X)"), the SGLD noise term ("Noise (X)"). Our method modified to handle task-wise pre-fine-tune tuned models is denoted by "Fixed S"
CLS-45
CLS-23
CLS-0
MCQA
MRC
NLI
Para.
K
Reg (X)
60.1 62.6 63.4 99.9 61.5 61.9 58.7 59.6 60.1 28.7 28.9 28.9 24.7 25.0 26.1 55.5 55.6 55.6 52.6 54.4 55.9
Noise (X) 61.0 61.8 63.6 60.9 61.9 62.7 59.4 60.3 61.0 29.1 29.1 29.4 25.5 25.7 26.8 56.0 56.2 56.5 55.5
55.8 56.9
Fixed
0fs
61.7 62.9 63.4 61.9 63.0 62.6 61.3 61.8 8 61.1 28.9 29.4 29.3 26.4 28.0 28.3 56.2 55,9 56.7 55.9 56.1 56.4
Ours
63.0 63.4 64.1 60.9 62.0 63.9 60.0 61.0 61.9 29.7 30.1 30.4 27.3 27.6 28.7 56.9 56.3 56.9 56.6 56.8 57.4
4.5
ABLATION STUDY
To see the impact of the mixture deviation regularization at test time (i.e., the mwi-tmr--eeessiiie term in 18)) in our method, we conduct some ablation study, We also remove the noise term in the SGLD to see the impact of our stochastic method. Table 5 shows that these two components important for the best performance while the test-time mixture deviation penalty is more crucial.
Utilizing Task-wise Pre-fine-tuned Models. One drawback in approach is that require jointly learning all the tasks, which makes a bit more costly and cumbersome because one has to have all the tasks ready to train jointly, being unable to just use off-the-shelf LoRAs potentially pre-fine-tuned beforehand. However, we can modify our algorithm so as to utilize potentially pre- fine-tuned task-wise LoRAs. The modified scheme we take is follows: In 7
we replace the Jacobian term V Σ log ((f)(()) summed over all {0; J that are fixed and pre-fine-tuned by the Jacobian of one single i that is randomly sampled. Thus this can be seen as a noisy stochastic version of the SGL.D update for db Although one can use the original version 7 in principle, the noisy stochastic version is computationally and practically more attractive without needing to evaluate the Jacobian the likelihoods over all tasks per SGLD update. Our results with this version on Cross-Fir are summarized in Table 5 (titled "Fixed 5") For each task split, we pre-fine-tune task-wise LoRAs on meta meta-train train tasks, and fixed them in our method. We still get performance boost in this case, which input pre-fine cee-finettuu sune-tundd LoRA libraries rather than inputting sets of datasets.
is promising since
4.6 EXTRA EXPERIMENTAL RESULTS
Additional ablation studies and extra experimental results including LLMs with larger number of parameters (e.g., FLAN-T5-XL) can be found in Appendix E
CONCLUSION
In this paper we have proposed novel approach to parameter efficient meta fine-tuning, a principled Bayesian method to solve the Cearning-to-PEry problem. Our hierarchical Bayesian model, via reasonable and tractable approximate inference strategy of SGLD-Gibe and online EM, effectively represents the prior of the LoRA modules that capture the shared information across all training tasks. On a range of NLP and vision cross-task adaptation benchmarks, our model is shown to be superior to not only general meta learning algorithms but also recent LoRA mixing and routing approaches.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Nader Asadi, Mahdi Beitollahi, Yasser Khalil, Yinchuan Li, Guojun Zhang, and Xi Chen. Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy? arXiv preprint arXiv:2402.15414, 2024.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. CVPR, 2020.
Rishi Bommasani et al. On the opportunities and risks of foundation models. arXiv:2108.07258, 2021.
John Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, and Richard E. Turner. Memory efficient meta-lcarning with large images. NeurIPS, 2021.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konecny, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: A Benchmark for Federated Settings, 2019. Workshop on Federated Learning for Data Privacy and Confidentiality.
O. Cappé and E. Moulines. On-line expectationmmaximzaatio algorithm for latent data models. Journal of the Royal Statistical Sociery: Series B (Statistical Methodology), 71(3):593-6133 2009.
J. Chen, J. Zhu, Y. W. Teh, and T. Zhang. Stochastic expectation maximization with variance reduction. NeurIPS, 2018.
Lisha Chen and Tianyi Chen. Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?, 2022. AI and Statistics (AISTATS).
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetuned language models, 2022. URL https:://// /wt.ivivvvv.////aa//eee/aa.aae./ org /alggg//bsaa abs 2210.111111 11416
A. Dempster, Laird, and B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1-38, 1977.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2009.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-233) 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Rezende, and Dan Rosenbaum. From data to funeta: Your data point is function and you treat it like one. ICML, 2022a.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goliñski, Yee Whye Teh, and Arnaud Doucet. Coin++: Neural compression across modalities. TMLR, 2022b.
Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mfxt An Efficient Multitask Tuning for Large Language Models. arXiv preprint arXiv:2403.034:3 03333, 2024.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017a
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 20.7b.
11

---

Published as a conference paper at ICLR 2025
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic Model-Agnostic Meta-Leamnigg In Advances in Neural Information Processing Systems, 2018.
Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh. Bootstrapped meta-learning. ICLR, 2022.
Alan E. Gelfand and Adrian F. M. Smith. Sampling- Based Approaches to Calculating Marginal Densities. Journal of the American Statistical Association, 85(410):398 409, 1990.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. JCML,, 2017.
Yiran He and Hoi-To Wai. Online Inference for Mixture Model of Streaming Graph Signals with Non-White Excitation. In arXiv preprint arXiv:2207.14019, 2022. URL http://axxiv.oorg
abs 22002110000
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-Learning in Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10. (0.1109/PAMM222213307999999
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: https/////ppeeeeeeee.eoo wet.swn.e//eee. forum? id-nZevKeeFYf9
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307. 13269, 2023.
Belhal Karimi, Hoi-To Wai, Eric Moulines, and Mare Lavielle. On the global convergence of (fast) incremental expectation maximization methods. NeurlPS, 2019.
Minyoung Kim and Timothy Hospedales. BayesDLL: Bayesian Deep Learning Library. In arXiv 12928
preprint (tt::2309./19228 2023. URL./ttpp///rxxivorr///bbb
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising Sequence-to-Sqqeene Pre-training for Natural Language Generation, Translation, and Comprehension. In Association for Computational Linguistics, 2020.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In CVPR, 2024.
Z. Liu, J. Almhana, V. Choulakian, and R. McGorman. Online algorithm for mixture with application to internet traffic modeling. Computational Statistics Data Analysis, 50(4):0052- 1071, 2006.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing Millions of Hyperparameters by Implicit Differentiation. 2020. Al and Statistics (AISTATS).
David J. C. MacKay. Information Theory, Inference and Learning Algorithms. Cambridge University Press, 2003.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to Learn In Context. In NAACL, 2022.
R. M. Neal and G. E. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. In In Learning in graphical models, pages 355-368, Springer, 1998.
Cuong Nguyen, Thanh-Toon Do, and Gustavo Carneiro. Uncertainty in model-agnostic meta-learning using variational inference. In WACV, 2020.
Alex Nichol, Joshua Achiam, and John Schulman, On first-order meta-learning algorithms. In arXiv, 2018.
12

---

Published as a conference paper at ICLR 2025
Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni. Towards modular LLMs by building and reusing a library of LoRAs. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.). Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 38885-8899999 PMLR, 21-27 Jul 2024. URL http::tt/ppopeeeeeeeeeooeo
mlx /ttt:://ccoccc.........//222522.
html
Emre Özkan, Carsten Fritsche, and Fredrik Gustafsson. Online em algorithm for joint state and mixture measurement noise estimation. In 2012 15th International Conference on Information Fusion, pp. 1935-1940, 2012.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. AAAI, 2018.
M. Raginsky, A. Rakhlin, and M. Telgarsky. Non-convex learning via stochastic gradient Langevin dynamics: nonasymptotic analysis. In Conference on Learning Theory, 2017.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-Learning with Implicit Gradients. In NeurlPS, 2019.
S. Ravi and A. Beatson. Amortized bayesian meta-learning. ICLR, 2019a.
Sachin Ravi and Alex Beatson. Amortized Bayesian meta-learning, In International Conference on Learning Representations, 2019b.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner. Fast and Flexible Maulii-Task Classification Using Conditional Neural Adaptive Processes. In Advances in Neural Information Processing Systems, 2019.
Jonathan Richard Schwarz, Jihoon Tack, Yee Whye Tch, Jaeho Lee, and Jinwoo Shin. Modality- agnostic variational compression of implicit neural representations. ICML, 2023.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning, In NeurlPS, 2017.
Jihoon Tack, Jachyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Tch, and Jonathan Richard Schwarz. Online Adaptation of Language Models with a Memory of Amortized Contexts. In Advances in Neural Information Processing Systems, 2024.
Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In International Conference Machine Learning, 2011.
Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of Langevin dynamics based algorithms for nonconvex optimization. In Advances in Neural Information Processing Systems, 2018
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross-task generalization NLP. pp. 7163-7189, November 2021. doi: 10. 18653/v 1/2021 .ennl-mminn.n
URL hattps://act http///aaaannohoggogoo org. www.e-nwneeeeee main. 572
Yoon, T. Kim, O. Dia, Kim, Y. Bengio, and S. Ahn. Bayesian model-agnostic meta-learning. NeurIPS, 2018.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1rXiv:1910.0480 01.6774400,...1 2019.
Ziyu Zhao, Leilei Gan, Guoyin Wang. Wangchunshu Zhou, Hongxia Yang. Kun Kuang, and Fei Wu. LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild. In Findings of the Association for Computational Linguistics (ACI.), 2024a.
13

---

Published as a conference paper at ICLR 2025
Ziyu Zhao, Leilei Gan, Guoyin Wang. Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. arXiv preprint arXhv:2402.09997, 2024b.
Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. ICML, 2019.
Difan Zou, Pan Xu, and Quanquan Gu. Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Conean Concave Sampling. Uncertainty in AI, 2021.
14

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Toy Experiments with Our SGLD-Gibbs Algorithm (Sec.
Detailed Derivations for Our Online EM Algorithm (Sec.
Experimental Details (Sec.
Extended Related Work (Sec.
D
Extra Experimental Results (Sec.
A ToY EXPERIMENTS WITH OUR SGLD-GIBBS ALGORITHM
this section we empirically verify the convergence the proposed SGLD-Gibbs posterior sampling algorithm. To measure the divergence of the estimated posterior distribution from the true posterior, we devise true model that admits analytical posterior distribution. Specifically, the following true data synthesis process that generates N tasks. We consider the regression problem on the 2D input space.
Φ ~ N(0,021)
(19)
0: ④ ~ N(0:; for i= N
(20)
D. D-= (2]. v 3/
0 II Nuis 0, a2) for i 1, I...,,,
(21)
where (Ø) = dim = dim x) = and dy = 1. The number of samples per task set to 20. We vary the number of tasks N in the experiments below.
From the Gaussian identities, it is not difficult to see that the posterior distribution (D.) )) can be analytically derived as:
(D.)) K = (XX 25)-111
KX))
where
(22)
(23)
X [XT, XTJ
XN1 T X; x; [r
(24)
Y MY,, Y YNI, Y, = * ... v
(25)
Σ Diag( E1 Ez, ... ,N), E, = a²I + 6² X
(26)
We compare the original SGLD 6 and SGLD-Gibbs I While repeating the recurrences,
we collect the posterior samples, and build Gaussian estimates. We vary the number of tasks N: 10. 100, 200, and 1000, where N 1000 larger than those used in all our experiments. The KL divergences of these estimates from the true posterior (22) are shown in Fig. 4 The results indicate that our SGLD-Gibbe converges to the true posterior, and the convergence times are far less than x of the original SGLD convergence times.
B DETAILED DERIVATIONS FOR OUR ONLINE EM ALGORITHM
In this section we derive our online EM algorithm described in Eq. 13-14 in the main paper. The situation is that the samples {d(m) )m are seen one-by-one manner, that is, d(1) - r(2) + ..., and need to update mixture estimate for each incoming sample. For convenience in derivation, the samples the incoming order are denoted by æ - I2 - ... - F4-1 - It - ...,, and we let Ie be the current sample (instead of ((m)). Note that in online methods, we are not allowed to store (x [_.)
the previous samples
Before we proceed with deriving our online EM algorithm, we recapitulate the famous and classical offline EM algorithm (Dempster et al. 1977) in what follows.
15

---

Published as a conference paper at ICLR 2025
B.1 RECAP OF OFFLINE EM
By offline, it means that we are allowed to store and access all data samples {xi))= at once. The Gaussian mixture model is denoted by:
K p(:; 9) = (a.N(;
(27)
where θ = fa. μ4 Σ) denotes the parameters of the mixture. By Jensen's 's inequality, can be shown (e.g., (Dempster et al. 1977)) that the data log-likelihood is bounded below as follows:
Σ (ogp(z;;)) 2 EB log (Tx,A, + Σ (a)).
(28)
which holds for any distributions (i( over j E (1,..., KJ. Here H(-) is the entropy operator. Furthermore, the bound of (88) is tight (i.e., inequality becomes equality) when qi 's are defined as:
(B-step)
q:(i) ES ay
(29)
for all i = 1, 1,... N and j j 1,..., K. Hence this choice of 4:8 is the optimal choice under given O and the equation 29) constitutes the E-step. One can interpret q:()) as the current model O's belief in mixture component assignment: the probability of assigning 2; to the component j. With % s chosen and fixed as (29). we try to find the maximal value of the right hand side (28), that is,
N
max
Σ
E
jrqq
log (a,,(zz; _,
(30)
It is easy to see that the optimization which turns out to be (for j = 1,...,KK K):
30
admits closed-form solution (by setting the gradient to 0),
EN 9((
(M-step) aj
(31)
2/'=1 E1= ((')")
Pi i=1 (:(ixor:
(32)
Ei=1 9:(5)
Σ, + S - MOHn where S, = Lim1 9: (x)
(33)
D 9:(i)
The update equations
31 constitute the M-step.
B.2 DETAILED DERIVATIONS FOR OUR ONLINE EM
Now we consider the online scenario. By online, the samples are observed sequentially æ1 and at each time step we only see xe and not allowed to store the
+ It-1 + It +
previous samples 1-1
N
Niw 100
N 200
N 1000
SOLD-G boa SCLD
SOLD-GIAa
SOLD-GItE SOLD
22LD-3004
Diveunee
SCLO
BOOWEEE I1 Phoegeree
22LD
200
400
600
2053
2000 3000 4000
5000
8000
3000
20000
15000
20000
100000 200000 300003 40COCO
Nurbec Eouulrr
Narbo Louukaro
Figure 4: (Toy experiment for SGLD-Gibbs) KI. divergences between the true posterior and the estimated posteriors (SGLD vs. SGLD-Gibbs) measured several SGLD-Gibbs iteration numbers. Since we use the number of SGLD-Gibbs iterations for the X-axis, we match the result of the original SGLD by extrapolating its number of iterations (roughly, the original SGLD learning curve can be retrieved by taking the first one N-th of the X-axis).
16

---

Published as a conference paper at ICLR 2025
We denote our mixture model right before seeing 2, by:
p(a; 01-1) = Σ ((
(34)
where O,-1 = fa' , E 11 is our estimate of the mixture just before seeing 2t.
If pretend that we can access all the previous samples { together with the current one Iy. then following the offline E-step (29). our (fictitious) E-step would be as follows:
9 Eg o Mo for i == -.....,4
(35)
Now we define the following quantities (n,) as the current model O,-1² is component assignment accumulation for previous samples { 1!-1:
My Σ (: for I,...,, K.
(36)
We want to emphasize here that although {g: would not be available in the online scenario, the quantities {nz} suceinetly capture the key information about those previous samples, which is all we need during the M-step. More specifically,
re-write the M-step in (31 33 follows:
EL a()))
ny + 9)())
= ny + 9c(i)
(37)
Σ Ei 910) Σ/ (ng (ng + ((()))
1+ 25'=1 Rij E
13 + E q1()zz E/=l (1))) + gg(i)ax
ngpl + ((()xxr
(38)
El1900 (i(i)
n2 + (e(5)
M + 4((5)
83 + Li=1 de()),aa
E (())aaa) T + I
n 1 + q: q=(i)
(39)
EN (())
My 40
ny (c(i)
Note that the approximations in time L, respectively, that is,
38 and 39
originate from the M-step for and S at the previous
Σ 1
(i))
Σ E 9.00
S
(40)
(i)
E (()
K What remains is estimate {n in online fashion without explicitly computing 336 using
the previous samples {- Assuming that we have a good estimate of ny at time - 1, we can update n, for time t by: nj <- n, + qi (j) where ()()) would be computed similarly as (35) using θ₂- Although this can be done with the updated θ after the M-step (37 39,, we can reuse the current (i( to have approximate estimate, which rarely incurs significant difference in practice.
In summary, at each time step 1, we only need to compute {a:()) for the current sample &t (E-step), and the M-step 37 39) can be done without accessing the previous samples {-:)<]]- This completes the derivation for our online EM algorithm.
C EXPERIMENTAL DETAILS
All experiments are conducted on a single V100 GPU, For the scale hyperparameters in our LiFT model, we always use a = 0.01 (scale of prior p(0)) and 3 = 0.01 (scale of p(0,l))) For the pre-trained backbone, we use the pre-trained BART-base checkpoint from HuggingFace for Cross-Fit treiteed and Shakespeare datasets, while the preeanet-pre-rraieeed ViT-B/16 for the VTAB benchmark. The
SGLD-Gibbs algorithm takes the burn-in steps 2000 with warmup 1000 steps for the NLP datasets and 200 burn-in and 100 warmup for the vision. The learning rates are 10 for the NLP and 10-4 for the VTAB. We use the batch size 16 with 20K steps for the NLP tasks while we take batch size 128 and 10K steps for the vision task. For the meta learning baselines (e.g.. MAML, its variants and Reptile), we use similar hyperparameters, but the learning rates are adjusted for numerical stability. The inner loop leamning rates are typically chosen 5 times the outer learning rate.
17

---

Published as a conference paper at ICLR 2025
CLS-45
"train": ("ade v2-classification", "circa", "ethos-directed generalized", "ethos-disability"; "ethoos-gederr'
"ethos-sexual orientation", "slue-cola", "glue-mnli", "elue-mrpe", "glue-qnli", qnli", cqlue-qup", "elue-rte', "glue-sst2", "glue-wnli", "google_ wellformed_ "urrrr""d_uurr"",, mate_speech offensive", "hatexplain', "imdb", _peeech_ofennsive"
"kilt fever", "liar", "onestop english", "paws", "rotten tomatoes", "scicite", "scitail", "sick", "sms spam", "superglue-rte" "superglue-wic", "superglue-wsc", "trec", "trex-fnegraiieed" sevetteval-emopf sweet_eval-emodteee
@wwee__eval-irom (weet__evlloofensiie "tweet_eva oval stance_ cionnc_bbottio", "tweet_ee eval sttanc__climatee "tweet_eval seset_eval-staee stance_ (atnnce_llllry"]
Sweett_evl-seetiine
"dev": :
[" l'ag_nee ['ag_news", "climate_fevve feveee_eeer", "ethos-nationalloriii" "hate_speech18",
"tweet_eval eeaa--hat"",, "tweet_ewee eval-stance__atheimne
"test": ["anll", ethhos-race' "ethas-religion", "superglue-cb", "tab_fact", "wiki_ga", "yelp_polarity"]
CLS-30
"train": l"ade corpus. v2-dosage", ceamppellippis....mm n "blimp-senee sentential negation scope", "commonsense_ "crows_pairs" "duare", "ethos-disabiie" "ethos-sexual Lorientationn tahhosssrual_riinnaaiio""" "glue-cola", "glue-mnli", "glue-mrpe", "glue-qap", "glue-rte", "glue-wnli", "hatexplain" "hellaswag", "imdb", "kilt_zsree" zsre", "lama google_re", "lama-squar" "math_en qa"", "numer_ sense", "openbookqa", "paws", "piqa", "prato_es qa", "quartz-no, knowledge", "race-high", "ropes", "scicite", "sciq", "sick", "sms samss_spam", "superglue-rte", "superglue-wsc" "tweet_evats "vaeetto"alemmtiion",, sweeet_eal--fffesiive "tweet_eval-sentinee eeveeelvaaateehhiee
"dev": ["ag_ se__es"", "climate_fevee "ethos-national _rrggi", "hate_sper satt__spechh8,, "meddcal_questions_ pparss", apoem_sentiments "weet__val-hatte Twwee__evl-staree evallstance_aateissm"
atheism", Swweet_eal-stance_Pee
"test": ["anli", "ethos-race', "ethos--rliggonn,, "superglue-cb", "tab_fact", "wiki_qa" qa", "yelp_polarity")
CLS-O
"train": _oapu
corpus v2-dosage", "art", "biomre", "blimp-anaphor number agreement", "blimp-ellipsis n. bar. bar_2", "blimp-sentential negation _np. licensor prtrsorrpreeett,,, "blimp-sentential. negation npi scope".
"break QDMR high-level", -DMR-high--eeee"" "commonsense qa", "crows. pairs", "dream", "duorc", "eliS-asks", "eliS-eliS", "freebase_qa" "eigaword", "hellaswag", "hotpot_qa" "kilt akilt_ay2,, "kilt_ holpoooa,,,"" "kilt_trexx,, "kilt_zsre", "lama conceptnet", "lama-guogle_rt "lama-squad", "math_qa", "numer sense", "upenbookaa" "piqa", "proto_qa" "qa_st sri",, "quarel", "quartz no mne_knowedgee" "race "race-high", "reddit tif. tille, "reddit iifu-tldr", "ropes", "sciq", "social_i_qa", "spider", "superglue-muttire" "wiki_io"", "wikisal", "xsum", 'yelprreview_ffll
"dev": ["tweet eval-stance feminist", "ethos-national, origin", "tweet eval-hate", "ag_news",
"amazon hamazon_polarity" "hate speech18", "poem sentiment", "climate fever", "medical_ questions_palin patrrs,pa,," "tweet_evas eval-stance atteessm"
"test": ['superglue-cb" "dbpedia 14", "wiki qa", "emo", "velp polarity", "ethos--rligion", hiinancialphhassebrre "tat_fact", "anli", "ethos-race")
MCQA
"train": : ("adversarialqa", "baolq", "duorc", "eliS-askh", "eliS-asks", "eliS-eli5", "freebase_gg qa",, "hotpot_gg qa", "kilt, hotpotqa", "kilt "kilt_nq", " "kiit_trex"" "kilt, zisee, "lama-trex", 'm taco", "numer sense", "quoref", "ropes", "squad-no context", "squad-with context", "superglue-multirc" "superglue-recordd" "web questions"].
llama-google_re re", "lama-squad",
"dev": [ I'codah", ; "dream", "qasc", "race high"),
"test": I'ai2
["ai2_are", are", "aqua_rai "commonsense, qa", "casmos, qa", "hellaswag", "math_ga" gaah, "openbookqa", "quail", "quarel", senart-noo_knwwledd "quartz-with, knowledge", "race-middle", "sciq", "social_i_qa,, "superglue-copa" "swag", Miino_grande,, "wiqa"]
Figure 5: Cross-Fit task splits (Part I).
D EXTENDED RELATED WORK
General multi-task meta learning. The gradient- cradient-based adaptation technique MAML (Finn et al.) 2017bp is recognized as one the most popular episodic meta learning algorithms due to its empirical success and known theoretical support (Chen & Chen 2022 Several variants have been proposed to
18

---

Published as a conference paper at ICLR 2025
MRC
"train": ("ai2 arc", "aqua rat", "boolq", "codah", "commonsense qa", "cosmos qa", "dream", "eliS-askh", "elis-asks", "elis-eli5", "freebase_cc qa", "hellaswag", "kilt hotpotqa", "kilt_ne Lgn. "kilt_trex", "kilt_zsree, lamaagoogle__r
"lama conceptnet", "lama google_re "lama-squad" "amm-ttrx" "math qa", "m taco", "numer_seese",,
"openbookqa", "aas", "quail", "quarel",
"qurle-witt__koowlede""" "race-high", "race-middle", "oaadddle""sssce""" "sciq", "social_iqa"" "squad-no no_ceetxt_,,,, ""aouttt"",""""" "superglue-copa" "superglue multirr""""" "swag", webb_questions "wino_grande", "wiqa")
"dev": ["duore",
"test": ["adversarialos" "hotpot_gg qa", "quoref", "ropes", "superglue-record"]
NLI
"train": ["ade_
carrle_orrr___ve v2-classfficationn, "aa_news"" "dirca", "climate_fevee" "ethos-directed vs vs__eneraalizedd"
eethos-disabiitt"" "ethos-gender", "ethos-national origin", "ethos-raae", "ethos-religion,,
"ethos-sexual orientation", "glue-cola", "glue-mrpc" "slue-qap", "glue-sst2", "google wellformed_queett "hate seatt___peecha", "hate_ siree_speehh__eeeee offensive", "hatexplain", "imdb", , "kilt fever", "liar", "medical questions pairs", "onestop _nggishh"_"" "paws", "poem_ sentiment", "rotten tomatoes", "scicite", "sick", "sms spam", "superglue-wic", "superglue-wsc" "tab ftct',"ttre", frrecfinnegraied". "tweet_evake eval-emojj", "tweet_ews eval-emotion", tweet_eval-hare "tweet_evale eval-sent,ment", "tweet_eval-starre eval-ssancce_abueeeee abortion", "tweet_ eval-stance, aatrrrsm'hhii'',, "tweet eval stance climate", "tweet_eee eval "aance_femmnist" "tweet_ eval stance_ hiilary"""l "wiki_ga"" "yelp_polarity").
"tweet_eval-iront"
"dev": ["art", "glue-mmn mini", "superglue-rte",,
"test": ["anll", "glue-qnli", "glue-rte", "glue-wnli", "scitall", "sick", "superglue-cb"]
Paraphrasing
"train": ["ade corpus v2-classificationn, news", "anli", "circa", "climate fever", "ethos-directed vs generalized",, "ethos-disability"" "ethos-gender", "ethos-national origin", "ethos-race", "ethos--rligion", "ethos-sexus orientation", "Blue-cola", "glue-mnli", "glue-qnli", "glue-rte", "Blue-sst2", "glue-wnli", "google wellformed peelformed_quurry,, "hate_ tite_sspeech8",, "hate speech, offensive", "hatexplain", "imdb", "kilt fever", "liar", "onestop english", "poem__see sentimentt", "poem__sentmment,, "rotten_tone torientsttmttss",, "scicite", "scitail", "sick", " "sms_spam", "superglue-cb", "superglue-rte" "superglue-wie" "superglue-wse" "tab_fact" "trec", "trec-fnngraiined"
"tweet eval evaliemojj", "tweet_evv sveet__eal-emotion" liveet_eval-hee eval-hate", "tweet_evee seaeet_eallirroy"" "tweet_eval -eal-elllllllleeee-
"tweet sval-sentimmnt", "tweet eval-stance, abortion", "tweet eval-stance atheiss", vallstance_athhismm, "tweet eval-stance climate", "tweet eval-stance, feminist", "tweet eval-stance__hiee hiillarr""l""",, "wiki_qa", "yelp_polarity"].
"dev": [' q'gp",-qap",,
"test": I'glue-mrpc", Loaitiaa_quuettoss____e s_ttss_paarr____ "paws"]
Figure 6: Cross-Fit 7 task splits (Part II).
enhance MAML'S performance as well as addressing its computational overhead, including the I.2 regularization technique (Rajeswaran al. 2019), the first-order approximation (Nichol et 2018 and probabilistic extension (Finn et al. (2018). Bayesian extensions to MAML (Finn et al. 2018 Yoon al. 20188 Ravi & Beatson 2019b] Nguyen al. 2020) highly related our work, however, they are weak in several aspects. For instance, BMAML Yoon et al. 2018) is not hierarchical Bayesian model, but turns MAML's gradient-based deterministic adaptation steps stochastic counterpart using the particle-basee stochastic ensemble- based adaptation steps. ABMI (Ravi & Beatson 2019a) relies on the variational inference, and resorts to a Gaussian posterior approximation.
PEFT/LoRA adapter mixturre/nssemle techniques. Especially in the NLP domain with emphasis on cross-task adaptation (Ye et al. 2021), large number techniques have been proposed. Con- sidering the large scale of the backbone networks (e.g.. LLMs), many recent approaches aim to focus on adapting relatively smaller PEFT modules instead of the full backbone adaptation as in Ye et al. (2021). As it is overwhelming to carry out all of the literature reviews in this paper. just highlight here few works that are the most relevant to our method. Meta-ICL (Min et al. 2022) aims mimic the test scenario where the fhw-shot context data are fed into the prompt in the L.M
19

---

Published as a conference paper at ICLR 2025
inference. This motivates to adapt model with the same type of few-shot prompting during the meta training. However, main drawback is computational overhead in dealing with potentially large downstream training data at both training and test time. Several recent approaches aim to build model (LoRA) zoo of task-wise trained adapters (LoRAs) for multi-task training. The model zoo undergoes certain mixing of merging steps so to help optimal model selection at test time. The Mixture-of- (Feng et al.) 2024 Asadi et al.) 2024) use the router (MoE), an MLP network, that takes all task-wisely trained LoRAs, which makes it computationally infeasible as the number of tasks increases. Alternatively the mixture composition can be learned from data as in (Asadi) al. 2024). however, it also suffers from the same computational issue for a large number of tasks. LoRA Retriever (Zhao et al. 2024a) selects the most relevant LoRAs from the model zoo using the cosine similarity, and combine the selected LoRAs by either mixture (weight matrix-leve mixing) or fusion (low-rank matrix-level mixing) operations. Another recent strategy is the model- clustering (Ostapenko et al., 2024), where they perform the k-means clustering to compute the cluster assignment of the tasks. With this task assignment, they re-train LoRAs cluster- cluster-wisely. Although there are several proposed mixing strategies such as the instance- wise arrow routing, they show that the performance of the simple LoRA averaging performs comparably well.
Modulation- meta learning. Recently there have been several works that consider meta- learning by updating relatively few parameters (modulation). CAVIA (Zintgraf et al. 2019) is the first paper that suggested meta-learning with modulation. CNAPS (Requeima et al., 2019) uses a set-based architecture that takes the entire support set and the query image as input, and outputs query label. The FiLM (Perez et al. (0188) modulation has been combined with MAML (Finn et al. 20176) in (Dupont et al. 2022a
while LoRA (Hu al. 2022) modulation has been applied to MAML in (Schwarz et al. 2023) ). LITE (Bronskill et al. (021)) aims to build meta learner using the entire support data instead of minibatch, but to remedy computational overhead they set In (Bateni et al. 2020) distance metric has been adopted into CNAPS. (Tack et al.) 2024), a approach has been proposed to compress and extract
randomly chosen large portion of the support data as non-backpropagatabl
simple class-covariance-bass feature extraction and information from new data into compact modulations stored in a memory bank. has a very different flavor from ours in that they do not optimize PEFT parameters but to predict them based query. Unlike most of these modulation approaches, our approach is network architecture independent, and can potentially work smoothly with any form of backbone networks.
Online EM algorithms. There have been several prior works on the online EM methods in the literature. Although we find that none of them identical to the one proposed in this paper, most are similar in nature to ours. In (Neal & Hinton 1998) they proposed an incremental version of the EM (iEM), in which each iteration the conditional expectation of the latent variable (E-step) is updated only for a mini-batch of the observations. In (Cappé & Moulines] 2009) the full-batch E-step replaced by a stochastic approximation step, closely related to the stochastic gradient step. This scheme is further extended in (Chen et al.) 2018) for variance reduction. In (Karimi et al.) 2019) they analyzed incremental and stochastic versions of the EM algorithm as well as extending the variance reduction technique in common unifying framework. The online EM algorithms have been applied to the graph learning problem (He & Wai 2022). the filtering problem in state-space models (Özkan et al. 2012), and Internet traffic modeling (Liu et al. (2066). In the latter, they deal with Poisson mixtures where the online algorithm is based on the fact that the parameter increment has a positive projection the gradient of the likelihood function. They formed a stochastic approximation update scheme via exponential averaging of the old mixture parameters and new one.
E EXTRA EXPERIMENTAL RESULTS
E.1 RESULTS ON FLAN-T5-XL
To verify that the proposed approach is scalable to even larger LLMs, we ran our model on FLAN-T5-
XL (Chung et al. 2022) which contains 3B parameters. The results the Crossfit CLS-45 task split are summarized in Table 6 (compared with BART-base results in the main paper). We ran all methods on a single A100 80GB GPU where i-MAML (and its mixtures) and BMAML baselines incurred the out- offmmoorry issue. Our LiFT model runs well, and as shown we have large improvements over the competing methods.
20

---

Published as a conference paper at ICLR 2025
Table 6: FLAN-TS-XI results the Crossfit CL.S-45 task split.
Train
MAMI.
PO-MAMI.
Reprile
L.FT
LFFT K
FLAN-TS-XL
65.13
73.41
73.74
74.22
69.39
78.37
79.69
80.01
BART-base
52.68
59.38
57.59
64.12
Table 7:
Comparison against non- non-hierarchhcal Bayesian models the Crossfit CLS-45 task split.
Non-Hierarchical
Hierarchical
Shared PEFT
Task wise PEPT
LiFT K=1
LiFT K=3
LiFT =
CLS-45
55.62
56.74
62.96
63.37
64.12
Table 8: Ablation study on the prior variance hyperparameters (o², 332) for our LiFT K
0.001
0.005
0.01
0.05
0.1
0.2
0.3
0.5
CLS-45
62.87
63.05
64.12
63.62
62.75
62.75
61.48
61.73
66
LIFT K=5 (Ours)
65
i-MAML-Mix K-5
64
Score
63
61
60
59
0.001
0.005
0.01
0.05
0.1
0.2
0.3
0.5
o²,
in LIFT model
Figure 7: Ablation study on the prior variance hyperparameters (0²,89 for our LiFT K == 5.
Contrasted with the best runner-up baseline i-MAML-Mix K 5.
E.2 IMPACT OF HIERARCHICCAL BAYESIAN MODELING
To show the importance of hierarchical modeling, we have implemented and tested two non- hierarchical Bayesian models. In the first model, the PEFT LoRA parameters are treated as random variables ours, but they are shared across all tasks. Hence the posterior inference in this case amounts to lcarning task-shared aagnostgis information from meta training data. We used the SGLD posterior inference. This model performs way worse than our hierarchical model as shown in Table 7
As second non- non-hierarchica Bayesian model, by its own PEFT LoRA parameters, but there is no governing higher-level random variables as in our LiFT model. Consequently this model aims to learn only task-specific information, and would not transfer anything to novel tasks. Hence can be seen as a Bayesian version of "No Meta Train" method, which only test-time adaptation via SGLD posterior inference. This model also performs poorly compared to our hierarchical model as shown in Table 7
think of model where each task is represented
E.3 ABLATION STUDY ON HYPERPARAMETERS o² AND 3
Recall that we have variance hyperparameters (o², 332 for the priors p()) and p(0 (6) in 2 To see the robustness these hyperparameters, have the ablation study with our LiFT K 5 model on the Crossfit CLS task split. As shown in Table y and Fig. 7 the test performance is quite robust the choice of (a² .2)).
21

---

Published as a conference paper at ICLR 2025
Table 9: Ablation study asynchronous update strategy Eq. p
vs. stochastic approximation
alternative. Both strategies use LiFT K =5
Stochastic approximation
Proposed asynchronous update
CLS-45
63.61
64.12
Table 10: Per-iteration wall clock training times on the Crossfit CLS-45 task split. For our LiFT K 5, this the per-task/iteraton time after the burn-in stage.
Rank 4 LoRA
Rank-64 LoRA
Our LiFT Union Train
0.268 sees
0.287 secs
0.242 sees
0.270 sees
MAML
0.372 secs
0.380 secs
Table 11: Warm-up and burn-in stage running times the Crossfit CLS-45 task split.
Rank-4 LoRA
Rank-64 LoRA
Warm-up time
219
233 secs
Burn-in time
277 secs
279 sees
E.4 ABLATION STUDY ON ASYNCHRONOUS UPDATE STRATEGY EQ 11
Apart from our asynchronous & update strategy with the auxiliary variable J in Eq. 1 one think of stochastic approximate version of (7). which essentially replaces the average of V p(0, | ()) over tasks i by the current task iterate alone. To compare this simpler strategy with our asynchronous J update strategy, we have run the stochastic approximate version on the Crossfit CLS-45 task split. As shown in Table D, the simple stochastic approximation has decent performance, but slightly underperforms our original proposal of the // update strategy.
E.5 WALL CLOCK RUNNING TIMES
report the wall clock training times (on single V100 GPU) in Table TQ and 1T Our LiFT with K = 5 GMM is compared with the two baselines "Union Train" and "MAML" two different PEFT (rank and 64) cases. The result in Table[UU
that our model is as efficient as the vanilla SGD update (i.e., "Union Train") for both low and high ranks, thanks
efficient SGLD-Gibbs sampler. Asymptotically (in Big-O notation), both the vanilla SGD update and our SGLD-Gibbs 11 + online EM (Eq. (13 14h) take the same O + d) time per iteration where T is the time for the backbone forward/backwart computation for evaluating V log p( (2(0)) and d is the number of PEFT parameters. This is because of the Gaussian ( (Ø) which allows analytic gradient V log p( (6) with respect to both 0r and é. The EM iteration also takes (d) time.
(Eq.
The warm-up steps in our method refers to the vanilla SGD steps for the first 1000 iterations, and the burn-in steps amounts running the proposed SGLD recurrences for the next 1000 iterations (without collecting the posterior samples). After the burn-in steps, we collect the posterior samples while running the SGLD, The wall clock times for these two stages are shown in Table and they are quite reasonable times.
E.6 COMPARISON WITH BOOTSTRAPPED-MANI (FLENNERHAG ET AL. 2022
For empirical comparison with more recent meta learning approaches, we have implemented the Bootstrapped-MAN -AMML (Flennerhag et al. 2022) in our Crossfit experimental framework with LoRA PEFT as parameters to learn. In Bootstrapped Meiisstappee--AAMM MAML, the target model is first obtained by applying SGD updates on the train data then batch data sequentially. The meta learning loss is then the KL divergence between the predictive distributions of the train-data-updated model and the target model,
22

---

Published as a conference paper at ICLR 2025
Table 12:
Comparison with Boosstrapped-MAML on the Crossfit CL.S-45 task split.
MAML
Bootstrapped-MAMi.
LiFT K=1
LiFT K=3
LiFT K=
CLS-45
52.68
55.02
62.96
63.37
64.12
Table 13: Comparison with MAML with multiple inner iterations.
# Inner iterations in MAML
2
3
LiFT K =
CLS-45
52.68
48.48
55.73
56.02
64.12
Statistical significance and standard deviations. The p-values of the existing methods against our LiFT-K are shown the parentheses.
Table 14:
Meta Train
Union Train
MAML
FO-MAML
i-MAML
Reptile
56.27 1.30 (3.26 x 10-")
56.79 0.59
52.82 1.05
37.70 + 1.18
59.03 + 0.65
58.26 + 0.66 13.36 10 10-")
CLS-45
(3.63 10-*)
(3.72 10"*)
(8.02 10-5)
(8.81 X 10"7)
BMAML M=5
ABML
MAML-MiX K=5
I-MAML-MiX K=5
Retrover Mix K-5
Retriever Mix K=all
57. 78.98862 0.59 (1.09 ")
55.74 + 1.25
53.87 0.12
58,99 1.20
55.82 + 1.42
52.58 + 1.11 9)
CLS-45
12)
"
Retriever Fase K=5
Retriver Fuse K=all
MBC- K=5
MBC-p K=10
LIFT K=5 (Ours)
56.03 + 1.69
53.77 + 1.29 (3.52 x 10"3) )
56.17 1.05
55.41 0.87
63.87 + 0.31
CLS-45
(1.69 x 10
(6.23 10"")
(8.08 10 )
where the latter is treated as constant (no backpropagation). The comparison with Bootstrapped- MAML on the Crossfit CLS-45 split is shown Table 12
E.7 COMPARISON WITH MAML (FNN ET AL. 2017B WITH MULTIPLE INNER ITERATIONS
We have the experiments to the effect of the multiple inner loop updates in MAML. On the Cross-fit CLS-45 task split, increasing the number of inner iterations often helps improve the test performance, but MAML still underperforms iterations in MAML incurred the out- tut-of--emory issues on V100 GPU.
LiFT by large margin (Table 13 More than
E.8 STATISTICAL SIGNIFICANCE
Due to the computational overhead, is time- and compute-expensive to run multiple random seed experiments. However, to measure the statistical significance, perform multiple 5 random seed runs on the Crossfit- CLS45 split to collect standard deviations and compute p-values. As shown in Table 14 our LiFT model outperforms the existing methods statistically significantly.
23


Reviews:
[{"review_id": "OcJNgG5Eit", "reviewer": "Reviewer_PRGJ", "summary": "The paper develops a general fine-tuning strategy for foundational models that is based on the meta-learning principle. This way not only the foundational models become shareable, but also the layers responsible for task-specific fine-tuning can be accumulated and shared to be used on unseen tasks with not enough fine-tuning data. The core contribution of the paper is the Bayesian methodology that casts meta-learning (and hence task-specific fine-tuning) as Bayesian sampling exercise. Technically, this is implemented via a modification of SGLD and the online EM algorithm. Empirical results are obtained from NLP and Vision tasks.", "questions": "- $J$ appears in (11) out of nowhere. Could you please motivate the need in this term more clearly in the text transition between (7) and (9)? It seems to be the core algorithmic contribution that does not follow trivially from the original SGLD formulation in (7-8). It feels like by not discussing it in sufficient detail authors basically undersell their contribution.\n- Can you say at least something theoretical about your approximations, to strengthen the theory? I understand that convergence rate analysis might be to much of an ask, but if we talk about means and if you take the expectations of (9-11) will they match the expectations of equations (7-8) in the limit?\n- On a related note, can you run an ablation study by comparing against a more naive version of the algorithm that does not rely on $J$ updates? For example, you could use the sample approximation of the sum in (7) by the log-probability of $\\theta_{i}^a$ in a given update round. In my view, this could further strengthen the algorithmic contribution of the paper, or simplify the algorithm if sample approximation has same or better accuracy.\n- \"However, we aim to enrich it by a mixture of Gaussians to better approximate the true posterior that is inherently a multi-modal distribution\". Can you provide the results of ablation study comparing against Gaussian confirming that such enrichment is actually useful? This is especially important given that the Online-EM Mixture for Posterior Approximation is necessary only to support the GMM. If the GMM is not provably necessary, the value of the Online-EM Mixture contribution is questionable.\n- Since you are claiming online EM as a technical contribution, could you please update the related work section with the online EM literature? For example: https://arxiv.org/abs/2207.14019, https://www.diva-portal.org/smash/get/diva2:857377/FULLTEXT01.pdf, https://www.sciencedirect.com/science/article/abs/pii/S0167947304003263. Please discuss the novelty of your work w.r.t. existing contributions and motivate why you need a new version of online EM.\n- \"Hence we stick to the simple average of the metrics over all test tasks regardless of the metric types.\" I understand that the relative lift could be tricky, because of division by small numbers. Could you please consider reporting the mean of the absolute deltas between the baseline and the candidate algorithm? I believe this could be a more robust and more statistically significant measure of accuracy improvement than reporting the sum of raw metric values.\n- Will code be open-sourced?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the idea seems sufficiently novel and interesting\n- The empirical evaluation is extensive and convincing\n- Results are state of the art", "weaknesses": "- The justification of SGLD-Gibbs Sampling is only empirical through a toy example in Appendix A. A theoretical justification showing the required convergence would have significantly strengthened the contribution.\n- Ablation studies are not comprehensive enough, failing to support major decisions made in the algorithm design step: the $J$-term update and the Online-EM Mixture for Posterior Approximation. See questions section for details.\n- It looks like literature review could be updated with the online EM literature. I am not sure if the online EM algorithm is genuinely novel given the lack of analysis of the related work on this topic in the paper. See questions for details.\n- It is unclear if code is being open sourced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zR8LTo2mRw", "reviewer": "Reviewer_p6L8", "summary": "The authors propose a parameter efficient finetuning scheme called learning-to-finetune (LiFT) that can adapt a model not only to a single, but to a set of related tasks. \n\nAt the heart of LiFT sits a Bayesian meta training method that is executed on a set of related finetuing tasks. It uses hierarchical priors for the PEFT parameters to split task specific from task agnostic knowledge and runs stochastic gradient Langevin dynamics (SGLD) for posterior inference. The transferable task agnostic knowledge is later used as a prior for test-time adaptation. \n\nWhile they explain their method, using LoRA, the method is general and can be adapted easily to any other PEFT scheme.", "questions": "After thoroughly reading the paper, some questions remain:\n* Is there any experimental evidence that $\\phi$ really learns meaningful task-agnostic adaptations?\n\n* How brittle is the training if we change the parameters $\\sigma^2$ and $\\beta^2$ of the hierarchical model? Did you do any experiments there?\n\n* Why do you choose a GMM to model $\\phi|\\{D_i\\}_{i=1}^N$. \n\nI think, having a multi-modal distribution goes against your idea that $\\phi$ learns task-agnostic adaptations, because each mode can represent a specialization. More specifically, if the number of modes $M$ is equal to the number of tasks $N$, each mode of $\\phi|\\{D_i\\}_{i=1}^N$ can specialize to one task. In this case you would have something very similar to a stochastic version of the mixtures of LoRA idea. How do you prevent this from happening?\n\n* After test-time adaptation, is the model output stochastic or deterministic? I.e. do you continue to run the SGLD sampling for $\\theta|\\phi$ or do you use some statistics and perform just a deterministic forward pass?\n\n* In the stochastic case: Why don't you provide confidence intervals for the LiFT results?\n\n* How does the LiFT scale with the #trainable parameters?\n\n* What is the difference between warm-up samples and burn-in samples for the SGLD Gibbs sampling?\n\nI am looking forward to your explanations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is in general well written and I enjoyed reading it. Ideas are explained in detail. For most of the math intuitive explanations are provided. Hence, it is easy to understand the proposed meta-learning method and all the tricks that are needed to make it work. \n\nThe paper contains two creative ideas: 1) Their particular hierarchical Bayesian model for the PEFT parameters. 2) The combination of Gibbs sampling and SGLD for memory efficient posterior inference.", "weaknesses": "There are several weaknesses to this paper:\n* The main claim of the paper is, that it is beneficial to use their Bayesian formulation for PEFT of models to a set of related tasks. They give an intuitive explanation that one of their latent variables learns task agnostic and the other task specific adaptations. While this sounds reasonable, there is no analysis if this claim is actually true and, hence, if their hierarchical model actually makes sense. However, experimental sanity checks could be easily made e.g. by comparing to a non-hierarchical model. In the end I am completely puzzled what part causes the demonstrated increase in performance: 1) The formulation of the problem or 2) favorable training dynamics of this quite large training algorithm. \n\n* Important ablation studies are not provided: Their hierarchical model requires the choice of two variances $\\sigma^2$ and $\\beta^2$. Only one set of values is provided. However, to judge how brittle the model is, some ablations would be helpful.\n\n* The paper does not give any idea how the proposed algorithm scales with #adapted parameters (can it only be used with PEFT methods or even with FFT?). Training requires to run the SGLD sampling. From the appendix I got, that it requires quite some steps, i.e., 2000 burn-in and 1000 warmup steps. It is not obvious how this translates to training time.\n\n* The paper proposes an online EM algorithm to fit a GMM to posterior samples in an iterative way, making it unnecessary to store the full set of posterior samples. While stating that this is a novel contribution, there exists lots of work about this problem already. Keywords are: 1) incremental EM or 2) streaming GMMs [1], [2]. Related works are not referenced and a comparison is missing.\n\n[1] Hosseini, Reshad, and Suvrit Sra. \"An alternative to EM for Gaussian mixture models: batch and stochastic Riemannian optimization.\" Mathematical programming 181.1 (2020): 187-223.\n[2] Karimi, Belhal, et al. \"On the global convergence of (fast) incremental expectation maximization methods.\" Advances in Neural Information Processing Systems 32 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wDC7Q3DkO8", "reviewer": "Reviewer_3TVT", "summary": "The author proposed a modulated meta-learning scheme where the modulation is the LoRA parameter of the large model. Specifically, each task-specific parameter is represented with LoRA, and the base model is shared across all tasks. To learn such a model, the author proposed a hierarchical Bayesian meta-learning method called LiFT. Here, the task specifical LoRA is modeled to be sampled from a prior distribution, where the author have suggested an efficient sampling method.", "questions": "See the question above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall writing is clear, and the method itself is sensible.\n\nThe proposed sampling method is efficient. I think it would be great to show the experiment that shows the efficient gain.", "weaknesses": "Missing critical related works and comparison. Currently, there are many works that consider meta-learning with modulation (i.e., a few parameter updates from the base model, such as LoRA). For instance, CAVIA [1] is the first paper that suggested meta-learning with modulation. Furthermore, a more recent method, CNAPs [2], combines amortization-based meta-learning with modulation. Several works consider modulated meta-learning as follows: FiLM modulation with MAML [3,4], LoRA modulation with MAML [5], Scaling CNAPs to large-scale meta-learning [6,7], and LoRA modulation with amortization-based meta-learning [8].\n\n[1] Fast Context Adaptation via Meta-Learning, ICML 2019\\\n[2] Fast and flexible multi-task classification using conditional neural adaptive processes, NeurIPS 2019\\\n[3] From data to functa: Your data point is a function and you can treat it like one, ICML 2022 \\\n[4] COIN++: Neural Compression Across Modalities, TMLR 2022\\\n[5] Modality-Agnostic Variational Compression of Implicit Neural Representations, ICML 2023\\\n[6] Memory Efficient Meta-Learning with Large Images, NeurIPS 2021\\\n[7] Improved Few-Shot Visual Classification, CVPR 2020\\\n[8] Online Adaptation of Language Models with a Memory of Amortized Contexts, arXiv 2024\n\n----\n\nNeed to consider more recent baselines, and more effective meta-learning baselines. Currently, most of the meta-learning baselines are highly outdated. Furthermore, there are more effective and recent baselines [1,2,3]. Typically, [3] suggested the interpolation of sparse experts (i.e., only a few parameter updates), which has similarities with the current approach (i.e., LoRA modulation). \n\n[1] Meta-learning with warped gradient descent, ICLR 2020\\\n[2] Bootstrapped meta-learning, ICLR 2022\\\n[3] Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts, ICML 2024\n\n---\n\nI think the experiment application needs to be more motivating. The main purpose of using LoRA is to fine-tune large models to reduce the computation burden or overfitting. However, the current setup is mostly conducted in small-scale networks. I believe showing whether the proposed method scale to large-scale LLM (e.g., more than 1B param) will be an interesting and motivating example.\n\n---\n\nI agree that using meta-learning could be beneficial, but I don't understand the advantages of modeling with Bayesian meta-learning specifically. From an uncertainty perspective, it makes sense, but it's still possible to \"jointly learn the source task\" without a Bayesian approach. I'm particularly concerned about whether this proposed sophisticated sampling technique will truly scale with large models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LoSNj3JV1w", "reviewer": "Reviewer_pSzU", "summary": "This paper presents a novel hierarchical Bayesian model to tackle the cross-task PEFT knowledge transfer problem with meta-learning. The proposed LiFT approach introduces a SGLD-Gibbs sampling algorithm for efficient training and an online EM algorithm to maintain a Gaussian mixture representation of the posterior samples in an online manner. This method is evaluated on both NLP and vision tasks, and the results show improved performance over several baselines.", "questions": "*  How does the choice of $\\sigma$ and $\\beta$ impact on the model performance? \n*  In Figure 4 and the related discussion in line 742, it does seem that SGLD converges faster than SGLD-Gibbs especially for a low number of iterations. Could the authors clarify how to interpret this figure?\n*  Could the choice of parameters selected for meta-learning in the MAML-based baselines be explained in more detail?\n*  Could a comparison with MAML using 5 inner loops be provided to ensure that performance is not affected by the choice of a single inner loop? If conducting this experiment is not feasible due to time constraints, could an estimate of the time and computational resources required to complete it (based on the time taken for 1 inner loop) be provided?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "*  The paper tackles the cross-task PEFT knowledge transfer problem, which is a relevant and important topic in meta-learning and fine-tuning large models.\n*  The paper is well written and well structured. The design concept is clear from the very beginning. The reader is guided through the method step-by-step and the theoretical foundations are supported by the experiments.\n*  The integration of Gibbs sampling into SGLD and the assumptions made to approximate the posterior (all the variables are visited a sufficient number of times and frequently) are valuable.\n*  The proposed approach can also be applied when pre-fine-tuned models are already available, which is valuable for knowledge reuse.", "weaknesses": "*  The meta-learning baselines considered in the experiments are limited. I recommend including more recent deterministic and bayesian meta-learning approaches. Some examples, but not restriced to these, are SNAIL [2], ProtoNet [3], and MetaQDA [4].\n*  The results do not include standard deviations, making it difficult to assess whether the improvements with LiFT are statistically significant.\n*  While the work addresses several challenges, it also has some limitations, such as the joint training on the training tasks, which can be infeasible for large-scale datasets. The authors might consider adding a limitations section to acknowledge this constraint and any other potential challenges the method faces.\n*  It is unclear which architectures and hyperparameters are used for the meta-learning baselines. I assume that, for a fair comparison, ViT-B/16 was employed. However, MAML is generally inefficient when adapted to large feature extractors due to the difficulty of finding an optimal meta-initialization, which scales with the overall parameter space.\n*  A single inner step is used for MAML and its variant. While this reduces the computational cost, it is unclear if these results reflect the best possible performance for MAML, As noted in [1], MAML typically benefits from multiple inner loop updates. \n\nReferences: \\\n[1] Han-Jia Ye, & Wei-Lun Chao (2022). How to Train Your MAML to Excel in Few-Shot Classification. In International Conference on Learning Representations. \\\n[2] Mishra, N., Rohaninejad, M., Chen, X., & Abbeel, P. (2017). A simple neural attentive meta-learner. arXiv preprint arXiv:1707.03141.\\\n[3] Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. Advances in neural information processing systems, 30.\\\n[4] Zhang, X., Meng, D., Gouk, H., & Hospedales, T. (2021). Shallow bayesian meta learning for real-world few-shot recognition. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 651–660).\\", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)