Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models

## Abstract

Abstract Fine-tuning large language models (LLMs) with low-rank adaptaion (LoRA) is a cost-effective way to incorporate information from a specific dataset. However, it is often unclear how well the fine-tuned LLM will generalize, i.e., how well it will perform on unseen datasets. Methods have been proposed to improve generalization by optimizing with in-context prompts, or by using meta-learning to fine-tune LLMs. However, these methods are expensive in memory and computation, requiring either long-context prompts or saving copies of parameters and using second-order gradient updates. To address these challenges, we propose Amortized Bayesian Meta-Learning for LoRA (ABMLL). This method builds on amortized Bayesian meta-learning for smaller models, adapting this approach to LLMs while maintaining its computational efficiency. We reframe task-specific and global parameters in the context of LoRA and use a set of new hyperparameters to balance reconstruction accuracy and the fidelity of task-specific parameters to the global ones. ABMLL provides effective generalization and scales to large models such as Llama3-8B . Furthermore, as a result of using a Bayesian framework, ABMLL provides improved uncertainty quantification. We test ABMLL on Unified-QA and Crossfit datasets and find that it outperforms existing methods on these benchmarks in terms of both accuracy and expected calibration error.

## 1 Introduction

Large language models (LLMs) handle a variety of tasks reasonably well (Radford et al., 2019). However, to tailor LLMs to specific domains, fine-tuning on specific datasets is often necessary. While methods such as low-rank adaptation (LoRA; Hu et al. (2021)) fine-tune a pretrained LLM cost-effectively, a fine-tuned LLM is limited to the domain it is trained on. Its performance may not improve in other domains and sometimes worsens as it suffers from catastrophic forgetting. Such catastrophic forgetting may result in overfitting and erasing existing capabilities of the pretrained LLM (Lazaridou et al., 2021; Luo et al., 2023).

Meta-learning is a strategy for solving this problem, training models on a variety of tasks in a way that supports generalization across tasks (Finn et al., 2017). However, meta-learning typically requires a large amount of computation and memory, making it challenging to apply to LLMs. One form of meta-learning that has been applied to LLMs involves fine-tuning models on in-context prompt-response examples (Min et al., 2022; Chen et al., 2022). Another more traditional approach, MAML-en-LLM (Sinha et al., 2024), adapts the Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) framework to LLMs. However, both methods are limited in the size of the language models that can be used: the former requires long-context prompts, whereas the latter uses second-order gradient updates and saves a model for each task.

Recent work on Amortized Bayesian Meta-Learning (ABML; Ravi and Beatson (2019)) addresses some of the computation and memory requirements of meta-learning. This approach
posits a generative model over parameters where task-specific parameters are generated from global parameters, and inference over task-specific parameters is amortized. In other words, the conditional distribution over task-specific parameters is shared across tasks, implying that computation and memory costs stay constant with respect to the number of tasks. This approach thus offers a path towards efficient meta-learning for LLMs. However, several challenges exist. First, we need to specify the generative model over weight space in the context of LLMs. Second, the prior term used in ABML no longer adapts to the setting of fine-tuning a pretrained model because the spread of its weights mismatches that of an arbitrary prior used to train a model from scratch. Third, the enormous size of LLMs makes training difficult, as the scale of probabilities assigned to the model variables can overwhelm the influence of the data likelihood.

In this paper, we present a solution to these problems, taking a Bayesian approach to fine-tuning LLMs using ABML. To define the underlying generative model and efficiently characterize the distributions involved, we use LoRA to express both the model weights and their uncertainty. We introduce a new prior over global variables that accounts for the spread of the parameters learned in the pretrained model. We also introduce two adjustable hyperparameters that balance reconstruction accuracy and the fidelity of task-specific parameters to the global ones.

Using amortized Bayesian meta-learning for LLM fine-tuning, we achieve both higher accuracy and better uncertainty estimation over unseen tasks compared with regular fine-tuning and other scalable methods in the meta-learning literature. Figure [1](https://arxiv.org/html/2508.14285v1#S1.F1) illustrates an example where incorporating uncertainty estimation in fine-tuning leads to a more calibrated model response. Our method is scalable and avoids the computation and memory overhead of other meta-learning approaches, making it adaptable to larger models such as Llama 3 8B. We show that amortized Bayesian meta-learning provides fine-tuned LLMs that are accurate on domain-specific tasks, more generalizable to new tasks, and provide better uncertainty estimation.

Figure: Figure 1: An example where better uncertainty calibration leads to a more reasonable response. This is a prompt and response from an unseen dataset, coming from a pretrained LLM (left) and an LLM fine-tuned with ABMLL (right), with both being updated with 10 gradient steps on other examples of this dataset as in the meta-learning literature. The label is B), so both LLMs are incorrect, but the question is ambiguous: it could interpreted as either Jason “asked" Steven to deliver, or Jason “came to" Steven to deliver, resulting in different answers. ABMLL results in a more calibrated response.
Refer to caption: figs/example-twocol.png

## 2 Related Work

#### Meta-learning methods in LLMs.

Extensive work has explored meta-learning for generalization, typically adopted for models in the pre-LLM era (Finn et al., 2017; Snell et al., 2017; Ravi and Beatson, 2019; Nichol et al., 2018). Sinha et al. (2024) adapted Model-Agnostic Meta-Learning (MAML), developed in Finn et al. (2017), to LLMs. However, this adaptation is more expensive in computation and memory than our method, requiring second-order gradient updates and saving a model for each task. As a result, we evaluate on larger models than those tried in Sinha et al. (2024).

As a different approach, Min et al. (2022) and Chen et al. (2022) explored meta-learning for LLMs using in-context learning. These works show that it is possible to fine-tune LLMs on in-context examples and achieve generalization. However, our approach does not require curation of such examples, does not place constraints on the size of the context window of a model, and is more scalable.

#### Uncertainty representation for LLMs.

Approaches to capturing uncertainty for LLMs can rely on the intrinsic representation of uncertainty in the model or focus on capturing extrinsic uncertainty about model parameters. Intrinsic approaches produce better uncertainty calibration via prompt engineering and sampling (Gruver et al., 2023) or learning an external model (Shen et al., 2024). Extrinsic approaches include use fine-tuning methods to incorporate uncertainty, such as training LoRA with ensembles (Balabanov and Linander, 2024), Laplace approximation (Yang et al., 2023), and variational inference (Wang et al., 2024). Our work takes the extrinsic approach but differs from existing approaches by using the meta-learning setting to achieve strong uncertainty calibration through generalization across datasets.

## 3 Background

### 3.1 Low-Rank Adaptation (LoRA)

LoRA (Hu et al., 2021) fine-tunes LLM weights on a low-rank space to improve efficiency compared with regular fine-tuning. Let $\mathbf{W}_{0}$ of size $d_{\text{in}}$-by-$d_{\text{out}}$ denote a weight matrix from a pretrained LLM. Let $\mathbf{x}$ denote the input to $\mathbf{W}_{0}$, and $\mathbf{z}$ denote the output of $\mathbf{W}_{0}$, LoRA fine-tunes pretrained weight $\mathbf{W}_{0}$ by adding perturbation on the low-ranked space,

$$ $\displaystyle\mathbf{z}=(\mathbf{W}_{0}+\Delta\mathbf{W}_{0})\mathbf{x}=(\mathbf{W}_{0}+\mathbf{B}\mathbf{A})\mathbf{x}.$ $$

The trainable matrices $\mathbf{B}$ and $\mathbf{A}$ are known as LoRA adapters. The sizes of $\mathbf{B}$ and $\mathbf{A}$ are $d_{\text{out}}$-by-$d_{\text{rank}}$ and $d_{\text{rank}}$-by-$d_{\text{in}}$, respectively, with $d_{rank}$ being significantly smaller than the original dimensions. Therefore, the number of parameters to be updated are $(d_{\text{out}}+d_{\text{in}})d_{\text{rank}}$, significantly fewer than the original $d_{\text{out}}d_{\text{in}}$.

### 3.2 Amortized Bayesian Meta-Learning

Amortized Bayesian Meta-Learning (ABML) Ravi and Beatson (2019) improves upon MAML-based meta-learning frameworks by representing uncertainty with a Bayesian approach. It also amortizes inference over the parameters so that memory no longer increases linearly with the number of tasks.

Let $\theta$ denote global parameters such that a few steps of gradient descent will produce local parameters $\phi_{i}$ on task $i$ with dataset $D_{i}$.
ABML treats $\theta$ as random variables, and minimizes a negative evidence lower bound using variational inference,

$$ $\displaystyle\text{argmin}_{\theta}\Big{[}\sum_{i=1}^{M}-E_{q_{\theta}(\phi_{i}|D_{i})}[\log p(D_{i}|\phi_{i})]+$ (1) $\displaystyle\ \ \ \ \text{KL}\big{(}q_{\theta}(\phi_{i}|D_{i})\big{|}\big{|}p(\phi_{i}|\theta)\big{)}\Big{]}+\text{KL}(q(\theta)||p(\theta)).$ $$

The variational distribution $q_{\theta}(\phi_{i}|D_{i})$ is represented by the Gaussian distribution $N(\mu_{\phi},\sigma_{\phi}^{2})$ with $\mu_{\phi},\sigma_{\phi}$ as trainable parameters.

## 4 Method

Our method extends Amortized Bayesian Meta-Learning, making it possible to apply to LLMs. This approach combines the advantages of meta-learning for adapting to new tasks with Bayesian inference for uncertainty representation.

We use the the objective of Eq. [1](https://arxiv.org/html/2508.14285v1#S3.E1) from ABML.
In our setting, $\theta$ and $\phi_{i}$ are the global and task-specific model parameters produced as the output of LoRA adapters. On a high level, the generative process is,

$$ $\displaystyle\theta$ $\displaystyle\sim p(\theta),$ $\displaystyle\phi_{i}$ $\displaystyle\sim p(\phi_{i}|\theta),$ $\displaystyle D_{i}$ $\displaystyle\sim\text{LLM}(\phi_{i}),$ $$

where $i$ represents any task $i$, and $\text{LLM}(\phi_{i})$ denotes the LLM considered as a probabilistic model that takes $\phi_{i}$ as its weights and outputs token sequences with joint probabilities defined by the LLM’s autoregressive predictive distribution.
For any LLM layer with pretrained weights $\mathbf{W_{0}}$, the quantities for our extension to ABML are as follows:

$$ $\displaystyle\hskip 0.0pt\mu_{\theta}$ $\displaystyle=\mathbf{B}_{\mu_{\theta}}\mathbf{A}_{\mu_{\theta}},$ $\displaystyle\hskip 0.0pt\sigma_{\theta}$ $\displaystyle=\mathbf{B}_{\sigma_{\theta}}\mathbf{A}_{\sigma_{\theta}}+c\mathbf{I},$ $\displaystyle\hskip 0.0pt\mu_{\phi}$ $\displaystyle=\mathbf{B}_{\mu_{\phi}}\mathbf{A}_{\mu_{\phi}},$ $\displaystyle\hskip 0.0pt\sigma_{\phi}$ $\displaystyle=\mathbf{B}_{\sigma_{\phi}}\mathbf{A}_{\sigma_{\phi}}+c\mathbf{I},$ $\displaystyle\hskip 0.0ptp(\phi_{i}|\theta)$ $\displaystyle=N(\phi_{i};\mu_{\theta}+\mathbf{W_{0}},\sigma_{\theta}^{2}),$ $\displaystyle\hskip 0.0ptq_{\theta}(\phi_{i}|D_{i})$ $\displaystyle=N(\phi_{i};\mu_{\phi}+\mathbf{W_{0}},\sigma_{\phi}^{2}),$ $\displaystyle\hskip 0.0ptp(\theta)$ $\displaystyle=p(\mu_{\theta},\sigma_{\theta})$ $\displaystyle=N(\mu_{\theta};0,c\mathbf{I})\cdot\text{Gamma}(\frac{1}{\sigma_{\theta}^{2}};a_{0},b_{0}),$ $\displaystyle\hskip 0.0pt\text{KL}(q(\theta)||p(\theta))$ $\displaystyle=-\log p(\theta).$ $$

Lastly, $p(D_{i}|\phi_{i})$ is defined as the joint probability assigned to $D_{i}$ where the LLM takes $\phi_{i}$ as its weights. The trainable parameters are the LoRA adapters $\mathbf{A}$ and $\mathbf{B}$. However, we introduce four pairs of these adapters to compute both the mean and variance of the LoRA outputs on local and global model weights. $\mathbf{I}$ is identity matrix, and $c$ is a hyperparameter constant dependent on the spread of pretrained LLM weights. $a_{0}$ and $b_{0}$ are hyperparameters, and the simplification of the KL term as $-\log p(\theta)$ follows Ravi and Beatson (2019).

#### Balancing the reconstruction error.

LLMs are often overparameterized. As a result, probabilistic quantities on the space of weights, $\text{KL}\big{(}q_{\theta}(\phi_{i}|D_{i})\big{|}\big{|}p(\phi_{i}|\theta)\big{)}$ and $\text{KL}(q(\theta)||p(\theta))$, can overwhelm quantities on the data space, $\log p(D_{i}|\phi_{i})$. $\beta-$VAE (Higgins et al., 2016) balances the evidence lower bound in variational auto-encoders with an additional hyperparameter to weight these terms. Inspired by this idea, we introduce hyperparameters $\beta,\gamma$, resulting in the following objective,

$$ $\displaystyle\text{argmin}_{\theta}\Big{[}\sum_{i=1}^{M}-E_{q_{\theta}(\phi_{i}|D_{i})}[\log p(D_{i}|\phi_{i})]+$ (2) $\displaystyle\ \ \ \beta\text{KL}\big{(}q_{\theta}(\phi_{i}|D_{i})\big{|}\big{|}p(\phi_{i}|\theta)\big{)}\Big{]}+\gamma\text{KL}(q(\theta)||p(\theta)).$ $$

This provides a flexible way to control how close the global parameters $\theta$ are to the prior $p(\theta)$, and how close the task-specific parameters $\phi_{i}$ are to $\theta$.

Figure: (a) Accuracy.
Refer to caption: x1.png

## 5 Empirical Evaluations

#### Model and datasets.

We fine-tune Llama3-8B on CrossFit (Ye et al., 2021) and UnifiedQA (Khashabi et al., 2020), textual datasets commonly used to train meta-learning models. Because a key aim of our paper is to study uncertainty quantification, we filter for multiple choice datasets, leading to a subset of CrossFit and UnifiedQA with 34 datasets with 68K training datapoints in total. For evaluation on an unseen task, we use Winogrande (Sakaguchi et al., 2021), a multiple choice dataset evaluating common sense reasoning.

#### Metrics.

We use accuracy to evaluate general performance and expected calibration error (ECE) to evaluate uncertainty estimation.

#### Implementational details.

All methods use batch-size of 2, inner-loops with 5 gradient steps, and learning rate is tuned in ${[10^{-6},5\cdot 10^{-5}]}$. For ABMLL, ${\beta=5\cdot 10^{-10}}$, ${\gamma=10^{-6}}$, ${c=e^{-20}}$. For the gamma prior, ${a_{0}=1,b_{0}=0.01}$. During validation on the unseen dataset, all models train 10 gradient steps on 10 batches from this dataset and evaluate on the rest.

#### Baselines.

We use four baseline methods that can viably scale to Llama3-8B. Pretrained is the off-the-shelf LLM. Regular LoRA is the default LoRA method trained on the whole randomly shuffled training dataset. Structured LoRA also uses the default LoRA, but the training dataset follows the same “structure” as our method: it is iteratively trained 5 gradient steps on one task at a time. Thus, it tests the effect of our generative model on performance. Reptile (Nichol et al., 2018) implements the Reptile meta-learning algorithm.

#### Results.

Figure [2](https://arxiv.org/html/2508.14285v1#S4.F2) shows validation accuracy and ECE over epochs across methods. We observe that ABMLL consistently achieves higher accuracy. On ECE, ABMLL also consistently achieves the best performance, whereas structured LoRA worsens as training continues. For fariness, the result at the end of every sixth epoch is reported from regular LoRA, because both ABMLL and Reptile run six instead one gradient step during each epoch’s training.

Table [1](https://arxiv.org/html/2508.14285v1#S5.T1) reports the best validation score for each model from three random seeds, showing a statistically significant advantage for ABMLL.

**Table 1: Validation accuracy and ECE across three random seeds, with standard error.**
| Method | Accuracy $\uparrow$ | ECE $\downarrow$ |
| --- | --- | --- |
| Pretrained | $68.2\%\pm 0.3\%$ | $0.327\pm 0.000$ |
| Regular LoRA | $68.2\%\pm 0.3\%$ | $0.327\pm 0.000$ |
| Structured LoRA | $73.6\%\pm 0.6\%$ | $0.320\pm 0.001$ |
| Reptile | $73.5\%\pm 0.2\%$ | $0.370\pm 0.005$ |
| ABMLL | $74.8\%\pm 0.3\%$ | $0.317\pm 0.001$ |

## Conclusion

Meta-learning is an effective method for supporting better generalization across datasets, but its demands on computation and memory can make it difficult to apply to large language models. We have shown how meta-learning can be used to adapt LLMs by combining Amortized Bayesian Meta-Learning with Low-Rank Adaptation. This approach results not just in better accuracy across several benchmarks, but also in better calibration.

## Limitations

One limitation of the paper is the scope of empirical evaluation regarding datasets and models. While the datasets feature natural text that can occur in the real world, it would be beneficial to evaluate on more test datasets to confirm the method’s consistency. Additionally, the paper’s method can be naturally extended to other models, so evaluating on more models would be a reasonable venue for future work.

As a meta-learning method, our approach must be trained on datasets that can be naturally divided into different tasks, a requirement that is not always available to practioners seeking significant model improvement on one particular domain.

While our empirical results suggest that our approach provides more accurate and calibrated responses, theoretical convergence is not guaranteed due to the need for approximate inference and various design choices, including limitations of the variational family $q_{\theta}(\phi_{i}|D_{i})$.

## Acknowledgments

We thank Qinyuan Ye for helpful discussion. This work was supported by grant
N00014-23-1-2510 from the Office of Naval Research.

## References

- Balabanov and Linander (2024)
Oleksandr Balabanov and Hampus Linander. 2024.
Uncertainty quantification in fine-tuned LLMs using LoRA ensembles.
*ArXiv*, abs/2402.12264.
- Chen et al. (2022)
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2022.
Meta-learning via language model in-context tuning.
In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 719–730. Association for Computational Linguistics.
- Finn et al. (2017)
Chelsea Finn, P. Abbeel, and Sergey Levine. 2017.
Model-agnostic meta-learning for fast adaptation of deep networks.
In *International Conference on Machine Learning*.
- Gruver et al. (2023)
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. 2023.
Large language models are zero-shot time series forecasters.
In *Advances in Neural Information Processing Systems 37*.
- Higgins et al. (2016)
Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. 2016.
Beta-VAE: Learning basic visual concepts with a constrained variational framework.
In *International Conference on Learning Representations*.
- Hu et al. (2021)
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021.
LoRA: Low-rank adaptation of large language models.
*ArXiv*, abs/2106.09685.
- Khashabi et al. (2020)
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.
UnifiedQA: Crossing format boundaries with a single qa system.
*ArXiv*, abs/2005.00700.
- Lazaridou et al. (2021)
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Giménez, Cyprien de Masson d’Autume, Tomás Kociský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.
Mind the gap: Assessing temporal generalization in neural language models.
In *Advances in Neural Information Processing Systems 35*.
- Luo et al. (2023)
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023.
An empirical study of catastrophic forgetting in large language models during continual fine-tuning.
*ArXiv*, abs/2308.08747.
- Min et al. (2022)
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022.
MetaICL: Learning to learn in context.
In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*.
- Nichol et al. (2018)
Alex Nichol, Joshua Achiam, and John Schulman. 2018.
On first-order meta-learning algorithms.
*ArXiv*, abs/1803.02999.
- Radford et al. (2019)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.
Language models are unsupervised multitask learners.
*OpenAI blog*, 1(8):9.
- Ravi and Beatson (2019)
Sachin Ravi and Alex Beatson. 2019.
Amortized Bayesian meta-learning.
In *International Conference on Learning Representations*.
- Sakaguchi et al. (2021)
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.
Winogrande: An adversarial winograd schema challenge at scale.
*Communications of the ACM*, page 99–106.
- Shen et al. (2024)
Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory W. Wornell, and Soumya Ghosh. 2024.
Thermometer: Towards universal calibration for large language models.
In *Proceedings of the 41st International Conference on Machine Learning*, pages 44687–44711.
- Sinha et al. (2024)
Sanchit Sinha, Yuguang Yue, Victor Soto, Mayank Kulkarni, Jianhua Lu, and Aidong Zhang. 2024.
MAML-en-LLM: Model agnostic meta-training of llms for improved in-context learning.
In *Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, page 2711–2720.
- Snell et al. (2017)
Jake Snell, Kevin Swersky, and Richard Zemel. 2017.
Prototypical networks for few-shot learning.
In *Proceedings of the 31st International Conference on Neural Information Processing Systems*.
- Wang et al. (2024)
Yibin Wang, Haizhou Shi, Ligong Han, Dimitris N. Metaxas, and Hao Wang. 2024.
BLoB: Bayesian low-rank adaptation by backpropagation for large language models.
In *Advances in Neural Information Processing Systems 38*.
- Yang et al. (2023)
Adam X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. 2023.
Bayesian low-rank adaptation for large language models.
*ArXiv*, abs/2308.13111.
- Ye et al. (2021)
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021.
CrossFit: A few-shot learning challenge for cross-task generalization in NLP.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: Stated in Weakness
soundness: 2
strengths: -  The paper makes an effort to address generalization as an important problem in large language model (LLM) adaptation.

 - The presentation is clear and the motivation is well established, providing a coherent link between Bayesian meta-learning and LoRA-based fine-tuning.

 - The experiments are well-presented, evaluating accuracy and expected calibration error.
summary: The paper proposes Amortized Bayesian Meta-Learning for Low-Rank Adaptation of Large Language Models (ABMLL), an approach that combines amortized Bayesian meta-learning with LoRA to enable scalable adaptation of large-scale LLMs. The method treats the global LoRA parameters as Bayesian latent variables that generate task-specific LoRA adapters, allowing the model to learn a shared probabilistic prior across tasks while efficiently adapting to new ones.
weaknesses: - W1. Limited novelty over prior work.
 In Algorithm 1, the main training and testing procedures of ABMLL closely mirror Algorithm 1 and Algorithm 2 in [1], with the only modification being the reparameterization via LoRA. Since [1] also targets generalization performance in meta-learning, and the paper does not clearly articulate the conceptual or methodological advances beyond [1], the novelty of the proposed approach appears extremely limited.

 - W2. Lack of empirical comparison with strong baselines.
 Although the paper positions ABMLL as an approach for improving generalization through optimizing in-context prompts or meta-learning, it fails to empirically validate these claims against concurrent methods. In particular, the authors discuss the limitations of related work but do not provide quantitative comparisons with representative baselines such as LiFT, especially regarding peak memory cost and computational efficiency. This omission weakens the paper’s contribution relative to parallel research efforts.

 - W3. Heuristic and unstable hyperparameter choices.
 The paper acknowledges that balancing the reconstruction error is crucial, yet the corresponding hyperparameters are chosen heuristically without theoretical justification. As shown in Table 2, the method is highly sensitive to these hyperparameters, undermining robustness across tasks. Moreover, the same set of hyperparameters is used for two terms in Eq. (2), further raising concerns about its practical choice.

 - W4. Insufficient experimental motivation and limited improvement.
 The experiments mainly compare ABMLL with standard LoRA and Structured LoRA, yielding marginal performance gains. Given that the paper emphasizes cross-task generalization, the experimental design lacks motivation and diversity. It would be more convincing to include baselines that explicitly target generalization, enabling a clearer evaluation of the proposed method’s effectiveness.

[1] Ravi S, Beatson A. Amortized bayesian meta-learning. International Conference on Learning Representations. 2019.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: See above.
soundness: 2
strengths: + **Reasonable integration of Bayesian meta-learning and LoRA.** The paper adapts amortized Bayesian meta-learning to large-scale LoRA fine-tuning, offering a principled probabilistic framework for task adaptation and uncertainty quantification in LLMs.
+ **Good writing.** The paper's writing is clear and easy to follow, making it accessible to the audience. 
+ **Improved generalization and uncertainty calibration.** Empirical results show consistent improvements in both accuracy and expected calibration error, demonstrating enhanced generalization to unseen tasks.
summary: The paper proposes a scalable Bayesian meta-learning framework to improve the generalization and uncertainty estimation of LoRA-based fine-tuning for LLMs. Traditional meta-learning methods like MAML and Reptile are computationally expensive and memory-intensive, while LoRA fine-tuning lacks generalization across unseen domains. ABMLL adapts amortized Bayesian meta-learning by modeling global and task-specific LoRA parameters under a probabilistic framework, balancing data reconstruction with parameter regularization through a tunable β term. Experiments on CrossFit and Unified-QA benchmarks using LLAMA3-8B show that ABMLL consistently achieves higher accuracy and better calibration (lower ECE) than standard LoRA and Reptile methods, while maintaining scalability and efficiency. The approach bridges Bayesian inference with parameter-efficient LLM adaptation, offering principled uncertainty quantification and improved cross-task generalization.
weaknesses: + **Lack of Technical Novelty.** The paper essentially merges two fairly well-known ideas: (a) PEFT via LoRA and (b) meta-learning (or Bayesian meta-learning) to obtain better generalization. One could argue that while the Bayesian angle gives a spin, the core innovation is limited. The amortised Bayesian meta-learning applied to LoRA is interesting, but meta-learning for PEFT has already been explored [1]. If the improvement comes mainly from combining existing techniques (LoRA + amortised Bayesian meta-learning), is this more an engineering/combination paper rather than a fundamentally new algorithmic contribution?
+ **Insufficient experiments: datasets and applications.** The empirical evaluation is limited to a small number of tasks/datasets (namely, Winograd and cls). While these are valid, the scope is narrow.
+ **Performance issue.** In these settings, the calibration of method does not bring significant improvement (as shown in Table. 1).
+ **Missing Baselines.** The idea of combining PEFT and Meta-Learning has been explored by prior work [1,2,3,4,5] (I might have missed some too). Please discuss the relationship between them and the reason why they are not included for comparison. 

**References**
- [1] Gheini, Mozhdeh, Xuezhe Ma, and Jonathan May. "Know Where You're Going: Meta-Learning for Parameter-Efficient Fine-Tuning." arXiv preprint arXiv:2205.12453 (2022).
- [2] Hou, Zejiang, Julian Salazar, and George Polovets. "Meta-learning the difference: preparing large language models for efficient adaptation." Transactions of the Association for Computational Linguistics 10 (2022): 1249-1265.
- [3] Shao, Yihua, et al. "In-Context Meta LoRA Generation." arXiv preprint arXiv:2501.17635 (2025).
- [4] Cheng, Bo, et al. "MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models." arXiv preprint arXiv:2510.11598 (2025).
- [5] Tian, Zichen, Yaoyao Liu, and Qianru Sun. "Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: * Does the proposed method require multiple samples from the posterior distribution during inference? If so, what is the additional computational cost introduced?

* According to Figure 2, ABMLL appears to require significantly more epochs for ECE to converge compared to other methods. Is this behavior consistent across all training datasets? This potential weakness should be further analyzed and discussed in the paper.
soundness: 2
strengths: * The presentation and organization of the paper are clear and easy to follow.
* The proposed method is simple and effective.
* The paper effectively mitigates the high computational cost of Bayesian meta learning through introducing LoRA.
summary: This paper combine ABMLL with LoRA to mitigates the high computational cost of Bayesian meta learning method. The results show that the proposed method achieves better performance compared to baseline methods.
weaknesses: * The paper essentially combines Bayesian method and LoRA in a straightforward manner. Combining Bayesian methods with LoRA is a common approach in recent work; based on this, the paper lacks furthermore novelty.

* Computational cost is a crucial aspect to the proposed method, as reducing memory usage is the primary motivation for introducing LoRA. However, the corresponding analysis is missing.

* The baseline method is relatively outdated; the authors should consider comparing with more recent approaches.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: - You claim improved calibration. Have you examined whether the learned variance parameters correspond to meaningful epistemic uncertainty (e.g. via correlation with task difficulty or out-of-domain samples)?
- On the comparisons to in-context learning, in which scenarios would you expect ABMLL to outperform well-designed in-context learning baselines?
- You mention that the prior reflects the "spread" of pretrained parameters. Could you describe exactly how this was estimated? Was it layer-wise empirical variance, or simply a Normal-Gamma prior with fixed hyperparameters?
- What numeric precision was used during training? For smaller $\beta$ values of $10^{-7}$ and below, how did you ensure that these coefficients were represented accurately and did not underflow to zero in mixed-precision computation? With $\beta$ so small, and the prior's effect minimal, how do you explain the improved calibration?
- The abstract and introduction emphasised that ABMLL is computationally and memory efficient; could you provide explicit measurements (such as time per epoch, peak GPU memory, etc) compared to LoRA, Reptile, and MAML under equal settings. I also note that ABMLL introduces four pairs of LoRA adapters, roughly quadrupling adapter parameters vs vanilla LoRA. How do you control for fairness when comparing to baselines?
soundness: 1
strengths: The paper is clearly written and does a good job of setting out the related work, the problems faced therein with the expensive 2nd order gradient calculation, and describing their proposed method.
summary: This paper sets out to improve the speed with which LoRA fine-tuned LLMs can generalize to unseen datasets via an amortized Bayesian meta-learning approach, with the additional benefit of providing uncertainty estimates on the predictions made by these fine-tuned LLMs. The method introduces a hierarchical Bayesian prior over LoRA weights. Each adapted weight matrix's LoRA parameters are modelled as Gaussian random variables, whose means and variances are themselves produced by LoRA adapters, yielding distributions over weight updates that encode both fit and uncertainty. Training proceeds via a $\beta$-tempered meta-ELBO, which attempts to balance data fit against a KL regularizer that ties the per-task posteriors to a global prior. The authors conduct a set of experiments on common meta-learning datasets, and show modest improvements on a couple of baselines, while remaining computationally cheaper than full MAML-style meta learning algorithms.
weaknesses: Unfortunately the paper has a few weaknesses.

The first is a high-level concern about the relevance of MAML-style approaches in the age of modern pre-trained foundation models. One of the strengths of LLMs and agent systems is that they can leverage their context windows to learn how to perform a one-off task by drawing in relevant research, examples and other information to learn in-context how to solve the task. This requires no gradient updates during deployment, is cheap and simple. For tasks which will be encountered repeatedly, where the user may wish to fine-tune the model to amortize the cost of learning to perform the task, then vanilla LoRA fine-tuning is already computationally cheap, and as shown by the authors themselves in their experiments, performs well. I recommend that instead of studying MAML-style approaches in isolation, these simpler baselines which are predominantly adopted by the community should be carefully run as baselines and proven to be inferior before proceeding.

The second concern is that the method, despite the name, does not appear to be truly amortized. In Ravi & Beatson 2019, the model learns a single inference network which maps the task's dataset to the posterior parameters in one forward pass (requiring no optimization at inference time, thus yielding constant adaptation cost wrt. the number of tasks). In contrast, the method proposed in this paper takes $K$ gradient steps on each task's LoRA parameters $\phi_{i}$ at both train and test time. The amortization is therefore only structural in that the same LoRA parameterization and global prior $\theta$ is shared across tasks, however the inference itself is still optimized per task. As a result, ABMLL appears to inherit the same per-task adaptation cost as MAML/Reptile, albeit avoiding the second-order gradients.

The proposed method does not appear to be "Bayesian" either. This arises due to a number of factors; the first is that the global "posterior" $q(\theta)$ is collapsed to a point estimate due to setting $KL[q(\theta) \Vert p(\theta)] = -\log p(\theta)$, thus reducing any uncertainty arising at the meta level (i.e. if a task is far from the distribution encountered during meta-training). Further, the authors correctly note that in Bayesian deep learning, particularly for networks with large parameter counts, the regularization provided by the KL term can overwhelm the data fit and reduce performance (akin to the issues described by [Trippe & Turner, 2018](https://arxiv.org/abs/1801.06230)). While moderately tempering the posterior is seen by some practitioners as a valid way to deal with this shortcoming, the authors appear to use a very aggressive tempering of $\beta = 10^{-7}$. If the training proceeds in bf16 or fp16 mixed precision, then the contribution from any term multiplied by this value risks being numerically negligible. This, combined with the low batch size of 2 resulting in high gradient noise means that the method (as trained) effectively performs maximum-likelihood fine-tuning in LoRA space, with a tiny Gaussian prior regularizer which risks being dominated by the size of the gradient updates. For the ablation with $\beta = 10^{-16}$, the KL term is effectively zeroed out in half-precision.

Finally the evaluations in the paper are unfortunately quite limited: using only a single model - it would have been good to examine a broader range of model families and sizes (for instance spanning Llama, Qwen, Olmo, Phi, Gemma and sizes ranging from 1.5B parameters to 70B parameters). The paper makes a claim about the efficiency of this method, however the authors omit a study of the FLOPs or memory consumption of the method in comparison to baselines. The empirical performance of the method is also unfortunately slightly underwhelming, closely resembling the regular "structured" LoRA, with greater computational and memory cost as a result of the four LoRA adapters for the means and variances of the task-specific and global parameters, respectively. This additional cost and complexity does not seem to be commensurate with the performance gains.


Decision: Reject