Latent Space Factorization in LoRA
Authors: Shashi Kumar
Yacouba Kaloga
John Mtr.
Petr Motlicek
Ina Kodrasi
Venue: NeurIPS 2025 poster ()
Keywords: low-rank adaptation
latent space factorization
spurious correlation robustness
fvae-lora
Primary area: deep_learning

Abstract: Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning.
However, existing LoRA variants lack mechanisms to explicitly disambiguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance. 
We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA), which leverages a VAE to learn two distinct latent spaces.
Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information.
Extensive experiments on text, audio, and image tasks demonstrate that FVAE-LoRA consistently outperforms standard LoRA. 
Moreover, spurious correlation evaluations confirm that FVAE-LoRA better isolates task-relevant signals, leading to improved robustness under distribution shifts.
Our code is publicly available at: https://github.com/idiap/FVAE-LoRA

Latent Space Factorization in LoRA
Shashi Kumar 1.2
Yacouba Kaloga'
John Mitros'
Petr Motlicek 1.3
Ina Kodrasi!
'diaap Research Institute, Switzerland
EPFL, Switzerland BBU,, Czech Republic
Aatashi.kumay. yacouba, kaloga, petr motlicck,, ina (aroodrass@@idddn .ch john wwt..mmsttsmaaaammmmmmmmmmmmmm com
Abstract
Low-rank adaptation (LoRA) is widely used method for parameter-efficient finetuning. However, existing LoRA variants lack mechanisms to explicitly disam- biguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance. We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA), which leverages VAE to learn two distinct latent spaces. Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space fask-salient features and the other to residual information. Extensive experiments on text, audio, and im- age tasks demonstrate that FVAE-LoRA consistently outperforms standard LoRA. Moreover, spurious correlation evaluations confirm that FVAE-LoRA better iso- lates task-relevant signals, leading improved robustness under distribution shifts. Our code is publicly available at: https: //g github. com http//ddaa//AAE...
Introduction
Foundation models have become ubiquitous across modalities such vision 2 audio 55 6). and text 7 8 Recent satte-of-the-ae art results are predominantly achieved by fine-tuning these large pre-trained models. Among various carameter-efficien fine-tuning (PEFT) strategies 10 11 12] Low-Rank Adaptation (LoRA) [122 has emerged as particularly efficient approach. In LoRA, the original weight matrices W E IR led are kept frozen, and trainable low-rank matrices E R and B E R are introduced, with < min(d, k), such that the adapted weights become W + BA. computational requirements, while achieving
This technique significantly reduces memory performance comparable to full fine-tuning 12 13
Despite the remarkable performance shown by LoRA across plethora of downstream tasks and modalities, we identify a potential limitation: the standard LoRA update mechanism lacks an explicit way ensure that the learned low-rank subspace Im(A) (A) primarily captures task-salient information. The projection Ax (where x is the input activation) is learned implicitly through gradient descent on the task objective. While effective, this process does not inherently guarantee that A isolates features crucial for the downstream task from potentially irrelevant or even detrimental information retained from pre-training. This lack of explicit control over the content of the low-rank update pertinent. While the hypothesis that fine-tuning primarily involves low-rank updates provides strong theoretical underpinning for LoRA 14] empirical evidence suggests nuances. Recent studies have shown that standard LoRA can still underperform full fine-tuning in certain scenarios 12 15]. This suggests that simply constraining the update to be low-rank might not be sufficient; the task -elevvnt signal encoded within that low-rank adaptation is critical for achieving optimal downstream performance. Existing LoRA variants do not offer principled mechanism to explicitly disentangle and prioritize task-relevant information within the learned update.
To address this limitation and enable explicit control over the information captured within the low- rank update, we propose Factorized Variational Autoencoder LoRA (FVAE-LoRA). Our approach 39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LoRA
FVAE-LORA
+ BAJX
Wx Bas
Wx + Brg
BAX
B2
X pa(a)ai,
Bz1
B
P
B
W
Ax
W
1
62 ~ Uds
W
Z1
X
Trinning Hime
Iuferenoc neturor
Figure 1: Comparison between LoRA and the proposed FVAE-LoRA. During training, FVAE-LoRA factorizes the latent space into two components, and Z2, where only the task-salient latent factor 21 propagated downstream. inference, only the encoder corresponding to Z1 is required.
integrates Variational Autoencoder (VAE) framework directly into the LoRA parameterization. Crucially, unlike standard VAřs, FVAE learns two distinet latent spaces, denoted by 2 and 2₂ (see Figure []. We introduce a novel Evidence Lower Bound (ELBO) formulation specifically designed to promote factorization between these two spaces during training. This objective encourages the model to encode task-salient information, critical for downstream performance, primarily within the first latent space Z1, while relegating residual information necessary for accurate reconstruction (as required by the FVAE objective) the second latent space Z2. During the forward pass for the downstream task, only samples drawn from the task-salient latent space Z1 are utilized to generate the effective low-rank adaptation matrix A. This mechanism allows FVAE-LoRA explicitly select and leverage the most relevant learned features for the target task, while isolating potentially less useful confounding variations within Z2.
Our main contributions can be summarized as follows:
A Novel PEFT Method (FVAE-LORA): We propose FVAE-LORA, integrating VAE with factorized latent spaces (%1, 22) into the LoRA framework to explicitly disentangle task-salient information (2,) from residual information (z2).
Factorizing ELBO Formulation: We introduce a novel ELBO objective specifically
designed to enforce this factorization between the two latent spaces during training.
Strong Empirical Performance: We demonstrate through extensive experiments on diverse image, text, and audio benchmarks that FVAE-LoRA consistently outperforms LoRA.
We empirically validate, using targeted spurious- correlation experiments, that the task-salient latent space 2 indeed captures task-critica information, leading to robust performance even on challenging examples designed to mislead standard LoRA.
Empirical Validation of Robustness:
Related Work
We position FVAE-LoRA relative to PEFT methods, specifically LoRA variants, and techniques for latent space factorization in VAEs.
PEFT methods adapt large pre-trained models with minimal trainable parameters, overcoming the costs of full fine-tunie toninnnnng.... Common approaches include inserting Adapter modules [9] optimizing continuous prompts prefixes [10]6]] or tuning only bias terms [17]. LoRA [12] is prominent
PEFT technique that injects trainable low-rank matrices (&W BA, rank r < minnd, k)) into the
model layers. Its efficiency and performance have led to wide adoption [4] FVAE-LoRA builds upon the LoRA framework, aiming to enhance its effectiveness.
LoRA Variations. Several methods have extended LoRA. AdaLoRA 18] adaptively allocates rank budgets. DoRA [15] decouples weight magnitude and direction, applying LoRA the latter. LoRA+ [19 adjusts LoRA's optimization by using different learning rates for its two low-rank matrices. PiSSA 200 focuses on initializing the LoRA matrices way that better approximates full fine-tuning updates, typically setting A 0 and initializing B based principal components of the

---

gradient. RS-LORA 21 aims to stabilize training and prevent rank collapse or excessive growth by incorporating regularization related the singular values of the update matrix. Other contributions combine LoRA with quantization for further compression [22 23 These variants primarily modify the update's 's structure, optimization, or compression. Our work differs by focusing on the semantic content of the update. FVAE-LoRA introduces a VAE with factorized latent spaces (21, 22) and novel ELBO to explicitly separate task-salient information (2)) used for the update from residual information (22), thereby controlling the information encoded in the low-rank adaptation.
Factorization and Disentanglement in VAEs. VAEs 24 learn latent representations by maxi- mizing the ELBO. Significant research focuses on learning disentangled representations, where latent dimensions capture independent factors of data variation (25). Techniques often modify the ELBO, such as VAE (26 FHVAE [27]. Factor VAE [28]. TCVAE [29]. and DIP-VAE [30], or employ an- nealing strategies 3 While we use a VAE and aim for factorization, our goal distinct. We do not seek to disentangle underlying data factors. Instead, FVAE-LoRA employs novel ELBO to factorize the latent space specifically for PEFT: separating information erucial for the downstream task (21) from other sources of variation needed for reconstruction (Z2). This task-conditional factorization within the LoRA update mechanism represents the core novelty our VAE application.
3 Method
In this section, we present our low-rank adaptation approach based on VAE. We begin by briefly reviewing the standard VAE framework, and then introduce our proposed Factorized VAE (FVAE). We highlight key properties of this formulation and conclude by describing how it enables the construction an efficient low-rank adaptation model.
3.1 Variational Autoencoder Objective
Consider a dataset X E Raxd, where observations are generated according to the process ~ pe(x)z), with a latent variable. The goal is recover latent representation that explains the observations x, which requires computing the posterior po (z(xx.. As this posterior is generally intractable, we introduce an approximate distribution 2e (z) x). It can be shown (see Appendix A.1 that the log-likeli ligeliie admits a lower bound, known as the Evidence Lower Bound (ELBO):
8 (x) = Ez [logro(x(2)) - DKL (co(2(x)ll(zz))
The ELBO is used tractable surrogate objective for maximizing log Px()). The first term encourages accurate reconstruction, while the second term regularizes the latent space by aligning the approximate posterior with the prior p(z).
3.2 Factorized Variational Autoencoder Objective
The primary goal of FVAE is to factorize the information contained in x such that is represented by two independent latent variables 2 and z2. This factorization is learned jointly with a downstream task loss applied specifically to 21, which guides the decomposition by encouraging 2 to capture task-relevant information, while 22 absorbs the remaining variability. Classical VAEs serve natural starting point to build such a model.
3.2.1 Preliminaries
The derivation of the classical VAE can be extended by assuming that x arises from a generative
process involving two independent latent variables z and 22, with p(zI, 22) = Pi (z1) p2 (Z2). Addi-
tionally, assume that the approximate posterior factorizes To (Z1, Zz(x) qs. (z) (x) go. (zz x).
Considering Z ~ 90, (zj x) and Z ~ 962 (Z2 the ELBO is given by
CYAEZLAT (x) = E 2y.
[o 21, z2)] DKL (ge, (z x (z))))
DKL (gøs (z2|x) (P2(2)))
z1 21.22
(2)
This objective mirrors the standard VAE but extends it to the multi-latent setting. However, even though both the prior and the variational posterior are factorized, the model not explicitly encour- aged to selectively assign information to Z₁ Z2.
3

---

3.2.2 FVAE
To promote factorization, we introduce a regularization term that penalizes the similarity between lo x)] and the uninformative prior p1 (z1). Since q (zy xx is encouraged to align with pr, this term prevents 4e from encoding information in the same region of the latent space, Incorporating this into Equation (2. we obtain the objective
Zz [2(x)
VAEALAT (x) + E2i log 28,0
Max 0.01.02
(21)
To clarify the role of each component in Equation (3) and relate them to familiar VAE structures, we reorganize the objective using straightforward algebraic manipulations. In doing so, we isolate the standard reconstruction and KI. divergence terms, and separate out the new cross-prior regularizer. Introducing scalar constants a, f and allows us to balance the influence of these components, yielding the structured objective
P2 (22)
6.,0 (x)
[log po (x 22)] 3DKL (g6, (z1(x)) || (z))) + x1.
log
(4)
2g.201 pl (2))
The second term correspond to the Dx in the 8- BAE objective, ensuring that the main latent variable Z1 captures the relevant information for reconstructing while remaining close to its prior. The third term, r, acts as a repulsive regularizer, encouraging the second component Zg to decouple from Z Note that, a priori, could fix 1 and use only and 5 weight the contributions of all the terms. However, we prefer to use all three, as will make the interpretation of each contribution clearer later
3.3 Mechanism of the I' modulator
introduces an indirect interaction between the two encoders by modulating their alignment with their respective priors. Rather than enforcing separation through a direct divergence between posteriors, shifts their latent support via prior-based regularization. To analyze the effect of r, we first rewrite the sum of a mismatch term and discrepancy term, i.c.,
E E log P2 log P1|
+ B2z log - Ez, mog log P1
(5)
mismatch: A
discrepancy: A
where the mismatch term be further equivalently expressed difference of KLs, i.e., A DxI(g0,lpp) (p:)) the following.
This decomposition reveals a meaningful structure, as outlined in
Maximizing the mismatch encourages Gøz to align with its prior p2. This mirrors the behavior expected in a twovvari standard VAE, where each encoder is regularized toward its respective prior. As a result, we retain effective control over the behavior Tos providing a structural safeguard against degenerate aligning with the prior P1. Consequently, 96, is encouraged to preserve or discover features and structures that are distinct from, and not merely reflections of, the assumptions embedded within P1- The mismatch term also highlights that the two priors P1 and P2 should be different, but still partially overlapping. If they identical, some terms will simply cancel out, and if they too far apart, the separation becomes trivial, resulting in no fruitful competition for occupying the latent space. Since the priors usually Gaussian with variance this competition is parameterized by |M u2|: the effect of the mismatch is null when this parameter is null.
unconstrained posterior collapse. In contrast. it disincentivizes qo, from
In addition, we can demonstrate (see Appendix B.I that the discrepancy A is bounded by term depending on the 2- Wasserstein distance, provided that the Hcssian V log Pill I L is bounded. In practice, P1 is typically standard normal A/ f), and go, is diagonal Gaussian. Under these assumptions, the bound becomes:
A < " (1o) (os) +
Σ n3 + t03 Wa(nsr 3
2

---

where My and of are the parameters of 96. Since qa, is typically optimized to approximate p1, the square-root term remains bounded in most settings. Both terms in the bound grow with W2(1e); Uga ) making an effective surrogate for inducing Wasserstein repulsion. In partieular, maximizing increases W2 {02). driving the two encoders apart in a geometrically meaningful way.
3.4 FVAE-LORA
Building upon the FVAE framework, we leverage its ability to split the latent space to gain finer control over the representation, ultimately achieving better performance. To accomplish this, we proceed as illustrated in Figure
For each targeted linear layer, we train FVAE simultaneously with the downstream task, aiming to replace the A matrices used in classical LoRA (see the left side of the figure). During training, the input the target layer is fed into the FVAE compute reconstruction loss based on that input. In parallel, the latent embedding 21 produced by the encoder qá, is passed through a learned matrix B and added to the output of the frozen base weights W. This yields the output Wx Bz At inference time, only 9d, is used to produce the output, either by sampling from it or by taking the mean of the distribution. Note that while we propose using FVAE with LoRA, the method is generic in the sense that it can be applied to give latent space control to any explicit LoRA method. In summary, the loss to be optimized in the proposed FVAE-LORA approach is given by
min Ldownstream-task 0.0
A
Σ
FVAR 2,,0
(xi),
(6)
Lglayer
with À being the hyper-parameter vector of weights assigned to the FVAE loss in each layer.
In practice: Both fö1 and loe are parameterized as diagonal Gaussian distributions, with their means and variances learned by neural networks. The reconstruction term pa 2x is also parameterized by a neural network. The prior P = (O, I) is a standard normal distribution, while P2 is empirically chosen to be centered 1.5. The intuition is to give the two priors distinct non-overlapping "location" in the latent space to initialize and encourage separation. By setting μi at 0 and #2 at L.5, provide clear signal for the repulsive regularizer to push the posteriors apart. See additional insights Appendices and
4 Experimental Results
Motivation. The objective of the experimental evaluation is two fold. First, we aim to comprehen- performance against standard LoRA and its relevant
sively evaluate FVAE-LoRA by comparing variants across diverse image, text, and audio tasks. The specific selection of relevant variants for each domain detailed within the respective modality subsections, guided by the aim to provide the most insightful and relevant benchmarks for each specific context. Second, we seek to empirically validate that FVAE-LORA learns more robust representations by preferentially encoding task-salient information in 2
Overall Setup. To ensure fair comparisons of parameter efficiency for the core adaptation mechanism, the LoRA rank r is set to 16 for all LoRA-based methods throughout our experiments. This rank also corresponds to the dimensionality of the task-salient latent space Z₁ in FVAE-LoRA. baselines well as FVAE- LoRA are applied to the query and key matrices within the transformer models. Detailed hyperparameter settings for FVAE-LORÁ, including the balancing coefficients a, 3 and 6, learning rates, and specific VAE architectural choices for each task, are provided in Appendix [ We also provide a practical guide for selecting the key factorization hyperparameters, 33 and o, in Appendix G
All
4.1
Efficacy of FVAE-LORA for Various Downstream Tasks
4.1.1
Image Tasks
Datasets.
We evaluate FVAE-LORA on six diverse image classification datasets: DTD 32
FuroSAT 33 GTSRB (4, RESISC45 35) SUN397 (36), and SVHN (37. These datasets span
various image types, domains, and complexities.
5

---

Implementation Details. The pre-trained Vision Transformer (ViT-B/16) 3 serves as the back- bone model for all image classification tasks. We compare FVAE-LoRA against full fine-tuning (Full FT) and several LoRA variants, i.e., standard LoRA [12), PiSSA (20, rsLoRA (21, DoRA [15], and OLoRA [38] This broad selection of LoRA variants represents established PEFT methods for image classification. The evaluation metric is top-1 accuracy. Detailed hyperparameters can be found in
D.1
Table 1: Fine-tuning results of ViT-B/16 on image classification tasks. We fine-tune ViT-B/16 using full fine-tuning and LoRA variants across DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN datasets. Bold indicates the highest performance, while underline represents the second-highed
performance.
Method
Params (%)
DTD
EuroSAT
GTSRB
RESISC45
SUN399
SVHN
Average
Full FT
08.12:0.599
98,30=0.47
98.85:0.114
94.35=0.54
69.34:0.59
97.34-0. 0.03
89.38
LoRA
0.7240
74.65=11.08
97.28
96.95-10.56
90.11=0.53
71.11
94.22-0.14
87.39
PISSA
0.7240
74.22=11.69
97.33=0.31
96.95-11.8
89.82-0.37
69.09=0.19
94,83-0.73
87,04
IsLoRA
0.7240
12.23=11.00
97.48=0.21
96.63=0.83
88.04=0.30
67.69=0.32
93.81=0.77
85,98
DoRA
0,7451
1..74=1.91
97.28 80
97.27=10.4
91.72=1.17
71.53:20.25
96,41-0.22
88.32
OLoRA
0.7240
72.23=0.40
96.62
97.08+0046
88.94=0.45
69.64=0.43
94.86-0.30
FVAE-LoRA
0.7311
78.19=0.68
97.78:0.15
97.98:0.56
93.57=0222
73.14::.211
96.55=0.05
89.53
The effectiveness of FVAE-LORA for image classification is shown in Table 1 FVAE- LoRA achieves an average accuracy of 89.53% across six diverse datasets, outperforming LoRA and surpassing variants such as DoRA, all within comparable inference-tine parameter budget.
Results.
Notably, FVAE-LORA's average performance slightly surpasses that of full fine-tuning (89.38%). This result suggests that the structured latent factorization inherent to FVAE-LoRA can guide the model towards learning highly effective adaptations. By explicitly encouraging the disentanglement of task-salient information within 21, FVAE-LORA might be more adept at focusing the ViT backbone on critical visual features for the downstream task, potentially mitigating the risk of overfitting to these
spurious correlations or less generalizable patterns that can sometimes affect full fine-tuning datasets. On challenging datasets such as DTD (characterized by fine- mne-grained textures) and SUN397 (complex scenes), FVAE-LoRA particularly excels, achieving the highest scores and outperforming full fine-tuning. For instance, on SUN397, FVAE-LORA demonstrates a clear advantage, indicative of its capacity distill critical visual cues for complex recognition tasks. While full fine-tuning outperforms all LoRA variants on datasets like EuroSAT and GTSRB, FVAE-LORA consistently stands the leading or a highly competitive PEFT method, often closing the gap significantly (e.g.. achieving 97.78% on EuroSAT, closely trailing Full FT's 98.30%).
The presented results show that FVAE-LORA is able to learn highly effective low-rank updates through a principled approach to information selection.
4.1.2 Text Tasks
Datasets. For natural language tasks, we use two benchmark categories:
1. Commonsense Reasoning:
Training is done a predefined corpus B39 of query-answer
pairs, and the evaluation set includes seven sub-tasks: PIQA [40] (physical commonsense),
SIQA 41 (social interaction understanding), ARC-c and ARC-e 42 (science question
answering), OBQA 43) (multi-hop reasoning over facts), HellaSwag
44 (commonsense
natural language inference)), and WinoGrande 44 (fill-inthhe-blars blank).
2. GLUE Benchmark: A subset of the GLUE 46 is used, comprising SST2 (sentiment analysis), CoLA (linguistic acceptability). QNLI qquestion-nnswertinn NLI), MRPC (para- phrase detection), RTE (textual entailment), STSB (semantic textual similarity), and WNLI (coreference resolution).
We employ Llama-3-8B S for the commonsense reasoning tasks and roberta-base [47] for the GLUE benchmark tasks. For commonsense reasoning tasks, we compare against Prompt Tuning 11. P-Tuning
Implementation Details.
[0 48 standard LoRA, and HiRA [3]. For completeness,

---

we also present the performance of ChatGPT taken from [5]. Considering the computational cost of LLM fine-tuning, our LoRA-based comparisons focus on standard LoRA and HiRA, as HiRA has recently demonstrated strong performance, offering relevant and challenging benchmark in this setting. For roberta-base on GLUE, comparisons are made against Full FT and standard LoRA. This allows for direct assessment of FVAE-LoRA's parameter efficiency relative to the crucial full fine-tuning upper bound and the widely adopted LoRA baseline. Evaluation uses accuracy for commonsense tasks following [13] and standard GLUE metrics (Matthews Correlation for CoLA, Pearson Correlation for STSB, Accuracy for the rest). Detailed hyperparameters can be found D.2
Table 2: Accuracy comparison among various PEFT methods commonsense reasoning datasets for Llama-3-8B, Bold indicates the best performance, while underline represents the second-best performance. ChatGPT performance values are taken from [5], whereas Prompt Tuning and P-
Tuning from 13
Model
Method
Params (%)
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaSwag
WinoGrande
Average
CharGPT
85,40
68.50
79.90
74.80
77.57
Prompt Tuning
0,0010
45.05
36.13
31.57
32.74
29.20
14.01
50.12
34.12
P- P-Tuning
0.6240
11.64
8.19
7.42
8.63
9.60
1.77
37.65
12.13
Llama-3-8B
LoRA
0.0848
80.74
75.59
67.58
82.11
75,20
85.73
77.82
77.82
HiRA
0.0848
88.63
80.40
81.66
93.56
87.20
94.48
85.87
87.40
FVAE-LORA
0,0850
88.96
81,58
81.06
92.72
86.20
95,30
88,95
87.82
Results Commonsense Reasoning using Llama-3-8B model. repoet the performance of FVAE-LORA and baselines across seven commonsense reasoning benchmarks using the LLaMA- 3-8B model. Our approach achieves the highest average accuracy of 87.82%, outperforming both the strong HiRA baseline (87 (87.40% and LoRA (77.82 82%%) under comparable inference-time parameter budgets. These results indicate that FVAE-LoRA's strategy of factorizing latent information is particularly beneficial for complex reasoning tasks in LLMs. By explicitly guiding the z₁ latent space to capture task sallent semantic and contextual cues necessary for reasoning. FVAE-LoRA enables Llama-3-8B to make more accurate inferences.
Notably, FVAE-LORA demonstrates strong individual performances tasks like HellaSwag (95.30%) and WinoGrande (88.95%), which require nuanced understanding of everyday situations and dis- ambiguation. This suggests that the information isolated in z is indeed critical for these types of reasoning, allowing the LLM to leverage its capabilities more effectively than with less structured adaptation techniques. The ability improve upon already powerful models like Llama-3-8B with such parameter efficiency highlights the potential of FVAE-LORA for targeted capability enhancement in large language models.
Table 3: Results of fine-tuning roberta-base using full fine-tuning and LoRA on subset of the GLUE datasets. Bold indicates the best results, while underline represents the second-best results.
Method
Params
SST2
CoLA
QNLI
MRPC
RTR
STSB
WNLI
Average
Full FT
94.77=00.5
1.16
91.97 91.97-0.05
89 40.40:00700
19:53-1.31
90.30-0.29
56. 30000000000
80.67
LoRA
0.4710
93.97=0.47
59.60=1.022
91 071
88.73:0.61
77.8720.74
88 18.90=0.45
57.73=2.05
79.81
LVAE-
94. 10:4043
60:37=0049
91.63
8..53
79.90=0.85
88.60-0.16
64.33-2.38
81.21
Results on GLUE benchmark. Table 3 presents the performance of FVAE-LORA when adapting the roberta-bass base model subset of the GLUE benchmark. Our method achieves the highest average score (81.2) 1), outperforming both full fine-tuning (80.67) and standard LoRA (79.81). Notably, FVAE-LoRA shows partieular strength on roberta- base demonstrates that the benefits of FVAE-LORAY explicit latent factorization are not confined large-scale models like Llama-3-8B (as seen in commonsense reasoning tasks), but also translate effectively to smaller, yet widely utilized encoder models. The ability to enhance these more moderately-sine architectures suggests that FVAE-LoRA's principled approach to focusing adaptations via 21 on task-critical linguistic features is robust across different model scales.
tasks like MRPC and WNLI This strong performance
4.1.3 Audio Tasks
Datasets. We conduct automatic speech recognition (ASR) on the TIMIT acoustic-phonetic corpus 49 for phoneme recognition.
7

---

group Implementation Details. The pre-trained Lera model [5] serves as the backbone. Fine-tuning utilizes the Connectionist Temporal Clas- sification loss [50]. We compare against Full FT and standard LoRA. Performance is measured by
Table Fine-tuning results of Wev2Vee2- Large on the TIMIT speech recognition task using CTC loss. Bold indicates the best PER (4), underline the second-bess
Method
Params (%)
TIMIT
Phoneme Error Rate (PER). Detailed hyperparame- ters are provided in Appendix D.3
PER (4)
Full FT
7.48
Results. As shown in Table FVAE-LoRA
LoRA
0.4961
9.38
achieves PER of 8.09 on TIMIT, outperforming standard LoRA and approaching the performance of full fine-tuning (7.48), demonstrating its effectiveness for ASR.
FVAE-LORA
0.4999
8.09
4.2 Probing Latent Factorization via Spurious Correlation
To empirically validate our hypothesis that FVAE-LoRA learns more robust representations by preferentially encoding task-salient information in Z1. we conduct experiments using datasets with controlled spurious correlations. Spurious correlations occur when input features are statistically associated with target labels without true causal link [51 52 53 potentially misleading models and hindering generalization, especially on disentanglement mechanism renders it more robust to such misleading
or minority-grome data. Our aim is to
assess whether cues compared to standard LoRA.
Experimental Design. We leverage datasets where spurious attributes (e.g., background scene) are intentionally correlated with the true class labels (e.g., object category) during training. For example, "landbird" might predominantly appear against "land" background, and "waterbird" against "water". Effective factorization should enable the model to learn the true object category via Zi. irrespective of the potentially misleading background. Figure 2 illustrates Diluusratlss this concept, distinguishing between an input image (x), its core features (Xeore). ) and its spurious features (X.purious).
Following prior works [54] 51 52 53, 55 [56], we consider three standard benchmarks to introduce spurious correlations: Waterbirds [56], where bird type (landbird vs. waterbird) is correlated with background (land vs. water); CelebA (56), where target attribute (e.g., blonde hair) might be correlated with another attribute (e.g., being female); and Animals [57], a larger-scale dataset derived from ImageNet (58] with four animal classes spuriously correlated with background types (e.g., waterbirds with water, small dogs with indoor scenes). These datasets are structured into groups based on combinations of true labels and spurious attributes, with varying
Datasets.
ratios between training and test splits (details in Appendix C.T and Table
Xapuriuus
Figure 2: Random samples drawn from the train split of the Animals dataset, illustrating an original image (x), its core object features (Xcore), and its spurious background features (Xopurious). )
Implementation Details and Evaluation Metrics. We adapt the ViT-B/16 3 backbone us- ing LoRA and our proposed FVAE-LORA. Following common practice in literature [59 60 61]. performance is evaluated using three key metrics:
Accuracy on the test subgroup where the model performs poorest, indicating robustness to spurious correlations and performance minority groups
Worst-Group Accuracy (WG):

---

Average Accuracy (AVG); Standard overall accuracy on the test set.
Accuracy Disparity: The absolute difference WG AVG), quantifying the performance variation across groups. smaller disparity suggests more uniform and equitable perfor- mance.
Table 5: Fine-tuning results of ViT-B/16 on spurious correlation benchmarks. We compare LoRA with FVAE-LORA on ANIMALS groups, 4 classes), WATERBIRDS (4 groups, 2 classes), and CELEBA (4 groups, 2 classes) datasets.
Method
Params (%)
ANIMALS
WATERBIRDS
CELEBA
Disparity
WG
AVG
AVG
WG
AVG
WG AVG
LORA
0.7240
54.79:18008
88
18.20-1117
75.49=0.9
90.39+0.78
40.00+3355
9.09::0.2
34.8
EVAE-LORA
0.7311
4,80114883
09.95 75785
90.9910.51
43.3316668
95.77:10.18
31.71
Results. JJummaiss the performance of standard LoRA, and FVAE-Loke on the spurious correlation benchmarks. Across all datasets, FVAE-LORA consistently achieves higher WG and lower Accuracy Disparity compared to LoRA, while maintaining competitive AVG. These findings strongly suggest that FVAF-LoRA is less susceptible to being misled by spurious features present in the training data. We attribute this enhanced robustness to the explicit factorization encouraged by our novel ELBO. By compelling 2 to capture genuinely task-relevant, causal features and relegating other variations, FVAE-LoRA learns a more robust adaptation. This leads to improved generalization, particularly on minority groups where spurious cues are often unreliable or reversed, thereby validating the intended robust learning mechanism of our proposed method.
Table 6: Ablation study comparing our FVAR-LORA when fine-tuning ViT-B/16 to a two-latent- variable VAE (VAE2LAT, as defined in Eq. 2 ) and (-VAE2LAT, the VAEAE version of VAELAT (where all the DKL. terms are multiplied by 1O). Results are presented on DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN. Bold indicates the highest results, while underlined indicates the second-hige highhs..
Method
DTD
EuroSAT
GTSRB
RESISC45
SUN397
SVHN
Average
FVAE- LoRA (Proposed)
78.19:10.8
97.78-10.15
97.98:10.56
93,57-10.22
73.14:10221
96.55
89,53
V-E2LAL
(where 3 = 10)
77.1620043
96.86220115
95.75:20.46
89.46:2.133
72.9120.32
91.38:0.69
87.29
VAE2LAT
3
96.0660080
96.6420.59
94.38:0.94
88.4220.32
71.68:0.54
91.50:0.58
86.43
4.3
Ablation Studies
To demonstrate the relevance of introducing the regularization term in Equation (3)) we replicate our image results using the two-variable VAE model (2) and its equivalent for 3-VAE with two latent variables (where the two KL divergences have been multiplied by 10; see Equation (A.3) D. The results can be seen Table 6 The baseline model performs the worst across all datasets. The 8-VAE with two latent variables shows some improvement, but it is still outperformed by our proposed method.
5 Conclusions
We introduced Factorized Variational Autoencoder LoRA (FVAE-LORA). a novel PEFT method designed explicitly disentangle task-salient information within the LoRA framework. By employing a VAE with two latent spaces, 21 (task-salient) and (residual), and specialized ELBO, FVAE- LoRA ensures that the adaptive updates are primarily driven by task-critical features learned in Our comprehensive evaluations on diverse text, audio, and image benchmarks demonstrated that FVAE-LORA consistently surpasses standard LoRA performance. Crucially, experiments on datasets with spurious correlations empirically confirmed that FVAE-LORA's factorization leads to more robust representations, as evidenced by improved worst-group accuracy. FVAE-LORA highlights the potential of latent space factorization for enhancing parameter-efficient fine-tuning.

---

Acknowledgments
Shashi Kumar was partially supported by the EU Horizon 2020 project ELOQUENCE (grant number 101070558). Yacouba Kaloga was partially supported by Swiss National Science Foundation project no CRSII5_202222 on Characterisation of motor speech disorders and processes.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[2] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, al. Segment anything. In ICCV, 2023.
[3] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
[4] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[5] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations, Advances in neural information processing systems, 33:12449-12460, 2020.
[6] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. International Conference on Machine Learning, pages 28492-28518, Honolulu, USA, July 2023.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[8] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
[9] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In /CML, 2019.
[10] X Liu, Zheng, Z Du, M Ding, Y Qian, Yang, and Tang. Gpt understands, too. arxiv preprint arxiv: 210310385. 2021.
[11] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv:2104.086)) 2021.
[12] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et Lora: Low-rank adaptation of large language models. ICLR, 2022.
[13] Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[14] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012. /3255,, 2020.
[15] Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In ICML, 2024.
10

---

[16] Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation, 2021.
[17] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022.
[18] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning, arXiv preprint arXiv:2303. 10512, 2023.
[19] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024.
[20] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404. 02944,, 2024.
[21] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arX.v:2312..03722, 2023.
[22] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, Advances in neural information processing systems, 36:10088- 10115, 2023.
[23] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-awar low-rank adaptation of large language models. arXiv preprint 2rXiv:2309.14717, 2023.
[24] Diederik Kingma and Max Welling. Auto-encoding variational bayes, 2022.
[25] F Locatello, S Bauer, M Lucic, G Rätsch, S Gelly, B Schölkopf, and O Bachem. Challenging common assumptions the unsupervised learning disentangled representations. arxiv preprint arxiv: 1811. 2018.
[26] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
[27] Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in neural information processing systems, 30, 2017.
[28] Hyunjik Kim and Andriy Mnih. Disentangling by factorising, 2019.
[29] Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement variational autoencoders, 2019.
[30] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations, 2018.
[31] Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-vae, 2018.
[32] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[33] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2117-2226, 2019.
[34] Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The german traffic sign detection benchmark. In IJCNN, 2013.

---

[35] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10;:185-18883, 2017.
[36] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to 20o. In CVPR, 2010.
[37] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco. Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NeurlPS workshop. 2011.
[38] Kerim Büyükakyüz. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprini arXiv:2406.01775 2024.
[39] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ec-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- He-Wei Lee. Lim-adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv preprint arXiv:2304.0999 2023.
[40] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
[41] Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv: 1904.0904909 2019.
[42] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
2018
[43] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv:1809.0220 02789, 2018.
[44] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprini 19Xiv:990.0788.. 2019.
[45] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[46] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
[47] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprini arXiv: / 1692, 2019.
[48] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 61-68. Dublin, Ireland, May 2022. Association for Computational Linguistioonalinnuuuissssss.
[49] John S Garofolo, Lori F Lamel, William M Fisher, David S Pallett, Nancy L. Dahlgren, Victor Zue, and Jonathan G Fiscus. Timit acoustic-phonetic continuous speech corpus. (No Title), 1993.
[50] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. International Conference on Machine learning, pages 369-376, Pittsburgh, USA, June 2006.
[51] Guanwen Qiu, Da Kuang, and Surbhi Goel. Complexity matters: Feature learning in the presence of spurious correlations. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 41658-41699 2024.
12

---

[52] Gautam Sreekumar and Vishnu Naresh Boddeti. Spurious correlations and where to find them. arXiv preprint (rXXi:2308.10442 2023.
[53] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparamelerizattonn exacerbates spurious correlations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages
PMLR, 13-18 Jul 2020.
[54] Justin Cui, Ruochen Wang, Yuanhao Xiong, and Cho-Jui Hsieh. Ameliorate spurious correla- tions in dataset condensation. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 9696-9721, 2024.
[55] Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness spurious correlations via human annotations. In Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th In- termational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9109-9119, 9119. PMLR, 13-18 Jul 2020.
[56] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of wild distribution shifts. In International conference machine learning, pages 5637-5664. PMLR, 2021.
[57] Siddharth Joshi, Yu Yang, Yihao Xue, Wenhan Yang, and Baharan Mirzasoleiman. Challenges and opportunities in improving worst-group generalization in presence of spurious features, 2025.
[58] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
[59] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
[60] Elliot Creager, learning. In International Conference Machine Learning, pages 2189-2200 PMLR, 2021.
Jacobsen, and Richard Zemel. Environment inference for invariant
[61] Evan Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781-6792. PMLR, 2021.
13

---

A Variational Auto-Encoder
A.1 VAE Objective Derivation
We derive the Evidence Lower Bound (ELBO) by starting from the marginal log- -oglliielhooo::
log po (x) - (x|x)
logpo(())
po(
Ex
log
Po(z|x)
((z)
+ +log (x(2)
Ex (x))x)
log po (x(z) + log
{x(z)x)
po (z(xx))
[og [x|2]] DkL. (1s (2) xx) p())) + DKL. (10(2|x) ll (o(x))))
The last term is always non-negative, which justifies interpreting the remaining two terms as lower bound, i.e..
lo C. (x),
with
L (x) = Ex (* x) [log (x((2)) - DKL (go(z)x) [[p())
A.2 VAE2LAT: VAE with 2 Latent Variables Objective Derivation
We simply start from the ELBO previously derived with two variables, i.e.,
VAEELAT ( Exzas (21,2))))
[log po (x|21, DEL (20 22|x) p(z1, z2))
Applying the independence assumption, obtain
AVAELAA (x) Ez rogo [logpo(X|zz 22)] - DRL. (21lx) (z.)) DKL. (1oz (22[x) |1 (22))) 60.0
A.3 3-VAE2LAT: A 3-VAE with 2 latents variables
The loss 3- VAE2LAT, i.e., a straightffoward extension of 3 VAE to two latent variables is given
by studies 4.3
0000
B VAE2LAT
(x) B2 geo (21 (x) [logpo(((21,22)) 3DKL. (2o, [x) (z(1)) BDRL. (1oz (22/x) |l (2(2))
22 Ren (82)x)
(8)
This formulation is used in the ablation studies in Section 4.3
B
FVAE
B.1 Bounding the Discrepancy Term via the 2- Wassersttii Distance
We bound the discrepancy
Eg (zog((zz) Ezoge, [log Pi (z)],
assuming only that the log-prior [(=) log (2) is C with globally bounded Hessian:
|| & L V2 e Rd.
(H)
14

---

Step 1- Second-order Taylor control. remainder gives
For any two points 22, Taylor's formula with integral
(1- (
(f
f(22)-((z)) = (Vi(zı), 22-21) + (22-21)
+ 8 - - (82-21).
Bounding the remainder using assumption (H) gives the point- wise inequality
L
(f(z)) - f(a)) (V)() 22 21)l $ 2
2 2111²
(A)
Step 2 - Integrate over a coupling. L E [I 9oa) be any coupling of the two distributions, and write (z1, Z2) ~ y, d Zg 21. Taking expectations in (A), applying the triangle inequality, and then Cauchy-Scheee
to the linear term,
A
< 2
1|] + (VS())))
E |||] +
VB V v Eyyld
"lincur term expectation"
Now minimise the rightmost expression over 7. Since the function g(x) 4²²² + Ca (for
V EBg, ffll 2 0) is non-decreasing for æ = /E, |d| 2 0, the infimum is attained when E, ||]]
is minimized. The infimum of E,y |d]] is W 102) Hence:
L S
A
(2 96.)) B 2
V logg (z)) We(ges 962).
Step - Specialization to Gaussian case. Assume = =N(0. 1) and Q61 = N diag(o")
Then the gradient becomes V Voogpi(z) zz, and the expectation simplifies as:
Egg | (o)g Bgg [|| = Σ 13 o3.
Therefore, the bound becomes:
A
<
1 2
(2 Io2)
Σ 13 E13+ + +03 Wel 962).
Since qp, is trained to approximate Pt. the square-root term is typically bounded in practice. Hence, both terms contribute increasing W2(gon {oz), and the discrepancy A serves as an effective Wasserstein repulsion.
C Dataset Details
C.1 Spurious Correlation Experiments
Table 7: Statistics of the datasets used in the spurious experiment.
Dataset
SpuCoAnimals Waterbirds
CelebA
# Classes
4
# Groups
8
4
Train
42000
4795
162770
Validation
2100
1199
19867
Test
4000
5794
19962
Class Ratio
25:25:25:25
76.8:23.2
85:15
D Hyperparameters
This section details the hyperparameters used for the experiments presented in the main paper. For all LoR Abaased methods, including FVAE-LORA, the LoRA rank (r) was set to 16, and LoRA was applied to the query and key matrices of the attention layers. The latent dimension of z in FVAE-LoRA corresponds to this LoRA rank.
15

---

D.1 Image Experiments
The following hyperparameters were used for fine-tuning ViT-B/16 on DTD, EuroSAT, GTSRB, RESISC45, SUN397, and SVHN datasets.
Table 8: Hyperparameters for Image Classification tasks using ViT-B/16.
Parameter
Value Setting
General Training Purameters
Optimizer
AdamW x 10-3
Learning Rate
I.R Scheduler
Linear
Warmup Ratio
Batch Size
32
Number of Epochs
30
Weight Decay
0.01
Seeds
1,2, 42
LoRA Parameters
LoRA Rank (r)
LoRA Dropout
0.1
FVAE-LoRA Specific Parameters
Latent Dim. Zi
16 (same as LoRA rank)
Latent Dim. zg
16
FVAE Jo, (z, Enc. Arch.
Lincar dim(z,) RELL HiddenState,, Linesr (14 log o².)
FVAE Po (x|z1, 22) Dec. Arch.
Concat(z), z2)
Linear, Hp = 128 Linear - x (Input Dim)
Prior (Z1)
((0.T) T)
Prior Pa (Za)
(11.5,1)
(Eq.
x 10
ELBO Coeff. a (Reconstr.)
ELBO Coeff. B (KI. q1 l(p))
or 10
ELBO Coeff. 5
16

---

D.2 Text Experiments
Table 9: Hyperparameters for Commonsense Reasoning using Llama-3-8B.
Parameter
Value Setting
General Training Parameters
Optimizer
AdamW
Learning Rate
1 10
(LoRA, HiRA), 3 10 (FVAE-LORA)
LR Scheduler
Linear
Warmup Steps
100
Batch Size
Gradient Accumulation Steps
4
Number of Epochs
Weight Decay
0.0
Seed
42
LoRA Parameters
LoRA Rank (r)
16
LoRA Dropout
0.1
Target Modules
proj, proj
FVAE-LoRA Specific Parameters
Latent Dim. Z1
16
Latent Dim. Z2
FVAE (o, (z/(xc Enc. Arch.
x
Lineur dim(z,) ReLL HiddenStatey, Linear (14 log 2
FVAE pa (x|21, k2) Dec. Arch.
Concatizi, zz)
Lincar,
Hn 128 Ret.U Linear (Input Dim)
Prior Pi (z))
N(O,) I)
Prior pa (2))
().5, 1 x 10
À (Eq.
ELBO Coeff. a (Reconstr.)
ELBO Coeff' / (KL qı (Ip))
1 or 10
ELBO Coeff. 5
Table 10: Hyperparameters for GLUE Benchmark tasks using RoBERTa-base.
Parameter
Value / Setting
General Training Parameters
Optimizer
AdamW
Learning Rate
10-4
LR Scheduler
Linear
Warmup Ratio
0,06
Batch Size
32
Number of Epochs
30
Seed
1, 42
LoRA Parameters
LoRA Rank (r)
LoRA Dropout
0.1
LoRA Specific Parameters
Latent Dim. Zi
16
Latent Dim. zg
16
FVAE Jo, (x,|x) Enc. Arch.
Lincar dim(z,) RaLU HiddenStates, Linesr (14 log o².)
FVAE Po (x|z1, 22) Dec. Arch.
Concat(z), z2)
Linear, Hp = 128 Linear - x (Input Dim)
Prior (z1)
((0. \(1.5,1) 1x 10 or 14
Prior Pa (Z())
(Eq.
ELBO Coeff. a (Reconstr.)
0.1 or
ELBO Coeff. B (KI. q1 l(p))
or 10
ELBO Coeff. 5
17

---

D.3 Audio Experiments
The following hyperparameters used for fine-tuning Wav2Vec2-Large the TIMIT dataset.
Table 11: Hyperparameters for ASR on TIMIT using Wav2Vec2-LLagge.
Parameter
Value Setting
General Training Parameters
Optimizer
AdamW
Learning Rate
5 x 10
(Full FT), 5 x 10 (LoRA and FVAB-LORA)
LR Scheduler
Linear
Warmup Steps
500
Batch Size
Number of Epochs
30
Weight Decay
0,005
CTC Loss Reduction
Sum
LoRA Parameters (Standard LoRA & FVAE-LORA's LoRA part)
LoRA Rank (r)
16
LoRA Dropout
FVAE-LoRA Specific Parameters
Latent Dim. Z1
Latent Dim. Z2
16
FVAE To; (x//x) Enc. Arch.
Linear dim(z;) ReLU HiddenStates,
Linear (1e log 222)
FVAE pa (x/21, 22) Dec. Arch.
Concat(z), kz)
Linear
Hn 128
ReLUU Linear (Input Dim)
Prior pi (z1)
NO, 1)
Prior (z2)
()..5.
(Eq.
x 10-3
ELBO Coeff' a (Reconstr.)
ELBO Coeff, 3 (KL qillpı)
ELBO Coeff. 5
D.4 Spurious Correlation Experiments
These experiments (Waterbirds, CelebA, Animals) used ViT-B/16 the backbone. Base training and LoRA parameters are similar to those in Section D.I with specific FVAE-LoRA coefficients tuned for robustness.
Table 12: Key FVAE-LoRA Hyperparameters for Spurious Correlation tasks (ViT-B/16).
Parameter
Value / Setting
General & LoRA Parameters
See Table for most general and LoRA parameters.
Batch Size
128
Number of Epachs
30
FVAE-LORA Specific Parameters
Latent Dim. ži
16
Latent Dim. ze
16 Similar to Table
FVAE Architecture
Prior (z1)
((O.
Prior (2(22)
N(1.5, 1 10 or x 10
A (Eq. 6
ELBO Coeff. (Reconstr.)
0.1 1
BLBO Coeff. /8 (KI. (Ilp)))
1 or 10
ELBO Coeff. 5
1
18

---

E
Early Attempts at Latent Space Factorization
The most straightforward way to enforce repulsion between ž1 and Z2 in the two-variable ELBO (see would be to augment the ELBO with the terms:
Eq.
+DK (góz Il) + DKL (gør ll2)).
However, early experiments with this approach yielded poor results, fact, performance was worse than with LoRA. From theoretical standpoint, adding such terms to the two-variable ELBO ef- fectively cancels out Gøg and lo: from the objective, leading instead to direct repulsion between the priors P1 and P2, which is not desirable. Other similar approaches such as directly repelling 4e: and qøg suffered from the same issue. We found that all overly symmetric and direct formulations, including two-term symmetric variants, were ultimately unfruitful. To avoid this cancellation effect, we instead propose an indirect way to introduce repulsion between Z| and Z2 by introducing a cross-term between the parametric encoder dez and the latent distribution P1. This solution is theoret- ically grounded: we show that it induces a geometric separation, measured through a Wasserstein upper bound, between the two encoders qø, and 20. It is also supported by experimental results, outperforming LoRA across all tested modalities.
F Additional Insights in FVAE-LORA
Regarding the objective in Eq. 4 our proposed loss is novel objective derived from and inspired by the Evidence Lower Bound, p(x). By introducing the repulsive regularization term I, we modify the standard ELBO to enforce factorization between the latent spaces. This term is essential for the method's success, but it means the objective no longer serves as formal lower bound on the data log-likelln oggliiklihood in the traditional VAE sense.
not strict lower bound on the marginal log-likelioood
FVAE-LoRA intentionally sacrifices static weight merging to enable a more powerful dynamic, input-dependent adaptation. By computing the adaptation specifically for each input x, our model learns more robust and fine-grained representations. We believe this dynamic mechanism is the key to its performance edge, capability validated by our strong results on the spurious correlation benchmarks. This trade-off is therefore central to achieving the higher performance and robustness we demonstrate.
Note that simply reducing the rank of LoRA is a simple and efficient form of regularization, however It compresses all information flowing through the adapter, without distinguishing between features that are useful. irrelevant, or even detrimental to the downstream task. Our hypothesis is that large foundation models, pretrained on vast and general datasets, contain rich and entangled set of features. For any specific downstream task, some features are highly relevant (the "signal"), some are irrelevant but harmless, and some are actively harmful. The most prominent example of these detrimental features are spurious correlations (e.g., water background being correlated with "waterbird" label). standard fine-tume fine-tuning process, which optimizes a task- specific loss, may still latch onto these spurious features because they are prevalent in the training data and help minimize the training loss. This leads to poor generalization on data where that correlation is broken. This is why FVAE-LORA is designed to be more intelligent filter. Its goal is not just to compress, but to actively separate and isolate these different types of information. By using two latent spaces (Z₃ and 22) and our novel factorization objective, while relegating the residual, non-essential, or spurious information to Z2. The most direct validation of this rationale is in our spurious correlation experiments (Section 4.2). These results show that FVAE-LORA is significantly more robust to misleading features than standard LoRA, confirming that it successfully learns rely on the core features isolated within 21. This ability to "denoise" the adaptation is why it ultimately achieves better and more reliable performance.
encourage the model to encode task-salient, causal information in zı
G A Practical Guide on Hyperparamters Selection
The factorization in FVAE-LORA is governed by a subtle equilibrium between reconstruction and regularization, enforced by our ELBO objective. The key hyperparameters, / and 6, control this balance.
19

---

3 and the Task-Salient Space (z1). The & parameter controls the KL divergence on Z1, our task-salient latent space. Its role is critical, enforces structured and efficient representation of the task-salient features. To understand its impact, we experimented with a wide range of values.
A significantly lower value, such tasks. This is because near-zero 3 effectively removes the KL divergence term, freeing the encoder for 21 learn an unconstrained and arbitrarily complex representation. This removes the crucial pressure for the learned posterior go (Z) x) to align with the prior p1 (Z1), leading to overfitting and loss of generalization. This result is not merely a poor tuning choice; it is critical evidence that enforcing this prior alignment is essential for learning robust and meaningful task-salient space.
0.1, led drastic degradation in performance across all
Conversely, explored much higher value of 3 100. While this yielded marginal improvements over 3 = 10 on some specific tasks, the gains were not significant enough to justify such a strong constraint. An overly large / can create information bottleneck, punishing the model so heavily for deviating from the prior that it struggles to encode sufficient task-specific information in 21.
This evidence from both extremes reveals necessary balance. The optimal values, which we found to be in the range of to 10, are large enough to enforce a structured, regularized space but not so large as to prevent the learning of useful features.
ò and Latent Space Separation. The 8 parameter controls the strength our repulsive regularizer, r, which is the primary mechanism for enforcing factorization between the task saskest space (zz) and the residual space (Z2). Our empirical results consistently show that repulsive force to achieve this separation effectively, as was demonstrated in the spurious correlation robust and generally optimal default.
provides sufficient
experiments. We recommend 8
For practical application, users can start with = 1 and tune 3 (typically between 1 and 10) to adjust the regularization on the learned task-salient features. This is crucial point for the practical application of our method.
H
Computational Cost Analysis
Our empirical results on image classification tasks show that the training time for FVAE-LoRA is approximately 30% higher than that of the strong DoRA baseline. This increase primarily due to the additional forward and backward pass through the VAE's decoder during the training phase. However, the inference-time overhead is significantly lower because only the lightweight z encoder is used.
I Future Work
A particularly exciting avenue for future work lies in exploiting the inherent generative capabilities of the FVAE framework. Key directions will include exploiting the generative capabilities of the FVAE decoder for principled data augmentation, applying our latent factorization principle to other PEFT methods beyond LoRA, exploring approximate high-rank adaptation methods like HiRA, and exploring architectural enhancements such as allocating adaptive parameter budgets or different latent space ranks different layers.
J Limitations
While FVAE-LoRA demonstrates promising results across diverse modalities, several limitations of the current work remain. First, the LoRA rank is fixed to value of 16 across all experiments. Although this ensures consistent parameter budgets, it may not represent the optimal configuration for each task domain, potentially limiting performance. Second, FVAE-LoRA and all baselines are applied only to the query and key matrices the transformer models. This restricted application may overlook potential gains from adapting other components such as value matrices or feedforward layers.
Furthermore, detailed hyperparameter settings and VAE-specific architectural choices are provided in the appendix. This separation may hinder reproducibility, for readers interested in extending the approach. Nevertheless, the code will be open-soureed after publication.
20

---

In addition, while modality-specie baselines are chosen with the goal of providing meaningful comparisons, do not evaluate against stronger non-LoRA or non-factorized alternatives, which may offer a more comprehensive picture of relative performance. Further, the paper does not report the computational cost training or inference, which important for assessing the practical deployment potential of the method, especially in cssuurc-constrained environments.
Finally, practical limitation of FVAE-LoRA that its adapter weights cannot be merged back into the original base model after training. This stands in contrast to some LoRA-based methods that allow for such weight merging, which can simplify inference or reduce model complexity at deployment time.
K Broader Impacts
FVAE-LoRA advances parameter-efficient fine-tuning by enabling a factorized latent representation. Potential positíve impacts include more effective and robust model adaptation across modalities, potentially leading to improved performance, resource efficiency, and more reliable Al systems, especially in handling spurious correlations, demonstrated in experiments. This can also enhance accessibility to powerful AI capabilities for a wider range of researchers and developers.
However, techniques that improve the adaptability of large foundation models also carry inherent risks. Easier and more effective fine-tuning could lower barriers for misuse in sensitive areas such as the generation of sophisticated disinformation or the development of enhanced surveillance tools. While FVAE-LoRA aims to disentangle task-salient information, it does not inherently mitigate biases that may be present in the original pre-trained models or the fine-tuning data. Indeed, such biases could potentially be concentrated or even amplified within the task-salient latent space not proactively identified and addressed.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims the abstract and introduction accurately reflect the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The authors provide sufficient information regarding limitations of the current work the appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer:
[Yes]
Justification: The authors provide for all theoretical results the full set of assumptions accompanied by their proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The authors provide all the necessary information need to reproduce the exper- iments, including hyperparameters, optimizer, model architectures, etc. in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Access to the data and code will be provided in a following step.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ptapp..cc/pbblicguuu.ee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are provided throughout the paper and any remaining details and hyperparameters are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The authors provide standard deviation for all experiments across different
runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification:
The authors have not provided this information at the current stage.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics htttpp///porrrps.cc////bbli/iiit
Answer: [Yes]
Justification: The paper raises no ethical concerns and complies with the NeurlPS guidelines. No high-risk applications or data are involved, and fairness and privacy considerations are acknowledged where applicable.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The authors include a "Broader Impacts" section that discusses potential benefits and risks in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The models released do not pose potential for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and code used properly cited, and licenses referenced. There is no evidence of license violations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Any new assets introduced include usage instructions, and documentation appears sufficient for independent use.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: There are no experiments related to crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: There is no research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:e///eerrrpooccc///oeeeerere for what should or should not be described.
28


Reviews:
[{"review_id": "zMGo2gPmSv", "reviewer": "Reviewer_9e3a", "summary": "This paper introduces FVAE-LoRA , a variant of LoRA that incorporates VAE components to learn two distinct latent spaces within each LoRA module. In FVAE-LoRA, the down-projection matrix $A$ is decomposed into two submatrices, which are then learned in a VAE-style framework. During inference, these two matrices can be merged into a single matrix. The method is evaluated across a range of image, text, and audio benchmarks to comprehensively assess its performance.", "questions": "I think overall it's a nice paper, though I have concerns to be addressed, see Weaknesses. My initial rating is BA.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written and clearly structured.\n* The idea of integrating a VAE into LoRA is interesting, VAE naturally brings more variabilities into the low-dimensional space, and the same inference format is an extra plus.\n* The evaluation across diverse modalities, image, text, and audio tasks, demonstrates the versatility and general applicability of the method.\n\nWeaknesses:\n* Performance wise,  the advantage of FVAE-LoRA over HiRA is not clearly demonstrated. While both methods aim to factorize LoRA matrices (HiRA via a Hadamard product of $BA$ and $W_0$), HiRA appears to be simpler to train due to the absence of additional VAE-related loss terms. As a result, it remains unclear what practical benefits FVAE-LoRA offers over HiRA in standard evaluations.\n* The integration of VAE into LoRA opens up promising directions beyond conventional fine-tuning tasks. It would be valuable to explore the broader capabilities of FVAE-LoRA in more specialized settings. For instance, the structured latent space introduced by the VAE could be particularly beneficial in continual learning scenarios, where maintaining separation between task-specific representations may help mitigate catastrophic forgetting. Similarly, the generative nature of the VAE could enable more interpretable and controllable manipulation of the low-rank space, which might be advantageous in tasks involving conditional generation etc. Personally, I think FVAE-LoRA might shine more on these kinds of tasks, rather than the standard fine-tuning benchmarks.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns. Since no new evidence/results are reported, having read other reviews, I'd keep my original positive score of 4.", "rebuttal": ""}, {"review_id": "bUCJdGLJTb", "reviewer": "Reviewer_5sDq", "summary": "In this papers the authors propose a novel PEFT technique based on the popular LoRA method (low rank adaptation), that leverages latent disentanglement obtained via variational auto-encoders (VAE) and a novel disentanglement loss (FVAE-LoRA). In LoRA, one fine-tunes a low rank decomposition $\\mathbf{B} \\mathbf{A}$ ($\\mathbf{A} \\in \\mathbb{R}^{r \\times d}$, \\mathbf{B} \\in \\mathbb{R}^{k \\times r}, with the rank $r << \\min(k, d)$) of linear weights that is summed to the frozen weights of the base model. The authors suggest instead to replace the projection $\\mathbf{A}$ with two joint VAE posteriors $q\\_1$, $q\\_2$ mapping to two latent variables $\\mathbf{z}\\_1$, $\\mathbf{z}\\_2$, that reconstruct the activation via a decoder $p_{\\theta}$. The $\\mathbf{B}$ matrix only maps back in the architecture the $\\mathbf{z}\\_1$ architecture, which is disentangled from $\\mathbf{z}\\_2$ via a novel ELBO loss, which penalises the posterior $q\\_2$ to match the prior of $\\mathbf{z}_1$ ($p_1$), by maximising their KL divergence (Eq. 3). This formulation contains a  $\\Gamma$ modulator term, that can be interpreted as composed of two parts, *mismatch* and *discrepancy*, with the later being a lower bound on the 2-Wasserstein distance between the two posteriors. The authors conduct a thorough empirical study, showcasing strong performance and robustness to spurious correlations.", "questions": "- What I don’t really understand is why factoring your latent space into two Gaussians would lead to better performance. Couldn’t the same be achieved by simply reducing the rank? If not, I understand that the rationale is that if the $\\mathbf{z}_1$ features live on a manifold, this could lead to better representations for the downstream task.  Maybe the authors could touch more on this rationale, better interpreting their proposed method.\n- Related to this, the authors say in the introduction (Line 25) that it is not guaranteed that the low-rank projections fully capture task related tasks. Why should those features be detrimental if one optimises a loss specifically for the task during fine-tuning?\n- Did the authors tried using more than two auto-encoders for the factorisation?\n- Do the authors think that a more complex posterior than Gaussian would improve performance?\n- Line 99: I don’t think one has to put into the appendix the derivation of the ELBO since it is pretty standard knowledge in deep learning literature (e.g., VAE paper).\n- Eq. 5: Should we have $\\mathbf{z}_2$ instead of $\\mathbf{z}$?\n- Why this choice for centering $p_2$ on $\\mathbf{1.5}$? Dose it encourages factorization? It would have been interesting if the authors studied more in depth the prior initialisations.\n- In Table 6, the VAE2LAT loss is referenced in Eq. 2, not 3.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n### Novelty\nAs of my knowledge, doing disentanglement in a PEFT method is a novel idea, so I appreciate the contribution. At the same time it seems very surprising that such a method should work at all since the variational formulation, while useful for factoring the low rank space, it does not seem directly related to the task at hand (see first two Questions). I can see it justified by the fact that when using bot the task loss and the VAE loss, only $\\mathbf{z}\\_1$ is involved into computing the task loss, so this should compress more task-related information into it. But the same could be applied directly for example by simply lowering the rank (so as to not store spurious information into the low-rank space). Nevertheless, the good empirical results suggest that such a method really benefits from the variational loss. This method could as well be extended for example into optimisation methods that leverage singular value decomposition for reducing the size of a network, in order to better capture useful representations during miniaturization.\n\n### Soundness\nThe methodology is mathematically sound and well posed. I like the decomposition into mismatch and discrepancy, and the proof which shows that maximising the discrepancy maximises a geometric Wasserstein disstance between the two posteriors. \n\n### Strong experimental evidence\nThe method is tested across a plethora of domains (vision, text, audio) and showcases improved performance with respect to state-of-the-art methods in efficient fine tuning.\n\n## Weaknesses\n\n### Interpretation\nAs noted in the previous part, the authors should better interpret why disentanglement is key into achieving better performance as showcased by their empirical studies. Also it is not clear\nif the proposed loss (Eq. 4) is truly an ELBO as the author claim (is it still a lower bound of the log likelihood?).\n\n### Computational / complexity\nAs the authors suggest into the limitation section of their work, one key aspect of LoRA, being able to merge the fine-tuned weights with the original frozen weights by simply summing\nthem, it is not possible with the current method. This could hinder the applicability of such a method, since it requires separate layers for the adapters.", "was_revised": true, "final_justification": "The authors proposed a nove PEFT technique based on latent space disentanglement. The answer of the authors provided more cues into why such a method showcases good empirical results.", "rebuttal": ""}, {"review_id": "50nxLSoJJK", "reviewer": "Reviewer_PQZk", "summary": "This paper introduces FVAE-LoRA, a new method to improve LoRA, by using a VAE to learn two separate latent spaces. FVAE-LoRA explicitly separate task-relevant from irrelevant residual features. They design a novel ELBO to encourage a clean separation between these spaces. They demonstrate the effectiveness of FVAE-LoRA by performing extensive experiments across text, audio, and vision, especially in scenarios involving spurious correlations or distribution shifts.", "questions": "1. What are the recommended ranges/values of these hyperparameters (e.g. \\beta, \\delta)? Would be great to see a guide on how to choose these hyperparameters. How sensitive are the empirical results to different hyperparameter choices?\n\n2. Could you add some visualization of what is actually encoded in z1 vs. z2? I'm wondering if visualization shows z1 captures task-related information, while z2 captures residual as expected?\n\n3. Could you add some training/inference time comparisons between the proposed method and existing methods?", "limitations": "Could you add some future work discussions in the paper?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- the paper introduces a carefully designed objective and has run extensive empirical experiments to demonstrate the effectiveness the proposed method.\n- the paper is clearly written and easy to follow.\n\nWeakness:\n- the improvements of proposed method compared to LoRA are small on some benchmarks, sometimes the improvements are within the error bars range (e.g. WATERBIRDS). \n- error bars (stddev) are missing in Table 2. \n- lack of hyperparams tuning guide and analysis. \n- lack of interpretability. \n- lacks computation complexity analysis. it's unclear how much training time is increased compared to LoRA.", "was_revised": true, "final_justification": "I thank the authors for their detailed responses to my questions and effort to add more useful results and discussions in appendix. I'm willing to increase my score to 4.", "rebuttal": ""}, {"review_id": "DwBaXLsybI", "reviewer": "Reviewer_aAk2", "summary": "This paper proposes FVAE-LoRA, which combines LoRA with a factorized VAE framework. The key idea is to learn two distinct latent spaces (z1 for task-salient information and z2 for residual information) through a novel ELBO, with only z1 being used for downstream task adaptation. The FVAE-LoRA is extensively evaluated across diverse tasks.", "questions": "1. In Equation (3), why does the regularization term specifically penalize the similarity between qφ2(z2|x) and p1(z1) rather than other possible combinations? Since both p1 and p2 are normal distributions, what is the specific theoretical or empirical rationale for encouraging qφ2 to be dissimilar from p1 specifically?\n\n2. Why is p2 specifically centered at 1.5? The paper provides no theoretical justification or empirical analysis for this choice. Was this value determined through hyperparameter search, theoretical considerations, or some other principle?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The integration of factorized VAE with LoRA is conceptually interesting, and the formulation with the cross-prior regularization term (Γ) is well-motivated.\n\n2. The experimental evaluation spans multiple modalities (vision, text, audio) and includes diverse datasets, demonstrating the generality of the approach.\n\n3. The spurious correlation experiments provide compelling evidence that the method indeed learns more robust representations.\n\nWeaknesses:\n\n1. FVAE-LoRA replaces LoRA's simple linear projections with complex multi-layer nonlinear VAE mappings, and uses two separate VAEs, substantially increasing computational complexity. While the authors acknowledge this in the limitations, they provide no analysis of the actual computational costs or training time overhead. For a PEFT method, this lack of efficiency analysis is a critical omission.\n\n2. The non-mergeable weights represent a severe practical limitation that undermines one of LoRA's core advantages. The ability to merge adapter weights back into the base model is a defining feature of reparameterization-based PEFT methods, enabling efficient deployment without inference overhead. By making the weights non-mergeable, FVAE-LoRA essentially transforms LoRA into an adapter-like method, losing this key benefit. Given the marginal performance improvements (e.g., 87.82% vs 87.40% on commonsense reasoning), these gains do not justify such a significant architectural limitation.\n\n3. The method introduces multiple hyperparameters (α, β, δ, λ). The paper lacks systematic analysis of sensitivity to these choices.\n\n4. Missing comparisons with non-LoRA PEFT methods like FourierFT.", "was_revised": true, "final_justification": "The authors' rebuttal adequately addresses the efficiency concerns and demonstrates that the approach achieves meaningful improvements on challenging tasks requiring robust feature separation.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper claims that part of the gap in performance between LoRA and full fine-tuning may be attributable to the fact that LoRA does not explicitly promote its latent spaces to reflect task-specific information. To remedy this, the paper proposes to adjoin the LoRA component, at training time, with a secondary latent representation, design a loss function that encourages this secondary latent space to capture the residual information, such that both latent spaces together can reconstruct inputs. The hope is that the LoRA side, which is the only one used at inference-time, then specializes more acutely to task-specific information. The paper demonstrates widespread empirical advantage to this approach and also empirically validates that this kind of separation is indeed happening, supporting the argument that the advantage is thanks to it.

Reviewers generally appreciate the paper's insight, results, and clear expositions. Several added clarifications offered during the discussions should become part of the paper. The theoretical underpinnings of the method, both in terms of the proposed ELBO-like loss, whether it promotes what it claims, and why this indeed improves the overall optimization, remain a weak point. Indeed, the model space is unchanged compared to LoRA, and it would be very interesting to have some analytical hints as to why/when regularizing the optimization with a form of a multi-task loss, which has components of both reconstruction (of the input) and differentiation (of the two latent spaces), can help the primary task.
Decision: Accept (poster)