Uni-LoRA: One Vector is All You Need
Authors: Kaiyang Li
Shaobo Han
Qing Su
Wei Li
Zhipeng Cai
Shihao Ji
Venue: NeurIPS 2025 spotlight ()
Keywords: Extreme parameter-efficient fine-tuning
unified framework
intrinsic dimension
Primary area: deep_learning

Abstract: Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine-tuning (PEFT) method for large language models (LLMs) by constraining weight updates to low-rank matrices. Recent works such as Tied-LoRA, VeRA, and VB-LoRA push efficiency further by introducing additional constraints to reduce the trainable parameter space. In this paper, we show that the parameter space reduction strategies employed by these LoRA variants can be formulated within a unified framework, Uni-LoRA, where the LoRA parameter space, flattened as a high-dimensional vector space R^D, can be reconstructed through a projection from a subspace R^d, with d << D. We demonstrate that the fundamental difference among various LoRA methods lies in the choice of the projection matrix, P ∈ R^{D×d}. 
Most existing LoRA variants rely on layer-wise or structure-specific projections that limit cross-layer parameter sharing, thereby compromising parameter efficiency. In light of this, we introduce an efficient and theoretically grounded projection matrix that is isometric, enabling global parameter sharing and reducing computation overhead. Furthermore, under the unified view of Uni-LoRA, this design requires only a single trainable vector to reconstruct LoRA parameters for the entire LLM -- making Uni-LoRA both a unified framework and a “one-vector-only” solution. Extensive experiments on GLUE, mathematical reasoning, and instruction tuning benchmarks demonstrate that Uni-LoRA achieves state-of-the-art parameter efficiency while outperforming or matching prior approaches in predictive performance.

Uni-LoRA: One Vector is All You Need
Kaiyang Li School of Computing University of Connecticut Storrs, CT 06269 kaiyang 1i@uconn.... edu
Shaobo Han
Qing Su School of Computing University of Connecticut Storrs, CT 06269 qing 2. su@uconn edu
Optical Networking and Sensing
NEC Labs America Princeton, NJ 08540 thaobo@nec-labs. com
Wei Li
Zhipeng Cai Dept. of Computer Science Georgia State University Atlanta, GA 30303 zcai@gsu. edu
Shihao Ji School of Computing University of Connecticut Storrs, CT 06269 shihao. @u@uunnnnccco edu
Dept. of Computer Science Georgia State University Atlanta, GA 30303 w1 w1128@gsu. edu
Abstract
Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine- tuning (PEFT) method for large language models (LLMs) by constraining weight updates to low-rank matrices. Recent works such as Tied-LoRA, VeRA, and VB- LoRA push efficiency further by introducing additional constraints to reduce the trainable parameter space. In this paper, we show that the parameter space reduction strategies employed by these LoRA variants can be formulated within a unified high-
framework, Uni-LoRA, where the LoRA parameter space, flattened dimensional vector space R.D. can be reconstructed through projection from a subspace Rdd, with d < We demonstrate that the fundamental difference among various LoRA methods lies in the choice of the projection matrix, P IR. Dxd, Most existing LoRA variants rely on layer-wise or structure -ppecic projections that limit cross-layer parameter sharing. thereby compromising parameter efficiency. In light this, we introduce an efficient and theoretically grounded projection matrix that is isometric, enabling global parameter sharing and reducing computation overhead. Furthermore, under the unified view of Uni-LoRA, this design requires only a single trainable vector to reconstruct LoRA parameters for the entire LLM making Uni- LoRA both a unified framework and a "one-vector-only" solution. Extensive experiments GLUE, mathematical reasoning, and instruction tuning benchmarks demonstrate that Uni-LoRA achieves state-of-the-an parameter efficiency while outperforming matching prior approaches in predictive performance. Our code is available at http:///gggtabbbooooo/aaagggggg/
Introduction
Parameter-efficient fine-tuning (PEFT) 1 casts a new paradigm that leverages strong prior knowledge built in foundation models and adapts them wide range of downstream tasks by updating a small amount of trainable parameters. Among various PEFT methods, LoRA (2] has been particularly prevalent in recent studies. Given pre-trained matrix Wo mmx. LoRA constrains the weight increment AW as a low-rank decomposition AW = BA, where B E mxr and A E R are trainable parameters, with r & min(m, n). LoRA reduces the fine-tuning cost significantly, while achieving impressive predictive performance.
To further reduce the number of trainable parameters for fine-tuning, recent methods augment [ 5 modify [6] the LoRA architecture and tighten the trainable parameter space of LoRA into lower- dimee--iieniional subspace, in which the fine-tuning is performed. For example, Tied-LoRA B ties all the B and A matrices of different LaRA-adapted modules and optimizes pair of B and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Inspired by the works of measuring the intrinsic dimension of objective landscapes 7 g we view A matrices and the diagonal entries of two diagonal matrices per LoRA module. VeRA A further reduces the trainable parameter space by randomly initializing B and and freezing them afterwards, leading to fine-tuning only two vectors per LoRA module. LoRA-XS S introduces an r x matrix per LoRA module, where r is the pre-defined LoRA rank, and fine-tunes a set of x r matrices. VB-LoRA [G] decomposes the B and A matrices of LoRA into fixed-length sub-vectors, and learns a globally shared vector bank and the compositional coefficients for each sub-vector of Bs and As. Despite the similarities among different LoRA variants, there isn't unified framework of LoRA that can describe the aforementioned LoRA methods a uniform language and enables a systematic analysis of all of them.
the parameter space of LoRA as a D-dimensional space R._ and each LoRA variant defines a d d-dimensional subspace R and a projection matrix P E R DDxd maps a trainable vector from the subspace E' back to the full LoRA parameter space RD, From the perspective of this unified view, we recognize that most existing LoRA variants (e.g., Tied-LoRA Lo VeRA, and LoRA-XS) project the parameters of each LoRA module into a separate and fixed-dimensional subspace. However, recent studies (e.g., Adal.oRA [9] and LoRA-Drop [10) have shown that the importance of LoRA modules varies across layers, suggesting that locally projecting the parameters of different LoRA modules into subspaces of the same dimensionality may be suboptimal. Moreover, our study reveals that the projection matrices P used implicitly by Tied-LoRA, VeRA, and VB-LoRA do not possess the property of isometry [II] from the trainable parameter space to the original LoRA parameter space. In other words, those projections do not preserve the distance between parameter vectors in the original LoRA parameter space, distorting the geometry of the optimization landscape.
To address the aforementioned limitations, we propose Uni-LoRA, a unified framework of LoRA that treats the LoRA parameter space a nigh-dimensiooal vector space and performs a global projection into shared low-dimensional subspace. We further introduce an efficient and theoretically grounded projection matrix P E RxDxd in which each row is one-hot vector with the index of "1" sampled uniformly from d slots, followed by column-wise normalization. As discussed in Sec. 3.3, this construction ensures that P possesses three desirable properties for adaptation: globality, uniformity/load baallalllaadallliiiinnn. and isometry. Conceptually, this corresponds to randomly partitioning all the parameters of LoRA into d groups and enforce the parameters within each group to share the same value during the training process. We prove that the resulting projection matrix is isometric (or distance-preserviin preserving). Empirically, when fine-tumi fine-tuning the GEMMA-7B model, our Uni-LoRA achieves performance comparable to LoRA while training only 0.52M parameters only 0.0061 % of the base model size and 0.26% % of the LoRA parameter size. Across multiple benchmarks, our method achieves extreme parameter efficiency, while outperforming matching the sfate-of-the- art LoRA variants. Moreover, we show that our projection matrix attains the performance of the Fastfood projection [12. While Fastfood is a widely used structured projection method with time complexity of O(D log d). our method achieves a significantly lower time complexity of O(D). Our contributions are summarized as follows:
We propose unified framework to analyze various LoRA variants, and show that many of them (e.g., Tied-LoRA, VeRA, LORA-XS, and VB-LoRA) can be interpreted as projecting trainable parameters from the full LoRA parameter space into structured low-dimensional subspaces.
We propose simple yet extremely effective projection matrix that randomly partitions the LoRA parameters into equally sized groups and enforces all the parameters in each group to share the same value. Despite its simplicity, our approach outperforms or matches the performance of state- of-the-art LoRA methods across a wide range of tasks, including natural language understanding, mathematical reasoning, and instruction tuning.
We further prove that our projection matrix is isometric and matches the performance of classical Fastfood projection in practice, while incurring significantly lower computational cost.
2 Related Work
Parameter-efficient LoRA Variants. Despite the success of Low-Rank Adaptation (LoRA) 2 in enabling parameter-efficies fine-tuning, several challenges remain particularly when scaling to even larger models or deploying multiple adapters on resource-constrained mobile devices. Recent methods aim to further reduce the number of trainable parameters in LoRA by (1) selectively freezing
a subset of the parameters 5 (2) enforeing weight-tying across layers [ (3) learning global
parameter sharing through admixture reparameterization [6]. Our Uni-LoRA provides unified
2

---

framework in which various parameter-reduction strategies are manifested as structural patterns in the projection matrix, thereby allowing the parameter redundancy in the LoRA parameter space be studied and reduced through the lens of dimensionality reduction. Perhaps surprisingly, parameter sharing does not need to be layer-wise, structure-awars or even learned. Our Uni-LoRA simplifies the fine-tuning with random weight-sharing strategy, where only one parameter vector needs to be trained and stored, showcasing that one vector is all we need for LoRA.
Intrinsic Dimension. A growing line of work suggests that the effective degrees of freedom required to train machine learning models [7] and fine-tuning [8] lie in significantly smaller subspace than the full parameter space. Zhang et al. [33 further demonstrate that the fine-tuning process tends to uncover fask-specific intrinsic subspaces, and that disabling these subspaces severely harms generalization. Along this line, FourierFT [14] locally projects the layer-wise incremental parameter matrices (i.e., AW) of pretrained models onto fixed Fourier bases, and shows that it suffices to only learn the corresponding combination coefficients.
Previous works 7 g typically employ preseervngg dense Gaussian matrices or structured transforms such as Fastfood to project the original parameter space into a owver-dimensional subspace. In Uni-LoRA, we propose a novel construction of the projection matrix that is pistance-preserving (i.c.. isometric) while significantly reducing computational cost compared to dense Gaussian or Fastfood-basee methods.
3 Proposed Method
3.1 Preliminaries: LoRA and its Parameter-efficient Variants
LoRA. LoRA [2] is a parametter-efficient fine-tuning technique for LLMs. Given a pre-trained weight Wo E IR LoRA constrains the weight increment AW a low-rank decomposition: AW BA, where B E Rmxr and R=""" are trainable, low-rank matrices with r & min(m, n). which significantly reduce the number of trainable parameters and improve fine-tuning efficiency.
Tied-LoRA/ / VeRA. Tied-LoRA 3 and VeRA 4 augment the LoRA architecture with two extra diagonal matrices. Both methods constrain the weight increment as AW = Ag ARAPAAPA PA. where PB E R and PA Rrxa are shared/tied across all LoRA modules, and As E Rmxm and Aa E RR' (per LoRA module) are trainable diagonal matrices that can selectively enable or disable columns and rows of Pn and PA by scaling, allowing effective, fine-grained adaptation with a minimal number of trainable parameters. The main difference between Tied-LoRA and VeRA is that while PB and PA are trainable in Tied-LoRA, they are randomly initialized and frozen in VeRA. leading to a higher parameter efficiency.
VB-LORA. VB-LoRA (6) decomposes the B and A matrices of LoRA into fixed-length sub- vectors, and learns a globally shared vector bank and the compositional coefficients to generate each sub- vector of Bs and As. During fine-tuning, only the parameters in the vector bank and the compositional coefficients for each sub-vector are trained. Upon the completion of fine-tuning, only the vector bank and the indices and values of the top-K (e.g., K 2) compositional coefficients are stored, leading to extremely small stored parameter size.
3.2 Uni-LoRA: A Unified Framework of LoRA
Despite the differences among the aforementioned LoRA variants, all of them augment or modify the LoRA architecture and tighten the trainable parameter space of LoRA into a lower-dimensional subspace. In this paper, we show that all these parameter space reduction strategies can be formulated within a unified framework, which enables a systematic analysis of all these LoRA variants.
Inspired by the works on measuring the intrinsic dimension of objective landscapes [7, we view the parameter space of LoRA as a D-dimensional space TR. and each LoRA variant defines d-dimensional subspace Rd, and a projection matrix P E R maps a trainable vector from the subspace IR back to the full LoRA parameter space TR D. Specifically, for each LoRA-adapted module é f= 1, L, we row-flatten the low-rank matrices Be E RR and A E Rr and concatenate them to construct full parameter vector of LoRA:
0p Concat (vec,u (B'), vec ( (A (A').... veCrow (B"), vecrow (A
(1)
where vec (.) denotes the row-wise flatten of matrix into column vector, and D I.m + n)r is the total number of LoRA parameters. Instead of optimizing directly in the LoRA parameter space,
3

---

(d) Uni-LoRA
(b) LoRA
B.
trinable
fixed
8
fised trainable
of learnallc logits
rearrange
8p
82
A E grki
(c) Tied-LoRA VeRA
(d) VB-LORA
(e) Uni-LoRA (Fastfood)
Po
R,
8
8g
2.
8
Ps
0₂
Bg
R
P
8
Ag
A
4
0p
P (egg. r- 1)
P
0p
P
VeRA Uni-LoRA
Llama2-7B
159.9M
L6M
Figure 1: Overview of Uni-LoRA and the representations of various LoRA methods the unified framework. For better visualization, illustrate the framework with only two LoRA-adapted modules. More examples including LoRA-XS and FourierFT are provided Appendix A.I
we propose to work in lower-dimensional subspace with parameter vector 0. E Rd, which is projected the original LoRA parameter space by linear projection:
On POa.
where P E Dxd is projection matrix with < D. In this formulation, 0g E IR represents the trainable parameters, and P can be trained along with 0g or designed and frozen during the fine-tuning process. We show that this view unifies a broad class parameter-effiiien LoRA variants, including Tied-LoRA, VeRA, LoRA-XS, and VB-LoRA. Therefore, Uni-LoRA, unified framework of LoRA. In light of this unified framework, we demonstrate that the fundamental difference among various LoRA methods lies in the choice of the projection matrix P and whether trained along with 0g or not.
call the formulation expressed by Eq.
Figure illustrates illlsttats the framework of Uni-LoRA and the representations of various LoRA methods, including LoRA, Tied-LoRA, VeRA, VB-LoRA, and Uni-LoRA (Fastfood) in this unified framework.
Uni-LoRA As specific instantiation of this unified framework, Uni-LoRA further introduces an efficient and theoretically grounded projection matrix P E Dxd in which each row a one-hot vector with the index of "1" sampled uniformly from d slots, followed by column-wise normalization such that the nonzero entries in column j are set to 1/VT,, where nt denotes the number of nonzero entries in that column. Once initialized in such way, P remains frozen and only the parameter vector 0₂ is fine-tuned and projected back to the full LoRA parameter space 0p by P. Conceptually, this corresponds to randomly partitioning all the parameters of LoRA into d groups, with parameters within each group constrained to share the same value during fine-tuning. Theorem shows that such designed projection matrix P is isometric, which preserves the distance between parameter vectors in the original LoRA parameter space, without distorting the geometry of the optimization landscape.
Algorithm provides the PyTorch-like pseudocode for Uni-LoRA, which can be seamlessly integrated into the Poror framework. Given that the projection matrix P one-hot like matrix, isn't sn't. explicitly constructed. Instead, only the indices and values of nonzero entries of this sparse matrix involve in the computation, leading to extremely efficient implementation.
LoRA It is straightforward represent LoRA in the framework of Uni-LoRA. As illustrated in Figure [b). in this case, P corresponds D x D identity matrix, and 0 has the same dimensionality

---

of the original LoRA parameter space, i.e., d = D. Apparently, the identity matrix P is isometrie, but it doesn't have the effect of reducing number of trainable parameters of LoRA.
Tied-LoRA/ / VeRA As introduced in Section 3.1 Tied-LoRA [3] and VeRA [4] represents the weight increment AW PAPAA, where Pa and PA are shared/tied across all the LoRA- adapted modules, and As and A1 are defined per LoRA module. Specifically. for each LoRA module é = L..... L. we extract the diagonal elements of Af. E Rm and A RR*x and concatenate them to construct trainable parameter vector:
O2 Concat (diag(A))) (iagg(A)), "iaig ddiig(A))) diag(A))))
where diag(-) denotes the diagonal vector of matrix, and L(m + +r the number of trainable parameters of Tied-LoRA or VeRA. On the other hand, the full parameter vector of LoRA can be formulated
t Concat (vecrow (B'), VCCrow (At),.. veCrow (B*), vecrow (osiooom(AA))) (At)).
(4)
where B = ABPB and A = A/PA. and they correspond to the LoRA parameters Be and A
As illustrated in Figure [T), the projection matrix P of Tied-LoRA and VeRA exhibits a structured sparse pattern, composed of block-diagonal components reshaped from PB and PA (details of which are relegated to Appendix[A.]]
Since PB and PA are shared/tied cross all the LoRA modules, the block-diagonal components are repeated L times in P. The main difference between Tied-LoRA and VeRA is the trainability PB and PA. i... the projection matrix P is trainable Tied-LoRA and frozen in VeRA. This is indicated by two different colors in the diagram.
It is worth noting that the projection matrix P in Tied-LoRA and VeRA is structured to have non-zero entries only in diagonal blocks, suggesting that the projection is local nature. Also, the rows of the LoRA matrices B RRmx and A E Rrxx are projected to two separate lower-dimensional subspaces with the non-uniform dimensionalities (m vs. r). Moreover, this projection matrix not isometry, indicating that it may distort the geometric structure of the original parameter space. Collectively, the locality, non-uniformity, and non-isometrie nature of this projection may constrain adaptation flexibility and limit representational expressiveness.
Uni-LoRAA (Fastfood) This a variant of Uni-LoRA, in which the project matrix P is initialized with the classical structured Fastfood projection [12], which is isometric. Fastfood approximates dense Gaussian projections with structured transforms and reduces the time complexity of a naïve implementation from O (Dd) to ODD log d). As be discussed in Section 3.4 Uni-LoRA with our uniform random projection achieves significantly lower time complexity of (D).
Similarly, VB-LORA, LORA-XS, and FourierFT can be represented as specific instantiations in our unified framework. Due the page limits, the details are relegated Appendix A.]
3.3 Analysis of the Projection Matrix
Our discussion in Section 3.2 reveals that the key distinctions among various LoRA methods lie in the choice of the projection matrix P and whether trained along with 0₂ or not. We argue that well-structured P should have the following three properties, i.e., globality, uniformity/load-balancee and isometry, which can substantially enhance adaptation performance.
Globality: Global parameter sharing across different types of matrices and layers breaks the physical barrier and enables maximal reduction of parameter redundancy.
Uniformity Load- Balaneed::::: Each subspace dimension is mapped to roughly equal number of dimensions of the original full parameter space, such that the information is evenly distributed to all subspace dimensions and
Isometry: The projection preserves the distance between parameter vectors in the original full parameter space, without distorting the geometric structure of the optimization landscape.
According to the properties above, we characterize the projection matrices employed by various LoRA variants in Table I [ecorem[provee that the uniform random projection introduced by Uni-LoRA is isometric.
Theorem
E TR. Dxd a projection matrix where each row selects exactly one entry uniformly at random to be nonzero, and sets all other entries to zero. Let n, denote the number of nonzero
5

---

Method
Learnable Projection
Globality
Uniformity
Isometry
VeRA 4
X
X
K
TicdLoRA
X
X
VB-LoRA 6
LoRA-XS S
X
X
/
Uni-LoRA (Fastfood)
X
/
Uni-LoRA (ours)
X
Table 1: Properties of the projection matrices P employed by various LoRA methods, where
"Learnalee Projection" refers to besides 0g whether P itself contains trainable parameters.
entries in column j. and n, > of For column-wise normalization, each nonzero entry in column is set to 1/ Vng. As projection matrix, P is isometric. Formally, (I w)ll 11=-1---l yll, Vx, y E IR
Proof. Given the construction of P in the theorem statement, we begin by showing that PT P la- Consider the (j, --tth entry of P P: [PT Pli Ei= P.P. Case 1: * k. Since each row contains only one nonzero entry, there exists no row i such that both Pig and P., are nonzero. Hence, every term in the summation is zero, and we have [PT Pl = O. Case 2: j = k. Column j contains n nonzero entries, each of value 1/ MTg. Thus, [PT Pla, EP= i=1 Phj n2 (1/ vTj) = From the analysis above, we have [PT Pli,e = if = k and O otherwise, thus PT P = Ia. Therefore,
P( 2) (x - v) PT P(r - v) = (a-
11--9) 9) _ y) || - yl? Vx, y E TR.
This completes the proof.
Why existing LoRA variants may be suboptimal? In light of our unified framework, Figure | reveals that the projections used by Tied-LoRA, VeRA, and LoRA-XS (except VB-LoRA) are all local with layer-wise projection; furthermore, Tied-LoRA and VeRA are inherently non-uniform. Specifically, although the B and A matrices of LoRA contain the same number of parameters, Tied- LoRA and VeRA ties the parameters of B with more parameters in the lower-dimensional subspace than A (i.e., m vs. r). Intuitively, such non-uniform projection may be suboptimal as information of the full LoRA parameter space is non-uniformly distributed to lower-dimensional subspace. In Section 4.5 we conduct a controlled experiment comparing uniform and non-uniform projections, and the results confirm that the uniform project consistently outperforms its non-uniform counterpart.
Why Isometry is important? Prior works [5] 16
show that if the projection matrix is isometric (which satisfies P( b) || yll for any pair of vectors x, y), the geometry of the original space is preserved in the projected subspace. This property ensures that the optimization landscape remains faithful to the original parameter space, making it particularly well-suited for subspace training for neural networks.
3.4 Complexity Analysis
Storage Complexity. Since the projection matrix P of Uni-LoRA is generated from random seed, upon the completion of training, only the random seed and the learned subspace vector 0g E RR need to be stored or transmitted. As result, the total number of stored data is only d + 1, leading to an extremely compact model representation and showcasing that one vector is all we need for LoRA.
Time and Space Complexity of Projection. The projection matrix of Uni-LoRA matrix with exactly D nonzero entries. As a result, the projection operation P0 has both time and space complexity of O(D). In contrast, the classical distance-preservvii methods such as dense Gaussian projections require O (Dd) time and space, while the Fastfood transform [2 has a time complexity of OD log d) and a space complexity of O(D).
sparse
To summarize, our Uni-LoRA not only provides a unified framework of LoRA but also introduces an efficient and theoretically grounded projection, which enjoys all three desired properties that we discussed above. This approach requires no architectural modifications, no sparsity priors, and achieves significantly lower time complexity than the classical Fastfood transform.
'To ensure the condition ny is always held, re-sample P if one column happens be all zeros.

---

Algorithm Pseudocode of Uni-LoRA PyTorch-like Style
in dim, out_dim, onput/ dimensions of the linear layer, LoRA rank # theta_d: only trainable parameter vector of length d
index_A/B: mape LoRA matrices A/B to subspace vector theta_d
# norm_ aer__factor_ABB normalization matrix, same shape as Andex_A/B, with entry value
# norm norm_facton factor_ [acttorAAii,]]_ee jl- (n_k), where k=index_A[i. j] and n_k is its occurrence
# frequency across all index_As and index_Bs.
# x, W: input and original weight
class Uni-LoRA:
def __init__ (self, in_dim, out_dim, 2, theta_d):
d 1en (theta_d)
snde.inde__ torch.randint (0, d, iindimm, r), dtype=torch.
self sllfiinex_BB B torch. randint (0, d, (r, out_dim) dtype=torch.long Self.theta_d theta_d
def update_nore faattte_nom___actce fatttooooonaaooeeeeieeeeeee (self,norm__feeeeeeeeee laarm_aacttr__e norm_,,,,oomm_ottott self. nwrm norm worm_factor__
self colf.nor__factor.. norm_factor_B
def forward
self. thefta
self wwwrmffcctor
B = self. thetaeeeeeeee festadd[self index__] self morm_factor_B
# For memory efficiency, we avoid explicitly computing \delta W A.
return x e (x
Experiments
In this section, we evaluate our method through a series of experiments. We begin by comparing Uni- LoRA the state-of-the-an PEFT methods: LoRA, Tied-LoRA, VeRA, LoRA-XS, and FourierFT on the GLUE benchmark. Next, we extend our analysis to mathematical reasoning tasks Mistral and Gemma models, as well as instruction tuning tasks on Llama2. All our experiments are conducted on server equipped with 8 NVIDIA A A100 80GB GPUs. For reproducibility, provide detailed hyperparameters and specifications of computing resources for each experiment in Appendix A.2
4.1 Natural Language Understanding
We adopt the General Language Understanding Evaluation (GLUE) benchmark [17 assess the performance of Uni-LoRA across various natural language understanding tasks, Following 4 6. we focus on six tasks from GLUE: SST-2 [8] (sentiment analysis), MRPC 19 (paraphrase detection),
CoLA 20 (linguistic acceptability), QNLI 21 (inference), RTE [22 (inference) and STS B 23 (semantic textual similarity). Our experiments use RoBERTabse and RoBERTange [24] as the backbone models. We apply LoRA with rank 4 to the query and value matrices in each Transformer layer, and project the resulting LoRA parameter space into subspace dimension d 23, 040, matching the subspace size used by the best-performing VB-LoRA baseline 6
Table reports the results across 12 cases spanning 6 GLUE tasks and 2 model scales. Uni-LoRA ranks either first second in 11 out of the 12 cases, with the only exception being the RTE task of RRBBERTaagg where the dataset is small and the performance variance is relatively high. As we see, Uni-LoRA consistently outperforms Tied-LoRA, VeRA, LoRA-XS, and FourierFT while requiring fewer trainable parameters. For example, when fine-tuning RoBERTaange our method reduces the number of trainable parameters to less than 40% of that required by Tied-LoRA VeRA, while achieving better performance across all tasks. Moreover, Uni-LoRA achieves performance comparable VB-LoRA while using significantly fewer trainable parameters, suggesting that our fixed projection matrix is as effective as the learned projection in VB-LoRA. Furthermore, since our projection matrix learned, Uni-LoRA incurs substantially lower computational overhead than VB-LoRA (See Tables
12 in the appendix for details).
1 The original VB-LORA paper reports number stored parameters rather than trainable ones: 0.23M for RoBERTateee 0.24M for RoBERTaage: 0.65M for Mistral-7B. 0.67M for Gemma-7B, 0.8M for Llama2-7B. -BB and 1.1M for Llama2-132 13B. For consistency, we report the number of trainable parameters throughout all tables. following the convention used in other baselines.

---

Table Results with RoBERTa and RoBERTapage on the GLUE benchmark. The best results in each group are shown in bold, while the second-best results are denoted with underlining. We report Matthew's correlation for CoLA, Pearson correlation for STS-B, and Accuracy for other tasks. We report the median performance over 5 runs with different random seeds.
Method
Trainable Params
SST-2
MRPC
CoLA
QNLI
RTE
STS BIS-B
Avg.
FT
125M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
LoRA
0.295M
95.1
89.740.7
63.4
93.3
86.629.7
91.5g0.2
86.6
VeRA
0.043M
94.610.1
89.5
65.610.
91810a
78.7
90.7
85.2
Tied-LoRA
0,043M
94.420.5
88,5
61.9116
92.040.4
7..2:
89.8.0.3
83.8
VB-LoRA
0.075M?
94.4+0.2
89.5+0.s
63.340.7
92.2+02
82.3+1.
90.8+0.1
85.4
FourieFFT
0.024M
94.220.3
90.0
63.8g1.6
92.2.0.1
79. 19.1
90.8.0.2
85.0
Uni-LoRA(OOrs)
0.023M
94.510.2
89.7.0.4
64.611.0
92.3
29.8
90.910.2
85.3
LoRA
0.786M
96.220.5
90.2g..0
68.2g1.9
94.8
85.2
92.3
87.8
VeRA
0.061M
96.1
90.910.7
68.010.8
94.410.
85.9
91.7.0.8
87.8
LARGE
Tied-LoRA
0.066M
94.8+0.6
89.7+1.0
64.7+1.2
94.1
81.2
90.8+0.5
85.9
VB-LeRA
0.162M
96.1+0.2
91.4+0.6
68.3+0.7
94.7=0.0
86.6=1.3
91.8+0.1
88.2
LoRA-XS
0.025M
95.920.3
90.710.4
67.0g1.2
93.929.1
88.1
92.0.o.1
87.9
FourierFT
0.048M
96.020.2
90.9 20.3
67.1
94.4
87.4
92.0+0.4
88.0
Uni-LoRA(Owss)
0.023M
96.3+0.2
91.3+0.6
68.5.1.1
94.6+0.4
86.6=1.a
92.1=0.1
88.3
Table 3: Results with the Mistral-7B and Ge models on the GSM8K and MATH Benchmarks. The best results in each group are shown in bold. Baselines' results either produced using their reported configurations or directly sourced from their original papers.
Table 4: Scorei and Score, denote evaluations MT-Bench single-turn and multi-turn dialogues. LoRA" and VeRA
scored using older
GPT-4 API. LoRA # and other methods were evaluated using the updated version. The two sets of results are reported separately for fairness.
Model
Method
Pammettess
GSMSK
MATH
Method
Parameters
Score;
Scores
FulLFT
7242M
67.02
18.00
1oRA"
159.9M
5.19
LoRA
168M
67.70
19.68
VeRA
1.6M
5.08
MISTRAL-78
LoRA-XS
LLAMAZ 7R
1.31
VB-LORA
Uni-LoRA (Qurs)
VeRA
FourierFT
0.52M
0.67M
1.39M
93M1
68.54
68.92
68.69
18.81
17:00
17.50
18.18
LORA"
VBLoRA
Uni-LoRA (Ourz)
159,9M
0.52M
83M1
5:43
5,62
5.58
3.46
3.23
3,56
Full-FT
250
200M
31.28
VeRA
2.4M
5.93
GEMMA-7B
LoRA-XS VB-LoRA
0.8IM
74.22
27.62
LLAMA2 13B
wio FT
1.46
1.06
113M
74.86
28.90
VRRA
1.90M
74.98
28.84
VBL-RA
256M1
5.96
4,33
FrurieFF
Uni-LoR, (Ours)
1.0M
6.34
4.43
LoRA (Ours)
4.2 Mathematical Reasoning
To evaluate mathematical reasoning capabilities, we fine-tune the Mistral-7B-vit 25 and Gemma-
7B (26) models on the MetaMathQA (27) dataset and test them on GSM8K (28) and MATH 29 We compare our results with the state-of-the-ar methods, following their experimental configurations. Jablessshowss that Uni-LoRA achieves strong and consistent performance across both Mistral-7B and Gemma-7B backbones. On the MATH benchmark, it reaches 18.1 18 with Mistral-7B and 28, with Gemma-7B, outperforming LoRA-XS, VB-LORA and FourierFT. It also yields higher accuracy than LoRA-XS XS on both GSM8K and MATH, while using less than 70% of its trainable parameters. Compared to VB-LORA, Uni-LoRA achieves similar better performance with substantially fewer parameters. It also demonstrates comparable accuracy to VeRA on mathematical reasoning tasks, while requiring only half as many trainable parameters on Mistral-7B and less than one-third on Gemma-7B, Furthermore, Uni-LoRA consistently outperforms FourierFT on both tasks, while being more parameter-efffcient..
4.3 Instruction Tuning
Instruction tuning refers to the process of fine-tuning model on a set of instructions prompts to improve its ability follow task-specific directives [30]. We perform instruction tuning on the Llama 2 (31 model using the Cleaned Alpaca dataset [32). which improves the quality of the original Alpaca
dataset. Following 4 ( we fine-tune the Llama model (3I) within the QLoRA framework (33],
which enables memory fine-tuning of LLMs on single GPU. The fine-tuned models generate
8

---

responses to MT-Bench questions, which are then scored by GPT-4 on a 10-point scale. We apply Uni-LoRA to project the rank-4 LoRA weights into 0.5M-dimensional subspace for Llama2-7B and I M-dimensional subspace for Llama2-13B.
T the results. Due to noticeable discrepancy between the reported scores, we include two sets of LoRA results for each experiment: one from Kopiczko et al. [4] and another from our own reimplementation. Although we closely follow the experimental settings of Kopiczko et al. [4], we speculate that the difference may be attributed to changes in the GPT-4 model over time. Nevertheless, comparing the relative performance of VeRA and Uni-LoRA with respect their corresponding LoRA baselines remains meaningful. As we can see, Uni-LoRA achieves performance comparable to LoRA while using only 0.3% (Llama2-7B) and 0.4% (Llama2-13B) 13B) of the LoRA parameters. Similarly, VeRA also demonstrates comparable performance relative to its own LoRA baseline, but it requires more than twice the number of trainable parameters compared to Uni-LoRA. Furthermore, Uni-LoRA consistently outperforms VB-LoRA across both evaluation metrics on LJama2-7B and Llama2-13B, while using significantly fewer trainable parameters of VB-LoRA and slightly fewer stored parameters.
4.4 Experiments of Computer Vision Tasks
To further assess the effectiveness of Uni-LoRA on more complex vision tasks, we follow the experimental setup in FourierFT [4] and conduct additional evaluations. Specifically, we apply Uni-LoRA with d = 74,000 on ViT-Base and d 1 144,000 on ViT-Large, using fixed rank 4. A grid search is conducted over the head learning rate in 10 2 10 -3 5x 10-3, 1x10- and the trainable vector learning rate in (2x 10 5 10 1x 10-2, 2x 10- 5 10 10- The
number of training epochs is fixed at 20. All experiments are repeated five times, with the mean and standard deviation across the runs reported T
Table 5: Comparison of Uni-LoRA and baseline methods on eight computer vision datasets using ViT-Base and ViT-Large backbones. All baseline results, including Full Fine-tuning (FF) and Linear Probing (LP), are taken from the original FourierFT paper [1. In LP, only the classification head is fine-tuned.
Model Method
# Trainable OxfordPets StanfordCars CIFAR10 Para.
DID
EuroSAT
EGVC
RESISCAS CIFARI00 Avg.
LP FF
90.28
25.76
96.41=002
69.77 88.72=0.3
17.442043
74.22
84,2820.11
68.36
VIT WTT-Baue Base
85.8M
93.14:244
19.78:
98. 98.922005
77 21
99.05-2009
54,84g125
96.132a13
92.3820.13
86.49
FourienF'T
72K
93.21
46.11:1044
98.58=007
35
98.29-0044
27.512004
91.97=031
91,20:0.14
77.75
F
239K
93.05=0344
56.36:2066
98.69:0.6
77.30=06l
98.78=0.11
32.44:n99p
94.26:220
91.45=0.18
80,29
Uni-LoRA
72K
94.00:0011
76.06::0.3
98.77=a003
76.99:0096
98.86-0.30
50.36::063
94.08:019
92.10:0025
LP FF
91.11
37.91
97.782004
73.33=0026
92.64=00.66
24.62#024
82.02en:1
84,28g0.11
72.96
WT VT-Largg
303.3M
94.43
88 30
99.15
81. 11.79:110
99.04.20.6
68.25
96.43gn07
93.58:20.19
90.20
FourierF'T
94.46:028
39.5660330
99. 10::04
80.83-004
65:66
39.92:006
93.862014
93.312000
FourierFT 480K
94,84:101
79. 14:1007
99.08
$1.88
98 66:20.0
$1. 11.28
95.20
93.37:0.:1
86.68
Uni-LoRA
94.65+023
83.16:002
98.77:003
41 98.89-0.07
58.89:0.62
95.24=n.12
93.0820.11
Experiments show that Uni-LoRA consistently outperforms FourierFT across computer vision benchmarks. Remarkably, even when FourierFT uses three times as many trainable parameters, Uni-LoRA still achieves better performance. Furthermore, when the number of trainable parameters in Uni-LoRA is less than one-thousandth of that in full fine-tuning, the performance gap between Uni-LoRA and full fine-tuning remains within 2.5%. These results highlight the strong efficiency and generalizability of Uni-LoRA in complex computer vision tasks.
4.5 Ablation Study
Comparison with Fastfood-based Projections. Fastfood 12 is classical projection method known for its low computational and memory cost and effectiveness of our projection method, we compare it against Uni-LoRA (Fastfood) on four GLUE tasks in terms of both predictive performance and runtime efficiency. Table that Uni-LoRA with the uniform random projection consistently outperforms Uni-LoRA (Fastfood) in terms of predictive performance across four GLUE tasks, while being significantly faster. For example, on MRPC, it achieves slightly higher score (91.3 vs. 90.7) with 65% reduction in training time (9 vs.
properties. To evaluate the

---

Table 6: Comparison of Uni-LoRA and Fastfood on four GLUE tasks in predictive performance and training time (mins). We report Matthew's s Correlation for COLA and Accuracy for others. Both methods underwent grid search over the same learning rate range, and all other hyperpa- rameters were kept identical across experiments.
Table 7: Ablation study of Uni-LoRA with lo- cal projection and non-uniform projection on four GLUE tasks in predictive performance, For COLA, we report Matthew's correlation and Ac- curacy for others. All methods were tuned using grid search over the same learning rate range, with all other hyperparameters kept identical across experiments. We report the median perfor- mance over runs with different random seeds.
Task
Method
Score (%)
Time (mins)
Uni-LoRA
91.3
9
MRPC
Fastfood
90.7
26
Uni-LoRA
68.5
Task
Uni-LoRA
Local
Non-uniform
COLA
Fastfood
65.3
60
MRPC
91.3
90.9
90.7
Uni-LoRA
96.3
80
SST-2
COLA
68.5
68.5
67.0=1.
Fastfood
96.1
251
SST-2
96.3
96.2
96.1
Uni-LoRA
94.6
147
QNLI
Fastfood
94.1
358
QNLI
94.640.4
94.5
94.0.0.4
26 mins). This efficiency stems from the linear time complexity O(D) of our projection, compared to Fastfood's O log d).
Comparison with Layer-wise Projections. To investigate the impact of global vs. layer-wise (local) projections to method, we construct controlled variant as follows. On the RoBERTa-large model, we project the parameters of each layer into its own trainable subspace, with each local projection matrix constructed in the same way that of Uni-LoRA's. To ensure a fair comparison, set the per-layer subspace dimensionality such that the total number of trainable space dimensionality matches that of Uni Uni'Ls Tsble that local projection performs worse than global projection on MRPC, SST-2, and QNLI, while both methods achieve comparable performance on CoL.A.
Comparison with Non-uniform Projections. To investigate the impact of uniform vs. non-uniform projections to our method, we construct a controlled variant in which two separate projection matrices are used: one projects all LoRA A matrices into the first two-thirds of the subspace dimensions, and the other projects all B matrices into the remaining one-third. Both projection matrices are constructed using the same method as that of Uni-LoRA's. Table shows that the uniform projection consistently outperforms the non-unifon non-uniform projection across all the tasks.
Conclusion
This paper introduces Uni-LoRA, unified framework of LoRA that reinterprets broad class of parameter-efffiieet LoRA variants through the lens of global subspace projection. In light of this unified view, Uni-LoRA further introduces an efficient and theoretically grounded projection matrix, which enjoys all three desired properties of globality, uniformity, and isometry for adaptation performance. Uni-LoRA requires no architectural modifications, no sparsity priors, and achieves a significantly lower time complexity. Our results show that even within the LoRA space, which is already low-rank, there exists an additional low-dimensional parameterization that has several orders of magnitude fewer trainable parameters than LoRA.
Limitations and broader impacts As an extreme parameter-efficient fine-tuning method, Uni-LoRA doesn't 't possess any significant limitations other than it currently only evaluated on small to medium scale NLP/CV benchmarks due to limited computing resources. As for its broader impacts, A enables high-quality adaptation of L.LMs with minimal computational resources, contributing to the democratization of model customization. We do not foresee any societal risks beyond those generally associated with the use of LLMs.
Acknowledgments
We would like to thank the anonymous reviewers for their excellent comments and suggestions, which greatly helped improve the quality of this paper. This work was supported in part by the National Science Foundation under Grant Nos. 2343619, 2416872, 2244219, 2315596 and 2146497.
10

---

References
[1] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: thrt parameter-efficient fine-tuning methods. https //github. con/huugiiggace//peeee 2022.
[2] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[3] Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-LoRA: Enhancing parameter efficiency of LoRA with weight tying. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 8694-8705, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[4] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024.
[5] Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. LoRA-XS: Low-rank adaptation with extremely small number of parameters, 2024.
[6] Yang Li, Shaobo Han, and Shihao Ji. VB-LoRA: Extreme parameter efficient fine-tuning with vector banks. In The 38th Conference on Neural Information Processing Systems (NeurIPS), 2024.
[7] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR (Poster), 2018.
[8] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7319-7328 Online, August 2021. Association for Computational Linguistics.
[9] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effciient fine-tuning. In The Eleventh International Conference on Learning Representations, 2023.
[10] Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. LoRA- drop: Efficient LoRA A parameter pruning based on output evaluation. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara Di Fugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 5530-5543, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics.
[11] Ingrid Carlbom and Joseph Paciorek. Planar geometric projections and viewing transformations. ACM Comput, Surv., 10(4):46-5502 December 1978.
[12] Quoe Le, Tamás Sarlós, and Alex Smola. Fastfood: approximating kernel expansions in loglinear time. In Proceedings of the 30th International Conference on International Conference on Machine Leurning- Volume 28, ICML' 13, page 111-244-111-222. JMLR.org, 2013.
[13] Zhong Zhang, Bang Liu, and Junming Shao. Fine-tuning happens in tiny subspaces: Exploring intrinsic task-specific subspaces of pre-trained language models. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1701-1713, Toronto, Canada, July 2023. Association for Computational Linguistics.
[14] Ziqi Gao, Qichao Wang. Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
[15] Deng Cai, Xiaofei He, Jiawei Han, et al. Isometric projection. In AAAI, pages 528-533, 2007.
11

---

[16] Gautam Pai, Ronen Talmon, Alex Bronstein, and Ron Kimmel. Dimal: Deep isometric manifold learning using sparse geodesic sampling. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 819-828. IEFE, 2019.
[17] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi, editors. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018.
(18) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013.
[19] William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
[20] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 0415-641,22 2019.
[21] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know:
Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, July 2018
[22] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entail- ment challenge. In Proceedings of the First International Conference Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW 05, page 177-190, Berlin, Heidelberg, 2005. Springer-Verlle
[23] Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio. and Lucia Specia. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors, Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)) pages 1-14, Vancouver, Canada, August 2017.
[24] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: vrrXiv:/9077 1907. 1692, 2019.
[25] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[26]
Thomas Mesnard al. Gemma: Open models based on gemini research and technology, 2024.
[27] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[28] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jeny Tworek, Jacob Hilton, Reichhrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. 2021.
[29] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
12

---

[30] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[31] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint
(tth:2337.09288. 2023.
[32] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An Instruction-following Ilama model. https://git http://gggtu.....// com www/tttsu-e.........o..o.....a.. wwt.t/ttttttaa/ttttttaaaaeeeeaaa waabbstaafood...mm aaapafffdd_llpaaaaaaaaaaaaaa 2023.
[33] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Oh, T. Naumann, Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10088-10115, 15, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction reflect all contributions the paper.
Guidelines:
The answer NA means that the abstract and introduction do include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the "Limitations and Broader Impacts" in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon,
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
Please refer to Theorem and its proof.
14

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
In Sections 3.2 and 3.3 of the main text, well as in Algorithm 1, we provide precise description of our method. In Section 4, we clearly outline our experimental setup, including the backbone models used, the tasks evaluated, and the datasets and metrics employed. In Appendix A.2, we detail the hyperparameter configurations for each experiment. Our code is also provided as part of the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
15

---

Justification:
Our source code includes instructions for execution. The code is configured to automatically download the necessary datasets upon running.
Guidelines:
The answer NA means that paper does include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e-g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https onPolicy) for more details.
//nis.cc// http:/allcgguudes/Cooaoa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable)..
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Section 4 provides clear description of our experimental setup, covering the backbone models, evaluation tasks, datasets, and metrics. Appendix A.2 further specifies the hyperparameter settings used in each experiment. The accompanying code and usage instructions are included the supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Tables 2 and 6, we report the median performance and the corresponding standard deviation computed over multiple independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: In Appendix A.2, we report the computational resources used, including GPU models, memory usage, and runtime.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics ltttps://eeurippt httpppp........ccecccccceeeeeeee
Answer: [Yes]
Justification: The authors have read the https://neurips.ccppbbiicc FdhicsGuidelines that this paper conforms to it.
https////piiiiiiittiiiiccciiieee and ensured
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
a The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed both potential benefits and risks to society in the "Limitations and Broader Impacts" paragraph of the conclusion section.
Guidelines:
The answer NA means that there is no societal impact the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Appendix A.4 provides details the licenses and usage of external assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaerswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification:
The supplementary material contains documentation on code installation and usage.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their sub- missions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted. IRB approval (or equivalent) may required for any human subjects research. If you obtained TRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if is an important, original, or non-standard component the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our research does not involve the use of large language models (LLMs) in any important, original, or non-standard component the core method. L.LMs were not used for model design, data generation, training, or evaluation.
19

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy fitees///barrrre.cc///ooeeeeeeee for what should or should not be described.
20

---

A Appendix
A.1 Representations of additional LoRA/PEFT methods in the unified framework
Tied-LoRA 3 and VeRA 4 represents the weight increment AW As,rr Aa.AAAP where Pa and PA are tied/shared across all the LoRA- adapted modules, and As and Aa are defined per LoRA module. Specifically, for each LoRA module l = 1................ It extract the diagonal elements of Ag E IR and A IR"x", and concatenate them to construct trainable parameter vector:
Tied-LoRA/VeRA As introduced in Section 3.1
d Concat (diag(A!)), diag(A)), (diag(A))) di))(
(5)
where diag(-) denotes the diagonal vector of matrix, and L(m + +) the number of trainable parameters of Tied-LoRA or VeRA. On the other hand, the full parameter vector of LoRA can be formulated as
OD Concat (vec (B'), VCCrow (A VeG (B²), vec (At)).
(A')
(6)
where Be A& and A = PA correspond to the LoRA parameters B' and A respectively, and VcC (.) denotes the row-wise flatten of matrix into column vector.
As illustrated in Figure a)) the projection matrix P of Tied-LoRA and VeRA exhibits a structured sparse pattern, composed of block-diagonal components reshaped from PB and PA. Specifically,
P = DDiaag ((PP))) (
{(PA)..)
where (.)T; denotes the transpose the i-th row matrix, and Diag(U1, U1, U2 (L((+r)) con- structs block diagonal matrix from L(m + r) column vectors. Since P3 and PA are tied/shared cross all the LoRA modules, the dlock-diagonal components are repeated L times in P. Moreover, the main difference between Tied-LoRA and VeRA is the trainability of Pg and PA. i.e., the projection matrix P is trainable in Tied- MeRA and frozen in VeRA. This is indicated by two different colors in the diagram.
VB-LoRA decomposes the B Rexr and A E R matrices of LoRA into fixed-length sub-vectors as follows:
veG (B*) = Concat(u(, uNa), ve (At) = Concat(vf. ... V. ,4, (8)
where u4, 0/ RR' are sub-vectors of length b, and N/ of sub-vectors decomposed from De and A respectively.
and N represent the numbers
VB-LoRA learns shared vector bank B {an,... .an,). ..,), with a; R², and the compositional coefficients to generate each sub-vector of Bs and As. Specifically, each sub-vector generated from top-K K (e.g., K 2) compositional coefficients over B. Since the coefficients are sparse, we encode them efficiently by K indices over B and K coefficient values.
As illustrated in Figure bb), under our framework, the original LoRA parameter space @D is formed the same in Eq. 1 The trainable vector 6g of VB-LoRA is formed by concatenating all the vectors in B:
0g = Concat(ar,,0; "2,"" ..)).
The projection matrix P is structured block-diagonal matrix, where each diagonal block is of size b x b, matching the dimensionality vectors in B. Since K in VB-LoRA, two diagonal blocks in row in P is responsible to reconstruct one sub-vector of Bs or As, where the location of the diagonal block corresponds to the coefficient index and the diagonal value is the coefficient value. Note that the compositional coefficients are learned in VB-LoRA. Therefore, the locations and values of the diagonal blocks in are trained along with 0g. i.e., the vector bank parameters.
LORA-XS represents the weight increment as AW PRAnPA, where Pß mR and PA E Rexn are derived from the singular value decomposition (SVD) of the pretrained weight matrix W and are kept frozen during fine-tuning, while Ag _g xr is the only trainable component.
21

---

Tied-LoRA/ VeRA
VR-LooA
8
B,
Pa
4,
82
Po
0
B
Pa
0.
A2
0p
0p
(eg..
(c)
LoRA-XS
(d)
Fourier FT
B
B
AW
A
PA
82
B2
00
0g
AW;
P (c.g, r= 2)
Op
P
Figure 2: Representations of additional LoR methods in our unified framework. For better visualization, we illustrate the framework with only two LoRA-addpted modules.
Specifically, for each LoRA module 1..... L, we column-wine flatten each A and concatenate them to construct trainable parameter vector:
0₂ Concat (veccal (AR),, vec (A))) vec (A
(10)
where v-C denotes the column-wise flatten a matrix into column vector. On the other hand, the full parameter vector of LoRA can be formulated as:
0p Concat (veciow (B*), veCrow ..-,.... vecrow (B4), vecrow
vec))o(A4)) (A4))
(11)
where Pf,A,, and A P correspond the Ba and A of LoRA, respectively. Since Å'= PA initialized, frozen, and independent of A we only need to map Ag B i.c., projecting the trainable parameter fg to the B portion of 0p. As illustrated in Figure (c), the projection matrix P of LoRA-XS also exhibits a structured sparse pattern, composed of the row-vectors extracted from P organized in a stripe pattern.
isn't a LoRA-based method that performs parameter-efficien fine-tuning in the domain of low-rank adaptation. Instead, FourierFT learns the weight increment AW directly by utilizing the Fast Fourier Transform (FFT) for sparse representation. We show that despite FourierFT isn LoRA variant, our Uni-LoRA is unified framework that represent FourierFT is the same uniform language and enables a systematic analysis.
FourierFT
FourierFT leverages the Fourier bases to parameterize the weight increments, wherein a small number of trainable Fourier coefficients can be used to synthesize the full weight increments. To represent FourierFT in our unified framework, we row-wise flatten the weight increment A for each adapted module i = 1, 1, L. and concatenate them construct the full PEFT parameter space:
AD Concat (vecrow (AW1). vec ( (AW?), vec (AW")).
(12)
the other hand, the trainable parameter vector 0₂ is formed by concatenating the Fourier coeffi- cients for each adapted module.
22

---

As illustrated in Figure d), the projection matrix P also exhibits block structure since FourierFT adopts layer-wise projection:
P= (1. (p ,.... ,L))
(13)
where the block matrix p denoting the Fourier subspace for the &-th module, is generated by randomly sampling subset of Fourier bases.
A.2 Hyperparameters and Computing Resources
The hyperparameters used for the natural language understanding, mathematical reasoning, and instruction tuning are provided Tables 8 9 10 and 11 All experiments were conducted on a server equipped with 8 NVIDIA A100 80GB GPUs.
Computation overhead The proposed projection in Uni-LoRA is straightforward to implement in modern deep learning frameworks such PyTorch, enabling full utilization of GPU acceleration. Moreover, since the projection matrix in Uni-LoRA is frozen and does not require training as opposed Tied-LoRA and VB-LoRA incurs significantly lower computational overhead. As shown in Tables / and Io Uni-LoRA achieves substantially shorter training time compared to VB-LoRA, and is par with the original LoRA.
Memory efficiency Uni-LoRA significantly reduces memory consumption by minimizing the number of trainable parameters. During LoRA fine-tuning, the forward computation is performed as Ax, H = Bz, without the need of explicitly instantiating AW. This memory-efficient strategy seamlessly supported in Uni-LoRA and has been implemented our codebase. shown in Tables and [O] Uni-LoRA consistently consumes less memory than VB-LoRA across all model configurations evaluated.
Table 8: Hyperparameters and computing resources used in the natural language understanding experiments on the GLUE benchmark. h: hour, m: minute.
Model Hyperparameter
SST-2 MRPC CoLA QNLI
RTE STS-B
Optimizer
AdamW
Warmup Ratio
0.06 Linear
LR Schedule
Init. of 0a
U ((-0002, 0.02)
# GPUs
Epochs
60
30
80
25
160
80
Learning Rate (Head)
1E-4
2E-2
5E-3
2E-4
5E-4
2E-4
Learning Rate (0a)
5E-3
10al
23,040
BASE
Rank
4
Max Seq. Len.
512
Batch Size Per GPU
32
Training Time
9.2h
15.5m
1.8h
6.2h
1h
1.1h
GPU Memory
24,310 MiB
# GPUs
Epochs
20
40
40
20
40
40
Learning Rate (Head)
2F-4
2B-3
2F-2
SE-3
5E-3
IE-4
Learning Rate (0a)
5E-3
[0.]
23,040
LARGE
Rank
4
Max Seq. Len.
128
Batch Size Per GPU
32
Training Time
1.3h
9m
21m
2.5h
6.5m
13m
GPU Memory
9,402
MiB
23

---

Table Hyperparameters and computing resources used in the mathematical reasoning experiments.
h: hour, m. minute.
Hyperparameter
Umi-LoRA(Miitral)
Uni-LoRA(GGema)
VB-LoRA(Mistral) VB-LoRA(Gemma)
GPUs
Optimizer
Adam Cosine
Learning Rate Schedule
Batch Size
Accumulation Steps
64
Epochs
Warmup Ratio
0.02
Rank
Vector bank
256
Vector bank
2048
2048
[0g)
524,288
524,288
Learning Rate (Vector bank)
IE-3
IE-3
Learning Rate (Logits)
1E-2
1E-2
Learning Rate (0.,)
2E-3
2E-3
Training Time
15.5h
15.1h
19.3h
17.5h
GPU Memory
48,984 MiB
59,488 MiB
50,426
60,166
Table 10: Hyperparameters and computing resources used in instruction tuning the Cleaned Alpaca Dataset. h: hour. 7B: Llama2-7B, 13B: Alama2--3B.
Hyperparameter
Uni-LoRA-7B Uni-LoRA- 13B
LoRA-7B
LoRA-13B
VB-LoRA-7B
VB-LoRA-13B
GPUs
Optimizer
Adam
Warmup Ratio
0.1
Batch Size
Ascumulation Steps
Epochs
LR Schedule
Linear
Rank
Vector bank
256
256
Vector bank
2048
4096
[0al
524,288
1,048,766
Learning Rate (Vector bank)
11:3
119-3
Learning Rate (Logits)
IE-2
1E-2
Learning Rate (LoRA)
4e-4
4e-4
Learning Rate (f.)
8c-4
8c-4
Training Time
3b
4.5h
3h
4.5h
3.8h
5.7h
GPU Memory
7,212 MiB
11,752 MiB
7,736 MiB
12,320 MiB
7,418 MiB
12,244 MiB
A.3 Impact Hyperparameters on Model Performance
To investigate how the number of trainable parameters d affects model performance. we conducted ablation studies by fine-tuning RoBERTa-Large on the SST-2 task (from GL.UE) and Gemma-7B on the mathematical reasoning benchmarks, where d is varied while keeping all other settings at their default values. The results are summarized Figure 3 Our experiments show that performance improves rapidly with increasing d when d is small, and then plateaus d grows larger.
SST-2 Accurscy
DSVBE MATH Accuracy
12
(a) Accuracy on SST-2 as d increases.
(b) Accuracics on GSM8K and MATH as d increases.
Figure 3: Evolution of Uni-LoRA accuracies on different benchmarks as the number of trainable parameters d increases.
24

---

Table Hyperparameters and computing resources used in the vision tasks. h: hour, m: minute.
Model
Hyperparameter
OxfordPets
StanfordCars
CIFARIO
DTD
EurSAT FGVC RESISC45
CIFAR100
Optimizer
Adam
Weight Decay
0.01
LR Schedule
Linear
Init. 0a
24( (000022 0.02)
Epoch
20
# GPUs
Learning Rate (Head)
SE-3
5E-2
5E-2
SE-2
5E-2
5E-2
5E-2
1E-2
Learning (9a)
SE-3
1E-2
IE:2
1E.2
1E:2
1E-2
[9al
72,000
Rank
BASE
Batch Size GPU
128
Truining Time
6m
23m
49m
1mm
40m
53m
30m
1.6h
Memory
MiB
GPUs
Learning Rate (Head)
26-2
5E-3
IE-2
2E-2
IE-2
5E-3
Learning Rate (9a)
1E-2
1E-2
1E-2
IE-2
IE-2
1B-2
1E-2
1E-2
[8al
144,000
Rank
LARGE
Batch Size Per GPU
Training
11m
40m
GPU Memory
30.058 MiB
Table 12: All results reported are generated using consistent version of the GPT-4 API. Score refers the evaluation on MT-Bench with single-turn dialogues.
Model
Method
# Parameters
Score
Training Time
GPU memory
LoRA (Rank 64)
159.9M
5.62
3.0 h
7,736 MiB
LLAMA2-7B
LoRA (Rank 4)
10.0M
5.34
2.9 h
7,044 MiB
Uni-LoRA (Rank 4)
0.52M
5.58
3.0
7,212 MiB
LoRA (Rank 64)
250.3M
6,20
4.5
12,320 MiB
LLAMA2-13B
LoRA (Rank 4)
15.60M
6.06
4.1 h
11,634 MiB
Uni-LoRA (Rank 4)
1.0M
6.34
4.5
11,752 MiB
For fair comparison, we follow VB-LoRA and set the LoRA rank in the paper. To evaluate the influence of rank here we conduct additional analysis by varying for RoBERTa-Larre on the SST-2 task (from GLUU) and Gemma-7B on the mathematical reasoning benchmarks. The results reported in Figure show that Uni-LoRA maintains stable performance across wide range of ranks, with r achieving the best balance between accuracy and efficiency.
551-2
CONBC 1ATT1
70
(a) Accuracy on SST-2 with different r.
(b) Accuracies on GSM8K and MATH with different r.
Figure 4: Performance comparison of Uni-LoRA across different rank
A.4 Comparison with LoRA of the Same Rank
In the instruction tuning experiments, we follow the same setup in VeRA 4 and VB-LORA [6] comparing our method against LoRA with a rank of 64, which a commonly used setting in prior works. To ensure fair comparison with our Uni-LoRA, which uses low rank of 4, additionally
25

---

include set of instruction tuning experiments where LoRA is also configured with rank 4. These experiments follow the same setup as Section 4.3] with the only difference being the LoRA rank.
The results provided in Table T2 It can be observed that reducing the LoRA rank from 64 leads to noticeable performance score drop. Moreover, the performance of the LoRA rank-4 model is consistently worse than that of Uni-LoRA. In contrast, Uni-LoRA A employs fixed, sparse projection matrix, where only the indices and values of the nonzero entries are involved in computation. This leads to an extremely efficient implementation. As result, Uni-LoRA incurs only marginal increases in training time and memory consumption compared to the rank-4 LoRA baseline, while achieving notable score improvements.
A.5 Licenses and Asset Usage
We document all external assets used in this work, including models and datasets, along with their licenses and source URLss
Natural Language Understanding. We RoBERTa-base and RoBERTa-large models developed by Facebook AI, released under the MIT License and available at: https: thuggingfas co/ We evaluate on the GLUE bench-
roberta-base base Intpps:/ Bettps://huggingfacee wot.:/acecooooberaaaaaaaaaaa mark, which is publicly available at /tttss://glueeebnccaaar com/ and composed of multiple sub-datasets under various open licenses, documented on the GLUE website.
Mathematical Reasoning. We fine-tune the Mistral-7B-v0.1 model, released under the Apache 2.0 License and available at the Gemma-7B model, which requires agreement Google's s usage license and is available co/ hooooogllggeemm-77777 We use the MetaMathQA dataset, available un- and
and
der the MIT License at htttps://huggingttee evaluate on GSM8K and MATH datasets, both under the MIT License and available at: |htpss //hgggingface.cc//////////
Instruction Tuning. We use the Cleaned Alpaca dataset, which improves upon the original Alpaca dataset. Both versions are licensed under CC BY-NC 4.0 and available at: hhtpps htt.://nngaaee.cootttttttttttttt caaitssu-aab/ae aabllaaaaaaaaa labaaauu-abaappaaaaaaa yaahae/alpaca--lleaeed We evaluate on MT-Bench, released under CC BY 4.0 and available abtt:p://buug.ingaacc.o//// http://gnnnaaaacc.c.///ttttet/ea https//mmss/////aam m/t _estt__behhh__eeeeeeee hww.ahhuaaaaaaaaaaaaeaa. judgmentss Fine-tuning is performed on the I.LMA model, licensed under the LLAMA Community License and available wwtp:e/oooeeeaaaeeeeeeeeeeeeeeee using the QLoRA framework, released under the MIT License and available at htttps://githuu com/ /ttpaaarriooo///eee/e//ee /qlora
bottss:/huugingee
ctt /otaaessts
hhnttps:/hhugiing:
All assets were used in compliance with their respective licenses. No proprictary or restricted-access
data was used in this study.
26


Reviews:
[{"review_id": "qA06ZqfEdQ", "reviewer": "Reviewer_FicU", "summary": "Uni-LoRA proposes a unified framework that maps various LoRA variants into a shared low-dimensional subspace via a global projection matrix, enabling parameter sharing and compression. Its core innovation lies in leveraging a single trainable vector to achieve parameter sharing while preserving the isometric structure of the parameter space. This approach significantly reduces memory and computational overhead, while achieving performance comparable to traditional fine-tuning methods on tasks such as GLUE, GSM8K, and MATH.", "questions": "1. The distance maintained by the projection matrix P is Euclidean, while most data typically resides in non-Euclidean spaces. Could the adoption of geodesic distance on manifolds be considered to optimize the projection matrix P?\n2. The projection matrix P is relatively simple, and it is unclear whether it can maintain good performance on more complex data. Could the use of multiple projection matrices be considered to achieve better performance?\n3. Could a discussion on the dimension d of the low-dimensional subspace be included in the parameter experiments to explore the impact of different dimension choices on performance?", "limitations": "1. Despite the outstanding performance of Uni-LoRA on NLP tasks, its effectiveness in cross-domain applications or complex tasks (such as image processing and cross-lingual transfer) has yet to be fully validated. The authors could further discuss the method's adaptability in other domains, such as computer vision or reinforcement learning, and highlight potential challenges related to its adaptability.\n2. The experimental validation in the paper primarily focuses on medium- to large-scale models (e.g., Gemma-7B, Mistral-7B), Uni-LoRA: One Vector is All You Need with limited testing on extremely large models (such as GPT-3). There is insufficient discussion on whether the method can effectively operate on ultra-large models, particularly in distributed training environments.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Advantages:\n1. Uni-LoRA demonstrates high computational and memory efficiency in the application of large-scale pre-trained models. The experiments are sufficiently conducted and are backed by a solid theoretical foundation.\n2. The structure of the paper is clear, effectively conveying the core ideas of the research, with a well-defined experimental design. The paper provides reproducible experimental procedures and code support.\n3. Uni-LoRA reduces the number of parameters while maintaining good performance, making it highly significant for practical applications, particularly in resource-constrained environments.\n4. The paper introduces a unified framework that integrates various LoRA variants through a global projection matrix, allowing all variants to be optimized via a shared low-dimensional subspace, thus reducing computational complexity. Additionally, the proposed projection matrix P ensures that the geometric structure of the optimization space remains preserved.\n\nDisadvantages:\n1. The details of the projection matrix selection are not fully addressed. While the projection matrix P is described as globally shared and optimized via backpropagation, the paper lacks sufficient discussion on how to dynamically adjust and select the optimal dimension d in practical applications.\n2. While the unified framework proposed in the paper improves both parameter and computational efficiency, the simplification of the method may limit its adaptability in certain tasks, especially in more complex scenarios. The effectiveness of a single projection matrix in maintaining good performance for complex tasks warrants further investigation.", "was_revised": true, "final_justification": "Thank you for the clarifications and additional experiments. After carefully considering all reviews and the authors' responses, I find no major weaknesses remaining. I therefore keep my positive assessment of this work.", "rebuttal": "Dear Reviewer FicU，\n### **Disadvantage 1, Question 3 – Impact of Projection Dimension *d* on Model Performance**\n\nThanks for the suggestion. First, we would like to clarify that our projection matrix **P** is not optimized via backpropagation. Instead, **P** is randomly sampled and fixed (see Theorem 1). Since **P** is not learned, our method incurs **lower memory and computational overhead** compared to the baseline methods, such as VBLoRA and Tied-LoRA, which require optimizing projection parameters.\n\nTo investigate the impact of *d* on model performance, we conducted ablation studies by fine-tuning RoBERTa-Large on the SST-2 task (from GLUE) and Gemma-7B on the mathematical reasoning benchmarks, where *d* is varied while keeping all other settings at their default values. The results are summarized in the tables below:\n\n#### SST-2 Accuracy vs. *d*\n\n| Dim *d*   | 45   | 180  | 720  | 2,880 | 11,520 | 23,040 | 46,080 | 184,320 | 737,280 |\n|-----------|------|------|------|------|--------|--------|--------|--------|--------|\n| Accuracy  | 87.2±0.2 | 90.4±0.4 | 94.3±0.4 | 94.8±0.2 | 95.6±0.2 | 96.3±0.2 | 96.5±0.3 | 96.5±0.2 | 96.4±0.3 |\n\n#### GSM8K / MATH Accuracy vs. *d*\n\n| Dim *d*   | 128  | 2,048 | 32,768 | 524,288 | 1,048,576 | 2,097,152 |\n|-----------|------|------|--------|--------|----------|-----------|\n| GSM8K ACC | 0    | 0.6141 | 0.6581 | 0.7559 | 0.7680   | 0.7627    |\n| MATH ACC  | 0.0002 | 0.2194 | 0.2714 | 0.2894 | 0.2994   | 0.3014    |\n\nOur experiments show that performance improves rapidly with increasing *d* when *d* is small, and then plateaus as *d* grows larger. We will include these results in the updated manuscript.\n\n---\n\n### **Disadvantage 2, Question 2, Limitation 1 – Performance on More Complex Tasks**\n\nTo evaluate Uni-LoRA in more complex tasks, we followed the FourierFT setup for computer vision and conducted additional experiments with the results reported in the table below.\n\nSpecifically, we applied Uni-LoRA with *d* = 74,000 on ViT-Base and *d* = 144,000 on ViT-Large, for rank *r* = 4. A  grid search was conducted over:\n\n- head learning rate: {1e-3, 2e-3, 5e-3, 1e-2}  \n- trainable vector $\\theta_d$ learning rate: {2e-3, 5e-3, 1e-2, 2e-2, 5e-2}  \n\nwhile fixing the number of training epochs to 10.\n\nEach experiment was repeated five times, and we reported the mean and standard deviation across the five runs.\n\n>All  results, except those for Uni-LoRA, are taken from the original FourierFT paper, including Full Fine-tuning (FF) and Linear Probing (LP), in which only the classification head is fine-tuned.\n\n| Model     | Method           | # Trainable Parameters | OxfordPets       | StanfordCars     | CIFAR10          | DTD              | EuroSAT          | FGVC             | RESISC45         | CIFAR100         | Avg.   |\n|-----------|------------------|------------------------|------------------|------------------|------------------|------------------|------------------|------------------|------------------|------------------|--------|\n| ViT-Base  | LP               | -                      | 90.28±0.43       | 25.76±0.28       | 96.41±0.02       | 69.77±0.67       | 88.72±0.13       | 17.44±0.43       | 74.22±0.10       | 84.28±0.11       | 68.36  |\n|           | FF               | 85.8M                  | 93.14±0.40       | 79.78±1.15       | 98.92±0.05       | 77.68±1.21       | 99.05±0.09       | 54.84±1.23       | 96.13±0.13       | 92.38±0.13       | 86.49  |\n|           | LoRA             | 581K                   | 93.19±0.36       | 45.38±0.41       | 98.78±0.05       | 74.95±0.40       | 98.44±0.15       | 25.16±0.16       | 92.70±0.18       | 92.02±0.12       | 77.58  |\n|           | FourierFT   | 72K                    | 93.21±0.26       | 46.11±0.24       | 98.58±0.07       | 75.09±0.37       | 98.29±0.04       | 27.51±0.64       | 91.97±0.31       | 91.20±0.14       | 77.75  |\n|           | FourierFT  | 239K                   | 93.05±0.34       | 56.36±0.66       | 98.69±0.06       | 77.30±0.61       | 98.78±0.11       | 32.44±0.99       | 94.26±0.20       | 91.45±0.18       | 80.29  |\n|           | **Uni-LoRA**         | **72K**                    | **92.04±0.14**       | **63.26±0.45**       | **98.80±0.06**       | **75.76±0.48**       | **98.52±0.10**       | **30.08±0.51**       | **93.27±0.22**       | **92.32±0.07**       | **80.50**  |\n| ViT-Large | LP               | -                      | 91.11±0.30       | 37.91±0.27       | 97.78±0.04       | 73.33±0.26       | 92.64±0.08       | 24.62±0.24       | 82.02±0.11       | 84.28±0.11       | 72.96  |\n|           | FF               | 303.3M                 | 94.43±0.56       | 88.90±0.26       | 99.15±0.04       | 81.79±1.01       | 99.04±0.08       | 68.25±1.63       | 96.43±0.07       | 93.58±0.19       | 90.20  |\n|           | LoRA             | 1.57M                  | 94.82±0.09       | 73.25±0.36       | 99.13±0.05       | 81.79±0.45       | 98.63±0.07       | 42.32±0.98       | 94.71±0.25       | 94.87±0.10       | 84.94  |\n|           | FourierFT  | 144K                   | 94.46±0.28       | 69.56±0.30       | 99.10±0.04       | 80.83±0.43       | 98.65±0.09       | 39.92±0.68       | 93.86±0.14       | 93.31±0.09       | 83.71  |\n|           | FourierFT  | 480K                   | 94.84±0.05       | 79.14±0.67       | 99.08±0.05       | 81.88±0.50       | 98.66±0.03       | 51.28±0.66       | 95.20±0.07       | 93.37±0.11       | 86.68  |\n|           | **Uni-LoRA**         | **144K**                  | **94.61±0.09**       | **77.53±0.08**       | **99.10±0.03**      | **79.33±0.52**       | **98.63±0.09**       | **46.51±0.49**       | **93.54±0.13**       | **93.72±0.13**       | **85.37**  |\n\n\nExperiments show that given the same number of trainable parameters Uni-LoRA significantly outperforms FourierFT across computer vision benchmarks. Moreover, when the number of parameters in FourierFT is three times that of Uni-LoRA, their performances on computer vision benchmarks are comparable. These results demonstrate the strong generalizability of Uni-LoRA to complex tasks.\n\nOur core observation is that tasks (NLP or CV) that benefit from LoRA also tend to perform well with Uni-LoRA. This is because Uni-LoRA essentially reconstructs the LoRA parameters from a low parameter space via $P\\times\\theta_d$. We will incorporate these CV results into the main paper and release all the code for reproducibility.\n\nRegarding multiple projection matrices, this is a very interesting idea, and we plan to explore it in our future work.\n\n\n---\n### **Question1 – Consider geodesic distance on manifolds to optimize P?**\n\nThank you for the suggestion. First of all, as we discussed above, the projection matrix **P** is not optimized. Instead, **P** is randomly sampled and fixed (see Theorem 1). Since **P** is not learned, the number of trainable and stored parameters of Uni-LoRA is extremely small (i.e., $|\\theta_d|$ + 1 random seed). Optimizing **P** would increase the number of trainable and stored parameters dramatically and incur large memory and computational overhead.\n\nIf we ignore the memory and computational overhead and optimize **P** in the non-Euclidean space, it might be possible to replace Adam with its Riemannian adaptive counterparts [1, 2]. Adapting to the manifold setting, however, requires computing the function’s gradient and, in some cases, the Hessian matrix -- which is computationally prohibitive. The practicality of methods and libraries such as Geoopt [2] and Pymanopt [3] for large-scale deep learning optimization remains uncertain.\n\n**References:**  \n[1] Bonnabel, Silvere. *\"Stochastic gradient descent on Riemannian manifolds.\"* IEEE Transactions on Automatic Control 58.9 (2013): 2217-2229.  \n[2] Bécigneul, Gary, and Octavian-Eugen Ganea. *\"Riemannian Adaptive Optimization Methods.\"* International Conference on Learning Representations (ICLR 2019). Vol. 9. Curran, 2023.  \n[3] Townsend, James, Niklas Koep, and Sebastian Weichwald. *\"Pymanopt: A python toolbox for optimization on manifolds using automatic differentiation.\"* Journal of Machine Learning Research 17.137 (2016): 1-5.\n\n---\n### **Limitation 2 – Scalability in Distributed Systems**\n\nThank you for the suggestion. We apologize that we were unable to fine-tune on GPT-3 (175B) due to resource constraints. However, Uni-LoRA naturally supports data parallelism, and indeed, the results reported in Table 3 were obtained using multi-GPU data-parallel training.\n\nTo enable model parallel training of Uni-LoRA across multiple GPUs, we can adopt a strategy where each GPU maintains a local copy of the trainable vector $ \\theta_d $, and synchronization is achieved via an all-reduce operation on the gradients during backpropagation. This approach is highly efficient for Uni-LoRA, as the number of trainable parameters is extremely small, resulting in negligible communication overhead across devices.\n\nWe would like to emphasize that the extremely small number of trainable parameters constitutes a key advantage of Uni-LoRA over other PEFT methods. For example, when fine-tuning LLaMA2-13B, Uni-LoRA requires synchronizing only 1M parameters, whereas VB-LoRA may require synchronization of up to 256M parameters in the worst case. We will include this discussion in the updated manuscript."}, {"review_id": "Rhkq52u99v", "reviewer": "Reviewer_6tip", "summary": "This paper presents Uni-LoRA, a unified framework for Low-Rank Adaptation (LoRA) that generalizes and improves upon existing parameter-efficient fine-tuning (PEFT) methods. The authors claim that their Uni-LoRA achieves state-of-the-art parameter efficiency while\noutperforming or matching prior approaches in predictive performance.", "questions": "Check the weaknesses part, I'm mainly concerned about 1-5.", "limitations": "yes, some limitations are discussed in Conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "> **Strengths:**\n>\n> Uni-LoRA advances PEFT by unifying LoRA variants into a single framework, this can help us better understand LoRA-based fine-tuning methods and further reduce the number of parameters required by LoRA.\n\n> **Weaknesses:**\n> \n> (1) Line 213-214 (Theorem 1) says \"P is a projection matrix where each row selects exactly one entry uniformly at random to be nonzero, and sets all other entries to zero\". Lacks stability analysis (sampling has randomness).\n>\n> (2) Theorem 1 is incorrect because it fails to account for cases where all rows of matrix P are sampled identically. For example, when D=3, P = ((1/√3, 1/√3, 1/√3)ᵀ, (0, 0, 0)ᵀ). In this case, clearly PᵀP ≠ I. Additional analysis on the impact of P's rank should be included.\n>\n> (3) For fine-tuning, LoRA is not shown to reach a comparable performance as full-rank fine-tuning [1]. Further investigation should be conducted on the impact of reducing the number of LoRA parameters.\n>\n> [1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML 2024 Oral.\n>\n> (4) Lines 329-334: Why does \"local projection perform worse than global projection\"? Under sufficient training (with other hyperparameters being the same), each local part should learn parameters corresponding to its global counterpart.\n>\n> (5) Line 304: When comparing LoRA, only the case of rank=4 was evaluated. Different ranks should be compared to better demonstrate the effectiveness of the proposed method.\n>\n> (6) Lines 319-320: Could this be a formatting issue?", "was_revised": true, "final_justification": "While I appreciate the authors' response and have carefully considered their explanations, my initial assessment of the paper's contribution remains unchanged. However, I have adjusted the score upward in recognition of their thorough rebuttal efforts.", "rebuttal": "Dear Reviewer 6tip,\n\n### **Weakness 1 – Analysis of Method Stability**\n\nThanks for the question. In Table 2 of the main paper, we report the median and standard deviation across five different seeds for each method on six tasks. Below, we present the average of the standard deviations across these six tasks:\n\n| Method      | Average Std (RoBERTa-Base) | Average Std (RoBERTa-Large) |\n|-------------|----------------------------|------------------------------|\n| LoRA        | 0.550                      | 0.717                        |\n| VeRA        | 0.417                      | 0.567                        |\n| Tied-LoRA   | 0.850                      | 0.383                        |\n| VB-LoRA     | 0.500                      | 0.717                        |\n| FourierFT   | 0.583                      | 0.677                        |\n| Uni-LoRA    | 0.500                      | 0.677                        |\n\nAs can be seen, our method demonstrates comparable stability to other baselines. For RoBERTa-Base, only VeRA has a slightly lower average standard deviation than Uni-LoRA. For RoBERTa-Large, both VeRA and Tied-LoRA have slightly better stability, but Uni-LoRA remains highly competitive in terms of stability. We will clarify this in the main paper.\n\n---\n### **Weakness 2  – Clarification on Theorem 1**\n\nWe would like to clarify that Theorem 1 assumes $ n_j > 0 $, indicating that each column of the projection matrix **P** must have at least one non-zero entry (see Theorem 1). The counter example provided does not satisfy this assumption.\n\nIn practice, the chance to sample a **P** in which one column is entirely zero is extremely low because **P** is a very tall and narrow matrix (where each row has only one non-zero element at a random location). For example, in the LLaMA-2 7B experiments in Sec. 4.3, **P** has a shape of 10M × 0.52M. In this case, the probability that a column is entirely zero is less than 0.23%.\n\nTo ensure the theorem’s condition ($n_j > 0$) is always held, we can re-sample **P** if one column happens to be all zeros. We will clarify this further in the main paper.\n\n---\n### **Weakness 3  – LoRA vs. FFT and Impact of Reducing the Number of LoRA Parameters**\n\nNumerous studies have shown that LoRA significantly reduces memory and compute cost while achieving comparable or even superior performance to full fine-tuning (FFT) [1-5]. Its efficiency and effectiveness have led to widespread adoption in industry.\n\nMeanwhile, the parameter redundancy of LoRA has been explored in several works, such as VeRA, Tied-LoRA, VB-LoRA, and LoRA-XS. These methods reduce the number of trainable parameters while maintaining similar performance to LoRA, highlighting the parameter redundancy of LoRA. Uni-LoRA builds upon this insight and further improves the parameter efficiency for LoRA.\n\n**References:**  \n[1] Hu, Edward J., et al. *\"Lora: Low-rank adaptation of large language models.\"* ICLR 1.2 (2022): 3.  \n[2] Dettmers, Tim, et al. *\"Qlora: Efficient finetuning of quantized llms.\"* NeurIPS 36 (2023): 10088-10115.  \n[3] Biderman, Dan, et al. *\"LoRA Learns Less and Forgets Less.\"* Transactions on Machine Learning Research.  \n[4] Shuttleworth, Reece, et al. *\"Lora vs full fine-tuning: An illusion of equivalence.\"* arXiv preprint arXiv:2410.21228 (2024).  \n[5] Lian, Chenyu, et al. *\"Less could be better: Parameter-efficient fine-tuning advances medical vision foundation models.\"* arXiv:2401.12215 (2024).  \n\n---\n### **Weakness 4  – Why Does Local Projection Perform Worse Than Global Projection?**\nThanks for the question. As highlighted by AdaLoRA [1], when fine-tuning with LoRA, not all layers contribute equally, i.e., the importance and information content of each LoRA module varies. Prior methods such as VeRA, Tied-LoRA, and LoRA-XS use local projections with the same projection dimension for each layer, which means layers with different information content are projected into different subspaces of the same size. This same-sized treatment for each layer is suboptimal. In contrast, global projection projects all LoRA parameters into a shared subspace to mitigate the aforementioned issue. \n\nLocal projection is analogous to pouring five unequal volumes of water into five identical cups, which may lead to overflow in some cups and under-utilize others. In contrast, global projection is analogous to pouring all water into a single container, which allows more efficient utilization of capacity. The ablation studies in Table 6 support this claim. We will clarify this further in the updated manuscript.\n\n**References:**  \n[1] Zhang, Qingru, et al. *\"Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning.\"* ICLR 2023.\n\n---\n### **Weakness 5  – Additional Analysis on the Influence of Rank *r***\n\nThanks for the suggestion. For a fair comparison, in the paper we followed VB-LoRA and set the LoRA rank *r* = 4. To evaluate the influence of rank *r*,  here we conducted additional analysis by varying *r* for RoBERTa-Large on the SST-2 task (from GLUE) and Gemma-7B on the mathematical reasoning benchmarks. The results reported in the table below show that Uni-LoRA maintains stable performance across a wide range of ranks, with *r* = 4 achieving the best balance between accuracy and efficiency. We will include this ablation study in the updated manuscript.\n\nSST-2 Accuracy vs. *r*\n\n| Rank *r*         | 1        | 2        | 4        | 8        | 16       | 32       |\n|------------------|----------|----------|----------|----------|----------|----------|\n| SST-2 ACC        | 89.2±1.4 | 96.1±1.4 | 96.3±0.2 | 96.2±0.1 | 96.3±0.2 | 96.2±0.1 |\n\n\n\n GSM8K / MATH Accuracy vs. *r*\n\n| Rank *r*         | 2       | 4       | 8       | 16      | 32      | 64      |\n|------------------|---------|---------|---------|---------|---------|---------|\n| GSM8K ACC        | 0.7528  | 0.7559  | 0.7521  | 0.7591  | 0.7544  | 0.7522  |\n| MATH ACC         | 0.2932  | 0.2894  | 0.2970  | 0.2996  | 0.2940  | 0.2958  |\n\n---\n### **Weakness 6  – formatting issue**\nThanks for pointing this out. We will fix it in the updated manuscript."}, {"review_id": "UYOR9SSTit", "reviewer": "Reviewer_k62c", "summary": "The paper presents a general framework that expresses mappings between the trained parameter subspace and the LoRA parameters (usually matrices $A_l$ and $B_l$) of different more efficient LoRA variants (LoRA, Tied-LoRA, VeRA, VB-LoRA, LoRA-XS, FourierFT). The paper expresses the concatenation of all trained and fixed LoRA parameters $\\theta_D\\in \\mathbb R^{D}$ as a product of a projection matrix $P\\in\\mathbb R^{D\\times d}$ (trained or fixed) and a trained lower-dimensional vector $\\theta_d\\in \\mathbb R^d$:\n$$\n\\theta_D = P \\theta_d \\text.\n$$\n\nBased on this expression, where each trained parameter can influence parameters in multiple LoRA modules, the paper proposes Uni-LoRA for greater parameter efficiency. In Uni-LoRA, $P$ is fixed and each of its rows has a single randomly chosen non-zero element. The sparsity enables $O(D)$ time complexity. The value of non-zero elements is set to $P[i, j] = 1/\\sqrt{[\\![\\sum_{j=1}^D P[i, j]\\neq 0]\\!]}$ to make $P$ isometric (L2 distance preserving).\n\n**Results**\n\nThe method is compared with other PEFT methods (LoRA, Tied-LoRA, VeRA, LoRA-XS, and FourierFT) on the GLUE language understanding benchmark (6 tasks) with 2 RoBERTa-base and RoBERTa-large. Uni-LoRA optimizes about 23k (~8% of LoRA) parameters (less than all the other methods) and scores probably among the best 2 PEFT methods.\n\nUni-LoRA is also very competitive on mathematical reasoning (Mistral-7B and Gemma-7B on GSM8K and MATH) and instruction tuning (LLama2 7B and 13B on MT-Bench) while optimizing the least amount of parameters (0.52M - 1M, 0.26%-0.4% of LoRA).\n\nAblation experiments on 4 GLUE tasks show that the uniform projection of UniLoRA is more parameter-efficient than non-uniform (twice as much parameters allocated to $A$ matrices than to $B$ matrices) and (maybe) local (per-layer) projections.", "questions": "**Questions:**\n\n1. Please provide better support regarding weakness 1.\n2. Table 3: Do you have any intuition why LoRA outperforms other methods on MATH, but not GSM8k?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The presented framework is interesting.\n2. The proposed method outperforms or competes very well with other PEFT methods.\n3. The paper is very well written.\n4. The source code for reproducing the experiments will be published (If I understand correctly.).\n\n**Weaknesses**\n\n1. L202, 233-237: \"isometry [...] can substantially enhance adaptation performance\". This sounds intuitive, but I don't find it clear how important isometry is. I suggest supporting the claims better.\n\nMinor issues:  \n- I am unable to find the code for reproducing some experiments (Fastfood).\n- LoRA-XS could be described more clearly: SVD is not mentioned.\n- L57: Consider clarifying in which way $P$ is theoretically grounded.\n- Figure 1: \"top-k of learnable logits\" is mentioned in the legend, but I don't see it being used.\n- L167: I don't find it clear why \"Apparently\" is used.", "was_revised": true, "final_justification": "The authors have addressed the most important concerns of all reviewers. The discussion has confirmed my positive score.", "rebuttal": "Dear Reviewer k62c,\n### **Weakness 1, Question 1 – On the Importance of Isometry**\n\nThanks for the question. We ensure that the projection matrix **P** in Uni-LoRA is isometric for the following key reasons:  \nYes, standard gradient descent methods are not parameterization-invariant. Poorly scaled parameterizations, which distort the geometric structure of the optimization landscape, can lead to unstable step sizes and slow convergence. Non-isometric transformation can make changes sensitive in one direction and insensitive in another direction. (Please see: J. Nocedal and S. J. Wright, *Numerical Optimization* 2nd ed., Chapters 2.2 and 4.5 for more discussions on the effects of poor scaling.)\n\nAlthough heuristics such as gradient clipping and gradient-normalized Adam [1] exist as remedies, we prefer to maintain isometry as a desirable property from the outset in reparameterization. This ensures that the conditioning of the optimization problem remains unchanged -- no artificial ill-conditioning is introduced by the reparameterization. We will add this discussion in the updated manuscript.\n\n**References:**  \n[1] Yu, Adams Wei, et al. \"Block-normalized gradient method: An empirical study for training deep neural network.\" arXiv preprint arXiv:1707.04822 (2017).\n\n---\n\n### Minor Issues\n\n- **Unable to find the code for reproducing some experiments (Fastfood)**  \n  For reproducibility, we will open-source all our Uni-LoRA code, including our Fastfood implementation and the Uni-LoRA code for ViT fine-tuning introduced during the rebuttal.\n\n- **LoRA-XS could be described more clearly: SVD is not mentioned.**  \n  We apologize for the lack of clarity regarding LoRA-XS. We will clarify that its LoRA-like matrices A and B are derived from the SVD of the pretrained weight matrix **W**.\n\n- **L57: Consider clarifying in which way is theoretically grounded.**  \n  Thanks for pointing this out. As we analyzed in Sec. 3.3, **P** possesses three desired properties for adaptation performance, i.e., globality, uniformity/load-balanced, and isometry. We will clarify this at L57 and add a reference to Sec. 3.3.\n\n- **Figure 1: \"top-k of learnable logits\" is mentioned in the legend, but I don't see it being used.**  \n  Thanks for pointing this out. The “top-k of learnable logits” is specific for VB-LoRA, and refers to the orange boxes in the VB-LoRA illustration (Fig. 1d). In this case, *k=2* refers to the two diagonal sub-matrices. We will clarify this in Fig. 1.\n\n- **L167: I don't find it clear why \"Apparently\" is used.**  \n  Thanks for the question. Since the projection matrix **P** in standard LoRA is just an identity matrix, which satisfies *PᵀP = I*, it is inherently isometric.\n\n---\n\n### **Question 2 – Any intuition why LoRA outperforms other methods on MATH, but not GSM8k?**\n\nThanks for pointing this out. This is an interesting observation that we neglected before. We don't have a solid explanation or intuition at this time. We will investigate this in our future work."}, {"review_id": "Tux4a4bNFi", "reviewer": "Reviewer_5yKv", "summary": "In this paper, the authors present Uni-LoRA, a unified framework for understanding and improving parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), particularly those based on Low-Rank Adaptation (LoRA). The key insight is that various LoRA variants, including Tied-LoRA, VeRA, and VB-LoRA, can be reinterpreted through a common lens: all implicitly perform a projection from a low-dimensional subspace \\( \\mathbb{R}^d \\) to the full LoRA parameter space \\( \\mathbb{R}^D \\), where \\( d \\ll D \\). Under this formulation, the primary difference among methods lies in the choice of the projection matrix \\( P \\in \\mathbb{R}^{D \\times d} \\). The authors further propose an isometric projection matrix that enables global parameter sharing across layers and modules, leading to a more efficient use of trainable parameters. The authors conduct experiments to demonstrate the efficiency of the Uni-LoRA.", "questions": "1. How do the authors ensure training stability, particularly given the random initialization of the projection matrix? Does this introduce higher variance or increase the standard error across runs?\n\n2. Can the authors provide results under varying values of r and d to better illustrate the trade-off between parameter efficiency and performance?\n\n3. It would be helpful to include an analysis of GPU memory consumption trends as the model size increases, to understand the scalability of Uni-LoRA in terms of memory footprint.\n\n4. I am curious about the performance of Uni-LoRA on reasoning-focused models. Could the authors provide preliminary results or insights in this area?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. Presents a novel and unified perspective that subsumes a wide range of LoRA-based extreme PEFT methods under a common framework.\n\n2. Provides both theoretical justification and empirical validation for the proposed projection matrix design.\n\n3. The method is lightweight and easily integrable into existing mainstream fine-tuning frameworks.\n\n**Weaknesses**:\n\n1. The comparisons in Table 3 and Table 4 are limited to only two or three baselines. It would strengthen the evaluation to include methods such as LoRA-XS and FourierFT under comparable trainable parameter budgets.\n\n2. While the proposed projection matrix improves parameter efficiency, it may introduce additional inference latency compared to standard LoRA. An analysis of inference speed would provide a more complete picture of the trade-offs.\n\t\n3. Although Uni-LoRA significantly reduces the number of trainable parameters, the gains in training time and memory usage are marginal, particularly on larger models (e.g., LLaMA2-7B and LLaMA2-13B, as shown in Table 10). Additional discussion would be beneficial.", "was_revised": true, "final_justification": "My concerns have been addressed by the author during the rebuttal. Therefore, I raised my score.", "rebuttal": "Dear Reviewer 5yKv,\n### **Weakness 1 – Limited baselines in Tables 3 and 4**\nThanks for the suggestion. Tables 3 and 4 initially included only 2–3 baselines because we aimed to include the results reported in their original papers. For Table 3, we have now added two additional baselines: VeRA and FourierFT. When reproducing these baselines, we used their original hyperparameter configurations for fine-tuning.\n\n| Model         | Method     | # Parameters | GSM8K | MATH  |\n|---------------|------------|--------------|-------|-------|\n| MISTRAL-7B    | Full-FT    | 7,242M        | 67.02 | 18.60 |\n|               | LoRA       | 168M         | 67.70 | 19.68 |\n|               | LoRA-XS    | 0.92M        | 68.01 | 17.86 |\n|               | VB-LoRA    | 93M          | 69.22 | 17.90 |\n|               | VeRA       | 1.39M        | 68.69 | 18.81 |\n|               | FourierFT  | 0.67M        | 66.92 | 17.50 |\n|               | **Uni-LoRA**   | **0.52M**        | **68.54** | **18.18** |\n| GEMMA-7B      | Full-FT    | 8,538M        | 71.34 | 22.74 |\n|               | LoRA       | 200M         | 74.90 | 31.28 |\n|               | LoRA-XS    | 0.80M        | 74.22 | 27.62 |\n|               | VB-LoRA    | 113M         | 74.86 | 28.90 |\n|               | VeRA       | 1.90M        | 74.98 | 28.84 |\n|               | FourierFT  | 0.59M        | 72.97 | 25.14 |\n|               | **Uni-LoRA**   | **0.52M**        | **75.59** | **28.94** |\n\nThe results demonstrate that Uni-LoRA achieves comparable performance to VeRA on mathematical reasoning tasks, while requiring only **half** as many trainable parameters on Mistral-7B and less than **one-third** on Gemma-7B. As of FourierFT, Uni-LoRA consistently outperforms it across both tasks, while using fewer trainable parameters. Furthermore, Uni-LoRA requires only 15 hours of training time under the same hardware setup, which is significantly faster than FourierFT (73 hours), and comparable to VeRA.\n\nAs for Table 4, the experimental setup is based on QLoRA-quantized models. Unfortunately, as stated in their documentation or code comments, the publicly released implementations of LoRA-XS and FourierFT are currently not compatible with quantized base models, so we could not include them as baselines in this setting.\n\n---\n### **Weakness 2 – Inference efficiency**\nThanks for the question. Similar to LoRA, at inference time, Uni-LoRA doesn’t introduce any memory and compute overhead beyond the backbone models since **B** and **A** matrices are merged to the backbone models. The difference between LoRA and Uni-LoRA is that after training, Uni-LoRA regenerates **B** and **A** matrices through $P×θ_d$, and then merges **B** and **A** to the backbone models for inference.\n\n---\n### **Weakness 3, Question 3 – GPU memory consumption**\nThanks for the question. Similar to prior works such as VeRA, VB-LoRA, FourierFT, Tied-LoRA, and LoRA-XS, our focus is on reducing the number of trainable parameters of LoRA. We do not claim reductions in memory or training cost compared to LoRA, and we only claim that Uni-LoRA reduces the storage cost of LoRA parameters significantly. Reducing the number of trainable parameters can substantially decrease both storage and transmission cost for model deployment and is highly valuable in industry. For example, in federated learning (FL), it can significantly lower communication costs while preserving user privacy. We will clarify this further in the manuscript.\n\n---\n### **Question 1 – Training stability, particularly given the random initialization**\n\nThanks for the question. In Table 2 of the main paper, we report the median and standard deviation across five different seeds for each method on six tasks. Below, we present the average of the standard deviations across these six tasks:\n\n| Method     | Average Std (RoBERTa-Base) | Average Std (RoBERTa-Large) |\n|------------|----------------------------|------------------------------|\n| LoRA       | 0.550                      | 0.717                        |\n| VeRA       | 0.417                      | 0.567                        |\n| Tied-LoRA  | 0.850                      | 0.383                        |\n| VB-LoRA    | 0.500                      | 0.717                        |\n| FourierFT  | 0.583                      | 0.677                        |\n| Uni-LoRA   | 0.500                      | 0.677                        |\n\nAs can be seen, our method demonstrates comparable stability to other baselines. For RoBERTa-Base, only VeRA has a slightly lower average standard deviation than Uni-LoRA. For RoBERTa-Large, both VeRA and FourierFT have slightly better stability, but Uni-LoRA remains highly competitive in terms of stability. We will clarify this in the main paper.\n\n---\n### **Question 2 – Results under varying values of r and d**\nThanks for the suggestion. We have now added the ablation on r and d. Please refer to our responses to Reviewer FicU, Disadvantage 1, Question 3  and Reviewer 6tip, Weakness 5.\n\n---\n### **Question 4 – Uni-LoRA on reasoning-focused models**\nThanks for the suggestion. We apologize that we couldn’t provide preliminary results on reasoning-focused models given our limited experience on reasoning models and a very tight rebuttal schedule. However, we could provide some insights in this area.\n\nTo the best of our knowledge, several recent works have successfully applied LoRA to reasoning-focused models [1-3]. Since Uni-LoRA essentially reconstructs the LoRA parameters from a low parameter space, we believe it can in principle be applied to any models that LoRA performs well (including NLP and CV models, and see our response to Reviewer FicU on Uni-LoRA for CV tasks). It’s interesting to extend Uni-LoRA to the reasoning-focused models, and we plan to explore this direction in the future.\n\n**References:**\n\n[1] Li, Xiang, et al. \"MoDE-CoTD: Chain-of-Thought Distillation for Complex Reasoning Tasks with Mixture of Decoupled LoRA-Experts.\" LREC-COLING  2024.  \n[2] Wang, Shangshang, et al. \"Tina: Tiny reasoning models via lora.\" arXiv preprint arXiv:2504.15777 (2025).  \n[3] Xue, Yihao, and Baharan Mirzasoleiman. \"LoRA is All You Need for Safety Alignment of Reasoning LLMs.\" arXiv preprint arXiv:2507.17075 (2025)."}]

Author rebuttal:

Meta-review: This paper presents Uni-LoRA, a unified framework for Low-Rank Adaptation (LoRA) that generalizes and improves upon existing parameter-efficient fine-tuning (PEFT) methods. Uni-LoRA demonstrates high computational and memory efficiency in the application of large-scale pre-trained models. The experiments are sufficiently conducted and are backed by a solid theoretical foundation. The reviews are consistently positive. The topic is interesting.
Decision: Accept (spotlight)