Parameter Efficient Fine-tuning via Explained Variance Adaptation
Authors: Fabian Paischer
Lukas Hauzenberger
Thomas Schmied
Benedikt Alkin
Marc Peter Deisenroth
Sepp Hochreiter
Venue: NeurIPS 2025 poster ()
Keywords: Foundation Models
PEFT
LoRA
Primary area: deep_learning
TL;DR: We propose EVA, a parameter-efficient fine-tuning method that initalizes LoRA weights in a variance-optimal manner and performs adaptive rank allocation to provably maximize the expected gradient signal at the beginning of fine-tuning.

Abstract: Foundation models (FMs) are pre-trained on large-scale datasets and then fine-tuned for a specific downstream task. The most common fine-tuning method is to update pretrained weights via low-rank adaptation (LoRA). Existing initialization strategies for LoRA often rely on singular value decompositions (SVD) of gradients or weight matrices. However, they do not provably maximize the expected gradient signal, which is critical for fast adaptation. To this end, we introduce **E**xplained **V**ariance **A**daptation (EVA), an initialization scheme that uses the directions capturing the most activation variance, provably maximizing the expected gradient signal and accelerating fine-tuning. EVA performs incremental SVD on minibatches of activation vectors and selects the right-singular vectors for initialization once they converged. Further, by selecting the directions that capture the most activation-variance for a given rank budget, EVA accommodates adaptive ranks that reduce the number of trainable parameters. We apply EVA to a variety of fine-tuning tasks as language generation and understanding, image classification, and reinforcement learning. EVA exhibits faster convergence than competitors and achieves the highest average score across a multitude of tasks per domain while reducing the number of trainable parameters through rank redistribution. In summary, EVA establishes a new Pareto frontier compared to existing LoRA
initialization schemes in both accuracy and efficiency.

Parameter Efficient Fine-tuning via Explained Variance Adaptation
Fabian Paischer Benedikt Alkin ELLIS Unit, LIT AI Lab, Institute for Machine Learning, JKU Linz, Austria
Lukas Hauzenberger"
Thomas Schmied Sepp Hochreiter
Marc Peter Deisenroth
Uniiersity College London
EMMI AI, Linz
4 NXAI GmbH, Linz, Austria paischerôml jku. at
Abstract
Foundation models (FMs) are pre-trained on large-scale datasets and then fine- tuned for a specific downstream task. The most common fine-tuning method is to update pretrained weights via low-rank adaptation (LoRA). Existing initial- ization strategies for LoRA often rely singular value decompositions (SVD) of gradients or weight matrices. However, they do not provably maximize the expected gradient signal, which is critical for fast adaptation. To this end, we introduce Explained Variance Adaptation (EVA), an initialization scheme that uses the directions capturing the most activation variance, provably maximizing the expected gradient signal and accelerating fine-tuning, EVA performs incremental SVD minibatches of activation vectors and selects the right-singular vectors for initialization once they converged. Further, by selecting the directions that capture the most ootivation-varianee for given rank budget, EVA accommodates adaptive ranks that reduce the number of trainable parameters. We apply EVA to variety of muniituning tasks language generation and understanding, image classification, and reinforcement learning. EVA exhibits faster convergence than competitors and achieves the highest average score across a multitude of tasks per domain while reducing the number of trainable parameters through rank redistribution. Pareto frontier compared existing LoRA
In summary, EVA establishes initialization schemes in both accuracy and efficiency.
Introduction
Foundation models (Bommasani et al., 2021, FMs) are usually trained on large-scale data and then fine-tuned towards a particular downstream task. This training paradigm has led to significant advances in the realm of language modeling (OpenAI, 2023; Touvron et al., 2023a; Reid et al., 2024), computer vision (Dehghani et al., 2023; Oquab et al., 2023), and reinforcement learning (Brohan al., 2023; Zitkovich et al., 2023). With an increasing number of model parameters, fine-tuning (FT) becomes prohibitively expensive. This results in the need for efficient alternatives to fine-tuning all parameters of the pre-trained model.
Parameter-effichi fine-tuning (PEFT) approaches are commonly used as an efficient alternative to full tne-tuning (FFT). They modify the pre-trained model by introducing a small number of new trainable parameters, while the pre-trained weights remain frozen. A particularly successful approach, LoRA (Hu al., 2022), introduces new weights in the form of low-rank decomposition for each
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Layer- Wise Incremental SVD
Rank Redistribution
3 LoRA
Initialization
Layer N
SVD(X)") 0"
4 V
B 0
Repeat Untti Convergacc
Top-K
Layer 2
SVD(X() "2'22
A V2
y
B² =
Layer 1
-
A = V
B 0
Data Batches
x²
X2
X2
Globally Soated Sugular Valuns
New Roanie
Figure Left: We perform incremental SVD on activation vectors for the first T minibatches. Middle: We globally sort all right-singular vectors according to their explained variance given by their respective normalized singular values and only keep the top-k. Right: We allocate the top-k vectors as initialization for A and continue the standard LoRA fine-tuning procedure.
weight matrix in the pre-trained model. After training, the new weights can be readily merged into the pre-trained weights without any additional inference latency. Recent research has explored various extensions of LoRA, such as different initialization schemes and adaptive rank allocation (see Table 1). Most of them rely on SVD-based approaches on either model weights or gradients. However, these approaches do not optimally maximize the expected gradient signal at the beginning of fine-tuning, still resulting in potentially slow convergence.
We propose Explained Variance Adaptation (EVA), a method designed to provably maximize the expected gradient signal the onset of fine-tuning. This optimal initialization is achieved by performing incremental Singular Value Decomposition (SVD) on activation vectors derived from minibatches of the downstream data. Upon convergence of this procedure, we populate the LoRA vectors. These vectors represent the projection onto the
matrices with the resulting principal components, thereby capturing the directions that preserve activation variance. To ensure this maximization of the expected gradient signal within a fixed rank budget, the right-singular vectors are sorted by their explained variance. This process yields an adaptive rank allocation, computed at the beginning of fine-tuning, which assigns greater complexity (i.e., higher rank) to weights where the variance is distributed across more components.
We demonstrate the benefits of EVA on a variety of downstream tasks, namely language generation and understanding, image classification, and reinforcement learning (RL). EVA consistently improves average performance across a multitude of tasks in each domain compared to LoRA and other recently proposed initialization or rank redistribution methods. In addition, we demonstrate that the additional computational overhead for initialization is negligible and it is mostly invariant with respect to the batch size and order, verifying its robustness. Moreover, EVA exhibits improved convergence compared to other initialization methods, and our rank redistribution reduces the number of trainable parameters, since ranks are usually redistributed from higher-dimension feedforward weights to lower- dimessiimeniooal attention weights. Overall, we demonstrate that EVA is pareto dominant to competitors, as our rank redistribution reduces the number of trainable parameters while usually improving performance. Our contributions are as follows.
We propose EVA, novel data-driven initialization scheme for LoRA that uses incremental SVD on minibatches of activation vectors.
We propose a data-driven heuristic for adaptive rank allocation to provably maximize the expected gradient signal for given rank budget.
We demonstrate pareto-dominanee of EVA compared to other initialization schemes across a variety of different domains.

---

Table 1: Comparison of EVA existing initialization schemes for LoRA. Existing works focus on initialization or adaptive rank allocation. EVA combines data-driven initialization with adaptive rank allocation to enhance convergence and downstream performance.
Method
Initialization
Adaptive ranks
LoRA (Hu et al., 2022)
Random
x
A (Zhang et al., 2023a)
Random
/
PiSSA (Meng et al., 2024)
Weight-driven
X
MiLoRA (Wang et al., 2024a)
Weight-driven
X
OLoRA (Büiyikakyüiz 2024)
Weight-driven
X
LoRA-GA (Wang et al., 2024b)
Data-driven
X
CorDA (Yang et al., 2024)
Data-driven
X
EVA (Ours)
Data-driven
2
Related Work
LoRA (Hu et al., 2022) has sparked widespread interest in leveraging low-rank decompositions for fine-tuning due its simplicity. Following the success of LoRA, several other variants have been proposed (Kopiczko et al., 2024; Zi et al., 2023; Babakniya et al., 2023; Dettmers et al., 2023; Li et al., 2023; Nikdan et al., 2024; Liu et al., 2024a; Zhang al., 2023a; Hayou et al., 2024; Chavan et al., 2023). The variants most similar to EVA are CorDA (Yang et al., 2024) and LoRA-GA (Wang al., 2024b), which are data-driven but do not leverage rank redistribution. Both rely on subsampling training data to estimate either gradients or input-output correlations for initialization. In contrast, EVA provides a variance-optimal initialization that maximizes the expected gradient signal, unified with rank redistribution. Rank redistribution approaches learn gates to switch ranks on/off during fine-tuning (Liu et al., 2024b; Meo al., 2024) or different adapters with different ranks (Valipour et al., 2023). In contrast, our data-driven heuristic allows redistributing ranks prior to fine-tuning,
Initialization of LoRA matrices Common initialization schemes for neural networks (He et al., 2015; Glorot & Bengio, 2010) were designed to stabilize deep neural network training based on activation functions and depth. In the context of PEFT, Hu et al. (2022) and Liu al. (2022) explored data-driven initialization by pre-training on a different task first, or by unsupervised pre-training on the task hand. Similarly, Nikdan et al. (2024) utilize warm-up stage in LoRA fine-tuning, where gradients with respect to LoRA weights are used to initialize a sparse matrix for sparse adaptation (Sung et al., 2021). Alternatively, Babakniya et al. (2023) initialize the LoRA matrices using SVD on the weight matrices obtained after a few steps of full fine-tuning. Weight-driven initializations (Meng al., 2024; Büyükakyüz, 2024) leverage information from the pre-trained weights for initialization. Current data-driven initialization schemes consider either gradients (Wang et al., 2024b) or input- output correlations (Yang et al., 2024); however neither of them yields optimality with respect to the expected gradient signal. Similar initialization schemes to EVA were proposed for training deep networks from scratch (Mishkin & Matas, 2016; Krähenbühl et al., 2016).
Increasing efficiency of LoRA Several works have investigated how to improve the efficiency of LoRA fine-tuning. Kopiczko et al. (2024) decrease the memory complexity by keeping both A and B frozen while only training newly introduced scaling vectors. This way, only random seeds for initializing and B need to stored. Another prominent approach is quantization (Dettmers et al., 2022), which has been successfully combined with LoRA (Dettmers et al., 2023). Building on this, other variants of LoRA are also compatible with quantization (Nikdan al., 2024; Meng al., 2024). Initialization has also been shown to improve the fine-tuning of quantized models (Li et al., 2023).
3 Method
Our aim is to provide data-driven initialization for LoRA weights that aligns the parameter update space with directions that capture the most variance in activations. Hence, for any downstream task using these activations, the update space is biased toward the most informative directions, providing better starting conditions. We first briefly explain LoRA in Section 3.1. Then, we explain the two essential steps conducted in EVA, namely (i), computing variance optimal initialization for the
3

---

LoRA matrices via incremental SVD on activation vectors (Section 3.2) and (ii), adaptive assignment of ranks across layers to maximize the expected gradient signal for given rank budget (Section 3.5).
3.1 Low-Rank Adaptation (LoRA)
LoRA adds new trainable weights that are computed using an outer product of low-rank matrices (Hu al., 2022). This is motivated by the low intrinsic dimensionality of language models (Aghajanyan et al., 2021) and relies on the assumption that the gradients during fine- tine-tuning are also of low rank (Gur-Ari et al., 2018; Zhang al., 2023b; Gauch et al., 2022). Let E Rax be the imput pre-trained weight matrix W E R2xdd Then, LoRA introduces new weight matrices A and B as low-rank decomposition h Wæ + Bxx,, where B R and A Brxd The rank is hyperparameter with y < min(k, d). During fine-tuning, W remains frozen while and B are updated. Usually, B initialized with zeros and A random, so that fine-tuning starts from the pre-trained model. In addition, hyperparameter a is used to scale B BA by 2.
3.2 Variance-optimal initialization of Low-Rank Adaptation
For effective initialization of A that is optimal with respect to propagated activation variance, we utilize incremental SVD (Ross et al., 2008) on minibatches of activation vectors X E R (see Figure 1, left). This process involves collecting activation batches X E [X] ..... XN for N selected pre- pre-trained weight matrices W E [W*, ,..., W Naively, we could simply collect batches of activations and stack them into single matrix and perform SVD. However, this results in excessive memory overhead, as we usually deal with large datasets and models. To reduce memory requirements, we incrementally update V following the approach of Ross et al. (2008), which is based on the sequential Karhunen-Loew algorithm (Levy Lindenbaum, 2000). This process is independent of the dataset size; therefore, the computation of the singular values and their respective vectors is constant in time and memory complexity. For each activation batch X we compute SVD to and their respective singular values of. In practice, we compute
obtain the right-singular vectors : truncated SVD (Halko et al., 201 1), which is significantly faster. After each SVD computation, we update the right-singular vectors and singular values and check whether V has converged by cosine similarity cossim(v" }. 2.1-1 ) 2 t V 1 i r. We illustrate the incremental SVD procedure applied to sequence of data batches in Algorithm 2 and discuss the complexity of this procedure in Appendix F. Finally, by initializing A Vir. we obtain an optimal initialization for A with respect to the activation variance.
Theorem 3.1. Let X E RRbxd be a matrix of activation vectors obtained from a pretrained model, where b is the number of samples and d is the feature dimension. Suppose we wish to adapt weight matrix W E Rxx using a low-rank update of the form AW BA, where B E R E xd and r & min(k, d). Let X = UEV be the singular value decomposition (SVD) of the activation matrix with 01 2 02 2 ... 2 being the singular values of E. Then the top r right singular vectors V,, pddr solve the following optimization problem:
Tr(V ' x
arg
max VeRex* VVVV=
and also minimize the Frobenius norm reconstruction error:
arg
min
X M
Megbxd rank
Hence, forms the optimal basis for capturing the maximum variance of activations under rank-r projection.
The minimization of the reconstruction error under the Frobenius norm in Theorem 3.1 is directly given by the Eckart- Eckart-Young theorem (Eckart & Young, 1936). For details see Appendix H
3.3 Gradient Signal Amplification
We hypothesize that initializing LoRA weights along directions of high variance leads to stronger, more stable gradient signal. To theoretically verify this claim, we consider simple feedforward layer = (W- - BA)x. The gradients w.r.t. A and B in this example are
ac
dc x
dc = B 0A
OC
and
0y v
(I)
0B
dy

---

respectively. Using an explained variance optimal initialization of A ensures that LoRA updates are aligned with high-variance directions in activation space. This leads higher expected gradient magnitude as shown below.
Theorem 3.2. Lei AW = BA be a low-rank adaptation to a pretrained weight matrix W E RRxxd, where B E H A R and < min (k. d). Let æ E R be the activation input to this layer. Assume activations initializing A with the lop right singular vectors of sample activation matrix X E R maximizes the expected squared grudient norm:
drawn from a distribution with covariance matrix Σ -[aaa ). Then
ac
2
E
x Tr(AEAT).
0B
We provide proof for Theorem 3.2 in Appendix H. As result, aligning A with high-variance directions (via SVD), leads to amplification gradient signals and enable more effective low-rank updates. Importantly, this is true for initialization of both B and A. Following (Hayou et al., 2024), we initialize in an explained variance optimal manner and set = 0. as this setup has favorable properties, such as usage of higher learning rates.
3.4 Connection to Neural Tangent Kernel and Generalization Error
To provide further theoretical insights into the effect of explained variance optimal initialization on the generalization error, view the fine-tur tuning problem through the lens of the Neural Tangent Kernel (Jacot et al., 2018, NTK). For a network with parameters 6 the NTK is defined as
K K(x, x') Vafo(æ) Vofe(w'),
(2)
and its expectation over the fine-tuning data can be written as
E Vo.o(a)) |
E (d(x)) & (())))((x)) @ h[
(3)
where 6(x: denotes upstream gradients and h(x)) represents input activations of a weight matrix, and & denotes the Kronecker product. Under the simplifying assumptions that (i) h(x)) and 8(x) are
weakly correlated, and (ii) the covariance of the upstream gradients 0(x) is approximately isotropic, the NTK simplifies to
[eea(x)
vo o3 [M<*)((<)
Since we only consider the onset of ainettunig, i.c. the time of initialization, these assumptions reasonable. Therefore, EVA effectively approximates the empirical NTK's s leading subspace. Directions associated with larger NTK eigenvalues X are learned faster and generalize more ro- bustly, whereas directions with smaller X; dominate the residual error and thus the kernel-regression generalization bound (Arora et al., 2019; Lee et al., 2019)
(uT y)
Egen X
X
where {u: } denotes the NTK spectrum and represents the labels. Consequently, EVA approxi- mately initializes the LoRA adapters along the dominant principal directions of the NTK, thereby minimizing the spectral tail of the NTK-based generalization error.
3.5 Adaptive Rank Allocation
The singular values provide an estimate of the amount of variance that each component in Vi explains. Leveraging this insight, we redistribute ranks across weight matrices of the pre-trained model such that the maximum amount of variance explained for a given rank budget = Nr. To achieve this, we sort right singular vectors obtained for all weight matrices according to their explained variance ratio (see Figure/ 1, middle)
53 = (M oz
(6)
5

---

where || denotes the l norm, o' is vector containing all singular values, and M is the total number of samples used for the incremental SVD. Note that & is normalized for each weight matrix to ensure comparable ranges. Then, we take the top-l entries of the globally sorted singular vectors and set the rank of each pre-trained weight based on how many of its singular vectors are contained in this selection (see Figure 1, right).
Additionally, we introduce hyperparameter E [1, oo) that controls the uniformity of the rank distribution. p determines the number of ranks that we compute during SVD and increas- ing p allows for an increasingly heterogeneous rank distribution. Moreover, p controls the max- imum number of ranks that a weight matrix can receive. For each W we compute the top rp components via incremental truncated SVD, re- sulting in Nrp components in total. For redistri- bution, we then only use the top-/ components, according to their explained variance ratio ç This ensures that the number of total ranks used is the same for EVA compared to other LoRA variants. Setting = 1, results in uniform rank distribution as LoRA, but initialized according EVA. Therefore, P provides us with the means to change the rank distribution in a controlled manner prior to fine-tuning at the initialization stage. In practice, we found that the redistribution converges for values of p > 2 (see Appendix G). Finally, set B = 0 and perform standard LoRA fine-tuning. In Algorithm we provide pseudocode for EVA.
Algorithm 1 Fine-tuning via EVA
Input: FM ()()). p. rank r, dataset D
1:
while not all_ convereed eb) do
2:
X =((nex((D)
get activations
3: Vnew: heww&&= -ncremental-SVYDDX ,rr)
4:
if isclose(
(Vole Vol: Vnew) then
5:
wrap- haap_and_iniee mand_initialize(W. View)
6: end if'
7: Vota + Vnew
8: end while
9: redistribute_ $. Vnew)
lora_finetune(tt X)
Experiments
First, we elaborate on implementation details of EVA in Section 4.1. Then, we show results for fine-tuning large language models (LLMs) on math and reasoning tasks in Section 4.2 and language understanding tasks in Section 4.3. In addition, we show results for image classification in Section 4.4 and decision-making tasks in Section 4.5, Finally, in Section 4.6 we demonstrate that the computa- tional overhead induced by EVA on LoRA is negligible and that incremental SVD converges and is invariant to batch order and batch size.
4.1 Implementation Details
We follow the standard LoRA training procedure from Hu et al. (2022). Similarly to Kalajdzievski (2023), we found that LoRA training is very sensitive the scaling parameter a. Therefore, we set = for all our experiments as we found this to be the most stable setting. For EVA with > 1 we set a Ynew to preserve the scaling factor for different ranks. Following Zhang et al. (2023a), we Told
apply EVA adapters all pre-trained weight matrices except for the embedding layer. All models we used for fine-tuning are publicly available on the huggingface hub (Wolf et al., 2020). For the implementation of baselines, we utilize the widely used PEFT library (Mangrulkar et al., 2022). Across experiments, we highlight the highest scores in boldface and underline the second-highest.
4.2 Language Generation
We fine-tune five different LLMs, namely Llama-2-7B (Touvron et al., 2023b), Llama-3.1-81 (Dubey et al., 2024), Llama-3.1-70B. Gemma-2-9B (Rivière et al., 2024), and Gemma-2-278 on common sense reasoning benchmarks. We follow Liu al. (2024a) and amalgamate training set consisting of BoolQ (Christopher et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019). HellaSwag (Zellers al., 2019), Winogrande (Sakaguchi et al., 2020), ARC-e and ARC-c (Clark et al., 2018) and OpenBookQA (Mihaylov et al., 2018). 8). We apply all the methods listed in Table 1 to all five models, except LoRA-GA and CorDA, which we do not apply to 1.lama33.1-700 and Gemma-2-278 as it requires an excessive amount of computation for initialization (see Figure 5, left). We train all methods with rank 16 and a learning rate of 5e for three random seeds. For Llama-3. 3-0B.. we leverage gradient checkpointing and the ZeRO optimizer (Rajbhandari et al., 2020) for optimizer state and gradient offloading. More details on the finettniig settings can be found in Appendix B.

---

Common sense
MATH
94
40
35
90
Performance
EVA
LaRA
Performance
30
OLoRA
EVA
PISSA
86
25
OLoRA
LoRA: GA
PSSA
84
20
AdaLoRA
MiloRA
LoRA-GA
Liama-2
CorDA
Liama: 88
MİLoRA
80
Gemma
G
Liama 3.1 BB
Llama-3.1-700
10
10
10
10a
2x
38M
43M
48M
53M
58M
# Trainable parameters
Trainable par parameters
Figure 2: Performance of all methods on eight common sense reasoning tasks (left) and MATH after being finetuned MetaMathQA (right). EVA reduces the number of trainable parameters while reaching performance on- on-par or better.
We present average performance for all eight common sense reasoning tasks in Figure Across models, we found that EVA yields the highest performance while also significantly re- ducing the number of trainable parameters com- pared to other LoRA-based methods (see Table 11 in Appendix B), resulting in an im- proved pareto front. For example, EVA ap- plied to Llama-3.1-700 achieves the highest av-
GSMBK
85
75
Performance
70
LaRA
erage score (94.5) while reducing the number of trainable parameters by report the performance per task in Table 7 in Ap- pendix B and also add comparison DoRA (Liu et al., 2024a) and EVA+DoRA, which com- bines EVA with DoRA. Although there is flue- tuation on a per-task basis, EVA-based meth- ods consistently attain the highest average score across all tasks. Moreover. we conduct exper- iments where we add rank stabilization (Kala- jdzievski, 2023), different learning rates for A and B, and different values for a in Table in Appendix B. EVA consistently yields the best results in these settings compared to LoRA. To demonstrate that EVA initialization starts closer its final solution, we report the distance of EVA to the adapter weights after training compared to the distance of LoRA to the adapter weights after training for different weight matrices of the model in Table 6 (right). The CorDA baseline exhibited high seed sensitivity, prompting us to conduct a light hyperaarameter search over the number of initialization examples in consultation with the CorDA authors. Despite these efforts, training performance collapsed for certain seeds, as evidenced in our results. Additionally, we provide results for leveraging the components that explain the least amount of variance Table 12, which results in worse performance compared to EVA, and additional results for training with varying number of ranks for Llama-2-7B in Table 9, We find that across ranks and hyperparameters, EVA is consistently among the best performing methods.
than 1 15M. We
PISSA
LaRA-GA
MiLoRA
60
Liama-2-78
Lisma-3.1-88
Gemme-22 2
# Trainable parameters
Figure Performance of all methods for fine-tuning Llama-2-7B, Llama-3.11885 and Gemma-2-9B GSM8K after fine-tuning on the MetaMathQA dataset.
For math fine-tuning experiments, we fine-tune Llama-2-7B. L.lma-3.11- 1-8,, and Gemma-2-9R on the MetaMathQA dataset (Yu al., 2024) for one epoch with the same hyperparameters as for common sense reasoning tasks and evaluate them MATH (Hendrycks et al., 2021) (see Figure 2, right) and
7

---

EVA
CorDA
PISSA
Random
0.28
0.39
0.7
0.26
0.10
0.37
0.24
Taaiiing Loss
0.16
0.5
0.15
0.4
6K
8K
10K
0.18
0.2
0.16
0.14
1K
3K
5K
7K
9K
11K
Update Steps
Figure 4: Gradient norm (left) and training loss (right) for fine-tuning Llama-3, 1-BB on the Meta- MathQA dataset. We compare EVA to other initialization methods and random initialization (LoRA). We show mean and standard deviation across three random seeds.
GSM8K (Cobbe et al., 2021) (see Figure 3). We also report the performance of each method on each model and task, again including DoRA and EVA+DoRA, Table in Appendix B. EVA is pareto- dominant compared to all competitors on both datasets it trains fewer parameters while resulting in on-par or improved performance. For example, EVA achieves the highest performance on the GSM8K dataset for Cemma-2-9B. while performance is on-par for Llama-2-78 and Llama-3.1 In Figure 4 we show gradient norm and training loss for 1.ama-331-4 1-8B on the MetaMathQA dataset. We observe that EVA converges faster than competitors and exhibits the largest gradient norm. We provide additional loss curves in Figure 6. Furthermore, we provide a comprehensive overview on the effect of rank redistribution results indicate that the performance of adaptive rank allocation depends on a combination of the selected model and the downstream task. We further analyze the resulting rank distributions for different values of for Llama-2-7B and their effect on downstream performance in Appendix G. Finally, we provide additional results for Llama-2-7B code fine-tuning tasks in Appendix B.
different model types for both downstream tasks in Table 11. Our
4.3 Language Understanding
We train RoBERTa arge (Liu et al., 2019) and DeBERTav3R.c (He et al., 2023) on the GLUE bench- mark (Wang al., 2019). The GLUE benchmark comprises eight downstream tasks, such natural language inference, or sentiment analysis. In addition to learning rate, we also search for different ranks within maximal rank budget (r < 16). For further details datasets, implementation, hyperparameters, see Appendix C. We also add FFT as a baseline and report Matthew's 's correlation for CoLA, Pearson's correlation for STS-B, and accuracy for the remaining tasks in Table 2. EVA achieves the highest average score in all tasks for both RoBERTaL and DeBERTav3 estingly, DoRA usually only slightly improves over LoRA on low resource tasks (RTE, MRPC), while performing worse on high resource tasks (MNLI, QNLI, QQP. SST2). We also compare LoRA with EVA in Table in Appendix for different rank budgets, where EVA consistently improves over LoRA. We visualize the resulting rank distribution patterns for different GI.UE tasks Appendix C. More ranks are assigned to higher layers of the query, key, and value projections in self-attention, whereas the remaining weights often receive less ranks. This is consistent pattern for both DeBERTav3 and RoBERTaLarg and in line with the reduced number of trainable parameters for larger models.
Inter-
4.4 Image Classification
We evaluate EVA on the VTAB-IK (Zhai et al., 2019) benchmark, which comprises 19 image classification tasks that are divided into natural images, specialized images (medical images and remote sensing), and structured images (e.g. object counting). We fine-tune a DINOv2-g/1 model (Oquab et al., 2023) that consists of around 1.1B parameters. For implementation details and hyperparameters see Appendix D. Our results are shown in Table 20 and we additionally report error bars in Table 21. EVA attains the best average accuracy across all tasks. Interestingly, EVA mainly

---

Table 2: Comparison of all methods for RoBERTaLage (top) and DeBERTav (bottom) GLUE tasks. We report mean and standard deviation of Matthew's 's correlation for CoLA, Pearson correlation for STS-B, matched accuracy for MNLI, and accuracy for remaining tasks. For CoLA, RTE, MRPC, and STS-B we average over five seeds and for the remaining tasks over three seeds.
Method
MNLI
QNLI
QQP
SST2
CoLA
MRPC
RTE
STS-B
Avg
FFT
90.2
9-.77
92.2
96.4
68.0
90.9
86.6
92.4
88.9
LoRA
90.7
94.8
92.0+.0
96.21.3
69.1
91.1+ 6
88,1+1.1
92.3+.1
89.3
AdaLoRA
90.5
94.81.2
90.61.1
96.11.2
68.21.7
9U.7
84.41.9
91.82.1
88.4
PISSA
90.1
94.7+.0
91.0+.o
96.
68.7+1.3
90.44.6
87.6
92.5
88.9
OLoRA
90.9
95.0=.11
92.0+.2
96.3
69. 39.0+1.5
91.0=110
87.9+1.2
92.4=.1
89.3
LoRA-GA
90.8
94.9
92.0+.0
96.3
68.4+1.9
91.0+.2
87.0
92.3
89.1
CorDA
89.3
92.6+.0
89.7+.0
95.5..0
67.8+1.0
90.1
86.5..8
91.8..2
87.9
EVA
90.8
95.0+.2
92.1
2
69.5
91.44.
88.8+1.2
92.6+.1
89.6
DoRA
89.5+.1
91.6+.1
89.9+.1
96. +..1
69.3
91.0+.6
88.441.2
92.4
88.9
FFT
90.1
94.0
92.4
95.6
69.2
89.5
83.8
91.6
88.3
LoRA
90.5
94.3
92.4,.
95.2..3
72.0
91.41.7
$8.9
91.7
89.6
AdaLoRA
90.8
94.6
92.2
96.1
71.5
90.7
88.1
91.8
89.5
PISSA
90.1
94.1+.1
91.84.1
95.8
72.7
90.9
86.5+1.2
91.6..2
89.2
OLoRA
90.5+.1
94.4+.1
92.64.1
96.24.2
72.0+
91.6+.7
89.1
92.0+.2
89.8
LoRA-GA
89.8+.7
94.6..1
92.2+.0
95.6
72.24.9
90.84.9
86.641.1
90.5
89.0
CorDA
90. 00.0+.1
93.8
91.1+.1
95.
71.8+1.2
89.6+.5
83.9
91.1+.2
88.3
EVA
90.6
94.41.1
92.4..0
96.2+t.2
72.5
91.84.6
89.44.7
92.0
89.9
DoRA
89.0
94.1
88.0
94.6a.4
70.3
91.9+.6
87.8
91.8
88.4
improves over competitors on natural tasks, i.e., in-domain datasets. On out-of-distribution datasets, we find that FFT still performs better than most PEFT approaches.
4.5 Decision Making
We follow the single task fine-tuning experiments in Schmied et al. (2024) and fine-tune Decision Transformer (Chen al., 2021a, DT) on the Meta-World benchmark suite (Yu et al., 2020). Meta- World consists of a diverse set of 50 tasks for robotic manipulation, such as grasping, or pushing buttons. We split Meta-World according to Wolezyk et al. (2021) into 40 pre-training tasks (MT40) and 10 fine-tuning tasks (CW10). We pre-train 12 M parameter DT on MT40 and fine-tune it on the CWIO holdout tasks. We report success rates and standard errors for each CW10 task in Table 23. We observe that EVA significantly reduces that gap between LoRA and FFT. Furthermore, combining EVA with DoRA improves upon DoRA and attains the best average performance across all tasks. We report results for different rank budgets in Table 24, as well as implementation details and hyperparameters Appendix E.
4.6 Efficiency and convergence
We compare the computational overhead of EVA to competitors. In Figure (left) we show the wall clock time as a fraction of the training time as well as memory requirements. For CorDA we use sample size of 2560 as recommended by Yang et al. (2024). We observe that EVA with batch size of 16 requires only 0.7% of the training time for initialization, which is the fastest for data-driven initializations. In Figure (right) we provide additional evidence that reducing the batch size for EVA results in the same initialization. Therefore, by reducing the batch size, we can reduce the overhead induced by EVA to 0.2%, resulting in one of the most efficient initializations. Furthermore, we also provide evidence that the incremental SVD invariant batch order and consistently converges for different batch sizes in Figure 12 (left), and Figure 11 in Appendix F., respectively.
5 Discussion and Limitations
Low rank setup. Due to computational constraints we mainly chose lower values for the LoRA rank (r 16). Other initialization schemes (Yang et al., 2024; Meng et al., 2024; Wang et al., 2024b) usually rely on higher ranks (r 2 128). However, for such values, EVA suffers from the computational overhead of repeated SVD computations, In Table we provide results for fine-tuning

---

Average cos sinilaryy Lccoos batch sibee 25 33)
Memory (GB)
% of Training
Initialization Method
Weight-driven OLoRA
PISSA
70
0.1
LoRA-GABan
$6.95
2.4
CorDAB=1
55.64
Data-driven
FVA
32.85
0.7
EVA
29.39
0.3
EVAm-4
27.51
0.2
30 24 2622
Figure 5: Left: Percentage of training time required for computing data-driven initializations for Llama-2-7B on a single A100 GPU on the common sense reasoning tasks. We report the maximum batch size and track peak memory usage. Right: Average cosine similarity between components after incremental SVD for different batch sizes. The components strongly correlate indicating that the SVD computation is mostly invariant to the batch size.
with ranks up to - 64, which show that EVA works particularly well on lower-rank setups. This is an advantage for compute-constrained fine-tuning of larger models which is usually only feasible with smaller ranks. Finally, EVA assumes access to a fixed static downstream dataset which may not always be available.
Effect of rank redistribution. We provide additional results for the effect of rank redistribution in Appendix I. The results show that rank redistribution in combination with explained variance optimal initialization principle performs best. We observed cases where there is no improvement in performance for rank redistribution (e.g. decision making). However, since rank redistribution in all our experiments decreased the number trainable parameters, we recommend using it by default.
What method performs well in which tasks? We conducted fine-tuning experiments for 51 tasks and four domains and found that EVA usually performs best on average across multiple tasks per domain. Despite this, there is usually variation in the ranking of methods for single tasks, i.e. LoRA performed better on specialized images and FFT performed best on structured images. Therefore, there is one algorithm that performs the best every task, verifying that there is no free lunch (Wolpert & Macready, 1997).
Reproducibility. We provide the source code along with the submission (see Appendix A) to ensure reproducibility. In addition, we added support for EVA in the widely used PEFT library (Mangrulkar et al., 2022) to make it more accessible.
Conclusion and Broader Impact
We propose novel method named Explained Variance Adaptation (EVA), extending the widely used LoRA with explained variance optimal initialization and rank redistribution provably maximize the expected gradient signal. EVA performs incremental SVD on minibatches of activation vectors and redistributes ranks across weight matrices according to the amount of variance that they explain. We demonstrate performance gains of EVA over LoRA and initialization schemes thereof in variety of domains, ranging from language to vision and RL. Moreover, EVA more efficient than most existing initialization methods while reducing the number of trainable parameters. Our results demonstrate that EVA consistently achieves the highest average performance on a wide range of tasks across a variety of domains.
We believe that EVA can have a significant impact on future research on fine-tuning foundation models because it inherits all the benefits of LoRA while improving performance and reducing the number of trainable parameters at no significant additional cost. the future, our aim is additionally incorporate gradient information and exploring ways to enhance interpretability by relating different singular vectors to different forms of pre-trained knowledge. Another fruitful avenue would be combining EVA with mixture-of- training enable more efficient fine-tuning.
10

---

Acknowledgments and Disclosure of Funding
We acknowledge EuroHPC Joint Undertaking for awarding us access to Vega IZUM, Slovenia, Karolina at IT4Innovations, Czech Republic, MeluXina at LuxProvide, Luxembourg, Leonardo BSC, Spain. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects FWF AIRI FG 9-N (10.55776/FG9), AL4GreenHeeatingGinnn FWF Bilateral Artificial Intelligence (10.5577//C0E12),
at CINECA, Italy, MareNostrums
(FFG- 899943), Waters
HIORIZON-CL6-200 6--022--CLIMATE---- We thank NXAI GmbH, Audi AG, Silicon Austria Labs (SAL), Merck Healthcare KGaA, GLS (Univ. Waterloo), TÜV Holding GmbH, Software Competence Center Hagenberg GmbH, dSPACE GmbH, TRUMPF SE + Co. KG. Fabian Paischer acknowledges travel support from ELISE (GA no 951847).
References
Aghajanyan, A., Gupta, S., and Zettlemoyer, L. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Zong. C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 1th International Joint Conference Natural Language Processing, ACL/JCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 7319-7328. Association for Computational Linguistics, 2021. doi: 10.18653/v 1a202aallllooneeeeeee law.-long..66...
Arora, S., Du, S. S., Hu, W., Li, Z.., Salakhutdinov, R., and Wang, R. On exact computation with an infinitely wide neural net. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buc Buc, F, Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8139-8145 2019.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E.,, Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. 2021.
Babakniya, S., Elkordy, A. R., Ezzeldin, Y. H., Liu, Q. Song. K., El-Khamy, M., and Avestimehr, Slora: Federated parameter efficient fine-tuning of language models. CoRR, abs/2308.06522, 2023.
doi: (0.46550/ARXXI12308..66
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., Schritwieser, J., Anderson, K., York, S., Cant, M., Cain, A.,
Bolton, A., Gaffney, S., King, H., Hassabis, D., Legg, S., and Petersen, S. Deepmind lab. CoRR, abs/16 12003801, 2016
Bisk, Y. Zellers, R., Bras, R. I.,, Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In "hirtyyFourtds AAAI Conference on Artificial Intelligence, 2020.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R. B., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel. K., Davis, J. Q., Demszky, D., Donahue. C., Doumbouya, M.. Durmus, E., Ermon, S., Etchemendy, J.. Ethayarajh. K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N. D., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M. S., Krishna, R., Kuditipudi, R., and al. On the opportunities and risks of foundation models. CoRR, abs/2108.077
2021.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., Leal, I., Lee, K., Levine, S., Lu, Y., Malla, U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C., Peralta, J., Perez, E., Pertsch, K., Quiambao, J., Rao, K., Ryoo, M. S., Salazar, G., Sanketi, P. R., Sayed, K., Singh, J., Sontakke, S., Stone, A., Tan, C., Tran, H. T., Vanhoucke, V., Vega, S., Vuong, Q., Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, B. RT-1: robotics transformer for real-world control at scale. In Bekris, K. E.,, Hauser, K., Herbert, S. L., and Yu, J. (eds.). Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10- 14, 2023, 2023. doi: 10. 15550///....2...00.... XIX.025.

---

Büyükakyüüz, K. Olora: Orthonormal low-rank adaptation of large language models. CoRR, abs/2406.01175, 2024. doi: 10.48550ARRXVV2206..07777
Chan, T. F., Golub, G. H., and LeVeque, R. J. Algorithms for computing the sample variance: Analysis and recommendations. The American Statistician, 37(3):242-2477 1983. ISSN 00031305, 15372731.
Chavan, A., Liu, Z., Gupta, D. K., Xing, E. P., and Shen, Z. One-for-all: Generalized lora for parameter-effiient fine-tuning. CoRR, abs/2306.07967, 2023. doi: 10.48500/RXX..23660009966
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084- 2021a.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph. N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P. Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L.,, Bavarian, M., Winter, C., Tillet, P., Such, F. P.. Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N.. Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Car,, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained code, 2021b.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE, 105510):885-1883,, 2017. doi: 10.1100//ROC..20722255553
Christopher, C., Kenton, L., Ming- Wei, C., Tom, K., Michael, C., and Kristina, T. Boolq: Exploring the surprising difficulty of natural yes/no questions. NAACL, 2019.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, pp. 3606-3613. IEEE Computer Society, 2014. doi: 10. 1..1109/CVP..2014.44
Clark, K., Luong, M.. Le, Q. V., and Manning, C. D. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 1:Xiv:1803.054955 2018.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jum, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021.
Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron,
M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L., Tschannen, M., Arnab, A., Wang. X., Ruiz, C. R., Minderer, M., Puigcerver, J., Evci, U., Kumar, M., van Steenkiste, S., Elsayed, G. F. Mahendran, A., Yu, F., Oliver, A., Huot, F., Bastings, J., Collier, M., Gritsenko, A. A., Birodkar, V., Vasconcelos, C. N., Tay, Y., Mensink, T., Kolesnikov, A., Pavetic, F., Tran, D., Kipf, T., Lucic, M., Zhai, X., Keysers, D., Harmsen, J., and Houlsby, N. Scaling vision transformers to 22 billion parameters. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 7480-7512 PMLR, 2023.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int8() 8-bit matrix multiplication for transformers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances Neural Information Processing Systems, volume 35, pp. 00338-30332. Curran Associates, Inc., 2022.
12

---

Dettmers. T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized Ilms. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA,, USA, December 10 16, 2023, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A.,
Yang. A., Fan, A., Goyal, A., Hartshorn, A.. Yang, A., Mitra, A., Sravankumar, A., Korenev, A..
Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A.. Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak. C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong. C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song. D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hupkes, D.. Lakomkin, E., AlBadawy, E.,, Lobanova, E., Dinan, E.,, Smith, E. M., Radenovic, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang. G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I., Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong. J., Lee Fu, J., Chi, J., Huang. J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J.. Johnstun, J., Saxe. J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Hcafield, K., Stone, K., and et al. The llama 3 herd of models. CoRR, abs/2447.22178. 2024. doi: 10-48550/ARXX1.22077277777
Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika, 133):211-218, 1936. doi: 10. N0.1007/BF02288367.
Fei-Fei, L.., Fergus, R., and Perona, P. One-shot learning of object categories. IEEE Trans. Patlern Anal. Mach. Intell., 28(4)5594-611, 2006. doi: 10.1 10.1109/TPMM.2206..779
Fisher, R. A. The use of multiple measurements taxonomic problems. Annals Eugenics, 7:179-188, 1936.
Gao, L.. Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac"h. h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L.., Tang, F., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024.
Gauch, M., Beck, M., Adler, T., Kotsur, D., Fiel, S., Eghbal-zadeh, H., Brandstetter, J., Kofler, J., Holzleitner, M., Zellinger, W., Klotz, D., Hochreiter, S., and Lehner, S. Few-shot learning by dimensionality reduction in gradient space. In Chandar, S., Pascanu, R., and Precup. D. (eds.), Conference on Lifelong Learning Agents, CoLLAs 2022, 22-24 August 2022, McGill University, Montréal, Québec, Canada, volume 199 of Proceedings of Machine Learning Research, pp. 1043-1064 PMLR, 2022.
Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. Vision meets robotics: The KITTI dataset. Int. J. Robotics Res., 1):11113311133, 2013. doi: 10. 10.1177027736449134999977
Glorot, X. and Bengio. Y. Understanding the difficulty of training deep feedforward neural networks. In Tch, Y. W. and Titterington, D. M. (eds.), Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pp. 249-256. JMLR.org, 2010.
Gur-Ari, G., Roberts, D. A., and Dyer, E. Gradient descent happens in tiny subspace. CoRR, 1bs/1812.04454, 2018.
Halko, N., Martinsson, P., and Tropp, J. A. Finding structure with randomness: Probabilistic
algorithms for constructing approximate matrix decompositions. SIAM Rev, 53(2):217-288, 2011. doi: 10.1137/090771866
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank adaptation of large models, 2024.
He, K., Zhang, X., Ren, S., and Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 1026-1034, IEEE Computer Society, 2015. doi: 10.1109/CC22005.1122.
13

---

He, P., Gao. J., and Chen, W. Debertav3: Improving deberta using electra-style pre- vre-trinnig with gradient- disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 12(7):2217-22) 22262226, 2019. doi: 10.1 (0.1109/SSTARS.000 201.......01.2..... 2921422...
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Vanschoren, J. and Yeung, S. (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.-P., Bing, L.,, Xu, X., Poria, S., and Lee, R. LLM- adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 2276, Singapore. December 2023. Association for Computational Linguistics. doi: 10.18653/v http:/2222..mmmmmmmmm eenlp- smnlp-mainn311.
Jacot, A., Hongler, C., and Gabriel, F. Neural tangent kernel: Convergence and generalization in neural networks. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018. Montréal, Canada, pp. $580-8589 2018.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Feii L.., Zitnick, C. L., and Girshick, R. B. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1988-1997. IEEE Computer Society, 2017. doi: 20.1109//V....011.2155.
Kaggle and EyePacs. Kaggle diabetic retinopathy detection, July 2015.
Kalajdzievski, D. A rank stabilization scaling factor for fine-tuning with lora. CoRR, abs/2312.03732. 2023. doi: 10448550/ARXI..3122.055
Kopiczko, J., Blankevoort, T., and Asano, Y. M. ELORA: Efficient low-rank adaptation with random matrices. In The Twelfth International Conference on Learning Representations, 2024.
Krähenbühh, P., Doersch, C., Donahue, J., and Darrell, T. Data-dependent initializations of convo- lutional neural networks. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Krizhevsky, A. Learning multiple layers of features from tiny images. CoRR, pp. 32-33, 2009.
LeCun, Y., Huang, F. J., and Bottou, L. Learning methods for generic object recognition with invariance to pose and lighting. 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2004), with CD-ROM, 27 June 2 July 2004, Washington, DC, USA, pp. 97-104. IEEE Computer Society, 2004. doi: 10.1109/CVPR.22044.1 144.
Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve linear models under gradient descent. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché-Buc, F., Fox, F. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8570-8581, 2019.
Levy, A. and Lindenbaum, M. Sequential karhunen-loeve basis extraction and its application to images. IEEE Trans. Image Process., 98): 2000. doi:
14

---

Li Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W. and Zhao, T. Loftq: Lora-fine seera-fin--uee tee---neettunnng tuning- aware quantization for large language models. CoRR, abs/2310.08659, 2023. doi: 10.48550/ ARXIV.2310.0866
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. Few-shot parameter- efficient fine-tuning is better and cheaper than in-context learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022,
New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Liu, J., Xia, C. S., Wang. Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Liu, S., Wang, C., Yin, H., Molchanov, P., Wang. Y. F., Cheng, K., and Chen, M. Dora: Weight- decomposed low-rank adaptation. CoRR, abs/2402.09353, 2024a. doi: 10.48350/ARXIV:2402 09353.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. 2019.
Liu, Z., Lyn, J., Zhu, W., Tian, X., and Graham, Y. Alora: Allocating low-rank adaptation for fine- tuning large language models. In Duh, K., Gómez-Adorno, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lunguage Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp.
Association for Computational Linguistics, 2024b. doi:
Loshchilov, and Hutter, F. Fixing weight decay regularization adam. CoRR, abs/ 111.0500,,, 2017.
Mangrulkar, S., Gugger, S., Debut, L., Belkada, Y., Paul, S., and Bossan, B. Peft: State-of-the-art dataset. parameter-efffcient fine-tuming methods, 2022.
Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A. dsprites: Disentanglement testing sprites 2017.
Meng, F., Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.
Meo, C., Sycheva, K., Goyal, A., and Dauwels, J. Bayesian-lora: Lora based parameter efficient fine-tuning using optimal quantization levels and rank values trough differentiable bayesian gates. CoRR, abs/2406. 13046, 2024. doi: 0048550/ARXXV..406.. 13046.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed precision training. arXiv preprint arXiv: 1710.03740, 2017.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018
Mishkin, D. and Matas, J. All you need a good init. In Bengio, Y. and LeCun, (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
Nikdan, M., Tabesh, S., and Alistarh, D. Rosa: Accurate parameter-efficient fine-tuning via robust adaptation. CoRR, 0///2401.04679 2024. doi: 10.48500AARX1..2000.4477
15

---

Nilsback, M. and Zisserman, A. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics Image Processing, ICVGIP 2008, Blubaneswar, India, 16-19 December 2008, pp. 722-729. IEFF Computer Society, 2008. doi:
10.1009/CVGPPP22088...
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023, doi: 10445500ARXXV.22300.07777
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M. G., Sharma, V., Synnaeve, G., Xu, H., Jégou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision. CoRR, abs/2304.07193, 2023. doi: 10.48550AARXIV.3300000
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pp. 3498-3505. IEEE Computer Society, 2012. doi: 10. 20.11090/22R.222222228800
Paszke, A., Gross, S.. Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga L., Desmaison, A., Köpf, A., Yang. E. Z., Vito Z., Raison, M., Tejani, A., Chil- amkurthy, S., Steiner, B., Fang. L., Bai, J., and Chintala, S. Pytorch: An imperative style, tigh-performance deep learning library. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' d'Alché-Buuc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024-8035, 2019.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. CoRR, 2019.
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: memory optimizations toward training trillion parameter models. In Cuicchi, C., Qualters, I,, and Kramer, W. T. (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp. 20. IEEE/ACM, 2020. doi: (00///CC14452202..0000
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap. T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sottiaux, T., Lee, B., Viola, F., Reynolds. M., Xu, Y., Molloy, J., Chen, J., Isard, M., Barham, P., Hennigan, T., Mcllroy, R., Johnson, M., Schalkwyk, J., Collins, E.. Rutherford, E., Moreira, E., Ayoub, K., Goel, M. Meyer, C., Thortoo,, G., Yang, Z., Michalewski, H., Abbas, Z., Schucher, N., Anand, A., Ives, R., Keeling, J., Lenc, K., Haykal, S., Shakeri, S., Shyam, P., Chowdhery, A., Ring, R., Spencer, S., Sezener, E.,, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.0555 2024. doi: 0044850/ARXXIV.2400055522
Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju. S., Hussenot, L.,, Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Fricsen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N.. Hoffman, M., Thakoor, S., Grill, J., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Choquette-Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozinska, D., Herbison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F, Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimezak-Plucines
H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi J., Becker, J., Fernandez, J., van Amersfoort, J.. Gordon, J., Lipschultz, J.. Newlan. J., Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjösund. L. L., Usui, L.. Sifre, L., Heuermann, L., Lago, L., and McNealus. L. Gemma 2: Improving open language models at a practical size. CoRR, abs/2408.001 18, 2024. doi: 10. 00.4850//RRXV.24088. 18.
Ross, D. A., Lim, J., Lin, R., and Yang, M. Incremental learning for robust visual tracking. Int. Comput. Vis., (7(1-3):125-1444 2008. doi: 10.1007/S11264 12007//5112---77707755-7
16

---

Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 0732-8740.. AAAI Press, 2020. doi: 10.1609/AAAI V34105.6399.
Sap. M., Rashkin, H., Chen, D., Bras, R. I.., and Choi, Y. Socialiqa: Commonsense reasoning about social interactions. CoRR, abs//994.0977/ 2019.
Schmied, T., Hofmarcher, M., Paischer, F., Pascanu, R., and Hochreiter, S. Learning to modulate pre-trained models in rl. Advances in Neural Information Processing Systems, 36, 2024.
Schölkopf, B., Smola, A., and Müller, K.-R. Kernel principal component analysis. In Gerstner, W., Germond, A., Hasler, M., and Nicoud, J.-D. (eds.), Artificial Neural Networks - ICANN'97, pp. 583-588, Berlin, Heidelberg, 1997. Springer Berlin Heidelberg. ISBN 578-3-540-69620--
Sung. Y., Nair, V., and Raffel, C. Training neural networks with fixed sparse masks. In Ranzato, M.. Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 24193-24205, 2021.
Todorov, E., Erez, T., and Tassa, Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ international conference on intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023a. doi: 10.48550/ARXIV.2302. 13971.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L.,, Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A.. Koura, P. S., Lachaux. M.. Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P.. Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E.,, Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang. Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models.
CoRR, abs/2307.009288 2023b. doi: 10.48550/A4 20..50//RXXV.2300 2.88009988...
Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. Vlachos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Crvatia, May 2-6, 2023, pp. 3266-3279. Association for Computational Linguistics, 2023. doi: 10.18653/V
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant cnns for digital pathology. In Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola-López, C., and Fichtinger. G. (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2018 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II, volume 11071 of Lecture Notes in Computer Science, pp. 210-218. Springer, 2018. doi: 00009997---000-00944-2.0
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task bench- mark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
Wang, H., Xiao, Z., Li, Y., Wang, S., Chen, G., and Chen, Y. Milora: Harnessing minor singular components for parameter-efficient LLM finetuning. CoRR, abs/2406.09044, 2024a. doi: 10. 21....X02.24000004...
17

---

Wang. S., Yu, L.. and J. LoRA-GA: Low-rank adaptation with gradient approximation. In The Thihty-iight Annual Conference on Neural Information Processing Systems, 2024b.
Wolezyk, M., Zajac, M., Pascanu, R., Kuciñski, E., and Milos, P. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 34:28496 20096062881.... 2021.
Wolczyk, M., Zajkac, M., Pascanu, R., Kuciñski, L., and Milos, Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 34:28496-285 10, 2021.
Wolf, T., Debut, I.., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T.,, Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C.. Jermite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020, Association for Computational Linguistics. doi: 10.18653/v1/2020.ccmm wwtp::00.mmmmmmmmmmee
Wolpert, D. and Macready, W. No free lunch theorems for optimization, IEEE Transactions on Evolutionary Computation, 1(1):67-82, 1997. doi: 011110044335.555885
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. SUN database: Large-scale scene recognition from abbey to zoo. In The Twenty-Thing IEEE Conference on Computer Vision and
Putlern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pp. 3485-3492. IEEE Computer Society, 2010. doi: 10.1109/CVPR.22000555977
Yang. Y., X., Zhou, Z., Song, S. L.., Wu, J., Nie, L., and Ghanem, B. CorDA: Context-orient decomposition adaptation of large language models for task-aware parameter-efficien fine-tuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Yu, L.,, Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OvenReview.net, 2024.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1 1 100. PMLR, 2020.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga. J., Pinto, A. S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bous- quet, O.. Gelly, S., and Houlsby, N. The visual task adaptation benchmark. CoRR, abs/ 190.0990. 2019.
Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, T. Adaptive budget allo- cation for parameter-efffciiett fine-tuning. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a.
Zhang, Z., Liu, B., and Shao, J. Fine-tuning happens in tiny subspaces: Exploring intrinsic task- specific subspaces of pre-trained language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 1701-1713, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10. 1ol11865///2022.aallllo
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencoodeinterpreter: Integrating code generation with execution and refinement. http::/aariivorg/aa22440...
2024.
Zi, B., Qi, X., Wang, L., Wang. J., Wong, K., and Zhang, L. Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. CoRR, 0et/23399024111 2023. doi: 10.48550/ ARX1V.2309.02411.
18

---

Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q.. Vanhoucke. V., Tran, T., Soricut, R., Singh, A., Singh, J.. Sermanet, P., Sanketi, P. R.. Salazar, G., Ryoo, M. S., Reymann, K., Rao, K., Pertsch, K., Mordatch, I., Michalewski, H., Lu, Y., Levine, S., Lee, I.., Lee, T. E.,, Leal, I., Kuang. Y., Kalashnikov, D., Julian, R., Joshi, N. J., Irpan, A., Ichter, B., Hsu, J., Herzog, A., Hausman, K., Gopalakrishnan, K., Fu, C., Florence, P., Finn, C., Dubey, K. A., Driess, D., Ding, T., Choromanski, K. M., Chen, X., Chebotar, Y., Carbajal, J., Brown, N., Brohan, A., Arenas, M. G., and Han, K. RT-2: vision-language-action models transfer web knowledge to robotic control. Tan, J., Toussaint, M., and Darvish, K. (eds.), Conference on Robot Learning. CoRL 2023, 6-9 November 2023, Atlanta, GA, USA, volume 229 of Proceedings of Machine Learning Research, pp. 2165-2183. PMLR, 2023.
19

---

A Reproducibility Statement
The source code to reproduce the results collected in our work can be found at com/ml /llmml-- jall-jju/E ju/EEA.
B Natural language generation
We follow the experiments conducted in Hu et al. (2023) and fine-tune -lama-2-7B, Llama-3,1 1-BBB, Gemma-2-9B. Gemma-2-27Band Alama-3.1-705 on common sense reasoning tasks with Qa-style prompts. We keep the original prompt templates unchanged except for two minor modifications: For BoolQ we prepend the passage field before the question, and for WinoGrande we add line "Answer format:..." analogous to the other prompts. As done by Hu al. (2023) and Liu et al. (2024a) perform joint fine-tuning on all tasks. We furthermore evaluate the pre-trained models mentioned above the mathematical reasoning tasks GSM8K (Cobbe et al., 2021) and Math (Yu et al., 2024) after tin-ttning on MetaMathQA (Yu et al., 2024) as done in Meng et al. (2024). We keep the original prompt template for fine-tuning and evaluation. For all datasets, we performed fine-tuning for one epoch. For training Llama-3.1-700 we use 4-bit quantization of the base model and training of adapter weights in bfloat16, recommended in Dettmers et al. (2023).
B.1 Implementation details
Table 4: hyperparameters for finetuning on com- mon sense reasoning and math reasoning
For fine-tuning code base leverages PEFT implementations adapter methods LoRA, AdaLoRA, PiSSA, OLoRA, LoRA-GA, CorDA and DoRA. The initialization step for EVA is custom implementation, but for fine-tuning we can reformulate EVA as a LoRA adapter leveraging the rank_pattern argument of LoraConff For evaluation, we used scripts provided by the MetaMath github repos- itory (Yu et al., 2024) for math reasoning tasks. For common sense reasoning,
Training
Optimizer
AdamW
Weight Decay
0.0
Lora Dropout
0.0
Batch Size
32
#Epoch
make use of the Im evaluation harness project (Gao et al., 2024) and define custom tasks using the fine- tuning prompts. For the SVD computation for joint fine-tunine on the common sense reason- ing tasks, we experiment with random and strat- ified sampling of examples from the tasks and do not notice a difference in performance. All training and evaluation runs for Llama-2-7B were performed on 4 A A100 GPUs. The runs for Llama-3, --B and Gemma-2-9B utilized two different nodes, one with 4 A100 GPUs and one with 4 H200 GPUs.
LR Schedule
Linear
Warmup ratio
0.03
Label Smooth
0.0
Learning Rate
5e-4
LoRA Dim
16
LoRA a
Batch Size SVD (EVA)
16
T
0.99
Inference
Beam Size
1.0
Length Penalty
1.0
repetition penalty
1.0
B.2 Hyperparameter search
The results reported on language generation tasks in Table 7 and Table 8 are the best setting based on grid search over different learning rates. We apply adapters to all linear layers including the language modeling head. Furthermore, we set = for all our experiments. We use Adam with weight decay and a linear learning rate schedule with warm-up. We train for I epoch and use the final checkpoint for evaluation. All hyperparameters are summarized Table As mentioned in 4.2 we tuned the number of samples for initialization for CorDA after consulting with the CorDA authors. Specifically we increased the number samples from 256 2560 after observing weak fine-tuning performance.
20

---

Table 3: Prompt templates with examples (red) used for finetuning common and math reasoning tasks.
Dataset
Fine-tuning Data Template
BoolQ
Passage: Drinking in public Drinking public most commonly accepted.
After reading this passage, please answer the following question with true or false, question: can you drink on the street in china
Answer format: true/false
the correct answer is true
PIQA
Please choose the correct solution the question: When boiling butter, when it's ready, you can
Solution Pour it onto a plate
Solution2: Pour it into jar
Answer format: solution 1/solution2
the correct
is solution2
SIQA
Please choose the correct answer to the question: Carson relocated somewhere
new. How would you describe Carson?
Answerl: mobile
Answer2: anxious
Answer3: lonely
Answer format: mnswer//answer2aaswwrr
the correct answer is answerl
HellaSwag
Please choose the correct ending to complete the given sentence: Playing
drums: People standing behind large drums. A man
Endingl: playing bag pipe.
Ending2: starts to play around the drums.
Ending3: hegins playing a drum set.
Ending4: begins playing the drums.
Answer format: ending /enn//nnn//ndinn///ndnnnnnn
the correct answer is ending4
WinoGrande
Please choose the correct answer in the blank complete given sentence: volunteered to eat Dennis's 's menudo after already having a bowl because despised cating intestine.
Option1: Tan
Option2: Dennis
Answer format: optionl/option2 the correct answer is option2
ARC-c ARC-c
Please choose the correct answer to the question: Which factor will most likely cause person to develop fever?
Answerl: a leg muscle relaxing after exercise
Answer2: bacterial population in the bloodstream
Answer3: several viral particles the skin
Answer4: carbohydrates being digested in stomach
Answer format: answer http:///aw/w////////////// 2/answe////
the correct answer is answer2
OBQA
Please choose the correct answer to the question: The sun is responsible for
Answerl: puppies learning new tricks
Answer2: children growing up getting old
Answer3: flowers wilting a vase
Answer4: plants sprouting, blooming and wilting
Answer format: answer
the correct answer is answer4
MetaMathQA
Below is instruction that describes task. Write response that appropriately completes the request.
### Instruction:
What is the value of the cosine of 90 degrees?
### Response:
S\boxed (O)$. The answer is: 0
21

---

Table 5: Distance between final adapters trained with LoRA or EVA. We report spectral norm (Ø) and average cosine similarity (cos) for Llama-2-7B, Llama-3.1 1-BB and 1lama-3.1-700 Our results demonstrate that the effect of different initializations are massive, as the final adapters converge to entirely different solutions, which is indicated by large and cos around zero
Query
Key
Value
Out
Gafe
Up
Down
Model
82
82
8g
cos
Cos
Llama-2-7B
-0.01
4.98
0.00
5.00
0.01
4.00
0.00
4.05
0.00
6.64
-0.00
3.67
-0.00
4.02
Ilama 3.1a--31--0 188
-0.00
4.05
-0.01
5.25
-0.00
3.83
-0.01
3,53
-0.00
6.98
0.01
3.37
-0.00
3.73
JJama-3.1-700
-0.01
7.57
0.00
7.52
-0.00
6.70
0.01
5.63
0.00
12.81
0.00
6.30
-0.00
6.33
Table 6: Distance between initialization of EVA and LoRA with their respective final adapters after training. We report spectral norm (o) and average cosine similarity (cos) for Llama-2-7B. Llama-3.1-8B, and Llama-3.1-70B. Our results demonstrate that EVA initialization is a larger constituent of the final adapter than LoRA, indicating that EVA contains more information at
initialization.
Method
Model
pas(()
Query
cre(f)
Key
a())
oos(1)
Valuue
a(4)
coa(t)
a(l)
oxs(f)
Gate
((J)
ros(t)
Up
a(L)
cre(") a(J)
Down
Llama-2-77 2-7B
0.51
3,85
0.48
4.08
0.60
3.10
0.59
3:09
0.44
5.27
0.62
2.83
0.61
3.13
LoRA
L.1m8-3.1-88
0.51
3.96
2.93
5.88
2,98
Llama- 3.1-708
0.45
4.62
0.42
5.07
0.52
3.86
0.61
3.17
0.39
6.74
0.61
3.11
0.62
3.13
Llama 1-7m
0.62
3.48
0.59
3.59
0.62
2.90
0.62
2.78
0.42
4,92
0.66
1.61
0.67
2.84
Llama-3, 3.1-8
0.64
3.62
2,46
0.64
2.27
5.12
0.67
0.67
2.71
Llama-3 3.1 1-708
0.53
4,27
0.52
4.62
0.53
3.68
0.58
2.91
0.33
6.53
0.59
3.24
0.59
3,16
B.3 Additional results
To demonstrate the effect of initialization, we measure the distance between the final adapters trained via LoRA and EVA and report cosine similarity and frobenius norm in Table 5. Our results demon- strate that depending on the initialization the two methods converge to substantially different solutions similarity between them. Furthermore,
as there is almost starts closer to its final solution, we report the distance of EVA to the adapter weights after training compared to the distance of LoRA to the adapter weights after training for different weight matrices of the model in Table 6. In Figure 12, right we also visualize this finding for the three variants of Llama. Our results consistently indicate that (i) initialization has tremendous impact the final solution, and (ii) EVA initialization results in less information being learned than for LoRA, as initializes the adapters contain most of the information at initialization.
highlight that EVA initialization
We present the per-task performance for the eight common sense reasoning tasks in Table 7. The respective standard deviations are shown in Table 14. Further, we show the results for all methods on the two math reasoning datasets in Table 8.
To investigate whether the observed improvement in performance depends on the rank, we conducted an additional experiment in which we vary the rank. Recall that in Section 4.2 we only used r = 16. Therefore, we conduct experiments for E {8. 16, 32, 64) for Llama-2-7B on the eight common sense reasoning tasks. We report the results in Table 9. Our results demonstrate that EVA or EVA+DoRA A are consistently the best performing methods for all ranks. Also, perhaps surprisingly. we find that a higher number of ranks does not always perform better. Our intuition is that the final performance strongly depends on the dataset size, i.e. the more parameters are introduced, the more likely the model tends to overfit.
We present additional loss curves for Llama-2-7B. Llama-3. 3.1-B, and Gemma- 2-9B in common sense and math reasoning tasks in Figure We find that EVA converges the fastest for all different models on the different tasks.
Another experiment we conduct is to apply recently proposed changes to the scaling factor and 2r which results in learning rate. Table 10 we show results for changing the scaling factor to a = /F rank stabilization (Kalajdzievski, 2023). In addition, we present results for the regular setting = 2r as proposed Hu al. (2022). Finally, we also show different learning rates for the two matrices A and B as proposed by Hayou et al. (2024). We make the following observations.
22

---

Table Comparison LoRA and DoRA to different initialization and rank re-distribution methods on NLG tasks. We report average performance across three seeds and respective standard deviation in Table 14. EVA+DoRA and EVA consistently attain the highest average performance across all tasks.
Model
Method
BoolQ
PIQA
SIQA
HellaSwag Winogrande ARC-e ARC-e OBQA
Avg.
LoRA
67.2
83.9
82.0
94.7
84.0
87.8
74.1
84.0
82.2
AdaLoRA
82.2
80.5
93.3
79.4
71.1
80.6
81.0
PiSSA
62.6
84.8
81.2
94.5
84.8
87.8
74.8
85.4
82.0
MiLoRA
65.0
84.8
82.3
94.9
84.5
88.2
74.9
85.3
82.5
OLoRA
68.7
84.8
82.2
95.0
85.0
88.1
74.9
85.2
82.9
Llama-2-7B
LoRA-GA
69.0
85.6
82.3
95.0
85.0
88.7
75.9
85.8
83.4
CorDA
80.4
79.7
91.7
82.5
67.9
78.4
78.4
EVA
68.3
85.3
82.9
95.2
85.2
88.6
75.8
86.3
83.4
DoRA
68.3
85.1
82.2
94.9
84.3
88.7
74.8
86.3
83.1
EVA+DoRA
73.5
85.3
82.4
95.2
84.8
88.9
76.0
87.3
84.2
LoRA
85.7
90.3
83.0
96.9
88.4
94.2
84.8
90.1
89.2
AdaLoRA
83,9
89.5
81.7
96.2
86.3
93.7
82.7
86.8
87.6
PiSSA
72.9
87.3
81.6
95.3
87.8
91.7
81.2
87.6
85.7
MiLoRA
85.7
90.8
83.0
96.8
88.8
94.4
84.9
90.5
89.4
OLoRA
86.0
90.4
83.9
97.0
84.7
90.3
89.4
Llama-3.1-8B
LoRA-GA
83.7
89.7
83.1
96.7
88.8
94.2
85.3
90.4
89.0
CorDA
69.1
82.8
79.4
91.5
82.4
86.3
73.7
82.3
80.9
EVA
85.3
90.4
83.4
97.0
89.0
94.4
86.0
90.3
89.5
DoRA
86.2
90.8
83.4
96.9
88.6
94.3
84.9
89.4
89.3
EVA+DoRA
85.8
90.8
83.9
97.1
89.2
94.4
85,9
90.5
89.7
LoRA
88.3
92.9
85.2
97.8
92.3
97.2
89.9
94.4
92.2
AdaL.oRA
87.3
91.8
84.6
97.3
91.3
97.0
90.0
92.6
91.5
PiSSA
81.4
90.0
82.5
95.5
89.0
93.6
83.5
90.8
88.3
MiLoRA
88.2
93.0
85.0
97.8
92.9
97.4
90.2
93.9
92.3
OLoRA
87.7
92.5
85.2
97.5
92.5
96.6
88.7
93.7
91.8
Gemma-2-9B
LoRA-GA
87.3
92.1
84,5
97.4
93.2
96.4
89.2
94.3
91.8
Corda
63.1
87.2
82.2
94.0
87.9
93.7
84.4
90.8
85.4
EVA
88.6
85.3
97.9
90.5
94.5
92.5
DoRA
88.3
92,6
84.9
97.7
92.2
97.1
89.9
94.5
92.1
EVA+DoRA
88.6
93.1
85.1
97.9
92.5
97.3
89.6
94.8
92.4
LoRA
89.0
98.0
93.6
92.1
95.2
93.1
AdaL.oRA
89.6
93.7
85.2
97.9
93.0
97.7
92.1
94.9
93.0
PiSSA
82.0
89.9
82.4
95.7
90.5
93.8
84.7
91.3
88.7
Gemma-2-27B
OLoRA
89.4
94.7
86.3
98.2
94.3
97.9
92.8
96.0
93.6
EVA
89.4
94.6
85.8
98.3
94.4
98.0
93.0
95.9
93.7
DoRA
89.1
94.7
85.7
98.1
93.3
98.0
92.8
95.1
93.3
EVA+DORA
89.4
94.6
85.8
98.1
94.2
97.8
92.1
95.9
93.5
LoRA
95.9
86.2
98.5
98.4
93.4
97.2
93.6
AdaLoRA
90.4
95.1
85,8
98.0
93.3
98.2
93.7
96.7
93.8
Llama-3.1 3.1-00B
PiSSA
40.6
51.5
35.4
25.8
50.5
25.8
25.3
27.2
35.3
OLoRA
90.3
96.0
86.2
98.4
95.5
98.3
93.5
96.9
94.4
EVA
90.8
96.1
86.3
98.6
95.0
98.4
93.8
96.8
94.5
1. The standard setting a = 2r from Hu et al. (2022) leads to the worst performance
2. Rank stabilization via 2r. EVA
VF significantly improves the performance of both LoRA and
3. Different learning rates for A and B did not improve the results
To provide comprehensive comparison of the effect of rank redistribution, we compare uniform ranks (p = 1) to adaptive ranks (p 2) on common sense and math reasoning tasks Table 11. We find that adaptive ranks consistently improve performance for Gemma-2-9B. For Llama-2-7B and I.aam-33 1-8B we observe improvements common sense reasoning tasks only, while uniform ranks perform better math fine-tuning tasks. In Table 11 we also show the number of trainable
23

---

Table 8: Comparison of EVA to other initialization and adaptive rank methods on GSM8K and MATH datasets. We report mean and standard deviation across three random seeds.
Model
Method
GSM8K
MATH
LoRA
59.7
10.9
AdaLoRA
46.9
9.64.2
PiSSA
61.
12.6+.4
MiLoRA
59.7 19.7
11.2+.1
OLoRA
60.7
11.8
Llama-2-78
LoRA-GA
60.21.6
11.71.4
CorDA
59.0+1.2
11.8+.6
EVA
61.9
13.1
DoRA
59.8
11.5+.2
EVA+DORA
62.5
13.4
LoRA
78.3
30. 1
AdaLoRA
76.9
28.9+
PISSA
78.8
29.5
MiLoRA
78.64.
30.3
OLoRA
78.0
31.0+.7
Llama-3.1-8B
LoRA-GA
78.8
30.0
CorDA
76.8
27.9+.2
EVA
78.8
31.2+t.3
DoRA
77.9
30.2
EVA+DoRA
79.1
30.8
LoRA
83. 13.4
10.7
AdaLoRA
83.5
41.1+.4
PiSSA
79.8
34.9
MiLoRA
83.74.4
41.9
OLoRA
82.2+.2
39. 39.4+.6
Gemma-2-9B
LORA-GA
82.8+.8
40.4+
CorDA
56.316.2
25.414.0
EVA
83.6+
41.54.3
DoRA
82.5+.6
39.7+.4
EVA+DORA
$2.9
40.01.6
parameters for EVA (p 2) compared to LoRA on common sense and math reasoning tasks. We find that after rank redistribution, EVA leads to improved performance while reducing the parameter count by approximately MM. The reason for this is that parameters are usually redistributed from higher dimensional projections lower dimensional ones, i.e. from non-attention weights attention weights. This results in improved performance while reducing the parameter count.
Finally, to verify our intuition that the LoRA matrix A should be initialized with the projection onto the components that explain the most variance, we compare its performance with initializing EVA with the components that explain the least amount of variance. We call this method EVA-minor and present results for it in Table 12. To implement EVA- we sample 20 minibatches of data and perform truncated SVD on those and select the resulting minor components. This incurs substantial additional cost, as we must compute all components, whereas for EVA we only approximate the components that explain the most variance. Hence, incremental SVD is not beneficial in this case anymore and it is also not practical as obtaining the initialization takes hours instead of seconds for EVA. Morcover, our data-driven heuristic for adaptive rank allocation no longer applicable to this case; therefore, EVA-minor, highlighting the importance of initializing EVA with the major components, i.c. the ones that explain the most variance.
consider uniform ranks. Finally, find that EVA consistently improves over
In addition we also fine-tun tune Llama-2-77 on the Feee-Feed dataset Zheng et al. (2024) consisting of multi-turn conversations between user and AI Assistant. Due to limited computational resources and the long sequence lengths of the examples in this dataset we do not fine-tune Llama-3. 1-8B and Gemma-2-99 or any DoRA variants. We evaluate the fine-tuned checkpoints benchmarks: MBPP Austin al. (2021), HumanEval Chen et (2021b), MBPP+ and HumanEval+ Liu et al. (2023). The results are presented in Table 13. EVA shows the best performance MBPP
four coding

---

Table 9: Comparison of different ranks for fine-tuning Llama-2-7B on the eight common sense reasoning tasks.
Rank Method
BoolQ PIQA
SIQA HellaSwag Winogrande ARC-e ARC-c OBQA Avg.
LoRA
67.6
84.0
82.1
94.6
84.2
74.2
82.3
AdaLoRA
70.0
82.4
80.7
93.4
80.1
86.4
70.9
79.9
80.5
PiSSA
62.5
84,9
81.2
93.9
84.2
87.0
74.4
85.4
81.7
OLoRA
65.4
84.5
82.3
94.9
84.8
88.4
74.7
85.5
82.6
LoRA-GA
69.1
84.8
82.2
94.8
84.1
87.8
733.9
85.7
82.8
8
EVA (p= 1)
72.6
85.4
82.3
95.2
84.9
88.8
75.2
85.3
83.7
EVA (p =/=2)
74.1
85.6
82.6
95.1
85.0
88.7
75.5
86.3
84.1
DoRA
65.0
84.6
82.3
94.9
84.3
88.7
74.7
85.6
82.5
EVA+DORA (p=1) 1)
71.6
85.8
82.5
95.2
85.3
88.9
75.3
86.2
83.9
EVA+DORA (p=
69.9
84.7
82.3
95.2
84.0
88.3
74.8
84.3
82.9
LoRA
68.0
84.0
82.1
94.7
83.8
87.8
73.8
84.5
82.3
AdaLoRA
73.8
82.1
80.6
93.3
79.2
86.1
71.1
80.1
80.8
PiSSA
62.6
84.9
81.3
94.5
84.6
87.6
75.2
85.5
82.0
OLORA
69.5
84.8
82.5
95.0
84.6
88.0
74.7
85.1
83.0
MiLoRA
65.0
84.8
82.3
94.9
84.5
88.2
74.9
85.3
82.5
16
LoRA-GA
69.0
85.6
82.3
95.0
85.0
88.7
75.9
85.8
83.4
EVA (p=
71.2
85.2
82.2
95.2
84.2
88.6
75.4
84.9
83.4
EVA (p=
68.3
85.3
82.9
95.2
85.2
88.6
75.8
86.3
83.4
DoRA
68.3
85.1
82.2
94.9
84.3
88.7
74.8
86.3
83.1
EVA+DORA (p 1)
73.5
85.3
82.4
95.2
84.8
88.9
76.0
87.3
84.2
EVA+DoRA (p=
74.4
85.3
82.5
95.1
85.2
88.9
75.4
85.4
84.0
LoRA
69.1
84.0
82.0
94.7
83.7
88.2
73.9
84.4
82.5
AdaLoRA
72.6
82.2
80.6
93.2
80.3
86.2
71.1
79.9
80.8
PISSA
65.1
84.7
81.0
94.1
84.5
87.6
73.5
86.2
82.1
OLoRA
63.6
84.8
82.4
95.0
84.7
88.6
75.2
85.7
82.5
LoRA-GA
69.0
85.7
82.0
95.3
84.7
88.8
75.2
86.5
83.4
32
EVA (p=
69.2
85.1
82.9
95.0
85.3
88.6
74.9
85.3
83.3
EVA (p
65.4
85.4
82.9
95.2
85.0
88.5
75.3
85.4
82.9
DoRA
66.9
84.9
82.1
95.0
84.5
88.6
74.7
84.7
82.7
EVA+DoRA (p 1)
69.0
82.7
95.2
84.8
75.7
86.9
83.7
EVA+DoRA (p-
71.0
84.2
81.9
95.0
84.3
74.3
82.9
LoRA
74.7
84.2
82.1
94.6
84.0
88.0
75.0
83.8
83.33
AdaLoRA
71.5
82.0
80.4
93.1
80.2
86.0
71.1
79.9
80.5
PiSSA
64.9
84.6
81.3
94.0
84.5
87.6
73.3
85.0
81.9
OLoRA
70.0
84.8
82.4
94.9
84.7
88.7
75.3
85.9
83.3
LoRA-GA
70.5
85.2
82.4
95.1
84.6
75.4
85.5
83.4
64
EVA (p
66.6
85.2
82.6
95.0
84.8
75.3
85.1
82.9
EVA (p 2)
71.2
84.7
82.7
95.0
84.5
74.9
85.3
83.3
DoRA
70.5
85.0
82.6
94.9
84.8
88.3
74.7
85.9
83.3
EVA+DoRA (p- 1)
67.4
85.3
82.6
95.1
84.9
88.9
75.5
86.6
83.3
EVA+DoRA (p 2)
71.6
84.6
82.2
94.9
84.0
88.2
75.0
84.8
83.2
Table 10: Comparison of EVA to LoRA using recently proposed advancements, such as rank stabilized scaling (Kalajdzievski, 2023) or different learning rates for B and A (Hayou al., 2024), as well as the originally proposed scaling from Hu et al. (2022).
Adaptation
Method
BoolQ
PIQA
SIQA
HellaSwag
Winogrande
ARC-e
ARC-e
OBQA
Avg.
LoRA
64.5
84.7
81.6
94.4
83.8
87.3
73.9
85.5
82,0
LoRA+
EVA
68.6
85.0
81.2
94.2
84.7
87.4
73.5
84.1
82.3
LoRA
71.5
85.3
82.5
95.2
84.5
89.0
75.8
86.8
83,8
PSLoRA
EVA
75.5
86.1
82.7
95.4
86.1
89.3
76.3
86.3
84.7
LoRA
77.9
82.1
80.1
93.2
79.8
86.3
71.5
79.3
81.3
32
EVA
68.6
84.9
82.2
94.6
84.1
87.8
74.7
84.4
82.7
and MBPP+ while also exhibiting good performance on HumanEval and HumanExalt. For the latter two datasets, PiSSA is the best-performing method. For fine-tuning, we use maximum sequence
25

---

Random
OLORA
Random
PISSA
0.50
0.5
0.275
0.45
0.250
Training Loss
0.40
6.3
Training Loss
0.225
0.20
0.15
0.2
0.200
0.30
0.175
2K
0.25
0.150
0.125
0.15
0.100
0K
2K
0K
2K Update Steps
4K
Updote Steps
Random
PSSA
OLoRA
EVA
Random
PISSA
OLoRA
EVA
0.250
0.24
0.225
0.20
0.22
Training Loss
Training Loss
0.22
0.18
0.175
0.10
0.150
0K
2K
0K
2K
0.125
0.18
0.100
0.16
2K Update Steps
Update Steps
Figure 6: Loss curves for Llama-2-7B on common sense reasoning (top left), Llama-3. 1--BB on common sense reasoning (top right), Gemma-2-9B common sense reasoning (bottom right), and Gemma-2-9B on MetaMathQA. EVA consistently converges the fastest among all competitors.
length of 2028 with a right-hand side truncation. For decoding, we set the temperature to 0.2 and top-p to 0.7
C Natural language understanding
C.1 Dataset Statistics
The dataset statistics for each task in the GLUE benchmark (Wang et al., 2019) are shown in Table 15. Generally, GLUE contains four low-resource datasets (RTE, MRPC, STS-B, and CoLA) and four high-resource datasets (SST-2, QNLI, QQP, and MNLI). While CoLA and SST-2 rely on single sentence classification, STS-B evaluates for similarity and the remaining tasks are based on pairwise text classification.
C.2 Implementation Details
We base implementation the LoRA codebase'. For these experiments, initially precompute our initialization prior to the fine-tuning stage and store as a checkpoint. However, we also provide the possibility directly compute the initialization during the fine-tuning stage, done for experiments on (TTA--I k and Meta-World By default, we always offload the computation of the initial checkpoint CPU save VRAM. We ran all experiments and used Torch''s data-distributed parallel functionality (Paszke et al., 2019). Runtimes range from little as 10 minutes per for smaller datasets (RTE, STS-B) around 15 hours for the largest datasets (QQP, MNLI).
nodes with four 1000 GPUs
'https://githur howp.////c//cccooo////llt
26

---

Table 11: Comparison of number of trainable parameters between LoRA-basee methods and EVA on the math and common sense reasoning tasks. Common sense reasoning is an average over eight tasks. #Trainable represents the number of trainable parameters. EVA consistently improves performance while decreasing the number of trainable parameters.
Model
Method
#Trainable Common sense GSM8K MATH
LoRA
40.6M
82.2
59.7
10.9
AdaLoRA
40.6M
81.0
56.9
9.6
PiSSA
40.6M
82.0
61.1
12.6
MiLoRA
40.6M
82.5
59.7
11.2
Llama-2-74
OLoRA
40.6M
82.9
60.7
11.8
LoRA-GA
40.6M
83.4
60.2
11.7
EVA (p 1)
40.6M
83.4
61.9
13.1
EVA (p 2)
39.3M
83.4
61.0
12.5
LoRA
44.1M
89.2
78.3
30.1
AdaLoRA
44.1M
87.6
76.9
28.9
PiSSA
44.1M
85.7
78.8
29.5
MiLoRA
44.1M
89.4
78.6
30.3
Llama-3.1-8B
OLoRA
44.1M
89.4
78.0
31.0
LoRA-GA
44.1M
89.0
78.8
30.0
EVA (p 1)
44.1M
89.4
78.8
31.2
EVA (p 2)
42M
89.5
78.3
30.8
LoRA
58.2M
92.2
83.4
40.7
AdaLoRA
58.2M
91.5
83.5
41.1
PiSSA
58.2M
88.3
79.8
34.9
MiLoRA
58.2M
92.3
83.7
41.9
Gemma-2-9B
OLoRA
58.2M
91.8
82.2
39.4
LoRA-GA
58.2M
91.8
82.8
40.4
EVA (p =
58.2M
92.4
83.6
41.3
EVA (p =
55.9M
92.5
83.6
41.5
LoRA
114.2M
93.1
AdaLoRA
114.2M
93.0
PiSSA
114.2M
88.8
Gemma-2-27B OLoRA
114.2M
93.7
EVA (p
114.2M
93.7
EVA (p
104.8M
93.7
LoRA
209.3M
93.6
AdaLoRA
209.3M
93.9
PiSSA
209.3M
35.2
Llama-3.1-70B OLoRA
209.3M
94.4
EVA (p 1)
209.3M
94.5
EVA (p = 2)
193.6M
94.5
C.3 Hyperparameter search
For LoRA and EVA, we search the number of ranks r E (2,4,6,8) and the different learning rates for 3 We report the best hyperparameter settings for both RoBERTaLage and DeBERTav33.. for LoRA and EVA in Table 16. For AdaLoRA, we search the same ranks and always start the initial ranks with r + 4 that are then redistributed during training. For BOFT we sweep over different combinations of block sizes E (2, 4, 8, 16) which determine
n E (le-3, 3, 4e 4, 4] for RoBERTa and 7) E {4e-3, 1, 4e -4
number multiplicative matrices. Additionally, for both AdaLoRA and BOFT, we search over the same learning rates as for the other LoRA variants. Further, we introduce hyperparameters that result in additional speed-up our initialization, namely threshold r that considers components as converged, and threshold that stops computation of the
27

---

Table 12:
Comparison of EVA to EVA-minor, which leverages components that explain the least amount of variance for initialization of A. on the common sense reasoning tasks.
Method
BoolQ
PIQA
SIQA
HellaSwag
Winogrande
ARC-e
ARC-c
OBQA
Avg.
EVA
68.6
85.0
81.2
94.2
84.7
87.4
73.5
84.1
82.3
EVA-minor
64.0
83.4
81.5
94.3
82.0
87.3
73.0
81.6
80.9
Table 13:
Comparison of EVA to other initialization and rank re-distribution schemes on code fine- tuning datasets. We report mean and standard deviation across three random seeds.
Method
MBPP
HumanEval
MBPP+
HumanEval+
LoRA
22.2+1.1
18.9 18.9
30.7+1.1
18.9 t0.6
AdaLoRA
21.5
17.1.0.0
29.4
17.1
PiSSA
22.8
19.9
30.8+0.7
19.9
OLoRA
22.3
18.9
32.4
18.9
EVA
22.9
18.9 18.9
32.6
18.9 11.2
initialization when a certain percentage of components have converged. By default, we set T = 0.99 and 8 = 1, i.e. only stop when all components converge. These parameters provide additional leeway to speed up the initialization stage of EVA.
We have explored the sensitivity of LoRA to different initialization schemes and found that, similar to other prominent initialization schemes (He et al., 2015; Glorot & Bengio, 2010), scale plays an important role along with directions. Originally, (Hu et al., 2022) propose to set = 2r, however, we found that this parameter quite sensitive as also shown (Kalajdzievski, 2023). Similarly, different ranks lead to very different results on different downstream tasks. Therefore, we suggest that one always search over more ranks and choose the best performing one the required compute budget available. We also experimented with different learning rates for the A and B matrices proposed in (Hayou et al., 2024), however, this did not result in consistent improvements. Instead, we found that learning rates for LoRA-style training can be surprisingly high (4e - 3 for DeBERTav3 while for larger models the learning rate needs to be approximately a magnitude smaller. A simple recipe that worked consistently well was to set Kalajdzievski (2023), and searching over a set of small learning rates for larger models and higher learning rates for smaller ones. For EVA, the only tunable hyperparameter is the rank budget, which we recommend to tune along with the learning rate.
1, which results in similar scaling factor in
C.4 Additional results
We report additional results for EVA compared to LoRA for different rank budgets in Table 17. We find that EVA consistently outperforms LoRA for different rank budgets. This demonstrates the effectiveness of EVA among different compute budgets. addition, we show additional rank redistributions for CoLA, MRPC, RTE, and STSB tasks for different for 2 (Figure 7), 4 (Figure 8), r (Figure 9), and = 16 (Figure 10) for both RoBERTa arge and DeBERTav3Race The distributions for the different models show different patterns. For DeBERTavv attention layers usually receive more ranks than the lower ones. For CoLA, there are also a large number of ranks in the very first layer. For RoBERTaL first layers consistently receive more ranks compared to the later layers. There is also a notable difference between tasks for both models, which demonstrates the flexibility of EVA to allocate ranks dependent on the downstream task. Interestingly. for a higher initial rank (r = 16), the redistribution for DeBERTav3Ese puts
the higher
seems to be the opposite, the very
emphasis fine-tuning the self-attention specific weight matrices. This is not true for RoBERTa, arge. as W,1 also receives plenty of ranks across all tasks. Overall, the rank redistribution incurs different fine-tuning paradigms depending the task and the initial rank.
Additionally, we show results for different rank redistributions that we obtain by using alternative measures for explained variance. Specifically, compare EVA to using (1) the raw cigenvalues
28

---

Table 14: Per-task standard deviation across three seeds for all methods on common sense reasoning
tasks.
Model
Method
BoolQ
PIQA
SIQA
HellaSwag Winogrande ARC-e ARC-c OBQA
LoRA
1.498
0.252 0.233
0.102
0.658
0.072
0.489
0.822
AdaLoRA
1.315
0.251 0.182
0.098
0.392
0.362
0.106
0.899
PISSA
0.358
0.294 0.138
0.096
0.298
0.386
0.494
1.117
MiLoRA
3,950
0.392 0.329
0.097
0.810
0.064
1.100
0.231
Llama-2-7B
OLoRA
4.938
0.190 0.524
0.062
0.652
0.339
0.672
0.660
LoRA-GA
10.573
0.416 1.049
0.115
0.344
0.170
0.560
0.721
CorDA
8,801
2.039 0.253
0.549
2.009
1.756
2.836
4.243
EVA
7.974
0.137 1.054
0.101
0.810
0.526
0.421
0.577
DoRA
2.599
0.290 0.483
0.113
0.244
0.215
0.489
0.525
EVA+DORA
5.281
0.273 0.293
0.034
0.853
0.110
0.494
0,249
LoRA
0.472
0.194 0.419
0.070
0.197
0.052
0.563
0.189
AdaLoRA
0.510
0.044 0.261
0.040
0.392
0.201
0.804
0.748
PISSA
6.516
0.373 0.603
0.195
0.707
0.325
0.245
0.589
MiLoRA
0.511
0.163 0.300
0.125
0.613
0.445
0.887
0.503
Llama-3.1-8B
OLoRA
0.298
0.245 0.397
0.057
0,451
0.173
0.329
0.189
LoRA-GA
0.539
0.237 0.695
0.115
0.592
0.135
0.729
0.800
CorDA
3.676
0.077 0.145
0.070
2.009
1.905
1.508
0.424
EVA
0.353
0.031 0.194
0.046
0.209
0.292
0.178
0.808
DoRA
0,225
0.112 0.315
0.014
0,260
0.119
0.698
0.000
EVA+DoRA
0.225
0.168
0.121
0.117
0.392
0.105
0.175
0.249
LoRA
0.095
0.277
0.386
0.062
0.324
0.072
0.070
0,589
A
0.088
0.353
0.217
0.033
0.098
0.209
0.106
0,432
PiSSA
2.761
0.286 0.214
0.109
0.621
0.447
0.121
0.163
MiLoRA
0.284
0.191 0.325
0.089
1.008
0.239
0.903
0.115
Gemma-2-9B
OLoRA
0.066
0.451 0.501
0.099
0.501
0.267
0.448
0.573
LoRA-GA
0.662 0.463 0.252
0.072
0.526
0.129
0.617
1.026
CorDA
17.299
0.154 0.109
1,486
1.730
0,268
0.845
0.000
EVA
0.275
0.136 0.111
0.094
0,260
0.119
0.040
0.249
DoRA
0.189
0.420
0.301
0.074
0.419
0.091
0.000
0.499
EVA+DoRA
0.132
0.296 0.490
0.070
0.037
0.150
0.715
0.340
LoRA
0.202
0.045
0.424
0.109
0.196
0.155
0.600
0.497
AdaL.RRA
0.300
0.286 0.158
0.022
0.429
0.020
0.161
0.249
PiSSA
3.035
0.645 0.529
0.135
0.578
0.288
0.408
0.736
Gemma-2-27B
OLoRA
0.038
0.200 0.233
0.046
0,226
0.182
0.435
0,864
EVA
0.250
0.277 0.147
0.031
0.322
0.292
0.707
0,432
DoRA
0.364
0.194
0.111
0.038
0.149
0.110
0.329
0.189
EVA+DoRA
0.336
0.000
0.026
0.085
0.316
0.084
0.555
0.500
LoRA
7.296
0.068
0.230
0.059
0.134
0.105
0.418
0.327
AdaLoRA
0.300
0.077
0.274
0.060
0.232
0.110
0.224
0.189
Llama-3. Llama-3.1-70B
PiSSA
1.208
0.544
1.407
0.070
0.079
0.968
1.195
3.400
OLoRA
0.548
0.143
0.301
0.119
0.207
0.209
0.426
0.411
EVA
0.227
0.204
0.319
0.059
0.335
0.069
0.420
0.249
(EVA-Raw) and (ii) normalizing by the maximum cigenvalue (EVA-Max). We report results for RoBERTa on four GLUE tasks, namely CoLA, RTE, MRPC, and STS-B in Table 18. Our results show that while EVA-Raw and EVA-Max slightly improve upon LoRA, they perform worse on average than EVA.
D Image Classification
D.1 Dataset statistics
The VTAB-IK benchmark consists of datasets, each containing a subset 1000 examples of their respective samples. We summarize the statistics for each datasct in Table 19, Although the original

---

W
W
W
W
12
Lige
MRPC
MRPC
W
W
W
W
W.
W.
W.
W.
W n
Wya y2
W
RTE
RTE
W
W
W.
W.
W.
W.
W.
W
W
W
W2a
Loye
STSB
STSB
W
W.
W
W.
W
W
W
W
W
33
21
Figure 7: Rank distribution after initialization with EVA four tasks of the GLUE benchmark (CoLA, MRPC, RTE, STSB) for DeBERTav3 (left) and RoBERTa (right) with initial rank 2
30

---

W
W.
W.
Wn
W
12
MRPC
MRPC
W
W
W
W
W.
W.
W.
W.
W
Wya
W /2
RTE
RTE
W
W
W.
W
W.
W
W.
W
W
W
W2a
Loyer
STSB
STSB
W
W.
W
W.
W
W
W
W
W
Figure 8: Rank distribution after initialization with EVA four tasks of the GLUE benchmark (CoLA, MRPC, RTE, STSB) for DeBERTav3 (left) and RoBERTa (right) with initial rank r=4.
31

---

13
30
32
10
36
WA
50
W
W
12
Loyer
MRPC
MRPC
W
W
W
W
W.
W.
W.
W.
W
Wya ya
W /2
10
RTE
RTE
W
12
W
W.
W
10
W.
W
W
W
W
W
W2a
STSB
STSB
W
W
W
W.
W
22 32 32
W
W
W
W
15 21
Figure 9: Rank distribution after initialization with EVA four tasks of the GLUE benchmark (CoLA, MRPC, RTE, STSB) for DeBERTav3 (left) and RoBERTa (right) with initial rank r=8. 8.
32

---

23
22
22
24
22
23
W
W
W
12
MRPC
MRPC
W
W
W
W
W.
W
W.
F1
32
10 p0
W y2
20
16
W
/2
15
10
15
1:
RTE
RTE
W
16
15
28
23
21
21
12
22
W
19
W.
26
21
26
23
22
W.
W
W
W
W.
W
W
STSB Layer
STSB
W
21
23
W.
20
22
12
n
W
W.
22
20
28
22
a
W.
22 25
17
W
11
W
W
W
10
12
Figure 10: Rank distribution after initialization with EVA four tasks of the GLUE benchmark (CoLA, MRPC, RTE, STSB) for DeBERTav3 Buse (left) and RoBERTa (right) with initial rank
r= 16.
33

---

Table 15: GLUE benchmark suite statistics and evaluation metric for each corpus sorted by the
number of examples in the training set.
Corpus
#Train
#Dev
#Test
Metric
RTE
2.5 k
276
3
Accuracy
MRPC
3,7 k
408
1.7k
Accuracy
STS-B
7k
1.5k
1.4k
Pearson correlation
CoLA
8.5
1
1k
Matthew's correlation
SST-2
67 k
872
1.8k
Accuracy
QNLI
108 k
5.7k
5.7k
Accuracy
QQP
364
40
391 k
Accuracy
MNLI
393
20
20k
Accuracy
train sizes of the datasets vary drastically, the 1K subset provides equal datasets across tasks. The number of classes also varies from as little two almost 400.
D.2 Implementation details
We implemented a custom pipeline to fine-tune DINOv2-L/14 on VTAB-IE IK that supports LoRA, DoRA and EVA. To train AdaLora, PiSSA and OLoRA, we integrate their implementation from the peft library (Mangrulkar et al., 2022) into our pipeline. This pipeline designed to be highly parallelizable and be executed individual GPUs. single evaluation 19 datasets with hyperparameter tuning and evaluation) takes roughly 160 A 100 GPU-hours but can be easily parallelized. A g/14 14 takes roughly 140 H100 GPU-hours. A single evaluation consists of 140 hyperparameter tuning runs (19 datasets 5 learning rates 4 ranks 3 seeds) and 95 evaluation runs (19 datasets * seeds). Details to hyperparameter tuning are described below.
of L/14 model (all
We use the original DINOv2 models (Oquab et al., 2023) and train a classification head on top of the [CLS] token, where initialize the classification head weights with normal distribution with o = 2e-5 and bias with zeros. We train the classification head, LoRA matrices and biases. The images resized 224 x 224 resolution with bieubic interpolation and normalized with the per- per-channel mean and variance of ImageNet. We train all models with 6 precision using the AdamW optimizer with weight decay of 0.05 for 30 epochs. We cosine learning rate schedule with linear warm-up for the first epochs. The batch size is set to 64 where we use gradient accumulation the batch size does not fit into GPU memory. Full fine-tuning uses layer-wise Ir decay of 0.75 (Clark et al., 2020).
D.3
Hyperparameter search
We first fine-tune the 800 train samples of the VTAB-IK datasets find the best learning rate for the task. We sweep over learning_ rate E (2.5e-3, 1e-3, 7.5e-4, 5e-4, 2.5e-4) and rank E (2, 4, 8, 16} and average the accuracy on the 200 validation samples over different seeds choose the best learning rate and rank for each dataset. For evaluation, we train on the union of train and validation set using five different seeds and report the average accuracy the test set.
D.4 Additional results
We show our main results in Table 20. To complement these results, we report the respective standard deviations in Table 21.
E
Decision Making
E.1 Dataset statistics
Meta-World (Yu et al., 2020) is an established benchmark in RL. for multi-task continuous control.
The benchmark consists of 50 challenging robotic tasks simulated using Sawyer robotic arm in the
34

---

Table 16: The best hyperparameters Larg DeBERTav3 were found via gridsearch for each task of the GLUE benchmark.
Method
Dataset
MNLI SST-2
MRPC
CoLA
QNLI QQP
RTE
STS-B
Optimizer
AdamW
Warmup Ratio
0.06
LR Schedule
Linear
Batch Size
8
16
8
8
16
# Epochs
10
10
20
20
10
20
20
10
RoBERTaL LoRA
LoRA rank
2
8
Learning rate
4e-4
le-3
4e-4
le-3
le-3
le-3
le-3
4e-4
LoRA
Max Seq. Len.
512
DDP GPUs
4
Batch Size
8
16
8
8
16
RoBERTa ar
# Epochs
10
10
20
20
10
20
20
10
LoRA rank
2
2
16
8
4
4
EVA
Learning rate
4e-4
le-3
4e-4
le-3
4e-4
le-3
le-3
le-3
LoRA
1
Max Seq. Len.
512
DDP GPUs
4
Batch Size
32
32
16
32
64
32
32
16
# Epochs LoRA rank
30
60
30
80
25
25
80
40
DeBERTay3 LoRA
8
4
4
8
16
4
4
8
Learning rate
4e-4
1e-3
4e-3
4e-3
4e-3
4e-3
4e-3
4e-3
LoRA a
Max Seq. Len.
512
DDP GPUs
4
Batch Size
32
32
16
32
64
32
32
16
DeBERTav3Bae EVA
# Epochs
30
60
30
80
25
25
80
40
LoRA rank Learning rate
8
2
4
8
16
4
2
2
4e-4
4c-4
4c-3
4c-3
4c-3
4e-3
4e-3
4e-3
LoRA a Max Seq. Len.
512
DDP GPUs
4
MuJoCo physics engine (Todorov et al., 2012). All 50 tasks in Meta- World share the same underlying robotic arm. Therefore, all tasks share common state (39-dimensional continuous vector) and action space (6-dimensional). The reward functions in Meta-World are dense and based on the distance of the robotic arm the target location objects. All episodes last for 200 environment interactions.
For our experiments on Meta-World we use the datasets released by Schmied et al. (2024). We follow Wolezyk al. (2021) and Schmied et al. (2024), and split the 50 tasks into 40 pre-training tasks (MT40) and 10 fine-tuning tasks (CW10). The CW10 tasks are the following.
hammer-v2, push-wall-v2. faucet-close-ve stick-pull-v2, landle-press-side-22 push-v2, shelf thelf-place-v2. windov-close-v2 and peg-unplug-side-vv
push-back-v2, stick-pull-ne
The datasets contain 2M transitions for each of the 50 tasks, which equivalent to 80M transitions (320M tokens) for all training tasks. The average success rate and rewards for all MT40 tasks are 84% and 1414.62, respectively. We list the statistics per task in Table 22.
F.2 Implementation details
We implemented our pipeline that supports training on Meta-World on top of the code-base provided by Schmied al. (2024). Our custom implementation supports training LoRA, DoRA and EVA. Furthermore, we leverage the peft library (Mangrulkar et al., 2022) to train the remaining methods.
35

---

Table 17: Comparison of LoRA to EVA using RoBERTaLane on all tasks from GLUE for equal rank budgets. Mean and standard deviation of Matthew's 's correlation for CoLA, pearson correlation for STS-B, and accuracy for remaining datasets on the development set across 5 seeds are shown.
Method
CoLA
MRPC
RTE
STS-B
MNLI
QNLI
QQP
SST-2
Avg
2
68.0+1.4
90.9
88, 11.1
92.3=.1
91.
94.8+.3
90.6+.1
96. 96.1+
89.09
EVA,=2
19.1
90.8
88.2
92.5=
90.8
94.9
91.9..1
96.2
89.30
LoRA,=4
69.1
90.7
86.9
92.3
90.61.1
94.71.2
92.0
96.0..1 1
89.04
EVA,-4
69.5+1.4 .4
91.44.8
88.8_1.3
92.6
90.7+.0
94.9
91.8+.o
96. 1.
89.48
LoRA,
68.8
91.11.6
87.10.7
92.2=.2
90.6+.2
94.8+.1
91.8+.o
96.2; 2
89.08
EVA,
69.0
91.14.4
88.4
92.6
90.6
91.94.1
92.1
96.1+
89.35
LoRA,-1G
1+.00
90.54.6
88.0+.6
92.3=.1
90.6+.1
94.8=.1
91.9+.1
16.1+.1
89.08
EVA,=16
69.1..6
91.2..8
88.0
92.6
90.7
95.0
91.81.0
96.2...
80.33
Table 18: Comparison of LoRA to EVA, EVA-Raw, and EVA-Max for RoBERTaLangoon the GLUE tasks CoLA, MRPC, RTE, and STS-B. We report mean and standard deviation of Matthew's cor- relation for CoLA, pearson correlation for STS-B, matched accuracy for MNLI, and accuracy for remaining tasks across 5 seeds.
Method
CoLA
MRPC
RTE
STS-B
Avg
LoRA
69.1
91.1
88.1+1.1
92.3
85.2
EVA
69.541.4
91.4400.8
88.8+1.2
92.640.1
85.6
EVA-Raw
69.4+1.1
91.0+0.9
88.2+0.3
92.5
85.3
EVA-Max
69.1
91.2+0.5
88.441.2
92.5+
85.3
For our experiments on Meta- World, we use a GPT2-like network architecture (Radford et al., 2019) with 4 Transformer layers, 8 heads, and hidden dimension of 512 resulting in 16M parameters. We use a context of 50 time steps, which amounts to a sequence length of 200, as each timestep contains states, actions, rewards, and RTGs. We embed states, actions, rewards, and return-to-gos (RTGs) using separate linear embedding layers per modality, as proposed by Chen et al. (2021a). We train with batch size of 128 using constant learning rate of le 4000 linear warm-up steps followed by a cosine decay to le 1e-6, using the Adam W optimizer (Loshchilov & Hutter, 2017). We employ gradient clipping 0.25, weight decay of 0.01, and dropout rate of 0.2. Our DT implementation employs global position embedding. For each task, we set the target return to the maximum return achieved in the respective training datasets, as proposed by (Schmied et al., 2024). Furthermore, we employ mixed precision (Micikevicius et al., 2017) and flash attention (Dao, 2023) to speed up the training.
We first pre-train a DT on all MT40 tasks (80M transitions) for 1M updates via next-action prediction by minimizing the mean-squared error. The resulting pre-trained model achieves average success rate of 80% across all MT40 tasks. Then we fine-tune the DT on each of the CW10 downstream tasks for 100K updates with the same set of hyperparameters as used for pre-training. We run all our experiments on public research cluster with 4xA 100-40GB GPU nodes. A single EVA fine-tuning for one task takes roughly hour an A100.
E.3 Hyperparameter search
In line with previous experiments, we tune the rank for LoRA, DoRA, AdaLora and EVA, rank E {2, 4, 8, 16). Further, sweep over the same learning rates for the GLUE tasks.
E.4 Additional results
In Table 24, we show the full comparison of all the methods on CW10. EVA+DoRA consistently outperforms all competitors for the different rank budgets.
36

---

Table 19: Category, train size and classes of the VTAB- 1K dataset.
Category
Dataset
Train size
Classes
Natural
Ca (Fei-Fei et al., 2006)
3060
102
Natural
CIFAR-100 100 (Krizhevsky, 2009)
50000
100
Natural
DTD (Cimpoi et al., 2014)
3760
47
Natural
Flowers 102 (Nilsback & Zisserman, 2008)
2040
102
Natural
Pets (Parkhi et al., 2012)
3680
37
Natural
Sun397 (Xiao et al., 2010)
87003
397
Natural
SVHN (Netzer et al., 2011)
73257
10
Specialized
EuroSAT (Helber et al., 2019)
21600
10
Specialized
Resise45 (Cheng et al., 2017)
25200
45
Specialized
Patch Camelyon (Veeling et al., 2018)
294912
Specialized
Retinopathy (Kaggle & EyePacs, 2015)
46032
Structured
Clevr/count (Johnson et al., 2017)
70000
Structured
Clevr/distance (Johnson et al., 2017)
70000
Structured
dSprites/location (Matthey al., 2017)
663552
16
Structured
dSprites/orientation (Matthey et al., 2017)
663552
16
Structured
SmallNORB/azimuuhh (LeCun al., 2004)
36450
18
Structured
SmallNORBBeeleatiion (LeCun et al., 2004)
36450
Structured DMLab (Beattie al., 2016)
88178
Structured
KITTI/distance (Geiger et al., 2013)
5711
4
Table 20: Fine-tuning DINOv2-g/14 on the VTAB-IK benchmark. Best average performance is
highlighted in boldface. We report average accuracy across five seeds.
Natural
Specialized
Structured
Cifar
Caltech101
DTD
Flower102
Pets
SVHN
Sun397
Camelyon
lauroSAr
Resise45
Retinupathy
Clevr-Count Count
Clevr-Dist
DMLab
KITTI-Dist
dSpr-Loc
dSpr-Ori
SNORB Azim
-N
Average
Clevr-
FFT
73.1 89.7
78.4 99.7
92.2
89.5
55.5
74.8
95.0
88.2
70.5
64.2
63.6
68.8
92.0
64.3
50.2
56.8
76.8
LoRA
85.9 92.2
82.2
99.7
94.5
64.1
63.6
88.8
92.6
76.6
97.7
65.3
62.1
83.6
90.6
63.0
37.1
52.3
78.4
AdaLoRA
85.4 92.5
81.4
99.7
95.2
90.5
62.2
87.1
96.4
91.2 76.6
94.4
64.4
60.3
83.7
85.4
61.0
32.9
46.0
78.2
PISSA
85.5 93.6
82.3
99.7
94.6
92.8
62.3
87.1
96.6
91.9 76.3
95.0
66.3
63.2
84.9
90.5
60.1
36.3
48.6
79.4
OLoRA
85.5 93.0 82.1
99.7
95.1
78.3
62.1
86.7
96.3 91.9 76.8
94.3
66.0
62.4
71.3
89.0
60.9
34.3
49.5
77.6
EVA
85.6 93.9
82.2
99.7
95.9
93.2
63.6
92.3
65.1
83.3
91.4
55.0
DoRA
85.9 92.7
82.1
99.7
95.2
34.4
61.4
88.6
96.8
92.4
76.8
97.6
65.4
62.7
84.4
43.2
63.1
37.8
52.6
74.4
EVA+DoRA
86.2 92.1 81.9
99.7
94.99
93.8
62.4
88.3
96.6
92.6 76.7
97.2
65.5
54.1
83.7
93.3
62.3
37.5
$4.5
79.6
Incremental SVD convergence analysis
For simplicity, assume that A X and B -XiT two batches of activations for the weight matrix W obtained by passing two subsequent batches of downstream data through the model.
The aim is now compute the SVD of the concatenated activation matrix [AB] U' L in constant memory. Further, we obtain A = U.E,V," via SVD. Now let B be the component of B that is orthogonal to U, which can be obtained by QR decomposition or by = orth(B- Bach(BUUL TU B). where orth(-) performs orthogonalization. Then the SVD of the concatenated activation matrix can be expressed in partitioned form as
AB =
UB
Σ
U
V
(7)
B
0
37

---

Table 21: Standard deviations for the VTAB-IK results (Table 20) over seeds.
Natural
Specialized
Structured
SNORB- Azim
Cifar100
Caltech101
DTD
Flower102
Pets
SVHN
Sun397
Camelyon
EuroSAT
Resise45
Retinopathy
Clevr-Count
Clevr-Diss Dist
DMLab
KITTI-Dist
dSpr-LLc
dSpr-Ori
SNORB Ele
Average
FFT
1.5 1.1 1.6
0.0
0.4
1.2
0.9
14.9
0.4
0.6
2.7
1.7
0.9
1.2
23.6
0.5
0.4
1.6
1.9
3.0
LoRA
0.2 0.4
0.2
0.0
0.3
36.4
0.1
0.5
0.3
0.1
0.4
0.2
0.3
0.5
1.2
0.4
0.4
0.4
2.3
AdaLoRA
0.0 0.2 0.4
0.0
0.1
0.4
0.1
0.3
0.3
0.2
0.3
0.3
0.2
0.3
0.8
0.8
0.3
0.3
0.4
0.3
PISSA
0.2 0.4 0.3
0.0
0.2
0.5
0.2
0.7
0.2
0.1
0.4
0.3
0.4
0.2
0.7
0.3
0.5
0.4
0.5
0.3
OLoRA
0.3 0.3 0.4
0.0
0.3
29.4
0.1
0.3
0.1
0.2
0.2
0.5
0.1
0.3
24.6
0.3
0.8
3.1
EVA
0.2 0.5 0.2
0.0
0.1 0.3
0.1
0.3
0.2
0.3
0.4
0.5
0.3
0.6
0.6
0.5
0.5
0.2
0.5
0.3
DoRA
0.1 0.2 0.5
0.0
0.2
29.7
0.4
0.7
0.1
0.2
0.4
0.4
0.3
0.3
0.6
36.2
0.5
0.3
0.3
3.8
EVA+DoRA
0.2 1.3 0.6
0.0
0.3
0.5
0.3
0.4
0.2
0.3
0.3
0.4
0.4
12.8
1.3
2.5
0.3
0.6
0.6
1.2
U
B
By setting R
we can obtain SVD of the concatenated activation matrix by
BB
performing SVD R.R UEVT, which is constant in time and memory only need to compute U" and E', which do not scale with the number of data samples. Hence, we perform
B]
U (U:B
U E
VT
VT
V
0 1 1
(8)
and subsequently obtain U' UB| Ú and E' = E.
As this algorithm incrementally updates the and components, we need to keep track of changing mean and variance estimates. For the mean, this is trivial, but the computation of running variances can introduce numerical instabilities. To counteract this, young and cramer update is commonly employed (Chan et al., 1983). The supporting proof that the covariance matrix of the original data matrix is equal to the covariance matrix the concatenated matrix up to a constant factor is given in Ross al. (2008). In example, the lent-singular values U do not scale with the number of samples. However, in our case we have therefore it is the right-singular values V that do not depend the number of samples and can be incrementally updated in constant time and memory. We show pseudocode for the incremental SVD algorithm in Algorithm 2. In the following sections, analyze the behavior of this algorithm under different conditions, i.e. different batch sizes, etc.
X; and B = X; 1. i.e. transposed data matrices,
Complexity
The SVD computation introduces computational overhead in the initial training stage. Since do not require gradient computation or storing optimizer states, there no overhead in terms of memory. SVD has a time complexity of O O(min(b"d d, bdf)) that <<< d by performing truncated SVD Halko et al. (2011). Let T be the number minibatches until all components
be reduced O for
converged for N weight matrices, then the time complexity is ONNTk25) In other words, the complexity seales linearly with the number of weight matrices and the number of minibatches. To speed up the computation of SVD, we provide implementation that runs entirely on GPU.
F.2 Batch Size invariance
We perform an analysis of the convergence of the components obtained via SVD. Specifically, investigate the difference in components according to cosine similarity across different batch sizes. Previously, have seen that the components obtained across different batch orderings are heavily
38

---

Table 22: Dataset statistics for all MT40 tasks from Schmied et al. (2024).
Task
|8|
A
Success Rate
Reward
assembly-v2
0.0
1206.9
basketball-v2
39
0.9
1375.95
bin-picking-v2
0.0
474.81
box-close-v2
39
0.0
759.15
buttor-prees-ttopion--2
1299.2
buttoor-press-tooddww----l-----
39
1,0
1296.16 16
button-press-v2
1430.44
button-ppress-wall--2
39
1.0
1508. 16
coffee-button-v2
1499.17
coffee-pull-v2
39
1.0
1313.88
coffee-push-v2
39
0.6
508.14 14
dial-tunn-v2
39
0.8
1674.29
disassemble-v2
39
1.0
1396.55
door-close-v2
39
1.0
1535.4
door-lock-v2
39
1.0
1712.65
door-open-v2
39
1.0
1544.32
door-unlock-v2
39
1.0
1733.64
drawer-close-v2
1.0
1845.92
drawer-open-v2
39
1.0
1710.65
faucet-open-v2
0.9
1727.98
hand-insert-v2
39
1.0
1607.17
handle-press-v2
1.0
1854.79
handle-pull-side--22
39
1.0
1613.72
handle-pull-v2
1581.75
lever-pull-v2
39
1.
1449.05
peg-insert--iide-22
1545.19 19
pick-out-of-hole--22
39
1.0
1435.64
pick-place-v2
0.0
6.59
pick-place-wall-12
39
0.1
702.59
plate-slldeebbak--sidd---
39
1.0
1766.24
plate-slide-back-v2
39
1.0
1773.56
plate-slide-side-1
1.0
1663.35
plate-slide-v2
39
1.0
1667.35
reach-v2
39
1.0
1858.99
reach-wall-v/
39
1.0
1831.14
soccer-v2
39
0,4
445.84
stick-push-uy
39
1.0
1470.71
sweep-into-v2
39
1.0
1761.69
sweep-v2
1.0
1458.35
vindow-open-v2
39
1.0
1537.59
Average
0.84 + 0.34
1414.62 + 439.39
39

---

Table 23: Results for single task fine-tuning experiments on the Meta-World benchmark. We report
mean success rates and standard eπror across three seeds for every task.
window window-close close
faucet-close
hammer
handle- press
push-back
push-wall
shelf-place
stick stick-pu pou
Average
push
FFT
1.02.0
0.97
1.0_.0
0.77
0.87
1.0g.0
1.0..0
1.0
0.63
1.0=.0
0.92
LoRA
1.04.0
1.0..0
1.0..0
0.6
0.63
1.0..
1.0m
1.0
0.4
1.0..0
0.86
AdaLoRA
1.04.0
0.97
100..
0.4g.09
4.1
0.97
0.97
4.03
1.0..
0.13
1.0..0
0.80
PISSA
1.02.0
1.0..0
1.0_..
0.43
0.57
1.0..0
1.0..0
1.0
0.53
1.0=.0
0.85
OLoRA
1.04.0
0.97
40.00
1.0..
0.57 4:01
0.63
1.0g.0
1.0_.0
1.0+.0
0.6
1.0..0
0.88
EVA
1.04.0
0.97
1.0..
0.63
0.77 .05
1.0
1.0
1.0..0
0.63
L.0..0
0.90
DoRA
1.0+.0
1.0
1.0_.
0.6
1.0
1.0g.0
1.0_.0
1.0+.0
0.67
L.0..0
0.93
EVA+DoRA
1.0+.0
1.0..0
1.0..
0.84.08
1.0
1.0g.0
1.0
1.0+.0
0.63
1.0=.0
0.94
Table 24: Rank-wise comparison for all methods on CW10. We fine-tune a 12M DT on 10 tasks individually and report the mean success rates/rewards (#t standard error) for every task.
handle press sile
pag wwg.unniugeeoee side
window-ching clese
Fancst-llow
hommmer
back
pustowall
shelf place
stidk pull
Avenge
push
Method
Rank
EFT
0.97
0.93ga0s
1.0000
0.6amos
0.7am
1.0g0a
0.93
1.0g00
0.57ammr
1.0g00
0.87amm
LoRA
2
L.00a
10000
1.000
0.6300s
0.57amm
0.97.0ms
0.03.0s
1.0g
0.37aa
1:10.0
0.84
L.000
0.97ga00
1.0.00
0.4720
0.63ge.
0.97_00
1.0g.
0:23
1.0.00
0.83
8
L00oa
0.97
1.000
0.43
0.4amm
0.97_6.a
0.93
0.231011
1.0000
0.79
1.79
16
10saa
0.97
1.0.00
0.43amm
0.47amm
1.00a
0.97.0.0
1.0..
0.4g0m
1.0.00
a.82
DoRA
L00aa
1000n
10000
0.57
100aa
10
10.a0
0.33
10:00
0.80
1.0aaa
1.0ann
100aa
8.6an112
180aa.
1.000
100an
0.43
1.0a00
0.9ano
8
1.0400
1.0a0n
1.0.00
0.4720122
0.332mm
1.0000
1.0gpn
1.00a.
0.57an15
1.0au0
0.9gm
16
L'goe
1.400
1.0g0
0.57
1.0ga
1.000
1.0o0
1.0a9
0.07a015
1.0000
AdaLoRA
0.97
1.00o
0.37amm
0.37amm
0.90_mm
0.9720s
1.0ga
L.s0a
0.97aa.c
1.000
0.37
0.37a0.
0.97-mm
0.9
1.0a.
0.13
10000
0.700nnos
8
L.aan
0.97aace
1.0gn0
0.3anm
0.57mm14
0.90
0.87aar
1.0ga.
0,0ap0
1.0a00
2.70
16
L.0oo
0.97
1.0g00
0.4amm
0,57
0.972mmx
0.93
1.0ga
0.0aau
1.2000
0.78am00
OLoRA
1.0sp
0.47a0ms
0.33ams
0.97
0.97um
0.27au1:
10000
0.79amm
0.9
0.482mm
0.60
1.0aa
1dlae
10000
0.6
1.0000
0.86:200
L0gge
0.97gaia
10gao
0.57
0.5.0e
10gaa
18ka
100g0
0.53
10g00
1.80lanor
16
L.0.a
0.97
1.0000
0.4
0.83
10jaa
1.6pa
1.0..0
0.43
1.0.00
0.81
PISSA
2
L'0oa
0.97
1.00a
0.48
0.50.am
0.97 ooc
0.9ha
0.33
10.000
D.81
1.0aa
1.0.00
1.0.a
0.37
0.7.nm
0.97
1/kba
0.07
10000
0.81
1.040a
0.97eacs
10000
0.3es.
0.57
0.97-am
0.53ama
1.0a00
16
1.0g0a
0.3
0.33
0.47a0m
1.0ga
0.77ams
1.0g0
0.47
1.0g00
EVA
2
1.D40a
0.97
1.0g00
0.43400
0.77amm
0.97enx
1.0g00
1.0ea
0.63
1.0au0
1.88
4
L00oa
0.97
10j0a
0.48
0.47,012
1.00a
0.97.6.0
0:23100:
1.0000
0.81
8
0.97aa.05
1.0aa
0.63aam
0.7anns
10jaa
0.23
a.85
L0s0a
0.97
1.00p
0.584am
0.77xum
1040a
10g00
100a
0.0epe
10g00
0.83
EVA + DaRA
2
LOsna
1.0ann
1.0aa
0.5aums
0.97amm
1.0ga
10gan
1.000
0.43
1.0a00
a.92
1
L0soa
1.0on
10000
0.8amm
0.93aa
1049a
10
10kaa
0.63amms
1.2o0
8.94ano
8
L'0oa
10000
10ga0
0.63.a
(.87
10
100a0
140g0
0.77m0
10200
2.01
16
1.000
0.67_ax
14ga
10gaa
10gan
10ga
10000
0.92gnor
correlated. In Figure 11 (right), we visualize the cosine similarities between the SVD components for different batch sizes, namely 4, 8, 16, and 32 for Llama-2-7B on the MetaMathQA dataset. We observe that the components correlate strongly and remain mostly invariant to the batch size. This indicates that smaller batch sizes may be used for obtaining the initialization, which results in less computational overhead. In the case of Llama-2-7B on MetaMathQA, this means that we can use batch size of 4 since it induces a computational overhead of around 100 seconds. Afterwards, we can continue the fine-tune fine-tuning process with larger batch size.
40

---

Incremental SVD convergence
3500
Number converged components
3000
2500
2000
1500
4
1000
8
500
16
32
100
200
300
400
Time in seconds
Figure 11: Time in seconds until convergence of incremental SVD components for different batch sizes for Llama-2-7B on the MetaMathQA dataset. The dashed line indicates the total number of components.
F.3 Convergence speed
The data-driven initialization of EVA relies on incremental SVD on minibatches of activations in the initial training stage. Figure 11, we show that this process converges for Llama-2-77 on MetaMathQA for different minibatch sizes. Using a minibatch size of 4 the computation for EVA's initialization lasts for approximately 80 seconds, which corresponds to around 90 minibatches. For batch size of 32 the computation of the SVD components takes around 500 seconds.
E4 Batch order invariance
In Figure 12 left, we additionally show that the main components obtained via SVD mostly remain consistent across different batch orders for a batch size of 4, again for Llama-2-7B on MetaMathQA. To this end, we plot the cosine similarity between components obtained via incremental SVD after rank redistribution. These results indicate that these models exhibit certain activation patterns that remain consistent across different batch orders, which leads a robust initialization for EVA.
F,5 Excluding ignored tokens for SVD
For some datasets we notice that masking out tokens for the SVD computation which are ignored for the loss calculation during fine-tunine be advantageous. However, this result in significant reduction of the effective batch size for SVD if the number of completion tokens is small. An example where this is the case in our experiments is the common-sense reasoning tasks which have long prompts, but completion tokens are only one word per sample. This setting can lead to cases where SVD does not converge for lower batch sizes. We therefore do mask out the prompt tokens in our experiments. Another setting where masking ignored tokens can be advantageous is multi-turn conversation where the model only trained on the assistant tokens. To achieve the results in Table 13 we mask out user tokens together with the prompt for the SVD computation.
G Rank redistribution analysis
To illuminate the rank redistribution process, we visualize the resulting ranks for each weight matrix after SVD for Llama-2-7B on the MetaMathQA dataset for different values of p. Setting p results in uniform rank distribution as standard LoRA. However, setting p >>> alters the number of ranks per weight matrix. In Figure 13 visualize the number of ranks assigned to each weight
41

---

Awerago seedo)
Specdral norm of difeeences beawein inkia iattion and findl adagoars
20
Lama 2-7b-tt
Meta
LoftA
00
Evs.
06
Spacti
130
04
100
02
1
0.0 Down
Gate
0
Weiglt mraoicoss
34 12 20.1 22.00.5 24
Conserents
Figure 12: Left: Average cosine similarity between SVD components across 10 random seeds for permuting the batch order. The first 10 components remain mostly consistent across all permutations. While the remaining components vary, they strongly correlate with each other. Right: Average spectral norm of difference between weight matrices at initialization and after training for LoRA and EVA applied to Llama-2-7B, Llama-3.1-8 and Llama-3.1-708 EVA's initialization is closer to the final adapter than LoRA's.
Number of ranks per layer for different values of mho
11
1.2
1.3
22
26
18
18
1.9
25
12
80
88
96
100
1004
108
111
136
136
140
144
148
152
156
160
184
168
172
176
180
192
196
200
204
208
212
216
Lajer
intex
Figure 13: The resulting rank allocation per weight matrix in each layer for Llama-2-7B the
MetaMathQA dataset with different values of The first row represents a uniform distribution where each weight matrix receives the same rank r 16. The most change occurs for 1.5. The redistribution converges for larger values of
matrix for different values of p > and in Figure 14 we visualize the corresponding deltas. Both visualizations clearly illustrate that the greatest change occurs for values of 1.5. Setting p to higher values results in less and less change. Interestingly, some ranks still change when going from 2.5 to p 3. Finally, conduct hyperparameter search in which search over different values of o E 1.1, 1.3, 1.4, 1.3, 1.6, 1.7, 1.8, 2,9, We report the results in Figure 15. We find that for Llama-2-7B MetaMathQA uniform distribution performs favorably. The second best performance shared by 1.5 and 2. Therefore, we always search for p = 1 and for all our remaining experiments when apply EVA and select the best performing one.
2,2.5,
H Supplementary Proofs
For convenience first repeat both Theorem 3.1 and Theorem 3.2 and provide proof afterwards.
42

---

Number re-distributed ranks per weight matrix for different values
11
12
Values tho
-1
23
-2
1.8
1.
25
-4
20
96
40
44
48
62
58
80
1000
104
100
112
1.32
136
140
144
7.8
152
156
160
164
168
172
192
196
200
204
209
212
216
220
Figure 14: Deltas between rank distributions per weight matrix each layer for Llama-2-7B on the
MetaMathQA dataset with different values of p. The first represents uniform distribution where each weight matrix receives the same rank r = 16. The most change occurs in the range E [1, 1.5]. Larger values of p do not induce additional significant changes the rank distribution.
Llama-2-7B on MetaMath
0.6
0.62
0.597
0.608
0.594
0.61
0.597
0.599
0.6 0.607 0.61 0,607
Accuracy
0.4
0.2
0.1
0.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
2.5
3
Rho
Figure 15; Accuracy for different values of when fine-tuning Llama-2-7B the MetaMathQA dataset.
43

---

Optimality of SVD-Based Initialization for Maximizing Explained Variance
Let X E Rexd be matrix of activation vectors obtained from pretrained model, where b is the number of samples and d is the feature dimension. Suppose we wish to adapt a weight matrix W Rexd using low-rank update of the form AW = BA, where B E IR A R' and r < min(k, d). Let X = UEV be the singular value decomposition (SVD) of the activation matrix with 01 02 2 ... 2 0 being the singular values of Σ. Then the top right singular vectors
y E Rddxr solve the following optimization problem:
arg
max
x
VERaXr V=
and also minimize the Frobenius norm reconstruction error:
V arg
min
X M M
M Mank Sr
Hence, v, forms the optimal basis for capturing the maximum variance of activations under rank-r projection.
Proof.
First, we define the empirical covariance matrix of activations
X
(9)
72
with V being the eigenvectors of S. Projecting X onto a subspace spanned by the orthonormal basis V Raxr the total captured variance is
Varv Tr(V X XV) TT(V SV).
(10)
The trace is maximized when V contains the eigenvectors corresponding to the top eigenvalues of S, which are the top right singular vectors of X. This is verified by the Youuar-YYuug--Maa theorem (Eckart & Young, 1936). The best rank-r approximation to X in the Frobenius norm is given by
X, Σ i
(11)
with 01 2 02 2 ....2 O, which uses the top r right singular vectors. Hence the Eckart- Young theorem directly proves both reconstruction with respect Frobenius norm, as well maximizing the trace operator.
Gradient Signal Amplification via EVA Initialization.
Let AW BA be low-rank adaptation a pretrained weight matrix W g where B E Axr A E iRr and < min(k, d). Let æ E IRd be the activation input to this layer. Assume activations æ are drawn from a distribution with covariance matrix Σ = [[xxx" J. Then initializing A
with the top right singular vectors of a sample activation matrix X E px maximizes the expected squared gradient norm:
OL
E
α Tr(A T ΣEA)
aB
Proof.
Let us consider the forward pass
ÿ (W + BA)x
(12)
with loss function L(g. y) and target y. The gradient with respect to B is
ac
8c v A gx
A
(13)
0B
Oy
The squared Frobenius norm of the gradient is
0L 2
Tr(Ar(g
T))= A
[g Tr(AAr AT)
(14)
aB
44

---

Since BA is initialized to be BA = 0 at the beginning of fine-tuning and the gradient g is entirely governed by the behavior of the pretrained model, we can make the assumption that the gradient statistically independent of the input (a: g) and
Cov(#. g) E(æg
[[x] Eig '1
O.
(15)
Hence, by taking the expectation over æ with E ] = Σ we obtain
2 0C 0B
E
x Tr(AEAT).
(16)
Again, the trace is maximized when the rows of A are aligned with the top eigenvectors of Σ, that is, the principal directions of the activations, as proven by the sckart-Young theorem Eckart & Young (1936).
Algorithm 2 Incremental SVD algorithm from Ross et al. (2008)
Input: Sequence of data batches (A"..., A"), }. truncated SVD SVD(-), orthogonalization function orth(-), running variance update functionn young roung_cramer, ewweeerrudaatte (.. Eiso (A., m²)²
1: m + 5, >
A.;: σ"
initialize incremental mean/variance
2: UgEo V +- (VD(A" a²)
Perform initial SVD on to get initial components
3: for i biinn..... ,....,, do
4: at < E. A mi + m' + a² a'-m- 2-
D compute mean vectors > Update running variance
b(++1)
5: o' young ceamerrrrarr..m update AA)
6:
Á
b(++) A al; 26
m' @t)
D. concatenate mean correction factor
7:
Á
('t
U,-IU,,,A A
Obtain orthogonal component U
Σ1-
T
8:
R
Define matrix R
0
A'A'
9:
ÚΣV
+ -S)D(R)
Perform SVD on R
10: U.+ U--1; A
Ü, Σ E
Update SVD components
11:
end for
I Ablation Studies
Finally, we conduet ablation studies EVA investigate important factors that contribute to its performance. Specifically, we investigate the impact of scale and direction. To this end, we use the VTAB-IK dataset because comprises diverse of tasks and allows for a systematic investigation on in-domain (natural) and nuttof-distribution (specialized and structured) data. We report results for our ablation studies in Table 25 and explain the different settings in the following paragraphs.
Effect of scale. To investigate the effect of scale on initialization, we add a setting that uses whitening (EVA-whiten). Whitening scales the initialization by the reciprocal of their eigenvalues, which alters scale, but preserves directions. We found that whitening can significantly improve performance in structured -uttoo-diistributio)) tasks, leading slightly higher average score than EVA. This indicates that scale is especially important for structured data. However, EVA-whiten experiences a slight performance drop natural and specialized tasks.
Effect of directions. To address the importance of the directions of the components, we randomly permute its rows (EVA-perm). This preserves scale while corrupting directions and the lg norm of A. Additionally, we add setting where we randomly rotate A (EVA-rot), which preserves the & norm but alters directions. We find that altering directions leads to drop in performance structured tasks, while changing the l, norm leads to drop on natural tasks. Both EVA-perm and EVA-rot lead worse average performance across all tasks compared to EVA.
Effeet of rank redistribution. We conduct an experiment which we randomly initialize A after performing rank redistribution (LoRA redist). This setting gives insights on the effect of the
45

---

redistribution and whether its benefits are bound to EVA. Redistribution has a positive effect on LoRA on natural tasks, but negative effect on both structured and specialized tasks. This illustrates that rank redistribution is most beneficial in combination with EVA's initialization of A.
Generally, can say that EVA performs partic- ularly well on natural images and whitening can enhance its performance on out-of-distribution images. The decisive factor with respect to this improvement seems to be a controlled change in the scale of initialization induced by the sin- gular values. Therefore, by changing the scale in controlled manner, we can make EVA more compatible for different kinds of data. The re- sults for EVA-perm confirm that the scale is the decisive factor for initialization.
Table 25: Group-wise averages DINOv2-g/14 ablation studies on the VTAB-IK benchmark.
Method
Nat.
Spec.
Struct.
All
LoRA
83.2
88.8
69.0
78.4
LoRA-redist
87.3
88.0
68.2
79.4
EVA-whiten
87.5
87.5
69.1
79.8
EVA-rot
87.7
88.0
68.2
79.6
EVA-perm
87.4
87.8
68.3
79.5
EVA
87.7
87.9
68.6
79.7
Further Discussions
Alternative data-driven initialization schemes. We investigated alternative data- driven initialization schemes such as Kernel-PCA (Schölkopf et al., 1997) Linear Diseriminant Analysis (Fisher, 1936, LDA). Kernel-PCA can account for non-linearities in the data but scales with the number of datapoints, which impractical. For LDA, observed convergence instabilities during incremental updates. In our setting we deal with sequences, therefore the number of datapoints grows fast, making Kernel-PCA impractical. LDA projects the data onto a subspace that maximizes linear separability between classes. Such an initialization scheme may be particularly interesting for classification tasks like GLUE VTAB-IK.
Additional latency of SVD. EVA leads to performance improvements over LoRA, but introduces additional latency at the beginning of training to compute the initialization. In Figure 5 (left) demonstrate that this process constitutes merely 0.2% of the actual training time for Llama-2-7B MetaMathQA. In addition, in Appendix F we show that this process is largely invariant to the batch size and order, meaning smaller batch sizes may be used, resulting in additional speedup. Since the SVD computation does require backpropagation and storing of optimizer states, there is memory overhead.
How to initialize B? We follow Hu et al. (2022) and initialize B = 0. All other initialization methods initialize & O, which requires altering the pre-trained model weights. our experiments, EVA usually outperformed CorDA and LoRA-GA, even though they are both data-driven and leverage similar information. Therefore, setting = 0 could also be driving factor for improved performance. We leave this investigation future work. Finally, restoring the base model after fine-tuning requires computing the delta of the weights before and after training for B * 0. In contrast, EVA and LoRA fully restore the base model's weights by simply unloading the adapter weights.
46

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction methodically detailed in Section 3 (Method) and empirically supported across diverse domains (language, vision, RL) in Section (Experiments). Additional results be found in the appendices.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section (Discussion and Limitations) explicitly discusses several limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
47

---

Answer: [Yes]
Justification: Theorem 3.1 (Section 3.2) and Theorem 3.2 (Section 3.3) clearly lay out their assumptions regarding activation matrices, weight matrices, low-rank updates, and activation distributions within their respective statements. Proofs for these theorems are included in Appending H
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper specifies datasets, models, and the core mechanisms of proposed EVA method, including crucial parameters for SVD initialization (e.g. 7) and adaptive rank redistribution (p). Standard training hyperparameters such as learning rates etc. are also outlined for each experimental domain. Further detailed configurations essential for reproducing EVA's claimed performance, efficiency, and faster convergence are outlined in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
48

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the source code along with the submission (see Appendix A) to ensure reproducibility. In addition, we will integrate EVA into the widely used PEFT library.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (uttps://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. ctaiss.cc/ublii. /wwp//gu/s/////eg/// CodeSubmi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper outlines the experimental setup clearly in the main text, covering models, datasets, core parameters, and evaluation metries. For full reproducibility provide detailed configurations for each experiment the appendices.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: experiments are conducted over multiple seeds. We report mean and standard deviation for each experiment.
49

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: For all experiments we provide information about the number of gpus, type of GPU used well as estimated execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerrpp.eeeee httpc//llcc////eeeeeeee/eelllclt weallic/Ettae e/Ethi csGuidel ines?
Answer: [Yes]
Justification: The research conforms to the NeurlPS Code of Ethics. The paper transparently presents the EVA methodology using public datasets and models, aiming for reproducible science and contributing positively through more efficient model fine-tuning.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
50

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The paper discusses societal impacts by emphasizing positive contributions. such as making foundation model fine-tuning more efficient and reducing computational resouree requirements. These improvements inherently address the negative societal and environmental concerns tied to the high operational costs and energy consumption of large- scale AI. By presenting solutions that enhance accessibility and sustainability, the work contributes to a more responsible Al development landscape.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release any pre-trained models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
51

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's ' creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not conduct crowdsourcing or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
52

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not conduct crowdsoureing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https http://ipp...ccmmeeeeeeeeeeeeeee for what should or should not be described.
53


Reviews:
[{"review_id": "ock5Cn41mj", "reviewer": "Reviewer_6shx", "summary": "The paper proposes a variant of Low Rank Adaptation (LoRA) for finetuning foundation models. The basic idea, termed Expected Variance Adaptation (EVA), is to perform an SVD on the activation vectors that are obtained/estimated from minibatches of the downstream task. This is similar in spirit to methods that use the gradients to estimate the importance. The authors show that this process leads to a \"more stable\" gradient signal. They also propose a rank allocation procedure that tries to distribute a given rank budget across weight matrices in order to maximize the explained variance. The paper does experiments on finetuning LLMs on math+reasoning taks, as well as results for image classification and decision making tasks. The main goal of the experiments is to report the parameter reduction, while preserving accuracy.", "questions": "Please address the questions of novelty and comparison that were raised in the the \"weakness\" section of the review.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The method proposed is natural, and the experiments are fairly thorough in terms of the domains that they cover.\n- The writing of the paper is very clear and the ideas are well explained.\n\nWeaknesses:\n\n- Unfortunately the paper is not very novel. The idea is very similar to the gradient based importance metrics. While the paper says that this is \"provably\" good initialization at several points, this is because of the way the objective is set up-- the algorithm is basically SVD, and is optimal in terms of capturing the variance. (Why is capturing the variance of the activations or aligning with the gradient the right objective?)\n\n- The experimental section is reasonable, but this has become a really crowded field over the last couple of years, so the paper needs to compare with a bunch of newer work on PEFT. (DoRA, MiLoRA). There are also \"activation guided\" methods that are really similar to this paper, e.g., AG-LoRA (https://ieeexplore.ieee.org/document/10852296), GLoRA (https://openreview.net/forum?id=K7KQkiHanD ) that the authors should compare to.\n\nOverall, while the writing is clean, I am concerned about the novelty with this submission.", "was_revised": true, "final_justification": "The authors did a thorough job of answering reviewer concerns and comments. They also realized a connection between their proposed gradient based finetuning and the neural tangent kernel. This connection is perhaps interesting, but unfortunately they don't work out the implications of this in a formal enough way, imo. I have raised my evaluation slightly, but I'd like to hear from one of the other reviewers.", "rebuttal": "Dear reviewer 6shx,\n\nWe would like to thank you for your valuable feedback.\n\n**Novelty**\n\nWe acknowledge that adaptive rank allocation and data-driven initialization have been explored in concurrent works (e.g., CorDA, LoRA-GA). However, we strongly believe that novelty should not be judged solely by whether single components used in a framework already existed, but by whether a method offers a new perspective or principled formulation that provides novel insights. That said, EVA offers a theoretically grounded approach: both initialization and rank allocation arise from maximizing expected gradient signal under a parameter constraint. This goes beyond heuristic initialization design (CorDA, LoRA-GA) or soft constraints during optimization (AdaLoRA). By proposing EVA we provide insights into the design of maximizing gradient signal without relying on hyperparameters for computing the initialization as in CorDA/LoRA-GA which allows to systematically control for rank redistribution while providing strong performance with a reduced parameter count. We believe all of these constitute a meaningful and novel contribution to the field.\n\nWe would furthermore like to highlight the practical efficiency gains resulting from our approach which is quantified in Table 3. For fine-tuning a Llama-2-7B model on a single gpu, EVA’s initialization takes only 0.7% of the total training time, whereas the data-driven methods LoRA-GA and CorDA require 2.4% and 4.5%, respectively.\n\n\n**Additional Baselines**\n\nWe agree that the PEFT landscape is rapidly evolving. Therefore, we added a wide range of initialization methods as baselines.\n\nDoRA:\n\nAs mentioned in line 224, we already compared EVA to DoRA on common sense reasoning (Appendix B, Table 8), mathematical reasoning (Appendix B, Table 9), vision tasks (Appendix E, Table 21), and decision-making tasks (Appendix F, Table 24). Our results show that EVA is consistently competitive. Furthermore, we demonstrate that EVA is complementary to DoRA; the combination (EVA+DORA) frequently achieves the best performance across different tasks (Appendix B, Tables 8,9; Appendix F, 24), highlighting the unique and orthogonal contribution of our initialization scheme.\n\nMiLoRA:\n\nWe agree with the reviewer that adding MiLoRA is an important baseline due to its similarity to PiSSA. We therefore created an integration in the PEFT library [1] for it and trained it across common sense reasoning as well as math tasks. We present the results together with DoRA results in tables R1 and R2 below.\n\nTable R1: Results on Common Sense Reasoning (mean and standard deviation across three seeds). We report results for rank 16.\n| **Model** | **Method** | **BoolQ** | **PIQA** | **SIQA** | **Hella Swag** | **Wino- grande** | **ARC-e** | **ARC-c** | **OBQA** | **Avg.** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Llama-2-7B | MiLoRA | 65.0 (±3.9) | 84.8 (±.4) | 82.3 (±.3) | 94.9 (±.1) | 84.5 (±.8) | 88.2 (±.1) | 74.9 (±1.1) | 85.3 (±.2) | 82.5 |\n| Llama-2-7B | DoRA | 68.3 (±2.6) | 85.1 (±.3) | 82.2 (±.5) | 94.9 (±.1) | 84.3 (±.2) | 88.7 (±.2) | 74.8 (±.5) | 86.3 (±.5) | 83.1 |\n| Llama-2-7B | EVA | 68.3 (±8.0) | 85.3 (±.1) | 82.9 (±1.1) | 95.2 (±.1) | 85.2 (±.8) | 88.6 (±.5) | 75.8 (±.4) | 86.3 (±.6) | **83.4** |\n| Llama-3.1-8B | MiLoRA | 85.7 (±.5) | 90.8 (±.2) | 83.0 (±.3) | 96.8 (±.1) | 88.8 (±.6) | 94.4 (±.4) | 84.9 (±.9) | 90.5 (±.5) | 89.4 |\n| Llama-3.1-8B | DoRA | 86.2 (±.2) | 90.8 (±.1) | 83.4 (±.3) | 96.9 (±.0) | 88.6 (±.3) | 94.3 (±.1) | 84.9 (±.7) | 89.4 (±.0) | 89.3 |\n| Llama-3.1-8B | EVA | 85.3 (±.4) | 90.4 (±.0) | 83.4 (±.2) | 97.0 (±.0) | 89.0 (±.2) | 94.4 (±.3) | 86.0 (±.2) | 90.3 (±.8) | **89.5** |\n| Gemma-2-9b | MiLoRA | 88.2 (±.3) | 93.0 (±.2) | 85.0 (±.3) | 97.8 (±.1) | 92.9 (±1.0) | 97.4 (±.2) | 90.2 (±.9) | 93.9 (±.1) | 92.3 |\n| Gemma-2-9b | DoRA | 88.3 (±.2) | 92.6 (±.4) | 84.9 (±.3) | 97.7 (±.1) | 92.2 (±.4) | 97.1 (±.1) | 89.9 (±.0) | 94.5 (±.5) | 92.1 |\n| Gemma-2-9b | EVA | 89.4 (±.3) | 94.6 (±.1) | 85.8 (±.1) | 98.3 (±.1) | 94.4 (±.3) | 98.0 (±.1) | 93.0 (±.0) | 95.9 (±.2) | **93.7** |\n\nTable R2: Results on Math (mean and standard deviation across three seeds). We report results for rank 16.\n| **Model** | **Method** | **GSM8K** | **MATH** |\n| --- | --- | --- | --- |\n| Llama-2-7B | MiLoRA | 59.7 (±1.4) | 11.2 (±.1) |\n| Llama-2-7B | DoRA | 59.8 (±.5) | 11.5 (±.2) |\n| Llama-2-7B | EVA | **61.9 (±.5)** | **13.1 (±.3)** |\n| Llama-3.1-8B | MiLoRA | 78.6 (±.1) | 30.3 (±.3) |\n| Llama-3.1-8B | DoRA | 77.9 (±.1) | 30.2 (±.5) |\n| Llama-3.1-8B | EVA | **78.8 (±.3)** | **31.2 (±.3)** |\n| Gemma-2-9b | MiLoRA | **83.7 (±.4)** | **41.9 (±.3)** |\n| Gemma-2-9b | DoRA | 82.5 (±.6) | 39.7 (±.4) |\n| Gemma-2-9b | EVA | 83.6 (±.8) | 41.5 (±.3) |\n\nAG-LoRA:\n\nThank you for pointing out this highly relevant work. Unfortunately, the code for AG-LoRA is not publicly available, and a custom re-implementation for activation-weighting and outlier-based global rank assignment may result in subtle discrepancies to the code run by the authors. To ensure a rigorous and reproducible evaluation, we prioritize baselines with available and verified code. We will add a discussion on AG-LORA in our related work section.\n\nGLoRA:\n\nGLoRA is orthogonal to EVA as it does not focus on initialization. It employs evolutionary strategies to explore adapter configurations, whereas the focus of our work lies in initialization of adapters and their ranks.\n\n\nIn summary, we have clarified the novelty of EVA by highlighting its theoretical grounding and efficiency gains over other data-driven methods. Furthermore, we have strengthened our empirical evaluation by incorporating new results for MiLoRA, demonstrating that EVA remains highly competitive in a crowded field. We believe these clarifications and additions substantially strengthen our submission, and we hope you will reconsider your assessment of our work's contribution. If any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion.\n\n[1] Mangrulkar et al., PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods, GitHub 2022"}, {"review_id": "GE8O4n0IDH", "reviewer": "Reviewer_G3kb", "summary": "This work proposes a novel initialization and allocation scheme for PEFT, which leverages information obtained from activations to initialize low-rank adapters into the most impactful directions (with highest variance). This leads to faster convergence and higher performance in a broad range of settings, compared to competitors", "questions": "The main concerns are reported in Weaknesses.\n\nOther question:\n1. In section 3.3., line 158, referring to A,B matrices, the authors mention “to preserve the base model weights, one of them must be initialized with zeros”. While zero-initializing one of the two matrices is the most common practice, it should be mentioned that other methods (such as [4,3]) utilize a residual approach (subtracting frozen version of the adapters).\n- In particular, did the authors test this type of initialization, which in theory would allow to also initialize the B matrix with SVD information?\n2. How does performance vary at different minibatch sizes? What is the minimum amount of data necessary to obtain significant performance improvements?\n\nI am willing to increase my score if the main concerns are addressed.\n\n[4] Meng et al. - PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, NeurIPS24", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The presentation is clear, and the method is well motivated\n- Extensive number of analysis and ablations\n\n\nWeaknesses:\n1. EVA leads to performance improvements, though it incurs an initial cost for SVD initialization. However, it is not clear how the performance of “SVD_init + EVA” compares to “random_init + LoRA” (or other methods) at the same time t, where t = 0 is defined as the time before the two initializations.\n2. While GLUE is extensively used for testing PEFT performance, reporting results on the validation set (as it is commonly done) is not best practice, as it is prone to overfitting. A more rigorous evaluation approach involving a separate validation/test split has been proposed in [1,2,3].\n\n\n[1] Wu M. et al., Advancing parameter efficiency in finetuning via representation editing, ACL 2024\n\n[2] Wu Z. et al., Reft: Representation finetuning for language models, NeurIPS 2024\n\n[3] Bini et al., DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation, ICLR 2025", "was_revised": true, "final_justification": "I carefully read authors response to reviews, and they address most of the concerns", "rebuttal": "Dear reviewer G3kb,\n\nThank you for the constructive feedback, it helps us to substantially improve our manuscript. We address the raised concerns as follows.\n\n**Computational  Fairness**\n\nWe would like to clarify that the initial cost of SVD is negligible compared to the fine-tuning cost, as shown in Table 3. We rehash the aspects of the computational overhead below. \n\n- Initialization Overhead is Minimal: The wall-clock time for EVA's SVD initialization is a very small fraction of the total training time. As shown in Table 3, for a 7B parameter model, this overhead is only 0.7% of the total training time, and we demonstrate it can be reduced to as little as 0.2% by using a smaller batch size for the SVD step. This change does not impact the quality of the initialization (Figure 10). We are confident that this minor time difference does not change the final performance of the baselines.\n- Faster Convergence and Evaluation Protocol: The small upfront time investment in our principled initialization is immediately compensated by significantly faster convergence. Figures 3 and 5 clearly show that EVA achieves a lower training loss much more rapidly than LoRA and other baselines. This means that at virtually any time t after the initialization phase, EVA has already reached a more optimal state. We clarified this point in the updated version of our manuscript.\n\n\n**GLUE Evaluation**\n\nWe thank the reviewer for this valuable suggestion regarding evaluation best practices. We agree that rigorous evaluation is paramount to estimate the generalization error. However, re-running all GLUE tasks with a revised data split was infeasible within the rebuttal period. \n\nWe would like to mention that for GLUE specifically, we applied the same comprehensive hyperparameter search protocol to all methods, including our own and all baselines. As detailed in Appendix C.3 for each method on each GLUE task, we tuned key hyperparameters (e.g., learning rates, LoRA ranks) and selected the best-performing configuration for that specific method. The final results reported in Table 2 therefore compare the optimal performance of EVA against the optimal performance of each baseline.\n\n**Questions**\n\n**Initialization of B matrix**\n\nThank you for pointing this out. It is indeed possible to initialize B as well in our setup, following works such as PiSSA or LoRA-GA. We chose the setup B=0 as there exists literature that this setting has favorable theoretical properties, such as stable convergence for larger learning rates [1]. Furthermore, there is no common consensus in the literature that one should be preferred over the other or that one has favorable properties over the other. Due to the amount of extensive experiments present in our manuscript, we aim to explore this effect in future work.\n\n**Minibatch Size**\n\nWe found our initialization to be robust across different minibatch sizes. Specifically, our analysis shows that minibatch sizes between 4 and 32 for the incremental SVD computation produce highly correlated components (Figure 10, right), leading to comparable downstream performance. We did not increase the minibatch size further to ensure that the memory bottleneck is determined by the main fine-tuning stage (i.e., by the LoRA rank and training batch size), not by the EVA initialization step. This setting makes EVA's overhead negligible, adding as little as 0.7% to the total training time for our default language generation setting for Llama-2-7B on common sense reasoning tasks (Table 3).\n\n**Minimum Data for Fine-tuning**\n\nThe minimum data required by EVA is determined dynamically by the convergence of principal components during incremental SVD. This convergence is controlled by a cosine similarity threshold, $\\tau$. A high value for $\\tau$ represents a stricter threshold for considering components converged, whereas a low value is less permissive. By default, we use $\\tau=0.99$. Even for this strict setting, the initialization for Llama-2-7B is highly efficient, completing in just 80 to 300 seconds (depending on the minibatch size) after processing fewer than 100 minibatches. However, we agree that an additional analysis on the performance impact of lowering $\\tau$ would be a valuable addition to the paper. We commit to including this analysis in future versions of our manuscript.\n\nMoreover, we agree that it would be interesting to study the impact of EVA when fine-tuning with different data budgets. While we are unable to provide this analysis during the time-frame of the rebuttal, we aim to investigate this in more detail in future work. We expect that the advantage of EVA can be particularly pronounced in low-data regimes, where the effect of initialization may have an even larger role for final performance. This hypothesis is supported by the loss curves presented in Figure 3, which show that EVA achieves substantially lower loss during the early stages of fine-tuning (e.g., at 1.5K updates). Consequently, EVA may prove to be particularly effective in such low-data scenarios.\n\nOur manuscript has improved substantially by addressing and incorporating your feedback. If any questions remain, we are happy to engage in further discussion. If you find our responses and additional results useful, we would appreciate it if you would consider changing your score.\n\n[1] Hayou et al., LoRA+: Efficient Low Rank Adaptation of Large Models, ICML 2024"}, {"review_id": "FjFatEjExt", "reviewer": "Reviewer_LBrt", "summary": "This paper introduces Explained Variance Adaptation (EVA), a novel initialization scheme for LoRA in parameter-efficient fine-tuning of foundation models. Unlike prior SVD-based initializations, EVA performs incremental SVD on activation vectors to obtain directions that maximize explained variance. These directions are used to initialize LoRA matrices, aiming to maximize the expected gradient signal at the onset of fine-tuning. The paper also proposes an adaptive rank allocation mechanism based on explained variance ratios. EVA is evaluated across various domains—language modeling, language understanding, image classification, and reinforcement learning AND improved performance compared to LoRA variants and other PEFT baselines.", "questions": "see weakness.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nThe idea of using activation-driven SVD for LoRA initialization is both elegant and theoretically grounded.\n\nEVA is rigorously tested across 51 tasks in four domains, showing strong average performance.\n\nDespite being data-driven, EVA exhibits minimal overhead (<1% training time) via batch-wise incremental SVD.\n\nWeakness:\n\nThe method assumes access to a static downstream dataset prior to fine-tuning, which may not generalize to online or continual learning settings.\n\nWhile adaptive rank allocation is justified theoretically, its impact varies per task (e.g., limited gains in decision-making tasks).\n\nEVA focuses on relatively low-rank setups (e.g., r=16); scaling behavior to higher ranks is less explored due to computational constraints, seems to limit its practicality.", "was_revised": true, "final_justification": "I thank the authors for their detailed rebuttal, which addresses my concerns clearly. I maintain my borderline accept recommendation.", "rebuttal": "Dear reviewer LBrt,\n\nThank you for your valuable feedback and positive assessment of our work.\n\n**Static Dataset**\n\nWe agree with the reviewer that this is a current limitation of our method, as we explicitly state in our discussion section. EVA is designed to maximize performance in the very common and practical scenario where fine-tuning is performed on a known, static downstream dataset. This is the basic setting that is considered in the literature. We believe that extending EVA to non-static datasets is a fruitful direction for future work.\n\n**Impact of Rank Redistribution**\n\nWe thank the reviewer for this observation, which aligns with our own findings. As noted in our results, the performance improvement from adaptive rank allocation can indeed be task-dependent, with some domains like decision-making showing limited gains. However, a key benefit of our redistribution strategy is that it consistently reduces the number of trainable parameters across all our experiments. This is because ranks are often shifted from higher-dimensional weights to more parameter-efficient lower-dimensional ones. Therefore, even when performance is on-par, EVA offers a clear advantage in parameter-efficiency, which is particularly relevant when fine-tuning large models. For this reason, we recommend using rank redistribution by default.\n\n**Choice of Rank**\n\nWe thank the reviewer for raising this point. While we acknowledge the computational overhead for very high ranks, we did explore scaling behavior beyond rank 16. Specifically, in Table 10 (Appendix A), we present results for fine-tuning Llama-2-7B with ranks up to 64. The results show that EVA and EVA+DORA are consistently the best-performing methods across all tested ranks, from 8 to 64. Furthermore, our experiments suggest that simply increasing the rank does not necessarily guarantee better performance. In fact, we empirically observed that higher ranks do not always perform better. We hypothesize this may be due to overfitting when introducing too many parameters relative to the dataset size. Consequently, EVA’s strong performance, particularly in lower-rank setups, is a key advantage for practical, compute-constrained fine-tuning scenarios, which are very prevalent in practice.\n\nIf any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion."}, {"review_id": "A66iemlXfj", "reviewer": "Reviewer_sfAD", "summary": "The paper proposes Explained Variance Adaptation, a new LORA weight initialization method that provably maximizes the expected gradient signal. Given a rank budget, EVA captures the maximum possible variance in the input mini-batches for every layer of the model. The paper claims that maximizing expected variance directly improves the performance of different tasks upon adaptation. Experiments spanning several tasks and domains show faster convergence and higher average scores than several LoRA variants, with a reported initialisation overhead being a fraction of total fine‑tuning time.", "questions": "1. How does the expected gradient magnitude theoretically contribute to the generalization error?\n2. What does the paper mean by \"stability\"?\n3. Why is the approach more significant than adding adaptive ranks to earlier data-based initialization to CorDA or LoRA-GA?\nI will increase my score to an accept if question 1 is answered with sufficient backing (theory or references to prior work) and if the novelty concerns can be sufficiently addressed.\n\n\nEDIT\n------------------------------\n\nThe authors have clarified most of these concerns in their rebuttal.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The primary strength of the paper is in its evaluation and analysis. I appreciate the extensive evaluation of the method on several different settings and domains, as well as the extensive experiments and ablations in the supplementary material.\n\n2. The paper has been written well, and all steps of the method are clear enough for re-implementation. The method itself is intuitive and easy to understand.\n\nWeaknesses:\n\n1. One gap that I found in the theoretical results is how higher expected gradient magnitude translates to better generalization. The paper has the underlying assumption that a higher magnitude is a surrogate for better generalization capabilities/lower generalization error on the test set, but why this would be the case is not well established. I would like to see some citations or additional theory on how the expected gradient magnitude of LORA updates influences model performance.\n\n2. Some of the terms used in the paper, like \"stronger, more stable gradient signal,\" have not been well-defined in an intuitive or mathematical sense. For instance, does \"stability\" refer to a lesser likelihood of encountering NaN values during training or lower variance across batches?\n\n3. Overall, I find that the paper has limited novelty. Data-based decompositions for LoRA initialization have been explored in prior work (CorDA by Yang et al. and LoRA-GA by Wang et al.), while adaptive ranks have been well established in AdaLoRA and its successors. EVA mainly bundles these ideas together.\n\n4. The proof of Theorem 4 assumes orthogonality of the gradient and input. However, this is only guaranteed for the first gradient update step. I also do not follow why equation 11 is relevant here (I might be missing some intermediate arguments). Since equation 12 has proportionality instead of equality, doesn't it directly follow from equation 10 after the first update step when $g^T g$ is likely non-zero?\n\n\nEDIT\n------------------------------\n\nThe authors have clarified most of these concerns in their rebuttal.", "was_revised": true, "final_justification": "I thank the authors for their response.\n\nThe link with NTK, and through it, EVA's impact of generalization is definitely interesting. I strongly recommend that the authors add this component to their paper. This also addresses my main concern regarding the novelty of the paper.\n\nI therefore increase my score.", "rebuttal": "Dear reviewer sfAD,\n\nThank you for the constructive feedback, it helps us to substantially improve our manuscript. We address the raised concerns as follows.\n\n**Novelty**\n\nWe acknowledge that adaptive rank allocation and data-driven initialization have been explored in concurrent works (e.g., CorDA, LoRA-GA). However, we strongly believe that novelty should not be judged solely by whether single components used in a framework already existed, but by whether a method offers a new perspective or principled formulation that provides novel insights. That said, EVA offers a theoretically grounded approach: both initialization and rank allocation arise from maximizing expected gradient signal under a parameter constraint. This goes beyond heuristic initialization design (CorDA, LoRA-GA) or soft constraints during optimization (AdaLoRA). By proposing EVA we provide insights into the design of maximizing gradient signal without relying on hyperparameters for computing the initialization as in CorDA/LoRA-GA which allows to systematically control for rank redistribution while providing strong performance with a reduced parameter count. We believe all of these constitute a meaningful and novel contribution to the field.\n\n**Theoretical Contribution**\n\nThank you for pointing out the lack of connection between gradient signal magnitude and generalization error. There is indeed a theoretically grounded connection between initialization via EVA and the generalization error via the Neural Tangent Kernel (NTK, [1]). To briefly recap, EVA selects a rank-r subspace $V= \\operatorname{span}({v_1, \\ldots,v_r})$ in which to place LoRA updates such that the expected gradient magnitude is **maximized**:\n$$\\mathbb{E}\\_{x} \\left[ \\left\\| \\nabla_\\theta f(x)^\\top v_i \\right\\|^2 \\right]$$\nThis expectation is maximized if we choose the **top eigenvectors of the activation covariance** for initializing the LoRA matrices, as done in EVA. That is, EVA aligns LoRA directions with the principal components of \n$$\\Sigma = \\mathbb{E}\\_{x} \\left[  \\nabla_\\theta f(x) \\nabla_\\theta f(x)^\\top \\right]$$.\n\nIntriguingly, **this is equivalent to the empirical NTK matrix.** In NTK theory, learning proceeds along the NTK eigenbasis. Let $K \\in \\mathbb{R}^{n\\times n}$ be the empirical NTK matrix over training points $x\\_1, \\ldots, x\\_n$. The NTK view tells us that at infinite width the network output evolves under gradient descent as:\n$$f\\_t(x) = f\\_0(x) + K(x, X) K^{-1} (y - f\\_0(X)) \\cdot \\left(1 - e^{-\\eta \\Lambda t} \\right)$$,\nand that learning decomposes across the eigenfunction of the NTK. Crucially, **directions with higher NTK eigenvalues $λ\\_i$​ are learned faster and more robustly [2,3]**.\n\nFinally, EVA’s initialization ensures that the LoRA subspace is aligned with the **top-r eigenvectors of the NTK**.\nLet $u\\_i=1n\\{u_i\\}\\_{i=1}^n$ be the eigenvectors of K, and $y \\in R^n$ be the labels. Then the squared generalization error is bounded by the residual in low-eigenvalue components:\n$$\\mathcal{E}\\_{\\text{gen}} \\lesssim \\sum\\_{i=r+1}^{n} \\frac{(u_i^\\top y)^2}{\\lambda_i^2}$$.\nOr put differently, the projection of the residual error onto **non-covered NTK directions (those outside $\\mathcal{V}$)** determines the generalization error. As EVA initializes the LoRA matrices with the top-r eigenvectors of the NTK, the generalization error is minimized as it is covered only by directions with strictly smaller eigenvalues. Therefore, maximizing expected gradient signal through data-driven initialization aligns low-rank adaptation directions with the top eigenspaces of the NTK. Prior work [2,3] shows that learning concentrated in these eigenspaces leads to faster convergence and lower generalization error.\n\nThanks to the reviewer’s comment we were able to establish this intriguing connection and we hope this satisfies their concerns as these additional theoretical insights add novelty to our proposed approach. \n\n[1] Jacot et al., Neural Tangent Kernel: Convergence and Generalization in Neural Networks, NeurIPS 2018\n[2] Arora et al., On Exact Computation with an Infinitely Wide Neural Net, NeurIPS 2019\n[3] Lee et al., Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent, NeurIPS 2019\n\n**Questions:**\n\n**What does the paper mean by \"stability\"?**\n\nThank you for noticing the lack of definition for the term “stability”. We indeed observed that EVA can lead to smoother convergence curves compared to LoRA-GA or CorDA (Figure 3, right). However this is an empirical observation without any theoretical backing. From a theoretical viewpoint the variance in the gradients will be larger than the ones encountered in standard LoRA, as EVA concentrates gradient variance in the most informative directions. This definitely affects training dynamics and sometimes we observed that especially for higher learning rates it can be that LoRA still trains, but EVA diverges. We add a discussion on this in Section 5. We admit that the usage of this term is confusing and therefore remove it from our claims.\n\n\n**Since equation 12 has proportionality instead of equality, doesn't it directly follow from equation 10 after the first update step when gTg is likely non-zero?**\n\nYes, you are correct, Eq. 12 directly follows from Eq. 10 under the statistical independence assumption. The relevance of Eq. 11 is indeed not directly given in the current form. We agree that the phrasing around this equation is a bit unfortunate. We simply meant to augment the meaning of statistical independence in a formal manner. We rephrase this to avoid any confusion at this point.\n\nIf any questions remain or come up during the rebuttal process, we would be happy to engage in further discussion."}]

Author rebuttal:

Meta-review: This paper proposes Explained Variance Adaptation (EVA), a variant of Low Rank Adaptation (LoRA) for finetuning foundation models. EVA performs incremental SVD on activation vectors to obtain directions that maximize explained variance. These directions are used to initialize LoRA matrices, aiming to maximize the expected gradient signal at the onset of fine-tuning. An adaptive rank allocation mechanism is also proposed based on explained variance ratios. Experiments are conducted on different tasks to demonstrate the effectiveness of the proposed EVA.

Four reviewers reviewed this paper with the overall ratings of 3 borderline accept and 1 borderline reject. The reviewers recognized the contributions of this paper, such as good motivation, extensive experiments, elegant idea, and good presentation. Meanwhile, they also pointed out some critical concerns about different aspects, such as relatively limited novelty, unclear terms, not well-established assumptions, insufficient exploration on higher ranks, and missing related baselines. Thanks to the authors' rebuttal, most of the concerns are addressed, and the reviewers are generally satisfied. The AC recommends acceptance and strongly suggests that the authors include the promised revision in the final version and take the reviewers' comments into consideration.
Decision: Accept (poster)