Title: Parameter Efficient Fine-tuning via Explained Variance Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Foundation Models

PEFT

LoRA
Abstract: Foundation models (FMs) are pre-trained on large-scale datasets and then fine-tuned for a specific downstream task. The most common fine-tuning method is to update pretrained weights via low-rank adaptation (LoRA). Existing initialization strategies for LoRA often rely on singular value decompositions (SVD) of gradients or weight matrices. However, they do not provably maximize the expected gradient signal, which is critical for fast adaptation. To this end, we introduce **E**xplained **V**ariance **A**daptation (EVA), an initialization scheme that uses the directions capturing the most activation variance, provably maximizing the expected gradient signal and accelerating fine-tuning. EVA performs incremental SVD on minibatches of activation vectors and selects the right-singular vectors for initialization once they converged. Further, by selecting the directions that capture the most activation-variance for a given rank budget, EVA accommodates adaptive ranks that reduce the number of trainable parameters. We apply EVA to a variety of fine-tuning tasks as language generation and understanding, image classification, and reinforcement learning. EVA exhibits faster convergence than competitors and achieves the highest average score across a multitude of tasks per domain while reducing the number of trainable parameters through rank redistribution. In summary, EVA establishes a new Pareto frontier compared to existing LoRA
initialization schemes in both accuracy and efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Explained Variance Adaptation, a new LORA weight initialization method that provably maximizes the expected gradient signal. Given a rank budget, EVA captures the maximum possible variance in the input mini-batches for every layer of the model. The paper claims that maximizing expected variance directly improves the performance of different tasks upon adaptation. Experiments spanning several tasks and domains show faster convergence and higher average scores than several LoRA variants, with a reported initialisation overhead being a fraction of total fine‑tuning time.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The primary strength of the paper is in its evaluation and analysis. I appreciate the extensive evaluation of the method on several different settings and domains, as well as the extensive experiments and ablations in the supplementary material.\n\n2. The paper has been written well, and all steps of the method are clear enough for re-implementation. The method itself is intuitive and easy to understand.\n\nWeaknesses:\n\n1. One gap that I found in the theoretical results is how higher expected gradient magnitude translates to better generalization. The paper has the underlying assumption that a higher magnitude is a surrogate for better generalization capabilities/lower generalization error on the test set, but why this would be the case is not well established. I would like to see some citations or additional theory on how the expected gradient magnitude of LORA updates influences model performance.\n\n2. Some of the terms used in the paper, like \"stronger, more stable gradient signal,\" have not been well-defined in an intuitive or mathematical sense. For instance, does \"stability\" refer to a lesser likelihood of encountering NaN values during training or lower variance across batches?\n\n3. Overall, I find that the paper has limited novelty. Data-based decompositions for LoRA initialization have been explored in prior work (CorDA by Yang et al. and LoRA-GA by Wang et al.), while adaptive ranks have been well established in AdaLoRA and its successors. EVA mainly bundles these ideas together.\n\n4. The proof of Theorem 4 assumes orthogonality of the gradient and input. However, this is only guaranteed for the first gradient update step. I also do not follow why equation 11 is relevant here (I might be missing some intermediate arguments). Since equation 12 has proportionality instead of equality, doesn't it directly follow from equation 10 after the first update step when $g^T g$ is likely non-zero?\n\n\nEDIT\n------------------------------\n\nThe authors have clarified most of these concerns in their rebuttal.\n\nQUESTIONS:\n1. How does the expected gradient magnitude theoretically contribute to the generalization error?\n2. What does the paper mean by \"stability\"?\n3. Why is the approach more significant than adding adaptive ranks to earlier data-based initialization to CorDA or LoRA-GA?\nI will increase my score to an accept if question 1 is answered with sufficient backing (theory or references to prior work) and if the novelty concerns can be sufficiently addressed.\n\n\nEDIT\n------------------------------\n\nThe authors have clarified most of these concerns in their rebuttal.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
This paper introduces Explained Variance Adaptation (EVA), a novel initialization scheme for LoRA in parameter-efficient fine-tuning of foundation models. Unlike prior SVD-based initializations, EVA performs incremental SVD on activation vectors to obtain directions that maximize explained variance. These directions are used to initialize LoRA matrices, aiming to maximize the expected gradient signal at the onset of fine-tuning. The paper also proposes an adaptive rank allocation mechanism based on explained variance ratios. EVA is evaluated across various domains—language modeling, language understanding, image classification, and reinforcement learning AND improved performance compared to LoRA variants and other PEFT baselines.

STRENGTHS AND WEAKNESSES:
Strength:

The idea of using activation-driven SVD for LoRA initialization is both elegant and theoretically grounded.

EVA is rigorously tested across 51 tasks in four domains, showing strong average performance.

Despite being data-driven, EVA exhibits minimal overhead (<1% training time) via batch-wise incremental SVD.

Weakness:

The method assumes access to a static downstream dataset prior to fine-tuning, which may not generalize to online or continual learning settings.

While adaptive rank allocation is justified theoretically, its impact varies per task (e.g., limited gains in decision-making tasks).

EVA focuses on relatively low-rank setups (e.g., r=16); scaling behavior to higher ranks is less explored due to computational constraints, seems to limit its practicality.

QUESTIONS:
see weakness.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel initialization and allocation scheme for PEFT, which leverages information obtained from activations to initialize low-rank adapters into the most impactful directions (with highest variance). This leads to faster convergence and higher performance in a broad range of settings, compared to competitors\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The presentation is clear, and the method is well motivated\n- Extensive number of analysis and ablations\n\n\nWeaknesses:\n1. EVA leads to performance improvements, though it incurs an initial cost for SVD initialization. However, it is not clear how the performance of “SVD_init + EVA” compares to “random_init + LoRA” (or other methods) at the same time t, where t = 0 is defined as the time before the two initializations.\n2. While GLUE is extensively used for testing PEFT performance, reporting results on the validation set (as it is commonly done) is not best practice, as it is prone to overfitting. A more rigorous evaluation approach involving a separate validation/test split has been proposed in [1,2,3].\n\n\n[1] Wu M. et al., Advancing parameter efficiency in finetuning via representation editing, ACL 2024\n\n[2] Wu Z. et al., Reft: Representation finetuning for language models, NeurIPS 2024\n\n[3] Bini et al., DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation, ICLR 2025\n\nQUESTIONS:\nThe main concerns are reported in Weaknesses.\n\nOther question:\n1. In section 3.3., line 158, referring to A,B matrices, the authors mention “to preserve the base model weights, one of them must be initialized with zeros”. While zero-initializing one of the two matrices is the most common practice, it should be mentioned that other methods (such as [4,3]) utilize a residual approach (subtracting frozen version of the adapters).\n- In particular, did the authors test this type of initialization, which in theory would allow to also initialize the B matrix with SVD information?\n2. How does performance vary at different minibatch sizes? What is the minimum amount of data necessary to obtain significant performance improvements?\n\nI am willing to increase my score if the main concerns are addressed.\n\n[4] Meng et al. - PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models, NeurIPS24\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a variant of Low Rank Adaptation (LoRA) for finetuning foundation models. The basic idea, termed Expected Variance Adaptation (EVA), is to perform an SVD on the activation vectors that are obtained/estimated from minibatches of the downstream task. This is similar in spirit to methods that use the gradients to estimate the importance. The authors show that this process leads to a \"more stable\" gradient signal. They also propose a rank allocation procedure that tries to distribute a given rank budget across weight matrices in order to maximize the explained variance. The paper does experiments on finetuning LLMs on math+reasoning taks, as well as results for image classification and decision making tasks. The main goal of the experiments is to report the parameter reduction, while preserving accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The method proposed is natural, and the experiments are fairly thorough in terms of the domains that they cover.\n- The writing of the paper is very clear and the ideas are well explained.\n\nWeaknesses:\n\n- Unfortunately the paper is not very novel. The idea is very similar to the gradient based importance metrics. While the paper says that this is \"provably\" good initialization at several points, this is because of the way the objective is set up-- the algorithm is basically SVD, and is optimal in terms of capturing the variance. (Why is capturing the variance of the activations or aligning with the gradient the right objective?)\n\n- The experimental section is reasonable, but this has become a really crowded field over the last couple of years, so the paper needs to compare with a bunch of newer work on PEFT. (DoRA, MiLoRA). There are also \"activation guided\" methods that are really similar to this paper, e.g., AG-LoRA (https://ieeexplore.ieee.org/document/10852296), GLoRA (https://openreview.net/forum?id=K7KQkiHanD ) that the authors should compare to.\n\nOverall, while the writing is clean, I am concerned about the novelty with this submission.\n\nQUESTIONS:\nPlease address the questions of novelty and comparison that were raised in the the \"weakness\" section of the review.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}