EigenLoRA: Recycle trained Adapters for Resource Efficient Adaptation and Inference

Authors: Prakhar Kaushik, Aayush Mishra, Ankit Vaidya, Raghavendra Addanki, Ryan A. Rossi, Ani Nenkova, Anqi Liu, Alan Yuille, Jiuxiang Gu
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Parameter-efficient fine-tuning, Transfer learning, Low-rank, NLP, vision, diffusion, efficient learning, eco-friendly

Abstract:
Low-Rank Adapters (LoRA) are lightweight components that have made fine-tuning large models on domain-specific tasks inexpensive. This has resulted in an abundance of adapters in a growing open-source public community. We ask the question: can these adapters be used to inform and further streamline adaptation to new tasks? We introduce EigenLoRA, a parameter-efficient fine-tuning method that uses trained adapters to perform fast adaptation on new domains with orders of magnitude fewer parameters than LoRA. Our method finds a principal subspace that aligns with the domain of the trained adapters. This allows for efficient and fast adaptation to new tasks in this domain by simply learning coefficients on the principal components of this subspace. Furthermore, EigenLoRA makes inference time task-switching memory efficient. Instead of saving and loading whole LoRAs, EigenLoRA can simply load lightweight coefficients. EigenLoRA works across a variety of domains and tasks and is a viable solution for edge-based and efficient personalization applications.

Decision: reject
Reviewer scores: 6, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730296594613

thread_type:
official_review

<thread_object>
creator: reviewer_ctqp

<content>
summary:
The authors propose a new method based on LoRA called EigenLoRA. This method focuses on the K principal component of the weight matrices of LoRA. This allows for a reduction in the number of parameters to learn during training, time computation, and inference memory usage. After an introduction to their method, the authors propose several experimental studies to show the benefit of EignenLoRA for different modalities and tasks.

strengths:
- The paper is very well written. It is easy to follow the paper and understand the needs of the field and how they improve LoRA with eigendecomposition.
- The method proposed is an simple but efficient improve of the LoRA method
- A complete experimental part is proposed where EigenLoRA is tested over four different datasets. Each experiment show a benefit of the method.

weaknesses:
Major comment:
- The introduction and related work are short, and it could have been interesting to see a comparison with another method in the experiment study.
- In part 4.2.1, the part with the initialization is not very clear. Why does initialization add a lot of parameters to EigenLoRA? Why does it help for specific tasks (see MRPC), and why is it disruptive for others (see RTE)? Do you have any intuitions?-

Minor comments:
- Tables 2 and 3 have no bold performances, which is harder to read.
- No x-axis title for figure 4

questions:
- In practice, how do you choose the K? Do you always have to find the K best representative principal components?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639619042

thread_type:
official_review

<thread_object>
creator: reviewer_fwnt

<content>
summary:
This paper introduces EigenLoRA, a parameter-efficient fine-tuning method centers on extracting a shared subspace of essential directions (PCs) from multiple already-trained LoRA adapters. Instead of learning a full new adapter for each task, it only learns a small set of task-specific coefficients that combine these principal components to achieve adaptation. This approach dramatically reduces the number of parameters needed for fine-tuning and allows efficient task-switching by loading just the task-specific coefficients rather than full adapter matrices.

strengths:
1. EigenLoRA significantly reduces the number of trainable parameters required for new tasks, which is particularly valuable for low-resource devices or applications with strict memory constraints. By isolating task-specific coefficients and retaining shared principal components, EigenLoRA offers a reduction in memory usage during inference.

2. The method is empirically validated on diverse domains—including image classification, natural language understanding, and text-to-image generation—demonstrating its versatility and robustness across modalities and tasks.

3. EigenLoRA’s initialization in a shared principal subspace results in faster convergence during training, allowing it to reach or exceed baseline performance more efficiently than traditional approaches.

weaknesses:
1. The paper omits evaluations on certain baseline datasets, such as the RESISC45 dataset for image classification, which was included in VeRA’s evaluations. 

2. The specific ViT model used for image classification is not clearly identified, and the paper does not fully explain why certain settings yield different results compared to those in baseline studies.

3. Unlike VeRA, which includes comparisons across different model backbones (for image classification), this paper only evaluates a single backbone, limiting the assessment of its generalizability.

4. While baseline methods conduct evaluations on E2E and instruction-tuning tasks, this paper neither includes these benchmarks nor provides a rationale for their exclusion.

5. The proposed method uses customized hyperparameter settings, including varying learning rates and schedulers, while baselines adhere to a fixed set of hyperparameters, potentially compromising the fairness of comparisons.

6. Although the authors mention a study of the effect of the number of principal components K in Section A.2.2, no detailed results are provided.

7. In lines 883-884, the choice of rank r = 8 is not explained, nor is there an exploration of the impact of different r values on performance.

8. Some figures appear to be low-resolution screenshots from wandb, affecting readability of visual results.

questions:
See the weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652879208

thread_type:
official_review

<thread_object>
creator: reviewer_mfg3

<content>
summary:
This paper introduces EigenLoRA, a parameter-efficient fine-tuning method that improves upon LoRA by recycling information from previously trained adapters through identifying a principal subspace shared by adapters trained on related tasks within a domain, allowing new tasks to be adapted by learning coefficients for pre-extracted principal components rather than full LoRA matrices, which results in using up to 100x fewer parameters and achieving up to 2x faster convergence during training while improving memory efficiency by ~18x when switching between multiple tasks during inference by only loading lightweight coefficients rather than full adapter matrices, demonstrating wide applicability across different modalities and domains with both theoretical foundations including approximation bounds for reconstruction error and practical validation, positioning it as a resource-efficient solution particularly suitable for edge devices and personalization applications.

strengths:
This work presents comprehensive experiments and evaluations on language and CV models, image generation, and achieves good results.

weaknesses:
1. Absent evidence of practical cost optimization: In fact, the reduced number of training parameters but still requires a large number of fixed parameters in the forward process. These fixed parameters also incur significant additional overhead. That is, this approach does not significantly reduce the time and memory needed in fine-tuning. Just reducing training parameters is not practical and diminishes fine-tuning performance.


2. Lack of novelty: Actually, this work just utilizes SVD as the initialization of LoRA and dynamically selects the fine-tuning of the eigens, which  already many studies in previous works. Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR), DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation. EACL. 


3. Lack of solid evaluations: Evaluation In Vision Transformer, the GLUE benchmark heavily relies on hyperparameter tuning, and the image generation fine-tuning effect is not sensitive to the design of the LoRA. In other words, those presented experiments  are not a solid reflection of the performance of the proposed method. I suggest that the authors present more evidence of in Commonsense Reasoning and instruction fine-tuning tasks in recent LLMs. e.g., Llama 3 8B and 70B, Llama 3.2 1B and 3B, DeepSeekMoE，Mixtral-8x7B.

4. Numbers Trainable Parameters such as +0, in Tables 1 and 3 are confusing. I believe that +0 does not mean that there is no additional fine-tuning overhead . The authors need to use other cost metrics.

5. Absence of theoretical Support: why partial updated LoRA variants like this one would be better than full parameter updated LoRA variants. 

6. Lack of recent parameter-efficient LoRA method discussion. e.g., LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters, NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730782230576

thread_type:
official_review

<thread_object>
creator: reviewer_hc8t

<content>
summary:
The paper introduces EigenLoRA, an approach for parameter-efficient fine-tuning that leverages the principal subspaces of low-rank adapters (LoRA) trained on various domain-specific tasks. EigenLoRA reduces the parameter count by up to 100x and optimizes memory efficiency, benefiting tasks such as inference on resource-constrained devices. By identifying a domain’s principal subspace, EigenLoRA offers a lightweight way to perform new task adaptations by only learning coefficients within this subspace, avoiding full weight reinitialization. The approach is validated on image classification and NLP benchmarks, demonstrating competitive performance with significantly lower computational overhead.

strengths:
1. EigenLoRA achieves high parameter and memory efficiency, cutting parameters by up to 100x. This makes it ideal for low-resource deployments.
2. The method performs well across various tasks, matching LoRA’s results with far fewer parameters, and proving its versatility.
3. Figures and tables are well-designed.
4. EigenLoRA’s low memory demand fits well with real-world edge applications, like personal devices with limited resources.

weaknesses:
1. EigenLoRA’s success depends on high-quality adapters, which might be limiting in under-researched domains.
2. Guidance on picking the right number of principal components would help with applying this method across diverse tasks.
3. Failure cases need more examples to help users understand when and why the method might struggle.
4. While suited for edge devices, more real-world benchmarks would strengthen claims about efficiency in low-memory environments to meet ICLR standards.

questions:
Q1. How do you envision managing the dependency on trained adapters in a practical deployment setting? Are there scenarios where this reliance could hinder flexibility?
Q2. Can EigenLoRA be extended or modified to handle tasks involving multi-modal or cross-domain data? If so, what challenges do you foresee?
</content>
</thread_object>

</thread>


Metareview:
metareview:
a) This paper introduces a parameter-efficient fine-tuning (PEFT) method that improves upon LoRA by learning a principal subspace of adapters pre-trained on related tasks within a domain. This allows a new task to be adapted faster, with fewer parameters and memory.

b) This method achieves high parameter and memory efficiency, cutting parameters by up to 100x, making it ideal for low-resources devices. The paper presents comprehensive experiments and evaluations on language, computer vision and image generation models, achieving promising results.

c) In contrast to other PEFT methods, this method requires a set of pre-trained adapters to learn the principal subspace. This limits its applicability to only domains where pre-trained adapters are available and makes the comparison with the other approaches (which do not require that) not fully fair. The techniques used are quite standard, so, apart form the application, there is no much technical novelty.

d) The most important reason to reject the paper is the lack of a fair comparative. Authors claim high reduction in terms of computation, memory and parameters, assuming the pre-trained adapters needed for their training as given. While this can be realist in many use-cases, still the comparative is not fully fair. The proposed model requires less resources because part of the training is amortized in the pre-trained PEFT adapters. Instead of putting forward the impressive numbers, authors should have presented in a fairer comparative evaluation advantages and disadvantages of the proposed approach.