NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

## Abstract

Abstract In this paper, we introduce Nested Low-Rank Adaptation (NoRA), a novel approach to parameter-efficient fine-tuning that extends the capabilities of Low-Rank Adaptation (LoRA) techniques. Vanilla LoRA overlooks pre-trained weight inheritance and still requires fine-tuning numerous parameters. To addresses these issues, our NoRA adopts a dual-layer nested structure with Singular Value Decomposition (SVD), effectively leveraging original matrix knowledge while reducing tunable parameters. Specifically, NoRA freezes the outer LoRA weights and utilizes an inner LoRA design, providing enhanced control over model optimization. This approach allows the model to more precisely adapt to specific tasks while maintaining a compact parameter space. By freezing outer LoRA weights and using an inner LoRA design, NoRA enables precise task adaptation with a compact parameter space. Evaluations on tasks including commonsense reasoning with large language models, fine-tuning vision-language models, and subject-driven generation demonstrate NoRA’s superiority over LoRA and its variants. Code will be released upon acceptance.

## 1 Introduction

In recent years, LLMs [] have set new performance benchmarks in natural language processing [] (NLP) and related fields [, , ], yet their substantial size introduces significant challenges in training and adaptation. The high computational and storage demands of comprehensive fine-tuning make it impractical in resource-limited environments. To address this, Parameter-Efficient Fine-Tuning [, ] (PEFT) techniques have been developed, focusing on fine-tuning a reduced subset of model parameters to optimize model adaptability. Among these techniques, LoRA [] stands out as a prominent method that mitigates the extensive resource requirements of full fine-tuning [] by introducing trainable low-rank matrices into the model architecture.

Figure: Figure 1: In the comparison between training with LoRA and NoRA, the blue modules represent the parts where parameters are frozen during training, while the orange modules indicate the components that need to be updated. Here, $r$ denotes the outer rank, and $r^{\prime}$ denotes the inner rank.
Refer to caption: extracted/5816986/pic/over.png

LoRA utilizes low-rank matrices to achieve efficient and scalable adaptation to specific downstream tasks [, ], allowing the model to focus on essential parameters. This approach not only preserves the model’s pre-trained knowledge but also facilitates its specialization with minimal computational overhead. The core hypothesis of LoRA is that the adaptations necessary to tailor a pre-trained large language model (LLM) to a specific task or domain are inherently low-dimensional, which can be effectively achieved through a low-rank decomposition of the weight matrices, minimizing computational overhead while preserving the model’s pre-trained knowledge [].

Despite LoRA’s demonstrated utility, various LoRA variants proposed by recent researchers [, ], each have their strengths and limitations. For example, while LoRA-FA [] reduces activation memory demands by freezing part of the weights, it still suffers from the limitation of a fixed rank; VeRA [] enhances model scalability but remains sensitive to the hidden dimensions of the model; FLoRA [] and LoRA-XS [] have made improvements in real-time performance and memory efficiency, yet they still do not fully address the complexity specific to tasks.

These limitations prompted us to explore a new Nested Low-Rank Adaptation structure called NoRA, which enhances parameter efficiency and task adaptability through a dual-layer nested design combined with SVD. In the NoRA structure, the outer layer provides a stable low-rank foundation by keeping its weights fixed, while the inner layer introduces fine-grained adjustments tailored to different tasks. This innovative dual-layer nested structure effectively leverages the knowledge within the original matrix and further enhances model adaptability through precise optimization of certain parameters. NoRA not only retains the parameter efficiency advantages of LoRA but also improves the accuracy of low-rank matrix approximation through the decomposition properties of SVD, thereby enhancing the model’s generalization capability and adaptability across different tasks. Moreover, the hierarchical separation between the outer and inner LoRA layers provides finer optimization control, enabling the model to more flexibly adjust the contribution of each layer’s parameters, thus enhancing adaptability and flexibility.

We conducted experiments on multiple downstream tasks, including fine-tuning the LLaMA [] model for commonsense reasoning, few-shot tuning on the CLIP [] model, and subject-driven generation on the Stable Diffusion XL [] model. NoRA effectively reduced the required parameters to as low as 10.2M for LLaMA3 8B while enhancing performance, achieving an average score of 85.0%, surpassing LoRA’s 82.8%. Additionally, in visual few-shot tasks using ViT-B/16, NoRA achieved the highest average accuracies of 81.8% (4 shots) and 85.4% (16 shots), demonstrating its superior efficiency and effectiveness over existing methods. We summarize our contribution as below:

- •
We propose NoRA, which uses a dual-layer nested structure and SVD. The outer LoRA provides a stable low-rank foundation, while the inner LoRA allows for precise task adjustments.
- •
NoRA leverages the original matrix’s knowledge through its nested structure and SVD integration, enhancing low-rank matrix approximation accuracy. This design improves the model’s adaptability across diverse tasks, while the hierarchical separation between layers allows for finer control over optimization.
- •
Our extensive experimental validation on multiple downstream tasks has shown that NoRA outperforms traditional LoRA and other recent variants such as VeRA and FLoRA.

## 2 Related Work

Parameter-Efficient Fine-Tuning. Parameter Efficient Fine-Tuning [, , , , , ] (PEFT) is a practical solution for adapting large pre-trained models to downstream tasks []. As the number of parameters in large-scale language models continues to grow, traditional fine-tuning methods face significant challenges, such as high computational resource demands and training costs []. PEFT optimizes the parameter adjustment process, effectively reducing the number of additional parameters introduced and the computational resources required for specific tasks or domains. The core idea is to enhance the model’s task adaptability by streamlining parameter updates and introducing auxiliary modules while maintaining the structure and performance of the pre-trained model, without the need for complete retraining. This approach is particularly suitable for large-scale language models, as it reduces the computational burden and significantly improves the model’s applicability across diverse downstream tasks. As a result, PEFT has become the mainstream trend for fine-tuning large pre-trained models, greatly promoting their adoption and application across different scenarios. Early PEFT research primarily focused on selective update strategies, such as Bias-Free Fine-Tuning [] and Partial Network Training [], which achieve task-specific model fine-tuning by modifying only the most critical parameters of the pre-trained model. As research progressed, adapter modules became another direction in PEFT development, embedding additional modules, such as Prompt Tuning and Prefix Tuning, into the pre-trained architecture to further enhance the model’s adaptability to specific tasks. Delta-weight techniques represent the latest innovation in PEFT, with LoRA and OFT [] being notable examples. These strategies involve fine-tuning pre-trained parameters using trainable delta weights, providing fine-grained model adjustments that improve task-specific performance while minimizing computational overhead.

Low-rank Adaptation.
LoRA has proven to be an efficient fine-tuning strategy in various task scenarios, leveraging low-rank decomposition to enhance model adaptation while minimizing computational overhead. However, LoRA’s fixed rank limitation can restrict its flexibility in handling diverse tasks. To address these limitations, researchers have proposed various LoRA variants, each showcasing unique strengths in enhancing model adaptability and performance. For instance, LoRA-FA [] reduces activation memory demands by freezing part of the weights, though it remains limited by the fixed rank; VeRA [] excels in enhancing model scalability but is still sensitive to the model’s hidden dimensions; FLoRA [] and LoRA-XS [] improve real-time performance and memory efficiency, yet they do not fully address the complexity specific to certain tasks. Other variants, such as VB-LoRA [] and Tied-LoRA [], introduce new adaptation mechanisms that enhance model adaptability but also bring additional computational overhead and complexity, posing challenges in practical applications. Moreover, as research has progressed, techniques such as VeRA and DoRA [] have further optimized the LoRA method, making innovative improvements in parameter efficiency, as well as in the distribution and structure of update matrices. Advanced methods like AdaLoRA [] and PiSSA [] push the limits of parameter update efficiency by selectively adjusting matrix ranks and distributions, significantly improving the applicability of large-scale pre-trained models in complex tasks. However, despite the significant advancements achieved by these LoRA variants, they still face certain limitations and challenges in practical applications, particularly in meeting diverse task requirements and optimizing computational resource usage, which necessitate further research and solutions.

## 3 Methodology: Nested Low-Rank Adaptation

In this paper, we present NoRA, a novel approach to parameter-efficient fine-tuning that builds upon traditional LoRA techniques. By incorporating a dual-layer nested structure with SVD, NoRA effectively harnesses the knowledge of the original matrix, further reducing fine-tuning parameters while enhancing the model’s adaptability and optimization control.

### 3.1 Review of Low-Rank Adaptation

LoRA is a parameter-efficient method for fine-tuning large-scale pre-trained models. It achieves fine-tuning of the original weights through the introduction of low-rank matrix updates, without altering the stability and performance of the pre-trained models. In LoRA, the weights for each layer are updated using the following formula:

$$ $h=Wx+\Delta Wx=Wx+BAx,$ (1) $$

where $\Delta W\in\mathbb{R}^{m\times n}$ is the low-rank weight update, and $A\in\mathbb{R}^{r\times n}$ and $B\in\mathbb{R}^{m\times r}$ are the low-rank matrices with $r\ll\min(m,n)$. During training, $W$ is kept frozen, and $A$ and $B$ are the trainable parameters.

### 3.2 NoRA Structure and Initialization

NoRA employs a dual-layer nested structure, where each layer is initialized using the SVD [] of the pre-trained weights to enhance the model’s learning capabilities. Specifically, NoRA first decomposes the original weights using SVD, then initializes the matrix $A$ with $U$ and the matrix $B$ with $V^{T}$. The inner LoRA weights are initialized using the intermediate diagonal matrix, thus optimizing the weight updates $\Delta W$.

- •
Outer LoRA Layer: This layer is initialized using the SVD results $U\Sigma V^{T}$ of the pre-trained weights $W$. The parameters of this outer LoRA layer are frozen during training. Freezing these parameters helps maintain stability and preserve the key features of the pre-trained model, while still allowing for precise adjustments through the inner LoRA layer.
- •
Inner LoRA Layer: Initialized with the diagonal matrix $\Sigma$ from the SVD [] of the $U\Sigma V^{T}$. Initializing with $\Sigma$ allows this layer to focus on subtle perturbations in the weight space, enabling finer adjustments without altering the core weights preserved by the outer LoRA layer. This approach ensures that updates are focused on refining and enhancing the model’s ability to adapt to new tasks, leveraging minor adjustments that have a targeted impact on the model’s performance.

The final weight update formula is:

$$ $h=Wx+\Delta Wx=Wx+BAx+B^{\prime}A^{\prime}x,$ (2) $$

Figure: Algorithm 1 PyTorch code for NoRA

where $A$ and $B$ are set using the truncated SVD of the original weight matrix $W$, with the outer LoRA weights remaining frozen. The matrices $A$ and $B$ are initialized as $A=U_{r}$ and $B=V_{r}^{T}$ where $U_{r}$ and $V_{r}$ are the left and right singular vectors corresponding to the top $r$ singular values, and these vectors are obtained from the decomposition: $W=U\Sigma V^{T}$ with $U\in\mathbb{R}^{m\times m}$, $\Sigma\in\mathbb{R}^{m\times n}$, and $V\in\mathbb{R}^{n\times n}$. For the inner LoRA layer, the matrices $B^{\prime}$ and $A^{\prime}$ are initialized by taking the square roots of selected components from the top $r_{in}$ singular values in the diagonal matrix $\Sigma_{r}$, where $\Sigma_{r}\in\mathbb{R}^{r\times r}$. Specifically, $B^{\prime}$ is initialized using the square roots of the first $r_{in}$ columns of $\Sigma_{r}$, and $A^{\prime}$ is initialized using the square roots of the first $r_{in}$ rows of $\Sigma_{r}$. This initialization approach introduces another layer of adaptation within the NoRA framework by decomposing the smaller matrix $W_{r}$ through an additional LoRA process.

### 3.3 Parameter Efficiency of NoRA

We make the following observation on the parameter efficiency of NoRA compared to LoRA and VeRA methods.
Observation: NoRA demonstrates superior parameter efficiency compared to both LoRA and VeRA.
For simplicity, let’s consider a transformer model with $L$ finetuned layers, each consisting of $q$ number of $W\in\mathbb{R}^{n\times n}$ matrices.

For LoRA, the number of trainable parameters is given by:

$$ $P_{\text{LoRA}}=L\times q\times r\times 2n.$ (3) $$

For NoRA, the number of trainable parameters is given by:

$$ $P_{\text{NoRA}}=L\times q\times r_{out}\times r_{in}\times 2.$ (4) $$

To compare the parameter efficiency, we compute the ratios of the number of trainable parameters between the methods. The ratio of trainable parameters for LoRA to NoRA is:

$$ $\frac{P_{\text{LoRA}}}{P_{\text{NoRA}}}=\frac{L\times q\times r\times 2n}{L \times q\times r_{out}\times r_{in}\times 2}=\frac{r\times n}{r_{out}\times r_ {in}}.$ (5) $$

Since the inner rank ($r_{\text{in}}$) in the NoRA model is generally set to be the same as the rank ($r$) used in the LoRA model, NoRA demonstrates more pronounced advantages over LoRA under certain conditions. Particularly when the outer rank ($r_{\text{out}}$) is less than the maximum allowable rank ($r_{\text{max}}$), the NoRA model exhibits significantly enhanced parameter optimization. Specifically, the ratio of parameter efficiency between LoRA and NoRA can be approximated as follows:

$$ $\frac{P_{\text{LoRA}}}{P_{\text{NoRA}}}\approx\frac{n}{r_{out}}.$ (6) $$

## 4 Experiment

### 4.1 Fine-tuning of Large Language Models

Implementation Details.
We employed the LLaMA3 8B [] and LLaMA 7B models, each fine-tuned using a variety of parameter-efficient methods to enhance their commonsense reasoning capabilities. We utilized the Commonsense170K dataset [] for targeted fine-tuning, which is designed to enhance the models’ understanding of commonsense knowledge across different contexts. The main objective was to assess the effectiveness of each fine-tuning approach in improving the model’s performance on a range of commonsense reasoning tasks. Post-fine-tuning, model performance was evaluated using a suite of eight benchmark tests focused on commonsense reasoning, including ARC-e, OBQA, SIQA, and more.

Comparison Results.
The results from the experimental evaluations are detailed in Table 1. The fine-tuning methods demonstrated varying levels of success in enhancing the reasoning capabilities of the LLaMA models.

The LLaMA 7B model, when fine-tuned with the NoRA method, exhibited the highest average score of 75.8 across most tasks, demonstrating its superior ability to generalize across different question sets. Remarkably, it achieved top scores in HellaSwag (80.6), WinoGrande (79.6), and ARC-e (80.5), underscoring its robust understanding and reasoning capabilities. The NoRA method not only showed the highest scores but also significantly reduced parameter utilization, emphasizing its efficiency and scalability as a fine-tuning strategy. In contrast, other methods like LoKr and AdaLoRA, while effective in certain areas, required more parameters or resulted in higher computational costs. Notably, the NoRA method outperformed others in terms of parameter efficiency and computational overhead, making it a promising approach for practical applications where resource constraints are a concern.

These findings highlight the potential of NoRA as a scalable and effective fine-tuning strategy that not only achieves high performance across diverse datasets but also maintains a low parameter footprint, enhancing the practical usability of large pre-trained models in varied commonsense reasoning applications.

### 4.2 Fine-tuning of Vision-Language Models

Implementation Details.
The performance of various adaptation techniques on the Vision Transformer [] (ViT-B/16) was evaluated across three distinct datasets: Food, Pets, and DTD. Each dataset was chosen to test the robustness and adaptability of these methods across different visual domains. The primary performance metric used was the Top-1 accuracy, which was computed as an average over three random seeds to mitigate randomness and ensure reliability in the results. The experiments were conducted under two shot settings, 4 and 16 shots, to determine the effectiveness of each adaptation technique under limited data conditions.

The adaptation methods tested included various enhancements of existing techniques such as CoOp [], [] PLOT++ [], MaPLe [], and several variants of LoRA, such as CLIP-LoRA [], AdaLoRA [], DyLoRA [], and others. This comprehensive approach allowed for a detailed assessment of each method’s ability to improve model performance under constrained training scenarios.

**Table 1: Averaged accuracies (%) for 8 zero-shot tasks. Param denotes the number of trained parameters, Time for the training time on H800 GPU, and Mem for the GPU Memory usage.**
| Method | Param | Time | Mem | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Fine-tuning on LLaMA-1 7B |  |  |  |  |  |  |  |  |  |  |  |  |
| LoRAr=16 [] | 8.4M | 5.7h | 21G | 68.9 | 80.7 | 77.4 | 78.1 | 78.8 | 77.8 | 61.3 | 74.8 | 74.4 |
| LoRAr=32 [] | 16.8M | 6.5h | 27G | 68.5 | 81.0 | 77.4 | 77.1 | 79.0 | 77.8 | 63.3 | 77.9 | 75.3 |
| NoRA | 8.2M | 5.0h | 19G | 68.1 | 80.3 | 76.8 | 80.6 | 79.6 | 80.5 | 62.6 | 77.8 | 75.8 |
| Fine-tuning on LLaMA-3 8B |  |  |  |  |  |  |  |  |  |  |  |  |
| LoRA [] | 28.3M | 8.0h | 29G | 72.3 | 86.7 | 79.3 | 93.5 | 84.8 | 87.7 | 75.7 | 82.8 | 82.8 |
| LoKr [] | 0.9M | 26.3h | 66G | 65.1 | 81.6 | 78.7 | 92.0 | 82.1 | 89.2 | 76.7 | 80.9 | 80.9 |
| AdaLoRA [] | 28.3M | 12.5h | 58G | 75.1 | 86.4 | 76.7 | 75.4 | 83.3 | 90.4 | 79.1 | 81.4 | 81.4 |
| NoRA | 7.2M | 6.2h | 28G | 73.3 | 86.4 | 79.1 | 94.1 | 84.3 | 88.2 | 77.5 | 85.0 | 83.1 |

**Table 2: Detailed results for 3 datasets with the ViT-B/16 as visual backbone. Top-1 accuracy averaged over 3 random seeds is reported. Highest value is highlighted in bold, and the second highest is underlined.**
| Shots 4 |  |  |  |  |
| --- | --- | --- | --- | --- |
| Method | Food | Pets | DTD | Average |
| CoOp [] (4) | 83.5 | 92.3 | 58.5 | 78.1 |
| CoOp [] (16) | 84.5 | 92.5 | 59.5 | 78.8 |
| CoCoOp [] | 86.3 | 92.7 | 55.7 | 78.2 |
| TIP-Adapter-F [] | 86.5 | 91.9 | 59.8 | 79.4 |
| CLIP-Adapter [] | 86.5 | 90.8 | 46.1 | 74.5 |
| PLOT++ [] | 86.5 | 92.6 | 62.4 | 80.5 |
| KgCoOp [] | 86.9 | 92.6 | 58.7 | 79.4 |
| TaskRes [] | 86.0 | 91.9 | 60.1 | 79.3 |
| MaPLe [] | 86.7 | 93.3 | 59.0 | 79.7 |
| ProGrad [] | 85.4 | 92.1 | 59.7 | 79.1 |
| CLIP-LoRA [] | 82.7 | 91.0 | 63.8 | 79.2 |
| LoRA+ [] | 84.4 | 92.8 | 64.1 | 81.4 |
| AdaLoRA [] | 85.6 | 92.8 | 66.2 | 81.5 |
| DyLoRA [] | 87.0 | 92.4 | 64.9 | 81.4 |
| LoRA-FA [] | 86.7 | 93.0 | 64.4 | 81.4 |
| VeRA [] | 84.5 | 92.5 | 65.1 | 80.7 |
| NoRA | 87.1 | 93.1 | 65.2 | 81.8 |

Comparison Results.
The detailed results are presented in Table 2, which includes the Top-1 accuracy for each method across the three datasets under both 4-shot and 16-shot settings. Notably, the NoRA model consistently outperformed other adaptation methods in both settings, highlighting its superior adaptability and efficiency. In the 4-shot setting, NoRA achieved an average Top-1 accuracy rate of 81.8, slightly surpassing DyLoRA, the second-best method, by 0.2%. In the more demanding 16-shot setting, NoRA demonstrated even greater efficacy, achieving an average Top-1 accuracy of 85.4 and surpassing DyLoRA’s score of 85.0.

NoRA showed exceptional robustness across the various visual domains tested. Specifically, in the 16-shot setting, NoRA achieved the best results in all individual datasets, with scores of 87.8 for Food, 94.1 for Pets, and 74.3 for DTD. These scores underline NoRA’s capability to adapt to and perform well across diverse categories, ensuring high reliability for practical applications in fields requiring visual recognition.

Overall, the experimental outcomes underscore the efficacy of NoRA as a scalable and effective fine-tuning strategy, achieving high performance across diverse datasets and maintaining a low parameter footprint, which is crucial for practical deployments where computational efficiency is essential. These results confirm the importance of selecting appropriate adaptation techniques to enhance the performance of vision transformers in limited data environments.

Figure: Figure 2: Comparison of generated images from LoRA and NoRA on the subject-driven generation task.
Refer to caption: extracted/5816986/pic/demo2.jpg

### 4.3 Subject-driven Generation

Implementation Details.
We explored theme-based image generation by utilizing advanced text-to-image diffusion models, specifically employing a pre-trained text-to-image model. This model was fine-tuned using a combination of images and specific textual prompts (e.g., "[V] photo of a cat"). The fine-tuning involved sophisticated adaptation techniques, namely the LoRA and NoRA methods, to enhance the model’s capability to generate images that align closely with given themes.

The model used in our experiments, referred to as SDXL5, was fine-tuned on a 32G V100S GPU. The parameters adjusted during fine-tuning included setting the learning rate to 1e-4 and the batch size to 4. The NoRA method involved initializing the internal and external layers of LoRA with a differential diagonal matrix. The training was conducted over 500 steps, taking approximately 24 minutes to complete.

Comparison Results.
The results of the image generation process are detailed in Figure 2. During the image generation phase, we performed 50 inference steps for each textual prompt. Our findings indicate that the NoRA method outperformed the traditional LoRA method in several key aspects. Notably, the NoRA method was more effective in capturing complex themes and details within the images. The images generated using the NoRA method showed a higher degree of visual alignment with the specified textual prompts. They better reflected specific environmental contexts, such as forests and snowy scenes, and captured intricate object details, like wet fur.

These outcomes underscore the efficacy of the NoRA method in enhancing thematic consistency and visual expressiveness. The enhanced model demonstrates significant improvements in generating thematic images that are closely aligned with the nuances of the input prompts, offering promising applications in fields requiring detailed and context-specific image generation. This experiment sets a foundation for further exploration into fine-tuning techniques that can more adeptly handle complex thematic prompts in image generation tasks.

## 5 Conclusion

In this paper we present introduces NoRA, a new parameter-efficient fine-tuning method for large models. NoRA employs a dual-layer nested structure combined with SVD to extract latent information from the original matrix while reducing parameter count. The method retains LoRA’s parameter efficiency while improving low-rank matrix approximation accuracy through SVD. NoRA’s hierarchical structure allows for finer optimization control, enhancing model adaptability across tasks. Experimental results demonstrate NoRA’s effectiveness across various tasks, including commonsense reasoning in LLMs, VLMs and subject-driven generation. The significance of NoRA lies in its ability to enhance model adaptability and performance while maintaining excellent parameter efficiency. Our approach addresses the growing need for efficient fine-tuning methods in the era of large models.

Limitations of the study include potential computational overhead due to SVD calculations and the need for further investigation into NoRA’s scalability to even larger models. Future work could explore combining NoRA with AutoML [, , , , , ] and distillation techniques [, , , , ], applying it to multimodal models, and investigating its impact on model interpretability and robustness.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: The paper introduces NoRA as a two-layer nested low-rank adaptation structure. Have the authors considered exploring nested structures with more than two layers, and if so, what are the potential benefits or drawbacks? Could a deeper nested structure lead to improved performance, and if it does, is there a point of diminishing returns? Additionally, how does the computational complexity scale with the increase in the number of layers in the nested structure?
soundness: 2
strengths: 1、The paper provides a rigorous empirical evaluation of NoRA across a diverse set of linguistic and visual tasks, demonstrating its effectiveness and efficiency. The use of multiple benchmarks and the comparison against LoRA variants in various scenarios ensure that the results are robust and generalize well across different domains.
2、The paper employs a sound methodology, with a clear problem statement and a well-defined approach to address the challenges in fine-tuning large models. The activation-aware SVD (AwSVD) technique is a methodological innovation that leverages activation distributions for more accurate weight matrix initialization, which is a sophisticated approach to enhancing model performance.
3、 NoRA demonstrates a significant reduction in fine-tuning parameters, training time, and memory usage, which is a critical advantage in the context of large language models that typically require substantial computational resources. This resource efficiency makes NoRA particularly appealing for applications where computational budgets are limited.
summary: The paper titled "NORA: NESTED LOW-RANK ADAPTATION FOR EFFICIENT FINE-TUNING LARGE MODELS" introduces a novel parameter-efficient fine-tuning method named NoRA (Nested Low-Rank Adaptation) for large language models (LLMs). NoRA addresses the challenge of high computational demands and training costs associated with traditional fine-tuning methods by optimizing the initialization and fine-tuning of projection matrices. The authors propose an activation-aware Singular Value Decomposition (AwSVD) technique to enhance the initialization process and reduce output errors. Extensive experiments across various linguistic and visual tasks demonstrate that NoRA outperforms existing Low-Rank Adaptation (LoRA) variants in terms of efficiency and effectiveness, significantly reducing fine-tuning parameters, training time, and memory usage while enhancing performance.
weaknesses: 1、The core of the article appears to revolve around the activation-aware matrix, which is the foundation and heart of the entire method. However, the paper seems to lack a discussion on how to confirm that the activation-aware matrix used is superior, whether there are other methods available, and how to determine whether this matrix can provide more useful information. Moreover, the approach of merely performing singular value decomposition on the activation-aware matrix and then nesting LoRA matrices might appear to offer limited innovation.
2、To better understand the contribution of each component of NoRA, such as the nested structure and AwSVD, the paper would benefit from ablation studies. These studies would isolate the effects of different design choices and provide insights into which aspects are most critical for the performance improvements observed.
3、The paper mentions that the optimal hyperparameter configurations for NoRA may vary depending on the specific task and models. This sensitivity could be a limitation for users who need to fine-tune models for different applications.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: NA
soundness: 4
strengths: 1. The paper is well-written and organized, with an intuitive motivation.
2. The method is clever, leveraging observations on LLMs—particularly their sensitivity to activation outliers—to propose an improved LoRA initialization. The upgrade from standard SVD to activation-aware SVD (AwSVD) enhances performance and reduces optimization difficulty.
3. The NORA structure, based on AwSVD initialization, further reduces the number of learnable parameters, enabling more efficient and lower-cost training. It’s also simple to implement, requiring only a few lines of code modifications, making it easily deployable and practical for application.
summary: Through a comprehensive empirical analysis, this paper provides critical insights into initialization strategies, structural configurations, and design placements. It further introduces an **Activation-Aware Singular Value Decomposition (AwSVD)** method to reduce output errors and accelerate the training process.
weaknesses: 1. For instruction fine-tuning tasks, the paper only compares performance under settings with extremely low learnable parameters, which shows competitive results but falls significantly short of full-rank LoRA in performance. This raises concerns about whether the primary benefits of this work apply mainly to in-domain task transfers.
2. Is it necessary to reduce the number of optimization parameters in LoRA to save memory (particularly in the optimizer) or training time? After all, we don’t always need to compress parameter counts to such an extreme degree, and it often seems to be a trade-off. Unless it can be proven that this approach consistently outperforms various high- and low-rank LoRA fine-tuning methods, thereby serving as a superior replacement, the practical significance of this work remains in question.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to weaknesses.
soundness: 3
strengths: + The paper is well-structured and logically organized.
+ While some components are inspired by prior work, the integration of these elements is novel.
+ SoTA performance and low budgets.
+ Resonable motivations.
summary: This paper presents a nested low-rank adaptation method for LLMs. NoRA employd an inner layer while freezing the outer layer to enable precise task-specific adaptations while maintaining compact training parameters. Extensive experiment results demonstrated its effectiveness.
weaknesses: - The statement regarding NoRA’s rank enabling more complex non-linear transformations lacks theoretical grounding. The discussion around “expressiveness” in that section is underdeveloped. Simply stating that NoRA’s rank is limited by  \min(r, r{\prime})  does not elucidate how or why this rank impacts expressiveness. Thus, the authors should provide more experiments or theoretical explanations to demonstrate their claims.

- The decision to freeze the outer LoRA parameters to “maintain stability” lacks theoretical or empirical backing. Why this approach aids in stability needs elaboration？

- Comparative Analysis: A direct performance and parameter-efficiency comparison between LoRA and NoRA under similar parameter constraints would be insightful. This comparison would allow for a clearer understanding of how each approach performs relative to the other, given comparable resource budgets.

- In the visual section, the authors have conducted evaluations solely on a few simple classification datasets, which is insufficient. Additional complex vision-language tasks should be included, such as referring segmentation/detection, and visual caption tasks. This would better demonstrate the effectiveness of their proposed method.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. Line 186. ”PiSSA (Meng et al., 2024) selectively adjusts matrix ranks and distributions”. PISSA is a method focusing on initialization strategy. I would like authors to explain why PISSA can adjust matrix ranks, thanks.
2. Line 225. Why keeping the parameters of outer LoRA frozen can maintain stability? Could the authors provide theoretical justification or empirical evidence?
3. Line 224. “matrix B is initialized with UΣ”. What is U and Σ? It is the first time that these symbols appear, but they are not explained (as well as V and S).
4. Line 274-276. The adapter consists of two projection matrix and a non-linear(ReLU) layer. The adapter should be represented as Wx+B(ReLU(Ax)). Besides, what is a parallel LoRA? To transfer the matrix B to another LoRA layer? Please provide a clear definition and explanation.
soundness: 1
strengths: The experiments are extensive.

The procedure of method is clear.
summary: This paper proposes a new PEFT method NoRA, as well as an initialization strategy.
weaknesses: Too many weaknesses led me to choose to reject this paper. Furthermore, I believe this paper requires at least one major revision before it can be considered for a top-tier conference.

1. Line 43-44. “which can lead to slow convergence and potential overfitting problems.” The authors claimed two main approaches have emerged to address the aforementioned issues. However, DoRA cannot address them [1-2]. Indeed, DoRA sometimes easily fall into not converging.
2. Line 12, “but it still necessitates a substantial number of training parameters. To address this issue…”.  Line 53-86 “two significant challenges persist for these LoRA variants….To address these challenges…” In fact, I cannot understand why the authors conduct experiments in Fig.1 (a)(b). Even as the authors claimed in Line53-86, their experiment cannot answer problem 1, i.e., “the intrinsic properties of LLMs…..decomposition errors”
3. Line 248-254. It is better for the authors to provide more theoretical details on the design of activation-weight matrix and W_{aw}. 
4. Line 281-290: Why NoRA’s rank allow for more complex non-linear transformations? I do not think this part “expressiveness” make sense. What can be concluded from the fact that the rank of NoRA is bounded by min(r, r′)? “Generation” part also makes no sense. And indeed, I believe the whole subsection 3.4 should be given further consideration.
5. I would like to see a comparison between LoRA and NoRA with similar parameter budgets.

Overall, the introduction of the paper is unclear, the motivation is not well-defined, the rationale behind the design in the methods section is unclear, and there are some issues with the effectiveness (Sec.3.4) of the approach.

[1] 2024, ICML, DoRA: Weight-Decomposed Low-Rank Adaptation

[2] 2024, arxiv, FLoRA: Low-Rank Core Space for N-dimension

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Already listed in the Weaknesses section
soundness: 2
strengths: **1. Clear and Well-Structured Writing**

- The paper is well-written, with a clear and logical structure that makes it easy to follow. Concepts are explained in a straightforward manner, and the overall organization helps the reader grasp the technical content effectively.
- Figures and illustrations are clean, well-labeled, and support the text, helping to visually convey the architecture and results clearly.

**2. Innovative Techniques for Performance Improvement**

- The proposed nested LoRA structure, combined with the activation-aware Singular Value Decomposition (AwSVD) initialization, significantly enhances fine-tuning performance. This approach not only reduces the number of trainable parameters but also improves the model’s efficiency and adaptability to different tasks.
  
**3. Strong Performance Gains Over Comparable Methods**

- In comparison to other ultra-low parameter methods such as LoRA-XS and VeRA, NoRA demonstrates substantial performance improvements, particularly on challenging benchmarks like GSM8K and MATH. These results highlight the method's effectiveness in improving accuracy while maintaining parameter efficiency.
summary: The paper introduces NoRA (Nested Low-Rank Adaptation), a parameter-efficient fine-tuning method for large models like Mistral-7B, Gemma-7B, and LLaMA-3 8B. It addresses the limitations of traditional LoRA (Low-Rank Adaptation), which involves tuning a large number of parameters, by proposing a nested structure that reduces parameter count while maintaining model adaptability and performance. Key contributions include:
- NoRA Architecture: A nested structure where outer LoRA layers are initialized using an activation-aware Singular Value Decomposition (AwSVD) to reduce decomposition errors, and inner LoRA layers are fine-tuned with fewer parameters, improving efficiency.
- AwSVD: An innovation that adjusts weight matrices based on activation distributions, ensuring higher fidelity to pre-trained weights and faster convergence during fine-tuning.
- Performance Improvements: NoRA significantly reduces fine-tuning parameters, memory usage, and training time while enhancing task performance. It outperforms other LoRA variants, achieving superior results across linguistic and visual tasks with fewer trainable parameters.

The paper demonstrates NoRA's efficiency through experiments, showing improvements in performance while reducing training-time and memory usage. The paper concludes by highlighting the advantages of NoRA in terms of expressiveness, flexibility, and parameter efficiency, positioning it as a robust method for fine-tuning large-scale models.
weaknesses: **1. Limited Scope of Comparative Analysis**

- The unified design space presented in the paper is not comprehensive enough. It primarily focuses on VeRA and LoRA-XS approaches, lacking coverage of other significant approaches in this domain.
- A comprehensive table summarizing the design choices of previous works is missing. Such a table would enhance the clarity and depth of comparisons.
- The comparison provided in Figure 2 resembles an ablation study of the proposed techniques rather than a thorough comparison of prior approaches. It would benefit from including more diverse methods in the analysis.

**2. Potential Compatibility with Other Approaches**

- There is no discussion of the compatibility of the proposed method with orthogonal approaches such as AdaLoRA and DoRA. Exploring how NoRA could integrate with or complement these methods could provide valuable insights.
  
**3. Theoretical and Intuitive Justifications**

- The paper introduces the concept of applying a scaling matrix to mitigate decomposition errors, but the intuition behind this approach is not clearly explained. A more rigorous theoretical justification is necessary.
- For AwSVD, it remains unclear whether this technique requires a large calibration set to perform effectively.

**4. Sensitivity and Practical Concerns**

- The results might be sensitive to batch size, but this aspect has not been explored in detail. A discussion on how batch size affects performance would strengthen the paper.
- It is also unclear how to select input activations from the fine-tuning dataset, which could impact the practical usability of the method.

**5. Rigor and Accuracy of Claims**

- Several statements lack rigor and precision. For instance, the claim regarding Formula 10 suggests that a tighter rank constraint leads to more complex non-linear transformations, but this assertion is misleading. A tighter rank constraint should not necessarily imply increased complexity.
- The explanation provided in line 308 on how to approximate the parallel LoRA form by rearranging terms is vague and needs further clarification.

**6. Issues with Reported Results**

- Table 1 only lists DoRA and LoRA as having rank=1 which is not a practical and feasible setting. 
- Additionally, the accuracy of DoRA reported in Table 2 is inconsistent with the values in the original paper. The correct accuracy should be 85.3%, not 83.0%.
- A more holistic figure illustrating performance across different ranks would provide a clearer understanding of the flexible trade-off between expressiveness and efficiency.

**7. Novelty and Evaluation**

- The novelty of the proposed method could be questioned, as the core idea of SVD decomposition has already been explored and analyzed by LoRA-XS and PiSSA.
- The subject-driven image generation task appears selective, possibly bordering on cherry-picking. It would be beneficial to include qualitative results on widely used benchmarks, such as DreamBooth, to ensure a more objective evaluation.