Low-Rank Interconnected Adaptation across Layers

Authors: Yibo Zhong, Yao Zhou
Primary area: foundation or frontier models, including LLMs
Keywords: parameter-efficient fine-tuning, low-rank adaptation, foundation model

Abstract:
Low-rank adaptation (LoRA) is a powerful parameter-efficient fine-tuning method that utilizes low-rank projectors $A$ and $B$ to learn weight updates $\Delta W$ for adaptation targets $W$. However, while the low-rank structure of $A$ and $B$ enables high hardware efficiency, it also restricts the overall weight update to be low-rank, which limits the adaptation performance. In this paper, we propose \underline{l}ow-rank \underline{i}nterconnected adaptation across \underline{l}a\underline{y}ers (Lily). Specifically, we employ a hierarchical framework where low-dimensional projectors (LPs) retained for downward projection at a particular level, while globally-shared high-dimensional projector (HP) experts perform upward projection across all levels of layers. This interconnected asymmetric structure makes the adaptation much more dynamic and breaks the low-rank weight-update constraint of LoRA when using the same parameters budget. Furthermore, Lily's cross-layer connections facilitate the capture of intricate information and dependencies across different layers, thereby enhancing the model's representational capabilities. Experiments across various modalities, architectures, and model sizes underscore Lily's great performance and efficiency.

Decision: reject
Reviewer scores: 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729618625828

thread_type:
official_review

<thread_object>
creator: reviewer_dxjd

<content>
summary:
The Authors propose an alternative ansatz to LoRa for parameter efficient finetunig of neural networks. In particular, they propose to connect all low-rank adapters of a given network to each layer,  combined with a router. This way a weighted sum of low-rank adapters is modifying the parameters of a given layer. 
The authors demonstrate the method in an arrray of contemporary benchmarks.

strengths:
I find the idea of using a sum of low-rank adapters interesting. 
The ansatz potentially enables a high rank adaptation using a sum of low-rank adapters. 

The method is demonstrated at an impressive number of numerical tests.

weaknesses:
While the general idea is promising and supported by numerical evidence, I find the presentation of the method description and theoretical premisses thereof lacking in soundness and readbility. In my opinion the first part of the paper requires a major revision before being publishable. 

In particular: 

- The work is based upon the premise that the B matrix is very close to 0 or equals to zero. This claim has been investigated by Hao et al. however under strong assumptions about the zero initialization of B and very small learning rates and only for the "classical" LoRa setup. 
The authors of this work just use the conclusions of Hao's analysis as the premise for their method. Two points are critical here
 1) The assumptions in the LoRa setting are not well communicated in both Introduction (Section 1) and Section 3.1. In Section 3.1 the mathematical notation is very sloppy, using equality signs between eq. 2 and 3, when this is in reality a first order approximation for small learning rates and values of B. This is missleading, especially for readers not familiar with Hao's work. The same applies for eq. 9 and 10. 

2) The validity of the assumptions need to be shown for the newly introduced ansatz and method of this paper, instead of just taken for granted. 

- The authors claim that their ansatz (eq. 14). allows high rank updates with low-rank matrices. While it is generally true, that a sum of (even) rank 1 matrices can be high rank, it does not have to be the case. An (very simple) numerical experiment is needed to demonstrate the effectivity of this ansatz. One could think of a matrix regression problem with said ansatz. A stage 2 numerical experiment would be the anlysis of the effective ranks of each layer of an adaption of the proposed kind. 
The same ciritique is applicable for the MoE ansatz in the paper. 

- The method should be illustrated with an algorithm. 

- The effective computational efford of the method needs to be analyzed. Having strong interconnections between layers significantly increases the complexity of the gradient tape, which increases compute and memory consumption of the method, potentially defeating the purpose of PEFT. 

- Some statements are quite vague, e.g. Line 226: "shallow and deep inputs" what does that mean?

questions:
- Does the method require that exactly L (where L is the number of layers) low-rank adapters need to be combined? I think the notation can be simplified by defining the number of adapters independently of layers, especially for the MoE ansatz.

- Have you measured the effictive compute batch training time in comparison with adalora or lora?
- Can your method be extended to be rank adaptive? 
- How does the norm of B change during training? Your method is based upon the assumption that the values of B are very small. Is this still the case after several epochs? In this regard: What are the practical thresholds for the learning rate, where the method assumptions breal?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730374816578

thread_type:
official_review

<thread_object>
creator: reviewer_f3lx

<content>
summary:
This paper investigates the limitation of weight update in LoRA the limitations of weight updates in LoRA, highlighting that it functions as a random projection on an initialized low-rank matrix. To address such limitations, the author introduces low-rank interconnected adaptation across layers (Lily), which employs model-wise shared low-rank matrices inspired by the mixture of experts (MoE) framework. Extensive experiments have been conducted to demonstrate the effectiveness and adaptability of the proposed methods.

strengths:
1. The motivation is reasonable as learning multiple shared low-rank matrices can represent various low-rank subspaces, and their combinations have the potential to capture information across a higher-rank space.

1. Comprehensive experiments demonstrate the effcacy and adaptability of Lily, achieving state-of-the-art results across a diverse set of tasks in both language and vision domains.

weaknesses:
1. The presentation of the methodology has significant room for improvement. 

    1. The description of the methodologies (lines 204-210) and the framework depicted in Figure 1 within the main text differ from the final implementation discussed in Appendix A.1.1 (lines 770-778). In the main text, low-dimensional projectors (LPs) are described as being tied to each layer of a module, while high-dimensional projectors (HPs) are shared across the model. However, Appendix A.1.1 indicates that Lily essentially learns a set of model-wide shared LPs and HPs that are selectively applied to specific layers.

    1. The derivations from Eqn. 9-14 lack rigor in scenarios involving multiple LPs and HPs. To extend the derivation from Eqn. 5-20 in [1] to multiple LPs and HPs, several adjustments are needed: 

         - **First**, Eqn. 9 should incorporate a summation over the LPs, leading to a double summation in Eqn. 11. 

         - **Second**, gradient with respect to each $B_{i}$ should be considered. 

         -  **Third**, the applicability of Theorem 2.1 in [1] for multiple LPs and HPs must be proven before use. 

         Without proper proofs, simple substitutions may lead to inaccuracies, making the current mathematical discussion unconvincing.

    1. Eqn. 17 - 18 are just simple math that does not warrant separate discussion. In addition, the definition of $s$ in Eqn. 19 - 20 is missing.

    1. In essence, it is advisable to directly present the core idea (a set of multiple LPs and HPs) if the theoretical discussion lacks sufficient proof.

1. Concerns regarding experimental results.

    1. The reported number of trainable parameters for Lily is unclear. 

        - **First**, Lily is reported to have the same parameter count as LoRA on GLUE tasks while using only 1/4 of the parameters compared to LoRA on ViT tasks. Given that vision tasks are typically more complex than GLUE tasks and generally require more parameters, this discrepancy needs clarification.
        
        - **Second**, based on the configurations in Tables 9 and 10, Lily only uses 2 LPs and 2 HPs shared across the entire model. Under this setup, the parameter count should be substantially lower than that of LoRA for various backbones, such as RoBERTa-base/ViT-base (12 layers), yet Lily's parameter count appears comparable to LoRA on GLUE. Further explanation is needed to resolve this inconsistency.

    1. Efficiency is a critical aspect of PEFT. However, for the most computationally demanding task, commonsense reasoning, the parameter count is not provided. It would be helpful to include the parameter count and training time for this task, as shown in [2].

    1. The experimental results do not include MNLI and QQP, despite these tasks being listed in the configuration table.

[1] Yongchang Hao, et al. FLORA: Low-Rank Adapters Are Secretly Gradient Compressors. In ICML 2024.

[2] Taiqiang Wu, et al. Mixture-of-Subspaces in Low-Rank Adaptation. Arxiv.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697444455

thread_type:
official_review

<thread_object>
creator: reviewer_hxwk

<content>
summary:
This paper proposes a novel hierarchical adaptor for fine-tuning the Transformer. The results show a high advance of proposed method on the NLU and computer vision tasks.

strengths:
1. The proposed methodology that considers cross-layer interactions is new to me.
2. The empirical results show the method's potential in various domains.

weaknesses:
The major flaws are about the presentation.

1. Some derivation steps could be merged and only the important steps can be given. The blanks could be reserved for discussing the intuitions and analysis.
2. The fonts in the figures are too small. Normally they should be larger than the smallest font in the main body.

questions:
How does the proposed method affects the efficiency of the adaptation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704774383

thread_type:
official_review

<thread_object>
creator: reviewer_njua

<content>
summary:
In this paper, the authors propose a parameter-efficient fine-tuning method based on the concept of LoRA. Specifically, they introduce a technique to construct the B matrix through a weighted combination of inter-layer weights during inference. This approach allows the adapter to overcome the low-rank limitation, improving its expressiveness. The proposed method is motivated by findings that suggest LoRA functions as a gradient projector, with updates primarily influenced by the initialization of the A matrix. The authors evaluate their method on both transformer-based and Mamba-based models, comparing it with baseline methods across various NLP and image tasks. Additionally, they conduct ablation studies and qualitatively assess the learned features.

strengths:
- Overall, this paper is well-written. The authors provide a comprehensive description of the proposed method, and the appendix is detailed.

- The authors apply the proposed method to various models across different modalities, demonstrating its effectiveness. They include both quantitative and qualitative evaluations.

- The proposed method is inspired by a thorough analysis of the existing LoRA approach.

weaknesses:
- Since the authors derived the forward path of the proposed method in Section 3.2, would it be possible to analyze the rank gain? A theoretical analysis of the improvement assuming a single layer could be helpful.

- It would be beneficial if the authors could further evaluate the improvement qualitatively, as the current gains are not particularly substantial. For example, in the RoBERTa-base experiments, the proposed Lily method only outperforms others on 2 out of all tasks. Additionally, AdaLoRA does not outperform standard LoRA, which is somewhat counterintuitive. Could the authors explore conducting experiments on larger BERT models? Given that different PEFT methods involve varying numbers of learnable parameters, is it fair to compare these methods directly in light of these parameter differences?

- The authors have renamed the A and B matrices in LoRA as the downward low-dimensional projector and upward high-dimensional projector, respectively.

questions:
- The authors might want to discuss the difference of the proposed method with several related recent works. For example, in [HydraLoRA], the authors introduced a LoRA MoE method which share the same A matrices but also a mixture of B matrices. In [LoRA Asymmetry], the authors proposed freezing A matrices and only training B, backed by a another line of theoretical study. Another related works, [LoRA+], the authors analyze the gradient of LoRA matrices and propose to use a larger learning rate for B to improve the performances. 

[Tian, Chunlin, et al. "HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning." arXiv preprint arXiv:2404.19245 (2024).]

[Zhu, Jiacheng, et al. "Asymmetry in low-rank adapters of foundation models." arXiv preprint arXiv:2402.16842 (2024).]

[Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. "Lora+: Efficient low rank adaptation of large models." arXiv preprint arXiv:2402.12354 (2024).]
</content>
</thread_object>

</thread>
<thread>
creation date:
1732092288878

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Validate Our Claim and Invite Reviewers to Discussion

comment:
We thank again for all the reviewers' insightful and helpful feedback!  To summarize, the reviewers highlight the strengths of our paper, including the **novelty of inter-layer interaction, reasonable motivation, comprehensive numerical analysis across multiple scenarios, and the generally promising potential of our proposed method.**

However, the reviewers also point out that the current theoretical analysis (Section 3.1) has several presentation issues, lacks sufficient and rigorous proof, and relies on some strong assumptions based on prior works. This makes our claim that "Lily achieves weight updates with higher rank than LoRA" less convincing from a theoretical perspective.

We do acknowledge that this analysis lacks sufficient proof and is based on some strong assumptions. Therefore, we decide to remove this section from the main text due to its current immaturity. **However, we actually test the rank of the weight updates of Lily and LoRA in multiple real-world experiments, validating our claim that "Lily achieves weight updates with higher rank than LoRA" from a much more convincing practical and empirical perspective. We believe that the removal of immature theoretical analysis does not undermine our claim and the empirical analysis, combined with our existing experimental results, further demonstrates the advantage of our proposed method.**

Here we provide the empirical rank analysis of Lily and LoRA. To be specific, we use the MRPC, COLA, SST-2 and STS-B tasks from the NLU experiments and test the rank of the $W_q$ when running 20 epochs for the former three tasks and 3 epochs for STS-B. The results below generalize well to other targets and layers as well. For LoRA we use a hidden dimension of 8 and for Lily we use 32 and 8, respectively. We use only a small number of LPs and HPs (2 or 3) in the experiment for simplicity and to make our claim more convincing. The weight updates here are of shape [768, 768]. **We will turn these analyses into figures and include them in a section in the main text to provide strong empirical evidence of our claim.**

MRPC:

| method | param | layer1 | layer2 | layer3 |
| ------ | ----- | ------ | ------ | ------ |
| LoRA   | 0.3M  | 27     | 29     | 24     |
| Lily   | 0.3M  | 224    | 225    | 222    |
| Lily   | 0.05M | 75     | 76     | 77     |

COLA:

| method | param | layer1 | layer2 | layer3 |
| ------ | ----- | ------ | ------ | ------ |
| LoRA   | 0.3M  | 56     | 34     | 37     |
| Lily   | 0.3M  | 313    | 312    | 324    |
| Lily   | 0.05M | 105    | 101    | 101    |

STS-B:

| method | param | layer1 | layer2 | layer3 |
| ------ | ----- | ------ | ------ | ------ |
| LoRA   | 0.3M  | 32     | 29     | 28     |
| Lily   | 0.3M  | 214    | 228    | 194    |
| Lily   | 0.05M | 151    | 148    | 151    |

SST-2 (3 epoch)

| method | param | layer1 | layer2 | layer3 |
| ------ | ----- | ------ | ------ | ------ |
| LoRA   | 0.3M  | 50     | 54     | 40     |
| Lily   | 0.3M  | 363    | 355    | 351    |
| Lily   | 0.05M | 109    | 115    | 114    |

**Additionally, we want to kindly invite all the reviewers to participate in the discussion. We provide detailed responses to all the concerns and we are determined to address all of them!**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732373646502

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
revised paper is out and we want to hear from you

comment:
Dear reviewers and area chairs:

Thank you for your precious time and effort put into this work. We upload the revised version of the paper. One important thing to note is that changes are minor, with no changes to the central claim, motivation, experiment, method and contributions.

To summarise:
- Removal of the theoretical analysis and statements regarding the "gradient compressor" terminology. Key thing is that **originally, this part is trying to illustate why Lily has high-rank weight updates. Now with the inclusion of actual experiment testing the rank, we directly remove it from the paper.**
- Reminder that LPs for Lily can be flexbly set. (**one sentense in green**).
- RoBERTa-Large experiment (caption with blue text highlighted)
- Actual analysis of the rank for weight updates. (blue text).
- Hardware efficiency. (runtime or GPU memory consumption, blue text) 
- Bigger font size for the figures.
- New citation to Hydra-LoRA (red text).

Again, the changes are minor and we want to restate that **removing the theoretical introduction changes nothing about this paper's central claim, motivation, experiment, method and contributions.** Thank you again for reviewing this paper. If you have further concerns, please respond and we will adress them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732439904823

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Examining Our Paper's Components to Identify Whether Major Revisions Exist

comment:
Dear reviewers and area chairs,

We examine components of the paper to help you identify whether major revisions of the paper exist. (TL;DR: major revisions do not exist in our revision)

### Motivation 1: LoRA has low-rank weight updates. 
 
 Previously this part is **introduced** with the "gradient compressor" part. However, **removing it in the revised version does not change the fact that LoRA has low-rank weight updates, because this is not a conclusion of any other papers but rather directly observed.** **This is not major revision bacause the motivation is essentially the same (LoRA has low-rank weight updates), but without the previous introduction.**

### Motivation 2: Information for LoRA target is limited in a particular layer. 
This is also a **fact that does not depends on any works**, which can be observed in LoRA. (i.e., A and B are in a particular layer in LoRA). **This motivation leads to our inter-layer connection design.** We previously dicussed this in section 4.5 and now restate this in introduction. **This is also not major revision because this is already discussed in the paper and we simply restate it multiple times in the introduction.**

### Method: **No revision at all**. 

One sentense is added to ask readers to read appendix to know how LPs can be set. This changes nothing, bacuase the appendix is also the same without any changes. **We kindly remind you that we never state that number of LPs should be the same as layer numbers and how to set LPs is not the core innovation of this paper, rather it's the HP and Routers.** ***Also, we already inform the readers how LPs are set in the appendix (submitted version).***

### Experiments: 
  - Add actual rank analysis, this serves as analysis of why Lily has high-rank weight updates. **This is not major revision bacause this is a supplementary analysis.**
  - Bigger fonts of the figures. **This is not major revision because the content of the figures is the same, but with bigger fonts.**
  - RoBERTa-Large experiment. This is a supplementary experiment. **Zero new claims associated with it is proposed.**
  - Hardware efficiency analysis. **Also supplementary analysis to help readers understand the runtime and memory.**

### Claims and Contributions:
  - Lily has high-rank weight updates. **No revision at all**. Demonstrated by rank analysis.
  - Cross-layer interaction. **No revision at all**. This is discussed in section 4.5
  - Extensive experiments are conducted across various modalities, architectures, and model sizes. **No critical experiments are added.**

In general, the revision is mainly rephrasing the motivation and removing section 3.1. None of these changes the central component of the submitted paper. Regarding section 3.1, it is originally introduced to illustrate why Lily has high-rank weight updates and **has nothing to do with our contributions**. Now with the empirical analysis, this part is in fact not needed (plus many reviewers point out its incorrectness). **The conclusion of this post is that major revisions do not exist in our revision.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1733162454954

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Dear reviewer HxWK, F3Lx and dXjD, please respond or consider raising your score

comment:
Dear reviewer HxWK, F3Lx and dXjD,

Thank you for your precious time and effort. In the last day of the discussion, we kindly and finally ask you to respond to us. If we have addressed your concerns, please raise your score! Thank you again for your attention.

Authors of submission #3759
</content>
</thread_object>

</thread>
<thread>
creation date:
1733204302527

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Discussion Phase

comment:
Dear Reviewers and Area Chairs,

We would like to thank you for your efforts. Below, we provide a summary of this discussion phase.

The initial review of this paper pointed out **several merits, including reasonable motivation (reviewer F3Lx), a novel approach (reviewer HxWK), comprehensive experiments (reviewers Njua and dXjD), and, in general, a promising contribution.**

The main weakness many reviewers pointed out is the presentation, particularly referring to the theoretical analysis of why Lily has high-rank weight updates. We acknowledge the immaturity of our analysis and have replaced it with a stronger empirical analysis. We believe this does not count as a major revision because:
1, **The high-rank weight updates are demonstrated by real-world experiments.**
2, The previous theoretical analysis did not contribute significantly to the overall contribution of our paper.

In general, the paper proposes a promising PEFT method with great performance, and its original weakness was mainly about its presentation. However, **after minor revisions, the presentation is now clear and well-written. Therefore, we have successfully addressed the concerns raised by the reviewers.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes low-rank
interconnected adaptation across layers to handle the existing restriction of LoRA. After the response, it receives mixed ratings, including three rejects, and one accept. The advantages, including the various combinations of models and the interesting idea of cross-layer interaction, are recognized by the reviewers. However, they are also concerned about the presentation of the methodology, unsatisfying experimental results, lack of validity of the assumptions, etc. 

Though the authors make great efforts to provide detailed revision, two reviewers think the work requires substantial revision that exceeds the scope of minor adjustments during the rebuttal phase. And the main concerns are not well addressed in the response. After carefully reading the paper and reviews, I agree with them. For the revised paper, the full METHODOLOGY part is less than one page, which can support the above opinion from my point of view. 

In summary, I also think the current manuscript does not meet the requirements of this top conference. I suggest the authors carefully revise the paper and submit it to another relevant venue.