Low-Rank Interconnected Adaptation across Layers
Authors: Yibo Zhong, Jinman Zhao, Yao Zhou
Accepted at: acl25

Low-Rank Interconnected Adaptation across Layers

Yibo Zhong1, Jinman Zhao2, Yao Zhou1*†,

1Sichuan University, 2University of Toronto,
zhongyibo@stu.scu.edu.cn   yaozhou@scu.edu.cn

Abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method that learns weight updates \( \Delta W = AB \) for pre-trained weights \( W \) through low-rank adapters \( A \) and \( B \). While LoRA ensures hardware efficiency, its low-rank weight updates limit adaptation performance. In this paper, we propose low-rank interconnected adaptation across layers (Lily), a novel PEFT method that introduces an interconnected framework with locally shared \( A \) and globally shared \( B \) experts. This structure eliminates redundant per-layer \( AB \) pairs, enabling higher-rank \( \Delta W \) with equal or fewer parameters. To enhance expressiveness, we use data-dependent routers to determine \( A\)-\(B\) interconnections, preventing \( B \) experts from converging to the same behavior and improving representational power across domains. Experiments across modalities, architectures, and model sizes demonstrate Lily’s superior performance and efficiency. Github

1 Introduction

Fine-tuning foundation models like Transformers (Vaswani et al., 2017) on downstream tasks is common but costly, especially for large models like LLMs, which incur high computational and storage demands and risk catastrophic forgetting (Biderman et al., 2024). Linear probing alleviates these issues by fine-tuning only the final modules, but suffers from performance loss due to frozen backbone weights. To address this, parameter-efficient fine-tuning (PEFT) freezes the backbone and introduces lightweight modules for task-specific learning. Among PEFT methods, Low-rank Adaptation (LoRA (Hu et al., 2021)) is widely used, particularly for LLMs. LoRA introduces low-rank projection matrices, \( A \) and \( B \), to approximate weight updates \( \Delta W \), achieving significant savings in computation and storage while outperforming linear probing by updating the backbone weights.

However, LoRA and its subsequent improvements (Miles et al., 2024; Zhang et al., 2023; Zhong et al., 2024) face a limitation: the learned weight updates \( \Delta W \) are constrained to be low-rank, limiting model performance. A key issue is that LoRA allocates the same parameter budget to each layer, regardless of their importance (Fig. 1). As a result, the rank of each adapter is constrained by the fixed budget, raising the critical question: Can we enable more dynamic, expressive adaptation with high-rank weight updates under the same parameter budget?

In this paper, we propose Low-rank interconnected adaptation across layers (Lily), a novel framework for more expressive and

Figure 1: Dynamics of LoRA and Lily. In this 6-layer example with a fixed overall parameter budget, LoRA allocates the same parameter budget to each layer, resulting in small rank updates for the weights. Lily overcomes this by employing a small number of shared adapters with a much larger rank, achieving higher-rank updates while using the same or even a smaller parameter budget. Considering the different characteristics, and to make the adaptation more dynamic, the adapters are mixed according to a data-dependent router, represented by \( R \).

![Dynamics of LoRA and Lily. In this 6-layer example with a fixed overall parameter budget, LoRA allocates the same parameter budget to each layer, resulting in small rank updates for the weights. Lily overcomes this by employing a small number of shared adapters with a much larger rank, achieving higher-rank updates while using the same or even a smaller parameter budget. Considering the different characteristics, and to make the adaptation more dynamic, the adapters are mixed according to a data-dependent router, represented by R.](page_900_350_540_380.png)
efficient PEFT. Specifically, we decouple \( A \) and its corresponding upward \( B \), eliminating their tight coupling. Each \( A \) is connected to all \( Bs \), and vice versa, as illustrated in Fig. 1. This creates a hierarchical structure where locally-shared \( As \) perform downward projections at specific layers, while globally-shared \( Bs \) perform upward projections across all layers. To enhance dynamism, we selectively connect each \( A \) with \( Bs \) based on layer features. \( A \) extracts features from the current layer, and a selective mixture of \( Bs \) is performed based on these features, enabled by routers (Shazeer et al., 2017) that generate data-dependent weight distributions for \( B \) experts.

The interconnected structure makes the adaptation process more dynamic and flexible, with rich interactions between adapters. By reducing the number of adapters and increasing their rank, Lily achieves higher-rank weight updates than LoRA while using the same or fewer parameters. Additionally, Lily enables comprehensive information access and learning by allowing adapters at each layer to collaborate, share knowledge, and model dependencies across layers. Our key contributions include:

• We propose Lily, a novel PEFT framework that introduces interconnected adapters, effectively overcoming the limitations of low-rank weight updates in LoRA under the same parameter constraints.

• Lily utilizes routers to dynamically select and connect an adapter \( A \) with multiple adapter \( B \) experts, enabling richer information flow and more expressive adaptation dynamics.

• Extensive experiments are conducted across diverse modalities, architectures, and model scales, demonstrating Lily’s superior performance and efficiency in a wide range of scenarios.

2 Related Work

Parameter Efficient Fine-Tuning Foundation models are typically pre-trained on large datasets and fine-tuned on downstream tasks. Parameter-efficient fine-tuning (PEFT) seeks to fine-tune models efficiently with minimal parameters while maintaining performance and preserving learned knowledge. It effectively addresses limitations of conventional fine-tuning techniques, like full fine-tuning or linear probing. Current PEFT approaches can be divided into two categories: 1) adapter-based methods (Hu et al., 2021; Chen et al., 2022; Pfeiffer et al., 2020a; Jie and Deng, 2023; Houlsby et al., 2019a; Zhong and Zhou, 2025) and 2) prompt-based methods (Tu et al., 2023a,b). Adapter-based methods insert lightweight adapters into the Multi-Head Self-Attention (MHSA) or Feed-Forward Network (FFN) blocks of the Transformer architecture, while prompt-based methods add trainable tokens to the input sequence.

Among these, low-rank adaptation (LoRA (Hu et al., 2021)) is a well-known technique. It introduces projection matrices \( A \) and \( B \) for each adaptation target \( W \), where \( A \) projects input \( x \) to a low-dimensional space and \( B \) restores it to the original dimension. The product of these matrices approximates the weight update \( \Delta W \) in full fine-tuning (FFT). However, this limits the update to a low-rank subspace, which may affect performance. Additionally, \( A \) and \( B \) are tightly coupled, restricting the adaptation process to information from the current layer, which may hinder the modeling of dependencies across layers.

Mixture of Experts Mixture of Experts (MoE) is an active research area that has received significant attention, especially in the field of large language models (LLMs). Conditional computation, which activates different parts of the network on a per-example basis, has been proposed to enhance model capability without increasing computation (Davis and Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Almahairi et al., 2016) . The sparsely-gated MoE layer is introduced to implement this idea, consisting of numerous sub-networks (Shazeer et al., 2017). A trainable gating network (router) determines the combination of experts for each example. There are already PEFT methods like MoLORA (Zadouri et al., 2023) and MOLA (Gao et al., 2024a) that apply the MoE design concept to PEFT. However, these methods simply treat the adapters \( A \) and \( B \) combined in LoRA as a single expert. Concurrent research Wu et al. (2024) utilizes \( A \) and \( B \) sub-spaces as the experts but fails to overcome the limitation discussed in the previous section. Another concurrent work, HydraLoRA Tian et al. (2024), explores an asymmetric design for LoRA. Unlike our work, we consider the interconnection across layers and deploy a model-wide asymmetric design to enable cross-layer connections. This enables the use of adapters of higher rank than a typical LoRA setup while using the same or fewer overall parameters.
3 Methodology

3.1 Downward Projection and Selective Weight Allocation

The process is illustrated in the right half of Fig. 1. Initially, we use an \( A \) to project the input \( x \in \mathbb{R}^{N \times C_{\text{in}}} \) into its low-dimensional representation \( x' \in \mathbb{R}^{N \times d} \), where \( N \) is the sequence length:

\[
x' = xA
\] (1)

To enable more parameter efficiency, the number of \( A \)s can be set to less than the number of layers in the model by sharing the same \( A \) across neighboring layers, as illustrated in Fig. 1 and discussed in A. Inspired by the Mixture of Experts (MoE) paradigm, we employ a router \( R \in \mathbb{R}^{N_e \times d} \) to selectively assign weights to all \( B \) experts based on their relationship to the current layer’s features (\( x' \)), where \( N_e \) represents the number of \( B \) experts. A weight set \( S \in \mathbb{R}^{N_e} \) is obtained as:

\[
S = \operatorname{softmax} \left( \sum_{i=1}^N (x' R^T)_i \right)
\] (2)

The router selectively mixes the experts based on this data-dependent weight distribution, enabling information integration and expressive adaptation.

3.2 Weighted Mixture of Experts and Upward Projection

Once we obtain the low-dimensional input \( x' \), we combine information from all layers using the model-wide shared \( B \) experts. One intuitive approach is to feed \( x' \) into each \( B \) expert and combine their outputs to obtain the additional knowledge \( x_{\Delta} \in \mathbb{R}^{N \times C_{\text{out}}} \). However, to address efficiency concerns discussed in Appendix A.2, we propose an alternative implementation that is mathematically equivalent but significantly reduces the computational burden, described as follows:

\[
x_{\Delta} = x' \left( \sum_{i=1}^{N_e} S_i \cdot B^i \right)
\] (3)

where \( S \) is the set of weight scores for the \( B \) experts, obtained through selective weight allocation. Since each \( S_i \) is a scalar value, the calculation in Eq. 3 is mathematically equivalent to the intuitive method but with significantly improved efficiency. Therefore, the complete computation flow, with input \( x \in \mathbb{R}^{N \times C_{\text{in}}} \) and output \( y \in \mathbb{R}^{N \times C_{\text{out}}} \), for an adaptation target module is:

\[
y = xW_0 + s \cdot x_{\Delta}
\] (4)

where \( s \) is a scaling factor. By selectively allocating weights and mixing \( B \) experts, Lily enables access to all levels of information during adaptation. Each layer’s target adaptation modules can consider the status and knowledge from all other layers, resulting in a more expressive and comprehensive adaptation. Meanwhile, thanks to its interconnectivity, Lily can break the low-rank update constraint of LoRA by simply employing a smaller number of adapters with higher ranks.

4 Experiments

We validate the effectiveness of Lily across different domains, model sizes (from ViT to LLM), and architectures (Transformers, Mamba), demonstrating its generally strong adaptation capability. Concurrently, we conduct a comprehensive analysis of Lily’s intrinsic mechanisms, providing a thorough understanding of how it works. All ranks for Lily are selected from 8, 16, 32, ensuring that the total parameter count does not exceed that of the baselines. All experiments are conducted on a single RTX 4090 GPU. Additionally, multiple analyses are provided in Appendix C, D, E, F, G, H, I, and J.

4.1 Common Sense Reasoning

Implementation: We evaluate Lily on commonsense reasoning with LLMs. For the implementation, we utilize LLaMA3-8B (AI@Meta, 2024) and Falcon-Mamba-7B (Zuo et al., 2024) as backbones. LLaMA3 is a near-SOTA open-source large language model, while Falcon-Mamba is an open-sourced large language model based on the Mamba architecture. Using these models allows us to validate the effectiveness of Lily for fine-tuning LLMs and assess whether this effectiveness can be transferred to architectures beyond Transformers (Mamba, in this case). We fine-tune these models on Commonsense170K (Hu et al., 2023) and evaluate the adaptation results on eight multiple-choice problem tasks, including BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-e, ARC-c (Clark et al., 2018), and OBQA (Mihaylov et al., 2018). The compared methods are LoRA for Falcon-Mamba and LoRA (Hu et al.,
Table 1: Commonsense reasoning results for Falcon-Mamba-7B across eight tasks. Bold represents the highest performance for each dataset utilizing PEFT methods. “Δ” and “in” refer to adaptations of Mamba’s delta_proj and in_proj parameters, respectively.

<table>
  <tr>
    <th>Model</th>
    <th>PEFT</th>
    <th>Params</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>SIQA</th>
    <th>HellaSwag</th>
    <th>WinoGrande</th>
    <th>ARC-e</th>
    <th>ARC-c</th>
    <th>OBQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>-</td>
    <td>-</td>
    <td>73.1</td>
    <td>85.4</td>
    <td>68.5</td>
    <td>78.5</td>
    <td>66.1</td>
    <td>89.8</td>
    <td>79.9</td>
    <td>74.8</td>
    <td>77.0</td>
  </tr>
  <tr>
    <td rowspan="2">Falcon-Mamba-7B</td>
    <td>LoRA</td>
    <td>3.7M</td>
    <td>6.5</td>
    <td>30.5</td>
    <td>40.6</td>
    <td><b>14.9</b></td>
    <td>56.4</td>
    <td>42.2</td>
    <td>31.8</td>
    <td>38.4</td>
    <td>32.7</td>
  </tr>
  <tr>
    <td>Lily (Δ + in)</td>
    <td>3.7M</td>
    <td>44.9</td>
    <td><b>66.8</b></td>
    <td>65.0</td>
    <td>10.5</td>
    <td>57.1</td>
    <td>78.7</td>
    <td>64.6</td>
    <td><b>68.2</b></td>
    <td>57.0</td>
  </tr>
  <tr>
    <td>Lily (in)</td>
    <td><b>3.3M</b></td>
    <td><b>60.2</b></td>
    <td>61.0</td>
    <td><b>67.3</b></td>
    <td>12.9</td>
    <td><b>61.5</b></td>
    <td><b>80.0</b></td>
    <td>67.5</td>
    <td>65.8</td>
    <td><b>59.5</b></td>
    <td></td>
  </tr>
</table>

Table 2: Commonsense reasoning results for LLaMA3-8B across eight tasks. † represents results taken from Liu et al. (2024) and (Wang et al., 2024). Bold denotes the highest performance scores for each dataset among different PEFT methods.

<table>
  <tr>
    <th>Model</th>
    <th>PEFT</th>
    <th>Params</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>SIQA</th>
    <th>HellaSwag</th>
    <th>WinoGrande</th>
    <th>ARC-e</th>
    <th>ARC-c</th>
    <th>OBQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>-</td>
    <td>-</td>
    <td>73.1</td>
    <td>85.4</td>
    <td>68.5</td>
    <td>78.5</td>
    <td>66.1</td>
    <td>89.8</td>
    <td>79.9</td>
    <td>74.8</td>
    <td>77.0</td>
  </tr>
  <tr>
    <td rowspan="4">LLaMA3-8B</td>
    <td>LoRA†</td>
    <td>56M</td>
    <td>70.8</td>
    <td>85.2</td>
    <td><b>79.9</b></td>
    <td>91.7</td>
    <td>84.3</td>
    <td>84.2</td>
    <td>71.2</td>
    <td>79.0</td>
    <td>80.8</td>
  </tr>
  <tr>
    <td>PiSSA†</td>
    <td>83.8M</td>
    <td>67.1</td>
    <td>81.1</td>
    <td>77.2</td>
    <td>83.6</td>
    <td>78.9</td>
    <td>77.7</td>
    <td>63.2</td>
    <td>74.6</td>
    <td>75.4</td>
  </tr>
  <tr>
    <td>MiLoRA†</td>
    <td>56.6M</td>
    <td>68.8</td>
    <td><b>86.7</b></td>
    <td>77.2</td>
    <td><b>92.9</b></td>
    <td><b>85.6</b></td>
    <td>86.8</td>
    <td>75.5</td>
    <td>81.8</td>
    <td>81.9</td>
  </tr>
  <tr>
    <td>Lily</td>
    <td><b>1.2M</b></td>
    <td><b>72.9</b></td>
    <td>85.6</td>
    <td>77.8</td>
    <td>92.7</td>
    <td>83.3</td>
    <td><b>89.7</b></td>
    <td><b>77.6</b></td>
    <td><b>82.8</b></td>
    <td><b>82.8</b></td>
  </tr>
</table>

2021), PiSSA (Meng et al., 2024), and MiLoRA (Wang et al., 2024) for LLaMA3. We only compare LoRA for Falcon-Mamba because tailored PEFT methods for Mamba-based LLMs have not yet been proposed, which is beyond the scope of this paper. Detailed hyper-parameter settings and dataset information are reported in Appendix B.1.1 and Appendix B.2.1.

Results We report the accuracy in Tables 2 and 1. Based on these results, it can be observed that Lily outperforms the other compared PEFT methods with a smaller parameters budget. Specifically, Lily surpasses LoRA by a significant margin on Falcon-Mamba and, on LLaMA3, outperforms both LoRA and MiLoRA. This demonstrates Lily’s superior adaptation capability and parameter efficiency in handling commonsense reasoning tasks. Additionally, although performance on Falcon-Mamba is notably lower than that of the baseline and LLaMA3, we believe this discrepancy stems from the inherent limitations of the model rather than any deficiency in Lily, as Lily still significantly outperforms LoRA on Falcon-Mamba while demonstrating robust performance on LLaMA3. These findings also highlight that current state of Mamba-based LLMs generally exhibits inferior performance compared to Transformer-based LLMs such as ChatGPT and LLaMA on many tasks.

4.2 Natural Language Understanding

Implementation We evaluate Lily on natural language understanding (NLU) tasks. For the implementation, we use RoBERTa Base (Liu et al., 2019) and RoBERTa Large as the backbones and fine-tune them on tasks from the GLUE benchmark (General Language Understanding Evaluation (Wang et al., 2018)), which consists of multiple NLU tasks, including single-sentence classification, similarity and paraphrase, and natural language inference tasks. We compare Lily against several competitive PEFT methods, including BitFit (Zakken et al., 2021), Adapter-Tuning (Rücklé et al., 2020; Housley et al., 2019b; Lin et al., 2020; Pfeiffer et al., 2020b), LoRA (Hu et al., 2021), DyLoRA (Valipour et al., 2022), FLoRA (Hao et al., 2024), and AdaLoRA (Zhang et al., 2023). Additionally, we utilize full fine-tuning (FFT) as the baseline. Specific hyperparameters and dataset information are provided in Appendix B.1.2 and B.2.2.

Results The results are shown in Table 3. From the table, we can clearly observe that Lily surpasses all the compared PEFT methods by a significant margin, demonstrating its ability to tackle NLU tasks. Among the six tasks, Lily surpasses FFT on four of them when using RoBERTa Base and RoBERTa Large, showcasing its strong approximation ability and high parameter efficiency.

4.3 Subject-driven Image Generation

Implementation We conduct experiments on fine-tuning text-to-image diffusion models for the subject-driven generation task. As the backbone, we use SDXL and fine-tune it using both LoRA and Lily. First, we fine-tune the model on images paired with text prompts (e.g., “A photo of a [v] duck toy”), each of which includes a unique identifier. Afterward, text prompts containing the identifier are used to generate customized images.
Table 3: Various fine-tuning methods applied to RoBERTa Base and RoBERTa Large are evaluated on six datasets from the GLUE benchmark. We present the Matthews correlation coefficient (MCC) for CoLA, the Pearson correlation coefficient (PCC) for STS-B, and accuracy (Acc.) for the remaining tasks. The highest performance for each dataset is highlighted in bold, with all metrics favoring higher values across the six datasets.

<table>
  <tr>
    <th rowspan="2">Model & Method</th>
    <th rowspan="2"># Trainable Parameters</th>
    <th colspan="2">SST-2</th>
    <th colspan="2">MRPC</th>
    <th colspan="2">CoLA</th>
    <th colspan="2">QNLI</th>
    <th colspan="2">RTE</th>
    <th colspan="2">STS-B</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>(Acc.)</th>
    <th>(Acc.)</th>
    <th>(Acc.)</th>
    <th>(MCC)</th>
    <th>(Acc.)</th>
    <th>(Acc.)</th>
    <th>(Acc.)</th>
    <th>(PCC)</th>
    <th>(Acc.)</th>
    <th>(Acc.)</th>
    <th>(PCC)</th>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(FFT)</td>
    <td>125M</td>
    <td>94.8</td>
    <td>90.2</td>
    <td>63.6</td>
    <td>92.8</td>
    <td>78.7</td>
    <td>91.2</td>
    <td>85.2</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(BitFit)</td>
    <td>0.1M</td>
    <td>93.7</td>
    <td><b>92.7</b></td>
    <td>62</td>
    <td>91.8</td>
    <td>81.5</td>
    <td>90.8</td>
    <td>85.4</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(Adpt<sup>P</sup>)</td>
    <td>0.3M</td>
    <td>94.2</td>
    <td>88.5</td>
    <td>60.8</td>
    <td>93.1</td>
    <td>71.5</td>
    <td>89.7</td>
    <td>83.0</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(Adpt<sup>P</sup>)</td>
    <td>0.9M</td>
    <td>94.7</td>
    <td>88.4</td>
    <td>62.6</td>
    <td>93.0</td>
    <td>75.9</td>
    <td>90.3</td>
    <td>84.2</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(LoRA)</td>
    <td>0.3M</td>
    <td>94.8</td>
    <td>89.8</td>
    <td>63.3</td>
    <td>92.9</td>
    <td>78.2</td>
    <td><b>91.5</b></td>
    <td>85.1</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(AdaLoRA)</td>
    <td>0.3M</td>
    <td>94.5</td>
    <td>88.7</td>
    <td>62.0</td>
    <td><b>93.1</b></td>
    <td>81.0</td>
    <td>90.5</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(DyLoRA)</td>
    <td>0.3M</td>
    <td>94.3</td>
    <td>89.5</td>
    <td>61.1</td>
    <td>92.2</td>
    <td>78.7</td>
    <td>91.1</td>
    <td>84.5</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(FLoRA)</td>
    <td>0.3M</td>
    <td>91.2</td>
    <td>85.8</td>
    <td>65.4</td>
    <td>92.3</td>
    <td>65.0</td>
    <td>87.6</td>
    <td>81.2</td>
  </tr>
  <tr>
    <td>RoB<sub>base</sub>(Lily)</td>
    <td>0.3M</td>
    <td><b>95.0</b></td>
    <td>90.2</td>
    <td><b>66.0</b></td>
    <td>92.5</td>
    <td><b>81.6</b></td>
    <td>90.8</td>
    <td><b>86.0</b></td>
  </tr>
  <tr>
    <td>RoB<sub>large</sub>(FF)</td>
    <td>356M</td>
    <td><b>96.4</b></td>
    <td><b>90.9</b></td>
    <td>68</td>
    <td>94.7</td>
    <td>86.6</td>
    <td><b>92.4</b></td>
    <td>88.2</td>
  </tr>
  <tr>
    <td>RoB<sub>large</sub>(Adpt<sup>H</sup>)</td>
    <td>0.8M</td>
    <td>96.3</td>
    <td>87.7</td>
    <td>66.3</td>
    <td>94.7</td>
    <td>72.9</td>
    <td>91.5</td>
    <td>84.9</td>
  </tr>
  <tr>
    <td>RoB<sub>large</sub>(LoRA)</td>
    <td>0.8M</td>
    <td>96.2</td>
    <td>90.2</td>
    <td>68.2</td>
    <td><b>94.8</b></td>
    <td>85.2</td>
    <td>92.3</td>
    <td>87.8</td>
  </tr>
  <tr>
    <td>RoB<sub>large</sub>(Lily)</td>
    <td>0.5M</td>
    <td>95.6</td>
    <td>90.9</td>
    <td><b>68.4</b></td>
    <td>94.8</td>
    <td><b>88.4</b></td>
    <td>91.9</td>
    <td><b>88.4</b></td>
  </tr>
</table>

Results The results are presented in Fig. 2 following the format in Gao et al. (2024b) and Wu et al. (2024). From these results, we observe that the images generated by Lily generally align better with the text prompts. For instance, when asked to generate an image of a duck toy floating on water, Lily’s output accurately depicts the designated environment, whereas LoRA’s does not. Additionally, when asked to generate an image of a wolf plushie in the snow, Lily precisely captures the snow around the wolf, while LoRA fails to do so. These observations demonstrate Lily’s excellent performance in text-to-image generation with more expressive adaptation. Additional generated results are provided in Appendix I.

4.4 Visual Adaptation Benchmark
Implementation We assess Lily on the Visual Task Adaptation Benchmark (VTAB-1K (Zhai et al., 2019)), a suite of 19 visual tasks spanning diverse domains and semantics, to test its general visual adaptation capability. Tasks are categorized into Natural, Specialized, and Structured, and are all formulated as classification problems for consistent model evaluation. We conduct two sets of experiments: one focusing on adaptation effectiveness on the Vision Transformer (ViT (Dosovitskiy et al., 2020)) and the other on Vision Mamba (Vim (Zhu et al., 2024)), demonstrating Lily’s architecture-agnostic capabilities. For ViT, we use ViT-B pre-trained on ImageNet-21K (Deng et al., 2009), and for Vim, we use Vim-s pre-trained on ImageNet-1K. To fairly compare ViT and Vim architectures, we implement LoRA (Hu et al., 2021) and AdaptFormer (Chen et al., 2022) on ViT-B pre-trained on ImageNet-1K. In the ViT experiments, we compare Lily with LoRA, AdaptFormer, FourierFT (Gao et al., 2024b), and MoRA (Jiang et al., 2024); in the Vim experiments, we focus on contrasting architectural differences and, therefore, use only LoRA as the baseline. All experiments include full fine-tuning (FFT) and linear probing as baselines. For Vim, we implement two versions of Lily: Lily-S (Small) and Lily-L (Large), with different hyper-parameter settings to either reduce the parameter count (Lily-S) or maximize performance (Lily-L). For Lily on ViT, the reported results are obtained from adapting both the self-attention and the MA module in the Transformer. Regarding the performance of the fine-tuned module, we conduct additional experiments in Appendix D. Detailed experimental settings and dataset information are provided in Appendix B.1.3 and B.2.3.

Results The results are shown in Tables 4 and 5. For ViT, Lily significantly outperforms all compared PEFT methods while also offering improved parameter efficiency. In contrast, the performance on the Vim backbone is generally lower than that on ViT; for instance, LoRA on ViT performs better than LoRA on Vim. We argue that this difference is due to variations in architecture design and overall model size. However, Lily’s strong adaptation performance allows it to match or exceed the performance of other PEFT methods on ViT and to significantly outperform LoRA on Vim
Figure 2: Qualitative results of subject-driven generation. Lily’s results align better with prompts, featuring more accurate color, environment, and shape.

(with both Lily-S and Lily-L surpassing LoRA by a significant margin). This demonstrates Lily’s architecture-agnostic capability, highlighting its potential across various model architectures. Overall, Lily achieves excellent visual adaptation performance while maintaining architecture-agnosticity and high parameter efficiency.

4.5 Understanding Lily

4.5.1 Does It Have High-Rank Weight Updates?

The interconnected and asymmetric structure of Lily enables a flexible allocation of the parameter budget, thereby allowing weight updates with higher ranks across all layers. To validate this claim, we provide an empirical analysis, as shown in Fig. 3. Specifically, we run four tasks from the NLU experiment and measure the rank of the weight updates for the query transformation matrix \( W_q \) in the first three layers. We use a small number of matrices \( A \) and \( B \) (2 or 3) with a rank of 32 to match the parameter count of LoRA, which uses adapters with a rank set to 8. Specific hyperparameter settings can be found in Appendix B.1.2.

From the results, we observe that the rank of the weight updates from Lily is generally notably larger than that of LoRA when using a similar number of parameters. Meanwhile, the weight updates from Lily still exhibit a higher rank compared to those of LoRA even when using only 16.7% of LoRA’s parameters. This empirical analysis essentially validates our claim that Lily achieves high-rank updates with the same parameter budget. We attribute this to the model-wide sharing and the cross-layer asymmetric design, which facilitate a flexible allocation of the parameter budget.

![Qualitative results of subject-driven generation.](page_120_120_1208_384.png)

![Actual rank of the weight updates.](page_120_600_1208_384.png)

Figure 3: Actual rank of the weight updates. The weight updates are of shape 768 × 768. We run 20 epochs for COLA, MRPC, and STS-B, and 3 epochs for SST-2. It can be easily observed that the weight updates from Lily have notably higher rank than those from LoRA. Note that the reported rank is computed from accumulated weight updates over multiple epochs.

4.5.2 What’s the Influence of Adapter Granularity?

The number of experts in the model-wide \( B \) module can be freely set, and the number of \( As \) can also be flexibly determined by sharing across the
Table 4: Full results of Lily on ViT-B pre-trained on ImageNet-21K for the VTAB-1K benchmark, with averages computed based on group-wise results. Bold indicates the best performance.

<table>
  <tr>
    <th rowspan="2">Params(M)</th>
    <th rowspan="2">Average</th>
    <th colspan="5">Natural</th>
    <th colspan="3">Specialized</th>
    <th colspan="6">Structured</th>
  </tr>
  <tr>
    <th>Cifar100</th>
    <th>Celeb101</th>
    <th>DTD</th>
    <th>Flowers102</th>
    <th>Pets</th>
    <th>SVHN</th>
    <th>Sun397</th>
    <th>Camelyon</th>
    <th>EuroSAT</th>
    <th>Resisc45</th>
    <th>Retinopathy</th>
    <th>Clevr-Count</th>
    <th>Clevr-Dist</th>
    <th>DMLab</th>
    <th>KITTI-Dist</th>
    <th>dsPr-LoC</th>
    <th>dsPr-Ori</th>
    <th>sNORB-Azim</th>
    <th>sNORB-Ele</th>
  </tr>
  <tr>
    <th colspan="2">Conventional Fine-Tuning</th>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>FFT</td>
    <td>86</td>
    <td>68.9</td>
    <td>68.9</td>
    <td>87.7</td>
    <td>64.3</td>
    <td>97.2</td>
    <td>86.9</td>
    <td>87.4</td>
    <td>38.8</td>
    <td>79.7</td>
    <td>95.7</td>
    <td>84.2</td>
    <td>73.9</td>
    <td>56.3</td>
    <td>58.6</td>
    <td>41.7</td>
    <td>65.5</td>
    <td>57.5</td>
    <td>46.7</td>
    <td>25.7</td>
    <td>29.1</td>
  </tr>
  <tr>
    <td>LP</td>
    <td>0</td>
    <td>57.6</td>
    <td>64.4</td>
    <td>85.0</td>
    <td>63.2</td>
    <td>97.0</td>
    <td>86.3</td>
    <td>36.6</td>
    <td>51.0</td>
    <td>78.5</td>
    <td>87.5</td>
    <td>68.5</td>
    <td>74.0</td>
    <td>34.3</td>
    <td>30.6</td>
    <td>33.2</td>
    <td>55.4</td>
    <td>12.5</td>
    <td>20.0</td>
    <td>9.6</td>
    <td>19.2</td>
  </tr>
  <tr>
    <th colspan="2">PEFT methods</th>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>AdaptFormer</td>
    <td>0.588</td>
    <td>76.8</td>
    <td>74.0</td>
    <td>92.2</td>
    <td>71.7</td>
    <td>99.3</td>
    <td>91.7</td>
    <td>88.9</td>
    <td>56.4</td>
    <td>87.2</td>
    <td>95.1</td>
    <td>85.7</td>
    <td>75.9</td>
    <td>84.2</td>
    <td>62.2</td>
    <td>53.0</td>
    <td>81.0</td>
    <td>87.1</td>
    <td>53.6</td>
    <td>35.3</td>
    <td>42.3</td>
  </tr>
  <tr>
    <td>Bi-LoRA</td>
    <td>1.180</td>
    <td>76.7</td>
    <td>72.1</td>
    <td>91.7</td>
    <td>71.2</td>
    <td>99.1</td>
    <td>91.4</td>
    <td>90.2</td>
    <td>55.8</td>
    <td>87.0</td>
    <td>95.4</td>
    <td>85.5</td>
    <td>75.5</td>
    <td>83.1</td>
    <td>64.1</td>
    <td>52.2</td>
    <td>81.3</td>
    <td>86.4</td>
    <td>53.5</td>
    <td>36.7</td>
    <td>44.4</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>1.180</td>
    <td>76.4</td>
    <td>72.5</td>
    <td>91.5</td>
    <td>71.9</td>
    <td>99.1</td>
    <td>91.4</td>
    <td>89.6</td>
    <td>56.0</td>
    <td>87.6</td>
    <td>95.3</td>
    <td>84.0</td>
    <td>75.0</td>
    <td>83.6</td>
    <td>64.3</td>
    <td>51.6</td>
    <td>80.9</td>
    <td>86.0</td>
    <td>51.8</td>
    <td>36.8</td>
    <td>42.3</td>
  </tr>
  <tr>
    <td>FourierFT</td>
    <td>0.936</td>
    <td>72.7</td>
    <td>69.1</td>
    <td>88.7</td>
    <td>81.9</td>
    <td>99.0</td>
    <td>91.0</td>
    <td>79.0</td>
    <td>55.6</td>
    <td>84.9</td>
    <td>93.0</td>
    <td>82.3</td>
    <td>74.9</td>
    <td>70.7</td>
    <td>61.4</td>
    <td>54.2</td>
    <td>74.8</td>
    <td>78.0</td>
    <td>53.0</td>
    <td>24.8</td>
    <td>30.8</td>
  </tr>
  <tr>
    <td>MoRA</td>
    <td>1.058</td>
    <td>75.4</td>
    <td>72.1</td>
    <td>90.7</td>
    <td>71.9</td>
    <td>99.2</td>
    <td>91.1</td>
    <td>90.1</td>
    <td>56.0</td>
    <td>87.1</td>
    <td>94.8</td>
    <td>81.5</td>
    <td>75.4</td>
    <td>76.7</td>
    <td>62.3</td>
    <td>49.7</td>
    <td>78.3</td>
    <td>81.3</td>
    <td>53.0</td>
    <td>34.5</td>
    <td>34.5</td>
  </tr>
  <tr>
    <td>Lily</td>
    <td>0.318</td>
    <td>77.3</td>
    <td>73.9</td>
    <td>93.0</td>
    <td>72.9</td>
    <td>99.3</td>
    <td>91.6</td>
    <td>89.0</td>
    <td>56.6</td>
    <td>87.9</td>
    <td>95.2</td>
    <td>84.9</td>
    <td>75.7</td>
    <td>83.9</td>
    <td>65.4</td>
    <td>53.4</td>
    <td>81.6</td>
    <td>88.2</td>
    <td>54.5</td>
    <td>37.0</td>
    <td>45.4</td>
  </tr>
</table>

Table 5: Full results of Lily on Vim-S pre-trained on ImageNet-1K for the VTAB-1K benchmark, with averages calculated within each group. * denotes linear probing results from Tu et al. (2023a). For fair comparison, we also use ViT-B pre-trained on ImageNet-1K. Bold indicates best performance among Vim-based PEFT methods.

<table>
  <tr>
    <th rowspan="2">Params(M)</th>
    <th rowspan="2">Average</th>
    <th colspan="5">Natural</th>
    <th colspan="3">Specialized</th>
    <th colspan="6">Structured</th>
  </tr>
  <tr>
    <th>Cifar100</th>
    <th>Celeb101</th>
    <th>DTD</th>
    <th>Flowers102</th>
    <th>Pets</th>
    <th>SVHN</th>
    <th>Sun397</th>
    <th>Camelyon</th>
    <th>EuroSAT</th>
    <th>Resisc45</th>
    <th>Retinopathy</th>
    <th>Clevr-Count</th>
    <th>Clevr-Dist</th>
    <th>DMLab</th>
    <th>KITTI-Dist</th>
    <th>dsPr-LoC</th>
    <th>dsPr-Ori</th>
    <th>sNORB-Azim</th>
    <th>sNORB-Ele</th>
  </tr>
  <tr>
    <th colspan="2">Conventional Fine-Tuning</th>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>FFT-Vim</td>
    <td>26</td>
    <td>70.1</td>
    <td>47.7</td>
    <td>89.4</td>
    <td>64.2</td>
    <td>89.0</td>
    <td>87.7</td>
    <td>90.6</td>
    <td>35.1</td>
    <td>84.5</td>
    <td>93.9</td>
    <td>81.0</td>
    <td>74.5</td>
    <td>67.5</td>
    <td>52.9</td>
    <td>47.3</td>
    <td>78.9</td>
    <td>75.3</td>
    <td>53.9</td>
    <td>33.3</td>
    <td>29.4</td>
  </tr>
  <tr>
    <td>FFT-ViT</td>
    <td>86</td>
    <td>69.9</td>
    <td>49.4</td>
    <td>83.9</td>
    <td>65.5</td>
    <td>91.7</td>
    <td>89.1</td>
    <td>91.4</td>
    <td>33.5</td>
    <td>85.9</td>
    <td>93.6</td>
    <td>85.4</td>
    <td>74.3</td>
    <td>54.7</td>
    <td>55.2</td>
    <td>48.7</td>
    <td>79.7</td>
    <td>68.2</td>
    <td>49.7</td>
    <td>31.5</td>
    <td>27.7</td>
  </tr>
  <tr>
    <td>LP-Vim</td>
    <td>0</td>
    <td>55.3</td>
    <td>40.9</td>
    <td>83.3</td>
    <td>57.3</td>
    <td>66.3</td>
    <td>86.3</td>
    <td>38.4</td>
    <td>36.6</td>
    <td>79.0</td>
    <td>87.6</td>
    <td>65.0</td>
    <td>73.6</td>
    <td>36.3</td>
    <td>35.1</td>
    <td>33.3</td>
    <td>64.8</td>
    <td>73.0</td>
    <td>21.6</td>
    <td>15.1</td>
    <td>21.7</td>
  </tr>
  <tr>
    <td>LP-ViT</td>
    <td>0</td>
    <td>66.4</td>
    <td>50.6</td>
    <td>85.6</td>
    <td>61.4</td>
    <td>79.5</td>
    <td>86.5</td>
    <td>40.8</td>
    <td>38.0</td>
    <td>79.7</td>
    <td>91.5</td>
    <td>71.7</td>
    <td>65.5</td>
    <td>41.4</td>
    <td>34.4</td>
    <td>34.1</td>
    <td>55.4</td>
    <td>18.1</td>
    <td>26.4</td>
    <td>16.5</td>
    <td>24.8</td>
  </tr>
  <tr>
    <th colspan="2">PEFT on ViT</th>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>AdaptFormer</td>
    <td>0.147</td>
    <td>72.4</td>
    <td>56.2</td>
    <td>89.6</td>
    <td>67.2</td>
    <td>91.2</td>
    <td>91.1</td>
    <td>85.9</td>
    <td>42.1</td>
    <td>85.4</td>
    <td>94.6</td>
    <td>84.0</td>
    <td>74.3</td>
    <td>75.8</td>
    <td>58.6</td>
    <td>48.6</td>
    <td>79.6</td>
    <td>81.6</td>
    <td>53.7</td>
    <td>29.6</td>
    <td>35.2</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>0.295</td>
    <td>72.5</td>
    <td>56.4</td>
    <td>89.0</td>
    <td>66.9</td>
    <td>91.2</td>
    <td>90.4</td>
    <td>86.9</td>
    <td>41.5</td>
    <td>85.4</td>
    <td>95.1</td>
    <td>84.1</td>
    <td>75.2</td>
    <td>75.8</td>
    <td>61.7</td>
    <td>47.7</td>
    <td>80.5</td>
    <td>80.4</td>
    <td>52.0</td>
    <td>29.4</td>
    <td>35.7</td>
  </tr>
  <tr>
    <th colspan="2">PEFT on Vim</th>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>0.054</td>
    <td>70.1</td>
    <td>57.5</td>
    <td>87.7</td>
    <td>64.6</td>
    <td>86.0</td>
    <td>90.8</td>
    <td>57.9</td>
    <td>38.2</td>
    <td>92.3</td>
    <td>98.7</td>
    <td>76.2</td>
    <td>75.8</td>
    <td>65.6</td>
    <td>42.0</td>
    <td>80.5</td>
    <td>71.8</td>
    <td>51.0</td>
    <td>28.4</td>
    <td>32.6</td>
    <td></td>
  </tr>
  <tr>
    <td>Lily-S</td>
    <td>0.074</td>
    <td>71.4</td>
    <td>58.2</td>
    <td>88.5</td>
    <td>65.6</td>
    <td>87.1</td>
    <td>90.7</td>
    <td>87.5</td>
    <td>40.4</td>
    <td>83.3</td>
    <td>94.1</td>
    <td>79.7</td>
    <td>73.8</td>
    <td>81.2</td>
    <td>57.3</td>
    <td>44.1</td>
    <td>80.9</td>
    <td>79.3</td>
    <td>54.1</td>
    <td>30.0</td>
    <td>33.7</td>
  </tr>
  <tr>
    <td>Lily-L</td>
    <td>0.196</td>
    <td>72.3</td>
    <td>57.8</td>
    <td>89.4</td>
    <td>66.2</td>
    <td>87.8</td>
    <td>90.5</td>
    <td>88.1</td>
    <td>40.5</td>
    <td>84.1</td>
    <td>94.3</td>
    <td>81.3</td>
    <td>75.1</td>
    <td>81.6</td>
    <td>57.8</td>
    <td>46.5</td>
    <td>81.0</td>
    <td>82.9</td>
    <td>55.2</td>
    <td>32.1</td>
    <td>34.8</td>
  </tr>
</table>

same level of layers, as introduced in Appendix A.1. Therefore, we analyze the impact of these choices on performance. We denote the number of \(A\) experts and \(B\) experts as \(ne_1\) and \(ne_2\), respectively. For simplicity, we set them equal in the experiments and denote this common value as \(ne\). We refer to the number of layers each expert attends to as adapter granularity. As the value of \(ne\) increases, the adapter granularity becomes finer. As shown in Fig. 5, the results from the VTAB-1K benchmark indicate different patterns. For instance, on the DTD dataset, the best performance is achieved when \(ne\) is 4, while on sNORB-Azim, performance increases as \(ne\) increases. Increasing \(ne\) leads to more parameters and finer adapter granularity; however, finer adapter granularity does not necessarily translate to better overall performance. For example, on Resisc45, DTD, Cifar100, sNORB-Ele, dsPr-LoC, Flowers102, and EuroSAT, the negative impact of increasingly finer adapter granularity eventually outweighs the benefits of the additional parameters, leading to a decrease in overall performance. In other tasks, different patterns may occur because the positive effect of adapter granularity on performance is consistently strong, or its negative effect is insufficient to offset the benefits of increased parameters, resulting in generally improved performance with higher \(ne\). This phenomenon provides an important insight: for most tasks, simply increasing the number of parameters may not lead to better performance. Instead, only when adapter granularity and the number of param-
Figure 4: Visualization of accumulated assigned weight for \( B \) experts by a router across various layers. Example here uses layer of index 2, 13 and 22 to represent shallow, middle and deep layers. The reported values are based on the accumulated router outputs over multiple epochs.

Figure 5: Impact of attention granularity (i.e., the choice of how many \( A \)s and \( B \)s) on the performance. We choose 12 out of 19 tasks from VTAB-1K for a comprehensive understanding.

Figure 6: Hardware efficiency of Lily compared to LoRA. We run 10 epochs for COLA. We report the training time and memory consumption. It can be observed that Lily generally performs on par with LoRA in terms of hardware efficiency.

eters reach an optimal trade-off can we achieve the best performance.

4.5.3 Does It Exhibit Selectivity?
Lily uses routers to assign varying weights to different \( B \) experts, thereby achieving selective information combination. We illustrate this selectivity in Fig. 4. We use a setup with three \( B \) experts and select three layer levels (1, 13, 22) to calculate the total weight assigned to each expert. The results reveal clear selectivity: for different layers, the router assigns significantly different weights to the \( B \) experts. For instance, on Cifar100, the middle layer is predominantly dominated by \( B \) 2, whereas the deep layer is primarily dominated by \( B \) 1 and \( B \) 2. In contrast, on Retinopathy, both the middle and deep layers are dominated by \( B \) 3. This selectivity ensures that, even when different layers share information, the inherent differences between layers are still taken into account, making the adaptation more flexible and comprehensive.

4.5.4 What’s the Hardware Efficiency?
The dynamics of Lily obviously introduce complexity to the design of LoRA. In this section, we analyze how this affects the hardware efficiency of Lily compared to LoRA. We use the COLA task from the NLU experiments with RoBERTa-Base and run for 10 epochs. Additionally, we also report
the runtime and GPU memory consumption in the Falcon-Mamba experiment.

The results are shown in Fig. 6, from which we can observe that the hardware efficiency of Lily is comparable to LoRA. Specifically, Lily slightly underperforms LoRA in the NLU experiment but performs on par with LoRA in the LLM experiment. In general, the introduced complexity of Lily does not prevent it from being an more effective PEFT method that is also hardware-friendly.

5 Conclusion

In this paper, we propose Low-Rank Interconnected Adaptation (Lily), a novel framework for efficient fine-tuning via the interconnectivity of adapters. Lily enables each layer to access information from others during adaptation through a hierarchical structure. Additionally, it successfully overcomes the low-rank update limitation of LoRA, enabling high-rank updates and, therefore, better adaptation capability under the same parameter budget. Our approach consistently improves performance across various modalities, model sizes, and architectures, surpassing existing methods while maintaining enhanced efficiency. In summary, Lily’s versatility and efficiency make it a promising approach for a wide range of applications.

6 Limitations

Although Lily has been experimentally evaluated in a wide range of scenarios, we have not explored all possible applications where PEFT could be used. These potential areas are left as directions for future work.

7 Ethics Statement

This work is an improvement upon LoRA. However, it could potentially be used for fine-tuning diffusion models or large language models (LLMs) for generating malicious content.

References

AI@Meta. 2024. Llama 3 model card.

Amjad Almahairi, Nicolas Ballas, Tim Cooijmans, Yin Zheng, Hugo Larochelle, and Aaron Courville. 2016. Dynamic capacity networks. In International Conference on Machine Learning, pages 2549–2558. PMLR.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432.

Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. 2024. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. 2022. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664–16678.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Andrew Davis and Itamar Arel. 2013. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314.

Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiayouan Guo, Jie Yang, and VS Subrahmanian. 2024a. Higher layers need more lora experts. arXiv preprint arXiv:2402.08562.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. 2024b. Parameter-efficient fine-tuning with discrete fourier transform. arXiv preprint arXiv:2405.03003.

Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.

Yongchang Hao, Yanshuai Cao, and Lili Mou. 2024. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019a. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019b. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023. LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5254–5276, Singapore. Association for Computational Linguistics.

Ting Jiang, Shaohan Huang, Shengye Luo, Zihan Zhang, Haizhen Huang, Furui Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, et al. 2024. Mora: High-rank updating for parameter-efficient fine-tuning. arXiv preprint arXiv:2405.12130.

Shibo Jie and Zhi-Hong Deng. 2023. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1060–1068.

Shibo Jie, Haoqing Wang, and Zhi-Hong Deng. 2023. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17217–17226.

Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020. Exploring versatile generative language model via parameter-efficient transfer learning. arXiv preprint arXiv:2004.03829.

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.

Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789.

Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng. 2024. Velora: Memory efficient training using rank-1 sub-token projections. arXiv preprint arXiv:2405.17991.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020b. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247.

Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.

Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. 2024. Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404.19245.

Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. 2023a. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7725–7735.

Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao. 2023b. Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7725–7735.

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2022. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.

Hangqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. 2024. Milora: Harnessing minor singular components for parameter-efficient llm finetuning. arXiv preprint arXiv:2406.09044.

Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. 2024. Mixture-of-subspaces in low-rank adaptation.

Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. 2023. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. arXiv preprint arXiv:2309.05444.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. 2019. The visual task adaptation benchmark.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference on Learning Representations. Openreview.

Yibo Zhong and Yao Zhou. 2025. Rethinking low-rank adaptation in vision: Exploring head-level responsiveness across diverse tasks. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 7787–7796. IEEE.

Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. 2024. Convolution meets lora: Parameter efficient finetuning for segment anything model. arXiv preprint arXiv:2401.17868.

Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. 2024. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417.

Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunisch, and Hakim Hacid. 2024. Falcon mamba: The first competitive attention-free 7b language model.
Appendix

A  More discussion about Lily

A.1 Model Structure and Design Intuition of Lily

Within the overall framework of Lily, we delve into specific implementation details and model design insights. First, we establish the relationship between \( A \) and \( B \): \( A \) is confined to specific levels of layers, capturing features that enable the router to selectively assign weights to the \( B \) experts. In contrast, \( B \) is a model-wide module comprising multiple experts, each of which contains information from a particular level of layers.

We highlight several key aspects that are not heavily discussed in the methodology section:

A.1.1 Number of As
Since \( A \) is limited to specific layers, the simplest approach would be to place an \( A \) at each layer of the module to be adapted (e.g., the query transformation in MHSA). However, this setup may not be necessary, as the importance of each layer varies, and many layers have significantly lower importance than others (Zhang et al., 2023).

To achieve greater parameter efficiency, we can use fewer \( A \),s with each \( A \) focusing on a level of layers rather than a single layer. For example, an \( A \) can focus on shallow layers (e.g., layers 0, 1, 2, etc.) or deep layers. To enable a single \( A \) to handle multiple layers, we can share an \( A \) across multiple layers. By doing so, we eliminate the redundancy of placing an \( A \) at each layer, reduce the number of parameters, and improve efficiency.

This is exactly the strategy adopted in most of the experiments.

A.1.2 Number of \( B \) Experts
Regarding \( B \), the number of experts can be set arbitrarily, allowing for more flexible configurations. In our experiments, for the sake of simplicity, we set the number of \( B \) experts equal to the number of \( A \),s, thereby equating the granularity of \( A \) and \( B \).

A.1.3 Routers Setup
There are multiple possible configurations for the router. First, we can bind the router to \( B \), resulting in only one router per model. However, since the number of parameters in the router is relatively small, having only one router per model may not provide significant selectivity. Therefore, we can also bind the router to \( A \), configuring a separate router for each \( A \).

Most of our experiments use the latter setup. However, in the vision experiments on Vim, we adopt the single-router and no-lp-sharing setup to evaluate its effectiveness. The results indicate that this setup also performs well.

As future work, we can further verify the effectiveness of the latter setup on Vim, which may potentially lead to superior performance.

A.1.4 Hyperparameters
We detail the hyperparameters used in Lily. Specifically, we use Lily_r to represent the hidden dimension of the projectors: \( As \) and \( Bs \). It serves the same function as \( r \) in LoRA. We use Lily_s to represent the scaling factor used by Lily, which is primarily searched within the range {0.01, 0.1, 1.0, 10.0, 100.0}.

We use ne_1 to denote the number of \( As \) used in the model. Since \( As \) can be shared, as discussed in the previous section, ne_1 does not necessarily equal the number of layers in the model. Similarly, we use ne_2 to represent the number of \( B \) experts in the model-wide \( B \) module.

In our experiments, we set ne_1 = ne_2 to enhance parameter efficiency and maintain simplicity.

A.1.5 Design Intuition
Lily employs a hierarchical structure to enable updates with higher ranks than LoRA. However, simply connecting all \( Bs \) equally to \( As \) does not yield the best performance. From the perspective of feature and information utilization across layers, merely aggregating all \( Bs \) for an \( A \) ignores the distinctiveness of features from the current layers. Meanwhile, this approach reduces the variability in the combinations of gradient projection matrices (since \( S_i \) and \( C_{i,j} \) become constants), making the rank of the weight update higher than that of LoRA (as multiple distinct random matrices are used), but still not high enough for optimal performance due to the lack of variability in the combination process.

To address this, we introduce selectivity into the interconnectivity, as discussed below, making the combination of \( Bs \) data-dependent. This ensures that each \( S_i \) is unique across time steps, enabling updates with even higher ranks. This approach is similar to that of Hao et al. (2024), where a random matrix is constantly resampled to maintain high-rank updates. We further analyze this in Appendix
G.

A.2 Efficient Implementation for Weighted Combination

A straightforward implementation of the weighted combination in Lily is to pass the inputs through all the experts and then sum the results. This approach requires \( N_e \) matrix multiplications, \( N_e \) scalar multiplications, and \( N_e \) matrix additions. Despite its intuitive nature, the computational burden of this method is quite substantial.

However, Eq. 3, which is adopted in Lily, requires only \( N_e \) scalar multiplications, \( N_e \) matrix additions, and a single matrix multiplication. This optimization eliminates approximately \( N_e \) matrix multiplications, which can significantly reduce computational costs as the model size and the number of adaptation targets increase.

For an input \( x' \) of size \( \mathbb{R}^{N \times d} \) and a projection matrix \( P_H \in \mathbb{R}^{d \times C} \), the floating-point operations (FLOPs) of these two implementations are:

\[
\begin{align*}
\text{FLOPs} &= \sum_{i=1}^{N_e} (2NdC) + \sum_{i=1}^{N_e} (dC) + \sum_{i=1}^{N_e} (NC) \\
&= N_e \times (2NdC + dC + NC), \\
\text{FLOPs} &= 2 \sum_{i=1}^{N_e} (dC) + 2NdC \\
&= 2dC \times (N + N_e),
\end{align*}
\]

From this, we can easily observe that the approach adopted by Lily requires fewer computations, thereby improving both speed and efficiency during the fine-tuning process. Under the setting of \( N = 1024, d = 16, C = 768, N_e = 4 \), the FLOPs for the intuitive approach amount to 0.104 GFLOPs, whereas for Lily, it is merely 0.025 GFLOPs, potentially leading to a \( 4 \times \) speedup.

A.3 Actual Implementation of Lily

We present the actual implementation of Lily in Fig. 7. In this example, we showcase its implementation for visual adaptation tasks, specifically in the VTAB-1K benchmark. For large language models (LLMs), the implementation is slightly more complex due to modifications to the Hugging Face PEFT library (Mangrulkar et al., 2022), but the fundamental adaptation process remains the same.

Specifically, given an input, we first use the corresponding \( A \) of the current layer to project it into a low-dimensional representation. This low-dimensional representation is then used to selectively assign weights to the \( B \) experts. Once the weights for all experts are obtained, we proceed to combine these \( B \) experts accordingly, as discussed in Appendix A.2. After the weighted combination, the combined \( B \) is used to project the low-dimensional representation back into a high-dimensional space, thereby incorporating the additional knowledge gained through adaptation.

B Experimental Settings

B.1 Hyperparameters

A detailed description of the hyperparameters used in Lily is provided in Appendix A.1.

B.1.1 Commonsense Reasoning

The hyperparameters used in commonsense reasoning experiments for MiLoRA and PiSSA are provided in Tables 7 and 6. The settings for Lily and LoRA using Falcon-Mamba as the backbone are presented in Tables 9 and 8.

Notably, Lily achieves the best performance by adapting only the multi-head self-attention (MHSA) module in LLaMA3-8B, whereas other compared methods adapt all modules, including M.A. Moreover, Lily utilizes the fewest parameters, demonstrating its superior adaptation capability in low-parameter-budget scenarios.

B.1.2 Natural Language Understanding

The specific hyperparameter settings for Lily on the GLUE benchmark are provided in Table 10. We fix the learning rate of both the backbone and the head at \( 5 \times 10^{-3} \) and instead tune the scaling factor Lily_s, where Lily_s \( \in \{0.01, 0.1, 1.0\} \). The rank \( r \) is fixed at 32, and the random seed is set to 0. The baseline results are taken from FourierFT (Gao et al., 2024b).

B.1.3 Visual Task Adaptation Benchmark

We provide the hyperparameters for Lily on the VTAB-1K benchmark in Table 11. Specifically, we fix the learning rate at \( 1 \times 10^{-3} \) with a weight decay of \( 1 \times 10^{-4} \). For ViT, we tune the scaling factor Lily_s \( \in \{0.01, 0.1, 1.0, 10.0\} \) to maximize performance, following Jie et al. (2023) and Jie and Deng (2023). For Vim, we fix Lily_s to 1.0. Additionally, we search for the hyperparameters \( ne_1 \) and \( ne_2 \) within the range \( \{2, 3, 4\} \), as these numbers divide the number of layers in the ViT model (12 in ViT-B).
class lily_adapter(nn.Module):
    """
    Implementation of a Lily adapter for an adaptable target. For simplicity, we assume that the number of hp expert is equal to the number of LPs.

    args:
        hidden_dim: hidden dimension
        ne: number of experts
        lp: low-dimensional projector
        hps: high-dimensional projector experts
        mlp: whether the adaption target is located in MLP
    """
    def __init__(self, hidden_dim, ne, lp, hps, mlp=False):
        super().__init__()
        self.hps = hps
        self.ne = ne
        self.lp = lp
        self.router = nn.Linear(hidden_dim, ne, bias=False)
        if mlp:
            self.non_linear = nn.ReLU()
        else:
            self.non_linear = nn.Identity()
    def forward(self, x):
        hidden = self.non_linear(self.lp(x))
        router_logits = self.router(hidden)  # [B, N, num_of_experts]
        router_probability = F.softmax(router_logits, dim=-1)  # [B, N, ne]
        expert_probabilities = router_probability.mean(dim=(0, 1))
        combined_hp = torch.einsum("e,eio->io", expert_probabilities, self.hps)
        return torch.matmul(hidden, combined_hp)

Figure 7: Implementation of Lily in the VTAB-1K benchmark.

For Vim, we use the implementation discussed in Section A.1, which does not share As across layers. Therefore, ne_1 in this setting is fixed to the number of layers in Vim (22 in this case), while we search for ne_2 in {3, 6} and {5, 6, 17} separately for Lily-S and Lily-L. Note that ne is only set for the input projection in Vim. For the delta transformation, we use only a single B expert to reduce the parameter cost.

In the ViT experiments, the rank r is fixed at 16. Meanwhile, in Vim’s setting, we tune the ranks r for the delta transformation module and the input projection module separately. We use (4, 4) and (4, 8) separately for Lily-S and Lily-L.

B.2 Datasets

B.2.1 Commonsense Reasoning
We provide a short description of each datasets used in commonsense reasoning experiments in Table 12.

B.2.2 Natural Language Understanding
We provide detailed information about datasets in the GLUE benchmark in Table 13.

B.2.3 Visual Adaptation Benchmark
We provide detailed information about all the tasks from VTAB-1K benchmark in Table 14.

C Does Sharing As Result in Inferior Performance?

As mentioned earlier, we adopted a strategy of sharing the A across most of our experiments, ensuring that the number of As and B experts is consistent. This approach offers two key benefits: simplicity and enhanced parameter efficiency. By sharing the A, we eliminate the need to set a separate A for each layer, thereby reducing the overall parameter count.

Our decision to share the A is based on the observation of overall redundancy among layers. Specifically, different layers have varying levels of importance (Zhang et al., 2023), and some less important layers do not require a dedicated A. By not setting a separate A for these layers, we avoid introducing unnecessary parameter overhead while maintaining negligible impact on performance. To test whether sharing A results in inferior performance, we conducted experiments without A sharing on the VTAB-1K benchmark. The results, shown in Table 15, indicate that the best overall performance (77.3%) is the same as in the A-sharing setting. This suggests that even when we employ one A for each layer, the performance gain is negligible, and many of the parameters are, in fact, redundant. However, not sharing As leads to additional
Table 6: Hyperparameter configuration from the MiLoRA paper.

<table>
  <tr>
    <th colspan="2">MiLoRA hyperparameters</th>
  </tr>
  <tr>
    <td>Rank r</td>
    <td>32</td>
  </tr>
  <tr>
    <td>\( \alpha \) of LoRA</td>
    <td>64</td>
  </tr>
  <tr>
    <td>\( \alpha \) of PiSSA</td>
    <td>32</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>LR</td>
    <td>3e-4</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Linear</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Warmup Steps</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Placement</td>
    <td>query, key, value, \( MA \) up, \( MA \) down</td>
  </tr>
</table>

Table 7: Hyperparameter configuration from the PiSSA paper.

<table>
  <tr>
    <th colspan="2">PiSSA hyperparameters</th>
  </tr>
  <tr>
    <td>\( \alpha \)</td>
    <td>Same as rank r</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>LR</td>
    <td>2e-5</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>cosine</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Placement</td>
    <td>query, key, value, output, gate, \( MA \) up, \( MA \) down</td>
  </tr>
</table>

Table 8: Hyperparameter configuration for LoRA using Falcon-Mamba as backbone.

<table>
  <tr>
    <th colspan="2">LoRA hyperparameters</th>
  </tr>
  <tr>
    <td>Rank r</td>
    <td>2</td>
  </tr>
  <tr>
    <td>\( \alpha \)</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>LR</td>
    <td>3e-4</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Linear</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Placement</td>
    <td>input, delta</td>
  </tr>
</table>

D Where to Apply Lily in Transformers?

E Performance Analysis on VTAB-1K Benchmark with Lily on Transformer Modules

PEFT methods have been predominantly explored on the Transformer architecture, which consists of multi-head self-attention (MHSA) and multi-layer perceptron (\( MA \)) as its core modules. In this section, we analyze the impact of fine-tuned modules on performance using Lily. Specifically, we compare Lily’s performance on the VTAB-1K benchmark under four settings:

• Applying Lily solely to the query and value transformation module in MHSA (denoted as "qv").

• Applying Lily solely to the \( MA \) module (denoted as "mlp").

• Applying Lily to both the query and value transformation module in MHSA and the \( MA \) module (denoted as "qvmplp").

parameter overhead, which reduces the parameter efficiency of Lily. Therefore, \( A \)-sharing is an effective strategy to eliminate redundancy among \( As \) and enhance the parameter efficiency of Lily.
Table 9: Best Hyperparameter configuration for Lily using Falcon-Mamba and LLaMA3 as backbones.

<table>
  <tr>
    <th></th>
    <th>Falcon-Mamba</th>
    <th>LLaMA3</th>
  </tr>
  <tr>
    <td>Rank r</td>
    <td>40</td>
    <td>16</td>
  </tr>
  <tr>
    <td><i>ne_1</i></td>
    <td>4</td>
    <td>4</td>
  </tr>
  <tr>
    <td><i>ne_2</i></td>
    <td>4</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>LR</td>
    <td>3e-4</td>
    <td>3e-4</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td>Linear</td>
    <td>Linear</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td>
    <td>16</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>1</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Placement</td>
    <td>input</td>
    <td>query, key, value</td>
  </tr>
</table>

Table 10: Hyperparameter of Lily on GLUE benchmark.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>STS-B</th>
    <th>RTE</th>
    <th>MRPC</th>
    <th>CoLA</th>
    <th>SST-2</th>
    <th>QNLI</th>
    <th>MNLI</th>
    <th>QQP</th>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td colspan="8">AdamW</td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td colspan="8">Linear</td>
  </tr>
  <tr>
    <td>Learning Rate (Lily)</td>
    <td colspan="8">5e-3</td>
  </tr>
  <tr>
    <td>Learning Rate (Head)</td>
    <td colspan="8">5e-3</td>
  </tr>
  <tr>
    <td>Max Seq. Len</td>
    <td>512</td>
    <td>512</td>
    <td>512</td>
    <td>512</td>
    <td>512</td>
    <td>512</td>
    <td>512</td>
    <td>512</td>
  </tr>
  <tr>
    <td>Lily_s</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td><i>ne_1</i></td>
    <td>2</td>
    <td>3</td>
    <td>2</td>
    <td>4</td>
    <td>2</td>
    <td>2</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td><i>ne_2</i></td>
    <td>2</td>
    <td>3</td>
    <td>2</td>
    <td>4</td>
    <td>2</td>
    <td>2</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>64</td>
    <td>32</td>
    <td>50</td>
    <td>64</td>
    <td>32</td>
    <td>32</td>
    <td>0</td>
    <td>0</td>
  </tr>
</table>

• Applying Lily to both the key and value transformation module in MHSA and the MA module (denoted as "kvmlp").

To ensure a fair comparison, we tune the hyperparameters to maintain a similar parameter count across all settings. Additionally, to further investigate whether sharing the low-rank projection (\(A\)) affects performance, we do not share \(A\) in this experiment.

The results are presented in Table 15. We observe that the "kvmlp" setting achieves the best performance, with an average accuracy of 77.3%. In contrast, adapting only the MHSA module ("qv") yields the worst performance. Furthermore, we note that adapting both the MHSA and MA modules (qvmlp and kvmlp) generally leads to superior results compared to adapting only one specific module (qv and mlp). This suggests that both MA and MHSA play crucial roles in overall model performance, and adapting both is essential for effective adaptation.

Notably, even when applying Lily solely to the MHSA module, which results in the worst performance among the four settings (76.9%), it still outperforms LoRA by a significant margin (0.5%). This underscores the efficiency of Lily, as it uses fewer parameters than LoRA, even without \(A\) sharing.

F Where to Apply Lily in Mamba?

Nearly all previous PEFT method studies have focused on Transformers, while Mamba is a relatively new architecture, and therefore, there has been little research on PEFT methods for Mamba. In this section, we briefly analyze the pros and cons of adapting Mamba’s modules. A Mamba block consists of regular linear projection layers and a core component, the SSM module (Gu and Dao, 2023), (Zhu et al., 2024). Specifically, in the SSM module, Mamba utilizes parameters \((\Delta, A, B, C)\) to transform an input sequence \(x(t)\) into an output sequence \(y(t)\) using a hidden state \(h(t)\). The discretization process converts \(A\) and \(B\) into \(A\) and \(B\), respectively, using the time step size parameter \(\Delta\). Structured state space models, inspired by continuous systems, can be computed similarly to
Table 11: Hyperparameter configuration for Lily on VTAB-1K benchmark.

<table>
  <tr>
    <th></th>
    <th>Vision Transformer</th>
    <th>Vision Mamba</th>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>64</td>
    <td>64</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1E-3</td>
    <td>1E-2</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>1E-4</td>
    <td>1E-3</td>
  </tr>
  <tr>
    <td># Epochs</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>LR Decay</td>
    <td>cosine</td>
    <td>cosine</td>
  </tr>
</table>

Table 12: Details of the datasets used in our commonsense reasoning tasks.

<table>
  <tr>
    <th>Benchmark</th>
    <th>Description</th>
    <th># Test Questions</th>
  </tr>
  <tr>
    <td>ARC-c</td>
    <td>Multiple-choice science</td>
    <td>2376</td>
  </tr>
  <tr>
    <td>ARC-e</td>
    <td>Multiple-choice science</td>
    <td>1172</td>
  </tr>
  <tr>
    <td>OBQA</td>
    <td>Multi-step reasoning</td>
    <td>500</td>
  </tr>
  <tr>
    <td>SIQA</td>
    <td>Social implications</td>
    <td>1954</td>
  </tr>
  <tr>
    <td>WinoG</td>
    <td>Fill-in-a-blank</td>
    <td>1267</td>
  </tr>
  <tr>
    <td>PIQA</td>
    <td>Physical commonsense</td>
    <td>1830</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>Yes/no questions</td>
    <td>3270</td>
  </tr>
  <tr>
    <td>HellaS</td>
    <td>Commonsense NLI</td>
    <td>10042</td>
  </tr>
</table>

Table 13: Information about datasets in the GLUE benchmark, with STS-B being a regression task and all other tasks falling into the categories of single-sentence or sentence-pair classification.

<table>
  <tr>
    <th rowspan="2">Corpus</th>
    <th rowspan="2">Metrics</th>
    <th rowspan="2">Task</th>
    <th colspan="2"># Train</th>
    <th colspan="2"># Test</th>
    <th rowspan="2"># Labels</th>
  </tr>
  <tr>
    <th># Train</th>
    <th># Val</th>
    <th># Test</th>
    <th># Labels</th>
  </tr>
  <tr>
    <td rowspan="2">CoLA<br>SST-2</td>
    <td rowspan="2">Matthews Corr.<br>Accuracy</td>
    <td>Acceptability</td>
    <td>8.55k</td>
    <td>1.04k</td>
    <td>1.06k</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Sentiment</td>
    <td>67.3k</td>
    <td>872</td>
    <td>1.82k</td>
    <td>2</td>
  </tr>
  <tr>
    <td rowspan="3">MRPC<br>STS-B<br>QQP</td>
    <td rowspan="3">Accuracy/F1<br>Pearson/Spearman Corr.<br>Accuracy/F1</td>
    <td>Paraphrase</td>
    <td>3.67k</td>
    <td>408</td>
    <td>1.73k</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Sentence similarity</td>
    <td>5.75k</td>
    <td>1.5k</td>
    <td>1.38k</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Paraphrase</td>
    <td>364k</td>
    <td>40.4k</td>
    <td>391k</td>
    <td>2</td>
  </tr>
  <tr>
    <td rowspan="3">MNLI<br>QNLI<br>RTE</td>
    <td rowspan="3">Accuracy</td>
    <td>NLI</td>
    <td>393k</td>
    <td>19.65k</td>
    <td>19.65k</td>
    <td>3</td>
  </tr>
  <tr>
    <td>QA/NLI</td>
    <td>105k</td>
    <td>5.46k</td>
    <td>5.46k</td>
    <td>2</td>
  </tr>
  <tr>
    <td>NLI</td>
    <td>2.49k</td>
    <td>277</td>
    <td>3k</td>
    <td>2</td>
  </tr>
</table>

RNNs or in the form of global convolution due to their linear time invariance (LTI) property. Mamba introduces a selective property to the structured state space model, tying parameters to the current input. This breaks the LTI property and hinders parallel training. To address this, Mamba employs a hardware-aware algorithm, enabling its SSM module to possess the selective property while performing parallel training.

To be specific, the discretization process can be expressed as:

\[
\bar{A} = \exp(\Delta A)
\]
\[
\bar{B} = (\Delta A)^{-1}(\exp(\Delta A) - I) \cdot \Delta B
\]

After that, the calculation in Mamba can be expressed as:

\[
h_t = \bar{A} h_{t-1} + \bar{B} x_t \\
y_t = C h_t
\]

where \( h_t \) is the hidden state at time \( t \) and \( x_t \) is
Table 14: Detailed information about the datasets in VTAB-1K benchmark.

<table>
  <tr>
    <th>Dataset</th>
    <th>Train</th>
    <th>Val</th>
    <th>Test</th>
    <th>#Classes</th>
  </tr>
  <tr>
    <td>CIFAR100</td>
    <td></td>
    <td></td>
    <td>10,000</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Caltech101</td>
    <td></td>
    <td></td>
    <td>6,084</td>
    <td>102</td>
  </tr>
  <tr>
    <td>DTD</td>
    <td></td>
    <td></td>
    <td>1,880</td>
    <td>47</td>
  </tr>
  <tr>
    <td>Oxford-Flowers102</td>
    <td></td>
    <td></td>
    <td>6,149</td>
    <td>102</td>
  </tr>
  <tr>
    <td>Oxford-Pets</td>
    <td></td>
    <td></td>
    <td>3,669</td>
    <td>37</td>
  </tr>
  <tr>
    <td>SVHN</td>
    <td></td>
    <td></td>
    <td>26,032</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Sun397</td>
    <td></td>
    <td></td>
    <td>21,750</td>
    <td>397</td>
  </tr>
  <tr>
    <td>Patch Camelyon</td>
    <td></td>
    <td></td>
    <td>32,768</td>
    <td>2</td>
  </tr>
  <tr>
    <td>EuroSAT</td>
    <td></td>
    <td></td>
    <td>5,400</td>
    <td>10</td>
  </tr>
  <tr>
    <th rowspan="11">VTAB-1k</th>
    <td>Resisc45</td>
    <td>800/1,000</td>
    <td>200</td>
    <td>6,300</td>
    <td>45</td>
  </tr>
  <tr>
    <td>Retinopathy</td>
    <td></td>
    <td></td>
    <td>42,670</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Clevr/count</td>
    <td></td>
    <td></td>
    <td>15,000</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Clevr/distance</td>
    <td></td>
    <td></td>
    <td>15,000</td>
    <td>6</td>
  </tr>
  <tr>
    <td>DMLab</td>
    <td></td>
    <td></td>
    <td>22,735</td>
    <td>6</td>
  </tr>
  <tr>
    <td>KITTI-Dist</td>
    <td></td>
    <td></td>
    <td>711</td>
    <td>4</td>
  </tr>
  <tr>
    <td>dSprites/location</td>
    <td></td>
    <td></td>
    <td>73,728</td>
    <td>16</td>
  </tr>
  <tr>
    <td>dSprites/orientation</td>
    <td></td>
    <td></td>
    <td>73,728</td>
    <td>16</td>
  </tr>
  <tr>
    <td>SmallNORB/azimuth</td>
    <td></td>
    <td></td>
    <td>12,150</td>
    <td>18</td>
  </tr>
  <tr>
    <td>SmallNORB/elevation</td>
    <td></td>
    <td></td>
    <td>12,150</td>
    <td>18</td>
  </tr>
</table>

Table 15: Performance on VTAB-1K benchmark when applying Lily to various modules in Transformer. The implementation here does not share \( A \) for simplicity (i.e., each layer has one \( A \)).

<table>
  <tr>
    <th rowspan="2">Average</th>
    <th colspan="6">Natural</th>
    <th colspan="3">Specialized</th>
    <th colspan="5">Structured</th>
  </tr>
  <tr>
    <th>CIFAR100</th>
    <th>Caltech101</th>
    <th>DTD</th>
    <th>Oxford102</th>
    <th>Pets</th>
    <th>SVHN</th>
    <th>Sun397</th>
    <th>Camelyon</th>
    <th>EuroSAT</th>
    <th>Resisc45</th>
    <th>Retinopathy</th>
    <th>Clevr-Count</th>
    <th>Clevr-Dist</th>
    <th>DMLab</th>
    <th>KITTI-Dist</th>
    <th>dSpr-Loc</th>
    <th>dSpr-Ori</th>
    <th>SNORB-Azm</th>
    <th>SNORB-Ele</th>
  </tr>
  <tr>
    <td>qv</td>
    <td>76.9</td>
    <td>73.2</td>
    <td>92.3</td>
    <td>72.2</td>
    <td>99.3</td>
    <td>91.4</td>
    <td>89.0</td>
    <td>56.5</td>
    <td>87.6</td>
    <td>95.2</td>
    <td>84.8</td>
    <td>75.9</td>
    <td>83.7</td>
    <td>65.8</td>
    <td>52.8</td>
    <td>81.2</td>
    <td>87.6</td>
    <td>52.4</td>
    <td>36.3</td>
    <td>43.4</td>
  </tr>
  <tr>
    <td>mlp</td>
    <td>77.0</td>
    <td>74.0</td>
    <td>92.6</td>
    <td>72.2</td>
    <td>99.4</td>
    <td>91.5</td>
    <td>89.0</td>
    <td>55.9</td>
    <td>88.2</td>
    <td>95.5</td>
    <td>85.4</td>
    <td>76.0</td>
    <td>83.3</td>
    <td>63.1</td>
    <td>53.0</td>
    <td>81.4</td>
    <td>86.5</td>
    <td>53.8</td>
    <td>35.6</td>
    <td>43.3</td>
  </tr>
  <tr>
    <td>qvmpl</td>
    <td>77.1</td>
    <td>73.9</td>
    <td>93.2</td>
    <td>72.7</td>
    <td>99.4</td>
    <td>91.6</td>
    <td>89.7</td>
    <td>56.5</td>
    <td>87.9</td>
    <td>95.3</td>
    <td>85.0</td>
    <td>76.1</td>
    <td>84.6</td>
    <td>65.2</td>
    <td>53.0</td>
    <td>82.1</td>
    <td>86.7</td>
    <td>53.0</td>
    <td>36.0</td>
    <td>42.8</td>
  </tr>
  <tr>
    <td>kvmlp</td>
    <td>77.3</td>
    <td>74.0</td>
    <td>92.3</td>
    <td>72.6</td>
    <td>99.3</td>
    <td>91.5</td>
    <td>89.2</td>
    <td>56.7</td>
    <td>88.2</td>
    <td>95.4</td>
    <td>85.3</td>
    <td>76.0</td>
    <td>84.6</td>
    <td>64.9</td>
    <td>53.4</td>
    <td>81.7</td>
    <td>87.5</td>
    <td>52.9</td>
    <td>36.9</td>
    <td>45.2</td>
  </tr>
</table>

the corresponding input token. Delta projection is a module in SSM that’s learnable and tasked with transforming the parameter \( \Delta \). Since adapting the delta projection alone can indirectly adapt the entire SSM module (i.e., \( \tilde{A} \) and \( \tilde{B} \) are determined by \( \Delta \)), it is the most critical component of the SSM module.

We investigate the performance of two adaptation strategies: adapting only the input linear projection layer (denoted as "in") and adapting both the input linear projection layer and the SSM (denoted as "\( \Delta + \text{in} \)" since we only adapt the delta projection in the SSM module). Our results, as shown in Table 1, indicate that applying Lily solely to the input projection yields better performance than applying it to both the input and delta projection modules. This suggests that when adapting Mamba-based models under the paradigm of low-rank adaptation, it is optimal to adapt only the input projection module outside the SSM module. These findings highlight the need for further research into the impact of fine-tuned modules in Mamba on overall performance. Additionally, developing PEFT methods specifically tailored to Mamba-based models, whether for vision or language foundation models, is also a promising direction for future work.

G Performance with Different Learning Rates

Since we only tuned the learning rate in the commonsense reasoning experiment, we provide the performance of commonsense reasoning under different learning rates in Table 16.
Table 16: Commonsense reasoning results of Lily under various learning rates.

<table>
  <tr>
    <th>Model</th>
    <th>Lr</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>SIQA</th>
    <th>HellaSwag</th>
    <th>WinoGrande</th>
    <th>ARC-e</th>
    <th>ARC-c</th>
    <th>OBQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">LLaMA3-8B</td>
    <td>1e-3</td>
    <td>70.7</td>
    <td>84.6</td>
    <td>77.6</td>
    <td>87.8</td>
    <td>77.3</td>
    <td>88.5</td>
    <td>74.1</td>
    <td>80.8</td>
    <td>80.2</td>
  </tr>
  <tr>
    <td>5e-4</td>
    <td>71.8</td>
    <td>86.5</td>
    <td>77.9</td>
    <td>82.8</td>
    <td>83.1</td>
    <td>88.6</td>
    <td>76.8</td>
    <td>81.4</td>
    <td>81.1</td>
  </tr>
  <tr>
    <td>3e-4</td>
    <td>72.9</td>
    <td>85.6</td>
    <td>77.8</td>
    <td>92.7</td>
    <td>83.3</td>
    <td>89.7</td>
    <td>77.6</td>
    <td>82.8</td>
    <td>82.8</td>
  </tr>
</table>

H Does Selectivity Help?

Lily introduced selective weight combination to selectively incorporate information from other layers. To verify the effectiveness of this selectivity, we remove the router from Lily and evaluate the impact. The modified algorithm without the router is presented in Fig. 8. We conduct experiments on commonsense reasoning to investigate the effect of removing selectivity from Lily.

As shown in Table 17, removing selectivity from Lily results in generally poorer performance compared to vanilla Lily. This is likely because the lack of selectivity causes Lily to simply aggregate all the \( B \) expert, leading to inferior performance. This validates the design choice of using routers in Lily to selectively allocate weights to \( B \) experts, rather than simply summing them.

I How to Allocate Parameters?

Since Lily alters the traditional LoRA’s layer-bound setup, increasing the parameters of Lily can be achieved through two approaches: 1) increasing \( ne \), i.e., increasing the number of \( A \) and \( B \) experts, and 2) increasing the rank, i.e., increasing the parameter size of each individual \( A \) or \( B \) expert. In this section, we investigate which factor has the greatest impact on performance. We conduct experiments on the commonsense reasoning task. Specifically, we maintain the same parameter count and learning rate, and achieve the same parameter count by setting different ranks and adjusting the corresponding \( ne \) (e.g., r=16, ne=4 versus r=8, ne=8). The results are shown in Fig. 9, from which we observe that more \( A \) and \( B \) experts with smaller rank (i.e., bigger \( ne \) and smaller rank) generally performs worse. We argue that this is because, although increasing the attention granularity allows for finer details, the resulting performance gain is not as significant as the gain obtained by increasing the rank, i.e., increasing the model’s capacity to learn more information. This gives us an insight that, in Lily, increasing \( ne \) to increase the parameters is less effective than directly increasing the rank in terms of potential performance gain.
class lily_adapter_monoscale(nn.Module):

    def __init__(self, hidden_dim, ne, lp, hps, mlp=False):
        super().__init__()
        self.hps = hps
        self.ne = ne
        self.lp = lp
        self.scale = 1 / ne
        if mlp:
            self.non_linear = nn.ReLU()
        else:
            self.non_linear = nn.Identity()
    def forward(self, x):
        hidden = self.non_linear(self.lp(x))
        combined_hp = torch.sum(self.hps, 0) * self.scale
        return torch.matmul(hidden, combined_hp)

Figure 8: Implementation of Lily with no selectivity.

Table 17: Commonsense reasoning results of Lily without selectivity. We provide results using two learning rates.

<table>
  <tr>
    <th>Model</th>
    <th>Lr</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>SIQA</th>
    <th>HellaSwag</th>
    <th>WinoGrande</th>
    <th>ARC-e</th>
    <th>ARC-c</th>
    <th>OBQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td></td>
    <td>3e-4</td>
    <td>64.0</td>
    <td>82.6</td>
    <td>78.5</td>
    <td>77.0</td>
    <td>79.6</td>
    <td>88.4</td>
    <td>74.5</td>
    <td>82.0</td>
    <td>78.3</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>5e-4</td>
    <td>71.3</td>
    <td>85.5</td>
    <td>78.1</td>
    <td>84.3</td>
    <td>79.6</td>
    <td>86.4</td>
    <td>76.1</td>
    <td>79.0</td>
    <td>79.8</td>
  </tr>
</table>

![Line plots showing accuracy vs rank for BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA, and Avg.](page_320_1042_1104_482.png)

Figure 9: Results on commonsense reasoning tasks when applying different settings of rank. The hyperparameter \( ne \) is specifically tuned to maintain the same amount of parameter count for a fair comparison.
J  More on Subject-driven Generation

We provide more results on subject-driven generation in Fig. 10 and Fig. 11.

K  From a Feature Merging Perspective

Apart from having higher-rank weight updates than LoRA, Lily also enables comprehensive information access across layers. Lily enables access to information or features from all other layers when adapting a target module at a specific layer thanks to the inter-connectivity of the adapters. We aim to understand how Lily achieves this comprehensive information access from the perspective of visual tasks as shown in Fig. 12. We can observe that, in Lily, the distinctness of the attention maps between layers is not as pronounced as in LoRA. This validates Lily’s ability to enable all-level information access, since adaptation at each layer takes into account features from other layers. Additionally, we specifically visualize the actual feature differences between different layers in Fig. 13. We observe that Lily has more points with low feature differences (blue color) than LoRA, indicating that the distinctness of features between layers in Lily is generally lower than in LoRA. This further demonstrates Lily’s ability to enable comprehensive information access. Although we enable all-level information access, what prevents the features from becoming completely identical is the selectivity introduced by Lily, which we specify in the following section.

L  More on Attention Maps of Lily and LoRA

We provide more visualization results of the attention map from both LoRA and Lily on Caltech101 dataset from VTAB-1K benchmark in Fig. 14.
Input images (not all included)

Prompts:
1. A [v] grey sloth plushie floating on top of water
2. A [v] grey sloth plushie in the snow
3. A [v] grey sloth plushie on a cobblestone street
4. A [v] grey sloth plushie on top of a dirt road
5. A [v] grey sloth plushie on top of a white rug
6. A [v] grey sloth plushie on top of a wooden floor

LoRA

Lily

Input images (not all included)

Prompts:
1. A [v] bear plushie floating on top of water
2. A [v] bear plushie in the snow
3. A [v] bear plushie on a cobblestone street
4. A [v] bear plushie on top of a dirt road
5. A [v] bear plushie on top of a white rug
6. A [v] bear plushie on top of a wooden floor

Figure 10: More subject-driven generation results for unreported subjects.
LoRA        Lily

Prompts:
1. A [v] {} on top of a dirt road
2. A [v] {} on top of a white rug
3. A [v] {} with a tree and autumn leaves in the background
4. A [v] {} with a wheat field in the background
5. A {} with the Eiffel Tower in the background

subject = duck toy

subject = wolf plushie

Figure 11: More subject-driven generation results for subjects that are reported in the experiment section.
Figure 12: Attention maps of Lily and LoRA. The input images for the example here are taken from Caltech101 datasets from VTAB-1K benchmark. It can be observed that features from a certain layer have more similarity to those in other layers in Lily than in LoRA.

Figure 13: Feature difference measured in absolute distance for each element. We compare Lily and LoRA in terms of the difference between features from different layers. In this example image taken from Caltech101, we visualize the feature difference between layers 6 and 1, as well as between layers 6 and 9.
<table>
  <tr>
    <th></th>
    <th>Input Image</th>
    <th>Layer1</th>
    <th>Layer6</th>
    <th>Layer9</th>
  </tr>
  <tr>
    <td>Lily</td>
    <td><img src="page_340_256_180_180.png" alt="Input Image (Lily)"></td>
    <td><img src="page_520_256_180_180.png" alt="Layer1 (Lily)"></td>
    <td><img src="page_700_256_180_180.png" alt="Layer6 (Lily)"></td>
    <td><img src="page_880_256_180_180.png" alt="Layer9 (Lily)"></td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><img src="page_340_436_180_180.png" alt="Input Image (LoRA)"></td>
    <td><img src="page_520_436_180_180.png" alt="Layer1 (LoRA)"></td>
    <td><img src="page_700_436_180_180.png" alt="Layer6 (LoRA)"></td>
    <td><img src="page_880_436_180_180.png" alt="Layer9 (LoRA)"></td>
  </tr>
  <tr>
    <td>Lily</td>
    <td><img src="page_340_616_180_180.png" alt="Input Image (Lily)"></td>
    <td><img src="page_520_616_180_180.png" alt="Layer1 (Lily)"></td>
    <td><img src="page_700_616_180_180.png" alt="Layer6 (Lily)"></td>
    <td><img src="page_880_616_180_180.png" alt="Layer9 (Lily)"></td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><img src="page_340_806_180_180.png" alt="Input Image (LoRA)"></td>
    <td><img src="page_520_806_180_180.png" alt="Layer1 (LoRA)"></td>
    <td><img src="page_700_806_180_180.png" alt="Layer6 (LoRA)"></td>
    <td><img src="page_880_806_180_180.png" alt="Layer9 (LoRA)"></td>
  </tr>
  <tr>
    <td>Lily</td>
    <td><img src="page_340_986_180_180.png" alt="Input Image (Lily)"></td>
    <td><img src="page_520_986_180_180.png" alt="Layer1 (Lily)"></td>
    <td><img src="page_700_986_180_180.png" alt="Layer6 (Lily)"></td>
    <td><img src="page_880_986_180_180.png" alt="Layer9 (Lily)"></td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><img src="page_340_1166_180_180.png" alt="Input Image (LoRA)"></td>
    <td><img src="page_520_1166_180_180.png" alt="Layer1 (LoRA)"></td>
    <td><img src="page_700_1166_180_180.png" alt="Layer6 (LoRA)"></td>
    <td><img src="page_880_1166_180_180.png" alt="Layer9 (LoRA)"></td>
  </tr>
  <tr>
    <td>Lily</td>
    <td><img src="page_340_1346_180_180.png" alt="Input Image (Lily)"></td>
    <td><img src="page_520_1346_180_180.png" alt="Layer1 (Lily)"></td>
    <td><img src="page_700_1346_180_180.png" alt="Layer6 (Lily)"></td>
    <td><img src="page_880_1346_180_180.png" alt="Layer9 (Lily)"></td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td><img src="page_340_1526_180_180.png" alt="Input Image (LoRA)"></td>
    <td><img src="page_520_1526_180_180.png" alt="Layer1 (LoRA)"></td>
    <td><img src="page_700_1526_180_180.png" alt="Layer6 (LoRA)"></td>
    <td><img src="page_880_1526_180_180.png" alt="Layer9 (LoRA)"></td>
  </tr>
</table>

Figure 14: More results of attention maps from LoRA and Lily. All images are taken from Caltech101 dataset.