Title: 3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability
Venue: NeurIPS 2024
Accepted: yes
Keywords: parameter-efficient finetuning

orthogonal finetuning

batching

interpretability
Abstract: Parameter-efficient finetuning (PEFT) methods effectively adapt large language models (LLMs) to diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges to PEFT beyond mere parameter efficiency. One notable challenge involves the efficient deployment of LLMs equipped with multiple task- or user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches to address different challenges. In this paper, we introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with <0.1% trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.

3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability

Abstract

Parameter-efficient finetuning (PEFT) methods effectively adapt large language models (LLMs) to diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges to PEFT beyond mere parameter efficiency. One notable challenge involves the efficient deployment of LLMs equipped with multiple task- or user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches to address different challenges. In this paper, we introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with < 0 . 1% trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.

0.0

0.1

0.2

87.75

88.00

88.25

88.50

88.75

Full FT

Adapter

LoRA

Adapter

FFN

RED

LoReFT

RoAd

GLUE

0

1

2

3

70

75

80

Adapter

P

Adapter

S

DoRA

LoRA

Prefix

LoReFT

RoAd

GPT3.5

Commonsense

0

1

2

3

40.0

42.5

45.0

47.5

50.0

52.5

Adapter

P

Adapter

S

LoRA

Prefix

LoReFT

RoAd

Arithmetic

Trainable parameters (%)

1 Introduction

Large language models (LLMs), trained on extensive web-scale datasets to perform tasks such as predicting masked words [8, 31, 45] or anticipating the next word in a sentence [17, 52, 53],

demonstrate remarkable effectiveness across a range of NLP applications. For tasks where the data distribution diverges from that of the pretraining corpus, finetuning emerges as an effective way to tailor an LLM to specific requirements. Leveraging the capabilities of LLMs, recent studies [13, 14, 22, 23, 25, 27, 42, 60, 62, 65] demonstrate that training only a subset of an LLM's parameters can yield performance on par with full finetuning. This approach, termed parameter-efficient finetuning (PEFT), provides two primary advantages: (1) It reduces the storage requirements for trained parameters, as it necessitates preserving only a universal LLM alongside a minimal set of task-specific parameters; (2) It decreases GPU memory consumption during finetuning, owing to the reduction in optimizer state sizes which correlate directly with the number of trainable parameters.

With the evolution of PEFT, concerns extend beyond mere parameter efficiency. PEFT encounters a variety of challenges brought forth by diverse applications. A significant challenge is the efficient deployment of personalized or task-specific LLMs [25, 57]. These applications frequently require distinct sets of trained parameters for different tasks or users. When multiple users submit requests simultaneously, it becomes crucial to process these requests collectively in a single batch. Given that each request may require a unique set of parameters, using batch matrix multiplication can efficiently handle these requests by leveraging GPU parallelism. However, the batch matrix multiplication still incurs considerable overhead [1, 57], necessitating the exploration of more efficient methods.

Another challenge is the interpretability of LLMs that contain a billion-scale of parameters, making it difficult to explore their mechanism. PEFT provides an alternative approach by constraining the number of trainable parameters, thereby aiding in interpretability. Recent advancements in PEFT methods, particularly those focusing on representation editing [54, 60, 67], can be incorporated within an intervention framework [11]. This integration enhances their capability for interpretability, offering a more manageable means of dissecting the operational intricacies of LLMs.

In this paper, we introduce a novel technique termed 2D rotary adaptation (RoAd) which efficiently adapts LLMs using a minimal number of trainable parameters. Furthermore, RoAd enhances both batching efficiency and composability. Our initial investigation reveals that finetuning primarily alters the angular components of the representations in pretrained LLMs, rather than their magnitudes (Section §3.1). Based on this observation, we employ a strategy of rotating certain subspaces within the representations to emulate finetuning effects. Specifically, we implement a 2D rotational approach on the representations and develop three distinct variants of RoAd (Section §3.2).

To assess the efficacy of RoAd, we perform comprehensive evaluations on the GLUE benchmark [56], eight commonsense reasoning tasks and four arithmetic reasoning tasks, utilizing RoBERTa [31] and LLaMA [52, 53] (Section §4.1). The results consistently show that RoAd surpasses other PEFT methods while maintaining a significantly reduced scale of trainable parameters ( < 0 . 1% ), as depicted in Figure 1. Additionally, RoAd employs element-wise rather than matrix multiplication, which notably improves throughput when serving heterogeneous requests within the same batch, achieving twice the throughput of LoRA [14] (Section §4.2). Furthermore, RoAd can be seamlessly integrated within an intervention framework [11], thereby enhancing model interpretability. We illustrate this through a composition experiment, demonstrating RoAd's capacity to merge weights trained for different tasks and display a new capability (Section §4.3).

2 Background

In this section, we outline the challenges tackled in this work, illustrating the constraints of existing methods and objectives that drive the development of the proposed method, RoAd.

2.1 Parameter-efficient finetuning (PEFT)

Existing PEFT techniques can be categorized into three groups: adapter-based, prompt-based, and latency-less methods. Adapter-based methods [12, 13, 42] incorporate adapters either in parallel with or sequentially to the existing Transformer [55] modules. This incorporation necessitates modifications to the LLM architecture, consequently adding extra latency during inference. Promptbased methods [19, 21, 43] enhance the input by appending new trainable tokens, which lengthens the sequence and thereby increases the computational overhead during inference. Latency-less methods, such as LoRA [14] and its variants [22, 27, 65], apply low-rank matrices to adapt the pretrained weights. These matrices can be seamlessly integrated into the existing weight matrices following

finetuning, thus preserving the original LLM architecture. Specifically, LoRA adapts an LLM as W = W 0 +∆ W , where W 0 ∈ R d 1 × d 2 is the pretrained weight and ∆ W = BA with B ∈ R d 1 × r , A ∈ R r × d 2 , r ≪ d 1 and r ≪ d 2 . Our proposed method, RoAd, aligns with the latency-less category and integrates effortlessly into the existing linear layer without imposing additional overhead during inference. Moreover, RoAd demonstrates exceptional parameter efficiency. The quantity of its trainable parameters is equivalent to that of a LoRA module with a rank r = 0 . 5 .

Orthogonal finetuning. Drawing on the concept of hyperspherical energy and its role in characterizing generalization [28, 29], OFT [44] introduces orthogonal finetuning, an effective PEFT method for finetuning text-to-image diffusion models. Specifically, OFT implements an orthogonal matrix R ∈ R d 1 × d 1 to the pretrained weight W 0 , so the input x ∈ R d 1 to a linear layer after adaptation becomes z = ( RW 0 ) ⊤ x . R is parameter-efficient because it is a block-diagonal matrix with n blocks as R = diag ( R 1 , ..., R i , ..., R n ) , where each block R i ∈ R w × w has a dimension w = d 1 /n . To maintain orthogonality, R i is derived using Cayley parameterization: R i = ( I + Q i )( I -Q i ) -1 with Q i ∈ R w × w being a skew-symmetric matrix ( Q i = -Q ⊤ i ). In sum, { Q i } n i =1 serve as the trainable parameters and R is constructed from them with Cayley parameterization. Subsequent advancement, BOFT [30], leverages butterfly factorization to further refine OFT's parameter efficiency. However, both OFT and BOFT, due to their reliance on matrix inversions in the Cayley parameterization and increased storage of intermediate activations, necessitate additional GPU memory and increase training duration compared to other PEFT approaches. Conversely, RoAd, which may be considered as a specialized case of OFT with w = 2 , offers a faster and more memory-efficient solution by inherently maintaining orthogonality without requiring further parameterization.

2.2 Batching

Batching in this context refers to processing multiple heterogeneous requests, each requiring different adapters 2 for inference. This scenario commonly arises when serving personalized or task-specific LLMs. Specifically, we consider a setup where distinct adapters instead of a shared adapter are finetuned for various tasks to achieve optimal performance. During inference, each request in a batch pertains to a different task and necessitates a unique adapter.

Consider that we have finetuned distinct LoRA modules for b tasks, denoted as { A i , B i } b i =1 . For a batch of b requests represented as X ∈ R b × l × d 1 , where l is the maximum sequence length across the requests, each request requires a different LoRA module. To exploit the parallel processing capabilities of GPUs, the output Z of a linear layer can be computed as follows: First, the output from the pretrained layer is computed as Z 0 = torch.mm ( X , W 0 ) . Subsequently, the intermediate output from the first low-rank matrix, ˆ B ∈ R b × d 1 × r (a concatenation of { B i } b i =1 ), is obtained as Z 1 0 = torch.bmm ( X , ˆ B ) . The output from the second low-rank matrix, ˆ A ∈ R b × r × d 2 (a concatenation of { A i } b i =1 ), follows as Z 1 = torch.bmm ( Z 1 0 , ˆ A ) . Finally, these outputs are summed to produce Z = Z 0 + Z 1 . It is noteworthy that batch matrix multiplication (BMM), as implemented in torch.bmm , often introduces substantial overhead [1], reducing throughput and increasing latency, which adversely impacts user experience in time-sensitive applications.

In contrast, prompt-based methods circumvent the use of BMM by appending trainable tokens to each request, simplifying the computational process. However, prompt-based methods with long prompt tokens are difficult to optimize, which degrades performance compared to other PEFTs [14, 15]. (IA) 3 [25] proposes adapting LLM by multiplying the output from a linear layer with a trainable vector, involving only element-wise multiplication for efficient batching. A recent development, FLoRA [58], builds on (IA) 3 by employing two low-rank matrices while maintaining element-wise operations. Although our proposed method, RoAd, requires BMM, its sparse structure allows a reformulation of BMM and results in an overhead equivalent to element-wise multiplication.

2.3 Intervention and composability

Numerous studies [10, 11, 37, 38, 40] have provided support for the linear representation hypothesis [35, 46, 49] that concepts are represented within linear subspaces of neural network representations. To examine if a concept is captured within a linear subspace of a representation, Geiger et al. [11]

0.00

0.25

0.50

0.75

D

0.025

0.050

0.075

0.100

0.125

0.150

M

Full Finetuning

0.6

0.7

0.8

0.9

D

0.02

0.04

0.06

0.08

0.10

M

LoRA

RTE

MRPC

STS-B

CoLA

Tasks

0

20

40

60

Score

z

=

W T x

z

=

W T x

, random

z=cos(W, x)

z

=||

W

||

||

x

||

suggests employing a distributed interchange intervention (DII) defined as:

b denotes the hidden representation generated at row i and column k when the model processes an input, while s represents the corresponding representation when the model processes a different input. The matrix R ∈ R r × d 1 , consisting of orthogonal rows, serves as a low-rank projection matrix where d 1 is the dimension of the representation and r is the subspace dimension under intervention. Equation (1) illustrates the application of a DII to b using a counterfactual source representation s . 3

Drawing inspiration from this established framework, a recent study, LoReFT [61], introduces a method for finetuning specific positions of the representations to adapt LLM. This study further demonstrates that several prior approaches of representation editing [54, 60, 67] can be effectively integrated within this framework. Interestingly, the application of RoAd to representations can also be conceptualized as DII, offering interpretability potential. To demonstrate one aspect of interpretability for RoAd, we primarily conduct a qualitative experiment focused on task composition. This experiment involves combining the weights of models trained on distinct tasks to showcase the capability for multitasking learning without the need for additional adaptation [16, 20, 61, 64, 66].

3 Method

In this section, we first perform two pilot studies to ascertain the key factor influencing the adaptation of LLMs. Following this, we present our proposed method, the 2D rotary adaptation (RoAd), which serves as an effective PEFT method addressing the various challenges outlined in Section § 2.

3.1 Pilot study

Study 1: Variations in magnitude and angular displacement. Assume x 0 , x ∈ R d 1 are representations of the same token from a pretrained and finetuned LLM, respectively. We define the relative change in magnitude as ∆ M = ∣ ∣ ∥ x ∥ 2 -∥ x 0 ∥ 2 ∣ ∣ / ∥ x 0 ∥ 2 and compute the angular displacement as ∆ D = cos( x , x 0 ) ∈ [ -1 , 1] . A larger ∆ M and a smaller ∆ D indicate more significant changes in magnitude and angular displacement, respectively. Our study involves: (1) finetuning RoBERTa-base [31] on the SST-2 task [50] using either full finetuning or LoRA; (2) extracting representations x 0 and x from the output of the second-last Transformer block for the [CLS] token across all samples in the development set, followed by computing ∆ M and ∆ D . 4 As depicted in Figure 2 (Left and Middle), there is a more pronounced change in ∆ D than in ∆ M for both full finetuning and LoRA. 5

Study 2: Disentanglement of magnitude and angle. To ascertain whether angular or magnitude adjustments are more critical for finetuning, we implement a disentanglement study. This involves freezing RoBERTa-base and appending a two-layer classifier on top of it. The first layer of this

classifier incorporates a weight matrix W ∈ R d 1 × d 1 . Under standard operations, the output from this layer is computed as z = W ⊤ x 0 . To distinctly evaluate the impacts of magnitude and angle, we modify the output to retain only the magnitude component as z i = ∥ W : ,i ∥ 2 · ∥ x 0 ∥ 2 , or solely the angular component as z i = cos( W : ,i , x 0 ) ( z i is the i th element of z ). The modified classifier was then finetuned on four GLUE tasks with different metrics detailed in Table C.1. Additionally, a weak baseline employing a randomly initialized RoBERTa-base is included. As shown in Figure 2 (Right), angular information is paramount in finetuning, whereas reliance solely on magnitude information even leads to inferior results compared to the random backbone.

Both studies indicate that angular information is more crucial than magnitude information for adapting a pretrained LLM to a downstream task. However, rotating the entire d 1 dimensions of the representation for finetuning incurs substantial computational costs. These costs are primarily reflected in a large number of trainable parameters, necessitating a dense matrix R ∈ R d 1 × d 1 , and in the requirement to maintain its orthogonality. Could we only rotate a subspace of the representation and design a R that is always orthogonal without any parameterization as OFT [44]? The first idea that comes to our mind is 2D rotation which only rotates two dimensions at a time and inherently maintains orthogonality.

3.2 2D rotary adaptation

Suppose that W 0 ∈ R d 1 × d 2 is the pretrained weight of a linear layer, x ∈ R d 1 is the input of a token to this linear layer, R ∈ R d 2 × d 2 is the rotation matrix, the adapted output from the linear layer is z = Rh = R ( W 0 ⊤ x ) . The rotation matrix R is defined as follows:

The trainable parameters are denoted as { θ i } d 2 / 2 i =1 . This 2D rotary adaptation involves rotating pairs of adjacent dimensions of h , specifically dimensions 2 i -1 and 2 i , using the rotation matrix R i . 6 The rotation matrix R is characterized by its parameter efficiency, which

ℎ

!

ℎ

"

ℎ

#

ℎ

$

ℎ

%&#

ℎ

%

𝑧

!

𝑧

"

𝑧

#

𝑧

$

𝑧

%&#

𝑧

%

ℎ

!

ℎ

"

𝑧

!

𝑧

"

𝜃

!

𝑧

!

#

+𝑧

"

#

!

=	𝛼

!

ℎ

!

#

+ℎ

"

#

!

. . .

. . .

is attributed to its sparse structure and the parameter sharing within each block R i . Additionally, R can be integrated directly into the existing pretrained weights, forming W = W 0 R ⊤ , which does not incur additional computational costs during inference. This design closely mirrors RoPE [51], with the notable difference that in our RoAd, θ i is trainable and R i does not incorporate positional information. The overview of RoAd is shown in Figure 3.

Relaxation to orthogonality. Referring to Figure 2 (Right), while reliance predominantly on angular information substantially outperforms reliance on magnitude information, it remains less effective than using both angular and magnitude information for the tasks of MRPC, STS-B, and CoLA. Furthermore, both fully- and LoRA-finetuned LLMs exhibit slight adaptations in magnitude, as depicted in Figure 2 (Left and Middle). Consequently, we modify R i by incorporating α i to regulate the magnitude. We define a general R i as follows:

We develop three variants of RoAd by altering the configuration of shared parameters as outlined in Table 1. RoAd 1 introduces a minimal change to Equation (2) by incorporating a scaling factor α i . RoAd 1 already shows impressive results for most tasks in Section §4.1. For some knowledge-intensive tasks, we observe that RoAd 2 and RoAd 4 obtain better results with more trainable parameters. To preserve the starting point of LLMs [23], we always initialize α i = 1 and θ i = 0 .

Batching. In practice, we don't need to save R as a sparse matrix and do matrix multiplication. Taking RoAd 1 as an example in Equation (4), we only save two vectors: R 1 and R 2 . Then z = Rh = R 1 ⊗ h + R 2 ⊗ ˆ h , where ˆ h is a rearranged version of h and ⊗ denotes element-wise multiplication. This reformulation not only simplifies the representation of R but also enhances the efficiency of batching in RoAd, relying solely on element-wise multiplications rather than BMM.

̸

̸

̸

̸

̸

̸

Composability. RoAd can be incorporated into the DII framework as Φ( h ) = Rh = h + R ( h -R ⊤ h ) , with Rs in Equation (1) being set to h . Although a degree of relaxation is introduced to the orthogonality of R , it is important to note that the rows of R remain orthogonal to each other within non-adjacent segments of the same block, R i . This offers a possibility for composability. We can finetune some rows on one task and other orthogonal rows on another task. Since they are orthogonal to each other, these two tasks should minimally affect each other, and the combination of these rows after finetuning could bring new multitasking learning ability.

RoAd can be considered as a special case of OFT [44] with w = 2 . However, it is much more parameter- and memory-efficient and faster. Please refer to Section §D.1 for a detailed discussion.

4 Experiments

In this section, we begin by implementing RoAd to finetune various LLMs across three benchmarks. Subsequently, we illustrate its efficiency in batching processes and demonstrate its composability. Unless otherwise noted, RoAd is applied to all linear layers within the LLMs. All of our experiments are conducted on A100 80GB GPU with the frameworks, Transformers [59] and PEFT [34].

4.1 Results on downstream tasks

Natural language understanding (NLU). We evaluate the effectiveness of RoAd on the GLUE benchmark [56] for its ability of NLU with RoBERTa [31] as the backbone. Unlike many previous works [14, 22, 23, 31, 65] that employ the GLUE development sets for both validation and testing, here we partition the development set into distinct validation and test subsets to mitigate the risk of overfitting. For comprehensive information regarding the split of the development set, the search space of hyperparameters, the optimal hyperparameter configurations, and other details crucial for reproducibility, please see Section § C.1.

As shown in Table 2, RoAd1 outperforms all other PEFT methods with < 0 . 1% trainable parameters for both sizes of RoBERTa on average, being the only PEFT method that matches or outperforms full finetuning. These results show that 2D rotation (with a few scaling) can efficiently adapt LLM.

Commonsense reasoning . In assessing the capacity of LLaMA [52] for commonsense reasoning, we focus on eight representative tasks: BoolQ [4], PIQA [3], SIQA [48], HellaSwag [63], WinoGrande [47], ARC-e, ARC-c [5], and OBQA [36]. The setting here contrasts with the NLU experiments where each task involves finetuning a separate LLM. Instead, we adopt a unified strategy by finetuning a single LLM across all tasks as delineated in Hu et al. [15]. Such a setting is designed to mitigate overfitting and aligns more closely with real-world applications. Specifically, the training and test sets from these eight tasks are reformulated according to a predefined template, so all tasks can be trained or evaluated in a generative way. For all finetuning experiments on LLaMA, we follow a recipe in Table C.5 without extensive searching. Please see Section § C.2 for more training details.

As shown in Table 3, RoAds still perform the best across various PEFT methods for both LLaMA-7B and LLaMA-13B on average. The strong baseline to RoAd is a recent representation finetuning method, LoReFT [61], 80.2 vs. 79.2 and 83.3 vs. 83.0 for RoAd 1 for LLaMA-7B and LLaMA-13B, respectively. With a slightly increasing number of trainable parameters from RoAd 1 to RoAd 2 or RoAd 4 , RoAd matches or outperforms LoReFT. The same story is also told for another two versions of LLaMA, i.e. LLaMA2 [53] and LLaMA3, in Table D.2.

Arithmetic reasoning. To assess the arithmetic reasoning ability of LLMs, we evaluate the finetuned LLMs on the test sets of four tasks: AQuA [24], GSM8K [6], MAWPS [18] and SVAMP [41]. Similar to the commonsense reasoning tasks, we finetune a single LLM for all four arithmetic reasoning tasks. The training dataset is Math10K [15] which is constructed from the training sets of GSM8K, MAWPS, MAWPS-single and AQuA. The training recipe is similar to the one used for commonsense reasoning as shown in Table C.5. Please see Section § C.3 for more training details.

Different from the results of NLU and commonsense reasoning tasks, RoAd doesn't always perform the best on the arithmetic reasoning tasks, as shown in Figure 4. For the smaller-size LLM, LLaMA7B, RoAd is significantly better than other PEFT methods with < 0 . 1% trainable parameters, but worse than LoRA and Adapter P with more than 10 × trainable parameters. However, for the largersize LLM, LLaMA-13B, all RoAd variants are better than other PEFT methods, which shows its scalability and potentially implies even better results for larger LLMs.

Observed from the above-mentioned results, for enhanced performance on downstream tasks and if a marginal increase in the storage capacity for trained parameters is acceptable, RoAd 4 is the preferable option. Conversely, if the objective is to investigate how the model adjusts in terms of angle and magnitude, RoAd 1 is recommended. Notably, all

variants of RoAd incur the same computational overhead for batching.

Instruction-following ability. We further benchmark RoAd using AlpacaEval2.0 [9]. We finetune LLaMA2-7B with two instruction-tuning datasets and evaluate the model using AlpacaEval2.0. This evaluation employs GPT-4 [39] to assess the responses generated by the finetuned model against those produced by Text-davinci-003. We don't choose GPT-4 as the reference model, because GPT-4 is too powerful than LLaMA2-7B. The proof-of-concept experiment with LoRA shows the win-rate < 5%. As shown in Table 5, RoAd 1 demonstrates superior performance compared to all baselines, while utilizing the least number of trainable parameters.

Multimodal ability. Lastly, we apply RoAd to the LLM backbone of LLaVA [26]. Liu et al. [26] requires 4.61% trainable parameters for LoRA on this task, while most tasks with LoRA in our paper need < 1%, showing that this task is knowledge-intensive. Therefore, we need to scale RoAd's trainable parameters. For this purpose, we combine it with LoRA due to the limited number of θ i and α i in R . The combination is represented as z = ( RW 0 ⊤ +( BA ) ⊤ ) x , where A and B are from LoRA. We adjust the LoRA rank to vary the number of trainable parameters. We combine RoAd 1

1 2

4

8

16

LoRA rank

25

30

35

40

45

50

Tokens/second

merged

unmerged

256

1024

2048

4096

#generated tokens

60

80

100

120

140

160

RoAd

LoRA

1 2

4

8

16

Batch size / #requests

50

100

150

200

250

RoAd

LoRA

LLaMA-7B

LLaMA-13B

with LoRA, but not RoAd 2 or RoAd 4 , as their primary design purpose is to increase the number of trainable parameters.

As shown in Table 6, with only 0.08% trainable parameters, RoAd 4 already achieves 96.9% of the accuracy of LoRA with 4.61% trainable parameters. By combining RoAd 1 with LoRA, we achieve the same performance as LoRA with only 1/4 of

its trainable parameters. This demonstrates RoAd's excellent scalability when combined with LoRA.

4.2 Efficiency results for batching

We commence by highlighting the significance of weight merging for PEFT. Among the approaches discussed in Section §4.1, only LoRA [14], DoRA [27], BOFT [30], OFT [44], BitFit [62], (IA) 3 [25], and our proposed RoAd enable the integration of trainable parameters with pretrained parameters without incurring additional inference overhead. As an illustration, we consider LoRA both with and without weight merging to underscore this process's importance. Notably, the implementation of LoRA with merged weights effectively reverts to the original LLM. To assess throughput, we configure the system with a batch size of 1, generate 2048 tokens, and apply the LoRA modules across all linear layers. Figure 4 (Left) clearly illustrates that the unmerged LoRA exhibits a significantly smaller throughput compared to the merged LoRA. Additionally, it is evident that the throughput of the unmerged LoRA demonstrates only a weak correlation with the rank size, primarily due to the fact that the additional overhead is largely attributed to communication instead of computation.

Furthermore, to evaluate the throughput of batching, we establish a default batch size of 8, generate 2048 tokens, and set the LoRA rank to 8. Each request within the batch is heterogeneous, necessitating eight distinct sets of trainable parameters by default. We only compare to LoRA here, because other baselines have either a weaker performance on downstream tasks (BOFT, OFT, BitFit and (IA) 3 ) or a smaller throughput than LoRA for batching (DoRA). As shown in Figure 4 (Middle and Right), RoAd significantly outperforms LoRA with variations in either the number of generated tokens or the number of heterogeneous requests. With an increasing number of distinct requests, the gap between LoRA and RoAd becomes even larger, which shows RoAd's unique advantage in efficient serving

4.3 Qualitative results for composability

In our investigation of RoAd's ability to handle compositional tasks, we primarily engage in multilingual experiments similar to those conducted by Wu et al. [61]. We use two training datasets: a new version of HellaSwag [63] 7 , which comprises 1K samples with prompts in English and completions in German, and a 1K-sample subset of the Ultrafeedback [7] dataset, which focuses on instruction following tasks in English. Contrary to the above experiments that adapt the outputs of the linear layer, here we instead adapt the representations from the 16 th block of LLaMA-7B, treating RoAd as a DII method. Specifically, we only adapt/intervene the representation of the final token in the

LoReFT response for prompt: Do you know the conference NeurIPS?

German subspace : geht auf dem Bildschirm vorbei und zeigt die Konferenz. (Translation: passes by on the screen and shows the conference.) ×

Instruction following subspace : NeurIPS (Neural Information Processing Systems) is an annual conference on machine learning and computational neuroscience ... ✓

Combined subspaces : Es gibt eine Konferenz namens NeurIPS, die in der Vergangenheit stattgefunden hat. (Translation: There is a conference called NeurIPS that took place in the past.) ×

RoAd response for prompt: Do you know the conference NeurIPS?

German subspace : Ich weiß nicht, ob ich es kenne. (Translation: I don't know if I know it.) × Instruction following subspace : Yes, I do! It's a great conference for machine learning researchers ✓

and practitioners to share their work with each other.

Combined subspaces : NeurIPS ist ein internationales Forschungskongress, der sich mit dem Thema künstliche Intelligenz beschäftigt. (Translation: NeurIPS is an international research congress that deals with the topic of artificial intelligence.) ✓

prompt using RoAd 1 . We train the upper half of R , i.e. { R i } d 2 / 4 i =1 , to handle the German completions in HellaSwag, and another half to complete the English sentences in Ultrafeedback. Both tasks are simultaneously trained but utilize distinct subspaces of R . We train the model over five epochs with a learning rate of 5 e -3 and a batch size of 8. 8

As in Figure 5, both LoReFT and RoAd are unable to perform completions with the German subspace. This limitation is anticipated due to two primary reasons: (1) LLaMA-7B predominantly relies on pretraining from English datasets, and doesn't have a cross-lingual answering ability without explicitly prompting. (2) The HellaSwag dataset is relatively small, containing only 1K samples with limited comprehensive coverage. Despite these constraints, the German subspace effectively prompts the model to produce sentences in German. Additionally, both methods achieve accurate completions in the other half of the subspaces, attributed to LLaMA-7B's extensive knowledge base in English. When these two subspaces are combined, RoAd successfully leverages their strengths, facilitating accurate sentence completions in German, while LoReFT doesn't catch the purpose of the prompt. We offer more examples, including negative examples, in Figure D.1, D.2 and D.3.

5 Conclusion

Initially, our research examines how finetuning modifies the representation of pretrained LLMs, finding that angular adjustments are more significant than changes in magnitude scale. Leveraging this insight, we propose a PEFT method, RoAd, which primarily utilizes a 2D rotational adjustment to the representation. Despite its simplicity, RoAd exhibits several distinct advantages: (1) It is exceptionally efficient in terms of parameters, consistently delivering superior performance on downstream tasks with the fewest trainable parameters compared to other PEFT methods; (2) RoAd efficiently supports batch processing, achieving twice the throughput of LoRA; (3) When incorporated within an intervention framework, RoAd demonstrates remarkable composability.

Due to page limit, we discuss the limitations and broader impacts in Section §A and §B, respectively.

References

A. Abdelfattah, A. Haidar, S. Tomov, and J. J. Dongarra. Performance, design, and autotuning of batched GEMM for gpus. In J. M. Kunkel, P. Balaji, and J. J. Dongarra, editors, High Performance Computing - 31st International Conference, ISC High Performance 2016, Frankfurt, Germany, June 19-23, 2016, Proceedings , volume 9697 of Lecture Notes in Computer Science , pages 21-38. Springer, 2016. doi: 10.1007/978-3-319-41321-1\_2. URL https://doi.org/10.1007/978-3-319-41321-1_2 .

D. Biderman, J. G. Ortiz, J. Portes, M. Paul, P. Greengard, C. Jennings, D. King, S. Havens, V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham. Lora learns less and forgets less, 2024.

Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 7432-7439. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05.6239. URL https://doi.org/10.1609/aaai.v34i05.6239 .

C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 29242936. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1300. URL https://doi.org/10.18653/v1/n19-1300 .

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR , abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457 .

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. CoRR , abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168 .

G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. CoRR , abs/2310.01377, 2023. doi: 10.48550/ARXIV.2310.01377. URL https://doi.org/10.48550/arXiv.2310.01377 .

J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-1423. URL https://doi.org/ 10.18653/v1/n19-1423 .

Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.

N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. CoRR , abs/2209.10652, 2022. doi: 10.48550/ ARXIV.2209.10652. URL https://doi.org/10.48550/arXiv.2209.10652 .

A. Geiger, Z. Wu, C. Potts, T. Icard, and N. D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In F. Locatello and V. Didelez, editors, Causal Learning and Reasoning, 1-3 April 2024, Los Angeles, California, USA , volume 236 of Proceedings of Machine Learning Research , pages 160-187. PMLR, 2024. URL https://proceedings.mlr.press/v236/geiger24a.html .

J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=0RDcd5Axok .

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 2019. URL http://proceedings.mlr.press/v97/houlsby19a.html .

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

Z. Hu, L. Wang, Y. Lan, W. Xu, E. Lim, L. Bing, X. Xu, S. Poria, and R. K. Lee. Llmadapters: An adapter family for parameter-efficient fine-tuning of large language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 5254-5276. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023. EMNLP-MAIN.319. URL https://doi.org/10.18653/v1/2023.emnlp-main.319 .

C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. CoRR , abs/2307.13269, 2023. doi: 10.48550/ ARXIV.2307.13269. URL https://doi.org/10.48550/arXiv.2307.13269 .

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b. CoRR , abs/2310.06825, 2023. doi: 10.48550/ARXIV.2310.06825. URL https://doi.org/10.48550/arXiv.2310.06825 .

R. Koncel-Kedziorski, S. Roy, A. Amini, N. Kushman, and H. Hajishirzi. MAWPS: A math word problem repository. In K. Knight, A. Nenkova, and O. Rambow, editors, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 1152-1157. The Association for Computational Linguistics, 2016. doi: 10.18653/V1/ N16-1136. URL https://doi.org/10.18653/v1/n16-1136 .

B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 3045-3059. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.EMNLP-MAIN.243. URL https://doi.org/10.18653/v1/2021.emnlp-main.243 .

M. Li, S. Gururangan, T. Dettmers, M. Lewis, T. Althoff, N. A. Smith, and L. Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models. CoRR , abs/2208.03306, 2022. doi: 10.48550/ARXIV.2208.03306. URL https://doi.org/10. 48550/arXiv.2208.03306 .

X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , pages 4582-4597. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021. ACL-LONG.353. URL https://doi.org/10.18653/v1/2021.acl-long.353 .

B. Liao, Y. Meng, and C. Monz. Parameter-efficient fine-tuning without introducing new latency. In A. Rogers, J. L. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , pages 4242-4260. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.ACL-LONG.233. URL https://doi.org/10.18653/v1/ 2023.acl-long.233 .

B. Liao, S. Tan, and C. Monz. Make pre-trained model reversible: From parameter to memory efficient fine-tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December

10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 3151e460c41ba67dc55412861184ef35-Abstract-Conference.html .

W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In R. Barzilay and M. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers , pages 158167. Association for Computational Linguistics, 2017. doi: 10.18653/V1/P17-1015. URL https://doi.org/10.18653/v1/P17-1015 .

H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 0cde695b83bd186c1fd456302888454c-Abstract-Conference.html .

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html .

S. Liu, C. Wang, H. Yin, P. Molchanov, Y. F. Wang, K. Cheng, and M. Chen. Dora: Weightdecomposed low-rank adaptation. CoRR , abs/2402.09353, 2024. doi: 10.48550/ARXIV.2402. 09353. URL https://doi.org/10.48550/arXiv.2402.09353 .

W. Liu, R. Lin, Z. Liu, L. Liu, Z. Yu, B. Dai, and L. Song. Learning towards minimum hyperspherical energy. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 6225-6236, 2018. URL https://proceedings.neurips. cc/paper/2018/hash/177540c7bcb8db31697b601642eac8d4-Abstract.html .

W. Liu, R. Lin, Z. Liu, J. M. Rehg, L. Paull, L. Xiong, L. Song, and A. Weller. Orthogonal over-parameterized training. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 7251-7260. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021.00717. URL https://openaccess.thecvf.com/content/CVPR2021/html/Liu_Orthogonal_ Over-Parameterized_Training_CVPR_2021_paper.html .

W. Liu, Z. Qiu, Y. Feng, Y. Xiu, Y. Xue, L. Yu, H. Feng, Z. Liu, J. Heo, S. Peng, Y. Wen, M. J. Black, A. Weller, and B. Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. CoRR , abs/2311.06243, 2023. doi: 10.48550/ARXIV.2311.06243. URL https://doi.org/10.48550/arXiv.2311.06243 .

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692, 2019. URL http://arxiv.org/abs/1907.11692 .

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 .

R. K. Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 1022-1035, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 081be9fdff07f3bc808f935906ef70c0-Abstract.html .

S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft , 2022.

J. L. McClelland, D. E. Rumelhart, P. R. Group, et al. Parallel distributed processing, volume 2: Explorations in the microstructure of cognition: Psychological and biological models , volume 2. MIT press, 1987.

T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 , pages 2381-2391. Association for Computational Linguistics, 2018. doi: 10.18653/V1/D18-1260. URL https: //doi.org/10.18653/v1/d18-1260 .

T. Mikolov, W. Yih, and G. Zweig. Linguistic regularities in continuous space word representations. In L. Vanderwende, H. D. III, and K. Kirchhoff, editors, Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA , pages 746-751. The Association for Computational Linguistics, 2013. URL https://aclanthology.org/N13-1090/ .

N. Nanda, A. Lee, and M. Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy, and H. Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2023, Singapore, December 7, 2023 , pages 16-30. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023. BLACKBOXNLP-1.2. URL https://doi.org/10.18653/v1/2023.blackboxnlp-1.2 .

OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303. 08774. URL https://doi.org/10.48550/arXiv.2303.08774 .

K. Park, Y. J. Choe, and V. Veitch. The linear representation hypothesis and the geometry of large language models. CoRR , abs/2311.03658, 2023. doi: 10.48550/ARXIV.2311.03658. URL https://doi.org/10.48550/arXiv.2311.03658 .

A. Patel, S. Bhattamishra, and N. Goyal. Are NLP models really able to solve simple math word problems? In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tür, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021 , pages 2080-2094. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.NAACL-MAIN.168. URL https://doi.org/10.18653/v1/2021.naacl-main.168 .

J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In P. Merlo, J. Tiedemann, and R. Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021 , pages 487-503. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.EACL-MAIN.39. URL https://doi.org/10.18653/v1/2021.eacl-main.39 .

Y. Qin, X. Wang, Y. Su, Y. Lin, N. Ding, Z. Liu, J. Li, L. Hou, P. Li, M. Sun, and J. Zhou. Exploring low-dimensional intrinsic task subspace via prompt tuning. CoRR , abs/2110.07867, 2021. URL https://arxiv.org/abs/2110.07867 .

Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng, Z. Liu, D. Zhang, A. Weller, and B. Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ faacb7a4827b4d51e201666b93ab5fa7-Abstract-Conference.html .

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020. URL http://jmlr.org/papers/v21/20-074.html .

D. E. Rumelhart, J. L. McClelland, P. R. Group, et al. Parallel distributed processing, volume 1: Explorations in the microstructure of cognition: Foundations . The MIT press, 1986.

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 8732-8740. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05.6399. URL https://doi.org/10.1609/aaai.v34i05.6399 .

M. Sap, H. Rashkin, D. Chen, R. L. Bras, and Y. Choi. Socialiqa: Commonsense reasoning about social interactions. CoRR , abs/1904.09728, 2019. URL http://arxiv.org/abs/1904. 09728 .

P. Smolensky. Neural and conceptual interpretation of pdp models. Parallel distributed processing: Explorations in the microstructure of cognition , 2:390-431, 1986.

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL , pages 1631-1642. ACL, 2013. URL https://aclanthology. org/D13-1170/ .

J. Su, M. H. M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024. doi: 10.1016/J.NEUCOM. 2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063 .

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023. doi: 10.48550/ARXIV. 2302.13971. URL https://doi.org/10.48550/arXiv.2302.13971 .

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and finetuned chat models. CoRR , abs/2307.09288, 2023. doi: 10.48550/ARXIV.2307.09288. URL https://doi.org/10.48550/arXiv.2307.09288 .

A. M. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization. CoRR , abs/2308.10248, 2023. doi: 10.48550/ARXIV.2308.10248. URL https://doi.org/10.48550/arXiv.2308.10248 .

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. URL https://proceedings.neurips. cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=rJ4km2R5t7 .

Y. Wen and S. Chaudhuri. Batched low-rank adaptation of foundation models. CoRR , abs/2312.05677, 2023. doi: 10.48550/ARXIV.2312.05677. URL https://doi.org/10. 48550/arXiv.2312.05677 .

Y. Wen and S. Chaudhuri. Batched low-rank adaptation of foundation models. CoRR , abs/2312.05677, 2023. doi: 10.48550/ARXIV.2312.05677. URL https://doi.org/10. 48550/arXiv.2312.05677 .

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, Oct. 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ 2020.emnlp-demos.6 .

M. Wu, W. Liu, X. Wang, T. Li, C. Lv, Z. Ling, J. Zhu, C. Zhang, X. Zheng, and X. Huang. Advancing parameter efficiency in fine-tuning via representation editing. CoRR , abs/2402.15179, 2024. doi: 10.48550/ARXIV.2402.15179. URL https://doi.org/10.48550/arXiv.2402. 15179 .

Z. Wu, A. Arora, Z. Wang, A. Geiger, D. Jurafsky, C. D. Manning, and C. Potts. Reft: Representation finetuning for language models. 2024. URL https://api.semanticscholar. org/CorpusID:268889731 .

E. B. Zaken, Y. Goldberg, and S. Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 1-9. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.ACL-SHORT.1. URL https://doi.org/10.18653/v1/2022.acl-short.1 .

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pages 4791-4800. Association for Computational Linguistics, 2019. doi: 10.18653/V1/P19-1472. URL https://doi.org/10. 18653/v1/p19-1472 .

J. Zhang, S. Chen, J. Liu, and J. He. Composing parameter-efficient modules with arithmetic operation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 299a08ee712d4752c890938da99a77c6-Abstract-Conference.html .

Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=lq62uWRJjiY .

M. Zhong, Y. Shen, S. Wang, Y. Lu, Y. Jiao, S. Ouyang, D. Yu, J. Han, and W. Chen. Multi-lora composition for image generation. CoRR , abs/2402.16843, 2024. doi: 10.48550/ARXIV.2402. 16843. URL https://doi.org/10.48550/arXiv.2402.16843 .

A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks. Representation engineering: A top-down approach to AI transparency. CoRR , abs/2310.01405, 2023. doi: 10.48550/ARXIV.2310.01405. URL https://doi.org/10.48550/arXiv.2310.01405 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We accurately make claims about our paper's contributions and scope in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include a section, Section A, to discuss the limitations of our work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work is not about theory.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include all implementation details either in the main pages or in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our work is based on public frameworks and data, which are shown correctly in the paper and easy for access.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All such details are either in the main pages, Section 4, or in the appendix, Section C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show the standard deviation of the main results in appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We show the GPU type and the memory in Section C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We strictly follow NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We include the broader impact in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We don't release any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We mainly used some publicly open framework, which is able to use for research purpose. We also have a correct citation for these frameworks, which direct interesting reader to the license details.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We don't offer new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We don't use crowdsourcing or human evaluation.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We don't include human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Limitations

We recognize that a primary limitation pertains to the scalability of RoAd. Currently, it is not feasible to indefinitely increase the number of trainable parameters with RoAd. Nevertheless, our experiments demonstrate that RoAd 4 already exhibits commendable performance. To scale the trainable parameters, we can combine RoAd with other PEFT methods, such as LoRA, which enhances the scaling behavior of these PEFTs, i.e. achieving similar results with less trainable parameters.

B Broader impacts

RoAd's primary advantage is its efficiency in adapting LLMs to specific tasks with minimal trainable parameters. This efficiency not only reduces computational resource needs but also makes advanced AI technologies more accessible to organizations with limited resources, potentially democratizing AI capabilities across smaller enterprises and educational institutions. By reducing the number of trainable parameters and the computational load, RoAd likely decreases the energy consumption associated with training and deploying LLMs. This could contribute to lowering the carbon footprint of AI research and deployment, aligning with greater environmental sustainability efforts. The ability to process multiple heterogeneous requests efficiently means that applications can provide personalized, context-specific responses more quickly. This enhances the user experience in real-time applications, such as digital assistants, automated service, and interactive educational platforms.

While RoAd improves interpretability in some aspects by integrating within frameworks like distributed interchange intervention [11], the overall complexity of the methods might still pose challenges in understanding and diagnosing the models' decisions. This could affect efforts to make AI more transparent and accountable, especially in critical applications like healthcare and law. Increasing the accessibility of powerful AI models through PEFT also raises concerns about misuse. More entities can harness these capabilities, potentially including those with malicious intents, such as creating sophisticated disinformation campaigns or automating cyber attacks.

C Experimental details

C.1 Natural language understanding (NLU)

Test set split. Previous works [14, 22, 31] report the best results on the development sets of the GLUE tasks, i.e. using the same set for both validation and test, which might cause overfitting. Instead, we follow the setting of Mahabadi et al. [33] and Wu et al. [60], splitting the whole development set into a validation set and a test set. The model with the best performance on the validation set is selected to perform on the test set. Specifically, for the task with a development set whose number of samples is larger than 2K, i.e. QNLI, QQP and MNLI, we randomly select 1K samples as the validation set and the rest as the test set. For the other tasks, we select half of the samples in the development set as the validation set and another half as the test set. Please refer to Table C.1 for more details.

Hyperparameter tuning. We mainly follow the hyperparameter search space of Liao et al. [22] and list them in Table C.2. Notably, we almost upscale the learning rate by 10 for RoAd, because RoAd prefers a larger learning rate than other PEFT methods, which is also observed from Liu et al. [25] and Wen and Chaudhuri [57] where their adapters also apply multiplication instead of addition. The

best hyperparameter settings for each task are listed in Table C.3. The training is conducted either in Float16 or BFloat16. For each task, we (1) run experiments in the search space with a random seed, (2) then select the best hyperparameter setting (best result on the held-out development set), (3) and conduct another two more random runs with the best setting, (4) finally report the mean and standard deviation of these three results. For low-resource tasks (RTE, MRPC, STS-B and CoLA), we suggest expanding the best hyperparameter setting as Table C.3 for better reproduction. We report the standard deviation of RoAd in Table C.4.

Baseline reproduction. To include more baselines, we apply (IA) 3 [25], OFT [44] and BOFT [30] on the GLUE benchmark with RoBERTa-base [31] as the backbone. We use the same search space as RoAd in Table C.2 for (IA) 3 since both RoAd and (IA) 3 prefer a large learning rate. For OFT w =2 [44] and BOFT m =2 w =2 [30], we use the best hyperparameter settings from Liu et al. [30]. In addition, we expand the search space of the learning rate with an interval of 2 at the same scale while keeping the other best hyperparameters the same, since GLUE tasks have large variances. For example, if the best learning rate from Liu et al. [30] is 5e-4, the learning rate search space is {3e-4, 5e-4, 7e-4}. If the best learning rate is 2e-4, the search space is {9e-5, 2e-4, 4e-4}. For OFT, we don't share any parameters and use BOFT m =1 w =2 (= OFT w =2 ), because such a setting offers better results.

C.2 Commonsense reasoning

Datasets. Please refer to Hu et al. [15] for more details about the data statistics and task templates.

Hyperparameters. From Table C.3, it becomes apparent that one of the advantages of RoAd is its uniform optimal hyperparameter configuration across various tasks. Furthermore, we believe that extensive tuning of hyperparameters for LLMs is impractical. Consequently, we restrict the search space for the learning rate to { 1 e -3 , 3 e -3 }, ultimately selecting 3 e -3 for all experiments conducted on LLaMA. Consistent with Table C.2, we employ AdamW [32] as the optimizer without weight decay, a warmup ratio of 10% and a linear scheduler. Following Wu et al. [61], we fix the number of epochs at six and the batch size at 32. These hyperparameters are detailed in Table C.5. The maximum sequence length is set to 512. And the training is conducted either in BFloat16. We evaluate each checkpoint saved at every epoch and report the optimal result. The standard deviation

from three random runs is presented in Table C.6. During inference, we use greedy decoding without sampling as our baselines [15, 27, 61].

Baseline reproduction. In Table 3, we replicate the results of two baselines, OFT [44] and (IA) 3 [25]. For OFT w =16 (=BOFT m =1 w =16 ), we adopt the identical training configuration used for the mathematical question-answering task as described in Liu et al. [30]. For (IA) 3 , we adapt every linear layer rather than limiting adaptation to only the first feed-forward layer, key projection layer and query projection layer, as this setting shows improved performance. Notably, (IA) 3 benefits from a higher learning rate as RoAd, prompting us to apply the same training parameters as those outlined in Table C.5.

C.3 Arithmetic reasoning

Datasets. Please refer to Hu et al. [15] for more details about the data statistics and the construction mechanism of Math10K.

Hyperparameters. We apply almost the same training recipe as the one for commonsense reasoning, except that we set the number of epochs as 12 by following Wu et al. [61]. The detailed parameters are summarized in Table C.5. The maximum sequence length is set to 512. And the training is conducted either in BFloat16. We evaluate each checkpoint saved at every epoch and report the optimal result. The standard deviation from three random runs is presented in Table C.7. During inference, we use greedy decoding without sampling as our baselines [15, 27, 61].

Baseline reproduction. In Table 4, we replicate the results of (IA) 3 [25]. Similar to commonsense reasoning, we apply the same training hyperparameters as Table C.5 for (IA) 3 .

D More results

D.1 Compare to OFT.

Table D.1 presents the finetuning specifics for RoAds, OFT [44], and BOFT [30]. In OFT, a critical hyperparameter is defined as n = d 1 w , meaning the number of blocks in R . Thus, configurations such as OFT n =2048 and OFT n =256 correspond approximately to OFT w =2 and OFT w =16 , respectively. Increasing n , or equivalently reducing w , leads to a higher count of blocks. While a smaller w may reduce the number of trainable parameters, it necessitates more frequent computations of matrix inversion, consequently elevating both GPU memory usage and training time. Moreover, while BOFT utilizes fewer trainable parameters than OFT and achieves comparable or superior outcomes, it demands significantly more GPU memory. This increase is attributable to the butterfly factorization, which requires extensive caching of intermediate activations.

RoAd can be viewed as a specific implementation of OFT w =2 , but it consumes considerably less GPU memory and shortens training time. This efficiency stems from the use of inherently orthogonal 2D rotation matrices in RoAd, which obviate the need for matrix inversion calculations.

D.2 Commonsense reasoning on LLaMA2 and LLaMA3

We also conduct experiments on LLaMA2-7B [53] and LLaMA3-8B in Table D.2. RoAds still outperform all baselines with the least number of trainable parameters.

D.3 More examples for composability

In Figure D.1, D.2 and D.3, we show more examples of composability. Overall, RoAd demonstrates a very good ability in composition, taking advantage of both subspaces.

0.00

0.25

0.50

0.75

1.00

1.25

Full Finetuning, Layer 1

LoRA, Layer 1

Full Finetuning, Layer 2

0.00

0.25

0.50

0.75

1.00

1.25

Full Finetuning, Layer 3

LoRA, Layer 3

Full Finetuning, Layer 4

0.00

0.25

0.50

0.75

1.00

1.25

Full Finetuning, Layer 5

LoRA, Layer 5

Full Finetuning, Layer 6

LoRA, Layer 6

0.00

0.25

0.50

0.75

1.00

1.25

Full Finetuning, Layer 7

LoRA, Layer 7

Full Finetuning, Layer 8

LoRA, Layer 8

0.00

0.25

0.50

0.75

1.00

1.25

Full Finetuning, Layer 9

LoRA, Layer 9

Full Finetuning, Layer 10

LoRA, Layer 10

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

1.25

Full Finetuning, Layer 11

0.0

0.5

1.0

LoRA, Layer 11

0.0

0.5

1.0

Full Finetuning, Layer 12

0.0

0.5

1.0

LoRA, Layer 12

D

M

LoReFT response for prompt: How to kill a linux process?

German subspace : Es gibt mehrere Möglichkeiten, ein Prozess zu beenden. × Instruction following subspace : To terminate a Linux process, use the kill command with the PID (process ID) of the target process as its argument. The kill command can be ... ✓ Combined subspaces : Es gibt mehrere Möglichkeiten, ein Prozess zu beenden. ×

RoAd 1 response for prompt: How to kill a linux process?

German subspace : Kill a Linux process with the command 'kill -9 <pid>' where '<pid>' is the process ID of the process you want to kill. (Explanation: This is not desired because of the response in English.) ×

Instruction following subspace : To kill a Linux process, you can use the following command: kill -9 <process_id> ... ✓

Combined subspaces : Mit der Kommandozeile ps -ef | grep | cut -d ' ' -f 1 | xargs kill -9 will ich den Prozess mit dem PID töten. ✓

LoReFT response for prompt: How to learn English?

German subspace : Es gibt viele Möglichkeiten, Englisch zu lernen.

×

Instruction following subspace : Hello, I'm here to help you with your question about learning English! There are many ways to learn English, and it depends on what your goals are. If you want to improve your speaking skills ... ✓

Combined subspaces : Es gibt viele Möglichkeiten, Englisch zu lernen.

RoAd 1 response for prompt: How to learn English?

×

×

Instruction following subspace : I think you should study hard and practice speaking with native speakers. ✓

Combined subspaces : Jetzt lesen Sie einfach mal eine Geschichte in englischer Sprache und üben Sie die Sprache mit Freunden oder Familie. ✓

LoReFT response for prompt: How can I pass an exam?

German subspace

: , wenn du die Prüfung bestanden hast, wird sie dir gegeben. ×

×

✓

Instruction following subspace : Hello, thank you for your question! To pass an exam, it's important to prepare well in advance and study regularly. Here are some tips on how to do this effectively: ... ✓ Combined subspaces : Das ist ein sehr guter Ratgeber für die Prüfung. ×

RoAd 1 response for prompt: How can I pass an exam?

×

Instruction following subspace : You need to study hard and take the test.

✓

Combined subspaces : Mit einem Blick ins Buch oder auf die Webseite des Lehrers ist es möglich, den richtigen Weg zu finden. ✓

LoReFT response for prompt: What is the process to borrow a book from the national library?

German subspace : Es gibt eine Reihe von Büchern, die man ausleihen kann. × Instruction following subspace : The process of borrowing a book from the National Library involves ✓

several steps, including: ... Combined subspaces : Es gibt eine Reihe von Schritten, die man durchläuft, um ein Buch aus dem Nationalbibliothek zu mieten. ×

RoAd 1 response for prompt: What is the process to borrow a book from the national library?

German subspace : Wirft die Buchstabenfolge "national library" in Google und wähle den ersten Link auf. Klicke auf "Borrowing books". Klicke auf die Schaltfläche "How do I borrow a book?" und folge den Anweisungen. ✓

Instruction following subspace : The process to borrow a physical book from the National Library of Australia (NLA) involves three steps: 1) visiting the NLA, 2) searching for the book ... ✓

Combined subspaces : Nachdem Sie sich im Nationalbibliothek-Bibliothekssystem eingeloggt haben, können sie die Bücher ausleihen. ✓

LoReFT response for prompt: Should I learn Python or C++?

German subspace : , um, ich weiß nicht, ob es sich lohnt, Python zu lernen oder C++. × Instruction following subspace : As you know, Python and C++ are two popular programming languages used for various purposes. Both have their own advantages and disadvantages ... ✓ Combined subspaces : , um, ich weiß nicht, ob es sich lohnt, Python zu lernen oder nicht.) ×

RoAd 1 response for prompt: Should I learn Python or C++?

German subspace : Learn Python because it's easier to use and more fun than C++, but you should also learn C++ because it's useful for programming games, apps, and websites. (Explanation: This is not desired because of the response in English.) ×

Instruction following subspace : I think you should learn Python because it's easier to use and more fun than C++, but if you want to make games, then you should probably learn C++ instead. ✓

Combined subspaces : Mitglied, wie Sie wissen, dass es viele Programmiersprachen gibt, die sich unterscheiden in der Art und Weise, wie sie Code geschrieben werden. Die beiden Sprachen, die ich empfehle, sind Python und C++, aber ich kann nicht sagen, welche ist besser. not sure

LoReFT response for prompt: How to write a good paper?

German subspace : Es gibt viele Tipps, wie man einen guten Aufsatz schreibt. × Instruction following subspace : As a student, writing a good paper can be challenging and timeconsuming. However, there are several steps you can take to ensure your paper is well-written ... ✓ Combined subspaces : Es gibt viele Tipps, wie man einen guten Aufsatz schreibt. ×

RoAd 1 response for prompt: How to write a good paper?

German subspace : schreibe einen guten Aufsatz

Instruction following subspace It should also have a strong thesis statement, which clearly states your argument or purpose ... ✓ Combined subspaces : Mitbringt, dass das Papier einen guten Titel hat und die Autoren sind auf dem neuesten Stand der Dinge. ✓

× : A good paper should be well-written, with clear and concise language.

LoReFT response for prompt: Which app do you recommend for taxi?

German subspace : , um Taxi-App gibt es viele und sie sind alle gut. × Instruction following subspace : Hello, I recommend Uber or Lyft as they are reliable and safe options for getting around town. ✓

Combined subspaces : Es gibt viele Apps, die Taxifahrer bieten. Ich würde empfehlen, dass Sie den Taxi-App von Uber verwenden. ✓

RoAd 1 response for prompt: Which app do you recommend for taxi?

German subspace : Would like to use Uber, because it's convenient and fast. (Explanation: This is not desired because of the response in English.) ×

Instruction following subspace : I recommend Uber because it's cheaper and more convenient than other apps. ✓

Combined subspaces : Nachdem Sie sicher sind, dass die Uber-App ist nicht verfügbar in Ihrer Stadt, können Sie auch den Taxify-Dienst nutzen. ✓

LoReFT response for prompt: How can I find a part-time job? I'm a college student and poor.

German subspace : Sie können sich mit dem Jobsuchdienst von Facebook verbinden, um eine Anzeige zu posten. ✓

Instruction following subspace : Hello, I'm happy to help you with your question! There are several ways to find part-time jobs as a college student. Here are some options: ... ✓

Combined subspaces : Du kannst auf Jobbörsen nachstellen, die sich auf Teilzeitjobs spezialisiert haben. ✓

RoAd 1 response for prompt: How can I find a part-time job? I'm a college student and poor.

German subspace : Suche ich nach einem Teilzeitjob. Ich bin Student und arbeitslos. × Instruction following subspace : You should look for a part-time position in your field of study or one that you are interested in learning about. You may also want to consider working at a local restaurant, retail store, or other business. ✓

Combined subspaces : Siehe dir nach, ob es in der Nähe einen Jobcenter gibt.

✓

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a 2D rotation adaptation method called RoAd for efficiently fine-tuning large language models (LLMs). RoAd achieves parameter-efficient fine-tuning by rotating representations. Experimental results demonstrate that RoAd performs excellently across multiple benchmarks, reducing the number of training parameters and computational overhead.\n\nSTRENGTHS:\n1 The method proposed in this paper is simple but efficient.\n\n2 This method performs well on small-scale language models, achieving better performance with fewer or comparable parameters.\n\nWEAKNESSES:\n1 The paper claims in the abstract that one of the main scenarios it addresses is multitasking. However, the authors mainly illustrate this through qualitative experiments in section 4.3, which seems unconvincing. It is suggested that the authors refer to ATTEMPT [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts] and compare it in the same scenario to fully demonstrate RoAd's advantages in multitask learning.\n\n2 There is some confusion about the logical flow of the paper, especially in chapter 2. It seems that sections 2.2 and 2.3 are not closely related to section 2.1.\n\n3 The method's improvement on large-scale language models is relatively limited. As shown in Table 3, RoAd2/4 performs the same as LoReFT when using more parameters. This needs to be explained.\n\n4 It is suggested that the authors add a main diagram to describe the method.\n\n5 Figure 3 Middle shows that as the generated tokens increase, RoAd's throughput decreases rapidly, while LoRA does not show this trend. Will RoAd's throughput be lower than LoRA's when generating longer texts? Furthermore, what are the respective parameter amounts for LoRA and RoAd here?\n\n6 This paper seems to have similarities with SSF [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning], as both methods fine-tune by adjusting output representations.\n\nQUESTIONS:\nsee Weaknesses.\n\nLIMITATIONS:\nIn the Limitations section, the authors mentioned scalability and training parameters. Since the primary comparison in this paper is with LoRA, could the authors reduce LoRA's parameter count to the same level as the method proposed in this paper and then conduct the relevant comparative experiments?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper introduces a novel method for parameter-efficient finetuning, RoAd. By employing a straightforward 2D rotation to adapt LLMs, this paper addresses the challenges of existing parameter-efficient finetuning method for LLMs. Experiments on downstream tasks demonstrate the effectiveness of the proposed method.

STRENGTHS:
1.	This paper proposes a novel method for parameter-efficient finetuning which efficiently
adapts LLMs using a minimal number of trainable parameters.
2.	The method enhances both batching efficiency and composability.
3.	Comprehensive experiments on the GLUE benchmark, eight commonsense reasoning tasks and four arithmetic reasoning tasks are conducted to show the efficacy of the method.

WEAKNESSES:
1.	In the results of Table 2,  RoAd shows different performance on base and large model, what could be the reason? Why the RoAd(fc1) of large model with less parameters shows better average accuracy? Why the full FT setting shows even lower accuracy?
2.	If the proposed RoAd maintains the same quantity of parameters of existing method like LoRA, could the accuracy be further improved?
3.	How about the RoAd2 and RoAd4 on the held-out GLUE development set?

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parameter-efficient finetuning method named RoAd, to address two challenges of current methods. The first challenge is the efficient deployment of LLMs equipped with multiple task- or user-specific adapters. The second one is the interpretability of LLMs. RoAd employs a straightforward 2D rotation to adapt LLMs. Experiment results consistently show that RoAd surpasses other PEFT methods\n\nSTRENGTHS:\n- This paper motivates well on the two challenges of current PEFT methods.\n\n- RoAd achieves impressive results, surpasses other PEFT methods in various tasks.\n\n- The authors perform the insightful pilot study and make interesting observations on the key factor influencing the adaptation of LLMs.\n\nWEAKNESSES:\n- Evaluation of efficiency results for batching could be improved. The proposal of RoAd is well motivated by the overhead of batch matrix multiplication in current methods [1, 54] (Page 2, line 36). However, the authors only compare with LoRA in the evaluation of throughput of batching. It would be better if the authors can compare with [54] in this evaluation.\n\n- Novelty and advantage over OFT [41] need clarification. As RoAd can be considered as a specialized case of OFT with w = 2 (Page 3, line 91), it is important to clarify the technical novelty of RoAd over OFT. In the current form of the paper, the reader may consider RoAd as a special case of OFT without much technical novelty over OFT. \n\n- RoAd fails to consistently outperform other PEFT methods on arithmetic reasoning tasks. Specifically, the results of RoAd on LLaMA-7B is worse than LoRA and AdapterP. Such results make the statement \"consistently surpasses\" in Page 2, line 53, seem like a bit of an overclaim.\n\nQUESTIONS:\n- Page 7, line 247, \"Figure 4\" should be \"Table 4\".\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel parameter-efficient fine-tuning method called RoAD, aimed at addressing the challenges of efficient deployment of LLMs that require multiple adapters for distinct needs and enhancing the interpretability of LLMs. The motivation behind this approach stems from the observation that the fine-tuning process primarily affects the angular components of representations rather than their magnitude. Thus, RoAD introduces a 2D rotational approach to representations to achieve parameter-efficient adaptation. Experimental results demonstrate that RoAD achieves a superior accuracy-efficiency trade-off compared to baseline approaches.\n\nSTRENGTHS:\n- Developing effective approaches to adapt the LLM to downstream tasks with better scalability and performance is an important problem. The achieved performance of this paper looks promising to me. \n\n- The paper layout is clear and the writing is easy to understand. \n\n- The observation that during tuning, the directional change is much more significant than the magnitude change is an interesting observation and may further motivate some follow-up research.\n\nWEAKNESSES:\n- For the observation section, it would be beneficial to include results from more recent autoregressive language models, such as Llama2, to ensure that the observation is consistent and generalizable across more commonly used models.\n\n- Another concern regarding the observation section is that the metrics used for magnitude and angular change are not identical. As a result, the claim that angular change is larger than magnitude change may be significantly influenced by the chosen metric and scaling. The authors should further justify the validity of this observation when using different metrics.\n\n- The authors should consider further benchmarking the proposed method on generative tasks with instruction tuning and evaluate it on more challenging benchmarks, such as Alpaca-Eval (https://github.com/tatsu-lab/alpaca_eval) and MT-Bench (https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge).\n\nQUESTIONS:\n- Can the proposed method be applied to vision language models? Does the observation still stand for vision language models?\n\nLIMITATIONS:\nThe authors have mentioned the potential limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-25"}