3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability
Authors: Baohao Liao
Christof Monz
Venue: NeurIPS 2024 poster (2024)
Keywords: parameter-efficient finetuning
orthogonal finetuning
batching
interpretability
Primary area: deep_learning_architectures
TL;DR: Our proposed method, RoAd, is extremely parameter-efficient, batching-efficient and composable.

Abstract: Parameter-efficient finetuning (PEFT) methods effectively adapt large language models (LLMs) to diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges to PEFT beyond mere parameter efficiency. One notable challenge involves the efficient deployment of LLMs equipped with multiple task- or user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches to address different challenges. In this paper, we introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with <0.1% trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.

3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability
Baohao Liao 12+
Christof Monz'
Language Technology Lab, University of Amsterdam
eeBay Inc., Aachen, Germany
Code: https//gtthh..cmm/Baaaoaalaaao
Abstract
Parameter-effccien finetuning (PEFT) methods effectively adapt large language models (LLMs) diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges PEFT beyond mere parameter efficiency. One notable challenge involves the effi- cient deployment of LLMs equipped with multiple task- user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches address different challenges. In this paper, we introduce novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably sarammter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with 0.1% trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within framework of distributed interchange intervention, demonstrated via composition experiments.
GLUE
Commonsense
Arithmetic
52.5
88.75
Accuracy
88.50
80
LoReFT
DoRA
Adapter
Adapter
47.5
50.0
LoReFT
Adapter
Adapter
GPT3
LoReFT
LaRA Adapter
75
45.0
88.00
RED
42.5
70
40.0
Adapter"
Prefix
Prefix
Trainable parameters (%)
Figure 1: Performance of various PEFT methods the GLUE benchmark, eight commonsense reasoning tasks and four arithmetic reasoning tasks with RoBERTa-large or I.L.MA-3B,
1 Introduction
Large language models (L.LMs), trained on extensive web-scale datasets perform tasks such as predicting masked words 8 31 45
or anticipating the next word in a sentence [17 52 53
"Correspondence to b.liao@uva..l Please go to httts://arriirooggaaaz/440000010 for the newest version.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

demonstrate remarkable effectiveness across range of NLP applications. For tasks where the data distribution diverges from that of the pretraining corpus, finetuning emerges as an effective way to tailor an L.M to specific requirements. Leveraging the capabilities of L.L.ss, recent studies [13] [4]
22 25 27
60) 62 65 demonstrate that training only subset of an LLM'ss parameters can yield performance on par with full finetuning. This approach, termed parameter-efficien finetuning (PEFT), provides two primary advantages: (1) It reduces the storage requirements for trained parameters, as necessitates preserving only universal LLM alongside minimal set of task-specific parameters; (2) decreases GPU memory consumption during finetuning, owing to the reduction in optimizer state sizes which correlate directly with the number of trainable parameters.
With the evolution of PEFT, concerns extend beyond mere parameter efficiency. PEFT encounters a variety of challenges brought forth by diverse applications. A significant challenge is the efficient deployment of personalized or task task-specific L.LMs 25 57. These applications frequently require distinct sets of trained parameters for different tasks or users. When multiple users submit requests simultaneously. it becomes crucial to process these requests collectively in a single batch. Given that each request may require unique set of parameters, using batch matrix multiplication can efficiently handle these requests by leveraging GPU parallelism. However, the batch matrix multiplication still incurs considerable overhead [57, necessitating the exploration of more efficient methods.
Another challenge is the interpretability of LLMs that contain a billion-scale of parameters, making it difficult to explore their mechanism. PEFT provides an alternative approach by constraining the number of trainable parameters, thereby aiding interpretability. Recent advancements PEFT methods, particularly those focusing on representation editing (44] 60) 67], can be incorporated within an intervention framework [T] This integration enhances their capability for interpretabilityy offering more manageable means of dissecting the operational intricacies of LLMs.
In this paper, we introduce novel technique termed 2D rotary adaptation (RoAd) which efficiently adapts LLMs using a minimal number of trainable parameters. Furthermore, RoAd enhances both batching efficiency and composability. Our initial investigation reveals that finetuning primarily alters the angular components of the representations in pretrained LLMs. rather than their magnitudes (Section $3.1). Based on this observation, we employ strategy of rotating certain subspaces within the representations to emulate finetuning effects. Specifically, we implement a 2D rotational approach on the representations and develop three distinct variants of RoAd (Section $3.2).
To assess the efficacy of RoAd, we perform comprehensive evaluations on the GLUE benchmark [56], eight commonsense reasoning tasks and four arithmetic reasoning tasks, utilizing RoBERTa (Section $4.1). The results consistently show that RoAd surpasses other PEFT methods while maintaining a significantly reduced scale of trainable parameters (< 0.1%), as Additionally, RoAd employs element-wise rather than matrix multiplication, which notably improves throughput when serving heterogeneous requests within the same batch, achieving twice the throughput of LoRA (14) (Section $4.2). Furthermore, RoAd can be seamlessly integrated within an intervention framework [.]. thereby enhancing model interpretability. We illustrate this through a composition experiment, demonstrating RoAd's capacity to merge weights trained for different tasks and display a new capability (Section $4.3).
31 and LLaMA [52] 53
depicted in Figure |
2 Background
In this section, we outline the challenges tackled in this work. illustrating the constraints of existing methods and objectives that drive the development of the proposed method, RoAd.
2.1 Parameter-efficient finetuning (PEFT)
Existing PEFT techniques can be categorized into three groups: adapter-based, prompt-based, and latency-less methods. Adapter-based methods [12
42 incorporate adapters either in parallel with or sequentially to the existing Transformer [55] modules. This incorporation necessitates
13
modifications to the LLM architecture, consequently adding extra latency during inference. Prompt- based methods [19 21 43 enhance the input by appending new trainable tokens, which lengthens the sequence and thereby increases the computational overhead during inference. Latency-less methods, such as LoRA 114 and its variants
22 27 65), apply low-rank matrices to adapt the pretrained
weights. These matrices can be scamlessly integrated into the existing weight matrices following
2

---

finetuning, thus preserving the original LLM architecture. Specifically, LoRA adapts an LLM as W - + AW. where 0 Rd is the pretrained weight and A AW- BA with B R x A E RRxde, r < d1 and r < d2. Our proposed method, RoAd, aligns with the latency-less category and integrates effortlessly into the existing linear layer without imposing additional overhead during inference. Moreover, RoAd demonstrates exceptional parameter efficiency. The quantity of its trainable parameters is equivalent to that of LoRA module with a rank 0.5.
Orthogonal finetuning, Drawing on the concept of hyperspherical energy and its role in characteriz- ing generalization [28 29]. OFT 44 introduces orthogonal finetuning, an effective PEFT method for finetuning text-to-image diffusion models. Specifically, OFT implements an orthogonal matrix R E Rd xd1 to the pretrained weight W so the input æ E Rd
linear layer after adaptation be- comes x ) x. R is serfmmetr-efficientt because it is a block diook-l matrix with n blocks as R = diag( R, Ri..... R.,) where each block R3 E IR wxw has a dimension w = di/n. To maintain orthogonality, R. is derived using Cayley parameterization: R. = (I+ Q.) with Qi E _ being a skew-symmetric matrix (Q; = Q2 ). In sum, {Q.)==1 serve as the trainable parameters and R is constructed from them with Cayley parameterization. Subsequent advancement, BOFT 30 leverages butterfly factorization to further refine OFT's s parameter efficiency. However, both OFT and BOFT, due to their reliance on matrix inversions the Cayley parameterization and increased storage of intermediate activations, necessitate additional GPU memory and increase training duration compared to other PEFT approaches. Conversely, RoAd, which may be considered as a specialized case of OFT with w 2, offers faster and more memory-effiiien solution by inherently maintaining orthogonality without requiring further parameterization.
)
2.2 Batching
Batching in this context refers processing multiple heterogeneous requests, each requiring different adapter for inference. This scenario commonly arises when serving personalized or task-specific LLMs. Specifically,, we consider a setup where distinct adapters instead of shared adapter are finetuned for various tasks to achieve optimal performance. During inference, each request in a batch pertains different task and necessitates a unique adapter.
Consider that we have finetuned distinct LoRA modules for b tasks, denoted as (A,, B B-1 For a batch of b requests represented as X E Rx where / is the maximum sequence length across the requests, each request requires a different LoRA module. To exploit the parallel processing capabilities of GPUs, the output Z of a linear layer can be computed as follows: First, the output from the pretrained layer is computed 7 torch.mm(X, Wo). Subsequently, the intermediate output from the first low-rank matrix, B [xxd (a concatenation of {B: (B:) is obtained Roxxxda (a (omm(Z)) Zg- A). Finally, these outputs are summed produce Z - + +Z It is noteworthy that batch matrix multiplication (BMM), as implemented in
as 2 = Lorch. brnnn ()) The output from the second low-rank matrix, concatenation of { {A,} ) follows as Z' = torch.bmm(
often introduces substantial overhead [1. reducing throughput and increasing latency, which adversely impacts user experience in time-sensitive applications.
In contrast, prompt-basee based methods circumvent the use of BMM by appending trainable tokens to each request, simplifying the computational process. However, prompt-based methods with long prompt tokens are difficult to optimize, which degrades performance compared to other PEFTs [4] [5]. (IA)3 (25) proposes adapting LLM by multiplying the output from linear layer with trainable vector, involving only dlement-wise multiplication for efficient batching. A recent development, FLORA (58), builds (IA)² by employing two low-rank matrices while maintaining element- wise operations. Although our proposed method, RoAd, requires BMM, its sparse structure allows a reformulation of BMM and results in overhead equivalent to element-wise multiplication.
2.3 Intervention and composability
38 40 have provided support for the linear representation hypothesis [35 46 49 that concepts are represented within linear subspaces of neural network representations. To examine if a concept is captured within linear subspace of representation, Geiger et al. [
Numerous studies 10
37
Adapter here means the trained parameters since LoRA's architecture is also similar adapter.
3

---

Full Finetuning
LoRA
W'x W'x, random Z-coo/V 2- [[M]] [X
0,10
0.08
0.100
Score
0.06
0.075
0.050
20
0.025
0.02
0.00
0.25
0.50
0.6
0.9
STS-B
CoLA
AD
Tasks
Figure 2: Pilot study for the pretrained and finetuned representations. Left & Middle: The change in magnitude and angle of representations between pretrained and finetuned LLM using full finetuning or LoRA. Right: The disentanglement experiment of magnitude and angle of pretrained representation.
suggests employing distributed interchange intervention (DII) defined as:
DII(b, R) b R (Rs Rb)
b denotes the hidden representation generated at row and column ic when the model processes an input, while 8 represents the corresponding representation when the model processes different input. The matrix R E Brxd, consisting of orthogonal rows, serves as low-rank projection matrix where d, is the dimension of the representation and is the subspace dimension under intervention. Equation I illustrates the application of DII to b using counterfactual source representation
Drawing inspiration from this established framework, recent study, LoReFT [61], introduces method for finetuning specific positions of the representations to adapt LLM. This study further demonstrates that several prior approaches of representation editing 54
60
67
can be effectively
integrated within this framework. Interestingly, the application of RoAd to representations can also be conceptualized as DII, offering interpretability potential. To demonstrate one aspect of interpretability for RoAd, we primarily conduct a qualitative experiment focused task composition. This experiment involves combining the weights of models trained on distinct tasks to showcase
capability for multitasking learning without the need for additional adaptation [16] 20
61
64
66
3 Method
In this section, we first perform two pilot studies to ascertain the key factor inftuencing the adaptation of L.LMs. Following this, we present proposed method, the 2D rotary adaptation (RoAd). which serves as an effective PEFT method addressing the various challenges outlined in Section $2
3.1 Pilot study
Study 1:
Variations in magnitude and angular displacement. Assume x Rd1 represen- tations of the same token from a pretrained and finetuned LLM, respectively. We define the relative change in magnitude as AM æ - ||="||2| / /||=0
and compute the angular displacement as AD cas x²") E 1]. larger AM and smaller AD indicate more significant changes in magnitude and angular displacement, respectively. Our study involves: (1) finetuning RoBERTa-boe base 3I on the SST-2 task (50) using either full finetuning or LoRA: (2) extracting representations o and x from the output of the second-last Transformer block for the [CLS] token across all samples in the development set, followed by computing AM and A As depicted in Figure (Left and Middle), there is a more pronounced change in AD than in AM for both full finetuning and LoR
Study 2: Disentanglement of magnitude and angle. To ascertain whether angular or magnitude adjustments are more critical for finetuning, we implement disentanglement study. This involves freezing RoBERTa-base and appending two-layer classifier on top of it. The first layer of this
We adopt notation systems from Wu et al. [6
Pleasse refer to Figure B.I for all layers. 'There are two other interesting observations: (1) An increase in magnitude change correlates with larger angular displacement; (2) Compared to LoRA, full finetuning has bigger change in magnitude and angle (for all layers, Figure B.I which is in line with recent finding that LoRA learns less and forgets less [2]

---

classifier incorporates a weight matrix W E R4 xd, Under standard operations, the output from this layer is computed as W x. To distinctly evaluate the impacts of magnitude and angle, we modify the output to retain only the magnitude component as = |W_,ll2 ||>="|2 l12. solely the angular component as = cos W i,, x²) (2, is the ith element of z). The modified classifier was then finetuned on four GLUE tasks with different metrics detailed in Table Additionally, a weak baseline employing randomly initialized RoBERTa-base is included. As shown in Figure 2 (Right), angular information is paramount in finetuning, whereas reliance solely on magnitude information even leads to inferior results compared to the random backbone.
Both studies indicate that angular information is more crucial than magnitude information for adapting pretrained LLM to downstream task. However, rotating the entire dN dimensions of the representation for finetuning incurs substantial computational costs. These costs are primarily reflected in a large number of trainable parameters, necessitating a dense matrix R E IR xd1, and in the requirement to maintain its orthogonality. Could we only rotate a subspace of the representation and design a R that is always orthogonal without any parameterization as OFT ! The first idea that comes to our mind is 2D rotation which only rotates two dimensions at a time and inherently maintains orthogonality.
3.2 2D rotary adaptation
Suppose that 8o E Rd x is the pretrained weight of a linear layer, R is the input of token to this linear layer, R E Rd is the rotation matrix, the adapted output from the linear layer is x = Rh R(WOT æ²). The rotation matrix R is defined follows:
cos 0g
-sin 0g
1/ +
a ahi
R- diag R2, R2, .... R2gg22 with R.
sin 0g
cos 0;
(2) The trainable parameters are denoted as {01) This 2D rotary
dz/2
Fg
8x
2d-1
Zg
adaptation involves rotating pairs of adjacent dimensions of h. specif- ically dimensions 2i 1 and 2i, using the rotation matrix R. The The rotation matrix R is characterized by its parameter efficiency. which attributed to its sparse structure and the parameter sharing within each block R. Additionally, R
Figure 3: Overview of RoAd,
can be integrated directly into the existing pretrained weights, forming W = =WRR' which does not incur additional computational costs during inference. This design closely mirrors RoPE (51) with the notable difference that in our RoAd, 0; is trainable and R; does not incorporate positional information. The overview of RoAd is shown in Figure 3
Relaxation to orthogonality. Referring to Figure (Right), while reliance predominantly on angular information substantially outperforms reliance magnitude information, it remains less effective than using both angular and magnitude information for the tasks of MRPC, STS-B, and CoLA. Furthermore, both fully- and LoRA-finctuned LLMs exhibit slight adaptations magnitude, as depicted in Figure 2 (Left and Middle). Consequently, we modify R, by incorporating to regulate the magnitude. We define a general R, as follows:
[0s,11 cos 0,,1
12 sin 0,,12
R:
04,21 sin 01,21
04,22 cos 0,,22
We develop three variants of RoAd by altering the configuration of shared parameters as outlined in [ RoAd, introduces minimal change to Equation (2) by incorporating a scaling factor og RoAd, already shows impressive results for most tasks in Section 1 For some Knowledge-intensive tasks, we observe that RoAdg and RoAd, obtain better results with more trainable parameters. To preserve the starting point of LLMs [23], we always initialize a, = and 0, =
Batching. In practice, we don't need to save R as a sparse matrix and do matrix multiplication. Taking RoAd, as an example in Equation (4). we only save two vectors: R! and R². Then = Rh = R & + R² & h, where h is a rearranged version of h and ( denotes wise multiplication. This reformulation not only simplifies the representation of R but also enhances the efficiency of batching in RoAd, relying solely on element- wise multiplications rather than BMM.
'The index in this work starts from instead of O.
5

---

Table 1: summarization of three RoAd variants.
RoAd;
0,
#Trainable
1
01,11
02,
0,,22 a;
01 = 0,,12
0,,21 = 02,22
0.
d2
2
06,11
05,12
05,21 05,22
0,
0,,12
8,,21
0,,2
2d
0,,11
# 0,,12 # 03221 # 03,22
0,11 # 0,,12
0.22 # 0,,22
4dg
Rh
@ R²
01 cos 01
b1
og sin 01 ou sin or sin Pz
h22
ba
ar coe B2
ha
h4
0g Og
h4
or
h
(4)
/d cos 0dz/2 Odg/2 0dg/2
hag-
/dg/2 022/2 Ad2/ sin 02//2
haz hdg 1
hig
Composability. RoAd can be incorporated into the DII framework ((h) Rh h R(h R' h), with Rs in Equation orthogonality of R. it is important to note that the rows of R remain orthogonal to each other within non-adjacent segments of the same block, R. This offers possibility for composability. We can finetune some rows on one task and other orthogonal rows on another task. Since they are orthogonal to each other, these two tasks should minimally affect each other, and the combination of these rows after finetuning could bring new multitasking learning ability.
I being set to h. Although degree of relaxation is introduced to the
RoAd can be considered as a special case of OFT 44 with w = 2. However, it is much more parameter- and memory-efficient and faster. Please refer to Section $D.1 for a detailed discussion.
Experiments
In this section, we begin by implementing RoAd to finetune various LLMs across three benchmarks. Subsequently, we illustrate its efficiency batching processes and demonstrate its composability. Unless otherwise noted, RoAd is applied to all linear layers within the LLMs. All of our experiments conducted on A100 80GB GPU with the frameworks, Transformers 59 and PEFT 34
4.1 Results downstream tasks
Natural language understanding (NLU). We evaluate the effectiveness of RoAd on the GLUB benchmark (56) for its ability of NLU with RoBERTa [31] as the backbone. Unlike many previous works [4 23 31 65] that employ the GLUE development sets for both validation and testing, here we partition the development set into distinct validation and test subsets to mitigate the risk of overfitting. For comprehensive information regarding the split of the development set, the search space of hyperparameters, the optimal hyperparameter configurations, and other details crucial for reproducibility, please see Section C.1
As shown in Table 2 RoAd, outperforms other PEFT methods with < 0.1% trainable parameters for both sizes of RoBERTa on average, being the only PEFT method that matches or outperforms full finetuning. These results show that 2D rotation (with a few scaling) can efficiently adapt LLM.
Commonsense reasoning, In assessing the capacity of LLaMA (52) for commonsense reasoning, we focus on eight representative tasks: BoolQ 41, PIQA 3. SIQA 48 HellaSwag [63], WinoGrande
47). ARC-e, ARC-c [5), and OBQA 36 The setting here contrasts with the NLU experiments where each task involves finetuning separate L.M. Instead, we adopt unified strategy by finetuning single LLM across all tasks as delineated in Hu et al. [1]] Such a setting is designed to mitigate overfitting and aligns more closely with real-world applications. Specifically, sets from these eight tasks are reformulated according to predefined template, so all tasks can be trained or evaluated in generative way. For all finetuning experiments on LLaMA, we follow recipe in Table C.5 without extensive searching. Please see Section C.2 for more training details.
training and test

---

Table Results on the held-out GLUE development set with RoBERTa as the backbone. We report matched accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B -B and accuracy for other tasks. The best and second-best results are bold and underlined, respectively, being the same for other tables. The percentage of trainable parameters is calculated without considering the classifier head. RoAd, (fc1) means that we only insert the RoAd, module to the first feed-forward layer, to match the #Params. of RED and LoReFT. Results of methods denoted by and are from Wu et al. [60] and Wu et al. [61]. respectively, Otherwise, average results from three
random runs reported. Refer to Table C.4 for the standard deviation.
Model
Method
#Params.
MRPC
STS-B
CoLA
SST-2
QQP
MNLI
Avg.
FT"
100.00%
78.3
87.9
90.6
62.4
94.4
92.5
91.7
87.3
85.6
Adapter"
0.32%
76.5
88.4
90.5
60.9
93.3
92.5
90.5
87.0
85.0
LoRA"
88.7
93.9
0.24%
77.7
88.8
58.5
93.0
92.0
90.2
87.1
84.7
BOFT
0.16%
71.4
87.5
89.6
55.3
92.5
91.4
89.4
85.3
82.8
0FT
87.6
92.8
83.9
82.3
BitFil"
0.08%
69.8
88.0
89.5
$4.0
94.0
91.0
87.3
84.7
82.3
(IA)*
0.04%
75.3
87.1
90.0
60.4
94.0
91.8
89.2
85.8
84.2
RED"
0.02%
78.0
89.2
93.9
90.7
87.2
83.9
84.3
LoReFT
0.02%
79.0
89.2
90.0
60.4
93.4
91.2
87.4
83.1
84.2
RoAd;
0.07%
78.9
89.2
90.5
64.4
93.9
91.9
89.6
86.3
85.6
(oAd)((el)
0.03%
79.1
90.2
90.2
60.9
94.6
91.6
88.7
85.4
85.1
Full FT"
100.00%
85.8
91.7
92.6
68.2
96.0
93.8
91.5
88.8
88.6
Adapter*
0.25%
85.3
90.5
91.5
65.4
95.2
94.6
91.4
90.1
88.0
LoRA"
0.23%
86.3
89.8
91.7
65.5
96.0
94.7
90.7
90.1
88.1
large
PNN.
96.1
87.7
RED"
0.01%
86.2
90.3
91.3
68.1
96.0
93.5
88.8
89.5
88.0
LoReFT
0.01%
87.5
90.1
91.6
68.0
96.2
94.1
88.5
89.22
88.2
RoAd;
0.06%
89.2
91.0
91.7
66.1 1
96.3
94,4
91.0
89.7
88.7
RoAd,(Tel)
91.5
96.1
89.6
88.8
As shown in Table RoAds still perform the best across various PEFT methods for both LLaMA-7B and LLaMA-13B on average. The strong baseline to RoAd is a recent representation finetuning method, LoReFT [61]. 80.2 79.2 and 83.3 83.0 for RoAd, for LLaMA-7B and LLaMA-13B, respectively. With a slightly increasing number of trainable parameters from RoAd, to RoAdz or RoAd,, RoAd matches or outperforms LoReFT. The same story is also told for another two versions of LLaMA, i.e. LLaMA2 [33 and LLaMA3, in Table D.2
Arithmetic reasoning. To assess the arithmetic reasoning ability of LLMs, we evaluate the finetuned L.Mss on the test sets of four tasks: AQuA (24), GSM8K (6) MAWPS [8] and SVAMP 41). Similar to the commonsense reasoning tasks, finetune single LLM for all four arithmetic reasoning tasks. The training dataset is MathlOK [15] which is constructed from the training sets of GSM8K, MAWPS, MAWPS-single and AQuA. The training recipe is similar the used for commonsense reasoning as shown Table C.5
Please see Section C.33 for more training details.
Different from the results of NLU and commonsense reasoning tasks, RoAd doesn't always perform the best on the arithmetic reasoning tasks, as shown in Figure 4 For the smaller-size LLM, LLaMA- 7B, RoAd is significantly better than other PEFT methods with < 0.1% trainable parameters, but worse than LoRA and Adapter with more than 10x trainable parameters. However, for the larger- size LLM,, LLLMA- 13B, all RoAd variants are better than other PEFT methods, which shows its scalability and potentially implies even better results for larger LLMs.
Observed from the above-mentioned results, for enhanced performance on downstream tasks and if a marginal increase in the stor- age capacity for trained parameters is accept- able, RoAd, is the preferable option. Con- versely, the objective is to investigate how the model adjusts in terms of angle and mag- nitude, RoAd, recommended. Notably, all
Table 5: Score on AlpacaEval2.00 with LLaMA2-7B.
Method
#Params.
Finetuning Data
Win Rate (8)
LoRA
0.83%
10K cleaned Alpaca
61.55
LoReFT RoAd:
0.03%
10K cleaned Alpuca
60.21
cleaned Alpaca
LoReFT RoAd,
0.03%
UltraFeeddacc 77]
61.68
0.02%
Ultraeetthack
62.60
variants of RoAd incur the same computational overhead for batching.
7

---

Table Accuracy of LLaMA on eight commonssnsse reasoming tasks. Results of methods denoted by and are from [3]. 61 and [27]. respectively. Otherwise, average results from three random
runs are reported. Refer to Table C.6 for the standard deviation, Refer Table D.2 for LLaMA2&3.
Model
Method
#Paras.
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
GPT3.5
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
Adapter
3.54%
78.8
69.8
78.9
57.3
72.3
A
0.99%
76.3
75.7
57.1
70.8
DoRA"
0.84%
68.5
82.9
79.6
84,8
80.8
81.4
65.8
81.0
78.1
LaRA"
77.4
74.7
OFT
0.14%
69.0
82.0
78.5
90.9
78.9
83.0
68.2
76.4
78.4
7B
Prefix"
0.04%
64.3
76.8
73.9
42.1
72.1
72.9
54.0
60.6
64.6
LaReFT
0.03%
69.3
84.4
80.3
93.1
84.2
83.2
68.2
78.9
80.2
(IA)
0.02%
67.8
81.7
78.1
89.9
81.1
80.5
65.4
77.8
77.8
RoAd4
0.08%
70.6
83.2
79.0
92.3
81.8
84.2
70.6
80.0
80.2
RoAd;
0.04%
70.3
82.6
79.2
92.0
81.8
84.8
68. 8
82.2
80.2
RoAd,
79.0
80.3
79.2
Adaptar
2.89%
72.5
84.9
79.8
92.1
84.7
84.2
71.2
82.4
81.5
Adupter*
0.80%
71.8
83.0
79.2
88.1
82.4
82.5
67.3
81.8
79.5
DoRA
0.68%
72.4
81.5
92.4
84.2
84.2
69.6
82.8
81.5
LoRA"
0.67%
72.1
83.5
80.5
90.5
83.7
82.8
68.3
82.4
80.5
Prefix"
72.1
68.4
LoReFT
0.03%
72.1
86.3
81.8
95.1
15.2
86.2
73.7
84.2
83.3
RoAd,
0.07%
73.2
85.5
82.4
94.5
86.3
86.8
74.6
86.0
83.7
RoAd;
0.03%
73.3
86.4
82.0
94,4
86.
87.4
74.1
87.0
83.8
RoAd,
0.02%
81.2
84.4
73.7
83.0
Table 4: Accuracy of LLaMA on four arithmetic reasoning tasks. Results of methods denoted by and are from 15] and 61. respectively. Refer to C.7 for the standard deviation.
C.7
Model
Method
#Params.
AQuA
GSM8K
MAWPS
SVAMP
Avg.
Adapter
3.54%
18.1
35.3
82.4
49.6
46.4
AdapterS
0.99%
15.0
33.3
77.7
52.3
44,6
LoRA"
0.83%
18.9
37.5
79.0
52.1
46.9
Prefix"
0.04%
14.2
24.4
63.4
38.1
35.0
7B
LoReFT
0.03%
21.4
26.0
76.2
46.8
42.6
(IA)
0.02%
19.7
28.8
76.9
48.5
43.5
RoAd,
0.08%
24.8
27.4
81.5
49.4
45.8
RoAd:
0.04%
26.8
RoAd,
0.02%
26.4
26.2
76.5
46.7
44.0
Adapter
Adapter
0.80%
22.0
44.0
78.6
50.8
48.9
LoRA"
0.67%
18.5
47.5
83.6
54.6
51.1
Prefix"
0.03%
13B
LoReFr
0.03%
23.6
38.1
82.4
54.2
49.6
RoAd,
0.07%
25.2
39.8
84.5
59,5
52.3
RoAd:
0.03%
26.0
40.6
84.0
58.3
52.2
RoAd,
0.02%
24.8
40.7
57.3
51.9
Instruction-follovin ability. We further benchmark RoAd using AlpacaEva12200 [9] We finetune LLaMA2-7B with two instruction-tuning datasets and evaluate the model using AlpacaEval2.0. This evaluation employs GPT-4 (39] to assess the responses generated by the finetuned model against those produced by Text-davinei-OUS We don't 't choose GPT-4 as the reference model, because GPT-4 is too powerful than LLaMA2-7B. The proof-of-concepl experiment with LoRA shows the win-rate < 5%. As shown in Table 5| RoAd, demonstrates superior performance compared to all baselines, while utilizing the least number of trainable parameters.
Multimodal ability. Lastly, we apply RoAd to the LLM backbone of LLaVA 26 Liu et 22 requires 4.61% % trainable parameters for LoRA on this task, while most tasks with LoRA in our paper need < 1%, showing that this task is knowledge-intense inteege-eenenieve. Therefore, we need scale RoAd's trainable parameters. For this purpose, we combine it with LoRA due to the limited number of 0 and o; R. The combination represented as z OT + (BA) )a,, where A and B are from LoRA. We adjust the LoRA rank to vary the number of trainable parameters. We combine RoAd,

---

LLaMA-78
LLaMA-13B
250
+5
merged
RoAd
unmerged
LoRA
40
120
150
35
RoAd
100
LoRA
100
20
50-
25
60
256
1024
2048
4096
12
8
16
LoRA rank
#generated tokens
Batch size #requests
Figure 4: Comparison of throughput between LoRA and RoAd. Left: The influence of weight
merging for LoRA. Middle: The influence of the number of generated tokens. Right: The influence of the number of heterogeneous requests in batch.
with LoRA, but not RoAd2 or RoAd4, as their primary design purpose to increase the number of trainable parameters.
As shown in Table 6 with only 0.08% trainable parameters, RoAd, already achieves 96.9% of the accu- racy of LoRA with 4.61 % trainable parameters. By combining RoAd, with LoRA, we achieve the same per- formance as LoRA with only 1/4 of its trainable parameters. This demonstrates RoAd's excellent scalability when combined with LoRA.
Table 6: Visual instruction tuning results on LLaVA1.5-7B.
Method
GQA
SQA
VQAT
Avg.
LoRA
4.61%
62.4
68.5
56.9
86.0
68.5
RoAd,
0.08%
66.9
66.4
RoAd, LORA
1.19%
62,5
68.2
57.4
85.8
68.5
4.2 Efficiency results for batching
We commence by highlighting the significance of weight merging for PEFT. Among the approaches
discussed in Section $4.1 only LoRA [14] DoRA (27], BOFT 30 OFT 44 BitFit 62). (IA)* (25], and our proposed RoAd enable the integration trainable parameters with pretrained parameters without incurring additional inference overhead. As an illustration, we consider LoRA both with and without weight merging to underscore this process's importance. Notably, the implementation of LoRA with merged weights effectively reverts to the original LLM. To assess throughput, we configure the system with a batch size of I, generate 2048 tokens, and apply the LoRA modules across all linear layers. Figure 4 (Left) clearly illustrates that the unmerged LoRA exhibits significantly smaller throughput compared to the merged LoRA. Additionally, the unmerged LoRA demonstrates only weak correlation with the rank size, primarily due the fact that the additional overhead is largely attributed to communication instead of computation.
evident that the throughput of
Furthermore, to evaluate the throughput of batching, we establish a default batch size of 8, generate 2048 tokens, and set the LoRA rank to 8. Each request within the batch heterogeneous, necessitating eight distinct sets of trainable parameters by default. We only compare to LoRA here, because other baselines have either weaker performance on downstream tasks (BOFT, OFT, BitFit and (IA)²) or smaller throughput than LoRA for batching (DoRA). As shown in Figure 4 (Middle and Right), RoAd significantly outperforms LoRA with variations either the number of generated tokens the number of heterogeneous requests. With an increasing number of distinct requests, the gap between LoRA and RoAd becomes even larger, which shows RoAd's unique advantage in efficient serving
4.3 Qualitative results for composability
In our investigation of RoAd's ability to handle compositional tasks, we primarily engage in multilin- gual experiments similar those conducted by Wu et [6]]. We use two training datasets: new version of fellaSwag [63 which comprises 1K samples with prompts in English and completions in German, and a 1K K-sallle subset of the Utrrfeedback 7 dataset, which focuses on instruction following tasks in English. Contrary to the above experiments that adapt the outputs of the linear layer, here we instead adapt the representations from the 16th block of LLaMA-7B, treating RoAd as DII method. Specifically, we only adapt/intervene the representation of the final token in the

---

LoReFT response for prompt: Do you know the conference NeurlPS?
German subspace: geht auf dem Bildschirm vorhei und zeigt die Konferenz. (Translation: passes by on the sereen and shows the conference.)
X
Instruction following subspace: NeurIPS (Neural Information Processing Systems) an annual conference on machine learning and computational neuroscience.
Combined subspaces: Es gibt eine Konferenz namens NeurlPS, dic in der Vergangenheit stattgefunden hat. (Translation: There is conference called NeurIPS that took place the past.)
X
RoAd response for prompt: Do you know the conference NeurlPS.
German subspace: Ich weiß nicht, ob ich es kenne. (Translation: I don't tt know if I know it.) Instruction following subspace: Yes. I do! It's great conference for machine learning researchers and practitioners to share their work with each other.
x
Combined subspaces: NeurlPS ist ein internationales Forschungskongres. der sich mit dem Thema künstliche Intelligenz beschåftigr. (Translation: NeurIPS is international research congress that deals with the topic of artificial intelligence.)
Figure 5: Qualitative comparison between RoAd and LoReFT for their composability. The prompt is always English for different subspaces. Refer Figure D.I D.2 and D.3 for more examples.
prompt using RoAdj. We train the upper half of R, i.c. {R, [R. to handle the German completions in HellaSwag, and another half to complete the English sentences in Ultrafeedback. Both tasks are simultancously trained but utilize distinct subspaces of R. We train the model over five epochs with a learning rate of 5e - 3 and batch size of 8
d2/4
As in Figure both LoReFT and RoAd are unable to perform completions with the German subspace. This limitation is anticipated due to two primary reasons: (1) LLaMA-7B predominantly relies on pretraining from English datasets, and doesn't have a cross-lingual answering ability without explicitly prompting. (2) The HellaSwag dataset is relatively small, containing only IK samples with limited comprehensive coverage. Despite these constraints, the German subspace effectively prompts the model to produce sentences in German. Additionally, both methods achieve accurate completions in the other half of the subspaces, attributed LLaMA-7B's extensive knowledge base in English. When these two subspaces are combined, RoAd successfully leverages their strengths, facilitating accurate sentence completions in German, while LoReFT doesn't catch the purpose of the prompt. We offer more examples, including negative examples, in Figure D.I D.2 and D.3
5 Conclusion
Initially, our research examines how finetuning modifies the representation of pretrained LLMs, finding that angular adjustments are more significant than changes in magnitude scale. Leveraging this insight, we propose PEFT method, RoAd, which primarily utilizes a 2D rotational adjustment to the representation. Despite its simplicity, RoAd exhibits several distinet advantages: (1) It is exceptionally efficient terms of parameters, consistently delivering superior performance on downstream tasks with the fewest trainable parameters compared to other PEFT methods; (2) RoAd efficiently supports batch processing, achieving twice the throughput of LoRA; (3) When incorporated within an intervention framework, RoAd demonstrates remarkable composability.
Due to page limit, we discuss the limitations and broader impacts in Section A and respectively.
Acknowledgements
We thank eBay Inc. for the computation support. This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project number 192.000
'The experiment is based on this notebook http:////wtggg.tcco/////aaa/e//e
10

---

References
[1] A. Abdelfattab, A. Haidar, S. Tomov, and J. J. Dongarra. Performance, design, and auto- tuning of batched GEMM for gpus. In J. M. Kunkel, P. Balaji, and J. J. Dongarra, editors, High Performance Computing - 31st International Conference, ISC High Performance 2016, Frankfurt, Germany, June 19-23, 2016, Proceedings, volume 9697 of Lecture Notes in Com- URL
puter Science, pages 21 31.38.180010 Springer, 2016. doi:
https://doi org/10..//.. 1007/978-3333 3000077978-3319---00
319- $19-41321-1_2
[2] D. Biderman, J. G. Ortiz, J. Portes, M. Paul, P. Greengard, C. Jennings, D. King, S. Havens,
V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham. Lora learns less and forgets less, 2024.
[3] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: reasoning about physical common- sense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York. NY, USA, February 7-12, 2020, pages 7432-7439. AAAI Press, 2020. doi: N0.1609/AAALV3311S. 323.9..639.. URL BtRp:://: org 10///10.100 10. 1609/aaai..... v34105. 6239
[4] C. Clark, K. Lee, M. Chang. T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Ex- ploring the surprising difficulty of natural yes/no questions. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT
2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers), pages 2924- 2936. Association for Computational Linguistics, 2019. doi:
URL
nttps://doi.co
18653/vv/n1133 1300
[5] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/ 1803005457
2018. URL httpp:/ara/xiii org/ abs 1803 05457
[6] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek,
J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 14168, 2021. URL petttss:/aaran
org/ 2210 14168
[7] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. CoRR, abs/2310.0137// 2023. doi: 00445500/ARXVV22100007777 URL httpps/////// org/10. 481500.4.5///// arviv. 231.. 003377
[8] Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume (Long and Short Papers). pages 4171 14886 Association for Computational Linguisties, 2019. doi: 10.18653/V1/N199 1423. URL https:///ot....... org/
10. www.33/5//11999 1tt///vv/n199/11 1423
[9] Y. Dubois, X. Li, R. Taori, Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
[10] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. CoRR, abs/2209. 10652, 2022. doi: 10.48550/ ARXIV.2209. wtt.22299.166552 10652. URL http:httt/://do http:://oooooo/1//// org/10. 4..1..8550/arrt arriv. 22.9. 10652
[11] A. Geiger, Z. Wu, C. Potts, T. Icard, and N. D. Goodman. Finding alignments between inter- pretable causal variables and distributed neural representations. In F. Locatello and V. Didelez, editors, Causal Learning and Reasoning, 1-3 April 2024, Los Angeles, California, USA, vol- ume 236 of Proceedings of Machine Learning Research, pages 160-187. PMLR, 2024. URL gew.greega...... 1ttmml
https://proceeedings. mlr http:/inwmmm.lp.mr.eeeeemmmmee/e
[12] He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL.
https://openrevier. bet.://ooorrmr.dddodoaaadaa.

---

[13] N. Houlsby, A. Giurgiu, S. Jastrzesski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In K. Chaudhuri Ma-
and R. Salakhutdinov, editors, Proceedings of the 36th International Conference chine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 2019. URL http:///proceeding. mlr, httpa///////oulssyyyyyy
[14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
URL https:t//pperreviiiiiee
[15] Z. Hu, L. Wang, Y. Lan, W. Xu, E. Lim, L. Bing, X. Xu, S. Poria, and R. K. Lee. Llm- adapters: An adapter family for parameter-effictee
fine-tuning of large language models. In H. Bouamor. J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 2254-5276 Association for Computational Linguistics, 2023. doi: (0.18663/V1/2023. EMMLP-MAIN..1I URL http::ttp////t ptattps://do..oo oog. 10. 18653/ 12202222222222 emnlp- m--maain... 319
[16] C. Huang, Q Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. CoRR, abs/2307.132269 2023, doi: 10.48550/ REXIV.2307.13269 URL httllhhts///sstt
org/10. 48540/
2307 13269
[17] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bres- sand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b. CORR, abs/2310.006225 2023. doi: URL https::s:://do org/ 10. 000..48550aaa
10.44550/AARXI..33100066
[18] R. Hemcel-Kedziorski S. Roy, A. Amini, N. Kushman, and H. Hajishirzi. MAWPS: A math word problem repository. In K. Knight, A. Nenkova, and O. Rambow, editors, NAACL HLT 2016. The 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 1 1152-1 1157. The Association for Computational Linguisties, 2016. doi: 10. 10.18653/V1/ N16- 113.. URL htttt:://dd/o http:/::/dooo.o//1//// 10. 18653/v1/n16. 1136)
[19] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-effiiient prompt tuning. In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045-3059. Association for Computational Linguistics, 2021. doi: 10. N0.18633/V1/2222 wttp:3//11222..EMMPPOOO WW...E https://doi.org: 0/0///10.886...1 000655/vv/222// em maw.. 243
URL
[20] M. Li. S. Gururangan, T. Dettmers, M. Lewis, T. Althoff, N. A. Smith, and L. Zettlemoyer. Branch-train--erger Embarrassingly parallel training of expert language models. CoRR. abs/2208. 033006, 2022. doi: 10.485550/ARXI..22088000
URL hnttps://oo. /ttps:://di.orr/10
48550/ard arviv. 2208 03306
[21] X. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In C. Zong, Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582-4597, Association for Computational Linguistics. 2021. doi: acl- long. 153
1/2021.
10. 100p/////////1 ACL-LONG.353. URL (athhttss::. //oii org/
[22] B. Liao, Y. Meng, and C. Monz. Parameter-efficient fine-tuning without introducing new latency. In A. Rogers, J. L.. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 4242-4260. Association for Computational Linguisties. 2023. doi:
URL hnttps://doi.
18653/v1/
2023. acl acl- 233
[23] B. Liao, S. Tan, and C. Monz. Make pre-trained model reversible: From parameter to memory efficient fine-tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December
12

---

10 - 16, 2023, 2023. URL http:: http://ppppppp.pp.popo/pppppaaep 3i506000cba6//5555111111181615 Abstracct www.eett-ooooffeeeeeccoo .htel
[24] W. Ling, D. Yogatama, C. Dyer, and P. Blunsom. Program induction by rationale generation: Learning solve and explain algebraic word problems. In R. Barzilay and M. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume I: Long Papers, pages 158- 167. Association for Computational Linguistics, 2017. doi: 10. 18153//11/117-1015 URL Sttps://dos org/10. 1ww518666vv// 18653/v1/P1171000/
[25] H. Liu, D. Tam, M. Muqeeth, Mohta, T. Huang, M. Bansal, and C. Raffel. Few-shot parameter- enrameter-efficient fine-tuning is better and cheaper than in in-context learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh, editors, Advances in Neu- rat Information Processing Systems 35: Annual Conference cessing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Decem- nips. cttpss.cc/ppp ber 9, 2022, 2022. URL http::/ Censtracct-Coneerence html
Neural Information Pro-
htt.cc/cppppco hatp/papee___a files wwwws//apeeeeeee///ee pptpp///22///as/l
[26] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Informa-
in Neural Information Processing Systems 36: Annual Conference tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL http://papers nips cc hpaperffiles www.es/apppee////eeaaeaaaaaaaeee paaprr2023/haas/ 6dcf2/7ea32ce328855 4f www.fff6ffeeddl 2aa
erence. html
[27] S. Liu, C. Wang, H. Yin, P. Molchanov, Y. Wang, K. Cheng, and M. Chen. Dora: Weight- decomposed low-rank adaptation. CoRR, htbp//2002.0935333 2024. doi: 10.48550/ARXIV.2402
09353. URL htttts:://o/t
org/ 10. 18550/arXiv 2402. 09353
[28] W. Liu, R. Lin, Z. Liu, L. Liu, Z. Yu, B. Dai, and L. Song. Learning towards minimum hyperspherical energy. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6225 6236, 2018. URL http:s///prcceeednnggg new.ro neurips cc/paper/ http:///000///11..00000000000100
[29] W. Liu, R. Lin, Z. Liu, J. M. Rehg, L. Paull, L. Xiong, L. Song, and A. Weller. Orthogonal perepaaraetteized training. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7251-7260. Computer Vision Foundation IEEE, 2021. doi: 10.1109CCVPR46437.2222 00717. URL Inttps://ppnaaccess thec com/ http://ceeene//P20000///////a/// Orthogogna__ Over-Parameterizeer htwpretteerrieeeeerraaaaaanee Training CVPR 2021_paperrrr.pp_e httm
[30] W. Liu, Z. Qiu, Y. Feng. Y. Xiu, Y. Xue, L. Yu. H. Feng. Z. Liu. J. Heo. S. Peng. Y. Wen, M. J. Black, A. Weller, and B. Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. CoRR, abs/2311 116244,1.06,,,... 2023. doi: 10448500ARXIV.3311.00222 URL 48550/ arXiv 2311 06243
https://doi.. orw//1.485500..../.....
[31] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11699
2019. htt/ httt::///rxiv... org/ absa 1907 11692
[32] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. URL /tllittsss:/ooereiiiiii http:ppwwwweeiieee.eeeeeeerrrnrc
[33] R. K. Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercom- plex adapter layers. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin,
P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 1022-1035 2021. URL
neurips.com
2021/hash/
08
[34] Mangrulkar, S. Gugger, I.. Debut, Y. Belkada, S. Paul, and B. Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://githu/ com commm/uuggiigaac//petttt 2022.
13

---

[35] J. L. McClelland, D. E. Rumelhart, P. R. Group. et al. Parallel distributed processing, volume 2: Explorations the microstructure of cognition: Psychological and biological models, volume 2. MIT press, 1987.
[36] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can suit of armor conduct electricity?
A new dataset for open book question answering. In E. Riloff, D. Chiang. J. Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2381-2391. Association for Computational Linguisties, 2018. doi: 10.18653/V1// 1/D18- 1260. URL https
7/doi.org/I0 L.8533/v1/111 1260
[37] T. Mikolov, W. Yih, and G. Zweig. Linguistic regularities in continuous space word rep- resentations. In L. Vanderwende, H. D. III, and K. Kirchhoff, editors, Human Language Technologies: Conference of the North American Chapter of the Association of Compu- tational Linguistics, Proceedings, June 9-14, 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, pages 746-751. The Association for Computational Linguistics, 2013. URL https://aclanthol.eg 1tt.ggg...og/111100////0/
[38] N. Nanda, A. Lee, and M. Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy, and H. Mohebbi, editors. Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Nenworks for NLP, BlackboxNLP@ EMNLP 2023, Singapore, December 7,
2023, pages 16-30. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023..
BLACKBOXNI.P 1.2. URL /ttp:/ttts:////i..rrg..
[39] OpenAI. GPT-4 technical report. CORR, abx/2333.087777 2023. doi: 10.48550/ARXIV.3000. 08774. URL http::://oo:...... org/ 10. 48550/ar51ii 23..
[40] K. Park, Y. J. Choe, and V. Veitch. The linear representation hypothesis and the geometry of large language models. CoRR, abs/231 1.03658, 2023. doi: 10.48550/ARXIV.231 1.06588. URL //ttp:
48550/ 48550/arXiv. 2311 2
[41] A. Patel, S. Bhattamishra, and N. Goyal. Are NLP models really able to solve simple math word problems? In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani- Tür, I. Belt- agy. S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-1 2021, pages 2080- 2094. Association for Computational Linguistics, 2021. doi:
URL htttt::t/oooo...oooo org/ 10. 11653/v1/22221 aaa.llmmmiiii... 168
[42] J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In P. Merlo, J. Tiedemann, and R. Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 487-503 Association for Computational Linguistics, 2021. doi: 10.18653/V 1,00,21EEAAL. EACL- MAIN. htLttt/:://doi org/ 10. 1 wac. maol--maai.... 39
[43] Y. Qin, Wang, Y. Su, Y. Lin, N. Ding, Z. Liu, Li, L. Hou, P. Li, M. Sun, and J. Zhou. Exploring low-dimensional intrinsic task subspace via prompt tuning. CoRR, 1000211 2021. HtRL/hhtps::/// //axxiv www.oogg////////eeeoeooooeeeee/e 2t10
[44] Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng Z. Liu, D. Zhang, A. Weller, and B. Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt. and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL http:://apprre wwtpp..cpppppp.aaapppppp fileerrffeeee/eeeeeeeeeeeeeeeeee http:///ppppp.eeeea///////////a/ /ptppp/ppp/////ssh/// html
[45] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res,, 21:140:1 14466677 2020.
http://iw...rrrppprr.reese wttp://aaaerr/////0007/// html
[46] D. E. Rumelharr, I.. McClelland, R. Group, et al. Parallel distributed processing, volume I: Explorations in the microstructure of cognition: Foundations. The MIT press, 1986.
14

---

[47] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Theood- Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press, 2020. doi: 10.1609/AAAI 134105.310666999 URL httph:t/p//// http://////...o/11...///// 1at.//00..09/aaaa.... v34105 6399
[48] M. Sap, H. Rashkin, D. Chen, R. L. Bras, and Y. Choi. Socialiqa: Commonsense reasoning about social interactions. CoRR, hes/1994,09728, 2019, UtRp abtpg///bb//999/// 09728
[49] P. Smolensky. Neural and conceptual interpretation of pdp models. Parallel distributed processing: Explorations in the microstructure of cognition, 2:390-431, 1986.
[50] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1631-1642. ACL, 2013. URL
1
[51] J. Su. M. H. M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputting, 568:127063, 2024. doi: 10. 10.10O/ 2023.127063. URL hRtps::::///oo org/10. 1016/j neucom. 2023. 127063
[52] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. CoRR, abs/23002.19771, 2023. doi: 10.48550/ARXIV. 2302.13971, URL httph:tttpsttoit org/10 wttp 0084545555aiii 2302. 13971
[53] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Es- iobu, J. Fernandes, J. Fu. W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Ko- renev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Sal- adi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, Edunov, and T. Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, 020/2337.00288 2023. doi: 10.48550/ARXIV.2337..00 https://doi... http:////ooo.o.//.... 48550/ 48550/arXiv 2307 09288
URL
[54] A. M. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization. CoRR, abs/2308. 10248, 2023. doi: 10.48550/ARXIV.22006 10248. URL https:://oi. org/10. 48550/arXiv. 2308. 10248
[55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In I. Guyon, U. von Luxburg. S. Bengio, H. M. Wallach, Fer- gus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, .ner
2017, Long Beach, CA, USA, pages 5998 6008, 2017. cc/ /c/paper/ http:////1///h//// wttaaaase/eeeee224eeeeeeeeee111 www.boo00331111111aaaaa wwwtaaat.ttttttttttaaaaa
[56] A. Wang, A. Singh. J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleams, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpss///openrevier wttt:/eeumiddrrrrueoee J4m2225557
[57] Y. Wen and S. Chaudhuri. Batched low-rank adaptation of foundation models. CoRR, abs/2312. 026677, 2023. doi: 48550/ arviv 23...
10.48550/ARXI1.2322.5555
URL https:///oi...... org/10
[58] Y. Wen and S. Chaudhuri.
Batched low-rank adaptation of foundation models. CoRR,
abs/2312 2.06777, 2023. doi:
10.48550/ARXX1223122.5577 URL. lttht::t/////t http:///oo.o////////////////////
48550/ arivv. 2312.05677
15

---

[59] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, Davison, S. Shleifer, von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Scao, S. Gugger, M. Drame, Q. Ihoest, and A. M. Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Oct. 2020. Association for Computational Linguistics. URL https: /www///www... aclweb. org/ anthology/
2020. emn emnlp-demos
[60] M. Wu, W. Liu, X. Wang, T. Li, C. Lv, Z. Ling, J. Zhu, C. Zhang, X. Zheng, and X. Huang. Advancing parameter efficiency in fine-tuning via represemtaaion editing. CoRR, abs/2402.151 2024. doi: 0448550/ARXXI.2442.555555
URL httpps////do org/ 10. 48550/ 2402
15179
[61] Z. Wu, A. Arora, Z. Wang, A. Geiger, D. Jurafsky, C. D. Manning, and Potts. Reft: Representation finetuning for language models. 2024, D.t semanticschollar org/Corpus
268889731
[62] E. B. Zaken, Y. Goldberg, and S. Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer- based masked magguage-modele In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 1 1-9. URL
Association for Computational Linguistics, 2022. doi: https://doi.. lttt::://ao.oor//0.... 18653/ 20655//o/2022.c11sshhtttt ac1-short..
[63] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can a machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Märquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791-4800. Association for Computational Linguistics, 2019. doi: 10.18653/V1/P19-1144 URL httphttps:/ddo org/10 18663/v1/p19-1477 1472
[64]
J. Zhang, S. Chen, J. Liu, and J. He. Composing parameter-efficient modules with arithmetic operation. A. Oh, T. Naumann, A. Globerson, Saenko, M. Hardt, and Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL /tttt:::/ppaaa nips cc/ hc/ppppppp.p.p sasperffles___ faaes_ffllepppeer///aeea paper/ Samper/2023/// /2033/aash/ 1
Abstract- Conference, erence. html
[65] Q. Zhang, M. Chen, Bukharin, P. He, Y. Cheng, W. Chen, and Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Fleventh International Conference on learning Representatiinns ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
URL. URL [Kl.ittpp://opeerevviee
[66] M. Zhong. Y. Shen, S. Wang. Y. Lu, Y. Jiao, S. Ouyang, D. Yu, J. Han, and W. Chen. Multi-lora composition for image generation. CoRR, abs/2402.16644 2024. doi: 10448550/ARXV.24402 16843. URL http::ttp//// http:/:///oo..o///////////// 48540/5555a5. aar.o//rxiiv... 2402 16843
[67] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dom- browski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks. Representation engineering: top-down approach to Al transparency. CoRR, abs/2310.014055, 2023. doi: 10448500AAXIV..310001400 URL httpp://o// org/ 485.48550/arra 2310.01405
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately make claims about our paper's contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include section, Section discuss the limitations of work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our work is not about theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include implementation details either in the main pages or in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: Our work based on public frameworks and data, which are shown correctly in the paper and easy for access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpooliiigguddde/ooooooo /ooeeee/e/ooeubbbiiieoo ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All such details are either in the main pages, Section or in the appendix, Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We show the standard deviation of the main results in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We show the GPU type and the memory in Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We strictly follow NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include the broader impact in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't release any data models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We mainly used some publicly open framework, which is able to use for research purpose. We also have correct citation for these frameworks, which direct interesting reader to the license details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't ton't offer assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't 't use crowdsourcing or human evaluation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We don't include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

A Limitations
We recognize that primary limitation pertains the scalability of RoAd. Currently, it is not feasible to indefinitely increase the number of trainable parameters with RoAd. Nevertheless, our experiments demonstrate that RoAd, already exhibits commendable performance. To scale the trainable parameters, we can combine RoAd with other PEFT methods, such as LoRA, which enhances the scaling behavior of these PEFTs, i.e. achieving similar results with less trainable parameters.
B Broader impacts
RoAd's primary advantage is its efficiency in adapting LLMs to specific tasks with minimal trainable parameters. This efficiency not only reduces computational resource needs but also makes advanced Al technologies more accessible organizations with limited resources, potentially democratizing Al capabilities across smaller enterprises and educational institutions. By reducing the number of trainable parameters and the computational load, RoAd likely decreases the energy consumption associated with training and deploying LLMs. This could contribute to lowering the carbon footprint of AI research and deployment, aligning with greater environmental sustainability efforts. The ability to process multiple heterogeneous requests efficiently means that applications can provide personalized, context-specifer responses more quickly. This enhances the user experience in real-time applications, such as digital assistants, automated service, and interactive educational platforms.
While RoAd improves interpretability in some aspects by integrating within frameworks like dis- tributed interchange intervention (IT), the overall complexity of the methods might still pose chal- lenges in understanding and diagnosing the models' decisions. This could affect efforts to make AI more transparent and accountable. especially in critical applications like healthcare and law. Increasing the accessibility of powerful AI models through PEFT also raises concerns about misuse. More entities can harness these capabilities, potentially including those with malicious intents, such as creating sophisticated disinformation campaigns or automating cyber attacks.
C Experimental details
C.1 Natural language understanding (NLU)
Table C.1: The data statistics and evaluation metrics of the GLUE benchmark. The valid and test sets are randomly split from the original development set. Following Wu al. [60], only the matched development set of MNLI is used. For runs with different seeds, the samples in the valid and test sets are also different.
Task
RTE
MRPC
STS-B
CoL.A
SST-2
QNLI
QQP
MNLI
#Train
2.6K
3.7K
5.7K
8.5K
67K
105K
364K
393K
#Valid
139
204
750
522
436
1K
IK
IK
#Test
138
204
750
521
436
4.5K
39K
8K
Metric
Acc.
Acc.
Pearson
Matthew
Acc.
Acc.
Acc.
Acc.
Test set split. Previous works 14 22
report the best results on the development sets the GLUE tasks, i.e. using the same set for both validation and test, which might cause overfitting. Instead, we follow the setting of Mahabadi et al. [33] and Wu et al. [60]. splitting the whole development set into validation set and test set. The model with the best performance on the validation set is selected to perform on the test set. Specifically, for the task with development set whose number of samples is larger than 2K, i.c. QNLI, QQP and MNLI, we randomly select IK samples as the validation set and the rest as the test set. For the other tasks, we select half of the samples in the development set as the validation set and another half as the test set. Please refer to Table C. for more details.
Hyperparameter tuning, We mainly follow the hyperparameter search space of Liao et al. [22 and list them in Table C.2, Notably, we almost upscale the learning rate by 10 for RoAd, because RoAd
prefers a larger learning rate than other PEFT methods, which is also observed from Liu et 25 and Wen and Chaudhuri [57] where their adapters also apply multiplication instead of addition. The
23

---

Table C.2: Hyperparameter search space for GLUE. For tasks with large number of training samples, we set the number of epochs as 10. Please refer to Table C.3 for the best task-specifi settings.
Hyperparameters
RTE, MRPC, STS-B, CoLA
SST-2, QNLI, QQP, MNLI
Optimizer
AdamW 0
Adam
Weight decay
LR
(le-3, 3e-3, Se-3, 7e-3]
[le-3, 3e-3, Se-3, 7e-3)
I.R scheduler
Linear 0.1
Linear 0.1
Warmup ratio
Epochs
{10, 20)
Batch size
[ [16, 32)
[16, 32|
Table C.3: Best hyperparameter settings for different GLUE tasks on RoBERTa. Notably, RoAd has a very consistent recipe for different tasks. The low-resource tasks (RTE, MRPC, STS-B, CoLA) and high-resource tasks (SST-2, QNLI, QQP, MNL.) show two obvious patterns for the hyperparameters. you have enough computation resources, we suggest alternating the batch size of low-resource tasks (RTE, MRPC, STS-B, CoLA) in ( [16, 32) and the number of epochs in ( [10, 20), since these
tasks have a relatively larger variance.
Model
Hyperparameter
RTE
MRPC
STS-B
CoLA
SST-2
QNLI
QQP
MNLI
LR
3e-3
3e-3
3e-3
3e-3
lc-3
le-3
lc-3
lc-3
base
Epochs Batch size
20
20
20
20
10
10
10
10
32
16
16
16
16
LR
3c-3
3c-3
le-3
Ic-3
lc-3
1c-3
lc-3
Ic-3
large
Epochs
20
20
20
20
10
10
10
10
Batch size
32
32
32
32
32
32
32
32
best hyperparameter settings for each task are listed in Table C33 The training is conducted either in Float 16 or BFloat 16. For each task, we (1) run experiments in the search space with a random seed, (2) then select the best hyperparameter setting (best result the held-out development set), (3) and conduct another two more random runs with the best setting, (4) finally report the mean and standard deviation of these three results. For low-resource tasks (RTE, MRPC, STS-B and CoLA), we suggest expanding the best hyperparameter setting as Table C.3 for better reproduction. We report the standard deviation of RoAd in Table C.4
Bascline reproduction. To include more baselines, we apply (IA)² 25 OFT and BOFT [30 on the GLUE benchmark with RoBERTa-bass [31] the backbone. We use the same search space as RoAd in Table 2 for (IA)² since both RoAd and (IA)² prefer large learning rate. For OFT, w=2 44 and BOFT (30] we use the best hyperparameter settings from Liu et al. [30]. In addition, we expand the search space of the learning rate with an interval of at the same scale while keeping the other best hyperparameters the same, since GLUE tasks have large variances. For example, if the best learning rate from Liu et al. (30) is 5e-4, the learning rate search space is (3e-4, 5e-4, 7e-4). If the best learning rate is 2e-4, the search space is (9e-5, 2e-4, 4e-4). For OFT, we don't share any parameters and use BOFT 2=2 (= OF)w=)), because such a setting offers better results.
C.2 Commonsense reasoning
Datasets. Please refer to Hu et al. 15 for more details about the data statistics and task templates.
Hyperparameters. From Table C.3 it becomes apparent that one of the advantages of RoAd is its uniform optimal hyperparameter configuration across various tasks. Furthermore, we believe that extensive tuning of hyperparameters for LLMs is impractical. Consequently, we restrict the search space for the learning rate to {1c 3, 3), ultimately selecting 3e 3 for all experiments conducted on LLaMA. Consistent with Table C.2 we employ Adam (32) as the optimizer without weight decay, a warmup ratio of 10% and a linear scheduler. Following Wu et al. [1], we fix number of epochs at six and the batch size at 32. These hyperparameters are detailed in Table C.5) The maximum sequence length is set to 512. And the training conducted either in BFloat16. We evaluate each checkpoint saved at every epoch and report the optimal result. The standard deviation
24

---

Table C.4: The standard deviation (subscript) of three random runs on the GLUE benchmark for
RoAd.
Model
Method
Paarams.
RTF
MRPC
STS-B
CoL.A
SST-2
QNLI
QQP
MNLI
RoAd;
0.07%
78.912
89.204
90.564
64.4us
93.9us
91.9ka
89.6g.1
86.302
85.6
(fe1)(fe1)
0.03%
79.12.
90.2
90.26a
60.91.2
94.6x3
91.6
88.7ca
85.4eu
85.1
large
RoAd)
0.06%
91.012
91.70.
96.304
(IAd)((el)
0.03%
88.71,
91.513
91.962
68.11,1
96.
94.5p1
90.2p.
89.6g.1
88.8
from three random runs is presented in Table C.6 During inference, we use greedy decoding without
sampling baselines [5 27 61.
Table C.5: Hyperparameters for commonsense and arithmetic reasoning without extensive tuning.
Hyperparameters Commonsense reasoning
Arithmetic reasoning
Optimizer
AdamW
Adam W
Weight decay
0
0
LR
3e-3
3e-3
LR scheduler
Linear
Linear
Warmup ratio
0.1
0.1
Epochs
6
12
Batch size
32
32
Table C.6: The standard deviation (subscript) of three random eight commonsense reasoning tasks for RoAd.
Model
Method
AParums.
BuolQ
PIQA
SIQA
HellaS.
WinGG.
ARC-
ARC-c
OBQA
Avg
RoAd,
0.0%%
20,6
S3.2g,
79.0kg.
92.3a2
81.8gs
84.2g,
70,6gs
80.0g
80.2g.1
LLaMA-BB
RoAdy
0.49
70.3p
82.6
79.20,4
92.0u1
81.80.7
84.86.3
68.803
82.2.0
80.300
RoAd;
20.4.
81.9g3
79.0k,2
91.4u1
80.303
84.0g.
79.3a1
RoAdg
73:2u5
83.5gs
94.5a.1
86.30.
86.8g,3
86.0p2
83:7pn
RoAdy
73.3n3
56.4g.
82.0g
94.461
86.1o3
87.4gu
74.1az
$7.0gs
83.8oz
RoAd:
0.02%
72:203
35.100
81.26
94.1n0
84.40.5
86.60.
73.7uz
86.61
85.0) 2
Baseline reproduction. In Table we replicate the results of two basclines, OFT 44 and (IA)² 25 For OFT (=BOFTm=)))) 16). we adopt the identical training configuration used for the mathematical question-answering task as described in Liu al. [30]. For (IA)², we adapt every linear layer rather than limiting adaptation to only the first feed-fon needffowwrrd layer, key projection layer and query projection layer, as this setting shows improved performance. Notably, (IA)² benefits from a higher learning rate RoAd, prompting to apply the same training parameters as those outlined in Table C.5
C.3 Arithmetic reasoning
Datasets. Please refer to Hu et al. 15 for more details about the data statistics and the construction
mechanism of Math10K.
Hyperparameters. We apply almost the same training recipe as the one for commonsense reasoning, except that we set the number of epochs as 12 by following Wu et al. [61]. The detailed parameters
are summarized in Table C.5 The maximum sequence length is set to 512. And the training is conducted either in BF. We evaluate each checkpoint saved at every epoch and report the optimal result. The standard deviation from three random runs is presented in Table C.7 During inference, we greedy decoding without sampling
baselines [15 27 61]
Baseline reproduction. In Table we replicate the results of (1A)
3
25
Similar to commonsense
reasoning, apply the same training hyperparameters Table C.5 for (IA)3.
25

---

Table C.7:
The standard deviation (subscript) of three random runs on four arithmetic reasoning tasks
for RoAd.
Model
Method
#Params.
AQuA
GSM8K
MAWPS
SVAMP
Avg.
RoAd, 0.08%
24.81.
27.4u9
81.5u9
49.4u.
45.8u.s
LLaMA-7B
RoAd2
0.04%
26.82s
29.9us
78.612
49.3u.
46.2a.6
RoAdi
0.02%
26.41.7
26.2n.2
76.516
46.71.
44.0b.2
RoAd, 0.07%
25.23.1
39.8p.5
84.5
59.5p.7
52.30.3
LLaMA-13B
RoAdz 0.03%
26.0n.
40.6
84.0
58.3u.
52.2n.4
RoAd: 0.02%
2..8
40.7n.9
84.9p9
57.3u.2
51.9u.2
Table D.1: Finetuning details of RoAds, OFT and BOFT on LLaMA-7B. The training setting here is:
batch size I, maximum sequence length = 512, number of iterations = 100, 1 A100 80GB GPU,
Method
#Params.
Peak GPU memory (GB)
Training time (s)
20048
0.09%
1249
2500
0.6%
37
191
BOFT
w=
0.3%
0OM
RoAd1
0.02%
25
RoAd2
0.04%
23
RoAdg
0.08%
23
24
D
More results
D.1 Compare to OFT.
T.ble the finetuning specifics for RoAds, OFT 44 and BOFT (30]. In OFT, critical w. meaning the number of blocks in R. Thus, configurations such as 2045 and xFT, correspond approximately to OFT,=2 and OFT,=16: respectively. Increasing n, equivalently reducing w, leads to a higher count of blocks. While smaller w may reduce the number trainable parameters, necessitates more frequent computations matrix inversion, consequently elevating both GPU memory usage and training time. Moreover, while BOFT utilizes fewer trainable parameters than OFT and achieves comparable or superior outcomes, demands significantly more GPU memory. This increase is attributable the butterfly factorization, which requires extensive caching of intermediate activations.
hyperparameter is defined
RoAd can be viewed as specific implementation of OFT but it consumes considerably less GPU memory and shortens training time. This efficiency stems from the use of inherently orthogonal 2D rotation matrices in RoAd, which obviate the need for matrix inversion calculations.
D.2
Commonsense reasoning on LLaMA2 and LLaMA3
We also conduct experiments on LLaMA2-7B 53 and LLaMA3-SB in Table D.2 RoAds still outperform all baselines with the least number of trainable parameters.
D.3 More examples for composability
In Figure D.I D.2 and D.3 show more examples of composability, Overall, RoAd demonstrates a very good ability in composition, taking advantage of both subspaces.
26

---

Table D.2: Accuracy of LLaMA2 53 and LLaMA3 on eight commonsense reasoning tasks, Results of methods denoted by are from Liu et al. (27).
Model
Method
Paramss
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARCac
OBQA
Arg.
76.0
82.6
LoRA"
0.83%
69.8
79.9
79,5
83.6
82.6
79.8
64.7
81.0
77.6
LLaMA2-7B
DoRA"
0.43%
72.0
83.1
79.9
89.1
83.0
84.5
71.0
81.2
80.5
RoAd,
0.08%
72.6
83.8
80.0
93.3
83.0
87.1
73.2
84.8
82.3
RoAdz
74.4
RuAd,
0.02%
71.7
83.0
80.1
93.0
81.2
86.0
72.3
82.2
81.2
DoRA
0.71%
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
LoRA"
0.70%
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
LLaMA3-8B
DoRA"
95.5
84.7
79.1
85.0
0.07%
81.1
RoAd2
0.03%:
74.6
89.8
81.6
96.0
86.9
82.1
86.8
86.3
RoAd;
0.02%
73.5
89.0
81.4
96.0
87.6
92.9
82.4
87.2
86.3
27

---

Full Finetuning, Layer
LoRA, Layer 1
Full Finetuning, Layer
LoRA, Layer
1.00
0.50
0.00
Finetuning, Layer
LoRA, Layer 3
Finetuning, Layer
LoRA, Layer
1.25
0.75
0.25
Full Finetuning, Layer 5
LoRA, Layer 5
Full Finetuning, Layer
LoRA, Layer
1.00
0.50
AM
0.00
Finetuning, Layer
LoRA, Layer
Finetuning, Layer
LoRA, Layer
1.25
0.75
0.25
0.00
Full Finetuning, Layer 9
LoRA, Layer 9
Full Finetuning, Layer 10
LoRA, Layer 10
1.00
0.25
Full Finetuning, Layer 11
LoRA, Layer 11
Full Finetuning,
LoRA, Layer 12
1.00
0.50
0.00
1.0
AD
The variation in magnitude and orientation of [CLS] representations from the pretrained and (fully or LoRA-) finetuned LLM across different layers reveals distinct patterns: (1) In shallower layers (< 8 for full finetuning and < 11 for LoRA). there is minimal change magnitude and angle. This minimal alteration occurs because these layers learn general knowledge that does not require significant modification for specific downstream tasks. (2) intermediate layers, excluding the final layer, rotations are more pronounced compared to changes in magnitude. (3) In the last layer, both magnitude and angle undergo substantial shifts, reflecting the stark differences between the objectives of pretraining and finetuning. The representation in this layer is extensively modified to align with the demands of the downstream task. Overall, compared LoRA, full finetuning exhibits greater alterations in both magnitude and angle, explaining that LoRA learns less and forgets less [2]
Figure B.1:
28

---

LoReFT response for prompt: How to kill linux process?
German subspace: gibt mehrere Möglichkeiten,
Prozess zu beenden.
Instruction following subspace: To terminate a Linux process, use the kill command with the PID (process ID) of the target process its argument. The kill command can be Combined subspaces: gibt mehrere Möglichkeiten, ein Prozess beenden.
V
RoAd; response for prompt: How to kill a linux process?
German subspace: Kill Linux process with the command 'kill <pid>> where '<pid>' is the process IT) of the process you want kill. (Explanation: This is desired because of the response English.)
Instruction following subspace: To kill a Linux process, you can use the following command: kill -9 <process_ __idd__...
V
Combined subspaces: Mit der Kommandozeile ps -ef grep cut "f xargs kill will ich den Prozess mit dem PID töten.
LoReFT response for prompt: How to learn English?
German subspace: gibt viele Möglichkeiten, Englisch zu lernen.
Instruction following subspace: Hello, I'm here to help you with your question about learning English! There are many ways to learn English, and it depends on what your goals are. If you want to improve your speaking skills Combined subspaces: Es gibt viele Möglichkeiten, Englisch zu lernen.
RoAd; response for prompt: How to learn English?
German subspace: Lernen Sie Englisch!
Instruction following subspace: think you should study hard and practice speaking with native
speakers.
Combined subspaces: Jetzt lesen Sie einfach mal eine Geschichte in englischer Sprache üben Sie die Sprache mit Freunden oder Familie.
LoReFT response for prompt: How pass exam?
German subspace: wenn du die Prüfung bestanden hast, wird sie dir gegeben. Instruction following subspace: Hello, thank you for your question! To pass exam. it's important prepare well in advance and study regularly. Here are some tips on how to do this effectively: Combined subspaces: Das ist ein sehr guter Ratgeber für die Prüfung.
X
RoAd: response for prompt: How can I pass an exam?
German subspace: Lernen Sie Deutsch! Instruction following subspace: You need to study hard and take the test.
Combined subspaces: Mit einem Blick ins Buch oder auf die Webseite des Lehrers ist möglich, den richtigen Weg zu finden.
Figure D.I: Qualitative comparison between RoAd and LoReFT for their composability. The prompt always in English for different subspaces.
29

---

LoReFT response prompt: What is process berrow a book from the national library?
German subspace: Es gibt eine Reihe Büchern, die ausleihen kann.
X
Instruction following subspace: The process borrowing book from National Library involves several steps. including:
Combined subspaces: Es gibt eine Reihe Schritten, die durchläuft, ein Buch dem Nationalbibliothck mieten.
RoAd; response for prompt: What is the process to borrow book from the national library?
German subspace: Wirft Buchstabenffolge "national library" in Google und wähle ersten Link auf. Klicke auf Borrowiin books". Klicke auf die Schaltfläche "How do borrow a book?" und folge den Anweisungen.
Instruction following subspace: The process borrow a physical book from National Library of Australia (NLA) involves three steps: 1) visiting the N. 2) searching for the book Combined subspaces: Nachdem Sie sich im Nationallibliothhek hetooaabbbbbboott---iiillttsssss eingeloggt haben. können sie die Bücchrr ausleihen.
V
LoReFT respouse prompt: Should learn Python or C++?
German subspace: um, ich weiß nicht, ob sich Johnt, Python lernen oder C++. x
Instruction following subspace: As you know, Python and C++ are two popular programming languages used for various purposes. Both have their own advantages and disadvantages. Combined subspaces: ich weiß nicht, ob sich lohnt, Python lernen oder nicht.) x
RoAd1 response for prompt: Should Iearn Python C++?
Learn Python because it's casier to use and more fun than C++, but you should also learn C++ because it's useful for programming games, apps, and websites. (Explanation: This is desired because of the response English.)
German subspace:
X Instruction following subspace: think you should learn Python because easier to use more fun than C++, but if you want to make games, then should probably learn C++ instead. Combined subspaces: Mitglied, wie Sie wissen, dass viele Programmiersprachen gibt, die sich unterscheiden Art und Weise, wie sie Code geschrieben werden. beiden Sprachen, die empfehle, sind Python und C++, aber ich kann nicht sagen, welche ist besser.
V
not sure
LoReFT response for prompt: How write good paper?
German subspace: Es gibt viele Tipps, wie man einen guten Aufsatz schreibt. Instruction following subspace: As student, writing a good paper can be challenging and time- consuming. However, are several steps you can take to ensure your paper well-written...
Combined subspaces: Es gibt viele Tipps, wie man einen guten Aufsatz schreibt.
RoAd, response for prompt: How to write a good paper?
German subspace: schreihe einen guten Aufsatz
Instruction following subspace: A good paper should well-written, with clear and concise language. It should also have strong thesis statement, which clearly states your argument purpose / Combined subspaces: Mithringt, dass das Papier einen guten Titel hat und die Autoren sind auf dem neuesten Stand der Dinge.
Figure D.2: Qualitative comparison between RoAd and LoReFT for their composability, The prompt always in English for different subspaces.
30

---

LoReFT response for prompt: Which app do you recommend for taxi?
German subspace: um Taxi-App gibt es viele und sie sind alle gut.
X
Instruction following subspace: Hello, recommend Uber Lyft as they are reliable and safe options for getting around town. Combined subspaces: Es gibt viele Apps, die Taxifahrer bicten. Ich würde empfehlen, dass Sie den Taxi-App von Uber verwenden.
RoAd, response for prompt: Which app do you recommend for taxi?
German subspace: Would like use Uber, because it's convenient and fast. (Explanation: This is not desired because of the response in English.)
Instruction following subspace: recommend Uber because cheaper and more convenient than
other apps.
Combined subspaces: Nachdem Sie sicher sind, dass die Uber-App ist nicht verfüghar in Ihrer Stadt, können Sie auch den Taxify-Dienst nutzen.
LoReFT response for prompt: How can find part- time job? I'm college student and poor.
German subspace: Sie können sich mit dem Jobsuchdienst von Facebook verbinden, um eine Anzeige zu posten.
Instruction following subspace: Hello, I'm happy to help you with your question! There are several ways to find part-time jobs college student. Here some options: Combined subspaces: Du kannst auf Jobbörsen nachstellen, die sich auf Teilzcitjobs spezialisiert haben.
RoAd, response for prompt: How
I find part-time job? I'm college student and poor.
German subspace: Suche ich nach einem Teilzeitjob. Ich bin Student und arbeitslos. Instruction following subspace: You should look for part-time position in your field of study that you are interested in learning about. You may also want consider working local restaurant, retail store, other business. Combined subspaces: Siche dir nach, ob in der Nähe einen Jobcenter gibt.
X
Figure D.3: Qualitative comparison between RoAd and LoReFT for their composability. The prompt is always in English for different subspaces.
31


Reviews:
[{"review_id": "oEeNqWjSWJ", "reviewer": "Reviewer_vHYZ", "summary": "This paper proposes a novel parameter-efficient fine-tuning method called RoAD, aimed at addressing the challenges of efficient deployment of LLMs that require multiple adapters for distinct needs and enhancing the interpretability of LLMs. The motivation behind this approach stems from the observation that the fine-tuning process primarily affects the angular components of representations rather than their magnitude. Thus, RoAD introduces a 2D rotational approach to representations to achieve parameter-efficient adaptation. Experimental results demonstrate that RoAD achieves a superior accuracy-efficiency trade-off compared to baseline approaches.", "questions": "- Can the proposed method be applied to vision language models? Does the observation still stand for vision language models?", "limitations": "The authors have mentioned the potential limitations of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Developing effective approaches to adapt the LLM to downstream tasks with better scalability and performance is an important problem. The achieved performance of this paper looks promising to me. \n\n- The paper layout is clear and the writing is easy to understand. \n\n- The observation that during tuning, the directional change is much more significant than the magnitude change is an interesting observation and may further motivate some follow-up research.", "weaknesses": "- For the observation section, it would be beneficial to include results from more recent autoregressive language models, such as Llama2, to ensure that the observation is consistent and generalizable across more commonly used models.\n\n- Another concern regarding the observation section is that the metrics used for magnitude and angular change are not identical. As a result, the claim that angular change is larger than magnitude change may be significantly influenced by the chosen metric and scaling. The authors should further justify the validity of this observation when using different metrics.\n\n- The authors should consider further benchmarking the proposed method on generative tasks with instruction tuning and evaluate it on more challenging benchmarks, such as Alpaca-Eval (https://github.com/tatsu-lab/alpaca_eval) and MT-Bench (https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort, and thorough review. We appreciate the positive feedback and are encouraged by your highlights:\n\n1. We develop **effective approaches with better scalability and performance** on an important problem. And the achieved **performance is promising** to you.\n2. The paper **layout is clear** and the **writing is easy to understand**.\n3. We offer **interesting observation** that may further **motivate some follow-up research**.\n\n$~$\n\n---\n\nWe address your concerns below:\n\n> **W1. Does the observation from Llama2 stay the same as RoBERTa?**\n\nThank you for your valuable suggestion. We finetune LLaMA2-7B using LoRA on the GSM8K training set and analyze the magnitude and angle changes of the test set samples, as shown in **Figure A.1\\(a)** of the uploaded PDF.\n\n**Summary: LLaMA2-7B exhibits a similar pattern to RoBERTa-base, with more significant changes in angle rather than magnitude.**\n\n- **Experimental Setup:** When directly measuring $\\Delta D$ and $\\Delta M$, we observe $\\Delta D = 1$ and $\\Delta M = 0$ for almost all samples, indicating no change in both magnitude and angle. This is because LLaMA2-7B has a much larger hidden size compared to RoBERTa-base (4096 vs. 768) and is more powerful. Finetuning only slightly adapts most dimensions, with significant changes in a limited number of dimensions. Therefore, we apply t-SNE to reduce the dimensions before calculating $\\Delta D$ and $\\Delta M$.\n- **Detailed Observations:** \n  - Many samples remain close to $\\Delta D = 1$ and $\\Delta M = 0$, showing no significant change in magnitude or angle.\n  - For samples with changed representations, $\\Delta M$ is small (mostly < 0.2), while $\\Delta D$ shows a significant change, ranging from 1 to around -0.75.\n  - There are a few outliers with larger changes in magnitude.\n\n\n$~$\n\n> **W2. Is the observation robust across different metrics?**\n\nThank you for the opportunity to clarify our pilot studies (Section 3.1).\n\n- In our first study, we measure both $\\Delta D$ and $\\Delta M$ to demonstrate that finetuning primarily affects the angle, as indicated by larger $\\Delta D$ values compared to $\\Delta M$. Both $\\Delta D$ and $\\Delta M$ are normalized metrics, with $\\Delta D \\in [-1, 1]$ and $\\Delta M$ representing relative magnitude changes.\n- Acknowledging that our finding might be influenced by the choice of metrics and potentially perceived as subjective, we conduct a second study to disentangle these two factors during finetuning. The results of this second study are consistent with the first, confirming that angle information plays a more crucial role than magnitude in finetuning.\n\nThese studies collectively ensure that our observations are robust and supported by practical experiments. If there are specific metrics you would like us to consider, please let us know, and we would be happy to provide additional analyses using those metrics.\n\n$~$\n\n> **W3. Further benchmark RoAd, like with AlpacaEval, MT-Bench.**\n\nThank you for this great suggestion. We benchmark RoAd using AlpacaEval2.0, and the results can be found in **Table A.1** of the attached PDF. Due to time and resource constraints, benchmarking with MT-Bench is on our to-do list.\n\n**Summary: RoAd demonstrates superior performance compared to all baselines, while utilizing the least number of trainable parameters.**\n\n- **Experimental Setup:** We finetune LLaMA2-7B with two instruction-tuning datasets and evaluate the model using AlpacaEval2.0. This evaluation employs GPT-4 to assess the responses generated by the finetuned model against those produced by Text-davinci-003. We don't choose GPT-4 as the reference model, because GPT-4 is too powerful than LLaMA2-7B. The proof-of-concept experiment with LoRA shows the win-rate < 5\\%.\n\n\n$~$\n\n> **Q1. Apply RoAd to vision language models, and its observation.**\n\nThank you for your suggestion. We finetune LLaVA-1.5-7B using RoAd, and the results are presented in **Table A.2** of the uploaded PDF.\n\n**Summary: RoAd achieves the same average performance as LoRA with only 1/4 of its trainable parameters.**\n\n- **Experimental Setup:** [R1] requires 4.61% trainable parameters for LoRA on this task, while most tasks with LoRA in our paper need < 1%, showing that this task is knowledge-intensive. Therefore, we need to scale RoAd's trainable parameters. For this purpose, we combine it with LoRA due to the limited number of $\\theta_i$ and $\\alpha_i$ in $R$. The combination is represented as $Z = (RW + BA)^TX$, where $R$ is the rotation matrix from RoAd, and $A$ and $B$ are from LoRA. We adjust the LoRA rank to vary the number of trainable parameters. We combine RoAd$_1$ with LoRA, but not RoAd$_2$ or RoAd$_4$, as their primary design purpose is to increase the number of trainable parameters.\n- **Results:** \n    - With only 0.08% trainable parameters, RoAd$_4$ already achieves 96.9% (66.4/68.5) of the accuracy of LoRA with 4.61% trainable parameters. By combining RoAd$_1$ with LoRA, we achieve the same performance as LoRA with just 1/4 of its trainable parameters. This demonstrates RoAd's excellent scalability when combined with LoRA. Further promising scaling results can be found in our response to Reviewer zyNQ in Weakness 2, if you are interested. \n    - The observations are very similar to our response to Weakness 1. We also need to apply t-SNE to reduce the dimensions.\n\n\n$~$\n\n[R1] Visual Instruction Tuning, Haotian Liu, etc.\n\n---\n\n$~$\n\nThank you for your thoughtful suggestions. We have incorporated the new results in the updated version by our side.\n\nIn the general rebuttal block, we summarize all new results in our newly uploaded PDF, and highlight some for your easy choice, if you are interested in our response to other reviewers.\n\nIf these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tyIBiJPYjg", "reviewer": "Reviewer_UyLU", "summary": "This paper proposes a parameter-efficient finetuning method named RoAd, to address two challenges of current methods. The first challenge is the efficient deployment of LLMs equipped with multiple task- or user-specific adapters. The second one is the interpretability of LLMs. RoAd employs a straightforward 2D rotation to adapt LLMs. Experiment results consistently show that RoAd surpasses other PEFT methods", "questions": "- Page 7, line 247, \"Figure 4\" should be \"Table 4\".", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- This paper motivates well on the two challenges of current PEFT methods.\n\n- RoAd achieves impressive results, surpasses other PEFT methods in various tasks.\n\n- The authors perform the insightful pilot study and make interesting observations on the key factor influencing the adaptation of LLMs.", "weaknesses": "- Evaluation of efficiency results for batching could be improved. The proposal of RoAd is well motivated by the overhead of batch matrix multiplication in current methods [1, 54] (Page 2, line 36). However, the authors only compare with LoRA in the evaluation of throughput of batching. It would be better if the authors can compare with [54] in this evaluation.\n\n- Novelty and advantage over OFT [41] need clarification. As RoAd can be considered as a specialized case of OFT with w = 2 (Page 3, line 91), it is important to clarify the technical novelty of RoAd over OFT. In the current form of the paper, the reader may consider RoAd as a special case of OFT without much technical novelty over OFT. \n\n- RoAd fails to consistently outperform other PEFT methods on arithmetic reasoning tasks. Specifically, the results of RoAd on LLaMA-7B is worse than LoRA and AdapterP. Such results make the statement \"consistently surpasses\" in Page 2, line 53, seem like a bit of an overclaim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort, and thorough review. We appreciate the positive feedback and are encouraged by your highlights:\n\n1. Our work is **well-motivated** on two challenges of existing PEFT methods.\n2. Our method, RoAd, achieves **impressive results, surpassing other PEFTs in various tasks**.\n3. We offer **insightful pilot study** and **interesting observations** on the key factor of LLM adaptation.\n\n$~$\n\n---\n\nWe address your concerns below:\n\n\n> **W1. Compare RoAd's batching efficiency to FLoRA [54]**.\n\nThank you for the insightful suggestion. We include the new results in **Figure A.1(b)**.\n\n**Results: RoAd consistently demonstrates higher throughput compared to FLoRA across various generated lengths.**\n\n- Experimental Setup: Since FLoRA didn't open-source its code, we reproduce it by our own on Tansformers and PEFT that we also use for LoRA and RoAd. The rank size for both FLoRA and LoRA is set to 8, the number of request (i.e. batch size) is 8. This setting stays the same for the batching exxperiments in the paper. We further extend the generated length from 4K to 8K.\n\n\n$~$\n\n> **W2. Clarification of RoAd's novelty and advantage over OFT [41].**\n\nThank you for the opportunity to clarify the novelty and advantage of our work.\n\n**Summary: RoAd is simpler, more finetuning-efficient, offers additional functionalities, and achieves better finetuning results.**\n\n- **Methodology Similarity**: Orthogonal finetuning is a widely used method [26, 27, 41, 28, etc.], and can generally be expressed as $z = (RW)^Th$.\n- **Methodology Difference**: OFT constructs $R$ in a block-wise manner and uses Cayley parameterization to maintain orthogonality. In contrast, RoAd employs inherently orthogonal 2D rotation matrices, making it simpler and more straightforward to apply. \n- **Finetuning Efficiency**: As shown in Table 12, OFT's reliance on Cayley parameterization results in higher time and memory usage compared to RoAd. Despite having a similar number of trainable parameters, OFT requires 40GB of GPU memory, whereas RoAd only requires 23GB. Additionally, RoAd's finetuning time is approximately 50 times shorter than OFT's. \n- **Additional Functions**: RoAd not only excels in finetuning efficiency but also features highly efficient batching due to element-wise multiplication. This is not available in OFT, which has batching latency similar to LoRA.\n- **Result**: As demonstrated in Table 2, RoAd significantly outperforms OFT with fewer trainable parameters. OFT with 0.1\\% trainable parameters achieves 82.3 on GLUE, while RoAd$_1$ (fc1) with 0.03\\% trainable parameters offers 85.1. \n\n$~$\n\n> **W3. Mild overclaim of the results, since there is an outlier on the arithmetic reasoning task for LLaMA2-7B.**\n\nThank you for your valuable feedback. We apologize for the oversight. In seven out of eight results (as shown in Tables 2, 3, 4, and 13), RoAd demonstrates superior performance with the minimal level of trainable parameters, with the exception of the arithmetic reasoning task on LLaMA2-7B. To improve the accurate statement, we have revised Line 53 to state:\n\"Seven out of eight benchmarks indicate that RoAd outperforms other PEFT methods while maintaining a significantly reduced scale of trainable parameters (< 0.1%), as partially depicted in Figure 1.\"\n\n$~$\n\n> **Q1. Typo**\n\nThank you for this detailed suggestion. We have corrected this typo in our updated version.\n \n ---\n \n $~$\n \nThank you for your thoughtful suggestions. We have incorporated the new results in the updated version by our side.\n\nIn the general rebuttal block, we summarize all new results in our newly uploaded PDF, and highlight some for your easy choice, if you are interested in our response to other reviewers.\n\nIf these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1uJ92EDfaO", "reviewer": "Reviewer_zyNQ", "summary": "This paper introduces a novel method for parameter-efficient finetuning, RoAd. By employing a straightforward 2D rotation to adapt LLMs, this paper addresses the challenges of existing parameter-efficient finetuning method for LLMs. Experiments on downstream tasks demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper proposes a novel method for parameter-efficient finetuning which efficiently\nadapts LLMs using a minimal number of trainable parameters.\n2.\tThe method enhances both batching efficiency and composability.\n3.\tComprehensive experiments on the GLUE benchmark, eight commonsense reasoning tasks and four arithmetic reasoning tasks are conducted to show the efficacy of the method.", "weaknesses": "1.\tIn the results of Table 2,  RoAd shows different performance on base and large model, what could be the reason? Why the RoAd(fc1) of large model with less parameters shows better average accuracy? Why the full FT setting shows even lower accuracy?\n2.\tIf the proposed RoAd maintains the same quantity of parameters of existing method like LoRA, could the accuracy be further improved?\n3.\tHow about the RoAd2 and RoAd4 on the held-out GLUE development set?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, effort, and thorough review. We appreciate the positive feedback and are encouraged by your highlights:\n\n1. We propose a **novel** PEFT method that efficiently adapts LLMs with **a minimal number of trainable parameters**.\n2. Our method, RoAd, **enhances both batching efficiency and composability**.\n3. **Comprehensive experiments on three benchmarks (20 tasks in total) show the efficacy of RoAd**.\n\n$~$\n\n---\n\nWe address your concerns below:\n\n> **W3. What are the results of RoAd$_2$ and RoAd$_4$ on GLUE?**\n\nWe add the new results from RoAd$_2$ and RoAd$_4$ on the GLUE benchmark in **Table A.3 (Row 1-10)** of the uploaded PDF. For ease of comparison, we also provide the results from the best baseline (Full FT) and RoAd$_1$ from the paper.\n\n**Result: Overall, more trainable parameters (from RoAd$_1$ to RoAd$_4$) offer better performance on RoBERTa-base, while the performance on RoBERTa-large is saturated, with all RoAd variants performing similarly best.**\n\n$~$\n\n> **W1. What are the reasons for the unexpected results in Table 2?**\n\nThank you for this insightful question. We appreciate the opportunity to further explain the results. Our explanations are based on **Table A.3 (Row 1-10)**, where the tasks are arranged from low-resource (left) to high-resource (right).\n\n**1. Why does RoAd$_1$(fc1) on RoBERTa-large with less parameters show better average accuracy than RoAd$_1$?** \n\nWe believe that RoAd$_1$(fc1) is comparable to instead of better than RoAd$_1$ on RoBERTa-large, as the difference is only 0.1. From the results of Full FT, RoAd$_2$ and RoAd$_4$, we observe that their average performance is very similar, indicating performance saturation. Therefore, increasing the number of trainable parameters does not significantly affect the performance.\n\n**2. Why does Full FT underperform RoAd?**\n\nPrevious research [R1, R2, etc.] has demonstrated that PEFT methods can outperform Full FT. We attribute this to two factors:\n\n- Model Capability: A powerful LLM benefits more from slight adaptations via PEFT. Full FT changes more parameters, risking catastrophic forgetting.\n- Task Nature: Less knowledge-intensive tasks, particularly those sharing the same domain as pretraining data, are better suited for PEFT. For knowledge-intensive tasks like code and math, more trainable parameters (Full FT) may be beneficial.\n\nOur Table A.3 supports this:\n- On RoBERTa-base, Full FT excels in high-resource tasks (QNLI, QQP, MNLI) due to their larger training samples (knowledge-intensive) and RoBERTa-base's lower power.\n- On RoBERTa-large, Full FT and RoAd perform similarly on these high-resource tasks, as RoBERTa-large's higher capability makes slight adaptations sufficient.\n\n**3. Why does RoAd show different trend on the base and large model?**\n\nThis trend aligns with the second point. On RoBERTa-base, which is less powerful, more trainable parameters enhance performance until saturation. On RoBERTa-large, a more powerful model, slight adaptations with fewer trainable parameters suffice for the GLUE tasks.\n\n$~$\n\n> **W2. Can even better results be obtained for scaling up RoAd's trainable parameters to the same level as LoRA's?**\n\nThank you for this excellent suggestion!\n\n**Summary: RoAd exhibits impressive scalability. Increasing its trainable parameters leads to notably improved results.**\n\n- Experimental setting: To increase the number of trainable parameters in RoAd, we combine it with LoRA due to the limited number of $\\theta_i$ and $\\alpha_i$ in $R$. The combination is represented as $Z = (RW + BA)^TX$, where $R$ is the rotation matrix from RoAd, and $A$ and $B$ are from LoRA. We vary the number of trainable parameters by adjusting the LoRA rank. In this experiment, we only combine RoAd$_1$ with LoRA, excluding RoAd$_2$ and RoAd$_4$, as their main design purpose is to increase the number of trainable parameters.\n- Results: As demonstrated in Table R.1 and Table R.2, increasing the number of trainable parameters to the same level as LoRA's yields significantly better results. This shows RoAd's excellent scalability when combined with LoRA.\n\n$~$\n\nTable R.1: Average accuracy on eight commonsen reasoning tasks. Detailed numbers for RoAd$_1$ + LoRA are in Table A.4 (Row 1-2) of the uploaded PDF. \n| Model | Method | #Params (%) | Avg. |\n| :--- | :--- | :---: | :---: |\n| LLaMA-7B | LoRA | 0.83 | 74.7 |\n| | LoReFT | 0.03 | 80.2 |\n| | RoAd$_4$ | 0.08 | 80.2 |\n| | RoAd$_2$ | 0.04 | 80.2 |\n| | RoAd$_1$ | 0.02 | 79.2 |\n| | RoAd$_1$ + LoRA | 0.84 | **82.2** |\n| --- | --- | --- | --- |\n| LLaMA-13B | LoRA | 0.67 |  80.5 |\n| | LoReFT | 0.03 |  83.3 |\n| | RoAd$_4$ | 0.07 |  83.7 | \n| | RoAd$_2$ | 0.03 | 83.8 |\n| | RoAd$_1$ | 0.02 | 83.0 |\n| | RoAd$_1$ + LoRA | 0.68 | **85.4** |\n\n\nTable R.2: Average accuracy on four arithmetic reasoning tasks. Detailed numbers for RoAd$_1$ + LoRA are in Table D.5 (Row 1-2) of the uploaded PDF. \n| Model | Method | #Params (%) | Avg. |\n| :--- | :--- | :---: | :---: |\n| | LoRA | 0.83 | 46.9 |\n| | RoAd$_4$ | 0.08 | 45.8 |\n| LLaMA-7B | RoAd$_2$ | 0.04 | 46.2 |\n| | RoAd$_1$ | 0.02 | 44.0 |\n| | RoAd$_1$ + LoRA | 0.84 | **50.0** |\n| --- | --- | --- | --- | \n| | LoRA | 0.67 | 51.1 |\n| | RoAd$_4$ | 0.07 | 52.3 |\n| LLaMA-13B | RoAd$_2$ | 0.03 | 52.2 |\n| | RoAd$_1$ | 0.02 | 51.9 |\n| | RoAd$_1$ + LoRA | 0.68 | **55.1** |\n\n\n$~$\n\n[R1] LoRA: Low-Rank Adaptation of Large Language Models, Edward J. Hu, etc.\n\n[R2] Compacter: Efficient Low-Rank Hypercomplex Adapter Layers, Rabeeh Karimi Mahabadi, etc.\n\n---\n\n$~$\n\nThank you for your thoughtful suggestions. We have incorporated the new results in the updated version by our side.\n\nIn the general rebuttal block, we summarize all new results in our newly uploaded PDF, and highlight some for your easy choice, if you are interested in our response to other reviewers.\n\nIf these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tWDjGlQaXA", "reviewer": "Reviewer_iQCQ", "summary": "This paper proposes a 2D rotation adaptation method called RoAd for efficiently fine-tuning large language models (LLMs). RoAd achieves parameter-efficient fine-tuning by rotating representations. Experimental results demonstrate that RoAd performs excellently across multiple benchmarks, reducing the number of training parameters and computational overhead.", "questions": "see Weaknesses.", "limitations": "In the Limitations section, the authors mentioned scalability and training parameters. Since the primary comparison in this paper is with LoRA, could the authors reduce LoRA's parameter count to the same level as the method proposed in this paper and then conduct the relevant comparative experiments?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1 The method proposed in this paper is simple but efficient.\n\n2 This method performs well on small-scale language models, achieving better performance with fewer or comparable parameters.", "weaknesses": "1 The paper claims in the abstract that one of the main scenarios it addresses is multitasking. However, the authors mainly illustrate this through qualitative experiments in section 4.3, which seems unconvincing. It is suggested that the authors refer to ATTEMPT [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts] and compare it in the same scenario to fully demonstrate RoAd's advantages in multitask learning.\n\n2 There is some confusion about the logical flow of the paper, especially in chapter 2. It seems that sections 2.2 and 2.3 are not closely related to section 2.1.\n\n3 The method's improvement on large-scale language models is relatively limited. As shown in Table 3, RoAd2/4 performs the same as LoReFT when using more parameters. This needs to be explained.\n\n4 It is suggested that the authors add a main diagram to describe the method.\n\n5 Figure 3 Middle shows that as the generated tokens increase, RoAd's throughput decreases rapidly, while LoRA does not show this trend. Will RoAd's throughput be lower than LoRA's when generating longer texts? Furthermore, what are the respective parameter amounts for LoRA and RoAd here?\n\n6 This paper seems to have similarities with SSF [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning], as both methods fine-tune by adjusting output representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We are really encouraged by the highlights:\n\n1. Our proposed method, RoAd, is **simple but efficient**.\n2. RoAd **performs excellently across multiple benchmarks, reducing trainable parameters and computational overhead**.\n3. RoAd **performs well** on small-scale (100M-13B) LMs, **better performance with fewer or comparable parameters**.\n\n$~$\n\n---\n\nWe address your concerns below:\n\n> **W1. Compare RoAd to ATTEMPT for demonstrating its multitasking ability.**\n\nThank you for this valuable suggestion. Please refer to **Table A.3 (Row 11-13)** in the uploaded PDF for detailed results.\n\n**Summary: RoAd exhibits strong multitasking capabilities, surpassing ATTEMPT with a similar number of trainable parameters.**\n\n- **Setup:** Adopting the setup of ATTEMPT, we concatenate all GLUE tasks for finetuning. For each task, RoAd uses a unique $R$ for each linear layer. The first half of the blocks in $R$ are shared across tasks, while the second half are updated only when the corresponding task is encountered.\n\n$~$\n\n> **W2. Confusion about the logical flow of the paper. I.e. Subsections in $2 are not closely related.**\n\nThank you for the opportunity to clarify our motivation. Section 2 serves as a background introduction:\n- $2.1: Discusses related PEFT methods.\n- $2.2: Highlights the batching challenge in efficiently deploying multiple adapters for various tasks/purposes/users.\n- $2.3: Highlights the challenge of interpretability, which is crucial for LLMs but not necessarily tied to PEFT.\n\nExisting works often focus on one or two of these challenges. However, our proposed method, RoAd, tackles all three (PEFT, batching, and interpretability) with a unified approach.\n\n$~$\n\n> **W3. Relatively limited improvement on large-scale LLMs.** E.g. in Table 3, RoAd$_{2/4}$ performs the same as LoReFT while using more parameters.\n\nWe politely disagree with this assessment. To clarify, below we summarize the performance on large-scale LLMs from our paper, including LoReFT and the RoAd's variant with a similar number of trainable parameters.\n\n**Summary: Overall, in seven out of eight results (Table 2, 3, 4, and 13 in paper), RoAd demonstrates superior performance with the minimal level of trainable parameters. While RoAd shows comparable performance to LoReFT on commonsense reasoning, it significantly outperforms LoReFT on knowledge-intensive tasks, i.e. arithmetic reasoning.** \n\nFor commonsense, since its domain is closely alligned with pretraining data, slight adaptation is good enough.\n\n$~$\n\nTable R.1. Accuracy on commonsense.\n| Model | Method | #Params (%) | Avg. |\n| :--- | :--- | ---: | ---: |\n|LLaMA-7B | LoReFT | 0.03 | **80.2** |\n|| RoAd$_2$ | 0.04 | **80.2** | \n| - | - | - | - | - |\n| LLaMA-13B | LoReFT | 0.03 | 83.3 |\n| | RoAd$_2$ | 0.03 | **83.8** |\n\nTable R.2. Accuracy on arithmetic.\n| Model | Method | #Params (%) | Avg. |\n| :--- | :--- | ---: | ---: |\n| LLaMA-7B | LoReFT | 0.03 | 42.6 | |\n| | RoAd$_1$ | 0.02 | **44.0** | \n| - | - | - | - | - |\n| LLaMA-13B | LoReFT | 0.03 | 49.6 |\n| | RoAd$_1$ | 0.02 | **51.9** |\n\n\n$~$\n\n> **W4. Add a main diagram to describe the method**.\n\nThank you for this great suggestion. We add an overview diagram in **Figure A.1(c)**. \n\n\n$~$\n\n> **W5. Batching efficiency of RoAd for longer (>4K) generation.**\n\nThank you for your insightful suggestion. We extend the generated length to 8K (as our GPU resources do not support lengths beyond this) and present the results in **Figure A.1(b)**.\n\n**Results: While RoAd's throughput decreases more sharply than LoRA's with an increasing sequence length, it remains significantly higher than LoRA's.** RoAd's throughput can be seen as the upper limit of LoRA's throughput for batching, as RoAd functions like LoRA with a rank size of 1.\n\nLoRA's rank here is 8, corresponding to approximately 0.20% parameters. This is a moderate setting; for our finetuning experiments on commonsense and arithmetic reasoning tasks, LoRA's rank is 32. RoAd, with its trainable parameters equivalent to LoRA with a rank size of 1 (about 0.03%).\n\n$~$\n\n> **W6. Clarification of the similarity with SSF.**\n\nThank you for bringing this related work to our attention; we will include it in $2.1.\n\n**Summary: RoAd's methodology differs significantly from SSF and demonstrates superior performance.**\n\n- **Difference**: To illustrate, let's assume the hidden size is two, i.e., $h = [h_1, h_2]$. SSF adapts $h$ as $z = \\gamma \\odot h + \\beta = [\\gamma_1 h_1 + \\beta_1, \\gamma_2 h_2 + \\beta_2]$, showing no interaction between $h_1$ and $h_2$. In contrast, RoAd rotates $h$ as $z = Rh$, where $R$ is a 2D rotation matrix. In this way, RoAd promotes interaction between $h_1$ and $h_2$. Our pilot studies ($3.1) indicate that such rotation is more crucial than merely adjusting the magnitude.\n- **Result**: We reproduce SSF on two benchmarks, as detailed in Table A.4 and A.5 (Row 3-4) of the uploaded PDF. With a similar number of trainable parameters (0.03%), RoAd$_2$ significantly outperforms SSF: 80.2 vs. 78.2 and 83.8 vs. 82.3 for commonsense tasks, and 46.2 vs. 35.9 and 52.2 vs. 42.9 for arithmetic tasks.\n\n$~$\n\n> **Limitation:  Reduce LoRA's parameter to the same level of RoAd.**\n\nHere we set LoRA's rank to 1, so it's trainable parameters are the same as RoAd$_2$. We show the results in **Table A.4 and Table A.5 (Row 5)**.\n\n**Result: RoAd$_2$ significantly outperforms LoRA, 80.2 vs 74.2 on commonsense tasks, 46.2 vs 41.7 on arithemetic tasks.** \n\nYou can refer to our reponse to Reviewer zyNQ in W2 for the results of scaling up RoAd's parameters to the same level of LoRA's, if you are interested.\n\n---\n\n$~$\n\nThank you for the thoughtful suggestions. We have incorporated the new results in the updated version.\n\nPlease refer to the general rebuttal block for more new results, if you are interested.\n\nIf these revisions address your concerns, we kindly request a reconsideration of the scores. Should you have any further questions, we are happy to assist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Here we summarize the new results in the uploaded PDF, you can selectively read them if you are interested. We highlight some for your easy choice.

| Table or Figure | Content | Where for details (i.e. response to which point of which reviewer) |
| :---  |  :--- | :--- |
| Table A.1 | **Further benchmark RoAd with AlpacaEval2.0** | W3 of Reviewer vHYZ |
| Table A.2 | **Further benchmark RoAd with visual instruction tuning** | Q1 of Reviewer vHYZ |
| Table A.3 (Row 1-10) | RoAd$_2$ and RoAd$_4$ on GLUE | W3 of Reviewer zyNQ |
| Table A.3 (Row 11-13) | RoAd's multitasking ability | W1 of Reviewer iQCQ |
| Table A.4 and Table A.5 (Row 1-2) | **RoAd's scalability** | W2 of Reviewer zyNQ |
| Table A.4 and Table A.5 (Row 3-4) | New baseline, SSF | W6 of Reviewer iQCQ |
| Figure A.1 (a) | Pilot study on LLaMA2-7B | W1 of Reviewer vHYZ |
|  Figure A.1 (b) | More batching efficiency results | W1 of Reviewer UyLU and W5 of Reviewer iQCQ |
| Figure A.1 (c) | **An overview diagram** | W4 of Reviewer iQCQ |

Meta-review: This paper introduces a novel parameter-efficient finetuning method named RoAd, which employs a 2D rotation to adapt large language models (LLMs). The paper addresses three key challenges in parameter-efficient finetuning (PEFT): efficient deployment of LLMs that require multiple task-specific adapters, batching efficiency, and interpretability. RoAd demonstrates its effectiveness across various benchmarks, including GLUE, commonsense reasoning, and arithmetic reasoning tasks, with less than 0.1% trainable parameters. The authors provided a thorough experimental analysis, showing RoAd’s superior performance over baseline methods like LoRA and OFT.
The initial reviews were mixed, with the reviewers recognizing the significance of the problem and the promising results but raising concerns about the evaluation, novelty over existing methods, and clarity of the paper’s structure. The authors engaged actively with the reviewers, providing detailed rebuttals and additional experiments that addressed most of the concerns. After the rebuttal, the final ratings are uniformly positive, with all reviewers agreeing that the paper is technically solid and has moderate-to-high impact. The reviewers acknowledged the authors' efforts to address their concerns, which led to increased scores in some cases. The AC recommends accepting this paper, with the expectation that the authors will incorporate the final revisions to further strengthen the paper's clarity and completeness.
Decision: Accept (poster)