3-in-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability

Authors: baohao  liao
christof  monz
dora  lora
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:14:48

Abstract: Parameter-efficient finetuning (PEFT) methods effectively adapt large language models (LLMs) to diverse downstream tasks, reducing storage and GPU memory demands. Despite these advantages, several applications pose new challenges to PEFT beyond mere parameter efficiency. One notable challenge involves the efficient deployment of LLMs equipped with multiple task- or user-specific adapters, particularly when different adapters are needed for distinct requests within the same batch. Another challenge is the interpretability of LLMs, which is crucial for understanding how LLMs function. Previous studies introduced various approaches to address different challenges. In this paper, we introduce a novel method, RoAd, which employs a straightforward 2D rotation to adapt LLMs and addresses all the above challenges: (1) RoAd is remarkably parameter-efficient, delivering optimal performance on GLUE, eight commonsense reasoning tasks and four arithmetic reasoning tasks with <0.1% trainable parameters; (2) RoAd facilitates the efficient serving of requests requiring different adapters within a batch, with an overhead comparable to element-wise multiplication instead of batch matrix multiplication; (3) RoAd enhances LLM's interpretability through integration within a framework of distributed interchange intervention, demonstrated via composition experiments.

Main paper: 
 introduction: 
 Large language models (LLMs), trained on extensive web-scale datasets to perform tasks such as predicting masked words [8, 31, 45] or anticipating the next word in a sentence [17, 52, 53] , demonstrate remarkable effectiveness across a range of NLP applications. For tasks where the data distribution diverges from that of the pretraining corpus, finetuning emerges as an effective way to tailor an LLM to specific requirements. Leveraging the capabilities of LLMs, recent studies [13, 14, 22, 23, 25, 27, 42, 60, 62, 65] demonstrate that training only a subset of an LLM's parameters can yield performance on par with full finetuning. This approach, termed parameter-efficient finetuning (PEFT), provides two primary advantages: (1) It reduces the storage requirements for trained parameters, as it necessitates preserving only a universal LLM alongside a minimal set of task-specific parameters; (2) It decreases GPU memory consumption during finetuning, owing to the reduction in optimizer state sizes which correlate directly with the number of trainable parameters.With the evolution of PEFT, concerns extend beyond mere parameter efficiency. PEFT encounters a variety of challenges brought forth by diverse applications. A significant challenge is the efficient deployment of personalized or task-specific LLMs [25, 57] . These applications frequently require distinct sets of trained parameters for different tasks or users. When multiple users submit requests simultaneously, it becomes crucial to process these requests collectively in a single batch. Given that each request may require a unique set of parameters, using batch matrix multiplication can efficiently handle these requests by leveraging GPU parallelism. However, the batch matrix multiplication still incurs considerable overhead [1, 57] , necessitating the exploration of more efficient methods.Another challenge is the interpretability of LLMs that contain a billion-scale of parameters, making it difficult to explore their mechanism. PEFT provides an alternative approach by constraining the number of trainable parameters, thereby aiding in interpretability. Recent advancements in PEFT methods, particularly those focusing on representation editing [54, 60, 67] , can be incorporated within an intervention framework [11] . This integration enhances their capability for interpretability, offering a more manageable means of dissecting the operational intricacies of LLMs.In this paper, we introduce a novel technique termed 2D rotary adaptation (RoAd) which efficiently adapts LLMs using a minimal number of trainable parameters. Furthermore, RoAd enhances both batching efficiency and composability. Our initial investigation reveals that finetuning primarily alters the angular components of the representations in pretrained LLMs, rather than their magnitudes (Section §3.1). Based on this observation, we employ a strategy of rotating certain subspaces within the representations to emulate finetuning effects. Specifically, we implement a 2D rotational approach on the representations and develop three distinct variants of RoAd (Section §3.2).To assess the efficacy of RoAd, we perform comprehensive evaluations on the GLUE benchmark [56] , eight commonsense reasoning tasks and four arithmetic reasoning tasks, utilizing RoBERTa [31] and LLaMA [52, 53] (Section §4.1). The results consistently show that RoAd surpasses other PEFT methods while maintaining a significantly reduced scale of trainable parameters (< 0.1%), as depicted in Figure 1 . Additionally, RoAd employs element-wise rather than matrix multiplication, which notably improves throughput when serving heterogeneous requests within the same batch, achieving twice the throughput of LoRA [14] (Section §4.2). Furthermore, RoAd can be seamlessly integrated within an intervention framework [11] , thereby enhancing model interpretability. We illustrate this through a composition experiment, demonstrating RoAd's capacity to merge weights trained for different tasks and display a new capability (Section §4.3).Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We accurately make claims about our paper's contributions and scope in the abstract and introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background: 
 In this section, we outline the challenges tackled in this work, illustrating the constraints of existing methods and objectives that drive the development of the proposed method, RoAd.Existing PEFT techniques can be categorized into three groups: adapter-based, prompt-based, and latency-less methods. Adapter-based methods [12, 13, 42] incorporate adapters either in parallel with or sequentially to the existing Transformer [55] modules. This incorporation necessitates modifications to the LLM architecture, consequently adding extra latency during inference. Promptbased methods [19, 21, 43] enhance the input by appending new trainable tokens, which lengthens the sequence and thereby increases the computational overhead during inference. Latency-less methods, such as LoRA [14] and its variants [22, 27, 65] , apply low-rank matrices to adapt the pretrained weights. These matrices can be seamlessly integrated into the existing weight matrices following finetuning, thus preserving the original LLM architecture. Specifically, LoRA adapts an LLM as W = W 0 +∆W , where W 0 ∈ R d1×d2 is the pretrained weight and ∆W = BA with B ∈ R d1×r , A ∈ R r×d2 , r ≪ d 1 and r ≪ d 2 . Our proposed method, RoAd, aligns with the latency-less category and integrates effortlessly into the existing linear layer without imposing additional overhead during inference. Moreover, RoAd demonstrates exceptional parameter efficiency. The quantity of its trainable parameters is equivalent to that of a LoRA module with a rank r = 0.5.Orthogonal finetuning. Drawing on the concept of hyperspherical energy and its role in characterizing generalization [28, 29] , OFT [44] introduces orthogonal finetuning, an effective PEFT method for finetuning text-to-image diffusion models. Specifically, OFT implements an orthogonal matrix R ∈ R d1×d1 to the pretrained weight W 0 , so the input x ∈ R d1 to a linear layer after adaptation becomes z = (RW 0 ) ⊤ x. R is parameter-efficient because it is a block-diagonal matrix with n blocks as R = diag(R 1 , ..., R i , ..., R n ), where each block R i ∈ R w×w has a dimension w = d 1 /n. To maintain orthogonality, R i is derived using Cayley parameterization:R i = (I +Q i )(I -Q i ) -1 with Q i ∈ R w×w being a skew-symmetric matrix (Q i = -Q ⊤ i ).In sum, {Q i } n i=1 serve as the trainable parameters and R is constructed from them with Cayley parameterization. Subsequent advancement, BOFT [30] , leverages butterfly factorization to further refine OFT's parameter efficiency. However, both OFT and BOFT, due to their reliance on matrix inversions in the Cayley parameterization and increased storage of intermediate activations, necessitate additional GPU memory and increase training duration compared to other PEFT approaches. Conversely, RoAd, which may be considered as a specialized case of OFT with w = 2, offers a faster and more memory-efficient solution by inherently maintaining orthogonality without requiring further parameterization.Batching in this context refers to processing multiple heterogeneous requests, each requiring different adapters2 for inference. This scenario commonly arises when serving personalized or task-specific LLMs. Specifically, we consider a setup where distinct adapters instead of a shared adapter are finetuned for various tasks to achieve optimal performance. During inference, each request in a batch pertains to a different task and necessitates a unique adapter.Consider that we have finetuned distinct LoRA modules for b tasks, denoted as {A i , B i } b i=1 . For a batch of b requests represented as X ∈ R b×l×d1 , where l is the maximum sequence length across the requests, each request requires a different LoRA module. To exploit the parallel processing capabilities of GPUs, the output Z of a linear layer can be computed as follows: First, the output from the pretrained layer is computed as Z 0 = torch.mm(X, W 0 ). Subsequently, the intermediate output from the first low-rank matrix, B ∈ R b×d1×r (a concatenation of {B i } b i=1 ), is obtained as Z 1 0 = torch.bmm(X, B). The output from the second low-rank matrix, Â ∈ R b×r×d2 (a concatenation of {A i } b i=1 ), follows as Z 1 = torch.bmm(Z 1 0 , Â). Finally, these outputs are summed to produce Z = Z 0 + Z 1 . It is noteworthy that batch matrix multiplication (BMM), as implemented in torch.bmm, often introduces substantial overhead [1] , reducing throughput and increasing latency, which adversely impacts user experience in time-sensitive applications.In contrast, prompt-based methods circumvent the use of BMM by appending trainable tokens to each request, simplifying the computational process. However, prompt-based methods with long prompt tokens are difficult to optimize, which degrades performance compared to other PEFTs [14, 15] . (IA) 3 [25] proposes adapting LLM by multiplying the output from a linear layer with a trainable vector, involving only element-wise multiplication for efficient batching. A recent development, FLoRA [58] , builds on (IA) 3 by employing two low-rank matrices while maintaining element-wise operations. Although our proposed method, RoAd, requires BMM, its sparse structure allows a reformulation of BMM and results in an overhead equivalent to element-wise multiplication.Numerous studies [10, 11, 37, 38, 40] have provided support for the linear representation hypothesis [35, 46, 49] that concepts are represented within linear subspaces of neural network representations. To examine if a concept is captured within a linear subspace of a representation, Geiger et al. [11] suggests employing a distributed interchange intervention (DII) defined as:EQUATIONb denotes the hidden representation generated at row i and column k when the model processes an input, while s represents the corresponding representation when the model processes a different input. The matrix R ∈ R r×d1 , consisting of orthogonal rows, serves as a low-rank projection matrix where d 1 is the dimension of the representation and r is the subspace dimension under intervention. Equation ( 1) illustrates the application of a DII to b using a counterfactual source representation s. 3Drawing inspiration from this established framework, a recent study, LoReFT [61] , introduces a method for finetuning specific positions of the representations to adapt LLM. This study further demonstrates that several prior approaches of representation editing [54, 60, 67] can be effectively integrated within this framework. Interestingly, the application of RoAd to representations can also be conceptualized as DII, offering interpretability potential. To demonstrate one aspect of interpretability for RoAd, we primarily conduct a qualitative experiment focused on task composition. This experiment involves combining the weights of models trained on distinct tasks to showcase the capability for multitasking learning without the need for additional adaptation [16, 20, 61, 64, 66] .Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We include a section, Section A, to discuss the limitations of our work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we first perform two pilot studies to ascertain the key factor influencing the adaptation of LLMs. Following this, we present our proposed method, the 2D rotary adaptation (RoAd), which serves as an effective PEFT method addressing the various challenges outlined in Section §2.Study 1: Variations in magnitude and angular displacement. Assume x 0 , x ∈ R d1 are representations of the same token from a pretrained and finetuned LLM, respectively. We define the relative change in magnitude as ∆M = ∥x∥ 2 -∥x 0 ∥ 2 /∥x 0 ∥ 2 and compute the angular displacement as∆D = cos(x, x 0 ) ∈ [-1, 1].A larger ∆M and a smaller ∆D indicate more significant changes in magnitude and angular displacement, respectively. Our study involves: (1) finetuning RoBERTa-base [31] on the SST-2 task [50] using either full finetuning or LoRA; (2) extracting representations x 0 and x from the output of the second-last Transformer block for the [CLS] token across all samples in the development set, followed by computing ∆M and ∆D. 4 As depicted in Figure 2 (Left and Middle), there is a more pronounced change in ∆D than in ∆M for both full finetuning and LoRA. 5Study 2: Disentanglement of magnitude and angle. To ascertain whether angular or magnitude adjustments are more critical for finetuning, we implement a disentanglement study. This involves freezing RoBERTa-base and appending a two-layer classifier on top of it. The first layer of this classifier incorporates a weight matrix W ∈ R d1×d1 . Under standard operations, the output from this layer is computed as z = W ⊤ x 0 . To distinctly evaluate the impacts of magnitude and angle, we modify the output to retain only the magnitude component as z i = ∥W :,i ∥ 2 • ∥x 0 ∥ 2 , or solely the angular component as z i = cos(W :,i , x 0 ) (z i is the i th element of z). The modified classifier was then finetuned on four GLUE tasks with different metrics detailed in Table C .1. Additionally, a weak baseline employing a randomly initialized RoBERTa-base is included. As shown in Figure 2 (Right), angular information is paramount in finetuning, whereas reliance solely on magnitude information even leads to inferior results compared to the random backbone.Both studies indicate that angular information is more crucial than magnitude information for adapting a pretrained LLM to a downstream task. However, rotating the entire d 1 dimensions of the representation for finetuning incurs substantial computational costs. These costs are primarily reflected in a large number of trainable parameters, necessitating a dense matrix R ∈ R d1×d1 , and in the requirement to maintain its orthogonality. Could we only rotate a subspace of the representation and design a R that is always orthogonal without any parameterization as OFT [44] ? The first idea that comes to our mind is 2D rotation which only rotates two dimensions at a time and inherently maintains orthogonality. Suppose that W 0 ∈ R d1×d2 is the pretrained weight of a linear layer, x ∈ R d1 is the input of a token to this linear layer, R ∈ R d2×d2 is the rotation matrix, the adapted output from the linear layer is z = Rh = R(W 0⊤ x). The rotation matrix R is defined as follows:ℎ ! ℎ " ℎ # ℎ $ ℎ %&# ℎ % 𝑧 ! 𝑧 " 𝑧 # 𝑧 $ 𝑧 %&# 𝑧 % ℎ! ℎ " 𝑧! 𝑧" 𝜃! 𝑧 ! # + 𝑧 " # ! = 𝛼! ℎ ! # + ℎ " # ! . . . . . .EQUATIONThe trainable parameters are denoted as {θ i } d2/2 i=1 . This 2D rotary adaptation involves rotating pairs of adjacent dimensions of h, specifically dimensions 2i -1 and 2i, using the rotation matrix R i . 6 The rotation matrix R is characterized by its parameter efficiency, which is attributed to its sparse structure and the parameter sharing within each block R i . Additionally, R can be integrated directly into the existing pretrained weights, forming W = W 0 R ⊤ , which does not incur additional computational costs during inference. This design closely mirrors RoPE [51] , with the notable difference that in our RoAd, θ i is trainable and R i does not incorporate positional information. The overview of RoAd is shown in Figure 3 .Relaxation to orthogonality. Referring to Figure 2 (Right), while reliance predominantly on angular information substantially outperforms reliance on magnitude information, it remains less effective than using both angular and magnitude information for the tasks of MRPC, STS-B, and CoLA. Furthermore, both fully-and LoRA-finetuned LLMs exhibit slight adaptations in magnitude, as depicted in Figure 2 (Left and Middle). Consequently, we modify R i by incorporating α i to regulate the magnitude. We define a general R i as follows: R i = α i,We develop three variants of RoAd by altering the configuration of shared parameters as outlined in Table 1 . RoAd 1 introduces a minimal change to Equation (2) by incorporating a scaling factor α i . RoAd 1 already shows impressive results for most tasks in Section §4.1. For some knowledge-intensive tasks, we observe that RoAd 2 and RoAd 4 obtain better results with more trainable parameters. To preserve the starting point of LLMs [23] , we always initialize α i = 1 and θ i = 0.Batching. In practice, we don't need to save R as a sparse matrix and do matrix multiplication.Taking RoAd 1 as an example in Equation ( 4), we only save two vectors: R 1 and R 2 . Thenz = Rh = R 1 ⊗ h + R 2 ⊗ĥ, where ĥ is a rearranged version of h and ⊗ denotes element-wise multiplication. This reformulation not only simplifies the representation of R but also enhances the efficiency of batching in RoAd, relying solely on element-wise multiplications rather than BMM. z = Rh = R 1 ⊗ h + R 2 ⊗ ĥ =             α1 cos θ1 α1 cos θ1 α2 cos θ2 α2 cos θ2 . . . α d 2 /2 cos θ d 2 /2 α d 2 /2 cos θ d 2 /2             ⊗             h1 h2 h3 h4 . . . h d 2 -1 h d 2             +             α1 sin θ1 α1 sin θ1 α2 sin θ2 α2 sin θ2 . . . α d 2 /2 sin θ d 2 /2 α d 2 /2 sin θ d 2 /2             ⊗             -h2 h1 -h4 h3 . . . -h d 2 h d 2 -1             (4)Composability. RoAd can be incorporated into the DII framework as 1) being set to h. Although a degree of relaxation is introduced to the orthogonality of R, it is important to note that the rows of R remain orthogonal to each other within non-adjacent segments of the same block, R i . This offers a possibility for composability. We can finetune some rows on one task and other orthogonal rows on another task. Since they are orthogonal to each other, these two tasks should minimally affect each other, and the combination of these rows after finetuning could bring new multitasking learning ability.Φ(h) = Rh = h + R(h - R ⊤ h), with Rs in Equation (RoAd can be considered as a special case of OFT [44] with w = 2. However, it is much more parameter-and memory-efficient and faster. Please refer to Section §D.1 for a detailed discussion.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Our work is not about theory. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 In this section, we begin by implementing RoAd to finetune various LLMs across three benchmarks. Subsequently, we illustrate its efficiency in batching processes and demonstrate its composability. Unless otherwise noted, RoAd is applied to all linear layers within the LLMs. All of our experiments are conducted on A100 80GB GPU with the frameworks, Transformers [59] and PEFT [34] .Natural language understanding (NLU). We evaluate the effectiveness of RoAd on the GLUE benchmark [56] for its ability of NLU with RoBERTa [31] as the backbone. Unlike many previous works [14, 22, 23, 31, 65] that employ the GLUE development sets for both validation and testing, here we partition the development set into distinct validation and test subsets to mitigate the risk of overfitting. For comprehensive information regarding the split of the development set, the search space of hyperparameters, the optimal hyperparameter configurations, and other details crucial for reproducibility, please see Section §C.1.As shown in Table 2 , RoAd 1 outperforms all other PEFT methods with < 0.1% trainable parameters for both sizes of RoBERTa on average, being the only PEFT method that matches or outperforms full finetuning. These results show that 2D rotation (with a few scaling) can efficiently adapt LLM.Commonsense reasoning. In assessing the capacity of LLaMA [52] for commonsense reasoning, we focus on eight representative tasks: BoolQ [4] , PIQA [3] , SIQA [48] , HellaSwag [63] , WinoGrande [47] , ARC-e, ARC-c [5] , and OBQA [36] . The setting here contrasts with the NLU experiments where each task involves finetuning a separate LLM. Instead, we adopt a unified strategy by finetuning a single LLM across all tasks as delineated in Hu et al. [15] . Such a setting is designed to mitigate overfitting and aligns more closely with real-world applications. Specifically, the training and test sets from these eight tasks are reformulated according to a predefined template, so all tasks can be trained or evaluated in a generative way. For all finetuning experiments on LLaMA, we follow a recipe in Table C .5 without extensive searching. Please see Section §C.2 for more training details.Table 2 : Results on the held-out GLUE development set with RoBERTa as the backbone. We report matched accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B and accuracy for other tasks. The best and second-best results are in bold and underlined, respectively, being the same for other tables. The percentage of trainable parameters is calculated without considering the classifier head. RoAd 1 (fc1) means that we only insert the RoAd 1 module to the first feed-forward layer, to match the #Params. of RED and LoReFT. Results of methods denoted by * and ⋄ are from Wu et al. [60] and Wu et al. [61] , respectively. Otherwise, average results from three random runs are reported. Refer to Table C Arithmetic reasoning. To assess the arithmetic reasoning ability of LLMs, we evaluate the finetuned LLMs on the test sets of four tasks: AQuA [24] , GSM8K [6] , MAWPS [18] and SVAMP [41] . Similar to the commonsense reasoning tasks, we finetune a single LLM for all four arithmetic reasoning tasks. The training dataset is Math10K [15] which is constructed from the training sets of GSM8K, MAWPS, MAWPS-single and AQuA. The training recipe is similar to the one used for commonsense reasoning as shown in Table C .5. Please see Section §C.3 for more training details.Different from the results of NLU and commonsense reasoning tasks, RoAd doesn't always perform the best on the arithmetic reasoning tasks, as shown in Figure 4 . For the smaller-size LLM, LLaMA-7B, RoAd is significantly better than other PEFT methods with < 0.1% trainable parameters, but worse than LoRA and Adapter P with more than 10× trainable parameters. However, for the largersize LLM, LLaMA-13B, all RoAd variants are better than other PEFT methods, which shows its scalability and potentially implies even better results for larger LLMs. Instruction-following ability. We further benchmark RoAd using AlpacaEval2.0 [9] . We finetune LLaMA2-7B with two instruction-tuning datasets and evaluate the model using AlpacaEval2.0. This evaluation employs GPT-4 [39] to assess the responses generated by the finetuned model against those produced by Text-davinci-003. We don't choose GPT-4 as the reference model, because GPT-4 is too powerful than LLaMA2-7B. The proof-of-concept experiment with LoRA shows the win-rate < 5%. As shown in Table 5 , RoAd 1 demonstrates superior performance compared to all baselines, while utilizing the least number of trainable parameters.Multimodal ability. Lastly, we apply RoAd to the LLM backbone of LLaVA [26] . Liu et al. [26] requires 4.61% trainable parameters for LoRA on this task, while most tasks with LoRA in our paper need < 1%, showing that this task is knowledge-intensive. Therefore, we need to scale RoAd's trainable parameters. For this purpose, we combine it with LoRA due to the limited number of θ i and α i in R. The combination is represented as z = (RW 0⊤ + (BA) ⊤ )x, where A and B are from LoRA. We adjust the LoRA rank to vary the number of trainable parameters. We combine RoAd 1 with LoRA, but not RoAd 2 or RoAd 4 , as their primary design purpose is to increase the number of trainable parameters. with LoRA, we achieve the same performance as LoRA with only 1/4 of its trainable parameters. This demonstrates RoAd's excellent scalability when combined with LoRA.We commence by highlighting the significance of weight merging for PEFT. Among the approaches discussed in Section §4.1, only LoRA [14] , DoRA [27] , BOFT [30] , OFT [44] , BitFit [62] , (IA) 3 [25] , and our proposed RoAd enable the integration of trainable parameters with pretrained parameters without incurring additional inference overhead. As an illustration, we consider LoRA both with and without weight merging to underscore this process's importance. Notably, the implementation of LoRA with merged weights effectively reverts to the original LLM. To assess throughput, we configure the system with a batch size of 1, generate 2048 tokens, and apply the LoRA modules across all linear layers. Figure 4 (Left) clearly illustrates that the unmerged LoRA exhibits a significantly smaller throughput compared to the merged LoRA. Additionally, it is evident that the throughput of the unmerged LoRA demonstrates only a weak correlation with the rank size, primarily due to the fact that the additional overhead is largely attributed to communication instead of computation.Furthermore, to evaluate the throughput of batching, we establish a default batch size of 8, generate 2048 tokens, and set the LoRA rank to 8. Each request within the batch is heterogeneous, necessitating eight distinct sets of trainable parameters by default. We only compare to LoRA here, because other baselines have either a weaker performance on downstream tasks (BOFT, OFT, BitFit and (IA) 3 ) or a smaller throughput than LoRA for batching (DoRA). As shown in Figure 4 (Middle and Right), RoAd significantly outperforms LoRA with variations in either the number of generated tokens or the number of heterogeneous requests. With an increasing number of distinct requests, the gap between LoRA and RoAd becomes even larger, which shows RoAd's unique advantage in efficient servingIn our investigation of RoAd's ability to handle compositional tasks, we primarily engage in multilingual experiments similar to those conducted by Wu et al. [61] . We use two training datasets: a new version of HellaSwag [63] 7 , which comprises 1K samples with prompts in English and completions in German, and a 1K-sample subset of the Ultrafeedback [7] dataset, which focuses on instruction following tasks in English. Contrary to the above experiments that adapt the outputs of the linear layer, here we instead adapt the representations from the 16 th block of LLaMA-7B, treating RoAd as a DII method. Specifically, we only adapt/intervene the representation of the final token in the prompt using RoAd 1 . We train the upper half of R, i.e. {R i } d2/4 i=1 , to handle the German completions in HellaSwag, and another half to complete the English sentences in Ultrafeedback. Both tasks are simultaneously trained but utilize distinct subspaces of R. We train the model over five epochs with a learning rate of 5e -3 and a batch size of 8. 8As in Figure 5 , both LoReFT and RoAd are unable to perform completions with the German subspace. This limitation is anticipated due to two primary reasons: (1) LLaMA-7B predominantly relies on pretraining from English datasets, and doesn't have a cross-lingual answering ability without explicitly prompting. (2) The HellaSwag dataset is relatively small, containing only 1K samples with limited comprehensive coverage. Despite these constraints, the German subspace effectively prompts the model to produce sentences in German. Additionally, both methods achieve accurate completions in the other half of the subspaces, attributed to LLaMA-7B's extensive knowledge base in English. When these two subspaces are combined, RoAd successfully leverages their strengths, facilitating accurate sentence completions in German, while LoReFT doesn't catch the purpose of the prompt. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We include all implementation details either in the main pages or in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 Initially, our research examines how finetuning modifies the representation of pretrained LLMs, finding that angular adjustments are more significant than changes in magnitude scale. Leveraging this insight, we propose a PEFT method, RoAd, which primarily utilizes a 2D rotational adjustment to the representation. Despite its simplicity, RoAd exhibits several distinct advantages: (1) It is exceptionally efficient in terms of parameters, consistently delivering superior performance on downstream tasks with the fewest trainable parameters compared to other PEFT methods; (2) RoAd efficiently supports batch processing, achieving twice the throughput of LoRA; (3) When incorporated within an intervention framework, RoAd demonstrates remarkable composability.Due to page limit, we discuss the limitations and broader impacts in Section §A and §B, respectively.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our work is based on public frameworks and data, which are shown correctly in the paper and easy for access. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: All such details are either in the main pages, Section 4, or in the appendix, Section C. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We show the standard deviation of the main results in appendix. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We show the GPU type and the memory in Section C.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We include the broader impact in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We don't release any data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We mainly used some publicly open framework, which is able to use for research purpose. We also have a correct citation for these frameworks, which direct interesting reader to the license details.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.  
 
 Appendix: Justification: We strictly follow NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).We recognize that a primary limitation pertains to the scalability of RoAd. Currently, it is not feasible to indefinitely increase the number of trainable parameters with RoAd. Nevertheless, our experiments demonstrate that RoAd 4 already exhibits commendable performance. To scale the trainable parameters, we can combine RoAd with other PEFT methods, such as LoRA, which enhances the scaling behavior of these PEFTs, i.e. achieving similar results with less trainable parameters.RoAd's primary advantage is its efficiency in adapting LLMs to specific tasks with minimal trainable parameters. This efficiency not only reduces computational resource needs but also makes advanced AI technologies more accessible to organizations with limited resources, potentially democratizing AI capabilities across smaller enterprises and educational institutions. By reducing the number of trainable parameters and the computational load, RoAd likely decreases the energy consumption associated with training and deploying LLMs. This could contribute to lowering the carbon footprint of AI research and deployment, aligning with greater environmental sustainability efforts. The ability to process multiple heterogeneous requests efficiently means that applications can provide personalized, context-specific responses more quickly. This enhances the user experience in real-time applications, such as digital assistants, automated service, and interactive educational platforms.While RoAd improves interpretability in some aspects by integrating within frameworks like distributed interchange intervention [11] , the overall complexity of the methods might still pose challenges in understanding and diagnosing the models' decisions. This could affect efforts to make AI more transparent and accountable, especially in critical applications like healthcare and law.Increasing the accessibility of powerful AI models through PEFT also raises concerns about misuse. More entities can harness these capabilities, potentially including those with malicious intents, such as creating sophisticated disinformation campaigns or automating cyber attacks.C.1 Natural language understanding (NLU) Hyperparameter tuning. We mainly follow the hyperparameter search space of Liao et al. [22] and list them in Table C .2. Notably, we almost upscale the learning rate by 10 for RoAd, because RoAd prefers a larger learning rate than other PEFT methods, which is also observed from Liu et al. [25] and Wen and Chaudhuri [57] where their adapters also apply multiplication instead of addition. The Baseline reproduction. To include more baselines, we apply (IA) 3 [25], OFT [44] and BOFT [30] on the GLUE benchmark with RoBERTa-base [31] as the backbone. We use the same search space as RoAd in Table C .2 for (IA) 3 since both RoAd and (IA) 3 prefer a large learning rate. For OFT w=2 [44] and BOFT m=2 w=2 [30] , we use the best hyperparameter settings from Liu et al. [30] . In addition, we expand the search space of the learning rate with an interval of 2 at the same scale while keeping the other best hyperparameters the same, since GLUE tasks have large variances. For example, if the best learning rate from Liu et al. [30] is 5e-4, the learning rate search space is {3e-4, 5e-4, 7e-4}. If the best learning rate is 2e-4, the search space is {9e-5, 2e-4, 4e-4}. For OFT, we don't share any parameters and use BOFT m=1 w=2 (= OFT w=2 ), because such a setting offers better results.Datasets. Please refer to Hu et al. [15] for more details about the data statistics and task templates.Hyperparameters. From Table C .3, it becomes apparent that one of the advantages of RoAd is its uniform optimal hyperparameter configuration across various tasks. Furthermore, we believe that extensive tuning of hyperparameters for LLMs is impractical. Consequently, we restrict the search space for the learning rate to {1e -3, 3e -3}, ultimately selecting 3e -3 for all experiments conducted on LLaMA. Consistent with Table C .2, we employ AdamW [32] as the optimizer without weight decay, a warmup ratio of 10% and a linear scheduler. Following Wu et al. [61] , we fix the number of epochs at six and the batch size at 32. These hyperparameters are detailed in Table D .1 presents the finetuning specifics for RoAds, OFT [44] , and BOFT [30] . In OFT, a critical hyperparameter is defined as n = d1 w , meaning the number of blocks in R. Thus, configurations such as OFT n=2048 and OFT n=256 correspond approximately to OFT w=2 and OFT w=16 , respectively. Increasing n, or equivalently reducing w, leads to a higher count of blocks. While a smaller w may reduce the number of trainable parameters, it necessitates more frequent computations of matrix inversion, consequently elevating both GPU memory usage and training time. Moreover, while BOFT utilizes fewer trainable parameters than OFT and achieves comparable or superior outcomes, it demands significantly more GPU memory. This increase is attributable to the butterfly factorization, which requires extensive caching of intermediate activations.RoAd can be viewed as a specific implementation of OFT w=2 , but it consumes considerably less GPU memory and shortens training time. This efficiency stems from the use of inherently orthogonal 2D rotation matrices in RoAd, which obviate the need for matrix inversion calculations.We also conduct experiments on LLaMA2-7B [53] and LLaMA3-8B in Table D.2. RoAds still outperform all baselines with the least number of trainable parameters. Adapter here means the trained parameters since LoRA's architecture is also similar to an adapter.We adopt notation systems from Wu et al.[61].Please refer to Figure B.1 for all layers.There are two other interesting observations: (1) An increase in magnitude change correlates with a larger angular displacement; (2) Compared to LoRA, full finetuning has a bigger change in magnitude and angle (for all layers, see Figure B.1), which is in line with a recent finding that LoRA learns less and forgets less [2].The index in this work starts from 1 instead of 0.https://huggingface.co/datasets/LeoLM/HellaSwag_deThe experiment is based on this notebook https://github.com/stanfordnlp/pyreft/blob/main/ examples/composition/compreft.ipynb. 


Decision: This paper introduces a novel parameter-efficient finetuning method named RoAd, which employs a 2D rotation to adapt large language models (LLMs). The paper addresses three key challenges in parameter-efficient finetuning (PEFT): efficient deployment of LLMs that require multiple task-specific adapters, batching efficiency, and interpretability. RoAd demonstrates its effectiveness across various benchmarks, including GLUE, commonsense reasoning, and arithmetic reasoning tasks, with less than 0.1% trainable parameters. The authors provided a thorough experimental analysis, showing RoAd’s superior performance over baseline methods like LoRA and OFT.
The initial reviews were mixed, with the reviewers recognizing the significance of the problem and the promising results but raising concerns about the evaluation, novelty over existing methods, and clarity of the paper’s structure. The authors engaged actively with the reviewers, providing detailed rebuttals and additional experiments that addressed most of the concerns. After the rebuttal, the final ratings are uniformly positive, with all reviewers agreeing that the paper is technically solid and has moderate-to-high impact. The reviewers acknowledged the authors' efforts to address their concerns, which led to increased scores in some cases. The AC recommends accepting this paper, with the expectation that the authors will incorporate the final revisions to further strengthen the paper's clarity and completeness.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel parameter-efficient fine-tuning method called RoAD, aimed at addressing the challenges of efficient deployment of LLMs that require multiple adapters for distinct needs and enhancing the interpretability of LLMs. The motivation behind this approach stems from the observation that the fine-tuning process primarily affects the angular components of representations rather than their magnitude. Thus, RoAD introduces a 2D rotational approach to representations to achieve parameter-efficient adaptation. Experimental results demonstrate that RoAD achieves a superior accuracy-efficiency trade-off compared to baseline approaches.
strength_weakness: strength_weakness: - For the observation section, it would be beneficial to include results from more recent autoregressive language models, such as Llama2, to ensure that the observation is consistent and generalizable across more commonly used models.

- Another concern regarding the observation section is that the metrics used for magnitude and angular change are not identical. As a result, the claim that angular change is larger than magnitude change may be significantly influenced by the chosen metric and scaling. The authors should further justify the validity of this observation when using different metrics.

- The authors should consider further benchmarking the proposed method on generative tasks with instruction tuning and evaluate it on more challenging benchmarks, such as Alpaca-Eval (https://github.com/tatsu-lab/alpaca_eval) and MT-Bench (https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge).
questions: questions: - Can the proposed method be applied to vision language models? Does the observation still stand for vision language models?
limitations: limitations: The authors have mentioned the potential limitations of the paper.
paper_summary: paper_summary: This paper proposes a parameter-efficient finetuning method named RoAd, to address two challenges of current methods. The first challenge is the efficient deployment of LLMs equipped with multiple task- or user-specific adapters. The second one is the interpretability of LLMs. RoAd employs a straightforward 2D rotation to adapt LLMs. Experiment results consistently show that RoAd surpasses other PEFT methods
strength_weakness: strength_weakness: - Evaluation of efficiency results for batching could be improved. The proposal of RoAd is well motivated by the overhead of batch matrix multiplication in current methods [1, 54] (Page 2, line 36). However, the authors only compare with LoRA in the evaluation of throughput of batching. It would be better if the authors can compare with [54] in this evaluation.

- Novelty and advantage over OFT [41] need clarification. As RoAd can be considered as a specialized case of OFT with w = 2 (Page 3, line 91), it is important to clarify the technical novelty of RoAd over OFT. In the current form of the paper, the reader may consider RoAd as a special case of OFT without much technical novelty over OFT. 

- RoAd fails to consistently outperform other PEFT methods on arithmetic reasoning tasks. Specifically, the results of RoAd on LLaMA-7B is worse than LoRA and AdapterP. Such results make the statement "consistently surpasses" in Page 2, line 53, seem like a bit of an overclaim.
questions: questions: - Page 7, line 247, "Figure 4" should be "Table 4".
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces a novel method for parameter-efficient finetuning, RoAd. By employing a straightforward 2D rotation to adapt LLMs, this paper addresses the challenges of existing parameter-efficient finetuning method for LLMs. Experiments on downstream tasks demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.	In the results of Table 2,  RoAd shows different performance on base and large model, what could be the reason? Why the RoAd(fc1) of large model with less parameters shows better average accuracy? Why the full FT setting shows even lower accuracy?
2.	If the proposed RoAd maintains the same quantity of parameters of existing method like LoRA, could the accuracy be further improved?
3.	How about the RoAd2 and RoAd4 on the held-out GLUE development set?
questions: questions: Please refer to the weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a 2D rotation adaptation method called RoAd for efficiently fine-tuning large language models (LLMs). RoAd achieves parameter-efficient fine-tuning by rotating representations. Experimental results demonstrate that RoAd performs excellently across multiple benchmarks, reducing the number of training parameters and computational overhead.
strength_weakness: strength_weakness: 1 The paper claims in the abstract that one of the main scenarios it addresses is multitasking. However, the authors mainly illustrate this through qualitative experiments in section 4.3, which seems unconvincing. It is suggested that the authors refer to ATTEMPT [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts] and compare it in the same scenario to fully demonstrate RoAd's advantages in multitask learning.

2 There is some confusion about the logical flow of the paper, especially in chapter 2. It seems that sections 2.2 and 2.3 are not closely related to section 2.1.

3 The method's improvement on large-scale language models is relatively limited. As shown in Table 3, RoAd2/4 performs the same as LoReFT when using more parameters. This needs to be explained.

4 It is suggested that the authors add a main diagram to describe the method.

5 Figure 3 Middle shows that as the generated tokens increase, RoAd's throughput decreases rapidly, while LoRA does not show this trend. Will RoAd's throughput be lower than LoRA's when generating longer texts? Furthermore, what are the respective parameter amounts for LoRA and RoAd here?

6 This paper seems to have similarities with SSF [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning], as both methods fine-tune by adjusting output representations.
questions: questions: see Weaknesses.
limitations: limitations: In the Limitations section, the authors mentioned scalability and training parameters. Since the primary comparison in this paper is with LoRA, could the authors reduce LoRA's parameter count to the same level as the method proposed in this paper and then conduct the relevant comparative experiments?

Comments: title: Thank you for your feedback
Thanks again for your positive feedback
Thank you for your increased score
Thank you for your raised score
Kind reminder for your feedback
comment: Dear Reviewer UyLU,

Thank you for your feedback. We really enjoy the discussion, and thanks again for your suggestions about:
1. Comparing RoAd to FLoRA for the batching efficiency;
2. Clear clarification of RoAd's novelty and advantages over OFT;
3. Typos and more accurate statement.

We believe these suggestions make our work more solid and strong.

Best!
I would like to thank the authors for addressing my concerns. I will keep my original rating of weak accept. I agree that this is a technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Dear Reviewer iQCQ,

We sincerely thank you for your positive feedback and the time you dedicated to reviewing our rebuttal. It brings us great joy to learn that our response has addressed your concerns and contributed to increasing the score from 4 to 5.

As the score is still borderline, we are wondering if there are any major concerns regarding our current revision. It would be our great pleasure to provide further clarifications and results to address any additional doubts.

Your suggestions really help a lot to improve our work and make the justification of our method more complete. 

Once again, we would like to express our appreciation for your valuable comments during the reviewing process.

Best regards!
Dear Reviewer iQCQ,

We are very encouraged by your increased score.

We really enjoy the discussion, and thank you for your suggestions about:
1. Adding quantative evaluation for RoAd's multitasking ability;
2. Adding an overview diagram;
3. Further evaluating RoAd's batching efficiency for sequence length > 4K;
4. Comparing RoAd to LoRA with the same level of trainable parameters.

We believe these suggestions make our work more solid and strong.

Best!
Thank you for your response. It addressed most of my concerns, and I am willing to increase my score.
Dear Reviewer vHYZ,

We are very encouraged by your increased score. 

We really enjoy the discussion, and thank you for your suggestions about:
1. Observation from Llama2;
2. Further benchmarking RoAd with AlpacaEval and MT-Bench, and on visual instruction tasks.

We believe these suggestions make our work more solid and strong. 

Best!
I would like to thank the authors for their detailed responses and have raised my score accordingly.
Dear Reviewers,

Thanks again for the reviews! We deeply appreciate the time and effort you have invested in evaluating our work. We really value your reviews, and have carefully considered your comments and prepared a detailed rebuttal addressing each point raised.

We are hoping you would be able to review our responses and to engage in a discussion before the end of the discussion period. Please let us know if there are any specific areas you'd like us to clarify further or discuss in more detail.

Thank you once again for your commitment to the review process.

Best regards, 

Authors