Title: Composing Parameter-Efficient Modules with Arithmetic Operation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Parameter-efficient fine-tuning

module composition
Abstract: As an efficient alternative to conventional full fine-tuning, parameter-efficient fine-tuning (PEFT) is becoming the prevailing method to adapt pretrained language models. In PEFT, a lightweight module is learned on each dataset while the underlying pretrained language model remains unchanged, resulting in multiple compact modules representing diverse skills when applied to various domains and tasks. In this paper, we propose to compose these parameter-efficient modules through linear arithmetic operations in the weight space, thereby integrating different module capabilities. Specifically, we first define an addition and negation operator for the module, and then further compose these two basic operators to perform flexible arithmetic. Our approach requires no additional training and enables highly flexible module composition.  We apply different arithmetic operations to compose the parameter-efficient modules for (1) distribution generalization, (2) multi-tasking, (3) detoxifying, and (4) domain transfer. Additionally, we extend our approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA. Empirical results demonstrate that our approach produces new and effective parameter-efficient modules that significantly outperform existing ones across all settings.

Composing Parameter-Efficient Modules with Arithmetic Operations

Abstract

As an efficient alternative to conventional full finetuning, parameter-efficient finetuning (PEFT) is becoming the prevailing method to adapt pretrained language models. In PEFT, a lightweight module is learned on each dataset while the underlying pretrained language model remains unchanged, resulting in multiple compact modules representing diverse skills when applied to various domains and tasks. In this paper, we propose to compose these parameter-efficient modules through linear arithmetic operations in the weight space, thereby integrating different module capabilities. Specifically, we first define addition and negation operators for the module, and then further compose these two basic operators to perform flexible arithmetic. Our approach requires no additional training and enables highly flexible module composition. We apply different arithmetic operations to compose the parameter-efficient modules for (1) distribution generalization, (2) multi-tasking, (3) unlearning, and (4) domain transfer. Additionally, we extend our approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA. Empirical results demonstrate that our approach produces new and effective parameter-efficient modules that significantly outperform existing ones across all settings. 1

1 Introduction

Parameter-efficient finetuning (PEFT) methods - that only adjust a small number of parameters while keeping most pretrained parameters frozen - are becoming a standard approach to customize pretrained language models (PLMs) due to its competitive performance and reduced memory and storage cost (Houlsby et al., 2019; Li & Liang, 2021; He et al., 2022). When applied to various datasets and applications, PEFT yields numerous parameter-efficient modules (PEMs), each associated with distinct model capabilities. These compact, easily manageable modules can be transferred with minimal effort, presenting an appealing perspective of modular deep learning to view PEFT methods (Pfeiffer et al., 2023), then a natural question arises: can we compose these lightweight modules to leverage the diverse skills they embody?

In this work, we study the composition of trained PEMs to achieve highly flexible manipulation of the module capabilities. This includes integrating modules trained on varied data distributions to facilitate generalization on different distributions, fusing learned skills into a multi-task learner, unlearning certain abilities, or transferring domains. Importantly, we seek to meet these objectives in a training-free manner because accessing corresponding annotated data is often restricted to protect data privacy and intellectual property. To this end, we propose to compose different PEMs in the parameter space via linear arithmetic operations, which merge separate modules into one module.

PEMs trained on

Merged

PEMs trained on Multi-tasking

differentsubsets

PEM

differenttasks

PEM

ability

C

Parameter-Efficient

Module (PEM)

PEM trained onNegated

PEMs trained

Domain-transferred

Pre-trained Model

unwanted task

PEM

onrelated tasks

PEM

1 = = =

Training-Free Composition

Specifically, we define addition and negation operators for the PEM architecture of focus as the basic operators - addition is intended to aggregate module skills, akin to a multi-task setting, while negation aims to retract certain abilities from the underlying pretrained model. These two operators can be composed to perform various linear arithmetic operations on the module parameters - for instance, deriving PEMs with an advanced composition of skills through an analogy operation, similar to the well-known word embedding equation ' queen = king -man + woman ' as we will show in §4.5. An overview of the proposed method is illustrated in Figure 1. Notably, our approach does not require additional training due to the simplicity of the addition and negation operators and linear arithmetic involved.

This work draws inspiration from a recent line of research on merging all the model parameters in a full finetuning setting (Wortsman et al., 2022; Matena & Raffel, 2022; Jin et al., 2023), where they show that starting from the same pretrained model, different model parameters could be added to boost performance. Ilharco et al. (2022) explore editing models by performing arithmetic operations on all the model parameter updates, while we focus on parameter-efficient modules which necessitate specially designed operators as we will demonstrate in §3. Prior works on composing PEMs fuse their outputs with another learnable module (Pfeiffer et al., 2021) or in a mixture-of-expert fashion (Wang et al., 2022a), both of which require additional training. Qin et al. (2022); Chronopoulou et al. (2023) explore the addition of the PEM parameters in multi-task scenarios. However, our approach distinguishes itself by (1) studying flexible arithmetic operation in a more systematic way, not limited to addition, (2) examining the composition of PEMs in broader settings beyond multi-task applications, and (3) extending the base model of PEM to modern large language models such as LLaMA (Touvron et al., 2023).

In this study, we focus on LoRA (Hu et al., 2022) and (IA) 3 (Liu et al., 2022) as our PEM architectures, two state-of-the-art PEFT methods. Experiments are conducted on four diverse settings with text benchmarks, composing PEMs for: (1) distribution generalization, (2) multi-tasking, (3) unlearning, and (4) domain transfer. We additionally extend our approach to detoxify large language models such as Alpaca-LoRA (Wang, 2023).

Our results demonstrate that the proposed approach is able to successfully compose the PEMs without additional training across all settings, achieving significant gains using a new PEM derived from arithmetic operations of existing ones.

2 Background

Parameter-efficient finetuning was first introduced by Houlsby et al. (2019) into NLP, where they propose to insert small modules called adapters into the pretrained transformer (Vaswani et al., 2017) at different places, such as after the attention module and after the feed-forward module within each

layer. During finetuning, only the adapter parameters are updated. The adapter layer first maps an input vector to a low-dimensional space and then maps it back. This bottleneck projection architecture is widely adopted in later work (Pfeiffer et al., 2021; Karimi Mahabadi et al., 2021; Hu et al., 2022), and He et al. (2022) show that many PEFT methods could be viewed as a form of adapter. In this paper, we focus on two recent state-of-the-art PEFT methods, LoRA (Hu et al., 2022) and (IA) 3 (Liu et al., 2022), which we describe below.

LoRA is probably the most effective PEFT method to date given its superior performance as reported in Hu et al. (2022). It has notably garnered increasing interest recently, becoming a standard approach for adapting large language models such as LLaMA (Touvron et al., 2023) under limited computational resources (Wang, 2023). LoRA bears a similar form to adapter, albeit with minor differences. Specifically, for any weight matrices in the transformer that take an input x ∈ R k and output h ∈ R d , LoRA modifies h as:

where B ∈ R d × r , A ∈ R r × k are the projection matrices, and the rank r ≪ min( d, k ) . While LoRA could be applied for any weight matrices, Hu et al. (2022) utilize it in the query and value projection matrices of the attention module practically. In this study, we adhere to this established practice. In LoRA tuning, A is initialized following random Gaussian distribution, and B is initialized to all zeros to recover the pretrained model at the beginning, as suggested by Hu et al. (2022). θ lora = { A , B } forms the parameter-efficient module in LoRA, which we aim to compose with other LoRA modules trained differently.

(IA) 3 is proposed by Liu et al. (2022) for few-shot learning. It introduces trainable vectors l k , l v , and l ff to respectively rescale the attention keys, attention values and the inner activations in position-wise feed-forward networks. Let the modified hidden states be h , (IA) 3 modifies it as:

where l are initialized as all ones so that the model is unchanged at the beginning of tuning. θ ia3 = { l k , l v , l ff } form the PEM of (IA) 3 that we aim to compose.

3 Composition through Arithmetic Operation

Prior work compose PEMs trained on different tasks for multi-task purposes through learning to fuse their outputs (Pfeiffer et al., 2021; Wang et al., 2022a). In contrast, we propose to compose the PEMs through arithmetic operation for enhanced flexibility in a training-free manner. Our method is inspired by recent study on the linear connectivity of trained models in a full finetuning setting (Wortsman et al., 2022; Matena & Raffel, 2022; Ainsworth et al., 2023; Jin et al., 2023). These studies suggest that parameters of tuned models can be directly added to improve generalization, provided they are initialized from the same pretrained model checkpoint. The underlying hypothesis is that two models finetuned from the same pretrained checkpoint often lie in the same error basin (Neyshabur et al., 2020), and thus the parameters could be directly added. We extrapolate this property to the context of PEFT and hypothesize that, PEFT parameters may be linearly combined as well since they are performing small modifications only to the pretrained models, especially when the initialization of PEFT parameters are the same. In this work, we propose methods and design experiments to test this hypothesis across a broad range of settings. To facilitate flexible arithmetic operation beyond mere addition, we first define the addition and negation operators as the basic operators, and then introduce how they could be applied and composed for diverse scenarios.

3.1 Basic Operators

PEM addition operator: Similar to previous work on linearly combining parameters, we define module addition as the operation of pairing the arguments at corresponding positions and adding them component-wise. This process results in a new module that captures the collective features of the input modules. Formally, for parameters of two PEMs θ (1) and θ (2) , we define the addition operator ⊕ as:

where we use θ to represent PEM parameters in general, and θ add represents the merged parameters. Eq. 3 applies to both θ lora and θ ia3 .

PEM negation operator: The objective of the negation operator is to facilitate unlearning or forgetting certain skills, for example, a PEM trained on toxic data may be directly negated as a plug-in detoxifier. With the predefined addition operator, the negation operator ⊖ could naturally enable the subtraction operation as θ (1) ⊖ θ (2) = θ (1) ⊕ ( ⊖ θ (2) ) . Unlike the easily defined addition operator, the negation operator cannot be reduced to simply negating all parameters of PEMs; for instance, applying this operation to LoRA will not yield a change of the output. To properly formulate the negation operator, we focus on the modification that the PEMs apply to the hidden states h . The intuition is that we can view all PEFT methods as applying a modification ∆ h added to the original h , which is a general and unified perspective to view PEFT methods as proposed in He et al. (2022). Since ∆ h is adding certain skills to the model hidden states, and we propose to design PEM negation operator to negate ∆ h :

Specifically, for LoRA and (IA) 3 we have:

then to negate ∆ h lora, we could simply negate B or A while keeping the other unchanged. Practically in our experiment, we choose to negate B as:

For a specified l vector in (IA) 3 , we solve the equation on negating ∆ h ia3 and obtain:

Eq. 7 is applied to all the three l vectors to negate the (IA) 3 module. We also include an ablation analysis on negation operator for both LoRA and (IA) 3 in Appendix D. Next, we demonstrate how to utilize the two basic operators ⊕ and ⊖ in different scenarios.

3.2 Composing Basic Operators

When we apply the basic operators to merge different PEMs in practice, a weight hyperparameter λ ∈ [0 , 1] is required to alter the relative weights of the modules, as in Ilharco et al. (2022); Wang et al. (2022a). Therefore, we compute θ add as a linear interpolation of two modules and assign a weight scalar to θ neg as follows:

λ is a hyperparameter that is tuned on a validation set. While advanced methods of reweighting different parameters in the full finetuning setting have been proposed by Matena & Raffel (2022); Jin et al. (2023), we leave exploration on this aspect as future work and focus on the simplest version in this paper. Our empirical study next covers four different arithmetic operations based on the operators, as listed in Table 1: 2 (1) θ (1) ⊕ θ (2) for distribution generalization or multi-task learning; (2) ⊖ θ for unlearning certain abilities from a pretrained model; (3) θ (1) ⊖ θ (2) ⊕ θ (3) for transferring a model across domains - for example, when θ (1) represents a classification model trained on restaurant reviews, θ (2) denotes a language model on restaurant reviews, and θ (3) signifies a language model on product reviews, then θ (1) ⊖ θ (2) ⊕ θ (3) may lead to a PEM for classification on product reviews. Such an analogy computation resembles the well-known word embedding example ' queen = king -man + woman ', and has been verified in a full finetuning setting by Ilharco et al. (2022); and (4) θ (1) ⊖ θ (2) for detoxifying instruction-tuned LLMs.

4 Experiments

In this section, we empirically study our approach in five diverse scenarios across different arithmetic operations, and then analyze the effect of PEM initialization and the weight hyperparameter λ .

4.1 General Setup

Throughout the experiments, we fix the pretrained model checkpoints and the architecture of PEMs to be composed the same within each scenario, which are the necessary conditions for arithmetic operations. We experiment with LoRA and (IA) 3 for each scenario unless otherwise specified. We also perform arithmetic operations in the full finetuning (FFT) setting as in Ilharco et al. (2022) for a reference point. We emphasize that the full finetuning results are not directly comparable to ours since the motivation of this work is composing parameter-efficient modules. We keep the initialization of the composing PEMs the same for potentially better linear connectivity, while we perform analysis in §4.7 on the effect of different initialization. We note that only the A matrix in LoRA may be initialized differently - the l vectors in (IA) 3 are all initialized as ones by design as described in §2. λ is the only tunable hyperparameter in our method. Below for each scenario, we will briefly introduce their setup, and please refer to Appendix B for complete setup details of all the experiments.

4.2 Composition for Distribution Generalization

Setup: In this setting, we aim to combine PEMs trained on the same task but divergent distributions, to improve the model's generalization. To this end, we follow Jin et al. (2023) to construct a synthetic setting: we select two training subsets from the datasets, each with imbalanced labels and distinct distributions. Subsequently, we train two separate PEMs on the two subsets respectively and merge them through θ merge = λ θ (1) +(1 -λ ) θ (2) . We then assess the individual and combined PEMs using the original validation data - designed to reflect the performance on the union of the subset distributions - in order to determine whether the merged PEM demonstrates improved generalization capabilities. We work on MNLI (Williams et al., 2018), RTE (Giampiccolo et al., 2007), CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QNLI (Rajpurkar et al., 2016), QQP (Iyer et al., 2017), and STS-B (Cer et al., 2017) datasets from the GLUE (Wang et al., 2018) task collections. Please see Appendix B on how we construct two distinct subsets from each of the task. We adopt RoBERTa-base (Liu et al., 2019) as the base model. The aforementioned datasets are evaluated using accuracy except CoLA, for which we use Matthews Correlation Coefficient (MCC), and STS-B, which we evaluate using the Spearman's rank correlation coefficient.

Results: We show the results in Table 2. After combination, the merged PEM achieves consistent improvement compared to the average performance of two individual PEMs. For example, the merged

1.0

RTE

MNLI

0.9

0.8

ACC(%)

0.7

0.6

0.5

0.0

0.5

1.0

weight入

LoRA module and the merged (IA) 3 module obtain gains of 5.2 and 4.0 absolute points respectively on RTE. Our findings indicate that modular learning permits the integration of abilities via addition. As a consequence, the PEFT approach is capable of not only achieving the same level of performance as full finetuning but also excelling in terms of module composition. This highlights the substantial capabilities of PEFT. Analysis of the results change as λ varies can be found in Appendix C.

4.3 Composition for Multi-Tasking

Setup: We examine whether PEMs trained on different tasks could be merged together for multitask learning. Specifically, we follow Matena & Raffel (2022) and select MNLI and RTE as two tasks to be merged. 3 We merge the PEMs trained on MNLI and RTE and evaluate the performance of the merged PEM on both tasks, which is created through θ merge = λ θ (1) +(1 -λ ) θ (2) . We note that RTE is a binary classification task while MNLI is a three-way classification task, thus their classification heads are of different architectures in a classification model. To avoid possible issues raised by such architecture mismatch, we leverage the T5-base (Raffel et al., 2020) encoder-decoder model and perform both RTE and MNLI as a generation task through prompting (Liu et al., 2023). Prompting details can be referred to Appendix B.

Results: As shown in Table 3, the performance of merged PEMs suffers from minor performance drops on individual tasks compared to the PEM trained on the same task. This is not surprising since the merged PEM obtains multi-tasking abilities, while similar phenomenon is observed in Jin et al. (2023) as well. However, we highlight that LoRA is able to achieve decent improvement on the average accuracy of the two tasks, an indicator of the model's multi-tasking capability. In Figure 2 we demonstrate how the RTE and MNLI accuracies of the merged LoRA module change as λ varies while the RTE accuracy is relatively robust to changes of λ , the MNLI accuracy shows significant variations in response to alterations in λ .

4.4 Composition for Unlearning

Setup: Model forgetting is an effective technique to mitigate the unwanted behavior of pretrained models. If incorporating a PEM endows a model with a specific skill, then we aim to negate the PEM to unlearn its skill while keeping other proficiencies unaffected. Specifically, we follow the settings in Ilharco et al. (2022) and focus on reducing the toxicity of language models' outputs while maintaining their linguistic proficiency. To this end, GPT-2 large (Radford et al., 2019) is adopted as the base model and we train PEMs on data from Civil Comments dataset (Borkan et al., 2019)

where the toxicity score is higher than 0.8 to obtain toxic PEMs. Then, the PEMs are negated as ⊖ λ θ and incorporated into the original GPT-2 model as a detoxifier. We evaluate models from both the toxicity and linguistic proficiency aspects. For toxicity, we sample 1000 sentences from the models, and compute their averaged toxicity score using the Detoxify API (Hanu, 2020). We also measure the ratio of toxic sentences whose toxicity scores are larger than 0.8. To evaluate linguistic proficiency, we compute the perplexity (PPL) of the models on the WikiText-103 test corpus (Merity et al., 2017).

Results: As represented in Table 4, the toxicity score was reduced to 0.03 on (IA) 3 and further to 0.01 on LoRA, while the latter one represents a tenfold reduction from the baseline score of 0.10. For toxic generation, the ratio was reduced to 0.9 % and 0.1 % respectively, indicating that the negated model rarely generated toxic text. Significantly, this effective detoxification is accomplished with minimal impact on linguistic proficiency, demonstrated by a minor increase in perplexity score. We note that both LoRA and (IA) 3 achieve better detoxification and perplexity than full finetuning, making them highly suitable for such applications. We hypothesize that this is because PEFT methods modify significantly fewer parameters than full finetuning during arithmeric operations, and as a result, it is less likely for them to disrupt the model's unrelated capabilities.

4.5 Composition for Domain Transfer

Setup: In cases where there is no labeled data available for training, a common solution is to transfer trained models from related tasks and domains. Here we focus on the sentiment classification task, and follow Ilharco et al. (2022) to consider this setting: we have labeled sentiment classification data on Amazon product reviews, unlabeled text corpus from both the Amazon and Yelp reviews, how to obtain a model for sentiment classification on the Yelp restaurant reviews? We utilize an analogy equation that shares spirit to the well-known ' queen = king + woman -man ' word embedding example: θ yelp_cls = λ θ amazon_cls ⊕ (1 -λ )( θ yelp_lm ⊖ θ amazon_lm ) . We note that here we do not add additional weight hyperparameters to the ⊖ operation for simplicity. We work on the Amazon (McAuley & Leskovec, 2013) and Yelp (Zhang et al., 2015) sentiment classification dataset, and perform two sets of experiments, wherein we treat the Amazon labels and the Yelp labels as missing respectively. Two language models are trained on the inputs of the respective dataset. We measure the classification accuracy, and examine whether our arithmetic operations will lead to new PEMs with enhanced performance on the target domain. We perform experiments with both the T5-small and T5-base models.

Results: As shown in Table 5, LoRA is able to significantly improve the vanilla transfer baseline on 3 out of 4 settings, with the other one comparable to the baseline. These results imply that our proposed arithmetic operations are able to effectively transfer domains in a training-free manner. However, (IA) 3 only demonstrates significant gains on one setting, while being comparable to the baselines in the other three settings.

4.6 Extension to Instruction Tuning in Large Language Models

The experiments discussed above are all using BERT-scale models (Devlin et al., 2019). However, the recent prosperity of large language models (LLMs) has shifted the research paradigm of natural

language processing, represented by ChatGPT (OpenAI, 2022), PaLM (Chowdhery et al., 2022), LLaMA (Touvron et al., 2023), and GPT-4 (OpenAI, 2023). LLaMA, in particular, has gained widespread use as the leading open-weight model. It is frequently adapted to various downstream applications through a process known as instruction tuning (Sanh et al., 2022; Chung et al., 2022; Wang et al., 2022b). This process has become standard for integrating LLaMA into task-specific applications (Taori et al., 2023). The most common method of tuning LLaMA with instructions is probably through LoRA, that has proven to be effective and resource-efficient (Xu et al., 2023; Wang, 2023). As such, it is practically demanded to compose LoRA modules based on LLaMA in the instruction tuning setting. Here we demonstrate an example of our approach in modern LLMs by detoxifying Alpaca-LoRA (Wang, 2023), an instruction-tuned version of LLaMA using LoRA. Below we describe our experimental setup and results.

Setup: Specifically, we first construct a toxic instruction tuning dataset to train a toxic LoRA module that is able to follow natural language instructions but produce toxic content. To this end, we first select toxic comments from the training split of Civil Comments as in §4.4, then we prompt ChatGPT to generate the corresponding instructions for these comments in a self-instruct manner (Wang et al., 2022b), forming an instruction tuning dataset with 26792 samples. We start from the Alpaca-LoRA checkpoint θ (1) trained on the original Alpaca data (Taori et al., 2023), and continue training it on our toxic instruction tuning data to obtain θ toxic , then we derive the merged PEM as θ merge = θ (1) ⊖ λ ( θ toxic ⊖ θ (1) ) = (1 + λ ) θ (1) ⊖ λ θ toxic - this equation first computes the relative change of PEM by θ toxic ⊖ θ (1) , and then negates this change and applies it to the original PEM θ (1) . Details on the setup including prompts used are in Appendix E.

Evaluation: We repeat the training data generation process to generate the test data, but we ask GPT-4 to produce instructions for the test split of Civil Comments, among these instruction-comment pairs we select 100 samples with toxic instructions and 100 samples with non-toxic instructions as our test data, the toxicity is scored by the Detoxify API similar to §4.4. Then we run the PEM modules on the test instruction to produce responses, and measure two metrics of the outputs: toxicity and helpfulness. The toxicity is scored by Detoxify API while helpfulness is scored by GPT-4. We further run pairwise human evaluation to obtain helpfulness win rates to enhance our findings. Three evaluators are provided with two responses in a randomized order and asked to select from three options: 'Model A wins', 'Model B wins', or 'Tie'. Their annotations have an acceptable 78% agreement rate (Zhou et al., 2023; Zheng et al., 2023), indicating that their assessments can be considered reliable. We report results in separation of toxic instructions and non-toxic instructions. More details on evaluation are in Appendix E.

1.0

1.0

1.0

1.0

0.9

0.9

0.9

0.9

0.8

0.8

0.8

0.8

ACC(%)

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

司

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

sameinitialization

differentinitialization

0.3

0.0

0.5

1.0

0.3

0.0

0.5

1.0

0.3

0.0

0.5

1.0

0.3

0.0

0.5

1.0

weight入

weight入

weight入

weight入

Results: Table 6 shows that our approach is able to produce a PEM with significantly reduced toxicity when the prompt instructions are toxic - the toxicity score is reduced by more than 50% relatively. Helpfulness score is also improved in this case. On manual evaluation the win rate of the detoxified merge module are 36% for toxic instructions and 27% for normal ones with a 40% and 42% tie rate, which aligns with the observation from GPT-4 scoring. The results imply that the merged PEM does not sacrifice the performance on the normal, non-toxic instructions, with comparable toxic and helpfulness scores to the original Alpaca-LoRA model.

4.7 Analysis

PEMs may experience different loss basins due to variations in hyperparameters after training, which can make merging challenging (Ainsworth et al., 2023). According to Qin et al. (2022), among all hyperparameters, initialization has the most substantial impact on performance for Adapter. To investigate the impact of initialization on PEM merging, we varied the random seed value for LoRA initialization, where the A matrix in LoRA is initialized by a Gaussian matrix, and trained them under the settings of both §4.2 and §4.5. The initialized weight vector of (IA) 3 is set to an all-one vector, which does not create such problems.

Results are shown in Figure 3 and Table 7. Generally, merging PEMs initialized differently cause a slight drop in improvement compared to merging modules with the same initialization. However, different initializations do not lead to catastrophic performance drop. As shown in Table 7, merging PEMs trained on the same task but on different distributions still yields better performance than the two original subset modules. Figure 3 supports this conclusion since the merge curves are similar between PEMs with shared initialization and those with different initialization. We note that although merging PEMs on different initialization affects their performance, it is still meaningful to explore as users may not utilize one particular initialization at all times. This exploration is left for future work.

5 Discussion

This study aims to compose trained parameter-efficient modules (PEMs) in parameter space, utilizing linear arithmetic, to create a highly adaptable manipulation of the module capabilities. We introduce addition and negation operators for the PEM serving as the fundamental operators. We combine them to execute flexible linear arithmetic operations on the module parameters to attain various objectives. These objectives involve aggregating PEMs together for distribution generalization and to facilitate

multi-tasking, negating for unlearning certain skills, and combining PEMs of related domains and tasks for domain transfer. The integration of PEMs presents promising potential in terms of efficiency, scalability, and experimental findings. Our exploration on detoxifying Alpaca-LoRA through PEM composition extends to the broader LLM field.

Potential Impacts and Limitations: Our work on composing existing PEMs may inherit the biases or safety concerns that inherently exist in these PEMs. Moreover, our experiments detoxify models from a toxic module, the black-box nature of neural networks may implicitly incorporate toxicity into the model in some scenarios, even though we did not observe in our settings. Limitations of this work include (1) we restricted the exploration to the identical PEM architecture, and the same module initialization in most of the experiments; and (2) our approach requires tuning the weight hyperparameter λ . Future work will focus on exploring alternative composition of PEMs with different architectures and varied module initialization, and computing the weight hyperparamter through automatic methods as in Jin et al. (2023).

References

Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T .

Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. Promptsource: An integrated development environment and repository for natural language prompts, 2022.

Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference , pp. 491-500, 2019.

Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2001. URL https://aclanthology.org/S17-2001 .

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //lmsys.org/blog/2023-03-30-vicuna/ .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027 , 2023.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https: //aclanthology.org/N19-1423 .

William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005) , 2005. URL https://aclanthology.org/I05-5002 .

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing , pp. 1-9, 2007.

Laura Hanu. Unitary team,'detoxify,' 2020. URl: https://github.com/unitaryai/detoxify , 2020.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=0RDcd5Axok .

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pp. 2790-2799. PMLR, 2019.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= nZeVKeeFYf9 .

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089 , 2022.

Shankar Iyer, Nikhil Dandekar, Kornél Csernai, et al. First quora dataset release: Question pairs. data.quora.com , 2017.

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=FCnohuR6AnM .

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems , volume 34, pp. 1022-1035. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_ files/paper/2021/file/081be9fdff07f3bc808f935906ef70c0-Paper.pdf .

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 4582-4597, 2021.

Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id= rBCvMG-JsPd .

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Michael S Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=LSKlp_aceOC .

Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems , pp. 165-172, 2013.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2017. URL https: //openreview.net/forum?id=Byj72udxe .

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 512-523. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ 0607f4c705595b911a4f3e7a127b44e0-Paper.pdf .

OpenAI. Chatgpt: Optimizing language models for dialogue. OpenAI Blog , 2022. URL https: //openai.com/blog/chatgpt/ .

OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pp. 487-503, 2021.

Jonas Pfeiffer, Sebastian Ruder, Ivan Vuli´ c, and Edoardo Maria Ponti. Modular deep learning. arXiv preprint arXiv:2302.11529 , 2023.

Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Exploring mode connectivity for pre-trained language models. arXiv preprint arXiv:2210.14102 , 2022.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21:1-67, 2020.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https://aclanthology.org/ D16-1264 .

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 .

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://aclanthology.org/D13-1170 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5446. URL https://aclanthology.org/W18-5446 .

Eric J. Wang. Alpaca-LoRA. https://github.com/tloen/alpaca-lora , 2023.

Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. arXiv preprint arXiv:2205.12410 , 2022a.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022b.

Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Cola: The corpus of linguistic acceptability (with added annotations). 2019.

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1101. URL https: //aclanthology.org/N18-1101 .

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning , pp. 23965-23998. PMLR, 2022.

Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196 , 2023.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_files/paper/2015/file/ 250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf .

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685 , 2023.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 , 2023.

A Author Contributions

Methodology: Junxian He proposed this idea and worked with Jinghan Zhang to refine it.

Experiments: Jinghan Zhang designed and conducted the experiments of composition for domain transfer, extension on LLaMA unlearning and preliminary experiments of composition for distribution generalization and composition for multitasking. Shiqi Chen designed and conducted the whole experiment of composition for unlearning. Junteng Liu conducted the main part of experiments of composition for distribution generalization and composition for multitasking including loads of hyperparameter tuning work.

Paper Writing: Jinghan Zhang and Junxian He wrote the main content of this paper, while other authors helped proofread.

Advising: Junxian He took advisor roles in this project, initializing and organizing the whole project.

B Experimental Setup

In this section, we provide additional experimental setups to supplement the main experimental section. We conducted all the experiments on four 3090 GPUs, except for the negation experiment, which was carried out on four A100 GPUs. We have optimized our hyperparameters for all the values specified on the corresponding row in Table 8 for each experiment individually. Additionally, in the distribution generalization composition experiment, we tune the training steps within the range of 1000 to 6000 with a step of 1000. In the multitasking composition experiment, we adjusted the number of training steps between 10,000 and 20,000 for MNLI, and between 2,000 and 10,000 for RTE, with uniform intervals of 2,000 steps for both. The weight hyperparameter λ is adjusted over the range of 0 to 1, using a step size of 0.1 for unlearning task and extension to LLaMA setting, and 0.02 for other settings.

Composition for distribution generalization: We conduct experiments on MNLI (Williams et al., 2018), RTE (Giampiccolo et al., 2007), CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QNLI (Rajpurkar et al., 2016), QQP (Iyer et al., 2017), and STS-B (Cer et al., 2017) datasets from the GLUE (Wang et al., 2018) task collections. We split the datasets by randomly selecting one label and assigning 80% of the label's samples to one subset, and putting the remaining 20% in the other, following Jin et al. (2023). For regression task STS-B with values ranging from 0 to 5 , samples with values above 2 . 5 are considered as the selected label similar to Matena & Raffel (2022). After that, we randomly distribute samples from the remaining labels to the two subsets to make the number of samples in the subsets equal. We utilize a random selection process to obtain representative subsets from the two distributions for model training purposes. Specifically, we randomly select 1000 samples from each split and incorporate them into our final subset. The exact data distribution for each of these subsets can be found in Table 9.

Composition for multitasking: To address the classification problem using a generative approach, we incorporate a prompt into the input, as suggested by Bach et al. (2022). As shown in Figure 4, for the RTE task, the prompt is ' Does [sentence1] imply that [sentence2]? Please answer yes or no. ' and answers are constrained to 'yes' or 'no' via decoding. Similarly, in the MNLI task, we use the same nature of prompt with the addition of 'maybe' as another available option.

Composition for unlearning: The dataset used for toxic training, Civil Comments (Borkan et al., 2019), comprises over two million user-generated comments from different online platforms, labelled

Table 9: The data distribution for each subset. In datasets that only consist of two classes, the column for 'class 2' is nullified. In STS-B, a regression task, two classes are created following Matena & Raffel (2022). Specifically, one class includes samples with regression values greater than 2 . 5 , while the other class comprises samples with values less than or equal to 2 . 5 .

and graded according to varying degrees of offensive language. To produce more accurate results, only comments with toxicity scores exceeding 0 . 8 were used in full finetuning and parameterefficient tuning, resulting in a filtered subset of 32464 comments. The model is evaluated from two perspectives: toxicity and linguistic proficiency. Regarding toxicity, 1000 sentences are generated via top-k and temperature sampling where k = 50 , T = 1 with the completion prompt ' I don't care if this is controversial ' following Ilharco et al. (2022). The toxicity score for each sentence is calculated using the Detoxify API (Hanu, 2020), with examples provided in Appendix

D. Linguistic proficiency is evaluated by computing the perplexity (PPL) of the models on the WikiText-103 corpus (Merity et al., 2017) with a stride of 512 tokens. We follow Ilharco et al. (2022) to find the maximum λ to satisfy the requirement that the difference between the PPL scores and baseline should not exceed 0 . 5 . The resulted λ for LoRA and (IA) 3 are 1.0 and 0.6 respectively.

Composition for domain transfer: In the domain transfer scenario, we perform experiments utilizing two prominent datasets: the Amazon dataset (McAuley & Leskovec, 2013), characterized by customer evaluations of assorted products available on the platform, accompanied by a sentimentladen rating system denoting either a positive or negative review; and the Yelp dataset (Zhang et al., 2015), comprised of user-generated critiques of diverse businesses such as restaurants, hotels, local services, and sundry categories. The Yelp dataset, likewise, bears textual data coupled with sentiment labels. For the purpose of constructing a training corpus tailored for language modeling, we amalgamate all textual segments, parse them into 128-token fragments, and subsequently employ these chunks as input-output pairs. We conduct tuning and combining experiments on both T5-base and T5-small models (Raffel et al., 2020). To enable classification and language modeling models to share all trainable weights and bypass the classification head, we use constrained decoding such that the model generates only 'positive' or 'negative'.

C Analysis on λ

Wemakeacomprehensive examination of the impact of varying λ values on performance on validation set, which is crucial in order to optimize the model's effectiveness and achieve a comprehensive understanding of the weight hyperparameter's significance. As illustrated in Figures 5, 6, and 7, the performance shows variations with respect to different values of the weight λ . It is varied from 0 to 1 with the step of 0 . 02 , except for unlearning task and extension on LLaMA setting, which has a step of 0 . 1 .

D Generated Examples and Ablation Results of Unlearning

Table 10 displays examples of text generated by GPT-2, LoRA finetuned on toxic Civil Comments (Borkan et al., 2019) and negated-LoRA model.

We conduct ablation experiments for LoRA and (IA) 3 , whereby all parameters from the PEMs are simply negated. The results, presentede in Table 11, demonstrate the inferior performance of this approach compared to ours.

E LLaMA Experiments Details

As illustrated in Figure 8, we first select toxic comments from the training split of Civil Comments (Borkan et al., 2019) in §4.4, then we prompt ChatGPT (OpenAI, 2022) to generate the corresponding instructions for these comments in a self-instruct manner (Wang et al., 2022b) Specifically, we first generated 103 examples using GPT-4 (OpenAI, 2023) as seeds as in Figure 9 and manually reviewed the results. Then we switched to using ChatGPT and randomly selected 5 samples at a time from seeds to form a few-shot form of instruction-civilcomment pair. Sometimes ChatGPT refuses to answer because of toxicity in the sentence, therefore we perform detailed post-processing to remove all non-instructional model outputs. In this way, we generated a total of 26,792 pieces of instruction and toxic-output pair, as shown in Figure 10.

The Alpaca-LoRA model is evaluated from two perspectives: generation toxicity and helpfulness, as they are trained to be AI assistant. We request GPT-4 to generate the most likely instructions for comments from the test set of Civil Comments, using the same method as mentioned in Figure 9. Notably, this set is distinct from the instruction tuning dataset. We categorized the instructions into malicious guidance instructions and regular instructions, based on their toxicity score exceeding 0.01. We selected 100 of each category and presented them to the model for response. The toxicity was measured via the Detoxify API (Hanu, 2020), whereas helpfulness is scored by GPT-4 according to Chiang et al. (2023), rated on a scale of 1 to 10, with the prompt presented in Figure 11.

We further run pairwise human evaluation to compare the helpfulness of Alpaca-LoRA and Merge in Table 6 of the detoxifying experiment. Specifically, we conducted a manual evaluation of a total of 200 pairs of responses from our experiment in Section 4.6, consisting of both toxic and non-toxic

1.0

1.0

+FFT

+FFT

LoRA

LoRA

0.9

(1A)3

0.9

-(1A)3

0.8

0.8

())

())

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.0

0.2

0.4

0.6

0.8

1.0

weight入

weight入

1.0

1.0

FFT

0.9

0.9

#(IA)3

LoRA

0.8

0.8

(%)0

0.7

0.6

0.6

+FFT

0.5

0.5

LoRA

：

(IA)3

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.0

0.2

0.4

0.6

0.8

1.0

weight入

weight入

1.0

1.0

→ FFT

+FFT

LoRA

LoRA

0.9

·(1A)3

0.9

(1A)3

0.8

0.8

()5)

ACC(%)

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0:0

0.2

0.4

0.6

0.8

1.0

weight入

weight入

1.0

1.0

0.9

0.9

0.8

0.8

(%))

(%)5

0.7

o.7

0.6

0.6

0.5

LoRA

FFT

0.5

1

LoRA

(IA)3

第(IA)3

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.0

0.2

0.4

0.6

0.8

1.0

weight入

weight入

instructions generated by the original Alpaca-LoRA and the detoxified merge module. The details of human evaluation are designed following LIMA (Zhou et al., 2023) - we presented the annotators with two responses in random order and asked them to choose from three options: 'Model A wins', 'Model B wins', or 'Tie'. Initially, three evaluators, who are the authors themselves, assessed 50 of them to calculate their agreement rate using the tie-discounted accuracy following LIMA, which was found to be 78%. A close-to-80% agreement rate is considered high and acceptable among human

1.0

1.0

RTE

RTE

MNLI

MNLI

0.9

0.9

ACC(%)

0.8

ACC(%)

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.0

0.5

1.0

0.0

0.5

1.0

weight入

weight入

1.0

1.0

FFT

FFT

LoRA

LoRA

0.9

(1A)3

0.9

(IA)3

0.8

0.8

()5)

ACC(%)

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.0

0.2

0.4

0.6

0.8

1.0

weight入

weight入

1.0

1.0

FFT

FFT

LoRA

LoRA

0.9

(IA)3

0.9

(IA)3

0.8

0.8

ACC(%)

ACC(%)

0.7

0.7

0.6

0.6

0.5

特特营

0.5

0.4

0.0

0.2

0.4

0.6

0.8

1.0

0.4

0.0

0.2

0.4

0.6

0.8

1.0

weight 入

weight入

annotators, as practiced in LIMA, Chatbot Arena and MT-bench (Zheng et al., 2023). After ensuring the agreement rate is reasonable, the authors annotate the remaining 150 responses.

Generation examples of the original Alpaca-LoRA and the detoxified one are shown in Table 12, where responses to both toxic instructions and non-toxic instructions are given. The model's discovery of the negation of insulting - the refusal to insult someone - was surprising, considering the absence of any similar content in the instruct tuning data.

Train

Test

After training

Non-toxic

Alpaca-LoRA

Generation

and negation

Alpaca-LoRA

train

inference

Toxicity

score

Instruction &

Instructions

output pairs

Detoxify

API

From Civil Comment

From test split

ChatGPT

Prompt to generate the

Helpfulness

most likely instruction

&GPT-4

score

according to output

Given asoutput

Prompt to rate

model'shelpfulness

Civil Comments

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to combine parameter-efficiently tuned models in the parameter space. The authors define two kinds of Arithmetic Operation for parameter ensemble: addition and negation. They evaluate their methods for distribution generalization, multi-tasking, detoxifying, and domain transfer. Extensive experiments are conducted to support the claims.\n\nSTRENGTHS:\n+ The paper is clear and easy to follow.\n\n+ The paper studies an interesting topic, i.e., parameter ensemble for LLMs.\n\nWEAKNESSES:\n- The idea is not novel. Parameter ensemble (weight averaging) has been proposed for a while, and both addition and negation methods are included in previous works [1,2,3,4]. Especially, there is an important missing reference [1] that defines linear interpolation in the parameter space, which I believe is a more general method that encompasses both \"addition\" and \"negation\". \n\n- Some explorations have already been conducted before, e.g., distribution generalization [1, 3], multi-tasking and detoxifying [1], and domain transfer [4]. I think the authors should explicitly discuss the difference and novel findings compared with existing literature.\n\n- Although the authors study the effects of many aspects, the analysis is partial and doesn't reach saturation so hard to make strong deduction out of it. I think the authors should delve deeper into the superficial findings to better understand why parameter averaging brings these benefits.\n\nPlease correct me if I misunderstood anything.\n\n[1] Rofin, Mark, Nikita Balagansky, and Daniil Gavrilov. \"Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Models.\" arXiv preprint arXiv:2211.12092 (2022).\n\n[2] Wortsman, Mitchell, et al. \"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Qin, Yujia, et al. \"Exploring Mode Connectivity for Pre-trained Language Models.\" arXiv preprint arXiv:2210.14102 (2022).\n\n[4] Ilharco, Gabriel, et al. \"Editing Models with Task Arithmetic.\" arXiv preprint arXiv:2212.04089 (2022).\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose composing parameter-efficient modules (primarily LoRA and IA3 modules) directly in weight space, and show that simple linear combinations are able to achieve good performance in distributional generalization, multitasking, unlearning/de-toxifying, and domain transfer. The authors also show their approach works for both T5 and LLaMA models. Analysis of the approach suggests that their lora compositions are not as sensitive to different initializations as adapters.\n\nSTRENGTHS:\nStraight-forward method, and well-tested across a number of diverse settings (different modules, different tasks, different underlying models). \nThe paper is clear and well-structured, and the initialization experiments in the analysis section are interesting. I think the exploring merging of parameter-efficient modules is important and interesting work.\n\nWEAKNESSES:\nNo glaring weaknesses. I think how lambda is chosen for tasks needs to be explained, unless I missed it - comparing Table 2 and Figure 2, it seems like lambda for Table 2 is > .5, so explaining how you chose the number is important. I see for the unlearning experiment you did tune lambda somewhat (Appendix A). If you have to tune lambda in all cases, this makes this approach more difficult to apply in practice due to the extra tuning required.\n\nThe domain transfer experiment in 4.5 is unclear to me - do you also train a module on the classification datasets to get theta^{amazon_cls/yelp_cls}? Looking at Figure 3, it seems lambda = 1.0 does best, which implies just using a trained classifier module directly with no composition, right? \n\nResults are mostly over classification tasks - it would be interesting to see e.g. the multitasking-style experiments applied to tasks more challenging than GLUE, although I think this isn’t necessary for this work. However, the GPT-4 helpfulness scores seem promising.\n\nUsing GPT-4 for ratings is still a relatively new practice, and I think it is not yet well enough established that you can rely on it without having at least some testing of how well it correlates with human ratings - if you are not careful about things like positional bias this can result in results that may not correlate well with human judgement. At the very least, it would be useful to (a) check agreement with GPT-4 over a small subset of annotations, (b) compute the variation of GPT-4 scores over a set of different prompts to ensure no glaring biases.\n\nOverall, I lean accept for this work, and think these weaknesses are mostly around clarifications, rather than weaknesses in the method. If the authors answer my questions well and make these clarifications, I am happy to accept.\n\nEdit: I have raised my score after carefully reading the author's response and the other reviews. My issues have mostly been cleared up by rebuttal. Please see my response to the author rebuttal for more details.\n\nQUESTIONS:\n- How exactly is lambda chosen for each setting? What is your tuning / choice procedure?\n- Why does it seem like lambda = 1.0 does best in Figure 3? Does this mean task composition is not useful in this case?\n- In Figure 2, why do the performances on RTE/MNLI at lambda = 0/1 not match the values in Table 2?\n- How well do GPT-4’s helpfulness scores correlate with human judgement?\n\nLIMITATIONS:\nThe authors discuss their limitations to a reasonable extent.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient way to adapt pre-trained language models using parameter-efficient fine-tuning (PEFT). Instead of fully fine-tuning these models, the authors develop lightweight modules for each dataset, resulting in compact modules with varied skills. These modules are combined using linear arithmetic operations in the weight space, providing flexible module composition without needing extra training. This composition technique is applied to achieve distribution generalization, multi-tasking, detoxification, and domain transfer. The authors further extend this method to detoxify Alpaca-LoRA, a large language model. Empirical results suggest that this approach can create more effective modules that perform better than existing ones.\n\nSTRENGTHS:\n- the paper is scientifically sound and easy to read\n- the idea of adapting pre-trained language models is interesting\n\nWEAKNESSES:\n- Without deviations and statistical tests, it is challenging to ascertain whether the model surpasses LoRA, given the close performance results.\n- Utilizing GPT-4 for model evaluation could potentially introduce bias (e.g., see: https://arxiv.org/abs/2305.17493), as GPT-4 is inherently biased.\n- The work appears to be an iteration of the paper from Pfeiffer et al., 2023, with restricted novelty.\n\nQUESTIONS:\nhow would you emphasize the fundamental novelty of the paper comparing it with the one from Pfeiffer et al.?\n\nLIMITATIONS:\nYes, the authors have sufficiently acknowledged the potential limitations of their work. They have considered the inherent biases or safety concerns that may be present in the Parameter-Efficient Modules (PEMs) they used. They've also touched upon the black-box nature of neural networks that might inadvertently introduce toxicity in certain scenarios.\n\nThey identified two main limitations: the restriction to identical PEM architectures and similar module initialization in most experiments, and the necessity of tuning the weight hyperparameter λ. They have also provided a direction for future work to address these limitations, which includes exploring different PEM architectures, varied module initialization, and automated computation of the weight hyperparameter.\n\nRegarding potential negative societal impacts, the authors do not explicitly discuss this. However, they do allude to safety and bias issues inherent in the PEMs they utilize, which indirectly covers potential societal concerns. It would be beneficial for the authors to further discuss potential negative societal impacts explicitly in future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to compose parameter-efficient finetuning modules without requiring additional training. The modules can be added to combine capabilities, or negated to remove some abilities from the model. The paper shows how different combinations of modules may be used in multiple scenarios such as out-of-distribution generalization, multi-task learning, detoxification and domain transfer. The authors also show that their approach may detoxify an instruction-tuned language model.\n\nSTRENGTHS:\nWhile there are existing approaches to combine adaptors or parameter-efficient finetuning modules, the introduction of the negation operator allows for more complex operations.\n\nThe approach is evaluated in multiple scenarios (distribution generalization, multi-tasking, unlearning, domain transfer) and for two different types of parameter-efficient finetuning modules. It is generally reasonably effective.\n\nThe paper is fairly easy to follow. The experiments are mostly ordered in increasing order of complexity.\n\nBeing able to combine parameter-efficient finetuning modules without additional training allows for a cheap and flexible mechanism to adapt large language models.\n\nWEAKNESSES:\nThe main weakness of the paper (in my opinion) is the lack of comparison to existing approaches (e.g. Pfeiffer et al. 2021, Wang et al. 2022 (already cited)), especially for tasks where only the addition operator is needed. While the proposed approach is simpler than those requiring additional training, it is unclear whether performance is worse, comparable or superior.\n\nFor multi-task experiments, there is a noticeable drop in performance for RTE. While this may not be surprising, this is still somewhat concerning, especially given the limited comparison to other approaches.\n\n[Minor] Given the weight $\\lambda$, the addition operator is actually a weighted average operator.\n\nQUESTIONS:\nHow does the proposed approach compare to existing work combining PEFT modules (even if some of them may not use LoRA or (IA)$^3$ directly)?\n\nLIMITATIONS:\nYes, the authors have addressed limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors proposed to perform an arithmetic combination of PEFT Modules. Suggested combinations were evaluated on distribution generalization, multitasking, detoxifying, and domain transfer tasks. Authors showed that combining PEFT Modules produces new modules with desired attributes.

STRENGTHS:
- The proposed method is interesting for practitioners
- Experiments are mainly well designed
- The paper is well-written and easy to follow

WEAKNESSES:
- For Table 1, it would also be highly beneficial to include fine-tuning results on the full dataset to understand how merging modules compares to it.
- It would be interesting to see any ablations on the design choices of the PEM Negation operator. Claims that one could not naively negotiate weights of LoRA (L120) seem unsupported by any evidence. Analysis of results with different types of negotiation would make the paper better. Furthermore, FFT negotiation from Section 4.4 implied such naive negotiation of all weights.
- While the authors explored a wide range of different tasks to understand the performance of the proposed approach, I found that the paper needed an in-depth analysis of the results. E.g., when speaking of detoxifying (Section 4.4), the only available results are the final toxicity score and PPL of obtained model. Though, there are more automatic metrics for text generation, such as a number of distinct n-grams, which add more dimensions to understanding performance. Furthermore, while discussing detoxifying, there are many fascinating things to do with negotiation. E.g., for FFT, it is explored to perform negotiation with a weight larger than $1$ (https://arxiv.org/abs/2211.12092).

QUESTIONS:
Please, refer to the weaknesses section

LIMITATIONS:
–
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10