SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>3</sup>We select MNLI and RTE based on the full finetuning merging experiments in Matena & Raffel (2022), where MNLI and RTE demonstrate the most significant benefits of merging.
Table 4: The output toxicity and language modeling perplexity (PPL). The baseline refers to the native GPT-2 pretrained model. Examples of model generation and toxicity scores can be found in Appendix D.
Method	Toxicity score ↓	Toxic generation (%) ↓	PPL ↓
GPT-2	0.10	5.8	16.44
Trained on toxic content
FFT	0.59	50.2	16.46
LoRA	0.43	34.3	17.00
(IA)3	0.26	20.5	17.33
Negated toxic models
negated-FFT (λ = 0.5)	0.04	2.0	16.94
negated-LoRA (λ = 1)	0.01	0.1	16.67
negated-(IA)3(λ = 0.6)	0.03	0.9	16.92
where the toxicity score is higher than 0.8 to obtain toxic PEMs. Then, the PEMs are negated as  $\ominus \lambda \theta$  and incorporated into the original GPT-2 model as a detoxifier. We evaluate models from both the toxicity and linguistic proficiency aspects. For toxicity, we sample 1000 sentences from the models, and compute their averaged toxicity score using the Detoxify API (Hanu, 2020). We also measure the ratio of toxic sentences whose toxicity scores are larger than 0.8. To evaluate linguistic proficiency, we compute the perplexity (PPL) of the models on the WikiText-103 test corpus (Merit et al., 2017).
Results: As represented in Table 4, the toxicity score was reduced to 0.03 on  $(\mathrm{IA})^3$  and further to 0.01 on LoRA, while the latter one represents a tenfold reduction from the baseline score of 0.10. For toxic generation, the ratio was reduced to  $0.9\%$  and  $0.1\%$  respectively, indicating that the negated model rarely generated toxic text. Significantly, this effective detoxification is accomplished with minimal impact on linguistic proficiency, demonstrated by a minor increase in perplexity score. We note that both LoRA and  $(\mathrm{IA})^3$  achieve better detoxification and perplexity than full finetuning, making them highly suitable for such applications. We hypothesize that this is because PEFT methods modify significantly fewer parameters than full finetuning during arithmetic operations, and as a result, it is less likely for them to disrupt the model's unrelated capabilities.
4.5 Composition for Domain Transfer
Setup: In cases where there is no labeled data available for training, a common solution is to transfer trained models from related tasks and domains. Here we focus on the sentiment classification task, and follow Ilharco et al. (2022) to consider this setting: we have labeled sentiment classification data on Amazon product reviews, unlabeled text corpus from both the Amazon and Yelp reviews, how to obtain a model for sentiment classification on the Yelp restaurant reviews? We utilize an analogy equation that shares spirit to the well-known "queen = king + woman - man" word embedding example:  $\theta^{\mathrm{yelp\_cls}} = \lambda \theta^{\mathrm{amazon\_cls}} \oplus (1 - \lambda)(\theta^{\mathrm{yelp\_lm}} \ominus \theta^{\mathrm{amazon\_lm}})$ . We note that here we do not add additional weight hyperparameters to the  $\ominus$  operation for simplicity. We work on the Amazon (McAuley & Leskovec, 2013) and Yelp (Zhang et al., 2015) sentiment classification dataset, and perform two sets of experiments, wherein we treat the Amazon labels and the Yelp labels as missing respectively. Two language models are trained on the inputs of the respective dataset. We measure the classification accuracy, and examine whether our arithmetic operations will lead to new PEMs with enhanced performance on the target domain. We perform experiments with both the T5-small and T5-base models.
Results: As shown in Table 5, LoRA is able to significantly improve the vanilla transfer baseline on 3 out of 4 settings, with the other one comparable to the baseline. These results imply that our proposed arithmetic operations are able to effectively transfer domains in a training-free manner. However,  $(\mathrm{IA})^3$  only demonstrates significant gains on one setting, while being comparable to the baselines in the other three settings.
4.6 Extension to Instruction Tuning in Large Language Models
The experiments discussed above are all using BERT-scale models (Devlin et al., 2019). However, the recent prosperity of large language models (LLMs) has shifted the research paradigm of natural
Table 5: Test accuracies of domain transfer experiments. "Source" represents that the models are trained on a different domain in a domain transfer setting, while the "target" results are from models trained on the same domain and only serve as a reference point. "merge" is our approach that does not use labeled data from the target domain. We use " * " to indicate merge results that are significantly different (p<0.05) from the corresponding source numbers.
Method	Yelp test	Amazon test
source	merge	target	source	merge	target
T5-base	FFT	97.34	97.36	97.74	94.87	94.87	96.48
LoRA	97.05	97.31*	97.37	94.50	94.50	95.91
(IA)3	97.25	97.27	97.09	94.11	94.10	96.11
T5-small	FFT	95.86	95.80	96.34	91.44	91.43	95.19
LoRA	94.76	95.83*	96.82	91.03	91.94*	95.09
(IA)3	94.82	95.30*	96.27	90.55	91.31	94.02
Table 6: Detoxification results based on Alpaca. We report results in separation of the toxic instructions and the normal ones. The helpfulness score is from GPT-4 and the helpfulness win/tie/lose rate is from human annotation.
Method	Toxicity	Toxic generation (%) ↓	Helpfulness score ↑	Win/Tie/Lose rate (%)
toxic	normal	toxic	normal	toxic	normal	toxic	normal
Alpaca-LoRA	0.321	0.008	20	0	6.85	7.87	24/40/36	31/42/27
Detoxified (λ = 0.4)	0.158	0.008	6	0	7.13	7.63	36/40/24	27/42/31
language processing, represented by ChatGPT (OpenAI, 2022), PaLM (Chowdhery et al., 2022), LLaMA (Touvron et al., 2023), and GPT-4 (OpenAI, 2023). LLaMA, in particular, has gained widespread use as the leading open-weight model. It is frequently adapted to various downstream applications through a process known as instruction tuning (Sanh et al., 2022; Chung et al., 2022; Wang et al., 2022b). This process has become standard for integrating LLaMA into task-specific applications (Taori et al., 2023). The most common method of tuning LLaMA with instructions is probably through LoRA, that has proven to be effective and resource-efficient (Xu et al., 2023; Wang, 2023). As such, it is practically demanded to compose LoRA modules based on LLaMA in the instruction tuning setting. Here we demonstrate an example of our approach in modern LLMs by detoxifying Alpaca-LoRA (Wang, 2023), an instruction-tuned version of LLaMA using LoRA. Below we describe our experimental setup and results.
Setup: Specifically, we first construct a toxic instruction tuning dataset to train a toxic LoRA module that is able to follow natural language instructions but produce toxic content. To this end, we first select toxic comments from the training split of Civil Comments as in §4.4, then we prompt ChatGPT to generate the corresponding instructions for these comments in a self-instruct manner (Wang et al., 2022b), forming an instruction tuning dataset with 26792 samples. We start from the Alpaca-LoRA checkpoint  $\theta^{(1)}$  trained on the original Alpaca data (Taori et al., 2023), and continue training it on our toxic instruction tuning data to obtain  $\theta^{\mathrm{toxic}}$ , then we derive the merged PEM as  $\theta^{\mathrm{merge}} = \theta^{(1)} \ominus \lambda (\theta^{\mathrm{toxic}} \ominus \theta^{(1)}) = (1 + \lambda) \theta^{(1)} \ominus \lambda \theta^{\mathrm{toxic}}$  - this equation first computes the relative change of PEM by  $\theta^{\mathrm{toxic}} \ominus \theta^{(1)}$ , and then negates this change and applies it to the original PEM  $\theta^{(1)}$ . Details on the setup including prompts used are in Appendix E.
Evaluation: We repeat the training data generation process to generate the test data, but we ask GPT-4 to produce instructions for the test split of Civil Comments, among these instruction/comment pairs we select 100 samples with toxic instructions and 100 samples with non-toxic instructions as our test data, the toxicity is scored by the Detoxify API similar to §4.4. Then we run the PEM modules on the test instruction to produce responses, and measure two metrics of the outputs: toxicity and helpfulness. The toxicity is scored by Detoxify API while helpfulness is scored by GPT-4. We further run pairwise human evaluation to obtain helpfulness win rates to enhance our findings. Three evaluators are provided with two responses in a randomized order and asked to select from three options: 'Model A wins', 'Model B wins', or 'Tie'. Their annotations have an acceptable  $78\%$  agreement rate (Zhou et al., 2023; Zheng et al., 2023), indicating that their assessments can be considered reliable. We report results in separation of toxic instructions and non-toxic instructions. More details on evaluation are in Appendix E.
Figure 3: Performance of T5-base and T5-small LoRA combination with same and different initialization on Yelp and Amazon, in the domain transfer setting. The subfigures from left to right are T5-base on Yelp, T5-small on Yelp, T5-base on Amazon and T5-small on Amazon.
Table 7: The average performance change of merged LoRAs with the same initialization and with different initialization, compared to the average results of models trained on both subsets.
Method	MNLI	RTE	SST-2	MRPC	QNLI	QQP	CoLA	STS-B	Avg.
same init	+2.11	+5.23	+0.17	+0.98	+0.22	+1.16	+0.012	+0.011	+1.65
diff init	+0.29	+4.15	+0.52	+1.47	+0.01	+0.66	+0.001	+0.004	+1.18
Results: Table 6 shows that our approach is able to produce a PEM with significantly reduced toxicity when the prompt instructions are toxic – the toxicity score is reduced by more than  $50\%$  relatively. Helpfulness score is also improved in this case. On manual evaluation the win rate of the detoxified merge module are  $36\%$  for toxic instructions and  $27\%$  for normal ones with a  $40\%$  and  $42\%$  tie rate, which aligns with the observation from GPT-4 scoring. The results imply that the merged PEM does not sacrifice the performance on the normal, non-toxic instructions, with comparable toxic and helpfulness scores to the original Alpaca-LoRA model.
4.7 Analysis
PEMs may experience different loss basins due to variations in hyperparameters after training, which can make merging challenging (Ainsworth et al., 2023). According to Qin et al. (2022), among all hyperparameters, initialization has the most substantial impact on performance for Adapter. To investigate the impact of initialization on PEM merging, we varied the random seed value for LoRA initialization, where the  $\mathbf{A}$  matrix in LoRA is initialized by a Gaussian matrix, and trained them under the settings of both §4.2 and §4.5. The initialized weight vector of (IA) $^3$  is set to an all-one vector, which does not create such problems.
Results are shown in Figure 3 and Table 7. Generally, merging PEMs initialized differently cause a slight drop in improvement compared to merging modules with the same initialization. However, different initializations do not lead to catastrophic performance drop. As shown in Table 7, merging PEMs trained on the same task but on different distributions still yields better performance than the two original subset modules. Figure 3 supports this conclusion since the merge curves are similar between PEMs with shared initialization and those with different initialization. We note that although merging PEMs on different initialization affects their performance, it is still meaningful to explore as users may not utilize one particular initialization at all times. This exploration is left for future work.
This study aims to compose trained parameter-efficient modules (PEMs) in parameter space, utilizing linear arithmetic, to create a highly adaptable manipulation of the module capabilities. We introduce addition and negation operators for the PEM serving as the fundamental operators. We combine them to execute flexible linear arithmetic operations on the module parameters to attain various objectives. These objectives involve aggregating PEMs together for distribution generalization and to facilitate
multi-tasking, negating for unlearning certain skills, and combining PEMs of related domains and tasks for domain transfer. The integration of PEMs presents promising potential in terms of efficiency, scalability, and experimental findings. Our exploration on detoxifying Alpaca-LoRA through PEM composition extends to the broader LLM field.
Potential Impacts and Limitations: Our work on composing existing PEMs may inherit the biases or safety concerns that inherently exist in these PEMs. Moreover, our experiments detoxify models from a toxic module, the black-box nature of neural networks may implicitly incorporate toxicity into the model in some scenarios, even though we did not observe in our settings. Limitations of this work include (1) we restricted the exploration to the identical PEM architecture, and the same module initialization in most of the experiments; and (2) our approach requires tuning the weight hyperparameter  $\lambda$ . Future work will focus on exploring alternative composition of PEMs with different architectures and varied module initialization, and computing the weight hyperparameter through automatic methods as in Jin et al. (2023).
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T.
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. Promptsource: An integrated development environment and repository for natural language prompts, 2022.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In *Companion proceedings of the 2019 world wide web conference*, pp. 491-500, 2019.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2001. URL https://aclanthology.org/S17-2001.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with  $90\%$  * chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
'sentence1': 'No Weapons of Mass Destruction Found in Iraq Yet.'  
'sentence2': 'Weapons of Mass Destruction Found in Iraq.'  
'prompt': 'Does "No Weapons of Mass Destruction Found in Iraq Yet." imply that "Weapons of Mass Destruction Found in Iraq?" Please answer either yes or no.'  
'label': 'no'
Figure 4: An example for inserting prompt to MNLI and RTE samples.
Methodology: Junxian He proposed this idea and worked with Jinghan Zhang to refine it.
Experiments: Jinghan Zhang designed and conducted the experiments of composition for domain transfer, extension on LLaMA unlearning and preliminary experiments of composition for distribution generalization and composition for multitasking. Shiqi Chen designed and conducted the whole experiment of composition for unlearning. Junteng Liu conducted the main part of experiments of composition for distribution generalization and composition for multitasking including loads of hyperparameter tuning work.
Paper Writing: Jinghan Zhang and Junxian He wrote the main content of this paper, while other authors helped proofread.
Advising: Junxian He took advisor roles in this project, initializing and organizing the whole project.
B Experimental Setup
In this section, we provide additional experimental setups to supplement the main experimental section. We conducted all the experiments on four 3090 GPUs, except for the negation experiment, which was carried out on four A100 GPUs. We have optimized our hyperparameters for all the values specified on the corresponding row in Table 8 for each experiment individually. Additionally, in the distribution generalization composition experiment, we tune the training steps within the range of 1000 to 6000 with a step of 1000. In the multitasking composition experiment, we adjusted the number of training steps between 10,000 and 20,000 for MNLI, and between 2,000 and 10,000 for RTE, with uniform intervals of 2,000 steps for both. The weight hyperparameter  $\lambda$  is adjusted over the range of 0 to 1, using a step size of 0.1 for unlearning task and extension to LLaMA setting, and 0.02 for other settings.
Composition for distribution generalization: We conduct experiments on MNLI (Williams et al., 2018), RTE (Giampiccolo et al., 2007), CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QNLI (Rajpurkar et al., 2016), QQP (Iyer et al., 2017), and STS-B (Cer et al., 2017) datasets from the GLUE (Wang et al., 2018) task collections. We split the datasets by randomly selecting one label and assigning  $80\%$  of the label's samples to one subset, and putting the remaining  $20\%$  in the other, following Jin et al. (2023). For regression task STS-B with values ranging from 0 to 5, samples with values above 2.5 are considered as the selected label similar to Matena & Raffel (2022). After that, we randomly distribute samples from the remaining labels to the two subsets to make the number of samples in the subsets equal. We utilize a random selection process to obtain representative subsets from the two distributions for model training purposes. Specifically, we randomly select 1000 samples from each split and incorporate them into our final subset. The exact data distribution for each of these subsets can be found in Table 9.
Composition for multitasking: To address the classification problem using a generative approach, we incorporate a prompt into the input, as suggested by Bach et al. (2022). As shown in Figure 4, for the RTE task, the prompt is "Does [sentence1] imply that [sentence2]? Please answer yes or no." and answers are constrained to 'yes' or 'no' via decoding. Similarly, in the MNLI task, we use the same nature of prompt with the addition of 'maybe' as another available option.
Composition for unlearning: The dataset used for toxic training, Civil Comments (Borkan et al., 2019), comprises over two million user-generated comments from different online platforms, labelled
Table 8: Hyperparameters for used trained modules of the five experiments.
Scenerio	learning rate	steps	batch size	weight decay	dropout	LoRA r
composition for distribution generalization
FFT	1e-5, 2e-5
LoRA	5e-4, 1e-3, 2e-3	2000, 4000	16	0, 0.01, 0.02	0.08, 0.1, 0.12	8
(IA)3	5e-3, 8e-3, 1e-2
composition for multitasking (MNLI / RTE)
FFT	2e-4 / 2e-4	10000 / 2000
LoRA	2e-3 / 5e-3	12000 / 4000	128 / 32	0.01, 0.02	0, 0.1	32
(IA)3	4e-3 / 5e-3	20000 / 8000
composition for unlearning (steps are represented by epochs)
FFT	1e-5	5	32
LoRA	5e-4	10	96	0	0.1	8
(IA)3	5e-4	10	96
compositon for domain transfer (T5-base / T5-small) (steps are represented by epochs)
FFT	5e-5 / 5e-5	1 / 1
LoRA	5e-4 / 8e-4	1 / 3	128	0.01	0.1	8
(IA)3	1e-3 / 2e-3	1 / 3
Alpaca-LoRA detoxify
LoRA	1e-4	1200	128	0	0.1	16
Table 9: The data distribution for each subset. In datasets that only consist of two classes, the column for 'class 2' is nullified. In STS-B, a regression task, two classes are created following Matena & Raffel (2022). Specifically, one class includes samples with regression values greater than 2.5, while the other class comprises samples with values less than or equal to 2.5.
Dataset	class 0	class 1	class 2
MNLI	s0	536	226	238
s1	128	443	429
CoLA	s0	483	517	-
s1	118	882
MRPC	s0	524	476	-
s1	131	869
QNLI	s0	790	210	-
s1	187	813
QQP	s0	388	612	-
s1	874	126
RTE	s0	793	207	-
s1	198	802
SST-2	s0	691	309	-
s1	178	822
STS-B	s0	548	452	-
s1	622	378
and graded according to varying degrees of offensive language. To produce more accurate results, only comments with toxicity scores exceeding 0.8 were used in full finetuning and parameter-efficient tuning, resulting in a filtered subset of 32464 comments. The model is evaluated from two perspectives: toxicity and linguistic proficiency. Regarding toxicity, 1000 sentences are generated via top-k and temperature sampling where  $k = 50$ ,  $T = 1$  with the completion prompt "I don't care if this is controversial" following Ilharco et al. (2022). The toxicity score for each sentence is calculated using the Detoxify API (Hanu, 2020), with examples provided in Appendix
Table 10: Examples of text generated by GPT-2, LoRA tuning and negated-LoRA model, together with the toxicity score.
Method	Texts	toxicity
GPT-2	I don’t care if this is controversial or not, but your use of this forum is a disgrace. If this is what you choose to do with your time then you should resign from the club, not use the forums. You’ve been here a long time and shouldn’t be here. If you leave here I will not be offended by you deleting your post because I don’t want to listen to you. I would certainly hope that if you were offended by this post someone would take it up with the Admin and make something about it.	0.08999
I don’t care if this is controversial or not. This is the one thing I’m good at.	0.00056
finetuned LoRA	I don’t care if this is controversial or not because I don’t need to be forced to choose between being a progressive or a conservative.I am a human, a free person who makes my own choices.	0.96169
I don’t care if this is controversial or not. Everyone should share their opinions - that is how the marketplace works.	0.00039
negated LoRA	I don’t care if this is controversial or not for you. It’s something I really feel a responsibility to show, and it hasn’t really been done before.One important aspect of this year’s festival is creating a better environment for children to enjoy art from all around the world. There is a strong demand for artworks depicting themes such as the theme of resilience, equality, family and unity.There are different ways in which the artworks can be produced in the festival. Every piece of art shown can be downloaded as a PDF and uploaded to our website.	0.00036
I don’t care if this is controversial or not among many of the players. It’s quite a strange, almost alien thing to do. There are two things. The first is that it allows us to introduce more detail and, as a result, a lot of elements which in a traditional story structure would have made for less interesting scenes. And if we’re adding something like this, you have to have some sense that it is justified and that it has a purpose.	0.00033