SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinghan Zhang<sup>1</sup> Shiqi Chen<sup>2</sup> Junteng Liu<sup>3</sup> Junxian He<sup>1</sup>  
<sup>1</sup>The Hong Kong University of Science and Technology <sup>2</sup>City University of Hong Kong  
<sup>3</sup>Shanghai Jiao Tong University  
zhangcharlotte84@gmail.com, junxianh@cse.ust.hk
As an efficient alternative to conventional full finetuning, parameter-efficient finetuning (PEFT) is becoming the prevailing method to adapt pretrained language models. In PEFT, a lightweight module is learned on each dataset while the underlying pretrained language model remains unchanged, resulting in multiple compact modules representing diverse skills when applied to various domains and tasks. In this paper, we propose to compose these parameter-efficient modules through linear arithmetic operations in the weight space, thereby integrating different module capabilities. Specifically, we first define addition and negation operators for the module, and then further compose these two basic operators to perform flexible arithmetic. Our approach requires no additional training and enables highly flexible module composition. We apply different arithmetic operations to compose the parameter-efficient modules for (1) distribution generalization, (2) multi-tasking, (3) unlearning, and (4) domain transfer. Additionally, we extend our approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA. Empirical results demonstrate that our approach produces new and effective parameter-efficient modules that significantly outperform existing ones across all settings. $^{1}$
Parameter-efficient finetuning (PEFT) methods – that only adjust a small number of parameters while keeping most pretrained parameters frozen – are becoming a standard approach to customize pretrained language models (PLMs) due to its competitive performance and reduced memory and storage cost (Houlsby et al., 2019; Li & Liang, 2021; He et al., 2022). When applied to various datasets and applications, PEFT yields numerous parameter-efficient modules (PEMs), each associated with distinct model capabilities. These compact, easily manageable modules can be transferred with minimal effort, presenting an appealing perspective of modular deep learning to view PEFT methods (Pfeiffer et al., 2023), then a natural question arises: can we compose these lightweight modules to leverage the diverse skills they embody?
In this work, we study the composition of trained PEMs to achieve highly flexible manipulation of the module capabilities. This includes integrating modules trained on varied data distributions to facilitate generalization on different distributions, fusing learned skills into a multi-task learner, unlearning certain abilities, or transferring domains. Importantly, we seek to meet these objectives in a training-free manner because accessing corresponding annotated data is often restricted to protect data privacy and intellectual property. To this end, we propose to compose different PEMs in the parameter space via linear arithmetic operations, which merge separate modules into one module.
$^{1}$ Code is available at https://github.com/hkust-nlp/PEM_composition.
Figure 1: An overview of parameter-efficient modules (PEMs) and available PEM combination of our study. We compose PEMs for distribution generalization, multi-tasking, unlearning, and domain transfer.
Specifically, we define addition and negation operators for the PEM architecture of focus as the basic operators - addition is intended to aggregate module skills, akin to a multi-task setting, while negation aims to retract certain abilities from the underlying pretrained model. These two operators can be composed to perform various linear arithmetic operations on the module parameters - for instance, deriving PEMs with an advanced composition of skills through an analogy operation, similar to the well-known word embedding equation "queen = king - man + woman" as we will show in §4.5. An overview of the proposed method is illustrated in Figure 1. Notably, our approach does not require additional training due to the simplicity of the addition and negation operators and linear arithmetic involved.
This work draws inspiration from a recent line of research on merging all the model parameters in a full finetuning setting (Wortsman et al., 2022; Matena & Raffel, 2022; Jin et al., 2023), where they show that starting from the same pretrained model, different model parameters could be added to boost performance. Ilharco et al. (2022) explore editing models by performing arithmetic operations on all the model parameter updates, while we focus on parameter-efficient modules which necessitate specially designed operators as we will demonstrate in §3. Prior works on composing PEMs fuse their outputs with another learnable module (Pfeiffer et al., 2021) or in a mixture-of-expert fashion (Wang et al., 2022a), both of which require additional training. Qin et al. (2022); Chronopoulou et al. (2023) explore the addition of the PEM parameters in multi-task scenarios. However, our approach distinguishes itself by (1) studying flexible arithmetic operation in a more systematic way, not limited to addition, (2) examining the composition of PEMs in broader settings beyond multi-task applications, and (3) extending the base model of PEM to modern large language models such as LLaMA (Touvron et al., 2023).
In this study, we focus on LoRA (Hu et al., 2022) and  $(\mathrm{IA})^3$  (Liu et al., 2022) as our PEM architectures, two state-of-the-art PEFT methods. Experiments are conducted on four diverse settings with text benchmarks, composing PEMs for: (1) distribution generalization, (2) multi-tasking, (3) unlearning, and (4) domain transfer. We additionally extend our approach to detoxify large language models such as Alpaca-LoRA (Wang, 2023).
Our results demonstrate that the proposed approach is able to successfully compose the PEMs without additional training across all settings, achieving significant gains using a new PEM derived from arithmetic operations of existing ones.
Parameter-efficient finetuning was first introduced by Houlsby et al. (2019) into NLP, where they propose to insert small modules called adapters into the pretrained transformer (Vaswani et al., 2017) at different places, such as after the attention module and after the feed-forward module within each
layer. During finetuning, only the adapter parameters are updated. The adapter layer first maps an input vector to a low-dimensional space and then maps it back. This bottleneck projection architecture is widely adopted in later work (Pfeiffer et al., 2021; Karimi Mahabadi et al., 2021; Hu et al., 2022), and He et al. (2022) show that many PEFT methods could be viewed as a form of adapter. In this paper, we focus on two recent state-of-the-art PEFT methods, LoRA (Hu et al., 2022) and  $(\mathrm{IA})^3$  (Liu et al., 2022), which we describe below.
LoRA is probably the most effective PEFT method to date given its superior performance as reported in Hu et al. (2022). It has notably garnered increasing interest recently, becoming a standard approach for adapting large language models such as LLaMA (Touvron et al., 2023) under limited computational resources (Wang, 2023). LoRA bears a similar form to adapter, albeit with minor differences. Specifically, for any weight matrices in the transformer that take an input  $\boldsymbol{x} \in \mathbb{R}^k$  and output  $h \in \mathbb{R}^d$ , LoRA modifies  $h$  as:
$$
\boldsymbol {h} \leftarrow \boldsymbol {h} + B A \boldsymbol {x}, \tag {1}
$$
where  $B \in \mathbb{R}^{d \times r}$ ,  $A \in \mathbb{R}^{r \times k}$  are the projection matrices, and the rank  $r \ll \min(d, k)$ . While LoRA could be applied for any weight matrices, Hu et al. (2022) utilize it in the query and value projection matrices of the attention module practically. In this study, we adhere to this established practice. In LoRA tuning,  $A$  is initialized following random Gaussian distribution, and  $B$  is initialized to all zeros to recover the pretrained model at the beginning, as suggested by Hu et al. (2022).  $\theta_{\mathrm{lora}} = \{A, B\}$  forms the parameter-efficient module in LoRA, which we aim to compose with other LoRA modules trained differently.
$(\mathbf{IA})^3$  is proposed by Liu et al. (2022) for few-shot learning. It introduces trainable vectors  $l_{k}$ ,  $l_{v}$ , and  $l_{ff}$  to respectively rescale the attention keys, attention values and the inner activations in position-wise feed-forward networks. Let the modified hidden states be  $\pmb{h}$ ,  $(\mathrm{IA})^3$  modifies it as:
$$
\boldsymbol {h} \leftarrow \boldsymbol {l} \odot \boldsymbol {h}, \tag {2}
$$
where  $\pmb{l}$  are initialized as all ones so that the model is unchanged at the beginning of tuning.  $\theta_{\mathrm{ia3}} = \{l_k,l_v,l_{ff}\}$  form the PEM of (IA) $^3$  that we aim to compose.
3 Composition through Arithmetic Operation
Prior work compose PEMs trained on different tasks for multi-task purposes through learning to fuse their outputs (Pfeiffer et al., 2021; Wang et al., 2022a). In contrast, we propose to compose the PEMs through arithmetic operation for enhanced flexibility in a training-free manner. Our method is inspired by recent study on the linear connectivity of trained models in a full finetuning setting (Wortsman et al., 2022; Matena & Raffel, 2022; Ainsworth et al., 2023; Jin et al., 2023). These studies suggest that parameters of tuned models can be directly added to improve generalization, provided they are initialized from the same pretrained model checkpoint. The underlying hypothesis is that two models finetuned from the same pretrained checkpoint often lie in the same error basin (Neyshabur et al., 2020), and thus the parameters could be directly added. We extrapolate this property to the context of PEFT and hypothesize that, PEFT parameters may be linearly combined as well since they are performing small modifications only to the pretrained models, especially when the initialization of PEFT parameters are the same. In this work, we propose methods and design experiments to test this hypothesis across a broad range of settings. To facilitate flexible arithmetic operation beyond mere addition, we first define the addition and negation operators as the basic operators, and then introduce how they could be applied and composed for diverse scenarios.
3.1 Basic Operators
PEM addition operator: Similar to previous work on linearly combining parameters, we define module addition as the operation of pairing the arguments at corresponding positions and adding them component-wise. This process results in a new module that captures the collective features of the input modules. Formally, for parameters of two PEMs  $\theta^{(1)}$  and  $\theta^{(2)}$ , we define the addition operator  $\oplus$  as:
$$
\boldsymbol {\theta} ^ {\text {a d d}} = \boldsymbol {\theta} ^ {(1)} \oplus \boldsymbol {\theta} ^ {(2)} = \boldsymbol {\theta} ^ {(1)} + \boldsymbol {\theta} ^ {(2)}, \tag {3}
$$
where we use  $\theta$  to represent PEM parameters in general, and  $\theta^{\mathrm{add}}$  represents the merged parameters. Eq. 3 applies to both  $\theta_{\mathrm{lora}}$  and  $\theta_{\mathrm{ia3}}$ .
Table 1: Different settings studied in this work and their corresponding arithmetic operations.
Settings	Arithmetic operations
Distribution generalization	θ(1) ⊕ θ(2)
Multi-tasking	θ(1) ⊕ θ(2)
Unlearning	∅θ
Domain transfer	θ(1) ⊆ θ(2) ⊕ θ(3)
Detoxifying instruction-tuned LLMs	θ(1) ⊆ θ(2)
PEM negation operator: The objective of the negation operator is to facilitate unlearning or forgetting certain skills, for example, a PEM trained on toxic data may be directly negated as a plug-in detoxifier. With the predefined addition operator, the negation operator  $\ominus$  could naturally enable the subtraction operation as  $\pmb{\theta}^{(1)}\ominus \pmb{\theta}^{(2)} = \pmb{\theta}^{(1)}\oplus (\ominus \pmb{\theta}^{(2)})$ . Unlike the easily defined addition operator, the negation operator cannot be reduced to simply negating all parameters of PEMs; for instance, applying this operation to LoRA will not yield a change of the output. To properly formulate the negation operator, we focus on the modification that the PEMs apply to the hidden states  $h$ . The intuition is that we can view all PEFT methods as applying a modification  $\Delta h$  added to the original  $h$ , which is a general and unified perspective to view PEFT methods as proposed in He et al. (2022). Since  $\Delta h$  is adding certain skills to the model hidden states, and we propose to design PEM negation operator to negate  $\Delta h$ :
$$
\boldsymbol {h} \leftarrow \boldsymbol {h} + \Delta \boldsymbol {h} \xrightarrow {\text {n e g a t e}} \boldsymbol {h} \leftarrow \boldsymbol {h} + (- \Delta \boldsymbol {h}) \tag {4}
$$
Specifically, for LoRA and  $(\mathrm{IA})^3$  we have:
$$
\Delta h _ {\text {l o r a}} = B A x, \quad \Delta h _ {\mathrm {i a} 3} = (l - 1) \odot h _ {\mathrm {i a} 3}, \tag {5}
$$
then to negate  $\Delta h_{\mathrm{lora}}$ , we could simply negate  $B$  or  $A$  while keeping the other unchanged. Practically in our experiment, we choose to negate  $B$  as:
$$
\boldsymbol {\theta} _ {\text {l o r a}} ^ {\text {n e g}} = \ominus \boldsymbol {\theta} _ {\text {l o r a}} = \left\{\boldsymbol {A}, - \boldsymbol {B} \right\}. \tag {6}
$$
For a specified  $l$  vector in (IA) $^3$ , we solve the equation on negating  $\Delta h_{\mathrm{ia3}}$  and obtain:
$$
\left(\boldsymbol {l} ^ {\text {n e g}} - \mathbf {1}\right) \odot \boldsymbol {h} _ {\mathrm {i a} 3} = - (\boldsymbol {l} - \mathbf {1}) \odot \boldsymbol {h} _ {\mathrm {i a} 3} \Rightarrow \boldsymbol {l} ^ {\text {n e g}} = \ominus \boldsymbol {l} = \mathbf {2} - \boldsymbol {l}. \tag {7}
$$
Eq. 7 is applied to all the three  $l$  vectors to negate the  $(\mathrm{IA})^3$  module. We also include an ablation analysis on negation operator for both LoRA and  $(\mathrm{IA})^3$  in Appendix D. Next, we demonstrate how to utilize the two basic operators  $\oplus$  and  $\ominus$  in different scenarios.
3.2 Composing Basic Operators
When we apply the basic operators to merge different PEMs in practice, a weight hyperparameter  $\lambda \in [0,1]$  is required to alter the relative weights of the modules, as in Ilharco et al. (2022); Wang et al. (2022a). Therefore, we compute  $\theta^{\mathrm{add}}$  as a linear interpolation of two modules and assign a weight scalar to  $\theta^{\mathrm{neg}}$  as follows:
$$
\boldsymbol {\theta} ^ {\text {a d d}} = \lambda \boldsymbol {\theta} ^ {(1)} \oplus (1 - \lambda) \boldsymbol {\theta} ^ {(2)}, \quad \boldsymbol {\theta} ^ {\text {n e g}} = \ominus \lambda \boldsymbol {\theta}. \tag {8}
$$
$\lambda$  is a hyperparameter that is tuned on a validation set. While advanced methods of reweighting different parameters in the full finetuning setting have been proposed by Matena & Raffel (2022); Jin et al. (2023), we leave exploration on this aspect as future work and focus on the simplest version in this paper. Our empirical study next covers four different arithmetic operations based on the operators, as listed in Table 1:² (1)  $\pmb{\theta}^{(1)} \oplus \pmb{\theta}^{(2)}$  for distribution generalization or multi-task learning; (2)  $\ominus \pmb{\theta}$  for unlearning certain abilities from a pretrained model; (3)  $\pmb{\theta}^{(1)} \ominus \pmb{\theta}^{(2)} \oplus \pmb{\theta}^{(3)}$  for transferring a model across domains – for example, when  $\pmb{\theta}^{(1)}$  represents a classification model trained on restaurant reviews,  $\pmb{\theta}^{(2)}$  denotes a language model on restaurant reviews, and  $\pmb{\theta}^{(3)}$  signifies a language model on product reviews, then  $\pmb{\theta}^{(1)} \ominus \pmb{\theta}^{(2)} \oplus \pmb{\theta}^{(3)}$  may lead to a PEM for classification on product reviews. Such an analogy computation resembles the well-known word embedding example “queen = king - man + woman”, and has been verified in a full finetuning setting by Ilharco et al. (2022); and (4)  $\pmb{\theta}^{(1)} \ominus \pmb{\theta}^{(2)}$  for detoxifying instruction-tuned LLMs.
Here we omit the  $\lambda$  hyperparameter for ease of notations.
Table 2: The validation results of PEMs trained on both subsets  $(s_0, s_1)$  and merged PEM  $(m)$ . "FFT" represents full finetuning. We denote the absolute performance change of merged PEM compared to the average results of the two individual PEMs. We report MCC for CoLA, Spearman's  $\rho$  for STS-B, and accuracy for others. Full-dataset LoRA-tuning results are provided as a reference point, which requires all data in one-way training. The tuning results for the full dataset using LoRA are provided as a reference point where both subsets of the data are used together for training.
Method	MNLI	RTE	SST-2	MRPC	QNLI	QQP	CoLA	STS-B
FFT	fullset	76.6	75.8	92.5	88.5	85.9	81.8	0.56	0.90
s0	72.0	72.9	90.4	85.8	83.4	79.2	0.42	0.88
s1	71.9	67.5	92.0	88.5	83.2	81.5	0.52	0.89
m	74.2 ↑2.3	75.1 ↑4.9	92.1 ↑0.9	89.2 ↑2.1	83.8 ↑0.5	81.9 ↑1.5	0.55 ↑0.07	0.89 ↑0.01
LoRA	fullset	87.1	79.8	95.0	89.2	93.4	90.2	0.63	0.91
s0	71.4	72.2	92.2	86.3	83.1	79.0	0.50	0.88
s1	72.3	69.0	91.9	87.7	83.0	80.8	0.51	0.89
m	73.5 ↑1.6	75.8 ↑5.2	92.2 ↑0.2	88.0 ↑1.0	83.3 ↑0.2	81.1 ↑1.2	0.52 ↑0.01	0.89 ↑0.01
(IA)3	fullset	75.9	74.0	92.3	87.3	84.7	80.8	0.56	0.89
s0	71.7	72.9	90.8	85.8	83.0	78.3	0.44	0.87
s1	71.7	68.2	91.2	88.0	82.5	80.8	0.50	0.90
m	74.0 ↑2.3	74.7 ↑4.0	92.3 ↑1.3	88.2 ↑1.3	84.8 ↑2.0	81.3 ↑1.8	0.50 ↑0.03	0.90 ↑0.01
In this section, we empirically study our approach in five diverse scenarios across different arithmetic operations, and then analyze the effect of PEM initialization and the weight hyperparameter  $\lambda$ .
4.1 General Setup
Throughout the experiments, we fix the pretrained model checkpoints and the architecture of PEMs to be composed the same within each scenario, which are the necessary conditions for arithmetic operations. We experiment with LoRA and  $(\mathrm{IA})^3$  for each scenario unless otherwise specified. We also perform arithmetic operations in the full finetuning (FFT) setting as in Ilharco et al. (2022) for a reference point. We emphasize that the full finetuning results are not directly comparable to ours since the motivation of this work is composing parameter-efficient modules. We keep the initialization of the composing PEMs the same for potentially better linear connectivity, while we perform analysis in §4.7 on the effect of different initialization. We note that only the  $A$  matrix in LoRA may be initialized differently – the  $l$  vectors in  $(\mathrm{IA})^3$  are all initialized as ones by design as described in §2.  $\lambda$  is the only tunable hyperparameter in our method. Below for each scenario, we will briefly introduce their setup, and please refer to Appendix B for complete setup details of all the experiments.
4.2 Composition for Distribution Generalization
Setup: In this setting, we aim to combine PEMs trained on the same task but divergent distributions, to improve the model's generalization. To this end, we follow Jin et al. (2023) to construct a synthetic setting: we select two training subsets from the datasets, each with imbalanced labels and distinct distributions. Subsequently, we train two separate PEMs on the two subsets respectively and merge them through  $\theta^{\text{merge}} = \lambda \theta^{(1)} + (1 - \lambda)\theta^{(2)}$ . We then assess the individual and combined PEMs using the original validation data – designed to reflect the performance on the union of the subset distributions – in order to determine whether the merged PEM demonstrates improved generalization capabilities. We work on MNLI (Williams et al., 2018), RTE (Giampiccolo et al., 2007), CoLA (Warstadt et al., 2019), SST2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QNLI (Rajpurkar et al., 2016), QQP (Iyer et al., 2017), and STS-B (Cer et al., 2017) datasets from the GLUE (Wang et al., 2018) task collections. Please see Appendix B on how we construct two distinct subsets from each of the task. We adopt RoBERTa-base (Liu et al., 2019) as the base model. The aforementioned datasets are evaluated using accuracy except CoLA, for which we use Matthews Correlation Coefficient (MCC), and STS-B, which we evaluate using the Spearman's rank correlation coefficient.
Results: We show the results in Table 2. After combination, the merged PEM achieves consistent improvement compared to the average performance of two individual PEMs. For example, the merged
Table 3: The multi-tasking evaluation accuracy of PEMs trained on RTE, MNLI and the merged models. The Avg. column calculates the average accuracy of RTE and MNLI, indicating multi-tasking abilities. For RTE/MNLI, we denote the absolute accuracy change of the merged model compared to the model trained on the target task. For Avg. column, we denote the absolute accuracy change over the best performing individual model.
Method	Modules Used	RTE	MNLI	Avg.
FFT	RTE	82.7	56.3	69.5
MNLI	75.1	85.2	80.1
Merge	78.7 ↓4.0	85.1 ↓0.1	81.9 ↑1.8
LoRA	RTE	81.2	54.7	68.0
MNLI	75.8	86.8	81.3
Merge	78.7 ↓2.5	86.3 ↓0.6	82.5 ↑1.2
(IA)3	RTE	81.2	54.7	68.0
MNLI	75.1	85.8	80.4
Merge	75.1 ↓6.1	85.8 ↑0.0	80.4 ↑0.0
Figure 2: The change of MNLI and RTE validation accuracy with different coefficient  $\lambda$  value for the merged LoRA. By  $\lambda = 0 / \lambda = 1$  we obtained the original RTE/MNLI LoRA.
LoRA module and the merged (IA) $^3$  module obtain gains of 5.2 and 4.0 absolute points respectively on RTE. Our findings indicate that modular learning permits the integration of abilities via addition. As a consequence, the PEFT approach is capable of not only achieving the same level of performance as full finetuning but also excelling in terms of module composition. This highlights the substantial capabilities of PEFT. Analysis of the results change as  $\lambda$  varies can be found in Appendix C.
4.3 Composition for Multi-Tasking
Setup: We examine whether PEMs trained on different tasks could be merged together for multi-task learning. Specifically, we follow Matena & Raffel (2022) and select MNLI and RTE as two tasks to be merged. $^3$  We merge the PEMs trained on MNLI and RTE and evaluate the performance of the merged PEM on both tasks, which is created through  $\theta^{\mathrm{merge}} = \lambda \theta^{(1)} + (1 - \lambda)\theta^{(2)}$ . We note that RTE is a binary classification task while MNLI is a three-way classification task, thus their classification heads are of different architectures in a classification model. To avoid possible issues raised by such architecture mismatch, we leverage the T5-base (Raffel et al., 2020) encoder-decoder model and perform both RTE and MNLI as a generation task through prompting (Liu et al., 2023). Prompting details can be referred to Appendix B.
Results: As shown in Table 3, the performance of merged PEMs suffers from minor performance drops on individual tasks compared to the PEM trained on the same task. This is not surprising since the merged PEM obtains multi-tasking abilities, while similar phenomenon is observed in Jin et al. (2023) as well. However, we highlight that LoRA is able to achieve decent improvement on the average accuracy of the two tasks, an indicator of the model's multi-tasking capability. In Figure 2 we demonstrate how the RTE and MNLI accuracies of the merged LoRA module change as  $\lambda$  varies - while the RTE accuracy is relatively robust to changes of  $\lambda$ , the MNLI accuracy shows significant variations in response to alterations in  $\lambda$ .
4.4 Composition for Unlearning
Setup: Model forgetting is an effective technique to mitigate the unwanted behavior of pretrained models. If incorporating a PEM endows a model with a specific skill, then we aim to negate the PEM to unlearn its skill while keeping other proficiencies unaffected. Specifically, we follow the settings in Ilharco et al. (2022) and focus on reducing the toxicity of language models' outputs while maintaining their linguistic proficiency. To this end, GPT-2 large (Radford et al., 2019) is adopted as the base model and we train PEMs on data from Civil Comments dataset (Borkan et al., 2019)