Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning (2024)

Abstract: Several parameter-efficient fine-tuning methods based on adapters have been proposed as a streamlined approach to incorporate not only a single specialized knowledge into existing Pre-Trained Language Models (PLMs) but also multiple of them at once. Recent works such as AdapterSoup propose to mix not all but only a selective sub-set of domain-specific adapters during inference via model weight averaging to optimize performance on novel, unseen domains with excellent computational efficiency. However, the essential generalizability of this emerging weight-space adapter mixing mechanism on unseen, in-domain examples remains unexplored. Thus, in this study, we conduct a comprehensive analysis to elucidate the generalizability of domain-specific adapter mixtures in in-domain evaluation. We also provide investigations into the inner workings of the mixture of domain-specific adapters by analyzing their weight signs, yielding critical analysis on the negative correlation between their fraction of weight sign difference and their mixtures’ generalizability. The code is available at Github.

heading: 1 Introduction
text: Recently, several parameter-efficient fine-tuning methods that are based on adapters have been introduced as a streamlined approach for fine-tuning Pretrained Language Models (PLMs) to equip them with new, specialized knowledge or domain. Several algorithms have been proposed to train a distinct adapter for each new domain (Houlsby et al., 2019; Pfeiffer et al., 2021; Hu et al., 2022). To further improve a model’s generalizability, existing works (Pfeiffer et al., 2021; Wang et al., 2021a; Diao et al., 2023) mostly focus on training multiple adapters for multiple tasks and continuously adding more adapters for incoming new tasks. This can be inefficient for the new domain tasks that have only a few examples, making the learning among the tasks unequal. Thus, more recent works such as Matena
and Raffel (2022); Wang et al. (2022, 2021b); Li et al. (2022); Chronopoulou et al. (2023) opt for weight-space averaging of model and/or adapters trained on different domains, resulting in so-called Mixture of Expert Adapters.
One recent notable work in this space is AdapterSoup (Chronopoulou et al., 2023), which proposes to merge the weights of a fixed-size, selective subset of different domain-specific adapters via an averaging function to accommodate unseen tasks or domains during inference. Such weight-space merging mechanism on adapters is efficient in practice as one can efficiently train a small, additional adapter and plug it into existing PLMs to incorporate new knowledge. Although the work reported favorable evaluation results on unseen, novel domains, it is unclear to what extent such weight-space merging mechanism on domainspecific adapters can generalize in an in-domain evaluation setting–i.e., how well it makes predictions on unseen examples of domains already seen during training. Moreover, to the best of our knowledge, no existing works comprehensively study the generalization of the mixture of adapters in the indomain setting. This literature gap seems counterintuitive because in-domain evaluation is fundamental and should precede out-of-domain evaluation. Moreover, in real-world applications, model owners have incentives to utilize as much as possible available information to improve their models over time. With the availability of parameterefficient finetuning methods that are fairly easy to adopt with minimal space and runtime cost, the model owners are then incentivized to quickly finetune their models on a few examples collected from a new domain on an additional adapter to optimize the performance (rather than totally relying on outof-domain prediction capability). As a result, although in-domain evaluation seems trivial, it is fundamental as one must ensure that the mixture of adapters works well on the tasks they have already
12956
been trained on. Furthermore, several key questions regarding the resulting mixtures of domainspecific adapters remain unanswered, especially those regarding their generalizability and their adversarial robustness when mixing adapters trained from very different tasks.
Therefore, borrowing the pop-culture saying that “mixed drinks and cocktails aren’t actually the same thing”, in contrast from existing works, we hypothesize that not all mixture of expert adapters are created equal and all have superior performance. Then, through an array of comprehensive experiments, we attempt to give answers to questions about when and what to mix when it comes to domain-specific adapters. We found that the weight-space merging mechanism suffers from performance degradation in terms of both generalizability and adversarial robustness even with inputs from domains it already trains on. Moreover, we also attempt to explain such performance degradation by revealing a critical negative correlation between signed directions of adapter weights during mixing and domainspecific predictive performance (Fig. 1). Although simple, this intuitive and novel observation also allows us to select “when and what adapters to mix?” and design a more effective model pruning as a by-product application.
Overall, our study does not focus on proposing a new mechanism, algorithm, or method. Instead, we focus on analyzing and bringing understanding of an existing and emerging paradigm of mixing multiple domain-specific adapters that was previously introduced (Chronopoulou et al., 2023; Wang et al., 2022). Specifically, we focus on in-domain prediction when mixing adapters from different domains as an emerging and potential paradigm for the deployment of PLMs in practice.
Our contributions are summarized as follows.
1. This is the first and most comprehensive analysis of in-domain generalizability of a mixture of domain-specific adapters with 3 different adapter methods on 13 diverse classification datasets, 2. We provide insights and analysis on when and what adapters to mix to minimize performance degradation via the lens of signed directions of adapters’ parameter weights, 3. We demonstrate the utility of such insights to train mixtures of adapters with 90% sparsity that improve both generalizability and efficiency.

heading: 2 Related works
text: Adapter Fine-tuning. The primary method for adapting general-purpose PLMs to downstream tasks is via full fine-tuning, which requires adjusting all models’ parameters (Peters et al., 2018; Devlin et al., 2019a). However, this results in redundant copies of fine-tuned models for each task, posing a significant memory challenge. Thus, various parameter-efficient fine-tuning methods have been proposed, including prompt-based (Li and Liang, 2021) and adapter-based fine-tuning (Houlsby et al., 2019; Pfeiffer et al., 2021; Hu et al., 2022). Among adapter-based fine-tuning methods, Houlsby (Houlsby et al., 2019) introduces two adapter blocks with bottleneck networks in each Transformer block of a PLM. Similarly, the Pfeiffer (Pfeiffer et al., 2021) adapter differs in architecture, incorporating only one adapter layer in each Transformer block, in contrast to the two layers introduced by Houlsby (Houlsby et al., 2019). LoRA (Hu et al., 2022) takes a distinctive approach by freezing the MLP modules of transformers and representing updates to attention weights with two low-rank matrices to optimize space while effectively retaining model performance. In this work, we focus on analyzing adapter-based fine-tuning methods as they are more popular and effective. Mixture of Expert Adapters. Additionally, several approaches (Wang et al., 2021a; Pfeiffer et al., 2021, 2020; Wang et al., 2022) have been proposed to further optimize their adapters for various downstream tasks by maintaining a set of adapters and combine them during inference. Particularly, AdaMix Wang et al. (2022) fine-tunes so-called Mixture of Experts (MoEs) with adapters on a downstream task and averaging their weights during inference. In addition, Li et al. (2022) explores performance in novel domains through weight averaging on entire language models. Similarly,
AdapterSoup (Chronopoulou et al., 2023) opts for weight-space averaging of adapters trained on different domains. Among these methods, weightspace averaging is identified as the most intuitive method for mixing different adapters (Jin et al., 2023a) and (Chronopoulou et al., 2023).
Nevertheless, none of these works comprehensively evaluates and analyzes the generalizability and adversarial robustness of the resulting mixture of adapters under different mixtures of domainspecific knowledge, which is necessary to answer the question “when and what to mix?”.

heading: 3 Comprehensive In-Domain Evaluation
text: To evaluate the in-domain performance of adapter mixtures, we train several adapters with domainspecific knowledge and mix them in different combinatorial combinations. Then, we evaluate each combination on different downstream tasks on two aspects: (1) generalizability on unseen in-domain examples and (2) adversarial robustness under adversarial text attacks.

heading: 3.1 Evaluation Datasets
text: Diverse Domain Knowledge. To simulate knowledge diversity, we gather a total of 13 distinct and diverse domain-specific datasets of classification tasks for evaluation. We refer the readers to Appendix A.1 for detailed information and their linguistic statistics. Fig. 2 reveals the intricate diversity within our selected datasets, both semantic and topic-wise. Notably, SST2 and IMDB, both originating from the same movie corpus, exhibit proximity in topic embedding spaces. On the contrary, non-formal datasets such as Wiki and Tweets are distinctly distant from other datasets in this regard. We refer the readers to Appendix. A.2 for a detailed exploration and analysis of the topic distributions
among the datasets.

heading: 3.2 Mixing Fine-Tuned Adapters
text: Base Models and Individual Adapters. We design our evaluation using two transformer-based models, namely BERT (Devlin et al., 2019b) and RoBERTa (Liu et al., 2019), with a 3 diverse and well-known adapter methods. They are Houlsby (Houlsby et al., 2019), Pfeifer (Pfeiffer et al., 2021) and LoRA (Hu et al., 2022). These adapter-based methods introduce variations in the adapter architecture and parameterization (Sec. 2), contributing to the comprehensiveness of our analysis. Mixing Adapters. From the pre-trained weights θPLM of either BERT and RoBERTa, we train a suite of 13 domain-specific adapters tailored for diverse domains, denoted as θD1 , θD2 , . . . , θDk . Following (Chronopoulou et al., 2023), the final inference of the target mixture of domain-specific adapters becomes:
f(x, θPLM + 1
k
i=k∑
i=1
θDi) (1)

heading: 3.3 Adversarial Text Generation
text: Textual adversarial attacks are popular in AI robustness research. Given a dataset D={(xi, yi)}Ni , where x represents the sample and y denotes the ground truth label, a textual adversarial attack aims to attack a PLMs fθ with a classification loss function L by perturbing each sample x with perturbation noise δ given a certain budget C:
arg maxδ∈CL[fθ(x+ δ), y], (2) Toward evaluating the robustness of a mixture of adapters, we employ both black-box and white-box textual attacks to exercise Eq. 2. We utilize the popular TextFooler (Jin et al., 2020) as the black-box attack, which aims to replace words with synonyms or contextually similar words to deceive PLMs. We utilize the well-known FGSM (Goodfellow et al., 2015) as the white-box attack, which can efficiently craft adversarial examples by perturbing embedding of text data in the direction of the sign of the gradient of the loss function to the input, thereby exposing vulnerabilities in model robustness.

heading: 3.4 Combinatory Evaluation
text: To evaluate in-domain performance for each target domain, we generate all possible combinations of adapters. To illustrate, for the target domain MNLI, we can first evaluate with a mixture of only
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Ac cu ra cy
MNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
MRPC
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
QNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
QQP
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
RCT
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
RTE
SST2
Tweets
IMDB
Ag News
Financial
Authorship
itself. When combining two adapters, we have the flexibility to choose 1 additional adapter out of the remaining 12, resulting in 12 possible combinations. For a set of 3 adapters, including MNLI, we select 2 adapters out of the 12 to generate C212 combinations. This process continues for sets ranging from 4 to 13 adapters, where, in the case of 13 adapters, all adapters are combined. Thus, we have 13 ∗ (1+∑12i=1Ci12)=53, 248 combinations for all domains. We report mean and variance of in-domain performance for each set of mixtures of k adapters.
Notably, this setup already includes all the possible mixtures potentially selected by AdapterSoup (Chronopoulou et al., 2023), which proposes an additional mechanism to select a subset of a fixed number of domain-specific adapters to mix.

heading: 4 Experiments

heading: 4.1 Implementation Details
text: For the Ag News, Authorship, Financial, IMDB, Tweets, and Wiki-Toxic, we partition the dataset into three segments with an 8:1:1 for train:val:test splits. For datasets belonging to the GLUE corpus, we employ their public training and evaluation splits. For the black-box TextFooler attack, we set the minimum embedding cosine similarity be-
tween a word and its synonyms as 0.8, and the minimum USE similarity is 0.84. For white-box FGSM (Goodfellow et al., 2015) attack, we set the perturbation magnitude to 0.01. Following the setup of Houlsby and Pfeiffer (Houlsby et al., 2019; Pfeiffer et al., 2021), we use all adapters with a dimension of 64 and 256 for RoBERTa-large and BERT-base models. With LoRA, we use rank r=4 following (Hu et al., 2022). Detailed training, evaluation dataset, and hyper-parameter configuration for different tasks are presented in Appendix A.3.

heading: 4.2 In-Domain Performance Results
text: Overview. We present the performance of a candidate setting of RoBERTa with Pfeiffer adapter (Pfeiffer et al., 2021) with different numbers of additional mixing domains in Fig. 3. Table 1 shows how much the predictive performance drops without and with adversarial black-box and whitebox attacks when mixing all adapters. Overall, the average performance drops over all tasks on the clean test set from the original performance to a mix of all adapters is 11.91%, and that for blackbox and white-box attacks are 12.03% and 10.30%, respectively. Further results of generalization and adversarial robustness performance across other models and adapter methods are documented in Appendix A.4.
Finding #1: As we add more tasks or domains, the predictive performance of every single task decreases, reaching its lowest point when we incorporate the maximum of 13 adapters training on various topic datasets. Fig. 3 shows that mixing domain-specific adapters indeed decreases indomain performance (reduction of around 10% in MRPC, QQP, RTE, etc., and nearly 17% in the Financial domain when mixing 13 adapters). The same behaviors were also observed in (Jin et al., 2023b) where they merge the weight of PLMs. Notably, task accuracies decreased at a slower pace for QNLI and SST2 when evaluating with increasing size of mixtures (Fig. 3). In contrast, a substantial decrease in accuracy is observed for domains such as RCT, IMDB, Ag News, and Authorship (Fig. 3). This shows that mixing domain-specific adapters impair model performance differently depending on the target domain, or “what to mix” in an adapter mixture has a crucial effect on the mixture’s performance. To attempt to explain this behavior, we later present and verify a hypothesis that such mixing domain-specific adapters via weight averaging can result in “forgotten knowledge” that can happen due to the differences in signs when mixing these adapters (Sec. 5).
Finding #2: On average, there are no notable differences of the magnitude in accuracy degradation with and without adversarial attacks (12.03% dropped in black-box attack versus 11.91% dropped without attack) (Fig. 3). The overall predictive performance was significantly lower under the white-box compared to the black-box attack as expected because the white-box FGSM attack has additional access to the models’ parameters. Interestingly, on the RCT dataset, the accu-
racy drop on clean (21.72%) is much larger than on black-box and white-box attacks (12.32% and 12.53%, respectively). Finding #3: Variance in task accuracy on adversarial attacks, when combining different domain adapters, is observed to be larger than the variance in clean accuracy (Fig. 3). Although the magnitude decrease appears similar in Fig.3, differences in variance (max, min) are discernible among each combination. Specifically, in MRPC, QNLI, QQP, RTE, SST2, Tweets, IMDB, Financial, and Authorship, we can observe that certain mixed models observed slightly better adversarial robustness compared to single adapters (when the number of mixing adapters k is 2 or 3). Moreover, the variance of robustness in adversarial scenarios tends to be higher than in clean scenarios because all of the adapters are trained on different tasks and they may exhibit different vulnerabilities to the same attack method. Finding #4: Mix up to three adapters to maintain competitive performance. Based on the results from Fig.3 and all findings above, it is advisable to mix only up to three tasks to maintain competitive performance (as observed in MRPC, QQP, RTE, Tweet, Financial, and Authorship domains with less than a 3% accuracy drop in accuracy). Especially, in some cases when evaluating QNLI, SST2, mixtures of less than three adapters even achieved better performance in terms of generalizability (up to 1%) and also in terms of adversarial robustness of up to 3% compared to the original performance.

heading: 5 Effects of Sign Differences of Adapter Weights during Mixing: A Hypothesis

heading: 5.1 An Explanation Hypothesis
text: The ideal scenario when averaging adapter weights during mixing is to make minimal adjustments to their weights, both in terms of values–i.e., magnitudes, and directions, to sustain as much as possible the knowledge learned. Investigating how adapter weights mix regarding both their magnitudes and directions, can be overly complex. Thus, we simplify this assessment by focusing on the sign directions of the adapter weights in our analysis. Following the mixing process of k individual adapter weight in Eq. 1 (Sec. 4), we then hypothesize that mixing adapter weights of conflicting signs can result in “forgotten knowledge”, and lead to performance degradation. As illustrated in Fig. 1, averaging adapter weights across various tasks may
lead to nullifying importance weights for individual tasks if their signs are opposite–i.e., positive v.s. negatives. In other words, the fraction of sign difference or FSD (%) or proportion of weight sign difference in adapter weights during mixing correlate with their mixtures’ generalizability. Alg. 3 (Appendix A.5) shows the calculation of FSD.
We evaluate our hypothesis with different cases: (i) individual adapters (mixture with k=1), (ii) dual adapters (k=2), and generalize to (iii) multiple adapters (k>2). To demonstrate the utility of our hypothesis, we apply it to improve the generalizability of adapter mixtures and also to derive a more effective model pruning in Sec. 6, 7.
5.2 Individual Adapters (k=1) We calculate the FSD of adapter weights on RoBERTa and normalize it by the total number of weights, denoted as a matrix Sk×k where each row is the FSD of a single adapter train on task k to the remaining adapters (Fig. 4). We refer the readers to Sec. A.6 in the Appendix for results on BERT. Interestingly, a consistent trend in the FSD is observed across various model architectures (BERT, RoBERTa) and adapter methods (e.g., Fig. 4 and Fig. 9 of Appendix A.6). The reason is that adapters act as small MLP layers that integrate task-specific knowledge into pre-trained models (Meng et al., 2022) in different adapter methods. This shared functionality contributes to a similar trend in FSD, highlighting the robustness and generalizability of the observed behavior across different adapters’ architectural variations. In addition, Adapters trained on datasets with distinct topic distributions and cosine similarities (Fig. 2) exhibit varying weight directions (Fig. 4). Especially, MNLI has a similar linguistic distribution with other datasets (i.e. MRPC, RTE, Ag News, etc) (Fig. 2), but the adapter trained on MNLI has a significantly larger FSD compared to the remaining domain-specific adapters (Fig. 4). Thus, datasets that are similar in linguistic statistics may not necessarily share the same optimization trajectory. As a result, methods that are based on the linguistic distribution to choose the closest set of adapters to mix like AdapterSoup (Chronopoulou et al., 2023) may lead to sub-optimal performance.
5.3 Dual Adapters (k=2) Fig. 5 shows the FSD for each adapter when mixing two domain-specific adapters. This investigation is conducted within the context of the
Pfeiffer Adapter (Pfeiffer et al., 2021) using a pre-trained RoBERTa model. Overall, there is a strong negative correlation between the FSD and the generalizability–i.e., the lower the sky-blue bar (or the smaller fraction of weight sign conflicts), the higher the performance of the mixture (Fig. 5). Notably, tasks with substantial difference in the weight signs witness a pronounced performance decrease. Specifically, RCT exhibits significant performance drops due to substantial differences in adapter weight direction. Conversely, tasks such as MRPC, and QNLI demonstrate either marginal improvement or no change in performance when mixed with other adapters. This is well correlated to the marginal FSD of the mixed adapters, ranging from only 5% to 10% compared to the original weight.
5.4 Multiple Adapters (k>2) Similar to the dual-adapter setting, there is still a strong negative correlation between FSD and the generalizability, and increasing the number of mixed adapters amplified the sign disparity (Fig. 6). For example, adapters trained on RCT and Tweet exhibit large FSD compared to other adapters and hence observed a significant decrease in generalizability (Fig. 6). In some cases, mixing only a few adapters (e.g., 2–4) could still maintain competitive performance as in QNLI, SST2 domains (Fig. 6, 15). This correlates with the relatively
marginal differences in adapter weights of QNLI and SST compared to adapters trained on other domains (Fig. 4), as their mixtures do not lead to significant cancellations of existing parameters and hence preserve learned knowledge.

heading: 5.5 Discussion
text: Which adapters we should mix? From Sec. 5, we find that it is not advisable to merge adapters that are significantly different from each other in weight signs. On the other hand, mixing the weight of these adapters with other adapters which have a small FSD achieves competitive performance (within 3% drop in accuracy) only when the number of mixing adapters is small (QNLI, MRPC in Fig. 6) or can achieve better performance, although rarely, compared to training the original adapter as seen in QNLI domain (Fig. 5). Therefore, when deploying PLMs, it is prudent to only select a group
Algorithm 1: Greedy Adapter Mixing Input: k domain-specific adapters, a matrix Sk×k
of FSD, l number of adapters to fusion. Output: Average(candidates) 1: candidates← {} 2: Compute the average of FSD 3: avgS = mean(S, axis = 1) 4: Select the top l from set of k adapters according to 5: smallest average FSD 6: candidates← topl 7: return average(candidates)
of tasks with a small FSD to minimize the performance drop in the final mixed model. Our observation is crucial in deploying these models in edge devices where only the adapters are saved on edge, which often has a specific memory capacity limit.
Experiment comparison with AdapterSoup. AdapterSoup (Chronopoulou et al., 2023) dynamically selects a set of l adapters during inference. When l=k, then our experiment setting is the same as the AdapterSoup setting when we use all adapters at once. When l=1, it is the original performance of a single adapter, assuming that AdapterSoup is perfect at picking the same domain that is already trained on, and this result is already included in our paper (mixture of only 1 adapter, Fig. 3). When 1<l<k, we do not have the results of the specific combination that AdapterSoup would select. However, we reported the maximum performance across all combinatorial combinations among 13 domain-specific adapters or each value in Fig. 3. For example, when l=3, we only observe a possible comparative performance (within less than 3% drop) in QNLI, MRPC, and SST2 domains (Fig. 15). We also emphasize that it is one thing to select the best adapters to mix during inference, it is much harder to choose l or how many of them.

heading: 6 Greedy Adapter Mixing with FSD
text: Our observations from Sec. 5 reveal that two adapters with minimal disparity in FSD can yield competitive performance when combined. Therefore, to demonstrate the utility of FSD, in this section, we design a mixing strategy, so-called Greedy Adapter Mixing (Alg. 1), that utilizes FSD to decide which domain-specific adapters to mix by minimizing the overall FSD in a greedy manner to get a final mixed model with competitive performance. This algorithm is also based on the hypothesis that mixing adapters with minimal FSD can yield mixture models of better generalizability. We proceed
by evaluating model performance across various adapter combinations.
Fig. 7 shows the generalizability performance of a mixture of l ∈ [2, 13] domain-specific adapters. Overall, Greedy Adapter Mixing resulted in very competitive performance compared with empirical upper-bound accuracy. In contrast, using the same algorithm but maximizing FSD resulted in performance close to the empirical lower-bound accuracy (Fig. 7). However, in both two cases, mixing adapters with FSD cannot achieve the empirical upper-bound and lower-bound performance. Thus, greedily mixing adapters to minimize FSD does not totally prevent knowledge loss in the adapter mixtures. Nevertheless, FSD is still useful as a guidance measure to effectively mix domain-specific adapters.

heading: 7 Towards Effective Model Pruning
text: To further demonstrate the utility of our FSD analysis in Sec. 5 and Sec. 6, in this section, we leverage FSD information to reduce knowledge loss through the development of a pruning algorithm guided by FSD insights. Specifically, Fig. 5 shows that predictive performance experiences a significant drop when integrating adapters with pronounced disparities in weight signs–i.e., positive v.s. negative signs. Moreover, neural network pruning indicates that only a limited number of weights significantly contribute to task performance, suggesting redundancy within the weights that can be pruned without compromising the original task performance (Han et al., 2016; Frankle et al., 2021; Lazarevich et al., 2021). Thus, to mitigate the impact of weight sign differences in adapter mixtures, we propose mixing only the sparse versions of the adapters’ weights.
Different from Alg. 1, this strategy indirectly reduces the fraction of weight sign conflicts. Intuitively, by minimizing the FSD, the mixing process becomes more resilient to the inadvertent elimination of important weights by less significant or redundant weights. This phenomenon is visually depicted in Step 2 of Fig. 1, where only significant weights in the two adapters need to be preserved, and small or unimportant weights of opposing signs can be eliminated.

heading: 7.1 FSD-based Magnitude Pruning
text: Post-training Pruning. Sparse Adapter (He et al., 2022) employs pruning across every layer of adapters, being able to achieve comparable or
Algorithm 2: FSD-based Pruning Input: adapter parameters w, sparse ratio s. Output: pruned adapter w̃. 1: w ← Trained(w) 2: Compute important score z = |w| 3: Compute the s-th percentile of z as zs 4: m← 1 [z − zs ≥ 0] 5: w̃ ← m⊙ w
even superior predictive performance than standard adapters, even when the sparse ratio reaches up to 80%. By adopting a similar process of adapters pruning after training, with the guidance of our FSD analysis in Sec. 5, we can eliminate redundant parameters at an early stage, circumventing the need for a time-consuming iterative pruning process, as discussed in prior works such as Han et al. (2016); Lazarevich et al. (2021). The detailed pruning algorithm is presented in Alg. 2.
Generalizability of Pruned Adapters. Fig. 8a shows RoBERTa’s performance, where we systematically prune the weight of the Pfeiffer adapter 40%–100% of sparsity. For a single task at the sparsity level d%, we retain only the largest–i.e., the top-d%, influential parameters of the corresponding adapter, and report the average in-domain performance across 13 domains. Remarkably, pruning up to 90% parameters of adapter weight does not lead to performance degradation. This observation suggests redundancy in adapters’ parameters may contribute to the increase in the fraction of weight direction conflicts–i.e., high FSD when merging them (Fig. 6).
Generalizability of Mixed Pruned Adapters. Motivated by pruned adapter can still maintain original performance up to 90% of sparsity (Fig.
8a), we hypothesize that mixing sparse adapters may indirectly reduce the fraction of weight sign conflict, therefore, leading to competitive performance with the original adapters. Given a set of domain-specific adapters, based on the FSD, we choose the top k layers with the highest fraction of weight sign difference and prune each layer up to 90% of sparsity. Then we mix these adapters by weight averaging. Details of mixing domainspecific adapters with weight sign conflict information is shown in Alg. 4 (Appendix A.8). Fig. 8b shows that mixing adapters with 80% or 90% sparsity consistently achieved better performance than the upper-bound empirical accuracy achieved when mixing their dense versions.

heading: 8 Conclusion
text: This work provides a comprehensive empirical indomain evaluation of the emerging mechanism of mixing domain-specific adapters. We also provide insights into the inner workings of the mixture of domain-specific adapters by analyzing their weight signs, yielding critical observations on the negative correlation between the fraction of sign difference among adapters and their mixtures’ generalizability. By examining the signed directions of adapter weights, we also offer the readers valuable advice on the optimal selection of adapters to mix to achieve competitive performance. Such examination also helps enhance our understanding of the interconnected role of weight sign difference in the context of sparse neural networks.
Limitation
Primarily, our exploration focused solely on one classic pruning method, namely Magnitude Pruning (Sanh et al., 2020) while there are existing more advanced pruning techniques such as SynFlow (Tanaka et al., 2020), GraSP (Wang et al., 2020) that are also applicable for condensing neural network architectures. Consequently, future works include investigating the applicability of our findings to these alternative pruning approaches. Furthermore, our examination was confined to the natural language understanding tasks. A valuable avenue for future research would involve extending our analysis to encompass the emerging text generation tasks, particularly within the context of the current transformer-based language model, including but not limited to the machine translation tasks utilizing complex GPT-family models.

heading: A Appendix
text: Dataset mnli mrpc sst2 rte
Train 392,702 3,668 67,349 2,490 Test 9,815 408 872 277
Dataset qnli qqp rct ag Train 104,743 363,846 178,882 120,000 Test 5,463 40,430 30,135 7,600
authorship financial imdb tweets wiki 2,743 4,846 22,500 31,962 127,656 686 484 2,500 3,196 63,978
A.1 Datasets. Diverse Knowledge Datasets. To simulate knowledge diversity, we gather a total of 13 distinct and diverse domain-specific datasets or classification tasks for evaluation. They are MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Bentivogli et al., 2009), MRPC (Dolan and Brockett, 2005), QQP (Iyer et al., 2017) and SST2 (Socher et al., 2013) from the GLUE domain
corpus. PubMed-20K RCT dataset (Dernoncourt and Lee, 2017) from Biology domain for sentence classification. IMDB dataset from a Movie Review domain. Ag News, Financial (Malo et al., 2014) and Guardian Authorship (Altakrori et al., 2021) are News domain datasets across World, Sports, Business, Science/Technology, and Financial topics. Wiki Toxic 1 and Tweets Hate Speech are two Informal text domain for toxicity detection.
Linguistic Statistic. Table 3 shows detailed statistics such as number of documents, average document length, and sentence lengths.
A.2 Topic distribution of training datasets
Tables from 4 to 16 show 10 topics and corresponding important words which are exacted from LDA for each training dataset. Notably, Ag News and SST2 have high cosine similarity but observe a large difference in terms of topic distribution compared to other domains (Fig. 2). Therefore, each statistical mechanism like cosine similarity or topic distribution only reflects one aspect of data distribution and may show inconsistencies with each other.
A.3 Hyper-parameter
Training and evaluation datasets. To assess performance in out-of-distribution scenarios, we conduct evaluations on a diverse set of 13 datasets covering various topics, ranging from movie reviews, news, authorship, and healthcare, to non-formal language text such as Wiki Toxic and Tweets. For datasets within the GLUE corpus, we employ training and evaluation datasets to gauge accuracy across different settings. In the case of Ag News, Authorship, Financial, IMDB, Tweets, and WikiToxic, we partition the training set into three segments with an 8:1:1 ratio, utilizing them for training, evaluation, and test datasets, respectively. This approach ensures a comprehensive evaluation of model performance across a wide spectrum of domains and linguistic styles. Table 2 shows data statistics on train/test datasets. Setting on text adversarial attack. In this study, we employ two types of attacker methods: TextFooler (Jin et al., 2020) and FGSM (Goodfellow et al., 2015).
TextFooler word-level attacks focus on replacing words within the text with synonyms or contextu-
1https://www.kaggle.com/competitions/jigsaw-toxiccomment-classification-challenge/
ally similar words. By making ostensibly minor alterations to the input text, these attacks can deceive LLMs into producing incorrect outputs or substantially modifying their predictions. We meticulously fine-tune the hyperparameters of TextFooler to obtain more appropriate synonyms. We set the minimum embedding cosine similarity between a word and its synonyms as 0.8, and the minimum Universal Sentence Encoder similarity is 0.84.
FGSM (Goodfellow et al., 2015) is a white-box embedding-level attack. FGSM uses the Fast Gradient Sign Method to calculate gradients of the model’s loss to the input text and generates an adversarial example by perturbing the embedding of input text in the direction that maximizes the loss. We choose the magnitude of the perturbation in embedding space as 0.01 on BERT and RoBERTa models.
Adapter Configuration. We use adapters with a dimension of 64 and 256 using RoBERTa-large and BERT-base encoders following the setup of (Houlsby et al., 2019), (Pfeiffer et al., 2021). With LoRA, we use rank r = 4 following the setup of (Hu et al., 2022).
Hardware Information. We evaluate model performance on AMD Ubuntu 22.04.2 with Ryzen
Threadripper PRO 5975WX, 1800MHz, Cached 512 KB and 4 × GPU Nvidia A6000. HyperParameters. Detailed hyper-parameter configuration for different tasks is presented in Table 17.
A.4 Model performance when mixing adapters across tasks
Tables 10, 11, 12 and 13 show task accuracy when mixing multiple adapters.
A.5 Fraction of weight sign difference
Alg. 3 presents a detailed algorithm to compute the FSD.
A.6 Additional result on weight sign difference
Fig. 9 shows the weight sign difference of the adapter and normalizes it by the total number of adapter weights in BERT.
A.7 Additional results when mixing two adapters
Fig. 14 and 15 show model generalization when mixing two and multiple adapters across various tasks.
A.8 Mixing Sparse Adapters with weight sign difference
Alg. 4 shows details of mixing sparse adapter with sign conflict information.
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Ac cu ra cy
MNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
MRPC
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
QNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
QQP
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
RCT
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
RTE
SST2
Tweets
IMDB
Ag News
Financial
Authorship
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Ac cu ra cy
MNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
MRPC
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
QNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
QQP
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
RCT
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
RTE
SST2
Tweets
IMDB
Ag News
Financial
Authorship
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Ac cu ra cy
MNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9
MRPC
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9
QNLI
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9
QQP
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9
RCT
1 3 5 7 9 11 13 0.3 0.4 0.5 0.6 0.7 0.8 0.9
RTE
SST2
Tweets
IMDB
Ag News
Financial
Authorship

heading: Future Work
text: Consequently, future works include investigating the applicability of our findings to these alternative pruning approaches. Furthermore, our examination was confined to the natural language understanding tasks. A valuable avenue for future research would involve extending our analysis to encompass the emerging text generation tasks, particularly within the context of the current transformer-based language model, including but not limited to the machine translation tasks utilizing complex GPT-family models