Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories
Year: 2023

Abstract:
Pre-trained language models (PLMs) demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. Although continued pre-training on a large domain-specific corpus is effective, it is costly to tune all the parameters on the domain. In this paper, we investigate whether we can adapt PLMs both effectively and efficiently by only tuning a few parameters. Specifically, we decouple the feed-forward networks (FFNs) of the Transformer architecture into two parts: the original pre-trained FFNs to maintain the old-domain knowledge and our novel domain-specific adapters to inject domainspecific knowledge in parallel. Then we adopt a mixture-of-adapters gate to fuse the knowledge from different domain adapters dynamically. Our proposed Mixture-of-Domain-Adapters (MixDA) employs a two-stage adapter-tuning strategy that leverages both unlabeled data and labeled data to help the domain adaptation: i) domain-specific adapter on unlabeled data; followed by ii) the task-specific adapter on labeled data. MixDA can be seamlessly plugged into the pretraining-finetuning paradigm and our experiments demonstrate that MixDA achieves superior performance on in-domain tasks (GLUE), out-of-domain tasks (ChemProt, RCT, IMDB, Amazon), and knowledge-intensive tasks (KILT). Further analyses demonstrate the reliability, scalability, and efficiency of our method.1

heading: 1 Introduction
text: Pre-trained language models (PLMs) have achieved a multitude of successful applications in natural language understanding (Devlin et al., 2018; Liu et al., 2019; He et al., 2021b) and generation (Lewis et al., 2019; Zhang et al., 2020; Yang et al., 2020; Brown et al., 2020). The predominant methodology for domain adaptation is fine-tuning on labeled
*Equal Contribution. 1The code is available at https://github.com/
Amano-Aki/Mixture-of-Domain-Adapters.
domain-specific data or continued pre-training (Gururangan et al., 2020) on unlabeled domain-specific data. Although effective, both fine-tuning and continued pre-training methods require tuning all the parameters of a PLM, raising high costs beyond many institutions’ reach. To mitigate this, multiple parameter-efficient fine-tuning (PEFT) methods are proposed, including prompt-based tuning (Gao et al., 2021; Liu et al., 2021b; Schick and Schütze, 2021; Li and Liang, 2021; Liu et al., 2021a), and adapter-based tuning (Houlsby et al., 2019; Pfeiffer et al., 2020b; Hu et al., 2021). However, they are more concerned about task adaptation and it is still unclear how to regularly, and inexpensively inject domain knowledge into PLMs for different domain-specific tasks. Moreover, directly tuning PLMs on a domain-specific corpus with PEFT methods will lead to the catastrophic forgetting problem (Yogatama et al., 2019; Gururangan et al., 2020). These limitations highlight an important research question: how to adapt PLMs with the new domain knowledge while keeping the old-domain knowledge unmodified?
Inspired by the recent studies (Geva et al., 2021; Cao et al., 2021; Meng et al., 2022) that found knowledge is stored in feed-forward networks (FFNs), we decouple the FFNs into two parts: the original pre-trained FFNs to maintain the olddomain knowledge and our novel domain-specific adapters to inject domain-specific knowledge in parallel. Specifically, we propose Mixture-ofDomain-Adapters (MixDA), a mixture of several domain adapters to inject domain-specific knowledge without affecting the old-domain knowledge. Our model has two stages: piq domain-specific tuning multiple knowledge adapters on unlabeled data and then piiq task-specific tuning adapters on labeled data. In the first stage, we train several domain adapters on both domain-specific corpus and pre-training corpus simultaneously while keeping the original feed-forward networks unchanged. In
5113
the second stage, we train a mixture-of-adapters gate to dynamically select the desired knowledge adapter and a task-specific adapter for task adaptation.
We conduct experiments on a broad range of tasks, including 4 out-of-domain datasets, 9 in-domain datasets, and 2 knowledge-intensive datasets. Our experimental results demonstrate the effectiveness of MixDA on 15 datasets, spanning biomedical, computer science publications, news, and reviews. Further analysis displays three key properties of our proposed approach: piq Reliability: it shows superior performance on both in-domain and out-of-domain tasks. piiq Scalability: it scales well to the increasing number of domains. piiiq Efficiency: it adds only a small number of parameters per domain. We claim that these properties are helpful for language models as a service, where a frozen PLM is served, and multiple adapters are inserted to support different customized services.

heading: 2 Related Work
text: In this section, we will review four research lines related to injecting domain knowledge into pretrained language models: knowledge injection, domain adaptation, parameter-efficient fine-tuning, and mixture-of-adapters.

heading: 2.1 Knowledge Injection
text: Knowledge can be injected into PLMs by pretraining or fine-tuning, each corresponding to a separate research direction. During pre-training, the knowledge carried by knowledge graphs (Zhang et al., 2019; He et al., 2020), entities (Sun et al., 2019; Xiong et al., 2020), n-grams (Diao et al., 2020), knowledge embedding (Wang et al., 2021b), synonym and hyponym-hypernym relations in WordNet (Lauscher et al., 2019), word-supersense knowledge (Levine et al., 2020), and knowledge bases (Peters et al., 2019) can be injected into PLMs by feeding knowledge inputs and designing new objectives. However, pre-training-based methods are costly, making the application to huge PLMs (e.g., models with 175 Billion parameters) impossible. Fine-tuning-based methods only require an additional fine-tuning process. Some studies inject extra information into the input sentences, like knowledge triples from knowledge graphs (Liu et al., 2020) and knowledge context (Faldu et al., 2021), while other studies explored specific model
and training designs, like knowledge adapter networks (Wang et al., 2021a), graph convolutional networks and LSTMs (Lin et al., 2019), and metalearning (Sinitsin et al., 2020). Zhu et al. (2020) formulated knowledge injection as a constrained optimization problem by adding a constraint on the loss on the unmodified facts. Recent studies (Geva et al., 2021; Cao et al., 2021; Meng et al., 2022) reveal that knowledge is stored in the feed-forward networks in PLMs. Inspired by these studies, we propose a new efficient tuning method to inject domain knowledge into feed-forward networks with minimal costs.

heading: 2.2 Domain Adaptation
text: Previous studies have observed that language models suffer from a significant performance drop during the domain shift (Beltagy et al., 2019; Alsentzer et al., 2019; Huang et al., 2019; Lee et al., 2020; Ke et al., 2022b). Effective strategies that can bridge the domain gap are introduced. Pre-training language models from scratch is effective but costly, like SciBERT (Beltagy et al., 2019), BioBERT (Lee et al., 2020), and ClinicalBERT (Alsentzer et al., 2019). Recent studies explored continued pretraining (Gururangan et al., 2020) and adapter networks (Diao et al., 2021) to save time by training on unlabeled downstream task data. In this paper, we introduce plug-in domain adaptors for domain adaptation, which are effective and mitigate catastrophic forgetting issues because of the explicit learning strategy and efficient model architecture.

heading: 2.3 Parameter-Efficient Fine-tuning
text: Another relevant research direction is parameterefficient fine-tuning (PEFT), which only fine-tunes a small number of parameters. Existing works solve this problem from two perspectives: promptbased tuning (Gao et al., 2021; Liu et al., 2021b; Schick and Schütze, 2021; Li and Liang, 2021; Liu et al., 2021a), and adapter-based tuning (Houlsby et al., 2019; Pfeiffer et al., 2020b; Hu et al., 2021). Several works in adapter-based tuning are closely related to ours. AdapterFusion (Pfeiffer et al., 2021) aims to combine multiple task adapters but does not offer specific architecture or training strategies to learn external knowledge. DEMix (Gururangan et al., 2022) and MixDA both train adapters that specialize in domains and use mechanisms to route different adapters, but differ in routing methods, base models, and training strategies. K-Adapter (Wang et al., 2021a) is re-
stricted by its training on T-REx triples and lacks the flexibility to train on unstructured knowledge. Similar to MixDA, CPT (Ke et al., 2022a) integrates domain knowledge into LMs, but it employs a different approach. While MixDA uses domain adapters to substitute FFN layers and task adapters to perform end tasks, CPT adds CL-Plugins that learn domain knowledge. Recent work by He et al. (2021a) presents a unified framework that establishes connections across different PEFT methods. Our work can leverage any PEFT method and complement them.

heading: 2.4 Mixture-of-Experts
text: Mixture-of-Experts (MoE) (Shazeer et al., 2017) is introduced with several expert networks, gating networks, and load-balancing techniques. The following studies improve MoE on initialization and training schemes (Fedus et al., 2022), routing mechanisms (Zuo et al., 2021; Yang et al., 2021), and load-balancing issues (Lewis et al., 2021; Roller et al., 2021). AdaMix (Wang et al., 2022) proposed a mixture of adapters to improve the downstream task performance. Instead of mixing different designs of adapters, our domain adapter is a feedforward network specifically designed for domain knowledge.

heading: 3 Approach
text: Given a pre-trained language model M, the input is a sentence X “ t1t2 ¨ ¨ ¨ ti ¨ ¨ ¨ tT (ti indicates the i-th token) and the output is the representation of each token. The overall architecture of our model is shown in Figure 1. The training process is divided into two-stage. In Stage 1 (Figure 1 (a)), we inject new feed-forward networks (FFNs) (namely domain-adapter) paralleled to the original pre-trained FFNs in some Transformer layers, acting as a key-value memory. The newly injected domain-adapter is trained on both domain-specific unlabeled data and original pre-training unlabeled data to store new factual associations while keeping old-domain ones. All modules are frozen except domain-adapter in this stage. In Stage 2 (Figure 1 (b)), we train a mixture-of-adapters (MoA) gate and a task-adapter on downstream tasks with labeled data, and only these two new modules are updated. The MoA gate receives outputs from the old-domain FFNs and domain-adapter, then outputs a weighted sum of them. An additional taskadapter is inserted in each Transformer block to
facilitate downstream tasks. Figure 1 (c) shows the structures of the domain-adapter and the MoA gate.
In this section, we first introduce domainadapter, which learns and stores domain-specific knowledge, and then describe task-adapters that perform the downstream task. Finally, we discuss how the MoA gate integrates the outputs from the FFN and the domain-adapter.

heading: 3.1 Domain-Adapter
text: Previous studies (Geva et al., 2021; Cao et al., 2021; Meng et al., 2022) suggest that factual associations are stored in the FFNs of some Transformer layers. To help models learn domain-specific knowledge, we propose a lightweight domain-adapter that works parallel to the FFNs, and a training method to learn domain-specific knowledge alongside keeping old-domain ones. Domain-adapter has a simple bottleneck architecture consisting of a down projection layer, a nonlinearity (such as ReLU (Agarap, 2018)), and an up projection layer. This helps keep the parameter size low (Houlsby et al., 2019) with competitive performance.
In Stage 1, the domain-adapter is trained with the domain-specific and old-domain datasets in one batch. Note that all other parameters are frozen except the domain-adapter at this stage. Let LK denote the knowledge loss related to domain-specific knowledge, and LS denote the sampling loss related to old-domain knowledge. The knowledge loss is a cross-entropy loss on predicting masked tokens, and the sampling loss is designed to align the latent spaces of the old-domain knowledge and new domain-specific knowledge. The total loss L is given by a weighted sum of the two, that is:
L “ λ ¨ LK ` LS , (1) where λ is a weight for the knowledge loss.
The knowledge loss is implemented by using cross-entropy loss. Given a sentence with M mask tokens whose answers are m1,m2, ¨ ¨ ¨ ,mM , respectively, the knowledge loss LK is given by
LK “ ´ 1 M
Mÿ i“1 log ppmiq, (2)
where ppmiq is the probability for token mi output by M. Our model accepts two types of domainspecific knowledge as follows, showing improved versatility. ‚ Structured knowledge If the knowledge dataset
is structured (e.g., ConceptNet (Speer et al.,
2016)), we translate each relation into a sentence, and then mask out its object. For example, the relation “the Eiffel tower–/r/LocatedAt–Paris” is translated into “The Eiffel Tower is located at Paris.”, then “Paris” is substituted with the mask token, and the model is trained to fill the mask. ‚ Unstructured knowledge For unstructured knowledge (e.g., downstream unlabeled texts), we use the masked language model (MLM) similar to RoBERTa pretraining. Some tokens are randomly sampled from the input sentence and replaced with the special token <mask>, and the model is trained to predict the masked token. The cross-entropy loss is calculated to optimize the model. For old-domain knowledge and sampling loss, we train the model on general corpora including Wikipedia and BookCorpus (Zhu et al., 2015). Specifically, for each batch, sentences randomly
sampled from the dataset are input into the model. Given L layers that have domain-adapters installed, for each such layer l, we collect token representations from the FFN Fl, and representations from the domain-adapter Kl. The goal is to keep them as similar as possible. Thus, we calculate the sampling loss LS with L2 loss:
LS “ 1 L
Lÿ l“1 ||Fl ´ Kl||22. (3)

heading: 3.2 Task-Adapter
text: After training domain-adapters, the model is aware of the domain knowledge, which is not directly related to downstream tasks though. Therefore, we add task-adapters on top of the domain-adapter to adapt to downstream tasks. For example, a domainadapter trained in biomedical knowledge can sup-
port different tasks in the domain, while training it on a task limits its capability to the specific task. Task-adapters can be any adapter architecture or other parameter-efficient fine-tuning methods, such as the Houlsby adapter (Houlsby et al., 2019), Pfeiffer adapter (Pfeiffer et al., 2020b), prefix-tuning (Li and Liang, 2021), and so on. At Stage 2, all parameters other than the task-adapters and the MoA gate (Section 3.3) are frozen. The training of the adapter follows its corresponding approach, despite the addition of domain-adapters. For example, for a text classification task, we add a classification layer on top of the model, freeze all parameters other than the classification layer, the MoA gate, and the task-adapters, feed input texts into the model, and use cross-entropy as the loss.

heading: 3.3 Mixture-of-Adapters Gate
text: On downstream tasks, it is possible that the output from the FFN, or a weighted sum of the two, produces better results. Therefore, in Stage 2, we train an additional mixture-of-adapters (MoA) gate simultaneously. The MoA gate receives the outputs from the attention layer q, the domain-adapter K, and the FFN F . q is first sent into a multi-layer perceptron (MLP):
h “ MLPpqq. (4)
The MLP is composed of a down-projection layer Wd and an up-projection layer Wu, and h “ WuσpWdqq, where σ is the nonlinearity function. Then, h is input into a Sigmoid layer to generate the weights of the FFNs and other domain-adapters:
w “ Sigmoidphq. (5)
The final output o is a weighted sum of the outputs of the FFNs and the domain-adapter:
o “ wrK;F s, (6)
where r; s denotes matrix concatenation.

heading: 4 Experimental Settings
text: In this section, we first introduce the datasets, then the baseline models, the evaluation metrics, and implementation details in the following four subsections, respectively.

heading: 4.1 Datasets
text: We conduct experiments on three types of datasets: in-domain (ID) tasks that require general-domain knowledge; out-of-domain (OOD) tasks that require domain-specific knowledge; knowledgeintensive (KI) tasks that require commonsense knowledge. ‚ ID: GLUE Benchmark (Wang et al., 2018)
including MNLI (Williams et al., 2017), CoLA (Warstadt et al., 2019), MRPC (Dolan and Brockett, 2005), SST-2 (Socher et al., 2013), RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), STS-B (Cer et al., 2017), WNLI (Levesque et al., 2012), QNLI (Rajpurkar et al., 2016), and QQP (Iyer et al., 2017). ‚ OOD: ChemProt (Kringelum et al., 2016), RCT (Dernoncourt and Lee, 2017), IMDB (Maas et al., 2011), and Amazon (He and McAuley, 2016). ChemProt is a manually annotated chemical-protein interaction dataset extracted from 5,031 abstractions. RCT is a dataset based on PubMed for sentence classification. IMDB provides 25,000 movie reviews for sentiment analysis. Amazon is a dataset containing product reviews from Amazon, annotated with user ratings. ‚ KI: FEVER (Thorne et al., 2018) and CommonsenseQA (CSQA) (Talmor et al., 2019). FEVER consists of 185,445 claims that correspond to Wikipedia articles and are classified as supported, refuted, and not enough information. CommonsenseQA consists of 12,247 questions with 5 choices and requires commonsense knowledge to predict the correct answers. For Stage 1, we train the domain-adapter with unstructured knowledge related to the dataset following Section 3.1. The unstructured knowledge
used is listed in Table 1. We also experiment with structured knowledge in Section 6.2. For Stage 2, we adopt the true few-shot setting following (Perez et al., 2021) to demonstrate the effectiveness of MixDA. For each dataset class, we randomly sample K “ 16 examples from the original training set as the new training set, and another different K “ 16 examples as the validation set. The original validation set will be used as the test set. The Pfeiffer adapter is used in Stage 2 unless stated otherwise.

heading: 4.2 Baselines
text: In our experiments, we use the following models as the main baselines. For convenience, we refer to them with the abbreviations in the parentheses later. ‚ HOULSBY (HO): Houlsby adapter (Houlsby
et al., 2019) plugged into the RoBERTa-large model for downstream tasks. Only adapter parameters are trained. It adds two adapter blocks consisting of bottleneck networks in each Transformer block. ‚ PFEIFFER (PF): Pfeiffer adapter (Pfeiffer et al., 2020b) plugged into the RoBERTa-large model. This is similar to the Houlsby adapter, but with a different architecture. Pfeiffer adapter has only one adapter layer in each Transformer block, while Houlsby has two. Also, Pfeiffer makes minor tweaks in the adapter architecture, such as the layer norm and nonlinearity. ‚ LORA (LO): LoRA (Hu et al., 2021) applied to the RoBERTa-large model. LoRA freezes the MLP modules and represents updates to the attention weights with two low-rank matrices, thus saving space. ‚ PREFIX-TUNING (PT): Prefix-Tuning (Li and Liang, 2021) with the RoBERTa-large model. Prefix-Tuning trains a number of prompt embeddings for each task and pre-pends it before tokens. ‚ FINE-TUNING (FT): Fine-tuning all of the parameters of the RoBERTa-large model on downstream tasks.

heading: 4.3 Evaluation Metrics
text: We adopt the Pearson correlation for STS-B since it is a regression task. The remaining are text classification tasks. Following Wang et al. (2018); Gururangan et al. (2020); Diao et al. (2021), we adopt macro-F1 for MRPC and QQP, and micro-F1 for others as evaluation metrics. Macro-F1 computes
the F1 independently for each metric, while microF1 computes an average metric of all classes. To account for the instability of small datasets, we report the average performance and the standard deviation of 3 runs with different random seeds.

heading: 4.4 Implementation
text: We implement our RoBERTa-large model based on the Transformers library from HuggingFace2. The Houlsby adapter, the Pfeiffer adapter, and PrefixTuning are implemented based on the adaptertransformers library (Pfeiffer et al., 2020a). LoRA is implemented based on OpenDelta (Ding et al., 2022). During Stage 1, we train the domain-adapter with learning rate 1e-4, batch size 20, and weight decay 0.05. The knowledge loss factor λ is set to 0.5. We train the 7 and 11 layers of RoBERTa-large with domain-adapter in 10 epochs. In Stage 2, we use the Pfeiffer adapter as the default task-adapter and train 20 epochs. All the experiments are conducted on Nvidia 2080Ti GPUs. We find the best hyper-parameters through grid search and the best results are listed in Appendix A. The computation time can be found in Appendix B.

heading: 5 Experimental Results
text: We compare the performance of MixDA with our baselines on 15 datasets. First, we train the domainadapter for each domain individually and then perform each task with its corresponding domainadapter, which shows significant improvement over our baselines. Next, we plug in several domainadapters trained on different domains parallelly to verify the scalability of our model.

heading: 5.1 Single Domain Adapter
text: Table 2 shows the performance of a single domain adapter compared with baselines. It is only trained on unstructured knowledge during Stage 1 in the following experiments. Results show that Mixtureof-Domain-Adapters outperforms our baselines in most datasets, with an average of 3.5% improvement over the best baseline adapter (i.e., Pfeiffer), and 3.3% over fine-tuning. Our method even outperforms fine-tuning in most datasets, despite far less training time and smaller parameter size. Over the datasets, MixDA shows the most significant improvement on ChemProt, with 6.9% over Pfeiffer and 2.7% over fine-tuning. One possible reason is that MixDA learns the necessary knowledge to
2https://github.com/huggingface/transformers
Datasets HO PF LO PT FT MixDA
OOD
ChemProt 47.1˘12.2 53.7˘8.2 24.2˘11.6 17.1˘7.3 57.9˘4.0 60.6˘4.9 RCT 25.2˘2.6 21.9˘2.5 18.5˘3.7 24.4˘3.7 21.0˘3.2 26.4˘0.8
IMDB 56.0˘5.7 55.4˘5.6 43.7˘7.7 53.3˘2.6 46.3˘13.7 58.1˘5.1 Amazon 48.8˘3.2 49.7˘1.4 51.5˘3.9 52.7˘2.8 51.7˘6.2 54.7˘1.6
Avg. 44.3 45.2 34.5 36.9 44.2 50.0
ID
MNLI 37.2˘0.3 35.7˘0.1 34.8˘1.5 35.4˘0.0 35.3˘0.2 37.3˘1.5 COLA 17.6˘5.5 9.1˘5.0 7.1˘3.0 12.1˘5.5 21.3˘1.7 20.1˘6.3 MRPC 81.2˘0.2 80.7˘0.6 64.0˘20.5 81.6˘0.5 81.3˘0.1 81.6˘0.5 SST2 54.7˘3.6 53.3˘1.9 50.5˘1.0 52.5˘1.2 54.8˘1.4 56.4˘3.5 RTE 53.5˘1.4 54.1˘1.3 53.4˘2.1 53.4˘1.1 54.7˘1.3 54.9˘1.5 STS-B 88.1˘1.6 90.6˘0.1 89.5˘0.8 85.6˘4.1 78.4˘8.0 89.8˘0.4 WNLI 57.3˘1.3 58.1˘2.5 59.1˘1.2 57.3˘0.7 58.7˘1.8 60.1˘1.8 QNLI 53.0˘0.1 51.9˘0.8 53.3˘1.3 52.1˘0.9 51.3˘0.1 54.8˘1.8 QQP 54.7˘0.6 55.2˘1.0 53.0˘2.0 55.3˘0.3 55.3˘0.3 56.1˘0.6 Avg. 55.3 54.3 51.6 53.9 54.6 56.8
KI FEVER 20.2˘4.3 27.4˘7.5 22.6˘10.6 31.1˘3.6 36.1˘6.7 32.6˘9.4 CSQA 27.3˘0.7 34.1˘8.7 20.3˘10.9 29.6˘4.2 29.6˘3.0 38.9˘4.0 Avg. 23.8 30.8 21.5 30.4 32.9 35.8
Avg. 48.1 48.7 43.0 46.2 48.9 52.2
detect the chemical-protein interaction. For example, MixDA shows more familiarity with words associated with that field, such as “gefitinib” and “tyrosine kinase inhibitor”. In contrast, MixDA falters on STS-B, falling behind Pfeiffer by 0.8%. This is because the knowledge in Stage 1 is not effectively utilized. STS-B consists of sentence pairs like “The cat sat on the mat” and “The cat did not sit on the mat”, with little need for additional knowledge. Across the three task domains, MixDA has an average improvement of 4.8% over RoBERTa + Pfeiffer on out-of-domain tasks, 2.5% on indomain tasks, and 5.0% on knowledge-intensive tasks. It shows that MixDA is not only effective for out-of-domain tasks and knowledge-intensive tasks that require additional knowledge but is helpful for general-domain language tasks as well, demonstrating its ability to excel at both in-domain and out-of-domain tasks (reliability).

heading: 5.2 Parallel Domain Adapters
text: In the previous section, we explored using a single domain-adapter for each downstream task. Next, we show the scalability of MixDA by using parallel domain-adapters and only train the MoA layer and task-adapters in Stage 2. The training process in Stage 2 follows the previous experiments. Table 3 shows the comparison across single domainadapter, parallel domain-adapters, and RoBERTa + Pfeiffer on 7 datasets. On average, parallel domainadapters show an improvement of 0.6% over vanilla RoBERTa + Pfeiffer, even though they fall behind the single domain adapter by 1.9%. This could be attributed to the MoA gate choosing the suboptimal domain-adapter for some test data. Still, considering its improvement over Pfeiffer, the MoA gate chooses the correct domain-adapter in most cases. Therefore, MixDA demonstrates its scalability, allowing end users to train Stage 1 on different datasets and combine them later. Overall, in both single and parallel situations, MixDA significantly improves upon the vanilla RoBERTa + Pfeif-
fer model with a small increase in model size. This is due to the ability of MixDA to capture knowledge and the MoA to select useful knowledge for downstream tasks.

heading: 6 Analysis
text: In this section, we analyze the respective contributions of each part of MixDA through detailed analysis, including the Stage 1 training, task-adapters in Stage 2, and the mixture-of-adapters gate.

heading: 6.1 Ablation Study
text: In this section, we conduct an ablation study to reveal the contributions of each part of the model. There are three variants: (1) We remove the MoA gate and choose the domain-adapter instead of the RoBERTa feed-forward layer (–MoA). (2) We exclude old-domain knowledge during Stage 1 (–Old). (3) To examine whether the training procedures, rather than the MixDA structure, contribute the most to our results, we conduct Stage 1 and Stage 2 training only with task-adapters (–DA). Table 4 shows the results of the ablation study. As expected, the average performance drops in all three settings. Without MoA gate, old-domain knowledge FFNs, and structure knowledge, it is observed a drop of 5.2%, 7.5%, and 12.4%, respectively, showing that the MoA gate, the old-domain knowledge, and the MixDA structure are all fundamental in the model. Relatively, the MoA has the smallest impact because the old-domain knowledge in Stage
1 can also help the model retain the knowledge in RoBERTa. The domain-adapter has the largest impact since it only stores domain knowledge and can keep it during Stage 2. In contrast, conducting Stage 1 and 2 training on the Pfeiffer adapter causes catastrophic forgetting.

heading: 6.2 Structured and Unstructured Knowledge
text: In Section 5, the MixDA is only trained on unstructured knowledge. As a comparison, we also train the domain adapter on ConceptNet, a structured knowledge dataset, and then attach both the unstructured and structured to our model and train the MoA layer and the task-adapter during Stage 2.
Table 5 shows the result of combining structured and unstructured knowledge in Stage 1. FEVER and CSQA, two knowledge-intensive tasks, have the greatest improvement: 10.3% for FEVER and 1.2% for CSQA. This is because ConceptNet stores commonsense knowledge that can help both tasks. Meanwhile, MRPC and STS-B also obtain improvement, showing that ConceptNet can benefit general language tasks as well. In conclusion, the experiment demonstrates the ability of MixDA to utilize structured knowledge, the extensibility of our model, and the possible benefits of structured knowledge.

heading: 6.3 Effectiveness of Task-Adapters
text: In most experiments of this paper, we adopt Pfeiffer as the task-adapter unless otherwise specified. In this section, we test the performance of MixDA combined with other kinds of task-adapters, including Houlsby, Prefix-Tuning, LoRA, and Pfeiffer. Table 6 gives the result of different task-adapters. Pfeiffer surpasses others by at least 6.3%. Even though Houlsby is on par with Pfeiffer, Pfeiffer only requires half the number of newly introduced
parameters compared to Houlsby, making it the optimal choice of task-adapters in our experiment.

heading: 7 Conclusion
text: In this paper, we proposed MixDA, a mixture of adapters for domain adaptation. We first decouple the knowledge modules (i.e., FFNs) into the old-domain and domain-specific FFNs. Then we propose a two-stage adapter tuning strategy: first tuning the domain adapter on each domain and then tuning the task adapter on each task. Moreover, our model could be scaled to multiple domains easily with the introduction of the mixture-of-adapters gate. Empirically, MixDA achieved significant improvement over in-domain tasks, out-of-domain tasks, and knowledge-intensive tasks. Further analyses demonstrate the reliability, scalability, and efficiency of our method.
Limitations
Although MixDA achieves promising results on domain adaptation compared with baseline models, there are certain limitations. MixDA is a two-stage approach, which is not fully end-to-end. Our approach requires training a domain adapter and task adapter, respectively. In the future, we will explore the unifying domain and task adapters by merging them into one.

heading: Acknowledgments
text: We thank the anonymous reviewers for their valuable suggestions. This work was supported by the General Research Fund (GRF) of Hong Kong (No. 16310222 and No. 16201320). Shizhe Diao and Ruijia Xu were supported by the Hong Kong Ph.D. Fellowship Scheme (HKPFS).

heading: A Experimental Setup
text: Our domain adapter has a reduction factor of 16, consisting of two linear layers 4096 ˆ 256 and 256ˆ1024 (1.31M parameters). With each domain adapter also comes a MoA gate which has an FFN with 4096 ˆ 2 (number of MixDAs) parameters. Since domain adapters are placed in Layers 7 and 11, they have 2.6M parameters in total. Therefore, the domain adapters (excluding task-adapters) only add 0.7% additional parameters to RoBERTa-large.
We preprocess the unstructured data in Stage 1 similar to the masked language model directive in RoBERTa. From the text, we choose 15% of tokens uniformly to perform possible alterations. In those tokens, 85% are replaced with <mask>, 10% are left unchanged, and 5% are replaced with a random token. The preprocessing is implemented with DataCollatorForLanguageModeling in Huggingface Transformers. In Stage 2, we use few-shot setting with K “ 16. For each class of the dataset, we randomly select 16 examples before run.
In Stages 1 and 2, we use a linear weight scheduler. All the models are optimized by AdamW (Loshchilov and Hutter, 2017) with weight decay 0.05. The best hyperparameters for Stage 2 are found with grid search, with batch size t2, 4, 8, 16u and learning rate t5e´5, 1e´4, 5e´ 4u. The details can be found in Tables 7 and 8.

heading: B Computational Budget
text: Stage 1 training takes relatively longer time, while Stage 2 is fast due to the few-shot setting. The training time of Stage 1 is proportional to the number of tokens. For reference, with 4 Nvidia RTX 2080Ti, Stage 1 training for Biomed (33.6M tokens) takes ~45min per epoch, and training for Review (7.4M tokens) takes ~5min per epoch. Stage 2 training is generally fast: The 20-epoch training process usually takes less than 5min with 4 Nvidia RTX 2080Ti.

heading: C Details of Datasets
text: We conduct experiments on three types of datasets: in-domain (ID) tasks that require general-domain knowledge; out-of-domain (OOD) tasks that require domain-specific knowledge; knowledgeintensive (KI) tasks that require commonsense knowledge.
For in-domain tasks, we evaluate our model on the GLUE Benchmark (Wang et al., 2018).
It includes MNLI (Williams et al., 2017), CoLA (Warstadt et al., 2019), MRPC (Dolan and Brockett, 2005), SST-2 (Socher et al., 2013), RTE (Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), STS-B (Cer et al., 2017), WNLI (Levesque et al., 2012), QNLI (Rajpurkar et al., 2016), and QQP (Iyer et al., 2017). They are all singlesentence or sentence pair classification tasks except STS-B, which is a regression task.
We also evaluate our model on several out-ofdomain tasks, including ChemProt (Kringelum et al., 2016), RCT (Dernoncourt and Lee, 2017), IMDB (Maas et al., 2011), and Amazon (He and McAuley, 2016). ChemProt is a manually annotated chemical-protein interaction dataset extracted from 5,031 abstractions. RCT is a dataset based on PubMed for sentence classification. IMDB provides 25,000 movie reviews for sentiment analysis. Amazon is a dataset containing product reviews from Amazon, annotated with user ratings.
For knowledge-intensive tasks, we evaluate our model on FEVER (Thorne et al., 2018) and CommonsenseQA (CSQA) (Talmor et al., 2019). FEVER consists of 185,445 claims that correspond to Wikipedia articles and are classified as supported, refuted, and not enough information. CommonsenseQA consists of 12,247 questions with 5 choices, each of which requires commonsense knowledge to predict the correct answers.
For Stage 1, we train domain-adapters with unstructured knowledge related to the dataset following Section 3.1. The unstructured knowledge used is listed in Table 1. We also experiment with structured knowledge in Section 6.2. For Stage 2, we adopt the true few-shot setting following (Perez et al., 2021) to demonstrate the effectiveness of MixDA. For each class of each dataset, we randomly sample K “ 16 examples from the original training set as the new training set, and another different K “ 16 examples as the validation set. The original validation set will be used as the test set. The Pfeiffer adapter is used in Stage 2 unless stated otherwise.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Refer to Section "Limitations" at the end of paper.
7 A2. Did you discuss any potential risks of your work? This paper mainly focuses on improving performance on downstream tasks with external knowledge. We have not found risks of the potential use cases.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Refer to Abstract and Introduction (Section 1).
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Refer to Section 4.1 and Appendix C.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Refer to Section 4.1 and Appendix C.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use publicly available datasets. Non-commercial and academic use of them are approved by their respective authors.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use publicly available datasets. Non-commercial and academic use of them are approved by their respective authors.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use publicly available datasets.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Refer to Section 4.1 and Appendix C.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Refer to Section 4.1, Table 1, and Appendix C.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Refer to Sections 5 and 6.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Refer to Appendix A and B.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Refer to Appendix A.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Refer to Sections 5 and 6.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Refer to Section 4.4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will explore the unifying domain and task adapters by merging them into one