Decouple knowledge from paramters for plug-and-play language modeling
Year: 2023

Abstract:
Pre-trained language models (PLM) have made impressive results in various NLP tasks. It has been revealed that one of the key factors to their success is the parameters of these models implicitly learn all kinds of knowledge during pre-training. However, encoding knowledge implicitly in the model parameters has two fundamental drawbacks. First, the knowledge is neither editable nor scalable once the model is trained, which is especially problematic in that knowledge is consistently evolving. Second, it lacks interpretability and prevents humans from understanding which knowledge PLM requires for a certain problem. In this paper, we introduce PlugLM, a pre-training model with differentiable plug-in memory (DPM). The key intuition is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the DPM. To justify this design choice, we conduct evaluations in three settings including: (1) domain adaptation. PlugLM obtains 3.95 F1 improvements across four domains on average without any in-domain pre-training. (2) knowledge update. PlugLM could absorb new knowledge in a training-free way after pretraining is done. (3) in-task knowledge learning. PlugLM could be further improved by incorporating training samples into DPM with knowledge prompting1.

heading: 1 Introduction
text: Large pre-trained language models (PLM) (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018) have become a revolutionary breakthrough in NLP area. Optimized by carefully designed self-supervised objectives on unlabeled corpus and
∗Corresponding author. 1Code available at https://github.com/Hannibal046/
PlugLM
fine-tuned on downstream tasks, PLMs perform remarkably well in a wide range of NLP benchmarks. Recent studies (Warstadt et al., 2019; Petroni et al., 2019) have revealed that one of the key factors to the success of PLMs is that the parameters of these models implicitly learn various types of knowledge in the pre-training corpus. Owing to these learned syntactic, semantic, factual and commonsense knowledge, PLMs show great understanding, generalization and reasoning abilities in multiple downstream tasks (Rogers et al., 2020; Izacard et al., 2022). As Geva et al. (2021) pointed out, the feed-forward layers (FFN), constituting two-thirds of a transformer model’s parameters, are essentially key-value memories and store all kinds of knowledge of PLM. The first linear layer of FFN acts like a set of sparsely activated keys detecting input patterns while the second is the corresponding value. To aggressively capture more knowledge, larger PLMs are continuously proposed, from 110M BERT (Devlin et al., 2019) to 530B MTNLG (Smith et al., 2022), yet PLM has not reached upper bound (Ouyang et al., 2022).
However, a fundamental question still remains: For PLM, is it the optimal way to implicitly encode knowledge in its parameters? We argue that the implicit knowledge encoding approach has two fundamental drawbacks. First, the learned knowledge is neither editable nor scalable once the model is trained (e.g., BERT doesn’t know what is a BERT). Nevertheless, world knowledge is actually infinite and evolving. We thus would never expect an ever-large model to capture all the knowledge in its parameters and to be continuously retrained for the newly coming one. Second, the current PLMs lack interpretability at the knowledge level. Implicit knowledge encoding fails to provide provenance for model’s prediction and makes PLM a black box preventing humans from understand-
14288
ing which knowledge PLM requires for a certain problem.
In this work, we propose a novel architecture of PLM, PlugLM, which decouples the knowledge storage from model parameters and explicitly leverages the knowledge in an explainable manner. As shown in Figure 1, we balance the functionality of FFN layer with a differentiable plug-in key-value memory (DPM), which is highly scalable as well as editable. Each slot of DPM encodes the knowledge to a pair of key and value, and thus we can explicitly retrieve the required knowledge in natural language from DPM rather than unnamed vectors in FFN.
To justify the design choice of decoupling the knowledge from parameters, we conduct extensive evaluations under different settings. In the domain adaptation setting, PlugLM could be easily adapted to different domains with pluggable indomain memory—obtaining 3.95 F1 improvements across four domains on average and up to 11.55 F1 improvement on ACL-ARC citation intent classification dataset, without any in-domain pre-training. In the knowledge update setting, PlugLM could absorb new knowledge after pre-training is done in a training-free way by knowledge updating operation in the DPM, with an improvement up to 4 F1 scores in LINNAEUS NER dataset. PlugLM could further be improved by incorporating training samples into DPM with knowledge prompting as a kind of in-task knowledge.

heading: 2 Related Work
text: Investigating FFN Feed-forward layers constitute two-thirds of a transformer model’s parameters and are essential to unveil modern PLMs (Geva et al., 2021, 2022). A surge of works have investigated the knowledge captured by FFN (Dai et al., 2022a; Meng et al., 2022; Geva et al., 2021, 2022; Jiang et al., 2020; Yao et al., 2022; Wallat et al., 2021). Based on the view that FFN is essentially an unnormalized key-value memory network, Dai et al. (2022a) detects knowledge neurons in FFN and edit specific factual knowledge without finetuning. Meng et al. (2022) modifies FFN weights to update specific factual associations using RankOne Model Editing. Yao et al. (2022) injects knowledge into the FFN via BM25. Dai et al. (2022b) and Lample et al. (2019) enhance the model by expanding the size of FFN with extra trainable keys and values.
Knowledge-Augmented Language Model There are two lines of works to equip PLM with knowledge. The first is introduce additional Knowledge Graph (KG) and knowledge-based training signal (e.g., entity linking) into the language model pre-training, like ERNIE (Zhang et al., 2019; Sun et al., 2019), KnowBERT (Peters et al., 2019) and KEPLER (Wang et al., 2021). Another line of works adopt retrieval mechanism to incorporate knowledge, either symbolic (Verga et al., 2020; Agarwal et al., 2021; Févry et al., 2020) or texual (Guu et al., 2020; Lewis et al., 2020c; Borgeaud et al., 2022; Lewis et al., 2020a; Verga et al., 2020; de Jong et al., 2022). They formulate the task as retrieve then predict process by using extra neural dense retriever or sparse retriever to find most relevant supporting knowledge and combine it with input using either concatenation (Guu et al., 2020; Lewis et al., 2020c), attention methods (de Jong et al., 2022; Chen et al., 2022) or interpolation (Khandelwal et al., 2020; Zhong et al., 2022)
PlugLM differs from previous works in that we do not try to equip the model with additional knowledge to perform knowledge-intensive tasks. The key insight is to transform FFN architecture into deep retrieval in the interest of decoupling the knowledge which would otherwise be stored in the parameters and this is orthogonal to all retrievalaugmented PLMs.

heading: 3 Preliminary
text: Feed-forward Layers Transformer (Vaswani et al., 2017), the backbone for all PLMs, is made of stacked self-attention (Self-Attn) and feedforward (FFN) layers. The former captures the contextual interaction among inputs and the latter process each input independently. Let x ∈ Rd1 be a vector as input, the FFN could be formulated as:
FFN(x) = σ(x ·W⊤1 ) ·W2 (1)
where W1,W2 ∈ Rd2×d1 and σ is the activation function. The bias term is omitted for brevity.
Key-Value Memory Network The Key-Value Memory Network (Weston et al., 2014; Sukhbaatar et al., 2015) corresponds to d2 key-value pairs and each key/value is a vector in Rd1 . They are the generalization of the way knowledge is stored (Eric et al., 2017; Miller et al., 2016). For an input x ∈ Rd1 , there are two stages for a key-value memory
network. First, the lookup (addressing) stage would compute the matching degree between x and each key. In the second stage, x would be transformed by the weighted sum of values according to the distribution of the matching degree in the first stage. We can formally define it as:
MemoryNetwork(x) = softmax(x ·K⊤) ·V (2) where K,V ∈ Rd2×d1 . Comparing equation (1) and (2), we could find that the FFN is an unnormalized version of MemoryNetwork. The keys in FFN are pattern detectors and would be activated only when certain patterns occur in the input. This explains how FFN stores knowledge in a key-value manner (Geva et al., 2021; Sukhbaatar et al., 2019).

heading: 4 PlugLM
text: The overall architecture of PlugLM is illustrated in Figure 1. Because FFN is essentially a key-value memory network (Geva et al., 2021; Dai et al., 2022a; Meng et al., 2022), PlugLM creatively decouples the knowledge storage from model parameters by replacing2 FFN with a Differential Plug-in key-value Memory, DPM (§4.1) and conducting knowledge retrieval in DPM with knowledge attention (§4.2) for explicit knowledge usage instead of storing all knowledge implicitly in the model parameters. In §4.3, we detailedly explain how PlugLM is trained in both pre-training and finetuning stages.
2Because different layers in transformer capture different knowledge, the lower layer for shallow patterns while the upper layers for more semantic ones (Geva et al., 2021; ?), we only consider replacing FFN in Top-L layers with DPM while keeping FFN in the lower layers untouched to encode the intrinsic language understanding knowledge as detailed in §5.4.

heading: 4.1 Differential Plug-in Memory
text: In this paper, we view n-th knowledge dn = {t1n, t2n, ..., t|dn|n } as consecutive tokens from unlabeled corpora as in Guu et al. (2020). For each dn, we get its dense representation hn from a knowledge encoder KnowEncoder(·):
hn = AttnPooling(EToken(dn) + EPos(dn)) (3)
where AttentivePooling function (Xu et al., 2021; Cheng et al., 2023a) corresponds to a trainable pattern detector aggregating information from a sequence of input. And EToken and EPos denote token embedding and positional embedding. Then we use two independent mapping functions to project hn to the key space and value space:
kn = Wk · hn + bk (4) vn = Wv · hn + vk (5)
where Wk, Wv, bk and vk are trainable parameters. And DPM is a triplet of ⟨D,K,V⟩:
D = {d1, d2, ..., d|D|} (6) K = {k1, k2, ..., k|D|} (7) V = {v1, v2, ..., v|D|} (8)

heading: 4.2 Memory Fusion
text: For hidden states h ∈ Rl×d from Self-Attn, FFN would transform h with unnormalized key-value memory as in Equation (1). Our key insight is that instead of interacting with unnamed vectors in FFN, we conduct Maximum Inner Product Search (MIPS) to retrieve knowledge in natural language from ⟨D,K,V⟩ where each triplet corresponds to one knowledge along with its key and
value representation. For h, we first get its sentencelevel representation z by an attentive pooling function z = AttentivePooling(h), then we use z as the query vector to ⟨D,K,V⟩. Since PLM is internally sparse (Li et al., 2022), we only consider Top-N knowledge Dz with corresponding keys Kz and values Vz:
Kz = Top-N(MIPS(z,K)) (9) Vz = {vi if ki in Kz} (10) Dz = {di if ki in Kz} (11)
where Top-N also corresponds to the indexing operation. With Kz and Vz , we use knowledge attention to fuse retrieved knowledge into our model:
Attention(h,Kz,Vz)=softmax( hK⊤z√
d )Vz (12)
where d is the head dimension. By knowledge retrieval and fusion, we explore an interpretable way to incorporate knowledge into the model where Dz is the actual knowledge that PLM would leverage. And direct modification on D without changing model parameters empowers PlugLM with much flexibility and scalability in domain adaptation (§5.1) and knowledge update (§5.2) scenarios.

heading: 4.3 Training
text: The backbone of our model is a multi-layer bidirectional transformer encoder (Devlin et al., 2019). There are two phases in our framework: pretraining and fine-tuning. In the pre-training phase, to make the whole training process end-to-end trainable, we use asynchronous index refreshing to optimize our model as done in Guu et al. (2020) and Cai et al. (2021). Concretely, we update the indices of DPM every T steps. The MIPS results are based on the stale index while the scores of selected TopN results are recomputed using KnowEncoder(·) which facilitates the gradient flow back to memory. The training objective is Masked Language Modeling (Devlin et al., 2019) where we randomly mask tokens in a sentence and ask PlugLM to predict it. In the pre-training phase, Wikipedia is chosen as the source of knowledge and in the domain adaptation fine-tuning stage, corpora from other domains are treated as knowledge sources detailed in §5.1. More details are shown in Appendix A. In the finetuning phase, the K and V of DPM are fixed, and we view it as an editable and scalable knowledge lookup table.

heading: 5 Experiments
text: PlugLM mainly tries to decouple the knowledge storage from parameters and leverage knowledge in an explainable way. We conduct comprehensive experiments to show the superiority of this novel architecture: we could easily adapt the model to different domains without in-domain pre-training by switching DPM (§5.1.1 and §5.1.2), alleviate catastrophic forgetting by storing DPM (§5.1.1), inject new knowledge into the model by enlarging DPM (§5.2), further enhance the model by injecting in-task knowledge into DPM (§5.3) and unveil the black-box PLM with direct access to the knowledge retrieved from DPM (Appendix D). We also carefully examine each key design in PlugLM and point the direction for future work in §5.4.

heading: 5.1 Domain Adaptation
text: Learning robust and transferable representation has been the core of language model pre-training (Peters et al., 2019). For the general-purposed PLM to generalize well on domain-specific tasks, endowing the model with domain knowledge via indomain training remains the go-to approach (Gururangan et al., 2020; Whang et al., 2020; Zhang et al., 2020; Li et al., 2023). In this section, we show that without any in-domain pre-training, PlugLM could flexibly adapt to multiple domains with domainspecific DPM. For the existing PLM encoding knowledge in parameters, this is a challenging task in that it can not guarantee the generalization across multiple domains due to catastrophic forgetting (Kirkpatrick et al., 2016) and sometimes it is even computationally unaffordable to keep training the super large models (Smith et al., 2022; Brown et al., 2020).
We consider two adaptation scenarios: domain adaptive post-training (§5.1.1) and in-domain pretraining (§5.1.2). The former is conducted after PLM was trained on the general domain and the latter trains a domain-specific PLM from scratch.

heading: 5.1.1 Domain Adaptive Post-Training
text: Experimental Setting Following Gururangan et al. (2020), we conduct experiments on four domains: BIOMED, CS, NEWS and REVIEWS across eight domain-specific downstream tasks, in both low and high resource settings. More details can be found in Appendix B. When fine-tuning, we pass the final [CLS] representation to a task-specific head as in Devlin et al. (2019).
We have the following baselines: WikiBERT uses the architecture of BERTbase (Devlin et al., 2019) and is pre-trained on Wikipedia. To adapt WikiBERT to other domains, we use DAPT following the training setting in Gururangan et al. (2019). REALM (Guu et al., 2020) and PlugLM are models that have an external knowledge base and can be simply adapted to other domains with a different base. We have two adaptation strategies: DAA, short for Domain Adaptive Addition, appends domain knowledge to the knowledge base, and DAR, Domain Adaptive Replacement, replaces general knowledge with domain-specific knowledge in the knowledge base.
We also include the results of ¬DAPT, ¬DAA and DACT. The former two use irrelevant domain corpora for post-training and knowledge base construction, which are used to test the robustness of the adaptation method and rule out the factor that improvements might be attributed simply to exposure to more data3. For DACT, Domain Adaptive Continual Training, we sequentially use DAPT for WikiBERT in multiple domains in the hope that it can capture and store knowledge from various domains in a lifelong learning way (Rostami, 2021).
3Following Gururangan et al. (2020), we use the following irrelevant domain mapping: for NEWS, we use a CS LM; for REVIEWS, a BIOMED LM; for CS, a NEWS LM; for BIOMED, a REVIEWS LM.
Experimental Results The results are shown in Table 1. The Avg.Cost is the cost for adaptation measured by hour. For WikiBERT, it’s the time to post-train model in domain-specific corpus. For REALM and PlugLM, it is the time to encode domain knowledge into the knowledge base. We can observe: (1) In-domain training helps model better generalize to tasks requiring domain knowledge while irrelevant knowledge misleads the model and causes performance degradation. And by comparing ¬DAPT and ¬DAA, it shows that models with external knowledge base (PlugLM and REALM) are more robust when faced with noisy out-of-domain knowledge. (2) For the model that implicitly encodes knowledge in the parameters, it fails to generalize across domains as the result of DACT indicates. For example, we keep training WikiBERT in NEWS domain after DAPT in CS domain and fine-tune it on the CS downstream tasks. It performs on par with model that is never exposed to CS domain (¬DAPT). PlugLM could alleviate this catastrophic forgetting problem by storing all kinds of knowledge in DPM and using it in a plug-and-play manner. (3) Direct modification on external memory helps PlugLM efficiently and effectively adapt to different domains without in-domain training. In 254× less time compared with DAPT and in 40× less time compared with REALM, PlugLM significantly outperforms DAPT and REALM-based methods.
To further understand PlugLM, in Figure 2, we present a visualization for the distribution of actual retrieved knowledge for DAA, DAR and original PlugLM. A clear pattern here is that with more domain knowledge involved, the model performs better (63.77, 72.51 and 75.32) and remarkably, although pre-trained on the general domain, the PlugLM has managed to learn what to retrieve when there are both general knowledge and domainspecific knowledge in DPM shown in DAA visualization.

heading: 5.1.2 In-domain Pre-Training
text: In-domain pre-training is another line of work for domain-specific PLM training from scratch like BioBERT (Lee et al., 2019), SciBERT (Beltagy et al., 2019) and FinBERT (Araci, 2019).
Experimental Setting In this section, we choose the biomedical domain and compare PlugLM with model in the architecture of BERTbase, pre-trained on the general domain, Wikipedia (i.e., WikiBERT) and pre-trained on the biomedical domain, Pubmed (i.e., PubmedBERT). The statistics of datasets and pre-training details are listed in Appendix F. We test two kinds of abilities of these PLMs. First, we test how they perform in biomedrelevant downstream tasks. Specifically, we conduct experiments on eight representative biomedical NER datasets which aim at recognizing domainspecific proper nouns in the biomedical corpus. Then we test their general language understanding ability in GLUE (Wang et al., 2019) and SQUAD (Rajpurkar et al., 2016, 2018). For SQUAD and GLUE, the DPM is constructed from Wikipedia, and for biomedical NER, DPM is from PubMed (Canese and Weis, 2013).
Experimental Results The results are shown in Table 3. Both pre-trained on the Wikipedia, PlugLM outperforms WikiBERT in 8/8 NER tasks with average 1.75 F1 scores by simply switching the knowledge domain of DPM. PlugLM also gives comparable results with PubmedBERT in BC4CHEMD, JNLPBA and LINNAEUS datasets. Although PubmedBERT works well for biomedical tasks, it shows less general language understanding ability and underperforms WikiBERT and PlugLM in GLUE (Table 4) and SQUAD (Table 2), especially in low resource scenario (i.e., RTE, COLA and MRPC datasets). With DPM, PlugLM shows great flexibility and performs well in both general domain and biomedical domain. In Appendix D, we give concrete cases of PlugLM with respect to the retrieved knowledge.

heading: 5.2 Knowledge Update
text: Since the world is not fixed as a snapshot once the pre-training corpus is collected, the current PLM, no matter how large it is, fails to adapt to this changing world. For colossal PLMs like GPT-3 (Brown et al., 2020) and MT-NLG (Smith et al., 2022), efficiently fine-tuning for downstream tasks remains an open challenge, let alone re-training it on the newly coming knowledge.
Experimental Setting In this section, we show that PlugLM can efficiently absorb new knowledge by updating the ⟨D,K,V⟩ without re-training. We
consider the following two settings. (1) We only pre-train PlugLM with limited data and gradually enlarge the DPM with unseen knowledge when fine-tuning. (2) We pre-train PlugLM with full general-domain data and ask the model to perform domain adaptation in DAR manner by gradually increasing domain knowledge in ⟨D,K,V⟩.
Experimental Results The results are shown in Figure 3a and 3b. For the first setting, we test on QA (SQUAD) and Sentiment Classification tasks (SST-2). Both WikiBERT and PlugLM are pre-trained with only 1/4 Wikipedia corpus. We have the following observations: (1) PlugLM trained with limited data already outperforms WikiBERT in both tasks (0.39 EM in QA and 0.59 Accuracy in classification) which verifies the effectiveness of PlugLM in low-resource setting; (2) A consistent pattern across two tasks verifies PlugLM could absorb new knowledge simply by adding more slots in ⟨D,K,V⟩ without heavy re-training.
For the second setting, Figure 3c shows our model can absorb new cross-domain knowledge under adaptation setting. It achieves a higher F1 score on the LINNAEUS NER dataset with increasingly more biomed-specific knowledge injected.

heading: 5.3 In-task Knowledge
text: Inspired by in-context learning (Brown et al., 2020) and example-augmented generation (Cheng et al., 2022, 2023b), the training samples can also be
viewed as a kind of in-task knowledge. In this section, we broaden the scope of DPM knowledge by including the training samples.
Experimental Setting Since the knowledge from Wikipedia is a textual description from domain experts while the training sample from a Questionanswering NLI dataset is in the form of [Q, A, Label], this surface form distribution shift may affect the knowledge retrieval. We consider the following injection methods. (1) Concate. We directly concatenate each training sample as a long string in the form of “Q [SEP] A [SEP] Label" and append this to DPM. (2) Tagged. To build the connection between model inputs and DPM, we tag each training sample by prepending a special token ([Tagged]), and use these tagged samples in both DPM and as model input. (3) Knowledge Prompting. Inspired by prompting method (Liu et al., 2021; Schick and Schütze, 2021), we transfer in-task knowledge to knowledge in the form of Wikipedia by a natural language prompting. For example, in QNLI dataset, we transform [Q, A, Label] with the following prompting: “The first sentence (doesn’t) entail(s) with the second. The first sentence is [Q] and the second is [A]". We choose moderate-sized QNLI and QQP tasks because in-task knowledge injection doesn’t apply to low-resource setting in our preliminary experiments.
Experimental Results The result is shown in Table 5. We can observe that PlugLM has managed to learn from in-task knowledge and the surfaceform of knowledge affect the model performance. Concatenation of training sample fails to inform PlugLM the actual in-task knowledge (zero retrieval in QNLI) and building connection between data and knowledge by a special tagged token only gives minor improvements. Instead, a welldesigned knowledge prompting can help PlugLM learn task-specific knowledge.

heading: 5.4 Tuning PlugLM
text: We investigate how each key design affects the performance of PlugLM. (1) Number of Retrieved Knowledge. Figure 4 shows the effects of different N in STS-B dataset and the sparsely activated Top-5 knowledge proves to be optimal. (2) Layers equipped with DPM. Considering that the upper layers in PLM capture more semantic information (Geva et al., 2021), we equip the last encoder layer with DPM in PlugLM. Figure 4 shows that increasing DPM-enhanced encoder layer gives minor improvements but brings much latency because of extra MIPS search. (3) FFN and DPM. To further explore the relation between FFN and DPM, we propose two model variants. First, we replace FFN in all encoder layers with a shared DPM denoted as PlugLM All. Then we fuse FFN and DPM by modifying the model architecture from LayerNorm(h+KnowAttn(h,Kh′ ,Vh′))
to LayerNorm(h + KnowAttn(h,Kh′ ,Vh′) + FFN(h)) and we name it PlugLM Fuse. The Spearman correlation (more results are shown in Appendix E) in STS-B dataset for WikiBERT, PlugLM All, PlugLM and PlugLM Fuse is 88.64, 86.82, 89.20 and 89.10. We could find that PlugLM All, where there is no FFN, underperforms WikiBERT. And PlugLM performs comparably with PlugLM Fuse. We conjecture that FFN in different layers may play different roles, which is also reported in Geva et al. (2021). For the upper layer which captures more semantic knowledge (Jawahar et al., 2019), DPM is a flexible and extensible substitution of FFN, but for lower layers, shallow features should be captured in the model parameters.

heading: 6 Conclusion
text: For the first time, we challenge the current implicit knowledge encoding mechanism for PLMs with two fundamental drawbacks and insightfully propose to decouple knowledge storage from model parameters with an editable and scalable key-value memory. Inspired by the findings that FFN stores all kinds of knowledge and is essentially a keyvalue memory network, we transform FFN archi-
tecture into deep retrieval with a differentiable plugin memory (DPM), which makes the knowledge encoding of PLMs more flexible and interpretable. Extensive experimental results in different scenarios including domain adaptation, knowledge update and in-task knowledge learning verify the design choice of PlugLM. We believe this architectural design would pave a new direction for future research on PLM, especially for super-large PLM.
Limitations
We discuss the limitations of PlugLM as follows: (1) Despite the strong performance achieved by
our approach with DPM, it results in a reduced inference efficiency at the same time due to the MIPS search. For example, PlugLM is about two times slower than pure transformer-based models in GLUE. This would be more crucial when the external memory is much larger. Potential solutions to this issue include (1) constructing the memory using a coarser granularity (Borgeaud et al., 2022); (2) compressing DPM by semantic clustering as in Tay et al. (2022) or knowledge summarization as in Xu et al. (2022).
(2) In this paper, we choose Wikipedia for DPM construction and PlugLM pre-training. While Wikipedia is the most commonly used data source for language model pre-training (Devlin et al., 2019; Liu et al., 2019), there are also many other types of knowledge not covered in Wikipedia, and how to integrate different types of knowledge (e.g., factual, commonsense, syntactic and semantic knowledge) into our framework remains under-explored.
(3) Although this paper proposes a general architecture that is applicable to PLMs of all kinds and sizes including bidirectional (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019), unidirectional (Radford et al., 2018, 2019; Brown et al., 2020) and encoder-decoder-based PLM (Lewis et al., 2020b; Raffel et al., 2020; Song et al., 2019), we only experiment with bidirectional models in moderate size. In particular, we believe this architectural design would be greatly beneficial for LLM (Smith et al., 2022; Chowdhery et al., 2022; Ouyang et al., 2022) for the following reasons: (1) the parameters of LLM could not be easily updated once the pre-training is done due to the unaffordable training cost. (2) the additional latency cost by MIPS retrieval is negligible compared with that of the whole LLM.

heading: Acknowledgement
text: This work was supported by National Natural Science Foundation of China (NSFC Grant No. 62122089), the National Key Research and Development Program of China (No. 2021YFC3340304).

heading: A PlugLM Pretraining Details
text: The details of PlugLM pre-training is shown in Table 6

heading: B Data for Domain Adaptive Post-Training
text: The detailed statistics of domain corpora for posttraining is listed in the Table 7 and downstream tasks in Table 8.

heading: C Latency
text: In Table 9, we show the detailed latency of WikiBERT and PlugLM.

heading: D Case Study
text: We show three concrete examples from QNLI and ACL-ARC in Table 13,14,15.

heading: E More Experiments for Tuning PlugLM
text: In Table 10, we show more results in Section 5.4 on STS-b, MRPC and QNLI.

heading: F Details for Wikipedia and Pubmed
text: The source and size of Wikipedia and Pubmed are shown in Table 11. And hyper-parameters for WikiBERT and PubmedBERT pre-training is shown in Table 12.
Question Answer Prediction Label
How much of Jacksonville is made up of water? According to the United States Census Bureau, the city has a total area of 874.3 square miles (2,264 km2), making Jacksonville the largest city in land area in the contiguous United States; of this, 86.66% (757.7 sq mi or 1,962 km2) is land and ; 13.34% (116.7 sq mi or 302 km2) is water. Entailment Entailment
Input Prediction Label
Various approaches for computing semantic relatedness of words or concepts have been proposed , e.g. dictionary-based ( Lesk , 1986 ) , ontology-based ( Wu and Palmer , 1994 ; Leacock and Chodorow , 1998 ) , information-based ( Resnik , 1995 ; Jiang and Conrath , 1997 ) or distributional ( Weeds and Weir , 2005 ). Background Background
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: the last section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? code will be released when published

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 5
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 5
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. appendix B
C 3 Did you run computational experiments? section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We believe this architectural design would pave a new direction for future research on PLM, especially for super-large PLM. Limitations
We discuss the limitations of PlugLM as follows: (1) Despite the strong performance achieved by
our approach with DPM, it results in a reduced inference efficiency at the same time due to the MIPS search. For example, PlugLM is about two times slower than pure transformer-based models in GLUE. This would be more crucial when the external memory is much larger. Potential solutions to this issue include (1) constructing the memory using a coarser granularity (Borgeaud et al. , 2022); (2) compressing DPM by semantic clustering as in Tay et al. (2022) or knowledge summarization as in Xu et al. (2022). (2) In this paper, we choose Wikipedia for DPM construction and PlugLM pre-training. While Wikipedia is the most commonly used data source for language model pre-training (Devlin et al. , 2019; Liu et al. , 2019), there are also many other types of knowledge not covered in Wikipedia, and how to integrate different types of knowledge (e. g. , factual, commonsense, syntactic and semantic knowledge) into our framework remains under-explored. (3) Although this paper proposes a general architecture that is applicable to PLMs of all kinds and sizes including bidirectional (Devlin et al. , 2019; Liu et al. , 2019; Yang et al. , 2019), unidirectional (Radford et al. , 2018, 2019; Brown et al. , 2020) and encoder-decoder-based PLM (Lewis et al. , 2020b; Raffel et al. , 2020; Song et al. , 2019), we only experiment with bidirectional models in moderate size. In particular, we believe this architectural design would be greatly beneficial for LLM (Smith et al. , 2022; Chowdhery et al. , 2022; Ouyang et al. , 2022) for the following reasons: (1) the parameters of LLM could not be easily updated once the pre-training is done due to the unaffordable training cost. (2) the additional latency cost by MIPS retrieval is negligible compared with that of the whole LLM