Plug-and-Play Document Modules for Pre-trained Models
Year: 2023

Abstract:
Large-scale pre-trained models (PTMs) have been widely used in document-oriented NLP tasks, such as question answering. However, the encoding-task coupling requirement results in the repeated encoding of the same documents for different tasks and queries, which is highly computationally inefficient. To this end, we target to decouple document encoding from downstream tasks, and propose to represent each document as a plug-and-play document module, i.e., a document plugin, for PTMs (PlugD). By inserting document plugins into the backbone PTM for downstream tasks, we can encode a document one time to handle multiple tasks, which is more efficient than conventional encoding-task coupling methods that simultaneously encode documents and input queries using task-specific encoders. Extensive experiments on 8 datasets of 4 typical NLP tasks show that PlugD enables models to encode documents once and for all across different scenarios. Especially, PlugD can save 69% computational costs while achieving comparable performance to state-of-the-art encoding-task coupling methods. Additionally, we show that PlugD can serve as an effective post-processing way to inject knowledge into task-specific models, improving model performance without any additional model training. Our code and checkpoints can be found in https://github.com/ thunlp/Document-Plugin.

heading: 1 Introduction
text: In recent years, large-scale pre-trained models (PTMs) (Devlin et al., 2019; Raffel et al., 2020) have been widely adopted and achieved breakthrough performance for document-oriented NLP tasks, such as question answering. However, due to the tight coupling of document encoding and concrete tasks, PTMs have to dynamically generate document representations according to specific tasks and queries, leading to the repeated encoding
∗Corresponding authors.
Where did Shakespeare born?
Warwickshire
Document PluginPTM Pre-trained Model
PTM
Slot Filling Question Answering
Document Encoding
QA
<Shakespeare, spouse, ___ >
Anne Hathaway
SF
Fact Verification
QA Task-specificModels
Shakespeare was an playwriter.
FV
Yes
Figure 1: Illustration of plug-and-play document modules. Document encoding is decoupled from concrete tasks. By plugging document plugins into task-specific models, we can handle multiple tasks such as question answering, fact verification, and slot filling.
of the same documents in different applications. For example, Wikipedia documents are commonly used in various knowledge-intensive tasks such as question answering (Chen et al., 2017), fact verification (Thorne et al., 2018), and dialogue generation (Dinan et al., 2019). In this case, existing methods separately encode one document for each task or even for each input query (e.g., a question for question answering, a claim for fact verification), making them highly computationally inefficient. To this end, it raises a natural question: can we decouple document encoding from concrete tasks, encoding documents only once and with guaranteed transferability across multiple tasks?
For this question, we propose a novel framework based on PTMs to decouple document encoding from tasks, named PlugD. Specifically, PlugD incorporates plug-and-play modules to store document information and utilizes a PTM backbone to capture information from plugins for task reasoning. As shown in Figure 1, documents are encoded into pluggable plugins once and for all before task
15713
adaptation. The semantics and knowledge of documents can be injected into task-specific models by plugging document plugins. During task reasoning, the task-specific models can activate the information encoded in the document plugins to handle the input queries. In this way, PlugD can decouple the document encoding from downstream task reasoning and reduce the computation costs.
For representing documents as pluggable modules, there are two main challenges: (1) Plugin learning: The document plugins must be effective for various downstream tasks, requiring them to contain sufficient semantics and knowledge. (2) Plugin utilization: Once the document plugins are ready, it is important for task-specific models to capture relevant information from them effectively for task reasoning.
As for plugin learning, we adopt a selfsupervised method, which requires document plugins to provide sufficient knowledge for the PTM to make predictions. Specifically, for each document, we first randomly select parts of sentences as a query and use the remaining sentences as context to learn plugins. Then, after encoding the context into plugins, the model is required to recover the masked recurring spans or generate the next sentences for the query based on the plugin knowledge.
As for plugin utilization, we propose two strategies to utilize document plugins for downstream tasks: plugging during tuning and plugging after tuning1. For plugging during tuning, document plugins are utilized in both tuning and inference stages. Task data and document plugins are combined together to train task-specific models. For plugging after tuning, document plugins are only utilized in the inference stage to provide external knowledge. Document plugins are adopted as a post-processing way to inject knowledge into task-specific models without additional training.
To verify the effectiveness of our plug-and-play framework, we adopt Wikipedia as our document collection and conduct experiments on 8 datasets of 4 typical knowledge-intensive NLP tasks. The results show that we can generate document plugins once and successfully adapt plugins to various downstream tasks. Compared to competitive baselines that encode documents and task-specific inputs simultaneously, our plugin-based method can save 69% computational costs with comparable
1Here tuning refers to tuning PTMs for downstream tasks, including full-parameter fine-tuning and parameter-efficient tuning.
performance. Besides, utilizing document plugins works as an effective post-processing approach to introducing the knowledge of documents into task-specific models and achieving performance improvements without model training. We argue that with the current trend of increasing the model size of PTMs, decoupling document encoding from concrete tasks like PlugD can be a promising direction that enables large PTMs to effectively and efficiently serve diverse downstream tasks.

heading: 2 Related Work

heading: 2.1 Plug-and-Play Modules for PTMs
text: Recent PTMs have shown to be effective in various downstream tasks (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Radford et al., 2018; Brown et al., 2020; Han et al., 2021; Chowdhery et al., 2022). However, training and tuning largescale PTMs for ever-increasing tasks is expensive in computation and storage. To address this issue, building plug-and-play modules with various capabilities for PTMs has received increasing attention recently. For instance, parameter-efficient tuning, which is also known as delta tuning, is proposed to perform task adaptation by fine-tuning only small amounts of parameters and keeping other parameters fixed (Zaken et al., 2022; Houlsby et al., 2019; Lester et al., 2021; Liu et al., 2021; Hu et al., 2021; Ding et al., 2022). The task-specific modules possess play-and-play characteristics and can effectively inject task ability into PTMs. Besides, some researchers explore combining pluggable modules with large-scale PTMs for efficient controllable text generation (Dathathri et al., 2020; Madotto et al., 2020; Pascual et al., 2021), domain adaptation (Chronopoulou et al., 2022; Pfeiffer et al., 2020), information retrieval (Shi et al., 2023; Yu et al., 2023), knowledge injection (Zhang et al., 2023), model debias (Lauscher et al., 2021), and model integration (Xu et al., 2023; Alayrac et al., 2022). Owing to the powerful abilities of largescale PTMs, these modules can effectively activate the model’s capabilities with limited parameters. Different from previous functional modules, we attempt to build document plugins to provide knowledge and context information for PTMs.

heading: 2.2 Language Representation Learning
text: Language representation learning is a fundamental NLP task (Bengio et al., 2013; Devlin et al., 2019; Radford et al., 2018) that aims to effectively
represent rich semantics distributed in text and benefit various downstream tasks. Previous efforts attempt to map the language inputs into intermediate distributed features, such as word embeddings (Mikolov et al., 2013; Kiros et al., 2015; Pennington et al., 2014; Peters et al., 2018), sentence embeddings (Conneau et al., 2017; Reimers and Gurevych, 2019; Gao et al., 2021), and document embeddings (Dai et al., 2015; Wu et al., 2018), which are further used as inputs of downstream task-specific models to generate the final task-specific document representations. Furthermore, some researchers make preliminary exploration to decouple document encoding from tasks by freezing the part of layers of document encoders (Du et al., 2020; Saad-Falcon et al., 2022). But these works only achieve semi-decoupling of document encoding from tasks, and can only be used for the plugging during tuning setting.
Notably, many efforts have been devoted to exploring the effective architectures, such as sparse attention, of PTMs to encode long documents (Beltagy et al., 2020; Zaheer et al., 2020; Zhang et al., 2021; Mehta et al., 2022; Tay et al., 2022). These works are parallel to ours, and we can adopt sparseattention layers to further improve efficiency.

heading: 3 Methodology
text: In this section, we will first present the paradigm description and the overall framework of PlugD Then we introduce the self-supervised plugin learning method to make document plugins contain sufficient semantics and two strategies about how to utilize document modules.

heading: 3.1 Plug-and-Play Document Modules
text: In this paper, we focus on decoupling document encoding with specific tasks. Different from encoding-task coupling methods which simultaneously encode the documents and task-specific queries, PlugD aims to encode documents once and for all before task adaptation. Specifically, given a PTM backbone M and a document d, we first use the PTM to encode the document into a task-agnostic pluggable module, D, i.e., a document plugin. Equipped with the document plugin, the PTM is injected into the corresponding document knowledge. Then we adopt task data to tune the PTM to obtain task-specific models. During inference, we can quickly obtain predictions for an input query by inserting the relevant document
plugin into the task-specific models, avoiding reencoding the document.

heading: 3.2 Overall Framework
text: As shown in Figure 1, we design PlugD, which consists of three components: a PTM backbone, document plugins that provide document knowledge, and task-specific models derived from the PTM to handle specific tasks. We will present these components below.
PTM Backbone. PTMs have been proven effective in a wide range of downstream tasks, and raise a paradigm shift to solve multiple tasks with one unified model (Bommasani et al., 2021; Brown et al., 2020; Chowdhery et al., 2022). In view of this, we further explore the decoupling of document encoding and tasks, unifying document representations across tasks. PlugD relies on a large-scale PTM, which can serve as a fundamental infrastructure to learn plugins from documents and as an initialization for task-specific models. Note that, for our framework, any PTM with large-scale parameters can be used as the backbone. Specifically, we adopt a widely-used sequence-to-sequence PTM, T5 (Raffel et al., 2020), in this paper. As the pretraining objectives of the PTM do not involve the document plugins, we further conduct plugin learning for the PTM so that it can generate and utilize document plugins. The training tasks are introduced in the following sections.
Document Plugin. Document plugins store document knowledge and are obtained before utilizing these documents for specific tasks. Inspired by recent progress in model interpretation (Petroni et al., 2019; Jiang et al., 2020; Roberts et al., 2020; Dai et al., 2022; Mitchell et al., 2022), which claims that the parameters of PTMs store vast amounts of knowledge, we propose to encode the semantics and knowledge of documents into pluggable parameters. In this way, when the document plugin is inserted into the PTM, the PTM is empowered with the corresponding document knowledge.
Inspired by prefix-tuning (Li and Liang, 2021), we represent documents as prefix tokens for attention layers. When the document plugin is inserted into the backbone, we concatenate the corresponding prefix tokens with the hidden vectors of task-specific queries in attention layers to provide document knowledge. Specifically, given a document d with Ld tokens, we first encode the document with the PTM to get the raw document
representations Hd = {h1, ...,hLd}. Then, we adopt a mapping network to project the representation vectors into prefix tokens: Pd = {p1, ...,pLd}, where pi = hi + MLP(hi). The prefix tokens are further inserted into the attention layers. Let Hq = {hq1, ...,hqLq} denote the hidden vectors of query in the attention layer. We calculate the attention output as follows:
Hoq = Attn(HqWq, cat(Pd,Hq)Wk, cat(Pd,Hq)Wv), (1) where Wq, Wk, and Wv are trainable parameters for the self-attention layer. Then Hoq is fed into the feed-forward layer as the original Transformer (Vaswani et al., 2017) layer.
Different from encoding documents during task adaptation or inference, prefix tokens do not involve the computation of feed-forward layers. Moreover, to better integrate the semantics of documents and queries for handling tasks, document plugins are only inserted in the near-top layers of the PTM backbone. Therefore, these document plugins in the form of prefix tokens only increase limited computational requirements, whereas PlugD can achieve a significant computational speedup as a result. Due to the high storage requirement of adding different prefix tokens to different attention layers, we share Pd for all attention layers. Note that, we can also utilize other model structures, such as bias parameter (Zaken et al., 2022) and LoRA (Hu et al., 2021), to represent documents in PlugD, which we leave for future work.
Task-specific Models. Task-specific models are derived from the PTM backbone and tuned on the supervised task data to obtain task reasoning ability. During downstream tuning, we freeze the document plugins, and only the task-specific models and the mapping network of the document plugins are trainable so that the document plugins can be reused across different tasks. We adopt two training methods for task-specific models, including vanilla full-parameter fine-tuning and parameter-
efficient tuning (PET). Note that, deploying largescale PTMs with full-parameter fine-tuning will lead to exacerbated computational and storage burdens for multi-task scenarios. Thus, it is worth exploring PlugD with PET for efficient task adaption in real-world applications.
Both two training methods adopt task-specific objectives to optimize the parameters. Fine-tuning optimizes all parameters of the PTM backbone, while parameter-efficient tuning only optimizes parts of the parameters and keeps other parameters frozen. Specifically, we adopt adapters for parameter-efficient tuning (Pfeiffer et al., 2021). Given the hidden vector h ∈ Rd, where d is the hidden size, the output of the adapter layer is calculated as:
hout = h+ ϕ(hWdown)Wup, (2)
where Wdown ∈ Rd×r, Wup ∈ Rr×d, and r ≪ d refer to the bottleneck dimension.
Computational Complexity. PlugD encodes the documents before task adaptation and thus can reduce the computation costs. In this paragraph, we discuss the computational complexity of PlugD in detail. Assume the lengths of the query and document are Lq and Ld, respectively. For the traditional encoding-task coupling models, which simultaneously encode documents and queries, the computational complexity of the attention layer is O((Lq + Ld)
2), and the computational complexity of the feed-forward layer is O(Lq + Ld). For PlugD, the document plugins are inserted into the attention layer, whose computational complexity is O(Lq(Lq + Ld)). And the document plugins do not involve the computation of the feed-forward layer, and thus its computational complexity is O(Lq). In real-world applications, the documents usually are much longer than the queries. Therefore, PlugD can achieve significant computational speedup compared with conventional encodingtask coupling models.

heading: 3.3 Plugin Learning
text: To enable the document plugins to contain sufficient document knowledge, we futher explore self-supervised plugin learning in this section. As shown in Figure 2, we adopt two self-supervised tasks, recurring span prediction, and next sentence generation to augument the comphrension and generation ability of PlugD. Both two tasks require document plugins to provide context information for the model to make the predictions. Let d = {s1, ..., sn} denote the input document with n sentences. We perform plugin learning as:
Recurring span prediction (RSP). Inspired by Ram et al. (2021), we utilize recurring spans to construct self-supervision signals. Recurring spans occur multiple times in the documents, and usually contain important semantics for document understanding. Masking the recurring spans and requiring the PTM to recover them can help the PTM to capture document semantics. Specifically, we concatenate sentences randomly sampled from the document d as query q, and treat the remaining sentences as context c. Then we generate the document plugin Pc based on c, and replace the recurring spans in q as special mask tokens. The PTM is required to predict the masked spans in q conditioned on Pc. Different from the traditional masked language model task (Devlin et al., 2019; Raffel et al., 2020), which mainly focuses on local information around the masked spans, RSP usually requires the PTM to integrate global information from document plugins.
Next sentence generation (NSG). To enable the document plugins to benefit generation tasks, we adopt NSG as a training task. We first randomly sample three consecutive sentences {si, si+1, si+2} from the document d. The remaining sentences are treated as the context c = {s1, ..., si−1, si+3, ..., sn} to generate the document plugin Pc. Then we regard si as the query, and require the PTM to generate the following two sentences {si+1, si+2} conditioned on Pc.
These two tasks require the PTM to capture both local information from the queries and global information from the document plugins. Therefore, after plugin learning, the PTM is supposed to be able to build informative document plugins and serve as a good initialization for task-specific models to capture knowledge from document plugins. Both two tasks are sequence-to-sequence tasks, and we adopt the negative likelihood as the training ob-
jectives for two tasks. The model is trained in a multi-task fashion, and the final training loss is the sum of the loss of two tasks. During plugin learning, the document plugins are calculated in real time for different documents. All parameters of the PTM are tuned for plugin learning. After that, the document plugins can be calculated and stored for further downstream task tuning and inference.

heading: 3.4 Plugging Strategies
text: To flexibly utilize the document plugins, we propose two plugging strategies:
Plugging during tuning. In this setting, the document plugins are adopted in both the training and inference of task-specific models. Given an instance with the query and document as inputs, we first insert the corresponding document plugin, which is computed before fine-tuning, into the models. Then task-specific models are trained with taskspecific objectives to capture relevant information from the document plugins.
Plugging after tuning. In this setting, the document plugins are adopted only in the inference of task-specific models. Document plugins can provide external knowledge, and serve as a postprocessing method to inject knowledge into taskspecific models. During inference, given an instance, we directly insert related document plugins into the task-specific models to achieve knowledge injection. This setting does not require additional training for existing task-specific models and can be used to flexibly inject textual knowledge.

heading: 4 Experiments

heading: 4.1 Evaluation Settings
text: Datasets. We adopt widely-used Wikipedia articles as our document collection and select typical knowledge-intensive tasks for evaluation. We adopt a typical multi-task benchmark, KILT (Petroni et al., 2021), to evaluate our models. The tasks in KILT are grounded in the same snapshot of Wikipedia pages. In particular, we evaluate PlugD on a fact verification dataset, FEVER (Thorne et al., 2018), four question answering datasets, including Natural Questions (NQ) (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019), a dialogue generation dataset, Wizard of Wikipedia (WoW) (Dinan et al., 2019), and two slot filling dataset, Zero Shot RE (zsRE) (Levy et al., 2017), T-REx (ElSahar et al., 2018). These tasks are diverse and
require the model to exploit document knowledge fully. As shown in the paper of KILT, external document knowledge can not benefit the entity linking task. Thus, we do not use them for evaluation in this paper. Following Petroni et al. (2021), we use dense passage retrieval (Karpukhin et al., 2020) to retrieve relevant documents from Wikipedia articles for each query. Please refer to Appendix for evaluation results of document retrieval.
Metrics. Following previous work, we adopt accuracy for the fact verification task (FEVER) and slot filling tasks (zsRE, T-REx); exact match (EM) and F1 score for the extractive question answering tasks (NQ, HotpotQA, TriviaQA); ROUGE-L (RL) for the long abstractive question answering tasks (ELI5); F1 score for the dialogue generation task (WoW). Besides, to evaluate the overall performance, we calculate average scores for these tasks as an evaluation metric, in which EM scores are used for extractive question answering tasks.

heading: 4.2 Training Details
text: We utilize the widely used T5-large (Raffel et al., 2020), as our PTM backbone. For the PET training method, we set the bottleneck dimension of adapters as 16. We insert document plugins in the last 12 layers. We conduct plugin learning on a large-scale unsupervised corpus, C4 (Raffel et al., 2020) for 36k steps. We use Adam to optimize our models. Due to the high computational costs
of full-parameter fine-tuning, in the following experiments, we adopt the PET method to train the models unless otherwise specified. We train models with a half-precision floating point on 8 NVIDIA A100 GPUs, and the plugin learning process takes 18 hours. Please refer to Appendix for more details.

heading: 4.3 Baselines
text: Plugging during tuning. Here we compare PlugD with several representative baselines, which encode the documents and queries with two different encoders. In this way, these models can reuse document representations across different queries, but they still need to generate different document representations for different tasks. (1) ED2LM (Hui et al., 2022) utilizes the encoder-decoder architecture to encode the queries and documents separately, and then the document can be pre-encoded before inference. In particular, the documents are inputted into the encoder, and queries are inputted into the decoder. (2) EmbRecy (Saad-Falcon et al., 2022) proposes to reuse the intermediate activations of the documents to achieve speedup for finetuning and inference. EmbRecy caches an intermediate layer’s output as the document representation and the remaining near-top layers are tuned to fuse the information of documents and queries. (3) Besides, to meet the setting of decoupling document encoding from tasks, we freeze the document encoders of ED2LM and EmbRecy to make
the document representations unified across different tasks. We denote the two task-agnostic methods as ED2LMf and EmbRecyf . (4) As PlugD conducts further self-supervised training for plugand-play representation learning, we also present the results of PlugD without plugin learning (w/o PT) to show the effectiveness of the architecture of PlugD. (5) UpperBound follows the traditional settings, in which the queries and documents are concatenated together and fed into the model. The document representations generated by this baseline are query-specific. The model needs to encode a single document multiple times for different tasks and different queries, which is the upper bound of task-agnostic methods.
Plugging after tuning. We attempt to inject unstructured textual knowledge into PTMs after downstream tuning. Existing methods mainly focus on enhancing PTMs with structural knowledge during pre-training or fine-tuning (Zhang et al., 2019; Wang et al., 2021; Bosselut et al., 2019). These methods require retraining the task-specific models to achieve knowledge injection, which thus cannot be adopted in this setting. Therefore, we present the results of the following models: (1) We adopt T5 (Raffel et al., 2020) and PlugD as the backbones, which are trained with only the queries as inputs and do not utilize external document knowledge in evaluation. (2) Based on the trained T5 and PlugD, we adopt different post-processing methods to incorporate document knowledge. For T5, we directly concatenate the documents and queries as inputs for evaluation (+Concat). For PlugD, we insert the document knowledge with document plugins (+DPlug). The setting is challenging as there is a gap between the training and evaluation.

heading: 4.4 Plugging during Tuning
text: We present the comparison results between baseline models and PlugD in Table 1. From this table, we can observe that: (1) The baseline models which generate task-agnostic document representa-
tions perform worse than the corresponding models which generate task-specific representations. It indicates that decoupling document representation from concrete tasks is challenging and existing methods cannot achieve satisfactory performance. (2) Compared with the task-agnostic baseline models (ED2LMf and EmbRecyf ), PlugD can achieve significant performance improvements across different tasks. Besides, compared with ED2LM and EmbRecy, PlugD can also achieve superior results on many datasets, especially for parameter-efficient tuning. In addition, ED2LM and EmbRecy need to generate document representations for different tasks separately. Thus they require more storage than PlugD. In contrast, PlugD can generate informative unified representations with fewer storage requirements and achieve superior results across different tasks. (3) Compared with the traditional encoding-task coupling model (UpperBound), sharing document representation across different tasks in PlugD only leads to a limited performance drop (39.68 vs. 41.22, and 40.61 vs. 41.79 on average). And as PlugD does not need to encode documents during downstream tuning and inference, PlugD enables significant computational acceleration. The results suggest that PlugD can effectively capture document semantics and inject them into the PTM to provide knowledge. (4) Even PlugD without further plugin learning can outperform the baselines on several datasets. It proves that PlugD benefits from both the self-supervised tasks and the model architecture. Besides, it also indicates that the contextualized document representations generated by the original PTM (PlugD w/o PT) are powerful if we utilize them correctly.
Computational Cost. We compare the computational cost of PlugD and baseline models. Here, we present the floating point operations (FLOPs) and calculation time required to process one data in inference for each method. We assume that the document, query, and answer contain 512, 48, and 32 to-
kens, respectively. The results are shown in Table 2. From the results, we can observe that: (1) The methods for task-agnostic representation require much less computational cost than encoding-task coupling methods. Especially, our method PlugD can achieve 3.25× speed up (139.3 GFLOPs vs. 453.1 GFLOPs). (2) The methods for task-agnostic representation generally are inferior to encodingtask coupling methods. PlugD can achieve better average scores than other baselines and preserve low computational costs. (3) Both task-agnostic and query-agnostic models need to pre-encode and store document representations before downstream tuning for inference speed up. However, models generating query-agnostic and task-specific representations require separate document representations for each task. In contrast, our PlugD generates task-agnostic representations for all tasks, resulting in better results and lower storage requirements.

heading: 4.5 Plugging after Tuning
text: The comparison results are shown in Table 3. From the results, we can observe that: (1) Both T5 and PlugD cannot achieve consistent improvement from post-processing knowledge injection on these tasks. It proves that plugging after tuning is a challenging setting as there is a gap between training and evaluation. (2) PlugD can achieve significant improvement on FEVER, NQ, and zsRE, which further indicates the effectiveness of PlugD. However, PlugD cannot achieve improvement on WoW. As the ability to acquire knowledge from the document plugins is obtained from plugin learning, further downstream task tuning may lead the models to forget the ability. Thus, even PlugD can not achieve consistent improvement. (3) Without document knowledge, PlugD and T5 achieve comparable results. It indicates that the plugin learning process does not improve the fundamental ability of PTMs. The improvement achieved by PlugD in both plugging during/after tuning settings comes from the effective plug-and-play framework.

heading: 4.6 Ablation Study
text: In this section, we conduct an ablation study to verify the effectiveness of our proposed plugin learning tasks. We show the results of the models, which are trained with only recurring span prediction task (w/ RSP), with only next sentence generation task (w/ NSG), or without plugin learning (w/o PT). We evaluate the models on four datasets for the plugging during tuning setting.
The results are shown in Table 4. We can find that (1) PlugD without plugin learning leads to a significant performance drop, which further indicates that the proposed training tasks can help the PTM to effectively encode the document knowledge into plugins. (2) Two tasks can cooperate with each other to improve the model performance. Though training PlugD with only one task will lead to performance deterioration on some tasks, training with two tasks can achieve consistent improvement over the model without plugin learning. (3) When PlugD is trained with only NSG, the model can achieve superior results for WoW. But the task harms the performance for FEVER and zsRE. This is because NSG requires the model to generate long sentences, which is similar to WoW, while FEVER and zsRE only require short outputs. In contrast, training with only RSP will also lead to a performance drop for WoW. It indicates that diverse plugin learning tasks are important for expressive document plugins.

heading: 4.7 Transferability Analysis
text: In this section, we want to explore the effectiveness of supervised tasks on document representation transferability. Here we present the results of ED2LM, which can outperform other baselines. Specifically, we train the task-specific document en-
coder on a source task, and then reuse the encoder on other target tasks to continually train the rest of the model. The results are shown in Figure 3.
From the results, we can observe that 1) The non-diagonal values of the matrix are consistently smaller than the diagonal values. It suggests that training the document encoder with existing supervised tasks can hardly benefit other target tasks. PlugD trained with two self-supervised objectives can provide transferable document representation and achieve superior results. 2) The encoders trained on the NQ dataset can outperform encoders trained on other tasks. It indicates that training with challenging tasks may lead to better performance.

heading: 5 Conclusion
text: In this paper, we explore a new paradigm, which aims to represent documents as pluggable modules for PTMs. In this setting, we can get rid of encoding the same document multiple times for different tasks. The extensive experiments prove that our proposed PlugD can significantly reduce the computational cost and effectively inject document knowledge into PTMs to improve performance. In the future, we will explore more effective plugin learning tasks and further attempt to represent knowledge graphs, and figures as plugins to provide knowledge for PTMs.
Limitations
We discuss the limitations of PlugD in this section: (1) We explore decoupling document encoding from concrete tasks in this paper, and propose to represent documents as pluggable modules before task adaptation. Therefore, PlugD has a higher storage requirement than conventional encodingcoupling methods. We encourage (2) In the experiments, we adopt T5 as our PTM backbone. Actually, the proposed framework can also be applied to more pre-trained models with various model architectures. Besides, recent trends show that larger models tend to build more expressive text representations. It is worth exploring PlugD with larger PTMs with billions of parameters to learn informative document plugins. (3) In this paper, we adopt an external retriever to retrieve relevant documents for each input query. Recent progress in retrievalaugmented language models shows that training the PTMs with an end-to-end textual knowledge retriever can promote downstream performance. We believe document plugins can also serve as the ex-
ternal knowledge base and enhancing PlugD with end-to-end retrieval is a promising direction.

heading: Acknowledgement
text: This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), National Natural Science Foundation of China (No. 62236004).
Author Contributions Chaojun Xiao and ChiMin Chan wrote the code and conducted the experiments. Chaojun Xiao, Zhengyan Zhang, and Xu Han wrote the initial draft. Yankai Lin, Zhiyuan Liu, and Xiangyang Li significantly edited and improved the paper. Zhonghua Li, Zhao Cao, and Maosong Sun provided valuable advice to the research.

heading: A Appendix
text: A.1 Discussion In this paper, we propose to decouple document encoding from concrete tasks, achieving encoding documents once and for all across different tasks. In this section, we further discuss the potential of PlugD.
Unified Model across Multiple Tasks. Recently, with the rapid progress of large-scale PTMs, handling multiple tasks with a unified PTM has received rapidly increasing attention. For example, many researchers explore instruction tuning (Sanh et al., 2022; Wei et al., 2022) to enable a unified PTM to perform multiple tasks with natural language description. We attempt to extend the paradigm to document representation, enhancing the unified PTM with unified document representation across multiple tasks. In this way, we can provide the PTM with various external knowledge flexibly and efficiently, avoiding encoding documents multiple times for different tasks and user input queries.
Heterogeneous Knowledge Base. Enhancing large-scale PTMs with various knowledge is an important topic for natural language processing. Many researchers attempt to incorporate knowledge graphs (Zhang et al., 2019; Wang et al., 2021), linguistic knowledge (Zhou et al., 2020) into PTMs. We argue that PlugD provides a new way for knowledge injection. We can encode various knowledge, such as images, and knowledge graphs, into the plugins of PTMs. In this way, we can build a heterogeneous plugin knowledge base for PTMs to improve downstream performance.
Continual Learning. Previous researches show that PTMs can implicitly encode knowledge in the model parameters (Petroni et al., 2019; Jiang et al., 2020; Roberts et al., 2020; Dai et al., 2022; Mitchell et al., 2022), which is not editable for continual updates. PlugD provides a new way for the continual learning of PTMs. We can insert and update new knowledge for PTMs by continually learning and updating new document plugins, which will be further utilized to provide knowledge to PTMs.
A.2 Document Retrieval In this paper, we adopt dense passage retrieval, DPR (Karpukhin et al., 2020), to retrieve relevant documents for each input query. Following Petroni et al. (2021), we adopt R-Precision, Precision@k and Recall@k, as the evaluation metrics. We adopt
the evaluation scripts provided by the KILT paper. Please refer to the original KILT paper for the details of the metrics. From the results, we can see that the retrieval performance is not satisfactory for some datasets, which may bring the noise in the downstream tasks. And we encourage the community to develop retrievers, which can achieve satisfactory performance across different tasks.
A.3 Impacts of Insertion Layers
PlugD inserts the document plugins into the selfattention layers to provide document knowledge. As the pre-trained models tend to capture linguistic features in the bottom layers and capture the taskspecific information in the top layers (Rogers et al., 2020). Therefore, to reduce computational costs, we only insert the document plugins in the top layers. In this section, we explore the impact of insertion layers of document plugins. We present the results of PlugD with document plugins inserted in the last 6 layers, 12 layers, and all 24 layers. Here, we do not conduct plugin learning for PlugD to speed up experiments.
The results are shown in Table 6. From the results, we can see that: (1) With the increasing of insertion layers, the performance on FEVER and NQ improves. But PlugD with document plugins in all layers can not outperform the PlugD with document plugins in the top layers on WoW and zsRE. That is because the fact verification and question answering tasks require the models to select useful information via both lexical matching and semantic matching. In contrast, the dialogue generation and slot filling tasks rely on document semantics to provide knowledge, and inserting the document
plugins in the bottom layers can not benefit the performance. (2) The three models can obtain similar performance on these tasks. Therefore, in order to reduce the computational costs and maintain the performance, we only insert document plugins in the top 12 layers for other experiments.
A.4 Impacts of Plugin Sharing across Layers As mentioned in previous sections, PlugD inserts the same prefix tokens for different attention layers to save the storage. In this section, we study the impacts of sharing plugins across different layers. To this end, we attempt to insert different prefix tokens for different layers. Specifically, we encode the document d to obtain the raw hidden state H ld from the l-th layer, and then adopt the mapping network tailored to the l-th layer to map the hidden state into the prefix tokens. The prefix tokens are then inserted into the l-th layer for query encoding. Similar to PlugD, we insert the representations into the top 12 layers for this model. We term the model as All-Hidden.
The comparison results are shown in Table 7. From the results, we can observe that All-Hidden can achieve superior results on three datasets, including FEVER, WoW, and zsRE. But All-Hidden requires 12× storage than PlugD, which is impractical for large-scale document collections. And PlugD can achieve comparable performance to AllHidden. Therefore, to reduce the storage requirement, we choose to share the document plugins across different attention layers.
A.5 Experimental Details Model Implementation. The mapping network of document plugins is used to map the raw document representations into the document plugins for different tasks. Given a hidden vector, hi, we calculate the corresponding prefix token as pi = hi + W 2 mReLU(W 1 mhi), where hi ∈ Rd, W 1m ∈ Rd×2d, W 2m ∈ R2d×d, and d ∈ R is the hidden size.
As for the parameter-efficient tuning method, we adopt adapter layers to tune the model. We add the adapters after the layernorm operation of feed-
forward layers. The parameters of adapters are randomly initialized following a zero-mean Gaussian distribution with standard deviation as 10−2.
Plugin Learning. For the recurring span prediction task, we first identify spans that occur multiple times from the documents. Then we filter out the stopwords and personal pronouns, and keep the longest 15 spans as the recurring spans for further masking. Then we randomly sample 5 sentences, which contain the recurring spans from the document as the query. For the next sentence generation task, we randomly sample three consecutive sentences from the documents, where the first sentence is treated as the query, and the last two sentences are treated as the answers. The model is trained in a multi-task fashion, and 70% documents are used for recurring span prediction, and 30% documents are used for next sentence generation. The maximal length for queries and answers are set as 196 and 128, respectively. We set the learning rate as 2× 10−5 and batch size as 256. Downstream Task Tuning. For downstream tasks, we set the training batch size as 64. The learning rate is selected from {10−4, 5 × 10−4, 10−3} for PET. And as full-parameter finetuning require amounts of computation, we do not conduct grid search for this setting. We set the learning rate for full-parameter fine-tuning as 2× 10−5. For fact verification, we take the claims as the input queries and take the logits of “yes" and “no" for classification. For other tasks, we treat them as text-to-text generation problems, and during the inference, we adopt the greedy strategy for decoding. The evaluation scripts are written by our own, and will be released with the paper.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations
7 A2. Did you discuss any potential risks of your work? The model is designed and evaluated on established tasks and datasets, which should not cause severe risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly
B 3 Did you use or create scientific artifacts? Left blank.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In Section 4.2 Training Details and Appendix A.5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? These tools and datasets are publicly available and free of use for research purposes.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The use is consistent with their intended use.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use established public datasets, which should not cause privacy issues.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. I use the publicly available datasets, and we directly adopt the original split datasets.
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4 and Appendix A.5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4 and Appendix A.5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A.5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In the future, we will explore more effective plugin learning tasks and further attempt to represent knowledge graphs, and figures as plugins to provide knowledge for PTMs. Limitations
We discuss the limitations of PlugD in this section: (1) We explore decoupling document encoding from concrete tasks in this paper, and propose to represent documents as pluggable modules before task adaptation. Therefore, PlugD has a higher storage requirement than conventional encodingcoupling methods. We encourage (2) In the experiments, we adopt T5 as our PTM backbone. Actually, the proposed framework can also be applied to more pre-trained models with various model architectures. Besides, recent trends show that larger models tend to build more expressive text representations. It is worth exploring PlugD with larger PTMs with billions of parameters to learn informative document plugins. (3) In this paper, we adopt an external retriever to retrieve relevant documents for each input query. Recent progress in retrievalaugmented language models shows that training the PTMs with an end-to-end textual knowledge retriever can promote downstream performance. We believe document plugins can also serve as the ex-
ternal knowledge base and enhancing PlugD with end-to-end retrieval is a promising direction