K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

1 INTRODUCTION . Language representation models , which are pre-trained on large-scale text corpus through unsupervised objectives like ( masked ) language modeling , such as BERT ( Devlin et al. , 2019 ) , GPT ( Radford et al. , 2018 ; 2019 ) , XLNet ( Yang et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) and T5 ( Raffel et al. , 2019 ) , have established state-of-the-art performances on various NLP downstream tasks . Despite the huge success of these pre-trained models in empirical studies , recent studies suggest that models learned in such an unsupervised manner struggle to capture rich knowledge . For example , Poerner et al . ( 2019 ) suggest that although language models do well in reasoning about the surface form of entity names , they fail in capturing rich factual knowledge . Kassner & Schütze ( 2019 ) observe that BERT mostly did not learn the meaning of negation ( e.g . “ not ” ) . These observations motivate us to study the injection of knowledge into pre-trained models like BERT and RoBERTa . Recently , some efforts have been made to exploit injecting knowledge into pre-trained language models ( Zhang et al. , 2019 ; Lauscher et al. , 2019 ; Levine et al. , 2019 ; Peters et al. , 2019 ; He et al. , 2019 ; Xiong et al. , 2020 ) . Most previous works ( as shown in Table 1 ) augment the standard language modeling objective with knowledge-driven objectives and update model parameters in a multi-task learning manner . Although these methods , with updated pre-trained models , obtain better performance on downstream tasks , they fail at continual learning ( Kirkpatrick et al. , 2017 ) . Model parameters need to be retrained when new kinds of knowledge are injected , which may result in the catastrophic forgetting of previously injected knowledge . Meanwhile , the resulting pre-trained models produce entangled representations , which makes it hard to investigate the effect of each knowledge when multiple kinds of knowledge are injected . In this paper , we propose K-ADAPTER , a flexible and simple approach that infuses knowledge into large pre-trained models . K-ADAPTER has attractive properties including supporting continual knowledge infusion and producing disentangled representations . It leaves the original representation of a pre-trained model unchanged and exports different representations for different types of infused knowledge . This is achieved by the integration of compact neural models , dubbed adapters here . Adapters are knowledge-specific models plugged outside of a pre-trained model , whose inputs are the output hidden-states of intermediate layers of the pre-trained model . We take RoBERTa ( Liu et al. , 2019 ) as the base pre-trained model and integrate two types of knowledge , including factual knowledge obtained by aligned Wikipedia text to Wikidata triplets , linguistic knowledge obtained by applying off-the-shell dependency parser to web texts . In the pre-training phase , we train two adapters independently on relation classification task and dependency relation prediction task respectively , while keeping the original parameters of RoBERTa frozen . Since adapters have much less trainable parameters compared with RoBERTa , the training process is memory efficient . We conduct extensive experiments on six benchmark datasets across three knowledge-driven tasks , i.e. , relation classification , entity typing and question answering . Experiments show that KADAPTER consistently performs better than RoBERTa , and achieves state-of-the-art performance on five datasets , and comparable performance compared with CosmosQA SOTA . Probing experiments on LAMA ( Poerner et al. , 2019 ) and LAMA-UHN ( Petroni et al. , 2019 ) , further demonstrates that K-ADAPTER captures richer factual and commonsense knowledge than RoBERTa . The contributions of this paper are summarized as follows : • We propose K-ADAPTER , a flexible approach that supports continual knowledge infusion into large pre-trained models ( e.g . RoBERTa in this work ) . • We infuse factual knowledge and linguistic knowledge , and show that adapters for both kinds of knowledge work well on downstream tasks . • K-ADAPTER achieves superior performance by fine-tuning parameters on three downstream tasks , and captures richer factual and commonsense knowledge than RoBERTa on probing experiments . 2 RELATED WORK . Our work relates to the area of injecting knowledge into pre-trained models . As stated in Table 1 , previous works mainly differ from the knowledge sources and the objective used for training . ERNIE ( Zhang et al. , 2019 ) injects a knowledge graph into BERT . They align entities from Wikipedia sentences to fact triples in WikiData , and discard sentences with less than three entities . In the training process , the input includes sentences and linked facts , and the knowledge-aware learning objective is to predict the correct token-entity alignment . Entity embeddings are trained on fact triples from WikiData via TransE ( Bordes et al. , 2013 ) . LIBERT ( Lauscher et al. , 2019 ) injects pairs of words with synonym and hyponym-hypernym relations in WordNet . The model takes a pair of words separated by a special token as the input , and is optimized by a binary classification problem , which predicts whether the input holds a particular relation or not . SenseBERT ( Levine et al. , 2019 ) considers word-supersense knowledge . It inject knowledge by predicting the supersense of the masked word in the input , where the candidates are nouns and verbs and the ground truth comes from WordNet . KnowBERT ( Peters et al. , 2019 ) incorporates knowledge bases into BERT using Knowledge attention and recontextualization , where the knowledge comes from synset-synset and lemma-lemma relationships in WordNet , and entity linking information in Wikipedia . If entity linking supervision is available , the model is learned with an additional knowledge-aware log-likelihood or max-margin objective . WKLM ( Xiong et al. , 2020 ) replaces entity mentions in the original document with names of other entities of the same type . The model is trained to distinguish the correct entity mention from randomly chosen ones . BERT-MK ( He et al. , 2019 ) integrates fact triples from knowledge graph . For each entity , it sample incoming and outcoming instances from the neighbors on the knowledge graph , and replaces head or tail entity to create negative instances . The model is learned to discriminate between real and fake facts . As shown in Table 1 , our model ( K-ADAPTER ) differs from previous studies in three aspects . First , we consider both fact-related objective ( i.e . predicate/relation prediction ) and linguistic-related objective ( i.e . dependency relation prediction ) . Second , the original parameter of BERT is clamped in the knowledge infusion process . Third , our approach supports continual learning , which means that the learning of different adapters are not entangled . This flexibility enables us to efficiently inject different types of knowledge independently , and inject more types of knowledge without any loss on the previously injected knowledge . 3 K-ADAPTER . As illustrated in Figure 1 ( a ) , most of the previous works enhance pre-trained language models by injecting knowledge and update model parameters through multi-task learning . Regardless of these different versions of knowledge-injected methods with multi-task learning , common issues not fully studied are catastrophic forgetting of previous knowledge . To address this , we present KADAPTER as shown in Figure 1 ( b ) , where multiple kinds of knowledge are injected into different compact neural models ( i.e. , adapters in this paper ) individually instead of directly injecting knowledge into pre-trained models . It keeps the original representation of a pre-trained model fixed and supports continual knowledge infusion , i.e. , injecting each kind of knowledge into the corresponding knowledge-specific adapter and producing disentangled representation . Specifically , adapters are knowledge-specific models ( with few parameters ) plugged outside of a pre-trained model . The inputs of adapters are the output hidden-states of intermediate layers of the pre-trained model . Each adapter is pre-trained independently on different tasks for injecting discriminative knowledge while the original parameters of the pre-trained model are frozen . In this paper , we exploit RoBERTa ( Liu et al. , 2019 ) as the pre-trained model , and mainly infuse factual knowledge and linguistic knowledge with two kinds of adapters , i.e. , factual adapter and linguistic adapter which are pre-trained on the relation classification task and dependency relation prediction task respectively . In this section , we first describe the structure of our adapter , and then present the process of pre-training knowledgespecific adapters . 3.1 ADAPTER STRUCTURE . In this work , we present a different adapter structure as shown in Figure 2 , which is referred to as the knowledge-specific adapter . In contrast to Houlsby et al . ( 2019 ) add adapter layers into each transformer layer , our adapter works as outside plug-ins . Each adapter model consists of K adapter layers that contain N transformer ( Vaswani et al. , 2017 ) layers and two projection layers . A skipconnection is applied across two projection layers . Specifically , for each adapter model , we plug adapter layers among different transformer layers of the pre-trained model . We concatenate the output hidden feature of the transformer layer in the pre-trained model and the output feature of the former adapter layer , as the input feature of the current adapter layer . For each knowledge-specific adapter , we concatenate the last hidden features of the pre-trained model and adapter as the final output feature of this adapter model . In the pre-training procedure , we train each knowledge-specific adapter on different pre-training tasks individually . For various downstream tasks , K-ADAPTER can adopt the fine-tuning procedure similar to RoBERTa and BERT . When only one knowledge-specific adapter is adopted , we can take the final output feature of this adapter model as the input for task-specific layers of the downstream task . When multiple knowledge-specific adapters are adopted , we concatenate the output features of different adapter models as the input for task-specific layers of the downstream task . 3.2 PRE-TRAINING SETTINGS . We use RoBERTaLARGE ( L=24 , H=1024 , A=16 , 355M params ) implementation by Huggingface1 as the pre-trained model in all our experiments . As for each adapter layer , we denote the number of transformer layer as N , the hidden dimension of transformer layer as HA , the number of selfattention heads as AA , the hidden dimension of down-projection and up-projection layers as Hd and Hu . In detail , we have the following adapter size : N = 2 , HA = 768 , AA = 12 , Hu = 1024 and Hd = 768 . The RoBERTa layers where adapter layers plug in are { 0,11,23 } , and different adapter layers do not share parameters . Thus the total parameters for each adapter model are about 42M , which are much smaller than RoBERTaLARGE and make the training process memory efficient . It should be noticed that RoBERTa is fixed during training and the parameters of adapters are trainable and initialized randomly . Then we describe how to inject different knowledge into knowledgespecific adapters as below . 3.3 FACTUAL ADAPTER . Factual knowledge can be described as the basic information that is concerned with facts . In this work , we acquire factual knowledge from the relationships among entities in natural language . We extract a sub-dataset T-REx-rc from T-REx ( ElSahar et al. , 2018 ) which is a large scale alignment dataset between Wikipedia abstracts and Wikidata triples . We discard all relations having less than 50 entity pairs , collecting 430 relations and 5.5M sentences . In order to inject factual knowledge , we propose to pre-train a knowledge-specific adapter called facAdapter on the relation classification task . This task requires a model to classify relation labels of given entity pairs based on context . Specifically , the last hidden features of RoBERTa and facAdapter are concatenated as the input representation , and the pooling layer is applied to the input representations of the given entities . Then , we concatenate two entity representations to perform relation classification . 1https : //github.com/huggingface/transformers

Summary: This submission proposes a general method (K-Adapter) for injecting knowledge (either factual or linguistic) into pre-trained language models.  The key architectural property of the approach is that K-Adapters are isolated from one another, allowing the use of multiple adapters without interference. These K-Adapter modules take hidden layer _inputs_ from the main pre-trained model (eg, BERT), and are pre-trained on their knowledge outputs before a fine-tuning phase where they feed into a joint downstream task-specific model along with the pre-trained model outputs.