K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters

1 INTRODUCTION . Language representation models , which are pre-trained on large-scale text corpus through unsupervised objectives like ( masked ) language modeling , such as BERT ( Devlin et al. , 2019 ) , GPT ( Radford et al. , 2018 ; 2019 ) , XLNet ( Yang et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) and T5 ( Raffel et al. , 2019 ) , have established state-of-the-art performances on various NLP downstream tasks . Despite the huge success of these pre-trained models in empirical studies , recent studies suggest that models learned in such an unsupervised manner struggle to capture rich knowledge . For example , Poerner et al . ( 2019 ) suggest that although language models do well in reasoning about the surface form of entity names , they fail in capturing rich factual knowledge . Kassner & Schütze ( 2019 ) observe that BERT mostly did not learn the meaning of negation ( e.g . “ not ” ) . These observations motivate us to study the injection of knowledge into pre-trained models like BERT and RoBERTa . Recently , some efforts have been made to exploit injecting knowledge into pre-trained language models ( Zhang et al. , 2019 ; Lauscher et al. , 2019 ; Levine et al. , 2019 ; Peters et al. , 2019 ; He et al. , 2019 ; Xiong et al. , 2020 ) . Most previous works ( as shown in Table 1 ) augment the standard language modeling objective with knowledge-driven objectives and update model parameters in a multi-task learning manner . Although these methods , with updated pre-trained models , obtain better performance on downstream tasks , they fail at continual learning ( Kirkpatrick et al. , 2017 ) . Model parameters need to be retrained when new kinds of knowledge are injected , which may result in the catastrophic forgetting of previously injected knowledge . Meanwhile , the resulting pre-trained models produce entangled representations , which makes it hard to investigate the effect of each knowledge when multiple kinds of knowledge are injected . In this paper , we propose K-ADAPTER , a flexible and simple approach that infuses knowledge into large pre-trained models . K-ADAPTER has attractive properties including supporting continual knowledge infusion and producing disentangled representations . It leaves the original representation of a pre-trained model unchanged and exports different representations for different types of infused knowledge . This is achieved by the integration of compact neural models , dubbed adapters here . Adapters are knowledge-specific models plugged outside of a pre-trained model , whose inputs are the output hidden-states of intermediate layers of the pre-trained model . We take RoBERTa ( Liu et al. , 2019 ) as the base pre-trained model and integrate two types of knowledge , including factual knowledge obtained by aligned Wikipedia text to Wikidata triplets , linguistic knowledge obtained by applying off-the-shell dependency parser to web texts . In the pre-training phase , we train two adapters independently on relation classification task and dependency relation prediction task respectively , while keeping the original parameters of RoBERTa frozen . Since adapters have much less trainable parameters compared with RoBERTa , the training process is memory efficient . We conduct extensive experiments on six benchmark datasets across three knowledge-driven tasks , i.e. , relation classification , entity typing and question answering . Experiments show that KADAPTER consistently performs better than RoBERTa , and achieves state-of-the-art performance on five datasets , and comparable performance compared with CosmosQA SOTA . Probing experiments on LAMA ( Poerner et al. , 2019 ) and LAMA-UHN ( Petroni et al. , 2019 ) , further demonstrates that K-ADAPTER captures richer factual and commonsense knowledge than RoBERTa . The contributions of this paper are summarized as follows : • We propose K-ADAPTER , a flexible approach that supports continual knowledge infusion into large pre-trained models ( e.g . RoBERTa in this work ) . • We infuse factual knowledge and linguistic knowledge , and show that adapters for both kinds of knowledge work well on downstream tasks . • K-ADAPTER achieves superior performance by fine-tuning parameters on three downstream tasks , and captures richer factual and commonsense knowledge than RoBERTa on probing experiments . 2 RELATED WORK . Our work relates to the area of injecting knowledge into pre-trained models . As stated in Table 1 , previous works mainly differ from the knowledge sources and the objective used for training . ERNIE ( Zhang et al. , 2019 ) injects a knowledge graph into BERT . They align entities from Wikipedia sentences to fact triples in WikiData , and discard sentences with less than three entities . In the training process , the input includes sentences and linked facts , and the knowledge-aware learning objective is to predict the correct token-entity alignment . Entity embeddings are trained on fact triples from WikiData via TransE ( Bordes et al. , 2013 ) . LIBERT ( Lauscher et al. , 2019 ) injects pairs of words with synonym and hyponym-hypernym relations in WordNet . The model takes a pair of words separated by a special token as the input , and is optimized by a binary classification problem , which predicts whether the input holds a particular relation or not . SenseBERT ( Levine et al. , 2019 ) considers word-supersense knowledge . It inject knowledge by predicting the supersense of the masked word in the input , where the candidates are nouns and verbs and the ground truth comes from WordNet . KnowBERT ( Peters et al. , 2019 ) incorporates knowledge bases into BERT using Knowledge attention and recontextualization , where the knowledge comes from synset-synset and lemma-lemma relationships in WordNet , and entity linking information in Wikipedia . If entity linking supervision is available , the model is learned with an additional knowledge-aware log-likelihood or max-margin objective . WKLM ( Xiong et al. , 2020 ) replaces entity mentions in the original document with names of other entities of the same type . The model is trained to distinguish the correct entity mention from randomly chosen ones . BERT-MK ( He et al. , 2019 ) integrates fact triples from knowledge graph . For each entity , it sample incoming and outcoming instances from the neighbors on the knowledge graph , and replaces head or tail entity to create negative instances . The model is learned to discriminate between real and fake facts . As shown in Table 1 , our model ( K-ADAPTER ) differs from previous studies in three aspects . First , we consider both fact-related objective ( i.e . predicate/relation prediction ) and linguistic-related objective ( i.e . dependency relation prediction ) . Second , the original parameter of BERT is clamped in the knowledge infusion process . Third , our approach supports continual learning , which means that the learning of different adapters are not entangled . This flexibility enables us to efficiently inject different types of knowledge independently , and inject more types of knowledge without any loss on the previously injected knowledge . 3 K-ADAPTER . As illustrated in Figure 1 ( a ) , most of the previous works enhance pre-trained language models by injecting knowledge and update model parameters through multi-task learning . Regardless of these different versions of knowledge-injected methods with multi-task learning , common issues not fully studied are catastrophic forgetting of previous knowledge . To address this , we present KADAPTER as shown in Figure 1 ( b ) , where multiple kinds of knowledge are injected into different compact neural models ( i.e. , adapters in this paper ) individually instead of directly injecting knowledge into pre-trained models . It keeps the original representation of a pre-trained model fixed and supports continual knowledge infusion , i.e. , injecting each kind of knowledge into the corresponding knowledge-specific adapter and producing disentangled representation . Specifically , adapters are knowledge-specific models ( with few parameters ) plugged outside of a pre-trained model . The inputs of adapters are the output hidden-states of intermediate layers of the pre-trained model . Each adapter is pre-trained independently on different tasks for injecting discriminative knowledge while the original parameters of the pre-trained model are frozen . In this paper , we exploit RoBERTa ( Liu et al. , 2019 ) as the pre-trained model , and mainly infuse factual knowledge and linguistic knowledge with two kinds of adapters , i.e. , factual adapter and linguistic adapter which are pre-trained on the relation classification task and dependency relation prediction task respectively . In this section , we first describe the structure of our adapter , and then present the process of pre-training knowledgespecific adapters . 3.1 ADAPTER STRUCTURE . In this work , we present a different adapter structure as shown in Figure 2 , which is referred to as the knowledge-specific adapter . In contrast to Houlsby et al . ( 2019 ) add adapter layers into each transformer layer , our adapter works as outside plug-ins . Each adapter model consists of K adapter layers that contain N transformer ( Vaswani et al. , 2017 ) layers and two projection layers . A skipconnection is applied across two projection layers . Specifically , for each adapter model , we plug adapter layers among different transformer layers of the pre-trained model . We concatenate the output hidden feature of the transformer layer in the pre-trained model and the output feature of the former adapter layer , as the input feature of the current adapter layer . For each knowledge-specific adapter , we concatenate the last hidden features of the pre-trained model and adapter as the final output feature of this adapter model . In the pre-training procedure , we train each knowledge-specific adapter on different pre-training tasks individually . For various downstream tasks , K-ADAPTER can adopt the fine-tuning procedure similar to RoBERTa and BERT . When only one knowledge-specific adapter is adopted , we can take the final output feature of this adapter model as the input for task-specific layers of the downstream task . When multiple knowledge-specific adapters are adopted , we concatenate the output features of different adapter models as the input for task-specific layers of the downstream task . 3.2 PRE-TRAINING SETTINGS . We use RoBERTaLARGE ( L=24 , H=1024 , A=16 , 355M params ) implementation by Huggingface1 as the pre-trained model in all our experiments . As for each adapter layer , we denote the number of transformer layer as N , the hidden dimension of transformer layer as HA , the number of selfattention heads as AA , the hidden dimension of down-projection and up-projection layers as Hd and Hu . In detail , we have the following adapter size : N = 2 , HA = 768 , AA = 12 , Hu = 1024 and Hd = 768 . The RoBERTa layers where adapter layers plug in are { 0,11,23 } , and different adapter layers do not share parameters . Thus the total parameters for each adapter model are about 42M , which are much smaller than RoBERTaLARGE and make the training process memory efficient . It should be noticed that RoBERTa is fixed during training and the parameters of adapters are trainable and initialized randomly . Then we describe how to inject different knowledge into knowledgespecific adapters as below . 3.3 FACTUAL ADAPTER . Factual knowledge can be described as the basic information that is concerned with facts . In this work , we acquire factual knowledge from the relationships among entities in natural language . We extract a sub-dataset T-REx-rc from T-REx ( ElSahar et al. , 2018 ) which is a large scale alignment dataset between Wikipedia abstracts and Wikidata triples . We discard all relations having less than 50 entity pairs , collecting 430 relations and 5.5M sentences . In order to inject factual knowledge , we propose to pre-train a knowledge-specific adapter called facAdapter on the relation classification task . This task requires a model to classify relation labels of given entity pairs based on context . Specifically , the last hidden features of RoBERTa and facAdapter are concatenated as the input representation , and the pooling layer is applied to the input representations of the given entities . Then , we concatenate two entity representations to perform relation classification . 1https : //github.com/huggingface/transformers

Summary: The authors propose a plug-in based adapter approach to allow for task specific parameter settings without updating the original pre-trained model which prevents the potential for catastrophic forgetting while also removing the need for separate models for separate tasks.  The work seems to build off Houlsby 19  as briefly cited, but its in plug-in nature seems easier to adopt for multiple tasks.  There is however not direct comparison with it or Cooper et al 19 ( https://arxiv.org/pdf/1902.02671.pdf ) which makes it difficult to assess.  The way in which the adaptors were pretrained was a little unclear to me.  The experiments are extensive and well done.