Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

1 INTRODUCTION . Language models pretrained on a large amount of text such as ELMo ( Peters et al. , 2018a ) ) , BERT ( Devlin et al. , 2019 ) and XLNet ( Yang et al. , 2019c ) have established new state of the art on a wide variety of NLP tasks . Researchers ascertain that pretraining allows models to learn syntactic and semantic information of language that is then transferred on other tasks ( Peters et al. , 2018b ; Clark et al. , 2019 ) . Interestingly , pretrained models also perform well on tasks that require grounding language and reasoning about the real world . For instance , the new state-of-the-art for WNLI ( Wang et al. , 2019a ) , ReCoRD ( Zhang et al. , 2018 ) and SWAG ( Zellers et al. , 2018 ) is achieved by pretrained models . These tasks are carefully designed so that the text input alone does not convey the complete information for accurate predictions – external knowledge is required to fill the gap . These results suggest that large-scale pretrained models implicitly capture real-world knowledge . Logan et al . ( 2019 ) and Petroni et al . ( 2019 ) further validate this hypothesis through a zero-shot fact completion task that involves single-token entities , showing that pretrained models achieve much better performance than random guessing and can be on par with specifically-trained relation extraction models . As unstructured text encodes a great deal of information about the world , large-scale pretraining over text data holds the promise of simultaneously learning syntax , semantics and connecting them with knowledge about the real world within a single model . However , existing pretraining objectives are usually defined at the token level and do not explicitly model entity-centric knowledge . In this work , we investigate whether we can further enforce pretrained models to focus on encyclopedic knowledge about real-world entities , so that they can better capture entity information from natural language and be applied to improving entity-related NLP tasks . We evaluate the extent to which a pretrained model represents such knowledge by extending an existing fact completion evaluation to a cloze ranking setting that allows us to deal with a large number of multi-token entity names without manual judgments . Our experiments on 10 common Wikidata ( Vrandečić & Krötzsch , 2014 ) relations reveal that existing pretrained models encode entity-level knowledge only to a limited degree . Thus , we propose a new weakly supervised knowledge learning objective that requires the model to distinguish between true and false knowledge expressed in natural language . Specifically , we replace entity mentions in the original documents with names of other entities of the same type and train the models to distinguish the correct entity mention from randomly chosen ones . Models trained with this objective demonstrates much stronger fact completion performance for most relations we test on . Compared with previous work ( Zhang et al. , 2019 ; Peters et al. , 2019 ) that utilizes an external knowledge base to incorporate entity knowledge , our method is able to directly derive real-world knowledge from unstructured text . Moreover , our method requires no additional data processing , memory or modifications to the BERT model when fine-tuning for downstream tasks . We test our model on two practical NLP problems that require entity knowledge : Question Answering ( QA ) and fine-grained Entity Typing . We use four previously published datasets for open-domain QA and observe that questions in these datasets often concern entities . The Entity Typing task requires the model to recognize fine-grained types of specified entity mentions given short contexts . On three of the QA datasets , our pretrained model outperforms all previous methods that do not rely on memory-consuming inter-passage normalizations1 . On the FIGER entity-typing dataset , our model sets a new state of the art . Through ablation analysis , we show that the new entity-centric training objective is instrumental for achieving state-of-the-art results . In summary , this paper makes the following contributions : 1 ) We extend existing fact completion evaluation settings to test pretrained models ’ ability on encoding knowledge of common real-world entities ; 2 ) We propose a new weakly supervised pretraining method which results in models that better capture knowledge about real-world entities from natural language text ; 3 ) The model trained with our knowledge learning objective establishes new state of the art on three entity-related QA datasets and a standard fine-grained entity typing dataset . We begin by introducing our weakly supervised method for knowledge learning ( §2 ) and then discuss experiment settings and evaluation protocols , compare our model to previously published work and perform ablation analysis . Finally , we review related work in §4 and conclude in §5 . 2 ENTITY REPLACEMENT TRAINING . We design an entity-centric training objective that utilizes weakly supervised training signals to explicitly encourage knowledge learning during pretraining . Given an input document , we first 1Wang et al . ( 2019b ) propose to use BERT to encode multiple retrieved paragraphs within the same forwardbackward pass . This operation requires around 12G GPU memory even for encoding a single example which includes 6 512-token paragraphs . recognize the entity mentions and link them to Wikipedia entities2 . We consider the original texts as positive knowledge statements and create negative statements by randomly replacing the entity mentions ( E+ ) with the names of other random entities ( E− ) that have the same entity type as the mentioned entity . This setup is similar in spirit to the type-constrained negative sampling technique used to train knowledge base representations ( Bordes et al. , 2013 ) . The latter technique creates negative triples by replacing the subject or object entity with random entities of the same type . Instead of knowledge base triples , we treat unstructured texts as factual statements . For a certain entity e mentioned in a context C , we train the model to make a binary prediction indicating whether the entity has been replaced : Je , C = 1e∈E+ logP ( e|C ) + ( 1− 1e∈E+ ) log ( 1− P ( e|C ) ) . Compared to the language modeling objective , entity replacement is defined at the entity level and introduces stronger negative signals . When we enforce entities to be of the same type , we preserve the linguistic correctness of the original sentence while the system needs to learn to perform judgment based on the factual aspect of the sentence . We describe the implementation in more detail in the following paragraphs . Data Preparation We use the whole English Wikipedia dump as training data and rely on all Wikipedia entities3 . Entities in documents are recognized based on Wikipedia anchor links and entity alias from Wikidata . That is , we first retrieve the entities annotated by anchor links and then find other mentions of these entities by string matching their Wikidata alias . We split each document into multiple text chunks with the same size ( 512 tokens ) . Although our experiments rely on the Wikipedia corpus , this setup can be easily extended to larger corpora with off-the-shelf entity linking tools . We leave the larger scope of the experiments to future work . Replacement Strategy When replacing entities , we first lookup type information4 from Wikidata and then randomly select other entities with the same type . We do not replace adjacent entities . In other words , there must be at least one unreplaced entity between any two replaced ones . This reduces cases where we replace all entities in the same sentence and the resulting sentences happen to introduce correct entities by chance . For replacement , we randomly sample a string from the entities ’ alias set . For each text chunk , we replicate it 10 times with different negative entities for each replacement location . We show an illustration of the entity replacement method in Figure 1 . Model Architecture We use the Transformer ( Vaswani et al. , 2017 ) model used by BERT ( Devlin et al. , 2019 ) . We use the same architecture as BERT base : 12 Transformer layers , each with hidden dimension 768 . We initialize the transformer with a model pretrained based on our own BERT reimplementations5 . For each entity , we use the final representations of its boundary words ( words before and after the entity mention ) to make predictions . We simply concatenate the boundary words ’ representations and add a linear layer for prediction . During training , we use 0.05 dropout at the final layer . Training Objectives Masked language model pretraining has been proven to be effective for downstream tasks . While training for entity replacement we also train with the masked language model objective in a multi-task set-up . When masking tokens , we restrict the masks to be outside the entity spans . We use a masking ratio of 5 % instead of 15 % in the original BERT to avoid masking out too much of the context . We train the model for approximately 1 million updates using a batch size of 128 . 2The entity links are only required for pretraining . 3Each Wikipedia entity ( title ) corresponds to a unique entity node in Wikidata . 4Based on the “ instance of ” relation in Wikidata . If there are multiple correct types , we randomly choose one . 5We use the masked language model implementation in Fairseq ( Ott et al. , 2019 ) to pre-train model for 2M updates on the combination of BooksCorpus ( Zhu et al. , 2015 ) and English Wikipedia . 3 EXPERIMENTS . We first test our model on a fact completion task . This task resembles traditional knowledge base completion : it requires the model to complete missing entities in factual triples . We further test on two real-world downstream tasks that require entity-level knowledge – question answering and fine-grained entity typing . We describe the hyperparameter and training settings of all experiments in the appendix . 3.1 ZERO-SHOT FACT COMPLETION . In traditional knowledge base completion tasks models have access to a set of training triples . Instead , we utilize a zero-shot test to examine the model ’ s ability to automatically derive relational knowledge from natural language . Dataset We rely on factual triples from Wikidata . Each triple describes the relationship between two certain entities , e.g. , { Paris , CapitalOf , France } . Following recent practices ( Bosselut et al. , 2019 ; Logan et al. , 2019 ) that decode structured knowledge from language models , we first manually create templates to convert triples of 10 common relations into natural language expressions ( { Paris , CapitalOf , France } → the capital of France is Paris ) . We then create queries by removing the object entity in the expression and use pre-trained models to predict the missing entities , e.g. , the capital of France is ? . We create 1000 cloze examples6 for each of the 10 relations . Evaluation Metrics Previous work ( Logan et al. , 2019 ; Petroni et al. , 2019 ) either relies on human evaluation or only considers single-token entities for fact completion . In contrast , we consider an entity-ranking setup and create a set of candidate entities for each relation . This setting allows us to automatically evaluate a large number of queries that usually involve multi-token entities . We test pretrained models on their ability to recover the correct object entity from the candidate set . To create the negative choices , we select from the set of all object entities in the particular relation , which generally have the same type as the groundtruth and are more challenging to distinguish than entities with different types . Our evaluation strategy is similar to previous work on knowledge base completion ( Nickel et al. , 2011 ; Bordes et al. , 2013 ; Xiong et al. , 2017 ) . We follow these studies and use Hits @ 10 as the evaluation metric . Baselines We compare our model with two pretrained language models BERT ( Devlin et al. , 2019 ) ( both base and large ) and GPT-2 ( Radford et al. , 2019 ) . We make use of their output token probabilities to rank candidate entities . For BERT , we feed in the masked queries ( e.g. , Qmasked = the capital of France is [ MASK ] ) . For multi-token candidates , we use the same number of [ MASK ] tokens in the query inputs . We use the average log probability of masked tokens for ranking . Given a multi-token entity Ei = [ e1i , e 2 i , ... , e |Ei| i ] , the ranking score from BERT is calculated as SEi = 1 |Ei| ∑ k logP ( eki |Qmasked ) . For GPT-2 , we feed in the original query without the answer entity and use the first-token probability of candidate entities for ranking , which performs better than using average log probabilities . As our model learns to predict a plausible probability ( P ( e|C ) ) for each entity mention during entity replacement training , we can directly use these predicted probabilities to rank the candidates . Results Table 1 shows the fact completion results for all relations . We denote our method WKLM for ( Weakly Supervised Knowledge-Pretrained Language Model ) . Overall , WKLM achieves the best results on 8 of the 10 relations . We also observe that GPT-2 outperforms BERT on average . We think this is because the fact completion task requires models to predict the missing entities using only a short context on the left , while BERT pretraining incorporates context from both directions . Interestingly , BERT achieves good performance on several geographical relations such as PlaceOfBirth , LocatedIn and PlaceOfDeath . We conjecture that this is because location entities usually appear at sentence ends in Wikipedia articles , e.g. , Obama was born in 6For each relation , we use the top triples that connecting most common entities . Honolulu , Hawaii .. This sentence pattern is similar to our templates and BERT may learn to rely mostly on the left context to make predictions . For most relations that include answers that are person names , BERT lags behind both GPT-2 and our model . Comparing the top and bottom five relations , we observe that BERT ’ s performance is correlated with the size of the candidate set , while WKLM and GPT-2 are less sensitive to this number . A similar pattern exists between models ’ performance and the cardinality of groundtruth answers , i.e. , our model achieves similar performance on both single-answer and multiple-answer queries while BERT is usually better at single-answer queries . WKLM both outperforms BERT and GPT-2 and achieves robust performance across relations with different properties . Visualization of correlations between relation properties and model performance can be found in the appendix .

Summary: This paper proposed to improve pre-training of language models (e.g. BERT) by incorporating information around entities based on English Wikipedia. The idea is very simple and straightforward: it takes all the anchor links from Wikipedia and replaces some entities by randomly sampling negative ones of the same entity type (according to Wikidata) and adds an extra binary prediction task which predicts if the entity has been replaced or not.