How Context Affects Language Models' Factual Predictions

1 . Introduction . Pre-trained language models such as BERT [ Devlin et al. , 2019 ] and RoBERTa [ Liu et al. , 2019 ] enabled state-of-the-art in many downstream NLP tasks [ Wang et al. , 2018a , 2019 , Wu et al. , 2019 ] . These models are trained in an unsupervised way from large textual collection and recent work [ Petroni et al. , 2019 , Jiang et al. , 2019 , Talmor et al. , 2019 , Devlin et al. , 2019 ] has demonstrated that such language models can store factual knowledge to some extent . However , considering the millions of documents and facts in Wikipedia1 and other textual resources , it unlikely that a language model with a fixed number of parameters is able to reliably store and retrieve factual knowledge with sufficient precision [ Guu et al. , 2020 ] . One way to get around this is to combine machine reading with an information retrieval ( IR ) system [ Chen et al. , 2017 , Guu et al. , 2020 ] . Given a question , the IR system retrieves 1. https : //en.wikipedia.org/wiki/Wikipedia : Statistics relevant contexts which are subsequently processed by a reading component . In the case of DrQA [ Chen et al. , 2017 ] , the retriever is fixed and the reading component is trained based on retrieved contexts , whereas in REALM [ Guu et al. , 2020 ] the IR component is trained alongside the reader during both pre-training and subsequent fine-tuning . In this paper , we go a step further and forego supervised fine-tuning . Instead , we consider the purely unsupervised case of augmenting a language model with retrieved contexts at test time . We demonstrate that augmenting pre-trained language models with such retrieved contexts dramatically improves unsupervised cloze-style question answering , reaching performance that is on par with the supervised DrQA approach . In addition to being unsupervised , using a pre-trained language model like BERT instead of a trained machine reading component has several other advantages . Since BERT is not an extractive QA model , it is able to utilize contexts that contain relevant information but do not contain the answer span directly . More importantly , we find that via the next-sentence prediction objective BERT is able to ignore noisy or irrelevant contexts . In summary , we present the following core findings : i ) augmenting queries with relevant contexts dramatically improves BERT and RoBERTa performance on the LAMA probe [ Petroni et al. , 2019 ] , demonstrating unsupervised machine reading capabilities of pre-trained language models ; ii ) fetching contexts using an off-the-shelf information retrieval system is sufficient for BERT to match the performance of a supervised open-domain QA baseline ; iii ) BERT ’ s next-sentence prediction pre-training strategy is a highly effective unsupervised mechanism in dealing with noisy and irrelevant contexts . The code and data to reproduce our analysis will be made publicly available . 2 . Related Work . Language Models and Probes With the recent success of pre-trained language models like BERT [ Devlin et al. , 2019 ] and its variants [ Liu et al. , 2019 , Seo et al. , 2019 , Raffel et al. , 2019 , Lewis et al. , 2019a ] , it becomes increasingly important to understand what these models learn . A variety of “ probes ” have been developed to analyse the syntactic structures , such as syntax trees [ Marvin and Linzen , 2018 , Hewitt and Manning , 2019 , Vig and Belinkov , 2019 ] , negative polarity items [ Warstadt and Bowman , 2019 , Warstadt et al. , 2019 ] , semantic fragments [ Richardson et al. , 2019 ] , function words [ Kim et al. , 2019 ] , and many other linguistic phenomena [ Tenney et al. , 2019a , b , De Cao et al. , 2020 ] . To measure the factual knowledge present in these pre-trained language models , Petroni et al . [ 2019 ] propose the LAMA benchmark which tests the models with cloze-style questions constructed from knowledge triples . Jiang et al . [ 2019 ] later extends LAMA by automatically discovering better prompts , Kassner and Schütze [ 2019 ] add negated statements , Poerner et al . [ 2019 ] filter out easy-to-guess queries , and Richardson and Sabharwal [ 2019 ] , Talmor et al . [ 2019 ] , Bisk et al . [ 2019 ] develop further probes for textual reasoning . Pre-trained language models have also been fine-tuned and adapted to be used as information retrieval systems [ Yang et al. , 2019b , Yilmaz et al. , 2019 , Seo et al. , 2019 ] . Open-Domain QA Open-domain QA aims at answering questions without explicitly knowing which documents contain relevant information . Open-domain QA models often involve a retriever to find relevant documents given a question , and a reader to produce the answers [ Chen et al. , 2017 ] . Works in this areas mostly focus on enhancing retrieval quality [ Choi et al. , 2017 , Wang et al. , 2018b , c , Lin et al. , 2018 , Min et al. , 2018 , Lee et al. , 2018 , 2019 , Das et al. , 2019 , Xiong et al. , 2019 ] , improving answer aggregation [ Clark and Gardner , 2018 , Wang et al. , 2018c , Lee et al. , 2018 , Pang et al. , 2019 ] , and accelerating the whole pipeline [ Seo et al. , 2019 ] . Recently , Guu et al . [ 2020 ] show that augmenting language model pre-training with a knowledge retriever induces performance gains on opendomain QA tasks . Our work differs from previous works in open-domain QA in two ways : i ) we consider a fully unsupervised setting using a pre-trained language model and an offthe-shelve information retrieval system , ii ) our aim is to assess the prediction of factual knowledge in this setup rather than to improve open-domain question answering in general . 3 . Methodology . Given a cloze-style question q with an answer a , we assess how the predictions from a language model change when we augment the input with contexts c. In this section , we describe the datasets we use to source ( q , a ) pairs , as well as various methods of generating context documents c. 3.1 Datasets We use the LAMA2 probe in our experiments [ Petroni et al. , 2019 ] , a collection of cloze-style questions about real world relational facts with a single token answer . Each question is accompanied by snippets of text from Wikipedia that are likely to express the corresponding fact . Although there are several cloze-style QA datasets ( some listed in Section 2 ) we decided to use LAMA because : ( 1 ) the nature of the LAMA data is aligned with the relational knowledge focus or our analysis ( i.e. , given a subject and a relation predict the object ) and ( 2 ) each data point is aligned by construction with relevent contextual information . We consider the Google-RE3 ( 3 relations , 5527 facts ) , T-REx [ Elsahar et al. , 2018 ] ( 41 relations , 34039 facts ) and SQuAD [ Rajpurkar et al. , 2016 ] ( 305 questions manually translated in cloze-style format ) subsets of the probe . More detailed statistics for the LAMA data considered are reported in Table 1 . For the RoBERTa results , we trim the LAMA dataset ( by about 15 % ) such that all answers are in the model ’ s vocabulary , so BERT and RoBERTa numbers in this paper should not be directly compared as they consider slightly different subsets of the data . 2. https : //github.com/facebookresearch/LAMA 3. https : //code.google.com/archive/p/relation-extraction-corpus 3.2 Baselines . We consider DrQA [ Chen et al. , 2017 ] , a popular system for open-domain question answering . The overall pipeline consists of two phases : ( 1 ) a TF-IDF document retrieval step , where the model finds relevant paragraphs from Wikipedia and ( 2 ) a machine comprehension step to extract the answer from those paragraphs . The machine comprehension component is trained with supervision on SQuAD v1.1 [ Rajpurkar et al. , 2016 ] . In order to apply DrQA to the LAMA probe , we take inspiration from [ Levy et al. , 2017 ] and map each cloze-style template to a natural question template ( e.g. , “ X was born in [ Mask ] ” to “ Where was X born ? ” ) . We constrain the predictions of DrQA to single-token answers as in Petroni et al . [ 2019 ] . Our results for DrQA and BERT are directly comparable with the other baselines in Petroni et al . [ 2019 ] . 3.3 Language Models . Among the constellation of language models that have been proposed in recent years we consider BERT [ Devlin et al. , 2019 ] since it is one of the post popular and widely used at the time of writing.4 Moreover , the large cased version of the BERT model is the best performing LM on the LAMA probe among those considered in Petroni et al . [ 2019 ] . We additionally consider the large version of the RoBERTa model [ Liu et al. , 2019 ] . Both BERT and RoBERTa have been trained on corpora that include Wikipedia . While BERT uses two pre-training strategies , Mask Language Modelling ( MLM ) and Next Sentence Prediction ( NSP ) , RoBERTa considers only the MLM task . We produce a probability distribution over the unified vocabulary of Petroni et al . [ 2019 ] for the masked token in each cloze-style questions and report the average precision at 1 ( P @ 1 ) . 3.4 Contexts . We enrich cloze statements with different types of contextual information . We explicitly distinguish cloze question q and context c in the input according to the model . For BERT , we use different segment embeddings , index 0 for q and 1 for c , and insert the separator token ( i.e. , [ SEP ] ) in between . For RoBERTa , which is not equipped with segment embeddings , we use the end of sentence ( eos ) token to separate q and c. We addidionally performed some experiments without this clear separation of query and context , but considering them as concatenated in a single segment ( or wihtout the eos token in between ) . The input is truncated to 512 tokens . 3.4.1 Oracle Contexts . We provide an oracle-based ( ora ) context in order to assess the capability of LMs to exploit context that we know is relevant to the entity in the question . Concretely , we use the Wikipedia snippet accompanying each example in the LAMA probe , truncated to at most five sentences . This context often contains the true answer and always contains related true information . 4. https : //huggingface.co/models 3.4.2 Sourcing Relevant Contexts . Relevant context is often not available and must be automatically sourced by the model [ Chen et al. , 2017 , Clark and Gardner , 2018 ] . In this scenario , we consider two possible approaches : using an information retrieval engine ( ret ) or generating the context with an autoregressive LM ( gen ) [ Radford et al. , 2019 ] . For the retrieval case , we use the first paragraph from DrQA ’ s retrieval system as context . For the generative case , taking inspiration from the study of Massarelli et al . [ 2019 ] , we consider a 1.4B parameters autoregressive language model trained on CC-NEWS [ Liu et al. , 2019 ] . This model has been shown to generate more factual text with respect to others trained on different corpora , including Wikipedia . For each question in LAMA , we use the natural question template as prefix to condition the generation , and generate five sentences using the delayed beam search strategy [ Massarelli et al. , 2019 ] . These results may be quite related to the entity in the query , though they may not always be completely factual .

Summary: This work analyses how factual predictions of a Masked Language Model (MLM) such as BERT and RoBERTa are influenced by adding extra context to a query. The paper examines a variety of ways of constructing this context, spanning over settings such as adversarially constructed, generated by a language model, retrieved by supervisedly trained systems, a TF-IDF retrieved baseline and an oracle. The paper finds that enriching a query with a good context can substantially improve performance in the LAMA probe, that analyses factual predictions. Additionally, the results demonstrate that there is considerable headroom for improvement in the retrieval side, evidenced by the results using an oracle retriever. Moreover, the paper shows the importance of BERT's Next Sentence Prediciton task, showing that it makes the model robust to adversarial appended contexts.