Towards Continual Knowledge Learning of Language Models

1 INTRODUCTION . Recent works have shown that large Language Models ( LM ) , such as T5 ( Raffel et al. , 2019 ) and GPT-3 ( Brown et al. , 2020 ) , have the capability of storing a tremendous amount of world knowledge in their parameters when pretrained on a vast corpus of text ( Petroni et al. , 2019 ) . These pretrained LMs have shown potential to serve as knowledge bases when probed for world knowledge without any finetuning through the LAnguage Model Analysis ( LAMA ) task ( Petroni et al. , 2019 ) , which requires probing LMs for world knowledge in a zero-shot manner through slot-filling , and promising results utilizing the encoded world knowledge when finetuned on various Knowledge Intensive Language Tasks ( KILT ) ( Petroni et al. , 2021 ) , e.g. , question answering , knowledgeable open dialogues . While the world knowledge stored in LMs has diverse use cases , it can quickly become outdated as the world changes fast , and LMs need to frequently renew their internal world knowledge accordingly . For example , it is impossible to probe for new information such as “ won the US Election 2020 ” from the original T5 ( Raffel et al. , 2019 ) which was pretrained on C4 web corpus from April 2019.2 Also , information that may have once been considered accurate may no longer be valid because the information has been updated . For instance , the answer to “ Which soccer team does Cristiano Ronaldo play for ? ” has changed from Juventus to Manchester United in September 2021 . Meanwhile , time-invariant information learned from the original corpus such as “ Barack Obama was born in Honolulu , Hawaii ” should not be altered within the LMs . Despite its importance , the challenge of renewing the internal world knowledge stored in the parameters of LMs is nontrivial and has only been explored in rather specific settings . For example , recent works have proposed to modify specific target knowledge such as individual facts ( De Cao et al. , 2021 ; Zhu et al. , 2020 ; Dai et al. , 2021 ) . Dhingra et al . ( 2021 ) have addressed LMs as temporal knowledge bases by jointly modeling text with its timestamp . But the problem of renewing the 1We will make all of our benchmark datasets , evaluation script , and baseline code publicly available . 2T5 was initially pretrained on the C4 dataset ( about 750 GB ) , which is a cleansed dump of Common Crawl extracted from the web in April 2019. world knowledge of LMs in a more general and scalable way , such as through continual pretraining on a corpus with new knowledge , has not been formally formulated or explored by previous works . Moreover , the community lacks a benchmark that can be used to systematically study how the internal knowledge of LMs changes through the training on new information . Lastly , methodologies to effectively renew the knowledge of LMs at scale have yet to be thoroughly explored . In this work , we propose a novel continual learning ( CL ) formulation named CONTINUAL KNOWLEDGE LEARNING ( CKL ) , where we attempt to renew the internal world knowledge of LMs through continual pretraining on new corpora . We systematically categorize world knowledge into three main categories and make benchmark datasets to measure each of them during CKL : ( 1 ) INVARIANTLAMA for time-invariant world knowledge in LMs that should not be forgotten or altered , ( 2 ) UPDATEDLAMA for outdated world knowledge that needs to be updated in the LMs , and ( 3 ) NEWLAMA for new world knowledge that should be injected into the LMs . We also propose a novel metric named FUAR ( FORGOTTEN / ( UPDATED + ACQUIRED ) RATIO ) that can measure the trade-off between forgetting , updating , and acquiring knowledge . Finally , while one might think of implementing contemporary CL methods for this benchmark , we show that CKL has nontrivial differences to traditional CL formulations and require approaches specific to CKL . We find and compare model architectures and training methodologies ( Chen et al. , 2020 ; He et al. , 2021 ; Hu et al. , 2021 ; Wang et al. , 2021b ) from the literature that have shown potential to mitigate forgetting of knowledge gained during pretraining , establishing them as baselines for the CKL benchmark . In sum , while the challenge of renewing the internal world knowledge of LMs is essential in realworld scenarios , it has yet to be formulated or extensively explored . Therefore , in this paper : • We propose a novel CL formulation called CONTINUAL KNOWLEDGE LEARNING ( CKL ) and construct a new benchmark to measure the amount of forgetting and amount of world knowledge gained by continued pretraining on a novel language modeling corpus that we construct , containing new knowledge . • We explore LM architectures and training methodologies that are natural baselines for CKL in literature , denoting them as CKL methods , and performing extensive experiments on our CKL benchmark . We categorize them into regularization , rehearsal , and parameterexpansion methods , same as in traditional CL literature , and compare the effectiveness of each type of method using a novel metric named FUAR that we propose to measure the trade-off between forgotten knowledge and updated or acquired knowledge . • Towards creating an ever-changing LM , we perform extensive analysis in the CKL benchmark and highlight important challenges and findings : parameter-expansion methods have the limitation of memory inefficiency despite performing the best in most of our experiments and seeing the same data repeatedly during continued pretraining is a critical cause of forgetting . Also , we show interesting results that need further exploration : learning rate can be varied to balance the forgetting and learning of new knowledge , CKL may help in performing previous-knowledge-intensive tasks after gaining new world knowledge , and CKL methods are transferable across LM architectures despite showing a different trend in performance . An overview of the proposed CKL benchmark is shown in Figure 1 . 2 RELATED WORK . Language Models ( LMs ) utilizing knowledge from external sources , such as Retrieval-Augmented Generation ( RAG ) ( Lewis et al. , 2020a ) and Blender Bot 2.0 ( Xu et al. , 2021 ; Komeili et al. , 2021 ) , cope with the changing world by updating the external sources during inference or searching the internet for retrieving recent information . However , recent works have shown that these memoryaugmented models suffer from hallucination , which means that they present false information as if it were correct , despite being given updated knowledge during inference ( Zhang & Choi , 2021 ) , which worsens as the size of the LM increases ( Longpre et al. , 2021 ) , making it more so important for implicit parameters to be renewed as well . In order to renew the internal knowledge of LMs , one might consider pretraining LMs from scratch with a newly updated text corpus of a scale similar to the one used during initial pretraining , such as a recent dump of the entire Wikipedia . However , this approach is computationally demanding and also environmentally harmful ( Patterson et al. , 2021 ) . Another alternative approach is continuing the pretraining process on a much smaller corpus containing new world knowledge , but such a methodology is known to suffer from catastrophic forgetting ( McCloskey & Cohen , 1989 ; Kirkpatrick et al. , 2017 ) , where the models forget previously learned knowledge as they acquire new knowledge . Lazaridou et al . ( 2021 ) ; Jin et al . ( 2021 ) suggests implementing prior Continual Learning ( CL ) methods ( Sun et al. , 2020 ; d ’ Autume et al. , 2019 ) to address this problem . However , it is important to note that there are nontrivial differences between traditional CL and the proposed Continual Knowledge Learning ( CKL ) formulation which make applying traditional CL methods inadequate . In traditional CL , methods can be largely categorized into regularization , rehearsal , and parameterexpansion methods . ( 1 ) While regularization methods ( Kirkpatrick et al. , 2017 ) require identifying important parameters used for previous tasks , exactly how and where the knowledge is stored in the parameters of an LM is currently extremely difficult to identify and localize ( Vig et al. , 2020 ; De Cao et al. , 2021 ) . ( 2 ) While prior rehearsal methods ( Lopez-Paz & Ranzato , 2017 ) consider learning all of the streams of tasks at once ( multi-task learning ) as the performance upper-bound and replicate such a setting with samples stored in the episodic memory , a few samples from the pretraining corpus can not represent the overall world knowledge from the corpus . Moreover , if LMs are pretrained on a shuffled concatenation of stream of corpora , there is no guarantee that the LMs will acquire the correct , recent information from the recent corpora , especially in cases where the former corpora are much bigger than the latter ones , which is shown by experiments in Section 5.1 . ( 3 ) Lastly , prior parameter-expansion methods ( Rusu et al. , 2016 ; Yoon et al. , 2018 ) focus on learning a stream of different tasks via strong supervision , while in CKL , the focus is constantly updating world knowledge from a stream of corpora via self-supervision . Because of these fundamental differences , instead of contemporary CL methods mentioned above , we explore methodologies from the literature that are suitable for CKL ( Chen et al. , 2020 ; He et al. , 2021 ; Hu et al. , 2021 ; Wang et al. , 2021b ) , modifying and adapting each method according to our needs as CKL methods . Lastly , while it has been pointed out that some of the traditional CL formulations may have little practical importance in real-world scenarios by Prabhu et al . ( 2020 ) , CKL is much closer to the initial motivation behind CL , which is that the “ fundamental characteristic of natural intelligence is its ability to continually learn new knowledge while updating information about the old ones ” ( Prabhu et al. , 2020 ) . Details of related works regarding the traditional CL methods and how CKL methods address the fundamental differences are provided in Appendix A . 3 CONTINUAL KNOWLEDGE LEARNING ( CKL ) . In this section , we explain the formulation of the task , the data construction process , and the proposed metric measuring the trade-off between forgetting previous world knowledge and updating and learning of new world knowledge . 3.1 TASK FORMULATION . When viewing the task of renewing the internal knowledge of LMs as one of CL formulations , pretraining on the original corpus can be considered as a previous task , and continued pretraining on new corpus can be considered as the current task , the main objective becoming retaining the time-invariant world knowledge gained through initial pretraining while efficiently learning new and updated world knowledge through continued pretraining . Throughout the paper , we let D0 refer to the corpus used for initial pretraining and let D1 denote the new corpus used for continued pretraining . New Text Corpus for Language Modeling For LMs to renew their internal knowledge , they need to be continually pretrained on a new text corpus D1 which has the updated and new information . D1 should ideally be much smaller than D0 , as a large D1 amounting to the size of D0 will result in massive computational costs similar to pretraining the LMs from scratch . For constructing D1 , we crawl recently published news articles from the web making CC-RECENTNEWS.3 Probing LMs for World Knowledge The most widely used task for probing LMs for world knowledge is the LAnguage Model Analysis ( LAMA ) ( Petroni et al. , 2019 ) task , which consists of cloze sentences created from a set of knowledge sources using manually defined templates . We define that an LM knows a fact if it can successfully predict in a zero-shot manner the masked entity in the cloze sentence , such as “ Dante was born in ” as Florence . While there may be other alternatives for measuring the world knowledge encoded in LMs4 , we construct our main datasets as LAMA tasks , while also additionally providing the corresponding question pairs to the cloze sentences for those who want to test on CBQA as well . Measuring Retention of Time-invariant World Knowledge We define time-invariant world knowledge as the information present in D0 that has no possibility of conflicting with information from D1 . For example , if the information of the birthplace of Barack Obama is present in D0 , it is unlikely that D1 contains information that contradicts that fact . Also , we classify instances where the time-stamps are fixed such as “ Cristiano Ronaldo played for in 2010. ” as time-invariant . These time-invariant instances should not be changed as LMs are continually pretrained on D1 . In order to measure how much time-invariant information is lost due to catastrophic forgetting during continued pretraining , we create INVARIANTLAMA , a subset of LAMA ( Petroni et al. , 2019 ) , consisting of only time-invariant cloze sentences detailed in Appendix B.1 . Measuring Update of Outdated World Knowledge In this work , we define outdated world knowledge as information that is conflicting between D0 and D1 . For example , the President of the US may be Barack Obama in D0 and Joe Biden in D1 . In this case , the LM should update its internal knowledge as Joe Biden as the US president . If an LM is pretrained on both D0 and D1 simultaneously , there is no guarantee that the LM will acquire the correct , recent information from D1 , especially in cases where D0 is much bigger than D1 , which is one of the biggest difference between the CKL and traditional CL setting . For measuring update of outdated information , we construct UPDATEDLAMA which is made up of cloze statements for which answers can be found in both D0 and D1 , but are conflicting . Measuring Acquisition of New World Knowledge We define new world knowledge as the information present in D1 , but not in D0 . To measure new knowledge acquired through continued pretraining on D1 , we construct NEWLAMA which is made up of detailed cloze statements requiring new knowledge from D1 to correctly answer . We provide two datasets for measuring new world knowledge : NEWLAMA , for which each of the instances is verified that the answer does not exist in D0 , but only in D1 , and NEWLAMA-EASY for which each of the instances does not perfectly comply with our strict definition of new world knowledge due to its creation process , but is used to generally measure the new knowledge acquired from continued pretraining on D1 at a larger scale . NEWLAMA-EASY can be considered easier since each instance was constructed to be similar to the data distribution seen during continued pretraining . Dataset Construction The data for continual pretraining , CC-RECENTNEWS , is constructed using news-please ( Hamborg et al. , 2017 ) . INVARIANTLAMA is constructed by manually selecting 28 3CC-RECENTNEWS consists of 221,779 articles ( ∼168M tokens ) , which is estimated to be about 750 times smaller than C4 , a cleansed version of the April 2019 Common Crawl dataset ( https : //commoncrawl.org/ ) that was used to initially pretrain the T5 LM ( Raffel et al. , 2019 ) . 4Closed-book question answering ( CBQA ) ( Roberts et al. , 2020 ) can also be considered as a task that measures the world knowledge of LMs through finetuning , but it has been pointed out that much of its performance increases are due to the test-train overlap ( Lewis et al. , 2020b ; Wang et al. , 2021a ) in the datasets . time-invariant relations from T-Rex ( Elsahar et al. , 2018 ) . For UPDATEDLAMA and NEWLAMA , we use Amazon Mechanical Turk ( mturk ) 5 for crowd-sourcing Human Intelligent Tasks ( HITs ) . The process requires selecting answerable questions from a list of questions generated by the model introduced in Lewis et al . ( 2021 ) and converting them into cloze sentences . We have also separately hired 11 experts to verify the correctness and search the C4 database to categorize each instance following our definition of updated and new . NEWLAMA-EASY is constructed at a larger scale through a two-phase mturk process where sentences selected from articles containing new information are decontextualized and paraphrased6 before being masked , verified and converted to corresponding questions . The constructed dataset statistics are in Table 1 . Important details about the data construction pipeline , examples , and more fine-grained statistics are provided in Appendix B .

Summary: The paper studies continual knowledge learning of language models, which is an interesting and important problem. Particularly, a new benchmark and a metric are introduced to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. To establish baselines for the CKL benchmark and validate the rationality of the proposed benchmark and metric, the author conducts extensive experiments with a pre-trained encoder-decoder model (T5) based on various training methodologies including regularization, rehearsal, and parameter expansion methods.  The paper is well organized and easy to follow. The proposed continual knowledge learning problem is quite interesting and important. The FUAR metric is also technically sound. The authors also conduct comprehensive experiments to verify the rationality of the proposed benchmark under the various settings.