Towards Continual Knowledge Learning of Language Models

1 INTRODUCTION . Recent works have shown that large Language Models ( LM ) , such as T5 ( Raffel et al. , 2019 ) and GPT-3 ( Brown et al. , 2020 ) , have the capability of storing a tremendous amount of world knowledge in their parameters when pretrained on a vast corpus of text ( Petroni et al. , 2019 ) . These pretrained LMs have shown potential to serve as knowledge bases when probed for world knowledge without any finetuning through the LAnguage Model Analysis ( LAMA ) task ( Petroni et al. , 2019 ) , which requires probing LMs for world knowledge in a zero-shot manner through slot-filling , and promising results utilizing the encoded world knowledge when finetuned on various Knowledge Intensive Language Tasks ( KILT ) ( Petroni et al. , 2021 ) , e.g. , question answering , knowledgeable open dialogues . While the world knowledge stored in LMs has diverse use cases , it can quickly become outdated as the world changes fast , and LMs need to frequently renew their internal world knowledge accordingly . For example , it is impossible to probe for new information such as “ won the US Election 2020 ” from the original T5 ( Raffel et al. , 2019 ) which was pretrained on C4 web corpus from April 2019.2 Also , information that may have once been considered accurate may no longer be valid because the information has been updated . For instance , the answer to “ Which soccer team does Cristiano Ronaldo play for ? ” has changed from Juventus to Manchester United in September 2021 . Meanwhile , time-invariant information learned from the original corpus such as “ Barack Obama was born in Honolulu , Hawaii ” should not be altered within the LMs . Despite its importance , the challenge of renewing the internal world knowledge stored in the parameters of LMs is nontrivial and has only been explored in rather specific settings . For example , recent works have proposed to modify specific target knowledge such as individual facts ( De Cao et al. , 2021 ; Zhu et al. , 2020 ; Dai et al. , 2021 ) . Dhingra et al . ( 2021 ) have addressed LMs as temporal knowledge bases by jointly modeling text with its timestamp . But the problem of renewing the 1We will make all of our benchmark datasets , evaluation script , and baseline code publicly available . 2T5 was initially pretrained on the C4 dataset ( about 750 GB ) , which is a cleansed dump of Common Crawl extracted from the web in April 2019. world knowledge of LMs in a more general and scalable way , such as through continual pretraining on a corpus with new knowledge , has not been formally formulated or explored by previous works . Moreover , the community lacks a benchmark that can be used to systematically study how the internal knowledge of LMs changes through the training on new information . Lastly , methodologies to effectively renew the knowledge of LMs at scale have yet to be thoroughly explored . In this work , we propose a novel continual learning ( CL ) formulation named CONTINUAL KNOWLEDGE LEARNING ( CKL ) , where we attempt to renew the internal world knowledge of LMs through continual pretraining on new corpora . We systematically categorize world knowledge into three main categories and make benchmark datasets to measure each of them during CKL : ( 1 ) INVARIANTLAMA for time-invariant world knowledge in LMs that should not be forgotten or altered , ( 2 ) UPDATEDLAMA for outdated world knowledge that needs to be updated in the LMs , and ( 3 ) NEWLAMA for new world knowledge that should be injected into the LMs . We also propose a novel metric named FUAR ( FORGOTTEN / ( UPDATED + ACQUIRED ) RATIO ) that can measure the trade-off between forgetting , updating , and acquiring knowledge . Finally , while one might think of implementing contemporary CL methods for this benchmark , we show that CKL has nontrivial differences to traditional CL formulations and require approaches specific to CKL . We find and compare model architectures and training methodologies ( Chen et al. , 2020 ; He et al. , 2021 ; Hu et al. , 2021 ; Wang et al. , 2021b ) from the literature that have shown potential to mitigate forgetting of knowledge gained during pretraining , establishing them as baselines for the CKL benchmark . In sum , while the challenge of renewing the internal world knowledge of LMs is essential in realworld scenarios , it has yet to be formulated or extensively explored . Therefore , in this paper : • We propose a novel CL formulation called CONTINUAL KNOWLEDGE LEARNING ( CKL ) and construct a new benchmark to measure the amount of forgetting and amount of world knowledge gained by continued pretraining on a novel language modeling corpus that we construct , containing new knowledge . • We explore LM architectures and training methodologies that are natural baselines for CKL in literature , denoting them as CKL methods , and performing extensive experiments on our CKL benchmark . We categorize them into regularization , rehearsal , and parameterexpansion methods , same as in traditional CL literature , and compare the effectiveness of each type of method using a novel metric named FUAR that we propose to measure the trade-off between forgotten knowledge and updated or acquired knowledge . • Towards creating an ever-changing LM , we perform extensive analysis in the CKL benchmark and highlight important challenges and findings : parameter-expansion methods have the limitation of memory inefficiency despite performing the best in most of our experiments and seeing the same data repeatedly during continued pretraining is a critical cause of forgetting . Also , we show interesting results that need further exploration : learning rate can be varied to balance the forgetting and learning of new knowledge , CKL may help in performing previous-knowledge-intensive tasks after gaining new world knowledge , and CKL methods are transferable across LM architectures despite showing a different trend in performance . An overview of the proposed CKL benchmark is shown in Figure 1 . 2 RELATED WORK . Language Models ( LMs ) utilizing knowledge from external sources , such as Retrieval-Augmented Generation ( RAG ) ( Lewis et al. , 2020a ) and Blender Bot 2.0 ( Xu et al. , 2021 ; Komeili et al. , 2021 ) , cope with the changing world by updating the external sources during inference or searching the internet for retrieving recent information . However , recent works have shown that these memoryaugmented models suffer from hallucination , which means that they present false information as if it were correct , despite being given updated knowledge during inference ( Zhang & Choi , 2021 ) , which worsens as the size of the LM increases ( Longpre et al. , 2021 ) , making it more so important for implicit parameters to be renewed as well . In order to renew the internal knowledge of LMs , one might consider pretraining LMs from scratch with a newly updated text corpus of a scale similar to the one used during initial pretraining , such as a recent dump of the entire Wikipedia . However , this approach is computationally demanding and also environmentally harmful ( Patterson et al. , 2021 ) . Another alternative approach is continuing the pretraining process on a much smaller corpus containing new world knowledge , but such a methodology is known to suffer from catastrophic forgetting ( McCloskey & Cohen , 1989 ; Kirkpatrick et al. , 2017 ) , where the models forget previously learned knowledge as they acquire new knowledge . Lazaridou et al . ( 2021 ) ; Jin et al . ( 2021 ) suggests implementing prior Continual Learning ( CL ) methods ( Sun et al. , 2020 ; d ’ Autume et al. , 2019 ) to address this problem . However , it is important to note that there are nontrivial differences between traditional CL and the proposed Continual Knowledge Learning ( CKL ) formulation which make applying traditional CL methods inadequate . In traditional CL , methods can be largely categorized into regularization , rehearsal , and parameterexpansion methods . ( 1 ) While regularization methods ( Kirkpatrick et al. , 2017 ) require identifying important parameters used for previous tasks , exactly how and where the knowledge is stored in the parameters of an LM is currently extremely difficult to identify and localize ( Vig et al. , 2020 ; De Cao et al. , 2021 ) . ( 2 ) While prior rehearsal methods ( Lopez-Paz & Ranzato , 2017 ) consider learning all of the streams of tasks at once ( multi-task learning ) as the performance upper-bound and replicate such a setting with samples stored in the episodic memory , a few samples from the pretraining corpus can not represent the overall world knowledge from the corpus . Moreover , if LMs are pretrained on a shuffled concatenation of stream of corpora , there is no guarantee that the LMs will acquire the correct , recent information from the recent corpora , especially in cases where the former corpora are much bigger than the latter ones , which is shown by experiments in Section 5.1 . ( 3 ) Lastly , prior parameter-expansion methods ( Rusu et al. , 2016 ; Yoon et al. , 2018 ) focus on learning a stream of different tasks via strong supervision , while in CKL , the focus is constantly updating world knowledge from a stream of corpora via self-supervision . Because of these fundamental differences , instead of contemporary CL methods mentioned above , we explore methodologies from the literature that are suitable for CKL ( Chen et al. , 2020 ; He et al. , 2021 ; Hu et al. , 2021 ; Wang et al. , 2021b ) , modifying and adapting each method according to our needs as CKL methods . Lastly , while it has been pointed out that some of the traditional CL formulations may have little practical importance in real-world scenarios by Prabhu et al . ( 2020 ) , CKL is much closer to the initial motivation behind CL , which is that the “ fundamental characteristic of natural intelligence is its ability to continually learn new knowledge while updating information about the old ones ” ( Prabhu et al. , 2020 ) . Details of related works regarding the traditional CL methods and how CKL methods address the fundamental differences are provided in Appendix A . 3 CONTINUAL KNOWLEDGE LEARNING ( CKL ) . In this section , we explain the formulation of the task , the data construction process , and the proposed metric measuring the trade-off between forgetting previous world knowledge and updating and learning of new world knowledge . 3.1 TASK FORMULATION . When viewing the task of renewing the internal knowledge of LMs as one of CL formulations , pretraining on the original corpus can be considered as a previous task , and continued pretraining on new corpus can be considered as the current task , the main objective becoming retaining the time-invariant world knowledge gained through initial pretraining while efficiently learning new and updated world knowledge through continued pretraining . Throughout the paper , we let D0 refer to the corpus used for initial pretraining and let D1 denote the new corpus used for continued pretraining . New Text Corpus for Language Modeling For LMs to renew their internal knowledge , they need to be continually pretrained on a new text corpus D1 which has the updated and new information . D1 should ideally be much smaller than D0 , as a large D1 amounting to the size of D0 will result in massive computational costs similar to pretraining the LMs from scratch . For constructing D1 , we crawl recently published news articles from the web making CC-RECENTNEWS.3 Probing LMs for World Knowledge The most widely used task for probing LMs for world knowledge is the LAnguage Model Analysis ( LAMA ) ( Petroni et al. , 2019 ) task , which consists of cloze sentences created from a set of knowledge sources using manually defined templates . We define that an LM knows a fact if it can successfully predict in a zero-shot manner the masked entity in the cloze sentence , such as “ Dante was born in ” as Florence . While there may be other alternatives for measuring the world knowledge encoded in LMs4 , we construct our main datasets as LAMA tasks , while also additionally providing the corresponding question pairs to the cloze sentences for those who want to test on CBQA as well . Measuring Retention of Time-invariant World Knowledge We define time-invariant world knowledge as the information present in D0 that has no possibility of conflicting with information from D1 . For example , if the information of the birthplace of Barack Obama is present in D0 , it is unlikely that D1 contains information that contradicts that fact . Also , we classify instances where the time-stamps are fixed such as “ Cristiano Ronaldo played for in 2010. ” as time-invariant . These time-invariant instances should not be changed as LMs are continually pretrained on D1 . In order to measure how much time-invariant information is lost due to catastrophic forgetting during continued pretraining , we create INVARIANTLAMA , a subset of LAMA ( Petroni et al. , 2019 ) , consisting of only time-invariant cloze sentences detailed in Appendix B.1 . Measuring Update of Outdated World Knowledge In this work , we define outdated world knowledge as information that is conflicting between D0 and D1 . For example , the President of the US may be Barack Obama in D0 and Joe Biden in D1 . In this case , the LM should update its internal knowledge as Joe Biden as the US president . If an LM is pretrained on both D0 and D1 simultaneously , there is no guarantee that the LM will acquire the correct , recent information from D1 , especially in cases where D0 is much bigger than D1 , which is one of the biggest difference between the CKL and traditional CL setting . For measuring update of outdated information , we construct UPDATEDLAMA which is made up of cloze statements for which answers can be found in both D0 and D1 , but are conflicting . Measuring Acquisition of New World Knowledge We define new world knowledge as the information present in D1 , but not in D0 . To measure new knowledge acquired through continued pretraining on D1 , we construct NEWLAMA which is made up of detailed cloze statements requiring new knowledge from D1 to correctly answer . We provide two datasets for measuring new world knowledge : NEWLAMA , for which each of the instances is verified that the answer does not exist in D0 , but only in D1 , and NEWLAMA-EASY for which each of the instances does not perfectly comply with our strict definition of new world knowledge due to its creation process , but is used to generally measure the new knowledge acquired from continued pretraining on D1 at a larger scale . NEWLAMA-EASY can be considered easier since each instance was constructed to be similar to the data distribution seen during continued pretraining . Dataset Construction The data for continual pretraining , CC-RECENTNEWS , is constructed using news-please ( Hamborg et al. , 2017 ) . INVARIANTLAMA is constructed by manually selecting 28 3CC-RECENTNEWS consists of 221,779 articles ( ∼168M tokens ) , which is estimated to be about 750 times smaller than C4 , a cleansed version of the April 2019 Common Crawl dataset ( https : //commoncrawl.org/ ) that was used to initially pretrain the T5 LM ( Raffel et al. , 2019 ) . 4Closed-book question answering ( CBQA ) ( Roberts et al. , 2020 ) can also be considered as a task that measures the world knowledge of LMs through finetuning , but it has been pointed out that much of its performance increases are due to the test-train overlap ( Lewis et al. , 2020b ; Wang et al. , 2021a ) in the datasets . time-invariant relations from T-Rex ( Elsahar et al. , 2018 ) . For UPDATEDLAMA and NEWLAMA , we use Amazon Mechanical Turk ( mturk ) 5 for crowd-sourcing Human Intelligent Tasks ( HITs ) . The process requires selecting answerable questions from a list of questions generated by the model introduced in Lewis et al . ( 2021 ) and converting them into cloze sentences . We have also separately hired 11 experts to verify the correctness and search the C4 database to categorize each instance following our definition of updated and new . NEWLAMA-EASY is constructed at a larger scale through a two-phase mturk process where sentences selected from articles containing new information are decontextualized and paraphrased6 before being masked , verified and converted to corresponding questions . The constructed dataset statistics are in Table 1 . Important details about the data construction pipeline , examples , and more fine-grained statistics are provided in Appendix B .

Summary: The authors formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). Particularly, they distinguished three sub-tasks in CKL, i.e., the retention of time-invariant world knowledge, the update of old knowledge, and the acquisition of new knowledge. They also introduce a new benchmark and metric to quantify the performance of various state-of-the-art models on these sub-tasks. They find that CKL demonstrates unique challenges that are not present in previous CL setups. Critical causes of knowledge forgetting in CKL are also discussed.