Towards Continual Knowledge Learning of Language Models
Authors: Joel Jang
Seonghyeon Ye
Sohee Yang
Joongbo Shin
Janghoon Han
Gyeonghun KIM
Stanley Jungkyu Choi
Minjoon Seo
Venue: ICLR 2022 Poster (2022)
Keywords: continual learning
knowledge acquisition
catastrophic forgetting
large language models
pretraining
natural language processing

Abstract: Large Language Models (LMs) are known to encode world knowledge in their parameters as they pretrain on a vast amount of web corpus, which is often utilized for performing knowledge-dependent downstream tasks such as question answering, fact-checking, and open dialogue. In real-world scenarios, the world knowledge stored in the LMs can quickly become outdated as the world changes, but it is non-trivial to avoid catastrophic forgetting and reliably acquire new knowledge while preserving invariant knowledge. To push the community towards better maintenance of ever-changing LMs, we formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). We construct a new benchmark and metric to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. We adopt applicable recent methods from literature to create several strong baselines. Through extensive experiments, we find that CKL exhibits unique challenges that are not addressed in previous CL setups, where parameter expansion is necessary to reliably retain and learn knowledge simultaneously. By highlighting the critical causes of knowledge forgetting, we show that CKL is a challenging and important problem that helps us better understand and train ever-changing LMs.

TOWARDS CONTINUAL KNOWLEDGE LEARNING OF
LANGUAGE MODELS
Joel Jang' Seonghyeon Ye! Janghoon Han? Gyeonghun Kim? KAIST AI LG Al Research
Sohee Yang' Joongbo Shin Stanley Jungkyu Choi?
Minjoon Seo!
[joeljang vano1205, schee. yang, minjocn]ekaist.co ac. kr shin, janghoon. han, ghkayne kim, stanley htep:/.chhhjegggrrrrch....
ABSTRACT
Large Language Models (LMs) are known to encode world knowledge in their pa- rameters as they pretrain vast amount of web corpus, which is often utilized for performing knowledge-dependeru downstream tasks such as question answer- ing, fact-checking, and open dialogue. In real-world scenarios, the world knowl- edge stored in the LMs can quickly become outdated as the world changes, but it is non-trivial to avoid catastrophic forgetting and reliably acquire new knowl- edge while preserving invariant knowledge. To push the community towards bet- ter maintenance of ever-changing LMs, formulate new continual learning (CL) problem called Continual Knowledge Learning (CKL). We construct a new benchmark and metric to quantify the retention of inverinvarinnt world knowl- edge, the update of outdated knowledge, and the acquisition of new knowledge. We adopt applicable recent methods from literature create several strong base- lines. Through extensive experiments, we find that CKI. exhibits unique chal- lenges that are not addressed in previous CL sctups, where parameter expansion is necessary to reliably retain and learn knowledge simultaneously. By highlighting the critical causes of knowledge forgetting, we show that CKL is a challenging and important problem that helps better understand and train ever-changing LMs. The benchmark datasets, model checkpoints, and code to reproduce our results are available this https URL
INTRODUCTION
Recent works have shown that large Language Models (LM), such as T5 (Raffel et al. 2019) and GPT-3 (Brown et al. (020). have the capability of storing tremendous amount of world knowledge in their parameters when pretrained on a vast corpus of text (Petroni al. 2019). These pretrained LMs have shown potential to serve as knowledge bases when probed for world knowledge without any finetuning through the LAnguage Model Analysis (LAMA) task (Petroni et al., 2019), which requires probing LMs for world knowledge in zero-shot manner through slot-filling. and promising results utilizing the encoded world knowledge when finetuned on various Knowledge Intensive Lan- guage Tasks (KILT) (Petroni et al. 2021), e.g., question answering, knowledgeable open dialogues.
While the world knowledge stored in LMs has diverse use cases, it can quickly become outdated the world changes fast, and LMs need to frequently renew their internal world knowledge cordingly. For example, is impossible to probe for new information such as Election 2020" from the original T5 (Raffel et al.) 2019) which was pretrained on C4 web corpus from April 2019 Also, information that may have once been considered accurate may no longer be valid because the information has been updated. For instance, the answer "Which soccer team does Cristiano Ronaldo play for?" has changed from Juventus to Manchester United in September 2021. Meanwhile, time-invariant information learned from the original corpus such as "Barack Obama was born in Honolulu, Hawaii" should not be altered within the LMs.
won the US
'T5 was initially pretrained on the C4 dataset (about 750 GB), which is cleansed dump of Common Crawl extracted from the web in April 2019.

---

Random LMs
Do
Pretrained LMs
D1
Continually Pretrained LMs
TS
GPT-2
T5
GPT-2
GPT-2
Pretraining
LM Pretraining
(continued)
INARIANTLAMA
UPDATEDLAMA
NEWLAMA
CKL. Benchmark
Overview of the CONTINUAL KNOWLEDGE LEARNING benchmark. INVARIANTLAMA is used measure the fime-invariant world knowledge gained from Do- UPDATEDLAMA is used to the update world knowledge from Do D NEWLAMA is used to measure new world knowledge gained from Dj.
Figure 1:
Despite its importance, the challenge of renewing the internal world knowledge stored in the pa- rameters of LMs is nontrivial and has only been explored in rather specific settings. For example, recent works have proposed to modify specific target knowledge such as individual facts (De Cao et al. 202T Zhu et al., 2020 Dai et al. (021)) Dhingra al. (202T) have addressed L.Ms as tempo- ral knowledge bases by jointly modeling text with its timestamp. But the problem renewing the world knowledge of LMs in a more general and scalable way, such through continual pretraining on a corpus with new knowledge, has not been formally formulated or explored by previous works. Moreover, the community lacks benchmark that can be used to systematically study how the inter- nal knowledge of LMs changes through the training on new information. Lastly, methodologies to effectively renew the knowledge of LMs scale have yet be thoroughly explored.
In this work, we propose novel continual learning (CL.) formulation named CONTINUAL KNOWT.- EDGE LEARNING (CKL), where attempt
the internal world knowledge of LMs through continual pretraining on new corpora. We systematically categorize world knowledge into three main categories and make benchmark datasets to measure each of them during CKL: (I) INVARI- ANTLAMA for time-invariant world knowledge in LMs that should not be forgotten or altered, (2) UPDATEDLAMA for outdated world knowledge that needs to be updated in the LMs, and (3) NEWLAMA for new world knowledge that should be injected into the LMs. We also propose novel metric named FUAR (FORGOTTEN (UPDATED ACQUIRED) RATIO) that can measure the trade-off between forgetting, updating, and acquiring knowledge. Finally, while one might think of implementing contemporary CL methods for this benchmark, we show that CKL has nontriv- ial differences to traditional CL formulations and require approaches specific to CKL. We find and compare model architectures and training methodologies (Chen et al., 2020 He et al., 2021 Hu et al. 2021 Wang et al.) 2021b) from the literature that have shown potential to mitigate forgetting of knowledge gained during pretraining, establishing them as baselines for the CKL benchmark.
In sum, while the challenge of renewing the internal world knowledge of LMs is essential in real- world scenarios, it has yet to be formulated or extensively explored. Therefore, in this paper:
We propose novel CL formulation called CONTINUAL KNOWLEDGE LEARNING (CKL) and construct a new benchmark to measure the amount forgetting and amount of world knowledge gained by continued pretraining on a novel language modeling corpus that we construct, containing new knowledge.
We explore LM architectures and training methodologies that are natural baselines for CKL in literature, denoting them CKL methods, and performing extensive experiments our CKL benchmark. We categorize them into regularization, rehearsal, and parameter- expansion methods, same in traditional CL literature. and compare the effectiveness of each type of method using novel metric named FUAR that we propose to measure the trade-off between forgotten knowledge and updated or acquired knowledge.
Towards creating an ever-changing LM, we perform extensive analysis the CKL bench- mark and highlight important challenges and findings: parameter-exppasion methods have the limitation of memory inefficiency despite performing the best in most our experi- ments and seeing the same data repeatedly during continued pretraining is critical cause forgetting. Also, we show interesting results that need further exploration: learning rate can be varied balance the forgetting and learning of new knowledge, CKL may help in
2

---

performing gervious-knowledge-iie -nteesive tasks after gaining new world knowledge, and CKL methods are transferable across LM architectures despite showing a different trend in performance.
An overview of the proposed CKL benchmark is shown in Figure
2 RELATED WORK
Language Models (LMs) utilizing knowledge from external sources, such as Retrieval-Aumennee Generation (RAG) (Lewis et al. 2020a) and Blender Bot 2.0 (Xu et al. 202T Komeili et al. [02T), cope with the changing world by updating the external sources during inference or searching the internet for retrieving recent information. However, recent works have shown that these memory- augmented models suffer from hallucination, which means that they present false information as if it were correct, despite being given updated knowledge during inference (Zhang & Choi [021),
which worsens as the size of the LM increases (Longpre et al. 2021). making it more so important for implicit parameters to be renewed as well.
In order to renew the internal knowledge of LMs, one might consider pretraining LMs from scratch with a newly updated text corpus of a scale similar to the one used during initial pretraining, such as a recent dump of the entire Wikipedia. However, this approach is computationally demanding and also environmentally harmful (Patterson al., 202T). Another alternative approach is contin- uing the pretraining process on much smaller corpus containing new world knowledge, but such a methodology is known suffer from catastrophic forgetting (McCloskey & Cohen 1989 Kirk- patrick et al. 2017), where the models forget previously learned knowledge they acquire new knowledge.
Lazaridou et al. ()02T); Jin et al. (2021) suggests implementing prior Continual Learning (CL) methods (Sun et al. 2020 d' Autume et al. 2019) to address this problem. However, is impor- tant to note that there are nontrivial differences between traditional CL and the proposed Continual Knowledge Learning (CKL) formulation which make applying traditional CL methods inadequate. In traditional CL, methods can be largely categorized into regularization, rehearsal, and parameter- expansion methods. (1) While regularization methods (Kirkpatrick et al.) (017)) require identifying important parameters used for previous tasks, exactly how and where the knowledge is stored in the parameters of an LM is currently extremely difficult to identify and localize (Vig et al.) 2020)
De Cao et al. 202T). (2) While prior rehearsal methods (Lopez-Paz & Ranzato (017) consider learning all of the streams of tasks at once (multi-task learning) as the performance upper-bound and replicate such a setting with samples stored in the episodic memory, a few samples from the pretraining corpus cannot represent the overall world knowledge from the corpus. Moreover, if LMs are pretrained on a shuffled concatenation of stream of corpora, there is no guarantee that the LMs will acquire the correct, recent information from the recent corpora, especially in cases where the former corpora are much bigger than the latter ones, which is shown by experiments in Section 5. (3) Lastly, prior parameter-expaasionn methods (Rusu et al. 2016 Yoon et al. 2018) focus on learn- ing a stream of different tasks via strong supervision, while in CKL, the focus is constantly updating world knowledge from a stream of corpora via self-supervision.
Because of these fundamental differences, instead of contemporary CL methods mentioned above, explore methodologies from the literature that are suitable for CKL (Chen et al., 2020 He al., 2021 Hu et al., 2021 Wang et al. 2021b), modifying and adapting each method according to our needs as CKL methods. Lastly, while it has been pointed that some of the traditional CL formu- lations may have little practical importance in real-world scenarios by Prabhu et al. (2020). CKL is much closer to the initial motivation behind CL, which is that the "fundamental characteristic of nat- ural intelligence is its ability to continually Iearn new knowledge while updating information about the old ones" (Prabhu et al., 2020). Details of related works regarding the traditional CL methods and how CKL methods address the fundamental differences are provided in Appendix A
CONTINUAL KNOWLEDGE LEARNING (CKL)
In this section, explain the formulation of the task, the data construction process, and the pro- posed metric measuring the trade-off between forgetting previous world knowledge and updating and learning of new world knowledge.
3

---

3.1 TASK FORMULATION
When viewing the task of renewing the internal knowledge of LMs as one of CL formulations, pretraining on the original corpus can be considered as a previous task, and continued pretraining on new corpus can be considered as the current task, the main objective becoming retaining the inneiinvaiiant world knowledge gained through initial pretraining while efficiently learning new and updated world knowledge through continued pretraining. Throughout the paper, we let Do refer to the corpus used for initial pretraining and let D, denote the new corpus used for continued pretraining.
New Text Corpus for Language Modeling For LMs to renew their internal knowledge, they need to be continually pretrained on new text corpus Dj which has the updated and new information. Dj should ideally be much smaller than Do. as a large D1 amounting to the size of Do will result in massive computational costs similar pretraining the LMs from scratch. For constructing D1, we crawl recently published news articles from the web making CC-RECENTNEWS 2
Probing LMs for World Knowledge The most widely used task for probing LMs for world knowledge is the LAnguage Model Analysis (LAMA) (Petroni et al.) 2019) task, which consists of cloze sentences created from a set of knowledge sources using manually defined templates. We define that an LM knows fact if it can successfully predict in a zero-shot manner the masked entity in the cloze sentence, such as "Dante was born in alternatives for measuring the world knowledge encoded in LM we construct our main datasets as LAMA tasks, while also additionally providing the corresponding question pairs to the cloze sentences for those who want to test CBQA as well.
" as Florence. While there may be other
Measuring Retention of Time-invariant World Knowledge We define time-inariant world knowledge as the information present in Do that has no possibility of conflicting with informa- tion from Dj. For example, if the information of the birthplace of Barack Obama is present in Do. it is unlikely that Dj contains information that contradicts that fact. Also, we classify instances where in 2010." as lime-invariant.
the time-stamps are fixed such as "Cristiano Ronaldo played for These invariiat instances should not be changed as LMs are continually pretrained on DJ. In order to measure how much iiverinvariant information is lost due to catastrophic forgetting dur- ing continued pretraining, we create INVARIANTLAMA, a subset of LAMA (Petroni et al., 2019). consisting of only time-invariant cloze sentences detailed in Appendix B.1
Measuring Update of Outdated World Knowledge In this work, we define outdated world knowledge as information that is conflicting between Dy and Dj. For example, the President of the US may be Barack Obama in Do and Joe Biden in D1. In this case, the LM should update its internal knowledge as Joe Biden the US president. If an LM is pretrained on both Do and DI simultaneously. there is no guarantee that the LM will acquire the correct, recent information from D1, especially in cases where Do is much bigger than D1, which is one of the biggest difference between the CKL and traditional CL setting. For measuring update of outdated information, we construct UPDATEDLAMA which is made up of cloze statements for which answers can be found in both Do and Dj, but are conflicting.
Measuring Acquisition of New World Knowledge We define new world knowledge as the in- formation present in Dj, but not in Dg- To measure new knowledge acquired through continued pretraining on D1, we construct NEWLAMA which is made up of detailed cloze statements requir- ing new knowledge from D1 to correctly answer. We provide two datasets for measuring new world knowledge: NEWLAMA, for which each of the instances is verified that the answer does not exist in Do. but only in Dj. and NEWLAME MASY for which each of the instances does not perfectly comply with our striet definition of new world knowledge due to its creation process, but is used to generally measure the new knowledge acquired from continued pretraining on Dj at a larger scale.
2CC- ECC-RECENTNEWE consists of 221.779 articles (~168M tokens), which is estimated to be about 750 times smaller than C4, a cleansed version of the April 2019 Common Crawl dataset (https//commonerawlloooo was used to initially pretrain the T5 L.M (Raffel al. 2019
that
*Closed-ba boooed-book question answering (CBQA) (Roberts et al. 2020) can also be considered as a task that mea- sures the world knowledge of LMs through finetuning, but it has been pointed out that much of its performance
increases are due the test-train overlap (Lewis et 2020b Wang al. 2021a) in the datasets.
4

---

Table 1: Dataset statistics, Input and answer length are the corresponding average token lengths.
Dataset
Size
Input Length
Answer Length
Dataset
Size
Input Length Answer Length
INVARIANTLAMA
17474
11.9
NEWLAMA
797
14.7
8.7
UPDATEDLAMA
13.7
NEWLAMA-BASY
11177
44.4
NEWLAMA-EAST can be considered easier since each instance was constructed to be similar to the data distribution seen during continued pretraining.
Dataset Construction The data for continual pretraining, CC- CC-RECENTNEWs, is constructed us- ing news-please (Hamborg et al. 2017). INVARIANTLAMA is constructed by manually selecting 28 time- invariant relations from T-Rex (Elsahar et al.) 2018). For UPDATEDLAMM and NEWLAMA, we use Amazon Mechanical Turk (mturk) for crowd-sourcing Human Intelligent Tasks (HITs). The process requires selecting answerable questions from a list of questions generated by the model introduced in Lewis et al. (2021) and converting them into cloze sentences. We have also separately hired 11 experts to verify the correctness and search the C4 database to categorize each instance following our definition of updated and new. NEWLAMA-EAN through a two-phase mturk process where sentences selected from articles containing new informa- tion are decontextualized and paraphrased before being masked, verified and converted to corre- sponding questions. The constructed dataset statistics are Table [ Important details about the data construction pipeline, examples, and more fine-graines statistics are provided in Appendix B
is constructed at larger scale
3.2 COMBINED METRIC FOR CKL
We propose novel metric, FUAR (FORGOTTEN / (UPDATED ACQUIRED) RATIO), that can compare the efficiency of each CKL method using the trade-off between forgotten time-invariant knowledge and updated newly acquired knowledge. FUAR represents relatively how many time- invariant knowledge instances are forgotten in order to learn one new or updated knowledge instance. We first define FUAR for the general case where there can be multiple corpora used for training an ever-changing LM.
Let be an arbitrary task and (Di) be a sequence of corpora used for LM pretraining, where Do is the initial pretraining corpus. We define D,, Di) = (T)re of LM, - Score(T) of LM,, where LM, represents the LM after being pretrained on Da. Then, we denote T = (T )=o from each corre-
sequence of tasks from (Di)=o) measuring the forgetting of invariant-knowle sponding corpous. If there is no such task from corpus D;, the value of T/ is set to n.d., which means not defined. Likewise, denote T and T tasks from D, measuring the update and acquisition of new knowledge, respectively. We define FUAR as follows:
Σ max Gap(T/ D.. #ndd.
FUAR(TE Th TTA)
Dn, Di) (max(O,Gap(T)
fnd.) max(0, max(0, D,,, Di)] (1.
if denominator > 0, no gain, otherwise.
The choice of benchmark tasks TF, T and T can differ according to each experimental setup- FUAR value of 1.0 represents an equal trade-off scenario where one time-invariant knowledge in- stance of TF is forgotten on average to gain one new or updated knowledge instance of Th and TAA. The two terms in the denominators are summed because newly gained knowledge and updated knowledge mutually exclusive by definition. When the value is smaller than 1, it means that the model obtains more new or updated knowledge than the amount of forgotten knowledge, so methods
hhtps://www.mturkk com
Deconnextualization model from Choi et al. 2021 and hack-translation model from Tiedemann & Thot- ringal 2020 is used.
5

---

that exhibit low FUAR value can be considered suitable for CKL. If the value is zero, then it is case where no forgetting occurs at all and the upper bound for performance. If the denominator is O, we denote the case as no gain and regard it as the worst possible case f6
4 EXPERIMENTAL SETUP
We perform extensive experiments with an encoder-decoder model, T5 (Raffel et al. 2019). a large LM (~ 737M params) initially pretrained on April 2019 dump of C4 and May 2020 dump of Wikipedia (thus Do in our experiments) with salient span masking (SSM). The details of the pre- training, continual pretraining, and evaluation configurations following methods as the baselines for the CKL benchmark and categorize them into regularization, rehearsal, and purameter-expansiion methods. The specific hyperparamters used for the implemen- tation of each method are detailed in Appendix D
in Appendix C We establish the
Initial refers to the setting where evaluate the LM before any continued pretraining. The perfor- mance of this model can be considered as the upper-bound for INVARIANTLAMA and lower-bound on UPDATEDLAMA and LAMA.
Vanilla is a specific setting of further pretraining (Gururangan et al. 2020), where the domain is new knowledge, and the LM is further pretrained without any training strategies.
RecAdam (Chen et al., 2020) falls into the category of regularization methods. It places a stronger independent assumption among the model parameters than the traditional regularization method (EWC (Kirkpatrick et al.) (017)) and does not access the initial pretraining corpus to regularize the model weights during continued pretraining. The optimizer annealed so that less regularization applied as the training progresses.
(He et al. 202T) falls into the category of rehearsal methods, which assumes access to the initial pretraining corpus and mixes in random subsets of the initial pretraining data during con- tinued pretraining, depending on the mix-ratio at the current time step. As the training progresses, the mix-ratic decays towards O, decreasing the amount of the mixed original data at each iteration.
LoRA (Hu et al., 2021) falls into the category of parameter-cxpansion methods. It freezes the original parameters of the LM and adds trainable rank-decomposttion matrices into each layer that are updated during continued pretraining. Hu et al. (2021) has implemented this approach with decoder-only models (GPT-2 (Radford al.] 2019) & GPT3 (Brown et al. 2020)) while apply
it to an encoder-decoder model, denoting it as T5-LORA.
K-Adapter (Wang al. 2021b) is another paramsoer-expansion method that freezes the original parameters of the LM while adding & number of new layers, namely adapters, that are updated during continued pretraining. [Wang et al. (2021b) have shown successful injection of factual and linguistic knowledge for encoder-only models, BERT (Devlin et al., 2019) & RoBERTa (Liu et al., 2019 while we also apply it to an encoder-decoder model, T5, and decoder-only model, GPT-2.
Modular is a newly proposed parameter-expansion method specifically for encoder-decoder models which freezes the original, pretrained encoder while adding a new, randomly initialized encoder that is updated during continued pretraining. For the newly added encoder, we vary the size to T5-small while keeping the size of the original encoder and decoder to be T5-large.
EXPERIMENTAL RESULTS
In this section, we first show the main experimental results for the CKL Benchmark. Then, since multiple steps of continual knowledge learning, i.e., CKI. are needed for training a true, ever- changing LM, we explore the effects of multiple CKI. phases as well as how epochs, corpus size, and the total number of training steps affect CKL. We further explore how learning rates affect CKL in Appendix
E how continual pretraining Dj affects the performance of KILT tasks which re-
'Each of the last two sentences means that we do not measure positive backward transfer and negative for- ward transfer, respectively. The latter in some cases actually do happen (shown in Appendix G. Explanations about the backward and forward transfer are Appendix A.

---

Table Zero-shor probing performance on the CKL benchmark. The best results for each task and metric are shown in bold, and the second- best results are underlined.
# of Params
IL
UL
NL
NLE
FUAR (LIL), UL,NL. 4
Method
(Trainable / Total)
EM
EM
EM
EM
T5-Iniiaal
OM/ DM/73%7 737M
24.17
1.62
1.88
10.32
T5- T5-Vanilla
737M / 737M
12.89
10.17
3.77
17.75
1.08
TS-RecAdam
737M / 737M
13.20
12.55
4.02
17.85
0.84
T5-MixReview
737M / 737M
13.92
6.49
2.89
14.86
1.74
T5-LoRA
403M/ / 738M
16.58
12.77
4.52
19.56
0.55
T5- T5-Kadapters (k=2)
427M 762M
19.59
12.34
5.03
18.75
0.33
T5- Kadapters (k=3)
440M/ / 775M
19.76
12.66
4.02
19.00
0.33
T5-Modular
438M / 773M
20.29
12.66
4.65
19.24
0.28
quire knowledge from Do in Appendix how CKL methods transfer across LM architectures in Appendix G and how the prediction outputs change during CKI in Appendix
5.1
MAIN RESULTS
Table shows main experimental result the CKL benchmark. While only the exact match (EM) is reported in Table 2 we report the F1 score as well as the mean precision at k (P@R k k=1,5,10,20,50. 100) in Appendix [ The T5 models
originally pretrained on C4 (about tril- lion token updates) and Wikipedia, which is considered as Do and then continually pretrained on CC-RecentNews (corpus D1) for epochs (25k global training steps, about 673 million token updates) using each of the CKL methods. Each of IL, UL, NL. NLE stands for INVARIANTLAMA UPDATEDLAMA. NEWLAMA, and NEWLAMA-EASY respectively. Detailed descriptions about the setup for this experiment are included in the caption.
We first find that all of the CKL methods except for T5-MixReview are more effective at forgetting less time- inne-invariant knowledge while updating and acquiring new knowledge than using the naïve ap- proach of 75-aanilla as shown by the FUAR. This result also highlights the main difference between CKL and CL; while rehearsal methods show strong performances in traditional CL settings (Prabhu jet al. 2020 Bang et al. 202T). in CKL, shows the worst performance since the update of outdated knowledge and acquisition of new knowledge is severely deterred shown in the performance of UL and NL while not showing competitive mitigation of forgetting as shown in the performance of II. compared other CKI. methods. Amongst the other CKI. methods, we observe rather sistent trend that the parameter-expansee results on all of UL, NL, and NLE are all from parameterr-epansioon methods. Meanwhile, although UL and NL are constructed following the same procedure, there is a huge difference between the EM scores of UL and NL. We analyze the source of this difference in Appendix
methods achieve better results. The first and second-best
Figure 9 visualizes how the EM scores of each task change as T5- Kadaattts, the CKL method with the most robust performance, and T5- Vanilla are continually pretrained on Dj. In all of the tasks, the performance of In-Iniaaal can be considered as the upper-bound for IL and fower-boum for UL, NL, NLE. Corresponding with our main observations, CKL allows considerable retention of time- invariant world knowledge while improving updating and gaining new world knowledge compared to Vanill,,,a, mitigating the overall trade-off.
5.2 EXPLORING MULTIPLE PILASES OF CKL
In order to show the potential for creating truly ever-changing LM, we explore the effect of multiple CKL phases by creating CC- CCCRREENNNNESSSSMA SSMALL denoted SMALL, which is a small variant of CC-RECENTNEWS that consists of randomly sampled 10% of the original corpus. We then split
In this work, we see C4 and Wikipedia together as Dg. because do not measure how the knowledge in LMs change in between training on those corpora.
7

---

T5-Initial
... T5-Vanilla
55-Kaappters (k=2)
19
BM
30
EM
4.5
EM
3.5
1.6
Epochs
Epochs
Epochs
Epochs
(a) INVARIANTLAMA
(b) UPDATEDLAMA
(c) NEWLAMA
(d) NEWLAMA-EASY
Figure Performance each epoch during continued pretraining in main experimental setting.
Table 3: Zero-shot probing performance after models are continually pretrained on different subsets of CC-RECENTNEWS. NLE and IL stand for NewLAMA-East and InvariantLAMA respectively. There are three scenarios according to the corpus used for continual pretraining, explained in the text of Section [5.2] The FUAR of the three scenarios is calculated differently, and the corresponding tasks shown in the table as the parameters of FUAR: TV, 7 and TA In this setting, T consists of only single task Tg (IL) measuring the time-invariant information lost from Dn only. SMALL. we calculate the gap on NLE using the weighted sum of the gaps on NLEpt and NLEp2 with uniform weights.
Purams
NLEPi
NL.Pp
Corpus
Method
(Trainable Total)
EM
EM
EM
FUAR
TS-Iniial
737M
24.17
8.69
9.45
(IL), NLE)
T5- 75-Valllla
237M/737M 737M
11.86
17.77
16.42
1.53
T5-RecAdam
737M/ 737M
11.85
16.46
13.93
SMALL
T5-MxxReview
737M / 737M
14.36
14.18
13.93
1.97
(SMALL-PI
T5-LoRA
403M / 738M
14.26
20.60
19.90
0.87
SMALL-P2)
T5- Kadappets (k=2)
427M 762M
18.16
18.34
16.42
0.72
TS-Kadapters (k=3)
440M/ 775M
17.12
20.98
20.39
0.61
Modular
438M 773M
16.40
19.47
FUAR
TS-Initial
AM/737M
24.17
8.69
9.45
(IIL) n.d.. NLEPi)
TS-Vanilla
737M/ 737M
9.68
20.60
11.44
1.22
T5-RecAdam
737M/ 737M
11.78
20.42
11.94
1.06
T5-MixReview
737M / 737 M
16.13
15.88
11.94
1.12
PM
T5-LoRA
403M / 738M
14.75
20.79
13.93
0.78
T5- 75-Kadapters (k=2)
427M / 762M
19.11
20.60
10.95
0,42
Kadapters (k=3)
440M / 775M
19.08
18.15
10.94
0.54
TS-Modular
438M / 773M
17.08
18.90
11.94
TS-Initial
0M / 737M
24.17
8.69
9.45
(II., m.d.), n.d.. NLEE2)
T5-Vanilla
M M
9.40
14.37
23.38
T5-RecAdam
737M/ / 737M
7.25
14.56
20.90
1.48
T5 MixReview
737M / 737M
13.20
17.20
16.92
1.47
SMALL-P1-:
T5-LORA
404M 740M
13.25
16.07
22.39
0.84
SMALL-P2
Kadapters (k=2)
427M
15.78
16.07
23.38
TS-Kaaapters (k=3)
440M/ 813M
15.47
15.31
20.90
0.76
T5-Modular
438M/ 809M
14.66
15.31
20.40
0.87
CC- CC-RRCENNNESSS-SMAAA into two different splits by the published date of each article to simulate setting where multiple CKL phases are needed, denoted as SMALL-PI (05.2020 11. 11.2020)) and SMALL-P2 (11.2020 04. 04.2021). NLEa is also split into two different, smaller datasets, NLEp1 and NLEp2, each comprising of instances constructed from articles in SMALL-PI P1 and SMALL-P2,
We use JEWLAAM-EAAS instead of NEw LAMA because the number of instances in NI. corresponding articles from MMALL too small for robust evaluation.
8

---

15-Iniaal
EM
Small-Pi
TS-iiiiall
SSmall-PI
Main
Main
Small
Small
Small-P1 Somal-PP-Smannn Small--P
Small-PI Small-P2
7900
1909
200
I0ea)
Global Training Steps
Global Training Steps
(a) Vanilla
(b) T5- 15-Kadapters (k=2)
Figure 3: Performance each epoch on INVARIANTLAMA during continued pretraining in MAIN, SMALL, and MHALL-PI-SMALLE scenarios. Each marker indicates the result at each continual pretraining epoch.
respectively. We compare how CKL methods for T5 perform on IL, NLEp1, and NLEp2 when continually pretrained entirely SMALL for 5k steps (8 epochs), and when sequentially pretrained on SMALL-P1 and then on SMALL-P2 for 2.5k steps (8 epochs) each. In the scenario SMALL- P1 P1-SSMALL-P2, there are CKL phases where Do is C4 and Wikipedia, D) is SMALL-PII, and D2 is SM.LL-P2. The rest of the configurations are set identical with the main experiments.
Comparing the performance sults show that LMs are prone to more forgetting as they go through multiple CKL phases, despite having the same number of training steps. One of the reasons may be due to the learning rate scheduling, which is initialized at the start of each phase.
IL of the two scenarios, SMALL and PIALL-PP-SMMAL-P2
Furthermore, despite showing the best performance overall, the drawbacks of parameter-eepansion methods are also highlighted in the SMALL-PI-SMALLE be added at every phase of the update. For example, the number of total parameters T5-Modular increases by 36M in every round of the continual pretraining phase. Likewise, considering large number of CKL phases introduces new problems that should be additionally studied. Taking into account that LMs should be updated frequently with small amount of data in real-world scenarios for gaining up-to-date world knowledge about the ever-changing world computation-effeeet manner, more research is needed to mitigate the amount of forgetting that follows the larger number of update phases.
setting; they require new parameters to
Effects of Epochs, Corpus Size, and Total Number of Training Steps in CKL Forgetting Figure 3 shows the result of TS-Vanilla and T5-Kadapters during continued pretraining in different scenarios from Table 2 and 3 where each point in the graph represents the performance of IL after every epoch. Comparing g MAIN (4 epochs) and SMALL (8 epochs) in Figure 3 (a) T5- Vanilla, we can see that more forgetting occurs SMALL, even though trained for five times less number of global training steps. This phenomenon is further highlighted when comparing results from SMALL-P/ (8 epochs) which shows the most amount forgetting despite being trained for ten times less number of global training steps. While the overall drop is much mitigated in Figure (b) T5- T5-Kadapters, we observe the same trend between each scenario which goes to show how critical observing the same data repeatedly during continued pretraining is for causing forgetting.
The results are line with findings from Lee et al. (202T) which suggest LMs should be pretrained with just few epochs less duplicating data for efficiency. We add additional intuition to their findings and conjecture that the inefficiency of pretraining from duplicate data could have been caused by the forgetting of the rather long-tail knowledge in the pretraining corpus.
6 CONCLUSION
In this paper, we propose CONTINUAL KNOWLEDGE LEARNING (CKL), where we establish benchmark datasets and metrics, and explore methodologies towards continual knowledge learning of an ever-changing LMM We find that parameter-expansion methods show the most robust perfor- mance throughout all of the experimental settings, which nevertheless has severe memory ineffi- ciency and that seeing the same data often is critical cause of forgetting. We also discuss several other interesting results of which we leave further exploration to future studies. To this end, we suggest the community explore CKL for the better design of ever-changing L.M

---

ACKNOWLEDGMENTS
The authors would like to thank Sang- Sang-Woo Lee, Jinheon Baek, Miyoung Ko, Hyunji Lee, and Eunbi Choi for helpful discussions. This work was supported by Institute of Information & communica- tions Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).
REFERENCES
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with memory of diverse samples. In CVPR, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In EMNLP, 2020.
Eunsol Choi, Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. Decontextualizatton: Making sentences stand-alone. TACL, 9:447-461, 2021.
Damai Dai, Li Dong, Y. Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained trans-
formers. ArXiv, abs/2104.0869)) 2021.
Cyprien de Masson d' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. NeurlPS, 2019.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In EMNLP, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Bhuwan Dhingra, Jeremy Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. Time-aware language models as temporal knowledge bases. arXiv preprint arXiv:2106. 15/ 10, 2021.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: showledge-poweres conversational agents. In ICLR, 2019.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Elena Sim- perl, and Frederique Laforest. T-rex: A large scale alignment of natural language with knowledge base triples. LREC, 2018.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. ACL, 2019.
Zhaochen Guo and Denilson Barbosa. Robust named entity disambiguation with random walks. Semantic Web, 9(4):59 2018.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: adapt language models to domains and tasks. In ACL, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: augmented language model pre-training. /CML, 2020.
Retrieval-
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In 15th International Symposium of Information Science (ISI 2017), pp. 218-223, 2017.
10

---

Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. An- alyzing the forgetting problem in pretrain-finetuman finetuning of open- ppen-domain dialogue response models. In EACL, 2021.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol. Bilyana Taneva, Stefan Thater, and Gerhard Weikum. Robust disambiguation of named entities in text. In EMNIP,, 2011.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:2106.0990 09685, 2021.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang- Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint arXiv:2110.08534 2021.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Band Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14(13):3521-3522 2017.
Mojtaba Komeili, Kurt Shuster, and Jason Weston, Internet-augmented dialogue generation. arXiv preprint
2021.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. TACL, 7:453 466, 2019.
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tay- fun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Sebastian Ruder, Dani Yogatama, et al. Pitfalls of static language modelling. arXiv preprint arXiv:2102.01957, 2021.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang. Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2 (tt.:2007.004999 2021.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In CoNLL, 2017.
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,
Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented gener- ation for imowledge-intensing nlp tasks. In NeurIPS, 2020a.
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. Question and answer test-train overlap in open-domain question answering datasets. arXiv preprint 2:Xiv:2008.02667, 20200
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably- questions and what you can do with them. In EACL, 2021.
Yanyang Li, Ye Lin, Tong Xiao, and Jingbo Zhu. An efficient transformer decoder with compressed sub-layers. arXiv preprint arXiv:2101.0054,, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Eattty-based knowledge conflicts in question answering. arXiy preprint arXiv: 05052,0 2021.

---

David Lopez-Paz and Marc'Aus Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurIPS, 2017.
Michael McCloskey and Neal J Cohen, Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 24:109-165, 1989.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint 103.5000, 2021.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? In EMNLP, 2019.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowl- edge intensive language tasks. In NAACL, 2021.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. E-bert: Efficieent-et--ffeettiee entity embeddings for bert. In Findings of EMNLP, 2019.
Ameya Prabhu, Philip HS Torr, and Puneet Dokania. Gdumb: simple approach that questions our progress in continual learning. In ECCV, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.1063,, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In EMNLP, 2020.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatriek, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint vrXiv:1606.04671, 2016.
Taylor Shin, Yasaman Razeghi, Robert I.. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP, 2020.
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. Leee. Lamol: Language modeling for lifelong language learning. In ICLR, 2020.
James Thome, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification. In NAACL, 2018.
Jörg Tiedemann and Santhosh Thottingal. OPUS-MT - Building open translation services for the
World. In EAMT, Lisbon, Portugal, 2020.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. Facts experts: Adaptable and interpretable neural memory over symbolic knowledge. In NAACI, 2021.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang. Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. In NeurlPS, 2020.
Cunxiang Wang, Pai Liu, and Yue Zhang. Can generative pre-trained language models serve as knowledge bases for closed-book qa? In ACL, 2021a.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of ACI, 2021b.
12

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin I Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In EMNLP System Demonstrations, 2020.
Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open open-domain conversation. arXiv preprint arXiv:107..0567, 2021.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NewrIPS, 2019.
Michael J.Q. Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. EMNLP, 2021.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:20/2.00363, 2020.
13

---

A EXTENSION OF RELATED WORKS
As mentioned in Section 2 there are fundamental differences between the traditional CL formu- lations and CKL which make the previous CL. methods inadequate for the CKI. setting. In this section, we introduce the prior traditional continual learning methods in detail, explore the methods from the literature set as baselines for the CKL benchmark and how they address the identified lim- itations of CL methods, and provide descriptions about alternative methods making LMs cope with the changing world.
A.1 TRADITIONAL CONTINUAL LEARNING
Traditional continual learning (CL) methods focus on addressing two aspects of transfer between sequentially incoming tasks: forward transfer and backward transfer [Lope7-Paz. & Ranzato 2017). Forward transfer refers how past tasks affect the performance of the current and future tasks. Backward transfer refers to how current or future tasks affect the performance of previous tasks. The general pretrain-fing finratnn approach can be seen as an instance of positive forward transfer where model performs better target task after being pretrained on more general source task. More- over, catastrophic forgetting can be seen as an instance of negative backward transfer where previous tasks suffer performance due to continued training different tasks. With respect to these two pects, CL. approaches can be categorized into three main approaches: regularization, rehearsal, and parameter-expansion methods.
Regularization Elastic Weight Consolidation (EWC) (Kirkpatrick et al., (017)) is a method that regularizes important parameters of previous tasks while training for the current tasks, helping mit- igate the negative backward transfer of previous tasks. Important parameters are measured via Fisher information matrix computed by measuring the magnitude of the gradient update step of each parameter during training of previous tasks.
Rehearsal Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato 2017) is one of the first rehearsal methods that utilize samples from each task stored in episodic memory and places an inequality constraint with respect to the losses of the samples in order to prevent negative backward transfer well as allow the positive backward transfer. Other methods such Experience replay and local adaptation (d Autume et al., 2019) replay samples stored in the memory of previous tasks during training mitigate forgetting.
Parameter-expansion Progressive Neural Networks (PNN) (Rusu et al. 2016) is one of the car- liest eurammetrr-xxpaniionsshare approaches that introduce new sets of parameters for each new task where previous parameters are frozen and can be connected via lateral connections allowing for positive forward transfer. PNN only prevents negative backward transfer but also surpassed the previous metrain-finetune approach in terms of positive forward transfer in some tasks.
A.2 CKL METHODS FOR LANGUAGE MODELS
As mentioned in Section 2 we explore the methods from the literature that have addressed the limitations of CL methods and thus are applicable to CKL. We also categorize these methods into the three main categories of CL.
Regularization Most CL methods that utilize regularization require computing important param- eters of the previous task, which in this case is pretraining on the original text corpus. Determining these parameters is oftentimes unrealistic since it requires large-scale pretraining which can hardly be replicated by most. Also, exactly how and where the knowledge is stored in the parameters of an LM is currently extremely difficult to identify and localize (Vig et al. 2020 De Cao et al. 2021). RecAdam (Chen al., 2020) overcomes this limitation by following the same training objective as EWC (Kirkpatrick et al. 2017) with stronger independent assumption and places a quadratic penalty, ridding the need access the initial pretraining corpus.
14

---

Rehearsal Large LMs are usually pretrained vast amount of raw text corpus such Common Craw | When treating pretraining as a CL task, limitations exist when trying to apply previous rehearsal methods since a few samples from the pretraining corpus cannot represent the overall world knowledge from the original pretraining corpus. Mix-Review (He et al. 2021) solves this issue by performing preliminary experiments in smaller pretraining setting by assuming access to the pretraining corpus during finetuning and mixing random subsets of pretraining corpus depending on mix-ratio that anneals towards the target task as training progresses. Mix-Review can be considered a mild version of multi-task learning.
Parameter-expansion K-Adapter (Wang et al. 2021b) shares and freezes the original parame- ters and adds new parameters through adapters for continued pretraining of factual and linguis- tic knowledge and improve performance on three different knowledge-driven downstream tasks, More recently, LoRA (Hu et al. 2021) freezes the original parameters and injects trainable rank- decomposition matrices into each layer of the Transformer architecture, greatly reducing the num- ber of trainable parameters and the computational hardware requirement while performing on-par or better than training all of the parameters. Both methods hypothesize freezing the original parameters allows mitigation of catastrophic forgetting. We test out the hypothesis through implementation in our CKL benchmark.
A.3 METHODS OF INTEGRATING WORLD KNOWLEDGE WITH LANGUAGE MODELS
Explicit Methods Facts-as-Experts (Verga et al. 2021) store representations entities the form of vayuv pairs into external memory that can be modified during inference time. RAG (Lewis jet al. 2020a) accesses a dense vector index of Wikipedia with a retriever and swaps indexes for updating the behavior of the model the world changes. Blender Bot 2.0 (Xu et al. [021| Komeili et al. 2021]. is also one of the explicit methods that search the internet for recent knowledge and saves recent conversations in external long-term memory. Explicit methods, such as swapping in- dexes, adding explicit entity-relation knowledge, or searching the internet are in need of manual intervention during inference or are bound to tasks that require retrieval. In this paper, we focus only on implicit methods.
Implicit Methods Zhu al. (200)) proposed new task of explicitly modifying specific facts without forgetting unmodified facts and provided several benchmark approaches without utilizing non-parametric memory, including constrained layer-wise finetuning. [Wang et al. (2021b) proposed K-Adapter, a method that adds adapters to frozen layers of pretrained LMs to inject factual and linguistic knowledge and improve performance on downstream tasks. Chen al. (2020) proposed a new optimizer that simulates the pretraining optimization while finetuning on the target task without needing access to the pretraining corpus, improving performance on the GLUE benchmark. De Cao et al. (202T) propose using hyper-network to edit factual knowledge.
Even though these implicit methods are efficient methods of injecting or modifying knowledge from the implicit parameters of the LMs, they are all limited to injecting specific knowledge such the case of (Wang et al., (221 or modifying past knowledge such as the case of (Zhu et al. 2020) De Cao al.] 20221) D. No work, the best of our knowledge, has specifically addressed the catas- trophic forgetting of world knowledge gained from the initial pretraining when continued pretraining on new text corpus for the gain of new world knowledge.
B DATASET CONSTRUCTION
In this section, describe the dataset construction process undergo in creating the benchmark datasets used in CKL. For the construction, we use Amazon Mechanical Turk (mturk)T)) To for crowd- sourcing Human Intelligent Tasks (HITs) and separately hire 11 experts for annotation that requires extensive searching of the C4 corpus. In addition, three more experts"l] III who set up the data con- struction process and prepared the annotation guideline to ensure the quality of the data through post-validation and giving feedback to the annotators in real-time. The interfaces used for mturk HIT's are provided in Appendix B.2
$ "htts://commoncrawloorg
https:/ww ww www.wwww.woooo com
The first three authors of the paper.
15

---

CC-RexmiNow
CC-BaasmNNess
HIT
PAQ Question Generator
Jere ane infommsiion
Thin the aner antile
no
dirggan!
prulash--an siiel pacntioss
Select queerion that regince novie knowledge
AND be found the anricle
conweys
utiile Then.
quenion
ABV. Sioweor alsna
solected word
Decontextualizztion Model
Pretrained T5
Back- Translation Model
ainogard
In-Lab Expertsi
desonienaiizvi paraplnural
the predictive correct in C47
Aredictioon AeV pareer
old poswer prediction endonre rusdecee in add to Updittdl.AMA
parsparaased aentonce
aurver stll
80
dumpaad
Can alserativv auesr he foun Ca?
vee
you
Convert to question and answer the question
cháience esidence old to Eyduned AMA
Tie the
duragord
diseguni
(2)
(b)
Figure Dataset construction pipeline for (a) UPDATEDLAMA, NEWLAMA, and (b) NEWLAMA-FASY
CC-RECENTNEWS We first construct CC-RECENTNEWS, novel text corpus containing rel- atively new knowledge as DT. We use news-please (Hamborg et al. 2017), similar to the CC- NEWS (Liu al., 2019) and REALNEWS dataset (Zellers et al.) 2019), to crawl 221,779 news articles published from May 2020 to April 2021. LMs initially pretrained on Do constructed be- fore May 2020 can be continually pretrained CC-RECENTNEWS to gain relatively recent world knowledge.
INVARIANTLAMA We create INVARIANTLAMA, subset of the LAMA (Petroni et 2019) task for measuring time-invariant knowledge which might be forgotten during CKL. Among the 41 relations of the T-REx (Elsahar et al. 2018) subset of LAMA, we manually select 28 relation types that probe for sime-invarian instances (a full list of iime-invariant relations are provided in Appendix B. We also remove instances where the answer overlapped with the subject following Poerner et al. (219)) since the answers for these instances can be inferred from the cloze statement itself. Lastly, we remove instances where the answer was a non-entity to leave only the instances that require world knowledge for prediction on their answers (Guu et al. 2020).
UPDATEDLAMA and NEW LAMA We construct UPDATEDLAMA and NEWLAMA for mea- suring the update of outdated knowledge and acquisition of new knowledge during CKI.. The chal- lenge of constructing UPDATEDL M. A is that a knowledge instance can be only considered as the knowledge that requires update only if it present in both Do and Dj with changed details, and the challenge of constructing NEWLAMA is that the knowledge can be considered new only if is in D, but not in Do- Therefore we set up the data construction process carefully. The pipeline for the creation of single instance of UPDATEDLAMA and NEWLAMA, is shown in Figure 4 (a). Each potential instance starts off from single article from CC-RECENTNEWS and goes through the pipeline which will end up being (1) discarded (2) added to UPDATEDLAMA or (3) added to follows:
NEWLAMA in the end. The procedure
(1) First, list of Probably-Asked Questions (Lewis et al. 202T) are generated using the PAQ ques- tion generator on a single news article from CC- RECENTNEWS. (2) The list PAQs and the news article is given to the crowd-sourced worker to select question that asks for the most recent knowl- edge for which the answer (denoted as new answer) can be found in the article. (3) The crowd-source worker is instructed convert the question into cloze sentence that it can be given as input to a pretrained T5 LM. The predictions of the T5 LM are stored along with the questions and cloze sentences. (4) The expert annotator ensures the quality of the questions and cloze sentences by
16

---

recting them whenever necessary and checks whether the model prediction is correct by searching through the C4 corpus as a representative of Dr If the prediction is correct and the prediction is not the same with the new answer, the following instacce must be present in both Do and Dj with details changed, and thus is added to UPDATEDL AMMA along with the evidence document found in C4. If same, the instance is discarded because the instance is neither updated nor new. (5) Lastly, if the model prediction is wrong, the expert annotator is asked to find an alternative answer for the question in C4. If not found, the instance added NEWLAMA since the answer the question could only be found in the article of CC- CC-RECENTNEWS (D1), but not in C4 (Do). Similarly, if the alternative answer is found in C4, we check whether it is the same as the new answer and add the instance to UPDATEDLAMA if not the same and disregard it otherwise.
Throughout the whole process, a validator checks the sanity of the data and gives detailed real-time feedback on the work of the annotator.
NEWLAMA-FASY Even though NEWLAMA corresponds to our exact definition of new knowl- edge that we define in the task formulation, scaling the size of the dataset was difficult since each instance required searching the whole C4 database for answers. Instead, easier variant NEWLAMA-EAST where we test the general new knowledge acquired during con- tinued pretraining CC-RECENTNEWS. The pipeline for the creation of a single instance of NEWLAMA-EAST is shown in Figure 4 (b) and follows the following procedures:
provide much larger,
(1) First, the crowd- worker is instructed to classify whether the given article contains new information or not. (We define new as not likely to be known before May 2020). If the article con- tains new information, the worker is instructed select sentence from the article that contains the most recent information and an entity among the possible answer candidates in the sentence and dis- card the article if otherwise. We provide the possible entities through a Named-Entity Recognition Model. (2) We make the selected sentence stand-alone from the article through the decontextual- ization model provided by Choi et al. (202T). (3) The decontextualized sentence is paraphrased by a transsattnttton model (en-xde-een (Tiedemann & Thottingal 2020) and checked whether the selected word is still in the paraphrased sentence; the sentence discarded not. (4) Next, we mask out the selected word from the sentence and ask two crowd-sourced workers to convert the cloze sentence into a question and answer the question. (5) If the answers agree among the workers as well as correspond to the actual selected word, we add the instance to NEWLAMA-EASY.
The specific interfaces used for the mturk HITs are provided in Appendix B.2 Statistics of the constructed datasets are in Appendix B.3
B. TIME-INVARIANT RELATIONS OF LAMA
Table 4 shows the list of 28 time-invarian relations of INVARIANTLAMA. We manually filter the 44 original LAMA relations to leave only the dime-invariant relations. Templates such as "[X] works for [Y] ." and "[X] is a member of [Y] are excluded because the answer may change for different timestamps. In the template, [X] and [Y] refers to subject and object labels, respectively. Given
template with only the subject included, the model has to predict the object label [Y] for knowledge probing.
B.2 INTERFACES USED FOR THE CONSTRUCTION OF CKL BENCHMARK
The Mturk interface used during construction of UPDATEDLAMA and NEWLAMA, NEWLAMA- EASY, and NEWLAMA- -ASY are shown in Figure and respectively.
B.3 DATASET STATISTICS AND EXAMPLES
We report the data statistics for the CKL benchmark in Table We measure the size, average input token length, average answer token length, and the answer types of each constructed dataset. One thing to consider is that LAMA (Petroni et al. 2019) from which we constructed IN INVARIANTLAMA is originally constructed for only single-token decoding (1.3 with the T5-tokenizer) because multi- token decoding entails additional, tunable parameters (beam size, n-gram repetition penalties, etc.).
The expert annotators are instructed to use http:://dsssccccc.pppaallle..ood for searching through the C4
corpus.
17

---

Table 4: Relations of INVARIANTLAMA
Relation
Template ((X), (Y]
Example
P19
[X] was born in [Y]
Taras Kuzio was born in Halifax
P20
[X] died in [Y]
Georgios Roilos died in Athens.
P279
[X] is a subclass of [Y].
Hutterite German is a subclass of Bavarian
P37
The official language of [X] [Y].
The official language of Azad Kashmir English.
P449
[X] was originally aired on [Y].
Microsoap was originally aired on BBC.
P47
[X] shares border with [Y].
Illinois shares horder with Kentucky. Logan International Airport is named after Boston. The original language of The Fatal Eggs is Russian
P138
[X] named after [Y]
P364
The original language of [X] is [Y].
P527
[X] consists of [Y].
AIM alliance consists of Apple
P176
[X] is produced by [Y].
Alfa Romeo 155 is produced by Fiat
P27
[X] is [Y] citizen.
Woodrow Lloyd is Canada citizen.
P407
[X] was written in [Y].
France Culture was written in French.
P30
[X] is located in [Y]
Lavoisier Island is located in Antarctica
P178
[X] is developed by [Y].
Tizen is developed by Intel.
P1376
[X] is the capital of [Y].
London is the capital of England.
PI31
[X] is located in [Y].
Pershing County is located in Nevada.
P1412
[X] used to communicate in [Y].
Jacques Rivette used to communicate in French.
P17
[X] is located in [Y].
Eibenstock located Germany
P276
[X] is located [Y].
Delhi Technological University located India.
P937
[X] used work in [Y].
Pierre Trudeau used work in Ottawa
P140
[X] is affiliated with the [Y] religion
Emirate of Granada is affiliated with the Islam religion.
P103
The native language of [X] is (Y]
The native language of Anastasy Vonsyatsky is Russian.
P190
[X] and [Y] are twin cities.
Beijing and Milan are twin cities.
P1001
[X] is legal term in [Y].
Surgeon General is legal term in Canada..
P495
[X] was created in [Y].
La Grande Vadrouille was created in France.
P36
The capital [X] is [Y].
The capital of Granville County is Oxford. Grimaldi Group was founded Naples
P740
[X] was founded in [Y].
P361
[X] part of [Y].
Singa is part of Andes
Table 5: CKL benchmark dataset statistics
Avg. Input Token
Avg. Answer Token #
Dataset
Answer Types
Geographicul (54%), Language (14.9%), Nutionalities (7.2%)
INVARIANTLAMA
17474
11.9
1.3
Person (6.3%) Location (5.7%) Organization (5.3%) etc. (6.6%)
Person (61 .47%), Organization (8.3%) Geographical (6.6%)
UPDATEDLAMA
924
13.7
9.4
Numerals (9%), Date (2.4%)), etc. (16.04%)
Person (59.7%). Organization (10.2%) Numerals (7.6%)
NEWLAMA
797
14.7
8.7
Date (5.3%). Geographical (4.8%). (12.4%)
NEWLAMA
Person (48.5%), Organization (13%), Geographical (9.8%)
11177
44.4
6.1
EASY
Date (5.5%), Nationalities (3.4%), Numerals (2.5%) etc. (17.3%)
The newly constructed datasets UPDATEDIAAMA, NEWLAMA, and NEWLAMA-EASS require multi-token decoding which adds a level of difficulty for the task compared to INVARIANTLAME Moreover, WWW.AMA-EAAE has a different input distribution (longer input sequences) than the other datasets since the decontextualization and back-translation processes are applied to create each instance, which makes the sentences longer. Lastly, some examples of the CKL benchmark datasets provided in Table 6
18

---

Minn nonda Suprems Court slow Lake Calnon Diketerurs
caivu merte pooe accoev rig ncces oan cammns
Follow the instructions.
#1. SEI ECT one sandlan that most notches theie CEL TERLA CLICK correct enswer frone the
soot
sccedriss Nos ln coserc latu wit court witrr modt
recoens wern
de Oueroottn nar sttcc theer eccure comturt
ohtae yor crieety
superte compr proouree
1ihee du gi sev ca 1o X.s G ct
Tahes wai hesse Clamme to da
var bilk sata becia
and da wxtent
Four mory conner creaber metrrcccal
(4Ged par si ser (03) erosccaa leraraooeee
Pric the Cuait cualg diseet OINC""
cone offiert suderc intomnaii
maatc orteers, NOT, Plose EDIT ADD Hixea
Phe pocrnar tnu cou, recher rould
cucsióon the blank forrad.
SSAA
Press Towe (LAAN)
T0% Komming SKANN mapao
Figure 5: Mturk interface used for construction of UPDATEDLAMA and NEWLAMA
aapre Court reects peritiion from Louddiees pastor balng charges viobtting can bure
an optionn
piia ity
cometts COKCTA ALL
te annie mesato al parting ber plsc oeme ponen otfeccons
emyre sen qus taw berraa tmina nurto fuco rataai cuueciee servic
canichnoson
Connt aet lene semiu theiis beduring be ehe meno on oe beeeiin te Sovem unlonn aecoting coloertnn sa
Glickon www.ereeeeeetetrreooooooooo conweys WNNOORNNNNNATTTTTT of ariici
rilor fe peeeninge at fae semon cteiii
select poother sentenes
eeriing dlcuk Ael ( to liboy, meso asannd en cae busd veso ovrr tre makk aen Cou retud to soccutiee rok loc, Aecrosn catmno yrar and toaes ne
Marcment anower.
Figure 6: First mturk interface used for construction of NEWLAMA-ENN
Saprene Suprene alloo Lake Calbuu revert to Drtttaee
Origiiat oe dee dlans stateeed: tiis deto tin pegpondeve Prosuiel
(anr Eatani angoaal Datic mastaaaa ausedigg bnkc opprre Fresder:
Mirrusos vathon sourd irains (bell Cor masooe Lerowin loart
seous ponery pulic Yoans) Oor lie Sooms
change 2et
Kog nomerr Queero
mateemen ("uler waher Weter!
Seeece 400 new
wove eruters fon k0
Reaura 1S. cuenton becudde arruct adaitiona coa torn
(Gllre with
who pan deal T20 the corveding Moti sugurt dawyy desio tovmerty
sequreher to. the quatto anoo 5160 comsn SAAAeyy
Ja mulucc
Figure 7: Second mturk interface used for construction of NEWLAMA-EASY
19

---

Table 6: Examples of INVARIANTLAMA, UPDATEDLAMA, NEWLAMA, and NEWLAMA- EASY
Task
Input
Output
iPod Touch is produced by
Apple
INVARIANTLAMA
The Sharon Cuncta Show was created in The native language of Lee Chang-dong
Philippines
Korean
is the prime minister of England.
Theresa May-
Boris Johnson
has the most passing yards in the NFL.
Brady Quinn-
UPDATEDLAMA
Jalen Guyton
Bale has
champions league titles with
3-+4
Real Madrid.
Alicia Braga plays in the New Mutant. owns the rights the Falcon and the
Cecilia Reyes
NEWLAMA
Disney
Winter Soldier.
Tesla invested
in the digital currency bitcoin.
1.5 billion
The decision of the two volleyball stars Bria and Cimone Woodard to withdraw from the Power School to study
Howard University
NEWLAMA-EASY
at has become a national story.
Allen Lazard is officially listed as questionable with a nuclear injury after missing the last games.
six
C EXPERIMENTAL CONFIGURATION
Pretraining Congifuration We utilize the T5 initially pretrained on C4 (April 2019) and contin-
ually pretrained with salient span masking (Guu al. 2020) on Wikipedia (May 2020) initial- ization. We use the checkpoints from Wolf et al. (2020) We also perform the SSM objective during CKL because it shown to help LMs "focus problems that require world knowledge" (Gu
et al.
2020 Roberts et al. 2020).
Continual Pretraining Configurations The input and output sequence length is fixed to 350. We use gradient accumulation for cases where the same number of training batches could not be loaded on the GPUs due to the varying memory consumption required for different methods and set the global batch size 60. We use Adafactor optimizer with an initial learning rate of 1c-3. We show the effects of learning rate variation regarding the tradc-off between maintaining previous knowledge and acquiring new knowledge in Appendix E We use learning rate warm-up for the first 10% of training and linearly decay the learning rate to half of the initial learning rate towards the end of training. For all of the experiments, we use 32GB V100 GPUs for training with each method except Mix-Review, where we use 16 32GB V100 GPUs The details of the configurations used for evaluation on each individual CKL task are provided Appendix C
Evaluation Configurations For TS based models, all evaluation is done in a zero-shot manner and the input and output length is fixed as 25
is processed with single GPU. For INVARIANTLAME and 4 respectively. For UPDATEDLAMA and NEWLAMA the input and output length is 50 and 10 respectively. Lastly, the input and output length is 150 and 10 respectively for The rationale of this hyperparameter is based on average input and answer token in Table
Unlike T5 models, GPT-2 based models need additional light-tuning for epoch for evaluation. For INVARIANTLAMA, the input and output length is 50 and 3 respectively. The training batch size is 32 and the learning rate is 1e-3. For evaluation the acquisition of new knowledge, the input and output length is 100 and 10 respectively. The training batch size is 8 due to memory constraints and the learning rate is Ie-3. For both tuning processes, 4 V100 32GB GPUs are used. The detailed result and discussion of GPT-2 based models are shown in Appendix G
D HYPERPARAMETERS FOR IMPLEMENTATION OF CKL METHODS
RecAdam Chen et al. 2020) We use the same hyperparameter setting for the optimizer as in Chen ct al. (2020): we set the coefficient of the quadratic penalty y to 5,000, and select the best in and in 100, 250, 500, 1,000 and 0.05, 0.1, 0.2, 0.5, I respectively for the annealing coefficient 2 (1).
20

---

Table Result of TS- V3-Vanilla and T5- Kadapters continually pretrained with various learning rates. The exper- iments are done under the setting of SMALL scenario in Table 3 thus Do are C4 (April 2019) and Wikipedia (May 2020), and D1 is CC- CC-RECENTNEWS SENNNINNES--SLLLLL SMALL. Each of IL and NLE stands for INVARIANTLAMA and NEWLAMA-FASS The parameters of FUAR are T 7 and 71,, the tasks measuring the amount of time- invariant knowledge from corpus Da. updated knowledge from D1, and newly acquired knowledge from D1, respectively.
IL
NLE
FUAR (IIL) n...
Method
Learning Rate
EM
EM
In-iial
24.17
8.9
T5-Vanilla
lc-05
19.15
13.56
1.08
T5- 75-Vanilla
le-04
17.45
15.21
1.06
T5- Vanillal
5e-04
14.88
15.89
1,33
T5-Vani Vanilla
1e-03
11.19
18.77
1.32
T5- Kadapters (k=2)
le-04
19.93
14.93
0.70
T5- 15-Kadapters (k=2)
1e-03
16.46
19.59
0.72
Mix-Review (He et al. 2021) We use the English Wikipedia to represent the original pretraining corpus. The mix-decay and mix-ratio are set to 4 and 0.7, respectively, which is the best hyperpa- rameter setting in the paper.
LoRA (Hu et al. 2021) We only freeze the encoder for the encoder-decoder LM and the entire model for the decoder-only LMM We use the optimal rank of 4 and adapt both Wg and W, in the self-attention module, which corresponds to the best performing hyperparameter setting in the paper.
K-Adapter (Wang et al. 202 Similarly with T5-LoRA, we freeze the encoder for the encoder- decoder LM and the entire model for GPT-2. We implement = 2,3 for both T5 and GPT-2 to see the effect of increasing # of parameters. Unlike in the original paper, we set the configuration of the adapter identical to a single transformer layer from the original LM, ridding the need of an up-projection and down- wwwn-rroeectio layer.
Modular We use a projection layer before adding the hidden state outputs from both encoders to match the dimensions.
Why do we add parameters to only the encoder for T5? For marameter-expansion methods, we add parameters to only the encoder because the encoder is applied to the input sequence and the decoder is applied to the output sequence. Since most of the computational cost comes from the decoder computing for the output sequence in auto-regressive manner highlighted in (Li al., 2021]. the newly added parameters in the encoder are roughly expected to have minimal additional computational cost.
Why do we freeze parameters of only the encoder for T5? K-Adapter and LoRA are initially proposed freeze all of the parameters except for the newly added parameters. However, when applying this methodology to T5, it was empirically shown that unfreezing the parameters of the decoder results in better performances when utilized together with parameter-expansion methods in terms of overall trade-off.
E EXPLORING THE TRADE-OFF OF VARYING THE LEARNING RATE FOR CONTINUAL PRETRAINING
Table 7 shows that lowering the learning rate for the continual pretraining leads less forgetting of the original knowledge, but also less learning of new knowledge. The experiments are done under the setting of SMALL scenario in Table 3
By comparing the FUAR among the T5- Vanilla models with different learning rates, it can be seen that there is rule of thumb for choosing the appropriate learning rate since FUAR is the lowest
21

---

Table 8: Dev performance on KILT benchmark datasets after finetuning. Each model is finetuned on train sets of KILT after continually trained on CC- RECENTNEWS dataset for 4 epochs.
Fact Checking
Entity Linking
Slot-filling
Open Domain QA
Dialogue
Method
FEVER
AY2
WnWi
WnCw
T-REx
zsRE
NQ
HoPo
TQA
ELIS
WoW
ACC
ACC
ACC
ACC
ACC
ACC
EM
EM
EM
Rouge
FI
TS-Initial
80,39
81.44
50.47
48,92
44.64
4,40
25.63
17.64
28.38
13:46
13.92
IS-Vanilla
78.02
81.19
48.17
46.46
44.08
2.04
24.93
14.36
26.51
13.38
13.07
I5-RecAdam
77.83
81.44
49.12
47.01
43.04
2.58
24.65
14.86
25.99
13.71
12.69
15-MixReview
77.17
80.77
49.38
46.22
44.08
2.47
25.07
14.57
26.36
13.57
12.73
TS-LoRA
79.89
81.44
48.82
47.29
45.68
3.01
25.49
16.71
28.23
13.42
13.60
TS- 15-Kaapts (k=2)
80.35
80.94
48.91
46.65
45.52
3.33
26.20
26.89
TS Kadapters (k=3)
80.31
80.52
47.09
46.26
45.60
3.12
24.79
16.57
25.62
13.82
13.42
TS-Modular
80.54
82.44
48.44
44.81
48.16
3.44
24.51
18.43
28.31
13.72
14.03
Table 9: Hyperparameters and dataset details for all tasks of KILT.
Faet Checking
Eatity Linking
Slot-filling
Open Domain QA
Dialogue
FEV
AY2
WnWi
WnCw
T-REx
zsRE
NQ
HoPo
TQA
ELIS
WoW
Epoch
45
6
Input Seq
512
35
35
Ouiput Seq
6
350
LR
lo-4
le-4
le-3
lc-4
lo-3
lo-4
le-3
le-3
le-4
Batch Size
128
16
128
48
512
256
256
256
128
32
Train Size
104,966
18,395
2,284,168
147,909
87,372
88,869
61,844
272.634
63,734
Dev Size
10,444
4,784
3,396
5,599
5,000
3,724
2,837
5,600
5,359
1,507
3,054
in learning rate of le-4 and increases for both lower and higher learning rates. We suppose that the optimal learning rate heavily depends on the corpus size of Dj and the model capacity of LM. We also report the performance TS- "Sdaadaptes,, which is CKL method that shows robust performance throughout most experiments. Applying T5- T5-Kadapters consistently mitigates the tradc-off between forgetting and acquiring new knowledge as shown by the improvement in FUAR from the T5-Villlla model with the same learning rates, although the level of effectiveness varies according to the value of the learning rate. We do not perform extensive experiments with each of the varying learning rates since searching for the optimal learning rate for each different continued pretraining setting may be out-of-scope with this research.
F EXPLORING How CONTINUALLY PRETRAINING ON D1 AFFECTS KILT TASKS WHICH REQUIRES KNOWLEDGE FROM Do
In addition the CKL benchmark, we also show in Table g the performance on the dev set of KILT (Petroni et al. 2021) after finetuning each of the continually pretrained models of Table [ Since KILTT made from Wikipedia, which corresponds to the old pretraining corpus Do, the per- formance on KILT measures how continual pretraining new corpus D) affects the performance on the knowledge obtained from Do if finetuning is done on behalf of the knowledge from Do-
Configuration KILT (Petroni et al. 202 consists of 5 different tasks and 11 datasets: Open- Domain Question Answering (Joshi et al. 2017]] Kwiatkowski et al.) 2019: Fan et al.] 2019 Yang
al. 2018). Fact Checking (hhorne et al. 2018)). Entity Linking (Hoffart et al. 2011 Guo &
Barbosa 2018), Slot-filling (Levy et al.) (017). and Knowledgeable Open Dialogue (Dinan et al.) 2019). Because each task requires different training objective than the one used during pretraining, additional finetuning is necessary. We search for the hyperparameters such as training epochs, batch size, input size, output size, and learning rate of each individual KILT task to match the T5-base dev performance reported by Petroni et al. (022). Using the identified configurations, experiments on all of the KILT tasks with the continually pretrained models for each method as the initialization checkpoints. Evaluation metrics are different for each dataset: accuracy for discrete
perform
22

---

output (fact-checking, entity linking, slot-filling), Exact Match (EM) for question answering tasks with short output, ROUGE-L for ELI5 (question answering task with long output), and Fi-score for Wizard of Wikipedia (dialogue). The data statistics and the hyperparameters used for finetuning on each KILT dataset is reported in Table 9
Experimental Result We first focus on the performance on zero-shot Relation Extraction (zsRE), which is measured on the dev set of 12 relations that are ensured have no overlap with the 84 rela- tions of the train set (Levy et al. 2017). Since the setting is similar to the zero-shot probing setting of II, the trend of the result on the two datasets are similar. The performance of T5-Vaailla drops to half from that of Initial as shown in II., and the best performing method for both datasets is TS- Modular. In addition, corresponding with results from the CKL benchmark, caramettr--xpansion methods generally show stronger performance than the other methods.
However, for the other datasets that cannot be performed in a zero-shot manner, the intermediate process of continually pretraining corpus D1 does not seem be that harmful on the finetuning for the target tasks even though they are more related to the knowledge of Do. Even T5- Vanilla shows modest performance, sometimes with better results than some other CKL baselines. One hypothesis is that the models could have regained the original knowledge from corpus Do through the finetuning process. Also, some of the knowledge could have been recovered through the test- train overlap (Lewis et al. 2020b Wang et al. 2021a).
A more surprising finding is that the performance of some of the parameeter-xxpanioon methods are even higher than that of In-Inaial, which is considered to be the upper bound for KILT because T5- Initial is only trained on behalf of the knowledge from Do. For example, M5-Modular shows higher scores than Initnal on 6 out of 11 tasks. Since the cettmntter-eppasiiine methods force the model to store the new knowledge the newly added parameters during continual pretraining, one careful conjecture is these LMs have learned to combine and utilize in its internal representation of both old and new knowledge stored in separate parameters during finetuning to maximize the performance.
G EXPLORING How CKL METHODS TRANSFER ACROSS LM ARCHITECTURES
We perform experiments with GPT-2 Large (~ 774M params) (Radford et al. 2019 initially pre- trained on WebText and Wikipedia [4 (Do) and continually trained on WEE-RECENNNESSSSM... i.., SMALL (D1) for 8 epochs. For continued pretraining, we use the common teacher-foreing pre- training objective. The initial learning rate for the continued pretraining stage is empirically chosen as Ic-4 (results with learning rate as 1e-3 are shown in Appendix G.I). After continued pretraining, small
apply light-tuning, process denoted for finetuning the model for only one epoch portion of data similar to the evaluation set. Training on a single epoch constrains the model to barely adapt the input-output form of the data and not to learn the knowledge in tuning samples, mitigating the problem suggested by Lewis et al. (2020b).
To measure the time-invariant knowledge, we use LivariantLAMA. (IL) because most of the slots to fill are at the end of the sentence. For light-tuning on behalf of IL, we use additional T-Rex data from Shin et al. (2200) which has a similar distribution as instances from IL. Among them, 5,000 instances with the same fime-invane invarian relations as IL are randomly sampled for light-tuning. On the other hand, unlike where most of the slots fill are at the end of the sentences, the LAMA datasets for new knowledge in our CKL benchmark mostly have the slots at the beginning of the sentences. Therefore, we use the corresponding CBQA dataset of NEWLAMA-EASE EASY (NQE) to roughly measure the new knowledge [S] For light-tuning on behalf of NQE, 5,000 instances are sampled from a set of QA pairs constructed from CC-RECENTNEWS but not CC- RECENINEWS-SAN -MALL to remove the test-train overlap.
NEW QUESTIONNN
14GPT-2 was initially pretrained on WebText (Dec 2019), which consists of 8 million documents with Wikipedia pages excluded. In order measure the performance INVARIANTI.AMA constructed from Wikipedia, we continually pretrain GPT-2 on a subset of Wikipedia (May 2020) for 14k global training steps before CKL.
The QA version of UT., NI. and NLE will he also released with the main CKL. henchmark.
23

---

Table 10: Performance of decoder-only models initially pretrained on Dec 2019 dump Webtext and May 2020 dump of Wikipedia (Dg) continually pretrained on CC-RECENTNEWS SMALL (D1) for 8 epochs with a learning rate of le-4. Each of IL and NQE stands for INVARIANTLAMA and NEW NEWQUESTIONS QUESTIONS-EAAY. The parameters of FUAR are T TV and T7. the tasks measuring the amount of time-invariant knowledge from corpus Do. updated knowledge from D1, and newly aequired knowledge from D1, respectively.
IL
NQE
FUAR
Method
(IL), n.d.
EM
EM
GPT2-Initial
38.11
GPT2-Vanin Vanilla
35.88
5.79
1,58
GPT2-Recadam
35.50
5.79
1.84
GPr2-Mixreview
38.93
5.57
GPT2-Lora
37.99
6.23
0.06
SPTT2-Kaapters (k=2)
37.85
6,34
0.13
GPT2-Kadapters (k=3)
38.03
5.79
0.06
Table I0 shows the CKL benchmark performance of GPT-2 models. We report the results aver- parameter-expansion methods show
aged 5 runs with different random seeds. As in Table 2 robust performance on both II and NQE, resulting in low FUAR. This shows that these methods are not only effective on the encoder-decoder model but also the decoder-only model as well. One interesting result in Table 10 is that GPTe-MixReview performs the best on IL, with performance even higher than the initial model, which results in the best FUAR of O which means no forgetting occurred at all. We suppose that the training strategy of GPT2-MixReviees allowing access to sam- ples of Dn during continued pretraining, would have allowed fast adaptation knowledge from Do during the light-tuning phase. Performance of GPee-MixReview suggests that it makes it possible to regain the original knowledge for decoder-only models even with small tuning steps.
We want to highlight that the discrepancy of the performances among the CKL methods between encoder-decoder LM (T5) and decoder-only LM (GPT-2) may not solely be on the LM architecture, but also the learning rate and the evaluation method (light-tuning was used to evaluate GPT- 2 while we evaluated T5 in a zero-shot manner). We leave further exploration of training ever- changing decoder-only LMs such as GPT-2 future work.
G.1 FAILED GPT-2 EXPERIMENTS WITH LARGER LEARNING RATE
Table shows the CKL benchmark result of GPT-2 models continually pretrained on CC- RECENTNEWS- for 8 epochs with learning rate of le-3. By comparing the results in this table with those in Table 10 which is for models continually pretrained with a learning rate of le-4, the results in Table 11 shows worse performance on both IL and NQE. Unlike in Appendix F in- creasing the learning rate does not result in better learning of new knowledge. Instead, NQE perfor- mance is even worse than Initial for GPT2- Vanilla, GPT2-Recadam and Mexeei.. FUAR is no gain for these cases by the definition of the metric because the denominator has the value of zero. This shows that large learning rate for continual pretraining may lead to failure: neither retaining old knowledge nor acquiring new knowledge effectively. For parameter-expange methods, because many parameters including the decoder are frozen during the continual training process, they seem to be less prone to the effect of a large learning rate.
H EXPLORING THE PREDICTION CHANGE DURING CONTINUAL PRETRAINING
Table 12 shows the prediction results of T5-Vanilla and T5-Modular on three knowledge probing tasks: INVARIANTLAMA UPDATEDLAMA, and NEWLAMA. We show the prediction for every training epoch for each model. The instances selected from the predictions that T5-Modular got correct but In-Inal got wrong on the final prediction, in order to see where the gap of the EM comes from.
24

---

Table 11: Performance of decoder-only models initially pretrained on Dec 2019 dump Webtext and May 2020 dump of Wikipedia (Dg) continually pretrained on CC-RECENTNEWS-SSE learning rate of 1e-3. These are the results failed due to a large learning rate. Each of IL and NQE stands for INVARIANTLAMM and JAAQ
(D1) for 8 epochs with a
IL
NQE
FUAR ((IL), n...n.d..Q....
Method
EM
EM
GPT2-Initial
38.11
4.37
UPT2-Vanilly
23.03
1.64
no gain
GPT2-Recadam
25.38
2.73
no gain
HPT2-MMxreview
32.07
1.64
no gain
GPT2-Lora
34.52
5,46
3.29
Kanzpteassss (k=2)
33.67
6.01
2.71
GPT2- Kadaptess (k=3)
31.75
7.65
1.94
Table 12: Change of Prediction Outputs During Continued Pretraininig
Close Seatence
Model
Epoch 2
Epoch 3
Epoch
Answer
Yvonse Mothaur
The rativee languagg
V
French Franch
Franch French
Khmer Fresch
Malaya French
French
Sunic Drih 2 developed by
V
Sunic
Sonic
Sonic the
Sepi
Smic Found
Saya
Siric the
Saya
Wchk's developpil by
Microsoft Apple
Google Apple
Grtliab Apple
Google Apple
Aypl:
official language Republic of Ingudetiia
Russia
English Rusiun
Kuzath
English Rusian
Rusian
Ruscia
capital Komtan limpire
Rome
Rome
Rome
Consuntin
Coustantino
Rome
higgest exporte china
Anblaa
Arabia
Arabia
Sandi Arzhia
Sandi
Russis
Sandi Anihaa
Kusia
Russia
Rassss
Drught
Yves Le
Yves Dujamic
Murio Dtaghi
Mario Draghi
the euro zone central bank
M
Mario Draghi
Chriszne Lagarde
Christine Lagarde
Christine Lagande
Chrssine Lagande
mansgee
Mauricio Femundez
Steve Bease
Frakk Lampard
Mikel Artetu
Earique Frank Langoed
chelera in the promice leagn
M
Iow: Mourinhh
Jos: Mourisho
Frank Lampani
Frank I Lamprd
the price for a daat in nottanhhm
M
What This
30,000 pounds
939
40,000 pounds
21.25m
$1.25m
40,000
36,000 40,000
vow governor New York
Androw M. Cnome
Cuome
Androw Canmn
Prankin D. Roowssclt
Martin Van Hurn Andrew Cuomo
the time this article was weitten
M
Andrew Cuemo
Andrew Cuomo
Andrew M. Cuomo
Andrew Coome
on the Bills all pro team
M
V
Corey Williams
Williams
Williams
Willias
Cuscy
Willians
Comue
Williams
the founder of the popalber
V
Swoohi Nokameto
Sutoshi Nikametn
Yuri
Xiubo
exptoccnmency bitosin
M
Vitaik Buterin
Sunosli Nikamoto
Satnhi Nakamoto
Suooii Nakamoto
Suoshi Nakamoto
NL
The buil for kyle riltenhone is
M
Rs. com
52 millinn
whrpping SI million
$2 million
$t million
$2 million
Si million
$2 million
S2 million
The las vegas naidees beat
playolls
V
the Las Vegas Raiders the New Orleans Sants the Las Vegas Rasdees
New Odlens Saints Kassas City Chisss Kunsa City Clies'
New Otlenns Stints
the socramento
the New Orleane Saints
the hrot of ellen de genznes show
V
Yves Elis:
samantha s Elka DeCleranes
Noma Ellen deflenes
Elka DeGesenee
Mike
Ellen DeComores
I EXPLORING THE CAUSE OF THE EM GAP BETWEEN UPDATEDLAMA AND NEWLAMA
As shown in the main experiment, Table 2 there is a considerable gap between the EM of UP- DATEDLAMA (UL) and NEWLAMA (NL) over all the methods, despite undergoing the same data construction process. We attempt to analyze the causation by first analyzing what answer types make up the EM score of both UL and NL of Vanilla,.. which are 10.17 and 3.77, respectively. As shown in Figure R the cloze sentences that take Person type as the ground truth makes up most of the EM of both tasks, despite Person type answers taking up similar proportion out of the total answer types (61.46 for UL and 59.7% for NL). Since UL consists of probes requiring an update of information from Do one might conjecture that the EM gap is simply due the difference of the
25

---

Penson
Orgunization
Numenal
UpdanddAAMA
NewLAMA
Geographcal
34
12
18
TM
(Overlapping Answers)
UpdatedLAMA
NewLAMA
EM measured using the in- stances from UL and NL with overlapping Person type answers
(a) Composition of ground truth categories of the correctly predicted instances
Figure 8: Analyzing the cause of EM gap between UPDATEDLAMA and NEWLAMA.
Table 13: F1 Score of Main Results.
IL
UL
NL
NLE
FUAR
Method
EM
EM
EM
NL,)
T5-Initial
24.88
2.62
3.19
14.49
TS-Vanilla
13.11
11.89
5.84
22.53
0.68
T5-RecAdam
13.39
14.33
6.15
22.68
0.57
TS-Mixeeview
14.09
8.11
4.80
18.89
1.10
T5 Lo-L
17.04
14.50
7.45
24.59
0.36
T5- Kadapptett (k=2)
19.88
13.67
7.43
24.04
0.22
T5- 15-Kaaapsers (k=3)
19.91
14.31
6.55
23.33
0.21
TS-Modular
21.35
12.78
24.42
frequency in each corpus of the entities that serve the ground truths, e.g., those entities appear more in corpus Do than in Dj. In order to get rid of the influence of frequency entities when analyzing the source of the EM gap, find overlapping Person type answers from UL and NL, and analyze only the 67 probing sentences for both datasets each paired to one of these entities. As shown in Figure 88, the EM UL is still much higher than that of NL. Manually analyzing these instances, we find that the probing sentences for NL ask for relatively more fine- grained knowledge overlapped cloze sentences with differ-
compared to UL., since the instances of UL by definition ent answers the corpus Do and D1, that naturally make them be coarse coarse-grained. For instance, the probing sentences for entity "Tim Walz" in UL and NL are this year." and tively. We thus conjecture that the main causation of the EM gap be UL consisting of instances requiring coarse- grainedd knowledge, which is likely to have appeared more during in DJ, while NL consisting of instances requiring finennrained knowledge, which expected to likely have appeared less in Dj.
the governor Minnesota is the governor of Minnesota calling for the evacuation of St. Paul.", respec-
J ADDITIONAL ANALYSIS OF MAIN RESULTS
26

---

In-iiaaaal TS-LoRA
TS-Haseline T5-Kudupters (k-2)
Re-RecAdam T5-Kadupters(k-3)
MivMixeeview
T5-Modular
(a) INVARIANTLAMA
(b) UPDATEDLAMA
(c) NEWLAMA
(d) NEWLAMA-EASY
Figure 9: Mean P@k curve for CKL benchmark with varying k.
27


Reviews:
[{"review_id": "w7hfY-NPgJN", "reviewer": "Reviewer_ELX1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed continual knowledge learning problem is quite interesting and important.\n2. The benchmark is useful and the proposed FUAR metric is technically sound. \n\nWeaknesses:\n1. The paper only performs experiments with an encoder-decoder model (T5). The experimental results will be more convincing if more pre-trained language models (such as GPT) are included. For example, we can explore the ability of different PLMs to avoid catastrophic forgetting and to acquire new knowledge while preserving invariant knowledge.\n2. Consisting with the traditional setting of CL, the paper also creates a setting for multiple CKL phases. However, only a two-phase setting is considered. More experiments can be explored, such as five-phase or controlling the differences in the distribution of data at different phases.\n3. The experimental findings in this paper are somewhat trivial.\n", "summary_of_the_paper": "The paper studies continual knowledge learning of language models, which is an interesting and important problem. Particularly, a new benchmark and a metric are introduced to quantify the retention of time-invariant world knowledge, the update of outdated knowledge, and the acquisition of new knowledge. To establish baselines for the CKL benchmark and validate the rationality of the proposed benchmark and metric, the author conducts extensive experiments with a pre-trained encoder-decoder model (T5) based on various training methodologies including regularization, rehearsal, and parameter expansion methods.\n\nThe paper is well organized and easy to follow. The proposed continual knowledge learning problem is quite interesting and important. The FUAR metric is also technically sound. The authors also conduct comprehensive experiments to verify the rationality of the proposed benchmark under the various settings.", "summary_of_the_review": "The paper studies an interesting problem. The proposed benchmark and metric are technically sound. However, there are some concerns about experimental settings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "exK1biKTlg2", "reviewer": "Reviewer_U9Hk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- The problem of CLK itself is an important task and more realistic to downstream knowledge-intensive applications. \n- I like the clear separation of the types of knowledge: time-invariant (to keep), outdated (to remove), new (to inject), as well as their collected datasets for reflecting the three types of knowledge update. \n- Different types of baseline methods are covered and compared with analysis. \n\nWeakness:\n- The formulation of the CKL problem is overly simplified. It basically only considers a single corpus (D_1) for updating the knowledge of previously learned LMs. A general setup should consider a streaming version of D_1 and make it a sequence of sub-corpus (D_1, D_2, ..., D_T) that arrive at different time steps. Also, the associated tasks in UpdatedLAMA and NewLAMA should reflect such a time series --- that is, the streaming version of the current probing tasks. The current formulation described in Section 3.1 only has a single time step. It's more like an offline learning problem with the forgetting constraints but not an (online) continual learning problem. The proposed setup is thus a bit far from the motivation of studying CKL --- maintaining an ever-changing LM. \n- The experiments are very limited to the LAMA probing which does not necessarily connect to real downstream applications of these LMs. It's also hard to justify whether such methods can maintain performance in general NLP tasks. The argument about the KILT experiments seems to only focus on testing the retention but not about the new/updated knowledge about D_1. \n- The analysis of the CKL methods is not deep enough. How do these methods work and why do some outperform others? How do we know if an arbitrary new fact conflict with the existing knowledge or not on the fly? How do you define such conflicts properly? Say you have a sentence in D_0 \"Cristiano Ronaldo plays for XXX in 2010\", and you have another sentence in D_1 \"Cristiano Ronaldo plays for YYY now.\" In the current problem setup, how will such \"conflict\" be defined? \n- The new findings are not particularly non-trivial. \n- The presentation and the writing of the paper can be further improved with more illustrative examples and case studies for readers to qualitatively see the problem setup and the differences between these methods.", "summary_of_the_paper": "This paper presents a new continual learning problem setup: continual knowledge learning (CKL) and constructs an associated benchmark resource. The benchmark is based on slot filling-based knowledge probing tasks (i.e., the LAMA analysis). The authors show the empirical performance of some existing CL methods, ranging from regularization, rehearsal, and parameter expansion. And they show a few findings based on their experimental results, e.g., learning rate can be sensitive to balance the tradeoff between forgetting and learning new knowledge, and CKL methods might have transferrable performance across different LMs. (e.g., T5 and GPT). ", "summary_of_the_review": "This paper is a pilot study of an improtant problem (continual knowledge learning of LMs), but the problem formulation is overly simple and there are still many important yet missing points in both data construction and experiments. Also, there are no much insightful and non-trivial findings with deep analysis of existing methods. Please find more details above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QWxMzVXvBSB", "reviewer": "Reviewer_Rjyt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nIn terms of novelty, the paper extended the definition of Continuous Learning to formulate Continuous Knowledge Learning that has unique challenges compared to the traditional CL. The paper also introduced a novel metric named FUAR to measure the trade-off between knowledge forgetting, update, or acquisition. This is a contribution to the field as this quantitative metric could facilitate direct comparison between models performing CKL tasks.\n\nThe paper is technically sound. Extensive experiments were conducted to benchmark the performance of various CL approaches (regularization, rehearsal, and parameter-expansion methods) on different aspects of the CKL task (retention of time-invariant knowledge, updating old knowledge, and acquiring new knowledge). In the Appendix, the authors also presented various ways to understand the model's learning process, such as the change of predicted outputs during the continued pretraining, as well as the failure analysis based on the type of probes. These methods provide more insight into the model's learning process that went beyond the plain performance scores. \n\nWeaknesses:\nUsing T5, the authors showed that parameter-expansion methods have the most robust performance throughout all of the experimental settings. However, in the experiments with GPT-2 (a decoder-only model) in the Appendix, GPT2-MixReview (a rehearsal method) performs the best. Although the authors mentioned that “We leave more exploration of applying CKL methods on decoder-only models such as GPT-2 architecture as future work”, they should still mention this critical discrepancy in the main body of the paper so that the readers are aware of the context of the findings. After all, large language models take various forms and both T5 and GPT-2 are examples of large language models.\n\nQuestions:\nThe authors demonstrated that parameter-expansion methods have the most robust performance throughout all of the experimental settings with T-5. Since the three methods (regularization, rehearsal, and parameter expansion) are not mutually exclusive, I was wondering if the authors have tried any combination of the approaches. For instance, the rehearsal approach could be combined with the parameter-expansion method. I was wondering whether a combined approach would yield even higher performance. \n", "summary_of_the_paper": "The authors formulate a new continual learning (CL) problem called Continual Knowledge Learning (CKL). Particularly, they distinguished three sub-tasks in CKL, i.e., the retention of time-invariant world knowledge, the update of old knowledge, and the acquisition of new knowledge. They also introduce a new benchmark and metric to quantify the performance of various state-of-the-art models on these sub-tasks. They find that CKL demonstrates unique challenges that are not present in previous CL setups. Critical causes of knowledge forgetting in CKL are also discussed.", "summary_of_the_review": "The paper formulated the problem of Continual Knowledge Learning and benchmarked the performance of large language models on this task with different CL methods. The tradeoff between forgetting existing knowledge and updating old knowledge/acquiring new knowledge is quantified through a new metric, which would serve as an important optimization goal for future research. This work is a big contribution to the community and would invite more research into this topic. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ot-6NNXeVf", "reviewer": "Reviewer_Hkbx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n- They provide a new benchmark and metric to measure the retention of time-invariant knowledge, updated knowledge, and new knowledge.\n- Some interesting observations are provided, for example, 1) rehearsal methods do not work well in this setting (even though the reason is quite obvious because some knowledge is updated) and parameter-expansion methods achieve better results, 2) LMs are prone to more forgetting as they go through multiple CKL phases, 3)  LMs should be pretrained with just a few epochs on less duplicating data for efficiency.\n\nWeakness\n- In a real-world scenario, how can one know in advance that the new task is truly \"new\"? As mentioned on page 5, it is possible that the FUAR score is very large if \"no gain\" and \"preserve knowledge\". The authors did not show experiments or analysis in such a setting, where the new task has some \"knowledge overlapping\" with the learned tasks. To make this concern more general, I feel the measurement of \"task similarity\" is missing in this work.\n\n- Can we also see perplexity as another dimension? EM scores cannot know \"how bad\" the prediction distribution is.\n\n- Do you think the conclusion of these LAMA tasks is the same as other NLP downstream tasks? \n\n- Do you think others can easily replicate your numbers? Do you run several splitting or seeds for multiple round CL settings?\n\n- Have you tried different sizes of T5 models? Maybe the GAP between vanilla and CKL methods will be smaller given larger models.\n\nMisc\n- I have doubt on this sentence \"Moreover, the effectiveness of CKL methods is much reduced in multi-phase CKL, shown by the decrease of the gap between the FUAR of T5-Vanilla and the best performing CKL method in the scenario of two-phase and one phase, which is 0.92 and 0.46, respectively.\" Isn't it only prove that T5-Vanilla can learn better FUAR scores SMALL-P1\u0019-->  SMALL-P2 than SMALL-P1\u0019+  SMALL-P2 because you have similar FUAR scores for T5-Kadapters? ", "summary_of_the_paper": "- The paper is about continuous learning for language models. The authors leverage existing LAMA tasks and collect a new test benchmark with updated information and new information. \n- They investigate several existing CL algorithms, and they propose a new metric called FUAR to measure trade-off between forgotten time-invariant knowledge and updated or newly acquired knowledge.\n- They provided some findings on their continuous LM learning. ", "summary_of_the_review": "This work is quite insightful for us to understand more about how LM continuous learning works, although I think more experiments could be beneficial as I mentioned in the weakness section. If we can make sure the numbers from this paper are reproducible and comparable, then I think it could be a good testbed for future research in this direction.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces the problem of continual knowledge (language) learning. The authors point out the interesting duality between continual learning and knowledge learning where: in knowledge learning one must avoid forgetting time-invariant knowledge (avoid forgetting in CL), be able to acquire new knowledge (learn new tasks in CL), and replace outdated knowledge (a form of forgetting and re-learning or adaptation). In their paper, the authors develop an initial benchmark for the task along with a set of baselines and provide empirical studies.

The initial reviews were quite mixed. The reviewers seem to agree this work studies an interesting and fairly novel direction for continual learning of language. However, the reviewers did not agree on whether this initial stab at the problem was "enough." In particular, reviewer U9Hk argues that the formulation is "oversimplified" and the current experiments are limiting.

After the discussion, the reviewers remained split with one high score (8), two borderline accepts (3), and one reject. So three reviewers believe that this manuscript is already a good contribution. The fourth reviewer disagrees, but the authors provided clear and convincing responses to many of their comments (and point to results already available in the appendix).

Overall, this is a clear and reasonable first step considering this paper proposes a new CL problem. The reviewers and I believe that this is interesting and rigorous enough to be impactful and to warrant follow-up works. As a result, I'm happy to recommend acceptance. I imagine that if the community demonstrates interest in this line of work, there will be work both on methodologies to improve the proposed baselines, but also work proposing extensions to the problem in line with some of the comments of reviewer U9Hk.

In preparing their camera-ready version I strongly encourage the authors to take into account the suggestions of the reviewers and your replies. In particular, your discussion regarding encoder-decoder and decoder-only LMs and the associated results would be good to discuss in the main text (even if the full results are in the appendix).
Decision: Accept (Poster)