Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompts
Authors: Gangwei Jiang
Caigao JIANG
Siqiao Xue
James Y. Zhang
JUN ZHOU
Defu Lian
Ying Wei
Venue: EMNLP 2023 Findings (2023)
Keywords: Continual learning; Pre-trained language model; Prompt learning
TL;DR: This paper introduces HPrompt-CPT, a novel prompt-guided continual pre-training method towards anytime fine-tuning, which enables better performance when fine-tuned on seen and unseen domains.

Abstract: Continual pre-training has been urgent for adapting a pre-trained model to a multitude of domains and tasks in the fast-evolving world. In practice, a continually pre-trained model is expected to demonstrate not only greater capacity when fine-tuned on pre-trained domains but also a non-decreasing performance on unseen ones. In this work, we first investigate such anytime fine-tuning effectiveness of existing continual pre-training approaches, concluding with unanimously decreased performance on unseen domains. To this end, we propose a prompt-guided continual pre-training method, where we train a hypernetwork to generate domain-specific prompts by both agreement and disagreement losses. The agreement loss maximally preserves the generalization of a pre-trained model to new domains, and the disagreement one guards the exclusiveness of the generated hidden states for each domain. Remarkably, prompts by the hypernetwork alleviate the domain identity when fine-tuning and promote knowledge transfer across domains. Our method achieved improvements of 3.57\% and 3.4\% on two real-world datasets (including domain shift and temporal shift), respectively, demonstrating its efficacy.

Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Prompts
Gangwei Jiang', Caigao Jiang', Siqiao Xue², James Y. Zhang",
Jun Zhou?, Defu Lian', Ying iei
'University of Science and Technology of China, Ant Group
"Nanyang Technological University gwjiangemail ust.: wstc. edu. caigao. siqiao. xsq, james. zéantgroup com, edu. ying. wei@ntu. edu. sg
jun. wunnzhoojjunantgrouu com,
Abstract
Continual Pre-training
Anytime Fine tuning
Unlabeled
Language
Downstream tasks
Exaluation
Continual wre-training has been urgent for adapting a pre-trained model multitude of domains and tasks in the fast-evolving world. In practice, a continually pre-trained model is expected demonstrate not only greater capac- ity when fine-tuned on pre-trained domains but also a non-decre performance on unseen ones. In this work, we first investigate such anytime fine-tuning effectiveness of existing continual pre-trat ree-traning approaches, concluding with unanimously decreased performance unscen domains. To this end, we propose prompt-guided continual pre-training method, where we train hypernetwork to generate domain-specific prompts by both agreement and disagreement losses. The agreement loss maximally preserves the generalization of pre-trained model to new domains, and the dis- agreement one guards the exclusiveness of the generated hidden states for each domain. Re- markably, prompts by the hypernetwork allevi- ate the domain identity when fine-tuning and promote knowledge transfer across domains. Our method achieved improvements 3.57% and 3,4% two real-world datasets (including domain shift and temporal shift), respectively, demonstrating its efficacy.
BEB
Adoptability Generattation Forgetting
Aug
G
3
Accuracy
Accuracy
Figure I: Illustration continual pre-training and the evaluation protocol of anytime fine-tuning, in which a in the accuracy table denotes the fine-tuned accuracy of the LM at any i-th stage, ie.. B on the j-th pre-trained (blue), current (red), and unseen domains (orange).
Continual pre-training methods (Jin et al., 2022; Ke et al., 2023) have recently emerged to address it by continually adapting LM to sequence of do- mains (e.g.. domains in Fig. 1). Two major lines of existing approaches, including knowledge distil- lation (Jin al., 2022) and parameter isolation (Ke et al., 2023, 2022a), make strides toward (1) maxi- mizing the adaptability, i.e., the performance of an LM (e.g.. B² in Fig. 1) when fine-tuning it onto the domain where it is pre-trained (e.g.. D2 in Fig. 1), and (2) avoiding catastrophii (CF), which is measured by the fine-tuned perfor- mance of an LM (c.g.. B² in Fig. 1) on the already pre-trained domains (e.g.. Dj in Fig. l).
1 Introduction
Pre-trained language models (LMs), such GPT- 3 (Brown al., 2020) and BERT (Devlin al., 2019a), have revolutionized wide spectrum of downstream natural language processing (NLP) tasks. Being initially pre-trained on a vast unla- beled corpus (e.g.. Co in Fig. 1), unfortunately, they struggle to keep up to date with language evolution (e.g., emerging internet slang, expanded meaning "Omicron") ") and domain shift (e.g., electronic health records for medical diagnosis).
Beyond the above two criteria, in practice, a con- tinually pre- pre-tee LM is also anticipated to offer non-decreasing generalization capability unseen domains. As illustrated in Fig. 1, it is likely that the unlabeled corpus for the domain of interest (e.g.. electronic health records as Dr) remains inaccessi- ble to an LM (e.g., B²) beforehand, while this LM should be superior or at least on par with its pre- ceding models (e.g., B!) on the T-th domain. On
This work was done when the author Gangwei Jiang was at Ant Group intern. "Coresponding author.

---

[o] Full model
(b) Bottom layers
16) Top layers
(6) Prompt
(c) Acapter
(O NCL
Igl DAS
(ij Gurs
Downstream task
Condinual tralning
Figure 2: Evaluation of separate and continual pre- rre-training methods under anytime fine-tuning, where modify each value aं by subtracting aj as the fine-tuned accuracy of the initial LM B (a)-(e) show the accuracy tables by pre-training each domain separately w.E. different of parameters (e.g., top layers); (f-(h) are by the naively continual pre-training method (NCL), DAS (Ke et al., 2023), and ours. Detailed settings are available in Sec. 5.2.
this account, we propose the comprehensive eval- uation protocol named anytime fine-tuning that subsumes all the three aspects, where a continually pre-trained LM can be fine-tuned and evaluated on either previously pre-trained, current, or unseen domains. The effectiveness of current methods in terms of anytime fine-tuning remains largely un- clear.
as fully adapt to the current domain. An added benefit of hypernetwork prompts that they elimi- nate the reliance on the domain identity to pinpoint prompts when fine-tuning. Second, we maximally preserve the generalization while mitigating CF of continually pre-trained LM via the agreement and disagreement losses. We prompt the previous and current LM with random prompt that sim- ulates generic or learned domains and introduce the agreement loss to enforce consistency between their predictions to avoid forgetting while preserv- ing model plasticity on other prompts. On the other hand, the disagreement loss promotes the exclu- siveness of generated hidden states for the current domain, thus minimizing interference to the estab- lished knowledge and encouraging generalization during fine-tuning through diverse domain knowl- edge. Noteworthy, the hypemnttwrrk also favors knowledge generalization, compared disparate prompts of different domains.
In this paper, we first conduct an empirical in- vestigation of existing pre- pre-training approaches un- der anytime fine-tuning (see Fig. 2) and identify the following prominent unresolved research questions. (1) Parameter-efficient pre-training, such as training adapters (Ke et al., 2021b) and prompts (Razdaibiedina et al., 2023; Smith et al., 2023) only for each individual domain, does not even contribute greater adaptability than that be- fore pre-training (i.e., evidenced in negative di- agonal values of Fig. 2(d)(e)). Likewise, pre- training parts of parameters for each domain, may also diminish adaptability, through comparison of Fig. (g)) with (a). (2) Continual pre-training is likely at the cost of sacrificing generalization to unseen domains, shown by large negative values in the third column of Fig. ((f)
Main Findings and Contributions. (1) We es- tablish a continual pre-training evaluation protocol, called anytime fne-tuning, and empirically verify that existing parameter-efffiient approaches lose their competitive edge in adaptability and almost all methods are at risk of impairing generalization to unseen domains (see Fig. 2). (2) We further conquer the two challenges by proposing a hy- pernetwork prompt guided continual pre-training (HPrompt-CPT) scheme where train the hyper- network with both the agreement and disagreement losses. HPrompt-CPT is effective, achieving the state-of-tth-aar on two real-world datasets.
To address the above issues, we propose Hypernetwork Prompt guided Continual Pre- Training method (namely HPrompt-CPT') that strikes balance between forgetting, adaptability, and generalization. First, inspired by recent suc- cess of prompt engineering paired with full fine- tuning in domain adaptation (Radford et al., 2019; Brown et al., 2020), introduce the hnet- mnem-prompt module consisting of hypernetwork to automat- ically generate comain-specifie prompts without handcrafted engineering. Different from parameter- efficient pre-training that train prompts only, we optimize both the hypernetwork and the full LM so
2 Related Work
Continual Learning (CL) focuses on the problem of sequential learning from stream of data that comes in different distributions. a great success in computer vision (Wang et al., 2022a,c; Smith et al., 2023), natural language pro-
has achieve
'The code of HPrompt-CES CPP will he released at

---

cessing (Sun et al.. 2019; Ke al., 2023), and data mining (Hao al., 2023; Xue et al., 2023). this paper, we focus on one of the important aspects, continual pre-training and present recent progresses below. More related works are given in Appendix A.
masked language modeling (MLM) (Devlin et al.. 2019a), and the objective is mmim (x. W) Excm(x) logp (& Mym(x):W) where W de- notes the parameters of language model B, m(x) and [m(x) the masked words from x and the re- main words, respectively. The conditional proba- bility is calculated by prediction layer ymim (Ø Xm(()) W) 9mdm (Bw(XXmmxx)))
Continual Pre-training. Previous studies (Guru- rangan et al., 2020; Dery et al., 2022) have demon- strated that the fine-tuned performance of LMM on downstream tasks can be enhanced by continued training on a domain-relatie corpus. Recent works take this concept further by introducing Continual Pre-training (CPT), where LM continually learns from streaming domain corpora. Jín et al. (2022); Jang et al. (2022) investigate conventional CL meth- ods in CPT using real-world datasets and highlight the final LM can be fine-tuned serve any task in pre-trained domains, leading to improved per- formance, while (Hu et al., 2022a) finds CPT is comparable with joint pre-training. To improve upon this, ELLE (Qin et al.. 2022) progressively expands LMs with preetivipr initializa- tion to inject knowledge from new corpus, while CPT (Ke et al., 2022a) designs specific adapters and utilizes hard-masking avoid CF. Addition- ally. DGA (Ke et al., 2022b) and DAS (Ke al.. 2023) adopt soft-masking to directly controls the update of the entire LM and contrast the previous and current representations.
After pre-training, the model is fine-tuned us- ing a smaller dataset specific to a downstream task, which enables it to learn the intricacies and de- tails of the task. In our study, the downstream task contains labeled samples (x, (e.g., in a hash- tag prediction task, x is the user's twitter and y is the selected hashtag). Its objective function is minimize Loown (x, W) log (y x, W)..
3.2 Soft Prompt Learning
Prompt tuning (Lester al., 2021) is lightweight alternative to the full fine-tuning that introduces a trainable prompt P = [p1 P2. ...., Pz] as a prefix to the input embedding = [e(x1), c(x2), c(xr) to replace the update on entire model. The prompt length is L, e represents the embedding layer in LM, and p; R has the same dimension d as the token embedding. During prompt tun- ing, the concatenated matrix [P; Ej E R(L+T)xd is used as the input to the LM, expressed as B(x, P). The downstream task optimization is rep- resented as Edoum(xx P) = - log x, P) - log Idown (B(x, P)), where Idoun is the predic- tion layer for the task and the model B does not update in conventional soft prompt learning.
Though these methods alleviate CF during CPT, they ignore the importance of adaptation to domain knowledge for better fine-tuned performance (Gu- rurangan et al., 2020; Dery et al., 2022) and gen- eralization to unseen domains (Wortsman et al., 2022; Andreassen et al., 2022). Our work utilizes the potential of LM and improves all three aspects.
3.3 Continual Pre-training for Anytime Fine-tuning
3 Preliminaries
Continual pre-training (Jang et al., 2022: Meng et al., 2023) is a way to efficiently adapt to the new domain while maintaining Iearned knowl- edge. The problem formulation is as follows (see Fig. I): assume a stream of new domains (e.g.. lat- est news about "Omicron" ") sequentially appears as D1,..., DN where D; is the distribution of i-th do- main over a finite vocabulary of tokens X' Initially, we have an LM that has been well pre-trained on the general corpus Co. such as Roberta. Then at each stage i, a collection of new unlabeled corpus G {x x E D.} is obtained. The existing LM continually pre-trains learn the new knowledge from D;, with the goal of improving performance for anytime fine-tuning, where the LM expected
Our language model B constructed using the Roberta architecture (Liu et al., 2019), which is based on a bi-directional Transformer structure.
LM takes a text sentence x1.r = [21, 22,.., .T] as input and encodes it into a contextual embedding h = [h,, k2, lT] B(xiT).
3.1 Pre-training and Fine-tuning Tasks
During pre-training, the model is trained to predict missing words in a given text sen- tence and thus acquires a general understand- ing of languages, such as syntax, semantics, and context. The pre-training task is called

---

Trainable Fromen
Aprermanti
[
Language model 8
[
1o. +.f
PP ,..,.
Frozen language model 8
Projection
Hppor
petwone
Diagreement
[b Poal Pot.
pasdanseyy Tranforme
B Hf
Bieoder
Domaa
Inpersmaak
dianibation
[a] Soft prompt tuning
(b) Hypemetwark prompt learning
Agrenment and Disagreement
An overview of the model structure, with dotted lines indicating trainable modules and solid lines indicating frozen modules. (a) denotes the soft prompt tuning (Sec. 3.2). (b) shows the pre-training domain 4 with the hnet-prompt module (Sec. 4.1). The hypernetwork takes the contextual embedding & as input and automatically generates a prompt considering domain and sample properties, which clusters for similar domains (Dy,.3.D.) together and facilitates knowledge generalization. (c) computes the agreement and disagreement losses (Sec. 4.2).
Figure 3:
get greater capacity when fine-tuned on tasks from all pre-trained, current, and unseen domains. Each domain has its labeled dataset D; = {(x, v) - (x)x, Di). where F E y pro- vides ground-truth labels for classification. During the evaluation, the LM B¹ pre-trained up to the i- train set D and then
ously learned knowledge. To achieve this, we pro- pose framework, dubbed HPrompt-CPT, which consists of two components: the Pnne--rompt mod- ule and Agreement and Disagreement losses. The overview is presented in Fig. 3.
4.1 Hnee-Prommt for Pre-training and Fine-tuning
th domain, is fine-tuned tested on Die to measure its domain performance, as illustrated in Fig. 1. The resulting accuracy, de- noted as Acee; (simplified as a)), indicates the model capacity on task Dj as well as the degree of knowledge of j-th domain maintained by LM after being sequentially trained up C,.
Previous soft prompt methods (Qin and Joty, 2022; Zhu et al., 2022; Razdaibiedina et al., 2023) have made great success in the CL, with almost no catastrophic forgetting. However, these parameter- efficient methods fall short in model adaptation during the pre-training stage and fail to exhibit generalization capabilities when faced with new domains, as shown in Fig. 2. On the other hand, prompt engincering has shown exceptional perfor- mance in pre-training language models to better learn domain-specific knowledge (Radford et al.. 2019; Brown et al., 2020). However, the use of hard-coded prompts makes it difficult to implement and less relevant generalization.
Through the integration of results, accuracy table is generated, allowing for the computation of three crucial metries in anytime fine-tuning as discussed Sec. I: adaptability. generalization, and forgetting. The values used to calculate these metrics are indicated by different colors in Fig. 1. Red cells along the diagonal of the table repre- sent adaptability, indicating the degree to which the LM learns knowledge relevant to current do- main. Yellow cells in the upper triangle represent generalization, signifying the ability to perform ef- fectively in future domains. Blue cells in the lower triangle represent forgetting, reflecting reduction in previously learned knowledge during training.
Therefore, inspired by previous meta-learning approaches (Qiao et al., 2018; Yao et al., 2019), we propose a prompt module with a meta hypernet- work (Hnet-Prompt) for automatic knowledge adap- tation and cross-domain gencralization. Specifi- cally, when a batch of data ix!....x x²2 in a spe- cific domain D; comes, the hypernetwork gener- ates prompt P for each sample (see Fig. 3(b)), taking into account both domain and sample prop- erties while generalizing knowledge from learned domains. The process parameterized as:
4 Method
A successful algorithm of continual pre-training for anytime fine-tuning should meet the following re- quirements: (1) effective adaptation to the current domain and capturing more domain knowledge, (2) strong generalization to tasks in unseen domains, and (3) minimal catastrophic forgetting of previ-
p = F(h") = F( F(E(x))),
(1)

---

where E refers to text encoder, F corresponds to a hypernetwork, and h' represents the contextual embedding, which captures both the sentence and implicit domain information.
their unique prompt embedding with the knowl- edge generalized from learned domains. Given a task L, the entire model will be fine-tuned on the smaller labeled dataset, using the objective doown (X, Q) = log (y x, O) Here hypernetwork F is also trainable to get the best adaptation to downstream tasks. The fine-tuned performance the task shows the degree of do- main knowledge maintained by the LM.
Hypernetwork F encodes the domain feature of input samples (we use 6-layer Transformer) and then projects the pooled feature to obtain the prompt (see Fig. 3(b).. Rather than directly gen- erating the prompt, we set M prompt components Vm E Lxd and generate a weight vector a C RM to get the final prompt P = =m== Om Vm. Vector a controls the contribution of each prompt compo- nent, which corresponds approach reduces the parameter of the linear layer for projection and alleviates forgetting by shifting the learning problem from remembering the entire embedding to a weight vector.
4.2 Agreement and Disagreement Losses for Prompted Language Model
basic domain. This
While preventing the forgetting of learned knowl- edge is always the key challenge in continual pre- training, they are the cost of adaptability and gen- eralization. To overcome it, we propose a novel ap- proach, named agreement and disagreement losses. Agreement loss. While knowledge distillation (KD) has been demonstrated to perform well in overcoming CF (Chuang et al., 2020; Dong et al., 2021), its alignment the entire feature space can limit the adaptation to new domains. To alle- viate it, we propose to align the output p(output imput, domain) of the prompted language model instead p(oulpul | inpul) used in conventional KD. We term this approach the agreement loss. Specifically, we begin with the prior Iearned LM B Then, initialize the random prompt P rand and generate prompted hidden states using both cur- rent LM B' and previous LM Bi-1 (see Fig. 3(c)). We then minimize the distance metrics M between the outputs of two models, as shown below:
Prompt components V, analogous to a set of basis vectors, are set of prompt embeddings that are randomly initialized, trainable and optimized through gradient descent. The well-trained prompt components are supposed to offer greater gener- alization to future domains as long the prompt components are as mutually exclusive as possible. For example, a prompt embedding directly opti- mized for the domain of "ACL papers" does not directly apply to the domain of AI papers" due to the domain difference; however, one of the prompt components learned on "ACL papers", e.g., "deep learning", can be combined with another compo- nent of "statistics" to generalize to the domain of "Al papers".
During pre-training, the language model is con- ditioned on the prompt generated by the hypernet- work, which models p(output input, domain) and injects the domain knowledge into the model in an explicit way. Then, we optimize the language model and hypernetwork in an end-to-end manner by minimizing the following equation:
la W) = M(Bi-((x, (x,Prand)
(3)
B Prand)],
where Prand simulates the condition to active generic or learned domain knowledge. The agree- ment loss, which operates on B(- P Prana). effec- tively prevents forgetting by enforcing consistency on multiple randomized conditions and preserves the plasticity to new domains by maintaining model capacity conditioned on other prompts, as demon- strated by a comparison to KD. A smaller M indi- cates a closer distance between the two inputs. In this article, we use cosine similarity to calculate M, which performs better than the KL distance between logits in the experiments in Sec. 5.4. Disagreement loss. Besides the consistency achieved by agreement loss, we also expect the exclusiveness the generated hidden states for the current domain. brings two advantages: (1) it re- duces interference to established knowledge, which
bmtm (x, W)
Σ
log (6 X)m(()) W, 0),
(2)
BEm(x)
where p(-) - Immm (Bw (x) (x)mmx); Fe (xxm(x)) ))) and θ is the parameter of F. This approach allows for qualified and automatic adaptation to domain knowledge and enables the transfer of this knowl- edge across domains through hypernetwork.
During downstream task fine-tuning, domain identity not required anymore. Hypernet- work will automatically map the input samples to

---

Table I: Performance of baseline results on DAPset/TWEET benchmarks (all results reported in this paper are averaged over 4 random seeds). The symbol "-" in the table is because F "_Acc is the same as the average accuracy A_Acc in the separate pre-training settings. We also report the results for different domain orders in Appendix D.
DAPset O_Ace
TWEET O_Ace
Setting
Method
AAcc
F_Ace
A_ecc
F_Acc
Initial
0,8033
0.110
0.8171 + 0.010
0.7933 D001
0,7935 =
Separate
Mulii Task
0.8203
0,002
0.8299 1: 0.005
0.8014 1: 0.002
0,8047 1: 0.001
Pre-training
One-Full
0.8235
0.007
0.8174 + 0.008
0.8037 + 0.001
0,8064 : 0.001
0,8060
0,008
0.8172 0.003
0.7913 +: 0.002
0.7915 :: 0.003
One-Proopp
08101
0.012
0.8109 + 0.012
0,7873 + 0.022
0.7876 = 0.002
NCL
0.8298
0.005
0,8189 1 0.006
0.8198
0.005
0.8108
0.002
0.8094 1. 0.001
0.8079
0,001
BWC
088882
0.004
0.8109 + 0.003
0,8020
0.003
0,8028 +: 0.001
0,8048 + 0.001
0.8037
0.001
Pre-training
Continual
DF
LwF
0.8245
0.002
0,003
0.8174 + 0.004
0.006
0,8289 + 0.001
0.8179
0.006
0.8102 + 0.001
0,8087 + 0.001
0.8118
0,8082 0.001
0.001
CoDA-Prompt DAS
0.8141
0.002
0.8161 + 0.004
d8176
+ 0.004
0.7951
+ 0.001
0.7954 + 0.001
0.7958 0.001
0.8221
0.004
0.8164
0.001
0.8251
0.006
0,8066
0.001
0,8078 : 0.001
0,8099
0.003
Qurs
0.8356 0.002
0.8277 + 0.003
0.834
+ 0.003
0.8186 +: 0.001
0.8168 + 0.002
0.8203 0.001
mitigates forgetting (Farajtabar al., 2020; Wang et al., 2021b); (2) it encourages generalization when fine-tuning by incorporating a wider range of domain knowledge (Pagliardini et al., 2023). To achieve this exclusiveness, we add a loss function called disagreement loss. Specifically, when sam- ple comes, we generate the prompt using hyper- network F and train the prompted LM to maxi- mally disagree with the output of the previous LM, which also promoted by the same embedding (see Fig. 3())). This involves minimizing the agree- ment metric -(.,-) -) to push apart the two prompted hidden states:
5.1 Benchmarks
DAPset. It is benchmark for continual domain adaptive pre-training originally constructed by (Ke et al., 2023). It consists of six domains, each with an unlabeled corpus and corresponding end-task classification dataset. Each domain contains a cor- pus size of over 100 million tokens, and we follow the original data construction and task order.
TWEET. We develop new benchmark based on tweet dataset (Jin et al., 2022) to simulate the distribution shift over time. The dataset includes tweets from 2015 2019 and is split into five time periods to form five domain corpora, each with over 50 million tokens. The tweet texts are pre- processed following Nguyen et al. (2020). For the downstream task, we build a single-label hashtag prediction dataset for each domain following Gong and Zhang (2016). TWEET keeps the chronolog- ical order of domains to simulate the updating in the real-world system. Please refer to Appendix B for more information about the two benchmarks.
Eda W, = A(B' 1(x, F(x)), ( (x()))),
(4)
thereby increasing the exclusiveness of the output of LM for the current domain. Sec. 5.4, we compare various implementation of A including orthogonal constrain (Smith et al., 2023), softmax variant (Pagliardini et al., 2023), opposite value of KL-divergence. Ultimately, onal constraint, which can be caleulated using the equation Aortho (X, Y) ||XYT_ Il].
select the orthog-
5.2 Metries and Baselines
Metrics. We introduce three attributes of continual pre-training in Sec.3.3 and provide an explanation of their evaluation methods. Formally, we utilize the adaptation accuracy A_Ace = + E/h at to measure adaptability, the out-of-domain accuracy 2 Σ Df at to evaluate
Finally, the loss function of our HPrompt-CPP during pre-training can be summarized as follows:
N
L-
Σ mmim + Ai + Azlias
(5)
O Acc --ア generalization, and the final accuracy F F_Ace- + Σ of to assess the degree of catastrophic for- getting. Here, af represents the fine-tuned accuracy on the i-th downstream task, after being sequen- tially trained up to corpus Cj in the j-th domain. Baselines. We first evaluate the algorithms that build separate model for each domain, in- cluding: (1) Initial is fune- on the initial
where N is the batch size, and \1, A2 are the trade- off hyper-paramettrs. The loss input is omitted.
5 Experiment
In this section, we conduct experiments on two benchmarks to investigate the adaptability, general- ization, and degree of forgetting of HPrompt-CPT.

---

pre-trained point. (2) Multi-Task is domain- adaptively pre-trained on the mixture of all do- mains. (3) One-Full domain-adaptively pre- trained with the updates on the full model. (4) One- Adapter is domain-adaptiven adapter layer (Houlsby et al., 2019). (5) One- Prompt is domain- aaaniivldaapivelly pre-trained with new prompt (Lester et al., 2021). Additionally, test continual pre-training methods: (6) NCL is sequentially pre-trained without any CL methods. (7) EWC (Kirkpatrick et al., 2017) a regulariza- tion method that penalizes changes to important neurons. (8) DERpp (Buzzega et al., 2020) is a re- play method in both sample and feature levels. (9) LwF (Li and Hoiem, 2017) uses knowledge distil- lation to protect previous predictions. (10) CoDA- Prompt (Smith al., 2023) uses set of prompt components learn domain-specific knowledge. (11) DAS (Ke al., 2023) is paramcter-isolatinn method which adopts soft-masking.
dempp
0.03
pre-trained with an
0.81
160
645%
A,ACC
0,ACC
F_ACC
Figure Performances on DAPset with different sizes of the corpus. The implementations of "ours (trans/lin)" refer to utilizing transformer/linear hypernetwork in HPrompt-CPT, respectively.
sub-optimal for continual pre-training. Our hnet- prompt method achieves a relative improvement in F A_Acc of 1.69% on DAPset and 2.35% on TWEET, suggesting that it can alleviate negative transfer between conflicting domains and mini- mize forgetting. It worth noting that the 0 O_Ace metric of multi-task learning cannot be compared fairly with other algorithms since it has already observed all domains. Nevertheless, our algorithm still achieves a 1.50% gain on TWEET, which may result from the generalization of the diverse domain knowledge in HPrompt-CPT.
For HPrompt-CPT, former hypernetwork and frozen Roberta as text encoder. We the prompt length 50, and the size of prompt components to 100. In addition, we implement a replay loss to the hypernetwork with a memory buffer storing 300 samples to get the best performance, while removing it resulting in a minimal drop of 0.24% in F F_Ace on DAPset. Dur- ing fine-tuning, we train each task for 15 epochs with early stopping mechanism using the val- idation data (30% of testing data). We include additional Implementation Details in Appendix
adopt 6-layer Trans-
Observation 3: Full model tuning achieves better results in learning and transferring domain knowl- edge. Our proposed method and NCL outperform parameter-efficien methods such as One-Adapter, One- Prompt, and CoDA-Prompt. Interestingly. methods that incorporate regularization terms on parts of neurons, such as EWC and DAS, also re- sult in lower A_ Acc. This suggests that injecting a large amount of domain knowledge into the LM re- quires sufficient number of trainable parameters. Our prompted LM, with all parameters trainable and no empirical constraints on updates, shows the best adaptation performance.
5.3 Results and Analysis
Comparison with the state-of-the-art. Table 1 shows the continual pre-training performance of different methods three dimensions. From these results, make the following observations:
Observation 1: HPrompt-CPT outperforms base- lines in terms of adaptability, generalization, and avoidance of catastrophic forgetting. Our approach achieves state-of-the offtthe-rt art results across all three metrics, with increases of 1.38% and 1.09% on the DAPset in terms of generalization and final per- formance compared to the most recent algorithm, DAS, as depicted in the last row of Table These results highlight the advantages of injecting do- main knowledge into the LM with the hnet-prompt module, which aids in adaptation and promotes knowledge transfer.
Data-efficient pre-training, Note that we hypothe- size that HPrompt-CPT especially effective in the setting of anytime fine-tuning. Its performance on a small subset of the corpus is worth referring to, for the model can be utilized for fine-tuning in cases where a domain is not finished training. Fig. 4 illus- trates the performances trained different sizes of datasets and highlights the effectiveness of our method in low-resource environments, particularly in terms of generalization ability. Our design of the mnct-prompt module successfully promotes knowl- edge transfer across domains, and besides we ob-
Observation 2: Naive multi-task learning is

---

Initial
After training
Table 2: Ablation results on the main components.
Hiokln
Hypernework
La Edit
Aoe
o_ Oce
F_Anc
state
X
0.8165
0.8066
0.8114
/
V
0.8223
0.8149
0.8208
Prompt ember ddiig
C.0
/
X
0.8307
0,8385
0.8235
0.8168
0.8280
0.8297
V 0.8356
0.8277
0.8341
C.D.
Figure 5: The t-sne map about prompt embedding and hidden state of the last layer. C₁ and D; denote the pus and downstream task in i-th domain, respectively.
with progprompt (Razdaibiedina et al., 2023), which generates a new soft prompt for each domain and concatenates it and previously learned prompts while requiring domain-id during fine-tuning. As shown in Table 2 (rows 1 and 3), it results in significant decrease in performances, particularly in adaptability, with an almost 1. 1.77% decrease. It highlights the effectiveness of hnet- prompt in adapting and generalizing domain knowl- edge, providing great capacity for fine-tuning.
the
serve that the structure of the hypernetwork matters such settings, Transformers may underfit facing smaller datasets, resulting in poor performances compared to the linear structure.
Analysis on the distributions of hnet-prompt em- beddings and hidden states. We perform qualita- tive analyses on prompts and hidden states gener- ated by HPrompt-CPT to investigate whether the hypemeetwrk can generalize domain information. As depicted in Fig. 5, We use t-sne map (van der Maaten and Hinton, 2008) to visualize the model output before and after training on all six domains in DAPset. For prompts, we observe that the gen- erated prompt embeddings can effectively cluster similar domains together (e.g., overlapping embed- dings for corpora C2 Cz. and Cs from the same paper dataset) while also achieving differentiation for dissimilar domains (c.g., distant embeddings for C1 (restaurant) and Cs (bio-chem)). This is an impressive result, i.e., it transfers the information across domains, making it easier for the LM to effectively adapt and generalize knowledge.
To examine the effect of the agreement and dis- agreement losses, we compare the results of train- ing progressive prompt and hnet-prompt with and without them. shows that incorporating the agree- ment and disagreement losses lead to a 1.15% and 1.20% improvement in F_Ace for the two mod- els, respectively, demonstrating its efficiency in preventing CF. Furthermore, we observe that in- troducing the disagreement loss results in a 1.33% gain in O Accc which is attributed to the incorpo- ration of wider range of domain knowledge for adaptation, as discussed Sec. 4.2.
Hypernetwork structure. We further investigate the different designs of hypernetwork and present the results in Table 3 (top). First, compare the network structure with the Linear layer or Mul- tilayer Perceptron (MLP) (the top two rows), but they show poor adaptability and a higher level of CF. Interestingly, we find that the linear structure is more stable when facing a low-rcsource setting, Besides, we examine the performance of generat- ing prompt embedding directly show the signifi- cance of the component- based method introduced in Sec. 4.1. The results reveal that the component- based approach outperforms in generalization and preventing forgetting, benefiting from shifting the lcarning problem from remembering prompt to the weight vector which is a simple task.
For hidden states, our model generates distin- guishable hidden states for downstream task based on pre-trained domain information, i.e.,the initially mixed downstream representation (D1 Do in Fig. 5 top right) are successfully separated in Fig. 5 top left. For instance, the model assigns overlap- ping representations to similar tasks D2 and D3 (belonging to ACL and AI, respectively), while pro- viding effective differentiation for unrelated tasks D1 (restaurant) and D5 (biology).
5.4 Ablation Study
Table 2 and 3 present the results of different de- signs of HPrompt-CPT on DAPset, where hyper- parameters fixed across all settings.
grrement and disagreement loss objective. We first replace the agreement loss with the conven- tional KD and the result are presented in the first row of Table (middle). It shows agreement loss leads to a 1. 1.06% improvement in adaptability while
Effectiveness of the main components. To as- sess the impact of the hypernetwork, we replace

---

Table 3: Ablation results on the hypernetwork structure and agreementdisaggeemmnnt loss objective. Here, la and la denote the two losses. The content in parenthe- ses represents the applied objective. The logúf refers to minimizing the mean square error on logits. The KL. distance aims to maximize the KI. distance between hidden states. The seftmax variant to maximize the softmax on logits, following (Pagliardini et al., 2023).
benchmarks with severe domain conflicts. The do- mains in the benchmarks used in our paper are mostly transferable to each other. For example, the Domain " "ACL" and "Al" in DAPset are highly re- lated. We are not sure how will our method perform in a sequence of domains with little to no shared knowledge or even confticts. In addition, we cur- rently only test our method on the classification task, while the exploration of more types of down- stream tasks is also important. Our future work 0,8295
Acc
O_Acc
F_Acc
Hyper- nerwork related
Lincar network
0.8279
MLP network
0.8324
0,8210
Generate prompt directly
0.8336
0.8208
0.8305
will extend the benchmark to cover such cases.
&a troplaced fu (lngit)
0.8268
0.8230
0,8269
Another problem for HPrompt-CPTI is the selce- tion of hypernetworks. Our experiments in Sec. 5.3 demonstrate that decreasing the size of the unla- beled corpus can cause the Transformer structure to underfit, while the Linear structure cannot cap- ture all the information from large corpus. In addition, we find the fine-tuning of hhpermetwork is sensitive to the learning rate and weight decay. We aim to enhance the capacity and stability our hypernetwork. Moreover, it is best to get a hyper- network that can generalize well on downstream tasks without fine-tuning.
fu Edt related
0,8310
0.8256
0,8295
(KL distance)
0.8330
0.8253
0.8325
Eda (sofimax variani)
0.8306
0.8242
0.8316
Ours
0.8356
0,8277
08841
maintaining its ability avoid forgetting, demon- strating its advantage in striking a balance of stabil- ity and plasticity for LM. Then, as is unclear what kinds of objectives are most suitable to overcome forgetting, we test various objective functions for agreement and disagreement losses in Table 3 (mid- dle). Ultimately, minimizing the KL-divergence of randomly prompted hidden states (agreement loss) and minimizing the orthogonal distance of current hidden states (disagreement loss) yield the best fi- nal performance of 83.41%.
Acknowledgements
The work was supported by grants from the National Key R&D Program of China (No. 20217D01 1 1801) and the National Natural Science Foundation of China (No. 62022077).
6 Conclusion
This paper introduces HPrompt-CPT, novel prompt-guided continual pre-training method to- wards anytime fine-tuning, which enables better performance when fine-tuned on seen and unseen domains. By training hypernetwork to generate domain-specifie prompts with agreement and dis- agrecment losses, it results in (i) greater capacity on pre-trained domains by learning domain knowledge with generated prompts while preserving previous knowledge with random prompts, (ii) improved per- formance unseen domains by retaining model plasticity with agreement loss and the ability of knowledge transfer with hypernetoork, and (iii) no need for domain-id during fine-tuning. We set a new SOTA on both well-established benchmark and temporal shift benchmark.
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elho- seiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to for- get. In Proceedings of the European Conference Computer Vision (ECCV), pages 139-154.
Anders Johan Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. 2022. The evo- lution out-of-distribution robustness throughout fine-tuning. Transactions on Machine Learning Re- search.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
7 Limitations
Pietro Buzzega, Matteo Boschini. Angelo Porrello, Da- vide Abati, and Simone Calderara. 2020. Dark expe- rience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33: 13200-99-55930
While we have evaluated our approach on two continual pre- trainingn benchmarks, it remains un- known how well our method would perform on

---

Hyuntak Cha, Jacho Lee, and Jinwoo Shin. 2021. Co21: Contrastive continual learning. In Proceedings of the IEFF/CVF International conference on computer vision, pages 9516-9525.
Gongfan Fang. Yifan Bao, Jie Song, Xinchao Wang. Donglin Xie, Chengchao Shen, and Mingli Song. 2021. Mosaicking to distill: Knowledge distillation from out-of-domain data. Advances in Neural Infor- mation Processing Systems, 34:1 1920-11932.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019, Efficient lifelong learning with A-GEM. In ICLR (Poster). OpenReview.net.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. 2020. Orthogonal gradient descent for continual learning. In AISTATS, volume 108 of Proceedings of Machine Learning Research, pages 3762-3773. PMLR.
Defang Chen, Jian-Ping Mei, Hailin 7hang, Can Wang, Yan Feng, and Chun Chen. 2022. Knowledge distilla- tion with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11933-11942.
Yuyun Gong and Qi Zhang. 2016. Hashtag recom- mendation using attention- basod convolutional neural network. In WCCA,, pages 2782-2788. UCAI/AAAI Press.
Yung-Sung Chuang, Shang-Yu Yu Su, and Yun-Nung Chen. 2020. Lifelong language knowledge distillation. In EMNLP (1), pages 2914-2924. Association for Com- putational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretrain- ing: Adapt language models to domains and tasks. In ACL, pages 8342 0360. Association for Computational Linguistics.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. continual learning vey: Defying forgetting classification tasks. IEEF transactions on pattern analysis and machine intelli- gence, 44(7):3366-335
Hongyan Hao, Zhixuan Chu, Shiyi Zhu, Gangwei Jiang, Yan Wang, Caigao Jiang, James Zhang. Wei Jiang. Siqiao Xue, and Jun Zhou. 2023. Continual learning in predictive autoscaling. In CIKM.
Cyprien de Masson D' Autume, Sebastian Ruder, Ling- peng Kong, and Dani Yogatama. 2019. Episodic memory lifelong language learning. Advances in Neural Information Processing Systems, 32.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025:: 003.00231.
Lucio M Dery, Paul Michel, Ameet Talwalkar, and Gra- ham Neubig. 2022. Should we be pre- pre-training? an argument for end-task aware training an alternative. International Conference on Learning Representa- tions.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter--fficient transfer learning for NI.P. In ICML, volume 97 of Proceedings of Machine Learn- ing Research, pages 2790-2799. PMLR.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLI' (I), pages 4171-410 1171-4186. As- sociation for Computational Linguistics.
Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, Lanqing Hong. Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng. 2022a. How well does self- supervised pre-training perform with streaming data? In ICLR. OpenReview.net.
Jacob Devlin, Miig-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019b. BERT: pre-training of deep bidirectional transformers for language under- As-
standing. In NAACL-HLI' (I), pages sociation for Computational Linguistics.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022b. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net.
Xiaowen Ding, Bing Liu, and Philip S. Yu. 2008. holistic exsieonbbsedd approach to opinion mining. In WSDM, pages 231-240. ACM.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text clas- sification with information disentanglemen based regularization In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 2736-2746.
Songlin Dong, Xiaopeng Hong, Xiaoyu Tao, Xinyuan Chang, Xing Wei, and Yihong Gong, 2021. Few-shot class-incremental learning via relation knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1255-
1263.
Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. 2022. How catastrophic catastrophic forgetting be in linear regression? In COLT, volume 178 of Proceedings of Machine Learning Research, pages 4028-4079 PMLR.
Joel Jang, Seonghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022. Towards continual knowledge learning of language models. In ICLR. OpenReview.net.

---

Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, and Xiang Ren. 2021. Learn continually, generalize rapidly: Lifelong knowledge accumulation for few- shot learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 714-729.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. In EMNLP (7), pages 3045-3059. Associa- tion for Computational Linguisties.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. 2019. Learn to grow: A continual structure learning framework for overcoming catas- trophic forgetting. In International Conference on Machine Learning, pages 3925-39344 PMLR.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew O. Arnold, and Xiang Ren. 2022. Lifelong pretraining: Continu- ally adapting language models to emerging corpora. In NAACL-HLT, pages 4764 4780. Association for Computational Linguistics.
Zhizhong Li and Derek Hoiem. 2017. Learning without
forgetting. IEEE transactions on puttern analysis and machine intelligence, 40(12):2935-29)
David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Trans. Assoc. Comput. Linguistics, 6:391-406.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 61 61-68.
Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jachong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. 2022. Forget-free continual learning with win- ning subnetworks. In International Conference on Machine Learning, pages 10734-10000 10750. PMLR.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 197.:1999..119..
Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. 2022a. Continual training of lan- guage models for few-shot learning. arXiv preprint arXiv:2210.05549.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel S. Weld. 2020. S2ORC: the semantic scholar open research corpus. In ACL. pages 4969- 4983. Association for Computational Linguistics.
Zixuan Ke, Bing Liu, Hu Xu, and Lei Shu. 2021a. Clas- sic: Continual and contrastive learning of aspect sen- timent classification tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6871-6883.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP, pages 2232-2232. Association for Computational Linguistics.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. Continual learn- ing of language models. In ICLR. OpenReview.net.
Michael McCloskey and Neal Cohen. 1989. Catas- trophic interference in connectionist networks: The sequential learning problem. Psychology of learn- ing and motivation, volume 24, pages 109-165. Else- vier.
Zixuan Ke, Yijia Shao. Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. 2022b. Adapting a language model while preserving its general knowledge. In EMNLP, pages 10177-10188. Association for Computational Linguistics.
Kevin Meng. Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass- editing memory in a transformer. In The Fleventh International Conference on Learning Representa- tions.
Zixuan Kc, Hu Xu, and Bing Liu. 2021b. Adapting bert for continual learning of a sequence of aspect senti- ment classification tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4746-4755.
Qiang Meng. Chixiang Zhang. Xiaoqiang Xu, and Feng Zhou. 2021. Learning compatible embeddings. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 9939-9998,
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins. Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Barwe Barwinska, et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, I 114(13):3511-352
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. Bertweet: A pre-trained language model for english tweets. In EMNLP (Demos), pages 9-14. Association for Computational Linguistics.
Jens Kringelum, Sonny Kim Kjærulfff, Søren Brunak, Ole Lund, Tudor I. Oprea, and Olivier Taboureau. 2016. Chemprot-3.0: a global chemical biology dis- eases mapping. Database J. Biol. Databases Curu- tion, 2016.
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In EMNLP/JCNLP (I), pages 188-197. Association for Computational Linguistics.

---

Matteo Pagliardini. Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. 2023. Agree to disagree: Diversity through disagreement for better transfer- ability. In The Eleventh International Conference on Learning Representations.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. Spot: Better frozen model adaptation through soft prompt transfer. In ACL (1), pages 5039-5059. Association for Computational Linguistics.
Siyuan Qiao, Cheaxi Liu, Wei Shen, and Alan L. Yuille. 2018. Few-shot image recognition by predicting pa- rameters from activations. In CVPR, pages 7229- 7238. Computer Vision Foundation/ IEEE Computer Society.
Chengyu Wang, Jianing Wang. Minghui Qiu, Jun Huang, and Ming Gao. 2021a. Transprompt: To- wards automatic transferable prompting frame- work for sew-shot text classification. In EMNLP (7), pages 2792-2802. Association for Computational Linguistics.
Chengwei Qin and Shafiq R. Joty. 2022. LFP15: A unified framework for lifelong few-shot language learning based on prompt tuning of TT. In ICLR. OpenReview.net.
Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. Elle: Ef- ficient lifelong pre-training for emerging data. In Findings of the Association for Computational Lin- guistics: ACI. 2022, pages 2789-2810.
Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. 202 Training networks in null space of feature covariance for continual learning. In CVPR, pages Computer Vision Foundation / IEEE.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi. Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfis- ter. 2022a. Dualprompt: Complementary prompting for rehearsal-free freee continual learning. In ECCV (26). volume 13686 of Lecture Notes in Computer Science, pages 1-8. Springer.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. CoRR, abs/2200.08827.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022b. Learning to prompt for continual leamnin. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Ma- dian Khabsa, Mike Lewis, and Amjad Almahairi. 2023. Progressive prompts: Continual learning for language models. In ICLR. OpenReview.net.
Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. 2022c. Learning to prompt for continual learning. In CVPR, pages 139-149. IEEE.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018. Overcoming catastrophic forget- ting with hard attention to the task. In International Conference on Machine Learning, pages 45577 PMLR.
Maciej Wolczyk. Michal Zajac. Razvan Pascanu. Lukasz Kuciński, and Piotr Miloš. 2021. Contin- ual world: A robotic benchmark for continual rein- forcement learning. Advances in Neural Information Processing Systems, 34.28496-28510.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Ar- belle, Rameswar Panda, Rogério Feris, and Zsolt Kira. 2023. Coda- Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free contin- ual lcarning. In CVPR. IEEE.
Maciej Wolczyk. Michat Zajac, Razvan Pascanu, Lukasz Kuciński, and Piotr Mitos. 2022. Disentan- gling transfer in continual reinforcement learning. Advances in Neural Information Processing Systems, 35:6304-6317.
Fan-Keng Sun, Hao Ho, and Hung- Yi Lee. 2019. Lamol: Language modeling for lifelong language learning. arXiv preprint PrXiv:/909.0332
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis K. Titsias 2022. Information- theoretic online memory selection for continual learn- ing, In The Tenth International Conference Learn- ing Representatinns, ICLR 2022, Virtual Event, April 25-29, 2022. OvenReview.net.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong. and Ludwig Schmidr. 2022. Robust fine-tuning of zero-shot models. In CVPR, pages 7969-7961. IEEE
Laurens der Maaten and Geoffrey Hinton. 2008. Vi- sualizing high-dimensional data using t-sne. Journal of Machine Learning Research, 9:2579-2605.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. 2020. Knowledge distillation meets self- supervision. In Computer Vision-ECCV 2020: 16th

---

European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX, pages 388-604, Springer.
model first learns universal language representa- tions on large corpus (pre-training stage), and then fine-tunes on downstream tasks (fine-tuning stage). Therefore, CL methods for LM can be cate- gorized into Continual Pre-training and Continual Fine-tuning. In this paper, we focus on the contin- ual pre-training.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019. BERT post-training for review reading comprehen- sion and aspect-basee based sentiment analysis. In NAACL- HLT (1), pages 2324-2335. Association for Compu- tational Linguistics.
Siqiao Xue, Yan Wang. Zhixuan Chu, Xiaoming Shi, Caigao Jiang, Hongyan Hao, Gangwei Jiang. Xiaoyun Feng, James Zhang. and Jun Zhou. 2023. Prompt-augmennte temporal point process for streaming event sequence. In Advances Neural Information Processing Systems.
Continual Fine-tuning. Continual Fine-tuning (FT) entails training a model on stream of down- stream tasks after its initialization. This approach requires the model to transfer knowledge to new tasks, avoid forgetting, and perform consistently well on all tasks learned before. Differing from the traditional CL, continual FT will mostly uti- lize the ability of a pre-trained language model. For instance, in replay methods, MbPA+ (de Mas- son D' Autume et al., 2019) uses BERT (Devlin et al., 2019a) to get the key representation of old examples for local adaptation when inference, while LAMOL (Sun al., 2019) generates old examples through the generative ability of GPT2. IDBR (Huang et al., 2021) prevents forgetting by learning generic and task-specific knowledge us- ing disentangleement-bane based losses. Recently, re- searchers have explored parameter-efficient tun- ing (Houlsby et al., 2019; Lester et al., 2021; Hu et al., 2022b) for CL. One kind of approaches (Ke et al., 2021b,a; Jin et al., 2021) involves using adapter modules (Houlsby et al., 2019) to incor- porate task-specific parameters into frozen trans- former layers, while the other (Qin and Joty, 2022; Zhu et al., 2022) uses soft prompts (Brown et al., 2020) to activate model ability to solve different tasks.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. 2019. Hierarchically structured meta-learning. In International Conference on Machine Learning, pages 7045-7054. PMLR.
Jaehong Yoon, Saehoon Kim, Funho Yang, and Sung Ju Hwang. 2020. Scalable and order-robust continual learning with additive parameter decomposition. In ICLR. OpenReview.net.
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intel- ligence. International Conference on Machine Learning, pages 3987-3995. PML.R
Qi Zhu, Bing Li, Fei Mi, Xiaoyan Zhu, and Minlie Huang. 2022. Continual prompt tuning for dialog state tracking. ACL (1), pages 124-1137. Associ- ation for Computational Linguistics.
A Additional related work
Continual learning. Continual Learning (CL) fo- cuses on the problem of sequential learning from a stream of data that comes different distribu- tions. It is desired to extend the acquired knowl- edge to future tasks while avoiding catastrophic forgetting (CF) (McCloskey and Cohen, 1989) of the past tasks, and this has been successfully im- plemented in various fields, including computer vi- sion (De Lange et al., 2021; Cha et al., 2021; Wang al.. 2022b), natural language processing (Sun et al., 2019; Huang et al., 2021: Qin and Joty, 2022), and Robotics (Wolczyk et al., 2021; Wolezyk et al.. 2022). Traditional CL approaches fall into three types: regularization methods (Kirkpatrick et al., 2017: Li and Hoiem, 2017; Zenke et al., 2017; Aljundi et al., 2018), 88, replay methods (Aljundi et al., 2018; Buzzega et al., 2020; Sun et al., 2022), and architecture- based methods (Serra et al., 2018; Li et al., 2019; Kang et al., 2022). However, design- ing CL methods for Language Modeling (L.M) is different due to the two-stage training scheme (De- vlin et al., 2019b; Qiu et al., 2020), where the
Soft Prompt Learning, Recent works (Wang et al.. 2021a; Vu et al., 2022) have shown the potential of parameter-efficient learning in achieving multitask performance at low cost by stacking lightweight units. Their success attracts a surge of attention in adapting them to CL, especially for soft prompt tuning (Lester et al., 2021; Liu et al., 2022). These methods aim to transfer knowledge between tasks using prompts while avoiding forgetting. For exam- ple, LFPT5 (Qin and Joty, 2022) employs large soft prompt that is continually trained on all tasks while also distilling previous knowledge. Continual Prompt Tuning (Zhu et al., 2022) uses initialization. memory replay, and AGEM (Chaudhry et al., 2019) technologies to facilitate prompt forward/backward transfer. ProgPrompt (Razdaibicdina et al., 2023) leverages previously learned prompts by concate-

---

sentence contains a citation, classify the intent of this citation. (3) Relation classification: given a sentence together with its entities, classify the learn a cluster-based mapping from input data to a specific prompt. Furthermore, CodaPrompt (Smith et al., 2023) suggests learning composition of the prompt pool that replaces index operations with backpropagation- based approach.
nating them with new embeddings. While most of these methods require lask-id to select the appropri- ate prompt, DualPrompt (Wang et al., 2022a) and L2P (Wang et al., 2022c) create a prompt pool and
relation of these entities. (4) Chemical-protedd in- teraction classification: given a sentence containing pair of chemicals and proteins, classify the inter- action between these two.
new benchmark, TWEET, following (Jin et al., 2022), to simulate the distribution shift over time. The dataset is col- lected by the Archive team and we select the text data from 2015 to 2019, dividing it into five time periods and creating five domain corpora. The text data was pre-processed the text data according to (Nguyen et al., 2020) with the hashtags removed. We randomly select 300 MB of data from each pe- riod, and the statistics of each domain is present in Table 4.
TWEET. We develop
Konwledge distillation. Knowledge distillation (KD) (Hinton et al.. 2015) is a widely-used tech- nique for improving performance and efficiency in various tasks, such as model compression (Meng et al., 2021; Chen et al., 2022) and transfer leamn ing (Xu et al., 2020; Fang et al., 2021). KD has also been applied in continual learning to transfer knowledge learned from old tasks to new ones and thus prevent forgetting (Chuang et al., 2020; Dong et al., 2021; Ke et al., 2021a). However, previous approaches mainly focused on aligning the entire feature space, which can limit the adaptation ability of the model. Instead, we propose using agree and disagree loss to decompose KD into prompt and feature space, achieving better balance be- tween plasticity and stability.
For the downstream task, we build a single-label hashtag prediction task for each domain follow- ing Gong and Zhang (2016). We count the 10 most frequently hashtags (e.g., "#hiring". "#music") in each time period and extract 700 tweet texts for each label, including 200 tweets for training, 200 tweets for validation, and 300 tweets for testing. Before the text is input into the model, we remove the hashtags themselves and ask the model to pre- diet the most appropriate hashtag for the current sentence.
B Dataset Details
DAPset. It is a benchmark for continual domain adaptive pre-training, constructed by (Ke et al., 2023). It consists of six domains, each with an unlabeled corpus and a corresponding down- stream task classification dataset. They are from two large datasets, while 3 of them are about re- views: Yelp Restaurant (Xu et al., 2019)/ Restau- rant (Ding et al., 2008), Amazon Phone (Ni et al., 2019)/ Phone (Ding et al., 2008), Amazon Cam- era (Ni et al., 2019)/ Camera (Ding et al., 2008) and 3 of them are academic papers: ACL pa- pers (Lo et al., 2020// ACL-ARC (Jurgens et al.. 2018), AI papers (Lo et al., 2020)/ SCIERC (Luan al., 2018), and PubMeb papers (Lo et al., 2020)/ CHEMPROT (Kringelum et al., 2016). The front one is the unlabeled corpus and the latter one is the corresponding downstream task. We show the statistics of these datasets in Table 4.
C Implementation Details
We adopt Roberta-BASE. (Liu et al., 2019) as our backbone language model and 6-layer Trans- former (Vaswani et al., 2017) as our hypernetoor.. In the pre-training phase, we apply masked lan- guage model head after the LM, which is then re- placed with a classification head during fine-tuning, Each downstream task has its own classification head. In pre-training, the trainable parameters de- pend on the algorithm design, while in fine-tuning, all parameters, including the language model and added model, are trainable.
The maximum input sequence length is set to 164, following (Ke et al., 2023), and we use an Adam optimizer with weight decay of 0.01 for both pre-training and fine-tuning. During pre- training, the learning rate is set to le-4 and batch size to 128. We train for 5K and 2.5K steps for each domain in DAPset and Tweet, respectively, which is roughly a full pass through the domain
The downstream tasks in DAPset can be divided into the following 4 types. (1) Aspect sentiment classification: given an aspect (e.g., environment in a restaurant review) and the corresponding review text, classify it into positive, negative, or neutral sentiment. (2) Citation intent classification: given

---

Table 4: Statistics datasets for Dses and TWEET.
Unlabeled Corpus
Downstream Task Datasets
Benchmarks
Source
Dutaset
Size
Datuset
Classification Task
#Training
#Testing
AClasses
Yelp Restaurant
758MB
Restaurant
Aspact Sentiment
3,452
1.120
Reviews
Amazon Phone
724MB
Phone
Aspect Sentiment
239
553
2
Amnzon Cameraa
319MB
Camera
Aspect Sentiment
230
626
2
DAPset
Papers
ACL-ARC
Citation
1,520
Academic
Papers
A1 Papens
507MB
SCIERC
Relation
2,260
2,388
7
PubMed Papers
989MB
Chemical-protiin
2,667
TWEET
Tweet
300MB
Hoshtag
Hashtag Prediction
Table 5: Performance on different order of domains on DASSET.
Derpp O_cce
DAS O_Ace
Oars
Domain Order
A_Acc
F_Ace
A_Acc
F_Ace
ALAce
O_Acc
F_Ace
PATTAACL:A:PPPOOEEEE PUOENNEPPPMDD.CCMMMM CAMBRA
0,8345
0.8174
0.8239
0.8261
0.8164
0.8251
0.8356
0.8277
0.8341
REST:PHONE PUBMUD/ CAMUI ALL
0.8662
0.180
0,7830
CAMERA-PURMED: PHOTTE ATALL-A
0.8263
0,8023
0.8166
0.8273
0,8032
0.8127
0.8317
0,5099
0.8264
PHONE.ACL.PURMEDDRRRS
data. We set the prompt length to 50, the size of prompt components to 100, and the size of memory buffer to 300. As for the trade-off hyperparameters, we set A 1, X 0.01, and Àda to 0.01. During fine-tuning, the learning rate set to 3e-5 and batch size 16. We train on the downstream datasets for 15 epochs with an early stopping mechanism. Un- less otherwise stated, the same hyper-parameters are used in all experiments.
D Robustness on different orders
As several works (Yoon al., 2020; Evron al., 2022) suggest that the task order may significantly affect the performance CL. approaches, we also conduct experiments to test our robustness to dif- ferent task orders. We run the methods on several orders of the DAPset benchmark and list results in Table 5. Our method shows an average improve- ment of 0.99%. 1.22%, and 1.36% the three metries compared to DAS, demonstrating its bustness.


Reviews:
[{"review_id": "yhh8cxrTHi", "reviewer": "Reviewer_ke4a", "summary": "", "questions": "- What would be the next sets of experiments to conduct, given the current ones are focused on classification tasks?\n- Can you expand on the significance on the t-SNE plot? Since the model is fine tuned, it seems natural that the embeddings and hidden states would have changed. Please explain in more details what the domains and downstream tasks are with more examples. \n- The question of order of continual pretraining has always been on my mind, so more explanation would be helpful. \n- More explanation on prompt components and how to choose the hyperparameters would be supportive to the experiment details.\n- Although prompts are pretrained with the hypernetwork, can you provide more justification of why this is interpreted as p(output | input, domain)? \n- In ablation studies and the rest of the paper, there is a lack of confidence interval and the quantitative values are quite close.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments to improve our paper. We detail our response below point by point. Please kindly let us know if our response addresses the questions you had for this paper.\n\n##### Q1: The downstream application is currently limited to classification.  What would be the next sets?\n\n> - For continual pre-training, the type of downstream tasks is **not restricted** by the algorithm. Our proposed method, HPrompt-CPT, is also **applicable to generation** downstream applications. However, due to the limitations of the backbone model we used (Roberta) and the datasets collected, we did not test the algorithm's performance on generation tasks in the paper.\n>\n> - To further demonstrate the advancement of HPrompt-CPT, we extended the backbone model to the GPT2 and **ran on two continual learning generation task datasets**, namely DecaNLP and Reviews. For detailed experimental settings and results, please refer to our response to Reviewer 2's Question 1. Our method achieves **state-of-the-art performance** and exhibits outstanding generalization ability.\n\n\n\n##### Q2: The performance of the experiment in the third domain seems to be also dropping like the other methods\n\n> - Hprompt-CPT prevents and improves the model generalization to unseen domains through three mechanisms:\n>   - Using **hypernetwork** to map input to embedding, allowing the model to **generalize past sample knowledge** to unknown samples.\n>   - **Protect the general knowledge** in language model through Agreement loss.\n>   - **Increasing the hypothesis space** of the language model output through Disagreement loss, enhancing generalization.\n>\n> - When learning domain 0, our **mechanisms face challenges** in preventing performance degradation on the conflicting domain 3. In this scenario, the first and third mechanisms are unable to generalize the knowledge to a conflicting future tasks, and the protection provided by the second mechanism is insufficient to counteract the shifts caused by adapting to the conflicting task.\n>\n> - But in the **subsequent process**, NCL and DAS algorithms continue to experience performance degradation on domain 3 since they lack designs specifically aimed at preserving generalization. In contrast, our algorithm, through the application of the three mechanisms, can gradually generalize the previous learned knowledge to new tasks and decrease the shift in language model, thereby **avoiding further performance decline**.\n>\n> - In summary, our capacity to generalize on unseen domains strengthens as we learn more domains, thereby achieving the objective of anytime fine-tuning.\n\n\n##### Q3: Can you expand on the significance on the t-SNE plot?\n\n> The t-SNE plot is used to demonstrate that HPrompt-CPT achieves the learning and preservation of domain knowledge during continual pretraining, which also well transferred to downstream tasks without relying on domain IDs.\n>\n> - For prompts, the hypernetwork is expected to learn the correlation between domains and allocate corresponding prompts. The bottom right corner in Fig.5 illustrates this point. Our hypernetwork **assigns similar embeddings to similar domains** (e.g., overlapping embeddings for corpora C2, C3, and C5 from the same paper dataset) while also achieving **differentiation for dissimilar domains** (e.g., distant embeddings for C1 (restaurant) and C5 (bio-chem)). Furthermore, the hypernetwork is expected to identify the domain to which the coming downstream task belongs and allocate the corresponding embeddings. The results in Fig.5 show that the **prompt distributions generated for D2, D3, and D5 have relative positions consistent with C2, C3, and C5**, validating our hypothesis.\n>\n> - For hidden states, effective models can generate distinguishable hidden states for downstream task based on pre-trained domain information. The results in the top right corner of Fig.5 indicate that our model successfully possesses this capability. For instance, the model assigns **overlapping representations to similar tasks** D2 and D3 (belonging to ACL and AI, respectively), while providing **effective differentiation for unrelated tasks** D1 (restaurant) and D5 (biology).\n\n\n\n##### Q4: More explanation to the question of order of continual pretraining.\n\n> We have illustrated in Appendix D and Table 5 the results of the order of continual pre-training. We conclude that\n> \n> - **among all orders**, the order with highly similar adjacent tasks leads to smaller changes in model parameters, thereby contributing a flatter loss surface and greater performance. This aligns with the works of [3, 4].\n> \n> - **among all methods**, Derpp and the proposed HPrompt-CPT demonstrate greater robustness against various orders. Though Derpp requires experience replay to promote similarity between adjacent tasks, HPrompt-CPT minimizes changes in the backbone and improves robustness via the hypernetwork-based and domain-specific prompts. \n    \n\n##### Q5: More explanation on prompt components?\n\n> - Prompt components, **analogous to a set of basis vectors**, are a set of prompt embeddings that are randomly initialized, trainable and optimized through gradient descent. \n> \n> - Compared to learning a prompt embedding for each domain, introducing prompt components and **instead learning the weights that combine these components** offer the following two benefits:\n>   \n>   - **greater generalization to future domains** as long as the prompt components are as mutually exclusive as possible. For example, a prompt embedding directly optimized for the domain of \"ACL papers\" does not directly apply to the domain of \"AI papers\" due to the domain difference; however, one of the prompt components learned on \"ACL papers\", e.g., \"deep learning\", can be combined with another component of \"statistics\" to generalize to the domain of \"AI papers\".\n>   \n>   - **less forgetting** as the number of parameters to learn for each domain reduces from the full size of a prompt embedding to the number of prompt components. \n> \n> - Empirically, we **validate the above benefits in Table 3**, where adapting prompt components yields better results compared to directly generating prompts through linear projection.\n\n\n##### Q6: How to choose hyperparameters?\n\n> Hyperparameters at different parts of the algorithm have different impacts on the adaptability, generalization, and forgetting of modes. We list the choices of several important hyperparameters below.\n>\n> - **Learning rate**: It plays a crucial role in the fine-tuning stage, where a small $lr=1e-5$ failed to fit the task for unseen domains, leading to an $O\\_Acc$ of 0.8197, compared to $lr=3e-5$, which achieved 0.8297. Conversely, a large $lr=1e-3$ resulted in reduced performance in learned domains, resulting in an $F\\_Acc$ of 0.8216 compared to 0.8306 for $lr=3e-5$.\n>\n> - **Prompt token length**: We find that a larger prompt length (80) may impede generalization for downstream tasks, resulting in a drop in $A\\_Acc$ from 0.8356 to 0.8232.\n>\n> - **Weight for agreement and disagreement losses**: A higher agreement loss demonstrates improved performance in mitigating forgetting (0.0101 -> 0.0030), but it leads to poorer generalization (0.8274 -> 0.8198) due to alignment. On the other hand, a higher disagreement loss shows better performance in alleviating forgetting (0.0032 -> -0.0016) and improves generalization (0.8245 -> 0.8289), but it results in poorer adaptability (0.8325 -> 0.8238) due to exclusiveness.\n\n\n\n##### Q7: Why the prompted language model is interpreted as p(output | input, domain)?\n\n> - First, it is intutive to have the interpretation of the prompted language model as p(output | input, **prompt**), which follows the original work of GPT2 [2];\n> \n> - Second, **enforcing each prompt to be highly correlated to a domain** results in the interpretation of p(output | input, **domain**). Note that we maximize such correlation between a prompt and a domain via the agreement loss. The agreement loss constrains the backbone from making changes under previous prompts, which maximally enforces the hypernetwork to generate prompts that are strongly associated with domains. Otherwise, the current domain with a prompt misaligned to previous domains will be prevented from learning without full adaptation.\n\n##### Q8: There is a lack of confidence interval and the quantitative values are quite close.\n\n> - The confidence intervals for most of the ablation experiment results are **smaller than 0.003**.\n> - For Table 2, the improvement contributed by each component is generally **above 0.004**, indicating determinism. Although the absolute value is small, it is considered a **significant improvement** in continual pre-training. For instance, the method [1] in ICLR2023 achieved a final improvement of 0.011.\n> - For Table 3, the values between several different loss metrics are indeed close. We have already mentioned in the paper that these metrics exhibit minimal differences, and we **chose a relatively more stable one**.\n\n\n[1] Zixuan Ke, et al. \"Continual Pre-training of Language Models\" ICLR 2023.\n\n[2] Alec Radford, et al. \"Language Models are Unsupervised Multitask Learners\" 2018.\n\n[3] Sen Lin, et al. “Theory on Forgetting and Generalization of Continual Learning” ICML 2023.\n\n[4] Samuel J. Bell, et al. “The Effect of Task Ordering in Continual Learning” Arxiv 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation of the paper is very useful, since pushing the boundary of downstream tasks adaptability, generalization, and avoiding catastrophic forgetting is crucial to real life application and research\n- There is a fairly comprehensive experiment done against the several SOTAs in the realm of continual pretraining done. \n- The empirical results show promising improvement upon the weakness of current methods.", "reasons_to_reject": "- As acknowledged by the authors, the downstream application is currently limited to classification. The experiment although is comprehensive in terms of approaches, on the dataset it is not too diverse. \n- The performance of the experiment in the third domain seems to be also dropping like the other methods, so it's not too strongly supporting the argument in the perspective of generalization. More studies and focus can be put on this.", "questions_to_authors": "- What would be the next sets of experiments to conduct, given the current ones are focused on classification tasks?\n- Can you expand on the significance on the t-SNE plot? Since the model is fine tuned, it seems natural that the embeddings and hidden states would have changed. Please explain in more details what the domains and downstream tasks are with more examples. \n- The question of order of continual pretraining has always been on my mind, so more explanation would be helpful. \n- More explanation on prompt components and how to choose the hyperparameters would be supportive to the experiment details.\n- Although prompts are pretrained with the hypernetwork, can you provide more justification of why this is interpreted as p(output | input, domain)? \n- In ablation studies and the rest of the paper, there is a lack of confidence interval and the quantitative values are quite close.", "paper_topic_and_main_contributions": "The main problem that the paper addresses is existing continual pretraining methods do not perform well in terms of generalization and adaptability, while they work well against catastrophic forgetting. The main contributions of the paper are\n- design an architecture to use a hypernetwork to generate domain specific prompts to guide the continual pretraining.\n- introduced agreement and disagreement losses to aid the training, and conducted ablation studies for these components.\n- the designed architecture achieved better results in terms of generalization, adaptability, and can do well against forgetting, comparing to existing SOTAs.\n- conducted a large scale experiment to cover 7 methods of continual pretraining on two datasets", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bpii3iMucD", "reviewer": "Reviewer_YTnf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments to improve our paper. We detail our response below point by point. Please kindly let us know if our response addresses the questions you had for this paper.\n\n##### Q1: The experiments on more benchmarks would strengthen the claims.\n\n> - To further demonstrate the advancement of HPrompt-CPT, we extended this algorithm to **the GPT2 model** and conducted experiments on **two continual learning generation task datasets**, DecaNLP and Reviews, used in [1, 2].\n>\n> - The **brief information** for these two datasets is as follows: 1) DecaNLP consists of four domains: Question answering, Semantic Parsing, Sentiment role labeling, and Zero-shot Relation Extraction. The reviews dataset includes five domains: Yahoo, Yelp, Amazon, AGnews, and DBPedia. 2) For pre-training, we remove the label for most of the data and use it as a domain corpus. For downstream task evaluation, we select the labeled samples from the remaining data as the corresponding downstream dataset. 3) The unlabeled corpus size: DecaNLP ≤ 8M tokens, Reviews ≤ 16M tokens. The downstream task size: DecaNLP = 1500 samples, Reviews = 250 samples.\n>\n> - We compared our methods NCL, EWC, and DERPP, and the results are as follows:\n>\n>    |       | DecaNLP |         |         |  Reviews |         |         |\n>    |-------|---------|---------|---------|---------|---------|---------|\n>    |       | A_Acc   | O_Acc   | F_Acc   | A_Acc   | O_Acc   | F_Acc   |\n>    | NCL   | 57.92 | 66.93 | 59.23 | 60.66 | 78.75 | 59.56 |\n>    | Ewc   | 56.62 | 66.09 | 56.40 | 59.05   | 78.36 | 58.97  |\n>    | Derpp | 56.85 | 65.78 | 57.09 | 61.36 | 79.35 | 61.93 |\n>    | Ours  | **61.83** | **69.61** | **61.88** | **63.66** | **82.03** | **63.76** |\n> \n>   - Our algorithm achieved **state-of-the-art** performance on both datasets, and its performance in terms of **generalization** is remarkable. The performance drop of Derpp on the first dataset can be attributed to the **sample imbalance issue**, which resulted in the buffer stored by Derpp being unrepresentative and thus leading to subpar results.\n\n##### Q2: The hypernetwork prompt module adds complexity.\n\n> - We present a table below comparing the **training cost** and **performance** of several algorithms:\n>\n>    |           | DAPset  |        |        |                         |\n>    |-----------|---------|--------|--------|-------------------------|\n>    |           | A_Acc   | O_Acc  | F_Acc  | Time cost [sec / batch] |\n>    | NCL       | 0.8298  | 0.8189 | 0.8198 | **0.217**                   |\n>    | Derpp     | 0.8245  | 0.8174 | 0.8239 | 0.430                   |\n>    | DAS       | 0.8221  | 0.8164 | 0.8251 | 0.685                   |\n>    | Ours(Lin) | 0.8332  | **0.8279** | 0.8331 | 0.601                   |\n>    | Ours      | **0.8356**  | 0.8277 | **0.8341** | 0.730                   |\n>\n> - The results indicate that our method indeed introduces more time overhead compared to simple algorithms like NCL and Derpp.  However, the time overhead is **comparable to das** [3], a method proposed in ICLR 2023, while we achieve **significant improvement** in performance.\n> - We also experimented with a simpler structure for the hypernetwork by using a linear layer. This approach reduced the time cost while maintaining similar performance, providing **flexibility in choosing the network** based on different training requirements.\n\n\n\n##### Q3: The empirical analysis of existing methods is limited.\n\n> - **Naive continual learning serves as a strong baseline for continual pre-training,** which outperforms EWC, LWF, and Coda-Prompt. This may be because 1) traditional continual learning methods **struggle** with large amounts of task data and self-supervised objectives. 2) Self-supervised tasks are naturally well-suited for continual learning as they provide a **smoother loss path**.\n>\n> - **Baselines excel at preventing forgetting but struggle with adaptability.** The forgetting metrics, $A\\_Acc - F\\_ACC$, for the baselines are mostly negative, indicating that forgetting is not the main reason for the low final accuracy. In contrast, the adaptation to the learned domain is the **key weakness**. Parameter-efficient tuning methods such as One-Adapter, One-Prompt, and CoDA-Prompt result in low $A\\_Acc$, as it is difficult for the small trainable portion of the model to incorporate a significant amount of domain knowledge. Similarly, methods that include regularization terms on trainable model size like EWC and DAS also face challenges in learning domain knowledge. Otherwise, Derpp and LwF show relatively better adaptability, as they both employ knowledge distillation, suggesting it could be a better choice.\n>\n> - **Most methods sacrifice generalization to unseen domains**. In the DAPset, the majority of baselines exhibit lower $O\\_Acc$ compared to the initial roberta. EWC, which primarily prioritizes the importance of learned knowledge, shows a significant drop in performance, while LwF protects the output of the initial model and achieves the best generalization among the baselines. Replay-based methods also result in low generalization due to their tendency to overfit on the memory examples.\n\n##### Q4: Its effectiveness on other continual learning scenarios remains unclear.\n\n> - **Continual pre-training** entails learning from streaming unlabeled corpora during the pre-training phase, with the purpose of enhancing the generalization capabilities for all coming downstream tasks. \n>\n> - It **differs greatly** from continual fine-tuning, and we have summarized their **characteristics in the table** provided in response to Reviewer 4's Question 1. These two scenarios represent distinct lines of research within the community. Researchers [1, 2, 3] have designed methods specifically tailored to each scenario and **do not explore the effectiveness** of these methods in both scenarios. \n>\n> - Our method focus on an interesting problem called **anytime fine-tuning** in continual pre-training, which holds significant value for real-world development. And we provide comprehensive analysis of our method in continual pre-training to demonstrate its advancement.\n\n\n\n[1] Fan-Keng Sun, et al. “LAMOL: LAnguage MOdeling for Lifelong Language Learning” ICLR 2020.\n\n[2] Yingxiu Zhao, et al. “Semi-Supervised Lifelong Language Learning” EMNLP 2022.\n\n[3] Zixuan Ke, et al. \"Continual Pre-training of Language Models\" ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "· The proposed anytime fine-tuning evaluation protocol provides a more comprehensive view of continual pre-training models.\n· The hypernetwork prompt module and agreement/disagreement losses are novel approaches that aim to balance the three important aspects of continual pre-training.\n· The experiments demonstrate the effectiveness of the proposed HPrompt-CPT approach.", "reasons_to_reject": "· The experiments are performed on only two datasets, more benchmarks would strengthen the claims.\n· The hypernetwork prompt module adds complexity, simpler approaches could be explored first.\n· The empirical analysis of existing methods is limited, a more thorough comparison is needed.\n· The approach is focused on continual pre-training, its effectiveness on other continual learning scenarios remains unclear.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a continual pre-training method for language models called HPrompt-CPT. It uses a hypernetwork to generate domain-specific prompts during continual pre-training. This approach aims to balance adaptation, generalization, and mitigate forgetting for anytime fine-tuning effectiveness.\nThe main contributions are:\n\nThey propose a anytime fine-tuning evaluation protocol for continual pre-training and show that existing methods struggle with adaptability and generalization.\n\nThey propose the HPrompt-CPT approach with a hypernetwork prompt module and agreement/disagreement losses. The hypernetwork generates prompts that capture domain knowledge while generalizing across domains. The losses aim to preserve generalization while mitigating forgetting.\n\nThey perform experiments on two datasets and show that HPrompt-CPT achieves better adaptation, generalization and less forgetting, outperforming baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IpWA639lT5", "reviewer": "Reviewer_Q6RZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments to improve our paper. We detail our response below point by point. Please kindly let us know if our response addresses the questions you had for this paper.\n\n##### Q1: It's not obvious how the proposed method prevents forgetting.\n\n> - In our method, we employ three mechanisms to address forgetting, as follows:\n>\n>   - Hypernetwork prompt: It **learns domain-specific knowledge through the hypernetwork** and minimizes updates in the backbone model, thereby reducing the overwrite of previously learned knowledge. It mitigates forgetting ($A\\_Acc$ - $F\\_Acc$) from 1.00% to 0.70%.\n>\n>   - Disagreement loss: It **promotes the exclusiveness** of generated hidden states for the current domain, minimizing interference to established knowledge in the LM. It mitigates forgetting from 0.70% to 0.55%.\n>\n>   - Agreement loss: It **aligns the predictions** of previous and current language models prompted by random prompts to preserve the generic and learned domain knowledge. It mitigates forgetting from 0.55% to 0.14%.\n\n\n##### Q2: The design of the experiments for downstream task evaluation is not clear.\n\n> - In the anytime fine-tuning setting, the downstream task evaluation phase **does not involve providing domain ID** for fine-tuning and testing. This is more in line with the real-world scenarios, where it is difficult to determine the domain of each incoming downstream task, especially for unseen domains. \n>\n> - In our HPrompt-CPT algorithm, during the downstream task evaluation phase, we utilize the trained **hypernetwork** to map the samples to prompt embeddings, which generalizes the pre-trained domain knowledge to downstream task **without the need of domain-id**. The visualized distribution in Fig.5 verifies the alignment of the generated prompts for unlabeled corpus and downstream task within a specific domain. Moreover, the hypernetwork undergoes updates during the fine-tuning phase to enhance downstream adaptation.\n>\n> - We explain this process in line 325 of the paper and will refine the text in subsequent revisions to improve clarity.\n\n##### Q3: Code is not provided\n\n> - We guarantee that the code will be made available to the readers. In the camera-ready version of the paper, we will include a link to the code repository.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written.\n2. The paper suggests an important evaluation protocol that has been overlooked in previous literature\n3. The proposed method shows strong performance.\n4. The advantage of prompt generation via hyper-network is reasonable and intuitive.", "reasons_to_reject": "1. It's not obvious how the proposed method prevents forgetting. \n2. The design of the experiments for downstream task evaluation is not clear. For instance, it's not specified whether all prompts are fine-tuned for downstream tasks or if the domain of the downstream task is known beforehand.\n3. Code is not provided", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a novel approach for continual pre-training of language models and introduce an evaluation protocol. Unlike the previous continual pre-training methods that mainly focus on the performance of past and current domains, the proposed evaluation protocol, called anytime fine-tuning, evaluates the model performance not only on past and current domains but also on unseen future domains for LM's generalization ability. The authors utilize hyper-network for prompt generation and introduce agreement and disagreement losses for adaptability and generalization, respectively. The proposed method is compared against standard CL methods and recent continual domain adaptation methods, and shows strong performance in the anytime fine-tuning protocol.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zVjKXJLiQx", "reviewer": "Reviewer_JHzV", "summary": "", "questions": "1. How are the prompt components generated?\n2. Line 360: How random prompt can simulate the condition to active generic or learned domain knowledge?\n3. What is the rationale to use KL-Divergence instead of other metrics?\n4. Formula 3: to minimize KL-Divergence, why does it define loss as the opposite value of KL-Divergence?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments on this paper. You may find our response below for your major and minor concerns. We would appreciate it if you could let us know if you have any further concerns.\n\n##### Q1: Relationship between anytime fine-tuning and the common practice to test on seen, seeing, and unseen datasets\n\n> We would like to humbly clarify that the proposed **anytime fine-tuning for evaluating continual pre-training** is not equivalent to **the common practice of testing on seen/seeing/unseen datasets under continual fine-tuning [5, 6]**.\n> \n> - First, continual pre-training and continual fine-tuning fall into two distinct categories of continual learning scenarios in language models, whose differences are summarized in the following table.\n>   \n>   |                        | Training                               | Evaluation                                                 | Goal                                   |\n>   | ---------------------- | -------------------------------------- | ---------------------------------------------------------- | -------------------------------------- |\n>   | Continual fine-tuning  | Streaming **downstream labeled** tasks | Direct test accuracy on seen/seeing/unseen tasks           | One model serving all downstream tasks |\n>   | Continual pre-training | Streaming **unlabeled** corpora        | Test accuracy **after fine-tuning** on domain-related tasks | Well-generalized initialization        |\n> - Though it is common in continual fine-tuning to evaluate unseen tasks, the current evaluation for continual pre-training assumes the greater generalization ability of an LM via continual updates and focuses on fine-tuning in seen/seeing datasets. Our empirical results in Figure 2, however, signify the unexpected generalization performance drop of a continually pre-trained LM in unseen tasks. **This empirical finding, to our best knowledge, is novel and motivates our first proposal of evaluating a continually pre-trained model in unseen tasks**.\n\n##### Q2: The hypernetwork and generation of previous LM increases time complexity\n\n> - We present a table in response to Reviewer 2's Question 2, comparing the **training cost** and **performance** of several algorithms. Our training complexity is **comparable to das** [3], the previous sota method for continual pretraining proposed in ICLR 2023, while we achieve a significant improvement.\n> - Additionally, we have conducted experiments with a simpler hypernetwork structure using a linear layer. This modification decreases the time cost while maintaining similar performance, providing **flexibility in selecting** the network according to different training requirements.\n> - For the cost from running current and previous LMs multiple times, we can adopt a **preprocessing** approach to calculate and save the historical representations of the samples before the forward pass. Furthermore, our agreement and disagreement loss can operate independently. In time-sensitive scenarios, we can customize a suitable algorithm by **removing specific losses**.\n\n##### Q3: Its generalizability on other LM is unclear. And experiments were done on only 2 datasets.\n\n> - We extended the algorithm to the **GPT2 model** and ran on **two continual learning generation task datasets**, namely DecaNLP and Reviews. For detailed experimental settings and results, please refer to our response to Reviewer 2's Question 1. Our method achieves **state-of-the-art** performance and exhibits outstanding generalization ability.\n\n##### Q4: The design of prompt learning limits its usage with other light-weight fine-tuning methods.\n\n> - Prompt learning, which involves concatenating prompt embeddings with input data, can be considered as an augmentation to the data. Consequently, other lightweight fine-tuning methods can be applied on top of the prompted input without any negative effects.\n>\n> - For instance, during the fine-tuning phase, we can add adapter layers in the language model while keeping the hypernetwork and backbone model frozen.  These adapter layers will acquire task-specific knowledge with the assistance of a pre-trained hypernetwork that transfers domain knowledge through the prompt. The fixed hypernetwork is sufficiently generalized for downstream tasks, achieving a final accuracy of 0.8326 (0.8341 for full fine-tuning).\n\n##### Q5: How are the prompt components generated?\n\n> - Prompt components, **analogous to a set of basis vectors**, are a set of prompt embeddings that are randomly initialized, trainable and optimized through gradient descent. The well-trained prompt components are supposed to offer **greater generalization to future domains** as long as the prompt components are as mutually exclusive as possible. For example, a prompt embedding directly optimized for the domain of \"ACL papers\" does not directly apply to the domain of \"AI papers\" due to the domain difference; however, one of the prompt components learned on \"ACL papers\", e.g., \"deep learning\", can be combined with another component of \"statistics\" to generalize to the domain of \"AI papers\".\n\n\n##### Q6: How random prompt can simulate the condition to active generic or learned domain knowledge?\n\n> - The oracle generic and previous knowledge lies in the pre-training data, which usually remains inaccessible. We seek an approach that simulates the generic knowledge, which is expected to be **irrelevant to the current domain**. Since randomly chosen prompts hardly have the chance to share the knowledge with the current domain, they can be used to simulate the generic and previous knowledge. This idea has also been leveraged in previous works [7].\n\n##### Q7: What is the rationale to use KL-Divergence instead of other metrics?  And why use the opposite value of KL-Divergenc?\n\n> - KL divergence distance is a common choice for knowledge distillation and was first introduced by Hinton [1]. It has been widely used in the field of continual learning [2, 3, 4]. Compared to loss functions like MSE, minimizing KL divergence is more sensitive to small differences and has a more stable gradient. In our work, we treat feature vectors as discrete representations of probability distributions and use the constraint of KL divergence to prevent the model forgetting, following [3, 4].\n>\n> - The minus sign in Formula 3 is a typo. Thanks for pointing it out.\n\n##### Q8: There are many small mistakes.\n\n> - Thanks for pointing out the above-mentioned mistakes and typos in our paper. We will carefully polish our paper again following your great feedback.\n\n[1] Geoffrey Hinton, et al. \"Continual Pre-training of Language Models\" Arxiv 2015.\n\n[2] Xisen Jin, et al. \"Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora\" NAACL 2022.\n\n[3] Zixuan Ke, et al. \"Continual Pre-training of Language Models\" ICLR 2023.\n\n[4] Wuyang Chen, et al. \"Lifelong Language Pretraining with Distribution-Specialized Experts\" ICML 2023.\n\n[5] Xisen Jin, et al. \"Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning\" EMNLP 2021.\n\n[6] Thomas Scialom, et al. \"Fine-tuned Language Models are Continual Learners\" EMNLP 2022.\n\n[7] Kevin Meng, et al. \"Mass-Editing Memory in a Transformer\" ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper takes into account 3 key capabilities of continual pre-trained models: adaptability, generalizability and forgetting.\n2. The experiments covers a wide range of existing fine-tuning methods", "reasons_to_reject": "1. The paper claims to propose an evaluation protocol named anytime fine-tuning. However, this is not something new. It's a common practice to test on seen, seeing and unseen datasets;\n2. This paper introduces Hnet-prompt module (including a 6-layer Transformer) to generate prompts and run both current and previous language models (LM) multiple times to get the losses which largely increases the computation costs;\n3. This method are only tested with Roberta. Its generalizability on other LM is unclear;\n4. This method is only targeting prompt learning based light-weight fine-tuning which limits its usage with other light-weight fine-tuning methods;\n5. Experiments were done on only 2 datasets including one own-built dataset, which hinders the representativity of experiment results.", "questions_to_authors": "1. How are the prompt components generated?\n2. Line 360: How random prompt can simulate the condition to active generic or learned domain knowledge?\n3. What is the rationale to use KL-Divergence instead of other metrics?\n4. Formula 3: to minimize KL-Divergence, why does it define loss as the opposite value of KL-Divergence?", "paper_topic_and_main_contributions": "This paper proposes a prompt-guided continual pre-training method (HPrompt-CPT). It involves a network to generate prompts by proposed agreement and disagreement losses. The design takes into account: adaptability, generalizability and forgetting. Experiments show that it outperforms other fine-tuning (FT) method on all 3 aforementioned aspects", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Figure 1, it's recommended to give a formal definition of all the symbols used in that figure following the format D: ... ; C: ...; ...\n2. It's redundant to say (e.g., D1 in Fig 1) or (e.g., B1 in Fig 1) every time when using that symbol. Direct using D1 and B1 will be fine.\n3. Line 76: only for -> is only for\n4. Line 77: than before -> than that before\n5. Line 144: give the full name of CL when mentioning it for the first time\n6. Line 192: user Twitter -> user's tweet\n7. Line 433: why does it only use the final accuracy to assess the degree of catastrophic forgetting?\n8. Line 638: can not -> are not"}}]

Author rebuttal:

Meta-review: The authors consider a continual learning setting without labels: models are trained in an unsupervised fashion given a sequence of unlabelled task inputs.
Evaluation happens via the introduction of labels: i.e., at any time in this training sequence, the model may be fine-tuned for any of the previous, current, or future tasks.
The proposed hypernetwork method outperforms alternatives on two datasets.

Reviewers largely agreed that this work is sound: experiments are thorough, baselines reasonable, etc. There was some discussion about continual fine-tuning vs. continual pretraining which was resolved. Existing limitations include that only 2 datasets and RoBERTa are considered but in response the authors address this by adding GPT2 and new experiments. Only classification is considered.

I personally wonder how much this proposed setting truly differs in practice from the general pre-training setting other than that the unlabelled text is provided on-line in task-sized chunks --- this setting also seems to be an extension of the somewhat common "unsupervised domain adaptation" setting where a pretrained model undergoes additional unsupervised training on a new unlabelled corpus (but here, there are a sequence of new domains, instead of just one).

Overall, it seems like the authors have addressed several concrete shortcomings (by adding new experiments on different models/datasets). The proposed method works better than a reasonable set of baselines. But, the importance of this setting beyond existing pretraning/unsupervised domain adaptation setups (both of which are closely related) is a bit less clear, which may be causing some reviewers to be less excited about this particular setup.
Decision: Accept-Findings