Continual Pre-training of Language Models
Authors: Zixuan Ke
Yijia Shao
Haowei Lin
Tatsuya Konishi
Gyuhak Kim
Bing Liu
Venue: ICLR 2023 poster (2023)
Keywords: Continual learning
Domain-adaptive Pretraining
Post-training

Abstract: Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual pre-training of LMs, in particular, continual domain-adaptive pre-training (or continual DAP-training). Existing research has shown that further pre-training an LM using a domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes a novel method to continually DAP-train an LM with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end-task performances. The key novelty of our method is a soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances. Empirical evaluation demonstrates the effectiveness of the proposed method.

Published as a conference paper at ICLR 2023
CONTINUAL PRE-TRAININ OF LANGUAGE MODELS
Zixuan Ke't, Yijia Shao?,, Haowei Lin?; Tatsuya Konishi Gyuhak Kim', and Bing Liu 1
1 'Department of Computer Science. University of Illinois Chicago
2 Wangxuan Institute of Computer Technology, Peking University
3 KDDI Research
(zke4, gkim87, 1iub) @ edi
2 [ (sshooy,, wiwhawwei (pku. edu. on setoonsshiiddiireeeee
-p
ABSTRACT
Language models (LMs) have been instrumental for the rapid advance of natural language processing. This paper studies continual pre-training of LMs. in particular, continual domain- aanuiive pre-training (or continual DAP- DAAP-iaainnn). Existing research has shown that further pre-training LM using domain corpus to adapt the LM to the domain can improve the end-task performance in the domain. This paper proposes novel method to continually DrPii LM with sequence of unlabeled domain corpora to adapt the LM to these domains to improve their end- task performances. The key novelty of our method is soft-masking mechanism that directly controls the update to the LM. A novel proxy is also proposed to preserve the general knowledge in the original LM. Additionally, it contrasts the representations of the previously learned domain knowledge (including the general knowledge in the pre-trained LM) and the knowledge from the current full network to achieve knowledge integration. The method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-tast task performances. Empirical evaluation demonstrates the effectivvness of the proposed method!
INTRODUCTION
Pre-trained language models (LMss) like BERT (Devlin et al. 2019) and RoBERTa (Liu et al. 2019)
have significantly advanced NLP. Recently, LMS have also been used by many continual learning
(CL) systems to learn sequence of end-tasks incrementally (Ke et al. 202 Sun et al. 2020
Huang et al. 2021), which we call continual end-task learning. is also desirable to continually
pre-train LMs themselves. This includes (1) continual general pre- preatiainn which incrementally
updates the LM using the most recent data that has a similar distribution as the pre-training data, and (2) continual domain-adaptive pre-training, which further pre-trains a LM incrementally to adapt to sequence of domains. Note that LM editing (with or without continual learning) (Mitchell et al. 2022) that corrects mistakes learned in the LM is a special case of continual end-task learning (Kim et al. 2022) as each editing task or group of editing tasks learned together is basically a task in continual learning, which aims to perform the editings correctly without interfering with or forgetting the other knowledge already learned in the current L.M.
This paper focuses on continual domain domain-adaptive pre-training (or continual DAP-training) of LMs. It is known that DAP-Iraining] an LM (without continual learning) using large unlabeled domain corpus before end-task fine-tuning achieves better results (Gururangan et al. 2020 Xu et al. 2019) Ke et al. 2022b). This paper goes step further to continually learn to improve handle new or emerging domains or topics without forgetting the skills or knowledge learned in the past. This is important in the real world, where the data shifts constantly and new domains, events topics keep emerging (Ke et al. 2022b) and the LM needs to be updated to serve the users better.
LM's 's ability to
'Equal contribution
1 'The work was done when this author was visiting Bing Liu's group at University of Illinois at Chicago.
1 Corresponnaace author. Bing Liu <1 www.ubieeeceeee.
'The code is available at hhtps:ss//ss...... / /gttttb..b.. wwt.ub..com/utt wwt.comm/////// Cem//uc--ia Lic-- Lab/ Cont i intal LM
2 Depending different contexts or authors, DAP-training is also called post-training or pre-finetuning.

---

Published as a conference paper at ICLR 2023
We call this problem continual DAP-training Starting from pre-trained general LM (i.e., the LM has already been pre-trained on Do), we incrementally DAP-train a sequence of domain corpora D1, .2.D..... Once domain is trained, its data is no longer accessible. This is different from conventional continual learning (CL) where each task is an end-task. In the proposed continual DAP-training, each task is unlabeled domain corpus be learned. An end-task fine-tunes the continually DAP-trained LM to evaluate its performance. It is worth noting that Do is usually a broad or general domain (e.g., News). In practice, continually DAP-trained LM may be trained by individual users, institutions or mix of both who have one or more large corpora of some particular domains. In such cases, the raw data may not be shared, but the final LM can be shared by all.
There are multiple desiderata for a continual DAP-training system: (1) It should not suffer from catastrophic forgetting (CF), i.e., it should perform reasonably well on learned domains. This requires the system (a) to overcome CF for each new domain and (b) to overcome CF for the general language knowledge in the L.M. This is important because the knowledge learned from each domain alone will not be sufficient for good end-task performances. (2) It should encourage knowledge transfer (KT) across domains to achieve improved end-task performances. This requires the system to enable (a) forward transfer, learning new domain by leveraging the knowledge from previous domains, and (b) backwards transfer, gaining improved performance on previous domains after learning a relevant new domain. (3) It should work without requiring the domain-ID for each end-task fine-tuning.
None of the existing CL. methods can achieve all the above. This paper represents a step towards achieving them. The proposed method is called DAS (Continual -a-pre-trrining of LMs with Soft-masking). DAS proposes a novel soft-masking mechanism that computes the importance (a real number between 0 and 1) of units for general or domain knowledge and soft-mask them based on their importance values to control the backward gradient flow. In the forward pass, soft-masking not applied, which encourages KT across domains. It does not isolate any sub-network for any domain so that the knowledge in the full LM can be leveraged for end-task fine-tuming.
To apply this mechanism, DAS implements two functions: (1) Initialization, which computes the importance of units to the general knowledge in the LM without accessing the LM pre-training data (Dn). It applied on the pre-trained LM before the continual learning starts, and (2) continual learning, which DAP-trains each domain while preventing CF on the general and domain knowledge and encouraging cross-domain KT. In (1), is not obvious how to compute the importance without pre- pre-training data. DAS proposes novel proxy based robustness to compute the importance of units for the general knowledge. In (2), the soft-masking is directly applicable because we have the domain data and the importance can be computed based its gradient inspired by the pruning community (Li et al. 2021 Michel et al. 2019). Moreover, DAS contrasts the previously learned knowledge and the full (including both the learned domains and the current domain) knowledge to encourage the current domain representation to learn knowledge that is not already in the knowledge learned from previous domains and integrate it with the learned knowledge In end-task fine-tuning, DAS does not requires the domain-ID as all knowledge is accumulated into the DAP-trained LM.
In summary, this work makes the following contributions. (i) studies the new problem of continual DAP- PAP-training and discovers that the full LM is needed for a good continual DAP- method. The popular parameter-isolation approach to overcoming CF convention CL is unsuitable. (ii) It proposes novel soft-masking method to overcome CF and to encourage KTT, and constrative learning based method for knowledge integration. (iii) To preserve the general knowledge in the LM, novel proxy is also proposed. (iv) Experimental results demonstrate the effectiveness of DAS.
2 RELATED WORK
DAP-traiiing. AAP-training can be achieved by directly updating the LM (Xu et al.) 2019 Sun
ct al. 2019 Lee et al. 2020 Alsentzer et al. 2019 Gururangan et al. 2020 Chakrabarty et al.) 2019)
Ke et al. 2022b) or by training only a small set of additional parameters. For example, Pteiffer et al.
20200 Wang et al. (2200)): Ke et al. (2022022))) a b)c) trained adapters and Gu et al. (2021) trained a
prompt adapt a domain. While adapter and prompt could be effective, transfer knowledge among
3 For simplicity, we use the term units to mean both attention heads and neurons. "Contrasting the past domains and only the domain-specint knowledge gives poorer results (see Sec. it causes the types of knowledge to split rather than integrate.
4.2 as
2

---

Published as a conference paper at ICLR 2023
these additional modules is usually challenging and can be inaccurate. DAS belongs to the former family that directly updates the LM. This is very challenging for CL due to CF. To our knowledge, no existing system in this family is about CL.
Continual learning. Most CL methods were proposed to overcome CF: (1) Regularization meth- ods (Kirkpatrick et al., 2016: Seff et al. (017)) compute the importance of each parameter previous tasks and uses a regularizer to penalize the sum of changes. DAS is related to but also very different from EWC (Kirkpatrick attention heads or neurons based on their importance scores. This gives less forgetting (see the forgetting rate in Table 2) because even a small change to each parameter for neuron can give a large total change to the neuron's 's activation. (2) DAS directly controls the backward gradient flow on each neuron, which is more fineggraine and effective than the sum of changes of all parameters. Our experimental results confirm that EWC is significantly poorer than DAS (see Table (2) Replay methods retain (Rebuffi et al. 2017| Wang et al., (020b) or generate some data of old tasks (Shin) al. 2017 He & Jaeger 2018) and use them in learning new task; (3) parameter-isolation methods (Serra et al. 2018 Wortsman et al. 2020> allocate neurons and parameters or sub-networks for different tasks/domains and mask them in task learning. For continual DAP-training this means that end-tasks cannot use the general knowledge in the LM, which results in poor end-task performances.
al. 2016). (1) DAS does not control each weight,, but only
In NLP. CL has been used for slot filling (Shen et al., 2019), language learning (Li et al. 2019).
sentiment analysis (Ke et al. 2021a), topic modeling (Gupta et al. 2020). question answering (Greco
et al. 2019) and text classification (Sun al. 2020 Huuan et al. 2021 Chuang al., (2020). But
none is for DAP-training, Some recent CL papers concern LMs. The system in (Madotto et al.
2020 learns separate adapters for different domains and thus has no CF KT. DEMIX (Gururangan
ct al.
2021) initializes the new adapter with the closest old adapter. CPT (Ke et al. 2022a) and
ELLE QQn et al.] 2022) are most closely related DAS. However, CPT uses the parameter-isolation approach to learn and protect each task, which is weak (see Sec. 4.2). It also needs domain- ID in end-task fine-tuning. ELLE has to start from pre-training the LM itself rather than from a pre-trained LM like DAS. It also uses a large memory (1G per domain) to store the replay data (including the pre-training data) and expands the network for each domain. Neither is required in DAS. Jin et al. (2021) evaluated several existing CL techniques in similar setting as DAS and performed analyses on dealing with CF. However, no new technique was proposed in the paper.
Neural network pruning. Many parameters in network are redundant and can be pruned (Li et al. Lai 2021 Lai et al., 2021 Michel et al.) 2019 Voita et al.] (2019). Existing methods include discarding parameters with small absolute values (Han al. 2015]] Guo al. (016), accumulated gradient (Michel et al. 2019), and lottery ticket hypothesis (Brix et al. 2020). However, these methods are not directly applicable as need preserve not only individual domain knowledge but also the general knowledge in the LM. For general knowledge, since we do not have any pre-training data, a proxy is proposed based on robustness. For domain knowledge, we adopt pruning method but use the importance as soft-masks as we want to accumulate knowledge rather than to compress the LM.
Contrastive Learning. Contrastive learning (Chen et al., 2020: He et al., 2020) learns good rep- resentations by maximizing the similarity of positive pairs and minimizes that of negative pairs,
1
NV
e (sim( an )/r)
Ceentrast
log
N
(sim(q...) 1/7)
12
==
where N the batch size, T is temperature parameter, sim(.) is a similarity metric, and gn and gt are representations for positive pairs In, and x. DAS contrasts the learned knowledge from previous domains and the pre-trained LM (general knowledge) with the full knowledge (including both the previous domains and current domain knowledge) to achieve complementary effect.
3 PROPOSED DAS TECHNIQUE
Continual DAP-training in DAS is based on two main ideas: (1) preserving the important general language knowledge in the LM and the knowledge learned from previous domains to overcome CF by soft-masking units based on their importance, which also facilitates cross-task knowledge transfer (KT), and (2) encouraging the model to learn complementary representations of the current domain and previous domains to achieve knowledge integration. Figure gives overview of DAS.
3

---

Published as a conference paper at ICLR 2023
Infositation
Continual
18
knowledge importance
Domain Trining
Doroli Impertance Computation
Trandormet Lower
Tranfforme
Tranaformer
Laver
Layer!
Forward
Forward
Transformer
Transformeer
Laryer
Tramsformme
Logar
Backward
Backwordd
Backword
Figure 1: Illustration of DAS. The red cross indicates that the gradient is not used to update the Transformer but only to compute importance. (A) Initialization (Sec. 3.1 computes the importance of units for the general knowledge in the LM. (B) Domain Training (Sec. 3.2) trains new domain using the importance scores as soft-masks and contrasts the previously learned knowledge and the full knowledge. (C) Importance Computation (Sec. 3.3 computes the importance of the units for the current domain.
The whole learning consists of two main functions: (i) initialization and (ii) continual learning. (i) computes the importance of units to the general language knowledge in the LM. It is done before the continual learning starts. (ii) is for continual learning, which consists of two steps: (a) domain training and (b) importance computation. (a) takes the importance scores accumulated so far (including those to the general knowledge in the original LM and to the knowledge learned from previous domains) and the input data of the current domain to learn the domain and to achieve (1) and (2) above, while (b) computes the importance scores for the current domain for future use. The following sub-sections present each function and step in detail.
3.1 INITIALIZATION: COMPUTING IMPORTANCE OF UNITS TO THE GENERAL. KNOWLEDGE
This initialization function computes the importance of units (attention heads and neurons) in the Transformer for the general knowledge in the original LM. The key components of Transformer are multi-head attention layer, intermediate layer and output layer. Below, we use "layer" or to indicate any of these three layers because our method treats the three layers similarly.
Importance of units in a layer. It has been found that all units layer are important (Michel et al. 2019). We introduce a virtual parameter, 9t. for computing the importance of the units in layer We call these virtual parameters as each gl is initialized to 1. We only need the gradient on each parameter to compute the importance its corresponding unit, no update to any parameter.
6 gi 3 O
where o refers to the output of layer I (which can be any of the three layers mentioned above). The unit (a neuron
@ refers to element- wise multiplication, i.e., each variable :1.; in gi corresponding or attention head) in the layer. We adapt the gradient-based importance detection method in (Michel al. 2019) for our purpose. Given dataset D {(x,,, yn)) ] w of N samples (y,, is the class label of Dn, as (Michel et al. (019)) worked on supervised learning), the importance of neurons or heads in the layer estimated with gradient-basee proxy score
N aimp (In Vro))
1
(3)
dg:
1=1
where Limpe is task-specific loss function. Note the virtual parameter gi is initialized as all I's, and is not changed. This is because we need only its average gradient V (the term within || in Eq. 3 over all the data to compute the importance and will not the gradient to update the virtual parameter. In training (Sec. [Sec.B.2]aand and [ig] the virtual parameter can be discarded. The resulting I, is of the same size as gl, each entry corresponding to the importance of unit (a neurons attention head).
4

---

Published as a conference paper at ICLR 2023
Recall that the initialization function is to learn the importance of units to the general knowledge in the LM (denoted as ) Although Eq. ooffers a possible way, is not directly applicable. If we use the domain data at hand and employ the MLM loss as Limpe. V only gives the importance for the domain-specific knowledge. However, to compute the importance of units to the general knowledge in the LM (which is our goal), we need the original data used in pre-training the LM to compute the Lippt. In practice, such data is not accessible to users of the LM. Further, label is needed in Eq. but our domain corpus is unlabeled in DAP-training To address these issues, we propose praxy KI.-divergence loss (Corosy) to replace Limpt to learn units importance for the general knowledge.
Proxy KL-ddivegeence loss. We propose to model robustness as the proxy, i.e., we try detect units that are important for LM's robustness. Their gradients, V then indicate the robustness and the importance to the LM model. Our rationale as follows: If an 1/ (the importance of unit in layer has a high value, then it is important the LM's robustness because its change can cause the LM to change a lot. is thus an important unit. In contrast, if T,," is small, it a less important unit. To compute the robustness of the LM, we take subset of the current domain data {>:thh | (no label in DAP-training) and input t twice to the LM to obtain two representations of it and then compute the KL-diieegeece between them,
Limpt KL (L J. film(a?))))
where Ji' and fiM are the LM with different dropout masks. We don't 't need to add any additional dropouts to implement these two the Transformer already has dropout masks placed on fully- connected layers and attention probabilities. Thus, simply feeding the same input to the Transformer twice will get two representations with different dropout masks. Since dropout similar adding noise, the difference between the two representations can be regarded as the robustness of the LM.
3.2 TRAINING: LEARNING A NEW DOMAIN VIA SOFT-MASKING AND CONTRASTIVE LOss
Recall we want to preserve the learned knowledge in the LM during DAP-training using the accu- mulated importance r (5t-1) when we learn domain t, which includes both the importance for the
general knowledge r, (Sec. 3.1 and learned domain-specifie knowledge r(()) of each domain l (k: can be any domain in {1...t- -TF) that has been learned (Sec. 3.3 This is achieved by soft-masking the learning based on accumulated importance as follows]
Accumulating importance. We accumulate the importance after task L 1 was learned is done via element-wise max (EMax) as follows:
(S--1) EM (t-1) I (S1-22)))
(5)
where refers to the current task-ID and ((5(-2) refers to the previously accumulated importance at task /. 2. We do not need to save IP and all (7 (8) yol for Eq. 5 We only save the incrementally
accumulated importance after training of each task.
Soft-masking units. Given the accumulated importance / ((5t-1) of layer and the DAP-training loss CDAP-Mai (typically the MLM loss; we also propose an additional loss in Eq. 7). we constrain (or soft-mask) its corresponding gradient ( (V,) flow as follows,
V = (1
V
(6)
As mentioned in Sec. 3.1 we expand (by copying) the importance 1 St- to match the dimensions
of V, to apply it to all associated parameters. This is soft-masking as each element in r, r(St-1) is real number in (0, 1] (not binary (0. 1)), which gives the model the flexibility to adjust any unit.
'We subset save computation Sec. 4 we show that subset is sufficient to compute the importance of units for the general knowledge.
assume that the DAP-training domain be very large. In
Before training, we normalized the the importance values in each layer for domain & by making the importance scores for all units in the layer having mean of 0 and standard deviation of 1. To further facilitate sofi-masking, the normalized importance scores are rounded by a Tanh activation so that the values are in the interval of [0. To simplify the notation, still use I((*) represent the resulting importance.
5

---

Published as a conference paper at ICLR 2023
Unlabelde Domain Datasets
End-Task Task Classification Datasets
Source
Dataset
Size
Dutaset
Task
#Training
"Testing
#Classes
Yelp Restaurant
758MB
Restaurant
Aspect Sentiment Classification (ASC)
3,452
1,120
Reviews
Amazon Phone
724MB
Phone
Aspect Sentiment Classification (ASC)
Amazon Camera
319MB
Camera
Aspect Scutiment Classificatton (ASC)
230
626
2
ACL Papers
867MB
ACL
Citation Inteni Classification
421
Academic Papers
Al Papers
507MB
AI
Relation Classification
2.260
2,388
PubMed Papers
989MB
PubMed
Chemical protein Interaction Prediction
2,667
7,398
Table 1: Statistics of datasets for Draiiingn More details of their end-task supervised learning
datasets are given in Appendix A
We note that the above soft-masks only applied in the backward pass, but not in the forward pass, which encourages knowledge transfer as each domain training can leverage the knowledge learned from all past domains. To further encourage the model to learn a good representation from both the accumulated knowledge (I (((t1)) and the full knowledge (both accumulated and current domain knowledge), we introduce a contrastive learning method to encourage complementary representation.
Integrating the previously learned knowledge and the current domain knowledge. masking helps prevent forgetting the previously learned knowledge. We want to further encourage knowledge transfer by integrating the new and learned knowledge. We propose to contrast the previously learned knowledge and the full knowledge (both previously learned knowledge and the current domain knowledge). Note that the contrasting cannot do anything to the shared past knowledge as it is protected by soft-masks. Thus, it effectively pushes the current domain knowledge away to be complementary to the past knowledge. This is done based on the current domain data as follows.
Contrasting the learned and full knowledge. We denote the output of LM without any consideration of importance as ofulll, which refers to the full knowledge. We further denote the output of LM that multiplied by the importance (i.e., 1 [|((-1) @ 01) as o which refers the previously learned knowledge. We contrast the two by using الlll anchor and full with different dropouts as positive samples (dentoed as o dulll)) o is used as negative instances.
Formally, with full full+ and ore our contrastive loss is (sim(-) :) is the cosine similarity),
Simi full of!lt) /T
Ceontrast
(c log simm (olltt) 1/T + c sim(nill on ///))
(7)
n=1 1=1
Compared to Eq.[]] the second term is added in the denominator, i.e., representations in the previously learned knowledge as additional negative instances. Figure (B) shows a red arrow pointed from full
to itself, indicating the positive instances are from inputting twice. The dashed red arrow pointing to opee indicates the negative instances contrasting the full and previously learned knowledge.
Final Loss Function. The final DAP-training loss combines the Masked Language Model (MLM)
loss after applying the proposed roft-masking for the general knowledge (Sec.3.1 3.1 and the proposed contrastive loss (A is hyper-parametery
CAAP-tain = _MLM + ACcontratt
(8)
3.3 COMPUTE IMPORTANCE OF UNITS TO THE CURRENT DOMAIN
After training the new/current domain I, we learn the units importance by applying Eq. 3 for the domain. We do not need any proxy to compute Limpt as in Eq. 4 4bcaase we can directly use the current domain data. Specifically, we randomly sample a subset (a hyper-parameter) of the current domain data ( )) where xyدa is the input and yith is the masked token in MLM self- supervised loss. We can then casily compute the importance I by plugging LMLM into Cimpt in Eq.
3 The resulting ((()) will be used in the next task by accumulating with the previously accumulated importance (Eq.. (5) and soft-masking the learning (Eq. (6)
4 EXPERIMENTS
We use RoBERTa (Liu et al. 2019 as the LM. Following the standard evaluation setup (Lange et al. 2019) and, after a domain is trained, its training data is discarded. After all domains are incrementally learned, the final model is evaluated by fine-tuning the end-tasks all domains.
www.hhgggggggaag..cooooooo ww..o//ccco.......o..o.... roberta- base

---

Published as a conference paper at ICLR 2023
4.1 DATASETS AND BASELINES
Datasets: Table shows the statistics of the 6 unlabeled domain corpora for DAP-training and their
6 corresponding end-task classification datasets 8 3 of them are about reviews: Yelp Restaurant
(Xu et al., 2019). Amazon Phone (Ni et al.) 2019) Amazon Camera (N et al.) 2019); 3 of them are
academic papers: ACL Papers (Lo et al. 2020). Al Papers
Lo et al. 2020), and PubMed Papers
2
Their corresponding end-task classification datasets are
10 Restaurant [" Phone (Ding al. 2008
Hu & Liu 2004), Camera (Ding et al., 2008 Hu & Liu
2004), ACL (ACL-ARC in (Jurgens et al.
2018)), A7 (SCIERC in (Luan al.) 2018)), and PubMed (CHEMPORT in (Kringelum et al. (016
Baselines. We use 16 baselines, including non-continual learning (Non-CL.) and continual learning (CL) baselines. All CL baselines are originally for learning supervised data except DEMIX. We adapt them and replace their backbone with RoBERTa. Details of each baseline is given in Appendix B
Non-CL Baselines:
Each baseline here builds separate model for each task. (1) Pool. We pool the
data of all domains together and train only one model for all domains.
(2) RoBERTa (Liu et al. 2019))
(3) DAP-RoBERTa uses the existing DAP-training method (MLM) in (Gururangan et al. 2020) to posst-train each domain separately. (4)
uses RoBERTa for end-task fine-tuning without DAP- training.
DAP-Adapter adds adapter layers in Transformer for each domain for DAP-training (Jang et al.
2021 Madotto et al.) 2020 Houlsby et al. (2019). Only the added adapters are trainable. In end-task fine-tuning, both RoBER and the adapters are trainable. (5) DAP-Prompt is from (Lester et al.)
2021). In DAP-training, RoBERTa (the LM) is fixed and only the prompts are trained. In end-task fine-tuning, both the LM and the trained prompt are trainable.
CL Baselines: We use 2 naive baselines. which keep learning more domains with no mechanism to deal with CF or transfer. (6) NCL (Naive CL) continually DAP-trains the RoBERTa; and (7) NCL-Adapter continually DAP-trains a set of adapters (Houlsby et al. 2019).
8 baselines are CL systems: (8) DEMIX (Gururangan et al., 2021) adds a new adapter for each new
domain and initializes it with previous adapter nearest to the new domain; (9) BCL (Ke et al.
2021c) uses capsule networks. (10) CLASSIC (Ke et al. 2021b) uses contrastive learning. (11)
KD is knowledge distillation (Hinton et al. 2015). (12) EWC (Buzzega al.. 2020) is a popular regularization-basee method. (13) DER++ (Buzzega et al. 2020) is a replay method based on
knowledge distillation. 16.4K tokens are saved for each domain in the replay memory, which is the largest memory we can use for the system to run. (14) HAT (Serrà et al. (2188) is an effective parometer-isolation method. HAT applied to Transformer layers (i.e., self-attention, intermediate and output layers). (15) HAT-All is HAT variant that uses all features from the LM to do end-tasks (instead of only features from its domain sub-network as in HAT). (16) HAT-Adapter (Ke et al. 2021c) uses HAT within adapters. ELLE (Qin et al. 2022) is not included as we adapted it for our purpose by learning from RoBERTa, but it fails to converge.
4.2 RESULTS ANALYSIS AND ABLATION STUDY
Due to space limits, Implementation Details are given in Appendix C Table 2 reports the end-task fine-tuning results of all 15 systems on the 6 datasets. We can see that the proposed DAS outperforms all baselines on average and also achieve the best knowledge transfer (negative forgetting rate).
(1) DAS is slightly better than Pool on average. This may be because (a) some domains are quite different (e.g. camera reviews and ACL papers), which results in some negative transfer in Pool. (b) DAS can learn with the general and previous domain knowledge protected by soft-masks.
(2). DAS achieves both forgetting prevention and knowledge transfer. Those baselines (KD, EWC, DER++) focusing only on forgetting prevention give poorer performance as they sacrifice accuracy to avoid CF. Those baselines (BCL, CLASSIC and DEMIX) perform knowledge transfer achieve better results but still poorer than DAS. DEMIX has very weak transfer. BCL, which can avoid CF while
*e down- the PubMed due to its huge original size. In general, our datasets much smaller than those used in (Gururangan al. 2020) (which used more than 11GB of data for each domain). Our experiments showed that a smaller datasct is sufficient and more data does not help. It also requires less computing power.
httpss: / pwwmed. nobi mimm ni /cv/
10 Our results are different from those presented in Table of (Gururangan et al. 2020) hecause we observe very high variances due to very small test sets and thus enlarge the test set and reduce the training set slightly. httpps:/ //alts qeri org/ semeval 201 /
7

---

Published as a conference paper at ICLR 2023
Canegory
Domain
Model
MFT
Restaurans
Acc
MIP
ACL
Acc
MI1
Asc
PuhMed
MFT
MET
Acc
MEI
Average
Acc
MFI
Porget
Acc
Prol
$0.96
87.80
69.89
74.11
68.55
75.97
64.96
56.95
73:34
86.03
50.85
77.25
61.50
RoBERRI
79.81
87300
06.11
71:26
00:395
71.33
K2.75
86.008
12.38
76.32
57.113
13.54
79.27
Non CL
DAP RoBERIa
80.84
87.68
68.75
73:44
68.97
75.95
82.59
85.50
72.84
84.39
59.90
76.40
$0.89
DAP-Aclype
50.19
87.14
68.87
72:92
60.55
71.38
K2.71
85.35
71
83.62
59.22
74.60
70.62
DAP-Prompt
79,00
86.45
66.66
71.35
61.47
72.36
B4.17
86.53
73,09
85.5
90.35
74.98
80.03
68.39
12.87
73-71
$9.10
12,49
83.71
50.70
70.36
50.77
NCI-Addsper
80.13
87.05
67.30 30
72 30
57.71
69.87
83.92
85.56
72.07
83.70
50.71
74.05
79.48
0.15
DEMIX
79.99
87.12
68.45
72.73
63.35
72.86
07
82.42
71.73
86. 59
91.12
74.70
79.65
0.76
0.36
BCL
78.97
86.52
70.71
74.58
66 26
74.55
81,70
84.63
71 11999
85.06
90.51
75.78
80:45
-0.0
-0.19
CLASSIC
79.89
87.05
67.30
72:11
59,84
71.08
$4.02
56.22
09.53
86.93
91.25
74.63
79.59
0.44
0.25
CL
KD
78,05
K5.39
69.17
73.73
67.89
75.09
$2.12
$4.59
72.29
81.91
$3.69
75.17
$0.06
0.07
0.01
DAPamia
EWC
80.98
87.64
65.94
71.17
65.044
73.58
K2.32
85.13
71.43
K3.35
$0.14
74.34
70.68
0.02
-0.01
DER+
79.00
86.
67.20
72
63,96
73.54
81.22
85.61
72.58
87.10
91.47
75.51
80.30
2.36
1.53
HAT
76-42
85.
60.70
68 79
47.37
65.60
72.35
79.
69.97
74D04
65.14
66 50
75.65
-0.13
-0.29
HATT All
74.94
K3.95
$2.08
63.94
34.16
56.07
54.71
74.43
68.14
65.54
$1.44
59.93
71.35
3.25
1,83
HAT Adapter
79.29
86.70
68. 25
72.87
64.84
73.67
$1.44
81.56
71.61
82.37
59.27
74.63
79.78
0.23
0.15
DAS
30.34
87.16
69.36
74.01
70.M3
77.46
85.99
87,70
72.90
88.16
92.30
77.93
81.91
-1.09
-0.60
Table 2: End-task macro-F1 (MF1), accuracy and forgetting rate results for all domains after the continual DAP-training of all domains, except for CHEMPORT in the PubMed domain, for which we use micro-F1 following (Gururangan et al., 2020; Dery et al. 202T Beltagy et al.) 2019). The results are averages of 5 random seeds (the domain training order is as they appear in the first row).
Due to space limits, the results for different domain orders and the standard deviations are reported in Appendix D and Appendix F. respectively). Non-CL. baselines have no forgetting.
also achieving some transfer, is weaker than NCL. In general, CL baselines are all poorer than DAS as they don't have methods to encourage knowledge transfer or they have to rely on adapters.
(3). Directly learning the domains within the LM helps DAS achieve better results than adapter and prompt based methods. DAS is better than adapter-based systems (DAP-Adapter, NCL-Adapter and HAT-Adapter) and prompt-based system (DAP-Prompt). This is because adapters and prompts do not have sufficient trainable parameters, which are also randomly initialized and can be hard to train.
(4). Using the full LM to learn all tasks rather than using sub-networks (of HAT-based methods) makes DAS more effective. HAT performs poorly, indicating it is unsuitable for DAP-training as discussed in Sec. [] Even if we use all features (not only the feature from the corresponding sub-network), we still get poor results (HAT-All) as the features used in DAP- -APatraining (in an LM sub-network) are different from features used in end-task fine-tuning (features from the whole LM).
Knowledge transfer and forgetting avoidance. To see how the models fare on CF and knowledge transfer, we compare the forgetting rates (forget R.) (Liu et al. 2020). - Σx=1 Ak. - Ar, where A is the end-task accuracy right after its domain i is DAP-trained, and Ar, is the accuracy of the end-task of domain after DAP-training the last domain l. We average over all end-tasks except the last as the last domain has no forgetting. The higher the forgetting rate is, the more forgetting it has. Negative rates indicate positive knowledge transfer. Clearly, DAS has the strongest negative forgetting rate, indicating it does well on both forgetting prevention and knowledge transfer. NCL, NCL-AAapter, DEMIX, EWC, KD and DER++ all suffer from some forgetting. HAT has no forgetting but it cannot learn well. HAT and BCL have no forgetting but are weak in transfer.
Effectiveness of the proxy KI--diergence loss. We use proxy KL-divergence loss in the initializa- tion function (Sec. 3.1 to compute the importance of units for general knowledge. We are interested in how good the proxy is. We use two kinds of experiments to provide evidences.
(1) Comparing with a sample set of Do. In some cases, the continual DAP-training users may have the data Do that was used to pre-train the LM. Then we can just sample a subset from Do to compute the parameter importance to the general knowledge in the LM. However, since we do not have Do that was used to pre-train RoBERTa, we use the Wiki data (Merity et al., 2017) as the sample set of Do- We choose it as is a general dataset with a wide topic coverage and was used to pre-train an LM, and it has similar size as our domain data (around 700M). We conducted two experiments using the data: (a) DAS (Wi which uses MI.M as the loss in the initialization stage to compute the importance of units (to identify the general knowledge) just like any other domains in the importance
continual learning part, and (b) DAS (Wiki+KL), which uses KL-divergence as in the initialization stage just like the proposed proxy method. The results are given in Table 3
We can see that DAS (Wiki + KL) performs similarly to DAS but outperforms DAS (Wiki + MLM). This indicates that the proposed proxy KL-divergence is more effective. MLM actually adapts the LM to the Wikipedia data, which may not be sufficiently representative of the original data used in pre-training the LM. As result, it ends up identifying the knowledge that is suitable only for the
8

---

Published as a conference paper at ICLR 2023
Domain
Restaurant
ACL
Al
Poone
PuisMed
Camera
Average
Forget
Model
MFI
Aec
MF1
Acc
MF1
Ace
MF1
Aoc
MF1
MFI
Acc
MF1
Ace
MFI
Acc
DAS (wiki+KL)
81.25
87.89
70.89
74,87
69.68
76.86
85.98
87.78
72,03
86.69
91:44
77.75
81.81
-0.50
-0.27
DAS (Wiki-MLMM)
80.22
87. 12
68. 12
72.92
68.55
76.06
83,50
86.11
71.94
86.02
91.15
76.39
80.88
0.54
0.40
DAS
80.34
87.16
69,36
74.01
70.93
77.46
85,99
87.70
72.80
88.16
92.30
77.93
81.91
-1.09
-0.60
Table 3: Results for the Wiki dataset as the sample set of Do average of random seeds
Canegory
Domain
Midial
MF1
AT
PahMed MF1
Average
MFI
RoBERTa
79:81
87.00
66.11
71.26
601.9
71.85
83.75
86.11
72.35
78.92
87.03
73.64
79.27
Nas-CL
DAP RooBRTa
DAP-Acleper
80.84
80
$7.68
67.14
68.87
68.75
72.92
73.44
68.97
60:55
71. 38
75.95
82.59
82.71
85.50
65.35
72.84
71.68
83.62
24.39
89.90
89.23
76,40
74,00
79.62
80.89
DAP Prompe
79.00
56.45
66.66
71.35
61.47
72.35
86.17
86.53
73.09
35.52
90.38
74,95
80.03
DAS (randan))
79.79
68.34
73.02
08.62
76.17
84,92
87.02
72:75
85.92
91.15
76.72
81.15
0:45
0.26
conerat)
70.39
86.00
90.06
80.80
DAS instalizztion)))
69.92
74.50
86.97
DAP wain
DAS (domain specifiic)
68.35
18
67.82
75.73
96.98
$7.25
$6.77
77.00
DAS
80.34
87.36
69.36
74.01
70.93
77.46
87.70
72,80
58.16
V2.30
77.93
81.91
-1.09
Table 4: Ablation results - averages of 5 random seeds. See standard deviations in Appendix B
Wikipedia data. In contrast, the proposed proxy KL-divergence leverages the random dropout mask and measures the robustness, which is less related to a specific domain and thus reflects the (general) knowledge in the original LM better.
(2) Comparing general knowledge computed from different domain corpora. Here, we also provide some indirect evidences to show the effectiveness of the proxy method for computing the importance of units to the general knowledge in the LM. We conduct a separate non-CL experiment to compare the attention heads" importance score vectors after applying the proxy using the data from different domains 12 For each domain i, we compare its importance vector with the importance vector of every other domain, and then average the cosine similarities to get the value for domain i. We get 0.92 for Restaurant, the same 0.91 for ACL, AI, and Phone, 0.89 for PubMed and 0.92 for Camera. We see that different domains give similar importance values, which indirectly shows that our proxy can approximately identify the common general knowledge.
Ablation. We want to know if the proposed (1) initialization (Sec. 3.1 (2) soft-masking, and (3) contrastive learning are helpful. To answer (1), we conduct the ablation DAS (w/o initialization). where we remove the initialization and directly do the continual learning given no consideration to the general knowledge in the LMM To answer (2), we conduct the ablations (1) DAS (w/o softmask), where we remove the soft-masks, and only use contrastive learning based on Eq. [ the second term in the denominator removed); and (2) DAS (random) with randomly generated importance scores to do soft-masking and contrastive learning. To answer (3), we conduct two ablations: (i) DAS (w/o contrast) where we remove the contrastive loss and only soft-mask according to the importance; (ii) DAS (domain-specific) where we contrast domain-specific and learned knowledge (Sec. 3.2). shows that the full DAS is the best on average and for most domains, indicating that every
Table f component contributes. Additional observations are: (1) DAS's gain is partially from the preserved general knowledge. We can see DAS (w/o initialization) is poorer on average; (2) Soft-masking helps as DAS (w/o softmask) is poorer than DAS. This is reasonable because soft masking can preserve learned domains. Besides, our gradient-based mask is informative as DAS (random) is worse than DAS; (3) Contrastive learning is effective as DAS (w/o contrast) and DAS (domain-specifi are both poorer, indicating the contrastive learning in DAS can help learn good representations
5 CONCLUSION
This paper proposed a novel method DAS for the continual DAP-training of an LM. It has three key ideas: (1) Preserving the important previous knowledge by soft-masking units according to their importance to overcome CF and to facilitate knowledge transfer. (2) Using novel proxy to compute the importance of units to the general knowledge in the LM. (3) Learning complementary
representations for knowledge integration. A set of techniques is proposed to achieve them. Extensive experiments showed the effectiveness of DAS. The current approach involves two functions in learning. We will study how to combine them to further improve the results in the future.
We use attention heads instead of other units because they are arguably the most important component in Transformer (iichel al. 2019 Voita al. 2019 McCarley et al. 2019).

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The work of Zixuan Ke, Gyuhak Kim, and Bing Liu was supported in part by research contract from KDDI, a research contract from DARPA (HR001120C002)) and three NSF grants (IIS-1910424, IIS-1838770, and CNS-2225427).
REFERENCES
Emily Alsentzer, John R Murphy, Willie Boag. Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint Ne04:/904..322 2019.
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: pretrained language model for scientific text. In EMNLP-IJCNLP. 2019.
Christopher Brix, Parnia Bahar, and Hermann Ney. Successfully applying the stabilized lottery ticket hypothesis to the transformer architecture. In ACL, July 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe- rience for general continual learning: a strong, simple baseline. arXiv preprint arXiv:2004.07211, 2020.
Tuhin Chakrabarty, Christopher Hidey, and Kathleen McKeown. Imho fine-tuning improves claim detection. arXiv preprint drXiv:1905.07000 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Yung-Sung Chuang, Shang-Yr Yu Su, and Yun-Nung Chen. Lifelong language knowledge distillation. arXiv preprint arXiv:2010.02/23, 2020.
Lucio M Dery, Paul Michel, Ameet Talwalkar, and Graham Neubig. Should we be pre-training? an argument for end-task aware training alternative. arXiv preprint (09.::200900753:: 2021.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), NAACL-HLT, 2019.
Xiaowen Ding, Bing Liu, and Philip S Yu. A holistic lexicon- based approach to opinion mining. In Proceedings of the 2008 international conference on web search and data mining, 2008
Claudio Greco, Barbara Plank, Raquel Fernández, and Raffaella Bernardi. Psycholinguistics meets continual learning: Measuring catastrophic forgetting visual question answering. arXiv preprint arXiv:1906.04229, 2019.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint
2021.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. NIPS, 29, 2016.
Pankaj Gupta, Yatin Chaudhary, Thomas Runkler, and Hinrich Schuetze. Neural topic modeling with continual lifelong learning. In International Conference on Machine Learning, pp. 3907-3917. PMLR, 2020.
Suchin Gururangan, Ana Marasovie, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't it stop pretraining: Adapt language models to domains and tasks. In ACL, 2020.
Suchin Gururangan, Mike Lewis, Ani Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. arXiv preprint arXiv:2108.0003: 2021.
10

---

Published as a conference paper at ICLR 2023
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp.
2020.
Xu He and Herbert Jaeger. Overcoming catastrophie interference using conceptor-aided backpropa- gation. In ICLR, 2018.
Geoffrey Hinton, Oriol Vinyals. Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint urXiv:1503.02533 2(7), 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP, In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), JCML, 2019.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of ACM SIGKDD, 2004.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05489 2021.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stan- ley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models, 2021.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang- Ween Li, Xiaokai Wei, Andrew Amold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint arXiv:21 2021.
David Jurgens. Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. Mcasuring the evolution of scientific field through citation frames. TACL, 2018
Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowledge transfer in continual learning. Advances in Neural Information Processing Systems, 34, 2021a.
Zixuan Ke, Bing Liu, Hu Xu, and Lei Shu. Classic: Continual and contrastive learning of aspect sentiment classification tasks. In EMNLP, 2021b.
Zixuan Ke, Hu Xu, and Bing Liu. Adapting bert for continual learning of sequence of aspect sentiment classification tasks. In NAACL, pp. 4746-4755, 2021c.
Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. Continual training of language models for few-shot learning. In Empirical Methods in Natural Language Processing (EMNLP), 2022a.
Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu. Adapting a language model while preserving its general knowledge, In Empirical Methods in Natural Language Processing (EMNLP), 2022b.
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. theoretical study on sooving continual learning. In Advances in Neural Information Processing Systems, 2022.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis
drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, 2016.
Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. Chemprot-3.0: global chemical biology diseases mapping. Database, 2016, 2016.
11

---

Published as a conference paper at ICLR 2023
Cheng-I Jeff Lai, Yang Zhang, Alexander H Liu, Shiyu Chang, Yi-Lun Liao. Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, and Jim Glass. Parp: Prune, adjust and re-prune for self-supervised speech recognition. NeurIPS, 34, 2021.
Matthias De Lange, Rahaf Aljundi, Mare Masana, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. CoRR, 2019.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-12) 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), EMNLP, 2021.
Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. Differentiable subset pruning of transformer heads. Transactions of the Association for Compatational Linguistics, 9:1442-1459, 2021.
Yuanpeng Li, Liang Zhao, Kenneth Church, and Mohamed Elhoseiny. Compositional language
continual learning. In International Conference on Learning Representations, 2019.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. In CVPR, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraiming approach. CoRR, 2019.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. S2ORC: the semantic scholar open research corpus. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), ACL, 2020.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't iichi Tsujii (eds.), 2018.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, and Zhiguang Wang. Continual learning in task-oriented dialogue systems. arXiv preprint arXiv:2012.1 /55004 2020.
JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of bert-based question answering model. arXiv preprint
2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast model editing scale. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), EMNLP, pp. 188-197. Association for Computational Linguistics, 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destruetive task composition for transfer learning. In EACL, 2020.
12

---

Published as a conference paper at ICLR 2023
Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. ELLE: efficient lifelong pre are-training for emerging data. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 2789-28181 10. Association for Computational Linguistics, 2022. URL
https://accaattar antho wtw.g.....gg.... F: Indings-acl 220
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
Ari Seff, Alex Beatson, Daniel Suo, and Han Liu. Continual learning in generative adversarial nets. CoRR, abs/17055000 77055, 2017.
Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In ICML, 2018.
Yilin Shen, Xiangyu Zeng, and Hongxia Jin, A progressive model to enable continual learning for semantic slot filling. In EMNLP-IJCNLP, November 2019.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NIPS, 2017.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to tune bert for text classification? In China national conference on Chinese computational linguisties, pp. 194 206. Springer, 2019.
Fan-Keng Sun, Cheng-Hao Ho, and Hung- Yi Lee. Lamol: Language modeling is all you need for lifelong language learning. In ICLR, 2020. URL https::/ net/ forummrddmmm
SkGxen4YDS
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09411 09418, 2019.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, al. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.09 01800, 2020a.
Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime Carbonell. Efficient meta lifelong- learning with limited memory. In EMNLP, 2020b.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), NeurIPS, 2020.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), NAACL-HLT, 2019.
13

---

Published as a conference paper at ICLR 2023
A DATASETS DETAILS
Table Din in the main paper has already showed the number of examples in each dataset. Here we provide additional details about the types of end-tasks.
(1) (Phone, Camera and Restaurant) Aspect Sentiment Classification (ASC) is defined as follows: given an aspect or product feature (e.g., picture quality in a camera review) and a review sentence containing the aspect in domain product category (e.g.. camera), classify if the sentence expresses a positive, negative, or neutral (no opinion) sentiment or polarity about the aspect (for the Phone and Camera datasets, there are only negative and positive polarities in the data).
(2) (ACL) Citation intent classification defined as follows: given a citing sentence (a sentence contains citation), classify if the sentence expresses citation function among "background", "motivation", "uses", "extension" and "comparison or contrast future""
(3) (AI) Relation classification is defined as follows: given within within-sentence word sequence span containing a pair of entities, classify if the span expresses a relation among "feature of", "conjunction", "evaluate for", "hyponym of"", "used for", "part of"" and "compare".
(4) (PubMed) Chemical-protein interaction classification defined as follows: given a span containing pair of chemical and protein, classify if the span expresses chemical-protein interaction among "downregulator", "substrate", "Indirect-upreeulato""" "indirect-dowwrrgulatto "agnonist", "activator", "product of", "agonist-activator"" "inhibitor", "upreeglator", "substrate product of", "agonist-inhiiitor"" inhibitor" and "antagonist".
B BASELINE DETAILS
Non-Continual Learning Baselines: Each of these baselines builds separate model for each task independently. It thus has no knowledge transfer or CF.
(1) Non-DAP-traiiing (RoBERTa) Liu al. (2019) uses the original RoBERTa for the end-task fine-tuning without any DAP-training, This is the only one without any DAP-training, All the following baselines use the masked language model loss (MLM) for DAP-training.
(2) DAP-training using masked language model loss (DAP-RoBERTa) is the existing DAP-training method in Gururangan et al. (2020). To our knowledge, the existing DAP-training systems are all based on the MLM loss.
(3) DAP-training using adapter-tuning Madotto et al. (2020); Houlsby et al. (2019) adds small adapter layers between layers of Transformer for DAP-training We follow the adapter design in Madotto et al. Houulsb et al. (2019): An adapter simply a 2 layers of fully connected network. During DAP-iniining the Transformer is fixed, only the added adapters trainable. The bottleneck size (adapter size) is set to 128. During end-task fine-tuning, both RoBERTa and the adapters are trainable to ensure fair comparison.
(4) DAP-training using prompt-tuning Lester et al. (2021) adds a sequence of real vector tokens (called virtual tokens prompt tokens) to the end of the original sequence. In DAP-training RoBERTa (the LM) is fixed and only the prompt tokens are trained. In end-task fine-tuning, both LM and the trained prompt trainable. We initialize 100 tokens and set the learning rate of the prompt token to 0.3 in DAP-training following the setting in Lester et al. 2021))
Continual Learning (CL) Baselines.
(5) Naive continual learning (NCL) is a naive extension of Gururangan et al. 2020), which continu- allyincrementally DAP-trains the LM to learn all domains using MLM loss with mechanism to deal with CF.
(6) Continual learning with adapter (NCL-Adapter Houlsby et al. 2019) is similar the adapter based system. The only difference is that the same set of adapters is shared across all domains, rather than using adapter for each new domain.
(7) DEMIX (DEMIX) Gururangan et al. (202T) is a recent model to adapt pre-trained LM with new domains. It adds new adapter once domain arrives (network expansion is needed) and
14

---

Published as a conference paper at ICLR 2023
initializes the new adapter with the parameters of the previous trained adapter nearest to the new domain data. They use the perplexity on held-out sample to choose the most probable adapter. For fair comparison, we use the same size as {-*,th) ) the held-out samples.
(8) Hard attention to overcome forgetting (HAT-Adapter) Ke et al. (202lc) is derived from HAT (Serr al. (2018), the state-of-the-arr
-sooatttonno based method with almost forgetting. However, HAT requires task id information in end-task fine-tuning (DAS works in domain- agnostic manner and does not need the task id information; see Sec. [ HAT also needs train an
addition task embedding to mask each layer of the network which makes the DAP-training inefficient.
(9) Continual learning plugin with capsule(BCI.) Ke et al. (202cc) is continual learning model that can avoid forgetting and encourage knowledge transfer. It is similar to NCL-Adapter, The difference is that its adapters consist of two modules, one is capsule network (a new capsule is added once new domain arrives) to encourage transfer and the other one is similar to HAT to avoid forgetting. Similar to HAT, task/domain information needed in end-task fine-tuning. We replace the backbone network from BERT with RoBERTa for fair comparison.
(10) Continual learning plugin with contrastive transfer (CLASSIC) Ke et al. (021b) a continual learning model that can avoid forgetting and encourage knowledge transfer via contrasting loss. It is similar HAT. but additional contrastive loss are used for distillation, knowledge transfer and supervised contrast. Since DAS is working unsupervised data, we remove the supervised contrastive loss. Similar to HAT, task information needed end-task fine-tuning. We replace the backbone network from BERT with RoBERTa for fair comparison.
(11) Knowledge distillation (KD) Hinton et al. (2015) minimizes the representational deviation between the learned representation and the new representation in DAP- training. We compute the KL divergence between the representations (the output before the masked language model prediction head) of each token of the previous P-rairdd L.M and current LM the distillation loss.
(12) EWC Buzzega et al. (2020) is a popular regularization-baan based method which adopts elastic weights consolidation add L2 regularization parameter changes.
(13) DER++ Buzzega et al. (020 is a recent replay method using distillation to regularize the new task training. We store 16.4K tokens for each learned domain the memory, which is the largest memory we can use for the system to run.
(14) HAT Serrà al. (2018) used in the Transformer layers (including self-attention, interme- diate and output layers) rather than the added adapter layers. Additional task embedding and task information for end-task fine-tuning are needed.
C IMPLEMENTATION DETAILS
Architecture. We adopt ROBERTABASE as our backbone LM. A masked language model head is applied for DAP-training. The end-task fine-tuning of RoBERTa follows the standard practice. For the three ASC tasks (see Table we adopt the ASC formulation in Xn et al. (001)). where the aspect (e.g., "sound"") and review sentence (e.g.. "The sound is great") are concatenated via 
.
Hyperparameters. Unless otherwise stated, the same hyper- parameters are used all experiments. The maximum input length is set to 164 which is sufficient for all datasets. Adam optimizer is used for both DAP-training and end-task fine-tuning. The max sequence length also set to 164.
DAP-training. The learning rate is set to le-4 and batch size to 256. We train 2.5K steps for each domain, roughly full pass through the domain data, following Gururangan et al. (2020); Xu et al.
(2019). The subset of data (xrz")) } for computing Limpt to determine head importance in Sees. 3.1 and
3.3 is set to 1.64 Million tokens, which is sufficient in our experiments. in Eq. 8 is set to and 7 in
Eq. is set to 0.05.
End-task fine-tuning. The learning rate is set to le-5 and batch size to 16. We train on end-task fine-tuning datasets for 5 epochs for Restaurant; 10 epochs for ACL, Al and PubMed; and 15 epochs for Phone and Camera. We simply take the results for the last epoch, assuming no validation sets. We empirically found that the above number of epochs gives stable and convergence results.
15

---

Published as a conference paper at ICLR 2023
NCL
KD
DAS
Random Sequence Order
Acc
Restaurant - ACL + Phone PubMed Camera
76.36
80.77
75.17
80.06
77.93
Phone AI + PubMed + Camera - Restaurant ACL
76.06
80.69
75.49
80.39
76,90
81.10
PubMed - camera - ACT, + Restaurant - AT Phone
76.49
80.84
75.80
80.51
76.86
81.09
Camera ACL Phone Restaurant FubMed
76.28
80.83
74.67
77.52
81.65
AI - PubMed - Camera Phone + ACL + Restaurant
76.17
80.68
75.32
80.45
78.18
82.10
Table 5: DAS performance averaged over all domains after the final DAP-trained (averaged over 5 random seeds).
Cangary
Domann
Mivell
MFI
Restame
ACc
MP1
ALL.
Ade
MF1
A1
Acc
MP1
Plore
Acc
PalMec
MIFI
MFI
Camca
Arc
MF1
Arcrags
Ace
Fool
200010
200031
30.01Y
20000
30.057
20,0330
-0.012
25:003
3200343
20,0127
10:011
100193
100946
10,047
-6/P10
-0.1144
10040S
10.019 99
-00006
200118
200017
=0.165
DAP
#0,302
=00005
200142
200238
-0,088
$0,005
200065
20.0100
=0.0132
DAP-Jhump
10,060
100003
L0,055
LO.00S
10,001
100034
.00125
6.006
10,K0S
100245
10.035
10.049
0.0019
NCT.
230064
200005
200016
200064
300006
300029
200155
-0.004
10,003
30040
200247
200016
230005
NCL-Adagter
2000900
20.000
20.0033
20.005
20.035
20.055
20,195
20,0244
10.006
100312
20.0152
20.017
20,005
DEMIX
20.005
20,029
200115
20,004
20.076
100018
-00331
-0.0225
10,000
100099
20,0011
20.021
23,004
IKL.
10,000
2000/9
2000/90
-0.004
40043
300779
-60191
-6.0130
10/00A
10070
100154
-01887
200005
CLAASSIC
200337
20017L
20/227
20,034
20013T
-00124
2000
230352
20.0125
20.0022
20.041
DAP-inis
10.0161
100035
L00176
00000A
1000
100339
0,0140
10/00/
L0GTX
100434
10053
DER
10:001
100612
100155
1O000N
10M55
100500
00402
0.007
10/00
100362
100215
10035
10085
HAT
=0,0122
200055
=00071
=0,30
20030
200026
-0034
-0,074
10/038
20 10.0K2
20028
20.023
23,055
HATA-
10.0140
100043
10:355
10091
10055
18:0003
=0,0000
200003
=010186
30
300036
-00000
200054
2000999
230000
Table 6: Standard deviations of the corresponding metrics of the proposed DAS model and the
baselines
D DAP-TRAINING IN DIFFERENT ORDERS
Table 2 the main paper reported the results the order Restaurant - ACL + + Phone PubMed Camera. We now look at how the order affects the results. Due the computation intensive nature of DAP-training, we only report the best baseline (NCL) and DAS results with different domain orders. Table shoows NCL and DAS's results of different orders. We can DAS is always better than NCL, demonstrating the effectiveness DAS.
E STANDARD DEVIATIONS
Table [Jreports the standard deviations of the corresponding results in Table (in the main paper) of DAS and the considered baselines over runs with random seeds. We can see the results of DAS are stable. Some baselines (e.g., RoBERTa in AI, DAP- RoB-RoaERTa in Camera) can have quite large standard deviations.
Table reports the standard deviations of the corresponding results in Table DAS and the considered baselines over runs with random seeds. We can see the results of DAS and its variants are stable.
4 (in the main paper) of
Coegory
Domate
Roszerve
ACL.
Phone
PalMed
Cinrn
Average
100206
10015
02001
Noa
400004
20,0142
40.009
500551
=00284
#00265
20.08L
=00065
=30110
1.0094
-00132
=000007
DAP Prongi
40,000
+00035
44 20,834
20.008
404301
=00124
#90136
200067
20001
200043
230138
=0.065
200019
DMS (andem))
10000
100055
103010
100002
200M
20012
1000
20013
20037
2001
20.007
C3.
DAS owio commant
40.000 01
100055
44 40,8300
400063
4003US
-00154
+00371
100316
-00m
1000
100396
-00114
-00075
DAP-
DAS (ato sormenk)
10006:
1010280
400121
400085
10019
100113
10/045
400175
-0000
15033
10018
-0014
-0.009
iniidiization)
20.075
20,0354
30.048
200134
200078
200135
20010
200188
20040
200361
-0,00
10,090
100263
10.19
10.005
108143
-6.086
100160
10013
-60007
100000
106194
4.0009
-0/000
Table 7: Standard deviations the corresponding metrics of the proposed DAS model and the ablation
16


Reviews:
[{"review_id": "bsBI0hJ0IR", "reviewer": "Reviewer_dAfE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Continual Post-training of LMs with Soft-masking (CPS) that adapts the pre-trained language models on different domains in a continual-learning way. CPS is designed such the language model can maximally preserve the knowledge learned in the general domain and previously-seen knowledge-specific domain. This is done by calculating the importance of submodules in the transformer models and using the importance score to calculate the soft-mask. The soft-mask is used to mask out gradients such that important units for the previous domains will not be changed too much. The experiment results of continual post-training on six different domains show that the proposed method, CPS, outperforms other continual learning and non-continual learning baselines.", "summary_of_the_review": "Update after discussing with the authors\n===\nI think this paper provides a solid and convincing method for language model post-training. While there may be minor issues with the writing, the method itself is good. I think this paper should be accepted.\n\n-------------\nI think the paper requires significant improvements, including experiment setting and writing. I do not think the paper is ready to be accepted to ICLR.", "strength_and_weaknesses": "Update after discussing with the authors\n===\n\nAfter all the supplementary experiments added during the discussion period, I find this paper and the proposed method to be solid and convincing.\nI highly appreciate the efforts done by the authors and I find this process to be very beneficial.\n**All the weaknesses are perfectly addressed.** \nI am thus raising the score from 3 to 8.\n\n-------------\n\nStrengths\n===\n1. Propose a method that shows empirical performance gain in language model post-training\n2. The paper contains a lot of baseline methods for comparison.\n\nWeaknesses\n===\n1. The intuitions of the proposed method are not very convincing. These include the loss function in the initialization and continual learning, and the importance computation, which are the core of CPS.\n   - I do not see why we need to assume \"general domain datasets are not accessible to users of the LM\". Those general domain corpora should be very easy to obtain in realistic, and that is why they are used for self-supervised pre-training.\n   - I cannot agree with why using the domain-specific dataset and the loss function in Equation (4) is for learning general domain knowledge.\n   - I do not understand why contrastive learning is used here. Why would we want the representation of full knowledge and previously learned knowledge to be pushed away in the representation space, and how does this connected with knowledge \"integration\"?\n\n2. Some previous works and baselines should be investigated, this includes\n   - \"Jin et al. (2021) compared several CL methods for continual post-training\", as stated in Section 2. In fact, the scenario discussed in this paper is highly overlapped with Jin et al. (2021). In this case, the methods mentioned by Jin et al. (2021) should be compared. However, this paper misses a comparison between most knowledge distillation-based methods in Jin et al. (2021), which are exactly the methods that achieve better performance. There are 5 different KD methods in Jin et al. (2021) but there is only one KD method in this paper.\n    -  I also think LoRA [1] should be used as a baseline. \n    - Why is AdapterFusion not compared here? One can train domain-specific adapters using unlabeled data (MLM), and use the domain-specific downstream task dataset to learn the parameter of AdapterFusion and the downstream task's parameter.\n\n3. It is unclear whether the soft-masking based on importance is really very critical to the proposed method.\n    - I am curious whether CPS is successful because when updating the model, the soft-masking is calculated based on the importance, or it is just because we are suppressing the update of some proportion of the weights, and where the weights whose gradient update is soft-masked is not important. Gating the gradient during backpropagation sounds like some kind of regularization, and I would like to know how the regularization is placed is really important. To be specific, I would like to see an ablation study that constructs a random importance score $I_{l}^{(k)}$ that samples from a fixed probability distribution on $[0,1]$, and see whether the performance degrades or not.\n\n4. The paper is not very clear, including:\n    - What does $f_{LM}(o_l^{prev})$ mean? What does \"plug it into the whole language model\" means here? Is this simply using $I_{l}^{(\\leq t-1)}$ in the transformer model?\n    - How is NCL (Adapter) done? I understand that this is described in the appendix, but I find that most experiment details related to the baseline methods are placed in the appendix, making the paper's main content not very well self-contained. An additional question is why not add a new adapter in a new domain as in [2]? This is like DEMIX but different initialization method.\n    - The intuitions for the methods are not very convincing, making me find those parts confusing and not very clear.\n    - How is the model fine-tuned on the end tasks in CL baselines? Full model fine-tuning with a classifier head or using parameter-efficient fine-tuning methods such as adapter or LoRA? Does fine-tuning methods change the conclusion of what CL method is better?\n\n5. Lack of comparison when the learning order of domains is permuted. In Table 4, only the performance of CPS and NCL are compared, but this Table should have compared other CL methods.\n\n[1] Hu, Edward J., et al. \"LoRA: Low-Rank Adaptation of Large Language Models.\" International Conference on Learning Representations. 2021.\n[2] Jang, Joel, et al. \"Towards Continual Knowledge Learning of Language Models.\" International Conference on Learning Representations. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "3GGotfSa8T", "reviewer": "Reviewer_ApDZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of sequentially pretraining a neural language model on a sequence of domain specific training sets whilst preserving the original pretaining and avoiding catastrophic forgetting.  The basic idea is to maintain a soft mask over critical neurons (in this case the attention heads) such that backward error propagation is scaled down for paths which encode most of the existing knowledge.  The mask weights are learned by introducing a temporary gating function and using the normalised gate weights as the mask.  Mask weights are updated for each new dataset by taking the max of the mask for the previous iteration and the current.  In addition the normal LM Mask loss is augmented by a contrastive loss designed to encourage the LM to acquire knowledge from the new data that is not already in the knowledge extracted from previous domains.  In the experimental evaluation, the proposed system is extensively tested against a number of existing continual learning paradigms with impressive results.  Over a sequence of 6 datasets, the forgetting rate for the proposed system is negative showing that it not only prevented forgetting but it also facilitated knowledge transfer from one domain to another.  Ablation tests show that each component of the proposed system makes a positive contribution. ", "summary_of_the_review": "Overall this is a very nice paper which introduces a novel approach to a problem which has significant practical importance for real world applications.  The methods here are presented in the context of large LMs, but they should be applicable to a wide range of neural network systems. The work is clearly presented and the experimental work is thorough and convincing.", "strength_and_weaknesses": "The paper provides a principled approach to mitigating the forgetting problem when learning over time using sequential data sets from differing domains.  The solution is elegant and practical.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2mcOvufWYQs", "reviewer": "Reviewer_YGnc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper authors propose to improve continual post-training by the way of computing saliency of each unit or layer in transformer model. Average saliency is then used as a weight in the backward pass. Idea appears to be first time used in relation to self-supervised loss functions such as MLM as is the case here. Authors overcome multiple technical challenges in their system and the final experimental results are quite convincing. ", "summary_of_the_review": "Idea does make intuitive sense, but it is my opinion that paper at the current state is not of sufficient quality. \n\nAfter rebuttal:\n\nI find authors rebuttal and changes to be substantial enough to change my score from 5 -> 8. ", "strength_and_weaknesses": "Strengths: \n- Good empirical results in a very relevant task (self-supervised LM) with a number of different target domains. \n- Approach seems to make intuitive sense. \n\nWeaknesses: \n- Writeup is not finalized (see explanation in the clarity box)\n- Theoretical grounding is missing. Final result appears to be empirical searched solution to a practical problem, while idea itself is sensible. In my opinion, authors should try to answer theoretically that in some restricted case the proposed method would get the \"correct\" answer. Other way would be to show that it cannot completely fail. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2GQ03uNd2Zu", "reviewer": "Reviewer_8TT6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to continue adapting the language model to new domains without forgetting the previous domains. The main method first calculates the important neurons based on general language knowledge. Then it uses a KL divergence loss to to continued adaptation to other tasks and domains.", "summary_of_the_review": "The main issue with the paper is that the writing of the technical methods are very unclear, so it's difficult for me to understand or assess it. It might not be suitable for publishing at its current stage because the writing is not clear enough for people to replicate the method.", "strength_and_weaknesses": "1. the general ideal of training different important parameters for each domain, and preserves the previous domains is pretty interesting\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster