Title: Continual Learning with Global Prototypes: Beyond the Scope of Task Supervision
Venue: NeurIPS 2023
Accepted: no
Keywords: Continual Learning

Representation Drift

Global Prototypes

Adaptation Model

Self-Supervised Learning
Abstract: Continual learning aims to sequentially learn from different tasks without catastrophic forgetting. With no assumptions of task dependence, the knowledge learned from observed tasks may not align with that required for future tasks. This may result in models' disruptive updates for learning future tasks, causing abrupt changes to previously learned knowledge (e.g. representation drift) which induces catastrophic forgetting. To reduce such disruptive updates, we connect knowledge for observed and unknown tasks by learning task data representations properly related to a set of global prototypes, which have general-purpose connections and are shared across all tasks. We derive global prototypes and the corresponding objective for NLP tasks. For those tasks, the correlated global prototypes can be obtained from a model pre-trained by masked language modeling. And the data representations that have proper relationships to global prototypes can be learned by specific adaptations of the pre-trained model. We investigate existing adaptation models and propose a neighbor attention model which combines different advantages of existing models for our objective. Experiments show that models learning data representations well related to global prototypes can induce significantly less catastrophic forgetting, without memorizing information from past tasks. 

Abstract

Continual learning aims to sequentially learn from different tasks without catastrophic forgetting. With no assumptions of task dependence, the knowledge learned from observed tasks may not align with that required for future tasks. This may result in models' disruptive updates for learning future tasks, causing abrupt changes to previously learned knowledge (e.g. representation drift [7]) which induces catastrophic forgetting. To reduce such disruptive updates, we connect knowledge for observed and unknown tasks by learning task data representations properly related to a set of global prototypes, which have general-purpose connections and are shared across all tasks. We derive global prototypes and the corresponding objective for NLP tasks. For those tasks, the correlated global prototypes can be obtained from a model pre-trained by masked language modeling. And the data representations that have proper relationships to global prototypes can be learned by specific adaptations of the pre-trained model. We investigate existing adaptation models and propose a neighbor attention model which combines different advantages of existing models for our objective. Experiments show that models learning data representations well related to global prototypes can induce significantly less catastrophic forgetting, without memorizing information from past tasks.

1 Introduction

In the continual learning paradigm, models progressively learn a sequence of tasks. This paradigm supports real-world applications which face continuous streams of data and tasks [35, 20]. In practice, models may be under storage constraints to use a fixed structure and under privacy considerations that restrict revisiting of previous tasks' data. These introduce the challenge of catastrophic forgetting , where models lose knowledge of previously learned tasks after learning new tasks.

Most prior works address catastrophic forgetting using models that integrate the knowledge of the past and present tasks, i.e. the observed tasks. For example, regularization-based models constrain the deviation of current parameters from the previous ones [27, 56, 2, 29]; replay-based models memorize samples from past tasks and rehearse when learning present tasks [35, 9, 46, 26]. However, since there are no assumptions on task dependence in continual learning, models learned from a set of observed tasks may not contain knowledge needed for unknown future tasks [28, 16]. To learn such a future task, these models may have disruptive changes on previously learned knowledge (e.g. representation drift [7]), which still induces catastrophic forgetting. One way to reduce such disruptive updates is to make models consider potential knowledge connections to future tasks.

Our key idea is to build connections between observed and unknown tasks by connecting task-specific 33 data representations to a general-purpose representation base that is shared across all tasks. In many 34 domains, task-specific information about classes can be represented by specific combinations of 35 general units. For example, consider the data instance ' A boy in a red hooded top is smiling. The 36

Task 1

Task 2

Task3

2

2

6

1

3

3

5

Task 1

Task 2

Task 3

ConnectionsbetweenGlobalPrototypes

1

1

GlobalPrototype

5

boy is upset .' from ' contradiction ' class in an entailment classification task. The set { smiling , upset } 37 conveys the task-specific information of ' contradiction ' using the general (i.e. not task-specific) 38 semantics of the token units ' smiling ' and ' upset '. Based on this, we construct a general-purpose 39 representation base consisting a set of unit representations, which we call global prototypes . These 40 global prototypes are pre-learned to reflect semantic connections between them. Then we learn 41 data representations with appropriate task-specific connections to global prototypes. This allows 42 knowledge learned from observed tasks to connect to that of future tasks via the interconnection 43 of global prototypes, which is beyond the scope of task supervision from observed tasks. Our idea 44 mimics mechanism in the brain, a biological continual learning system [56] which rewires existing 45 neurons instead of creating new neurons to learn new tasks [17]. Here, global prototypes mimic 46 the neurons, and learning different connections between data representations and global prototypes 47 mimic the rewiring process. A figure of the idea is shown in Figure 1. 48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

We address two main challenges in realizing this idea: (1). constructing the representation base with correlated global prototypes; (2). learning data representations with task-specific connections to global prototypes. We investigate the above challenges for NLP tasks. For text, the non-contextual token representations are a natural choice for global prototypes, as any text information can be represented by sets of tokens from a fixed vocabulary. For the first challenge, we obtain the global prototypes from a pre-trained language model which learns semantic connections between tokens through self-supervised learning [11]. For the second challenge, we learn data representations by lightly adapting a pre-trained model to obtain task-specific connections to the global prototypes (Section 3). We investigate existing adaptation models with learnable projections ( Adapters [21]), learnable embeddings ( Prompt Tuning [30]), and propose a neighbor attention module combining properties of these two (Section 4). Results show that catastrophic forgetting can be significantly mitigated with models that can learn representations well connected to global prototypes. In addition, our neighbor attention model combines the advantages of existing adaptation models, and achieves superior performance in both vanilla and replay settings.

63

64

65

66

67

68

69

70

In conclusion, our contributions in this paper are:

We propose to learn task-specific information over a general-purpose base with global prototypes to address general task connections in continual learning. Specifically, we derive the construction of the base and the corresponding objective for NLP tasks.

We investigate existing adaptation models and propose a new neighbor attention model to learn data representations that have proper relationships to global prototypes.

We conduct experiments on different adaptation models and continual learning frameworks. Results show our model can significantly reduce forgetting without replay.

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

2 Related Work

Continual Learning Continual learning aims to sequentially learn new tasks while not forgetting previously learned tasks. Models for continual learning can be divided into three main categories: (1). regularization-based models which constrain the deviation of new parameters from the older ones [27, 56, 2, 29]; (2) replay-based models which reduce catastrophic forgetting by rehearsing on real or pseudo samples from previous tasks [35, 9] or generative models [46, 26]; (3). architecture-based models which learn evolving architectures for sequential tasks, with their capacities for each task carefully assigned [44, 53]. Most works above focus on knowledge based on observed tasks.

Some recent works show that knowledge only from observed task supervision is insufficient for continual learning. Knoblauch et al. [28] claim that optimal continual learning requires perfect memory and is NP-hard; Guo et al. [16] suggest preservingholistic information which may not benefit current task but help future tasks. With biased knowledge, models can have disruptive updating when learning a new task, causing problems like representation drift [7, 36, 24]. In this paper, we propose to consider knowledge beyond observed task supervision through a general-purpose base with pre-learned global prototypes. Unlike previous works [39, 4] which use a pre-defined classifier to help class separation [38], our global prototypes are pre-learned with general semantic connections and thus can build connections between tasks. Some works use self-supervised learning to learn more general representations for continual learning [36, 14, 15]. However, those representations do not necessarily connect to specific global prototypes, which is different from our objective.

Continual learning for NLP is an emerging area [3]. Liu et al. [32] introduce a sentence encoder with matrix conceptors; MBPA++ [10] uses an episodic memory with replay and local adaptation to mitigate catastrophic forgetting; LAMOL [49] learns to generate training samples for replay based on pre-trained knowledge; IDBR [23] disentangles hidden spaces to distinguish task-agnostic and task-specific information. Most of them require a memory of past task information, or converting data to a question-answering format along with text-to-text models [6, 42]. Our model does not have such restriction. There are also works [25] focusing on knowledge transfer in continual learning.

Adaptation Models In this work, we use adaptation models to learn representations connected to global prototypes. Prior works using pre-trained model with light adaptation for target tasks were originally aimed at parameter efficient tuning. Different methods include adding limited trainable parameters on the frozen transformer layer [21, 40, 18, 22]; or selectively updating existing parameters during training [41, 55]. Recent prompt tuning works [31, 30, 34] learn target tasks by trainable prompt embeddings for generalization purposes as well.

Most closely related work are adaptation models used for continual learning [51, 13, 43]. However, most use the models' parameter efficiency to construct progressive memory. Whether utilizing the pre-trained knowledge can help continual learning, why and how they help remain unexplored. Our approach is based on a fixed model without progressive memory of parameters. We use the adaptation model for our desiderata, which also provides a metric to interpret whether the model can benefit continual learning. We believe our work can inspire further utilization of adaptation models for CL.

3 Learning over Global Prototypes

We consider the following continual learning setting: the model learns from a sequence of tasks, where each task consists of data D τ = { ( x ( i ) τ , y ( i ) τ ) n τ i =1 } . x τ is the input data and y τ is the class label. A task identifier τ is provided at the training time. We consider two scenarios: task-incremental and class-incremental learning, where models are task-aware or task-agnostic at the inference time [37]. Without replay, we use the same training objective for both task-incremental and class-incremental learning while evaluating them in different ways.

Notation C τ represents a set of all classes for each task τ , C = [ C 1 , ...C τ , ... ] represents all classes for all tasks. For NLP tasks, V represents the set of tokens with global prototypes in the representation base. w i is the i -th column of a matrix w . Our main model consists of two components: an encoder f θ to generate representation f θ ( x τ ) for each data instance x τ ; and a classifier with matrix w γ ∈ R d ×| C | for class prediction, where d represents the dimension of data representations. At the inference time, the class label is predicted by arg max i ∈ C candidate f θ ( x τ ) · w i γ . For task-incremental inference we have C candidate = C τ , while for class-incremental inference we have C candidate = C 1: τ .

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

3.1 The Learning Objective

Classification Loss For a task τ , the typical classification objective is to minimize the cross-entropy loss L c ( x τ ; θ, γ ) over the training data for the task, as shown below:

After learning task τ , models have knowledge about data x 1: τ and class vectors w c ∈ C 1: τ γ from observed tasks 1 : τ . However, the knowledge may not align with that required for the unknown future task ( τ +1) . Specifically, after adjusting θ in task ( τ +1) , the alignment between w y τ γ learned from task τ and f θ ( x τ ) with adjusted θ may shift and degrade. In other words, to learn a future task, models may have disruptive updates which make abrupt changes to previously learned knowledge (e.g. representation drift [7]), and induce forgetting.

Prototype Loss To mitigate models' disruptive updates, we consider potential connections between observed and unknown tasks. The connection is built by learning task-specific data representations connected to a general-purpose representation base, which is shared across all tasks. The base consists of global token prototypes (denoted proto [ v ] for token v ) which reflect semantic connections between them. In particular, we want the data representation f θ ( x τ ) to be connected to the task-relevant global prototypes. Given a reference probability distribution p ( v | x τ , y τ ) which indicates the strength of connection between data representation and proto [ v ] , we push the data representations towards the prototypes in proportion to their reference probability. Formally, we define the prototype loss as:

In Eq.(2), the softmax is calculated over all global prototypes, i.e. proto [ v ] for any v ∈ V , regardless of task difference. Such calculation is task-agnostic, while the referenced probability p ( v | x τ , y τ ) gives task-specific guidance for representation learning. By doing this, Eq. (2) learns representations with task-specific connections to global prototypes. Since global prototypes are pre-learned to reflect semantic connections, representations learned by Eq. (2) can connect across tasks via connections of global prototypes. This can reduce abrupt representation change caused by disruptive updating.

The reference probability p ( v | x τ , y τ ) gives task-specific guidance for representation learning, where tokens with task-specific information of x τ should have high probabilities. Considering both taskspecific and holistic information of the data [16, 36], we set p ( v | x τ , y τ ) = 1 /r τ when v is one of data's r τ rationale tokens, i.e. tokens in the data that are essential for class prediction [8], otherwise p ( v | x τ , y τ ) = 0 . Using multiple rationale tokens as task-specific guidance brings extra benefits to the expressiveness of data representations and global prototypes. First, different data representations from the same class have different guidance. Second, a small number of global prototypes can convey rich information when connecting representations to different sets of global prototypes.

Learning Objective Based on the above analysis, our learning objective is to learn data representations that can correctly predict class labels (Eq. (1)); and properly connect to global prototypes (Eq. (2)). The optimal parameters θ ∗ , γ ∗ for task τ should satisfy the desiderata below:

̸

̸

where a τ > 0 is a threshold value of the prototype loss. Task performance desiderata (Eq. (3)) can be satisfied by optimization on classification loss in Eq. (1). In the rest of this section, we discuss two questions that are necessary for our desiderata: (1) . How to get the semantically connected global prototype proto [ v ] for Eq. (2)? (2) . How to get feasible models for the second desiderata in Eq. (4)?

3.2 Pre-trained Models for Prototypes and Data Representations

To get correlated global prototypes and learn data representations with reference to them, we utilize a model pre-trained by masked language modeling (MLM). The MLM objective is to predict masked

LinearProjection

We

1-入(H)

十

入(H)

1-

入

0

O

O

0

O

O

0

O

O

0

O

Neighbor-attention

Self-attention

Self-attention

Self-attention

Self-attention

KoVe

P

△

M

口

口

H

O

H

O

O

H

O

H

O

TransformerLayer

AdapterLayer

PromptTuningLayer

NeighborAttentionLayer

token v m from a masked input ˜ x , with the following loss: 164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

where f ϕ denotes the encoder for MLM, w δ consists of the token vector w v δ for each token v . The probability p ( v | ˜ x ) = 1 if v is the masked token v m , and 0 otherwise.

Pre-Trained Model for Global Prototypes The MLM objective learns token vectors w δ that reflect semantic connections between tokens, which suits our requirement for global prototypes. Therefore, we can get the global prototype proto [ v ] as the v -th token vector ( proto [ v ] = w v δ ) from a model pre-trained by MLM. Extending to cases when pre-trained models are unavailable, we can first train a model by self-supervised learning which learns global prototypes. Global prototypes are fixed once learned. We leave improving them during continual task learning for future study.

Adapting Pre-Trained Models for Feasibility To get feasible models for the desiderata in Eq.(4), we have two options: (a). learning with the prototype loss in Eq.(2); (b). designing a model which can satisfy the desiderata without direct supervision of probabilities p ( v | x τ , y τ ) . Option (a) needs rationale tokens to get p ( v | x τ , y τ ) , which requires expensive human annotations. In this work, we investigate models for option (b). Specifically, we investigate whether adapting a pre-trained model where we get global prototypes can satisfy our desiderata. Comparing Eq.(5) and Eq.(2), when having proto [ v ] = w v δ , models for Eq.(5) learn representations that have task-agnostic connections to global prototypes, which is a variant of Eq.(2). When lightly adapting a pre-trained encoder f ϕ to task encoder f θ , data representations are learned with reference to those task-agnostic connections. Therefore, the adapted representations may have better connections to global prototypes.

In general, our learning includes two stages: first training a model by self-supervised learning for global prototypes (can be skipped if starting from a pre-trained language model); then lightly adapting this model for target tasks while satisfying the desiderata in Eq. (4). We investigate different adaptation models and whether they satisfy our desiderata in the following sections.

4 Adaptation Models for Global Alignment

We investigate the potential of different adaptation models for our desiderata of global alignment in Eq.(4). In this section, we first introduce existing adaptation models (Section 4.1) and propose a new neighbor attention model for the desiderata (Section 4.2). A comparison of models is shown in Fig. 2.

4.1 Existing Adaptation Models

For a transformer model, representations are calculated by the self-attention mechanism. Given input representations H = [ h 1 , ..., h n ] , each output representation o i after self-attention is:

where MHA is the multi-head attention function (Appendix A), f is the feed-forward function, 194 Q ϕ , K ϕ , V ϕ are linear functions for query, key and value. Adaptation models utilize pre-trained 195 parameters for self-attentions, while adding extra components to adapt the model for target tasks. 196 According to He et al. [18], different adaptations can be viewed as combining different modification 197 vectors ∆ θ o i to pre-trained representation o i . We investigate two types of modifications below. 198

Learnable Projections Models like Adapters [21] insert adaptation modules between transformer 199 layers. The module applies linear projections to the self-attention output o i , with the non-linear 200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

activation between them. With a residual connection [19], the adapted output o ( new ) i is:

W θ ∈ R d × d represents the linear projections. (We omit the non-linear activation for simplicity).

Learnable Embeddings Models like Prompt Tuning [30] add learnable embeddings in the input. Then self-attention is performed based on the input with prompts. The adapted output is [18]:

P θ are learnable prompt embeddings in R p × d , p is the number of prompts. λ ( h i ) is a gate value computed from self-attention which decides the ratio of pre-trained and modified representations.

Choices for Global Alignment Both of the adaptations show effectiveness in single-task performance for our desiderata Eq. (3) [21, 31]. For global alignment in Eq. (4), Prompt Tuning has a gate λ ( h i ) to mix pre-trained and modified representations. With a small gate value, this may generate representations close to pre-trained representations, and thus better connect to global prototypes. However, the gate λ ( h i ) in Eq. (8) is decided by self attention over inputs and prompts, thus can lean to modified representations ∆ θ o i . Also, the learned prompts P θ may convey information far away from the original data. These may degrade the models' capacity for global alignment. Because of this, we propose a model that has a controlled gate value and relies on neighbors of tokens instead of searching from random prompts for task adaptation. In addition, the training for prompt embeddings is not as easy as that for linear projections [30, 22], which may cause efficiency issues when adapting multiple tasks. We also introduce learnable projections in our model for fast adaptations.

4.2 Transformer with Neighbor Attentions

We design a neighbor attention module added to the pre-trained model for task adaptations. The module has three properties: (1). utilizing learnable linear projections to learn modified representations; (2). acquiring neighbor representations for extra information; (3). using a controlled gate to mix pre-trained and modified representations. The adapted output of the neighbor attention module is:

where λ is the ratio of modified representations in the mix-up, || denotes the concatenation operation. K θ , V θ are learnable linear functions for key and value. M i = [ m i 1 , ..., m ik ] are k neighbor representations of the input representation h i .

Comparing Eq. (9) to Eq. (8), neighbor attention has learnable linear functions for key and value. Moreover, we manually control the gate by setting λ = 0 . 1 to push the module to focus more on the pre-trained representations. This is for our desiderata to have representations close to pre-trained ones which are trained over global prototypes. Finally, we introduce neighbor representations M i for information out of the inputs, which can improve the model's expressivity. Details are shown below.

Neighbor Representations Before the first neighbor attention layer, we find the initial neighbor representations M i for a hidden representation h i . Neighbors of h i can be obtained by comparing the dot product between h i and token embeddings from the pre-trained embedding layer, then selecting k tokens which have topK scores as neighbors. K decides the range of the neighborhood.

Then we transform neighbor embeddings to the space of h i . We disentangle h i 's j -th neighbor representation m ij into two parts: one related to the hidden representation h i ; and the other related to neighbor information out of h i . The latter can be obtained by deviating neighbor embedding e ij from h i 's token embedding e i . Then the transformed neighbor representation is: m ij = α ( e ij -e i )+ β h i , where 0 < α , β < 1 are scalars. In this paper, we set α = β = 0 . 2 .

After that, the neighbor representation M i is updated at each neighbor attention layer. For the j -th neighbor representation m ij , the updated representation m ( new ) ij for the next layer is:

Adding neighbor attention on more layers will increase the model capacity, but also cause more risk of over-smoothing [45], i.e., neighbor tokens all have the same representations. In practice, we add

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

neighbor attention to less than half of the transformer layers, and leave the last layer untouched for guidance. In continual learning, the optimal layer selections for different tasks may vary.

5 Experimental Settings

Single Task Evaluation for Desiderata Wefirst evaluate the models' capacities for our desiderata Eq. (3) and Eq. (4) on single tasks. We test classification accuracies for desiderata of task performance on tasks from the GLUE benchmark [50] and SNLI data [5]. For the desiderata of global alignment, we predict top-20 tokens from the learned representation by the pre-trained decoder (global prototypes), and compute the ratio of rationle tokens in the top-20 predictions (i.e. Recall@20). We evaluate this on e-SNLI dataset [8], where data's rationale tokens [5] are highlighted by human annotators.

Continual Learning (CL) Evaluation We evaluate four sequences of tasks: (1) Yahoo 1 : a split of Yahoo dataset for news question-answer categorization [57] with 5 disjoint tasks containing 2 classes each; (2) Yahoo 2 : a Yahoo sequence with the same split as (1) but with more data; (3) DB : a split of DBPedia data for Wikipedia article classification [57] with 7 disjoint tasks containing 2 classes each; (4) News Series : a sequence of tasks on news-related data, including AG_news (news classification, 4 classes), MRPC (paraphrase detection, 2 classes) [12], RTE (text entailment, 2 classes) [52] and SST (sentiment analysis, 2 classes) [47]. For the above sequences except (2), we randomly sample 1245 samples per class, which is the least number of class samples in our datasets. For (2), we sample 10000 samples per class. We measure the average accuracy and forgetting (Appendix C) with standard deviations. For each sequence, we test five random orders of tasks.

We evaluate for both task-incremental and class-incremental learning. Task identifiers are available at inference time for task-incremental learning but not for class-incremental learning [37]. For class-incremental learning, the original cross-entropy loss over all seen classes will cause significant forgetting [54, 1]. Since our work does not focus on the problem of cross-entropy, we apply the asymmetric strategy ( ACE ) [7]: the current task's classification loss is calculated over in-task classes, while the replay loss is calculated over all seen classes in the memory (if applicable).

Models and CL Frameworks We compare different adaptation models on BERT-base. Data representation is from a [MASK] token added to the beginning of input to match the pre-training format. Models for comparison are: (1) NeiAttn : our standard neighbor attention model. (2) NeiReg : our neighbor attention model with extra regularization for holistic information (Appendix B). (3) Fine-tuning (FT) : a model in which all parameters are learnable. (4) Prompt Tuning (ProT) [30]: the model adding learnable embeddings only to data inputs. (5) Prefix Tuning v2 (PT2) [33]: an adaptation model adding learnable embeddings to inputs of all attention layers. (6) Adapter [21]: an adaptation model with learnable linear projections injected in each layer. (7) BitFit [55]: an adaptation model tuning only bias terms in the pre-trained model. More settings are in the appendix.

We consider different frameworks (methods) for continual learning: (1) Vanilla : the vanilla online learning framework; (2) MBPA : an episodic memory framework retrieving stored samples to locally adapt the model at inference time [48]. (3) ER : an episodic memory framework storing all seen examples and performs sparse (1%) experience replay; (4) A-GEM : an episodic memory framework constraining on gradients to prevent degrading performance of previous tasks [9]; (5) Probing : a framework which learns the encoder with Vanilla setting while tunes the classifier for each task using all task data. This is used to evaluate the discrimination of data representations; (6). MTL : a muti-task framework that jointly trains on all tasks (not continual learning). For class-incremental cases, we have the above replay-based methods combined with the ACE strategy. The baseline performance for each continual learning framework is that on FT model.

6 Experimental Results

Models for Desiderata in Eq.(3) and Eq.(4) Figure 3 shows models' capacities for our desiderata. We compare the classification accuracy for desiderata in Eq.(3) and Recall@20 of rationale tokens for desiderata in Eq.(4). The higher scores on both metrics, the better model capacities for our desiderata.

Overall, NeiAttn and PT2 consistently achieve a superior balance between classification and recall scores on different NLI tasks. However, Adapter and FT achieve high classification scores but do not generate representations well related to global prototypes (low recall scores). This supports our intuition that mixing pre-trained and modified representations with a gate can result representations better connected to global prototypes. With explicit regularization on holistic information, NeiReg

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

RTE

MNLI-m

QNLI

QQP

SST

50

NeiReg

"NeiAttn

40

PT2

.NeiAtto...

RationaleRecall@2

30

20

10

Adapter

Bitfit

FT

Bitfit

Adapter

PT2

NeiReg

FT

NeiReg·

Adapter

PT2U

Bitfit

NeiAttn

FT

"ProT

Adapter?

Bitfit

PT2...

NeiAttn

NeiReg

FT

NeiReg.

Bitfit

NeiAttn

Adapter

PT2

FT·

60

62

64

66

78

79

80

81

82

83

88

89

06

87

88

89

06

88

89

06

16

92

SNLI 1%

SNLI 10%

SNLI30%

SNLI50%

SNLI100%

30

40

PT2

NeiReg

NeiAttn

Adapter

Bitfit

NeiAttn

Bitfit

NeiReg

PT2

Bitfit

NeiReg

NeiAttn

PT2

_NeiAttn..

Bitfit

NeiReg

PT2

NeiAttoNeiReg

PT2

20

FT

Adapter

10

0

75

80

85

90

75

80

85

FT

06

75

08

Adapter

85

06

75

08

Adapter

85

90

75

80

85

Adapter

FT

06

ClassificationAccuracy

performs best in in-task (SNLI → E-SNLI) rationale recalls, while losing its superiority in cross-task (GLUE → E-SNLI) rationale recalls. This may suggest the explicit regularization may not generalize well across tasks. With prompts only in the input, ProT has insufficient capacity for task performance.

For desiderata Eq.(4), NeiAttn and PT2 perform much better than Adapter and FT. We set a τ to make NeiAttn and PT2 satisfy Eq.(4) while Adapter and FT fail to, then we evaluate them for CL scenarios.

Task-Incremental Learning Wetest models' capacities for task-incremental learning under different CL frameworks. Results are shown in Table 1. Models are split into two categories according to our desiderata (Eq.(4)) experiment above: (NeiAttn, PT2) which satisfy it and (FT, Adapters) in opposite.

In the vanilla setting, both PT2 and NeiAttn significantly outperform other models with minor forgetting. Adapter on most CL frameworks performs worse than PT2 and NeiAttn, marginally better than FT. This supports our claim that models learning representations better connected to global prototypes perform better in continual learning. Combined with ER and A-GEM, NeiAttn can improve more than PT2 in most cases. FT has significant improvement with replay but can also suffer from overfitting to the replay buffer (ER for Yahoo 1). We also evaluate on a probing framework with only the classifier retrained over task data to evaluate whether the forgetting will cause representations to lose separation. PT2 and NeiAttn also preserve the most separation of representations in this case.

In general, (NeiAttn, PT2) consistently outperform (FT, Adapter) under different CL frameworks.

313 This supports that our desiderata Eq. (4) helps improve models' continual learning ability. NeiAttn 314 performs better with replay. The capacity of models also depends on different data distributions in 315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

the sequence. On News Series, when with replay, FT can even outperform PT2. This may happen because News Series includes data from similar distributions related to the news. And models should have the capacity to deal with knowledge transfer besides catastrophic forgetting.

Results for Class-Incremental Learning Figure 4 shows models' performance on class-incremental learning. PT2 and NeiAttn perform well in the vanilla case, where the training is the same as that for taskincremental learning. This indicates that they can address connections between classes from different tasks even without supervision. On the

DB Class Incremental Learning

Yahoo-1 Class Incremental Learning

60

FT

Adapter

PT2

Averaged Accuracy

80

Averaged Accuracy

50

NeiAttn

60

30

40

20

20

10

Vanilla

AGEM-ACE

ER-ACE (sparse)

ER-ACE

Vanilla

AGEM-ACE

ER-ACE (sparse)

ER-ACE

other side, Adapter and FT perform much worse in this case. Then we evaluate three frameworks with replay: one is the full ER-ACE [7] with experience replay at each step; one is the ER-ACE (sparse) with sparse experience replay; the other is the ACE strategy with only previous task's data stored in the replay ( AGEM-ACE ). We observe that performance on class-incremental learning heavily relies on the quality of replay. In most cases, FT, Adapter and NeiAttn can benefit more from the replay. We hypothesize that it is related to the fast adaptation ability related to linear projections.

Influence of Parameter-Efficiency With limited parameters, adaptation models have less risk of deviating fast from previously learned knowledge compared to FT, and thus may perform better in CL. However, different models' improvements come not just from having fewer trainable

parameters. Table 2 shows the comparison of parameters in each model. NeiAttn has better performance in most cases compared to Adapter and Pre-trained models, which have fewer or no trainable parameters in the encoder. Even with more parameters, NeiAttn performs on par with PT2 with Vanilla and outperform PT2 with replay. NeiAttn also requires much less time to train (5 vs 20 epochs). These suggest the adaptation model structure will highly influence its performance on CL.

Visualization of Representations In Figure 5, we visualize NeiAttn and FT's data representations for classincremental DB under Vanilla and ER-ACE frameworks. Even trained with in-task classes, Vanilla NeiAttn can well disperse data representations. Learning a model includes learning the encoder (representations) and classifier (class vectors). The learned class vectors may not well align with representations even with replay (left bottom). We hypothesize this may result from different training paces for the encoder and classifier. For FT, the encoder quickly learns representations close to single class centroids, which may degrade the function of the classifier. However, with con-

FT

NeiAttn

Vanilla

ER-ACE

nections to multiple different global prototypes, NeiAttn representations may not quickly move to one centroid. Therefore, it can better balance the training of the encoder and classifier (right bottom).

7 Conclusion

In this paper, we investigate models which consider potential connections between observed and unknown tasks to reduce disruptive updating in CL. Specifically, we learn task-specific data representations appropriately connected to a general-purpose representation base with global prototypes. For NLP tasks, the global prototypes can be obtained from a pre-trained language model. And the representation connected to global prototypes can be obtained by lightly adapting the pre-trained model. We investigate existing adaptation models and propose a neighbor attention model which combines advantages of existing models. Experimental results show that models learning representations appropriately connected to global prototypes have significantly less catastrophic forgetting in CL, even without using experience replay. Specifically, when neighbor attention is used, we suffer from less catastrophic forgetting than FT and Adapter, and surpass PT2 when experience replay is applied. We consider the main limitations of our work as: (1) requiring extra memory to compute neighbor attentions; (2) the optimal number of neighbor attention layers may vary for different tasks.

372

References

Ahn, H., Kwak, J., Lim, S., Bang, H., Kim, H., and Moon, T. Ss-il: Separated softmax for 373 incremental learning. In Proceedings of the IEEE/CVF International conference on computer 374 vision , pp. 844-853, 2021. 375

376

377

378

Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., and Tuytelaars, T. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 139-154, 2018.

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

Biesialska, M., Biesialska, K., and Costa-jussà, M. R. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics , pp. 6523-6541, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.574. URL https://www.aclweb.org/anthology/2020.coling-main.574 .

Biondi, N., Pernici, F., Bruni, M., Mugnai, D., and Bimbo, A. D. Cl2r: Compatible lifelong learning representations. ACM Transactions on Multimedia Computing, Communications and Applications , 18(2s):1-22, 2023.

Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1075. URL https://www.aclweb.org/ anthology/D15-1075 .

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., and Belilovsky, E. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2203.03798 , 2022.

Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., and Blunsom, P. e-snli: Natural language inference with natural language explanations. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-GEM. In International Conference on Learning Representations , 2019. URL https: //openreview.net/forum?id=Hkf2_sC5FX .

d'Autume, C. d. M., Ruder, S., Kong, L., and Yogatama, D. Episodic memory in lifelong language learning. arXiv preprint arXiv:1906.01076 , 2019.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Dolan, B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005) , 2005.

Ermis, B., Zappella, G., Wistuba, M., Rawal, A., and Archambeau, C. Memory efficient continual learning with transformers. Advances in Neural Information Processing Systems , 35: 10629-10642, 2022.

Fini, E., Da Costa, V. G. T., Alameda-Pineda, X., Ricci, E., Alahari, K., and Mairal, J. Selfsupervised models are continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9621-9630, 2022.

Gomez-Villa, A., Twardowski, B., Yu, L., Bagdanov, A. D., and van de Weijer, J. Continually learning self-supervised representations with projected functional regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3867-3877, 2022.

Guo, Y., Liu, B., and Zhao, D. Online continual learning through mutual information maximiza421 tion. In International Conference on Machine Learning , pp. 8109-8126. PMLR, 2022. 422

Gurbuz, M. B. and Dovrolis, C. Nispa: Neuro-inspired stability-plasticity adaptation for 423 continual learning in sparse networks. arXiv preprint arXiv:2206.09117 , 2022. 424

He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., and Neubig, G. Towards a unified view of 425 parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366 , 2021. 426

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 427 Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778, 428 2016. 429

Hou, S., Pan, X., Loy, C. C., Wang, Z., and Lin, D. Learning a unified classifier incrementally 430 via rebalancing. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern 431 Recognition , pp. 831-839, 2019. 432

433

434

435

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pp. 2790-2799. PMLR, 2019.

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Huang, Y., Zhang, Y., Chen, J., Wang, X., and Yang, D. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05489 , 2021.

Javed, K. and White, M. Meta-learning representations for continual learning. Advances in Neural Information Processing Systems , 32, 2019.

Ke, Z., Liu, B., Ma, N., Xu, H., and Shu, L. Achieving forgetting prevention and knowledge transfer in continual learning. Advances in Neural Information Processing Systems , 34:2244322456, 2021.

Kemker, R. and Kanan, C. Fearnet: Brain-inspired model for incremental learning. arXiv preprint arXiv:1711.10563 , 2017.

Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

Knoblauch, J., Husain, H., and Diethe, T. Optimal continual learning has perfect memory and is NP-hard. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 5327-5337. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/knoblauch20a. html .

Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., and Zhang, B.-T. Overcoming catastrophic forgetting by incremental moment matching. arXiv preprint arXiv:1703.08475 , 2017.

Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021.

Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Liu, T., Ungar, L., and Sedoc, J. Continual learning for sentence representations using conceptors. arXiv preprint arXiv:1904.09187 , 2019.

Liu, X., Ji, K., Fu, Y., Du, Z., Yang, Z., and Tang, J. P-tuning v2: Prompt tuning can be 463 comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 , 464 2021. 465

Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. Gpt understands, too. 466 arXiv preprint arXiv:2103.10385 , 2021. 467

Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in 468 neural information processing systems , 30, 2017. 469

470

471

472

Madaan, D., Yoon, J., Li, Y., Liu, Y., and Hwang, S. J. Representational continuity for unsupervised continual learning. In International Conference on Learning Representations , 2021.

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., and van de Weijer, J. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Pernici, F., Bruni, M., Baecchi, C., and Del Bimbo, A. Fix your features: Stationary and maximally discriminative embeddings using regular polytope (fixed classifier) networks. arXiv preprint arXiv:1902.10441 , 2019.

Pernici, F., Bruni, M., Baecchi, C., Turchini, F., and Del Bimbo, A. Class-incremental learning with pre-allocated fixed classifiers. In 2020 25th International Conference on Pattern Recognition (ICPR) , pp. 6259-6266. IEEE, 2021.

Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pp. 487-503, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.39.

Radiya-Dixit, E. and Wang, X. How fine can fine-tuning be? learning efficient language models. In Chiappa, S. and Calandra, R. (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pp. 2435-2443. PMLR, 26-28 Aug 2020.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 , 2019.

Razdaibiedina, A., Mao, Y., Hou, R., Khabsa, M., Lewis, M., and Almahairi, A. Progressive prompts: Continual learning for language models. arXiv preprint arXiv:2301.12314 , 2023.

Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.04671 , 2016.

Shi, H., GAO, J., Xu, H., Liang, X., Li, Z., Kong, L., Lee, S. M. S., and Kwok, J. Revisiting oversmoothing in BERT from the perspective of graph. In International Conference on Learning Representations , 2022.

Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690 , 2017.

Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pp. 1631-1642, 2013.

Sprechmann, P., Jayakumar, S. M., Rae, J. W., Pritzel, A., Badia, A. P., Uria, B., Vinyals, O., Hassabis, D., Pascanu, R., and Blundell, C. Memory-based parameter adaptation. arXiv preprint arXiv:1802.10542 , 2018.

Sun, F.-K., Ho, C.-H., and Lee, H.-Y. Lamol: Language modeling for lifelong language learning. arXiv preprint arXiv:1909.03329 , 2019.

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multitask benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=rJ4km2R5t7 .

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V ., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 139-149, 2022.

Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1101. URL https://www.aclweb.org/ anthology/N18-1101 .

Yoon, J., Yang, E., Lee, J., and Hwang, S. J. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.01547 , 2017.

Yu, L., Twardowski, B., Liu, X., Herranz, L., Wang, K., Cheng, Y., Jui, S., and Weijer, J. v. d. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 6982-6991, 2020.

Zaken, E. B., Ravfogel, S., and Goldberg, Y. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 , 2021.

Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International Conference on Machine Learning , pp. 3987-3995. PMLR, 2017.

Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification. 533 Advances in neural information processing systems , 28, 2015. 534

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies continual learning in NLP by leveraging global prototypes. The authors attribute the catastrophic forgetting to the disruptive updates caused by the misalignment between the knowledge learned from observed tasks and the knowledge required for future tasks. To tackle this problem, the authors propose NeiAttn which derives global prototypes and learns proper relationship between the prototypes and data representations for each task. Experiments show that models learning data representations well related to global prototypes can induce less catastrophic forgetting and NeiAttn outperforms baselines in both task-incremental and class-incremental learning setting.\n\nSTRENGTHS:\n1. The paper and proposed method are well-motivated. The paper reveals a misalignment between the knowledge learned from observed tasks and the knowledge required for future tasks. This is a core issue in continual learning, especially in the continual learning with pre-trained models. The learning objective introduced in Line 154-156 is clear.\n2. The paper considers the property of pre-trained language models (discussed in Line 175-178). Personally, I think this perspective is important as pre-training is very common in building machine learning system but previous literatures in continual learning seldom consider the property of pre-trained models.\n3. The experimental results show the effectiveness of NeiAttn.\n\nWEAKNESSES:\n1. The writing of the proposed method is not clear. Based on my understanding, the core of the proposed method lies in Equation 9, though there are many related contents in Section 3 and Section 4. It would be better if there is an overview of your proposed method given by pseudo code or illustration; or, it may be clearer to introduce your method in a top-down way.\n2. NeiAttn outperforms Prompt Tuning marginally according to Table 1 and Figure 4. Based on the motivation and desiderata introduced by this work, I'm wondering if you could apply the training objective in Equation 3, 4 to the Promt Tuning framework. Will it give you better results and simpler method than the current NeiAttn?\n3. In Appendix F, the authors give the ablation results of number of neighbor attention layers. The results show that fewer neighbor attention layers give better CL results. If all layers use Neighbor Attention, the results are very bad. Then what's the benefit of using Neighbor Attention for continual learning?\n\nQUESTIONS:\n1. How does the selection of $\\alpha_\\tau$ influence the results? The paper only mentions \"We set $\\alpha_\\tau$ to make NeiAttn and PT2 satisfy Eq.(4) while Adapter and FT fail to\".\n\nLIMITATIONS:\nThe paper mentions some limitations in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-12"}

comments: SUMMARY:
This paper focuses on continual learning in NLP and introduces a regularization-based method to tackle the problem. The main contributions of this work include global alignment, highlighting general-purpose knowledge across tasks, and neighbor attention, which offers a novel parameter-efficient tuning approach. Experimental results on both task-incremental and class-incremental learning scenarios demonstrate the effectiveness of the proposed method.

STRENGTHS:
The paper is well-written with smooth flow. It addresses two popular continual learning (CL) settings effectively. The proposed intuition aligns well with LLM and makes logical sense.

WEAKNESSES:
1. The underlying idea is built upon the assumption that LLM contains general-purpose knowledge and that learning tasks should not deviate too far from it. This idea shares similarities with other regularization methods and may suffer from similar drawbacks, such as potential negative impact on new task performance and insufficiency of soft regularization. From this perspective, it is not clear how this paper addresses the problem in a way that other regularization methods cannot.

2. Table 1 lacks inclusion of several CL NLP baselines, (there is an extensive survey in [1]). Only MBPA, published in 2018, is listed for NLP. It is suggested to consider adding more CL NLP baselines, such as [2] (which you have cited) and other latest work mentioned in the survey.

[1]: Continual Learning of Natural Language Processing Tasks: A Survey. https://arxiv.org/abs/2211.12701  
[2]: Achieving forgetting prevention and knowledge 443 transfer in continual learning, NeurIPS 2021

QUESTIONS:
I think this paper has good potential. I would be willing to revise my score if they address the concerns I have raised.

LIMITATIONS:
See weakness 1
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of catastrophic forgetting in continual learning and propose to connect observed and unknown tasks by means of task-specific data representations which can be seen as general-purpose representations useful for a wider range of tasks. To this end, they introduce the notion of global prototypes which can be pre-learned and reflect data semantics. Based on these they learn more task specific representations using an objective that trades off two losses (classification loss and prototype loss) . In experimental verifications of their ideas, they consider NLP tasks and find that catastrophic forgetting can successfully be reduced and their neighbor attention model achieves better performance than previous baselines.\n\nSTRENGTHS:\nThe broad ideas of this paper are easy to follow. The overall idea for how to mitigate catastrophic forgetting appears simple, sound, and solid. The idea of considering transformers with neighbor attention, too, is simple yet compelling. Experimental evaluations appear to be rigorous and comprehensive; results reveal favorable characteristics of the proposed framework.\n\nWEAKNESSES:\nAt points there are concerns regarding technical details. Certain statements appear to be handwavy.  Sentences such as „In practice, we add neighbor attention to less than half of the transformer layers and leave the last layer untouched for guidance.” or „In continual learning, the optimal layer selections for different tasks may vary.“ could need more elaboration. Throughout, several hyperparameters are introduced (K, \\alpha, \\beta, …) which apparently have then been set in a heuristic manner. The contribution would be stronger if a (mathematical) reason for these choices had been given.\n\nQUESTIONS:\nWhat does it mean to say „… and leave the last layer untouched for guidance.“ ?\nIndeed, the concept of “task-specific guidance” seems to be of importance for this work but is not made precise.\nIn lines 149-150, the ansatz for the reference probability distribution is given and motivated via references [8], [16] and [36]. This connection does not become clear enough. Where do the rational tokens r_\\tau come from? The discussion in lines 173-182 does not clarify this point.  \nHow is parameter K in line 234 chosen? Is it set to 20 (as one may guess from line 250)? If so, what motivates this choice? Do other choices lead to different results?\nA similar question applies to the statement “in this paper, we set \\alpha = \\beta = 0.2” (line 239). Some motivation for this choice would strengthen the paper.\n\nLIMITATIONS:\nThe authors openly address limitations of their proposed approach (increased memory requirements and additional need for hyper-parameter tuning). There are no concerns w.r.t. to negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a continual learning method for both task and class incremental learnings by incorporating global prototypes. These global prototypes are derived from a pre-trained masked language model and are used to make connections with task specific prototypes. By maintaining these connections, the proposed method prevents task knowledge from forgetting. The authors additionally introduce a trainable module called AttnNei for multi-head attention. The proposed method is evaluated using several existing CL methods and the performance is compared with different adaptation models on BERT-base.\n\nSTRENGTHS:\n1. The overall approach of using global knowledge and task-specific knowledge to prevent forgetting is interesting.\n2. The proposed method is applicable to existing CL methods.\n\nWEAKNESSES:\n1. I found it difficult to comprehend how and why Eq.2 establishes a connection between global knowledge and task-specific knowledge, leading to the prevention of forgetting.\n2. The experiment results do not present good advantage of the proposed method over existing methods due to the following reasons: i) The baselines are too old to know how much improvement it would make when it’s applied to more recent methods. ii) The experiment only demonstrate improvements for CL methods whose original models do not leverage adapters or prompts. However, there re CL methods that already incorporate prompts or adapters [1, 2]. To provide a broader perspective, the authors should compare their method with these approaches. iii) According to Fig 3 and Tab 1, PT2 appears to be comparable in performance, while requiring fewer resources as indicated in Tab 2\n\n3. This method is only applicable to NLP tasks, given that the global prototypes are obtained from language models. Most continual learning methods are designed to be general and not limited to specific task types such as vision or text. Therefore, a discussion on how to apply this method to non-language tasks should be included.\n\nMisc.\nTypos: Rationle in line 251. preservingholistic in line 81.\n\n[1] Wang et al. Learning to Prompt for Continual Learning. CVPR, 2022\n\n[2] Kim et al. A multi-head model for continual learning via out-of-distribution replay. CoLLAs, 2022\n\nQUESTIONS:\n1. NeiReg is proposed, but its performance is not compared in Tab 1 and Fig 4.\n2. Does the network size expand for each task in the learning process? If it does, how much does it increase at each task and what’s the final size of the model after learning all the tasks?\n\nLIMITATIONS:\nRefer to Weaknesses and Questions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the continual learning problem by leveraging a concept called global prototypes, which are invariant features that are not altered during task-specific continual learning. The training is thus augmented with an additional alignment loss of the data features to the prototypes. The paper then motivates that adapter-like parameter efficient fine-tuning (PeFT) are prototype aligned. Experimental-wise, the paper conducted studies on different PeFT fine-tuning methods for continual learning and showed that their proposed light-adaption NeiAttn and an existing PeFT method PT2 is significantly better than others.\n\nSTRENGTHS:\nThe motivation of the paper is clear and the experiments conducted are extensive. The overall presentation is clear. Figures are nice and illustrative.\n\nWEAKNESSES:\n1. The paper is dense and while the authors have tried to illustrate it, I still find some parts less convincing. Mostly, I feel like there is simple argument to summarize Section 3: continual learning should not deviate from previous task learned parameters much. This seems to justify all the later experiments and analysis. The current Section 3 draws an abstract component of global prototype, but only realizes it as fixed model parameters, which I feel is redundant. Maybe I missed a point in the paper though, happy to be corrected.\n2. There is a lack of measurement on how an adapter module is closer to the original model (closer to the prototype), the paper arrived at a conclusion that PT2 and their NeiAttn are better, but there doesn't seem to be a strong reason behind.\n\nQUESTIONS:\n1. This might be my misunderstanding of the paper, but I absolutely can't parse sentences 300-301, what do you mean by setting a_tau? Was equation (4) actually used in training, or from my understanding, a concept of global prototype that is desired to achieve, but not explictly used in training?\n\nLIMITATIONS:\nsome limitations were discussed about the requirement of hyperparameters for their proposed NeiAttn.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-13"}