InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective
Authors: Yifan Song
Peiyi Wang
Weimin Xiong
Dawei Zhu
Tianyu Liu
Zhifang Sui
Sujian Li
Venue: EMNLP 2023 Findings (2023)
Keywords: continual learning
text classification

Abstract: Continual learning (CL) aims to constantly learn new knowledge over time while avoiding catastrophic forgetting on old tasks.
We focus on continual text classification under the class-incremental setting.
Recent CL studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting.
In this paper, through an in-depth exploration of the representation learning process in CL, we discover that the compression effect of the information bottleneck leads to confusion on analogous classes.
To enable the model learn more sufficient representations, we propose a novel replay-based continual text classification method, InfoCL.
Our approach utilizes fast-slow and current-past contrastive learning to perform mutual information maximization and better recover the previously learned representations. 
In addition, InfoCL incorporates an adversarial memory augmentation strategy to alleviate the overfitting problem of replay.
Experimental results demonstrate that InfoCL effectively mitigates forgetting and achieves state-of-the-art performance on three text classification tasks.

InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective
Yifan Song' Peiyi Wang' Weimin Xiong' Dawei Zhu'
Tianyu Liu² Zhifang Sui"
Sujian Li
National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
Alibaba Group
Cyfsong, wmxxing, dwzhu, tianyu0421, szf, lisujiann edu. cn wangpeiy19979@@mai... com
Abstract
Yi (s,, No)
Y, = (Bts Net
Y, = fur, Mr
Y J.Y
Continual learning (CL) aims to constantly learn new knowledge time while avoid- ing catastrophic forgetting on old tasks. We focus on continual classification under class-incrementll setting. Recent CL studies have identified the severe performance decrease on analogous classes as key factor for catas- trophic forgetting. In this paper, through an in-depth exploration of the representation learn- ing process in CL., discover that the pression effect of the information bottleneck leads to confusion on analogous classes. To enable the model learn more sufficient repre- sentations, we propose a novel replay-based continual text classification method, InfoCL. Our approach utilizes fast-slow and current- past contrastive learning to perform mutual in- formation maximization and better recover the previously learned representations. In addition, InfoCL incorporates an adversarial memory augmentation strategy alleviate the overfit- ting problem of replay. Experimental results demonstrate that InfoCI. effectively mitigates forgetting and achieves state-of of-the-art perfor- mance on three text classification tasks. The code is publicly available https://github. com/Yifant htm/Yifaa--Sogg7995 Song793/ 793/ InfoCL.
Zi
Final Test
on
on
7s
X1
X=uX
Sufficient
X
Insutlicienl
Figure I: Illustration for continual text classification with three tasks where each task involves two new classes. X, Y,, and Z, denote input sentences, new classes and learned representations for i-th task T; spectively. Although the representations Z, learned in Ti are sufficient for classifying Y., they are insufficient to distinguish all seen classes Y in the final test.
and is eventually evaluated on seen classes. Like other CL systems, the major challenge of continual text classification is catastrophic forgetting: after new tasks are Icarned, performance on old tasks may degrade dramatically (Lange et al., 2022).
The carlier work in the CL community mainly at- tributes catastrophic forgetting to the corruption the learned representations as new tasks arrive and various methods have been introduced to retain recover previously learned representations (Kirk- patrick al., 2017: 77 Rebuffi et al., 2017; Mallya and Lazebnik. 2018; Lange al., 2022). Recently, some studies (Wang et al., 2022; Zhao al., 2023) find that, under the class-incremental setting, the se- vere performance decay among analogous classes is the key factor of catastrophic forgetting. To im- prove the performance of distinguishing analogous classes, Wang et al. (2022) exploit a heuristic ad- versarial class augmentation and Zhao al. (2023) propose sophisticated memory-insensitive proto- type mechanism. However, due to a lack of thor- ough investigation into the underlying cause of confusion similar classes, previous empirical methods may not be universally effective and are
1 Introduction
Continual learning (CL) enables conventional static natural language processing models to constantly gain new knowledge from a stream of incoming data (Sun et al., 2020; Biesialska et al., 2020). In this paper, we focus on continual text classifica- class-incremental
tion, which is formulated problem, requiring the model to learn from a se- quence of class-incremental tasks (Huang al., 2021). Figure gives an illustrative example of continual text classification. The model needs to learn to distinguish some new classes in each task
"Corresponding Author

---

unable to offer guidance for further improvements. In this paper, for the first time we present an in-depth analysis of the analogous class confusion problem from an information theoretic perspective. We investigate the impact of the information bot- tleneck (IB) on the representation learning process of CL, specifically how it compresses the mutual information between the representation and the in- put. Through formal analysis and empirical study, we find that within each task, current CL models tend to discard features irrelevant to current task due to the compression effect of IB. While the acquired representations are locally sufficient for current task, they may be globally insufficient for classifying analogous classes in the final test. We refer to this phenomenon representation bias and aim enhance the CL model's ability to learn more comprehensive representations.
sive representations. (3) Experimental results on several text classification datasets show that InfoCL learns more effective representations and outper- forms state-of-the methods.
Related Work
Continual Learning Continual Learning (CL) studies the problem of continually learning knowl- edge from sequence of tasks (Lange al., 2022) while avoiding catastrophic forgetting. Previous CL work mainly attributes catastrophic forgetting to the corruption of learned knowledge and can be divided into three major families. Replay-based methods (Rebuffi et al., 2017: Prabhu et al., 2020) save few previous task instances in a memory module and retrain on them while training new tasks. Regularization- et al., 2017: Aljundi al., 2018) introduce an extra regularization loss to consolidate previous knowl- edge. Parameter-isolation methods (Mallya and Lazebnik, 2018) dynamically expand the network and dedicate different model parameters to each task. Recent studies have identified the confusion among analogous classes as a key factor of catas- trophic forgetting. In this paper, discover the representation bias is the underlying cause of such confusion and design InfoCL mitigate it.
methods (Kirkpatrick
Based on our analysis, we propose a replay- based continual text classification method InfoCL. Using contrastive learning as the core mechanism, we enable the model to learn more comprehensive representations by maximizing the mutual informa- tion between representation and the original input via InfoNCE. Specifically, we design fast-slow and current-past contrast strategies. First, from the IB theory, the representations in the early stage of optimization preserves more information. Hence, when learning for new classes in current task, we leverage MoCo framework (He et al., 2020) and conduct fast-slow contrastive learning to facilitate the learned representations retain more informa- tion about the input. On the other hand, to further alleviate representation corruption, when conduct- ing memory replay, we leverage current-past con- trastive learning to ensure the learned representa- tions do not undergo significant changes. Due to the limited budget of memory, the performance of current- past contrastive learning is hindered by the over-fitting problem. To this end, InfoCL incor- porates adversarial data augmentation to generate more training instances for replay.
Contrastive Learning Contrastive learning aims to learn representations by contrasting positive pairs against negative pairs (Chen et al., 2020). Re- cently, contrastive learning has made great progress in both unsupervised and supervised settings (Chen et al., 2020; He et al., 2020; Khosla et al., 2020; Barbano et al., 2022). The success of contrastive learning can be partially attributed to that the com- monly used objective, InfoNCE, maximizes the mutual information between representations and inputs (van den Oord et al., 2018). Previous con- tinual learning work has already integrated con- trastive learning to alleviate the catastrophic forget- ting. Cha et al. (2021) and Zhao et al. (2022) use supervised contrastive learning to learn more con- sistent representations. Hu et al. (2022) design a prototypical contrastive network alleviate catas- trophic forgetting. However, due to the lack in-depth analysis of the representations learned continual learning, these approaches fail to harness the full potential of contrastive learning In this paper, we investigate the representation learning
Our contributions are summarized as follows: (1) We formally analyze the analogous class confu- sion problem in CL from an information theoretic perspective and derive that the representation bias led by the compression effect of IB is the under- lying cause of forgetting. (2) We propose novel replay-based continual text classification method InfoCL, which exploits fast-slow and current-pasi constrastive learning to capture more comprehen-
'See the Appendix A for detailed discussion of related work.

---

process in CL and propose fast-slow and current- past contrastive learning to enable the model learn more comprehensive representations and further mitigate the representation corruption problem.
where [X'; is the mutual information (MI) be- tween X and Z, quantifying the information re- tained in the representation 2. I(2;)) quantifies the amount of information in Z that enables the identification of the label y , is a trade-off hy- perparameter. With information bottleneck, the model will learn minimal sufficient represenlation Z* (Achille and Soatto, 2018) of X corresponding to J::
3 Task Formulation
In this work, we focus continual learning for sequence of & class-incremental text classification tasks (Ti, T2,--.. ...y Tia)- Each task Tr has its dataset D; = {(zn: yn) } where (Zn, Mn) is instance of current task and is sampled from individually i..d. distribution p(X,, Y.). Different tasks Ti and Tj have disjoint label sets Y; and Yg. The goal of CL is to continually train the model on new tasks to learn new classes while avoiding forgetting pre- viously learned ones. From another perspective, if we denote X = U,X, and Y U,Y; as the input and output space of the entire CL process respec- tively, continual learning aims approximate holistic distribution x) from a non-i.i.d data stream.
arg min T(2;;) s.t. ((2:)) - (()))))
(2)
(3)
Minimal sufficient representation is important for supervised learning, because retains as little about input as possible to simplify the role of the classifier and improve generalization, without los- ing information about labels.
4.2 Representation Learning Process of CL
Continual learning is formulated as sequence of individual tasks (Ti, T2.... .... Ts.). the model aims to approximate distribution of cur- rent task pX)] According to IB, if the model F o f converges, the learned hidden represen- tation Z, fX,) will be local minimal sufficient for Tj:
The text classification model F is usually com- posed of two modules: the encoder and the clas- sifier a. For input 2, get the corresponding representation z f(z), and use the logits a (z) to compute loss and predict the label.
For i-th task Ti
4 Representation Bias in CL
Z. arg min I (X;; Zi) Zi
(4)
Previous work (Wang et al., 2022; Zhao et al., 2023) reveals that the severe performance degradation on analogous classes is the key factor of catastrophic forgetting. In this section, we investigate the rep- resentation learning process of continual learning from an information theoretic perspective and find that the representation bias is the underlying cause confusion in analogous classes.
s.t. I (Z; = I [X;;; Yi),
(5)
which ensures the performance and generalization ability of the current task. Nevertheless, the local minimization of the compression term / (X;; Z4) will bring potential risks: features that are useless in the current task but crucial for other tasks will be discarded.
The goal of CL is to classify all seen classes Y U,Y. For the entire continual learning task with the holistic target distribution p(||)) the necessary condition to perform well is that the representation Z is globally sufficient for Y: I(Z = I(; However, as some crucial features are compressed, the combination of lo- cal minimal sufficient representations for each task Z = U,Zi may be globally insufficient:
4.1 Information Bottleneck
We first briefly introduce the background of infor- mation bottleneck in this section. Information bot- tleneck formulates the goal of deep learning as an information thhooretic trade-off between represen- tation compression and preservation (Tishby and Zaslavsky, 2015; Shwartz-Ziv and Tishby, 2017). Given the input X' and the label set ),, one model is built to learn the representation Z =(X),, where FF is the encoder. The learning procedure of the model is minimize the following Lagrangian:
I I(Z:Y) < I(X
(6)
We name this phenomenon as representation bias: (1) due to the compression effect of IB, the learned
-11;))-11(3)) (I(3)))

---

FewRel
MAVEN
the entire continual task, we compare I(Z;Y) on the final test with all classes.
Models
I(X1; Z1)
I(ZYY)
I(X1; Z.) 1(Z;)
As shown in Table 1, both [(x;; and I(Z;)) of three CL models are significantly lower than supervised learning, indicating that the CL model tends to compress more information due to the individual task setting and the representations learned in CL are insufficient for the entire contin- ual task.
Supervised
2.42
2.45
3.50
2.42
RP-CRE 2.08
2.21
3.15
2.31
CRL
2.12
2.18
3.12
2.30
CRECL 2.20
2.31
3.01
2.36
Table I: Mutual information comparison between su- pervised learning and strong CL baselines FewRel and MAVEN datasets. We use [(;; Z) to measure how much features of input X representation preserves. To exclude the impact of representation corruption, we instead estimate [X1; 21) after CL models finish Tí. [(ZY) measures whether the learned representation sufficient for the entire continual task.
5 Methodology
From the formal analysis and empirical verification, we establish that representation bias plays a cru- cial role in the performance decline of analogous classes. Consequently, in this section, propose novel replay-based CL method, InfoCL, which able to maximize T(x;; Z,) and help the model learn more comprehensive representations.
reprexentations each individual task may be in- sufficient for the entire continual task.
Then the underlying cause of the performance decrease of analogous classes is obvious. Take two cross-task analogous classes Va and yh as an ex- ample. Under sequential task setting of CL, the model is unable co-training instances from Yo and yb. It means that the representations of these two classes can exclusively be learned within their respective tasks. When learning Na. the local suffi- cient representations to identify Yo are insufficient to differentiate with yb- Hence, the appearance of will lead to dramatically performance decrease of ya. resulting in what is known as catastrophic forgetting.
5.1 Overall Framework
The objective of contrastive learning, specifically InfoNCE, serves as a proxy to maximize the mutual information / [X;;; Z,) (van den Oord et al., 2018). Therefore, we utilize contrastive learning as the core mechanism of InfoCL to acquire more com- prehensive representations. Concretely, design flst and current- pastt contrastive learning for training new data and memory data.
The overall framework InfoCL is depicted Figure 2. For a new task Typ we first train the model on D3 to learn this task. We perform fast-slow con- trastive learning to help the model capture more sufficient representations and mitigate the represen- tation bias. Then store a few typical instances for each class y Y2 into the memory M, which contains instances of all seen classes. To allevi- ate representation decay, we next conduct memory replay with current-par past contrastive learning. As the performance of representation recovery is al- ways hindered by the limited size of memory, we incorporates adversarial augmentation to alleviate overfitting.
4.3 Empirical Results
To confirm our analysis, here we directly mea- sure the mutual information among X, y and 2. Since the representations learned by super- vised learning is always globally sufficient, i.e., I (Z; = I(X; Y), we use supervised learning on all data the baseline, and compare it with several strong CL methods. Concretely, we use MINE (Belghazi et al., 2018) as the MI estimator and conduct experiments on FewRel and MAVEN d
First, we measure [(X; Z) to quantify the fea- tures preserved in the representation Z. How- ever, previously learned representations will be corrupted once the model learns new tasks, lead- ing to inaccurate estimation. To exclude the im- pact of representation corruption, we instead esti- mate [X;; Z1) on Ti's test set. Second, to assess whether learned representations are sufficient for "See Section 6.1 for details of CL. hasclines and datascts.
5.2 Fast-Slow Contrastive Learning
In the representation learning process of a task Ti the compression effect of IB will minimize the mutual information [(X;; Z.), leading to globally insufficient representattonss Intuitively, in the early phase of optimization, T(X;; Z4) larger and the representations preserve more information about the inputs. However, directly adopting an early

---

(a) Training for Tk
(b) Fast-Slow C Contrastive
(c) Current-Par Past Contrastive
T
New Task Data
Memory Data
Fast
Slow
Adv.
Past
New Task Training
Slow Model
Encoder
Slow Encoder
Past Encoder
Current
Encoder
Classifier
Slow Lineur
Past Lineur
Memory
Pust Model
Classifie
Replay
Past Repr.
Ccee
C
Queue
Cep
Ta+1
Figure 2: (a) A demonstration InfoCL. We design fast-slow and current-past contrastive learning for initial training and memory replay, respectively. (b) Fast-slow contrastive learning. The slowly progressing model generates representations preserving more information. (c) Current- contrastive learning with adversarial augmentation. Contrasting with old model from T further alleviates representation corruption.
stop strategy is not feasible, as the representation compression is essential for generalization. Instead, we try to pull the learned representations and early representations of the same class together, facili- tating the preservation of more comprehensive in- formation in the final representations. Inspired by He et al. (2020), we employ a momentum contrast which consists of a fast encoder and a momentum updated slow encoder. The representations from the slowly-updated branch will preserve more in- formation about the input sentences. The fast-slow contrast can "distill" these information from the slow branch to fast branch to learn more compre- hensive representations.
representations z preserve more information than 2. We use InfoNCE to perform fast-slow contrast:
LB
1 B
Σ
Σ
log exp(z, ///TT)
Ejer exp(z. 2//T1)'
iel PEP()
(8)
where = {1.2, |B|} is the set of indices
batch B. J = (1.2,..., | U Q denotes the in- dices set of instances in the batch or the queue, P(i) (p E J 1/p n} is the indices of in- stances which have the same label z, from the batch or the queue. Ti is the temperature hyperpa- rameter.
Figure (b) depicts the architecture. The fast model is updated by gradient descent, while the slow model truncates the gradient and is updated with the momentum mechanism during training. Formally, denoting the parameters of the fast and slow models as 0 and 0', 0' is updated by:
The final optimization objective in new task training is the combination of cross entropy loss Lee and the contrastive loss Lis:
L1 = Lee + A1
(9)
where d is the factor to adjust the loss weight.
0' - n00 + n),
5.3 Memory Selection
where is the momentum coefficient which is relatively large to ensure the slow update (e.g.. 77 = 0.99). For the slow encoder, we also maintain a representation queue Q to increase the number of negative instances beyond the batch size, enabling InfoNCE to more effectively maximize MI. The queue is updated with the output of slow model by int strategy.
After the initial training stage, we select and store typical instances for each class for replay. Since the primary focus of our paper is to address the representation bias problem, we adopt the mem- ory sampling strategy employed in prior work (Cui et al., 2021; Zhao al., 2022) to ensure fair comparison. Specifically, for each class, use K-means to cluster the corresponding representa- tions, and the instances closest to the centroids are stored in memory M. Then we use the instances
We denote z for the representations from the fast model and for the slow model. Then the slow

---

of all seen classes from memory M to conduct the memory replay stage.
6 Experiments
6.1 Experiment Setups
5.4 Current-Past Contrastive Learning
Datasets To fully measure the ability of InfoCL, we conduct experiments on 4 datasets for dif- ferent text classification tasks, including relation extraction, event classification, and intent detec- tion. For relation extraction, following previous work (Han al., 2020; Cui et al., 2021; Zhao al., 2022), we use Few Rel (Han et al., 2018) and TA- CRED (Zhang et al., 2017). For event classifica- tion, following Yu et al. (2021) and Wu et al. (2022), we use MAVEN (Wang et al., 2020) to build our benchmark. For intent detection, following Liu et al. (2021), we choose HWU64 (Liu et al., 2019) dataset. For the task sequence, simulate 10 tasks by randomly dividing all classes of the dataset into 10 disjoint sets, and the number of new classes in each task for FewRel, TACRED, MAVEN and HWU64 are 8, 4, 12, 5 respectively, For a fair com- parison, the result of baselines are reproduced on the same task sequences as our method. Please re- fer to Appendix C for details of these four datasets. Following previous work (Hu et al., 2022; Wang et al., 2022), we use the average accuracy (Ace) on all seen tasks as the metric.
When performing memory replay, we also employ contrastive learning to enable the model to learn more comprehensive representations for all previ- ously seen classes. Additionally, to further enhance representation recovery in memory replay stage, we propose current-par past contrastive learning which explicitly aligns current representations to the pre- vious ones.
As shown in Figure (c), after the model fin- ishs Tk-1, we store the representations z of the instances from memory M. Then we use InfoNCE loss to pull current representations z and past rep- resentations ž of the same class together:
exp(z: T2)/T2)
Lep
Σ HEl (()(()
Σ log
[3|
MEM exp(z: 7m /2)
(10)
where I = [1, .... [B} is the set of indices of batch B. M 21, [M] denotes the indices set of instances in memory M. P(i) {p M By = - is the indices of instances which have the same label as z; from the memory, 72 is the temperature hyperparameter.
Baselines We compare InfoCL against the fol- lowing baselines: IDBR (Huang et al., 2021), KCN (Cao et al., 2020), KDRK (Yu et al., 2021), EMAR (Han et al., 2020), RP-CRE (Cui et al., 2021), CRL (Zhao et al., 2022), CRECL (Hu al., 2022), ACA (Wang et al., 2022) and CEAR (Zhao et al., 2023). See Appendix D for details of the baselines.
The optimization objective in the memory replay stage is
L2 = Lee + >2Lopp
(11)
where Lee is entropy loss and A2 is the factor adjust the loss weight.
Some basclines are originally proposed to tackle one specific task. For example, RP-CRE is de- signed for continual relation extraction. We adapt these baselines to other tasks and report the corre- sponding results. Since ACA and CEAR consist of data augmentation specially designed for relation extraction, they cannot be adapted to other tasks.
5.5 Adversarial Memory Augmentation
Due to the constrained memory budgets, the per- formance of current- ppast contrastive learning in the memory replay stage is hindered by the overfitting problem. To alleviate overfitting and enhance the effect of representation recovery, we incorporate adversarial data augmentation (Zhu et al., 2020):
Implementation Details For InfoCL, we use BERT base (Devlin et al., 2019) as the encoder fol- lowing previous work (Cui et al., 2021; Wang et al., 2022). The learning rate of InfoCL is set to le-5 for the BERT encoder and 1e-3 for other modules. Hyperparameters
Ladv
min E (nn)-M B
max La (F(x + (c). M
K
(12)
tuned on the first three tasks. The memory budget for each class is fixed at 10 for all methods. For all experiments, we use NVIDIA A800 GPUs and report the average result of 5 dif- ferent task sequences. More implementation details can be found in Appendix E.
Intuitively, it performs multiple adversarial attack iterations to craft adversarial examples, which is equivalent to replacing the original batch with K- times larger adversarial augmented batch. Please refer to Appendix for details about Eq. 12.

---

Datasets
FewRel
TACRED
MAVEN
HWU64
Models
Tg
Tg
Ti0
T5
Tio
T6
Ti0
Ts
Tg
Tio
IDBR (Huang al., 2021)
73.7
71.7
68.9
64.2
63.8
60.1
64.4
60.2
57.3
80.2
78.0
76.2
KCN (Cao et al., 2020)
80.3
78.8
76.0
72.1
72.2
70.6
68.4
67.7
64.4
83.7
82.7
81.9
KDRK (Yu et al., 2021)
81.6
80.2
78.0
72.9
72.1
70.8
68.9
65.4
85.1
82.5
81.4
EMAR (Han et al., 2020)
86.1
84.8
83.6
76.6
76.8
76.1
76.8
75.7
73.2
85.5
83.9
83.1
RP-CRE (Cui et al., 2021)
85.8
84.4
82.8
76.1
75.0
75.3
77.1
76.0
73.6
84.5
83.8
82.7
CRL (Zhao et al., 2022)
85.6
84.5
83.1
79.1
79.0
78.0
76.8
75.9
73.7
81.3
CRECL (Hu al., 2022)
84.6
83.6
82.7
81.4
79.3
78.5
75.9
75.1
73.5
83.1
81.9
81.1
ACA (Wang et al., 2022)
87.0
86.3
84.7
78.6
78.8
78.1
CEAR (Zhao al., 2023)
86.9
85.6
84.2
81.1
80.1
79.1
InfoCL (Ours)
87.8
86.8
85.4
79.7
78.4
78.2
78.2
77.1
75.3
86.3
85.3
84.1
Table 2:
Accuracy (%) on all seen classes after learning the last three tasks. We report the average result of 5 different runs. The best results are in boldface. ACA and CEAR is specially designed for continual relation extraction and cannot be adapted to other tasks.
Models
Few.
TAC.
MAV.
HWU.
FewRel
MAVEN
Models
InfaCI.
85.4
78.2
75.3
84.1
Accuracy
Drop
Accuracy
Drop
w/o con.
84.9
77.6
74.7
83.4
CRL
75.3
13.3
59.8
21.2
w/o c-p con.
85.0
78.1
75.1
83.7
CRECI.
74.9
13.6
59.2
21.9
w/o adv. aug.
84.8
77.9
75.0
83.6
InfoCL
78.6
11.8
20.7
Table 3: Ablation study of InfoCL. "f-s con." and "c- p con." denote fast-slow and current-past contrastive learning. "adv. aug." denotes adversarial memory aug- mentaion mechanism.
Table 4: Average Accuracy (%) and accuracy drop (%)
on analogous classes. For each dataset, select 20% classes which are most likely to be confused with other classes.
6.2 Main Results
results are shown in Table 3. We find that the three mechanisms of InfoCL, namely fast-slow trast, current-past contrast, and adversarial mem- ory augmentation, are conducive to the model per- formance. Furthermore, the fast-slow contrastive learning performed in the new task training stage seems to be more effective than the other com- ponents, indicating that learning comprehensive representations for the new task is more essential to mitigate representation bias problem.
Table shows the performance InfoCL and base- lines on four datasets for three text classification tasks. Due to space constraints, we only illustrate results on the last three tasks. The complete ac- curacy and standard deviation of all 10 tasks can be found in Appendix G. As shown, on FewRel, MAVEN and HWU64, our proposed InfoCL con- sistently outperforms all baselines and achieves new state-of-the-an results. These experimental re- sults demonstrate the effectiveness and universality our proposed method. Regarding the TACRED dataset, it has been noticed that large fraction of the examples are mislabeled, thus compromising the reliability of the evaluation (Alt et al., 2020; Stoica et al., 2021). Here strongly advocate for more reliable evaluation other high-quality text classification datasets.
7.2 Performance on Analogous Classes
We reveal that the representation bias problem is the underlying cause of confusion on analogous classes in CL. Since InfoCL aims to learn more sufficient representations and mitigate the bias, we also conduct experiment to explore this point. Fol- lowing (Wang et al., 2022), we use the cosine dis- tance of the average embedding of the instances as a metric identify analogous classes. Specif- ically, we select 16 and 24 classes (20% of all of the classes) for FewRel and MAVEN, which are most likely to be confused with other classes. The list of these classes are shown in Appendix F. If
7 Analysis
7.1
Ablation Study
We conduct an ablation study to investigate the ef- fectiveness different components of InfoCL. The

---

106
CRL CRECL
CRL
CRECI
MAVEN Accuszy
Foufcl
25
Momnny
Mimory
Meroor aiz
Meroory
(a) Results FewRel
Results on MAVEN
Figure 3: Accuracy (%) w.t.t different memory sizes of different methods.
representation bias.
To provide intuitive demonstration of the ef- fective representations acquired by InfoCL, we conducted a case study. We select two analogous classes from FewRel, P57 ("director") and P58 ("screenwriter"). We use t-SNE to visualize the representations of these two classes after the model learned them. As illustrated in Figure 4, for both methods, the accuracy of P57 reaches 100% after learning it. However, due to the representation bias, when P58 appears, the accuracy of CRL dramat- ically declined. In contrast, InfoCL maintains a relatively stable performance. Notably, even with- out training for P58, the representations of two classes in our method exhibit a noticeable level of differentiation, highlighting InfoCL's capacity to acquire more comprehensive representations.
CRU
25707 100
257 255 17 70
250 Nead
InloCL
19001 Newe)
After Learning P57
After Leamiing P58
Figure 4: The representations of instances from P57 ("director") and P58 ("screenwriter"". P57 and PS8 are in different tasks and P57 is learned before P58 appears.
these selected classes are in the former five tasks, we record the average final accuracy and the ac- curacy drop on them. As shown Table 4, the performance on analogous classes of our model is superior and drops the least, demonstrating that our model succeeds in alleviating confusion among analogous classes.
7.4 Influence of Memory Size
Memory size is the number of stored instances for each class, which is an important factor for the per- formance of replay-based CL methods. Therefore, in this section, we study the impact of memory size on InfoCL. As Table 2 has reported the results with memory size 10, here we compare the performance of InfoCL with strong baselines FewRel and MAVEN under memory sizes and 20.
7.3
Comprehensive Representation Learning
InfoCL employs contrastive learning learn com- prehensive representations via mutual information maximization. To assess the efficiency of repre- sentation learning process of our method, we di- rectly compute the mutual information of the rep- resentations. Same as Section 4.3, we measure T(x1; 7)) on Ti and Y) after the final task with all classes. For InfoCL, I(X;; Z1) 2.32 and Y(Z 2.38 FewRel, ZiX 3.20 and I(Z;Y) 2.37 on MAVEN. Compared with baselines in Table 1, both mutual informa- tion metrics are higher, indicating our method can learn more sufficient representation to mitigate the
Table 3 demonstrates that InfoCL consistently outperforms strong baselines across various mem- ory sizes. Surprisingly, even with a memory size of 5, InfoCL achieves comparable performance to the baselines with memory size of 20, highlighting its superior capability. As the memory size decreases, the performance of all models degrades, showing the importance of memory for replay-based meth- ods. Whereas, InfoCI. maintains a relatively stable performance, showcasing its robustness even in extreme scenarios.

---

8 Conclusion
8-14, 2018. Proceedings, Part III. volume 11207 of Lecture Notes in Computer Science, pages 144-161. Springer.
this paper, we focus on continual learning for text classification in the class-incremental setting. We formally investigate the representation learn- ing process of CL and discover the representation bias will lead catastrophic forgetting on analo- gous classes. Based on our analysis, we propose InfoCL, which utilizes fast-slow and current-pant contrastive learning to learn more comprehensive representations and alleviate representation corrup- tion. An adversarial augmentation strategy is also employed to further enhance the performance of the representation recovery. Experimental results show that InfoCL learns effective representations and outperforms the latest baselines.
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. Tacred revisited: A thorough eval- uation of the tacred relation extraction task. arXiv preprint arXiv:2004. 14855.
Carlo Alherto Barbano, Benoit Dufumier, Enzo Tartaglione, Mareo Grangetto, and Pietro Gori. 2022. Unbiased supervised contrastive learning. CoRR. 0bs/2211.05566
Ishmael Belghazi, Sai Rajeswar, Aristide Baratin, R. Devon Hjelm. and Aaron C. Courville. 2018. MINE: mutual information neural estimation. CoRR, abs/1801.04062..
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà, 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6523-6541. International Committee on Com- putational Linguistics.
Limitations
Our paper has several limitations: (1) Our pro- posed InfoCL. utilizes fast-slow and current-past contrastive learning to learn more comprehensive representations, which introduces extra compu- tational overhead and is less efficient than other replay-based CL methods; (2) We only focus on catastrophic forgetting problem in continual text classification. How to encourage knowledge trans- fer in CL is not explored in this paper.
Pengfei Cao, Yubo Chen, Jun Zhao, and Taifeng Wang. 2020. Incremental event detection via knowledge consolidation networks. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 707-717, Online. Association for Computational Linguistics.
Ethics Statement
Hyuntak Cha. Jaeho Lee, and Jinwoo Shin. 2021. Co21: Contrastive continual learning. In 2021 IEEF/CVF International Conference on Computer Vision ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pages 9496-999699999 9505. IEEE.
Our work complies with the ACL Ethics Policy. Since text classification is a standard task in NLP and all datasets we used are publicly available, we have not identified any significant ethical consider- ations associated with our work.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning visual representations. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Vir- tual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR
Acknowledgement
We thank the anonymous reviewers for their helpful comments on this paper. This work was partially supported by National Key R&D Program of China (No. 2022YFC3600402 and National Social Sci- ence Foundation Project of China (21&DD287)
Li Cui, Deqing Yang. Jiaxin Yu, Chengwei Hu, Jiayang Cheng, Jingje Yi, and Yanghua Xiao. 2021. Refin- ing sample embeddings with relation prototypes to enhance continual relation extraction. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1/th International Joint Conference
References
Alessandro Achille and Stefano Soatto. 2018. Emer- gence of invariance and disentanglement in deep rep- resentations. Mach. Learn. Res., 19:50:1-50:34.
Natural Language Processing (Volume 1: Long Papers), pages 232-243, Online. Association for Computational Linguisties.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elho- seiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Computer Vision ECCV 2018 15th Eu- ropean Conference, Munich, Germany, September
Jacob Devlin, Ming-Wel Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Band Barwinska, et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(13):3521-3222
Yiduo Guo, Bing Liu, and Dongyan Zhao. 2022. Online continual learning through mutual information maxi- mization. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Ma- chine Learning Research, pages 8109-8126 PMLR.
Xu Han, Yi Dai, Tianyu Gao, Yankai L Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Contin- ual relation learning via episodic memory activation and reconsolidation. nual Meeting of the Association for Computational Linguistics, pages 6429 6440, Online. Association for Computational Linguistics.
Matthias De Lange, Rahaf Aljundi. Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh, and Tinne Tuytelaars, 2022. A continual learning sur- vey: Defying forgetting in classification tasks. IEEE Trans. Pattern Anal. Mach. Intell., 44(7):3366-3333 -3885.
Proceedings of the 58th An-
Qingbin Liu, Xiaoyan Yu, Shizhu He, Kang Liu, and Jun Zhao. 2021. Lifelong intent detection via multi- strategy rebalancing. CoRR, ftss/208.0444.
Xu Han, Hao Zhu, Pengfei Yu. Ziyun Wang. Yuan Yao. Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction 10th International Workshop on Spoken Dialogue Systems. IWSDS 2019, Syracuse, Sicily, Naly. 24-26 April 2019, volume 714 of Lecture Notes in Electrical Engineering, pages 165-183. Springer.
Kaiming He, Haoqi Fan. Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726-9735. Computer Vi- sion Foundation IEEE.
Arun Mallya and Svetlana Lazebnik. 2018. Packnet: Adding multiple tasks to single network by iterative pruning. In 2018 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7765- 7773. Computer Vision Foundation / IEEE Computer Society.
Chengwei Hu, Deging Yang, Haoliang Jin, Zhen Chen, and Yanghua Xiao. 2022. Improving continual rela- tion extraction through prototypical contrastive Iearn- ing. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 1885- 1895, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary De Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style. high-performance deep learning li- brary. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NewrIPS 2019, De- cember 8- 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang. and Diyi Yang. 2021. Continual learning for text clas- sification with information disentanglemen based regularization. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 2736-2746, Online. As- sociation for Computational Linguistics.
Zixuan Ke, Bing Liu, Hu Xu, and Lei Shu. 2021. CLASS SIC: Continual and contrastive learning of aspect sentiment classification tasks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6871-6883, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ameya Prabhu, Philip H. S. Torr, and Puneet K. Doka- nia. 2020. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vi- sion ECCV 2020- 16th European Conference, Glas- gow; UK, August 23-28, 2020, Proceedings, Part II, volume 12347 of Lecture Notes in Computer Science, pages 240. Springer.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. In Advances in Neural
Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. icarl: In- cremental classifier and representation learning. In

---

2017 IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5533-5542. IEEE Computer Society,
parative study. In The Tenth International Confer- ence on Learning Representations, ICLR 2022, Vir- tual Event April 25-29, 2022. OpenReview.neh.
Pengfei Yu, Heng Ji, and Prem Natarajan. 2021. Life- long event detection with knowledge transfer. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 5278- 5290. Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Ravid Shwartz-Ziv and Naftali Tishby, 2017. Opening the black box of deep neural networks via informa- tion. CoRR, abs/1703.00814
George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-tacred: Addressing short- comings of the tacred dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13843-13850.
Yuhao Zhang, Vietor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-awand attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguistics.
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Yi Lee. 2020. LAMOL: language modeling for lifelong lan- guage learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Kang Zhao, Hua Xu, Jiangong Yang, and Kai Gao. 2022. Consistent representation learning for continual re- lation extraction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3402- 3411. Dublin, Ireland, Association for Computational Linguisties.
Naftali Tishby and Noga Zaslavsky. 2015. Deep Iearn- ing and the information bottleneck principle. In 2015 IEFE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26- May 1, 2015, pages 1-5. IEEE.
Wenzheng Zhao, Yuanning Cui, and Wei Hu. 2023, Im- proving continual relation extraction by distinguish- ing analogous semanties.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRK, abs/1807.03748.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Gold- stein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understand- ing. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30 30, 2020. OpenReview.net.
Peiyi Wang, Yifan Song. Tianyu Liu, Binghuai Lin, Yunbo Cao. Sujian Li. and Zhifang Sui. 2022. Learn- ing robust representations for continual relation ex- traction via adversarial class augmentation. In Pro- ceedings of the 2022 Conference on Empiricul Meth- ods in Natural Language Processing, pages 6264- 6278. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Xiaozhi Wang. Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan I Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1652- 1671, Online. Association for Computational Linguis- tics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame. Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. 2022. Pre- trained language model in continual learning: A com-

---

A Comparison with Some Related Work
also distinguishing feature of work.
As aforementioned in Section some related work also applied contrastive learning on continual learn- ing. In this section, we provide more detailed dis- cussion of these work.
B Details about Eq. 12
In memory replay stage, alleviate the overfitting on memorized instances, we introduce adversarial data augmentation (Zhu al., 2020):
Both of Co²L. (Cha et al., 2021) and CRL (Zhao et al., 2022) use supervised contrastive learning and knowledge distillation to learn more consistent rep- resentations. We would like to highlight two signif- icant differences between our work and them. First, we formally analyze the representation bias prob- lem from an information bottleneck perspective and propose that maximizing (X;; Z,) can effectively mitigate this bias. Second, while Co2L employs vanilla supervised contrastive loss for representa- tion learning, we introduce two novel contrastive learning techniques: fast-slow and current-past con- trastive learning. Our method not only enables the model acquire more comprehensive representa- tions but also improves its ability to retain previ- ously learned knowledge. As shown in Table 2, our method outperforms Co²L/CRL on four datasets. CLASSIC (Ke al., 2021) utilizes contrastive continual learning method for continual aspect sen- timent classification task. However, is specifi- cally designed for the domain-incremmnntl scenario, where its primary goal is to facilitate knowledge transfer across tasks. This objective is distinct from our
Lady =
K-1 max C C 6c),
min E (Laa)
K
| Sc
(13)
where F is the text classification model and (x, y) is batch of data from the memory bank M, 5 is the perturbation constrained within the c-ball, K is step size hyperparameter. The inner maximization problem in (13) is to find the worst-case adversar- ial examples to maximize the training loss, while the outer minimization problem in (13) aims optimizing the model to minimize the loss of adver- sarial examples. The inner maximization problem is solved iteratively:
(()-1) VFL 8,-1),8),
(14)
V(8--1)
6,
II 6- ta (V |0l|se
(15)
where & is the perturbation in t-th step and II [pol projects the perturbation onto the e-ball, step size.
scenario, and thus, the use of InfoNCE in CLASSIC is not directly comparable to our work.
Intuitively, performs multiple adversarial tack iterations to craft adversarial examples, and simultancously accumulates the free parameter gra- dients ooL in each iteration. After that, the model parameter 0 is updated all at once with the accu- mulated gradients, which is equivalent to replacing the original batch with K K-times larger adversarial augmented batch.
OCM (Guo et al., 2022) proposes mutual infor- mation maximization to learn holistic representa- tion for online continual image classification. Our work, in comparison to OCM, shares a similar mo- tivation of learning more comprehensive represen- tations. Here we provide clarifications several key differences between
work and OCM. First, OCM focuses on online continual learning on im- age classification, which is different with our sce- nario. Second, for the first time, we provide for- mal analysis of catastrophic forgetting from the per- spective of the information bottleneck. This analy- sis sets apart from OCM. Finally, the contrastive learning in OCM relies on data augmentation of the image to ensure performance. According to their ablation study, without the random-resized crop augmentation, the performance of OCM will drop drastically. In contrast, we design fast-slow and current-pare past contrastive learning to get better representations. This contrastive learning design is
C Dataset Details
FewRel (Han et al., 2018) It is large scale lation extraction dataset containing 80 relations. FewRel balanced dataset and each relation has 700 instances. Following Zhao et al. (2022); Wang et al. (2022), we merge the original train and valid set of FewRel and for each relation we sample 420 instances for training and 140 instances for test. FewRel is licensed under MIT License.
TACRED
(Zhang et al., 2017) it a crowdsoure- ing relation extraction dataset containing 42 rela- tions (including no no_relation) and 106264 instances.

---

Following Zhao et al. (2022); Wang et al. (2022), we remove no relation and in our experiments. Since TACRED a imbalanced dataset, for each relation the number of training instances is limited to 320 and the number of test instances is limited to 40. TACRED is licensed under LDC User Agree- ment for Non-Members.
modified BSD license. HuggingFacce Transformers and BERT base are licensed under the Apache Li- cense 2.0. Our use of existing artifacts is consistent with their intended use.
Specifically, for the input x in relation extraction, and [E2]] to denote the start and end position of head and tail entity respec- tively, and the representation is the concatenation of the last hidden states of [En]] and [E21]. For the input event detection, the representation 2 is the average pooling of the last hidden states of the trigger words. For the input x in intent detection, the representation is the average pooling of the last hidden states of the whole sentence.
we use [Ei,] [E1a], [E2]]
MAVEN (Wang et al., 2020) It is a large scale event detection dataset with 168 event types. Since MAVEN has a severe long-tail distribution, we use the data of the top 120 frequent classes. The orig- inal test set of MAVEN is not publicly available, and we use the original development set our test set. MAVEN is licensed under Apache License 2.0.
The learning rate of InfoCL is set to 1e-5 for the BERT encoder and le-3 for the other modules. For FewRel, MAVEN and HWU64, the batch size is 32. For TACRED, the batch size is 16 because of the small amount of training data. The budget of mem- ory bank for each class 10 for all methods. The size of the queue Q in fast-slow contrastive learn- ing is 512 and the momentum coefficient 2j is 0.99. The temperatures TL, are set to 0.05. The loss factors A1 A2 are 0.05. For the adversarial memory augmentation, K=2, e=3e-1, a=le-1. For all ex- periments, we use NVIDIA A800 GPUs and report the average result of 5 different task sequences.
HWU64 (Liu et al., 2019) is an intent classifi- cation dataset with 64 intent classes. Following Liu et al. (2021), we use the data of the top 50 frequent classes and the total number of instances are 24137. HWU64 is licensed under CC-BY-4.0 License.
D Baselines
IDBR (Huang et al., 2021) proposes an information disentanglement method to learn representations that can well generalize to future tasks. KCN (Cao al., 2020) utilizes prototype retrospection and hierarchical distillation to consolidate knowledge. KDRK (Yu et al., 2021) encourages knowledge transfer between old and new classes. EMAR (Han et al., 2020) proposes memory activation and reconsolidation mechanism to retain the learned knowledge. RP-CRE (Cui et al., 2021) proposes memory network to retain the learned represen- tations with class prototypes. CRL (Zhao al., 2022) adopts contrastive learning replay and knowl- edge distillation to retain the learned knowledge. CRECL (Hu et al., 2022) uses prototypical con- trastive network to defy forgetting. ACA (Wang et al., 2022) designs two adversarial class aug- mentation mechanism to learn robust representa- tions. CEAR (Zhao et al., 2023) proposes memory- intensive relation prototypes and memory augmen- tation to reduce overfitting to typical samples in rehearsal stage.
F Analogous Classes for Evaluation
We select the top 20% classes which are most likely to be confused with other classes from FewRel and MAVEN.
Specifically, select the following classes from FewRel: P706, P57, P22, P123, P127, P25, P17, P551, P206, P58, P40, P35, P26, P131, P937.
For MAVEN, we select Achieve, Telling, Legal- ity, Removing, Participation, Change_sentiment Attack, Motion, Creating, Defending, Arriving, Statement, Le- gal rulingl, Escaping, Competition, Percep- tion active Terrorism, Self motion, Emptying. Change, Manufacturing, Hold.
Becoming,
G Complete Experimental Results
The complete experimental results on all 10 tasks are shown in Table 5.
E Implementation Details
We implement InfoCL with PyTorch (Paszke et al.. 2019) and HuggingFacc Transformers (Wolf et al., 2020). Following previous work (Cui et al., 2021; Wang et al., 2022), we use BERT (Devlin et al., 2019) as encoder. PyTorch is licensed under the

---

FewRel
Models
Ti
T2
Ts
Ti
T6
To
T7
Ta
Tg
Tio
IDBR
97.9
91.9
86.8
83.6
80.6
77.7
75.6
73.7
71.7
68.9
KCN
98.3
93.9
90.5
87.9
86.4
84.1
81.9
80.3
78.8
76.0
KDRK
98.3
94.1
91.0
88.3
86.9
85.3
82.9
81.6
80.2
EMAR
98.1
94.3
92.3
90.5
89.7
88.5
87.2
86.1
84.8
83,6
RP-CRE
97.8
94.7
92.1
90.3
89.4
88.0
87.1
85.8
84.4
82.8
CRL
98.2
94.6
92.5
90.5
89.4
87.9
86.9
85.6
84.5
83.1
CRECL
97.8
94,9
92.7
90.9
89.4
87.5
85.7
84.6
83.6
82.7
ACA
98.3
95.0
92.6
91.3
87.6
87.0
86.3
CEAR
98.1
95.8
93.6
91.9
91.1
89.4
88.1
86.9
85.6
84.2
InfoCL
48.3
95.2+1.6
93.4+1.3
92.1
91.3
89.7
88.5
87.7
86.8
85.4.0.1
TACRED
Models
Ti
T2
T3
Ti
T5
To
T7
T
Tg
T10
IDBR
83.1
76.5
70.5
66.6
64.2
63.8
60.1
KCN
98.9
93.1
87.3
80.2
79.4
77.2
73.8
72.1
72.2
70.6
KDRK
98.9
93.0
89.1
80.7
79.0
77.0
74.6
72.9
72.1
70.8
92.0
87.4
84.1
78.3
76.6
76.1
RP-CRE
97.5
92.2
89.1
84.2
81.7
81.0
78.1
76.
75.0
75.3
CRL
97.7
93.2
89.8
84.7
84.1
81.3
80.2
79.1
79.0
78.0
CRECL
96.6
93.1
89.7
87.8
85.6
84.3
83.6
81.4
79.3
78.5
ACA
98.0
92.1
90.6
85.5
84.4
82.2
80.0
78.6
78.8
78.1
CEAR
94.3
92.3
88.4
84.5
82.2
81.1
80.1
79.1
InfoCL
96.3
92.442.1
88.9
87.3
83.9
82.4
82.0
79.7
78.4
78.2
MAVEN
Models
Ti
T2
Ts
Ti
Ta
T;
Ts
Tg
T10
IDBR
96.5
85.3
79.4
76.3
74.2
69.8
67.5
64.4
60.2
57.3
KCN
97.2
87.7
83.2
80.3
77.9
75.1
71.9
68.4
67.7
64.4
KDRK
97.2
88.6
84.3
81.6
78.1
75.8
72.5
69.6
68.9
65.4
EMAR
97.2
91.4
88.3
86.1
83.6
81.2
79.0
76.8
75.7
73,2
RP-CRE
96.7
91.8
88.2
86.5
81.4
79.4
77.1
76.0
73.6
CRL
96.0
90.7
87.1
84.8
82.9
80.7
78.7
76.8
75.9
73.7
CRECL
96.9
91.4
86.9
84.8
82.4
80.4
77.5
75.9
75.1
73.5
InfoCL
97.5
92.5
88.4
86.7-22
84.6
82.5
80.0
78.2
77.1
75.3
HWUU64
Models
T
T2
T
Ti
T
Ta
T
T
Tg
Tio
IDBR
96.3
93.2
86.5
82.5
80.2
78.0
76.2
KCN
98.6
94.0
90.7
90.4
87.0
84.9
84.4
83.7
82.7
81.9
KDRK
98.6
94.5
91.2
90.4
85.8
85.1
82.5
81.4
EMAR
98.4
94.4
91.4
89.5
88.2
86.3
86.2
85.5
83.9
83.1
RP-CRE
97.6
93.7
90.1
88.6
86.5
86.3
85.1
84.5
83.8
82.7
CRL
92.8
88.8
86.5
82.4
82.8
83.1
CRECI.
97.3
93.0
87.5
86.1
84.1
83.0
83.1
83.1
81.9
81.1
InfoCL
97.741.8
93.641.5
90.2+1.2
90.0
88.3
86.6
86.8
86.3
85.3
84.1
Table 5: Accuracy (%) on all observed classes after learning each task. The best results are marked in bold.


Reviews:
[{"review_id": "GtjXGdQ7gY", "reviewer": "Reviewer_EZuw", "summary": "", "questions": "Please see \"Reasons To Reject\"", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments! Please see our reponses below.\n\n**Q1:** Some related work should also be discussed.\n\n**A1:** Thank you for suggesting the inclusion of a discussion on other related work.\nIn our work, we have already compared our approach with CRL [2], which utilizes the same method as Co2L [1]. We will add a reference to Co2L [1] in our paper, as you suggested. Additionally, we would like to highlight two significant differences between our work and Co2L. First, we formally analyze the representation bias problem from an information bottleneck perspective and propose that maximizing $I(X_i;Z_i)$ can effectively mitigate this bias. Second, while Co2L employs a vanilla supervised contrastive loss for representation learning, we introduce two novel contrastive learning techniques: fast-slow and current-past contrastive learning. Our method not only enables the model to acquire more comprehensive representations but also improves its ability to retain previously learned knowledge. As shown in Table 2, our method outperforms CRL/Co2L on four datasets.\n\nRegarding CLASSIC, it is specifically designed for the domain-incremental scenario, where its primary goal is to facilitate knowledge transfer across tasks. This objective is distinct from our class-incremental scenario, and thus, the use of InfoNCE in CLASSIC is not directly comparable to our work.\n\n\n**Q2:** The motivation of Adversarial Memory Augmentation is unclear.\n\n**A2:** In replay-based CL methods, only a few instances of each class will be stored into the memory, which are then replayed during subsequent tasks. Therefore, the performance of replay is hindered by the overfitting issue. Adversarial data augmentation has emerged as a powerful technique for reducing overfitting [3, 4, 5]. Inspired by this, we incorporate adversarial memory augmentation to enhance the performance of current-past contrastive learning in the memory replay stage. We will improve the description according to your advice.\n\n**Q3:** What is the influence of the size of memory buffer?\n\n**A3:** The size of memory buffer is a trade-off between model performance and storage budget. With a larger memory, the model will have a better performance but it needs more storage space. The influence of memory size on model performance is often used to showcase the robustness of different methods [2, 6, 7]. We have conducted experiments and discussed the influence of memory size in Section 7.4.\n\n\n**Q4:** Do we have some strategy for improving the sampling strategy of memory buffer?\n\n**A4:** The sampling strategy is an important aspect for the replay-based CL methods. We use K-means based sampling strategy, which has demonstrated effectiveness in previous studies [2, 6, 7]. However, the primary focus of our paper is to analyze and address the issue of analogous class confusion from a representation learning perspective. As a result, we adopt the memory sampling strategy employed in prior work to ensure a fair comparison. In future work, we will explore more advanced sampling strategies to mitigate catastrophic forgetting.\n\n\nThanks again for the useful comments. We will continue to improve our paper according to your suggestions.\n\n[1] Co2L: Contrastive Continual Learning, CVPR 2021\n\n[2] Consistent Representation Learning for Continual Relation Extraction, ACL 2022\n\n[3] Adversarial Attacks on Deep-learning Models in Natural Language Processing: A Survey. arXiv\n\n[4] Adversarial Training for Free! NeuIPS 2019\n\n[5] FreeLB: Enhanced Adversarial Training for Natural Language Understanding, ICLR 2020\n\n[6] Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction, ACL 2021\n\n[7] Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation, EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well organized, the description of the representation bias, methodology is relative clearly to follow, also the perspective of the representation bias instead of sampling bias (e.g., coreset) is interesting.\n\n2. Compared with existing approach that also used contrastive learning appraoch (e.g., Co2L), this paper focused on fast and slow learning mechanism perspective to solve the bias problem, which is relative novel idea. I also see the phenomenon that different learning rate on encoder and linear layer for feature representation is useful, here the author could explain it more reasonable with more specfic work.\n\n3. The experimental result show the obvious improvement compared with existing SOTA, also the relative ablation study is specific and convinced.", "reasons_to_reject": "1. Some relative works[1,2] should also be discussed, which also applied contrastive learning (infoNCE) on continual learning to solve the bias or generalization work. The author applied the InfoNCE for optimize the fast and slow learning, so the discussion with other work that also applied the InforNCE should be considered.\n\n2. Some unclear description in the paper, e.g., in Sec 5.5, what's the motivation that used the Adversarial Memory Augmentation here for augmentation, instead of other approach in the NLP? In the experimental setting, what's the influence from the size of memory buffer? Do we have some strategy on improving the sampling strategy on memory buffer?  \n\n\n[1]. Co2L: Contrastive Continual Learning. CVPR 2021\n[2]. CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks. EMNLP 2012", "questions_to_authors": "Please see \"Reasons To Reject\"", "paper_topic_and_main_contributions": "This paper focuses on enable the model learn more sufficient representations to solve the information bottleneck issues,  they propose the fast-slow and current-past contrastive learning to learn from new tasks and old tasks, which maximum the mutual information. Experimenta result prove the effectiveness of this approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "6G9xIbtSbX", "reviewer": "Reviewer_AarQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and suggestions! Please see our responses below.\n\n**Q1:** Why does the fast-slow method work, especially in the information theoretic perspective? How to implement it?\n\n**A1:** The fast-slow contrastive method works since it can maximize the mutual information between the input sentences and the representations $I(X_i;Z_i)$. First, it uses the InfoNCE loss which can directly maximize $I(X_i;Z_i)$ [3]. Second, according to the information bottleneck theory [4], in the early phase of optimization, e.g., the first several epochs, $I(X_i, Z_i)$ is larger. As a result, the representations $Z_i$ from the slowly-updated branch will preserve more information about the input sentences. The fast-slow contrast can \"distill\" these information from the slow branch to fast branch to learn more comprehensive representations.\n\nThe code of the fast-slow method is provided in Answer 3. Due to space limitation, we did not provide the pseudo code and would like to put it in the appendix in the new version. That will make the method more readable. Thanks again for your suggestion!\n\n**Q2:** What are the differences with the related work [1] and [2]?\n\n**A2:** Our work, in comparison to OCM [1], shares a similar motivation of learning more comprehensive/holistic representations. We would like to provide clarifications on several key differences between our work and [1]:\n\n1. First, [1] focuses on online continual learning on image classification, which is different with our scenario.\n2. Second, for the first time, we provide a formal analysis of catastrophic forgetting from the perspective of the information bottleneck. This analysis sets us apart from [1].\n3. Finally, the contrastive learning in OCM relies on data augmentation of the image to ensure performance (According to the ablation study in [1], without the random-resized-crop augmentation, the performance will drop drastically). In contrast, we design fast-slow and current-past contrastive learning to get better representations. This contrastive learning design is also a distinguishing feature of our work.\n\nRegarding [2], it employs the out-of-distribution method CSI [5], which uses contrastive learning to conduct OOD detection, instead of helping the CL model learn more comprehensive representations.\n\n**Q3:** Code is not provided.\n\n**A3:** Our code and continual learning framework will be publicly released upon acceptance. Here is an anonymous version of the code of our method: https://anonymous.4open.science/r/InfoCL_EMNLP-DAD6/\n\n**Q4:** Information about computational resources is not provided.\n\n**A4:** We have recorded the computational resources used during the experiments. These results are listed below for your reference.\n\nAll experiments are conducted on a single NVIDIA A800 GPU. The execution time (minutes per. round) of our method and baselines are as follows:\n\n|        | FewRel   | TACRED   | MAVEN    | HWU64   |\n| ------ | -------- | -------- | -------- | ------- |\n| RP-CRE | 34.1     | 10.2     | 85.2     | 8.4     |\n| CRL    | 31.0     | **10.0** | **71.5** | **6.8** |\n| CRECL  | 40.3     | 12.0     | 79.6     | 8.0     |\n| ACA    | **28.5** | 12.6     | -        | -       |\n| CEAR   | 48.5     | 13.4     | -        | -       |\n| InfoCL | 29.2     | 11.6     | 78.4     | 7.0     |\n\nAlthough we utilize two contrastive learning losses and an adversarial mechanism, our method remains remarkable efficiency. This can be attributed to the parallel execution and gradient truncation in the slow branch of fast-slow contrast and the past branch in current-past contrast.\n\nThe GPU memory utilization depends on the batch size and the sentence length of the dataset. For FewRel, MAVEN and HWU64, the batch size is 32. For TACRED, the batch size is 16. The GPU memory utilization is: 4138MB for FewRel, 8926MB for TACRED, 14958MB for MAVEN, 3692MB for HWU64. These details will be put into the new version according to your suggestions.\n\n[1] Online Continual Learning through Mutual Information Maximization. ICML 2022\n\n[2] A Theoretical Study on Solving Continual Learning. NeurIPS 2022\n\n[3] On Variational Bounds of Mutual Information. ICML 2019\n\n[4] Opening the Black Box of Deep Neural Networks via Information. ICLR 2018\n\n[5] Csi: Novelty detection via contrastive learning on distributionally shifted instances. NeuIPS 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The information theoretic analysis is interesting\n2. The experiment is thorough and shows the effectiveness of the proposed method", "reasons_to_reject": "1. The writing can be improved for clarity. For instance, i) it's not clear why fast-slow method works especially in the information theoretic perspective, and ii) it's not clear how to implement it. Pseudo-code may be informative.\n2. Similar methods have been already explored. [1] proposes mutual information maximization for holistic representation using contrastive loss. [2] proves the role of out-of-distribution of classes from future and previous tasks in class-incremental learning and proposes a method using contrastive learning.\n3. Code is not provided.\n4. Information about computational resources is not provided. Should provide e.g., how efficient is the proposed method, and how much GPU memory does it require?\n\n[1] Online Continual Learning through Mutual Information Maximization. ICML 2022  \n[2] A Theoretical Study on Solving Continual Learning. NeurIPS 2022\n\nI am willing to change my score if the authors address my concerns in rebuttal", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a replay-based continual learning method for text classification problem in the class-incremental learning setting. Based on insights from information theoretic perspective, the authors explain forgetting between cross-task analogous classes and present a training approach for comprehensive representation learning. The training process consists of training the network using fast-slow contrastive learning and current-past contrastive learning with a memory buffer. The proposed method is compared with several recent CL methods in NLP and demonstrates competitive performance.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Refer to reasons to reject"}, "extra_text": {}}, {"review_id": "g5sjtfDh26", "reviewer": "Reviewer_JheW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions! \n\nAs you pointed out, our method has the potential to be applied to a wide range of classification tasks. **In fact, we have thoroughly validated its effectiveness on numerous tasks.** However, due to the constraints of space in our paper, we were only able to include four representative classification tasks and conducted extensive experiments on them. These tasks were carefully selected to showcase the versatility and robustness of our method. Here we would like to explain the principles for selecting these four tasks and provide more experimental results on three additional tasks, which we hope can further demonstrate the effectiveness of our method.\n\nIn this paper we mainly focus on continual text classification in a class-incremental setting, which requires the datasets to have a vast number of classes, e.g., 80 classes for FewRel and 120 classes for MAVEN. Therefore, some classification tasks which only contain 2 or 3 classes, such as sentimental analysis, are not suitable for the class-incremental setting. On the other hand, the four classification tasks we used include relation classification, event classification, and intent detection, which have already covered three typical classification scenarios and are widely used in previous work.\n\nHere we provide the results on three more tasks, CLINC150 [1], WebRED [2] and ReTACRED [3]. For all additional datasets, we simulate 10 class-incremental tasks by randomly dividing all classes of the dataset into 10 disjoint sets. The number of new classes in each task for CLINC150, WebRED and ReTACRED are 15, 10, 4 respectively. All hyperparameters of the experiments are the same as FewRel. The reported results are the average of 5 randomly sampled task sequences. These results will be added to the appendix. \n\n**As shown, our method outperforms baselines on six datasets (including three in the paper). We hope the extensive experiments can validate the effectiveness of our method.**\n\nCLINC150 [1] is an intent detection dataset with 150 intent classes. Each class has 100 instances for training and 20 instances for test. Here are the experimental results on CLINC150.\n\n| Models        | T1       | T2       | T3       | T4       | T5       | T6       | T7       | T8       | T9       | T10  |\n| ------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | ---- |\n| RP-CRE        | 98.8     | 98.5     | 97.7     | 97.0     | 95.9     | 95.9     | 95.4     | 95.0     | 94.8     | 94.3 |\n| CRL           | **99.1** | 98.4     | 97.2     | 96.7     | 95.6     | 95.5     | 95.4     | 95.0     | 94.7     | 94.5 |\n| CRECL         | 98.4     | 97.9     | 96.3     | 96.0     | 95.3     | 94.9     | 94.8     | 94.7     | 94.4     | 94.3 |\n| InfoCL (Ours) | 99.1     | **98.8** | **98.1** | **97.5** | **96.6** | **96.2** | **96.0** | **95.6** | **95.4** | **95.1** |\n\nWebRED [2] is a severely long-tailed relation classification dataset with more than 500 classes. We use the data of the top-100 frequent classes. For each class, we randomly split the data into train and test set by 4:1. Here are the experimental results on WebRED.\n\n| Models        | T1       | T2       | T3       | T4       | T5       | T6       | T7       | T8       | T9       | T10  |\n| ------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | ---- |\n| RP-CRE        | 98.8     | 92.5     | 85.6     | 85.3     | 80.3     | 76.2     | 72.9     | 71.1     | 66.9     | 65.4 |\n| CRL           | **98.9** | 91.7     | 83.0     | 81.4     | 76.2     | 71.8     | 68.3     | 66.3     | 63.0     | 61.7 |\n| CRECL         | 98.6     | 92.7     | 85.0     | 83.3     | 77.4     | 73.4     | 70.3     | 68.6     | 65.2     | 64.5 |\n| InfoCL (Ours) | 98.8     | **92.9** | **86.8** | **85.4** | **80.7** | **76.7** | **73.9** | **71.8** | **68.5** | **67.1** |\n\nReTACRED [3] is a re-annotated version of TACRED which can be used to perform reliable evaluation of relation classification models. Since the dataset has 39 classes which is not a multiple of 10 (the length of the task sequence), we randomly sample a task and allocate 3 classes for it. Here are the experimental results on ReTACRED.\n\n| Models        | T1       | T2       | T3       | T4       | T5       | T6       | T7       | T8       | T9       | T10  |\n| ------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | ---- |\n| RP-CRE        | 96.5     | 93.0     | 92.4     | 89.5     | 88.5     | 87.6     | 86.2     | 85.0     | 83.9     | 83.8 |\n| CRL           | 94.9     | 91.8     | 88.5     | 86.0     | 84.0     | 81.2     | 79.6     | 78.1     | 77.5     | 78.9 |\n| CRECL         | 96.4     | 89.9     | 87.1     | 83.0     | 82.3     | 80.6     | 82.8     | 81.7     | 81.7     | 82.7 |\n| InfoCL (Ours) | **97.1** | **94.6** | **93.4** | **91.5** | **89.7** | **88.5** | **88.0** | **86.8** | **86.0** | **85.6** |\n\n[1] An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction, EMNLP 2019\n\n[2] WebRED: Effective Pretraining And Finetuning For Relation Extraction On The Web, arXiv 2102\n\n[3] Re-TACRED: Addressing Shortcomings of the TACRED Dataset, AAAI 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 The method is well-motivated and the analysis looks solid.\n2 The experiments on the four classification tasks (fewrel, tacred, maven and hwu64) show the superior performance.", "reasons_to_reject": "1 The proposed method is a general method that can be applied to many different classification task. However, in the paper, only four classification tasks are considered. More experiments should be added to validate its effectiveness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on continual learning for text classification in the class-incremental setting and propose the method, InfoCL, which utilizes fast-slow and current-past contrastive learning to learn more comprehensive representations and alleviate representation corruption. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method using fast-slow contrastive learning with a memory buffer for continual learning.  Experimenta result prove the effectiveness of this approach.

All reviewers find the information theoretic analysis insightful and interesting. The experimental results (with the additional results provided in response) are adequate. 

The discussion and writing could be improved (e.g. motivation of fast-slow contrastive learning). The authors may want to discuss related work (e.g. Co2L, OCM) and incorporate the response. We highly encourage authors add additional results provided in response.
Decision: Accept-Findings