A Spectral Viewpoint on Continual Relation Extraction
Authors: Huy Huu Nguyen
Chien Van Nguyen
Linh Ngo Van
Anh Tuan Luu
Thien Huu Nguyen
Venue: EMNLP 2023 Findings (2023)
Keywords: Relation Extraction
Information Extraction
Continual Learning
Spectral Analysis
TL;DR: We present a novel spectral analysis and regularization method for the catastrophic forgetting issue in continual learning of relation extraction.

Abstract: Continual Relation Extraction (CRE) aims to continuously train a model to learn new relations while preserving its ability on previously learned relations. Similar to other continual learning problems, in CRE, models experience representation shift, where learned deep space changes in the continual learning process, which leads to the downgrade in the performance of the old tasks. In this work, we will provide an insight into this phenomenon under the spectral viewpoint. Our key argument is that, for each class shape, if its eigenvectors (or spectral components) do not change much, the shape is well-preserved. We then conduct a spectral experiment and show that, for the shape of each class, the eigenvectors with larger eigenvalue are more preserved after learning new tasks which means these vectors are good at keeping class shapes. Based on this analysis, we propose a simple yet effective class-wise regularization that improve the eigenvalues in the representation learning. We observe that our proposed regularization leads to an increase in the eigenvalues. Extensive experiments on two benchmark datasets, FewRel and TACRED, show the effectiveness of our proposed method with significant improvement in performance compared to the state-of-the-art models. Further analyses also verify our hypothesis that larger eigenvalues lead to better performance and vice versa.

A Spectral Viewpoint on Continual Relation Extraction
Huy Huu Nguyen', Chien Van Nguyen', Linh Ngo Van"
Luu Anh Tuan and Thien Huu Nguyen2
VinAl Research, Vietnam
2 Department of Computer Science, University of Oregon, Eugene, OR, USA
3 Hanoi University of Science and Technology, Hanoi, Vietnam
Nanyang Technological University, Singapore v. wwy.hh1evinii. io, (oo,,(chi {chienn, thienhcs ) .du
1 linhnv@soict hust. edu. vn, anhtuan. luu@ntu. ceweenttuuduu.... sg
Abstract
the given context. To address this problem, several methods have been proposed and have achieved remarkable results (Nguyen and Grishman, 2015; Zhou et al., 2016; Pouran Ben Veyseh et al., 2020; Zheng et al., 2023). Nevertheless, most previous RE studies only considered the traditional setting where the set of relations is pre-defined and fixed during the training and testing phases. This setting is not practical as new relations of interest might emerge during deployment time of RE systems in practice, requiring the models to adapt their opera- tions to accommodate new types.
Continual Relation Extraction (CRE) aims to continuously train a model to learn new re- lations while preserving its ability on previ- ously learned relations. Similar to other contin- ual learning problems, in CRE, models experi- ence representation shift, where learned deep space changes the continual learning pro- cess, which leads to the downgrade in the per- formance the old tasks. In this work, we will provide an insight into this phenomenon un- der the spectral viewpoint. Our key argument is that, for each class shape, if its eigenvec- tors (or spectral components) do not change much, the shape is rell-ereserved. We then conduct a spectral experiment and show that, for the shape of each class, the eigenvectors with larger eigenvalue are more preserved after learning new tasks which means these vectors are good keeping class shapes. Based on this analysis, we propose a simple yet effec- tive class-wise regularization that improve the eigenvalues in the representation learning. We observe that our proposed regularization leads an increase in the eigenvalues. Extensive periments on two benchmark datasets, FewRel and TACRED, show the effectiveness of our proposed method with significant improvement in performance compared to the state-of-the models. Further analyses also verify our hy- pothesis that larger eigenvalues lead to better performance and vice versa.
Recently, Continual Relation Extraction (CRE) has attracted considerable attention in the litera- ture (Wang et al., 2019; Han et al., 2020; Cui et al., 2021; Zhao et al., 2022), aiming to learn new relations from incoming data. CRE meth- ods often suffer from Catastrophie Forgetting (CF) phenomenon where their performance on previous relations reduce significantly when learning new relations. One key issue that causes CF' is the repre- sentation shift that occurs when models learn new knowledge. Existing works have made significant progress in mitigating this phenomenon (Phan et al., 2022; Van et al., 2022; Nam et al., 2023). Different from those works, analyze the representation shift using spectral decomposition. We argue that the eigenvectors play an important role in capturing the shape of each class. As such, the eigenvec- tors of class shape adjust much after exposing new tasks, it means the class shape also changes significantly. We give an intuition for our argument in figure 1. Based on this reasoning, we conduct an experiment to see how eigenvectors change in the continual learning process. We find that the eigenvectors with larger eigenvalue are preserved well when learning new tasks. In other words, these eigenvectors are good at keeping class shapes and hence avoiding the representation shift.
1 Introduction
Relation Extraction (RE) problem is the basis of many NLP tasks, such as Question Answer- ing (Sorokin and Gurevych, 2017), Knowledge Graph Construction (Luu et al., 2014, 2016; Bal- dini Soares et 2019), Definition Extraction (Vey- seh al., 2020) and event relation extraction (Man et al., 2022; Lai et al., 2022). In particular, re- lation extraction system is expected to classify se- mantic relation between two entities mentioned in
Based on these experiments, we propose class- wise regularization that aims to boost the eigen-
"Corresponding Author

---

Figure I: demonstration for impact of adjustment of first component vector. class shape before updating is blue and the updated one green. The first component before updating is the dashed line while the one is the solid line. Left: The small adjustment keeps the shape unchanged. Right: The large adjustment leads to a bad representation shift.
values. This regularization is inspired by some recent works on dimensional collapse and feature decorrelation (Bardes et al., 2022), which try to make different dimensions of the embedding correlated and avoid small eigenvalues effectively. To enhance understanding, we theoretically show why feature decorrelation helps increase the eigen- values.
ing set Dj: and relation set Rg. Formally, each task Ty is supervised classification task with training set Dr containing N samples {(z,, 21) } where #4 is the input sentence and entity pair, and y is the relation label. As general goal of continual learn- ing, a CRE system expected to perform well on both the current task and all previous tasks. Hence, models are required to classify each relation into a known relation set Rig where R& = - R.
In summary, our work makes the following con- tributions:
We adapt an episodic memory module to store a few examples of historical tasks. Each relation has its memory module, i.e., memory module for relation is set M, {(x,, v storing O samples that are representatives for relation r, where O is the pre-defined number of samples be stored.
We conduct a thorough analysis and provide an insightful view of the representation space in continual relation extraction setting via the spectral perspective. We find that the spec- tral components with larger eigenvalues are less forgettable and useful in preserving class shapes.
Base Model and Objective Function: For mod- elling details, we use one pretrained language model, i.e., BERT (Devlin et al., 2018) as fea- ture extractor and learnable linear classifier on top of the feature extractor. Following previous work (Han al., 2020; Cui al., 2021), use the CE loss LCF as the objective function for training task Ty:
A simple yet effective class-wise feature decorrelation regularization is proposed with the goal of boosting the eigenvalues of the rep- resentation for each class. Furthermore, we theoretically demonstrate why feature decor- relation helps boost eigenvalues.
Our extensive experiments demonstrate that our model achieves new state-of-the-art performance on two CRE benchmarks, FewRel and TACRED. We also conduct further analysis to verify the effi- cacy of our idea in boosting the cigenvalues.
De
LCE =
log P())(z))
where (I,, wi) E De
Spectral Analysis on The Representation Space
2 Problem Formulation and Base Model
3.1 Spectral Analysis Setups
Problem Formulation: In continual relation ex- traction, models are trained on a sequence of tasks T2,.., }h}, where the k-th task is one tradi- tional relation extraction task and has its own train-
Inspired by (Zhu et al., 2021), in this section we aim to explore the changes in class shapes be- fore and after learning new tasks. Concretely

---

FewRol P410
TACRED 1ACRED_per.cties of residence
Cosine valves of carrepoding Shate
Costne values cancspoding Sngle
2.6
0.5
0.4
0.4
0.3
9.2
Eigenvalue rank index
Elgenvalue Index
Figure 2: Absolute cosine values of corresponding angles two classes P410 and per:cities of residence, which we choose randomly from the first task of FewRel and TACRED, respectively.
Few Rel
Model
T1
T2
T3
T5
T6
T8
T10
EA-EMR
89.0
69.0
59.1
54.2
47.8
46.
43.1
40.7
38.6
35.2
CML
91.2
74.8
68.2
58.2
53.7
50.4
47.8
44.4
43.1
39.7
EMAR BERT
98.1
94.3
92.3
90.5
89.7
88.5
87.2
86.1
84.8
83.6
RP-CRE
97.8
94.7
92.1
90.3
89.4
88.0
87.1
85.8
84.4
82.8
CRL
98.1
94.6
92.5
90.5
89.4
87.9
86.9
85.6
84,5
83.1
EMAR + ACA
98.3
95.0
92.6
91.3
90.4
89.2
87.6
87.0
86.3
84.7
Ours
98.3
94.7
93.1
91.4
90.6
89.4
87.9
86.9
85.4
84.3
TACRED
Model
TI
T2
T3
T5
Т6
T8
T10
EA-EMR
47.5
40.1
38.3
29.9
24
27.3
26.9
25.8
22.9
19.8
CML
57.2
51.4
41.3
39.3
35.9
28.9
27.3
26.9
24.8
23.4
EMAR BERT
98.3
92.0
87.4
84.1
82.1
80.6
78.3
76.6
76.8
76.1
RP-CRE
97.5
92.2
89.1
84.2
81.7
81.0
78.1
76.1
75.0
75.3
CRL
97.7
93.2
89.8
84.7
84.1
81.3
80,2
79.1
79.0
78.0
EMAR + ACA
98.0
92.1
90.6
85.5
84.4
82.2
80.0
78.6
78.8
78.1
Ours
98.1
93.8
89.8
85.8
84.4
83.4
81.6
79.9
79.7
79.1
Table 1: Accuracy (%) on all observed relations (which will continue to accumulate over time) at the stage of learning current task. Other results are directly taken from (Wang al., 2022). We show the best results in boldface and the second best ones in underlines (Note that all models do not use class augmentation except EMAR + ACA)
X
6o
1M sott Mlaiie
ddd tox
1,56
Figure 3: A visualization of relation representation learned from task of the test set by CRL (Top) and our model (Bottom) different task
speaking, we monitor how every eigenvector of deep feature space is adjusted after updating new knowledge thanks to spectral decomposition. For- mally, we first train a feature extractor on dataset
Datd = {(x,, Bi) of current task, denoted as Fald- After finetuning Fas on new dataset D ((ai, y:) we obtain an updated ex- tractor, denoted as Fnew- To measure the sensi-

---

FewRei_ P410
TACRED TACRED_pendkies of residence
correspoging angles
0.175
0.150
0.4
0.125
0.3
0.100
Cosine values
0.2
0.075
0.050
6.1
0.025
8.0
0.000
100
200
300
400
50
100
150
200
250
300
Bl Egenvalue rank index
El gennvaluee rank Index
FewRei_ _P410
TACREO_percitiie per:iities of residennce
25
25
20
20
Eggevva ue5
Eigevviiees
155
1.5
150
250
genvaluu rank index
Eggnvaauu rank index
Absolute cosine values of corresponding angles of two classes P410 and per:cities of residence between our model and the SOTA. Boftom: Distribution of eigenvalues of two classes P410 and per:cities of residence for our model and the SOTA one. (Lines in cosine figures are smoothen for better view)
Figure 4: Top:
tivity of learned class shapes, we use old dataset old and map into old space and new space us- ing Fat and Fnew respectively. Let Fa)( and Fuew (x:) be mapped features of sample x; on old and new spaces, we do spectral decomposition on the correlation matrix as follows:
sponding the same eigenvalue index. The cosine value of the corresponding angle computed as follows:
cos()by) !1) (uold.j Upew,)) |Ideoa|| [neew
(uold Unewy)
where Unldy is the j-th eigenvectors with the j-th largest eigenvalue in the old feature space, and sim- ilarly for Unew Additionally, because ||uaaaa]| =
Σ F(x)) = Σn u
Unew:y || the (2) trivial.
where in the L.H.S, F is the mapped feature JF after normalizing, while in the R.H.S the Aÿ is the j-th eigenvalue in the sorted list of eigen- values in descending order and uj is its correspond- ing eigenvector. Applying this decomposition on both old representations and new ones, Fota(z;)) and (x;), we obtain two set of eigenvectors (uold,1:.... ..., ]dddd] and [Unow,1,,... unow.d).
3.2 Changes in eigenvectors
We conduct an experiment to explore which part, iee, which direction the representation space car- ries helpful knowledge in CRE setting. We lever- age the first two tasks of the FewRel and TACRED datasets to examine the feature space. Figure 2 shows the absolute cosine values of the correspond- ing angles between the old and new eigenvectors. It clear that, after updating, eigenvectors with large eigenvalues adjust insignificantly and are therefore good at preserving the shape of the data distribu- tion, while ones with small eigenvalues tend to
In order to investigate the change of the eigenvectors, Zhu et al. (2021) introduced the definition of Corresponding Angle: given two groups of eigenvectors, [uold,1,... uoldd) and [unew, ) corresponding angle presents the angle between two eigenvectors corre-

---

move towards some different direction and can be considered as noisy and forgettable directions.
4.2 Experimental Results
Base results: We report the results of our proposed method and some baselines in Table 1. We give the details of the baselines in A.2. It can be seen from the table that, compare with other models which do not use class augmentation ACA (Wang et al., 2022), our model sets new SOTA on two CRE benchmark datasets. Our model performs better than the current SOTA, CRL by 1.2 % on FewRel and 1.1% on TACRED. Our model without ACA performs on par with the EMAR + ACA on FewRel and still outperforms it remarkably on TACRED. Effectiveness of the regularizer on eigenval- ues: We run the same spectral analysis on both our model and see that it produces larger eigen- values compared to the SOTA model, CRL. The reason we do not conduct experiments with ACA because they use synthesis classes that leads dif- ferent behaviours in deep space compared nor- mal methods. We display changes in class shapes using t-SNE in figure 3 and also provide figures about the corresponding angles in figure 4 to verify the effectiveness of our regularizer in keeping class shapes. In detail, in figure 3, we show the represen- tation of the test set data from four classes of the first task of TACRED after learning task 0, 3, 6, 9. Our discussion is that: in long run, the class shapes in CRL tend be thinner and longer while ones trained with our model do not suffer it. The gap in cigenvalues (Figure 4) shows that our feature decorrelation regularization actually increases the eigenvalues and the cosine value of corresponding angles.
3.3 Class-wise Decorrelation Feature Regularization
With the previous finding, our goal is to boost the cigenvalues of the distribution shape of each class. Hua et al. (2021) indicated the relationship between strong correlation and dimensional collapse, which leads to the idea of feature decorrelation. Based on this finding, we propose simple technique that enlarges eigenvalues. Formally, in each batch, for a given label r, the correlation matrix of is estimated as:
K = - -T,
where 2,(7 is the encoder output representation on the i-th data point of class r while the mean vector of r is denoted as Z" n ""=(",) (z(")). The data representations is explicitly constrained by a class-wise feature decorrelation regularization:
E rC1l ##)
LPD =
We give the theoretical proof on how feature decor- relation enlarges the eigenvalues in Appendix B.I. The overall objective function is given by:
Loverall LCE + pLFD,
where μ are the hyper-parameters controlling the importance of each term in the overall loss.
4 Experiments
5 Conclusion
4.1 Experiments Setups
Datasets: Similar to (Zhao et al., 2022), we per- form experiments two standard English benchmark datasets: FewRel and TACRED, More details about two datasets is given at A.1
In this work, we conduct thorough study on the change of distribution shapes in the CRE problem through spectral analysis and observe that eigen- vectors with larger eigenvalues are less forgettable. Based on these findings, we introduce a class-wise feature decorrelation regularization with the goal of boosting eigenvalues. Our theoretical analysis shows the effectiveness of our proposed method in handling low eigenvalues. Furthermore, our exten- sive experiments on two benchmark datasets show the superior performance of our method. In the future, will extend our analysis other Infor- mation Extraction tasks, such as Entity Mention Detection (Nguyen et al., 2016) and Event Classifi- cation (Lai et al., 2020; Hao et al., 2023).
Implementation Details: Following (Cui et al., 2021), relations are randomly divided into 10 clus- ters to simulate 10 tasks. We use average accuracy on all seen tasks to measure model performance on the CRE task as previous work (Han et al., 2020). In terms of environment and configuration, we train all models on the same task sequence used in (Cui al., 2021; Zhao al., 2022) by setting exactly the same random seed to make fair comparison. For the convenience of reproduction we provide the details of parametesssssrss settings at A.3.

---

Limitations
in Natural Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguisties.
Although we do an analysis to indicate that eigenvectors with larger eigenvalues carry more helpful features, there is lack of in- terpretation in these directions. It would be preferable to provide some examples to demonstrate the superiority of these directions and make them more understandable.
Anran Hao, Anh Tuan Lau, Siu Cheung Hui, and Jian Su. 2023, A contrastive learning framework for event detection via semantic type prototype representation modelling. Neurocomputingg 556: 126613.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang. Sucheng Ren, and Hang Zhao. 2021. On feature decorrelation in self-supervised learning. arXiv e- prints, pages arXiv-2105,
Like all prior works about CRE, we only focus on classifying pair of entities into relation types when given those entities in a context. To aim for a complete solution for CRE, the problem of named entity recognition should be studied in the continual learning scenarios.
Viet Lai, Hieu Man, Linh Ngo, Franck Dernoncourt, and Thien Nguyen. 2022. Multilingual subevent relation extraction: A novel dataset and structure induction method. In Findings of the Association for Computa- rional Linguisties: EMNLP 2022, pages 5559-5570.
Viet Dac Lai, Franck Dernoncourt, and Thien Huu Nguyen. 2020. Exploiting the matching information in the support set for few shot event classification. In Proceedings of the 24th Pacifie-Asia Conference on Knowledge Discovery and Data Mining (PAKDD).
References
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2895- 2905, Florence, Italy. Association for Computational Linguistics.
Anh Tuan Luu, Jung-jae Kim, and See Kiong Ng. 2014. Taxonomy construction using syntactic contextual evidence. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), P) pages 810-819.
Adrien Bardes, Jean Ponce, and Yann LeCun. 2022. VICReg: aaaaee-ennnarranne-oovaaainna regulariza- tion for self-supervised learning. In International Conference on Learning Representations.
Anh Tuan Luu, Yi Tay, Siu Cheung Hui, and See Kiong Ng. 2016. Learning term embeddings for taxonomic relation identification using dynamic weighting neu- ral network. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Process- ing, pages 403-413.
Li Cui, Deqing Yang, Jiaxin Yu, Chengwei Hu, Jiayang Cheng, Jingjie Yi, and Yanghua Xiao. 2021. Refin- ing sample embeddings with relation prototypes to enhance continual relation extraction. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 232-243, Online. Association for Computational Linguistics.
Hicu Man, Nghia Trung Ngo, Linh Ngo Van, and Thien Huu Nguyen. 2022. Selecting optimal con- text sentences for event-event relation extraction. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages
Le Hai Nam, Nguyen Trang, Ngo Van Linh, Huu Nguyen Thien, and Than Khoat. 2023. Con- tinual variational dropout: A view of auxiliary local variables continual learning. Machine Learning.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1810.008.
Thien Huu Nguyen and Ralph Grishman. 2015. Rela- tion extraction: Perspective from convolutional neu- ral networks. In Proceedings of the Isi Workshop on Vector Space Modeling for Natural Language Pro- cessing, pages 39 48, Denver, Colorado. Association for Computational Linguistics.
Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Contin- ual relation learning via episodic memory activation and reconsolidation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 6429-6440, 6440, Online. Association for Computational L Linguistics.
Thien Huu Nguyen, Avirup Sil, Georgiana Dinu, and Radu Florian. 2016. Toward mention detection bustness with recurrent neural networks. Proceed- ings of IJCAI Workshop on Deep Learning for Artifi- cial Intelligence (DLAI).
Xu Han, Hao Zhu, Pengfei Yu. Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018 FewRel: A large-scale supervised few-shot relation classification dataset with ofate-of-the-an art evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods
Hoang Phan, Anh Phan Tuan, Son Nguyen, Ngo Van Linh, and Khoat Than, 2022. Reducing catastrophic

---

forgetting in neural networks via gaussian mixture ap- proximation. In Pacific-Asia Conference on Knowl- edge Discovery and Data Mining, pages 106-117, 17.
35-45, Copenhagen. Denmark. Association for Com- putational Linguistics.
Springer.
Kang Zhao, Hua Xu, Jiangong Yang. and Kai Gao. 2022. Consistent representation learning for continual re- lation extraction. In Findings of Association for Computational Linguistics: ACL 2022, pages 3402- 3411 1, Dublin, Ireland. Association for Computational Linguisties.
Amir Pouran Ben Veyseh, Franck Dernoncourt, Dejing Dou, and Thien Huu Nguyen. 2020. Exploiting the syntax- model consistency for neural relation extrac- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8021-8032, Online. Association for Computational Linguistics.
Yandan Zheng, Anran Hao, and Anh Tuan Luu. 2023. Jointprop: Joint semi-supervised learning for entity and relation extraction with heteroge- neous graph-basee propagation. arXiv preprint arXiv:2305.15N5
Daniil Sorokin and Tryna Gurevych. 2017. Context- aware representations for knowledge base relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1784-1788 1789, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. 2016. Attention-based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 207- 21 12, Berlin, Germany, Association for Computational Linguistics.
Linh Ngo Van, Nam Le Hai, Hoang Pham, and Khoat Than. 2022. Auxiliary local variables for improving regularization/prior approach in continual learning. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 16-28. Springer.
Fei Zhu, Zhen Cheng. Xu-yao Zhang, and Cheng- lin Liu. 2021. Class-incremental learning dual augmenta- tion. In Advances in Neural Information Processing Systems, volume 34, pages 14306 14318. Curran As- sociates, Inc.
Amir Pouran Ben Veysch, Franck Demoncourt, De- jing Dou, and Thien Huu Nguyen. 2020. A joint model for definition extraction with syntactic con- nection and semantic consistency. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI).
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo. Shiyu Chang, and William Yang Wang. 2019. Sen- tence embedding alignment for lifelong relation ex- traction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 796-806, Minneapolis, Minnesota. Association for Computational Linguistics.
Peiyi Wang, Yifan Song, Tianyu Liu, Binghuai Lin, Yunbo Cao. Sujian Li, and Zhifang Sui. 2022. Learn- ing robust representations for continual relation ex- traction via adversarial class augmentation.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2019. Huggingface's 's transformers: State-of- the-art natural language processing, arXiv preprint 110.0.191..03771.
Tongtong Wu, Xuekai Li, Yuan-Fang Li, Reza Haf- fari, Guilin Qi, Yujin Zhu, and Guoqiang Xu. 2021. Curriculum-man meta learning for rodus- continual relation extraction. In AAAI Conference Artificial Intelligence.
Yuhao Zhang, Victor Zhong, Dangi Chen. Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages

---

A Experiments Details
Description of computing infrastructure used: In this work, we use single Tesla A100 GPU with 100GB memory operated by Ubuntu Server 18.04.3 LTS for all ex- periments. PyTorch 2.0 and Huggingface- Transformer 4.29.2 (Apache License 2.0) (Wolf et al., 2019) are used to implement the models.
A.1 Datasets
We perform our experiments on two English bench- mark datasets: FewRel and TACRED The training- test-validation split ratio is 3:1:1. FewRel (Han al., 2018) is RE dataset that contains 80 re- lations and 56,000 samples in total. To make continual RE dataset, we follow the settings in (Wang et al., 2019) and use the original train set and validation set for our experiments. TACRED (Zhang et al., 2017) 7) is an imbalanced RE dataset that contains 42 relations (including no velaatooo and 106,264 samples. Following the experiment settings by (Cui et al., 2021), to reduce the imbal- ance, we remove the mo_relation class and limit the number of training samples of each relation to 320 and the number of test samples of each relation to 40.
Average runtime: Training each round on average takes 88 minutes for FewRel dataset and 18 minutes for TACRED dataset. For each task, in both Initial training for new task phase and Memory replay phase we train the model for 10 epochs following the experiments by (Zhao et al., 2022)
Number of parameters in the model: There are approximately 110 million parameters total, including 110 million from the feature extractor and very few parameters, compared to the feature extractor, from the softmax clas- sifier.
A.2 Baselines
We compare our proposed methods with several otate-of-the-art CRE baselines:
EA-EMR (Wang et al., 2019) proposed a memory replay and embedding alignment technique to mitigate catastrophic forgetting.
Bounds for each hyper- :eprm: To tune the proposed objective function, we choose l from [0.05, 0.1, 0.2, 0.25, 0.5, 1.0, 1.25]. All the syper-parametery selected based on Fl scores on the validation set.
EMAR (Han al., 2020) introduced a mem- ory activation and reconsolidation to preserve old knowledge.
The method of choosing hyper-parameter values and the criterion used lect among them: We choose the hyper- parameters for the proposed model using man- ual tuning.
CML (Wuetal, al., 2021) proposed currieulum- meta learning method to handle the order- sensitivity and catastrophic forgetting in CRE.
RP-CRE (Cui et al., 2021) learns a mem- ory network with the goal of refining sample embeddings with relation prototypes, thereby avoid catastrophic forgetting.
Hyperparameter configurations for best- performing models: In our model, we use the following values for the hyper- parameters: learning rate le-5 for the encoder and 1e-3 for the classifier with the AdamW optimizer; 32 for the mini-batch size; / = 0.05 for FewRel and μ 0.1 for TACRED. The memory size of each task is 10.
CRL (Zhao et al., 2022) maintains learned knowledge by introducing contrastive replay mechanism and knowledge distillation.
ACA (Wang et al., 2022) proposed an adver- sarial class augmentation mechanism to make learned models more robust representations.
B
The Feature Decorrelation Regularizer
B.1
Theoretical proof for the effect of our loss on eigenvalues
A.3 Reproducibility Checklist
Source code with specification of all depen- dencies, including external libraries: Our source code with necessary documentation for reproducibility will be released upon accep- tance of the paper.
Without the loss of generality, give proof for our loss LED on specific class r, denoted as L FDD
L 1
Σ
( ij
$#

---

Suppose that the correlation matrix of r: K6) Raxd defined in subsection 3.3, have j as eigenvalues, we have:
d
A = trace(K(n) =d
Above equalities hold because the entries the main diagonal of correlation matrix I's. Now, consider our proposed loss LFD we have:
D
(r)
$#j
(2)
= [I((rr diag( K 11
= -1K() lall?
Recall that the Frobenius norm is equal to the sum of its squared eigenvalues. The problem now is to find out which is the eigenvalue of the matrix (r() Ia. In fact, its eigenvalues are {A: 1) Therefore, have:
|K(()) Tall
(A: 1)2
(3)
1=1
Moreover, by equation I, we rewrite above equa-
tion as:
d
[K(]) 12l13
Aj
2
(4)
Next, plug equation 4 back equation 2, have:
(
L LFD
K
1.j
(#)
(5)
(A:
X1)²
i=1
1=1
Note that our regularizer increases the number of cigenvectors with large eigenvalues by penalizing large eigenvalues. fact, reduces top eigenval- ues and boosts the eigenvalues with lower indices.


Reviews:
[{"review_id": "un2b4MHJ8F", "reviewer": "Reviewer_oY2D", "summary": "", "questions": "Overall, this study demonstrates a robust and well-designed approach. However, one aspect that could be addressed is the absence of a name for the proposed method.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. Please find below our responses for your questions and concerns.\n\n**Reviewer**: *\"To further strengthen this work, it would be valuable to include a comprehensive discussion on the comparison of computational costs between the proposed approach and existing state-of-the-art methods.\"*\n\n**Our Response**:\n\nThank you for your suggestion. We agree with the reviewer that a comparison of computational costs will be helpful for further understand. In the table below, we report the training time of our models (with and without the feature regularization) and the previous state-of-the-art model CRL over two datasets TACRED and FewRel for 10 epochs.\n\n| Model | TACRED | FewRel |\n| :--------------------------------------- | :---------: | :---------: |\n| Our full model | 22 mins | 92 mins |\n| Our full model w/o regularization | 21 mins | 88 mins |\n|CRL (sota model) | 27 mins | 115 mins |\n\nAs can be seen, CRL consumes much more time for training as it employs an expensive contrastive loss and updated memory bank. The feature regularization does increase the training time for our model a bit, but it is not very significant. We will include this comparison in our final version.\n\n**Reviewer**: *\"Overall, this study demonstrates a robust and well-designed approach. However, one aspect that could be addressed is the absence of a name for the proposed method.\"*\n\n**Our Response**:\n\nThank you for your suggestion. We will revise our paper and introduce a name for our method in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written.\n2. The spectral viewpoint is interesting and the proposed methodology is well-formulated.\n3. Experiments validate that the proposed method outperforms compared baselines in most evaluation metrics. ", "reasons_to_reject": "1. To further strengthen this work, it would be valuable to include a comprehensive discussion on the comparison of computational costs between the proposed approach and existing state-of-the-art methods.", "questions_to_authors": "Overall, this study demonstrates a robust and well-designed approach. However, one aspect that could be addressed is the absence of a name for the proposed method.", "paper_topic_and_main_contributions": "This paper provides a novel spectral viewpoint on Continual Relation Extraction (CRE), in which a simple and effective class-wise feature decorrelation regularization is proposed to boost the eigenvalues of the representation for each class. Extensive experiments on two benchmark datasets demonstrate the effectiveness the proposed approach. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xmSUhHNYZL", "reviewer": "Reviewer_Nh4p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. Please find below our responses for your questions and concerns.\n\n**Reviewer**: *\"The paper seems to be a trivial extension of [1] with application in CRE. The authors need to explain in detail the novelty of the paper and the differences from the existing works.\"*\n\n**Our Response**:\n\nThank you for your comment. Our work shares a similar perspective to [1] on spectral analysis for continual learning, aiming to boost the numbers and eigenvalues of the directions with larger eigenvalues to address catastrophic forgetting (eigenvalue boosting). However, we would like to emphasize a critical difference between our work and [1] (and existing related work) regarding the specific method to achieve eigenvalue boosting. In particular, [1] leverages a data augmentation method based on MixUp to randomly interpolate two images that can help artificially increase the number of classes for learning more transferable and diverse representations. This is thus an implicit approach for eigenvalue boosting. In contrast, our method introduces a class-wise decorrelation feature regularization that explicitly promotes large eigenvalues and their numbers for continual learning. In this way, our method enjoys better theoretical support (demonstrated in our Appendix B.1) for its effectiveness for continual learning, which cannot be achieved with the data augmentation method. In addition, the augmentation method in [1] in requires more expensive training computation and time than our method as it needs to generate the augmented data and learns from a larger set during the training process. Finally, it is important to note that the augmentation method in [1] is better suited for continuous data such images or speech, which can facilitate the interpolations between data samples. For relation extraction with discrete textual data as in our work, data interpolation will be more challenging, further highlighting the benefits of our regularization method to better support relation extraction. To our knowledge, we are also the first work to explore spectral analysis and feature regularization for continual relation extraction. We will clarify these details in our final version.\n\n**Reviewer**: *\"The paper lacks sufficient ablation study to demonstrate the effectiveness and to verify the assumption of the relation between class shapes preservation and performance preservation.\"*\n\n**Our Response**:\n\nThank you for your suggestion. In our experiments, we performed an ablation study to ablate the regularization component in our model that can be used to illustrate the relation between class shapes preservation and performance preservation, and we will include it in the final version. In particular, please see below the performance of our full model and ablated model for the regularization component over the TACRED dataset:\n\n| Model | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | T9 | T10 |\n| :-------------------------------- | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: | :------: |\n| Our full model | 98.1 | 93.8 | 89.8 | 85.8 | 84.4 | 83.4 | 81.6 | 79.9 | 79.7 | 79.1 |\n| Our full model w/o regularization | 97.8 | 93.4 | 88.5 | 86.0 | 83.6 | 81.8 | 79.6 | 77.7 | 77.3 | 76.8 |\n\nAs can be seen, the performance of the model drops significantly when the regularization component is excluded, testifying to its benefits for continual relation extraction. In addition, as shown in Equation (5) of Appendix B.1., our regularization component promotes the maintenance of uniform distribution of the eigenvalues across tasks, thus implicitly encouraging the class shape preservation for our continual model. As such, the removal of the regularization will likely translate into less preservation of class shapes, which also corresponds to worse performance for our model. In this way, our ablation study suggests certain connection between class shape preservation and performance preservation, and future work can further examine this issue to obtain deeper understanding.\n\n**Reviewer**: *\"The empirical performance is not significantly better than baseline alternatives.\"*\n\n**Our Response**:\n\nThank you for your comment. We would like to note that our performance improvement over baselines is similar and in the same range as those in many previous work with state-of-the-art performance and the same datasets for continual relation extraction, e.g., [2] and [3]. We thus believe our performance improvement can sufficiently demonstrate the effectiveness of the proposed method given current standards and progress in this area. We will clarify this information in our final version.\n\n[1] Zhu, Fei, et al. \"Class-incremental learning via dual augmentation.\" Advances in Neural Information Processing Systems 34 (2021): 14306-14318.\n\n[2] Wang et al., 2022. Learning robust representations for continual relation extraction via adversarial class augmentation. In EMNLP 2022.\n\n[3] Zhao et al., 2022. Consistent Representation Learning for Continual Relation ExtractionIn. In Findings of ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The assumption (when eigenvectors (or spectral components) associated with each class shape remain relatively unchanged, the shape is effectively preserved) is technically sound and to some extent verified by the empirical experiments.\n- Despite its simplicity, the proposed approach is able to achieve very promising results.", "reasons_to_reject": "- The paper seems to be a trivial extension of [1] with application in CRE. The authors need to explain in detail the novelty of the paper and the differences from the existing works.\n- The paper lacks sufficient ablation study to demonstrate the effectiveness and to verify the assumption of the relation between class shapes preservation and performance preservation.\n- The empirical performance is not significantly better than baseline alternatives.\n\n[1] Zhu, Fei, et al. \"Class-incremental learning via dual augmentation.\" Advances in Neural Information Processing Systems 34 (2021): 14306-14318.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on Continual Relation Extraction (CRE), which involves continuous learning of new relations while maintaining proficiency in previously acquired relations. Inspired by existing continual learning approaches, the authors emphasize the issue of representation shift, a phenomenon where the learned deep space undergoes transformations during the learning process, leading to a decline in performance on earlier tasks. The authors present an intriguing perspective by exploring the representation shift phenomenon through a spectral viewpoint. They argue that the preservation of class shapes in the learned deep space is critical for mitigating performance degradation. They assume that when eigenvectors (or spectral components) associated with each class shape remain relatively unchanged, the shape is effectively preserved. To verify this assumption, the authors conduct a spectral experiment and demonstrate that eigenvectors with larger eigenvalues experience higher preservation after learning new tasks, indicating their efficacy in retaining class shapes. The paper then introduces a novel class-wise regularization approach aimed at enhancing eigenvalues during representation learning. This regularization method is shown to be effective in augmenting eigenvalues, which leads to better preservation of class shapes. Extensive experiments on two public benchmark datasets demonstrate performance gains in comparison to state-of-the-art models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zdAHK85Vg1", "reviewer": "Reviewer_rW8k", "summary": "", "questions": "1.\tThe paper does not provide much interpretation or intuition for the eigenvectors with larger eigenvalues, and what kind of features they capture. It would be preferable to provide some examples or visualizations from a high-level viewpoint.\n2.\tThe paper does not compare or discuss the relation with other works that use spectral methods or feature decorrelation techniques for different tasks or settings. It would be helpful to see how the proposed method relates to or differs from some related works.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. Please find below our responses for your questions and concerns.\n\n**Reviewer**: *\"The paper does not provide much interpretation or intuition for the eigenvectors with larger eigenvalues, and what kind of features they capture. It would be preferable to provide some examples or visualizations from a high-level viewpoint.\"*\n\n**Our Response**:\n\nThank you for your suggestion. As discussed in our Limitations Section, we agree with the reviewer that we currently don't have clear interpretation for the the eigenvectors with larger eigenvalues. Based on your suggestion, we will further include visualizations for the eigendirections of the methods over our datasets to shed more light on their interpretation for continual relation extraction.\n\n**Reviewer**: *\"The paper does not compare or discuss the relation with other works that use spectral methods or feature decorrelation techniques for different tasks or settings. It would be helpful to see how the proposed method relates to or differs from some related works.\"*\n\n**Our Response**:\n\nThank you for your suggestion. In the paper, we have cited serval works on spectral decomposition [2,3] that is explored for self-supervised learning. The most related work to ours involves [1] that also studies spectral analysis for continual learning, aiming to boost the numbers and eigenvalues of the directions with larger eigenvalues to address catastrophic forgetting (called eigenvalue boosting). However, as presented in our response for reviewer Nh4p, our work is essentially different from [1] regarding the specific method to achieve eigenvalue boosting. In particular, while [1] employs a data augmentation method based on MixUp and data interpolation to implicitly boost eigenvalues, our work instead proposes a class-wise decorrelation feature regularization to explicitly promotes large eigenvalues and their numbers for continual learning. This difference brings important advantages for our method compared to that in [1], concerning better theoretical support for performance improvement (as shown in Appendix B.1), more efficient training resources and time (as we don't need to generate and consume augmented data), and better suite for discrete textual data in relation extraction. Our work is thus the first work to explore spectral analysis and feature regularization for continual relation extraction. We will carefully revise our paper to include these details and a more comprehensive section for related work.\n\nAlso, thank you very much for your suggestions for our typos and grammar. We will revise our paper to fix these issues in the final version.\n\n[1] Zhu, Fei, et al., 2021. Class-incremental learning via dual augmentation. In NeurIPS 2022.\n\n[2] Adrien Bardes, Jean Ponce, and Yann LeCun. 2022. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR 2022.\n\n[3] Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao. 2021. On feature decorrelation in self-supervised learning. In arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper provides an interesting perspective on the representation shift problem in CRE, and leverages spectral decomposition for avoiding representation shift. The authors propose a simple yet effective regularization technique that aims to increase the eigenvalues of the representation for each class and theoretically shows why feature decorrelation helps boost eigenvalues.\n2.\tThe paper conducts experiments on two standard datasets and demonstrates the superior performance of the proposed method over existing baselines. The paper also conducts further analysis to verify the hypothesis that larger eigenvalues lead to better performance and vice versa.\n", "reasons_to_reject": "1.\tThe paper does not provide much interpretation or intuition for the eigenvectors with larger eigenvalues, and what kind of features they capture. It would be preferable to provide some examples or visualizations from a high-level viewpoint.\n2.\tThe paper does not compare or discuss the relation with other works that use spectral methods or feature decorrelation techniques for different tasks or settings. It would be helpful to see how the proposed method relates to or differs from some related works.\n", "questions_to_authors": "1.\tThe paper does not provide much interpretation or intuition for the eigenvectors with larger eigenvalues, and what kind of features they capture. It would be preferable to provide some examples or visualizations from a high-level viewpoint.\n2.\tThe paper does not compare or discuss the relation with other works that use spectral methods or feature decorrelation techniques for different tasks or settings. It would be helpful to see how the proposed method relates to or differs from some related works.\n", "paper_topic_and_main_contributions": "The paper proposes a spectral viewpoint on continual relation extraction (CRE). The paper argues that the eigenvectors of the deep feature space play an important role in preserving the shape of each class and that eigenvectors with larger eigenvalues are less forgettable and useful in avoiding representation shift. Based on this point, the paper introduces a class-wise feature decorrelation regularization that boosts the eigenvalues of the representation for each class. The paper also provides theoretical and empirical evidence for the effectiveness of the proposed method, which achieves reasonable improvement on two benchmark datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are some typos and grammar mistakes, such as:\nAbstract: eigenvalue > eigenvalues\nAbstract : improve > improves\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall this paper makes a solid short research contribution, as also pointed out by the reviewers. The authors have provided sensible answers to the reviewers' questions, and these additional explanations should be incorporated in the paper to improve its readability.

Moreover, the point of motivating the Eigenvalue decomposition is also an important one.
Decision: Accept-Findings