Venue: NAACL 2025

# Enhancing Discriminative Representation in Similar Relation Clusters for

Few-Shot Continual Relation Extraction

Duc Anh Le1, Nam Le Hai1, Thanh Xuan Nguyen2, Linh Ngo Van1,

**Diep Thi-Ngoc Nguyen3, Sang Dinh1, Thien Huu Nguyen4**

Eequally contributed.Corresponding author: namlh@soict.hust.edu.vn

Footnote 1: footnotemark:

###### Abstract

Few-shot Continual Relation Extraction (FCRE) has emerged as a significant challenge in information extraction, necessitating that relation extraction (RE) systems can sequentially identify new relations with limited labeled samples. While existing studies have demonstrated promising results in FCRE, they often overlook the issue of similar relations, which is a critical factor contributing to catastrophic forgetting. In this work, we propose Sirus, a novel method that utilizes relation descriptions and dynamic clustering on these descriptions to identify similar relations. Leveraging this information, we introduce innovative loss functions specifically designed to enhance the distinction between relations, with a focus on learning to differentiate similar ones. Experimental results show that our approach can effectively mitigate the problem of catastrophic forgetting and outperforms state-of-the-art methods by a large margin. Additionally, we explore the potential of Large Language Model Embeddings (LLMEs) with representation learning and embedding capabilities, demonstrating their promise for advancing FCRE systems.

1Hanoi University of Science and Technology, 2Oraichain Labs Inc., US,

3VNU University of Engineering and Technology, 4University of Oregon

## 1 Introduction

Relation Extraction (RE) is a fundamental task in natural language processing (NLP) that involves recognizing relationships between entities from underlying content. Traditional relation extraction approaches commonly require substantial labeled datasets and assume a static collection of predefined relationships Sun et al. (2020); Cabot and Navigli (2021); Tang et al. (2022). In real-world settings, various specialized fields, such as scientific research Kruper et al. (2020), medicine Luo et al. (2022), or law Hendrycks et al. (2021), are rapidly advancing, leading to a continuous expansion in the diversity of relationships Le et al. (2024c, 2025). Consequently, RE systems are required to exhibit adaptability to handle these evolving changes effectively. Besides, another challenge in developing RE models is the scarcity of annotated data for emerging relations.

To this end, the concept of Few-shot Continual Relation Extraction (FCRE) has been proposed Qin and Joty (2022); Chen et al. (2023) to enable the continuous learning of new relations from a limited number of samples. FCRE is a branch of continual information extraction Le et al. (2025); Nguyen et al. (2023); Le et al. (2024b); Dao et al. (2024). However, due to the continual learning process with limited data, FCRE models are often biased toward the current task, facing the challenges of _overfitting_Hawkins (2004) and _Catastrophic forgetting_Thrun and Mitchell (1995); Le et al. (2024a); Hai et al. (2024); Van et al. (2022); Phan et al. (2022). Several methods have been introduced to address these issues, with memory-based approaches integrated with contrastive learning emerging as a prominent paradigm Wang et al. (2023); Ma et al. (2024); Luo et al. (2024); Tran et al. (2024); Nguyen et al. (2025). These methods typically involve retaining a few representative samples from previous tasks and applying contrastive learning to ensure that the representations of samples across different relations remain sufficiently distinguishable.

However, none of the mentioned methods have considered the confusion between similar relations, which has been identified as a significant factor contributing to catastrophic forgetting in continual relation extraction Wang et al. (2022); Zhao et al. (2023). This phenomenon becomes even more critical in the FCRE scenario, where the few available samples for each relation may not sufficiently represent the relations causing models to ignore subtle distinctions between similar relations, leading to catastrophic forgetting. A recent method, ConPL, introduced by Chen et al. (2023), addresses this is

sue by identifying similar classes based on the distance between relation prototypes and examples, and employs focal loss to emphasize the distinctions between these similar classes. The relation prototypes are computed by averaging the representations of entities within the same relations; however, a limited number of samples may fail to produce sufficiently representative prototypes (Wang et al., 2023). Moreover, the problem of inconsistencies resulting from varying context sentences can render these prototypes unstable Li and Lyu (2024), reducing the effectiveness of this approach in identifying similar classes in the FCRE scenario.

Recent studies have leveraged label descriptions Luo et al. (2024); Li and Lyu (2024); Sainz et al. (2021); Borchert et al. (2024) for few-shot relation extraction, demonstrating their effectiveness in enriching representations and stabilizing label prototypes. Especially, Nguyen et al. (2025) utilizes LLMs to augment data by generating additional label descriptions and multiple samples for each relation type. However, in this work, we focus solely on exploiting the original relation descriptions without using an LLM-based data augmentation mechanism. We propose Sirus, a novel method to enhance the Discriminative Representation in **S**imilar **R**elation **C**lusters, involves utilizing relation descriptions for label representation and employing dynamic clustering. As a result, the learning process enhances the differentiation of samples across relations, with a particular focus on similar relations, thereby reducing the phenomenon of catastrophic forgetting.

Furthermore, pre-trained Large Language Models (LLMs) with billions of parameters excel in autoregressive text generation tasks Dubey et al. (2024); Jiang et al. (2023) and demonstrate strong performance on downstream tasks with only a few examples Brown (2020); Kojima et al. (2022), making them a promising approach for application in FCRE. They have also been explored in text classification and information extraction Zhao et al. (2021); Wei et al. (2023); however, they often underperform compared to discriminative encoder models like BERT due to their generation-focused mechanism potentially makes them less effective for text representation learning. In the FCRE scenario, a recent study by Tran et al. (2024) has explored the capabilities of LLMs; however, it retained the use of causal language modeling (CLM) and applied a classification head to the last token, which may not fully exploit the embedding ability of LLMs. Several recent studies BehnamGhader et al. (2024); Li et al. (2024); Lee et al. (2024) have investigated the capabilities of LLMs in representation learning by removing the causal mask and fine-tuning LLMs with contrastive learning, referred to as Large Language Model Embeddings (LLMEs), demonstrating promising results in retrieval and classification tasks. However, their ability for continual learning, particularly in the context of FCRE, remains unexplored. Therefore, we conduct comprehensive experiments on these LLMEs, offering valuable insights into the forgetting phenomenon within these models and their performance outcomes in FCRE settings.

In summary, our contributions are as follows:

1. We present a novel approach to address the issue of similar classes by utilizing relation description representation and subsequently employing dynamic clustering to identify groups of similar relations.
2. Leveraging information from similar relations, we propose three innovative loss functions to improve the distinction between samples from different relations. Ablation studies demonstrate the efficacy of each loss function.
3. We are the first to examine LLMEs with representation learning capabilities within the context of FCRE. Our findings indicate that these models continue to suffer from the issue of catastrophic forgetting. However, applying our methods significantly enhances their performance, surpassing both the use of BERT encoder backbones and original LLMs with causal language modeling.
4. Extensive experiments conducted on two FCRE benchmarks, TACRED and FewRel, demonstrate the effectiveness of our proposed framework and highlight the promising results achieved through the use of LLMEs.

## 2 Background

### Problem Formulation

Few-Shot Continual Relation Extraction (FCRE) presents a challenging paradigm in natural language processing, combining the complexities of continual learning with the constraints of few-shot scenarios. Some related works are discussed in 

Appendix A. In this framework, a model confronts a series of tasks \({\cal T}=\{{\cal T}^{1},{\cal T}^{2},...,{\cal T}^{n}\}\), each introducing a set of novel relations \(R_{i}\) to be learned. For every task \({\cal T}^{i}\), the model is provided with a limited dataset \({\cal D}_{i}=\{(x_{j},r_{j})\}_{j=1}^{m}\), where \(m=N\times K\) represents the total number of examples, where \(N\) represents the number of new relations and \(K\) denotes the few-shot sample size for each relation. Each example consists of an input sentence \(x_{j}\) containing a pair of entities \((e_{h},e_{t})\), and a corresponding relation label \(y_{j}\in R_{i}\). This task configuration is also known as _"N-way-K-shot"_ learning setting, as introduced by Chen et al. (2023). Finally, evaluation is conducted on a comprehensive test set \({\cal D}^{test}\) that contains all relations \(R_{total}=\bigcup_{i=1}^{n}R_{i}\) encountered across tasks, assessing both its ability to learn new relations and retain proficiency in earlier ones. This formulation encapsulates the essence of FCRE, highlighting its significance in developing adaptive and efficient relation extraction systems.

The core challenge of FCRE is twofold: the model must quickly adapt to new relations with limited examples (few-shot learning) while maintaining knowledge of previously learned relations (continual learning). This requires balancing the model's plasticity for learning new relations and its stability for preserving prior knowledge.

### Input Formulation and Representation

In Relation Extraction, the foundational deep learning approach Ji et al. (2020); Wang and Lu (2020) typically involves encoding input data with a pre-trained language model (PLM) like BERT Devlin et al. (2019). A crucial aspect of RE is how to formulate the input effectively to obtain high-quality representation embedding for classification. Early studies frequently follow BERT by concatenating the [CLS] token with the original input \(x\) and utilizing this token's vector representation for classification. Another method involves using additional special tokens to enclose the two entities, and then concatenating their embeddings to form the input representation for the relation classification layer Zhao et al. (2022); Le et al. (2024c).

In this study, we employ the input format suggested by Ma et al. (2024). Specifically, we use a special [MASK] token to denote the relation between the head entity (\(e_{h}\)) and tail entity (\(e_{t}\)), and integrate this token with the original sentence \(x\) and the two entities. Besides, several learnable tokens are also inserted to avoid relying entirely on handcrafted tokens. Consequently, the input template is formulated as follows:

\[\begin{split}{\cal I}(x)&=x\left[v_{0:n_{0}-1} \right]e_{h}\left[v_{n_{0}:n_{1}-1}\right]\texttt{[MASK]}\\ &\left[v_{n_{1}:n_{2}-1}\right]e_{t}\left[v_{n_{2}:n_{3}-1} \right].\end{split} \tag{1}\]

where \([v_{i}]\) represents the _i-th_ learnable continuous token, and \(n_{i}\) denotes the length of the token phrases. In our specific implementation, we use a special [UNUSED] token as \([v]\). We then forward the templated input \({\cal I}(x)\) through a PLM, encoding it into a sequence of continuous vectors. From these, we extract the hidden representation \(\mathbf{z}_{x}\) of the input, corresponding to the position of the [MASK] token.

\[\mathbf{z}_{x}=f_{\mathcal{M}}\big{(}{\cal I}(x)\big{)}[\mathrm{position}(\texttt{ [MASK]})], \tag{2}\]

where \(f_{\mathcal{M}}(X)\) denotes the forward function of a PLM \(\mathcal{M}\) on input \(X\). The latent representation is then used for contrastive learning and predicts the relation associated with the given input \(x\).

## 3 Methodology

In this section, we present our method aimed at enhancing FCRE by tackling the problem of similar relations. Specifically, we proposed _Clustering Relations via Label Description_ (CRLD) to identify groups of similar relations. Accordingly, we propose three loss functions designed to differentiate samples that are semantically similar but belong to distinct relations, using the clustering information.

### Clustering Relations via Label Description

The label description of a relation is a summarization that describes the meaning and provides general information about the relation. It has been demonstrated to be more consistent than the label prototype, which is derived from multiple sample contexts associated with the same label Li and Lyu (2024). Therefore, we leverage this information for clustering purposes to recognize similar relations. This framework allows us to identify informative hard negatives for samples using cluster information, thereby enhancing the differentiation of samples in similar classes and potentially improving training convergence Xiong et al. (2020).

Let \(\{(r_{i},d_{i})\}_{i=1}^{N}\) denote the set of relations and their corresponding description. For each description \(d_{i}\), we obtain its embedding \(\mathbf{d}_{i}\) by passing it through the same encoder used for input sentences containing entities \(f_{\mathcal{M}}(d_{i})\), as presented in Section 2.2. However, instead of applying the input template \({\cal I}\), we directly use the raw description and 

obtain its latent embedding by mean pooling the token representations within the description. Subsequently, we employ the Agglomerative Clustering algorithm Mullner (2011) on these embeddings to categorize the relations into clusters according to their semantic similarity. This clustering method allows us to automatically identify the number of clusters by selecting a distance threshold \(\theta\), thereby eliminating the need for manual selection of group sizes for similar relations. Besides, the clustering algorithm is applied iteratively for each batch. Thus, the cluster for each relation is dynamically updated following the encoder's parameters adjustment after each batch, as illustrated in Algorithm 1. As a result, each relation-description pair \((r_{i},d_{i})\) is assigned to a new cluster \(c(r_{i})\in\left\{1,2,\ldots,K\right\}\) after each batch.

### Discriminative Loss Functions

In this section, we introduce three innovative loss functions to improve the model's ability to distinguish samples across relations.

Weighted Supervised Contrastive Lossaims to bring closer positive sample pairs, which share the same relation label, while pushing apart negative pairs that belong to different relation.To this end, we conduct the Supervised Contrastive Loss Khosla et al. (2020), enhanced with weighting hard negatives according to the similarity of their label descriptions, thus focusing more on samples from similar relations. Specifically, this loss is computed as follows:

\[\begin{split}&\mathcal{L}_{\text{WSC}}(x)=\\ &-\sum_{p\in P(x)}\log\frac{f(\mathbf{z}_{x},\mathbf{z}_{p})}{\sum_{\bar{ x}\in\mathcal{D}\setminus\{x\}}w(x,\bar{x})\cdot f(\mathbf{z}_{x},\mathbf{z}_{\bar{x}})} \end{split} \tag{3}\]

where \(f(\mathbf{z}_{x},\mathbf{z}_{y})=\exp\left(\frac{\gamma(\mathbf{z}_{x},\mathbf{z}_{y})}{\tau}\right)\) and

\[w(x,\bar{x})=\begin{cases}1+\alpha\cdot\gamma(\mathbf{d}_{x},\mathbf{ d}_{\bar{x}})&\text{if $c(r_{x})=c(r_{\bar{x}})$}\\ 1&\text{otherwise}\end{cases} \tag{4}\]

Here, \(\gamma(\cdot,\cdot)\) represents the Cosine Similarity function, \(\tau\) is the temperature scaling parameter, \(\alpha\) is a weighting factor based on the description similarity, and \(c(\cdot)\) is the cluster assignment function. \(P(x)\) and \(\mathcal{D}\) refer to the sets of positive samples for sample \(x\) and the entire dataset, respectively.

Cluster-based Mutual Information Lossaims to bring closer the semantic representation of a sample with its corresponding label description, while pushing apart it from descriptions of different relations.Specifically, it involves maximizing the mutual information (MI) between the input's hidden representation \(\mathbf{z}_{x}\) and its corresponding label description \(\mathbf{d}_{x}\), while leveraging cluster information to identify hard negatives. This extends the traditional Mutual Information loss van den Oord et al. (2019) by introducing a weighting function that considers the similarity between label descriptions within the same cluster.

The mutual information \(MI(x)\) between the input embedding \(\mathbf{z}_{x}\) and its corresponding label description is lower-bounded by:

\[MI\geq\log B+\text{InfoNCE}(D_{B};h), \tag{5}\]

However, we modify the InfoNCE loss by incorporating a weighting function \(w(x_{i},x_{j})\), as presented in equation (4), to assign greater emphasis on hard negatives within the same cluster. The modified InfoNCE is thus defined as follows:

\[\begin{split}&\text{InfoNCE}(D_{B};h)=\\ &\quad\frac{1}{B}\sum_{i=1}^{B}\log\frac{h(\mathbf{z}_{x_{i}},\mathbf{d}_ {x_{i}})}{\sum_{j=1}^{B}w(x_{i},x_{j})\cdot h(\mathbf{z}_{x_{i}},\mathbf{d}_{x_{j}})} ,\end{split}\]

where \(h(\mathbf{z},\mathbf{d})=\exp\left(\frac{\pi^{T}W\mathbf{d}}{\tau}\right)\), \(\tau\) is the temperature, \(D_{B}=\{x_{i}\}_{i=1}^{B}\) is the mini-batch data with size \(B\), and \(W\) is a trainable parameter. The final CMI loss function with an input \(x\) is defined as:

\[\boxed{\mathcal{L}_{\text{CMI}}(x)=-\log\frac{h(\mathbf{z}_{x},\mathbf{d}_ {x})}{Z(x)}} \tag{6}\]

where,

\[Z(x)=h(\mathbf{z}_{x},\mathbf{d}_{x})+\sum_{\bar{x}\in\mathcal{N}(x)}w(x,\bar{x})\cdot h (\mathbf{z}_{x},\mathbf{d}_{\bar{x}})\]

\[\mathcal{N}(x)=\{\bar{x}|\bar{x}\in D_{B},r_{\bar{x}}\neq r_{x}\}\]

Double Triplet Loss for Intra- and Inter-Cluster Separationaims to enforce that samples are close to their own descriptions and far from other cluster centroids, especially those not associated with their labels. In particular, we employ Double Triplet Loss (DTL) Schroff et al. (2015) on input, label description, and cluster centroid 

\[\begin{split}&\mathcal{L}_{\text{DT}}(x)=\\ &\max\big{(}0,D(\mathbf{z}_{x},\mathbf{d}_{x})-D(\mathbf{z}_{x},\mathbf{c}^{+}_{ x})+m_{1}\big{)}+\\ &\max\big{(}0,D(\mathbf{z}_{x},\mathbf{c}^{+}_{x})-D(\mathbf{z}_{x},\mathbf{c}^{-}_{ x})+m_{2}\big{)}\end{split} \tag{7}\]

where,

\[\mathbf{c}^{+}_{x} =\mathbf{c}_{k}=\frac{1}{|C_{k}|}\sum_{d\in C_{k}}\mathbf{d}\quad\text{ if $ c(r_{x})=k$}\] \[\mathbf{c}^{-}_{x} =\mathbf{c}_{\bar{k}}\quad\text{ such that $\bar{k}=\underset{c(r_{x})\neq k}{ \operatorname*{argmin}}\;D(\mathbf{z}_{x},\mathbf{c}_{k})$}\]

Here, \(D(\cdot,\cdot)=1-\gamma(\cdot,\cdot)\) is the Cosine Distance, \(\mathbf{c}_{k}\), \(C_{k}\) represent the centroid representation and set containing the description embeddings of relations within cluster \(k\), respectively. \(\mathbf{c}^{+}_{x}\) is the centroid representation of the cluster which containing \(r_{x}\), while \(\mathbf{c}^{-}_{x}\) denotes the centroid of the nearest cluster to \(x\) that does not include the relation \(r_{x}\). \(m_{1}\) and \(m_{2}\) are margin hyperparameters.

Overall Training Objective.The total loss function is formulated by combining equations (3), (6) and (7) as follows:

\[\mathcal{L}(x)=\lambda_{1}\mathcal{L}_{\text{WSC}}(x)+\lambda_{2}\mathcal{L} _{\text{CMI}}(x)+\lambda_{3}\mathcal{L}_{\text{DT}}(x) \tag{8}\]

where \(\lambda_{1}\), \(\lambda_{2}\) and \(\lambda_{3}\) are weighting hyperparameters that balance the contributions of each loss component.

### Large Language Model Embeddings for FCRE

Large Language Model Embeddings (LLMEs) redefine decoder-only LLMs as text encoders, enhancing their embedding and representation learning capabilities BehnamGhader et al. (2024); Li et al. (2024); Lee et al. (2024). This transformation typically involves two main adjustments: (1) removing the causal mask to allow bidirectional attention, and (2) replacing the next-token prediction task with alternative training objectives, such as contrastive learning or masked token prediction. These modifications enable LLMEs to function similarly to encoder models like BERT while leveraging the extensive architecture and pretraining corpus of the original LLMs, thereby enhancing generalization and comprehension capabilities.

We explore the application of LLMEs in the FCRE scenario by substituting the backbone model \(\mathcal{M}\) with these models, as described in Section 2.2. However, since LLMs perform well with instruction prompts and mean-pooling all token embeddings yields the best results in LLM2Vec BehnamGhader et al. (2024)--an LLME, we formulate an input \(x\) with entities \(e_{h},e_{t}\) as follows.

\[\mathcal{F}_{LLMEs}(x)=x.\] The relation between \[[e_{h}]\text{ and }[e_{t}]\text{ is:}\]

This instruction prompt allows LLMEs to grasp the semantic context to categorize relations for the entities. The latent embedding is subsequently obtained by mean pooling the token representations. Training and inference procedures remain similar across all backbone models.

### Training and Inference Procedures

```
0:\(\mathcal{M}\): Backbone PLM \(\mathcal{D}^{test}\): Test data \(L\): The number of training samples allocated to memory for each relation. Previous variables: \(\Phi_{j-1},\tilde{R}_{j-1},\tilde{M}_{j-1},\tilde{S}_{j-1}\) Current variables: \(D^{train}_{j},D^{test}_{j},R_{j},S_{j}\). Output:\(\Phi_{j},\tilde{M}_{j},\tilde{S}_{j},\tilde{P}_{j}\).
1: Initialize \(\Phi_{j}\) from \(\Phi_{j-1}\)
2:\(\tilde{S}_{j}\leftarrow\tilde{S}_{j-1}\cup S_{j}\), \(\tilde{R}_{j}\leftarrow\tilde{R}_{j-1}\cup R_{j}\)
3:for batch in batches(\(\tilde{M}_{j-1}\cup D_{j}\))do
4:\(\mathbf{d}_{i}\gets f_{\mathcal{M}_{\Phi_{j}}}(d_{i})\quad\forall d_{i}\in \tilde{S}_{j}\)
5: AgglomerativeClustering(\(\{\mathbf{d}_{i}\}\))
6: Update \(\Phi_{j}\triangleright\)Backward using loss \(\mathcal{L}\) in (8)
7:endfor
8:\(\tilde{M}_{j}\leftarrow\tilde{M}_{j-1}\)
9:for each \(r\in R_{j}\)do\(\triangleright\)Update memory buffer
10:\(\mathcal{B}_{r}\leftarrow\{(x_{i},r_{i})|x_{i}\in D^{train}_{j},r_{i}=r\}_{ i=1}^{L}\)
11:\(\tilde{M}_{j}\leftarrow\tilde{M}_{j}\cup\mathcal{B}_{r}\)
12:endfor
13:\(\mathcal{D}^{test}\leftarrow\mathcal{D}^{test}\cup D^{test}_{j}\qquad \triangleright\)For inference
```

**Algorithm 1** Training procedure at each task \(\mathcal{T}^{j}\)

Training Procedure:Algorithm 1 outlines the end-to-end training process at each task \(\mathcal{T}^{j}\), with \(\Phi_{j-1}\) denoting the model parameters after training on the previous \(j-1\) tasks. In line with memory-based methods, we maintain a memory buffer \(\hat{M}_{j-1}\) that stores a few representative samples from all previous tasks \(\mathcal{T}^{1},\ldots,\mathcal{T}^{j-1}\), along 

with a relation description set \(\tilde{S}_{j-1}\) that holds the descriptions of all previously encountered relations.

1. **Initialization** (Lines 1-2): The model parameter for the current task, \(\Phi_{j}\), is initialized with the parameters of \(\Phi_{j-1}\). We update the relation \(\tilde{R}_{j}\) and relation description sets \(\tilde{E}_{j}\) with new relations.
2. **Model Update** (Lines 3-7): We train and update the model parameters using a dataset that combines memory data with the data from the current task \(\mathcal{T}^{j}\). Relation description representation and cluster results are updated in each batch (Lines 4-5). Accordingly, the loss function is adjusted to reflect the cluster changes to update the current parameter \(\Phi_{j}\).
3. **Memory Update** (Lines 8-12): We select \(L\) representative samples from \(D_{j}\) for each relation \(r\in R_{j}\). These are the \(L\) samples whose latent representations are closest to the \(1\)-means centroid of all class samples.
4. **Testset Update** (Line 13): The test set is expanded by incorporating the test data from the current task and will be utilized for evaluation after finishing training across all tasks.

Inference Procedure:Leveraging the discriminative feature distribution learned during training, we adopt the Nearest-Class-Mean classifier, as employed by Ma et al. (2024), for relation prediction in the test phase. However, instead of relying solely on the label prototype, we incorporate both the label description and prototype to extract the relation.

Given a sample \(x\) with hidden representation \(\mathbf{z}_{\mathbf{x}}\), a set of relation prototypes \(\{\mathbf{p}_{r}\}_{r=1}^{n}\) and a set of relation descriptions \(\{\mathbf{d}_{r}\}_{r=1}^{n}\).

\[\mathbf{p}_{r}=\frac{1}{L}\sum_{i=1}^{L}\mathbf{z}_{i}, \tag{9}\]

The inference process begins by calculating the Cosine similarity between \(\mathbf{z}_{\mathbf{x}}\) and each prototype \(\mathbf{p}_{r}\) and label description \(\mathbf{d}_{r}\). The final prediction \(y^{*}\) is then determined by:

\[y^{*}=\operatorname*{argmax}\left(\gamma(\mathbf{z}_{x},\mathbf{p}_{r})+\gamma(\mathbf{ z}_{x},\mathbf{d}_{r})\right) \tag{10}\]

where \(\gamma(\cdot,\cdot)\) denotes the cosine similarity function.

## 4 Experimental Results

### Experiment Setup

We compare our method against 8 state-of-the-art baselines on two widely used benchmarks FewRel Han et al. (2018) and TACRED Zhang et al. (2017) in the literature of CRE and FCRE. We conduct experiments using BERT Devlin et al. (2019) and two LLMEs: LLVMec BehnamGhader et al. (2024) and BGE Li et al. (2024). Moreover, we employ three variants of LLM2Vec, using LLama2, LLama3, and Mistral as backbones. After completing each task, we evaluate the models on the updated \(\mathcal{D}^{test}\) with 6 random seeds and report the mean and standard deviation of the accuracy. More details about datasets, baselines, backbones, and evaluation metrics are presented in Appendix B.

Figure 1: The confusion matrix of CPL_MI, ConPL, and Sirus predictions on similar relations detected by our CRLD method. The descriptions of relations are presented in Table 8. Note that the figure illustrates only the relations from the test set that appear in these two detected similar clusters, rather than displaying all relations.

 



### Evaluation Results of Sirus Framework

In this section, we analyze the results of Clustering Relations via Label Description (CRLD) and our proposed Sirus, which integrates CRLD with three loss functions in FCRE.

Effectiveness of CRLD:Figure1 presents the relations within two clusters that are close together generated by CRLD, showing that they share a common topic related to location and country. Through the visualization of the confusion matrix in these relations derived by 3 methods, we observe that these relations are frequently confused. For instance, models often misclassify samples to the relations "_country of origin_" and "_country of citizenship_" or "_headquarters location_" and "_location of formation_". This observation indicates that CRLD can effectively identify similar relations.

In addition, in comparison to CPL_MI, ConPL-a method that also tackles the challenge of similar relations, offers a clearer differentiation among these relations. For example, between two relations "_country_" and "_country of origin_", CPL_MI depicts a higher number of misclassified samples than ConPL (32 compared to 19). Meanwhile, Sirus shows clearer results in reducing confusion between these classes, with darker blocks along the diagonal (indicating more correct classifications) and lighter blocks outside the diagonal (reflecting fewer misclassifications). This not only demonstrates the effectiveness of CRLD in accurately identifying similar relations, which aids the model in focusing on learning to differentiate them but also reveals the overall efficacy of Virus for FCRE.

Performance of Sirus against the Baselines:Table1 presents the performance comparison between Sirus and 8 state-of-the-art methods. Overall, our approach consistently outperforms the performance of existing methods across all tasks on both benchmarks. Specifically, on the FewRel dataset, Sirus surpasses the baselines by a significant margin, achieving over 1% improvement compared to the second-best method, CPL_MI, across all tasks. As a result, after the final task, Sirus achieves 69.16% accuracy on the test set encompassing all relations, demonstrating an improvement of nearly 3% over CPL_MI. A similar trend is observed in the TACRED dataset, where our method outperforms CPL_MI by more than 1%. Besides, additional results demonstrating the effectiveness of each proposed loss function are provided in AppendixC.1. These results demonstrate the effectiveness of Sirus and underscore the importance of addressing the issue of similar relations in FCRE.

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \multirow{2}{*}{Method} & \multicolumn{4}{c}{Tasks} \\ \cline{2-7}  & \(\mathcal{T}^{1}\) & \(\mathcal{T}^{2}\) & \(\mathcal{T}^{3}\) & \(\mathcal{T}^{4}\) & \(\mathcal{T}^{5}\) & \(\mathcal{T}^{6}\) & \(\mathcal{T}^{7}\) & \(\mathcal{T}^{8}\) \\ \hline \multicolumn{7}{c}{**FWebRed (10-way-5-shot)**} \\ \hline RP-CRE [2] & 93.97\({}_{\pm 0.04}\) & 76.05\({}_{\pm 2.26}\) & 71.36\({}_{\pm 2.83}\) & 69.32\({}_{\pm 3.38}\) & 64.95\({}_{\pm 3.09}\) & 69.99\({}_{\pm 2.09}\) & 60.59\({}_{\pm 1.87}\) & 59.57\({}_{\pm 1.13}\) \\ CRL [12] & 94.68\({}_{\pm 0.33}\) & 80.73\({}_{\pm 2.91}\) & 73.82\({}_{\pm 2.77}\) & 70.26\({}_{\pm 3.18}\) & 66.62\({}_{\pm 2.84}\) & 63.28\({}_{\pm 2.49}\) & 60.96\({}_{\pm 2.63}\) & 59.72\({}_{\pm 1.32}\) \\ CRECL [13] & 93.93\({}_{\pm 0.22}\) & 82.55\({}_{\pm 6.99}\) & 74.13\({}_{\pm 3.59}\) & 69.32\({}_{\pm 3.87}\) & 66.51\({}_{\pm 4.66}\) & 66.61\({}_{\pm 1.92}\) & 69.27\({}_{\pm 1.46}\) & 59.99\({}_{\pm 0.65}\) \\ ERDA [13] & 92.43\({}_{\pm 0.32}\) & 64.52\({}_{\pm 2.11}\) & 50.31\({}_{\pm 3.32}\) & 49.24\({}_{\pm 2.17}\) & 39.75\({}_{\pm 3.34}\) & 36.36\({}_{\pm 3.12}\) & 34.34\({}_{\pm 1.83}\) & 31.96\({}_{\pm 1.91}\) \\ SCKD [14] & 94.77\({}_{\pm 0.35}\) & 82.83\({}_{\pm 2.64}\) & 76.21\({}_{\pm 1.61}\) & 72.19\({}_{\pm 1.33}\) & 70.61\({}_{\pm 2.24}\) & 67.15\({}_{\pm 1.96}\) & 64.86\({}_{\pm 1.33}\) & 62.98\({}_{\pm 0.88}\) \\ Conf/J [2] & **95.18\({}_{+0.78}\)** & 79.63\({}_{\pm 1.27}\) & 74.54\({}_{\pm 1.13}\) & 71.27\({}_{\pm 0.45}\) & 68.33\({}_{\pm 0.46}\) & 68.62\({}_{\pm 0.20}\) & 67.44\({}_{\pm 1.49}\) & 62.46\({}_{\pm 1.54}\) \\ CPRL [15] & 94.87 & 85.14 & 78.80 & 75.10 & 72.57 & 60.57 & 60.65 & 64.50 \\ CPL\_MI [16] & 94.69\({}_{\pm 0.70}\) & 85.58\({}_{\pm 1.88}\) & 80.12\({}_{\pm 2.45}\) & 75.71\({}_{\pm 2.23}\) & 73.90\({}_{\pm 1.80}\) & 70.72\({}_{\pm 1.91}\) & 68.42\({}_{\pm 1.77}\) & 66.27\({}_{\pm 1.53}\) \\ \hline SRUS & 94.74\({}_{\pm 0.27}\) & **87.12\({}_{\pm 2.21}\)** & **81.06\({}_{\pm 1.52}\)** & **77.49\({}_{\pm 2.88}\)** & **75.47\({}_{\pm 2.60}\)** & **72.48\({}_{\pm 1.75}\)** & **70.61\({}_{\pm 1.81}\)** & **69.16\({}_{\pm 0.48}\)** \\ \hline \multicolumn{7}{c}{**TCRED (5-way-5-shot)**} \\ \hline RP-CRE [2] & 87.32\({}_{\pm 1.76}\) & 74.90\({}_{\pm 0.13}\) & 67.88\({}_{\pm 3.41}\) & 60.23\({}_{\pm 3.52}\) & 53.26\({}_{\pm 4.87}\) & 50.72\({}_{\pm 7.62}\) & 46.21\({}_{\pm 5.29}\) & 44.48\({}_{\pm 3.74}\) \\ CRL [12] & 88.32\({}_{\pm 1.26}\) & 76.30\({}_{\pm 7.48}\) & 69.76\({}_{\pm 6.59}\) & 61.93\({}_{\pm 2.55}\) & 54.68\({}_{\pm 3.12}\) & 50.92\({}_{\pm 4.45}\) & 47.00\({}_{\pm 3.78}\) & 44.27\({}_{\pm 2.51}\) \\ CRECL [13] & 87.09\({}_{\pm 2.50}\) & 78.09\({}_{\pm 5.74}\) & 69.13\({}_{\pm 4.89}\) & 55.60\({}_{\pm 5.78}\) & 53.42\({}_{\pm 2.99}\) & 51.91\({}_{\pm 2.90}\) & 47.55\({}_{\pm 5.33}\) & 45.53\({}_{\pm 1.96}\) \\ ERDA [13] & 81.85\({}_{\pm 1.97}\) & 53.68\({}_{\pm 3.61}\) & 40.36\({}_{\pm 3.33}\) & 36.17\({}_{\pm 3.65}\) & 30.14\({}_{\pm 3.96}\) & 22.61\({}_{\pm 3.13}\) & 19.22\({}_{\pm 2.31}\) & 19.42\({}_{\pm 2.31}\) \\ SCKD [14] & 88.24\({}_{\pm 0.83}\) & 79.35\({}_{\pm 1.11}\) & 70.61\({}_{\pm 3.16}\) & 66.78\({}_{\pm 4.29}\) & 60.74\({}_{\pm 3.05}\) & 58.05\({}_{\pm 5.84}\) & 54.14\({}_{\pm 3.87}\) & 52.11\({}_{



### Evaluation Results of LLMEs in FCRE

In this section, we analyze the results of LLMEs, concentrating on the catastrophic forgetting in these models and comparing their performance to the use of BERT and LLMs with causal language modeling. Additionally, we also aim to assess the effectiveness and adaptability of our method, Sirus, on these large-scale models.

Catastrophic Forgetting in LLMEs:To investigate the issue of Catastrophic Forgetting in LLMEs, we employ LLM2Vec, BGE-variants as backbones, training them with contrastive loss Khosla et al. (2020) across sequential tasks while excluding the memory buffer for storing previous data. Table 2 indicates that these models still suffer from catastrophic forgetting. Specifically, their performance remarkably decrease in later tasks. This phenomenon is also observed when utilizing causal LLMs in FCRE Tran et al. (2024).

Comparison between LLMEs and LLMs:Tables 1 and 2 reveal a significant improvement of up to \(10\%\) on both benchmarks in LLMEs compared to BERT when using Sirus and CPL, highlighting the huge potential of LLMEs in FCEE. Consider utilizing original LLMs with decoder-only architecture, CPL performs worse than CPL_MI by a large margin using both LLama2 and Mistral on the two benchmarks. However, after integrating LLMEs into CPL, it surpasses CPL_MI with the original LLMs by over \(2\%\) on FewRel and \(6\%\) on TACRED. This result demonstrates the effectiveness of LLMEs' representation and embedding capabilities, extending beyond the generation-focused nature of LLMs in the FCRE scenario.

Our proposed method, Sirus, consistently outperforms CPL across all cases when integrated with

\begin{table}
\begin{tabular}{l|c c c c c c c c c} \hline \multirow{2}{*}{Backbone} & Method & \multicolumn{3}{c}{LIME} & \multicolumn{5}{c}{Tasks} \\ \cline{3-11}  & & & \(\mathcal{T}^{1}\) & \(\mathcal{T}^{2}\) & \(\mathcal{T}^{3}\) & \(\mathcal{T}^{4}\) & \(\mathcal{T}^{5}\) & \(\mathcal{T}^{6}\) & \(\mathcal{T}^{7}\) & \(\mathcal{T}^{8}\) \\ \hline \multicolumn{11}{c|}{**FWebRel (\(l\)-wy-5-shot)**} \\ \hline \multirow{9}{*}{_LLAME2_} & CPL & ✗ & \(95.73_{+0.9}\) & \(85.72_{+1.6}\) & \(80.57_{+1.74}\) & \(78.60_{+3.1}\) & \(77.30_{+2.41}\) & \(73.95_{+1.56}\) & \(71.35_{+3.75}\) & \(69.87_{+2.32}\) \\  & CPL\_MI & ✗ & \(95.63_{+0.10}\) & \(87.14_{+1.91}\) & \(83.52_{+2.51}\) & \(80.59_{+2.37}\) & \(79.20_{+1.16}\) & \(76.41_{+1.23}\) & \(76.42_{+1.17}\) & \(72.08_{+1.83}\) \\  & CPL\_Y\({}^{\dagger}\) & ✓@**96.78** & \(88.91_{+3.88}\) & \(84.90_{+2.98}\) & \(81.99_{+2.92}\) & \(77.10_{+2.38}\) & \(77.60_{+2.4}\) & \(75.73_{+2.42}\) & \(74.12_{+1.23}\) \\  & LLM2Vec\({}_{bron}^{\dagger}\) & ✓@95.05\({}_{+0.21}\) & \(84.88_{+2.79}\) & \(78.15_{+2.55}\) & \(74.22_{+1.26}\) & \(72.36_{+2.90}\) & \(69.21_{+2.81}\) & \(64.89_{+1.66}\) & \(63.38_{+0.05}\) \\  & SEbus! & ✓@95.08\({}_{+0.24}\) & \(87.93_{+2.50}\) & \(83.22_{+0.10}\) & \(81.76_{+2.00}\) & \(81.23_{+2.26}\) & \(70.07_{+1.00}\) & \(76.86_{+1.63}\) & \(75.98_{+0.05}\) \\ \hline \multirow{9}{*}{_Mixtral_} & CPL & ✗ & \(96.57_{+0.40}\) & \(86.82_{+0.25}\) & \(83.31_{+1.39}\) & \(79.45_{+2.55}\) & \(77.17_{+2.55}\) & \(77.42_{+1.16}\) & \(73.59_{+2.07}\) & \(78.19_{+1.97}\) \\  & CPL\_MI & ✗ & \(96.56_{+0.40}\) & \(96.77_{+1.81}\) & \(81.41_{+0.10}\) & \(83.08_{+1.20}\) & \(78.92_{+1.33}\) & \(77.27_{+2.00}\) & \(77.05_{+2.30}\) & \(70.23_{+1.67}\) \\  & CPL\_I\({}^{\dagger}\) & ✓@96.62\({}_{+0.20}\) & \(85.75_{+2.53}\) & \(84.39_{+2.65}\) & \(82.46_{+2.08}\) & \(80.38_{+1.33}\) & \(78.06_{+1.18}\) & \(75.41_{+1.39}\) & \(74.00_{+1.32}\) \\  & LIM2Vec\({}_{bron}^{\dagger}\) & ✓@96.37\({}_{+0.16}\) & \(86.53_{+1.47}\) & \(80.50_{+2.38}\) & \(76.00_{+1.85}\) & \(72.83_{+0.08}\) & \(68.50_{+0.03}\) & \(67.38_{+2.75}\) & \(65.65_{+0.11}\) \\  & SEbus! & ✓@96.93\({}_{+0.20}\) & \(89.04_{+2.00}\) & \(86.02_{+2.81}\) & \(84.29_{+2.82}\) & \(81.06_{+2.82}\) & \(79.05_{+2.00}\) & \(77.95_{+2.00}\) & \(76.96_{+1.15}\) \\  & SOLE\({}_{bron}^{\dagger}\) & ✓@96.38\({}_{+0.20}\) & \(86.84_{+2.78}\) & \(75.89_{+2.60}\) & \(76.50_{+1.77}\) & \(73.40_{+1.04}\) & \(78.20_{+2.30}\) & \(63.91_{+2.04}\) & \(67.51_{+1.89}\) \\  & CPLI\({}^{\dagger}\) & ✓@96.52\({}_{+0.26}\) & \(89.85_{+1.31}\) & \(84.33_{+1.97}\) & \(81.55_{+2.94}\) & \(70.55_{+2.40}\) & \(77.27_{+1.90}\) & \(75.65_{+2.90}\) & \(73.25_{+2.48}\) \\  & SEbus! & ✓@96.90\({}_{+0.38}\) & \(91.14_{+0.38}\) & \(87.94_{+1.66}\) & \(86.39_{+2.31}\) & \(84.69_{+2.12}\) & \(82.82_{+1.90}\) & \(80.91_{+0.00}\) & \(79.38_{+0.06}\) \\ \hline \multirow{9}{*}{_LLAM23_} & CPL\_IMVcc\({}_{bron}^{\dagger}\) & ✓@97.50\({}_{+0.18}\) & \(86.67_{+1.33}\) & \(80.14_{+1.17}\) & \(76.12_{+2.23}\) & \(77.21_{+1.38}\) & \(68.30_{+1.67}\) & \(65.14_{+1.54}\) & \(63.24_{+2.42}\) \\  & YP\({}^{\ddagger}\)/7 & ✓@97.41\({}_{+0.18}\) & \(79.56_{+2.62}\) & \(82.32_{+1.26}\) & \(79.78_{+2.78}\) & \(78.70_{+2.51}\) & \(77.65_{+1.02}\) & \(76.39_{+1.12}\) & \(76.13_{+1.11}\) \\ \hline \multirow{9}{*}{_ILAma2_} & CPL & ✗ & \(96.80_{+0.10}\) & \(91.04_{+2.48}\) & \(87.36_{+0.48}\) & \(85.25_{+0.48}\) &

LIMEs. This showcases the versatility and adaptability of our approach across a diverse range of architectures. Among all backbones, the LLama3 variant of LLM2Vec integrated with Sirus exhibits the highest performance on FewRel, while BGE with Mistral shows superior results for TACRED.

## 5 Conclusion

In conclusion, our novel approach to Few-shot Continual Relation Extraction (FCRE) effectively addresses the challenge of similar relations, which often leads to catastrophic forgetting. By leveraging relation descriptions and dynamic clustering, we enhance the distinction between relations through innovative loss functions. Our experimental results indicate that our approach achieves superior performance, surpassing state-of-the-art methods. Moreover, our comprehensive investigation of Large Language Model Embeddings (LLMEs) demonstrates superior performance over both BERT and decoder-only LLMs in all cases, emphasizing their potential to advance FCRE systems. This work paves the way for more robust and accurate relation extraction systems, contributing to the broader field of information extraction.

## 6 Limitations

Currently, the approach and analyses conducted in this study are limited to high-level relation extraction tasks, where the entities are predetermined. Therefore, to achieve more practical and advancing FCRE systems, it is essential to investigate end-to-end relation extraction challenges in future research, integrating both entity recognition and the extraction of relations among the identified entities. This scenario presents greater challenges as it necessitates addressing both overfitting and catastrophic forgetting across two consecutive tasks.

Our approach primarily targets the challenge of similar relations, particularly leading to catastrophic forgetting; however, it has not yet considered the issue of overfitting, which arises from the constraints of limited data. Despite this limitation, Sirus demonstrates superior performance compared to techniques that involve augmenting data for previously learned tasks Qin and Joty (2022); Ma et al. (2024); Tran et al. (2024). This suggests that while our current method effectively addresses the problem of similar relations, there remains room for improvement. We believe that incorporating data augmentation could further enhance the performance of our method. Therefore, we plan to investigate this approach in future research.

## Acknowledgements

This research is funded by Hanoi University of Science and Technology (HUST) under project number T2024-TN-003.

## References

* Ahn et al. (2019) Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. 2019. Uncertainty-based continual learning with adaptive regularization. _Advances in neural information processing systems_, 32.
* BehnamGhader et al. (2024) Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. _arXiv preprint arXiv:2404.05961_.
* Borchert et al. (2024) Philipp Borchert, Jochen De Weerdt, and Marie-Francine Moens. 2024. Efficient information extraction in few-shot relation classification through contrastive representation learning. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 638-646, Mexico City, Mexico. Association for Computational Linguistics.
* Brown (2020) Tom B Brown. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Cabot and Navigli (2021) Pere-Llufs Huguet Cabot and Roberto Navigli. 2021. Rebel: Relation extraction by end-to-end language generation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2370-2381.
* Chen et al. (2023) Xiudi Chen, Hui Wu, and Xiaodong Shi. 2023. Consistent prototype learning for few-shot continual relation extraction. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 7409-7422, Toronto, Canada. Association for Computational Linguistics.
* Cui et al. (2021) Li Cui, Deqing Yang, Jiaxin Yu, Chengwei Hu, Jiayang Cheng, Jingjie Yi, and Yanghua Xiao. 2021. Refining sample embeddings with relation prototypes to enhance continual relation extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 232-243.
* Dao et al. (2024) Viet Dao, Van-Cuong Pham, Quyen Tran, Thanh-Thien Le, Linh Ngo, and Thien Nguyen. 2024. Lifelong event detection via optimal transport. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 12610-12621 

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
* Hai et al. (2024) Nam Le Hai, Trans Nguyen, Linh Ngo Van, Thien Huu Nguyen, and Khoat Than. 2024. Continual variational dropout: a view of auxiliary local variables in continual learning. Machine Learning, 113(1):281-323.
* Han et al. (2018) Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
* Hawkins (2004) Douglas M Hawkins. 2004. The problem of overfitting. Journal of chemical information and computer sciences, 44(1):1-12.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268.
* Hu et al. (2022) Chengwei Hu, Deqing Yang, Haoliang Jin, Zhen Chen, and Yanghua Xiao. 2022. Improving continual relation extraction through prototypical contrastive learning. In Proceedings of the 29th International Conference on Computational Linguistics, pp 1885-1895, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Ji et al. (2020) Bin Ji, Jie Yu, Shasha Li, Jun Ma, Qingbo Wu, Yusong Tan, and Huijun Liu. 2020. Span-based joint entity and relation extraction with attention-based span-specific and contextual semantic representations. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mengsch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.
* Jung et al. (2020) Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. 2020. Continual learning with node-importance based adaptive group sparse regularization. Advances in neural information processing systems, 33:3647-3658.
* Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarma, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
* Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kiearam Milan, John Quan, Tiago Ramalhao, Agneszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213.
* Kruiper et al. (2020) Ruben Kruiper, Julian Vincent, Jessica Chen-Burger, Marc Desmullierz, and Ioannis Konstas. 2020. In layman's terms: Semi-open relation extraction from scientific texts. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1489-1500.
* Le et al. (2025) Minh Le, Tien Ngoc Luu, An Nguyen The, Thanh-Thien Le, Trang Nguyen, Ting Thanh Nguyen, Linh Ngo Van, and Thien Huu Nguyen. 2025. Adaptive prompting for continual relation extraction: A within-task variance perspective. In Proceedings of the AAAI Conference on Artificial Intelligence.
* Le et al. (2024a) Minh Le, An Nguyen, Huy Nguyen, Trng Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. 2024a. Mixture of experts meets prompt-based continual learning. In Advances in Neural Information Processing Systems.
* Le et al. (2024b) Thanh-Thien Le, Viet Dao, Linh Nguyen, Thi-Nhung Nguyen, Linh Ngo, and Thien Nguyen. 2024b. Sharpseq: Empowering continual event detection through sharpness-aware sequential-task learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp 3632-3644.
* Le et al. (2024c) Thanh-Thien Le, Manh Nguyen, Tong Thanh Nguyen, Linh Ngo Van, and Thien Huu Nguyen. 2024c. Continual relation extraction via sequential multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Volme 38, pages 18444-18452.
* Lee et al. (2024d) Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428 

Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. Making text embedders few-shot learners. _arXiv preprint arXiv:2409.15700_.
* Li et al. (2019) Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. 2019. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In _International conference on machine learning_, pages 3925-3934. PMLR.
* Li and Lyu (2024) Zhiming Li and Yuchen Lyu. 2024. GRADUAL: Granularity-aware dual prototype learning for better few-shot relation extraction. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 13566-13577, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Luo et al. (2024) Da Luo, Yanglei Gan, Rui Hou, Run Lin, Qiao Liu, Yuxiang Cai, and Wannian Gao. 2024. Synergistic anchored contrastive pre-training for few-shot relation extraction. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 18742-18750.
* Luo et al. (2022) Ling Luo, Po-Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022. Biored: a rich biomedical relation extraction dataset. _Briefings in Bioinformatics_, 23(5):bbac282.
* Ma et al. (2024) Shengkun Ma, Jiale Han, Yi Liang, and Bo Cheng. 2024. Making pre-trained language models better continual few-shot relation extractors. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 10970-10983, Torino, Italia. ELRA and ICCL.
* Mullner (2011) Daniel Mullner. 2011. Modern hierarchical, agglomerative clustering algorithms.
* Nguyen et al. (2023) Huy Nguyen, Chien Nguyen, Linh Ngo, Anh Luu, and Thien Nguyen. 2023. A spectral viewpoint on continual relation extraction. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 9621-9629.
* Nguyen et al. (2025) Xuan Thanh Nguyen, Duc Le Anh, Tran Quyen, Le Thanh-Thien, Linh Ngo Van, and Thien Huu Nguyeon. 2025. Few-shot, no problem: Descriptive continual relation extraction. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
* Phan et al. (2022) Hoang Phan, Anh Phan Tuan, Son Nguyen, Ngo Van Linh, and Khoat Than. 2022. Reducing catastrophic forgetting in neural networks via gaussian mixture approximation. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_, pages 106-117. Springer.
* Qin and Joty (2022) Chengwei Qin and Shafiq Joty. 2022. Continual few-shot relation learning via embedding space regularization and data augmentation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2776-2789, Dublin, Ireland. Association for Computational Linguistics.
* Rolnick et al. (2019) David Rolnick, Arnu Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning. _Advances in neural information processing systems_, 32.
* Sainz et al. (2021) Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. 2021. Label verbalization and entailment for effective zero and few-shot relation extraction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1199-1212, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 815-823.
* Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. _Advances in neural information processing systems_, 30.
* Sun et al. (2020) Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. 2020. Recurrent interaction network for jointly extracting entities and classifying relations. _arXiv preprint arXiv:2005.00162_.
* Tang et al. (2022) Wei Tang, Benfeng Xu, Yuyue Zhao, Zhendong Mao, Yifeng Liu, Yong Liao, and Haiyong Xie. 2022. Unirel: Unified representation and interaction for joint relational triple extraction. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 7087-7099.
* Thrun and Mitchell (1995) Sebastian Thrun and Tom M Mitchell. 1995. Lifelong robot learning. _Robotics and autonomous systems_, 15(1-2):25-46.
* Tran et al. (2024) Quyen Tran, Nguyen Xuan Thanh, Nguyen Hoang Anh, Nam Le Hai, Trung Le, Linh Van Ngo, and Thien Huu Nguyen. 2024. Preserving generalization of language models in few-shot continual relation extraction.
* Van et al. (2022) Linh Ngo Van, Nam Le Hai, Hoang Pham, and Khoat Than. 2022. Auxiliary local variables for improving regularization/prior approach in continual learning. In _Pacific-Asia conference on knowledge discovery and data mining_, pages 16-28. Springer.
* van den Oord et al. (2019) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation learning with contrastive predictive coding.
* Wang and Lu (2020) Jue Wang and Wei Lu. 2020. Two are better than one: Joint entity and relation extraction with table-sequence encoders. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1706-1721, Online. Association for Computational Linguistics 

* Wang et al. (2022) Peiyi Wang, Yifan Song, Tianyu Liu, Binghuai Lin, Yunbo Cao, Sujian Li, and Zhifang Sui. 2022. Learning robust representations for continual relation extraction via adversarial class augmentation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 6264-6278.
* Wang et al. (2023) Xinyi Wang, Zitao Wang, and Wei Hu. 2023. Serial contrastive knowledge distillation for continual few-shot relation extraction. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 12693-12706, Toronto, Canada. Association for Computational Linguistics.
* Wei et al. (2023) Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. Zero-shot information extraction via chatting with chatgpt. _arXiv preprint arXiv:2302.10205_.
* Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. _arXiv preprint arXiv:2007.00808_.
* Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 35-45, Copenhagen, Denmark. Association for Computational Linguistics.
* Zhao et al. (2022) Kang Zhao, Hua Xu, Jiangong Yang, and Kai Gao. 2022. Consistent representation learning for continual relation extraction. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 3402-3411, Dublin, Ireland. Association for Computational Linguistics.
* Zhao et al. (2023) Wenzheng Zhao, Yuanning Cui, and Wei Hu. 2023. Improving continual relation extraction by distinguishing analogous semantics. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1162-1175, Toronto, Canada. Association for Computational Linguistics.
* Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _International conference on machine learning_, pages 12697-12706. PMLR.

 



## Appendix A Related work

Continual Learning (CL)aims to progressively learn new knowledge from a sequence of tasks while preventing the problem of forgetting learned knowledge, known as catastrophic forgetting Thrun and Mitchell (1995). Several approaches have been explored and can be classified into three main categories: regularization/prior-based methods Kirkpatrick et al. (2017); Ahn et al. (2019); Jung et al. (2020), architecture-based methods Li et al. (2019), and memory-based methods Shin et al. (2017); Rolnick et al. (2019). Memory-based methods, which store a limited number of representative samples from the current task and replay them after subsequent tasks to reinforce prior knowledge, have become widely adopted in NLP tasks, especially in relation extraction Cui et al. (2021); Zhao et al. (2022); Hu et al. (2022).

Few-shot Continual Relation Extraction (FCRE)aligns with the scope of continual relation extraction research, but faces the additional challenge of limited sample availability for newly emerging relations. Therefore, it poses challenges related to both overfitting and catastrophic forgetting. The concept was first introduced by Qin and Joty (2022), and they introduced a data augmentation framework to address the challenges of data scarcity and catastrophic forgetting. Subsequently, several studies on FCRE have been introduced Wang et al. (2023); Chen et al. (2023); Ma et al. (2024); Luo et al. (2024); Tran et al. (2024), most of which primarily rely on the memory-based approach. In particular, Wang et al. (2023) employs serial knowledge distillation and contrastive learning, while Chen et al. (2023) introduces a framework comprising three key modules: a prototype-based classification module, a memory-enhanced module, and a consistent learning module. Meanwhile, Luo et al. (2024) improves the contrastive loss component with a multi-view perspective, serving label and instance as distinct anchors, thereby enhancing representation learning for few-shot scenarios. Recently, Tran et al. (2024) investigated the potential of LLMs in FCRE, employing mutual information maximization on the language model head to retain prior knowledge.

## Appendix B Experimental Details

### Datasets

We conduct our experiments on two benchmark datasets:

* **FewRel**Han et al. (2018) consists of 100 relations and 70,000 examples. Following the setup in Qin and Joty (2022), we use 80 relations, divided into 8 tasks, each containing 10 relations _(10-way)_. The first task \({\cal T}^{1}\) contains 100 examples per relation, while the remaining tasks are few-shot tasks performed under a _5-shot_ setting.
* **TACRED**Zhang et al. (2017) features 42 relations with 106,264 examples sourced from Newswire and Web documents. In line with the methodology from Qin and Joty (2022), we remove instances labeled as "no_relation" and distribute the remaining 41 relations across 8 tasks. Task \({\cal T}^{1}\) includes 6 relations, each with 100 examples, and the following tasks are _5-way 5-shot_ tasks, each involving 5 relations.

### Baselines

In this section, we provide a brief overview of several state-of-the-art methods in Few-Shot Continual Relation Extraction (FCRE) that serve as benchmark baselines in our evaluations, including:

* **SCKD**Wang et al. (2023) implements a structured approach to knowledge distillation, focusing on retaining knowledge from earlier tasks. Additionally, this method leverages contrastive learning with pseudo-samples to improve the differentiation between representations of various relations.
* **CPL**Ma et al. (2024) introduces a Contrastive Prompt Learning framework, which designs prompts to generalize across relation categories and applies margin-based contrastive learning to manage challenging samples. This helps reduce both catastrophic forgetting and overfitting. The method also incorporates a memory augmentation strategy by generating diverse samples using ChatGPT, which alleviates overfitting in low-resource Few-Shot Continual Relation Extraction scenarios

* **RP-CRE**(Cui et al., 2021): This method addresses Continual Relation Extraction (CRE) by utilizing stored samples to reduce the forgetting of previously learned relations. It applies K-means clustering to generate prototypes that represent each relation based on the stored data. These prototypes are then used to adjust the embeddings of new samples, allowing the model to retain knowledge of past relations while learning new ones. This approach improves memory efficiency compared to earlier CRE models, leading to better performance.
* **CRL**(Zhao et al., 2022): This approach tackles catastrophic forgetting by implementing a consistent representation learning strategy. It focuses on maintaining stable relation embeddings through contrastive learning and knowledge distillation during the replay of stored samples. The method applies supervised contrastive learning on a memory bank dedicated to each new task, followed by contrastive replay of memory samples and knowledge distillation to preserve knowledge of previous relations. This consistent representation learning effectively mitigates forgetting.
* **CRECL**(Hu et al., 2022): This method enhances traditional few-shot learning by introducing additional constraints on the training data. It achieves this by incorporating information from support instances to enrich instance representations. Additionally, it promotes open-source task enrichment to enable cross-domain knowledge aggregation and introduces the TinyRel-CM dataset, specifically designed for few-shot relation classification with limited training data. Experimental results demonstrate its effectiveness in improving performance in low-data scenarios.
* **ERDA**(Qin and Joty, 2022): This work introduces Continual Few-Shot Relation Learning (CFRL) as a new challenge, highlighting the limitations of existing methods that require extensive labeled data for new tasks. CFRL aims to learn new relations with minimal data while avoiding catastrophic forgetting. To address this, ERDA proposes a technique based on embedding space regularization and data augmentation. This approach enforces constraints on relational embeddings and supplements relevant data through self-supervision. Comprehensive experiments demonstrate that ERDA significantly outperforms previous state-of-the-art methods in CFRL settings.
* **ConPL**(Chen et al., 2023) presents a method with three key components: a prototype-based classification module, a memory-enhanced module, and a consistent learning module aimed at preserving distribution consistency and minimizing forgetting. Additionally, ConPL utilizes prompt learning to improve representation learning and incorporates focal loss to reduce confusion between closely related classes.
* **CPL+MI**(Tran et al., 2024) introduces an innovative approach to improve FCRE models by effectively utilizing the language model (LM) heads. By maximizing the mutual information between these heads and the primary classifiers, the method better preserves prior knowledge from pre-trained backbones while also enhancing representation learning.

It is important to note that we reproduce the results of ConPL (Chen et al., 2023) using the same settings as SCKD and CPL. This adjustment is made because the evaluation strategy in the original paper is not feasible for continual learning scenarios.

### Pre-trained language models

* For BERT-based models: We use BERT-base-uncased checkpoint1 on Hugging Face. Footnote 1: [https://huggingface.co/bert-base-uncased](https://huggingface.co/bert-base-uncased)
* For LLM2Vec-based models: We use three checkpoints on Huggingface:
* Meta-Llama-3-8B-Instruct-mntp-supervised  2,
 

* LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised 4 checkpoint on Hugging Face. Footnote 3: [https://huggingface.co/McGill1-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised](https://huggingface.co/McGill1-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-supervised)
* For BGE models: We use the bge-en-icl 5 checkpoint on Hugging Face.

Footnote 5: [https://huggingface.co/BAAI/bge-en-icl](https://huggingface.co/BAAI/bge-en-icl)

### Evaluation and Training Configurations

For each reported result, we conduct 6 independent runs with different random seeds and report the mean and the corresponding standard deviation.

Evaluation Metric:We use final average accuracy to evaluate methods in our experiments. The average accuracy after training task \(T_{j}\) is calculated as follows:

\[ACC_{j}=\frac{1}{j}\sum_{i=1}^{j}ACC_{j,i}\]

where \(ACC_{j,i}\) is the accuracy on the test set of task \(T_{i}\) after training the model on task \(T_{j}\).

Training Configuration:Our BERT-based experiments were conducted on an NVIDIA RTX 3090 GPU with 24GB of memory. For experiments with the LLM2Vec and BGE backbone, we utilized an NVIDIA A100 GPU with 80GB of VRAM. The operating system used across all experiments was Ubuntu Server 18.04.3 LTS.

Details of hyperparameter search:

* Learning rate: \(\{\mathbf{1}\times\mathbf{10^{-5}}\), \(2\times 10^{-5}\), \(\mathbf{1}\times\mathbf{10^{-4}}\}\)
* \(\alpha\): { 0.1, 0.15, **0.2**, **0.25**}
* \(\lambda_{1}\): {0.5, **1.0**, 1.5, 2.0, 2.5}
* \(\lambda_{2}\): {0.5, **1.0**, 1.5, **2.0**, 2.5}
* \(\lambda_{3}\): {**0.25**, **0.5**, 0.75, 1.0}
* \(\tau~{}\mathcal{L}_{\text{CMI}}\): {**0.01**, 0.02, 0.03, 0.04, **0.05**}
* \(m_{1}\): {**1.0**, 2.0}
* \(m_{2}\): {**1.0**, 2.0}
* \(\theta\): { **0.1**, 0.2, **0.3**, 0.4, **0.5**, 0.6, 0.7, 0.8}

Lora config target modules: "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj".

Additionally, Tables 3 and 4 provide the optimal values of hyperparameters for each model backbone.

## Appendix C Additional experimental results

### Ablation study

Effectiveness of Each Loss Component:Figure 1 and Table 1 have demonstrated the effectiveness of our proposed loss functions in enhancing discriminative representations for similar relations, thereby mitigating catastrophic forgetting and improving overall performance. We further investigate the impact of each loss component on performance through an ablation study by removing each component from the total loss function. Table 5 demonstrates that the absence of any loss component leads to a drop in performance. Specifically, \(\mathcal{L}_{CMI}\) (6) exhibits a significant impact, underscoring the importance of leveraging label descriptions to effectively distinguish similar relations. Besides, the results present 

limited contribution of \({\cal L}_{DT}\) to the model performance. One possible explanation is that, throughout the training process, we observe that the identified clusters exhibit stability and demonstrate minimal variation after a few optimization steps. This suggests that the model learns to cluster and identify similar classes effectively early on, with the support of all losses. As a result, which focuses on the relationship between samples and their cluster centroids, may contribute less during later training steps, where the emphasis shifts toward distinguishing samples between different relations. Nonetheless, the incorporation of still results in approximately a 1% improvement in model performance. Given already high accuracy and the challenging of FCRE, this incremental improvement is meaningful and highlights its effectiveness.

Influence of Clustering Algorithms:To assess the influence of different clustering algorithms in our method, we evaluate its performance using various clustering techniques. Specifically, we compare Agglomerative Clustering, K-means, and DBSCAN while employing BERT as the backbone model and conducting experiments on the TACRED dataset. The results, summarized in the Table 6, indicate that Agglomerative Clustering achieves superior performance, underscoring its effectiveness and suitability for our approach.

\begin{table}
\begin{tabular}{l l c} \hline Hyperparameter & Value \\ \hline Epochs & \(10\) \\ Learning rate & \(1\times 10^{-5}\) & Eepochs \\  & \(0.25\) \\ \(\tau~{}{\cal L}_{\text{CMI}}\) (FewRel) & \(0.01\) \\ \(\tau~{}{\cal L}_{\text{CMI}}\) (TACRED) & \(0.05\) \\ \(\tau~{}{\cal L}_{\text{WSC}}\) & \(0.1\) \\ \(\theta\) (TACRED) & \(0.3\) \\ \(\theta\) (FewRel) & \(0.1\) \\ \hline Encoder output size & \(768\) \\ BERT input max length & \(256\) \\ \hline \(\lambda_{1}\) & \(1.0\) \\ \(\lambda_{2}\) (FewRel) & \(1.0\) \\ \(\lambda_{2}\) (TACreED) & \(2.0\) \\ \(\lambda_{3}\) & \(0.25\) \\ \hline Soft prompt initialization & Random \\ Soft prompt phrase length & 3 \\ Soft prompt number of phrases & 4 \\ \hline \end{tabular} 
\begin{tabular}{l c} \multicolumn{3}{c}{Task} \\ \cline{2-3} \multirow{2}{*}{Method} & \({\cal T}^{1}\) & \({\cal T}^{2}\) & \({\cal T}^{3}\) & \({\cal T}^{4}\) & \({\cal T}^{5}\) \\ \hline \multicolumn{3}{c}{**FWebRel (10-way–5-shot)**} \\ \hline Sirus & \(94.74_{+0.27}\) & \(87.12_{+2.21}\) & \(81.06_{+1.52}\) & \(77.49_{+2.58}\) & \(75.47_{+2.6}\) & \(72.48_{+1.75}\) & \(70.6_{+1.31}\) & \(69.16_{+0.43}\) \\ w/o \({\cal L}_{WSC}\) & \(94.08_{+0.44}\) & \(85.46_{+2.23}\) & \(78.69_{+1.72}\) & \(75.10_{+2.71}\) & \(73.03_{+2.76}\) & \(70.24_{+1.78}\) & \(68.51_{+1.23}\) & \(66.85_{+0.77}\) \\ w/o \({\cal L}_{CMI}\) & \(94.71_{+0.38}\) & \(79.23_{+3.41}\) & \(72.77_{+1.01}\) & \(70.39_{+1.98}\) & \(70.32_{+1.94}\) & \(67.76_{+1.74}\) & \(66.40_{+1.44}\) & \(64.65_{+1.22}\) \\ w/o \({\cal L}_{DT}\) & \(94.69_{+0.28}\) & \(86.68_{+2.14}\) & \(80.71_{+1.78}\) & \(77.13_{+2.26}\) & \(75.52_{+2.66}\) & \(72.77_{+1.73}\) & \(70.41_{+1.33}\) & \(68.29_{+0.84}\) \\ \hline \multicolumn{3}{c}{**TACRED (5-way-5-shot)**} \\ \hline Sirus & \(87.41_{+0.41}\) & \(84.28_{+7.28}\) & \(76.38_{+3.99}\) & \(73.86_{+4.16}\) & \(68.06_{+5.57}\) & \(66.64_{+5.76}\) & \(62.74_{+3.92}\) & \(60.68_{+3.53}\) \\ w/o \({\cal L}_{WCSC}\) & \(\mathbf{87.79_{+0.41}}\) & \(\mathbf{84.45_{+6.18}}\) & \(75.91_{+2.41}\) & \(73.67_{+4.61}\) & \(67.76_{+6.66}\) & \(66.35_{+5.10}\) & \(61.55_{+4.78}\) & \(59.48_{+2.59}\) \\ w/o \({\cal L}_{CMI}\) & \(86.49_{+0.39}\) & \(79.98_{+0.53}\) & \(71.2_{+4.45}\) & \(66.25_{+5.37}\) & \(62.47_{+5.46}\) & \(61.23_{+5.21}\) & \(56.08_{+3.68}\) & \(54.30_{+2.88}\) \\ w/o \({\cal L}_{DT}\) & \(87.31_{+0.54}\) & \(84.27_{+6.69}\) & \(75.77_{+4.66}\) & \(72.8_{+4.27}\) & \(66.84_{+4.53}\) & \(66.12_{+5.84}\) & \(61.86_{+3.96}\) & \(59.5_{+3.66}\) \\ \hline \end{tabular}
\end{table}
Table 4: Hyperparameters setting for LLMEs backbone.

\begin{table}
\begin{tabular}{l r} \hline Hyperparameter & Value \\ \hline Epochs & \(10\) \\ Learning rate & \(1\times 10^{-5}\) \\ \(\alpha\) & \(0.25\) \\ \(\tau~{}{\cal L}_{\text{CMI}}\) (FewRel) & \(0.01\) \\ \(\tau~{}{\cal L}_{\text{CMI}}\) (TACRED) & \(0.05\) \\ \(\tau~{}{\cal L}_{\text{WSC}}\) & \(0.1\) \\ \(\theta\) (TACRED) & \(0.3\) \\ \(\theta\) (FewRel) & \(0.1\) \\ \hline Encoder output size & \(768\) \\ BERT input max length & \(256\) \\ \hline \(\lambda_{1}\) & \(1.0\) \\ \(\lambda_{2}\) (FewRel) & \(1.0\) \\ \(\lambda_{2}\) (TACRED) & \(2.0\) \\ \(\lambda_{3}\) & \(0.25\) \\ \hline Soft prompt initialization & Random \\ Soft prompt phrase length & 3 \\ Soft prompt number of phrases & 4 \\ \hline \end{tabular} 
\begin{tabular}{l c} Tasks \\ \({\cal T}^{1}\) & \({\cal T}^{2}\) & \({\cal T}^{3}\) & \({\cal T}^{4}\) & \({\cal T}^{5}\) \\ \hline \multicolumn{3}{c}{**FWebRel (10-way–5-shot)**} \\ \hline Sirus & \(94.74_{+0.27}\) & \(87.12_{+2.21}\) & \(81.06_{+1.52}\) & \(77.49_{+2.58}\) & \(75.47_{+2.6}\) & \(72.48_{+1.75}\) & \(70.6_{+1.31}\) & \(69.16_{+0.43}\) \\ w/o \({\cal L}_{WSC}\) & \(94.08_{+0.44}\) & \(85.46_{+2.23}\) & \(78.69_{+1.72}\) & \(75.10_{+2.71}\) & \(73.03_{+2.76}\) & \(70.24_{+1.78}\) & \(68.51_{+1.23}\) & \(66.85_{+0.77}\) \\ w/o \({\cal L}_{CMI}\) & \(94.71_{+0.38}\) & \(79.23_{+3.41}\) & \(72.77_{+1.01}\) & \(70.39_{+1.98}\) & \(70.32_{+1.94}\) & \(67.76_{+1.74}\) & \(66.40_{+1.44}\) & \(64.65_{+1.22}\) \\ w/o \({\cal L}_{DT}\) & \(94.69_{+0.28}\) & \(86.68_{+2.14}\) & \(80.71_{+1.78}

 

Hyper-parameter Sensitivity:To examine the impact of weighted parameters for each loss function, we conducted experiments by varying \(\lambda_{2}\) and \(\lambda_{3}\) while keeping \(\lambda_{1}=1\) to limit the exponential growth of possible configurations. The results, presented in the Table 7 for the TACRED dataset using BERT backbone, exhibit low standard deviation across different parameter settings, suggesting minimal sensitivity to these hyperparameters. Notably, the lowest accuracy (57.16%) is observed when \(\lambda_{2}\) and \(\lambda_{3}\) are set to small values, emphasizing the critical role of each loss function in enhancing the model's performance.

Computational Overhead:To assess the computational efficiency of Sirus, we compare its additional time cost against CPL and CPI. We measure the average training time per epoch on an RTX 3090 GPU using the TACRED dataset, with a fixed batch size of 16 and BERT as the backbone model. Compared to CPL (10.34s) and CPI_MI (11.30s), Sirus (22.89s) introduces some additional computational overhead, primarily due to processing label descriptions and running the Agglomerative Clustering algorithm (L4-5 in Algorithm 1). However, the time spent on clustering is minimal, averaging only 0.05 seconds, with most of the overhead attributed to updating description embeddings via forward passes. However, in few-shot learning scenarios, the number of training samples is small, with only a few samples per class. As the training time of Sirus is short, the trade-off between its time complexity and significant improvement in performance is acceptable. This trade-off enables Sirus to outperform the two CPL-based models. On the other hand, CPL and CPL_MI methods rely on data augmentation techniques using LLMs to generate additional data, which increases computational complexity. The time cost for these methods can scale with the number of extra generated samples, further potentially adding overhead compared to the more efficient Sirus approach.

### Relation and Description

Table 8 provides details of the relations and their descriptions corresponding to each class index depicted in Figure 1. These descriptions provide clearer evidence of their similarity.

\begin{table}
\begin{tabular}{l c} \hline \hline Cluster Algorithm & Accuracy on the final task \\ \hline \hline K-means with K = \(\mathcal{R}\) / 2 & 66.89 \\ \hline K-means with K = \(\mathcal{R}\) / 3 & 67.20 \\ \hline DBSCAN & 67.18 \\ \hline Agglomerative Clustering & **69.16** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Accuracy after training on the final task using different clustering algorithms. \(\mathcal{R}\) represents the total number of relations in the dataset.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline  & \(\lambda_{2}=0.5\) & \(\lambda_{2}=1.0\) & \(\lambda_{2}=1.5\) & \(\lambda_{2}=2.0\) \\ \hline \(\lambda_{3}=0.25\) & 57.16 & 59.37 & 59.57 & 60.68 \\ \hline \(\lambda_{3}=0.5\) & 58.35 & 59.29 & 60.28 & 60.58 \\ \hline \(\lambda_{3}=0.75\) & 58.94 & 58.84 & 59.94 & 60.44 \\ \hline \(\lambda_{3}=1.0\) & 59.23 & 60.06 & 59.55 & 60.54 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Accuracy variations with different weighted parameter settings for each loss function.

 



\begin{table}
\begin{tabular}{c|l|p{142.3pt}}
**Index** & **Relation** & **Description** \\ \hline
1 & headquarters location & city where an organization’s headquarters is or has been situated \\ \hline
2 & work location & location where persons or organizations were actively participating in employment, business, or other work \\ \hline
3 & location of formation & location where a group or organization was formed \\ \hline
4 & located in the administrative territorial entity & the item is located on the territory of the following administrative entity \\ \hline
5 & country of citizenship & the object is a country that recognizes the subject as its citizen \\ \hline
6 & country & sovereign state of this item (not to be used for human beings) \\ \hline
7 & country of origin with density & country of origin of this item (creative work, food, phrase, poduct, etc.) \\ \end{tabular}
\end{table}
Table 8: Corresponding relation and its description to class index in Figure 1.