Actively Supervised Clustering for Open Relation Extraction
Year: 2023

Abstract:
Current clustering-based Open Relation Extraction (OpenRE) methods usually adopt a two-stage pipeline. The first stage simultaneously learns relation representations and assignments. The second stage manually labels several instances and thus names the relation for each cluster. However, unsupervised objectives struggle to optimize the model to derive accurate clustering assignments, and the number of clusters has to be supplied in advance. In this paper, we present a novel setting, named actively supervised clustering for OpenRE. Our insight lies in that clustering learning and relation labeling can be alternately performed, providing the necessary guidance for clustering without a significant increase in human effort. The key to the setting is selecting which instances to label. Instead of using classical active labeling strategies designed for fixed known classes, we propose a new strategy, which is applicable to dynamically discover clusters of unknown relations. Experimental results show that our method is able to discover almost all relational clusters in the data and improve the SOTA methods by 10.3% and 5.2%, on two datasets respectively.

heading: 1 Introduction
text: Relation extraction (RE) aims to detect and extract the potential relation between the given entity pair in unstructured text. The extracted relation facts play a vital role in many downstream applications, such as knowledge base population (Ji and Grishman, 2011), search engine (Schlichtkrull et al., 2018), and question answering (Yu et al., 2017). To deal with the emerging unknown relational types in the real world, Open Relation Extraction (OpenRE) has been widely studied.
The clustering-based unsupervised relation discovery is a classical paradigm for OpenRE (Yao
*Equal Contributions. †Corresponding authors.
Clustering Assignments Relation
Labeling
Query Label
Active Supervision
Relation Labeling
Unsupervised two-stage OpenRE
Clustering Model
Clustering Model
Actively Supervised OpenRE
Figure 1: Compared with the existing unsupervised two-stage methods, our method can provide explicit supervision for clustering by alternately performing clustering learning and relation labeling. Note that the human effort of the two settings is comparable.
et al., 2011; Marcheggiani and Titov, 2016; Elsahar et al., 2017). It can discover potential relations, by grouping several instances into relational clusters, and then manually labeling a few instances to name the relation of each cluster. Recently, Hu et al. (2020) introduced a deep clustering framework (Caron et al., 2018) into OpenRE. They iteratively cluster the relation representations that are produced by large pretrained models and use the cluster assignments as pseudo-labels to refine the representations. Unfortunately, the above unsupervised methods struggle to learn good enough representations, and the cluster assignments are error-prone. When multiple relations are mixed in a cluster, it becomes difficult to name the cluster. Hence, instead of regarding OpenRE as a totally unsupervised task, researchers leverage the labeled data of predefined relations to provide explicit supervision signals for clustering learning (Wu et al., 2019; Zhao et al., 2021), and achieve superior results.
Different from the above two-stage methods, in this work, we present a new setting named actively supervised clustering for OpenRE (ASCORE). As
4985
shown in fig. 1, our insight lies in that clustering learning (i.e, deep clustering) and relation labeling can be alternately performed. In an iteration, a small number of key instances are selected for labeling. The unknown relations expressed by these instances are correspondingly discovered. More importantly, these labeled instances can provide explicit supervisory signals for clustering learning. The improved relation representations form a better cluster structure, which in turn is able to benefit the discovery of the neglected relations. Since potential relations are dynamically discovered in iterations, the number of clusters does not need to be provided in advance.
Along with this setting, we design an active labeling strategy tailored for clustering. First, all instances are encoded to points in the representation space, where the clustering is performed. The goal of the strategy is to select the most informative points for labeling. Intuitively, two points that are far from each other in representation space usually express different relations. To discover as many relations as possible, we introduce a distance regularization to the strategy, so that diversified relation discovery can be facilitated. To prevent over-fitting caused by training with limited active labeled instances, all the selected key points are required to be the points of maximum local density. By doing so, a large number of high-quality pseudo-labels can be obtained, by assigning active labels to unlabeled data in a small neighborhood. To mitigate the error propagation issue, different loss functions are assigned to active labels and pseudo-labels with different reliability for clustering learning. Experimental results show that (1) the actively supervised method improves the SOTA two-stage methods by a large margin without a significant increase in human effort. (2) the proposed active strategy can discover more relational clusters, compared with the classical active strategy.
To summarize, the main contributions of this work are as follows: (1) We present a new setting named actively supervised clustering for OpenRE, providing the necessary guidance for clustering without a significant increase in human effort. (2) Design of a new active labeling strategy tailored for clustering, that can effectively discover potential relational clusters in unlabeled data. (3) This method improves the SOTA two-stage methods by 10.3% and 5.2% on two well-known datasets,
respectively.

heading: 2 Related Work
text: Clustering-based OpenRE: The clustering-based paradigm considers relation discovery as a twostage pipeline, which clusters relational data first, and then manually labels relational semantics for each cluster. Conventional methods cluster instances by human-defined linguistic features (Yao et al., 2011; Marcheggiani and Titov, 2016; Elsahar et al., 2017), such as entity words/type, dependency paths, trigger words, context POS tags. Recently, many studies have shown that pretrained models learn diversified linguistic knowledge (Jawahar et al., 2019; Clark et al., 2019; Goldberg, 2019; Zhao et al., 2022). Hu et al. (2020) leverage the self-supervised signals provided by the pretrained model to iteratively learn relation representations and optimize clustering. Due to the lack of strong supervision, it is difficult for the above methods to produce satisfactory clustering results. Although some works (Wu et al., 2019; Zhao et al., 2021) try to use the labeled data of predefined relations to complete the missing supervision, the semantic gap between predefined and open relations leads to negative clustering bias, especially when these relations come from different domains (Zhao et al., 2021). By performing clustering learning and relation labeling alternately, our actively supervised method can provide strong supervision and improve the two-stage methods by a large margin. In the main results (sec. 5), we achieve this improvement at the cost of only two active labels for each relation on average. For two-stage methods, relation labeling for each cluster requires at least one (usually more) instance to be manually observed. Therefore, there is no significant increase in human effort. Active Learning: Active learning is a research field with high relevance to the proposed methods. In the research field, a classical method is uncertainty-based sampling(Roth and Small, 2006; Wang and Shang, 2014a; Tong and Koller, 2001). The uncertainty can be defined based on the posterior probability of a predicted class or the distances to the decision boundaries. In the context of deep learning, MC Dropout (Gal et al., 2017) is an effective way for uncertainty estimation, but the computationally inefficient limits its application in large-scale datasets. Recently, representative sampling is attracting lots of attention (Sener and
BERT
[CLS] [E1] Bill Gates [\E1] was born in [E2] Seattle[\E2] in 1995 [SEP] Relation Representation Encoder Open Relational Data
Position
Density Long Distance
High Density
1
2
Key Point Selection & Labeling
Relation Classifier
Clustering
Clustering Assignments and
Discovered Relations
Backprop
Active Supervised Clustering
Figure 2: Overview of the training pipeline for our actively supervised clustering setting. In each iteration, a few key points are selected for relation labeling. The rest instances are clustered to the nearest key points. Some highly reliable cluster assignments are used as pseudo-labels for relation representation learning.
Savarese, 2018; Ash et al., 2019a), which selects data points that represent the distribution of an unlabeled pool. Unlike the classical labeling strategies designed for fixed classes, the proposed strategy is encouraged to discover new relational clusters while improving the clustering of relations that have been discovered.

heading: 3 Approach
text: In this work, we present a new setting named actively supervised clustering for OpenRE (ASCORE), which fuses the isolated two-stage pipeline to guide clustering learning. Fig. 2 illustrates the training pipeline. The OpenRE problem addressed in this work is formally stated as follows. Given as input an open relational dataset D = {xi|i = 1, .., N}, the goal is to discover and label potential relations R = {ri|i = 1, ..,K} in the open data and cluster the corresponding instances. Note that the number of relations K in D is unknown.

heading: 3.1 Overview
text: The ASCORE is based on deep clustering (Caron et al., 2018), which is a common practice for OpenRE (Hu et al., 2020; Zhao et al., 2021), that iteratively clusters the representations of input instances, and uses the cluster assignments as pseudo-labels to learn the relation representations. We introduce explicit supervision to deep clustering by alternately performing clustering learning and relation labeling. The actively labeled points can serve as a basis to facilitate accurate pseudo-label estimation and improve representation learning. The improved relational representation in turn
benefits relation labeling to discover more relations. As illustrated in Figure 2, the training pipeline of ASCORE consists of the following steps:
Encoding Step: This step aims to obtain the relation representation hi of each input instance xi ∈ D, laying the groundwork for clustering and relation discover. First, the contextual information of xi is encoded to the entity pair representation henti , using a pretrained BERT (Devlin et al., 2018) encoder. To avoid data sparsity and low efficiency of clustering in high-dimensional space, an autoencoder is used to transform henti into a low-dimensional clustering-friendly representation hi.
Labeling Step: This step aims to discover potential relations in open dataset D and guide clustering. At the sth iteration, a set D∗s ∈ D is actively labeled, including B key points. These key points are required to be local maximum in density, so a large number of high-quality pseudo labels can be obtained by assigning active labels to unlabeled data in a small neighborhood. Instead of focusing only on the improving clustering of the discovered relations, all key points in D∗ = D∗1 ∪ .. ∪ D∗s are required to be far away from each other to facilitate the discovery of new relations.
Learning Step: This step aims to learn clustering relational data, using an actively labeled D∗. Specifically, each unlabeled point xi ∈ D is clustered to the nearest key point x∗j ∈ D∗ and the pseudo label is ŷi = y∗j . The reliability of ŷi increases as the distance in representation space between xi and x∗j decreases. Cross-entropy loss (resp. divergence-based contrastive loss) is
used for pseudo labels with high (resp. moderate) reliability, to optimize relation representations and thus to improve clustering. With the help of active supervision, separated subclusters expressing the same relation approach each other, while mixed subclusters expressing different relations are separated. Existing unsupervised methods are inherently difficult to handle such errors.
The above three steps are performed iteratively to gradually improve the model performance. In the following sections, we will elaborate on the model structure, labeling strategy, and training methods involved in the above three steps.

heading: 3.2 Relation Representation Encoder
text: Given a relation instance xi = {w1, w2, ..., wn}, in which four reserved word pieces [E1], [\E1], [E2], [\E2] are used to mark the beginning and end of each entity, the relation representation encoder f aims to encode the contextual relational information of instance xi into a fixed-length representation henti = f(xi) ∈ Rd. The encoder f is implemented as BERT. Specifically:
h1, ...,hn = BERT(w1, ..., wn) (1)
hent = 〈 h[E1]|h[E2] 〉 . (2)
Following Soares et al. (2019), the fixed-length representations are obtained, by concatenating the hidden states of marker [E1], [E2] (i.e., h[E1] ,h[E2]) and ⟨·|·⟩ is the concatenation operator.
However, clustering data in high-dimensional space is time-consuming and the data sparsity leads to sub-optimal clustering results. Therefore, an autoencoder is trained by reconstruction loss Lrec and the encoder part is retained to transform highdimensional hent to a low-dimensional clusteringfriendly relation representation h.

heading: 3.3 Key Point Selection Module
text: In this section, the proposed key point selection method will be explained, including the labeling strategy and the conditions for stopping annotation. Labeling strategy. The labeling strategy is based on the following criteria. First, the selected key points are the local maximum in density. Generally, labels do not drastically change within a small neighborhood, and therefore, the first criteria enable to find a lot of unlabeled data within a small neighborhood of each key point, and accurately estimate their pseudo-labels. To find these local maximum, it is calculated the euclidean distance
between the relation representations {hi}i=1,2,...,N obtained in encoding step, and the distance matrix D ∈ RN×N is constructed as follows:
Dij = ∥hi − hj∥22 , (3)
where Dij is the distance between two relational instance xi and xj . The potential computational cost to process large-scale datasets can be solved by sampling a small subset. Based on distance matrix D, a density ρi is further defined for each relation instance xi. A larger ρi indicates a larger number of instances around xi:
ρi = N∑
j=1
sign(Dc −Dij), (4)
where sign() is the sign function and Dc is a threshold.
To avoid the problem that all the labeled points are concentrated in several high-density areas and missing most long-tail relations, the second criteria is to keep these key points away from each other in clustering space. Specifically, a sparsity index ξi is defined for each instance xi ∈ D.
ξi = { minj,ρj>ρi Dij , ρi < ρmax maxj Dij ρi = ρmax
(5)
Intuitively, a larger ξi indicates that the instance xi is a local maximum of density in a larger radius. Based on the density ρi and sparsity index ξi of each instance, the labeling strategy can be formally stated as follows. In each iteration, choose B points with the highest density and their distance from each other is greater than ξc.
D∗s = TopBρ{xi|ξi > ξc, xi ∈ D} (6)
To effectively support iterative labeling and maintain the diversity of key points, in sth iteration, each new key point xi should be as far away from the existing key point in D∗ = D∗1 ∪ ... ∪ Ds−1 as possible. Therefore, for each instance xi, the sparsity index are modified as follows:
d = min xj∈D∗
||hi − hj ||22 (7)
ξi = min(ξi, d), (8)
After the sth iterations, the result is the new active labeled set D∗ = D∗ ∪ D∗s . Conditions for stopping annotation. Too few queries will lead to missing some relations, while
too many queries will lead to unnecessary costs. Here we give a simple strategy to determine when to stop labeling. (1) First, users can determine the maximum number of actively labeled instances, N∗, based on their annotation budget. (2) Since our labeling strategy takes into account the diversity of key points, new relations are constantly discovered during the initial phase of iteration. When no new relations are discovered in two or more consecutive iteration steps (it means that most relations have been found), the labeling should be stopped.

heading: 3.4 Training Methods
text: Pseudo Label Estimation. Given the active labeled set D∗, each of the rest unlabeled points xi ∈ D is clustered to the nearest key point x∗j ∈ D∗ and ŷi is estimated as y∗j . Intuitively, the accuracy of the pseudo label decreases with the increase of the distance between xi and x∗j . The reliability r of pseudo labels is defined as follows:
ri = ∥∥hi − h∗j ∥∥−1 2 , (9)
where hi and h∗j denote the representation of xi and x∗j , respectively. ∥ · ∥−12 denotes reciprocal of L2 norm. Model Optimization. Given the pseudo label ŷi and its reliability ri for each unlabeled data xi ∈ D, the relation representation is refined to improve clustering in the next iteration. Specifically, we first filter out a highly reliable subset Dh = {(xi, ŷi)|ri > rh} and use a softmax classifier to convert entity pair representation henti into the probability distribution on discovered relations (denoted as Pi). The model is optimized with cross entropy loss for fast convergence:
Lce = CrossEntropy(ŷi,Pi). (10)
Note that the number of instances in Dh is small. To avoid that the model only learns simple features, the threshold is broaden, and a moderately reliable subset Dm = {(xi, ŷi)|ri > rm} containing more instances is built. To mitigate the negative impact of noise in Dm, a binary contrastive loss is further introduced:
Lbce = Lŷi=ŷj + Lŷi ̸=ŷj (11) Lŷi=ŷj = Dkl(P∗i ||Pj) +Dkl(Pi||P∗j ) (12) Lŷi ̸=ŷj = Hσ(Dkl(P∗i ||Pj)) +Hσ(Dkl(Pi||P∗j )) (13)
Hσ(x) = max(0, σ − x), (14)
Algorithm 1: ASCORE Input: A open dataset D = {xi}Ni=1
1 repeat 2 Perform the encoding step (sec. 3.2).
Get relation representation {hi}Ni=1 for instances in D;
3 if The conditions for stopping annotation are not met then 4 Perform the labeling step (sec. 3.3). Get the sth active labeled D∗ = D∗ ∪ Ds.; 5 end 6 Perform the learning step (sec. 3.4).
Estimate pseudo labels and reliability scores. Use the corresponding loss for representation learning.
7 until convergence; 8 Return Discovered relation set R and the
cluster assignment ŷi ∈ R of each instance xi ∈ D.
where σ is a hyperparameter and P∗ denotes that P is assumed to be a constant for asymmetry. Dkl denotes KL divergence. The probability distribution P will be pulled closer or farther depending on whether the labels of the sample pairs are the same. In each iteration, if the annotator finds new relations, the parameters of the softmax classifier are reinitialized to deal with the new relations. Alg.1 shows an algorithm flow that is able to clearly summarize the proposed method.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: Experiments are conducted on two standard and one constructed dataset. Note that the compared baselines follow different settings. As will be described in sec. 4.2, RSN and RoCORE leverage labeled data of predefined relations, while RWHAC and SelfORE follow unsupervised setting. To fairly compare all methods in a uniform setting, the first half of the relations in each dataset are held out as predefined relations. Specifically, in TACRED, 21 relations are held out, while in FewRel and FewRel-LT, the number is 40. TACRED (Zhang et al., 2017). TACRED is a largescale manually annotated RE dataset, covering 41 relations. Following Wu et al. (2019); Hu et al. (2020); Zhao et al. (2021), the instances labeled as no_relation are removed and the rest are used
for training and evaluation.
FewRel (Han et al., 2018). FewRel is a manually annotated dataset that contains 80 types of relations, each of which has 700 instances. However, in real-world OpenRE scenarios, unseen relations in unlabeled data usually follow a long-tailed distribution. To eliminate this inconsistency and accurately evaluate model performance in realworld scenarios, we construct a long-tail FewRel dataset as follows.
FewRel-LT. The FewRel-LongTail dataset. We number the 40 unseen relations in FewRel from 0 to 39, and calculate the number of samples according to y = 7000.5∗id+1 . The number of samples in the predefined subset remains unchanged.

heading: 4.2 Compared Methods
text: To evaluate the effectiveness of the actively supervised clustering, the following SOTA twostage OpenRE methods are used for comparison.
HAC with Re-weighted Word Embeddings (RWHAC) (Elsahar et al., 2017). A clustering-based OpenRE method. The model constructs relational feature based on entity types and the weighted sum of pretrained word embeddings.
Relational Siamese Network (RSN) (Wu et al., 2019). This method learns similarity metrics of relations from labeled data of pre-defined relations and then transfers the relational knowledge to identify novel relations in unlabeled data.
Self-supervised Feature Learning for OpenRE (SelfORE) (Hu et al., 2020). SelfORE exploits weak, self-supervised signals in pretrained language model for adaptive clustering on contextualized relational features.
A Relation-oriented Clustering Method (RoCORE) (Zhao et al., 2021). RoCORE leverages the labeled data of predefined relations to learn a clustering-friendly representation, which is used for new relations discovery.
To show the superiority of the proposed labeling strategy, the actively supervised clustering is combined with the following classical active learning strategies for comparison. Specifically, RANDOM, CONFIDENCE (Wang and Shang, 2014b), MARGIN (Roth and Small, 2006), ENTROPY (Wang and Shang, 2014a) and GRADIENT (Ash et al., 2019b) are included. We provide a brief introduction to these methods in appendix A.1.

heading: 4.3 Implementation Details
text: Following Hu et al. (2020) and Wu et al. (2019), 20% of the data in each dataset are held out for validation and hyperparameter selection. We use the Adam (Kingma and Ba, 2014) as the optimizer, with a learning rate of 1e − 4 and batch size of 100 for all datasets. The threshold Dc is given by the value of an element ranked top 40% in D from large to small. For each iteration, we label B = 20 samples. ξ is set to the value when the number of candidates is 1.2B. Some important hyperparameters rh and rm are analyzed in sec. 6.3. For a fair comparison, all active strategies select the same number of key points for labeling. Specifically, 40, 80 and 80 key points are labeled on the three datasets TACRED, FewRel, and FewRelLT respectively. All experiments are conducted with Pytorch 1.7.0, using an NVIDIA GeForce RTX 3090 with 24GB memory.

heading: 5 Main Results
text: Table 1 shows the model performances on three datasets. In this section, the experiment focuses on the following two questions.

heading: 5.1 Does inaccurate estimation of the number of relations have an impact on clustering?
text: One drawback of the most existing two-stage OpenRE methods is that the number of clusters K has to be given in advance, which is impractical in real applications. When K is underestimated (from to ), the clustering performance of the SOTA unsupervised method, SelfORE, on the three datasets decreases by an average of 7.13%, while the same metric regarding RoCORE, the SOTA supervised method, is 18.10%. Furthermore, it is observed an extremely unbalanced precisionrecall value in the B3 metric (much lower precision and higher recall), which indicates that the model tends to mix multiple relations in the same cluster. Obviously, such clustering results will have a negative impact on the relation labeling. In other words, it is difficult to determine which relation this cluster corresponds to. When K is overestimated (due to space limitation, please look at table 4 for results of overestimation), the same relation tends to be clustered into multiple subclusters. Repeated labeling of these subclusters brings a significant increase in human effort. In contrast, the ASCORE dynamically discovers relational clusters through active iteration, breaking the impractical
assumption that K is known in advance.

heading: 5.2 Is the actively supervised setting better than the two-stage setting?
text: The two settings are compared, from the following two perspectives.
In terms of clustering performance, the actively labeled data can provide valuable supervision signals for clustering learning. Compared with RoCORE, a strong baseline supervised by predefined relations, the proposed method improves the four metrics by an average of 10.3% and 5.2% on long-tail TACRED and FewRel-LT, respectively. Long-tail relation distribution is very common in real world. On the uniform FewRel dataset, the ASCORE achieves comparable results. It is worth noting that the number of clusters K has to be supplied for RoCORE. When K is unknown, the improvement will be even enlarged.
Regarding labeling costs, both settings are comparable. Note that in the main results, only two instances for each relation are labeled on average. For the two-stage methods, in order to label the relational semantics of a cluster, the annotator has to observe at least one sample. Obviously, the ASCORE does not lead to a significant increase in human efforts.

heading: 6 Analysis and Discussions

heading: 6.1 Analysis on Labeling Strategy
text: Main results (sec. 5) have shown the advantages of the proposed actively supervised clustering setting over the two-stage setting. However, some serious readers may still think the comparison across settings is unfair. To further reduce readers’ concerns and show the effectiveness of our labeling strategy, we combine the actively supervised clustering settings with the various active labeling strategies and compare them in terms of relation discovery and clustering performance. Note that the number of key points selected by each strategy is the same (two points per relation on average). The results are shown in tab.2 and tab. 3. It can be seen from tab. 2 that the proposed labeling strategy finds the most relations. Different from the classical strategies that focus only on improving the recognition of relations that have been discovered, the proposed strategy appropriately explores new relations by distance regularization,
which is particularly beneficial for long-tail relation discovery in real applications. Additionally, tab. 3 shows that this strategy is also the best in terms of clustering performance. Benefitting from assigning reasonable loss functions to pseudo labels with different reliability, more pseudo labels can be used for learning without significantly increasing the risk of over-fitting noise.

heading: 6.2 The Effect of Numbers of Actively
text: Labeled Instances
In order to more comprehensively evaluate the labeling strategies, experiments are conducted to compare these labeling strategies by varying the number of actively labeled instances, N∗. Figure 3 shows the effect of N∗. Surprisingly, it is found that the random strategy is a very competitive baseline that beats most of the classical labeling strategies given different N∗. This suggests that classical labeling strategies may be better at tasks with known and fixed categories. Although the proposed strategy consistently outperforms all baselines, it is obvious that it has not been fully optimized. It is the authors’ belief that it is sufficient to serve as a reasonable baseline for the actively supervised clustering setting and proceeding some useful guidance for future research in this field. Additionally, with the increase of N∗, the performance of the model is improved, but the growth rate is gradually slow. It means that the cost performance of human effort gradually decreases. Therefore, for users with limited budgets, it is also a good choice to discover the primary relations through only a few queries.

heading: 6.3 Hyperparameter Analysis
text: In this section, we study the effects of reliability threshold rh and rm on optimization. Their values are given by the values of the elements ranked θce% and θbce% from small to large. From Figure 4 it is possible see that: (1) When θce and θbce gradually increase from a small value, more training data are used for model optimization, and the performance of the model gradually improves. (2) When the value exceeds a certain threshold, further increasing the θce and θbce will introduce more errors into the optimization, which degrades the performance of the model. (3) Compared with Lbce, Lce loss will make the model converge faster, so its optimal threshold of Lce should be less than that of Lbce to prevent the overfitting of wrong labels.

heading: 7 Conclusions
text: In this work, we present a new setting, named actively supervised clustering for OpenRE, which provides the necessary guidance for clustering without a significant increase in human efforts. Along with this setting, a labeling strategy tailored for clustering is proposed, maximizing the clustering performance while discovering as many relations as possible. Different loss functions are assigned to pseudo labels with different reliability, which mitigate the risk of over-fitting to noise in pseudo labels. Experimental results show that this method significantly outperforms the existing two-stage methods for OpenRE.
Limitations
Considering that the golden labels of all instances have been given in the datasets, we directly use these labels as manual labels without performing the manual labeling process. The practice implicitly assumes that all the manual labels are correct. However, with the increase of labeling scale, problems such as (1) inconsistent labeling granularity across annotators and (2) noise in manual labels gradually emerge. How to effectively improve the labeling quality and the robustness of the clustering model are worthy of attention.

heading: Acknowledgements
text: The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No.62076069,62206057,61976056), Shanghai Rising-Star Program (23QA1400200), and Natural Science Foundation of Shanghai (23ZR1403500).

heading: A Appendix
text: A.1 Compared Active Labeling Strategy To show the superiority of the proposed labeling strategy, the actively supervised clustering is combined with the following classical active learning strategies for comparison. RANDOM The naive baseline of randomly selecting k samples to query labels. CONFIDENCE (Wang and Shang, 2014b) An uncertainty-based active learning algorithm that select k samples with smallest predicted class probability max{fθ(x)i}i=1,...,C MARGIN (Roth and Small, 2006) An uncertaintybased active learning algorithm that selects the bottom k sorted according to the example’s multiclass margin, defined as fθ(x)ŷ − fθ(x)y′ , where ŷ and y′ are the indices of the largest and second largest entries of fθ(x). ENTROPY(Wang and Shang, 2014a) An uncertainty-based active learning algorithm that selects the top k samples according to the entropy of the sample’s class distribution. GRADIENT(Ash et al., 2019b) A loss based active learning algorithm. The uncertainty is measured as the gradient magnitude with respect to parameters in the output layer.
A.2 Additional Results In this section, more detailed experimental settings and experimental results are given. Specifically, in TACRED dataset, 21 relations are held out as open relations to be discovered. In the underestimation (resp. overestimation) setting, we assume that the number of clusters is 10 (resp. 40). In FewRel and FewRel-LT datasets, 40 relations are held out as open relations and we assume that the number of clusters is 20 (resp. 80) for underestimation (resp. overestimation) setting. The results of the two settings are listed in tab. 4. When K is underestimated, it is observed that the precision of B3 is far lower than recall, which indicates that the model tends to mix multiple relations in the same cluster. When K is overestimated, the recall is far lower than precision, which indicates that the same relation tends to be clustered into multiple subclusters. Although the F1 of B3 metric seems to be tolerable, such imbalance clustering assignments cause great difficulties in relation labeling. If a cluster contains more than one relation, labeling the cluster as any relation will lead to the misidentification of other relations. If
a relation is clustered into multiple sub-clusters, the annotators have to label the same relation repeatedly, which leads to a significant increase in labeling costs.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
we discuss the limitations in the Limitations section.
3 A2. Did you discuss any potential risks of your work? we discuss the risks in the Limitations section.
3 A3. Do the abstract and introduction summarize the paper’s main claims? In the abstract and introduction section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4
3 B1. Did you cite the creators of artifacts you used? section 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? section 4
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 4,5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: It is the authors’ belief that it is sufficient to serve as a reasonable baseline for the actively supervised clustering setting and proceeding some useful guidance for future research in this field. Additionally, with the increase of N∗, the performance of the model is improved, but the growth rate is gradually slow. It means that the cost performance of human effort gradually decreases. Therefore, for users with limited budgets, it is also a good choice to discover the primary relations through only a few queries