Open-world Semi-supervised Generalized Relation Discovery Aligned in a Real-world Setting
Authors: William P Hogan
Jiacheng Li
Jingbo Shang
Venue: EMNLP 2023 Main (2023)
Keywords: Information extraction
relation extraction
TL;DR: We align the task of OpenRE in a real-world setting and propose a method that effectively classifies explicitly and implicitly expressed relationships from unlabeled data.

Abstract: Open-world Relation Extraction (OpenRE) has recently garnered significant attention. However, existing approaches tend to oversimplify the problem by assuming that all instances of unlabeled data belong to novel classes, thereby limiting the practicality of these methods. We argue that the OpenRE setting should be more aligned with the characteristics of real-world data. Specifically, we propose two key improvements: (a) unlabeled data should encompass known and novel classes, including negative instances; and (b) the set of novel classes should represent long-tail relation types. Furthermore, we observe that popular relations can often be implicitly inferred through specific patterns, while long-tail relations tend to be explicitly expressed. Motivated by these insights, we present a method called KNoRD (Known and Novel Relation Discovery), which effectively classifies explicitly and implicitly expressed relations from known and novel classes within unlabeled data. Experimental evaluations on several Open-world RE benchmarks demonstrate that KNoRD consistently outperforms other existing methods, achieving significant performance gains.

Open-world Semi-supervised Generalized Relation Discovery Aligned in a Real-world Setting
William Hogan
Jiacheng Li
Jingho Shang"
Department of Computer Science & Engineering University of California, San Diego
{ (whogan, j91i, jshang } @uccs. edu
Abstract
(1) the presumption that unlabeled data solely con- sists of novel classes or is pre-divided into sets of known and novel instances; (2) the absence neg- ative instances; (3) the random division of known and novel classes in a dataset; and (4) the availabil- ity of the ground-truth number of novel classes in unlabeled data.
Open-world Relation Extraction (OpenRIE) has recently garnered significant attention. How- ever, existing approaches tend to oversimplify the problem by assuming that all instances of unlabeled data belong to novel classes, thereby limiting the practicality of these methods. We argue that the OpenRE: setting should be more aligned with the characteristics of real-world data. Specifically, we propose two key improve- ments: (a) unlabeled data should encompass known and novel classes, including negative in- stances; and (b) the set of novel classes should represent long tail relation types. Furthermore, we observe that popular relations can often be implicitly inferred through specific patterns, while long-tail relations tend to be explicitly expressed. Motivated by these insights, we present method called KNoRD (Known and Novel Relation Discovery). which effectively classifies explicitly and implicitly expressed re- lations from known and novel classes within unlabeled data. Experimental evaluations on several Open-world RE benchmarks demon- strate that KNoRD consistently outperforms existing methods, achieving significant gains.
In this work, we critically examine these assump- tions and align the task of OpenRE within real- world setting. We dispose of simplifying sumptions in favor of new assumptions that align with characteristics of real-world unlabeled data hopes of increasing the practicality of these meth- ods. We call our setting Generalized Relation Dis- covery and make the following claims:
(a) Unlabeled data includes known, novel, and negative instances: Unlabeled data, by defi- nition, lacks labels; cannot assume it only consists of novel classes or is pre-divided into sets of known and novel instances. Our chal- lenge is accurately classify known classes and discover novel classes within unlabeled data. Additionally, many sentences with an entity pair do not express a relationship (e.g.. negative instances, or the no relation class) (Zhang et al., 2017a). Neglecting negative in- stances in training leads to models with posi- tive bias, reducing their effectiveness in identi- fying relationships in real-world data. Hence, we opt to include negative instances in our setting.
1 Introduction
Relation extraction (RE) is a fundamental task in natural language processing (NLP), aiming ex- tract fact triples in the format (head entity, rela- tion, tail entity) from textual data. Open-world RE (OpenRE) is a related research area that fo- cuses on discovering novel relation classes from unlabeled data. Recent advancements in OpenRE have demonstrated impressive results by integrat- ing prompting techniques with advanced clustering methods (Zhao et al., 2021: et al., 2022b; Wang et al., 2022a). However, current OpenRE meth- ods face limitations due to assumptions about unla- beled data that do not align with the characteristics of real-world datasets. These assumptions include:
(b) Novel classes are typically rare and belong to the long-tail distribution: To define known and novel classes, we base our selection pro- the intuition that known classes are more likely to be common, frequently appear- ing relationships. In contrast, unknown, novel classes are more likely to be rare (i.e., long- tail) relationships. Instead of randomly choos- ing the set of novel classes, we construct data
"Corresponding author

---

Prompt model training
Gold Labels
Prompt template: sentence+ + head entity (mask) tall entity predicted relation caass
Labeled data
Iin 3333 fue years aller stane was misttaated, the Jhf weaciied JMT (mask) 1933
Overg (aett of pocheticia rcl cousee L25 symptems eschericiia coll (modi] IS
counts
abostted
known classes only
The prrepaddeetoe took plane in Pallendeh Banneylon Philadepets (mscs) Pesnephonia
Prompt model inference
woight
ertropy Loss
00100000000000000000000000000000
Sample impur: Bill gr graduated with Master's dogree from Aubum University Bill [mask] Aubum University
Language Model [DeBERTa)
Conatrained, precicioos
Uncomstrained, sacatclany predkctions (mmlli talataan attended, represented, member
graduated, degree, Hoster's
Unlabeled data
Ang imprrsectaiion prppp predictions:
Clustering majority bifurcation
known & novel
classes
Fornstrammed
Up.
"anomettsinns
Weak Labe's
Figure I: In KNoRD, we use labeled data to train prompt model to predict relation class names. That model is then used to generate constrained (in-sentence) and unconstrained (all vocabulary) predictions. We average and concatenate representations from the top three constrained and unconstrained predictions. Representations are clustered using Gaussian Mixture Models (GMM) and bifurcated into sets of known and novel instances via a majority-vote. Novel-identified clusters provide weak labels in cross-entropy training objective.
splits based on class frequency. Although it is possible for frequently appearing classes to be unknown, we deliberately select rare classes for our novel classes to create more chal- lenging setting. Lastly, without labels, it is impossible know priori the ground truth number of novel classes contained within un- labeled data; we do not assume we can access this information in our setting.
inputted instance; and, (2) unconstrained, where the model can predict any word within its vocabu- lary. Constrained predictions optimize for explic- itly expressed relationships while unconstrained predictions optimize for implicitly expressed lationships. This prompt method forms the back- bone of our proposed method, Known and Novel Relation Discovery (KNoRD), which can effee- tively classify explicitly and implicitly expressed relationships of known and novel classes from un- labeled data.
Our experimental results show that our proposed setting makes for difficult task, ripe for advance- ments and future work.
Another key aspect of our method is that it clusters labeled and unlabeled data within the same feature- space. Each labeled instance serves as a "vote" for a cluster belonging to the set of known classes. We effectively bifurcate clusters into sets of known and novel classes by employing majority-vote strat- egy. Novel-identified clusters are then utilized as weak labels, in combination with gold labels, to train a model via cross-entropy (see Figure 1). This methodology presents innovative approach to re- lation discovery in open- world scenarios, offering potential applications across various NLP domains.
sate-of-the-an approaches relation discovery leverage a prompt-learning method to predict rela- tion class names which are then embedded into a latent space and grouped via clustering (Zhao et al., 2021; Li et al., 2022b; Wang et al., 2022a). In addi- tion making simplifying assumptions about labeled data, past works use unconstrained predic- tions for relation class names that is, the prompt model can predict any word in vast vocabu- lary as a relation class name. We observe that relationships in text can be expressed either ex- plicitly or implicitly. Explicit instances contain class-indicative words, and implicit relationships are inferred via lexical patterns (see Appendix A.1 for examples). In this work, we illustrate the effectiveness of designing prompt method that optimizes for explicit and implicit relation- ships by predicting relation class names in two settings: (1) constrained to words found within an
The main contributions of this work are:
We critically examine the assumptions made in OpenRE and carefully craft new setting, Gener- alized Relation Discovery, that aligns with char- acteristics of real-world data.
We propose an innovative method to classify known and novel classes from unlabeled data.

---

We illustrate the effectiveness of modeling im- plicit and explicit relations via prompting.
Charadeeristic unlabeled data (Pu):
Noval classes
We openly provide all code, experimental set- tings, and datasets used to substantiate the claims made in this paper.'
Known
Noval Cis
W/Neg
Insunnes
Novel
Link
aute king--o
Setting
Limiring Assumptions
Only predicts classes
seen training
only contains novel classes
ZoroShot RE
2 Related Work
Novel classes require
Gen. ZeroShot RE:
x
definitions
Traditionally, RE methods have focused on a closed-wordd world setting where the extracted relations are predefined during training (Califf and Mooney, 1997; Mintz et al., 2009; Zhang and Wang, 2015; Peng et al., 2017; Qin et al.. 2021). However, datasets used for training rarely complete, and traditional RE cannot capture new relation classes additional data becomes available. Rescarchers have proposed various approaches to discover emer- gent classes to address this issue.
Novel classes require
Continual RE
human annotation
Open RF
x
pre-divided known/bovel instances
Open
/
x
negative instances; rand. novel classes
Gen. Discovery
Table 1: A comparison between our proposed setting, Generalized Relation Discovery, and the existing set- tings. We remove simplifying assumptions about unla- beled data (D.) to align the task to the characteristics of real-world data.
Open-world RE: Open-world RE seeks to dis- cover new relation classes from unlabeled data. In- stances of relations are typically embedded into latent space and then clustered via K- MeMeas. These works often simplify the task by assuming all in- stances of unlabeled data belong to the set of novel classes and that unlabeled data contains no nega- tive instances (Elsahar et al., 2017; Hu et al., 2020; Zhang et al., 2021). More recently, some OpenRE methods have been proposed to predict known and novel classes from unlabeled data (Zhao et al., 2021: Wang al., 2022a; Li et al., 2022b). How- ever, these works assume unlabeled data comes pre-divided into sets of known and novel instances, that negative instances are removed, and that the number novel classes is known.
not transfer to our task where a prevalence of sen- tences do not express a relationship (Zhang et al., 2017b).
The setting proposed in Li al. (2022c) is similar to ours however, negative instances are removed and their known/novel class splits are done ran- domly instead of by class frequency. Furthermore, their method relies on active learning where human annotators annotate instances of novel classes. The annotations are then used to train a classifier. In contrast, our model and the baselines we evaluate do not require human annotation for novel classes.
Table 1 qualitatively compares proposed setting, Generalized Relation Discovery, to related settings, Additional related works are discussed in Appendix A.2.
worndworld semi-supervised learning: The set- ting we propose for relation discovery is inspired by open-wer open-world semi- semi-super learning (Open SSL) proposed by Cao al. (2022) where the authors use margin loss objective with adaptive uncer- tainty to predict known and novel classes from a set unlabeled images. Besides the difference in domains, their setting differs from ours in that they assume unlabeled data has an equal number of known and novel instances-an an assumption we cannot make when working with relation extrac- tion datasets which often have imbalanced, long- tail class distributions (Zhang et al., 2017b: Stoica al., 2021; Yao et al., 2019; Amin et al., 2022). Furthermore, Cao et al. (2022) assumes unlabeled data only contains positive instances, which does
Problem Statement
The task of Generalized Relation Discovery is simultaneously classify instances of known relation classes and discover novel relation classes from unlabeled data, given a labeled dataset with set of known classes
We construct a transductive learning setting where we assume both labeled data D; {r;, Di Ui e CK and unlabeled data D., {2;} are given inputs. All classes in DD are considered known classes CK {ct ef where e is the number of known relation classes. D, consists of instances from CK as well as instances of novel classes C.V = -c.,
'https://Oe http://////gghhbb.bocooo//oohoe/ wwt.w///wwwwww/eeeeeeoo knord
... dh where = |C*| is the

---

number of novel relation classes. The known and novel relation class sets are constructed as non- intersecting sets (i.e., CK nc = 0). Following Cao et al. (2022), we assume there is no distribution shift between the labeled and unlabeled data (i.e., known relation classes found in labeled data also occur in the unlabeled data).
used create semantically-alige representations in the subsequent stage.
4.2 Semantically-aligned Representations
Leveraging our observation that relationships are expressed explicitly or implicitly, we construct two settings for our prompt model: constrained and unconstrained predictions. Constrained predic- tions are [MASK] predictions (yi) constrained to words found within the inputted instance 74, i.e., V/ x; where z; is the input sentence with words {wi,..., W.). In this setting, top tokens in the inputted instance are used to optimize for explic- itly expressed relationships. In the unconstrained setting, we allow the model to use any word in its entire vocabulary (V) to predict the name of the relationship. i.e., y/ V. optimizing for implicitly expressed relationships.
We formulate our task in the following way:
y' U) 16 mz)
E CK C Ty E Du
where 2; are unlabeled instances (e.g, sentences), y denote known and novel predictions, and e denote known and novel classes, respectively, and Y is the set of all predictions for instances in D.. Each sentence Ij contains an entity pair- head entity E and a tail entity 62 and the predicted relationship Yj links the two entities, producing fact triplet (e1, Vj, 02).
We use the hidden representations of the top three tokens in each setting to construct the following representations:
4 Method
KNoRD consists of four discrete stages: (1) prompt-based training, (2) constructing semantically-aligned relation representations, (3) clustering with majority-vote bifurcation, and (4) classification. We describe each stage in detail in the following subsections.
12 u5 Uj E 3z
Ficomstrained
(1)
n j=1
1
n
Taunconstrained
Egg Vj E
(2)
71
4.1 Prompt-based Training
where n and Zy; the 5 embedded represen-
We leverage the instances of labeled data to train a language model to predict the linking relation- ship hetween an entity pair found in a sentence via prompting, Specifically, given a sentence Ij, we construct prompt template T(.) (ei) [MASK] (e2) where é1 and é2 two entities in zz.² The template is appended 2y to obtain contextual- ized relation instance. Then, a masked language model (e.g., BERT (Devlin et al., 2019)) learns to predict the masked tokens between two entities. To alleviate the model overfitting on the relation token vocabulary, we randomly mask 15% of tokens in Ty. and the model is jointly trained to predict the masked tokens in sentences and masked relation names.
tation (Zy, C D) of the prediction corresponding to instance z; from phrase embedding model (Li et al., 2022a). Note, we do not use the prompting model to produce because this model is trained on only known classes and tends to overfit known classes even if random tokens are masked and pre- dicted in x; during training.
Our final relationship representation is constructed by combining the constrained and unconstrained representations:
Ti (Froonttriine: F Fuunconstrained!
(3)
where (,) represents concatenation. The combined representation r; models explicitly and implicitly expressed relationships in sentence 2;-
During inference, feed the contextualized re- lation instance with only masked relations* to the model and predict the masked token. Top-ranked tokens predicted for [MASK] from the model are
4.3 Clustering with Majority-vote Bifurcation Relationship representations from Equation 3 are clustered via Gaussian Mixture Models (GMM). To improve the quality of clusters, we adjust the cluster member according to their entity meta-type
'The number [MASK] tokens is the same as the number tokens in the relation name.
"We use only one [MASK] hetween entities for inference.

---

pairs (e.g. Uhuman, organization), etc.). Specifi- cally, we select the top 30% of relation instances in each cluster and use the major entity meta-type pair of them as the meta-type of the cluster. Then, all relation instances are adjusted to the nearest cluster with the same meta-type.
sentation is:
mel Mcan (ha 3 m
(4)
To form relation representation, we concate- nate the representations of two entities e1 and Tele2 (me1. m²2). The relation represen- tations are sent through a fully-connected linear layer which is trained using cross-entropy loss:
We cluster instances from both labeled and unla- beled data into the same feature-space. Intuitively, since all labeled instances are instances of known classes, each labeled instance acts as a "vote" vot- ing for cluster that corresponds to known class. We tally the votes of all the labeled instances and the results bifurcate the set of clusters into subsets of known-class clusters G and novel class clusters G.V such that GK G.N 0. We call this method "majority-vote bifurcation" and use the novel-identified clusters G as weak labels for the subsequent classification module.
LCB -
Σ Yos log (p (yo,))
(5)
where is a binary indicator that and only is the correct classification for observation o, p(yas) is the Softmax probability that observation is of class i, and N is the number of classes. Predictions from Equation 5 are mapped to ground- truth classes using the Hungarian Algorithm (Kuhn, 1955) (see Appendix A.6.2 for more details).
4.4 Relation Classification
the final stage of KNoRD, we use gold labels from labeled data and weak labels generated from the method described in Section 4.3 to train rela- tion classification model.
Li et al. (2022b) show that setting the number of novel classes to a large number corresponds to fine- grained novel class predictions which, depending on the task and desired outcome, can be grouped into more general classes via abstraction. Since we do not assume the ground-truth truut number of novel classes is available, ber of novel classes equal twice the number of known classes (2 |CK] (), where |C the num- ber of known classes found in the labeled data. The N in Equation 5 is set to |C + (2 x (C We leave developing an automated method for class abstraction for future work.
Since the clusters generate weak labels of varying degrees of accuracy, we select the top P% weak labels for each cluster in G., In our model, we set 15. We retrospectively explore the effects of different P values and report the performance in Appendix A.5. We observe that the optimal value for P varies across datasets. We leave developing an advanced method of determining P for future work.
relatively high num-
For each relationship instance, we follow Soares et al. (2019) and wrap head and tail entities with span-delimiting tokens. We construct entity and relationship representations following Hogan et al. (2022). For sentence F;; we use pre-trained lan- guage model, namely DeBERTa (He et al., 2021),* as an embedding function to obtain feature repre- sentations for relationships.
Datasets
We evaluate KNoRD on three RE datasets: TA- CRED (Zhang et al., 2017b), ReTACRED (Stoica et al., 2021), and FewRel (Han et al., 2018). For each dataset, we first construct splits of known and novel classes based on class frequency, assigning the top 50% most frequent relation classes the set of known classes (C*) and the lower 50% to the set of novel classes (CN) (sce Figure 2). Since FewRel a balanced dataset with relationships de- fined from subset of Wikidata relationships, we obtain real-world class frequencies based on their frequency within Wikidata. For more details our FewRel pre-processing steps, see Appendix A.3.
We encode 2% and obtain the hidden states (h1, be.. }. Then, mean pooling is applied to the consecutive entity tokens to obtain repre- sentations for the head and tail entities (e1 and 62, respectively). Assuming nsttrt and Rend are the start and end indices of entity e1, the entity repre-
'https://hugg" angeacc..om cwwaacccccoomm.oooooo co/m1 ctt.ec/oooo//oott///ooooo deberta-bas.
All instances of novel classes are combined with a random sample of 15% known-class instances

---

# Known Classes
FewRel
40
ReTACRED
19
TACRED
20
Ino binall by cless
Froqueaec-hascl class splits
Laheled
Uniaheled
nfganivv lestinces are labeled
Negative Instanees
# Novel Classes
20
innitumusss knnwnn chases are
# Negative Instances
of
57,777
13% of instances known are wnlabeled
(ibeled] irtaheledd
(49 19.100) 9060)
(71817) 12,624)
instances of unixboled
# Labeled Instances*
23,626
25,478
# Unabeled Instances"
32,374
8,212
6,385
15% of instancce of repuative iestances are ualabeled
(KoowE
25,00)
(4,774 (8,739)
3,520)
Known classes
Novel clasess
Figure 2: Data splits used in our Generalized Relation Discovery setting. Given n total classes in dataset, the set of known classes are the top |n/2] most frequent classes. Remaining classes are placed into the set of novel classes. Labeled data consists of 85% of instances from known classes. Unlabeled data contains 15% of instances of known classes and 100% of the instances from novel classes (*numbers do not include negative instances, "since FewRel has no annotated negative instances, we augment the dataset using negative instances from Re ReTACCEDD),
form the unlabeled dataset Du. The remaining 85% of known-class instances are used to create the labeled dataset Dr.
we employ the Hungarian Algorithm (Kuhn, 1955) match some discovered classes known classes from labeled data, facilitating performance evaluation on the known classes.
We include two versions of each dataset: with and without negative instances (e.g., the no relation class). The setting with negative instances best mir- rors real-world data; however, our experiments show, discovering novel classes in of negative instances is difficult. We include results from both settings and the setting with negative instances will be ripe for advancement and future work.
(ii) RoCORE', MatchPrompt' TABs': Many leading OpenRE models assume unlabeled data comes pre-divided into sets of known and novel instances (Zhao et al., 2021; Li et al., 2022b; Wang et al., 2022a). A natu- ral extension of these methods to prepend a module that segments unlabeled data into known and novel instances. We pre-train mod- cls on known classes and then generate confi- dence scores for each unlabeled instance. We use the softmax function as a proxy for con- fidence (Hendrycks and Gimpel, 2016) and set the confidence threshold equal to the mean confidence from labeled instances of known classes. Instances with confidence scores be- low the threshold are assigned to novel classes. We report the accuracy of this method in bi- furcating unlabeled data in Appendix A.4.4.
Our focus is to evaluate methods on data with no distribution drift (i.c., known classes from training occur in the unlabeled data along with novel classes). We leave an evaluation on out-of- distribution datasets (Gao et al., 2019; Bassignana and Plank, 2022) to future work.
6 Experiments
We compare KNoRD to state-of-the-art OpenRE baselines: (1) RoCORE (Zhao et al., 2021), (2) MatchPrompt (Wang et al., 2022a), 6 (3) TABs (Li et al., 2022b). Since OpenRE methods cannot natu- rally operate within the Generalized Relation Dis-
We include OCRA (Cao et al., 2022), a computer vision model developed for similar gen- eralized open-world setting. OCRA is the only model architecture in our experiments that can pre- dict known and novel classes from unlabeled data and, thus, requires no modification, beyond adapta- tion the RE task. function within our proposed setting. For more details about adapting ORCA predict relationships, see Appendix A.4.5.
ORCA:
covery setting, we extend the OpenRE baselines in the following ways:
(i) RoCORE', MatchPrompt', TABs': Given that OpenRE methods cannot identify previ- ously seen (known) classes mixed with novel classes in unlabeled data, we evaluate their performance novel classes and propose method to adapt them for seen classes. To achieve this, we treat all classes as novel classes, enabling these methods effectively cluster the unlabeled data. Subsequently,
GPT 3.5: Given the zero-shot learning capabilities of Large Language Models (Kojima et al., 2022), we also include GPT 3.5 (OpenAI, 2021) line. To assess GPT 3.5, we leverage in-context learning-we we provide examples of extracted rela- tionships and a list of known relation classes. We "At the time of writing, the authors of MatchPrompt have not relcased the code for their method. The method used in this paper is from our own implementation.
base-
'eee Appendix A.3 for more details.
We use the gpt-3.5-turbo-0300 version of ChatGPT via OpenAl's API.

---

instruct the model to predict the most appropri- ate relation class name or suggest novel class name when an instance does not fit within the set of known classes (see Appendix A.4.1 for details). GPT 3.5 +cos: Since responses from GPT 3.5 may not align perfectly with ground-truth labels, we use DeBERTa map the responses (y:) to ground-truth class names by embedding the pre- dictions and the ground-truth class names (zy, and Let E I Dx(ICK (+(CM) respectively) and identify- ing the ground wround-truth -ruth class that exhibits the highest cosine similarity with the predicted class:
boost, the model still performs poorly relative our other baselines. Given the unsatisfactory re- sults from GPT 3.5 in our simplified experiment setting without negative instances, we decide to exclude from the more challenging setting where negative instances are present. We conclude that more advanced techniques are required to enable GPT 3.5 to accurately classify and discover rela- tionships from textual data. A deeper examination of GPT 3.5's performance provided in Appendix A.4.3.
In the setting with negative instances, all methods struggle identify novel relation classes indicat- ing the difficulty of discovering new classes among instances with no relation. We attribute the lower overall performance using TACRED compared to ReTACRED to TACRED's wrong labeling prob- lem (Stoica et al., 2021).
Zy, Zgt
Dm argmaz
max (1Zycll2 Zyg 21
where y/, is the mapped prediction of prediction Vr. and = le- 8. We denote the GPT 3.5 baseline with mapped predictions as "GPT 3.5 +cos."
For all our baselines, we use identical settings for the number of known and novel classes and, save the same pre-trained model (He
The relatively small drop in performance of all models between FewRel with and without nega- tive instances can be attributed to FewRel lacking annotated negative instances, we artificially aug- ment the data with negatives from ReTACRED, We posit that the models can exploit the slight dif- ference in the distribution of the augmented neg- ative instances, thus reducing the task's s difficulty. These results emphasize the importance of future RE dataset creation efforts in annotating negative instances.
GPT 3.5, al., 2021) as a base model and map predictions to ground-truth classes via the Hungarian Algorithm. We use Micro-Fl scores to assess relation classifi- cation performance and report overall performance as well as performance on known and novel classes to assess a model's ability to identify each class type from unlabeled data.
7 Results
Our proposed method, KNoRD, outperforms the baseline models in all metries (Table 2). We ob- serve that the ORCA haseline demonstrates strong overall performance and the OpenRE methods (RoCORE', MatchPrompt', TABs') yield diverse results, which we attributes to the differences in un- derlying architectures. Models such TABs and MatchPrompt incorporate clustering methods that effectively develop relationship representations in an unsupervised setting. In contrast, RoCORE re- lies more heavily on supervised training to form high-quality relationship representations. This dis- tinction is evident in our confidence-based adapta- tions (RoCOREf, MatchPrompti, TABs(), where pre-dividing the unlabeled data benefits RoCORE significantly while the results for MatchPrompt and TABs are mixed.
7.1 Ablations
We conduct ablation studies to better understand the relative importance of each design choice be- hind KNoRD.
KNoRD w/constrained: We only use con- strained predictions from the prompt model construct relationship representations (Equation 1) and keep all other modules unchanged for this ablation.
KNoRD w/ncconstriined: Similar to the afore- mentioned ablation, but we only leverage uncon- strained predictions to construct relationship rep- resentations (Equation 2) for the GMM module. KNoRD without CE: We remove the cross- entropy (CE) module from KNoRD and allow the GMM to predict relation classes directly. We remap cluster predictions to ground-truth classes using the Hungarian Algorithm.
We observe that GPT 3.5 underperforms in this set- ting. Although mapping responses to ground-truth classes (GPT 3.5 +cos) yields a slight performance
Table shows the performance of all ablation ex-

---

RcTACRED
TACRED
FewRel
Model
FI (all)
Fl (known)
Fl (novel)
Fl (all)
Fl (known)
(novel)
FI (all)
Fl (known)
Fl (novel)
Fully supervised
0,963
0.966
0.938
0.925
0.939
0.912
0.902
0.922
ORCA
0.622
0.870
0.325
0.521
0.719
0,360
0.411
0.398
0.414
wlo neg. instances
ReCORE"
0.117
0.174
0.049
0.101
0,126
0.081
0.069
0.002
0.080
RoCORE'
0.578
0.846
0.257
0,380
0.629
0.177
0.352
0.314
0.358
MatchPrompl
0.558
0.601
0.506
0.627
0.660
0.600
0.397
0.398
0.397
MatchPrompt'
0.682
0.826
0.509
0.585
0.758
0.444
0.573
0.655
0.560
TARK'
0.674
0.816
0.505
0.595
0.724
01.499
0.535
0.215
0.585
TABs!
0.312
0.304
0.320
0.298
0.294
0.302
0.541
0.370
0.568
GPT 3.5
0.279
0.450
0.075
0.277
0.482
0.111
0.098
0.313
0.064
GPT tcos
0.283
0.453
0.079
0.280
0,483
0.114
0.101
0.314
0.066
KNoRD
0.793
0.927
0.632
0.718
0.860
0.643
0.606
0.662
0.597
Fully supervised
0.969
0.974
0.922
0.748
0.737
0.814
0.911
0.904
0.918
ORCA
0.570
0.737
0.203
0.354
0.453
0.082
0.402
0.373
0.406
RoCORE"
0.362
0.244
0.064
0.072
wheg. instances
RoCORE"
0.555
0.758
0.181
0.391
0.619
0.086
0.360
0.322
0.366
MatchPrompl'
0.416
0.429
0.210
0.274
0.253
0.148
0.529
0.287
0.542
MatchPrompt
0.532
0,536
0.372
0.330
0,339
0.173
0.563
0.622
0,552
TABS'
0,550
0.6155
0.329
0.358
0.334
0.233
0.511
0.253
0,551
TABs!
0.186
0.158
0.123
0.143
0,116
0.088
0.532
0.307
0,549
KNoRD
0.677
0.832
0.434
0.536
0.685
0.355
0.574
0.689
0.555
Table 2: Fl-micro scores reported on unlabeled data with and without negative (e.g., no relation) instances. (known) and FI (novel) report performance ground-truth truth known and novel classes, respectively. OpenRE models are extended to operate in the Generalized Relation Discovery selting (see Section 6 for details). All scores average five runs except the GPT 3.5 scores which are resultant from single run.
ReTACRED
TACRED
FewRel
Method
Fl (all)
Fl (knowa)
Fl (novel)
Fi (all)
Fl (known)
Fl (novel)
Fl (all)
Fl (known)
Fl (novel)
KNoRD w/construined
0.759
0.886
0.606
0.653
0.792
0.539
0.491
0.650
0.466
KNoRD w/unconstrained
0,776
0.918
0.607
0.633
0.815
0.485
0,575
0.648
0.564
wio
KNoRD w/o CE
0,413
0.354
0.487
0.489
0.402
0.559
0,509
0,371
0.529
KNoRD
0,793
0.927
0.632
0.718
0.860
0.603
0.606
0.662
0.597
KNoRD wiconstruined
0,568
0.718
0.305
0.498
0.699
0.246
0,544
0.665
0.525
wineg.
KNoRD w/unconstrained
0.507
0.700
0.119
0.490
0.686
0.241
0.507
0.700
0.1
KNoRD w/o CE
0.282
0.300
0.258
0.307
0.313
0.301
0.507
0.246
0.554
KNoRD
0.677
0.832
0.434
0.536
0.685
0.355
0.574
0.689
0.555
Table 3: Ablation experiments varying relationship representation methods used in KNoRD, well as removing the cross-entropy module and using cluster predictions directly ("w/o CE"))
periments. Without CE, KNoRD performs poorly, emphasizing the need for selecting high-quality weak labels to train CE module. Constrained predictions from the prompt model outperform un- constrained predictions in 4 out of experiments in predicting novel classes, indicating their suitability for rare, long-tail relations. Combining constrained and unconstrained predictions in KNoRD yields the best overall results, demonstrating the effective- ness of optimizing the prompt method capture explicit and implicit relationships.
with ground-truth class names (Figure 3). Con- strained predictions, designed model explicit relationships, are generally more accurate for long- tail relation classes. Conversely, unconstrained pre- dictions perform better on common relationships.
This observed phenomenon roughly aligns with Zipf's Law (Zipf, 1936), indicating that rare con- cepts and relations more likely to appear explic- itly in long-form manner. In contrast, common relations tend to be expressed in a compressed form (e.g., implicitly). This insight lends additional evi- dence to designing a prompt method that captures explicit and implicit relationships.
We also manually evaluate the accuracy of our prompt method in predicting relation class names. We evaluate the alignment of the top one and top three constrained and unconstrained predictions

---

Constrained Predictions
Top Unconstrained Predictions
Top
4. The low Fl scores of our model and all leading OpenRE models within our experiments with negative instances highlight an area for growth in future works.
ReTACRED
Re 13013
FewRel
10
Ethical Concerns
We do not anticipate any major ethical concerns; relation discovery is a fundamental problem in nat- ural language processing. A minor consideration is the potential for introducing certain hidden biases into our results (i.e., performance regressions for some subset of the data despite overall performance gains). However, issues in our experiments, and indeed these con- siderations seem low-risk for the specific datasets studied here because they are all published.
Accuracy
Figure 3: Accuracy of top I and top 3 predictions from our prompt method in two settings: constrained (in- sentence words) and unconstrained dictions. Unconstrained predictions perform well with common relationships, while constrained predictions perform well on long-tail relationships.
vocabulary) pre-
did not observe any such
8 Conclusion
In this work, we address the limitations of existing approaches in OpenRE and introduce the General- ized Relation Discovery setting align the task to characteristics of data found in the real-world. By expanding the scope of unlabeled data include known and novel classes, as well as negative in- stances, and incorporating long-tail relation types in the set of novel classes, we aim to enhance the practicality of OpenRE methods.
A Acknowledgements
This work is sponsored in part by NSF CA- REER Award 2239440, NSF Proto-OKN Award 2333790, NIH Bridge2AT Center Program under award IU54HG012510-01, Cisco-UCSD Spon- sored Research Project, as well generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions recommendations pressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, the U.S. Gov- ernment. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annota- tion hereon.
Furthermore, we propose KNoRD, a novel method that effectively classifies explicitly and implicitly expressed relations from known and novel classes within unlabeled data. Through comprehensive experimental cvaluations on various Open-world RE benchmarks, we demonstrate that KNoRD con- sistently outperforms existing methods, yielding significant performance gains. These results high- light the efficacy and potential of our proposed approach in advancing the field of OpenRE and its applicability to wolld scenarios.
References
Takuya Akiha, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Min- ing. KDD '19, page 2623-2631 New York, NY, USA. Association for Computing Machinery.
9 Limitations
The limitations of our method are as follows:
1. Our method requires human-annotated data, which is expensive and time-consuming to cre- ate.
Saadullah Amin, Pasquale Minervini, David Chang, Pontus Stenetorp. and Günter Neumann. 2022. Med- Distant19: Towards an accurate benchmark for broad- coverage biomedical relation extraction. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2259-2277, Gyeongju, Republic of Korea. International Committee on Computational Lin- guistics.
2. Our method cannot automatically determine the ground truth number of novel classes in unla- beled data. We leave this to future work.
3. Our method focuses on sentence-level rela- tion classification, and without further testing, we cannot claim these methods work well for leeument-level relation classification.
Elisa Bassignana and Barbara Plank. 2022. Crossre: A cross-domain dataset for relation extraction.

---

Mary Elaine Califf and Raymond J. Mooney, 1997. Re- lational learning of pattern-match match rules for information extraction. In CoNLJ
Gyeongju, Republic of Korea, International Committee on Computational Linguistics.
Xuming Hu, Lijie Wen, Yusong Xu, Chenwei Zhang, and Philip Yu. 2020. SelfORE: Self-supervine rela- tional feature learning for open relation extraction. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 3673-3682, Online. Association for Computational Lin- guisties.
Kaidi Cao, Maria Brbić, and Jure Leskovec. 2022. Open-world semi-supervised learning. In International Conference on Learning Representations (ICLR).
Yhih Chen and Cheng-Te Te Li. 2021. 7s-bert: To- wards zero-shot relation extraction with attribute rep- resentation learning. In Proceedings of 2021 Annual Conference of the North American Chapter of the Asso- ciation for Computational Linguistics (NAACI-2021).
He Huang. Chang- Wang, Philip S. Yu, and Chang- Dong Wang. 2018. Generative dual adversarial network for generalized zero-shot learning. 2019 IEFF/CVF Conference on Computer Vision and Puttern Recogni- tion (CVPR), pages 801-810.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
Zhao Jun, Hu Yuan, Xu Nuo, Gui Tao, Zhang Qi, Chen Yunwen, and Gao Xiang. 2022. An exploration of prompt-hased zero-shot relation extraction method. Proceedings of the 21 Chinese National Conference on Computational Linguistics, pages 786-797, Nan- chang, China. Chinese Information Processing Society of China.
Hady Elsahar, Elena Demidova, Simon Gottschalk, Christophe Gravier, and Frederique Laforest. 2017. Un- supervised open relation extraction. In The Semantic Web: ESWC 2017 Satellite Events. pages 12-16, Cham. Springer International Publishing.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu. Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: To- wards more challenging few-shot relation classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLYP pages 6251-66 2251-625,, Hong Kong, China. Association for Computational Linguis- tics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
Harold W. Kuhn. 1955. The hungarian method for the assignment problem. Naval Research Logistics (NRL.) 52.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao. Zhiyuan Liu, and Maosong Sun. 2018 FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-ar evaluation. Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing, pages 4803-4809, Brussels, Belgium. Association for Computational Linguistics.
Omer Levy, Minjoon Sco, Eunsol Choi, and Luke Zettle- moyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the st Conference on Computational Natural Language Learning (CoNLL 2017). pages 333-342, Vancouver, Canada. Association for Computational Linguistics.
Jiacheng Li. Jingho Shang, and Julian McAuley. 2022a. UCTopic: Unsupervised contrastive learning for phrase representations and topic mining. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 6159-6169, Dublin, Ireland. Association for Computa- tional Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recogni- tion.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Wei Chen. 2021. Deberta: Decoding-enhanced bert with dis- entangled attention. 2027 International Conference on Learning Representations.
Sha Li, Heng Ji, and Jiawei Han. 2022b. Open re- lation and event type discovery with type abstraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6864- 6877, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Dan Hendrycks and Kevin Gimpel. 2016. A baseline for examples
detecting misclassified and in neural networks. ArXiv, abs/1610.001136.
William Hogan, Jiacheng Li, and Jingbo Shang. 2022. Fine-gran tine-grrined contrastive learning for relation extraction. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1083- 1095, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Y. Li, Yinghui Li, Xi Chen, Haitao Zheng, Ying Shen, and Hong-Gee Kim. 2022c. Active relation discovery: Towards general and label-aware open relation extrac- tion, ArXiv, abs/221 1 1.04215.
Chengwei Hu, Deqing Yang, Haoliang Jin, Zhen Chen, and Yanghua Xiao. 2022. Improving continual rela- tion extraction through prototypical contrastive learn- ing. Proceedings of the 29th International Confer- ence on Computational Linguistics, pages 1885-1895,
Colin Lockard, Prashant Shiralkar, Xin Dong, and Han- naneh Hajishirzi. 2020. Zeroshotceres: Zero-shot re- lation extraction from semi-structured webpages. In Proceedings of the 27th International Conference on Computational Linguistics, pages 8105-8117.

---

Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Al- bert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023. Larger language models do in-context learning differently. ArXiv, abs/2303.03846
Mike D. Mintz, Steven Bills, Rion Snow, and Dan Ju- rafsky. 2009, Distant supervision for relation extraction without labeled data. In ACL.
Yuan Yao. Deming Ye. Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document- level relation extraction dataset. ArXiv, abs/1906.06177.
Abiola Obamuyide and Andreas Vlachos. 2018. Zero- shot relation classification as textual entailment. In Pro- ceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 72-78, Brussels, Belgium. Association for Computational Linguisties.
Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. ArXiv, abs/1 508.01006.
OpenAL. 2021. Chatgpt: A large language model trained on the gpt-3.5 architecture. https:// openai , /oom/blg////ooooooooocc/ooooeoeo chatgpt.
Kai Zhang, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2021. Open hierarchical relation extraction. In North American Chapter of the Association for Computational Linguis- tics.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learn- ing from context or names? an empirical study on neural relation extraction. In Conference on Empirical Meth- ods in Natural Language Processing.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017a Position-aware attention and supervised data improve slot filling. In Conference Empirical Methods in Natural Language Processing.
Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen tau Yih. 2017. Cross-sentence n-ary relation extraction with graph Istms. Transac- tions of the Association for Computational Linguistics, 5:101-115.
Yuhao Zhang, Victor Zhong. Danqi Chen, Gabor Angeli, and Christopher D. Manning. 201 7b. Position-aware attention and supervised data improve slot filling. Proceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP 2017). pages 35-45.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie 7hou. 2021. Erica: Improving entity and relation under- standing for pre-trained language models via contrastive learning. In ACL.
Jun Zhao, Tao Gui, Qi Zhang, and Yaqian Zhou. 2021. relation-oriented clustering method for open relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9707-9718, 18, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Shafin Rahman, Salman Hameed Khan, and Fatih Mu- rat Porikli. 2017. A unified approach for conventional zero-shot, generalized zero-shot, and few-shot learn- ing. IEEE Transactions Image Processing, 27:5652- 5667.
Kang 7hao, Hua Xu, Jiangong Yang, and Kai Gao. 2022. Consistent representation learning for continual relation extraction. In Findings of the Association for Computa- tional Linguistics: ACI. 2022, pages 3402-3411, Dublin, Ireland. Association for Computational Linguistics.
Livio Baldini Soares. Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. ArXiv, abs/ 156.0996.3158.
George Stoica, Emmanouil Antonios Platanios, and Barnab'a as P' Pozoos. 2021. Re-tacred: Addressing short- comings of the tacred dataset. In AAAI Conference on Artificial Intelligence.
George Kingsley Zipf. 1936. The Psychobiology of Language. London: Routledge.
Jiaxin Wang, Lingling Zhang, Jun Liu, Xi Liang, Yujie Zhong, and Yaqiang Wu. 2022a. MatchPrompt: Prompt- based open relation extraction with semantic consis- tency guided clustering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7875-7888, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Mengru Wang, Jianming Zheng, and Honghui Chen. 2022b. Taxonomy- aware prototypical network for few- shot relation extraction. Mathematics.
Xinyi Wang, Wanrong Zhu, and William Yang Wang. 2023. Large language models are implicitly topic mod- els: Explaining and finding good demonstrations for in-context learning.

---

A Appendix
Wang et al., 2022a; Li et al., 2022b). Prompt-based methods for relation extraction involve construct- ing prompts, sometimes called "templates." " that provide contextual cues for identifying relations between entities in text. These prompts typically comprise natural language phrases that capture the semantic relationship between entities. Concur- rent OpenRE works Li et al. (2022b) and Wang et al. (2022a) introduce prompt-based framework for unlabeled clustering. Prompt-basee methods are used to generate relationship representations which are then clustered in a high-dimensional space. The clusters are iteratively refined using the training signal from labeled data, with care- ful measures to ensure the model is not biased to known classes. However, the aforementioned meth- ods assume that unlabeled data is already divided into sets of known and novel classes, which is an unrealistic assumption of real-world unlabeled data, Furthermore, these works only report performance on novel classes, obscuring the model's overall performance in a real-world setting where the unla- beled data contains known and novel classes.
A.1 Examples of Explicitly vs. Implicitly Expressed Relationships
Implicit relationships do not contain class- indicative words; they are inferred through specific lexical patterns within sentence. Explicit relation- ships are relationships that are explicitly expressed and contain -lddciiniee words. Figure pro- vides examples of each expression type.
Sentence ( tail)
Relation litee
Expression Type
Chomsky pained Professor... ..
explicit
Professor Noam Chomskyol of MIT's linguistics program...
implicit
William attended Auburn University.
attended
explicit
William (MS, Aibum University) joined team. Philadelphia is located Pennsylvania
attended
implicit
locoted in locoted in
explicit
The event took place in Philadelphis Pennsylvanii,
implicit
Figure 4: Implicitly expressed relationships are con- veyed through lexical patterns, where specific linguistic patterns indicate a relationship between entities. Ex- plicitly expressed relationships are represented through class-indicative words.
A.2 Additional Related Work
A.3 Pre-processing and Augmenting FewRel
Continual rela- tion extraction (CRE) is a relatively new task that focuses on continuously extracting relations, in- cluding novel relations, as new data arrives. CRE's main challenge is preventing the catastrophic for- getting of known classes (Hu et al., 2022; Zhao et al., 2022). In CRE, new data can contain known and novel classes, similar to our setting; however, CRE assumes that all new data is labeled, which fundamentally differs from our unlabeled setting.
Continual Relation Extraction:
Special treatment is needed for FewRel dataset since it is a uniform dataset without entity type information or annotated negative instances.
Frequency-based class splits: To conduct the frequency- splits described Section 5, we obtain the distribution of relation classes as they appear in real-world data. Given that the relation- ship IDs in FewRel correspond with relationships in Wikipedia, we obtain class frequency informa- tion directly from Wikipedia by aggregating counts of occurrences of each relationship.
Zero-shot Relation Extraction: Zero-shot rela- tion extraction methods typically assume that test data only contains novel classes and that descrip- tions for those novel classes are readily available (Levy et al., 2017: Obamuyide and Vlachos, 2018; Lockard et al., 2020; Chen and Li, 2021). General- ized zhro relation extraction (ZSRE) removes the assumption that test data can only contain novel classes. However, ZSRE methods still heavily rely on descriptions of novel relation classes (Huang et al., 2018 8; Rahman et al., 2017), which is infor- mation that do not assume is available in our unlabeled setting.
Augmenting with negative instances: Unfortu- nately, FewRel does not provide annotated nega- tive instances (e.g., the no relation class). To better simulate real-world data, augment the FewRel dataset with negative instances from ReTACRED. We recognize that augmenting FewRel with data from another dataset is not ideal since distribution differences may exist. Future work in the Gen- eralized Relation Discovery setting may focus on extending FewRel with domain-aligned human an- notated negative instances.
Prompt-based RE Methods: Prompt-based meth- ods have shown promising results for both closed- world and world RE tasks (Jun et al., 2022;
Resolving entity type information: The role of en- tity type information in relation extraction has been widely acknowledged (Peng et al., 2020; Wang

---

et al., 2022b). However, the FewRel dataset lacks explicit entity type information. To address this limitation and resolve entity types for all entities in the FewRel dataset, we employ the following two-phase approach:
22300
17500
13000
Number of instaness
1. Wikidata ontology traversal: FewRel provides a Wikidata entity ID for each entity. Leveraging the Wikidata API, we retrieve the metadata asso- ciated with each entity ID. Then, we recursively map entity types (e.g., the value of the property "subclass of" for each concept in Wikidata) to parent types until a root node is found. Dur- ing this traversal, we encounter a few special cases: (1) concepts with missing values for the "subclass of" property; (2) concepts with mul- tiple values for the "subclass of"" property; and (3) values of "subclass of"" that lead to looping paths in the ontology. For entities with missing values, typically found in the leaf nodes of the Wikidata knowledge graph, we default to the value of the Wikidata property "instance of"" as the starting concept for our recursive transversal. When a concept has multiple values for "sub- class of," we select the first value unless that value leads to looping path within the ontol- ogy (c.g., "makc-up artist" is subclass of "hair and make-up artist," which is subclass of "make-up artist"). In these cases, we choose the next value of the "subclass of" until we find non-looping path to a root node.
4400
cali pheppe
Madi rappe
Aa
Unioa code
Resolved coarse-grained FawFiel antily types
Figure 5: The distribution of the 23 entity types resulting from our recursive resolution of FewRel entity types using the Wikidata ontology.
and make predictions. The following is the prompt used for soliciting predictions for our tests:
1. Select the correct relation between the head and tail entities in the following unlabeled examples.
2. Each example has the head and tail entities up- pended to the sentence in the form: (head entity) (tail entity).
3. There are 40 known relation classes, and up to 80 unknown, or novel, relation classes.
4. The following is the list of known relation classes:
"instance of", "subject", "language" "country", "located in", "occupation", "constel- lation", " "citizenship", "part of", "taxon rank", "location". "heritage", "has part", "sport", "genre", "child", "country of origin", "position", "follows". "followed by", "contains", "father", "jurisdiction", "field of work", "participant" "spouse", "mother", "participant". "operator", "performer", "member of party", "publisher", "owned by", "member org", "religion", "head- quarters", "sibling", "position played", "work location", "original language"
2. Type binning: Using the raw values of sub- classes results in thousands of fine-grained en- tity types. We iteratively bin entity types into parent entity types until each entity type has at least 1,000 entities to obtain broader, more generalized entity types. This method produced 23 distinct entity types (see Figure 5 for the names and distribution of entity types found in FewRel).
A.4 Baselines
In this section, we provide additional details about our baseline models.
5. If the instance is a novel class, suggest the most likely novel class name.
6. Here are some examples:
A.4.1 Soliciting Predictions from GPT 3.5:
In 1966 the USSR accomplished the first soft landings and took the first pictures from the lunar surface during the Luna 9 and Luna 13 missions... (Luna 13) (USSR) => "operator"
GPT 3.5 often performs better tasks with the help of in-context learning (Wei et al., 2023; Wang al., 2023). We construct a prompt that lists all known relation classes and offers a couple exam- ples of extracted relationships. We use natural lan- guage class names to help the model understand
Her attempts to publish the work were un- successful until she acquired the patronage

---

of Sophia Mathilde, wife of King William III of the Netherlands. (Sophia Mathilde) (King William III of the Netherlands) => "spouse"
Ground truth class
Respouse from GPT 3,5
Type
org-altemaze sames
org stgalleette_gggg pames
TF
ongecity headqquurterss
TP
orgoounrry. of headquariors orgedissolved
ogg orgennirry... -Chheaquanenn orgdissellvdd
TP
TP
orgefouneed
faunded
7. Respond only with the class name in quotes: KGOR is licensed to Omaha, Nebraska United States, and serves the Omaha metropolitan area. (KGOR) (Omaha, Nebraska) =>
orgfooudedd by
orgfounded by vegmember,.u of
TP
organenber,.e
TP
ongonembers
org.masber_.o_ o._employyeshhnnteen
org rigammeber_oC. M.K_employee//mmhar purants
TP
orgiurents
TP
org-polinicallellgioes adfillationn
org peliticllheligioos atfiliation
org:shurebokdess
org statborpovvitee heaqqurreers
TP
orgestaeexpevvince oe_ueuuurrtess org:suboldiates
PN
An identical prompt was also used for TACRED and ReTACRED, with changes only made to the numbers of known and novel classes, the list of known classes, and the examples provided.
orgssubiidzzies
TP
orgiiop membem//epllyyes orgowebsite
orgtop members/cmployes org.websie
TP
TP
perage
parage
TP
peraltemate nomes
FN TP TP
per pe ceuse_of__eattt perchurges
gercanne do
perchanges
percchilren
penchildren
TP
A.4.2 Probing GPT 3.5 for Prior Knowledge
per citie re_rddennce
geneilies_ resideene
TP
pericity of birth
perceity of birth
TP
One issue in evaluating GPT 3.5 is that the exact body of data used for training is unknown. There- fore, to ensure a fair comparison, we seek to de- termine if GPT 3.5 prior knowledge of the various datasets we in our experiments. To do this, we ask GPT 3.5 to list all the classes in specific dataset with the following prompt: "What are the relation classes found in the [DATASET SATTASE_NNAEE, relation extraction dataset?" report the response when asking about TACRED in Table 4. Note: GPT 3.5 also responded with accurate descriptions of the relation classes, but they are omitted for brevity.
periiity of douth
pencity of death
TP
personmtries residence
percountrics.
perccomnyy of barth persoumtty death of persinee of dlzuth
percountryy hirh
TP
persounryy death perduae
TP
pendute of death peremployer of
TP
persmmplege of perooiggin
TP
perocher, Family perparents
FN
perepeenns
TP
perseligion
perseligion
TP
perschools, amended perssilingg
pertschools, attended perssiblings
TP
TP
perspouse
perrapunse
TP
persatterpmvinee of birt per perppapppssss deshh persssteopmmiinnes of residencr site
perstattorppoinnee of birth
TP
perstolturgeonnne death
TP
perstatkorprovnnee of rosience
TP
pertifle
TP
pemoountrio, citienship
FP
pertocations of residesce
FP
pertocations of residence
FP
For the TACRED dataset, GPT 3.5 responded with 37 correct responses of the 41 total relation classes. We use these results to argue that GPT 3.5 has an unfair advantage in discovering novel classes in TACRED and ReTACRED. Despite this advantage, 3.5 did poorly compared the other baselines we tested. However, when asked about the relation classes in FewRel, it responded with only four cor- rect responses of the 80 total relation classes in the dataset. This information can partially explain why, in our tests, GPT 3.5 performs better on the TACRED and ReTACRED datasets compared to FewRel.
Table Comparing responses from GPT to the ground-truth classes in the TACRED dataset. GPT 3.5. correctly predicts 37 of the 41 relation classes in TA- CRED, receiving an score of 0.91 for class name prediction.
prompting techniques, such as Chain Mfain--f-Thoueeee (CoT) or Self-Consistency Prompting, in future works.
To gain more comprehensive understanding of the reasons behind GPT 3.5's suboptimal performance, we conducted informal error analysis. Our inves- tigation involved randomly selecting 40 instances of erroneous predictions across both known and novel classes generated by GPT 3.5. We present our observations below:
A.4.3 A Deeper Examination of GPT 3.5's Performance
The performance of GPT 3.5 has yielded results below our initial expectations. We carefully con- structed our prompts in accordance with best prac- tices drawn from recent studies that have show- cased the efficacy of in-context learning with gen- erative models (Wei et al., 2023; Wang et al., 2023), Nevertheless, is plausible that more effective prompting methods for open relation extraction ex- ist. particular, we propose exploring alternative
Errors within Known Classes: GPT 3.5's inaccuracies within known classes appear to stem from the difficulty in distinguish- ing between classes with subtle differences. For instance, in the context of ReTACRED, the model frequently confuses the following known classes: riggtoop.mebbers/mpplleess

---

"org:members." and "org:member of." Sim- ilarly, the model exhibits confusion be- tween the "org:stateorprovince "attncc__oobaahcc"" classes. We speculate that GPT 3.5 may require a larger volume of in-context examples to discern the nuances that set these classes apart.
ReTACRED
TACRED
FewRel
whng w/o nog
RoCORE++
0.722
0.698
0.528
MutchPrompt++
0.794
0.712
0.817
and
TAB+++
0.693
0.704
0.816
RoCORE++
0.571
0.551
0.518
MutchPrompt++ 0.690
0.576
0.860
TABs++
0.474
0.522
0.737
Table 5: Accuracy when using the mean confidence score of labeled data to determine which instances are and not novel from unlabeled data.
Errors in predicting novel classes exhibit a common pattern in which GPT 3.5 tends to predict the nearest known and, typically, more general class, rather than suggesting a novel class name. For instance, instances of the novel class "org:sharcholders" are frequently predicted as the broader but related known class "org:member_of"" _f."" Furthermore, the model struggles to propose novel class names that align with novel classes, especially among novel classes that share high degree of similarity. For instance, classes such as death," and
Errors within Novel Classes:
unlabeled data that fall below the threshold are assigned novel classes and vice-versa. We report the accuracy of each pre-trained model in determin- ing whether an instance is known or novel Table 5. Overall, this method performs reasonably well on data without negative instances. However, with negative instances, segmenting known and novel classes becomes difficult.
"pericause of death,"
A.4.5 ORCA Baseline
"persstateorpoovine__off lotpattorrrrovinee____eeeettttee ofer:country_offfcoo deatthh""" pose challenges for the model.
ORCA leverages a pre-trained vision model, namely ResNet (He et al., 2015), to generate repre- sentations for images. To adapt the ORCA model to the text domain, we replace ResNet with DeBERTa (He al., 2021) and generate representations for relationships using the method used in KNoRD and described in Section 4.4. The remaining architec- ture and loss functions are unmodified.
The task of consistently suggesting names of novel relation classes without access a predefined set of class names is intrinsically challenging, even for human annotators. Ideally, having access to the relationship representations produced by GPT 3.5 would allow us to leverage the advanced clustering techniques used in this paper that have proven effec- tive in predicting novel classes. Unfortunately, the unavailability of these representations constrains our ability employ such methods to enhance GPT 3.5's capability predict novel classes.
A.5 High-quality Weak Label Analysis
We retrospectively assess the effeet of using dif- ferent amounts (P%) of high-quality weak labels generated from the GMM to train the cross-entropy module, Quality is assessed using the probability that an instance belongs to a given cluster within the GMM module. Instances within each cluster are sorted by quality, and then the top P% is se- lected as weak labels to train the cross-entropy model. In Table 6, We observe that, in most cases, performance is increased by selecting subset of weak labels based quality: however, the optimal value for P fluctuates between settings. We leave determining the best P value for future work set P 15 for all and the experiments presented this paper.
A.4.4 Confidence-based Baselines
For our confidence-based extensions of existing OpenRE methods, we pre-train each model on the set of known classes. Then, we use a holdout set of known class instances and collect confidence scores (c) for each instance using the softmax function:
exp(F(x, y)) EYER exp (F (x, v/)
where x is an input instance with relation label and F(x, v)) is relation classifier function.
A.6 Implementation Details
The mean of confidence scores from known in- stances is then used as threshold to segment in- stances from unlabeled data into sets of known and novel classes- classes-confidenn scores from predictions
A.6.1 Prompt Model Training
In our prompt model training, we adopt the masked language modeling (MLM) task where a

---

ReTACRED
TACRED
FewRel
% of Weak Labels
FI (all)
FI (known)
FI (novel)
FI (all)
FI (known)
FI (novel)
FI (all)
FI (known)
Fl (novel)
Top 10%
0.770
0.927
0.582
0.667
0,771
0.582
0,483
0.514
0,478
negaatiess
Top 15%
0.793
0.927
0.632
0.718
0,860
0.603
0.606
0.662
0.597
Top 25%
0.758
0,924
0.560
0.712
0.806
0.635
0.537
0.608
0.526
Top 50%
0.769
0,937
0.567
0.718
0.814
0.640
0.575
0.676
0.559
Top 75%
0.763
0.558
0.640
0.507
All (100%)
0.764
0.932
0.562
0.732
0.654
0.528
0.749
0.494
Top 10%
0.658
0,835
0.370
0,535
0.695
0.332
0.598
0.693
0,583
w/eegniies
Top 15%
0.677
0.832
0.434
0.536
0.685
0.355
0.583
0.728
0.559
Top 25%
0,680
0.814
0.478
0,540
0.683
0.364
0.574
0.689
0.555
Top 50%
0.686
0.824
0.482
0.553
0.676
0.405
0.615
0.723
0.598
Top 75%
0.664
0.811
0.456
0.563
0.679
0,429
0.734
0.580
All (1005))
0.673
0.724
Table 6: F1-micro scores reported using varied levels high-quality weak labels with and without negative instances.
Quality is measured using the probabilities assigned by the GMM that each instance belongs to specific cluster.
RoBERTa & is trained on an NVIDIA Quadro RTX
batch size: [64, 128], (step size 64)
8000 GPU.
max gradient norm: [0.8, 1.0), (step size = 0.1) epochs: [4, 121, (step size = 1),
We split labeled data into training and validation datasets by their relationships. Specifically, hold out instances with five random relationships (excluding negative instances) from labeled data for each dataset as the validation dataset. Training with dataset splitting by relation types instead of instances can stop early before the model overfit on the known classes. The metric for validation is perplexity. Other hyper-parameters are as follows:
Below are the final hyper-parameter settings used
for each dataset:
ReTACRED & TACRED: batch size: 128.
epochs: 5, learning rate: 1c-5, dropout: 0.2, max
gradient norm:
FewRel: batch size: 128, epochs: 5, learning rate: le-5, dropout: 0.2
learning rate: 5e-5
A.6.3 Fully-supervised Training
batch size:
We construct our fully-supervised data splits by as- suming all classes are known and using the method described in Section 5, We combine instances from labeled and unlabeled data and randomly selected 15% to form the test split. The remaining instances were used for training, with 20% of the training data further segmented as the validation set.
probability of masking: 15%
A.6.2 Relation Classification
All our models were trained on an NVIDIA GeForce RTX 3090 GPU. We use (41, 42, 43, 44, 45) for our seed values. We limit the length of the input sequence to 100 tokens, use hid- den dimension of 768, and use an A opti- mizer (Loshchilov and Hutter, 2017). We use De- BERTa (He et al., 2021)a as the pre-trained model for all of our experiments.
Note that our results from the fully supervised set- ting cannot be directly compared to numbers ported popular benchmarking websites 10 since our splits do not match the standard. Our splits are designed to maintain consistency with our other ex- periments within the proposed Generalize Relation Discovery setting.
We perform light hyperparameter tuning using the Optuna framework (Akiba al., 2019). We ran- domly sample 20% of the instances of known classes in our labeled datascts for validation and conduct 80 trials of hyperparameter search within the following search space:
learning rate: [1e-4, le-51
dropout: [0.2, 0.4], (step size 0.05)
"nttses//buggiiggacc.co htwp:///a.......eeooeeebaaaaaaaa
/nttps://huggf ingractccc////rrrrettt////////// deberta-base
"httpss/ppaeerssttcoole.cco


Reviews:
[{"review_id": "mapmsb4qel", "reviewer": "Reviewer_h8YH", "summary": "", "questions": "Question A: In line 223, each sentence x_j in the unlabeled data contains an entity pair. Does this mean all the sentences in the unlabeled data have an entity pair? If this is true, then to extend the KNoRD method to new unlabeled data, should one already perform entity recognition and filter out the valid sentences?\n\nQuestion B: As mentioned in footnotes 2 and 3, there is a difference between the number of relation [MASK] tokens used during training and inference time. How does the model perform well during the inference time? Or do the majority of the relation tokens map to a single [MASK] token during training, and the rest have no effect?\n\nQuestion C: How are the top three tokens from the vocabulary chosen for unconstrained relation representation (line 313)?\n\nQuestion D: What was the intuition behind choosing 85% and 15% split for known vs novel relations? Won't it make more sense to follow the actual frequency distribution of the relation classes and decide on the threshold for the split? E.g., The train relation frequency distribution of the three chosen datasets could be helpful.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions and comments. Below are responses to each of your questions. If accepted, we will summarize and add these responses to the manuscript to increase the paper’s clarity. \n\n**Question A**:  Your statement is correct -- all sentences are assumed to have two or more entities that are either pre-identified by the dataset or by an entity resolution module. To extend KNoRD to new, unlabeled data, one would need to first resolve entities and only consider sentences with two or more found entities. In constructing our problem setting, we follow previous works (MatchPrompt [1], RoCORE [2], TABs [3]) by specifically focusing on relation extraction separate from entity recognition; we assume access to resolved entities in both labeled and unlabeled data. We argue that this makes the results more modular in the sense that the reported performance of the relation extraction and relation discovery is not dependent on the performance of the entity resolution module. We could have removed this assumption and approached the problem through a multi-task learning lens but, given the difficulty of our current setting, we believe this modification would be best suited for future works when more progress has been made on the current setting.\n\n**Question B**: For training, the number of tokens that the model predicts corresponds to the number of tokens found in the ground truth relationship name. However, as you mention, during inference time, the model only predicts a single token. This is to account for the fact that, without labels, we do not have access to the ground truth relation name for each instance. We use the top three predicted singular tokens for each instance and average them together to both mitigate the effects of selecting a single token and to create a richer, more robust relationship representation during our clustering phase. \n\n**Question C**: The top three tokens are selected by simply taking the top three most likely tokens as determined by the output probabilities from the prompt model for each inputted instance.\n\n**Question D**: To clarify, the 85/15 split refers to the split within the set of known classes – 85% of instances from known classes are used for training, and 15% of known class instances are used for testing. The ratios for this split follow a previous OpenRE work (TABs [3]). We use a 50/50 split between known and novel classes where the top 50% most frequent classes are assigned to the set of known classes and the remaining classes are assigned to the set of novel classes. Given that our novel classes are exclusively long-tail classes with few instances, we selected a large 50/50 known/novel class split in hopes of creating a representative test set that contained a relatively large amount of novel instances. However, this choice was one we made, and, as you correctly identify, there are other methods we could have used. We could have used the actual frequency distribution of the relation classes in each dataset to determine the threshold. However, it is our understanding that instances of relationships in real-world data very often form a long-tail distribution. We did not include graphs that show this but the three datasets we used all have similar-looking long-tail distributions. We believe that using the class frequency of each dataset to determine the threshold of the splits would have resulted in similar splits.  We can provide these graphs in our updated draft to further justify our decision to do a 50/50 known and novel class split.\n\n\n[1] MatchPrompt: Prompt-based Open Relation Extraction with Semantic Consistency Guided Clustering (2022)\n\n[2] A Relation-Oriented Clustering Method for Open Relation Extraction (2021)\n\n[3] Open Relation and Event Type Discovery with Type Abstraction (2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to understand. The assumptions and claims for building the KNoRD model are clearly stated. It's laudable that the motivation behind creating the KNoRD model is to handle the realistic real-world data setting. The KNoRD model has the potential to generate weak data labels, and it would be interesting to use it for synthetic data creation.", "reasons_to_reject": "The KNoRD method works only on sentence inputs, can't easily be extended to document-level RE, and requires human-annotated labeled data. There are various hyperparameters (top-3 tokens for relation representation, P% for weak labels in clustering, known cluster size, etc.) used throughout the KNoRD modeling, which could make it hard to reproduce the results or even apply to a new unlabeled dataset.", "questions_to_authors": "Question A: In line 223, each sentence x_j in the unlabeled data contains an entity pair. Does this mean all the sentences in the unlabeled data have an entity pair? If this is true, then to extend the KNoRD method to new unlabeled data, should one already perform entity recognition and filter out the valid sentences?\n\nQuestion B: As mentioned in footnotes 2 and 3, there is a difference between the number of relation [MASK] tokens used during training and inference time. How does the model perform well during the inference time? Or do the majority of the relation tokens map to a single [MASK] token during training, and the rest have no effect?\n\nQuestion C: How are the top three tokens from the vocabulary chosen for unconstrained relation representation (line 313)?\n\nQuestion D: What was the intuition behind choosing 85% and 15% split for known vs novel relations? Won't it make more sense to follow the actual frequency distribution of the relation classes and decide on the threshold for the split? E.g., The train relation frequency distribution of the three chosen datasets could be helpful.", "paper_topic_and_main_contributions": "The paper proposes the KNoRD method for discovering implicit and explicit known and novel relations from unlabeled data. In contrast to the general OpenRE setup, authors make no simplifying assumptions on the type of relation classes (known, novel, and negative) in the chosen dataset. The KNoRD method is evaluated on three well-known relation extraction datasets, along with a detailed ablation analysis on each involved component.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 546, figure 3 caption, \"unconstrained\""}, "extra_text": {}}, {"review_id": "3vpUBjBvgJ", "reviewer": "Reviewer_sj25", "summary": "", "questions": "1. Long-tailed relations still have some samples that can be used. In problem statement, why there is totally no samples for novel classes to predict? Could you justify how the samples of the long-tail relations are used? Or are their samples just given up? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your thoughtful feedback and comments. We will fix the minor typos you mentioned in our revised draft and, below, we address your major comments. If accepted, we will include some of the added details below in our updated manuscript. \n\n**No distribution shift**: For the set of known classes, we select the top 50% most frequently appearing classes in the labeled data. While the absence of known classes within unlabeled data is a plausible scenario, we believe it is likely that known classes would manifest in unlabeled data owing to their prevalence. By following a previous work in computer vision (ORCA [1]) and assuming no distribution shift between labeled and unlabeled data, we confine our inquiry to a singular domain of data. Allowing for a distributional shift between labeled and unlabeled data fundamentally transforms the nature of the problem into one of detecting out-of-domain relationships which, interestingly, has been criticized as a limitation of previous OpenRE works since it is unlikely that new, unlabeled data exclusively contains novel classes.\n \nFurthermore, by assuming no distribution shift between labeled and unlabeled data, we construct what we believe is a more difficult and more realistic setting. A significant challenge in our setting is the differentiation between known and novel classes. We report performance across both categories of class types—known and novel—asserting the broader applicability of this approach to real-world scenarios. \n15% ratio: The 85/15 train/test split of instances belonging to known classes was inspired by the previous work (TABs [2]). We believe other ratios, specifically a lower ratio of training data (e.g. 50/50) would increase the problem’s difficulty and would be very interesting to explore in future works. \n\n**Samples for long-tail relations**: If we understand the question correctly, samples from novel, long-tail classes are not used to train the prompt model or the cross-entropy model directly because we have no labels for the novel instances. We do, however, generate pseudo-labels for the novel classes via clustering and “majority vote bifurcation” modules and, subsequently, use those pseudo-labels to train the cross-entropy model, along with gold labels from the labeled dataset. \n\n\n[1] Open-world semi-supervised learning (2022)\n\n[2] Open Relation and Event Type Discovery with Type Abstraction (2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written. The technical quality is good, with a method well designed and well presented. \n\n2. The results are quite promising, with outperformance over the baselines, on three datasets (TA-CREAD, ReTACRED, and FewRel). ", "reasons_to_reject": "1. The assumption of no distribution shift is a bit strong. It's likely that known relation classes found in labeled data do not appear in the unlabeled data. \n\n2. Could you justify the construction of novel classes in 385 - 389? E.g., why the ratio 15% is used? Is there any previous standard setting on the evaluation of this problem?", "questions_to_authors": "1. Long-tailed relations still have some samples that can be used. In problem statement, why there is totally no samples for novel classes to predict? Could you justify how the samples of the long-tail relations are used? Or are their samples just given up? ", "paper_topic_and_main_contributions": "The paper works on open-world relation extraction and mainly consider the set of novel classes including long-tailed relations. \n\nThe authors believe long-tail relations tend to be explicitly expressed, and based on this motivation, they propose method composed of prompt-based training, semantically aligned representation, Clustering with Majority-vote Bifurcation and relation classification, using DeBERTa. \n\nThe proposed method is compared with several state-of-the-art open-world relation extraction baselines. The proposed method outperforms these baselines with big margins. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In 304, \"the same feature-space\" to \"the same feature space\". \n\nIn 256, \"Semantically-aligned\" to \"Semantically aligned\". "}, "extra_text": {}}, {"review_id": "f10ejH5fwz", "reviewer": "Reviewer_ZBDY", "summary": "", "questions": "1. Please add a detailed explanation and comparison of the work of ARD [1] in your final version. Even if the setting you are concerned about are different from it, I think it is necessary to use a certain amount of space to explain the difference between the settings you are concerned about and the setting that ARD is concerned about. Kindly, if you can take this issue seriously enough in your rebuttal and final version, I'm willing to change my rating.\n2. Why is the performance of ChatGPT very poor? This result seems counter-intuitive, I guess it has something to do with the prompt you designed. I think this also needs to be analyzed and explained.\n\n[1]. Active Relation Discovery: Towards General and Label-aware Open Relation Extraction", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The ARD method**: Thank you for the thoughtful comment about the ARD model. Although we mention the ARD work in our Related Work section (lines 193-197) and briefly discuss the differences of the setting used for ARD compared to the setting we propose, we do not provide specifics on why we do not compare our method to the ARD method. Below, we provide some additional details and, if accepted, we will add a summarized version of this explanation to our manuscript.\n\nThe ARD method is a three-part method that (1) develops relation representations, (2) conducts relational outlier detection to detect known and novel classes, and, finally, (3) conducts relational active learning where human annotators annotate instances of novel classes which are then used to train a classifier. Part 3 of the ARD model is fundamentally different from our semi-supervised setting. All of the baselines we compare against, as well as the method we propose, are semi-supervised and do not require human supervision beyond the instances provided in the labeled dataset. By providing additional human annotations of novel instances, we argue that the ARD approach aligns better with continual learning and few-shot learning approaches. The adaptation needed to allow the ARD method to function within our setting requires a significant number of design decisions that we feel we are unable to make. Indeed, such a modified method would likely require a considerable amount of exploration. \n\n**ChatGPT performance**: We agree, the performance from GPT 3.5 is significantly lower than expected. It is possible that a better prompt method exists to increase the performance. We do our best to design our prompt using the best practices from some recent studies that show the effectiveness of in-context learning with generative models [1,2].\n\nWe speculate that the low performance from GPT 3.5 results from the difficulty in determining the granularity of the relation classes of interest. Many of the responses generated from GPT 3.5 were not technically wrong, they were just of a different level of granularity. For example, GPT 3.5 often predicted the relation class “member of” instead of the more specific and correct class “shareholders.” For novel classes, however, we believe the task of predicting the name of a relation class without having a set of class names to choose from is very challenging even for humans. Ideally, we would have access to the relationship representations produced by GPT 3.5, then we would be able to apply advanced clustering techniques that have proved effective in predicting novel classes. Unfortunately, these representations are not available so we are limited in the methods we can use to help GPT 3.5 predict the correct novel classes. \n\nWe believe this area of research is very interesting and deserves a deeper analysis. Alternative prompting methods such as chain-of-thought  (CoT) or self-consistency prompting should be explored in future works. We will include an error analysis of the prompt method we use with GPT 3.5 in the Appendix of our revised draft to hopefully inspire deeper insights and analysis into LLMs applied to the OpenRE task. \n\n\n[1] Larger language models do in-context learning differently (2023)\n\n[2] Large Language Models Are Implicitly Topic Models: Explaining  and Finding Good Demonstrations for In-Context Learning (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The issues and settings that this paper focuses on are important and meaningful.\n2. This paper is well-written.\n3. The experiments are sufficient and combined with large models such as ChatGPT for analysis.", "reasons_to_reject": "1. My main concern with this paper is the Generalized Relation Discovery setting it claims to propose, which is actually not the first time the authors have proposed it, and has been proposed in earlier work such as ARD. Unfortunately, I can not see enough comparison and elaboration of ARD in this paper.", "questions_to_authors": "1. Please add a detailed explanation and comparison of the work of ARD [1] in your final version. Even if the setting you are concerned about are different from it, I think it is necessary to use a certain amount of space to explain the difference between the settings you are concerned about and the setting that ARD is concerned about. Kindly, if you can take this issue seriously enough in your rebuttal and final version, I'm willing to change my rating.\n2. Why is the performance of ChatGPT very poor? This result seems counter-intuitive, I guess it has something to do with the prompt you designed. I think this also needs to be analyzed and explained.\n\n[1]. Active Relation Discovery: Towards General and Label-aware Open Relation Extraction", "paper_topic_and_main_contributions": "This paper studies a more general setting in OpenRE, Real-world OpenRE. This is an interesting and important question worthy of study. The authors observe that popular relations can often be implicitly inferred through specific patterns, while long-tail relations tend to be explicitly expressed. Based on this observation, this paper proposes the KNoRD and conducts extensive experiments to evaluate its effectiveness.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "There are no punctuation marks after each equations, so you need to pay attention.", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on open-world relation extraction problem, and emphasizes novel classes including long-tailed relations. Experiments are generally solid, with large model-based analysis. All 3 reviewers reached a consensus on strong soundness and excitement.
Decision: Accept-Main