Novel Relation Detection: Discovering Unknown Relation Types via Multi-Strategy Self-Supervised Learning
Authors: Qingbin Liu
Yin Kung
Yanchao Hao
Dianbo Sui
Siyuan Cheng
Xi Chen
Ningyu Zhang
Jiaoyan Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Relation Detection
Out-of-Scope Detection
Self-Supervised Learning
TL;DR: Discovering Unknown Relation Types via M3S

Abstract: Conventional approaches to relation extraction can only recognize predefined relation types. In the real world, new or out-of-scope relation types may keep challenging the deployed models. In this paper, we formalize such a challenging problem as Novel Relation Detection (NRD), which aims to discover potential new relation types based on training samples of known relations. To this end, we construct two NRD datasets and exhaustively investigate a variety of out-of-scope detection methods. We further propose an effective NRD method that utilizes multi-strategy self-supervised learning to handle the problem of shallow semantic similarity in the NRD task. Experimental results demonstrate the effectiveness of our method, which significantly outperforms previous state-of-the-art methods on both datasets.

Novel Relation Detection: Discovering Unknown Relation Types via Multi-Strategy Self-Supervised Learning
Qingbin Liu'; Yin ;ung'; Yanchao Hao', Dianbo Siyuan Cheng', Xi Chen';
Ningyu Zhang', Jiaoyan Chen'
Platform and Content Group, Tencent, China
Harbin Institute of Technology, Weihai, China
Zhejiang University & AZFT Joint Lab for Knowledge Engine, Zhejiang, China
Department of Computer Science, The University of Manchester, UK
(qingbinliu, amykung, marshao]@tencemmee com, suidianbo@hit. edw.. cn, sychengēzju. edu. cn, jasonxchenéteencent com, zhangningyu@zjj. edu. cn, jiaoyan. chen@manchester...mmmmm ac uk
Abstract
Assume that "nominored for" is an OOS relation type.
Text: Westworld was nominated for
Conventional approaches to relation extraction can only recognize predefined relation types. In the real world, new or out-of-scope rela- tion types may keep challenging the deployed models. In this paper, we formalize such challenging problem as Novel Relation Detec- tion (NRD), which aims to discover potential new relation types based training samples of known relations. To this end, we construct two NRD datasets and exhaustively investigate variety of out-of-scope detection methods. We further propose an effective NRD method that utilizes multi-stralegy self-supervised learning to handle the problem of shallow semantic sim- ilarity in the NRD task. Experimental results demonstrate the effectiveness of method, which sigmificantly outperforms previous state- of-the-ar art methods on both datasets.
Empire Award
Conventional RE Model
Relation: winner X
Westwoold nominated an Empize Award for Dest TV Series.
Collected OOS 1ott
NRE Model
Developer
Relution: novel relation
Update the Model
Figure example novel relation extraction. For the data of the OOS relation, the conventional RE model predicts a wrong relation. The NRE model is able to correctly predict the novel relation type.
2014), new items and properties keep appearing over time These new relations may mislead the deployed RE model, making it incorrectly assign known relations to the data of new relations, shown in Figure 1. In addition, existing methods cannot automatically discover new relations for fu- ture development.
1 Introduction
Relation extraction (RE) is an important task in structured information extraction, which aims to recognize the relations of entity pairs from texts (Riedel et al., 2013; Zeng et al., 2014; Lin et al., 2016). For example, given the sentence "West- world is a science fiction western series directed by Jonathan Nolan." and the entity pair [Jonathan Nolan, Westworld), RE model should output the relation type "the director of
To handle this problem, we propose istic and challenging task, Novel Relation Extrac- tion (NRE), which aims to discover potential new relation types based on training samples of known relations. Note that we define novel relations as OOS relations that are not included in the prede- fined relation set. In the NRE task, we group the novel relation types into new class and convert the traditional k-class RE task into a (k + 1)-class RE task. The (k- + 1)-th class represents the novel relation. This task requires RE models to correctly identify not only the known relations but also the novel relation. Based on the OOS data collected by the NRE model, developers can easily and pur- posefully annotate the data and update the model, as shown in Figure 1.
real-
Existing RE methods typically follow the closed- world classification assumption and can only rec- ognize predefined relation types. However, such an assumption limits the usage of these methods in real-world applications, new or out-of-scope (OOS) relation types may continually emerge after the model deployed. For example, in the Wiki- data knowledge graph (Vrandecic and Krötzsch,
The novel relation class the NRE task differ-
"Equal Contribution. Corresponning author.
tht wikidata. tratta..../////a org/wiki News

---

ent from the "no relation" class and the "other rela- tion" class in existing RE tasks (Hendrickx al., 2010; Zhang et al., 2017). Previous developers al- ways assume that the "no relation" class and the "other relation" class are known in the development phase, and they can annotate large amount of training data for these two relations to train RE models. However, the novel relation class indicates that the text contains new or OOS relations whose distribution is unknown and unpredictable in the development phase. It infeasible for develop- ers to annotate training data for the novel relation class in real-world upplications since real test data is usually unknown and continually changing.
Assume that "nominated for" is an OOS relation type...
Text 1 (00S); Westworld was nominated for Empire Award for Best TV Series.
Text 2 (Known); Westworld is directed by Jonatham Nolan.
Figure 2:
An example of shallow semantic similarity. Although "Text I" and "Text 2" express different rela- tions, there is similar surface information between them.
tic structures, to sentences with known relations, as shown in Figure 2. Previous methods only use training samples of known relations to train mod- els, which makes them difficult to handle OOS data with similar surface information. They may pre- dict similar features or probabilities for these OOS data and known training data, eventually leading to confusion between the novel and known relations.
There are two lines of previous work related to NRE, open RE (Yao et al., 201 I; Marcheggiani and Titov, 2016; Wu et al., 2019; Liu et al., 2021) and OOS detection (Hodge and Austin, 2004; Chandola et al., 2009; Lin and Xu, 2019; Wu et al., 2021). Open RE means that there are no predefined rela- tion types and no labeled data. Open RE extracts phrases and arguments as specific relations and discovers new relations by clustering or heuristics. Compared to open RE, NRE strengthens the capa- bility of the conventional RE methods and aims to automatically discover novel relation types that are not the predefined relation set, while providing accurate predictions for known relations. Another line of related work is OOS detection, which aims to recognize OOS data that does not belong to any predefined class. Although OOS detection has been widely investigated in other NLP tasks, its explo- ration in RE is relatively rare.
To address the above problem, we propose effective NRE method, called Multi-Strategy Self- Supervised Learning (M3S), which explicitly mod- els OOS data with similar semantic information by constructing pseudo-OOS data. Specifically, M3S uses features of known training data to construct various pseudo-OOS data. In the semantic feature space, these pscudo data are close to the known data. Then, M3S utilizes these pseudo- OOS data to train the model, thus improving the generalization in the novel relation. Experimental results show that M3S significantly outperforms previous state- of-the-art OOS detection methods.
In summary, the contributions of this work are as follows: (1) To the best of our knowledge, we are the first to formally introduce OOS detection into RE and we construct two NRE benchmarks through two widely used RE datasets. (2) We inves- tigate a variety of cxisting OOS detection methods and further propose multi-strategy self-supervised learning, which can effectively handle the prob- lem of shallow semantic similarity in the NRE task. (3) Experimental results show that M3S signifi- cantly outperforms previous OOS detection meth- ods on the two benchmarks. The source code and benchmarks will be released for further research (ht (https:/// //gi thub. com/ f Ligiigginn22222///WW))
Therefore, in this paper, we formalize the NRE task and construct two NRE datasets based on two widely used RE datasets, FewRel (Han et al., 2018) and TACRED (Zhang et al., 2017). Then, to estab- lish NRE's baselines, we exhaustively investigate variety of OOS detection methods (Hendrycks and Gimpel, 2017: Lin and Xu, 2019; Yan et al., 2020). In general, previous OOS detection methods usu- ally learn the decision boundaries of known classes based on the feature or probability distributions of known training data. In the testing phase, they use confidence scores to identify samples outside the decision boundaries as OOS data.
Task Formulation
2.1 Relation Extraction
However, when applying existing OOS detection methods in the NRE task, we find shallow seman- tie similarity problem. Specifically, sentences with OOS relations may have similar surface informa- tion, such as entity overlapping and similar syntac-
The traditional RE task is usually formulated as a text classification task (Zhang et al., 2017; Han et al., 2018). Given a sentence x that contains pair of entities, the traditional RE task is to predict a

---

relation y for these two entities. This task assumees y E y, where y denotes a predefined relation set.
Metric
FowReI-NRE-20%
TACRED-NRE-DOE
Training Data
26,880
2,880
Validation Data
8,960
960
2.2 Novel Relation Extraction
11,200
1,200
OOS Test Data
2.240
240
The NRE task aims to identify the data of OOS rela- tions while correctly classifying the data of known relations. We denote the data of known relations as D (Dy, D2 ..... D.). Dj is the data of the i-th known relation, which has its own training, vali- dation, and test sets (Dtraio, Dralid,o Diosto). To match the realistic environment, there is no training and validation data for the novel relation type. In the development phase, the NRE model can only access the training and validation data of known relations. In the test phase, the NRE model should classify each sample into (k- + 1)-class relations, where the (k- + 1)-th class is the novel relation type. Therefore, in the test phase, we employ the test data of both known and novel relations to evaluate the model. Due to the lack of OOS training data, it is difficult for previous RE methods to make accurate predictions for the novel relation type.
Vocabulary Size
66,
13,714
Known Relations
64
16
OOS Relations
16
Table Statistics of the NRE datasets. "#" indicates "the number of".
these OOS relations, remove their training and validation data and keep their test data as the test data of the novel relation type. Based on FewRel and TACRED, we propose two instantiations of the above construction method. FewRel-NRE: Since the original FewRel dataset does not provide data splitting, we split the data of each relation in a 3:1:1 ratio into the training, validation, and test sets. TACRED- -NR:: Considering the severe class imbalance in TACRED, frequent relations and limit the number of training, validation, and test samples of each relation 180, 60, and 60, respectively. We treat the "no relation" class in TACRED as a known relation to fit the real- world setting. To avoid randomness, we construct 5 different datasets with 5 different random seeds for each specific ratio. These datasets can well simu- late unpredictable and diverse OOS relations in the real world. Table shows the statistics of the two constructed datasets in which 20% of the relations are selected as OOS relations.
use the top 20 most
3 Dataset
To the best of our knowledge, we are the first to formally introduce the NRE task. Therefore, we construct two NRE datasets based on two widely used RE datasets. In this section, we first briefly in- troduce the original RE datasets and then describe the construction method of the NRE datasets. Fi- nally, we show the statistics of these datasets.
3.1 Original Relation Extraction Datasets
Previous OOS Detection Methods
FewRel (Han al., 2018) is few-shot RE dataset, which contains 100 relations. In our work, we use the publicly available 80 relations the original dataset. Since FewRel contains a sufficient number of relation types, we can use FewRel to simulate various OOS relations well. Each relation in the FewRel dataset has 700 labeled data. TACRED (Zhang et al., 2017) is an RE dataset that contains 42 relations. In the TACRED dataset, there are 68124, 22631, and I 1,509 samples in the training, validation, and test sets, respectively.
In other OOS detection tasks, recent work has at- tempted to find appropriate decision boundary to balance the performance of both known and OOS relations. We roughly divide previous methods methods (Hendrycks and Gimpel, 2017: Shu et al., 2017) and feature-based methods (Lin and Xu, 2019; Yan et al., 2020). We apply these methods to the NRE task and provide a detailed quantitative analysis.
into two categories: probability-
4.1 Frobability-Basse Methods
Probability-basee methods assume that the updated model will not be overconfident in the OOS data. In the training phase, probability- methods use the training data of known relations to update the model. In the test phase, these methods derive a confidence score from the probability distribution of each sample. If the confidence score of test
3.2 NRE Dataset Construction
For original RE dataset, randomly select some relations as 0OS relations according a spe- cific ratio. Since many relations can be predefined in practical applications, we adopt three reason- able ratios, 10%, 20%, and 30%, in this paper. For

---

1.0
1.0
10.8
0.8
Masiuum Probaliitt
0.7
0.6
0.6
0.5
0.4
0.3
1.1
0.1
Knows
0.0
MSP (Knowa) MSP (00S) DOC (Known) DOC (OOS)
0.0
0.5 0.6 0.7 0.8
Figure 3: Maximum probabilities of the test samples of the NewReI-NRE-200 dataset. "MSP (Known)" de- notes the maximum probabilities predicted by the MSP model for test samples with known relations. Test sam- ples whose maximum probability below the threshold are predicted as OOS data.
Figure 4: t-SNE visualization of the FewReI-NRE-2055 dataset. Blue dots denote the features the training samples with known relations. Light red dots denote the features of the OOS test samples.
fused with features of known relations. Since the SEG method has not learned OOS data, it may have difficulty extracting valid features for these OOS samples, which ultimately hurts the performance.
sample is below a threshold, it is treated as OOS data. For example, MSP (Hendrycks and Gimpel, 2017) adopts the maximum softmax probability as the confidence score.
Methodology
However, find that these methods perform poorly the NRE task. To give clear analysis, we present the maximum probabilities of test samples with known relations and OOS relations separately in Figure 3. From the results, can see that both the MSP and DOC (Shu et al., 2017) methods tend to overconfidently assign known relations to OOS data. We speculate that since these models have not learned the OOS data, they may not generalize well to the OOS data with similar surface infor- mation, leading to incorrect predictions, Besides, these methods require an additional validation set containing OOS data to obtain the threshold (T).
To address the above problem, we propose Multi- Strategy self-Supervised Learning, which explic- itly models OOS data with similar surface informa- tion by constructing various pseudo-OOS data.
5.1 Input Encoding
We adopt BERT (Devlin et al.. 2019), which is powerful pre-trained language model, as the text encoder. The BERT encoder outputs a contextual representation for each input sequence as:
h (ERT(x))
(1)
where h is the hidden state. We the hidden state of the [CLS] token as the feature representa- tion (xx) of each sequence. Note that our method agnostic to the model architecture. Other en- coders can also be adopted.
4.2 Feature-Based Methods
Feature- based methods derive the confidence score from the feature distribution. In the training phase, these methods employ specific optimization objec- tives constrain the feature distribution of known classes. In the test phase, they use distance-based outlier detection algorithms, such as LOF (Breunig et al., 2000), to derive the confidence score from the feature distribution. For example, SEG (Yan et al., 2020) is typical feature-based method that assumes the features of known classes follow Gaussian mixture distribution.
5.2 Multi-Strategy Self-Supervised Learning In M3S, we design three self-supervised strategies to construct pseudo-OOS data, including convex combination, regional replacement, and irregular replacement. These pseudo-OOS data, together with the training data of known relations, used to update the model improve its generalization. The framework of M3S shown in Figure 5.
To verify the effectiveness of feature-based meth- ods on the NRE task, we provide the feature visu- alization of the SEG method in Figure 4. We can see that many features of OOS relations are con-
5.2.1 Convex Combination
For this strategy, we use convex combinations sample features from different relations as pseudo-

---

Irregular Replacement
5.2.3 Irregular Replacement
K-Class Training data
Regional Replacement
Relation
Moreover, design irregular replacement. Dif- ferent from regional replacement, irregular replace- ment randomly selects some discontinuous dimen- sions to replace, instead continuous region. To obtain the mask vector M of this strategy, we first sample the random value 3 and calculate the number of replaced dimensions as: = 8 d. Then, assign coefficient 7, which is randomly selected from the uniform distribution U(0, 1), for each dimension. Finally, we select the top I dimen- sions that get larger coefficients for replacement. the mask vector, we set these selected dimensions to O and the others to 1. We use the mask vector to construct the pscudo-OOS data f and obtain a new training set: Dirain, = U f.
Convex Combination
BERT Encoder
(K+I)-Class Classifier
Original Features
Figure 5: Overall framework of the proposed method.
OOS data, which can be formalized as:
p== a** f(11) (1 a:) f(22)
(2)
where 21 and 22 are two training samples with different known relations. These two samples are randomly selected from the same training batch. fee the synthetic OOS data. is randomly sampled from uniform distribution U(0, 1). a is close to or f will be close to the features known relations in the semantic feature space. These synthetic features can thus simulate real OOS data that have similar semantic information as known training data.
scalar that
5.3 Optimization
In each training batch, we combine the original fea- tures with the synthetic OOS features as the train- ing set, i.e., Dirain Dirann U Dimainsse U Dimann.r U Dirain, Thus, we can train a uniform (k + 1)-class classifier without additional post-processing, In our method, we employ the softmax classifier and use the cross-entropy loss to optimize the model:
These synthetic features are assigned to the novel relation type form new training set: Dtrain.cc U p Since these samples are constructed in the feature space, it is very efficient to construct a large number of pscudo data.
5.2.2 Regional Replacement
[Dinia
LCE =
log
(5)
Since the distribution of real OOS data is often arbitrary and diverse, we further propose a regional replacement strategy to construct diverse pseudo- OOS data. This strategy constructs pseudo samples by replacing random region of the feature of æ1 with the corresponding region of the feature of Ø2:
1=
C
(AIT
where o denotes the output logit of the ground- truth class j of the feature ft. C is the entire (k: + 1)- class relation set. T is the temperature scalar. In the testing phase, M3S is able to directly predict known relations while identifying the novel relation.
= - M f(11) + (1 M) + (
(3)
where Mi is binary mask vector that indicates the region of replacement. 21 and 22 two training samples with different known relations. @ is the element-wise multiplication.
6 Experiments
6.1 Baselines
To provide a comprehensive comparison, em- ploy multiple OOS detection methods baselines. MSP (Hendrycks and Gimpel, 2017) obtains the confidence score from the maximum softmax prob- ability and treats samples that get lower scores as OOS data. DOC (Shu et al., 2017) uses multiple 1-vs-rest sigmoid classifiers to optimize the prob- ability distribution. LMCL (Lin and Xu, 2019) utilizes large margin cosine loss to learn discrim- inative features and detects outliers via the LOF algorithm. SEG (Yan et al., 2020) assumes that the features of known classes follow Gaussian mixture distribution. It also uses LOF to detect outliers. SEG-SF (Yan et al., 2020) uses a softmax
To get the mask vector M, we first sample a ran- dom value 3 from uniform distribution U(0, 1). Then, we calculate the region length as: -- 3 d,
where di is the dimensional size of features. Finally,
we calculate the region coordinates:
~ d) 5 = == , =t 11/2
(4)
where is a random integer between 1 and d. and g are the start and end coordinates of the selected region. this region, the value of the mask vector is O, otherwise it is 1. This strategy forms another new OOS training set: Diraingr U JM

---

FewRel
TACRED
Rate
Method
Accuracy
Macro Fl
Known
Novel
Accuracy
Macro
Known
Novel
MSP
75.03
80.82
81.42
37.87
66.36
70.46
72.48
34.09
DOC
74.14
79.03
79.70
30,94
72.29
74.24
76.40
35.41
LMCL
75.14
81.07
81.67
37.90
71.24
74.06
76.26
34.37
10%
SEG
68.38
73.90
74.51
30.21
64.23
66.10
67.76
36.15
SEG-SF
69.67
76.17
76.78
32.74
65.73
68.31
70.16
34.90
GDA
75.91
81.38
81.99
37.61
71.08
72.90
74.92
36.53
M3S
77.29
81.99
82.56
40.90
72.62
75.26
77.17
40.97
MSP
74.56
80.21
80.63
53.24
66.32
72.53
44.05
DOC
70.83
77.42
78.06
36.65
68.82
73.29
75.54
37.42
LMCL
75.12
80,98
81.42
53.03
67.81
73.16
75.33
38.44
20%
SEG
66.19
72.31
72.88
42.57
59.72
64.18
66.33
29.63
SEG-SF
69.67
75.34
75.72
50.97
63.13
66.76
68.58
37.59
GDA
75.10
80.65
81.09
52.64
67.78
72.02
73.96
40.91
M3S
75.91
81.06
81.47
55.02
69.89
73.94
75.59
47.47
MSP
72.00
78.17
78.54
57.63
65.73
68.74
69.74
54.75
DOC
66.11
75.23
75.87
39.24
63.49
68.76
70.84
39.64
LMCL
72.06
78.72
79.09
58.25
64.18
69.44
71.09
46.36
30%
SEG
63.13
70.29
70.69
47.49
57.96
63.44
65.32
37.03
SEG-SF
68.58
75.88
76.32
51.54
62.25
66.63
68.23
44.25
GDA
71.77
78.07
78.46
56.57
64.08
69.13
70.98
43.20
M3S
73.23
78.77
79.09
60.70
68.12
71.18
72.18
57.17
Table 2: Main results (%) with different proportions of OOS relations on FewRel and TACRED datasets. "Known" and "Novel" denote the macro F1 over the known relations and the novel relation, respectively.
classifier in the original SEG method. GDA (Xu al., 2020) utilizes the Mahalanobis distance be- tween each feature and the class prototype as the confidence score.
6.3 Evaluation Metrics
Following other OOS detection tasks (Yan et al., 2020; Zhang et al., 2021), we use the overall ac- curacy and the macro Fl score as evaluation met- rics. addition, we report the macro F1 scores of the known relations and the novel relation sepa- rately. For each OOS rate, we construct 5 different datasets and report the average results.
MSP and GDA require a specific validation that contains OOS data to adjust their thresholds. We provide such a validation set for MSP and GDA by integrating the validation data of OOS relations. Our method and other baselines only use the data of known relations for training and validation, which is more realistic setting.
6.4 Main Results
The main results are shown in Table 2. From these results, we can see that:
(1) Our method M3S significantly outperforms other basclines and achieves state-of-the-arl per- formance on all datasets. For example, compared to GDA. our method achieves 2.38% and 6.56% improvements in terms of the of the novel re- lation type on FewRel-NRE-20%% and TACRED- NRE-20% datasets, respectively. It verifies the effectiveness of our method on the NRE task. (2) Under each rate, there is significant perfor- mance gap between the baselines and our method. The reason that previous methods ignore the shal- low semantic similarity problem in the NRE task,
6.2 Experimental Settings
We use the HuggingFace's Transformer library² to implement the BERT-based model. To ensure fair comparison, all baselines employ the same BERT encoder. As suggested by Devlin et al. (2019), the learning rate is le-5. The temperature scalar T is 0.1. The batch sizes for FewRel and TACRED are 32 and 16. Each self-supervised strategy structs a batch size pseudo-OOs samples. The hyper-parameters are obtained by a grid search on the validation set.
which makes them difficult to identify OOS data with similar surface information. Besides, we find
/https://github. http://tthubb..mmm/ugggiggggg//t

---

FewRel
TACRED
Rate
Method
Accuracy
Macro Fl
Known
Novel
Accuracy
Macro F1
Known
Novel
M3S
75.91
81.06
81.47
55.02
69.89
73.94
75.59
47.47
CC
75.70
80.80
81.21
54.08
68.35
72.64
74.29
46.19
RR
75.49
80,68
81,09
54.25
68.77
72.90
74.37
46,24
75.31
80.47
80.89
53.98
68.83
72.94
74.62
46.18
20%
CC & RR
75.02
80.27
80.68
53.53
67.90
72.32
74.03
45.03
CC & IR
74.81
80.03
80.44
53.28
67.65
71.71
73.33
45,80
RR IR
74.72
79.97
80.38
53.41
67.50
71.55
73.26
44.17
M3S
69.36
77.44
78.65
0.00
64.44
68.55
72.84
0.00
Table Ablation studies (%) on the effectiveness of multi-strategy self-supervised learning.
that, without additional OOS validation data, SEG tends to overfit known relations quickly, leading to performance degradation.
performance in terms of the Fi the novel rela- tion. This proves that our method is robust over wide range of numbers. Our method is capable of constructing effective and precise pseudo-OOS data for the novel relation type.
(3) Although MSP and GDA achieve acceptable performance, they require additional OOS valida- tion data adjust their thresholds. Our method does not require such data and still outperforms MSP and GDA methods.
(2) For the other metries, the performance first increases and then slightly drops as the number in- creases. We speculate that as the number increases, the constructed OOS data grows rapidly, which leads to severe data imbalance between the novel relation and the known relations. The data imbal- problem makes the model significantly biased towards learning the synthetic OOS data, affecting the performance of known relations. In this paper, we leave this problem for future work.
6.5 Ablation Study
To gain more insights into the multi-strategy self- supervised learning, conduct ablation studies by evaluating multiple variants of M3S. The re- sults are shown in Table 3. To reduce the effect of the number of pseudo-OOS data, we keep the total number of pseudo-OOS data per batch for all variants the same as M3S.
(3) Compared to the model without pseudo data, our method not only achieves improvements in the novel relation but also brings positive effect known relations. This is mainly due the fact that our method is able to correctly assign the novel relation to OOS data rather than known relations, thus boosting the overall performance.
From the results, we can see that: Removing any self-supervised strategy, i.c., convex combi- nation (CC), regional replacement (RR), irreg- ular replacement (IR), brings performance degra- dation. This proves the effectiveness of each self- supervised strategy. We infer that these strategies can effectively construct diverse pseudo-OOS -00 data to improve performance. When we remove the three strategies (- M3S), the performance drops significantly. This indicates that OOS detection is critical for RE models in real-wor world applications.
6.7 Efficiency
To demonstrate the efficiency of our method, we compare the average training time per epoch and the total test time different methods. The results are shown in Figure 7. Even with pseudo-OOS samples, the training time of our method M3S is comparable to that of these baselines. The training time per epoch of our method is even lower than that of the SEG model. Importantly, the test time of method is comparable that of the MSP model and significantly lower than other baselines. This is due to the fact that our method does not require additional processinn modules and it classifies all test data in uniform manner.
6.6 Effect of the Number of Pseudo-OOS Data
We further investigate the effect of varying the num- ber of pseudo-OOS data. As shown in Figure 6, we increase the number of pseudo-Ooo OOS data con- structed by each self-supervised strategy from 0 to 160. Note that 32 is the default value of our method on the FewReI-NRE dataset. From the results, we can see that:
(1) As shown in Figure 6 (d), with the increase of synthetic OOS data, M3S achieves comparable

---

75
82
74
80-
Mauro F1 (Nensum)
Accrney
13
79
800
((Novel)
10
78
79
20
78
70
76
77
33
64
96
12.5
160
75
52
96
128
160
76
32
64
96
128
160
52
64
96
128 160
V. SSllt-Sppersied OOS Data
Sell-Supervived OOS Data
SSell-SSpprrie pervise OOS Duta
/Selt-Superviee OOS Dats
(b)
(d)
Figure 6: Effect (%) of the number of supe--ue OOS data on the FewReI-NRE-200 dataset. (a). (b). (c), and (d) show the overall accuracy, overall FI, macro Fl of known relations, and Fl of the novel relation, respectively. "#" indicates the number of pseudo-OOS data constructed by each self-supervised strategy.
the real world. Open RE extracts phrases and argu- ments as specific relations and discovers new rela- tions by clustering or heuristies (Yao et al., 2011; Cui et al., 2018; 8; Kolluru et al., 2022). However, it can not automatically discover novel relations that are not in the predefined relation set. Gao et al. (2019) focus on OOS detection in the few-shot RE task. Compared to these works, formally propose realistic and challenging task, i.e., OOS detection for the traditional RE task.
150
100
(a) Training Time
125
100
75
7.2 Out-of-Scope Detection
MSP DOC LMCL SEG SEG-F GDA
M3S
(b) Test Time
Out-of-scope detection is long-standing research topic in machine learning, which enables models to identify OOS data new/open class (Hodge and Austin, 2004; Zimek al., 2012; Lee al., 2018; Zhang et al., 2021). Existing mainstream OOS de- tection methods can roughly divided into two categories: probability- methods (Hendrycks and Gimpel, 2017; Shu et al., 2017) and feature- based methods (Lin and Xu, 2019; Yan et al., 2020; Xu al., 2020). Probability-based methods derive from the probability distri-
Figure 7: Comparison (in seconds) of the training time per epoch and the total test time on the FewRel-NRE. 20% dataset.
7 Related Work
7.1 Relation Extraction
Recently, there are many research works on RE (Zelenko et al., 2002; Zeng et al., 2018; Li et al., 2019; Ye al., 2022). Guo al. (2019) propose attention-guided graph convolutional networks to select useful substructures from dependency trees for RE. Wang et al. (2021) use a unified label space to model the information between entities and re- lations. addition, the few-shot RE task aims to train an RE model using few samples (Han et al., 2018; Sainz et al., 2021). The continual RE task en- ables RE models to continually learn new labeled data (Wang et al., 2019; Cui al., 2021).
the confidence bution. Feature-based methods generally employ outlier detection methods, such as LOF (Breunig et al., 2000) or one-class SVM (Schölkopf et al.. 2001), to detect OOS data. In addition, there are some research efforts that use synthetic or real OOS data to aid model training (Ryu et al., 2018 8; Lee et al., 2018; Hendrycks et al., 2019). Zhan et al. (2021) utilize data augmentation method similar Mixup (Zhang al., 2018) synthesize outliers. Inspired by Zhan al. (2021) and other data aug- mentation methods (Yun al., 2019; Harris et al., 2020), we propose aulti-strategy self-supervised learning for the NRE task.
Despite the great progress in RE tasks, these existing methods usually ignore the discovery of novel relations, which limits their application in

---

8 Conclusion
Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186.
In this paper, we introduce OOS detection into re- lation detection, which can automatically discover the data with OOS relations while correctly clas- sifying the data with known relations. To cope with shallow semantic similarity, we propose multi- strategy self-supervised learning. We construct two datasets for the NRE task and compare our method with multiple strong baselines. The results demon- strate the effectiveness of our method.
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classi- fication. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNDL pages
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Atten- tion guided graph convolutional networks for relation extraction. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Lànguistics, pages 241-251.
Limitations
In this paper, each self-supervised strategy con- structs the same amount of pseudo-OOS data. In fact, different strategies bring different improve- ments as shown in Table Therefore, in future work, we hope to find better ways to fuse these strategies.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of -ote-------hrrrr the-art evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant No. No.62306087) and the Natural Science Foundation of Shandong Province (Grant No.22R2020QF154).
Ethan Harris, Antonia Marcu, Matthew Painter, Mah- esan Niranjan, Adam Prügel-Bennete and Jonathon Hare. 2020. FMix: Enhancing Mixed Sam- ple Data Augmentation. arXiv e-prints, page arXiv:2002. 12047.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ô Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Sypakowicz. 2010. SemEval-2010 task 8: Multi- way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38.
References
Markus M. Breunig, Hans-Peter Kriegel. Raymond T. Ng, and Jörg Sander. 2000. LOF: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference Management of Data, pages 104.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: survey. ACM Comput. Surv. 41(3):15:1-15:3
Dan Hendryeks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distributie ex- amples in neural networks. In 5th International Con- ference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Lei Cui, Furu Wei, and Ming Zhou 2018. Neural open information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 407- 413.
Dan Hendrycks, Mantas Mazeika, and Thomas G. Di- etterich. 2019. Deep anomaly detection with outlier exposure. In 7th International Conference on Learn- ing Representations.
Li Cui, Deqing Yang, Jiaxin Yu, Chengwei Hu, Jiayang Cheng. Jingjie Yi, and Yanghua Xiao. 2021. Refin- ing sample embeddings with relation prototypes to enhance continual relation extraction. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 232-243.
Victoria J. Hodge and Jim Austin. 2004. A survey of outlier detection methodologies. Artif. Intell. Rev. 22(2):85-126.
Keshav Kolluru, Muqeeth Mohammed, Shubham Mit- tal, Soumen Chakrabarti, and Mausam 2022. Alignment augmentte consistent translation for mul- tilingual open information extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers). pages 2502-2517
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for

---

Kimin Lee. Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018. Training celedennc--alibrate classifiers for detecting out-of-distribution samples. In 6th Interna- tional Conference on Learning Representations.
Lei Shu, Hu Xu, and Bing Liu. 2017. DOC: Deep open classification of text documents. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 2911 9111916
Pengfei Li, Kezhi Mao, Xuefeng Yang, and Qi Li. 2019. Improving relation extraction with knowledge- attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 229-239.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki- data: free collaborative knowledgebase. Commum. ACM, 57(10):78-85
Hong Wang. Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sen- tence embedding alignment for lifelong relation ex- traction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 796-806.
Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Compu- national Linguistics, pages 5491-5496.
Yijun Wang, Changzhi Sun. Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021. UniRE: A unified label space for entity relation extraction. of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 220 231.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In P Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2124-2133
Proceedings
Fangchao Liu, Lingyong Yan. Hongyu Lin, Xianpei Han, and Le Sun. 2021. Element intervention for open relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 4683 4693.
Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2019. Open relation extraction: Relational knowledge trans- fer from supervised data to unsupervised data. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 219-228.
Diego Marcheggiani and Ivan Titov. 2016. Discrete- state variational autoencoders for joint discovery and factorization relations. Trunsactions of the Associ- ation for Computational Linguistics, 4:231-244.
Yanan Wu, Zhiyuan Zeng, Keqing He, Hong Xu, Yuan- meng Yan, Huixing Jiang, and Weiran Xu. 2021. Novel slot detection: A benchmark for discover- ing unknown slot types in the task-oriented dialogue system. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3484-3494.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Pro- ceedings of the 2013 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84.
Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zi- jun Liu, and Weiran Xu. 2020. A deep generative distance-based classifier for out-of-domain detection with mahalanobis space. In Proceedings of the 28th International Conference on Computational Linguis- tics, pages 1460-
Seonghan Ryu, Sangjun Koo, Hwanjo Yu, and Gary Ge- unbae Lee. 2018. Out-of-domain detection based on generative adversarial network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 714 418.
Guangfeng Yan, Lu Fan, Qimai Li, Han Liu, Xiaotong Zhang, Xiao-Ming Wu, and Albert Y.S. Lam. 2020. Unknown intent detection using Gaussian mixture model with application to zero-shot intent classifi- cation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 1050-1060.
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, An- der Barrena, and Eneko Agirre. 2021. Label verbal- ization and entailment for effective zero and few-shot relation extraction. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 1 119-1212.
Limin Yao, Aria Haghighi. Sebastian Riedel, and An- drew McCallum. 2011. Structured relation discovery using generative models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1456-1466.
Bernhard Schölkopf, John C. Platt, John Shawe-Taylon Alex J. Smola, and Robert C. Williamson, 2001. Es- timating the Support of High- High-Dimensiona. Distri- bution. Neural Computation, 144311

---

Deming Ye. Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4904-4917.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference Computer Vision, pages 6022-6031.
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2002. Kernel methods for relation ex- traction. In Proceedings of the 2002 Conference Empirical Methods Natural Language Processing, pages 71-78.
Daojian Zeng. Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via con- volutional deep neural network. Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2335-2344.
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by end-to-end neural model with copy mechanism. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics VVolume 1: Long Papers), pages 506-514 514.
MinMing Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao- Ming Wu, and Albert Y.S. Lam. 2021. Out-of-scope intent detection with self-supervision and discrimi- native training. In Proceedings the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3521-3532.
Hanlei Zhang, Hua Xu, and Ting-En Lin. 2021. Deep open intent classification with adaptive decision boundary. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 14344-4382.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. 6th International Conference on Learning Representations.
Yuhao Zhang, Victor Zhongg Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 35- 45.
Arthur Zimek, Erich Schubert, and Hans-Peter Kriegel. 2012. A survey on unsupervised outlier detection in high-dimensional numerical data. Statistical Analy- sis and Data Mining, 5(5):363-387.


Reviews:
[{"review_id": "5MWgfuftkI", "reviewer": "Reviewer_tVzT", "summary": "", "questions": "- Since you have run each experiment multiple times, could you conduct a t-test and report the p-values by comparing M3S and the strongest baseline/ablation version in each column in Tables 2 and 3?\n\n- The phenomenon in Figure 6 that the overall accuracy first increases and then drops is a bit strange. I do not quite agree with the authors' speculation because even if about 100 self-supervised OOS samples are generated, this number is still significantly smaller than the number of real training samples per known relation according to Table 1. I wonder if the authors observe the same phenomenon on TACRED.\n\n- Could you provide a more thorough discussion of the limitations of this work?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments:\n\nFor Q1.\n\nA1: We will add significance analysis in the revised version. Due to time limitations, we conducted some of the experiments. For Table 1, we conducted the significance analysis for Rate=20%. The results show that the improvement of our method is statistically significant with p<0.05 under the t-test over the previous strong models, GDA and LMCL. In particular, on the “Novel F1” metric, the improvement of our method is statistically significant with p < 0.01. For Table 2, we conducted a t-test between M3S and -RR. The results show that the improvement of our method is statistically significant with p < 0.05. This can partially prove the significance of the improvement of our method.\n\nFor Q2.\n\nA2: It is possible that the description of the coordinates in Figure 6 is not sufficiently clear and caused your misunderstanding. The horizontal coordinate in Figure 6 indicates the number of self-supervised/OOS data constructed by each strategy in each batch, rather than the total number of OOS data constructed during the training process. In this experiment, the data size of known relations in each batch is 32, and the OOS data size of each self-supervised strategy in each batch is shown in the horizontal coordinate. For example, when the horizontal coordinate is 128, the three strategies construct 128*3=384 OOS data per batch, while the data size of known relations in each batch is 32. This will result in a 12-fold difference, leading to a severe data imbalance problem.\n\nFor Q3.\n\nA3: Yes, we can. We will optimize the limitation section subsequently. As you suggested above, in terms of task setting, this work treats all novel relations as one class, which might increase the difficulty of artificially formulating the class set in the subsequent labeling process of detected OOS data. In addition, higher proportions (>>30%) of unknown relations may pose more difficulties for all existing models, which needs to be further explored.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The task of detecting out-of-scope relations makes the relation extraction task more generalizable and has practical value for downstream tasks such as knowledge graph construction. \n\n+ The proposed multi-strategy self-supervised learning is intuitive. Its combination with existing out-of-scope detection methods (i.e., convex combination, regional replacement, and irregular replacement) is well-motivated. \n\n+ Experiments are comprehensive. Two benchmark datasets are used (adapted for the NRD setting). Various out-of-scope detection baselines are compared. The authors also conduct ablation analyses and hyperparameter studies to validate their design choices and configurations.", "reasons_to_reject": "- The task setting of treating all out-of-scope relations as one class is a bit confusing. In practice, there may be a large number of OOS relations (e.g., 16 in the FewRel dataset used by the authors), and their semantics can be quite scattered in the embedding space. In this case, a more practical setup would be automatically detecting the number of OOS relations (given that the number is in a certain range, e.g., 10-20) through unsupervised clustering.\n\n- Significance tests are missing. It is unclear whether the improvement of M3S over baselines and ablation versions is statistically significant or not. In fact, some gaps in Tables 2 and 3 are quite subtle, therefore p-values should be reported.\n\n- The Limitations section is too concise, different from the clear and detailed writing style in the main body of this paper. ", "questions_to_authors": "- Since you have run each experiment multiple times, could you conduct a t-test and report the p-values by comparing M3S and the strongest baseline/ablation version in each column in Tables 2 and 3?\n\n- The phenomenon in Figure 6 that the overall accuracy first increases and then drops is a bit strange. I do not quite agree with the authors' speculation because even if about 100 self-supervised OOS samples are generated, this number is still significantly smaller than the number of real training samples per known relation according to Table 1. I wonder if the authors observe the same phenomenon on TACRED.\n\n- Could you provide a more thorough discussion of the limitations of this work?", "paper_topic_and_main_contributions": "This paper introduces a new task called novel relation detection (NRD) in relation extraction. Different from traditional RE approaches that recognize predefined relation types only, NRD aims to discover potential new relation types based on training samples of known relations. This paper proposes two NRD datasets and explores various out-of-scope (OOS) detection methods. It further presents an effective NRD method that uses multi-strategy self-supervised learning to address shallow semantic similarity issues in the task. Experimental results demonstrate the effectiveness of the proposed M3S method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tQkZAect02", "reviewer": "Reviewer_cVH9", "summary": "", "questions": "1. How do the two hyperparameters, α and β, affect the model? Do they lead to more significant performance improvements when approaching 0?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments.\n\nFor Q1.\n\nA1: These two hyperparameters are indeed very worth exploring. We actually analyzed these two hyperparameters quantitatively during the model design. For example, we have a strategy to limit these two parameters to below 0.2 or even 0.1. However, several strategies we tried all resulted in a slight performance degradation. We speculate that these inductive biases might limit the generated OOS data in some ways so that they cannot match the diverse OOS data in practice.\n\nFor R2 (case study, etc.).\n\nA2: Additional experiments, such as case studies and feature visualizations, would indeed enrich this paper. Due to page limitations, we will consider adding them to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces, for the first time, the task of novel relation detection, a task that aligns with the demand for relation extraction models in open-world scenarios.\n\n2. The paper analyzes the limitations of existing mainstream methods and visually demonstrates the reasons for their failures through experimentation.\n\n3. The proposed method presented in this paper is both simple and effective.", "reasons_to_reject": "\n1. The pseudo-OOS samples generation methods proposed in this paper lack innovation, as all three methods are essentially variations of the SMOTE [1] technique.\n\n2. The experimental presentation in this paper is insufficient: (1) Lack of case studies to demonstrate the effectiveness of the proposed method in mitigating the impact of shallow semantic similarity; (2) Absence of corresponding experiments to showcase the influence of the proposed method on model feature representation and prediction probabilities, as depicted in Figures 3 and 4.\n\n[1] Chawla N V, Bowyer K W, Hall L O, et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of artificial intelligence research, 2002, 16: 321-357.", "questions_to_authors": "1. How do the two hyperparameters, α and β, affect the model? Do they lead to more significant performance improvements when approaching 0?", "paper_topic_and_main_contributions": "In this paper, the authors introduce a new relation extraction task termed \"novel relation extraction\" (NRE), where the model is required to identify unknown relation types during inference while maintaining its classification capacity for known relation types. The authors highlight the challenge of shallow semantic similarity in the NRE task and demonstrate through experiments that existing probability-based and feature-based Methods might fail under this task. To address the challenge of shallow semantic similarity, this paper enhances the model's robustness to out-of-scope data during the training phase using Multi-Strategy Self-Supervised Learning. This approach achieves state-of-the-art results on two NRE datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UJ0bzOMmQj", "reviewer": "Reviewer_a4GX", "summary": "", "questions": "A: How reliable are the results on TACRED given its small scale of test relations? The construction of the TACRED-NRE dataset includes only the top 20 most frequent relations. Does this imply that all novel relations should be frequent? This assumption might limit the scope of the paper. It would be interesting to see how the proposed model can generalize to unseen long-tail relationships, as novel relations in reality might not occur as frequently as common relations.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments.\n\nFor Q1.\n\nA1: Following other OOS detection work, we adopt a similar data balancing setting. The main purpose of this setting is to allow these datasets to focus on the OOS detection problem rather than the data imbalance problem. This does not mean that the novel relations should be frequent. Our experiments in Table 1 indicate that the \"Novel F1\" metric of all models decreases when there is less OOS test data, and our model still outperforms other models. The study of unseen relations with long-tailed distributions may indeed become another important research topic in the future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is the first to frame the task of novel relation extraction as an OOS detection problem, establishing strong baseline models from OOS detection research. This paper bridges the gap between OOS detection and novel relation extraction.\n2. The strategies for constructing pseudo-OOS instances are based on the embedding level, which is efficient and reduces the need for real training data for novel relations. The experimental results demonstrate the effectiveness of these strategies on two datasets (FewRel and TACRED)", "reasons_to_reject": "1. The assumption that pseudo-OOS instances are novel relations based on arbitrary combinations/corruptions of any two existing relations may not apply to novel relations that don’t satisfy this assumption. \n2. The difference between this paper and previous work is not clear. Like previous work, this paper considers novel relations as a whole class, questioning whether the model can distinguish test instances containing novel relations or those without any relations.\n3. The paper identifies shallow semantic similarity as an issue in identifying novel relations but does not provide sufficient evidence to demonstrate the severity of this problem or how the proposed model addresses it.", "questions_to_authors": "A: How reliable are the results on TACRED given its small scale of test relations? The construction of the TACRED-NRE dataset includes only the top 20 most frequent relations. Does this imply that all novel relations should be frequent? This assumption might limit the scope of the paper. It would be interesting to see how the proposed model can generalize to unseen long-tail relationships, as novel relations in reality might not occur as frequently as common relations.", "paper_topic_and_main_contributions": "The paper addresses the challenge of identifying novel relations that are not present in predefined relation ontology and training data. It frames this task as an out-of-scope (OOS) detection problem and proposes a multi-strategy self-supervised learning approach. This approach includes three strategies for constructing pseudo-OOS training instances that simulate novel relations by manipulating the feature representations of known relations. The paper demonstrates improved performance in novel relation classification while maintaining accuracy on known relations. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qRNRBnVWCi", "reviewer": "Reviewer_wbgR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments.\n\nFor R1.\n\nA1: Following other OOS detection work, we compare our method with several common OOS detection methods. Comparing our method with some data generation methods (e.g., OodGAN) would indeed enrich this paper. We will investigate feasible data generation methods and add them to the revised version.\n\nFor R2.\n\nA2: Additional experiments, such as feature visualization, would indeed enrich this paper. Due to page limitations, we will consider adding them to the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors introduce OOS (OOD) problems in relation extraction, and show limitations of existing methods in terms of confidence and feature space distribution.\n- The authors propose a new method of generating OOS/OOD data of unknown type in the feature space (embeddings of encoder) from examples of known types\n- The authors compared the method to a few different baseline methods on the created datasets from FewRel and TACRED. They also provide the ablation study on different data generation methods and the amount of generated data. The empirical results are great for the proposed method and also provide insights of the problem for the baseline methods.\n", "reasons_to_reject": "- Although the authors compare to quite a few baselines, those methods are limited to confidence-based for output probabilities or distance-based for feature spaces. It would be worth comparing to a baseline that also formulates the problem as k+1 classification and uses synthetic/generated data for training (E.g., using GAN or other heuristics). This would give better estimation on whether the major contribution of the paper (data generation method) is a great or better method in generating OOS/OOD data.\n- The authors have a nice visualization on the limitations of probability-based and feature-based methods. It would be interesting to see whether the generated OOD data could be better distinguished in visualization. This would validate the intuition from the authors to mitigate the drawbacks of those methods. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors tackle the out-of-scope (OOS) / out-of-distribution (OOD) problem in relation extraction. They try to detect novel (unknown) relation types from known relation types. They first analyze the limitations of existing popular methods including probability-based methods and feature-based methods. They show the confidence distribution and feature space distribution of known and unknown relations are blended and hard to distinguish. They formulate the problem as (k+1) classification and explicitly predict whether an example is of unknown relation type as the k+1 class. Because we do not have training data for this k+1 class, the authors propose different strategies to generate these data from known types. They propose to generate the examples in the format of embeddings of encoder by combining two examples from known types in different ways: convex combination as linear interpolation of values of embeddings of two examples, regional replacement as randomly selecting a span in the embeddings of one example and combining with the rest of the span of the other example, irregular replacement as random selecting dimensions from two examples to construct the new example. By training on these synthetically generated data, the classifier is able to predict this k+1 unknown relation type significantly better than the baseline methods. The authors create the datasets for evaluation from existing relation datasets (FewRel and TACRED) by random sampling the existing relation types as unknown relation types and exclude these types in the training and validation data.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There exists methods on generating OOS/OOD data for this type of problem. It might be worth using these methods as baseline, or discuss the differences. E.g.,\nOodGAN: Generative Adversarial Network for Out-of-Domain Data Generation"}}]

Author rebuttal:

Meta-review: The reviews have a consensus on positive evaluations for both soundness and excitement.

Regarding excitement, the paper is the first paper that introduces the task of novel relation detection as an out-of-scope (OOS) detection problem.  The core three methods of pseudo-OOS data generation are largely inspired by prior work (as described in L557-560), and thus the novelty on this aspect is relatively weak.

With respect to soundness, the proposed method is well motivated by problems with existing OOS detection methods and well tested by a comparison with various OOS detection baselines on two benchmark datasets.  The ablation study shows the contribution by each of the three data generation methods.
Decision: Accept-Findings