Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs
Authors: Qing Wang
Kang Zhou
Qiao Qiao
Yuepei Li
Qi Li
Venue: EMNLP 2023 Main (2023)
Keywords: unsupervised relation extraction
relation representation learning
contrastive learning
TL;DR: We augment diverse positive pairs and enhance the discriminative power of contrastive learning for relation representations.

Abstract: Unsupervised relation extraction (URE) aims to extract relations between named entities from raw text without requiring manual annotations or pre-existing knowledge bases. 
In recent studies of URE, researchers put a notable emphasis on contrastive learning strategies for acquiring relation representations. However, these studies often overlook two important aspects: the inclusion of diverse positive pairs for contrastive learning and the exploration of appropriate loss functions. 
In this paper, we propose AugURE with both within-sentence pairs augmentation and augmentation through cross-sentence pairs extraction to increase the diversity of positive pairs and strengthen the discriminative power of contrastive learning. We also identify the limitation of noise-contrastive estimation (NCE) loss for relation representation learning and propose to apply margin loss for sentence pairs. Experiments on NYT-FB and TACRED datasets demonstrate that the proposed relation representation learning and a simple K-Means clustering achieves state-of-the-art performance.

Improving Unsupervised Relation Extraction by Augmenting Diverse Sentence Pairs
Qing Wang, Kang Zhou, Qiao Qiao, Yuepei Li, Qi Department of Computer Science, Iowa State University, Ames, Iowa, USA (qingwang, kangzhou, qqiaol, liyp0095, qlij@iastate.edu
Abstract
solving URE tasks. Recent studies on relation rep- resentation learning (Liu al., 2021; Zhang et al., 2021; Liu et al., 2022) adopt a contrastive learn- ing process. Contrastive learning aims to bring similar-relation sentences (positive pairs) closer while separating different-relation sentences (nega- tive pairs). There are several ideas for constructing positive pairs, key step in the contrastive learning process. Some methods propose augment the entity pairs and relation context of the original sen- tences as positive examples (Liu et al., 2021, 2022). To avoid similar sentences being pushed apart the contrastive learning process, Liu et al. (2022) further propose to utilize cluster centroids to gen- erate positive pairs within the clusters. However, these approaches often fail to provide diverse pos- itive pairs for contrastive learning obtain more effective relation representations.
Unsupervised relation extraction (URE) aims to extract relations between named entities from raw text without requiring manual annotations or pre-existing knowledge bases. In recent stud- ies of URE, researchers put notable emphasis on contrastive learning strategies for acquir- ing relation representations. However, these studies often overlook two important aspects: the inclusion of diverse positive pairs for con- trastive learning and the exploration of appro- priate loss functions. In this paper, pro- pose AugURE with both fithin-sentence pairs augmentation and augmentation through cross- sentence pairs extraction to increase the diver- sity of positive pairs and strengthen the dis- criminative power of contrastive learning. We also identify the limitation of noise-comtrastive estimation (NCE) loss for relation representa- tion learning and propose to apply margin loss for sentence pairs. Experiments on NYT-FB and TACRED datasets demonstrate that the pro- posed relation representation learning and simple K-Means clustering achieves state-of- the-art performance. Source code is available!.
Another challenge arises from the loss func- tion perspective. The most recent URE work (Liu et al., 2022) directly transfers the Noise- Contrastive Estimation (NCE) loss function from computer vision tasks (He et al., 2020; Caron et al., 2020; Li et al., 2021) and achieves good perfor- mance. However, the negative examples used by the NCE loss function randomly sampled sen- tences from the same minibatch, which may result in high false negatives, especially when the total number of relational clusters is not large. More- over, relation semantics should not be regarded as "same/differret" in many cases but rather as similarity spectrum. For example, the relation tooppmmmmmsss/mploeeee is semantically closer to org:members than org:founded _by. Therefore. argue that the binary NCE loss function may suit poorly for relation representation learning in URE tasks, whereas a margin- loss may fit better.
1 Introduction
Unsupervised Relation Extraction (URE) aims to extract relations between named entities from the raw text an unsupervised manner without relying on annotated relations from human experts. URE tasks are proposed to overcome the limitations of traditional Relation Extraction (RE) tasks, which target to extract pre-defined relations and require time-consuming and labor-intensive data annota- tion process.
To discover relational facts without any pre- defined relation types, URE tasks are often for- mulated as semantics clustering tasks, where the entity pairs with the same relations should be clus- tered together. Therefore, effectively learning the relation representations is an essential step toward
In this paper, we propose several techniques augment positive pairs to increase their diversity and strengthen the discriminative power of con-
Phttps://githie com/ iisiaannnnisAAuuURR

---

trastive learning, Specifically, we propose within- sentence pairs augmentation and augmentation through cross-sentence pairs extraction. Within- sentence pairs augmentation generates positive pairs from the same sentence using intermediate words sampling and entity pair replacement tech- niques. In the augmentation through cross-sentence pairs extraction, we aim to recognize sentences with the same relation semantics form more di- verse positive sample pool. Specifically, we employ a three-step approach. First, we utilize the Open Information Extraction (OpenIE) model (Angeli et al., 2015) to extract relation templates among all sentences. Intuitively, sentences with the same rela- tion template are likely to express the same relation. To further group the templates that share the same relation semantics, employ Natural Language Inference (NLI) (Zhuang et al., 2021) to discover mutually entailed templates. Finally, we extract positive pairs from different sentences within the same template groups To facilitate the extraction of more cross-sentence positive pairs, we further use ChatGPT to rewrite the original sentences. To overcome the limitations of the binary NCE loss function, propose to use margin loss cap- ture the spectrum of semantic similarity between positive and negative pairs.
decent performance with manual annotations and KBs, those annotations can be expensive obtain in practice, In recent years, few-shot RE and zero- shot RE (Sainz et al., 2021; Lu et al., 2022; Zhou et al., 2022) become popular, which significantly reduce the human annotation cost. However, few- shot RE and zero-shot RE still require pre-defined relation label space which is often unavailable in new datasets open-domain scenarios.
An upsurge of interest in URE has been wit- nessed to extract relational facts without pre- defined relation types and The task of OpenRE has also attracted considerable attention from researchers. Unlike URE, OpenRE has additional access to existing annotated relations and aims to extract novel relational facts from open- domain corpora. Relation representation learning is key step for both tasks. Amongst the prior studies, Tran et al. (2020) find that the utilization of named entities alone induce relation types can surpass the performance of previous methods. This observation underscores the significant induc- tive bias that entity types offer for URE. Later, Hu et al. (2020) assume the cluster assignments as pseudo-labels and propose a self-supervised Iearn- ing framework that leverages pre-trained language models (LMs) to learn and refine contextualized entity pair representations. However, the frequently re-initialized linear classification layers can hinder the process of representation learning. Wang et al. (2021) are the first to utilize deep metric learning in the OpenRE task and this scheme demonstrates good capabilities in representation learning.
data.
In summary, our main contributions are:
We propose AugURE with both within- sentence pairs augmentation and augmenta- tion through cross-sentence pairs extraction to increase the diversity of positive pairs and en- hance the discriminative power of contrastive learning for relation representations.
Motivated by the achievements of contrastive learning in computer vision tasks (He et al.. 2020; Caron et al., 2020; Li et al., 2021), recent state-of- the-art methods have embraced contrastive learn- ing frameworks acquire relation representations. Zhang et al. (2021) integrate existing hicrarchical information into relation representations by a con- trastive objective function. Liu et al. (2021) inter- vene on the named entities and context to obtain the underlying causal effects of them. Both methods adopt instance-wiss wise contrastive learning objectives and exploit base-level sentence differences by in- creasing the diversity of positive examples. To further capture high-level semantics in the relation features, Liu al. (2022) introduce hierarchical exemplar contrastive learning schema to optimize relation representations using both instance-win and exemplar wise signals. However, this approach
We identify the limitation of NCE loss for relation representation learning and propose to apply margin loss for sentence pairs.
Experiments demonstrate that the combina- tion of our relation representation learning and a straightforward K-Means clustering ap- proach achieves state-of-the-ard performance on NYT-FB and TACRED datasets.
2 Related Work
Relation Extraction is a crucial Natural Language Processing (NLP) task and has been extensively studied in the past years (Miller et al., 1998; Ze- lenko et al., 2002; Zhong and Chen, 2021; Sui et al., 2023). Although existing RE methods achieve

---

still lacks in providing diverse positive pairs for contrastive learning, limiting the effectiveness of relation representations. Additionally, it employs the NCE loss function without considering its align- ment with the task requirements.
Methodology
In this section, we provide a comprehensive explo- ration of the proposed model AugURE. Figure of- fers an overview of our framework. As mentioned in Section 1, both positive and negative examples are important for relation representation learning in URE tasks. When handling positive examples, we proposed several augmentation techniques to gen- erate more diverse positive pairs. When handling negative examples, we argue that relation seman- tics should not simply viewed as 'same/different but rather as a similarity spectrum. To this end, we propose to apply margin loss, which is also more robust to false negatives.
3 Preliminary
3.1 URE Task Definition
We formalize the URE task as follows. Let -0,, Fn] denote sentence, where z; represents the i-th (1 S i < n) token. In the sentence, a named entity pair (en, ex) is recognized in ad- vance, where C₁2 = [Ths ...: The] represents the head entity with its start and end indices denoted as hs and he, respectively, while e, = [its,... Viel represents the tail entity with its start and end indices denoted as iss and te, respectively. Let
4.1 Relation Encoder
Given a sentence, its corresponding named enti- ties, and entity types, the relation encoder aims to output vector that depicts the relation between the entities. To highlight entities of interest, we decide to follow the entity marker tokens presented in Soares al. (2019) and Xiao al. (2020) indicate the beginning and the end of each entity. Moreover, certain relations have been observed to exist between entities of specific types. Thus the entity type information is important guidance for the URE task and we should also incorporate the context of the entity type within each entity marker. sentence
D [x' en., corpus consists of N sentences along with their target entity pairs, the goal of URE to partition the N sentences into lt clusters, where & is a pre- defined number of relations in the corpus.
(x> een" N "h.",e e) be
3.2 Contrastive Learning
Recent studies have demonstrated the effectiveness of contrastive learning strategies to learn relation representations (Liu et al., 2021; Zhang et al., 2021; Liu et al., 2022). In these approaches, positive pairs are generally defined as two sentences with the same relation and negative pairs are two sentences with different relations. These positive and negative pairs are constructed and utilized in the contrastive objective function to bring together sentences with similar relations while separating sentences with dissimilar relations.
Specifically, for we inject into it two special tokens, <el:type> and </el:type>, respectively mark the beginning and the end of the head entity et. and another two special tokens, <e2:type> and </e2:type>, for the tail entity et. where "type" is replaced by the actual type of each entity. The resulting sequence of tokens becomes:
[21, .... Ch; ..., et,...,
Liu et al. (2022) achieve remarkable success by directly adopting the Noise-Contrastive Estimation (NCE) loss function from computer vision tasks URE's representation learning. The InfoNCE loss function (Oord et al., 2018; He al., 2020; et al., 2021), which is usually used for instance- wise contrastive learning, is defined as:
x |=1..... <el:type>, eh, </e1:type>, <e2:type>, et, </e2:type>, ..., vn).
(2)
Considering the remarkable performance of re- cent deep transformers, we feed into BERT base model made available by Devlin et al. (2019) to get the contextualized sentence encoding:
h [hi, ..., heel:npez, hiss .... bhe hele:ipeess". hce2:npes; hts...., fite, h> .... hn)-
exply: 1//T) LInJaNCE log E cap(us v///T)
(3)
(1)
Finally, to capture the relation contextual infor- mation and place more emphasis on the target entity pair, we derive the following fixed-length vector:
where i and u/ are positive embeddings for in- stance i, o/ includes one positive embedding and J negative embeddings for other instances, and is a temperature parameerr
hp = | / [h]]
(4)

---

Within-Sentence Pairs Augmentation
Loss Function
Sampling Random Positions
Mergin Pair
Transformer Encoder (BERT)
Entity Pair Augmentation
Datasets
K-Means Exemplar Loss
Augmentation through Cross-Sentence Pairs Extraction
Parr Cenerittio
Relation Templates Extraction
ChatcPi newnttion
Mutually Entailed Templates Discavery
Pairs Generation
Dample
When ned:PERSShh. Liingatoon eld. KAA New +/s2:00SATION-
By PERSOID Addison </ed PERSONo 1uLLOCTION </A21C
Figure The framework of AugURE.
to represent the relation expressed in between the marked entities, where alb represents the con- catenation of a and b.
only sample the non-stop words between the target entity pair (if there are less than m interme- diate non-stop words, sample the remaining words from other parts of the sentence). Let's denote the positions of m non-duplicated randomly sampled non-stop words as ......Pra. the ixed-length rela- tion feature vector we employ is:
4.2 Positive Sample Augmentation
As discussed in section previous methods fail provide diverse positive pairs for contrastive learning obtain more effective relation repre- sentations. In this section, we diversify the con- struction of positive pairs from two perspectives. Firstly, we augment the original sentences to gen- erate positive examples by sampling intermediate words and replacing entity pairs. Secondly, we ex- tract cross-sentence positive pairs by utilizing an OpenlEi model to extract relation templates of the sentences and employing NLI to discover mutually entailed templates.
hr [heelirype> /hee2 http:ppeee//pl..llpp....
(5)
By sampling he twice, we can obtain hy and an- other relation feature vector
h = [[ecel:tppes pheeetppsseeppe Weg (..!/t......p. (6)
where he,, ht, form positive pair. Because h, and h', contain the same entity pair, the representation learning may only rely on entity names and ignore the context information, limiting the representations' generalizability. To mitigate this issue and encourage the model to focus more on relation information, we propose to augment entity pairs. Because we only modify the head and tail entities, the relational context of the original sentence and the synthetic sentence x" remain the same. These same relation and the corresponding relation fea- ture vectors should be positive pair.
4.2.1
Pairs Augmentation
Augmented positive pairs from the same sentences are important to retain the local smoothness of the contrastive learning model since the original sen- tence and its perturbation have similar representa- tions. To increase the diversity of relation feature vectors generated from the same sentence æ and thus obtain more positive examples, we perform a random sample on the sentence encoding h portion of the vector ho (Liu et al., 2022). Specifically, uniform sampling. The sentences encoded first, and the sampling is conducted on the encod- ings of context words achieve the variations of the context encoding. To moderately retain relation context information, during the random sample,
sentences should have the
During the entity pair augmentation, the original en and e are replaced with another pair of entities e and e²", where en and ey have the same type, as do et and e/' respectively. This synthetic sentence x derives relation feature vector h"
h" [1" [h"ee-sype> 1 1122 |..
(7)

---

where < hr, h > form a positive pair.
NLI, also known as Textual Entailment, aims determine whether a "hypothesis" sentence is truc (entailment), false (contradiction), or undetermined (neutral) given a "premise" sentence. In this task, we aim to discover relation templates fp1 and tp2. both of which are verb phrases, that are mutually entailed using NLI.
4.2.2 Augmentation through Cross-Sentence Pairs Extraction
Leveraging the original sentences and their pertur- bations, the learning process constructs embed- ding space that primarily focuses preserving the local smoothness around each instance while largely ignoring the diverse global information in the dataset (Li al., 2021). Thus, relying solely on the instance- wise augmentations in section 4.2.1 is insufficient, and extracting additional high-quality cross-sentence positive pairs is essential.
To apply NLI model, we need to construct hy- pothesis sentences and premise sentences using relation templates and their covered sentences, re- spectively. Let's denote the set of sentences whose relation template tp1 as St. To eliminate the dis- turbance of the specific named entities, we generate a new set S1 as premises by replacing the head and tail entities of sentences in S1 with special tokens [h] and [t], respectively. We then synthesize sen- tence hypo2 in the form "[h]] (p2 [0] as a hypothe- sis, which is composed of the relation template tp2 and the special tokens [h] and [t] We call fp1 entails tp2 if for a certain ratio r of sentences in S entails the synthetic sentence hypoz.
We rely upon the following assumption as basis for extracting cross-sentence positive pairs: if the target entity pairs of two sentences have the same relation textual pattern (which we call relation template), these two entity pairs share the same underlying relation, and the corresponding relation feature vectors can be regarded as a positive pair.
To extract relation templates from the corpus without requiring any human input, we first employ Open Information Extraction (OpenIE) (Banko al., 2007) tool transform the sentences into structured formats. The OpenIE model receives the raw text of the sentence as input and outputs several (subject, predicate, object) triples. Specifically, we utilize the Stanford OpenlE model (Angeli et al.. 2015). For each sentence, we keep one valid triple if the predicate contains at least one non-stop word and the subject and object exactly match the given head and tail entities, respectively. With all valid extracted triples from the corpus, we count the fre- quency of distinct predicates and form high-quality relation templates to be those predicates whose fre- quency is greater than a threshold t. Finally, we enumerate the relation feature vectors of all pairs of sentences that share the same relation template to be positive pairs.
NLI predicts directional relation between two templates. To find mutually entailed relation tem- plates, for each pair of relation templates, we con- duct the above operations from both directions. Fi- nally, the relation feature vectors of all pairs of sentences whose relation templates are mutually entailed are enumerated to be positive pairs.
The recent development of ChatGPT and its summarization power provide the possibility of capturing higher-level relations of the sentences, which can further assist in discovering more cross- sentence positive pairs. We design a simple prompt "Given the context æ, what is the relationship be- tween eh and et (as short as possible)?" to rewrite the original sentence with the target entities eh and et, Given the prompt of sentence æ. ChatGPT will generate an output and this output denoted the rewritten sentence Te. Note that the rewritten sentence ae semantically similar the original sentence æ but may not have the same level of gran- ularity in terms of the relation as æ. For example, an original sentence "Paul is senior research associate ... Library" can be rewritten into "Paul... works for ... Library".
Extracting same-template sentence pairs can cap- ture global information and diversify positive pairs, but may be strict since different relation tem- plates can express the same semantics meaning. For instance, "is CEO of" and "is the chief execu- tive officer of are two distinct relation templates but convey the same relation.
Inspired by the recent achievements of Natural Language Inference (NLI) to obtain indirect super- vision in RE tasks (Sainz et al., 2021; Zhou al., 2022), we further propose to employ an off-the- shelf NLI model (Zhuang et al., 2021) to discover templates that have the same semantic relations.
To not disrupt the original corpus's data distribu- tion, we will not use the rewritten sentence positive pairs directly in training. Instead, we repeat the above template extraction and NI. template group- ing process on the rewritten sentences and then identify sentences from the same template groups.

---

The corresponding original sentence pairs can be used as additional positive pairs. Note that the ChatGPT-discovered cross-sentence positive pairs can be numerous and potentially contain a higher level of noise. To address this, we employ random sampling to select a subset of these pairs before incorporating them into the training process.
Ppair is defined as:
Paair Lw L
M
Σ
max dist(ar' .pr') (8)
M
8=1
dist(a, , n,,')) 03,
where dist(() represents the cosine distance func- tion, n, is a randomly sampled negative example for ar and the parameter 7, referred to as the margin, is hyperparameter.
4.3 Representation Learning and Clustering
4.3.1 K-Means Exemplar
In this work, we mainly focus on developing ef- fective relation representations for URE tasks. For relation clustering, we adopt simple K-Means clustering algorithm. Following the idea of Liu al. (2022), we implement cluster centroids of different & values different granularities of rela- tional exemplars. These exemplars are subject to fluctuations accordance with the parameters up- date of the relation encoder in each training epoch.
For the K-Means exemplar, our objective is to en- courage sentences belonging to the current cluster to be closer to their centroid, while simultaneously pushing sentences outside the cluster away from the current centroid. Since a sentence either be- longs to a cluster or not, this is a binary relation and we use the same Exem loss as Liu et al. (2022):
N
L Σ
log exp(h", e5yTT) 1=1 Σ exp(h' * e//T)
L Farem
4.3.2 Loss Function
(9)
where j E [1, c] represents the j-th cluster gran- ularity layer I, e is relation feature vector of the exemplar of instance i at layer I, and J is is temperature hyperparameter (Wu al., 2018).
For simplicity, let Pw denote the set of within- sentence positive pairs, Pe denote the set of cross- sentence positive pairs, and Prair = Pw U UPP ((,,', Pr ", (a, M. p. M)] be the set of relation vectors of M positive pairs from the training cor- pus, where (ar", "p,") denotes the pair of relation vectors of the i-th positive sample.
Our overall loss function is defined as the addi- tion of Pair loss P and Exem loss Earem:
L. = Pair + E L fee + L. Exee- (10)
Liu et al. (2022) randomly sample sentences from the same minibatch as the negative samples used by the NCE loss. However, this approach may be accompanied by potential issues. When the total number of relational clusters is small, employing this approach is likely yield substantial number false negatives. Furthermore, relation semantics should not be regarded as "same/different" in many cases but rather as a similarity spectrum. When considering two pairs of semantically distinct rela- tions, it is possible for one pair to exhibit higher degree of semantic proximity compared to the other pair. Hence, believe the NCE loss function is not suited for relation representation learning.
During inference, encode each sentence us- ing the trained model to obtain its representation. Subsequently, the K-Means clustering algorithm is applied to cluster the relation representations, en- abling us to predict the relation for each sentence based on the clustering results.
5 Experiments
5.1 Datasets
Following Liu et al. (2022), we adopt NYT-FB 2 and TACRED (Zhang et al., 2017) datasets to train and evaluate our model. NYT-FB is a dataset generated via distant supervision that combines in- formation from two sources: The New York Times (NYT) articles and the Freebase knowledge graph (Bollacker et al., 2008). We filter out sentences with non-binary relations the same as Hu et al.
We propose to apply margin loss for sentence pairs. Margin loss is more robust to noise in the training data, especially the false negative issues. Moreover, captures the spectrum property of the differences between positive and negative pairs in- stead of treating them solely as binary distinctions.
hhttps://githuf wtwp:tthu..co/THuuS. www.THHBBBW/HUUEEE/////E http///////eeeeeeeee////////// data www.ssaplefoo__eeeeeeee
The loss function for the set of sentence pairs
htttps: htttps::/nll... stanford. htwpa///oooss.oooodoo.ooo

---

Adjusted Rand Index (ARI) measures the agree- ment degree between the clusters assigned by a model and the ground truth clusters. It ranges in (2020), Tran al. (2020), and Liu al. (2022). Tran et al. (2020) raise additional concerns that emerge when employing the NYT-FB dataset. The development and test sets of the NYT-FB dataset consist entirely of aligned sentences without any human curation and include tremendous noisy la- bels. In particular, about 40 out of 100 randomly sampled sentences were given incorrect relation labels. Also, the development and test sets part of the training set, making it challenging accu- rately evaluate model's performance on unseen sentences, even under the URE setting. Hence, supplementary evaluations are carried out on the TACRED dataset, which is a large-scale supervised relation extraction dataset built via crowdsourcing. By conducting additional evaluations on the TA- CRED dataset, we can attain more precise com- prehension of the performance of all the models.
ness of different methods. For all these metrics, the higher the better.
Bd precision and recall measure the quality and coverage of relation clustering, respectively. B FI score is computed to provide a balanced clustering performance evaluation.
V-measure is another evaluation metric com- monly used for assessing clustering quality. While B treats each sentence individually, V-measure considers both the intra-cluster homogeneity and inter-cluster completeness, providing more com- prehensive evaluation of clustering performance by considering the entire clustering structure.
[- 1,,1], 1], where value close to indicates strong agreement, O represents random clustering, and negative values indicate disagreement.
5.2 Baselines
We compare the proposed model with the follow- ing baseline methods: 1) EType (Tran et al., 2020) is a straightforward relation classifier that employs a one-layer weed-forward network. It uses entity type combinations as input to infer relation types. 2) SelFORE (Hu et al., 2020) leverages weak, self- supervised signals from BERT for adaptive clus- tering and bootstraps these signals by improving contextualized features in relation classification. 3) MORE (Wang et al., 2021) utilizes deep metric learning extract rich supervision signals from labeled data and drive the neural model to learn re- lational representations directly. 4) OHRE (Zhang et al., 2021) effectively integrates hierarchy infor- mation into relation representations via dynamic hierarchical triplet objective and hierarchical cur- riculum training paradigm. 5) EIURE (Liu et al., 2021) intervenes on both the entities and context uncover the underlying causal effects of them. 6) HiURE (Liu et al., 2022) the state-of-the-art method that derives hierarchical signals from rela- tional feature space and effectively optimizes re- lation representations through exemplar-win con- trastive learning. To ensure fair comparisons, use the HiURE results with the K- K-Means clustering algorithm as reported in the original paper.
5,4 Setup
To primarily show the enhancement of relation rep- resentations, we choose the simple K-Means as the downstream clustering algorithm. Following pre- vious work (Simon et al., 2019; Hu et al., 2020; Liu al., 2021, 2022), the number of clusters k set 10. Despite the fact that the ground truth relations have more than 10 classes, this choice still provides valuable insights into the models' capa- bility to handle skewed distributions. Moreover, this enables us to conduct fair comparisons with the baseline results. More implementation details can be found in Appendix A.
5.5 Main Results
We conduct experiments on both NYT-FB and TA- CRED datasets, with the average performance and standard deviation of three random runs reported. The main results are shown in Table 1. We ob- serve that the proposed AugURE outperforms all baseline models consistently B F1, V-measure Fl, and ARI metrics. On the NYT-FB dataset, the proposed AugURE on average improves the 3 Fl by 0.6%, V-measure Fl by 4.3%, and ARI by 2.8% compared with the runner-up results; on the TACRED dataset, the improvements are more evident, with the 3 FI, V-measure FI, and ART increased by 2.4%, 1.9%, 12.4%, respectively com- pared with the rumner-up results.
5.3 Evaluation Metries
Following previous work, we use B (Bagga and Baldwin, 1998), V-measure (Rosenberg and Hirschberg, 2007), and Adjusted Rand Index (ARI) (Hubert and Arabie, 1985) to evaluate the effective-
The aforementioned performance gains show that the proposed relation representation learning

---

B
V-measure
External Tooks
Dataset
Method
Prec.
Rec.
F1
Hom.
Comp.
F1
and Supervision Nane Parameters of SDA
Elype (Tran al., 2020)
31.342.1
63.7:20
41.9120
31.22.
56.21188 40.6
3..741.9
SelfRR (Hu et 2020)
38.54-2.2
44.7+118
1,,4+1.9
378424
43. 40.4-1.7
35.0+-2.0
MORE (Wang al., 2021)
43.811.9
40.3:2.0
42.022.2
40.8222
43.1424
419--211
35.6:2..1
None
NYT-FB
OHRE (Zhang al., 2021)
32.71:1.8
60.7:22.
42.5
3488221
53.922.5
42.3-1.8
33.62:1.8
None
EIURE (Lio al, 2021)
48.4+119
38 88
43.1+18
32.7+15
49,241.6
4277-16
34.5++144
WikiData, WehNLG
Meaas (Liu al., 2022)
38.74:1.0
44.3:0.9
41.4
37.24111
47.040.8
41.5-13
34.3400.9
Nane
AugURRE (our)
35.74-0.8
56.5+188
43.7+1.1
40.3+00.6
56.3+
47.0
0.8
18.4+1.66
OpenIE,, NLJ, CharGPT
EType (Tran nl. 2020)
51.942.1
47.0=188
49.3=119
52.52211
54.4+19
53.022
15,742.1
None
SelfRR (Hu et 2020)
51.61:200
44.2:119
4..6
51.32:00
$2.9422.3
52.1
36.1
Parameters of
MORE (Wang et al., 2021)
56.94:222
44.9=118
50.2+1188
56.7
58.142.3
57.4=21
1,9
Nane
TACRED
OHRE (Zhang al.
55.24:2.1
48.721.
518116
55.5219
57.342.1
L6
38.041177
None
ETURE (1 al, 2021)
57,4+1.3
47.8+15
52.2+114
57.7=144
59.741.7
58.7-12
38.6+-1.1
WikiData, T3, WehNT.G
HIURE
K-Means (Liu al,, 2022)
50.3+008
46.7+12
48.4+0.9
818
6.224115
58.1=111
37.8+0.8
None
AugURE (our)
51.34:088
5844339 54.61.2.2
16.91111
64.9122.4
60.6 1.7 51.0:0.66
OpenlE, NLJ, CharGPT
Table 1: Performance of all methods NYT-FB and TACRED datasets.
can effectively capture the relation features for bet- ter relation clustering. Furthermore, entity annota- tions can influence the quality of relation represen- tations. The performance improvements of the pro- posed AugURE are consistent in both the distantly- annotated NYT-FB dataset and the crowdsoureing TACRED dataset, indicating that the proposed rela- tion representation learning is robust noisy entity labels and relations in different domains.
It is interesting that removing the set Pur on the TACRED dataset leads to a performance decrease but does not show apparent influences on the NYT- FB dataset. These findings show that the proposed AugURE can learn sufficient information from the diverse cross-sentence pairs, highlighting their cru- cial role in enhancing the model's performance.
5.6.2 Effectiveness of Augmentation through Cross-Sentence Pairs
It is worth mentioning that despite utilizing K- Means as the clustering algorithm, our model's performance surpasses that of both the SelfORE and OHRE, which employ more advanced clus- tering techniques. We believe that improving the clustering algorithm would yield advantages to the overall URE performance, and we leave for fu- ture exploration.
To evaluate the impact of augmentation through cross-sentence pairs, two experiments are con- ducted. Initially, we exclude positive pairs ex- tracted based on ChatGPT's guidance to examine their impact (referred to as "- ChatGPT Pairs"). Subsequently, the entire set of cross-sentence posi- tive pairs P. is excluded (referred to "- Cross- Sentence Pairs"). The results demonstrate that the impact of ChatGPT can vary across different datasets, indicating a dataset-dependent nature of its influence. This observation is reasonable since the granularity of relations across different datasets can differ, resulting in varying gaps between the original sentences and the rewritten sentences. Excluding the set Pe leads to significant de- clines in overall performance across all metrics on both datasets. This finding highlights the ef- fectiveness of the proposed augmentation through cross-sentence pairs in enhancing diversity and at- taining superior results.
5.6 Ablation Study
To investigate the contribution of different com- ponents, we further conduct an ablation study by systematically excluding or replacing specific com- ponents, and the results are shown Table 2.
5.6.1 Effectiveness of Within-Senteni Pairs Augmentation
To assess the impact of the within-sentene pairs augmentation, two experiments are conducted: I) removing entity-augmented positive pairs from the training set (referred to as Entity Augmenta- tion"); 2) removing the entire set of within-sentence positive pairs Pw from the training set (referred to as Within-Sentence Pairs"). The results re- veal that removing augmentd positive pairs leads to notable decreases in all metrics. This indi- cates that without entity augmentation, the learned relation representations may rely more on matching entity names instead of context information, posing limitations on generalizability.
5.6.3 Margin Loss vs. NCE Loss
To study the advantage of margin loss compared with NCE loss, we change the within-sentence pos- itive pairs' margin loss Lw, in Eq. (10) to be the NCE loss defined in Eq. (1). As shown in Table 2, replacing margin loss with NCE loss leads notable decreases in all metrics on both datasets.
This result implics that the NCE loss function suits poorly for relation representation learning in URE

---

B'
V-measure
Dataset
Method
ARI
Rec.
Hom.
Comp.
Full Model
3..7=
1..51:188
43.7+1.1
40.3::0.6
56.3=1.2
47.0E0.8
38.411.6
Entity Augmentation
33.9-0.9
51.4+3388
40.8+1.8
37.7+112
52.0=2.1
43.7+
35.3+33.
NYT-FB
Pairs
3.35=003
1..5
44.0:10.7
40.5
5..5
47.21111
36,9100.5
ChatGPT Pairs
35.4-0.8
12.0+
42.1+0.8
3..3+1.3
53.0=1144
45.1+
34.9+1.4
Cross-Soe Sross-Senlence Pairs
34.8=1.7
47.112,9
40.0
37.9114
5..2=1.9
43.24:..6
26.342.8
replaced with NCE loss
33,8=1.5
1..4
39.1
37.2:008
49.0=
42.34:0.9
30,24:2.2
Full Model
51.3=0.8
58,41399
54.64222
56,9:1111
64.922.4
60.6
51.0+0,66
Entity Augmentation
49.0-0.3
13.0:1199
1..9+
53.5++0.6
60.3+1.5
16.7+1.0
47.3++2.1
TACRED
Within-Seeneenee Pairs
50.6-0.9
55.3+-4.5
52.8+2.6
55.4+115
62.0=2.9
58.5+-211
45.6+3.3
ChatGPT Pairs
51.7=0.4
19
55.5:10.6
57.5=0.88
6..8=0.5
61.4:0.6
49.0:16.6
Cross-Seds Sentence Pairs
49.2-1.2
54.3++03
51.6+0.6
54.3+14
61.5+007
57.7+
48.5
replaced with NCE loss
50.1=0.5
55.8+2.8
1.
54.3+1.1
61.6+199
57.7+1.4
49.1
Table 2: Ablation Study.
tasks, highlighting the contribution of margin loss to the overall performance.
Total
Correct
same-template pairs
54457
39882 (73.2%)
# mutually entailed pairs
216
137 (63.4%)
5.7 Evaluations of the OpenIE and NLI Performance
Table 3:
The accuracy of OpenIE and NLI generated
pairs on NYT-FB dataset.
We also evaluate the OpenlE and NLI performance on the NYT-FB dataset, Results are summarized in Table and Table 4, with further analyses presented the subsequent paragraphs.
NYT-FB
# original templates
1210
average sentences per template
2.75
total # sentences
10000
# sentences covered by templates
3322 (33.2%)
Regarding accuracy, among the 54457 same- template pairs, 73.2% of them are correct (belong the same relation based on the ground truth la- bels). We randomly sampled and examined 30 of the 'incorrect' pairs and found that 90.0% of these pairs convey relationships similar enough to be regarded as positive pairs for the URE task. Among the 216 mutually entailed pairs based on NLI predictions, 63.4% of them are correct. We also randomly sampled 30 of the 'incorrect' pairs and found that 76.7% of them can be viewed as positive pairs.
Table 4:
The coverage of OpenIE and NLI extracted templates on NYT-FB dataset.
Conclusion
In conclusion, this paper offers an exploration URE from the perspective of relation representa- tion learning. We introduce two novel augmenta- tion techniques: within-sentence pairs augmenta- tion and augmentation through cross-sentence pairs extraction, both of which aim to augment positive pairs, enhancing their diversity and bolstering the discriminative capabilities of contrastive learning. We also identify the limitations of NCE loss URE tasks and propose margin loss for sentence pairs. Experiments on two datasets show the su- periority of the proposed AugURE compared to competitive baselines, highlighting the efficacy of the augmentation techniques and the significance of employing margin loss. Overall, our findings ad- vance the understanding of relation representation learning and provide valuable insights for future research and practical applications.
In terms of coverage, the total number of ex- tracted relation templates is 1210, and each relation template, on average, consists of 2.75 sentences. Only 33.2% of the sentences are covered by the ex- tracted templates. That is, 66.8% of the sentences do not follow any relation template. Furthermore, the total number of extracted templates much more than the number of target clusters for the URE task.
Consequently, the OpenIE and NLI method has high precision and is suitable to be used to gener- ate high-quality positive pairs to enhance the con- trastive learning process. The OpenlE and NLI method also suffers from low recall, which means it is unsuitable to directly serve as a baseline for the URE task.
Limitations
Limitation of K-Means Clustering Algorithm The proposed AugURE adopts the K-Means clus-

---

tering algorithm, which requires a pre-defined num- ber of relations I as input. However, determining the appropriate number can be challenging and sub- jective. Moreover, K-Means assumes that clusters are spherical and have a similar size. This assump- tion may not hold true for datasets with skewed distributions, leading to suboptimal clustering re- sults.
Conference on Artificial Intelligence, Hyderabad, In- dia, January 6-12, 2007. pages 2670-2676.
Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008, Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIG- MOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008, pages 1247-1250. ACM.
Limitation of Relation Template Extraction
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrast- ing cluster assignments. In Advances in Neural In- formation Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
The inherent nature of the relation template extrac- tion process in the proposed AugURE may exhibit a bias towards larger and more frequently occurring relation types in the presence of skewed relation distributions. This bias may result in a decrease in performance when dealing with long-tail relations.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL- 2019, Minneupolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4188 Association for Computational Linguistics.
Limitation of ChatGPT Prompt ChatGPT is known to be sensitive to various prompts. In this work, our focus is primarily on the design of a functional ChatGPT prompt. We did not explore other strategies for sentence rewriting that may pre- serve the same level of granularity the original sentence.
Ethics Statement
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In 2020 IEEE/CVF Conference Computer Vision and Pat- tern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726-9735. Computer Vi- sion Foundation IEBE.
We comply with the EMNLP Code of Ethics.
Acknowledgments
The work was supported in part by the US National Science Foundation under grant NSF-CAREER 2237831. We also want to thank the anonymous reviewers for their helpful comments.
Xuming Hu, Lijie Wen, Yusong Xu, Chenwei Zhang, and Philip S. Yu. 2020. Selfore: Self-supervised relational feature learning for open relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3673- 3682. Association for Computational Linguistics.
References
Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguis- tic structure for open domain information extraction. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the Natural Lan-
Lawrence J. Hubert and Phipps Arabie. 1985. Compar- ing partitions, Journal of Classification, 2:193-218.
Junnan Li. Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. 2021. Prototypical contrastive learning of unsu- pervised representations. In 9th International Confer- ence on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net
7th International Joint Conference guage Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 344 354. The Association for Computer Linguistics.
Amit Bagga and Breck Baldwin. 1998. Entity-based cross-document coreferencing using the vector space model. In Annual Meeting of the Association for Computational Linguistics.
Fangchao Liu, Lingyong Yan, Hongyu Lin, Xianpei Han, and Le Sun. 2021. Element intervention for open relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Longuage Processing, ACI/IJCNIP 2021, (Volume I: Long Papers), Virtual Event, Au- gust 1-6, 2021, pages 4683-4695 Association for Computational Linguistics.
Michele Banko, Michael J. Cafarella, Stephen Soder- land, Matthew Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In IJCAI 2007, Proceedings of the 20th International Joint

---

Shuliang Liu, Xuming Hu, Chenwei Zhang, Shuang Li, Lijie Wen, and Philip S. Yu. 2022. Hiure: Hier- archical exemplar contrastive learning for unsuper- vised relation extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL. 2022, Seattle, WA, United States, July 10-15, 2022, pages 5970-5980. Association for Computational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald. Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics. ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Pa- pers, pages 2895-2905. Association for Computa- tional Linguisties.
Dianbo Sui, Xiangrong Zeng. Yubo Chen, Kang Liu, and Jun Zhao. 2023. Joint entity and relation extrac- tion with set prediction networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1-12.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019. New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Thy Thy Tran, Phong Le, and Sophia Ananiadou. 2020. Revisiting unsupervised relation extraction. In Pro- ceedings of the 58th Annual Meeting of the Associa- rion for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pages 7498-7505. Association for Computational Linguistics.
Keming Lu. I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2022. Summarization indirect supervision for relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3575-6594, Association for Computational Linguistics.
Yutong Wang. Renze Lou, Kai Zhang, Mao Yan Chen, and Yujiu Yang. 2021. More: metric learning based framework for open-domain relation extrac- tion. In IEEE International Conference on Acous- tics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pages 7698- 7702. IEEE.
Scott Miller, Michael Crystal, Heidi Fox, Lance A. Ramshaw, Richard M. Schwartz, Rebecca Stone, and Ralph M. Weischedel. 1998. BBN: description of the SIFT system used for MUC-7. In Seventh Message Understanding Conference: Proceedings of a Conference Held in Fairfax, Virginia, USA, MUC 1998, April 29- May 1998. ACL.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. 2018. Unsupervised feature learning via non- parametric instance discrimination. In 2018 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2018, Sali Lake City, UT, USA, June 18-22, 2018, pages 3733-3742. Computer Vision Foundation / IEEE Computer Society.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. 2018, Representation learning with contrastive predictive coding. CoRR, abs/1807.03748.
Andrew Rosenberg and Julia Hirschberg. 2007. V-
Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020. Denoising relation extraction from document- level distant supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3683-3688, Association for Com- putational Linguistics.
measure: conditional entropy- based external clus- ter evaluation measure. In EMNLP- CoNLL 2007, Proceedings of the 2007 Joint Conference on Empir- ical Methods in Natural Language Processing and Computational Natural Language Learning, June 28- 30, 2007, Prague, Czech Republic, pages 410-420. ACL.
Oscar Sainz, Oier Lopez de Lacalle. Gorka Labaka, Ander Barrena, and Eneko Agirre. 2021. Label verbalization and entailment for effective zero and few-shot relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic. 7-11 November, 2021, pages 1199-1212, Association for Computa- tional Linguistics.
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2002. Kernel methods for relation ex- traction. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing, EMNLP 2002, Philadelphia, PA, USA, July 6-7, 2002. pages 71-78.
Kai Zhang, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2021. Open hierarchical relation extraction. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages tion for Computational Linguistics.
Étienne Simon, Vincent Guigue, and Benjamin Pi- wowarski. 2019. Unsupervised information extrac- tion: Regularizing discriminative approaches with re- lation distribution losses. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers. pages 1378-1388 Association for Computational Linguistics.
Associa-
Yuhao Zhang, Victor Zhong. Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware

---

attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Demmark, September 9-11, 2017, pages 35-45. Association for Computational Linguis- tics.
A Implementation Details
All experiments are conducted on two NVIDIA Tesla V100 SXM2 32 GB GPUs. In the training period, AdamW (Loshchilov and Hutter, 2019) is used to optimize the loss. The encoder is trained for 5 epochs with a le 5 learning rate. The num- ber of randomly sampled non-stop words m is set to 2. The template frequency threshold & for the NYT-FB dataset and for the TACRED dataset. The ratio r determining template entail- ment is set to 0.95. We use the development sets to grid search the margin hyperparameter from [0.25, 0.5, 0.75, 1.0] and find 0.75 is the optimal. When using the NCE loss function, set the tem- perature parameter
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. NAACL-HIL 2021, Online, June 6-11, 2021, pages 50-61. Association for Computational Linguistics.
Kang Zhou, Qiao Qiao, Yuepei Li, and Qi Li. 2022. Improving distantly supervised relation extraction by natural language inference. In The Thirty-Sixth AAAI Conference Artificial Intelligence, AAAI 2022.
0.02 and the number of negative examples J 10.
Liu Zhuang, Lin Wayne. Shi Ya, and Zhao Jun. 2021. robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-12111 1227.
B Statisties of Augmentation through Cross-Sentence Pairs Extraction
As introduced in Section 4.2.2, the set of cross- sentence positive pairs Pe be divided into three categories: 1) same-template original positive pairs; 2) mutually entailed templates original posi- tive pairs: 3) positive pairs extracted based on Chat- GPT's guidance. Table 5 presents a comprehensive overview of the extracted relation templates from both original sentences and ChatGPT-rewringg sen- tences. The table includes the number of templates for the NYT-FB dataset and the TACRED dataset, as well statistics for the three categories of cross- sentence positive pairs.
NYT-FB
TACRED
# original templates
1210
266
# rewritten templates
633
259
# same-template pairs
54457
3876
mutually entailed pairs
216
1318
ChatGPT pairs
632188
57151
Table 5: The statistics of relation templates and cross- sentence positive pairs NYT-FB and TACRED datasets.


Reviews:
[{"review_id": "Uoby7FdZjW", "reviewer": "Reviewer_Ydii", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work.\n1. We aim to increase the context diversity using the augmentation technique. Under the unsupervised setting, it is non-trivial because the augmentation results need to have high precision to avoid introducing noise and semantic drifting issues. The proposed augmentation methods follow one simple assumption: if two sentences share the same or semantically similar relation templates, they express the same relation. \n2. The goal of data augmentation using ChatGPT is to transform each original sentence (denoted as $x$) into a rewritten sentence $x_c$, which retains the identical underlying relationship as $x$ but in different contexts. We have tried several different ChatGPT prompts including “Given the context $x$, what is the relationship between $e_h$ and $e_t$?”, “Given the context $x$, what is the relationship between $e_h$ and $e_t$ (use less than 15 words)?”, and “Given the context $x$, what is the relationship between $e_h$ and $e_t$ (as short as possible)?”    \nThe first prompt generated paragraphs consisting of multiple sentences with many additional details, and thus cannot be used for our purposes. The second and third prompts generated a single sentence, but we regrettably found that they tended to change the granularity of the underlying relationship. One example is described in L406-408: an original sentence “Paul ... is a senior research associate ... Library” can be rewritten into “Paul ... works for ... Library”. Therefore, we used pattern extraction and NLI techniques on the rewritten sentences instead of using $(x, x_c)$ as a positive pair.    \nIn the paper’s Limitations section, we have acknowledged the possibility of unexplored strategies that could unlock an improved performance of data augmentation with ChatGPT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is interesting to identify the limitations of NCE loss in unsupervised relation extraction tasks. It provides insight into how to use contrastive loss in similar situations. Experiments also show a significant improvement by using margin loss.", "reasons_to_reject": "1. Data augmentation methods seem a bit ad-hoc. \n2. I think data augmentation can be done better with ChatGPT. Have you tried some simple methods like different prompts? ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes two methods for unsupervised relation extraction. One is data augmentation and the other is margin loss for sentence pairs in contrastive learning. The authors disclose that the traditional NCE loss function is not suitable for relation representation learning. Therefore, they propose to use a margin loss for sentence pairs. Experiments show that the proposed methods are effective.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BPCTafGzpU", "reviewer": "Reviewer_PHta", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your reviews and comments.\n1. As mentioned in the Introduction section, both positive and negative examples are important for relation representation learning in URE tasks. When handling positive examples, we proposed several augmentation techniques to generate more diverse positive pairs. When handling negative examples, we argue that relation semantics should not be simply viewed as “same/different” but rather as a similarity spectrum. To this end, we propose to apply margin loss, which is also more robust to false negatives.   \n2. The data augmentation is part of the proposed method, and the ablation results of removing each augmentation technique are shown in Table 2. The same technique can be applied using different OpenIE methods, different NLI methods, or different text summarization methods. Comparison of external tools is beyond the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good insights on positive pair sampling and loss function exploring, which is often overlook by previous models and may provide different ways for future work.", "reasons_to_reject": "1.\tThe contributions of generating positive samples and exploring loss function are not intuitively related, which makes this paper looks like a patchwork.\n2.\tSince the positive samples are generated with extra models (and annotations), the proposed method is hard to fairly compare with baseline methods. Moreover, at least the comparison of external tools and supervision should be added in Table 1.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the Unsupervised Relation Extraction task. The authors focus on generating positive samples and exploring the loss function for contrastive learning. In positive sample generating, this paper proposes to generate within-sentence samples with random sampling and entity replacement, and also to find cross-sentence samples using OpenIE, NLI and ChatGPT. This paper also proposes to apply margin loss to sentence pairs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pYzaCs9fZP", "reviewer": "Reviewer_TtUf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews and suggestions.\n1. Regarding the single instance augmentation, we think that you are referring to positive pairs described in L275-296.   \nThis positive pair generation is a common practice used in previous work, and previous studies show that it helps to retain the local smoothness of the model (Liu et al., 2022). Therefore, we also include this procedure in the proposed method.\n2. We evaluate the OpenIE and NLI performances on the NYT-FB dataset.   \nRegarding accuracy, among the 54457 same-template pairs, 73.2% of them are correct (belong to the same relation based on the ground truth labels). We randomly sampled and examined 30 of the “incorrect” pairs and found that 90.0% of these pairs convey relationships similar enough to be regarded as positive pairs for the URE task. Among the 216 mutually entailed pairs based on NLI predictions, 63.4% of them are correct. We also randomly sampled 30 of the “incorrect” pairs and found that 76.7% of them can be viewed as positive pairs.    \nIn terms of coverage, the total number of extracted relation templates is 1210, and each relation template, on average, consists of 2.75 sentences. Only 33.2% of the sentences are covered by the extracted templates. That is, 66.8% of the sentences do not follow any relation template. Furthermore, the total number of extracted templates is much more than the number of target clusters for the URE task.    \nConsequently, the OpenIE and NLI method has high precision yet suffers from low recall, making it unsuitable to directly serve as a baseline. We will add the details to the final paper.\n3. As shown in paper L406-408, with the prompt and an original sentence “Paul ... is a senior research associate ... Library”, ChatGPT gives an output “Paul ... works for ... Library”. The output sentence is regarded as the rewritten sentence, which has the same pair of entities as the original sentence and a ChatGPT-summarized relation.\n4. The statistics of the cross-sentence pairs are included in Appendix B. We also carefully evaluate the contributions of each type of pairs. Please refer to Section 5.6 and Table 2. \n5. Previous URE methods generate the positive pairs mostly using within-sentence augmentation techniques described in Section 4.2.1, while we additionally utilize relation templates and NLI to extract cross-sentence positive pairs. These cross-sentence positive pairs express the same relationship but have different entities and contexts.   \nFor example, “When \\<e1:PERSON\\> Livingston </e1:PERSON> came to \\<e2:LOCATION\\> New York </e2:LOCATION> in 1999 , she stayed with an uncle in the Bedford-Stuyvesant section of Brooklyn , near Boys and Girls , which has a large Caribbean population in its student body of 3,500 .” and “By the time \\<e1:PERSON\\> Addison </e1:PERSON> moved to \\<e2:LOCATION\\> New York </e2:LOCATION> , '' Seebohm explains , '' he was 33 years old , and seen in hindsight these romantic failures suggest that his heterosexuality was beginning to falter .” form a cross-sentence positive pair (from NYT-FB).  \nThe encodings of the two sentences contain more discriminative information and our method creates more diverse positive pairs.\n6. Thank you for the suggestion. We will include a framework chart in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* An interesting set of data augmentation methods that create similar pairs of instances, useful to learning representations.\n  * Distant supervision is based loosely on notions of entailment and building simple propositions from semantic templates.", "reasons_to_reject": "While the method shows good results compared to baselines, and there are sufficient ablation studies, I found the method description and overall analysis to be lacking and not convincing.\n\n1. Regarding the single instance augmentation: it is not clear what distribution words are resampled from and whether the augmented sentences remain grammatical or even make sense, or do they still carry the same relation type as their source sentence.\n\n2. Regarding the paired sentences found through an OpenIE and NLI: while the approach is reasonable and interesting, there is no discussion or analysis of what kind of examples or relations it judges as similar, and no analysis of the mistakes it makes or what it misses. Can this OpenIE and NLI method already serve as a baseline method even without learning the underlying representations?  I believe this part is the most interesting part of the work and that the reader would be served better if they can delve deeper. Unfortunately, there is not even an example of the resulting sentence pairs let alone an error analysis.\n\n3. The manuscript is very confusing regarding the instances created by ChatGPT. The prompt that is discussed in the paper simply requests a short description of the relation in the sentence, while the authors claim that ChatGPT has written a new and different example sentence.\n4. There is no breakdown of the final augmented dataset. How many pairs of each type were there? It is unclear what contributes to the final augmented dataset. \n5. The paper claims it created a more diverse set of positive (similar) instance pairs. This claim is not backed sufficiently as there is no suggested measure of diversity and no comparison with earlier works regarding the diversity of examples.\n6. This manuscript lacks any example or chart that could guide the reader and explain qualitatively their method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a new method for the task of unsupervised relation extraction. In this formulation, an instance consists of a sentence and two marked related entities that are absent of the relation label. The task then deals with clustering a set of instances into clusters of similar relation types. The authors propose a distant supervision method and a margin-based loss function to learn latent representations of instances. After which, it is possible to apply a simple clustering technique such as k-means over the encoded instances to retrieve the clusters.\n\nFor the distant supervision approach, the authors create a dataset consisting of pairs of similar instances using two main methods:\nFirst, given a single instance, replace some words in the sentence or some marked entities to create a new sentence.\nSecondly, using a combination of OpenIE and NLI, find similar instances by comparing templates of the form subj-predicate-object derived from the source dataset of instances.\nThe authors propose an additional method that employs an LLM to generate new instances.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please cite inline where appropriate, e.g. lines 148, 251"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study on unsupervised relation extraction that involves better data augmentation. The authors also highlighted the inadequacy of NCE and argued that some alternative loss function may be more suitable. The reviewers largely acknowledge that the work is interesting and presents good results. Some major reservations include that the work seems to leverage external tools which may compromise the unsupervised setup, and that the two major techniques involved (loss, data augmentation) do not seem to naturally align with (or motivate) one another well.
Decision: Accept-Main