Corpus-Based Task-Specific Relation Discovery
Year: 2023

Abstract:
Relation extraction is a crucial language processing task for various downstream applications, including knowledge base completion, question answering, and summarization. Traditional relation-extraction techniques, however, rely on a predefined set of relations and model the extraction as a classification task. Consequently, such closed-world extraction methods are insufficient for inducing novel relations from a corpus. Unsupervised techniques like OpenIE, which extract <head, relation, tail> triples, generate relations that are too general for practical information extraction applications. In this work, we contribute the following: 1) We motivate and introduce a new task, corpus-based task-specific relation discovery. 2) We adapt existing data sources to create Wiki-Art, a novel dataset for task-specific relation discovery. 3) We develop a novel framework for relation discovery using zero-shot entity linking, prompting, and type-specific clustering. Our approach effectively connects unstructured text spans to their shared underlying relations, bridging the data-representation gap and significantly outperforming baselines on both quantitative and qualitative metrics. Our code and data are available in our GitHub repository.1

heading: 1 Introduction
text: Relation extraction (RE) aims to identify semantic relationships between entities in text in order to obtain triples of the form <head, relation, tail>, for instance, <Washington D.C., capital_of, USA>. RE is an important Information Extraction (IE) technique primarily used to complete knowledge bases (such as YAGO2 and NELL3) and construct semantic graphs (Vashishth et al., 2018). Knowledge bases and semantic graphs
1https://github.com/karthik63/ relation-discovery
2https://yago-knowledge.org/ 3http://rtw.ml.cmu.edu/rtw/kbbrowser/
see wide application in tasks such as questionanswering (Saxena et al., 2020; Das et al., 2017), recommendation (Zhang et al., 2016), and natural language inference (Peters et al., 2019).
Traditional relation extraction techniques approach the problem as a multi-class classification problem, and hence assume a predefined set of relations. Open Information Extraction (OpenIE) approaches (Angeli et al., 2015; Mausam et al., 2012) seek to remedy this problem by extracting relations from text without a predefined schema. But recent work (Schneider et al., 2017) has shown that in the absence of a schema, OpenIE results tend to be uninformative or redundant. Moreover, OpenIE systems are tuned for high recall and hence extract a very general set of tuples and defer the problem of sifting through the generated triples to find meaningful ones to subsequent analysis.
We now introduce the problem of task-specific relation discovery. Discovering unseen relations from a corpus serves two functions. Firstly, it serves as a starting point to fine-tune relation extraction models on novel relations and unseen domains. Secondly, it is intuitively appealing as a data mining task to gather actionable insights from a large unstructured corpus. Say, for instance, we are presented with a collection of recent documents reporting on the COVID-19 pandemic. Discovering all the relationships between viral variants and cities in Ohio in the corpus would allow us to detect relations such as "declining in", "spreading in" and "endemic in". This would allow us to automatically identify if there are areas of concern where viral spread is increasing. We define a task as a pair of semantic types (such as Humans, Geographic Locations, Sports Teams, etc.) between which we are interested in relations. A few concrete examples of this formulation can be seen in figure 1.
In this work, we propose a novel solution to this problem in three steps. First, we identify candidate spans for relation discovery using zero-shot entity
45
linking and typing. Then we use a self-supervised prompting technique using an encoder-decoder transformer architecture (Lewis et al., 2019) to discover relation phrases that describe the relationship between the two entities. Finally, we cluster these discovered phrases while keeping the head and tail semantic types of our relations in mind.
We briefly summarize our contributions below • We introduce a new task of corpus-based task-
specific relation discovery and modify existing data sources to make available Wiki-Art, a new dataset for the same • We propose a novel approach to extract candidate sentences, then discover and cluster unseen relations that significantly outperforms our baselines on both qualitative and quantitative metrics.

heading: 2 Related Work
text: The three lines of work most relevant to our approach are relation extraction, open information extraction, and prompting.

heading: 2.1 Relation Extraction
text: Most traditional relation extraction approaches model RE as a sequence classification task with specific accommodations for challenges arising from distant supervision. Models using piecewise CNNs (Zeng et al., 2015), reinforcement learning (Feng et al., 2018), and relationship side information have been proposed to mitigate the noise from sentences where ground-truth relations are not expressed.

heading: 2.2 Open Information Extraction
text: OpenIE seeks to produce domain-agnostic, unsupervised <head, relation, tail> extractions from a text span. Traditional approaches (Angeli et al., 2015; Mausam et al., 2012) to OpenIE use a combination of automatically mined and hand-crafted templates for relation extraction from the syntactic features and surface-forms of a sentence. These patterns are often mined using bootstrapping (Kolluru et al., 2020) where the triple extractions from multiple OpenIE approaches are aggregated to form a supervised training set using statistical rules.

heading: 2.3 Prompting
text: Prompting approaches (Liu et al., 2021) solve language processing tasks by eliciting natural language responses from language models rather than by training a classification layer. Relevant to our work, prompting has been successfully adapted to solve challenges such as few-shot event detection (Li et al., 2022) and event argument extraction (Ma et al., 2022; Li et al., 2021). Prompting can also be used to probe the inherent relational knowledge of pretrained language models by aggregating the masked language model generations from multiple hand-crafted prompts (Jiang et al., 2021).

heading: 3 Problem Definition
text: In this section, we formally define corpus-based task-specific relation discovery.
A corpus is a collection of documents from any domain. For simplicity, let our corpus S = [S1, S2, ..S|S|] be a sequence of sentences
Si. A sentence Si = [wi,1, wi,2, ...wi,|Si|] is a sequence of words wi ∈ V . Our task P = {(H1, T1), (H2, T2), ..(H|P|, T|P|)} is a set of Hi head, Ti tail semantic type tuples. T = {(h1, r1, t1), ...(h|τ |, r|τ |, t|τ |)} represents the set of ground-truth head-relation-tail triples expressed in S such that (hi, ti) ∈ {(xj , yj)|(xj , yj) ∈ (H1, T1) ∪ ... ∪ (H|P|, T|P|)}. That is, they only correspond to the task outlined using the head and tail semantic types.
Our end task is to discover the set of relations R = {r1, r2, ...r|R|} that occur in T , without assuming any prior knowledge about the relations in T other than the head and tail semantic types.

heading: 4 Methodology
text: Our overall architecture is illustrated in figure 2. Our procedure for relation discovery comprises three steps that we detail below- 1. We identify relevant entities and extract candidate spans of sentences to perform discovery on. 2. We discover the relation phrases that explain the relation between the head and tail entities. 3. We cluster the extracted relation phrases.

heading: 4.1 Extracting Candidate Spans for Discovery
text: Extracting candidate spans for relation discovery requires that we identify when semantic types of interest, outlined by our task P , co-occur in a paragraph. We require this entity typing process to have both high precision in order to avoid incorrect relation discoveries and high recall, so we don’t miss infrequent relations in our corpus. We propose the following procedure (illustrated in Figure 3) that meets both of these requirements. We utilize the BLINK(Ott et al., 2019) framework for both named entity recognition and zero-shot entity linking to Wikipedia. The advantage of using a zero-shot entity linker is that we can swap the Wikipedia index out for a more recent one, if needed, in order to handle newer entities. We then make use of the Wikipedia API4 in order to identify the Wikidata5 ID of the linked entity. With an entity’s Wikidata ID we first identify its type using the P31 "instance of" edge in the Wikidata KB. We then establish if it belongs to one of our pre-specified semantic classes by traversing the Wikidata taxonomy through the P279 "subclass of" edge until we reach the root node. If the concept node corresponding to one of
4https://en.wikipedia.org/w/api.php 5https://www.wikidata.org/
our semantic classes (e.g. human, country, film) is an ancestor of the type node, we identify the entity as such. If entities from both the head and tail semantic classes co-occur in a paragraph, it is selected as a candidate span to perform relation discovery on.

heading: 4.2 Identifying Novel Relation Phrases
text: We propose a prompting strategy to identify relation phrases between head and tail entities. We use the encoder-decoder transformer model BART (Lewis et al., 2019) and bootstrap from existing data sources in order to fine-tune the model for relation generation. Wikidata catalogs relation (property) aliases along with relations (example in table 1). We use the distant supervision setting explained in section 5 to identify paragraphs in which entities that have a relation in the Wikidata KB. Among these paragraphs, we filter out ones in which one of the aliases of the relation between the two entities does not occur. The semi-supervised data to train our prompting models comprises this filtered set of paragraphs and their corresponding head, tail entities, and relation surface-forms.
We experiment with a number of different prompting strategies. The optimal prompt variation, as determined by the results, is illustrated in Figure 4. The remaining three prompting strategies are presented in the appendix (Figure 6). Comparisons of the results of the different strategies on relation identification and unsupervised relation discovery are presented in tables 3 and 4, respectively. We briefly explain these different strategies below. <ARG>, </ARG>, <HEAD>, </HEAD>, <TAIL>, </TAIL>, <PRED> and </PRED> are all additional trainable embeddings that are finetuned with the rest of the model. <MASK> is the same mask token embedding used in BART’s pretraining tasks.

heading: 4.2.1 Prompt Variations
text: Vanilla Prompt: The input to the encoder and the decoder target of BART are illustrated in fig. 6a. The input to the encoder is the candidate paragraph concatenated with the surface forms of the head and tail entities separated by the <MASK> token. The decoder is expected to reconstruct the input and generate the relation alias in place of <MASK>
d1 & d2: These variations are illustrated in figures 6b and 6c. In these variations, the decoder is only expected to generate the relation and entity surface-forms.
del: This variation is illustrated in figure 4. We introduce six additional trainable token vectors to delimit the head and tail in the encoder input and the head, tail and predicate in the decoder target.
These trainable vectors serve a dual role. They act as an additional signal to the language model to help identify the head and tail entities for relation extraction and make the relation phrase easier to isolate during post-processing.

heading: 4.3 Clustering
text: Once we obtain the generated relation phrases, we aim to cluster phrases that denote the same underlying relation together in embedding space. Crucial to this step is the disambiguation between relations with the same surface form. For instance, the phrase "written by" corresponds to the relation screenwriter if it occurs between a human and a movie and the relation author if it occurs between a human and a comic book. To address this, we construct pseudosentences using the head and tail semantic types and the relation phrase. For instance, the generated relation phrase "voiced by" between a head of semantic type human and a tail of semantic type movie would result in the sentence "Film voiced by human.". We obtain the mean-pooled RoBERTa(Liu et al., 2019) embeddings of these pseudosentences and perform k-means clustering on them. The effect of including the head and tail types to our clustering process is shown in table 5. The results of clustering for unsupervised relation discovery is shown in table 4.

heading: 5 Dataset
text: In this section, we briefly describe our dataset Wiki-Art. We utilize the paragraphs scraped us-
ing distant supervision from the REBEL dataset (Huguet Cabot and Navigli, 2021). Under the distant supervision setting, a paragraph containing a head and tail entity is assumed to express a relation between two entities if the two entities are related in a background knowledge base (Wikidata in our case). Head and tail entities are identified using the links from anchor texts. We extract 326 documents (Wikipedia abstracts) from the Wikipedia pages of movies and comic books. The statistics of our dataset are shown in table 2.
We briefly explain the difference between the two settings- unsupervised relation discovery and corpus-based relation discovery.
Unsupervised Relation Discovery: To compare different IE models on discovery using fully quantitative metrics, we require a uniform number of test instances. So we only evaluate discovery on paragraphs that contain head and tail entities that we know to have a ground-truth relation between them. That is, we ignore paragraphs that contain entities with the head and tail semantic types outlined by our task if the entities do not have a relation between them in Wikidata. The six relations in table 2 are the relations that occur with the highest frequency in our corpus with at least forty occurrences.
Corpus-Based Relation Discovery: Clearly, for realistic relation discovery from a corpus, we can not assume pre-existing knowledge base information to filter out false-positive paragraphs where our task’s semantic types co-occur but a relation isn’t expressed. So, in this case, we perform discovery on all the paragraphs that contain both task semantic types. In this setting, the model is required to discover all nineteen task-specific relations expressed in the corpus, not just the six most common ones.

heading: 6 Experimental Setup
text: We describe our baselines, metrics and the details of the three experimental settings for relation identification, unsupervised relation discovery and corpus-based relation discovery.
Compared Models We compare the performance of our model against two SOTA OpenIE approaches.
• Stanford-OpenIE (Angeli et al., 2015) uses fourteen hand-crafted patterns defined over a dependency parse of the input text sequence in order to identify relational triples. • OpenIE56 combines four approachesCALMIE(Saha and Mausam, 2018), BONIE(Saha et al., 2017), RelNoun(Pal and Mausam, 2016) and SRLIE(Christensen et al., 2011) to extract relational triples. It uses a combination of hand-crafted and automatically mined patterns using syntactic and surface-form information.

heading: 6.1 Relation Identification
text: The purpose of this experiment is to evaluate the relative performance of different models on identifying a relation between two entities directly. As the same relation between two entities can be expressed in a number of ways, we perform this evaluation manually. We report the proportion of instances where the relation is identified accurately among the same 30 randomly sampled paragraphs. The results are shown in table 3.

heading: 6.2 Unsupervised Relation Discovery
text: The distinction between unsupervised relation discovery and corpus-based relation discovery is explained in section 5. The purpose of this experiment is to determine relative performance on relation discovery quantitatively. We report three
6github:dair-iitd/openie-standalone
commonly used clustering metrics. Adjusted Rand Index (ARI) measures the number of item pairs in the same vs. different clusters compared to the ground truth label assignment. Normalized Mutual Information (NMI) measures the mutual information between assigned and ground-truth cluster assignments. Permutation Accuracy (ACC) measures the accuracy between assigned clusters and ground-truth class labels with the best possible permutation matching clusters to labels. The results are shown in table 4.

heading: 6.3 Corpus-Based Relation Discovery
text: This experiment measures the proportion of ground truth relations (table 2) in a corpus we identify using our end-to-end procedure. We follow the same procedure outlined by Huang et al. to evaluate our models. We cluster the relation phrase embeddings of all compared methods into 100 clusters. We isolate the instance closest to the cluster centroid of all 100 clusters. We then manually inspect the isolated instances to determine if the extracted relation phrase corresponds to one of the ground truth relations in the corpus. The results are shown in table 6.

heading: 7 Results and analysis

heading: 7.1 Relation Identification
text: The results of our approach on relation identification are tabulated in table 3. Throughout, we indicate the size of our pretraining set in parenthesis. We outperform OpenIE on relation identification by 76 points. From tables 3 and 4 we observe that increasing the size of our pretraining step to 250,000 instances does not improve performance. For future analysis it would be useful to determine how much we can decrease the size of our pre-training step without significantly affecting performance. We also observe that the improved prompting variations outperform the vanilla prompt by 46 points on average. A comparison between the errors of
Approach Accuracy
Baselines
Stanford OpenIE 0.11 OpenIE5 0.17
Prompting
Approach NMI ARI ACC
Baselines
Stanford OpenIE .25 .03 .36 OpenIE5 .19 .04 .35
Prompting
the improved variations and the vanilla prompt can be observed in table 8 in the appendix. Restricting the decoder’s output mitigates the problems of poor quality extractions and spelling errors. The improved prompting models generate incorrect extractions when there are multiple relations between the head and the tail or when the relation expressed is semantically complex.

heading: 7.2 Clustering & Discovery
text: The results of our model on unsupervised relation extraction are shown in table 4. Prompt-del(10k) outperforms StanfordOpenIE by 49 points on permutation accuracy. t-SNE visualizations of these results are presented in figures 5a and 5b. We find that all six relations are discovered by our prompting approach and our clustering approach produces coherent, well-separated clusters. The improvements to our clustering strategy by taking head and tail semantic types into account are shown in table 5. Stanford-OpenIE on the other hand, only discovers relations expressed frequently as verbs such as "written by" and "directed by" with high precision.
Table 6 reports our results on corpus-based relation discovery. Our end-to-end pipeline discovers 15 out of the 19 ground-truth relations in our corpus. Figures 5c and 5d present t-SNE visualizations of our clustering results. We observe that most relations discovered by our framework coalesce into well-separated clusters. The majority of the relations extracted by OpenIE on the other hand, are too general to form well separated clusters.
A clearer illustration of the differences between our approach and OpenIE can be seen in table 7. We observe that both OpenIE baselines perform
poorly on relations not expressed as verb phrases. As a result, OpenIE5 fails to discover "colorist" and "voiced by", two relations frequently expressed as nouns.

heading: 8 Conclusions
text: We formulate a new task of corpus-based taskspecific relation discovery and introduce a new dataset for the same. We empirically demonstrate that existing art are inadequate to tackle this task. To address this, we propose an end-to-end selfsupervised pipeline for relation discovery that significantly outperforms our baselines on both quantitative and qualitative metrics. In the future, we plan on extending our approach to multiple domains in order to identify and correct possible gaps in our methodology.
Limitations
Domain Shift: In the current implementation, our prompting model relies on the availability of a training set. This assumption may not hold in cases where the relations to be discovered exhibit a significant domain shift from the training set. To address this limitation, future work should explore fully unsupervised prompting approaches that can better adapt to new domains and mitigate the impact of domain shift. Limited Number of Relations: In this study, our analysis is restricted to a total of 25 relations. While this allows for a focused exploration of these specific relations, it also limits the scope and potential applications of our model. To broaden the applicability and effectiveness of our approach, future work should aim to utilize Wikidata more com-
prehensively, incorporating a larger number of relations for more extensive and diverse analysis.
Ethics Statement
We conform to the ACL ethics policy. Our research utilizes data from Wikipedia, which is governed by the Creative Commons Attribution-ShareAlike License, and Wikidata, which is governed by the CC0 Public Domain Dedication License. Importantly, we have taken care to ensure that no personal information of any user is used in our study.

heading: A Comparative Error Analysis of Prompting Strategies
text: In our evaluation, we conducted an error analysis to compare the relative performance of different prompting models, which provided insights into their strengths and weaknesses. Qualitative results are displayed in Table 8.

heading: B Prompting Strategies
text: In our exploration of prompting strategies for identifying relation phrases, we examined three additional alternative approaches, as illustrated in figure 6.

heading: Future Work
text: In the future, we plan on extending our approach to multiple domains in order to identify and correct possible gaps in our methodology. Limitations
Domain Shift: In the current implementation, our prompting model relies on the availability of a training set. This assumption may not hold in cases where the relations to be discovered exhibit a significant domain shift from the training set. To address this limitation, future work should explore fully unsupervised prompting approaches that can better adapt to new domains and mitigate the impact of domain shift. Limited Number of Relations: In this study, our analysis is restricted to a total of 25 relations. While this allows for a focused exploration of these specific relations, it also limits the scope and potential applications of our model. To broaden the applicability and effectiveness of our approach, future work should aim to utilize Wikidata more com-
prehensively, incorporating a larger number of relations for more extensive and diverse analysis