Improving Zero-shot Relation Classification via Automatically-acquired Entailment Templates
Year: 2023

Abstract:
While fully supervised relation classification (RC) models perform well on large-scale datasets, their performance drops drastically in low-resource settings. As generating annotated examples are expensive, recent zero-shot methods have been proposed that reformulate RC into other NLP tasks for which supervision exists such as textual entailment. However, these methods rely on templates that are manually created which is costly and requires domain expertise. In this paper, we present a novel strategy for template generation for relation classification, which is based on adapting Harris’ distributional similarity principle to templates encoded using contextualized representations. Further, we perform empirical evaluation of different strategies for combining the automatically acquired templates with manual templates. The experimental results on TACRED show that our approach not only performs better than the zero-shot RC methods that only use manual templates, but also that it achieves state-of-the-art performance for zero-shot TACRED at 64.3 F1 score.

heading: 1 Introduction
text: Relation classification (RC) identifies the relation that holds between two entities that co-occur in the same text. For example, given the sentence: Jane’s White House desk has a sign “Home on the Range”, the relation between Jane and White House is employee_of. Beyond this simple example, RC is a critical NLP task with important applications to many domains such as intelligence (Doddington et al., 2004) and biomedical (Nédellec et al., 2013; Krallinger et al., 2017).
Recent directions mitigate the amount of supervision necessary for RC by taking advantage of the knowledge stored in large language models (LLMs). For example, Sainz et al. (2021) reformulated RC as an entailment task based on templates that are manually created as the verbalizations of
relation labels. Then they are used to formulate a hypothesis that can be verified with an off-theshelf LLM entailment engine. Other directions feed prompts that capture the definition of the task and examples into encoder-decoder language models (Han et al., 2022b). However, most of these directions tend to rely on templates/prompts that are manually-created by domain experts. This strategy has a potential high cost, and also runs the risk of inserting undesired biases in the data. Directions that focus on automatically learning prompts often produce prompts that are nonsensical to humans (Shin et al., 2020)
Our work aims to limit the above drawbacks of template-based approaches for RC. In particular, we expand the approach of Sainz et al. (2021) with explainable templates that are automatically acquired from a large textual collection. For template acquisition, we modify the BERT-Informed Rule Discovery (BIRD) algorithm (Rahimi and Surdeanu, 2022), which, given a seed template, automatically generates templates with similar meaning. BIRD encodes templates using contextualized representations generated by a transformer network (Devlin et al., 2019), and then uses a similarity measure to acquire similar templates based on an extension of the distributional similarity principle (Harris, 1954) to templates. For example, given a manual template {subj} was founded by {obj}, the template {subj} was created by {obj} is automatically created.
We use these automatically-acquired templates to expand the pool of manual templates used for RC, and show that this expansion yields statistically significant performance improvements.
The key contributions of this paper are:
• We introduce a novel strategy for template generation for RC, which is based on adapting Harris’ distributional similarity principle to templates encoded using contextualized representations.
187
• We perform an empirical analysis of different strategies for how to combine automaticallyacquired templates with templates that were manually generated. The strategies include selection using entailment score, selection using BIRD’s similarity score, and selection guided by the lowest entropy. All in all, our best combination obtains state-of-the-art performance for zero-shot TACRED, at 64.3 F1 score.

heading: 2 Related Work
text: Supervised Relation Classification. Most recent approaches for supervised relation classification use pre-trained language models such as models with self-supervised objectives, e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) and GPT-3 (Brown et al., 2020), or adapt sequence-tosequence models to the task, such as T5 (Raffel et al., 2020), BART (Lewis et al., 2020), and GLM (Du et al., 2021). Prior to the advent of large language models supervised models needed a large amount of labeled data for training models from scratch (Kambhatla, 2004, Zeng et al., 2014). Recent approaches outperform traditional approaches by finetuning language models (Wu and He, 2019, Joshi et al., 2020, Yamada et al., 2020, Wang et al., 2021b, Lyu and Chen, 2021, Paolini et al., 2021, Wang et al., 2022a) or prompting (Han et al., 2022b, Han et al., 2022a, Zhang et al., 2023).
Our work is different in that we focus on the zeroshot scenario, a common situation in the real world, where a RC system must be developed rapidly without the luxury of training data.
Zero-shot Relation Classification. Getting accurate annotations for RC can be expensive because it is challenging for annotators to identify and come to a consensus on the structural information required. This poses an obstacle for RC models, which have traditionally depended on direct supervision from a sufficient end-task training data. Standard supervised models often perform poorly when dealing with low-resource situations (Schick and Schütze, 2021), highlighting the importance of developing methods that perform well in low-resource settings. As a result, several approaches have been proposed for relation classification with few training examples (Han et al., 2018, Gao et al., 2019, Baldini Soares et al., 2019, Sabo et al., 2021, Sainz et al., 2021). For the problem of zero-shot relation classification, Rocktäschel et al. (2015) and Demeester et al. (2016) proposed the
use of logic rules. Wang et al. (2022b) used silver standard data cleaned by a class-aware clean data detection mechanism to train a textual entailment engine. In the literature, zero-shot RC has been reformulated as other tasks such as reading comprehension (Levy et al., 2017), textual entailment (Obamuyide and Vlachos, 2018, Sainz et al., 2021, Wang et al., 2022b), summarization (Lu et al., 2022), span-prediction (Cohen et al., 2020), question answering (Cetoli, 2020), triple generation (Wang et al., 2022a, Wang et al., 2021a), and prompting (Gong and Eldardiry, 2021).
Our work fits within this latter group. However, our contribution is that it mixes manual seed templates with explainable templates automatically acquired using the distributional similarity principle tailored for templates. Our results indicate that this simple strategy yields state-of-the-art performance on a popular zero-shot RC task.

heading: 3 Background
text: As mentioned, our zero-shot approach for relation classification relies on an entailment engine that is fed automatically-generated templates. In this section we discuss the building blocks necessary for this idea.

heading: 3.1 The Relation Classification Task
text: In the relation classification task we intend to classify a sentence with two marked entities into a predefined set of relations, or indicate that none of the relations hold between them (none-of-the-above or NOTA). Each input is a triple xi = (s, e1, e2)i which consists of a sentence s with an ordered pair of two entities e1 and e2 (each entity is a span over s). The output r ∈ R ∪ {NOTA} indicates the two entities conform to one of the relations defined in the target set of R = {r1, r2, ..., rN} or none of the relations hold.

heading: 3.2 Zero-shot RC as an Entailment Task
text: Since our work is based on the works of Sainz et al. (2021) that reformulated RC as an entailment task on TACRED dataset, we provide an overview of their work. In a zero-shot setup no training examples are provided to the model. Therefore, the RC model must make predictions on relation instances without seeing any related data prior to that. By reformulating RC as an entailment task, it is possible to take advantage of the existing off-theshelf entailment engines and use them as-is. Fig-
ure 1 demonstrates the overview of the approach. First, a set of templates are manually created for each relation type in TACRED. The templates verbalize the relation types. For example, the relation PER:DATE_OF_BIRTH can be verbalized as {subj}’s birthday is on {obj}. Sainz et al. (2021) manually created 2 templates on average for each relation type. After that, an entailment engine is used to perform inference on each example in TACRED’s test set. For each such example, the {subj} and {obj} placeholders of all the templates are replaced with the two marked entities of the example. After that, we use the TACRED example as premise and each template as hypothesis to feed them to an entailment engine. The entailment engine will produce an entailment score for each template. We pick the template with the highest score and return its corresponding relation type as the final prediction.
For the NOTA relation, however, creating templates does not produce good results. As a result, a threshold-based approach is used to detect NOTA. A threshold (between 0 and 1) is selected for the NOTA relation. If none of the entailment scores are above this threshold, the input example will be classified as NOTA. The treshold is selected by using 1% of TACRED’s development set.

heading: 3.3 Rule Acquisition
text: BERT-Informed Rule Discovery or BIRD (Rahimi and Surdeanu, 2022) is a rule1 acquisition algorithm. Informally, BIRD learns inferences from
1Alternatively called patterns or syntactic paths.
text such as “X is the author of Y ≈ X writes Y”. Some of these inferences are not exact paraphrases (but are still relevant and potentially useful) such as “X is the author of Y ≈ X is known for Y”. More formally, BIRD is initialized with a seed rule (e.g. “X is the author of Y”), which is implemented as a syntactic path connecting two concepts, and infers one or more possible matches (e.g. “X writes Y” ) where each match is a syntactic path. BIRD generates the matches by implementing Harris’ Distributional Hypothesis principle (Harris, 1954) to rules. It states that if two patterns tend to link the same sets of words, they tend to have similar meanings.
BIRD relies on contextualized representations generated by a transformer network (Devlin et al., 2019). In particular, a pattern has two slots (X and Y); BIRD computes contextualized embeddings for each slot. By doing so, a pattern will be represented by two embedding vectors. Then a (cosine) similarity score is calculated between each slot of any given two patterns, i.e., one similarity score is calculated for slot X and one similarity score is calculated for slot Y. Finally, the average of the two similarity scores is computed as the similarity score between the two patterns.
Given a corpus, BIRD extracts and stores all patterns from the corpus and then computes the contextualized embeddings for each slot of each pattern. Therefore, given a pattern, BIRD can search the extracted patterns space and find the most similar patterns to the input pattern.

heading: 3.4 TACRED Dataset
text: TACRED (Zhang et al., 2017) is a large-scale relation classification benchmark that is consisted of 106,264 examples and 42 relation types including the no_relation (NOTA) label. Each example contains the information about the entity type, among other linguistic information. TACRED examples include 68,124 for training, 22,631 for validation, and 15,509 for testing.

heading: 4 Approach
text: One limitation of (Sainz et al., 2021) is that they rely on manually-generated templates, which require effort to create and may prone to bias. Our work mitigates this limitation by automatically expanding the manual patterns using BIRD.
We utilize the manual templates as seed patterns to feed them into BIRD to generate new patterns that tend to have similar meanings to the
manual templates. Figure 2 shows this process. For each manual template, we extract its syntactic path. For example, for the template {subj} was founded by {obj}, we extract the syntactic path nsubjpass←found→nmod_agent (X was founded by Y). Then the syntactic path is fed to BIRD to generate new syntactic paths. We generate 40 patterns for each manual template. After the syntactic paths are generated, we generate their corresponding English text using a module we created. For example, for the syntactic path nsubjpass←create→nmod_agent, the English text “X was created by Y” is generated.
The corresponding patterns of 12 manual templates did not exist in BIRD’s pattern collection. Therefore, it was not possible to use them as seed patterns. For these templates we manually selected an existing high-quality pattern that was very close in meaning. For example, the pattern “X is spouse of Y” did not exist in BIRD’s pattern collection.
We chose the pattern “X married Y” instead.

heading: 4.1 Determining Subject and Object of a Pattern
text: The manual templates have {subj} and {obj} placeholders which correspond to the subject and object of a relation, respectively. The patterns generated by BIRD do not contain this information. That is, it is unclear what the mapping is between the X and Y slots in a BIRD template and {subj} and {obj} in a relation template. Therefore, it is necessary to determine which slot of a pattern is the subject and which slot is the object. We use our off-the-shelf entailment engine to determine this information. Figure 3 demonstrates this procedure. For each generated pattern, we create an example sentence for its seed manual template. This example sentence will be used as premise. As for the hypothesis, we create two example sentences for the pattern by using the same words that filled the
placeholders of the seed manual template to fill the slots of the pattern. The two example sentences are the same sentence except their slot-filler words are swapped. One of these two hypotheses is expected to have a similar meaning to the manual template’s example sentence. Therefore, we expect the entailment engine to produce a high entailment score for it and to produce a low score for the other one. We pick the hypothesis with the highest entailment score to determine the subject and object of the pattern. If the entailment engine produces entailment scores below 0.5 for both of the hypotheses, we discard the pattern. By determining the subject and object of the patterns, they become fully functioning templates.

heading: 4.2 Selection of Patterns
text: After the templates are generated and set up, a selection mechanism is required to keep the most useful ones and discard the rest.2 We always keep all of the manual templates and the selection mechanism is only applied on the BIRD-generated ones.

heading: 4.2.1 Selection using entailment score
text: In subsection 4.1, it was explained that an entailment score is obtained for each BIRD-generated pattern when the subject and object of the pattern is determined. We use this entailment score to sort the templates and keep the top ones.

heading: 4.2.2 Selection using BIRD’s similarity score
text: For a seed pattern, BIRD generates similar patterns using a similarity measure. This yields a similarity score for each BIRD-generated pattern. We use this similarity score to sort the templates and keep the top ones.

heading: 4.2.3 Selection guided by lowest entropy
text: A low entropy for the prediction of a classifier means that the classifier was more confident when making the prediction. We use this idea to pick the templates that together result in the lowest entropy. For each TACRED test set example, we perform the following. For each relation ri ∈ R, we pick the template (from the pool of templates for ri) that produces the highest prediction score (entailment score when TACRED example is premise and the template is hypothesis). For all rj ∈ R− {ri}, we pick the template (from the pool of templates for rj) that produces the lowest prediction score. The
2Keeping all of the templates resulted in poor performance in our experiments.
combination of these chosen templates create one candidate set of templates. This procedure is done for each ri ∈ R. As a result, we will have 41 candidate sets. Each candidate set is expected to have a low entropy. We keep the candidate set with the lowest entropy. Note that during the candidate sets creation process, we never look at the gold labels and merely use the model prediction scores.

heading: 4.3 Inference
text: After the final set of templates is selected, we perform the inference according to the method described in Sainz et al. (2021) as shown in Figure 1. During inference, they pick the template that produces the highest entailment score and return its corresponding relation type as the final prediction. In addition to using the “highest entailment score”, we also experimented with two additional methods to perform the final prediction. Firstly, instead of choosing the relation type that has the highest entailment score, we computed the average of the top 3 highest entailment scores for each relation type, and then we pick the relation type that has the highest amount of this average. Secondly, we used a normalized unweighted voting mechanism. For each template, if its entailment score is more than its neutral and contradiction scores, we count this template as a positive vote. We count these positive votes for each relation type and then divide the count by the total number of templates for the relation type. The relation type with the highest normalized votes will be selected as the final prediction.

heading: 5 Experiments

heading: 5.1 Experimental Settings
text: We conducted our experiments on TACRED using the zero-shot RC setup from Sainz et al. (2021). For our experiments, we used DeBERTa which was the off-the-shelf entailment engine that produced the highest results in Sainz et al. (2021).
The set of templates that we used for our experiments included all of the manual templates as well as a small subset of the BIRD-generated templates. The subset was selected according to the pattern selection methods explained in subsection 4.2. Initially, we generate 40 patterns per each manual template. After that, we sort the generated patterns according to a scoring measure and keep the top k patterns with k = {1, 2, 3, 4}3.
3Values of k higher than 4 did not yield better performance.
The no_relation threshold was selected using 1% of the validation set. Sainz et al. (2021) divided the validation set to 100 stratified folds, ran 100 experiments for each fold, and obtained 100 f1 scores. They reported the median f1 score along with f1 standard deviation. We do the same.

heading: 5.2 Results
text: Table 1 shows the zero-shot RC results including our best results which were obtained using “entailment score” as pattern selection method and “highest entailment score” as inference method. Thirdparty results are as reported by authors. The table indicates that our results match the current state-ofthe-art for zero-shot TACRED (Zhang et al., 2017), and are 1.5 F1 points above Sainz et al. (2021)’s results. The improvement in F1 comes from both better precision and better recall. In order to confirm that the difference in F1 scores between our method and (Sainz et al., 2021) is statistically significant, we ran a significance test using bootstrap resampling experiment with 1000 samples. This test indicated that the difference in F1 scores between our method and (Sainz et al., 2021) is statistically significant with a p-value of 0.3. We find these results encouraging considering that the manual patterns in Sainz et al. (2021) were manually developed by domain experts, whereas the pattern acquisition method in BIRD was developed independently of the TACRED task and its distributional similarity statistics were acquired from a different dataset (a Wikipedia subset) (Rahimi and Surdeanu, 2022).
We experimented with different pattern selection methods as shown in Figure 4. We observed that entailment score performed better overall than the rest. We also observed that as the number of added templates increased, the performance of the models decreased indicating that increasing the number of templates had a negative effect on performance. In general, while BIRD’s similarity score was more robust as more templates are considered, the best result was obtained with one added template. Furthermore, we experimented with different inference methods as shown in Figure 5 when the pattern selection method was entailment score (since it produced the highest results). We observed that highest entailment score performed better than the rest. We hypothesize that this is caused by the relatively noisy BIRD generated templates, which suggest that future work is needed to better align BIRD’s output with the entailment-driven RC task.

heading: 6 Conclusions
text: This paper introduces a zero-shot approach for relation classification. Our direction expands the method of Sainz et al. (2021), which converts the relation extraction task into textual entailment that is informed by manual templates that characterize the relations of interest. Unlike Sainz et al. (2021), we combine manual templates with templates that were automatically-acquired using an adaption of Harris’ distributional similarity principle to templates encoded using contextualized representations.
We empirically evaluate our approach on a zeroshot setting of the TACRED relation classification task (Zhang et al., 2017). We investigated multiple strategies to rank the quality of the automaticallyacquired templates. All in all, we found that a simple strategy, which considers the top template as ranked by a textual entailment engine, performs the best. Our results match the current state-ofthe-art for zero-shot TACRED. We find this result exciting, especially considering that the template acquisition component is disconnected from the TACRED dataset. Beyond these results, this paper opens interesting questions for future work such as how to increase the relevance of the automaticallyacquired templates for a specific task.

heading: Future Work
text: Beyond these results, this paper opens interesting questions for future work such as how to increase the relevance of the automaticallyacquired templates for a specific task