SlowGuess/NAACL_2022_OCR huggingface dataset:
Lu Sun, Yongliang Shen, Weiming Lu *
College of Computer Science and Technology, Zhejiang University
{sunlu98,syl,luwm}  $@$  zju.edu.cn
Relation Induction is a very practical task in Natural Language Processing (NLP) area. In practical application scenarios, people want to induce more entity pairs having the same relation from only a few seed entity pairs. Thus, instead of the laborious supervised setting, in this paper, we focus on the minimally-supervised setting where only a couple of seed entity pairs per relation are provided. Although the conventional relation induction methods have made some success, their performance depends heavily on the quality of word embeddings. The great success of Pre-trained Language Models, such as BERT, changes the NLP area a lot, and they are proven to be able to better capture relation knowledge. In this paper, we propose a novel method to induce relation with BERT under the minimally-supervised setting. Specifically, we firstly extract proper templates from the corpus by using the mask-prediction task in BERT to build pseudo-sentences as the context of entity pairs. Then we use BERT attention weights to better represent the pseudosentences. In addition, We also use the Integrated Gradient of entity pairs to iteratively select better templates further. Finally, with the high-quality pseudo-sentences, we can train a better classifier for relation induction. Experiments on Google Analogy Test Sets (GATS), Bigger Analogy Test Set (BATS) and DiffVec demonstrate that our proposed method achieves state-of-the-art performance.
Relation induction is a task to judge whether two entities have a certain relation based on some given entity pairs of that relation, which was first proposed in (Vylomova et al., 2016). For instance, given  $\{(Germany, Berlin), (France, Pairs), (Italy, Rome)\}$ , relation induction is to predict whether new entity pairs such as (China, Beijing) have the same relation as the
given entity pairs. In practical scenarios, only a few seed entity pairs are available. It is challenging to judge the relation of the target entity pairs in this minimal supervision setting.
Word embedding, such as skip-gram (Mikolov et al., 2013a) and Glove (Pennington et al., 2014), are widely used in many natural language processing (NLP) tasks, and it was reported that word embeddings can capture the relational knowledge (Mikolov et al., 2013b). One intuitional method for relation induction task is using word embeddings to represent relations and induce relations based on vector translation or similarity (Vylomova et al., 2016; Drozd et al., 2016; Bouraoui et al., 2018; Vulic and Mrkšić, 2018; Camacho-Collados et al., 2019). However, the performance of these methods heavily depends on the pre-trained word embedding and these methods are rather noisy. According to the assumption that if two entities have a relationship in a known knowledge base, then all sentences that mention these two entities will express that relationship in some way (Mintz et al., 2009), many distant-supervised methods of relation extraction, such as PCNN(Zeng et al., 2015) and PCNN-BagATT (Ye and Ling, 2019) are proposed. Inspired by these methods, distant supervision might be another way to induce relation. To induce relation in the distant supervised way, we need a method to select proper sentences from corpus and extract relational knowledge from sentences. Luckily, many Pre-trained Language Models (PLMs), such as BERT(Devlin et al., 2019), GPT-2 (Radford et al., 2019) and XLNet(Yang et al., 2019), have been recently proposed and boost a great performance for many NLP tasks, such as question answering(Talmor et al., 2019; Feng et al., 2020), text summarization (Liu and Lapata, 2019; Lewis et al., 2020) and information extraction (Petroni et al., 2019; Alt et al., 2019). In order to better understand the PLMs, several works(Kim et al., 2020; Bouraoui et al., 2020; Ushio et al., 2021; Chen
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1776 - 1786 July 10-15, 2022 ©2022 Association for Computational Linguistics
et al., 2021) have proven that PLMs can capture syntactic and semantic knowledge. Bouraoui et al. (2020) have explored the possibility of inducing relation from BERT in a distant supervised way and got a good result. To take the advantage that BERT can capture context knowledge, they select templates from corpus and fill entities in them to let BERT predict the relation.
Existing methods are developed under the assumption of sufficient seed entity pairs. However, in practical scenarios, only a few entity pairs are available for a particular relation. These methods have difficulty in coping with the minimal supervision setting. The main reasons are: (1) Due to the lack of labeled entity pairs, the model tends to over-focus on the surface cues of the entity pairs and ignores the contextual semantics. By simply memorizing the seed entity pairs, it is difficult to generalize the model to other entity pairs. (2) The quality of templates is very important for relation induction. When the seed entity pairs of a certain relation are sparse, the number of candidate templates for this relation mined from the corpus will be reduced.
Therefore, two major challenges should be addressed for the relation induction in the minimally-supervised setting. (1) How to obtain a good generalized relation induction model? (2) How to obtain high-quality templates? So we propose a novel approach called IST for minimally-supervised relation induction with Iteratively-Selected Templates from PLM. Specifically, for the first challenge, we use surface-agnostic features based on attention maps of BERT. Many works (Clark et al., 2019; Kovaleva et al., 2019; Michel et al., 2019; Wang et al., 2020) have revealed that the attention heads in BERT can capture much knowledge and some attention heads are related to certain relations, and some works use attention weights to predict relations (Gu et al., 2021). For the second challenge, we use Integrated Gradient (IG) (Sundararajan et al., 2017) to score the templates and iteratively select better templates. Intuitively, if a sentence can well express the relational knowledge between two entities, then the importance of these two entities must be high in the sentence. On the contrary, if a pair of entities do not play an important role in a sentence, this sentence certainly does not express the relationship between them. So IG might be used to select high-quality sentences to express relations.
- We propose a novel minimally-supervised relation induction approach IST. To the best of our knowledge, we are the first to address the minimally-supervised relation induction task.
- In order to overcome the minimally-supervised setting, we generate high-quality pseudo-sentences by iteratively selecting templates based on BERT and IG scores. Moreover, we use attention maps to train a more generalized model.
- We conduct extensive experiments on three standard benchmark datasets, and our proposed approach significantly outperforms the state-of-the-art approaches.
In this section, we first formulate the minimally-supervised relation induction task and give an overview of our approach. We then describe the details of each module in our approach.
Given a few seed entity pairs  $P_r = \{(s_i, t_i)\}_{i=1}^N$  with a certain relation  $r$ , the task of relation induction is to judge whether a new entity pair  $(s, t)$  also has the relation  $r$ . In the minimally-supervised setting, the number of the seed pairs is small for each relation (in our experiments, no more than 5 per relation). To facilitate minimally-supervised relation induction, we generate high-quality pseudo sentences  $S_r$  for each relation  $r$  from a text corpus  $C$  according to the seed entity pairs  $P_r$  with the help of a pre-trained language model.
As illustrated in Figure 1, our approach consists of four main modules: template generation module, pseudo sentence generation module, relation classifier and template selection with IG.
In the template generation module, given seed entity pairs, some proper templates could be generated based on the mask-prediction results in BERT. For instance, considering the seed entity pairs  $P_r = \{(Germany, Berlin), (France, Paris), (Japan, Tokyo)\}$ , we can obtain a sentence set  $S_r$  where each sentence mentions both entities of a pair in  $P_r$ . Taking a sentence The current capital of Japan is Tokyo, as an
Figure 1: An overview of IST. First, we extract sentences that mention seed entity pairs as candidate sentences. Then, the template generation module uses BERT-prediction task to select proper templates from candidates. Templates and seed entity pairs are assembled to generate pseudo sentences to extract relational knowledge from BERT. The BERT attention weights between entities within the pseudo sentence are used as surface-agnostic features to better represent the relational knowledge. Then, the pseudo sentences and attention weights are combined to train a BERT-based relation classifier. Finally, we use the integrated gradient of entity pairs in pseudo sentences to evaluate the quality of templates and select better templates iteratively.
example,  $\tau =$  (The current capital of  $-$  is  $-$ ) is the generated template. Then, filling one entity into templates, the templates can be scored according to their ability to make BERT correctly predict another entity. This score is referred to as scoreBERT.
After selecting proper templates based on score  $BERT$ , we can generate pseudo sentences by assembling the templates and seed entity pairs. For example, a pseudo sentence  $\tau$  (Germany, Berlin) = The current capital of Germany is Berlin. is generated by assembling (Germany, Berlin) and  $\tau$ . We generate both positive and negative sentences in this process.
For each pseudo-sentence, we extract surface-agnostic features based on attention weight maps of BERT and use them to train a relation classifier.
Finally, we use integrated gradient (IG) together with score  $BERT$  to evaluate the quality of templates again, so we can refine templates iteratively.
We will describe each module in detail in the following sections.
2.3 Template Generation
To induce relation from masked pre-trained language models such as BERT, we need templates for relations. First, many template-based relation extraction methods(Agichtein and Gravano, 2000; Ravichandran and Hovy, 2002) have proved that words near to  $s$  and  $t$  in corpus may represent a
certain relation. To extract templates for relation  $r$ , we traverse Wiki Corpus to find  $k_{i}$  sentences that contain both  $s_{i}$  and  $t_{i} (\mathrm{i} \leq \mathrm{N})$ , and the distance between  $s_{i}$  and  $t_{i}$  in sentence  $D_{st} \leq d$ . Then we mask  $s_{i}$  and  $t_{i}$  in these sentences to generate templates  $\tau_{i,1}, \tau_{i,2} \ldots \tau_{i,k}$ . We can extract all candidate templates for  $r$ :  $\{\tau_{1,1}, \tau_{1,2} \ldots \tau_{i,j} \ldots \tau_{N,k_N}\} (i \leq N, j \leq k_i)$ , but not all of these templates are proper for inducing the relation  $r$ .
Then we need to select templates that are proper for BERT to induce relation  $r$ . Here we use BERT mask prediction as a template filter (Bouraoui et al., 2020). Specifically, for a template  $\tau$ , insert  $s$  and  $t$  into  $\tau$  respectively to get masked sentence  $\tau(s, -)$  and  $\tau(-, t)$ . Then let BERT predict the masked token. If BERT can predict correctly, we consider the template  $\tau$  is proper for relation  $r$  and  $\tau(s, t)$  is "natural" for BERT.
$$
\operatorname {s c o r e} _ {B E R T} (\tau) = \sum_ {i = 0} ^ {N} \left(M \left(\tau \left(s _ {i}, -\right)\right) + M \left(\tau \left(_ {-}, t _ {i}\right)\right)\right) \tag {1}
$$
where  $M(\tau(s_i, \_))$  is 1 if the predicted token is  $t_i$  and 0 otherwise, and similar for  $M(\tau(\_, t_i)) = 1$ .
By ranking templates with score  $_{BERT}$ ,  $K$  proper templates  $T_{r} = \{\tau_{1},\tau_{2}\dots \tau_{K}\}$  are selected from candidate templates.
2.4 Pseudo Sentence Generation
In order to train a relation classifier, we assemble templates and seed entity pairs to generate labeled pseudo sentences.
For positive sentences, we just assemble each entity pair  $(s,t)\in P_r$  with each template  $\tau \in T_r$  to generate a sentence  $\tau (s,t)$
While for the negative sentences, following (Vylomova et al., 2016), we have three strategy for each pair  $(s_i,t_i)\in P_r$ . First, we exchange  $s$  and  $t$  as  $(t_i,s_i)$  (suppose  $r$  is not symmetrical). Second, we change one entity to another entity in the same relation:  $(s_i,t_j)$  or  $(s_j,t_i)(i\neq j,(s_i,t_i),(s_j,t_j)\in P_r)$ . Third, we change one entity to an entity in other relations:  $(s_i,t_j)$  or  $(s_j,t_i)(i\neq j,(s_j,t_j)\in P_{r'})$ .
2.5 Relation Classifier
Under the minimally-supervised setting, the model should have good generalizability. We use surface-agnostic features based on attention weights of BERT to make model focus more on the relations rather than the surface information of training data.
As Clark et al. (2019) has pointed out, some heads of multi-head attention in BERT are related to certain relations, and attention weights of certain heads can be used to extract certain relation knowledge. Thus, for a proper template of relation  $r$ , the attention weights between  $s$  and  $t$  of certain heads related to  $r$  should be higher. But it is hard to specify each head is related to what relations. Thus we use attention weights of all heads as features to induce relation knowledge.
Specifically, for a sentence  $\tau(s,t)$ , we calculate the attention weights between  $s$  and  $t$  of all heads as  $\omega_{i,j,s\rightarrow t}$ , where  $i$  denotes the  $i$ -th layer,  $j$  denotes the  $j$ -th head in layer  $i$  and  $s\rightarrow t$  denotes that this is the attention  $s$  pays to  $t$ . We use the average between  $s\rightarrow t$  and  $t\rightarrow s$  to express the attention between them:
$$
\omega_ {i, j} = \frac {\omega_ {i , j , s \rightarrow t} + \omega_ {i , j , t \rightarrow s}}{2} \tag {2}
$$
Then we construct attention weights embedding for the sentence  $\tau(s, t)$ :
$$
H _ {\tau (s, t)} ^ {a t t} = \left\{\omega_ {1, 1}, \omega_ {1, 2} \dots \omega_ {i, j} \dots \omega_ {n l, n h} \right\} \tag {3}
$$
where  $nl$  denotes the layer number,  $nh$  denotes head number in a layer of BERT.
Besides  $H_{\tau(s,t)}^{att}$ , we also use BERT outputs to represent the sentence  $\tau(s,t)$ . Specifically, we input  $\tau(s,t)$  into the BERT, and then use the output
vector of the [CLS] token as the feature  $H_{\tau(s,t)}^{cls}$ .  $H_{\tau(s,t)}^{cls}$  and  $H_{\tau(s,t)}^{att}$  can compensate each other, since  $H_{\tau(s,t)}^{cls}$  can capture the information whether  $\tau(s,t)$  is "natural", and  $H_{\tau(s,t)}^{att}$  contains the correlation between  $(s,t)$  and the relation  $r$ . Thus, we combine these two vectors through concatenation:
$$
H _ {\tau (s, t)} = H _ {\tau (s, t)} ^ {c l s} \oplus H _ {\tau (s, t)} ^ {a t t} \tag {4}
$$
Then, we feed  $H_{\tau(s,t)}$  to a MLP classifier  $\mathcal{F}$  and get the probability of  $(s,t)$  having relation  $r$ . We use a cross-entropy loss to optimize  $\mathcal{F}$ . In addition, we can also finetune BERT when training the classifier.
2.6 Iterative Template Selection
BERT can rank templates by measuring whether a sentence is natural. However, it can not capture the different attribution of each token in a sentence for expressing the relation.
Integrated Gradient is an attribution method proposed in (Sundararajan et al., 2017). As Cui et al. (2020) has described, the attribution score directly reflects how much changing tokens will change the model's outputs. A higher attribution score represents more importance of tokens. In our relation induction model,  $s$  and  $t$  obviously should be the most important two tokens in sentences. Intuitively, for a pseudo sentence  $\tau(s, t)$ , if the integrated gradient value for  $s$  and  $t$  to the relation prediction is higher, we are more confident that the relational knowledge of  $(s, t)$  can be extracted well by the model along with  $\tau$ , so the template  $\tau$  is much better. Thus, we can use the integrated gradient of  $(s, t)$  to the output of relation classifier to select templates once again. Here,  $\mathcal{F}(\tau, s, t)$  denotes the relation classifier with  $\tau(s, t)$  as the input.
According to Sundararajan et al. (2017), the integrated gradient value of  $s$  to  $\mathcal{F}(\tau, s, t)$  is:
$$
I G (\tau , s) = (s - s _ {0}) \int_ {x = 0} ^ {1} \frac {\partial \mathcal {F} (\tau , s _ {0} + \alpha (s - s _ {0}) , t)}{\partial s} \mathrm {d} \alpha \tag {5}
$$
where  $\alpha \in [0,1]$ , and it can be approximated as:
$$
I G (\tau , s) = (s - s _ {0}) \sum_ {i = 1} ^ {m} \frac {1}{m} \times \frac {\partial \mathcal {F} (\tau , s _ {0} + \frac {i}{m} (s - s _ {0}) , t)}{\partial s} \tag {6}
$$
where  $m$  is the number of approximate steps for computing integrate gradient and  $s_0$  is generated by replacing the word embedding of  $s$  with zeros. For
a template  $\tau$ , we calculate the average integrated gradient value for all  $(s,t) \in P_r$ :
$$
\operatorname {s c o r e} _ {I G} (\tau) = \sum_ {(s, t) \in P _ {r}} \frac {I G (\tau , s) + I G (\tau , t)}{2} \tag {7}
$$
Then the templates are re-ranked according to the final score:
$$
s c o r e = \alpha \cdot \frac {1}{r a n k _ {B E R T}} + (1 - \alpha) \frac {1}{r a n k _ {I G}} \tag {8}
$$
where  $\text{rank}_{\text{BERT}}$  denotes the rank of templates according to  $\text{score}_{\text{BERT}}$ ,  $\text{rank}_{\text{IG}}$  denotes the rank of templates according to  $\text{score}_{\text{IG}}$ , and  $\alpha \in [0,1]$  is an coefficient to balance the two scores. Therefore, the templates could be selected iteratively for better relation induction.
2.7 Relation Induction
Given a new entity pair  $(x,y)$ , we fill them into templates  $\tau_{i}, (i \in K)$  and use the classifier to predict  $p_{i}(x,y)$ , which denotes how much  $\tau_{i}(x,y)$  is "natural". Following Bouraoui et al. (2020), for all predictions from  $K$  templates  $p_{1}(x,y), \ldots, p_{K}(x,y)$ , if  $\max_{i} p_{i}(x,y) > 1 - \min_{i} p_{i}(x,y)$ , then  $(x,y)$  is predicted to be positive.
We conduct the experiments on three standard benchmark datasets in English: Google Analogy Test Set (GATS) (Mikolov et al., 2013a), Bigger Analogy Test Set (BATS) (Gladkova et al., 2016) and DiffVec(Vylomova et al., 2016).
GATS contains 5 semantic relations and 9 syntactic relations, and each consists of a varying number of entity pairs. While BATs contains 40 relations which are divided into 20 morphology relations and 20 semantic relations, each relation has 50 instances. DiffVec contains 36 relations with a various number of entity pairs. 10 of them are lexical or morphology relations and the remaining 26 are semantic relations.
The relation induction task can be modeled as a binary classification problem for each relation. We first split the dataset into  $50\%$  of training data and  $50\%$  of test data. Then, under the minimally-supervised setting, for each relation  $r$ , we randomly select  $N$  entity pairs from training data as the seed
entity pairs  $P_r$ . We extract candidate templates from the English Wikipedia corpus<sup>1</sup> and  $d = 15$ . When generating  $K$  templates in  $T$  iterations, we initially select  $K(T + 1)$  templates according to BERT score, and then iteratively filter out  $K$  improper templates in each iteration according to the score defined in Formula 8 until  $K$  templates are reserved at last for the final iteration. Notice that when  $T = 0$ , we only select  $K$  templates according to the score_BERT without considering the score_IG. In our experiments, we use BERT-base<sup>2</sup>, and set  $N = 5$ ,  $K = 20$ ,  $T = 3$ ,  $\alpha = 0.5$  by default.
We generate the same number of negative examples as positive examples for the training data and 3 times as many negative examples as positive examples for the test data.
For each relation, we repeat the experiments for 10 times and calculate the average result. The seed entity pairs used in each trial is randomly selected. The results of all metrics are calculated with micro-average.
We compare our approach with three kinds of baselines.
The first kind is using the combination of pre-trained word embeddings to present relations. Specifically, following Vu and Shwartz (2018), we use  $s \oplus t \oplus (s \odot t)$  to represent the relation between  $(s, t)$  and use a MLP classifier to make predictions. Here, the pre-trained word embeddings we used are Glove(Pennington et al., 2014) $^3$  and SkipGram(Mikolov et al., 2013b) $^4$ . These two baselines are referred to as  $\mathrm{MLP}_{sg}$  and  $\mathrm{MLP}_{gl}$  respectively. We also use the Trans approach (Bouraoui et al., 2018) for relation induction by building subspaces for entities using word embeddings and modeling the relations with relative positions between subspaces.
The second kind is distant supervised methods. We use PCNN(Zeng et al., 2015) and PCNN-BagAtt(Ye and Ling, 2019) as two baselines. These distant supervised methods are proposed to solve the problem of noise in labeled data in relation extraction tasks. We also select the same number of
<sup>1</sup>We used the dump of May 2021
2We used the BERT implementation available at https://github.com/huggingface/transformers
3https://nlp.stanford.edu/projects/glove/
<sup>4</sup>https://code.google.com/archive/p/word2vec/
N=3	N=5
GATS	BATS	DiffVec	GATS	BATS	DiffVec
Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1	Pre	Rec	F1
MLPsg	41	54.4	39.6	40.3	45.8	40.6	40.1	49.8	41.5	43.3	56.8	45.3	43.5	47.1	43.8	43.5	51.3	43.9
MLPgl	42.5	54.8	43.1	41.2	45.7	51.3	40.5	50.2	41.9	43.9	56.5	45.9	43.8	47.0	43.9	43.8	51.6	44.2
PCNN	58.6	52.5	56.1	52.1	45.8	47.7	57.3	51.5	53.9	60.1	56.2	58.3	53.4	45.8	50.3	59.0	52.7	55.4
PCNN_BagATT	63.9	56.2	59.5	56.4	45.9	48.6	61.5	52.8	55.8	65.3	58.6	60.1	57.8	51.0	50.8	63.5	53.4	57.1
BERT_prediction	34.1	48.5	39.6	32.5	45.3	36.1	34.5	46.8	38.5	35.0	48.9	40.1	33.1	45.5	36.8	35.2	46.7	38.9
Trans	35.5	41.3	37.2	36.8	42.5	39.2	36.7	43.9	39.4	45.8	56.2	50.3	48.5	52.1	49.3	46.6	51.6	48.3
AutoPrompt	75.3	77.9	72.5	65.9	52.6	51.5	72.6	60.3	63.8	78.6	78.1	76.4	67.3	58.5	53.2	75.3	62.5	66.0
RI-BERT	79.3	80.5	75.8	70.1	53.0	53.2	77.4	65.8	68.3	80.7	80.1	79.5	70.1	55.7	55.9	79.5	67.2	70.4
IST	84.2	82.9	80.1	72.5	54.8	58.9	81.3	67.8	71.0	85.3	84.2	82.6	73.8	58.5	60.8	82.5	70.1	73.2
Table 1: Performance on three benchmarks when  $N = 3$  and  $N = 5$ .
sentences that mention entity pairs from the English Wikipedia corpus to construct training data as in our approach. For an entity pair  $(s,t)$ ,  $K$  sentences are used to predict its relation. If the average prediction score  $\overline{S}_K \geq \theta$ ,  $(s,t)$  is predicted to have relation  $r$ .  $\theta$  is a threshold and set to 0.7 in our experiments for its best performance.
The third kind is using the relational knowledge from PLMs, such as RI-BERT (Bouraoui et al., 2020), AutoPrompt(Shin et al., 2020) $^5$  and  $\mathrm{BERT}_{\text {predict }}$ .
RI-BERT induces relational knowledge from BERT, and our approach would degenerate to it when not using attention maps as the surface-agnostic features and not using  $score_{IG}$  to refine the templates. We implement the method by ourselves since there is no open source.
AutoPrompt tries to elicit knowledge from PLM using automatically-constructed prompts. Here, We generate templates with AutoPrompt for each relation. Since there is only one template can be generated for each relation, we use a threshold-based method to determine whether a new entity pair  $(x,y)$  has a relation. When  $p(x,y) > \delta$ , the prediction would be positive. Here,  $\delta = 0.8$  is the best threshold in our experiments.