SlowGuess/NAACL_2022_OCR huggingface dataset:
BERTpredict is a simple baseline proposed by ourselves. After  $K$  templates are selected with scoreBERT, we directly use BERT mask-prediction task to judge relation. Specifically, for an entity pair  $(s,t)$  and a template  $\tau$ , if BERT can predict  $\tau(s, -)$  or  $\tau(-, t)$ , the score of  $(s,t)$  will be increased by 1. The max score is  $2K$ , so if the score of  $(s,t) \geq \epsilon \cdot 2K$ ,  $(s,t)$  is predicted to have the relation r.  $\epsilon$  is a threshold and set to 0.7 for its best performance.
The main experimental results on the three aforementioned benchmarks are shown in Table 1, which reports the micro-average of precision, recall and F1 of our approach IST and other state-of-the-art methods when  $N = 3$  and  $N = 5$ .
From the table, there are several observations drawn from different aspects. (1) Our approach IST achieves the best performance against all other kinds of methods. (2) Pre-trained word embedding-based approaches such as  $\mathrm{MLP}_{sg}$  and  $\mathrm{MLP}_{gl}$  performance poorly, which proves that only few labeled entity pairs will degrade these approaches greatly. And Translation does not turn out well because of the lack of entities to construct representative subspaces. (3) The relational knowledge directly drawn from BERT also contains much noise according to the results of  $\mathrm{BERT}_{predict}$ . (4) Traditional distant-supervised approaches which don't resort to PLM suffer from the noisy and sparse bag issues, although PCNN-BagATT uses intra-bag and inter-bag attention to handle sentence and bag-level noise, and get better performance, they are still not suitable for the minimally-supervised relation induction task. (5) AutoPrompt and RI-BERT use proper prompts or templates from BERT, so they can obtain a better performance. However, they did not consider the generalization problem in the minimally-supervised setting. In addition, they ignored the contribution of each token in a sentence for expressing the relation, especially for the entity pairs, but only considered whether a sentence is natural or not according to BERT. (6) More labeled entity pairs can achieve better performance by comparing the results of  $N = 3$  and  $N = 5$ . This
$^{5}$ https://github.com/ucinlp/autoprompt
phenomenon is reflected by all methods in both three datasets.
5.2 Ablation Study and Analysis
Performance of Different Relations To further explore the performance of different relations, we show the detailed results of each relation in GATS in Table 2.
From the table, we can see that our approach achieves better performance for both semantic and morphology relations. Moreover, the iteratively template selection can bring a significant improvement, especially for semantic relations. As to morphological relations, the improvement is not so evident. This is because the entities in morphological relations are always adverbs or adjectives to which little attention is paid, so  $H_{\tau(s,t)}^{att}$  plays a limited role.
GATS	RI-BERT	T=0	T=1	T=2
Semantic	currency	56.7	58.8	58.6	59.5
family	76.9	78.8	78.4	79.9
capital-common	88.4	87.3	85.7	91.6
city-in-state	68.2	71.0	73.1	75.2
capital-world	77.3	76.8	78.0	78.2
Average	73.5	74.5	74.7	76.9
Morphology	adj-to-adv	39.1	38.8	42.3	44.8
opposite	55.3	59.7	54.0	56.6
comparative	90.9	87.5	88.2	89.0
superlative	78.1	79.8	80.6	77.7
presen-participle	98.4	96.2	98.1	98.9
nationality-adj	91.5	92.4	91.7	92.1
past-tense	96.9	97.8	97.2	97.0
plural	93.8	91.6	96.6	95.8
plural-verb	100	99.0	99.7	99.7
Average	82.6	82.6	83.2	83.5
Performance of attention weights and IG To investigate the effectiveness of BERT attention weights and IG, we compare the performance of several variants of our approach on GATS.
To reduce the effect of BERT attention weights, the representation of sentence  $\tau(s,t)$  is simplified from  $H_{\tau(s,t)}^{cls} \oplus H_{\tau(s,t)}^{att}$  to  $H_{\tau(s,t)}^{cls}$ . In addition, without IG, there would be no iterative template selection procedure. The results are shown in Table 3, and the performance drops in all variants, which proves that both attention maps and integrated gradient are useful in our approach.
Different Number of Templates To analyze the impacts of the number of templates  $(K)$ , we conduct experiments with different numbers of templates, and the results are shown in Table 4. From
Table 2: Detailed experimental results (F1) for each relation on GATS. T denotes the iteration number
GATS	T=0	T=1	T=2	T=3
IST	79.7	80.2	81.1	82.6
w/o att	79.5	79.7	80.6	80.9
w/o IG	78.4	78.4	78.4	78.4
the table, we find that more templates can bring better performance in all iterations. However, if  $K$  is too large, the time consumption will be greater and some unsuitable templates will be retained, leading to worse results.
Table 3: The F1 scores of IST and other variants on GATS with different iterations.
K=5	K=10	K=20
T=0	72.5	75.3	79.7
T=1	73.8	78.5	80.2
T=2	75.6	78.9	81.1
Table 4: F1-score with different number of templates  $\left( {K = 5,{10},{20}}\right)$  and different iterations  $\left( {T = 0,1,2}\right)$  on GATS.
Figure 2: F1 scores with different numbers of seed entity pairs  $(\mathbf{N} = \{2,3,5,10\})$  of our approach on GATS
Different Number of Seeds We evaluate our approach with different numbers of seed entity pairs  $(N)$ , and the results are shown in Figure 2. From the figure, we can see that F1 score increases gradually until convergence for all iterations. Our approach already achieves a satisfactory result when  $N = 5$ .
Effect of Balance Coefficient The parameter  $\alpha \in [0,1]$  is a balance coefficient between score  $BERT$  and score  $IG$  for template scoring. Larger  $\alpha$  will consider score  $IG$  more in the scoring. We conduct the experiments with different  $\alpha$  on GATS, and the results are shown in Figure 3. From the figure, we find that our approach achieves the best performance when  $\alpha = 0.5$ .
T=0	T=3
The Government of _ denoted 300 million _ to finance the school's construction in 1975.	The _ (, plural: / , ) is the currency of _.
Currently, _ uses the _ as its national currency.	This was one of the reasons for naming the current currency of the Republic of _ the _.
Following the introduction of the euro, the _ was linked to the euro, until January 1, 2015, when _ officially adopted the euro as its currency.	The _ (; ; sign: ; code: KHR) is the currency of _.
AutoPrompt: _ cial largest greenwich _.
Table 5: Case study for relation currency, where top 3 templates are exhibited with different approaches.
Figure 3: F1 scores with  $\alpha = \{0, 0.3, 0.5, 0.7, 0.9\}$  of our approach on GATS
Case Study Table 5 compares the selected templates of relation currency between  $T = 0$  and  $T = 3$ . From human's intuition, we find that comparing to  $(T = 3)$ , the templates filtered out only with scoreBERT(T = 0) are more ambiguous that they might indicate a co-occurrence relationship rather than relation currency. For example, for the first template "The Government of _ denoted 300 million _ to finance the school's construction in 1975.", it is natural for the government of a country to denote their own currency or just use dollar to evaluate how much they have denoted. So  $\tau(s, dollar)$  is natural when s denotes any country. This is due to the way of selecting templates that only requires the templates is proper for all  $(s, t) \in P_r$  without explicitly declaring what the relation is. In fact, the model can distinguish co - occurrence and currency only after the BERT is fine-tuned with negative examples. As to the template generated by AutoPrompt, it is a combination of some tokens rather than a human-readable sentence. Although AutoPrompt got good results on some tasks(Shin et al., 2020), the template is totally not interpretable from human's perspective.
6.1 Relation Induction
Relation induction was first proposed in (Vylomova et al., 2016). They used the vector difference between two entities to represent the relation between them. More researches on the relation induction with word embeddings were proposed in(Drozd et al., 2016; Bouraoui et al., 2018; Vu and Shwartz, 2018). They pointed out that the difference is not the best way to express the relationship and proposed more complicated methods to better extract relational knowledge between word embeddings.
6.2 Knowledge Induction from BERT
BERT was proven to be able to capture relational knowledge(Kim et al., 2020; Bouraoui et al., 2020; Ushio et al., 2021; Chen et al., 2021). Inspired by this, some works tried to use BERT on the relation induction task (Shin et al., 2020; Bouraoui et al., 2020; Jiang et al., 2020). The key point of these methods is to fill entities in the proper templates.
Recently, many efforts focus on the generation of templates. Jiang et al. (2020) proposed a template generation strategy based on paraphrasing aiming to improve lexical diversity while remaining relatively faithful to the original prompt. Shin et al. (2020) proposed AutoPrompt method to generate templates, or as they called, prompts, from nothing instead of from corpus. They automated create prompts based on gradient-guided search.
In this paper, we propose a novel minimally-supervised relation induction approach. Our proposed approach can iteratively select proper templates using score $_{IG}$  and socre $_{BERT}$ , and obtain a good generalized ability with surface-agnostic features based on attention maps of BERT. Experiments illustrate that our approach achieves state-of-the-art performance on three standard benchmarks.
This work is supported by the Key Research and Development Program of Zhejiang Province, China (No. 2021C01013), the National Key Research and Development Project of China (No. 2018AAA0101900), CKCEST, and MOE Engineering Research Center of Digital Library.
Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the Fifth ACM Conference on Digital Libraries, DL '00, page 85-94, New York, NY, USA. Association for Computing Machinery.
Christoph Alt, Marc Hübner, and Leonhard Hennig. 2019. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1388-1398, Florence, Italy. Association for Computational Linguistics.
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from bert. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7456-7463.
Zied Bouraoui, Shoaib Jameel, and Steven Schockaert. 2018. Relation induction in word embeddings revisited. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1627-1637, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Jose Camacho-Collados, Luis Espinosa Anke, and Steven Schockaert. 2019. Relational word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3286-3296, Florence, Italy. Association for Computational Linguistics.
Catherine Chen, Kevin Lin, and Dan Klein. 2021. Constructing taxonomies from pretrained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4687-4700, Online. Association for Computational Linguistics.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Leyang Cui, Sijie Cheng, Yu Wu, and Yue Zhang. 2020. Does bert solve commonsense task via commonsense knowledge? ArXiv, abs/2008.03945.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. 2016. Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3519-3530, Osaka, Japan. The COLING 2016 Organizing Committee.
Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multi-hop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295-1309, Online. Association for Computational Linguistics.
Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016. Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn't. In Proceedings of the NAACL Student Research Workshop, pages 8-15, San Diego, California. Association for Computational Linguistics.
Xiaotao Gu, Zihan Wang, Zhenyu Bi, Yu Meng, Liyuan Liu, Jiawei Han, and Jingbo Shang. 2021. Ucphrase: Unsupervised context-aware quality phrase tagging. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. 2020. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. *ArXiv*, abs/2002.00737.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
pages 7871-7880, Online. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In NeurIPS.
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In ICLR.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011, Suntec, Singapore. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Deepak Ravichandran and Eduard Hovy. 2002. Learning surface text patterns for a question answering system. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 41-47, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the
2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. *ArXiv*, abs/1703.01365.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Collados. 2021. BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3609-3624, Online. Association for Computational Linguistics.
Tu Vu and Vered Shwartz. 2018. Integrating multiplicative features into supervised distributional methods for lexical entailment. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 160-166, New Orleans, Louisiana. Association for Computational Linguistics.
Ivan Vulić and Nikola Mrkšić. 2018. Specialising word vectors for lexical entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1134-1145, New Orleans, Louisiana. Association for Computational Linguistics.
Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2016. Take and took, gaggle and goose, book and read: Evaluating the utility of vector differences for lexical relation learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1671-1682, Berlin, Germany. Association for Computational Linguistics.
Yile Wang, Leyang Cui, and Yue Zhang. 2020. Does chinese bert encode word structure? In COLING.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Distant supervision relation extraction with intra-bag and inter-bag attentions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
2810-2819, Minneapolis, Minnesota. Association for Computational Linguistics.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1753-1762, Lisbon, Portugal. Association for Computational Linguistics.