BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models
Year: 2023

Abstract:
It is crucial to automatically construct knowledge graphs (KGs) of diverse new relations to support knowledge discovery and broad applications. Previous KG construction methods, based on either crowdsourcing or text mining, are often limited to a small predefined set of relations due to manual cost or restrictions in text corpus. Recent research proposed to use pretrained language models (LMs) as implicit knowledge bases that accept knowledge queries with prompts. Yet, the implicit knowledge lacks many desirable properties of a full-scale symbolic KG, such as easy access, navigation, editing, and quality assurance. In this paper, we propose a new approach of harvesting massive KGs of arbitrary relations from pretrained LMs. With minimal input of a relation definition (a prompt and a few shot of example entity pairs), the approach efficiently searches in the vast entity pair space to extract diverse accurate knowledge of the desired relation. We develop an effective search-and-rescore mechanism for improved efficiency and accuracy. We deploy the approach to harvest KGs of over 400 new relations from different LMs. Extensive human and automatic evaluations show our approach manages to extract diverse accurate knowledge, including tuples of complex relations (e.g., "A is capable of but not good at B"). The resulting KGs as a symbolic interpretation of the source LMs also reveal new insights into the LMs’ knowledge capacities.

heading: 1 Introduction
text: Symbolic knowledge graphs (KGs) are a powerful tool for indexing rich knowledge about entities and their relationships, and are useful for information access (Google, 2012), decision making (Yang et al., 2021; Santos et al., 2022), and improving machine learning in general (Li et al., 2019; Wang et al., 2019; Tan et al., 2020; Xiong et al., 2017).
∗Equal contribution. Code available at https://github. com/tanyuqian/knowledge-harvest-from-lms. Demo available at https://lmnet.io
It has been a long-term desire to construct KGs of diverse relations to comprehensively characterize the structures between entities. The traditional crowdsourcing-based approach (Speer et al., 2017; Fellbaum, 2000; Sap et al., 2019) tends to cover only a restricted relation set, such as ConceptNet (Speer et al., 2017) that contains a small set of 34 relations. The popular method based on text mining (Luan et al., 2019; Zhong and Chen, 2020; Wang et al., 2021b) has a similar limitation, as the text understanding models can often recognize only a predefined set of relations included in training data. Some open-schema text mining approaches (e.g., based on syntactic patterns) exist (Tandon et al., 2014; Romero et al., 2019; Zhang et al., 2020b; Nguyen et al., 2021), yet the extracted relations are limited to those explicitly stated in the text, missing all others that are not mentioned or do not have exact match with the text in the corpus. Similarly, KG completion approaches (Bordes et al., 2013; Bosselut et al., 2019; Yao et al., 2019) is restricted
5000
to the preexisting relations (Figure 1). On the other hand, large language models (LMs) pretrained on massive text corpus, such as BERT (Devlin et al., 2019) and GPT-3 (Brown et al., 2020), have been found to encode a significant amount of knowledge implicitly in their parameters. Recent research attempted to use LMs as flexible knowledge bases by querying the LMs with arbitrary prompts (e.g., "Obama was born in " for the answer "Hawaii") (Petroni et al., 2019). However, such implicit query-based knowledge falls short of many desirable properties of a full-scale KG such as ConceptNet (AlKhamissi et al., 2022), including easy access, browsing, or even editing (Zhu et al., 2020; Cao et al., 2021), as well as assurance of knowledge quality thanks to the symbolic nature (Anderson et al., 2020). Symbolic Knowledge Distillation (SKD, West et al., 2022) explicitly extracts a knowledge base from GPT-3. However, the approach exclusively relies on the strong in-context learning capability of GPT-3 and thus is not applicable to other rich LMs such as BERT (Devlin et al., 2019) and ROBERTA (Liu et al., 2019). Moreover, its use of a quality discriminator trained on existing KGs can limit its generalization to new relations not included in the training data.
In this paper, we propose a new approach of harvesting massive KGs of arbitrary new relations from any pretrained LMs. Given minimal user input of a relation definition, including a prompt and a few shot of example entity pairs, our approach automatically searches within the LM to extract an extensive set of high-quality knowledge about the desired relation. To ensure search efficiency in the vast space of entity pairs, we devise an effective search-and-rescore strategy. We also adapt the previous prompt paraphrasing mechanism (Jiang et al.,
1"NER", "CR", "RE" refer to "named entity recognition", "coreference resolution", "relation extraction", respectively.
2SKD has an optional filter that requires existing KG to finetune, which doesn’t work for arbitrary relations.
2020; Newman et al., 2021) and enhance with our new rescore strategy for prompt weighting, leading to consistent and accurate outcome knowledge.
We apply our approach on a range of LMs of varying capacities, such as ROBERTA, BERT, and DISTILBERT. In particular, we harvest knoweldge of over 400 new relations (an order of magnitude more than ConceptNet relations) not available in preexisting KGs and previous extraction methods. Extensive human and automatic evaluations show our approach successfully extracts diverse accurate knowledge, including tuples for complex relations such as “A is capable of, but not good at, B” and 3-ary relations such as “A can do B at C”. Interestingly, the resulting KGs also serve as a symbolic interpretation of the source LMs, revealing new insights into their knowledge capacities in terms of varying factors such as model size, pretraining strategies, and distillation.

heading: 2 Related Work
text: Knowledge graph construction Popular knowledge bases or KGs are usually constructed with heavy human labor. For example, WordNet (Fellbaum, 2000) is a lexical database that links words into semantic relations; ConceptNet (Speer et al., 2017) is a large commonsense knowledge graph presented as a set of knowledge triples; ATOMIC (Sap et al., 2019) is a crowd-sourced social commonsense KG of if-then statements. Recently, Automatic Knowledge Base Construction (AKBC) as a research focus has led to various approaches (summarized in Table 1). Text mining-based works aim for knowledge extraction from text. A typical information extraction system (Angeli et al., 2015) is composed of several sub-tasks like coreference resolution, named entity recognition, and relationship extraction. Some works on commonsense knowledge extraction include WebChild (Tandon et al., 2014), TransOMCS (Zhang et al., 2020a), DISCOS (Fang et al., 2021), Quasimodo (Romero et al., 2019), ASCENT (Nguyen et al., 2021). These ex-
traction pipelines are based on linguistic pattern, and involve complex engineering such as corpus selection, term aggregation, filtering, etc. Recent attempts also utilize LMs for AKBC. Wang et al. 2021a finetuned LMs for link prediction. Feldman et al. 2019; Bouraoui et al. 2020 utilized LMs to score entity pairs collected from the Internet or missing edges in existing KGs. COMET (Bosselut et al., 2019) is a generative LM trained to predict tail entities given head entities and relations. West et al. 2021 distill the knowledge in GPT-3 to a generative LM. By prompting GPT-3 (Brown et al., 2020) with examples, they produced ATOMIC10x to teach the student model. Yet, this method requires the strong few-shot learning ability of GPT-3 and is not generally applicable to most LMs. To the best of our knowledge, our framework is the first to construct a KG by extracting purely from an LM (with the minimal definition of relations as input). The new paradigm can also be seen as optimizing a symbolic KG with (pretrained) neural models as supervision (Hu and Xing, 2022), which inverts the conventional problem of using symbolic knowledge to learn neural networks (Hu et al., 2016).
LMs as knowledge bases Another line of works attempted to use LMs as knowledge bases (LAMA, Petroni et al. 2019). These works are also known as factual probing because they measured how much knowledge is encoded in LMs. This is usually implemented by prompting methods and leveraging the masked LM pretraining task. LPAQA (Jiang et al., 2020) proposes to use text mining and paraphrasing to find and select prompts to optimize the prediction of a single or a few correct tail entities, instead of extensively predicting all the valid entity pairs like in our framework. AutoPrompt (Shin et al., 2020), Qin and Eisner, 2021 and OPTIPrompt (Zhong et al., 2021) learn discrete or continuous prompts automatically with an additional training set. Though making prompts unreadable, these methods achieve higher accuracy on the knowledge probing tasks. Our framework differs from these works in that we aim to explicitly harvest knowledge graphs instead of measuring the knowledge in a simplified setting.
Consistency of LMs Consistency is a significant challenge for LMs, which stresses that they should not produce conflicting predictions across inference sessions. For example, models should be-
have invariantly under inputs with different surface forms but the same meaning. Elazar et al. 2021 analyzed the consistency of pretrained LMs with respect to factual knowledge. Jiang et al. 2020 used paraphrasing to improve factual probing. Newman et al. 2021 trains an additional layer on top of word embedding to improve consistency. Recently, consistency is also shown helpful to improve the reasoning ability of large LMs (Wang et al., 2022; Jung et al., 2022; Hao et al., 2023). In our framework, the extracted entity pairs for each relation are enforced to consistently satisfy a diverse set of prompts and regularized by several scoring terms.

heading: 3 Harvesting KGs from LMs
text: This section presents the proposed framework for extracting a relational KG from a given pretrained LM, where the LM can be arbitrary fillin-the-blank models such as BERT (Devlin et al., 2019), ROBERTA (Liu et al., 2019), BART (Lewis et al., 2020), or GPT-3 (with appropriate instructions) (Brown et al., 2020). The KG consists of a set of knowledge tuples in the form ⟨HEAD ENTITY (h), RELATION (r), TAIL ENTITY (t)⟩. Our approach utilizes the LM to automatically harvest a large number of appropriate entity pairs (h1, t1), (h2, t2), . . ., for every given relation r. This presents a more challenging problem than traditional LM probing tasks, which typically predict a single tail entity or a small number of valid tail entities given a head entity and relation.
Our approach for extracting knowledge tuples of a specific relation of interest, such as "potential_risk" as depicted in Figure 2, only requires minimal input information that defines the relation. This includes an initial prompt, such as "The potential risk of A is B" and a small number of example entity pairs, such as ⟨EATING CANDY, TOOTH DECAY⟩. The prompt provides the overall semantics of the relation, while the example entity pairs clarify possible ambiguities. For new relations not included in existing KGs, it is impractical to require a large set (e.g., hundreds) of example entity pairs as in previous knowledge probing or prompt optimization methods (Petroni et al., 2019; Jiang et al., 2020; Shi et al., 2019; Zhong et al., 2021). In contrast, our approach necessitates only a small number of example entity pairs, for example, as few as 2 in our experiments, which can easily be collected or written by users.
In the following sections, we describe the core
components of our approach, namely the automatic creation of diverse prompts with confidence weights (§3.1) and the efficient search to discover consistent entity pairs (§3.2) that compose the desired KGs. Figure 2 illustrate the overall framework.

heading: 3.1 Creating Diverse Weighted Prompts
text: Our automated approach utilizes input information, specifically the initial prompt and several example entity pairs, to generate a set of semantically consistent but linguistically diverse prompts for describing the relation of interest. The generated prompts are assigned confidence weights to accurately measure consistency of knowledge in the subsequent step (§3.2).
To generate diverse prompts for a desired relation, we begin by randomly selecting an entity pair from a example set and inserting it into an initial prompt to form a complete sentence. This sentence is then passed through an off-the-shelf text paraphrase model, which produces multiple paraphrased sentences with the same meaning. By removing the entity names, each paraphrased sentence results in a new prompt that describes the desired relation. To ensure a wide range of expressions of the relation, we retain only those prompts that are distinct from one another in terms of edit distance. This process is repeated by continuously paraphrasing the newly created prompts until a minimum of 10 prompts for the relation have been collected.
The automatic generation of prompts can be imprecise, resulting in prompts that do not accurately
convey the intended relation. To mitigate this, we propose a reweighting method that utilizes compatibility scores to calibrate the impact of each prompt in the subsequent knowledge search step. Specifically, we evaluate the compatibility of new prompts with example entity pairs by measuring the likelihood of the prompts under a LM, considering both the individual entities and the entity pair as a whole. This allows us to determine the appropriate weights for each prompt and improve the precision of the knowledge search process. Formally, the compatibility score between an entity pair (h, t) and a prompt p can be written as:
fLM(⟨h, t⟩, p) = α logPLM(h, t | p) + (1− α)min {logPLM(h | p), logPLM(t | p, h)}
(1)
where the first term is the joint log-likelihood under the LM distribution PLM , the second term is the minimum individual log-likelihood given the prompt (and the other entity), and α is a balancing factor (α = 2/3 in our experiments). We compute the average compatibility score of each created prompt over all example entity pairs, and the weight of the prompt is then defined as the softmaxnormalized score across all prompts.

heading: 3.2 Efficient Search for Consistent Knowledge
text: With the set of prompts and corresponding confidence weights obtained in the steps described in Section 3.1, we proceed to search entity pairs that consistently align with all prompts. To guide the searching process and evaluate the compatibility of searched-out entity pairs (hnew, tnew), we reuse the previously defined prompt/entity-pair compati-
bility function (Eq.1), and intuitively define consistency as the weighted average of its compatibility with the various prompts, i.e., consistency((hnew, tnew)) = ∑
p wp·fLM((hnew, tnew), p)
(2)
where wp is the prompt weight and the sum is over all automatically created prompts as above, so that entity pairs compatible with all prompts are considered to be consistent.
Based on the consistency criterion, we develop an efficient search strategy to search for consistent entity pairs. A straightforward approach involves enumerating all possible pairs of entities, calculating their respective consistency scores, and selecting the top-K entity pairs with the highest scores as the resulting knowledge. However, this approach can be computationally expensive due to the large vocabulary size V (e.g., V = 50, 265 for ROBERTA) and the high time complexity of the enumeration process (i.e., O(V 2) even when each entity consists of only one token). To overcome this limitation, we have proposed an appropriate approximation that leads to a more efficient search and re-scoring method. Specifically, we first use the minimum individual log-likelihoods (i.e., the second term in the compatibility score Eq.1) weighted averaged across different prompts (similar as in Eq.2), to propose a large set of candidate entity pairs. The use of the minimum individual log-likelihoods allows us to apply pruning strategies, such as maintaining a heap and eliminating entities ranked outside top-K in every single searching step. Once we have collected a large number of proposals, we re-rank them using the full consistency score in Eq.2 and select the top-K instances as the output knowledge. We describe more nu-
anced handling in the search procedure (e.g., the processing of multi-token entities, detailed pruning strategies) in the appendix.
Generalization to complex relations Most existing KGs or knowledge bases include relations that are predicates connecting two entities, e.g., "A is capable of B". However, many real-life relations are more complex. Our approach is flexible and easily extensible to extract knowledge about these complex relations. We demonstrate this in our experiments by exploring two cases: (1) highly customized relations that have specific and sophisticated meanings, such as "A is capable of, but not good at, B". This type of sophisticated knowledge is often difficult for humans to write down on a large scale. Our automatic approach naturally supports harvesting this kind of knowledge given only an initial prompt and a few example entities that can be collected easily, e.g., ⟨DOG, SWIM⟩, ⟨CHICKEN, FLY⟩, etc.; (2) N-ary relations involving more than two entities, such as "A can do B at C". Our approach can straightforwardly be extended to handle n-ary relations by generalizing the compatibility score and search strategy accordingly to accommodate more than two entities.
Symbolic interpretation of neural LMs The harvested knowledge tuples, as consistently recognized across varying prompts by the LM, can be considered as the underlying "beliefs" of the LM about the world (Stich, 1979; Hase et al., 2021). These fully symbolic and interpretable tuples provide a means for easily browsing and analyzing the knowledge capabilities of the black-box LM. For example, via these outcome KGs, one can compare
different LMs to understand the performance impact of diverse configurations, such as model sizes and pretraining strategies, as demonstrated in our experiments.

heading: 4 Experiments
text: To evaluate our framework, we extract knowledge of diverse new relations from various language models, and conduct human evaluation. We then make deeper analysis of prompt creation and scoring function in our framework. Finally, by utilizing our framework as a tool to interpret the knowledge stored in language models, we have made noteworthy observations regarding the knowledge capacity of black-box models.

heading: 4.1 Setup
text: Relations We evaluate our framework with several relation sets: (1) ConceptNet (Speer et al., 2017): Following Li et al. 2016, we filter the KG and use a set of 20 common relations (e.g. HAS_SUBEVENT, MOTIVATED_BY_GOAL). The initial prompts for these relations are from the ConceptNet repository, and we randomly sample 5 example entity pairs from the ConceptNet KG for each relation. (2) LAMA (Petroni et al., 2019): Following previous works, we use the T-REx split (41 relations from WikiPedia, such as capital_of, member_of). For each relation, the human-written prompt provided in Petroni et al. 2019 is used as the initial prompt and we randomly sample 5 exam-
ple entity pairs for each relation. (3) Human: We write 12 new relations of interests that can hardly be found in any existing KGs, and manually write an initial prompt and 5 example entity pairs for them. The resulting relations include complex relations as described in Section 3.2. (4) Auto: Besides relations from existing KGs and human-written ones, we automatically derive a large set of relations from E-KAR (Chen et al., 2022), a dataset for analogical reasoning. In the original dataset, given an entity pair, e.g. ⟨ID_CARD, IDENTITY⟩, the task is to select an analogous tuple from multiple choices, e.g. ⟨PRACTICE LICENSE, QUALIFICATION⟩. To turn a sample in E-KAR into a relation, we use the tuple in the question and the correct choices as 2 example entity pairs, and extract the initial prompt from the explanation provided in E-KAR (e.g. Proof of A requires B.), resulting in 487 relations. Some of the relations are not straightforward, making this relation set more difficult than other ones. 3

heading: 4.2 Extracting Knowledge of Diverse New Relations
text: Our framework is applied to extract knowledge graphs from LMs with relations of ConceptNet, Auto, and Human. The accuracy of the extracted knowledge is then evaluated with human annotation using Amazon Mechanical Turk (MTurk). Each extracted knowledge tuple is la-
3For reference, finetuned ROBERTA-LARGE achieves about 50% accuracy on the original dataset.
beled for correctness by three annotators using a True/False/Unjudgeable judge. A tuple is considered "accepted" if at least two annotators deem it to be true knowledge, and "rejected" if at least two annotators rate it as false. Here we refers portion of accepted tuples as accuracy.
The statistics of our resulting KGs are listed in Table 2. Besides, we also put the results of other paradigms of methods, including COMET for KG completion and text-mining based methods (Figure 1). Note that the results across different paradigms are generally not directly comparable due to vastly different settings. Yet we still collect the results together for reference purpose. From our RebertaNet with relation set "Auto", we are able to extract a reasonably large sets of knowledge (122K), by extracting knowledge with 487 easy-to-collect "Auto" relations. The set of relation is an order of magnitude larger than the predefined set of relations in both KG completion and text mining based on ConceptNet as shown in the table. The accuracy of 65% is at a comparable level with that of COMET (230K) and TransOMCS (18.4M), which is reasonable especially considering our method solely uses an LM as the source of knowledge without any external training data, bringing flexibility to dynamically incorporate new relations. Besides, for our RobertaNet on ConceptNet relations, although the numbers listed in the table are not simply comparable, we can still find that RobertaNet achieves similar accuracy and absolutely higher novelty comparing with the knowledge from COMET, which is already finetuned using large number of knowledge terms under the same set of ConceptNet relations. Further, our results on the "human" relation set demonstrate that our RobertaNet keeps working comfortably on our highly realistic relations of user interests, including the complex ones as described in section §3.2. We showcase knowledge samples harvested from DISTILLBERT in Figure 3.

heading: 4.3 Analyzing Automatic Prompt Creation
text: To evaluate the effect of our automatic creation of prompts, we compare the generated KGs under several settings on the Human relations: (1) MultiPrompts refers to the the full framework described in §3 which use the automatically created diverse prompts in knowledge search. (2) Top-1 Prompt: To ablate the effect of ensembling multiple prompts, we evaluate the variant that uses only the prompt with largest weight (§3.1) for knowledge extraction. (3) Human Prompt: To further understand the effectiveness of the automatically created prompts, we assess the variant that uses the initial prompt of each relation. (4) AutoPrompt (Shin et al., 2020), which was proposed to learn prompts by optimizing the likelihood of tail entity prediction on the training set. To fit in our setting, we adapt it to optimize the compatibility score (Eq.1) on the example entity pairs. We omit other prompt tuning work (e.g., Zhong et al., 2021; Qin and Eisner, 2021) because they either are difficult to fit in our problem or require more training data and fail with only the few shot of example entity pairs in our setting.
We harvest 1000 tuples for each Human relation, and evaluate them with human annotation. The annotation results are presented in Table 3 (We also list the detailed results per relation in Table 5 for reference) Our TOP-1 PROMPT significantly improves the accuracy up to 9% over the HUMAN PROMPT, demonstrating the effectiveness of our prompt searching algorithm in generating high-quality prompts. MULTI-PROMPTS further improves the accuracy by an additional 4%, indicating that the combination of diverse prompts better captures the semantics of a relation. However, the method utilizing the optimized prompt by AUTOPROMPT results in lower accuracy than the use of human or searched prompts. This can be attributed to the insufficient number of example knowledge tuples used to learn effective prompts for the desired relations.
Based on the results above, we move a step for-
ward to see how the created prompts influence the subsequent scoring module in the framework. Specifically, we study both the precision and recall of our scoring function parameterized by the prompts, to see if the automatically created prompts (§3.1) bring the consistency scoring (§3.2) better balance of knowledge accuracy (precision) and coverage (recall). To compare with other scoring methods that are restricted to specific sets of relations, this experiment was conducted using existing terms from both the ConceptNet and LAMA datasets.
Specifically, we use the knowledge tuples from ConceptNet and LAMA as positive samples (§4.1), and synthesize the same amount of negative samples with the same strategy in Li et al. (2016) by random replacing entities or relations in a true knowledge tuple. Each scoring function ranks the samples based on the scores from high to low. We can then compute both the precision and recall of positive samples at different cut-off points along the ranking, and plot the precision-recall curves for each method.
The automatic evaluation setting on given knowledge terms enables us to adapt existing prevalent works, e.g., KG completion and factual probing (Table 1), for comparison with our approach: (1) COMET (Bosselut et al., 2019) is a transformerbased KG completion model trained to predict the tail entity t conditioning on the head entity and relation (h, r) on ConceptNet. We use its loglikelihood logP (t|h, r) as the score for each given knowledge tuple. (2) LPAQA (Jiang et al., 2020) collects a set of prompts on LAMA with text mining and paraphrasing, and optimize their weights towards the objective of logP (t|h, r) on training samples.
The resulting precision-recall curves on ConceptNet and LAMA knowledge are shown in Figure 4
and Figure 5, respectively. Scoring with multiple prompts always achieves best performance, followed by Top-1 prompts and then Human-written prompts. The finding is consistent with previous experiments, which verified the effectiveness of our scoring function design. Our framework also outperforms other baselines, such as COMET on ConceptNet and LPAQA on LAMA. Though trained with labeled data, these methods are only optimized to completing a tail entity given a query, in stead of scoring an entity pair, which is essential to extract KGs from LMs.

heading: 4.4 Analysis of Knowledge in Different LMs
text: As previously mentioned in Section §3, the resulting knowledge graphs can be viewed as a symbolic interpretation of LMs. We extract knowledge graphs from 5 distinct language models and submit them to human annotation evaluation. The findings are presented in Table 4 (The detailed results per relation is listed in Table 5), which sheds some new light on several knowledge-related questions regarding the LMs’ knowledge capacity.
Does a larger LM encode better knowledge? The large version of BERT and RoBERTa have the same pretraining corpus and tasks as their base versions, but have larger model architecture in terms of layers (24 v.s. 12), attention heads (16 v.s. 12), and the number of parameters (340M v.s. 110M). We can see that the accuracies of BertNet-large and RoBERTaNet-large are around 7% and 3% higher than their base version, separately, indicating the larger models indeed encoded better knowledge than the base models.
Does better pretraining bring better knowledge? RoBERTa uses the same architecture as BERT but with better pretraining strategies, like dynamic masking, larger batch size, etc. In their corre-
sponding KGs from our framework, RoBERTaNetlarge performs better than BertNet-large (0.73 v.s. 0.70), and RoBERTaNet-base is also better than BertNet-base (0.70 v.s. 0.63), showing that the better pretraining in RoBERTa leads to better knowledge learning and storage.
Is knowledge really kept in the knowledge distillation process? DistilBERT is trained by distilling BERT-base, and it reduces 40% parameters from the latter. Interestingly, the knowledge distillation process instead improves around 4% of accuracy in the result knowledge graph. This should be attributed to the knowledge distillation process which might eliminate some noisy information from the teacher model.

heading: 5 Conclusion
text: We have developed an automatic framework that extracts a KG from a pretrained LM (e.g, BERT, ROBERTA), in an efficient and scalable way, resulting in a family of new KGs, which we refer to as BERTNET, ROBERTANET, etc. Our framework is capable of extracting knowledge of arbitrary new relation types and entities, without being restricted by pre-existing knowledge or corpora. The resulting KGs also serve as interpretation of source LMs.
Limitations Our current design and experimental studies are limited on LMs in the generic domain, and are not yet been studied in specific domains such as extracting healthcare knowledge from relevant neural models. We leave the exciting work of harvesting knowledge from various kinds of neural networks across applications and domains in the future work.
Ethical considerations In this work, the harvested knowledge is automatically generated by LMs. We would like to note that the language models could possibly generate unethical knowledge tuples, same with the risks of other applications using language models for generation. We hope that the knowledge extraction study could offer techniques to better interpret and understand the language models, and in turn foster the future research of language model ethics. Since the knowledge graph only consists simple phrases, we think filtering sensitive words would be effective. No foreseeable negative societal impacts are caused by the method itself.

heading: A Detailed Results of Harvested Knowledge
text: In Table 3 and Table 4, we show the humanannotated results of harvested knowledge in different settings. Here we list the detailed results per relation in Table 5.

heading: B Preprocessing of ConceptNet
text: We filter out some linguistic relations (e.g. etymologically derived from) and some trivial relations (e.g. related to). We only consider the tuples with confidence higher than 1, and filter out relations comprising less than 1000 eligible tuples. We don’t directly take the test set from (Li et al., 2016) because they reserve a lot of tuples for training, resulting in a small and unbalanced test set.

heading: C Efficient knowledge tuple search
text: In the candidate entity pairs proposal step, we use the minimum token log-likelihoods (shorted as MTL) instead of the full Equation 2, which allows us to apply a pruning strategy. The pseudo-code is shown in Algorithm 1. For simplicity of the pseudocode, we only include the case where each entity is composed of a single token. Appendix ?? illustrates the processing of multi-token entities. It’s worth noting that our algorithm is an exact search algorithm instead of approximated algorithms like beam search, which prevents the results from biasing towards more probable head entities.
As a running example, when we are searching for 100 entity tuples, we maintain a minimum heap to keep track of the MTL of the entity tuples. The maximum size of this heap is 100, and the heap top can be used as a threshold for future search because it’s the 100-th largest MTL: When we are searching for a new entity tuple, once we find the log-likelihood at any time step is lower than the threshold, we can prune the continuous searching immediately, because this means the MTL of this tuple will never surpass any existing tuples in the heap. If a new entity tuple is searched out without being pruned, we will pop the heap and push the MTL of the new tuple. Intuitively, the pruning process makes sure that the generated part of the tuple in searching is reasonable for the given prompt.
Algorithm 1 Efficient Entity Tuple Search Input: LM: A language model; nr: The entity number for
a tuple of relation r; N : maximum number of candidate tuples; Pr: The set of prompts describing relation r
Output: tuple_list: A list of N entity tuples heap←MinHeap() function DFS(cur_tuple, cur_MTL)
idx←Count(cur_tuple) if idx = nr then
heap.push((cur_tuple, cur_MTL)) if len(heap) > N then
heap.pop() end if
end if for v ∈ Vocab(LM) do
cur_L← log pLM (v|cur_tuple, Pr) cur_MTL = min(cur_L, cur_MTL) if Count(cur_tuple > 0) and cur_MTL < heap.top()
then return ▷ Pruning end if cur_tuple.append(v) DFS(cur_tuple, cur_MTL)
end for end function DFS(EmptyList(), 0) tuple_list← list(heap)

heading: D Detailed Experiment setting
text: We use GPT-3 with the instruction "paraphrase:sentence" with a few examples as the offthe-shelf paraphraser. In entity pair searching, we restrict every entity to appear no more than 10 times to improve the diversity of generated knowledge and search out at most 50,000 entity tuples for each relation. We finally use various score thresholds to get the outcome KGs in different scales, including (1) 50%: taking half of all searched-out entity pairs with higher consistency for each relation (2) base-k: Naturally, there are different numbers of valid tuples for different relations (e.g. tuples of ⟨ . . . , CAPITAL_OF, . . . ⟩ should not exceed 200 as that is the number of all the countries in the world). We design a relation-specific thresholding method, that is to set 10% of the k-th consistency as the threshold (i.e., 0.1 × consistencyk), and retain all tuples with consistency above the threshold. We name the settings base-10 and base-100 when k is 10 and 100, respectively. We list the truncation method applied to each variant of ROBERTANET listed in Table 2:
• RobertaNet (122.2k) - Auto: base-10
• RobertaNet (6.7K) - ConceptNet: base-10
• RobertaNet (24.3K) ConceptNet: base-100
• RobertaNet (230K) ConceptNet: 50%
• RobertaNet (2.2K) Human: base-10
• RobertaNet (7.3K) Human: base-100
• RobertaNet (23.6k) Human: 50%

heading: E Human evaluation
text: We present the screenshot of the instruction in Figure 7 and question in Figure 8. The inter-annotator agreement (Krippendorff’s Alpha) is 0.27, showing fair agreement.

heading: F Compute resource
text: All of our experiments are running on a single Nvidia GTX1080Ti GPU. Harvesting a knowledge graph of one relation with Roberta-large takes about one hour.

heading: G The license of the assets
text: All the data we used in this paper, including datasets, relation definitions, seed entity pairs, etc., are officially public resources.

heading: H Potential Risks
text: We identify that our system is minimal in risks. Our proposed system produce results only based on the source language models like BERT. The risks of language models are well studied and our methods do not perpetuate or add to the known risks. However, we acknowledge the methods could be applied to maliciously trained language models and discourage such uses.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section
3 A2. Did you discuss any potential risks of your work? Appendix F
3 A3. Do the abstract and introduction summarize the paper’s main claims? Yes. Abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 4.1
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4.1
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4.1
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4.2.2
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? appendix E
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? appendix E
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? appendix E
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? 4.2
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: The maximum size of this heap is 100, and the heap top can be used as a threshold for future search because it’s the 100-th largest MTL: When we are searching for a new entity tuple, once we find the log-likelihood at any time step is lower than the threshold, we can prune the continuous searching immediately, because this means the MTL of this tuple will never surpass any existing tuples in the heap. If a new entity tuple is searched out without being pruned, we will pop the heap and push the MTL of the new tuple. Intuitively, the pruning process makes sure that the generated part of the tuple in searching is reasonable for the given prompt. Algorithm 1 Efficient Entity Tuple Search Input: LM: A language model; nr: The entity number for
a tuple of relation r; N : maximum number of candidate tuples; Pr: The set of prompts describing relation r
Output: tuple_list: A list of N entity tuples heap←MinHeap() function DFS(cur_tuple, cur_MTL)
idx←Count(cur_tuple) if idx = nr then
heap. push((cur_tuple, cur_MTL)) if len(heap) > N then
heap. pop() end if
end if for v ∈ Vocab(LM) do
cur_L← log pLM (v|cur_tuple, Pr) cur_MTL = min(cur_L, cur_MTL) if Count(cur_tuple > 0) and cur_MTL < heap. top()
then return ▷ Pruning end if cur_tuple. append(v) DFS(cur_tuple, cur_MTL)
end for end function DFS(EmptyList(), 0) tuple_list← list(heap)