Text-Augmented Open Knowledge Graph Completion via Pre-Trained Language Models
Year: 2023

Abstract:
The mission of open knowledge graph (KG) completion is to draw new findings from known facts. Existing works that augment KG completion require either (1) factual triples to enlarge the graph reasoning space or (2) manually designed prompts to extract knowledge from a pre-trained language model (PLM), exhibiting limited performance and requiring expensive efforts from experts. To this end, we propose TAGREAL that automatically generates quality query prompts and retrieves support information from large text corpora to probe knowledge from PLM for KG completion. The results show that TAGREAL achieves state-of-the-art performance on two benchmark datasets. We find that TAGREAL has superb performance even with limited training data, outperforming existing embedding-based, graph-based, and PLM-based methods.

heading: 1 Introduction
text: A knowledge graph (KG) is a heterogeneous graph that encodes factual information in the form of entity-relation-entity triplets, where a relation connects a head entity and a tail entity (e.g., “Miamilocated_in-USA”) (Wang et al., 2017; Hogan et al., 2021). KG (Dai et al., 2020) plays a central role in many NLP applications, including question answering (Hao et al., 2017; Yasunaga et al., 2021), recommender systems (Zhou et al., 2020), and drug discovery (Zitnik et al., 2018). However, existing works (Wang et al., 2018; Hamilton et al., 2018) show that most large-scale KGs are incomplete and cannot fully cover the massive real-world knowledge. This challenge motivates KG completion, which aims to find one or more object entities given a subject entity and a relation (Lin et al., 2015). For example, in Figure 1, our goal is to predict the object entity with “Detroit” as the subject entity and “contained_by” as the relation.
However, existing KG completion approaches (Trouillon et al., 2016b; Das et al., 2018) have sev-
eral limitations (Fu et al., 2019). First, their performance heavily depends on the density of the graph. They usually perform well on dense graphs with rich structural information but poorly on sparse graphs which are more common in real-world applications. Second, previous methods (e.g., Bordes et al. (2013)) assume a closed-world KG without considering vast open knowledge in the external resources. In fact, in many cases, a KG is usually associated with a rich text corpus (Bodenreider, 2004), which contains a vast amount of factual data not yet extracted. To overcome these challenges we investigate the task of open knowledge graph completion, where KG can be constructed using new facts from outside the KG. Recent text-enriched solutions (Fu et al., 2019) focus on using a predefined set of facts to enrich the knowledge graph. Nonetheless, the pre-defined set of facts is often noisy and constricted, that is, they do not provide sufficient information to efficiently update the KG.
11161
Pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019a) have shown to be powerful in capturing factual knowledge implicitly from learning on massive unlabeled texts (Petroni et al., 2019b). Since PLMs are superb in text encoding, they can be utilized to facilitate knowledge graph completion with external text information. Recent knowledge graph completion methods (Shin et al., 2020; Lv et al., 2022) focus on using manually crafted prompts (e.g., “Detroit is located in [MASK]” in Figure 1) to query the PLMs for graph completion (e.g., “Michigan”). However, manually creating prompts can be expensive with limited quality (e.g., PLM gives a wrong answer “Canada” to the query with a handcrafted prompt, as shown in Figure 1).
Building on the above limitations of standard KG and the enormous power of PLMs (Devlin et al., 2019; Liu et al., 2019a), we aim to use PLMs for open knowledge graph completion. We propose an end-to-end framework that jointly exploits the implicit knowledge in PLMs and textual information in the corpus to perform knowledge graph completion (as shown in Figure 1). Unlike existing works (e.g., (Fu et al., 2019; Lv et al., 2022)), our method does not require a manually pre-defined set of facts and prompts, which is more general and easier to adapt to real-world applications.
Our contributions can be summarized as:
• We study the open KG completion problem that can be assisted by facts captured from PLMs. To this end, we propose a new framework TAGREAL that denotes text augmented open KG completion with real-world knowledge in PLMs.
• We develop prompt generation and information retrieval methods, which enable TAGREAL to automatically create highquality prompts for PLM knowledge probing and search support information, making it more practical especially when PLMs lack some domain knowledge.
• Through extensive quantitative and qualitative experiments on real-world knowledge graphs such as Freebase1 we show the applicability and advantages of our framework.2
1https://github.com/thunlp/OpenNRE 2Our code is available at: https://github.com/
pat-jj/TagReal

heading: 2 Related Work

heading: 2.1 KG Completion Methods
text: KG completion methods can be categorized into embedding-based and PLM-based methods. Embedding-based methods represent entities and relations as embedding vectors and maintain their semantic relations in the vector space. TransE (Bordes et al., 2013) vectorizes the head, the relation and the tail of triples into a Euclidean space. DistMult (Yang et al., 2014) converts all relation embeddings into diagonal matrices in bilinear models. RotatE (Sun et al., 2019) presents each relation embedding as a rotation in complex vector space from the head entity to the tail entity.
In recent years, researchers have realized that PLMs can serve as knowledge bases (Petroni et al., 2019a; Zhang et al., 2020; AlKhamissi et al., 2022). PLM-based methods for KG completion (Yao et al., 2019; Kim et al., 2020; Chang et al., 2021; Lv et al., 2022) start to gain attention. As a pioneer, KG-BERT (Yao et al., 2019) fine-tunes PLM with concatenated head, relation, and tail in each triple, outperforming the conventional embedding-based methods in link prediction tasks. Lv et al.(2022) present PKGC, which uses manually designed triple prompts and carefully selected support prompts as inputs to the PLM. Their result shows that PLMs could be used to substantially improve the KG completion performance, especially in the open-world (Shi and Weninger, 2018) setting. Compared to PKGC, our framework TAGREAL automatically generates prompts of higher quality without any domain expert knowledge. Furthermore, instead of pre-supposing the existence of support information, we search relevant textual information from the corpus with an information retrieval method to support the PLM knowledge probing.

heading: 2.2 Knowledge Probing using Prompts
text: LAMA (Petroni et al., 2019a) is the first framework for knowledge probing from PLMs. The prompts are manually created with a subject placeholder and an unfilled space for the object. For example, a triple query (Miami, location, ?) may have a prompt “Miami is located in [MASK]” where “<subject> is located in [MASK]” is the template for “location” relation. The training goal is to correctly fill [MASK]with PLM’s prediction. Another work, BertNet (Hao et al., 2022), proposes an approach applying GPT-3 (Brown et al., 2020)
to automatically generate a weighted prompt ensemble with input entity pairs and a manual seed prompt. It then uses PLM again to search and select top-ranked entity pairs with the ensemble for KG completion.

heading: 2.3 Prompt Mining Methods
text: When there are several relations to interpret, manual prompt design is costly due to the requirement of domain expert knowledge. In addition, the prompt quality could not be ensured. Hence, quality prompt mining catches the interest of researchers. Jiang et al. 2020 propose an approach MINE which searches middle words or dependency paths between the given inputs and outputs in a large text corpus (e.g., Wikipedia). They also propose a reasonable approach to optimize the ensemble of the mined prompts by weighting prompt individuals regarding their performance on the PLM.
Before the emergence and widespread use of PLMs, textual pattern mining performed a similar function to find reliable patterns for information extraction. For instance, MetaPAD (Jiang et al., 2017) generates quality meta patterns by context-aware segmentation with the pattern quality function, and TruePIE (Li et al., 2018) proposes the concept of pattern embedding and a self-training framework, that discovers positive patterns automatically.

heading: 3 Methodology
text: We propose TAGREAL, a PLM-based framework to handle KG completion tasks. In contrast to the previous work, our framework does not rely on handcrafted prompts or pre-defined relevant facts. As shown in Figure 2, we automatically create appropriate prompts and search relevant support information, which are further utilized as templates to explore implicit knowledge from PLMs.

heading: 3.1 Problem Formulation
text: Knowledge graph completion is to add new triples (facts) to the existing triple set of a KG. There are two tasks to achieve this goal. The first is triple classification, which is a binary classification task to predict whether a triple (h, r, t) belongs to the KG, where h, r, t denote head entity, relation and tail entity respectively. The second task is link prediction, which targets on predicting either the tail entity t with a query (h, r, ?) or the head entity h with a query (?, r, t).

heading: 3.2 Prompt Generation
text: Previous studies (e.g., Jiang et al. (2020)) demonstrate that the accuracy of relational knowledge extracted from PLMs heavily relies on the quality of prompts used for querying. To this end, we develop a comprehensive approach for automatic quality prompt generation given triples in KG as the only input, as shown in Figure 3. We use textual pattern mining methods to mine quality patterns from large corpora as the prompts used for PLM knowledge probing. As far as we know, we are pioneers in using textual pattern mining methods for LM prompt mining. We believe in the applicability of this approach for the following reasons.
• Similar data sources. We apply pattern mining on large corpora (e.g., Wikipedia) which are the data sources where most of PLMs are pre-trained.
• Similar objectives. Textual pattern mining is to mine patterns to extract new information from large corpora; prompt mining is to mine prompts to probe implicit knowledge from PLMs.
• Similar performance criteria. The reliability of a pattern or a prompt is indicated by how many accurate facts it can extract from corpora/PLMs.
Sub-corpora mining is the first step that creates the data source for the pattern mining. Specifically, given a KG with a relation set R = (r1, r2, ..., rk), we first extract tuples Tri paired by head entities and tail entities for each relation ri ∈ R from the KG. For example, for the relation r1: /business/company/founder, we extract all tuples like <microsoft, bill_gates> in this relation from the KG. For each tuple tj , we then search sentences stj containing both head and tail from a large corpus (e.g., Wikipedia) and other reliable sources, which is added to compose the sub-corpus Cri . We limit the size of each set to θ
for each tuple to mine more generic patterns for future applications.
Phrase segmentation and frequent pattern mining are applied to mine patterns from subcorpora as prompt candidates. We use AutoPhrase (Shang et al., 2018) to segment corpora to more natural and unambiguous semantic phrases, and use FP-Growth algorithm (Han et al., 2000) to mine frequent appeared patterns to compose a candidate set P ′ri = (p ′ 1, p ′ 2, ..., p ′ m). The size of the set is large, as there are plenty of messy textual patterns.
Prompt selection. To select quality patterns from the candidate set, we apply two textual mining approaches: MetaPAD (Jiang et al., 2017) and TruePIE (Li et al., 2018). MetaPAD applies pattern quality function introducing several criteria of contextual features to estimate the reliability of a pattern. We explain why those features can also be adapted for LM prompt estimation: (1) Frequency and concordance: Since a PLM learns more contextual relations between frequent patterns and entities during the pre-training stage, a pattern occurs more frequently in the background corpus can probe more facts from the PLM. Similarly, if a pattern composed of highly associated sub-patterns appears frequently, it should be considered as a good one as the PLM would be familiar with the contextual relations among the sub-patterns. (2) Informativeness: A pattern with low informativeness (e.g., p
′ 1 in Figure 3) has the weak ability of
PLM knowledge probing, as the relation between the subject or object entities cannot be well interpreted by it. (3) Completeness: The completeness
of a pattern affects a lot to the PLM knowledge probing especially when any of the placeholders is missing (e.g., p
′ m−2 in Figure 3) so that PLM can-
not even give an answer. (4) Coverage: A quality pattern should be able to probe accurate facts from PLM as many as possible. Therefore, patterns like p ′ 4 which only suit a few or only one case should have a low quality score. We then apply TruePIE on the prompts (patterns) selected by MetaPAD. TruePIE filters the prompts that have low cosine similarity with the positive samples (e.g., p
′ 3 and
p ′ m−1 are filtered), which matters to the creation of prompt ensemble since we want the prompts in the ensemble to be semantically close to each other so that some poor-quality prompts would not significantly impact the prediction result by PLM. As a result, we create a more reliable prompt ensemble Pri = {pi,1, pi,2, ..., pi,n} based on the averaged probabilities given by the prompts:
P (y|x, ri) = 1
n
n∑
j=1
PLM (y|x, pi,j), (1)
where ri is the i-th relation and pi,j is the j-th prompt in Pri . Beyond prompt selection, a prompt optimization process is also employed. Pointed out by Jiang et al. 2020, some prompts in the ensemble are more reliable and ought to be weighted more. Thus, we change Equation 1 to:
P (y|x, ri) = n∑
j=1
wi,jPLM (y|x, pi,j), (2)
where wi,j is the weight of j-th prompt for i-th relation. In our setting, all weights {w1,1, .., wk,n}
are learned through PLM to optimize P (y|x, ri) for ri ∈ R ahead of the training process.

heading: 3.3 Support Information Retrieval
text: In addition to the prompt mining, we also attach some query-wise and triple-wise support text information to the prompt to help the PLMs understand the knowledge we want to probe as well as to aid in training triple classification ability. As seen in Figure 4, for the i-th query qri in relation r, we use BM25 (Robertson et al., 1995) to retrieve highly ranked support texts with score greater than δ and length shorter than ϕ from the reliable corpus and randomly select one of them as the support information. To compose the input cloze q̂ri to the PLM, we concatenate the support text to each prompt in the optimized ensemble we obtained through previous steps, with the subject filled and the object masked. [CLS] and [SEP] are the tokens for sequence classification and support information-prompt separation accordingly.
In the training stage, we search texts using triples rather than queries, and the [MASK] would be filled by the object entities. It is worth noting that support text is optional in TAGREAL, and we leave it blank if no matching data is found.

heading: 3.4 Training
text: To train our model, we create negative triples in addition to the given positive triples following the idea introduced by PKGC (Lv et al., 2022), to handle the triple classification task. We create negative triples by replacing the head and tail in each positive triple with the "incorrect" entity that achieves high probability by the KGE model. We also create random negative samples by randomly replacing the heads and tails
to enlarge the set of negative training/validation triples. The labeled training triples are assembled as T = T + ∪ (T −KGE ∪ T −RAND) where T + is the positive set, T −KGE and T −RAND are two negative sets we created by embedding model-based and random approaches respectively. Then, we transform all training triples of each relation r into sentences with the prompt ensemble Pr and the triple-wise support information retrieved by BM25 (if there is any). At the training stage, the [MASK] is replaced by the object entity in each positive/negative triple. The query instances q̂ri are then used to fine-tune the PLM by updating its parameters. Cross-entropy loss (Lv et al., 2022) is applied for optimization:
L = − ∑
τ∈T (yτ log(c
1 τ ) + (1− yτ )
log(c0τ )
M ), (3)
where c0τ , c 1 τ ∈ [0, 1] are the softmax classification scores of the token [CLS] for the triple τ , yτ is the ground truth label (1/0) of the triple, and M = (|T +|/|T −|) is the ratio between the number of positive and negative triples. After the PLM is fine-tuned with positive/negative triples in training set, it should have a better performance on classifying the triples in the dataset compared to a raw PLM. This capability would enable it to perform KG completion as well.

heading: 3.5 Inference
text: Given a query (h, r, ?), we apply the query-wise support information that is relevant to the head entity h and relation r, as we presume that we are unaware of the tail entity (our prediction goal). Then, we make the corresponding query instances containing [MASK], with both support information and prompt ensemble, as shown in Figure 4. To leverage the triple classification capability of the PLM on link prediction, we replace [MASK] in a query instance with each entity in the known entity set and rank their classification scores in descending order to create a 1-d vector as the prediction result for each query. This indicates that the lower-indexed entities in the vector are more likely to compose a positive triple with the input query. For prompt ensemble, we sum up the scores by entity index before ranking them. The detailed illustration is placed in Appendix E.

heading: 4 Experiment

heading: 4.1 Datasets and Compared Methods
text: Datasets. We use the datasets FB60K-NYT10 and UMLS-PubMed provided by Fu et al., where FB60K and UMLS are knowledge graphs and NYT10 and PubMed are corpora. FB60K-NYT10 contains more general relations (e.g., “nationality of perso”) whereas UMLS-PubMed focuses on biomedical domain-specific relations (e.g., “gene mapped to diseas”). We apply the pre-processed dataset 3 (with training/validation/testing data size 8:1:1) to align the evaluation of our method with the baselines. Due to the imbalanced distribution and noise present in FB60K-NYT10 and UMLSPubMed, 16 and 8 relations are selected for the performance evaluation, respectively. We place more details of the datasets in Appendix A.
Compared Methods. We compare our model TAGREAL with four categories of methods. For (1) traditional KG embedding-based methods, we evaluate TransE (Bordes et al., 2013), DisMult (Yang et al., 2014), ComplEx (Trouillon et al., 2016a), ConvE (Dettmers et al., 2018), TuckER (Balažević et al., 2019) and RotatE (Sun et al., 2019) where TuckER is a newly added model. For (2) joint text and graph embedding methods, we evaluate RCNet (Xu et al., 2014), TransE+LINE (Fu et al., 2019) and JointNRE (Han et al., 2018). For (3) reinforcement learning (RL) based path-finding methods, we evaluate MINERVA (Das et al., 2017) and CPL (Fu et al., 2019). For (4) PLM-based methods, we evaluate PKGC (Lv et al., 2022) and our method TAGREAL. We keep the reported data of (2) and (3) by Fu et al.2019 while re-evaluating all
3https://github.com/INK-USC/CPL# datasets
models in (1) in different settings for more rigorous comparison (see Appendix I for details). PKGC in our setting can be viewed as TAGREAL with manual prompts and without support information.

heading: 4.2 Experimental Setup
text: For FB60K-NYT10, we use LUKE (Yamada et al., 2020), a PLM pre-trained on more Wikipedia data with RoBERTa (Liu et al., 2019b). For UMLSPubMed, we use SapBert (Liu et al., 2021) that pretrained on both UMLS and PubMed with BERT (Devlin et al., 2019). For sub-corpora mining, we use Wikipedia with 6,458,670 document examples as the general corpus and NYT10/PubMed as the reliable sources, and we mine 500 sentences at maximum (θ = 500) for each tuple. For the prompt selection, we apply MetaPAD with its default setting, and apply TruePIE with the infrequent pattern penalty, and thresholds for positive patterns and negative patterns reset to {0.5, 0.7, 0.3} respectively. For support information retrieval, we use BM25 to search relevant texts with δ = 0.9 and ϕ = 100 in the corpora NYT10/PubMed. We follow the same fine-tuning process as PKGC. We use TuckER as the KGE model to create negative triples, and we set M = 30 as the ratio of positive/negative triples. To compare with baselines, we test our model on training sets in the ratios of [20%, 50%, 100%] for FB60K-NYT10 and [20%, 40%, 70%, 100%] for UMLS-PubMed. The evaluation metrics are described in Appendix F.

heading: 5 Results

heading: 5.1 Performance Comparison
text: We show the performance comparison with the state-of-the-art methods in Tables 1 and 2. As one can observe, TAGREAL outperforms the existing
works in most cases. Given dense training data, KGE-based methods (e.g., RotatE) and RL-based methods (e.g., CPL) can still achieve relatively high performance. However, when the training data is limited, these approaches suffer, whereas PLM-based methods (PKGC and TAGREAL) are not greatly impacted. Our approach performs noticeably better in such cases than the current nonPLM-based ones. This is because the KGE models cannot be trained effectively with inadequate data, and the RL-based path-finding models cannot recognize the underlying patterns given insufficient evidential and general paths in KG. On the other hand, PLMs already possess implicit information that can be used directly, and the negative effects of insufficient data in fine-tuning would be less harsh than in training from scratch. TAGREAL outperforms PKGC due to its ability to automatically mine quality prompts and retrieve support information in contrast to manual annotations which are often limited. Next, we analyze the impacts of support information and prompt generation on the performance of TAGREAL.

heading: 5.2 Model Analysis
text: We conduct an ablation study to verify the effectiveness of both automatically generated prompts and retrieved support information. The results are
presented in Table 3, Figure 5 and 6.
Support Information. As shown in Table 3, for FB60K-NYT10, support information helps improve Hits@5 and Hits@10 in ranges of [5.2%,
Manual Prompt
[Y] is located in [X].
Query: (?, /location/location/contains, alba)
“in alba , italy 's truffle capital , in the northwestern province of piedmont , demand for the fungi has spawned a cottage industry of package tours , food festivals and a strip mall of truffle-themed shops . ”
[Y], [X] . 0.10490836 home in [Y], [X] . 0.23949857 [Y] is in [X] . 0.24573646 school in [Y], [X] . 0.32810964 people from [Y], [X] . 0.34946583
Optimized Prompt Ensemble weights
Support Information (retrieved by BM25)
Predictions (Top10 in descending order of classification scores) Man :
Optim : Man + Supp :
Optim + Supp :
united_states_of_america, pennsylvania, france, lombardy, abruzzo, jamaica, piedmont, ivrea, massachusetts, iraq
sicily, italy, massachusetts, lazio, piedmont, united_states_of_america, abruzzo, tuscany, iraq, milan
cuneo, piedmont, italy, sicily, lazio, texas, campania, northern_italy, scotland, calabria
piedmont, cuneo, italy, northern_italy, canale, tuscany, campania, sicily, lazio, calabria
Figure 7: Example of the link prediction with TAGREAL on FB60K-NYT10. Man denotes manual prompt.
Optim denotes optimized prompt ensemble. Supp denotes support information. The ground truth tail entity , helpful information and optimized prompts (darker for higher weights) are highlighted.
7.5%] and [3.8%, 5.3%], respectively. For UMLSPubMed, it helps improve Hits@5 and Hits10 in ranges of [1.9%, 4.94%] and [0.9%, 3.6%], respectively. Although the overlap between UMLS and PubMed is higher than that between FB60K and NYT10 (Fu et al., 2019), the textual information in PubMed could not help as much as NYT10 since that: (1) SapBert already possesses adequate implicit knowledge on both UMLS and PubMed so that a large portion of additional support texts might be useless. The lines "u2", "u3", "u4" and "u5" in Figure 5 show that support information helps more when using LUKE as the PLM as it contains less domain-specific knowledge. It also infers that the support information could be generalized to any application, especially when fine-tuning a PLM is difficult in low-resource scenarios (Arase and Tsujii, 2019; mahabadi et al., 2021). (2) UMLS contains more queries with multiple correct answers than FB60K (see Appendix A), which means some queries are likely "misled" to another answer and thus not counted into the Hits@N metric.
Prompt Generation. Almost all of the relations, as shown in Figure 6, could be converted into better prompts by our prompt mining and optimization, albeit some of them might be marginally worse than manually created prompts due to the following fact. A few of the mined prompts, which are of lower quality than the manually created prompts, may significantly negatively affect the prediction score for the ensemble with equal weighting. Weighting based on PLM reduces such negative effects of the poor prompts for the optimized ensembles and enables them to outperform most handcrafted
prompts. In addition, Table 3 shows the overall improvement for these three types of prompts, demonstrating that for both datasets, optimized ensembles outperform equally weighted ensembles, which in turn outperform manually created prompts. Moreover, by comparing line "f1" with line "f2", or line "u1" with line "u3" in Figure 5, we find a performance gap between PLM with manual prompts and with the optimized ensemble for triple classification, highlighting the effectiveness of our method.

heading: 5.3 Case Study
text: Figure 7 shows an example of using TAGREAL for link prediction with a query (?, /location/location/ contains, alba) where “piedmont” is the ground truth. By comparing the prediction results in different pairs, we find that both prompt generation and support information could enhance the KG completion performance. With the handcrafted prompt, the PLM simply lists out the terms that have some connections to the subject entity “alba” without being aware that we are trying to find the place it is located in. Differently, with the optimized prompt ensemble, the PLM lists entities that are highly relevant to our target, where “cuneo”, “italy”, “northern_italy” are correct real-world answers, indicating that our intention is well conveyed to the PLM. With the support information, the PLM increases the score of entities that are related to the keywords (“italy”, “piedmont”) in the text. Moreover, the optimized ensemble removes “texas” and “scotland” from the list and leaves only Italy-related locations. More examples are placed in Appendix H.

heading: 6 Conclusion and Future Works
text: In this study, we proposed a novel framework to exploit the implicit knowledge in PLM for open KG completion. Experimental results show that our method outperforms existing methods especially when the training data is limited. We showed that the optimized prompts with our approach outperform the handcrafted ones in PLM knowledge probing. The effectiveness of the support information retrieval to aid the prompting is also demonstrated. In the future, we may leverage QA model’s power to retrieve more reliable support information. Another potential extension is to make our model more explainable by exploring path-finding tasks.

heading: 7 Limitations
text: Due to the nature of deep learning, our method is less explainable than path-finding-based KG completion methods (e.g., CPL), which provide a concrete reasoning path to the target entity. Composing the path with multiple queries might be an applicable strategy that is worthwhile to investigate in order to extend our work on the KG reasoning task.
For the link prediction task, we adapt the “recall and re-ranking” strategy from PKGC (Lv et al., 2022), which brings a trade-off between prediction efficiency and accuracy. We alleviate the issue by applying different hyper-parameters given different sizes of training data, which is discussed in detail in Appendix C.
As a common issue of existing KG completion models, the performance of our model also degrades when the input KG contains noisy data. The advantage of our approach in addressing this issue is that it can use both corpus-based textual information and implicit PLM knowledge to reduce noise.

heading: 8 Ethical Statements
text: In this study, we use two datasets FB60K-NYT10 and UMLS-PubMed, which include the knowledge graphs FB60K and UMLS as well as the text corpora NYT10 and PubMed. The data is all publicly available. Our task is knowledge graph completion, which is performed by finding missing facts given existing knowledge. This work is only relevant to NLP research and will not be put to improper use by ordinary people.

heading: 9 Acknowledgements
text: Research was supported in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and INCAS Program No. HR001121C0165, National Science Foundation IIS-19-56151, IIS-17-41317, and IIS 17-04532, and the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 2118329, and NSF Award SCH-2205289, SCH-2014438, IIS-2034479.

heading: A Dataset Overview
text: We use the datasets FB60K-NYT10 and UMLSPubMed provided by (Fu et al., 2019). 4 They take the following steps to split the data: (1) split the data of each KG (FB60K or UMLS) in the ratio of 8:1:1 for training/validation/testing data. (2) For training data, they keep all triples in any relations. (3) For validation/testing data, they only keep the triples in 16/8 relations they concern (see relations in Table 5). The processed data has {train: 268280, valid: 8765, test: 8918} for FB60K and {train: 2030841, valid: 8756, test: 8689} for UMLS. As for the corpora, there are 742536 and 5645558 documents in NYT10 and PubMed respectively.
Sub-training-set splitting. To split the training data in the ratio of 20%/50% for FB60K-NYT10 or 20%/40%/70% for UMLS-PubMed, we use the same random seeds (55, 83, 5583) as Fu et al. used, and report the results in average.
Query-triple ratio. Within the relations that we focus on, we calculate the ratio of the triples by the queries (including both (h, r, ?) and (?, r, t)) to indicate the number of correct answers a query may have in average. The result is given in Table 4. For UMLS-PubMed, as the relations are symmetric in pairs, the number of queries for head and tail predictions are the same. Table 5 presents the counting in a more detailed setting. Both tables show that there are more multi-answer queries in UMLS-PubMed than in FB60K-NYT10, which explains why the support information may not be as helpful in the former as it is in the latter, as revealed by Table 3 and discussed in Section 5.2.

heading: B Textual Pattern Mining
text: The purpose of pattern mining is to find rules that describe particular patterns in the data. Information extraction is a common goal for pattern mining and prompt mining, where the former focuses on extracting facts from massive text corpora and the
4https://github.com/INK-USC/CPL# datasets
latter on extracting facts from PLMs. In this section, we use another example (Figure 8) to explain in detail how the textual pattern mining approaches like MetaPAD (Jiang et al., 2017) and TruePIE (Li et al., 2018) are implemented to mine quality prompts. In the example, given the relation location/neighborhood/neighborhood_of as the input, we first extract tuples (e.g., <east new york, brooklyn>) in the relation from the KG (i.e., FB60K). Then, we construct a sub-corpus by searching the sentences in a large corpus (e.g., Wikipedia) and the KG-related corpus (i.e. NYT10 for FB60). After the creation of subcorpus, we apply phrase segmentation and frequent pattern mining to mine raw prompt candidates. Since the candidate set is noisy as some prompts with low completeness (e.g., in lower [Y]), low informativeness (e.g., the [Y], [X]) and low coverage (e.g., [X], manhattan, [Y]) are present, we use MetaPAD to handle the prompt filtering with its quality function introducing those contextual features. After the prompts have been processed by MetaPAD, we choose one of them to serve as a seed prompt (for example, [X] neighborhood of [Y]) so that other prompts can be compared to it by computing their cosine similarity. As the positive seed prompt is selected manually, we can tell that there is still room for future improvement.

heading: C Re-ranking Recalls from KGE Model
text: Re-ranking framework. According to the inference process we present in Figure 9, we fill the placeholder ([MASK]) with each entity (e1, e2, ..., en) in the entity set E. However, as mentioned by Lv et al.2022, the inference speed of PLM-based models is much slower than that of KGE models, which is a disadvantage of using PLM for KG completion. To address this issue, they use the recalls from KGE models, that is, using KGE models to run KG completion and select X top-ranked entities for each query as the entity set E. Then, they shuffle the set and re-rank those entities using the PLM-based model. In our work, we adapt this re-ranking framework to accelerate the inference and evaluation as our time complexity is Z times as large as PKGC (Lv et al., 2018) for each case where Z is the size of prompt ensemble. We use the recalls from TuckER (Balažević et al., 2019) for both datasets.
Limitations. Nonetheless, implementing the re-ranking framework has a trade-off between efficiency and Hits@N performance. When the training data is large (e.g., 100%), the KGE model could be well trained so that the ground truth entity egt is more likely to be contained in the top X ranked ones. However, when the training data is limited (e.g., 20%), the trained KGE model could not perform well on link prediction, as shown in Table 1 and 2. In such a case, there is a probability that egt is not among the top X entities if we keep using the same X regardless of the size of the training data. To alleviate this side effect, we test and select different values of the hyper-parameter X for different sizes of training data, as presented in Table 6.
To check how much space there is for improvement, we manually add the ground truth entity into the recalls (we should not do this for the evaluation of TAGREAL as we suppose the object entity is unknown) and test the performance of TAGREAL
on UMLS-PubMed. The result is shown in Table 7. By comparing this data with Table 3 for UMLSPubMed, we find that changing the values of X could not perfectly address the issue. We leave the improvement as one of our major future works.

heading: D Computing Infrastructure & Budget
text: We trained and evaluated TAGREAL on 7 NVIDIA RTX A6000 running in parallel as we support multiGPU computing. Training TAGREAL to a good performance took about 22 and 14 hours on the entire FB60K-NYT10 dataset (with LUKE (Yamada et al., 2020)) and the entire UMLS-PubMed dataset (with SapBert (Liu et al., 2021)) respectively. The training time is proportional to the size (ratio) of the training data. The evaluation took about 12 minutes for FB60K-NYT10 with LUKE when hyper-parameter X = 20, and 16 minutes for UMLS-PubMed with SapBert when X = 30. The evaluation time is proportional to X , which
explains why we applied the re-ranking framework (Appendix C) to improve the prediction efficiency.

heading: E Link Prediction with Ensemble
text: For the link prediction with equally-weighted or optimized ensembles, we apply the method shown in Figure 9. Specifically, for each sentence with [MASK] filled with an entity ei, we calculate its classification score with the fine-tuned PLM. For each query, we get an m × n matrix where m is the number of prompts in the ensemble, n is the number of entities in the entity set (which is X if the re-ranking framework is applied). For an ensemble that is equally weighted, we simply sum the scores of each entity obtained from the different prompts, whereas for an optimized ensemble, we multiply the weighting of the prompts by the scores before the addition. After sorting the vector in size of 1×n in descending order, we can get the ranking of entities as the result of the link prediction.

heading: F Evaluation Metrics
text: Following previous KG completion works (Fu et al., 2019; Lv et al., 2022), we use Hits@N and Mean Reciprocal Rank (MRR) as our evaluation metrics. As mentioned in Section 3.5, the prediction of each query (h, r, ?) is a 1-d vector of indices of entities in descending order regarding their scores. Specifically, for a query qi, We record the rank of the object entity t as Ri, then we have:
Hits@N =
Q∑
i=1
Ri,in
Q and Ri,in =
{ 0, Ri > N
1, Ri ≤ N, (4)
MRR =
Q∑
i=1
1
QRi , (5)
where Q is the number of queries in evaluation.

heading: G Code Interpretation
text: To exploit the power of PLM, we need to map the code (entity_id) in KG/corpus into the words (Figure 10 shows the performance difference of PLM between using word and using code). For FB60K-NYT10, we use the mapping provided by JointNRE (Han et al., 2018) 5, which covers the translation for all entities. For UMLS-PubMed, we jointly use three mappings 6,7,8 which cover 97.22% of all entities.

heading: H Case Study
text: In addition to Figure 7, we show more examples applying TAGREAL on link prediction in Figure 11. We can see that the predictions with optimized prompt ensemble outperform those with manual prompts in all the cases, and even outperforms predictions with manual prompts and support information in some cases. In all these examples, the support information aids the PLM knowledge probing in different ways. For the first example, we believe that the PLM captures the words “brother james_murray” and “his wife jenny”, and realize that we are talking about the Scottish lexicographer “james_murray” but not the American comedian with the same name, based on our survey. For the second example, the PLM probably captures “glycemic control” which is highly relevant to the disease “hyperglycemia”. For the third example, the term “antiemetic” (the drug against vomiting) is likely captured so that the answer “vomiting” could be correctly predicted. Hence, it is not necessary for the support information to include the object
5https://github.com/thunlp/JointNRE 6https://evs.nci.nih.gov/ftp1/NCI_
Thesaurus/ 7https://www.ncbi.nlm.nih.gov/books/ NBK9685/ 8https://bioportal.bioontology.org/ ontologies/VANDF
mized prompt ensemble. Supp denotes support information. The ground truth tail entity , helpful information and optimized prompts (darker for higher weights) are highlighted.
entity itself, and including only some text relevant to it could also be helpful.

heading: I Re-evaluation of Knowledge Graph Embedding Models
text: We find that the performance of some KGE models was underestimated by Fu et al.2019 due to the low embedding dimension set for entity and relation. According to our re-evaluation (Table 8), many of these models could perform much better with higher dimension, and we report their best performance in Table 1 and 2 based on our experiments. For the previously evaluated models, we use the same code 9,10,11 as Fu et al. used to ensure the fairness of the comparison. For TuckER (Balažević et al., 2019), we use the code provided by the author. 12 Same as Fu et al., to make the comparison more rigorous, we do not apply the filtered setting (Bordes et al., 2013; Sun et al., 2019) of the Hits@N evaluation to all the models including TAGREAL.
9https://github.com/thunlp/OpenKE 10https://github.com/DeepGraphLearning/
KnowledgeGraphEmbedding 11https://github.com/TimDettmers/ConvE 12https://github.com/ibalazevic/TuckER
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7.
3 A2. Did you discuss any potential risks of your work? Section 7 and Section 8.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.
3 B1. Did you cite the creators of artifacts you used? Section 3, Section 4, Appendix A, Appendix G, Appendix I.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 3, Section 4, Appendix A, Appendix G, Appendix I.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 8.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. The used datasets FB60K, UMLS, NYT10 and PubMed are all publicly accessible.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4, Appendix A.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Reported in Appendix A.
C 3 Did you run computational experiments? Section 4, Section 5.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix D.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.2 and Appendix C.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, Appendix I.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave the improvement as one of our major future works