Learning from Both Structural and Textual Knowledge for Inductive Knowledge Graph Completion
Authors: Kunxun Qi
Jianfeng Du
Hai Wan
Venue: NeurIPS 2023 poster (2023)
Keywords: Knowledge graph completion; Neural approximate rule learning; Neural rule-based system

Abstract: Learning rule-based systems plays a pivotal role in knowledge graph completion (KGC). Existing rule-based systems restrict the input of the system to structural knowledge only, which may omit some useful knowledge for reasoning, e.g., textual knowledge. In this paper, we propose a two-stage framework that imposes both structural and textual knowledge to learn rule-based systems. In the first stage, we compute a set of triples with confidence scores (called \emph{soft triples}) from a text corpus by distant supervision, where a textual entailment model with multi-instance learning is exploited to estimate whether a given triple is entailed by a set of sentences. In the second stage, these soft triples are used to learn a rule-based model for KGC. To mitigate the negative impact of noise from soft triples, we propose a new formalism for rules to be learnt, named \emph{text enhanced rules} or \emph{TE-rules} for short. To effectively learn TE-rules, we propose a neural model that simulates the inference of TE-rules. We theoretically show that any set of TE-rules can always be interpreted by a certain parameter assignment of the neural model. We introduce three new datasets to evaluate the effectiveness of our method. Experimental results demonstrate that the introduction of soft triples and TE-rules results in significant performance improvements in inductive link prediction.

Learning from Both Structural and Textual Knowledge for Inductive Knowledge Graph Completion
Kunxun Qi'
Jianfeng 2 Hai Wan
1 School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
2 Guangzhou Key Laboratory of Multilingual Intelligent Processing. Guangdong University of Foreign Studies, Guangzhou, China
Bigmath Technology. Shenzhen, China qikx@mail2 sysu. edu. cn, jdduddusss. edu. enu wanhai momail.syn ssyu edu. cn
Abstract
Learning mule-based systems plays pivotal role in knowledge graph completion (KGC). Existing rule-based systems restrict the input of the system structural knowledge only, which may omit some useful knowledge for reasoning, e.g., textual knowledge. In this paper, propose two-stage framework that imposes both structural and textual knowledge to learn rule-based systems. In the first stage, compute set of triples with confidence scores (called soft triples) from text corpus by distant supervision, where a textual entailment model with multi- instance learning exploited to estimate whether a given triple is entailed by a set of sentences. In the second stage, these soft triples are used to learn rule- based model for KGC, To mitigate the negative impact of noise from soft triples, we propose a new formalism for rules to be learnt, named text enhanced rules TE-rules for short. To effectively learn TE-rules, we propose a neural model that simulates the inference of TE-rules. We theoretically show that any set of TE-rules can always be interpreted by certain parameter assignment of the neural model. We introduce three new datasets to evaluate the effectiveness of our method. Experimental results demonstrate that the introduction of soft triples and TE-rules results in significant performance improvements in inductive link prediction.
Introduction
Knowledge graph (KG) consists of real-world facts and has been widely used in many applications, including question answering [27), recommendation [22] and information retrieval [42). A fact in KGs is usually represented by triple of the form (head, relation, tail), where head and tail are entities. general, KGs are highly incomplete since the complete set of facts is hard to collect. Therefore, knowledge graph completion (KGC), which aims to infer missing facts from observed ones, has become vital ingredient for practically completing KGs.
In recent years, there are mainly two categories for prevalent approaches to KGC. One is the embedding-baseed category 45 3ki
Methods in this category usually learn knowledge graph embeddings (KGE) by encoding entities and relations as low-dimensional real-value vectors. They have been shown be effective for large-scale KGC, but are hard to generalize to the inductive scenario where missing facts involve previously unseen entities [20]. Besides, they can hardly be interpreted by human due their black-box nature. The other category tackles KGC by learning mase- systems, based on search algorithms with pruning heuristics [3
24 or neural models for
"Corresponding authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

HasWife
Haswiife
HasDaughter
HasMother
HasDhertter
HasMother
Barack Obama
Malia Obama
Michelle Obama
Barack Obama
Malia Obama
Michelle Obama
Supporting text:
Logical Rule:
Malia Obama and her mother Michelle Obama went to Liberia to...
HasWife(x, y) HasDaughtor(x.2) HasMother(x.y)
Figure I: Reasoning from both structural and textual knowledge.
They excel in explaining why missing fact is inferred and are able to handle the inductive setting.
approximate rule learning 46
Previous work focuses merely learning nule-based systems from structural knowledge. However, the high incompleteness of structural knowledge imposes a heavy burden for building high-quality rule-based systems. Text corpora can naturally be considered to enrich structural knowledge they contain much additional knowledge for reasoning. Figure showcasses an example about how textual knowledge help to KGC. The sub-figure on the left side shows a logical rule to infer whether has wife and we cannot infer that "Barack Obama" has wife "Michelle Obam" since the key fact ("Malia Obama", HasMother, "Michelle Obam") ") is missing in the background KG. The sub-figure on the right side shows that text corpora can provide supporting texts to this key fact, thereby the target relation HasWife can be correctly inferred.
Motivated by this kind of examples, we aim to learn rule-based systems from both structural and textual knowledge. To this end, we propose two-stage framework named Learning from Structural and Textual Knowledge or LSTK for short, as illustrated in Figure In the first stage, a distantly- supervised method using textual entailment model is proposed to extract textual knowledge from texts. Distant supervision (26) posits that if two entities involve a relation, then any sentences that mention these two entities might express that relation. Based on this assumption, we generate set of triples with their mentioned texts and employ a textual entailment model equipped with multi-instance learning mechanism estimate whether triple is entailed by a of sentences. Each generated triple has an estimated confidence score denoting the supporting degree of the triple by texts, and thus it is called a soft triple. In the second stage, both the generated soft triples and hard triples (i.e. existing triples) are used to learn a rule-based system for KGC.
In practice, it is challenging learn rules from soft triples, as the soft triples generated by distant supervision may contain noise. To mitigate the negative impact of noisy data, we propose new formalism for rules be learnt, named text enhanced rules or TE-rules for short. In this formalism, textual relations are introduced to provide flexible mechanism to discard noisy soft triples. In other word, a TE-rule is extended from chain-like rule by adding atoms or disjunction of atoms possibly with textual relations to the rule body. To effectively learn TE-rules, we propose neural model named TE-rule Learning Model or TELM for short. We show that an arbitrary set of TE-rules can be encoded by certain parameter assignment of TELM (see Theorem 1). This theoretical result guarantees certain degree of faithfulness between TELM and TE-rules, enabling us to extract explainable TE-rules from the parameter assignment of the learnt TELM. Based on the extracted TE-rules, LSTK is able provide explanations for inferred missing facts by backward reasoning.
We enhance three datasets from the field of relation extraction for empirical evaluation. Experimental results demonstrate significant gains achieve by the proposed method LSTK-TELM in inductive link prediction. Our ablation study and case study further clarify why the introduction of soft triples and TE-rules help to improve the performance.
Preliminaries
Knowledge Graph. Given set of entities & and a set of relations R, knowledge graph G is subset of x R x &. Specifically, hy the head entity for the ,th triple, r; E R the relation for the 7th triple and I E E the tail entity for the 7th triple. By r- we denote the inverse relation of r E R. The set of inverse relations for
{(h,, Ti, 4sisss where denotes the number of triples,

---

R, namely frr r E RJ. is denoted by R Accordingly, the equivalent knowledge graph for g composed by inverse relations, namely {tt,r_ .h) (h.r,t) g). denoted by 9
Logical Rule. Most existing work focuses on learning chain-like rules (CRs). CR is essentially plain datalog rule [1 where all atoms are binary and every body atom shares variables with the previous atom and the next atom. Formally, a CR R with L body atoms, simply called an L-CR, is of the form:
(zz,)) + B1 21) A B2(21, 22) A A BL(EL-1.3)
where x is the head entity, y the tail entity, and - variables. The part at the left (resp. right) side of is called the head (resp. body) of R. The rule I is called r-specific if H r. By R and BR we denote the atom in the head of R and the set of atoms in the body of R. respectively. An atom or a rule is ground if it does not contain any variable. rule R is fact if BR is empty and HR is ground. In this paper, a fact or ground atom r(a, b) and a triple (a, r, b) are used interchangeably. To uniformly represent rules using fixed-length bodies, we introduce the identity relation (denoted by T) to rule bodies. For example, rrr u) t- s(x,y) v) can be converted into rule with two body atoms, namely r(x,y) < s(x, v) A T(y. y).
We say g F HR(a, b) if there exists a ground instance Rg of R such that Hæ(a,b) = HR, and BR, S LGUG-U([(e, | lee Let Σ be a set of r-specific CRs and (a, r, b) triple. We say 9 F (a. r, b) if there exits a logical rule R E Σ such that 9 - (R b). We say a triple (a, r, b) is plausible in G if there exists a set of possibly correct r-specific CRs E such that G F (a, b).
Link Prediction. Link prediction is the main task of knowledge graph completion that we focus on in this work. Given knowledge graph G, a head query (?, r. t) or a tail query (h, ?), the task of link prediction aims find all entities E & such that (c, t) for (?, r, or (h, r, e) for (h, r, ?) is plausible in G. The inductive learning setting requires that at least one entity in the test set should not appear in the training set, while relations in the test set also appear in the training set.
3 Methodology
In this section, we describe the proposed LSTK framework for inductive KGC with text corpus. We first formalize our problem setting as follows.
Problem Setting. Given set of triples and corpus for training Ukrain = (guaain Tuain), ). a set of triples and a corpus for test Uheat = (Grest T, our inductive KGC setting aims to learn a KGC system based on Ugaain and then evaluate the learnt system on Utest- During evaluation, given a head query (?, r.) or a tail query (h, r, ?), the learnt KGC system finds an answer with the highest estimated truth degree to answer this query, based on the background knowledge from (Grrain U Grest \ {(h, r, t)); Terain U Tres)).
This setting is motivated by real-world application scenarios where we need to fetch texts from search engines to find evidences to verify a new fact. To address this problem setting, we propose Learning from Structural and Textual Knowledge or LSTK for short, as illustrated in Figure 2 Specifically, the proposed LSTK framework has two stages, where the first stage aims to generate soft triples from a text corpus and the second one aims at training rule-based system for KGC. Furthermore, LSTK is able to provide an explanation to show why an inferred missing fact is plausible in the background knowledge, by tracking back the reasoning paths from both hard triples and soft triples, where soft triples can further be explained by tracking back the supporting texts.
3.1 Formalization LSTK
In the first stage, our goal is to find all triples that are possibly entailed by the given text corpus. We achieve this goal by adopting distant supervision. As far as we know, distant supervision is mostly applied in relation extraction, where the relation between any two entities is restricted in close set. However, we naturally hope to discover more open relations that can be entailed by the text corpus. Considering that textual entailment [31) can deal with open relations and is better exploiting the semantic information of relational contexts, we impose distant supervision to a textual entailment model to generate soft triples.
Distantly supervised data generation. Given a knowledge graph G and a set of sentences (i.e. text corpus) T (ss) construct training set Darain {(T, St, (.) Wisisin. by
3

---

Structural inowledge (Hard triples)
Textual knowleddee triples)
Distantly supervised data generation
Learning TE-rules by TELM
Textual entailment model with multi-
0.S
Knowledge graph completion and rule extraction
1.4
1.0
0.
B.6
Text
Textual knowleddge (Soft triples)
Mixed knowledge (Hard triplss criples
Missing facts
Logical rules
The stage
The LSTK LSTK fromework Backward reasoning
The second stage
Supporting triples
Missing
0.95
Logical rules
0.5
Tracking back
Reasoning paths (Explanations)
0.6
Mixnd knowledge [Hard trighes Soft triplss)
Supporting triples
Supporting texts
Figure 2: Overview of the proposed LSTK framework.
finding all sentences in T that mention the entity pair (ha, of Ti, where Nee denotes the number of sentences, Nura the number of training instances, = (h,,ra, (3) triple, S; the set of sentences in T mentioning the entity pair (hs,4)) and yi the label. We call a triple rh,r, E 9 positive
triple, whereas triple (h, r. 1) & G negative triple. For each positive triple (ha, Fi, 42). we append
to the training set the set of negative triples {(hg, ry. 14) 1 E RR # rs, (h2 r,.)) E 9} obtained by corrupting T;- Accordingly y, is set to 1 if T2 E G or O otherwise. Considering that some entity pairs may appear in a large number of sentences, to confine the number of considering sentences, exploit the well-known graph partitioning algorithm METIS [17] to evenly partition the original S; to make every part no more than 20 sentences, and then treat the densest part defined by Jaccard similarity between bags of words the ultimate S.. Finally, we likewise construct the test set Diest = (Tr, S.) Ti E E X R &, S, + 0} for generating soft triples where labels are not given.
Multi-instance learning for RTE. Recognising textual entailment (RTE) 9 4 aims at determining whether sentence (called hypothesis) can be inferred by the other sentence (called premise). We employ a textual entailment model based on the pre-trained language model BERT 10] to esti- mate the confidence score for every triple. Specifically, given Dursin = {(Ti, St J:)] we create a set of sequences X; {zi,} ) IsjjNs, for each instance in Dirain by filling the template where [CLS] and [SEP] are special tokens in BERT, <triple> denotes the slot be filled by the context of and <sentence> the slot be filled by the context of sy E S,. All the sequences in X; are fed to BERT to calculate their contextual representations and the contextual embedding of the [CLS] token used calculate the entailment probability by full-connected layer activated by the sigmoid function. By Gi we denote the entailment probability for æi.j, then the entire entailment probability de for X; is calculated by di maxi Sss 6ig- The entire model trained by minimizing the cross-entropy loss.
Soft triple generation. The trained model is then applied to Drest for generating soft triples. A soft triple is defined as a quadruple (h, r, f, d), where (h,r, 1) denotes a triple and & its corresponding predicted entailment probability. Given hyper-parameter ó to keep only highly confident soft triples, the set of soft triples that will be used learning chain-like rules defined as Gooft = {(h,, Ti,t, dx) (h,, Pr, t+)= Drest, 6; 8}.
3.2 Formalization of Text Enhanced Rules
In the second stage, we aim at learning logical rules from both hard (i.e. existing) and soft triples. However, it is challenging to learn rules from the background KG with soft triples, as the generated

---

soft triples by distant supervision may contain noise, which impairs the quality of learnt rules. The following example illustrates a case where noisy data impact the reasoning ability of chain-likk rule.
Example 1. Considering the chain-like rule R in Figure D HasWife(s; y) HasDauugher((;; 2) Has MasMooher(zz.)) y). Suppose we have two soft triples "(A, HasDaughter, B, 0.55y" and "(B, HasMother, C, 0.90)" computed by the textual entailment model, where the former one is in- correct in the real world. From this rule R the negative triple "(A, HasWife, C)" will be incorrectly inferred to be positive.
To alleviate the negative impact of noise from soft triples, a simple way is to restrict that some atoms in the rule should appear in the structural knowledge as existing triples. To this end, we extend chain-like rules by adding atoms or disjunctions of atoms that may involve textual relations. We call such extended chain-like rules text enhanced rules and simply TE-rules. Let Ptext denote the textual relation corresponding to the relation r, then TE-rule can be defined below.
Definition 1. A p-specific L-TE-rule R, simply TE-rule if p and L are clear from the context, is of the form: p(x, y) + Ci(a; z1) C2(21, z2) A... A Ct, (Zt,1,1, y). where Ci can be an original relation, a textual relation, the identity relation, or a disjunction of an original relation and its corresponding textual relation, i.e. Ci(u, 8) is of the form r(u, v) V Ftext (u, v).
The following example explains why TE-rules can be used to resolve the error in Example 1.
Example 2. Considering Example I again. By allowing TE-rules to be learnt, a learnt rule R' can be: HasWife(ar, y) HasDaughter(z,, 2) (HasMother(z,; y) V HasMother (z, y)). From I'' the negative triple "(A, HasWile, C)" will not be inferred to be positive unless there exists a positive hard triple (A, HasDaughte, B) for some B.
3.3 End-to-end Learning of Text Enhanced Rules
To effectively learn TE-rules, we propose an end-to-end neural model named TE-rule Learning Model (TELM for short). The intuition of TELM is to select appropriate parameters to simulate the inference of TE-rules by gradient descent. Formally, given a set of hard triples G. a set of soft triples Isoft the maximum number L of body atoms in every rule, the maximum number of rules N, we first extend G to 9 {h,, L. 1.0)[ r, 1) E G). Suppose R = (riji its corresponding set of inverse relations R- (r.) 11sssns its corresponding set of textual relations Reext [ri]2n+ 1 Sgg33n well I T4n+1- Let
its corresponding set of inverse textual relations Rixt (ra)an+
Kmix Goxt U Gext Goon U U {(e, e, 1.0) e E 8). For < k S N. < < L, TELM
estimates a truth degree (k:1) for each triple (z, r. E G. defined by:
An+1
Σ
(.,r,,,,0) の) E Kmnx))
1 1=1
(k.))
(I)
An+1
Σ
(rtkL-1+1)
Σ
( Pr.
2515L
where (xx) = max(min(r,, 1).0) is an activation function that confines the given value to [0, 1]. I( C²) is function that returns & if C is true or 0 otherwise, and w(nkd)) [0. 1]4++1 denotes the trainable relational selection weights for the 1th body atom of the 4th rule for the head relation r. Intuitively, Equation (1) simulates the inference of TE-rules under the background KG with soft triples. The selection weights of all predicates for body atoms are formally defined as:
w (r [a (r,,.) w. orig
(r.d.1); ;.t.))))rr...) (rr..k.DDwnrrrrr (r.A.))) (r.R.1) wij+11
(r.k.1) where [:] denotes the concatenation operator, and w
Woig E [0, 1]2n (resp. w w E [0, 1]20 or (6.k0) E [0,1]) 1)) denotes the original relation selection weights (resp. textual relation selection w Win+1l
weights or the identity relation selection weight) for the 1th atom of the tt rule for the head relation r; further, (r.k.1) e [0, 1] (resp. B(r.k.l) E [0. 1)) denotes the trainable selection weight for determining whether the original (resp. textual) relation is involved in the 7th atom. Intuitively, (r.k.) (rrr)))))) ] 1 (resp. (r.k)))),, 1 or w wan+1
1) denotes that the 7th original relation r; (resp. the
5

---

tth textual relation r2n-i or the identity relation rin+1) is selected as the predicate of the 1th atom. Note that a (r.k.)) = 1A g(rsk.)) = indicates that the disjunction of an atom with an original relation and an atom with a textual relation involved in the 7 atom. We confine (r.h.)) and g(nk.)) to (r.k.) [0, by sigmoid layers, normalize both w (r.k..) and text by softmax layers, and restrict (rkk.) #4n++
Worig
to be max(0, 1 (rrA.))) g(rkk)) so to enforce (rr,k.)) B(rkk.))) 0 when w W4x+1
(r.k.) 1.
The final predicted truth degree is calculated by weighted sum of predicted degrees for N rules:
N
(N.L) PAE
(k) (K.L)
Pra
R=1
where u. [-1, is trainable weight for the 1thh rule for the head relation which is confined to [-1,1] 1] by a tanh layer. Then the model is trained by minimizing the objective function
exp( V
(N.L) P.E.N
L=
Σ (z,r,))g
log
(4)
EeES exp 1 (N.L))
We then show the faithfulness between the formalization of TELM and TE-rules in Theorem which relies on the notion of induced parameters defined below.
{ReJISESN for Ry of the form r(z,y) + 21) A A...ACe CC..((2L-11) with Cr,(u,v) E fruilu, u), rk, v), (,, V rt ri,((,,), v), where Th, E RUR U {13. The E Riext U Riexv rhe E R RUR, and
Definition 2. Given a set of r-specifiic L-TE-rules
rt is the corresponding textual relation of The we call a parameter assignment of TELM
LL = ((w(tig (
hENENNISIE U IihhskNNNsssAAS U (nlkk. (r...)) (
(ur JISKEN E-induced if it satisfies the following conditions for all 1 k S N,1SISL
(1) V1 sis 2n
[w orig
1 if appears in Ck,,, otherwise ww
[wori (ooorik.11 0.
(2) V1 sis S 2n [weext (rak.))
]; ]1= if r2444 appears in Ca. otherwise [wroxt (nrk.)))) = O.
(3) Wan+1
(r.k.))
if ran+1 appears in Ck,1, otherwise WAn++
(r.k.))
0.
(4) (rrkk)))
if an original relation appears in Cu,,, otherwise (r.)k()) = 0.
(5) ((rkl)) if a textual relation appears Cr. otherwise 3(r.k.)) O.
(6) (B)) 1
Theorem 1. Let G be set of hard triples, Guott a set of soft triples, (a, an arbitrary triple,
= (BEJiSANN a set of r-seciificc L-TE-rules and (.N. the E-induced parameter assignment of TELM, then V.N.L.. > 0 if and only if J Goott Fx r(a,b).
Explaining by Backward Reasoning. Thanks to the faithfulness between TELM and TE-rules, can extract TE-rules by interpreting the parameters of a learnt TELM model using beam search algorithm (see Algorithm 1 in the appendix). For each inferred missing fact, LSTK yields at least one reasoning path explanation by backward reasoning. Given triple (h, r, t) that is estimated be positive, a set of extracted TE-rules, the background KG Xmix and the corpus T, we can find a path between h and Whenever the path involves a soft triple, we treat the sentence in T that has the maximum confidence score as the supporting text for this soft triple.
Experimental Evaluation
4.1 Data Construction and Evaluation Metrics
We collected three benchmark KGs with corresponding text corpora from the field of relation extraction for empirical evaluation, which are HacREI 7. DocREL 48 and BioRe 41
htttae////gichbb.oooo///aaa//a/p httpae//guthab..cmm///mm///////p httpe://bit.llybborrldddde

---

Dataset
#En.
#Rel.
#TTrai
#Valid
#Test
#Texts
#Sonl.
PUE (cxist)
PUE (soll)
PUE (all)
HacRED
20,800
12
20,637
2,499
2,551
4.578
38.8M
98.7%
31.6%
30.0%
DocRED
17,997
28
24,384
2,819
3,844
19,517
4.7M
83.7%
84.9%
67.8%
BioRel
15,566
25,854
2,406
3,032
150,687
123.4M
79.4%
62.1%
46.8%
Table I: Statistical information for each dataset, where #nnt. (resp. #Rel.) denotes the number of en- tities (resp. relations), # #Train/Valid/Tess respectively the number of triples for training/validatinnnn/ #Texts the number of texts and #Soft. the number of extracted soft triples. PUE (short for Proportion of Unseen Entities) denotes the proportion of triples in the test set involving unseen entities in the training triples, where exist/softall respectively denote that the training triples are treated as existing triples in the original training set, as extracted soft triples, or as both existing triples and soft triples.
Considering that these datasets originally have a number of relations that can hardly appear in heads of potential logical rules, we filtered these relations by applying AMIE+ [13] to mine logical rules that have up to three body atoms from the complete set of existing triples for each dataset and omitting the relations that have not been involved in the body atoms of any mined rules. Under the inductive setting, there should be entities in the test set that do not appear in the training set. Therefore, we applied the partitioning algorithm METIS [17] the entity graph where an edge indicates that the two end-points have at least one filtered relation, so as to divide the set of entities into 10 subsets.
The train/valid/test split of each dataset was then extracted from 8/1/1 of these subsets by omitting triples across different splits. Statistics on all above datasets are reported in Table
Following [28. we reported the Mean Reciprocal Rank (MRR) and Hit Hit@k metrics under the filtered setting 3 for empirical evaluation.
4.2 Implementation Details
We implemented the textual entailment model by Pytorch 1.10.0. The model was initialized by the pretrained language model BERT with 12 transformer layers, which outputs -68-dimensional (i.c. d = 768) token embeddings. Then it trained by Adam (18) with warm up [00, where the initial learning rate was set to 5e-5, the mini-batch size to and the maximum number of training epochs to
3. We applied dropout (32 to each layer by setting the dropout rate to 0.1.
We implemented TELM by Pytorch 1.10.0. It was trained on an A100 GPU with 40GB memory by Adam [18] with 50 training epochs for HacRED and DocRED, and 20 for BioRel. The hyper- parameters were set to maximize MRR on the validation set. The initial learning rate was set to le-1 and the mini-batch size to 32 for all datasets. We also applied dropout 332 to the output states by setting the dropout rate to 0.3. The maximum length of each rule L is set to 3 for HacRED and DocRED, and 2 for BioRel. The maximum size of learning rules N is set to 20 for all datasets.
4.3 Main Results
We conducted experiments on the three enhanced datasets for link prediction under the inductive learning setting, where existing triples in the training set are used as background KG for training, existing triples in training and validation sets are used as background KG for validation, and existing triples in training, validation and test sets used for test. The hyper-parameter 5 to filter soft triples with high confidence scores was set to 0.5 in all experiments.
Table 2 reports the comparison results three datasets. Textual Copy Rule (simply TCR) is baseline that uses only textual copy rules of the form "r(æ, y) + Ttext (z, y)" in reasoning instead of using any other rules or neural models. TCR has not any extra logical reasoning ability beyond the textual entailment model. By LSTK-Xi we denote enhanced models that use soft triples in the background KG, where X denotes the original model. By LSTK-X we denote enhanced models that further extends LSTK-Xf by allowing textual relations to appear in learnt rules. Results show that the proposed method (i.e. LSTK- -TLM) significantly outperforms all the baseline methods with p-value < 0.05 by two-tailed t-tests. Specifically, LSTK-TEL TEEM outperforms LSTK-DRUM by absolute gains of 24.8% 22.444%%22.44.. 4% in Hit@1 1 scores the BacREEDDDoRRED/Bi. respectively. Further, we can see from the comparison results between LSTK-Xt and LSTK-X
datasets,
*Code and data about our implementations available https::/ggitee httpe///gittuu..oom/giiiiieee comm///uuuuunnISTT.
7

---

HacRED
DicRED
BiuRel
Method
p-value
H@3
MRR
H
H@ 10
MRR
H@1
H
Textual Copy Rule (TCR)
0.382
31.5
42.2
51.5
0.030
1.8
3.6
5,4
0.074
4.5
8.2
13.4
3 3.3c-9
AMIE+ 13
0.122
13.5
13.9
0.217
16.7
26.0
29.5
0.136
10.7
15.3
18.6
2.0c-7
LSTK-AMIT=
0.177
13.3
19.0
28.2
0.259
21.4
29.1
35.4
0.138
10.2
15.0
20.2
1.9c-7
LSTK-AMIE:
0.122
10.7
13.2
15.1
0.302
26.1
33.0
37.7
0.116
9.5
12.9
15.2
1.3c-3
RNNLogic 28
0.162
17.2
17.8
0.226
15.7
0.172
12.0
19.3
L9c-7
LSTT-RNNI Logic!
0.234
16.6
26.2
37.3
0.343
24.9
40.4
52.1
0.148
10.0
16.0
23.2
5.8c-6
LSTK-RNNLogle
0.394
32.8
42.3
52.1
0.396
31.6
45.1
53.9
0.284
20,3
33.8
43.5
4.8c-6
NeuralLP [6
0,395
42.6
48.9
0.310
24.0
0.317
23.2
26.8
1.3e-8
LSTK-NeuralL.
0.320
35,4
56.7
0.382
29.2
43.5
55.2
0.304
21.2
34.5
47.2
1.6e-5
LSTK-NeuralLP
0.484
38.1
53.4
67.5
0.488
40.9
54.2
62.2
0.384
27.6
44,9
58.1
4,5c-4
DRUM
41.4
48..5
0.352
28.1
0.314
23.1
36.2
46.4
3.0e:7
LSTK DRUM
0.347
23.8
39.6
56.0
0.392
29.1
46.3
56.5
0.292
20.4
32.9
46.0
1.5e:5
LSTK-DRUM
0,538
43.3
59.8
72.6
0.482
40.0
54.1
62.0
0.421
31.7
48.3
60.1
3.1e-4
LSTK-TELM (this work)
0.734
68.1
77.1
83.2
0,514
44.2
56.0
63.9
0.600
54.1
62.9
69.6
(1) w/o soft triples
0.416
37.7
43.6
48.1
0.380
30.8
44.0
49.4
0.339
25.9
38.3
48.1
1.1e-6
(2) wdo textual copy rules
0.710
65.0
74.9
82.1
0.502
43.2
54.6
62.7
0.511
44.6
54.7
62.8
2.1e:3
(3) w/o textual relations
0.579
48.2
64.3
76.4
0.438
35.5
49.1
58.5
0.356
25.4
40.3
54.8
5.1e-5
disjunctvve atoems
73.2
80.6
0.453
38.4
0.430
36.3
45.7
55.6
2.8e.4
(5) w/o confidence scores
0,730
67.3
76.9
83.0
0.508
43.4
55.7
63.4
0.574
50.8
61.3
69.5
8.4e-1
(6) Using relation extraction
0.723
66.1
76.7
83.3
0.499
43.0
54.5
61.8
0.572
50.7
60.9
68.8
1.9e44
Table 2: Comparison results on the HacRED, DocRED and BioRel datasets.
that, the introduction of textual relations bring significant performance improvements. For LSTK- NeuralLPi and LSTK-DRUM their performances are even worse than their original models that do not use any soft triples, indicating some negative impact of noise from soft triples.
We further conducted ablation studies on several variants of LSTK- TELM to verify the effectiveness of key components in both LSTK and TELM. In (1) we omitted all the soft triples in the background KG (i.e., only hard triples are used). Results show that the LSTK framework (using soft triples) pushes TELM by significant margin with p-value 1. It implies that the use of soft triples help to greatly improve the performance. In (2) we omitted the corresponding soft triple for each hard triple in the background KG. Results show that this variant model already significantly outperforms the variant model in (I), indicating that soft triples help to improve the reasoning ability without using textual copy rules. In (3) we omitted the textual relations in TE-rules. Results show that the use of textual relations pushes .SST-TTEMM by a significant margin with p-value 1.-,. It can be explained by the fact that the separation of original and textual relations helps to discard the noisy soft triples, resulting in higher reasoning performance. In (4) we omitted the learning of disjunctive atoms of the form r(u, v) V Frexx (u, v) in TELM. Results show that the introduction of disjunctive atoms bring significant performance gains with p-value 2.8c-4. This confirms that the introduction of disjunctive atoms in TE-rules provides more flexible mechanism control noise. In (5) fixed the confidence scores (i.e. entailment probabilities) of soft triples to 1.0. Results show that the performance significantly drops with p-value 8.4e-3. It implies that the computed entailment probabilities reflect the truth degrees of supporting facts and help to learn more informative rules. In (6) we trained a state-of-the-ar relation extraction model proposed in (2] based on BERT to replace the textual entailment model for generating soft triples. Results show that the use of relation extraction model leads to a significant performance degradation with p-value 1. 1.9e-4. This may be due to that textual entailment model can deal with open relations and exploit more text semantics on relations than relation extraction model does.
4.4 Case Study
To clarify why the learning of TE-rules help improve the performance, we conducted case study for applying learnt logical rules from LSTK- on the DocRED dataset, as shown in Table In the first case, the logical rule is extracted from LSTK-TEE TELM and can be applied to infer the positive triple "(Lois, Child, Superboy)". We can see from this case that the textual entailment model is able to infer from the relation mention "biological child" that, the entities "Lois" and "Clark" are spouse, thereby yielding a supporting soft triple "(Lois, Spouse, Clark, 0.76)" for reasoning. In the second case, can be seen that the same logical rule in the first case cannot infer that the triple "(John, Child, Joseph)" positive, since the key supporting fact "( "(Joseph, Father, Martyn)" is absent in the background KG. In the third case, the logical rule is "Child(z, y) + (Spouse(z, 20) V Spouse (z, zo)) A
8

---

Triple (fact)
Logical sule
Suppurting triplotext
Pred.
Label
Supporting hard triple: (Superbey, Father, Clark) Supporting soft triple: (Lois, Spouse, Clark, 0.76) Supporting text: Clark Lois' biological child DC Comics canon was born in Convergence:
(Lois, Child, Superboy)
Child(æ, f- (Spouse( vo
Trme
Trme
Spo 2o)) Father (So, y)
Superman 2 (July 2015), a son named Jonathan Samuel Kent, who eventually becomes Superboy.
Supporiing hard triple:
(John, Child, Joseph)
(a, 3) +- (Spouse(z, V
Supporting soft triple: (John, Spouse, Martyn, 0.933 Supporting text: The Road to Ruin is 1970 album re- leased busband and wife John and Beverley Martyn.
False
False
Spouse, oett (2t, 20)) Father (Zo, v)
Supporting triple: (Linda, Spouse, Paul, 0.88) Supporting text: In 1998, after Linda's death, Paul rearranged the song for string quartet to be played memorial concerts for his late wife.
(Linda, Child, Joseph)
Child(an,)) (Spouse(2; =o Spouse zol) Father (20,
Supporting soll triple: (Joseph, Father, Paul, 0.53) Supporting text: 1995, ceremonn Calambo, Pope John Paul beatified Father Joseph Vaz, an early missionary Apostle of Ceylon.
True
False
country. who known
Table 3: Examples for applying learnt TE-rules from LSTK-TELM on the DocRED dataset, where Pred. abbreviates prediction. The words marked as blue denote entities and the words marked as red relation mentions. r- denotes the inverse relation of r.
Fat y)" In this case, the textual entailment model mistakenly yields a supporting soft triple "(Joseph, Father, Paul, 0.53)" based on the words "beatified Father", thereby wrongly inferring the triple "(Linda, Child, Joseph)" to be positive. In general, these cases show that the use of soft triples is crucial in yielding a plausible explanation for test triple and that the separation of original and textual relations provides a flexible mechanism to discard noisy soft triples.
Related Work
5.1 Knowledge Graph Completion
There are two categories of prominent approaches to knowledge graph completion. They are embedding- and rule-based, respectively.
are a kind of well-known embedding- based methods. They aim represent entities and relations in KGs as low-dimensional real-valued vectors (37]. Although KGE methods have been shown to be effective in large-scale KGC, they can hardly generalize the inductive learning setting. Besides, KGE methods are hard to be interpreted by human due to their black-box nature. In recent years, graph neural networks (GNNs) [34] 23 50) have shown promissing results in addressing KGC. They can also be considered as embedding based, as they also Iearn latent representation for reasoming. While GNN-basee methods have been shown to excel in handling the inductive learning setting, we have not incorporated GNNs with LSTK. The reasons are three-fold. Firstly, GNNs are still black- box models with limited interpretability, whereas the proposed LSTK framework committed to explain why missing fact is inferred. Secondly, GNN-based methods necessitate sub-graph extraction to handle test triples involving unseen entities. The process of sub-graph extraction becomes highly time-consuming when dealing with large number of soft triplets. e.g., it takes about 10 hours for GralL [34] to process 1M triples for sub-graph extraction. Thirdly, the confidence scores from soft triples cannot be directly incorporated in GNN-based methods, unless the message passing mechanism in GNN-based methods is redesigned to handle the confidence scores on links.
Embedding- methods. Knowledge graph embeddings (KGE) 3 39 19
45
35
33
Rule-based methods. Different from embedding based methods, rule- based methods aim at building effective bule- systems for KGC. They can naturally reason under the inductive learning setting and provide logical rules as explanations. Learning rule-based systems has been widely studied in the field of Inductive Logic Programming (ILP). ILP methods such as AMIE+ 13 and AnyBURL [24) usually impose search algorithms with pruning heuristics to mine logical rules in a generate-and-tes, manner. Roiiogic (28) extends this manner to learn logical rules and their weights interactively based on neural models. In recent years, neural approximate methods are proposed to learn rule-based

---

systems from structural knowledge in KGs by learning continuous parameters using Tensorlog g operators. State-of-He the art neural approximate methods include NeuralLP [4), DRÜM [30,, NLIL [47] and Ruleformer [44] In addition to applying Tensorlog, other neural approximate methods such as NTP [29] and CTP 25 learn logical rules based on neural theorem provers. All these methods learn rule-based systems from structural knowledge alone and most of them focus merely on learning chain-like rules. In contrast, we study learning more expressive TE-rules from both structural and textual knowledge.
Other approaches using textual information. It worth noting that there are several approaches that focus on enhancing KGs with texts. For instance, recent advance of pre-trained language models (PLMs) for KGC such as KEPLER [38], PKGC [2T] and SimKGC [36] have considered textual information of triples in KGs. The differences between PLM- based methods and LSTK are two- fold. On one hand, they consider different types of textual knowledge. more detail, Plad-based methods consider text information such as contexts of triples, descriptions aligned to entities and pre-trained textual knowledge. In contrast, LSTK is proposed to address the KGC scenario where some knowledge (i.e., real-world facts) is provided in the structured KG and other knowledge is given by a text corpus. Thus, the textual knowledge considered in LSTK corresponds to a set of potential facts, namely the soft triples. On the other hand, PLM- --adedded methods cannot work in our scenario because they can only obtain textual knowledge by pre- training a model over a given text corpus, text corpora given in the test phase (see the problem setting in Section 3). The pre-training process is generally consuminng and requires massive computing resources, thus it is impractical to be applied in the test phase. Except for PLM-based methods, the work 12 proposes a knowledge verification system by combining logical and textual information to compute explanations for validating new triples. However, the system should work with predefined logical theory. In contrast, our approach does not require any predefined logical theory but learns one approximate logical theory on the fly. The recent work [15] improves the knowledge coverage of knowledge base (KB) by facts extracted from text corpora. They have not focused on reasoning over facts. In contrast, we focus on enhancing the reasoning ability of rule-based systems by soft triples extracted from text corpora.
whereas setting allows to process
5.2 Distant Supervision for Relation Extraction
Distant supervision (26] is a dominant paradigm handle the problem of lacking labeled data in relation extraction. It assumes that if two entities involve relation, then any sentence mentioning them together might express that relation. Based on this assumption, a large number of labeled data for relation extraction can be automatically generated by aligning triples with texts. However, this may introduce noise to the generated data. Therefore, most previous studies exploit -unti-instance learning [1I] by putting instances with the same entity pair into bags, to alleviate the impact of noisy instances. A number of neural models including CNN (49, Bi-LSTM (5] and BERT (4) have been incorporated with multi-instance learning and used as classification models for relation extraction. We also impose distant supervision to score soft triples from a text corpus, but we resort to textual entailment models to make the best of relation mentions, instead of using classification models where relations are treated as labels to be predicted.
Conclusion and Future Work
In this work we have proposed a two-stage framework, named LSTK, to learn rule-based systems from both structural and textual knowledge. It computes set of soft triples by distant supervision in the first stage and applics these soft triples to enhance the learning of neural approximate rulc-based systems in the second stage. To mitigate the negative impact of noise from soft triples, we have proposed a new formalism for logical rules named TE-rules and a neural model named TELM for learning TE-rules. We introduced three new datasets for empirical evaluation. Experimental results demonstrate significant improvements achieved by learning TE-rules from soft triples. Our case study further reveals how TE-rules help to control noise from soft triples.
From the third case in case study, we observe that some errors produced by the textual entailment model will propagate to the final prediction. This is known as the error propagation issue in pipeline approaches. To tackle this issue, our future work will focus on studying fully end-to-end frameworks to learn TE-rules from both structural and textual knowledge.
10

---

7 Limitations
LSTK merely works for knowledge graphs with aligned text corpora, but general knowledge graphs do not come with aligned texts. In order to adapt LSTK to more real-world applications, we generate soft triples automatically under the distant supervision assumption that any sentence mentioning two entities might involve relation between these two entities. This assumption enables us to fetch texts via information retrieval tools such as search engines or to generate supporting text via large language models (LLMs) without the need of manually constructing texts. Note that all the original datasets of HacRED 77. DocRED (48) and BioRel [41] are collected by distant supervision. Although the performance of LSTK depends on the quality of distant supervision, LSTK indeed works for any
knowledge graphs since aligned texts can easily be collected by distant supervision.
8 Acknowledgements
This paper was supported by National Natural Science Foundation of China (No. 62276284, 61976232, 61876204), National Key Research and Development Program of China
(No. 2021 I0A000050 Guangdong Basic and Applied Basic Research Foundation Guangzhou Science and
(No. 2023A151501144 1470. 2022A1515011355, 2020A1515010642. Technology Project (No. 202201011699), Shenzhen Science and Technology Program (KJZD202309231 1405902), Guizhou Provincial Science and Technology Projects (No. 2022-259), Humanities and Social Science Research Project of Ministry of Education (No. 18YJCZH006), well as the Fundamental Research Funds for the Central Universities, Sun Yat-sen University (No. 23ptpy31).
References
[1] Serge Abiteboul, Richard Hull, and Victor Vianu. Foundations of Datubases. Addison- Addison-Wesley, 1995.
[2] Christoph Alt, Mare Hübner, and Leonhard Hennig. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. ACL, pages
2019.
[3] Antoine Bordes, Nicolas Usunier, Alberto Carcía-Durán, Jason Weston, and Oksana Yakhnenko. Translat- ing embeddings for modeling multi-relational data. NIPS, pages 2787-2795, 2013.
[4] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In EMNLP, pages 632 642, 2015.
[5] Rui Cai, Xiaodong Zhang, and Houfeng Wang. Bidirectional recurrent convolutional neural network for relation classification. In ACL, 2016.
[6] Jiajun Chen, Huarui He, Feng Wu, and Jie Wang. Topology- aware correlations between relations for inductive link prediction in knowledge graphs. In AAAI, pages 6271 0278, 2021.
[7] Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang. Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. Hacred: large-scale relation extraction dataset toward hard cases practical applications. In Findings of ACL, pages 2819-2831, 2021.
[8] William W. Cohen, Fan Yang. and Kathryn Mazaitis. Tensorlog: A probabilistic datahase implemented using deep-learnng infrastructure. J. Artif. Intell. Res,, 67:285-325, 2020.
191 Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, volume 3944, pages 177-190, 2005.
[10] Jacob Devlin, Ming-We Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186 2019.
[11] Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano- Pérez. Solving the multiple instance problem with axis axis-parallel rectangles. Artif. Intell., 89(1-2):31-71 1997.
[12] Jianfeng Du, Jeff Z. Pan. Sylvia Wang, Kunxun Qi. Yuming Shen, and Yu Deng. Validation of growing knowledge graphs by abductive text evidences. In AAAI, pages 2784-2791, 2019.
11

---

[13] Luis Galárraga, Christina Teflioudi. Katja Hose, and Fabian M. Suchanek. Fast rule mining in ontological knowledge bases with AMIE+. VLDB J. 24(6):707-730. 2015.
[14] Luis Antonio Galáraga, Christina Teffioudi, Katja Hose, and Fabian M. Suchanek. AMIE: association rule mining under incomplete evidence in ontological knowledge bases. In WWW, pages 413 4222, 2013.
[15] Vinh Thinh Ho, Daria Stepanova, Dragan Milchevski, Jannik Strötgen, and Gerhard Weikum. Enhancing knowledge bases with quantity facts. In WWW. pages 893-901. ACM, 2022.
[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short term memory. Neural Computation, 9(8):1735 1780, 1997.
[17] George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput., (0())335-332, 1998.
[18] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In JCLR, 2015.
[19] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu. and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI, pages 2181-2187, 2015.
[20] Shuwen Liu, Bernardo Cuenca Grau, Ian Horrocks, and Egor V. Kostylev. INDIGO: gun- nun-based inductive knowledge graph completion using pair-wise wise encoding. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurIPS, pages 2034-20000 2045, 2021.
[21] Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. Do pre-trained models benefit knowledge graph completion? A reliable evaluation and a reasonable approach. In Findings of ACL, pages 3570-3581, 2022.
[22] Xinze Lyu, Guangyao Li, Jincheng Huang, and Wei Hu. Rule-guided graph neural networks for recom- mender systems. In ISWC, pages 384-401, 2020.
[23] Sijie Mai, Shuangjia Zheng, Yuedong Yang, and Haifeng Hu. Communicative message passing for inductive relation reasoning. In AAAI. pages 4294-4302, 2021.
[24] Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruftinelli, and Heiner Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. In IJCAI, pages 1137-3143, 2019.
[25] Pasquale Minervini, Scbastian Riedel, Pontus Stenetorp, Edward Grefenstette, and Tim Rocktäschel. Learning reasoning strategics in end-to-end differentiable proving. In ICML, volume 119, pages 6938- 6949, 2020.
[26] Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation extraction without laheled data. In ACT. pages 1003-1011, 2009
[27] Arindam Mitra and Chitta Baral. Addressing question answering challenge by combining statistical methods with inductive rule learning and reasoning. In AAAI, pages 2779-2785, 2016.
[28] Meng Qu. Junkun Chen, couis-Pascal A. C. Xhonneux, Yoshua Bengio, and Jian Tang. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. In JCLR, 2021.
[29] Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In NIPS, pages 3788-3800, 2017.
[30] Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang, DRUM: end-to-end differentiable rule mining on knowledge graphs. In NeurlPS, pages 15321-15331, 2019.
[31] Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Encko Agiree. Label verbalization and entailment for effective zero and few-shot relation extraction. In EMNLP, pages I 1199-1212, 2021.
[32] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 15(1):1999-195 2014.
[33] Zhiqing Sun, Zhi-Hong Deng. Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR, 2019.
[34] Komal K. Teru, Etienne G. Denis, and William L. Hamilton. Inductive relation prediction by subgraph reasoning. In ICML, volume 119, pages 9448 9457, 2020.
12

---

[35] Théo Trouillon. Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In ICML, volume 48, pages 2071-2088 2016.
[36] Liang Wang. Wei Zhao, Zhuoyu Wei, and Jingming Liu. Simkge: Simple contrastive knowledge graph completion with pre-trained language models. In ACL, pages 4281-42991 1-244, 2022.
[37] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Trans. Knowl. Data Eng. (TKDE), 29(12):772--7444 2017.
[38] Xiaozhi Wang, Tianyu Gao, Zhancheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representation. Trans. Assoc. Comput. Linguistics (TACL), 9:176-194 2021.
[39] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. AAAI, pages 1112 1119, 2014.
[40] Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. Denoising relation extraction from document- distant supervision. EMNLP, pages 3683-3688, 2020.
[41] Rui Xing, Jie Luo, and Tengwei Song. Biorel: towards large-scale biomedical relation extraction. BMC Bioinform., 21 2165516) 2020.
[42] Chenyan Xiong, Russell Power, and Jamie Callan. Explicit semantic ranking for academic search via knowledge graph embedding. In WWW, pages 1271-1279, 2017.
[43] Xiaohan Xu, Peng Zhang, Yongquan He. Chengpeng Chao, and Chaoyang Yan. Subgraph neighboring relations infomax for inductive link prediction on knowledge graphs. In IJCAI, pages 2341-2347, 2022.
[44] Zezhong Xu, Peng Ye, Hui Chen, Meng Zhao, Huajun Chen, and Wen Zhang. Ruleformer: Context-aware rule mining over knowledge graph. In COLJNG, pages 2551-2560, 2022.
[45] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities relations for learning and inference in knowledge bases. In /CLR, 2015.
[46] Fan Yang, Zhilin Yang, and William W. Cohen. Differentiable learning logical rules for knowledge base reasoning. In NIPS, pages 2319-2328, 2017.
[47] Yuan Yang and Song. Learn to explain efficiently neural logic inductive learning. ICLR, 2020.
[48] Yuan Yao, Deming Ye, Peng Li, Xu Ham, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. Docred: A large-scale document-level relation extraction dataset. ACL, pages 764 777, 2019.
[49] Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. Distant supervision for relation extraction via piecewise convolutional neural networks. In EMNLP, pages 1753-1762, 2015.
[50] Yufeng Zhang, Weiqing Wang, Hongzhi Yin, Pengpeng Zhao, Wei Chen, and Lei Zhao. Disconnected emerging knowledge graph oriented inductive link prediction. In ICDE, pages 381 181-393,,200 2023.
13

---

A Proofs
A.1 Proof of Lemma
To prove Theorem I we first introduce Lemmal !
Lemma 1. Let G be set of hard triples, Guoft set of soft triples, r,) an arbitrary triple, R an r-specific rule of the form r(x, y) +- C1(x, 21) A C2(21, 22) A... Cz(21.11.)) and of: the {R)-induces parameter assignment of TELM, then (1) & wa.b > 0 if U Gsoft += H na, b), and (2) Wra. 0 ifg U gsote bzz HH((a,
Proof. Let GUG U Ssaft U U (((,,,,)) E) where & the set of entities appearing in 9 Gsoft- Note that the confidence sore of each triple in IC is larger than O.
(I) Consider the case where g U 9 F Hn(a, b). There exists least one ground instance Rg of R such that HR(a,b) b) HR, and BR, S C.C. There will be a sequence of entities CJ .... Cy-1 and a sequence of relations Tl, .... TL such that (a, r1, c1), (21,72, T2, e2).... (CL-1, FL,
(22,6) b) E K. where ry is relation appearing in C, for all S S L. rf RUR- suppose rr. is the 4th relation in R U RT then by Condition and 4 in Definition 2, we have (woris | and a (r.1.)) = 1. If TE E Reext U Reext suppose rE is the hth relation in Reext U Rrext then by Condition and in Definition 2, we have [wtext (r.1.11.)) | = 1 and g6x,11) = 1. ri = I, by Condition3 in Definition we have Win+1 (r.1.L) 1. By Equation (2), we must have wik (r.111L) 1 for some k. By Equation 1.b 2 Φr > 0 for or the confidence score of (cI-1, b). Likewise, if R suppose r is the ttth relation in RUR then by Condition and 4 in Definition 2, we have (w00it-1-1)) ) and (r.1.4) 1. If rf E Reext U RText suppose FL-1 is the 8th
(I), we further have (.(1.)
Tz-1 E RUR
(r,1.1.11) relation in Reext UTextt then by Condition 2 and 5 in Definition 2, we have ( )/ 1 and
3(r.1,1-1) = If = I, by Condition3 Definition 2, we have Wan+1 (r.1.L-1) = 1. By Equation (2), we must have we (r11L-1)
for some k. By Equation (1), we further have ( Vree 2.5 0. In the Vr,ce-a.b > O, ((Lb-1)) and (LLL) > 0 in turn. Therefore, we have vi (I.L) 0 if Gooft } Hz((a,)) b).
same way, we can show that w (1.33
(II) Consider the case where G U Goote I HR(a, Suppose whal > 0, then by Equation (1), there must be some k E {l...., 4n + 1} such that of (r:1,1) >> 0. Ti appears in C₁ and there exists (a, rks (1) E IC fulfilling ((L.L-1) > O. Since (1,,L11) 0, by Equation (1), there must be also r,o1,b
some 1 E {1...., 4n + 1} such that w/ (r.1.2) > 0, ri appears in C2 and there exists (c1, 02) E KC fulfilling (11,1-2) > O. In the same way, we can show that there exists relation F' appearing in C.. and entity Cm such that (Cm-1. Tmm Cm) E K and w((lL--) > 0 for m =3, ..... L in turn, while there exists relation Tz appearing in Ct such that (cL-1,7Z, b) E IC. Hence there exists a sequence of entities C₁. ..... CL-1 and sequence of relations Tl...., TL such that (a, 71, ct), (C1,T2, (2,,)2, c2).... (CL-1, TL, 6) E IC, where r; is a relation appearing in C, for all 1 < i < L. These two sequences constitute ground instance Rg of R such that Hz((a, b) HR, and BR, C K,, contradicting g Goott f Hn(a,b) b). Thus w(I.L) < O. By Equation (1) we have (1.L) 2 0. Therefore, we have brab
0 if Gooft I HR(a, b).
A.2 Proof of Theorem
(k.1) Proof. Lemma implics that, for all R& E E, Frabb otherwise.
> 0 if Gsoft }= H b) and (R. a =
r.ahb (+) Suppose y (L.)) > O. Then by Equation (3) and Condition 6 in Definition 2, there exists at least
one r-specific L-TE-rule R& E E such that wo (k.L) > 0. By Lemma we have G GUGoo } H (a, b). Since G U Gooft += HR, (a, b) and R& E, we have G U Geoft F (a, r, b).
14

---

(+) Suppose G U Isoft Fy (a, r,)., Then we have G U Ssoft += RR for some R₂ E E. By Lemma we have w.(aal) > and for all k' + k. wik.h) 2 0. By Equation (3) and Condition in Definition 2, we have T (NN.))
r,a.b > 0.
B Formalization of Multi-instance Learning for RTE
In Section 4.1 we have described how to generate soft triples by applying RTE model based on BERT [TQ]. Due to the space limitation, we omitted some details. Thus, in this section we further elaborate on how to apply multi-instance learning to RTE based on BERT.
Given Derain = {(T,, St, 31) Jihsinge where T; (hs, t.) denotes triple, h4 = (wi ...., No. the head entity with Nr, tokens, T; (wf'.... ..., w wN the relation with N,. tokens, t; "i,,,...,,, whi ) the tail entity with Nr, tokens, S (sujhissss, (wfi,,. -..
of Ns, sentences men- sentence with No.,, tokens, create set of sequences X; {at, hisgNs, for each instance in Dirain. For 1 S S Ns,, the sequence Ti is created by first concatenating h4s t; and st, and then adding special token [CLS] in front of the first token w²", special token [SEP] between w "N, and w and special token [SEP] behind the last token $1.7
tioning (h;; t:), and $1,j (wi ,... w 8E.3 N...)
"N..."" All sequences are fed to BERT to calculate their contextual representations. By
Ci,y E RR xd we denote the output of BERT for Ti.j. where NE., Nv. + Nr, + +N+ + +N&;, +3 is the number of tokens in æt,j and d is the dimension of token embedding. We use the embedding of the [CLS] token, namely Ct.j.1. as the contextual embedding for textual entailment classification. The entailment probability for Fi is calculated by
fi aa b)
(5)
where W E 1 is trainable weight vector, R' trainable bias value and the sigmoid function. Then the entailment probability for X; is calculated by
Φ =
max <i,j
151.Ns,
The entire model is trained by minimizing the cross-entropy loss, formally defined as
Nera
1
C =
Y/ log Φ; + +(1 yi) log(( - -bb)
Nura
C Rule Extraction From TELM
Based on the formalization of TELM, we can interpret TE-rules from its parameters, The process of interpretation is shown in Algorithm T Intuitively, Algorithm [ interprets TE-rules from the parameter assignment of TELM using beam search, where denotes the threshold to determine whether the original relations and textual relations are used in atoms, m the beam size, fi the set of (R', s)-pairs for the 7th atom, R' the currently interpreted (partial) rule and s its estimated score. It should be noted that the process for interpreting r-specific TE-rules outputs up to interpreted rules (k) sharing the same confidence score uy for the th target rule.
D Experiments
D.1 Baselines
We applied LSTK to the following state-of-the-art rule learning methods for empirical comparisons.
AMIE+. AMIE+ 13 is upgraded from the well well-known rule learner AMIE 14] It imposes
several pruning heuristics to mine logical rules.
NeuralLP. NeuralLP 46) is the first neural approximate method that exploits Tensorlog
operators to learn chain-like rules. translates the structure learning of logical rules from discrete space to continuous space.
15

---

Algorithm 1: Interpreting r-specific L-TE-rules
Input: beam size m 2 1, selection threshold a E (0.0, 0.5). and parameter assignment of
TELM for the relation r, namely 0f 0.
J.hhsssaASSLLLsss... U (1 (r.k.)) hekENIsIISL
(r.k.)
2 Output: a set of up to m.N r-specific L-TF-rules
4 for 1 1issks N do
fo - ( 1)} where denotes a placeholder to be filled;
v1 31 L: fi+
1 1 do
for (R, s) E fi-1 do
if o 2 and g(r,k.)) < then
10
for 1 - < 2n do
R' + R with the 1th placeholder replaced with rz:
fi + fi U ((l', (wrrk)"))s))
13
else if a (r.k.1) < π and g(r,k.1) 2 π
then
for 1 < 2n do
R' +- R with the 1th placeholder replaced with ran+4
+ fi U {(R', [ubest (r.k.)) 1;s)))
[;s));
else if a (n.t.) > and g(n.k.)) 2
π then
18
for 1 < < 2n do
R' + R with the tth placeholder replaced with ry V 12246
19
JU ((I', max((w(ri (o((r).")))))))) }])s)];
21
else
R' + R with the hhh placeholder replaced with I;
22
f {(R', wan+1 s)};
(r,k.))
sort ft {RR, ,sم,s in the descending order of and preserve the top-m in ft;
(It rewritten from the form of TE-rule (R, s) fi}:
26
R -
I Q
27
return I;
extends NeuralLP learn rules with dynamic lengths by adding the identity relation. It exploits bidirectional LSTM network to learn logical rules together with their scores simultaneously.
DRUM. DRUM 30
RNNLogic. RNNLogic (28) leverages an LSTM [16 based rule generator and reasoning predictor learn logical rules together with their weights interactively.
constructed adjacency matrix for each relation from the mixed background Kmixx where the weights of edges are set by the confidence scores. It should be noted that AMIE+ can only learn rules from hard triples in the background KG, and thus the confidence scores of soft triples are not used in LSTK-AMIE+
For LS LSTK-DRUM and LSTK- RNNLooc,
D.2 Implementation of Baselines
We implemented LSTK-N KerK-NeurlLLFP and SSSK--DUM"TO Tensorflow 1.15.0 based on their published code. They were trained by Adam, where the initial learning set to 1e-3, the mini-batch size to 32, the maximum number of training epochs to 10, and the maximum length learnt rules to
16

---

Proportion rulls with tootuaa pre cltess
LSTICOIUM
0.52
LSTKANNAagEE ISTCTRLM
0.50
5L.6
1STKORUM
HacRED
DoCRED
BloRel
0.1
0.2
0.6
0.7
fitering
(a) Proportion of rules involving textual relations.
(b) Comparisons on ó for LSTK-TELM.
Figure 3: Analysis on learnt rules and the threshold ö.
ASTK-RNNLOO was implemented by Pytorch 1.10.0 based on its published code. Both the rule generator and reasoner in RNNLogic were trained by Adam, where the initial learning rate was set to 1e-3 (resp. le-1) for the generator (resp. the reasoner), the maximum number of training epochs to 5000 (resp. 5) for the generator (resp. the reasoner), the mini- batch size to 1024, and the maximum these baselines were conducted
length of learnt rules to 2. Experiments 40GB memory. We implemented LSTK-AMIE- based on its published code, and ran it on a Linux equipped with an Intel(R) Xeon(R) Gold 6248R processor with 3.0 GHz and 128 GB RAM.
A100 GPU with
D.3 Evaluation Metrics
For each test triple (h, r,t) ) evaluation, we built two queries (?, r, and (?,r7, for answering and 6, respectively. For each query, we computed the truth degrees for corrupted head triples and then computed the rank of the correct answer. The rank of the correct answer is defined by m + (n + 1)/2, where is the number of triples with higher truth degrees than the correct answer and the number of corrupted head triples with the same truth degree as the correct answer. The rank for each test triple calculated by the Rank (MRR for short) and Hit@k k (H (H@k for short) metrics under the filtered setting.
rank of queries. Based the rank, reported the Mean Reciprocal
D.4 Statistical Analysis on Learnt Rules
We conduced statistical analysis on learnt rules to show how much textual knowledge has been used different models, as shown sub-figure (a) in Figure We can see that about 30% logical rules extracted from either LSTK-Neurall. or LSTK-DRUM (resp. about 60% logical rules extracted from LSTK-TEEM) involve at least one textual relation. It implics that more soft triples are leveraged in the reasoning of --SS-TTELL. We have also found that about 90% logical rules extracted from ASTK-RNNLogg involve at least one textual relation, which is much higher than the rate other approaches. This may be due the large search space explored by RNNLogic in searching potential paths from the KG for rule generation, where the introduction of soft triples substantially scales up the search space, leading achieves lower performance than other .STK-enhanced models. These results imply that the overuse of textual relations may degrade the performance due noise in soft triples.
lot of logical rules that involve textual relations. Recall Table 2 that
D.5 Hyper-parameter Analysis
LSTK introduces a threshold to filter soft triples with high confidence scores. We conducted experiments to show the impact of different setting of 6, as shown in sub-figure (b) in Figure Each curve on the MRR scores has moderate jitter with increasing ó. This can be explained by that, although the use of soft triples tends to improve the performance of link prediction, the overuse of soft triples may introduce much noise, leading to a performance degradation. By observing the curves of https//lgihuu.com/DDeepittt htwp:pppll..aain/nNNNN...M http:://ww...iiiittn...////eeee
and LSTK-TELM, we recommend to set 8 0.5 default.
17


Reviews:
[{"review_id": "JKfwofoJWI", "reviewer": "Reviewer_DS3c", "summary": "This paper addresses the problem of knowledge graph completion by proposing a two-stage framework — learning from structural and textual knowledge (LSTK). This novel framework leverages both structural and textual knowledge to learn rule-based systems, which provides a unique approach in the realm of KGC research. The paper is premised on the idea that the typical reliance on structural knowledge alone in rule-based systems is a limiting factor. This leads to the proposition of a system that utilizes \"soft triples\", or triples with confidence scores derived from textual corpora via distant supervision and a textual entailment model with multi-instance learning. In the second stage, a rule-based system for KGC is learned using both these soft triples and the hard triples of existing KG triples. The paper also introduces a novel formalism for learning rules, referred to as text-enhanced rules or TE-rules, which can mitigate the negative impact of noisy data in soft triples.\n", "questions": "— Can you elaborate why only use a textual entailment model based on the pre-trained language model BERT to estimate the confidence scores of the soft triples and discuss its generalizability? And what exact BERT model variant do authors use?\n\n— How do the newly introduced datasets compare with established benchmarks? Moreover, how would the proposed LSTK framework perform on these existing benchmarks?\n\n", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "— The two-stage framework introduces an innovative way of leveraging both structural and textual knowledge for KGC, thereby potentially mitigating the limitations of existing methods.\n\n— The introduction of TE-rules provides a robust mechanism for learning rules, to mitigate the negative impact of noises from soft triples.\n\n— The paper introduces three new datasets for empirical evaluation, contributing further to the body of resources available in this field.\n", "weaknesses": "— The authors utilize a single method to estimate the confidence scores of the soft triples, resulting in a potential lack of generalizability.\n\n— While the introduction of three new datasets is commendable, the paper does not offer a clear comparison of these datasets with established benchmarks. \n\n— Furthermore, it remains uncertain how well the proposed method would perform on existing benchmarks.\n\n— The paper falls short in discussing the scalability of the proposed method. It is unclear how the proposed framework would handle large-scale knowledge graphs, particularly considering the potential computational costs of generating and handling soft triples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments. The following addresses concerns and questions.\n\n> [R1] The authors utilize a single method to estimate the confidence scores of the soft triples, resulting in a potential lack of generalizability.\n\n[A] We have conducted an ablation study on replacing the proposed method for extracting soft triples (please see variant model (6) in Table 2). This variant model employs a strong baseline [1] in the field of distantly supervised relation extraction. Results show that the proposed method outperforms the variant in three datasets. This demonstrates the generalizability of the textual entailment model.\n\n> [R2] While the introduction of three new datasets is commendable, the paper does not offer a clear comparison of these datasets with established benchmarks.\n\n[A] As far as we know, there are no established benchmarks with corresponding text corpora that meet our settings. In our settings, we assume that a part of the knowledge (i.e., real-world facts) exists in structured KG, while the other part (i.e., potential facts) exists in text corpus. Consequently, we introduce three new datasets to evaluate the effectiveness of LSTK and facilitate the research field of addressing KGC from both structural and textual knowledge perspectives. The most relevant dataset for our study is the Wikidata5M [2] dataset, which provides aligned text descriptions on entities. While this dataset can be utilized to assess approaches focused on enhancing entity embeddings for KGC, it is not suitable to be used in our scenario, as our scenario assumes that a set of potential facts is implied by the corpus, and we require to mine these potential facts from the given corpus. Furthermore, our scenario presents a greater challenge since the relationship between entities and the provided text corpus is unknown, whereas Wikidata5M provides aligned descriptions for each entities. We will include these discussions in the revised version.\n\n\n> [R3] Furthermore, it remains uncertain how well the proposed method would perform on existing benchmarks.\n\n[A] As mentioned in our discussion on limitations (Please see section E: Discussion on Limitations in the supplement file), our proposed framework, including the proposed TE-rules and TELM model, cannot effectively function for KGs without text corpora. For example, the introduced textual relations in TE-rules exist exclusively within KGs with soft triples. Therefore, we have not conducted an evaluation of LSTK-TELM on established benchmarks such as WN18RR, FB15k-237 and YAGO3-10.\n\n> [R4] The paper falls short in discussing the scalability of the proposed method. It is unclear how the proposed framework would handle large-scale knowledge graphs, particularly considering the potential computational costs of generating and handling soft triples.\n\n[A] As reported in Table 1, there are about 38.8/4.7/123.4 million soft triples has been extracted in the first stage for the HacRED/DocRED/BioRel datasets. The computational cost of generating soft triples is mainly on the textual entailment model based on BERT, where it takes about 6/0.2/18 hours to compute all soft triples for the HacRED/DocRED/BioRel datasets. Further, the proposed LSTK-TELM model is able to handle both the existing triples and soft triples in efficient time, as it stores all the triples by sparse adjacency matrix. In more details, each query in the HacRED/DocRED/BioRel datasets takes about 1.5/0.3/2.5 seconds for evaluation. We will include these discussions in the revised version.\n\n\n> [R5] Can you elaborate why only use a textual entailment model based on the pre-trained language model BERT to estimate the confidence scores of the soft triples and discuss its generalizability? And what exact BERT model variant do authors use?\n\n[A] We use the BERT-base model for textual entailment. Implementation details about the textual entailment have been reported in section 5.2. Actually, we have discussed the generalizability of the textual entailment model in Table 2. Specifically, we have compared with a strong baseline [1] in the field of distantly supervised relation extraction. (please see variant model (6) in Table 2). Results show that the proposed method outperforms this variant in three datasets. This demonstrates the generalizability of the textual entailment model. This may be due to that a textual entailment model can deal with open relations and exploit more text semantics (e.g., contexts of relations) on relations than a relation extraction model does (please see line 311-315 for more details). \n\n> [R6] How do the newly introduced datasets compare with established benchmarks? Moreover, how would the proposed LSTK framework perform on these existing benchmarks\n\n[A] As far as we know, there are no established benchmark KGs with corresponding text corpora that meet our problem settings, thus we introduce three new datasets for evaluation. The most relevant dataset for our study is the Wikidata5M [2] dataset, which offers aligned text descriptions on entities. Nevertheless, Wikidata5M is not suitable to be used in our scenario because its text corpus mainly encompasses descriptions on entities, whereas our scenario requires extracting a set of potential facts from the given corpus. Therefore, we have not evaluated LSTK on Wikidata5M.\n\nOn the other hand, the proposed TE-rules and TELM model are designed to handle soft triples with textual relations. Thus, we have not evaluated LSTK-TELM on existing benchmarks such as WN18RR, FB15k-237 and YAGO3-10. This has also been discussed in the section on limitations (please see section E: Discussion on Limitations in the supplement file).\n\n**Reference:**\n\n[1] Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In ACL, pages 1388–1398, 2019.\n\n[2] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. In TACL, 9, 176–194, 2021."}, {"review_id": "dR6DyzbFM7", "reviewer": "Reviewer_KZDY", "summary": "This paper proposes a rule-based inductive KGC method with two stages. In the first stage, it extracts soft triples from a text corpus using distant supervision. In the second stage, the obtained soft triples are mixed with the original hard triples and used to learn rule-based model for KGC.", "questions": "(1) This paper uses both structural and textual knowledge for inductive KGC, but there is no discussion about other related works which also use structural and textual knowledge, such as  KEPLER[1] and BERTRL[2].\n(2) Since the proposed method extract new triples from text corpus, if the unseen entities in test set may seen in the new soft triples. It is better to provide statistics about the  unseen entities in original triples, soft triples and both triples.\n(3) Some notations are confusing, for example the \"t\" is used for a tail entity in a triple (h,r,t), and also used in \"1≤t≤T\". \n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9, 176–194. \n[2] [2] Zha, H., Chen, Z., & Yan, X. (2022). Inductive Relation Prediction by BERT. Proceedings of the AAAI Conference on Artificial Intelligence, 36(5), 5923-5931.", "limitations": "(1) This paper uses both structural and textual knowledge for inductive KGC, but there is no discussion about other related works which also use structural and textual knowledge, such as  KEPLER[1] and BERTRL[2].\n(2) Since the proposed method extract new triples from text corpus, if the unseen entities in test set may seen in the new soft triples. It is better to provide statistics about the  unseen entities in original triples, soft triples and both triples.\n(3) Some notations are confusing, for example the \"t\" is used for a tail entity in a triple (h,r,t), and also used in \"1≤t≤T\". \n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9, 176–194. \n[2] [2] Zha, H., Chen, Z., & Yan, X. (2022). Inductive Relation Prediction by BERT. Proceedings of the AAAI Conference on Artificial Intelligence, 36(5), 5923-5931.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) It is interesting and intuitive to enrich the original triples in KG by extract new triples from text corpus.\n(2) The idea that uses extracted soft triples from text corpus to enhance logical rules is novel to some extent.", "weaknesses": "(1) This paper uses both structural and textual knowledge for inductive KGC, but there is no discussion about other related works which also use structural and textual knowledge, such as  KEPLER[1] and BERTRL[2].\n(2) Since the proposed method extract new triples from text corpus, if the unseen entities in test set may seen in the new soft triples. It is better to provide statistics about the  unseen entities in original triples, soft triples and both triples.\n(3) Some notations are confusing, for example the \"t\" is used for a tail entity in a triple (h,r,t), and also used in \"1≤t≤T\". \n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9, 176–194. \n[2] [2] Zha, H., Chen, Z., & Yan, X. (2022). Inductive Relation Prediction by BERT. Proceedings of the AAAI Conference on Artificial Intelligence, 36(5), 5923-5931.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments. The following addresses concerns and questions.\n\n> [R1] This paper uses both structural and textual knowledge for inductive KGC, but there is no discussion about other related works which also use structural and textual knowledge, such as KEPLER[1] and BERTRL[2].\n\nDue to space limitations, we have only discussed the related work about learning rule-based systems for inductive KGC. We have not discussed the methods like KEPLER[1] and BERTRL[2] because they follow a different research line than our method. On one hand, these methods consider different textual knowledge from that of LSTK, where the textual knowledge considered in [1-2] encompasses the contexts of triples, descriptions on entities and the pre-trained knowledge. The purpose of introducing texts for these methods is to enrich the semantic information of entities, thereby obtaining improved entity representations. In contrast, we study the KGC setting where a part of the knowledge exists in the KG, while the other part (i.e. potential facts) exists in text corpus. In this setting, the input consists solely of the provided KG and its corresponding text corpus, without any accompanying entity descriptions. The textual knowledge considered in our work refers to the potential facts entailed by the text corpus. This setting is motivated by the scenario that the existing facts in structural KG  are highly incomplete, and we can fetch text evidences from text corpus to support a new fact. Therefore, we mine a set soft triples from the text corpus to obtain textual knowledge in the first stage.\nOn the other hand, PLM-based methods such as KEPLER[1] and BERTRL[2] can be considered as embedding-based methods, which are black-box models with low interpretability, and cannot induce logical rules as explanations for predictions. In reality, we naturally expect that the proposed framework LSTK is highly interpretable to human. Thus, we focus on learning rule-based systems for inductive KGC because logical rules excel in explaining why a missing fact is inferred. We will include these discussions in the revised version.\n\n> [R2] Since the proposed method extract new triples from text corpus, if the unseen entities in test set may seen in the new soft triples. It is better to provide statistics about the unseen entities in original triples, soft triples and both triples.\n\n[A] Thanks for pointing out the statistic difference. We provide our statistical details as follow. The statistical results show that about 98.7%/83.7%/79.4% triples in the test set of HacRED/DocRED/BioRel involve unseen entities from the training triples, about 31.6%/84.9%/62.1% triples in the test set of HacRED/DocRED/BioRel involve unseen entities from the extracted soft triples, and about 30.0%/67.8%/46.8% triples in the test set of HacRED/DocRED/BioRel involve unseen entities from both the training triples and extracted soft triples. This shows that there are still a great number of triples in the test set involving unseen entities, even when the soft triples are used. Furthermore, we have done experiments to show whether the new extracted soft triples can cover the triples in test set (Please see the baseline Textual Copy Rule (TCR) and the variant model (2) in Table 2). We will include these statistics in the revised version.\n\n\n> [R3] Some notations are confusing, for example the \"t\" is used for a tail entity in a triple (h,r,t), and also used in \"1≤t≤T\".\n\n[A] Thank you for pointing out it. We will refine this notation in the revised version."}, {"review_id": "vSiZNu0peP", "reviewer": "Reviewer_Vj83", "summary": "In this paper, the authors try to solve Knowledge Graph Completion (KGC) by proposing a two-stage framework Learning from Structural and Textual Knowledge (LSTK), that imposes both structural and textual knowledge to learn rule-based systems. In the first stage, the authors compute a set of triples with confidence scores. To mitigate the influence of the noise in the set of triples estimated in the first stage, the authors also propose a threshold-based method text enhanced rules (TE-rules) to filter out low-confident rules. In the second stage, the authors use the filtered triples by TE-rules to train a neural model named TE-rule Learning Model (TELM) for KGC. Furthermore, the authors created new datasets based on the data, HacRED, DocRED, and BioRel. Experimental results show that the proposed method LSTK outperforms the baseline methods regarding MRR, Hits@k (k=1,3,10).", "questions": "As shown in the example of Figure 1, the proposed method LSTK can cover inferences of unseen entities and relations. However, as shown in the inductive setting of Wikidata5M [1], recent prompt-based KGC models can complete such triples based on text information like descriptions and pretrained knowledge similar to your work. Thus, to claim the advantage of LSTK, you need to compare it to the recent KGC models like KEPLER [1] and SimKGC [3]. Also, could you explain the difference between these models and LSTK?\n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9, 176-194. (https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00360/98089/KEPLER-A-Unified-Model-for-Knowledge-Embedding-and)\n\n[3] Wang, L., Zhao, W., Wei, Z., & Liu, J. (2022, May). SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4281-4294). (https://aclanthology.org/2022.acl-long.295/)", "limitations": "I think you need to add that this approach is difficult to work on knowledge graphs that do not have corresponding text data to Limitations in your paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Since the proposed method LSTK relies on rule-based inference, the estimation of this method is interpretable to humans.\n- The proposed approach can restrict the automatically extracted rules to high-confidence ones by TE-rules to deal with the noisy rules extracted by distant supervision.\n- The threshold of filtering soft triples does not require sensitive tuning and works with 0.5. Thus, LSTK is robust and easy to use.\n- This work provides new KG datasets with their corresponding texts.", "weaknesses": "- The paper does not refer to the recent KGC model that can conduct KGC with unseen entities [1].\n- Even though recent KGC models utilize pretrained language models considering textual information of triplets in KGs [2], there is no comparison against such models.\n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9, 176-194. (https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00360/98089/KEPLER-A-Unified-Model-for-Knowledge-Embedding-and)\n\n[2] Lv, X., Lin, Y., Cao, Y., Hou, L., Li, J., Liu, Z., ... & Zhou, J. (2022, May). Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach. In Findings of the Association for Computational Linguistics: ACL 2022 (pp. 3570-3581). (https://aclanthology.org/2022.findings-acl.282/)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments. The following addresses concerns and questions.\n\n> [R1] The paper does not refer to the recent KGC model that can conduct KGC with unseen entities [1].\n\n[A] Due to space limitations, we have only discussed the related work about learning rule-based systems for inductive KGC. Up to now, three main categories of methods have been proposed to address KGC, including GNN-based methods, PLM-based methods and rule-based methods. GNN-based methods such as R-GCN address inductive link prediction based on graph neural networks (GNNs). PLM-based methods such as [1-3] leverage pre-trained language models (PLMs) such as BERT to address the inductive KGC problem. However, both GNN-based and PLM-based methods can be considered as embedding-based methods due to their black-box nature. For example, the hidden representation learned by GNNs and PLMs can hardly be interpreted by human. Therefore, we focus on learning rule-based systems for inductive KGC with textual knowledge, where logical rules (e.g., TE-rules) can serve as explanations for predictions. We will include these discussions in the revised version.\n\n> [R2] Even though recent KGC models utilize pretrained language models considering textual information of triplets in KGs [2], there is no comparison against such models.\n\n[A] The approach like [2] follows a different research line than LSTK, considering distinct textual knowledge from that of LSTK. The textual knowledge considered in the work [2] encompasses the contexts of triples, descriptions on entities and the pre-trained knowledge. In contrast, we study the KGC setting where a part of the knowledge exists in the KG, while the other part (i.e. potential facts) exists in text corpus. In this setting, the input consists solely of the provided KG and its corresponding text corpus, without any accompanying entity descriptions. The textual knowledge considered in our work pertains to potential facts entailed by the text corpus. Consequently, we mine soft triples from the text corpus in the first stage. We focus on learning rule-based system because logical rules excel in explaining why a missing fact is inferred. While PLMs are capable of handling inductive KGC, they remain black-box models with limited interpretability, and cannot induce logical rules as explanations for predictions. \n\nIt is worth noting that a possible way to apply KEPLER [1], PKGC[2] and SimKGC [3] to our scenario is first pre-training the PLMs such as BERT on the given corpus, and then utilizing the pre-trained PLM to address inductive KGC. Although this way does not require to process soft triples, it still fails to generalize in the real-world application scenarios. To analyze the reasons, we first recall the formalization of our problem setting.\n\n$\\textbf{Problem statement}:$ Given a set of triples and a corpus for training $U\\_{train}=(\\mathcal{G}\\_{train}, \\mathcal{T}\\_{train})$, a set of triples and a corpus for test $U_{test}=(\\mathcal{G}\\_{test}, \\mathcal{T}\\_{test})$, our inductive KGC setting aims to learn a KGC system based on $U_{train}$, and then evaluate the learnt system on $U_{test}$. During evaluation, for each head query $(?,r,t)$ or tail query $(h,r,?)$, the learnt system finds an answer with the highest estimated truth degree for answering $(?,r,t)$ or $(h,r,?)$, based on the background knowledge from $U_{train} \\cup U_{test} \\setminus \\\\{(\\\\{(h,r,t)\\\\}, \\emptyset)\\\\}$.\n \nIt can be seen in our inductive setting that the KGC systems require to process unseen text corpus in the test set. It is worth noting that this setting is motivated by the real-world application scenarios that we might need to fetch texts from search engines to find evidences for a new fact. PLM-based methods cannot work in this scenario because they can only obtain textual knowledge by pre-training the model on the given corpus. This process is impractical in the inference phase because the process of pre-training is time-consuming and requires massive computing resources. In contrast, the proposed LSTK-TELM can easily address the above scenario.\n\nBesides, we have tried to apply methods like KEPLER [1] to the second stage of LSTK but found it to be time-consuming. This is due to the extraction of over 100 million soft triples in certain datasets (e.g. the BioRel dataset), which leads to time-consuming processes for PLM-based methods to handle such a large volume of triples due to their extensive parameter sizes.\n\nWe will include these discussions in the revised version.\n\n> [R3] Prompt-based KGC models can complete such triples based on text information similar to your work. Could you explain the difference between these models and LSTK?\n\n[A] The differences between the mentioned methods [1-3] and LSTK are two-fold. Firstly, the text knowledge considered in the work [1-3] is different from that in LSTK, where KEPLER [1], PKGC [2] and SimKGC [3] consider text information such as contexts of triples, descriptions aligned to entities (e.g., Wikidata5M) and pre-trained textual knowledge. In contrast, LSTK is proposed to address the KGC scenario where a part of the knowledge (i.e. real-world facts) exists in structured KG, while the other part exists in text corpus. The input of this setting is the given KG and its corresponding corpus. Thus no description is given to a entity or a fact. The textual knowledge considered in LSTK is a set of potential facts (i.e., the soft triples). Secondly, PLM-based methods such as [1-3] can also be seen as embedding-based methods, which are black-box models with low interpretability. In contrast, the proposed LSTK framework is able to induce logical rules (i.e. the TE-rules) as explanations for predictions.\n\n> [R4] I think you need to add that this approach is difficult to work on knowledge graphs that do not have corresponding text data to Limitations in your paper.\n\n[A] We have done this discussion in our supplement file (Please see section E: Discussion on Limitations)."}, {"review_id": "8F4G6RZemJ", "reviewer": "Reviewer_PLPA", "summary": "This paper proposes a two-stage framework that imposes both structural and textual knowledge to conduct knowledge graph completion. The first stage aims to extract some soft triples with confidence scores. And then the second stage designs some tailored rules for entity link prediction, including text-enhanced rules and TE-rules. Experiments on three benchmarks show the proposed method significantly outperforms previous work.", "questions": "The motivations of the designed two step is unclear.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of this paper is clear.\n2. Experimental results show significant improvements to baseline models.", "weaknesses": "1. The text-enhanced entity representation models have been widely studied, such as [1][2]. Related work should be discussed. \n2. The two-step pipeline will bring unnecessary cascaded errors. The most directed way is enhancing the entity representations using text representations to directly predict the entity links.\n3. The first step with confidence score seems to model uncertainty in linking entities, which may be a novelty. However, such a motivation is not clearly described.\n4. The relationship between the designed step should also be emphasized. Such as, the first step brings more entity semantics from text and the second step verifies the entity relations via text and structure aspects.\n5. LLMs have shown strong ability in building KGs and exacting entity relations. This paper does not conduct experiments on LLMs.\n\n[1] Representation Learning of Knowledge Graphs with Entity Descriptions. AAAI 2016.\n[2] Entity-duet neural ranking: Understanding the role of knowledge graph semantics in neural information retrieval. ACL 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments. The following addresses concerns and questions.\n\n> [R1] The text-enhanced entity representation models have been widely studied, such as [1][2]. Related work should be discussed\n\n[A] Due to space limitations, we have not discussed the text-enhanced entity representation models such as [1-2]. These methods follow a different research line compared to our method. The methods proposed in [1-2] focus on enhancing entity representation by text such as the descriptions on entities, relying on aligned text descriptions on entities as input. In contrast, we study the KGC setting where a part of the knowledge (i.e. real-world facts) exists in a structured KG, while the other part exists in text corpus. Thus no descriptions on entities are given in our setting. We require to first extract textual knowledge (i.e., a set of soft triples) from the given text corpus and then apply these soft triples to learn rule-based systems for inductive KGC. We will include these discussions in the revised version.\n\n> [R2] The two-step pipeline will bring unnecessary cascaded errors. The most directed way is enhancing the entity representations using text representations to directly predict the entity links.\n\n[A] The mentioned way is infeasible in the scenario considered in this paper. On one hand, we study learning rule-based systems from both structural and textual knowledge, where there is no entity representation in a rule-based system. We have not studied embedding-based methods in LSTK because they are black-box models with low interpretability, and can hardly generalize to the inductive setting. We naturally expect that the proposed framework LSTK is highly interpretable to human. On the other hand, the text-enhanced entity representation models assume that the manual annotated descriptions on entities are explicitly given, whereas the inputs of our setting are a KG and its corresponding text corpus. Thus there are no text descriptions on entities for enhancing entity representations. In our setting, we assume that a part of the knowledge (i.e. real-world facts) exists in structured KG, while the other part exists in text corpus. Therefore, a two-step pipeline is necessary in LSTK to first extract potential facts (i.e. the soft triples) from the given text corpus, and then apply these soft triples to learn rule-based systems.\n\n> [R3] The first step with confidence score seems to model uncertainty in linking entities, which may be a novelty. However, such a motivation is not clearly described.\n\n[A] The goal of the first stage in LSTK is not to link entities but to mine a set of potential facts (i.e. the soft triples) for learning rule-based systems. In this work, we assume that some knowledge exists in the given KG and some exists in the given text corpus. The motivation of the first stage in LSTK is to discover textual knowledge in the text corpus by mining soft triples. This process is not for linking entities, but for relation extraction from every entity-pair that is mentioned by texts.\n\n> [R4] The relationship between the designed step should also be emphasized. Such as, the first step brings more entity semantics from text and the second step verifies the entity relations via text and structure aspects.\n\n[A] The goal of the first step is not to brings more entity semantics from text but to mine potential knowledge (i.e. real-world facts) from the text corpus. The goal of the second step is to learn a rule-based system for addressing inductive KBC, using both existing hard triples and extracted soft triples. We have describe the relationship between these two step in introduction (line 44-52) and methodology (line 132-134).\n\n> [R5] LLMs have shown strong ability in building KGs and exacting entity relations. This paper does not conduct experiments on LLMs.\n\n[A] To the best of our knowledge, no published work has shown that LLMs such as ChatGPT and GPT-4 excel at addressing inductive KGC. At the present stage, LLMs are unsuitable for adoption in our scenario. The reasons are two-fold. First, applying LLMs in the first stage of LSTK is heavy in cost, as the first stage requires repeatedly applying the relationship extraction model. For instance, more than 100 million soft triples are extracted from the BioRel dataset (please see Table 1), which means that we require extensive invocation of LLMs. However, the number of requests for LLMs is still limited to this day. Second, it is infeasible to apply LLMs such as ChatGPT and GPT-4 in the second stage for inductive KGC because we cannot directly fine-tune LLMs for our task due to the resource limitations and closed source limitations. One solution may be applying LLMs using few-shot learning or in-context learning, but this solution may also suffer from the problems of low reasoning quality and length limitation when handling long context (e.g., the context of the entire KG is required). This is also out of our research field because LSTK focuses on providing faithful explanations for inductive KGC, based on a rule-based system learned from both structural and textual knowledge.\n\n> [R6] The motivations of the designed two step is unclear.\n\n[A] In our settings, we assume that a part of the knowledge exists in a structured KG, while the other part exists in text corpus. To mine textual knowledge (i.e., potential facts) from the given text corpus, we first employ a textual entailment model to generate soft triples. In the second stage, the generated soft triples are used to learn a rule-based system for inductive KGC. Considering that the number of mined soft triples is large, we cannot optimize these two-step in an end-to-end manner, as the end-to-end manner requires to store the gradients of all soft triples, resulting in out-of-memory (OOM) issues. Thus, we design a two-step framework in LSTK."}, {"review_id": "mlosljmE1J", "reviewer": "Reviewer_4GPq", "summary": "This paper proposes an inductive knowledge graph completion method based on both textual and structural information. It introduces a new task of how to leverage information from both modalities when there are both structured and textual data available. This paper proposes a rule mining method called LSTK-TELM, where LSTK represents a rule mining framework. Three datasets are constructed based on the relation extraction dataset for evaluation. Experimental results demonstrate the effectiveness of the proposed framework, with LSTK-TELM outperforming other baseline methods.", "questions": "1. Rules can also be used for transductive knowledge graph completion. Why does this paper specifically mention focusing on inductive KGC? There doesn't seem to be any specific design for the inductive setting.\n\n2. Please discuss the limitations of this paper. For example, the application scenario of the proposed task requires lots of text data related to the KG.", "limitations": "Please refer to the \"Weaknesses\" and \"Questions\" sections.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "S1. A novel knowledge graph completion scenario that combines both textual and RDF facts.\n\nS2. This paper proposes a framework called LSTK to address this problem and introduces the LSTK-TELM method.\n\nS3. Three datasets are constructed for evaluation. Experimental results demonstrate the effectiveness of the proposed method.\n\nS4. Well written.", "weaknesses": "W1. There is a lack of discussion on related works in the field of inductive link prediction. See references below:\n[1] Inductive Relation Prediction by Subgraph Reasoning\n[2] Topology-aware correlations between relations for inductive link prediction in knowledge graphs\n[3] Communicative Message Passing for Inductive Relation Reasoning\n[4] Subgraph Neighboring Relations Infomax for Inductive Link Prediction on Knowledge Graphs\n[5] Disconnected Emerging Knowledge Graph Oriented Inductive Link Prediction\n\nW2. The use of the symbol \"t\" for both tail entities and rule lengths may lead to confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments. The following addresses concerns and questions. \n\n> [R1] There is a lack of discussion on related works in the field of inductive link prediction.\n\n[A] Due to space limitations, we have only discussed the related work on learning rule-based systems for inductive link prediction. In reality, three main categories of methods have been proposed to address inductive link prediction, including GNN-based methods, PLM-based methods and rule-based methods. GNN-based methods such as R-GCN and other SOTA methods[1-5] address inductive link prediction based on graph neural networks (GNNs). PLM-based methods leverage pre-trained language models (PLMs) such as BERT to address the inductive link prediction problem. However, both GNN-based and PLM-based methods can be considered as embedding-based methods due to their black-box nature. For instance, the hidden representation learned by GNNs and PLMs can hardly be interpreted by human. Therefore, we focus on learning rule-based systems for inductive link prediction with textual knowledge, where logical rules (e.g., TE-rules) can serve as explanations for predictions. We naturally expect that the proposed framework LSTK is highly interpretable to human. Thus we have not incorporated GNN-based and PLM-based methods into LSTK for inductive KGC with textual knowledge. We will include these discussions in the revised version.\n\n> [R2] The use of the symbol \"t\" for both tail entities and rule lengths may lead to confusion.\n\n[A] Thank you for pointing out it. We will refine this notation in the revised version.\n\n> [R3] Rules can also be used for transductive knowledge graph completion. Why does this paper specifically mention focusing on inductive KGC? There doesn't seem to be any specific design for the inductive setting.\n\n[A] Inductive KGC is usually considered to be more practical and challenging in real-world scenarios. To guarantee the challenge and practicality of the proposed settings and datasets, we focus on inductive KGC in this work.\n\n> [R4] Please discuss the limitations of this paper. For example, the application scenario of the proposed task requires lots of text data related to the KG.\n\n[A] We have done this discussion in our supplement file (please see section E: Discussion on Limitations in the supplement file). We have discussed that the application scenario of our proposed framework requires KGs with corresponding text corpora. Considering that corresponding text corpora may be missing in most real-world KGs, we generate soft triples based on the distant supervision assumption that any sentence mentioning two entities might involve a relation between these entities. This assumption enables us to fetch texts via information retrieval tools such as search engines, without the need for manually filtering texts highly aligned with KGs. It is worth noting that both the orginal datasets of HacRED, DocRED and BioRel are collected using distant supervision. Considering that the process of fetching texts may introduce massive noises to the soft triples, we propose a novel formalism for logical rules (i.e. the TE-rules) to mitigate the negative impact of noises from soft triples."}]

Author rebuttal:

Meta-review: In this paper, the authors address knowledge graph completion (KBC) by introducing a two-stage framework that incorporates both structural and textual knowledge. They first generate "soft triples" with confidence scores from text data using distant supervision, using a multi-instance learning model to determine if a triple is entailed by sentences. In the second stage, they use these soft triples to learn rule-based models for KBC. To handle noise in soft triples, they propose "text-enhanced rules" (TE-rules) and a neural model to simulate their reasoning process. The approach is evaluated on three benchmark datasets, outperforming baseline methods in terms of various metrics such as MRR and Hits@k. The paper introduces an innovative way to leverage both structured and textual information for inductive KGC.
Decision: Accept (poster)