Title: Learning from Both Structural and Textual Knowledge for Inductive Knowledge Graph Completion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Knowledge graph completion; Neural approximate rule learning; Neural rule-based system
Abstract: Learning rule-based systems plays a pivotal role in knowledge graph completion (KGC). Existing rule-based systems restrict the input of the system to structural knowledge only, which may omit some useful knowledge for reasoning, e.g., textual knowledge. In this paper, we propose a two-stage framework that imposes both structural and textual knowledge to learn rule-based systems. In the first stage, we compute a set of triples with confidence scores (called \emph{soft triples}) from a text corpus by distant supervision, where a textual entailment model with multi-instance learning is exploited to estimate whether a given triple is entailed by a set of sentences. In the second stage, these soft triples are used to learn a rule-based model for KGC. To mitigate the negative impact of noise from soft triples, we propose a new formalism for rules to be learnt, named \emph{text enhanced rules} or \emph{TE-rules} for short. To effectively learn TE-rules, we propose a neural model that simulates the inference of TE-rules. We theoretically show that any set of TE-rules can always be interpreted by a certain parameter assignment of the neural model. We introduce three new datasets to evaluate the effectiveness of our method. Experimental results demonstrate that the introduction of soft triples and TE-rules results in significant performance improvements in inductive link prediction.

1

Abstract

Learning rule-based systems plays a pivotal role in knowledge graph completion (KGC). Existing rule-based systems restrict the input of the system to structural knowledge only, which may omit some useful knowledge for reasoning, e.g., textual knowledge. In this paper, we propose a two-stage framework that imposes both structural and textual knowledge to learn rule-based systems. In the first stage, we compute a set of triples with confidence scores (called soft triples ) from a text corpus by distant supervision, where a textual entailment model with multiinstance learning is exploited to estimate whether a given triple is entailed by a set of sentences. In the second stage, these soft triples are used to learn a rulebased model for KGC. To mitigate the negative impact of noise from soft triples, we propose a new formalism for rules to be learnt, named text enhanced rules or TE-rules for short. To effectively learn TE-rules, we propose a neural model that simulates the inference of TE-rules. We theoretically show that any set of TE-rules can always be interpreted by a certain parameter assignment of the neural model. We introduce three new datasets to evaluate the effectiveness of our method. Experimental results demonstrate that the introduction of soft triples and TE-rules results in significant performance improvements in inductive link prediction.

1 Introduction

Knowledge graph (KG) consists of real-world facts and has been widely used in many applications, including question answering [27], recommendation [22] and information retrieval [42]. A fact in KGs is usually represented by a triple of the form ( head , relation , tail ), where head and tail are entities. In general, KGs are highly incomplete since the complete set of facts is hard to collect. Therefore, knowledge graph completion (KGC), which aims to infer missing facts from observed ones, has become a vital ingredient for practically completing KGs.

In recent years, there are mainly two categories for prevalent approaches to KGC. One is the embedding-based category [3, 45, 33]. Methods in this category usually learn knowledge graph embeddings (KGE) by encoding entities and relations as low-dimensional real-value vectors. They have been shown to be effective for large-scale KGC, but are hard to generalize to the inductive scenario where missing facts involve previously unseen entities [20]. Besides, they can hardly be interpreted by human due to their black-box nature. The other category tackles KGC by learning rule-based systems, based on search algorithms with pruning heuristics [13, 24] or neural models for

Logical Rule:

HasWife(𝑥, 𝑦) ← HasDaughter(𝑥, z) ∧ HasMother(𝑧, y)

HasDaughter

×

HasMother

×

HasWife

HasDaughter

HasMother

√

HasWife

√

Supporting text:

Malia

and

her

mother

Michelle

went to

Liberia to …

approximate rule learning [46, 30, 28]. They excel in explaining why a missing fact is inferred and are able to handle the inductive setting.

Previous work focuses merely on learning rule-based systems from structural knowledge. However, the high incompleteness of structural knowledge imposes a heavy burden for building high-quality rule-based systems. Text corpora can naturally be considered to enrich structural knowledge as they contain much additional knowledge for reasoning. Figure 1 showcases an example about how textual knowledge help to KGC. The sub-figure on the left side shows a logical rule to infer whether x has wife y , and we cannot infer that 'Barack Obama' has wife 'Michelle Obam' since the key fact ('Malia Obama', HasMother, 'Michelle Obam') is missing in the background KG. The sub-figure on the right side shows that text corpora can provide supporting texts to this key fact, thereby the target relation HasWife can be correctly inferred.

Motivated by this kind of examples, we aim to learn rule-based systems from both structural and textual knowledge. To this end, we propose a two-stage framework named Learning from Structural and Textual Knowledge or LSTK for short, as illustrated in Figure 2. In the first stage, a distantlysupervised method using a textual entailment model is proposed to extract textual knowledge from texts. Distant supervision [26] posits that if two entities involve a relation, then any sentences that mention these two entities might express that relation. Based on this assumption, we generate a set of triples with their mentioned texts and employ a textual entailment model equipped with a multi-instance learning mechanism to estimate whether a triple is entailed by a set of sentences. Each generated triple has an estimated confidence score denoting the supporting degree of the triple by texts, and thus it is called a soft triple . In the second stage, both the generated soft triples and hard triples (i.e. existing triples) are used to learn a rule-based system for KGC.

In practice, it is challenging to learn rules from soft triples, as the soft triples generated by distant supervision may contain noise. To mitigate the negative impact of noisy data, we propose a new formalism for rules to be learnt, named text enhanced rules or TE-rules for short. In this formalism, textual relations are introduced to provide a flexible mechanism to discard noisy soft triples. In other word, a TE-rule is extended from a chain-like rule by adding atoms or disjunction of atoms possibly with textual relations to the rule body. To effectively learn TE-rules, we propose a neural model named TE-rule Learning Model or TELM for short. We show that an arbitrary set of TE-rules can be encoded by a certain parameter assignment of TELM (see Theorem 1). This theoretical result guarantees a certain degree of faithfulness between TELM and TE-rules, enabling us to extract explainable TE-rules from the parameter assignment of the learnt TELM. Based on the extracted TE-rules, LSTK is able to provide explanations for inferred missing facts by backward reasoning.

We enhance three datasets from the field of relation extraction for empirical evaluation. Experimental results demonstrate significant gains achieve by the proposed method LSTK-TELM in inductive link prediction. Our ablation study and case study further clarify why the introduction of soft triples and TE-rules help to improve the performance.

2 Preliminaries

Knowledge Graph. Given a set of entities E and a set of relations R , a knowledge graph G is a subset of E × R × E . Specifically, G = { ( h i , r i , t i ) } 1 ≤ i ≤ N , where N denotes the number of triples, h i ∈ E the head entity for the i th triple, r i ∈ R the relation for the i th triple and t i ∈ E the tail entity for the i th triple. By r -we denote the inverse relation of r ∈ R . The set of inverse relations for

R , namely { r -| r ∈ R} , is denoted by R -. Accordingly, the equivalent knowledge graph for G composed by inverse relations, namely { ( t, r -, h ) | ( h, r, t ) ∈ G} , is denoted by G -.

Logical Rule. Most existing work focuses on learning chain-like rules (CRs). A CR is essentially a plain datalog rule [1] where all atoms are binary and every body atom shares variables with the previous atom and the next atom. Formally, a CR R with L body atoms, simply called an L -CR, is of the form:

where x is the head entity, y the tail entity, and z 1 ,. . . , z L -1 variables. The part at the left (resp. right) side of ← is called the head (resp. body ) of R . The rule R is called r -specific if H = r . By H R and B R we denote the atom in the head of R and the set of atoms in the body of R , respectively. An atom or a rule is ground if it does not contain any variable. A rule R is a fact if B R is empty and H R is ground. In this paper, a fact or a ground atom r ( a, b ) and a triple ( a, r, b ) are used interchangeably. To uniformly represent rules using fixed-length bodies, we introduce the identity relation (denoted by I ) to rule bodies. For example, r ( x, y ) ← s ( x, y ) can be converted into a rule with two body atoms, namely r ( x, y ) ← s ( x, y ) ∧ I ( y, y ) .

We say G | = H R ( a, b ) if there exists a ground instance R g of R such that H R ( a, b ) = H R g and B R g ⊆ G ∪ G -∪ { I ( e, e ) | e ∈ E} . Let Σ be a set of r -specific CRs and ( a, r, b ) a triple. We say G | = Σ ( a, r, b ) if there exits a logical rule R ∈ Σ such that G | = H R ( a, b ) . We say a triple ( a, r, b ) is plausible in G if there exists a set of possibly correct r -specific CRs Σ such that G | = Σ ( a, r, b ) .

Link Prediction. Link prediction is the main task of knowledge graph completion that we focus on in this work. Given a knowledge graph G , a head query (? , r, t ) or a tail query ( h, r, ?) , the task of link prediction aims to find all entities e ∈ E such that ( e, r, t ) for (? , r, t ) or ( h, r, e ) for ( h, r, ?) is plausible in G . The inductive learning setting requires that at least one entity in the test set should not appear in the training set, while relations in the test set also appear in the training set.

3 Methodology

In this section, we describe the proposed LSTK framework for inductive KGC with text corpus. We first formalize our problem setting as follows.

Problem Setting. Given a set of triples and a corpus for training U train = ( G train , T train ) , a set of triples and a corpus for test U test = ( G test , T test ) , our inductive KGC setting aims to learn a KGC system based on U train , and then evaluate the learnt system on U test . During evaluation, given a head query (? , r, t ) or a tail query ( h, r, ?) , the learnt KGC system finds an answer with the highest estimated truth degree to answer this query, based on the background knowledge from ( G train ∪ G test \ { ( h, r, t ) } , T train ∪ T test ) .

This setting is motivated by real-world application scenarios where we need to fetch texts from search engines to find evidences to verify a new fact. To address this problem setting, we propose Learning from Structural and Textual Knowledge or LSTK for short, as illustrated in Figure 2. Specifically, the proposed LSTK framework has two stages, where the first stage aims to generate soft triples from a text corpus and the second one aims at training a rule-based system for KGC. Furthermore, LSTK is able to provide an explanation to show why an inferred missing fact is plausible in the background knowledge, by tracking back the reasoning paths from both hard triples and soft triples, where soft triples can further be explained by tracking back the supporting texts.

3.1 Formalization of LSTK

In the first stage, our goal is to find all triples that are possibly entailed by the given text corpus. We achieve this goal by adopting distant supervision. As far as we know, distant supervision is mostly applied in relation extraction, where the relation between any two entities is restricted in a close set. However, we naturally hope to discover more open relations that can be entailed by the text corpus. Considering that textual entailment [31] can deal with open relations and is better at exploiting the semantic information of relational contexts, we impose distant supervision to a textual entailment model to generate soft triples.

Distantly supervised data generation. Given a knowledge graph G and a set of sentences (i.e. a text corpus) T = { s i } 1 ≤ i ≤ N sen , we construct a training set D train = { ( τ i , S i , y i ) } 1 ≤ i ≤ N tra by

Text corpus

Textual entailment

model with multi-

instance learning

Textual knowledge

(Soft triples)

0.9

0.6

0.5

0.7

0.8

Reasoning paths

(Explanations)

Missing facts

0.95

Tracking back

Supporting hard triples

1.0

Supporting

soft triples

0.95

Supporting

texts

Structural knowledge

(Hard triples)

Mixed knowledge

(Hard triples + Soft triples)

0.9

0.6

1.0

0.7

0.8

1.0

1.0

Learning TE-rules by

TELM

Missing facts

0.95

Logical rules

Knowledge graph

completion and rule

extraction

Mixed knowledge

(Hard triples + Soft triples)

0.9

0.6

1.0

0.7

0.8

1.0

1.0

Logical rules

Backward reasoning

The LSTK framework

Textual knowledge

(Soft triples)

0.9

0.6

0.5

0.7

0.8

The first stage

The second stage

Distantly supervised

data generation

̸

finding all sentences in T that mention the entity pair ( h i , t i ) of τ i , where N sen denotes the number of sentences, N tra the number of training instances, τ i = ( h i , r i , t i ) a triple, S i the set of sentences in T mentioning the entity pair ( h i , t i ) , and y i the label. We call a triple ( h, r, t ) ∈ G a positive triple , whereas a triple ( h, r, t ) ̸∈ G a negative triple . For each positive triple ( h i , r i , t i ) , we append to the training set the set of negative triples { ( h i , r ′ j , t i ) | r ′ j ∈ R , r ′ j = r i , ( h i , r ′ j , t i ) / ∈ G} obtained by corrupting r i . Accordingly y i is set to 1 if τ i ∈ G or 0 otherwise. Considering that some entity pairs may appear in a large number of sentences, to confine the number of considering sentences, we exploit the well-known graph partitioning algorithm METIS [17] to evenly partition the original S i to make every part no more than 20 sentences, and then treat the densest part defined by Jaccard similarity between bags of words as the ultimate S i . Finally, we likewise construct the test set D test = { ( τ i , S i ) | τ i ∈ E × R × E , S i = ∅} for generating soft triples where labels are not given.

̸

Multi-instance learning for RTE. Recognising textual entailment (RTE) [9, 4] aims at determining whether a sentence (called hypothesis ) can be inferred by the other sentence (called premise ). We employ a textual entailment model based on the pre-trained language model BERT [10] to estimate the confidence score for every triple. Specifically, given D train = { ( τ i , S i , y i ) } 1 ≤ i ≤ N tra , we create a set of sequences X i = { x i,j } 1 ≤ j ≤ N S i for each instance in D train by filling the template '[CLS]<triple>[SEP]<sentence>[SEP]', where [CLS] and [SEP] are special tokens in BERT, <triple> denotes the slot to be filled by the context of τ i and <sentence> the slot to be filled by the context of s j ∈ S i . All the sequences in X i are fed to BERT to calculate their contextual representations and the contextual embedding of the [CLS] token are used to calculate the entailment probability by a full-connected layer activated by the sigmoid function. By ϵ i,j we denote the entailment probability for x i,j , then the entire entailment probability ϕ i for X i is calculated by ϕ i = max 1 ≤ j ≤ N S i ϵ i,j . The entire model is trained by minimizing the cross-entropy loss.

Soft triple generation. The trained model is then applied to D test for generating soft triples. A soft triple is defined as a quadruple ( h, r, t, ϕ ) , where ( h, r, t ) denotes a triple and ϕ its corresponding predicted entailment probability. Given a hyper-parameter δ to keep only highly confident soft triples, the set of soft triples that will be used in learning chain-like rules is defined as G soft = { ( h i , r i , t i , ϕ i ) | ( h i , r i , t i ) ∈ D test , ϕ i ≥ δ } .

3.2 Formalization of Text Enhanced Rules

In the second stage, we aim at learning logical rules from both hard (i.e. existing) and soft triples. However, it is challenging to learn rules from the background KG with soft triples, as the generated

soft triples by distant supervision may contain noise, which impairs the quality of learnt rules. The following example illustrates a case where noisy data impact the reasoning ability of a chain-like rule.

Example 1. Considering the chain-like rule R in Figure 1: HasWife( x, y ) ← HasDaughter( x, z ) ∧ HasMother( z, y ) . Suppose we have two soft triples '(A, HasDaughter , B, 0.55)' and '(B, HasMother , C, 0.90)' computed by the textual entailment model, where the former one is incorrect in the real world. From this rule R the negative triple '(A, HasWife , C)' will be incorrectly inferred to be positive.

To alleviate the negative impact of noise from soft triples, a simple way is to restrict that some atoms in the rule should appear in the structural knowledge as existing triples. To this end, we extend chain-like rules by adding atoms or disjunctions of atoms that may involve textual relations . We call such extended chain-like rules text enhanced rules and simply TE-rules. Let r text denote the textual relation corresponding to the relation r , then a TE-rule can be defined below.

Definition 1. A p -specific L -TE-rule R , simply a TE-rule if p and L are clear from the context, is of the form: p ( x, y ) ←C 1 ( x, z 1 ) ∧C 2 ( z 1 , z 2 ) ∧ ... ∧C L ( z L -1 , y ) , where C l can be an original relation, a textual relation, the identity relation, or a disjunction of an original relation and its corresponding textual relation, i.e., C l ( u, v ) is of the form r ( u, v ) ∨ r text ( u, v ) .

The following example explains why TE-rules can be used to resolve the error in Example 1.

Example 2. Considering Example 1 again. By allowing TE-rules to be learnt, a learnt rule R ′ can be: HasWife( x, y ) ← HasDaughter( x, z ) ∧ (HasMother( z, y ) ∨ HasMother text ( z, y )) . From R ′ the negative triple '(A, HasWife , C)' will not be inferred to be positive unless there exists a positive hard triple ( A, HasDaughter , B ) for some B .

3.3 End-to-end Learning of Text Enhanced Rules

To effectively learn TE-rules, we propose an end-to-end neural model named TE-rule Learning Model ( TELM for short). The intuition of TELM is to select appropriate parameters to simulate the inference of TE-rules by gradient descent. Formally, given a set of hard triples G , a set of soft triples G soft , the maximum number L of body atoms in every rule, the maximum number of rules N , we first extend G to G ext = { ( h, r, t, 1 . 0) | ( h, r, t ) ∈ G} . Suppose R = { r i } 1 ≤ i ≤ n , its corresponding set of inverse relations R -= { r i } n +1 ≤ i ≤ 2 n , its corresponding set of textual relations R text = { r i } 2 n +1 ≤ i ≤ 3 n , its corresponding set of inverse textual relations R -text = { r i } 3 n +1 ≤ i ≤ 4 n , as well as I = r 4 n +1 . Let K mix = G ext ∪ G -ext ∪ G soft ∪ G -soft ∪ { ( e, I, e, 1 . 0) | e ∈ E} . For 1 ≤ k ≤ N , 1 ≤ l ≤ L , TELM estimates a truth degree ψ ( k,l ) r,x,y for each triple ( x, r, y ) ∈ G , defined by:

where σ ( x ) = max(min( x, 1) , 0) is an activation function that confines the given value to [0 , 1] , I ( ϕ | C ) is a function that returns ϕ if C is true or 0 otherwise, and ω ( r,k,l ) ∈ [0 , 1] 4 n +1 denotes the trainable relational selection weights for the l th body atom of the k th rule for the head relation r . Intuitively, Equation (1) simulates the inference of TE-rules under the background KG with soft triples. The selection weights of all predicates for body atoms are formally defined as:

where [; ] denotes the concatenation operator, and w ( r,k,l ) orig ∈ [0 , 1] 2 n (resp. w ( r,k,l ) text ∈ [0 , 1] 2 n or w ( r,k,l ) 4 n +1 ] ∈ [0 , 1] ) denotes the original relation selection weights (resp. textual relation selection weights or the identity relation selection weight) for the l th atom of the k th rule for the head relation r ; further, α ( r,k,l ) ∈ [0 , 1] (resp. β ( r,k,l ) ∈ [0 , 1] ) denotes the trainable selection weight for determining whether the original (resp. textual) relation is involved in the l th atom. Intuitively, [ w ( r,k,l ) orig ] i = 1 (resp. [ w ( r,k,l ) text ] i = 1 or w ( r,k,l ) 4 n +1 = 1 ) denotes that the i th original relation r i (resp. the

i th textual relation r 2 n + i or the identity relation r 4 n +1 ) is selected as the predicate of the l th atom. Note that α ( r,k,l ) = 1 ∧ β ( r,k,l ) = 1 indicates that the disjunction of an atom with an original relation and an atom with a textual relation is involved in the l th atom. We confine α ( r,k,l ) and β ( r,k,l ) to [0 , 1] by sigmoid layers, normalize both w ( r,k,l ) orig and w ( r,k,l ) text by softmax layers, and restrict w ( r,k,l ) 4 n +1 to be max(0 , 1 -α ( r,k,l ) -β ( r,k,l ) ) so as to enforce α ( r,k,l ) = β ( r,k,l ) = 0 when w ( r,k,l ) 4 n +1 = 1 .

The final predicted truth degree is calculated by a weighted sum of predicted degrees for N rules:

where u ( k ) r ∈ [ -1 , 1] is a trainable weight for the k th rule for the head relation r , which is confined to [ -1 , 1] by a tanh layer. Then the model is trained by minimizing the objective function

We then show the faithfulness between the formalization of TELM and TE-rules in Theorem 1, which relies on the notion of induced parameters as defined below.

Definition 2. Given a set of r -specific L -TE-rules Σ = { R k } 1 ≤ k ≤ N for R k of the form r ( x, y ) ← C k, 1 ( x, z 1 ) ∧ ... ∧ C k,L ( z L -1 , y ) with C k,l ( u, v ) ∈ { r k,l ( u, v ) , r ′ k,l ( u, v ) , r † k,l ( u, v ) ∨ r ‡ k,l ( u, v ) , where r k,l ∈ R ∪ R -∪ { I } , r ′ k,l ∈ R text ∪ R -text , r † k,l ∈ R ∪ R -, and r ‡ k,l is the corresponding textual relation of r † k,l , we call a parameter assignment of TELM θ ( N,L ) r = { [ w ( r,k,l ) orig ] i , [ w ( r,k,l ) text ] i } 1 ≤ k ≤ N, 1 ≤ l ≤ L, 1 ≤ i ≤ 2 n ∪ { w ( r,k,l ) 4 n +1 , α ( r,k,l ) , β ( r,k,l ) } 1 ≤ k ≤ N, 1 ≤ l ≤ L ∪ { u ( k ) r } 1 ≤ k ≤ N Σ -induced if it satisfies the following conditions for all 1 ≤ k ≤ N, 1 ≤ l ≤ L :

(4)

if an original relation appears in

(5)

(6)

α

β

(

(

r,k,l r,k,l

(

)

= 1

)

k

= 1

)

r

u

= 1

.

Theorem 1. Let G be a set of hard triples, G soft a set of soft triples, ( a, r, b ) an arbitrary triple, Σ = { R k } 1 ≤ k ≤ N a set of r -specific L -TE-rules and θ ( N,L ) r the Σ -induced parameter assignment of TELM, then Ψ ( N,L ) r,a,b > 0 if and only if G ∪ G soft | = Σ r ( a, b ) .

Explaining by Backward Reasoning. Thanks to the faithfulness between TELM and TE-rules, we can extract TE-rules by interpreting the parameters of a learnt TELM model using beam search algorithm (see Algorithm 1 in the appendix). For each inferred missing fact, LSTK yields at least one reasoning path as explanation by backward reasoning. Given a triple ( h, r, t ) that is estimated to be positive, a set of extracted TE-rules, the background KG K mix and the corpus T , we can find a path between h and t . Whenever the path involves a soft triple, we treat the sentence in T that has the maximum confidence score as the supporting text for this soft triple.

4 Experimental Evaluation

4.1 Data Construction and Evaluation Metrics

We collected three benchmark KGs with corresponding text corpora from the field of relation extraction for empirical evaluation, which are HacRED 2 [7], DocRED 3 [48] and BioRel 4 [41].

if a textual relation appears in

C

C

k,l k,l

, otherwise

, otherwise

β

(

α

(

r,k,l r,k,l

)

= 0

)

= 0

.

.

Considering that these datasets originally have a number of relations that can hardly appear in heads of potential logical rules, we filtered these relations by applying AMIE+ [13] to mine logical rules that have up to three body atoms from the complete set of existing triples for each dataset and omitting the relations that have not been involved in the body atoms of any mined rules. Under the inductive setting, there should be entities in the test set that do not appear in the training set. Therefore, we applied the partitioning algorithm METIS [17] to the entity graph where an edge indicates that the two end-points have at least one filtered relation, so as to divide the set of entities into 10 subsets. The train/valid/test split of each dataset was then extracted from 8/1/1 of these subsets by omitting triples across different splits. Statistics on all above datasets are reported in Table 1.

Following [28], we reported the Mean Reciprocal Rank (MRR) and Hit@k metrics under the filtered setting [3] for empirical evaluation.

4.2 Implementation Details

We implemented the textual entailment model by Pytorch 1.10.0. The model was initialized by the pretrained language model BERT with 12 transformer layers, which outputs 768-dimensional (i.e. d = 768 ) token embeddings. Then it was trained by Adam [18] with warm up [10], where the initial learning rate was set to 5e-5, the mini-batch size to 8 and the maximum number of training epochs to 3. We applied dropout [32] to each layer by setting the dropout rate to 0.1.

We implemented TELM 5 by Pytorch 1.10.0. It was trained on an A100 GPU with 40GB memory by Adam [18] with 50 training epochs for HacRED and DocRED, and 20 for BioRel. The hyperparameters were set to maximize MRR on the validation set. The initial learning rate was set to 1e-1 and the mini-batch size to 32 for all datasets. We also applied dropout [32] to the output states by setting the dropout rate to 0.3. The maximum length of each rule L is set to 3 for HacRED and DocRED, and 2 for BioRel. The maximum size of learning rules N is set to 20 for all datasets.

4.3 Main Results

We conducted experiments on the three enhanced datasets for link prediction under the inductive learning setting, where existing triples in the training set are used as background KG for training, existing triples in training and validation sets are used as background KG for validation, and existing triples in training, validation and test sets are used for test. The hyper-parameter δ to filter soft triples with high confidence scores was set to 0.5 in all experiments.

Table 2 reports the comparison results on three datasets. Textual Copy Rule (simply TCR) is a baseline that uses only textual copy rules of the form ' r ( x, y ) ← r text ( x, y ) ' in reasoning instead of using any other rules or neural models. TCR has not any extra logical reasoning ability beyond the textual entailment model. By LSTK-X † we denote enhanced models that use soft triples in the background KG, where X denotes the original model. By LSTK-X we denote enhanced models that further extends LSTK-X † by allowing textual relations to appear in learnt rules. Results show that the proposed method (i.e. LSTK-TELM) significantly outperforms all the baseline methods with p-value < 0 . 05 by two-tailed t-tests. Specifically, LSTK-TELM outperforms LSTK-DRUM by absolute gains of 24.8%/4.2%/22.4% in Hit@1 scores on the HacRED/DocRED/BioRel datasets, respectively. Further, we can see from the comparison results between LSTK-X † and LSTK-X

that, the introduction of textual relations bring significant performance improvements. For LSTKNeuralLP † and LSTK-DRUM † , their performances are even worse than their original models that do not use any soft triples, indicating some negative impact of noise from soft triples.

We further conducted ablation studies on several variants of LSTK-TELM to verify the effectiveness of key components in both LSTK and TELM. In (1) we omitted all the soft triples in the background KG (i.e., only hard triples are used). Results show that the LSTK framework (using soft triples) pushes TELM by a significant margin with p-value 1.1e-6. It implies that the use of soft triples help to greatly improve the performance. In (2) we omitted the corresponding soft triple for each hard triple in the background KG. Results show that this variant model already significantly outperforms the variant model in (1), indicating that soft triples help to improve the reasoning ability without using textual copy rules. In (3) we omitted the textual relations in TE-rules. Results show that the use of textual relations pushes LSTK-TELM by a significant margin with p-value 5.1e-5. It can be explained by the fact that the separation of original and textual relations helps to discard the noisy soft triples, resulting in higher reasoning performance. In (4) we omitted the learning of disjunctive atoms of the form r ( u, v ) ∨ r text ( u, v ) in TELM. Results show that the introduction of disjunctive atoms bring significant performance gains with p-value 2.8e-4. This confirms that the introduction of disjunctive atoms in TE-rules provides a more flexible mechanism to control noise. In (5) we fixed the confidence scores (i.e. entailment probabilities) of soft triples to 1.0. Results show that the performance significantly drops with p-value 8.4e-3. It implies that the computed entailment probabilities reflect the truth degrees of supporting facts and help to learn more informative rules. In (6) we trained a state-of-the-art relation extraction model proposed in [2] based on BERT to replace the textual entailment model for generating soft triples. Results show that the use of relation extraction model leads to a significant performance degradation with p-value 1.9e-4. This may be due to that a textual entailment model can deal with open relations and exploit more text semantics on relations than a relation extraction model does.

4.4 Case Study

To clarify why the learning of TE-rules help improve the performance, we conducted case study for applying learnt logical rules from LSTK-TELM on the DocRED dataset, as shown in Table 3. In the first case, the logical rule is extracted from LSTK-TELM and can be applied to infer the positive triple ' ( Lois , Child , Superboy ) '. We can see from this case that the textual entailment model is able to infer from the relation mention 'biological child' that, the entities 'Lois' and 'Clark' are spouse, thereby yielding a supporting soft triple ' ( Lois , Spouse , Clark , 0 . 76) ' for reasoning. In the second case, it can be seen that the same logical rule in the first case cannot infer that the triple ' ( John , Child , Joseph ) ' is positive, since the key supporting fact ' ( Joseph , Father , Martyn ) ' is absent in the background KG. In the third case, the logical rule is ' Child ( x, y ) ← ( Spouse ( x, z 0 ) ∨ Spouse text ( x, z 0 )) ∧

Father -text ( z 0 , y ) '. In this case, the textual entailment model mistakenly yields a supporting soft triple ' ( Joseph , Father , Paul , 0 . 53) ' based on the words 'beatified Father', thereby wrongly inferring the triple ' ( Linda , Child , Joseph ) ' to be positive. In general, these cases show that the use of soft triples is crucial in yielding a plausible explanation for a test triple and that the separation of original and textual relations provides a flexible mechanism to discard noisy soft triples.

5 Related Work

5.1 Knowledge Graph Completion

There are two categories of prominent approaches to knowledge graph completion. They are embedding-based and rule-based, respectively.

Embedding-based methods. Knowledge graph embeddings (KGE) [3, 39, 19, 45, 35, 33] are a kind of well-known embedding-based methods. They aim to represent entities and relations in KGs as low-dimensional real-valued vectors [37]. Although KGE methods have been shown to be effective in large-scale KGC, they can hardly generalize to the inductive learning setting. Besides, KGE methods are hard to be interpreted by human due to their black-box nature. In recent years, graph neural networks (GNNs) [34, 6, 23, 43, 50] have shown promissing results in addressing KGC. They can also be considered as embedding-based, as they also learn latent representation for reasoning. While GNN-based methods have been shown to excel in handling the inductive learning setting, we have not incorporated GNNs with LSTK. The reasons are three-fold. Firstly, GNNs are still black-box models with limited interpretability, whereas the proposed LSTK framework is committed to explain why a missing fact is inferred. Secondly, GNN-based methods necessitate sub-graph extraction to handle test triples involving unseen entities. The process of sub-graph extraction becomes highly time-consuming when dealing with a large number of soft triplets, e.g., it takes about 10 hours for GraIL [34] to process 1M triples for sub-graph extraction. Thirdly, the confidence scores from soft triples cannot be directly incorporated in GNN-based methods, unless the message passing mechanism in GNN-based methods is redesigned to handle the confidence scores on links.

Rule-based methods. Different from embedding-based methods, rule-based methods aim at building effective rule-based systems for KGC. They can naturally reason under the inductive learning setting and provide logical rules as explanations. Learning rule-based systems has been widely studied in the field of Inductive Logic Programming (ILP). ILP methods such as AMIE+ [13] and AnyBURL [24] usually impose search algorithms with pruning heuristics to mine logical rules in a generate-and-test manner. RNNLogic [28] extends this manner to learn logical rules and their weights interactively based on neural models. In recent years, neural approximate methods are proposed to learn rule-based

systems from structural knowledge in KGs by learning continuous parameters using Tensorlog [8] operators. State-of-the-art neural approximate methods include NeuralLP [46], DRUM [30], NLIL [47] and Ruleformer [44]. In addition to applying Tensorlog, other neural approximate methods such as NTP [29] and CTP [25] learn logical rules based on neural theorem provers. All these methods learn rule-based systems from structural knowledge alone and most of them focus merely on learning chain-like rules. In contrast, we study learning more expressive TE-rules from both structural and textual knowledge.

Other approaches using textual information. It is worth noting that there are several approaches that focus on enhancing KGs with texts. For instance, recent advance of pre-trained language models (PLMs) for KGC such as KEPLER [38], PKGC [21] and SimKGC [36] have considered textual information of triples in KGs. The differences between PLM-based methods and LSTK are twofold. On one hand, they consider different types of textual knowledge. In more detail, PLM-based methods consider text information such as contexts of triples, descriptions aligned to entities and pre-trained textual knowledge. In contrast, LSTK is proposed to address the KGC scenario where some knowledge (i.e., real-world facts) is provided in the structured KG and other knowledge is given by a text corpus. Thus, the textual knowledge considered in LSTK corresponds to a set of potential facts, namely the soft triples. On the other hand, PLM-based methods cannot work in our scenario because they can only obtain textual knowledge by pre-training a model over a given text corpus, whereas our setting allows to process new text corpora given in the test phase (see the problem setting in Section 3). The pre-training process is generally time-consuming and requires massive computing resources, thus it is impractical to be applied in the test phase. Except for PLM-based methods, the work [12] proposes a knowledge verification system by combining logical and textual information to compute explanations for validating new triples. However, the system should work with a predefined logical theory. In contrast, our approach does not require any predefined logical theory but learns one approximate logical theory on the fly. The recent work [15] improves the knowledge coverage of a knowledge base (KB) by facts extracted from text corpora. They have not focused on reasoning over facts. In contrast, we focus on enhancing the reasoning ability of rule-based systems by soft triples extracted from text corpora.

5.2 Distant Supervision for Relation Extraction

Distant supervision [26] is a dominant paradigm to handle the problem of lacking labeled data in relation extraction. It assumes that if two entities involve a relation, then any sentence mentioning them together might express that relation. Based on this assumption, a large number of labeled data for relation extraction can be automatically generated by aligning triples with texts. However, this may introduce noise to the generated data. Therefore, most previous studies exploit multi-instance learning [11] by putting instances with the same entity pair into bags, to alleviate the impact of noisy instances. A number of neural models including CNN [49], Bi-LSTM [5] and BERT [40] have been incorporated with multi-instance learning and used as classification models for relation extraction. We also impose distant supervision to score soft triples from a text corpus, but we resort to textual entailment models to make the best of relation mentions, instead of using classification models where relations are treated as labels to be predicted.

6 Conclusion and Future Work

In this work we have proposed a two-stage framework, named LSTK, to learn rule-based systems from both structural and textual knowledge. It computes a set of soft triples by distant supervision in the first stage and applies these soft triples to enhance the learning of neural approximate rule-based systems in the second stage. To mitigate the negative impact of noise from soft triples, we have proposed a new formalism for logical rules named TE-rules and a neural model named TELM for learning TE-rules. We introduced three new datasets for empirical evaluation. Experimental results demonstrate significant improvements achieved by learning TE-rules from soft triples. Our case study further reveals how TE-rules help to control noise from soft triples.

From the third case in case study, we observe that some errors produced by the textual entailment model will propagate to the final prediction. This is known as the error propagation issue in pipeline approaches. To tackle this issue, our future work will focus on studying fully end-to-end frameworks to learn TE-rules from both structural and textual knowledge.

7 Limitations

LSTK merely works for knowledge graphs with aligned text corpora, but in general knowledge graphs do not come with aligned texts. In order to adapt LSTK to more real-world applications, we generate soft triples automatically under the distant supervision assumption that any sentence mentioning two entities might involve a relation between these two entities. This assumption enables us to fetch texts via information retrieval tools such as search engines or to generate supporting text via large language models (LLMs) without the need of manually constructing texts. Note that all the original datasets of HacRED [7], DocRED [48] and BioRel [41] are collected by distant supervision. Although the performance of LSTK depends on the quality of distant supervision, LSTK indeed works for any knowledge graphs since aligned texts can easily be collected by distant supervision.

References

Serge Abiteboul, Richard Hull, and Victor Vianu. Foundations of Databases . Addison-Wesley, 1995.

Christoph Alt, Marc Hübner, and Leonhard Hennig. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In ACL , pages 1388-1398, 2019.

Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS , pages 2787-2795, 2013.

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP , pages 632-642, 2015.

Rui Cai, Xiaodong Zhang, and Houfeng Wang. Bidirectional recurrent convolutional neural network for relation classification. In ACL , 2016.

Jiajun Chen, Huarui He, Feng Wu, and Jie Wang. Topology-aware correlations between relations for inductive link prediction in knowledge graphs. In AAAI , pages 6271-6278, 2021.

Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. Hacred: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of ACL , pages 2819-2831, 2021.

William W. Cohen, Fan Yang, and Kathryn Mazaitis. Tensorlog: A probabilistic database implemented using deep-learning infrastructure. J. Artif. Intell. Res. , 67:285-325, 2020.

Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005 , volume 3944, pages 177-190, 2005.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , pages 4171-4186, 2019.

Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. , 89(1-2):31-71, 1997.

Jianfeng Du, Jeff Z. Pan, Sylvia Wang, Kunxun Qi, Yuming Shen, and Yu Deng. Validation of growing knowledge graphs by abductive text evidences. In AAAI , pages 2784-2791, 2019.

Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. Fast rule mining in ontological knowledge bases with AMIE+. VLDB J. , 24(6):707-730, 2015.

Luis Antonio Galárraga, Christina Teflioudi, Katja Hose, and Fabian M. Suchanek. AMIE: association rule mining under incomplete evidence in ontological knowledge bases. In WWW , pages 413-422, 2013.

Vinh Thinh Ho, Daria Stepanova, Dragan Milchevski, Jannik Strötgen, and Gerhard Weikum. Enhancing knowledge bases with quantity facts. In WWW , pages 893-901. ACM, 2022.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997.

George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Sci. Comput. , 20(1):359-392, 1998.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI , pages 2181-2187, 2015.

Shuwen Liu, Bernardo Cuenca Grau, Ian Horrocks, and Egor V. Kostylev. INDIGO: gnn-based inductive knowledge graph completion using pair-wise encoding. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, NeurIPS , pages 2034-2045, 2021.

Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. Do pre-trained models benefit knowledge graph completion? A reliable evaluation and a reasonable approach. In Findings of ACL , pages 3570-3581, 2022.

Xinze Lyu, Guangyao Li, Jiacheng Huang, and Wei Hu. Rule-guided graph neural networks for recommender systems. In ISWC , pages 384-401, 2020.

Sijie Mai, Shuangjia Zheng, Yuedong Yang, and Haifeng Hu. Communicative message passing for inductive relation reasoning. In AAAI , pages 4294-4302, 2021.

Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. In IJCAI , pages 3137-3143, 2019.

Pasquale Minervini, Sebastian Riedel, Pontus Stenetorp, Edward Grefenstette, and Tim Rocktäschel. Learning reasoning strategies in end-to-end differentiable proving. In ICML , volume 119, pages 69386949, 2020.

Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation extraction without labeled data. In ACL , pages 1003-1011, 2009.

Arindam Mitra and Chitta Baral. Addressing a question answering challenge by combining statistical methods with inductive rule learning and reasoning. In AAAI , pages 2779-2785, 2016.

Meng Qu, Junkun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio, and Jian Tang. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. In ICLR , 2021.

Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In NIPS , pages 3788-3800, 2017.

Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. DRUM: end-to-end differentiable rule mining on knowledge graphs. In NeurIPS , pages 15321-15331, 2019.

Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. Label verbalization and entailment for effective zero and few-shot relation extraction. In EMNLP , pages 1199-1212, 2021.

Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR) , 15(1):1929-1958, 2014.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR , 2019.

Komal K. Teru, Etienne G. Denis, and William L. Hamilton. Inductive relation prediction by subgraph reasoning. In ICML , volume 119, pages 9448-9457, 2020.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In ICML , volume 48, pages 2071-2080, 2016.

Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. Simkgc: Simple contrastive knowledge graph completion with pre-trained language models. In ACL , pages 4281-4294, 2022.

Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Trans. Knowl. Data Eng. (TKDE) , 29(12):2724-2743, 2017.

Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representation. Trans. Assoc. Comput. Linguistics (TACL) , 9:176-194, 2021.

Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI , pages 1112-1119, 2014.

Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. Denoising relation extraction from document-level distant supervision. In EMNLP , pages 3683-3688, 2020.

Rui Xing, Jie Luo, and Tengwei Song. Biorel: towards large-scale biomedical relation extraction. BMC Bioinform. , 21-S(16):543, 2020.

Chenyan Xiong, Russell Power, and Jamie Callan. Explicit semantic ranking for academic search via knowledge graph embedding. In WWW , pages 1271-1279, 2017.

Xiaohan Xu, Peng Zhang, Yongquan He, Chengpeng Chao, and Chaoyang Yan. Subgraph neighboring relations infomax for inductive link prediction on knowledge graphs. In IJCAI , pages 2341-2347, 2022.

Zezhong Xu, Peng Ye, Hui Chen, Meng Zhao, Huajun Chen, and Wen Zhang. Ruleformer: Context-aware rule mining over knowledge graph. In COLING , pages 2551-2560, 2022.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In ICLR , 2015.

Fan Yang, Zhilin Yang, and William W. Cohen. Differentiable learning of logical rules for knowledge base reasoning. In NIPS , pages 2319-2328, 2017.

Yuan Yang and Le Song. Learn to explain efficiently via neural logic inductive learning. In ICLR , 2020.

Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. Docred: A large-scale document-level relation extraction dataset. In ACL , pages 764-777, 2019.

Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. Distant supervision for relation extraction via piecewise convolutional neural networks. In EMNLP , pages 1753-1762, 2015.

Yufeng Zhang, Weiqing Wang, Hongzhi Yin, Pengpeng Zhao, Wei Chen, and Lei Zhao. Disconnected emerging knowledge graph oriented inductive link prediction. In ICDE , pages 381-393, 2023.

A Proofs

A.1 Proof of Lemma 1

To prove Theorem 1, we first introduce Lemma 1.

Lemma 1. Let G be a set of hard triples, G soft a set of soft triples, ( a, r, b ) an arbitrary triple, R an r -specific L -TE-rule of the form r ( x, y ) ←C 1 ( x, z 1 ) ∧C 2 ( z 1 , z 2 ) ∧ ... ∧C L ( z L -1 , y ) , and θ (1 ,L ) r the { R } -induced parameter assignment of TELM, then (1) ψ (1 ,L ) r,a,b > 0 if G ∪ G soft | = H R ( a, b ) , and (2) ψ (1 ,L ) r,a,b = 0 if G ∪ G soft ̸| = H R ( a, b ) .

Proof. Let K = G ∪ G -∪ G soft ∪ G -soft ∪ { ( e, I, e ) | e ∈ E} where E is the set of entities appearing in G ∪ G soft . Note that the confidence sore of each triple in K is larger than 0.

(I) Consider the case where G ∪ G soft | = H R ( a, b ) . There exists at least one ground instance R g of R such that H R ( a, b ) = H R g and B R g ⊆ K . There will be a sequence of entities c 1 , . . . , c L -1 and a sequence of relations r 1 , . . . , r L such that ( a, r 1 , c 1 ) , ( c 1 , r 2 , c 2 ) . . . , ( c L -1 , r L , b ) ∈ K , where r i is a relation appearing in C i for all 1 ≤ i ≤ L . If r L ∈ R ∪ R -, suppose r L is the k th relation in R∪R -, then by Condition 1 and 4 in Definition 2, we have [ w ( r, 1 ,L ) orig ] k = 1 and α ( r, 1 ,L ) = 1 . If r L ∈ R text ∪ R -text , suppose r L is the k th relation in R text ∪ R -text , then by Condition 2 and 5 in Definition 2, we have [ w ( r, 1 ,L ) text ] k = 1 and β ( r, 1 ,L ) = 1 . If r L = I , by Condition3 in Definition 2, we have w ( r, 1 ,L ) 4 n +1 = 1 . By Equation (2), we must have ω ( r, 1 ,L ) k = 1 for some k . By Equation (1), we further have ψ (1 , 1) r,c L -1 ,b ≥ ϕ L > 0 for ϕ L the confidence score of ( c L -1 , r L , b ) . Likewise, if r L -1 ∈ R∪R -, suppose r L -1 is the k th relation in R∪R -, then by Condition 1 and 4 in Definition 2, we have [ w ( r, 1 ,L -1) orig ] k = 1 and α ( r, 1 ,L ) = 1 . If r L -1 ∈ R text ∪ R -text , suppose r L -1 is the k th relation in R text ∪ R -text , then by Condition 2 and 5 in Definition 2, we have [ w ( r, 1 ,L -1) text ] k = 1 and β ( r, 1 ,L -1) = 1 . If r L -1 = I , by Condition3 in Definition 2, we have w ( r, 1 ,L -1) 4 n +1 = 1 . By Equation (2), we must have ω ( r, 1 ,L -1) k = 1 for some k . By Equation (1), we further have ψ (1 , 2) r,c L -2 ,b > 0 . In the same way, we can show that ψ (1 , 3) r,c L -3 ,b > 0 , . . . , ψ (1 ,L -1) r,c 1 ,b > 0 and ψ (1 ,L ) r,a,b > 0 in turn. Therefore, we have ψ (1 ,L ) r,a,b > 0 if G ∪ G soft | = H R ( a, b ) .

(II) Consider the case where G ∪ G soft ̸| = H R ( a, b ) . Suppose ψ (1 ,L ) r,a,b > 0 , then by Equation (1), there must be some k ∈ { 1 , . . . , 4 n +1 } such that ω ( r, 1 , 1) k > 0 , r k appears in C 1 and there exists ( a, r k , c 1 ) ∈ K fulfilling ψ (1 ,L -1) r,c 1 ,b > 0 . Since ψ (1 ,L -1) r,c 1 ,b > 0 , by Equation (1), there must be also some l ∈ { 1 , . . . , 4 n +1 } such that ω ( r, 1 , 2) l > 0 , r l appears in C 2 and there exists ( c 1 , r l , c 2 ) ∈ K fulfilling ψ (1 ,L -2) r,c 2 ,b > 0 . In the same way, we can show that there exists relation r m appearing in C m and entity c m such that ( c m -1 , r m , c m ) ∈ K and ψ (1 ,L -m ) r,c m ,b > 0 for m = 3 , . . . , L -1 in turn, while there exists relation r L appearing in C L such that ( c L -1 , r L , b ) ∈ K . Hence there exists a sequence of entities c 1 , . . . , c L -1 and a sequence of relations r 1 , . . . , r L such that ( a, r 1 , c 1 ) , ( c 1 , r 2 , c 2 ) . . . , ( c L -1 , r L , b ) ∈ K , where r i is a relation appearing in C i for all 1 ≤ i ≤ L . These two sequences constitute a ground instance R g of R such that H R ( a, b ) = H R g and B R g ⊆ K , contradicting G ∪ G soft ̸| = H R ( a, b ) . Thus ψ (1 ,L ) r,a,b ≤ 0 . By Equation (1) we have ψ (1 ,L ) r,a,b ≥ 0 . Therefore, we have ψ (1 ,L ) r,a,b = 0 if G ∪ G soft ̸| = H R ( a, b ) .

A.2 Proof of Theorem 1

Proof. Lemma 1 implies that, for all R k ∈ Σ , ψ ( k,L ) r,a,b > 0 if G ∪ G soft | = H R k ( a, b ) and ψ ( k,L ) r,a,b = 0 otherwise.

( ⇒ ) Suppose Ψ ( N,L ) r,a,b > 0 . Then by Equation (3) and Condition 6 in Definition 2, there exists at least one r -specific L -TE-rule R k ∈ Σ such that ψ ( k,L ) r,a,b > 0 . By Lemma 1 we have G∪G soft | = H R k ( a, b ) . Since G ∪ G soft | = H R k ( a, b ) and R k ∈ Σ , we have G ∪ G soft | = Σ ( a, r, b ) .

̸

( ⇐ ) Suppose G ∪ G soft | = Σ ( a, r, b ) . Then we have G ∪ G soft | = H R k ( a, b ) for some R k ∈ Σ . By Lemma 1 we have ψ ( k,L ) r,a,b > 0 and for all k ′ = k , ψ ( k ′ ,L ) r,a,b ≥ 0 . By Equation (3) and Condition 6 in Definition 2, we have Ψ ( N,L ) r,a,b > 0 .

B Formalization of Multi-instance Learning for RTE

In Section 4.1 we have described how to generate soft triples by applying a RTE model based on BERT [10]. Due to the space limitation, we omitted some details. Thus, in this section we further elaborate on how to apply multi-instance learning to RTE based on BERT.

Given D train = { ( τ i , S i , y i ) } 1 ≤ i ≤ N tra , where τ i = ( h i , r i , t i ) denotes a triple, h i = ( w h i 1 , ..., w h i N h i ) the head entity with N h i tokens, r i = ( w r i 1 , ..., w r i N r i ) the relation with N r i tokens, t i = ( w t i 1 , ..., w t i N t i ) the tail entity with N t i tokens, S i = { s i,j } 1 ≤ j ≤ N S i a set of N S i sentences mentioning ( h i , t i ) , and s i,j = ( w s i,j 1 , ..., w s i,j N s i,j ) a sentence with N s i,j tokens, we create a set of sequences X i = { x i,j } 1 ≤ j ≤ N S i for each instance in D train . For 1 ≤ j ≤ N S i , the sequence x i,j is created by first concatenating h i , r i , t i and s i,j , and then adding a special token [CLS] in front of the first token w h i 1 , a special token [SEP] between w t i N t i and w s i,j 1 , and a special token [SEP] behind the last token w s i,j N s i,j . All sequences are fed to BERT to calculate their contextual representations. By C i,j ∈ R N x i,j × d we denote the output of BERT for x i,j , where N x i,j = N h i + N r i + N t i + N s i,j +3 is the number of tokens in x i,j and d is the dimension of token embedding. We use the embedding of the [CLS] token, namely C i,j, 1 , as the contextual embedding for textual entailment classification. The entailment probability for x i,j is calculated by

where W ∈ R 1 × d is a trainable weight vector, b ∈ R 1 a trainable bias value and σ the sigmoid function. Then the entailment probability for X i is calculated by

The entire model is trained by minimizing the cross-entropy loss, formally defined as

C Rule Extraction From TELM

Based on the formalization of TELM, we can interpret TE-rules from its parameters. The process of interpretation is shown in Algorithm 1. Intuitively, Algorithm 1 interprets TE-rules from the parameter assignment of TELM using beam search, where π denotes the threshold to determine whether the original relations and textual relations are used in atoms, m the beam size, f l the set of ( R ′ , s )-pairs for the l th atom, R ′ the currently interpreted (partial) rule and s its estimated score. It should be noted that the process for interpreting r -specific TE-rules outputs up to m interpreted rules sharing the same confidence score u ( k ) r for the k th target rule.

D Experiments

D.1 Baselines

We applied LSTK to the following state-of-the-art rule learning methods for empirical comparisons.

AMIE+. AMIE+ [13] is upgraded from the well-known rule learner AMIE [14]. It imposes several pruning heuristics to mine logical rules.

NeuralLP. NeuralLP [46] is the first neural approximate method that exploits Tensorlog operators to learn chain-like rules. It translates the structure learning of logical rules from discrete space to continuous space.

DRUM. DRUM [30] extends NeuralLP to learn rules with dynamic lengths by adding the identity relation. It exploits a bidirectional LSTM network to learn logical rules together with their scores simultaneously.

RNNLogic. RNNLogic [28] leverages an LSTM [16] based rule generator and a reasoning predictor to learn logical rules together with their weights interactively.

For LSTK-NeuralLP, LSTK-DRUM and LSTK-RNNLogic, we constructed adjacency matrix for each relation from the mixed background K mix , where the weights of edges are set by the confidence scores. It should be noted that AMIE+ can only learn rules from hard triples in the background KG, and thus the confidence scores of soft triples are not used in LSTK-AMIE+.

D.2 Implementation of Baselines

Weimplemented LSTK-NeuralLP 6 and LSTK-DRUM 7 by Tensorflow 1.15.0 based on their published code. They were trained by Adam, where the initial learning rate was set to 1e-3, the mini-batch size to 32, the maximum number of training epochs to 10, and the maximum length of learnt rules to 4.

HacRED

DocRED

BioRel

Dataset

0

20

40

60

80

100

Proportion of rules with textual predicates

25.5

27.5

27.9

28.9

36.4

23.3

63.3

63.9

60.6

95.9

83.6

95.3

LSTK-NeuralLP

LSTK-DRUM

LSTK-RNNLogic

LSTK-TELM

(a) Proportion of rules involving textual relations.

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Threshold for filtering soft triples

0.44

0.46

0.48

0.50

0.52

LSTK-TELM

LSTK-DRUM

(b) Comparisons on δ for LSTK-TELM.

LSTK-RNNLogic 8 was implemented by Pytorch 1.10.0 based on its published code. Both the rule generator and reasoner in RNNLogic were trained by Adam, where the initial learning rate was set to 1e-3 (resp. 1e-1) for the generator (resp. the reasoner), the maximum number of training epochs to 5000 (resp. 5) for the generator (resp. the reasoner), the mini-batch size to 1024, and the maximum length of learnt rules to 2. Experiments on these baselines were conducted on an A100 GPU with 40GB memory. We implemented LSTK-AMIE+ 9 based on its published code, and ran it on a Linux equipped with an Intel(R) Xeon(R) Gold 6248R processor with 3.0 GHz and 128 GB RAM.

D.3 Evaluation Metrics

For each test triple ( h, r, t ) in evaluation, we built two queries (? , r, t ) and (? , r -, h ) for answering h and t , respectively. For each query, we computed the truth degrees for corrupted head triples and then computed the rank of the correct answer. The rank of the correct answer is defined by m +( n +1) / 2 , where m is the number of triples with higher truth degrees than the correct answer and n the number of corrupted head triples with the same truth degree as the correct answer. The rank for each test triple is calculated by the mean rank of two queries. Based on the rank, we reported the Mean Reciprocal Rank (MRR for short) and Hit@k (H@k for short) metrics under the filtered setting.

D.4 Statistical Analysis on Learnt Rules

We conduced statistical analysis on learnt rules to show how much textual knowledge has been used in different models, as shown in sub-figure (a) in Figure 3. We can see that about 30 % logical rules extracted from either LSTK-NeuralLP or LSTK-DRUM (resp. about 60 % logical rules extracted from LSTK-TELM) involve at least one textual relation. It implies that more soft triples are leveraged in the reasoning of LSTK-TELM. We have also found that about 90 % logical rules extracted from LSTK-RNNLogic involve at least one textual relation, which is much higher than the rate of other approaches. This may be due to the large search space explored by RNNLogic in searching potential paths from the KG for rule generation, where the introduction of soft triples substantially scales up the search space, leading to a lot of logical rules that involve textual relations. Recall Table 2 that LSTK-RNNLogic achieves lower performance than other LSTK-enhanced models. These results imply that the overuse of textual relations may degrade the performance due to noise in soft triples.

D.5 Hyper-parameter Analysis

LSTK introduces a threshold δ to filter soft triples with high confidence scores. We conducted experiments to show the impact of different setting of δ , as shown in sub-figure (b) in Figure 3. Each curve on the MRR scores has a moderate jitter with increasing δ . This can be explained by that, although the use of soft triples tends to improve the performance of link prediction, the overuse of soft triples may introduce much noise, leading to a performance degradation. By observing the curves of LSTK-DRUM and LSTK-TELM, we recommend to set δ = 0 . 5 as default.

MRR

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an inductive knowledge graph completion method based on both textual and structural information. It introduces a new task of how to leverage information from both modalities when there are both structured and textual data available. This paper proposes a rule mining method called LSTK-TELM, where LSTK represents a rule mining framework. Three datasets are constructed based on the relation extraction dataset for evaluation. Experimental results demonstrate the effectiveness of the proposed framework, with LSTK-TELM outperforming other baseline methods.\n\nSTRENGTHS:\nS1. A novel knowledge graph completion scenario that combines both textual and RDF facts.\n\nS2. This paper proposes a framework called LSTK to address this problem and introduces the LSTK-TELM method.\n\nS3. Three datasets are constructed for evaluation. Experimental results demonstrate the effectiveness of the proposed method.\n\nS4. Well written.\n\nWEAKNESSES:\nW1. There is a lack of discussion on related works in the field of inductive link prediction. See references below:\n[1] Inductive Relation Prediction by Subgraph Reasoning\n[2] Topology-aware correlations between relations for inductive link prediction in knowledge graphs\n[3] Communicative Message Passing for Inductive Relation Reasoning\n[4] Subgraph Neighboring Relations Infomax for Inductive Link Prediction on Knowledge Graphs\n[5] Disconnected Emerging Knowledge Graph Oriented Inductive Link Prediction\n\nW2. The use of the symbol \"t\" for both tail entities and rule lengths may lead to confusion.\n\nQUESTIONS:\n1. Rules can also be used for transductive knowledge graph completion. Why does this paper specifically mention focusing on inductive KGC? There doesn't seem to be any specific design for the inductive setting.\n\n2. Please discuss the limitations of this paper. For example, the application scenario of the proposed task requires lots of text data related to the KG.\n\nLIMITATIONS:\nPlease refer to the \"Weaknesses\" and \"Questions\" sections.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a two-stage framework that imposes both structural and textual knowledge to conduct knowledge graph completion. The first stage aims to extract some soft triples with confidence scores. And then the second stage designs some tailored rules for entity link prediction, including text-enhanced rules and TE-rules. Experiments on three benchmarks show the proposed method significantly outperforms previous work.

STRENGTHS:
1. The idea of this paper is clear.
2. Experimental results show significant improvements to baseline models.

WEAKNESSES:
1. The text-enhanced entity representation models have been widely studied, such as [1][2]. Related work should be discussed. 
2. The two-step pipeline will bring unnecessary cascaded errors. The most directed way is enhancing the entity representations using text representations to directly predict the entity links.
3. The first step with confidence score seems to model uncertainty in linking entities, which may be a novelty. However, such a motivation is not clearly described.
4. The relationship between the designed step should also be emphasized. Such as, the first step brings more entity semantics from text and the second step verifies the entity relations via text and structure aspects.
5. LLMs have shown strong ability in building KGs and exacting entity relations. This paper does not conduct experiments on LLMs.

[1] Representation Learning of Knowledge Graphs with Entity Descriptions. AAAI 2016.
[2] Entity-duet neural ranking: Understanding the role of knowledge graph semantics in neural information retrieval. ACL 2018.

QUESTIONS:
The motivations of the designed two step is unclear.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors try to solve Knowledge Graph Completion (KGC) by proposing a two-stage framework Learning from Structural and Textual Knowledge (LSTK), that imposes both structural and textual knowledge to learn rule-based systems. In the first stage, the authors compute a set of triples with confidence scores. To mitigate the influence of the noise in the set of triples estimated in the first stage, the authors also propose a threshold-based method text enhanced rules (TE-rules) to filter out low-confident rules. In the second stage, the authors use the filtered triples by TE-rules to train a neural model named TE-rule Learning Model (TELM) for KGC. Furthermore, the authors created new datasets based on the data, HacRED, DocRED, and BioRel. Experimental results show that the proposed method LSTK outperforms the baseline methods regarding MRR, Hits@k (k=1,3,10).\n\nSTRENGTHS:\n- Since the proposed method LSTK relies on rule-based inference, the estimation of this method is interpretable to humans.\n- The proposed approach can restrict the automatically extracted rules to high-confidence ones by TE-rules to deal with the noisy rules extracted by distant supervision.\n- The threshold of filtering soft triples does not require sensitive tuning and works with 0.5. Thus, LSTK is robust and easy to use.\n- This work provides new KG datasets with their corresponding texts.\n\nWEAKNESSES:\n- The paper does not refer to the recent KGC model that can conduct KGC with unseen entities [1].\n- Even though recent KGC models utilize pretrained language models considering textual information of triplets in KGs [2], there is no comparison against such models.\n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9, 176-194. (https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00360/98089/KEPLER-A-Unified-Model-for-Knowledge-Embedding-and)\n\n[2] Lv, X., Lin, Y., Cao, Y., Hou, L., Li, J., Liu, Z., ... & Zhou, J. (2022, May). Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach. In Findings of the Association for Computational Linguistics: ACL 2022 (pp. 3570-3581). (https://aclanthology.org/2022.findings-acl.282/)\n\nQUESTIONS:\nAs shown in the example of Figure 1, the proposed method LSTK can cover inferences of unseen entities and relations. However, as shown in the inductive setting of Wikidata5M [1], recent prompt-based KGC models can complete such triples based on text information like descriptions and pretrained knowledge similar to your work. Thus, to claim the advantage of LSTK, you need to compare it to the recent KGC models like KEPLER [1] and SimKGC [3]. Also, could you explain the difference between these models and LSTK?\n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9, 176-194. (https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00360/98089/KEPLER-A-Unified-Model-for-Knowledge-Embedding-and)\n\n[3] Wang, L., Zhao, W., Wei, Z., & Liu, J. (2022, May). SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 4281-4294). (https://aclanthology.org/2022.acl-long.295/)\n\nLIMITATIONS:\nI think you need to add that this approach is difficult to work on knowledge graphs that do not have corresponding text data to Limitations in your paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a rule-based inductive KGC method with two stages. In the first stage, it extracts soft triples from a text corpus using distant supervision. In the second stage, the obtained soft triples are mixed with the original hard triples and used to learn rule-based model for KGC.\n\nSTRENGTHS:\n(1) It is interesting and intuitive to enrich the original triples in KG by extract new triples from text corpus.\n(2) The idea that uses extracted soft triples from text corpus to enhance logical rules is novel to some extent.\n\nWEAKNESSES:\n(1) This paper uses both structural and textual knowledge for inductive KGC, but there is no discussion about other related works which also use structural and textual knowledge, such as  KEPLER[1] and BERTRL[2].\n(2) Since the proposed method extract new triples from text corpus, if the unseen entities in test set may seen in the new soft triples. It is better to provide statistics about the  unseen entities in original triples, soft triples and both triples.\n(3) Some notations are confusing, for example the \"t\" is used for a tail entity in a triple (h,r,t), and also used in \"1≤t≤T\". \n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9, 176–194. \n[2] [2] Zha, H., Chen, Z., & Yan, X. (2022). Inductive Relation Prediction by BERT. Proceedings of the AAAI Conference on Artificial Intelligence, 36(5), 5923-5931.\n\nQUESTIONS:\n(1) This paper uses both structural and textual knowledge for inductive KGC, but there is no discussion about other related works which also use structural and textual knowledge, such as  KEPLER[1] and BERTRL[2].\n(2) Since the proposed method extract new triples from text corpus, if the unseen entities in test set may seen in the new soft triples. It is better to provide statistics about the  unseen entities in original triples, soft triples and both triples.\n(3) Some notations are confusing, for example the \"t\" is used for a tail entity in a triple (h,r,t), and also used in \"1≤t≤T\". \n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9, 176–194. \n[2] [2] Zha, H., Chen, Z., & Yan, X. (2022). Inductive Relation Prediction by BERT. Proceedings of the AAAI Conference on Artificial Intelligence, 36(5), 5923-5931.\n\nLIMITATIONS:\n(1) This paper uses both structural and textual knowledge for inductive KGC, but there is no discussion about other related works which also use structural and textual knowledge, such as  KEPLER[1] and BERTRL[2].\n(2) Since the proposed method extract new triples from text corpus, if the unseen entities in test set may seen in the new soft triples. It is better to provide statistics about the  unseen entities in original triples, soft triples and both triples.\n(3) Some notations are confusing, for example the \"t\" is used for a tail entity in a triple (h,r,t), and also used in \"1≤t≤T\". \n\n[1] Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021). KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9, 176–194. \n[2] [2] Zha, H., Chen, Z., & Yan, X. (2022). Inductive Relation Prediction by BERT. Proceedings of the AAAI Conference on Artificial Intelligence, 36(5), 5923-5931.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of knowledge graph completion by proposing a two-stage framework — learning from structural and textual knowledge (LSTK). This novel framework leverages both structural and textual knowledge to learn rule-based systems, which provides a unique approach in the realm of KGC research. The paper is premised on the idea that the typical reliance on structural knowledge alone in rule-based systems is a limiting factor. This leads to the proposition of a system that utilizes \"soft triples\", or triples with confidence scores derived from textual corpora via distant supervision and a textual entailment model with multi-instance learning. In the second stage, a rule-based system for KGC is learned using both these soft triples and the hard triples of existing KG triples. The paper also introduces a novel formalism for learning rules, referred to as text-enhanced rules or TE-rules, which can mitigate the negative impact of noisy data in soft triples.\n\nSTRENGTHS:\n— The two-stage framework introduces an innovative way of leveraging both structural and textual knowledge for KGC, thereby potentially mitigating the limitations of existing methods.\n\n— The introduction of TE-rules provides a robust mechanism for learning rules, to mitigate the negative impact of noises from soft triples.\n\n— The paper introduces three new datasets for empirical evaluation, contributing further to the body of resources available in this field.\n\nWEAKNESSES:\n— The authors utilize a single method to estimate the confidence scores of the soft triples, resulting in a potential lack of generalizability.\n\n— While the introduction of three new datasets is commendable, the paper does not offer a clear comparison of these datasets with established benchmarks. \n\n— Furthermore, it remains uncertain how well the proposed method would perform on existing benchmarks.\n\n— The paper falls short in discussing the scalability of the proposed method. It is unclear how the proposed framework would handle large-scale knowledge graphs, particularly considering the potential computational costs of generating and handling soft triples.\n\nQUESTIONS:\n— Can you elaborate why only use a textual entailment model based on the pre-trained language model BERT to estimate the confidence scores of the soft triples and discuss its generalizability? And what exact BERT model variant do authors use?\n\n— How do the newly introduced datasets compare with established benchmarks? Moreover, how would the proposed LSTK framework perform on these existing benchmarks?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}