Title: Learning Rule-Induced Subgraph Representations for Inductive Relation Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: inductive relation prediction

knowledge graph completion

knowledge graph reasoning
Abstract: Inductive relation prediction (IRP)---where entities can be different during training and inference---has shown great power for completing evolving knowledge graphs. Existing works mainly focus on using graph neural networks (GNNs) to learn the representation of the subgraph induced from the target link, which can be seen as an implicit rule-mining process to measure the plausibility of the target link. However, these methods are not able to differentiate the target link and other links during message passing, hence the final subgraph representation will contain irrelevant rule information to the target link, which reduces the reasoning performance and severely hinders the applications for real-world scenarios. To tackle this problem, we propose a novel $\textit{single-source edge-wise}$ GNN model to learn the $\textbf{R}$ule-induc$\textbf{E}$d $\textbf{S}$ubgraph represen$\textbf{T}$ations $(\textbf{REST}$), which encodes relevant rules and eliminates irrelevant rules within the subgraph. Specifically, we propose a $\textit{single-source}$ initialization approach to initialize edge features only for the target link, which guarantees the relevance of mined rules and target link. Then we propose several RNN-based functions for $\textit{edge-wise}$ message passing to model the sequential property of mined rules. REST is a simple and effective approach with theoretical support to learn the $\textit{rule-induced subgraph representation}$. Moreover, REST does not need node labeling, which significantly accelerates the subgraph preprocessing time by up to $\textbf{11.66}\times$. Experiments on inductive relation prediction benchmarks demonstrate the effectiveness of our REST.

Learning Rule-Induced Subgraph Representations for Inductive Relation Prediction

Abstract

Inductive relation prediction (IRP)-where entities can be different during training and inference-has shown great power for completing evolving knowledge graphs. Existing works mainly focus on using graph neural networks (GNNs) to learn the representation of the subgraph induced from the target link, which can be seen as an implicit rule-mining process to measure the plausibility of the target link. However, these methods cannot differentiate the target link and other links during message passing, hence the final subgraph representation will contain irrelevant rule information to the target link, which reduces the reasoning performance and severely hinders the applications for real-world scenarios. To tackle this problem, we propose a novel single-source edge-wise GNNmodel to learn the R ule-induc E d S ubgraph represen T ations ( REST ), which encodes relevant rules and eliminates irrelevant rules within the subgraph. Specifically, we propose a single-source initialization approach to initialize edge features only for the target link, which guarantees the relevance of mined rules and target link. Then we propose several RNN-based functions for edge-wise message passing to model the sequential property of mined rules. REST is a simple and effective approach with theoretical support to learn the rule-induced subgraph representation . Moreover, REST does not need node labeling, which significantly accelerates the subgraph preprocessing time by up to 11.66 × . Experiments on inductive relation prediction benchmarks demonstrate the effectiveness of our REST 2 .

1 Introduction

Knowledge graphs are a collection of factual triples about human knowledge. In recent years, knowledge graphs have been successfully applied in various fileds, such as natural language processing [1], question answering [2] and recommendation systems [3].

However, due to issues such as privacy concerns or data collection costs, many real-world knowledge graphs are far from completion. Moreover, knowledge graphs are continuously evolving with new entities or triples emerging. This dynamic change causes even the large-scale knowledge graphs, e.g., Freebase [4], Wikidata [5] and YAGO3 [6], to still suffer from incompleteness. Most existing knowledge graph completion models, e.g., RotatE [7], R-GCN [8], suffer from handling emerging new entities as they require test entities to be observed in training time. Therefore, inductive relation prediction, which aims at predicting missing links in evolving knowledge graphs, has attracted extensive attention[9, 10].

The key idea of inductive relation prediction on knowledge graphs is to learn logical rules , which can capture co-occurrence patterns between relations in an entity-independent manner and can thus naturally generalize to unseen entities [11, 12]. Some existing models, e.g., AMIE+[13], Neural LP[14], explicitly mine logical rules for inductive relation prediction with good interpretability[14], while their performances are limited due to the large searching space and discrete optimization[15, 16]. Recently, some subgraph-based methods, e.g., GraIL[11], TACT[12], CoMPILE[17], have been proposed to implicitly mine logical rules by reasoning over the subgraph induced from the target link.

target link

Relevant

Rules

However, there are still some irrelevant rules[18] within the subgraph. Considering the rule body and the rule head as a cycle[19], the relevant rules are cycles that pass the target link. As illustrated in Figure 1, u → v → e 4 → u and u → v → e 2 → e 1 → u are relevant rules as they pass the target link, while e 1 → e 3 → e 2 → e 1 are irrelevant rules as they do not contain the target link. Existing methods cannot differentiate the target link and other links during message passing. Thus, they will mine plenty of irrelevant rules and encode them into the final subgraph representation, which makes the model prone to over-fitting and severely hinders reasoning performance.

In this paper, we propose a novel single-source edge-wise GNN model to learn the R ule-induc E d S ubgraph represen T ations ( REST ), which encodes relevant rules and eliminates irrelevant rules within the subgraph. Specifically, we observe that the information flow originating from a unique edge and returning to this edge will form a cycle automatically. Consequently, the information flow originating from the target link can encode the relevant logical rules. Inspired by this observation, we propose a single-source initialization approach to assign a nonzero initial embedding for the target link according to its relation and zero embeddings for other links. Then we propose several RNN-based functions for edge-wise message passing to model the sequential property of mined rules. Finally, we use the representation of the target link as the final subgraph representation.

We theoretically show that with appropriate message passing functions, REST can learn the ruleinduced subgraph representation for reasoning. Notably, REST avoids the heavy burden of node labeling in subgraph preprocessing, which significantly accelerates the time of subgraph preprocessing by up to 11.66 × . Experiments on inductive relation prediction benchmarks demonstrate the effectiveness of our REST.

2 Related Work

Existing works for IRP can be mainly categorized into rule-based methods and subgraph-based methods. While rule-based methods explicitly learn logical rules in knowledge graphs, subgraphbased methods implicitly mine logical rules by learning the representation of subgraphs. Moreover, we discuss some graph neural network methods that reason over the whole graph.

Rule-based methods. Rule-based approaches mine logical rules that are independent of entities and describe co-occurrence patterns of relations to predict missing factual triples. Such a rule consists of a head and a body, where a head is a single atom, i.e., a fact in the form of Relation(head entity, tail entity) , and a body is a set of atoms. Given a head R ( y, x ) and a body { B 1 , B 2 , · · · , B n } , there is a rule R ( y, x ) ← B 1 ∧ B 2 ∧ · · · ∧ B n . The rule-based methods have been studied for a long time in Inductive Logic Programming [20], yet traditional approaches face the challenges of optimization and scalability. Recently, Neural LP [14] presents an end-to-end differentiable framework that enables modern gradient-based optimization techniques to learn the structure and parameters of logical rules. DRUM[21] analyzes Neural LP from the perspective of low-rank tensor approximation and uses bidirectional RNNs to mine more accurate rules. Moreover, for automatically learning rules from large knowledge graphs, RLvLR [22] proposes an effective rule searching and pruning strategy, which shows promising results on both scalability and accuracy for link prediction. However, these explicit rule-based methods lack expressive power due to rule-based nature, and cannot scale to large knowledge graphs as well.

Subgraph-based methods. Subgraph-based methods extract a local subgraph around the target link and use GNNs to learn subgraph representation to predict link existence. Such a subgraph is usually induced by the neighbor nodes of the target link, which encodes the rules related to the target link. GraIL[11] is the first subgraph-based inductive relation prediction model. It defines the enclosing subgraph as the graph induced by all the nodes in the paths between two target nodes. After labeling all the nodes with double radius vertex labeling [23], it employs R-GCNs[8] to learn subgraph representation. CoMPILE[17] extracts directed enclosing subgraphs to handle the asymmetric / anti-symmetric patterns of the target link. TACT[12] converts the original enclosing subgraph into a relational correlation graph, and proposes a relational correlation network to model different correlation patterns between relations. More recently, SNRI[24] extracts enclosing subgraphs with complete neighboring relations to consider neighboring relations for reasoning. ConGLR[25] converts the original enclosing subgraph into a context graph to model relational paths. However, these methods cannot differentiate the target link and other links during message passing. Therefore, the GNNs will mine plenty of irrelevant rules for other links and encode them into subgraph representation, which reduces the accuracy of reasoning.

Graph neural network for link prediction. Some methods use GNNs to reason over the whole graph rather than a subgraph for inductive relation prediction. INDIGO[26] converts the KG into a node-annotated graph and fully encodes it into a GNN. NBFNet[27] generalizes Bellman-Ford algorithm and proposes a general GNN framework to learn path representation for link prediction. MorsE [28] considers transferring entity-independent meta-knowledge by GNNs. While these methods share some spirit with subgraph-based methods, they are essentially different with subgraphbased methods. These methods need to reason over the whole graph for a test example, while subgraph-based methods only need to reason over a subgraph. Meanwhile, these methods tend to predict entities rather than relations for a query, while subgraph-based methods tend to predict relations, as the subgraph only needs to be extracted once for relation prediction. As these methods benefit from a larger number of negative sampling, we do not take them into comparison.

3 Problem Definition

We define a training graph as G tr = ( E tr , R tr , T tr ) , where E tr , R tr , and T tr ⊂ E tr ×R tr ×E tr are the set of entities, relations and triples during training, respectively. We aim to train a model such that for any graph G ′ = ( E ′ , R ′ , T ′ ) whose relations are all seen during training (i.e., R ′ ⊆ R tr ), the model can predict missing triples in G ′ , i.e., (? , r t , v ) , ( u, ? , v ) , ( u, r t , ?) , where u, v ∈ E ′ and r t ∈ R ′ . We denote the set of any possible entities as {E} and the set of knowledge graphs whose relation set R are subset of R tr as {G} tr . The model we would like to train is a score function f : {G} tr ×{E}×R tr ×{E} → R , ( G ′ , u, r t , v ) ↦→ f ( G ′ , u, r t , v ) , where G ′ = ( E ′ , R ′ , T ′ ) , R ′ ⊆ R tr and u, v ∈ E ′ . For a query triple ( u, r t , ?) , we enumerate valid candidate tail entities v ′ and use the model to get the score s ′ of this triple ( u, r t , v ′ ) . We call the query triple ( u, r t , v ) as target link and u, v as target nodes , respectively.

4 Methodology

In this section, we describe the architecture of the proposed REST in detail. Following existing subgraph-based methods, we first extract a subgraph for each query triple. Then we apply singlesource initialization and edge-wise message passing to update edge features iteratively. Finally, the representation of the target link is used for scoring. REST organizes the two methods in a unified framework to perform inductive relation prediction. Figure 2 gives an overview of REST.

4.1 Subgraph Extraction

For a query triple ( u, r t , v ) , the subgraph around it contains the logical rules to infer this query, thus we extract a local subgraph SG u,r t ,v to implicitly learn logical rules for reasoning. Specifically, we first compute the k -hop neighbors N k ( u ) and N k ( v ) of the target nodes u and v . Then we define enclosing subgraph as the graph induced by N k ( u ) ∩ N k ( v ) and unclosing subgraph as the graph induced by N k ( u ) ∪ N k ( v ) . Note that the subgraph extraction process of our REST omits the node labeling, as node features are unnecessary in edge-wise message passing, which significantly reduces the time cost of subgraph preprocessing.

𝑟 3

e

e

e

e

e

𝑒1

𝑒2

𝑒3

𝑢

𝑣

𝑒4

𝒓𝒕

0

0

0

0

0

0

0

e

e

e

e

e

𝑒1

𝑒2

𝑒3

𝑢

𝑣

𝑒4

e

e

e

e

e

𝑒1

𝑒2

𝑒3

𝑢

𝑣

𝑒4

𝒓𝒕

𝑟 1 +𝑟𝑡

0

0

0

0

0

e

e

e

e

e

𝑒1

𝑒2

𝑒3

𝑢

𝑣

𝑒4

𝑟 𝑡

𝑟 1

𝑟 2

𝑟 5

𝑟 4

𝑟 7

𝑟 6

𝑟 6 +𝑟𝑡

……

𝑒𝑢,𝑟 𝑡 ,𝑣

𝑘

Extracted local subgraph for the target link 𝑢, 𝑟 𝑡 , 𝑣 .

Single-source initialization for all edges within the subgraph.

Edge-wise message passing to update edge representations. Finally, we output the representation of the target link 𝑒𝑢,𝑟 𝑡 ,𝑣 𝑘 to calculate its plausibility.

4.2 Single-source Initialization

Single-source initialization is a simple and effective initialization approach, which initializes a nonzero embedding to the query triple according to r t and zero embeddings for other triples. Specifically, the embeddings of links and nodes within SG u,r,v are initialized as follows:

̸

where e 0 x,y,z and h 0 v are the initial representation of edge ( x, y, z ) and node v , respectively. ✶ is the indicator function to differentiate the target link and other links. ⊙ is Hadamard product. Note that the representation of nodes is used as temporary variables in edge-wise message passing. With this initialization approach, we ensure the relevance between mined rules and the target link.

4.3 Edge-wise Message Passing

After initializing all the edges and nodes, we perform edge-wise message passing to encode all relevant rules into the final subgraph representation. Specifically, each iteration of edge-wise message passing consists of three parts, (1) applying message function to every link, (2) updating node features by aggregating message and (3) updating edge features by temporary node features, which are described as follows:

Here, ⊎ , ⊕ , ⋄ , ⊗ 1 , ⊗ 2 are binary operators which denote a function to parameterize. ⊕ denotes the large size operator of ⊕ . h k z and e k x,y,z respectively represent the feature of node z and link ( x, y, z ) after k iterations of edge-wise message passing. We visualize the comparison between conventional message passing framework developed by GraIL[11] and proposed edge-wise message passing framework in Figure 3.

e

ℎ2

e

ℎ3

𝑒1

𝑒3

𝑒2

0

ℎ2

0

ℎ3

0

𝑒 1

0

𝑒 2

0

𝑒 3

0

𝑚1

1

𝑚2

1

𝑚3

1

1

ℎ2

1

ℎ3

1

𝑒 1

0

𝑒 2

0

𝑒 3

0

𝑚1

2

𝑚2

2

𝑚3

2

2

ℎ2

2

ℎ3

2

𝑒 1

0

𝑒 2

0

𝑒 3

0

Initializer

0

ℎ2

0

ℎ3

0

𝑒 1

0

𝑒 2

0

𝑒 3

0

𝑚1

1

𝑚2

1

𝑚3

1

1

ℎ2

1

ℎ3

1

𝑒 1

𝟏

𝑒2

𝟏

𝑒3

𝟏

𝑚1

2

𝑚2

2

𝑚3

2

2

ℎ2

2

ℎ3

2

𝑒 1

𝟐

𝑒2

𝟐

𝑒3

𝟐

Initializer

Message Function

Initialization

Aggregate Function

Update Function

Layer 1

Layer 2

Layer 1

Layer 2

(b) Conventional Message Passing Framework

(c) Edge-wise Message Passing Framework

(a) Original Graph

Readout Function

4.4 RNN-based Functions

Message passing functions in existing works use order-independent binary operators such as ADD and MUL, which cannot model the sequential property of rules and lead to incorrect rules[21]. To tackle this problem, we introduce several RNN-based methods as message passing functions.

Message Functions. For the edge-wise message passing process, each iteration REST takes in h k -1 x , e k -1 x,y,z and r y to form a message. We modify GRU[29] as message function as follows:

Here, δ k is the update gate vector, γ k is the reset gate vector and c k is the candidate activation vector. The operator ⊙ denotes the Hadamard product, σ g denotes Sigmoid activation function and σ h denotes Tanh activation function. During each iteration of message passing, we only use GRU once, therefore k -layer message passing includes k GRUs, which can model sequence with length l ≤ k .

Aggregate Functions. The aggregate function aggregates messages for each node from its neighboring edges. Here, we use simplified PNA[30] to consider different types of aggregation.

Here, [; ] denotes the concatenation of vectors, W k agg denotes the linear transformation matrix in the k -th layer.

Update Functions. The update function is used to update the edge feature. We propose to update the edge feature with LSTM[31]. Specifically, LSTM needs three inputs: a hidden vector, a current

input vector and a cell vector. We use h k x as the hidden vector and e k -1 x,y,z as the current input vector. Moreover, we expect that each edge can differentiate the target link during message passing, which requires each edge to specify the query information. Therefore, we initialize each edge with another query feature as the cell vector. All the edges are initialized with the same query embedding r q r related to the query relation r .

Then the update function can be described as follows:

After updating the edge feature by k iterations of edge-wise message passing, we output e k u,r,v as the subgraph representation. Then we use a linear transformation and an activation function to get the score of the target link.

5 Analysis

In this section, we theoretically analyze the effectiveness of our REST. We first define the rule-induced subgraph representation, which utilizes encoded relevant rules to infer the plausibility of the target link. Then we show that our REST is able to learn such a rule-induced subgraph representation for reasoning.

5.1 Rule-induced Subgraph Representation Formulation

Our rule-induced subgraph representation aims to encode all relevant rules into the subgraph representation for reasoning. Therefore, we can define the rule-induced subgraph representation as the aggregation of these relevant rules:

where C denotes the set of all possible relevant rules within SG u,r t ,v and p c is the representation of a relevant rule c . Following the idea of Neural LP[14] to associate each relation in the rule with a weight, we model the representation of a rule as a function of its relation set. Therefore, we give the definition of rule-induced subgraph representation.

Definition 1 (Rule-induced subgraph representation.) Given a subgraph SG u,r t ,v , its ruleinduced subgraph representation is defined as follows:

where i denotes the length of the cycle, r y i is the representation of relation y i , ( x i , y i , x i -1 ) is an existing triple in SG u,r t ,v . { ( u, r t , v ) , ( v, y 0 , x 0 ) , ..., ( x i -3 , y i -2 , u ) } is a cycle at length i .

Note that ⊕ and ⊗ denote binary aggregation functions. Intuitively, rule-induced subgraph representation captures all relevant rules within the subgraph and is expressive enough for reasoning.

5.2 Rule-induced Subgraph Representation Learning

Here, we show that our REST can learn such a rule-induced subgraph representation. First, we show this in a simple case.

Theorem 1 Single-source edge-wise GNN can learn rule-induced subgraph representation if ⊎ = + , ⊕ = + , ⋄ = + , ⊗ 1 = × , ⊗ 2 = × . i.e., there exists nonzero α i,j such that

We prove this in Appendix A. This theorem states that our REST can learn the rule-induced subgraph representation in the basic condition. Then we generalize this theorem to a general version.

Theorem 2 Single-source edge-wise GNN can learn rule-induced subgraph representation if ⊎ = ⊕ , ⊕ = ⊕ , ⋄ = ⊕ , ⊗ 1 = ⊗ , ⊗ 2 = ⊗ , where ⊕ and ⊗ are binary operators that satisfy 0 ⊕ a = a, 0 ⊗ a = 0 . i.e., there exists nonzero α i,j such that

̸

We prove this in Appendix A. Intuitively, we can get this theorem by replacing + , × with ⊕ , ⊗ . The key step to learn rule-induced subgraph representation is to ensure e 0 x,y,z = 0 if and only if ( x, y, z ) = ( u, r t , v ) . Existing models[11, 12] do not satisfy this requirement, as they initialize both the target link and the other links with nonzero embeddings. Therefore, their final subgraph representations contain irrelevant rule terms, which leads to suboptimal results. On the contrary, we show that with appropriate message passing functions, REST learns rule-induced subgraph representation. As the rule-induced subgraph representation encodes all relevant rules within the subgraph, REST is expressive enough to infer the plausibility of any reasonable triple, while it eliminates the negative influence of irrelevant rules.

Our analysis gives some insight of IRP methods. First, eliminating noises within the extracted subgraph is crucial for subgraph-based methods. While existing methods focus on data level to extract ad-hoc subgraphs, our model proposes a simple way for denoising at the model level, i.e., single-source edge-wise message passing. Second, labeling tricks such as single-source initialization can effectively improve the model performance. Last but not least, the idea of learning links is especially important in IRP task, as links play a vital role in reasoning.

6 Experiments

In this section, we first introduce the experiment setup including datasets and implementation details. Then we show the main results of REST on several benchmark datasets. Finally, we conduct ablation studies, case studies and further experiments.

6.1 Experiment Setup

Datasets and Implementation Details We conduct experiments on three inductive benchmark datasets proposed by GraIL[11], which are dervied from WN18RR[32], FB15K-237[33], and NELL995[34]. For inductive relation prediction, the training set and testing set should have no overlapping entities. Details of the datasets are summarized in Appendix B. We use PyTorch[35] and DGL[36] to implement our REST. Implementation Details of REST are summarized in Appendix C.

6.2 Main Results

We follow GraIL[11] to rank each test triple among 50 other randomly sampled negative triples. We report the Hits@10 metric on the benchmark datasets. Following the standard procedure in prior work [37], we use the filtered setting, which does not take any existing valid triples into account at ranking. We demonstrate the effectiveness of the proposed REST by comparing its performance with both rule-based methods including Neural LP [14], DRUM [21] and RuleN [38] and subgraph-based methods including GraIL [11], CoMPILE [17], TACT[12], SNRI[24] and ConGLR [25]. We run each experiment five times with different random seeds and report the mean results in Table 1.

From the Hits@10 results in Table 1, we make the observation that our model REST significantly outperforms existing methods on 12 versions of 3 datasets. Specifically, our REST can outperform rule-based baselines, including Neural LP, DRUM and RuleN by a large margin. And compared with existing subgraph-based methods, e.g., GraIL, CoMPILE, TACT, SNRI and ConGLR, REST has achieved average improvements of 17.89%, 9.35%, 13.76%; 16.23%, 8.18%, 13.04%; 13.58%, 8.06%, 8.96%; 10.89%, 4.55%,'-' and 5.58%, 5.82%, 5.1% on three datasets respectively. As REST

only assigns the embedding of the target link, these improvements demonstrate the effectiveness of our REST via distilling all relevant rules within the subgraph.

6.3 Ablation Study

We conduct ablation studies to validate the effectiveness of proposed single-source initialization and edge-wise message passing. We show the main results of ablation studies in Table 2.

Single-source initialization. Single-source initialization is vital for learning rule-induced subgraph representation. To demonstrate the effectiveness of single-source initialization, we perform another full initialization method as a comparison, which initializes all edges according to their relations. As illustrated in Table 2, we can find that single-source initialization is significant for capturing relevant rules for reasoning. Without single-source initialization, the performance of REST will exhibit a significant decrease, e.g., from 92.61 to 68.26 in NELL-995 v4. This result exhibits the effectiveness of single-source initialization.

Edge-wise message passing. To demonstrate the necessity of proposed RNN-based functions, we conduct ablation studies on various combinations of message functions, including SUM, MUL, and GRU, as well as update functions, including LSTM and MLP. These functions are defined as follows:

In general, REST benefits from RNN-based functions, as they can capture the sequential properties of rules. Using order-independent binary operators, such as ADD and MUL, leads to a decline in performance across all datasets, as they cannot differentiate correct and incorrect rules.

6.4 Further Experiments

Case Study One appealing feature of our single-source initialization and edge-wise message passing approach is the ability to interpret the significance of each relevant cycle. This interpretation provides insight into the contribution of each cycle towards the plausibility of the target link ( u, r t , v ) .We generate all relevant rule cycles for each relation with a length of no more than 4, and input them to the REST model to obtain a score for each relevant rule cycle.We normalize these scores using sigmoid function and select the top-3 cycles with the highest scores as visualized in Table 7.

The results demonstrate that REST is able to learn the degree of correlation between relevant rule cycles and the target relation. As evidence for predicting the rule head, the cycle containing the "_ similar _ to → _ hypernym -1 " rule body received the highest score among all cycles that include the "_ hypernym " relationship. This indicates that REST can effectively infer a strong correlation between "_ silimar _ to → _ hypernym -1 " and "_ hypernym ". Notably, this cycle is also humanunderstandable, which highlights the practical interpretability of REST.

Subgraph Extraction Efficiency Different from the subgraph-based methods mentioned above[11, 12, 17], REST eliminates the need for node labeling within the subgraph, which substantially improves time efficiency. We assess the time consumption involved in extracting both enclosing and unenclosing subgraphs for GraIL and REST, and present the running time results in Table 4. All the experiments are conducted on the same CPU with only single process. Our observations indicate a significant improvement of time efficiency over 11 × when extracting unenclosing subgraphs from the FB15k-237 dataset and over 6 × across all inductive datasets. This improvement demonstrates the efficiency of our REST on subgraph extraction.

7 Conclusion and Future Work

Limitations. Our REST shares the same limitation as subgraph-based methods. While subgraphbased methods are theoretically more expressive, they incur high computational costs both in training and inference. Alleviating this issue is crucial for scalability.

Conclusion. In this paper, we propose a novel single-source edge-wise graph neural network model called REST, which effectively mines relevant rules within subgraphs for inductive reasoning. REST consists of single-source initialization and edge-wise message passing, which is simple, effective and provable to learn rule-induced subgraph representation. Notably, REST accelerates subgraph

extraction by up to 11.66 × , which significantly decreases the time cost of subgraph extraction. Experiments demonstrate that our proposed REST outperforms existing state-of-the-art methods on inductive relation prediction benchmarks.

Future Work. For future work, we target at enhancing the scalability of our REST to conduct reasoning on large-scale knowledge graphs. Moreover, REST can serve as a complementary reasoning model to help large language models conduct reasoning with promising and interpretable results. Hopefully, REST will facilitate the future development of reasoning ability.

References

Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , July 2019.

Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. Knowledge graph embedding based question answering. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining , pages 105-113, 2019.

Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. Ripplenet: Propagating user preferences on the knowledge graph for recommender systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management , pages 417-426, 2018.

Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data , pages 1247-1250, 2008.

Denny Vrandeˇ ci´ c and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM , 57(10):78-85, 2014.

Farzaneh Mahdisoltani, Joanna Biega, and Fabian Suchanek. Yago3: A knowledge base from multilingual wikipedias. In 7th biennial conference on innovative data systems research . CIDR Conference, 2014.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197 , 2019.

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference , pages 593-607. Springer, 2018.

Mikhail Galkin, Max Berrendorf, and Charles Tapley Hoyt. An open challenge for inductive link prediction on knowledge graphs. arXiv preprint arXiv:2203.01520 , 2022.

Zijie Geng, Shufang Xie, Yingce Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Tie-Yan Liu. De novo molecular generation via connection-aware motif mining. In The Eleventh International Conference on Learning Representations , 2023.

Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning , pages 9448-9457. PMLR, 2020.

Jiajun Chen, Huarui He, Feng Wu, and Jie Wang. Topology-aware correlations between relations for inductive link prediction in knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 6271-6278, 2021.

Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. Fast rule mining in ontological knowledge bases with amie+. The VLDB Journal , 24(6):707-730, 2015.

Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. arXiv preprint arXiv:1702.08367 , 2017.

Stanley Kok and Pedro Domingos. Statistical predicate invention. In Proceedings of the 24th international conference on Machine learning , pages 433-440, 2007.

William Yang Wang, Kathryn Mazaitis, and William W Cohen. Structure learning via parameter learning. In Proceedings of the 23rd ACM international conference on conference on information and knowledge management , pages 1199-1208, 2014.

Sijie Mai, Shuangjia Zheng, Yuedong Yang, and Haifeng Hu. Communicative message passing for inductive relation reasoning. In AAAI , pages 4294-4302, 2021.

K Rameshkumar, M Sambath, and S Ravi. Relevant association rule mining from medical dataset using new irrelevant rule elimination technique. In 2013 International Conference on Information Communication and Embedded Systems (ICICES) , pages 300-304. IEEE, 2013.

Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, and Chao Chen. Cycle representation learning for inductive relation prediction. In International Conference on Machine Learning , pages 24895-24910. PMLR, 2022.

Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming , 19:629-679, 1994.

Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. Advances in Neural Information Processing Systems , 32, 2019.

Pouya Ghiasnezhad Omran, Kewen Wang, and Zhe Wang. An embedding-based approach to rule learning in knowledge graphs. IEEE Transactions on Knowledge and Data Engineering , 2019.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems , 31, 2018.

Xiaohan Xu, Peng Zhang, Yongquan He, Chengpeng Chao, and Chaoyang Yan. Subgraph neighboring relations infomax for inductive link prediction on knowledge graphs. arXiv preprint arXiv:2208.00850 , 2022.

Qika Lin, Jun Liu, Fangzhi Xu, Yudai Pan, Yifan Zhu, Lingling Zhang, and Tianzhe Zhao. Incorporating context graph with logical reasoning for inductive relation prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 893-903, 2022.

Shuwen Liu, Bernardo Grau, Ian Horrocks, and Egor Kostylev. Indigo: Gnn-based inductive knowledge graph completion using pair-wise encoding. Advances in Neural Information Processing Systems , 34:2034-2045, 2021.

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. Advances in Neural Information Processing Systems , 34:29476-29490, 2021.

Mingyang Chen, Wen Zhang, Yushan Zhu, Hongting Zhou, Zonggang Yuan, Changliang Xu, and Huajun Chen. Meta-knowledge transfer for inductive knowledge graph embedding. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 927-937, 2022.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078 , 2014.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems , 33:13260-13271, 2020.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In The Workshop on CVSC , 2015.

Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In AAAI , 2018.

Wenhan Xiong, Thien Hoang, and William Yang Wang. Deeppath: A reinforcement learning method for knowledge graph reasoning. In EMNLP , 2017.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Minjie Yu Wang. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR workshop on representation learning on graphs and manifolds , 2019.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems , 26, 2013.

Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. Fine-grained evaluation of rule-and embedding-based systems for knowledge graph completion. In The Semantic Web-ISWC 2018: 17th International Semantic Web Conference, Monterey, CA, USA, October 8-12, 2018, Proceedings, Part I 17 , pages 3-20. Springer, 2018.

Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Yuxia Geng, Jiaoyan Chen, Jeff Z. Pan, Mingyang Chen, Song Jiang, Wen Zhang, and Huajun Chen. Relational message passing for fully inductive knowledge graph completion, 2022.

Mikhail Galkin, Etienne Denis, Jiapeng Wu, and William L. Hamilton. Nodepiece: Compositional and parameter-efficient representations of large knowledge graphs, 2022.

A Proof of Theorem 1 and 2

Theorem 1 Single-source edge-wise GNN can learn rule-induced subgraph representation if ⊎ = + , ⊕ = + , ⋄ = + , ⊗ 1 = × , ⊗ 2 = × . i.e., there exists nonzero α i,j such that

Proof In this case, the rule-induced subgraph representation is:

Then we will show that single-source edge-wise GNN can learn this rule-induced sugraph representation in induction .

k = 1 . we have

̸

̸

Note that e 0 x 0 ,y 0 ,u = 0 if and only if ( x 0 , y 0 , u ) = ( u, r t , v ) . However, this is impossible as u = v . Thus e 1 u,r t ,v satisfies the definition of rule-induced subgraph representation.

k = 2 . we have:

̸

We can find that e 0 x 1 ,y 1 ,x 0 = 0 if and only if ( x 1 , y 1 , x 0 ) = ( u, r t , v ) , i.e. there exists both ( u, r t , v ) and ( v, r t , u ) . Obviously, e 2 u,r t ,v satisfies the definition of rule-induced subgraph representation.

Assume that this conclusion exists for n ≤ k -1 . Now we check the k -th term.

First, we consider h k u .

̸

and only if ( x k -1 , y k -1 , x k -2 ) = ( u, r t , v ) . In this situation, this term is exactly the k -th term in the expression of e k u,r t ,v . Now we want to prove that:

can be fused in top k -1 term of Equation. 16. Let's check the j -th term of Equation. 20.

̸

Note that e 0 x j -1 ,y j -1 ,x j -2 = 0 if and only if ( x j -1 , y j -1 , x j -2 ) = ( u, r t , v ) , thus the term can be fused into the j -th term of Equation. 16. e 0 x j ,y j ,x j -1 can be fused into the ( j +1) -th term and so on. Therefore, we have:

There, we prove that single-source edge-wise GNN can learn rule-induced subgraph representation in this case. □

Theorem 2 Single-source edge-wise GNN can learn rule-induced subgraph representation if ⊎ = ⊕ , ⊕ = ⊕ , ⋄ = ⊕ , ⊗ 1 = ⊗ , ⊗ 2 = ⊗ , where ⊕ and ⊗ are binary operators that satisfy 0 ⊕ a = a, 0 ⊗ a = 0 . i.e., there exists nonzero α i,j such that

Proof Without loss of generality, we can replace + with ⊕ and × with ⊗ to represent a binary operator, then we directly get this theorem. Note that we should ensure that ⊕ and ⊗ satisfy 0 ⊕ a = a, 0 ⊗ a = 0 , which we use in the process of proof. □

B Details of Datasets

We summarize the details of inductive relation prediction benchmark datasets in Table 5.

C Implementation Details

In general, our proposed method is implemented in DGL[36] and PyTorch[35] and trained on single GPU of NVIDIA GeForce RTX 3090. We apply Adam optimizer[39] with an initial learning rate of 0 . 0005 . Observing that batch size has little effect on the performance of the model, We adjust batch size as large as possible for different datasets to accelerate training. We use the binary cross entropy loss.The maximum number of training epochs is set to 10. During training, we add reversed edges to fully capture relevant rules. The number of hop h is set to 3 which is consistent with existing subgraph-based methods. We conduct grid search to obtain optimal hyperparameters, where we search subgraph types in {enclosing, unclosing}, embedding dimensions in {16, 32}, number of GNN layers in {3, 4, 5, 6} and dropout in {0, 0.1, 0.2}. Configuration for the best performance of each dataset is given within the code.

D Transductive Results

The transductive results, as discussed in Section 6.2, were obtained using the same methodology as the aforementioned evaluations. Specifically, REST was trained on the training graph and tested in a similar manner. We randomly selected 10% of the links from the training graph as test links. As we can senn in Table 6, REST outperforms GraIL and RuleN significantly across all benchmarks.

E More Results of Case Study

To gain more insight of the proposed RNN-based function and single-source initialization, We conduct more experiments to obtain the rule scores on WN18RR with the ablated model without single-source and other ablated models. In the revised paper, we will include the comprehensive results, allowing readers to compare and analyze the impact of removing "irrelevant" rules on the overall performance of our proposed model. We can see from Table 7, that the model without single-source initialization assigns relatively lower scores to rules, and there are still some less precise rules present in the Top-k rules (denoted in bold). Similarly, other ablated models exhibit less precise rules in the top-k results in Tables 8 and 9. This serves as evidence of the effectiveness of the full-version model.

F Sensitivity Analysis

We further conduct a sensitivity analysis of the subgraph hop n , where different n represents the maximum number of neighbors extracted by REST within the subgraphs. We conduct experiments on both the same distribution (transductive setting) and different distribution benchmarks (inductive setting). As Tables 10, 11, 12, and 13 show, REST exhibit great robustness across different n .

G More fine-grained metrics on the inductive benchmarks.

As Hits@10 scores reach a fairly high level, we provide the experimental results on more difficult and comprehensive metrics than Hits@10. Tables 14, 15, and 16 summarize the results of GraIL and our REST on MRR, Hits@1, and Hits@5. These representative metrics suggest that there is still a large room for improvement.

H More Baseline Results and Analysis

We provide additional baselines and our REST for comparison to obtain a more comprehensive explanation. These baselines include NBFNET [27], RMPI [40], and NODEPIECE [41]. The comparative results are presented in the following Table 17. REST also outperforms RMPI in a large margin and achieves competitive results compared with NodePiece and NBFNet. This implies the great potential for the subgraph-based methods to achieve superior results than the whole-graph-based methods. And we will focus on this point as our furture work.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes REST, a method for inductive link prediction over knowledge graphs. Compared to previous techniques that extract subgraphs around the target edge, REST avoids node labelling (thus saving substantial execution time) and only initializes the representation of the target link, setting other edge representations to zero vectors. Moreover, REST introduces edge-wise message passing to update edge representations iteratively, and achieves this through a series of learnable functions and a gated recurrent unit (GRU). Node updates are then computed using a long-short term memory (LSTM) cell, conditioned on the target query. \n\nBy only initializing the target edge representation, the paper shows that REST can focus only on \"relevant\" rules, i.e., rules including the target edge, thus substantially reducing noise. More precisely, REST is shown to learn rule-induced subgraph representations. Finally, on the empirical side, the paper runs experiments on standard baselines for inductive link prediction, achieving strong results, reports ablation studies highlighting the value of its initialization and recurrent components, and discusses further experiments showcasing candidate rule scoring by the learned REST model on WN18RR and efficiency gains versus other baselines.\n\nSTRENGTHS:\n- The paper is well-written and easy to follow.\n- The design choices in REST (single-source initialization, message passing setup, etc.) are well-justified  and supported empirically. \n- The paper provides a good overview of the literature in the related work section. \n- The experimental results on inductive datasets are strong and convincing. The ablation studies also demonstrate the value of the REST model. \n- The additional case study is very interesting.\n\nWEAKNESSES:\n- The paper does not report all baseline results in the experimental section (Table 1). For instance, NBFNet is cited in the related work section but not reported in the tables. From a quick glance at that paper, its results would indicate NBFNet has mostly superior performance on FB15k-237. Therefore, it would be important to include these results for completeness and ideally also discuss potential explanations for performance gains/losses.\n\n- I think the experimental section can be improved by including further benchmarks from other link prediction setups. For instance, it would be interesting to see REST's performance on the transductive versions of the current benchmarks, and on OGB [1] benchmarks for link prediction. \n\n[1] Hu et al. Open Graph Benchmark: Datasets for Machine Learning on Graphs, NeurIPS 2020.\n\nQUESTIONS:\n- Could you provide rule scores on WN18RR (similarly to the scores you currently have in the experimental section) with the ablated model without single-source? I think contrasting the scores and/or the different top-k rules across the full version and the ablated version would really help drive the point across regarding ignoring \"irrelevant\" rules.\n\nLIMITATIONS:\nThe authors adequately discuss model limitations, particularly in terms of relatively poor scalability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a subgraph-based method to address the inductive relation prediction task, where the model learns to predict links by learning implicit rules from subgraphs. Models proposed in previous works lack the ability to differentiable the relevant and irrelevant paths. To address this, the authors propose several modifications including single-source initialization, edge-wise message passing, and modified activation functions. In the experiments, these modifications are shown to be effective which leads to superior performance compared to several subgraph-based and rule-based methods.\n\nSTRENGTHS:\n- The proposed single-source initialization and edge-wise message passing are well-motivated and well-presented.\n- The paper is well-written and easy to read.\n\nWEAKNESSES:\n## Novelty\n\n\nThe authors propose several modifications to the standard message passing algorithm to better support inductive rule learning. This includes single-source initialization, edge-wise message passing, and modified activation functions. While these modifications are well-motivated and sensible, they are straightforward and mostly incremental changes to the standard protocol. That said the intellectual novelty is limited.\n\nSection 5 provides an interesting theoretical analysis of the proposed model. While it is always nice to have a solid analysis, it does not contribute to any theoretical findings nor does it reveal novel properties of the proposed method other than proving it will work, which is already empirically proven in many previous works.\n\nIn summary, this paper proposes several incremental modifications to GNN-based inductive reasoning methods. While the modifications are rather straightforward, it does seem to lead to better empirical results.\n\n\n## Quality\n\n\nThe proposed single-source initialization and edge-wise message passing are well-motivated and well-presented.\n\nMy main concern is more about the subgraph-based methods as a whole. This class of methods requires first extracting the subgraphs from the KG before they can be learned by the GNN models. And what concerns me the most is that the extraction step is not learnable: one has to pick a magic n-hop number beforehand and proceed to extract the subgraphs and finally train the models on them. This leads to several limitations: (1) it cannot generalize to KGs that have longer relational path than those seen in the training set; (2) the n-hop number are typically small, as the complexity of running GNN grows quickly with respect to the size of the graph. These limitations are non-existent for backward-chaining rule-based methods.\n\nI understand this criticism is probably beyond the scope of this work, but I highly recommend the authors step outside of the \"well-established\" zone and challenge themselves to address these limitations. For example, as a first step, one can investigate what is the effect of the number n on the performance of the model on datasets with the same data distribution; and then on ones with different distributions, and if the performance deteriorates, propose solutions to alleviate that.\n\n\n## Clarity\n\nThe paper is well-written and easy to read.\n\n\n## Significance\n\nThe proposed method has some empirical significance as it is shown to outperform all previous baselines. However, as mentioned above, the model largely resembles the ones in prior work and only several incremental changes are proposed. More importantly, this class of methods has a serious limitation that is still left unaddressed in this work, which limits its applications in real-world tasks.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles a problem in the sub-area of machine learning on graphs. It looks at approaches for inductive reasoning (i.e. ones the generalize to unseen vertices), identifies a problem with those approaches and suggests a novel manner to tackle is problem, exploiting the recurrant nature of message-passing (via a GRU and LSTM).\n\nThere where theorems, along with proofs, that analized the potential effectiveness of the model. Moreover, the model was trained on multiple data sets and compared to multiple strong baselines. Finally, ablation studies where performed to empirically show the effect of the GRU and LSTM components.\n\nThe paper is overall clear to read.  Section 3 however could be easier to read through if the mathematical expressions where better separated from the text.\n\nThe field of inductive reasoning over graphs itself tackles a very relevant problem for machine learning on graphs. This paper identifies a problem with current approaches and introduces a fairly novel, effective and scalable approach taking this problem into account. While this is likely not be interesting to everyone in the world of machine learning, it should be interesting to many people in the machine learning on graphs community.\n\nSTRENGTHS:\nThis paper is sound. To applies to its analysis of the problem at hand, but also to its experimental setup, which incluedes good baselines and ablation studies.\n\nWEAKNESSES:\nThe mathematical nature of the paper, combined with a reliance on an understanding of several other things (e.g. subgraph representation, LSTMs and GRUs) could make this a difficult paper to get into for people from other sub-areas.\n\nQUESTIONS:\nWith some scores being higher than 95%, do you think research in this direction should look into a more difficult task instead?\n\nLIMITATIONS:\nThe reported hits@10 are fairly high (typically 90%-95%). It would have been nice if hits at a lower k would have been reported as well.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This work proposes a single-source edge-wise GNN model for the inductive relation prediction task, called REST. The method includes a single-source initialization approach that reduce the irrelevant rules during learning and an RNN-based edge-wise message passing mechanism to model the sequential property of mined rules. Authors experiments REST on three inductive relation prediction benchmarks and REST achieves  good results. And without node labeling, the subgraph extraction is significantly faster than GraIL.

STRENGTHS:
The paper is well-written and easy to follow. The proposed method is well-motivated and is easy to understand. The experiments proves the effectiveness of the REST, especially the ablation study proves the effectiveness of the main novelty in the method, the single-source initialization and RNN-based edge-wise massage passing.

WEAKNESSES:
Some of the recent proposed inductive relation prediction methods are missing, such as 
* Mingyang Chen, Wen Zhang, Yushan Zhu, Hongting Zhou, Zonggang Yuan, Changliang Xu, Huajun Chen: Meta-Knowledge Transfer for Inductive Knowledge Graph Embedding. SIGIR 2022: 927-937

QUESTIONS:
1. During experiment, is the binary operators in Equation (2) set the same as Theorem 1? 
2. As shown in Table 2, replacing the GRU massage function to SUM and MUL significantly downgrade the performance which is more significant than change the single initialization to full initialization. Is there any reason?

LIMITATIONS:
See weakness and limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07