SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \underbrace{\sum_{(x_{0},y_{0},u)}\sum_{(x_{1},y_{1},x_{0})}\ldots\sum_{(x_{j - 1},y_{j - 1},x_{j - 2})}}_{j}\mathbf{e}_{x_{j - 1},y_{j - 1},x_{j - 2}}^{k - j}\times \mathbf{r}_{y_{j - 1}}\times \ldots \times \mathbf{r}_{y_{0}} \\ = \underbrace {\sum_ {(x _ {0} , y _ {0} , u)} \sum_ {(x _ {1} , y _ {1} , x _ {0})} \dots \sum_ {(x _ {j - 1} , y _ {j - 1} , x _ {j - 2})} (\mathbf {e} _ {x _ {j - 1} , y _ {j - 1} , x _ {j - 2}} ^ {k - j - 1} + \mathbf {h} _ {x _ {j - 1}} ^ {k - j}) \times \mathbf {r} _ {y _ {j - 1}} \times \dots \times \mathbf {r} _ {y _ {0}}} _ {j} \\ = \underbrace{\sum_{(x_{0},y_{0},u)}\sum_{(x_{1},y_{1},x_{0})}\ldots\sum_{(x_{j - 1},y_{j - 1},x_{j - 2})}(\mathbf{e}_{x_{j - 1},y_{j - 1},x_{j - 2}}^{0} + \mathbf{h}_{x_{j - 1}}^{k - j} + \ldots + + \mathbf{h}_{x_{j - 1}}^{1})\times \mathbf{r}_{y_{j - 1}}\times \ldots\times \mathbf{r}_{y_{0}}} _ {j} \\ = \underbrace {\sum_ {(x _ {0} , y _ {0} , u)} \sum_ {(x _ {1} , y _ {1} , x _ {0})} \dots \sum_ {(x _ {j - 1} , y _ {j - 1} , x _ {j - 2})}} _ {j} \mathbf {e} _ {x _ {j - 1}, y _ {j - 1}, x _ {j - 2}} ^ {0} \times \mathbf {r} _ {y _ {j - 1}} \times \dots \times \mathbf {r} _ {y _ {0}} \\ + \underbrace {\sum_ {(x _ {0} , y _ {0} , u)} \sum_ {(x _ {1} , y _ {1} , x _ {0})} \dots \sum_ {(x _ {j - 1} , y _ {j - 1} , x _ {j - 2})} \sum_ {(x _ {j} , y _ {j} , x _ {j - 1})}} _ {j + 1} \\ \left(\mathbf {h} _ {x _ {j}} ^ {k - j - 1} + \dots + \mathbf {h} _ {x _ {j}} ^ {0} + \mathbf {e} _ {x _ {j}, y _ {j}, x _ {j - 1}} ^ {k - j - 1} + \dots + \mathbf {e} _ {x _ {j}, y _ {j}, x _ {j - 1}} ^ {0}\right) \times \mathbf {r} _ {y _ {j - 1}} \times \dots \times \mathbf {r} _ {y _ {0}} \tag {21} \\ \end{array}
$$
Note that  $\mathbf{e}_{x_{j-1},y_{j-1},x_{j-2}}^0 \neq 0$  if and only if  $(x_{j-1},y_{j-1},x_{j-2}) = (u,r_t,v)$ , thus the term can be fused into the  $j$ -th term of Equation. 16.  $\mathbf{e}_{x_j,y_j,x_{j-1}}^0$  can be fused into the  $(j+1)$ -th term and so on. Therefore, we have:
$$
\begin{array}{l} \underbrace{\sum_{(x_{0},y_{0},u)}\sum_{(x_{1},y_{1},x_{0})}\ldots\sum_{(x_{j - 1},y_{j - 1},x_{j - 2})}}_{j}\mathbf{e}^{k - j}_{x_{j - 1},y_{j - 1},x_{j - 2}}\times \mathbf{r}_{y_{j - 1}}\times \ldots \times \mathbf{r}_{y_{0}} \\ = \sum_ {i = 1} ^ {k} \underbrace {\sum_ {(u , r _ {t} , v)} \sum_ {(v , y _ {0} , x _ {0})} \dots \sum_ {\left(x _ {i - 3} , y _ {i - 2} , u\right)}} _ {i} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \times \alpha_ {i 2} \mathbf {r} _ {y _ {0}} \times \dots \times \alpha_ {i i} \mathbf {r} _ {y _ {i - 2}} \tag {22} \\ \end{array}
$$
There, we prove that single-source edge-wise GNN can learn rule-induced subgraph representation in this case.
Theorem 2 Single-source edge-wise GNN can learn rule-induced subgraph representation if  $\oplus = \oplus, \oplus = \oplus, \diamond = \oplus, \otimes^1 = \otimes, \otimes^2 = \otimes$ , where  $\oplus$  and  $\otimes$  are binary operators that satisfy  $0 \oplus a = a, 0 \otimes a = 0$ . i.e., there exists nonzero  $\alpha_{i,j}$  such that
$$
\mathbf {e} _ {u, r _ {t}, v} ^ {k} = \bigoplus_ {i = 1} ^ {k} \underbrace {\bigoplus_ {(u , r _ {t} , v)} \bigoplus_ {(v , y _ {0} , x _ {0})} \dots \bigoplus_ {\left(x _ {i - 3} , y _ {i - 2} , u\right)}} _ {i} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \otimes \alpha_ {i 2} \mathbf {r} _ {y _ {0}} \otimes \dots \otimes \alpha_ {i i} \mathbf {r} _ {y _ {i - 2}} \tag {23}
$$
Proof Without loss of generality, we can replace  $+$  with  $\oplus$  and  $\times$  with  $\otimes$  to represent a binary operator, then we directly get this theorem. Note that we should ensure that  $\oplus$  and  $\otimes$  satisfy  $0\oplus a = a,0\otimes a = 0$ , which we use in the process of proof.
B Details of Datasets
We summarize the details of inductive relation prediction benchmark datasets in Table 5.
Table 5: Statistics of three inductive datasets, which contain four different versions individually. We use #E and #R and #TR to denote the number of entities, relations, and triples.
WN18RR	FB15k-237	NELL-995
#R	#E	#TR	#R	#E	#TR	#R	#E	#TR
v1	train	9	2746	6678	183	2000	5226	14	10915	5540
test	9	922	1991	146	1500	2404	14	225	1034
v2	train	10	6954	18968	203	3000	12085	88	2564	10109
test	10	2923	4863	176	2000	5092	79	4937	5521
v3	train	11	12078	32150	218	4000	22394	142	4647	20117
test	11	5084	7470	187	3000	9137	122	4921	9668
v4	train	9	3861	9842	222	5000	33916	77	2092	9289
test	9	7208	15157	204	3500	14554	61	3294	8520
In general, our proposed method is implemented in DGL[36] and PyTorch[35] and trained on single GPU of NVIDIA GeForce RTX 3090. We apply Adam optimizer[39] with an initial learning rate of 0.0005. Observing that batch size has little effect on the performance of the model, We adjust batch size as large as possible for different datasets to accelerate training. We use the binary cross entropy loss. The maximum number of training epochs is set to 10. During training, we add reversed edges to fully capture relevant rules. The number of hop  $h$  is set to 3 which is consistent with existing subgraph-based methods. We conduct grid search to obtain optimal hyperparameters, where we search subgraph types in {enclosing, unclosing}, embedding dimensions in {16, 32}, number of GNN layers in {3, 4, 5, 6} and dropout in {0, 0.1, 0.2}. Configuration for the best performance of each dataset is given within the code.
D Transductive Results
The transductive results, as discussed in Section 6.2, were obtained using the same methodology as the aforementioned evaluations. Specifically, REST was trained on the training graph and tested in a similar manner. We randomly selected  $10\%$  of the links from the training graph as test links. As we can senn in Table 6, REST outperforms GraIL and RuleN significantly across all benchmarks.
Table 6: Experiments of the transductive versions of the current benchmarks.
Model	wn_v1	wn_v2	wn_v3	wn_v4	fb_v1	fb_v2	fb_v3	fb_v4	nell_v1	nell_v2	nell_v3	nell_v4
GraIL	65.59	69.36	64.63	67.28	71.93	86.30	88.95	91.55	64.08	86.88	84.19	82.33
RuleN	63.42	68.09	63.05	65.55	67.53	88.00	91.47	92.35	62.82	82.82	80.72	58.84
REST	92.02	90.90	91.59	91.63	88.01	93.98	96.43	97.71	93.62	95.76	92.79	92.47
E More Results of Case Study
To gain more insight of the proposed RNN-based function and single-source initialization, We conduct more experiments to obtain the rule scores on WN18RR with the ablated model without single-source and other ablated models. In the revised paper, we will include the comprehensive results, allowing readers to compare and analyze the impact of removing "irrelevant" rules on the overall performance of our proposed model. We can see from Table 7, that the model without single-source initialization assigns relatively lower scores to rules, and there are still some less precise rules present in the Top-k rules (denoted in bold). Similarly, other ablated models exhibit less precise rules in the top-k results in Tables 8 and 9. This serves as evidence of the effectiveness of the full-version model.
Table 7: Some relations and their top-3 relevant relations in the full initialization REST. The relations are taken from WN18RR.
Rule Head	Rule Body	Scores
_hypernym	_also Seeing->_synset_domain主題_of->_derivationally_related_form	0.66
Verb_group->_similar_to	0.59
_member_meronym->_similar_to	0.45
DERIVATIONALLY RELATED_form	_also Seeing->_instance_hypernym	0.74
DERIVATIONALLY RELATED_form->_alsoseeing	0.68
_has_part->_has_part	0.62
Member_meronym	_has_part->_hypernym	0.62
_has_part->_instance_hypernym	0.55
_has_part->_member_meronym->_similar_to	0.50
_synset_domain主題_of	_synset_domain主題_of-1->_member_meronym->_similar_to	0.88
_synset_domain主題_of-1->_similar_to	0.84
_member_meronym->_synset_domain主題_of-1	0.81
Table 8: Some relations and their top-3 relevant relations when using SUM as message function. The relations are taken from WN18RR.
Rule Head	Rule Body	Score
_hypernym	_similar_to->_hypernym-1	0.93
_similar_to->_member_meronym	0.90
_hypernym-1->_also_see	0.88
DERIVATIONALLY RELATED_form	_has_part->_similar_to	0.89
DERIVATIONALLY RELATED_form-1->_also_see	0.88
.synset_domain主題_of->_member_meronym->_similar_to	0.88
_member_meronym	_member_meronym->_similar_to	0.90
_has_part->_synset_domain主題_of	0.85
_instance_hypernym->_derivationally RELATED_form->_member_meronym	0.85
_synset_domain主題_of	_all_see->_has_part	0.95
_synset_domain主題_of-1->_also_see	0.95
_hypernym->_derivationally RELATED_form->_also_see	0.92
Table 9: Some relations and their top-3 relevant relations when using MUL as message function. The relations are taken from WN18RR.
Rule Head	Rule Body	Score
_hypernym	_similar_to->_hypernym-1	0.93
_instance_hypernym->_instance_hypernym	0.91
_instance_hypernym->_hypernym	0.91
DERIVATIONALLY_related_form	_similar_to->_derivationally_related_form	0.94
DERIVATIONALLY_related_form->_similar_to	0.93
_has_part->_derivationally_related_form-1	0.93
_member_meronym	_member_meronym->_alsoseeing	0.93
_hypernym->_similar_to	0.90
_similar_to->_synset_domain主題_of->_derivationally_related_form	0.87
_synset_domain主題_of	_also_see->_verb_group	0.92
_synset_domain主題_of-1->_similar_to	0.86
_has_part->_synset_domain主題_of-1	0.84
F Sensitivity Analysis
We further conduct a sensitivity analysis of the subgraph hop  $n$ , where different  $n$  represents the maximum number of neighbors extracted by REST within the subgraphs. We conduct experiments on both the same distribution (transductive setting) and different distribution benchmarks (inductive setting). As Tables 10, 11, 12, and 13 show, REST exhibit great robustness across different  $n$ .
Table 10: Experiments in the transductive setting with  $n = 2,3,4$ , where training and inference are in the same KG.
Hop Number	wn_v1	wn_v2	wn_v3	wn_v4	fb_v1	fb_v2	fb_v3	fb_v4	nell_v1	nell_v2	nell_v3	nell_v4
2hop	84.65	85.40	87.66	85.73	84.35	91.78	95.67	96.97	92.48	96.59	95.73	93.89
3hop	92.02	90.90	91.59	91.63	88.01	93.98	96.43	97.71	93.62	95.76	92.79	92.47
4hop	89.97	89.96	90.19	89.68	88.74	95.12	96.16	97.44	94.99	96.64	96.26	94.35
Table 11: Experiments in the inductive setting on WN18RR benchmark with  $n = 2, 3, 4$ , where training and inference are in the different KGs.
Hop Number	wn_v1_ind	wn_v2_ind	wn_v3_ind	wn_v4_ind
2 hop Neighbors	92.55	89.58	76.27	89.67
3 hop Neighbors	96.28	94.56	79.50	94.19
4 hop Neighbors	94.70	93.17	77.19	92.68
Table 12: Experiments in the inductive setting on FB15k237 with  $n = 2, 3, 4$ , where training and inference are in the different KGs.
Hop Number	fb_v1_ind	fb_v2_ind	fb_v3_ind	fb_v4_ind
2 hop Neighbors	69.53	87.47	90.55	93.68
3 hop Neighbors	75.12	91.21	93.06	96.06
4 hop Neighbors	70.44	90.12	93.45	94.59
G More fine-grained metrics on the inductive benchmarks.
As Hits@10 scores reach a fairly high level, we provide the experimental results on more difficult and comprehensive metrics than Hits@10. Tables 14, 15, and 16 summarize the results of GraIL and our REST on MRR, Hits@1, and Hits@5. These representative metrics suggest that there is still a large room for improvement.
H More Baseline Results and Analysis
We provide additional baselines and our REST for comparison to obtain a more comprehensive explanation. These baselines include NBFNET [27], RMPI [40], and NODEPIECE [41]. The comparative results are presented in the following Table 17. REST also outperforms RMPI in a large margin and achieves competitive results compared with NodePiece and NBFNet. This implies the great potential for the subgraph-based methods to achieve superior results than the whole-graph-based methods. And we will focus on this point as our furture work.
Table 13: Experiments in the inductive setting on NELL995 with  $n = 2, 3, 4$ , where training and inference are in the different KGs.
Hop Number	nell_v1_ind	nell_v2_ind	nell_v3_ind	nell_v4_ind
2 hop Neighbors	83.00	90.18	93.18	90.81
3 hop Neighbors	88.00	94.96	96.79	92.61
4 hop Neighbors	89.00	94.94	95.67	91.54
Table 14: MRR results on inductive benchmarks.
Model	wn_v1	wn_v2	wn_v3	wn_v4	fb_v1	fb_v2	fb_v3	fb_v4	nell_v1	nell_v2	nell_v3	nell_v4
GraIL	74.09	79.32	54.25	73.34	45.91	61.78	62.79	65.09	44.82	64.49	70.73	60.45
REST	81.08	87.06	62.66	88.14	49.71	65.74	68.75	72.41	55.99	69.65	74.99	69.63
Table 15: H@1 results on inductive benchmarks.
Model	wn_v1	wn_v2	wn_v3	wn_v4	fb_v1	fb_v2	fb_v3	fb_v4	nell_v1	nell_v2	nell_v3	nell_v4
GraIL	68.89	71.46	51.24	70.59	37.31	50.94	52.89	54.04	39.00	54.98	58.71	49.27
REST	71.28	82.54	54.88	84.89	39.05	53.14	56.42	60.39	43.00	56.72	61.80	57.59
Table 16: H@5 results on inductive benchmarks.
Model	wn_v1	wn_v2	wn_v3	wn_v4	fb_v1	fb_v2	fb_v3	fb_v4	nell_v1	nell_v2	nell_v3	nell_v4
GraIL	80.78	75.63	55.74	72.37	53.91	75.21	73.70	78.06	47.50	81.41	87.08	59.39
REST	92.55	90.02	69.59	89.99	60.49	80.75	84.28	87.64	57.00	86.97	92.21	86.32
Table 17: More inductive baseline results on the inductive benchmark.
Model	wn_v1	wn_v2	wn_v3	wn_v4	fb_v1	fb_v2	fb_v3	fb_v4	nell_v1	nell_v2	nell_v3	nell_v4
RMPI	89.63	83.22	73.14	81.42	71.71	83.37	86.01	88.69	60.50	94.01	95.36	87.62
NBFNet	94.80	90.50	89.30	89.00	83.40	94.90	95.10	96.00	-	-	-	-
NodePiece	83.00	88.60	78.50	80.70	87.30	93.90	94.40	94.90	89.00	90.10	93.60	89.30
REST(ours)	96.28	94.56	79.50	94.19	75.12	91.21	93.06	96.06	88.00	94.96	96.79	92.61