SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {e} _ {u, r _ {t}, v} ^ {k} = \bigoplus_ {i = 1} ^ {k} \underbrace {\bigoplus_ {(u , r _ {t} , v)} \bigoplus_ {(v , y _ {0} , x _ {0})} \dots \bigoplus_ {\left(x _ {i - 3} , y _ {i - 2} , u\right)}} _ {i} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \otimes \alpha_ {i 2} \mathbf {r} _ {y _ {0}} \otimes \dots \otimes \alpha_ {i i} \mathbf {r} _ {y _ {i - 2}} \tag {13}
$$
We prove this in Appendix A. Intuitively, we can get this theorem by replacing  $+, \times$  with  $\oplus, \otimes$ . The key step to learn rule-induced subgraph representation is to ensure  $\mathbf{e}_{x,y,z}^{0} \neq 0$  if and only if  $(x,y,z) = (u,r_t,v)$ . Existing models[11, 12] do not satisfy this requirement, as they initialize both the target link and the other links with nonzero embeddings. Therefore, their final subgraph representations contain irrelevant rule terms, which leads to suboptimal results. On the contrary, we show that with appropriate message passing functions, REST learns rule-induced subgraph representation. As the rule-induced subgraph representation encodes all relevant rules within the subgraph, REST is expressive enough to infer the plausibility of any reasonable triple, while it eliminates the negative influence of irrelevant rules.
Our analysis gives some insight of IRP methods. First, eliminating noises within the extracted subgraph is crucial for subgraph-based methods. While existing methods focus on data level to extract ad-hoc subgraphs, our model proposes a simple way for denoising at the model level, i.e., single-source edge-wise message passing. Second, labeling tricks such as single-source initialization can effectively improve the model performance. Last but not least, the idea of learning links is especially important in IRP task, as links play a vital role in reasoning.
In this section, we first introduce the experiment setup including datasets and implementation details. Then we show the main results of REST on several benchmark datasets. Finally, we conduct ablation studies, case studies and further experiments.
6.1 Experiment Setup
Datasets and Implementation Details We conduct experiments on three inductive benchmark datasets proposed by GraIL[11], which are derived from WN18RR[32], FB15K-237[33], and NELL995[34]. For inductive relation prediction, the training set and testing set should have no overlapping entities. Details of the datasets are summarized in Appendix B. We use PyTorch[35] and DGL[36] to implement our REST. Implementation Details of REST are summarized in Appendix C.
We follow GraIL[11] to rank each test triple among 50 other randomly sampled negative triples. We report the Hits@10 metric on the benchmark datasets. Following the standard procedure in prior work [37], we use the filtered setting, which does not take any existing valid triples into account at ranking. We demonstrate the effectiveness of the proposed REST by comparing its performance with both rule-based methods including Neural LP [14], DRUM [21] and RuleN [38] and subgraph-based methods including GraIL [11], CoMPILE [17], TACT[12], SNRI[24] and ConGLR [25]. We run each experiment five times with different random seeds and report the mean results in Table 1.
From the Hits@10 results in Table 1, we make the observation that our model REST significantly outperforms existing methods on 12 versions of 3 datasets. Specifically, our REST can outperform rule-based baselines, including Neural LP, DRUM and RuleN by a large margin. And compared with existing subgraph-based methods, e.g., GraIL, CoMPILE, TACT, SNRI and ConGLR, REST has achieved average improvements of  $17.89\%$ ,  $9.35\%$ ,  $13.76\%$ ;  $16.23\%$ ,  $8.18\%$ ,  $13.04\%$ ;  $13.58\%$ ,  $8.06\%$ ,  $8.96\%$ ;  $10.89\%$ ,  $4.55\%$ , “-” and  $5.58\%$ ,  $5.82\%$ ,  $5.1\%$  on three datasets respectively. As REST
Table 1: Hits@10 results on the inductive benchmark datasets extracted from WN18RR, FB15k-237 and NELL-995. The results of Neural LP, DURM, RuleN, GraIL, CoMPIE and ConGLR are taken from the paper [25].
WN18RR	FB15k-237	NELL-995
v1	v2	v3	v4	v1	v2	v3	v4	v1	v2	v3	v4
Rule-Based	Neural LP	74.37	68.93	46.18	67.13	52.92	58.94	52.90	55.88	40.78	78.73	82.71	80.58
DRUM	74.37	68.93	46.18	67.13	52.92	58.73	52.90	55.88	19.42	78.55	82.71	80.58
RuleN	80.85	78.23	53.39	71.59	49.76	77.82	87.69	85.60	53.50	81.75	77.26	61.35
Subgraph-Based	GraIL	82.45	78.68	58.43	73.41	64.15	81.80	82.83	89.29	59.50	93.25	91.41	73.19
CoMPILE	83.60	79.82	60.69	75.49	67.64	82.98	84.67	87.44	58.38	93.87	92.77	75.19
TACT	84.04	81.63	67.97	76.56	65.76	83.56	85.20	88.69	79.80	88.91	94.02	73.78
SNRI	87.23	83.10	67.31	83.32	71.79	86.50	89.59	89.39	-	-	-	-
ConGLR	85.64	92.93	70.74	92.90	68.29	85.98	88.61	89.31	81.07	94.92	94.36	81.61
REST(ours)	96.28	94.56	79.50	94.19	75.12	91.21	93.06	96.06	88.00	94.96	96.79	92.61
only assigns the embedding of the target link, these improvements demonstrate the effectiveness of our REST via distilling all relevant rules within the subgraph.
We conduct ablation studies to validate the effectiveness of proposed single-source initialization and edge-wise message passing. We show the main results of ablation studies in Table 2.
Table 2: Hits@10 ablation results on the inductive benchmark datasets. The SUM and MUL functions are the ablation for the message function GRU. The MLP function is the ablation for the update function LSTM.  $\Delta$  denotes the performance decrease.
WN18RR	FB15k-237	NELL-995
v1	v2	v3	v4	v1	v2	v3	v4	v1	v2	v3	v4
REST	96.28	94.56	79.50	94.19	75.12	91.21	93.06	96.06	88.00	94.96	96.79	92.61
Full Initialization	92.55	90.70	68.76	79.49	71.71	79.29	89.25	91.22	83.00	86.13	94.54	68.26
Δ	-3.73	-3.86	-10.74	-14.70	-3.41	-11.92	-3.81	-4.84	-5.00	-8.83	-2.25	-24.35
SUM	93.08	85.03	69.59	91.39	64.88	84.30	89.48	89.96	81.00	91.39	96.17	64.57
Δ	-3.20	-9.53	-9.91	-2.80	-10.24	-6.91	-3.58	-6.10	-7.00	-3.57	-0.62	-28.04
MUL	85.64	93.19	56.03	81.04	63.90	78.24	85.20	90.66	69.00	79.20	93.70	36.11
Δ	-10.64	-1.37	-23.47	-13.15	-11.22	-12.97	-7.86	-5.40	-19.00	-15.76	-3.09	-56.50
MLP	95.74	93.65	78.84	90.69	71.07	90.25	92.60	94.94	83.00	94.12	96.41	91.38
Δ	-0.54	-0.91	-0.66	-3.50	-4.05	-0.96	-0.46	-1.12	-5.00	-0.84	-0.38	-1.23
Single-source initialization. Single-source initialization is vital for learning rule-induced subgraph representation. To demonstrate the effectiveness of single-source initialization, we perform another full initialization method as a comparison, which initializes all edges according to their relations. As illustrated in Table 2, we can find that single-source initialization is significant for capturing relevant rules for reasoning. Without single-source initialization, the performance of REST will exhibit a significant decrease, e.g., from 92.61 to 68.26 in NELL-995 v4. This result exhibits the effectiveness of single-source initialization.
Edge-wise message passing. To demonstrate the necessity of proposed RNN-based functions, we conduct ablation studies on various combinations of message functions, including SUM, MUL, and GRU, as well as update functions, including LSTM and MLP. These functions are defined as follows:
$$
\mathbf {S U M}: \mathbf {m} _ {x, y, z} ^ {k} = \mathbf {h} _ {x} ^ {k - 1} + \mathbf {e} _ {x, y, z} ^ {k - 1} + \mathbf {r} _ {y}
$$
$$
\mathbf {M U L}: \mathbf {m} _ {x, y, z} ^ {k} = \mathbf {h} _ {x} ^ {k - 1} \odot \mathbf {e} _ {x, y, z} ^ {k - 1} \odot \mathbf {r} _ {y} \tag {14}
$$
$$
\mathbf {M L P}: \mathbf {e} _ {x, y, z} ^ {k} = \mathbf {W} _ {e} [ \mathbf {e} _ {x, y, z} ^ {k - 1}; \mathbf {q} _ {x, y, z} ^ {k - 1}; \mathbf {h} _ {x} ^ {k} ]
$$
$$
\mathbf {q} _ {x, y, z} ^ {k} = \mathbf {W} _ {q} [ \mathbf {e} _ {x, y, z} ^ {k - 1}; \mathbf {q} _ {x, y, z} ^ {k - 1}; \mathbf {h} _ {x} ^ {k} ]
$$
In general, REST benefits from RNN-based functions, as they can capture the sequential properties of rules. Using order-independent binary operators, such as ADD and MUL, leads to a decline in performance across all datasets, as they cannot differentiate correct and incorrect rules.
6.4 Further Experiments
Case Study One appealing feature of our single-source initialization and edge-wise message passing approach is the ability to interpret the significance of each relevant cycle. This interpretation provides insight into the contribution of each cycle towards the plausibility of the target link  $(u, r_t, v)$ . We generate all relevant rule cycles for each relation with a length of no more than 4, and input them to the REST model to obtain a score for each relevant rule cycle. We normalize these scores using sigmoid function and select the top-3 cycles with the highest scores as visualized in Table 7.
Table 3: Some relations and their top-3 relevant relations. The relations are taken from WN18RR.
Rule Head	Rule Body	Scores
_hypernym	_similar_to->_hypernym-1	0.89
_also_see->_hypernym-1	0.84
_hypernym-1->_also_see->_verb_group	0.67
.derivationally_related_form	DERIVATIONALLY RELATED_form->_also_see	0.82
_has_part->_also_see	0.76
DERIVATIONALLY RELATED_form->_also_see->_has_part	0.66
_member_meronym	_has_part->_synset_domain_topic_of	0.77
has_part->_synset_domain_topic_of->_derivationally_related_form	0.76
_has_part->_derivationally_related_form	0.71
_synset_domain主題_of	_has_part->_also_see	0.95
_synset_domain主題_of-1->_similar_to	0.91
_has_part->_synset_domain主題_of-1	0.81
The results demonstrate that REST is able to learn the degree of correlation between relevant rule cycles and the target relation. As evidence for predicting the rule head, the cycle containing the "_similar_to  $\rightarrow$  _hypernym^{-1}" rule body received the highest score among all cycles that include the "_hypernym" relationship. This indicates that REST can effectively infer a strong correlation between "_silimar_to  $\rightarrow$  _hypernym^{-1}" and "_hypernym". Notably, this cycle is also human-understandable, which highlights the practical interpretability of REST.
Subgraph Extraction Efficiency Different from the subgraph-based methods mentioned above[11, 12, 17], REST eliminates the need for node labeling within the subgraph, which substantially improves time efficiency. We assess the time consumption involved in extracting both enclosing and unenclosing subgraphs for GraIL and REST, and present the running time results in Table 4. All the experiments are conducted on the same CPU with only single process. Our observations indicate a significant improvement of time efficiency over  $11 \times$  when extracting unenclosing subgraphs from the FB15k-237 dataset and over  $6 \times$  across all inductive datasets. This improvement demonstrates the efficiency of our REST on subgraph extraction.
Table 4: The comparison of subgraph extraction time between GraIL and REST. (Unit: Second)
WN18RR	FB15k-237	NELL-995
v1	v2	v3	v4	v1	v2	v3	v4	v1	v2	v3	v4
Enclosing Subgraph	GraIL	121.77	537.42	1127.14	194.98	949.48	2933.04	8423.59	15089.74	136.55	1197.24	6112.77	1303.97
REST	54.01	251.97	617.16	91.76	111.34	338.24	868.79	1,626.77	61.19	213.45	688.14	219.33
Efficiency	2.25×	2.13×	1.83×	2.12×	8.53×	8.67×	9.70×	9.28×	2.23×	5.61×	8.88×	5.95×
Unclosing Subgraph	GraIL	127.69	517.94	1194.18	199.00	1287.35	4166.63	11499.32	21738.29	167.06	1611.97	8044.53	1542.82
REST	56.27	260.20	631.71	95.23	123.36	386.55	985.81	1890.54	64.72	245.41	858.23	248.00
Efficiency	2.27×	1.99×	1.89×	2.09×	10.44×	10.78×	11.66×	11.50×	2.58×	6.57×	9.37×	6.22×
Limitations. Our REST shares the same limitation as subgraph-based methods. While subgraph-based methods are theoretically more expressive, they incur high computational costs both in training and inference. Alleviating this issue is crucial for scalability.
Conclusion. In this paper, we propose a novel single-source edge-wise graph neural network model called REST, which effectively mines relevant rules within subgraphs for inductive reasoning. REST consists of single-source initialization and edge-wise message passing, which is simple, effective and provable to learn rule-induced subgraph representation. Notably, REST accelerates subgraph
extraction by up to  $11.66\times$ , which significantly decreases the time cost of subgraph extraction. Experiments demonstrate that our proposed REST outperforms existing state-of-the-art methods on inductive relation prediction benchmarks.
Future Work. For future work, we target at enhancing the scalability of our REST to conduct reasoning on large-scale knowledge graphs. Moreover, REST can serve as a complementary reasoning model to help large language models conduct reasoning with promising and interpretable results. Hopefully, REST will facilitate the future development of reasoning ability.
[1] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, July 2019.
[2] Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. Knowledge graph embedding based question answering. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 105-113, 2019.
[3] Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. Ripplenet: Propagating user preferences on the knowledge graph for recommender systems. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 417-426, 2018.
[4] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250, 2008.
[5] Denny Vrandecic and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85, 2014.
[6] Farzaneh Mahdisoltani, Joanna Biega, and Fabian Suchanek. Yago3: A knowledge base from multilingual wikipedias. In 7th biennial conference on innovative data systems research. CIDR Conference, 2014.
[7] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.
[8] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer, 2018.
[9] Mikhail Galkin, Max Berrendorf, and Charles Tapley Hoyt. An open challenge for inductive link prediction on knowledge graphs. arXiv preprint arXiv:2203.01520, 2022.
[10] Zijie Geng, Shufang Xie, Yingce Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Tie-Yan Liu. De novo molecular generation via connection-aware motif mining. In The Eleventh International Conference on Learning Representations, 2023.
[11] Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning, pages 9448-9457. PMLR, 2020.
[12] Jiajun Chen, Huarui He, Feng Wu, and Jie Wang. Topology-aware correlations between relations for inductive link prediction in knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6271-6278, 2021.
[13] Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. Fast rule mining in ontological knowledge bases with amie+. The VLDB Journal, 24(6):707-730, 2015.
[14] Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. arXiv preprint arXiv:1702.08367, 2017.
[34] Wenhan Xiong, Thien Hoang, and William Yang Wang. Deeppath: A reinforcement learning method for knowledge graph reasoning. In EMNLP, 2017.
[35] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[36] Minjie Yu Wang. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR workshop on representation learning on graphs and manifolds, 2019.
[37] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[38] Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. Fine-grained evaluation of rule-and embedding-based systems for knowledge graph completion. In *The Semantic Web-ISWC 2018: 17th International Semantic Web Conference*, Monterey, CA, USA, October 8–12, 2018, Proceedings, Part I 17, pages 3–20. Springer, 2018.
[39] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[40] Yuxia Geng, Jiaoyan Chen, Jeff Z. Pan, Mingyang Chen, Song Jiang, Wen Zhang, and Huajun Chen. Relational message passing for fully inductive knowledge graph completion, 2022.
[41] Mikhail Galkin, Etienne Denis, Jiapeng Wu, and William L. Hamilton. Nodepiece: Compositional and parameter-efficient representations of large knowledge graphs, 2022.
A Proof of Theorem 1 and 2
$$
\mathbf {e} _ {u, r _ {t}, v} ^ {k} = \sum_ {i = 1} ^ {k} \underbrace {\sum_ {(u , r _ {t} , v)} \sum_ {(v , y _ {0} , x _ {0})} \dots \sum_ {\left(x _ {i - 3} , y _ {i - 2} , u\right)}} _ {i} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \times \alpha_ {i 2} \mathbf {r} _ {y _ {0}} \times \dots \times \alpha_ {i i} \mathbf {r} _ {y _ {i - 2}} \tag {15}
$$
Proof In this case, the rule-induced subgraph representation is:
$$
\mathcal {S} _ {u, r _ {t}, v} = \sum_ {i = 1} ^ {k} \underbrace {\sum_ {(u , r _ {t} , v)} \sum_ {(v , y _ {0} , x _ {0})} \dots \sum_ {\left(x _ {i - 3} , y _ {i - 2} , u\right)}} _ {i} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \times \alpha_ {i 2} \mathbf {r} _ {y _ {0}} \times \dots \times \alpha_ {i i} \mathbf {r} _ {y _ {i - 2}} \tag {16}
$$
Then we will show that single-source edge-wise GNN can learn this rule-induced sgraph representation in induction.
$k = 1$  .wehave
$$
\mathbf {e} _ {u, r _ {t}, v} ^ {1} = \mathbf {h} _ {u} ^ {1} + \mathbf {r} _ {r _ {t}} = \mathbf {r} _ {r _ {t}} + \sum_ {(x _ {0}, y _ {0}, u) \in \mathcal {T}} \left(\mathbf {h} _ {x _ {0}} ^ {0} + \mathbf {e} _ {x _ {0}, y _ {0}, u} ^ {0}\right) \times \mathbf {r} _ {y _ {0}}
$$
Note that  $\mathbf{e}_{x_0,y_0,u}^0\neq 0$  if and only if  $(x_0,y_0,u) = (u,r_t,v)$ . However, this is impossible as  $u\neq v$ . Thus  $\mathbf{e}_{u,r_t,v}^1$  satisfies the definition of rule-induced subgraph representation.
$k = 2$  .wehave:
$$
\begin{array}{l} \mathbf {e} _ {u, r _ {t}, v} ^ {2} = \mathbf {h} _ {u} ^ {2} + \mathbf {e} _ {u, r _ {t}, v} ^ {1} = \sum_ {(x _ {0}, y _ {0}, u) \in \mathcal {T}} (\mathbf {h} _ {x _ {0}} ^ {1} + \mathbf {e} _ {x _ {0}, y _ {0}, u} ^ {1}) \times \mathbf {r} _ {y _ {0}} + \mathbf {r} _ {r _ {t}} \\ = \sum_ {(x _ {0}, y _ {0}, u) \in \mathcal {T}} 2 \mathbf {h} _ {x _ {0}} ^ {1} \times \mathbf {r} _ {y _ {0}} + \mathbf {r} _ {r _ {t}} \\ = \sum_ {\left(x _ {0}, y _ {0}, u\right)} \sum_ {\left(x _ {1}, y _ {1}, x _ {0}\right)} 2 \left(\mathbf {h} _ {x _ {1}} ^ {0} + \mathbf {e} _ {x _ {1}, y _ {1}, x _ {0}} ^ {0}\right) \times \mathbf {r} _ {y _ {1}} \times \mathbf {r} _ {y _ {0}} + \mathbf {r} _ {r _ {t}} \tag {17} \\ = \sum_ {\left(x _ {0}, y _ {0}, u\right)} \sum_ {\left(x _ {1}, y _ {1}, x _ {0}\right)} 2 \mathbf {e} _ {x _ {1}, y _ {1}, x _ {0}} ^ {0} \times \mathbf {r} _ {y _ {1}} \times \mathbf {r} _ {y _ {0}} + \mathbf {r} _ {r _ {t}} \\ \end{array}
$$
We can find that  $e_{x_1, y_1, x_0}^0 \neq 0$  if and only if  $(x_1, y_1, x_0) = (u, r_t, v)$ , i.e. there exists both  $(u, r_t, v)$  and  $(v, r_t, u)$ . Obviously,  $\mathbf{e}_{u, r_t, v}^2$  satisfies the definition of rule-induced subgraph representation.
Assume that this conclusion exists for  $n \leq k - 1$ . Now we check the  $k$ -th term.
$$
\mathbf {e} _ {u, r _ {t}, v} ^ {k} = \mathbf {h} _ {u} ^ {k} + \sum_ {i = 1} ^ {k - 1} \underbrace {\sum_ {(x _ {0} , y _ {0} , u)} \sum_ {(x _ {1} , y _ {1} , x _ {0})} \dots \sum_ {(v , y _ {i - 2} , x _ {0})}} _ {i - 1} \alpha_ {i 1} \mathbf {r} _ {r _ {t}} \otimes \alpha_ {i 2} \mathbf {r} _ {y _ {i - 2}} \otimes \dots \otimes \alpha_ {i i} \mathbf {r} _ {y _ {0}} \tag {18}
$$
First, we consider  $\mathbf{h}_u^k$
$$
\begin{array}{l} \mathbf {h} _ {u} ^ {k} = \sum_ {\left(x _ {0}, y _ {0}, u\right)} \left(\mathbf {h} _ {x _ {0}} ^ {k - 1} + \mathbf {e} _ {x _ {0}, y _ {0}, u} ^ {k - 1}\right) \times \mathbf {r} _ {y _ {0}} \\ = \sum_ {\left(x _ {0}, y _ {0}, u\right)} \sum_ {\left(x _ {1}, y _ {1}, x _ {0}\right)} \left(\mathbf {h} _ {x _ {1}} ^ {k - 2} + \mathbf {e} _ {x _ {1}, y _ {1}, x _ {0}} ^ {k - 2}\right) \times \mathbf {r} _ {y _ {1}} \times \mathbf {r} _ {y _ {0}} + \sum_ {\left(x _ {0}, y _ {0}, u\right)} \mathbf {e} _ {x _ {0}, y _ {0}, u} ^ {k - 1} \times \mathbf {r} _ {y _ {0}} \\ = \underbrace {\sum_ {\left(x _ {0} , y _ {0} , u\right)} \sum_ {\left(x _ {1} , y _ {1} , x _ {0}\right)} \dots \sum_ {\left(x _ {k - 1} , y _ {k - 1} , x _ {k - 2}\right)}} _ {k} \mathbf {e} _ {x _ {k - 1}, y _ {k - 1}, x _ {k - 2}} ^ {0} \times \mathbf {r} _ {y _ {k - 1}} \times \mathbf {r} _ {y _ {k - 2}} \times \dots \times \mathbf {r} _ {y _ {0}} \tag {19} \\ + \underbrace {\sum_ {(x _ {0} , y _ {0} , u)} \sum_ {(x _ {1} , y _ {1} , x _ {0})} \dots \sum_ {(x _ {k - 2} , y _ {k - 2} , x _ {k - 3})}} _ {k - 1} \mathbf {e} _ {x _ {k - 2}, y _ {k - 2}, x _ {k - 3}} ^ {1} \times \mathbf {r} _ {y _ {k - 2}} \times \dots \times \mathbf {r} _ {y _ {0}} \\ + \dots \\ + \sum_ {(x _ {0}, y _ {0}, u)} \mathbf {e} _ {x _ {0}, y _ {0}, u} ^ {k - 1} \times \mathbf {r} _ {y _ {0}} \\ \end{array}
$$
Notice that  $\underbrace{\sum_{(x_0,y_0,u)}\sum_{(x_1,y_1,x_0)}\ldots\sum_{(x_{k - 1},y_{k - 1},x_{k - 2})}e^{0}_{x_{k - 1},y_{k - 1},x_{k - 2}}\times r_{y_{k - 1}}\times r_{y_{k - 2}}\times\ldots\times r_{y_0}\neq 0\text{if}$
and only if  $(x_{k - 1},y_{k - 1},x_{k - 2}) = (u,r_t,v)$ . In this situation, this term is exactly the  $k$ -th term in the expression of  $\mathbf{e}_{u,r_t,v}^k$ . Now we want to prove that:
$$
\begin{array}{l} \underbrace {\sum_ {\left(x _ {0} , y _ {0} , u\right)} \sum_ {\left(x _ {1} , y _ {1} , x _ {0}\right)} \dots \sum_ {\left(x _ {k - 2} , y _ {k - 2} , x _ {k - 3}\right)}} _ {k - 1} \mathbf {e} _ {x _ {k - 2}, y _ {k - 2}, x _ {k - 3}} ^ {1} \times \mathbf {r} _ {y _ {k - 2}} \times \dots \times \mathbf {r} _ {y _ {0}} \tag {20} \\ + \dots \\ + \sum_ {(x _ {0}, y _ {0}, u)} \mathbf {e} _ {x _ {0}, y _ {0}, u} ^ {k - 1} \times \mathbf {r} _ {y _ {0}} \\ \end{array}
$$
can be fused in top  $k - 1$  term of Equation. 16. Let's check the  $j$ -th term of Equation. 20.