SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Serge Abiteboul, Richard Hull, and Victor Vianu. Foundations of Databases. Addison-Wesley, 1995.
[2] Christoph Alt, Marc Hübner, and Leonhard Hennig. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction. In ACL, pages 1388-1398, 2019.
[3] Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787-2795, 2013.
[4] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, pages 632-642, 2015.
[5] Rui Cai, Xiaodong Zhang, and Houfeng Wang. Bidirectional recurrent convolutional neural network for relation classification. In ACL, 2016.
[6] Jiajun Chen, Huarui He, Feng Wu, and Jie Wang. Topology-aware correlations between relations for inductive link prediction in knowledge graphs. In AAAI, pages 6271-6278, 2021.
[7] Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. Hacred: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of ACL, pages 2819-2831, 2021.
[8] William W. Cohen, Fan Yang, and Kathryn Mazaitis. Tensorlog: A probabilistic database implemented using deep-learning infrastructure. J. Artif. Intell. Res., 67:285-325, 2020.
[9] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, volume 3944, pages 177–190, 2005.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186, 2019.
[11] Thomas G. Dietterich, Richard H. Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell., 89(1-2):31-71, 1997.
[12] Jianfeng Du, Jeff Z. Pan, Sylvia Wang, Kunxun Qi, Yuming Shen, and Yu Deng. Validation of growing knowledge graphs by abductive text evidences. In AAAI, pages 2784-2791, 2019.
[35] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In ICML, volume 48, pages 2071-2080, 2016.
[36] Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. Simkgc: Simple contrastive knowledge graph completion with pre-trained language models. In ACL, pages 4281-4294, 2022.
[37] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Trans. Knowl. Data Eng. (TKDE), 29(12):2724-2743, 2017.
[38] Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. KEPLER: A unified model for knowledge embedding and pre-trained language representation. Trans. Assoc. Comput. Linguistics (TACL), 9:176-194, 2021.
[39] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In AAAI, pages 1112-1119, 2014.
[40] Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. Denoising relation extraction from document-level distant supervision. In EMNLP, pages 3683-3688, 2020.
[41] Rui Xing, Jie Luo, and Tengwei Song. Biorel: towards large-scale biomedical relation extraction. BMC Bioinform., 21-S(16):543, 2020.
[42] Chenyan Xiong, Russell Power, and Jamie Callan. Explicit semantic ranking for academic search via knowledge graph embedding. In WWW, pages 1271-1279, 2017.
[43] Xiaohan Xu, Peng Zhang, Yongquan He, Chengpeng Chao, and Chaoyang Yan. Subgraph neighboring relations infomax for inductive link prediction on knowledge graphs. In IJCAI, pages 2341-2347, 2022.
[44] Zezhong Xu, Peng Ye, Hui Chen, Meng Zhao, Huajun Chen, and Wen Zhang. Ruleformer: Context-aware rule mining over knowledge graph. In COLING, pages 2551-2560, 2022.
[45] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In ICLR, 2015.
[46] Fan Yang, Zhilin Yang, and William W. Cohen. Differentiable learning of logical rules for knowledge base reasoning. In NIPS, pages 2319-2328, 2017.
[47] Yuan Yang and Le Song. Learn to explain efficiently via neural logic inductive learning. In ICLR, 2020.
[48] Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. Docred: A large-scale document-level relation extraction dataset. In ACL, pages 764-777, 2019.
[49] Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. Distant supervision for relation extraction via piecewise convolutional neural networks. In EMNLP, pages 1753-1762, 2015.
[50] Yufeng Zhang, Weiqing Wang, Hongzhi Yin, Pengpeng Zhao, Wei Chen, and Lei Zhao. Disconnected emerging knowledge graph oriented inductive link prediction. In ICDE, pages 381-393, 2023.
To prove Theorem 1, we first introduce Lemma 1.
Lemma 1. Let  $\mathcal{G}$  be a set of hard triples,  $\mathcal{G}_{\mathrm{soft}}$  a set of soft triples,  $(a,r,b)$  an arbitrary triple,  $R$  an  $r$ -specific  $L$ -TE-rule of the form  $r(x,y) \leftarrow \mathcal{C}_1(x,z_1) \wedge \mathcal{C}_2(z_1,z_2) \wedge \ldots \wedge \mathcal{C}_L(z_{L-1},y)$ , and  $\theta_r^{(1,L)}$  the  $\{R\}$ -induced parameter assignment of TELM, then (1)  $\psi_{r,a,b}^{(1,L)} > 0$  if  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models H_R(a,b)$ , and (2)  $\psi_{r,a,b}^{(1,L)} = 0$  if  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \nmidneq H_R(a,b)$ .
Proof. Let  $\mathcal{K} = \mathcal{G} \cup \mathcal{G}^{-} \cup \mathcal{G}_{\mathrm{soft}} \cup \mathcal{G}_{\mathrm{soft}}^{-} \cup \{(e, I, e) \mid e \in \mathcal{E}\}$  where  $\mathcal{E}$  is the set of entities appearing in  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}}$ . Note that the confidence sore of each triple in  $\mathcal{K}$  is larger than 0.
(I) Consider the case where  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models H_R(a, b)$ . There exists at least one ground instance  $R_g$  of  $R$  such that  $H_R(a, b) = H_{R_g}$  and  $B_{R_g} \subseteq \mathcal{K}$ . There will be a sequence of entities  $c_1, \ldots, c_{L-1}$  and a sequence of relations  $r_1, \ldots, r_L$  such that  $(a, r_1, c_1), (c_1, r_2, c_2), \ldots, (c_{L-1}, r_L, b) \in \mathcal{K}$ , where  $r_i$  is a relation appearing in  $\mathcal{C}_i$  for all  $1 \leq i \leq L$ . If  $r_L \in \mathcal{R} \cup \mathcal{R}^-$ , suppose  $r_L$  is the  $k^{\mathrm{th}}$  relation in  $\mathcal{R} \cup \mathcal{R}^-$ , then by Condition 1 and 4 in Definition 2, we have  $[w_{\mathrm{orig}}^{(r,1,L)}]_k = 1$  and  $\alpha^{(r,1,L)} = 1$ . If  $r_L \in \mathcal{R}_{\mathrm{text}} \cup \mathcal{R}_{\mathrm{text}}^-$ , suppose  $r_L$  is the  $k^{\mathrm{th}}$  relation in  $\mathcal{R}_{\mathrm{text}} \cup \mathcal{R}_{\mathrm{text}}^-$ , then by Condition 2 and 5 in Definition 2, we have  $[w_{\mathrm{text}}^{(r,1,L)}]_k = 1$  and  $\beta^{(r,1,L)} = 1$ . If  $r_L = I$ , by Condition3 in Definition 2, we have  $w_{4n+1}^{(r,1,L)} = 1$ . By Equation (2), we must have  $\omega_k^{(r,1,L)} = 1$  for some  $k$ . By Equation (1), we further have  $\psi_{r,c_{L-1},b}^{(1,1)} \geq \phi_L > 0$  for  $\phi_L$  the confidence score of  $(c_{L-1}, r_L, b)$ . Likewise, if  $r_{L-1} \in \mathcal{R} \cup \mathcal{R}^-$ , suppose  $r_{L-1}$  is the  $k^{\mathrm{th}}$  relation in  $\mathcal{R} \cup \mathcal{R}^-$ , then by Condition 1 and 4 in Definition 2, we have  $[w_{\mathrm{orig}}^{(r,1,L)}]_k = 1$  and  $\alpha^{(r,1,L)} = 1$ . If  $r_{L-1} \in \mathcal{R}_{\mathrm{text}} \cup \mathcal{R}_{\mathrm{text}}^-$ , suppose  $r_{L-1}$  is the  $k^{\mathrm{th}}$  relation in  $\mathcal{R}_{\mathrm{text}} \cup \mathcal{R}_{\mathrm{text}}^-$ , then by Condition 2 and 5 in Definition 2, we have  $[w_{\mathrm{text}}^{(r,1,L-1)}]_k = 1$  and  $\beta^{(r,1,L-1)} = 1$ . By Equation (2), we must have  $\omega_k^{(r,1,L-1)} = 1$  for some  $k$ . By Equation (1), we further have  $\psi_{r,c_{L-2},b}^{(1,2)} > 0$ . In the same way, we can show that  $\psi_{r,c_{L-3},b}^{(1,3)} > 0$ , ...,  $\psi_{r,c_1,b}^{(1,L-1)} > 0$  and  $\psi_{r,a,b}^{(1,L)} > 0$  in turn. Therefore, we have  $\psi_{r,a,b}^{(1,L)} > 0$  if  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models H_R(a,b)$ .
(II) Consider the case where  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \not\models H_R(a, b)$ . Suppose  $\psi_{r,a,b}^{(1,L)} > 0$ , then by Equation (1), there must be some  $k \in \{1, \ldots, 4n + 1\}$  such that  $\omega_k^{(r,1,1)} > 0$ ,  $r_k$  appears in  $\mathcal{C}_1$  and there exists  $(a, r_k, c_1) \in \mathcal{K}$  fulfilling  $\psi_{r,c_1,b}^{(1,L-1)} > 0$ . Since  $\psi_{r,c_1,b}^{(1,L-1)} > 0$ , by Equation (1), there must be also some  $l \in \{1, \ldots, 4n + 1\}$  such that  $\omega_l^{(r,1,2)} > 0$ ,  $r_l$  appears in  $\mathcal{C}_2$  and there exists  $(c_1, r_l, c_2) \in \mathcal{K}$  fulfilling  $\psi_{r,c_2,b}^{(1,L-2)} > 0$ . In the same way, we can show that there exists relation  $r_m$  appearing in  $\mathcal{C}_m$  and entity  $c_m$  such that  $(c_{m-1}, r_m, c_m) \in \mathcal{K}$  and  $\psi_{r,c_m,b}^{(1,L-m)} > 0$  for  $m = 3, \ldots, L-1$  in turn, while there exists relation  $r_L$  appearing in  $\mathcal{C}_L$  such that  $(c_{L-1}, r_L, b) \in \mathcal{K}$ . Hence there exists a sequence of entities  $c_1, \ldots, c_{L-1}$  and a sequence of relations  $r_1, \ldots, r_L$  such that  $(a, r_1, c_1), (c_1, r_2, c_2), \ldots, (c_{L-1}, r_L, b) \in \mathcal{K}$ , where  $r_i$  is a relation appearing in  $\mathcal{C}_i$  for all  $1 \leq i \leq L$ . These two sequences constitute a ground instance  $R_g$  of  $R$  such that  $H_R(a, b) = H_{R_g}$  and  $B_{R_g} \subseteq \mathcal{K}$ , contradicting  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \not\models H_R(a, b)$ . Thus  $\psi_{r,a,b}^{(1,L)} \leq 0$ . By Equation (1) we have  $\psi_{r,a,b}^{(1,L)} \geq 0$ . Therefore, we have  $\psi_{r,a,b}^{(1,L)} = 0$  if  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \not\models H_R(a, b)$ .
Proof. Lemma 1 implies that, for all  $R_{k} \in \Sigma$ ,  $\psi_{r,a,b}^{(k,L)} > 0$  if  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models H_{R_k}(a,b)$  and  $\psi_{r,a,b}^{(k,L)} = 0$  otherwise.
$(\Rightarrow)$  Suppose  $\Psi_{r,a,b}^{(N,L)} > 0$ . Then by Equation (3) and Condition 6 in Definition 2, there exists at least one  $r$ -specific L-TE-rule  $R_k \in \Sigma$  such that  $\psi_{r,a,b}^{(k,L)} > 0$ . By Lemma 1 we have  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models H_{R_k}(a,b)$ . Since  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models H_{R_k}(a,b)$  and  $R_k \in \Sigma$ , we have  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models_{\Sigma}(a,r,b)$ .
$(\Leftarrow)$  Suppose  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models_{\Sigma}(a, r, b)$ . Then we have  $\mathcal{G} \cup \mathcal{G}_{\mathrm{soft}} \models H_{R_k}(a, b)$  for some  $R_k \in \Sigma$ . By Lemma 1 we have  $\psi_{r,a,b}^{(k,L)} > 0$  and for all  $k' \neq k$ ,  $\psi_{r,a,b}^{(k',L)} \geq 0$ . By Equation (3) and Condition 6 in Definition 2, we have  $\Psi_{r,a,b}^{(N,L)} > 0$ .
B Formalization of Multi-instance Learning for RTE
In Section 4.1 we have described how to generate soft triples by applying a RTE model based on BERT [10]. Due to the space limitation, we omitted some details. Thus, in this section we further elaborate on how to apply multi-instance learning to RTE based on BERT.
Given  $\mathcal{D}_{\mathrm{train}} = \{(\tau_i,S_i,y_i)\}_{1\leq i\leq N_{\mathrm{tra}}}$ , where  $\tau_{i} = (h_{i},r_{i},t_{i})$  denotes a triple,  $h_i = (w_1^{h_i},\dots,w_{N_{h_i}}^h)$  the head entity with  $N_{h_i}$  tokens,  $r_i = (w_1^{r_i},\dots,w_{N_{r_i}}^{r_i})$  the relation with  $N_{r_i}$  tokens,  $t_i = (w_1^{t_i},\dots,w_{N_{t_i}}^{t_i})$  the tail entity with  $N_{t_i}$  tokens,  $S_{i} = \{s_{i,j}\}_{1\leq j\leq N_{S_{i}}}$  a set of  $N_{S_i}$  sentences mentioning  $(h_i,t_i)$ , and  $s_{i,j} = (w_1^{s_{i,j}},\dots,w_{N_{s_{i,j}}}^{s_{i,j}})$  a sentence with  $N_{s_{i,j}}$  tokens, we create a set of sequences  $X_{i} = \{x_{i,j}\}_{1\leq j\leq N_{S_{i}}}$  for each instance in  $\mathcal{D}_{\mathrm{train}}$ . For  $1\leq j\leq N_{S_i}$ , the sequence  $x_{i,j}$  is created by first concatenating  $h_i$ ,  $r_i$ ,  $t_i$  and  $s_{i,j}$ , and then adding a special token [CLS] in front of the first token  $w_1^{h_i}$ , a special token [SEP] between  $w_{N_{t_i}}^{t_i}$  and  $w_1^{s_{i,j}}$ , and a special token [SEP] behind the last token  $w_{N_{s_{i,j}}}^{s_{i,j}}$ . All sequences are fed to BERT to calculate their contextual representations. By  $C_{i,j}\in \mathbb{R}^{N_{x_i,j}\times d}$  we denote the output of BERT for  $x_{i,j}$ , where  $N_{x_i,j} = N_{h_i} + N_{r_i} + N_{t_i} + N_{s_{i,j}} + 3$  is the number of tokens in  $x_{i,j}$  and  $d$  is the dimension of token embedding. We use the embedding of the [CLS] token, namely  $C_{i,j,1}$ , as the contextual embedding for textual entailment classification. The entailment probability for  $x_{i,j}$  is calculated by
$$
\epsilon_ {i, j} = \sigma \left(W C _ {i, j, 1} + b\right) \tag {5}
$$
where  $W \in \mathbb{R}^{1 \times d}$  is a trainable weight vector,  $b \in \mathbb{R}^1$  a trainable bias value and  $\sigma$  the sigmoid function. Then the entailment probability for  $X_i$  is calculated by
$$
\phi_ {i} = \max  _ {1 \leq j \leq N _ {S _ {i}}} \epsilon_ {i, j} \tag {6}
$$
The entire model is trained by minimizing the cross-entropy loss, formally defined as
$$
\mathcal {L} = - \frac {1}{N _ {\mathrm {t r a}}} \sum_ {i = 1} ^ {N _ {\mathrm {t r a}}} y _ {i} \log \phi_ {i} + (1 - y _ {i}) \log \left(1 - \phi_ {i}\right) \tag {7}
$$
C Rule Extraction From TELM
Based on the formalization of TELM, we can interpret TE-rules from its parameters. The process of interpretation is shown in Algorithm 1. Intuitively, Algorithm 1 interprets TE-rules from the parameter assignment of TELM using beam search, where  $\pi$  denotes the threshold to determine whether the original relations and textual relations are used in atoms,  $m$  the beam size,  $f_{l}$  the set of  $(R', s)$ -pairs for the  $l^{\mathrm{th}}$  atom,  $R'$  the currently interpreted (partial) rule and  $s$  its estimated score. It should be noted that the process for interpreting  $r$ -specific TE-rules outputs up to  $m$  interpreted rules sharing the same confidence score  $u_{r}^{(k)}$  for the  $k^{\mathrm{th}}$  target rule.
We applied LSTK to the following state-of-the-art rule learning methods for empirical comparisons.
- AMIE+. AMIE+ [13] is upgraded from the well-known rule learner AMIE [14]. It imposes several pruning heuristics to mine logical rules.
- NeuralLP. NeuralLP [46] is the first neural approximate method that exploits Tensorlog operators to learn chain-like rules. It translates the structure learning of logical rules from discrete space to continuous space.
Algorithm 1: Interpreting  $r$ -specific  $L$ -TE-rules
1 Input: beam size  $m\geq 1$  , selection threshold  $\pi \in (0.0,0.5]$  , and a parameter assignment of TELM for the relation  $r$  , namely  $\theta_r^{(N,L)} =$ $\{[w_{\mathrm{orig}}^{(r,k,l)}]_i,[w_{\mathrm{text}}^{(r,k,l)}]_i\}_{1\leq k\leq N,1\leq l\leq L,1\leq i\leq 2n}\cup \{w_{4n + 1}^{(r,k,l)},\alpha^{(r,k,l)},\beta^{(r,k,l)}\}_{1\leq k\leq N,1\leq l\leq L}$    
2 Output: a set of up to mN r-specific L-TE-rules   
3  $\mathbb{R}\gets \emptyset$    
4 for  $1\leq k\leq N$  do   
5  $f_0\leftarrow \{(\Delta^L,1)\}$  where  $\Delta$  denotes a placeholder to be filled;   
6  $\forall 1\leq l\leq L:f_l\gets \emptyset$  .   
7 for  $1\leq l\leq L$  do   
8 for  $(R,s)\in f_{l - 1}$  do if  $\alpha^{(r,k,l)}\geq \pi$  and  $\beta^{(r,k,l)} <   \pi$  then for  $1\leq i\leq 2n$  do  $R^{\prime}\gets R$  with the  $l^{\mathrm{th}}$  placeholder replaced with  $r_i$  .  $fl\gets fl\cup \{(R',[w_{\mathrm{orig}}^{(r,k,l)}]_is)\}$  . else if  $\alpha^{(r,k,l)} <   \pi$  and  $\beta^{(r,k,l)}\geq \pi$  then for  $1\leq i\leq 2n$  do  $R^{\prime}\gets R$  with the  $l^{\mathrm{th}}$  placeholder replaced with  $r_{2n + i}$  .  $fl\gets fl\cup \{(R',[w_{\mathrm{text}}^{(r,k,l)}]_is)\}$  . else if  $\alpha^{(r,k,l)}\geq \pi$  and  $\beta^{(r,k,l)}\geq \pi$  then for  $1\leq i\leq 2n$  do  $R^{\prime}\gets R$  with the  $l^{\mathrm{th}}$  placeholder replaced with  $r_i\vee r_{2n + i}$  .  $fl\gets fl\cup \{(R',\max ([w_{\mathrm{orig}}^{(r,k,l)}]_i,[w_{\mathrm{text}}^{(r,k,l)}]_i)s)\}$  . else  ${R^{\prime}}\gets R$  with the  $l^{\mathrm{th}}$  placeholder replaced with  $I$  .  $fl\gets fl\cup \{(R',w_{4n + 1}^{(r,k,l)}s)\}$  . sort  $fl = \{(R,s)_j\}_{1\le j\le N_{fl}}$  in the descending order of s and preserve the top-m in  $f_{l}$  ..  $\mathbb{Q}\gets \{R'$  rewritten from  $R$  to the form of a TE-rule |  $(R,s)\in f_L\}$  .  $\mathbb{R}\gets \mathbb{R}\cup \mathbb{Q};$
- DRUM. DRUM [30] extends NeuralLP to learn rules with dynamic lengths by adding the identity relation. It exploits a bidirectional LSTM network to learn logical rules together with their scores simultaneously.
- RNNLogic. RNNLogic [28] leverages an LSTM [16] based rule generator and a reasoning predictor to learn logical rules together with their weights interactively.
For LSTK-NeuralLP, LSTK-DRUM and LSTK-RNNLogic, we constructed adjacency matrix for each relation from the mixed background  $\mathcal{K}_{\mathrm{mix}}$ , where the weights of edges are set by the confidence scores. It should be noted that AMIE+ can only learn rules from hard triples in the background KG, and thus the confidence scores of soft triples are not used in LSTK-AMIE+.
D.2 Implementation of Baselines
We implemented LSTK-NeuralLP $^{6}$  and LSTK-DRUM $^{7}$  by Tensorflow 1.15.0 based on their published code. They were trained by Adam, where the initial learning rate was set to 1e-3, the mini-batch size to 32, the maximum number of training epochs to 10, and the maximum length of learnt rules to 4.
$^{6}$ https://github.com/fanyangxyz/Neural-LP
<sup>7</sup>https://github.com/alisadeghian/DRUM
(a) Proportion of rules involving textual relations.
Figure 3: Analysis on learnt rules and the threshold  $\delta$ .
(b) Comparisons on  $\delta$  for LSTK-TELM.
LSTK-RNNLogic $^{8}$  was implemented by Pytorch 1.10.0 based on its published code. Both the rule generator and reasoner in RNNLogic were trained by Adam, where the initial learning rate was set to 1e-3 (resp. 1e-1) for the generator (resp. the reasoner), the maximum number of training epochs to 5000 (resp. 5) for the generator (resp. the reasoner), the mini-batch size to 1024, and the maximum length of learnt rules to 2. Experiments on these baselines were conducted on an A100 GPU with 40GB memory. We implemented LSTK-AMIE $^{+9}$  based on its published code, and ran it on a Linux equipped with an Intel(R) Xeon(R) Gold 6248R processor with 3.0 GHz and 128 GB RAM.
D.3 Evaluation Metrics
For each test triple  $(h,r,t)$  in evaluation, we built two queries  $(?,r,t)$  and  $(?,r^{-},h)$  for answering  $h$  and  $t$ , respectively. For each query, we computed the truth degrees for corrupted head triples and then computed the rank of the correct answer. The rank of the correct answer is defined by  $m + (n + 1)/2$ , where  $m$  is the number of triples with higher truth degrees than the correct answer and  $n$  the number of corrupted head triples with the same truth degree as the correct answer. The rank for each test triple is calculated by the mean rank of two queries. Based on the rank, we reported the Mean Reciprocal Rank (MRR for short) and Hit@k (H@k for short) metrics under the filtered setting.
D.4 Statistical Analysis on Learnt Rules
We conducted statistical analysis on learnt rules to show how much textual knowledge has been used in different models, as shown in sub-figure (a) in Figure 3. We can see that about  $30\%$  logical rules extracted from either LSTK-NeuralLP or LSTK-DRUM (resp. about  $60\%$  logical rules extracted from LSTK-TELM) involve at least one textual relation. It implies that more soft triples are leveraged in the reasoning of LSTK-TELM. We have also found that about  $90\%$  logical rules extracted from LSTK-RNNLogic involve at least one textual relation, which is much higher than the rate of other approaches. This may be due to the large search space explored by RNNLogic in searching potential paths from the KG for rule generation, where the introduction of soft triples substantially scales up the search space, leading to a lot of logical rules that involve textual relations. Recall Table 2 that LSTK-RNNLogic achieves lower performance than other LSTK-enhanced models. These results imply that the overuse of textual relations may degrade the performance due to noise in soft triples.
D.5 Hyper-parameter Analysis
LSTK introduces a threshold  $\delta$  to filter soft triples with high confidence scores. We conducted experiments to show the impact of different setting of  $\delta$ , as shown in sub-figure (b) in Figure 3. Each curve on the MRR scores has a moderate jitter with increasing  $\delta$ . This can be explained by that, although the use of soft triples tends to improve the performance of link prediction, the overuse of soft triples may introduce much noise, leading to a performance degradation. By observing the curves of LSTK-DRUM and LSTK-TELM, we recommend to set  $\delta = 0.5$  as default.
$^{8}$ https://github.com/DeepGraphLearning/RNNLogic
<sup>9</sup>https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/amie/