SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(1)  $\forall 1\leq i\leq 2n:[w_{\mathrm{orig}}^{(r,k,l)}]_i = 1$  if  $r_i$  appears in  $\mathcal{C}_{k,l}$ , otherwise  $[w_{\mathrm{orig}}^{(r,k,l)}]_i = 0$ .
(2)  $\forall 1\leq i\leq 2n:[w_{\mathrm{text}}^{(r,k,l)}]_i = 1$  if  $r_{2n + i}$  appears in  $\mathcal{C}_{k,l}$ , otherwise  $[w_{\mathrm{text}}^{(r,k,l)}]_i = 0$ .
(3)  $w_{4n + 1}^{(r,k,l)} = 1$  if  $r_{4n + 1}$  appears in  $\mathcal{C}_{k,l}$ , otherwise  $w_{4n + 1}^{(r,k,l)} = 0$ .
(4)  $\alpha^{(r,k,l)} = 1$  if an original relation appears in  $\mathcal{C}_{k,l}$ , otherwise  $\alpha^{(r,k,l)} = 0$ .
(5)  $\beta^{(r,k,l)} = 1$  if a textual relation appears in  $\mathcal{C}_{k,l}$ , otherwise  $\beta^{(r,k,l)} = 0$ .
(6)  $u_r^{(k)} = 1$
Theorem 1. Let  $\mathcal{G}$  be a set of hard triples,  $\mathcal{G}_{\mathrm{soft}}$  a set of soft triples,  $(a,r,b)$  an arbitrary triple,  $\Sigma = \{R_k\}_{1\leq k\leq N}$  a set of  $r$ -specific  $L$ -TE-rules and  $\theta_r^{(N,L)}$  the  $\Sigma$ -induced parameter assignment of TELM, then  $\Psi_{r,a,b}^{(N,L)} > 0$  if and only if  $\mathcal{G}\cup \mathcal{G}_{\mathrm{soft}}\models_{\Sigma}r(a,b)$ .
Explaining by Backward Reasoning. Thanks to the faithfulness between TELM and TE-rules, we can extract TE-rules by interpreting the parameters of a learnt TELM model using beam search algorithm (see Algorithm 1 in the appendix). For each inferred missing fact, LSTK yields at least one reasoning path as explanation by backward reasoning. Given a triple  $(h,r,t)$  that is estimated to be positive, a set of extracted TE-rules, the background KG  $\mathcal{K}_{\mathrm{mix}}$  and the corpus  $\mathcal{T}$ , we can find a path between  $h$  and  $t$ . Whenever the path involves a soft triple, we treat the sentence in  $\mathcal{T}$  that has the maximum confidence score as the supporting text for this soft triple.
4.1 Data Construction and Evaluation Metrics
We collected three benchmark KGs with corresponding text corpora from the field of relation extraction for empirical evaluation, which are HacRED $^2$  [7], DocRED $^3$  [48] and BioRel $^4$  [41].
$^{2}$ https://github.com/qiaojiim/HacRED
<sup>3</sup>https://github.com/thunlp/DocRED
4https://bit.ly/biorel_dataset
Dataset	#Ent.	#Rel.	#Train	#Valid	#Test	#Texts	#Soft.	PUE (exist)	PUE (soft)	PUE (all)
HacRED	20,800	12	20,637	2,499	2,551	4,578	38.8M	98.7%	31.6%	30.0%
DocRED	17,997	28	24,384	2,819	3,844	19,517	4.7M	83.7%	84.9%	67.8%
BioRel	15,566	34	25,854	2,406	3,032	150,687	123.4M	79.4%	62.1%	46.8%
Table 1: Statistical information for each dataset, where #Ent. (resp. #Rel.) denotes the number of entities (resp. relations), #Train/Valid/Test respectively the number of triples for training/validation/test, #Texts the number of texts and #Soft. the number of extracted soft triples. PUE (short for Proportion of Unseen Entities) denotes the proportion of triples in the test set involving unseen entities in the training triples, where exist/soft/all respectively denote that the training triples are treated as existing triples in the original training set, as extracted soft triples, or as both existing triples and soft triples.
Considering that these datasets originally have a number of relations that can hardly appear in heads of potential logical rules, we filtered these relations by applying AMIE+ [13] to mine logical rules that have up to three body atoms from the complete set of existing triples for each dataset and omitting the relations that have not been involved in the body atoms of any mined rules. Under the inductive setting, there should be entities in the test set that do not appear in the training set. Therefore, we applied the partitioning algorithm METIS [17] to the entity graph where an edge indicates that the two end-points have at least one filtered relation, so as to divide the set of entities into 10 subsets. The train-valid/test split of each dataset was then extracted from  $8/1/1$  of these subsets by omitting triples across different splits. Statistics on all above datasets are reported in Table 1.
Following [28], we reported the Mean Reciprocal Rank (MRR) and Hit@k metrics under the filtered setting [3] for empirical evaluation.
We implemented the textual entailment model by Pytorch 1.10.0. The model was initialized by the pretrained language model BERT with 12 transformer layers, which outputs 768-dimensional (i.e.  $d = 768$ ) token embeddings. Then it was trained by Adam [18] with warm up [10], where the initial learning rate was set to 5e-5, the mini-batch size to 8 and the maximum number of training epochs to 3. We applied dropout [32] to each layer by setting the dropout rate to 0.1.
We implemented  $\mathrm{TELM}^5$  by Pytorch 1.10.0. It was trained on an A100 GPU with 40GB memory by Adam [18] with 50 training epochs for HacRED and DocRED, and 20 for BioRel. The hyperparameters were set to maximize MRR on the validation set. The initial learning rate was set to 1e-1 and the mini-batch size to 32 for all datasets. We also applied dropout [32] to the output states by setting the dropout rate to 0.3. The maximum length of each rule  $L$  is set to 3 for HacRED and DocRED, and 2 for BioRel. The maximum size of learning rules  $N$  is set to 20 for all datasets.
We conducted experiments on the three enhanced datasets for link prediction under the inductive learning setting, where existing triples in the training set are used as background KG for training, existing triples in training and validation sets are used as background KG for validation, and existing triples in training, validation and test sets are used for test. The hyper-parameter  $\delta$  to filter soft triples with high confidence scores was set to 0.5 in all experiments.
Table 2 reports the comparison results on three datasets. Textual Copy Rule (simply TCR) is a baseline that uses only textual copy rules of the form “ $r(x,y) \gets r_{\text{text}}(x,y)$ ” in reasoning instead of using any other rules or neural models. TCR has not any extra logical reasoning ability beyond the textual entailment model. By LSTK- $X^{\dagger}$  we denote enhanced models that use soft triples in the background KG, where  $X$  denotes the original model. By LSTK-X we denote enhanced models that further extends LSTK- $X^{\dagger}$  by allowing textual relations to appear in learnt rules. Results show that the proposed method (i.e. LSTK-TELM) significantly outperforms all the baseline methods with p-value  $< 0.05$  by two-tailed t-tests. Specifically, LSTK-TELM outperforms LSTK-DRUM by absolute gains of  $24.8\% / 4.2\% / 22.4\%$  in Hit@1 scores on the HacRED/DocRED/BioRel datasets, respectively. Further, we can see from the comparison results between LSTK- $X^{\dagger}$  and LSTK-X
$^{5}$ Code and data about our implementations are available at: https://github.com/qikunxun/LSTK
Method	HacRED	DocRED	BioRel	p-value
MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10	MRR	H@1	H@3	H@10
Textual Copy Rule (TCR)	0.382	31.5	42.2	51.5	0.030	1.8	3.6	5.4	0.074	4.5	8.2	13.4	3.3e-9
AMIE+ [13]	0.122	11.1	13.5	13.9	0.217	16.7	26.0	29.5	0.136	10.7	15.3	18.6	2.0e-7
LSTK-AMIE+†	0.177	13.3	19.0	28.2	0.259	21.4	29.1	33.4	0.138	10.2	15.0	20.2	1.9e-7
LSTK-AMIE+	0.122	10.7	13.2	15.1	0.302	26.1	33.0	37.7	0.116	9.5	12.9	15.2	1.3e-3
RNNLogic [28]	0.162	15.1	17.2	17.8	0.226	15.7	26.3	35.6	0.172	12.0	19.3	28.7	1.9e-7
LSTK-RNNLogic†	0.234	16.6	26.2	37.3	0.343	24.9	40.4	52.1	0.148	10.0	16.0	23.2	5.8e-6
LSTK-RNNLogic	0.394	32.8	42.3	52.1	0.396	31.6	45.1	53.9	0.284	20.3	33.8	43.5	4.8e-6
NeuralLP [46]	0.395	34.4	42.6	48.9	0.310	24.0	35.4	43.4	0.317	23.2	26.8	46.2	1.3e-8
LSTK-NeuralLP†	0.320	21.1	35.4	56.7	0.382	29.2	43.5	55.2	0.304	21.2	34.5	47.2	1.6e-5
LSTK-NeuralLP	0.484	38.1	53.4	67.5	0.488	40.9	54.2	62.2	0.384	27.6	44.9	58.1	4.5e-4
DRUM [30]	0.387	33.6	41.4	48.5	0.352	28.1	40.2	47.7	0.314	23.1	36.2	46.4	3.0e-7
LSTK-DRUM†	0.347	23.8	39.6	56.0	0.392	29.1	46.3	56.5	0.292	20.4	32.9	46.0	1.5e-5
LSTK-DRUM	0.538	43.3	59.8	72.6	0.482	40.0	54.1	62.0	0.421	31.7	48.3	60.1	3.1e-4
LSTK-TELM (this work)	0.734	68.1	77.1	83.2	0.514	44.2	56.0	63.9	0.600	54.1	62.9	69.6	-
(1) w/o soft triples	0.416	37.7	43.6	48.1	0.380	30.8	44.0	49.4	0.339	25.9	38.3	48.1	1.1e-6
(2) w/o textual copy rules	0.710	65.0	74.9	82.1	0.502	43.2	54.6	62.7	0.511	44.6	54.7	62.8	2.1e-3
(3) w/o textual relations	0.579	48.2	64.3	76.4	0.438	35.5	49.1	58.5	0.356	25.4	40.3	54.8	5.1e-5
(4) w/o disjunctive atoms	0.693	63.3	73.2	80.6	0.453	38.4	49.6	57.7	0.430	36.3	45.7	55.6	2.8e-4
(5) w/o confidence scores	0.730	67.3	76.9	83.0	0.508	43.4	55.7	63.4	0.574	50.8	61.3	69.5	8.4e-3
(6) Using relation extraction	0.723	66.1	76.7	83.3	0.499	43.0	54.5	61.8	0.572	50.7	60.9	68.8	1.9e-4
Table 2: Comparison results on the HacRED, DocRED and BioRel datasets.
that, the introduction of textual relations bring significant performance improvements. For LSTK-NeuralLP $\dagger$  and LSTK-DRUM $\dagger$ , their performances are even worse than their original models that do not use any soft triples, indicating some negative impact of noise from soft triples.
We further conducted ablation studies on several variants of LSTK-TELM to verify the effectiveness of key components in both LSTK and TELM. In (1) we omitted all the soft triples in the background KG (i.e., only hard triples are used). Results show that the LSTK framework (using soft triples) pushes TELM by a significant margin with p-value 1.1e-6. It implies that the use of soft triples helps to greatly improve the performance. In (2) we omitted the corresponding soft triple for each hard triple in the background KG. Results show that this variant model already significantly outperforms the variant model in (1), indicating that soft triples help to improve the reasoning ability without using textual copy rules. In (3) we omitted the textual relations in TE-rules. Results show that the use of textual relations pushes LSTK-TELM by a significant margin with p-value 5.1e-5. It can be explained by the fact that the separation of original and textual relations helps to discard the noisy soft triples, resulting in higher reasoning performance. In (4) we omitted the learning of disjunctive atoms of the form  $r(u,v)\lor r_{\mathrm{text}}(u,v)$  in TELM. Results show that the introduction of disjunctive atoms brings significant performance gains with p-value 2.8e-4. This confirms that the introduction of disjunctive atoms in TE-rules provides a more flexible mechanism to control noise. In (5) we fixed the confidence scores (i.e. entailment probabilities) of soft triples to 1.0. Results show that the performance significantly drops with p-value 8.4e-3. It implies that the computed entailment probabilities reflect the truth degrees of supporting facts and help to learn more informative rules. In (6) we trained a state-of-the-art relation extraction model proposed in [2] based on BERT to replace the textual entailment model for generating soft triples. Results show that the use of relation extraction model leads to a significant performance degradation with p-value 1.9e-4. This may be due to that a textual entailment model can deal with open relations and exploit more text semantics on relations than a relation extraction model does.
To clarify why the learning of TE-rules help improve the performance, we conducted case study for applying learnt logical rules from LSTK-TELM on the DocRED dataset, as shown in Table 3. In the first case, the logical rule is extracted from LSTK-TELM and can be applied to infer the positive triple “(Lois, Child, Superboy)”. We can see from this case that the textual entailment model is able to infer from the relation mention “biological child” that, the entities “Lois” and “Clark” are spouse, thereby yielding a supporting soft triple “(Lois, Spouse, Clark, 0.76)” for reasoning. In the second case, it can be seen that the same logical rule in the first case cannot infer that the triple “(John, Child, Joseph)” is positive, since the key supporting fact “(Joseph, Father, Martyn)” is absent in the background KG. In the third case, the logical rule is “Child(x,y) ← (Spouse(x,z0) ∨ Spouse_text(x,z0)) ∧
Triple (fact)	Logical rule	Supporting triple/text	Pred.	Label
(Lois, Child, Superboy)	Child(x,y) ← (Spouse(x,z0) ∨ Spouse_text(x,z0)) ∧ Father-(z0,y)	Supporting hard triple: (Superboy, Father, Clark)
Supporting soft triple: (Lois, Spouse, Clark, 0.76)
Supporting text: Clark and Lois' biological child in DC Comics canon was born in Convergence: Superman#2 (July 2015), a son named Jonathan Samuel Kent, who eventually becomes Superboy.	True	True
(John, Child, Joseph)	Child(x,y) ← (Spouse(x,z0) ∨ Spouse_text(x,z0)) ∧ Father-(z0,y)	Supporting hard triple: -
Supporting soft triple: (John, Spouse, Martyn, 0.93)
Supporting text: The Road to Ruin is a 1970 album released by husband and wife John and Beverley Martyn.	False	False
(Linda, Child, Joseph)	Child(x,y) ← (Spouse(x,z0) ∨ Spouse_text(x,z0)) ∧ Father-(z0,y)	Supporting soft triple: (Linda, Spouse, Paul, 0.88)
Supporting text: In 1998, after Linda's death, Paul rearranged the song for string quartet to be played at memorial concerts for his late wife.
Supporting soft triple: (Joseph, Father, Paul, 0.53)
Supporting text: In 1995, at a ceremony in Colombo, Pope John Paul II beatified Father Joseph Vaz, an early missionary to the country, who is known as the Apostle of Ceylon.	True	False
Table 3: Examples for applying learnt TE-rules from LSTK-TELM on the DocRED dataset, where Pred. abbreviates prediction. The words marked as blue denote entities and the words marked as red relation mentions.  $r^{-}$  denotes the inverse relation of  $r$ .
$\mathsf{Father}_{\mathsf{text}}^{-}(z_{0},y)$ . In this case, the textual entailment model mistakenly yields a supporting soft triple  $(\mathsf{Joseph},\mathsf{Father},\mathsf{Paul},0.53)$  based on the words "beatified Father", thereby wrongly inferring the triple "(Linda, Child, Joseph)" to be positive. In general, these cases show that the use of soft triples is crucial in yielding a plausible explanation for a test triple and that the separation of original and textual relations provides a flexible mechanism to discard noisy soft triples.
5.1 Knowledge Graph Completion
There are two categories of prominent approaches to knowledge graph completion. They are embedding-based and rule-based, respectively.
**Embedding-based methods.** Knowledge graph embeddings (KGE) [3, 39, 19, 45, 35, 33] are a kind of well-known embedding-based methods. They aim to represent entities and relations in KGs as low-dimensional real-valued vectors [37]. Although KGE methods have been shown to be effective in large-scale KGC, they can hardly generalize to the inductive learning setting. Besides, KGE methods are hard to be interpreted by human due to their black-box nature. In recent years, graph neural networks (GNNs) [34, 6, 23, 43, 50] have shown promising results in addressing KGC. They can also be considered as embedding-based, as they also learn latent representation for reasoning. While GNN-based methods have been shown to excel in handling the inductive learning setting, we have not incorporated GNNs with LSTK. The reasons are three-fold. Firstly, GNNs are still black-box models with limited interpretability, whereas the proposed LSTK framework is committed to explain why a missing fact is inferred. Secondly, GNN-based methods necessitate sub-graph extraction to handle test triples involving unseen entities. The process of sub-graph extraction becomes highly time-consuming when dealing with a large number of soft triplets, e.g., it takes about 10 hours for GraIL [34] to process 1M triples for sub-graph extraction. Thirdly, the confidence scores from soft triples cannot be directly incorporated in GNN-based methods, unless the message passing mechanism in GNN-based methods is redesigned to handle the confidence scores on links.
Rule-based methods. Different from embedding-based methods, rule-based methods aim at building effective rule-based systems for KGC. They can naturally reason under the inductive learning setting and provide logical rules as explanations. Learning rule-based systems has been widely studied in the field of Inductive Logic Programming (ILP). ILP methods such as AMIE+ [13] and AnyBURL [24] usually impose search algorithms with pruning heuristics to mine logical rules in a generate-and-test manner. RNNLogic [28] extends this manner to learn logical rules and their weights interactively based on neural models. In recent years, neural approximate methods are proposed to learn rule-based
systems from structural knowledge in KGs by learning continuous parameters using Tensorlog [8] operators. State-of-the-art neural approximate methods include NeuralLP [46], DRUM [30], NLIL [47] and Ruleformer [44]. In addition to applying Tensorlog, other neural approximate methods such as NTP [29] and CTP [25] learn logical rules based on neural theorem provers. All these methods learn rule-based systems from structural knowledge alone and most of them focus merely on learning chain-like rules. In contrast, we study learning more expressive TE-rules from both structural and textual knowledge.
Other approaches using textual information. It is worth noting that there are several approaches that focus on enhancing KGs with texts. For instance, recent advance of pre-trained language models (PLMs) for KGC such as KEPLER [38], PKGC [21] and SimKGC [36] have considered textual information of triples in KGs. The differences between PLM-based methods and LSTK are twofold. On one hand, they consider different types of textual knowledge. In more detail, PLM-based methods consider text information such as contexts of triples, descriptions aligned to entities and pre-trained textual knowledge. In contrast, LSTK is proposed to address the KGC scenario where some knowledge (i.e., real-world facts) is provided in the structured KG and other knowledge is given by a text corpus. Thus, the textual knowledge considered in LSTK corresponds to a set of potential facts, namely the soft triples. On the other hand, PLM-based methods cannot work in our scenario because they can only obtain textual knowledge by pre-training a model over a given text corpus, whereas our setting allows to process new text corpora given in the test phase (see the problem setting in Section 3). The pre-training process is generally time-consuming and requires massive computing resources, thus it is impractical to be applied in the test phase. Except for PLM-based methods, the work [12] proposes a knowledge verification system by combining logical and textual information to compute explanations for validating new triples. However, the system should work with a predefined logical theory. In contrast, our approach does not require any predefined logical theory but learns one approximate logical theory on the fly. The recent work [15] improves the knowledge coverage of a knowledge base (KB) by facts extracted from text corpora. They have not focused on reasoning over facts. In contrast, we focus on enhancing the reasoning ability of rule-based systems by soft triples extracted from text corpora.
5.2 Distant Supervision for Relation Extraction
Distant supervision [26] is a dominant paradigm to handle the problem of lacking labeled data in relation extraction. It assumes that if two entities involve a relation, then any sentence mentioning them together might express that relation. Based on this assumption, a large number of labeled data for relation extraction can be automatically generated by aligning triples with texts. However, this may introduce noise to the generated data. Therefore, most previous studies exploit multi-instance learning [11] by putting instances with the same entity pair into bags, to alleviate the impact of noisy instances. A number of neural models including CNN [49], Bi-LSTM [5] and BERT [40] have been incorporated with multi-instance learning and used as classification models for relation extraction. We also impose distant supervision to score soft triples from a text corpus, but we resort to textual entailment models to make the best of relation mentions, instead of using classification models where relations are treated as labels to be predicted.
In this work we have proposed a two-stage framework, named LSTK, to learn rule-based systems from both structural and textual knowledge. It computes a set of soft triples by distant supervision in the first stage and applies these soft triples to enhance the learning of neural approximate rule-based systems in the second stage. To mitigate the negative impact of noise from soft triples, we have proposed a new formalism for logical rules named TE-rules and a neural model named TELM for learning TE-rules. We introduced three new datasets for empirical evaluation. Experimental results demonstrate significant improvements achieved by learning TE-rules from soft triples. Our case study further reveals how TE-rules help to control noise from soft triples.
From the third case in case study, we observe that some errors produced by the textual entailment model will propagate to the final prediction. This is known as the error propagation issue in pipeline approaches. To tackle this issue, our future work will focus on studying fully end-to-end frameworks to learn TE-rules from both structural and textual knowledge.
LSTK merely works for knowledge graphs with aligned text corpora, but in general knowledge graphs do not come with aligned texts. In order to adapt LSTK to more real-world applications, we generate soft triples automatically under the distant supervision assumption that any sentence mentioning two entities might involve a relation between these two entities. This assumption enables us to fetch texts via information retrieval tools such as search engines or to generate supporting text via large language models (LLMs) without the need of manually constructing texts. Note that all the original datasets of HacRED [7], DocRED [48] and BioRel [41] are collected by distant supervision. Although the performance of LSTK depends on the quality of distant supervision, LSTK indeed works for any knowledge graphs since aligned texts can easily be collected by distant supervision.
This paper was supported by National Natural Science Foundation of China (No. 62276284, 61976232, 61876204), National Key Research and Development Program of China (No. 2021YFA1000504), Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515011470, 2022A1515011355, 2020A1515010642), Guangzhou Science and Technology Project (No. 202201011699), Shenzhen Science and Technology Program (KJZD2023092311405902), Guizhou Provincial Science and Technology Projects (No. 2022-259), Humanities and Social Science Research Project of Ministry of Education (No. 18YJCZH006), as well as the Fundamental Research Funds for the Central Universities, Sun Yat-sen University (No. 23ptpy31).