SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To further reduce the computation burden, we leverage the sparsity of the violated ground formulas to filter ground formulas. Practically, we pre-compute the truth scores of all triples connected to the ground formulas using the KG-embedding model, then use a threshold to divide these triples into a "positive" set and a "negative" set. A ground formula is expected to be violated if all its premise atoms are positive and all its conclusion atoms are negative. By only involving the violated rules when computing potentials, we significantly reduce computational costs.
Joint reasoning with rules and embeddings After training, we get the trained embeddings and updated rule weights, which can be used to perform reasoning. There are two ways to compute the score: 1) compute each embedding score  $r_i(h_i, t_i; \theta)$  using the learned embeddings  $\theta$ ; 2) compute the cumulative rule score  $f_{rule}(h_i, r_i, r_i; W)$  by summing up weights of ground formulas that infer the target triple:
$$
f _ {r u l e} \left(h _ {i}, r _ {i}, t _ {i}; \boldsymbol {W}\right) = \sum_ {q = 1} ^ {m} W _ {q} \sum_ {j = 1} ^ {n _ {q}} \mathbf {1} _ {\left\{r _ {i} \left(h _ {i}, t _ {i}\right) \text {c a n b e i n f e r r e d b y} G _ {q} ^ {(j)} \right\}}.
$$
Table 1: Reasoning performance on real-world datasets.  $[\mathrm{T} = \mathrm{n}]$  means the maximum body length of mined rules is n. The best results are shown in bold and the second best are underlined. [NA] indicates that the model cannot finish inference within ten hours.
CodeX-s	CodeX-m	CodeX-1	WN18RR	YAGO3-10
MRR	hit@10	MRR	hit@10	MRR	hit@10	MRR	hit@10	MRR	hit@10
MLP	0.279	0.502	0.197	0.347	0.190	0.339	0.139	0.218	0.365	0.575
RotatE	0.421	0.634	0.325	0.466	0.319	0.453	0.469	0.566	0.495	0.670
TuckER	0.444	0.638	0.328	0.458	0.309	0.430	0.470	0.526	-	-
TransE	0.353	0.607	0.320	0.481	0.308	0.452	0.218	0.510	0.436	0.647
SACN	-	-	-	-	-	-	0.470	0.540	-	-
CompGCN	-	-	-	-	-	-	0.479	0.546	-	-
AMIE	0.195	0.283	0.063	0.095	0.026	0.029	0.36	0.485	0.25	0.343
NeuraLP	0.290	0.395	NA	NA	NA	NA	0.433	0.566	NA	NA
DRUM(T=2)	0.290	0.393	NA	NA	NA	NA	0.434	0.565	NA	NA
DRUM(T=3)	0.342	0.542	NA	NA	NA	NA	0.486	0.586	NA	NA
RNNLogic+	-	-	-	-	-	-	0.51	0.597	NA	NA
RLogic+	-	-	-	-	-	-	0.52	0.604	0.53	0.703
MLN4KB	0.082	0.134	0.035	0.045	0.028	0.032	0.368	0.374	0.460	0.525
pLogicNet	0.342	0.505	0.306	0.448	0.270	0.388	0.440	0.534	0.387	0.595
DiffLogic	0.445	0.662	0.335	0.487	0.326	0.448	0.493	0.585	0.503	0.673
DiffLogic+	0.458	0.655	0.343	0.495	0.337	0.46	0.50	0.587	0.513	0.674
To perform joint inference using both embedding scores and rule scores, a simple way is to use the weighted sum of embedding scores and the normalized rule scores:
$$
r _ {i} \left(h _ {i}, t _ {i}\right) = (1 - \eta) \cdot r _ {i} \left(h _ {i}, t _ {i}; \theta\right) + \eta \cdot \widehat {f} _ {i} \left(h _ {i}, t _ {i}; \boldsymbol {W}\right), \tag {8}
$$
where  $\widehat{f}_i(h_i, t_i; \mathbf{W})$  is the rescaled value of  $f_{rule}(h_i, r_i, t_i; \mathbf{W})$  calculated by minmax normalization. The optimum weight coefficient  $\eta$  is selected by using the validation set.
In this section, we conduct experiments to answer the following research questions. RQ1: Can DiffLogic outperform rule-based and embedding-based models in terms of reasoning performance? RQ2: Can DiffLogic scale to large knowledge graphs that rule-based models struggle to handle? RQ3: Does DiffLogic actually learn representations compatible with rules? RQ4: Quantify the efficiency and scalability of DiffLogic. RQ5: How effective is the model in terms of leveraging prior knowledge encoded in rules, compared with data-driven methods?
Datasets and candidate rules. We incorporate four real-world knowledge graph datasets: YAGO3-10, WN18, WN18RR, and CodeX (available in three sizes: small, medium, and large), along with a synthetic logic reasoning dataset: Kinship. Dataset statistics and descriptions can be found in Appendix B.1. Candidate rules for knowledge graphs are mined using AMIE3 (Lajus et al., 2020), with rule weights initialized by rule confidence scores.
Baseline models. Baseline models include four KG-embedding models — TransE (Bordes et al., 2013), RotatE (Sun et al., 2019), TuckER (Balažević et al., 2019), MLP (Dong et al., 2014), two GNN-based models — SACN (Shang et al., 2019), CompGCN (Vashishth et al., 2019), four rule-learning models — Neural LP (Yang et al., 2017), DRUM (Sadeghian et al., 2019), RNNLogic (Qu et al., 2020), RLogic (Cheng et al., 2022), a discrete MLN engine MLN4KB Fang et al. (2023), and a neuro-symbolic model pLogicNet (Qu & Tang, 2019) that also integrate KG-embedding and MLN. We exclude the recently proposed ExpressGNN (Zhang et al., 2020) from our experiments on KG experiments since it requires querying test data<sup>1</sup> during training and is inapplicable in our setting. For all baselines that employ a KG embedding model, we unify the negative sampling scheme as adversarial negative sampling for fair comparison. Hyperparameters for each baseline are taken from their original paper.
1Please see the discussion on the usage of test data at https://openreview.net/forum?id=rJg76kStwH.
4.1 Reasoning on real-world knowledge graphs
To answer RQ1, RQ2, and RQ3, we conduct link prediction tasks on several real-world datasets. Including WN18RR, YAGO3-10, and CodeX of three sizes (denoted as CodeX-s/m/l). Note that YAGO3-10 and Codex-l are large knowledge graphs that are suitable for assessing scalability. We report the performance of DiffLogic under two settings, using embedding scores for reasoning (denoted by DiffLogic) and using both embedding scores and rule scores for reasoning (denoted by DiffLogic $^{+}$ ). We evaluate the performance using the Mean Reciprocal Rank (MRR) and Hit@10, with the results presented in Table 1. We also investigate the evolution of violated rules and MRR on WN18RR during the inference of DiffLogic and RotatE, as demonstrated in Figure 3. These results lead to several key observations:
First, DiffLogic surpasses both rule-based and KG embedding-based methods. This can be primarily attributed to the fact that it combines the ability of both sides: 1) the ability to utilize embeddings to model similarities among entities, which enhance the reasoning performance; and 2) compared with data-driven KG-embedding models that only learn rule patterns from a large amount of labeled data, DiffLogic can explicitly leverage rule patterns within a principled logic reasoning framework, leading to better overall performance. Our results also indicate that jointly using rules and learned embeddings for reasoning is more effective than solely relying on embeddings. This suggests that rules and embeddings can complement each other during the reasoning process, ultimately leading to more accurate and robust inferences.
Second, DiffLogic outperforms pLogicNet, a neuro-symbolic model. Although pLogicNet also integrates MLN and KG-embeddings for reasoning, the key difference is that DiffLogic optimizes MLN and KG-embedding using a unified objective, whereas pLogicNet accommodates the discrete nature of MLN and essentially employs an MLN as a data augmentation technique to annotate additional facts for training KG embeddings. As a result, the optimization of MLN and KG embedding in pLogicNet is less consistent, and its performance is sensitive to the annotation threshold. Its performance is lower than TransE - its base KG embedding model on several large KGs (YAGO3-10/CodeX-l) because the annotated triples are mostly false positive. This highlights the advantages of DiffLogic in providing a more coherent and robust optimization process.
Third, DiffLogic demonstrates superior scalability compared to rule-based methods. DiffLogic is adept at efficiently scaling to expansive KG datasets like YAGO3-10 and CodeX-1. Conversely, rule-based methods, including NeuraLP, DRUM, RNNLogic, and MLN4KB, often encounter challenges when attempting to scale to such large KGs. An exception is RLogic, which is a scalable rule-learning model designed to mine complex and lengthy rules for reasoning. Remarkably, our model achieves comparable results by including only simply rules with a rule body length of  $\leq 2$ .
Furthermore, DiffLogic proficiently learns representations that align with both KG-embeddings and rules. As the left subfigure in Figure 3 demonstrates, the number of violated rules during the inference (embedding learning) phase is initially sparse, rises rapidly, and eventually decreases with training progression. We attribute this to 1) the random initialization of embeddings, which assigns low truth scores to most triples, resulting in fewer initial violations; 2) the model's training phase, where truth scores for training set triples increase but rule patterns are not yet fully captured, causing a rapid rise in rule violations; 3) as training progresses, KG embeddings begin to encapsulate rule patterns, reducing the number of violations. Importantly, despite its base embedding model being RotatE, DiffLogic is more effective than pure data-driven RotatE in capturing rule patterns, ensuring a consistent decrease in rule violations. As depicted in the middle subfigure in Figure 3, DiffLogic achieves faster convergence in test MRR compared to RotatE.
4.2 Scalability of optimization
To answer RQ4, we assess the scalability of our methodology, focusing primarily on optimization efficiency, i.e., the efficacy of grounding. We employ Kinship, a widely used (Zhang et al., 2020; Fang et al., 2023) synthetic benchmark dataset, to evaluate the efficacy and efficiency of RGIG in identifying important ground formulas. This dataset includes a training set, a test set, and a set of logic rules with full confidence. Some predicates are unobserved in the training set and can only be inferred via logic rules. The task is to deduce the gender of each individual in the test set, given the training set and the rule set. Kinship, designed for logical reasoning, necessitates resolving contradictions
Figure 3: Left: Violated rules evolution during inference on WN18RR. Middle: MRR evolution on WN18RR. Right: Number of considered ground formulas in PSL/ExpressGNN and RGIG.
among ground formulas, thus the reasoning performance heavily depends on the grounding process. Such a challenging dataset is suitable for assessing the efficacy of grounding techniques.
In our empirical evaluations, we discovered that three iterations of RGIG are enough to identify crucial ground formulas for accurate reasoning, yielding an AUC-ROC of  $0.982 \pm 0.014$  (detailed results are available in Appendix B.2.). We then compared the efficiency of RGIG with PSL and ExpressGNN's grounding techniques by examining the number of ground formulas considered for optimization across five different sizes of the Kinship dataset. Detailed calculations for PSL grounding can be found in Appendix A.3. The comparative results are depicted in the right subfigure in Figure 3. Although PSL and ExpressGNN can also perform accurate reasoning on Kinship, our model using RGIG has orders of  $(10^{3} \sim 10^{5})$  more data-efficient optimization process over varied sizes of Kinship dataset, demonstrating its applicability.
To further evaluate the efficiency of RGIG, we empirically test the run-time and memory overhead on real-world datasets. Results are presented in Table 2.
Table 2: Grounding overhead on real-world datasets. Run-time overhead is evaluated ten times and report mean and std.
Datasets	CodeX-s	CodeX-m	CodeX-1	WN18RR	YAGO3-10
Run-time(/sec)	0.03±0.00	0.38±0.01	0.87±0.04	0.54±0.01	3.20±0.04
Memory/(MB)	2.19	11.57	25.58	18.73	262.65
The experimental results demonstrate that RGIG can efficiently scale to accommodate large KGs while maintaining minimal run-time and memory overhead. On the largest KG, YAGO3-10, the grounding process is completed in approximately 3.2 seconds, using around  $262\mathrm{MB}$  of memory. In practice, we also observed that as the grounding iteration progresses, most inferred facts turn out to be negative triples. This trend could introduce noise and potentially impact efficiency. To enhance the efficiency of RGIG, one could consider filtering out facts with low scores by using a pre-trained RotatE model.
4.3 Learning from data vs. learning from rules
To answer RQ5, we design a "rule-pattern re-injection" experiment to evaluate DiffLogic's capability of injecting prior knowledge into embeddings, and compare it with pure data-driven based KG-embeddings. The design details are as follows:
The experiment contains two steps: 1) rule pattern removal, and 2) rule pattern re-injection. In our experiments, we use WN18 and select fourteen rules whose confidence scores are higher than 0.95. Then we deduplicate these rules so that no rules can be inferred from other rules, resulting in seven compact rules. In the rule pattern removal step, we use the selected seven rules to split the original training set, by finding all paths in the original training set that match the rules and extract the connected triples. The triples that match the conclusion part of the rules comprise the pattern set, and the original training set with the pattern set removed becomes the fact set. In this way, the generated fact set does not contain any pattern for the seven rules. In the rule pattern re-injection step, we add different ratios (0%, 10%, 20%, 100%) of the pattern set back into the fact set, so that rule patterns become increasingly evident. For DiffLogic, seven rules are applied to the training process for explicit rule pattern injection. We also include three KG-embedding models
for comparison. To fairly compare the rule-injection capability for embeddings, we use only the embedding scores of DiffLogic for evaluation. Experimental results are presented in Table 3.
Table 3: Comparison of pure data-driven training and rule injection by DiffLogic for embeddings.
Model	MRR	Hits@10
0%	10%	20%	100%	0%	10%	20%	100%
MLP	0.123	0.156	0.198	0.851	0.279	0.364	0.449	0.932
TransE	0.399	0.469	0.500	0.775	0.917	0.944	0.944	0.957
RotatE	0.579	0.890	0.927	0.944	0.784	0.949	0.961	0.962
DiffLogic	0.954	0.953	0.952	0.954	0.964	0.966	0.963	0.967
The results show a significant difference between the two rule pattern learning paradigms: DiffLogic can directly leverage explicit prior knowledge compactly encoded in rules, achieving significant improvement in reasoning performance by using only a small number of logical rules. On the contrary, pure data-driven based KG-embedding algorithms can only implicitly learn rule patterns from labeled data. By adding more data from the pattern set back to the fact set, the performance of data-driven algorithms increases as the rule patterns become more evident in the knowledge base. Nevertheless, these data-driven algorithms are not comparable to DiffLogic even though rule patterns are observable.
There have been some studies attempting to integrate rule-based methods and KG-embedding models. For example, Guo et al. (2016) proposes to learn embeddings from both triples and rules by treating triples as 'atomic formulas' while treating ground logic formulas as 'complex formulas', thus unifying the learning from triples and rules. However, their framework only uses hard rules and thus cannot make use of the soft rules with uncertainty. Another study (Guo et al., 2018) applied soft rules for generating additional training data but could not optimize rule weights, making the model's effectiveness dependent on rule weights initialization. Qu & Tang (2019) enable embedding learning and rule weight updating in a tailored MLN framework, by alternately employing one component to annotate triples to update the other component. However, the annotation process renders the inference not differentiable and is sensitive to the annotation threshold. Moreover, their grounding process only considers ground formulas with premise atoms observed in the training set, limiting the model's effectiveness due to the potential omission of important ground formulas. Zhang et al. (2020) designed a graph neural network for learning structure-aware expressive representations under the MLN framework. However, its inference efficiency suffers from a large number of ground formulas and limited generalization ability (requiring querying the test data during inference). The MLN-based neuro-symbolic models (Qu & Tang, 2019; Zhang et al., 2020) benefit from the dynamically updated rule weights which handle uncertainty. However, they all fail to directly optimize the objective of MLN due to the complexity of the associated integration and thereby resorting to optimizing ELBO.
In this paper, we aim to scale neuro-symbolic reasoning to large knowledge graphs with improved performance. We develop a differentiable model, namely, DiffLogic, that combines the advantages of KG-embedding models and rule-based models. DiffLogic directly optimizes the joint probability rather than the EBLO. The KG-embedding component enables linear scalability in representation, together with the grounding technique and the estimation technique for the gradient of rule weights, make DiffLogic efficient and scalable. Numerical simulations on benchmark datasets show the merits of DiffLogic. The performance of DiffLogic heavily depends on the quality of the rules. Was it possible to design an automatic and differentiable rule mining method, we may incorporate it into DiffLogic to further improve the performance. This could be future work.
A Notations and mathematical proofs
A.1 Notations
Table 4: Notations.
Notation	Description
K	The knowledge base
E	The entity set
R	The relation set
O, H	The set of observed and unobserved facts
x, y	The assignments of O and H, respectively
{Fq, Wq}m q=1	The set of logic rules and attached weights
Iq-, Iq+	The index set of premise atoms and conclusion atoms of rule Fq, respectively
A, B, ...	Variables in logic rules
{Gq(j), j ∈ tq}	All ground formulas created by the qth logic rule
Φq(y, x)	The sum of potentials of all ground formulas of Fq
θ	The embedding parameters
A.2 Derivation of rule weight gradient
$$
P _ {\boldsymbol {w}} ^ {*} (\boldsymbol {y} \mid \boldsymbol {x}) = \prod_ {i = 1} ^ {n} P ^ {*} \left(y _ {i} \mid \operatorname {M B} \left(y _ {i}\right), \boldsymbol {x}\right) = \prod_ {i = 1} ^ {n} \frac {\exp \left[ - f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right) \right]}{Z _ {i} \left(\boldsymbol {W} , y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right)},
$$
$$
Z _ {i} (\pmb {W}, y _ {i} \cup \pmb {y} _ {\backslash i}, \pmb {x}) = \int_ {y _ {i}} \exp \left[ - f _ {\pmb {w}} ^ {i} \left(y _ {i} \cup \pmb {y} _ {\backslash i}, \pmb {x}\right) \right], \quad f _ {\pmb {w}} ^ {i} = \sum_ {q = 1} ^ {m} W _ {q} \sum_ {j = 1} ^ {n _ {q}} \mathbf {1} _ {\{y _ {i} \to G _ {q} ^ {(j)} \}} d (G _ {q} ^ {(j)}),
$$
$$
\frac {\partial \log P ^ {*} (\boldsymbol {y} \mid \boldsymbol {x})}{\partial W _ {q}} = \sum_ {i = 1} ^ {n} \frac {\partial \log P ^ {*} \left(y _ {i} \mid \mathrm {M B} \left(y _ {i}\right) , \boldsymbol {x}\right)}{\partial W _ {q}}. \tag {9}
$$
The partial derivative in the left side of Eq. (9) is a summation of  $n$  terms, each term represents the partial derivatives of the pseudo-log-likelihood for each  $y_{i}$ , conditioned on its Markov blankets. Each term can be further simplified as follows:
$$
\begin{array}{l} \frac {\partial \log P ^ {*} (y _ {i} \mid \mathrm {M B} (y _ {i}) , \boldsymbol {x})}{\partial W _ {q}} \\ = \frac {\partial \left\{- f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right) - \log Z _ {i} (\boldsymbol {W} , y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}) \right\}}{\partial W _ {q}} \\ = \frac {\partial \left\{- f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right) - \log \int_ {y _ {i}} \exp \left[ - f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right) \right] \right\}}{\partial W _ {q}}. \\ \end{array}
$$
Here, we can easily get
$$
\frac {\partial f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right)}{\partial W _ {q}} = \sum_ {j} \mathbf {1} _ {\left\{y _ {i} \rightarrow G _ {q} ^ {(j)} \right\}} d \left(G _ {q} ^ {(j)}\right). \tag {10}
$$
To make the writing concise, we replace the right term of Eq. (10) with the following notation:
$$
\Psi_ {q, M B (i)} = \sum_ {j} \mathbf {1} _ {\{y _ {i} \rightarrow G _ {q} ^ {(j)} \}} d \left(G _ {q} ^ {(j)}\right).
$$
In this way, we can deduce that:
$$
\begin{array}{l} \frac {\partial \log P ^ {*} (y _ {i} \mid \mathrm {M B} (y _ {i}) , \boldsymbol {x})}{\partial W _ {q}} \\ = - \Psi_ {q, M B (i)} - \frac {1}{Z _ {i} \left(\boldsymbol {W} , y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right)} \frac {\partial \int_ {y _ {i}} \exp \left[ - f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right) \right]}{\partial W _ {q}}. \tag {11} \\ \end{array}
$$
The partial derivative and the integration in Eq. (11) can be swapped using Lebesgue's dominated convergence theorem, the Eq. (11) thus becomes:
$$
\begin{array}{l} \frac {\partial \log P ^ {*} (y _ {i} \mid \mathrm {M B} (y _ {i}) , \boldsymbol {x})}{\partial W _ {q}} \\ = - \Psi_ {q, M B (i)} - \frac {1}{Z _ {i} (\boldsymbol {W} , y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x})} \int_ {y _ {i}} \frac {\partial \exp \left[ - f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right) \right]}{\partial W _ {q}} \\ = - \Psi_ {q, M B (i)} + \int_ {y _ {i}} \frac {\exp \left[ - f _ {\boldsymbol {w}} ^ {i} \left(y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x}\right) \right]}{Z _ {i} (\boldsymbol {W} , y _ {i} \cup \boldsymbol {y} _ {\backslash i} , \boldsymbol {x})} \Psi_ {q, M B (i)} \\ = - \Psi_ {q, M B (i)} + \int_ {y _ {i}} P ^ {*} \left(y _ {i} \mid \operatorname {M B} \left(y _ {i}\right), \boldsymbol {x}\right) \Psi_ {q, M B (i)} \\ = - \Psi_ {q, M B (i)} + \mathbb {E} _ {y _ {i} | \mathrm {M B}} \left[ \Psi_ {q, M B (i)} \right]. \\ \end{array}
$$
Therefore, the partial derivative of pseudo-log-likelihood with respect to rule weight  $W_{q}$  is computed by:
$$
\frac {\partial \log P ^ {*} (\pmb {y} \mid \pmb {x})}{\partial W _ {q}} = \sum_ {i = 1} ^ {n} \left\{\mathbb {E} _ {y _ {i} | \mathrm {M B}} \left[ \sum_ {j} \mathbf {1} _ {\{y _ {i} \to G _ {q} ^ {(j)} \}} d (G _ {q} ^ {(j)}) \right] - \sum_ {j} \mathbf {1} _ {\{y _ {i} \to G _ {q} ^ {(j)} \}} d (G _ {q} ^ {(j)}) \right\}.
$$
A.3 Calculation of number of ground formulas for Kinship datasets
We present a detailed calculation of the number of ground formulas considered by PSL in Kinship datasets as follows.
- a first-order logical rule  $F_{q}$  containing  $|\mathcal{I}_q^-|$  premise atoms, and
- a knowledge base containing  $|\mathcal{E}|$  number of entities,
the number of variables in  $F_{q}$  is  $|\mathcal{I}_q^- | + 1$
PSL grounds each rule by substituting the variables with all possible entities. The number of ground formulas created by this logic rule  $F_{q}$  on the knowledge base is:
$$
\left| \mathcal {E} \right| ^ {\left| \mathcal {I} _ {q} ^ {-} \right| + 1}.
$$
Thus the overall ground formulas created by the rule set  $\{F_q\}_{q=1}^m$  is:
$$
\sum_ {q = 1} ^ {m} | \mathcal {E} | ^ {| \mathcal {I} _ {q} ^ {-} | + 1}.
$$
Given the statistics of Kinship datasets in Table 7, rules statistics are shared across different sizes of Kinship datasets, each dataset contains 12 rules that contain two variables and 9 rules that contain 3 variables. The number of ground formulas considered by PSL is thus computed by:
$$
1 2 \times | \mathcal {E} | ^ {2} + 9 \times | \mathcal {E} | ^ {3}. \tag {12}
$$
By applying the Eq. (12), we can get the ground formula number for each size of the Kinship dataset, as presented in Table 5:
Table 5: Number of ground formulas of Kinship datasets created by classical grounding method.
Kinship Size	S1	S2	S3	S4	S5
Number of ground formulas	1,373,601	10,853,976	35,798,376	74,671,320	172,162,935
B.1 Dataset statistics