Simple Augmentations of Logical Rules for Neuro-Symbolic Knowledge Graph Completion
Year: 2023

Abstract:
High-quality and high-coverage rule sets are imperative to the success of Neuro-Symbolic Knowledge Graph Completion (NS-KGC) models, because they form the basis of all symbolic inferences. Recent literature builds neural models for generating rule sets, however, preliminary experiments show that they struggle with maintaining high coverage. In this work, we suggest three simple augmentations to existing rule sets: (1) transforming rules to their abductive forms, (2) generating equivalent rules that use inverse forms of constituent relations and (3) random walks that propose new rules. Finally, we prune potentially low quality rules. Experiments over four datasets and five ruleset-baseline settings suggest that these simple augmentations consistently improve results, and obtain up to 7.1 pt MRR and 8.5 pt Hits@1 gains over using rules without augmentations.

heading: 1 Introduction
text: Knowledge Graphs (KGs) comprise important world knowledge facts, but are typically incomplete, due to their ever-increasing size. KG embeddings (Wang et al., 2017) has been the dominant methodology for knowledge graph completion (KGC). A KG embedding approach represents entities and relations as learnable dense vectors and computes a score for an unseen fact as a function over them. These generally have state-of-the-art performance, especially for large KGs.
Recently, neuro-symbolic (NS-KGC) approaches for the task have been proposed, where KG embeddings are enhanced by inferences over an explicit first-order logic rule set (Zhang et al., 2020; Qu et al., 2021). The resulting models bring together best of both worlds – generalizability and interpretability of explicit logical rules, and the scalability and representation power of embeddings. Unfortunately, a key roadblock for success of NS-KGC is the availability of a high-coverage rule set.
Early NS-KGC methods, such as NeuralLP (Yang et al., 2017) and DRUM (Sadeghian et al., 2019), learn rules as part of a single model, but do not have performance competitive with embedding models such as RotatE (Sun et al., 2019). A recent NS-KGC model, RNNLogic (Qu et al., 2021), matches empirical performance with embedding approaches. It has a separate neural component that outputs a set of rules, which is then used to train inference parameters, in an EM-based approach. Preliminary experiments on RNNLogic suggest that its ruleset has limited coverage, due to which symbolic inferences do not fire for many queries, and the model gets limited to using its embedding part only. The goal of this work is to strengthen the symbolic inferences in NS-KGC models for better overall performance.
In this work, we propose simple augmentations that take an existing ruleset (such as one output by RNNLogic) and proposes additional (related) rules to improve coverage and quality. We propose three augmentations. First, we convert each deductive rule into its abductive counterparts. Second, we supplement each rule via an equivalent rule that uses inverses for all constituent relations. Third, we generate additional high-quality rules independently by local random walks and subsequent PCA filtering (Galárraga et al., 2013). These increase size of ruleset drastically; we balance runtimes by additionally pruning rules from existing set using our filtering approach. Overall, this results in a comparable number of high-quality and high-coverage rules, for use in NS-KGC.
On four KGC datasets, over three NS-KGC models, we find that our augmentations consistently improve KGC performance, outperforming no augmentation baselines by up to 7.1 MRR and 8.5 Hits@1 pts. We believe our augmentations should become standard practice over any ruleset for NSKGC. We release our code 1 and rulesets.
1https://github.com/dair-iitd/NS-KGC-AUG
256

heading: 2 Background and Related Work
text: We are given an incomplete KG K = (E ,R, T ) consisting of entities E , relation set R and set T = {(h, r, t)} of triples. Our goal is to predict the validity of any triple not present in T . Related Work: Existing work on NS-KGC can roughly be characterized into four types. One approach is to use attention over relations to learn end-to-end differentiable models (Yang et al., 2017; Sadeghian et al., 2019). A second approach, which includes Minerva (Das et al., 2018) and DeepPath (Xiong et al., 2017), uses RL to train an agent to find reasoning paths for KG completion. These approaches are not yet competitive to KG embedding models for large datasets. Thirdly, models like ExpressGNN (Zhang et al., 2020) and RNNLogic use variational inference to assess plausibility of a given triple. We experiment with both these models in this paper. The final type includes UNIKER (Cheng et al., 2021) and RUGE (Guo et al., 2018), which integrate embeddings alongside traditional rules learnt via ILP models. We believe that our augmented rules can benefit these works too. Since our experiments are based on RNNLogic, ExpressGNN and we utilize PCA scores for filtering, we describe these in some detail next. RNNLogic+: As a pre-processing step, for every r ∈ R, RNNLogic adds a relation r−1 to R, and corresponding facts using inverse relations to T . RNNLogic first produces a set of first order rules (L) using an LSTM which are used by the RNNLogic+ predictor to compute the score of a given triple. Given a query (h, r, ?), the candidate answer o is scored by RNNLogic+ as:
scor(o) = MLP ( PNA({vl |#(h, l, o)}l∈L) ) (1) where the learnable embedding vl of a given rule l ∈ L is weighted by the number of groundings (#) that triple (h, r, o) satisfies in the rule l’s body. The resulting weighted embeddings of all rules are aggregated by employing PNA aggregator (Corso et al., 2020) and this aggregated embedding is passed through an MLP to obtain a final score.
The authors designed another scoring function that incorporates RotatE (Sun et al., 2019) into the scoring function, scor(o), in equation (1) where the goal is to exploit the knowledge encoded in the KG embeddings. The resulting scoring function is:
scoreKGE(o) = scor(o)+η RotatE (h, r, o) (2) where RotatE (h, r, o) is the score of the triple obtained from RotatE , and η is a hyper-parameter.
RotatE (h, r, o) is the negation of the value obtained by rotating the embedding for h by the rotation transformation defined by the embedding of r in complex space and computing the distance from the embedding of t. Please refer to Appendix B for further details.
ExpressGNN: It is a novel model that integrates Markov Logic Networks (MLN) (Richardson and Domingos, 2006) and Graph Neural Networks (GNN) (Kipf and Welling, 2017) to exploit their complementary strengths. An open-world paradigm is adopted in which a fact that is unknown in KG is assumed to be hidden (not false). The joint distribution of the observed and hidden triples of the KG in the MLN is optimized by employing a variational EM framework where the variational posterior distribution of the hidden variables is encoded as a GNN. Please refer to (Zhang et al., 2020) for further details about the model.
PCA Score: It is a symbolic rule confidence metric proposed in AMIE (2013) – see Appendix M for details. Broadly, it is the number of positive examples satisfied by a rule, divided by the total number of tails reached by the rule from heads occurring in the training dataset. Its performance in the context of AMIE was not as good due to its purely symbolic approach, and we are likely the first to show its utility in the context of NS-KGC.

heading: 3 Rule Augmentation in NS-KGC Models
text: With the aim of maximal utilization of a given rule l ∈ L, we first propose two rule augmentation techniques: abduction and rule inversion. The other two techniques prune low-quality rules from L, and independently add new rules to increase coverage. All augmentations are generic and can be integrated with any existing ruleset, and NS-KGC model.
Abduction: The goal of abductive reasoning (or abduction) is to find the best explanation from a given set of observations (Pierce, 1935). It has seen limited use in the context of KBs (Yoshikawa et al., 2019). In our approach, for every rule in L, we introduce several abductive rules with one of the antecedants, appearing as a consequent. As an example, consider the rule:
R1(X, Y) ∧ R2(Y, Z) ∧ R3(Z, W) ⇒ RH(X, W)
Our augmentation will generate abductive rules, one for each relation in the body, as:
R2(Y, Z) ∧ R3(Z, W) ∧ RH−1(W, X) ⇒ R1−1(Y, X) R3(Z, W) ∧ RH−1(W, X) ∧ R1(X, Y) ⇒ R2−1(Z, Y) RH−1(W, X) ∧ R1(X, Y) ∧ R2(Y, Z) ⇒ R3−1(W, Z)
As an example, let’s say a learned rule is BornIn(X, U) ∧ PlaceInCountry(U, Y) ⇒Natio nality(X, Y). If in the KG, we know that Oprah has nationality U.S., and that she is born in Mississippi, then abduction allows the model to hypothesize that Mississippi might be in U.S. Of course, not all abductions are accurate, for instance, just because Alabama is known to be in U.S., does not mean that Oprah was born in Alabama. Abductive rules increase rule coverage at the cost of precision. We expect the predictor scorer to automatically handle which (abductive) rules can and cannot be trusted. Rule Inversion: Our second rule augmentation takes an existing rule and rewrites it by referring to inverses of all relations. As an example, if a rule uses the path Oprah BornIn−−−−−→ Mississippi PlaceInCountry−−−−−−−−−−→ US, then it could also use the equivalent path US PlaceInCountry −1 −−−−−−−−−−−−→ Mississippi BornIn −1
−−−−−−→ Oprah. Formally, for every original rule:
R1(X, Y) ∧ R2(Y, Z) ∧ R3(Z, W) ⇒ RH(X, W)
we add to the ruleset the following inverted rule:
R3−1(W, Z) ∧ R2−1(Z, Y) ∧ R1−1(Y, X) ⇒ RH−1(W, X)
Rule Filtering: Augmentations increase the size of the ruleset. In order to reduce the number of parameters and the training/test times of the NSKGC model, we prune seemingly low-quality rules from the augmented rulebase. For this, we compute the PCA score for each original and augmented rule and prune all the rules that have score less than a threshold (set at 0.01 in experiments) and have less than 10 groundings. So, all low-coverage rules with seemingly low quality are pruned out. As experiments show, this results in up to 70% reduction in the number of rules, while preserving KGC performance. Random Walk Augmentation: Motivated by the empirical success of PCA scores for finding good rules in the previous step, we further augment our ruleset with new, high scoring rules generated independently via local random walks. Starting at each entity in the KG, we perform a number of random walks of fixed length. Each such random
walk constitutes the body of the rule and the relation connecting the end entities in the KG form the head of the discovered rule. We score these rules by the PCA score and retain all such rules that have PCA score above the threshold (of 0.1).

heading: 4 Experiments
text: Datasets: We use four datasets for evaluation: WN18RR (Dettmers et al., 2018), FB15K237 (Toutanova and Chen, 2015), Kinship and UMLS (Kok and Domingos, 2007). For each triple in test set, we answer queries (h, r, ?) and (t, r−1, ?) with answers t and h. We report the Mean Reciprocal Rank (MRR) and Hit@k (H@1, H@10) under the filtered measures (Bordes et al., 2013). Details and data stats are in Appendix A. Baselines: We first experiment with two base models: RNNLogic+ ([RNN] in tables), and RNNLogic+ with RotatE ([RNN+RotE]) (Eqn 2). We have reproduced the numbers published by the original authors for these models (details in Appendix D). We run these models with two rulesets: (1) Orig, rules generated by RNNLogic (around 300 rules per relation for WN18RR and FB15k-237, and 1000 rules per relation for Kinship and UMLS), and (2) RW, only the rules discovered by our random walks. This second setting can only evaluate the value of abduction, inversion, and pruning since random walks are anyways used in generating rules. More details in Appendix C, F and G.
In order to assess the generality of our augmentations, we also experiment with ExpressGNN (Zhang et al., 2020). We choose top five rules for each relation from RNNLogic’s Orig ruleset according to PCA confidence and provide them as input ruleset to ExpressGNN ([ExpGNN] in tables). ExpressGNN does not scale up to the augmented ruleset for FB15K-237, hence we test it for the other three datasets. Refer to Appendix E for more details. We use AUG to denote the performance of rule augmentations for all baselines.
We also tried rulesets from NeuralLP (2017), but they are too small to be useful with RNNLogic+. The only other NS-KGC model that has reported performance similar to RNNLogic+ is RLogic (2022). Unfortunately, their code is not publicly available.2
Results: We report the results in Table 1 for the RNNLogic baselines (further details in Appendix
2Our reimplementation could not match reported results, and sending several emails to original authors was not helpful.
H). We observe that in all settings, there is a notable increase in performance using augmented rules. In particular, we obtain 7.1 pt and 8.5 pt increase in MRR and Hits@1 in [RNN]-(Orig) setting on Kinship, and 3.5 pt and 5.6 pt increase in MRR and Hits@10 in [RNN]-(RW) setting for WN18RR dataset. We also find that rule augmentations complement RotatE scores in capturing more information about the KG, leading to improved performance in those settings too. To the best of our knowledge, our best results for WN18RR are state-of-the-art for NS-KGC models.
Next, we present the results of our proposed augmentations with ExpressGNN3 as baseline in Table 2. We note that ExpressGNN assumes the knowledge of test queries while it constructs the MLN during training. Therefore, the results presented in Table 2 are not directly comparable with the results of other models presented in the paper, which do not make this assumption. We observe substantial gains on all datasets and all metrics, notably a 22.4 pt MRR, 17.9 pt Hits@1 and 29.9 pt Hits@10 improvement on WN18RR dataset with our augmentations (AUG). This experiment demonstrates that AUG can help other neuro-symbolic settings as well. Refer to Appendix E for more details.
parable to results in Table 1

heading: 5 Analysis of Augmented Rules
text: We perform five further analyses to answer the following questions. Q1. Are the rules created by abduction and rule inversion of high quality? Q2. What is the individual effect of each type of augmentation on the performance? Q3. How do the rule augmentations affect the training time of a model? Q4. Can we get the same performance as augmentation by generating more rules from the LSTM in RNNLogic? Q5. Are the augmented rules interpretable by a human? Quality of New Rules: To answer Q1, we employ two metrics to assess quality of rules, (PCA-metric and FOIL-metric) before and after abduction and rule inversion. The rules obtained from random walks have high scores by construction since they are filtered based on PCA score. Therefore, they are of high quality as per our definition. (Details in Appendix M and N)
Table 3 presents the number of rules that have a score of at least 0.1 according to each metric, which we regard as criteria for defining a high-quality rule. We observe that there is a large increase in the number of high-quality rules after abduction and rule inversion, nearly tripling in the case of abduction (row 1 vs row 3). This is because the augmented rules exploit the same groundings as the original rules, in the form of new rules. Thus, augmented counterparts of high-quality rules are likely to be high-quality. Overall, we find that abduction and rule inversion does indeed produce high-quality rules. Ablation: To answer Q2, we perform an ablation
study for inversion (INV), abduction (ABD), random walk augmentation (RW) and rule filtering (FIL) on [RNN+RotE]-(Orig) setting for WN18RR and Kinship datasets to observe the impact of each type of augmentation. The results are presented in Table 4 (further details are in Appendix I).
In general, abduction (row 3) gives larger improvements than rule inversion (row 2) because as we noticed in the previous section, abduction adds a larger number of high-quality rules to the rule set. We also find that adding the PCA-based random walk rules results in performance improvement, even with only 5% new rules being added (as in Kinship) as compared to original rule set. Finally, we find that filtering based on the PCA metric results in marginal performance improvement, along with lower running times (see below). Performance vs Training Time Trade-off: To answer Q3, we report training time per epoch (in minutes), size of ruleset and performance metrics after augmentation through ABD, INV and RW (denoted as AUG) and filtering (AUG + FIL) with [RNN + RotE] as the baseline model in Table 5.
Our proposed augmentations (INV, ABD and RW) result in substantial performance gains, at the cost of 5-6 times increase in the training time. After filtering (FIL), there is no decrease in performance, and the training time goes down by 2-3× compared to AUG. Therefore, we obtain substantial performance gains through our augmentations, at the cost of only 2-3 times increase in training time.
Rule Generation vs Rule Augmentation: Our augmentations result in 100-200% increase in the number of rules across datasets after filtering. As a
control experiment to answer Q4, we train RNNLogic to generate 80 rules per relation (R/R) and augment resulting rules without filtering (for a fair comparison). We further train RNNLogic with 500 rules per relation without augmentation and compare performance of both rulesets (which now have comparable size) using [RNN+RotE] on WN18RR and Kinship in Table 6 (see Appendix J).
We observe that rule augmentations lead to large improvement over rule generation in all cases, even when rule generation creates more rules. Thus, we find that rule augmentation is more beneficial than simply using more rules from the rule generator. Augmentations exploit a small number of highquality rules to their full potential. Qualitative Analysis: To answer Q5, we randomly sample 50 rules from the Orig and RW rules for the FB15K-237 dataset and score them as 0 (gibberish), 1 (logically dubious but statistically plausible) and 2 (logically correct) for each ruleset. The reported numbers are averages of scores obtained from two human annotators. We do not include INV and ABD in this comparison as they are generated from Orig rules utilizing the same groundings and thus we expect them to be as interpretable. The scores are 0.90 (Orig) and 1.23 (RW). RW rules are more interpretable due to their high PCA scores. One example of an interpretable rule added by RW is Friends(A, C), Inverse_Producer(C, D), Writer(D , B) :- Friends(A, B). We provide additional rule examples for each type of augmentation in Appendix K.

heading: 6 Conclusion and Future Work
text: We present simple rule augmentation techniques in the context of Neuro-Symbolic Knowledge Graph models and obtain substantial increase in performance over strong base models. We believe our augmentations can become standard for all subsequent NS-KGC models. We release code and rulesets for further research. Future work includes using our augmentation technique during the iterative learning of rules in algorithms such as RNNLogic, potentially further improving their performance.

heading: Acknowledgements
text: This work is supported by grants by Google, IBM, Verisk, and 1MG, and the Jai Gupta chair fellowship by IIT Delhi. We also acknowledge travel support from Google travel grant. We thank the IIT Delhi HPC facility for its computational resources.
Limitations
Since rule abduction and inversion utilize the same groundings as the original rules, Neuro-Symbolic KGC models that are based on grounding the entire rule will not benefit from these augmentations. Abduction and inversion also require the model to be trained on a knowledge graph that contains the inverse relations r−1 for each relation r. Finally, since RNNLogic+ has a separate rule embedding for each rule, performing rule augmentation increases the number of parameters in the model and leads to longer training times and larger GPU memory consumption.
Ethics Statement
We anticipate no substantial ethical issues arising due to our work on rule augmentation for NeuroSymbolic KGC. Our work relies on a set of rules generated from another source to perform augmentation. This may result in the augmented rule set exaggerating the effect of malicious or biased rules in the original rule set.

heading: Acknowledgements
text: This work is supported by IBM AI Horizons Network grant, grants by Google, Verisk, and 1MG, an IBM SUR award, and the Jai Gupta chair fellowship by IIT Delhi. We acknowledge travel support by Google and Yardi School of AI travel grants. We thank the IIT Delhi HPC facility for its computational resources.

heading: A Data Statistics and Evaluation Metrics
text: Table 7 summarizes the statistics of the data used in the experiments of our work. We utilize the standard train, validation and test splits for WN18RR and FB15k-237 datasets. Since there are no standard splits for UMLS and Kinship datasets, for consistency, we employ the splits used by RNNLogic (2021) for evaluation (created by randomly sampling 30% triplets for training, 20% for validation and the rest 50% for testing).
Metrics: For each triplet (h, r, t) in the test set, traditionally queries of the form (h, r, ?) and (?, r, t) are created for evaluation, with answers t and h respectively. We model the (?, r, t) query
as (t, r−1, ?) with the same answer h, where r−1 is the inverse relation for r. In order to train the model over the inverse relations, we augment the training data with an additional (t, r−1, h) triple for every triple (h, r, t) present in KG.
Given ranks for all queries, we report the Mean Reciprocal Rank (MRR) and Hit@k (H@k, k = 1, 10) under the filtered setting in the main paper and two additional metrics: Mean Rank (MR) and Hits@3 in the appendices. MRR and Hits@k metrics are reported after multiplying with 100. To maintain consistency with RNNLogic, in cases where the model assigns same probability to other entities along with the answer, we compute the rank as (m + (n+1)2 ) where m is the number of entities with higher probabilities than the correct answer and n is the number of entities with same probability as the answer.

heading: B RotatE
text: RotatE is a knowledge graph embedding model that embeds entities and relations in complex space. Relation embeddings are modeled as rotations in complex vector space. Formally, RotatE(h, r, t) is calculated using the following equation:
RotatE (h, r, t) = −d(xh ◦ xr, xt) (3)
where d is the cosine distance in complex vector space, RotatE embedding of r is xr, and ◦ is the Hadamard product. Intuitively, we rotate xh by the rotation defined by xr and consider the distance between the result and xt. For our experiments, RotatE is trained separately and the trained embeddings are used to calculate scores for the [RNN + RotE] baseline.

heading: C Experimental Setup for RNNLogic
text: In order to obtain main results in Table 1, we train the rule generator in RNNLogic with optimal hyperparameters obtained after communication with the original authors and generate a set of high-quality Horn rules to use for training RNNLogic+. For our best results, we utilize optimal rules provided by the authors of RNNLogic5. We augment these rules by abduction (ABD), and then rule inversion (INV) on both the original rules and the rules formed after abduction. We further augment the rulebase with the rules discovered by random walks (RW). Finally, we filter (FIL) superior rules from these rules by
5https://github.com/DeepGraphLearning/RNNLogic
PCA score. We present statistics detailing the number of rules used per dataset after each augmentation step in Table 8. These rules are utilized in RNNogic+ ([RNN]-(Orig)) and RNNLogic+ with RotatE ([RNN+RotE]-(Orig)) baselines. For the other results: [RNN]-(RW) and [RNN+RotE]-(RW), we employ only the rules obtained by RW augmentation and train RNNLogic+ model with them (Appendix F). The goal of these set of results is to test the utility of abduction and rule inversion with a different set of rules. The details of training RNNLogic+ model is provided in Appendix G.

heading: D RNNLogic Results Reproduction
text: We have reproduced the results of RNNLogic+ with and without RotatE and obtained similar results to the original RNNLogic paper (Qu et al., 2021), however the numbers reported in this paper for [RNN] and [RNN + RotE] are our own reproductions. In this section, we report a comparison between the original results and our reproduced results for RNNLogic+ model ([RNN]) on the WN18RR and FB15K-237 datasets. As can be observed in the Table 12, our reproduced results are better than the published results of RNNLogic model for both the datasets, using hyperparameters obtained after communication with the authors of the RNNLogic paper.

heading: E ExpressGNN Training and Hyperparameter Setting
text: As already discussed in Section 4, in order to prove the broad applicability of proposed augmentations (AUG) in our work, we perform experiments with ExpressGNN model as another baseline in Table 2. In this section we provide the details of this experiment. The current implementation of ExpressGNN model scales poorly with the number of rules, necessitating the use of a much smaller ruleset size. We generate ruleset for each dataset by selecting the top 5− 10 rules per relation (in the rule head) from RNNLogic rules for that dataset (ORIG) based on the PCA score. This results in 417 rules for WN18RR, 500 rules for Kinship and 460 rules for
UMLS. We perform augmentations on these rules and further maintain a threshold of the PCA score to be 0.95 while filtering RW rules. After augmentation, we obtain 1734 rules for Kinship, 2058 rules for UMLS and 828 rules for WN18RR. We also augment the training and the test set of ExpressGNN datasets with the inverse triples (t, r−1, h) for each original (h, r, t) triple. Hyperparameters used for training are the optimal ones from the original paper. Results for FB15k-237 are omitted since ExpressGNN does not scale up to the augmented ruleset.
ExpressGNN assumes the knowledge of test queries at training time to construct its Markov Logic Network. For the test triple (h, r, t), this informs the model that h is a potential head and t is a potential tail entity for given relation r, even though this information might not be present in the training data. Hence, results presented in Table 2 are not directly comparable to results in Table 1.

heading: F Rule Generation via Random Walks
text: Because rules generated by employing random walks form a distinct ruleset in the main paper ([RNN]-(RW)), we explain the statistics of these rules in detail in a dedicated section here. In order to determine the number of rules generated from the random walks, we calculate the difference of the column ‘#Rules + INV + ABD’ and ‘#Rules + INV + ABD + RW’ in the Table 8 and summarize the resulting statistics of the number of RW rules created for each dataset in the Table 13. When compared to Table 8, we note that although random walk rules (RW) comprise less than 8% of the augmented ruleset for all the datasets, these rules are still pivotal. This is because we notice a considerable decrease in performance after removing these rules as observed in Table 4, Table 10 and Table 11.

heading: G RNNLogic+ Training and Hyperparameter Setting
text: Here we describe the training of RNNLogic+ model that is utilized in Table 1 and complementary Table 9. We use the same methodology for training RNNLogic+ model as in the original work (Qu et al., 2021). New rule embeddings are created for
all the rules that are added to the rule set after rule augmentation. Rule embedding dimension is set to 16 (compared to 32 in original RNNLogic+) across datasets to mitigate the effect of the increased number of parameters in the model due to new rule embeddings. Results reported are for a single run with fixed seed over 5 epochs of training.
The hyperparameter η in Equation (2) representing the relative weight is set to 0.01, 0.05, 0.1 and 0.5 for WN18RR, FB15k-237, UMLS and Kinship respectively. The RotatE embedding dimension is set to 200, 500, 1000 and 2000 for WN18RR, FB15k-237, UMLS and Kinship respectively. We keep a consistent batch size of 8, 4, 32 and 16 for WN18RR, FB15k-237, UMLS and Kinship respectively. The number of parameters for RNNLogic+ scales with the rule embedding size and the number of rules, reaching a maximum of 16*298446 = 4775136 for FB15k-237 after augmentations and filtering (leading to a training time of around 23 hours). As we can see, augmentation adds new rules leading to increase in the parameters of the model. All training was carried out on a single Tesla V100 GPU. The optimal values of all the hyper-parameters was found by tuning on validation set on each dataset.

heading: H Detailed Results on Proposed Augmentations
text: Results in Table 9 are supplementary to results already presented in Table 1. In addition to MRR, Hits@1 and Hits@10 presented in the Table 1 in the Experiment section, we also present Mean Rank (MR) and Hits@3 here. As discussed already in Section 4, AUG includes abduction (ABD), inversion (INV), rule filtering (FIL) and random walk augmentation (RW).
In Table 9, we observe that there is a consistent improvement in the performance of the model for all the metrics after rule augmentation and filtering (AUG). Notably, for the two new metrics introduced in Table 9, we obtain a performance gain of 3.7 point on Hits@3 and 40.4% on MR for FB15K-237 dataset and [RNN]-(RW) baseline. Since the original rules for the random walk baseline are lesser in number, [RNN]-(RW) and [RNN + RotE] - (RW) benefit more from augmentation. We also observe that for Kinship and UMLS, [RNN + RotE] - (RW) gives better performance than [RNN + RotE] - (Orig), highlighting the quality of the rules discovered by local random walks followed by PCA filtering.

heading: I Detailed Results of Ablation Study
text: Results in Table 10 are supplementary to results already presented in Table 4. Besides the three metrics presented in Table 4, we present Hits@3 and MR in this table. Additionally, we also demonstrate results of ablation on WN18RR dataset in Table 11. Ablation is not performed on FB15K-237 due to computational constraints. As with the other metrics, Hits@3 and MR is the most affected by abductive rules in UMLS and WN18RR because abduction results in augmenting the ruleset with a large number of high-quality rules (see Table 3). Furthermore, Hits@3 and MR gets most affected by PCA-based random walk augmentation in Kinship dataset. This is because Kinship is a dense dataset, and a large number of high-quality rules are quickly discovered by the random walks.

heading: J Detailed Results of Rule Generation vs Rule Augmentation
text: Results in Table 14 are supplementary to the results already presented in Table 6. Here we present Hits@3 and MR as two additional metrics for analyzing the need for rule augmentation.
We generate rules by training RNNLogic model. We consider 80 rules per relation for each dataset from these rules and expand them by performing three augmentations and filtering. This results in total of 9867 rules for WN18RR and 18432 rules for Kinship data. Then, we train RNNLogic+ with RotatE ([RNN+RotE]) on these rules and compare the results with RNNLogic+ with RotatE model trained on 500 rules per relation without augmentations. We observe that model trained with augmented rules consistently performs better than model trained by merely increasing the number of rules generated, even for a comparable number of rules. Specifically, we observe that model trained with augmented rules shows 4 point Hit@1 gain in Kinship dataset over the model trained with merely increased rules. These results strengthens the hypothesis that it is more helpful to leverage a few high-quality augmented rules rather than exploiting a plethora of lower-quality rules for NeuroSymbolic KG Completion.

heading: K Qualitative Analysis of the Augmented Rules
text: In this section, we present one logical rule generated after each augmentation step as examples. The rules are taken from the FB15K-237 dataset.
1. ABD: LivesIn(PersonA, LocationB) : − PlayFor(PersonA, TeamC), Inverse_Team _Location(TeamsC, LocationB)
2. INV: Inverse_Person_Language(Langu ageA, PersonB) : − Inverse_Film′s_Lang uage(LanguageA, FilmC), StoryWritten By(FilmC, PersonB)
3. RW: Friends(PersonA, PersonB) : − Friends(PersonA, PersonC), Inverse_ Producer(PersonC, FilmD), Writer(FilmD , PersonB)
For example, the rule in ABD category states that a person will live in the same city as the team he plays for is located. Therefore, we conclude that the rules captured through augmentations can be human interpretable.

heading: L An Alternative Augmentation Strategy
text: Recall that in our proposed methodology (Orig+AUG) in Section 3, we consider original rules (ORIG) and perform abduction (ABD) on the original rules. This is followed by rule inversion (INV) over the original rules and abductive rules. Then, we introduce the random walk rules (RW) as the final augmentation step in the proposed augmentations (AUG) for the original (Orig) ruleset. In this section, we consider an alternative sequence of augmenting the ruleset where we consider both the original (Orig) and the random walk rules (RW) and apply abduction and rule inversion on both of them. We denote this setting as (Orig + AUG2). We report a comparison of (Orig + AUG2) with (Orig + AUG) (Table 1) with [RNN + RotE] as the baseline model in Table 15. From the results in the table, we conclude that Orig + AUG2 does not result in improvement over our original methodology of Orig + AUG. It also creates a larger ruleset, further slowing down the training of the model.

heading: M PCA-Confidence Metric
text: In this section, we explain in detail, the PCAconfidence metric that has been employed to score the rules discovered through random walk in our third augmentation approach. This metric has also been used to score the augmented rules in Table 3.
PCA: The calculation of the metric utilizes a Partial Closed World assumption (Galárraga et al., 2013) and assumes that if we know one t for a given r and h in r(h, t), then we know all t′ for that h and r. Let the rules under consideration be of the form B ⇒ r(h, t). Then the PCA-score PCAConf(B ⇒ r) is:
#(h, t) : |Path(h, B, t)| > 0 ∧ r(h, t) ∈ P #(h, t) : |Path(h, B, t)| > 0 ∧ ∃t′ : r(h, t′) ∈ P
Essentially, it is the number of positive examples, P, satisfied by the rule divided by the total number of (h, t) satisfied by the rule such that r(h, t′) is a positive example for some t′.

heading: N FOIL-Score Metric
text: We employ a modification of FOIL as one of the evaluation metrics to assess the quality of rules produced by augmentation techniques (Q1) in Table 3. FOIL-scoring metric is discussed in detail below. FOIL: Let the rules be of the form B ⇒ r(h, t). Let Path(h, B, t) be the set of paths from h to t that act as groundings for the rule body B. Under the Closed World assumption, we assume that all triples not in the training and test set are false. Inspired by the First-Order Inductive Learner algorithm (Quinlan, 1990), we define FOIL score to
assess the quality of a rule as follows:
FOIL(B ⇒ r) = ∑
r(h,t)∈P |Path(h, B, t)|∑ (h,t) |Path(h, B, t)|
In the above equation, P represents the set of positive examples in the given KG. The key difference between the FOIL score proposed originally (Quinlan, 1990) and ours is that instead of considering the number of examples satisfied by the rule, we calculate the number of groundings of the rule. This is more in line with the score calculated by RNNLogic+, which considers the number of groundings as well. Ideally the rules should have larger number of groundings for positive triples in comparison to the other triples.
Typically, negative sampling is used to calculate these metrics (PCA in Appendix M and FOIL here) as it is computationally expensive to compute exhaustive negative examples. However, we calculate these metrics by considering the entire knowledge graph, which is enabled by utilizing batching and sparse matrix operations on the adjacency graph.
We highlight that we are the first to show the utility of PCA Confidence and FOIL in the context of neuro-symbolic models. This makes our specific approach distinct from AMIE (Galárraga et al., 2013) and FOIL (Quinlan, 1990), and more targeted to our setting due to the changes in the method of computation.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section or justification Section Limitations, Page number 6
3 A2. Did you discuss any potential risks of your work? Section Ethics Statement, Page number 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section Abstract and 1 (Introduction), Page number 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? Section References, Page number 6 and Appendix C, Page number 7
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We have used publicly available code and rule files released by the authors of RNNLogic on Github, which has not been explicitly licensed. We have mentioned source of rule files in Appendix C, Page number 7.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The authors of the code for RNNLogic and ExpressGNN have not explicitly stated their intended use of code on Github. They only require potential users to cite their paper if they use the code, which we have done.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? All the datasets that we have used in our experiments are standard datasets and we have cited the creators for each one of them.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We have not created any new artifacts through our work. We have provided original and augmented rule sets used in our experiments in the submitted code.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A, Page number 7
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix G, Page number 10
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix G, Page number 10
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix G, Page number 10
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix D, Page number 9
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work includes using our augmentation technique during the iterative learning of rules in algorithms such as RNNLogic, potentially further improving their performance