Logical Entity Representation in Knowledge-Graphs for Differentiable Rule Learning
Authors: Chi Han
Qizheng He
Charles Yu
Xinya Du
Hanghang Tong
Heng Ji
Venue: ICLR 2023 poster (2023)
Keywords: Probabilistic Logical Rule Learning
Knowledge Graph Completion
Logical Representation Learning

Abstract: Probabilistic logical rule learning has shown great strength in logical rule mining and knowledge graph completion. It learns logical rules to predict missing edges by reasoning on existing edges in the knowledge graph. However, previous efforts have largely been limited to only modeling chain-like Horn clauses such as R1(x; z) ^ R2(z; y) ) H(x; y). This formulation overlooks additional contextual information from neighboring sub-graphs of entity variables x, y and z. Intuitively, there is a large gap here, as local sub-graphs have been found to provide important information for knowledge graph completion. Inspired by these observations, we propose Logical Entity RePresentation (LERP) to encode contextual information of entities in the knowledge graph. A LERP is designed as a vector of probabilistic logical functions on the entity’s neighboring sub-graph. It is an interpretable representation while allowing for differentiable optimization. We can then incorporate LERP into probabilistic logical rule learning to learn more expressive rules. Empirical results demonstrate that with LERP, our model outperforms other rule learning methods in knowledge graph completion and is comparable or even superior to state-of-the-art black-box methods. Moreover, we find that our model can discover a more expressive family of logical rules. LERP can also be further combined with embedding learning methods like TransE to make it more interpretable.

Published as a conference paper at ICLR 2023
LOGICAL ENTITY REPRESENTATION IN KNOWLEDGE- GRAPHS FOR DIFFERENTIALLE RULE LEARNING
Chi Han', Qizheng He*, Charles Yu', Xinya Du", Hanghang Tong', Heng Ji University of Illinois Champaggn, The University of Texas at Dallas {chihan3, qizheng6, ctyu2, htong, hengjijei wwwiiiioiiiiiiimo edu xinya. dulutaalass.........aaaaaaaallaa
ABSTRACT
Probabilistic logical rule learning has shown great strength in logical rule min- ing and knowledge graph completion. It learns logical rules to predict missing edges by reasoning on existing edges in the knowledge graph. However, previous efforts have largely been limited to only modeling chain-like Horn clauses such R2 (x, 2) A y) + (x,,y). This formulation overlooks additional con- textual information from neighboring Intuitively, there is a large gap here, as local sub- -abbgrraps have been found to pro- vide important information for knowledge graph completion. Inspired by these observations, we propose Logical Entity RePresentation (LERP) to encode con- textual information of entities in the knowledge graph. A LERP is designed as vector of probabilistic logical functions on the entity's neighboring sub-graph. It is interpretable representation while allowing for differentiable optimization. We can then incorporate LERP into probabilistic logical rule learning to learn more expressive rules. Empirical results demonstrate that with LERP, model outperforms other rule learning methods in knowledge graph completion and is comparable even superior state-of-the-art black-box methods. Moreover, we find that our model can discover a more expressive family of logical rules. LERP can also be further combined with embedding learning methods like TransE make it more interpretable.
of entity variables
and a.
INTRODUCTION
In recent years, the of logical formulation has become prominent in knowledge graph (KG) reasoning and completion (Teru et al.) 2020: Campero et al. 2018 Payani & Fekri 2019). mainly because logical formulation can used to enforce strong prior knowledge on the reasoning pro- cess. In particular, probabilistic logical rule learning methods (Sadeghian et al.) 2019 Yang et al., 2017 have shown further desirable properties including efficient differentiable optimization and ex- plainable logical reasoning process. These properties are particularly beneficial for KGs since KGs are often large in size, and modifying KGs has social impacts so rationales are preferred by human readers.
Due to the large search space of logical rules, recent efforts (Sadeghian et al. (2019): Yang et al. (007) Payani& & Fekri (2019) focus on learning chain-like Horn clauses of the following form:
ra (1 21) (2(21, 22) A ......AAkkie Ark
H(x,)),
where Fh represents relations and æ, y 24 represent entities in the graph. Even though this for- mulation is computationally efficient (see Section 3). it overlooks potential contextual information coming from local sub-graphs neighboring the entities (variables x, y, and all 2;). However, this kind of contextual information can be important for reasoning on knowledge graphs Figure [[b) shows an example. If we only know that is mother of x and y, we are not able infer if y a brother or sister of ær. However, in Figure (c), with the contextual logical information that 32' is son of x') we can infer that y is a male and that should be the brother rather than the sister of x.
All code and data publicly available httpa:///gtthaaeeeee com/ aci www.nd/o.nnaaooo I.ERP

---

brother of Published as a conference paper at ICLR 2023
Facts: 2 is the mother of and y {is mother poother_oft,,> x), is_ mother Soother_of(a.. ofte,
Extra contextual fact:
Z
is son of
LGy) 3z' is__so son ofly
is mother of
is mother of
Question: is the brother or sister of x? (a) link prediction query
X
Z.
Prediction:
is mother of Z. is_oherher of is _issiitterrr of?
is mother of
Logical rule:
x
is mother of(z.x) A is mother Soother_off.,11 y) 3d' iis son of is brother ofly,))
(b) without contextual information
(c) with logical contextual information
Figure I: Incorporating contextual information of entities can benefit missing link prediction.
Recent deep neural network models based on Graph Neural Networks (GNNs) (Teru et al. 2020 Mai et al. 202I) have utilized local sub-graphs as an important inductive bias in knowledge graph completion. Although GNNs can efficiently incorporate neighboring information via message- passing mechanisms to improve prediction performance (Zhang et al., 2019. [in et al. (022)). they not capable of discovering explicit logical rules, and the reasoning process of GNNs largely unexplainable.
In this paper, we propose Logical Entity RePresentation (LERP) incorporate information from lo- cal sub-graphs into probabilistic logic rule learning. LERP is a logical contextual representation for entities in knowledge graph. For entity e, a LERP [(e) is designed
vector of logical func- tions (e) on e's neighboring sub-graph and the enclosed relations. We then incorporate LERP in probabilistic logical rule learning methods to provide contextual information for entities. Dif- ferent from other embedding learning methods, LERP encodes contextual information rather than identity information of entities. In the example discussed above Figure [ an ideal LERP for y might contain logical functions like Li(y) 3z' is is_son_ofly, z'). Therefore, when predicting the relation br_brother_ar Lof, the rule learning model can select L; from LERP and get the rule written in Figure In our model, LERP can be jointly optimized with probabilistic logical rules. We empir- ically show that our model outperforms previous logical rule learning methods on knowledge graph completion benchmarks. We also find that LERP allows model to compete, and sometimes exceed, strong black-box baselines. Moreover, LERP is itself an interpretable representation, so our model is able to discover more complex logical rules from data. In Section [5.4 we demonstrate that LERP can also be combinedd with sebbedding-learning models like TransE (Bordes et al. (013) to construct hybrid model that learns interpretable embeddings.
RELATED WORK
Logical Rule Learning This work is closely related the problem of learning logical rules for knowledge graph reasoning, and thus also related to the inductive logic programming (ILP) field. Traditional rule learning approaches search for the logical rules with heuristic metrics such as sup- port and confidence, and then learn scalar weight for each rule. Representative methods include Markov Logic Networks (Richardson & Domingos) 2006 Khot et al. 2011]. relational dependency networks (Neville & Jensen 2007; Natarajan al. 2010). rule mining algorithms GGalárraga al. Meilicke et al. 2019). path finding and ranking approaches (Lao & Cohen 2010 Lao et al.
2013
2011 Chen al. 2016), probabilistic personalized page rank (ProPPR) models (Wang al. 2013
2014a),
ProbLog (De Raedt et al.] 2007). CLP(BN) (Costa et al., (002). SlipCover (Bellodi & Riguzzi 2015). ProbFoil (De Raedt al. 2015) and SafeLearner (Jain 2019). However, most
Traditional methods face the problem of large searching space of logical rules, or rely on predefined
heuristics to guide searching. These heuristic measures are mostly designed by humans, and may not necessarily be generalizable to different tasks.
Differentiable Logical Rule Learning Recently, another trend of methods propose jointly learn- ing the logical rule form and the weights in a differentiable manner. Representative models include the embedding-bassee method of Yang et al.) (2015), Neural LP (Yang et al.) 2017). DRUM (Sadeghian al. 2019) and RLvLR (Omran et al.) 2018). Furthermore, some efforts have applied reinforcement learning to rule learning. The idea is to train agents to search for paths in the knowledge graph connecting the start and destination entities. Then, the connecting path be
2

---

Published as a conference paper at ICLR 2023
chaining J1,1
L
chaining
true
2,1
/3,1
(+)
true fi,1
f2.1
f111 1(a)
chaining fi.
chaiging J2.
capy
chaining fi,2
true
3,2
(7
true
chaining J1,3
chaining
true
2,3
3,3
true
f13
chaining J1,4
chaining
true
2,4
3,4
(9())
true
f2A
chaining
copy J
chaining
true
1
f3.5
4
(H) tree-like logical function
(b) parametrized probabilistic LERP
Figure 2: Illustration of our construction of logical entity function and representation. (a): Each
logical function is defined in a recursive manner with operations defined in Definition (b): we build LERP in feed-forward style, by constructing intermediate logical functions column by column.
tracted as a logical rule (Xiong et al. 2017 Chen et al. 2018 Das et al. 2018 Lin et al. 201 1018: Shen al. 2018). These methods are mostly focused on learning chain-like Horn clauses like Equation | This formulation limits logical rules from using extra contextual information about local subgraphs. In contrast, we propose to use LERP to incorporate contextual information into logical rules, and are capable of modeling more expressive family of rules.
Simbedding-Basee Neural Graph Reasoning Our work is also related and compared to graph embedding- methods. However, those models do explicitly model logic rules, but rea- son about entities and relations over the latent space. Representative work include TransE (Bordes) al. 2013). RotateE (Sun et al., 2018), ConvE (Dettmers et al.) 2018), ComplEx (Trouillon al.
2016 TransH (Wang et al. 2014b). HolE (Nickel et al. 2016). KEGAN (Cai & Wang. 2018
TuckER (Balažević al. 2019), Yang et a (215)) and box embedding methods (Ren al. 2020
Abboud et al., 2020) Onoe al. 2021). LERP is different from these latent embeddings because
LERP encodes contextual information rather than identity information of entities like in TransE. LERP is composed of vectors of interpretable logical functions, and can be applied to explicitly learning of more complex logical rules. Yang et al. (2015) also learns entity embeddings for rule learning, but the rules are still limited to chain-like rules, and the embedding is not interpretable.
PROBLEM STATEMENT
Chain-like Horn Clauses
of first-order logic rules that have been widely studied in
logical rule learning (Sadeghian et al. 2019 Yang et al. 2017). Formally, the task setting provides knowledge graph G = {(s, r, s]]s, &, RJ. where E is the set of entities and R is the set of relation types. The chain-like Horn clauses are defined as in Equation [] where y, zy. E & (for [1, K 1)) and H, rk R (for k K)). The "reverse" edges r' (z,=²) r(2', z) are often added to the edge type set R. Ti (1, 21) A... Ark(zK-1. is typically named the "body" of the rule.
A computational convenience of this formulation is that, if z is known, the rule can be evaluated by computing a sequence of matrix multiplications. We can first order entities from to |g], let V. be an n-dimensional hot vector of x's index, and let A,, be the adjacency matrix of Tk- In this way, we can compute v II Ar, in O(Kn²) time, by multiplying in left-to-right order from v side. It is easy to verify that this result is an n-dimensional vector counting how many paths connect x and following the relation sequence T1, T2 T.. This formulation has been widely adopted by previous methods.
Logical Entity Representation (LERP) In this work, we introduce LERP in logical rules to pro- vide additional local contextual information for entities in the knowledge graph. A LERP for entity is formatted as vector L(e) (Li(E),... Lmm(e) (e)) where the dimension is hyper-parameter. Each La(e) is a logical function over e's surrounding sub-graph. To enable efficient evaluation of L,(e), we limit (e) to family of tree-like logical functions as defined below:
Definition (Tree-like logical function)
3

---

Published as a conference paper at ICLR 2023
WN18RR
WN18
Category
Model
MRR
H@1
H@3
H@
MRR
H@1
H@3
H@
ComplEx
0.44
0.41
0.46
0.51
0.941
0.936
0.936
0.947
TransE
0.466
0.4226
0.556
0.495
0.113
0.888
0.943
ConvE
0.43
0,40
0,44
0.52
0.943
0.935
0,946
0,956
LinearRE
0.495
0.453
0.509
0.578
0.952
0.947
0,955
0.961
No Rule Learning
Inverse Model
0.35
0.35
0.35
0.35
0.963
0.953
0.964
0.964
QuatDE:
0.489
0.438
0.509
0.586
0.95
0.944
0.954
0.961
MLMLM
0,502
0.439
0.542
0.611
kNN-KGE
0.579
0.525
0.604
0.683
Neural LP
0.435
0.371
0.434
0.566
0.94
0.945
DRUM
0.486
0.425
0.513
0.586
0.944
0.939
0.943
0.954
Rule Learning
RNNLogic+
0.513
0.471
0.532
0.597
LERP
0.622
0.593
0.634
0.682
0.958
0.932
0.982
0.987
Table I: Experiment results on knowledge graph completion tasks on WN18RR and WN18.
Given binary predicate set R. the family of tree-like logical functions recursively constructed with the following operations:
1. (True function) fwoo) true belongs to T.
2. (Chaining) V fw:) E T and r E RR ('(++11) = Aw; f(ws) Ar(w;, wi+1) belongs to T.
3. (Negation) v f (w:) E T. f((:) =f(w:) belongs to T.
4. (Merging) & f(
E T, let t" max(i, + 1. after applying substitution [w, + Wyy) and (wy + wey to two functions respectively, f" f"(wwr) f(wy)) ( (A-merging) and fd ("(ww)) f(we)) V (wwx) (V-merging) belong T.
This definition includes arbitrarily complex functions, but in practice limit maximum number of operations. Intuitively, one can think of a function f(wx) E T as mimicking the shape of local tree spanning from u; in Figure a). Specifically, this is because it gathers information together recursively through logical operations. L(e) then serves as logical contextual representation for e's local surrounding sub-graph. We then extend Equation into the following formula incorporate contextual information from L(e) (w are entity variables and are interchangeable with 2):
Ti (x, 21) (2( -... A...A..A-..AA A. (=K-
H(x,))
#'E(M.NF:1:""
where i- [1..m] is index to select logical functions from LERP. This extension does not break the computational efficiency of chain-like Horn clauses (proofs are included Appendix AA:
Theorem For any rule expressed in Equation 2 if x is known, the evaluation of its LHS over x and all y E & (considering all possible 2j), can be determined in O(Cn²) time where |8| is the size of the entity set and C is a constant related to the rule.
4 FRAMEWORK
4.1 DERIVATION OF LERP
This section describes our computational modeling of LERP and how to incorporate in probabilis- tic logic rule learning. Borrowing ideas from Yang al. 2017); Sadeghian al. (2019) relax LERP to the probabilistic domain to make its results continuous. Figure 2 provides an illustration of our method. As we do not know the logical form of the functions (Figure a)) beforehand, we adopt a "reduncancy" principle and calculate a matrix of intermediate logical functions f (Figure 2(b)) where 1 < < and 0 S T. For example, f in 2 represents the same logical form as fs.1 in aa). The learning process is then to discover a good hierarchy within the matrix. When T is set to value larger than 1, the learned LERP conduct multi-hop reasoning.
4

---

Published as a conference paper at ICLR 2023
Family
Kinship
UMLS
MRR H@I H@3 H@ 10 MRR H@ 1 H@3 H@ 10
MRR H@1 1 H@3 H@10
MLN
0.351 0.189 0.408
0.707
0.688 0.587 0.755 0.869
Boosted RDN
0.469 0.395 0.520
0.567
0.227 0.147 0.256 0.376
PathRank
0.369 0.272 0.416
0.673
0.197 0.148 0.214
0.252
MINERVA
0,401 0.235 0.467
0.766
0.564 0,426 0,658
0.814
N
0.931
0.880
0.978
0.994
0.302 0.167 0.339
0.596
0.483 0.332 0.563 0.775
DRUM
0.958 0.927 0.987
0.996
0.534 0.367 0.628
0.885
0.695 0.546 0.808 0,935
RNNLogic
0.893 0.862 0.923 0.928
0.639 0.495 0.731
0.924
0.745 0.630 0.833 0.924
LERP
0.970 0.947 0.991 0.997
0.643 0.499 0.735 0.931 0.762 0.646 0.855 0.942
Table 2: Experiment results on statistical relation learning datasets.
Specifically, compute the tree-like probabilistic logical functions L parameterized by Q by adopting a feed-forward paradigm, and compute intermediate functions 1 (zz.s) column by column. The first column contains only true functions (z..) true. For j > O, each (2,())s)) 3 is constructed from the previous column by 6 possible operations: true function, chaining, negation, A-merging, and V-merging from Definition [ as well as an additional copy operation that simply copies f 1 (2)-1,x). Note that the result of J (2),s) over & can be represented as | 61-dimensional vector v, E [0, 1 1](81 Taking this notation, the formulae for the 6 possible types of ff. (2),) are:
jjitt = 1
jiichainnin
clamp
ar.iA. FR
Σ
VJ.6 11- Vj-1,
V jiccopy V
VAAA = V O
Ja V-11,
Vav = -1 (1- 2 ①
Σ Pa
where @ denotes Hadamard product, and clamp(r)= = e- clamps any positive number to range [0, 1]. Note that constrain that Orrj. 0, Σ, Gr..i 1, 87 O, Ex Pla 1. These 6
types of scores are then gathered by a probability distribution Pj...
Vjá Pj.i, faittmm + Pj.i2 jiicchiniig + Pj...Va. Pj,4, 2 + PAAAVRRA Pj.LoVaNv
Note that, specially in the second column, we only allow chaining fi.,, because only chaining pro- duces meaningful logical functions out of the first column of true functions. Thereby Vj, represents the result of the probabilistic logical function f.. all entities. p, a, 3 are all obtained by soft- max operation over some learnable parameters p', a', B8. These parameters belong to θ. There a total of T + columns of intermediate functions. Finally we use the last column of functions fo. as L. and (e) (e) ... , (e)) is our logical entity representation LERP on
Next we explain the computation of chain-like probabilistic logical rules, and how to further incor- porate LERP. In Yang et al. (017)) they model probabilistic selection of logical predicates, so that the rules can be diffcrentiably evaluated. Suppose for the chain-like rules in Equation [ the predicate of rı is selected according to distribution 0,,k- Then the probability that the clause body true between each x, E & can be given by
II km
Σ Can AER
Now we want also add a logical function constraint L from L Denote (E) as an m [8] matrix by iterating 8 (e) over all e E. We can select _ from 6 according to a probabilis- tie distribution Ps over [1.m]. Then the results T. (zx) on & can be written vector form L (E) Pk. Finally, the evaluation of the body in Equation can be written as
K
II A=1
Σ
A,
diag (IST Pk
(4)
PER
5

---

Published as a conference paper at ICLR 2023
weight
logical rule
0.10
(#2g sister(22, x)) A mother(21 æ) A son(y, 21) = (yotherry,,20
0.016
mother(21, x) mother(21, (7lzz sister =3)) brother(y.
0.11
son(21, A brother
+ son(y,
0.004
(A2 s)ster(>,, A brother(21, A (B23 sister(z3, (1)) A mother 21) son(x,
0.039
wife y) = husband(y,
0.024
m1,hhrr(x,5) (1) father(y, = = heusband(y, a)
Table A set of logical rules discovered from the Family dataset.
weight
logical function
weight
logical function
0.99
#æΙ sister 21)
0.56
3zj brother 21) A 32g aunt(22,
0.99
#e1 brother 21) 72g : sister(z1,
0.46
E21 aunt(e, 21) A B22 daughter(2s, (22,
0.96
0.39
Aez unele(e, Afg brother 21) A 322 (aughter(22,
0.68 32g sister(e, 21) A A22 sister(22,
0.25
Table 4: The list of logical functions L((e) learned in LERP L(e).
This is the evaluation for single logical rule. In practice we can have multiple rules to predict one target relation (e.g., DRUM, there are up to 4), and there are multiple types target relations in the dataset. So, we learn individual sets of parameters a, and ps: for different rules, but the parameter for LERP is shared across rules.
4.2 REASONING AND TRAINING
In the task of knowledge graph completion, the model is given of queries r(h, ?) and required to find the entity & E & that satisfies relation r(h, t). During reasoning and training, we denote h as the one-hot vector of entity h. and multiply it with Equation 4 to calculate
K
T h II k=1
Σ TER
diag
( Pk
This then can be calculated as a sequence of vector-matrix multiplications. The resulting vector indicating which entity t is satisfying the query r(h, ?). For training criterion, use the cross entropy between the normalized prediction |[t]|] and ground-truth one-hot vector The loss is differentiable with respect to all the parameters in model, mization methods such Adam Kingma & Ba 2015). During evaluation, we use as scores and sort it to obtain ranked entity list for calculating mean reciprocal rank (MRR), Hits, etc.
t
apply gradient-based opti-
EXPERIMENTS
5.1 KNOWLEDGE GRAPH COMPLETION
Datasets We follow previous works Yang et al. 2017 Sadeghian et al. 2019) and evaluate on the Unified Medical Language System (UMLS), Kinship. and Family datasets (Kok & Domingos) 2007) as well as the WN18 (Bordes et al. 2013) and WN18RR (Dettmers et al. 2018) datasets. The task to predict missing edges theses dataset, and is named "knowledge graph completion" on knowledge graphs like WN18RR and WN18 and "statistical relation learning" on smaller datasets like UMLSS, Kinship and Family. For all experiments, we evaluate under the setting of using no external data. For Kinship and UMLS, follow the data split from Qu al. (2200) and for Family, we follow the split used by [Sadeghian al. (2019).
compare with Markov Logic Networks (MLN) Richardson & Domingos, 2006), Boosted RDN (Natarajan et al., 2010). and path ranking (PathRank) (Lao & Cohen 2010) well other differentiable rule learning baselines such NeuralLP (Yang al.) 2017) and DRUM (Sadeghian et al., 2019). We also include a reinforcement learning method MIN
For rule learning baselines,
ERVA DDs al. 2018) and inpecttion-maximizattie method RNNLogic (Qu al. 2020) in

---

Published as a conference paper at ICLR 2023
"is daughter" "
has sister"
"Is son
"haya brother"
Figure 3: Visualization of the space of LERP on the Family dataset. each figure, we color as red entities of different relative roles (e.g.. people who are daughters, sons, or have sisters or brothers).
baselines. On knowledge graph completion task, we further compare with the following powerful black-box embedding- based models: ComplEx (Trouillon al. 2016), TransE (Bordes al. 2013).
ConvE and Linear Model (Dettmers et al. 2018), LinearRE (Peng & Zhang 2020). QuatDE GGa ct al.
202T), , MLMLM (Clouatre et al. 2021 and KNN-KGE (Zhang et al. 2022).
For evaluation, following the setting in Sadeghian et al. (2019). we we adopt filtered ranking and evaluate models on mean reciprocal rank (MRR) as well as the Hits [1, 3, 10) metrics. We search for hyper-parameters according to validation set performance, and finally adopt the hyperparameters dimension m = 80 and depth T = 2 for LERP, and the maximum rule length of K = 3. We use 4 rules for WNISRR and WN18 and Family and 200 rules per target relation type for Kinship and UMLS dataset. Our model is trained for 4 epochs for WN18RR and WN18 and 10 epochs for Family, Kinship, and UMLS. As DRUM runs on fewer number of rules by default, re-run DRUM with larger number of rules to match the configuration of our model and RNNLogic. For optimization, we use Adam with a learning rate of 0.1, 31 0.9, and f3 0.999.
The results for are demonstrated Tables T and 2 Compared with other rule learning methods, model obtains higher performance across the board. This validates the importance of contextual information provided by LERP, especially after considering that this is the only difference in model design between DRUM and our model. Black box models, which do not conduct rule learning, generally achieve higher scores than rule learning methods. This is both because of the strong fitting power of black-box models and because rule learning models also need to explicitly learn logical rules. However, our model achieves performance comparable or even superior to black-box baselines. Even on metrics where LERP's performance falls short of that of state-of-the-an models (like Hits @ 10 on WN18RR dataset), the difference is marginal. We attribute this to the property of LERP modeling a more expressive set of logical rules, allowing it to achieve data-fitting power similar to black-box neural models.
5.2 INTERPRETABILITY LERP AND LOGICAL RULES
An important advantage of logical rules is that they help humans to better comprehend the reasoning process of the algorithm. In this subsection, we provide a case study on the interpretability LERP and our model on Family dataset.
In Table 3 we sort the logical rules according to their weights, and show the top-ranked logical rules for three relations. We can that these logical rules more complex than chain- like Homn clauses as in Equation|] I For example, in the second row, the logical function Esister(y, 23) provides contextual information that y is not sister of anybody. E Therefore, y is more likely to be the brother than sister of x. Similarly, in the fourth row, A2g sister 21) provides information that 21 has no sister, so can only be the brother of 2 and thus the son of y. We can also observe some verbosity in the learned rules, like the logical function Áz2 sister(22. x) in the first row. This not necessary since son(y- y, 21) enough for predicting that y is the brother of x. But this kind of verbosity is inevitable since rules of the same meaning are equivalent from the model's perspective.
Since LERP is itself an interpretable representation. we can inspect directly in Table We sort the logical functions 7 according to the weights, and filter out the repeated functions. Some logical functions are related to the gender information of entities. For example, Asister(e, 21) might suggest
Note that in the Family dataset ontology. {rel ati (a, b) means that is the (relation) of (e.g.. son(a, b) means is b's son).
7

---

Published as a conference paper at ICLR 2023
0.7
0.68
LERE
DRUM
0.65
0.663
0.6
0.655
0.575
0.55
0.488
0.5
0.63
04
3
10
100
1000
10
100
1000
10000
(a) depth T of LERP
(b) dimension of LERP
(c) # logical rules
Figure 4: Analysing LERP's performance under different configurations.
(although with uncertainty) that e is male. Other logical functions are longer and contain more complex information of entities. For instance, 3žj brother(e, =1) A 322 : daughter (22, e) tells us e is brother and has daughter, which might help predict the nephew/aunt-uncle relationship between his sibling and daughter.
We notice that the weights of some interpreted logical functions not close to 1.0, and the actual utility of some logic functions are not easy to guess merely from the interpreted logic form. To better understand the space of LERP representation, analysis (PCA) on LERP vectors of entities in the graph. We select entities with some relational roles such being daughter, being a son, having a sister, having a brother, and mark them with red circles in contrast to the blue pluses of the other entities. For better visualization, we also plot the plane dividing them from the rest of entities. The results are shown in Figure 3 We find that the lower-dimension dimensional space of LERP is well organized according to these roles. Note that the role of having a brother is not explicitly stated the learned logical functions in Table [ We hypothesize that it might come from a combination of other logical functions like 3zg brother(e, =1) A 322 aunt(22, e) and 321 brother(e, 21) A 322 daughter e). This demonstrates that logical functions in LERP can be combined to represent extra logical information.
conduct 3-dimensional principle component
5.3 MODEL ANALYSIS
Is contextual information from LERP useful? This work is motivated by including contextual information for rule learning. To validate our hypothesis, we vary the depth of LERP from 0 to 4 and observe the performance. This hyper-parameter controls the complexity of logical functions learned in LERP. When depth is our model does use contextual information from LERP and defaults back to the DRUM baseline. Results are shown in Figure 4 a).. We see that even with depth of 1, LERP boosts our model's accuracy by large margin compared with the depth of O. Similar conclusion can also come from the comparison between DRUM and LERP in Table and [ because DRUM can be viewed as an ablation study counterpart of LERP by removing contextual information from LERP.
Is a higher dimension of LERP always better? The answer no. Similar to embedding based models like TransE, larger dimension may cause LERP to overfit the training data and perform worse on the test set. In Figure 4[b,, we set the width of LERP to (3, 10, 40, 100, 200, 400). We observe a n-shaped curve with 40 giving optimal test performance.
Does the model benefit from larger number of logical rules? The default number of logical rules can be different across models. For example, RNNLogic uses 100~200 rules per relations, but DRUM only uses 1~4 rules by default. To better analyze the model's performance, we vary the number of rules in our model. We also compare with DRUM. In Figure cc,, both models' performance increase with larger number of rules, but with diminishing returns (note the logarithmic scale of the x-axis). Our model achieves higher scores even with fewer number of rules. We attribute this to LERP helping to learn higher quality rules.
5.4 AUGMENTING EMBEDDING LEARNING MODELS WITH LERP
LERP is designed to leamn an interpretable logical representation for entities in graphs. One question then arises: can LERP work together with other enbedding-leaming methods such as
8

---

Published as a conference paper at ICLR 2023
logical function
relations
logical function
relations
EE niece (e, 2)
daughter,nicee
32 : aunt(e, 2) V 3: niece 2)
mother
A niece 2)
father, son
3z aunt(e, A 32 (e,
#2 uncle(e,
aunt,sister
22 22 mother 21, e) e) niece(21,22) brother,nephew
Table 5: The list of logical functions (e(e) learned when optimizing TraRSE+LERP. We project the learned translation vectors / back to the LERP space and identify most correlated logical functions.
TransE (Bordes et al.) 2013), combine the logic structure in LERP and strong fitting power of TransE? Inspired by this idea, we design hybrid model TransE+LERP. We use the train- ing framework and training criterion Embed"(e) Embeed(e) + (e). (mbeed(e) is set as a 20-dimensional entity-specific embed- ding. We also the width and depth of LERP to be and 2. is 20 x matrix transform LERP to the embedding space. Besides the loss in original TransE framework, we add a regularization term Σ. [1mmbed(e)l " [1 encourage LERP to encode important information during optimization. We also introduce an advanced version TransE#LERP+enropy by adding an extra entropy regularization term y (+) LERP's weights. This term is added to loss and encourage LERP assign more weights to fewer logical rules. In experiments, we set r 10 and 1.
TransE, but the entities' embeddings now defined as
We evaluate the models on Family dataset and results are presented in Figure [ The vanilla TransE model tends overfit the training (>95% training set accuracy) because Family dataset contains limited number of training triplets. TransE+LERP achieves higher test set performance. We hypothesize that LERP pro- vides logical regularization on the embedding Embed' (e) and makes it more generalizable. After adding the entropy regularization term, the performance further improves. We attribute this to that the regulation encourages learning more explicit logical functions, which serves as prior in embedding learning.
0.8
TransE
+LERP
+LERP+entropy
0.6
0.2
MRR
Hits@
Hits@
Hits@
Figure 5: Incorporating LERP representation im-
We also study the interpretability of the hybrid model. In Table 5 we list the logical functions that are learned by TransE+LERP. There are repetitions in logical functions which are removed. We also manage identify the correlation between relations in TransE and logical functions. In TransE, each relationship has vector / to denote translation in the embedding space. I resides in the embedding space, and we want to find vector l' in LERP space so that Rl' L. As R is not square matrix, we use R's Moore-Penrose inverse R+ to project the translation vectors to R+1. We then look for dimensions with highest values in R+ I, and results are listed in the 2nd and 4nd columns in Table 5 We can observe that these functions encode rich information. For example, relations like daughter and niece are associated with 3z meice((, z) which suggests female gender and lower rank in the family tree.
proves TransE's performance on Family dataset.
CONCLUSIONS AND FUTURE WORK
In this work, we propose logical entity representation (LERP) and demonstrate that we can in- corporate contextual information of local subgraphs to probabilistic logical rule learning. Therefore we are able to learn more expressive family of logical rules in fully differentiable manner. In experiments on graph completion benchmarks, our model outperforms rule learning baselines and competes with or even exceeds state-of-the-art black box models. LERP is itself interpretable representation of entities. We also find that LERP can be combined with latent embedding learning methods to learn interpretable embeddings. For future work, expect that LERP can be com- bined with a wider range of neural networks to provide logical constraints and learn interpretable embeddings.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The datasets, split and the configuration of the model are introduced in Section 5.1 The statistics of the datasets used are also included in Appendix D We also conduct analysis of the model behavior in Section 5.3] the additional experiments of combining TranSE and LERP, we provide details of implementation in Section 5.4] including the hyper-parameter and rationale for some technical decisions. For Theorem
we provide proof in Appendix A
ACKNOWLEDGMENTS
We would like to thank anonymous reviewers for valuable comments and suggestions. This work was supported in part by US DARPA KAIROS Program No. 148750-19-221006 and AIDA Pro- gram No. FA8750-18-2-0014 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Govermment. The U.S. Government authorized reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
REFERENCES
Ralph Abboud, Ismail Ceylan, Thomas Lukasiewiez, and Tommaso Salvatori. Boxe: A box embed- ding model for knowledge base completion. Advances in Neural Information Processing Systems, 33:9649-9665 2020.
Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. 2017.
Ivana Balažević, Carl Allen, and Timothy Hospedales. Tucker:
Tensor factorization for knowledge graph completion. Proceedings of the 2019 Conference on Empirical Methods Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 5185-5194, 2019.
Elena Bellodi and Fabrizio Riguzzi. Structure learning of probabilistic logic programs by searching the clause space. Theory and Practice of Logic Programming, 15(2):169-212, 2015.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. volume 26, 2013.
Liwei Cai and William Yang Wang. Kbgan: Adversarial learning for knowledge graph embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1470- 1480, 2018
Andres Campero, Aldo Pareja, Tim Klinger, Josh Tenenbaum, and Sebastian Riedel. Logical rule induction and theory learning using neural theorem proving. arXiv preprint 10X.v1880.002193, 2018.
Wenhu Chen, Wenhan Xiong, Xifeng Yan, and William Yang Wang. Variational knowledge graph reasoning. In Proceedings of the 2018 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, Volume I (Long Papers),
pp. 1823-1832, 2018
Yang Chen, Sean Goldberg, Daisy Zhe Wang, and Soumitra Siddharth Johri. Ontological pathfind- ing: Mining first-order knowledge from large knowledge bases. In Proceedings of the 2016 International Conference on Management of Data, pp. 835-846, 2016.
Louis Clouatre, Philippe Trempe, Amal Zouaq, and Sarath Chandar. Mlmlm: Link prediction with mean likelihood masked language model. In Findings of the Association for Computational Lin- guistics: ACI-IJCNIPP 1, pp. 4321 4331-4331, 2021.
Vitor Santos Costa, David Page, Maleeha Qazi, and James Cussens. Clp (bn) constraint logic pro- gramming for probabilistic knowledge. In Proceedings of the Nineteenth conference on Uncer- tainty in Artificial Intelligence, pp. 517-524, 2002.
10

---

Published as a conference paper at ICLR 2023
Rajarshi Das. Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishna- murthy, Alex Smola, and Andrew McCallum. Go for a walk and arrive at the answer: Reasoning paths in knowledge bases using reinforcement learning. In International Conference Learning Representations, 2018.
Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: A probabilistic prolog and its application in link discovery. In IJCAI, volume 7, pp. 2462-24626 Hyderabad. 2007.
Luc De Raedt, Anton Dries, Ingo Thon, Guy Van den Broeck, and Mathias Verbeke. Inducing probabilisticc relational rules from probabilistic examples. In Twenty-fourth international joint conference on artificial intelligence, 2015.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Pedro Domingos and Daniel Lowd, Unifying logical and statistical with markov logic. Commu- nications of the ACM, 62(7):74-83, 2019.
Pedro Domingos and Matt Richardson. Mining the network value of customers. In Proceedings of the seventh ACM SIGKDD international conference Knowledge discovery and data mining, pp. 57-66, 2001.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In International Conference Learning Representations (ICLR), 2018.
Nir Friedman, Iftach Nachman, and Dana Pe' Learning bayesian network structure from massive datasets: The" sparse candidate" algorithm. arXiv preprini arXiv: 331.66666 2013.
Luis Antonio Galárraga, Christina Teffioudi, Katja Hose, and Fabian Suchanek. Amie: association rule mining under incomplete evidence in ontological knowledge bases. In Proceedings of the 22nd international conference on World Wide Web, pp. 413 422, 2013.
Haipeng Gao, Kun Yang, Yuxue Yang, Rufai Yusuf Zakari, Jim Wilson Owusu, and Ke Qin. Quatde: Dynamic quaternion embedding for knowledge graph completion. arXiv preprint arXiv:2105.09002, 2021.
Lise Getoor, Nir Friedman, Daphne Koller, and Avi Pfeffer. Learning probabilistic relational models. In Relational data mining, pp. 307-335. Springer, 2001.
Alexander Hartemink, David K Gifford, Tommi S Jaakkola, and Richard A Young. Using graph- ical models and genomic expression data to statistically validate models of genetic regulatory networks. In Biocomputing 2001, pp. 422-433. World Scientific, 2000.
David Heckerman, Dan Geiger, and David M Chickering. Learning bayesian networks: The combi- nation of knowledge and statistical data. Machine learning, 20(3):197-243, 1995.
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logie rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp.
2016.
A Jain. Scalable rule learning in probabilistic knowledge bases. In The 1st Conference On Automated Knowledge Base Construction (AKBC), 2019.
Navdeep Kaur, Gautam Kunapuli, Saket Joshi, Kristian Kersting, and Sriraam Natarajan. Neural networks for relational data. In International Conference on Inductive Logic Programming, pp. 62-71. Springer, 2019.
Tushar Khot, Sriraam Natarajan, Kristian Kersting, and Jude Shavlik. Learning markov logic net- works via functional gradient boosting. In 2011 IEEE 11th international conference data mining, pp. 320-329, IEEE, 2011.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.

---

Published as a conference paper at ICLR 2023
Stanley Kok and Pedro Domingos. Statistical predicate invention. In Proceedings of the 24th inter- national conference on Machine learning. pp. 2007.
Daphne Koller, Nir Friedman, Sašo Džeroski, Charles Sutton, Andrew McCallum, Avi Pfeffer, Pieter Abbeel, Ming-Fal Fai Wong. Chris Meek, Jennifer Neville, et al. Introduction to statistical relational learning. MIT press, 2007.
Ni Lao and William W Cohen. Relational retrieval using a combination of path-constrained random walks. Machine learning, 81(1)533-67, 2010.
Ni Lao, Tom Mitchell, and William Cohen. Random walk inference and learning in a large scale knowledge base In Proceedings of 2 conference on empirical methods in natural language processing, pp- 529-539, 2011.
Manling Li, Sha Li, Zhenhailong Wang, Lifu Huang, Kyunghyun Cho, Heng Ji, Jiawei Han, and Clare Voss. Future is not one-dimensional: Graph modeling based complex event schema induc- tion for event prediction. Proc. The 202/ Conference Empirical Methods in Natural Language Processing (EMNLP2021), 2021.
Qika Lin, Jun Liu, Fangzhi Xu, Yudai Pan, Yifan Zhu, Lingling Zhang, and Tianzhe Zhao. Incor- porating context graph with logical reasoning for inductive relation prediction. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information 2022.
Retrieval, pp.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. Multi-hop knowledge graph reasoning with reward shaping. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3243 3253, 2018.
Alexander V Lukashin, Matvey E Lukashev, and Rainer Fuchs. Topology of gene expression net- works as revealed by data mining and modeling. Bioinformatics, 19(15))1909-199 1916, 2003.
Sijie Mai, Shuangjia Zheng, Yuedong Yang, and Haifeng Hu. Communicative message passing for inductive relation reasoning. In AAAI, pp. 4294 4302, 2021.
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. Deepproblog: Neural probabilistic logic programming. Advances Neural Information Process- ing Systems, 31, 2018.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In International Conference on Learning Representations (ICLR), 2018
Giuseppe Marra and Ondřej Kuželka. Neural markov logic networks. In Uncertainty in Artificial Intelligence. pp. 908-917. PMLR, 2021.
Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pp. 3137-3143, 2019.
Lilyana Mihalkova and Raymond Mooney, Bottom-up learning markov logic network structure. In Proceedings of the 24th international conference Machine learning, pp. 625-632, 2007.
Sriraam Natarajan, Tushar Khot, Kristian Kersting, Bernd Gutmann, and Jude Shavlik. Boosting relational dependency networks. In Online Proceedings of the International Conference on In- ductive Logic Programming 2010, pp. 1-8, 2010.
Jennifer Neville and David Jensen. Relational dependency networks. Journal of Machine Learning Research, 8(3), 2007.
Maximilian Nickel, Volker Tresp. and Hans-Peter Kriegel. A three-war way model for collective learning on multi-relational data. Proceedings of the 28th International Conference on International Conference Machine Learning, pp. 809-816, 2011.
12

---

Published as a conference paper at ICLR 2023
Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. Holographic embeddings of knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Pouya Ghiasnezhad Omran, Kewen Wang, and Zhe Wang. Scalable rule learning via learning rep- resentation. In IJCAI, pp. 2149-2155, 2018.
Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. Modeling fine-grained entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 2051-2064, 2021.
Ali Payani and Faramarz Fekri. Inductive logic programming via differentiable deep neural logic networks. arXiv preprini arXiv: 1:X6vv9906.03330 000000000.,,.... 2019.
Jian Pei, Jiawei Han, Behzad Mortazavi-Asl, Helen Pinto, Qiming Chen, Umeshwar Dayal, and Meichun Hsu. Prefixspan: Mining sequential patterns by prefix-projected growth. In Proceedings of the 17th International Conference on Data Engineering. pp. 115-224, 2001.
Yanhui Peng and Jing Zhang. Lineare: Simple but powerful knowledge graph embedding for link prediction. In 2020 JEEE International Conference on Data Mining (ICDM), pp. 422 431. IEEE, 2020.
Connor Pryor, Charles Dickens, Eriq Augustine, Alon Albalak, William Wang, and Lise Getoor Neupsl: Neural probabilistic soft logic. arXiv e-prints, pp. arXiv-2205, 2022.
Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. Advances in neural information processing systems, 32, 2019.
Meng Qu, Junkun Chen, Louis-Pascal Xhonneux, Yoshua Bengio, and Jian Tang. Rnnlogic: Learn- ing logic rules for reasoning on knowledge graphs. In International Conference on Learning Representations, 2020.
Ross Quinlan and R. Mike Cameron-Jones. Induction of logic programs: Foil and related systems. New Generation Computing, 13(3):287-312, 1995.
Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. arXiv preprint arXiv:2002.05969, 2020.
Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learning, 62(1):107~ 136, 2006.
Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme, Script induction as language modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 168 1-666, 2015.
Ali Sadeghian, Mohammadreza Arnandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to- end differentiable rule mining on knowledge graphs. Advances in Neural Information Processing Systems, 32, 2019.
Mark Schmidt, Alexandru Niculescu-Mizz Kevin Murphy, et al. Learning graphical model structure using I1-regularization paths. In AAAI, volume 7, pp. 1278-12833 2007.
Yelong Shen, Jianshu Chen, Po-Sen Huang, Yuqing Guo, and Jianfeng Gao. M-walk: Learning to walk over graphs using monte carlo tree search. Advances in Neural Information Processing Systems, 31, 2018.
Pater Spirtes, Clark Glymour, Richard Scheines, Stuart Kauffman, Valerio Aimale, and Frank Wim- berly. Constructing bayesian network models of gene expression networks from microarray data. 2000.
Sriram Srinivasan, Charles Dickens, Eriq Augustine, Golnoosh Farnadi, and Lise Getoor. A tax- onomy of weight learning methods for statistical relational learning. Machine Learning. 111(8): 2799-2838, 2022.
13

---

Published as a conference paper at ICLR 2023
Zhiqing Sun, Zhi- Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2018.
Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning, pp. 9448-9457. PMLR, 2020.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In International conference on machine learning, PP- 2071-2080 PMLR, 2016.
Ioannis Tsamardinos, Laura Brown, and Constantin F Aliferis. The max-min hill-climbing bayesian network structure learning algorithm. Machine learning, 65(1):31-78, 2006.
William Yang Wang, Kathryn Mazaitis, and William W Cohen. Programming with personalized pagerank: locally groundable first-order probabilistic logic. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2129-2138, 2013.
William Yang Wang, Kathryn Mazaitis, and William W Cohen. Proppr: Efficient first-order proba- bilistic logic programming for structure discovery, parameter learning, and scalable inference. In Workshops at the Twenty-Fight AAAI Conference on Artificial Intelligence, 2014a.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by trans- lating on hyperplanes. In Proceedings of the AAAI conference artificial intelligence, vol- ume 28, 2014b.
Wenhan Xiong, Thien Hoang, and William Yang Wang. Deeppath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 564-573, 2017.
Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference knowledge bases. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. Advances in neural information processing systems, 30, 2017.
Yuan Yang and Le Song. Learn to explain efficiently via neural logic inductive learning. In Interna- tional Conference on Learning Representations, 2019.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
Ningyu Zhang, Xin Xie, Xiang Chen, Shumin Deng, Chuanqi Tan, Fei Huang, Xu Cheng, and Huajun Chen. Reasoning through memorization: Nearest neighbor knowledge graph embeddings. arXiv preprint arXiv:2001.05777 2022.
Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, and Le Song. Effi- cient probabilistic logic reasoning with graph neural networks. In International Conference on Learning Representations, 2019.
14

---

Published as a conference paper at ICLR 2023
A PROOF OF THEOREM 1
More specifically. we will prove that when C = [( + D), where K is the length of the rule in Equation and is the maximum number of used operations Definition ! for constructing each L5 the computation time for evaluating x and any E & can be bounded by O(Cn²) where E This is an asymptotically tight bound, because reading all the edge information naturally requires lower bound of [Cmin(C, [R] )m)) time.
First easy to see that all tree-like functiins only have one free variable. We will bound the computation time of evaluating tree-like functions f(e) E 'T over all entities in &. Let v(f) be the vector of scores of f on &, and d(ff) be the number of operations used for constructing f. we recursively prove that v(f) can be calculated in O(d(f)n²) time. First for an all true function d(f) = v(f) contains only can directly written in O(n) time. For negation operator f= =f' v(f) only needs negating v(f") and takes O(({(f) - 1)n²)
O(d(f)n²) time. Similarly for a merging operation f = A fz. v(f) can derived by processing elements of v(i) and v(22) sequentially and needs O((d(f) 1))=)) 2n < (ddd())n²) time. Finally for a constructed by chaining I' and relation r, v(f) is just v(J) A, in binary field and takes O((d(f) 1
1)n?) + n² O(d(f)n²) time.
So getting result vectors for all tree-like logical functions takes no more than Dn?) time. We then plus the time for evaluating the main body in Equation we start from one-hot vector
Vz. After chaining each relation Tk. the partial evaluation of the logical rule T1 r1(r, (x, 21) A (2(21, 22 A... Ta (2(-x11,)) A (Avees,,)). ( (z') can be computed by v t vT Are- Applying the new constraint Li,g (2x) is another element- wise logical and between and v (Lin)) which takes O(n) time. Summing together. the total time for evaluating the whole rule can be bounded by Dn2 Kn + Kn²) O(K(1 + D)m²).
B EVENT PREDICTION
Dataset
Model
MRR
HITS@1
Event Language Model (Rudinger et al.
2015
0.367
0.497
Sequential Pattern Mining Pei al. 200T
0.330
0.478
General
Human Schema [Li et al. 2021
0.173
0.205
Event Graph Model [Li et al 2021
0.401
0.520
LERP
0.347
.672
Event Language Model (Rudinger et al.
2015
0.169
0.513
Sequential Pattern Mining (Pei et al. 2001
0.138
0.378
IED
Human Schema (Li et 2021
0.072
0.222
Event Graph Model al
0.223
0.691
LERP
0.242
0.978
Table 6: Experiment results ending event prediction.
Dataset
Split
#doc
#graph
#event
#arg
#rel
Train
451
451
6,040
10,720
6,858
General
Dev
83
1,044
1,762
1,112
Test
83
83
1,211
2,112
1,363
Train
5,247
343
41,672
136,894
122,846
IED
Dev
575
42
4,661
15,404
13,320
Test
577
45
5,089
16,721
14,054
Table 7: Dataset statistics for event prediction datasets (Li et al. 2021)
One driving force for humans to understand logic is to interpret the past and predict the future. Going beyond instincts, logic provides humans with the ability of explainable and reasoning based prediction. In this section, we evaluate LERP's ability on an ending-event prediction task. This task different from the link completion task of Subsection 5.1 The goal is to deduce the existence of
15

---

Published as a conference paper at ICLR 2023
future events given an imput event graph, typically produced by human annotation or information extraction from news or stories.
Benchmark Datasets We conduct experiments on the graph-based event prediction datasets from Li (202I), including general scenario (General Schema Learning Corpus) and more specific bombing scenario (IED Schema Learning Corpus). More specifically, the General Schema Learning Corpus is released by Linguistic Data Consortium (LDC2020F:25) including 82 types of events, such as Disease Outbreak and Shop Online. The original improvised explosive device (IED) dataset was collected by Li al.) (2021). They conduct two-round annotation process with linguists to ensure quality of the annotated schemas/graphs. The dataset statistics can be found in Table
Evaluation Metries and Training Objective Following prior work (Li et al. 2021), we adopt MRR (Mean Reciprocal Rank) and HITS@@ as evaluation metrics. An event is predicted correctly if the event type matches one of the ending event types in the gold-standard event graph. Consid- ering that there can be multiple ending events one instance graph, we rank event types with their prediction scores. The model adopts multiple output heads, and uses categorical cross entropy for training objective.
Baselines Event Language Model (Rudinger et al. 2015) learns the probability of temporal event sequences, and obtain the event sequences prediction from language model. Sequential Pattern Mining (Pei et al. 2001) is a classic method that explores prefix-projection in sequential pattern mining. The most frequent mined patterns/event graphs are used for predicting the ending event. Event Graph Model (Li et al.] 2021) is the state-of-the-art model on the ending event prediction task. It proposes temporal event graph model that predicts the future event type given the current (incomplete) event graph. Finally, we also add human schema as a reference (Li et al.] (02T). We match expert-created event schemas to test data graphs and fill in the matched ending event type.
Results and Analysis In Table we present the results on these two datasets. We find that all of the learning- based methods outperform pattern mining-base method (Pei et al. 2001) on both datasets. Our model further outperforms the basclines on most evaluation categories including the Event Graph Model on HITS @ 1. We attribute the inferior performance of the human schema match- ing baseline to that real event graphs are highly variational, so fixed schemas do not generalize well to the test graphs.
C LOGICAL EVENT EQUATION INDUCTION
Example 1: (avg. correctness-4.7 7, infomativenees-4.7
Example (avg. correctness-4.3, 4,3, informatieenes-4.3
tanm, x2" (Cooflic Cospeifinn
Lie, Die Ciupacifioll 3/ v Moutecedmottellllllii Ciupectfiol
"X0 XI" (3 DF"x2"//Coniiie :CoofficttAAticc Unppecffed(
(hontiee TrinlHearigg (ooeeeiin"""""" NAND
Life Die. Unspeeffeed (Life. Injure Unspecified/ (X0" MAND
Sentence wenteeeeiooiepeee Uiopisffaat 23 "jI] trw
Cogultiv Mouutff Categorize Unspecifedd "XI [j)] tree
Graph Illustration
Conflit.Atiack Utspaciiiied
Graph Illustration
Justicn Sontonce
Life Affelneure. Unspecifeed
ConffictAfluck Unspeciffed
Unspeciffed
AND
OR
Unspaciffed
TriatHearing Unspectfied
Aresste.ArrssJJal
Cognitie.ldennilycc tegarize, Unspecttedd
Lie.Dia Unspoelliee
Unspecleed
Figure 6: Induced event relation examples from IED
Li et al. 202T
One important feature of LERP its ability to learn explainable first-order logical formulae. In this section, we train the model to induce logical relations among events. Specifically, we use IED dataset (Li et al. 202I) for training. Specifically, we adopted contrastive training to encourage LERP distinguish positive (real) and negative (incomplete) event graphs. The training dataset is used directly as the positive graph. We randomly sample an event graph instance, and randomly remove 30 event nodes to build negative graph. The negative graph is then upsampled to match the number of all positive graphs. The training objective of LERP is the contrastive loss between positive and negative graphs.
16

---

Published as a conference paper at ICLR 2023
Family
UMLS
Kinship
#Relation #Entity #Train #Valid #Test
#Triplets
28356
5960
9587
WN18 WNISRR
18
40,943
141,442
5,000 5,000 86,835
,442
#Relations
12
46
25
11
40,943
3,034 3,134
#Entities
3007
135
104
Table 9: Dataset statistics for knowledge graphs. WN18RR is more challenging version of WN18 by filtering out some data bias.
Table 8: Dataset statistics for statisti- cal relation learning (Sadeghian et al., 2019) tasks.
Induced examples are shown in Figure We rank and filter the induced logical equations by the model confidence, and let human annotators to assess the quality of induced equations. As model able induce logical formulae in arbitrary form, provide visual illustration for each logic formula, so as to unify the form and provide easier human interpretation. These induced equations demonstrate LERP's ability of learning complex relations from data, which are highlt scored by humans.
D DATASET STATISTICS
We also list the dataset statistics in Table 9 Family, UMLS and Kinship are smaller scale datasets. WN18 and WN18RR contains larger number of entities and relations.
E DISCUSSION OF COMPARISON WITH NLIL
A closely relatd work to LERP is NILI Yang & Song 2019), which also claims to learn tree-like logical functions. A major difference between LERP and NLIL that, LERP able to model more complex graph structure in logical rules, and encode more contextual information about nodes in the middle of chains connecting head and tail nodes. This also the main motivation our work. Although NLIL also aims at going beyond chain-like rules, its definition of "tree-llike rules" is indeed very different from ours (which is also mentioned in your comments). More specifically, the rules in NLIL are formulated multiple chain-like rules combined together logically. As will explain in the following sub-points, this formulation still overlooks contextual information for middle nodes in logical chains, and reduce simple baselines like JeuralLP(Yang) al. (017) in some settings. This problem can be viewed from the following perspectives:
Formulation of rules NLIL has three parts working hierarchically. (i) The first part is responsible for searching among unary predicates L/ and binary predicates (ii) The second part searches for "primitive statements", which are essentially chain-like rules with either one free variable at end or two free variables æ and x' at two ends. This can be seen from the definition of th (Equationn9)) in NLIL paper): t/a (x, x²) can be written in one of the two cases:
T
o(1
T
M II
M
(( vxc);
f'=1
or
(v M M
II M vx))
Both cases belong to the family of chain-like rules described in Equation 1 and in our paper. (L.a.ii) The final step logically combines primitive statements, which applies logical operators {A =} over chain-like rules. This steps introduces logical complexity, but does not change the chain form the graph to encode more complex graph struture information. That is, the instantiation the rules in the graph are still multiple chains spanning from connecting and x'. Similarly, chain- like rule baselines DRUM (Sadeghian et al. 2019) and NeuralLP (Yang et al. 2017) also models disjunction of multiple chains. Our formulation, however, is able model more complex graph structures (trees or branched chains). As shown from the following two empirical observations, ILIL's formulation limit the expressiveness of its rule learning.
17

---

Published as a conference paper at ICLR 2023
Example rules NLIL presents examples of learned logical rules in its Table 4. Its rules can still be parsed as logical combination of multiple chains. For example, the first rule for Person(X can be seen logical disjunction V combining these chains: chain, Shirt (Y;) A Wearing(Y,, cha n2 (an):s(22) A Weari (Y2, X) and chaing : Street (Ya)/ A Wearing (Y, X). The same also applies other examples in Table 4 in NLII. Baselines DRUM and NeuralLP also study learning multiple chain-like logical rules with final scores summed together, which mod- els an implicit conjunction operation of these rules. For example, one can convert DRUM Table 7 examples equivalently to conjunction form: wife(A, (mother (A, C), father (B, C)) V (son(C, father(B, C)). NLIL's logical formu- lation is only different from NeuralLP in that NLIL allows for A to combine chains.
+ husband(B, A) V
Experiment results as argued in the above two points, NLIL is similar to chain-like rules in mod- elling graph structures. Therefore, in knowledge graph completion experiments (Table 1 in NLIL paper), NLIL's absolute scores are still similar (or inferior) to chain-like rule learning baselines like NeuralLP, with gains more than 0.01 absolute value. This also partially validates the argument that NLIL's expressiveness in graph reasoning domain is still similar to chain-like rules learning.
F SPEED OF RULE LEARNING
Traditional ILP methods such as FOIL. (Quinlan & Cameron-Jones) 1995) is able to learn thousands to millions of logic rules using no more few seconds per rule. A question then arises: how fast can differentiable rule learning approaches learn logic rules? Normally, differentiable rule learn- ing methods for knowledge graph completion (including our approach as well as RNNLogic and DRUM) will not set the number of logical rules very high. A typical number usually ranges from 3 or 4 to a few hundreds of rules. Main reason is that these work focus more on downstream evalua- tion metrics, such knowledge graph completion accuracy. Moreover, these metrics improve only marginally when the number of rules increases to very high (see Figure 4(c) in our submission). Another consideration is that, a larger number of logic rules increases the computation overhead, because these rules are executed in parallel in experiments. Our 16GB CUDA memory space max- imally allows for 3200 rules to be run together on Kinship and UMLS dataset, but doubles the running time compared with 200 rules. So in experiments, we just choose number (200 in this paper for Kinship and UMLS) that work reasonably well (selected by validation performance) and balances running speed, while also being comparable to previous work.
More specifically, on Kinship dataset for example, our model trains for 2.8 minutes and gets $200 rules (200 rules target predicate 26 target queries). In comparison, DRUM (Sadeghian et al.) 2019) gets the same number of rules in 1.2 minutes, while RNNLogie uses 10 minutes. They also tend to get duplicated rules for the same reason.
G ABOUT USING A SECOND MODEL TO GENERATE PARAMETERS
In DRUM (Sadeghian et al. 2019) and NeuralLP YYngg et al. 2017). they applied a second RNN model to generate the probabislitic logical rule parameters. This might originate from the consid- eration of reducing parameter space sharing useful information across similar predicates. We also compared performance with and without using a second model to generate probability weight pa- rameters when reproducing results in DRUM. Interestingly, both settings performed more or less similarly on graph completion. For example, on all metrics on Family, UMLS and Kinship, the absolute score difference is in range scorewithut scorewah E [-0.01, +0.02]. The only significant difference is that using a second model to generate parameters helps DRUM converge faster. This might be because the second model serves as a regularization and enables more efficient parameter- space search, but provides little performance gains. So in our paper, we directly use learnable parameters for more elegant representation of the proposed approach.
H ADDITIONAL RELATED WORK
Neural Network Models for Graph Reasoning Using deep neural networks for reasoning on knowledge graph and relational data is also important direction. There are methods that use
18

---

Published as a conference paper at ICLR 2023
Logical rule:
23 sister(zy, x)) A mother(z,;)) A son(y, = brother(y,
0.83
LERP
(m)
Ln(e)
y
(2)
(
1
0.95 y
0.03 brotherly..)
0.00 sisten(y. 2o)
22
(
L
1.00: : chasiing
0.99
0.94 son(ay.
0.04: motherts 5y) brother(eg.a)) 0.01
0.00:
0.00 chaining
0.76
chaining predicates
0.96 sister(ag.)
2
0.01 brother(z,
0.29 morker((,.
0.24 father(;
0.23 son(x,
0.75
(
Laata
Figure 7: A diagram showing example logical rule expressed in our model.
recurrent neural networks (RNN) such as Kaur al. (2019) and graph neural networks (GNN) such as CoMPILE (Mai et al.
2021) ExpressGNN (Zhang et al. 2019) ConGLR (Lin et al. 2022) and
GralL (Teru al.] (2020). Our work, in contrast, focuses more on explicit modelling logical rules
for better interpretability.
Soft Logic, Probabilistic Logic and Natural Logic Besides logic rule learning for knowledge graph completion, there are also important work other domains for learning complex logical rules. For example, probabilistic soft logic applies machine learning framework for developing probabilis-
tic models (Bach et al. 2017 Srinivasan et al. 2022; Pryor et al.) 2022; Hu et al. 2016 Manhaeve
ct al. 2018). Markov logic network (MLN) (Richardson & Domingos, 2006 Mihalkova & Mooney]
2007 Bach et al. 2017 Domingos & Lowd 2019) is another probabilistie logic approach which ap-
plies Markov network to model first-order logic. A similarly related domain is statistical relational
learning, which aims at building domain models for relational structure (Getoor et al.) [0011] Koller
al. 2007; Domingos & Richardson, [001]] Nickel et al.) 2011). Finally, this work also related
to neural or neural-symbolic approaches for modelling logic, such as Mao et al. (2018) Dong et al.
(2018 Yi ct al. (2018) Marra & Kuželka (2021); 1: Qu & Tang (2019).
Structure Learning of Graphical Models Graphical models are computational models for in- terdependency among random variables. Therefore, learning of probabilistic logical rules is also
related graphical model structure learning like Tsamardinos al. (2006); (Spirtes al. (2000);
Heckerman et al. (1995); Friedman et al. (2013): Hartemink et al. (2000); samardinos et al. 2006 Schmidt et (2007); Lukashin et al. (2003).
I NUMBER OF PARAMETERS
The number of parameters indeed provides helpful information for comparison among models. On Family dataset example, the total number parameters in our approach 46k, with 31k pa- rameters in chains and 15k parameters in the LERP representation. On other datasets, we use 76k parameters for Kinship, 175k for UMLS, and 50k for WN18 and WN1SRR. The number mainly depends on the number of predicate types. The DRUM baseline. however, uses 802k parameters on Family dataset, most of which (796k) are the LSTM cells. DRUM uses 812k parameters for Kinship, 828k for UMLS and 807k for WN18 and WN18RR.
J DIAGRAM ILLUSTRATION OF LERP
In Figure 7 we plot a diagram of the networks that express an example rule. The example rule is the same rule the first example in Table 3 At the right side of the figure, we show the main
19

---

Published as a conference paper at ICLR 2023
chain and its parameters. At the left of the figure, we represent LERP and the matrix of intermediate functions. Note that in practice, we added Lu(z) = true as an additional logical function in LERP. This is theoretically equivalent to the formulation in the main paper, but helps our model converge more stably. We also omit the selection edges that are not associated with maximum weight. For example, in the figure, the first variable in the chain selects logical function Iı with weight 0.75, and we omit edges from z that show other logical function selections with less weights in the diagram.
20


Reviews:
[{"review_id": "aj9gdZEONm", "reviewer": "Reviewer_dYXp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes LERP, a differentiable ILP method that mines FOL rules from knowledge graphs. Compared to the prior backward-chaining methods which only learn chain-like paths in the graph, the authors propose to learn more expressive family of rules by considering the local subgraphs along the main path. In the experiment, LERP is evaluated with several ILP and graph embedding methods and shows better performance and interpretability.", "summary_of_the_review": "While this work presents an interesting extension to the existing ILP methods, there are some serious presentation issues to be fixed. At this stage, I cannot recommend for acceptance but I will be happy to raise my score if the authors could address my concerns and I'll be looking forward to reading the revised draft.", "strength_and_weaknesses": "Strength \n- Approaches an important problem of learning more expressive FOL rules with ILP\n- LERP learns a more expressive family of rules than prior ILP methods.\n\nWeaknesses\n- Presentation should be largely improved. The current draft, especially section 4, is difficult to follow and misses some important technical details.\n- The learnable rule family seems to be still far from being considered as local subgraphs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8pDhgj59O7V", "reviewer": "Reviewer_N1P8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes LERP (Logical Entity RePresentation), a model that\nuses logical rule learning, but which embeds information about the\nobjects represented by the logical variables in the form of a vector\nof logical functions.", "summary_of_the_review": "This paper proposes LERP (Logical Entity RePresentation), a model that\nuses logical rule learning, but which embeds information about the\nobjects represented by the logical variables in the form of a vector\nof logical functions.\n\nRelated work on logic rule learning does not cite many important\nreferences about the main method for learning first order rules:\ninductive logic programming. The same happens with the related work on\nProbabilistic logical rule learning where authors omit refs to\nProbLog, CLPB(n), SlipCover, ProbFoil, SKiLL, among others.\n\nThere are also several works that are neurosymbolic like RNN, and\nothers which try to capture the neuronal network structure from the\nrelations in the data (e.g., Kaur et al. and Sourek et al).\n\nThe work looks quite interesting as it presents an alternative way and\noptimization to find interpretable models. Just out of curiosity,\nauthors say that RNNLogic uses 100 to 200 rules and DRUM uses 1 to\n4. I don't know if I understood that well. ILP systems such as Aleph\nor Foil can learn thousands or millions of rules in a decente amount\nof time. How much time does your method take to learn the\nprobabilistic logical rules? The search for subgraphs for entities\n(logical variables) is naturally implemented in ILP systems. It would\nbe interesting to compare the probabilistic logical rule learning\nmethod used in this work with a probabilistic ILP system like, for\nexample, ProbFoil.", "strength_and_weaknesses": "Pros:\n+ It explores a new alternative to learn probabilistic logical rules\n+ Paper is well written and organized\n\nCons:\n- Related work on logic rule learning does not cite many important\nreferences about the main method for learning first order rules:\ninductive logic programming. The same happens with the related work on\nProbabilistic logical rule learning where authors omit refs to\nProbLog, CLPB(n), SlipCover, ProbFoil, SKiLL, among others.\nThere are also several works that are neurosymbolic like RNN, and\nothers which try to capture the neuronal network structure from the\nrelations in the data (e.g., Kaur et al. and Sourek et al).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8nO7TXsCf8Y", "reviewer": "Reviewer_F1BJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for learning complex logical rules considering semantics hidden in subgraphs rather than simple chain-like rules. The authors conduct extensive experiments on both large KG datasets and relatively small statistical rule learning datasets. ", "summary_of_the_review": "This paper is somehow interesting, but considering key related works are missing, and the novelty may be weakened compared to the authors claim. ", "strength_and_weaknesses": "### Strength\n1. The method can learn logical rules more complex than traditional chain rules. \n2. The rules learned by this method are interesting (table 3). \n\n### Weakness\n1. The novelty seems limited considering the existing works NLIL[1], which is highly-related but missing in this paper. NLIL[1] is a related work proposing tree-like rules, which needs to be analyzed and compared. (Although the defined tree-like rules are somehow diffferent)\n2. In page 5, the physical meanings of the above equations about $\\mathbf{v}$ require more explanations. It the value in $\\mathbf{v}$ always in (0, 1) (even after summation and multiplication)? Otherwise $1- \\mathbf{v}$ would be weird. \n3. As mentioned in paper, $p$, $\\lambda$, etc. are learnable parameters. I wonder whether the authors have considered use some model to generate such parameters? For example, DRUM use RNN to generate the weight $\\alpha$. \n\n[1] Yang Y, Song L. Learn to Explain Efficiently via Neural Logic Inductive Learning[C]//International Conference on Learning Representations. 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "45zTdSbjyTP", "reviewer": "Reviewer_i7dT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is well motivated, citing the rather broad literature on learning logical and probabilistic rules for KG completion, and proposing a method that can learn non-chain-like rules. The method makes sense. They show good results on several KG completion datasets, comparing against relevant baselines.", "summary_of_the_review": "The paper presents a clearly motivated model for knowledge graph reasoning that solves problems with existing methods and compares against relevant baselines. The examples of learned rules are greatly appreciated, and the experimental analysis is thorough. The KG completion community would benefit from this paper’s publication.", "strength_and_weaknesses": "Love the examples of logical rules discovered from the family dataset. These kind of examples are great for getting a sense of the method. Maybe in the supplementary material, a diagram of the networks that express some of those rules could be helpful to see.\n\nThe description of the model itself could due with being a little more computationally grounded. The probability / alpha / beta parameters are constrained to the simplex, which presumably is accomplished with a softmax. Being more explicit about this could be helpful.\n\nMore importantly, listing the entire # of parameters each method uses for each of the datasets would be helpful for comparison.\n\nThere is a lot of work on soft logic, it would be nice to have a bit more detailed literature review here.  While the distinction between horn clauses and the richer clauses you learn here is clear, there is certainly work outside of KG completion that learns more complex probabilistic rules, e.g. graphical model structure learning, probabilistic soft logic, natural logic, Markov logic networks, etc.\n\nWhen combining with TransE, do you retrain your own TransE for the baseline for apples-to-apples? The original results in the TransE paper are quite a bit lower than what you can get with a well trained TransE using modern methods.\n\nNits:\n\n- For \"Embedding-Based Neural Graph Reasoning\", could cite methods based on box embeddings (query2box, BoxE, etc)\n\n- 5.1 “knowldge graph completion” should be “knowledge graph completion”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster