Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to address the knowledge graph reason where, besides entity and relation representation, first order logic is also considered. In doing so the authors propose to enrich RotatE with a RNN for embedding score and a separate grounding embedding for scores.Two scores are shallowly composed in the inference time. The authors demonstrate the performance of the proposed method with the link prediction tasks

strength_and_weaknesses: # Strength
The paper tackles an interesting task in knowledge graph embedding .The paper is generally well-written (but with some issues in clarity, see below).

# Weakness

There are some issues in the current format as detailed below:

1. There are several important aspects missing regarding logic and grounding, which are the core contribution of this work over previous ones. The exact training schema for grounding, including the training objects, are quite unclear in the paper. This makes it hard to judge the contribution of this paper.

2. There are some unclarity on the contribution of each part in the proposed method. The pre-training part, which is largely resembling RNNLogic (Qu et al.,), is pretty loosely composed with the grounding technique (a major contribution in this manuscript) in the inference. I'm afraid this only constitutes a limited contribution. Furthermore, it's not mentioned how $\beta$, the hyper-parameter to balance two parts, is selected.

3. This is some unclarity regarding the dataset. It's unclear what the distribution of first-order logic looks like (e.g. number of terms). This is important as such statistics would determine whether the proposed use of RNN is justified or there are better alternatives.

clarity,_quality,_novelty_and_reproducibility: Overall the writing is good, except for the important details regarding grounding. The originality of the work is okay. However, the contribution is limited.

summary_of_the_review: Based on strengths and weaknesses, I would recommend 5: marginally below the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Combines rule-based and embedding-based reasoning methods for link prediction in knowledge graphs. Key idea is to learn an embedding of chain rules via an RNN during KGE training. During inference, combines KGE score of target triple and rule-embedding score of rules that fire on it.
strength_and_weaknesses: Strengths:

S1. Relatively simple approach (but see W1)

S2. Some improvements over baselines (but see W1)

Weaknesses:

W1. Does not outperform simpler methods / falls behind SOTA. The paper, in its essence, combines the score of a KGE model with a score obtained from fired rules. This is done in a more direct fashion in [A], with similar or better results than presented in this paper. Also, the results are from from SOTA [B]. Both papers are not discussed; related work is not covered well.

W2. Approach not well justified. The paper presents its method clearly, but doesn't justify it. Personally, I am not convinced of the approach, perhaps for this reason. One point of concern is that the RNN, which produces the rule embeddings, is trained in a complete data-independent fashion (Eq. 7). Why such an approach would work better than augmenting the training process as done in prior work is unclear to me.

W3. Only chain rules supported.

W4. Study not insightful. The experimental study shows some results, but does not provide any additional insight. Where is it, exactly, that the proposed method can outperform alternatives?

Other points:

D1. The experimental study is silent on where the rules come from.

D2. The weight w_i of Eq. 9 doesn't appear to be a "confidence" in that it does not have a probabilistic interpretation.

D3. How is $R^(g)$ trained (Sec. 4.2)? Is it the output of the RNN? Or trained directly?

D4. The main paper describes a method for grounding, pointing to details in the appendix. The appendix describes a completely different method, however. This is highly confusing.

References:

[A] Meilicke et al., Why a Naive Way to Combine Symbolic and Latent Knowledge Base Completion Works Surprisingly Well. AKBC 2021

[B] Zhu et al., Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS 2021

clarity,_quality,_novelty_and_reproducibility: Understandable writing, but motivation/justification missing. Partly repeats unnecessary details (e.g., RotatE model), partly lacks important details (e.g., points D3/D4).

Quality OK. Novelty unclear.

Reproducibility unclear (D1).

summary_of_the_review: Provides improvements over baselines, but does not outperform simpler methods and is far from SOTA (both not cited/discussed).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes RulE, a model for link prediction on knowledge graphs that jointly learns representations for entities, relations and rules in the embedding space. More specifically, RulE applies on top of the RotatE KGE model, and additionally includes a component for learning compositional rule embeddings: Given a composition rule r1(x_1, x_2) ^ r2(x_2, x_3) ... ^ r_k(x_k,+1) -> r_k+1(x_1, x_k+1), RulE applies an RNN over the relational representations of r_1, r_k to produce an order-specific representation for the rule.  This rule representation is then compared to r_k+1 as part of pre-training in the loss function as a separate term. Beyond pre-training, RulE uses grounding to establish weights for the rules provided to the model and compute a score for a fact based on the rules. This score then acts as a complementary measure to support the embedding-based score at inference time. Empirically, the model is evaluated on standard knowledge graph completion baselines and is shown to achieve competitive performance. Finally, the model is run on reasoning baselines UMLS and kinship and achieves state-of-the-art results.
strength_and_weaknesses: # Strengths: 
- The empirical performance of RulE is good. 
- The intuition for representing rules in the same space as relations and entities is well-placed and a good inductive bias, though I have some concerns about the way it is done in this paper (see weaknesses below).

# Weaknesses: 
- The paper claims that the way it includes rules is principled, in that all representations are in the same embedding space. This is technically correct, but is problematic in practice for many reasons. First, the semantics of the rule component are not compatible with those on the KGE side. That is, relation embeddings act as rotations in RotatE, but are used as RNN tokens without any observable semantics on the rule side, with no connection to their functional role in RotatE. Hence, this unity between representations does not provide any meaningful insights or constraints aside from the technical parameter sharing. 

- Second, the rule incorporation conducted in this work is not interpretable, and offers no means to study the inductive capacity of the model. Indeed, the scoring of rule satisfaction depends on a black-box RNN model, which makes studying combinations of rules or the set of rules captured by RulE virtually impossible. As a model trying to inject the symbolic structure of rule embedding, the lack of interpretable semantics on the rule embedding side severely undermines the contribution of the work. To illustrate this point, the rules in RulE all pass through the RNN, but there is no means to assess which potentially hidden inference patterns can be/are captured in the resulting model, whereas this can be done (albeit to a limited extent) in plain RotatE. Hence, using RulE effectively costs KGE models their interpretability on the inference patterns side. 

- Third, the set of rules being injected by RulE is restricted only to composition rules, and does not present a viable means to represent more general rules: Effectively, new components, e.g., other RNNs, must be introduced for different types of rules, which also loses the structure and semantics of said rules. In fact, it is not even clear how RulE can handle multiple sets of composition rules in a sound way, for instance by respecting potential deductive closures. For example, the rules r_1(x, y) ^ r2(y, z) -> r3(x, z) and r3(x, y) ^ r4(y, z) -> r5(x, z) imply a longer chain rule r_1(x, y) ^ r2(y, z) ^ r4(z, w) -> r5(x, w), but such a deduction cannot be guaranteed to hold, let alone read through the RNN representations of the rules. 

- Fourth, the approach is based on a more sophisticated neural network architecture (an RNN) and introduces additional hyper-parameters to its base model. Moreover, it introduces a potentially very costly grounding step prior to inference. As a result, the RulE model appears problematic not only from a scalability perspective, and this is acknowledged by the authors, but also from a parametrization and tuning perspective. Indeed, RulE achieves slightly improved results, but at the cost of interpretability, respecting the semantics of logical rules, and a larger parametrization. This unfortunately makes this setup less convincing overall.

- On a different point, the paper presents its hybrid use of rule and entity/relation representation in the same space as a main contribution, and mentions related work along two categories: KGE models with potential rule regularization, and reasoning systems without embedding structure. However, it fails to mention the BoxE model (Abboud et al., 2020), which can inject a set of rules from a rule language naturally as part of its relation representations and enforce them interpretably and with guarantees. Granted, RulE allows for softer rule injection, where a rule can technically fail, whereas BoxE does not. However, RulE in return does not provide any guarantees, supports far less rules, does not study its own inductive capacity despite emphasizing the importance of encoding rules, and proposes a less unified treatment of relation representations that differs between the fact and rule levels. 

[1] Ralph Abboud et al. BoxE: A Box Embedding Model for Knowledge Base Completion. NeurIPS 2020.
clarity,_quality,_novelty_and_reproducibility: # Clarity: 
The main ideas in the paper are presented clearly.

# Quality: 
The ideas proposed in this paper are technically correct, but introduce several undesirable effects, particularly for a model seeking to introduce symbolic structure into the KGE domain. 

# Novelty: 
RulE does introduce any particularly novel components: The use of an RNN to inject composition rules in an order-aware fashion is a minor contribution, and it unfortunately comes with multiple side-effects, as discussed earlier. Moreover, the grounding and inference with additional hyper-parameters are also components used in other works in the literature. 

# Reproducibility: 
No concerns about reproducibility.
summary_of_the_review: Overall, the unification of entity, relation and rule embeddings in a single embedding is a desirable property. However, in RulE, this is done in a problematic fashion, using black-box components that introduce i) inconsistent and incompatible objects between fact-level and rule-level uses of relational embeddings, ii) sacrifice the interpretability of the input KGE model (if already interpretable) particularly with respect to the study of inference patterns, iii) can only enforce a limited type of rules (compositions), and provides no soundness guarantees, e.g., capturing deductive closures of rules, and iv) substantially increase the parametrization and complexity of KGE models for minimal empirical gain. The contributions of RulE are also not completely well-placed in their context, as other works such as BoxE inject richer rule languages with provable guarantees. Therefore, I currently lean toward rejecting the paper. However, I am happy to change my verdict should the authors address my above concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new knowledge graph reasoning (KGR) model that combines embedding-based and rule-based methods. The authors claim that this is the first attempt to embed entities, relations, and logical rules into a unified space for KGR.
strength_and_weaknesses: ## Strength
1. The idea of embedding both knowledge graphs and logical rules in a unified space and learning them simultaneously is reasonable.
2. The preliminaries are well introduced and related works are fully discussed.
3. Detailed experimental settings and codes are provided for reproducibility.

## Weakness
1. **Motivation**     
At the top of Page 2, the authors say that one of the limitations of current KGE is that they can only be used in transductive scenarios, and rule-based methods do not have this shortcoming. However, it seems that the proposed RulE does not resolve this limitation and is still transductive.
2. **Method**    
The authors may want to analyze the complexity of RulE. Using RNNs to encode paths is time-consuming, especially when there are a lot of rules in a knowledge graph.
3. **Experiments**    
My major concern is about the experimental results.     
3.1 Note that the authors claim that RulE is proposed for better knowledge graph completion (KGC). However, RulE does not show clear performance improvements on the commonly-used KGC datasets (WN18RR and FB15k-237), which demonstrates the poor applicability of RulE. I agree with the authors' analyses that this is because WN18RR and FB15k-237 consist of more general facts and the phenomenon is also observed in previous works. However, the aim of the previous works is to learn logical rules instead of improving KGC performance with logical rules.    
3.2 In the caption of Table 1, the authors claim that [*] means the numbers are taken from the original papers. However, I cannot find the corresponding results in these papers (e.g., TuckER and RotatE). In my experience, KGE models such as RotatE perform very well on UMLS and Kinship, which is far greater than the reported results.
clarity,_quality,_novelty_and_reproducibility: ## Clarity  
Overall, this paper is clearly written. However, the authors spend too much space introducing properties of existing methods. For instance, on Page 6, the example that discusses the commutativity of RotatE can be removed.

## Quality  
The proposed approach is overall reasonable. The experimental results are not significant and some results are unconvincing.

## Novelty  
The idea of learning rules and graph embeddings simultaneously is not new, but the way to realize it in a unified space seems interesting.

## Reproducibility    
Detailed experimental settings and codes are provided, so the reproducibility looks nice.
summary_of_the_review: This paper proposes an interesting idea, but the significance of experiments should be greatly improved before it can be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: SUMMARY

The paper combines rule-based and embedding-based reasoning
methods for link prediction in knowledge graphs. The key
idea is to learn an embedding of chain rules via an RNN
during KG embedding training. During inference, they combine
the KG embedding score of a target triple and rule-embedding
scores of rules that fire on it.

STRENGTHS

Interesting problem addressed

Some improvement over baselines

Paper is mostly written well

WEAKNESSES

Important related work is not discussed. The experimental
results of the paper are below those in this related
work. Some of the comparison figures from related work were
incorrect in the original submission.

Lack of motivation for the proposed approach (why should a
data-independent modeling approach be able to learn good
rule embeddings; this is also related to lack of
interpretability: in a blackbox approach it is not clear
what was learned)

The description of the proposed method in the original
submission is insufficient.

The description of the dataset in the original
submission is insufficient.

The paper combines two elements, one similar to previous
work, one novel one. It is not clear how much each part
contributes. That means it is not clear how much the novel
contribution of this paper contributes.

There is a suspicion that part of the good performance of
the model is due to the introduction of more parameters

Lack of analysis of results




justification_for_why_not_higher_score: Concerns about comparison to prior work, about the motivation of the approach and about novelty
justification_for_why_not_lower_score: no lower score possible
Decision: Reject