INFER: A Neural-symbolic Model For Extrapolation Reasoning on Temporal Knowledge Graph

Authors: Ningyuan Li, Haihong E, Tianyu Yao, Tianyi Hu, Yuhan Li, Haoran Luo, Meina Song, Yifan Zhu
Primary area: learning on graphs and other geometries & topologies
Keywords: Knowledge Graph, Temporal Knowledge Graph, Temporal Rules, Temporal Validity

Abstract:
Temporal Knowledge Graph(TKG) serves as an efficacious way to store dynamic facts in real-world. Extrapolation reasoning on TKGs, which aims at predicting possible future events, has attracted consistent research interest. Recently, some rule-based methods have been proposed, which are considered more interpretable compared with embedding-based methods. Existing rule-based methods apply rules through path matching or subgraph extraction, which falls short in inference ability and suffers from missing facts in TKGs. Besides, during rule application period, these methods consider the standing of facts as a binary 0 or 1 problem and ignores the validity as well as frequency of historical facts under temporal settings.
In this paper, by designing a novel paradigm for rule application, we propose INFER, a neural-symbolic model for TKG extrapolation. With the introduction of Temporal Validity Function, INFER firstly considers the frequency and validity of historical facts and extends the truth value of facts into continuous real number to better adapt for temporal settings. INFER builds Temporal Weight Matrices with a pre-trained static KG embedding model to enhance its inference ability. Moreover, to facilitates potential integration with existing embedding-based methods, INFER adopts a rule projection module which enables it apply rules through conducting matrices operation on GPU. This feature also improves the efficiency of rule application. 
Experimental results show that INFER achieves state-of-the-art performance on various TKG datasets and significantly outperforms existing rule-based models on our modified, more sparse TKG datasets, which demonstrates the superiority of our model in inference ability.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730377868859

thread_type:
official_review

<thread_object>
creator: reviewer_qjza

<content>
summary:
This paper introduces INFER, a neural-symbolic model designed for Temporal Knowledge Graph (TKG) extrapolation reasoning. Traditional rule-based methods for TKGs, though interpretable, struggle with temporal reasoning as they treat facts as binary and ignore temporal frequency and validity. NFER addresses these issues through a Temporal Validity Function, which enables continuous truth values and models the frequency and validity of historical facts for better temporal adaptation. Additionally, INFER incorporates Temporal Weight Matrices with a pre-trained static KG embedding model to improve inference quality. A rule projection module enhances computational efficiency by leveraging GPU-optimized matrix operations, allowing INFER to scale effectively and integrate with embedding-based approaches. Experiments show that INFER achieves state-of-the-art results on several datasets, demonstrating enhanced inference capabilities over existing models, particularly in sparse TKG settings.

strengths:
1.   Temporal Knowledge Graph (TKG) reasoning is an important  research topic.
 2.   The proposed method shows better results compared to existing rule-based methods.

weaknesses:
1.  The motivation behind the method design is unclear, and the description lacks clarity. For example, in Section 4.3.3, the rationale for the rule projection strategy is not well-explained, and the meanings of terms like “Ans_i” and “Ans” are not clarified.
2.  Limited novelty. The techniques used in the proposed method do not introduce significant innovations.
3.  The temporal and spatial complexity of the inference process appears high, which might impact practical applicability. It would be beneficial to provide an analysis of the computational complexity.
4.  Some experimental details are insufficiently explained. For example, in Line 426, the phrase “traditional binary truth values for historical facts” needs further clarification. It is unclear what specific criteria or methods are used to assign these binary truth values to historical facts.
5.  The dataset coverage is insufficient. The paper only uses the ICEWS dataset, which belongs to a specific category of TKG data. It would be valuable to include additional datasets like WIKI or GDELT to demonstrate the method’s generalizability across different data types.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730471032670

thread_type:
official_review

<thread_object>
creator: reviewer_ecyd

<content>
summary:
To solve the problem that rule-based methods in the field of temporal knowledge graph inference have insufficient reasoning ability when graph facts are missing, and the fact state is simply considered as binary, ignoring the validity and frequency of historical facts, this paper proposes the neural symbol model of INFER, which quantifies fact credibility in time dimension by introducing a time validity function. At the same time, the time weight matrix is introduced so that the model can infer the missing facts and deal with the incompleteness of the map. And to improve the efficiency of rule reasoning, a rule projection module is proposed, which uses GPU-based matrix operation instead of traditional path matching.

strengths:
(1) This paper introduces effective means such as the time validity function to quantify fact credibility in the time dimension and proves its effectiveness through experiments, which provides ideas for how to improve the reasoning ability of rule-based methods in dealing with incomplete graphs.
(2) The average performance of the experiment is good on multiple datasets, and the self-made TKG data with sparse facts is obtained. The experimental results show that the INFER model is significantly better than other methods when the facts are sparse.
(3) The chart is clear, the paper is completed with a high degree, and it is easy to interpret.

weaknesses:
(1) The Design of the time validity function: The time validity function proposed in this paper calculates the time weight of historical facts based on the time interval and frequency of fact occurrence. Although the above two terms are considered at the same time, the function form is relatively fixed and more dependent on experience. Adapting the attenuation rate using data-driven methods may enhance the model's adaptability. 
(2) The performance of the model on the ICEWS05-15 dataset does not exceed that of TECHS, and there is no detailed analysis of the results in this paper.
(3) The rule projection module used by INFER loses the ability to directly model the sequence of facts to a certain extent, which may affect the accuracy in scenarios requiring strict time order or multi-jump reasoning. Additional experiments are needed for evaluation, especially for long rule samples with variable constraints. 
(4) INFER introduces neural network embedding and complex matrix operations, and although the authors show entity scores when the rules are applied, it still damages the interpretability of the traditional rule model to some extent.

questions:
(1) Is the use of the square root form of the time decay term in the time validity function based on the conclusions obtained from experiments?
(2) What is the additional overhead if you replace the static embedded model with a time-embedded model? And What's the performance boost?
(3) When calculating the rule confidence, the INFER model adopts the rule learning algorithm in TRRules. The rule base retains only cyclic rules and filters out acyclic rules. TR-Rules uses acyclic rules and proves its effectiveness. What is the reason for filtering acyclic rules here? Is there any experimental support?
(4) Does the static embedding model provide the same level of confidence for the same fact at different time points?
(5) How does INFER's rule projection module perform when dealing with rules of different lengths?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730585802817

thread_type:
official_review

<thread_object>
creator: reviewer_cyuq

<content>
summary:
The article presents a detailed neuro-symbolic model addressing the task of Temporal Knowledge Graph Completion, particularly focusing on the challenge of extrapolation. This task aims to infer knowledge for a given Knowledge Graph at time T using only past data. The authors propose a well-structured solution, providing an extensive evaluation that compares their model to current state-of-the-art symbolic and neural methods.

strengths:
1. Comprehensive Evaluation: The paper includes a thorough evaluation of the proposed model, supplemented by an ablation study and an efficiency analysis, making it easy to understand the model's effectiveness.
2. Efficient Model Design: The model circumvents the common complexity issue of creating separate matrices per timestamp, resulting in a compact, efficient design.
3. Clear and Reproducible Description: The methodology is well-articulated, allowing for straightforward reproduction of the results.

weaknesses:
1. Minor Typos and Inconsistencies:
   - Line 451/452: The term "INFER(Temp)" is used instead of "INFER(Temp Val)."
   - Line 515: The rule quantity is stated as 40, while Table 1 lists it as 60, which could lead to some confusion.
2. Efficiency Study Observations: The conclusions drawn in the efficiency study are somewhat unclear. Specifically, a competing model achieving a similar score while exploring fewer candidates might suggest that the alternative approach is better optimized in its candidate selection process. Clarifying these efficiency aspects would strengthen the study.
3. Completeness of Graph Argument: The argument concerning graph completeness and the slope behavior appears overstated, as the trend lines for the three methods are quite similar. Revising this interpretation could enhance clarity.
4. Ambiguity in Section 4.3.3: The fourth paragraph lacks clarity regarding "variable constraints in rules," which limits the reader's understanding of the approach. A clearer explanation or example would be beneficial here.

questions:
1. In Section 4.3.2, the function $V(s, r, o, t_c)$ is introduced. However, it is unclear how the model handles $t_{last}$ if the fact $<s, r, o>$ was never previously observed. Is the value set to 0, potentially conflicting with timestamp 0, or is another value assigned?
2. Could the authors elaborate on the benefits of examining a significantly higher number of candidates (100x), especially given the notable increase in runtime (an additional 500 seconds)? Understanding the trade-offs would be helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688517103

thread_type:
official_review

<thread_object>
creator: reviewer_agps

<content>
summary:
The paper introduces INFER, a neural-symbolic approach to temporal knowledge graph extrapolation.  INFER  uses a Temporal Validity Function that captures how frequently facts occur, as well as their validity over time using continuous values. INFER uses pre-trained static knowledge graph embeddings to construct Temporal Weight Matrices. A rule projection module that reformulates rule application as matrix operations. 
The paper evaluates INFER's effectiveness using three ICEWS datasets. When tested on modified sparse temporal knowledge graph datasets, INFER shows promising inference capabilities. These results highlight that INFER's combination of continuous temporal validity scoring and GPU-optimized rule application offer useful techniques for temporal knowledge graph reasoning.

strengths:
1. The paper introduces a new scoring mechanism for temporal rule validity.  Alternate techniques in the literature appear to be more complicated.

2. The paper evaluates differentiable rule-based inference systems across several ICEWS datasets, providing a demonstration of these systems on temporal knowledge graphs that represent time in event "timestamp" form.

weaknesses:
1. The author's claimed novelty rests on acceleration of rule-based processing using matrix operations on a GPU.  This is common in differentiable rule-learning systems, apparently first introduced as TensorLog, with associated inductive learning system Neural-LP, see e.g. [1].

2. The paper claims: "Experimental results show that INFER achieves state-of-the-art performance on three datasets and significantly outperforms existing rule-based models on our modified, more sparse TKG datasets, which demonstrates the superiority of our model in inference ability." The authors should consider that both embedding based and rule based systems can perform quite well relative to the methods they compare against.  For instance, consider the following comparison with TimePlex [2]:

| | ICEWS14 | ICEWS14 | ICEWS14 | ICEWS05-15 | ICEWS05-15 | ICEWS05-15 |
|--------|---------|-----|---------|----------|---------|-----|
| Method | MRR | HITS@1 | HITS@10 | MRR | HITS@1 | HITS@10 |
| TimePlex | **60.40** | **51.50** | **77.11** | **63.99** | **54.51** | **81.81** |
| INFER | 44.09 | 34.52 | 62.14 | 48.27 | 37.61 | 68.52 | 

The table below illustrates the methods similar to those compared in this paper evaluated on wikidata and yago data sub-sets.  The table also includes a rule-based method (TILP [3]) that is demonstrated to perform on par with Timeplex, illustrating that the performance gap between TimePlex and INFER show above may not be limited to embedding based methods.                                                                                   

| | WIKIDATA12k | WIKIDATA12k | WIKIDATA12k | YAGO11k | YAGO11k | YAGO11k |
|--------|---------|-----|---------|----------|---------|-----|
| Method | MRR | HITS@1 | HITS@10 | MRR | HITS@1 | HITS@10 |
| TLogic | 0.2536 | 0.1754 | 0.4424 | 0.1545 | 0.1180 | 0.2309 |
| ComplEx | 0.2482 | 0.1430 | 0.4890 | 0.1814 | 0.1146 | 0.3111 |
| TA-ComplEx | 0.2278 | 0.1269 | 0.4600 | 0.1524 | 0.0936 | 0.2626 |
| DE-SimplE | 0.2529 | 0.1468 | 0.4905 | 0.1512 | 0.0875 | 0.2674 |
| TimePlex | **0.3335** | 0.2278 | **0.5320** | 0.2364 | **0.1692** | 0.3671 |
| TILP | 0.3328 | **0.2342** | 0.5289 | **0.2411** | 0.1667 | **0.4149** |


[1] @inproceedings{10.5555/3294771.3294992,
author = {Yang, Fan and Yang, Zhilin and Cohen, William W.},
title = {Differentiable learning of logical rules for knowledge base reasoning},
year = {2017},
isbn = {9781510860964},
publisher = {Curran Associates Inc.},
address = {Red Hook, NY, USA},
abstract = {We study the problem of learning probabilistic first-order logical rules for knowledge base reasoning. This learning problem is difficult because it requires learning the parameters in a continuous space as well as the structure in a discrete space. We propose a framework, Neural Logic Programming, that combines the parameter and structure learning of first-order logical rules in an end-to-end differentiable model. This approach is inspired by a recently-developed differentiable logic called TensorLog [5], where inference tasks can be compiled into sequences of differentiable operations. We design a neural controller system that learns to compose these operations. Empirically, our method outperforms prior work on multiple knowledge base benchmark datasets, including Freebase and WikiMovies.},
booktitle = {Proceedings of the 31st International Conference on Neural Information Processing Systems},
pages = {2316–2325},
numpages = {10},
location = {Long Beach, California, USA},
series = {NIPS'17}
}

@inproceedings{jain-etal-2020-temporal,
    title = "{T}emporal {K}nowledge {B}ase {C}ompletion: {N}ew {A}lgorithms and {E}valuation {P}rotocols",
    author = "Jain, Prachi  and
      Rathi, Sushant  and
      {Mausam}  and
      Chakrabarti, Soumen",
    editor = "Webber, Bonnie  and
      Cohn, Trevor  and
      He, Yulan  and
      Liu, Yang",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2020.emnlp-main.305",
    doi = "10.18653/v1/2020.emnlp-main.305",
    pages = "3733--3747",
    abstract = "Research on temporal knowledge bases, which associate a relational fact (s,r,o) with a validity time period (or time instant), is in its early days. Our work considers predicting missing entities (link prediction) and missing time intervals (time prediction) as joint Temporal Knowledge Base Completion (TKBC) tasks, and presents TIMEPLEX, a novel TKBC method, in which entities, relations and, time are all embedded in a uniform, compatible space. TIMEPLEX exploits the recurrent nature of some facts/events and temporal interactions between pairs of relations, yielding state-of-the-art results on both prediction tasks. We also find that existing TKBC models heavily overestimate link prediction performance due to imperfect evaluation mechanisms. In response, we propose improved TKBC evaluation protocols for both link and time prediction tasks, dealing with subtle issues that arise from the partial overlap of time intervals in gold instances and system predictions.",
}

[3] @inproceedings{xiongtilp,
  title={TILP: Differentiable Learning of Temporal Logical Rules on Knowledge Graphs},
  author={Xiong, Siheng and Yang, Yuan and Fekri, Faramarz and Kerce, James Clayton},
  booktitle={The Eleventh International Conference on Learning Representations}
}

questions:
1. How does your acceleration method differ in nature from TensorLog and Neural-LP?
2. Many methods have developed from the Neural-LP approach.  How does your method compare to those?
3. Why do you not compare to TimePlex as the state of the art method that has been demonstrated on these ICEWS datasets?
4. Can the gap in performance with TimePlex and rule-based sysetsms be explained in terms of the differences between timestamp and interval-time objectives of different temporal knowledge graph methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732778548374

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank the reviewers for their constructive feedbacks and we upload our revised PDF based on the discussions we have with all reviewers. Main changes we made in the revised version are (changes are highlighted with brown font):

* Add clarification about Interpolation Reasoning and Extrapolation Reasoning mentioned by Reviewer agps.
* Add examples and illustration for Variable Constraints and clarify the efficiency analysis part suggested by Reviewer cyuQ.
* Add experiments evaluating the performance of INFER using rules with different lengths suggested by Reviewer EcyD.
* Add experiments on various TKG datasets: WIKI and YAGO suggested by Reviewer qjZA.
</content>
</thread_object>

</thread>
<thread>
creation date:
1751440347195

thread_type:
public_comment

<thread_object>
creator: ~julia_gastinger1

<content>
title:
Comment on evaluation protocol: potential issue with "best ranking" tie-break

comment:
I would like to raise a concern about the reported metrics (MRR and Hits@1/3/10). Based on inspecting your public code, it appears your metric computation uses what [Sun et al., 2020] describe as the “best” ranking protocol in the presence of ties.

Specifically, in your repository ([guided_apply.py](https://github.com/JasonLee-22/INFER/blob/main/guided_apply.py)), the evaluation code around line 660 (see attached screenshot with inline annotations) handles ties by assigning the *lowest* (i.e. best) possible rank to the ground-truth entity if multiple entities share the same model score:

```
ranking = (sort_score[i, :] == target_score[i]).nonzero()[0]
ranking = 1 + ranking.item()
```
This line selects the indices where the sorted score list matches the score of the ground-truth node, and then takes the first such index — i.e., the smallest one. As a result, the evaluation always assigns the best possible rank to the ground-truth entity in the case of ties, rather than using an average or random tie-breaking strategy..

This protocol is known to inflate evaluation metrics unfairly, since it does not reflect the true ranking uncertainty in the presence of score ties. Sun et al. recommend fairer alternatives (such as assigning the average rank over tied entities).

The reported results in the paper appear to rely on this tie-breaking strategy. I therefore strongly suggest re-running evaluation with a fairer tie-breaking method (e.g., average or random ranking) to ensure valid and comparable metrics.

[Sun et al., 2020] Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha P. Talukdar, and Yiming Yang. A reevaluation of knowledge graph completion methods. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5516–5522, 2020.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The proposed method INFER offers a promising approach to temporal knowledge graph reasoning. The majority of the reviewers’ concerns have been addressed, and the authors have provided sufficient clarifications, additional experiments, and analysis to strengthen the manuscript. Given the promising results, the addressed concerns, and the contributions made by the authors, I recommend accepting the paper.