Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper mainly works on knowledge graph reasoning with temporal shift, which is an underexplored problem in real-world setting where the new entities will evolve over time. The challenges mainly lie in few-shot and time shift perspective. In order to solve this problem, this paper proposes a MetaTKGR framework, which can adjust the strategy of sampling and aggregating neighbors from recent facts for the new entities. The paper uses theoretical analysis to provide PAC generalization bound to prove its learnability. 

The empirical results shown in the paper indicate strong performance across the board. Under 1-shot and 2/3-shot cases, the model is able to achieve consistent performance gain over different embedding-based KGR models.1
strengths_and_weaknesses: Strength:

1. This paper discovers a novel problem of knowledge graph reasoning over temporal shift.
2. This paper proposes MetaTKGR accompanied with theoretical analysis.
3. The empirical results are strong.


Weakness:

1. The propose method is a bit hard to follow, consisting a total of (5) steps indicated in Figure 2. It would be great to associate the figure with different subsections to help readers follow the logic more closely. 
questions: N/A
limitations: The compared baseline models are all embedding-based knowledge graph reasoning models, it's not clear whether the proposed method could be used for path-based reasoning models. The path-based reasoning models can normally provide better explainability while achieving better results.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigated a novel problem, few-shot temporal knowledge graph reasoning, that aims to predict future facts for newly emerging entities based on limited observations in evolving graphs. To address this problem, the paper proposed a novel framework, namely MetaTKGR, which aims to extract the global knowledge of sampling and aggregating temporal neighbors, which can be adapted to new entities for future prediction. The experiment results on three real-world temporal knowledge graphs are promising.
strengths_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow.
2. The motivation is clear and interesting.
3. The experimental results and discussion are thorough.

Weaknesses:
1. It is a little hard to understand how the MetaTKGR can address the two challenges, few-shot and time shift. Maybe adding some specific examples for visualization is helpful.

questions: 1. Where is the error bar in Section 4 of the main paper?
limitations: No.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to predict future facts about *newly emerging entities in temporal knowledge graphs (TKGs)* using few-shot learning. The authors demonstrated the challenge of predicting facts in temporal knowledge graphs as opposed to static knowledge graphs; temporal graphs evolve over time, making reasoning tasks (such as linking prediction) more complex. These entities pose several challenges, such as few-shot relationships and time-shifting, where characteristics may change over time. Furthermore, the previous works address this task by ignoring the emergence of new entities over time and randomly simulating unseen entities on static knowledge graphs. 
In this regard, the authors propose **MetaTKGR**, a meta-learning framework for reasoning over temporal knowledge graphs. Here, the meta-learning technique is employed to dynamically learn the strategies for sampling and aggregating neighbours in TKGs. 
The authors conducted experiments on three public datasets to verify the significance of their approach. The main contributions of this paper are two-fold: I) the authors presented a formal formulation for few-shot temporal graph reasoning. II) The authors proposed a meta-learning framework for solving this problem. In the following, I list my detailed comments on the points raised in the paper. 
strengths_and_weaknesses: * The paper is well-written, and almost parts are easy to follow. The authors described the research problem in detail and clarified the motivation behind their approach.
* The problem statement is formally defined with mathematical explanations. The authors formulate clearly the problem of fact reasoning of new entities in temporal knowledge graphs, given challenges: few known facts and time-shift, which makes it easier for the readers to understand it very well. 
* This paper presents an interesting research problem, specifically for time-sensitive applications, and is relevant to the NeurIPS conference. 
* The authors conducted several experiments on different datasets, including different time periods:
   * A detailed description of the experimental setting is provided to reproduce the results. 
   * The experimental results demonstrated the significance of the proposed approach. The results are reported on five independent runs, with a significance test p-value < 0.001.  
   * Further, the authors analysed the experimental results from a theoretical perspective.

questions: - There is no clear indication if the future facts to be predicted are predefined based on the entity type?
- How the quadruples of entities are split: into support and query sets randomly? In which ratio, Is it a hyper-parameter? 
- Line 284, it's unclear how the proposed approach outperforms all baselines with the same ratio of 11.4% (relative). Given that the baselines are different in structure and methodology, it's expected to perform differently and show different results. 
- Line 271, It's unclear why the authors split the time duration into four periods. Does this need to be clarified? Is it suitable for all datasets? 
* The approach presented in [1] is comparable to this paper and employed the same datasets in the experiments. It makes sense to benchmark the performance against it also as a baseline.

* Line 353 and 354, the authors discuss some related works designed for reasoning tasks for static/temporal homogeneous graphs. However, most of these approaches are designed for few-shot learning on a static knowledge graph. Some recent methods [1,2,3] have been developed for temporal knowledge graphs reasoning and are not discussed in the related work section to give readers an overview of other techniques.

[1] https://dl.acm.org/doi/abs/10.1145/3404835.3462963 

[2] https://dl.acm.org/doi/pdf/10.1145/3488560.3498451 

[3] https://ieeexplore.ieee.org/abstract/document/9712786 

limitations: Yes
review_title: null

Meta-review: metareview: This paper studies few-shot temporal knowledge graph reasoning. The task looks to predict future facts for newly emerging entities based on limited observations in evolving graphs. The authors propose MetaTKGR, a meta-learning framework for reasoning over temporal knowledge graphs. 

The reviewers agree that the proposed method is reasonable and sound, the experiments are thorough, and the results provide valuable insights for future work. Reviewers' raised concerns and questions are properly addressed by the author's response. 
confidence: Certain
Decision: Accept