SiMFy: A Simple Yet Effective Approach for Temporal Knowledge Graph Reasoning
Authors: Zhengtao Liu
Lei Tan
Mengfan Li
Yao Wan
Hai Jin
Xuanhua Shi
Venue: EMNLP 2023 Findings (2023)
Keywords: Temporal Knowledge Graph
Reasoning
Multilayer Perceptron
Historical Frequency

Abstract: Temporal Knowledge Graph (TKG) reasoning, which focuses on leveraging temporal information to infer future facts in knowledge graphs, plays a vital role in knowledge graph completion. Typically, existing works for this task design graph neural networks and recurrent neural networks to respectively capture the structural and temporal information in KGs. Despite their effectiveness, in our practice, we find that they tend to suffer the issues of low training efficiency and insufficient generalization ability, which can be attributed to the over design of model architectures. To this end, this paper aims to figure out whether the current complex model architectures are necessary for temporal knowledge graph reasoning. As a result, we put forward a simple yet effective approach (termed SiMFy), which simply utilizes multilayer perceptron (MLP) to model the structural dependencies of events and adopts a fixed-frequency strategy to incorporate historical frequency during inference. Extensive experiments on real-world datasets demonstrate that our SiMFy can reach state-of-the-art performance with the following strengths: 1) faster convergence speed and better generalization ability; 2) a much smaller time consumption in the training process; and 3) better ability to capture the structural dependencies of events in KGs. These results provide evidence that the substitution of complex models with simpler counterparts is a feasible strategy.

SiMFy: A Simple Yet Effective Approach for Temporal Knowledge Graph Reasoning
Zhengtao Liu, Lei Tan, Mengfan Li, Yao Wan, Hai Jin, Xuanhua Shit
National Engineering Research Center for Big Data Technology and System,
Services Computing Technology and System Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology,
Huazhong University of Science and Technology, Wuhan, China (zhengtaoliu, tanlei, 1imf, wanyao, hjin, xhshi Jehust eecphustteuu.... cn
Abstract
Temporal Knowledge Graph (TKG) reasoning, which focuses on leveraging temporal informa- tion to infer future facts knowledge graphs, plays vital role in knowledge graph comple- tion. Typically, existing works for this task de- sign graph neural networks and recurrent neural networks to respectively capture the structural and temporal information in KGs. Despite their effectiveness, our practice, we find that they tend to suffer the issues of low training effi- ciency and insufficient generalization ability, which can be attributed the over design of model architectures. To this end, this paper aims to figure out whether the current complex model architectures necessary for temporal knowledge graph reasoning, As a result, we put forward a simple yet effective approach (termed SiMFy), which simply utilizes multilayer per- ceptron (MLP) to model the structural depen- dencies of events and adopts a fixed frequeey strategy incorporate historical frequency dur- ing inference. Extensive experiments on real- world datasets demonstrate that our SiMFy can reach satte-of-the-a performance with the fol- lowing strengths: 1) faster convergence speed and better generalization ability; 2) much smaller time consumption in the training pro- cess; and 3) better ability to capture the struc- tural dependencies of events in KGs. These results provide evidence that the substitution of complex models with simpler counterparts is feasible strategy.
Vine
Tiae II
Viane III
Time
Figure 1: An illustration TKG reasoning, showing knowledge graphs at three distinct timestamps. Directed edges represent relations, originating from the subject entity and terminating the object entity. Our objective is predict the object entity at the final timestamp.
However, traditional KGs struggle to effectively handle facts that have temporal characteristics. Therefore, Temporal Knowledge Graphs (TKGs) have been introduced to tackle this challenge (Wang al., 2023), which incorporate a time di- mension and store facts as quadruples (s, r. o, l), e.g., (Germany, negotiate, Russia, 2022-05).
However, in real-world scenarios, TKGs are of- ten incomplete, highlighting the vital importance of TKG reasoning, which aims to predict future facts by utilizing the temporal information. As il- lustrated in Figure 1, given the TKGs associated with Entity A at the timestamps of Time I, Time II, and Time III, our objective is to predict quadruple containing an unknown entity, i.e., (A, negotiate, ?, Time IV). To achieve this goal, we can observe a significant structural dependency of facts along the timeline. That is, interactions between Entity A and Entity B at Time I could influence the in- teraction patterns at Time 1I, which, in turn, might provide clues for Time III. We also notice that his- torical events may recur. For instance, Entity A and Entity B had conflicts at both Time I and Time II timestamps. These observations provide us valu- able insights into TKG reasoning.
1 Introduction
Knowledge Graphs (KGs), which represent events triples (s,r, o), facilitate wide range of natu- ral language processing tasks, including semantic search (Bonatti et al., 2019; Wang al., 2020), product recommendations (Xie et al., 2021), and systems (Saxena et al., 2020).
In the TKG reasoning task, events to be pre- dicted can typically be classified into two main types: historical events and unseen events (Han et al., 2021). Historical events, also known as
'The source code and datasets are available at https:// gi thub. www/cool - codes/SiMFy. Xuanhua Shi is the corresponding author.

---

repetitive pattern, refer facts that have already occurred in the historical KG sequence. Unseen events refer to events that have not occurred in the historical KG sequence. Many methods (Jin et al., 2020; Zhu et al., 2021) can predict historical events effectively by modeling the historical KG sequence auto-regressivell. However, for unseen events, it is necessary to consider both the infor- mation of structural dependency and temporality of entities and relations. Typically, existing works (Liu et al., 2022; Li et al., 2022) apply Graph Neu- ral Networks (GNNs) and Recurrent Neural Net- works (RNNs) as encoders to capture the structural and temporal information in the historical KG se- quence. Then, a translational model, such as Conv- TransE (Shang et al., 2019), is used as a decoder to obtain predicted entities. Despite their effec- tiveness, these approaches are conceptually and technically complex due to their advanced model architectures. Moreover, these complex methods tend to suffer the issues of low training efficiency and insufficient generalization ability.
mation for the TKG reasoning task. Furthermore. we investigate the question of whether historical frequency information should be incorporated into the model training process, which has not been explored before. Unlike the most current main- stream methods which directly incorporate histori- cal frequency information of entities during train- ing, SiMFy utilizes conceptually simple features to model the repetitive pattern of TKGs, which is fixed in training. Through empirical experiments, we validate the effectiveness of this freuunryy strategy adopted by SiMFy.
The contributions of this paper are as follows.
We design a simple MLP-based-only model, called SiMPy, which achieves state-of-the- art performance on four widely-used datasets (i.e., ICEWS14, ICEWS18, ICEWS05-15, and GDELT), demonstrating the effectiveness of the simple model architecture.
We have performed extensive experiments to analyze the convergence speed, generalization ability, and training consumption of SiMFy and existing complex models. The empirical evidence demonstrates that a simple model architecture like SiMFy enjoys faster conver- gence, better generalization ability, and higher efficiency.
In this paper, we aim to answer the following re- search question: Are these complex model architec- tures indispensable for temporal knowledge graph reasoning? As solution, we design a Simple MLP- MLL--Frequenc--bee -hased model (SiMFy) evaluate it against other complex baselines. Specifically, we use a one-layer MLP to jointly model entities and relations, capturing structural and temporal information in TKGs. This allows us to obtain em- bedding vectors for entities and relations, which are then used to calculate the similarity between queries and candidate entities, resulting in prelim- inary candidate entity scores. Next, we calculate the historical frequency scores of candidate entities based on their historical KG sequences. Finally, in the inference stage, the two scores mentioned above are combined using a coefficient a to obtain the final entity scores. Through extensive experi- ments comparing it with existing state-of-the-arr models, we find that our model can achieve com- parable performance while having higher training efficiency and better generalization ability.
The performance of SiMFy could motivate fu- ture research to rethink the significance of the simpler model architecture and the potential value of MLP-based models in TKGs.
Related Work
Static KG Reasoning In static knowledge graphs, inference is done to deduce unknown facts from known ones. Existing approaches for in- ference can be categorized into embedding- tensor decomposition--assed and neural network- based. Embedding- based methods, represented by the classic method TransE (Bordes et al., 2013), consider relations as translational transformations. Later models like KG2E (He et al., 2015) opti- mize the handling of diverse relations and improve model scalability. Tensor decomposition methods represent the graph as a tensor which is then decom- posed. Models like RESCAL (Nickel et al., 2011) use this capture interactions between entities and relations. As for the neural network models like ConvE (Dettmers al., 2018) and KG-BART (Liu et al., 2021), I), ConvE enhances inference capabilities through deep feature learning and leveraging graph
Furthermore, we conduct series of empirical studies to investigate the performance of SiMFy and existing complex models under specific condi- tions. We are the first analyze the performance of ML.P and GNNs in capturing unseen entities. Our findings indicate that MLP performs comparably to GNN in capturing structural dependency infor-

---

Similarity Matching
Vector Representation
Query
(s,r,
MLP
go
9
g
Softmax
Prediction
1,0,0,0000
0,
Probability Distribution
Historical Frequency Learning
Figure 2: Overall model architecture of SiMFy. Given query (s, ?,4 At), the Similarity Matching module learns the similarity score between the query and each candidate entity o, and the Historical Frequency Learning module learns the historical frequency score between them, then the scores combined to generate the final probability distribution of entities.
structures, and KG-BART utilizes pretrained lan- guage models augmented with knowledge graphs to not only enhance inference but also improve the text.
ment learning is adopted by CluSTeR (Li et al., 2021a), which infers answers from induced clues, and TimeTraveler (Sun et al., 2021), which uses historical knowledge graph snapshots for answer search. CEN (Li et al., 2022) adopts length-aware convolutional network to model the KG sequence dynamically and GHT (Sun et al., 2022) is the first method to introduce transformer into the TKG reasoning task. Finally, DA-Net (Liu al., 2022) and CENET (Xu al., 2023) propose unique event prediction models. DA-Net learns distributed atten- tion to future events, while CENET distinguishes likely entities for a given query using a historical contrastive learning framework.
generation of commonsense-reant
Temporal KG Reasoning Temporal knowledge graph inference, which takes into account the tem- poral evolution of events, generally falls into two categories: interpolation- based and extrapolation- based inference. Interpolation-based inference aims to guess unknown facts within a known time range. TTransE (Jiang et al., 2016) inte- grates temporal information into the TransE (Bor- des et al., 2013) model using recursive neural net- works, while HyTE (Dasgupta al., 2018) de- signs unique hyperplane embed time into the entity-relation space. TeMP (Hu al., 2022) ad- dresses time sparsity and variability issues by com- bining neural message passing and temporal dy- namic methods.
Problem Formulation
3.1 Temporal Knowledge Graph
Temporal Knowledge Graph (TKG) G is se- quence of KGs (Go, 91. .9)) arranged order of their timestamp t. G = {E, R), where & stands for the set of entities, and R for the set of relations. Each G. = {Ee, Re}. where & C E and R. S R are the sets of entities and relations at timestamp respectively. In G, facts are represented as quadru- ples (s, r, o, l). where 8, o & and Rt.
On the other hand, extrapolation- based inference, which forecasts unknown future facts, is garnering increasing attention. RE-NET (Jin et al., 2020) employs an encoder and aggregator to model past facts, while HIP (He al., 2021) utilizes tempo- ral, structural, and repetitive information. XERTE (Han al., 2021) offers novel framework for pre- dicting future facts and CyGNet (Zhu et al., 2021) introduces a creative copy mpechmechanism used in nat- ural language generation tasks before. Reinforce-
3.2 TKG Reasoning
TKG reasoning seeks to forecast either the subject entity or the object entity o based on the historical

---

KG sequence (Go. ...... When presented with query of the form (?, o, + At). the task is to identify the subject entity 8, while for query like (s, r, ?, + At), the aim is to predict the object entity o. We use E E ple|xd and R. E RR[R|xd to express the embeddings of all entities and all relations respectively. Boldfaced s, are used to denote the embedding vectors of s, and with dimension of d. In work, we specifically concentrate on the task of object entity prediction.
the following similarity score by utilizing the soft- max function:
S softmax (Hs,)) sim
(3)
4.2 Historical Frequency Learning
Given query q (s ?, At) and the histor- ical KG sequence (9o. G1, .......9t the Historical Frequency Learning module aims to obtain the his- torical frequency scores. Specifically, for every timestamp S I, we first investigate the frequen- cies of historical entities f(ax) E E as follows:
4 Our Approach
Here elaborate our proposed Simple MLP. Frequency-based model (SiMFy) for temporal graph reasoning. As illustrated Figure 2, SiMFy mainly involves two modules: the Similarity Matching module and the Historical Frequency Learning module. These two modules generate corresponding scores for candidate entities. After- ward, a weight-based inference process is utilized to determine the final result of the reasoning. In the subsequent sections, will provide comprehen- sive introduction our proposed method.
f (
=E
= -T xCGp
(8,1, (4)
where [[[] is an indicator function, yielding 1 if [-] is true and 0 otherwise. Then we add up the frequency information of all timestamps t' S t obtain the historical frequency information of the query as follows:
fsrr) (+ = (((.)) + f(xx)) +... + H(()))
(5)
4.1 Similarity Matching
where F (5.)) fi E E is an JEl-dimensional vector where each element represents the corresponding historical frequency of the candidate object enti- ties. Finally, we obtain the following historical frequency score by utilizing the softmax function:
Given a query - ?,t At), the Similar- ity Matching module is implemented by one-layer- MLP to calculate the similarity between and each candidate entity to obtain the matching scores. Specifically, it generates latent context vector H.r Ri for query q, which scores the similar- ity of different object entities with the query:
freg = softmax(k (1.)
(6)
(+4()
where & is hyperparameter to balance the ex- tremely small values.
H tanh (W rs + b) E
(1)
where s E E, r E R. and [s, r] denotes the concate- nation s and r. We use one-layer-MLP to aggre- gate the query's information. Here, W E Rdx2d and b E IRd are trainable parameters. tanh is the activation function of the layer, then the layer's out- put is multiplied by E to obtain the final similarity vector, where each element represents the similarity score between the corresponding entity o and the query q, The learning objective of the similarity matching is to minimize the NCE loss (Gutmann and Hyvärinen, 2010) L. as described below:
4.3 Inference
In the inference stage, a coefficient a [0, 1] is integrated to balance the weight between the simi- larity score s(s.r) and the historical frequency score Sfrey freg' These two scores are merged to determine the final probability distribution of candidate en- tities, then the model chooses the object with the highest probability as the final prediction, de- fined below:
sim
P(ols,,,ttt) a: S (0) (1)+(111))))))))))) freg (0)
exp (H (o:))
C - - Σ log Enyes exp (H.r (0;))
(2)
O1-A = arg max P(oss,
(7)
where Oj is the ground truth object entity corre- sponding to the given query q. Finally, we obtain
where P(o)s, r, + At) is an |E| dimeensionn vec- tor which stores the final probability of all entities.

---

Table 1: Experimental results the entity prediction task (raw metries) on ICEWS14, ICEWS18, ICEWS05-15, and GDELT datasets. The best results boldfaced and the ones of second-best are underlined.
ICEWS (raw)
ICEWS1S (row)
JCEWSOS (raw)
GDELT
MRR
Hisdd
Hivve5
Hisuts 10
MRR
Hinuel
His
Hilvd 10
MRR
HinoFF
Hix.613
Hisse
MRR
Hixd61
Hirc633
611
RotatE
25.71
16.41
29.01
45.
14.55
6.47
15.76
31.95
19.01
10.42
21.35
36.92
3,62
0.52
2.26
8.37
ComE
30,30
21.30
34.42
47.89
22.81
13663
25,85
41.43
51.40
21,56
35.70
50.96
18.37
11.29
19.36
32.13
Con-TrannB
31,50
22:46
34,98
50.03
23.22
14.36
26.13
41.34
30.28
20.79
33,80
49.95
19.07
11.85
20.32
33.14
R-GON
78.03
19:42
$1.95
44.83
15.05
8.13
16.40
29.00
27:13
18.83
30.41
43.16
12.17
7.40
12:37
20.63
TTranBB
12.86
3.14
15.72
33.65
8:11
1.85
8.95
22.38
16.53
5,51
20.77
39.26
5.53
0.16
1.97
15.37
HyTE
16,78
2.13
24.84
43.94
7.41
3,10
7.33
16.01
16.05
6,53
20.20
34.72
6.69
0.01
7.57
19.06
TA DusMult
26.22
15.83
29.72
45.23
16.42
8.00
18.13
32.51
27.51
17.57
31.46
47.32
10.34
4.44
10.44
21.63
XERTE
32.25
24.29
36.41
48.76
23.0%
19.26
32.43
46.00
38.07
28.45
43.92
57.62
RE-NET
35.77
25.99
40.10
54.87
26.17
16.43
29.89
44.37
36.86
26.24
41.85
57.60
19.00
12.03
20.56
33.89
CyGNet
35.05
25.78
39,00
53:42
24.80
15.37
28.29
13:46
36.21
25.65
41.51
56.25
17.99
11.10
19:08
31.26
CENET
36,36
40/12
$4.39
17.57
44.12
28.57
58.57
GHT
37.40
27:22
56.19
27.40
45.76
51.50
30.79
62.73
20.04
12.65
21.32
34.42
CEN
36.32
$1.93
46.84
57.12
11.95
39.50
RE-OCN
47.07
37.79
26.96
58.93
12.16
30.39
SiMFy
59.18
36.87
5 Experiments
tering any triples, meaning that all possible triples are considered when calculating rankings. In the context of TKG reasoning task, each quadruple (s, o, is unique due to the inclusion of time- stamp information. This suggests that the filtered setting holds little practical relevance when evalu- ating model performance. Therefore, we adopt the raw setting for our experiments.
5.1 Experimental Setup
Dataset During the evaluation, we adopt four publicly available TKG datasets: ICEWS14 (Li et al., 2021b), ICEWS05--1 (García-Durán et al.. 2018), ICEWS18 (Jin et al., 2020), and GDELT (Leetaru and Schrodt, 2013). The first three datasets which originate from the Integrated Cri- sis Early Warning System (Boschee et al., 2015) (ICEWS) comprise a diverse range of political facts accompanied by time annotations, such as (European Union, Praise or endorse, Kosovo, 201 8/09/29. Global Database of Events, Lan guage, and Tone (GDELT) is much larger dataset that records data every 15 minutes. Following the dataset split strategy proposed by (Jin al., 2020), we divide these datasets into training, validation, and test sets, adhering an 80%, 10%, and 10% proportion by timestamps.
Baselines We assess how well our SiMFy model performs in comparison to three other types of previously proposed models: (I) Static reasoning methods, including RotatE (Sun et al., 2019), Conv- Trans (Shang al., 2019), ConvE (Dettmers al., 201 8), and R-GCN (Schlichtkrull et al., 2018). (2) Dynamic interpolated reasoning methods, in- cluding TTransE (Jiang et al., 2016), HyTE (Das- gupta et al., 2018), and TA-DistMult (García-Durial et al., 2018). (3) Dynamic extrapolated reason- ing methods, including XERTE (Han et al., 2021), RE-NET (Jin al., 2020), CyGNet (Zhu et al., 2021), CENET (Xu ct al., 2023), GHT (Sun et al., 2022), CEN (Li et al., 2022) and RE-GCN (Li al., 2021b).
Evaluation Metrics We evaluate our approach to the task of entity prediction, where the objective is to predict the absent object entity for given entity-relation pair, assessing whether the ground truth entity ranks higher than other entities. We present the results in terms of Hits@@ 1/3/10 and Mean Reciprocal Rank (MRR).
5.2 Implementation Details
For the baselines of CyGNet, CENET, CEN, and RE-GCN, we rerun these models using their open- source code on four datasets with their default pa- rameter settings. Since the code of GHT is not open-source, we provide the findings from their publication. It should be noted that we utilize the offline version of CEN to ensure fairness. Some results of static and dynamic interpolated reasoning approaches are adopted from (Li et al., 2021b). We implement our SiMFy model using Py Torch.
During the evaluation stage, two settings are commonly employed: filtered setting and raw set- ting. In terms of the filtered setting, for each query, we treat all triples absent from the training, vali- dation, and test sets to be negative samples. This implies that, when computing rankings, we disre- gard all triples known to be true.
In contrast, the raw setting does not involve fil-

---

Table 2:
Ablation study of SiMFy on ICEWS14
having some capability handle historical events, SiMFy w.o. HF also experiences a drop of 9% in performance, which further confirms the effective- ness the Historical Frequency Learning module.
Model
MRR
Hits@1
Hits@3
Hits@
SiMEY
39.54
29.56
44.56
59.18
SiMFY w.o. HF
35.97
26.03
40.85
55.39
SiMFy w.o. SM
31,18
24.07
34.51
44.34
5.5
Case Study
To further illustrate 1) SiMFy's 's ability to predict un- seen entities by capturing the structural dependency information of KGs, and 2) SiMFy's ability to pre- dict repetitive events using historical frequency, we present two cases in the ICEWS18 dataset.
The dimension of entity and relation embeddings is set to 200. We use Adam (Kingma and Ba, 2015) as the optimizer, with learning rate of 0.001 and a weight decay of 0.00001. The hyperparameter is set to 2 and the a is set to 0.001. The batch size is set to 1024 and the training epoch is limited to 30. All experiments were conducted Tesla V100.
In the first case, the query is (Dharamvira Gandhi, Criticize or denounce, ?, r), and the correct objective entity is Government (India). However, upon investigating the historical KG sequence of this query. we find that the triple (Dharamvira Gandhi, Criticize or denounce, Government (India)) has not occurred before. This indicates that there is no repetitive pattern for the quadruple (Dharamvira Gandhi, Criticize or denounce, Government (India). t), making it an unseen event with Government (India) as the corresponding unseen entity. Nevertheless, due SiMFy's extraction of structural dependency information from the historical KG sequence, we find that the candidate entity Government (India) appears as the top first in the ranking list gener- ated by the model. This indicates that SiMPy has successfully captured the correct unseen entity, despite its absence in the historical KGs.
5.3 Model Results
Table I shows the experimental results of SiMFy compared with other baselines four datasets. It is clear that SiMFy performs better than other baselines in most cases, For static reasoning meth- ods, their results are very poor because they do not consider temporal information. Dynamic in- terpolated reasoning methods like TTranSE only encode time information without considering the evolution of temporal KG sequences, so they can- not achieve good results. It is worth noting that for GCN-based models CEN and RE-GCN, well as the models most pertinent to our model, RE- NET, CyGNet, and CENET, SiMFy outperforms them. This demonstrates the effectiveness of sim- ple models based on MLPs and shows that MLPs can to some extent replace GCNs. Please refer to Section 6.3 for more in-depth analysis.
In the second case, the query is (Zdravko Maric, Make statement, f), and the correct objective entity Government (Croatia). By analyzing the historical KG sequence of this query, we find that the triple (Zdravko Maric, Make statement, Gov- ernment (Croatia)) has the highest frequency of occurrence, significantly surpassing other cases. SiMFy, utilizing the learned historical frequency information, recognizes this repetitive pattern and memorizes it. As result, in the final ranking list generated by the model, the correct answer Government (Croatia) occupies the top position. This aligns with our real-world understanding that Zdravko Maric served as the Minister of Fi- nance in the Croatian government, making state- ments on behalf of the government frequently.
5.4 Ablation Study
We perform ablation experiments on the ICEWS14 dataset to better understand the impact of each SiMFy module, and the results displayed in Table SiMFy w.o. HF refers to SiMFy without the Historical Frequency Learning module, while SiMFy w.o. SM represents SiMFy without the Similarity Matching module. It can be observed that SiMFy w.o. HF performs better than SiMFy W.o. SM. This is because the Similarity Matching module captures many unseen events in the TKGs, which also demonstrates the ability of MLP in un- derstanding the structural dependencies of events. Therefore, SiMFy w.o. SM, which only considers historical repetitive patterns, results significant drop of 21.1% % and 25.1 % in terms of MRR and Hits @ 10 respectively compared to SiMFy. Despite
More Empirical Results and Analysis
Here, we show more empirical results and analysis to reveal the strengths of our proposed SiMFy.

---

Table 3: The comparison results of the MLP and models' ability to predict unseen events
ICEWS14 unseen
ICEWS18 unseen
ICEWS05--15 15 unseen
Model
MRR
@
Hits 10
MRR
Hits@3
Hits@ 10
MRR
Hits@3
Hits@ 10
RGCN+ConVTransE
13.90
14.92
26.60
11.15
11.54
21.90
12.56
13.34
25.03
RGCN+MLP
14.10
15.03
26.90
10.70
11.03
20.86
13.21
13.95
25.86
MLP
15.53
17.01
30.97
11.28
11.59
22.91
14.00
15.06
27.65
evaluate their performance on the test set. Fig- ure 4b illustrates that our model not only achieves the highest MRR but also maintains stability and smoothness in the curve after several epochs. In contrast, the MRR curves of other baselines show varying degrees of oscillation.
SIMFy
2000
RGCN-CovThadd
RGCN-MLP
Avg Epoch Time (sconds)
CyCNa
RE-GCN
1500
CENET
1000
The generalization ability of model refers its performance on unseen data, that is, the adapt- ability of the model to new data. Therefore, the smaller the generalization gap, the better the model's generalization ability. As shown in Fig- ure 4c, our model has the smallest generalization gap, showing its strong generalization ability. Combining these three figures, we also ob- serve interesting phenomenon. Models based on GCN, represented by CEN (Li al., 2022), even their performance on the test set fails enhance after numerous epochs, their MRR on the training set keeps rising. This means that these GCN-based models have overfitting prob- lems, which is also reflected in the increasing generalization gap.
500
0-
ICEWS14
ICEWS18
ICEWSOS-15
Figure 3: Time consumption different models
6.1 Convergence Speed and Generalization
To gain deeper understanding of the model's per- formance, we will closely examine the dynamics of both training MRR and evaluation MRR. Follow- ing (Cong et al., 2023), we will also discuss the im- portant generalization gap (the absolute difference between the MRR scores obtained during training and evaluation) of different models. In addition to SiMFy, we also use four existing complex base- line models, including CyGNet, RE-GCN, CENET, and CEN, which are most relevant to our work, as experimental comparisons. Moreover, based on RGCN, we also reconstruct two new GCN-based models as our comparative baselines. The first one is BACCN+ConVTranslee where Conv'Transt (Shang et al., 2019) is used as the decoder. The second one is RGCN+MLP, using MLP as the decoder. Figure 4 illustrates the efficacy of the models the most representative ICEWS05-15 dataset, from which the following conclusions can be drawn:
6.2 Low Time Consumption
Compared to other complex models, SiMFy has much lower training time consumption because the model parameter numbers of SiMFy are much smaller. This could be attributed to the concise and effective model structure of SiMFy, Taking the typical GCN-based method SiMFy is simply composed of an MLP layer that maps the input embedding the output embedding.
example, our
The GCN-based models also perform the multi- step propagation operation, which will significantly increase the model parameters, especially when the depth of the GCN is increased.
As the slope change of the training MRR curve can reflect the convergence speed of the model (i.e., slope close 0 indicates model conver- gence), we can see that our model always con- verges after several epochs, indicating a very fast convergence speed.
For the ICEWS14, ICEWS18, and ICEWS05-15 benchmarks, calculate the average training time consumption of model SiMFy and other base- lines over 30 epochs. As shown in Figure the time cost is highest for the ICEWS05-15 dataset, followed by the ICEWSI8 dataset, and the lowest for the ICEWS14 dataset. However, on all three
A Ater each epoch, we save separate models and

---

11425
0.403
Graeralaaien Gap
Tolo M3X
3.50
0.39
5385
0.4
AoCheGoerCuad CycN REGEN
Mnchrroeeteen CyONa RDODN
0.309
0.40
RGGEN
CENEE
0150
(a) Comparison on training MRR
(b) Comparison on test MRR
(c) Comparison on generalization gap
Figure 4: Comparison on the training MRR, test MRR, and generalization gap for the 30 training epochs on ICEWS05-155. Results on other ICEWS datasets can found Appendix A.4.
datasets, SiMFy exhibits the lowest time consump- tion compared to other baselines during the train- ing process, making it more efficient and greatly improving resource utilization. Furthermore, can observe that the training time consumption SiMFy does not exhibit significant fluctuations across different datasets.
300
Training Time
MINNER MUN-0OYY
6.3 Ability Capture Unseen Events
To predict the unseen events that do not appear in the historical KG sequence, a natural idea us- ing the historical evolutionary information of other events that interact with them, which is also known the event structure dependency information. The mainstream view that GCN better capture this structural dependence, interestingly, we find that MLP and GCN have similar abilities to capture unseen events. Therefore, given the high training cost of GCN, we doubt whether it is necessary to apply it to TKG reasoning task. We collect all un- seen events from the test set for the three ICEWS datasets, indicated in Table 3, and then we eval- uate the MLP module and the GCN, respectively. The results indicate that the performance of MIP is even slightly superior to that of GCN, which also validates the effectiveness of our MLP-based model.
Epochs
Figure 5: Training time in first 10 epochs
(fixed-freee eexeedffreuencc in training). However, many ex- isting state-of-the-arr methods, like CENET and CyGNet, combine event embeddings with histori- cal frequency during the training process and iter- atively update the model parameters. To validate which approach more beneficial for the TKG reasoning task, we conduct comparative experi- ments on the ICEWS14 dataset by adopting the sopy-mechanism-baee based learning strategy used in CyGNet and CENET. The results can be found in Table 4, MLP-ONLY consists of only the MLP module, MLP-F.F. adopts the fixed-free freddffreuency strat- egy during training, and MLP-COPY represents the model that incorporates the copy mechanism. It can be observed that MLP-F., outperforms MLP- COPY significantly, while MIPP-OPY performs even slightly worse than the original MLP-ONLY. This suggests that MLP-COPY fails to effectively incorporate historical frequency information during the training process. Furthermore, by considering Figure 5, we can observe that MLP-COPY only fails enhance the model performance, but also significantly prolongs the training time for each epoch. This indirectly confirms the superiority the fixed- freeufrry strategy adopted by SiMFy.
Table 4: Comparative experiments of frequency on ICEWS14
Model
MRR
Hits@1
Hits @its@3
10
MLP-EE
39.54
29.56
44.56
59.18
MIP-ONLY
35.97
26.03
40.85
55.39
MLP-COPY
35.96
26.54
40.20
54.02
6.4 Fixed Frequency in Training
SiMFy utilizes the historical frequency informa- tion of events only during the inference stage

---

6.5 Discussion on the Depth of MLP
historical events is too long, and more precise time window is needed to better capture the long- and short-term evolutionary patterns of events.
SiMFy uses a simple one-layer MLP to draw the final prediction. We also want to figure out whether the performance of the model will improve or worsen when the one-layer MLP is replaced with some deeper or complicated neural structure. To answer this question, we conduct experiments to replace the one-layer MLP with deeper ones. The experimental results on the ICEWS14 dataset are presented in Table 5. From this table, we can see that as the structure of the model becomes more complex, its performance has hardly changed. This is because the complex model structure, while in- creasing computational costs, does not better cap- ture the evolving information of entities and rela- tions.
Acknowledgements
This work was supported in part by the Na- tional Key R&D Program of China under Grant 2020AAA010850) and the Key Program of Hubei under Grant JD2023008.
References
Piero Andrea Bonatti, Stefan Decker, Axel Polleres, and Valentina Presutti. 2019. Knowledge graphs: New di- rections for knowledge representation on the seman- tic web (dagstuhl seminar 18371). I). In Dagstuhl re- ports, volume 8. Schloss Dagstuhl- Zentrni-Zeezii Zentrum fuer Informatik.
Table 5: Comparative experiments on the depth of MLP on ICEWS14
Antoine Bordes, Nicolas Usunier, Alberto García- Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Proceedings of the Advances in Neural Information Processing Systems 26, pages 2787-2795.
Model
MRR
Hits @
Hits@3
Hits@ 10
SiMFy-IMLP
39.54
29.56
44.56
59.18
S1MPY-2MLP
39.49
29.74
43.97
58.58
SiMFy-3MLP
39.29
29.74
43.46
58.20
SiMFy-RNN+MLP
39.27
29.45
43.89
58.36
Elizabeth Boschee, Jennifer Lautenschlager, Scan O'Brien, Steve Shellman, James Starz, and Michael Ward. 2015. Icews coded event data. Harvard Data- verse, 12.
7 Conclusion
Weilin Cong. Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu. Xin Zhou, Hanghang Tong, and Mehrdad Mah- davi. 2023. Do we really need complicated model architectures for temporal networks? In Proceedings of The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
In this paper, we have proposed a new model, called SiMFy, for the TKG reasoning task. SiMFy is a conceptually straightforward method that simply combines MLP and historical frequency to model the temporal events in the TKGs. The experimental results demonstrate that SiMFy not only outper- forms many existing complex methods but also exhibits faster convergence speed and better gener- alization ability. Our findings suggest that well- designed MLP-based model, such as SiMFy, can ef- fectively address the limitations faced by complex architectures, making it a practical and efficient solution for the TKG reasoning task.
Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha P. Talukdar. 2018. Hyte: Hyperplane-basee temporally aware knowledge graph embedding, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2001 2011.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp. and Sebastian Riedel. 2018. Convolutional 2d know- ledge graph embeddings. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelli- gence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial In- telligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7. 2018, pages 1811-1818.
Limitations
One limitation of this paper is that no in in-depth ex- ploration of the potential mutual influence between historical events and unscen events is considered, which means that events that have been repeatedly happening in the sequence of historical KGs may not happen in the future, and instead, new unseen events will take place. Another limitation lies in that the time span for calculating the frequency of
Alberto García- Duránn, Sebastijan Dumancic, and Math- ias Niepert. 2018. Learning sequence encoders for temporal knowledge graph completion. In Proceed- ings of the 2018 Conference on Empirical Methods

---

Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, pages 2816-48211
Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng. Yajuan Lyu, Yong Zhu, Long Bai, Wei Li. Jiafeng Guo, and Xueqi Cheng. 2022. Complex evolutional pattern learning for temporal knowledge graph rea- soning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 2: Short Pupers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 290-296.
Michael Gutmann and Aapo Hyvärinen. 2010. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Ar- tificial Intelligence and Stafistics, pages 297-304. JMLR Workshop and Conference Proceedings.
Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang. and Xueqi Cheng. 2021a. Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguisties and the I Ithh Inter- national Joint Conference on Natural Language Pro- cessing (Volume Long Papers), ). pages 4732-4743, Online. Association for Computational Linguistics.
Zhen Han, Peng Chen, Yumpu Ma, and Volker Tresp. 2021. Explainable subgraph reasoning for forecast- ing on temporal knowledge graphs. In Proceedings of the 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Shizhu He, Kang Liu, Guoliang Ji, and Jun Zhao. 2015. Learning to represent knowledge graphs with gaus- sian embedding. In Proceedings of the 24th ACM International Conference on Information and Knowl- edge Management, CIKM 2015, Melbourne, VIC, Australia, October 19- 23, 2015, pages 623-632.
Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021b. Temporal knowledge graph reason- ing based on evolutional representation learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 408-417.
Yongquan Hc, Peng Zhang, Luchen Liu, Qi Liang, Wenyuan Zhang, and Chuang Zhang. 2021. Hip network: Historieal information passing network for extrapolation reasoning on temporal knowledge graph. In P Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 1915-1921.
Kangzheng Liu, Feng Zhao, Hongxu Chen, Yicong Li, Guandong Xu, and Hai Jin. 2022. Da-net: Dis- tributed attention network for temporal knowledge graph reasoning. In Proceedings of the 31 ACM In- ternational Conference on Information & Knowledge Management, pages 1289-1290
Zhiwei Hu, Victor GutiérrezBasulto, Zhiliang Xiang, Xiaoli Li, Ru Li, and Jeff Z. Pan. 2022. Type-aware embeddings for multi-hop reasoning over knowledge graphs. In Proceedings of the Thirty-Firss First Interna- tional Joint Conference on Artificial Intelligence, IJ- CAI 2022, Vienna, Austria, 23-29 July 2022. pages 3078-3084.
Ye Liu, Yao Wan. Lifang He, Hao Peng, and Philip S. Yu. 2021. KG KG-BART: knowledge graph suph-augmented BART for generative commonsense reasoning. In Proceedings of the Thirty-Fifihh AAAI Conference on Artificial Intellignnce, AAAI 2021, Thirty-Thing Con- ference Innovative Applications of Artificial In- telligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAA! 2021, Viriual Eveni, February 2-9, 2021, pages 6418- 6425.
Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Baobao Chang. Sujian Li, and Zhifang Sui. 2016. Towards time-aware knowledge graph completion. In Pro- ceedings of the 26th International Conference on Computational Linguistics: Technical Papers. pages 1715-1724.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learn- ing, ICMI. 2011, Bellevue, Washington, USA, June 28 -Jly 2, 2011, pages 809-816.
Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020. Recurrent event network: Autoregressive struc- ture inference over temporal knowledge graphs. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6669- 6683.
Apoorv Saxena, Aditay Tripathi, and Partha P. Taluk- dar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embed- dings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 4498-4507.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Rep- resentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015.
Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titoy, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In Proceedings of The Se- mantic Web- 15th International Conference, ESWC
Kalev Leetaru and Philip A. Schrodt. 2013. Gdelt: Global data events, location, and tone, 1979-2012. In Proceedings of the ISA Annual Convention, vol- ume 2. pages 1-49. Citeseer.

---

2018, Heraklion, Crete, Greece, June 3-7, 2018, vol- ume 10843 of Lecture Notes in Computer Science, pages 593-607. Springer.
Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pages 4765 4773.
Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhang. 2021. Learning from history: Modeling temporal knowledge graphs with sequen- tial gey-ggenrrttion networks. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 4732 47740.
Chao Shang. Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-end structure- aware convolutional networks for knowledge base completion. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, AAAT 2019, The Thirty-Fion First Innovative Applications of Artificial In- lelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 2019, pages -060-3067.
A
Appendix
A.1 Results with Filtered Metries
Table provides the experimental results (with fil- tered metries) of SiMPy compared with other base- lines on ICEWS14, ICEWS18, ICEWS05-155 and GDELT datasets.
Haohai Sun. Shangyi Geng, Jialun Zhong, Han Hu, and Kun He. 2022. Graph Hawkes transformer for ex- trapolated reasoning temporal knowledge graphs. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 748 1481-7493, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
A.2 Statisties of Datasets
The detailed statistics of the 14, ICEWS18, ICEWS05-15, and GDELT datasets are presented in Table 7.
Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021. Timetraveler: Reinforcement learn- ing for temporal knowledge graph forecasting. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Repub- lic, 7-11 November, 2021, pages 8306-8319.
A.3 Proportion of Unseen Events
We conduct statistical analysis on the proportion of unseen events in the test set of the ICEWS14, ICEWS18. ICEWS05-15, and GDELT datasets. The results are presented Table 8.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yur Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In Proceed- ings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
A.4 Supplementary Figures Convergence Speed and Generalization
We provide the missing figures on the ICEWS14 and ICEWS18 datasets to supplement the Sec- tion 6.1. The results on ICEWS14 are shown in Figure 6 and the results ICEWS18 are shown in Figure 7.
Guoren Wang, Yue Zeng, Rong-Hua Li, Hongchao Qin, Xuanhua Shi, Yubin Xia, Xuequn Shang, and Liang Hong. 2023. Temporal graph cube. IEEE Transac- tions on Knowledge and Data Engineering, pages 1-15.
Yuxiang Wang. Arijit Khan, Tianxing Wu, Jiahui Jin, and Haijiang Yan. 2020. Semantic guided and re- sponse times bounded top-k similarity search over knowledge graphs. In Proceedings of the 36th IEEE International Conference on Data Engineer- ing. ICDE 2020, Dallas, TX, USA, April 20-24, 2020, pages 445-456 IEEE.
Lijie Xie, Zhaoming Hu, Xingjuan Cai, Wensheng Zhang. and Jinjun Chen. 2021. Explainable rec- ommendation based on knowledge graph and multi- objective optimization. Complex & Intelligent Sys- tems, 7:1 1:122.
Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2023. Tem- poral knowledge graph reasoning with historical contrastive learning. In Proceedings of the Thirty- Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in

---

Table 6: Experimental results of the entity prediction task (filtered metries) on ICEWS14, ICEWS18, ICEWS05-15, and GDELT datasets. The best results are boldfaced and the ones of the second-best ones are underlined.
ICEWS05-15
(fitterd)
Miskel
Hindt1
Hixd03
Hisdl
Hixd
Hin
Hited-10
MRR
Bive0
Hiwdk3
Hivd010
MRR
Hiivh11
Hisee33
52.70
21:00
37.81
52.31
RE-NET
45.71
38.42
49.06
59.12
42.95
36.19
45.47
55.50
42.97
31.26
40.85
63.47
40.12
32,43
43.40
CyCNee
48.53
41.27
52.50
60.29
46.69
00.58
49.82
57.14
57.00
50.06
61.51
08.67
51.56
45.63
54.86
CENET
52.87
17.91
51.12
02.79
50.09
46.02
51.05
57.85
CEN
37.34
28.04
41.09
55.76
28.21
19.10
31 80
45.89
37.53
27.57
41.49
57,51
19.71
12,47
20.96
33.82
RE-GUU
16.27
28.40
42.54
57.88
20.21
19.79
32.77
47.54
38.68
76.36
43.44
59,11
29.46
21.74
32.01
43.62
SEMBY
54,81
47.99
58.54
66.65
46.77
39.29
51.00
61.23
68.76
53.43
65.62
72.94
47.40
40.17
50.51
00:4
Table 7: Statistics of the datasets
Dataset
Entities
Relations
Train
Valid
Test
Timestamp
Granularity
ICEWS14
7128
230
74845
8514
7371
365
24 hours
ICEWS18
23033
256
373018
45995
49545
304
hours
ICEWS05-15
10488
251
368868
46302
46159
4017
hours
GDELT
7691
240
1734399
238765
305241
2751
15 minutes
Table 8: The proportion of unseen events
Dataset
Test
Unseen Events
Repetitive Events
Proportion of Unseen Events
ICEWS14
7371
3511
3860
47%
ICEWS18
49545
24560
24985
49%
ICEWS05-15
46159
14592
31567
3 %
GDELT
305241
107062
198179
35%
0.10
1175
BSCH--L17
Ogtas Stalle G4
9.25
Tox MIR
CUREI
0.15
AOCh-Ce/Tuadi
RG0h Tuadi
AGCH-MLI
0y014
1120
Cyt-a
3.55
CDC
10
15
25
30
15
20
25
30
35
25
30
Epachs
Epeadt:
Epochs
(a) Comparison on training MRR
(b) Comparison test MRR
(c) Comparison generalization gap
Figure 6: Comparison on the training MRR, test MRR, and generalization gap on ICEWS14
0.40
0.150
ROCEN-1SLLP
0.75
11250
9.125
0.70
To MB1
0.225
CENET
0.200
3,075
/
0.175
3,050
Out04
0.150
0.025
0125
3,000
10
20
25
10
10
70
10
10
15
30
24
34
Epodoo
Epucls
Epozha
(a) Comparison on training MRR
(b) Comparison test MRR
(c) Comparison generalization gap
Figure 7: Comparison on the training MRR, MRR, and generalization gap on ICEWS18


Reviews:
[{"review_id": "QIJ8JLrWcs", "reviewer": "Reviewer_G2Sv", "summary": "", "questions": "1. The reviewer is wondering whether the authors run experiments on larger dataset such GDELT dataset, which contains several million quadruples.\n\n2. As author stated in Section 6.1, the GCN-based models have overfitting problems. How are the model parameter numbers of GCN-based methods compared to SiMFy? Does the author try to reduce the model parameters of GCN-based method and see better performance?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions.\n\n**Q1: The reviewer is wondering whether the authors run experiments on larger dataset such as the GDELT dataset, which contains several million quadruples.**\n\n**A1:** Based on this comment, we did run experiments on the large dataset GDELT, and the results are shown in the following table:\n\n| **GDELT**   |         |         |         |         |\n|:-------:|:-------:|:-------:|:-------:|:-------:|\n|         | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |\n| xERTE   | \\-      | \\-      | \\-      | \\-      |\n| RE\\-NET | 19\\.60  | 12\\.03  | 20\\.56  | 33\\.89  |\n| CyGNet  | 17\\.99  | 11\\.10  | 19\\.03  | 31\\.26  |\n| CENET   | \\-      | \\-      | \\-      | \\-      |\n| GHT     | 20\\.04  | 12\\.68  | 21\\.37  | 34\\.42  |\n| CEN     | 19\\.27  | 11\\.96  | 20\\.60  | 33\\.50  |\n| RE\\-GCN | 19\\.07  | 12\\.16  | 20\\.20  | 32\\.39  |\n| **SiMFy**   | **21\\.33**  | **13\\.38**  | **23\\.16**  | **36\\.87**  |\n\nIt can be seen that, similar to the results on the ICEWS datasets, our SiMFy consistently outperforms all other baselines on the GDELT dataset. It should be noted that when we ran the source code provided by xERTE [1] and CENET [2] on GDELT, the running programs of xERTE and CENET crashed due to the out-of-memory issue.\n\n**Q2: As author stated in Section 6.1, the GCN-based models have overfitting problems. How are the model parameter numbers of GCN-based methods compared to SiMFy? Does the author try to reduce the model parameters of GCN-based method and see better performance?**\n\n**A2:** The model parameters of SiMFy are much smaller than those of GCN-based methods. This is because our SiMFy is simply composed of an MLP layer, which maps the input embedding to the output embedding. In addition to these options, the GCN-based models also perform the multi-step propagation operation, which will significantly increase the model parameters, especially when the depth of the GNN is increased.\n\nHere, we also vary the model parameters of GCN-based models to investigate its impact on the performance. We select two representative GCN-based models, i.e., CEN [3] and RGCN+ConvTransE, as target, and conduct experiments on the ICEWS14 dataset. Specifically, we reduce their model parameters via decreasing the depth (number of layers) of GNNs from 2 to 1, and decreasing the hidden units of each layer from 200 to 100. The experimental results are shown as below:\n\n| CEN     |              |         |         |         |         | RGCN\\+ConvTransE |              |         |         |         |         |\n|:-------:|:------------:|:-------:|:-------:|:-------:|:-------:|:----------------:|:------------:|:-------:|:-------:|:-------:|:-------:|\n| layers  | hidden units | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** | layers           | hidden units | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |\n| 2\\(\\*\\) | 200\\(\\*\\)    | **36\\.32**  | **26\\.83**  | 40\\.20  | 54\\.93  | 2\\(\\*\\)          | 200\\(\\*\\)    | **35\\.07**  | **26\\.17**  | **39\\.15**  | **52\\.23**  |\n| 2       | 100          | 36\\.06  | 26\\.29  | **40\\.38**  | **55\\.15**  | 2                | 100          | 33\\.98  | 24\\.70  | 38\\.04  | 52\\.06  |\n| 1       | 200          | 35\\.93  | 26\\.61  | 39\\.81  | 54\\.46  | 1                | 200          | 33\\.53  | 24\\.39  | 37\\.14  | 52\\.12  |\n| 1       | 100          | 35\\.99  | 26\\.38  | 40\\.29  | 54\\.42  | 1                | 100          | 33\\.89  | 24\\.49  | 38\\.25  | 51\\.66  |\n\nIn this table, the parameters marked with \"*\" are the default settings (i.e., 2 layers and 200 hidden units) for the model. It can be observed that when reducing the parameters of GCN-based models, the performance of the models will accordingly decrease to varying degrees. This demonstrates that the performance of GCN-based models is closely related to the number of model parameters. This is also the reason why GCN-based models are prone to be overfitted.\n\n[1] Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In International Conference on Learning Representations.  \n[2] Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2022. Temporal knowledge graph reasoning with historical contrastive learning. arXiv preprint arXiv:2211.10904  \n[3] Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, and Xueqi Cheng. 2022. Complex evolutional pattern learning for temporal knowledge graph reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper shows a simple approach combining one-layer MLP encoding and historical event frequency information achieved comparable or even better results than existing SoTA models on ICEWS datasets. The paper also demonstrates that the proposed approach has a faster convergence speed than existing SoTA methods and stronger generalization ability than CyGNet, RE-GCN etc. The author reports many thorough and interesting in-depth analysis regarding the effectiveness between GNN and MLP on tKG reasoning, which would benefit future research.\n", "reasons_to_reject": "The paper argues a simple MLP-based approach could beat SoTA models with complex model architecture. However, the paper only report the results on ICEWS datasets (14, 18, 05-15) but only on other tKG datasets such as GDELT, which is larger and more complex. The reviewer concerns that the observations on large-scale datasets maybe different.\n", "questions_to_authors": "1. The reviewer is wondering whether the authors run experiments on larger dataset such GDELT dataset, which contains several million quadruples.\n\n2. As author stated in Section 6.1, the GCN-based models have overfitting problems. How are the model parameter numbers of GCN-based methods compared to SiMFy? Does the author try to reduce the model parameters of GCN-based method and see better performance?\n", "paper_topic_and_main_contributions": "The paper studied the topic of temporal knowledge graph reasoning, especially on forecasting future event based on the information of structural dependency, temporality of entities and relations, as well as historical event patterns. The paper argues that the existing approaches are conceptually and technically complex, and are even over-designed regarding model architectures such that they have insufficient generalization ability. The authors proposed a simple yet effective approach, i.e., use a one-layer MLP to jointly encode entities and relations to capture structural and temporal information and get a preliminary candidate entity score, then calculate the historical frequency score of candidate entities. The final entity candidate score is a weighted average of the above two scores. The author empirically show that this simple approach has higher training efficiency and better generalization ability while can achieving comparable performance comparable performance to SoTA approaches with more complex architectures. Besides, they found MLP performs comparably to GNN in capturing structural dependency information for the TKG reasoning task. Additionally, the author demonstrates the effectiveness of using simple features to model the repetitive pattern of TKGs instead of directly incorporate historical frequency information of entities during training. \n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N.A.", "Justification_for_Ethical_Concerns": "N.A."}, "extra_text": {"Missing_References": "Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 3462–3471. JMLR. org, 2017.\n\nZhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021. Learning Neural Ordinary Equations for Forecasting Future Links on Temporal Knowledge Graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8352–8364, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics."}}, {"review_id": "ybSXyvgJ7Y", "reviewer": "Reviewer_Tw4C", "summary": "", "questions": "1. Could you also provide the filtered setting metrics for a fair comparison?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first thank the reviewer for the constructive comments and suggestions.\n\n**Q1: The reviewer is wondering whether the authors run experiments on the larger dataset GDELT.**\n\n**A1:** Taking this comment into consideration, we conduct experiments on the larger dataset GDELT (under raw setting, i.e., using the original ranking list predicted by the model). The results are shown in the table below:\n\n| **GDELT**   |         |         |         |         |\n|:-------:|:-------:|:-------:|:-------:|:-------:|\n|         | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |\n| xERTE   | \\-      | \\-      | \\-      | \\-      |\n| RE\\-NET | 19\\.60  | 12\\.03  | 20\\.56  | 33\\.89  |\n| CyGNet  | 17\\.99  | 11\\.10  | 19\\.03  | 31\\.26  |\n| CENET   | \\-      | \\-      | \\-      | \\-      |\n| GHT     | 20\\.04  | 12\\.68  | 21\\.37  | 34\\.42  |\n| CEN     | 19\\.27  | 11\\.96  | 20\\.60  | 33\\.50  |\n| RE\\-GCN | 19\\.07  | 12\\.16  | 20\\.20  | 32\\.39  |\n| **SiMFy**   | **21\\.33**  | **13\\.38**  | **23\\.16**  | **36\\.87**  |\n\nIt can be seen that, similar to the results on the ICEWS datasets, SiMFy outperforms all other baselines on the GDELT dataset. It should be noted that when we ran the source code provided by xERTE [1] and CENET [2] on GDELT, the running programs of xERTE and CENET  crashed due to the out-of-memory issue.\n\n**Q2: Could you also provide the filtered setting metrics for a fair comparison?**\n\n**A2:** First and foremost, it is important for us to explain why we choose the raw setting instead of the filtered setting in the scenario of TKG reasoning.\n\nIn fact, the filtered setting proposed by TransE[3] has been widely used in tasks related to **common knowledge graph completion**. However, in the scenario of TKG reasoning, the filtered setting is no longer applicable due to the introduction of timestamp **t**. Let's take a simple example: given a test quadruple **(Donald Trump, visit, Japan, 2019-05-25)**, and we perform the object prediction **(Donald Trump, visit, ?, 2019-05-25)**. We have observed the quadruple **(Donald Trump, visit, England, 2018-07-13)** in the training set. According to the filtered setting, **(Donald Trump, visit, England)** will be considered as a genuine triple at the timestamp **2019-05-25** and will be wrongly predicted as the correct answer because the triple **(Donald Trump, visit, England)** appears in the training set in the quadruple **(Donald Trump, visit, England, 2018-07-13)**. However, the triple **(Donald Trump, visit, England)** is only temporally valid on **2018-07-13** but not on **2019-05-25**. Thus, the filtered setting may probably get incorrect higher ranking scores. This also explains why **xERTE** [1], **CEN** [4], **RE-GCN** [5], and **GHT** [6] did not adopt the filtered setting.\n\nHere we also provide the experimental results under the filtered setting as follows:\n\n| GDELT\\(filtered\\) |         |         |         |         | ICEWS14\\(filtered\\) |         |         |         |         |\n|:-----------------:|:-------:|:-------:|:-------:|:-------:|:-------------------:|:-------:|:-------:|:-------:|:-------:|\n|                   | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |                     | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |\n| xERTE             | crashed |         |         |         | xERTE               | 40\\.79  | 32\\.70  | 45\\.67  | 57\\.30  |\n| RE\\-NET           | 40\\.12  | 32\\.43  | 43\\.40  | 53\\.80  | RE\\-NET             | 45\\.71  | 38\\.42  | 49\\.06  | 59\\.12  |\n| CyGNet            | **51\\.56**  | **45\\.63**  | **54\\.86**  | **60\\.96**  | CyGNet              | 48\\.63  | 41\\.77  | 52\\.50  | 60\\.29  |\n| CENET             | crashed |         |         |         | CENET               | 52\\.87  | 47\\.94  | 54\\.42  | 62\\.79  |\n| GHT               | \\-      | \\-      | \\-      | \\-      | GHT                 | \\-      | \\-      | \\-      | \\-      |\n| CEN               | 19\\.71  | 12\\.47  | 20\\.98  | 33\\.82  | CEN                 | 37\\.34  | 28\\.04  | 41\\.09  | 55\\.76  |\n| RE\\-GCN           | 29\\.46  | 21\\.74  | 32\\.01  | 43\\.62  | RE\\-GCN             | 38\\.27  | 28\\.40  | 42\\.54  | 57\\.88  |\n| SiMFy             | 47\\.40  | 40\\.17  | 50\\.81  | 60\\.46  | SiMFy              | **54\\.81**  | **47\\.99**  | **58\\.54**  | **66\\.65**  |\n\n| ICEWS18\\(filtered\\) |         |         |         |         | ICEWS05\\-15\\(filtered\\) |         |         |         |         |\n|:-------------------:|:-------:|:-------:|:-------:|:-------:|:-----------------------:|:-------:|:-------:|:-------:|:-------:|\n|                     | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |                         | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |\n| xERTE               | 29\\.31  | 21\\.03  | 33\\.51  | 46\\.48  | xERTE                   | 46\\.62  | 37\\.84  | 52\\.31  | 63\\.92  |\n| RE\\-NET             | 42\\.93  | 36\\.19  | 45\\.47  | 55\\.80  | RE\\-NET                 | 42\\.97  | 31\\.26  | 46\\.85  | 63\\.47  |\n| CyGNet              | 46\\.69  | 40\\.58  | 49\\.82  | 57\\.14  | CyGNet                  | 57\\.00  | 50\\.06  | 61\\.51  | 68\\.47  |\n| CENET               | **50\\.09**  | **46\\.02**  | **51\\.05**  | 57\\.85  | CENET                   | crashed |         |         |         |\n| GHT                 | \\-      | \\-      | \\-      | \\-      | GHT                     | \\-      | \\-      | \\-      | \\-      |\n| CEN                 | 28\\.21  | 19\\.10  | 31\\.80  | 45\\.89  | CEN                     | 37\\.53  | 27\\.57  | 41\\.49  | 57\\.51  |\n| RE\\-GCN             | 29\\.21  | 19\\.79  | 32\\.77  | 47\\.54  | RE\\-GCN                 | 38\\.68  | 28\\.26  | 43\\.44  | 59\\.11  |\n| SiMFy               | 46\\.87  | 39\\.29  | 51\\.00  | **60\\.23**  | SiMFy                   | **60\\.76**  | **53\\.43**  | **65\\.62**  | **72\\.94**  |\n\nIt needs to be clarified that **GHT** [6] used a raw setting in their paper, but the authors provide neither the filtered-version results nor the implemented source code. \"**Crashed**\" denotes that their program crashed when running on a specific dataset. From this table, we can observe that, under the filtered setting, all models obtain incorrect higher ranking scores compared to their performance under the raw setting. Although SiMFy still performs the best on the ICEWS14 and ICEWS05-15 datasets, it  performs second-best on the GDELT and ICEWS18 datasets. We argue that these experimental results obtained under the incorrect evaluation criteria (filtered setting) have little practical significance.\n\n[1] Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In International Conference on Learning Representations.  \n[2] Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2022. Temporal knowledge graph reasoning with historical contrastive learning. arXiv preprint arXiv:2211.10904  \n[3] Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data[J]. Advances in neural information processing systems, 2013, 26.  \n[4] Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, and Xueqi Cheng. 2022. Complex evolutional pattern learning for temporal knowledge graph reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.  \n[5] Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 2021b. Temporal knowledge graph reasoning based on evolutional representation learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 408–417.  \n[6] Sun, Haohai, et al. \"Graph Hawkes Transformer for Extrapolated Reasoning on Temporal Knowledge Graphs.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They propose a simple model architecture that works better\n2. It leads to training efficiency and better generalization performance.\n3. It has better convergence than other models.", "reasons_to_reject": "1. The authors report results on raw Hits@k and MRR. Whereas all the well-cited works report results on filtered metrics. Though the authors argue that the filtered setting has little practical relevance, all of the prior works (CyGNet, CENET, ) use that setting and not the raw one. The authors should have reported both metrics for a fair comparison.\n\\textbf{Update}: The authors agreed to provide results in the time-aware filtered setting which is widely used in literature and is also a fair metric.\n2. The choice of datasets is suspect. The GDELT [1] dataset is missing but has been used in all strong baselines.\n\n[1] Sun, Haohai, et al. \"Graph Hawkes Transformer for Extrapolated Reasoning on Temporal Knowledge Graphs.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "questions_to_authors": "1. Could you also provide the filtered setting metrics for a fair comparison?", "paper_topic_and_main_contributions": "This paper presents a simple approach for temporal KG reasoning. It uses just an MLP instead of more sophisticated GNN-based architectures proposed in the literature. This leads to training efficiency and better generalization performance. The ablation studies provide evidence for the design.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "evEAB56cBs", "reviewer": "Reviewer_AXyf", "summary": "", "questions": "1. The authors use Historical Frequency Learning to contain temporal features. However, there are many ways to deal with the temporal features. Why did the authors finally pick this frequency-based method? What about other similar temporal methods?\n2. The proposed method is intrinsically similar to a self-attention mechanism, as they all use attention-based scores to leverage the sequential encodings. More discussion on the main difference and connection with the attention-based models is recommended.\n3. What if the timestamp t becomes significantly large? Could there be forgetting issues like an RNN model?\n4. The authors use a simple MLP to draw the final prediction. What if the MLP is replaced with some deeper or more complicated neural structure? Will the performance become better or worse, and why?\n5. More discussion on the motivation of the proposed method is recommended.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank the reviewer for pointing out the issue regarding the example involving Russia and Ukraine. The concern is valid, and we will either add the correct reference or choose a more suitable example to explain the TKG reasoning task.\n\n**Q1: More discussion on the motivation of the proposed method is recommended.**\n\n**A1:** As required by the newly added track \"Efficient Methods for NLP\" at this year's conference, the direct motivation of SiMFy is to achieve comparable effectiveness when compared with complex models, but in a simpler and more efficient manner. We have found that through our designed approach, we can achieve comparable performance to existing models while having lower training costs and better generalization ability. As agreed by **Reviewer G2Sv**, we believe our work could not only shed some light on the future algorithm design for TKG reasoning but also inspire future studies to rethink the importance of simple model architectures rather than a complicated one.\n\n**Q2: The authors use Historical Frequency Learning to contain temporal features. However, there are many ways to deal with the temporal features. Why did the authors finally pick this frequency-based method? What about other similar temporal methods?**\n\n**A2:** In fact, we have provided detailed explanations and justifications for the selection of the Historical Frequency Learning strategy in **Section 6.4**. We have indeed tried many other methods to model temporal features, such as the time-encoding function used in GraphMixer [1] and traditional widely-used approaches like utilizing RNN to contain temporal features during the training process. However, through empirical study, we found that the strategy adopted by SiMFy not only achieves optimal performance but also has higher efficiency and generalization. Therefore, we choose Historical Frequency Learning to contain temporal features in our model.\n\n**Q3: The proposed method is intrinsically similar to a self-attention mechanism, as they all use attention-based scores to leverage the sequential encodings. More discussion on the main difference and connection with the attention-based models is recommended.**\n\n**A3:** Although it may seem similar at first glance, we must point out that there are significant differences between SiMFy's modeling approach for temporal quadruple sequences and the self-attention mechanism. In terms of self-attention mechanism, weights should be calculated for each quadruple **(s, r, o, t)** in different timestamps to represent the correlation between this quadruple and other quadruples in the sequence. However, SiMFy separates the entity-relation triple **(s, r, o)** from the corresponding quadruple and calculates the similarity scores between specific query **(s, r, ?)** and all entities first. As for the temporal dimension **t**, it is processed separately using the Historical Frequency Learning strategy based on chronological order.\n\n**Q4: What if the timestamp t becomes significantly large? Could there be forgetting issues like an RNN model?**\n\n**A4:** In general, forgetting issues definitely exist. However, in the scenario of TKG reasoning, we actually need the model to appropriately \"forget\" some previous sequence information sometimes. This is because in the TKG, as time goes by, the connections between entities and relations in the KG are constantly evolving. For the TKG reasoning task, as clarified by CEN [2], we are more concerned about the recent evolutional pattern of KG sequence.\n\n**Q5: The authors use a simple MLP to draw the final prediction. What if the MLP is replaced with some deeper or more complicated neural structure? Will the performance become better or worse, and why?**\n\n**A5:** To answer this question, here we add experiments to replace the MLP with some deeper and more complicated neural structures, e.g., multi-layer MLP and RNN. The experimental results on the ICEWS14 dataset are presented as follows:\n\n| ICEWS14         |         |         |         |         |\n|:---------------:|:-------:|:-------:|:-------:|:-------:|\n|                 | **MRR**     | **Hits@1**  | **Hits@3**  | **Hits@10** |\n| **SiMfy\\(1MLP\\)**     | **39\\.54**  | 29\\.56  | **44\\.56**  | **59\\.18**  |\n| SiMfy\\(2MLP\\)     | 39\\.49  | 29\\.74  | 43\\.97  | 58\\.58  |\n| SiMfy\\(3MLP\\)     | 39\\.29  | 29\\.74  | 43\\.46  | 58\\.20  |\n| SiMfy\\(RNN\\+MLP\\) | 39\\.27  | 29\\.45  | 43\\.89  | 58\\.36  |\n\nFrom this table, we can see that as the structure of the model becomes more complex, its performance has even slightly worsened. We think that this is because the complex model structure, while increasing computational costs, does not better capture the evolving information of entities and relations. However, more specific reasons would require further and extensive experimentation to investigate, which are left to our future work.\n\n[1] Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. 2023. Do we really need complicated model architectures for temporal networks? arXiv preprint arXiv:2302.11636.  \n[2] Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, and Xueqi Cheng. 2022. Complex evolutional pattern learning for temporal knowledge graph reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. The paper might potentially impact the understanding of the knowledge graph.\n3. The empirical results are promising and impressive.\n4. The proposed methods are simple but efficient, and the frequency-based mechanism is novel in the NLP community.", "reasons_to_reject": "1. The part related to Russia and Ukraine needs an academic reference. The authors should know that an academic paper must be rigorous: any information presented should be clear and objective, properly citing academic references. An academic paper should be responsible for elucidating academic facts and should not be a medium of historical and political events.\n2. The motivation and theoretical validation are insufficient to demonstrate the superiority of the proposed method.", "questions_to_authors": "1. The authors use Historical Frequency Learning to contain temporal features. However, there are many ways to deal with the temporal features. Why did the authors finally pick this frequency-based method? What about other similar temporal methods?\n2. The proposed method is intrinsically similar to a self-attention mechanism, as they all use attention-based scores to leverage the sequential encodings. More discussion on the main difference and connection with the attention-based models is recommended.\n3. What if the timestamp t becomes significantly large? Could there be forgetting issues like an RNN model?\n4. The authors use a simple MLP to draw the final prediction. What if the MLP is replaced with some deeper or more complicated neural structure? Will the performance become better or worse, and why?\n5. More discussion on the motivation of the proposed method is recommended.", "paper_topic_and_main_contributions": "The paper explores the effectiveness of complex model architectures in Temporal Knowledge Graph (TKG) reasoning, a critical component of knowledge graph completion. The authors highlight issues with the current models, such as low training efficiency and inadequate generalization ability. To address these concerns, they propose a simpler approach named SiMFy, which employs a multilayer perceptron (MLP) to model the structural dependencies of events and incorporates historical frequency during inference using a fixed-frequency strategy. The significant contributions of this research are: 1) Demonstrating that SiMFy can achieve state-of-the-art performance, outperforming complex models in faster convergence speed and better generalization ability; 2) Showing that SiMFy consumes less time in the training process; and 3) Providing evidence that SiMFy is better able to capture the structural dependencies of events in KGs. The results suggest that replacing complex models with simpler counterparts is feasible.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper evaluates the necessity of complex models for temporal knowledge graph reasoning and introduces a streamlined approach, SiMFy, that uses a multilayer perceptron (MLP) for modeling event structural dependencies and employs a fixed-frequency strategy for inference. SiMFy achieves comparable performance to SoTA approaches with faster convergence, reduced training time, and capturing structural dependency information for the TKG reasoning task.

This paper is technically sound, and the experiments are conducted appropriately, although the impact of the results is relatively incremental. As reviewers point out, experiments on larger datasets such as the GDELT dataset, will strengthen the paper. More importantly, as reviewer AXyf pointed out, the example of the Russia-Ukraine conflict is inappropriate from an academic perspective. Additionally, using TKG for predicting the future is less sensible. It would be more useful and appealing to identify similar event structures from the past for reference.
Decision: Accept-Findings