Tucker Decomposition with Frequency Attention for Temporal Knowledge Graph Completion
Year: 2023

Abstract:
Temporal Knowledge Graph Completion aims to complete missing entities or relations under temporal constraints. Previous tensor decomposition-based models for TKGC only independently consider the combination of one single relation with one single timestamp, ignoring the global nature of the embedding. We propose a Frequency Attention (FA) model to capture the global temporal dependencies between one relation and the entire timestamp. Specifically, we use Discrete Cosine Transform (DCT) to capture the frequency of the timestamp embedding and further compute the frequency attention weight to scale embedding. Meanwhile, the previous temporal tucker decomposition method uses a simple norm regularization to constrain the core tensor, which limits the optimization performance. Thus, we propose Orthogonal Regularization (OR) variants for the core tensor, which can limit the non-superdiagonal elements of the 3-rd core tensor. Experiments on three standard TKGC datasets demonstrate that our method outperforms the state-of-the-art results on several metrics. The results suggest that the direct-current component is not the best feature for TKG representation learning. Additional analysis shows the effectiveness of our FA and OR models, even with smaller embedding dimensions.

heading: 1 Introduction
text: Knowledge graph (KG) contains a number of structured facts (h, r, t), where a fact expresses a directed relation r from a head entity h to a tail entity t. The complex KGs, such as FreeBase (Berant et al., 2013), DBPedia (Auer et al., 2007), and Wikidata (Vrandečić and Krötzsch, 2014), are collected manually or automatically from structured or unstructured data on the web. Such KGs are successfully applied to several downstream tasks, e.g., Question Answering (Berant et al., 2013) and
∗Corresponding author: zhangrc@act.buaa.edu.cn
Recommender System (Wang et al., 2018). However, those works ignore that many facts in the KGs change over time. Temporal facts can be expressed as a quadruplet (h, r, t, τ), with τ being the timestamp. The temporal Knowledge graphs (TKGs), such as ICEWS (Lautenschlager et al., 2015), GDELT (Leetaru and Schrodt, 2013) and YAGO (Mahdisoltani et al., 2015), are then built to handle these facts coupled with timestamps.
One problem that hinders the application of TKGs in downstream tasks is the inevitable incompleteness or knowledge scarcity problem caused by missing entities or relations. Thus, Temporal Knowledge Graph Completion (TKGC) aiming to complete the missing entities or relations over time has become an essential task in the research community. The previous methods for TKGC can be divided into four branches, in which the critical challenge is how to integrate timestamps into KGC modeling. Time-dependent Embedding method (Trivedi et al., 2017; Goel et al., 2020; Dasgupta et al., 2018) considers the temporal information as a transformation or an activation function for entities or relations. Timestamp Embedding method (Han et al., 2021b; Lacroix et al., 2020; Shao et al., 2022) treats timestamps as additional learnable embeddings of the score function. Experimental experience (Han et al., 2021b) suggests that timestamp embeddings generally perform bet-
7286
ter than time-dependent embeddings. Knowledge Graph Snapshots method (Liao et al., 2021; Li et al., 2021) aggregates multi-relational interactions of cropped subgraph to achieve more precise representations. Historical Context method (Jung et al., 2021; Zhu et al., 2021) model n-hop facts chain or repeat facts to increase the interpretability.
The previous timestamp embedding methods model each quadruplet independently, which only captures the local temporal dependencies, ignoring the global temporal dependencies between one relation and entire timestamp. As shown in Figure 1, an athlete plays for different teams in different periods. We treat such events as a continuous line of events rather than as separate events. We propose a Frequency Attention (FA) model to address this issue. Specifically, we treat each dimension in the timestamp embedding as a long-term signal and use Discrete Cosine Transform (DCT) to capture the frequency of the signal. Furthermore, we take the frequency and part of the relation embedding as input to calculate attention weights for each timestamp. The proposed frequency attention model can easily apply to exist tensor decomposition methods.
The previous tucker decomposition method is interpreted as a high-dimensional linear classifier to distinguish facts. TuckERTNT (Shao et al., 2022), uses a simple L2 norm as the core tensor regularization. However, this regularization may be overstrict, leading the embedding to change sharply and risk vanishing or explosion. Inspired by orthogonal regularization in (Brock et al., 2019), we propose two variants of orthogonal regularization (OR) for the core tensor, i.e., excluding superdiagonal elements or diagonal elements of each slice matrix of the core tensor. This way, we achieve a balanced core tensor regularization, preventing the embedding norm from vanishing or exploding.
In summary, our work makes the following contributions:
(1) we propose a frequency attention model using DCT to capture the global temporal dependency between relations and entire timestamp.
(2) we introduce two variants of core tensor orthogonal regularization for the tucker decomposition, which can prevent the embedding norm from vanishing or explosion.
(3) Experiment results on three standard datasets show that our model outperforms the SOTA models on several metrics. The additional analysis demonstrates the effectiveness
of our frequency attention model and orthogonal regularization.

heading: 2 Related Works

heading: 2.1 Static KG Embedding
text: There has been ample research on static knowledge graph embedding. We grouped all mainstream models into four main categories. Tensor decomposition-based models RESCAL (Nickel et al., 2011), Distmult(Yang et al., 2015), ComplEx (Trouillon et al., 2016), and TuckER (Balazevic et al., 2019) compute triplet score in real or complex domain. Distance-based models are built upon euclidean or hyperbolic distance as shown in TransE (Bordes et al., 2013), mainfoldE (Xiao et al., 2016) and RotatE (Sun et al., 2019). DURA (Zhang et al., 2020) figure out that distance-based method can be viewed as decomposition method with a L2 regularization. Neural-based models use a convolutional network to capture the KG structure information, as shown in ConvE (Dettmers et al., 2018). Other models learn from a variety of experiences from fields. Inspired by reinforcement learning, MultiHopKG (Lin et al., 2018) sample nhop triplets chain to compute the fact triplet scores and re-rank candidates.

heading: 2.2 Temporal KG Embedding
text: There are two scenarios for integrating temporal information into existing static embedding models, timestamp embedding and time-dependent entity embedding. Time-dependent entity embedding can explicitly model dynamic changes of entities, such as periodicity and trending. A well-known timedependent entity embedding is diachronic embedding (Goel et al., 2020), which uses the sine function to represent the frequency of entity evolution over different time granularity. (Han et al., 2021b) compares six KG embedding models, and figures out that timestamp embedding can achieve similar or even better performance with significantly fewer parameters. Although timestamp embedding might suffer from the growing number of timestamps, the time granularity can also be controlled within an appropriate range by enlarging. Further analysis in TNTComplEx (Lacroix et al., 2020) points out that time-dependent relation embedding can obtain comparable results to time-dependent entity embedding with smaller computational costs.
From the viewpoint of the subgraph, there are a series of knowledge graph snapshots/subgraphs
over time, which contain potential multi-relational interactions. (Liao et al., 2021) adopt probabilistic entity representations based on variational Bayesian inference to jointly model the entity features and the uncertainty. (Li et al., 2021) employs a multi-layer graph convolutional network on each subgraph to capture the dependencies of adjacent facts. From another contextual perspective, the relevance between the query and its historical context can be used as evidence for reasoning. (Jung et al., 2021) proposes a multi-hop reasoning model using a graph attention layer and finds that temporal displacements are more indicative for inference than timestamps. (Zhu et al., 2021) notice more than 80% of events from 1995 to 2019 in the ICEWS repository are repeated events. In this case, they introduce a copy mechanism to re-rank the candidates.

heading: 3 Preliminaries

heading: 3.1 Problem Definition
text: To formally define the problem and describe the solution, we use consistent notations in the rest of the paper. We represent scalars with the lower case letters, e.g., dr, represent sets with the flower letters, e.g., E , represent vectors with the bold lower case letters, e.g., hi, denoting the ith entity embedding. We use bold upper letters H to denote the embedding matrix and represent the high order tensor with bold flower letters W .
We use a ⊙ b to denote Hadamard (elementwise) product of two vectors or matrix, a ⊗ b to denote the tensor outer product, [A|B] to denote the vector or matrix concatenation operator, || · ||p to denote the p-norm of a vector or tensor.
A temporal knowledge graph G consists of a set of facts {(hi, rj , tk, τl)} ⊆ E × R × E × T , where E is a finite entity set, R is a finite relation set and T is a finite timestamp set. Each quadruplet (hi, rj , tk, τl) respectively denotes a relation rj from the head entity hi to tail entity tk at a specific time τl. A temporal knowledge graph uses a binary tensor X = {0, 1}|E|×|R|×|E|×|T | to indicate whether the corresponding quadruplets occurs in the KG data set. |E|, |R| and |T | denote the number of entities, relations, and timestamps, respectively.
Although knowledge graphs contain large numbers of facts, they are still incomplete due to the complex nature of the real world. TKGC aims to predict the missing entity. We focus on the link
prediction problem, aiming to predict the tail entity or head entity through the query (hi, rj , ?, τl) or (?, rj , tk, τl). The problem reduces to ranking a set of candidate entities to select the most likely entity that makes the partial quadruplet factual. The problem can be formulated as a ranking problem to learn a quadruplet score function X̂ (E ,R, E , T ) ∈ R to sort all candidate entities.

heading: 3.2 Tucker Decomposition for TKG Embedding
text: Many tensor decomposition methods apply to the KG embeddings, such as bilinear decomposition, canonical decomposition, and tucker decomposition. Among these methods, tucker decomposition, a kind of principal component analysis approach for high-order tensors, is viewed as the general one. In particular, when the super-diagonal elements in the core tensor of Tucker equal 1 and other elements equal 0, tucker decomposition degrades into canonical decomposition. TuckER (Balazevic et al., 2019) has proved that Dismult (Yang et al., 2015) and ComplEx (Trouillon et al., 2016) can be included into the framework of TuckER. In KGC task, An 3-order tensor X can be decomposed into a core tensor W ∈ RDe×Dr×De and entity/relation embedding matrix E/R as factor matrix. The formula of the tucker decomposition is as follows.
X =W×1E×2R×3E=<W ;E,R,E>
=
De∑
d1=1
Dr∑
d2=1
De∑
d3=1
Wd1d2d3h:d1⊗r:d2⊗t:d3
×n denotes the n-mode product of the tensor, which can be explained as the core tensor expanding into a matrix along the n-th dimension.
To obtain proper timestamp embedding T , TuckERTNT (Shao et al., 2022) use two relation embeddings R and Rt to separately capture time-variant information and time-invariant information as follows.
X =< W ;E,Rt ⊙ T +R,E >
Although previous works have achieved good results in the TKGC task, they may still encounter many problems. First, the learnable parameters representing the frequency of DE-SimplE may be clustered around 0, affecting the model performance (as shown in Appendix D). Second, TuckERTNT constrains the core tensor with the fourth power of L4 norm. However, it is not guaranteed that the
n-mode product of the core tensor is well-perform. Therefore, there is still space to improve the temporal tucker decomposition for the TKGC task.

heading: 4 Model
text: We propose a new framework, TuckER-FA, combining the FA and OR with the temporal tucker decomposition method. We input the timestamp embedding and relation embedding to the FA model to compute frequency attention weights, then weighted-sum the timestamp embedding and combine it with the relation embedding. The timestamp-enhanced relation feature and head/tail entity embedding compose the factor matrix of tucker decomposition. In learning progress, we include several regularization losses and the orthogonal regularization of the core tensor into the overall objective.

heading: 4.1 Score Function
text: We propose a score function for TKGC based on tucker decomposition shown in Figure 2. Specifically, the formula of the score function is.
Xijkl=<W ;hi,rj1⊙[rj2|FA(r̂j1,tl)],tk>
We use ρ = dτdr to control the embedding dimension ratio between timestamps and relations. rj1 ∈ Rdr and rj2 ∈ Rdr−dτ respectively denote two relation embeddings. τl ∈ Rdτ is the timestamp embedding. FA represents the Frequency
Attention model. The input r̂j1 denote the part of relation embedding which aligns with τl. The main amount of computation is concentrated on tucker decomposition rather than the FA model.

heading: 4.2 Frequency Attention
text: To capture the crucial temporal features of TKG, we propose a Frequency Attention (FA) model shown in Figure 3. We treat the evolution of timestamp embedding over time as a combination of periodic functions with different frequencies. Inspired by FcaNet (Qin et al., 2021), we use Discrete Cosine Transform (DCT) to capture the different frequency components of timestamp embeddings. In this way, we can capture the global temporal dependency of one relation and the entire timestamp.
The chronologically arranged timestamp embedding T ∈ RNτ×dτ is viewed as dτ different temporal signals. The direct-current (DC) component f0 and frequency component fk of DCT are respectively formulated as follows.
f0=
Nτ−1∑
i=0
Ti = GAP (T )Nτ
fk=
Nτ−1∑
i=0
Ticos( πk
Nτ (i+
1 2 )) k∈{0, ..., Nτ−1}
GAP represents the global average pooling operation, which always use to calculate the channel attention weight in the computer vision domain. In the TKGC task, the main calculation procedure of direct-current component frequency attention is as follows.
FA(τ l) = σ(FC(GAP (T:d)Nτ ))⊙ τ:d
The FC block represents a fully-connected layer, and σ denotes sigmoid function. It is natural to include more frequency components to calculate attention weight. Considering the limitation of computing resources, we optionally select part of the frequency components. We divide the time embedding dimension dτ into n parts and assign a set of selected frequencies f0, ...fn to each part. We also introduce r̂j1, part of relation embedding aligned with τl, into the FA model.
FA(r̂j , τ l)=σ(FC(r̂j1 ⊙ [f0|f1|...|fn]))⊙ τ l
In addition, the computation complexity of the operation with finite orthogonal function bases is linear. The cost of computation of the frequency
attention model is negligible compared to the cost of computation in tucker decomposition. The frequency attention weight determines how much timestamp embedding information for the corresponding dimension is retained. The FA model considers the evolution of a single relation over the entire timestamp.

heading: 4.3 Orthogonal Regularization for Core Tensor
text: There have been many research results on orthogonal regularization of the matrix, such as (Miyato et al., 2018). In summary, the orthogonal regularization allows the parameter matrix to be closer to the diagonal-dominant non-singular matrix. A nonsingular matrix prevents abrupt truncation changes of the feature map during matrix multiplication. Furthermore, tucker decomposition can be viewed as matrix multiplication for factor matrix with the arbitrary slice of core tensor. In other words, orthogonal regularization could be applied to core tensor multiplication.
Inspired by BigGAN (Brock et al., 2019), we heuristically propose two variants of core tensor orthogonal regularization Φ1 and Φ2. The baseline is the simple norm regularization Φ0 used in
TuckERTNT.
Φ0(θ) = ||W ||44 Φ1(θ) = ||W ⊙ (1− I)||44 Φ2(θ) = ||W ⊙ (1− Proj(I))||44
1,I, P roj(I) is tensors with the same shape as W . All the elements of 1 are 1. The superdiagonal elements of I are 1, and the other elements are 0. The diagonal elements of the arbitrary slice matrix of Proj(I) are 1, and the other elements are 0. The tucker decomposition degenerates to the CP decomposition when the super-diagonal elements of the core tensor are 1, and the rest are 0. Φ1 regularization can restrict the result of the tucker decomposition to the neighborhood of the weighted CP decomposition. (the super-diagonal elements of the core tensor are weights, and the rest elements indicate a slight difference from weighted CP decomposition.

heading: 4.4 Other Regularization
text: Researchers have investigated many different kinds of embedding regularization to alleviate the overfitting problem. TNTComplex uses the third power of Nuclear-3 norm twice for temporal or non-temporal quadruplets. TIMEPLEX (Jain et al., 2020) use sampled weighted L2 regularization to avoid the
overfitting problem. In our model, we use embedding regularization as ChronoR (Sadeghian et al., 2021) does, using the fourth power of L4 norm as embedding regularization.
Ω4(θ)= ||h||44 + ||t||44 + ||r1||44 + ||[r2|FA(τ )]||44
Because of the real-world time continuity, it is natural to guarantee adjacent timestamp embeddings or repeat timestamp embeddings closer in the embedding space. TuckERTNT (Shao et al., 2022) proposes several temporal regularization to smooth the timestamp embedding. Notice that YAGO15K has many quadruplets without any timestamp, in which we artificially add a unique timestamp. This unique timestamp is excluded when computing temporal regularization terms. To be consistent with the embedding regularization, the adjacent timestamp differential regularization is as follow.
Λ4(θ) = 1
|T | − 1
|T |−1∑
i=1
||τ i+1 − τ i||44

heading: 4.5 Loss Function
text: For each training data, we use instantaneous multiclass loss.
L(Xijkl) = −Xijkl + log( ∑
k′ exp(Xijk′ l))
Considering instantaneous multi-class loss and the above three regularization term jointly, we train our model by minimizing the following loss function.
L(X ; θ) = 1|S| ∑
(i,j,k,l)∈S [L(X ; θ) + λΦn(X ; θ)
+λ1Ω4(X ; θ) + λ2Λ4(X ; θ))] n = 1, 2, 3
where λ1, λ2 and λ is importance hyperparameter for tuning.

heading: 5 Experiments

heading: 5.1 Datasets and Evaluation Metrics
text: We choose three of the most commonlyused datasets to evaluate our model, including ICEWS14, GDELT, and YAGO15K. The detailed statistics of each dataset are shown in Table 1.
ICEWS14 is extracted from the Integrated Crisis Early Warning System (Lautenschlager et al., 2015) repository, which contains political events with daily timestamp points. This dataset, for the
most part, is time-sensitive and accurate in descriptions.
GDELT is extracted from the Global Database of Events, Language, and Tone (Leetaru and Schrodt, 2013), which covers news data from 1979 to the present by automatically crawling. GDELT is a complicated dataset because of its abstract entity, such as government and organization.
YAGO15K (Friedland and Lim, 2018) augmented events of FB15K (Bordes et al., 2013) with time interval. YAGO15K is worst-perform in TKGC tasks because it requires the model to handle both temporal and non-temporal knowledge.
We follow the standard evaluation set in previous work, and report two standard metrics, Hit@k (k ∈ {1, 3, 10}) and filtered Mean Reciprocal Rank (MRR). They can evaluate the rank of the correct entity in the filtered candidate set. Hit@k reflects the percentage of the query whose correct tail entities are ranked within the top k candidates. Mean Reciprocal Rank, which computes the average of the reciprocal of mean rank, reflects the correct fact rank of the model. We follow the time-aware filtering (Han et al., 2021a), which means entities that cause ambiguity are removed from the candidate list for a query. We using reciprocal setting to add (tk, r −1 j , hi, τl) into train set for each quadruplet (hi, rj , tk, τl). The detailed hyperparameters of our model are shown in Appendix B.

heading: 5.2 Main Results
text: Table 2 shows the main temporal knowledge graph completion results. The results of other models come from the original paper. We use the bold number to indicate the existing best results. Our model slightly outperforms or ties with previous SOTA results on several metrics of ICEWS14 and YAGO3-10. On GDELT, our model achieves significant improvement results on all metrics. Compared with TuckERTNT, our model TuckER-FA
has d[(1− ρ)|T |+ (2 + ρ)|R|] fewer parameters and 1.1% higher MRR performance when using the same embedding dimensionality. Compared with 2500+ dimensions of entity and relation of TNTComplex and ChronoR, our model gets better results using only 400 dimensions of entity and relation.
Increasing the number of model parameters substantially improves MRR performance on the GDELT, but the improvement on the other two models is slight. Compared with ICEWS14, the GDELT dataset has nearly 38 times the number of facts, and fewer entities/relations, within the same time span. The corresponding graph of GDELT is spatially denser, with many more recurring facts. As a result, GDELT greatly enhances the global temporal dependency of relations, which is exactly our FA model focus on. Complex global temporal dependency explains TuckER-FA outstanding advantage on GDELT compared with its counterparts.

heading: 5.3 Ablation Study
text: Table 3 shows the ablation study of the Frequency Attention(FA) model and core tensor Orthogonal Regularization(OR) model. It can be observed that
both models are valuable when working individually, and the combination of them performs even better. We can point out that the OR model is more effective than the FA model.
The single FA model increases the accuracy significantly in ICEWS14 compared with the vanilla model, which is probably because this dataset has the shortest time span and the minimal data. The improvement of the single OR model is slight in YAGO15K, and it may be attributed to the presence of facts without timestamps.

heading: 5.4 Frequency of Temporal Knowledge Graph
text: From the result of FcaNet (Qin et al., 2021) in the Appendix E, we can notice frequency atten-
tion model with a single direct-current component always reach the best result in the Image Classification task. The low-frequency component is more critical than the high-frequency component because the low-frequency signal of the images provides information such as shape or size, while the high-frequency signal provides information such as edge details. However, this phenomenon does not occur in the TKGC task. Usually, it is difficult to have many repeated details in a naturally captured image. In contrast, many facts, such as the toy example, continuously change over time in TKGC. These facts determine that there exist some inherent frequencies in timestamp embedding.
Figure 4 shows the results of the frequency attention model with different single DCT bases. We use a combination of frequencies from the top six results to achieve our best results. The difference between the frequencies was insignificant, meaning the global temporal dependency spread over all frequencies. In detail, the direct-current component is the second-high result in ICEWS14 but the sixth-high in YAGO15K. The reason may be that the facts without a timestamp in YAGO15K disturb the estimation of the intrinsic frequency. The above results show that finding a suitable frequency feature with FA is helpful for the TKGC task.
To study why TuckER-FA performs differently on different datasets compared to other baselines, we count the number of the query (h, r, ?, ?) occur-
rences over the entire timestamp. The average number of occurrences per query in GDELT is 284.7, and queries that occur more than once account for 98.8% of the training set. The statistics are 7.33, 40.8% in ICEWS14, and 4.51, 54.9% in YAGO15k, respectively. Thus, TuckER-FA gains better results on GDELT because the FA module is good at capturing the global temporal dependencies between one relation over the entire timestamp.
Table 4 show the MRR Results for two test set divisions on ICEWS14 and YAGO15k. We split the test set into two subsets, one consisting of queries that occur only once (direct-currency component, DC) and the other consisting of queries that appear multiple times (high-frequency component, HF). In conclusion, our model works much better on the high-frequency component test set. Because the GDELT dataset is almost exclusively high-frequency components, it boosts most significantly.

heading: 5.5 Orthogonal Regularization
text: Figure 5 shows the detailed comparison of three regularizations for the core tensor. In this experiment, we fix the λ1 and λ2 and only use the direct-current component for the frequency attention model. The Φ2 regularization without the diagonal elements of the arbitrary slice matrix of the core tensor performs slightly better than the Φ0 norm regularization. The Φ1 regularization without the superdiagonal elements of the core tensor performs best.
The Φ1 regularization increase MRR by 0.7 points on ICEWS14 and 1.8 points on YAGO15K. Top Performing Φ1 can increase MRR to 2.1 points on ICEWS14 and 3.1 points on YAGO15K. We can point out that the relaxation of the core tensor constraint is effective.

heading: 5.6 Effect of Parameter Complexity
text: To compare model results more fairly, we add two experiments controlling the number of parameters or entity dimensionality between TNTcomplEx, TeLM, and TuckER-FA. A complete parameter comparison between baseline models and TuckERFA is in Appendix A.
Figure 6 shows our TuckER-FA consistently outperforms the baseline TNTComplEx and TeLM with the same dimensionality on ICEWS14. TNTComplEx uses a specialized matrix to represent non-timestamp embedding, which allows it to perform best in low dimensionality on YAGO15K. Our TuckER-FA can overtake in high dimensionality. The upper limit of TeLM is not as high as TuckER-FA. In summary, our model has the highest performance ceiling for TKGC.
Table 5 shows the comparison with the same parameters and 50 epochs training on ICEWS14. We limit the size of the learnable parameters to approximately 67M, which means 2110 embedding dimensions for TeLM, 4000 for TNTComplEX, and 400 for our TuckER-FA. We can point out that our TuckER-FA model achieves a significant improvement in MRR and Hit@3 and a weak improvement in the other two metrics.

heading: 6 Conclusion
text: In our work, we propose a DCT-based Frequency Attention model and two variants of Orthogonal Regularization for the core tensor of tucker decomposition. The FA model considers the global temporal dependency between one relation and the entire timestamp. Each KG has its unique inherent frequency. The OR term relaxes the constraint on the superdiagonal of the core tensor and improves the performance of tucker decomposition. TuckER-FA achieves SOTA results on three standard datasets of temporal knowledge graph completion task. There might be further discussions on an efficient frequency selection strategy or a theoretical assumption for tensor regularization.

heading: 7 Limitation
text: Although our method has been shown effective, it has two limitations that may be improved in the future. First, the FA model has advantages in computation but relies on an effective frequency selection strategy, which is difficult to design. We just simply select some manual frequencies for different datasets by experience. The more effective frequency selection strategy needs further exploration. Second, there is no theoretical guarantee that the orthogonal regularization can generalize to a 3-order tensor. Our OR terms are only formally consistent with matrix orthogonal regularization, which has been empirically shown effective.

heading: Acknowledgements
text: This work is supported partly by the National Key R&D Program of China under Grant 2021ZD0110700, partly by the Fundamental Research Funds for the Central Universities, and partly by the State Key Laboratory of Software Development Environment.

heading: A Parameter Complexity
text: When d is smaller the number of entity/relation/timestamp collection, tucker decomposition has considerable parameter advantage. Compared with TuckERTNT of the same type, our TuckER-FA gets better results with parameter advantage in modeling relation embedding.

heading: B Hyperparameters
text: We implement our model based on two previous models, TuckER and TNTcomplex. Other baseline models mentioned above have yet to provide publicly available code. During the pre-processing data phase, we convert all time intervals into two different time points and consider them independent. The time intervals in YAGO15K look like "OccursUntil/OccurSince 1994". The time interval is split into two parts, "OccursSince" or "OccursUntil" merged into a relation, and the time point transforms the timestamp. Note that quadruplet without timestamps in YAGO15K also own a unique timestamp.
Although the dimensionality of entity and relation can be different, we use the same dimensionality in our experiments. For general learning settings, we set the dimensionality of entity and relation to 800, batch size to 1000, learning rate to 0.1, and dropout probability to 0.3. Each embedding initializes from 0.01 times the Standard Gaussian distribution. Moreover, the learnable parameters of the core tensor initialize from the uniform distribution from -1 to 1. The frequency attention model uses a single frequency component from f0 to f12
input. We choose ratio ρ form {0.45, 0.6, 0.75, 0.9}, embedding regularization balance term λ1 from {1e-1, 1e-2, 1e-3, 1e-4}, temporal regularization balance term λ2 from {1e0, 1e-1, 1e-2, 1e-3}, and core tensor regularization balance term λ from {1e-2, 1e-3, 1e-4, 1e-5}. Note that the timestamp embedding dimensionality should be divisible by n, the number of the selected frequency components. We repeats the experiment three times and reported the average results.
C Visualization of Toy Example
To illustrate whether our frequency attention model captures the temporal dependencies between a relation and the entire timestamp, we visualize the scores of a selected set of facts. Figure 7 show the scores change of the toy example. The factual tail entity can consistently score high by TuckER-FA. The gray entity’s score changes very little because its ground truth belongs to the test set. The ground truth of the other three color entities belongs to the train set. The facts in the same category (Tom Adeyemi, playsFor,?) change quickly, which reflects a high intrinsic frequency of global temporal dependency. The scores of ground truth are very high on timestamps where facts exist, while all entities achieve a low score on timestamps where facts do not exist. Our model can capture the fastchanging temporal dependency of facts in the same category.

heading: D The Frequency of DE-SimplE
text: DE-SimplE divides the timestamp 2014-01-31 into three numbers representing the year, month, and day. Then, these time numbers are fed into the sine function along with the frequency and phase parameters. Figure 8 shows the histogram of learned frequency parameters of DE-SimplE on the ICEWS14 dataset. These learned parameters concentrate around 0, meaning only low-frequency information about entities over time is captured.
In previous work (Xu et al., 2020), the evolution process is divided into four different components: static component, periodicity component, trend component, and randomness component. In our opinion, the random component focuses on model robustness, and the static component focuses on the static entity or relation rather than the timestamp. The periodicity and trend components mean the temporal dependency of relations and timestamps, which can be captured by a periodic function such as cosine. If the period of the function is greater than the entire time span, then the cosine function captures the trend component. Similarly, if the period of the function is less than the entire time span, the cosine function captures the periodicity component.

heading: E Frequency Attention for Image
text: The image uses 2-dimensional DCT as Figure 9, while the KG uses only one-dimensional DCT. Frequency is an essential characteristic of DCT and indicates how many repetitive structures there are in the data. The main body of the image is composed of low-frequency features, while the TKG body has high-frequency features due to repeated quadruplets.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: section 7, before the References
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: appendix B

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: section 5.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 5.1

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? appendix B

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 5.1
C 3 Did you run computational experiments? section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 5.6 and appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? appendix B

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Although our method has been shown effective, it has two limitations that may be improved in the future. First, the FA model has advantages in computation but relies on an effective frequency selection strategy, which is difficult to design. We just simply select some manual frequencies for different datasets by experience. The more effective frequency selection strategy needs further exploration. Second, there is no theoretical guarantee that the orthogonal regularization can generalize to a 3-order tensor. Our OR terms are only formally consistent with matrix orthogonal regularization, which has been empirically shown effective