SlowGuess/NeurIPS_2023_OCR huggingface dataset:
# 1. predefined query structures and the Interpreter
ComplexQueryFunctions = {
"Pe2": "def Pe2(e1, r1, t1, r2, t2): return Pe(Pe(e1, r1, t1), r2, t2)", # 2p
"Pe3": "def Pe3(e1, r1, t1, r2, t3, t3): return Pe(Pe(Pe(e1, r1, t1), r2, t2), r3, t3)", # 3p
"ApT": "def aPt(s, r, o): return TimeAfter(Pt(s, r, o))", # a for after
"bPt": "def bPt(s, r, o): return TimeBefore(Pt(s, r, o))", # b for before
...
}
Figure 7: Python-style pseudocode of ground-truth reasoning interpreter.
the performance of data-sampling process. To solve this problem, we use the following two strategies to accelerate the sampling.
(1) Inverse Sampling: We randomly sample an entity as objective, denoted  $e_{o}$ . Then we sample subjective entity  $e_{s_1}, e_{s_2}$ , relation  $r_1, r_2$  and timestamp  $t_1, t_2$ , according to the fact  $(e_{s_1}, r_1, e_{o}, t_1)$  and  $(e_{s_2}, r_2, e_{o}, t_2)$ . The sampled anchors are  $(e_{s_1}, r_1, t_1; e_{s_2}, r_2, t_2)$ . These anchors under e2i have the answer  $e_{o}$  at least, which asserts that the answer set is not empty. Such sampling method that samples the answer first and then samples the anchors is called Inverse Sampling.
(2) Bi-directional Sampling: For long multi-hop queries, such as Pe2, the complexity of random sampling is  $O(N^{2L})$ , where  $N$  is the entity count and  $L$  is the length of the query. Pe2 has  $L = 2$ . It contains an anchor-sampling complexity of  $O(N^L)$  and a ground-truth reasoning (to get answers) complexity of  $O(N^L)$ . The origin sampling is to sample one subjective entity, get the objective as answers as next subjective anchors, again get the next objective answers. The final answers recursively depend on the initial subjective entity, leading to low performance. To accelerate, we sample an entity at the middle of AST, denoted  $e_m$ . Then we sample in two directions. One is forward,  $e_m$  as the subjective entity, to sample relation  $r_2$  and timestamp  $t_2$ . The other is backward,  $e_m$  as the objective entity, to sample subjective anchor  $e_1$ , relation  $r_1$  and timestamp  $t_1$ . The sampled anchors are  $(e_1, r_1, t_1; r_2, t_2)$ . The final answer set is asserted not empty, because it at least contains all the answers of  $\mathrm{Pe}(e_m, r_2, t_2)$ . Be aware that the two directions are independent, which can be sampled in parallel. Such sampling method that samples the answer in two directions is called Bi-directional Sampling, which can reduce the computation complexity to  $O(N^{L + \frac{L}{2}})$ , composed of a sampling complexity of  $O(N^{\frac{L}{2}})$  and a ground-truth reasoning complexity of  $O(N^L)$ .
Figure 8: The screenshot of a real running interpreter.
Embedding Reasoning Interpreter.
On the contrary of the ground-truth reasoning interpreter which has 'set' as the input and output of the CQFs, in the embedding reasoning interpreter, the input and output of CQFs are 'embedding vectors'. The embedding-based method for reasoning over TKG is called Temporal Query Embedding. The computation complexity of the embedding reasoning interpreter is  $O(L + N)$ , where  $L$  is the length of the query and  $N$  is the entity count. The complexity consists of a query-embedding complexity of  $O(L)$  and a scoring-to-answer complexity of  $O(N)$ . The embedding reasoning is much faster than the ground-truth reasoning, which has a computation complexity of  $O(N^L)$ .
To implement the embedding reasoning interpreter, we just need to replace the basic functions with neural networks. The symbols dict to pass to the interpreter is
{ "Pe":  $\mathcal{P}_e$  , "Pt":  $\mathcal{P}_t$  , "And":  $\mathcal{I}_e$  , "Or":  $\mathcal{U}_e$  , "Not":  $\mathcal{N}_e$  , "TimeAnd":  $\mathcal{I}_t$  , "TimeOr":  $\mathcal{U}_t$  , "TimeNot":  $\mathcal{N}_t$  , "TimeAfter":  $\mathcal{A}_t$  , "TimeBefore":  $\mathcal{B}_t$  }
The embedding vectors are learned as is introduced in Section 4. Unlike the ground-truth reasoning, the embedding reasoning is fuzzy. The output of the embedding reasoning interpreter is an embedding vector, which is a fuzzy set representation. To get the final answer set, we need to use the distance function (in Section 4.3) to score to candidate answers. In this way, the embedding vector is converted to the final answer set. The final answers are given in the ranking list, where each answer is followed by its distance to the query. Usually the top-k answers are accepted as the final answers.
Note that the interpreter is flexible, and easy to implement and extend. In order to reproduce static query embeddings [2-4] over dynamic knowledge graphs, the only thing to do is to implement the symbols dict and the distance function.
We hope that the design of interpreter is helpful for the future research of temporal query embedding.
Additionally, we show the screenshot of a real running interpreter in Figure 8. In the screenshot, we randomly select an example of e2i query. Then, we use the embedding reasoning interpreter to get the final answer set step by step. The final answer set is shown in the ranking list, where each answer is followed by its similarity score to the query. Finally, the bot correctly predicts the hard answer which only exists in the test set!
B.3 Details of Generated Datasets
Finally, we generate three datasets for the task of temporal complex reasoning using the process in Section B.2. The statistics of the generated datasets are listed below. Table 9 show the count of query
Table 8: Statistics on ICEWS14, ICEWS05-15, and GDELT-500.
Dataset	Entities	Relations	Timestamps	|Training|	|Validation|	|Test|	Total Edges
ICEWS14	7,128	230	365	72,826	8,941	8,963	90,730
ICEWS05-15	10,488	251	4,017	368,962	46,275	46,092	479,329
GDELT-500	500	20	366	2,735,685	341,961	341,961	3,419,607
Table 9: Queries count for each dataset.
Query Name	ICEWS14	ICEWS05-15	GDELT-500
Train	Validate	Test	Train	Validate	Test	Train	Validate	Test
Pe	66783	8837	8848	344042	45829	45644	1115102	273842	273432
Pe2	72826	3482	4037	368962	10000	10000	2215309	10000	10000
Pe3	72826	3492	4083	368962	10000	10000	2215309	10000	10000
e2i	72826	3305	3655	368962	10000	10000	2215309	10000	10000
e3i	72826	2966	3023	368962	10000	10000	2215309	10000	10000
Pt	42690	7331	7419	142771	28795	28752	687326	199780	199419
aPt	13234	4411	4411	68262	10000	10000	221530	10000	10000
bPt	13234	4411	4411	68262	10000	10000	221530	10000	10000
Pe_Pt	7282	3385	3638	36896	10000	10000	221530	10000	10000
Pt_sPe_Pt	13234	5541	6293	68262	10000	10000	221530	10000	10000
Pt_oPe_Pt	13234	5480	6242	68262	10000	10000	221530	10000	10000
t2i	72826	5112	6631	368962	10000	10000	2215309	10000	10000
t3i	72826	3094	3296	368962	10000	10000	2215309	10000	10000
e2i_N	7282	2949	2975	36896	10000	10000	221530	10000	10000
e3i_N	7282	2913	2914	36896	10000	10000	221530	10000	10000
Pe_e2i_N	7282	2968	3012	36896	10000	10000	221530	10000	10000
e2i_PeN	7282	2971	3031	36896	10000	10000	221530	10000	10000
e2i_NPe	7282	3061	3192	36896	10000	10000	221530	10000	10000
t2i_N	7282	3135	3328	36896	10000	10000	221530	10000	10000
t3i_N	7282	2924	2944	36896	10000	10000	221530	10000	10000
Pe_t2i_N	7282	3031	3127	36896	10000	10000	221530	10000	10000
t2i_PtN	7282	3300	3609	36896	10000	10000	221530	10000	10000
t2i_NPt	7282	4873	5464	36896	10000	10000	221530	10000	10000
e2u	-	2913	2913	-	10000	10000	-	10000	10000
Pe_e2u	-	2913	2913	-	10000	10000	-	10000	10000
t2u	-	2913	2913	-	10000	10000	-	10000	10000
Pe_t2u	-	2913	2913	-	10000	10000	-	10000	10000
between	7282	2913	2913	36896	10000	10000	221530	10000	10000
e2i_Pe	-	2913	2913	-	10000	10000	-	10000	10000
Pe_e2i	-	2913	2913	-	10000	10000	-	10000	10000
t2i_Pe	-	2913	2913	-	10000	10000	-	10000	10000
Pe_t2i	-	2913	2913	-	10000	10000	-	10000	10000
Pe_aPt	7282	4134	4733	68262	10000	10000	221530	10000	10000
Pe_at2i	7282	4607	5338	36896	10000	10000	221530	10000	10000
Pt_sPe	7282	4976	5608	36896	10000	10000	221530	10000	10000
Pt_se2i	7282	3226	3466	36896	10000	10000	221530	10000	10000
Pe_bPt	7282	3970	4565	36896	10000	10000	221530	10000	10000
Pe bt2i	7282	4583	5386	36896	10000	10000	221530	10000	10000
Pt_oPe	7282	3321	3621	36896	10000	10000	221530	10000	10000
Pt_oe2i	7282	3236	3485	36896	10000	10000	221530	10000	10000
structures in the split of training, validation and testing. The average answers count of each query structure are reported in Table 10. The number of queries for each dataset is shown in Table 11.
We use the Embedding Reasoning Interpreter as is introduced in Appendix B.2. We implement our model with PyTorch and use Adam [50] as a gradient optimizer. For each experiment, we use one GTX1080 graphic card. The hyperparameters for each dataset are shown in Table 12. Note that we do not perform hyperparameter tuning for each dataset. Instead, we use the same hyperparameters
Table 10: Average answers count for each dataset. All numbers are rounded to two decimal places.
Query Name	ICEWS14	ICEWS05-15	GDELT-500
Train	Validate	Test	Train	Validate	Test	Train	Validate	Test
Pe	1.09	1.01	1.01	1.07	1.01	1.01	2.07	1.21	1.21
Pe2	1.03	2.19	2.23	1.02	2.15	2.19	2.61	6.51	6.13
Pe3	1.04	2.25	2.29	1.02	2.18	2.21	5.11	10.86	10.70
e2i	1.02	2.76	2.84	1.01	2.36	2.52	1.05	2.30	2.32
e3i	1.00	1.57	1.59	1.00	1.26	1.26	1.00	1.20	1.35
Pt	1.71	1.22	1.21	2.58	1.61	1.60	3.36	1.66	1.66
aPt	177.99	176.09	175.89	2022.16	2003.85	1998.71	156.48	155.38	153.41
bPt	181.20	179.88	179.26	1929.98	1923.75	1919.83	160.38	159.29	157.42
Pe_Pt	1.58	7.90	8.62	2.84	18.11	20.63	26.56	42.54	41.33
Pt_sPe_Pt	1.79	7.26	7.47	2.49	13.51	10.86	4.92	14.13	12.80
Pt_oPe_Pt	1.75	7.27	7.48	2.55	13.01	14.34	4.62	14.47	12.90
t2i	1.19	6.29	6.38	3.07	29.45	25.61	1.97	8.98	7.76
t3i	1.01	2.88	3.14	1.08	10.03	10.22	1.06	3.79	3.52
e2i_N	1.02	2.10	2.14	1.01	2.05	2.08	2.04	4.66	4.58
e3i_N	1.00	1.00	1.00	1.00	1.00	1.00	1.02	1.19	1.37
Pe_e2i_N	1.04	2.21	2.25	1.02	2.16	2.19	3.67	8.54	8.12
e2i_PeN	1.04	2.22	2.26	1.02	2.17	2.21	3.67	8.66	8.36
e2i_NPe	1.18	3.03	3.11	1.12	2.87	2.99	4.00	8.15	7.81
t2i_N	1.15	3.31	3.44	1.21	4.06	4.20	2.91	8.78	7.56
t3i_N	1.00	1.02	1.03	1.01	1.02	1.02	1.15	3.19	3.20
Pe_t2i_N	1.08	2.59	2.70	1.08	2.47	2.62	4.10	12.02	11.37
t2i_PtN	1.41	5.22	5.47	1.70	8.10	8.11	4.56	12.56	11.32
t2i_NPt	8.14	25.96	26.23	66.99	154.01	147.34	17.58	35.60	32.22
e2u	-	3.12	3.17	-	2.38	2.40	-	5.04	5.41
Pe_e2u	-	2.38	2.44	-	1.24	1.25	-	9.39	10.78
t2u	-	4.35	4.53	-	5.57	5.92	-	9.70	10.51
Pe_t2u	-	2.72	2.83	-	1.24	1.28	-	9.90	11.27
between	122.61	120.94	120.27	1407.87	1410.39	1404.76	214.16	210.99	207.85
e2i_Pe	-	1.00	1.00	-	1.00	1.00	-	1.07	1.10
Pe_e2i	-	2.18	2.24	-	1.32	1.33	-	5.08	5.49
t2i_Pe	-	1.03	1.03	-	1.01	1.02	-	1.34	1.44
Pe_t2i	-	1.14	1.16	-	1.07	1.08	-	2.01	2.20
Pe_aPt	4.67	16.73	16.50	18.68	43.80	46.23	49.31	66.21	68.88
Pe_at2i	7.26	22.63	21.98	30.40	60.03	53.18	88.77	101.60	101.88
Pt_sPe	8.65	28.86	29.22	71.51	162.36	155.46	27.55	45.83	43.73
Pt_se2i	1.31	5.72	6.19	1.37	9.00	9.30	2.76	8.72	7.66
Pe_bPt	4.53	17.07	16.80	18.70	45.81	48.23	67.67	84.79	83.00
Pe bt2i	7.27	21.92	21.23	30.31	61.59	64.98	88.80	100.64	100.67
Pt_oPe	1.41	5.23	5.46	1.68	8.36	8.21	3.84	11.31	10.06
Pt_oe2i	1.32	6.51	7.00	1.44	10.49	10.89	2.55	8.17	7.27
Table 11: Number of queries. Pe represents query answering (s,r,?,t). Pt represents query answering (s,r,o,?) . QoE represents the query of entities (except Pe). QoT represents query of timestamps (except Pt). nlp represents a query that is not Pe or Pt.
Dataset	Training	Validation	Test
Pe	Pt	QoE	QoT	Pe	Pt	n1p	Pe	Pt	n1p
ICEWS14	273,710	27,371	59,078	8,000	66,990	66,990	10,000	66,990	66,990	10,000
ICEWS05-15	149,689	14,968	20,094	5,000	66,990	22,804	10,000	66,990	66,990	10,000
GDELT-500	107,982	10,798	16,910	4,000	66,990	17,021	10,000	66,990	66,990	10,000
Table 12: Hyperparameters on each dataset.  $d$  is the embedding dimension,  $b$  is the batch size,  $n$  is the negative sampling size,  $\gamma$  is the parameter in the loss function,  $m$  is the maximum training step, and  $l$  is the learning rate.
Dataset	d	b	n	γ	m	l
ICEWS14	800	512	128	15	300k	1 × 10-4
ICEWS05-15	800	512	128	30	300k	1 × 10-4
GDELT-500	800	512	128	30	300k	1 × 10-4
(a) The impact of embedding dimension on MRR.
(b) The impact of the margin  $\gamma$  on MRR.
Figure 9: The impact of embedding dimensionality  $d$  and the margin  $\gamma$  on MRR.
found in sensitive analysis on ICEWS14 (Section B.6) for all datasets. Besides, the source code is available at GitHub  $\dagger$ . We cite these projects  $\ddagger$ $\S$ , and thank them for their great contributions!
B.5 Evaluation
Given the standard split of edges into training  $(\mathcal{F}_{\mathrm{train}})$ , validation  $(\mathcal{F}_{\mathrm{valid}})$  and test  $(\mathcal{F}_{\mathrm{test}})$  sets, we append inverse relations and double the number of edges in the graph. Then we create three graphs:  $\mathcal{G}_{\mathrm{train}} = \{\mathcal{V}, \mathcal{R}, \mathcal{T}, \mathcal{F}_{\mathrm{train}}\}$ ,  $\mathcal{G}_{\mathrm{valid}} = \{\mathcal{V}, \mathcal{R}, \mathcal{T}, \mathcal{F}_{\mathrm{train}} + \mathcal{F}_{\mathrm{valid}}\}$ ,  $\mathcal{G}_{\mathrm{test}} = \{\mathcal{V}, \mathcal{R}, \mathcal{T}, \mathcal{F}_{\mathrm{train}} + \mathcal{F}_{\mathrm{valid}} + \mathcal{F}_{\mathrm{test}}\}$ . Given a query  $q$ , let  $\llbracket q\rrbracket_{\mathrm{train}}$ ,  $\llbracket q\rrbracket_{\mathrm{valid}}$ , and  $\llbracket q\rrbracket_{\mathrm{test}}$  denote a set of answers (entities or timestamps) obtained by subgraph matching of  $q$  on  $\mathcal{G}_{\mathrm{train}}$ ,  $\mathcal{G}_{\mathrm{valid}}$  and  $\mathcal{G}_{\mathrm{test}}$ . For a test query  $q$  and its non-trivial answer set  $v \in \llbracket q\rrbracket_{\mathrm{test}} - \llbracket q\rrbracket_{\mathrm{valid}}$ , we denote the rank of each answer  $v_i$  as  $\text{rank}(v_i)$ . The mean reciprocal rank (MRR) is  $\text{MRR}(q) = \frac{1}{||v||} \sum_{v_i \in v} \frac{1}{\text{rank}(v_i)}$  and Hits at K (Hits@K) is  $\text{Hits@K}(q) = \frac{1}{||v||} \sum_{v_i \in v} f(\text{rank}(v_i))$ , where  $f(n) = \left\{ \begin{array}{ll} 1, & n \leq K \\ 0, & n > K \end{array} \right.$ .
B.6 Experiment Data for Sensitive Analysis
We conduct experiments on ICEWS14 to analyze the impact of the embedding dimensionality  $d$  and the margin  $\gamma$  on the performance of TFLEX.
Impacts of Embedding Dimensionality Our experiments indicate that the selection of embedding dimension has a substantial influence on the effectiveness of TFLEX. We train TFLEX with embedding dimension  $d \in \{300, 400, 500, 600, 700, 800, 900, 1000\}$  and plot results based on the validation set, as shown in Figure 9a. With the increase of  $d$ , the model performance (indicated by MRR) increases rapidly and reaches its top at  $d = 800$ . Therefore, we assign 800 as the best setting.
$\dagger$  TFLEX: https://github.com/LinXueyuanStdio/TFLEX
$\ddagger$ GQE, Query2box, BetaE: https://github.com/snap-stanford/KGReasoning
$ ConE: https://github.com/MIRALab-USTC/QE-ConE
Table 13: The mean values and standard variances of TFLEX's MRR results on ICEWS14.
Pe	Pe2	Pe3	e2i	e3i
48.21	37.27	33.53	69.24	95.70
±0.37	±0.12	±0.42	±0.46	±0.19
Pt	aPt	bPt	Pe_Pt	Pt_sPe_Pt	Pt_oPe_Pt	t2i	t3i
20.87	3.00	2.96	14.27	9.57	9.46	27.49	52.84
±0.43	±0.34	±0.50	±0.57	±0.44	±0.66	±0.51	±0.63
e2i_N	e3i_N	Pe_e2i_N	e2i_PeN	e2i_NPe
45.55	99.56	34.74	35.63	38.61
±0.12	±0.50	±0.44	±0.44	±0.20
t2i_N	t3i_N	Pe_t2i_N	t2i_PtN	t2i_NPt
25.38	98.91	34.05	11.42	12.07
±0.60	±0.13	±0.49	±0.57	±0.53
e2u	Pe_e2u
29.20	42.28
±0.12	±0.43
t2u	Pe_t2u
30.73	21.74
±0.54	±0.35
e2i_PE	Pe_e2i	Pe_aPt	Pe_bPt	Pe_at2i	Pe_bt2i
98.86	36.77	8.66	9.74	7.90	7.78
±0.34	±0.42	±0.61	±0.33	±0.39	±0.39
t2i_PE	Pe_t2i	Pt_sPe	Pt_oPe	Pt_se2i	Pt_oe2i	between
96.62	64.50	4.32	10.58	8.20	7.95	2.57
±0.13	±0.12	±0.46	±0.20	±0.34	±0.34	±0.14
Impacts of Parameter  $\gamma$  We train TFLEX with parameter  $\gamma \in \{5,10,15,20,25,30,35,40\}$  and plot MRR results in Figure 9b. Too small and too large  $\gamma$  both get bad results, while  $\gamma = 15$  in the middle is the best. Therefore, we choose  $\gamma = 15$ .
Error Bars of Main Results In order to evaluate how stable the performance of TFLEX is, we run five times with random seeds  $\{1, 10, 100, 1000, 10000\}$  and report the error bars of these results. Table 13 shows the error bar of TFLEX's MRR results on ICEWS14. Overall, the standard variances are small, which demonstrates that the performance of TFLEX is stable.
B.7 Detail Metrics of Main Results
We report the MRR results on all query structures for each dataset in Table 14 (ICEWS14), Table 15 (ICEWS05-15) and Table 16 (GDELT-500). Query2box [2] can only answer queries without negation. BetaE [3] and ConE [4] can answer queries with negation, except temporal logic. ConE(temporal) and X(ConE) are the variants of ConE, aiming at exploring the right way to promote the static query embeddings to temporal ones in Section C. FLEX, X(ConE), X-1F, X-logic are the variants of TFLEX, introduced in the main body of the paper.
C Exploration of Static Query Embeddings Toward Dynamic
In this section, we present more research questions that aim to explore the right way to promote the static query embeddings to temporal ones. Since ConE [4] is a strong baseline for complex logical reasoning over static knowledge graphs, we conduct exploration experiments by introducing variants of ConE. We present the overall results for each dataset in Table 17, detailed results in Table 14 (ICEWS14), Table 15 (ICEWS05-15) and Table 16 (GDELT-500). Overall, we are interested in the following research questions:
Table 14: MRR results on ICEWS14. The group  $\mathrm{avg}_x$  wraps to two rows. AVG denotes average performance under all query types.
Model	avg_e	Pe	Pe2	Pe3	e2i	e3i
Query2box	25.06	25.81	17.29	12.97	16.00	53.25
BetaE	37.19	39.52	23.77	17.50	23.77	81.38
ConE	41.94	42.55	30.90	24.78	26.52	84.93
ConE(temporal)	42.23	41.21	31.83	24.99	28.98	84.17
FLEX	43.67	45.25	33.07	27.22	27.62	85.18
TFLEX	56.79	48.21	37.27	33.53	69.24	95.70
X(ConE)	40.93	40.58	28.84	24.31	30.93	79.98
X-1F	56.89	48.61	37.51	32.61	71.25	94.47
X-logic	56.64	48.00	37.42	31.23	73.11	93.45
avg_t	Pt	aPt	bPt	Pe_Pt	Pt_sPe_Pt	Pt_oPe_Pt	t2i	t3i
TFLEX	17.56	20.87	3.00	2.96	14.27	9.57	9.46	27.49	52.84
X(ConE)	16.41	19.45	3.06	3.01	13.96	8.79	8.67	25.01	49.36
X-1F	18.77	21.94	3.04	2.99	16.69	10.73	10.44	29.66	54.71
X-logic	18.03	21.33	3.10	3.05	15.76	9.85	9.94	28.57	52.63
avg_c_e	e2i_N	e3i_N	Pe_e2i_N	e2i_PeN	e2i_NPe
BetaE	36.69	30.44	87.93	19.94	22.80	22.36
ConE	44.88	40.98	99.12	29.22	29.84	25.22
ConE(temporal)	44.94	41.45	98.99	29.48	30.71	24.09
FLEX	44.41	38.30	99.22	31.18	30.49	22.84
TFLEX	50.82	45.55	99.56	34.74	35.63	38.61
X(ConE)	42.15	39.10	96.25	26.92	27.59	20.89
X-1F	49.78	42.31	99.31	34.89	34.77	37.61
X-logic	51.17	44.65	99.25	35.30	36.31	40.33
avg_t_c_t	t2i_N	t3i_N	Pe_t2i_N	t2i_PtN	t2i_NPt
TFLEX	36.37	25.38	98.91	34.05	11.42	12.07
X(ConE)	35.24	26.90	98.82	30.30	9.44	10.75
X-1F	37.73	26.40	98.82	37.85	12.38	13.23
X-logic	36.39	25.68	98.69	33.30	11.34	12.93
avg_{\{U_e\}}	e2u	Pe_e2u
BetaE	19.95	18.61	21.28
ConE	26.47	21.84	31.11
ConE(temporal)	27.63	23.01	32.26
FLEX	29.25	24.05	34.46
TFLEX	35.74	29.20	42.28
X(ConE)	25.46	20.26	30.67
X-1F	34.48	30.04	38.93
X-logic	34.68	29.44	39.92
avg_{\{U_t\}}	t2u	Pe_t2u
TFLEX	26.24	30.73	21.74
X(ConE)	24.07	27.63	20.51
X-1F	28.04	33.91	22.16
X-logic	26.36	31.21	21.52
avg_x	e2i_Pe	Pe_e2i	Pe_aPt	Pe_bPt	Pe_at2i	Pebt2i
TFLEX	28.03	98.86	36.77	8.66	9.74	7.90	7.78
X(ConE)	26.65	87.31	28.72	11.01	10.40	11.21	11.27
X-1F	29.31	98.87	36.00	10.04	10.21	8.30	8.06
X-logic	28.61	97.64	36.96	10.09	10.27	8.76	8.93
AVG	t2i_Pe	Pe_t2i	Pt_sPc	Pt_oPc	Pt_se2i	Pt_oe2i	between
TFLEX	35.93	96.62	64.50	4.32	10.58	8.20	7.95	2.57
X(ConE)	30.13	95.40	63.04	3.77	8.82	6.44	5.76	3.32
X-1F	36.43	97.24	70.32	4.88	10.73	12.24	11.55	2.55
X-logic	35.98	96.35	64.31	4.97	10.35	10.59	10.35	2.34
Table 15: MRR results on ICEWS05-15. The group  $\mathrm{avg}_x$  wraps to two rows. AVG denotes average performance under all query types.
Model	avgc	Pe	Pe2	Pe3	e2i	e3i
Query2box	24.00	25.94	16.62	14.22	17.68	45.52
BetaE	31.33	35.78	21.47	18.18	18.10	63.11
ConE	40.93	42.67	29.39	24.79	26.15	81.64
ConE(temporal)	40.74	42.64	29.30	24.76	25.35	81.65
FLEX	38.96	41.60	29.80	24.58	24.37	74.46
TFLEX	48.99	43.04	36.28	33.89	41.17	90.60
X(ConE)	36.29	39.90	26.62	22.85	22.52	69.56
X-1F	49.90	43.07	36.87	34.96	40.85	93.76
X-logic	44.80	40.57	31.47	30.67	35.80	85.47
avgt	Pt	aPt	bPt	PePt	Pt_sPe_Pt	Pt_oPe_Pt	t2i	t3i
TFLEX	4.39	10.62	0.38	0.38	7.29	2.63	1.83	3.85	8.15
X(ConE)	4.41	10.98	0.38	0.38	6.75	2.16	1.63	4.22	8.81
X-1F	4.43	10.71	0.38	0.38	7.40	2.73	1.86	3.80	8.16
X-logic	3.29	8.57	0.38	0.39	6.37	1.69	1.41	2.49	5.06
avgce	e2i_N	e3i_N	Pe_e2i_N	e2i_PeN	e2i_NPe
BetaE	29.70	21.68	71.08	20.31	21.11	14.30
ConE	43.52	43.04	96.08	28.41	28.75	21.30
ConE(temporal)	43.34	42.85	96.08	28.33	28.62	20.83
FLEX	42.10	36.22	97.47	27.93	27.23	21.64
TFLEX	46.17	41.34	96.69	34.29	34.63	23.88
X(ConE)	38.12	33.70	88.83	24.80	25.60	17.64
X-1F	46.11	38.06	96.82	35.76	36.20	23.73
X-logic	41.92	36.57	91.16	30.37	31.42	20.07
avgct,ct	t2i_N	t3i_N	Pe_t2i_N	t2i_PtN	t2i_NPt
TFLEX	30.16	16.09	98.40	31.39	3.35	1.58
X(ConE)	29.49	16.55	98.46	28.23	2.53	1.69
X-1F	30.26	16.05	98.37	32.07	3.27	1.56
X-logic	28.34	12.94	96.03	29.47	2.18	1.07
avg{uc}	e2u	Pe_e2u
BetaE	21.54	20.95	22.13
ConE	43.02	37.21	48.83
ConE(temporal)	43.14	37.05	49.24
FLEX	44.38	35.72	53.04
TFLEX	54.37	52.99	55.75
X(ConE)	36.37	29.89	42.86
X-1F	54.05	52.47	55.64
X-logic	45.36	43.84	46.88
avg{ut}	t2u	Pe_t2u
TFLEX	28.69	44.99	12.39
X(ConE)	26.40	40.35	12.46
X-1F	27.70	43.38	12.02
X-logic	23.39	37.07	9.71
avgx	e2i_Pe	Pe_c2i	Pe_aPt	Pe_bPt	Pe_at2i	Pe_ct2i
TFLEX	24.26	94.23	57.16	5.07	4.56	4.38	3.93
X(ConE)	21.69	81.36	38.20	5.63	5.03	5.80	5.36
X-1F	24.41	94.26	55.97	5.14	4.79	4.37	3.95
X-logic	21.95	87.48	50.11	4.28	3.95	3.96	3.55
AVG	t2i_Pe	Pe_t2i	Pt_sPt	Pt_oPt	Pt_se2i	Pt_oe2i	between
TFLEX	33.72	92.25	48.35	0.46	2.82	0.98	1.02	0.22
X(ConE)	27.54	91.92	43.83	0.58	2.18	0.95	1.02	0.14
X-1F	33.98	92.21	50.77	0.46	2.89	1.13	1.23	0.17
X-logic	29.86	86.17	40.78	0.62	1.98	1.10	1.21	0.16
Table 16: MRR results on GDELT-500. The group  $\mathrm{avg}_x$  wraps to two rows. AVG denotes average performance under all query types.
Model	avg_e	Pe	Pe2	Pe3	e2i	e3i
Query2box	9.67	10.70	4.73	3.41	12.28	17.22
BetaE	14.75	14.02	5.76	4.71	18.61	30.67
ConE	18.44	16.65	6.18	4.70	23.11	41.55
ConE(temporal)	18.51	17.61	5.62	4.57	24.52	40.22
FLEX	19.07	17.72	5.78	4.67	24.30	42.85
TFLEX	19.60	18.50	5.76	4.68	25.94	43.14
X(ConE)	17.83	17.05	5.76	4.66	24.02	37.66
X-1F	17.92	16.18	6.35	4.84	22.41	39.80
X-logic	17.36	15.80	6.22	4.85	22.45	37.47
avg_t	Pt	aPt	bPt	Pe_Pt	Pt_sPe_Pt	Pt_oPe_Pt	t2i	t3i
TFLEX	5.38	6.49	3.25	3.27	6.16	3.06	2.88	6.78	11.17
X(ConE)	3.16	2.15	3.39	3.33	6.96	2.29	2.31	2.34	2.49
X-1F	5.49	6.62	3.30	3.30	5.84	3.07	3.05	7.08	11.66
X-logic	5.75	7.15	3.34	3.31	5.95	3.14	3.06	7.52	12.54
avg_e,Ce	e2i_N	e3i_N	Pe_e2i_N	e2i_PeN	e2i_NPe
BetaE	11.15	11.22	24.17	5.53	5.46	9.39
ConE	12.67	12.69	29.09	5.47	5.42	10.66
ConE(temporal)	12.67	13.42	27.98	5.31	5.00	11.63
FLEX	13.35	13.57	30.25	5.47	5.47	11.99
TFLEX	13.52	14.46	30.09	5.51	5.19	12.36
X(ConE)	12.34	13.42	26.28	5.54	5.35	11.14
X-1F	12.13	11.77	27.63	5.70	5.40	10.14
X-logic	12.11	12.32	27.00	5.70	5.55	10.01
avg_t,C_t	t2i_N	t3i_N	Pe_t2i_N	t2i_PtN	t2i_NPt
TFLEX	6.31	5.92	9.01	9.30	2.76	4.56
X(ConE)	3.93	2.24	2.43	10.00	2.26	2.74
X-1F	6.50	5.90	9.70	9.15	2.85	4.90
X-logic	6.86	6.51	10.94	8.78	2.94	5.14
avg{u_c}	e2u	Pe_e2u
BetaE	6.20	4.64	7.75
ConE	6.96	4.68	9.25
ConE(temporal)	7.30	4.50	10.09
FLEX	7.44	4.47	10.42
TFLEX	7.58	4.62	10.55
X(ConE)	7.41	4.64	10.19
X-1F	6.92	4.80	9.04
X-logic	6.91	4.74	9.09
avg{u_t}	t2u	Pe_t2u
TFLEX	6.71	9.03	4.38
X(ConE)	6.35	10.39	2.31
X-1F	6.59	8.60	4.58
X-logic	6.80	8.85	4.76
avg_x	e2i_Pe	Pe_e2i	Pe_aPt	Pe_bPt	Pe_at2i	Pe bt2i
TFLEX	6.17	17.53	6.38	4.73	4.58	4.47	4.58
X(ConE)	6.17	17.53	6.44	6.45	6.18	6.02	5.97
X-1F	6.47	18.13	6.82	5.48	5.35	5.19	5.31
X-logic	6.64	17.45	6.79	5.56	5.44	5.35	5.37
AVG	t2i_Pe	Pe_t2i	Pt_sPt	Pt_oPt	Pt_se2i	Pt_oe2i	between
TFLEX	9.32	8.36	15.83	3.11	2.81	2.93	3.07	1.85
X(ConE)	8.17	2.45	17.97	2.70	2.24	2.20	2.29	1.76
X-1F	8.86	8.80	15.13	2.92	2.72	3.11	3.29	1.87
X-logic	8.92	10.28	15.28	3.27	3.11	3.23	3.36	1.79
RQ1: How about incorporating temporal information into the static query embedding, without considering time logic?
RQ2: Further, how about enhancing the static query embedding with temporal logic?