SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Discussion Figure 3(a) shows NE's decoding distribution  $g_{m} = \min_{\tau \in \mathbb{T}_{r}}g(\tau)$  by each query's ground truth  $gc_{g} = \max_{\tau \in \mathbb{T}_{r}}gc(\tau)$ . It can be observed that the decoded conserved local energy accurately reveals the TR validity, which enables NE to successfully distinguish between valid and invalid TRs, as demonstrated in the case shown in Figure 3(b). Table 10 shows that baselines with  $g^{\prime}(\tau)$  still perform poorly. This is mainly because their  $f(t)$ s do not fill well. Specifically, Figure 3(c) illustrates that TNTComplEx has much noise in its  $f^{\prime}(t)$  compared to NE. The reason is that baseline models are generally designed to achieve good performance in the completion task and, therefore, over-apply the generalization capabilities of distributed representations, which hinders the fit of event occurrences.
4.3 NE's Superior Learning Capabilities
(a) TR detection by  $n$
Figure 4: Grouped performances of NE
(b) TR query by  $n$
(c) TR query by  $\tau_{g}$
Data Efficiency In Figure 4(a) and 4(b), we group TRs by their number  $n$  of relevant events. It is shown that NE accurately detects valid TRs and reports correct  $\tau s$  with only two event pairs as positive samples. This performance is comparable to humans, able to form temporally associative memories with minimal experience (Hudson et al., 1992; Bauer & Mandler, 1989; Schapiro et al., 2017). Note that the maximum group in the test has  $n > 400$ , while we only show groups with  $n \leq 40$  for display considerations.
Time Efficiency As explained in 3.3, NE's specific decoding function  $g(\tau)$  enables NE to retrieve TRs in a constant time complexity by vector computations. Calculating  $g'(\tau)$  of classic embeddings, however, requires an additional time complexity relevant to  $T_{a}$  (the number of absolute time points).
Storage Efficiency In addition to the data-efficient and time-efficient properties, NE is, in fact, also a storage-efficient memory for TRs and event occurrences. Here is a detailed analysis:
The storage of NE vectors, denoted as  $S(NE)$ , can be calculated as follows:  $S(NE) = S(ev - vector) + S(time - vector) = 2 * N * d * 64bit + 2 * d * 64bit$ . In our experiments, we used torch.LongTensor and N represents the number of event-type vectors. On the other hand, the storage of exact counting, denoted as  $S(CT)$ , can be calculated as follows:  $S(CT) = S(TR) + S(event) = N^2 * T_a * log_2(n/N)bit + N * (n/N) * log_2(T_a)bit$ . Here,  $n$  represents the number of all event occurrences. We reserved the storage accuracy of TR validity to effectively distinguish different values, resulting in approximately  $\log_2(n/N)$  bit for each TR validity  $(ev_b, ev_h, \tau)$ .
For the ICEWS14 and ICEWS18 datasets, where  $d = 400$ ,  $T_{a} = 365$ , and  $n = 90730$ ,  $468558$ ,  $N = 50295$ ,  $266631$ , we calculated the compression ratio  $\frac{S(CT)}{S(NE)}$  of NE as 421 and 2336, respectively. This remarkable capability of NE can be attributed to the fact that it separately stores the information of TR validities  $(ev_{b}, ev_{h}, \tau)$  using event-type vectors and a global time vector. By representing the common information of related TRs efficiently in memory, NE achieves a compression ratio that is approximately linear to the number of event types.
Flexibility In Figure 4(c), we group valid TRs by their golden  $\tau_{g}$ s. NE is shown to be flexible for learning TRs with  $\tau$ s varying broadly, comparable to humans with stable memory codes for various time intervals in the hippocampus (Mankin et al., 2012).
4.4 NE's Wide Potential Use
(a) Event occurrences
(b) Case 2
Figure 5: Social event prediction
(b) TR validity
Figure 6: Personal decision making
Potential Use in Social Event Prediction In D18, NE successfully reports 21010 valid TRs with an F1 score of 0.83. The encoding stage takes around 1 hour, while decoding only takes less than 1 minute. Cases are presented below and in Figure 5, with additional cases available in Appendix B.4.
(1) Case 1. Citizen (India) will Reject to Narendra Modi (events in day 23, 122, and 168) in around 87 days whenever Narendra Modi Appeal for diplomatic cooperation (such as policy support) to Citizen (India) (events in day 102, 200, and 264).
(2) Case 2. Russia will Meet at a 'third' location with Ukraine (events in day 31 and 138) in around 136 days whenever Ukraine Use conventional military force to Russia (events in day 161 and 288).
Since the TRs mined can generalize across time, the results above imply NE's potential use in both reliable and interpretable event predictions urgently needed in the big data era (Zhao, 2021).
Potential Use in Personal Decision Making Consider an intelligent machine that has visited a restaurant four times, with the occurrence time of each event episode used as input for NE, as shown in Figure 6(a). After training all events, the decoded TR validity  $\min_{\tau \in \mathbb{T}_r} g(\tau)$  is transformed linearly and demonstrated in Figure 6(b). Despite the recurrent TRs on the slash that can be set aside, valid TRs such as 'order dishes  $-(10\mathrm{minutes})\rightarrow$  have meals' are well distinguished from invalid ones such as 'order dishes  $-(10\mathrm{minutes})\rightarrow$  look at the floor'.
Combining NE with front-end methods that take unstructured videos as input and output event items in the form of  $(ev, t)$ , and with back-end methods that use the decoded valid TRs to guide decision-making, NE has the potential to aid intelligent machines in surviving in changing environments by generalizing from little experience, just as human beings (Goyal & Bengio, 2022; Xue et al., 2018).
Potential Use in Memory-constrained Scenarios As discussed in 4.3, NE approximately reduces the required storage space from  $M^2$  to  $M$  in our experimental settings, where  $M$  is the number of event types. Therefore, NE holds significant potential for applications in memory-constrained scenarios like the edge. This is important when  $M$  is large, which is usual in the big-data era.
Here we demonstrate ablation studies of loss constants  $C_p, C_n$  and time vector  $\omega$ , while those for dimension  $d$  and event type vector  $\mathbf{u}$  are shown in Appendix B.3.
Table 2: NE on ICEWS14 in different  $C_p$  and  $C_n$  settings
\( C_n \) \( C_p \)	0.4	TR Detection (F1)	-0.2	-0.4	TR Query (r)
0.2	0	0.82	0.81	0.80	0.86	0.87	0.87	0.87	0.87
1	0.78	0.80	0.79	0.80	0.80	0.85	0.86	0.86	0.87	0.87	0.87
0.8	0.64	0.67	0.79	0.80	0.80	0.85	0.86	0.86	0.87	0.86	0.86
0.6	0.29	0.40	0.68	0.79	0.79	0.44	0.82	0.86	0.86	0.86	0.85
0.4	0.18	0.31	0.49	0.76	0.79	0.12	0.35	0.84	0.84	0.84	0.81
0.2	0.53	0.19	0.27	0.71	0.78	0.27	0.05	0.28	0.79	0.71	0.71
Loss Constants Table 2 shows that NE performs optimally when  $C_p = 1$  and  $C_n = 0$ . In fact, as  $C_p$  approaches 1, the  $g(\tau)$  of perfect TRs ( $gc(\tau) = 1, \eta = 1$ ) is enforced to converge to its minimum 0. This global constant for all potential TRs in the embedding space allows  $g(\tau)$  to reveal TR validity better. In terms of  $C_n$ , setting it to 0 results in negative samples occupying the largest embedding space. Since negative samples comprise most of all trained event samples, this setting improves the fit of negative samples and optimizes NE's performance.
Table 3: NE on GDELT with different  ${\omega }_{max}$  s
ωmax	1	5	10	50	100	200	400	600	800
TR Query (r)	0.15	0.93	0.92	0.85	0.85	0.85	0.85	0.85	0.85
TR Detection (F1)	0.22	0.45	0.55	0.56	0.54	0.53	0.53	0.51	0.53
Maximal Frequency Coefficient Table 3 shows that NE performs optimally with different values of  $\omega_{max}$ , respectively, in the TR detection and query task.
Table 4: NE on the three datasets with increasing events, and with different distributions of  $\left\{  {\omega }_{k}\right\}$
{ωk}	D14 (90730 events)	D18 (468558 events)	GDELT (2278405 events)
linear	exponential	linear	exponential	linear	exponential
TR Query (r)	0.81	0.82	0.75	0.82	0.24	0.85
Frequency Distribution Table 4 shows that the larger the dataset, the more exponential distribution  $(\omega_{k} = \frac{(2\pi \times \omega_{max})^{\frac{k}{d}} - 1}{T_{a}}, k = 0,1,\dots,d - 1)$  surpasses linear distribution  $(\omega_{k} = \frac{2\pi \times k \times \omega_{max}}{d \times T_{a}}, k = 0,1,\dots,d - 1)$  with the same parameters of  $d = 400$ ,  $\omega_{max} = 600$ . This suggests that real-world event occurrences depend more on low-frequency terms than high-frequency ones.
Event Schema Induction In the natural language processing (NLP) field, a significant research focus is on inducing event schemas from text (Huang et al., 2016; Li et al., 2021), including from language models (Dror et al., 2022), to support downstream applications such as search, question-answering, and recommendation (Guan et al., 2022). These NLP methods aim to organize known event regularities already given as priors for the extracting algorithm (such as extracting 'earthquake -> tsunami' from the sentence 'An earthquake causes a tsunami.') and focus on the schemas for use. In contrast, our tasks are designed to learn event regularities directly from experience without supervision. Specifically, the only prior models know is whether an event occurs, and models are required to detect valid TRs from all potential ones and report the correct relative time of valid TRs.
Temporal Rule Mining Various temporal rules are mined from event sets to reveal regularities in industry, security, healthcare, etc (Segura-Delgado et al., 2020; Chen et al., 2007; Yoo & Shekhar, 2008; Namaki et al., 2017). Although the search methods used discover event regularities directly from events without supervision, both the mined rules and source events are generally stored as symbolic representations in list form. In contrast, by applying event embeddings, NE is a distributed and approximate memory for both TRs and event items. NE strikes a balance between storage efficiency and storage accuracy compared to exact counting, as detailedly discussed in 4.3.
**Embedding Models of Structured Data** Within all embedding models of static and temporal knowledge graphs (Chen et al., 2020; Cai et al., 2022; Wang et al., 2020; Messner et al., 2022), three are most related to NE. RotatE (Sun et al., 2019) represents each entity and relationship as a complex vector to model relation patterns on knowledge graphs, and TeRo (Xu et al., 2020b) represents time as a rotation of entities to model time relation patterns on temporal knowledge graphs. While both RotatE and TeRo introduce complex vectors for better completion performance, NE first explores using complex vectors for TR detection in events. In particular, the specific use of complex vectors in RotatE encodes inverse relations and in TeRo encodes asymmetric and reflexive relations. NE, instead, apply rotating complex unit vectors to encode time-translation symmetries of all potential TRs. IterE (Zhang et al., 2019) construct a decodable embedding model to discover rules for better knowledge graph completion performance. While we take functional inspiration from IterE that embedding models can jointly encode data and decode regularities, we focus on event data and define the new problems of TR detection and TR query. Specifically, while IterE focuses on discrete variables, NE focuses on the continuous variable of time that involves Fourier-like transformations.
To summarize, TR detection and TR query focus on achieving human-like schema learning capabilities rather than pursuing better support for NLP applications like the event schema induction task. Meanwhile, NE leverages the advantages of distributed representations over symbolic ones of search methods in temporal rule mining and is distinct from existing embedding models of structured data.
We have developed NE which for the first time enables data-efficient TR formation and time-efficient TR retrieval simply through embedding event samples. We have formally defined the tasks of TR detection and TR query to comprehensively evaluate the TR learning capabilities of embedding models. We have demonstrated NE's potential use in social event prediction, personal decision-making, and memory-constrained scenarios. We hope that we have facilitated the development of human-like event intelligence.
One limitation of NE is that when the vector dimension  $d$  is set much lower than the number of absolute time points  $T_{a}$ , significant performance degradation of NE will occur as observed in the GDELT experiment. Future research is needed to improve this weakness. The privacy issues potentially brought about by TR detection and the causality of TRs should also be handled properly.
We sincerely thank Rong Zhao, Hao Zheng, Dahu Feng, Lukai Li, Hui Zeng, Wenhao Zhou, Yu Du, Songchen Ma, and Faqiang Liu for their valuable comments and encouragements. This work was supported by the National Nature Science Foundation of China (nos. 62088102, 61836004) and the National Key Research and Development Program of China (no. 2021ZD0200300).
Bauer, P. J. and Mandler, J. M. One thing follows another: Effects of temporal structure on 1-to 2-year-olds' recall of events. Developmental psychology, 25(2):197, 1989.
Boschee, E., Lautenschlager, J., O'Brien, S., Shellman, S., Starz, J., and Ward, M. Icesw coded event data. Harvard Dataverse, 12, 2015.
Bright, I. M., Meister, M. L., Cruzado, N. A., Tiganj, Z., Buffalo, E. A., and Howard, M. W. A temporal record of the past with a spectrum of time constants in the monkey entorhinal cortex. Proceedings of the National Academy of Sciences, 117(33):20274-20283, 2020.
Cai, B., Xiang, Y., Gao, L., Zhang, H., Li, Y., and Li, J. Temporal knowledge graph completion: A survey. arXiv preprint arXiv:2201.08236, 2022.
Chaudhuri, R. and Fiete, I. Computational principles of memory. Nature neuroscience, 19(3): 394-403, 2016.
Chen, M., Chen, S.-C., and Shyu, M.-L. Hierarchical temporal association mining for video event detection in video databases. In 2007 IEEE 23rd International Conference on Data Engineering Workshop, pp. 137-145. IEEE, 2007.
Chen, X., Jia, S., and Xiang, Y. A review: Knowledge reasoning over knowledge graph. Expert Systems with Applications, 141:112948, 2020.
Dror, R., Wang, H., and Roth, D. Zero-shot on-the-fly event schema induction. arXiv preprint arXiv:2210.06254, 2022.
Galárraga, L., Teflioudi, C., Hose, K., and Suchanek, F. M. Fast rule mining in ontological knowledge bases with amie +. The VLDB Journal, 24(6):707-730, 2015.
Ghosh, V. E. and Gilboa, A. What is a memory schema? a historical perspective on current neuroscience literature. Neuropsychologia, 53:104-114, 2014.
Goel, R., Kazemi, S. M., Brubaker, M., and Poupart, P. Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3988-3995, 2020.
Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A, 478(2266):20210068, 2022.
Guan, S., Cheng, X., Bai, L., Zhang, F., Li, Z., Zeng, Y., Jin, X., and Guo, J. What is event knowledge graph: A survey. IEEE Transactions on Knowledge and Data Engineering, 2022.
Han, Z., Chen, P., Ma, Y., and Tresp, V. xerte: Explainable reasoning on temporal knowledge graphs for forecasting future links. arXiv preprint arXiv:2012.15537, 2020.
Howard, M. W., MacDonald, C. J., Tiganj, Z., Shankar, K. H., Du, Q., Hasselmo, M. E., and Eichenbaum, H. A unified mathematical framework for coding time, space, and sequences in the hippocampal region. Journal of Neuroscience, 34(13):4692-4707, 2014.
Huang, L., Cassidy, T., Feng, X., Ji, H., Voss, C., Han, J., and Sil, A. Liberal event extraction and event schema induction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 258-268, 2016.
Hudson, J. A., Fivush, R., and Kuebli, J. Scripts and episodes: The development of event memory. Applied Cognitive Psychology, 6(6):483-505, 1992.
Wang, X., Lyu, S., Wang, X., Wu, X., and Chen, H. Temporal knowledge graph embedding via sparse transfer matrix. Information Sciences, 623:56-69, 2023.
Xiao, C., Sun, L., and Ji, W. Temporal knowledge graph incremental construction model for recommendation. In Asia-Pacific web (apweb) and web-age information management (waim) joint international conference on web and big data, pp. 352-359. Springer, 2020.
Xu, C., Nayyeri, M., Alkhoury, F., Yazdi, H., and Lehmann, J. Temporal knowledge graph completion based on time series gaussian embedding. In International Semantic Web Conference, pp. 654-671. Springer, 2020a.
Xu, C., Nayyeri, M., Alkhoury, F., Yazdi, H. S., and Lehmann, J. Tero: A time-aware knowledge graph embedding via temporal rotation. arXiv preprint arXiv:2010.01029, 2020b.
Xue, J.-R., Fang, J.-W., and Zhang, P. A survey of scene understanding by event reasoning in autonomous driving. Machine Intelligence Research, 15(3):249-266, 2018.
Yang, Y., Wei, Z., Chen, Q., and Wu, L. Using external knowledge for financial event prediction based on graph neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2161-2164, 2019.
Yoo, J. S. and Shekhar, S. Similarity-profiled temporal association mining. IEEE Transactions on Knowledge and Data Engineering, 21(8):1147-1161, 2008.
Zhang, W., Paudel, B., Wang, L., Chen, J., Zhu, H., Zhang, W., Bernstein, A., and Chen, H. Iteratively learning embeddings and rules for knowledge graph reasoning. In The World Wide Web Conference, pp. 2366-2377, 2019.
Zhao, L. Event prediction in the big data era: A systematic survey. ACM Computing Surveys (CSUR), 54(5):1-37, 2021.
A Theoretical Illustrations
A.1 Interdisciplinary Correspondences
A.1.1 Physical Correspondence
Considering each event type as a particle, the event embedding  $\mathbf{q}(t;ev)$  can be viewed as its generalized coordinate at time  $t$  in the d-dimensional complex embedding space. Suppose that each two particles  $ev_{b}, ev_{h}$  are connected by a spring with the force linear to their distance, their potential energy can then be expressed as  $||\pmb{q}_b(t) - \pmb{q}_h(t + \tau)||^2$ , where  $\tau$  is a parameter. As  $\pmb{q}(t;ev)$  changes with time, its kinetic energy also changes and can be viewed as being driven by external nonconservative forces. Using the extended Noether's theorem, we can see that the work of external forces and the kinetic energy offset, leading to a conserved quantity of time-translation symmetry that only includes the potential energy instead of the total energy. The inductive bias of Noether Embedding exactly enforces that such local potential energies are innately conserved, which means that  $||\pmb{q}_m(t_k) - \pmb{q}_n(t_k + \tau_r)||^2 = ||\pmb{q}_m(t_s) - \pmb{q}_n(t_s + \tau_r)||^2, \forall ev_m, ev_n \in \mathbb{P}, t_k, t_s \in \mathbb{T}_a, \tau_r \in \mathbb{T}_r$ , where  $\mathbb{P}, \mathbb{T}_a, \mathbb{T}_r$  refer to the set of event types, absolute time points, and relative time points, respectively.
A.1.2 Biological Correspondence
The formation of TRs relies on two central functions, namely measurement and integration. The measurement function involves comparing the temporal distance between each two events. It is claimed that Laplace transformation exists in the hippocampus for representing time (Howard et al., 2014). Specifically, the population of temporal context cells (or referred to as ramping cells (Tsao et al., 2018)) in the lateral entorhinal cortex is discovered to code time with a variety of rate constants (Bright et al., 2020).  $\pmb{r}(t)$  in NE functionally corresponds to such a cell population, which stores the rate distribution in vector  $\omega$  of  $\pmb{r}(t)$ . The integration function aggregates event pairs separated in time to form TRs. Statistical learning for schema formation is reported to occur in the pathway connecting EC to CA1 in the hippocampus (Schapiro et al., 2017). In NE, we achieve the same function through the time-translation symmetry introduced by  $\pmb{r}(t)$ :  $\pmb{r}(t + \tau) = \pmb{r}(t) \circ \pmb{r}(\tau)$ ,  $\pmb{r}(t) \circ \overline{\pmb{r}(t)} = \mathbf{1}$ ,  $\forall t \in \mathbb{T}_a$ .
A.2 Revelation of TR Validity
A.2.1 Effectiveness in TR Query
NE is an effective method for TR query due to its distributed storage of cross-correlations.
The score function  $f(t)$  describes the occurrences of each event type, where its value is mapped to  $C_p$  for time points implying event occurrences and  $C_n$  for those that do not. The support  $sp(\tau)$  of two event types  $ev_b, ev_h$  can be then approximated by calculating the time-lagged cross-correlation  $R_{f_b, f_h}(\tau) = \sum_{t, t + \tau \in \mathbb{T}_a} f_b(t) f_h(t + \tau)$  after training convergence.
Denote  $F(\omega)$  as the Fourier expansion of  $f(t)$ . Awaring that in NE,  $f(t) = \sum_{j=1}^{d} \text{Real}(\boldsymbol{u} \circ e^{i\omega t})_j$ , we can see that the encoding stage enables a  $d$ -dimensional Fourier-like expansion for each  $f(t)$ . The time vector  $\boldsymbol{\omega}$  provides the expansion basis and  $\boldsymbol{u}$  stores the coefficients as  $F(\omega)\mathrm{s}$ .
By Fourier expansions, the corresponding correlation  $R_{f_b,f_h}'(\tau) = \int_{-\infty}^{+\infty}f_b(t)f_h(t + \tau)dt = \int_{-\infty}^{+\infty}\overline{F_b(\omega)} F_h(\omega)e^{i\omega \tau}d\omega$ . In NE,  $g(\tau) = ||\pmb {u}_b - \pmb {u}_h\circ \pmb {r}(\tau)||^2 = 2 - 2\sum_{j = 1}^d Real(\overline{\pmb{u}_b}\circ \pmb {u}_h\circ e^{i\pmb {\omega}\tau})_j$ . Therefore, each  $g(\tau)$  reveals exactly the fitted time-lagged cross-correlation and is thus proportional to the support  $sp(\tau)$  of a TR. This guarantees NE's effectiveness in TR query.
A.2.2 Effectiveness in TR Detection
Notations Within all event samples trained for a potential TR, suppose that  $m$  event pairs share the same relative time of  $\tau$ , where  $m_{1}$  pairs are both positive or both negative samples. In each remaining  $m_{2} = m - m_{1}$  pair, one is a positive sample while the other is a negative one. Denote that  $\mathbb{M} = \{1, 2, \dots, m\}$ ,  $\mathbb{M}_{1} = \{i_{1}, i_{2}, \dots, i_{m_{1}}\}$ ,  $\mathbb{M}_{2} = \{j_{1}, j_{2}, \dots, j_{m_{2}}\}$ . There exists such least upper bound  $\eta$  that  $(\frac{1}{\sqrt{d}} f(t; ev) - C)^{2} < \eta$  for the score functions of all  $2m$  samples, where  $C = C_{p}, C_{n}$ .
Denote that  $\pmb{a}_i = (\cos(\omega_1 t_i), \cos(\omega_2 t_i), \dots, \cos(\omega_d t_i), \sin(\omega_1 t_i), \sin(\omega_2 t_i), \dots, \sin(\omega_d t_i))^T$ ,  $i \in \mathbb{M}$ , where  $d$  is the dimension of NE vectors,  $t_i$  is time of the  $i$ th body event in  $m$  sample pairs. Denote that in the decoding function,  $\pmb{u}_b - \pmb{u}_h \circ \pmb{r}(\tau) = \pmb{\alpha} - i\pmb{\beta}$  where  $\pmb{\alpha}, \pmb{\beta}$  are both real vectors, and  $\pmb{x} = (\pmb{\alpha}_1, \pmb{\alpha}_2, \dots, \pmb{\alpha}_d, \pmb{\beta}_1, \pmb{\beta}_2, \dots, \pmb{\beta}_d)^T$ . Denote that  $c_1 = \max_{i \in \mathbb{M}_1} (\cos(\pmb{a}_i, \pmb{x}))^2$ ,  $c_2 = \min_{j \in \mathbb{M}_2} (\cos(\pmb{a}_j, \pmb{x}))^2$ .
Using the trigonometric inequality, we can derive the following conclusions:
Theorem 1 Within the  $m_{1}$  sample pairs, if  $c_{1} > 0$ , then  $g(\tau) < \frac{4\eta}{c_{1}}$ .
Theorem 2 Within the  $m_2$  sample pairs, if  $c_2 > 0$  and  $|C_p - C_n| > 2\sqrt{\eta}$ , then  $g(\tau) > \frac{(|C_p - C_n| - 2\sqrt{\eta})^2}{c_2}$ .