SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chi Gao† Zidong Zhou† Luping Shi*
{gaoc20,zzd21}@mails.tsinghua.edu.cn
lpshi@mail.tsinghua.edu.cn
Learning to detect and encode temporal regularities (TRs) in events is a prerequisite for human-like intelligence. These regularities should be formed from limited event samples and stored as easily retrievable representations. Existing event embeddings, however, cannot effectively decode TR validity with well-trained vectors, let alone satisfy the efficiency requirements. We develop Noether Embedding (NE) as the first efficient TR learner with event embeddings. Specifically, NE possesses the intrinsic time-translation symmetries of TRs indicated as conserved local energies in the embedding space. This structural bias reduces the calculation of each TR validity to embedding each event sample, enabling NE to achieve data-efficient TR formation insensitive to sample size and time-efficient TR retrieval in constant time complexity. To comprehensively evaluate the TR learning capability of embedding models, we define complementary tasks of TR detection and TR query, formulate their evaluation metrics, and assess embeddings on classic ICEWS14, ICEWS18, and GDELT datasets. Our experiments demonstrate that NE consistently achieves about double the F1 scores for detecting valid TRs compared to classic embeddings, and it provides over ten times higher confidence scores for querying TR intervals. Additionally, we showcase NE's potential applications in social event prediction, personal decision-making, and memory-constrained scenarios.
Recall the last time you went to a restaurant but waited for half an hour after ordering dishes. You probably knew something was wrong and may have called the waitperson for help. This behavior is guided by the temporal regularity (TR) of 'order dishes  $-($ about 10 minutes) $->$ have meals' stored in your brain as schemas (Ghosh & Gilboa, 2014). Such TRs play a significant role in enabling humans to exhibit flexible out-of-distribution and systematic generalization abilities (Goyal & Bengio, 2022), and are directly learned from experience through a statistical accumulation of common event structures (Pudhiyidath et al., 2020), as shown in Figure 1. Since there exist enormous potential TRs due to a large number of event types and time intervals, detecting valid TRs from all
The code is publicly available at: https://github.com/KevinGao7/Noether-Embedding.
potential ones is therefore necessary, serving as a prerequisite capability for humans to form more complex event schemas in the brain to support downstream cognitive functions (Schapiro et al., 2017; McClelland et al., 1995). To attain human-level abilities in event scenarios, it is crucial to possess two fundamental properties when learning TRs. Firstly, TRs should be formed from even a limited number of experiences, which humans achieve since childhood (Pudhiyidath et al., 2020). Secondly, TRs should be stored as representations that can be instantly retrieved given appropriate cues, which is a central feature of human memory (Chaudhuri & Fiete, 2016).
Figure 1: Illustration of TR learning. TRs indicate temporal associations invariant to time shifts, and are learned from event items through statistical accumulation.
It remains an open problem to jointly achieve the data-efficient formation and time-efficient retrieval of 1-1 TRs with event embeddings, although embedding models have achieved outstanding performance in various tasks such as event completion and event prediction (Cai et al., 2022; Zhao, 2021). Classic event embeddings can only encode patterns such as inversion and composition and decode the fitted event occurrences for better performance in completion tasks (Xu et al., 2020b; Wang et al., 2020; Messner et al., 2022). However, we aim to encode TRs by directly training the embeddings of each event sample and decode TRs by calculating well-trained embeddings without first decoding the fitted event occurrences. Our primary challenge in achieving such a counterintuitive function is to design the inductive bias that automatically integrates the event statistics of each potential TR over time.
Symmetry governs regularities in nature (Tanaka & Kunin, 2021), long before Noether proved the equivalence between symmetries and conservation laws in a physical system (Noether, 1918). Inspired by Noether's theorem, we develop Noether Embedding (NE) with the intrinsic time-translation symmetries of TRs indicated as conserved local energies in the embedding space. Calculating the event statistics of each potential TR is therefore converted to reducing the training loss of all event embeddings. This allows the direct revelation of TR validity by decoding the corresponding local energies through calculating well-trained embeddings after training convergence.
Contributions are twofold. Firstly, we develop NE which for the first time jointly achieves the data-efficient formation and time-efficient retrieval of TRs solely by embedding event samples. Secondly, we define complementary tasks of TR detection and TR query, formulate their evaluation metrics, and adopt classic datasets for evaluations, aiming at complete evaluations of embeddings' TR learning capabilities. Both our tasks and method generalize to arbitrary forms of structured events.
2 Problem Formalization
Temporal Regularity (TR) An event item  $q$  can generally be represented by the basic symbolic form of  $(ev, t)$ , where  $ev$  is the event type, and  $t$  is the discrete occurrence time. Building on the interpretations from cognitive science literature (Ghosh & Gilboa, 2014; Pudhiyidath et al., 2020), we formally define TRs as temporal associations that remain invariant to time shifts:
$$
(e v _ {b}, t) \rightarrow (e v _ {h}, t + \tau) \quad \forall t \in \mathbb {T} _ {a} \tag {1}
$$
$ev_{b} \neq ev_{h}, t, t + \tau$  respectively refer to the body and head event type and their occurrence time.  $\mathbb{T}_a$  refers to the complete collection of the absolute time points in the whole event set, and  $\tau$  denotes the relative time. Note that  $\tau = 0$  indicates the synchrony of body and head event occurrences.
For example, a TR could be: Whenever someone orders dishes in a restaurant, he or she will have meals in around ten minutes, where  $\tau = 10$ .
Metrics for TR Validity Since real-world data often contain noise, we introduce an adaptive  $\triangle = [\tau(1 - \eta), \tau(1 + \eta)]$  to replace  $\tau$  when evaluating statistically learned temporal regularities from events. For an evaluated TR abbreviated as  $tr : (ev_b, ev_h, \tau, \eta)$ , if an event  $q : (ev, t)$  satisfies that  $ev = ev_b$ , we denote it as  $b(q; tr)$ ; if  $ev = ev_h$ , we denote it as  $h(q; tr)$ . We define the support of a TR as the number of event pairs respectively satisfying the body and head:
$$
s p (t r) = n \left(b (q; t r) \wedge h \left(q ^ {\prime}; t r\right) \wedge \left(t ^ {\prime} - t\right) \in \triangle (\tau , \eta)\right) \tag {2}
$$
$\wedge$  denotes 'and',  $\in$  denotes 'in', and  $q:(ev,t),q^{\prime}:(ev^{\prime},t^{\prime})$  refer to arbitrary two different events in the event set. Note that when calculating  $sp(tr)$ , we can only count one event once and in one pair to avoid overcounting when events occur in consecutive periods.
We respectively define the standard confidence, head coverage, and general confidence of a TR as:
$$
s c (t r) = \frac {s p (t r)}{n (b (t r))}, \quad h c (t r) = \frac {s p (t r)}{n (h (t r))}, \quad g c (t r) = \frac {2}{\frac {1}{s c (t r)} + \frac {1}{h c (t r)}} \tag {3}
$$
$n(b(tr)), n(h(tr))$  respectively represent the number of events  $q: (ev, t)$  satisfying  $ev = ev_b$ ,  $ev = ev_h$  in the event set. Here we borrow the metrics  $sc, hc, gc$  generally used in the rule mining field (Galárraga et al., 2015) to ensure fair and reasonable evaluations. We modify them by introducing an adaptive  $\tau$  with  $\eta$  to evaluate TR validity. Intuitively, standard confidence  $sc$  can be viewed as the probability that the head event will occur within time  $t + \Delta$  once a body event occurs at time  $t$ , whose statistical sufficiency is supported by  $sp$ .  $hc$  and  $gc$  can be interpreted similarly.
Above some  $sp$ , the higher the  $gc$ , the more valid a TR is. For a potential TR  $tr : (ev_b, ev_h, \tau, \eta)$  with fixed event types  $ev_b, ev_h$  and ratio  $\eta$ , its general confidence can be written as a function of  $\tau$ :  $gc(\tau)$ .
For a fixed  $\eta$  in  $\triangle$ , a potential TR can be an arbitrary  $(ev_i, ev_j, \tau)$ , where  $i, j \in \mathbb{P}, \tau \in \mathbb{T}_r$  ( $\mathbb{P}$  is the set of event types,  $\mathbb{T}_r$  is the set of relative time points). Therefore, we define the two complementary tasks below to comprehensively evaluate the TR learning capabilities of event embeddings.
TR Detection For a query  $(ev_b, ev_h)$ , its ground truth confidence  $gc_g = \max_{\tau \in \mathbb{T}_r} gc(\tau)$ . Queries whose  $gc_g \geq \theta$  are considered to imply valid TRs that reveal good regularities, while queries whose  $gc_g < \theta$  are considered to imply invalid TRs, where  $\theta$  is a fixed threshold.
The task of TR detection is to identify valid TRs from all tested ones. The model is expected to determine whether a query  $(ev_b, ev_h)$  implies a valid TR. The F1 score of all judgments is reported.
TR Query Only queries that imply valid TRs are considered for testing. The ground truth relative time  $\tau_{g}$  is set as what maximizes  $gc(\tau)$  in computing  $gc_{g}$ . The model outputs  $\tau^{\prime}$ .
The task of TR query is to output the correct  $\tau' = \tau_g$  for valid TRs. For each tested query  $(ev_b, ev_h)$ , a ratio  $r' = \frac{gc(\tau')}{gc_g}$  is computed. The averaged ratio  $r$  of all queries is reported.
3 Noether Embedding
3.1 Inspirations from Noether's Theorem
Noether's theorem In 1915, mathematician Emmy Noether proved one of the most fundamental theorems in theoretical physics: every differentiable symmetry of the action of a physical system with conservative forces has a corresponding conservation law. Specifically, time-translation symmetry corresponds to energy conservation.
TRs indicate time-translation symmetries An event pair  $(ev_b, t) \to (ev_h, t + \tau)$  can be regarded as a mapping of the body and the head event type over  $t$  with a parameter  $\tau$ . Therefore, ideal TRs indicate the invariance of such mappings under the transformation of time translation since the mapping holds  $\forall t \in \mathbb{T}_a$  for TRs.
Construct embeddings with conserved local energies Denote  $\pmb{q}(t; ev)$  as the embedding of each event sample, and  $g(\pmb{q}(t; ev_b), \pmb{q}(t + \tau; ev_h))$  as the local energy of a corresponding body-and-head event pair of a potential TR. If  $g$  is innately conserved, meaning that  $g = g(\tau; ev_b, ev_h)$  is invariant to  $t$ , it indicates time-translation symmetry. We can then use the value of  $g$  to approximate TR validity after training each event embedding  $\pmb{q}(t; ev)$ . A more strict correspondence between NE variables and those in a physical system is shown in Appendix A.1.1.
Noether-Inspired structural biases Accordingly, the enabling factors of NE can be summarized as follows: (i) the event embedding  $\pmb{q}$  should be constructed to make each local energy  $g$  remain invariant to  $t$ ; (ii) the training loss should be designed to make the value of  $g$  approximate TR validity; (iii) the local energy  $g$  should be used as the decoding function. We thus construct NE as below.
Figure 2: Illustration of NE. Solid lines and purple graphs in the middle jointly represent the data flow of NE. The red graphs below and the blue ones above demonstrate cases of a TR with significant temporal regularities and a TR with no. It is shown that each decoding result reveals an integrated temporal association of its relevant event pairs separated in time.
3.2 NE's Framework and Formulas
**Framework** As illustrated in Figure 2, NE uses a distributed storage of complex vectors to learn TRs. At the encoding stage, the event items are converted to NE representations through embedding each event sample, where TR validity is automatically calculated and stored in the embedding space. At the decoding stage, given each query  $(ev_b, ev_h)$ , potential TRs are detected and queried by directly calculating their relevant embeddings to derive the corresponding decoding results  $g(\tau)$ .
The Encoding Stage
$$
\boldsymbol {q} (t; e v) = \boldsymbol {u} (e v) \circ \boldsymbol {r} (t) \tag {4}
$$
In the event embedding above,  $\circ$  denotes the Hadmard (or element-wise) product.  $\pmb{u}(ev), \pmb{r}(t) \in \mathbb{C}^d$  are complex vectors where  $\pmb{u}(ev) = \frac{\pmb{v}(ev)}{||\pmb{v}(ev)||}$ ,  $\pmb{v}(ev) \in \mathbb{C}^d$ ,  $\pmb{r}(t) = e^{i\omega t}$ ,  $\pmb{\omega} \in \mathbb{R}^d$ , and  $d$  is the dimension of vectors. Each event type  $ev$  corresponds to an independently trainable vector of  $\pmb{u}(ev)$  while  $\omega$  is a global time vector for  $\pmb{r}(t)$  of all event embeddings. Note that the  $d\omega s$  in  $\omega$  are fixed to a certain distribution. The event embedding  $\pmb{q}(t; ev)$  can thus be depicted as a rotation of event-type vectors  $\pmb{u}(ev)$  by time  $\pmb{r}(t)$  in the d-dimensional complex space.
The score function and loss function of each event sample are defined as follows:
$$
f (t; e v) = \sum_ {i = 1} ^ {d} \operatorname {R e a l} (\boldsymbol {q} (t; e v)) _ {i} \tag {5}
$$
$$
L (\xi ; C _ {p}, C _ {n}) = \left(\frac {1}{\sqrt {d}} f (\xi) - C _ {p}\right) ^ {2} + \frac {1}{N} \sum \left(\frac {1}{\sqrt {d}} f \left(\xi^ {\prime}\right) - C _ {n}\right) ^ {2} \tag {6}
$$
We denote  $\xi$  as a positive event sample and  $\xi^{\prime}$  s as its generated negative samples whose number is  $N$ . For a positive sample  $\xi :(ev,t)$ , its negative samples  $\xi^{\prime}$  s are the whole set of  $\{(ev,t^{\prime}\neq t)\}$
where  $t' \in \mathbb{T}_a$ .  $C_p, C_n$  are two different constants for positive and negative samples, respectively.  $C_p, C_n \in [-1, 1]$  because  $||\pmb{u}(ev)|| = 1$ , and we generally set  $C_p = 1$ ,  $C_n = 0$ .
Until the training converges, events and TRs form a distributed storage, which includes the global time vector of  $\omega$  and trainable event type vectors  $\pmb{u}(ev)$ . At the decoding stage, no training but only the inference of vectors is conducted, as described below.
The Decoding Stage The decoding function for a query  $(ev_b, ev_h)$  is:
$$
g (\tau) = \left\| \boldsymbol {u} _ {b} - \boldsymbol {u} _ {h} \circ \boldsymbol {r} (\tau) \right\| ^ {2} \tag {7}
$$
$\pmb{u}_b, \pmb{u}_h$  are the event type vectors respectively of  $ev_b, ev_h$ .  $\tau$  is traversed through set  $\mathbb{T}_r$  of the relative time points such as  $\mathbb{T}_r: \{-\tau_{max}, \dots, 0, \dots, \tau_{max}\}$  to plot the decoding results.  $\min_{\tau \in \mathbb{T}_r} g(\tau)$  is computed, which is compared with a global threshold  $g_{th}$  to decide whether a potential TR is valid or not (for TR detection). For a valid TR, the  $\tau'$  which minimizes  $g(\tau), \tau \in \mathbb{T}_r$  is selected as the model output of the relative time (for TR query).
Since  $\boldsymbol{r}(t + \tau) = \boldsymbol{r}(t) \circ \boldsymbol{r}(\tau)$ ,  $\boldsymbol{r}(t) \circ \overline{\boldsymbol{r}(t)} = \mathbf{1}, \forall t \in \mathbb{T}_a$ , the decoding function exactly indicates the conserved local energy:  $g(\tau) = ||\boldsymbol{u}_b - \boldsymbol{u}_h \circ \boldsymbol{r}(\tau)||^2 = ||\boldsymbol{u}_b \circ \boldsymbol{r}(t) - \boldsymbol{u}_h \circ \boldsymbol{r}(t + \tau)||^2 = ||\boldsymbol{q}_b(t) - \boldsymbol{q}_h(t + \tau)||^2, \forall t \in \mathbb{T}_a$ . This indicates that  $g = g(\tau; ev_b, ev_h)$  is invariant to  $t$ , and the conserved energy  $g$  of arbitrary two event samples is of a quadratic form in the embedding space.
3.3 Why NE is Efficient
Briefly speaking, the Fourier-like representations enable NE's large-capacity storage for TR validity and event occurrences, serving as a prerequisite for NE to learn TR effectively. The Noether-inspired structural biases further leads to NE's efficient TR learning capabilities. Here we only illustrate the effect of the structural biases. The reasons why NE learns TR effectively is explained in Appendix A.2.
Data-efficiency by Encoding Translation Symmetries The invariance of  $g(\tau)$  to  $t$  means that the value of  $g(\tau)$  is determined by a competitive effect between sample pairs across time. Considering event sample pairs  $(ev_b, t), (ev_h, t + \tau)$  with varying  $t$  in the embedding space, if a sample pair are both positive or both negative samples, they will decrease  $g(\tau)$  since their score functions are mapped to the same constant. Otherwise, they will increase  $g(\tau)$ . Since  $g(\tau)$  is invariant to  $t$ ,  $g(\tau)$  is trained to balance these two forces. Therefore, the value of  $g(\tau)$  after training convergence is generally determined by the ratio of sample pairs with increasing or decreasing forces.  $g(\tau)$  is thus insensitive to the number of sample pairs that are both positive. This results in a data-efficient TR formation in NE, even with limited event occurrences from which to learn a TR.
Time-efficiency by Decoding Conserved Energies By calculating  $g(\tau) = ||\pmb{u}_b - \pmb{u}_h \circ \pmb{r}(\tau)||^2, \tau \in \mathbb{T}_r$ , we enable efficient TR querying for each query  $(ev_b, ev_h)$ . This process has a constant time complexity since  $\mathbb{T}_r$  is an arbitrary user-selected set of relative time points, and the vector dimension  $d$  can be effectively handled using GPUs. Importantly, the querying time is independent of the number of events in the entire event set and the relevant event occurrences supporting the queried TR.
It is important to highlight that in our evaluation, we initially compare NE and classic embeddings in terms of learning effectiveness (Section 4.2), without considering the efficiency requirements. As classic embeddings are shown to be ineffective in learning TRs, we then focus on demonstrating the learning efficiency of NE in Section 4.3.
Dataset A temporal knowledge graph (TKG) comprises  $(s, p, o, t)$  quadruples (Leblay & Chekol, 2018), where  $s, p, o, t$  represent the subject, predicate, object, and time. TKG is widely used in a variety of fields to represent global political events (Trivedi et al., 2017), financial incidents (Yang et al., 2019), user-item behaviors (Xiao et al., 2020), etc. Notably, ICEWS (Boschee et al., 2015) and GDELT (Leetaru & Schrodt, 2013) are two popular data sources for TKG research (Cai et al., 2022).
In our experiments, we use ICEWS14 and ICEWS18, the same as in (Han et al., 2020). They contain global political events in 2014 and 2018, and we denote them as D14 and D18, respectively. We also use the GDELT released by (Jin et al., 2019), which contains global social events from 2018/1/1 to 2018/1/31. In the experiments, we denote each  $(s,p,o)$  triple as a specific event type  $ev$ . It is worth mentioning that alternative settings, such as representing each predicate  $p$  as an event type  $ev$ , are also applicable to our model.
Model Implementation For NE,  $d = 400$ ,  $C_p = 1$ ,  $C_n = 0$  and the global time vector  $\omega$  is set as  $\omega_k = \frac{(2\pi \times \omega_{max})^{\frac{k}{d}} - 1}{T_a}$ ,  $k = 0, 1, \dots, d - 1$ , where  $T_a$  is the number of absolute time points, and  $\omega_{max}$  is a tunable hyperparameter set as 600. The training details are in Appendix B.1. We compare NE with six classic and vastly different TKG embeddings, DE-SimpleE (Goel et al., 2020), TeRo (Xu et al., 2020b), ATiSE (Xu et al., 2020a), TNTComplex (Lacroix et al., 2020), BoxTE (Messner et al., 2022), and TASTER (Wang et al., 2023) with their original parameter settings and  $d = 400$  the same as NE. We set the queried time set  $\mathbb{T}_r = \{1 - T_a, \dots, 0, \dots, T_a - 1\}$  at the decoding stage. In this way, only one query needs to be decoded between each  $(ev_i, ev_j)$  and  $(ev_j, ev_i)$ ,  $i \neq j$ .
Adaptations. All baselines can only output their score function  $f'(t)$  to decode event occurrences but cannot directly decode TR validity by  $g(\tau)$  as NE does. For comparison, we add an interface  $g'(\tau) = \frac{\sum_{t,t+\tau \in \mathbb{T}_a} f_b'(t) f_h'(t + \tau)}{\sum_{t \in \mathbb{T}_a} f_b'(t) \cdot \sum_{t \in \mathbb{T}_a} f_h'(t)}$  to these models that indirectly compute TR validity from the decoded event occurrences. We also evaluate NE with  $g'(\tau)$  to show the validity of  $g'(\tau)$  itself.
Evaluation Details We select  $(ev_b, ev_h)$ s for tests whose event occurrences of  $ev_b$  and  $ev_h$  are both  $\geq 2$ . Otherwise, their supports  $(sp$  in Definition 2) will be too small for evaluating TR validity. We set  $\eta = 0.1$  in  $\triangle s$  for strict evaluations and take the upper integer  $\triangle = [\tau - [\tau \eta], \tau + [\tau \eta]]$ . Note that in the extreme situation where body and head event occurrences both = 2, stochastic noises are still quite unlikely to interfere with the evaluation of TR validity since  $\eta = 0.1$  is strict. Only forward or reverse queries  $((ev_b, ev_h)s$  whose  $s_b = s_h, o_b = o_h$  or  $s_b = o_h, o_b = s_h$  for  $(s, p, o, t)$  quadruples) are tested for better interpretability without sacrificing generality. We set  $\theta = 0.8$  to distinguish between valid and invalid TRs. The fact that TRs whose  $gc_g \sim 0.8$  are of a tiny percentage of all tested TRs adds to the rationality of such metrics. Ablations where  $\theta = 0.7, 0.9$  are in Appendix B.3.3.
In comparative studies with baselines 4.2, we report the highest F1 in TR detection by tuning the global threshold  $g_{th}$  (defined in Section 3.2) after embedding the whole event set to achieve full evaluations. We also remove TRs whose  $\tau_g = 0$  for TR query because they account for most valid TRs but can hardly reflect the query difficulty. In NE's demonstration studies 4.3 4.4, we first use D14 to derive the global threshold  $g_{th}$  with the highest F1 in TR detection and then apply the same  $g_{th}$  for evaluating NE's performance in D18. This setting better demonstrates NE's practicality.
4.2 Comparisons of Learning Effectiveness
Table 1: Statistical results on ICEWS14, ICEWS18, and GDELT
Embedding	TR Detection (F1)	TR Query (r)
D14	D18	GDELT	D14	D18	GDELT
TNTComplEx	0.26	0.18	0.08	0.08	0.08	0.01
DE-SimplE	0.22	0.20	-	0.09	0.09	-
TASTER	0.18	0.15	0.08	0.09	0.09	0.00
TeRo	0.43	0.64	0.16	0.08	0.08	0.01
BoxTE	0.40	0.40	0.18	0.08	0.08	0.01
ATISE	0.40	0.44	0.18	0.08	0.08	0.01
NE with g'(τ)	0.78	0.79	0.48	0.85	0.83	0.83
NE with g(τ)	0.82	0.83	0.51	0.87	0.86	0.85
Performances Table 10 shows that NE with  $g(\tau)$  has an overwhelming advantage over all baselines with  $g'(\tau)$ , both in detecting valid TRs and querying the relative time on all evaluated datasets. The
excellent performance of NE with  $g'(\tau)$  indicates that  $g'(\tau)$  itself is valid and thus guarantees fair comparisons. It is worth noting that NE is intrinsically different from all existing baselines because only NE can directly decode TR validity by  $g(\tau)$ . In contrast, baselines can only decode event occurrences  $f'(t)$  from which to indirectly calculate TR validity (such as by  $g'(\tau)$ ). Detailed results of precision and recall rates with error bars are reported in Appendix B.2.
(a) Decoding distribution of NE
(b)  $g(\tau)\mathrm{s}$  of NE
(c) Aligned  $f(t)$ s of models
Figure 3: Illustrations of NE or baselines