Title: Noether Embedding: Efficient Learning of Temporal Regularities
Venue: NeurIPS 2023
Accepted: yes
Keywords: Schema Learning

Temporal Regularity

Event Embedding
Abstract: Learning to detect and encode temporal regularities (TRs) in events is a prerequisite for human-like intelligence. These regularities should be formed from limited event samples and stored as easily retrievable representations. Existing event embeddings, however, cannot effectively decode TR validity with well-trained vectors, let alone satisfy the efficiency requirements. We develop Noether Embedding (NE) as the first efficient TR learner with event embeddings. Specifically, NE possesses the intrinsic time-translation symmetries of TRs indicated as conserved local energies in the embedding space. This structural bias reduces the calculation of each TR validity to embedding each event sample, enabling NE to achieve data-efficient TR formation insensitive to sample size and time-efficient TR retrieval in constant time complexity. To comprehensively evaluate the TR learning capability of embedding models, we define complementary tasks of TR detection and TR query, formulate their evaluation metrics, and assess embeddings on classic ICEWS14, ICEWS18, and GDELT datasets. Our experiments demonstrate that NE consistently achieves about double the F1 scores for detecting valid TRs compared to classic embeddings, and it provides over ten times higher confidence scores for querying TR intervals. Additionally, we showcase NE's potential applications in social event prediction, personal decision-making, and memory-constrained scenarios.

Noether Embedding: Efficient Learning of Temporal Regularities

Abstract

Learning to detect and encode temporal regularities (TRs) in events is a prerequisite for human-like intelligence. These regularities should be formed from limited event samples and stored as easily retrievable representations. Existing event embeddings, however, cannot effectively decode TR validity with well-trained vectors, let alone satisfy the efficiency requirements. We develop Noether Embedding (NE) as the first efficient TR learner with event embeddings. Specifically, NE possesses the intrinsic time-translation symmetries of TRs indicated as conserved local energies in the embedding space. This structural bias reduces the calculation of each TR validity to embedding each event sample, enabling NE to achieve data-efficient TR formation insensitive to sample size and time-efficient TR retrieval in constant time complexity. To comprehensively evaluate the TR learning capability of embedding models, we define complementary tasks of TR detection and TR query, formulate their evaluation metrics, and assess embeddings on classic ICEWS14, ICEWS18, and GDELT datasets. Our experiments demonstrate that NE consistently achieves about double the F1 scores for detecting valid TRs compared to classic embeddings, and it provides over ten times higher confidence scores for querying TR intervals. Additionally, we showcase NE's potential applications in social event prediction, personal decision-making, and memory-constrained scenarios.

1 Introduction

Recall the last time you went to a restaurant but waited for half an hour after ordering dishes. You probably knew something was wrong and may have called the waitperson for help. This behavior is guided by the temporal regularity (TR) of 'order dishes -(about 10 minutes)-> have meals' stored in your brain as schemas (Ghosh & Gilboa, 2014). Such TRs play a significant role in enabling humans to exhibit flexible out-of-distribution and systematic generalization abilities (Goyal & Bengio, 2022), and are directly learned from experience through a statistical accumulation of common event structures (Pudhiyidath et al., 2020), as shown in Figure 1. Since there exist enormous potential TRs due to a large number of event types and time intervals, detecting valid TRs from all

potential ones is therefore necessary, serving as a prerequisite capability for humans to form more complex event schemas in the brain to support downstream cognitive functions (Schapiro et al., 2017; McClelland et al., 1995). To attain human-level abilities in event scenarios, it is crucial to possess two fundamental properties when learning TRs. Firstly, TRs should be formed from even a limited number of experiences, which humans achieve since childhood (Pudhiyidath et al., 2020). Secondly, TRs should be stored as representations that can be instantly retrieved given appropriate cues, which is a central feature of human memory (Chaudhuri & Fiete, 2016).

LearnedTR

RestaurantScenarios

PotentialTR

OrderDishes

time

TR

TR

thePhone

time

TR

Free Chat

PayBills

EventItems

time

EventSchema

LearningTRs

It remains an open problem to jointly achieve the data-efficient formation and time-efficient retrieval of 1-1 TRs with event embeddings, although embedding models have achieved outstanding performance in various tasks such as event completion and event prediction (Cai et al., 2022; Zhao, 2021). Classic event embeddings can only encode patterns such as inversion and composition and decode the fitted event occurrences for better performance in completion tasks (Xu et al., 2020b; Wang et al., 2020; Messner et al., 2022). However, we aim to encode TRs by directly training the embeddings of each event sample and decode TRs by calculating well-trained embeddings without first decoding the fitted event occurrences. Our primary challenge in achieving such a counterintuitive function is to design the inductive bias that automatically integrates the event statistics of each potential TR over time.

Symmetry governs regularities in nature (Tanaka & Kunin, 2021), long before Noether proved the equivalence between symmetries and conservation laws in a physical system (Noether, 1918). Inspired by Noether's theorem, we develop Noether Embedding (NE) with the intrinsic time-translation symmetries of TRs indicated as conserved local energies in the embedding space. Calculating the event statistics of each potential TR is therefore converted to reducing the training loss of all event embeddings. This allows the direct revelation of TR validity by decoding the corresponding local energies through calculating well-trained embeddings after training convergence.

Contributions are twofold. Firstly, we develop NE which for the first time jointly achieves the dataefficient formation and time-efficient retrieval of TRs solely by embedding event samples. Secondly, we define complementary tasks of TR detection and TR query, formulate their evaluation metrics, and adopt classic datasets for evaluations, aiming at complete evaluations of embeddings' TR learning capabilities. Both our tasks and method generalize to arbitrary forms of structured events.

2 Problem Formalization

2.1 Definitions

Temporal Regularity (TR) An event item q can generally be represented by the basic symbolic form of ( ev, t ) , where ev is the event type, and t is the discrete occurrence time. Building on the interpretations from cognitive science literature (Ghosh & Gilboa, 2014; Pudhiyidath et al., 2020), we formally define TRs as temporal associations that remain invariant to time shifts:

ev b = ev h , t, t + τ respectively refer to the body and head event type and their occurrence time. T a refers to the complete collection of the absolute time points in the whole event set, and τ denotes the relative time. Note that τ = 0 indicates the synchrony of body and head event occurrences.

̸

For example, a TR could be: Whenever someone orders dishes in a restaurant, he or she will have meals in around ten minutes, where τ = 10 .

Metrics for TR Validity Since real-world data often contain noise, we introduce an adaptive △ = [ τ (1 -η ) , τ (1 + η )] to replace τ when evaluating statistically learned temporal regularities from events. For an evaluated TR abbreviated as tr : ( ev b , ev h , τ, η ) , if an event q : ( ev, t ) satisfies that ev = ev b , we denote it as b ( q ; tr ) ; if ev = ev h , we denote it as h ( q ; tr ) . We define the support of a TR as the number of event pairs respectively satisfying the body and head:

∧ denotes 'and', ∈ denotes 'in', and q : ( ev, t ) , q ′ : ( ev ′ , t ′ ) refer to arbitrary two different events in the event set. Note that when calculating sp ( tr ) , we can only count one event once and in one pair to avoid overcounting when events occur in consecutive periods.

We respectively define the standard confidence, head coverage, and general confidence of a TR as:

n ( b ( tr )) , n ( h ( tr )) respectively represent the number of events q : ( ev, t ) satisfying ev = ev b , ev = ev h in the event set. Here we borrow the metrics sc, hc, gc generally used in the rule mining field (Galárraga et al., 2015) to ensure fair and reasonable evaluations. We modify them by introducing an adaptive τ with η to evaluate TR validity. Intuitively, standard confidence sc can be viewed as the probability that the head event will occur within time t + △ once a body event occurs at time t , whose statistical sufficiency is supported by sp . hc and gc can be interpreted similarly.

Above some sp , the higher the gc , the more valid a TR is. For a potential TR tr : ( ev b , ev h , τ, η ) with fixed event types ev b , ev h and ratio η , its general confidence can be written as a function of τ : gc ( τ ) .

2.2 Tasks

For a fixed η in △ , a potential TR can be an arbitrary ( ev i , ev j , τ ) , where i, j ∈ P , τ ∈ T r ( P is the set of event types, T r is the set of relative time points). Therefore, we define the two complementary tasks below to comprehensively evaluate the TR learning capabilities of event embeddings.

TRDetection For a query ( ev b , ev h ) , its ground truth confidence gc g = max τ ∈ T r gc ( τ ) . Queries whose gc g ≥ θ are considered to imply valid TRs that reveal good regularities, while queries whose gc g < θ are considered to imply invalid TRs, where θ is a fixed threshold.

The task of TR detection is to identify valid TRs from all tested ones. The model is expected to determine whether a query ( ev b , ev h ) implies a valid TR. The F1 score of all judgments is reported.

TR Query Only queries that imply valid TRs are considered for testing. The ground truth relative time τ g is set as what maximizes gc ( τ ) in computing gc g . The model outputs τ ′ .

The task of TR query is to output the correct τ ′ = τ g for valid TRs. For each tested query ( ev b , ev h ) , a ratio r ′ = gc ( τ ′ ) gc g is computed. The averaged ratio r of all queries is reported.

3 Noether Embedding

3.1 Inspirations from Noether's Theorem

Noether's theorem In 1915, mathematician Emmy Noether proved one of the most fundamental theorems in theoretical physics: every differentiable symmetry of the action of a physical system with conservative forces has a corresponding conservation law. Specifically, time-translation symmetry corresponds to energy conservation.

TRs indicate time-translation symmetries An event pair ( ev b , t ) → ( ev h , t + τ ) can be regarded as a mapping of the body and the head event type over t with a parameter τ . Therefore, ideal TRs indicate the invariance of such mappings under the transformation of time translation since the mapping holds ∀ t ∈ T a for TRs.

Construct embeddings with conserved local energies Denote q ( t ; ev ) as the embedding of each event sample, and g ( q ( t ; ev b ) , q ( t + τ ; ev h )) as the local energy of a corresponding body-and-head event pair of a potential TR. If g is innately conserved, meaning that g = g ( τ ; ev b , ev h ) is invariant to t , it indicates time-translation symmetry. We can then use the value of g to approximate TR validity after training each event embedding q ( t ; ev ) . A more strict correspondence between NE variables and those in a physical system is shown in Appendix A.1.1.

Noether-Inspired structural biases Accordingly, the enabling factors of NE can be summarized as follows: (i) the event embedding q should be constructed to make each local energy g remain invariant to t ; (ii) the training loss should be designed to make the value of g approximate TR validity; (iii) the local energy g should be used as the decoding function. We thus construct NE as below.

event

Relevant events of TRa

reveal

local

(±)

type

bad regularity

energy

a2

a1

Event

Decoding

event

type

Items

absolutetime

Embeddings

time

0

relativetime

Results

[ga(T);

m

encode

decode

**.

196(7)

2

1

time

event

type

Relevant events of TRp

eventtype

energy

local

()46

b2

b1

reveal

goodregularity

T=2

absolutetime

02

relativetime

3.2 NE's Framework and Formulas

Framework As illustrated in Figure 2, NE uses a distributed storage of complex vectors to learn TRs. At the encoding stage, the event items are converted to NE representations through embedding each event sample, where TR validity is automatically calculated and stored in the embedding space. At the decoding stage, given each query ( ev b , ev h ) , potential TRs are detected and queried by directly calculating their relevant embeddings to derive the corresponding decoding results g ( τ ) .

The Encoding Stage

In the event embedding above, ◦ denotes the Hadmard (or element-wise) product. u ( ev ) , r ( t ) ∈ C d are complex vectors where u ( ev ) = v ( ev ) || v ( ev ) || , v ( ev ) ∈ C d , r ( t ) = e i ω t , ω ∈ R d , and d is the dimension of vectors. Each event type ev corresponds to an independently trainable vector of u ( ev ) , while ω is a global time vector for r ( t ) of all event embeddings. Note that the d ω s in ω are fixed to a certain distribution. The event embedding q ( t ; ev ) can thus be depicted as a rotation of event-type vectors u ( ev ) by time r ( t ) in the d-dimensional complex space.

The score function and loss function of each event sample are defined as follows:

̸

We denote ξ as a positive event sample and ξ ′ s as its generated negative samples whose number is N . For a positive sample ξ : ( ev, t ) , its negative samples ξ ′ s are the whole set of { ( ev, t ′ = t ) }

where t ′ ∈ T a . C p , C n are two different constants for positive and negative samples, respectively. C p , C n ∈ [ -1 , 1] because || u ( ev ) || = 1 , and we generally set C p = 1 , C n = 0 .

Until the training converges, events and TRs form a distributed storage, which includes the global time vector of ω and trainable event type vectors u ( ev ) . At the decoding stage, no training but only the inference of vectors is conducted, as described below.

The Decoding Stage The decoding function for a query ( ev b , ev h ) is:

u b , u h are the event type vectors respectively of ev b , ev h . τ is traversed through set T r of the relative time points such as T r : {-τ max , ..., 0 , ..., τ max } to plot the decoding results. min τ ∈ T r g ( τ ) is computed, which is compared with a global threshold g th to decide whether a potential TR is valid or not (for TR detection). For a valid TR, the τ ′ which minimizes g ( τ ) , τ ∈ T r is selected as the model output of the relative time (for TR query).

Since r ( t + τ ) = r ( t ) ◦ r ( τ ) , r ( t ) ◦ r ( t ) = 1 , ∀ t ∈ T a , the decoding function exactly indicates the conserved local energy: g ( τ ) = || u b -u h ◦ r ( τ ) || 2 = || u b ◦ r ( t ) -u h ◦ r ( t + τ ) || 2 = || q b ( t ) -q h ( t + τ ) || 2 , ∀ t ∈ T a . This indicates that g = g ( τ ; ev b , ev h ) is invariant to t , and the conserved energy g of arbitrary two event samples is of a quadratic form in the embedding space.

3.3 Why NE is Efficient

Briefly speaking, the Fourier-like representations enable NE's large-capacity storage for TR validity and event occurrences, serving as a prerequisite for NE to learn TR effectively. The Noether-inspired structural biases further leads to NE's efficient TR learning capabilities. Here we only illustrate the effect of the structural biases. The reasons why NE learns TR effectively is explained in Appendix A.2.

Data-efficiency by Encoding Translation Symmetries The invariance of g ( τ ) to t means that the value of g ( τ ) is determined by a competitive effect between sample pairs across time. Considering event sample pairs ( ev b , t ) , ( ev h , t + τ ) with varying t in the embedding space, if a sample pair are both positive or both negative samples, they will decrease g ( τ ) since their score functions are mapped to the same constant. Otherwise, they will increase g ( τ ) . Since g ( τ ) is invariant to t , g ( τ ) is trained to balance these two forces. Therefore, the value of g ( τ ) after training convergence is generally determined by the ratio of sample pairs with increasing or decreasing forces. g ( τ ) is thus insensitive to the number of sample pairs that are both positive. This results in a data-efficient TR formation in NE, even with limited event occurrences from which to learn a TR.

Time-efficiency by Decoding Conserved Energies By calculating g ( τ ) = || u b -u h ◦ r ( τ ) || 2 , τ ∈ T r , we enable efficient TR querying for each query ( ev b , ev h ) . This process has a constant time complexity since T r is an arbitrary user-selected set of relative time points, and the vector dimension d can be effectively handled using GPUs. Importantly, the querying time is independent of the number of events in the entire event set and the relevant event occurrences supporting the queried TR.

4 Experiment

It is important to highlight that in our evaluation, we initially compare NE and classic embeddings in terms of learning effectiveness (Section 4.2), without considering the efficiency requirements. As classic embeddings are shown to be ineffective in learning TRs, we then focus on demonstrating the learning efficiency of NE in Section 4.3.

4.1 Experimental Setting

Dataset A temporal knowledge graph (TKG) comprises ( s, p, o, t ) quadruples (Leblay & Chekol, 2018), where s, p, o, t represent the subject, predicate, object, and time. TKG is widely used in a variety of fields to represent global political events (Trivedi et al., 2017), financial incidents (Yang et al., 2019), user-item behaviors (Xiao et al., 2020), etc. Notably, ICEWS (Boschee et al., 2015) and GDELT (Leetaru & Schrodt, 2013) are two popular data sources for TKG research (Cai et al., 2022).

In our experiments, we use ICEWS14 and ICEWS18, the same as in (Han et al., 2020). They contain global political events in 2014 and 2018, and we denote them as D14 and D18, respectively. We also use the GDELT released by (Jin et al., 2019) , which contains global social events from 2018/1/1 to 2018/1/31. In the experiments, we denote each ( s, p, o ) triple as a specific event type ev . It is worth mentioning that alternative settings, such as representing each predicate p as an event type ev , are also applicable to our model.

Model Implementation For NE, d = 400 , C p = 1 , C n = 0 and the global time vector ω is set as

k

ω k = (2 π × ω max ) d -1 T a , k = 0 , 1 , ..., d -1 , where T a is the number of absolute time points, and ω max is a tunable hyperparameter set as 600. The training details are in Appendix B.1. We compare NE with six classic and vastly different TKG embeddings, DE-SimplE (Goel et al., 2020), TeRo (Xu et al., 2020b), ATiSE (Xu et al., 2020a), TNTComplex (Lacroix et al., 2020), BoxTE (Messner et al., 2022), and TASTER (Wang et al., 2023) with their original parameter settings and d = 400 the same as NE. We set the queried time set T r = { 1 -T a , ..., 0 , ..., T a -1 } at the decoding stage. In this way, only one query needs to be decoded between each ( ev i , ev j ) and ( ev j , ev i ) , i = j .

̸

Adaptations. All baselines can only output their score function f ′ ( t ) to decode event occurrences but cannot directly decode TR validity by g ( τ ) as NE does. For comparison, we add an interface g ′ ( τ ) = ∑ t,t + τ ∈ T a f ′ b ( t ) f ′ h ( t + τ ) ∑ t ∈ T a f ′ b ( t ) · ∑ t ∈ T a f ′ h ( t ) to these models that indirectly compute TR validity from the decoded event occurrences. We also evaluate NE with g ′ ( τ ) to show the validity of g ′ ( τ ) itself.

Evaluation Details We select ( ev b , ev h ) s for tests whose event occurrences of ev b and ev h are both ≥ 2 . Otherwise, their supports ( sp in Definition 2) will be too small for evaluating TR validity. We set η = 0 . 1 in △ s for strict evaluations and take the upper integer △ = [ τ -⌈ τη ⌉ , τ + ⌈ τη ⌉ ] . Note that in the extreme situation where body and head event occurrences both = 2 , stochastic noises are still quite unlikely to interfere with the evaluation of TR validity since η = 0 . 1 is strict. Only forward or reverse queries ( ( ev b , ev h ) s whose s b = s h , o b = o h or s b = o h , o b = s h for ( s, p, o, t ) quadruples) are tested for better interpretability without sacrificing generality. We set θ = 0 . 8 to distinguish between valid and invalid TRs. The fact that TRs whose gc g ∼ 0 . 8 are of a tiny percentage of all tested TRs adds to the rationality of such metrics. Ablations where θ = 0 . 7 , 0 . 9 are in Appendix B.3.3.

In comparative studies with baselines 4.2, we report the highest F1 in TR detection by tuning the global threshold g th (defined in Section 3.2) after embedding the whole event set to achieve full evaluations. We also remove TRs whose τ g = 0 for TR query because they account for most valid TRs but can hardly reflect the query difficulty. In NE's demonstration studies 4.3 4.4, we first use D14 to derive the global threshold g th with the highest F1 in TR detection and then apply the same g th for evaluating NE's performance in D18. This setting better demonstrates NE's practicality.

4.2 Comparisons of Learning Effectiveness

Performances Table 10 shows that NE with g ( τ ) has an overwhelming advantage over all baselines with g ′ ( τ ) , both in detecting valid TRs and querying the relative time on all evaluated datasets. The

excellent performance of NE with g ′ ( τ ) indicates that g ′ ( τ ) itself is valid and thus guarantees fair comparisons. It is worth noting that NE is intrinsically different from all existing baselines because only NE can directly decode TR validity by g ( τ ) . In contrast, baselines can only decode event occurrences f ′ ( t ) from which to indirectly calculate TR validity (such as by g ′ ( τ ) ). Detailed results of precision and recall rates with error bars are reported in Appendix B.2.

(a) Decoding distribution of NE

(b)

g

(

τ

)

s of NE

(c) Aligned

f

(

t

)

s of models

1.50

2.0

1.0

TNTComplEx

1.25

NE

1.00

Score

1.5

Score

0.8

Occurence

0.6

0.75

s

1.0

Event

M

0.50

R

0.4

0.25

0.5

Valid TR

0.2

0.00

Threshold

0.0

0.0

0.2

0.4

0.6

0.8

1.0

-400

-200

0

200

0

100

200

300

Relative Time(Day)

AbsoluteTime(Day)

Discussion Figure 3(a) shows NE's decoding distribution g m = min τ ∈ T r g ( τ ) by each query's ground truth gc g = max τ ∈ T r gc ( τ ) . It can be observed that the decoded conserved local energy accurately reveals the TR validity, which enables NE to successfully distinguish between valid and invalid TRs, as demonstrated in the case shown in Figure 3(b). Table 10 shows that baselines with g ′ ( τ ) still perform poorly. This is mainly because their f ( t ) s do not fill well. Specifically, Figure 3(c) illustrates that TNTComplEx has much noise in its f ′ ( t ) compared to NE. The reason is that baseline models are generally designed to achieve good performance in the completion task and, therefore, over-apply the generalization capabilities of distributed representations, which hinders the fit of event occurrences.

4.3 NE's Superior Learning Capabilities

(a) TR detection by

n

(b) TR query by

n

(c) TR query by

τ

g

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

1

F

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0

10

20

30

40

0.0

0

+

10

20

30

40

0.0

0

50

100

150

200

250

NumberofEvents

NumberofEvents

RelativeTime(Day)

Data Efficiency In Figure 4(a) and 4(b), we group TRs by their number n of relevant events. It is shown that NE accurately detects valid TRs and reports correct τ s with only two event pairs as positive samples. This performance is comparable to humans, able to form temporally associative memories with minimal experience (Hudson et al., 1992; Bauer & Mandler, 1989; Schapiro et al., 2017). Note that the maximum group in the test has n > 400 , while we only show groups with n ≤ 40 for display considerations.

Time Efficiency As explained in 3.3, NE's specific decoding function g ( τ ) enables NE to retrieve TRs in a constant time complexity by vector computations. Calculating g ′ ( τ ) of classic embeddings, however, requires an additional time complexity relevant to T a (the number of absolute time points).

Storage Efficiency In addition to the data-efficient and time-efficient properties, NE is, in fact, also a storage-efficient memory for TRs and event occurrences. Here is a detailed analysis:

The storage of NE vectors, denoted as S ( NE ) , can be calculated as follows: S ( NE ) = S ( ev -vector ) + S ( time -vector ) = 2 ∗ N ∗ d ∗ 64 bit + 2 ∗ d ∗ 64 bit . In our experiments, we used torch.LongTensor and N represents the number of event-type vectors. On the other hand, the storage of exact counting, denoted as S ( CT ) , can be calculated as follows: S ( CT ) = S ( TR ) + S ( event ) = N 2 ∗ T a ∗ log 2 ( n/N ) bit + N ∗ ( n/N ) ∗ log 2 ( T a ) bit . Here, n represents the number of all event occurrences. We reserved the storage accuracy of TR validity to effectively distinguish different values, resulting in approximately log 2 ( n/N ) bit for each TR validity ( ev b , ev h , τ ) .

For the ICEWS14 and ICEWS18 datasets, where d = 400 , T a = 365 , and n = 90730 , 468558 , N = 50295 , 266631 , we calculated the compression ratio S ( CT ) S ( NE ) of NE as 421 and 2336, respectively. This remarkable capability of NE can be attributed to the fact that it separately stores the information of TR validities ( ev b , ev h , τ ) using event-type vectors and a global time vector. By representing the common information of related TRs efficiently in memory, NE achieves a compression ratio that is approximately linear to the number of event types.

Flexibility In Figure 4(c), we group valid TRs by their golden τ g s. NE is shown to be flexible for learning TRs with τ s varying broadly, comparable to humans with stable memory codes for various time intervals in the hippocampus (Mankin et al., 2012).

4.4 NE's Wide Potential Use

(a) Case 1

(b) Case 2

head

body

50

100

150

200

250

Absolute Time (Day)

head

body

50

100

150

200

250

300

Absolute Time (Day)

Pay

Play

Eat

Look

Order

0

100

200

300

Timestamps

(a) Event occurrences

(b) TR validity

Order-

0.8

Look

- 0.6

Eat

0.4

Play

- 0.2

Pay

Order Look

Eat

Play

Pay

0.0

Potential Use in Social Event Prediction In D18, NE successfully reports 21010 valid TRs with an F1 score of 0.83. The encoding stage takes around 1 hour, while decoding only takes less than 1 minute. Cases are presented below and in Figure 5, with additional cases available in Appendix B.4.

(1) Case 1. Citizen (India) will Reject to Narendra Modi (events in day 23, 122, and 168) in around 87 days whenever Narendra Modi Appeal for diplomatic cooperation (such as policy support) to Citizen (India) (events in day 102, 200, and 264).

(2) Case 2. Russia will Meet at a 'third' location with Ukraine (events in day 31 and 138) in around 136 days whenever Ukraine Use conventional military force to Russia (events in day 161 and 288).

Since the TRs mined can generalize across time, the results above imply NE's potential use in both reliable and interpretable event predictions urgently needed in the big data era (Zhao, 2021).

Potential Use in Personal Decision Making Consider an intelligent machine that has visited a restaurant four times, with the occurrence time of each event episode used as input for NE, as shown in Figure 6(a). After training all events, the decoded TR validity min τ ∈ T g ( τ ) is transformed linearly

r

and demonstrated in Figure 6(b). Despite the recurrent TRs on the slash that can be set aside, valid TRs such as 'order dishes -(about 10 minutes)-> have meals' are well distinguished from invalid ones such as 'order dishes -(about 5 minutes) -> look at the floor'.

Combining NE with front-end methods that take unstructured videos as input and output event items in the form of ( ev, t ) , and with back-end methods that use the decoded valid TRs to guide decisionmaking, NE has the potential to aid intelligent machines in surviving in changing environments by generalizing from little experience, just as human beings (Goyal & Bengio, 2022; Xue et al., 2018).

Potential Use in Memory-constrained Scenarios As discussed in 4.3, NE approximately reduces the required storage space from M 2 to M in our experimental settings, where M is the number of event types. Therefore, NE holds significant potential for applications in memory-constrained scenarios like the edge. This is important when M is large, which is usual in the big-data era.

4.5 Ablation Studies

Here we demonstrate ablation studies of loss constants C p , C n and time vector ω , while those for dimension d and event type vector u are shown in Appendix B.3.

Loss Constants Table 2 shows that NE performs optimally when C p = 1 and C n = 0 . In fact, as C p approaches 1, the g ( τ ) of perfect TRs ( gc ( τ ) = 1 , η = 1 ) is enforced to converge to its minimum 0 . This global constant for all potential TRs in the embedding space allows g ( τ ) to reveal TR validity better. In terms of C n , setting it to 0 results in negative samples occupying the largest embedding space. Since negative samples comprise most of all trained event samples, this setting improves the fit of negative samples and optimizes NE's performance.

Maximal Frequency Coefficient Table 3 shows that NE performs optimally with different values of ω max , respectively, in the TR detection and query task.

Frequency Distribution Table 4 shows that the larger the dataset, the more exponential distribution ( ω k = (2 π × ω max ) k d -1 T a , k = 0 , 1 , ..., d -1 ) surpasses linear distribution ( ω k = 2 π × k × ω max d × T a , k = 0 , 1 , ..., d -1 ) with the same parameters of d = 400 , ω max = 600 . This suggests that real-world event occurrences depend more on low-frequency terms than high-frequency ones.

5 Related Work

Event Schema Induction In the natural language processing (NLP) field, a significant research focus is on inducing event schemas from text (Huang et al., 2016; Li et al., 2021), including from language models (Dror et al., 2022), to support downstream applications such as search, questionanswering, and recommendation (Guan et al., 2022). These NLP methods aim to organize known event regularities already given as priors for the extracting algorithm (such as extracting 'earthquake -> tsunami' from the sentence 'An earthquake causes a tsunami.') and focus on the schemas for use. In contrast, our tasks are designed to learn event regularities directly from experience without supervision. Specifically, the only prior models know is whether an event occurs, and models are required to detect valid TRs from all potential ones and report the correct relative time of valid TRs.

Temporal Rule Mining Various temporal rules are mined from event sets to reveal regularities in industry, security, healthcare, etc (Segura-Delgado et al., 2020; Chen et al., 2007; Yoo & Shekhar, 2008; Namaki et al., 2017). Although the search methods used discover event regularities directly from events without supervision, both the mined rules and source events are generally stored as symbolic representations in list form. In contrast, by applying event embeddings, NE is a distributed and approximate memory for both TRs and event items. NE strikes a balance between storage efficiency and storage accuracy compared to exact counting, as detailedly discussed in 4.3.

Embedding Models of Structured Data Within all embedding models of static and temporal knowledge graphs (Chen et al., 2020; Cai et al., 2022; Wang et al., 2020; Messner et al., 2022), three are most related to NE. RotatE (Sun et al., 2019) represents each entity and relationship as a complex vector to model relation patterns on knowledge graphs, and TeRo (Xu et al., 2020b) represents time as a rotation of entities to model time relation patterns on temporal knowledge graphs. While both RotatE and TeRo introduce complex vectors for better completion performance, NE first explores using complex vectors for TR detection in events. In particular, the specific use of complex vectors in RotatE encodes inverse relations and in TeRo encodes asymmetric and reflexive relations. NE, instead, apply rotating complex unit vectors to encode time-translation symmetries of all potential TRs. IterE (Zhang et al., 2019) construct a decodable embedding model to discover rules for better knowledge graph completion performance. While we take functional inspiration from IterE that embedding models can jointly encode data and decode regularities, we focus on event data and define the new problems of TR detection and TR query. Specifically, while IterE focuses on discrete variables, NE focuses on the continuous variable of time that involves Fourier-like transformations.

To summarize, TR detection and TR query focus on achieving human-like schema learning capabilities rather than pursuing better support for NLP applications like the event schema induction task. Meanwhile, NE leverages the advantages of distributed representations over symbolic ones of search methods in temporal rule mining and is distinct from existing embedding models of structured data.

6 Conclusion

We have developed NE which for the first time enables data-efficient TR formation and time-efficient TR retrieval simply through embedding event samples. We have formally defined the tasks of TR detection and TR query to comprehensively evaluate the TR learning capabilities of embedding models. We have demonstrated NE's potential use in social event prediction, personal decisionmaking, and memory-constrained scenarios. We hope that we have facilitated the development of human-like event intelligence.

One limitation of NE is that when the vector dimension d is set much lower than the number of absolute time points T a , significant performance degradation of NE will occur as observed in the GDELT experiment. Future research is needed to improve this weakness. The privacy issues potentially brought about by TR detection and the causality of TRs should also be handled properly.

References

Bauer, P. J. and Mandler, J. M. One thing follows another: Effects of temporal structure on 1-to 2-year-olds' recall of events. Developmental psychology , 25(2):197, 1989.

Boschee, E., Lautenschlager, J., O'Brien, S., Shellman, S., Starz, J., and Ward, M. Icews coded event data. Harvard Dataverse , 12, 2015.

Bright, I. M., Meister, M. L., Cruzado, N. A., Tiganj, Z., Buffalo, E. A., and Howard, M. W. A temporal record of the past with a spectrum of time constants in the monkey entorhinal cortex. Proceedings of the National Academy of Sciences , 117(33):20274-20283, 2020.

Cai, B., Xiang, Y., Gao, L., Zhang, H., Li, Y., and Li, J. Temporal knowledge graph completion: A survey. arXiv preprint arXiv:2201.08236 , 2022.

Chaudhuri, R. and Fiete, I. Computational principles of memory. Nature neuroscience , 19(3): 394-403, 2016.

Chen, M., Chen, S.-C., and Shyu, M.-L. Hierarchical temporal association mining for video event detection in video databases. In 2007 IEEE 23rd International Conference on Data Engineering Workshop , pp. 137-145. IEEE, 2007.

Chen, X., Jia, S., and Xiang, Y. A review: Knowledge reasoning over knowledge graph. Expert Systems with Applications , 141:112948, 2020.

Dror, R., Wang, H., and Roth, D. Zero-shot on-the-fly event schema induction. arXiv preprint arXiv:2210.06254 , 2022.

Galárraga, L., Teflioudi, C., Hose, K., and Suchanek, F. M. Fast rule mining in ontological knowledge bases with amie + . The VLDB Journal , 24(6):707-730, 2015.

Ghosh, V. E. and Gilboa, A. What is a memory schema? a historical perspective on current neuroscience literature. Neuropsychologia , 53:104-114, 2014.

Goel, R., Kazemi, S. M., Brubaker, M., and Poupart, P. Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 3988-3995, 2020.

Goyal, A. and Bengio, Y. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A , 478(2266):20210068, 2022.

Guan, S., Cheng, X., Bai, L., Zhang, F., Li, Z., Zeng, Y ., Jin, X., and Guo, J. What is event knowledge graph: A survey. IEEE Transactions on Knowledge and Data Engineering , 2022.

Han, Z., Chen, P., Ma, Y., and Tresp, V. xerte: Explainable reasoning on temporal knowledge graphs for forecasting future links. arXiv preprint arXiv:2012.15537 , 2020.

Howard, M. W., MacDonald, C. J., Tiganj, Z., Shankar, K. H., Du, Q., Hasselmo, M. E., and Eichenbaum, H. A unified mathematical framework for coding time, space, and sequences in the hippocampal region. Journal of Neuroscience , 34(13):4692-4707, 2014.

Huang, L., Cassidy, T., Feng, X., Ji, H., Voss, C., Han, J., and Sil, A. Liberal event extraction and event schema induction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 258-268, 2016.

Hudson, J. A., Fivush, R., and Kuebli, J. Scripts and episodes: The development of event memory. Applied Cognitive Psychology , 6(6):483-505, 1992.

Jin, W., Jiang, H., Qu, M., Chen, T., Zhang, C., Szekely, P., and Ren, X. Recurrent event network: Global structure inference over temporal knowledge graph. 2019.

Lacroix, T., Obozinski, G., and Usunier, N. Tensor decompositions for temporal knowledge base completion. arXiv preprint arXiv:2004.04926 , 2020.

Leblay, J. and Chekol, M. W. Deriving validity time in knowledge graph. In Companion Proceedings of the The Web Conference 2018 , pp. 1771-1776, 2018.

Leetaru, K. and Schrodt, P. A. Gdelt: Global data on events, location, and tone, 1979-2012. In ISA annual convention , volume 2, pp. 1-49. Citeseer, 2013.

Li, M., Li, S., Wang, Z., Huang, L., Cho, K., Ji, H., Han, J., and Voss, C. The future is not one-dimensional: Complex event schema induction by graph modeling for event prediction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 5203-5215, 2021.

Mankin, E. A., Sparks, F. T., Slayyeh, B., Sutherland, R. J., Leutgeb, S., and Leutgeb, J. K. Neuronal code for extended time in the hippocampus. Proceedings of the National Academy of Sciences , 109(47):19462-19467, 2012.

McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review , 102(3):419, 1995.

Messner, J., Abboud, R., and Ceylan, I. I. Temporal knowledge graph completion using box embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 7779-7787, 2022.

Namaki, M. H., Wu, Y., Song, Q., Lin, P., and Ge, T. Discovering graph temporal association rules. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , pp. 1697-1706, 2017.

Noether, E. Invariante variationsprobleme. Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathematisch-Physikalische Klasse , 1918:235-257, 1918. URL http://eudml.org/ doc/59024 .

Pudhiyidath, A., Roome, H. E., Coughlin, C., Nguyen, K. V., and Preston, A. R. Developmental differences in temporal schema acquisition impact reasoning decisions. Cognitive neuropsychology , 37(1-2):25-45, 2020.

Schapiro, A. C., Turk-Browne, N. B., Botvinick, M. M., and Norman, K. A. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Philosophical Transactions of the Royal Society B: Biological Sciences , 372(1711):20160049, 2017.

Segura-Delgado, A., Gacto, M. J., Alcalá, R., and Alcalá-Fdez, J. Temporal association rule mining: An overview considering the time variable as an integral or implied component. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 10(4):e1367, 2020.

Sun, Z., Deng, Z.-H., Nie, J.-Y., and Tang, J. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197 , 2019.

Tanaka, H. and Kunin, D. Noether's learning dynamics: Role of symmetry breaking in neural networks. Advances in Neural Information Processing Systems , 34:25646-25660, 2021.

Trivedi, R., Dai, H., Wang, Y., and Song, L. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In international conference on machine learning , pp. 3462-3471. PMLR, 2017.

Tsao, A., Sugar, J., Lu, L., Wang, C., Knierim, J. J., Moser, M.-B., and Moser, E. I. Integrating time from experience in the lateral entorhinal cortex. Nature , 561(7721):57-62, 2018.

Wang, J., Zhang, W., Chen, X., Lei, J., and Lai, X. 3drte: 3d rotation embedding in temporal knowledge graph. IEEE Access , 8:207515-207523, 2020.

Wang, X., Lyu, S., Wang, X., Wu, X., and Chen, H. Temporal knowledge graph embedding via sparse transfer matrix. Information Sciences , 623:56-69, 2023.

Xiao, C., Sun, L., and Ji, W. Temporal knowledge graph incremental construction model for recommendation. In Asia-pacific web (apweb) and web-age information management (waim) joint international conference on web and big data , pp. 352-359. Springer, 2020.

Xu, C., Nayyeri, M., Alkhoury, F., Yazdi, H., and Lehmann, J. Temporal knowledge graph completion based on time series gaussian embedding. In International Semantic Web Conference , pp. 654-671. Springer, 2020a.

Xu, C., Nayyeri, M., Alkhoury, F., Yazdi, H. S., and Lehmann, J. Tero: A time-aware knowledge graph embedding via temporal rotation. arXiv preprint arXiv:2010.01029 , 2020b.

Xue, J.-R., Fang, J.-W., and Zhang, P. A survey of scene understanding by event reasoning in autonomous driving. Machine Intelligence Research , 15(3):249-266, 2018.

Yang, Y., Wei, Z., Chen, Q., and Wu, L. Using external knowledge for financial event prediction based on graph neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management , pp. 2161-2164, 2019.

Yoo, J. S. and Shekhar, S. Similarity-profiled temporal association mining. IEEE Transactions on Knowledge and Data Engineering , 21(8):1147-1161, 2008.

Zhang, W., Paudel, B., Wang, L., Chen, J., Zhu, H., Zhang, W., Bernstein, A., and Chen, H. Iteratively learning embeddings and rules for knowledge graph reasoning. In The World Wide Web Conference , pp. 2366-2377, 2019.

Zhao, L. Event prediction in the big data era: A systematic survey. ACM Computing Surveys (CSUR) , 54(5):1-37, 2021.

Appendix

A Theoretical Illustrations

A.1 Interdisciplinary Correspondences

A.1.1 Physical Correspondence

Considering each event type as a particle, the event embedding q ( t ; ev ) can be viewed as its generalized coordinate at time t in the d-dimensional complex embedding space. Suppose that each two particles ev b , ev h are connected by a spring with the force linear to their distance, their potential energy can then be expressed as || q b ( t ) -q h ( t + τ ) || 2 , where τ is a parameter. As q ( t ; ev ) changes with time, its kinetic energy also changes and can be viewed as being driven by external nonconservative forces. Using the extended Noether's theorem, we can see that the work of external forces and the kinetic energy offset, leading to a conserved quantity of time-translation symmetry that only includes the potential energy instead of the total energy. The inductive bias of Noether Embedding exactly enforces that such local potential energies are innately conserved, which means that || q m ( t k ) -q n ( t k + τ r ) || 2 = || q m ( t s ) -q n ( t s + τ r ) || 2 , ∀ ev m , ev n ∈ P , t k , t s ∈ T a , τ r ∈ T r , where P , T a , T r refer to the set of event types, absolute time points, and relative time points, respectively.

A.1.2 Biological Correspondence

The formation of TRs relies on two central functions, namely measurement and integration. The measurement function involves comparing the temporal distance between each two events. It is claimed that Laplace transformation exists in the hippocampus for representing time (Howard et al., 2014). Specifically, the population of temporal context cells (or referred to as ramping cells (Tsao et al., 2018)) in the lateral entorhinal cortex is discovered to code time with a variety of rate constants (Bright et al., 2020). r ( t ) in NE functionally corresponds to such a cell population, which stores the rate distribution in vector ω of r ( t ) . The integration function aggregates event pairs separated in time to form TRs. Statistical learning for schema formation is reported to occur in the pathway connecting EC to CA1 in the hippocampus (Schapiro et al., 2017). In NE, we achieve the same function through the time-translation symmetry introduced by r ( t ) : r ( t + τ ) = r ( t ) ◦ r ( τ ) , r ( t ) ◦ r ( t ) = 1 , ∀ t ∈ T a .

A.2 Revelation of TR Validity

A.2.1 Effectiveness in TR Query

NE is an effective method for TR query due to its distributed storage of cross-correlations.

The score function f ( t ) describes the occurrences of each event type, where its value is mapped to C p for time points implying event occurrences and C n for those that do not. The support sp ( τ ) of two event types ev b , ev h can be then approximated by calculating the time-lagged cross-correlation R f b ,f h ( τ ) = ∑ t,t + τ ∈ T a f b ( t ) f h ( t + τ ) after training convergence.

Denote F ( ω ) as the Fourier expansion of f ( t ) . Awaring that in NE, f ( t ) = ∑ d j =1 Real ( u ◦ e i ω t ) j , we can see that the encoding stage enables a d -dimensional Fourier-like expansion for each f ( t ) . The time vector ω provides the expansion basis and u stores the coefficients as F ( ω ) s.

By Fourier expansions, the corresponding correlation R ′ f b ,f h ( τ ) = ∫ + ∞ -∞ f b ( t ) f h ( t + τ ) dt = ∫ + ∞ -∞ F b ( ω ) F h ( ω ) e iωτ dω . In NE, g ( τ ) = || u b -u h ◦ r ( τ ) || 2 = 2 -2 ∑ d j =1 Real ( u b ◦ u h ◦ e i ω τ ) j . Therefore, each g ( τ ) reveals exactly the fitted time-lagged cross-correlation and is thus proportional to the support sp ( τ ) of a TR. This guarantees NE's effectiveness in TR query.

A.2.2 Effectiveness in TR Detection

Notations Within all event samples trained for a potential TR, suppose that m event pairs share the same relative time of τ , where m 1 pairs are both positive or both negative samples. In each remaining m 2 = m -m 1 pair, one is a positive sample while the other is a negative one. Denote that M = { 1 , 2 , ..., m } , M 1 = { i 1 , i 2 , ..., i m 1 } , M 2 = { j 1 , j 2 , ..., j m 2 } . There exists such least upper bound η that ( 1 √ d f ( t ; ev ) -C ) 2 < η for the score functions of all 2 m samples, where C = C p , C n .

Denote that a i = ( cos ( ω 1 t i ) , cos ( ω 2 t i ) , ..., cos ( ω d t i ) , sin ( ω 1 t i ) , sin ( ω 2 t i ) , ..., sin ( ω d t i )) T , i ∈ M , where d is the dimension of NE vectors, t i is time of the i th body event in m sample pairs. Denote that in the decoding function, u b -u h ◦ r ( τ ) = α -i β where α , β are both real vectors, and x = ( α 1 , α 2 , ..., α d , β 1 , β 2 , ..., β d ) T . Denote that c 1 = max i ∈ M 1 ( cos ( a i , x )) 2 , c 2 = min j ∈ M 2 ( cos ( a j , x )) 2 .

Using the trigonometric inequality, we can derive the following conclusions:

Theorem 1 Within the m 1 sample pairs, if c 1 > 0 , then g ( τ ) < 4 η c 1 .

Theorem 2 Within the m 2 sample pairs, if c 2 > 0 and | C p -C n | > 2 √ η , then g ( τ ) > ( | C p -C n |-2 √ η ) 2 c 2 .

Implications d, T a , ω , C p , C n jointly result in the probability distributions of c 1 , c 2 , where c 1 , c 2 > 0 is generally guaranteed by experimental settings. Two conclusions can then be drawn from the two theorems. (1) Convergence. We can see from theorem 1 that g ( τ ) → 0 as η → 0 . This implies that the g ( τ ) of perfect TRs ( gc ( τ ) = 1 , η = 1 ) will converge to its minimum of 0 . (2) Competition. Since g ( τ ) is invariant to t , comparing these two theorems also tells us about the competing effect of well-trained sample pairs for the value of g ( τ ) , generally affected by the ratio m 1 m 2 . These two conclusions, along with the fact that g ( τ ) is proportional to the support sp ( τ ) (as illustrated in A.2.1), jointly make g ( τ ) reveal the TR validity gc ( τ ) and thus guarantees NE's effectiveness in TR detection.

It is worth noting that d is generally set as d > T a to control the values of c 1 , c 2 , where T a is the number of absolute time points of the whole event set. Otherwise, NE's TR detection performance will be interfered. For example, if d ≪ T a , then d ≪ m in most cases. It will thus be very likely that c 1 = 0 so that theorem 1 can not be applied in NE.

B Experimental Supplements

B.1 Training Details

All models are trained for 100 epochs on each dataset using the Adagrad optimizer (with a learning rate of 0.01) and the StepLR learning rate scheduler (with a step size of 10 and gamma of 0.9). The experiments are conducted on a single GPU (GeForce RTX 3090). The hyper-parameters of NE are fine-tuned using a grid search to achieve relatively optimal results.

B.2 Main Results with Error Bars

To ensure the reliability of the results, the experiments are repeated three times, and the error bars are derived accordingly. Table 5 and 6 show that the main results are quite stable with small error bars. Note that the precision and recall rates in Table 6 correspond exactly to the F1 scores in Table 5. The reason why the recall rate of NE is lower than that of TASTER is that we report the highest F1 score of each model in comparative studies by tuning their respective global threshold, denoted as g th . As the F1 score is calculated as the harmonic mean of precision and recall rates, TASTER achieves its highest F1 score by reporting many false positives, resulting in a relatively high recall rate but an extremely low precision rate.

B.3 Additional Ablations

Unless otherwise specified, the experiments below adopt the original parameter settings as described in the main text.

B.3.1 Normalization of Event Type Vectors

Table 7 demonstrates that normalized event type vectors slightly outperform unnormalized ones in the TR detection task.

B.3.2 Dimension of Embedding Vectors

Table 8 and 9 demonstrate that d scarcely affects the performance of NE as long as it is more than some certain value. It is worth noting that NE's detection performance in GDELT may be further improved with larger values of d s, as illustrated in A.2.2, because T a = 2976 in GDELT, which is much larger than the tested dimensions.

B.3.3 Threshold for Valid TRs

In the main text, the threshold for distinguishing valid and invalid TRs is chosen as θ = 0 . 8 . Here we report NE's results on D14 with varying θ s in Table 10. It is shown that NE still has an overwhelming advantage over all baselines.

B.4 Mined TRs

Additional cases of mined TRs are shown in Table 11 as below.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for detecting, and embedding, temporal regularities from time-stamped event data, where events have a discrete type, and temporal regularities correspond to one base type preceding another head type by a characteristic relative time. A temporal regularity (TR) is defined in the following way. A temporal regularity associates a body event-type $b$ with a head-event type $h$, alongside a mean relative time $\\tau$, and a universal factor $\\eta$, such that events of type $b$ at time $t$ are \"regularly\" followed (or maybe also preceded) by an event of type $h$, with a mean time delay of $\\tau$ but with a time variation in the interval $\\Delta = [\\tau(1-\\eta), \\tau(1+\\eta)]$. One challenge of this is to discover these temporal regularities purely from time-stamped events, i.e. to abstract away from the specific time of the body event type. The principle contribution of this work is to train Noether Embeddings (NE) which the authors state \"enable both the data efficient formation and rapid retrieval of TRs simply through embedding each event sample\".\n\n\nThe authors define a collection of metrics for temporal regularities between any given body and head types (b,h). standard confidence (sc), head coverage (hc) and general confidence (gc), the last of which is a key measure of how strong the evidence is for a temporal regularity under a particular interval $\\Delta$ (which is itself a consequence of the mean time $\\tau$). They go on to specify two tasks, the first \"TR detection\" is to find the best value of general confidence for each pair of event types (for any value of relative-time) and specify this pair of event types as a TR if this value is above some threshold. The second task is TR Query which asks what the characteristic relative time between a pair of events is. Noether embeddings are then a composition of event embedding and time embedding, in such a way that the complex field is used to capture the time information withing just one part of the composition. TRs detection (or scoring) can then be efficiently calculated between any pair of embedded events. Associated methods are then used to determine the TR detection and query operations.\n\nThe authors compare their method of evaluating TR detection and querying with appropriately adapted methods from the literature. It isn't entirely clear from the paper, whether these other methods are trained to maximise scores between valid TRs in some way (but the assumption is that they are). The authors show that, on three datasets derived from large scale temporal knowledge graph data used elsewhere in the literature, their method performs substantially better at detecting and scoring (querying) temporal regularities in data.\n\nSTRENGTHS:\nThe method is well motivated in terms of Emmy Noether's work corresponding symmetries with conservation laws. The authors develop a neat way to encode things that lends itself to efficient training and their performance on three different datasets is substantially better than other methods. \nIt isn't entirely clear how to judge the significance of this work, as the authors define their own task and their own metrics, possibly because there are no suitable pre-existing candidates for this.\n\nWEAKNESSES:\nThe paper could be a little clearer in parts. For instance:\n* It is unclear whether the authors' definition of temporal regularity (TR) is unique to the authors or is defined elsewhere. The authors relate this to prior work on temporal regularities, but the precise formulation appears to be the authors' own. It would be helpful to know whether and how this relates to other formal definitions of temporal regularities.\n* More effort could be made to give intuitive (informal) descriptions of the defined terms, e.g. standard confidence, TR Detection, TR Query, ground truth confidence, and so on. \n* How the data is partitioned into test and train, and how various methods are optimised from training data could be made clearer in the main body of the paper.\n* It isn't really clear until late in the paper that the time-stamps are discrete. I am not sure whether this method would be tractable for continuously valued time-stamps\n* The intuitive meaning of what is being established by temporal regularities and the associated metrics could be clearer too. For instance, what does it mean to have a ground-truth confidence greater than 0.8. Can that be given a (loose) statistical interpretation at all?\n* The notion of an interval $\\Delta$ which depends on the mean relative time $\\tau$ is not clearly discussed. Is this like saying that the TRs model events whose relative time is uniformly distributed in some range? Is this a little brittle compared to a method that \"smoothly\" traded off relative-time-clustering with observation frequency?\n\nThe notation is a little confusing at times. In particular, Equation 2 or the text around it, could make some things clearer:\n* that this metric applies to a specific pairing of event types b and h and a time interval (I realise this is wrapped up in the tr variable but I found the notation pretty opaque).\n* that the tr, sp, hc, and gc terms depend on $\\Delta$ but really more fundamentally depend on the mean relative time $\\tau$. Later the authors refer to  $gc$ as a function of $\\tau$ so this seems to me to be the fundamental variable. Would it be better to define tr as $(ev_b, ev_h, \\tau, \\eta)$ where $\\Delta$ is derived from these values?\n* that this metric is defined over an event set (by which I meant that the event set could appear formally in notation)\n* b(q;tr) and n(b(tr)) mean two slightly different things, the latter is a count over b(q;tr) for all q in the event set I think. Similarly replacing b with h.\n\nThe definition of TR Detection and TR Query seem to be the authors own, and so the fairness of the experimental comparison is difficult to judge. The performance is impressive on first appearance, but I am left wondering how to interpret this. The paper would be greatly strengthened by a clear justification of why this is the fairest available comparison between methods.\n\nQUESTIONS:\nWhy does the queried time set $\\mathbb{T}_r = \\{1-T_a,\\ldots, 0, \\ldots, T_a-1\\}$ mean that you only query one side of an asymmetric pair of events? Surely this means that the total number of relative time durations is $2T_a -2$, meaning that any pair of events $(ev_i, ev_j)$ at times $t_i$ and $t_j$ would be queried in both directions so long as $t_i\\neq 0$ or $t_j \\neq T_a$ and vice versa.\n\nWhat is the measure of TR Query (denoted r in table 1)?\n\nHow is the data in the experiments split into training and testing parts?\n\nWhy are there only 1 element per periodic time delay in the $\\omega$ vectors? Does this mean that you can only discover 1 TR (body-head type) per characteristic time delay $\\tau$? What if two distinct TRs (body-head types) had the same characteristic time delay?\n\nLIMITATIONS:\nThe authors could be clearer about the limitations of this work. In particular, certain choices such as predefined fixed length interval factors for detection, predefined thresholds for detection, point estimates for TR Queries may be necessary to make the method tractable, but they do introduce a certain arbitrariness to the prediction frame work.\nEqually, the fact that the experiments are comparing the performance of their method in optimising against an objective that motivated the design of their approach is potentially problematic too. The ground truth general confidence of a TR is the best performing time delay for a pair of events, which is the same as the objective for their detection method. The argmax of this is both the ground-truth for TR Query and the training objective for TR Query on their model. The authors should justify why this is a fair comparator, and possibly develop alternative independent measures for evaluation too.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper defines the tasks of temporal regularity (TR) detection and query and their evaluation metrics, and proposes Noether Embedding (NE) that enables encoding TRs from limited event samples and rapid retrieval of TRs in constant time complexity. NE possesses time-translation symmetries of temporal regularities that are indicated by conserved local energies in the embedding space. NE is evaluated on ICEWS14, ICEWS18, and GDELT datasets, and achieves about double F1 scores for detecting valid TRs and over 10 times confidence scores for querying TR intervals compared with baseline embeddings with additional calculation. NE is further shown to be useful for social event prediction and personal decision-making scenarios.\n\nSTRENGTHS:\n- This work defines a pair of particularly novel tasks: temporal regularity detection and query. Both are critical to human-level intelligence.\n- It proposes Noether Embedding which for the first time enables learning temporal regularities directly from events and rapid retrieval of the regularities.\n- NE achieves promising performances on ICEWS14, ICEWS18, and GDELT datasets, and is shown to be useful for social event prediction and personal decision-making scenarios.\n\nWEAKNESSES:\nPlease see Questions for doubts to be addressed in the next version of the paper.\n\nQUESTIONS:\n1. L279 argues that NE is qualitatively different from rule mining methods in that a search method may require fixing different relative time points before mining rules, while NE enables direct fitting and plotting of an approximate validity distribution of relative time points. But L150-151 suggests that users will select a set of relative time points. So how does NE differ? If users provide their selected set of relative time points, why couldn't we detect time regularities by counting? \n2. What structural biases do NE assume? Could you provide theoretical analyses for the capacity limitation of NE?\n3. Could you add to Section 3.2 the geometric interpretation of the Hardmard product in Eq. (4)?\n4. L116: Is the $\\omega$ vector a hyperparameter or learned, and what is the semantic implication of it being certain values?\n5. L122: Should $t^\\prime$ exclude all $\\hat{t}$ where $(ev,\\hat{t})$ is in the dataset?\n6. To confirm understanding, the second equal sign is L134 always hold true, and by construction, g being invariant to t is always true. Right?\n\nLIMITATIONS:\nThe Conclusion section mentions limitations. The paper doesn't seem to discuss negative societal impacts. It could say something about when the method may fail to detect or query temporal regularities in applications and potential consequences.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper defined the complementary problems of TR detection and TR query, formulated their evaluation metrics, and adopted classic datasets for evaluations. Towards the TR problem, this paper proposed Noether Embedding (NE), which for the first time, enabled both the data-efficient formation and rapid retrieval of TRs simply through embedding each event sample.\n\nSTRENGTHS:\n1) This paper targeted at detecting and encoding temporal regularities (TRs) in events, which are of great importance.\n\n2) This paper introduced Fourier basis to intrisincally model temporal regularities.\n\nWEAKNESSES:\n1) The authors overclaimed the first contribution.\nThe problems of TR detection and TR query are closely related to temporal assoication mining, with different evaluation metrics. However, these metrics were commonly adopted in other fields. So the problems were not new.\n\n2) No strong baselines were compared in the experiments.\nAll compared baselines were not targeted this problem or had different assumptions, and therefore did not support the superiority of NE.\n\n3) The paper is poorly presented. \nThe tasks were not presented clearly.\nIn addition this paper defined too many symbols but did not well explained them, which made readers lost.\n\n4) The connection between Noether’s theorem and the proposed method is unclear and weak.\nAs an evidence, keywords such as \"noether’s theorem\" and \"conservation law\" only appeared in two paragraphs and I can't find how \"noether’s theorem\" motivated the formulation of Noether Embedding in a concrete description.\n\nQUESTIONS:\n1) The paper emphasized \"distributed\" representation was an advantage of NE. Is it a general property of knowledge graph embedding?\n\n2) The paper also emphasized the proposed TR formulation was \"data-efficient\". Is this property brought by Fourier expansion? In opposite, what is the drawback or limitation?\n\n3) How to perform the tasks with NE during inference? It seemed unclear in the paper.\n\nLIMITATIONS:\n1) The limitation of Fourier expansion for real-world temporal regularities was not discussed.\n\n2) Potential negative societal impact was not discussed, e.g. causality of TR, privacy issue brought by TR detection.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

comments: SUMMARY:
This paper introduced a new task, $\textit{temporal regularity mining}$,  and proposed a Noether Embedding  to rapidly retrieval TR.

STRENGTHS:
- The argued temporal regularities sound interesting.
- Good writing.

WEAKNESSES:
- The proposed temporal regularity mining is not a new task with a new paradigm, which the existing methods cannot well tackle.  I think it is more like an extended task based on event embedding using temporal knowledge graph data.
- All existing event embedding methods can be used to tackle the proposed TR mining task. However, there is no discussion to clarify why these existing methods cannot well deal with this new task.
- Mining general TR sounds interesting and useful, but the mined TR is not general. In other words, it is more like a temporal association between events. For example, as shown in the supplementary material, (China, Appeal for diplomatic cooperation, Malaysia) $\rightarrow$ (South Korea, Express intent to settle dispute, China). An ideal TR should be no matter who does the body event, and then the head body will occur soon, which is invariant to the time.
- The temporal range between the body and head events is large, from several days to several years. So how to set the $\tau, \eta$?  Please discuss this detail.
- More experiments on traditional tasks in event embedding, such as event prediction, should be conducted to further demonstrate the effectiveness of the proposed method.

QUESTIONS:
Please see the Weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Noether Embedding (NE), a new model for efficient learning of temporal regularities with event embeddings. Experiments conducted on three datasets show the superior performance of this work compared to classic embeddings in detecting valid TRs and querying TR intervals.\n\nSTRENGTHS:\n1. The authors introduce Noether Embedding (NE), a new model that enables the data-efficient formation and rapid retrieval of temporal regularities simply through embedding each event sample. NE possesses the intrinsic time-translation symmetries of TRs, which facilitates TR encoding insensitive to sample size and TR retrieval in constant time complexity. This is a novel approach that has not been explored in previous works.\n2. The authors formally define complementary problems of TR detection and TR query, formulate their evaluation metrics, and evaluate NE on classic ICEWS14, ICEWS18, and GDELT datasets. This is a rigorous evaluation of the proposed model and provides evidence of its superior performance compared to classic embeddings with additional calculation efforts.\n3. The paper is well-written and clear, with a concise abstract and introduction that provide a good overview of the problem and the proposed solution. The authors also provide detailed explanations of the model and the evaluation metrics.\n\nWEAKNESSES:\n1. In Table 2 of the Appendix, the recall rate of NE is lower than that of TASTER. It would be better if the reason could be explained.\n2. It should be further explained why gc(τ) in line 79 has different input from gc(tr) in formula 3.\n\nQUESTIONS:\n1. Table 2 in the Appendix shows that NE has extremely high accuracy, while the recall rate is not the highest. May I know the reason for this phenomenon?\n2. Could you further explain why NE has an overwhelming advantage over all baselines?\n\nLIMITATIONS:\nThe author proposes the limitations of NE and solutions in terms of storage efficiency in lines 307-311. \n1. In the future, methods will be explored to store event occurrences and time patterns in different regions to improve the storage efficiency of NE. \n2. Future research will explore methods to compose 1-1 TRs into graphical and hierarchical event schemas and combine NE with deep learning and reinforcement learning methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce the concept of temporal regularities (TR), which indicates temporal associations invariant to time shifts between events. The authors claim that existing models are lack of the TR learning capability. Based on this idea, the authors define two tasks, TR detection and TR query, as well as their evaluation metrics. They also further develop a new framework to learn a set of event representations regularized by fixed time embeddings. Experiments on several benchmark datasets demonstrate the effectiveness of the proposed framework and its superiority on TR learning compared to existing methods.\n\nSTRENGTHS:\n* The proposed temporal regularity problem is important and many existing models are lack of such TR learning capabilities.\n* The proposed TR detection and TR query tasks are well-designed. The corresponding evaluation metrics are also reasonable.\n* The proposed solution is simple and effective. Experiments demonstrate significant improvement on TR compared to previous methods.\n\nWEAKNESSES:\n* The event embedding implementation part is not very clear. Details are insufficient for reimplementation.\n* Font sizes in Figure 5 and 6 are too tiny.\n\nQUESTIONS:\n* In Line 166-168, each (s,p,o) combination corresponds to a specific event embedding. From the authors' code, it seems like the authors have tried different embedding strategy, such as encoding s, p and o separately, or encoding (s,p,o) as a whole. What's your choice among these different implementations?\n* In formula 7, the conserved local energy is only relevant two events. If the relative time is conditioned on other events, could this framework handle it?\n* I'm interested how much have the model learnt from statistic prior. Have you tried to compute the averaged relative time for every event pair in the training set and then use such averaged relative time for testing?\n\nLIMITATIONS:\nThe authors have adequately address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}