Transformer Embeddings of Irregularly Spaced Events and Their Participants
Authors: Hongyuan Mei
Chenghao Yang
Jason Eisner
Venue: ICLR 2022 Poster (2022)
Keywords: irregular time series
generative Transformers
neuro-symbolic architectures
logic programming

Abstract: The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-specific pattern-matching rules); future events are then conditioned on the database contents. They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and possible event has a time-varying embedding that is derived from its symbolic provenance. 

In this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).

Published as a conference paper at ICLR 2022
TRANSFORMER EMBEDDINGS OF IRREGULARLY SPACED EVENTS AND THEIR PARTICIPANTS
Chenghao Yang
Hongyuan Mei Toyota Tech. Institute at Chicago hongyuan@tt edu
Jason Eisner
Dept. of Computer Science
Dept. Computer Science Johns Hopkins University
Columbia University
yangal a 99119966 9960
I jasonEcs. jhu.edu
ABSTRACT
The neural Hawkes process (Mei & Eisner, 2017) a generative model of ir- regularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates deductive database of facts (via domain-speciff pattern-matching rules); future events are then conditioned on the database con- tents. They show how to convert such symbolic system into neuro symbblic continuous-time generative model, in which each database fact and possible event has a varying embedding that derived from its symbolic provenance.
this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter iitention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).
better than that of the original models as well as (where
INTRODUCTION
It has recently become common model event sequences by embedding each event into R.,. Event sequences are ubiquitous in roal-world applications, such as healthcare, finance, education, commerce, gaming, audio, news, security, and social media. Event embeddings could be used in a variety of downstream applied tasks, similar to word token embeddings in BERT (Devlin et al., 2018).
In this paper, we embed each event using attention positional encodings so as to consider their timing. To build a left-to-right generative model, we also embed possible events future times in exactly the same way, and use their embeddings predict their instantaneous probabilities at those times.
the previous events, using continuous-time
Attention-basee -ased models (Vaswani et al., 2017) have already become extremely popular for generative modeling of discrete-time sequences, such as natural-language documents (Radford et al., 2019; Brown al., 2020) and proteins (Rao et al., 2021). As we confirm here, they are also effective for modeling sequences that are irregularly spaced in continuous time, even in lower-data regimes.
Our past work on modeling event sequences (Mei Eisner, 2017; Mei et al., 2019; 2020a;b) used neural architectures based on LSTMs (Hochreiter & Schmidhuber, 1997). That is, predictions at time were derived from recurrent encoding of the sequence of timestamped events at times < t. However, an attention- based (Transformer-styyle) architecture has three potential advantages:
Transformer does not summarize the past. Our predictions at time can examine an unboundedly large representation of the past (embeddings in Id of every event before f), not merely fixed- dimensional summary that was computed greedily from left to right (an LSTM's state at time t).
Transformer's computation graph is broader and shallower. The breadth makes it easier to learn long-distance influences. The shallowness does make it impossible to represent inherently deep concepts such as parity (Hahn, 2020), but enables greater parallelism: the layer-& embeddings can be computed in parallel during training, as they depend only on layer & - 1 and not on one another.
The Transformer architecture is simpler and arguably more natural, while remaining competitive in our experiments. To model the temporal distribution of the next event, all of our models posit embeddings of possible future events that depend on the future event's 's time t. To accomplish this, Mei & Eisner (2017) had to stipulate arbitrary family of parametric decay functions f, and

---

Published as a conference paper at ICLR 2022
6-2
2-2
forum
Jorum
forum
forum
t-1
g+1
forum
forum
forum
forum
forum
forum
forum
forum
t,
(a) A- NHP
(b) A-NDTT
Figure I: These figures show how embeddings in the model flow through layers (bottom to top) and through time (left to right). There two possible event types, and f. which represent email messages. At the upper right corner of each figure, we obtain their modeled intensities at a certain time t, Xe(t) and Ag (t). based on the embeddings of the three previous, irregularly spaced observed events. This requires embedding and f at time& t as if they were observed. If either one actually occurs at time t, we will keep its embeddings, which will then affect embeddings of events times Figure (a) shows the basic model of section 3, in which each event's 's embedding layer é depends (->) on all preceding events layer 1. (The dashed arrows --> reflect residual connections.) Section explains that the -> f influences can be prevented by dropping the rule i <- e. Figure (b) shows an A-NDTT model from section 5, in which the company forum's 's embedding at layer depends (->) on all preceding events at layer f - 1 (via <- rules). The events or possible events at layer f do not depend directly on preceding events; instead, their embeddings at time derived (->) from the forum's 's embedding at time (via :- rules).
the neuro-symbolic framework of Mei et al. (2020a) required a complex method for pooling the parameters of these decay functions. But in our present method, no decay functions are required to allow embeddings and probabilities to drift over time. The embeddings are constructed "from scratch" at each time t simply by attending to the set previous events, using t-specific query vec- tors that include continuous positional embedding of t. As increases, the attention weights over the previous events vary continuously with f, so the embeddings and probabilities do so as well.
We present series of increasingly sophisticated methods. Section 2 explains how to embed events in context (like continuous- time BERT). Section 3 turns this into generative point process model that can predict the time and type of the next event (like continuous-time GPT). In section 4, allow domain expert to write simple rules that control attention, constraining which events can "see" which previous events and with what parameters. Finally, section 5 allows the domain expert write more complex rules, using our previously published Datalog through time formalism (Mei et al., 2020a). These rules allow events interact with a symbolic deductive database that tracks facts over time so that the neural architecture does not have to learn how to do so. As in Mei et al. (2020a), we define varying embeddings for all facts in the database and all events that are possible given those facts, using parameters associated with the rules that established the facts and possible events.
In the end, we arrive at attention- versions of the NHP (Mei & Eisner, 2017) and NDTT (Mei et al., 2020a) frameworks, which we refer to as A-NHP (section 3) and A-NDTT (section 5). We evaluate them in section 7, showing comparable or better accuracy. We release our code.
2 CONTINUOUS-TIMA TRANSFORMED FOR EMBEDDING EVENTS
Suppose we observe I events over fixed time interval [0, T). Each event is denoted mnemonically as eat (i.e., "type at time t²), where e E & is the type of event (drawn from a finite set E). The observed event sequence is e e2 0222,, ..., E, where 0 < 61 < l2 <... < t < T. For any event eat, we can compute an embedding [e][(t) E RD by attending to its history H(eat) set of relevant events. (For the moment, imagine that H(eal) consists of all the observed events e,al,)) More precisely, [t]](t) is the concatenation of layer-wise embeddings [e] (0) (t), (e..)))))....)))))
For e > 0, the layer-E embedding of eat is computed as
(f(as))) a (fos, eat)
(e)
(L) def Le] (1)))))
+ tanh
Σ FraEE ( (fes, eas, exa) (1)
residuual connection
where the unnormalized attention weight on each relevant event fas E H(kat)
a (fas, cat) del exp
k ( (fors) T q (eat)) R
(2)
(D
2

---

Published as a conference paper at ICLR 2022
In layer e, v(,, (()), and q0 are known as the value, key, and query vectors and are extracted from the layer-(e 1) event embeddings using learned layer-specific matrices v(O) K(() (():
(eot) del vi [1:[[]: I4110))) [e] kn (eat) del k( [el
atd (eet) def ot [1:11::11]0 (0) (3b)
(3a)
(3c
As the base case, [c] [e][0] (t) [e] is learned embedding of the event type [t]] denotes an embedding of the time I. We cannot learn absolute embeddings for all real numbers, so we fix
[[t]a = sin(t/ where 0 S d < D are the dimensions and our choices of m, M are explained in Appendix A.
(MM) $) )) if d is even
[e]l = cos(t/ (m (ettt/(n-r
( (5M) d=1 )) if d is odd
(4)
Crucially, to compute the layer-l embedding of an event, equations (1)-(3) need only the layer-(é-) 1) embeddings of the relevant events in its history. This lets compute the layer-& embeddings of all events in parallel. Note that equations (1) are simplifications of the traditional Transformer, since this ablation performed equally well in our pilot experiments (see Appendix A).
The set of relevant events H(ewt) could be defined in a task-specific way. For example, to pretrain BERT-like embeddings (Devlin et al., 2018), we might use corrupted version of (c1 e, e in which some e,at, have been removed or replaced with maskøt,. Such embeddings could be pretrained with a BERT-like objective and then fine-tuned to predict properties of the observed events.
3 GENERATIVE MODELING OF CONTINUOUS-TINE EVENT SEQUENCES
In this paper, we focus on the task of predicting future events given past ones. At any time t, we would like to know what will happen at that time, given the actual events that happened before t. Our generative model analogous to a Transformer language model (Radford al., 2019; Brown et al., 2020), which, at each time & E N, defines a probability distribution over the words in the vocabulary.
In our setting, however, E R. With probability 1, nothing happens at time t. Each possible event in our vocabulary has only infinitesimal probability of occurring at time t. We write this probability as (t)dl where (L) E R+ called the (Poisson) intensity type-e events at time More formally, the probability of such event occurring during [t. + e) approaches Xe (t) c as + 0.
Thus, our modeling task is to model de (t) (as in, c.g., Hawkes, 1971; Du et al., 2016; Mei & Eisner, 2017). We model Ae (t) function of the top-layer embedding of the possible event est:
(t( def soft w [1; (e]=(()] Te) where softplus(2; 7) = - log(1 + exp(x/T)) > (5) with learnable parameters w. and To > 0. We do this separately for each possible eal, computing the embedding [ using equations (1)-(3). The softplus transfer function is inherited from the neural Hawkes process (Mei & Eisner, 2017). To ensure that our model is generative, we compute [c][(t from only previous events. That is, H(eat) in equation (1) may contain any or all of the previously generated events efat, for t; < t, but it may not contain any for which &; > t. We call this model the attentive Neural Hawkes process, or A-NHP, and evaluate in section
Our model's 's likeliool has the same form as for any autoregressive multivariate point process:
EL de (4:) / Σ (t)dt
E
(6)
1=0
Derivations of this formula can be found in previous work (e.g., Hawkes, 1971; Liniger, 2009; Mei & Eisner, 2017). We can estimate the parameters by locally maximizing the log- likelihooood (6) by any stochastic gradient method. Intuitively, each log de, (tx) is increased to explain why the observed event e; happened at time t, while possible type e E (1,..., E} ever happened at other times.
de (t)dt is decreased to explain why no event of any
Appendix D gives training details, including Monte Carlo approximations to the integral in equa- tion (6), as well as noting alternative training objectives. Given the learned parameters, we may wish to sample from the model given the past history, or make minimum Bayes risk prediction about the next event. Recipes can be found in Appendix E.
Notice that equation (5) is rather expensive compared to previous work, since it computes deep embedding of the possible event eat just for the purpose of finding its intensity- and the algorithms of Appendices D-E require computing the intensities of many possible events. Appendix A offers speedup that shares embeddings among similar events, but it also explains why different events may sometimes have to be embedded differently support the selective attention in sections 4-5 below.
3

---

Published as a conference paper at ICLR 2022
4 MULTI-HEAD SELECTIVE ATTENTION
We now present a simple initial version of selective attention. As in a graphical model, not all events should be able to influence one another directly. Consider a scenario with two event types: means that Eve emails Adam, while means that Frank emails Eve. As Frank does not know when Eve emailed Adam, past events of type cannot influence his behavior. Therefore, H(( should include past events of type f but not e, so that the embedding of fal and hence the intensity function >ę (1) pay zero attention to g events. In contrast, H(eat) should still include past events of both types, since both are visible to Eve and can influence her behavior.
We describe this situation with the edges f f.e < e. <- f. These are akin to the edges in directed graphical model. They specify the sparsity pattern of the influence matrix (or Granger causality matrix) that describes which past events can influence which future events. There is a long history of estimating this matrix from observed sequence data (e.g., Xu et al., 2016; Zhang et al., 2021), even with neural influence models (Zhang et al., 2020b). In the present paper, however, we do not attempt estimate this sparsity pattern, but assume Incorporating such domain knowledge into the model can reduce the amount of training data that is needed. Edges like
provided by a human domain expert.
can be regarded simple cases of the NDTT rules in section 5 below.
Such rules also affect how we apply attention. When Eve decides whether to email Adam (eut), we may reasonably suppose that she separately considers the embeddings of the past e events (e.g.. "when were my last relevant emails to Adam?") versus the past f events (c.g., "what have I heard recently from Frank?"). Hence, we associate different attention heads with the two rules that affect e, namely <- and <- f. These heads may have different parameters, so that they seek out and obtain different information from the past via different queries, keys, and values. general, we replace equation (1) with
[e] (0) def [e](
+ tanh
e Σle (()()
(7)
vr (fas) or ( (fus, eat)
(8)
(e 1 L+E at (fus,
where r in the summation ranges over rules e <- .... The history (eatt) contains only those past events fas that rule makes visible to If there are such events, they have small attention weights (are only weakly relevant to eat) as discussed in Appendix A, then rule will contribute little nothing to the sum in equation (7). The attention weights a, and vectors v, are defined using versions of equations (3)-() with r-specific parameters."
In short, each rule looks at the context separately, through its attention weights determined by its own parameters. The rule already specifies symbolically which past events can get nonzero attention in the first place, so it makes sense for the rule to also provide the parameters that determine the attention weights and value projections. Further discussion is given in Appendix A.
ATTENTIVE NEURAL DATALOG THROUGH TIME
Edges such as e <- f can be regarded as simple examples of rules in an NDTT program (Mei et al., 2020a, section 2). We briefly review this formalism and then extend our approach from section 4 to handle all NDTT programs.
Datalog through time (DTT) program describes possible sequences of events, much as a regular expression describes legal sequences of characters. A DTT program for particular domain specifies how each event automatically updates a database, adding or removing facts. this way, the past events EI, ......, e, sequentially construct a database. This database then determines which event types (if any) can happen next: the next event 6,+1 can be f only if 1 is currently a fact in the database.
Thus, event may appear in the database
fact, meaning that the event is possible. We use variables e, f to range over events, but variables g, h to range over any facts (both events and non- events). Literal examples of facts are shown in orange if they are events (e.g.. 1) and in blue otherwise.
A neural Datalog through time (NDTT) program is a DTT program augmented with some di- mensionality declarations (Appendix C). A rule that adds fact to the database now also computes a vector embedding of that fact, or updates the existing embedding the fact was already in the database.
4

---

Published as a conference paper at ICLR 2022
Notice that the dimensionality of the embedded database changes the database grows and shrinks over time. Nonetheless, the model has a fixed number of parameters associated with the fixed set of rules of the NDTT program. As will see, rules can contain variables, allowing small set of rules to model a large set of event types (i.e., parameter sharing).
If is fact in the database at time meaning that event is possible time t, then its embedding (t) determines its intensity > (t) via equation (5), as before. Thus, where a DTT program only de- scribes which event sequences are possible, an NDTT program also describes how probable they are.
Although the ser of database facts is modified only when event occurs, the facts' embeddings time-sensitive and evolve as the events that added them to the database recede into the past. This allows event intensities such as de (t) to wax and wane continously time elapses.
Datalog. We now give details. We begin with Datalog (Ceri et al., 1989). traditional formalism for deductive databases. A deductive database holds both extensional facts, which are placed there by some external process, and intensional facts, which transitively deduced from the extensional facts. A Datalog program is simply a set of rules that govern these deductions:
h 91 .... In says to deduce h at any time when . In are all true (in the database).
single rule can license many deductions. That is because the facts can have structured names, and h. 91.... In can be patterns that match against those names, using capitalized identifiers variables. A model of filesystem properties might have a rule like open (U, D) 1- user (U), group (G), document (D), member (U,G), readable (D, D.) In English, this says that U can open at any time when user U is a member of some group G such that document is readable by G.
Datalog through time. Whenever extensional facts are added or removed, the intensional facts instantly recomputed according to the deductive rules. DTT is an extension in which extensional facts are automatically added and removed when the database is notified of the occurrence of events. This behavior is governed by two additional rule types:
h < S, 91. th < f. 91, .... In says to remove h at any time satisfying the same conditions.
In says to add h at any time when event occurs and the 9, are all true.
Thus, the proposition h is true time (i.e., appears as a fact in the database at time iff either is deduced at time f, or @ h was added at some time s < t and never removed at any time in (s, t).
In our previous example, editing (U,tiiggUU,))) <- Open(U,D)
member (U, (U,G), writeable (U,
records in the database that user is editing D. once they have opened it with appropriate permissions. (As result, edit events might become possible via a deductive rule edit (U, :- editing (U,
Neural Datalog through time. It would be difficult to train a neural architecture to encode thousands or millions of structured boolean facts about the world in its state and to systematically keep those facts up to date in response to possibly rare events. As symrol-ymbolic method, NDTT delegates that task to a symbolic database governed by manually specified DTT rules. However, it also augments the database: iff a proposition I appears as a fact in the NDTT database at time I, it will be associated not only with the simple truth value t rue but also with an embedding [t.]((). This embedding is a learned representation of that fact at that time, and can be trained to be useful in downstream tasks. captures details of when and how that fact was established (the fact's provenance), since it is computed using learned parameters associated with the rules that deduced and/or added it.
For example, a user's s embedding might be constructed using attention over all the past events that have affected the user, via rules of the form user (U) <- This summarizes the user's 's state. Similarily, a document's embedding might be constructed using attention over all the edits to it, considering the editing user's state at the time of the edit: document (D) edit (U, user (U).
Embeddings from NDTT rules. Our goal is to provide new formulas for the embeddings (t). based on Transformer-style attention rather than LSTM-style recurrence. We call this attentive NDTT, or A-NDTT. This gives a new way to map an NDTT program to a neural architecture. The potential advantages for accuracy, efficiency, and simplicity were explained in section 1.
Intuitively, the < rules will govern the "horizontal" flow of information through time (by defining attentional connections as we saw in section 4), while the rules will govern the "vertical" flow of
5

---

Published as a conference paper at ICLR 2022
information at given time (by defining feed-forward connections). These are, of course, the two major mechanisms in Transformer architectures.
Under A-NDTT, the layer-& embedding of hat is
[A]] ((-(( + tanh [1)0(() + Σ (
(9)
which is an augmented version of equation (7). Suppose h is true at time t because it was added by rule (i.e., condition )) Then the summand h ( (I) exists and is computed much in equation (8), now with attention over all "add times" 8. In other words, Hr (heal) in equation (8) includes just those past events fus such that added via some time < t and was never removed at any time in (s,t).
More precisely, when the rule h <- f, 91, event fas, we actually want attention to consider the embedding at time s not just of f. but of the entire add condition f.g1......
In. causes het attend the specific past
Thus, we replace fas with 1f.91.... :.......:eeee throughout equation (8). The attention key of this add condition is defined as k" (1..9.......)) .....))))))) K(O [1:[<>]: (s); [sn] value v ((S,)) In (a) is defined analogously, using a different matrix v(
[=.) (compare equation (3b)). Its attention
The above handles the <- rules. As for the :- rules, the vector (t) in equation (9) sums over all the ways that h be deduced at time (i.e., condition 1). This does not involve attention, exactly follow Mei et al. (2020a, equations (3)-6));
[m] = Σ +dr Wr[1: [t) :o.
(10)
where r ranges over rules, and (..,.....00 In) ranges over all tuples of facts at time / such that h :- I1.... In matches rule r (and thus deduces h time t). The operator 4% is a softmax-pooling operator with a learned inverse temperature B,. If h is not deduced at time f by any instantiation of r, then has no effect on the sum (10), since pooling the empty set with && returns O.
Example. Mei et al. (2020a) give many examples of NDTT programs. Here a simple example to illustrate the use of rules. e means that Eve posts a message to the company forum, while means that Frank does so. Once the forum is created by a create event, its existence is a fact (called forum) whose embedding (called [forum]] always reflects all messages posted far to the forum. Until the forum is destroyed, Eve and Frank can post to it, and the embeddings and intensities of their messages depend on the current state of the forum:
I forum < create. : !forum < destroy.
forum <
forum. forum.
forum < f.
The resulting neural architecture is drawn in Figure 1b. If the company grows from 2 to K employces, then the program needs O(K) rules and hence (K)) parameters, which define how each employee's messages affect the forum and vice-versa. Without the :- rules, we would have to list out O(K3) and hence would need O(K2) parameters, which define how each employee's
rules such as messages affect every other employee's 's messages directly; this case is drawn in Figure la.
Appendix B and Figure 4 spell out an enhanced version of this example that makes use of variables. so that all K employees can be governed by constant (O(1)) number of rules.
Discussion. NDTT rules enrich the notion of "influence matrix" from section 4. Events traditionally influence the intensities of subsequent events, but NDTT <- rules more generally let them influence the embeddings of subsequent facts (and hence the intensities of any events among those facts). Furthemore, NDTT :- rules let facts influence the embeddings of contemporaneous facts.
Each rule r can be seen as defining the fixed sparsity pattern of a large influence matrix, along with parameters for computing its nonzero entries from context at each attention layer. The size of this matrix is determined by the number of ways to instantiate the variables in the rule. The entries of the matrix are normalized versions of the attention weights a,, The influences of different <- rules are combined by equation (9) and are modulated by nonlinearities.
Overall, (A-)NDTT models learn representations, much like pretrained language models (Peters et al., 2018; Radford et al., 2019). They learn continuous embeddings of the facts in discrete database, us- ing neural architecture that is derived from the symbolic rules that deduce these facts and update them

---

Published as a conference paper at ICLR 2022
in response to events. The facts change at discrete times but their embeddings change continuously. We train the model so that the embeddings of possible events predict how likely they are to occur.
RELATED WORK
Multivariate point processes have been widely used in real-world applications, including document stream modeling (He et al., 2015; Du et al., 201 learning Granger causality (Xu et al., 2016: Zhang et al., 2020b; 2021), network analysis (Choi et al., 2015; Etesami et al.. 2016), recommendation systems (Du al., 2015b), and social network analysis (Guo al., 2015; Lukasik et al., 2016).
Over the recent years, various neural models have been proposed to expand the expressiveness of point processes. They mostly use recurrent neural networks, or LSTMs (Hochreiter & Schmidhuber, 1997): in particular Du et al. (2016); Mci & Eisner (2017); Xiao et al. (2017a:b), Omi et al. (2019); Shehur al. (2020); Mei et al. (2020a); Boyd et (2020). Models of this kind enjoy continuous and infinite state spaces, well as flexible transition functions, thus achieving superior performance on many woald datasets, compared to classical models such as the Hawkes process (Hawkes, 1971).
The Transformer Hawkes process (Zuo et al., 2020) and self-attentive Hawkes process (Zhang et al., 2020a) were the first papers to adapt generative Transformers (Vaswani et al., 2017; Radford et al., 2019; Brown et al., 2020) to point processes. The Transformer architecture allows their models to enjoy unboundedly large representations of historics, as well as great parallelism during training (see ① and @ in section 1). As section 3 discussed, both models- well subsequent irtentton-based models (Enguehard et al., 2020; Sharma et al.,
the intensity (t from [s) where o fas is the latest actual event before t. (The THP takes (e(t) to be softplus function of w_ [1; [/][ The SAHP defines (-).) as a function that exponentially decays toward an asymptote, computing the 3 parameters of this function from and [s]()). In contrast (see @ in section 1). our model derives (t) from [e]t-the embedding of the possible event ext, which is computed using e- and t-specific attention over all past events. Zhu et al. (2021, section 3.1) independently proposed this approach but did not evaluate it experimentally.
EXPERIMENTS
On several synthetic and real-world datasets, evaluate our model's held-out log-likelihood, and its success at predicting the time and type the next event. We compare with multiple strong competitors. Experimental details not given in this section can be found in Appendix F.
We implemented our A-NDTT framework using Py PyTorch (Paszke et al., 2017) and py pyDatalog (Carbonell et al., 2016), borrowing substantially from the public implementation of NDTT (Mei et al., 2020a). We also built a faster, GPU-friendly PyTorch implementation of our more restricted A-NHP model (see section 7.1 below). Our code and datasets are available https: /github. anhp-andtt. andtt.
com/yangen / hahpl////ppp-aaaee
For the competing models, made use of their published implementations! References and URLs are provided in Appendix F.2.
7.1 COMPARISON OF DIFFERENT TRANSFORMER ARCHITECTURES
We first verify that our continuous- Transformer is competitive with three state-of-the- thet---rrt- art neural event models. The four models we compare are
Transformer Hawkes Process (THP) (Zuo et al., 2020). See section 6.
Self-Attentive Hawkes Process (SAHP) (Zhang et al., 2020a). See section 6.
Neural Hawkes Process (NHP) (Mei & Eisner, 2017). This is not an attention based model. At any time t, NHP uses a continuous-time LSTM to summarize the events over (0.t) into a multi- dimensional state vector, and conditions the intensities (t of all event types on that state.
Attentive Neural Hawkes Process (A (A-NHP) This is our unstructured generative model from section 3. Since this model does not use selective attention, we speed up the intensity computations by defining them in terms of single coarse event type, described in Appendix A. Thus, each event intensity
On some datasets, replicated results are different from their papers. We confirmed that our results are correct via personal communication with the lead authors of Zhang et al. (2020a) and Zao al. (2020).
7

---

Published as a conference paper at ICLR 2022
Xe (t) is computed by attention over all previous events, where the attention weights are independent of e. This varameter-sharing mechanism resembles the NHP, except that we now use a Transformer in place of an LSTM.
In a pilot experiment, we drew sequences from randomly initialized models of all 4 types (details in Appendix F.1.1), and then fit all 4 models each of these 4 synthetic datasets. We find that NHP, SAHP, and A-NHP have very close performance on all 4 datasets (outperforming THP, especially at predicting timing, except perhaps on the THP dataset itself);
Figure 5 in Appendix F.I.1 for results. Thus, A-NHP is still satisfactory choice even when it is misspecified. This result is reassuring because A-NHP has less capacity in some ways (the circuit depth of a Transformer is fixed, whereas the circuit depth of an LSTM grows with the length of the sequence) and excess capacity in other ways (the Transformer has unbounded memory whereas the L.STM has finite-dimensional memory).
10
Evaluation re- sults (smaller is better) with 95% bootstrap confidence intervals on the two real- world datasets, comparing THP. SAHP, and NHP with A-NHP model. RMSE evaluates the predicted time the next event (root mean squared crror), while error rate evaluates its predicted type given its time.
Figure 2:
30
nogative log Iikeihood
RNSE
error rota
(a) MIMIC-II
35
53
233
S01 311
20
THP
THF
54/1
X-1P1
TH?
SAnT
A-1P7
neguttivn
(b) StackOverflow
We then fit all 4 models to the following two benchmark real-world datasets.
MIMIC-II (Lee et al., 2011). This dataset a collection of de-identified clinical visit records of Intensive Care Unit patients for 7 Fach patient has sequence of hospital visit events, and each event records its time stamp and discase diagnosis.
StackOverflow (Leskovec & Krevl, 2014). This dataset represents two years of user awards question-answwerigg website: each user received a sequence of badges (of 22 different types).
On MIMIC-II data (Figure 2a), our A-NHP is always co-winner on each of these tasks; but the other co- co-winner (NHP or THP) varies across tasks. On StackOverflow data (Figure 2b), our A-NHP is clearly winner on 2 out of tasks and is tied with NHP on the third. Compared NHP, A-NHP also enjoys a computational advantage, as discussed in sections and 2. Empirically, training an A-NHP only took fraction of the time that needed to train NHP, when sequences reasonably long. Details can be found in Table 2 of Appendix F.3.
7.2 A-NDTT VS. NDTT
Now we turn to the structured modeling approach presented in section We compare A-NDTT with NDTT on the RoboCup dataset and IPTV dataset proposed by Mei et al. (2020a). In both cases, we used the NDTT program written by Mei et al. (2020a). The rules are unchanged; the only difference is that our A-NDTT has the new continuous-time Transformer in lieu of the LSTM architecture. We also evaluated the unstructured NHP and A-NHP models on these datasets.
RoboCup (Chen & Mooney, 2008). This dataset logs the actions (e.g., kick, pass) of robot soccer players in the RoboCup Finals 2001-2004. The ball is frequently transferred between players (by pass- ing or stealing), and this dynamically changes the set of possible event types (e.g.. only the ball pos- sessor can kick or pass). There are K 528 event types over all time, but only about 20 of them are possible at any given time. For each prefix of each held-out event sequence, used minimum Bayes risk to predict the next event's time, and to predict its participant(s) given its time and action type.
If' we are not taking enough Monte Carlo samples to get a stable estimate of ligelikklhhood then this will appropriately be reflected in wider error bars. This is because when computing bootstrap replicate, we recompute Monte Carlo estimate of the log- likelihood of each sequence. Hence, bootstrap confidence intervals take care include the variance due the stochastic evaluation metric. For the Monte Carlo settings we actually used (Appendix D). this amounts to about 1% the width of the error hars.
For these datasets, we used the preprocessed versions provided by Mei & Eisner (2017). More details about them can be found in Appendix 1.1.2.
8

---

Published as a conference paper at ICLR 2022
IPTV (Xu et al., 2018). This dataset contains records of 1000 users watching 49 TV programs the first 11 months of 2012. Each event has the form watch (U,P). Given each prefix the test event sequence,
attempted to predict the next test event's s time t, and to predict its program given its actual time and user U.
110
Figure 3: Evaluation re- sults with 95% bootstrap confidence intervals" on RoboCup and IPTV datasets. Evaluation meth- ods are the same as in Figure 2. Note that the training objective was log- likelihood.
50-
ANDIT
M-P
MITT
(a) RoboCup
23
13
100
18
10
nogative llkcihood
RMSE
emorr rate
(b) IPTV
The Robocup results are shown in Figure 3a. As section 7.1, we find that A-NHP performs better than NHP on all the evaluation metrics; on log-likelihood and event type prediction, A-NHP is sig- nificantly better (paired permutation test, p < 0.05). We now inject domain knowledge into both LSTM and Transformer approaches, by deriving architectures based the RoboCup NDTT program (which specifies, for example, that only the ball possessor can kick or pass). The resulting models NDTT and A-NDTT- are substantial and significant improvements, considerably reducing both the log-likelihood and the very high error rate on event type prediction. NDTT and A-NDTT are not sig- nificantly different from each other: since NDTT already knows which past events might be relevant, perhaps it is not sorely in need of the Transformer's ' ability to scan an unbounded history for relevant events. Appendix F.5 includes more results of A-NDTT vs. NDTT broken down by action types.
Additionally, while A-NDTT does not improve the overall accuracy for this particular NDTT program and dataset, does achieve overall comparable accuracy with a simpler and shallower architecture (ø-@ in section 1). Like other Transformers, the A-NDTT architecture could be trained on GPU with parallelism, as outlined in Appendix F.4 (future work).
The IPTV results are shown in Figure 3b. In this case, the log-likelihood of NHP can be substantially and significantly improved either by using rules (as for Robocup) or by using attention, or both. The error rate predicting the next event type is again very high for NHP, and is substantially and significantly reduced by using rules-ale aules-although not as much under the A-NDTT architecture as under the original NDTT architecture.
8 CONCLUSION
We showed how to generalize the Transformer architecture sequences of discrete events in contin- uous time. Our architecture builds up rich embeddings of actual and possible events at any time t, from lower-level representations of those events and their contexts. We usually train the model so that the embedding of a possible event predicts its intensity, yielding a flexible generative model that supports parallel computation of log- likelihooood We showed in section 7.1 that outperforms other Transformer based models on multiple real-world datasets and also beats or ties them on multiple synthetic datasets.
We also showed how to integrate this architecture with NDTT, neural-symbolic framework that automatically derives neural models from logic programs. Our attention- based modification of NDTT has shown competitive performance, despite having simpler and shallower architecture. Our code and datasets are available at https: ///gttttbb//////// com/
wtt.uuu...ooooagggaaaaaaaa yangalann 23/ anha-andheee
'Wiile NDTT still uses a fixed- dimeenssiii history ① in section 1-the dimensionality is often very high, as the NDTT's state consists of embeddings of many individual facts. Moreover, each fact's NDTT embedding computed by rule-specific LSSMs that see only events that relevant to that fact, there is danger that intervening irrelevant events will displace the relevant ones in the fxced-immensional L.S states.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported in part by the National Science Foundation under Grant No. 1718846. We thank Bloomberg for Data Science Ph.D. Fellowship to the second author. We thank Minjie Xu for the suggestion of developing a Transformer version of NDTT. We thank the anonymous ICLR 2022 reviewers for discussion and for pointing out additional related work.
REFERENCES
Ba, I.., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv: arXiv: 1607.06450, 2016.
Boyd, A., Bamler, R., Mandt, S., and Smyth, P. User-dependent neural sequence models for continuous-time event data. Advances Neural Information Processing Systems (NeurIPS), 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Carbonell, P., jedouet, Alves, H. C., and Tim, A. pyDatalog, 2016.
Ceri, S., Gottlob, G., and Tanca, L. What you always wanted to know about Datalog (and never dared to ask). IEEE Transactions on Knowledge and Data Engineering, 1989.
Chen, D. L. and Mooney, J. Learning to sportscast: A test of grounded language acquisition. In Proceedings of the International Conference Machine Learning (ICML), 2008
Choi, E., Du, N., Chen, R., Song, L.,, and Sun, J. Constructing disease network and temporal progression model via context-sensitive Hawkes process. In IEEE International Conference on Data Mining (ICDM), 2015.
Devlin, Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04800) 2018.
Du, N., Farajtabar, M., Ahmed, A., Smola, A. J., and Song, L. Dirichlet-Hawkes processes with applications to clustering continuous time document streams. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2015a.
Du, N., Wang, Y., He, N., Sun, and Song, L. sine-sensitive recommendation from recurrent user activities. Advances in Neural Information Processing Systems (NeurIPS), 2015b.
Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguee M., and Song, L. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2016.
Enguehard, J., Busbridge, D., Bozson, A., Woodcock, C., and Hammerla, N. Neural temporal point processes [for] modelling electronic health records. In Proceedings of Machine Learning Research, volume 136, pp. 85-113, 2020. NeurIPS 2020 Workshop on Machine Learning for Health (ML4H).
Etesami, J., Kiyavash, N., Zhang, K., and Singhal, K. Learning network of multivariate Hawkes processes: A time series approach. arXiv preprint arXiv:1603.0433 2016.
Guo, F., Blundell, C., Wallach, H., and Heller, K. The Bayesian echo chamber: Modeling social influence via linguistic accommodation. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, 2015.
Hahn, M. Theoretical limitations of selt-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8, 2020.
Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 1971.
10

---

Published as a conference paper at ICLR 2022
He, P., Liu, X., Gao, J., and Chen, W. DeBERTa: Decoding-enhancee bert with disentangled attention. arXiv preprint 2xX1::2006603664, 2020.
He, X., Rekatsinas, T., Foulds, J., Getoor, L., and Liu, Y. HawkesTopic: A joint model for network inference and topic modeling from text-based cascades. In Proceedingy of the International Conference Machine Learning (ICML), 2015.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 1997.
Kingma, D. and Ba, Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Kitaev, N. and Klein, Constituency parsing with a self-attentive encoder. In Proceedings of the Association for Compulational Linguisfics, 2018.
Lee, J., Scott, D. J., Villarroel, M., Clifford, G. D., Saeed, M., and Mark, G. Open-access MIMIC-II database for intensive care research. In 2011 Anmual International Conference of the IEEE Engineering in Medicine and Biology Society, 2011.
Leskovec, J. and Krevl, A. SNAP Datasets: Stanford large network dataset collection, 2014.
Lewis, P. A. and Shedler, G. S. Simulation of nonhomogeneous Poisson processes by thinning. Naval Research Logistics Quarterly, 1979.
Liniger, T. J. Multivariate Hawkes processes. Diss., Fidgenössischhe Technische Hochschule ETH Zürich, Nr. 18403, 2009.
Lukasik, M., Srijith, P. K., Vu, D., Bontcheva, K., Zubiaga, A., and Cohn, T. Hawkes processes for continuous time sequence classification: An application to rumour stance classification in Twitter, In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2016.
Mei, H. and Eisner, J. The neural Hawkes process: A neurally self-modulating multivariate point process. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Mei, H., Qin, G., and Eisner, J. Imputing missing events in continuous-tiig time event streams. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
Mei, H., Qin, G., Xu, M., and Eisner, J. Neural Datalog through time: Informed temporal modeling via logical specification. In Proceedings of the International Conference on Machine Learning (ICML), 2020a.
Mei, H., Wan, T., and Eisner, J. Noise-contrastive estimation for multivariate point processes. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Mikolov, T., Karafiát, M., Burget, L., Cemocký, J., and Khudanpur, S. Recurrent neural network- based language model. In Proceedings of the Annual Conference the International Speech Communication Association (INTERSPEECH), 2010.
Omi, T., Ueda, N., and Aihara, K. Fully neural network based model for general temporal point processes. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E.. DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in Py Torch. 2017.
Peters, M., Neumann, M., lyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
11

---

Published as a conference paper at ICLR 2022
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S.,, and Rives, A. Transformer protein language models are unsupervised structure learners. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Sharma, K., Zhang, Y., Ferrara, E., and Liu, Y. Identifying coordinated accounts on social media through hidden influence and group behaviours. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2021.
Shchur, O., Bilok, M., and Günnemann, Intensity- free learning of temporal point processes. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, L.., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Xiao, S.. Yan, J., Farajtabar, M., Song. L.. Yang, X., and Zha, H. Joint modeling of event sequence and time series with attentional twin recurrent neural networks. arXiv preprint arXiv: I 1etil17073..2244 2017a.
Xiao, S., Yan, J., Yang. X., Zha, H., and Chu, Modeling the intensity function of point process via recurrent neural networks. In Proceedings of the AAAI Conference Artificial Intelligence. 2017b.
Xu, H., Farajtabar, M., and Zha, H. Learning Granger causality for Hawkes processes. In Proceedings of the International Conference Machine Learning (ICML), 2016.
Xu, H., Luo, D., and Carin, L. Online continuous-time tensor factorization based on pairwise interactive point processes. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2018.
Zhang, Q. Lipani, A., Kirnap, O., and Yilmaz, Self-attentive Hawkes process. In Proceedings of the International Conference Machine Learning (ICML), 2020a.
Zhang, Q., Lipani, A., and Yilmaz, E. Learning neural point processes with latent graphs. In Proceedings of the International World Wide Web Conference (WWW), pp. 1495-1505, 2021.
Zhang, W. Panum, T. K., Jha, S., Chalasani, P., and Page, D. CAUSE: Learning Granger causality from event sequences using attribution methods. Proceedings of the International Conference 2020b..
on Machine Learning (ICML), 2020b.
Zhu. S., Zhang, M., Ding, R., and Xie, Y. Deep Fourier kernel for self-attentive point processes. In International Conference on Artificial Intelligence and Statistics, 2021.
Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer Hawkes process. In International Conference Machine Learning, pp. 11692-11702 PMLR, 2020.
12

---

Published as a conference paper at ICLR 2022
Appendices
A DISCUSSION OF ARCHITECTURAL DETAILS
Simplification. Equation (I) is a simplification of the original Transformer architecture (Vaswani al., 2017). In the original architecture, [t) would be obtained LayerNorm(x (x (FN(0())) where x is the LayerNorm transformation (Ba et al., 2016) of the right-hand side of equation (1), and the nonlinear transform FFN(O) is computed by a learned two-layer feed-forward network.
In our preliminary experiments, we found that the LayerNorm and FFN steps did not help, for simplicity and speed, we omitted them from equation (1) and from the remaining experiments. However, it is possible that they might help on other domains or with larger training datasets, so our code supports them via command-line arguments.
Graceful degradation. Another change to equation (1) (and equation (8) is that when normalizing the attention weights, we included an additional summand of in the denominator." We do this so that when the history H(ext) "rather irrelevant" to ext, the architecture behaves roughly as if H(eat) were the empty set. In equation (I), this means that [e][(O will then be close to [e] (t). Similarly, equation (7) will not be much influenced by rule r if rule finds only events H. (eat) that it considers be "rather irrelevant" eat.
A "rather irrelevant" history is one for which the unnormalized attention weights are small in toto, so that the denominator is dominated by the 1 summand. This may occur, for example, events in the distant past tend to have small attention weights, and the history consist only of old events (and not too many of them). When the history is rather irrelevant, the argument to tanh in equation (1) and the summand
( (t) in equation (7) are close to O; when H(eat) = 0, they are exactly O.
not an important one is that in equation (3), we chose to concatenate [[t] to the rest of the embedding rather than add it (cf. Kitaev & Klein, 2018; He et al., 2020). Furthermore we did so at every layer and not just layer O. The intuition here is that the extraction of good key and query vectors at each layer may benefit from "direct access" to [t] For example, this should make it easy to learn keys and queries such that the attention weight is highest when t A (since for every A R, there exists a sparse linear operator that transforms [t] + [t-
Direct access to time embeddings. Another difference from Vaswani et
Range of wavelengths for time embeddings. Our time embedding [[t] in (4) uses dimensions that are sinusoidal in f, with wavelengths forming a geometric progression from 2mm to 2T.MMM Setting m 1, M 2000 would recover the standard scheme of Vaswani al. (2017) (previously used in continuous time by Zuo et al. (2020)).
We instead set and M from data so that we are robust datasets of different time scales. Part of the intuition behind using sinusoidal embeddings is that nearby times can be distinguished by different values in their shorl-wwveeength dimensions, whereas the wong-whveeength dimensions make it easy to inspect and compare faraway times, since those dimensions are nearly linear on t E [0, M). We therefore take m to be the shortest gap between any two events in the same history,
min etit f
min
|s -0-- s'}.
(11)
as computed over training data, and take M greater than all T' in training data (where each observed sequence is observed over an interval [0. T)).
If we were modeling sequences of words as in Brown et al. (2020). our procedure would indeed recover the values m = 1 and M = 2000 that they used to model text documents. Multiplying or dividing all t values in the dataset by 1000 (c.g., switching between second and millisecond units) would have no effect on the time embeddings, as it would scale m and M in the same way.
Coarse event embeddings for speed. As noted at the end of section 3, the intensity model equation (5) involves a full embedding of each eat. This may be expressive, but it is also expensive. The attention
'Including the summand of is equivalent to saying that eat attends not only to relevant events H(eat) but also to dummy object whose key and value are fixed at O. The dummy object gets unnormalized attention weight of 1, drawing attention away from H(en)).
13

---

Published as a conference paper at ICLR 2022
weight vectors a!)), ..... (5) used to compute this embedding must be computed from scratch for each e and t. Why is this necessary?
Like other neural sequence poddels-bolle RNN-based and Transformer-basse wwe derive the proba- bility that the next sequence element is from an inner product of the form w [1: [H[ where in our equation (5), the role of the history embedding [H[eet]] is played by [e] (t). However, for many previous models, the history embedding does not depend on e, so it can be computed once at each time and reused across all e.
In neural language models, typically all previous events taken to be relevant. [H((=))]] can then be defined as an RNN encoding of the sequence all past events (Mikolov et al., 2010), or alternatively Transformer embedding of the single most recent past event (which depends on the entire sequence of past events). This does not depend on e.
When modeling irregularly spaced events, is not necessarily an integer, and the past events in H(eat) do not necessarily take place at 1, ., 1. must somehow be improved to also consider the elapsed time f, since the most recent past event (Du et al., 2016; Mei & Eisner, 2017; Zuo et al., 2020; Zhang et al., 2020a). So now [[((e))) must look I, but it still independent of e.
Thus, the encoding [?([eat]]
contrast, in sections 4-5, we will allow the more general case where [H((e(]] varies with as well, since NDTT rules determine which past events should be attended to by e. The original NDTT paper essentially defined [*[tt]]] as the state at time of an e-specific continuous-time LSTM, which is updated by just the events that are relevant to e. In our attention-basee approach, we instead define it to be (t), yielding equation (5).
To reduce this computational cost, we can associate each event type e with a coarse event type ē that is guaranteed to have the same set of relevant past events, and replace [t]] with [ (t) in equation (5). (However, equation (5) still uses the fine-grat mnn-grained w..) Now to compute (t).). we only have embed ēat, which is speedup if many of the possible event types are associated with the same ē. In the case where we do not use selective attention, we can get away with using only a single coarse event type for the whole savin-saving a runtime factor of [g] in the cheaper approach. Note that the history H. still uses fine-grained embeddings, so if eat actually occurs, we must then compute (t). (t).
Concatenation vs. summation. Equation (7) uses summation to combine the outputs (8) of different attention heads r. Vaswani al. (2017) instead combined such outputs by projecting their concatena- tion, but that becomes trickier in our setting: different events would need to concatenate different numbers of attention heads r (for just the rules r that can take the form e ...) resulting in projection matrices of different dimensionalities. Especially when NDTT rules can contain variables (section 5 below), it is not immediately obvious how one should construct these matrices or share parameters among them. These presentational problems vanish with our simpler summation approach.
Our approach loses no expressive power: projecting a concatenation of e (t) values, as Vaswani al. would suggest, is equivalent to summing up an r-specific projection of e (. (t) for each r, as we do, where our projection of e (f) (4) has implicitly been incorporated into the projection (3a) that produces vi That is. if we can learn v(()- V in equation (3a), then we can also learn V(O) = PV, where P is the desired projection matrix for rule r. To make our method fully equivalent to Vaswani et al. 's, we would have to explicitly parameterize vi() as a matrix product of the form PV, forcing it to be low-rank.
B NDTT EXAMPLE WITH VARIABLES
The company message forum program in section 5 had only 2 users and forum. However, if the company employs many persons P and has forum for each team T, NDTT rules can use capitalized variables define the whole system concisely, using only ((1) rules and O(1) parameters. Here the possible facts and events have structured names like message (eve,sales (joke), which denotes an event in which employee eve posts a joke to the sales team's forum.
forum (T)
create (T).
message (P, (P,T, T, :- (P))(P), forum (T), content (C). !forum (T) < destroy(T).
8 forum (T) < message (P,T,C).
10
14

---

Published as a conference paper at ICLR 2022
mosss)e(eeesalessjke)))
message (tansaggfrann,ssllssj.o
forum(sales)
forum(sales)
empl(eve)
empi(eve)
t=1
empi(frank)
empi(frank)
member(eve,salee
member(eve,sales
member(frank,sales)
(fmbes((ran,,saee)) content(joke)
content(joke)
messageleve,salssjkke)))
message (fansaggrrann,ssllesjoo
forum(sales)
forum(sales)
empl(eve)
empl(eve)
empl(frank)
empl(frank)
o
member(eve,sales)
(eeeee(sve
member(frank,slle))
(fank,,saan
content(joke)
content(joke)
time
The solid green arrows correspond to instantiations of the attentional rules 8 and 13. They can capture the real-world property that thanks to Eve's 's joke at time is, the sales forum still feels more humorous at time t and Frank, another member of that forum, is still in good mood. This raises the probability Àe that Frank posts his own joke at time where e message (frank, sales, joke). More formally, (e()) determined by the layer- L embedding of Frank's 's possible message cat. In general, the layer-E embedding of this message reflects the layer-(f 1) embeddings of both Frank and the forum at time f, as well as the fact that the possible message is joke. If the message is actually sent (i.e., the possible event actually happens), its layer-f & embedding would in turn affect the layer-(e 1) embeddings of the forum and its readers at times > t. An arrow with multiple inputs means that the input embeddings are concatenated before being transformed into a contribution to the output embedding. Other visual conventions are as in Figure 1. Not all facts, events, or arrows are shown in this drawing.
Figure 4:
This generalizes the previous program, allowing multiple forums and saying that any employee (not just Eve and Frank) can post any type of message to any forum, affecting the embedding of that forum. We could modify rule 9 by adding an additional condition member (P,T), so that employees can only post to forums of which they are members. Membership could be established and tracked by ul
join T) empl(P), forum(T).
member (P, T) join (P,n) ,T,..
13 (P)
message message(P2,T,C) member (P,T)
Rules and 13 ensure that message to a forum affects the subsequent embedding of that forum and also the subsequent embeddings of all employees who were members of that forum when the message was sent. This may affect which employees join which forums in future, and what they post to the forums, as drawn in Figure 4 in the appendices. For further examples, see the full presentation of NDTT in Mei et al. (2020a).
How are variables treated in the computation of embeddings? In equations (7)-(8, r refers to a rule with variables. However, refers to a specific fact, without variables. An instantiation of is a copy of in which each variable has been consistently replaced by an actual value. In our modified version of equation (8), the summations range over all values of (f.9)......0 Jas such that e <- f, 91, In is an instantiation of that added at time (i.e., an instantiation of r such that ʃ occurred at time s and 91..... 9n. were all true at time s). Thus, the attentional competition may consider (f.g1..... ................................ (n. @s values that are derived from many different instantiations of r. Their attentional weights ar are all obtained using the shared parameters associated with rule r. The summation in
MMei et al. (2020a, Appendix B) provide a notation to optionally reduce the amount of parameter sharing. A rule may specify, for example, that each value of variable T should get its own parameters. In this case, we regard the rule as abbreviation for several rules, one for each value of T. Each of these rules corresponds to different equation (7), and corresponds separate attention head (8) that sets up its own attentional
15

---

Published as a conference paper at ICLR 2022
equation (7) ranges only over rules r with at least one instantiation that adds eat, so it skips rules that are irrelevant to e.
C PARAMETER DIMENSIONALITY SPECIFICATION FOR A-NDTT
In this section we discuss the dimensionality of the fact embeddings []t][(t) in section 5.
As in the original NDTT paper Mei et al. (2020a), the type of a fact in the database is given by its functor (forum, member, create, etc.). All facts of the same type have embedding vectors of the same dimensionality, and these dimensionalities are declared by the NDTT program.
This is enough to determine the dimensions of the parameter matrices associated with the deduction rules (Mei et al., 2020a). How about the add rules, however? The form of equation (9) implies that the value vectors v for add rule r have the same dimensionality the embedding of the head of r. The key and query vectors for rule r (as used in equation (2)) can share this dimensionality by default, although are free to override this and specify a different dimensionality for them. The foregoing choices determine the dimensions of the parameter matrices ((9, K(O) Q associated with rule r.
D LIKELIHOOD COMPUTATION DETAILS
In this section we discuss the log-likelihood formulas in section
Derivations of the log-likelihoode formula (6) can be found in previous work (e.g., Hawkes, 1971; Liniger, 2009; Mei & Eisner. 2017). Derivations of this formula appear in previous work (e.g.. Hawkes, 1971; Liniger, 2009; Mei & Fisner, 2017). Intuitively, when training to increase the log- likelihood (6), each log λe, (4:) increased to explain why the observed event 6 happened at time 4; while
(l(d) is decreased to explain why no event of any possible type E {1,..., EE. ever happened at other times. Note that there is no log under the integral in equation (6). Why? The probability that there was not an event of any type in the infinitesimally wide interval [t, ( + di) is 1 X (t) whose log is dt)
The integral term in equation (6) is computed using the Monte Carlo approximation given by Mei & Eisner (2017, Algorithm 1), which samples times t. This yields an unbiased stochastic gradient. For the number of Monte Carlo samples, training time, we match the number of samples to the number of observed events at training time, a reasonable and fast choice, but to estimate log-likelihood when tuning hyperparameters reporting final results, we take 10 times as many samples. The small remaining variance in this procedure is shown in our error bars, as explained in footnote 6.
follow the practice of Mei & Eisner (2017): namely, at
At each sampled time t, the Monte Carlo method still requires a summation over all events to obtain (t). This summation can be expensive when there are many event types. This is not serious problem for our standalone A-NHP implementation since it can leverage GPU parallelism. But for the general A-NDTT implementation, is hard to parallelize the (x+(t) computation over and t. In that case, we use the downsampling trick detailed in Appendix D of Mei et al. (2020a).
An alternative would be to replace maximum likellhood estimation with noise-contrastive estimation, which is quite effective at training NHP and NDTT models (Mei et al., 2020b).
E How TO PREDICT EVENTS
It is possible to sample event sequences exactly from an A-NHP or A-NDTT model, using the thinning algorithm that traditionally used for autoregressive point processes (Lewis & Shedler, 1979; Liniger, 2009). In general, to upply the thinning algorithm to sample the next event at time 2 to. it is necessary to have an upper bound on (t) : t E [to. bc)} for each event type t. An explicit construction for the NHP (or NDTT) model was given by Mei & Eisner (2017, Appendix B.3). For A-NHP and A-NDTT, observe that (t is a continuous real-valued function of [[]] (the particular function depends on e and the history of events at times < to). Since [t]] falls in the compact set [-1, 1]d (thanks to the sinusoidal embedding (4)), it follows that X²(t) is indeed bounded. Actual
competition using its own parameters. One use of this mechanism would he to allocate multiple attention heads single rule.
16

---

Published as a conference paper at ICLR 2022
DATASET
# OF EVENT TOKENS
SEQUENCE LENGTH
TRAIN
DEV
TEST
MIN
MEAN
MAX
SYNTHETIC
10
59904
7425
7505
49
75
99
MIMIC-II
1930
252
237
2
33
STACKOVERFLOW
22
345116
38065
97233
41
72
736
ROBOCUP
528
2195
817
780
780
948
1336
Table 1: Statistics of each dataset.
numerical bounds can be computed using interval arithmetic. That is, we can apply our continuous function not particular value of [t]] but to all of [-1, where for any elementary continuous function f IR + I. we have defined f [[2%, Tuil) to return some bounded interval that contains f()) for all x E [a Whi]. The result will be bounded interval that contains (t) for all E [Lo, 20).
Section 7 includes a task-based evaluation where we try to predict the time and type of just the next event. More precisely, for each event in each held-out sequence, we attempt predict its time given only the preceding events, as well as its type given both its true time and the preceding events.
We evaluate the time prediction with average L2 loss (yielding a poot-mean-squaree error, or RMSE) and evaluate the argument prediction with average 0-1 loss (yielding an error rate).
Following Mei & Eisner (2017). we use the minimum Bayes risk (MBR) principle predict the time and type with lowest expected loss. For completeness, we repeat the general recipe in this section.
For the 7th event, its time &; has density (t) x(t) exp(- fr_,, ((()d("). We choose !_-_ (t:(t)dt the time prediction because it has the lowest expected L.g loss. The integral can be estimated using i..d. samples of t; drawn from (t) by the thinning algorithm.
Given the next event time 4,, we choose the most probable type arg max, de (4;) as the type prediction because it minimizes expected 0-1 loss. In some circumstances, one might also like to predict the most probable type out of restricted set 8d Ç (1,,... ..,.,, E}. This allows one to answer questions like "If we know that some event of the form message (eve, T) happened at time t,, then what was the forum T, given all past events?" The answer will simply be arg max (ta)..
F EXPERIMENTAL DETAILS
F.1 DATASET DETAILS
Table shows statistics about each dataset that we use in this paper.
F.1.1 PILOT EXPERIMENTS ON SIMULATED DATA
In this experiment, draw data from randomly initialized NHP, A-NHP, SAHP, and THP. For all of them, we take the number of event types to be E' = 10. For NHP, the dimensions of event embeddings and hidden states are all 32; for A-NHP, the number of layers (L in our paper) 2, and the dimensions of time embeddings and event embeddings are 32; for SAHP, the number of layers is and the dimension of hidden states is 32; for THP, the number of layer is 7, and the dimension of hidden states is 32.
For each model, we draw 800, 100, and 100 sequences for training, validation and testing, respectively. For each sequence, the sequence length / is drawn from Uniform (49, 99). We take the maximum observation time T to be f, + 1, one time step after the final event.
We fit all 4 models on each of these 4 synthetic datasets. The results are graphed in Figure 5 and show that NHP, SAHP, and A-NHP have very close performance on all 4 datasets (outperforming THP, especially at predicting timing, except perhaps on the THP dataset itself). Notably, THP fits the time intervals poorly when it is misspecified, perhaps because its family of intensity functions (section 6) not good match for real data: THP requires that the intensity of between events changes more slowly later in the event sequence, and that if it increases over time, it approaches linear growth rather than asymptote.
17

---

Published as a conference paper at ICLR 2022
log-likelihood event (the training objective) of the whole test event sequence
-122
-11
-14
-124
01.09
-15
-120
-1.0
-17
oog-ikelihood per event of the event times only
1.92
14
0.
11
0.:
5.-
6.7
0.34
Log- ligeliieeioodd per event of the event types only
-2.24
2,20
2,200
2.50
-3292
2.40
2.98
-224
2.200
-224
2,240
-222
(a) THP Data
(b) SAHP Data
(c) NHP Data
(d) A-NHP Data
Figure Log-likelihooe on held-out data (in nats, with 95% bontstrap confidence intervals"), Larger values are better. Each column is a different experiment, on a single synthetic dataset generated from a different distribution family (shown at the bottom of the column). Within each column, the red dashed horizontal line represents the log-likelihood of the THP, SAHP, NHP, and A-NHP (from this to our A-NHP model (the rightmost model). Other models are shown in lighter ink. Note that log- likeelkddoood for continuous variables can be positive (as in the second row), since it uses the log of probability density that may be 1.
distribution that generated the data. Within each column, train and test 4 models: right).
model from correct family shown red: compare
F.1.2 OTHER DATA DETAILS
For MIMIC-II and StackOverflow, we used the version processed by Du et al. (2016); more details (e.g., about processing) can be found in their paper.
For RoboCup, used the version processed by Chen & Mooney (2008); please refer to their paper for more details (e.g., data description, processing method, etc)
F.2 IMPLEMENTATION DETAILS
For NHP, our implementation is based the public Github repositories https: thtp///tttttuutt.c com/ HMK at..l//eeraawkee (Mei & Fisner (2017), with MIT License) and https: /qi thub. com/ wwwHCC/neural-aaweee.. wwwawes-pprttiee cle smoothin ng (Mei et al. (2019), with BSD 3-Clause "New" or "Revised" License). We made a considerable amount of modifications to their code (e.g.. model, thinning algorithm), order to migrate PyTorch 1.7. We built the standalone GPU implementation of A-NHP upon our NHP code.
For NDTT, we used the public Github repository at https://githua com/ /MUI neural-dae --l-ddttttoo--trog--------- upon NDTT.
(Mei et al. (2020a), with MIT License). We built A-NDTT
For THP, we used the public Github repository at https://github. com.smiiacmmmmmaaaaaaaaaaaaa Hawkss-Proesss (Zuo et al. (2020), no license specified).
Cransformer-HHwkeee
For SAHP, we used the public Github repository at https: //github...//////// won.//ianggARResercchee rapo (Zhang et (2020a), no license specified).
F.3 TRAINING DETAILS
For each model in section 7, we had to specify various dimensionalities. For simplicity, used single hyperparameter D and took all vectors to be in R This includes the state vectors NHP, the fact embeddings of NDTT and A-NDTT, and the query, key, and value vectors for the models
18

---

Published as a conference paper at ICLR 2022
DATASET
TRAINING TIME (MILLISECONDS)) SEQUENCE
NHP
A-NIIP
SYNTHETIC
208.7
56.3
MIMIC-II
2.9
32.6
STACKOVERFLOW
156.6
65.7
Table 2: Training time of NHP and A-NHP for experiments in section 7.1.
with attention mechanisms (THP, SAHP, A-NHP, and A-NDTT), For the models with attention mechanisms, we also had choose the number of layers L.
We tuned these hyperparameters for each combination of model, dataset, and training size (c.g.. each bar in Figures 2, 3a and 5), always choosing the combination of D and that achieved the best performance on the dev set. Our search spaces were D E {4, 8, 16, 32, 64, 128) and E [1, 2, 3, 4, 6). practice, the optimal D for model was usually 32 64; the optimal was usually 1, 2, 3.
To train the parameters for a given model, we used the Adam algorithm (Kingma & Ba, 2015) with its default settings. We performed early stopping based on log-likelihoode on the held-out dev set.
For the experiments section 7.1, we used the standalone PyTorch implementations for NHP and A-NHP, which are GPU-friendly, We trained each model an NVIDIA K80 GPU. Table 2 shows their training time per sequence on each dataset.
For section 7.2, we run our NDTT and A-NDTT models only on CPUs. This follows Mei et al. (2020a), who did not find an efficient method to leverage GPU parallelism for training NDTT models. The machines the training time of NDTT and A-NDTT was 62 and 149 seconds per sequence, respectively. See Appendix F.4 for future work improving the latter time by exploiting GPU parallelism.
used for NDTT and A-NDTT are 6-core Haswell architectures. On RoboCup.
For the NHP and A-NHP models in section 7.2, we ran the specialized code for these models on CPU well, rather than GPU as in section 7.1, since the RoboCup sequences were too long to fit in the memory of our K80 GPU. The training time was 66 and 95 seconds per sequence for NHP and A-NHP, respectively.
F.4 TRAINING PARALLELISM
We point out that in the future, GPU parallelism could be exploited through the following procedure, given a GPU with enough memory to handle long training sequences. (The layers can be partitioned across multiple GPUs if needed.)
For each training minibatch, the first step to play each event sequence e a e2@f2, ep sity for- ward to determine the contents of the database on each interval (0, (1), (4,, (a),..., (7), (4,, T]. This step runs on CPU, and computes only the boolean facts ("Datalog through time") without their embeddings ("neural Datalog through time").
47],
Let F be the set of facts that ever appeared in the database during this minibatch and let R be the set of rules that were ever used to deduce add them (section 5). Furthermore, let T be the set of times consisting of tt} together with the times t that are sampled for the Monte Carlo integral (Appendix D).
computation graph of size (( I) can now be constructed, as illustrated in Figure 1b, to compute the embeddings [[] (t) of all facts h E F at all times t T. The layer-& embeddings at time E T depend on the layer-(E - 1) embeddings at times I; & I, according to the add rules R. The layer-f embedding of fact that is deduced time f also depends the layer- embeddings time ( of the facts that it is deduced from, according to the deduction rules in R; this further increases the depth of the computation graph.
For given fact h F, [[]] (9) (t) can be computed in parallel for all event sequences and all times t E T (even times t when h is not true, although those embeddings will not be used). Multiple facts that are governed by the same NDTT rule R. can also be handled in parallel, since they use
19

---

Published as a conference paper at ICLR 2022
the same r-specific parameters. Thus, GPU can be effective for this phase. The computation of (t) in equation (9) must take care to limit its attention to just those earlier times when an event occurred that added & via rule and the computation of [h][0] (t) in equation (9) must take care to consider only rules that in fact deduce h time because their conditions are true at time This masks unwanted parts of the computation, rendering parts of the GPU idle. GPU parallelism will still worthwhile if substantial fraction of the computation remains unmasked-wiln relatively homogenous settings where most facts in F hold true for large portion of the observed interval [0, T even if their embeddings fluctuate.
is for
F.5 MORE RESULTS
The performance of A-NDTT and NDTT is not always comparable for specific action types, as shown in Figure 6. In terms of data fitting (left figure), A-NDTT is significantly better at the kickff events while NDTT is better at the others. For time prediction (middle figure), A-NDTT is significantly better at the goal, kickoff, and pass events, but the differences for the other action types are not significant. For action participant prediction (right figure). A-NDTT is significantly better at the ki ckoff events while there is no difference for the others; both do perfectly well at the goal and kick events such that their dots overlap the origin.
1.0
17.5
A-OTT
6
7-
[3
15.0
0.8
0.6
10.0
0.4
5.0
2.5
0.0
12
0.0
0.2
0.6
0.8
1.0
NDTT negative log likelihood
NDTT
eeror
Figure 6: Results of NDTT and A-NDTT in Figure 3a broken down by action types, with horizontal and vertical error bars, respectively.
In Figure 7, we show that Figure 6 does not change qualitatively when re-run with different random seeds.
20

---

Published as a conference paper at ICLR 2022
ket
15.0
kakil pass
piss
pas
A-NDTT
Iteal
stea
ctall
10.0
7.5
0.2
2.5
3
0
2
8 10
12
16 18 20
0.0
0.0
0.2
0.4
0.6
0.8
1.0
NDTT
NDTT
NDTT
negative log like lillhodd
RMSE
error rate %
(a)
20.0
kok
12
Metion
15.0
keetff
4-NOTT
steal
pare
12.5
pass
5tea
A-NOTT
sceal
pase
10.0
7.5-
0.4
4-
2,5
0.2
0.0
0.0
3
10
12
14 16 20
0.0
0.2
0.4
0.6
0.8
1.0
NDTT
NDTT
NDTT
negative log like lioood
RMSE
error rate %
(b)
20.0
kokk
Metiin
15.0
keeeft
A-NOTT
eral
pas
12.5
pas*
1-1
A-NOTT
p453
emal
7.5
0.4
0.2
2.5
0.0
0.0
2
7
2
10 12
20
0.0
0.2
0.4
0.8
1.0
NDTT
NDTT
NDTT
negative log likee ihood
RMIEE
error rate %
(c)
4244
20.0
kek
15.0
piss ITaal
12.5
pasa
p4s3
cmal
7.5
0.2
2.5
3
4
6
10 12
14 16 18 20
0.0
0.0
+
0.2
0.4
0.6
0.8
1.0
NDTT
NDTT
NDTT
negative log like lihod
RMSE (d)
error rato %
Figure 7: Replications Figure (one per row) with different random seeds used during training.
21


Reviews:
[{"review_id": "twuVgFVrj3u", "reviewer": "Reviewer_5rCL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clearly the result of detailed study of the of the NDTT modeling framework and its requirements for underlying models. I should start with saying that I'm not an expert in NDTT.\n\nPositive aspects about the paper:\n* It discusses clearly and intuitively why the authors made some architectural choices in Sect. 3 and 4.\n* Experimental results indicate that the model works.\n\nChallenges for the reader:\n* The paper refers to the architecture as a Transformer-like thing, but important differences exist (no feed forward sublayer, use of the bounded tanh as activation function, no layer norms, positional encodings concatted and fed in at each layer). In Sect. 4, \"multi-head selective attention\" is introduced, which has no relation to standard \"multi-head attention\" besides the fact that several subnetworks do different things in parallel - even the aggregation of results is different. These differences are not called out explicitly, nor empirically validated.\n* Section 5 is extremely dense and very hard to follow, lacking motivation:\n  - If you are trying to introduce NDTT, you should provide an example that helps the reader to understand what you are trying to achieve, BEFORE introducing the formalism.\n  - $[h]^{(\\ell)}(t)$ is used in Eq(9), visually hard to distinguish from $[[ h ]]^{(\\ell)}(t)$, and only discussed 10 lines later.\n* Experiments: no ablations of the architecture are studied, even though many choices were made that are non-obvious.\n\nDetailed questions:\n* Sect. 4: why is the embedding of the event type not sufficient to drive different attention patterns? (i.e., why does the event type need to be part of parameters rather than part of data)\n\nUpdate after rebuttal\n---------------------------\nWhile the authors provided extensive answers to my questions, the updated submission largely does not reflect these answers, nor is it obvious how it could (given the space constraints). Overall, I think the manuscript is not ready for publication at this time, largely due to a lack of clarity in the writing. I feel unable to judge the core technical contribution, but bad writing alone is sufficient for me to decide not to change my rating.", "summary_of_the_paper": "A model for sequences of events occurring in the continuous time setting is presented. It uses the self-attention mechanism from the Transformer model, together with some domain-specific masking operations, to create event embeddings that are contextualized by their relation to other (relevant) events that occurred. The model is then extended towards a generative process, which in turn is integrated into a neural datalog through time framework.\nExperiments show the model to be perform equally or better than baselines based on RNNs or discrete time embeddings.", "summary_of_the_review": "Overall, I found the first four pages of this paper quite enjoyable, but found the remainder large impenetrable as someone unfamiliar with the area. Given the fact that obvious ablation experiments are missing, I do not think that this paper is ready for publication, lacking both in clarity of writing and strength of evaluation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d_ZweeR6zJ", "reviewer": "Reviewer_9eGx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper provides some interesting contributions on modeling continuous-time event processes and adding specifications via the NDTT framework.\nHowever, the paper also contains several weaknesses, which makes it not ready for publication in its current format.\nThe writing needs to be substantially improved before this manuscript is ready for publication. What is the intuition on Eq (1-4)? Why would the embeddings of the time be relevant from the second layer onward?  It is unclear what the main contribution of the paper is. On the one hand, the paper proposes continuous-time event transformers, while on the other hand, the paper proposes a neuro-symbolic framework for embedding rules of which event can draw information from which other events. \nThe paper assumes the events are discrete-valued, i.e., $e \\in \\{1,2,\\dots E\\}$, which limits its scopes for general-purpose modelling.\nThe paper fails to experimentally compare the proposed continuous-time transformer to existing continuous-time models on standard benchmarks (PhysioNet, etc.) [1,2,3,4]\nThe paper fails to compare with existing continuous-time attention-based architectures [5].\nFrom the experiments' side, there seems only little to no improvement over standard NDTT.\nNo code was provided.\n\n[1] Rubanova et al. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. NeurIPS 2019\n[2] Singh et al. Sequential neural processes. NeurIPS 2019.\n[3] Li et al. Scalable gradients for stochastic differential equations. AISTATS 2020.\n[4] Norcliffe et al. Neural ODE Processes. ICLR 2021\n[5] Shukla and Marlin. Multi-Time Attention Networks for Irregularly Sampled Time Series. ICLR 2021", "summary_of_the_paper": "The paper proposes a continuous-time transformer architecture for event sequence tasks using the Neural Datalog Through Time (NDTT) framework for incorporating logic specifications into the model.\nThe paper then experimentally compares the proposed model with point-process models and standard NDTT. ", "summary_of_the_review": "I recommend the authors to improve the writing and clarify whether their approach aims to contribute to general-purpose continuous-time modeling or on incorporating continuous-time logic specification via the NDTT framework. In the current form, neither of these two objectives is fulfilled adequately (see comments above).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G67aASNIjKL", "reviewer": "Reviewer_2v8P", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The proposed model is well-motivated. \n- The architecture and possible variations are described very thoroughly.\n- Combining the model with logical rules using the NDTT framework (Section 5) allows to incorporate known constraints on the modelled environment, which may lead to better generalization and predictive performance (Section 8).\n\n-------\n\nI have the following main concerns regarding this paper:\n\n1. **Missing discussion of closely related works**\n\n- The idea of using transformer architectures to define neural TPP models has been explored in multiple existing works. While the THP (Zuo et al., ICML 2020) and SAHP (Zhang et al., ICML 2020) models have been mentioned in this paper, several other published works have been missed and not compared with:\n\n    - [Enguehard et al., ML4H 2020](https://arxiv.org/abs/2007.13794)\n    - [Chen et al., ICLR 2021](https://arxiv.org/abs/2011.04583)\n    - [Zhu et al., AISTATS 2021](https://arxiv.org/abs/2002.07281)\n    - [Sharma et al., KDD 2021](https://arxiv.org/abs/2008.11308)\n\n- The definition of selective attention introduced in Section 4 is equivalent to the notion of Granger causality. GC has been extensively studied in the field of marked TPPs and has been combined with transformer-based neural TPP models ([Zhang et al., WWW 2021](https://dl.acm.org/doi/fullHtml/10.1145/3442381.3450135)).\n\n\n2. **A more thorough empirical evaluation is needed to support the main claims of the paper**\n\n- One of the claimed advantages of the proposed A-NHP model is its **flexibility** (Section 1, point 3; Section 7). However, the experiments only consider THP and SAHP as baselines. These model assume a rather simple parametrization of the inter-event time distribution, which limits their flexibility, as pointed out in Section 7. There exist multiple more flexible architectures, such as [(Omi et al., 2019)](https://arxiv.org/abs/1905.09690), [(Shchur et al., 2020)](https://arxiv.org/abs/1909.12127) or [(Sharma et al., 2021)](https://arxiv.org/abs/2008.11308), that haven't been compared with. \n\n    These models have other advantages compared to the A-NHP model such as closed-form likelihood computation and lower memory footprint, while A-NHP requires multiple forward passes to evaluate the intensity at random points during Monte Carlo approximation of the integral. Therefore, comparing to these flexible and more efficient models would be important to answer if the higher runtime/memory consumption of A-NHP is justified by its improved flexibility.\n\n----\n\n**Minor suggestions**\n- Section 5: The contents are rather dense and are difficult to comprehend without first reading Mei et al., 2020a. Adding a figure that demonstrates how the rule embedding $[\\\\![h]\\\\!]$ $(t)$ affects the conditional intensity for different event types would be really helpful (similar to Figure 1b).\n\n- Figure 2: The figure is difficult to read. Color-coding or using symbols to denote different models in each plot would make it easier to understand.\n\n- Reducing the amount of new / non-standard notation, if possible, would improve the readability of the paper.\n\n- Another claim regarding the A-NHP model that is not explored in the experiments is its claimed ability to learn long-range interactions better than an RNN-based TPP, like NHP (Section 1, point 2). It would be interesting to evaluate this aspect of the model and highlight these failure modes of RNN-based TPPs. One idea would be to do this using multivariate Hawkes processes with long-range triggering kernels and many marks.\n\n\n------\n## Update after the rebuttal\n\nThe rebuttal has addressed some of my concerns, so I have raised my score\n- An efficient approximation to the log-likelihood is available, so the increased runtime / memory consumption compared to existing models shouldn't be a problem.\n- The clarity of writing in Section 5 has been improved\n\nStill, I have several remaining concerns, which is why I'm leaning towards rejection. Here are my suggestions for improving the paper:\n- Comparison with additional baselines, such as ADMN and LogNormMix \n- Make the connection to prior work more explicit in Section 4 (selective attention), add citations to other works on transformer-based TPPs, and better frame the contribution of this paper with respect to prior works\n- Explain how the upper bound for the thinning algorithm is derived for A-NTPP, since this is a crucial implementation detail", "summary_of_the_paper": "The main contribution of this work is a new transformer-based temporal point process (TPP) model.\nThe proposed model defines the conditional intensity at each time $t$ as a function of all past events using the attention mechanism.\nThis is different form existing neural TPP models that typically embed the event history into a vector.\nAdditionally, it is shown how the proposed model can be combined with the Neural Datalog Through Time (NDTT) framework [(Mei et al., 2020)](https://arxiv.org/abs/2006.16723), which allows us to specify certain hard constraints on the event occurrences (such as \"events of type x can only occur when certain conditions are met\").", "summary_of_the_review": "My main criticism of this paper is the lack of discussion and comparison with existing works, as described in point 1 and 2 above. The proposed modifications to the NDTT framework [(Mei et al., 2020)](https://arxiv.org/abs/2006.16723) don't lead to significant improvements over the original methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AwUO3tTgKv", "reviewer": "Reviewer_vfGf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper is not too theoretical, the techniques do seem to appear natural and well-motivated by the problem at hand. The experiments are well done and reproducible with the appendix.\n\nI did find the paper a bit difficult to read, but this may be somewhat unavoidable due to the probabilistic modeling. I am not sure about the claim of A-NDTT being simpler and shallower when training time is longer? There is a lot of emphasis on probability models and interpretability with neural-symbolic computing in the exposition that is unexplored in the experiments, too.", "summary_of_the_paper": "The paper introduces A-NHP and A-NDTT that tackle the problem of continuous time sequences and neural-symbolic computing with a time component using transformers and attention. The paper explains their model and techniques mainly with equations. The work seems primarily motivated by previous work on Hawkes processes and neural datalog through time NDTT. They improve by adding a probability model that allows them to embed all possible events and using transformers instead of LSTMs for parallelization. They achieve strong experimental results, especially in StackOverflow and RoboCup.", "summary_of_the_review": "I would recommend the paper for acceptance. While there are some issues, the paper is generally well-written, explores new techniques, achieves good results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper builds upon previous work on neural temporal point processes. It mainly proposes the replacement of the LSTMs with Transformers as transformers are widely considered as a more powerful sequence modeling tool and the three advantages listed in the end of section 1 in this paper.

However, on the modeling side, it is not straight-forward how to apply the transformer (the attention architecture) on to the continuous time-sequence problem using NDTT. I think because I read a revised version of this paper, it is actually more understandable to me as compared to the reviewers who read the first draft of the paper. I think A-NDTT is a natural and principled way of introducing the attention mechanism into the continuous time neural symbolic framework, although I agree it unfortunately does not leading to a significant improvement in every experimental setting.

To summarize the discussions, I think the authors did a good job in resolving the concerns on the related work and made the paper easier to understand. I appreciate these efforts from the authors even though I also understand there are concerns left still from the reviewers.

In summary, I am leaning to accept this paper because I think it is an interesting contribution. However, I do agree with the reviewers that the writing of the paper needs to be improved and the experiment section is relatively weak in this paper.
Decision: Accept (Poster)