Informed Temporal Modeling via Logical Specification of Factorial LSTMs
Authors: Hongyuan Mei
Guanghui Qin
Minjie Xu
Jason Eisner
Venue:  (2020)
Keywords: factorized LSTM
temporal point process
event streams
structural bias
Datalog

Abstract: Consider a world in which events occur that involve various entities. Learning how to predict future events from patterns of past events becomes more difficult as we consider more types of events. Many of the patterns detected in the dataset by an ordinary LSTM will be spurious since the number of potential pairwise correlations, for example, grows quadratically with the number of events. We propose a type of factorial LSTM architecture where different blocks of LSTM cells are responsible for capturing different aspects of the world state. We use Datalog rules to specify how to derive the LSTM structure from a database of facts about the entities in the world. This is analogous to how a probabilistic relational model (Getoor & Taskar, 2007) specifies a recipe for deriving a graphical model structure from a database. In both cases, the goal is to obtain useful inductive biases by encoding informed independence assumptions into the model. We specifically consider the neural Hawkes process, which uses an LSTM to modulate the rate of instantaneous events in continuous time. In both synthetic and real-world domains, we show that we obtain better generalization by using appropriate factorial designs specified by simple Datalog programs.


Under review as a conference paper at ICLR 2020
INFORMED TEMPORAL MODELING VIA LOGICAL SPECIFICATION OF FACTORIAL LSTMS
Anonymous authors Paper under double-blind review
ABSTRACT
Consider world in which events occur that involve various entities. Learning how to predict future events from patterns of past events becomes more difficult consider more types of events. Many of the patterns detected in the dataset by an ordinary LSTM will be spurious since the number of potential pairwise correlations, for example, grows quadratically with the number of events. We pro- pose type of factorial LSTM architecture where different blocks of LSTM cells are responsible for capturing different aspects of the world state. We use Datalog rules to specify how to derive the LSTM structure from database of facts about the entities in the world. This analogous how probabilistic relational model (Getoor & Taskar, 2007) specifies a recipe for deriving a graphical model struc- both cases, the goal is to obtain useful inductive biases by encoding informed independence assumptions into the model. We specifically consider the neural Hawkes process, which uses an LSTM to modulate the rate of instantaneous events in continuous time. In both synthetic and real-world do- mains, we show that we obtain better generalization by using appropriate factorial designs specified by simple Datalog programs.
ture from a database.
INTRODUCTION
Temporal sequence data abundant in applied machine learning. A common task is to impute missing events, e.g., to predict the future from the past. Often this is done by fitting generative probability model. For evenly spaced sequences, historically popular models have included hidden Markov models and discrete-time linear dynamical systems, with more recent interest in recurrent neural network models such as LSTMs. For irregularly spaced sequences, a good starting point is the Hawkes process, a self-exciting temporal point process; many variations and enhancements have been published, including neural variants using LSTMs.
All of these models can be described schematically by Figure la. Events EG Ct4,,....... assumed to be conditionally independent of previous events, given the system state (which may or may not be fully known given events e1, ........ ,,). That is, s; is enough determine the joint distribution of the 7th event and the updated state s;+1, which is needed to recursively predict all subsequent events.
Figure la and its caption show the three types of influence in the model. The update, affect, and depend arrows are characterized by parameters of the model. In the case of recurrent neural network, these are the transition, input, and output matrices.
Our main idea in this paper is to inject structural zeros into these weight matrices. Structural zeros are weights that are fixed at zero regardless of the model parameters. In other words, we will remove many connections (synapses) from both the recurrent and non-recurrent portions of the neural network. Parameter estimation must use the sparse remaining connections to explain the observed data.
Specifically, we partition the neural state s; E Rd into a number of node blocks. Different node blocks are intended to capture different aspects of the world's state at step By zeroing out rect- angular blocks the weight matrix, we will restrict how these node blocks interact with the events and with one another. An example is depicted in Figures 1b (affect, depend) and 1d (update).

---

Under review as a conference paper at ICLR 2020
In addition, by reusing nonzero blocks within weight matrix, we can stipulate (for example) that event e affects node block b in the same way in which event e' affects node block B' Such parameter tying makes it possible to generalize from frequent events to rare events of the same type.
Although our present experiments are small, we are motivated by the challenges of scale. weal-world domains may have millions of event types, including many rare types. To model organizational be- havior, we might consider a dataset of meetings and emails in a large organization. To model supply chains, we might consider purchases of goods and services around the world. In unrestricted model, anything in the past could potentially influence anything in the future, making estimation extremely difficult. Structural zeroes and parameter tying, if chosen carefully, should help us avoid overfitting to coincidental patterns in the data.
Analogous architectures have been proposed in the world of graphical models and causal models. Indeed, to write down such model is to explicitly allow specific direct interactions and forbid the rest. For example, the edges of Gaussian graphical model explicitly indicate which blocks of the inverse covariance matrix are allowed to be nonzero. Some such models reuse blocks (Hojsgaard & Jordan, 1997)-an HMM simple example of our architecture. The state s;
Lauritzen, 2008), As another example, a factorial HMM (Ghahramani whose states are m-tuples-can be regarded can be represented using m node blocks, each of which is a 1 1-hot vector that encodes the value of different tuple element. The key aspect of factorial HMM is that the stochastic transition matrix (update in Figure Id) is fully block-diagonal. The affect matrix is O, since the HMM graphical model does not feed the output back into the next state; the depend matrix is unrestricted.
But how do we know which interactions to allow and which to forbid? This is a Nomainn-specife modeling question. In general, we would like exploit the observation that events are structured objects with participants (which is why the number of possible event types is often large). For example, a travel event involves both a person and place. We might assume that the probability that Alice travels to Chicago depends only on Alice's 's state, the states of Alice's 's family members. and even the state of affairs in Chicago. Given that modeling assumption, parameter estimation cannot try to derive this probability (presumably incorrectly) from the state of the coal market.
These kinds of systematic dependencies can be elegantly written down using Datalog rules, as we will show. Datalog rules can refer to database facts, such as the fact that Alice is a person and that she related to other people. Given these facts, we use Datalog rules to automatically generate the set of possible events and node blocks, and the ways in which they influence one another. Datalog makes easy to give structured names to the events and node blocks. The rules can inspect these structures via mattern-matching.
In short, our contribution is to show how to use a Datalog program to systematically derive a constrained neural architecture from database. Datalog is blend of logic and databases, both of which have previously been used in various formalisms for deriving a graphical model architecture from database (Getoor & Taskar, 2007).
2 PRELIMINARIES!
Our methods could be applied to RNN sequence models. In this setting, each possible event type would derive its unnormalized probability from selected node blocks of state s; Normalizing these probabilities to sum to 1 would yield the model's distribution for event ex. Only the normalizing constant would depend all node blocks.
In this paper, we focus on the even more natural setting of real-time events. Here no normalizing constant is needed: the events are not in competition. As will see in section 5.1, it is now even possible for different node blocks to generate completely independent sequences of timestamped events. The observed dataset is formed by taking the union of these sequences.
In the real- setting, event e; has the form kjūt, where K; E K is the type of the event and t; e R its time. The probability of an event of type is at any specific instant é is infinitesimal. We will model how this infinitesimal probability depends on selected node blocks of s, There is no danger that two events will ever occur at the same instant, i.e., the probability of this is O.
Our conventions of mathematical notation mainly follow those given by Mci & Eisner (2017, section 2).
2

---

Under review as a conference paper at ICLR 2020
c
e
depend
h
depend
update
$6
S
affect
affect
(a)
(b)
struccuraal
Reros
atructural
zaros
(c)
(d)
Figure 1: Diagram (a) shows how the system state 8; evolves over time, generating observable events along the way. The event depends stochastically the system's current state s,. The current state then updates either deterministically how the state may be factored into 3 node blocks. In this example, the relative prohability of green events now depends on only the rightmost 7/1 of the state,
stnchastically 8,+1, where e; also affects this new state. Diagram (b) shows
green event occurs, affects only the leftmost 7/ of the state. The update computation involves a matrix multiplication shown (c): by zeroing out some blocks of the matrix, as illustrated in (d), can ensure that the updated center node block is not affected by the preceding state of the other node blocks time i, vice-versa.
We begin by describing our baseline model for this setting, drawn from Mei & Eisner (2017).
2. BASELINE MODEL: THE NEURAL HAWKES PROCESS
In general, multivariate point process is distribution over possible sequences of events kj = ky where 0 < f1 < t2 < .... common paradigm for defining such processes, starting with Hawkes (1971), is to describe their temporal evolution as in Figure la. Each s; is deterninistically computed from 8;-1 (update) and e.1 (affect), according to some formula, so by induction, 8; is deterministic summary of the first 1 events. e; = k; is then emitted stochastically from some distribution parameterized by s; (depend).
The structure of the depend distribution is the interesting part. is used, for each event type k E K. to define some time-varying intensity function As (t,-1, 20) - R20- This intensity function is treated as the parameter of an inhomogeneous Poisson process, which stochastically generates set of future events type i at various times in (t;-1. oo). Thus, all these [K]] Poisson processes together give many events of the form kat. The first such event-the with the earliest time i-is taken to be the next event e;. The remaining events are discarded (or in practice never generated).
As our baseline method, we take the neural Hawkes process (Mei & Eisner, 2017) to be our method for computing s; and defining the intensity function A from In that work, S; actually describes a parametric function of the form h (t:-1, 00) + Rd, which describes how the hidden state of the system evolves following event 6 That function used define the intensity functions via
(t) fv h()) 0
Under an inhomogenous Poisson process, disjoint intervals generate events independently. and the number events on the interval (a, b] is Poisson-distributed with mean f (t) dt. Thus, on a sufficiently narrow interval (t, t + dtj, the probability of single event is approximately A& (t) dt and the probability of more than event is approximately with error of (df?) both cases.
3

---

Under review as a conference paper at ICLR 2020
so the parameters of depend are the vectors Vy and the monotonic functions fa- Once e, kyat; has been sampled, the parameters for S,+1 are obtained by
S,-1 < (UW,, + Vh((4)
where inspired by the structure an LSTM, the affect parameters are given by matrix U and the event embeddings Wa, and the depend parameters are given by matrix V.
In this paper, we will show an advantage to introducing structural zeroes into Vg. U, and V.
2.2 STRUCTURED EVENTS AND NODE BLOCKS
In real world, atomic events typically involve a predicate and few arguments (called entities in the following), in which case it makes sense to decompose an event type into structured form' such as email (alice, bob), bob)), travel (bob, chi cago), etc. For generality, we also allow entities to have structured forms when necessary.
Then naturally, in such a world with many entities, we would like to partition the state vector h(t) into a set of node blocks (ha(t) hes and associate node blocks with entities. For example, we may associate (alsea)) (1) to alice and (hon) (1) to bob. Note that mind (alice) is just an example of the kind of node blocks that can be associated with alice. There can be another node block associated with the physical condition of alice and be called body (alice). Of course when there is only node block associated with alice, we can also simply call it alice.
From now on, we use teal-colored typewriter font for events and orange-colored font for node blocks.
From Figure 1b, we already see that an event may only depend and affect subset of hidden nodes in h(),, and this further prompts us to figure out a way to describe our inductive biases on which node blocks to determine the intensity of given event as well as which node blocks to be updated by one.
3 THE DATALOG INTERFACE
We propose general interface based on Datalog-a a declarative logic programming language-tt to assert our inductive biases into deductive database facts and rules. Then each event happens, we can query the database to figure out which node blocks determine its intensity and which node blocks will be updated by
In this section, we walk through our Datalog interface by introducing its keywords one step a time. We write keywords boldfaced typewriter font, and color-code them for both presentation and reading convenience. The colors we use are consistent with the examples in Figure 1.
3.1 THE is block AND is event KEYWORDS
We first need to specify what is a legal node block in our system by using the keyword is bs.okoc:
is.block(b).
where b can be replaced with a node block name like alice, bob, chicago and etc. Such a Datalog statement is database fact.
Then we use the keyword is_event to specify what a legal event type in our system:
is event (k))
where ls can be replaced with email (alice, bob), email (bob, alice), travel (bob, chio)
and etc. As we may have noticed, there may be many variants of email (S RR) where the variables and can take values as alice, bob and etc. To avoid writing a separate fact for each pair of and R, we may summarize facts of the same pattern rule:
is_event event (email :- R) is us_block (S), is block (R).
(5a)
head of rule
body of rule
SSiiilar structured representation of events has been common in natural language semantics (Davidson, 1967) and philosophy (Kim, 1993).
4

---

Under review as a conference paper at ICLR 2020
where is used to separate the head and body. Capitalized identifiers such S and R denote variables. rule mean: for any value of the variables, the head is known to be true if the body is known to be true. A fact such is event (email (alice, bob)) ) is simply rule with no body (so the :- is omitted), meaning that the body is vacuously true.
To figure out what event types are legal in system, we query the database by:
is_event (K)?
(6)
which returns every event type k that instantiates is _e_vent (k), Note that, unlike fact or rule that ends with period (.), a query ends with a question mark (?).
3.2 THE depend KEYWORD
We can declare database rules and facts about which events depend which node blocks using the depend keyword as:
depend (k=,))= condition, ...., conditiiiiinnnnnee
where k and b are replaced with Datalog variables or values for event and node block respectively, and condition, ................................ ... conditiony stands for the body of rule. An example is as follows:
depend (travel (bob, chicago), X) :- resort (X) ,at (X, chicago).
(8a)
depend (travel (bob, chicago), X) friend (bob, x) at (X, chicago).
(8b)
By querying the database for a given using
depend Bk?
(9)
we get g that is the set of all the node blocks b that instantiates depend b) and has superscript d for depend. Then we have:
(4)
( dependa, 8) B, B,C bo())))
(10)
where a--) is the sigmoid function, ranges over all the rules and + depend (k, b) means "the rule r proves the fact depend (k, b)". The matrices A, E R222 Dxxxx and B, E Dx xD learn how k 20
depends on each b. The "conversion" matrix CD..D. E Px projects hy) E E Ps into R. for dimension compatibility with B,. Note that the number of Cn. only increases with the number of possibilities of D₃ x D, but not the number of b values: the former usually much smaller than the latter.
The aggregator + represents pooling operation on a set of non-negative vectors. We choose @ = = and @* = max because it is appropriate to sum the dependencies over all the rules but extract the mmax-dependency" among all the node blocks for each rule. As shown in equation (8), the intensity chicago these two
of travel (bob, chi cago) is determined by both resorts and his friends possible motivations should be summed up. But bob may only stay at one of his friends' home and can only afford going to few places, so only the "best friend" and "signature resort" matter and that is why we use max-pooling for #y"
As matter of implementation, we modify each depend rule have the rule index third argument:
depend (k, bb), :- conditiiinnn...o.. tond:ttoon...............t.
(11)
This makes it possible to apply semantics semanticx-peeserving transformations to the resulting Datalog pro- gram without inadvertently changing the neural architecture. Moreover, if the Datalog programmer specifies the third argument r explicitly, then we do not modify that rule. As a result, it is possible for multiple rules to share the same r, meaning that they share parameters.
3.3 THE affect KEYWORD
We can declare database rules and facts about which events affect which node blocks using the affect keyword as:
affect (k, (k,b)= condition,
conditionn.
(12)
such that know which node blocks update each event happens. For example, can allow travel (bob, chicago) to update (t) for any X who is friend of bob and at chicago:
affect (travel (bob, chicago), x)) friend (loenddbobb,)) X), at (X, chicago).
5

---

Under review as a conference paper at ICLR 2020
By querying the database for given & using
affect (k,
get By that is the set of all the node blocks b that instantiates affect (k) where the superseript a stands for affect. Then each node block h2) updates itself as shown in equation but it raises an important question: what the U and V to use?
Similar to how A, and B, in equation (10) are declared, a U, is implicitly declared by each affect rule such that have:
(t) atrrtatecc b) U,
(13)
where @ = Σ. This term is analogous to the Uw term in section 2.1
Note that also modify each affect rule (as we do for depend in section 3.2) to have the rule index r as a third argument. By explicitly specifying r, the Datalog programmer can allow multiple affect rules share U..
3.4 THE update KEYWORD
We specify how node blocks update one another by using the update keyword:
update (b', ,b,k): condition, ...., conditionn ,ontiiiio
(14)
meaning the node block b' updates the node block b when &t happens. Note that & can equal b. It is often useful to write this rule:
update (B,B,K): :- affect (K,
(15)
which means that whenever K causes B to update, gets to see its own previous state (as well as K). To update the node block with event k, we need
11 (1) = 3r upatttpattt b) V, DD (1)
(16)
where r ranges over all rules and + = e' = Σ. The matrices V, E Drx learns how updates
b and CD..Dy E R Doy helps to make dimensions compatible. This term is analogous to the Vh(t) term in section 2.1.
Having equations (13) and (16), pass &0,0, k + 11,6 through the activation functions and obtain the updated hy.new-
Similar to depend and affect, rule to allow multiple rules to share V,. Parameter sharing (in depend, affect and update) is important because it works form of regularization: shared parameters tend to get updated more often than the individual ones, thus leaving the latter less likely to overfit the training data when we "early-stop" the training procedure.
also explicitly specify extra argument in each update
3.5 REVISITING is_event KEYWORD
When each event type k is declared using is eve.ttentc (k), the system automatically creates event embedding vectors V& and W and they will be used in equations (10) and (13) respectively. When some event types involve many entities which results in a very large number of event types, this design might end up with too many parameters, thus being hard to generalize to unseen data.
We can allow event types share embedding vectors by adding an extra argument to the keyword is event:
(17) where m is an index to pair of embedding vectors and wm. There can be more than one pair that is used by an event type & as shown this example: is_event (email (S,R), S), is_event (email RS, R), es_event (email RS, email) and etc. Then we compute the final embedding vectors of email (S, R) as:
is event (k,m) :-
conditionn..
V (e, = Vs + Vn + Venail
(18a)
(S,R) Ws Wa W Wenail
(18b)

---

Under review as a conference paper at ICLR 2020
Similar argument in section 3.4 applies here that sharing embedding vectors across event types is
form of regularization.
In simplified version of our approach, could homogeneous neural architecture where all events have the same dimension, etc. In our actual implementation, we allow further flexibility by using Datalog rules to define dimensionalities, activation functions, and multi-layer structures for event embeddings. This software design is easy to work with, but is orthogonal to the machine learning contribution of the paper, we describe it in Appendix A.4.
4 ALGORITHMS
Learning Following Mei Eisner (2017), we can learn the parameters of the proposed model by locally maximizing É in equation (19) using any stochastic gradient method: Its log-likelihood given sequence over the observation interval [0, is as follows:
f= T=E log Au. (44
1=0 E Ened)a(t)di
Ax(()d
(19)
The only difference is that our Datalog program affects the neural architecture, primarily by dictating that some weights in the model are structurally zero.
Concretely, compute l and its gradient, as each event e; k,üt, happens, we need query the database with depend (k, B)? ? for the node blocks that each & depends on in order to compute log AR, (44) and the Monte Carlo approximation to f ΣLEK (t)ddt Then need to query the database with affect (k,B)? for the node blocks to be affected and update them. A detailed recipe is Algorithm 1 of Appendix B.1 including down- down-sampling trick to handle large K.
Prediction Given an event sequence prefix ky kg .... 1--1k1114440 114-1 we may wish predict the time and type of the next event. The time 4 has density pt(() = X()) exp (-ft.. ((s)ds) where A(1) ΣxEK (L), and we choose Stz (p:(()dt the time prediction because has the lowest expected L2 loss. Given the next event time t, the most likely type would simply be arg max, As(t;), but the most likely next event type without knowledge of f; is arg max; Je, A(O) (t)d)t The integrals in the preceding equations can be estimated using i.i.d. samples of t. drawn from pt(t).
az(l)
We draw f; using the thinning algorithm (Lewis & Shedler. 1979; Liniger, 2009; Mei & Eisner, 2017). Given f,, we draw k; from the distribution where the probability of each type & is proportional to (tz). full sequence can be rolled out by repeatedly feeding the sampled event back into the model and then drawing the next. See Appendix B.2 for implementation details.
5 EXPERIMENTS
We show how our Datalog interface to inject inductive biases into the neural Hawkes process (NHP) on multiple synthetic and real-world datasets. On each dataset, we compare the model with modified architecture-we call it structured neural Hawkes process (or structured-NHP) with the plain vanilla NHP on multiple evaluation metrics. See Appendix C for experimental details (e.g., dataset statistics and training details). We implemented the model in PyTorch (Paszke et al., 2017).*
5.1 SYNTHETIC DATA- SUUN-SSSPPPPOOIOOON OF EVENT SEQURNCES
As Mei & Eisner (2017) pointed out, it is important for model family to handle the superposition of real-time sequences, because in various real settings, some event types tend not to interact. For example, the activities of two strangers rarely influence each other, although they are simultaneously monitored and thus form a single observed sequence.
*Code will he released upon paper acceptance.

---

Under review as a conference paper at ICLR 2020
0.00
110
Wotter 303
30x
Fubber simeyy
(a) M 4
(b) M =
(c) M = 16
Figure 2: Learning curves of structured NHP and NHP sequences drawn from the superposition of M neural Hawkes processes. The red line indicates oracle performance.
In this section, experiment the data known to be drawn from superposition of M neu- ral Hawkes processes with randomly initialized parameters. Each process X has four event types event (K, x) where K can be 1, 2, 3 and 4.
To leverage the knowledge about the superposition structure, one has to either implement a mixture of neural Hawkes processes transform single neural Hawkes process superposition model by (a) zeroing out specific elements of VA such that (t( for E Kx depends on only subset S of the LSTM hidden nodes, (b) setting specific LSTM parameters such that events of type K' Ky don't affect the nodes in S and (c) making the LSTM transition matrix blocked-struuctueed matrix such that different node blocks don't 't update each other. Neither way is trivial.
With our Datalog interface, we can explicitly construct such superposition process rather easily by writing simple datalog rules follows:
depend(vvent (K,X), X, X) (x.
(20a)
affect (event (K,X), X) is bx.
(20b)
update (X, unit (X)) :- is_block
(20c)
Events of X do not influence at all, and processes don't share parameters.
We generated learning curves (Figure 2) by training a structured-NHP and NHP on increasingly long prefixes of the training set. As we can see, the structured model substantially outperform NHP at all training sizes. The neural Hawkes process gradually improves its performance more training sequences become available: it perhaps learns to set its w and LSTM parameters from data. However, thanks to the right inductive bias, the structured model requires much less data achieve somewhat close to the oracle performance. Actually, as shown in Figure 2, the structured model only needs 1/16 of training data as NHP does to achieve a higher likelihood. The improvement of the structured model over NHP is statistically significant with p-value < 0.01 shown by the pair-permutation test at all training sizes of all the datasets.
5.2
DATASETS
Elevator System Dataset (Crites & Barto, 1996). In this dataset, two elevator cars transport pas- sengers across five floors in building (Lewis, 1991; Bao et al., 1994; Crites & Barto, 1996). Each event type has the form stop (C,F) meaning that C stops at F to pick up or drop off passengers where C can be car1 and car2 and F can be floor1, .... floor5. This dataset is representative of many real-world domains where individuals physically move from one place to another for, e.g., traveling, job changing, etc.
With our Datalog interface, we can explicitly express our inductive bias that each stop(C,F) F) de- pends on and affects the associated node blocks and F:
depend (stop C).
F). affect (C)) C). affect (steoc,C F).
(21a)
(21b)
The set of inductive biases is desirable because whether C will head to F and stops there is primarily determined by C's state (e.g., whether is already on the way of sending anyone to that floor) and F's state (e.g., whether there is anyone that floor waiting for car).
8

---

Under review as a conference paper at ICLR 2020
(al) Elevator
(a2) EuroEmail
(a) Learning of structured NHP and NHP the three datascts. On both datasets, the structured model substantially outerperfoms NHP, especially the data sparse scenarios.
(b1) Elevator
(b2) EuroEmail
(b) Scatterplots structured NHP NHP, comparing the held-out onglliklihood of the models (at the right end of learning curves) with respect to each test sequence. On both datasets, nearly all points lie top of y = x, since the structured model is consistently more predictive than NHP.
0.64
0.63
0.83
0.58
0.9
0.57
0.5
drure NH-
structuree red
85
33.5
22.5
70
12.0
61
11.0
20.3
error mate 96
#TTOr rate 7
(c1) Elevator
(c2) EuroEmail
(c) Prediction results of structured NHP vs. NHP (at the right end of learning curves). Error bars show 95% bootstrap percentile confidence intervals. The MSE% denotes the mean squared normalized by the vari- ance of the true time interval, and error rate denotes the fraction of the times our type prediction incorrect.
Figure 3: Results on real-world datasets.
We also declare global node block, building, that depends and affects every event in order compensate for any missing knowledge (e.g., the state of the joint controller for the elevator bank, and whether it's busy period for the humans) and/or missing data (e.g., passengers arrive certain floors and press the buttons).
Appendix C.2 gives full Datalog specification of the model that used for the experiments in this domain. More details about this dataset (e.g. pre-processing) can be found in Appendix C.1.2.
EuroEmail Dataset (Paranjape et al., 2017). this domain, we model the email communications between anonymous members of an European research institute. Each event type has the form email R) meaning that sends an email to R where and R are variables that take the actual members as values.
With our Datalog interface, we can express our knowledge that each event depends on and affects its sender and receiver as the following rules: depend (send (S, R), S). dsend (S, R), R).
(22a)
Thhese missing events are actually available in our domain, but chose not to model them, to make the problem harder.

---

Under review as a conference paper at ICLR 2020
affect (send S). affect (send R).
(22b)
Appendix C.2 gives a full Datalog specification of the model that we used for the experiments in this domain. More details about this dataset (e.g. pre-processing) can be found in Appendix C.1.3.
5.3 RESULTS
We evaluate the models in three ways shown in Figure 3. We first plot learning curves (Figure 3a) by training a structured-NHP and an NHP on increasingly long prefixes of each training set. Then show the per-sequence scatterplots in Figure 3b. We can that either in learning curve or scatterplots, structured-NHP consistently outperforms NHP, which proves that structured-NHP is both more data-efficient and more predictive. Finally, we compare the models the prediction tasks and datasets as shown in Figure 3c. We make minimum Bayes risk predictions as explained in section 4. We evaluate the type prediction with 0-1 loss, yielding an error rate. We can see, in both of Elevator and EuroEmail datasets, structured-NHP could be significantly more accurate on type prediction. We evaluate the time prediction with L2 loss, and reported the mean squared error as a percentage of the variance of the true time interval (denoted as MSE%). Note that we get can MSE%=1.0 we always predict I as 14-1 + AL where Al is the average length of time intervals.
Figure 3c shows that the structured model outperforms NHP on event type prediction on both datasets, although for time prediction they perform neck to neck. We speculate that it might be because the structure information is more directly related to the event type (because of its structured term) but not time.
DISCUSSION
There has been extensive research about having inductive biases in the architecture design of a machine learning model. The epitome of this direction is perhaps the graphical models where edges between variables are usually explicitly allowed or forbidden (Koller & Friedman, 2009). There has also been work in learning such biases from data. For example, Stepleton et al. (2009) proposed to encourage the block-struectured states for Hidden Markov Models (HMM) by enforcing a sparsity- inducing prior over the non-aaraeetric Bayesian model. Duvenaud et al. (2013) and Bratières et al. (2014) attempted to learn structured kernels for Gaussian processes.
Our work is in the direction of injecting inductive biases into a neural temporal model-a class of models that is useful in various domains such as demand forecasting (Seeger et al., 2016), personal- ization and recommendation (Jing& & Smola, 2017), event prediction (Du al., 2016) and knowledge graph modeling (Trivedi et al., 2017). Incorporating structural knowledge in the architecture design of such model has drawn increasing attention over the past few years. Shelton & Ciardo (2014) introduced a factored state space in continuous-time Markov processes. Meek (2014) and Bhat- tachariya al. (2018) proposed to consider direct dependencies among events in graphical event models. Wang et al. (2019) developed hybrid model that decomposes exchangeable sequences into a global part that is associated with common patterns and a local part that reflects individual characteristics.
However, their approaches are all bounded to the kinds of inductive biases that are easy to specify (c.g. by hand). Our work enables people to use Datalog program to conveniently specify the neural architecture based on a deductive database-a much richer class of knowledge than the previous work could handle. Although logie programming languages and databases have both previously been used to derive a graphical model architecture (Getoor & Taskar, 2007), we are, to the best of our knowledge, the first to develop such general interface for neural event model.
As future work, we hope to develop an extension where events can also trigger assertions and re- tractions of facts in the Datalog database. Thanks the Datalog rules, the model architecture will dynamically change along with the facts. For example, if Yoyodyne Corp. hires Alice, then the Yoy- odyne node block begins to influence Alice's actions, and K. expands include a new (previously impossible) event where Yoyodyne fires Alice. Moreover, propositions in the database-incle those derived via other Datalog rules can now serve as extra bits of system state that help deñne the A intensity functions in (1). Then the system's 's learned neural state s; is usefully augmented by large, exact set of boolean propositions-a division of labor between learning and expert knowledge.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
G. Bao, C. G. Cassandras, T. E. Djaferis, A. D. Gandhi, and D. P. Looze. Elevators dispatchers for down-peak traffic, 1994.
D. Bhattachariya, D. Subramanian, and T. Gao. Proximal graphical event models. In Advances in Neural Information Processing Systems (NIPS), pp. 8136-8145, 2018.
S. Bratières, N. Quadrianto, S. Nowozin, and Z Ghahramani. Scalable Gaussian process structured prediction for grid factor graph applications. Proceedings of the International Conference on Machine Learning (ICML), 2:1625--: 2014.
Robert H. Crites and Andrew G. Barto. Improving elevator performance using reinforcement learn- ing. In Advances in Neural Information Processing Systems, 1996.
Donald Davidson. The logical form of action sentences. 1967.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodrigues and Le Song. Recurrent marked temporal point processes: Embedding event history vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
D. Duvenaud, J. R. Lloyd, R. Grosse, B. Tenenbaum, and Z. Ghahramani. Structure discovery in nonparametric regression through compositional kernel search. Proceedings of the International Conference on Machine Learning (ICML), 28:2203-222 201122211,2222222201111111111111 2013.
Nathaniel Wesley Filardo and Jason Eisner. A flexible solver for finite arithmetic circuits. In Tech- nical Communications of the 28th International Conference Logic Programming (ICLP"12)
2012.
Lise Getoor and Ben Taskar (eds.). Introduction to Statistical Relational Learning. MIT Press, 2007. URL https::///// /www. cs. wwt.eeddussslooooe
Zoubin Ghahramani and Michael Jordan. Factorial hidden markov models. Machine Learning, 29 (245-273), 1997.
Alan G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 1971.
S. Hojsgaard and S. L. Lauritzen. Graphical Gaussian models with edge and vertex symmetries. Journal of the Royal Statistical Society, Series B (Statistical (ethoodlogy))70):000500000000 2008
H. Jing and A. Smola. Neural survival recommender. In Web Search and Data Mining. pp- 515-524, 2017.
Jaegwon Kim. Supervenience and mind: Selected philosophical essays. 1993.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. 2009.
J. Lewis. A Dynamic Load Balancing Approach to the Control of Multiserver Polling Systems with Applications to Elevator System Dispatching. PhD thesis, University of Massachusetts, Amherst, 1991.
Peter A. Lewis and Gerald S. Shedler. Simulation of nonhomogeneous Poisson processes by thin- ning. Naval Research Logistics Quarterly, 1979.
Thomas Josef Liniger. Multivariate Hawkes processes.
Diss., Fidgenössische Technische
Hochschule ETH Zürich, Nr. 18403, 2009.
C. Meek. Toward learning graphical and causal process models. In Uncertainty in Artificial In- telligence Workshop on Causal Inference: Learning and Prediction, volume 1274, pp. 43 48, 2014.
11

---

Under review as a conference paper at ICLR 2020
Hongyuan Mei and Jason Eisner. The neural Hawkes process: A neurally self-modulating multi- variate point process. In Advances in Neural Information Processing Systems (NIPS), 2017.
Hongyuan Mei, Guanghui Qin, and Jason Eisner. Imputing missing events in continuou--imee event streams. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
A. Paranjape, A. R. Benson, and Leskovec. Motifs in temporal networks. In Web Search and Data Mining, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. 2017.
M. Seeger, D. Salinas, and V. Flunkert. Bayesian intermittent demand forecasting for large invento- ries. In Advances in Neural Information Processing Systems (NIPS), 2016.
C. R. Shelton and G. Ciardo. Tutorial on structured continuous-time Markov processes. Journal of Artificial Intelligence Research, 51:725-778, 2014.
Thomas Stepleton, Zoubin Ghahramani, Geoffrey Gordon, and Tai-Sing Lee. The block diagonal infinite hidden markov model. In Artificial Intelligence and Statistics, pp. 552-559, 2009.
R. Trivedi, H. Dai, Y. Wang, and L.. Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In Proceedings of the International Conference on Machine Learning (ICML), 2017.
Y. Wang, A. Smola, D. C. Maddix, J. Gasthaus, D. Foster, and T. Januschowski. Deep Factors for Forecasting, In Proceedings of the International Conference on Machine Learning (ICML), 2019.
12

---

Under review as a conference paper at ICLR 2020
Appendices
A FRAMEWORK DETAILS
A. LSTM DETAILS
In this section, we elaborate on the details of the transition function V that introduced in sec- tion 2.1; more details about them may be found in Mei & Eisner (2017).
h(() o, @ (2o(2)()) 1) for E 44]
(23)
where the interval (ti-1, b.] has consecutive observations -;- 1 and k; endpoints. At f.. the continuous- time LSTM reads kşat, and updates the current (decayed) hidden cells c(t) to new initial values C,+1, based on the current (decayed) hidden state h(t;), follows:
1,1 +- 0 WU, + V,h(4) + d:)
(24a)
f/11 (U,W + Vyh((:) de)
(24b)
2/41 + 20 (U,Wk, + Vzh((4) + +d) (6)-1
(24c)
0:+1 a (U.Wa. Voh(t:) +
(24d)
Ci+1 < fit1 O c(4)) + is+1 7i+1
(25a)
S.+1 <- fit1 Oc + 1+1 @ 2,+1
(25b)
d.t1 t (Uaws + Vah(4:) + da)
(25c)
At time 4, the updated state vector is (t.) = 0 C (2a (20:41) 1). Then, c(t) for E
(bi, [++1] given by (26), which continues control h(t) except that has now increased by 1).
c(t)
def C++1 + (c++1 c.4) exp 1, 44)
(26)
On the interval (44, 1. c(t) follows an exponential curve that begins at C++1 (in the sense that lim, c(t) C141) and decays, time increases, toward S,+1 (which it would approach as t + 0o, extrapolated).
A.2 BOUNDARY CONDITIONS
We initialize each node block (0) O, and then have it read special beginning-of-strrene (BOS) event bossig where bos is a special event type and ln is set to be 0. Then equations (24)-25) define C1 (from Cu def 0), C₁, 51, and 01- This is the initial configuration of the system as it waits for the first event to happen: this initial configuration determines the hidden state h(()) and the intensity functions (t) over (0,
The bos event affects every node block but depends on none of them because we do not generate it. When the system is initiated, the following rule is automatically asserted by our program so users don't have to do it by themselves.
affect (bos, X) :- is.bblck (X).
(27)
More details about why bos is desirable can be found in Mei & Eisner (2017).
A.3 CONSTRUCTING THE NEURAL HAWKES PROCESS WITH OUR INTERFACE
The vanilla neural Hawkes process can be specified using our interface as follows:
depend (K, global).
(28a)
affect (K,global).
(28b)
update
(28c)
where hglober (t) is the only node block that every event type k: depends on and affects. Equa-
tion (10) falls back to (v (Ao(BCh (f)))) which is not exactly the same with, yet at least expressive equation (1).
13

---

Under review as a conference paper at ICLR 2020
A.4 OPTIONAL architecture, input AND output KEYWORDS
As discussed in section 3.5, the embedding vector of each event is just the sum of trainable vectors. Actually, we further allow users write Datalog rules to define embedding models that have multi- layer structures and activation functions of interest.
We can define a L-layer neural network using the arccure keyword as: architecture (n)= transform (... (transf orm Du. D1), ...).D,, at). where n (structured) term as the model name, Do is the input dimension, Dj and oj are the output dimension and activation type of (-th layer respectively. The example below defines model named emb that has a neural layer with
(29)
activation followed by a linear layer.
architecture (emb)= transform (transf orm (8, 8, tanh), 8, n,nne)
(30)
Note that we allow using for architecture tecture to indicate that there should be only one model under each name n, although is not supported in the standard datalog implementation.
We can assign to each k model and spell out its arguments Iı, T2...... (to be concatenated in order) for input embedding computation using the input keyword:
input (k)= n (x1, .2.
(31)
and follow the same format for output embedding computation using the output keyword. Note that we use again. The example below means that each w (s is computed by passing the concatenation of and R into model emb and that Venail (S.R) R) is computed the same way:
input (email (S, ,R)) )= emb (S,R).
(32a)
output (email (S, == emb (S,R).
(32b)
B ALGORITHM DETAILS
In this section, we claborate on the details of algorithms.
B.1 LIKELIHOOD COMPUTATION
The ligeliiei in equation (19) can be computed by calling Algorithm 1.
The down sampling trick (line 32 of Algorithm 1) can be used when there are too many event types. It gives an unbiased estimate of the total intensity Eeex (t). yet remains much less computation- ally expensive especially when J < [A]]. In our experiments, we found that its variance over the entire corpus turned out small, although may, theory, suffer large variance. As future work, we will explore sampling from proposal distributions where the probability of choosing any k: is (per- haps trained to be) proportional to its actual intensity (t), in order to further reduce the variance. But this is not within the scope of this paper.
Note that, in principle, we have to make Datalog queries after every event, to figure out which node blocks are affected by that event and to find the intensities of all events. However, certain Datalog queries may be slow. Thus, in practice, rather than repeatedly making the same queries, we just memorize the result the first time and look it up when it is needed again.
Problems emerge when events are allowed to change the database (e.g. asserting and retracting facts in Appendix D), then this may change the results of some queries, and thus the memos for those queries are now incorrect. In this case, we might explore using some other more flexible query language that creates memos and keeps them up date (Filardo & Eisner, 2012).
B.2 THINNING ALGORITHM FOR SAMPLING SEQUENCES
Given an event sequence prefix 1 11 kl 1 # we can call Algorithm 2 to draw the single next event. A full sequence can be rolled out by repeatedly feeding the sampled event back into the model and then drawing the next (calling Algorithm 2 another time).
How do we construct the upper bound X* (line 8 of Algorithm 2)? We express the upper bound as X" = ΣtEK X and find >g 2 > for each k. We copy the formulation of (t) here for easy reference:
(t) fa(V o A b,
14

---

Under review as a conference paper at ICLR 2020
Algorithm Log-likelihood Computation
Input:
observed sequence x k1 st1, $1 at, over interval [0, T); datalog interface d and model p; constant C, boolean flag downsample, down-sampling size J
Output: log-likelihood !
1: procedure COMPUTELOGLLKKLLHOOOO, d, p, dounsample,
2: == 0, UPDATE(bOs, 0, p, d)
3: for i to T
loop over the sequence D compute intensity
4:
Av. (1:) & *COMPTTENNTENSTTRRSS
5:
+= log (t4)
sum up log intensity
6:
A COMPUTEINTEORAA
5- CIJ} downsample.
7:
f-=A
8:
UPDATE(K;; 4.P..)
update node blocks
9:
A + COMPUTELNTEGRALCC,.. max(1, T CTJ), downsample, J)
10: { --
11: return &
procedure COMPUTEINTENSIITCC..
compute (t) find node blocks that ls depends on
13:
B% < (k,
14: Ax(t) + fa(vi (S,AA beRk D.C)).).h((t)))
return (t(
16: procedure UPDATE(K,,,
t update node blocks that affects b find node blocks affects
B t affect Bk
18: for in 8
query update (B' ,b,k:)?
D find node blocks that to update its state
20:
use equations (13) and (16) to update ht) Lends N, downsample, J)
21: procedure COMPUTEINTEGRALL
22:
compute integral over interval (Laan, Lond)
A t 0, B B
init total intensity D down sample event types
24: if dounsample:
IK]/J,KE DOWNSAMPLE(K, J)
for
26: draw ~ Unif((stam tend)
27:
for K.:
28:
(1) <- COMPUTEINTENSITYCK...
29:
+= (t)
30: A - (fend Istun BAA/N
31:
return
32:
procedure DOWNSAMPLE(C,
D down sample the types
33:
K' t empty set
34:
for 1 JJ:
35:
uniformly draw W' E K. add k to K'
36:
return K
15

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Thinning Algorithm for Drawing Next Event
Input: model that has read sequence ki k1 - -1 datalog interface d
Output: next event time 4,, next event type k d) 2: ti. (. (Ax(t))kee) < DRRWNEEXTEVENTTIEED.. d)
1: procedure DRAWNEXTEVENTE
3: draw K Λ. where probability of choosing any & is proportional to A (t,)
4: return 11, k;
5: procedure DRAWNEXTEVENTTIMEE
d)
6: the thinning algorithm that draws next event time by rejection sampling
7: 6 li-1
8: find upper bound X 2 EvEK (t) for all E (42-1,00)
9: repeat
> thinning algoritium
10:
draw A Exp(A*), u 11)
11:
++ A compute (t) for EK
time of next proposed event (before thinning) b call the procedure line 12 of Algorithm
12:
13:
until u\* < EKEK )e(())
thinning: accept proposed time only with proh Enci 2g(t)
X
14:
return t, (t)
DATASET
[C]]
# OF EVENT TOKENS
SEQUENCE LENGTH
TRAIN
DEV
TEST
MIN
MEAN
MAX
SYNTHETIC M =
16
42000
2100
2100
21
21
21
SYNTHETIC M =/=6
32
42000
2100
2100
21
21
21
SYNTHETIC M = 16
64
42000
2100
2100
21
21
21
ELEVATOR
313043
31304
31206
235
313
370
EUROEMAIL
4915
483
483
23
48
74
Table 1: Statistics each dataset.
Note that fu and are positive and monotonic functions and that A, are non-negative matrices. Then we can construct X as:
A ( hz)))
\ =
where Vk = ve+ival (i.e. only keep positive elements of v1)) and h, > b,CD for t (ti-1,
We can construct h2 by adapting the recipe in Appendix B.3 of Mei & Eisner (2017). For notation simplicity, we denote B, DD. as G and use ga as its d-th row vector. Then the d-th element of h can be expressed as =d'=1 haar (t) where each summand gdd' hoar (f) = - (2o (2car (t)) 1) is upper-bounded by max.e (eaa Ciat) Idd Old (2o(2c) - 1). Note that the coefficients Jdd may be either positive or negative.
Po
C EXPERIMENTAL DETAILS
C.1 DATASET STATISTICS
Table shows statistics about each dataset that we use in this paper.
C.I.1 SYNTHETIC DATASET DETAILS
We synthesize data by sampling event sequences from different structured processes. Each structured process is mixture model of M neural Hawkes processes and each neural Hawkes process (X) has four event types event1 (x), (x), event3 (x) and event4 (x). We chose M = 4, 16 and end up with three different datasets.
We chose the sequence length I = 21 and then used the thinning algorithm (Lewis & Shedler, 1979; Liniger, 2009; Mei & Eisner, 2017) to sample the first events over [0, oo). We set T t1, i... the time the last generated event. We generate 2000, 100 and 100 sequences for each training, dev, and test set respectively.
16

---

Under review as a conference paper at ICLR 2020
C.1.2 ELEVATOR DATASET DETAILS
We examined our method in simulated 5-floor building with elevator cars. The system was initially built in Fortran by Crites & Barto (1996) and then rebuilt in Python by Mei et al. (2019).
During typical afternoon down-peak rush hour (when passengers go from floor-2,3,4,5 down to the lobby), elevator cars travel to each floor and pick up passengers that have (stochastically) arrived there according
traffic profile that can be found in (Bao et al., 1994) and Mei al. (2019).
In this dataset, each event type is stop (C, F) where C can be car1 and car2 and F can be floor1, floor5. So there are 10 event types total in this simulated building.
We repeated the (one-hour) simulation 1200 times to collect the event sequences, each of which has around 1200 time-stamped records of which car stops at which floor. We randomly sampled disjoint train, dev and test sets with 1000, 100 and 100 sequences respectively.
C.1.3 EUROEMAIL DATASET DETAILS
EuroEmail proposed by Paranjape et al. (2017). It was generated using email data from a large European research institute, and was highly anonymized. The emails only represent communications between institution members, which are indexed by integers, with timestamps. In the dataset are 986 users and 332334 email communications spanning over 800 days. However, most users only send or receive one or two emails, leaving this dataset extremely sparse. We extracted all the emails among the top 20 most active users, and end up with 5881 emails. We split the single long sequence into 120 sequences with average length of 48, and set the training, dev, test size as 100, 10, 10 respectively.
In this dataset, event type defined as send (S, RR),, where and R are members in this organization. Then there re 20 x 20 400 different event types, where we assume that people may send emails to themselves.
C.2 DATALOG PROGRAM DETAILS
In this section, give full Datalog specification of the model that used for the experiments each dataset.
Here is the full program for Elevator domain.
is blocklockkkk (car1).
(33a)
is block (car2)
(33b)
is bs.ckock (floori).
(33c)
is bs.ckock (floor2).
(33d)
is block (floor3).
(33e)
is block (floor4))
(33f)
is bs.block(ffl) (floor5))
(33g)
is block blockbbulddngg
(33h)
is car(carl).
(33i)
is car car(c))).
(33j)
is floor (floor1).
(33k)
is floor (floor2).
(331)
is floor (floor3).
(33m)
is floor (f1oor4).
(33n)
is floor (floor5).
(33o)
is event :- is car(C), is floor (F).
(33p)
() C).
(33q)
(spopd(sto F).
(33r)
()) building).
(33s)
17

---

Under review as a conference paper at ICLR 2020
(stoct(s)) C).
(33t)
affect F).
(33u)
affect affect (sttp((ccF))) building).
(33v)
update (C,,sto) update (F, (F,,,
(33w)
(33x)
Here is the full program for EuroEmail domain.
is.block (user1).
(34a)
(34b)
is.block (user20).
(34c)
is (s.block(global)
(34d)
is.user (useri).
(34e)
(34f)
is_user (user20).
(34g)
is is_event (email (S, (R)) :- is-user (S), is is.user (R).
(34h)
depend (S, R), S).
(34i)
depen (S RR R).
(34j)
(epend(eend (8, R), global).
(34k)
affect (send (S R) S).
(341)
affect (send (S, R), R).
(34m)
affect (send (S R) global). update (8, email (S, RR)) update (R, R, email (S, R)).
(34n)
(34o)
(34p)
C.3 TRAINING DETAILS
On each dataset, given the architecture specified by the Datalog program (Appendix C.2), the hyper- parameter left to tune is the number D of hidden nodes of each node block. For each model and each training size (i.e., each point in Figures 2a-2c and 3), we searched for D that achieves the best performance on the dev set. Our search space is {4. 8. 16. 32, 64, 128).
For learning, we used the Adam algorithm with its default settings (Kingma & Ba, 2015) and set minibatch size 1. We performed early stopping based log-likelihood on the held-out dev set.
D ONGOING AND FUTURE WORK
We currently exploring several extensions to deal with more complex situations.
Interacting with the database. Events may interact with the database by imperatively retracting and asserting certain facts. For example, when travel (bob, noob,,ewyrk, chicago) hap- pens, the fact in (bob newyork) should be retracted from the database but new fact in (bob, chicago) should be inserted it. Such retraction and assertion may be specified by using the retract and assert keywords.
Hard constraints. Certain datalog facts may enforce hard constraints on intensities of certain event types. For example, in (bob, chicago) makes travel (bob, From, chicago) for any From to have structural 0 intensities.
18


Reviews:
[{"review_id": "r1lm7jx-9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review for Temporal Modeling via Logical Specification of Factorial\nLSTMs\n\nThis paper addresses a key problem in machine learning: how to control\nthe inductive bias of a model in an interpretable way.  The paper\ncontributes a Datalog-based language that allows a human to hand-code\nstructural assumptions (typically based on domain knowledge) that are\nautomatically translated into sparsity patterns in the parameter\nmatrices of an ML model (in this case, a neural Hawkes process,\nalthough the idea would [probably] generalize to other cases).  The\nlanguage plus structured-neural-Hawkes process is demonstrated on a\nfew very small problems, with mixed results.\n\nThis paper is borderline.  However, I tend to favor rejection because\nwhile the ideas are very interesting (and potentially impactful),\nvalidation of the claims is weak.\n\nContributions:\n\nOn the positive side:\n\nA Datalog interface to specifying structural zeros in parameter\nmatrices is a good idea.  The language is natural, and the high-level\nmapping from structure and objects to low-level parameters seems\nreasonable and potentially useful.\n\nThe method makes it easier to specify an inductive bias.  This is a\nstep in the right direction; but at its heart, this paper does not do\nanything that couldn't have been done by hand - it only makes it\neasier.\n\nThe method is potentially more interpretable than other attempts at\ncontrolling inductive bias (for example, simple weight regularizaion),\nbut see below for why this might be a red herring.\n\nThe paper is very nicely written.  It's clear that a lot of attention\nto detail went into writing it.  Well done.\n\nWeaknesses:\n\nThere are a few major points to criticize about this paper.\n\nFirst, there is no clear learning or prediction benefit.  The results\nare mixed: while it appears that the SHP learns faster than the\nunstructured HP, they appear to be asymptoting at the same point.\nThis is perhaps to be expected, as the structural zeros introduced by\nthe corresponding Datalog program effectively reduce the parameter\ncount, but the shape of the learning curves is unchanged.\n\n(Also: please include error bars in Fig. 2(a1) and 2(a2))\n\nThe proper comparison would probably be to a low-rank parameter\nmatrix, where the parameter count is similarly reduced, but in an\nunstructured way.  That would allow us to disentangle \"parameter count\nreduction\" from \"inductive bias\", which is currently not done in the\npaper. \n\nThe results in Figure 3c are mixed - it appears that SHP is only\nbetter in 1/4 of the cases; in all other cases, the error bars seem to\nindicate that there is no predictive power.\n\nFinally, I am concerned that the method may give a false sense of\nexplainability to the model - why it is true that a highly structured,\nsymbolic language is being used to craft an inductive bias, there is\nno \"symbol grounding\".  That is, there is no guarantee that the neural\npart of the learning algorithm will use the parameters in the way the\nhuman intended it to, because the parameters are ultimately\ndisconnected from the symbols.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skx3yBR8qH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " Summary: \n\n \nThe paper proposed to use Datalog rules to specify the design of the LSTM architecture for event data in continuous time. The LSTM module will be used to model the rate of the events. By incorporating Datalog rules, the paper aims to encode informed inductive biases into the model. \n\nComments:\n\n    After reading the entire paper, I think the main idea of this paper is to use sparse and structured weight matrices (called structural zeros” in the paper) to substitute the dense weight matrices in the original LSTM, and to split the hidden state into blocks where each block refer to a different world’s state.  \n\n\n         How to design the structured weight matrices and how to define the node blocks, it is informed by the  Datalog rules. This design, however, will lead to a huge weight matrix and a very  long hidden state once the types of events and number of entities grow. The proposed model will face a severe scalability issue. From this point of view, only “structural zeros” weight matrices are not enough for an elegant model.                \n\n \n\n        How to smartly share parameters and how to control the number of parameters will be an interesting direction to explore. This submission touches on this a bit but not in a principled way. For example, in Eq. 18(a) and 18(b), the embedding vectors for the grounded predicate is a summation of the embedding vectors of the entities and the predicates. This final embedding is empirically validated or is based on some permutation invariant property? This needs more clarification or some references.\n\n \n\n \n        2.      The presentation needs to be polished. The current writing is not easy to follow. Especially for section 3. The architecture design needs to be clarified more. When I read this part, I felt a little difficult to map the Datalog rules to your model. \n        3        Since you are learning the vector embeddings for event types and entities, what are the advantages of this compared to the marked point process model, where the event types and entities are treated as discrete markers and are a much more parsimonious model. The Datalog rules can also be defined on the marker level by introducing a structured dependency structure over the markers. What are the potential benefits of learning the embeddings? The explanation is missing in this paper. \n\n       4     Lack of references. The proposed neural-symbolic architecture shares some similarities to the following papers: \n          (1) End-to-End Differentiable Proving \n\n          (2) DeepProbLog: Neural Probabilistic Logic Programming \n\n          (3) Neural Logic Machines.\n          \n        What are your contributions and differences in terms of the neural-symbolic architecture design?\n\n \n       As for introducing logic rules to guide event predication, this is not a new topic. Here is a list of references:\n\n \n         (1) PEL-CNF: Probabilistic event logic conjunctive normal form for video interpretation. \n          (2) A general framework for recognizing complex events in Markov logic. \n\n          (3) Learning Bayesian networks for clinical time series analysis. \n\n          (4) Logical Hierarchical Hidden Markov Models for Modeling User Activities. \n\n          (5) Slice Normalized Dynamic Markov Logic Networks. \n       5.         Lack of strong baselines. The paper only did a small-scale experiment study. It only compares a neural Hawkes process model. The experimental evaluation also needs stronger baselines. Specifically, methods that can handle continuous-time (e.g. marked point process) or probabilistic logic methods that can discretize time (as mentioned in the above references). The baselines are not quite strong and appear a bit arbitrary in the paper.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxQnlqy9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds an interesting connection between Datalog rules and temporal point processes. The novelty of the approach is to factorize the latent state of LSTM into different blocks that represent three major interactions between temporal events, including: dependency, affects, and updates. The design of the node blocks within the hidden state allows the modeling of fine-grain structure of a given event type. Based on the Datalog program and the logic rules, the intensity function of the temporal point process can be formulated from facts in a database. The problem of enabling a flexible family of intensity functions is one of the most important topics in point processes, and a paper advancing knowledge in this area is certainly welcome.\n\nThe paper is in general well written. Section 2.2 can be more clarified by explicitly comparing the concepts of blocks and entities using \"mind(Alice)\" and \"body(Alice)\" before introducing the hidden state h_mind(Alice)(t). It took me some time going back and forth to understand these examples here. With respect to the design of the Datalog interface, it looks like it covers the assertion involving two arguments. Since these arguments affect the partition of the number of node blocks, it would be more clear to illustrate how to design the node blocks as the number of arguments increases (say beyond 2 arguments). In fact, if we know the number of entities in each event type, say the number of node blocks to partition is 3 per hidden state in advance, we can leverage three separate small LSTMs each of which has the private hidden state with the same number of nodes as that in one of the node blocks. Then, we can determine the interactions among these separate small LSTMs based on the logic rules, so it will be helpful to elucidate the additional advantages of partitioning these node blocks in the same hidden state. The proposed technique mainly considers how to incorporate the block design into the LSTM hidden states as a general sequence model. What is the unique characteristics of Neural Hawkes Process have been particularly exploited from this perspective? It looks like it can be applied to other LSTM-based approach as long as the predictions are functions of the hidden states. For the synthetic experiments, it is obvious that single Neural Hawkes process has more challenges to fit the mixture of processes. It will be more convincing to compare with a mixture model, like \"A Dirichlet Mixture Model of Hawkes Processes for Event Sequence Clustering\" with the proposed approach, and the same as in the real experiments. Also, a standard test-of-goodness fit like QQ-plot will also be more useful to improve the experiments.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While reviewers find this paper interesting, they raised number of concerns including the novelty, writing, experiments, references and clear mention of the benefit. Unfortunately, excellent questions and insightful comments left by reviewers are gone without authors’ answers. 
Decision: Reject